|
I first started searching for this error message: Currently unreadable pending sectors
I found a link to Bad block HOWTO for smartmontools.
I read it and decided I needed to grab more information from smartd.
[root@bast:/home/dan] # smartctl -l selftest /dev/ad2
smartctl 5.39 2009-12-09 r2995 [FreeBSD 8.2-STABLE i386] (local build)
Copyright (C) 2002-9 by Bruce Allen, http://smartmontools.sourceforge.net
=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 14921 -
# 2 Extended offline Completed without error 00% 14920 -
# 3 Short offline Completed without error 00% 6914 -
# 4 Short offline Completed without error 00% 6914 -
That shows not much at all. Just that previous tests have run fine. Let's go for the big picture.
[root@bast:/home/dan] # smartctl -a /dev/ad2
smartctl 5.39 2009-12-09 r2995 [FreeBSD 8.2-STABLE i386] (local build)
Copyright (C) 2002-9 by Bruce Allen, http://smartmontools.sourceforge.net
=== START OF INFORMATION SECTION ===
Model Family: Maxtor DiamondMax Plus D740X family
Device Model: MAXTOR 6L040J2
Serial Number: 362129580341
Firmware Version: A93.0500
User Capacity: 40,027,029,504 bytes
Device is: In smartctl database [for details use: -P show]
ATA Version is: 5
ATA Standard is: ATA/ATAPI-5 T13 1321D revision 1
Local Time is: Fri Aug 19 16:36:40 2011 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 34) seconds.
Offline data collection
capabilities: (0x1b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
No Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
No General Purpose Logging support.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 20) minutes.
SMART Attributes Data Structure revision number: 11
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x0029 100 253 020 Pre-fail Offline - 0
3 Spin_Up_Time 0x0027 081 081 020 Pre-fail Always - 2437
4 Start_Stop_Count 0x0032 100 100 008 Old_age Always - 154
5 Reallocated_Sector_Ct 0x0033 100 100 020 Pre-fail Always - 1
7 Seek_Error_Rate 0x000b 100 093 023 Pre-fail Always - 0
9 Power_On_Hours 0x0012 059 059 001 Old_age Always - 27416
10 Spin_Retry_Count 0x0026 100 100 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0013 100 100 020 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 008 Old_age Always - 144
13 Read_Soft_Error_Rate 0x000b 100 093 023 Pre-fail Always - 0
194 Temperature_Celsius 0x0022 080 077 042 Old_age Always - 51
195 Hardware_ECC_Recovered 0x001a 100 001 000 Old_age Always - 89569576
196 Reallocated_Event_Count 0x0010 100 100 020 Old_age Offline - 0
197 Current_Pending_Sector 0x0032 100 100 020 Old_age Always - 1
198 Offline_Uncorrectable 0x0010 100 253 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x001a 200 200 000 Old_age Always - 0
SMART Error Log Version: 1
ATA Error Count: 1
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.
Error 1 occurred at disk power-on lifetime: 27400 hours (1141 days + 16 hours)
When the command that caused the error occurred, the device was in an unknown state.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 59 18 e8 ef 54 e0 Error: UNC 24 sectors at LBA = 0x0054efe8 = 5566440
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
c8 00 00 00 ef 54 e0 54 00:03:49.769 READ DMA
c8 00 00 00 ee 54 e0 54 00:03:49.765 READ DMA
c8 00 00 00 ed 54 e0 54 00:03:49.755 READ DMA
c8 00 00 00 ec 54 e0 54 00:03:49.751 READ DMA
c8 00 00 00 eb 54 e0 54 00:03:49.747 READ DMA
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 14921 -
# 2 Extended offline Completed without error 00% 14920 -
# 3 Short offline Completed without error 00% 6914 -
# 4 Short offline Completed without error 00% 6914 -
Device does not support Selective Self Tests/Logging
Next, I decided to run a short test.
[root@bast:/home/dan] # smartctl -t short /dev/ad2
smartctl 5.39 2009-12-09 r2995 [FreeBSD 8.2-STABLE i386] (local build)
Copyright (C) 2002-9 by Bruce Allen, http://smartmontools.sourceforge.net
=== START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===
Sending command: "Execute SMART Short self-test routine immediately in off-line mode".
Drive command "Execute SMART Short self-test routine immediately in off-line mode" successful.
Testing has begun.
Please wait 2 minutes for test to complete.
Test will complete after Fri Aug 19 16:47:16 2011
Use smartctl -X to abort test.
I wanted two minutes, and then I saw:
[root@bast:/home/dan] # smartctl -l selftest /dev/ad2
smartctl 5.39 2009-12-09 r2995 [FreeBSD 8.2-STABLE i386] (local build)
Copyright (C) 2002-9 by Bruce Allen, http://smartmontools.sourceforge.net
=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 27416 -
# 2 Extended offline Completed without error 00% 14921 -
# 3 Extended offline Completed without error 00% 14920 -
# 4 Short offline Completed without error 00% 6914 -
# 5 Short offline Completed without error 00% 6914 -
Hmmm, still no useful information. Let's try the long test.
root@bast:/home/dan] # smartctl -t long /dev/ad2
smartctl 5.39 2009-12-09 r2995 [FreeBSD 8.2-STABLE i386] (local build)
Copyright (C) 2002-9 by Bruce Allen, http://smartmontools.sourceforge.net
=== START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===
Sending command: "Execute SMART Extended self-test routine immediately in off-line mode".
Drive command "Execute SMART Extended self-test routine immediately in off-line mode" successful.
Testing has begun.
Please wait 20 minutes for test to complete.
Test will complete after Fri Aug 19 17:08:04 2011
Use smartctl -X to abort test.
After 20 minutes, I found:
[root@bast:/home/dan] # smartctl -l selftest /dev/ad2
smartctl 5.39 2009-12-09 r2995 [FreeBSD 8.2-STABLE i386] (local build)
Copyright (C) 2002-9 by Bruce Allen, http://smartmontools.sourceforge.net
=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed: read failure 90% 27416 786767
# 2 Short offline Completed without error 00% 27416 -
# 3 Extended offline Completed without error 00% 14921 -
# 4 Extended offline Completed without error 00% 14920 -
# 5 Short offline Completed without error 00% 6914 -
# 6 Short offline Completed without error 00% 6914 -
Now that's some information we can use in our fix. The LBA of the [first]
bad sector is 786767. How can we use this information?
Let's look at the output of gpart:
[dan@bast:~] $ gpart show
=> 63 78177708 mirror/gm0 MBR (37G)
63 78172227 1 freebsd [active] (37G)
78172290 5481 - free - (2.7M)
=> 0 78172227 mirror/gm0s1 BSD (37G)
0 1048576 1 freebsd-ufs (512M)
1048576 4141712 2 freebsd-swap (2.0G)
5190288 4167680 4 freebsd-ufs (2.0G)
9357968 1048576 5 freebsd-ufs (512M)
10406544 67765683 6 freebsd-ufs (32G)
Please note that /dev/ad2 is part of the gmirror mirror/gm0. I'm told that gmirror LBAs are used
unchanged. Let's proceed based on that perhaps dangerous assumption.
It looks like this LBA is in the first 512M of the HDD. Which is part of / I'm sure. Let's see
$ df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/mirror/gm0s1a 496M 405M 51M 89% /
devfs 1.0K 1.0K 0B 100% /dev
/dev/mirror/gm0s1e 496M 61M 395M 13% /tmp
/dev/mirror/gm0s1f 31G 6.7G 22G 23% /usr
/dev/mirror/gm0s1d 1.9G 513M 1.3G 28% /var
ngaio:/usr/ports/distfiles 138G 100G 27G 79% /usr/ports/distfiles
devfs 1.0K 1.0K 0B 100% /var/named/dev
devfs 1.0K 1.0K 0B 100% /var/db/dhcpd/dev
Yep, that looks like it's in / for sure.
Let's find out sector size:
[root@bast:/home/dan] # diskinfo -v ad2
ad2
512 # sectorsize
40027029504 # mediasize in bytes (37G)
78177792 # mediasize in sectors
0 # stripesize
0 # stripeoffset
19161 # Cylinders according to firmware.
16 # Heads according to firmware.
255 # Sectors according to firmware.
362129580341 # Disk ident.
And now for some partition information:
# fdisk -t /dev/ad2
******* Working on device /dev/ad2 *******
parameters extracted from in-core disklabel are:
cylinders=19161 heads=16 sectors/track=255 (4080 blks/cyl)
Figures below won't work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=19161 heads=16 sectors/track=255 (4080 blks/cyl)
Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
start 63, size 78172227 (38170 Meg), flag 80 (active)
beg: cyl 0/ head 1/ sector 1;
end: cyl 1023/ head 254/ sector 63
The data for partition 2 is:
The data for partition 3 is:
The data for partition 4 is:
But wait, as Wyze pointed out on IRC, fixed ad2 doesn't make sense. Instead, let's be sure
ad0 is solid with no errors.
[root@bast:/home/dan] # smartctl -t long /dev/ad0
smartctl 5.39 2009-12-09 r2995 [FreeBSD 8.2-STABLE i386] (local build)
Copyright (C) 2002-9 by Bruce Allen, http://smartmontools.sourceforge.net
=== START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===
Sending command: "Execute SMART Extended self-test routine immediately in off-line mode".
Drive command "Execute SMART Extended self-test routine immediately in off-line mode" successful.
Testing has begun.
Please wait 20 minutes for test to complete.
Test will complete after Fri Aug 19 20:14:56 2011
Use smartctl -X to abort test.
And some time later, I had my answer: all OK
[root@bast:/home/dan] # smartctl -l selftest /dev/ad0
smartctl 5.39 2009-12-09 r2995 [FreeBSD 8.2-STABLE i386] (local build)
Copyright (C) 2002-9 by Bruce Allen, http://smartmontools.sourceforge.net
=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 51193 -
# 2 Extended offline Completed without error 00% 38670 -
# 3 Short offline Completed without error 00% 38670 -
But I figured I wanted to do more. I wanted to read all of ad0, the HDD which seems to be OK:
[root@bast:/home/dan] # dd of=/dev/null if=/dev/ad0 bs=1m
38172+1 records in
38172+1 records out
40027029504 bytes transferred in 1157.080130 secs (34593135 bytes/sec)
[root@bast:/home/dan] #
No errors appeared in /var/log/messages during this time. I'm pretty confident and ad0 is solid and
reliable.
Now, let's try ad2:
[root@bast:~] # dd of=/dev/null if=/dev/ad2 bs=1m
dd: /dev/ad2: Input/output error
2717+0 records in
2717+0 records out
2848980992 bytes transferred in 127.232046 secs (22392008 bytes/sec)
[root@bast:~] #
Oh. And in /var/log/messages, I see:
Aug 19 22:15:50 bast kernel: ad2: FAILURE - READ_DMA status=51<READY,DSC,ERROR> error=40<UNCORRECTABLE> LBA=5566208
But compared to ad0, there is much less data... that's because dd stopped on the error. Let us try this:
[root@bast:~] # dd of=/dev/null if=/dev/ad2 bs=1m conv=noerror
dd: /dev/ad2: Input/output error
2717+0 records in
2717+0 records out
2848980992 bytes transferred in 127.128503 secs (22410246 bytes/sec)
dd: /dev/ad2: Input/output error
38170+1 records in
38170+1 records out
40025063424 bytes transferred in 1544.671423 secs (25911701 bytes/sec)
[root@bast:~] #
But still, only one error in /var/log/messages:
Aug 19 22:26:39 bast kernel: ad2: FAILURE - READ_DMA status=51<READY,DSC,ERROR> error=40<UNCORRECTABLE> LBA=5566208
But I also noticed this:
Aug 19 22:36:34 bast smartd[1575]: Device: /dev/ad2, 2 Currently unreadable (pending) sectors
Aug 19 22:36:34 bast smartd[1575]: Device: /dev/ad2, ATA error count increased from 1 to 3
At this point, I gave up for the day and sent an email off to the freebsd-stable mailing list.
| 