Changes between Version 3 and Version 4 of BadBlockHowto

Timestamp:

Mar 25, 2017, 1:55:25 PM (8 years ago)

Author:

Gabriele Pohl

Comment:

ext2/ext3 second example

BadBlockHowto

@@ -3 +3 @@
 This article describes what actions might be taken when smartmontools detects a bad block on a disk. It demonstrates how to identify the file associated with an unreadable disk sector, and how to force that sector to reallocate. 
 
-[[PageOutline(2,Table of Contents, inline)]]
+[[PageOutline(2-3,Table of Contents, inline, unnumbered)]]
 
 
@@ -198 +198 @@
 }}}
 
+=== ext2/ext3 second example ===
+
+On this drive, the first sign of trouble was this email from `smartd`:
+
+{{{
+    To: ballen
+    Subject: SMART error (selftest) detected on host: medusa-slave166.medusa.phys.uwm.edu
+    This email was generated by the smartd daemon running on host:
+    medusa-slave166.medusa.phys.uwm.edu in the domain: master001-nis
+    The following warning/error was logged by the smartd daemon:
+    Device: /dev/hda, Self-Test Log error count increased from 0 to 1
+}}}
+
+Running `smartctl -a /dev/hda` confirmed the problem:
+
+{{{
+Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
+# 1  Extended offline    Completed: read failure       80%       682         0x021d9f44
+Note that the failing LBA reported is 0x021d9f44 (base 16) = 35495748 (base 10)
+ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
+  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
+196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
+197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       3
+198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       3
+}}}
+
+and one can see above that there are 3 sectors on the list of pending sectors that the disk can't read but would like to reallocate.
+
+The device also shows errors in the SMART error log:
+
+{{{
+Error 212 occurred at disk power-on lifetime: 690 hours
+  After command completion occurred, registers were:
+  ER ST SC SN CL CH DH
+  -- -- -- -- -- -- --
+  40 51 12 46 9f 1d e2  Error: UNC 18 sectors at LBA = 0x021d9f46 = 35495750
+  Commands leading to the command that caused the error were:
+  CR FR SC SN CL CH DH DC   Timestamp  Command/Feature_Name
+  -- -- -- -- -- -- -- --   ---------  --------------------
+  25 00 12 46 9f 1d e0 00 2485545.000  READ DMA EXT
+}}}
+
+Signs of trouble at this LBA may also be found in `SYSLOG`:
+
+{{{
+[root]# grep LBA /var/log/messages | awk '{print $12}' | sort | uniq
+ LBAsect=35495748
+ LBAsect=35495750
+}}}
+
+So I decide to do a quick check to see how many bad sectors there really are. Using the `bash` shell I check 70 sectors around the trouble area:
+
+{{{
+[root]# export i=35495730
+[root]# while [ $i -lt 35495800 ]
+        > do echo $i
+        > dd if=/dev/hda of=/dev/null bs=512 count=1 skip=$i
+        > let i+=1
+        > done
+<SNIP>
+35495734
+1+0 records in
+1+0 records out
+35495735
+dd: reading `/dev/hda': Input/output error
+0+0 records in
+0+0 records out
+<SNIP>
+35495751
+dd: reading `/dev/hda': Input/output error
+0+0 records in
+0+0 records out
+35495752
+1+0 records in
+1+0 records out
+<SNIP>
+}}}
+
+which shows that the seventeen sectors `35495735-35495751` (inclusive) are not readable.
+
+Next, we identify the files at those locations. The partitioning information on this disk is identical to the first example above, and as in that case the problem sectors are on the third partition `/dev/hda3`. So we have:
+
+{{{
+     L=35495735 to 35495751
+     S=5269320
+     B=4096
+}}}
+
+so that `b=3778301` to `3778303` are the three bad blocks in the file system.
+
+{{{
+[root]# debugfs
+debugfs 1.32 (09-Nov-2002)
+debugfs:  open /dev/hda3
+debugfs:  icheck 3778301
+Block   Inode number
+3778301 45192
+debugfs:  icheck 3778302
+Block   Inode number
+3778302 45192
+debugfs:  icheck 3778303
+Block   Inode number
+3778303 45192
+debugfs:  ncheck 45192
+Inode   Pathname
+45192   /S1/R/H/714979488-714985279/H-R-714979984-16.gwf
+debugfs:  quit
+}}}
+
+Note that the first few steps of this procedure could also be done with a single command, which is very helpful if there are many bad blocks (thanks to Danie Marais for pointing this out):
+
+{{{
+debugfs: icheck 3778301 3778302 3778303
+}}}
+
+And finally, just to confirm that this is really the damaged file:
+
+{{{
+[root]# md5sum /data/S1/R/H/714979488-714985279/H-R-714979984-16.gwf
+md5sum: /data/S1/R/H/714979488-714985279/H-R-714979984-16.gwf: Input/output error
+}}}
+
+Finally we force the disk to reallocate the three bad blocks:
+
+{{{
+[root]# dd if=/dev/zero of=/dev/hda3 bs=4096 count=3 seek=3778301
+[root]# sync
+}}}
+
+We could also probably use:
+
+{{{
+[root]# dd if=/dev/zero of=/dev/hda bs=512 count=17 seek=35495735
+}}}
+
+At this point we now have:
+
+{{{
+ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
+  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
+196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
+197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
+198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
+}}}
+
+which is encouraging, since the pending sectors count is now zero. Note that the drive reallocation count has not yet increased: the drive may now have confidence in these sectors and have decided not to reallocate them..
+
+A device self test:
+
+{{{
+  [root#] smartctl -t long /dev/hda
+(then wait about an hour) shows no unreadable sectors or errors:
+Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
+# 1  Extended offline    Completed without error       00%       692         -
+# 2  Extended offline    Completed: read failure       80%       682         0x021d9f44
+}}}
+
 == Footnotes ==