Changes between Version 2 and Version 3 of BadBlockHowto

Timestamp:

Mar 25, 2017, 1:45:01 PM (7 years ago)

Author:

Gabriele Pohl

Comment:

ext2/ext3 first example

BadBlockHowto

@@ -22 +22 @@
 The authors would like to thank Sergey Vlasov, Theodore Ts'o, Michael Bendzick, and others for explaining this approach. The authors would like to add text showing how to do this for other file systems, in particular XFS, and JFS: please email if you can provide this information. 
 
+=== ext2/ext3 first example ===
+
+In this example, the disk is failing self-tests at Logical Block Address `LBA = 0x016561e9 = 23421417`. The LBA counts sectors in units of 512 bytes, and starts at zero.
+
+{{{
+root]# smartctl -l selftest /dev/hda:
+SMART Self-test log structure revision number 1
+Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
+# 1  Extended offline    Completed: read failure       90%       217         0x016561e9
+}}}
+
+Note that other signs that there is a bad sector on the disk can be found in the non-zero value of the `Current_Pending_Sector` count:
+
+{{{
+root]# smartctl -A /dev/hda
+ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
+  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
+196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
+197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       1
+198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       1
+}}}
+
+First Step: We need to locate the partition on which this sector of the disk lives:
+
+{{{
+root]# fdisk -lu /dev/hda
+Disk /dev/hda: 123.5 GB, 123522416640 bytes
+255 heads, 63 sectors/track, 15017 cylinders, total 241254720 sectors
+Units = sectors of 1 * 512 = 512 bytes
+   Device Boot    Start       End    Blocks   Id  System
+/dev/hda1   *        63   4209029   2104483+  83  Linux
+/dev/hda2       4209030   5269319    530145   82  Linux swap
+/dev/hda3       5269320 238227884 116479282+  83  Linux
+/dev/hda4     238227885 241248104   1510110   83  Linux
+}}}
+
+The partition `/dev/hda3` starts at `LBA 5269320` and extends past the ''problem'' LBA. The ''problem'' LBA is offset `23421417 - 5269320` = `18152097` sectors into the partition `/dev/hda3`.
+
+To verify the type of the file system and the mount point, look in `/etc/fstab`:
+
+{{{
+root]# grep hda3 /etc/fstab
+/dev/hda3 /data ext2 defaults 1 2
+}}}
+
+You can see that this is an `ext2` file system, mounted at `/data`.
+
+Second Step: we need to find the block size of the file system (normally 4096 bytes for `ext2`):
+
+{{{
+root]# tune2fs -l /dev/hda3 | grep Block
+Block count:              29119820
+Block size:               4096
+}}}
+
+In this case the block size is 4096 bytes. Third Step: we need to determine which File System Block contains this LBA. The formula is:
+
+{{{
+  b = (int)((L-S)*512/B)
+where:
+b = File System block number
+B = File system block size in bytes
+L = LBA of bad sector
+S = Starting sector of partition as shown by fdisk -lu
+and (int) denotes the integer part.
+}}}
+
+In our example, `L=23421417`, `S=5269320`, and `B=4096`. Hence the ''problem'' LBA is in block number
+
+{{{
+   b = (int)18152097*512/4096 = (int)2269012.125
+so b=2269012.
+}}}
+
+Note: the fractional part of `0.125` indicates that this problem LBA is actually the second of the eight sectors that make up this file system block.
+
+Fourth Step: we use `debugfs` to locate the inode stored in this block, and the file that contains that inode:
+
+{{{
+root]# debugfs
+debugfs 1.32 (09-Nov-2002)
+debugfs:  open /dev/hda3
+debugfs:  testb 2269012
+Block 2269012 not in use
+}}}
+
+If the block is not in use, as in the above example, then you can skip the rest of this step and go ahead to Step Five.
+
+If, on the other hand, the block is in use, we want to identify the file that uses it:
+
+{{{
+debugfs:  testb 2269012
+Block 2269012 marked in use
+debugfs:  icheck 2269012
+Block   Inode number
+2269012 41032
+debugfs:  ncheck 41032
+Inode   Pathname
+41032   /S1/R/H/714197568-714203359/H-R-714202192-16.gwf
+}}}
+
+In this example, you can see that the problematic file (with the mount point included in the path) is: `/data/S1/R/H/714197568-714203359/H-R-714202192-16.gwf`
+
+When we are working with an `ext3` file system, it may happen that the affected file is the journal itself. Generally, if this is the case, the inode number will be very small. In any case, `debugfs` will not be able to get the file name:
+
+{{{
+debugfs:  testb 2269012
+Block 2269012 marked in use
+debugfs:  icheck 2269012
+Block   Inode number
+2269012 8
+debugfs:  ncheck 8
+Inode   Pathname
+debugfs:
+}}}
+
+To get around this situation, we can remove the journal altogether:
+
+{{{
+tune2fs -O ^has_journal /dev/hda3
+}}}
+
+and then start again with Step Four: we should see this time that the wrong block is not in use any more. If we removed the journal file, at the end of the whole procedure we should remember to rebuild it:
+
+{{{
+tune2fs -j /dev/hda3
+}}}
+
+Fifth Step NOTE: '''This last step will permanently and irretrievably destroy the contents of the file system block that is damaged''': if the block was allocated to a file, some of the data that is in this file is going to be overwritten with zeros. You will not be able to recover that data unless you can replace the file with a fresh or correct version.
+
+To force the disk to reallocate this bad block we'll write zeros to the bad block, and sync the disk:
+
+{{{
+root]# dd if=/dev/zero of=/dev/hda3 bs=4096 count=1 seek=2269012
+root]# sync
+}}}
+
+Now everything is back to normal: the sector has been reallocated. Compare the output just below to similar output near the top of this article:
+
+{{{
+root]# smartctl -A /dev/hda
+ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
+  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       1
+196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       1
+197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
+198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       1
+}}}
+
+Note: for some disks it may be necessary to update the SMART Attribute values by using `smartctl -t offline /dev/hda`
+
+We have corrected the first errored block. If more than one blocks were errored, we should repeat all the steps for the subsequent ones. After we do that, the disk will pass its self-tests again:
+
+{{{
+root]# smartctl -t long /dev/hda  [wait until test completes, then]
+root]# smartctl -l selftest /dev/hda
+SMART Self-test log structure revision number 1
+Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
+# 1  Extended offline    Completed without error       00%       239         -
+# 2  Extended offline    Completed: read failure       90%       217         0x016561e9
+# 3  Extended offline    Completed: read failure       90%       212         0x016561e9
+# 4  Extended offline    Completed: read failure       90%       181         0x016561e9
+# 5  Extended offline    Completed without error       00%        14         -
+# 6  Extended offline    Completed without error       00%         4         -
+}}}
+
+and no longer shows any offline uncorrectable sectors:
+
+{{{
+root]# smartctl -A /dev/hda
+ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
+  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       1
+196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       1
+197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
+198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
+}}}
+
+== Footnotes ==
+
 [=#footnote1 [1]] Self-Monitoring, Analysis and Reporting Technology -> SMART