jonathan.hanks@LIGO.ORG - posted 13:04, Monday 04 March 2019 - last comment - 14:22, Monday 04 March 2019(47273)
Disk failure on h1tw3
H1tw3 (the new tw) is reporting a failed disk.
[530922.324644] sd 0:0:3:0: [sdd] Unhandled error code [530922.324649] sd 0:0:3:0: [sdd] [530922.324653] Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK [530922.324657] sd 0:0:3:0: [sdd] CDB: [530922.324661] Read(10): 28 00 3d 15 ab 68 00 00 80 00 [530922.324674] end_request: I/O error, dev sdd, sector 1024830312 [530922.833251] md/raid:md2: read error corrected (8 sectors at 1024830312 on sdd) [530922.833260] md/raid:md2: read error corrected (8 sectors at 1024830320 on sdd) [530922.833264] md/raid:md2: read error corrected (8 sectors at 1024830328 on sdd) [530922.833268] md/raid:md2: read error corrected (8 sectors at 1024830336 on sdd) [530922.833270] md/raid:md2: read error corrected (8 sectors at 1024830344 on sdd) [530922.833273] md/raid:md2: read error corrected (8 sectors at 1024830352 on sdd) [530922.833277] md/raid:md2: read error corrected (8 sectors at 1024830360 on sdd) [530922.833280] md/raid:md2: read error corrected (8 sectors at 1024830368 on sdd) [530922.833282] md/raid:md2: read error corrected (8 sectors at 1024830376 on sdd) [530922.833287] md/raid:md2: read error corrected (8 sectors at 1024830384 on sdd) [2037012.740535] md: data-check of RAID array md0 [2037012.740541] md: minimum _guaranteed_ speed: 1000 KB/sec/disk. [2037012.740544] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check. [2037012.740551] md: using 128k window, over a total of 1950720k. [2037012.772561] md: delaying data-check of md1 until md0 has finished (they share one or more physical units) [2037012.786333] RAID conf printout: [2037012.786339] --- level:6 rd:6 wd:6 [2037012.786353] disk 0, o:1, dev:sda [2037012.786356] disk 1, o:1, dev:sdb [2037012.786358] disk 2, o:1, dev:sdc [2037012.786360] disk 3, o:1, dev:sdd [2037012.786362] disk 4, o:1, dev:sde [2037012.786364] disk 5, o:1, dev:sdf [2037012.786414] md: data-check of RAID array md2 [2037012.786418] md: minimum _guaranteed_ speed: 1000 KB/sec/disk. [2037012.786421] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check. [2037012.786430] md: using 128k window, over a total of 512674304k. [2037022.277393] md: md0: data-check done. [2037022.281423] md: data-check of RAID array md1 [2037022.281427] md: minimum _guaranteed_ speed: 1000 KB/sec/disk. [2037022.281429] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check. [2037022.281434] md: using 128k window, over a total of 242114560k. [2038242.597751] md: md1: data-check done. [2041160.323778] md: md2: data-check done. [2041160.335370] RAID conf printout: [2041160.335391] --- level:6 rd:6 wd:6 [2041160.335393] disk 0, o:1, dev:sda [2041160.335395] disk 1, o:1, dev:sdb [2041160.335397] disk 2, o:1, dev:sdc [2041160.335398] disk 3, o:1, dev:sdd [2041160.335400] disk 4, o:1, dev:sde [2041160.335401] disk 5, o:1, dev:sdf
To aid in finding it:
ata-Crucial_CT525MX300SSD1_1710162E56E0 -> ../../sdd
I'll check with Dave but we will probably take care of this during the daq downtime tomorrow.
/dev/sdd was not marked by the raid as failed. But with the errors in dmesg, we are going to pull it out and take a look at it. I have added the warm spare to the raid array, marked /dev/sdd as failed. We will pull the disk tomorrow when the daq is down so that we can test it out of line.
Status:mdadm --query --detail /dev/md2 /dev/md2: Version : 1.2 Creation Time : Wed Feb 6 12:08:06 2019 Raid Level : raid6 Array Size : 2050697216 (1955.70 GiB 2099.91 GB) Used Dev Size : 512674304 (488.92 GiB 524.98 GB) Raid Devices : 6 Total Devices : 8 Persistence : Superblock is persistent Intent Bitmap : Internal Update Time : Mon Mar 4 14:13:49 2019 State : clean, degraded, recovering Active Devices : 5 Working Devices : 7 Failed Devices : 1 Spare Devices : 2 Layout : left-symmetric Chunk Size : 512K Rebuild Status : 0% complete Name : h1tw3:2 (local to host h1tw3) UUID : 75304e90:3eaee815:04060b83:a8133c8b Events : 945 Number Major Minor RaidDevice State 0 8 0 0 active sync /dev/sda 1 8 16 1 active sync /dev/sdb 2 8 32 2 active sync /dev/sdc 7 8 224 3 spare rebuilding /dev/sdo 4 8 64 4 active sync /dev/sde 5 8 80 5 active sync /dev/sdf 3 8 48 - faulty /dev/sdd 6 8 96 - spare /dev/sdgAs noted above, I marked sdd as failed, not the raid system.