Dan, Richard, Jonathan, Carlos, Dave:
h1fw0 is back online writing frames. Dan configured E18-0 POD-1 to rebuild its raid while it is being used by the frame writer.
Attached E18 log shows that the problem started at 21:16 PST Mon night, with a repeat at 21:29 and then a disk failure at 05:47 today, at which point I suspect the audible alarm sounded. Problem then became critical at 08:42 with a pod failure, and the frame writer stopped running at 08:44 (all times PST).
Attached E18 status screen shows POD-1 disk3 and disk2 are actively being rebuilt (green bar is animated growing left-to right), the other disks of POD1 are flashing green-yellow showing they belong to a degraded raid.
Frames gap on hfw0:
-rw-r--r-- 1 controls controls 1.6G Feb 19 08:46 /frames-0/frames/full/12346/H-H1_R-1234629632-64.gwf
-rw-r--r-- 1 controls controls 1.6G Feb 19 11:16 /frames-0/frames/full/12346/H-H1_R-1234629952-64.gwf
FRS12353 was reopened following new developments.
Dan sent our E18-0 logs to the manufacturer (Nexsan) who responded with a recommendation that we 1) upgrade the firmware in the controllers and 2) reseat the controllers.
WP8093 was opened to cover these operations. In staggered sequence, Dan upgraded the firmware and rebooted both controllers. He then removed and reseated the controllers. Controller-0 had no issues, but when controller-1 was reseated it failed. We are hoping the original problem was due to a flakey controller-1, which only came to a head when the card was hard rebooted.
Dan is in contact with Nexsan for a replacement controller card to be shipped to LHO, in the mean time we will operate h1fw0's raid with no redundant controller.
Throughout these operations h1fw0 continues to write frames with no interruption.
I'll extend WP8093 to cover the controller replacement.
Image of E18-0 status showing failed controller-1
