At 21:13:58 we had a crash of h1susb123 (ITMX, ITMY, BS) which in turn caused a SWWD DACKILL of the seismic front ends for these chambers.
Unlike the h1susb123 crash Sunday 11 May 2025 in which lscpi was slow but showed good cards, in this case lspci is responsive and shows a possible problem with the 2nd 18bit-DAC card.
Like the Sunday crash, dmesg does not give much information (listing below).
First thing to try would be a power cycle of the IO Chassis to see if the 18bit DAC is OK. Given the lateness of the hour we could delay this until the morning.
[Tue May 20 21:13:56 2025] rts_cpu_isolator: LIGO code is done, calling regular shutdown code
[Tue May 20 21:13:56 2025] h1iopsusb123: ERROR - An ADC timeout error has been detected, waiting for an exit signal.
[Tue May 20 21:13:57 2025] h1susitmpi: ERROR - An ADC timeout error has been detected, waiting for an exit signal.
[Tue May 20 21:13:57 2025] h1susbs: ERROR - An ADC timeout error has been detected, waiting for an exit signal.
[Tue May 20 21:13:57 2025] h1susitmy: ERROR - An ADC timeout error has been detected, waiting for an exit signal.
[Tue May 20 21:13:57 2025] h1susitmx: ERROR - An ADC timeout error has been detected, waiting for an exit signal.
To get the DAC drives running again on h1seib[1,2,3] I've bypassed the SWWDs for these front ends.
Power Cycle Computer ----------------------------------------------
EJ, Dave:
First thing to try was a power cycle of the front end computer.
07:26 Fence h1susb123 from Dolphin fabric, stop models, power down cpu
07:28 Power h1susb123 back up using IPMI
At this point lspci is reporting the 2nd 18bit-DAC slot as empty, as expected IOP model stops with insufficient cards.
So, looking like bad slot or bad DAC card.
Power Cycle Computer And IO Chassis -----------------------------------------------
EJ, Jonathan, Dave:
Second thing to try, power down computer as above, then power cycle the IO Chassis for about 30 seconds.
Result: the same, lspci does not show anything in the slot the 2nd 18bit-DAC is in.