Ryan C, Dave:

At 20:14:41 Tue 25 Feb 2025 PST we had a corner station issue which put the IOP models for h1sush34 and h1lsc0 into a DACKILL state. First attachment shows the status before any resets had been issued, second shows overview and the iop models after DIAG_RESET and DAQ_CRC_RESET had been issued.

There were no dmesg messages on either h1sush34 and h1lsc0 from today. I checked that both of these front ends had their full complement of cards in their IO Chassis.

In prep for restarting the IOP models, which requires restarting all the models on these front ends, Ryan put HAM3,4 into a safe state. I engaged the SWWD bypass on the SEI system for HAM 3,4.

After restarting the models on h1sush34 and h1lsc0 they came back with no issues.

The initial cause of the crash is not clear. Timing of the IOP STATE_WORDs going offline suggests sush34 crashed first glitching lsc0 at that time, but lsc0 didn't go fully offline until 6 seconds later.