Jonathan, Erik, Tony, Rahul, Richard, Dave:

Last night we had another one of the corner station glitches which took down all the models on h1susb123, h1sush2a, h1sush34 and h1sush56.

The glitch happened at 18:45:55 PST Mon 29th Jan 2024 during the O4a,b break.

This has happened several times before, details in FRS30324, with a regularity of every 80 to 100 days.

After the glitch, the dolphin IPC continued to function, which meant that the SWWDs did not trip the associated Seismic systems.

DMESG logs did not show anything.

I ran a complete set of Dolphin DIS_DIAGS (previous run was 24th Jan 2024), which only showed the issues with OAF0 and SEIH16 from last Friday's crashes. No problems with any SUS was seen.

Recovery from the crash was:

• Set all SEI IOP SWWD system into BYPASS mode with a very long bypass time (999999999S)
• Restart all the models on h1sus[b123, h2a, h34, h56]. All started with no problems.
• Unbypass the SEI SWWDs.
• Hand over to control room for SUS recovery.