H1SUSETMX ADC channels zeroed out at 21:11:39. SWWDs did not trip because there is no RMS on the OSEM signals, but the HWWD completed its 20 minute countdown and powered down the three ISI coil drivers at 21:32. This indicates ETMX's top stage OSEMs have lost power.

I've opened WP12692 to cover Fil going to EX to investigate.

During the recovery the +24VDC power supply for the SUS IO Chassis was glitched which stopped all the h1susex and h1susauxex models. To recover I first did a straight forward reboot of h1susauxex (no Dolphin), it came back with no issues.

To reboot h1susex was more involved, remember that the EX Dolphin switch was damaged by the 06 April 2025 power outage and has no network control. The procedure to reboot h1susex I used was:

• caput H1:IOP-SEIEX_REMOTE_IPC_PAUSE 1
• caput H1:IOP-ISCEX_REMOTE_IPC_PAUSE 1
• caput H1:IOP-SUSEX_REMOTE_IPC_PAUSE 1
• on h1susex:
• rtcds stop --all
• sudo systemctl reboot

When h1susex came back, I verified all the IO Chassis cards were present (they were all there)

I unpaused the SEI and ISC IPC by writing a 0 to their IPC_PAUSE channels.

The HWWD came back in nominal state.

I reset the SUS SWWD DACKILLs and unbypassed the SEI SWWD.

DIAG_RESET to clear all the IPC errors (it did so) and clear DAQ CRCs (they cleared).

Handed systems over to control room (Oli and Ryan S).

From Fil:

-18VDC Power supply had failed and was replaced.

Power supply is in rack VDD-2, location U25-U28, right-hand supply, label [SUS-C1 C2]

old supply (removed) S1202024

new supply (installed) S1300288

Last night's HWWD sequence is shown below. Reminder that at +40mins the SUS part of the HWWD trips, which sets bit2 of the STAT. This opens internal relay switches, but since we don't route the SUS drives through the HWWD unit (too noisy) this has no effect on operations. The delay between 22:52 and 23:20 is because h1iopsusex was down between 23:01 and 23:20.

FRS34786

Fan motor seized on failed power supply.