We had two site-wide power gltiches at 04:15:48 and 04:52:39 PDT Mon 21oct2024.
Both were seen in CS and EX MAINSMON channels. EX saw voltage drop for all three phases, CS saw this for two phases. Voltage drop persisted for about 3 cycles and then recovered.
h1susauxh2 and h1susauxex rebooted on the first glitch at 04:15, their models restarted around 04:17. No other model restarted at these times. There were no IPC errors and no long cpu-runs. We currently don't know why these two susaux machines rebooted by themselves.
Opened FRS32411
Notes from Fil:
The HAM6 PZT had to be restarted
Work on Beckhoff DAC power monitors
Both GC and CDS UPS units switched to battery backup power for both of these times. The GC unit sent email notifications, the CDS unit did not.
GC was on battery for 5 seconds, CDS for only 1 second.
FMC-EX_MAINS Channels seeing the outage at the reported times.
Outage 1: 4:15:48
Outage 2: 4:52:39
