We had two site-wide power gltiches at 04:15:48 and 04:52:39 PDT Mon 21oct2024.
Both were seen in CS and EX MAINSMON channels. EX saw voltage drop for all three phases, CS saw this for two phases. Voltage drop persisted for about 3 cycles and then recovered.
h1susauxh2 and h1susauxex rebooted on the first glitch at 04:15, their models restarted around 04:17. No other model restarted at these times. There were no IPC errors and no long cpu-runs. We currently don't know why these two susaux machines rebooted by themselves.
Opened FRS32411
Notes from Fil:
The HAM6 PZT had to be restarted
Work on Beckhoff DAC power monitors
Both GC and CDS UPS units switched to battery backup power for both of these times. The GC unit sent email notifications, the CDS unit did not.
GC was on battery for 5 seconds, CDS for only 1 second.
FMC-EX_MAINS Channels seeing the outage at the reported times.
Outage 1: 4:15:48
Outage 2: 4:52:39
TITLE: 10/21 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 9mph Gusts, 3mph 3min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.11 μm/s
QUICK SUMMARY:
IFO is in MAINTENANCE due to NPRO and AMPs being disabled likely because of some power fault. Seems like the OWL shift was quite busy and that there are some PSL and LASER_PWR issues - will try to figure out what was going on.
Dave, Ibrahim
A significant power glitch on all 3 phases occured at 4:15:48 Local. PSL power issues Ryan C had were due to this (alog 80782). Plots below.
H1 called for help after "Not ready for 15 minutes" at 11:38 UTC. IMC_LOCK and LASER_PWR are in fault. My medms aren't opening on my no machine unless I do guardmedm in the console. IMC lock is saying the PMC and FSS can't lock, LASER_PWR is in fault, unable to get to 2Ws "IMC has no power". Following the IMC locking issues wiki page I've tried DOWN initing ISC_LOCK, IMCLOCK, and LASER_PWR, the FSS noise eater is green.
The MCs and IMs alignment all looks fine. The AMPs and NPRO all went to disabled about an hour ago.
The NPRO and AMPs are all disabled as of 11:15 UTC
Running PSL weekly scripy yielded the following:
Laser Status:
NPRO output power is 0.1373W (nominal ~2W)
AMP1 output power is -0.4512W (nominal ~70W)
AMP2 output power is 0.1128W (nominal 135-140W)
NPRO watchdog is RED
AMP1 watchdog is RED
AMP2 watchdog is RED
PDWD watchdog is GREEN
PMC:
It has been locked 0 days, 0 hr 1 minutes
Reflected power = -0.1761W
Transmitted power = -0.02753W
PowerSum = -0.2036W
FSS:
It has been locked for 0 days 0 hr and 0 min
TPD[V] = -0.01628V
ISS:
The diffracted power is around 3.0%
Last saturation event was 0 days 1 hours and 26 minutes ago
Possible Issues:
NPRO power is low
AMP1 power is low
AMP2 power is low
NPRO watchdog is inactive
AMP1 watchdog is inactive
AMP2 watchdog is inactive
FSS TPD is low
NPRO error, see SYSSTAT.adl
After talking to Jason, there's not much to do about this right now. More investigation needs to be done to see what happened to the laser, potentially needs to be swapped.
There power issues on the Hanford site that could have easily glitched our power and caused the npro to trip.
Fil re-enabled the PMC high voltage after arriving on site, then I brought the PSL back up with little issue. One hiccup I ran into was with the new PDWD for the amplifiers; when the NPRO shut off this morning, the regular power watchdogs tripped the system off, but the PDWD did not. This meant I had to disable the PDWD before I was able to reset the system and turn the NPRO back on. After that, there were no issues recovering the PSL.
TITLE: 10/21 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 157Mpc
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY: Two locklosses this shift, one from the PSL glitching, each followed by fully automatic relocks. Otherwise, it's been a fairly quiet evening.
Lockloss @ 03:25 UTC - link to lockloss tool
The NPRO/FSS had been glitching with high frequency for about a minute prior to the lockloss, so I feel confident saying it's the cause here.
H1 back to observing at 04:23 UTC. Fully automatic relock.
Lockloss @ 01:09 UTC - link to lockloss tool
No obvious cause; doesn't look like the FSS, and I don't see evidence of an ETM glitch or DARM wiggle.
H1 returned to observing at 02:18 UTC after a fully automatic relock.
TITLE: 10/20 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 156Mpc
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY:
IFO is in NLN and OBSERVING as of 21:31 UTC
Mostly quiet shift with one fussy lock acquisition.
Lock Acquisition had no special problems but rather, a slew of small annoyances (unknown cause Lockloss alog 80773):
Other:
LOG:
None
TITLE: 10/20 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 157Mpc
OUTGOING OPERATOR: Ibrahim
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 21mph Gusts, 16mph 3min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.15 μm/s
QUICK SUMMARY: H1 has been observing for about 1.5 hours.
Closes FAMIS 26307. Last checked in alog 80649. Service mode error, see SYSSTAT.adl is the only new difference between last week's check.
Laser Status:
NPRO output power is 1.827W (nominal ~2W)
AMP1 output power is 64.44W (nominal ~70W)
AMP2 output power is 137.4W (nominal 135-140W)
NPRO watchdog is GREEN
AMP1 watchdog is GREEN
AMP2 watchdog is GREEN
PDWD watchdog is GREEN
PMC:
It has been locked 1 days, 22 hr 40 minutes
Reflected power = 24.82W
Transmitted power = 100.8W
PowerSum = 125.7W
FSS:
It has been locked for 0 days 0 hr and 4 min
TPD[V] = 0.8048V
ISS:
The diffracted power is around 2.4%
Last saturation event was 0 days 0 hours and 5 minutes ago
Possible Issues:
AMP1 power is low
PMC reflected power is high
Service mode error, see SYSSTAT.adl
Looks like I had left the PSL computer in service mode after work on Tuesday; fortunately this doesn't actually effect anything operationally, so no harm done. I've just taken the computer out of service mode.
I also added 100mL of water into the chiller since Ibrahim had gotten the warning from Verbal today.