Quick summary - I tested my theory of the dust monitor pump running backwards and spewing contaminate with a pump at EY and was able to get it to run backwards.

A day after the power outage Dave noticed that the PSL dust counts were still very elevated, so I went to check on the corner station dust monitor vacuum pump and found it in an odd state (alog86857). The pump was running very hot, read 0inHg on the dial, and had some loose connections. I turned it off, tightened things up, it turned it back on with good pressure. Thinking more about it after I had walked away, my theory is that the pump started to run backwards when the power went out and the low pressure of the vacuum pulled the pump in reverse. The power then came back on and the motor started and continued in that direction.

Today I wanted to check on the end station pumps, so I took this opportunity to bring a pump that needed to be rebuilt anyway with me to the end station and try to recreate my theory above. I found no pump at EX, so I went to EY where I found the pump completely locked up and the motor, not pump, very hot. I unplugged this, hooked up the one that needed a rebuild and plugged it in. It pulled to -12inHg. I then tried a few times unplugging the power, listening to hear if it started to spin backwards, then pulling it back in. Third time I got it and it was running while pushing a bit of air out of the bleed valve and the pressure read 0inHg.

I didn't test at the dust monitor end of this system if it was spewing out any contaminate, most likely the graphite from the vanes in the pump, but I'd guess if it was running backwards over night it would make enough for us to notice. I'm looking into check valves, in line filters, or some type of relay to avoid this in the future.

TITLE: 09/15 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
OUTGOING OPERATOR: Ryan S
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 9mph Gusts, 5mph 3min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.09 μm/s
QUICK SUMMARY:

IFO is in IDLE for CORRECTIVE MAINTENANCE

Promise! Today, we were able to get to LASER_NOISE_SUPPRESSION!

Witness to the culprit (In Jenne's words): The IMC_REFL power increases with IMC power increase.

There was a dust monitor that blew back some dust into the PSL that we think may have been the culprit but the Jason and Keita just came back from the PSL after looking for signs of this contamination but found none.

The plan now is to look at the IMC_REFL power increase again by setting the IMC power to 60W then to 63, whilst also trying a new combo:

• IMC Power to 60W
• ISS 2nd loop ON
• ISS 2nd loop OFF
• IMC Power to 63W

Did IMC_REFL increase? We shall find out.

Since the power outage, when we've been in the higher 2W locking states, we have been seeing some 'breathing' of a spot in the upper right of the PRM camera. Before the power outage, there was some scattering seen there, but it looked different (now it looks like a thumbprint and has a clear outline when fully there) and didn't 'breathe' like we see now.

Pre-outage (2025-09-09 07:01 UTC) - in PREP_DC_READOUT

Post-outage (2025-09-15 19:44 UTC) - In PREP_DC_READOUT

Following the steps as detailed here: 86935 we were able to get to the engage ASC full IFO state.

I ran the code for engaging IFO ASC by hand, and there were no issues. I did move the alignment around by hand to make sure the buildups were good and the error signals were reasonable. Ryan reset the green references once all the loops, including soft loops engaged.

We held for a bit at 2W DC readout to confer on the plan. We decided to power up and monitor IMC REFL We checked that the IMC REFL power made sense:

• we reduce the IMC REFL power by half on IOT2L on Friday
• at 2 W, we saw that IMC REFL was 1 mW of power
• At 25 W we saw that IMC REFL was 12.7 mW
• before the power outage IMC REFL was 8 mW
• Daniel reports that the IMC REFL power tripled after the power outage, so we would expect 24 mW
• but with the power down by half, we get about 12 mW- this is reasonable
• We checked that the power followed this trend as we powered up to full power- 60 W from PSL

I ran guardian code to engage the camera servos so we could see what the low frequency noise looked like. It looked much better than it did the last time we were here!

We then stopped just before laser noise suppression. With IMC REFL down by half, we adjusted many gains up by 6 dB. We determined that on like 5939, where the IMC REFL gain is checked if it is below 2 should now be checked to see if it is below 8. I updated and loaded the guardian.

We rain laser noise suppression with no issues.

Then, I realized that we actually want to increase the power out of the PSL so that the power at IM4 trans matches the value before the power outage- due to the IMC issues that power has dropped from about 56 W to about 54 W.

I opened the ISS second loop with the guardian, and then stepped up PSL requested power from 60 W to 63 W. This seemed to get us the power out we wanted.

Then, while we were sitting at this slightly higher power, we had a lockloss. The lockloss appears to be an IMC lockloss (as in the IMC lost lock before the IFO).

The IMC REFL power had been increasing, which we expected from the increase of the input power. However, it looks like the IMC refl power was increasing even more than it should have been. This doesn't make any sense.

Since we were down, we again took the IMC up to 60 W and then 63 W. We do not see the same IMC refl power increase that we just saw when locked.

I am attaching an ndscope. I used the first time cursor to show when we stepped up to 63 W. You can see that between this first time cursor and second time cursor, the IMC refl power increases and the IM4 trans power drops. However, the iss second loop was NOT on. We also did NOT see this behavior when we stepped up to 60 W during the power up sequence. Finally, we could not replicate this behavior when we held in down and increased the input power with the IMC locked.

After engaging full IFO ASC and soft loops this afternoon, I updated the ITM camera and ALS QPD offsets and accepted them in the appropriate SAFE SDF tables. After Beckhoff reboots and PSL optic inspections, we'll run an initial alignment to solidify these alignment setpoints. They will need to be accepted in the OBSERVE tables once eventually back to NLN.

We had a lockloss while in LASER_NOISE_SUPPRESSION (575), and looking at ASC-AS_A, the light on the PD dropped at the same time as DARM lost lock, so it was an IMC lockloss (lockloss webpage is still unaccessible but command line tool worked for me after waiting a while)

The previous channel list was channels which were non-zero before the glitch and became zero afterwards.

I've extended the analysis to look for channels which were varying before the glitch and became flat-lined afterwards, the flat-line value is shown.

auxcs.txt:  H1:SYS-ETHERCAT_AUXCORNER_INFO_CB_QUEUE_2_PERCENT (varying→flat:8.000e-05)
auxcs.txt:  H1:SYS-TIMING_C_FO_A_PORT_12_NODE_GENERIC_PAYLOAD_1 (varying→flat:9.100e+06)
auxcs.txt:  H1:SYS-TIMING_C_FO_A_PORT_12_NODE_XOLOCK_MEASUREDFREQ (varying→flat:9.100e+06)
auxcs.txt:  H1:SYS-TIMING_C_FO_B_PORT_11_NODE_GENERIC_PAYLOAD_0 (varying→flat:1.679e+04)
auxcs.txt:  H1:SYS-TIMING_C_FO_B_PORT_11_NODE_GENERIC_PAYLOAD_17 (varying→flat:2.620e+02)
auxcs.txt:  H1:SYS-TIMING_C_FO_B_PORT_11_NODE_PCIE_HASEXTPPS (varying→flat:1.000e+00)
auxcs.txt:  H1:SYS-TIMING_C_FO_B_PORT_2_NODE_GENERIC_PAYLOAD_13 (varying→flat:5.830e+02)
auxcs.txt:  H1:SYS-TIMING_C_FO_B_PORT_3_NODE_GENERIC_PAYLOAD_13 (varying→flat:6.470e+02)
auxcs.txt:  H1:SYS-TIMING_X_GPS_A_DOP (varying→flat:3.000e-01)
auxcs.txt:  H1:SYS-TIMING_Y_GPS_A_DOP (varying→flat:3.000e-01)
sqzcs.txt:  H1:SQZ-FIBR_LOCK_BEAT_FREQUENCYERROR (varying→flat:2.000e+00)
sqzcs.txt:  H1:SQZ-FREQ_ADF (varying→flat:-3.200e+02)
sqzcs.txt:  H1:SQZ-FREQ_LASERBEATVSDOUBLELASERVCO (varying→flat:2.000e+00)
sqzcs.txt:  H1:SYS-ETHERCAT_SQZCORNER_CPUUSAGE (varying→flat:1.200e+01)
tcsex.txt:  H1:SYS-ETHERCAT_TCSENDX_CPUUSAGE (varying→flat:1.200e+01)
tcsey.txt:  H1:AOS-ETMY_BAFFLEPD_4_ERROR_CODE (varying→flat:6.400e+01)
tcsey.txt:  H1:AOS-ETMY_BAFFLEPD_4_ERROR_FLAG (varying→flat:1.000e+00)
tcsey.txt:  H1:AOS-ETMY_ERROR_CODE (varying→flat:1.000e+01)
 

Comparing IM4 Trans alignment at 2W in before the power outage to now. Plot attached.

IM4 trans Pitch alignment is the same, Yaw alignment is 0.06 different. So alignment changes are minimal.

Power on IM4 trans is slightly (~3%) lower: NSUM Power on IM4 trans was 1.891 for 2.026W in (ratio 0.933), now is 1.780 for 1.974 in (ratio 0.902).

Here are the steps Sheila and I took that almost got us to prep ASC for full ifo:

• take guardian to carm_5_picometers
• change H1:LSC-TR_CARM_OFFSET to -52
• ramp H1:LSC-TR_CARM_GAIN to 2.1
• change H1:LSC-TR_CARM_OFFSET to -56

We measured all the LSC gains and found that maybe PRCL gain was dropping and causing locklosses so

• increase PRCL2 gain from 1 to 1.6
• increase DHARD P gain to -30, DHARD Y gain to -40
• run CARM_TO_REFL state in guardian
• reduce H1:LSC-TR_REFLAIR9_OFFSET (make sure to set a ramp time first!) slowly to zero
• ran RESONANCE state in guardian
• ran CARM_TO_ANALOG

We then went to prep_ASC_for_Full_ifo but we forgot to reduce the PRCL2 gain, so this probably caused a lockloss since DRMI moved to POP

We will keep trying this and maybe do full IFO ASC next

On the next attempt, I added this additional step:

• before DRMI to POP, I realized that I should drop the PRCL2 gain
• by hand, I lowered the PRCL2 gain slightly, and then increased the input matrix value for REFLAIR27 I accordingly until that was 1.6 times higher than its nominal value
• then, I ran DRMI to POP, which went fine.

Hi y'all!

Link to full report here.

Summary:

• 73%, 44.6%, 71.8%, 43.02%, 0%, 0%, 0% Observing per day
• Large EQ off coast of OR, as well as an electrical storm, caused lockloss/glitches on Tues
• Power outage on Wednesday caused many issues with IFO
• Greatly reduced range on Thursday, as well as extremely increased glitchrate
• IFO down for corrective maintenance from Thursday onwards. 
• One HVeto repeated winner for Mon, Tues, Wed:H1:LSC-REFL_A_LF_OUT_DQ. On Thursday, following the power outage, there were 12 round winners, 9 of which belonged to IMC
• Only three days of FScan-Observing: Mon - Wed. Tuesday was the lowest and Wednesday was the highest. All three hovered around ~2500 lines

Mon Sep 15 10:07:58 2025 INFO: Fill completed in 7min 54secs

I** have written a program to compare slow controls channels before and after with Wed 10sep2025 power glitch to see if there are any which look like they may have been broken and need further investigation.

(** Full disclosure, it was actually written by AI (Claude-code) and runs on my GC laptop safely contained in a virtual machine running Deb13 Trixie)

As a first pass, the code is looking for dead channels. These are channels which were active before the glitch, and are flat-line zero following.

The code gets its channels to analyze from the slow controls INI files.

Using Jonathan's simple_frames python module, it reads two minute trend GWF frame files. For before I'm using the 10:00-11:00 Wednesday file (an hour before the glitch) and for after I'm using the Thu 00:00-01:00 file. In both cases H1 was locked.

I'll post the results as comments to this alog.

I posted the following message in the Detchar-LHO mattermost channel:

Hey detchar! We could use a hand with some analysis on the presence and character of the glitches we have been seeing since our power outage Wednesday. They were first reported here: https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=86848 We think these glitches are related to some change in the input mode cleaner since the power outage, and we are doing various tests like changing alignment and power, engaging or disengaging various controls loops, etc. We would like to know if the glitches change from these tests.