Start Time: 1442034629

End Time: 1442034940

2025-09-15 22:15:22,374 bb measurement complete.
2025-09-15 22:15:22,375 bb output: /ligo/groups/cal/H1/measurements/PCALY2DARM_BB/PCALY2DARM_BB_20250916T051011Z.xml
2025-09-15 22:15:22,375 all measurements complete

Ibrahim, Elenna, Jenne, Ryan S

We made it to LASER_NOISE_SUPPRESSION!

Here's how we got here again and here's what we're monitoring

After much needed help from Ryan S, Elenna and Jenne in initial alignment (manual initial alignment needed), we managed to get back to CARM_5_PICOMETERS but encountered the same issues with an SRM ringup upn getting to RESONANCE, which caused a Lockloss.

Elenna guided me through the following combination of steps tried this morning:

1. Get to CARM_5_PICOMETERS

2. Set TR_CARM_OFFSET to -52

3. Set the REFLAIR_B_RF27 I PRCL matrix to 1.6*(value-it's-at). Press LOAD MATRIX. This is reached by going to LSC->LSC_OVERVIEW->AIR3F (under input matrices area on left of screen, you want to look for REFLAIR_B_RF27 on the left).

4. Continue locking - SRM rang up as before and saturated once but with the new gain, it was enough to not cause a LL.

This worked, and now we're at OMC_WHITENING with high violins (so understandable, sorry fibers).

Now, I'm monitoring H1:IMC-REFL_DC_OUT16 and watching for an increase in power, which is bad. We've been at 63W (the new 60) for 18 minutes - here's a plot of that.

I don't know the calibration, but CDS OVERVIEW is reading out 147MPc

[Fil, Jeff, Dave, Erik, Patrick]

As part of the hunt for problems related to the power outage on September 10, Dave notices that the channel H1:ISC-RF_C_AMP24M1_POWEROK had moved from 1 before the outage to 0 after. 

Jeff determined that the output of the amplifier was nominal, so that likely it was a problem with Beckhoff and not with the amplifier itself.

Fil inserted a breakout board between the amp the beckhoff cable and recorded a voltage drop from 3.3 to 2.2 volts on the power ok signal (pin 7). 

Some more testing shows the problem was definitely at Beckhoff end.  The associated terminal may need to be replaced.

J. Oberling, K. Kawabe

This afternoon we went into the PSL enclosure to inspect things after last week's power outage.  We concentrated on the IOO side of the PSL table, downstream of the PMC.  Our results:

We did a visual inspection with both the IR viewer and the IR-sensitive Nikon camera and did not find any obvious signs of damage on any of the optical surfaces we had access to; the only ones we couldn't see were the crystal surfaces inside the ISC EOM, we could see everything else.  I looked at the optics between Amp2 and the PMC and everything there looked normal, no signs of anything amiss.

While the beam was not perfectly centered on every optic, we saw no clipping anywhere in the beam path.  The irises after the bottom periscope mirror were not well centered, but they've been that way for a while so we didn't have a good reference for assessing alignment in that path (these irises were set after the O4 PSL upgrade, but there have been a couple of alignment shifts since then and the irises were not reset).  For reference, the beam is in the -X direction on the HWP in the power control rotation stage and in the -Z direction (but centered horizontally) on the PZT mirror after the power control stage.  We do have a good alignment reference on the ALS path (picked off through mirror IO_MB_M2, the mirror just before the ISC EOM), as those were set as part of the HAM1 realignment during this year's vent.  By my eye the first iris looked a tiny bit off in yaw (-Y direction) and pitch (+Z direction), while the second iris looked perfectly centered.  We found this odd, so Keita used the IR-sensitive camera to get a better angle on both irises and took some pictures.  With the better angle the beam looked well centered in yaw and maybe a little off in pitch (+Z direction) on that first iris, so I think my eye was influenced by the angle from which I was viewing the iris.  The second iris still looked very well centered.  Edit to add: Since the ALS path alignment looks good, to me this signals that there was not an appreciable alignment shift as a result of the change in PMC temperature.  If the PMC was the source of the alignment shift we would see it in both the main IFO and ALS paths.  If there is a misalignment in the main IFO path, its source is not the PMC.  Upon further reflection, a more accurate statement is:  If the PMC is the source of an alignment shift, the shift is too small to be seen on the PSL table (but not necessarily too small to be seen by the IMC).

The other spot of note is the entrance aperture for the ISC EOM.  It's really bright so it's hard to make a definitive determination, but it could be argued there's a very slight misalignment going into the ISC EOM.  I couldn't make anything out with the IR viewer, but Keita's picture shows the typical ring around the aperture a little brighter on the left side versus the right.  Despite this, there is no clipping in the beam, as we set up a WinCAM beam profiler to check.

The WinCAM was set behind IO_AB_L4, which is the lens immediately behind the bottom periscope mirror (this is the path that goes to the IMC_IN PD).  The attached picture shows what the beam looks like there.  No signs of clipping in the beam, so it's clearing all the apertures in the beam path.  I recall doing a similar measurement at this spot several years ago, but a quick alog search yields nothing.  I'll do a deeper dive tomorrow and add a comment should I find anything.

So to summarize, we saw no signs of damage to any visible optical surfaces.  We saw no clear evidence of a misalignment in the beam; the ALS path looks good, and nothing in the main IFO path looks suspicious outside of the ISC EOM entrance aperture (a lack of good alignment irises makes this a little difficult to assess; once we get the IFO back to a good alignment we should reset those irises).  We saw no clipping in the beam.

Keita has many pictures that he will post as a comment to this log.

TITLE: 09/15 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: Ibrahim
SHIFT SUMMARY: More progress today towards recovering H1 back to low noise! We've been able to make it up to LASER_NOISE_SUPPRESSION once so far and updated alignment offsets. Unfortunately we did see the slow increase of IMC REFL again once increasing the input power to 63W during LNS, and shortly after there was an IMC-looking lockloss. Since then, Jason and Keita inspected optics in the PSL and didn't find anything alarming. We attempted to reproduce the IMC REFL behavior with just the IMC locked up to 63W with the ISS secondloop on and off, but did not see it, interestingly. We then decided to try locking again, so an initial alignment with the new green offsets is ongoing.
LOG:

After finding the error code changes for the AMP241M1 sensor, Erik suggested listing channels which have single values before and after, but the value changed.

This table shows a summary of the channel counts, the detailed lists at in attached text files. non-zero to zero is called dead, varying to flatline is called broken, single to diff single is called suspicious.

Quick summary - I tested my theory of the dust monitor pump running backwards and spewing contaminate with a pump at EY and was able to get it to run backwards.

A day after the power outage Dave noticed that the PSL dust counts were still very elevated, so I went to check on the corner station dust monitor vacuum pump and found it in an odd state (alog86857). The pump was running very hot, read 0inHg on the dial, and had some loose connections. I turned it off, tightened things up, it turned it back on with good pressure. Thinking more about it after I had walked away, my theory is that the pump started to run backwards when the power went out and the low pressure of the vacuum pulled the pump in reverse. The power then came back on and the motor started and continued in that direction.

Today I wanted to check on the end station pumps, so I took this opportunity to bring a pump that needed to be rebuilt anyway with me to the end station and try to recreate my theory above. I found no pump at EX, so I went to EY where I found the pump completely locked up and the motor, not pump, very hot. I unplugged this, hooked up the one that needed a rebuild and plugged it in. It pulled to -12inHg. I then tried a few times unplugging the power, listening to hear if it started to spin backwards, then pulling it back in. Third time I got it and it was running while pushing a bit of air out of the bleed valve and the pressure read 0inHg.

I didn't test at the dust monitor end of this system if it was spewing out any contaminate, most likely the graphite from the vanes in the pump, but I'd guess if it was running backwards over night it would make enough for us to notice. I'm looking into check valves, in line filters, or some type of relay to avoid this in the future.

TITLE: 09/15 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
OUTGOING OPERATOR: Ryan S
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 9mph Gusts, 5mph 3min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.09 μm/s
QUICK SUMMARY:

IFO is in IDLE for CORRECTIVE MAINTENANCE

Promise! Today, we were able to get to LASER_NOISE_SUPPRESSION!

Witness to the culprit (In Jenne's words): The IMC_REFL power increases with IMC power increase.

There was a dust monitor that blew back some dust into the PSL that we think may have been the culprit but the Jason and Keita just came back from the PSL after looking for signs of this contamination but found none.

The plan now is to look at the IMC_REFL power increase again by setting the IMC power to 60W then to 63, whilst also trying a new combo:

• IMC Power to 60W
• ISS 2nd loop ON
• ISS 2nd loop OFF
• IMC Power to 63W

Did IMC_REFL increase? We shall find out.

Since the power outage, when we've been in the higher 2W locking states, we have been seeing some 'breathing' of a spot in the upper right of the PRM camera. Before the power outage, there was some scattering seen there, but it looked different (now it looks like a thumbprint and has a clear outline when fully there) and didn't 'breathe' like we see now.

Pre-outage (2025-09-09 07:01 UTC) - in PREP_DC_READOUT

Post-outage (2025-09-15 19:44 UTC) - In PREP_DC_READOUT

Following the steps as detailed here: 86935 we were able to get to the engage ASC full IFO state.

I ran the code for engaging IFO ASC by hand, and there were no issues. I did move the alignment around by hand to make sure the buildups were good and the error signals were reasonable. Ryan reset the green references once all the loops, including soft loops engaged.

We held for a bit at 2W DC readout to confer on the plan. We decided to power up and monitor IMC REFL We checked that the IMC REFL power made sense:

• we reduce the IMC REFL power by half on IOT2L on Friday
• at 2 W, we saw that IMC REFL was 1 mW of power
• At 25 W we saw that IMC REFL was 12.7 mW
• before the power outage IMC REFL was 8 mW
• Daniel reports that the IMC REFL power tripled after the power outage, so we would expect 24 mW
• but with the power down by half, we get about 12 mW- this is reasonable
• We checked that the power followed this trend as we powered up to full power- 60 W from PSL

I ran guardian code to engage the camera servos so we could see what the low frequency noise looked like. It looked much better than it did the last time we were here!

We then stopped just before laser noise suppression. With IMC REFL down by half, we adjusted many gains up by 6 dB. We determined that on like 5939, where the IMC REFL gain is checked if it is below 2 should now be checked to see if it is below 8. I updated and loaded the guardian.

We rain laser noise suppression with no issues.

Then, I realized that we actually want to increase the power out of the PSL so that the power at IM4 trans matches the value before the power outage- due to the IMC issues that power has dropped from about 56 W to about 54 W.

I opened the ISS second loop with the guardian, and then stepped up PSL requested power from 60 W to 63 W. This seemed to get us the power out we wanted.

Then, while we were sitting at this slightly higher power, we had a lockloss. The lockloss appears to be an IMC lockloss (as in the IMC lost lock before the IFO).

The IMC REFL power had been increasing, which we expected from the increase of the input power. However, it looks like the IMC refl power was increasing even more than it should have been. This doesn't make any sense.

Since we were down, we again took the IMC up to 60 W and then 63 W. We do not see the same IMC refl power increase that we just saw when locked.

I am attaching an ndscope. I used the first time cursor to show when we stepped up to 63 W. You can see that between this first time cursor and second time cursor, the IMC refl power increases and the IM4 trans power drops. However, the iss second loop was NOT on. We also did NOT see this behavior when we stepped up to 60 W during the power up sequence. Finally, we could not replicate this behavior when we held in down and increased the input power with the IMC locked.

After engaging full IFO ASC and soft loops this afternoon, I updated the ITM camera and ALS QPD offsets and accepted them in the appropriate SAFE SDF tables. After Beckhoff reboots and PSL optic inspections, we'll run an initial alignment to solidify these alignment setpoints. They will need to be accepted in the OBSERVE tables once eventually back to NLN.

We had a lockloss while in LASER_NOISE_SUPPRESSION (575), and looking at ASC-AS_A, the light on the PD dropped at the same time as DARM lost lock, so it was an IMC lockloss (lockloss webpage is still unaccessible but command line tool worked for me after waiting a while)

The previous channel list was channels which were non-zero before the glitch and became zero afterwards.

I've extended the analysis to look for channels which were varying before the glitch and became flat-lined afterwards, the flat-line value is shown.

auxcs.txt:  H1:SYS-ETHERCAT_AUXCORNER_INFO_CB_QUEUE_2_PERCENT (varying→flat:8.000e-05)
auxcs.txt:  H1:SYS-TIMING_C_FO_A_PORT_12_NODE_GENERIC_PAYLOAD_1 (varying→flat:9.100e+06)
auxcs.txt:  H1:SYS-TIMING_C_FO_A_PORT_12_NODE_XOLOCK_MEASUREDFREQ (varying→flat:9.100e+06)
auxcs.txt:  H1:SYS-TIMING_C_FO_B_PORT_11_NODE_GENERIC_PAYLOAD_0 (varying→flat:1.679e+04)
auxcs.txt:  H1:SYS-TIMING_C_FO_B_PORT_11_NODE_GENERIC_PAYLOAD_17 (varying→flat:2.620e+02)
auxcs.txt:  H1:SYS-TIMING_C_FO_B_PORT_11_NODE_PCIE_HASEXTPPS (varying→flat:1.000e+00)
auxcs.txt:  H1:SYS-TIMING_C_FO_B_PORT_2_NODE_GENERIC_PAYLOAD_13 (varying→flat:5.830e+02)
auxcs.txt:  H1:SYS-TIMING_C_FO_B_PORT_3_NODE_GENERIC_PAYLOAD_13 (varying→flat:6.470e+02)
auxcs.txt:  H1:SYS-TIMING_X_GPS_A_DOP (varying→flat:3.000e-01)
auxcs.txt:  H1:SYS-TIMING_Y_GPS_A_DOP (varying→flat:3.000e-01)
sqzcs.txt:  H1:SQZ-FIBR_LOCK_BEAT_FREQUENCYERROR (varying→flat:2.000e+00)
sqzcs.txt:  H1:SQZ-FREQ_ADF (varying→flat:-3.200e+02)
sqzcs.txt:  H1:SQZ-FREQ_LASERBEATVSDOUBLELASERVCO (varying→flat:2.000e+00)
sqzcs.txt:  H1:SYS-ETHERCAT_SQZCORNER_CPUUSAGE (varying→flat:1.200e+01)
tcsex.txt:  H1:SYS-ETHERCAT_TCSENDX_CPUUSAGE (varying→flat:1.200e+01)
tcsey.txt:  H1:AOS-ETMY_BAFFLEPD_4_ERROR_CODE (varying→flat:6.400e+01)
tcsey.txt:  H1:AOS-ETMY_BAFFLEPD_4_ERROR_FLAG (varying→flat:1.000e+00)
tcsey.txt:  H1:AOS-ETMY_ERROR_CODE (varying→flat:1.000e+01)
 

Comparing IM4 Trans alignment at 2W in before the power outage to now. Plot attached.

IM4 trans Pitch alignment is the same, Yaw alignment is 0.06 different. So alignment changes are minimal.

Power on IM4 trans is slightly (~3%) lower: NSUM Power on IM4 trans was 1.891 for 2.026W in (ratio 0.933), now is 1.780 for 1.974 in (ratio 0.902).

Here are the steps Sheila and I took that almost got us to prep ASC for full ifo:

• take guardian to carm_5_picometers
• change H1:LSC-TR_CARM_OFFSET to -52
• ramp H1:LSC-TR_CARM_GAIN to 2.1
• change H1:LSC-TR_CARM_OFFSET to -56

We measured all the LSC gains and found that maybe PRCL gain was dropping and causing locklosses so

• increase PRCL2 gain from 1 to 1.6
• increase DHARD P gain to -30, DHARD Y gain to -40
• run CARM_TO_REFL state in guardian
• reduce H1:LSC-TR_REFLAIR9_OFFSET (make sure to set a ramp time first!) slowly to zero
• ran RESONANCE state in guardian
• ran CARM_TO_ANALOG

We then went to prep_ASC_for_Full_ifo but we forgot to reduce the PRCL2 gain, so this probably caused a lockloss since DRMI moved to POP

We will keep trying this and maybe do full IFO ASC next

On the next attempt, I added this additional step:

• before DRMI to POP, I realized that I should drop the PRCL2 gain
• by hand, I lowered the PRCL2 gain slightly, and then increased the input matrix value for REFLAIR27 I accordingly until that was 1.6 times higher than its nominal value
• then, I ran DRMI to POP, which went fine.

Hi y'all!

Link to full report here.

Summary:

• 73%, 44.6%, 71.8%, 43.02%, 0%, 0%, 0% Observing per day
• Large EQ off coast of OR, as well as an electrical storm, caused lockloss/glitches on Tues
• Power outage on Wednesday caused many issues with IFO
• Greatly reduced range on Thursday, as well as extremely increased glitchrate
• IFO down for corrective maintenance from Thursday onwards. 
• One HVeto repeated winner for Mon, Tues, Wed:H1:LSC-REFL_A_LF_OUT_DQ. On Thursday, following the power outage, there were 12 round winners, 9 of which belonged to IMC
• Only three days of FScan-Observing: Mon - Wed. Tuesday was the lowest and Wednesday was the highest. All three hovered around ~2500 lines

I** have written a program to compare slow controls channels before and after with Wed 10sep2025 power glitch to see if there are any which look like they may have been broken and need further investigation.

(** Full disclosure, it was actually written by AI (Claude-code) and runs on my GC laptop safely contained in a virtual machine running Deb13 Trixie)

As a first pass, the code is looking for dead channels. These are channels which were active before the glitch, and are flat-line zero following.

The code gets its channels to analyze from the slow controls INI files.

Using Jonathan's simple_frames python module, it reads two minute trend GWF frame files. For before I'm using the 10:00-11:00 Wednesday file (an hour before the glitch) and for after I'm using the Thu 00:00-01:00 file. In both cases H1 was locked.

I'll post the results as comments to this alog.