TITLE: 10/26 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 153Mpc
INCOMING OPERATOR: Corey
SHIFT SUMMARY: Two lock acquisitions today, both of which had locklosses during TRANSITION_FROM_ETMX on the first try. This made the relocking times a bit longer, but generally the process was automatic. H1 has been locked for 2.5 hours.

• 14:49 - Starting initial alignment
• 15:10 - Starting lock acquisition
  • 15:50 - Lockloss in TRANSITION_FROM_ETMX
• 17:13 - Reached NLN, but moved to high-gain ASC to quell the 1 Hz ringup and fixed nominal SQZ Guardian states - alog87751
• 17:23 - Observing
• 18:54 - Lockloss - alog87753
  • 20:06 - Lockloss in TRANSITION_FROM_ETMX
• 20:59 - Observing

Now that we're back to observing with regularity after a week full of intense troubleshooting, I decided it would be nice to have a summary of the events that contributed to IFO locking issues in the past week and what was done along the way to fix them all in one place, so that's what I'll attempt do in this alog. Anyone should be encouraged to add comments with things I missed or additional commentary. Things I believe to be the most significant changes either contributing to or fixing IFO locking issues I've bolded.

• Monday Oct. 20
  • H1 is generally locking reliably despite high microseism
  • CSOFT_P and INP1_P gains updated - alog87582
  • Power scaling reference channel used in LASER_PWR Guardian is changed to IM4 TRANS - alog87581
  • H1 has a lockloss and things look strange immediately after - alog87595
    • Issues discovered to be a result of the power scaling change from earlier in the day, which is reverted - alog87598
    • While troubleshooting, the dark offset script is run and many SDF diffs are accepted, including filter changes on the LSC POP A RF45 diode - see first screenshot in alog87598
      • This changes whitening settings on this diode which are now incorrectly set when going through SDF_REVERT
      • Dark offsets are doubled as a result of different settings which are also accepted in SDF
  • H1 is unable to relock with high microseism as the reason given
• Tuesday Oct. 21 (Maintenance Day)
  • Power scaling reference channel function implemented in LASER_PWR Guardian to avoid previous issue - alog87603
  • Power supply for EX HEPI pump controller and Beckhoff vacuum controls replaced - alog87606
  • HAM7 CPS switched to local timing scheme - alog87608
  • Sat amps for lower stages of MC1, MC2, MC3, PR3, and SR3 updated - alog87620
  • P/SR3 estimators installed and engaged - alog87616
    • This began causing oscillations at 0.6 Hz while locking
  • Power scaling channel update reverted due to LSC instability while relocking - alog87624
  • BRSY moved out-of-loop - alog87634
  • H1 is unable to relock overnight with unstable LSC as the reason given - alog87640 and others
• Wednesday Oct. 22
  • ASC gain changes reverted, 2x MICH ASC gain added, A2L gains in MOVE_SPOTS changed - alog87645 and comments
  • P/SR3 estimators discovered to be the culprit of 0.6 Hz instability, HBW ASC discovered to fix 1 HZ ASC ringup when it appears - alog86758
  • P/SR3 estimators disengaged - alog86759
  • H1 struggles to relock overnight with ETMY roll mode and 1 Hz ASC ringups as the reasons given - alog87660 and alog87672
• Thursday Oct. 23
  • Problematic roll mode is found to actually be on ETMX, not ETMY - alog87698
  • Most of the day is spent finding good damping settings for the ETMX roll mode; H1 does no observing and no locking at all overnight - alog87708
• Friday Oct. 24
  • DHARD_P steps commented out of LOWNOISE_ASC - alog87712
  • ETMY sat amp discovered to be causing saturations on reaction chain - alog87713
    • Sat amp swapped, fixing saturations - alog87722
  • Incorrect whitening settings and drastically different dark offsets on LSC POP A RF45 are finally discovered and fixed - alog87728
    • MICH ASC gain un-doubled
  • DAMP_BOUNCE_ROLL state is moved from right before powerup to after LOWNOISE_LENGTH_CONTROL - not logged, but evidenced in ISC_LOCK code history
    • Since we've changed the roll mode damping to ETMX, we don't want it on during ESD transitions, hence this change in state order
    • Unbeknownst to those involved at the time, this state has been turning on DARM actuation to ETMY which would cause its bounce mode to ring up, but later states had been fixing this before this change
  • H1 is unable to relock from ETM bounce modes now being rung up and high winds; no observing or locking overnight - alog87734
• Saturday Oct. 25
  • Accidental ETMY actuation from DAMP_BOUNCE_ROLL discovered; state removed from Guardian path - alog87738
    • CSOFT_P gain updated and gain change removed from THERMALIZATION node
    • This means no damping is currently being done on any bounce or roll modes, but these have not been seen to be a problem since
  • H1 returns to observing for an appreciable length of time and locking resumes its automatic nature through the weekend

Lockloss @ 18:54 UTC after 1:41 locked - link to lockloss tool

No obvious cause, but I did see a couple glitches in the PRG a few seconds before the lockloss.

FAMIS 27400

Laser Status:
    NPRO output power is 1.858W
    AMP1 output power is 70.71W
    AMP2 output power is 140.2W
    NPRO watchdog is GREEN
    AMP1 watchdog is GREEN
    AMP2 watchdog is GREEN
    PDWD watchdog is GREEN

PMC:
    It has been locked 33 days, 0 hr 29 minutes
    Reflected power = 25.05W
    Transmitted power = 106.7W
    PowerSum = 131.8W

FSS:
    It has been locked for 0 days 2 hr and 11 min
    TPD[V] = 0.5278V

ISS:
    The diffracted power is around 3.4%
    Last saturation event was 0 days 3 hours and 41 minutes ago

Possible Issues:
    PMC reflected power is high

Sun Oct 26 10:09:38 2025 INFO: Fill completed in 9min 34secs

TITLE: 10/26 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Wind
OUTGOING OPERATOR: Tony
CURRENT ENVIRONMENT:
    SEI_ENV state: USEISM
    Wind: 31mph Gusts, 23mph 3min avg
    Primary useism: 0.05 μm/s
    Secondary useism: 0.52 μm/s 
QUICK SUMMARY: H1 has been down since 07:09 UTC and struggled to lock overnight due to high winds. Gusts are still high this morning, but I'll try to run H1 through an initial alignment and locking to see how far it gets.

• Wind high (recently a few gusts >40mph), microseism around 90th percentile
• All other systems nominal

H1 called me on my owl shift.
When I logged on I was suprised to learned we were only having troubles with the SQZ system in the wind that is howling and gusting to over 40mph.

Since 

The SQZ_MAN was trying to get the SQZr back to FDS but kept getting hung up on FC_WAIT_FDS and would drop back down. 
I was going to do Observing without squeezing to troubleshoot but we had a lockloss that I'm gonna blame on the wind.

https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi?event=1445499787

Running an initial alignment now.

TITLE: 10/26 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 151Mpc
INCOMING OPERATOR: Tony
SHIFT SUMMARY:

H1 finally back in buisiness.  Had one lockloss during the shift, but relock was mostly automatic.  Winds picked up in the last hour & microseism is still high.  Owl Shift is on for tonight.  H1 range is hovering around 153Mpc.
LOG:

• 2343 EQ alert for M6 South Pacific/Vanuatu EQ AND it was right on the line for ASC HiGn on the EQ Response, so plans made to transition to ASC Hi Gn.
  • 0009 EQ Mode Acivated Verbal Alarm ---switched to USEISM_EARTHQUAKE
  • 0010 ASC Hi Gn button pushed---OUT OF OBSERVING
  • 0014 R-Wave rolls thru PLUS see:  oscillations on PRMI scope (nuc30), asc scope noise, picket fence noise (no color changes though).
  • 0029 Seismic back to USEISM
  • 0043 ASC Low Noise---> Back to OBSERVING!
  • NOTE:  Perhaps a bit conservative on this one, but with high microseism and rough week locking, didn't want to lose lock.
• 0115 Lockloss (alogged earlier)
• 0224 OBSERVING
• 0357 Start seeing ASC oscilations as well as LSC oscillations.  Winds had picked up for the last hour and around this time the Corner had gusts above 30mph.  Microseism is also above the 90th percentile

H1 just had a lockloss, this was after observing for 30min with H1 ASC Hi Gn to ride out an EQ.

TITLE: 10/25 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 148Mpc
OUTGOING OPERATOR: Ryan S
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 15mph Gusts, 12mph 3min avg
    Primary useism: 0.04 μm/s
    Secondary useism: 0.55 μm/s 
QUICK SUMMARY:

H1 has been locked at NLN over 2hrs (yay!) after a rough week of no observing since Mon night.  Ryan gave me a great summary of the issues/saga that went on all the way up till this morning when RyanS, Sheila, & Elenna fixed H1!

OPS ASSUMPTION:  With H1 appearing back to normal, will assume there's an OWL shift for Tony (unless I hear otherwise)

OPS Handoff from RyanS:

• Keep an eye on Bounce & roll modes---use the scopes on the sitemap/SUS/bounce & roll medm.
• 1hz ASC (look at ASC scope from ISC LOCK medm or the ASC scopes on nuc29

We did just get a warning of an EQ and this one M6 in the South Pacific, AND the EQ Response graph has this EQ squarely on the "HIGH ASC" line.  Because of this (and high microseism and the locking issues of the week), will proactively take H1 out oObserving to transition the ASC Hi Gn button a few minutes before the R-wave arrives (timer set!)

TITLE: 10/25 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 150Mpc
INCOMING OPERATOR: Corey
SHIFT SUMMARY: Finally was able to get H1 back to observing today; see my earlier alogs for details on the efforts there. Since getting recovered, we had one lockloss from an unknown source but were able to relock easily. H1 has now been locked for just over 2 hours.

• 14:47 - Starting initial alignment
• 15:15 - Starting lock acquisition
• 15:56 - Reached LOWNOISE_LENGTH_CONTROL
• 17:28 - Lockloss from ETMY bounce mode in INCREASE_DARM_OFFSET; see alog87737
• 19:06 - Reached NLN; see alog87738 for things that happened around here
• 19:35 - Observing
• 20:20 - Lockloss - alog87739
• 21:49 - Observing

Lockloss @ 20:20 UTC after 1:15 locked - link to lockloss tool

Sadly a lockloss shortly after a triumphant return to observing. Nothing ringing up that I can see, environment is generally calm except for the consistently elevated microseism, and the lockloss felt quick, so no obvious cause here.

Executive summary: H1 returned to observing as of 19:35 UTC after a challenging week.

Following my lock acquisition and troubleshooting attempts this morning, H1 was able to relock fairly easily again all the way up to LOWNOISE_LENGTH_CONTROL. I requested LOWNOISE_ASC and began watching carefully for any rise in the ~9.8 Hz bounce mode, which I saw quickly start to come up in the early seconds of LOWNOISE_ASC. Once the state finished, my first reaction to try and stop this increase was to transition back to high-gain ASC, which I did using the script on the ISI config screen. Looking back in the ISC_LOCK Guardian log, I was reminded that right before LOWNOISE_ASC we had gone through DAMP_BOUNCE_ROLL, which I assumed was not really doing anything since the roll mode damping gain was commented out. However, I saw that in this state, an entry in the LSC output matrix was being set to -1, which I traced down to being the entry for sending DARM control to ETMY, meaning we likely have been unintentionally exciting the ETMY bounce mode. After confirming that this was very much incorrect, I set the matrix value back to 0 and almost immediately saw the 9.8 Hz peak and the bounce mode monitors start to drop. After a few minutes of watching things calm down, I transitioned back to lownoise ASC and saw no sign of the mysterious hump around 60 Hz that had been seen yesterday. With things looking good so far, I requested NOMINAL_LOW_NOISE, which H1 made it to without issue.

It appears the line in the DAMP_BOUNCE_ROLL state code to send DARM control to ETMY has been there for some time, and a comment there says it's to allow for roll damping on ETMY, which would use DARM as the error signal, so this makes some sense. Before yesterday, this state was being run right before we power up from 2W, and a later state would correct the LSC matrix settings so that no erroneous actuation was being sent to ETMY. To remedy this, I have commented out the line that sets the LSC matrix from DAMP_BOUNCE_ROLL. The roll damping we have been using is still commented out as well, so this state essentially does nothing right now and remains between LOWNOISE_LENGTH_CONTROL and LOWNOISE_ASC.

Soon after reaching NLN, I noticed the familiar 1 Hz ASC oscillation was starting to ring up, seen mostly in CSOFT_P and INP1_P.  At Elenna's direction, I increased the CSOFT_P gain from 20 to 25, and the oscillation started to turn around and subside after a few minutes. Elenna has updated ISC_LOCK to set the final gain of CSOFT_P to 25, and commented out the 30 minute reduction of the gain in the THERMALIZATION node.

Elenna and I then tackled the outstanding SDF diffs, which all ended up being accepted, and are documented in the attached screenshot.

Since squeezing looked like it could have been improved, I ran SQZ_MANAGER through 'SCAN_SQZANG_FDS', which noticeably improved DARM at high frequency.

With things looking about as wrapped up as they could be, I set H1 to start observing for the first time in a few days at 19:35 UTC.

Started the day by running an initial alignment and relocking up to LOWNOISE_LENGTH_CONTROL with no issues along the way. Eventually spent 1hr 20min in this state and took a PUM/ESD crossover measurement using the template userapps/lsc/h1/templates/PUM_crossover_2024.xml, see attachment. Sheila confirms even with the lower coherence around the crossover frequency of 20 Hz, this is a good measurement and lines up well enough with the 2024 reference.

I then requested 'INCREASE_DARM_OFFSET' since that's where Ryan C. notes he started to see the ~9.8 Hz bounce modes start to ring up and waited. I soon started to see the ETM bounce modes (at least according to the monitors) increasing. I tried applying a damping gain of 1 on ETMX with the filters already set and saw a small response, but the mode was still increasing. Sheila suggested lowering the DARM offset to 7 from 10.75 to possibly give more time, so I did. Changing the ETMX damping gain around some more did affect the mode (especially turning it off), so it may be possible to damp these, but it's also very possible these filters are very outdated. I also transitioned to high-gain ASC between my damping gain steps of 1 and 2 and didn't see an appriciable change in the mode. I didn't get a chance to try many different settings on either ETM before we lost lock, and since the OMC DCPDs weren't close to saturating, lowering the DARM offset may not have helped after all. See ndscope screenshot for a summary of these attempts. The roll mode damping is still commented out in ISC_LOCK, so it never came on during all of this, but I never saw the mode increase at all.

On the next lock acquisition, I'll go slower after LOWNOISE_LENTH_CONTROL to see if I can get a better idea of where this ringup starts.