I** have written a program to compare slow controls channels before and after with Wed 10sep2025 power glitch to see if there are any which look like they may have been broken and need further investigation.

(** Full disclosure, it was actually written by AI (Claude-code) and runs on my GC laptop safely contained in a virtual machine running Deb13 Trixie)

As a first pass, the code is looking for dead channels. These are channels which were active before the glitch, and are flat-line zero following.

The code gets its channels to analyze from the slow controls INI files.

Using Jonathan's simple_frames python module, it reads two minute trend GWF frame files. For before I'm using the 10:00-11:00 Wednesday file (an hour before the glitch) and for after I'm using the Thu 00:00-01:00 file. In both cases H1 was locked.

I'll post the results as comments to this alog.

TITLE: 09/15 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
OUTGOING OPERATOR: None
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 6mph Gusts, 3mph 3min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.09 μm/s
QUICK SUMMARY: I'll start by taking H1 through an initial alignment then relock up through CARM_5_PICOMETERS, since it sounds like that's still the latest stable state H1 can get to.

TITLE: 09/15 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: None
SHIFT SUMMARY:

IFO is in IDLE for CORRECTIVE MAINTENANCE

The mission today was to get to RESONANCE. We got really close!

The idea was that the calculated TR_CARM offset was likey incorrect so we had to step the offset to maintain stability as we went into resonance. So attempts below are all from CARM_TO_5_PICOMETERS.

While sitting in CARM_TO_REFL during handoff, took RF_DARM TF (attached).

I'm stealing Ryan S's format for his IFO troubleshooting today since it was so easy to read! (Thanks Ryan!)

Reference for healthy CARM to RESONANCE transition pre-outage: 2025/09/09 20:01 UTC, GPS: 1441483301

Relocked,

• TR_CARM_OFFSET set -52
• Lockloss ~2-3s into CARM_TO_REFL
• Could be a 5.3 EQ but unlikely

Relocked,

• TR_CARM_OFFSET set -52
• Lockloss ~2-3s into CARM_TO_REFL
• Alignment looked bad judging from INP1_INMON, buildups and AS_AIR so touched PRM to make better, which worked.
• Lockloss at 10s after CARM_TO_REFL, but the state finished
• Conclusion: alignment was still bad and more touching didn't help

Ran Initial Alignment - fully auto

Relocked,

• TR_CARM_OFFSET to -52
• CARM_TO_REFL Successful
• Sheila's advice was to step slowly into RESONANCE with the first line of code and see what happens, which I did
• SRM Saturated at 02:21:18 UTC (attached)
• Lockloss at 02:21:20 (DRMI)
• Looked at OPLEVS but no indication that they saw anything (attached)

Relocked,

• Took DHARD_P SS 2W OLG TF per Shela's instruction - no indication and the measurement didn't really look good due to low coherence. Phase was ok.
• Sheila came online
• TR_CARM_OFFSET to -52
• CARM_TO_REFL reached but PRG unstable
• Lockloss ~3s after

Plan now was to try to step the TR_CARM_GAIN up as we were stepping the TR_CARM_OFFSET down further from -52 to keep the UGF consistent and not cause a LL. Sheila plannd to measure this each time we stepped it.

Relocked,

• TR_CARM_OFFSET to -52, UGF measured, TR_CARM_GAIN to 1.2 (from 1)
• TR_CARM_OFFSET to -53, UGF measured, TR_CARM_GAIN to 1.4
• TR_CARM_OFFSET to -54, UGF measured, TR_CARM_GAIN to 1.7
• TR_CARM_OFFSET to -56, UGF measured, TR_CARM_GAIN to 2.1
• Sheila stepped into resonance by trying to split up the offset reduction (1st line of RESONANCE) but this failed either due to the problem or because there was 0 ramp time
• Lockloss

IFO in IDLE per Sheila's Instruction. We will try again tomorrow.

LOG:

None

TITLE: 09/14 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: Ibrahim
SHIFT SUMMARY: More troubleshooting today on H1 locking, primarily focused on getting through the CARM_TO_REFL transition and RESONANCE states. See my morning alog for details on the first half of the day; the rest I list here:

• After a discussion online between people, decided using a different TR CARM offset may be the path forward
• Reverted my earlier change of LSC_TR_REFLAIR9_CARM_TO_REF gain in lscparams back to -0.8
• Relocked - held in CARM_5_PICOMETERS and stepped TR_CARM offset
  • Took TR CARM transfer function between steps to check UGF doesn't get too low
  • Settled on -52 as a good offset
  • Moved to CARM_TO_ANALOG, PRG got noisy, waited here about 10 sec
  • Moved to RESONANCE, PRG less noisy, but DHARD control signals got glitchy and lockloss
  • 2 Hz ringup seen in DHARD and DARM
• Relocked - after CARM_5_PICOMETERS lowered TR CARM offset to -52
  • Requested CARM_TO_ANALOG, lockloss soon after entering RESONANCE
• Relocked - lowered TR CARM offset again after CARM_5_PICOMETERS
  • Turned off DHARD loops
  • Requested CARM_TO_ANALOG; lockloss at same place in RESONANCE
  • Realized RESONANCE turns DHARD gains back on, temporarily commented these lines out for next attempt
• Relocked - did NOT lower TR CARM gain
  • Turned off DHARD loops
  • Requested RESONANCE, lockloss at same place
• Relocked - lowered TR CARM offset again after CARM_5_PICOMETERS
  • Turned off DHARD loops
  • Requested RESONANCE, lockloss at same place
• Relocked - lowered TR CARM offset even lower to -55
  • Left DHARD on
  • Lockloss before even leaving CARM_5_PICOMETERS
  • -55 is too low of an offset
• Relocked - lowered TR CARM offset to -52
  • Left DHARD on
  • Lockloss at start of RESONANCE again
• Relocked - lowered TR CARM offset to -52
  • Uncommented DHARD gains in RESONANCE
  • Moved to CARM_TO_ANALOG, PRG noisy but otherwise stable
  • Ibrahim starts taking measurements

Generally locking has been very easy today up through CARM_5_PICOMETERS and alignment has been good.

TITLE: 09/14 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
OUTGOING OPERATOR: Ryan S
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 10mph Gusts, 6mph 3min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.12 μm/s
QUICK SUMMARY:

IFO is DOWN for CORRECTIVE MAINTENANCE

Ryan S has tried an exhaustive combination (I will link the alog when posted) of lock attempts to get beyond where we were yesterday, which was losing lock upon loading the matrix necessary to get from CARM_5_PICOMETERS to CARM_TO_REFL to RESONANCE.

I don't have any ideas of what to try but will be in communication with comissioners.

Below is my outline of locking attempts so far this morning, with any changes I made in Guardian or otherwise in bold:

• Plan to start w/ initial alignment then allow ISC_LOCK to go up to 'RESONANCE' if possible
  • Sheila messages saying CARM_TO_ANALOG won't work; changed ISC_LOCK line 2826 from 2 to 8
• Alignment goes without issue, just needed some SRM adjustment
• Lockloss early in 'RESONANCE', maybe actually something at end of CARM_TO_REFL like Ibrahim found last night
  • DHARD control signals got very glitchy
• Relocked - lockloss while waiting in CARM_TO_REFL after it finished; definitely end of this state (matrix loading?) is the issue
• While relocking, lockloss in DHARD_WFS, seems like alignment was just bad
  • Added SRC loops back into DRMI ASC since they should work after Elenna's dark offset updates
  • They do
• Relocked - holding in CARM_5_PICOMETERS while stepping CHARD P and Y to try and increase LSC-REFL
  • Not seeing much appreciable improvement
  • Reverted Sheila's REFLAIR phase changes from yesterday
  • Tried CARM_TO_REFL again; lockloss
  • SDF_REVERT reverts my phase reversion
• Relocked - held in CARM_5_PIVOMETERS to measure TR CARM transfer function
  • Gain looks too high compared to Feb 2023 reference (screenshot 1), changed LSC_TR_REFLAIR9_CARM_TO_REF gain in lscparams from -0.8 to -0.46 (might undo this later)
  • Remeasured (screenshot 2)
  • Tried CARM_TO_REFL again with new gain; lockloss

Summary of what we've changed so far since the overnight lock where IMC visibility degraded: (strting some notes, plan to add links later).

• reduced power on IMC refl diode
• rephased REFLAIR 9 + 45 for DRMI acquisition
• dark offsets for AS_C and AS 72, this makes DRMI ASC work again
• reduced tr_REFLAIR gain to match a reference from 2023, this did not help us make it through that transition, might be something we want to undo.

Sun Sep 14 10:08:17 2025 INFO: Fill completed in 8min 14secs

TITLE: 09/14 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
OUTGOING OPERATOR: None
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 30mph Gusts, 24mph 3min avg
    Primary useism: 0.04 μm/s
    Secondary useism: 0.11 μm/s
QUICK SUMMARY: Windy and rainy morning on-site. I'll start by running H1 through a fresh initial alignment, then continue troubleshooting where Ibrahim left off last night.

TITLE: 09/14 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: None
SHIFT SUMMARY:

IFO is in IDLE and CORRECTIVE MAINTENANCE

The "good" news:

DHARD P loop has been successfully automatic since closing it this afternoon. We can lock all the way to CARM_5_PICOMETERS quite quickly. DRMI has also been good to me this evening.

The bad news:

We thought we were losing lock at RESONANCE, but we were really losing lock at CARM_5_PICOMETERS.

Stepping through this state has showed that its only at the end of this, where the matrix involving REFL_BIAS, TR_REFL9 and TR_CARM is loaded, we lose lock roughly 10s later due to SRM becoming unstable quickly.

I investigated this via alogs 86909, 86910, 86911, 86912, which are comments to my OPS shift start (alog 86908). After being led down a rabbit hole of ramp times, from 3 other times where that was the problem, I can confirm that this isn't it. Curiously, lock was lost faster with a longer ramp time.

With confirmation from TJ and Jenne, I'm leaving IFO in IDLE with the plan being to solve this problem in tomorrow's shift.

I do feel like we're close to solving this problem or at least figuring out where the issue lies. Specific details are in the alogs listed below.

LOG:

None

TITLE: 09/13 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
OUTGOING OPERATOR: Ryan S
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 17mph Gusts, 11mph 3min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.15 μm/s
QUICK SUMMARY:

IFO is DOWN in CORRECTIVE MAINTENANCE

There was a lot of good work done today. Primarily rephasing REFLAIR (alog 86903) and closing DHARD_P loop. This has allowed us to easily get past DRMI. It seems that we are still losing lock around resonance due to a 35hz fast ringup.

The Procedure for the EVE:

1. Try DHARD_WFS and step to resonance (state before lockloss). If DHARD_WFS doesn't woprk,
   1. comment out line 2430, 2404 of ISC_LOCK.py
   2. Then go to CARM_5_PICOMETERS and turn DHARD_P Gain to -1, then -10. This is slow
2. If DHARD_WFS works, keep going and step through states by:
   1. once at previous state of the one you want to step
   2. "guardian -i ISC_LOCK"
      1. Copy the ezca lines
      2. copt intrix load lines
      3. do NOT copy "self.lines" or the library - that's done by the guardian -i
   3. Copy line by line and see it work. If state is done, MANUAL to state AFTER desired state

What has happened so far:

Same PR_G ringup caused lockloss at RESONANCE and CARM_TO_REFL so I'm going to the state before, which is CARM_5_PICOMETERS. Will follow the blueprint above. Thankfully DHARD_P worked 2x now.

I posted the following message in the Detchar-LHO mattermost channel:

Hey detchar! We could use a hand with some analysis on the presence and character of the glitches we have been seeing since our power outage Wednesday. They were first reported here: https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=86848 We think these glitches are related to some change in the input mode cleaner since the power outage, and we are doing various tests like changing alignment and power, engaging or disengaging various controls loops, etc. We would like to know if the glitches change from these tests.