Result 1: We can make it fine up through DHARD WFS. Reminder, DHARD WFS now comes after CARM OFFSET REDUCTION. CARM 5 PM still is the problem. The AS air camera shows lots of moving beams during offset reduction without DHARD on, but that seems repeatably ok for locking. (The first time we went through this sequence we were still in earthquake mode, which made this a fun stress test.)

Below are the locklosses experienced.

Lockloss #1:

• occurred during CARM 5 PM state. guardian log:

2025-09-19_00:05:44.210977Z ISC_LOCK executing state: CARM_5_PICOMETERS (309)
2025-09-19_00:05:44.211404Z ISC_LOCK [CARM_5_PICOMETERS.enter]
2025-09-19_00:05:44.222412Z ISC_LOCK [CARM_5_PICOMETERS.main] timer['CARM_ramp'] = 10.0
2025-09-19_00:05:48.263712Z ISC_LOCK [CARM_5_PICOMETERS.run] ['REFL_DC at reference offset: ', '1.81979900598526']
2025-09-19_00:05:48.263712Z ISC_LOCK [CARM_5_PICOMETERS.run] ['Using LSC-TR_CARM_OFFSET of ', '-47.13799969222334', ' for last step of CARM reduction.']
2025-09-19_00:05:48.263712Z ISC_LOCK [CARM_5_PICOMETERS.run] ezca: H1:LSC-TR_CARM_TRAMP => 5
2025-09-19_00:05:48.263712Z ISC_LOCK [CARM_5_PICOMETERS.run] ezca: H1:LSC-TR_CARM_OFFSET => -52
2025-09-19_00:05:53.400224Z ISC_LOCK [CARM_5_PICOMETERS.run] timer['CARM_ramp'] = 5.0
2025-09-19_00:05:57.263417Z ISC_LOCK [CARM_5_PICOMETERS.run] ['REFL_DC at reference offset: ', '0.8649354875087738']
2025-09-19_00:05:58.400264Z ISC_LOCK [CARM_5_PICOMETERS.run] timer['CARM_ramp'] done
2025-09-19_00:06:01.261270Z ISC_LOCK [CARM_5_PICOMETERS.run] ['REFL_DC at reference offset: ', '0.8532866835594177']
2025-09-19_00:06:01.261939Z ISC_LOCK [CARM_5_PICOMETERS.run] ezca: H1:LSC-TR_CARM_GAIN => 2.1
2025-09-19_00:06:01.262711Z ISC_LOCK [CARM_5_PICOMETERS.run] timer['CARM_ramp'] = 5.0
2025-09-19_00:06:05.262141Z ISC_LOCK [CARM_5_PICOMETERS.run] ['REFL_DC at reference offset: ', '0.8590858280658722']
2025-09-19_00:06:06.262933Z ISC_LOCK [CARM_5_PICOMETERS.run] timer['CARM_ramp'] done
2025-09-19_00:06:09.261966Z ISC_LOCK [CARM_5_PICOMETERS.run] ['REFL_DC at reference offset: ', '0.7854519486427307']
2025-09-19_00:06:09.263923Z ISC_LOCK [CARM_5_PICOMETERS.run] ezca: H1:LSC-TR_CARM_OFFSET => -56
2025-09-19_00:06:14.401909Z ISC_LOCK [CARM_5_PICOMETERS.run] timer['CARM_ramp'] = 5.0
2025-09-19_00:06:14.402503Z ISC_LOCK [CARM_5_PICOMETERS.run] ezca: H1:LSC-PD_DOF_MTRX_SETTING_4_17 => -1.952
2025-09-19_00:06:14.505997Z ISC_LOCK JUMP target: LOCKLOSS
2025-09-19_00:06:14.512463Z ISC_LOCK [CARM_5_PICOMETERS.exit]
2025-09-19_00:06:14.573087Z ISC_LOCK JUMP: CARM_5_PICOMETERS->LOCKLOSS
 

So this happened right after the ramp up of PRCL gain by 60% and CARM offset stepped to -56. However, the ring up was a ~63 Hz ring up, and PRCL doesn't go unstable at that frequency, so it's hard to see how the PRCL gain ramp was the cause.

Plan: measure OLGs before CARM 5 and figure out what the issue is. Possible culprits: TR CARM, RF DARM, PRCL

Lockloss #2:

• this time we went to DHARD WFS without intervention
• measured RF DARM, TR CARM and PRCL OLGs
  • PRCL looked fine
  • TR CARM looked fine (little higher than reference trace)
  • RF DARM looked low, increased DARM2 gain from 1 to 1.5
• Did CARM 5 pm steps by hand:
• tr carm offset to -52, fine
• remeasured loops
  • PRCL UGF down to 30 Hz, possibly a bit low, 20 deg phase margin
  • RF DARM gain now a little high, reduced DARM2 gain to 1.4
  • TR CARM gain low by about a factor of 2, now I understand Sheila's next guardian step which is to increase TR CARM gain from 1 to 2.1
• Increased TR carm gain to 2.1 following guardian, fine
• Ramped TR CARM offset from -52 to -56 following guardian
  • BAD, lockloss
  • same ring up, near 65 Hz

Seems like the -56 TR CARM offset is the culprit. Commented out this step in the guardian code.

Decided to reduce the PRCL gain increase from a 60% increase to a 30% increase (since based on the measurement, 60% might be too much). Edited guardian code.

Plan: let the code run with above adjustments, do not adjust DARM gain.

Result 2: SUCCESS. Guardian code tested without intervention.