Got called at 10:57PM (PT) and IFO was actually locked in NLN with one SDF Diff, which I accepted and is screenshot is below (IMC_DOF_1_P TRAMP from 0 to 3 and IMC_DOF_2_Y TRAMP from 0 to 3).

Good timing for this since the 1Hz ringup was happening. I saw earlier today that turning on the ASC High Gain might help with this so I turned it on and it seems to be working.

SQZ also looked terrible inthe high-frequency domain. I ran the OPO temp scan and the SQZ Angle, but this did not seem to improve anything (plot attached).

Additionally, the EY Roll Ringup that plagued Ryan C's EVE shift is indeed happening again. I'm unsure what to do about this since it seems he and Rahul tried a multitude of damping techniques without avail.

ASC High Gain was turned off ~5 minutes after it seemed to be working. A minute later, we it rang up again and we lost lock.

So there's two issues (or maybe they're related I don't know):

1. 1Hz Ringup that can be survived while locked but only in High Gain settings

2. ETMY Roll mode which just rings up consistently

Out of ideas - will continue to investigate but nothing seems to be helping.

TITLE: 10/23 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: Ibrahim
SHIFT SUMMARY: We made it to NLN but the ETMY roll mode took us down, it appears to have rung up each time we get into OMC_WHITENING since maintenance.

I found new settings for ITMX mode13, FM1+FM2+FM10 G= -0.5. 

The first 2 aquisitions we saw the Roll mode ringup but this 3rd time we did not, we slowly brought the IFO up, pausing to look at the mode in CHECK_VIOLINS, 10Ws, 25Ws, MAX_POWER,  TRANSITION_FROM_ETMX, and LOWNOISE_ESD_ETMX.
LOG: No log.

• 00:43 UTC We reached OMC_WHITENING
  • ETMY roll mode starts to ringup
  • 01:35 UTC lockloss from ETM glitch?
• 02:21 UTC OMC_WHITENING again, we're close to the limit so it shouldn't take very long
  • The ETMY roll mode looks to slowly increasing again, I'm going to try to damp it
  • 02:39 UTC NLN
    • SQZ looked bad, so I ran SCAN_OPO_TEMP then SCAN_SQZ_ANG, still looked bad after
    • I had some SDFs to accept, mostly offsets (ALSEY & LSC, ISCEY & ASC & ASCIMC, ALSEX & ISCEX & OMC)
    • I never made into Observing as I was fighting the ever ringing up ETMY roll mode until it made us lose lock
      • The 1Hz oscillation also showed up, as soon as I turned FM4 back on the Oscillation returned
• 03:13 UTC lockloss from ETMY roll mode
  • Rahul logged on to take a look with me, we paused at various states
  • 04:40 UTC NLN
    • 1Hz showed up, I turned off FM8 waited until it was fully off then FM4, as soon as FM4 is turned off and on the 1Hz seems to appear on DHARD
    • To turn them back on go in reverse, slowly
    • Something is growing, as seen in ASC-INP1 and LSC-PR_GAIN, Its at 1Hz even though I turned off the DHARD_P filters?
    • 04:52 UTC lockloss from the oscillation I'm assuming

I had been in NLN for 35 minutes fighting the ETMY roll mode, I also ran some SQZ stuff since it looked terrible (OPO_TEMP_SCAN, then ANG_SCAN) while in NLN. I was unsuccessful in damping the mode and it eventually grew saturated the DCPDs and killed the lock, there was also increased ground motion from a 5.6 from eastern Russia during this. In the middle of fighting the roll mode I saw the 1Hz oscillation start in DHARD_P so I turned off FM8 then FM4 on DHARD_P which fixed it.

Turns out that I had forgotten to turn the input on for the PR3 P estimator contribution from L (H1:SUS-PR3_M1_EST_P_FUSION_MODL_DRV_L_2GAP) (thanks to Edgard and Brian for having us even consider the estimators!), and that was causing the 0.6 Hz ASC ringups (87658). Even though we're pretty sure we found the cause, we've decided to temporarily turn off the PR3 and SR3 L and P estimators until we're back up and can verify that they are now working correctly.

TITLE: 10/23 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY: Currently relocking and at LOWNOISE_COIL_DRIVERS. Still struggling with a 1 Hz oscillation(87658), but we've solved issues with MICH phase margin and with a 0.6 Hz oscillation caused by incorrect estimator settings, so 2/3 of the issues solved?
LOG:

14:30UTC Started relocking
    Initial alignment
    Lockloss from OMC_WHITENING due to 1 Hz ringup
    Multiple locklosses from DRMI and under due to ALSX getting noisy and ALSY having some glitchy drops in power
    Locklosses from LOWNOISE_LENGTH_CONTROL, MOVE_SPOTS, etc.                                                                                                                                                                

We managed to track down the source of the 0.6 Hz oscillation that we see at move spots and full power. It seems to be the result of the new pitch and length estimators on PR3 and SR3 that were installed on Tuesday. We tested this by sitting at move spots (after the spots converged), and turning on and off the estimators for a few minutes at a time. When turning ON the estimators, the 0.6 Hz ringing is not immediately apparent- it will slowly grow. However, once the 0.6 Hz oscillation is visible in the controls, turning OFF the estimator immediately stops the ringing. Therefore, we think this is the culprit for this instability. Oli will be turning off the length and pitch estimators for PR3 and SR3 for now. The yaw estimators are still on because they have been operating stably for a long time and were not changed this Tuesday.

We have also put the factor of 2 increase of MICH2 gain into the guardian, since we kept forgetting to add it in. Several locklosses that we thought were due to various other things were actually due to the MICH loop ringing up.

Finally, we just had a lockloss after moving to camera servos because the 1 Hz oscillation began to grow. Here is everything I tried to stop the ringing (none of it worked):

• CSOFT P gain was already at 30. I dropped it to 25 and re-increased to 30 several times, no change either way
• INP1 P gain was at -1, I doubled it and then tripled it, no change
• INP1 Y gain was at -0.1, doubled it, no change
• DSOFT P gain was at 5, I increased to 10, then 15, no change
• DSOFT Y gain was at 5, increased to 10, no change

I then had an idea to go to high bandwidth ASC using the seismic script, but we lost lock as soon as I thought of that.

Next plan: go to lownoise length control, skip lownoise asc, go to the final spots and camera servos, stay in HBW ASC and wait for the 1 Hz ring up and see if we can ride it out this way.

Attached scope screenshot shows the 0.6 Hz oscillation disappearing as soon as Oli flipped the length and pitch estimator switches OFF.

TITLE: 10/22 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
OUTGOING OPERATOR: Oli
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 21mph Gusts, 17mph 3min avg
    Primary useism: 0.05 μm/s
    Secondary useism: 0.30 μm/s 
QUICK SUMMARY:

• Microseism is decreasing over the past ~14 hours
• Wind has been rising the past 2.5 hours
• We're at MAX_POWER currently

Sheila, TJ, Oli, Matt

We reverted the ASC gain changes (87582) setting INP1 P to 1 and CSOFT P to 25 at MAX power (rather than 30).  Most of our locklosses overnight were from states before Max power, so the CSOFT P gain can't have been the problem, but we are undoing it for now just for the sake of reverting recent changes. 

We looked at one of the overnight locklosses, it seems that MICH is ringing up at 4.7 Hz.  Since Oli is sitting at 2W DCreadout to damp violin modes, we measured the MICH OLG there, and we see it is pretty close to unstable at 5Hz, with low coherence there.  We added a factor of 2 gain in LSC MICH2, to bring the UGF to 10 Hz similar to the references, and give us 35 degrees phase margin. 

Next we checked the SRCL UGF, which is also locked on POP45, and it's gain looks low by a factor of 5 compared to the reference.  This loop has a large phase bubble so it's still stable. POP 45 whitening gain and filters seem to be the same as a few days ago, and the POP LF power is the same as it was a few days ago at 2W.  PRCL gain also looks low compared to the reference, but is stable.  

We were able to get to OMC_WHITENING (almost to low noise), but while we were sitting there we had the old ring up.  We tried manually increasing the CSOFT P gain to 30, but the osciallation had already grown large and we lost lock anyway.  I've now put the csoft P gain back to 30 in max power in the guardian.