Still Relocking after last NLN lockloss at 23:15 UTC - experiencing a whole host of problems

In the meantime we've had the following issues, with locklosses listed in the lockloss alog 77615:

- BS Glitch and other DRMI locklosses 8 times. The glitch locklosses (4 of these 8 losses) were overcome by waiting at ENGAGE_DRMI_ASC while ASC converges to mitigate the glitch kick. That being said, DRMI is extremely unstable in ENGAGE_DRMI_ASC and lost lock here 2 times right as ASC converged.

- Unknown locklossest at TRANSITION_FROM_ETMX and after initial alignment, LOWNOISE_ESD_ETMX

- IR Not found (also after initial alignment).

I already ran initial alignment and don't think this is the problem. Have troubleshot using Jenne's ppt to no avail. The problem is that I essentially have to wait 1 hr (for the EX Transition issue) before experiencing a lockloss. Having done an initial alignment between EX Transition locklosses, I think it's worth assuming that that didn't fix the issue so I'll resort to the call list when I'm near EX Transition.

While trying to re-lock, I ran into the same issues that Ryan C and Sheila noted in their alog 77574 (most recent lock) where a glitch in the state ENGAGE_DRMI_ASC to the state TRANSITION_DRMI_TO_3F thought to be BS ST2 causes a lockloss - elevated primary microseism probably makes us more susceptible.

As done in the last Lock Acquisition (survey), staying at ENGAGE_DRMI_ASC until ASC converges (such that lockloss chance from the glitch can be minimized) worked to overcome the glitch, which had a markedly smaller effect.

Once we get to NLN, I will investigate this as noted in Sheila's comment (alog 77575). Additionally, alog 77573 talks briefly about this issue and lists where the glitch was seen. While this happens, here are some more trends of the glitch being seen in multiple BS channels (again, and again and again). Will plan to post another alog summarizing any interesting findings.

TITLE: 05/04 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 151Mpc
INCOMING OPERATOR: Ibrahim
SHIFT SUMMARY:
H1 is Still locked and Observing for almost 43 Hours!

SUS Violin update:
After the Calibration sweep I increased the magnitude of the gain on EXmode9 again from -3 to -6.
I kept FM1 and FM10 selected. It's been working all day with values between -0.1 and -6.
Basically all day I was incrementally increasing this gain I started at -0.01 and exponentially increased it until I hit what I thought was the drive thresh hold of 1100-1200 until the Calibration sweep.  

Dropped into Commissioning to adjust the squeeze angle.
22:43 UTC back to OBSERVING.

LOG:
No log

TITLE: 05/04 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 139Mpc
OUTGOING OPERATOR: Tony
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 21mph Gusts, 19mph 5min avg
    Primary useism: 0.09 μm/s
    Secondary useism: 0.11 μm/s
QUICK SUMMARY:

IFO is in NLN and OBSERVING (42 min 51 hr lock!)

Nothing else of note

Calibration Sweep notes:
 I waited until the Drive Actuations for the Sus EX mode9 Violin were below 800 before running the Calibration sweep to hopefully minimize lockloss potential.
Ran the Following at 20:40 UTC ~GPS: 1398890536:
pydarm measure --run-headless bb 

notification: end of test
diag> save /ligo/groups/cal/H1/measurements/PCALY2DARM_BB/PCALY2DARM_BB_20240504T204058Z.xml
/ligo/groups/cal/H1/measurements/PCALY2DARM_BB/PCALY2DARM_BB_20240504T204058Z.xml saved
diag> quit
EXIT KERNEL

2024-05-04 13:46:08,970 bb measurement complete.
2024-05-04 13:46:08,970 bb output: /ligo/groups/cal/H1/measurements/PCALY2DARM_BB/PCALY2DARM_BB_20240504T204058Z.xml
2024-05-04 13:46:08,970 all measurements complete.

Then the following was ran :
gpstime;python /ligo/groups/cal/src/simulines/simulines/simuLines.py -i /ligo/groups/cal/src/simulines/simulines/newDARM_20231221/settings_h1_newDARM_scaled_by_drivealign_20231221_factor_p1.ini;gpstime

Start:
GPS: 1398890967.961527
Stop:
GPS: 1398892259.616687

2024-05-04 21:10:41,542 | INFO | File written out to: /ligo/groups/cal/H1/measurements/DARMOLG_SS/DARMOLG_SS_20240504T204910Z.hdf5
2024-05-04 21:10:41,550 | INFO | File written out to: /ligo/groups/cal/H1/measurements/PCALY2DARM_SS/PCALY2DARM_SS_20240504T204910Z.hdf5
2024-05-04 21:10:41,555 | INFO | File written out to: /ligo/groups/cal/H1/measurements/SUSETMX_L1_SS/SUSETMX_L1_SS_20240504T204910Z.hdf5
2024-05-04 21:10:41,560 | INFO | File written out to: /ligo/groups/cal/H1/measurements/SUSETMX_L2_SS/SUSETMX_L2_SS_20240504T204910Z.hdf5
2024-05-04 21:10:41,565 | INFO | File written out to: /ligo/groups/cal/H1/measurements/SUSETMX_L3_SS/SUSETMX_L3_SS_20240504T204910Z.hdf5

TITLE: 05/04 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 152Mpc
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 20mph Gusts, 17mph 5min avg
    Primary useism: 0.06 μm/s
    Secondary useism: 0.11 μm/s
QUICK SUMMARY:
H1 is still locked after 40 hours, and still OBSERVING!

SUS Violin update:
EX mode9 is still elevated but is going down. I have been adjusting the gain H1:SUS-ETMX_L2_DAMP_MODE9_GAIN to increase the rate that it declines while also trying to make the drive actuations ( H1:SUS-ETMX_L2_DAMP_MODE9_OUTPUT )  less than 1200 counts but greater than 800 to maximize the rate that we damp the Violin mode.

CAL Sweeps:
I have requested (from Virgo & LLO)  that the Saturday Calibration time happen at 20:30 UTC to allow time for the Violin to damp more before calibration sweeps. All sites have agreed to 20:30 UTC Commissioning time.


 

Sat May 04 10:12:16 2024 INFO: Fill completed in 12min 12secs

Tony, Rahul

ETMX09 was rung up (even though Ibrahim had set the gains off last night) and Tony was trying to fight the Guardian to try and damp this mode early this morning. We were out of Observing (for 15mins while we were figuring it out) and struggling to get in (violins too high message). Hence, at first we set the nominal values of gain in lscparam to zero, then set the violin damping guardian to Damp_Violins_Full_Power and then applied a fraction of the nominal gain. This seems to be slowly damping the mode.

Also then Tony was able to move back to Observing.

Will keep a watch over this mode (and others if needed) until we fully damp it.

TITLE: 05/04 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 151Mpc
OUTGOING OPERATOR: Ryan S
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 11mph Gusts, 7mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.09 μm/s
QUICK SUMMARY:

H1 has been locked and OBSERVING all night.
But the Violin ETMX mode 9 has been on the rise fore the past 34 hours! and is now very high!
I will be trying to damp this mode first thing!

TITLE: 05/04 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 155Mpc
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY:

IFO is in NLN and OBSERVING (26hr 46 min lock)

Violin Ring-Up:

From 04:40 UTC to 05:14 UTC, went in and out of observing to try to damp violin ETMX mode 9. The nominal gain did not work nor did testing different gains and (and turning on the angle filters). Checked the wiki/alog also but to no avail (initially went to the violins wiki page and Rahul's ppt to go to damp simple when Guardian was fighting me with gain issues). This violin has been growing slowly since the lock acquisition 26 hrs and 39 minutes ago. (screenshot). During this time, the violin count magnitude increased seven fold. Pinged Rahul and expecting help with it tomorrow.

SQZ Unlock:

Squeezer unlocked at 05:20 UTC but managed to relock at 5:29 UTC. I did have to play around with guardian states because guardian notified me that the SQZ ISS had saturated and to go to alog 70050 from May 2023. As I was playing around with the states, the SQZ relocked automatically. SQZ lockloss was potentially caused by a time corellated local earthquake that showed up on the EQ peak_outmon very sharply but was not high magnitude enough to cause us to go into EQ mode.

LOG:

IFO is still in NLN and OBSERVING (22 hr 40 min lock) at 156 MPc

Nothing else of note

TITLE: 05/03 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 133Mpc
INCOMING OPERATOR: Ibrahim
SHIFT SUMMARY:
H1 was in Observing the entire length of my shift. 
H1 has been locked and Obsereving for almost 19 hours.
LOG:

TITLE: 05/03 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 155Mpc
OUTGOING OPERATOR: Tony
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 14mph Gusts, 10mph 5min avg
    Primary useism: 0.04 μm/s
    Secondary useism: 0.08 μm/s
QUICK SUMMARY:

IFO is in NLN and OBSERVING (18hr 45 min lock)

Nothing else of note

[Anamaria, Tony, Jenne]

Anamaria has updated the L1 trace in our DARM FOM, so we can more easily see the difference between the sites. Tony made sure it's on the front wall TV via the launcher.

Attached is a comparison of the DARM spectrum of our good time April 11th (164 Mpc), compared to now that we have mostly recovered from our OFI incident (161Mpc).  Our sensitivity below 25Hz is slightly better than before, and above 1kHz the squeezing is a bit worse.

There is a bruco running which will appear here soon: https://ldas-jobs.ligo.caltech.edu/~sheila.dwyer/brucos/CLEAN_1398770931/

TITLE: 05/03 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 156Mpc
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 10mph Gusts, 7mph 5min avg
    Primary useism: 0.04 μm/s
    Secondary useism: 0.09 μm/s
QUICK SUMMARY:
H1's current lock has just hit 17 hours of Observing.
Wind speed is ramping up.
No plans to drop out of Observing.

We had several locklosses from DRMI in the last hour.  PRCL was oscillating at just above 100 Hz, because it's gain was too high.  I reduced the PRCL1 gain from 8 to 4.  I then remeasured once DRMI was transitioned to 3F, the gain was now too low by a factor of 2, so I set it back to 8. 

I won't make this change in the guardian, because I don't know if this is consistent or not.