TITLE: 11/11 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 26mph Gusts, 19mph 3min avg
    Primary useism: 0.05 μm/s
    Secondary useism: 0.40 μm/s
QUICK SUMMARY:
Lockloss from NLN during comissioning tagged as windy: https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi?event=1415382893

Holding H1 in Down with the IMC locked but the ISS turned off to test if the ISS is the cause of the IMC glitches we have been seeing that breaks our locks.
The secondary Microseism is elevated and the Wind is gusting above 30 MPH, so it's a good time to try to test this.
 

We seem to be having lots of locklosses during the transition from ETMX/lownoise ESD ETMX guardian states. With Camilla's help, I looked through the lockloss tool to see if these are related to the IMC locklosses or not. "TRANSITION_FROM_ETMX" and "LOWNOISE_ESD_ETMX" are states 557 and 558 respectively.

For reference, transition from ETMX is the state where DARM control is shifted to IX and EY, and the ETMX bias voltage is ramped to our desired value. Then control is handed back over the EX. Then, in lownoise ESD, some low pass is engaged and the feedback to IX and EY is disengaged.

In total since the start of O4b (April 10), there have been 26 locklosses from "transition from ETMX"

• 14 of which have been since the vent recovery on Aug 21
• 5 of these locklosses are tagged "IMC", all occuring on or after October 4.
• The first lockloss in this state since the vent recovery occured on Sept 13
• There is a variety of state duration times in these locklosses, ranging from 4 seconds to 2 minutes 57 seconds
• most of these locklosses before and after the vent have additional tags, such as "windy", "earthquake" etc. Only three appear to have only "refined" as a tag

From the start of O4b (April 10) to now, there have been 22 locklosses from "lownoise ESD ETMX"

• 5 of these occured before the vent
• none are tagged as IMC
• All but two of the locklosses occured with state durations between 1-3 seconds in the state
• before the vent, the 5 locklosses have additional tags besides "refined", such as "OMC DCPD", or "WINDY". After the vent, most of the locklosses are only tagged as "refined".
• Since the vent, one of these locklosses occured on Aug 23, the rest have occurred on or after Sept 27

Trending the microseismic channels, the microseism increases seems to correspond with the worsening of this lockloss rate for lownoise ESD ETMX. I think, when correcting for the IMC locklosses, the transition from ETMX lockloss rate is about the same. However, the lownoise ESD ETMX lockloss rate has increased significantly.

Checked that there was no significant downstream alignment shift after the IMC WFS were centered on September 10th 80018. Checking because this because this was the same week the "IMC" locklosses started. 

Al downstream (IM4 TRANS and POPA/B)  QPD PIT/YAW values are the same within 0.05 (range -1 to 1) before and after the IMC WFS centering. Some NSUM change in IM4 TRANS, probably as this PD is clipped. Plot attached.

Mon Nov 11 10:17:12 2024 INFO: Fill completed in 17min 9secs

Jordan confirmed a good fill.

After discussions with control room team: Jason, Ryan S, Sheila, Tony, Vicky, Elenna, Camilla 

• Prior to September 13th:
  • No IMC locklosses 80951
  • NPRO/FSS glitches present but not bothering us (June 80520, last year 67718).
• September 12th: IMC gain redistribution 79998
• September 13th: First IMC lockloss
• September to October:
  • We start noticing FSS glitches and do of tests to understand these.
  • They were present in NPRO only with everything else unlocked (part of reason for swapping NPRO)
  • Found that these glitches are not new, present in June 80520 and even last year 67718. Maybe they got worse, or maybe we just stopped surviving them as well.
  • IMC has periods of being unstable (have to wait in DOWN)
• October 9th: IMC gain redistribution reverted 80566
  • We have 6 days with only one IMC lockloss 
• October 15th: NPRO current increased 80687
  • IMC lockloses started again 
• Ocotber 18th: NPRO current reverted 80746
  • IMC lockloses continued
• October 23rd to 29th: NPRO swapped 80929
  • Swapped to new faraday and EOM, both found to be dodgy
• Ocotber 30th: Back to NLN
  • Had issues with NPRO 81073: found to be NPRO mode hopping 
  • IMC locklosses continued 
• October 30th to November 1st: Tests.
  • CARM OLG 80956
  • IMC OLG 80979
  • FSS OLG 80993 
  • If the PSL ISS is unlocked the IMC can stay locked: 81056
  • Attempt to swap TTFSS (reverted) 80983
  • Hardboot of PSL chassis h1psl0: 81048 
• November 6th: PSL mode hopping was fixed by changing NPRO temperature, 81107. 
  • PSL now stable
  • IMC has periods of being unstable
  • IMC locklosses continue 

Conclusions: The NPRO glitches aren't new. Something changed to make us not be able to survive them as well in lock. The NRPO was swapped so isn't the issue. Looking the timing of these "IMC" locklosses, they are caused by something in the IMC or upstream 81155.

Tagging OpsInfo: Premade templates for looking at locklosses are in /opt/rtcds/userapps/release/sys/h1/templates/locklossplots/PSL_lockloss_search_fast_channels.yaml and will come up with command 'lockloss select' or 'lockloss show 1415370858'.

Sheila and I went out to the floor to plug in the cable to run frequency noise injections (AO2).

I switched us to REFL B (79037) only to run the injections. I first ran the frequency noise injection template. Then, I disenaged the 10 Hz pole and engaged the digital gain to run the ISS injections (low, medium, and high frequency). I was fiddling with the IMC WFS pitch injection template to get it to higher frequency for a jitter injection when we lost lock.

Next step is to at least use the intensity and frequency measurements to determine how the noises are coupling to DARM (would be nice to have jitter too, but we'll see).

TITLE: 11/11 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 161Mpc
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 6mph Gusts, 4mph 3min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.33 μm/s
QUICK SUMMARY:

Ryan C was called last night at 9:24 UTC.
When I walked in Sheila was in the Team speak chat, so maybe she was helping out with getting it locked first thing in the morning. 

NLN reached at 15:37  UTC
Observing reached at 15 :40 UTC


 

H1 called for assistance at 09:29 UTC after the initial alignment timer expired from the IMC unlocking an not relocking, we were in PREP_FOR_SRY. I took us to down and the IMC locked and I then went back into locking where DRMI locked very quickly and looked pretty decent.

TITLE: 11/11 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Aligning
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY:

IFO is LOCKING at CHECK_AS_SHUTTERS

Overall a smooth shift where we were locked for two thirds of the shift until the PSL glitch most likely took us down. Since then, the glitch has been happening without much gap for relocking. We've had gaps of 20 or so mins without glitching but that's been the maximum so far since LL. Lockloss alog 81181.

Other than that, we dropped out of OBSERVING very briefly due to a TCSY CO2 Laser SDF diff that fixed itself.

LOG:

None

Closes FAMIS 26016. Last checked in alog 80912. 

Changes from last check:

• HAM8 CPS slightly noisier
• IY ST1 CPSINF2 channel noisy
• BS ST2 CPSINF H1 has a new bump 1-3Hz
• EY ST1 less noisy

Lockloss seemingly PSL caused judging from IMC and ASC-A signals losing lock within 10ms (attached). About a 700ms after the glitching started, we lost lock. Interestingly, IMC is not glitching for minutes post-LL as it usually does, with IMC locking and staying locked in the new acquisition.

TITLE: 11/11 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 157Mpc
INCOMING OPERATOR: Ibrahim
SHIFT SUMMARY:
The morning was a little shakey, but we persevered through the problematic IMC glitches to get to a lock by 22:35 UTC

LOG:                                                                                                                                                                                                                                                                                               

TITLE: 11/11 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 157Mpc
OUTGOING OPERATOR: Tony
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 7mph Gusts, 4mph 3min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.37 μm/s
QUICK SUMMARY:

IFO is in NLN and OBSERVING as of 22:35 UTC

Recovered from a massive 6.8 EQ. Nothing else of note.

Lockloss @ 11/06 00:58UTC

Reran all O4b locklosses on the locklost tool with the most up to date code, which includes the "IMC" tag.

I used the lockloss tool data since the emergency OFI vent (back online ~23rd August) until today and did some excel wizardry (attached) to make the two attached plots, showing the number of locklosses per day tagged with IMC and without tag IMC. I made this plots for just locklosses from Observing and all NLN locklosses (can include commissioning/maintenance times), including:

• No IMC locklosses August to September 13th (Iain checked prior to August G2401576)
• Setpeber 12th: IMC gain redistribution 79998
• October 9th: IMC gain redistribution reverted 80566
• October 15th: NPRO current increased 80687
• Ocotber 18th: NPRO current reverted 80746