TITLE: 02/18 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
    SEI_ENV state: USEISM
    Wind: 2mph Gusts, 1mph 3min avg
    Primary useism: 0.05 μm/s
    Secondary useism: 0.47 μm/s
QUICK SUMMARY: Lost lock at 1507 UTC (0707 PT). Maintenance day today. No active alarms.

Workstations were updated and rebooted.  This was an OS packages update.  Conda packages were not updated.

TITLE: 02/18 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 149Mpc
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY: Currently Observing and have been Locked for almost 1 hour.

Two locklosses during my shift, and they both had that weird oscillation in DARM right before the lockloss (see 82847). Relocking went okay as long as I ran an initial alignment.

First relocking:
At the start of the initial alignment during green arms, I did get the ALSX issue where it's locked but the WFS won't turn on. I Forced the auto-centering for the WFS and that was all it needed.
Once we got back up to NLN we were having issues with the filter cavity losing lock (similar to the issues from the rest of this weekend). It was losing lock at different places in the locking process (FIND_IR, GR_VCO_LOCKING, TRANSITION_IR_LOCKING, etc). I tried just running SQZ_MANAGER through DOWN and back up but that didn't work. I then tried moving FC1&2 a bit to get them back to where they were during our last lock but that also didn't work. I then just tried what Ryan S did yesterday (82840) - I paused SQZ_MANAGER in LOCK_CLF and adjusted the OPO temp. That worked well and we were then able to lock the filter cavity, start squeezing, and get back into Observing. Tagging sqz
Second relocking:
I just made sure to immediately run an initial alignment and there were no issues getting back to NLN and Observing.
LOG:

00:06 Lockloss
    - started an inital alignment
        - ALSX was locked (but oscillating weirdly) around 1 but the WFS wouldn't turn on - the auto-centering was off (we've been seeing this recently sometimes). I Forced the auto-centering and the WFS turned on and took ALSX up to 1.2 right away
    - Lockloss from TURN_ON_BS_STAGE2
    - Lockloss from PRMI_ASC

02:20 NOMINAL_LOW_NOISE
    - FC losing lock during different stages in the locking process (FIND_IR, GR_VCO_LOCKING, TRANSITION_IR_LOCKING, etc.)
    - Ran SQZ_MANAGER through DOWN and back up - didn't work
    - Adjusted FC1&2 a bit to get them back to where they were during our last lock - didn't work (did not revert these changes)
    - Did what Ryan S did yesterday (82840) - paused SQZ_MANAGER in LOCK_CLF and adjusted OPO temp. We were then able to lock the filter cavity, start squeezing, and get back into Observing
03:05 Observing
    03:36 Popped out of Observing to reset the sqz angle since we had a message about it and sqz was slowly getting worse
    03:36 Back into Observing

    03:47 Popped out of Observing to reset the sqz angle since we had a message about it and sqz was slowly getting worse
    03:48 Back into Observing

04:00 Lockloss
    - Ran initial alignment

05:16 NOMINAL_LOW_NOISE
05:19 Observing
                                                                                                            

In our ongoing quest to figure out what causes the ETMX glitch locklosses, I've rerun all O4 NLN locklosses with our most up-to-date locklost code to see if we can pinpoint when it started and how/if it comes and goes.

To do this I made a histogram plot for all of O4 showing the amount of locklosses we had each day in blue, and then over that plotted the number of  those locklosses that have ETMX glitches right before the lockloss (plot). Basically, it looks like it's been with us since the start of O4. There are times where we go a week or maybe almost a whole month without getting locklosses from them, but they're still pretty regular and we see them every few days otherwise. In O4b, it looks like we saw them very often between end of August to early October and then saw less of them in December, but it's also hard to quantify because we had less locklosses in December, but that could've been either due to more time locked, or due to more time when the detector was down (same with November - that was when we had all the NPRO stuff).

I also have this plot as a pdf, and here's the link to my lockloss tool in case anyone wants to peruse the early O4 ETM glitch locklosses. Also keep in mind that a few ETMX glitch locklosses will have been missed being tagged by the lockloss tool - our code relies on the glitch surpassing a threshold and then going back down and staying below another threshold for a set amount of time, but in a few cases the glitch leads right into the lockloss, meaning it doesn't stay below that threshold long enough for the lockloss tool to realize it was an ETM glitch and thus not tagging it. This doesn't happen very often but it's still important to note that there are more ETM glitch locklosses than we have plotted on this histogram.

Lockloss @ 02/18 04:00 UTC

A few seconds before the lockloss I saw ASC-INP1_P_INMON get pretty large, and ASC-CHARD_P_OUT also started oscillating. I also saw the lockloss in LSC-POP_A_LF_OUT and LSC_PR_GAIN_OUT a few seconds before the lockloss. Pretty sure this is was one of the weird oscillation locklosses like the ones in 82847

I replaced the viewport cover I had removed in case someone needs the laser safe state in the morning.

Lockloss @ 02/18 00:06 UTC after almost 5.5 hours locked

TITLE: 02/17 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 151Mpc
OUTGOING OPERATOR: Ryan S
CURRENT ENVIRONMENT:
    SEI_ENV state: USEISM
    Wind: 5mph Gusts, 3mph 3min avg
    Primary useism: 0.06 μm/s
    Secondary useism: 0.59 μm/s
QUICK SUMMARY:

Currently Observing at 151Mpc and have been Locked for 4:20 now. Wind is very low and ssecondary microseism is very high, but not too bad and turning slightly back downward.

TITLE: 02/17 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 153Mpc
INCOMING OPERATOR: Oli
SHIFT SUMMARY: Only one lockloss this shift, but spent time this morning out of observing to troubleshoot SQZ issues, details of which can be found in alog82853. Fortunately, it seems to have helped and H1's range is higher than it's been in a few days.
LOG:

• 15:38 - Observing after Corey fixed up SQZ enough to go to observing - alog82849
• 16:13 - Dropped observing to further troubleshoot
• 17:04 - Lockloss - alog82856
• 18:53 - Observing
• 20:17 - Dropped observing for more SQZ work
• 20:32 - Observing w/ range improved

FAMIS 31073

Only event of note I can see is PMC REFL suddenly had a rise of about 0.5W starting 4 days ago that has almost leveled out.

Lockloss @ 17:04 UTC - link to lockloss tool (late entry for the lockloss this morning)

Happened while Camilla and I had H1 out of observing to troubleshoot SQZ (see alog82853), but very unlikely anything we were doing contributed to the lockloss. This looks like another of the newer ~11Hz oscillation locklosses that Oli discusses in alog82847.

H1 returned to observing at 18:53 UTC (but later dropped again for more SQZ troubleshooting, see above).

Ryan S, Camilla

Corey 82849, Ryan and I have had some issues with SQZ this morning. Corey adjusted the SHG temp to get enough green power to lock the OPO but after that the SQZ and range was bad. FR3 signal and FC WFS were very noisy.

Ryan tried to take SQZ out, checked OPO temp (fine), reset the SQZ angle from 220deg back to a more sensible 190deg and then put SQZ back in again but he couldn't get the FC to lock. Followed steps in the SQZ wiki to touch up FC1/2 alignment and got  H1:SQZ-FC_TRANS_C_LF_OUTPUT to >120, plot. but we still were loosing the FC at TRANSION_TO_IR_LOCKING. THE FC also seemed unstable when locked on green. While we were troubleshooting, the IFO lost lost. Unsure if this is an ASC issue, FC_ASC trends attached (POS for Y and P were moving much more than usual), SQZ ASC trends (ZM4 PIT changes alot).

After the lock loss, the SQZ_FC seemed to lock stably in green with H1:SQZ-FC_TRANS_C_LF_OUTPUT = 160. Plot. This is higher than usual and it's not clear what changed!

Ryan mentioned that something happened to Oli at the weekend where the range was bad but SQZ unlocked and re-locked and it came back good, plot, but this seemed to the the OPO PZT changing to a better place (we know it likes ot be ~90 rather than 50s).

After relocking, everything was fine but the FC ASC wasn't turning on as the threshold was too low, 2.5. Ryan decreased H1:SQZ-FC_ASC_TRIGGER_THRESH_ON from 3.0 to 2.0, this has been slowly decreasing, maybe as we decreased the OPO trans from 80uW to 60uW last week. Plot. Ryan accepted in sdf and then checked SQZ_MANAGER, SQZ_FC, sqzparams to check this isn't set in GRD.

I was looking at the lockloss sheet and noticed that this weekend, there were three locklosses that were similar to each other. Ryan Short noted one in an alog a couple days ago: 82809.

The locklosses look similar to what we sometimes see when we lose lock due to ground motion from an earthquake or wind, but in all three cases there was no earthquake motion, and the wind was below 20mph. There will be an 11 Hz oscillation starting about 1 second before the lockloss, and it can be seen in DARM, all QUADs, MICH, PRCL, and SRCL.

Lockloss times:

2025-02-14 07:15 UTC

2025-02-14 21:15 UTC

2025-02-16 09:48 UTC

There was another instance of a lockloss that looked just like these on February 7th 2025 at 05:47UTC as well.