Sheila, Camilla

The operator team has been having a lot of issues with the SQZ FC locking recently, e.g. 86153. We hoped this was due to incorrect SEI settings 86120, but this didn't solve the issue.

The issue isn't with green locking as green locking can stay for ~minutes. It appears to be when we get to state FC_ASC_ON, although the ASC is not the issue (Oli had tried keeping it off). Comparing an unsuccessful to successful time, both the green and IR powers are lower when we are unsuccessful, as well as the FC_WFS_A_SUM Q_OUT channel, both the WFS and the IR signal get noisy with a growing ~15Hz oscillation before the lockloss plot, which happens before the ASC is turned on.

Sheila notes that this means that the FC IR LSC loop was probably unstable. We measured the crossover and could see it was very close to unstable at 15Hz, see plot. We increased the gain in at the FC LSC input matrix a factor of 1.3, H1:SQZ-FC_LSC_INMTRX_RAMPING_2_7 via sqzparams.py (fc_wfs_a_ir_gain) from -0.86 to -1.12.  This brought the crossover back to the reference with 50deg of phase margin. Ideally we would compleatly redesign this loop but it was hard to design originally 66092.

To get FC to lock successfully, Sheila did had to trend and revert FC2 and ZM3 alignments. It seems that while the FC locking is struggling, the fC ASC can pull these away from nominal.

Interestingly the successful lock, as the ASC comes on, the green light decreases as the IR light increases, maybe this si a sign that green and IR aren't well co-aligned. There is picos in the green FC path, but we don't want to touch these as it's more likely that different OPO crystal voltages or spots change the alignment.

Other things that have changed over the past ~month:

• FC launch power is lower and correlated to the FC_TRANS green power being lower, plot. We were having FC issues at the end of June when this was low as well. We can't increase this without taking power from the green pump.
• FC IR power is degrading quickly throughout a lock, the operators adjusting the OPO temperature brings this up again, plot, but this only sometimes fixes issues with FC locking, e.g.  86153.

Hopefully this has fixed our issues, but if this happens again, steps operators should take are (added to wiki):

• Check OPO temperature via taking SQZ_OPO_LR guardian to SCAN_OPOTEMP (will take 3 minutes, can watch via !SQZ Scopes > OPO temp) once it's done you can take SQZ_OPO_LR back to LOCKED_CLF_DUAL and  check the OPO temp plot that's in https://lhocds.ligo-wa.caltech.edu/exports/SQZ/GRD/OPOTEMP_SCAN/
• If H1:SQZ-FC_TRANS_C_LF_OUTPUT is a lot lower than 80, should check alignments and revert or optimize of FC2 and ZM3 (explicit instructions in the wiki (under "Issues with SQZ_FC Guardian").

Mon Aug 04 10:07:01 2025 INFO: Fill completed in 6min 57secs

Note that, similar to yesterday, TC-B does not saturate at -200C. Perhaps due to lower OAT (24C, 76F).

FAMIS26054

The script reported ITMY_ST1_CPSINF_H3 is high. This was the same as last time this was ran (alog85969).

TITLE: 08/04 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 148Mpc
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 14mph Gusts, 8mph 3min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.07 μm/s
QUICK SUMMARY: Locked for 6.5 hours after an earthquake took us out 10 hours ago. The DARM FOM is currently not working. Violin modes, most notably ITMY mode 6, are elevated but the DCPD window FOM shows that they are damping.

3:47:39 UTC unknown Lockloss  

 

TITLE: 08/04 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Earthquake
INCOMING OPERATOR: Corey
SHIFT SUMMARY:
H1 was locked for a good 6 hours then had an unknown lockloss.
Relocking was interupted at LOWNOISE_COIL_DRIVERS, due to a 6.2 M  earth_quake whcih took H1 to DOWN.

H1 is currently waiting in Earthquake mode.


LOG:
                                                                                                            

We have many peaks below 40 Hz that couple, at least partly, through input beam jitter. Last summer Sam and Genevieve determined that a wide variety of site equipment produced these peaks, including the office area air handler, mini-splits in the CER, and chiller compressors for the main HVAC (LIGO-G2402140).  The figure shows that there is coherence between DARM and the IMC WFS that might be used to clean these low frequency peaks, but that they are currently not being cleaned.

Eventually, we would rather not have peaks that need cleaning, but instead, reduce the source vibration and/or the vibration coupling to DARM. I think that the best plan is to reduce the source vibration of the largest peaks, but to mainly focus on reducing the coupling, because many of these peaks are just 2-5 times the vibration background at the coupling sites, so even eliminating the vibration of the sources will not be enough to get us to our design sensitivity.

The coupling of relatively low amplitude vibrations at low frequencies seems to be associated with coupling resonances. For example, when one of the frequencies of the office area air handler drifted into the 35Hz peak frequency of one of these coupling resonances, the peak in DARM was huge, but was greatly reduced by changing the operation frequency of the air handler (82986).  Ill try to map out these low frequency coupling resonances during commissioning periods as a step in understanding their cause.  But for now, it would be nice to see how much we can reduce the peaks with cleaning.

TITLE: 08/03 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 152Mpc
OUTGOING OPERATOR: Oli
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 14mph Gusts, 9mph 3min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.07 μm/s
QUICK SUMMARY:

H1 has been locked for 2 Hours and 20 minutes.
All systems seem to be functioning well.
Hoping for a smooth shift of Observing.

TITLE: 08/03 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 155Mpc
INCOMING OPERATOR: Tony
SHIFT SUMMARY: Currently Observing at 155Mpc and have been Locked for a little over 2 hours. One lockloss today and after some issues with ALS not wanting to stay locked (partially due to ground motion, partially unknown), after an initial alignment we were able to relock fine. We dropped out of Observing once due to the squeezer unlocking but it was able to fix itself.
LOG:

18:05 Lockloss
    - Lockloss from PRMI/DRMI multiple times due to ALSX or Y unlocking
    - Ran an initial alignment
21:19 NOMINAL_LOW_NOISE
    21:21 Observing
    22:10 Out of Observing due to SQZer unlock
    22:15 Back into Observing

Lockloss at 2025-08-03 18:05UTC after over 7.5 hour locked

Sun Aug 03 10:08:00 2025 INFO: Fill completed in 7min 56secs

After LLO dropped out, Jenne gave the OK to try to get squeezing back. The FC took a handful of minutes of trying and failing but then just worked. I used <a href="https://cdswiki.ligo-wa.caltech.edu/wiki/ObservationWithOrWithoutSqueezing">Ryan S's script</a> to get us back to being able to observe with sqz and we are back to Observing.

Out of Observing from 1539-1546UTC.

TITLE: 08/03 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 130Mpc
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 9mph Gusts, 3mph 3min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.06 μm/s
QUICK SUMMARY: The IFO is still in Observing without squeezing as Corey had left it. I'll give a squeezer a call as soon as I can organize my already impromptu morning.

Looks like H1 went down due to an EQ (probably M6.8 Russia EQ) 4+hrs ago, and was at NLN but having SQZ Issues:

• SQZ FC was having trouble getting past FC ASC ON.
• Took MAnager to DOWN & started following SQZ Wiki.
• OPO_LR needed to be taken to Nominal state.
• CLF_LR taken to Nominal state
• FC taken to GR LOCKED, but trans_c_lf_output was only at about 75 (less than nominal 100), so worked on FC Alignment.
  • Took me a while to find the "FC Servo" button on the SQZ Overview---my display has everything reall small, and I don't have my glasses on...still waking up).
  • Once I got things set up, I was only able to maximize the trans c lf output up to about 80.  Didn't feel like I improved it much.  And while doing this I was looking at the FC Trans GR camera, but that just looked like a 00 spot, so not really giving me an idea of if pitch or yaw needed.  (I'm guessing the camera is useful if you are not on a 00 mode.)
  • Moved on
• Took Manager to FDS, but FC was not able to get GR LOCKED, so went back to wiki since "everything was not fine" yet.
• Started trending FC1/2 pointing.....but also looked alog for any recent FC issues
• FC's trans has not been at 100 for almost 2-weeks.  :(
  • Because of this, this is why I'm hesitant to move FC1 in yaw (that is the only thing which is currently a little off since the last lock.....but maybe I should at this point?  Because for FC alignment, I was only moving FC2 for alignment touch up. 
  • Moved FC1's Yaw to get it from a witness value of -199 back to -206 (where it was for the last lock).
• In the alogs, soundsing like TJ had the same exact situation I'm having on 7/31(alog), but it looks like SQZ randomly came back for him.
  • For my 2nd go around with touching up FC alignment, I was only getting up to about 50 (vs 80 my first time).  :(
  • Going to try going to FDS....but still had FC issues. 
  • But did happen to get the FC trans relock to a better value back to just under 80 (but only for a few seconds and then it lost lock, and then relocked at this higher value.....these locks stopped me from restoring FC1 yaw from the move I made to get it to where it was before the EQ.
• At this point, I feel like I have not progressed anywhere with FC (still loses lock at the ASC_ON step.
• The last thing I was considering, was trying the OPO temperature adjustment (which TJ mentions from an Oli alog).....but at this point, H1 has been waiting for SQZ for 2hrs & I have been up that long, too (at 535am).  So I'm also considering just not Squeezing....
  • Also was hoping SQZ would just come back on its own as it did for TJ on 7/31 while he was troubleshooting, but no luck on that so far.
  • NOTE:  For this OPO Temperature adustment, Oli did have a much better trans output (of 115! vs our sub-80).  So not really going to be surprised if this does anything in this case.
• Did optimize the OPO Temperature a bit, but this had no improvement for the FC locking (this is even after a few more rounds of FC2 alignment touch-ups....where it would sort of get a little over 80.....but still no luck).
• Running the RyanS script to go to Observing WITHOUT Squeezing  (this automatically took H1 to Observing as soon as I took 3 Sqz nodes to their No Squeezing NOMINAL states.
• And after about 2.5hrs (1024-1255utc) of trying, H1 is back to Observing (without Squeezing) at 555am PDT.  Now to see if I should stay up or try to get sleep (as the sun is now in my eyes as I click "POST TO LOGBOOK").
• I'm attaching screenshots of FC1 & FC2 alignment and also the OPO Temperature change.

Frustration NOTES: 

• My NoMachine is not as fast as working in the Control Room.  So everything was a pain while trying to wake up
• Making ndscopes for restoring FC1 & 2 took me time (still not sure if that was even going to be worth the trouble since the FC output hasn't been much above 80 anyways (vs the desired 100+ in the wiki).
  • Really the only FC pointing which looks like it took a big step since the last time the SQZ was locking was FC1 yaw.
  • BUT.....is this really a thing since the Trans has only been at about 80 lately (last 12 days)??
• In the wiki for FC, I don't know what this lines means:
  • "FC ASC "gracefully clear history" is this correct?"
• Only having one screen to work with also makes it tough.  For this particular squeezer issue, I have tons of scopes, guardians and medms open and its hard to keep track of all my windows. 

TITLE: 08/03 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 151Mpc
INCOMING OPERATOR: Corey
SHIFT SUMMARY:
H1 has reached 24 Hours of continuous lock!
All systems lookin good.
No issues all night.
Passing Corey a locked IFO for his OWL shift.
 
LOG:                                                                                                           

TITLE: 08/02 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 152Mpc
OUTGOING OPERATOR: Oli
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 16mph Gusts, 10mph 3min avg
    Primary useism: 0.04 μm/s
    Secondary useism: 0.06 μm/s
QUICK SUMMARY:
H1 has been Locked for 19 Hours now with decent range.
No issues to report.

 

TITLE: 08/02 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 154Mpc
INCOMING OPERATOR: Tony
SHIFT SUMMARY:We are Observing at 149 Mpc and have been Locked for almost 19 hours. Besides the calibration measurement today, we've been Observing the rest of my shift.
LOG:

14:30UTC Observing and have been Locked for 9:44 hours
    14:30 Earthquake mode activated
    15:25 Back to CALM
    18:18 Earthquake mode
    18:31 Left Observing to run calibration suite
    18:31 NLN_CAL_MEAS
    19:03 NOMINAL_LOW_NOISE
    19:03 Observing
    20:25 Superevent S250802dn

Lockloss from NLN 2025-08-02 02:57:54
Looks like maybe it's an ASC excursion?