ETM Glitch Lockloss. Relocking now.

During comissioning, I ran a2l script for all. Results and screenshots below.

ETMX P
Initial:  3.23
Final:    3.22
Diff:     -0.01

ETMX Y
Initial:  4.9
Final:    4.95
Diff:     0.05

ETMY P
Initial:  5.52
Final:    5.4
Diff:     -0.12

ETMY Y
Initial:  1.35
Final:    1.4
Diff:     0.05

ITMX P
Initial:  -0.53
Final:    -0.56
Diff:     -0.03

ITMX Y
Initial:  3.21
Final:    3.24
Diff:     0.03

ITMY P
Initial:  0.06
Final:    0.02
Diff:     -0.04

ITMY Y
Initial:  -2.74
Final:    -2.77
Diff:     -0.03

Sheila, Camilla

NLG Measurements (follow 76542):

For 25uW we needed to reduce H1:SQZ-OPO_PZT_1_SCAN_TRIGGER_CHANNEL_1_LEVEL from 0.5 to 0.25

FC2 is misaligned during this dataset. Aimed for 3m30 of data for each point.

In the Mean SQZ data, there was a strange peak around 300Hz, we checked if tis was seeding by moving H1:SQZ-LO_SERVO_SLOWOUTOFS, this made no difference

Fri Mar 14 10:13:51 2025 INFO: Fill completed in 13min 48secs

TITLE: 03/14 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 151Mpc
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 4mph Gusts, 0mph 3min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.27 μm/s
QUICK SUMMARY:

IFO is in NLN and OBSERVING as of 23:59 UTC (14hr 40 min lock!)

We have planned comissioning today from 6AM - 2PM (though will stay in OBSERVING until a comissioner shows up).

TITLE: 03/14 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 147Mpc
INCOMING OPERATOR: Corey
SHIFT SUMMARY: Currrently Observing at 148Mpc and have been Locked for 5 hours. The range is still low and fluctuating a bit, but it's not as bad as it was before I made the changes to the multiple phases (83367). Secondary microseeism is going back down, and the wind is low.
LOG:

23:30UTC Relocking

23:50 NOMINAL_LOW_NOISE
    - Ran SQZ ANG ADJUST to check if the SQZ servo was in a location to pull the ADF phase away
23:59 Observing

02:55 Out of Observing to check sqz angle
    - Ran SQZ ANG ADJUST and had to adjust some phases (83367)

03:13 Back into Observing

We are Observing at 150Mpc and have been Locked for 4 hours.

We just got back into Observing after I took us out to touch up some sqz phases. This morning Camilla and Sheila had found (83350) that the SQZ servo was pulling the ADF phase away and causing large range drops. The same thing also happened the lock after that.

When we first got to NOMINAL_LOW_NOISE, I tried running SQZ ANG ADJUST to check if we were in a spot where the ADF phase would end up getting pulled away. At the location where the squeezing was best, the ADF sqz angle was pretty good, and the demod error was near 0 (ndscope1). I didn't adjust anything because of this, just went back into FDS and then into Observing. I planned on monitoring the ADF phase angle and then do these tests again once we were thermalized.

Over the course of thermalization, the same issue with the ADF phase was happening, although luckily the servo wasn't taking the phase in circles like it had been doing the previous couple of locks (ndscope2). Once we were thermalized, I took us out of Observing and ran SQZ ANG ADJUST. From that I saw that the best location for the sqz phase was now very close to a turning point (ndscope3). To move away from this location, I adjusted SQZ-ADF_VCXO_PLL_PHASE from 25 down to 10, and then adjusted SQZ-ADF_OMC_TRANS_PHASE from -107 to -117 to get SQZ-CLF_REFL_RF6_PHASE_PHASEDEG to oscillate around 0. The yellow BLRMS looks a lot better now and our range has gone up by 10Mpc, but my changes did make the purple BLRMS worse by a bit (ndscope4, ndscope5). I accepted the phase changes in SDF and we went back into Observing (SDF diffs).

Janos, Gerardo, Dave:

The slope trip level for some LVEA gauges was reduced from 1.0e-10Torr to 3.0e-12Torr such that if an event similar to this morning were to happen again VACSTAT would alarm on it.

Trending the SLOPE channels over the past week showed that BSC3's gauge (PT132) cannot be lowered without causing many false positive alarms. Attachments show trend and which gauges now have the lower trip level.

TITLE: 03/13 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Oli
SHIFT SUMMARY: Calibration and commissioning time this morning, but we have also been struggling a bit with relocking. We aren't entirely sure what particular problem is causing the lock losses on acquisition, but the elevated wind and useism aren't helping. We are now on DC readout though, so we're making progress.
LOG:

• 1559 Lock loss - 83351
  • Since this was during commissioning time, we did some ALS linearization testing on the way up (83357).
• 2012 Observing
• 2132 Lock loss - 83360
  • Recovery has been a bit of a struggle. ALS hasn't been staying locked, perhaps the DARM tramp that Sheila put in wasn't working, so we put the tramp back to 5 from 2. We also were losing lock at other various places, so we aren't too sure what the issue is yet. The wind and useism are both elevated, so perhaps this is factoring in.

TITLE: 03/13 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 25mph Gusts, 16mph 3min avg
    Primary useism: 0.06 μm/s
    Secondary useism: 0.34 μm/s
QUICK SUMMARY:

At ENGAGE_ASC_FOR_FULL_IFO and working on relocking with lots of lower state locklosses. These seem to be due to various causes, including high-ish winds& gusts. Secondary microseism is also a bit elevated.

I replaced the three pole, 25 amp circuit breaker which controls the line voltage for an HVAC circuit in the H2. Voltage output was 490 volts.The previous breaker was damaged by heat caused by a loose connection. 

1425936756

No obvious cause but there was a small change in the PRG a second or two before lock loss.

We lost lock from the calibration, so we tried to lock ALS without the linearization (some background in this alog: 83278.)  An active measurement of the transfer function from DRIVEALIGN_L to MASTER out was 1 without the linearization, and -0.757 with the linearization on.  So I've changed the DRIVEALIGN gain to -1.3 in the ALS_DIFF guardian when the use_ESD_linearization is set to false.  

We tried this once, and it stayed locked for a DARM gain of 400, but unlocked as the UIM boosts were turning on. We tried this again but it also didn't lock DIFF, so it is now out of the guardian again.

I looked at a few more of the past ALS DIFF locks, both sucsesful and unsucsesful attempts we are saturating the ESD (either the DAC or the limiter in the linearization) in the first steps of locking DIFF.  We do these steps quite slowly, stepping the darm gain to 40 waiting for the DARM1 ramp time, stepping it to 400, then waiting twice the ramp time, then engaging the boosts for offloading to L1.  I reduced the ramp time from 5 seconds to 2 seconds to make this go faster.  This worked on the first locking attempt, but that could be a coincidence. 

We will leave this in for a while, so that we can compare how frequently we loose lock at LOCKING_ALS.  In the last 7 days we've had 48 LOCKING_ALS locklosses, and 19 locklosses from NLN, so roughly 2.5 ALS locklosses per lock stretch. 

The calibration measurement caused another lock loss at the end of the measurement (alog83351)

Simulines start:

PDT: 2025-03-13 08:36:53.447733 PDT
UTC: 2025-03-13 15:36:53.447733 UTC
GPS: 1425915431.447733
 

End of script output:

2025-03-13 15:58:59,923 | INFO | Drive, on L2_SUSETMX_iEXC2DARMTF, at frequency: 42.45, and amplitude 0.41412, is fin
ished. GPS start and end time stamps: 1425916738, 1425916753
2025-03-13 15:58:59,923 | INFO | Scanning frequency 43.6 in Scan : L2_SUSETMX_iEXC2DARMTF on PID: 795290
2025-03-13 15:58:59,923 | INFO | Drive, on L2_SUSETMX_iEXC2DARMTF, at frequency: 43.6, is now running for 23 seconds.
2025-03-13 15:59:01,039 | INFO | Drive, on DARM_OLGTF, at frequency: 1083.3, and amplitude 1e-09, is finished. GPS st
art and end time stamps: 1425916738, 1425916753
2025-03-13 15:59:01,039 | INFO | Scanning frequency 1200.0 in Scan : DARM_OLGTF on PID: 795280
2025-03-13 15:59:01,039 | INFO | Drive, on DARM_OLGTF, at frequency: 1200.0, is now running for 23 seconds.
2025-03-13 15:59:02,168 | INFO | Drive, on L1_SUSETMX_iEXC2DARMTF, at frequency: 11.13, and amplitude 13.856, is fini
shed. GPS start and end time stamps: 1425916735, 1425916753
2025-03-13 15:59:02,168 | INFO | Scanning frequency 12.33 in Scan : L1_SUSETMX_iEXC2DARMTF on PID: 795287
2025-03-13 15:59:02,169 | INFO | Drive, on L1_SUSETMX_iEXC2DARMTF, at frequency: 12.33, is now running for 25 seconds
.
2025-03-13 15:59:07,162 | ERROR | IFO not in Low Noise state, Sending Interrupts to excitations and main thread.
2025-03-13 15:59:07,163 | ERROR | Ramping Down Excitation on channel H1:SUS-ETMX_L2_CAL_EXC
2025-03-13 15:59:07,163 | ERROR | Ramping Down Excitation on channel H1:SUS-ETMX_L3_CAL_EXC
2025-03-13 15:59:07,163 | ERROR | Ramping Down Excitation on channel H1:CAL-PCALY_SWEPT_SINE_EXC
2025-03-13 15:59:07,163 | ERROR | Ramping Down Excitation on channel H1:SUS-ETMX_L1_CAL_EXC
2025-03-13 15:59:07,163 | ERROR | Aborting main thread and Data recording, if any. Cleaning up temporary file structu
re.
2025-03-13 15:59:07,163 | ERROR | Ramping Down Excitation on channel H1:LSC-DARM1_EXC
ICE default IO error handler doing an exit(), pid = 795246, errno = 32
PDT: 2025-03-13 08:59:11.496308 PDT
UTC: 2025-03-13 15:59:11.496308 UTC
GPS: 1425916769.496308
 

We had a vacuum glitch in the LVEA at 01:57 Thu 13mar2025 PDT which was coincident with a lockloss. The glitch was below VACSTAT's alarm levels by an order of magnitude, so no VACSTAT alert was issued.

The glitch is seen in most LVEA gauges, and took about 10 minutes to pump down.

Attached plots show a sample of LVEA gauges, the VACSTAT channels for LY, and the ISC_LOCK lockloss.

We noticed that the range was dropping even lower than our already low ~145Mpc. There was a lot of low frequency noise that was creeping up. Sheila suggested that we look at the PIs and sure enough, PI31 had started to slowly creep up the same time as the range degredation (see attached). The SUS_PI guardian node will turn on the damping at 3 according to the H1:SUS-PI_PROC_COMPUTE_MODE31_RMSMON channel and turn it back off when it gets below 3. For now we just tried changing that threshold to 0 to all for continuous damping. This let the mode damp down completely and it brought our range back up.

We should think more about damping this down lower and having two thresholds, an on an off.

I have recently reported that the “mystery” beam on the spool piece wall near HAM3 was coming from the direction of ITMX (82252). To further this investigation, I started photographing the area around the ITMs and BS as best I coluld through our viewports (there is not a good view towards HAM3).  I found several unexpected distributions of light in the vertex:

1. 20 degree conical annular beams from ITMs

Both ITMX/CPX (Figure 1) and ITMY/CPY (Figure 3) cast an expanding annular “beam” towards the BS with a cone half angle of roughly 20 degrees from the main beam. My best guess is that it is produced by arm cavity light hitting the bevel of the ITMs (see cartoon in Figure 1). A good test of this would be to install the new test mass cage baffles at one or more of the ITMs at LLO (presuming Anamaria finds this beam at LLO) this upcoming break. The baffle should hide the bevel and eliminate the ring of light.

2. 45 degree conical annular beams from BS

The BS appears to cast an expanding annular “beam” with a cone half angle of 45 degrees, centered around the -X, -Y direction (Figure 2), and likely another annular beam, also with a half angle of 45 degrees, centered around  the  -X, +Y direction (evidence in Figure 3). I tried to find a geometry where the bevels were also the source of these beams but didn’t. My best guess is that the annular cone is produced by reflections of light from PR3 and the ITMs off of the inner surface of the circular cage around the BS, or the inside surface of the circular barrel of the BS itself (see drawings on third page of Figure 2). 

3. Reflection of BS in ITM elliptical baffles likely visible at ITMY and HAM3

The BS beam spot is reflected towards ITMY by the slanted piece of the ITMX elliptical baffle (Figure 3). While the actual beam is not reflected towards ITMY (it isn’t clipped), the baffle reflects light towards ITMY that is scattered out of the main beam by only a few degrees.