Once Sheila pushed a new SRC detuning, I ran a broadband measurement, then started simulines. We lost lock a little over a minute after starting, not sure of the cause yet.

Simulines start:

PDT: 2024-09-27 11:08:44.473513 PDT
UTC: 2024-09-27 18:08:44.473513 UTC
GPS: 1411495742.473513
 

Lock loss 1810UTC

Lockloss 1411472854

The lock loss tool tagged FSS oscillation. Trending H1:PSL-FSS_FAST_MON_OUT_DQ shows that something started to get noisy 35seconds before lock loss. I didn't see any other strangeness in our usual LSC, ASC, ETMX, DARM signals.

Took another data set of FIS with different SRCL offsets, to try to set the SRCL detuning for calibration measurement, similar to 79903. 

• no squeezing: 1411485337 (120 seconds) (ref 4)
• FIS, ran scan alignment, ressults here , ran scan sqz ang with the blrms changed to 6 (we normally use 3, to minimize the 300 Hz noise, but in this case we want to min 1.7kHz sqz)  SRCL at -290 start at 1411486541 ref 5  (SRCL offset we have been observing with)
• FIS, SRCL at -190 start at 1411487041 ref 6
• FIS, SRCL at -90 start at 1411487385 ref 7
• FIS, SRCL at -340 start at 1411487954 ref 8 (manual sqz ang adjust)
• FIS, SRCL at -390 start at 1411488251 ref 9
• FIS SRCL at -490 start at  1411488585  ref10  (22.2k counts on POP A 45I ADC)

Fri Sep 27 08:14:05 2024 INFO: Fill completed in 14min 1secs

TITLE: 09/27 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 145Mpc
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
    SEI_ENV state: USEISM
    Wind: 4mph Gusts, 2mph 3min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.44 μm/s
QUICK SUMMARY: Locked for one and a half hours, some shorter locks overnight as well. Verbal didn't mention any PIs for the last lock loss, but the length of the lock is suspicious. More investigation needed. Our range isn't optimal and the squeezing looks poor in the higher frequencies based on the nuc33 FOM.

TITLE: 09/27 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY:

IFO is LOCKING at ENGAGE_ASC_FOR_FULL_IFO. Fully auto so far...

Shift was mostly quiet. Despite locking issues today, IFO locked pretty automatically following Tony's initial alignment. We did lose lock during LOWNOISE_ESD_ETMX (558). Though, guardian brought it to NLN shortly after and we were in observing swiftly.

In terms of PI modes, there was one very harsh ringup 23 minutes into NLN (or 34 mins after MAX_POWER). This gave 3 verbal PI 24 alarms but the damping, even though at maximum, was able to bring it down. No other ringups.

There was one lockloss 04:08 UTC. probably attributed to the environment and rising secondary microseism/over 35mph wind combo - alog 80323.

TCS Work from today has 2 SDF Diffs (screenshot attached).
LOG:

Weekly In-Lock SUS Charge Measurement - Closes FAMIS 28372

Observations:

• EY Gamma and Alpha vs time have been steadily going down, lower than last week too, where they were markedly lower than 2 weeks ago.
• IX still having experiencing this data gap so only 3 data points and this week's aren't on this plot (not new)
• IY no changes
• EX Gamma and Alpha vs time have been higher this week and last (by a lot in the case of alpha) than in the previous few months. beta2 vs time slightliy elevated too in last 2 weeks but not a big jump.OK

Lockloss after a 3hr lock. A few details:

• Not due to PI modes despite one successfully damped PI24 ringup at around 23 minutes after NLN.
• There was a low magnitude (5.1) EQ in El Salavdor but we lost lock 3 mins before EQ mode.
• Winds in the last hour have gone above 30mph
• Microseism (secondary) in the last hour went above the 0.9 line

From the above, I think this lockloss may be environmentally cause rather than due to PI issues we have been experiencing. Now having trouble locking ALS due to high winds.

TITLE: 09/26 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Commissioning
INCOMING OPERATOR: Ibrahim
SHIFT SUMMARY:

OMC issues
Ether Cat issues

NLN reached at 20:27 UTC!

CAL and PEM Team were able to get some Comissioning done in the later half of the day.... but....
PI 24 Ring up.... -> Lockloss   before Sheila could finish her work. :(
ETMY Ring Heater turned on, so ETMY Will start to drift.

LOG:

TITLE: 09/26 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Calibration
OUTGOING OPERATOR: Tony
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 16mph Gusts, 10mph 3min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.24 μm/s
QUICK SUMMARY:

IFO is in MAINTENANCE (Calibration and OMC Whitening Chassis work) but now just trying to get locked.

We're in initial alignment now and prepping to lock for observing!

To try to avoid the 10kHz PI (MODE 24), I've increased the power on both segments of ETMY ring heater from 1W to 1.1W.  I did this after the calibration measurements when the PI was already ringing up, and we've now lost lock from the PI.  (for a recent history, see 80299)

I measured the power of the beam comming out of the HAM3 illuminator viewport and found it to be 19mW, as compared to 17mW measured in this alog 78878. The beam is a reflection off of the scraper baffle of the part of the PR2 to PR3 beam that is clipped by the aperture of the scraper baffle. We had minimized it from 47mW to 17mW for the referenced alog, and wanted to see if it was clipping more - not much.

pydarm measure --run-headless bb
diag> save /ligo/groups/cal/H1/measurements/PCALY2DARM_BB/PCALY2DARM_BB_20240926T212332Z.xml
/ligo/groups/cal/H1/measurements/PCALY2DARM_BB/PCALY2DARM_BB_20240926T212332Z.xml saved
diag> quit
EXIT KERNEL

2024-09-26 14:28:43,409 bb measurement complete.
2024-09-26 14:28:43,409 bb output: /ligo/groups/cal/H1/measurements/PCALY2DARM_BB/PCALY2DARM_BB_20240926T212332Z.xml
2024-09-26 14:28:43,410 all measurements complete.
anthony.sanchez@cdsws29:

21:30 UTC gpstime;python /ligo/groups/cal/src/simulines/simulines/simuLines.py -i /ligo/groups/cal/H1/simulines_settings/newDARM_20231221/settings_h1_newDARM_scaled_by_drivealign_20231221_factor_p1_asc.ini;gpstime
PDT: 2024-09-26 14:30:05.062848 PDT
UTC: 2024-09-26 21:30:05.062848 UTC
GPS: 1411421423.062848

2024-09-26 21:53:35,296 | INFO | Finished gathering data. Data ends at 1411422832.0
2024-09-26 21:53:36,077 | INFO | It is SAFE TO RETURN TO OBSERVING now, whilst data is processed.
2024-09-26 21:53:36,077 | INFO | Commencing data processing.
2024-09-26 21:53:36,077 | INFO | Ending lockloss monitor. This is either due to having completed the measurement, and this functionality being terminated; or because the whole process was aborted.

2024-09-26 21:53:36,077 | INFO | It is SAFE TO RETURN TO OBSERVING now, whilst data is processed.
2024-09-26 21:53:36,077 | INFO | Commencing data processing.
2024-09-26 21:53:36,077 | INFO | Ending lockloss monitor. This is either due to having completed the measurement, and this functionality being terminated; or because the whole process was aborted.
2024-09-26 22:01:43,741 | INFO | File written out to: /ligo/groups/cal/H1/measurements/DARMOLG_SS/DARMOLG_SS_20240926T213006Z.hdf5
2024-09-26 22:01:43,754 | INFO | File written out to: /ligo/groups/cal/H1/measurements/PCALY2DARM_SS/PCALY2DARM_SS_20240926T213006Z.hdf5
2024-09-26 22:01:43,764 | INFO | File written out to: /ligo/groups/cal/H1/measurements/SUSETMX_L1_SS/SUSETMX_L1_SS_20240926T213006Z.hdf5
2024-09-26 22:01:43,774 | INFO | File written out to: /ligo/groups/cal/H1/measurements/SUSETMX_L2_SS/SUSETMX_L2_SS_20240926T213006Z.hdf5
2024-09-26 22:01:43,784 | INFO | File written out to: /ligo/groups/cal/H1/measurements/SUSETMX_L3_SS/SUSETMX_L3_SS_20240926T213006Z.hdf5
ICE default IO error handler doing an exit(), pid = 2864258, errno = 32
PDT: 2024-09-26 15:01:43.864814 PDT
UTC: 2024-09-26 22:01:43.864814 UTC
GPS: 1411423321.864814
anthony.sanchez@cdsws29:

After the EtherCAT reboot we had a few SDF Diffs that needed to be accepted.
Screenshot.

This morning Tony and TJ had a hard time locking the OMC.

We've found that OMC DCPD A and B output are very assymmetric only when there was a fast transient (1st attachment) but not when the OMC length was slowly brought close to the resonance (2nd attachment), which suggested whitening problem.

The transfer function from OMCA to B suggested that the switchable hardware whitening was ON for DCPD_A and OFF for B when it was supposed to be OFF for both. 3rd attachment shows the transfer function from DCPD_A to B, and 4th attachment shows the anti-whitening filter shape.

Switching ON the anti-whitening only for DCPD_A made the frequency response flat. Trying to switch the analog whitening ON and OFF by toggling H1:OMC-DCPD_A_GAINTOGGLE didn't change the hardware whitening status, it's totally stuck.

We tried to lock the IFO by only using DCPD_B, but IFO unlocked for some reason.

After IFO lost lock, people found on the floor that it's the problem of the whitening chassis, not the BIO. It's not clear if we can fix the board in the chassis (which is preferrable) or have to swap the whitening chassis (less preferable as calibration group needs to measure the analog TF and generate a compensation filter).

We'll update as we make progress.

Daniel has the Beckhoff slow controls system offline for investigation.

I've bypassed the following alarms:

Bypass will expire:
Thu Sep 26 10:54:08 PM PDT 2024
For channel(s):
    H1:PEM-C_CER_RACK1_TEMPERATURE
    H1:PEM-C_MSR_RACK1_TEMPERATURE
    H1:PEM-C_MSR_RACK2_TEMPERATURE
    H1:PEM-C_SUP_RACK1_TEMPERATURE
 

Looking at data from a couiple of days ago, there is evidence of some transient bumps at multiples of 11.6 Hz. Those are visible in the summary pages too around hour 12 of this plot.

Taking a spectrogram of data starting at GPS 1410607604, one can see at least two times where there is excess noise at low frequency. This is easier to see in a spectrogram whitened to the median. Comparing the DARM spectra in a period with and without this noise, one can identify the bumps at roughly multiples of 11.6 Hz.

Maybe somebody from DetChar can run LASSO on the BLRMS between 20 and 30 Hz to find if this noise is correlated to some environmental of other changes.