TITLE: 10/23 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 163Mpc
OUTGOING OPERATOR: Camilla (OWL)
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 8mph Gusts, 7mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.33 μm/s
QUICK SUMMARY:

• We've been locked for 4:26, we survived a 6.0 from New Zealand ~4.5 hours ago
• All systems look good

The new high pass filter for the SRCL FF (73622) has improved the DARM RMS between 2 and 4 Hz a bit, although it's not clear why there is a new ~4 Hz peak. Now the largest contribution to the DARM RMS are the lines at 1.3 Hz, which we know are caused by the ETMX ISI (73625) and by the peak at 3.44 Hz which is coherent with CHARD_Y.

The lower DARM RMS in the 2-4 Hz region seems to have improved a little bit the non-stationary noise above 10 Hz, see spectrogram and scatter plot of DARM RMS in the 20-50 Hz region vs RMS in the 2.5-4 Hz region.

It's worth trying to further improve the DARM RMS by

• fixing the ETMX ISI problem
• trying to reduce CHARD_Y motion at 3.4 Hz
• increase DARM gain at frequencies < 5 Hz and see if that's enough to reduce the non-stationariity (depends on where's the source of the non-linearity)

Looking at the DARM bicoherence, there are three main components to the noise modualtion above 20 Hz: 1.3 Hz (ETMX ISI), 2.7-2.8 Hz  and 3.4 Hz (CHARD_Y). Not sure what's causing the DARM motion below 3 Hz, but there is a lot of coherence with ETMX M0 and R0 damping signals.

TITLE: 10/22 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 162Mpc
INCOMING OPERATOR: Camilla (OWL)
SHIFT SUMMARY:

n/a
LOG:

2332-2351 Attempted Broadband + simullines calibration measurements per Jenne's request while L1 was down, but had errors (as noted in previous alog).

Below are the steps for a (failed) attempt at running a broadband + simmulines calibration mesaurement w/ H1 was locked for 3+hrs and with L1 still in recovery from a power outtage:

• Following instructions (link)
• Screenshot of Cailbratoin Monitor medm taken (see attached)
• 2332 Out of OBSERVING and ISC LOCK to NLN_CAL_MEAS
• 2333 in a terminal started broadband calibration meas: pydarm measure --run-headless bb
  • 2339 this ended with:  ICE default IO error handler doing an exit(), pid = 99711, errno = 32
  • (Listing last few lines from this measurement below)
• 2340 Assuming this was the "end" of the broadband measurement, so I next ran simmulines measurement (in same terminal, per instructions):  gpstime;python /ligo/groups/cal/src/simulines/simulines/simuLines.py -i /ligo/groups/cal/src/simulines/simulines/settings_h1.ini;gpstime 
  • This had a similar error after 2-min.  Here are the last few lines from the terminal:

2023-10-22 23:42:28,941 | INFO | Drive, on L3_SUSETMX_iEXC2DARMTF, at frequency: 544.99, is now running for 23 seconds.
2023-10-22 23:42:35,388 | INFO | Drive, on PCALY2DARMTF, at frequency: 8.69, is finished. GPS start and end time stamps: 1382053346, 1382053370
2023-10-22 23:42:35,389 | INFO | Scanning frequency 8.99 in Scan : PCALY2DARMTF on PID: 106437
2023-10-22 23:42:35,389 | INFO | Drive, on PCALY2DARMTF, at frequency: 8.99, is now running for 31 seconds.
^C2023-10-22 23:42:40,347 | ERROR | Ramping Down Excitation on channel H1:SUS-ETMX_L1_CAL_EXC
2023-10-22 23:42:40,347 | ERROR | Ramping Down Excitation on channel H1:SUS-ETMX_L3_CAL_EXC
2023-10-22 23:42:40,347 | ERROR | Ramping Down Excitation on channel H1:SUS-ETMX_L2_CAL_EXC
2023-10-22 23:42:40,347 | INFO | Ending lockloss monitor. This is either due to having completed the measurement, and this functionality being terminated; or because the whole process was aborted.
2023-10-22 23:42:40,348 | ERROR | Ramping Down Excitation on channel H1:LSC-DARM1_EXC
2023-10-22 23:42:40,348 | ERROR | Aborting main thread and Data recording, if any. Cleaning up temporary file structure.
2023-10-22 23:42:40,348 | ERROR | Ramping Down Excitation on channel H1:CAL-PCALY_SWEPT_SINE_EXC
ICE default IO error handler doing an exit(), pid = 106373, errno = 32
PDT: 2023-10-22 16:42:44.751376 PDT
UTC: 2023-10-22 23:42:44.751376 UTC
GPS: 1382053382.751376

• 2351 After ligo.chat with Jenne, decided to stop Calibration Measurements (she mentioned possibly the dolphin crash from Fri night could be a possible culprit?)
• 2351 OBSERVING (and I restored H1 Manager to MANAGED)

Here were the last few lines at the end of the broadband measurement:

notification: new test result
notification: new test result
notification: new test result
notification: end of measurement
notification: end of test
diag> save /ligo/groups/cal/H1/measurements/PCALY2DARM_BB/PCALY2DARM_BB_20231022T233325Z.xml
/ligo/groups/cal/H1/measurements/PCALY2DARM_BB/PCALY2DARM_BB_20231022T233325Z.xml saved
diag> quit
EXIT KERNEL

INFO | bb measurement complete.
INFO | bb output: /ligo/groups/cal/H1/measurements/PCALY2DARM_BB/PCALY2DARM_BB_20231022T233325Z.xml
INFO | all measurements complete.
ICE default IO error handler doing an exit(), pid = 99711, errno = 32

TITLE: 10/22 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 160Mpc
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 5mph Gusts, 2mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.21 μm/s
QUICK SUMMARY:

H1's been locked 3hrs and got the rundown of what Ryan needed to do with the Squeezer with Sheila/Naoki's help earlier.  Useism is a bit lower than last night and winds are low.

TITLE: 10/22 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 157Mpc
INCOMING OPERATOR: Corey
SHIFT SUMMARY: One lockloss, automated relock, but there was a small SQZ issue.

Lockloss at 18:59UTC

The only issue I had relocking was the SQZ_LO_LR guardian reported that there was "no 3MHz on the OMC, could be a bad alignment", I talked to Sheila and I did a !graceful_clear_history on the SQZ_ASC_IFO screen and that fixed the issue, Naoki also recommended adjusting the OPO temperature again like yesterday which I also did and accepted in SDF.

LOG:

https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi?event=1382036394

Sun Oct 22 10:11:26 2023 INFO: Fill completed in 11min 22secs

TITLE: 10/22 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 164Mpc
OUTGOING OPERATOR: Camilla (OWL)
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 5mph Gusts, 3mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.23 μm/s
QUICK SUMMARY:

• We've been locked for 16:11, in Observing since 23:03UTC on 10/21
• All systems are looking good, range above 160!
• I had to restart both cdsws13 & 12 as neither of them were responding when I got in

TITLE: 10/21 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 146Mpc
INCOMING OPERATOR: Camilla (OWL)
SHIFT SUMMARY:

Locked entire shift with Lock Clock at just over 8hrs.  There was one M4.9EQ from Mexico which H1 rode through.
LOG:

• 2301 H1 to OBSERVING
• 0225 SRM saturations, picket fence, and seisblrms see 4.9 Mexico EQ
  • 0227-0247 EQ Mode (but this is about 4min after the R-wave passed)

All looks well for the last week for all site HVAC fans (see attached).

H1's been locked for more than half the shift thus far & has even rode through another Mexico EQ.  The range has been increasing the entire shift and it looks like are finally touching 160Mpc.  Violin mode (~500Hz) is now below 10^-17.

TITLE: 10/21 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 144Mpc
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 3mph Gusts, 1mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.34 μm/s
QUICK SUMMARY:

Literally just took H1 to Observing (at2303utc, after Camera Servo completed at 2301utc) after chatting with Ryan.  (Looks like I totally lucked out with RyanC getting H1 to NLN after all the work he did along with Camilla & Dave during the Owl shift due to last night's dolphin crash!).

Violins look a little elevated since last remember (currently just under 10^-16.

For this current lock, Ryan mentioned needing to possibly be mindful of PR3 drifting (because of PRMI not looking good during acquisition).  Mainly want to watch H1 to get past an hr and get back to a longer lock!

TITLE: 10/21 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Corey
SHIFT SUMMARY: Spent most of the shift continuing to try and recover alignments from the dolphin glitch at 8:05UTC. Currently waiting for ADS to converge to go into observing.

I had lots of issues to start in the morning, ALS_X would lose lock for no apparent reason or its WFS would oscilate too hard and kill its lock, then Diff_IR kept being "not really found" then PRMI buildups were very unstable and we couldn't get good enough flashes to get DRMI. I tried an IA, and then moving SR3 and PR3 based on their OPLEVS to where they were before the dolphin glitch, adjusting SR3 made PRMI better and PR3 made it worse.

I ended up trying to a full slider restore to a time when we were locking for the lock that was killed by the computer crashes (10/19 18:09UTC), still no luck getting PRMI or DRMI. I was considering doing another IA or restoring the sliders to after Camilla's IA this morning, after calling and talking with Jenne I did an initial alignment and not another restore. After this IA we went back up to NLN without any interventions, I stepped slowly through the states just in case. The violins had rung up a bit, but they damped down pretty quickly and weren't much of a hinderance.

Once I got to NLN, the SQZ manager had the notification "SQZ ASC AS42 not on?" So I went to RESET_SQZ_ASC then back to FREQ_DEP_SQZ following Vickys instructions in alog71083. While waiting for the ADS to converge I went through the large amout of SDF diffs for BOS and OAF from 8:50UTC which looked to be when Dave restarted their models, I reverted almost all of them after talking to Jenne (they were all COEFF) except for 1 in BOS. Naoki also hopped on to optimize the squeezing before we went back into observing, I optimized the OPO temperature when he was done and accepted it in SDF then went into observing at 20:49UTC.

Lockloss at 21:22UTC

Lost lock during PRMI_ASC from Yarm losing lock. PRMI keeps locking then losing it seconds later, flashes were super low. Tried CHECK_MICH after a few PRMIs. I trended PRM, PR3, SR3, and the BS to see if anything had moved since the LL, I ended up adjusting PR3 in yaw by .2 (155.5 ->155.3) since it looked like it drifted a little bit and then the flashes looked much better. Not sure why it moved? I also had to tap PRM a little but in pitch (2 microradians, -1502.4 -> -1504.4) and it caught PRMI then DRMI. Guardian brought us up the rest of the way no problem.

LOG:

No log

No obvious cause, https://ldas-jobs.ligo-wa.caltech.edu/~lockloss/index.cgi?event=1381958566

Kept losing lock from green arms, the ALS error signals didn't look great, so I started an Initial alignment at 15:12UTC which finished at 15:33UTC. I kept losing it at ALS from XARM_WFS_DOF1_P so I turned it off (I had to do this every lock attempt) and then DIFF_IR kept reported "IR not actually found" so I had to move it around by hand. During PRMI I misaligned PRM to check on the BS and it looked great on AS_AIR, guardian took us to CHECK_MICH while I was in the middle of checking the BS.

Defiantly something is misaligned, I couldn't get PRMI for a while and finally I tried to adjust SR3 in yaw to match its OPLEV from before the glitch and that got PRMI to finally lock, but it was unstable and lost it soon after, so something else is still badly aligned, the PIT signal looked good for SR3 based on the OPLEVS. PR3 OPLEV looks a little different as well so I adjusted it back to where it was.

I have to slowly step up in states to not lose lock going through ALS and IR, even with the DOF dance Xarm just loses it for seemingly no reason. I tried requesting LOCK_SLOW_NO_WFS let it settle in LOCKING_GREEN_ARMS then requested LOCKING_ALS and I did not have to turn off any WFS for Xarm. In CHECK_IR DIFF IR was not really found as it hasn't been every attempt. As I was trying to find DIFF_IR we went to EQ mode and lost lock from the quick ground motion increase (5.0 from Mexico?). I going to go make a cup of coffee while the ground motion calms down.

Investigating a dolphin crash of several corner station front ends.