TITLE: 07/16 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 132Mpc Been at NLN 3h10.
SHIFT SUMMARY: Unknown lockloss and a lot of time damping violins today. LLO is reporting stormy weather so we may have a TOO later for some of Robert's injections.
LOG: ESD_EXC_{ETMX,ETMY,ITMX,ITMY} can reloaded for 71330 if we loose lock or come out of observing.

• 17:06 - Lockloss 1373562320 after 2h30 in NLN.
  • Coincided with CP1 fill 71380
  • 1h40m in OMC_WHITENING for violins to damp, mainly EX04 and EX07
  • Relock fully automatic
• 18:17 - 19:25 OMC injections 71382
• 19:26 - 19:46 EQ mode activated
• 19:48 NLN and Observing

Add on to yesterdays work 71354

We will schedule a reboot. The web snapshots are frozen from 04:14PDT

Sun Jul 16 10:05:27 2023 INFO: Fill completed in 5min 26secs

1373562320 After 2h30 in NLN. No obvious cause.

Locklosses from NLN 07/09 to 07/16:

• 2023-07-09 13:00:54 ISI Issue FRS28481 Link
• 2023-07-10 07:57:37 Unknown Link NLN for 2h29
• 2023-07-10 15:57:45 Unknown Link NLN for 4h57
• 2023-07-10 20:45:37 Earthquake Link
• 2023-07-11 15:08:16 Tues Maintenance
• 2023-07-11 22:29:29 Unknown Link NLN for 2h00
• 2023-07-12 17:25:48 Interlock Tripped Link
• 2023-07-13 08:09:53 Unknown Link NLN for 7h05
• 2023-07-14 09:36:44 Earthquake Link
• 2023-07-14 16:18:25 Camilla Error Link
• 2023-07-14 22:29:00 Earthquake Link
• 2023-07-15 19:06:32 Unknown Link NLN for 18h32
• 2023-07-16 06:54:21 Earthquake Link
• 2023-07-16 12:40:58 Unknown Link NLN for 0h10

Total 14: 6 unknown, 4 earthquake, 2 user error, 1 hardware issue, 1 expected (Tues Maintenance).

TITLE: 07/16 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 135Mpc
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 9mph Gusts, 5mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.04 μm/s
QUICK SUMMARY: Been in NLN 30minutes.

Dust monitors, VAC, CDS, SUS, SEI all good.

TITLE: 07/16 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Earthquake
SHIFT SUMMARY:

Shift started with a whopper Alaskan earthquake and it took 5.5hrs to get back to OBSERVING.  (...but sadly, H1 had a lockloss 10min later!).  There was a Tsunami Warning triggered (albeit retracted/downgraded shortly afterward).

The above lockloss was a bit of mystery (Lockloss file is noted as a "FAIL", but it also lists a "Board Saturation"

Kind of forced to run an Initial Alignment because ISC LOCK quickly makes the call of whether to give PRMI a chance or not.  (but in hindsight, did get H1 to Observing after an Initial Alignment)
LOG:

• (0654) 7.4M Alaska EQ takes down H1 (seismic FOM attached):  tripped 7-SEI ISIs, RED picket fence, SEI Peakmon mean hit about 100,000 counts (which is offscreen on the ISI_config medm which only goes up to 2,000 counts!); IMC has trouble locking for dozens of minutes (but this could also be due to low diffracted power that's been noted), keeping ISC LOCK in PREP FOR LOCKING for about an hour.
  • 756 Have all ISIs back to their nominal states
  • 0803 Attempting LOCKING ARMS GREEN....Y-arm miraculously locked for a handful of seconds...but then both x&y went into INCREASE FLASHES.
  • 0914 Attempted to LOCK ARMS GREEN again---Xarm offloaded (Woo woo!), Yarm took a little more time, but also offloaded a few minutes later.
  • 0935 CHECK MICH FRINGES offloaded, but then lockloss.
  • 0950 First look at PRMI---pretty misaligned in yaw...quickly tried steering PRM in yaw, but then ISC_LOCK quickly went to CHECK MICH FRINGES again :-/
    • I don't like that ISC LOCK only gives ACQUIRE PRMI a few seconds (I was able to improve PRMI flahses, but then ISC LOCK went to CHECK MICH FRINGES -again-!  Guess I have to run an alignment since ISC LOCK won't allow me to tweak PRM without being rushed. :(
  • 1003-1101 Initial Alignment (on a shaky planet) started  (all the while Picket Fence continues to flash yellow here and there)
    • SRC ALIGNdidnt' work on first try (stuck at acquire sry).  On 2nd try, it went through an SR2 align (it skipped that step the first time.  Hmmm.)
    • Still would not acquire SRY.
    • So aligned SR2 & SRM by hand and tried SRC_ALIGN again....but it would not LOCK SRY.
    • Skipped!  Ended Alignment at 1101
  • 1050 Seismic to CALM
• 1108 LOCK #1 (post EQ): On first DRMI acquire it locked within seconds!
  • In contrast to the Chiapas eq Fri morning, ISC LOCK proceeded right to powering up (Fri violins were so rung up that H1 hung out at CHECK VIOLINS BEFORE POWERUP for quite a bit of time).
  • 1143 Waiting at OMC WHITENING, DCPDs are at 1,842,980 counts!! (target is under 524k!)
    • 1201Have all violin modes to their nominal gains (counts continue to drop, but still at about 1million!)
    • 1223 OMC Whitening completes after violins are good.
  • 1230 OBSERVING
• 1240 LOCK #2:  LOCKLOSS from OBSERVING
• 1301 LOCK #3:  LOCKLOSS at OFFLOAD DRMI ASC
• 1311 LOCK #4:  LOCKLOSS at START TR CARM
• 1312 LOCK #5:  OMC WHITENING had DCPDs at OVER 2-million counts(!) 
  • 1435 OBSERVING
• 1405 EQ Mode Activated (5.1M EQ in Mexico)
• 1425 EQ Mode to Calm

Lockloss at 1240utc.

There were no environmental issues and all the ISC FOMs looked fine. :-/

Much of the first half of the shift involved waiting for the earth to stop ringing due to the Alaska earthquake. 

Did my darnedest to avoid an Initial Alignment, but ISC_LOCK pretty much forced me to.  INITIAL ALIGNMENT successful for every step except for SRC ALIGN (I later aligned SR2 & SRM by hand, but SRC ALIGN was still NOT HAPPY & FAILED, so I skipped it and ended the INITIAL ALIGNMENT.).

[SEI_ENV finally went to calm at 1050utc (350amPT)---almost 4hrs later!]

After Alignnment, DRMI locked immediately when prompted!  Now continuing through ISC LOCK.

TITLE: 07/15 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 132Mpc
SHIFT SUMMARY: Been in NLN 1h55, still haven't had a TOO to take the 'pydarm measure --run-headless bb' calibration measurement.
LOG:

• Lockloss @ 19:06UTC 1373483210 alog 71353
  • 20:03-20:07UTC IMC_LOCK was struggling during laser noise suppression with log message "Diffracted power jumped too much, toggling secondloop", didn't look into how long this has been going on for Tagging PSL
  • During relock while damping violins I did some OMC shaking 71354
• Back to NLN and Observing at 21:08UTC
  • Small EQ from Japan at ~21:50UTC

[Jenne, Naoki, Brina, Caden, Robert, Lance, Camilla]

Since alignment changes were so impactful yesterday (alog 71302) and we still have higher noise than when OM2 was hot last week, and also higher noise than our April 60W times with cold OM2 (eg, Elenna's alog 71284 yesterday), we tried some alignment shifts today with some PEM vibration injections going, to see if we could find a better alignment.  For now, we've left the IFO with the same settings it had last night since there's nothing that's significantly better.  Nothing we've found comes close to matching the broadband (good) noise level of the April 60W time, or the last week hot OM2 times.

• The camera servos are very slow, so I wanted to switch back to ADS. However, just asking the camera servo to do so dramatically changed the alignment of the IFO; arm buildups were going down and some of the ASC looked unstable.
  • There isn't a handy way to switch back to the camera servos without having the offsets be set to your current position, so we used the Camera servo guardian to go back to using the cameras (which reset the camera offsets), and then Naoki by hand set the camera servo offsets to where we had been overnight.
  • I then turned on the A2L excitations (but not the servos), and changed the A2L gains to zero the error signals.
  • Then we again used the camera servo to switch over to ADS, and we saw much less change in the circulating power buildups. The third attachment shows the situation at that time. 
    • Note that this means that after a long time of thermalization, the place the ADS would keep the IFO is different than the place the camera servos keep the IFO.  I *think* that it's better with the camera servos, since that's where we're maintaining better buildups. 
• Once back on ADS, I tried moving A2L setpoints to see if that would change the jitter peak around 120 Hz, and see if it would change the broadband excess noise we seem to have. I also tried changing the AS36Q yaw offset, but I think that even with A2L moves, the current value of 35k count offset is the best.
  • I can pretty dramatically change the circulating arm powers.  In particular, changing the ETM Y2L set points was giving us an extra 10kW circulating, but that seemed to not be the best place.
• Robert reminded me / pointed out that the 120 Hz jitter peak isn't necessarily the only thing to be mindful of, since he can likely move that by adjusting damping and weights on the PSL periscope, so we should probably focus on the broadband noise increase and try to fix that. Unfortunately, none of the moves we made today seemed to affect it. 
• Around 21:50:00 UTC seemed to be a pretty good time, but again not better than with the hot OM2.
• We ended things by trying to improve some of the low freq by changing the ITMY A2L gains.  The second attachment shows where I left things when we lost lock due to a closeby earthquake. 

The first attachment shows a big-picture of the moves we made today. 

Daniel points out that the time with OM2 hot may have been some of our best-ever sensitivity around 70 Hz.  It would be helpful to have a plot like alog 71309, but including Elenna's time from April 6th, to see if pre-O4 we ever had as good of sensitivity around 70 Hz. 

It's also a little tricky to compare our HWS data using our plotter to what Elenna and Cao posted yesterday.  It would be helpful to either have the times from Cao's plot, or have Cao plot where it looks like our ITMX spot was at 21:50 UTC today. 

The TRANSITION_BACK_TO_ETMX state of the SUS_CHARGE  guardian has been causing the majority of our locklosses prior to Tuesday Maintenance, e.g. 69437, 70861.

I looked into this again and found that the ITMX LOCK_L gain is being turned off before the ITMX to ETMX transition has completed. The ramp time of the swap is 20 seconds but the code only waits 15 seconds before moving on, see attached plot and code.

We are using a ezca.get_LIGOFilter and the wait=True parameter isn't waiting the full ramp_time specified, created Issue 15. I've changed the wait to be False and added a 20s sleep timer. The SUS_CHARGE guardian will need to be reloaded when we are out of observing.