Recieved a call at 10:38UTC regarding the detector having issues relocking. It was stuck in an Initial Alignment at Green Arms, and right when it started the initial alignment, almost every suspension saturated, including the RMs and SQZ suspensions. We were also getting constant IFO OUT and MC2 saturations while in green arms. At 10:39UTC the ISI BS Stage 2 WD had tripped after many BS saturations.

We also have multiple large earthquakes coming in from the Phillipines and so the first thing I did was take the detector to DOWN. The R-waves of the earthquakes are now hitting us, and ITMY Stages 1 and 2 WDs have also tripped due to a very sudden and large spike in ground motion. At 11:36 everything is calmer in the chambers so I reset the watchdogs on both ITMY stages and BS stage 2.

TITLE: 12/03 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: LOCKING
INCOMING OPERATOR: Oli
SHIFT SUMMARY:

Shift started out decently, but degraded with H1's lock durations decreasing to a point of only staying at NOMINAL LOW NOISE for a handful of minutes (usually as range was dropping).

Something new tried this evening was trying to reset the SQZ AS42 immediately upon reaching NOMINAL LOW NOISE (but had a short lock on the one attempt of this).

Had around 2.5hrs of Observing during the shift (mainly from the first lock of the shift).
LOG:

• 0033 NLN
• 0042 OBSERVING
• 0102 ASC SQZ needed a reset (there's no alarm for this but the range drops ~40Mpc)
• 0109-0113 Out of OBSERVING to reset ASC for SQZ (Naoki)
• 0209 Heard trouble alarm in the Computer Users room, but I was too slow to see anything on the display.
• 0300 LOCKLOSS
  • LOCK #1:  DRMI & PRMI looked bad, check mich fringes, but lockloss
  • LOCK #2:  check mich fringes completed, PRMI acquired with POP18 touching 60counts and went up to 100+ after ASC & NLN at 0354utc.
• 0358 OBSERVING
• 0412 Went out of Observing after watching DARM's 10-60Hz bump increasing & bringing range down for about 15min, until
  • SQZ ASC went OFF, but SQZ was unlocked this time.  So, I watched it as it was trying to recover, but then there was a lockloss.
• 0415 LOCKLOSS
  • 0503 random lockloss at TRANSITION FROM ETMX
• 0547 NLN & Observing at 0553
  • 0557 Lockloss while trying to do a SQZ AS42 reset
• 0646 NLN & 0654 back to Observing
• 0709 Lockloss while watching the range do sort of decently for almost 30min
• For the next lock, tried to walk PR3 + PR2, but no improvements, so undid any changes tried.

for the three locks since last night, h1 gets to 140Mpc and then starts to nosedive in range over the first 15-20min of lock.  For the 3rd lock, this happened---basically a big noise hump in DARM would grow and grow (from 10-60Hz).  I waited for the SQZ ASC to happen (it did, but I also think the squeezer lost lock---that was different from last night's lock and the lock at the beginning of the shift).  So I was waiting for the SQZ to relock so I could resest the SQZ ASC, but then there was a lockloss.

Naoki suggested for the next lock, to preemptively Reset AS42 (per 71083).  So I did this as soon as the Squeezer was Squeezing and all Guardians were GREEN.  To get to Observing I had to ACCEPT a list of OFFSET SDF diffs (attached), and then went to OBSERVING.

But as I was typing this alog, sadly H1 lost lock after only 10min (or 4min of Observing).

H1 was locked about 2.5hrs and then had a lockloss (similar to last night's lock of ~3hrs).  Just recovered from the lockloss with H1 back to OBSERVING in under one hour.  Will keep an eye on SQZ to see if the SQZ ASC turns OFF.  Range is just above 140Mpc.

L1 is now back up and OBSERVING.

Winds have calmed and microseism still high (@95th percentile).

As mentioned earlier, Tony (w/ Keita & Sheila on TeamSpeak) had H1 well on its way to NLN (after the big Phillipines EQ + Hardware Watchdog Recovery + the other "ongoing" issues all happening at the same time!).

Now that H1 is at NLN, it's been decided to keep H1 in OBSERVING at this lower BNS range (due to 20-50Hz noise seen on DARM, tagging DetChar) to get some single-detector-OBSERVING in ....BUT Fri night, Robert was allowed COMMISSIONING time if he wants it (H1 range is lower (~140Mpc), so H1 may possibly be too low for his work).

The other change is the Vertical Offset changes from yesterday have been reverted for ITMx, ITMy, & BS (see attached for SDF diffs which were ACCEPTED for this).  H1 also had Gaurdians for automated operation from me last night; IFO NOTIFY and H1Manager are now set to IDLE.

Finally, a new NOTE:  IMC_WFS has a notification:  "IMC WFS not centered" now flashing every few seconds.  I hadn't seen that yesterday.

TITLE: 12/03 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Earthquake
OUTGOING OPERATOR: Tony
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 17mph Gusts, 14mph 5min avg
    Primary useism: 0.05 μm/s
    Secondary useism: 0.54 μm/s
QUICK SUMMARY:

Walking in to seeing H1 locked w/ ASC improving DRMI & ISC_LOCK currently starting to power up H1.  Keita was called in earlier (& he and Sheila are chatting on TeamSpeak).

Definitely see effects of an EQ-rung Earth (starting 9.5hrs ago!)  PLUS   Windy day (gusts mostly only up to 20mph but 40+mph at 8amPST  PLUS  Microseism continues to be steady right at the 95th percentile.

(ISC_LOCK currently at MOVE_SPOTS)

TITLE: 12/03 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Earthquake
INCOMING OPERATOR: Corey
SHIFT SUMMARY:
 

Removing offsets that were applied yesterday.

ITMX
H1:SUS-ITMX_M0_TEST_V_OFFSET was set back to 0.
H1:SUS-ITMX_R0_TEST_V_OFFSET was set back to 0.

ITMY
H1:SUS-ITMY_M0_TEST_V_OFFSET was set back to 0.
H1:SUS-ITMX_R0_TEST_V_OFFSET was set back to 0.

and Beam Splitter's H1:SUS-BS_M1_TEST_V_OFFSET was set back to 0.
H1:SUS-BS_M1_TEST_V_SW1 was turned OFF

I Accepted all the SDFs for these channels, screenshot1 and screenshot2

After this I put ISC_LOCK into Manual green arms and Started to search for Light on ASL-Y arm. Fought with that for a while, until I realized that the Beam splitter needs to be aligned FIRST, which made getting good flashed on the arms much simpler.
Once I got good Flashes on both arms I started an inital alignment.

Initial Alignment completed, and it did go through SR2_ALIGN.

Started Locking! Got Past AQUIRE_DRMI_1F and we are currently at ENGAGE_ASC_FOR_FULL_IFO!

LOG:
Tony went to EX to UnTrip the Hardware Watchdog
Mike was giving a tour today.

The attached plot shows the ST1 H1,H2,H3 DAC drives for, from upper to lower, ETMX, ITMX, ETMY, ITMY.

H1=blue, H2=orange, H3=green.

ETMY does not rail at all, ITMY all 3 rail for 6 seconds, ITMX H1,H3 rail for 7 seconds, ETMX all 3 rail for 12 seconds.

ETMX only differs from ITMY in the length of time the DAC drives are saturated at 10V (12 sec vs 6 sec).

Sat Dec 02 10:05:06 2023 INFO: Fill completed in 5min 3secs

Ok First thing I did is start trying to get things into a safe state.
ETMX was in a bad SEI State, due to ISI being tripped and HEPI being tripped.
Simply resetting the watchdog on the HEPI didnt work because the Hardware WatchDog was tripped:  H1:SUS-ETMX_HWWD_TTF_MIN
This also stopped me from untripping the following:
H1:IOP-SUS_ETMX_DACKILL_STATE
H1:IOP-SEI_ETMX_DACKILL_STATE

While talking with Dave, Jim, And Fil about the Sequence that these things need to be turned back on.
noticed that the DCPD screen was showing an increase in light on the PDs. I snapped a before picture of the DCPD screen.
I went down to the End station to Reset the Hardware Watchdog while on teamspeak with Dave.

I took pictures of the Hardware Watchdog and the CoilDriver chassis before I touched anything.
Simply pushing and holding the Fault Reset button resets the Hardware watchdog. 
From there, I could reset the ISI and then the HEPI.
Coming back to the control room the ring up had been stopped.


Over the course of the morning 12 earthquakes have rolled through. 3 of which have been over 6.0 Magnitude.
 

We are investigating why the ETMX HWWD saw high SUS RMS for 20 minutes which caused it to trip, whereas all the other HWWDs saw less than 1 minute of high RMS. It would appear that ETMX's high RMS stopped within one minute of the time the HWWD powered down the ETMX ISI coil drivers.

Just like gauge at X2-8, the gauge at Y2-8 is not reporting a pressure as of 1 hour ago, due to the lack of sunlight needed to charge batteries via solar panels. No action is needed at this time.