Ryan, Rahul, Fil, Dave:

All h1sush7 models stopped running at:

PDT: 2023-09-21 03:48:52.000000 PDT
UTC: 2023-09-21 10:48:52.000000 UTC
GPS: 1379328550.000000
 

From this time onwards H1 was still in lock with a depressed range of ~130MPc and was out of OBSERVE. h1seih7 SWWDs were tripped.

Recovery process was:

Stop models, fence h1sush7 from the Dophin fabric, reboot h1sush7.

When h1sush7 came back, I verified that the IO Chassis could not be seen. I then fenced and powered the front end down.

Fil went onto the mech room mezzanine and verified that the SUS side of the Kepco dual-power-supply had tripped (SEI side was OK). He powered the IO Chassis back on, I powered h1sush7 computer and all came back correctly.

I untripped the SWWDs to get h1seih7 driving again, Ryan and Rahul recovered the SUS models.

Famis 26458
HEPI Pump Trends for the last 45 days are attached.
H1:HPI-PUMP_L0_CONTROL_VOUT has seen a change of 8[units], in the last 3 days. Im not sure if movement of 8 is unreasonable or not. Also what are the units of these channels?
 

Below is the summary of the LHO DQ shift for the week of 2023-09-11 to 2023-09-17

• In observing for the week ~78%. (58.6%, 71.7%, 68.5%, 89.3%, 91.7%, 88.7%, 79.0%)
• Range: Mainly in the 140 Mpc to 150 Mpc range, with some slow/short-term fluctuations
• Lock Losses are mostly due to maintenance work/commissioning/calibration
• Combs: 29.96 Hz, 9.48 Hz, 4.98 Hz on days when Fscan are available. Line counts are mostly below mean or on average.
• Hveto: H1:ASC-DHARD_Y_OUT_DQ was round winner for 6 days. The rest of the winning channels are similar to that of last week except a new channel: H1:ISI-GND_STS_ITMY_Z_DQ showing up in round winner.
• Few pyCBC triggers with SNR > 10 (primarily in short) 

The full DQ shift report with day by day details is available at: https://wiki.ligo.org/DetChar/DataQuality/DQShiftLHO20230911

TITLE: 09/21 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 11mph Gusts, 8mph 5min avg
    Primary useism: 0.04 μm/s
    Secondary useism: 0.25 μm/s
QUICK SUMMARY: As mentioned by RyanC in his alog, IOPSUSH7 is showing several issues, causing the SWWD for HAM7 to be tripped; tagging CDS. This has left H1 without squeezing and out of observing since 10:48 UTC, but it has still been locked at low noise for almost 13 hours.

HAM7 ISI watchdog tripped at 10:53UTC from the ACT, the ACT actuators saw a large glitch multiple times. Each time I reset the watchdog the act goes crazy. Also I just noticed that HAM7 isn't on the SUS software watchdog medm screen. There's an IOP dackill as well. I gave Jim a call and he's gonna hop on and check it out.

TITLE: 09/20 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 156Mpc
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY:

- Relocking upon arrival, back to NLN @ 23:36, OBSERVE @ 23:47

• Violins are bit rung up from today's earlier locklosses, but they look to be damping ok
• EX saturation @ 00:12 UTC
• Lockloss @ 0:42

- 1:18 - inc 5.4 EQ from Alaska

• 1:27 - EQ mode activated, back to CALM @ 1:47
• Lockloss @ LASER NOISE SUPPRESION from the EQ 

- Reaquisition was automated, NLN @ 2:21, OBSERVE @ 2:39

• 2:42 - GRB-Short E439530 - STAND DOWN alert
• EX saturation @ 2:59

LOG:

No log for this shift.

Following a couple locklosses, H1 is back in observing mode. Seismic motion seems to have leveled out from the EQ earlier and all systems appear to be stable. Currently in a STAND DOWN due to a GRB short detection.

Lockloss @ 0:42, cause unknown. Not seeing a whole lot of evidence for an ASC ringup. On the LSC side of things, it would seem like LSC DARM IN1 is the first to see the instability followed shortly by the ASC AS A DC PD, scope attached. Looking at the locklost website, it would seem that the ETMX L2 UL/UR OSEM stages are seeing the elevated oscillations ~30 before the lockloss occurred.

H1EPICS_HWS.ini has 88 ITMY channels, all are connecting except the following 6:

H1:TCS-ITMY_HWS_BEAM_POS_X
H1:TCS-ITMY_HWS_BEAM_POS_Y
H1:TCS-ITMY_HWS_CO2_POS_X
H1:TCS-ITMY_HWS_CO2_POS_Y
H1:TCS-ITMY_HWS_RH_POS_X
H1:TCS-ITMY_HWS_RH_POS_Y
 

TITLE: 09/20 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Austin
SHIFT SUMMARY: H1 was unlocked for most of the morning following an earthquake, then commissioning time this afternoon. One commissioning-caused lockloss has H1 currently relocking.

• 14:38 Lockloss - ETMX saturation right before
• 15:02 Entered EQ mode - 5.2, 6.0, and 5.7 off the coast of Ecuador
• 16:30 Left EQ mode, started locking
• 17:02 Bubba noted RPD is testing ordinance at HAMMER until 00:00 (17:00 PT) this evening (~8 miles away), tagging DetChar
• 17:51 Reached NLN
• 18:10 Started observing
• 19:12 Dropped observing for coordinated commissioning time
• 20:29 Lockloss - alog - from commissioning activities
• 21:42 Reached NLN, resuming commissioning
• 22:45 Lockloss - from commissioning activities

LOG:

TITLE: 09/20 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Ryan S
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 19mph Gusts, 12mph 5min avg
    Primary useism: 0.06 μm/s
    Secondary useism: 0.28 μm/s 
QUICK SUMMARY:

- H1 is currently relocking, wirh DRMI just locked

- CDS/SEI ok

This was a good test of our new Picket Fence monitor EPICS channels. The PKT circle on the CDS Oveview is red, and the Picket Fence MEDM (attached) shows that the last update is many minutes ago (should never exceed 4 seconds).

I'm taking a look at the server and will restart it.