• TITLE: 08/16 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
  STATE of H1: Observing at 156Mpc
  INCOMING OPERATOR: Corey
  SHIFT SUMMARY: Failed EX/TMSX DAC card was swapped. It's been a windy day, we had 2 locklosses, both with windy tags. All three aquisitions were automatic. We've been locked for ~1.5 hours.
  LOG:                                                                                                                                                                        

  Start Time	System	Name	Location	Lazer_Haz	Task	Time End
  18:49	LASER	LVEA is LASER HAZARD	LVEA	YES	LVEA IS LASER HAZARD	09:49
  14:40	EE	Marc	EndX	N	Replace failed DAC card	15:12

• 16:24 UTC Observing
  • 16:32 - 16:35 UTC We dropped Observing as the H1:CAL-INJ_CW_OUTPUT was not outputting anything, it restarted on its own after a few minutes? I toggled the master switch off and back on but I don't think that did anything.
  • 19:05 UTC lockloss right before the calibration measurement (I had asked LLO if we could delay from 18:30 to 19:15)
• 20:27 UTC Observing
  • 20:51 UTC lockloss

• 21:56 UTC Observing

TITLE: 08/16 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 155Mpc
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
    SEI_ENV state: SEISMON_ALERT
    Wind: 24mph Gusts, 12mph 3min avg
    Primary useism: 0.06 μm/s
    Secondary useism: 0.20 μm/s
QUICK SUMMARY:

H1's been locked ~1.5hrs and we continue to have the fairly constant breeze hovering around 20mph (with some gusts over 30mph--no doubt contributing to the locklosses today perhaps).

Ryan was not able to run the Sat calibration due to locklosses and not having a thermalized H1.  If L1 happens to have a lockloss this evening, I can run one (otherwise, we waiting game/TOI).

Microseism has been trending up the last ~12hrs (currently flattened out above the 50th percentile....it had been below it).  Perhaps this has teamed up with the winds for the locklosses today.  (Have a standard M5.5 Tonga EQ inbound, and just transitioned to EQ Mode.  So have winds + increased microseism + incoming EQ).  Been a while since we've juggled multiple seismic sources.

20:51 UTC lockloss

19:05 UTC lockloss while we were in the last 15 minutes of thermalizing before the calibration measurement.

Sat Aug 16 10:09:02 2025 INFO: Fill completed in 8min 59secs

Marc replaced the bad 18bit DAC, system was powered back up 07:57

I can see all the cards in the IO Chassis, h1iopsusex started with no problems, followed by the user models.

I reset the SWWD on SUS and SEI.

HWWD had not tripped overnight.

Marc is heading out of EX, Ryan is untripping the user model watchdogs and preparing for IFO recovery.

I called Marc this morning and he headed out to site. Dave confirmed (86389) that the card that needs to be replaced is the one we swapped in a couple weeks ago (86086) while trying to solve the TMSX ASC locklosses (86079). Since the issues stopped after the old card was swapped out, we know that it was what was causing the issues. Unfortunately this means that that old DAC card won't be a good replacement for this failed card, so Marc is hunting around the EE lab looking for a different 18-bit DAC.

TITLE: 08/16 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
OUTGOING OPERATOR: Oli
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 29mph Gusts, 18mph 3min avg
    Primary useism: 0.04 μm/s
    Secondary useism: 0.12 μm/s
QUICK SUMMARY:

• We've been down for a few hours as a DAC card at EX has failed alog86389
  • Marc is finding a spare to swap in
  • We were locked for just over 25hours when the card failed
• Wind has been moderate overnight, but we rode through all of it
• Secondary microseism is starting to creep up a little

WP12760

Marc, Oli, Fil, Ryan C, Dave:

Marc is onsite preparing to drive to EX to replace the failed 18bit-DAC card in h1susex IO Chassis.

Front end is fenced and powered down.

At approximately 23:55 Friday 15 August 2025 PDT all models on h1susex failed with ADC timeout.

lspci showed that Adnaco2 slot4 looked not quite correct. This is the 3rd 18bit-DAC which is only used by the h1sustmsx model and was replaced recently during lock-loss investigations.

I power cycled h1susex following the procedure:

fence from Dolphin, stop all models, power down via terminal, wait, power up via IPMI.

Now lspci is not reporting any card in A2-S4, indicating this 18bit DAC has failed.

I was alerted to an issue by H1 MANAGER at midnight. One of the DACs failed at ETMX and took down ETMX and TMSX, and tripped the SEI software watchdog. I called Dave and he tried restarting the computer, but it didn't work. Richard said to try calling Fil and if he didn't answer, to wait until morning, and Fil didn't answer, so I'll try again at 6am to get him on the phone to see what can be done. For now, I've put ISC_LOCK in IDLE, and I've bypassed the software watchdog to restore the ISI and HEPI to their nominal states

TITLE: 08/15 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 156Mpc
INCOMING OPERATOR: Oli
SHIFT SUMMARY:

Smooth sailing shift with H1 locked over 23hrs even with our windy night (it's been 20-25mph steady winds at the corner for the last 12hrs).
LOG:

• 0131 GRB-Short E590468

For FAMIS #26591: All fan trends look nice and flat!