Sat Aug 16 10:09:02 2025 INFO: Fill completed in 8min 59secs

Marc replaced the bad 18bit DAC, system was powered back up 07:57

I can see all the cards in the IO Chassis, h1iopsusex started with no problems, followed by the user models.

I reset the SWWD on SUS and SEI.

HWWD had not tripped overnight.

Marc is heading out of EX, Ryan is untripping the user model watchdogs and preparing for IFO recovery.

I called Marc this morning and he headed out to site. Dave confirmed (86389) that the card that needs to be replaced is the one we swapped in a couple weeks ago (86086) while trying to solve the TMSX ASC locklosses (86079). Since the issues stopped after the old card was swapped out, we know that it was what was causing the issues. Unfortunately this means that that old DAC card won't be a good replacement for this failed card, so Marc is hunting around the EE lab looking for a different 18-bit DAC.

TITLE: 08/16 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
OUTGOING OPERATOR: Oli
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 29mph Gusts, 18mph 3min avg
    Primary useism: 0.04 μm/s
    Secondary useism: 0.12 μm/s
QUICK SUMMARY:

• We've been down for a few hours as a DAC card at EX has failed alog86389
  • Marc is finding a spare to swap in
  • We were locked for just over 25hours when the card failed
• Wind has been moderate overnight, but we rode through all of it
• Secondary microseism is starting to creep up a little

WP12760

Marc, Oli, Fil, Ryan C, Dave:

Marc is onsite preparing to drive to EX to replace the failed 18bit-DAC card in h1susex IO Chassis.

Front end is fenced and powered down.

At approximately 23:55 Friday 15 August 2025 PDT all models on h1susex failed with ADC timeout.

lspci showed that Adnaco2 slot4 looked not quite correct. This is the 3rd 18bit-DAC which is only used by the h1sustmsx model and was replaced recently during lock-loss investigations.

I power cycled h1susex following the procedure:

fence from Dolphin, stop all models, power down via terminal, wait, power up via IPMI.

Now lspci is not reporting any card in A2-S4, indicating this 18bit DAC has failed.

I was alerted to an issue by H1 MANAGER at midnight. One of the DACs failed at ETMX and took down ETMX and TMSX, and tripped the SEI software watchdog. I called Dave and he tried restarting the computer, but it didn't work. Richard said to try calling Fil and if he didn't answer, to wait until morning, and Fil didn't answer, so I'll try again at 6am to get him on the phone to see what can be done. For now, I've put ISC_LOCK in IDLE, and I've bypassed the software watchdog to restore the ISI and HEPI to their nominal states

TITLE: 08/15 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 156Mpc
INCOMING OPERATOR: Oli
SHIFT SUMMARY:

Smooth sailing shift with H1 locked over 23hrs even with our windy night (it's been 20-25mph steady winds at the corner for the last 12hrs).
LOG:

• 0131 GRB-Short E590468

For FAMIS #26591: All fan trends look nice and flat!

TITLE: 08/15 Eve Shift: 2330-0500 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 154Mpc
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 22mph Gusts, 14mph 3min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.08 μm/s
QUICK SUMMARY:

Easy & smooth shift handoff (as it should be) with H1 locked 17.75hrs!  And it has even been windy the last 7hrs (Corner winds over 20mph).

TITLE: 08/15 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 154Mpc
INCOMING OPERATOR: Corey
SHIFT SUMMARY: We stayed locked the entire shift, 17.5 hours.
LOG:                                                                                                                                                                                                     

20:55 UTC There was both a plane and a helicopter flying over site (DETCHAR)

Summary:

This post provides an estimate of the thermal transition caused by changing the JAC input power from 0 W to 100 W. The result will be used for the design of the JAC heater.
The main conclusion is :Changing the JAC input power from 0 W to 100 W results in a cavity length change of ≈ 0.2 µm (corresponding to ≈ 100 V PZT actuation), which remains within the PZT range. No heater-based compensation is required. 

Details:

Two possible types of state change are considered. One is the effect in which light scattered from the mirrors is absorbed by the cavity body, heating it and causing the entire cavity to expand. This is expected to have a long transition time. The other is expansion of the mirrors themselves due to absorption, which shortens the cavity length and is expected to have a relatively short time constant.

The attached plot shows one hour of trend data starting from 2024-11-14 19:30 UTC. From top to bottom, the traces are: PMC transmitted power, PMC temperature from the thermistor, heater input, and PZT input.
This lock occurred after the PMC had been unlocked for about three hours, so it represents the transition from a cold state to a hot state. Just before locking, the PMC temperature was about 307.0 K, and immediately after locking, the PZT input voltage was 370 V.

Although the heater input changes appear to cause rapid changes in the PMC temperature, these are not reliable indicators of the actual body temperature change. The observed change is about 0.4 K, but given aluminum’s thermal expansion coefficient of 10^{-5} m/K, a 0.1 K change would shift the cavity length by one FSR, which would exceed the PZT range and cause lock loss. This suggests that the rapid apparent temperature changes are due to the thermistor picking up radiation from the heater or sensing the local temperature near the heater, rather than the true body temperature.

Therefore, to evaluate the body temperature change, we need to disregard these fast variations and look at the average thermistor signal. This shows that the change is less than 0.1 K over one hour, with the highest temperature occurring immediately after lock. This indicates that the observed temperature change is due to the heater input change, and that the body expansion can be neglected.

The important point is that although the temperature just after lock and at t = 3500s is almost the same, the PZT input voltage changes by about 100 V. This is interpreted as a change in the mirror state. A 100 V change corresponds to a cavity length change of about 0.2 µm, which can be attributed to the change in input power.

TITLE: 08/15 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 157Mpc
OUTGOING OPERATOR: Oli
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 5mph Gusts, 3mph 3min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.09 μm/s
QUICK SUMMARY:

• We've been locked for 8.5 hours
• WInd is low, secondary microseism is low
• The range has been fairly steady this lock
• 11:49 UTC GRB-Short E590227

Today Oli and I saw that the ADS convergence checker for the beamsplitter was taking forever to return True during initial alignment, despite the fact that the signals appeared well-converged. The convergence threshold is set on line 1158 of ALIGN_IFO.py, and it is 1.5 for both pitch and yaw. Watching the log, the yaw output seemed to quickly be below this value, while the pitch output hovered between about 1.8 to 6. I tried raising the value to 5, but pitch still stayed mostly above that value. I finally changed it to 10, and the state completed. Overall, we were waiting for convergence for over 16 minutes. It seems like the convergence values for pitch and yaw should be different. It took about two minutes for the pitch and yaw ADS outputs to reach their steady value. On ndscope minute trend, the yaw average value appears to be around zero, while for pitch the average value is around -4. The convergence checker averages over 20 seconds.

The value is still 10, but that might be too high.