Relocking has been a struggle, high state locklosses, LOW_NOISE_COIL_DRIVERS, TRANSITION_FROM_ETMX, MAX_POWER.... Currently at 25W

DIAG_MAIN reports that the HWS ITMY code has stopped, violin modes were rung up during a MAX_POWER lockloss and have been giving me trouble since. I've also been getting lots of IY saturations during higher states (MOVE_SPOTS, DARM_TO_DC_READOUT, ...)

Sun Dec 24 10:06:47 2023 INFO: Fill completed in 6min 43secs

TCmins -79C/-73C

I restarted the slow-controls SDF system h1cdssdf on h1ecatmon0, running now with its full list of channels. It had been running without the lock-loss-alert channels since Thursday's cdslogin reboot.

TITLE: 12/24 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Earthquake
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 13mph Gusts, 10mph 5min avg
    Primary useism: 0.11 μm/s
    Secondary useism: 0.59 μm/s
QUICK SUMMARY:

• Microseism is high
• We're relocking following an EQ lockloss

After a 44 hour lock, the IFO needed some help to keep the green arms locked. The useism has been getting pretty high, so I tried the USEISM state in SEI_CONF and green was immediately able to hold. I changed the default states in SEI_ENV to be the Useism and Useism_Earthquake states, but these should be reverted when the useism is back down.

We have finished an alignment and are up to full power with this change.

TITLE: 12/24 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 158Mpc
INCOMING OPERATOR: TJ
SHIFT SUMMARY: Very quiet evening with a few earthquakes passing through. H1 was locked and observing the whole time; current lock strecth is coming up on 44 hours.

• 00:07 Incoming EQ from Tonga
  • 00:25 EQ mode activated
  • 01:03 Seismic to CALM
• 02:18 Incoming EQ from New Caledonia
  • 02:43 EQ mode activated
  • 03:24 Seismic to CALM
• 03:42 Incoming EQ from Chile
  • 04:13 EQ mode activated
  • 04:33 Seismic to CALM

Over the course of the day today, Ryan and I have noticed the BNS inspiral range dropping starting around noon PST. Since it's mostly come back at this point almost 12 hours later, it seems to be correlated to a drop in LVEA temperature which in turn has raised the large corner station optics (BS and ITMs) according to their vertical top-mass OSEMs (see one- and six- day trends attached). Perhaps the HVAC system lowered the heat into the LVEA in response to the outside temperature rising? Since H1 has remained locked and observing over this time, I imagine the angular control loops have been keeping up with alignment changes, but possibly altering noise in DARM which has been affecting range. I don't believe any action is needed here, but I think it's still worth noting here how corner station temperature changes can affect our observation performance.

State of H1: Observing at 151Mpc

H1 has been locked for almost 40 hours. EQ mode activated twice for quakes passing through, but otherwise it's been a quiet evening so far.

TITLE: 12/23 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 150Mpc
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 4mph Gusts, 3mph 5min avg
    Primary useism: 0.04 μm/s
    Secondary useism: 0.40 μm/s
QUICK SUMMARY:

H1 has been locked for 35.5 hours. I'll keep an eye on LVEA temps since they seem to have dropped slightly in the past couple hours. BNS range also seems to have dropped a bit.

TITLE: 12/23 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 152Mpc
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY: Quiet day, I ran a calibration measurement, and we stayed locked the whole shift (35:40). The range has been dropping slightly over the past hour or so, maybe related to vea small temperature changes?

16:35 I went into the warehouse by the VPW to restart the cluster thermostats after the Jonathon called to let me know it was getting warm in there which I had also done last Saturday the 16th.

18:08 I went out to the H2 enclosure and power cycled the 2 in use thermostats and adjusted the fan.

18:00 EQ from Japan, EQ mode at 18:08 UTC, back to CALM 20 minutes later

19:01 I dropped Observing to run the calibration sweep while LLO was relocking following the EQ and went back into Observing at 19:31UTC

20:30 to 21:00UTC I did a quick corner station building walk through, OSB, LSB, mech room, LExC, and staging (I did the H2 encl and VPW in the morning) and saw nothing noteworthy.

H1:PEM-Y_EBAY_RACK2_TEMPERATURE high temperature alert, almost 7 degrees above nominal but its trending down, tagging FMCS, PEM

STATE of H1: Observing at 163Mpc

• We've been locked for 31:42
• Low wind, microseism is trending down
• I ran a cal sweep from 19:01 to 19:30UTC

I started the cal sweep at 19:01 with the Broadband then I did the simulines.

BB:

/ligo/groups/cal/H1/measurements/PCALY2DARM_BB/PCALY2DARM_BB_20231223T190155Z.xml

19:08 Simulines started, GPS start: 1387393698.4

File written out to: /ligo/groups/cal/H1/measurements/DARMOLG_SS/DARMOLG_SS_20231223T190802Z.hdf5
2023-12-23 19:30:03,533 | INFO | File written out to: /ligo/groups/cal/H1/measurements/PCALY2DARM_SS/PCALY2DARM_SS_20231223T190802Z.hdf5
2023-12-23 19:30:03,544 | INFO | File written out to: /ligo/groups/cal/H1/measurements/SUSETMX_L1_SS/SUSETMX_L1_SS_20231223T190802Z.hdf5
2023-12-23 19:30:03,555 | INFO | File written out to: /ligo/groups/cal/H1/measurements/SUSETMX_L2_SS/SUSETMX_L2_SS_20231223T190802Z.hdf5
2023-12-23 19:30:03,565 | INFO | File written out to: /ligo/groups/cal/H1/measurements/SUSETMX_L3_SS/SUSETMX_L3_SS_20231223T190802Z.hdf5
ICE default IO error handler doing an exit(), pid = 42939, errno = 32
GPS end: 1387395021.653981

Sat Dec 23 10:11:09 2023 INFO: Fill completed in 11min 5secs

At 19:58 Fri 22 Dec 2023 the h1hwsex computer went down. This caused two issues:

1. EDC was disconnected from the 84 HWS_ETMX channels
2. h1tcshwssdf slow_controls SDF had disconnected channels

Camilla has confirmed that the HWS_ETMX camera is not needed over the break.

My HWS camera control code had put this camera into external_trigger mode when H1 locked early Friday morning, it is still in this mode, log entry:

• Fri 22Dec2023 04:12:04: Turned camera OFF

To make EDC and SDF green again, I have started a simulation HWS_ETMX IOC on cdsioc0, similar to what we were already doing for HWS_ETMY.

The simulation has all of its PVs set with VAL=0.0 except for those channels with non-zero values in the OBSERVE.snap and safe.snap SDF files. This ensures the SDF diff count is zero.

To get this going quickly I am running it on cdsioc0 as user ioc in a tmux session. Later I'll work on getting it under systemd control.
 

TITLE: 12/23 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 160Mpc
OUTGOING OPERATOR: Tony
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 5mph Gusts, 4mph 5min avg
    Primary useism: 0.04 μm/s
    Secondary useism: 0.42 μm/s
QUICK SUMMARY:

• CDS has 84 channels not connecting, Seems to be TCS-ETMX_HWS channels, tagging CDS, TCS
• MX temperature alarm but it looks fine now
• We've been locked for 27:43

TITLE: 12/23 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 156Mpc
INCOMING OPERATOR: Tony
SHIFT SUMMARY:

H1 was locked and observing the entire shift; current lock stretch coming up on 20 hours. One event candidate, S231223j, but nothing else of note on this very quiet evening.

State of H1: Observing at 161Mpc

H1 has been locked for almost 16 hours. Very quiet evening so far.

TITLE: 12/22 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 160Mpc
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
    SEI_ENV state: SEISMON_ALERT
    Wind: 8mph Gusts, 5mph 5min avg
    Primary useism: 0.12 μm/s
    Secondary useism: 0.39 μm/s
QUICK SUMMARY:

H1 has been locked and observing for almost 12 hours. All other systems look good.