Reports until 16:55, Monday 08 May 2023
H1 CAL
jameson.rollins@LIGO.ORG - posted 16:55, Monday 08 May 2023 (69421)
offline vs. online issues with H1:GDS-CALIB_STRAIN data over the weekend

At some point either last friday or over the weekend we started to see some very strange behavior in the H1:GDS-CALIB_STRAIN_NOLINES channel in the low-latency frames.  It manifested as what appeared to be a large DC offset in the channel, and polution of the spectrum at higher frequencies.  The problem was not seen in the data on disk, and only in the data in the low-latency frames.  Due to a quirk in the DMT systems, the low-latency hoft frames and the to-disk hoft frames are created by different DMT machines, running their own gstlal-calibration processes. It appears that the calibration process on h1dmt1 which writes out the low-latency hoft frame somehow got into a bad state.  The issue was resolved by restarting the gstlal-calibration processes on the two DMT machiens (h1dmt1 and h1dmt2).  It's unclear why this happened, the calibration group is looking into it.