corey.gray@LIGO.ORG - posted 17:24, Tuesday 02 January 2024 - last comment - 17:31, Tuesday 02 January 2024(75136)
Continuous Wave (CW) INJ Gain Changes Dropping H1 From Observing (starting at 0037utc)
At 0037utc, H1 was bumped out of OBSERVING due to an SDF change. The change has been this channel:
- H1:CAL-INJ_CW_GAIN
- It has been dropping from its nominal value of 1.0 and going to 0.0. This happens about every 1min and it stays at 0.0 for a few seconds.
- Every time this channel goes to 0.0 it is an SDF diff which does not allow us to go to OBSERVING.
I chatted with Louis (phoned him because it is a CAL channel), but he mentions this is not an "online calibration issue", and this has to do with the Continuous Wave Injection. It also does not appear to be guardian-related. After chatting with Louis, since Jonathan and Erik are on-site, I just notified them about the issue.
We have currently been OUT of OBSERVING for about 40+min.
Attached is another screenshot with:
- an ndscope of the channel,
- H1CALING_INJ_CW.adl filter bank during one of the gain changes
- CALINJ sdf page
Currently, Dave, Erik, & Jonathan are investigating.
Images attached to this report
Comments related to this report
0128: Back to OBSERVING! (thanks to Dave, Erik, Jonathan, and also Louis for my first chat with him).
Dave, Erik, Jonathan, This was caused by an expired leap second file on the h1hwinj1 machine. It looked like the CW injection process was restarting. After investigation we noticed that the psinject service was failing, with an error loading the leap second database file. The solution was to update the tzdata package. On SL7 this is "yum update tzdata". After we did that the injection process was able to startup and run.