TJ, Dave:
Follow up on the loss of the CW injection last Wednesday morning which caused H1 to drop out of observe ( alog-54917)
At 07:18:10 Wednesday 05 Feb 2020 PDT the CW injection process on h1hwinj1 stopped running, which caused one SDF difference for the h1calinj model which took H1 out of observe mode.
The CW injection is controlled by a psinject process running on the h1hwin1 machine. It is under systemd control, if the process stops systemd will restart it 300 seconds (5 mins) later. The orderly shutdown and startup scripts ramp the output of the CW filter module over 10 seconds to prevent IFO jarring.
Looking at the psinject.services systemd logs for the time in question, rather than the process spontaneously dying and being restarted, the logs say the process was shut down in a orderly way by running the controlled_psinject_stop script. The logs then show the orderly startup script being ran by systemd at 07:23.
This is the third occurrance of a spontaneous psinject restart during O3, I have reopened the FRS which covers these (https://services.ligo-la.caltech.edu/FRS/show_bug.cgi?id=13294) and added this event to the ticket.
Jonathan, Erik, Dave:
After further digging we found that these restarts do have associated failure conditions. The log files can be found on h1hwinj1 in the /home/hinj/Details/log/ directory.
05feb2020: Wed Feb 5 07:18:05 2020: SIStrAppend() error adding data to stream: Block time is already past
21jul2019: Sun Jul 21 06:06:31 2019: SIStrAppend() error adding data to stream: Block time is already past
18jul2019: Thu Jul 18 00:20:04 2019: Error while closing SIStream: Error clearing test point AND freeing awg slot
Tagging CAL group for follow up on these errors.
Trending last Wednesday's CW injection signal and CW_GAIN shows that the cw process had actually stopped sending an excitation signal about 5 seconds before systemd ran the controlled_cw_stop script (which ramped the CW_GAIN from 1.0 to 0.0 over 10.0 seconds). Trend is shown in attachment.