Reports until 15:58, Monday 30 November 2015
H1 CDS (CAL)
david.barker@LIGO.ORG - posted 15:58, Monday 30 November 2015 - last comment - 20:44, Tuesday 01 December 2015(23837)
psinject restarts of CW hardware injection update

Last Tuesday (24th Nov) Jim and I modified the monit on h1hwinj1 machine such that when it restarts the psinject process it smoothly ramps the excitation amplitude over a time period of 10 seconds. We manually started the new system on Tuesday and since then there have been no crashes of psinject until the last 24 hours. There have been 4 stops (with subsequent automatic restarts) in the past 24 hours, each stop was logged as being due to the error:

SIStrAppend() error adding data to stream: Block time is already past

Here are the start and crash times (all times PST). Monit automatic restarts are maked with an asterix

time of start time of crash
Tue 11/24 14:55:47 Sun 11/29 17:15:56
Sun 11/29 17:16:00* Mon 11/30 00:00:14
Mon 11/30 00:01:13* Mon 11/30 13:09:07
Mon 11/30 13:09:36* Mon 11/30 13:12:43
Mon 11:30 13:13:39* still running
Comments related to this report
jeffrey.kissel@LIGO.ORG - 16:42, Monday 30 November 2015 (23841)INJ
Link 

Adding INJ tag so the hardware injection team sees this.
keith.riles@LIGO.ORG - 20:44, Tuesday 01 December 2015 (23887)INJ
Link 

Using a script written to flag CW injection transitions
(start or stop) from psinject log files on h1hwinj, I found 
all of the transitions below to occur during observing mode. 
I'll ask Laura Nuttall to insert these in the database.

1132905630 1132905633 Nov 30 2015 08:00:13 UTC
1132905689 1132905692 Nov 30 2015 08:01:12 UTC
1132952963 1132952966 Nov 30 2015 21:09:06 UTC
1132952992 1132952995 Nov 30 2015 21:09:35 UTC
1132953179 1132953182 Nov 30 2015 21:12:42 UTC
1132953235 1132953238 Nov 30 2015 21:13:38 UTC

Peter Shawhan reported via e-mail that these restarts
happened sporadically in iLIGO days and may be related
to a busy network. Presumably, these restarts were not
disruptive in those days because they were not injected
via a time-domain inverse actuation filter. 

Looking at the first of the crashes above (see attached
time-domain zooms of the CW injection channel,
the IAF-filtered version of it and DARM), I'm surprised
that the effect in DARM isn't larger.
Images attached to this comment