Andy, Duncan, Laura, Ryan, Josh,
In alog 28299 Andy reported that we were seeing the ER9 range deteriorate due to glitches every 2 seconds. Figure 1 shows the glitches turning on in DARM at 2016-07-09 05:49:34 UTC.
We think the ALS system not being shuttered and changing state in lock is to blame. Here's why.
Excavator pointed us to a strong coupling between DARM and the ALS channels. Raw data confirmed a correlation, Figure 2 shows the ALS glitches tuning on at that same time and figure 3 shows that both DARM and ALS are glitching at the same times.
When we investigated the Guardian ALS state for this time (figure 4), it was not in a nominal configuration to start with and that got worse around the time the glitches started. The shutter was not "Active" and at 05:49:34 UTC the ALS X state changed from "-15 locked transition" to "-19 locking WFS" and at that same time the glitches started in DARM. So at some point, ALS X decided it needed to lock the arm (looks like Y followed an hour or so later). We did not track down exactly how the glitches originated or made it into DARM because this seems non-standard enough that a configuration fix should make it go away.
Figure 5 shows a summary page plot for nominal ALS X Guardian behavior from O1. So the shutter should be active and we don't expect to see "locking WFS" come on during an analysis ready state.
It seems like the ALS didn't think that the IFO was locked on IR anymore. The ALS-X state suddenly drops from 'Locked on IR' to 'PLL locked' (state 6 to state 2), then the requested state changes from 'Locked on IR' to 'Lock Arm' (state 6 to state 3). It seems like something went wrong in the communication and the ALS started to try to lock the arm. I don't think it would have helped if it were shuttered, because it would have unshuttered when trying to 'relock'. The attachment is just a plot of the two EPICS channels. As Josh said, the change corresponds to the time the glitching started.
Two additional notes: Here are the full Excavator results for the time period: https://ldas-jobs.ligo-wa.caltech.edu/~rfisher/Excavator_Results/Jul11_Tests/1152079217/ (Note: Excavator was run over unsafe channels as we were running a test of the code and then we started to follow up why something in ALS ODC popped up.) We were pointed to the source of the problems by the ALS-X ODC channel indicating ADC overflows on a 2 second interval with precise timing. The ADC overflows reported by the EPICs system at this time had timing fluctuations relative to the actual overflows of +/- .6 seconds in just the 5 minutes we looked at by hand.
We have been leaving the green light into the arms because sometimes it is usefull to see the power build ups and green WFS signals as we are trying to understand alignment problems. As people pointed out we would normally have these shuttered if things were really nominal, and in the shuttered state we don't check if the green arm is still ocked or not so it would not try to relock causing the glitches.