Dan, Travis
Around 06:50 UTC we started to observe frequent glitching in the PRCL and SRCL loops that generated a lot of nonstationary noise in DARM between 20 and 100Hz. The glitches occur several times a minute, it's been two hours of more or less the same behvaior and counting. Our range has dropped by a couple of megaparsecs. The first plot has a spectrogram of DARM compared to SRCL that shows a burst of excess noise at 08:08:20 UTC.
The noise shows up in POP_A_RF45_I and POP_A_RF9_I, but not so much in the Q phases, see second plot. (MICH is POPA_RF45_Q, PRCL and SRCL are from the I-phases.) A quick look at the PRM and SRM coil outputs doesn't reveal a consistent DAC value at the time of the glitches, so maybe DAC glitching isn't a problem, see third plot. The three optical levers that we're using in the corner station (ITMX, ITMY, SR3, all in pitch) don't look any different now than they did before 0700 UTC.
I'm pretty sure these come from PRM, but one stage higher than you looked, M2. Attached is a lineup of PRM M2 UR and the glitches in CAL DELTAL. Looks like 2^16 = 65536 counts DAC glitches to me. I'll check some other times and channels, but wanted to report it now.
After a lot of followup by many detchar people (TJ, Laura, Duncan, and Josh from a plane over the Atlantic), we haven't really been able to make DAC glitches work as an explanation for these glitches. A number of channels (especially PRM M2 and M3) start crossing +/- 2^16 around 7 UTC, when the glitches begin. Some of these glitches line up with the crossings, but there are plenty of glitches that go unexplained, and plenty of crossings that don't correspond to a glitch. It's possible that DAC glitches are part but not all of the explanation. We'll be following this up further since one of these glitches corresponds to an outlier in the burst search. Duncan does report that no software saturations (channels hitting their software limit, as with the tidal drive yesterday) were found during this time, so we can rule those out.
Andy, Laura, TJ, Duncan, Josh: To add to what Andy said, here are a few more plots on the subject,
1) The glitching being bad does coincide with SRM and PRM channels being close to 2^16 (plotted only positive values here, negative values are similar). Of course this is pretty weak evidence as lots of things drift.
2) A histogram of the number of glitches versus the DAC value of the PRM M2 UR channel has a small line at 2^16. Almost not significant. Statistically, with the hveto algorithm, we find only a weak correlation with +/-2^16 crossings in the PRM M2 suspensions. Again, all very weak.
3) As you all reported, the glitches are really strong in SRCL and PRCL, ASC REFL channels, etc. hveto would veto ~60% of yesterday's glitches using SRCL and PRCL, as shown in this time-frequency plot. But the rest of the glitches would still get through and hurt the searches.
So we haven't found the right suspension, or there is something more complicated going on. Sorry we don't have a smoking gun - we'll keep looking.
