In an ideal world we would have been able to systematically investigated each stage of the SR3 chain to find where the glitch was coming from. But with the pressure of needing to get the system up and running I decided to replace all electronics and do a followup in the lab. So this morning we took down the SR3 Front end and installed a new DAC card. Replace the coil driver SN100186 with SN 100192. The AI chassis had been replaced last night. So aside from cabling replacement the SR3 drive chain is new and after 10 minutes we do not see the glitches. Of course I could go 10 min this morning and not see them. Also there was a problem with the Voltage Monitor for T2. This was in the IO chassis ADC interface so we replaced the ADC the ribbon cable and ADC interface card for the SUS Aux chassis. This fixed problem.
This screenshot show the OPlev and witness sensor for SR3 pit. The problem shows up clearly early monday morning, things got slightly better around 2 am last night for unknown reasons. This morning after the electronics were swapped, the frames are bad, and now the problem seems to be fixed.
The second screenshot is a lockloss from last night, in which you can see that before the lockloss, SR3 gets something like 1 urad of pitch in about half a second. This is visible in both the witness sensor and the oplev although the calibrations don't agree. We also had smaller glitches that weren't large enough to lock us out of lock, but we think that the large ones were happening about once every 20 minutes or more frequently.
Here's a different view of the same thing that Sheila just posted.
I have 3 1-hour plots of SR3 M3 Wit Pit, all with the same y-axis scale: (1) Aug 26th at some time when the Cage servo was running, (2) sometime last night, and (3) right now. The "right now" plot shows the lock aquisition sequence, including the cage servo being engaged, which is why the DC value changes.
The salient point is that last week, the M3 Wit Pit looked nice and clean. Last night, even though there weren't huge glitches in the time that I chose, the optic was clearly moving around much more than normal. Now, the data looks clean again.