Jenne, Dan, control room people
Attached are some additional plots of the glitches that Sheila posted about earlier today. These are the same 'DRMI' glitches which we observed back in ER7 -- or, at least, they are similar enough that I can't tell the difference.
In our most recent lock stretches, started around 0400 UTC, the glitches have gone away. The mystery continues.
The excess noise is loudest in POP_A_RF9_I_ERR, which is what we use for PRCL in low noise. We also see noise in POPA_45_I_ERR (used for SRCL), and not so much in the Q-phases. See the first plot. So, they're not an issue with the PD or the demod electronics. It seems like real length noise, except the noise is very flat. It's hard to imagine a glitchy actuator making flat noise in the PRC and SRC lengths.
FWIW the noise is also visible in POPAIR, with the same pattern as POPA. We also see the excess noise in REFLA_45_I and Q (second plot).
Since these glitches appeared in ER7 we have implemented the offloading of the PRM and SRM M3 stages, so the drives to the last stage are mostly around zero. We looked for any correlation with zero-crossings in the SRM and PRM M3 coils (third and fourth plots). No smoking gun.
Jenne, Dan
Maybe the PRM M3 coil driver is bad? Or something else in the analog electronics...
The gliches in the DRMI signals are correlated with bursts of noise in the PRM M3 NOISEMONs. Compare the first plot (a glitching time) with the second plot (a quiet time).
This isn't surprising - if the PRM drive is compensating for a burst of noise in the PRCL error signal, we'd expect to see something in the coils. But the NOISEMON readbacks for PRM M3 are different for each coil, and LL is the worst offender at the glitchy times. See the third plot. This plot includes a comparison with the SRM M3 NOISEMON, which also shows a difference between glitchy and quiet times, but the coil NOISEMONs agree.
The variation between the coils is certainly due to something in the analog electronics. In the fourth plot we show the PRM M3 MASTER_OUT signals. The drive to the optic from the digital side is the same for all the coils.
It could be that the PRM M3 NOISEMONs are just wrong, but the correlation with the glitches is suspicious.
Not sure if this is enough to justify swapping the coil driver.
The fifth plot is a time-series of the PRM NOISEMONs at the time of a glitch. The y-axis has been scaled the same for all channels to highlight the large value of the LL readback. Note that the LR channel has two dropouts, this happens frequently (there are LR dropouts in both of the first two plots) but is not correlated to the DRMI glitches.
Agreed that the PRM M3 LL driver looks like it's broken. The excess noise in all of the ISC channels is a shelf up to about 70 Hz, and that's what is being sent out to the PRM drive. But in just the LL noisemon, the noise bursts actually go up past 1000 Hz. In UR and UL, the noisemons look just like the drive signal. The attached PDF lines up the two spectrograms to show this. Unless the noisemon is very nonlinear, LL is generating excess noise. Is there a transfer function somewhere of PRM M3 to different ISC degrees of freedom that would tell us whether this could be causing all the noise we see? The second page shows that the noise from LL is probably not major-carry glitches. They don't line up with zero crossings in the drive. The third page shows that the LR noisemon is broken. It looks like it's got ADC overflows or some other kind of saturation. I'll try to check it for ADC overflows.
For completeness, I checked if the glitches show up in the ASC signals that are sensitive to the PRC alignment. They do: the plot attached shows that during glitch times (dashed references) the REFL WFS have excess noise in the same band as the LSC sensors.
So, it's not pure length motion -- consistent with the picture of a single glitchy quadrant on PRM M3.
Dan and Jenne were wondering if this happened in ER7. I took a time when we saw similar mysterious DRMI glitches (June 7 19 UTC) and made some spectra and spectrograms. It looks like the same thing happening, but less severely. The noisemon in UR has the same shape as the drive signal, but LL clearly has a different shape and more excess noise at high frequency. Maybe some other Detcharians can look into this in more detail.
