The glitchy periods at 10 UTC today, and 8 UTC on July 20, seem to be due to overflows in all of the test mass L2 coils. The cause of the overflows looks to be CSOFT pitch. The attached plots show timeseries of CSOFT pitch and ITMX L2 master for one quadrant, then a spectrogram of the coil signal. It looks like most of the power is in the 2.8 Hz pitch mode of the test masses. From the summary pages, it looks like the RMS of CSOFT pitch has crept up over the last few months. When the coil drive goes over 131,072 counts, the DAC overflows and the coil glitches. It's possible that I have things backwards, and the glitch comes first and the overflow is a symptom, but that doesn't seem to match the behavior of the RMS at non-glitchy times. Is there a way to get CSOFT pitch lower, or maybe suppress some of the feedback at 2.8 Hz (which I think is the biggest contributor)?
FRS Ticket 8591 has been open on this subject.
Laura, Andy, TJ, Duncan
I looked in to the test mass L2 stage DAC overflows and found ITMY is the main culprit. At least the overflows of the ITMY L2 stage encompass the times when ETMX/Y and ITMX are also overflowing. I ran gwdetchar-overflow from the 16th July to today and the attached plots show an omicron glitchgram before and after the ITMY L2 overflows are removed.
I opened our common ASC template for CSOFT_P (in /ligo/svncommon/IscSVN/iscmodeling/trunk/ALIGOH1/ASC_loops/Measurements/CSOFT/), and see that Sheila measured CSOFT pitch on June 26th. It looks from that measurement like we have more than 10dB of gain peaking at 2.8Hz. Ooops. Although, if that's how we've been running since at least mid-June, I guess we haven't noticed it since our microseism has been very low, so we've had much less motion to suppress, so there has been less output from the loops, so we've been narrowly avoiding saturations?
The peak is not present in the soft loop spectra on June 15th, but it is there on the 17th. This coincides most closely with the ISS 3rd loop being re-engaged.