FRS9496
Jeff K, Richard, Dave:
At 07:30 PST Thursday 23rd November (Thanksgiving Day) PR3 rang up over two minutes. The T3 top OSEM shadow sensor RMS exceeded the 110mV threshold for 20 minutes continually, which caused the IOP Software Watchdog (SWWD) to DACKILL the h1sush2a DACs, at which time the oscillation stopped. The h1sush2a system remained in this state for the rest of the holiday weekend until the SWWD was reset this morning. Within an hour the ring-up occurred again. This cycle has repeated itself several times today. Investigation is continuing and an FRS has been opened.
Attached plot shows a 30 minute second trend plot of the Thursday morning event. Data shown is: T3's shadow sensor ADC channel (Ch 3), the h1suspr3 T3 DAC drive (Ch 9) and the h1susauxh2 M1 T3 Voltmon. As can be seen, the OSEM shadow sensor and the model drive ring up in Unisom, driving the coil driver. After the SWWD trips the DACs at the 20 minute mark, the coil driver is quickly zeroed, which causes a bumpy ring-down of the shadow sensor (which the model's output mimics).
Open FRS Ticket 9497.
PR3 Damping goes unstable because the Top Mass (M1) T3 binary IO switch for its analog low pass is stuck with the filter ON, implying the relay has became suddenly de-energized at 15:30 UTC (07:30 PST) on 2017-11-23. It has remained stuck there since. Logic thread that got me there: (1) The L, T, and Y damping loops (composed of LF, RT, and SD sensor/actuators) work just fine. Turn any one of the V, R, or P loops on, and they eventually (over the course of 1 to 2 minutes) ring up and cause the software watchdog to trip. (2) Had the binary IO screen open, just to show Siddhesh how they work. Noticed T3 was in the wrong state. (3) Toggling the state request (H1:SUS-PR3_BIO_M1_STATEREQ) from 1 (analog low pass OFF) to 2 (analog low pass ON), changes all the 6 OSEMs to be in the low-pass ON state. In this state, the digital compensation matches the analog configuration, so all is normal w.r.t. the damping loop plant. This is the configuration we ran in for the rest of the afternoon without problem. (4) Now, having a little time to think, I realize: when in state 1, the T3 OSEM's analog filtering is still stuck with its low pass ON, but the digital compensation compensates for the low pass OFF state. This changes the damping plant for *one* of the OSEMs, which is involved in all three, V, R, and P loops, and causes the loop to be unstable. Concluding theory as to what happened: The triple-top circuit diagram D0902747 (pg2, a zoom of the relevant portion of the circuit is attached for convenience) shows the energized (+5V to the relay, a binary, digital 1 sent through the BIO card) configuration, which is the low-pass OFF condition. When power is lost, the switch flipped to the de-energized (0V to the relay, a binary, digital 0 sent through the BIO card) configuration, which is the low-pass ON. This is what I think happened Thanksgiving morning: the relay (due to analog electronics failure) failed, switching the analog circuit to low-pass ON with the digital compensation still compensating for the low pass OFF condition, V, R, and P damping loops when unstable, rings up the suspension, eventually tripping the Software Watchdog. I'll update the FRS ticket and let the analog CDS team to put it on their "to-fix" list. However, because we can happily run in state 2, I don't suggest we take the time to fix it until we're done with IFO alignment, unless it can be fixed some morning quickly before the team gets started. For now, I've left PR3 in STATE 2, with the damping loops ON.
