J. Kissel, B. Weaver

I've found that the R0 tracking for the ETM QUAD suspensions had been left on, had gone unstable on Thursday Oct 21 2021 ~15:23 UTC / 08:23 PDT and were driving the undamped suspension 20-30 [um] peak-to-peak on all DOFs for 10 days since just 60 minutes ago. 
This happened within 30 seconds after ETMY was "restored" from a site-wide collection of watchdog trips [LHO:60354], well after the Oct 19 EY Vent [LHO:60332], but many hours before the course of EY Pump work and Door removal [LHO:60353].

There was *no* danger, and *no* damage, to the suspension during this time, but it's just something to be mindful about to turn off when it's not needed (namely when the chamber is vented and the SUS it in air). 
Also -- this is zero people's fault. This R0 tracking infrastructure was installed at the tail end of O3, there was a pandemic in between the then and when we vented EY for the first time, institutional memory didn't even set in (let alone get lost), person power was/continues to be already spread incredibly thin with the HAM567 vent, the EY vent was supposed to be "only this or that" but dragged on and expanded in scope, we've had a great amount of person power turnover, and I was out working remotely on family leave. C'est la vie these days...

No, the watchdogs did not trip at all once the oscillation began. For one, the watchdogs are using a busted old calculation of a supposed "RMS" that is actually meaningless, and its value did not exceed ~1500 [ct whatevers "RMS"], and the threshold (as it has been for *years*) is set to 25000 [ct whatevers "RMS"]. An ECR to fix this calculation has been approved a long time ago, and will be installed sometime between now and before O4 starts (see LHO:38948, Item (b) in IIET:9392, and ECR:E1700387). For two, we have seen this watchdog trip on many other occasions, so it's not as though the watchdog is insensitive or not doing it's job as a hardware protection device. Thus, that the watchdog did *not* trip here is also indicative that there was no danger or damage to the suspensions. It merely *looked* bad, because the OSEM sensors have non-zero whitening gain at the frequency the loops were oscillating, and so 20-30 [um] peak-to-peak of physical motion is quite close to saturating the OSEM ADC. We were just giving the actuators and SUS a nice bit of exercise ;-).

Here's how I found it, for an educational walkthrough of the trouble-shooting process:
While taking a quick glance at ETMY to ensure it was ready for pumpdown, I noticed that the UIM "COIL DRIVER DEATH" red light was flashing constantly on the overview screen (a big red box that appears around the COILOUTF banks). 
 :: This is usually indicative of -- you guessed it -- that the given coil driver has shut off for some reason. However, the trigger for this is that the OSEM *sensor* values are identically zero (because the coil drivers power the sensor satellite amplifiers, it was an easy way to tell). The fact that the warning was *flashing* was fishy 'cause -- y'know -- for electronics, OFF is OFF). NOT COIL DRIVER DEATH. 
 :: Also, there was no drive coming out the UIM stage anyways, so having the coil driver ON or OFF wouldn't change that there's no drive coming from the UIM. NOT ERRANT DRIVE FROM UIM.
 :: So I glanced over at the UIM OSEM sensor values -- then saw those were oscillating to the point of saturation / semi-clipping. These were alive and functional, also confirming that the coil driver chassis was ON. But, the oscillations meant that the OSEM sensor values were dropping below 0.0 -- because of the afore mentioned non-negligible whitening. Naively, one would expect that the OSEM sensor signal, which nominally goes from 0 ADC counts to 2^15 ADC counts *at DC* shouldn't be going to zero. Alas. Frequency-dependent Whitening. THE SUSPENSION IS REALLY MOVING.
 :: Which led me to glance at the top mass, M0, damping loops which were off and the R0 damping loops were off and the M0/R0 OSEMs were also oscillating to the point of semi-saturation / clipping. This led me to think "Oh, it must be that the ISI had accidentally been left in FULLY_ISOLATED with the high-bandwidth Stage 2 loops on." Why there? Because the ISI stage 2 loops ON and fully isolated go unstable when the QUAD and/or TMS are left undamped for more than an hour or so. Nope --  the ISI is cruising along in "ISI_DAMPED_HEPI_OFFLINE." NOT ISI'S FAULT.
 :: So, I tried turning on some of the damping loops that were off. This did not help, and made things oscillate and saturate worse -- because OSEMs don't work well as damping loop error signals when they're saturating. Most loops don't work well when the error signal is going non-linear under saturations. DAMPING LOOPS DON'T WORK
 :: By this point, I pulled up an ndscope trace of some of the two top mass OSEMS. I just used F1, F2, and F3, for both, just to see *something* of a time-series of what was going on to commission whether the damping loops were making things worse (sometimes indicative of a sign error, or rubbing, or some other bad health condition of the SUS).
 :: After I turned *off* the damping loops, but then noticed that there was still large, oscillating drive coming out of the Reaction Chain's DAC, from which I traced the source to the be the R0 tracking, "L2 WIT DAMP," "ETMY_R0_L2DAMP_L" lognitudinal filter bank. So I turned the INPUT and the OUTPUT OFF of that filter bank, and the oscillations disappeared. 
 :: I then, with the R0 tracking OFF, slowly (as in, DOF by DOF) began to turn the damping loops back on, and saw both via the damping loop output control request as well as in the time-series traces on NDS scope that motion was beginning to decrease.
 :: I continued until all the DOFs on both chains were damped, and motion was calm.
 :: Once the *L2* PUM OSEMs were no longer saturating, then -- just as a test -- I turned the R0 tracking back ON. And -- although the requested drive was large at the start, it also began to quiet within 30-60 seconds.

I've now left the R0 tracking ON, because we're pumping down and we don't anticipate much more activity that would cause with the PUM, UIM, M0, or R0 OSEMs to saturate.