J. Driggers, J. Kissel
We don't have the fortitude to investigate further in this scary snafu, but we want to note it for future reference in case it happens again:
Once we arrive at nominal low noise, the SDF system informed us that ETMX's global yaw control output for the L2 stage only (H1:SUS-ETMX_L2_LOCK_OUTSW_Y) was OFF. This meant that there was no global yaw control (DOFs DHARD, DSOFT, CHARD, CSOFT) getting out to the PUM.
We noticed it fast enough to
- set the ramp time to 4.0 seconds
- ramp down the gain of the L2 LOCK Y bank from 1.0 to 0.0,
- turn on the output switch, as is nominal, and
- ramp the L2 LOCK Y bank gain back to 1.0
in rapid succession such that we saw no noticeable effect on the IFO, and we were able to enter in to OBSERVATION READY unscathed.
I attach a trend of this channel, and it's similar channel for pitch over the course of O3B, since November. This switch has only been switched OFF 3 times (including today), and *other* than today, it happened while the IFO was down:
ISC_LOCK N State 11
2019-11-12 18:35:25 UTC OFF Ghost
18:38:37 UTC ON
ISC_LOCK N State 12
2019-11-26 20:35:57 UTC OFF Ghost
20:39:09 UTC ON
ISC_LOCK N State 600
2019-12-05 23:19:58 UTC OFF
23:51:27 UTC Jenne and Jeff turn it back ON
However, as one can see, these three times happen in three different ISC_LOCK guardian states, (11: READY, 12: LOCKING_ARMS_GREEN, and now in 600: NOMINAL_LOW_NOISE), and it's changing far LESS times that the ISC_LOCK guardian goes through the acquisition, implying that this switching has nothing to do with a rogue guardian.
We've also ruled out the ISC_LOCK guardian entirely, by perusing it for L2_LOCK_OUTSW toggles, and it's only in DOWN and DHARD_WFS that this switch is touched, and in both places, it's forced to 1.0 i.e. ON (maybe indicative of ghosts having spooked us in the past).
Jim, Camilla and I had a problem a few weeks ago where because the output buttons are the only clickable things on the OM overview, a stray mouse click can easily turn them off and it is not super obvious (late at night) that they are wrong on the screen. One proposal to prevent this is to remove the buttons from the OM overview screen, although I can't remember if anyone is planning to do this.
I seem to remember that for the quads these outsw buttons are one of the only things clickable from the overview screen as well. Perhaps these ghosts are just stray mouse clicks and we can eliminate them with medm science.