Reports until 16:35, Thursday 05 December 2019
H1 CDS
jeffrey.kissel@LIGO.ORG - posted 16:35, Thursday 05 December 2019 - last comment - 15:52, Friday 06 December 2019(53717)
SUS ETMX L2 LOCK OUTSW Y Snafu
J. Driggers, J. Kissel

We don't have the fortitude to investigate further in this scary snafu, but we want to note it for future reference in case it happens again:
Once we arrive at nominal low noise, the SDF system informed us that ETMX's global yaw control output for the L2 stage only (H1:SUS-ETMX_L2_LOCK_OUTSW_Y) was OFF. This meant that there was no global yaw control (DOFs DHARD, DSOFT, CHARD, CSOFT) getting out to the PUM. 

We noticed it fast enough to 
    - set the ramp time to 4.0 seconds
    - ramp down the gain of the L2 LOCK Y bank from 1.0 to 0.0, 
    - turn on the output switch, as is nominal, and
    - ramp the L2 LOCK Y bank gain back to 1.0
in rapid succession such that we saw no noticeable effect on the IFO, and we were able to enter in to OBSERVATION READY unscathed.

I attach a trend of this channel, and it's similar channel for pitch over the course of O3B, since November. This switch has only been switched OFF 3 times (including today), and *other* than today, it happened while the IFO was down: 
    ISC_LOCK N State 11
    2019-11-12 18:35:25 UTC OFF Ghost
               18:38:37 UTC ON

    ISC_LOCK N State 12
    2019-11-26 20:35:57 UTC OFF Ghost 
               20:39:09 UTC ON

    ISC_LOCK N State 600
    2019-12-05 23:19:58 UTC OFF 
               23:51:27 UTC Jenne and Jeff turn it back ON

However, as one can see, these three times happen in three different ISC_LOCK guardian states, (11: READY, 12: LOCKING_ARMS_GREEN, and now in 600: NOMINAL_LOW_NOISE), and it's changing far LESS times that the ISC_LOCK guardian goes through the acquisition, implying that this switching has nothing to do with a rogue guardian.

We've also ruled out the ISC_LOCK guardian entirely, by perusing it for L2_LOCK_OUTSW toggles, and it's only in DOWN and DHARD_WFS that this switch is touched, and in both places, it's forced to 1.0 i.e. ON (maybe indicative of ghosts having spooked us in the past). 

  
Images attached to this report
Comments related to this report
sheila.dwyer@LIGO.ORG - 15:52, Friday 06 December 2019 (53729)

Jim, Camilla and I had a problem a few weeks ago where because the output buttons are the only clickable things on the OM overview, a stray mouse click can easily turn them off and it is not super obvious (late at night) that they are wrong on the screen.  One proposal to prevent this is to remove the buttons from the OM overview screen, although I can't remember if anyone is planning to do this.  
 

I seem to remember that for the quads these outsw buttons are one of the only things clickable from the overview screen as well.  Perhaps these ghosts are just stray mouse clicks and we can eliminate them with medm science.