Displaying report 1-1 of 1.
Reports until 15:25, Friday 16 August 2019
H1 SEI (DetChar, ISC, OpsInfo, SEI)
jeffrey.kissel@LIGO.ORG - posted 15:25, Friday 16 August 2019 (51338)
In Retrospect It's Obvious! Looking at Existing BLRMS of BRS EX Would Have Revealed The Problem
J. Kissel, J. Warner

In the spirit "lessons learned" -- how could we have caught the BRS EX problem (FRS Ticket 13417) sooner -- Jim reminds us that there are pre-made BLRMS EPICs channels of each BRS in the standard ground motion frequency bands, e.g. 
     H1:ISI-GND_BRS_ETMX_RY_BLRMS_30M 
     H1:ISI-GND_BRS_ETMX_RY_BLRMS_30M_100M 
     H1:ISI-GND_BRS_ETMX_RY_BLRMS_100M_300M 
     H1:ISI-GND_BRS_ETMX_RY_BLRMS_300M_1 
     H1:ISI-GND_BRS_ETMX_RY_BLRMS_1_3 
     H1:ISI-GND_BRS_ETMX_RY_BLRMS_3_10 
     H1:ISI-GND_BRS_ETMX_RY_BLRMS_10_30

The first attachment is an "All of O3" trend of these channels, and one can "clearly!" (in the "in retrospect, it's obvious!" sense) that the BLRMS in all bands gets constantly elevated above regular ambient ground motion on Aug 15 2019. 
Second attachment is a time zoomin, and an amplitude zoom out, of the past week.

The BRS rarely has extended (> 1 hour) periods in which all bands are elevated 5 nrad_rms, and during this glitchy badness, the RMS soared well above 20 in all bands.
It's not going to catch everything, but for starters, we suggest a DIAG_MAIN alarm that looks at the collection BLRMS channels, and if they're collectively seen above 20 for 1 hour, then we certainly will know to look there immediately.

Also, we plan on adding these BLRMS channels to both the BRS screens and to create a monitor screen similar to the SUS actuator saturations screen, the Violin Mode Monitor screens, and the ISI stage performance matrices, where there's a color-coded report of the BLRMS level shown to report good and/or bad levels.

Hopefully this'll help us four years down the road when this happens again!
Images attached to this report
Displaying report 1-1 of 1.