Sheila, Jenne
We had a large EQ at just about 19:00 UTC, a couple of ISIs tripped but no suspensions so far. I set the seismic configuration to LARGE_EQ_NO_BRSXY after some BSC ISIs had already tripped, while I was doing that some more BSC ISIs tripped as well as BS HEPI, but I don't know if the change in state caused the trips or the earthquake did it. I reset all of the ISI watchdogs, they were various triggers for the trips including GS13s, ST2 CPS, T240s and ST1 actuators. A few minutes later (perhaps at 19:05) ITMX and ITMY tripped again. I have just reset them now at about 19:09, however they are not re-isolating because the guardians are waiting for the T240s to settle.
This should be some interesting data to see how the changes in the ISI models have changed the way things respond.
Our seismic FOM is not updating even when I hit update, and Terramon is down, but the USGS says there is a 7.6 in Honduras.
At around 19:35 ITMX and ITMY tripped to damping only again, both because of the T240s and ST2 CPSs. I am going to leave things this way and head home.
Because it wasn't mentioned here, I want to bump my alog 38921 , where I detail the VERY_LARGE_EQ button on the SEI_CONF screen. This is probably the kind of earthquake we should use that button for. Unfortunately, it probably wouldn't have worked this time, as we had changed the names of the DAMPED state for the chambers. I've updated that now to take all of the ISI's to ISI_DAMPED_HEPI_OFFLINE.
To reiterate what the button does:
1. Turns off all the sensor correction via the SEI_CONF guardian.
2. Switches all the ISI chamber guardians to ISI_DAMPED_HEPI_OFFLINE
3. Switches all the GS13s to low gain (except the BS & HAM6), and all BSC ST1 L4Cs to low gain as well
4. Puts HAM1 HEPI in READY
Also, Jenne reported that the local SEISMON code gave a verbal alarm about 3 minutes before the S-waves arrived and ISIs started tripping. If the code is alive, I'm not surprised it reported the earthquake before USGS or the Terramon webpages.
I took a look at the BS HEPI trip. This trip is very clearly caused by saturations in the vertical actuators. More work is needed to figure out what to do, but I put together a set of plots which show why I think the actuators signals generate the trips.
I'll note that saturated hydraulic actuators should be treated seriously, and are a good reason to turn things off.
But, hopefully we can keep this from happening in a smarter way than just turning everything off.