It tripped when we reached the RESONANCE state in ISC_LOCK. This trip happened later in the reacquisition from what we have seen in the past.
I also found SEI_DIFF stalled in the DOWN state, I'm a bit confused how it got there, but I'll look into it later.
Because SEI_CONF is now managed by SEI_ENV, I don't know if you can change SEI_CONF manually like we did in the past. Probably the proper way to force the earthquake controls on is to move SEI_ENV (the manager node) to EARTHQUAKE, not SEI_CONF (the subordinate node). This is something that hasn't been communicated clearly since we started the earthquake automation. This maybe also affects the function of the big red button, I haven't checked, but Patrick's earlier alog made it sound like we didn't trip too many platforms.
Looking at the trends for the second earthquake last night, there is some strange behavior from SEI_ENV. See attached image, SEI_ENV is top left, peakmon is top right, SEI_CONF is bottom left. Before the earthquake, the guardian is rapidly switching between the SEISMON_ALERT state and the CALM state. Peakmon crosses the 400 nm/s threshold to transition SEI_CONF to EARTHQUAKE from SEISMON_ALERT, but SEI_ENV doesn't do that in a normal way. Eventually, SEI_ENV tries to go to EARTHQUAKE, but is again switching rapidly between EARTHQUAKE and CALM, but well after peakmon crosses 1000 nm/s, which is the threshold that we use for the "ground only" test in SEI_ENV. Even then, SEI_CONF never goes to EARTHQUAKE in an normal way. Not sure what is going on, this all seems very wrong.
The SEI_ENV log is filled with a bunch of loops of this:
2020-03-09_06:40:00.638948Z SEI_ENV [CALM.enter]
2020-03-09_06:40:00.703947Z SEI_ENV [CALM.run] Peakmon high, no seismon alert, jumping to EARTHQUAKE
2020-03-09_06:40:00.760043Z SEI_ENV JUMP target: EARTHQUAKE
2020-03-09_06:40:00.760702Z SEI_ENV [CALM.exit]
2020-03-09_06:40:00.817673Z SEI_ENV JUMP: CALM->EARTHQUAKE
2020-03-09_06:40:00.818388Z SEI_ENV calculating path: EARTHQUAKE->CALM
2020-03-09_06:40:00.818388Z SEI_ENV new target: CALM
2020-03-09_06:40:00.818388Z SEI_ENV GOTO REDIRECT
2020-03-09_06:40:00.819659Z SEI_ENV REDIRECT requested, timeout in 1.000 seconds
2020-03-09_06:40:00.820828Z SEI_ENV REDIRECT caught
2020-03-09_06:40:00.821624Z SEI_ENV [EARTHQUAKE.redirect]
2020-03-09_06:40:00.883430Z SEI_ENV EDGE: EARTHQUAKE->CALM
2020-03-09_06:40:00.884153Z SEI_ENV calculating path: CALM->CALM
2020-03-09_06:40:00.888961Z SEI_ENV executing state: CALM (10)
2020-03-09_06:40:00.889582Z SEI_ENV [CALM.enter]
I can't find where it must have been doing the same with the SEISMON_ALERT state, must be too many pages back in the log.
None of this should affect the cps diff, but the ISI trips may have been related. The SEI_DIFF guardian turns off the cps diff if any ISIs trip or if the chamber guardian isn't nominal, maybe that caused a problem and crashed the node?