This morning after a lock-loss, the ETMX ESD railed, so Fil and I drove to the end station to reset it. To do this, Fil first powered off the chassis by the chamber (3rd chassis from the bottom on the right rack, power is red button on the left), then unplugged the right most cable on the chassis. He then powered the chassis back on and plugged the ESD cable back in. This seemed to fix the ESD, until it tripped just a little while ago.
And here is a longer trend of locklosses showing that the railing does not occur at all lockloss events - it's inconsistent.
Betsy and Jeff tell me the trip was maybe unnecessary. If you click the HV on/off button a few times, on the bottom of the SUS ETMX overview, it may reset itself.
The ETMx ESD railing happened numerous times after Jim's reset at Ex this morning. To clear it, I hit the HV ON/OFF bitton at the bottom of the SUS screen from the controlroom: I hit the button once and verified the ESD signals went to 0 (actually a few hundred instead of it's usual thousands), then hit the button again and confirmed the signals came back to "normal" (roughly -32k on DC and not -16k on the 4 quadrant channels). Note, I was slow to push the buttons which may have helped it be sucessful.
GUARDIAN UPDATED: Sheila, Jeff, Jaimi and I then added to the guardian DOWN portion of the ISC_LOCK code to check for this railing and clear it before making any decisions on whether the state of the ESD should be on or off. This has yet to be tested as the IFO is still working it's way back up.
Attached is a trend showing the numerous railing events of the morning - some of which we think happened because the timing in the guardian code was not quite right.