J. Kissel, C. Vorvick, H. Radkins (remotely)

While struggling to damping 2nd harmonic violin modes having just transitioned to DC readout for the first time after yesterday's Montana earthquake, the X-END seismic system suddenly tripped all three stages of its watchdog (ISI ETMX ST1 ST2, and HEPI). Totally unrelated random other problem to solve.

The ISI and HEPI overview screen quickly revealed that the HEPI pump station pressure was NOT OK. A call in to Hugh helped us realize this was a re-occurrence of an over voltage error of that pump station's Variable Frequency Drive (VFD) system (see previous occurences -- seemingly once a year in the summer -- documented in LHO aLOGs 29113, 28653, 12337, 7536).

To fix, I followed the recipes described in the above mentioned aLOGs, repeated here for clarity and a refresher:
- DO NOT UNTRIP HEPI Watchdog! Untripping the ISI watchdogs are fine, but if you haven't changed the SEI manager state, as soon as you untrip all watchdogs, the manager will trying to bring everything back up, and you can't without pump pressure.
- On HEPI Pump Controller MEDM screen, switch pump controller's PID loop over to Manual, ramp down "Manual State Output Tweak" by selecting any large "Count Size Tweak" (say, 10s or 100s) and hitting the "-" until to output (H1:HPI-PUMP_EX_CONTROL_VOUT) reads zero (it was frozen at the max output of 2048).
- Drive to end station (bring CDS laptop!!), turn on wireless network once there, by plugging the dangling red cable into the bottom right most ethernet switch port.
- Head into pump controller room with big flathead screw driver. 
- Confirm that HEPI pump controller box shows a green heartbeat blinking light, and a red "status OK" light.
- Confirm that the translucent fluid in the pump fluid indicator is around 8.5 inches (there's a ruler built into the side of the indicator). Don't be fooled by the little red level indicator thing, those aren't the droids for which you're looking.
- Open up big gray VFD control box (Hugh calls it a "DD box" ...), but confirm on your way that the big 60's NASA light is on (it's RED when ON and OK)
- The VFD controller shows a red "OU3" on its top digital LED display.
- Hit RESET button in bottom, left corner, hit FWD in upper right corner of keypad. "OU3" should change to "0.00", and the green LED screen should read RUN. Nothing else will happen until you start increasing the State Output Tweak again.
- From you CDS laptop, back on the HEPI pump controller overview, set "Count Size Tweak" to 10s, and slowly (one click every 5-10 [s]) click "+" above "Manual State Output Tweak" to increase differential pressure readout (H1:HPI-PUMP_EX_DIFF_PRESS) to 70 [PSI] (a red trace will begin to appear on the long black window once you get it within range ~1-2 [PSI] of the nominal 70)
- Once you get above ~50 ish [PSI] you can click a little faster, we're just looking to prevent large turn-on transients in the servo which'll move the platform.
- Once you're close (within 10 [PSI]) you can flip the PID back over to manual.
- Once the pressure is with 1-2 [PSI] of nominal, untrip the HEPI and ISI watchdogs.

Pictures of all of these things can be found in the above referenced aLOGs.