TITLE: 06/26 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
SHIFT SUMMARY:

• 7:40 lock - lockloss from a power glitch, Currently relocking
• DetChar safety injections were done

Lock#1:

Out of Observing at 16:20 for some DETCHAR safety injections, back into Observing at 16:30

The PCALX OFS saturated during the injection at 16:26 and was toggled off and on at 16:29

Out of Observing at 17:16 where we went to NLN_CAL_MEAS to rerun the DetChar safety injections LHO:70817, back into Observing at 17:26

19:33 Hanford Site emergency monthly test call

20:40 Within a minute, we had a glitch with corresponding EX saturation, the lights flickered in the control room and boths arm HEPI and both stage ISI watchdogs all tripped and then we lost lock. There was a small fast storm that was passing over the site at the time. I tried to reset the watchdogs, it worked fine for the ISIs but the HEPIs retripped after a few minutes. I put everything into SAFE for Team CDS to try and restart the models, to no avail, after calling Jim we sent Fil and Erik to the Ends to hit the physical reset button for the VFD panels after seeing that the HEPIs were reporting 0 pressure (H1:HPI-PUMP_EX_PRESS_DIFF_PSI) instead of the nominal 70.

Once the button was pressed the voltage was recovered manually from the medm X/YArm->HEPI->Beckoff Pump Controller

I opened an ndscope to monitor the pressure H1:HPI-PUMP_EX_PRESS_DIFF_PSI and set the right most controller to manual and set the step size to 0.1V and stepped fairly quickly up to 1V then slowly up to about 2 watching the pressure the whole time. Once the pressure reached about 68 I set the controller back to auto and the automation took over and overshot a bit but recovered and we got back to a steady 70.00 +/- .03 [V].

Lock#2:

Yarm wasn't terrible but there was no light on Xarm, I started off by trending the OPLEVs for the ETMX and tried to return to the same positions which yielded some light but it wasn't great. After some tapping around I decided just to do an initial alignment and to recover Xarm I used the restore script and set it back to when green arms locked during the last acquisition and it immediately looked better. Increase flashes then came on and brought it back.

LOG: