J. Kissel, G. Merano, C. Vorvick, R. McCarthy, K. Kawabe, C. Compton, K. Ryan
The most recent lock loss (from Nominal Low Noise at 22:59 UTC) is very likely due to a failure in the RGA (or its controller) on HAM6, which had "temporarily" been used as the vacuum interlock signal for the high voltage ("temporarily" -- a month or more now...). Unfortunately, due to the "temporary" nature of the set up, we don't have a read-back on that RGA, so we can't confirm, but all indirect evidence during the aftermath points to it.
Richard, Kyle, and Gerardo have now switched the HAM6 HV interlock signal back to the "nominal" configuration, i.e. using the HAM6 PT110 ion gauge (H0:VAC-LX_Y0_PT110_MOD1_PRESS_TORR), which is also measuring the HAM6 pressure. We can confirm that there was no significant pressure increase in the vacuum system, i.e. it has remained around ~2e-7 Torr during this entire event (though we did see a small, ~0.05 Torr, sharp increase at the time of lock loss.)
This brought the HAM6 HV back, and we're now re-acquiring.
Details will be in the comments by me and others mentioned above.
Here's the breadcrumb trail we followed in order to figure out this problem: (1) Upon attempting to re-acquire, the ISC_LOCK guardian got stuck on PRMI ASC, throwing a notification that that the ISC_DRMI guardian was throwing a notification. (2) The ISC_DRMI notification was "WFS DC centering railed." So, Cheryl and I went in to the DC centering screen and tried to clear the OMs, which were indeed receiving very huge numbers. The reset buttons on that DC centering screen didn't work, and -- Jenne says later when she joins the diagnosis party -- it's "because the integrators have been moved to the suspensions, the DC centering screen won't look like it's sending out garbage signals any more." (3) After reseting, Cheryl noticed that the AS_A and AS_B signals were quite zero. (4) Simultaneously, we say the DIAG_MAIN was throwing a notification that the high voltage for the Fast Shutter was off. At this point, I "remembered" (i.e. searched "HAM6 high voltage" in the aLOG) the problem we'd back in October 26 (see LHO aLOG 52786): ""(3) Our next stumping point after initial alignment of the green ams was done, was the failure to repair the HAM6 PT110 pressure sensor today (LHO aLOG 52775). This sensor was serving as our high voltage interlock sensor, so since it didn't work suddenly near the end of maintenance, it tripped the high voltage off. Since the high voltage was off, the fast shutter was closed, which meant that the QPDs at the AS port (AS_A, AS_B, AS_C) were blocked, and the input alignment wouldn't work -- with the symptom being that the ASC DC centering loops were railing. It took Jenne putting two-and-two together to understand this connection path as to why the automated ALIGNING_INPUT step wasn't progressing. TJ has now added a check for the fast shutter being open during this step to the automated initial alignment guardian so that we don't get bit by this again."" Confident that "the" HAM6 vacuum pressure gauge (not yet knowing/remembering about the temporary setup situation) had tripped the high voltage in HAM6, which popped up the FAST SHUTTER, which blocked the light to the AS_A and AS_B QPDs, i.e. the WFS DC centering error signals, like had happened during initial alignment back in October, (5) Camilla and I went out the CER mezzanine to look at the high voltage power supply. They were indeed off. We tried the "level one" fix of just turning the supplies, which didn't do anything. (6) Keita then joined us, remembering more about the interlock, and used a nearby ladder to look *on top of the rack* and found the HV interlock system. (7) We took a few pictures, and then hit the red reset button which did nothing (i.e. did not change the lights on the beckhoff module that serves as the interlock, nor did it allow the power supplies to come back on) (8) We power cycled the interlock chassis, and hit the reset button again, and while it changed the status lights on the beckhoff module, it still didn't allow the power supplies to be turned on. (We found out later, that nominally, *all* the lights on the interlock module will be on if things are functioning correctly.) Time to call in the cavalry: Richard, Gerardo, and Kyle. Before Richard arrived, he had me (9) check the power supply to the RGA's ion pump controller to confirm all was well. Not knowing anything about this, I brought Gerardo with me -- but he didn't hear the conversation -- so he took me out to look at the PT110 ion pump controller, and we confirmed all was happy with *that* controller. Richard went out to the mezzanine and (10) Tested the interlock module functionality by removing the real pressure signal and replacing it with a consistently jumpered mimicked high pressure gauge input, and the interlock became functional, thus he concluded that something was wrong with the pressure signal coming from HAM6. At this point, Gerardo and Kyle had in parallel gone out to the HAM6 area to check on things, and Richard met them out there, concluding that (11) Indeed, the RGA's ion pump and/or its controller was indeed quite dead. Thus -- they took action as described in the main aLOG and reverted the pressure signal to the PT110, instead of the RGA's ion pump. In order to hopefully shorten the wild goose chase next time, I've added 4 more words to the ISC_DRMI warning about "WFS DC centering railed" -- it now says "WFS DC centering railed, consider checking HAM6 HV." Attached figures: (A) IMG_2295.jpg -- the Fast Shutter and HAM6 high voltage power supplies in the Mezzanine above the CER, shown as they were found by Camilla and I in step (5): The HAM6 SHUTTER power supply power switch was in the ON position, but no other lights were on, and the analog dials were reading zero. The digital HAM6 High Voltage power supply's power switch was in the OFF position, and nothing appeared alive. (B) IMG_2296.jpg -- the back of the High Voltage Interlock chassis located on top of the high voltage power supply rack ("C6"). Lights were green... (C) IMG_2297.jpg -- A view inside the interlock chassis, showing the yellow beckhoff interlock module. This picture is before we power cycled the chassis -- step (7) from above -- and only on the "Power" and "K2" indicators were lit green on the interlock module. (D) IMG_2298.jpg -- The interlock module after power cycling the chassis, (8). Now *only* the power light was green. (E) IMG_2299.jpg -- This is a picture of the PT110 ion gage controller that Gerardo and I checked in the mechanical room, (9)FRS Ticket #14013 filed.
In the attached, loss of the OMC PZT2 HV at t~0 eventually kicked OMC and IFO out of lock.
Good to see that the fast shutter worked at t~0.09 even when HV is lost, as the fast action of the FS comes from the energy storage capacitor charged at 250V.