TITLE: 01/15 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 152Mpc
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY:

IFO is in NLN and OBSERVING as of 05:40 UTC (20 min lock!)

Overall, bad recovery from maintenance. Here’s what happened in 7 short stories:

1. Late LN2 Fill: NORCO arrived to fill CP8 at 00:34 UTC (16:34 PT) over 3 hours late while we were locking. Jim made EX as robust as possible such that we would survive their truck moving through. This worked though the BRS was all over the place as expected. Due to them driving <5mph as requested, they took 20 minutes to arrive at EX, then over 1 hr to do the fill and then another 20 minutes to get back. They left site at 02:34 UTC (18:34 PT). After NORCO left, I undid Jim’s seismic configuration changes to go back to nominal.
2. Changes made prevented locking: Earlier in the day, (before NORCO arrived), we lost lock at state 557/8, TRANSITION_FROM_ETMX. I was told by the DAY operator that if this happened, I was to revert Camilla’s changes in alog 82277 with regard to the DARM Boost. Since Camilla and Sheila were in the room when this happened, they decided not to do it since it could have just been bad luck. Erik had also made changes regarding ramp times that affect this state. Camilla and Sheila left for the day and then NORCO arrived. While NORCO was filling, we lost lock again at state 557, TRANSITION_FROM_ETMX. Camilla told me to revert her changes and Jenne told Erik to revert his. This was done by 1:31 UTC (17:31 PT).
3. Changes reverted prevented locking: We could not lock ALS until Norco left the EX area, which happened ALS locked finally at 02:10 UTC (while Norco was en route from EX). Locking then happened automatically until state 500, PREP_DC_READOUT_TRANSITION, where for 10 minutes, I was getting the error message “OMC not ready”. I then called Camilla, who didn’t answer. I then called Erik, who was still on site and came to help troubleshoot the issue (since his changes involved resetting OMs - the idea was that maybe this is related, which it is likely not). After some fiddling to no avail, I called Sheila (the first person on the call list), who walked me through the OMC troubleshooting process.
4. Troubleshooting with Sheila and Erik: I informed Sheila that the error was that the OMC TF was not letting guardian continue. We did a manual OMC scan while waiting at ASC_QPD_ON to see that there indeed were triplets, with 2 sidebands and one carrier with the right magnitude. We let guardian lock it and it did, yielding the expected signal magnitude for the DCPD_SUM and PZT2 Monitors. Looking closely at the error, Erik realized that it was the phase that was below the threshold (150) and was measuring at 144. Sheila, Erik and I then decided to lower the phase threshold to 140. This worked immediately until we also lost lock just as immediately when we transitioned to DARM_TO_DC_READOUT, which saturated DCPD, EX, EY, IX and HAM6. This happened at 04:13 UTC (20:42 PT). At 4:49 UTC, I reverted the phase changes to OMC_LOCK guardian, since clearly something else was awry.
5. “Did you try turning it off and back on?”: After the lockloss and the reversion, we went up to lock again, preparing to face the same issue, DTT template and TF in hand and then… it worked. Perhaps the lockloss from earlier was from transitioning the OMC with whitening to the OMC without whitening (which I did after achieving the full OMC lock during troubleshooting). Now our next hurdle is state 557/558 (TRANSITION_FROM_ETMX).
6. The DARM Boost is actually still on: It turns out that the DARM boost had to be turned off in two places, rather than just line 3058 (alog 82277), there was another later boost turn-on at line 3774. Sheila discovered this during troubleshooting at 5:15 UTC. Since we didn’t want to tempt H1 by turning off abruptly, we chose to leave it on and see what would happen at 557/558. After some anticipatory sighs, it worked! We automatically went all the way to NLN.
7. 05:40 UTC - reached NLN! We had one node that wasn’t right, which was the FSS being in INIT and not knowing how to go to IDLE. It wasn’t on the SDF page (no other SDF diffs). Sheila manualled into IDLE, which worked. Strangely enough, we also got a “check PSL chiller” alert but since I know that this alert has some weird timing feature with respect to its thresholds, I will just tag PSL.

Big thanks to Erik and Sheila who helped a lot with troubleshooting.

LOG:

None