Reports until 06:22, Monday 11 September 2023
H1 SQZ (OpsInfo)
oli.patane@LIGO.ORG - posted 06:22, Monday 11 September 2023 - last comment - 08:52, Monday 11 September 2023(72798)
Trying to get to Observing with SQZ ISS Issues

09/11 07:05UTC Vicky and Tony made changes to SQZ_MANAGER to take it to NO_SQUEEZING to allow us to be in Observing overnight without squeezing. (SDF Diffs1)

07:10 Vicki further made some edits (I already forgot what the difference was for these, but it was along the same vein)(SDF Diffs2)

07:15UTC While figuring out the workaround for the squeezer issues, we noticed that the ALIGN_IFO and INIT_ALIGN nodes were also keeping us out of Observing since they were reading as being NOT_OK according to the IFO guardian, even though they were in their nominal states. INIT_ALIGN for example is nominally in IDLE, and had been in IDLE since the previous lockloss at 05:53UTC (72797). Tony attempted to get IFO to show INIT_ALIGN as OK by requesting INIT_ALIGN to DOWN and then to IDLE, but this caused the detector to lose lock.

07:48UTC Running INITIAL_ALIGNMENT because the alignment was a mess (stuck going through ACQUIRE_PRMI 3 times) as well as hoping that it would clear the issues with INIT_ALIGN, but we lost lock at PREP_FOR_PRX

07:58 restored optics to settings from time 1378428918 (~3hrs previous, during lock)
- restarting INITIAL_ALIGNMENT again

08:06 INITIAL_ALIGNMENT couldn't find green arms so we took the detector out of INITIAL_ALIGNMENT and into GREEN_ARMS_MANUAL and found green by hand for both arms
08:14 green found, we requested it to DOWN and then into INITIAL_ALIGNMENT
08:39 Finished INITIAL_ALIGNMENT, requested NOMINAL_LOW_NOISE

09:20 Reached NOMINAL_LOW_NOISE
- ISC_LOCK is NOT_OK due to SQZ_MANAGER having been changed
- INIT_ALIGN is again listed as NOT_OK even though it is in its nominal state
- ALIGN_IFO listed as NOT_OK and is currently trying to go from SET_SUS_FOR_FULL_FPMI -> SET_SUS_FOR_FULL_FPMI (attachment3)
    - we later figured out that INIT_ALIGN and ALIGN_IFO are somehow connected to the squeezer and squeezer ISS and so the ISS issue somehow causes them to be NOT_OK

09:41 Trying to revert all changes that were made tonight to see if that will get rid of the NOT_OKs so we can get the detector into Observing. We were hoping that we would then be able to bypass the ISS failing
- Tony set SQZ_MANAGER's nominal to NO_SQUEEZING to see if that would help
    - it did not (nominal was changed back to FREQ_DEP_SQZ)

09:42 - 10:20 Various other methods tried and thought out (sdf3)
- ex) taking SQZ_MANAGER to DOWN and then back to FDS - didn't work

10:27UTC - Finally got into Observing! Did this by following alog 70050 to try to get around the squeezer ISS issue the same way Tony had been doing earlier that evening:
- Since the script settings were what they were supposed to be, Tony just used the alog as a reference for taking SQZ_MANAGER to NO_SQUEEZING, taking SQZ_OPO_LR to LOCKED_CLF_DUAL_NO_ISS, then to LOCKED_CLF_DUAL to help the ISS pump relock
- ISS pump relocked, ALIGN_IFO, INIT_ALIGN, and ISC_LOCK changed to OK, and we accepted sdf diffs(sdf4) and got into Observing

After that, the ISS has unlocked and locked back up multiple times and the currently accepted SDF diffs are for the ISS being ON. We know that this isn't ideal because of how often the ISS is losing lock and subsequently taking us out of Observing, but after VickY and Tony troubleshooted this issue for 6 hours, with me then working with Tony for a further 3.5 hours, we felt that there was no more that we would be able to do on a weekend night at 3:30am, and that although the ISS will keep losing lock and taking us out of Observing, SQZ_MANAGER will eventually get the ISS back up, putting us back into Observing, and this way we could at least get some amount of Observing time in the next few hours until more people can come in and fix the issue when the workday starts.

Things to note/tldr:
- SQZ_MANAGER's nominal state is back to being FREQ_DEP_SQZ, so that doesn't need to be changed
- SQZ_MANAGER needs to be taken out of IFO ignore list
- SQZ ISS needs to be fixed (obviously)

Thank you to Tony for staying for 3.5 hours past his shift end and Vicky staying up until the early hours to help troubleshoot!!

Images attached to this report
Comments related to this report
ryan.short@LIGO.ORG - 08:52, Monday 11 September 2023 (72803)OpsInfo

SQZ_MANAGER has been removed from the exclude_nodes list and the IFO top node was loaded at 15:37 UTC.