Reports until 19:29, Tuesday 29 October 2019
H1 ISC (CAL, ISC, Lockloss, OpsInfo)
jeffrey.kissel@LIGO.ORG - posted 19:29, Tuesday 29 October 2019 (52786)
Recovery Notes: ALS Fiber PLL, SRY During Initial Alignment, Bad HAM6 Pressure Sensor, REFLAIR Clipping on ISCT1, Wild ISS 2nd Loop... Oh My ...
S. Dwyer, J. Driggers, S. Dwyer, J. Kissel, N. Lecoeuche, T. Shaffer, D. Sigg

Recovery after maintenance today was pretty rough. The following disparate reasons were the cause of the trouble thus far, and we're not yet back in business. 
We started attempting to recover around 12:30p PDT.
At the moment, we're close to (mostly) nominal low noise, but having problems with the ISS 2nd loop once we request past 20-30W of PSL output power.

(1) Upon starting initial alignment with the green arms, Niko and I were immediately stumped by the ALS ARM guardians complaining about errors with their PDHs and PLLs. Daniel saved us on this one, with some details in LHO aLOG 52784. See his aLOg for explanation of what is happening -- anew systemic problem as a result of the new way to pick off the ALS fiber power from prior to the ref cav. His solution was to set laser crystal frequencies (e.g. H1:ALS-X_LASER_HEAD_CRYSTALFREQUENCY) to be some value close to its nominal. 

(2) After recovering the ETMY platform from the model restarts that happened today (see LHO aLOG 52768), we forgot to turn on the damping loops of the SUSTMSY, before attempting to restore the ISI to FULLY ISOLATED. This rung up some interaction between the two, and tripped the platform hard a few times. Once figured out and fixed with everyone damped and isolated as per normal, ALSY needed a lot of help in alignment (many mircoradians in pitch) before we saw any flashes. But, once Niko had the patience to iterate long enough with the TMSY and ETMY alignment, we were able to restore things enough for the greenWFS to take over.

(3) Our next stumping point after initial alignment of the green ams was done, was the failure to repair the HAM6 PT110 pressure sensor today (LHO aLOG 52775). This sensor was serving as our high voltage interlock sensor, so since it didn't work suddenly near the end of maintenance, it tripped the high voltage off. Since the high voltage was off, the fast shutter was closed, which meant that the QPDs at the AS port (AS_A, AS_B, AS_C) were blocked, and the input alignment wouldn't work -- with the symptom being that the ASC DC centering loops were railing. It took Jenne putting two-and-two together to understand this connection path as to why the automated ALIGNING_INPUT step wasn't progressing.
TJ has now added a check for the fast shutter being open during this step to the automated initial alignment guardian so that we don't get bit by this again.

(4) The final problem with Initial alignment was with aligning the SRC cavity. Sheila reports that there has been trouble with SRY (the IFO DOF used during initial alignment to align SR2 and SRM) since the recovery, in that it needs more length loop gain. The symptoms of the problem were that SRM was getting blasted and saturated by the LSC control, and the SRC alignment control signals were quite non linear. It's unclear if the saturated SRM was really a problem though, because smoourshing SRM hard tricked the INIT_ALIGN automated guardian in to thinking that SRM was *misaligned,* and thus began the "SRC is really misaligned, so I'll do the 'misalign SRM and adjust SR2 alignment' trick," and ended up slowly but surely bringing the SRC in to alignment. Regardless, Sheila added some more gain in SRY length loop in to the initial alignment guardian step. We don't have any clues yet as to why SRY needs more gain that it used to.

(5) Once past initial alignment, we got stumped on locking DRMI. The symptom was very low/small flashes in PRMI. We did the typical "shoulder shrug" redo of initial alignment, but to no avail. Once back to the same point we tried manually searching the BS and PRM all over the place for better flashes, questioning IM4 and PR3's alignment, the path of "look at the error signals" led us to suspecting REFLAIR9, whose DC light signal was really low. From there, we looked at the REFL camera image and saw next to nothing. Then we remembered that we haven't locked up the IFO since the phase camera work last night on ISCT1 -- the same table on which REFLAIR9 lives. Turns out it was that phase camera work mitigating scattered light (see LHO aLOG 52759): they had placed beam dumps in lots of locations in order to mitigate the scattered light, but did so during (mostly) nominal low noise (no squeezer), when we've got the REFLAIR path shuttered from in vacuum. Thus a dump inadvertently ended up right in the middle of this REFL path. As soon as a particular dump was removed, the camera image and light level on REFLAIR9 DC restored, and PRMI / DRMI locked up quite smoothly.

(6) We lost lock a few times during DRMI and/or it was a rocky turn on. In ENGAGE DRMI ASC state, we turn on SRC1, then an integrator to MICH, and then 5 secs later turn on SRC2. But our impression is that SRC1 and SRC2 should come on at the same time. Moved SRC1 engagement, into the next counter step, so the engagement order is now MICH integrator, then 5 seconds later SRC1 and SRC2 are engaged. This has been loaded in the guardian and confirmed successful once. Not sure if this'll fix such lock losses permanently, but we'll try it with this.

(7) Now, we're dealing with 2nd loop ISS problems. Once the request for PSL power up reaches past 20-30 W, the ISS 2nd loop freaks out, opens, closes, goes crazy, and kills the lock.
Jenne and Daniel are now investigating, and will aLOG what they find (hopefully a solution!).