J. Driggers, S. Dwyer, C. Gray, J. Kissel, T. Shaffer, P. Thomas, C. Vorvick, J. Warner, H. Yu Corey and I (starting about 13:00 PST), then Patrick, Sheila, and I (after 16:00 PST), plus a supporting cast listed above, recovered the interferometer after maintenance today. I detail all the catch points below and their cause / gotcha / solution if it was identified. The message: nothing broken, just the usual challenges of recovery from a heavy/scattered maintenance day. - PSL Periscope PZT swap: After PSL team installed a new PZT at the top of the PSL periscope (LHO aLOG 45853), they ran out of time, and left the beam going into the vacuum system a little bit off in yaw translation. - Cheryl, Sheila, and I confirmed on several metrics (IM4 TRANS QPDs, MC WFS DC "QPDs", ISS INPUT QPDs) with the IMC locked that the alignment in and out of the mode cleaner had changed. Solution: after driving the PZTs and MC1 to recover the spot position on the REFL MC WFS "QPDs," we locked up the IMC. The "gotcha": for beginnings of the locked IMC time, with all IMC ASC running, somehow the input to the PZT YAW loop had been turned off. This drove the IMC alignment, and downstream alignment metrics into very different / wrong places. Once we found the gotcha, the IMC ASC system restored the IM4 trans P & Y alignment to 0.2 to 0.1 "QPD units" (proportional to beam radii), respectively, where they had been 0.0 and 0.0. Further, the ratio between MC2 TRANS SUM and IM4 TRANS SUM was within 1% of that before the PZT swap, so we elected to move on. - Initial Alignment of Arms on Green: Corey and I are still new at this, so we struggled to figure out the three-mirror alignment system against several expert methods and metrics. Just stupid user error (i.e. by me) slowed us down on getting the Y arm aligned, but this needed extra work because of the ETMY ISI in the Y arm had been taken up and down as necessary for a model restart like today (LHO aLOG 45844). - What we learned: remember that if all is working perfectly and the arm(s) is (are) already well aligned, then there are 1 LSC + 3 Cavity Axis DOFs * (P and Y) = 7 control loops that help make this first step easy. "Easy" means -- just click "INITIAL ALIGNMENT" on the arm guardian state, and all the work is done for you. HOWEVER, if optics are not well aligned, then there are several layers of debugging configurations that you can try to get you back to perfect functionality: - Level 1 bad: if - the green arm transmission is between ~0.75 and 0.8, - the 7 (likely the ASC) loops running in INITIAL_ALIGNMENT are only making the arm power worse, but - the green camera spot image shows mostly 00 modes, and - the error signal for one or more of the green ASC loops are very large (say if the green ITM camera gets jostled like last week; LHO aLOG 45699), then You can request the normal INITIAL_ALIGNMENT state, but intentionally and manually disable the problematic loop, and slowly adjust the alignment of the uncontrolled optic / DOF in order to reduce the error signal and increase the green arm transmission above 0.8. Then request UNLOCKED and return to INITIAL_ALIGNMENT as normal (removing whatever you've done to temporarily disable the problematic loop). - Level 2 bad: if - the green arm transmission is between ~0.75 and 0.8, - the 7 (likely the ASC) loops running in INITIAL_ALIGNMENT are only making the arm power worse, - the error signals for the green ASC loops are large, and - the green camera image shows some flashes on non-00 modes, then You can run the LOCKED_NO_SLOW_NO_WFS state (you have to open the "ALL" button on the mini-guardian screen to see this state). In this state, *only* the LSC loop is used to lock the arms, but you're not fighting bad ASC loops. Here, adjust all three optics until you get the green arm transmission up above 0.8. Then revert to Level 1 bad instructions (or if you're lucky, you can go back to the "easy" instructions). - Level 3 bad: if - the green transmission is below ~0.75 (say, at the 0.6 level or lower), - you're seeing very few 00 modes on the green camera spot - requesting LOCKED_NO_SLOW_NO_WFS doesn't work (i.e. the guardian goes into "dance party" mode, quickly switching between many states) You should just leave the arm in the UNLOCKED state, and try - using the alignment slides to restore the ITM, ETM, and TMS to its top-mass OSEM values (not just the slider values) at a known good time of green arms locked, - focusing on aligning the optics / chambers that you know were disturbed today, and/or - keep an eye on the *red* transmitted power (if you get decent RED flashes, then the ITM and ETM are probably well-aligned, and your TMS is bad. If you have no red flashes, focus on the arms first) until you get the green arm transmission flashes about 0.8. Then revert to Level 2 bad instructions (or if you're lucky, you can go back to level 1, or even "easy"). Corey took the X arm, and I took the Y Arm. Corey was able to get by with "easy," I have to go to Level 3 Bad (some of which was my fault by accidentally looking at the RED transmitted power instead of the GREEN on my ndscope). - Initial Alignment of MICH / the BS: The "gotcha": over the past few weeks, the functionality of the MICH_DARK alignment states have been broken. Folks are working actively on it, but it's not yet fixed. The solution: one needs to simply request MICH_DARK_LOCKED, reduce darken the MICH spot on the AS port camera as much as possible by moving the beam splitter, and *skip to the next alignment step* SRC_ALIGN. - Running the DOWN state vs. the ETMX ESD: Once initial alignment is complete, one goes back to the main ISC_LOCK guardian and requests DOWN / PREP_FOR_LOCKING, and there it may get stuck constantly repeating the same commands on a loop over and over. The "gotcha": the charge measurement scripts (which create data that informs, e.g. those run today LHO aLOG 45843 that helped check the functionality of the ESDs after the power-supply configuration swap LHO aLOG 45845) do not gracefully handle exiting prematurely, and often leave the ETMX ESD in a mid-measurement configuration. This is a regular sticking point, and folks are working on making these scripts better, but it's not yet fixed. The solution: Thus, in order for the ISC_LOCK guardian to complete the execution of the DOWN state, you need to check that the following things are OK: - There is some requested drive signal coming out of the ESD system - Make sure that the bias voltage is set to "the right value" (currently -9.3; check SDF) - The binary IO configuration of every quadrant is in the "HI VOLTS" and "ENGAGED" state - Make sure the ESD driver is ON by looking that there are non-zero(or non-small) HV MON values in the very bottom right corner of the ETMX SUS overview. Once these are good, then the ISC_LOCK guardian should be able to proceed. - Locking PRMI / DRMI when it's been very slow/flaky for weeks, and you otherwise don't know if something from maintenance day is preventing it from working, or it's just the normal very slow/ flaky PRMI/DRMI: Here's where you go into the standard maintenance day tail spin of many different possibilities that launch many parallel investigations from various maybe-related clues, and then you end up changing nothing, and the problem fixes itself. After ~30 minutes of "the usual" attempts at increasing POP18/POP90 flashes by tweaking PRM / BS / SRM alignment, we were unsuccessful. The thread of clues from several folks investigating: - Sheila notices that the Tidal error signal is large, which we "know" is informed by IMC_F, which -- in the middle-of-the-lock-acquistion state of locking PRMI / DRMI with arms controlled by ALS, but off resonance -- is informed by ALS_COMM, which is probing the arms. - Sheila starts looking at optical lever motion, and notices that ETMX optical lever is moving "a lot," at ~1 urad pk2pk. - The excess motion is intermittent, on the ~5 minute time scale. "It's fixed! Who changed something? No one... Oh, it's back again..." - That launches Jim and I on a "which ISI is poorly performing?" investigation, because we know that sensor correction has been under construction for a week or two (e.g. LHO aLOGs 45641 45815), and folks were at the end stations today that may have rung up the BRSs into non-functionality. - The BRSs turn out to be fine, but it takes some effort to tell because of the sensor correction construction zone. - I find (via the SUSPOINT channels for the globally controlled cavity optics) that ITMY (not ETMX) is moving a factor of 2 to 4 more in the 30mHz to 100mHz (EQ) band. - In parallel, folks are rifling through a very red SDF system looking for settings clues. Lots of differences (from normal, fast-paced commissioning), questions asked about Beckhoff restarts not restoring settings, and too-little-expertise makes this a challenge to come up with any substantial clues. - Questions about last night's settings changes from commissioning the DRMI distract us for a bit - In parallel, Sheila and Patrick request the ISC_LOCK guardian to DOWN and use unorthodox guardian configurations and manually taking various suspensions to ALIGNED, they lock PRMI and DRMI alone, without arms. It locks up well, so they adjust the PRM / BS / SRM alignment in lock, but the adjustments are tiny. - Just as I'm trying to show when the ITMY ISI performance started to degrade, Sheila and Patrick restart the ISC_LOCK acquisition sequence, and DRMI locks right up with out them changing anything. Thus, we made it to ENGAGE_SOFT_LOOPS -- i.e. IFO is fully resonant and all ASC loops are in engaged -- no problem. We declared recovery complete at that point, but lost lock shortly after, and we've not been able to recover...