Reports until 17:44, Tuesday 04 December 2018
H1 ISC (CDS, OpsInfo, PSL, SEI)
jeffrey.kissel@LIGO.ORG - posted 17:44, Tuesday 04 December 2018 - last comment - 10:20, Wednesday 05 December 2018(45696)
Rough Recovery of IFO
J. Driggers, J. Kissel, N. Lecoeuche, E. Merilh, H. Yu, C. Vorvick

We've had a rough time recovering from maintenance activity today, and are still struggling. 

Thus far (as far as we can tell) the biggest offender appears to be debugging work on the Y-ARM Red camera system that appears to have moved the Y-ARM Green camera, thus (we think) spoiling our green alignment reference. Thus initial alignment was particularly painful (the error signals on the P/Y green camera loops were in the ~5-6 range, when they should be below 0.1), which we eventually "solved" by 
    - reverting the ITMY, ETMY and TMSY alignment (via moving sliders and steering to top mass OSEM values) to the last time the green arms were locked (yesterday, Dec 4 2018 @ 10:00 UTC), 
    - with ISC_LOCK set to INITIAL_ALIGNMENT, (and thus ALIGN_IFO already in SET_SUS_FOR_ALS_FPMI), set ALS_YARM to LOCK_NO_SLOW_NO_WFS and confirm that the transmitted power in the YARM is above 1.0
    - while confirming the the camera loops are OFF (using your favorite method, today we chose to hold the outputs of the camera loop filters at 0.0), set ALS_YARM to INITIAL_ALIGNMENT. This turns on the green WFS, but forces a zero to the control output of the camera servos (even though it looks like they've been turned on by the guardian.)
    - wait for the ALS WFS error signals to converge toward oscillating around zero with small amplitude. Accept that the camera loop error signals are large, and request GREEN_WFS_OFFLOADED
    - move on with the rest of initial alignment.
As such, aligning MICH was a little more difficult, and aligning the SRC was more difficult.
With this path forward, we'll not know that we have a good alignment -- and PRMI/DRMI lock acquisition will be more challenging -- until we *get* PRMI/DRMI locked, start running *red* WFS, get through the CARM reduction, get the arm HARD and SOFT loops engaged. Once we get the full red ASC systems running -- then we'll be sure to check the green camera offset settings and confirm that this was indeed the cause of our problems.

We've also noticed that the arm cavities are flashing a lot faster than normal during all of this. We suspected the great ISI model change today (LHO aLOG 45641), the potential resulting spoiling of sensor correction, and/or or the binary IO miswiring (LHO aLOG 45687), but have triple-confirmed that all settings are as normal, even though the user interface is not yet functional (thanks to TJ and Jim!).

We also have suspected the temporary settings changes on the PSL loops (PMC, ISS, FSS) that were required after the long PSL incursion today (LHO aLOG 45692), but those have been restored several hours later (i.e. after the PSL has re-thermalized) and we believe we're operating under normal conditions and performance (thanks to Jason and Cheryl).

Once we got past initial alignment, there was a small hiccup with the ETMX ESD system, which just needed to be turned ON after this morning's charge measurements were stopped prematurely. (This usually confuses the ISC_LOCK guardian's DOWN state, which has some out-of-date checks for ESD driver functionality.) This has been fixed.

Finally -- As I'm writing this log, we've also suddenly started seeing large, ~28 Hz oscillations in the red arm powers when locked on ALS COMM and DIFF.... and then now they've disappeared. The times at which we saw this problem were from about Dec 5 12:30 UTC to Dec 5 01:24 UTC.

While Jenne and Hung are sticking it out, the investigation continues with renewed vigor thanks to Sheila and Stefan, and we've only once achieved a DRMI lock.
    
Comments related to this report
sheila.dwyer@LIGO.ORG - 10:20, Wednesday 05 December 2018 (45709)

Just a note about the ESD checks in the PREP_FOR_LCOKING state (not down).  They are correct checks, it correctly did it's job (told us the ESD wasn't on) before we started trying to use it to lock ALS.  

Also, the noisy ALS problem that Jeff mentioned at the end of his log was due to some bounce and roll notches in MC2 M2 L that were turned off on monday night to try to diagonse the problems we were having powering up.  We've turned them back on.