J. Kissel, for just about everyone on site

Today was the official start of the so-called "re-locking team." As such, I've folded Ed, Jim (the operators on shift) and Betsy into the routine that I've done in the past for particularly busy Tuesdays and we've made an effort to focus more on speedy recovery from each task. Each part of the following tasks is necessary if we expect to robustly get back to locking ASAP:
- Best effort basis understanding of all planned maintenance activities, to obtain
    - an assessment of what the activities impact others
    - an assessment of those activities impact on the IFO
    - an assessment of how we'll recover from and what to test after those activities
- Preparing the IFO for maintenance
- Keeping track of the chaos during maintenance
- Beginning regression testing and recovery on as much of the IFO as soon as possible
- Following through with activities making sure they're closed out, performing regression testing, until the IFO performs as good as it did the night before
- aLOGging anything and everything you can about the day, highlighting the big stopping points and hold ups.

Here's how today when down.
First, refresh your memory with how the plan was supposed to go: LHO aLOG 19600
(All times PDT)
7:45a Ed, Jim, and I prep for maintenance (LHO aLOG 19616)

~8:00a Rick, Jeff, and Jason head into the PSL to do RefCav and PMC Alignment
      Fil installs new 9.1 MHz OXCO, and hooks it up to the timing system
      Dave, Jim, and Richard go to EX to replace EX SUS front end
      Bubba heads to EX for TMDS Piping
      Hugh begins checking corner station HEPI accumulator bladder pressures

~10:00a PSL team takes longer than expected, and only gets through RefCav alignment, forgoes PMC alignment until next week (LHO aLOG 19622
       Hugh is finished with Corner Station HEPI, moves on to EX and EY (LHO aLOG 19632)
       Fil moves to running BNC cables with Jordan and Vinny near HAM2 and HAM4
       Dave reports back that EX went terribly because they got shipped the wrong type of card (they got a DAC not an ADC) for the new PI damping ADC, moves to Y (LHO aLOG 19659)
       Recovery 1:Robert needs IMC recovered for PSL periscope tuning, and it's part of the plan anyway to recover the IMC after PSL work is done, so Relocking Team recovers corner station SEI, with a focus on HAM2/HAM3 so the IMC can get back up and running

10:15a TCS X LASER trips because of finnicky temperature sensor in racks near HAM4, suspected Fil / Vinny / Jordan activity (LHO aLOG 19624)

10:30a Jim still having trouble relocking the IMC (LHO aLOG 19623), we call in Keita he diagnoses as an alignment problem (LHO aLOG 19627), likely because the IMC REFL camera as a *little* low, and HAM2 had tripped during recovery, and we've see IMC REFL periscope problems in the past.

11:00a Jeff begins compiling, installing, and restarting ITM models
      Jim and Dave finish with EY and EX SUS front end replacement
      Recovery 2:Leo measures charge to confirm ETM functionality before front-end model changes, confirms functionality but a sign flip from an out-of-date snap of non-monitored filter gain
      Stefan installs ODC MASTER changes (LHO aLOG 19654)
      Jordan, Vinny, Katie perform PEM tap-test calibrations in all VEAs

11:30a Jeff finished with ITMs, moves onto ETMs for SUS model changes

12:00p Kiwamu and Sudarshan begin install of ISS outerloop electronics
      Dave finds bug in Jeff's code changes for ITMs that caused front end to stay stalled after startup, fixes them

12:15p IMC does OK for a bit, but continues to putter, Robert Kiwamu and Sudarshan is still delayed
      Jeff done with end station models, (LHO aLOG 19655)
      Dave runs quickly through all other computer reboots and a DAQ restart (LHO aLOG 19659)
      Recovery 3:Jenne and Ed begin recovery of end stations and green locking

12:30p Recovery 4:TCS X LASER recovered (LHO aLOG 19624)
      PSL trips because a shutter closes (of course this means we lose the IMC), Ed investigates and recovers

12:40p IMC locked, but remains squirrelly

1:00p  Jenne and Ed, with the help of realize that alignment offsets and M0 settings for QUADs are bogus because of out-of-date SDF snap of non-monitored alignment offsets and M0 LOCK filter gains

2:00p  Still battling the IMC Robert calls Rick; Rick and Jeff discover that the real problem with the IMC had been the RefCav realignment causing FSS oscillations all along (LHO aLOG 19631) and alignment was a read herring. 

2:11p  Recovery 1 complete Stable IMC lock, Robert, Kiwamu, and Sudardshan finally get started

2:52p  First of many models get killed by SORT ON SUBSTRING SDF bug, starting with ETMY, further stalling green arm locking recovery (LHO aLOG 19628)

3:30p  Diagnosed the problem of the SORT ON SUBSTRING SDF bug, so that stopped (LHO aLOG 19650)
      Ed, Jenne, Sheila also get stalled with OFFSET RAMPING BUG (LHO aLOG 19653)
      
4:00p  Jim and Ed's shift is up, Jim has to go, Jim stick around for a little, still neck deep in green ALS recovery

4:50p  Ed has to go, Jenne and Sheila "take over" though they've already been heavily involved
      Recovery 1 ... AgainFSS oscillations start again, Sheila has to reduce the FSS gain by 3 dB (LHO aLOG 19641)

6:30p  Initial alignment complete, begin lock acquisition attempts. 
      Sheila / Jenne identify that ALS COMM won't lock because error signal for VCO frequency is identically zero

7:20p  Recovery 3 complete After call to Dave, we figure out that at 8:00a, during Fil's install of the new 9.1 MHz OXCO, the timing comparator signal from which the COMM (and DIFF) VCO frequencies are derived had been moved to a different port of the timing fanout, foiling the information hardcoded Beckhoff PLC that converts the fanout channels to ALS channels. So we reverted the comparator to the right port and moved the oscillator to another port. (LHO aLOG 19646)
      Moving on to DRMI lock in the acquisiton sequence

9:20p  Find bug in ISC_LOCK guardian, a failing conditional statement from code installed a few days ago

9:38p  Get to RESONANCE (i.e. full IFO RF locked), find that the ITMY Bounce Mode is EXTREMELY rung up. Trace it down to bug in ITMY's Bounce / Roll / Violin mode error signal mapping (LHO aLOG 19657)

10:00p ITMY SUS and SEI recovered, 

10:30p Evan / Sheila Continue to wait for DRMI to lock, having trouble and confusion about PRM alignment, ASC start up / lock acquisition, 
       | 
       V
1:30a Resonance! Finally! Can begin to damp bounce and roll modes.

Still haven't fully tested and completed Recovery 2 as of this entry, because we haven't confirmed that we can get to ETMY low-noise lock.

I attach a panoramic of the white board, that Betsy to picture of yesterday (see LHO aLOG 19603), just to show how the best laid plans begin to fall to pieces as reality sets in throughout the maintenance day.

So, although got a ton done (as shown in LHO aLOG 19600 there were double the number of tasks that I mention, but they didn't come up because they didn't *end up* having an impact on immediate recovery), including all of Daniel's major tasks (from LHO aLOG 19451), we still lost a large fraction of time to the following:
   - Much belated diagnosis of problems resulting from PSL alignment activity
   - New /green users rediscovering the problems with computer restarts because we still don't have a good solution for holding alignment offsets and other important unmonitored SDF filter bank channels through the computer reboots
   - Much belated diagnosis of impact of moving timing comparator from PORT 11 of timing fanout
   - Brand new problems with SDF monitoring system
   - Much belated diagnosis new QUAD software bugs
   - The "relocking team" fizzling out after 4-5:00p.

This being said, I don't think this is any different from any other heavy maintenance day that I've planned / coordinated to this level of detail ahead of time, e.g. LHO aLOG 16165, LHO aLOG 10849, etc.