[S. Biscans, K. Izumi, J. Kissel, A. Pele, J. Rollins]

Log of how we brought the IMC back up to locking today after the RCG 2.7 upgrade in rough chronological order:


- SUS (for each SUS --  MC1, MC2, MC3, PRM, PR2, PR3, BS, ITMY, IM1-IM4)
   - Given to us as it comes up from a reboot, in SAFE -- watchdogs tripped, masterswitch off, (stale*) alignment offsets OFF. ** 
   - Reset IOP watchdog
   - Reset Inner-most USER watchdogs
   - Reset USER DACKILL
   - Turned on Masterswitch (damping loops should already be trying to run)
   - Turned stale alignment offsets (in anticipation of having to burt restore ripe offsets that are not that different)
   - Recovered ripe alignment offsets for MC1, MC2, and MC3
       - snapped to 2013-07-29 20:00 (Local)
       - /ligo/cds/lho/h1/burt/2013/07/29/20:00/h1susmc*epics.snap

- HPI-HAM2 & HPI-HAM3
   - Given to us as it comes up from a reboot, in SAFE -- watchdogs tripped, masterswitch off ***
   - Make sure no offsets are requested, ramp down master gain
   - reset watchdog
   - turned on master switch
   - manually restored offsets in OUTF 
      - put in a 10 sec ramp 
      - entered and turned on offsets from LHO aLOG 7180 (offsets turn on use the ramp time now as well)


- ISI-HAM2 & ISI-HAM3
   - Given to us as it comes up from a reboot, in SAFE -- watchdogs tripped, masterswitch off
   - Make sure no offsets are requested, ramp down master gain
   - reset watchdogs
   - Turned on Master Switch
   - Used Command window, to 
      - damp ISI
      - isolate ISI Level 2 

- PSL
   - Runs entirely on Beckhoff, and was unaffected by the the RCG 2.7 upgrade so there was no recovery needed

- IOO
   - Given to us as it comes up from a reboot
   - burt restored h1ascimc.snap (mainly to restore alignment of Beckhoff controlled mirror at the top of the periscope -- all other mirrors are fixed)
      - 2013-07-29 20:00 (Local)
      - /ligo/cds/lho/h1/burt/2013/07/29/20:00/h1susascimc.snap
   - turned off IMC WFS loops
      - ramp down master gain in bottom left corner
   - turned on guardian ****
      - go to IMC screen, have the guardian screen handy/visible
      - In a terminal
         ]$ ssh h1gaurdian0
         ]$ screen -RAad imcguardian
           (opens a "screen" environment, which looks just like a new terminal)
         ]$ cd /opt/rtcds/userapps/release/ioo/h1/scripts/
         ]$ ./runIMCGuardian
           (some lights should go green on the IMC Guardian Screen)
           (hit successive crtl+a, then ctrl+d which "detaches" the screen session, taking you back to the original h1gaurdian0 terminal)
         ]$ exit 
           (logs you out of h1guardian)
      - the guardian starts up not doing anything (state request or current state are blank)
      - hit "manual", "quiet," or "damped" on the IMC Guardian Screen (since there are currently no "goto_" scripts for "locked," hitting "locked" doesn't do anything)
      - Manually Ramp IMC WFS master gain back up (guardian doesn't know to touch it)
      - IMC Locks up like a charm


* Because the safe.snaps for most suspensions had been captured a while ago, one can never trust the alignment offsets. 

** For MC1, MC2, BS, and ITMY, the safe state came up with the BIO settings frozen and red -- specifically the Test/Coil Enable switches were red, and changing the state request didn't change the filter settings. Scared that this was a bug in RCG's interaction with RCG 2.7, I had considered going through a full boot process and/or declaring a failure mode of the upgrade. However, since this did *not* appear on MC3, PRM, PR2, or PR3 (which ruled out that the bug was common to a given computer), my quick test to see whether it was frozen on a given SUS was open open the BIO screen and randomly pick a field to try to change. When I got to the beam splitter, my randomness drove me to try changing the TEST/COIL Enable bit instead of the State Request. When I requested to turn on the SUS-BS M3 TEST/COIL Enable, it flipped the M1 and M2 TEST/COIL enable and all BIO functionality returned, and the stages began behaving independent as expected and as normal. I went back to the other suspensions, and saw similar behavior i.e. that the BIO EPICs record was stuck, and the SAFE burt restore didn't sufficiently tickle the EPICS database to get the BIO request and monitors to report back correct information. Once tickled in the right way, all functionally returned. YUCK!

*** Sebastian, having never seen spectra of LHO IPS, we had assumed that the HPIs were locked, or the IPS were broken. Because of the nonsense in described in ** I told him to wait a bit. Then we convinced ourselves that we needed to power-cycle the H1SEIH23 I/O Chassis (because we'd heard that Dave had to restart the PSL I/O chassis this morning because it complained it had lost connection with the dolphin network or something). 

**** We tried to follow Chris' instructions from LHO aLOG 6920, but this failed because the h1script0 machine is still running Ubuntu 10.4 (as opposed to 12.04 that h1guardian has been updated to recently, like the rest of the control room machines) -- so we don't know how Chris did it. Thankfully Jamie was helping and identified the problem immediately after querying Jim.