TITLE: 09/11 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 115Mpc
INCOMING OPERATOR: Ed
SHIFT SUMMARY:

• Whew!  What a night!  A bit of a rough night after the first hour!!  Thank you to Jamie & Jenne for helping from afar, Dave for taking a middle-of-the-night phone call, and Richard helping out with some final work  at EY to get us through the computer problem. 
• From 9:30-13:25 discovered and addressed issue related to h1boot1 crashing on us (see plethora of other logs from overnight).
• Unfortunately, during the last hour of the shift, after a nice Initial Alignment, H1 is having issues getting past ASC setps (it's locking in DRMI quick and fine though).  For the ENGAGE ASC locklosses it was at a time before the "saturation period" we get as we walk the spot through the ITMy point absorber.  :-/

LOG:

• 8:06 Lockloss (this was preceded by 17min with about 5-7 glitches.)
  • 8:06-8:32 LOCK#1 
    • DRMI flashing didn't look good, so went to CHECK MICH FRINGES (BS was off...mainly yaw) & then PRMI (PRM need a little yaw which I did, but then let ASC optimize).
    • Lockloss at RESONANCE
  • 8:33-8:59 LOCK#2
    • DRMI didn't lock & guardian locked PRMI to get DRMI to lock
    • Lockloss at ENGAGE ASC FOR FULL IFO

• 9:00-XXXX LOCK#3:  THIS IS WHEN h1boot1 issue began...in the middle of running an Initial Alignment
  • This is when the h1boot1 issues first started as we lost EPICS read/write capbabilities, now we transition to Observatory Mode to CORRECTIVE MAINTENANCE to address h1boot1.  After talking with Jamie/Jenne, there was nothing we could do, so now reached out to Dave.
  • Saw suggestion of taking OP to STOP & then EXEC (this was listed as a suggestion in ALIGN_IFO guardian log)...but this didn't appear to do anything.
  • Ran a LOAD do clear the fault.  then ran a DOWN again, and this brought back the error.  :-/
  • ~ 9:30utc Issues Begin:  For MICH BRIGHT, ALIGN IFO went into FAULT/red with a EZCACONNECTION ERROR (something due to not observeing effect of writing value to switch channel PRC1_P within a time out)
  • Initial Alignment run
• h1boot1 Recovery Begins  (see specifics from David's alogs)
  • 10:22(3:22amPDT): Left Message with David for h1boot1 recovery
  • 11:20-11:55 Rebooting front ends (sus, sei, isc) at EY (Corey)
  • 13:08 Richard off to EY to address communication issue between Corner & EY
  • 13:25 Dave calls to say we look all recovered as far as from the CDS side.  (my first order of business is Green EY....I don't see any flashes in that arm....Green EX has been locked).
  • Observatory Mode is remaining in CORRECTIVE MAINTENANCE until we get back to NLN.
• Restored sliders to 6:00 utc. via Patrick's script:  This immediately brought back flashing to Green ETMy!

• 13:40-13:59 Ran Initial Alignment (ran through all steps with no issues this time!)
• 13:59-14:10 LOCK#4
  • 14:10 Lockloss at  CARM TO TR (NOTE:  This is 1st lock after an Initial Alignment!)
• 14:10-14:23 LOCK#5
  • 14:23 Lockloss at ENAGE ASC FOR FULL IFO as LSC signals like PR GAIN all began to quickly nosedive.
• 14:25-14:41 LOCK#6
  • Paused at PREP_ASC for a minute or three.
  • 14:41 Lockloss at ENAGE ASC FOR FULL IFO as LSC signals slowly drifted down over the order of 30sec (did not see a growing Control Signal as an obvious culprit to address, via Jeff K's Control Signal Reduction script...but maybe I was not looking hard enough).
• 14:41 - 15:03 LOCK#6
  • Holding at PREP ASC FOR FULL IFO for about 7 minutes.
  • 15:03 Lockloss at ENAGE ASC FOR FULL IFO for a 3rd time!
  • Tried looking at ASC signals and maybe  both DHARD pit & yaw could be culprits, but PR Gain (& other LSC signals) all dropped very fast that using the Control Reduction Script would probably not help.  At a loss for what to try next...maybe another alignment?  This will be the gift I hand over to Ed.