ticket 12867

I'll add log results analysis soon.

TITLE: 05/08 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: Jeff
SHIFT SUMMARY:

This shift went super fast....thanks to a dolphin crash!  Ha.  But it was actually kind of fun to go through all the problems; definitely a learning experience!  Sadly, because I am slow I am only handing off an H1 with DRMI locked.

Back Story:  Just as Niko was about to leave after his handoff from me, had a major dolphin crash in the LVEA (Dave will have a more informed entry on the specifics of the crash...he did mention we hadn't had one of these since January where we had been suffering from these fairly regularly before then).

I tried to post specifics & timing of events in other alogs for what I found and did during my shift for this recovery.  In broad strokes, here were my method to the madness and areas I focused on in order through the night:

1. Proceed as far as I could with locking as usual, until I ran into an issue.  (I was stopped early due to IR cavity for X!)
2. Alog search for similar issue (Basically looked at last instances for recovery from Dolphin recoveries (thanks, Sheila, Dan, & JimW for the two entries I found).
3. Dark Offsets!  (I should have realized this was an issue early on (TR_X [blue on striptool] was up at 0.3 instead of 0!)
4. Check pit/yaw sliders for SUS (If I did this sooner (via Time Machine?) I could have saved some time scratching my head about why certain cavities were not locking.))
5. ISI Config!  (If I didn't read about this in the Dan/Jim alog, I would have been dead in the water.)

LOG:

• 7:04 Lockloss due to Dolphin crash
• 7:27 Recovery starts for CDS
• 7:35 Begin untripping stuff!
• 8:17 Begin first attempts at Locking (starting with the IMC).
• 11:05 Dark Offset investigation (because of previous dolphin recoveries Dec1, Jan17) 
  • Zeroed by hand:  H1:LSC-TR_X_QPD_B_SUM_OFFSET (from alog 46124)
  • Ran dark offset script:  dark_offsets_exe.py (at  /opt/rtcds/userapps/release/lsc/h1/scripts/dark_offsets/)
  • Adjusted Xarm dark offset again by hand since it looks like it was changed during dark_offsets_exe.py script  (from -1.1 to +0.1)
    • Adjusted Yarm dark offset since I was at it.
    • But I did both of these with IMC locked and ITMs aligned (i.e. not dark).....wonder if I need to back these out?  ---->I don't think so, because I unlocked IMC & misaligned ITMs and dark offsets looked fine.
• 11:55 - 14:05 Initial Alignment
• 14:13 LLO going down a few hours for Ion Pump replacement
• 14:05 Starting to lock again
  • IR was not found, so had to do by hand.
  • Had to do a PRMI due to BS not being well aligned (even after the alignment!) 
  • DRMI eventually locked, but we were getting OPO saturations (!), then broke lock---So something is still awry.
  • Looks like Ed/Richard fixed this by Clearing History for something on a Squeezer screen
  • H1 is now past DRMI to PREP_DC_READOUT *crossing fingers*
• 14:35 Seismon note:  Busy with recovery, but we noticed a sizeable Peruvian EQ roll through.  didn't get Verbal.  I didn't have seismon up to notice it.

Since was not able to progress past FIND_IR for locking, decided to do an Initial Alignment.  Here is that story:

11:55 - 14:05 Initial Alignment

• 11:55 - 12:30 ALS X took forever (25min) to offload (I'm assuming due to dark offsets [or just bad alignment]).
• INPUT_ALIGN notes:  
• 12:30 Input Align was stuck attempting to Acquire X-arm.  
  • Tried reducing Fiber polarization for X-arm, but had no luck as usual (I went ahead and zeroed Y-arm since I was there).
• 12:59 Noticed Messages from SEI nodes on SEI_CONF which were saying to "rerequest state" (went to SEI_CONF because this happened for Dec1 dolphin crash, so it forced me to look, and Jim has a mention of requesting other state and go back to one you want), so went to LARGE_EQ_NOBRSXY & then back to WINDY and this cleared issues.
  • Taking care of SEI_CONF gave me back IR X-arm!!  So INPUT_ALIGN_OFFLOADED from the alignment completed!  Whew!  
  • Dan B mentioned some FIR filter banks having a zero gain for ITMy in Dec, and I tried looking for that and didn't see it, but eventually noticed Jim's subentry about requesting different state, then rerequesting your desired state w/ SEI_CONF.
• MICH_DARK_ALIGN notes:
  • Completed PREP_FOR_MICH
  • For ACQUIRE_MICH_DARK:  Ran 3 commands for LSC-MICH1
  • Then it would stop.  AS spot was bright with fringes moving in pitch.
  • So could not get to MICH_DARK_LOCKED
  • Skipped!......almost...
  • When moved on to SRC_ALIGN saw it was not working either.  And this is when I happen to notice AS_AIR video looking fairly pitched UP.  So this got me to check pit/yaw sliders for optics from BS down to ISCT6 (i.e. BS, SR3, SR2, & SRM).  Restored all of these by using time machine to time of last lock...probably a better way, but it's easy, if a little time consuming.  This fixed/centered the AS_AIR video spot!
  • Dark Mich was way off in Yaw, but eventually moved BS to get a nice dark mich!
• SRC_ALIGN Notes:
  • Fairly off, so used SR2 to center spot on AS_C pd.

LOCKING UPDATE:

• Had to FIND_IR.
• DRMI wouldn't lock; PRMI did and needed lots of BS yaw (even after my fabulous alignment!) 
• When DRMI locked were getting "OPO" Verbal alarms for saturations 

This is where we currently stand.  Ed was in early and he is walking H1 through ISC_LOCK as I type.

I imagine there are still more post-dolphin-crash time bombs to be found (Ed, Richard, & Jeff are looking at our current issue:  OPO Saturations (on the Saturations medm ZM1 & ZM2 are railed!)

11:05 - 11:45 Dark Offset investigation (because of previous dolphin recoveries Dec1, Jan17) 

• Zeroed by hand:  H1:LSC-TR_X_QPD_B_SUM_OFFSET (from alog 46124), because I remembered doing this for one of the recoveries before.
• Ran dark offset script:  dark_offsets_exe.py (at  /opt/rtcds/userapps/release/lsc/h1/scripts/dark_offsets/)
  • While sort of following steps in Dark Offset wiki.
• Adjusted H1:LSC-TR_X_QPD_B_SUM_OFFSET dark offset again by hand since it looks like it was changed during dark_offsets_exe.py script  (from -1.1 to +0.1 [hindsight:  maybe this was changed during DOWN script?])
  • Adjusted Yarm dark offset since I was at it.
  • But I did both of these with IMC locked and ITMs aligned (i.e. not dark).....wonder if I need to back these out?  ---->I don't think so, because I unlocked IMC & misaligned ITMs and dark offsets looked fine.
• Moved back to locking with no luck....and then an alignment (alog forthcoming).

• Slowly pecking away.  
• Since having issues with FIND_IR, Looking at dark offset situation.  May run an alignment after dark offset work.

X & Y green arms look good.  But having issues getting to the next step.  In the Troubleshooting document there is mention of adjusting the COMM Offset, but that made no change (as well as adjusting the DIFF Offset).  I'm tempted to just run an alignment at this point, but I'm worried that won't address anything here since problem seems to be related to X-Arm for IR or PLL for ALS.  I feel like I should be focusing more on them instead of alignment.

Below are notes and observations thus far:

FIND IR Notes:

• TR_X looks to be in la la land at around 0.3.  (Should normally be around 0?  Wondering if a dark offset is not correct.)
• (per troubleshooting guide) Adjusting ALS COMM first
  • initial values:  COMM @-200.0 & DIFF @-804.2
• Adjusting COMM did nothing.  
• There is a DIAG_MAIN message for "waiting for DIFF PLL to come into range"  (But I'm not seeing any issues for PLL related stuff for X & Y arms from the ALS Overview.)
  • Delving deeper into sub-windows, do see some "redness" for the following in COMM PLL:
    • ERRORs for COMM PLL related to Beat note*
    • And further upstream see COMM VCO is sweeping the Tune Ofs slider until it reaches limits

* In the Troubleshooting Wiki there is mention of zeroing the Crystal Freq if COMM has no Beatnote, but I'm not totally familiar with that.  

At this point, I'm thinking I'm at a loss.  I could atleast cross something off the list, by doing an alignment, so I will run an alignment (after I have some breakfast).

During the shift change, we had multiple things fail on us.  While I can remember going to list what we saw/noticed in order:

• 7:04:05 "TCS[x & y] chiller trip imminent" on Verbal Alarm
• 7:04:17 All LVEA ISIs tripped, HAM236 HEPIs tripped
• 7:04:49 BS ITMy HEPIs tripped
• Looked up and we saw lots of RED on CDS overview for LVEA (& a few bits for EX & EY)
• Ran out to check TCS chillers (I knew these needed their weekly check, so wanted to check)
• 7:08 Contacted LLO & Virgo via TeamSpeak
• 7:10 Called Keita, and he asked me to call Dave
• 7:15 Called Dave
  • gave him rundown of what we saw here, and he was going to check things
• Took OBSERVATORY MODE to CORRECTIVE MAINTENANCE
• Screenshot of main windows we saw.

Just as I began to start locking, I remember Niko/Jenne requested some LOADs for  SQZ_MANAGER & HIGH_FREQ_LINES.  I did this while I was in the LOCKING_ARMS_GREEN state. 

By eye, the LOAD for the FREQ_LINES cleared the red ERROR for the node, and LOAD for SQZ_MANAGER (which did not have red ERROR) showed no visible change for the node.  Attached are log files for each node related to the LOAD.

Looks like all front ends on the corner station dolphin network are having issues, I'm starting the recovery.

Pep, Lilli,

Pep has taken 4 sets of HF measurements. The data are in ^trunk/Runs/O3/H1/Measurements/FullIFOSensingTFs/ (HF0, HF1, HF2, HF3).

Adding those HF measurements into the multi-meas GPR fitting, the results are shown in the first two pdf files. (Script: ^trunk/Runs/O3/H1/Scripts/Uncertainty/process_allmeas_writeGPRHDF5_20190502_addHighFreq.py)

Note that this fitting also includes the 0502 high-freq sweep (coherence is worse). By removing the 0502 sweep, the GPR fitting results are in the third pdf. The difference can hardly be told by eye. Including the poor high-freq sweep slightly improves the fitting.

The updated uncertainty estimate is shown in the first png (cf. the previous results in the 2nd png). The high-freq systematic error improves a lot.

With M Wade.

The DMT computers have patched and updated to gds-2.18.17; GDS calibration restated to remove Cal lines in GDS CLEANED STRAIN. This completes WP 8191.