TITLE: 01/01 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
OUTGOING OPERATOR: Ed
CURRENT ENVIRONMENT:
    SEI_CONF state: WINDY
    Wind: 30mph Gusts, 20mph 5min avg
    Primary useism: 0.12 μm/s
    Secondary useism: 0.48 μm/s
QUICK SUMMARY: relocking H1 complicated by wind gusts up to 60mph over the last 24 hours

TITLE: 01/01 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: Cheryl
SHIFT SUMMARY:
LOG:

Locking cont....

13:44 Initial Alignment

• I had to intervene with SRC again

14:19 Re-Locking again

Vacuum Alarm - PT140PressureTorr

15:19 paused at CARM_TO_TR

15:31 Made it past CARM_150_PICOMETERS

15:34 Paused @ ENGAGE_ASC_FOR_FULL_IFO

15:39 Paused @ ENGAGE_SOFT_LOOPS

15:42 Advancing to DARM_TO_DC_READOUT

15:46 DARM_TO_DC_READ....lockloss

Handing off to Cheryl

At ENGAGE_DRMI_ASC the SUS OM2 WD tripped and has been followed by a never ending stream of OMC and OPO saturations.

I'm going to have to abort this attempt and try again.

Report will be entered in tracking

08:48 in the midst of high winds, high uSeism, and uncertainties with H1 pursuant to the Dolphin crash...

• Incoming EQ from Tonga 5.4

09:00 IA

• Switched SEI_CONF back to WINDY
• 09:24 I acheived a much more robust Y arm-alignment
• i had to manually change the LSC Triggring of the Upper Threshold Enable from .5 to .3 (Corey gave me this information during hand-off)
  • This change corrected a severly unstable SRC alignment routine

09:49 I began to re-lock

• I didn't use the lock acquisition tracking at first because I didn't want to flood it with uncertainties surround extreme circumstances
• i manually had to manipulate at FIND_IR to finally arrive at my first DRMI attempt
  • 10:13 first DRMI lock
    • 10:15 multiple locklosses occurred at TURN_ON_BS_STAGE2
      • I found this in aLog (TURN_ON_BS_STAGE2) but couldn't locate the channels in MEDM to verify the info contained in the log entry
      • It became a non- issue
      • I then began using the tracking s H1 made it through various stages of lock acquisition

At this point, uSeism is still rising and the winds are currently still gusting in the 30mph range. H1 has made it as far as DARM_TO_RF

TITLE: 01/01 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
    SEI_CONF state: MORE_WINDY
    Wind: 41mph Gusts, 28mph 5min avg
    Primary useism: 0.14 μm/s
    Secondary useism: 0.51 μm/s
QUICK SUMMARY:

• Wind gusts are actually more on the order of 45-50mph.
• Sheila checked in and informed me of possible issues at ENGAGE _ASC_...
  • not even going to get there with this wind s ALS is very unstable and H1 won't move past FIND_IR at this point

TITLE: 01/01 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: Ed
SHIFT SUMMARY:

New Catches During Shift: 

1. SUS_OMC guardian node in STALLED state
2. SRCL Trigger for Upper Threshold needed to be lowered from 0.5 to 0.3 for SRC ALIGN

Summary Notes:

• Rough night.  Tried to post the main points I came across during the shift, so those alogs are better for specifics.  
• Currently Ed is taking the helm and Sheila is going to log on remotely.  Ed's seeing quite a bit of wind and we're seeing lots of tumbleweeds rolling by.
• Operator NOTE:  I've not been filling out the Intervention Surveys, because this is a unique period (post computer crash & heavy winds).

Sorry, Ed---didn't want to hand off H1 to ya in the new year like this!  Hope you have better luck overnight!
LOG:

• 0:00 Hand-off from Cheryl, assessing H1's status, & chatting with Keita, 
• 1:02 Begin re-locking again.
• 2:04 SEI_CONF from MORE WINDY to WINDY
  • Winds finally drop below 15mph!  *knock on wood*
• 2:49 lockloss at MOVE_SPOTS
• 3:15 lockloss at DARM TO DC READOUT
  • CSOFT pit became noisy toward end of MOVE SPOTS
• 3:16 VAC EY Solar Ion Pump pressure alarm (this is normal)
• 3:40 lockloss at PREP DC READOUT TRANSITION
• 4:13 Initial Alignment
  • SRC failed on multiple attempts, even after SR2 hand-align
• 5:19 SUS Slider RESTORE
  • ISC LOCK to DOWN
  • 5:21---restored to 5:02 yesterday (or 12/30 at 9:02 pm local).  This was via Patrick T's "!RESTORE" button on IFO Align Compact.
  • Took PZT offsets to their values from same time, but IMC wouldn't lock.  So returned them to where they were taken to after last night's crash & then IMC WFS took over.
• 6:00-7:22 Another Initial Alignment 
  • Green arms were way out (ALSx was at 0.4 & ALSy was 0)
    • Decided to work on alignment by hand (vs wait for INCREASE FLASHES.)
  • Had a trigger issue with ACQUIRE SRY (see earlier alog)
• 6:10 Winds picking up again to around 20mph.
• 7:22 Locking again
• 7:51 Lockloss at ENGAGE ASC FOR FULL IFO  :(
• Another lockloss as I leave at Turn ON BS Stage2  :(

Since "lunch", I've done a few things (with help from Sheila via text/phone):

• RESTORED all SUS sliders to last night before-crash.
• Attempted to restore IMC PZT offsets, but this gave us an IMC which would not lock, so went back to the post-crash offset values.  IMC locked and IMC WFS took over.
• Since the whole alignment changed, ran an Initial Alignment.
• Green Arms:
  • These were way off! 
  • Didn't waste time waiting for INCREASE FLASHES, and intervened to tweak both arms by hand (X-arm was at 0.4 and Y-arm was at an ugly 0.0).
• SRC ALIGN
  • Once again, we had the issues of ALIGN IFO aligning SR2, but then it would get stuck at ACQUIRE SRY. 
  • Sheila asked me to align by hand, which means to take ALIGN IFO to DOWN and then tweak SRM in a way which:  (1) increases the AS_A_DC signal & (2) make AS AIR video spot "better" or "symmetric"
  • This was hard for me to resolve, but slowly & surely, with 0.3-0.6 urad clicks for SRM I was able to increase the ASDC signal...and the video spot actually did get symmetric!  (see AS_A_DC signal improvement).
  • But tried ACQUIRE SRY, and still no locking (where "locked" is where the AS_A_DC signal doesn't flash up and down, but "locks" and stay at a flat value).  Since I aligned it pretty well, felt something else was awry...and at this time Sheila checked back up with me (& was also watching remotely).
  • She agreed SRM alignment was optimized, and something else was probably wrong.  She mentioned it looked like a triggering issue.  Sure enough, she was able to lock SRY! 
  • On LSC OVERview / TRIG MTRX:  For SRCL_TRIG, the Upper Threshhold (ENABLE) was set at 0.5.  She changed it to 0.3, and SRY immediately locked!
• I then continued/completed the alignment

Currently returned to locking. 

Since the Alignment took me so long (almost 90min), the BS was pretty misaligned, so immediately went to CHECK MICH FRINGES.  Now I have PRM locked and will continue and see how we look.

In case we need to look at the IMC, want to show Time Machine Differences for MC1/2/3 between NOW & 24hrs ago.

Attachment shows current slider values on top row for all the MC optics & the slider windows below them are their values 24hrs ago.

Winds have finally started to subside, and have managed to get H1 to states further in ISC_LOCK, but still short of DC Readout.  

News Thus Far: 

• Transitioned SEI_CONF to WINDY (altho Etmy St1 is not in a NOMINAL state...see alog).
• BRSy has been in a DAMPING state since 10:30am this morning.  (maybe this is related to item above)   NOTE:  Sheila mentioned this shouldn't be related to my recent locklosses, since locklosses are probably squarely to blame on ASC.
• Recent locklosses have been around ASC steps.
• Worried if there is any other straggling item we missed from Model Crash last night.

Have sent out texts to Sheila & Jenne to give them a status.

Sheila suggested an alignment, too---I was prepping to do that since I had 3 locklosses in a row.

Jenne also pointed me to her alog about SDF Diffs (here), in case I want to use this to revert some channels after last night's model crash.  Attached is what the SDF screen looks like as I am in the READY state.  Perhaps if I have some more issues I can do some reverting?  Will do this only with consultation with others.

Winds have died down (under 15mph).  Transitioned SEI_CONF to WINDY.  

But have had 3-locklosses in a row around MOVE SPOTS.  (see two most recent in 2nd attachment of nuc24).

• 2:49 lockloss at MOVE_SPOTS
• 3:15 lockloss at DARM TO DC READOUT
  • CSOFT pit became noisy toward end of MOVE SPOTS
  • See attached ndscope (1st attachment)
• 3:40 lockloss at PREP DC READOUT TRANSITION

Will start an alignment.

Decided to take SEI_CONF to the nominal WINDY state, but while there, notice that it has a FALSE state (i.e. orange box) for the NOMINAL box.   See attached screenshot with all suspects involved.

My guess would be that maybe it would be related to the BRSy Status being in the DAMPING state (N=30).  I trended this channel (H1:GRD-BRSY_STAT_STATE_N), and it's been in this state since 10:37am earlier today.  Maybe we need the winds to die down some more for it to get back to the READY (N = 40) state?  2nd Attachment is the last 12-14hrs showing BRSy switching to Damping for the last ~9hrs.

Texted Jim about this and he said we look normal, so will continue on and keep an eye on this.

Perhaps another remnant of last night's model crash (since other Guardian Nodes were found to be Stalled by Ed & Cheryl), but Keita found the SUS_OMC guardian node in the STALLED state (see attached). 

To get it "un-STALLED", I first tried an INIT, but this didn't change the state.  Keita then took the node to MISALIGNED, and this removed the STALLED bit.  He then took the node to ALIGNED, and it was back to a nominal state. 

NOTE: If you are searching for STALLED Guardian Nodes, the easy way to do a quick scan is to bring up the Guardian Overview and then for each node, if the 3rd "State Word" from the left is not-GREEN, then you have a Stalled Guardian Node.   We have no more Stalled Guardian Nodes now!  :)

TITLE: 01/01 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
OUTGOING OPERATOR: Cheryl
CURRENT ENVIRONMENT:
    SEI_CONF state: MORE_WINDY
    Wind: 19mph Gusts, 13mph 5min avg
    Primary useism: 0.13 μm/s
    Secondary useism: 0.37 μm/s

Very windy drive in!  Average speeds are currrently under 20mph (but with gusts up to 20mph)...there have been avg speeds at 25mph for a good chunk of the day with gusts over 40mph a few times.
QUICK SUMMARY:

H1's been down since ~12utc (4am Pacific time) or almost 13hrs so far. 

Summarizing Current Status of H1 and what Cheryl mentioned during hand-off:

• ISC_LOCK:  READY (since 23:51)
• OBSERVATORY MODE state:  CORRECTIVE MAINTENANCE
• SEI_CONF state:  VERY_WINDY
• IMs:  Cheryl said these had slight shift after the Model Crash, have not been changed, and should be fine.
• IMC:  has been locked after SUS adjustments (IM2 pit by Keita/Ed & IM3 yaw by Cheryl/Ed)
• Ifo Alignment: 
  • Cheryl's gone through (2) alignments.  For the second one she noted that INIT_ALIGN did not complete SRC. 
  • She went through the AS_CENTERING_OFFLOADED section by hand
  • She tried going through SQZ_ALIGN_OFFLOADED
• Ifo Locking:  Has made it up to CARM ON TR a couple of times
• SDFs:  Noted SEI SDF diffs come up for TRamps with a Set Point of 0 (wondering if it should be non-zero

My Plans For The Evening:

• Review all SDF Diffs
• Contact Keita to give him a status update.
• Continue Ifo locking as is.
• Transition SEI_CONF to WINDY & try Ifo locking
• Run another Initial Alignment
• Look at IMC, IMs, anything else?

TITLE: 12/31 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: Corey
SHIFT SUMMARY:
LOG:

• 31-12-19 21:58 21:57 switched SEI to WINDY_NOBRSX since BRSX vel is running between +/-1500 - just a test to see if H1 locks or holds lock better
  • not better, reverted back to MORE_WINDY
  • on the SEI medm there's a suggestion that there's a VERY_WINDY SEI state, however it's not in the guardian states
• 31-12-19 22:39 finished ushering H1 through Initial Alignment
  • to the credit of all who've worked to make H1 robust, after the alignment, it made it to CARM_TO_TR, twice!!!
  • the second time, guardian thought there was no IR in the arms, and I tried to correct it, and then H1 lost lock, reason, likely wind, but it's possible my attempt at correcting the not-really-missing IR caused the lockloss
• wind plots attached - gusts over 40mph are still happeneing about every 5 minutes of so.  It looked like the wind would decrease, but no...

OUTSTANDING:

• there are a number of HEPI and ISI TRAMP times that are either different or zero
  • this seems like it'a related to the dolphin crash, however, leaving TRAMP times at zero is not typical
  • I'm wondering if we've overlooked TRAMP values in our burt restore files, or if we are relying on the SPM differences?
  • this needs to be corrected before going to NLN
  • I emailed Dave Barker, but didn't yet call, since H1 is not yet locking and going to NLN
  • handed off to Corey
• during recovery last night, the input optics ended up at alignments that have some changes, befoe the dolphin crash to after
  • I checked IM4 Trans and it's different by 0.01 to 0.02, which I don't know what that means in real space, but the change isn't 0.3
  • should H1 have locking issues after the wind dies down, it wouldn't be extreme to restore the IMC optics (and PZT) and IM 1-3, to pre-dolphin crash alignments, and then doing a new Initial Alignment
• during both Initial Alignments, SRY would not lock, and during the second IA, I noticed that the log said that it was waiting for the optics to settle
  • I believe that the ground motion was so great, that guardian thought that the optics were swinging due to locking
  • maybe we could think about changing the threshold for "swinging" from alignment, vs excess motion due to ground motion?