SUMMARY:  Have been locked for almost 4hrs and in the last hour noticed our range slowly taking a dip in range over a 30min span (went from 112Mpc down to 102Mpc!).  Cause for the dip was a noticeable line on DARM which was sweeping between 65-120Hz (along with a few of its harmonics).  It's gone now.

TIME Of Event:  13:27 - 14:02utc Stopped abruptly at 14:02.

DARM BLRMS Striptool really show the effect.  H1 had a normal glitch and right after the glitch the the 60-100Hz band remained elevated.  100-450Hz band would go up a little due to the harmonic, but when the fundamental swept above 100Hz, then the 100-450Hz band became the noisiest.  (nuc6 Note:  100-450Hz has been off-scale on this StripTool, so I increased the vert axis and saved the file).

I took power spectra of a quiet time (blue) & during this noisy time (red) & you can clearly see the fundamental and it's harmonics.  But like I said, it was noticeably sweeping across DARM (I seriously looked at the CDS Overview for any excitations, but everything looks green in here!).  

Attached is a screenshot showing the power spectra, DARM BLRMS StripTool, & the BNS Range.

SUMMARY:  Wish I could say I actively curred H1's issue tonight, but all I can say is my breakfast got us back to OBSERVING after ~4hrs of being DOWN.   Below is a summary of with highlights in bold.

• Locking
  • 6:58 Lockloss due to EQ
  • 7:03 Ran Jenne's ASC WFS Phase command
  • 7:30 transitrioned to WINDY (since the nearby EQ clearly had moved on.  Seismon gave location of "Canada", but USGS probably has it right with a 4.8 EQ near Ferndale, CA)
  • 7:34 LOCKLOSS:  During ENGAGE_ASC_FOR_FULL_IFO, POP90 started to go up & then caused a lockloss (first of several!)
  • 7:34 - 9:00 Had 4-5 Locklosses all around ENGAGE_ASC.
  • 9:09- (about) 10:05 Kept H1 at OFFLOAD_DRMI_ASC while I made an egg sandwich.
  • 10:09 After reading alogs to see if the ASC WFS Settings need to be changed (they don't...with our current protocol for the weekend.  We manual toggle them between states via Jenne's command line), I didn't have anything else to try, so I went to ISC_LOCK to NLN.  
    • NOTE:  Over the ~1hr h1 was at OFFLOAD_DRMI_ASC, several LSC signals on the strip tool (notably POP18) slowly trended down (& then I touched up the DRMI alignment, to get POP18 up to around 60).  [See attached photo of strip tool]
  • 10:33 Made it to NLN (Guardian hovered a bit in a few steps to wait for ASC & ADS signals to converge).
• SDF Diffs:
  1. Had (4) for ASC WFS phases (expected), so I ACCEPTED these (will change this after H1 heats up and change the phases again).
  2. Had (1) for LSC's H1:LSC-SRCLFF1.  It has several filters engaged.  I spent (too much) time trying to find this filter bank to no avail.  Since H1's performance looks decent, I ACCEPTED this diff (of course screenshot-ing the difference first, [see attached]).  (What and where is this channel??)
• 10:50 Back to OBSERVING
• Have my alarm set for 12:00utc so I can pop out of OBSERVING and change the ASC WFS phases.

First attempt at locking had a lockloss at ENGAGE_ASC_FOR_FULL_IFO (where, POP90 started to increase & then a lockloss).

Giving it another try and watching during the step noted above.  If it drops out, will try an alignment.

TITLE: 05/05 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Earthquake
OUTGOING OPERATOR: Patrick
CURRENT ENVIRONMENT:
    Wind: 15mph Gusts, 12mph 5min avg
    Primary useism: 0.07 μm/s
    Secondary useism: 0.34 μm/s
QUICK SUMMARY:

Arrived to see Patrick dealing with a quick and nearby earthquake (& then H1 dropped out of lock a few minutes later).  I was scared about ALSy....but it looked normal.  :) Had to manually FIND IR (Patrick also had to do that earlier). 

NOTE:  OH, I wasn't sure if our new protocol is to ALWAYS run Jenne's AS_A command after every lockloss (and run other command after things heat up for 90min).  So I just ran it when we were at LOCKING_ALS.

TITLE: 05/04 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Earthquake
INCOMING OPERATOR: Corey
SHIFT SUMMARY: Keita came in and fixed the issue with the ALS Y arm. Finished initial alignment. Had to find IR by hand. Had to turn on squeezer noise eater. Made it to observing. Just lost lock from an earthquake in Canada.
LOG:

23:03 UTC Changed observatory mode to corrective maintenance
23:05 UTC Jeff J. back from biking down X arm
Keita fixed ALS Y arm.
02:42 UTC Keita left. Set observatory mode to initial alignment. Starting initial alignment.

Cleared excitations on ITMY:
patrick.thomas@zotws7:~$ diag -l -c
supported capabilities: testing  testpoints  awg  
diag> awg clear 30 *
reset succeeded
diag> awg clear 30 *
reset succeeded
diag> quit
EXIT KERNEL

The excitation bit went away but there are still two test points open. Not sure how to clear them.

03:21 UTC Green WFS offloaded
03:49 UTC Initial alignment done
03:55 UTC IR not found
While changing DIFF offset to find IR, got quick flash, and guardian immediately went on. Not sure if I really found it.
Lost lock engaging DRMI ASC. POPAIR_B_RF18_I fell off.
Lock loss engaging DRM_ASC again. POPAIR_B_RF18_I fell off again.
Going back to check IR
04:29 UTC Found IR by hand

Squeezer guardian is giving 'TTFSS EOM noisy' squeezer notifications. Jenne had me turn of squeezer noise eater. Accepted SDF differences for this.

05:20 UTC Observing
06:42 UTC Ran "z step H1:ASC-AS_A_RF36_SEG1_PHASE_R +1,40 H1:ASC-AS_A_RF36_SEG2_PHASE_R +1,40 H1:ASC-AS_A_RF36_SEG3_PHASE_R +1,40 H1:ASC-AS_A_RF36_SEG4_PHASE_R +1,40 -s 1" per Jenne's alog. Took us out of observing.
06:44 UTC Observing
06:53 UTC Earthquake arriving from Canada. Switched SEI_CONF to EARTH_QUAKE. Hope it isn't too late.
06:58 UTC Lockloss

Noticed a notification of an earthquake from Canada after seeing the DARM spectrum get noisy. Quickly transitioned SEI_CONF to EARTH_QUAKE per the plot of responses to take. Managed to hold look through the transition, but lost it a couple of minutes later.

06:42 UTC Ran "z step H1:ASC-AS_A_RF36_SEG1_PHASE_R +1,40 H1:ASC-AS_A_RF36_SEG2_PHASE_R +1,40 H1:ASC-AS_A_RF36_SEG3_PHASE_R +1,40 H1:ASC-AS_A_RF36_SEG4_PHASE_R +1,40 -s 1" per alog 48948. Took us out of observing.

Accepted attached SDF differences.

06:44 UTC Observing

05:20:39 UTC

Accepted the attached SDF differences. The squeezer manager was giving 'TTFSS EOM noisy' notifications and was not proceeding past LOCK_TTFSS. Jenne had me turn on the squeezer noise eater. The SDF differences in the second screenshot are for that. The differences in the first screen shot are for the RF36 phasing. They will come back after running the script in Jenne's alog 48948 in a couple of hours.

Keita has fixed the ALS Y arm. Proceeding with initial alignment.

1. While working from home talking to Ed over the phone:

Changed IY, EY and TMSY alignment back to in-lock position. Then manually align things -> Y green transmission 0.6, locks, but WFS doesn't like it. Disabling DOF3 didn't help, enabling just DOF1 or just DOF2 didn't help, it just ran away.

Asked Ed to do baffle alignment but I don't see that it actually completed.

2. I came in and sit with Patrick.

Did baffle alignment for TMSY and ITMY, then manually aligned ETMY -> Y green transmission 0.9-ish, locked. Started with WFS DOF1 and DOF2, very low gain, didn't run away. Enabled DOF3 but the servo wasn't doing anything for the camera position. Increased the gain to nominal (0.2) and it worked.

Now TRY is about 1.1, looks good, ready for initial alignment.

TITLE: 05/04 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
OUTGOING OPERATOR: Ed
CURRENT ENVIRONMENT:
    Wind: 14mph Gusts, 11mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.11 μm/s 
QUICK SUMMARY:

Ed reported that Keita is on the way in to help with troubleshooting.

TITLE: 05/04 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Aligning
INCOMING OPERATOR: Patrick
SHIFT SUMMARY:

H1 never recovered after the near 42 hour lock. There's something amiss with the Y arm alignment. Keita and I have been on the phone for ~4 hours trying to do things remotely before deciding to come out to the site.

LOG:

17:00 Started initial alignment. Y arm alignment was way out

16:17 GRB alert - LLO contacted H1 still in alignment

16:23 Y arm locked but at the expense of ITMY - trying to correct

16:43 Dither Align TMSY

16:58 Y arm locked once again. Better but may take some time to converge ITM YAW. - still no satisfaction

22:50 Been on the phone with Keita most of the day. Y arm locking has been weak to say the least. He is on his way in now.

TITLE: 05/04 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 112Mpc
INCOMING OPERATOR: Ed
SHIFT SUMMARY:

Wildlife observed:  coyote & porcupine around parking lot.  Also noticed an owl standing just outside the Control Room entrance (it flew away as Patrick opened door to OSB main entrance.
LOG:

• 9:57 - 10:57 GRB Alarm (via FERMI),confirmed receipt of notification w/ LLO.
  • INJ_TRANS automatically takes care of injections (takes node to EXTTRIG_ALERT_ACTIVE)
• 10:24 MX low temperature/FMCS alarms begin.
• 10:54-10:56 OUT of Observing due to Squeezer!  (relocked entirely on its own within 60-80seconds).
• 10:56 OBSERVING
• 13:18 Lockloss!
• Spent the rest of the shift dealing with Y-arm misalignment!  (see earlier alog)
  • Cheryl called during this recovery and mentioned team of elk heading toward EY!  (not sure I can blame them though)
• During Y-arm green recovery, I ran Jenne's command for relocking (from her alog).
  • Will need to run 2nd command when H1 at NLN for ~90min.

Symptoms:

• Do not see a spot for ALSy on the video
• Error signal for CAM ITM PIT is above 0.2 (it is at -1.2).  Trouble shooting document said I should Clear History, but that does not appear to work any more (the file for it is no longer in the folder....so maybe this is an old solution?)
  • Clear History file was:  /opt/rtcds/userapps/release/asc/h1/scripts/softCH_arm.py  (but softCH_arm.py is no longer in that folder)
• Continue investigating