TITLE: 03/07 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 116Mpc
INCOMING OPERATOR: Patrick
SHIFT SUMMARY:

Quiet shift with H1 locked 18.5+hrs.  Only one Squeezer drop & I missed it for a few minutes!  

Microseism looks like it turned back around to below the 90th percentile; yay!
LOG:

• 17:42-18:29 Out of Observing due to Squeezer and me being out of the Control Room.
• 20:31 One of the roof cameras went into a "spin" & stayed that way (LiveClient lost connection to the server).  I phoned Richard and we had to go restart the computer for the camera system at about 21:10utc
• 21:27-21:59 Stand Down with INJ_TRANS for Short GRB
• 21:25 Noticed "Stand Down" on Lock Clock...think this was due to INJ_TRANS activity a while after I had taken it to SUCCESS.

H1's been locked 14.75hrs (out 47min due to Squeezer/Operator absence as noted earlier).  Robert & Dave are also  on site.

I was out of the Control Room this morning and we dropped out of Observing due to the Squeezer.  I didn't notice this until 18:29.  In hindsight, I should have made sure my Lockloss Alert was set for my phone while I was away.  Anyway, here are some notes on the timing:

• 17:42 Out of Observing due to Squeezer. 
• 17:42:34 - 17:44:03 Squeezer was activingly changing to get us back to NLN.
• I was back in the Control Room at about 18:26 & happened to notice we were out of Observe (on GWIstat & then Dave noticed Ops Overview), I took us back to Observe.
• 18:29 Back to Observing

TITLE: 03/07 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 115Mpc
OUTGOING OPERATOR: Jim
CURRENT ENVIRONMENT:
    SEI_CONF state: WINDY
    Wind: 19mph Gusts, 15mph 5min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.59 μm/s

Microseism has taken a step up in the last 6hrs (so it's now at the 90th percentile).
QUICK SUMMARY:

H1's been locked for 11+hrs.  Range is between 115-120Mpc.

GWIstat says H1's been in Observing 45min which is a little odd.  I scanned through Guardian nodes & Summary Pages to see anything obvious over the last 9hrs, here are some notes:

• Summary Page shows a Squeezer drop at 8:03utc (see below) & then there is a drop at ~15:35utc which flags the Calib.  (see screenshot of "sqz_calib_summary.png attached)
  • Additionally I'm attaching a Summary Page's h(t) generation page's screenshot. (see screenshot of calib.png)
• 3:55utc See a few changes on AWG_LINES Guardian node:
  • 2020-03-07_03:55:47.021585Z AWG_LINES REQUEST: IDLE
    2020-03-07_03:55:47.021585Z AWG_LINES calculating path: IDLE->IDLE
    2020-03-07_03:58:23.513761Z AWG_LINES REQUEST: IDLE
    2020-03-07_03:58:23.513761Z AWG_LINES calculating path: IDLE->IDLE
    2020-03-07_04:27:21.830281Z AWG_LINES REQUEST: IDLE
    2020-03-07_04:27:21.830281Z AWG_LINES calculating path: IDLE->IDLE
• 8:03utc-8:04utc (i.e. 12:03amPT) Squeezer issue which briefly dropped us from OBSERVING.

TITLE: 03/06 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 116Mpc
INCOMING OPERATOR: Jim
SHIFT SUMMARY: One lock loss from observing. Briefly out of observing due to squeezer at end of shift. No other issues.
LOG:

03:55 UTC Lock loss. Likely from earthquake in Mexico.
04:01 UTC WDs have started tripping.
04:11 UTC Started untripping WDs.
04:14 UTC Done untripping WDs.
04:15 UTC Started relocking.
04:27 UTC Lock loss from LOCKING_ALS.
05:18 UTC Observing. No SDF differences.

I have scheduled two detchar injections per site, the first is at 10:00 UTC and the second at 12:00 UTC (both on March 7th). Here are the lines that were added to the schedule file:

1267610418 H1 INJECT_DETCHAR_ACTIVE 1 1.0 detchar/detchar-hwinj-schedule_LIGO-T1900555-PROPOSED_{ifo}_INJECTIONS-1251586818-390.txt
1267610418 L1 INJECT_DETCHAR_ACTIVE 1 1.0 detchar/detchar-hwinj-schedule_LIGO-T1900555-PROPOSED_{ifo}_INJECTIONS-1251586818-435.txt
1267617618 H1 INJECT_DETCHAR_ACTIVE 1 1.0 detchar/detchar-hwinj-schedule_LIGO-T1900555-PROPOSED_{ifo}_INJECTIONS-1251586818-390.txt
1267617618 L1 INJECT_DETCHAR_ACTIVE 1 1.0 detchar/detchar-hwinj-schedule_LIGO-T1900555-PROPOSED_{ifo}_INJECTIONS-1251586818-435.txt

I also restarted the INJ_TRANS guardian while the IFO was recovering from the earthquake, although this would not have affected the lock or observing mode. The request has been set back to INJECT SUCCESS.

Just lost lock for the first time this shift, likely from an earthquake in Mexico that hit too fast to respond to in time.

TITLE: 03/06 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 115Mpc
OUTGOING OPERATOR: Cheryl
CURRENT ENVIRONMENT:
    SEI_CONF state: WINDY
    Wind: 10mph Gusts, 6mph 5min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.43 μm/s 
QUICK SUMMARY: No issues.

TITLE: 03/07 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 116Mpc
INCOMING OPERATOR: Patrick
SHIFT SUMMARY:
LOG:

• 06-03-20 16:50 Marc to MY to pick up lids
• 06-03-20 17:22 Tyler and crew heading to X arm, between MY and EY, to work on tumbleweeds (shredding)
• 06-03-20 17:44 Ed transitioning LVEA to Laser Hazard
• 06-03-20 17:50 Ed done transitioning LVEA to Laser Hazard
• 06-03-20 18:01 Robert and Jenne to LVEA - HAM5 - pictures of beam on septum
• 06-03-20 18:01 WIFI in LVEA is ON
• 06-03-20 18:01 Marc back (around 17UTC)
• 06-03-20 18:02  H1 to Commissioning for Robert - about 2 minutes after Robert and Jenne left the Control Room, so the last minute of Observe may be pollutted
• 06-03-20 18:37 H1 lockloss
• 06-03-20 19:16 Gerardo to MY
• 06-03-20 19:24 H1 in Observe
• 06-03-20 19:41 Carlos turning on WIFI in MSR - will not disturbe Observe
• 06-03-20 20:31 Gerardo back from MY, heading to MY after lunch and Filiberto will be joining him
• 06-03-20 20:47 Filiberto and Ed to optics lab to test electronics
• 06-03-20 21:57 Filiberto and Ed done in Optics lab
• 06-03-20 21:57 Filiberto going to MY to help Gerardo
• 06-03-20 21:58 H1 lockloss
• 06-03-20 22:11 Filiberto to LVEA to measure a physical distance on HAM3, out at 21:11
• 06-03-20 22:50 Robert out to LVEA to take pictures as H1 relocks
• 06-03-20 23:28 Robert out of the LVEA
• 06-03-20 23:33 H1 to Observe

Currently:

• no plans for commissioning
• Gerardo and Fil beack from MY
• Tyler and crew on X arm, done for today

Since there has been recent activity adjusting EX electronics to try mitigating some of the narrow lines and combs, I've been trying to check in periodically with the comb tracking page (see H1 summary page "DetChar: Combs" tab). While adjustment work and investigations are still ongoing, I noticed a concerning uptick in the 1 Hz comb prevalence.

Attached is a plot from the comb tracking page that shows an uptick starting February 26. I'm concerned there was some adjustments that have increased some coupling, but I can't point to an exact culprit. Fil's aLOG 55291 from Feb. 25 said there was a change in the 24V supply for the safety chassis. Could this have caused something to change or added anything?

TITLE: 03/06 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 119Mpc
OUTGOING OPERATOR: Jim
CURRENT ENVIRONMENT:
    SEI_CONF state: WINDY
    Wind: 9mph Gusts, 7mph 5min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.55 μm/s
QUICK SUMMARY: H1 locked in Observe

• planned Commissioning starting at 10AM

TITLE: 03/06 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 115Mpc
INCOMING OPERATOR: Jim
SHIFT SUMMARY:

Out of Observing for once during the shift and for both of the locklosses the FSS had no problem relocking.  

NOTE:  If there is an issue with the FSS, protocol is still to call Jason for assistance (Rick is the back up).
LOG:

• 4:16 Lockloss
  • FSS locked right up!
  • Y-arm needed help (prompted by Verbal)
  • Initial DRMI flashes were ugly
  • Lockloss at ENGAGED SOFT LOOPS
  • NLN on 2nd attempt
• 5:39 OBSERVING

As seen in the attached 10-day pressure trend, we had some sort of pressure transient around 2100 hrs. local time on Tuesday.  This doesn't appear to correspond with any of the normal suspects, i.e., a loss of pump speed due to an ion pump HV switching event or from an ambient temperature increase (corresponding VEA temperture compared separately).  Note that the amount of gas needed to cause this trivial increase is of no concern.  Rather, it is the "unexplained" nature of this that is of concern.  These circumstances are similar to past events in which we theorized that ice may have fallen off of the cryopump's inner vessel, evaporated and then re-condensed.   

TITLE: 03/06 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 118Mpc
OUTGOING OPERATOR: Camilla
CURRENT ENVIRONMENT:
    SEI_CONF state: WINDY
    Wind: 2mph Gusts, 1mph 5min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.48 μm/s 

Microseism holding steady from 24hrs ago (just under 90th percentile) & low breezes.
QUICK SUMMARY:

H1's been locked for almost 7 hrs with a range of 118Mpc.  

Still under FSS protocol to phone Jason if we have issues with locking the FSS (Rick is his back-up).

There was a small earthquake this morning that passed through, which I expected to have cause the seismic system to transition to the earthquake state. It didn't, and probably didn't need to, but it did indicate that the seismon test in SEI_ENV was too strict. That test stopped checking whenever the seismon arrival time went to 0, so wouldn't trigger on an earthquake if arrived after the seismon prediction. There's an independent "passive" test that triggers when peak mon goes above 1000 nm/s, but for moderate earthquakes, that could be too late to prevent a lockloss.

I've changed the test to give us a window of 2 minutes after the seismon prediction. So line 49 used to be:

           ezca['SEI-SEISMON_LHO_R35_ARRIVALTIME_TOTAL_SECS_{}'.format(i)] > 0:

is now: 

           ezca['SEI-SEISMON_LHO_R35_ARRIVALTIME_TOTAL_SECS_{}'.format(i)] > -120: