TITLE: 09/22 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 113Mpc
OUTGOING OPERATOR: Patrick
CURRENT ENVIRONMENT:
    SEI_CONF state: WINDY
    Wind: 27mph Gusts, 15mph 5min avg
    Primary useism: 0.05 μm/s
    Secondary useism: 0.14 μm/s

Winds have started picking up a little over the last 2hrs.  Other than that, microseism continues its downward trend to quiet levels.
QUICK SUMMARY:

H1 locked/observing for almost 10hrs.  All looking nice.

As I brought up SEI_CONF & SEISMON windows, I did notice that the SEI_CONF node ALL window has a non-True NOMINAL state (it's outer box for WINDY is orange, not the usual green).  Having trouble tracking down how long it's been "orange". 

Patrick noted that on the ISI_CONF medm, the BRSY has a status of a yellow "DAMPING" (vs a green "READY").

TITLE: 09/22 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 114Mpc
INCOMING OPERATOR: Corey
SHIFT SUMMARY: Remained locked and in observing entire shift. No issues.
LOG:

17:41 UTC Ethan to optics lab
Rahul leading family on tour through control room

Have remained locked and in observing. No issues.

TITLE: 09/22 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 113Mpc
OUTGOING OPERATOR: Niko
CURRENT ENVIRONMENT:
    SEI_CONF state: WINDY
    Wind: 2mph Gusts, 1mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.17 μm/s 
QUICK SUMMARY: No issues.

TITLE: 09/22 Owl Shift 07:00 – 15:00 (00:00-08:00), all times posted in UTC

STATE of H1: Observing

INCOMING OPERATOR: Patrick

SHIFT SUMMARY: Rode through a 5.6 EQ near Tonga early in the shift, then had a sudden lockloss about halfway through. Relocked and in Observing for 1.5 hours.

LOG:

07:00 (00:00) Start of shift

08:04 (01:04) Going to EQ mode for 5.6 near Tonga

08:37 (01:37) Transitioning back to windy.

11:41 (04:41) Sudden lockloss, so sign of seismic or wind activity, no ADS excursions, no rung up ASC channels.

11:57 (04:57) Issues with green arms and finding IR, going into initial alignment

12:18 (05:18) Initial alignment complete, starting re-lock

12:31 (05:31) Lockloss from CARM_TO_TR

12:47 (05:47) Lockloss from ENGAGE_ASC_FOR_FULL_IFO

13:24 (06:24) At NLN, going to Observing. This time I didn’t adjust DRMI to optimize gains, and I waited for a few minutes at PREP_ASC_FOR_FULL_IFO.

15:00 (08:00) End of shift

Ops Shift Transition: 09/22/2019, Owl Shift 07:00 – 15:00 (00:00-08:00) - UTC (PT)

State of H1: Locked

Intent Bit: Observing

Weather: 0-5 mph wind

Primary 0.03 – 0.1Hz: 0.01 um/s

Secondary 0.1 – 0.3Hz: 0.2 um/s

Outgoing Operator: Corey

Quick Summary: Locked and Observing for ~24 hours.

TITLE: 09/21 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 115Mpc
INCOMING OPERATOR: Niko
SHIFT SUMMARY:

Approaching 24hrs of lock/observing.  Microseism is trending down & all is looking well.
LOG:

• Dan B. is on-site working on phase camera in LSB optics lab; out at 4:45.
• 2:09:  Fast EQ from Guatamala alarm received and I immediately switched to EARTHQUAKE just in case at 2:09, too.
  • 3:02 Transitioned back to WINDY 

Had a small EQ (which turned out to be nothing) roll through, but other than that, it's been smooth sailing for H1 this evening with almost 20.5hrs of lock/observing.

TITLE: 09/21 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 19Mpc
OUTGOING OPERATOR: Patrick
CURRENT ENVIRONMENT:
    SEI_CONF state: WINDY
    Wind: 11mph Gusts, 9mph 5min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.32 μm/s

Microseism is making a slight downward trend after its step up 22hrs ago.
QUICK SUMMARY:

• H1's been locked/observing for 16.25hrs.  Had a short recent triple-coincident stretch, but lost L1---they are battling the environment & EY HVAC.
• Continue to have Guardian Message for SZQ Manager node regarding the railed green power stabilization (ops are to ignore this).
• While going through the checksheet, noticed an error for the external Mech Room door at EY (probably the usual comms error); cleared by cycling a "Force" on it.

TITLE: 09/21 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 109Mpc
INCOMING OPERATOR: Corey
SHIFT SUMMARY: Remained locked and observing entire shift. No issues. Bee keeper came to take hives.
LOG:

15:42 UTC SEI_CONF to WINDY
16:37 UTC Bee keeper through gate to retrieve bee hives
16:46 UTC GRB-Short (E351160) 590777160 TRIGGER_DUR:     1.024 [sec] Stand down
16:52 UTC Set INJ_TRANS to INJECT_KILL
17:31 UTC Ethan to optics lab
17:32 UTC Set INJ_TRANS to INJECT_SUCCESS
20:00 UTC SEI_CONF to EARTH_QUAKE for eq from Timor-Leste
21:37 UTC SEI_CONF to WINDY

Have remained locked and in observing. No issues.

TITLE: 09/21 Owl Shift 07:00 – 15:00 (00:00-08:00), all times posted in UTC

STATE of H1: Observing

INCOMING OPERATOR: Patrick

SHIFT SUMMARY: Quiet shift, 5.6 EQ from Albania is currently underway. EQ mode with high microseism seems to cause some large ASC oscillations.

LOG:

07:00 (00:00) Start of shift

07:17 (00:17) Locked at NLN, going into Observing (no sdf diffs)

13:16 (06:16) Kyle to mechanical shop/mid-stations -- adjust bake temperature/loosen bolts

13:45 (06:45) Kyle back from mid-stations

14:29 (07:29) Switching to EQ mode for 5.6 EQ in Albania.

15:00 (08:00) End of shift

TITLE: 09/21 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 112Mpc
OUTGOING OPERATOR: Niko
CURRENT ENVIRONMENT:
    SEI_CONF state: EARTH_QUAKE
    Wind: 4mph Gusts, 2mph 5min avg
    Primary useism: 0.49 μm/s
    Secondary useism: 0.38 μm/s 
QUICK SUMMARY: Riding through eq in Albania.

TITLE: 09/20 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 111Mpc
INCOMING OPERATOR: Niko
SHIFT SUMMARY:

H1 was locked 49hrs before EY had a timing glitch which ended the lock.

Noticed "green power stabilization railed" message for the SQZ Manager.  This is a known issue for the last week & Sheila mentioned operators should ignore this message.
LOG:

• Optics Lab work 23:10-23:29, 0:20-0:47 (Dripta, Ethan & Dan later in the evening)
• 23:21 Begin seeing "green power stabilization railed" messages (in ISC_LOCK log) for SQZ Manager.  Mentioned to Sheila & Daniel.  (Might try to live with it until the fiber can be addressed during the Oct break.)
• 23:38 HAM6 RGA bake-out begins in OSB Mech Room (Kyle, WP#8359)
• 23:55-0:45 Restarted Access/Camera computer (access system was giving error and cameras were only updating every 3-sec).
• 0:49-1:07 MX trip (Kyle)
• 4:19:  Timing glitch takes down EY (after 49 hours of lock).
  • 4:23 Called Dave
  • Frontends look good, so this timing glitch just took down models.  
  • Dave restarted the down-ed models
  • 4:48 Dave calls to say I'm free to start redamping EY stuff & relocking H1.
    • T240s were railed, and this took a bit of time for them to calm down (not seen this before), but they did come off the rails.
    • 4:59 ISI trips again
    • Going to try for DAMPED instead of FULLY ISOLATED.  This was actually easy.  
    • Then went for ISOLATED_DAMPED
    • 5:01 Texted Hugh & Jim.
    • Jim texted instructions and then I called him to confirm:
      • INIT St1 & St2
      • INIT SEI_ETMY
      • SEI_ETMY to ISOLATED_DAMPED
      • SEI_ETMY finally to FULLY ISOLATED
      • SEI_ETMY is back!!
      • Just in case, finally took SEI_CONF to SC_OFF_NOBRSXY and then back to WINDY
      • 5:28 All EY SEI stuff is good to go!
  • 5:28 TMSy, ETMy, HEPIy, ISIy recovered from TRIPs
  • LOCK#1 5:31-5:54
    • ALSy had no flashing.  Restored ETMy & TMSy to slider values from 2hrs ago and this brought back beam.  Then optimized flashes (by adjusting only pitch for ETMy & this locked the arm up easily)
    • Took to PRMI & it locked quickly.
    • DRMI locked locked in 5min
    • Lockloss at DARM to RF
  • LOCK#2 5:55-6:11
    • Noticed zotws3 froze (ops work station), so rebooted it.  (couldn't change any medm screens)
    • DRMI locked in 2min
    • Lockloss at CARM 150 PICOMETERS
  • ALIGNMENT 6:14-6:31
    • SRC Align took multiple times, but it eventually completed.
  • LOCK#3 6:35-6:45
    • DRMI locked in 2min
    • Lockloss at CARM to TR  (note:  first lock after alignment)
  • LOCK#4 6:46-
    • DRMI locked in 2min
    • Handing off to Niko as we are....nevermind, don't wanna jinx!  :)

Ops Shift Transition: 09/21/2019, Owl Shift 07:00 – 15:00 (00:00-08:00) - UTC (PT)

State of H1: Locking

Intent Bit: Commissioning

Weather: 0-5 mph wind

Primary 0.03 – 0.1Hz: 0.01 um/s

Secondary 0.1 – 0.3Hz: 0.3 um/s

Outgoing Operator: Corey

Quick Summary: Locking after EY timing error. At ENGAGE_SOFT_LOOPS.

After Dave's recovery from the Timing Glitch at EY, brought back all the SUS & SEI subsystems, except for SEI_ETMY---it would trip ST 1 & 2 watchdogs when trying to isolate.  I ended up texting Hugh/Jim; Jim walked me through what I should do since I was having issues.  Here are Jim's instructions I followed:

• INIT St1 & St2 guardian nodes  (he originally suggested also the HEPI node, but since it was already isolated, we opted to not touch it this time).
• INIT SEI_ETMY
• Take SEI_ETMY to:  ISOLATED_DAMPED
• Take SEI_ETMY finally to:  FULLY ISOLATED
• SEI_ETMY is back!!
• Just in case, took SEI_CONF to:  SC_OFF_NOBRSXY and then back to WINDY
• 5:28 All EY SEI stuff is good to go!

We have had a timing at EY, which in turn caused a Dolphin glitch stopping the SUS, SEI and ISC models.

Dmesg on all three machines shows ADC timeouts. All IO Chassis are visible, so it looks like we just need to restart all the models.