R Schofield WP#8575

TITLE: 03/25 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: OBSERVING
INCOMING OPERATOR: Ed
SHIFT SUMMARY:

Nice shift with H1 holding lock since just after 1am (locally) & Observing that whole time.  Will be dropping out of Observing tonight so Robert can run some close-out measurements on H1 during the EVE shift.
LOG:

• 14:53 GRB Long, confirmed w/ LLO operator (they were calibrating), & Trigger Duration was too long.  IGNORING
  • Richard acknowledged the alarm in the Control Room
• 17:36 Incoming EQ (5.1 aftershock from Russia), but looks like we'll ride through it.
  • 20:11 Similar one as above and a non-issue.

TITLE: 03/25 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 119Mpc
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
    SEI_CONF state: WINDY
    Wind: 6mph Gusts, 2mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.23 μm/s
QUICK SUMMARY:

Communicated with Corey for handoff

• Robert is on site taking measurements
  • I will be coordinating remotely via Teamspeak

Smooth running with H1 locked almost 13hrs.  Had a little bit of wind this morning, but nothing too tough for Observing.  Microseism is low--just above the 50th percentile.

Richard M. was able to check the Chillers for me.  Crystal Chiller was right in between Min & Max, so he added 150mL.

TITLE: 03/25 Day Shift: 15:00-23:00 UTC (8:00-16:00 PST), all times posted in UTC
STATE of H1: CALIBRATION
OUTGOING OPERATOR: Cheryl
CURRENT ENVIRONMENT:
    SEI_CONF state: WINDY
    Wind: 29mph Gusts, 3mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.21 μm/s
QUICK SUMMARY:

• (Operating remotely.)
• H1's been in Observing for about 8hrs after the big Russia EQ last night & there were no issues overnight.
• On nuc28, while reloading the browser for GWIstat, I do see the Lock Clock is missing.
• Called into the Control Room & chatted with Richard for status.
• All else looks well.

Checked the dust monitor vacuum pumps - Tuesday 03/24/20

   End-X - Slight tweak in the air bypass valve to adjust vacuum pressure. Filter is clean. All looks and sound normal.
   End-Y - Slight tweak in the air bypass valve to adjust vacuum pressure. Filter is clean. All looks and sounds normal. 
   Corner Station - Air bypass valve fully is closed (No additional air to set vacuum pressure). However, at 16.5inHg, the the vacuum pressure is below normal operating pressure of 18.0inHg. This would indicate a leak somewhere in the LVEA or the pump has an internal problem. When the pump was overhauled, not that long ago, there were no signs of internal damage noted. The pump passed its post rebuild pressure checks. 
   Given all the work going on in the LVEA at this time, the possibility of damage to one of the vacuum lines is not out of the question. Will do an inspection of the LVEA vacuum lines as soon as I can get into the LVEA. 

Closing FAMIS #13009     

 Checked the HEPI pump stations on 03/24/20.

   CS - Level 6 4/16 - Down 1/16  from last check
   EY - Level 8 14/16 - No change from last check
   EX - Level 7 13/16 - No change from last check

 Did not observe any new or increasing leaks. All platforms looked normal.

Close FAMIS #13499

Checked both chillers on 03/24/20. 

   TCS-Y - Level at 30.0 - Did not add any water
   TCS-X - Level at 10.0 - Did not add any water

   All looked normal.

Closing FAMIS #11536 

TITLE: 03/25 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 118Mpc
INCOMING OPERATOR: Cheryl
SHIFT SUMMARY:

• First half of the shift was quiet
• EQ from Japoan/Russia area made for a tough 2.5 hours of down time
• Got my first remote re-lock!

LOG:

5:17 recovered from Very Large Earthquake and untripped ISI's

5:27 Started relocking at about .8-.9 um/sec

Locklosses @: 

• Locking ALS
• Offload DRMI ASC

Third attempt the green WFS for Y arm weren't engaging despite the arm locking.

• I noticed the fiber polarization was at the limit for that arm - corrected polarization for both arms

07:10 H1 back to OBSERVING

Just checking in.

H1 up and observing....

While i was writing this...

• 7.8mag EQ from Japan/Russia area broke lock before R3.5 waves arrived.
  • I called Jim for a possible preemptive course of action whereby I would break the lock intentionally and set the "Big Red Button". 
  • Lockloss occurred @ 2:58 UTC before I could even ask the question
• SEI_ENV placed in Maintenance Mode and Sensor correction is OFF
• ISC_LOCK is holding in DOWN
• Obervatory mode is in Environment - EARTHQUAKE
• SUS: OF1; ZM2; TMSY; TMSX untripped
  • There was continuing tripping of ISIs but only 5 of them tripped
    • EY; BS; HAM5; ITMX; EX
• IMC has relocked
• EQ FOM shows about 225 microns on the Z axis! 

Guess I'm waiting this one out. Screenshot doesn't want to work for me tonight.

TITLE: 03/24 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: OBSERVING
INCOMING OPERATOR: Ed
SHIFT SUMMARY:

Light activities on-site, but with that said, we had (2) LN2 trucks, tumbleweed burning by the Fire Dept, and landscaping contractor visiting the site.  H1 operation was fairly smooth with most of it being controlled remotely.  Attached is the Reservation System items from the day.
LOG:

• See Maintenance Log for activity covering from 15:00 - 20:00
• Hanford FD was onsite for tumble weed burning (from Bubba email).
• 20:28 Adding Niko to remote alert system (affects H1 up time briefly, but will be restored) [Dave]
• ~21:00 Logged into new NoMachine remote server per Jonathan H. email.
• 21:30 nuc24 rebooted by Richard M. (40min ndscope was froze).  He also rebooted the DARM spectra computer (nuc30) earlier in the day because that was froze.

TITLE: 03/24 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 116Mpc
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
    SEI_CONF state: WINDY
    Wind: 20mph Gusts, 16mph 5min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.24 μm/s
QUICK SUMMARY:

at 13:28 PDT. Tested alerts are now being sent to Niko.

WP8572 h1tw0 raw minute trend data offload

Dave:

today I completed the h1tw0 offload process which I had started last Friday. Today at 09:55 PDT I restarted h1nds0 (IFO was unlocked) with its new daqdrc configuration serving the past 9 months of minute trends from their final archive location. I was then able to delete the 256,674 old files from h1tw0, taking its SSD raid usage from 94% down to 2% (this took just over 2 hours). h1tw0 is now good for another 9 months.

Maintenance Summary:

Operating H1 on this Maintenance Morning remotely (used Reservation System to display activities on H1 I was aware of, logged into TeamSpeak in a room ("LHO COMM/OPS") with LHO staff, and had phone/text contact with reduced staff on-site, mainly Richard & Gerardo).

Overall for H1, it was a light day with our current LIGO Lab Operational Situation.  Kept H1 locked as long as we could.  When it did drop a main activity was reviewing/updating the SDF for all relevant subsystems.  There was commissioning work for HAM SEI Feed Forward.  And Maintenance was pretty much over just after noon (with commissioning work (Jenne) and activity on the site [relayed by Richard M via text] only lasting a few minutes beyond noon).  Locking was straightforward with only really two attempts needed to get back to NLN.  Below is the specific log of activities as  I recorded them.

LOG:

• 14:52-14:59 PEM Injection
• 15:00 Maintenance
• Gerardo VAC work all over each End Station
  • 15:19-15:38 EX visit.  (BRS kept "ON")
  • 15:48-16:06 EY visit.  (BRS kept "ON")
• 16:00 "TEST T369506" acknowledged
• 16:05 Shaking HAMs for SEI Feedforward work (Jenne)
• 16:13 Talked/check up with Richard M who is in Control Room.  He acknowledged the "TEST" on Verbal.  Gerardo just arrived back from EY.
• 16:28 Lockloss
  • Jenne had excitations which I heard say was probably due to lockloss.
  • Richard said we had (2) LN trucks on site at the time (parked)
• 16:29-17:20  Holding ISC LOCK in READY state, for SDF snapshots work (Sheila, Jenne, Jeff, Jim, Rahul, etc...)
• 16:39 Received "GRB-Long", but out of lock.  Also confirmed with LLO.
• 17:20 Begin locking to DRMI.
  • Green arms were both off in yaw & needed several rounds of INCREASE FLASHES
  • 17:37 lockloss
  • 17:53 @OFFLOAD DRMI ASC & handed back to jenne for her HAM sesimc feedforward work.
  • 19:06 HAM seismic feedforward work comlete (Jenne) & moved on to NLN.
  • 19:09 Lockloss at RESONANCE
  • 19:46 NLN & 19:48 OBSERVING!

owl_op is now replaced by remote_op, which sets up the operator's LLA settings for any of the three shifts (owl, day and eve).

usage: remote_op [-h] [-s SHIFT] [-d] name

setup lock-loss alerting parameters for remote operator

positional arguments:
  name                  operator's ligo.org name

optional arguments:
  -h, --help            show this help message and exit
  -s SHIFT, --shift SHIFT
                        Shift [owl, day, eve]
  -d, --disable         Disable the alerts for the listed person

Example:

Setup john.doe's evening shift

remote_op -s eve john.doe

Disable alerts for john.doe

remote_op -d john.doe
 

I reset both power watchdogs at 17:38 UTC (10:38 PDT).  This completes FAMIS 10755.

The TCSY rotation stage was cycling through adjusting power, so I did the usual fix of slightly tweaking the calibration. I changed the power in value from 3.8W to 3.75W 3.65W. One day we will need to get the calibration correct with a full search with this rotation stage.

In SDF there the H1:TCS-TCSY_LASERPOWER_COMMAND channel was flickering as the rot. stage was rerequesting to change powers. I unmonitored this channel.

EDIT: spoke too soon, changed to 3.65W.