TITLE: 08/30 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 119Mpc
INCOMING OPERATOR: Cheryl
SHIFT SUMMARY: Remained locked and in observing entire shift. Just noticed the DIAG MAIN message: TIDAL_LIMITS: Y_TIDAL_CTRL ouput within 10 percent of limit
LOG:

15:26 UTC Tyler to optics lab
16:09 - 16:20 UTC Niko and Ethan to optics lab
16:21 - 16:44 UTC Chandra to mid Y
17:05 UTC Niko and Ethan to optics lab

Expecting a light Tuesday maintenance. Only known activities to date:

OPS: PEM INJ @ 7:45 AM
OPS/DETENG: Reconcile safe snaps
Richard: EX green ALS camera housing replacement
Rahul: ETM charge measurements
VAC: VE cutting/drilling in LVEA

Since Monday is a holiday, all else will be dealt with on Tuesday.

(Related to FRS12067)

I wanted to do another investigation of the rate of connection errors recently, and how they compare to before and after h1guardian1 was upgraded to a 40core machine [alog46464]. This is just a quick look at the connection errors in ISC_LOCK from the last tow months. A further study will follow at some other time.

In the past I had determined that the connection errors had severely decreased post machine upgrade, and that ISC_LOCK was the biggest offender for connection errors (sorry no link at the moment). This is also seen by simply trending the CONNECT channel for ISC_LOCK (1st attachment). You can see that after the machine upgrade on January 16, 2019, the rate was reduced and after the start of O3 it was brought down even further. The problem is that they are not completely gone, so I looked at the last 2 months of connection errors from ISC_LOCK to determine where/when these happen.

Out of the 20 connection error instances, 11 were true connection errors. Most of these are from restarts on maintenance days, the others are events the lightning strike, dolphin crash, or similar.

The other 9 all occur in the state CHECK_MICH_FRINGES (48). It is always at the same line:

        ezca.get_LIGOFilter('SUS-BS_M3_LOCK_L').only_on('INPUT','FM1', 'FM2','OUTPUT', 'DECIMATION')

There is nothing special about this, these only_on LIGOFilter calls are all over ISC_LOCK. Yesterday I toggled that filter bank to see if for some very odd reason, it was the bank causing trouble. Nothing abnormal. I then ran this state manually a few times yesterday and during the lock reacquisition, and I still didn't see anything (no connection errors or abnormal machine loads).

These sparse connection errors don't seem to cause any trouble, and they are very brief (see log attachment 2). I will continue to monitor these to see if their rate increases or if they become problematic.

Have remained locked and in observing. No issues.

TITLE: 08/30 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 118Mpc
OUTGOING OPERATOR: Jeff
CURRENT ENVIRONMENT:
    SEI_CONF state: WINDY
    Wind: 15mph Gusts, 9mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.06 μm/s 
QUICK SUMMARY: No issues.

   No issues or problems to report at the mid point in this shift. 
   IFO remains locked and observing. Environmental conditions are good.  

TITLE: 08/29 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Preventive Maintenance
INCOMING OPERATOR: Ed
SHIFT SUMMARY:

Took the first half of the shift to finish Thursday Maintenance Full Day & then about 2.5hrs to recover from it (which honestly wasn't very bad considering--H1 wanted to get back into Observing!).

NOTE:  Teamspeak is down due to planned power outtage at MIT.  Should be back around 4am PDT.  So in the meantime phone other Control Rooms, when needed.

LOG:

• LVEA transitioned to Hazard and swept (Corey)
• 0:30 1st gate valve closed (GV15)
• 0:34 2nd gate valve closed (GV14)
• 0:34-1:56 Initial Alignment begins
  • 0:42 Paused Green arm locking
    • 0:44-1:17 Sheila/Keita touch up COMM Beat note out on ISCT1 (needed after PeterK's ALS path work on the PSL earlier today)
• 1:57 Lock #1
  • Tried CHECK_MICH_FRINGES because TJ was still here and wanted to see how it looks nowadays.
  • For CHECK MICH FRINGES, my experience (last month) was that we want a dark & symmetric spot made by moving the BS
  • Tonight, we had a bright spot on AS AIR.  This looked new.  So, I did not touch the BS & we just moved on.  
• 2:24 GRB received (we were not Observing)--discovered Teamspeak was down, so called LLO.
• 2:55 Back to Observing
• 5:00-5:02 TCSy CO2 laser drops us out of observing.
• 6:07 VERBAL EQ notification for Atlantic EQ of magnitude 5.7 (but i'm seeing it listed as 5.2)
  • 6:08 Transition to EARTHQUAKE
  • 6:54 Back to Windy
• 6:24 GRB notice received from FERMI & will Stand Down until 6:39  (phoned LLO to let them know & update them on power outage affecting Teamspeak)

5:00:02 - 5:01:50 utc Out of Observing

Looks like the laser unlocked.  Didn't see anything in SDF, but this node came up in the "NOT OK" Guardian Node List on the h1 Intent Bit.

Here's the initial note in the Log for this Node:

2019-08-30_05:00:02.397615Z TCS_ITMY_CO2 [LASER_UP.run] laser unlocked. jumping to find new locking point

And here's where it locked up on its own:

2019-08-30_05:01:18.702604Z TCS_ITMY_CO2 executing state: LASER_UP (-15

C. Gray, S. Dwyer, K. Kawabe, J. Kissel, P. Thomas, B. Weaver

After a hefty, special thursday maintenance day induced by the need to replace EX's HEPI pump station, recovery went rather smoothly. 
The major time sucks after the HEPI Pump Station was replaced and ETMX SEI system was completely restored (by 21:51 UTC, i.e. ~03:00 pm PDT, see LHO aLOG 51617):
    (1) the troubles the vacuum crew had in chasing down a sneaky leak, LHO aLOG 51624 [unavoidable commissioning of a new vacuum system component; extra 2.5 hrs]
    (2) the recovery of the COMM beatnote (LHO aLOG 51626) as a result of the change to PSL ALS path (LHO aLOG 51610) [0.5 hrs]
    (3) Initial alignment troubles with PRC and MICH (see LHO aLOG 51625), saturating suspensions. Both were fixed by just trying a 6th time. Probably should implement the suggested further changes in the last line of LHO aLOG 51409 [0.5 hrs]
    (4) Confusion about what to do in the CHECK_MICH_FRINGES state after recent commissioning work by Jenne, since MICH loop is on but not triggered. Banter about "just make flashes symmetric" or (the way it used to be when that state locked on a dark fringe) "minimize light as AS port, and make dark pattern symmetric. We didn't need to go through this state anyways, just wanted to check load on guardian computer during this state. [0.17 hrs]

I want to call out the brief outage of the LSC computer's model crash (LHO aLOG 51616) miraculously gave us no delay -- we were waiting for (2) anyway, and we now know that all settings apparently came back excellent (and our were under control by the ISC_LOCK guardian and its subordinates).

So -- even after 
     - the GIANT local earthquake this morning (LHO aLOG 51601) 
     - the sputtery start to initial alignment (EX recovery was a breeze, just a couple of small, normal-sized tweaks to ETMX and TMSX, and ALS XARM locked up beautifully), 
     - with the recovered COMM beatnote we sailed through LOCKING_ALS, 
     - PRMI acquisition, its ASC, the transition to DRMI, and its ASC all working perfectly, 
     - no lock loss at CARM_ON_TR b/c of the recent force of all subordinate IFO nodes to go *through* their down state (LHO aLOG 51465), 
     - a slow but sure trek through ENGAGE_SOFT_LOOPS, 
     - *zero* problems with violin modes before engaging DC readout, 
     - a smooth increase in power and all subsequent noise tunings, and 
We're at NOMINAL_LOW_NOISE with no striking setting differences.

Excellent work All (both today, and leading up to today)!

I would say interferometric recovery began at 00:30 UTC (17:30 PDT), and we hit NOMINAL_LOW_NOISE at 02:52 UTC (19:52 PDT), so recovery time is 2.3 hours, not so very much different than a "standard" run through an initial alignment + lock acquisition sequence these days as we wait for ASC loops to converge. But as highlighted above -- it makes for a *very* robust acquisition sequence, so it might be worth the wait.

Attached is a more detailed blow-by-blow, in case needed for future reference.

Jeff is compiling a thorough summary of today's recovery after being down for 9.5hrs due today's Planned Corrective Maintenance.

At 2:55utc (7:55pm PDT), H1 was put back to OBSERVING!  (So roughly 2.5hrs needed for recovery once the MX gate valves were opened.)

Time for lunch!

J. Kissel for S. Dwyer

As we were hovering in the ENGAGE_SOFT_LOOPS step for ~10 minutes as it creeps through the 4 spot positions in which we demand convergence of the ADS loops, we got impatient. Sheila loosened the threshold on the convergence checker for these loops (appearing on line 3032 of ISC_LOCK) from 3.0 to 6.0.
We'll see how this works for several lock stretches, and/or consider even looser thresholds on the points between initial alignment position and final position.

H1 continues with is recovery, but while doing this, we received a GRB notification.  One of our first steps in protocol for GRB notifications is to contact LLO & Virgo to make sure they also received the notification, but I noticed the Operator workstation with the TeamSpeak application was totally blank.  I tried to reconnect and received a "failure to connect to server" error.  Thinking the computer was down, I rebooted the computer.  

While waiting for reboot, I phoned LLO, and Joe Hanson confirmed he also received the alarm, but also noticed his Teamspeak was also not connected.  (not sure who we contact about this, so thought I'd atleast alog it.)

TJ, Sheila, Jeff K Corey Keita

We were seeing the typical difficulties with MICH_BRIGHT not locking (see Jenne's investigations of the cause of the problem here 51409 ) When ALIGN_IFO recognized that MICH isn't locked, it goes to DOWN, but then immediately tried to relock, when the beam splitter is still swinging too fast.  TJ and I edited INIT ALIGN so that it will set the request of ALIGN_IFO to DOWN in this case and wait 30 seconds before re-requesting MICH_BRIGHT.  

This doesn't solve the problem with the MICH locking, it just prevents us from pointlessly cycling between DOWN and MICH locking while the BS is still swinging too much. 

{Kyle, Gerardo, Chandra}

Today we capitalized on short-notice unscheduled maintenance day to remove and replace a legacy turbo station at mid-X station. Removal and installation went smoothly and we were leak checking by 3:15 pm when we discovered a large leak at the 12" flange of CETEC gate (turbo side). We replaced the gasket but flange still leaks at a rate of e-3 Torr-L/s. We concluded that the seal weld on turbo-side GV flange is cracked from bottoming out studs via double nutting, due to poor GV design. The tapped holes in flanges go all the way through the flange and the flange is weakly stitch welded to the GV body (see attached photo). Side note - Gerardo found that the 14" CETEC valves on ion pumps have partially threaded thru holes to prevent this from happening.

We will need to vent the MX spool at next opportunity to replace the CETEC valve. Meanwhile, the new turbo is bolted to the valve and volume is isolated via foreline safety valve. We are currently more vulnerable than normal operation with near atmospheric pressure on the other side of turbo isolation valve. The good news is the o-ring is seated in favorable direction so that atmospheric pressure pushes it in closed direction.

Note that we have replaced all legacy CETEC valves on site except at midstations' turbo and ion pumps (and EY turbo which will be replaced next time we vent EY).

While working in the CER rack ISC-C3, I noticed that that 24V LED on the rear panel of both the 35.5MHz and the 80MHz RF Oscillator Source's were not illuminated while the 9.1Mhz, 25.5Mhz and the rest of the sources all have illuminated 24V LED's so I measured the 35.5MHz and 80MHz signals out of their respective RF Distribution Amplifiers and they look good therefore this may be a non issue, but it seemed worth noting.

Prior to the installation of the pick off, the power in the ALS beam was measured to be 1.054 +/- 0.001 W in front of
ALS-M2.  Installed a zero order, half waveplate and polarising beamsplitter cube in the beam path.  Adjusted the
half waveplate to measure 1.040 W in front of ALS-M2.  The powers were measured with an Ophir 30A-BB-18 power
meter (S/N 805795).

    At present there is an HR mirror directing the beam to where the fibre would be.  This optic should be replaced
with an optic that has a higher transmission (5-10% perhaps) to allow for a monitoring photodiode.  I found such an
optic in the Optics Lab (unfortunately only after I exited the PSL Enclosure).

distances:
ALS-M1 to half waveplate: 29"
half waveplate to polarising beamsplitter cube: 3"
beamsplitter cube to mirror: 9"

    There is a lens in the ALS beam, labelled LaserOptik R=75 mm 532 nm + 1064nm.  Located ~20" from ALS-M1.