M. Pirello, R. McCarthy

We measured phase noise on the 3.125MHz source located in the CER using the BluePhase 1000 Phase Noise Test System.  Unfortunately data taken with the SR795 was corrupted on the thumb drive.  I did take notes, though they are not as thourough as the spectrum data.  I have attached a plot of the phase noise measured yesterday on the Tektronix function generator, and compared it to the phase noise measured today (3 datapoints).  Daniel wants more spectrum measured between zero Hz and 10kHz, so we will continue this measurement into next week.  I have attached what we have as of today.

WP8315

TITLE: 08/28 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 118Mpc
OUTGOING OPERATOR: Ed
CURRENT ENVIRONMENT:
    SEI_CONF state: WINDY
    Wind: 9mph Gusts, 5mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.06 μm/s
QUICK SUMMARY:  in Observe 16+ hours, monitoring HEPI pump pressure at EX

TITLE: 08/28 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 119Mpc
INCOMING OPERATOR: Cheryl
SHIFT SUMMARY:

Not much to summarize past my last logs. There was one more GRB alert that was over the trigger duration threshold - not worth mentioning.
LOG:

Swift Alert - E348658 - Trigger Duration 1.024 - H1 already in stand down

Fermi Alert E346568 - Trigger duration =1.024 (very close to the limit for stand down). Initiating stand down. INJ_TRANS to INJECT_KILL - 15min

Other control rooms not responsive on TS.

We are currently monitoring the HEPI pump at EX for what I infer to be an imminent failure. For upcoming shifts, operators should monitor channel H1:HPI-PUMP_EX_CTRL_OUTPUT with ndscope and verify that the level stays in the ~ 6.7 range. If there are any sharp fluctuations up or down this may result in a lockloss that is unrecoverable from ops. The instructions, as I understand them, should this occur is  as follows: From the SITEMAP>X-ARM -HEPI>Beckhoff Pump Controller>HPI_PUMP_EX_CUSTOM.adl screen select 'MANUAL' from the REMOTE side (right half) of the screen and change the "TWEAK SIZE" to "1". Then use the minus(-) button below the output value display to ramp the SETPOINT to(or close to) "0". Hugh should then be notified. 

TITLE: 08/28 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 118Mpc
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
    SEI_CONF state: WINDY
    Wind: 7mph Gusts, 5mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.06 μm/s
QUICK SUMMARY:

Currently in standown from Super Event S190828L

TITLE: 08/27 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 117Mpc
INCOMING OPERATOR: Ed
SHIFT SUMMARY:

EX HEPI Pump:  Want to monitor the output for this pump (see Hugh's alog) since it sounds like this pump is having issues.  I passed on to Ed what channel to look at for this.  If this pump happens to crash, Hugh will need to address (possibly swap out) the pump.

EY BRS:  This was balanced yesterday.  Jim mentioned we should also be mindful of this since there is the possibility it could drift out of range---he was guessing we should be fine, but just gave us a heads up about it.
LOG:

• 23:20 Out of Optics Lab (PeterK)
• Noticed a couple of doors open at EY (so I forced them shut...similar to yesterday)
• 0:05 Verbal for 6.8 EQ
  • 0:30 H1 to EARTHQUAKE state
  • 1:54 Transition back to WINDY state
  • 1:58 Transitioned back to EARTHQUAKE state since it was still seismically noisy from the earthquake.
  • 3:08 Transition back to WINDY state again
• Last hour of shift was lively due to a couple of candidates which rolled in:  S190828j & S190828l.

H1 rode through a 6.6 EQ & just recently transitioned back to the WINDY state. 

We now are back to triple coincidence as well.

H1 was was able to ride through a 6.6EQ (it was transitioned to the EARTHQUAKE state).  Here is a timeline:

0:05 Verbal for 6.8 EQ (was later listed as 6.6)

• ~0:15 EQ begins to be show up on ndscope, tidal strip tool, and seis blrms.
• 0:22 L1 goes down
• 0:30 H1 to EARTHQUAKE state
• 0:32 V1 goes down
• 1:06 R-wave begins to roll through as predicted
• 1:35 R-wave mostly gone
• 1:54 Transition back to WINDY state
• 1:58 Transitioned back to EARTHQUAKE since H1 started to have many glitches, tidal signals increased and ASC Control signals increased....will give it more time.

Attached is a screenshot of the desktop after the R-wave rolled through.

 J. Jones,  J. Kissel, P. Thomas

Some notes on recovery today:
- After lock loss for unknown reasons at 2019-08-27 14:38 UTC (see LHO aLOG 51536), we proceeded with maintenance day as planned. Note, several plans involved using the IFO during maintenance (i.e. it would stay up in to maintenance period, and maintenance/commissioning would be performed with the full IFO), but could not proceed due to the unknown lock loss.

- As usual, ground motion sensor correction for the seismic isolation systems (which includes input from BRS) was turned off site wide, as a is normal practice to avoid the cleaning crew and others impacting chamber motion by ringing up/exciting ground sensors.

- The following *unplanned* things happened:
    (a) Early in the morning, H1 ETMX HEPI Pump Tripped. Took a long time to recover, sucking up ETMX from use with the IFO, slowing recovery.
    (b) Mid-morning, TCSY laser tripped. Short time to recover, did not impact recovery.
    (c) Mid-morning, planned activity on BRS EY went south [only diagnosably so by late morning], needed further adjustment late in the morning

- The recovery began at 18:43 UTC [11:34 PDT], (still with all sensor correction OFF), when activity ceased enough that I could begin Initial alignment in green on Y-ARM (IMC was already up, locked, stable, and running). 
    - I knew that gym was done with his BRS work at EY, and that initial alignment of the green arms takes ~10-20 minutes, so I wanted to launch at least some of the long poles. At this point, we still didn't have end-X.
    - Instead of engaging the automated initial alignment system as per normal, I
        - took the ISC_LOCK guardian to INITIAL_ALIGNMENT, which used ALIGN_IFO to begin configuring the IFO for initial alignment, and triggered the start of the INIT_ALIGN automatic initial alignment sequence.
        - HOWEVER, I think switch the INIT_ALIGN guardian to AUTO, so I was the manager, and held it on PREP_FOR_GREEN, let it finish, then brought the ALS_YARM guardian (normally managed by INIT_ALIGN) to INITIAL_ALIGNMENT.
        - Didn't work, because (i) the arm motion was a little too high, and (b) the alignment of the ETM / TMS was such that the flashes were too low.
        - Requested ALS_YARM to UNLOCKED, manually touched up the alignment of ETM/TMS to get green flashes high,
        - Re-requested ALS_YARM to INITIAL_ALIGNMENT, and with success held there, in holding pattern waiting for ETMX to be recovered.

- 19:52 UTC 
    - Hugh, Jeff B and Richard M recovered HEPI operation with a temporary solution (i.e. fixed (a) from above), and brought ETMX seismic platform back to nominal configuration. 
    - Also configured site-wide system sensory correction "ON", except for BRS Y, which is still off.
    - Began initial alignment of X-arm with some confusion about how to bring INIT_ALIGN guardian to the right state vs. management of ALS_XARM, ended up requesting INITIAL_ALIGNMENT_COMPLETE, which essentially restarted everything and lost progress on Y-ARM. Should have requested "GREEN_ARMS_OFFLOADED" from ALS_YARM before pushing forward with the INIT_ALIGN guardian.
   
- 20:19 UTC 
    - Both green arms aligned and offloaded. On the rest of initial alignment (which no longer needed the arms), thus
    - Launched Jim traveling to EY to continue addressing EY BRS (i.e. (c) from above)
    - brought site-wide observatory sensor correction configuration to WINDY_NOBRSY (i.e. nominal for everyone but ETMY, which could not yet use the currently non-functional BRS EY). Jim manually further modified that end station's sensor correction configuration
    
- 20:35 UTC
    - Problems with MICH_BRIGHT, in that repeated attempts to lock flip-flopped (as it should) ALIGN_IFO between MICH_BRIGHT_LOCKED and DOWN in order to stop oscillations of MICH_BRIGHT. Tried holding in DOWN, touching up the beam splitter in pitch using AS_B and the AS camera as my metric, re-requesting MICH_BRIGHT_LOCKED and the oscillations continued.... 

- 20:40 UTC
    - .... until they didn't. Problem resolved itself. Jenne suspects that there is still tuning to do with optical lever damping that wasn't coded up in guardian after last week's commissioning of the same issue (see LHO aLOG 51409)

- 20:46 UTC
    - Rest of initial alignment went automatically and completely.

- 21:48 UTC
    - Back to NOMINAL_LOW_NOISE -- once back on the normal lock acquisition sequence, post-initial alignment, the acquisition sequence was entire automatic, save for one Patrick flip (instead of automated flip) from DRMI lock acquisition to PRMI acquisition.

(Went into commissioning PRCL Feed-Forward activity instead of directly in to observing. During that commissioning, Jim deemed BRS EY fixed, and we re-engaged it by requesting site-wide sensor correction configuration to WINDY).

Total time between lockloss and regaining NOMINAL_LOW_NOISE: 7.17 hours.
Total time between recovery start and finish: 3.08 hours.
Delay due to unplanned FAULTS (all on HEPI EX): 1.15 hours.
Issues that impacted recovery time as a result of planned Maintenance Day activity: None (-ish***). 

(***You could claim that the extra BRS EY activity "impacted" recovery, but really not, since we were able dis-engage it from forward progress and parallelize its recovery and the lock acquisition sequence).
 

Summary--HEPI is running and the platform is isolated but, the Pump Station pump could be on its last legs.  The motor is now turning at 3x the previous speed to achieve nominal pressures; and, based on the PID controller, the plant is different.

Cronological Details [edits in brackets]

1531utc (0831pdt) HEPI Tripped [there was activity in the LVEA at this time]; JeffB entered the EndX Mech Room, from the VEA at 1533 (percomm RichardM.)  Jeff recorded HEPI Reservoir Fluid level (after doing Dust Monitor Pump checks) after the HEPI had tripped based on the level he recorded.  It is very noisy in this space and he did not realize when he read the reservoir level that the pump was not running.  Based on conditions later found, this first trip was a fluid level trip.

Almost an hour later, JeffB and JimW initiated restarting the pump station.  This was not successful mainly likely because the fluid level was still low and would trip before reaching nominal pressures.  After these operators had reached their maximum cpu output, I was called in.  Observation and test revealed the fluid level was in fact low and this was mitigated by adding ~1/4" more fluid to the revervoir. [The accumulator pressures were checked and a thorough look for fluid leaks was performed--accumulators were fine  and no leaks were found.]

Once fluid level was comfortably above trip level, the drive was manually run up to achieve nominal differential pressure.  At this point, in hind site with trends to review, the return pressure was running low almost 10% and the supply pressure was ~nominal but much noisier.  Further, at this point the output of the pump station was running almost 15psi higher than normal while the pump was running at close to nominal speed!  Does this mean that something was obstructing the flow bewteen the pump station and the chamber pressure sensors?

After running like this for about 10 minutes and meanwhile having swept the VEA, JeffB and I were about the drive away, Patrick informed us the Platform had tripped again.  The trends reveal that at this time§, the supply pressure (at the chamber) spiked almost 15 psi (return pressure did as well.)  Of course this spiked the differential pressure and the PID drove the output towards zero.  Within 5 seconds, the pressures drop and the PID turns the output around and it now maxes out at long established output limits.  blah blah...

At this point RichardM checked the power supply confirming we still had all phases and we also checked that the rotational speed of the motor matched the VFD output.  So it seems the pump is not now able to pressure the system as before.  Once we figured out Patrick's PID controller we got the output pressures nominal and got the SEI platforms isolated.  But, to achieve this pressure, the motor/pump is now spinning +3x faster than before. Pressures are ~nominal and no more or less noisy than they were before, except, the first two pressure points after the pump is down 0.4  and 0.2psi suggesting with a stretch that some obstruction cleared...

Attached are two graphs, the first is 6 hours from before the trip to final operation state and the second is 20 seconds around the big change point discussed above §.  I'll take some time to annotate these graphs but for now I'll get these into the log.

[While the motor/pump here is now running 3x faster than before, it is still only running at 2/3 maximum.  For reference, the corner station pumps all run at ~1/2 maximum. So, there should be no issues until the pump falls apart. Comforting, right!]

TITLE: 08/27 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 118Mpc
OUTGOING OPERATOR: Cheryl
CURRENT ENVIRONMENT:
    SEI_CONF state: WINDY
    Wind: 8mph Gusts, 6mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.07 μm/s

Microseism continues it's trend down for the last 48hrs.
QUICK SUMMARY:

Arrived to an H1 in Observing after a Maintenance Day (Yay!), and Patrick mentioned it wasn't too bad getting us back with JeffK doing an Initial Alignment  and Patrick getting us back to NLN.

Operator Note:  Jim mentioned BRS-Y rebalancing work today and possibility it could drift out of range (If that happens, ISI will trip & we will want to take SEI_CONF to WINDY_NO_BRSY).

Check of HEPI Pump Stations this morning: 
   (1). Corner Station fluid level = 6 6/16, change = -1/16
   No new leaks or drips observed

   (2). End-Y fluid level = 9 0/16, change = 0/16
   No new leaks or drips observed

   (3). End-X fluid level = 8 11/16, change = +3/16 
   No new leaks or drips observed
   The increase in fluid level was due to the pump having tripped off 2 minutes before I entered the building. Hugh and I worked on recovering the pump station. The fluid level was actually down and hitting the trip levels. We added 4/16" of fluid to the reservoir. This appeared to bring the running level to 8 6/16. Need to let it run for a bit to get a good fluid level reading. The current trip level is between 8 5/16 and 8 4/16.  

Hugh is writing an aLOG on the incident.