• H1 in SRC when I took over
• ushered H1 into NLN
• optimized IM1, IM2, and IM3 alignments, as much as I could in an hour
• there's more to do, but LLO came back up so I put H1 in Observe
• ran a2l, pitch coherence was 0.9+
• accepted a lot of SDFs, will attach
• H1 range peaked at 72.7Mpc, before Hanford noise started
• H1 range currently at 71.3Mpc

TITLE: 05/05 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Other
INCOMING OPERATOR: Cheryl
SHIFT SUMMARY: A large lightning storm swept through the area and caused numerous power glitches to the site. Since then it has been an effort to recover. Just to make things a bit more hectic, we had a 5.8M earthquake in Tajikastan to shake up the recovery effort. I unfortunately didn't write down times for th erecovery process, but it went something like this:

• Power glitch caused the lock loss and tripped the PSL and the IOPISC end station models.
• Dave was quick on the phone and restarted the models.
• Attempts to recover from here got worse when all of the front ends went down.
• Many calls were made, high voltage was recovered on their power supplies, and so many watch dogs were reset and then re-reset.
• At one point I had to get Patrick on the phone to help get the slow controls at end X going again. A quick remote login and restart were all it needed luckily.
• After Jason got the PSL up and Dave did a few more model restarts, We eventually got to the point where I could maybe think about locking.
• I then had to run down to EX to turn on the HV for the ESD and now I have a horribly misaligned IFO to straighten out.

Thank you for the help of Dave Barker, Jason Oberling, Patrick Thomas, Richard McCarthy, and Keita Kawabe!

I have restored all suspensions back to where they were during our last lock and made it through initial alignment up to SRC align and without any major issues. Now handing off to Cheryl.

LOG:

• 02:07 Turned on fan in MSR and propped open the doors.
• 03:02 Lock loss due to power glitch from lightning.

I managed to clear the DAQ errors for h1pemm[x,y] by restarting the IOP models way more than should be necessary for it to latch to GPS time via EPICS. I suspect its EPICS GPS source was having issues earlier. Anyhow, all DAQ data is now good.

Greg checked on the LDAS and GDS systems, he found no problems. We should be good to go to observation mode when ready.

I checked the HEPI pump controller computers, they did not reboot and seem to have rode through the whole thing. BTW, they are running a 2.6.18 kernel, which is prior to the day208bug.

In summary, I don't think we need to do anything more at this point.

We think the worst of the storm may be over. I'll let TJ fill in the details, I know of at least three glitches which took down front end models. The first just took out the ISC models at the end stations (suspect timing card power supplies), the second (at 8:40pm) was the biggest and it rebooted every front end computer. After that most front ends came back up after the reboots, h1ioppsl0 had a timing excursion which took about 15 minutes to clear. At that point the only problem was bad DAQ data from both PEM systems at the midstations. A further glitch caused h1ioplsc0 to DAC error, I restarted all the LSC models to clear this.

The Beckhoff slow control machines also rebooted. All code restarted except for the three PLCs at EX. These have just been started.

At time of writing, the only computer issues I see is the bad data from the midstation PEM front end computers to the DAQ. If memory serves, these send their DAQ data back to the end stations via ethernet-fibre_optics converters, and then piggy-back on the main switch-to-switch link to the MSR. Maybe the converters need a power cycle.

Lightning strike. I just came back inside and looked at the camera to see it hit us. Dave is on the phone now.

Forgot I was logged in from when we were in maintenance for cleanroom cleaning. Logged out at ~23:51 UTC.

TITLE: 05/04 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 64Mpc
OUTGOING OPERATOR: Travis
CURRENT ENVIRONMENT:
    Wind: 7mph Gusts, 6mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.22 μm/s
QUICK SUMMARY: LLO is still down but we are running at 65Mpc for 4hrs

TITLE: 05/04 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 63Mpc
INCOMING OPERATOR: TJ
SHIFT SUMMARY:  In and out of Observing a few times for cleanroom work and OpLev swap and one lockloss likely caused by the cleaning crew (blame withheld).  Otherwise no issues to report.   
LOG:

17:45 Nutsinee to both ends to reinstall HWS plate

17:50 Hugh to MX

Looking at the trends posted in LHO alog 36024, the problem with the ITMy oplev appears to be a failing laser diode.  Since LLO was down, it was decided to perform a swap of this laser with one I recently stabilized in the lab.  The motivation is so we have a before/after record of the ITMy pointing for the vent beginning next week; it was thought to be better to perform the swap now, versus during the vent (either due to preventive maintenance or laser failure) and this effecting the alignment reading of the oplev (in theory it shouldn't, but we all know that old saying about theory and practice...).  New laser SN is 120-2, old laser SN is 189-1.  As is usual, the laser will need a couple hours to come to thermal equilibrium with its new home, then I can assess it for glitchiness.  I will leave WP 6616 open until the laser is operating glitch-free.

OpLev laser swap is complete.  Back to Observing.

For Jason to swap ITMy OpLev laser.  WP#6616.

Shifter: Beverly Berger

LHO fellow: Evan Goetz

For full results see here.

• DQ shift duty cycle: (52.4% + 57.8% + 96.0%)/3 = 68.7% with BNS range 65 - 70 Mpc.
• Most Summary Page data was missing on 1 May and Omicron triggers were missing on 2 May and some data updated only for part of the day. Everything appeared to work on 3 May.
• The ETMX oplev was seen to glitch whenever that data was available. However, when hVeto was also available, the ETMX oplev did not dominate glitching as strongly as ETMY had done previously. However, it was the Round 1 winning channel for the 16-hour hVeto on 3 May and Round 3 for the 24-hour run. The ETMY oplev was well behaved during this shift.
• A peculiar wandering band (see alog 35981) appeared on 2 May, evolving from about 500 Hz to 2 kHz (with a 2nd harmonic up to 4 kHz). The origin is unknown and does not appear to be MIC channels.
• On 3 May, a seismic noise spike appeared at about 5:50 UTC in the earthquake band and at about 10 - 30 Hz in h(t). The apparent earthquake was not visible in any other seismic band and was seen in all the usual places at earthquake band frequencies. The mechanism of upconversion is not clear although the bounce and roll,mode amplitudes spiked at the time of the h(t) noise spike. The winning channel for the noise spike was the round 1 hVeto winner H1:ASC-REFL_B_RF9_I_YAW_OUT_DQ although many other alignment channels have comparable significance.
• Evan heard an airplane fly over the control room on 2 May. The event appeared in h(t) and was recorded by PlaneMon.
• Evan performed a Bruco scan to try to understand the band of low SNR 50 Hz glitches that first appeared on 8 April (see alog 36006). There was no bivouac correlation. It should be noticed that the PSL table spectrogram shows noise features at about 50 Hz. The h(t) glitches are absent on 4 May.
• We point out that the 12:00 UTC range drop due on Monday through Thursday due to truck traffic on route 10 is accompanied by a similar drop at 0:00 UTC. Both features shift by + 1 h for PST. The 0:00 UTC occurrence must be the trucks returning from work. The coupling in this case to h(t) at 50 - 100 Hz may be weaker. Note that Robert has speculated that this upconversion (and others) are due to scattering from the Swiss cheese baffle (see alog 35735).

We are back to being locked at NLN.  Nutsinee is still running some measurements and the cleaning crew is still working on the cleanroom.  We will remain in Commissioning mode until either the cleaning crew is done or LLO come back on line, which ever occurs first.

• Oplev Pit Notes:   HAM2 is out of range above -10 (it is ~ -16)
• Oplev Yaw Notes:  HAM2 is out of range above -10 (it is at ~ -29)
• Oplev Sum Notes:  ITMy looks like it has a pretty steep slope over the last couple of days.  

Attaching screenshot of plots.  CLOSING out FAMIS #4726.

Possibly related to cleaning activities.  Nutsinee is taking advantage of the coincident downtime to reinstall HWS plates at both end stations and take some measurements during relocking.

Sheila helped me sort out the issue from yesterday's attempt at these measurements.  Today they were performed successfully while we were in Maintenance for cleaning for the upcoming vent.  Results can be found in /ligo/home/sheila.dwyer/PEM/HPI_injections_fire_pump:

HPI_HAM2_X4May2017.xml

HPI_HAM2_Y4May2017.xml

HPI_HAM2_Z4May2017.xml

TJ Massinger, Derek Davis, Laura Nuttall

Summary: the intervention on April 26th cleaned up the ETMY oplev glitching but the ETMX oplev is still causing problems. Both noise sources have been seen to cause loud background triggers for transient searches when they're glitching.

Since the end station oplev laser glitches have been seen to couple into h(t), we had to develop vetoes based on the BLRMS of the OPLEV_SUM channels to flag and remove these transients from CBC and Burst searches. Using the thresholds that capture the coupling into h(t), we were able to take some long trends of the impact of the oplev glitching using the veto evaluation toolkit (VET).

ETMY results: https://ldas-jobs.ligo-wa.caltech.edu/~tjmassin/detchar/VET/ETMY_OPLEV/1176422418-1177779618/65/

The take home message from this result page is the Omicron glitchgrams. In the first attached image, every blue point is an Omicron trigger. Those in red are triggers coincident with the ETMY L3 OPLEV SUM having a high 10-50 Hz BLRMS. This population of glitches in the bucket was severely damaging for transient searches, but the problem seems to have gone away since the intervention on April 26th (alog 35798). Looking at the segments where the OPLEV SUM BLRMS was high (attachment 2), we see that after April 26th there are few times when the flag is active, which indicates that there are fewer fluctuations in the OPLEV SUM readout.

ETMX results: https://ldas-jobs.ligo-wa.caltech.edu/~tjmassin/detchar/VET/ETMX_OPLEV/1176422418-1177779618/7_5/

Once again, looking at the Omicron glitchgram (attachment 3), every blue point is an Omicron trigger. Those in red are trigger coincident with ETMX L3 OPLEV SUM having a high 45-100 Hz BLRMS (chosen to capture the power fluctuations in the oplev). The ETMX oplev glitches aren't quite as damaging as the ETMY oplev glitches, but they're still producing loud background triggers in the transient searches. Looking at the segments where this flag was active (attachment 4), we see that the ETMX oplev laser has been glitching on and off over the last two weeks and coupling into h(t).