The PSL tripped 4 times last night, 3 of them in quick succession.  I've done some preliminary forensics, and all 4 were trips of the "Head 1-4 Flow" watchdog, with the Head 3 flow sensor being the culprit in every trip.

Trip #1: 5:07 UTC (21:07 PST 2017-3-5)

PSL tripped due to a trip of the "Head 1-4 Flow" interlock, caused by the flow reading from the Head 3 flow sensor dropping below the trip point.  While restarting the laser after this trip, it took a couple attempts to get the crystal chiller to start and stay running.  After the first attempt, the reported flow from the Head 3 flow sensor did not get above the trip point, so the chiller shut off.  The chiller restarted without issue on the 2nd attempt and the PSL restart went smooth from there.  Once the system was up and running, Jim had to reset the noise eater.

The first attachment shows the flow readings from the active laser head flow sensors along with the "Head 1-4 Flow" interlock signal.  The second attachment shows a slightly zoomed in view of the reading from the Head 3 flow sensor at the time of the trip.  As can be seen, the flow signal from Head 3 became ragged and dropped below the trip point, shutting down the system.

Trip #2: 5:47 UTC (21:47 PST 2017-3-5)

The PSL tripped again, for the same reason as trip #1.  This was also caused by the flow reading from the Head 3 flow sensor dropping below the trip point.  The third and fourth attachments show info for this trip, showing that again Head 3 was the cause.  The restart went smooth, and Jim had gone to the LVEA to reset the noise eater when...

Trip #3: 6:02 UTC (22:02 PST 2017-3-5)

The third PSL trip occurred while Jim was in the LVEA resetting the noise eater.  This trip was identical to the last 2.  The fifth and sixth attachments show that once again the flow reading from the Head 3 flow sensor was the cause of the trip; I also included the power output from the HPO in the fifth attachment.  As can be seen in the fifth attachment (PSL_Trip3_Head1-3_Flow_2017-3-6_06:02:06.png) there was another issue with restarting the crystal chiller; just over 30 seconds after restarting the chiller it shut down due to the flow reading from the Head 3 flow sensor.  The second chiller restart attempt was successful.  There were then zero issues in restarting the PSL; Jim had to reset the NPRO noise eater again once the rest of the system was up and running.

Trip #4: 7:59 UTC (23:59 PST 2017-3-5)

The fourth and final trip occurred just as TJ was taking over for the OWL shift.  Once again this was a trip of the "Head 1-4 Flow" interlock caused by the flow reading from the Head 3 flow sensor dropping below the trip point; this is shown in the final 2 attachments.  There were no issues restarting the PSL after this trip.  Once everything was up and running, TJ had to reset the noise eater twice and was then on to locking the IFO.

J. Kissel, E. Goetz

I've gathered our "bi-weekly" calibration suite of measurements to track the sensing function, and ensure all calibration is within reasonable uncertainty and to have corroborating evidence for a time-dependent detuning spring frequency & Q. Evan will post analysis results later.

The data have been saved and committed to:
/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O2/H1/Measurements/SensingFunctionTFs
    2017-03-06_H1DARM_OLGTF_4to1200Hz_25min.xml
    2017-03-06_H1_PCAL2DARMTF_4to1200Hz_8min.xml

    2017-03-06_H1_PCAL2DARMTF_BB_5to1000Hz.xml

This suite, again for time-tracking, took ~35 minutes from 2017-03-06, 18:02 - 18:35 UTC.

Lock was broken intentionally to begin Jenne's arm cavity scans.

We are still locked but we have transitioned to CALIBRATION for JeffK's calibration sweeps.  We have started the commissioning window as of 18:00 UTC with Keita's permission.  I assume this means the window will shift to 18:00-22:00 UTC.

TITLE: 03/06 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 65Mpc
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
    Wind: 26mph Gusts, 22mph 5min avg
    Primary useism: 0.06 μm/s
    Secondary useism: 0.24 μm/s
QUICK SUMMARY: Locked for 7 hours.  Other than the PSL issue over the night, no issues were handed off.

model restarts logged for Sun 05/Mar/2017
2017_03_05 07:14 h1broadcast0

unexpected crash of h1broadcaster, was rebooted to recover.

model restarts logged for Sat 04/Mar/2017 - Wed 01/Mar/2017 No restarts reported

TITLE: 03/06 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 58Mpc
INCOMING OPERATOR: Travis
SHIFT SUMMARY: PSL tripped as Jim handed the IFO off to me, but I got Jason on the phone and he got it right back up. PI mode 23 has popped up a few times but gone down before I have had to do anything. I think I remember someone saying this is associated with wind gusts somehow. I just ran a2l to fix pitch while LLO is down. It is still windy outside.

LOG:

• 09:08 Observing
• 14:08 GRB
• 15:34 Out of Observing to run a2l while LLO is down.
• 15:39 Back to Observing

Laser Status:
SysStat is good
Front End Power is 33.98W (should be around 30 W)
HPO Output Power is 169.3W
Front End Watch is GREEN
HPO Watch is GREEN

PMC:
It has been locked 0 days, 7 hr 13 minutes (should be days/weeks)
Reflected power = 14.49Watts
Transmitted power = 62.65Watts
PowerSum = 77.15Watts.

FSS:
It has been locked for 0 days 7 hr and 0 min (should be days/weeks)
TPD[V] = 1.868V (min 0.9V)

ISS:
The diffracted power is around 4.5% (should be 3-5%)
Last saturation event was 0 days 7 hours and 13 minutes ago (should be days/weeks)

Possible Issues:

None

Observing at ~58Mpc. Winds are gusting into the 30s and the Handford commute is going. Locked for 4hours so far

TITLE: 03/06 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 58Mpc
OUTGOING OPERATOR: Jim
CURRENT ENVIRONMENT:
    Wind: 15mph Gusts, 9mph 5min avg
    Primary useism: 0.08 μm/s
    Secondary useism: 0.30 μm/s
QUICK SUMMARY: PSL tripped as my shift started. I called Jason and he got things going again. We are now back to low noise and ~60Mpc.

TITLE: 03/06 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Corrective Maintenance
INCOMING OPERATOR: TJ
SHIFT SUMMARY: Bad news PSL
LOG:

Jeff was recovering from an EQ when I arrived. I managed to get the IFO back up for a few hours, but then the PSL went on the fritz. I called Jason, after a few rounds of filling the chiller and resetting the noise eater I was able to get locked again. As soon as TJ arrived, though, the PSL went down again. He's working with Jason (again) on getting back up. Re-locking has been easy though.

Shift Summary: Ran A2L script – Pitch is close to 0.6 and Yaw is at 0.5. PSL AC Temp alerts. Per Jason, went into LVEA to check Makeup Air fan and AC controls. See aLOG #34586.

Lost lock due to a Mag 6.5 earthquake near Kandrian, Papua New Guinea. Primary microseism shot up to 2.0 um/s. Put IFO into DOWN until tremors subside.          

Mag 6.5 Earthquake near Kandrian, Papua New Guinea

    Seen on USGS, Seismon, and Terramon
    USGS - EQ at 22:47 UTC
    Terramon - EQ at 22:47 UTC
    Rise in the BLRMS at around 23:00 UTC, to 3.0um/s
    H1 lost lock, LLO was already down

  In Observing for 7.5 hours. IFO seems to be running well at this time. Wind is a steady Moderate Breeze (up to 18 mph) with gusts up in the low 30s. Primary microseism in X and Y is a bit elevated, secondary microseism is flat at about 0.35um/s. A2L Pitch is up to 0.7, Yaw is below the reference. Everything else is OK.   

I'm working with Cheryl on current CDS issues. Looks like h1broadcast0 machine locked up, Cheryl is rebooting this machine.

The timing system has no errors currently, looks like this was a transient problem. 

• suspected PSL's HVAC might not be in Science Mode configuration, and went out to check. 
• makeup air was set to 100%
• set the makeup air to the 20% we use for Science Mode
• this seems a likely reason for the 2degF temperature drops
• talked to PeterK before and after heading out to the LVEA

• waiting for temperatures to come back up
• picture of refcav trans is attached
• very off center, worse than last night, worse than 3 hours ago when I started my shift

J. Kissel, S. Kandhasamy

As we begin to produce the systematic error budget for H1's response function, we're looking to figure out how to address the clipping of light going into H1 PCAL Y's RX PD that had been revealed in early January (see LHO aLOG 33108 and 33187).

Because we got extremely lucky and took the 2017-01-03 reference measures at a time when the clipping -- which had been found to be slowly varying as a function of time -- had varied briefly back to nominal, this problem does not create any impact on the static, frequency-dependent part of the calibration pipeline. However, because the small, time-dependent correction factors are calculated using the H1 PCALY RXPD, these correction factors are systematically biased. 

Working through the math of T1500377, specifically Eqs. 9, 12, 15, and 16, one can see that while the cavity pole frequency estimate would not be impacted, the scalar correction factors are:
    If 
        x_pcal' = eps x_pcal
    where eps is a real number (the number to convert the apparent displacement, x_pcal, into the real displacement, x_pcal'), then
        kappa_TST' = eps kappa_TST
    and
         kappa_PU' = (1 / A_0^PU) [A_total' - kappa_TST' A_0^TST]
                        (A_total' = eps A_total, from Eq. 11)
                    = (eps / A_0^PU) [A_total - kappa_TST A_0^TST]
                    = eps kappa_PU
    and finally,
                 S' = (1 / C_res) [ x_pcal' / d_err - D_0 (kappa_TST' A_0^TST + kappa_PU' A_0^PU)]^-1
                 S' = eps S
    such that
          kappa_C' = |S'|^2 / Re[S'] = (1 / eps) kappa_C
which means that we can simply scale the entire response function with this time-dependent systematic error:
                 R' = [1 / (kappa_C' C_0) ] + [kappa_PU' A_0^PU + kappa_TST' A_0^TST]
                    = [eps / (kappa_C C_0)] + eps [kappa_PU A_0^PU + kappa_TST A_0^TST]
                 R' = eps R

I attach a 77 day minute trend of the ratio between RXPD and TXPD that's been obtained from data viewer***. I'm not yet advocating that time-series this be used as the representative systematic error, but I post it to be demonstrative. Stay tuned on how this data is encorporated into the uncertainty / error budget.

*** Because it's data viewer, it spits out some dumb Julian calendar time vector. Just subtract off the first value, and you get a time vector in days since the start of the trend, which is Nov 30 2016 00:54:44 UTC.

The data also lives in the CAL repo here:
/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O2/H1/Measurements/PCAL/
    2017-02-14_H1PCALY_RXPD_Trend.txt
    2017-02-14_H1PCALY_TXPD_Trend.txt

The script used to process this data and plot the figure is here:
/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O2/H1/Scripts/PCAL/
    plot_h1pcaly_RXvsTXPD_trend_20170214.m

Tom Dent, Miriam Cabero

We have identified a sub-set of blip glitches that might originate from PSL glitches. A glitch with the same morphology as a blip glitch shows up in the PSL-ISS_PDA_REL_OUT_DQ channel at the same time as a blip glitch is seen in the GDS-CALIB_STRAIN channel.

We have started identifying times of these glitches using omicron triggers from the PSL-ISS_PDA_REL_OUT_DQ channel with 30 < SNR < 150 and central frequencies between ~90 Hz and a few hundreds of Hz. A preliminary list of these times (on-going, only period Nov 30 - Dec 6 so far) can be found in the file

https://www.atlas.aei.uni-hannover.de/~miriam.cabero/LSC/blips/O2_PSLblips.txt

or, with omega scans of both channels (and with a few quieter glitches), in the wiki page

https://www.lsc-group.phys.uwm.edu/ligovirgo/cbcnote/PyCBC/O2SearchSchedule/O2Analysis2LoudTriggers/PSLblips

Only two of those times have full omega scans for now:

https://ldas-jobs.ligo-wa.caltech.edu/~detchar/hveto/day/20161204/1164844817-1164931217/scans/1164876856.97/

https://ldas-jobs.ligo-wa.caltech.edu/~detchar/hveto/day/20161204/1164844817-1164931217/scans/1164882018.54/

The whitened time-series of the PSL channel looks like a typical loud blip glitch, which could be helpful to identify/find times of this sub-set of blip glitches by other methods more efficient than the omicron triggers:

https://ldas-jobs.ligo-wa.caltech.edu/~detchar/hveto/day/20161204/1164844817-1164931217/scans/1164876856.97/1164876856.97_H1:PSL-ISS_PDA_REL_OUT_DQ_1.00_timeseries_whitened.png