At approximately 15:30 UTC (8:30 PDT) on Saturday, 7/8/2017, the flow through the HPO laser heads dropped by ~0.12 lpm; see the first attachment.  This is worrying as the trip point for the laser head flow is set to 0.4 lpm; the flow through heads 2 and 3 is sitting at ~0.45 lpm, just 0.05 lpm above the trip point.  Looking at the other channels in the cooling system the picture becomes a little clearer (maybe).

The 2nd attachment shows the pressures in the PSL cooling manifold; H1:PSL-OSC_PRESS1 is the pressure at the manifold input, H1:PSL-OSC_PRESS2 is the pressure at the manifold output.  As seen in the attachment, the pressure at the inlet increased slightly, and the pressure at the outlet dropped slightly.  The third attachment shows the flow through the HPO power meter circuit (H1:PSL-OSC_PWRMETERFLOW), the 35W FE laser circuit (H1:PSL-AMP_FLOW), and the overall flow out of the PSL crystal chiller (H1:PSL-OSC_XCHILFLOW); the flow out of the chiller dropped slightly (~0.1 lpm) while the flow through the HPO power meter and 35W FE circuits both increased very slightly (interestingly, the signal from the FE flow sensor cleaned up after the event, and the frequency of drops in the power meter flow sensor signal also decreased).  The most likely cause of this behavior is some kind of flow restriction, probably somewhere in the laser head cooling circuit (although it is possible there could be a blockage in the filters under the PSL table).

The 4th attachment shows the temperature of the individual laser heads.  All saw an increase of ~0.2 °C, which indicates that this is a real drop in the flow through the HPO laser heads.  Fortunately this was not enough to effect the output power of the laser (5th attachment), and the temperatures have remained steady since the loss of flow on Saturday.

In the short term, we can try increasing the overall flow at the crystal chiller to see if it clears whatever is restricting the flow; at the very least we can increase the flow so we are a bit further from the trip point.  We will also inspect the filters beneath the PSL table when we are in the enclosure during tomorrow's maintenance window.  Any further investigation will require opening up the cooling system, which is highly invasive.

J. Kissel

Just posting an update on this -- the H1 PCAL Y RX PD reported displacement continues to decay, likely due to some temperature dependent clipping. This had been a part of the FRS Ticket 8328, but there's little we can do during an observation run to fix this. We've since closed the issue as LONGTERMFIX. AS such, I'll raise to an integration issue, and mark as WHEN VENT (similar to the promised work in Integration Issue 4700).

For now, recall that we've switched over to using TXPD as the calibration reference (see LHO aLOG 37166).

TITLE: 07/10 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Earthquake
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
    Wind: 15mph Gusts, 11mph 5min avg
    Primary useism: 0.02 μm/s
    Secondary useism: 0.05 μm/s
QUICK SUMMARY:

TITLE: 07/10 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Earthquake
INCOMING OPERATOR: Ed
SHIFT SUMMARY: Trying to damp violin modes. Had to recover a bit from the earthquake in Travis's shift since it rang up the violins a bit more, but we seem to be back to where we were, 6.5hr lock at 53Mpc.
LOG:

• 07:52 Lock loss. It didn't like the 4735hz damping that I tried.
• 08:23 Back to NLN
• 08:40 Back up to 50Mpc. Following the violin damping wiki, damping 997.89hz seemed to really help our range even though it wasn't the most rung up 2nd harmonic.

Damping notes:

• 4735.02hz is highest 4.7k mode and does not appear to be in the wiki (it is now). After much trial and error I was able to damp it down with the ETMY MODE10 filter bank, but slightly rang up 4735.04. Overall a big improvement to the 4.7k modes though
• 997.717 I was able to slowly damp with the settings in the wiki. I tried other combinations but nothing seemed to work any better.
• 994.646 Is still unknown and one of the largest second harmonics.
• There are still many third harmonics that are unknown and rung up. My reference from the beginning of my shift isn't much different than what it is now.

Damping continues. Range is now up to 52Mpc, but there are many modes that I can't figure out how to damp.

TITLE: 07/10 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Earthquake
OUTGOING OPERATOR: Travis
CURRENT ENVIRONMENT:
    Wind: 20mph Gusts, 16mph 5min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.04 μm/s
QUICK SUMMARY: Still rough going since the Montana quake earlier in the week, and a recent quake ~5hours ago made things worse. I'll see what I can damp and if I can get it locked again, but I don't have high hopes.

TITLE: 07/10 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Earthquake
INCOMING OPERATOR: TJ
SHIFT SUMMARY:  After the lockloss halfway through the shift, the violin modes that have been worked on all week seem to have rang back up.  I worked successfully on several of them, but as has been the story, the worst offenders seem to be unresponsive to my damping efforts.
LOG:  See previous aLogs.

VerbalAlarms has been announcing "OMC DCPD saturation" every few seconds for ~10 minutes.  I have never received this notification without a lockloss following.  This is not the case today.  Not sure what to do about this.

TITLE: 07/09 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Earthquake
OUTGOING OPERATOR: Ed
CURRENT ENVIRONMENT:
    Wind: 19mph Gusts, 15mph 5min avg
    Primary useism: 0.06 μm/s
    Secondary useism: 0.05 μm/s
QUICK SUMMARY:  Same state, different day.  Range around 50 MPc with higher order violin modes still rung up.  Lower Q fibers anyone?

TITLE: 07/09 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Earthquake
INCOMING OPERATOR: Travis
SHIFT SUMMARY:

Most if the day spent trying o damp violin modes. Some successful, some not. Elevated noise in DARM between 20-80Hz has reduced the range back into the 50Mpc area. Handing off to Travis.
LOG:

19:18 The bucket looks funny and the a2l measurement is showing some weird spikey things.

16:15 oops. Lockloss, My fault. It seems that the ramp time on the mode9 filter bank I was trying to use wasn't set at 10 seconds as it was yesterday. It was 0. !!!!

• funny thing is, guardian set all these ramp times to 10 when re-locking. I don't know why it was different.

16:25 The MC Trans camera isn't showing me a spot at low power even though the REL camera looks normal for the IMC being locked and the StripTool verifies. Strange.

• Just noticed Cheryl's aLog about the camera

17:01 55Mpc

17:04 accepted an adjustment to FSS to controll oscillations at lockloss. Range at 54.1 Mpc

17:22 begin damping ITMY mode10 - FM8 off, FM6 on (997.89)

• got some good responsive damping from that effort - gain to -100

17:56 attempt at suppressing ITMY mode9 (994.897)

• also some good damping reponse from a gain increase from 10 to 100

19:04 still playing whack-a-mole in the violin section of this orchestra. If nothing else I'm getting damping practice.

• Range is oscillating between 53 and 54Mpc

TITLE: 07/09 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Earthquake
OUTGOING OPERATOR: Jeff
CURRENT ENVIRONMENT:
    Wind: 7mph Gusts, 4mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.05 μm/s
QUICK SUMMARY:

TITLE: 07/09 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Earthquake
INCOMING OPERATOR: Jeff
SHIFT SUMMARY:  Locked at NLN with a range around 53 MPc.
LOG: Bubba and Kyle were on site around 2:30 UTC.