Yesterday I got a call from Ed that the TCSY flow rate was reporting under 2gpm, lower than the usual 3gpm (alog51806). While the flow at the flow meter was reporting lower, the chiller was still reporting its usual flow rate, ~4gpm. I was immediately concerned of a leak, but Ed found no water missing in the chiller. We are still full today as well.

My next suspect was the flow meter. Our water in the system is not the cleanest, and we have had issues in the past with the paddles in these flow meters getting gunked up (see Peter's alog32777 for pictures). This particular flow meter was replaced December 20, 2016, for similar issues, and the TCSX flow meter was replaced October16, 2018. The situation looks a bit different now because we have less of what we have been calling glitches in the flow. The difference is that yesterday and on the 5th we had this flow meter report the flow as ~1gpm lower for a few tens on minutes, before returning to normal. Meanwhile, the chiller flow doesn't seem to be affected. The laser temperature goes up slightly when this happened, about 0.5C, but this has been happening more frequently when we replaced the chiller.

The current, running TCSY chiller is actually our spare. We swapped it out on January 15, 2019 due to a leaky shaft seal. The flow on the current chiller seems to be less consistent than on the normal TCSY chiller. As Jason notes in alog50789, this may be part of the reason we have had more relocks from this laser. The normal chiller has be repaired with a new motor and pump, but still needs to be tested.

Going forward:

It definitely looks like we need to swap out and rebuild the TCSY flow meter soon. This isn't hugely invasive, but runs the risk of breaking pipes because the meter is dry-fit into the pipe and each time has been more difficult than expected. Because of this, we are hoping to push this off till the October vent, when we have more time. In the mean time we would like to swap the TCSY chiller as soon as we can get a long term test on the backup unit. The inconsistency of the current chiller can't be helping the flow problems, and it's an easy swap.

Attachment 1 - The flow rate from the flow meter, the flow from the chiller, the laser temp., and the laser output from the last few days.

Attachment 2 - Same channels as above, but in December, before the chiller swap.

[J. Kissel, A. Viets]

I just restarted the primary and redundant calibration pipelines around GPS time 1252087593, or Mon Sep 09 18:06:15 GMT 2019.  They are using the new filter described in LHO aLOG 51795.  This change is coordinated with the front-end changes noted in LHO aLOG 51819.

YES THIS WILL DROP THE FRONT-END COMPUTED BNS RANGE, because we're improving the accuracy of the real-time pipeline in order to better matrch the current IFO (more power than when it was last updated, no detuning, good spot positions, SRCL offset, accounting for new UIM boost, etc.)

Should be done in an hour or three. When we come back in to OBSERVATION READY, the new model will be in place.

See details of the model in LHO aLOG 51784, and references there in.
The decision point to change the model is documented in LHO aLOG 51782, which has yet more references and explanation.

TITLE: 09/09 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 116Mpc
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
    SEI_CONF state: WINDY
    Wind: 5mph Gusts, 4mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.13 μm/s
QUICK SUMMARY:

TITLE: 09/09 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 116Mpc
INCOMING OPERATOR: Ed
SHIFT SUMMARY:

Nice shift with H1 in Observing for nearly the entire shift.  This current lock has a different input pointing (via IMs), so if there are any issues you can easily revert that pointing (see SDF diffs here).
LOG:

• 6:59 H1 at NLN
• 7:08 H1 Taken to OBSERVING
• 7:35-8:07 48Hz line noisier.

Smooth running for H1 with a range averaging between 118-119Mpc.  Small EQ rolled through, but not requiring a change to the SEI_CONF.

During shift Hand-off of an H1 in NOMINAL LOW NOISE, I accepted SDF diffs which came up.  We were at a nice range of 118Mpc. 

SDF Diffs

The Diffs are attached and were:

1) SUSIM for IM1, IM2, & IM3

These are known and related to a change Cheryl made for relocking.  She will make an alog about this.

2) ASC:  RPC gains

Have a few small gain changes for ASC-RPC_DHARD & CHARD pit/yaw.  These looked new to me (what's "RPC"?) & Cheryl was not familiar with them as well.  I did a quick scan of ASC medms, but I could not track them down.  Nevertheless, I went ahead and accepted them.

Note for OBSERVATORY MODE:  The last 2.5-3hrs of downtime was listed as OBSERVING (vs. LOCK ACQUISITION or EARTHQUAKE).  I switched it very briefly to LOCK ACQUISITION so we can mark that we were in fact down due to a lockloss.

TITLE: 09/09 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 118Mpc
INCOMING OPERATOR: Corey
SHIFT SUMMARY: dropped lock during EQ, relocked with small changes to IMs, in Observe
LOG:

• 04:20:17 - lockloss, SEI was in EQ, DCPD and EX verbal alarms
• plot attached shows that bounce mode for ETMX increased right before lockloss
  • IX and EX bounce were hovering arounf 1.0, and about 8 seconds before lockloss EX bounce shot up to 1.7
• first attempt at relock ended before DRMI
• second attempt I made some small changes to IM1, IM2, IM3
  • maximized to a smooth signal POPAIR_B_RF18_I_NORM_MON, minimized POPAIR_B_RF18_I_NORM_MON, maximized LSC_REFL_A_LF_OUT16
• third attempt didn't survive ENGAGE SOFT LOOPS
• fourth attempt went to NLN
  • range is around 119Mpc

TITLE: 09/09 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 116Mpc
OUTGOING OPERATOR: Cheryl
CURRENT ENVIRONMENT:
    SEI_CONF state: WINDY
    Wind: 7mph Gusts, 6mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.12 μm/s
QUICK SUMMARY:

Cheryl had H1 at NOMINAL LOW NOISE a few seconds before my shift, and I took H1 to Observing shortly thereafter.

BS check has an increase in noise between 1Hz and 6Hz, on 3 sensors, when compared to May 2019 spectra (alog 49425), no issues at high frequency.

• BS_ST2_CPSINF_H1_I
• BS_ST2_CPSINF_V2_I
• BS_ST2_CPSINF_V2_I

Spectra for other sensors look OK.

There are 8 T240 proof masses out of range ( > 0.3 [V] )!

• ETMX T240 2 DOF X/U = -0.841 [V]
• ETMX T240 2 DOF Y/V = -0.596 [V]
• ETMX T240 2 DOF Z/W = -0.801 [V]
• ETMX T240 3 DOF Y/V = -0.323 [V]
• ITMX T240 1 DOF X/U = -1.028 [V]
• ITMX T240 3 DOF X/U = -1.048 [V]
• ITMY T240 3 DOF X/U = -0.435 [V]
• ITMY T240 3 DOF Z/W = -1.221 [V]

All other proof masses are within range ( < 0.3 [V] ):

• ETMX T240 1 DOF X/U = -0.188 [V]
• ETMX T240 1 DOF Y/V = -0.175 [V]
• ETMX T240 1 DOF Z/W = -0.285 [V]
• ETMX T240 3 DOF X/U = -0.236 [V]
• ETMX T240 3 DOF Z/W = -0.192 [V]
• ETMY T240 1 DOF X/U = -0.222 [V]
• ETMY T240 1 DOF Y/V = 0.118 [V]
• ETMY T240 1 DOF Z/W = -0.188 [V]
• ETMY T240 2 DOF X/U = -0.114 [V]
• ETMY T240 2 DOF Y/V = -0.179 [V]
• ETMY T240 2 DOF Z/W = -0.12 [V]
• ETMY T240 3 DOF X/U = -0.217 [V]
• ETMY T240 3 DOF Y/V = -0.242 [V]
• ETMY T240 3 DOF Z/W = -0.005 [V]
• ITMX T240 1 DOF Y/V = 0.118 [V]
• ITMX T240 1 DOF Z/W = 0.066 [V]
• ITMX T240 2 DOF X/U = 0.117 [V]
• ITMX T240 2 DOF Y/V = 0.076 [V]
• ITMX T240 2 DOF Z/W = 0.155 [V]
• ITMX T240 3 DOF Y/V = 0.059 [V]
• ITMX T240 3 DOF Z/W = 0.002 [V]
• ITMY T240 1 DOF X/U = 0.113 [V]
• ITMY T240 1 DOF Y/V = 0.093 [V]
• ITMY T240 1 DOF Z/W = 0.125 [V]
• ITMY T240 2 DOF X/U = 0.119 [V]
• ITMY T240 2 DOF Y/V = 0.235 [V]
• ITMY T240 2 DOF Z/W = 0.144 [V]
• ITMY T240 3 DOF Y/V = 0.149 [V]
• BS T240 1 DOF X/U = -0.054 [V]
• BS T240 1 DOF Y/V = -0.184 [V]
• BS T240 1 DOF Z/W = 0.199 [V]
• BS T240 2 DOF X/U = -0.012 [V]
• BS T240 2 DOF Y/V = 0.182 [V]
• BS T240 2 DOF Z/W = -0.092 [V]
• BS T240 3 DOF X/U = -0.007 [V]
• BS T240 3 DOF Y/V = -0.203 [V]
• BS T240 3 DOF Z/W = -0.274 [V]
• Assessment complete.

2019-09-08 18:32:37.355305
All STSs prrof masses that within healthy range (< 2.0 [V]). Great!

• Here's a list of how they're doing just in case you care:
• STS A DOF X/U = -0.887 [V]
• STS A DOF Y/V = -0.781 [V]
• STS A DOF Z/W = -0.533 [V]
• STS B DOF X/U = 0.27 [V]
• STS B DOF Y/V = -0.65 [V]
• STS B DOF Z/W = -0.414 [V]
• STS C DOF X/U = 0.18 [V]
• STS C DOF Y/V = 0.508 [V]
• STS C DOF Z/W = 0.445 [V]
• STS EX DOF X/U = -0.2 [V]
• STS EX DOF Y/V = 0.296 [V]
• STS EX DOF Z/W = 0.063 [V]
• STS EY DOF X/U = 0.174 [V]
• STS EY DOF Y/V = -0.186 [V]
• STS EY DOF Z/W = 0.579 [V]
• Assessment complete.

TITLE: 09/08 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 115Mpc
OUTGOING OPERATOR: Ed
CURRENT ENVIRONMENT:
    SEI_CONF state: WINDY
    Wind: 21mph Gusts, 16mph 5min avg
    Primary useism: 0.05 μm/s
    Secondary useism: 0.12 μm/s
QUICK SUMMARY: locked in Observe, wind spike to around 30mph

TITLE: 09/08 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 113Mpc
INCOMING OPERATOR: Cheryl
SHIFT SUMMARY:
LOG:

I got this verbal at 18:23UTC. I contacted TJ before trotting off to the mechanical room. We found that the 'CO2 Y' & 'CO2Y CHILLERS' screens show a discrepancy in flow rates. I went out to the chiller and found that the water level was at 9 (down from 10) so I added 50ml. At 18:55, after I returned to the CR I got another verbal. Trending shows this sudden drop in flowrate (possible leak) or possibly sudden malfunction in flow meter?

TITLE: 09/08 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 119Mpc
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
    SEI_CONF state: WINDY
    Wind: 9mph Gusts, 7mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.14 μm/s
QUICK SUMMARY:

H1 locked and reading 119+ Mpc. It just started raining again. All seems well at the moment.