TITLE: 08/26 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Aquisition
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 17mph Gusts, 14mph 5min avg
    Primary useism: 0.05 μm/s
    Secondary useism: 0.06 μm/s
QUICK SUMMARY:

- H1 has just lost lock as I am typing this...cause unknown

- SEI motion is a bit elevated in the EQ band, hovering around 0.1 um/s

Happened to look up and see a "Stand Down" state.  Did not receive any alerts for this.  The graceDB link for this lists this "GRB-SHORT" as NOTGRB.  Here's more info on this particular event (E432741):  https://gracedb.ligo.org/events/E432741/view/

PSL Report FAMIS report:
Laser Status:
    NPRO output power is 1.829W (nominal ~2W)
    AMP1 output power is 67.15W (nominal ~70W)
    AMP2 output power is 135.5W (nominal 135-140W)
    NPRO watchdog is GREEN
    AMP1 watchdog is GREEN
    AMP2 watchdog is GREEN

PMC:
    It has been locked 20 days, 6 hr 1 minutes
    Reflected power = 16.33W
    Transmitted power = 109.2W
    PowerSum = 125.5W

FSS:
    It has been locked for 0 days 10 hr and 48 min
    TPD[V] = 0.9056V

ISS:
    The diffracted power is around 2.4%
    Last saturation event was 0 days 10 hours and 48 minutes ago

Possible Issues: None

H1's been locked for 8.5hrs.  At around 1630utc (930amPT) H1 took a step down in range.  There was also a brief drop out of Observing due to the Squeezer (noted earlier).

At 17:41:44, H1 was dropped out of OBSERVING...after quickling taking OBSERVATORY_MODE to "COMMISSIONING", H1 was ready to go back to OBSERVING at 17:42:21.

Using the Guardian Control command, there was a brief SDF diff due to syscssqz.  I am not sure how to find which channel dropped us out.  I ran guardctrl on Squeezer nodes and a few other guardian nodes, but could not find the culprit.  (see attached).

Sat Aug 26 10:10:48 2023 INFO: Fill completed in 10min 44secs

TITLE: 08/26 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 158Mpc
OUTGOING OPERATOR: Tony
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 13mph Gusts, 11mph 5min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.06 μm/s
QUICK SUMMARY:

H1's been locked 4hrs.  H1 range continues at increased value of 150Mpc as was for the lend of the last lock.  There was a Labs Dust alarm needing to be acknowledged (it had been OK for last3hrs).  Winds slowly beginning to pick up.

Hanford site road work continues on a Saturday on Route10 (crack sealing) and I was turned around and had to take the long way around to get to work (this work was normally M-Th 7am-3pm, but they've been working Fri & Sat this week).

TITLE: 08/25 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 162Mpc
INCOMING OPERATOR: Tony
SHIFT SUMMARY:

- EX saturation @ 11:10/1:48/2:13 UTC

- Lockloss @ 5:56 - new record for a O4 lock at LHO - 60:29!

- Issues getting light on the ALS Y PD - might be related to the chiller issue we had earlier today?

• Attached witness sensors and slider values over the course of the past day - does not seem to be much drift (the most recent movement was from me trying to find light on the PD)
• The ALS Y guardian is showing a ReflPD A/PDH error
• Looking on the trouble shooting wiki, Tony and I tried running the dither align script (under TMS pointing/dithering) to try and get some power on the PD - no luck
• Attaching oplevs - looks like there is movement/drift - mostly in pitch - scope attached
  • Tried moving the sliders back to their original oplev positions from ~2 days ago - still no light
• Followed Camilla's steps here, found in the wiki to be used "if there is no image on the ALS camera", still no luck
• At this point, we're out of ideas and will be contacting a commissioner to help troubleshoot, stay tuned for a further alog

LOG:

No log for this shift.

Lockloss @ 5:56 UTC, DCPD saturation right before. Unknown cause. New record for a H1 lock during O4 - 60:29!

H1 is still chugging along, and has officially set a new O4 lock record of 57:30 (and counting)! Barring a few EX saturations; systems appear stable, and ground motion/wind is low.

At 9:45am local time Chiller 1 at End Y went into alarm for "Evaporator Water Flow Lost". When I arrived at the EY chiller yard I observed that neither chiller was running but chilled water pump 1 was continuing to run. I noted the alarm and headed for the mezzanine above the AHU's to asses what the supply and return pressures were nearest the evaporator coil. 

Immediately I read 0 (or what I thought was 0) at the return line. This would generally indicate that there has been enough glycol loss within the system that makeup is necessary via the local tank (though I've never seen it get to 0). Until I read that both supply lines were at an alarming 140psi (normal operating pressures for all 4 supply and return float around 30). I immediately phoned Richard to have him command the chilled water pump 1 off to stop the oversupply of chilled water. For reasons not clear to me the disable command via FMCS Compass was not taken at the pump.

I went back to the chiller yard and observed that 1: the pump had not been disabled and 2: pressures at the pump were at around 100psi (normal operating for the current frequency is about 50).

Following that, I manually threw the pump off at the VFD to prevent further runaway of the system. Between the time of noting 140psi and manually throwing the pump off, the system pressure increased to 160psi.

After a thorough walk-down of the system, I elected not to utilize our designed redundancy in chiller 2 and chilled water pump 2 as I was still unaware what was causing the massive overpressure at all supply and return lines. It was also found that the return line was not actually at 0, but instead had made a full rotation and was pegged on the backside of the needle (all of these need replacement now).

Macdonald-Miller was called on site to help asses what the issue might be. Given that there was recent incursions to flow via R. Schofield the strainer was the primary point of concern. We flushed the strainer briefly at the valve and noted a large amount of debris. after a second flush, much less/next to none was noted. This alleviated the system pressure substantially. 

The exact cause of the fault and huge increase of pressure is still not clear. There are a number of flow switches at the chiller. Bryan with Mac-Miller suspects part of the issue may live there, and we are going to pursue this further during our next maintenance window. Work was also performed at the strainer within the chiller where rubber/latex-esque debris was found. Work on Chiller 1 to continue but for now the system and end station is happy on chiller2/CHWP2. Looking at the FMCS screen shows temp's have normalized as of the writing of this log.

T. Guidry B. Haithcox. R. Thompson C. Soike R. McCarthy

Closes 26247, last completed in alog 72330

VAC_FAN5_170_2 has had a few spikes over the course of the last week, but the rest of the fans appear stable.

Inspired by Valera's alog demonstrating the improvement in the LLO sensitivity from O3b to O4a (LLO:66948), I have made a similar plot.

I chose a time in February 2020 for the O3b reference. I'm not aware of a good time without calibration lines, so I used the CALIB_STRAIN traces from both times. Our range today just got a new bump up, either due to the commissioning work today or the EY chiller work (72414). Note: I am quoting the GDS calib strain range!

I am adding a second plot showing O3a to O4 a as well, using a reference time from May 2019.

There is a significant amount of work that gave us this improvement. I will try to list what I can recall:

• ITMY replacement in 2021 to remove large point absorber
• Removal of septum window, new output faraday isolator, new OM2 TSAMS
• Addition of filter cavity for frequency dependent squeezing- squeezing levels around 3.7 dB (or more)
• Increase in circulating arm power ~200kW to 375kW
• Commissioning of camera servos- no more 20 Hz dither lines
• Reduction of ASC loop bandwidths, commissioning of HAM1 feedforward
• Reduction of LSC loop bandwidths, improvement of LSC feedforward
• Improvement of HAUX, HSTS and HLTS damping loops, reduced injection of BOSEM noise
• Damping of various baffles
• HAM1 table work to clamp RM blade springs, power attenuation on POP and REFL to allow power up
• Optimized test mass ESD biases to minimize electromagnetic interference coupling, improved grounding at end stations and corner to reduce ground loop noise
• Ring heater and CO2 laser commissioning to reduce frequency noise coupling and contrast defect

Closes FAMIS 26434, last completed in alog 71777

All BRS channels are within their nominal regions.

TITLE: 08/25 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 150Mpc
INCOMING OPERATOR: Austin
SHIFT SUMMARY:

H1's been locked 53.5hrs (1hr from our O4 longest lock *knock on wood*).

Main news of today was the H1 Chiller Pump#1 going down.  Tyler as well as a contractor spent most of the last 6hrs dealing with this and we are currently set and running with Chiller Pump #2.

There was 2hrs of commissioning by Gabriele & Elenna as well.

We appear to have a ~5Mpc increase in range....waiting to hear if this is due to the commissioning or working with different chiller pump!

LOG:

• 1707-2240 EY visit to address HVAC chilled water issues with incursion to Mechanical room (tyler)
  • 1814-1826 quick run to EX to compare chiller.
  • Richard at EY
  • Contractor contacted to come out to work on chiller
  • 1838 Richard walking into electronics bay to check on temperature
  • 1839 Propping door between E-Bay & Garb Room (Richard)
  • 1850 "Move & Cool" Portable air cooler being forklifted into the EY E-bay (the area you enter (tyler, chris, randy)
  • Ryan autoscaled the temperature scope on nuc32
  • 2011 Mac Miller (hvac contractor) starts driving down to EY with tyler
  • 2041 Heading out to try to turn on more AC units (i.e. EY Lab room) [Richard]
  • 2240 Done for the weekend.  Tyler said we are looking good and returning to a normal state with only Chiller Pump2.  (Chiller Pump1 still needs to be looked at.  In n
• 2018 COMMISSIONING---LSC & MICH Feed Forward measurements by Gabriele & Elenna

Recently we've been receiving groups of SubGRBs or long grbs from our igwn-alert subscription system that are often old events or repeated. Since we don't normally react to these and will ignore them, we don't need to even have them update in epics. I've added more filtering for these types of events in igwn_response.py and also in the tests.py for VerbalAlarms. The new code is running for both of these systems.

Another related issue I'm looking into is why we haven't received any superevents from out igwn-alert listener in maybe a month. I don't think the issue is on our end, but I have a few more checks. We still get phone calls in the control room, so operators are still informed.