We've had short locks today, and the veritcal osems suggest that the test masses are at an extreme not seen since last Febuary.  Eric noted that he had to change settings for the air handler 82763

We've had very short locks today, Jim has reverted his change from just over a week ago although we don't think that's likely to be the problem. 

Perhaps one of our optics is rubbing.

Yesterday 4-2-2024 the HVAC pre-filters at each of the 4 out station buildings along the arms were replaced with new pre-filters. The new pre filters are the same size (24"x24"x2") and filtration level (MERV-13) as the old pre-filters that were taken out. 
During inspection of the pre-filters an inspection was done on the "Main Filter Cartridge" which is the harder, more permanent, deeper, and heavier duty material type cartridge. It's installed in front of the pre-filter that is used for air filtration in the HVAC system. These are nearing the end of their life cycle. These will likely be replaced in the coming couple months.

(Randy, Mitchell, Betsy)

The West Bay cleanroom work is complete so the large cleanroom has been turned off again.  Room turned off ~9:30am this morning.

(The 4x 3IFO QUAD upper structures have now been moved to their smaller storage containers, relabeled , and ICS records re-organized.  The large ISI storage container that they came out of still houses the 2 Elliptical baffle down tube assemblies and partial TMS telescope subassemblies.  Photo pictures have been also added to the containers for our future selves.)

Over the course of the day today, Ryan and I have noticed the BNS inspiral range dropping starting around noon PST. Since it's mostly come back at this point almost 12 hours later, it seems to be correlated to a drop in LVEA temperature which in turn has raised the large corner station optics (BS and ITMs) according to their vertical top-mass OSEMs (see one- and six- day trends attached). Perhaps the HVAC system lowered the heat into the LVEA in response to the outside temperature rising? Since H1 has remained locked and observing over this time, I imagine the angular control loops have been keeping up with alignment changes, but possibly altering noise in DARM which has been affecting range. I don't believe any action is needed here, but I think it's still worth noting here how corner station temperature changes can affect our observation performance.

TITLE: 11/28 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Preventive Maintenance
INCOMING OPERATOR: Ryan S
SHIFT SUMMARY:

1. 16:35 UTC - Dave noticed an EY GPS timing error on the CDS overview that was flashing. The PPS A GPS clock was flashing an error due to a timing difference past tolerance. Error disappeared after 16:40 UTC.
2. 17:00 UTC SNEWS alert test T456241 caught on both Diag main and verbal
3. 18:11 UTC HAM7 Watchdog trip, likely due to SQZ rack work that Fil is doing (pulling cables).
4. 18:46 UTC IOC server temp change: While Erik was updating the IOC LVEA temperature, he realized that the entire IOC server needed to be rebooted.
   1. IOC server reboot happened at 18:46 UTC and connection was reestablished at 18:49 UTC.
   2. IOC reboot temporarily took down:
      1. Remote power
      2. Check violins
      3. Wi-Fi
      4. Ops GraceDB Standown - did not restart as expected, TJ turned it back on
      5. Picket Fence - did not restart as expected prompting further investigation by Erik - 
   3. Wi-Fi was separately turned on during reboot in order to maintain control room work.
   4. 19:09 UTC Erik updating IOC temp. Done at 19:16 UTC
5. Two GRB Short Alerts
   1. 19:25 UTC
   2. 19:37 UTC
6. 19:00 UTC (ish) Temp excursion in CER and SUP Rack 1 - still within “nominal/tolerance” but visibly on the come-up. 
   1. 20:10 UTC Eric investigated and found that the lead unit had a fault and so he reset the system. Temperatures visibly on the come-down, both in CER and SUP.
7. 19:46 UTC - during Beckhoff work, 
   1. Beckhoff restart caused a connection error with guardian ALSY node. Guardian 1 seems to have lost connection, despite Beckhoff showing everything as normal. There are 7 SPM diffs, all pertaining to H1:ALSY (and nothing else). 
   2. Apparently it fixed itself at 19:46 UTC and then the same issue happened at 20:27 UTC
   3. Error message reads as: “CONNECTION ERRORS, see SPM DIFFS for dead channels". See screenshot.
   4. The workstations have no problems and everything else seems normal.
   5. Dave and Daniel suggested doing a Guardian reboot, though Dave is looking into it and seeing if it will/won’t help.
   6. 20:45 UTC: This is currently stopping us from locking
   7. 20:51 (ish) UTC - TJ fixed the issue by bringing the guardian node to STOP and then EXEC, instead of reloading
      1. Apparently this is an issue where sometimes when connections are lost and reconnected a complete STOP rather than a PAUSE/RELOAD is necessary to bring the connection back.
      2. No Guardian reboot necessary
8. DAQ Restart: NUC23 isn’t loading and is yielding an error message - Ryan C couldn’t get into it during the previous maintenance Tuesday.
   1. Ryan C is power cycling it and then attempting to connect
   2. Worked fine after restart
9. Locking at 20:53 UTC 
   1. Starting with an initial alignment - Initial Alignment went fine
      1. Had to touch ALS_Y Yaw to get it caught
10. Lock acquisition 1 at 21:48 UTC: Lock loss due to DRMI lockloss at Turning_BS_Stage2
    1. Caused by DRMI_ASC issue that Sheila caught and is working on.
11. Lock acquisition 2 at 21:52 UTC: Sheila and Naoki - DRMI_ASC won’t work until they fix something so guardian only taken to DRMI_Locked_Prep_ASC
    1. Sheila said it was potentially fixed -
    2. Lockloss at the same stage again 21:57 UTC
12. Lock acquisition 3 at 21:59 UTC
    1. IMC put to DOWN at 22:00 UTC while Sheila and Naoki investigate the DRMI ASC issues
    2. Back to lock acquisition at 22:17 UTC
    3. Paused at DRMI_LOCKED_PREP_ASC - Sheila and Naoki noticed ASC SRC1 Pitch and Yaw offset(s) were off by orders of magnitude but they seem to have been fixed during their investigation. See Naoki’s alog 74457.
    4. 22:28 UTC Continuing lock - it worked!
    5. NLN Achieved at 23:04 UTC
    6. LOCKLOSS at 23:14 UTC while ASC clearing SDF Diffs.
       1. TJ noticed noisy power recycling gain
13. 14:54 UTC: FMCS Air Handler 3B (Reheat) - FMCS alarm handler came onto red and then 3 mins later went back to normal (tagged FMP)
14. Lock acquisition 4 at 23:15 UTC
    1. Power recycling gain is being noisy (again).

LOG:

Tyler "has recently changed the chiller pumps to not be on at all times and come on when needed. He gave me some data for when the EX chiller has been turning on and off over the last 2.5 weeks" (TJ, 2023). I took this data and plotted it against our range as well as the average internal temperature at EX to see if we would lose range when the chiller at EX turned on(attachment1).

Over this date range, the longest amount of time that the chiller was on consistently was for just over two hours. With both these longer segments and the more regular shorter segments of 'on' time (sometimes just 10 minutes on), there is no noticeable change in the range when the chiller turns on or off. There are multiple times where the range trends downward after the chiller starts turning on for longer stretches of time (red attachment2), but it is not consistent - yellow(attachment3) and green(attachment4) rectangles show the range rising with chiller 'on' times. Therefore I am inclined to attribute the lowering of the range during these times as being due to changes in indoor/outdoor temperatures and wind and not the chiller.

All looks nominal for the site HVAC fans (did catch a transition between fans at MY for this one).  Closing FAMIS 26251.

State of H1: Lock Acquisition

Winds have died down enough to the point where I'm attempting to lock H1. So far so good; there was one lockloss while trying to lock DRMI, but this time currently up to ENGAGE_ASC_FOR_FULL_IFO.

At 9:45am local time Chiller 1 at End Y went into alarm for "Evaporator Water Flow Lost". When I arrived at the EY chiller yard I observed that neither chiller was running but chilled water pump 1 was continuing to run. I noted the alarm and headed for the mezzanine above the AHU's to asses what the supply and return pressures were nearest the evaporator coil. 

Immediately I read 0 (or what I thought was 0) at the return line. This would generally indicate that there has been enough glycol loss within the system that makeup is necessary via the local tank (though I've never seen it get to 0). Until I read that both supply lines were at an alarming 140psi (normal operating pressures for all 4 supply and return float around 30). I immediately phoned Richard to have him command the chilled water pump 1 off to stop the oversupply of chilled water. For reasons not clear to me the disable command via FMCS Compass was not taken at the pump.

I went back to the chiller yard and observed that 1: the pump had not been disabled and 2: pressures at the pump were at around 100psi (normal operating for the current frequency is about 50).

Following that, I manually threw the pump off at the VFD to prevent further runaway of the system. Between the time of noting 140psi and manually throwing the pump off, the system pressure increased to 160psi.

After a thorough walk-down of the system, I elected not to utilize our designed redundancy in chiller 2 and chilled water pump 2 as I was still unaware what was causing the massive overpressure at all supply and return lines. It was also found that the return line was not actually at 0, but instead had made a full rotation and was pegged on the backside of the needle (all of these need replacement now).

Macdonald-Miller was called on site to help asses what the issue might be. Given that there was recent incursions to flow via R. Schofield the strainer was the primary point of concern. We flushed the strainer briefly at the valve and noted a large amount of debris. after a second flush, much less/next to none was noted. This alleviated the system pressure substantially. 

The exact cause of the fault and huge increase of pressure is still not clear. There are a number of flow switches at the chiller. Bryan with Mac-Miller suspects part of the issue may live there, and we are going to pursue this further during our next maintenance window. Work was also performed at the strainer within the chiller where rubber/latex-esque debris was found. Work on Chiller 1 to continue but for now the system and end station is happy on chiller2/CHWP2. Looking at the FMCS screen shows temp's have normalized as of the writing of this log.

T. Guidry B. Haithcox. R. Thompson C. Soike R. McCarthy

J. Kissel, for T. Guidry, R. McCarthy

Just wanted to get a clear separate aLOG in regarding what Corey mentioned in passing in his mid-shift status LHO:72423:

The EY HVAC Air Handler's chilled water pump 1 of 2 failed this morning 2023-08-25 at 9:45a PDT, and thus the EY HVAC system has been shut down for repair at 17:35 UTC (10:35 PDT). The YVEA temperature is therefore rising as it equilibrates with the outdoor temperature; thus far from 64 deg F to 67 deg F.

Tyler, Richard, and an HVAC contractor are on it, actively repairing the system, and I'm sure we'll get a full debrief later.

Note -- we did not stop our OBSERVATION INTENT until 2h 40m hours later 2023-08-25 20:18 UTC (13:18 PDT), when we've gone out to do some commissioning.

Following up on TJ's alog from 8/17 (72293), it was noted that a garbage truck at LSB was quite loud. The seismometers in the LVEA clearly picked up the noise, and it seems to coincide with noise in DARM. It's hard to be certain with only one signal, but it probably warrants further investigation.

Spectrogams attached of one seismometer and DARM for 2 minute time span around when noise was reported:
-Of signal
-Zoomed in with boxes on noise regions
-Stacked with some boxes around correlated noise

Genevieve, Lance, Robert

To further understand the roughly 10Mpc lost to the HVAC (https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=72308), we made several focussed shutdowns today. These manipulations were made during observing (with times recorded) because such HVAC changes happen automatically during observing and also, we were reducing noise rather than increasing it. The times of these manipulations are given below.

One early outcome is that the peak at 52 Hz in DARM is produced by the chilled water pump at EX (see figure). We went out and looked to see if the vibration isolation was shorted, it was not, though there are design flaws (the water pipes arent isolated). We switched from CHWP-2 to CHWP-1 to see if the particular pump was extra noisy. CHWP-1 produced a similar peak in DARM at its own frequency. The peak in accelerometers is also similar in amplitude to the one from the water pump at EY. One possibility is that the coupling at EX is greater because of the undamped cryobaffle at EX.

Friday HVAC shutdowns; all times Aug. 18 UTC

15:26 CS SF1, 2, 3, 4 off

15:30:30 CS SF5 and 6 off

15:36 CS SF5 and 6 on

15:40 CS SF1, 2, 3, 4 back on

16:02 EY AH2 (only fan on) shut down

16:10 EY AH2 on

16:20 EY AH2 off

16:28 EY AH2 on

16:45 EY AH2 and chiller off

16:56:30 EY AH2 and chiller on

17:19:30 EX chiller only off, pump stays on

17:27 EXwater pump CHWP-2 goes off

17:32: EX CHWP-2 back on chiller back on right after

19:34:38 EX chiller off, CHWP-2 pump stays on for a while

19:45 EX chiller back on

20:20 EX started switch from chiller 2 to chiller 1 - slow going

21:00 EX Finally switched

21:03 EX Switched back to original, chiller 1 to chiller 2

Lance, Genevieve, Robert

PEM injections

We are redoing part of the formal PEM injection program, which took place for a week before the run, because the 75-60W change reduced vibration coupling by factors of 2 or 3. On Wednesday, during the commissioning slot, we finished acoustic injections at EY and started shaker injections at EY. Thursday we finished the main shaker injections at EY.

ETMY bias sweep with electronic ground injections

Fig. 1 shows that the ETMY bias setting that minimizes our electronic ground injection coupling has possibly changed a little, but not much, since January.

HVAC shutdown

I shut down the HVAC, site-wide, from about 15:23 to 15:37 Aug 17 UTC. The range increased by a little less than 10 Mpc to a little over 160 Mpc. Elenna posted a spectrum for the best part of the site-wide shutdown (see below). The shutdown and especially the re-start took time, so Fig. 2 shows just the effect of of the chilled water system at EX. Also, we shut down SF4 only (the fan pushing 11,000 CFM at the CS, from 21:18 to 21:32 UTC, but we did not see an improvement in range. We will continue with a focussed study tommorrow in order to better understand the locations of the problems.

Robert did an HVAC off test. Here is a comparison of GDS CALIB STRAIN NOLINES from earlier on in this lock and during the test. I picked both times off the range plot from a time with no glitches.

Improvement from removal of 120 Hz jitter peak, apparent reduction of 52 Hz peak, and broadband noise reduction at low frequency (scatter noise?).

I have attached a second plot showing the low frequency (1-10 Hz) spectrum of OMC DCPD SUM, showing no appreciable change in the low frequency portion of DARM from this test.