The nominal value of the squeezer laser diode current was change to 1.863 from 1.95. The tolerance is unchanged at 0.1. Looking at trends we sometimes read a value just below 1.85 leading to a failed laser condition which in turn triggers a relocking of the squeezer laser. However, since we are already locked, all we see is the fast and commen gains ramping down and up.

2016 H1 lost lock due to 7.1 south pacific earthquake.

We rode through the P & S waves....the R wave is still about 30+min away and it will be "spicy" ~Jim W.

I have taken H1 to IDLE....while we wait for the R wave to pass & for the planet to calm down after that.

Attached are some screenshots Tony snapped.

Closes FAMIS 21128

•  TCS X
  • Before: 29.5
  • After: 30
  • Added: 60 ml
• TCS Y
  • Read 10.1, did not add any water.

The cup in back corner of mezzanine was empty

Mon Aug 28 10:10:25 2023 INFO: Fill completed in 10min 20secs

Travis confirmed a good fill curbside.

FAMIS 19991

Jason was in the anteroom last Tuesday for inventory, which is seen by the environmental trends. Since then, the differential pressure between the anteroom and the laser room has been elevated, but not alarmingly so.

Well pump is running to replenish the fire water tank. The pump will run for 4 hours and automatically shut down. 

TITLE: 08/28 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Tony
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 4mph Gusts, 2mph 5min avg
    Primary useism: 0.01 μm/s
    Secondary useism: 0.12 μm/s
QUICK SUMMARY:

H1 lost lock at 1152utc (452amPT) and literally as I type it was at Nominal Low Noise & waiting for the CAMERA_SERVO to have its channels converge!  Camera Servo completed, so I just took H1 to OBSERVE at 1506utc (806amPT).  Scanning Verbal it looks like H1 made several lock attempts over the 3hrs and made it fairly far into locking each time (Power Up and a little beyond) before locklosses.  Does not look like an Initial Alignment was run overnight.  Looks like Tony received no locking alerts overnight.

Big picture for the range is our locks over the last day or two (153Mpc and about 146Mpc at the very beginning of ths current lock).

TITLE: 08/27 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 154Mpc
INCOMING OPERATOR: Tony
SHIFT SUMMARY:

- Arrived with H1 locked for 17.5 hours

- 0:21 UTC - ISS pump turned off, followed alog 70050 to troubleshoot

• Reconciled, saved the OPO temp SDF here, back in OBSERVE @ 0:34 - Tagging SQZ

- Lockloss @ 1:08

- Relocking was troublesome, had to tinker a lot with ALS Y to try and find light on the PD. After a few slider restorations and trial/error attempts later, I was finally able to get it to catch. Writing down the golden sus values here for future reference:

• ETMY P: 135.2 Y: -160.8, TMSY P: 66.2 Y: -280.9, ITMY P: -10.7 Y: -17.1
• Seeing a slight swing in temps at EY, but nothing too egregious and looks to have tapered off - Tagging FMCS, scope attached

- Rest of unlocking went unaided, acquired NLN @ 2:38, OBSERVE @ 2:54

• Attaching some SQZ SDF DIFFS, these were caused by me accidently reverting the wrong suspensions trying to do a sus revert. Will be reverting all these changes - 1 / 2 / 3

- 102 Hz peak is apparent for this lock, screenshot

- EX saturation @ 4:43/5:07

- Microseism looks to be on the rise, but otherwise all looks nominal. Setting H1 to managed to handing off to Tony for the night.

LOG:                                                                                                            

Following a lockloss followed by a cumbersome relocking process, we have just made it back to observing as of 2:54 UTC. 102 Hz peak is apparent for this lock - ss attached.

Lockloss @ 1:08, no obvious cause.

TITLE: 08/27 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 160Mpc
INCOMING OPERATOR: Austin
SHIFT SUMMARY:

Nice shift riding through breezes and Colombian seismic waves.  If there's a lockloss, Dave may take the opportunity to clear up CRC on the CDS overview for SUSH2B.
LOG:

• 1819utc/1119amPT Stand Down alert appears (1034-1134amPT stand down time listed)
• 2153 Hints of an EQ with a glitch, picket fence, Peakmon, and L1 going down. (M5.5 from Colombia which did not come up on USGS for many minutes)
  • 1503 Colombia EQ on Verbal (still not on USGS)
  • Also seeing on PRM & SRM on nuc28's Tidal Striptool
  • 1517 EQ Mode Activated
  • 1527 Back to CALM
• 22:45 Giving a tour for my salsa dancing friend in town.

TITLE: 08/27 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 152Mpc
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 9mph Gusts, 6mph 5min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.10 μm/s
QUICK SUMMARY:

- H1 has been locked for just under 17 hours

- DMs/CDS ok

- Was able to ride out a 5.7 EQ from Colombia, just riding out the remnants of it

Quiet shift; winds slightly picking up.  Chatted with Dave regarding (1) RED CRC (Cyclic Redundancy Check) for SUSH2B (He will address this after a future lockloss.) and (2) h1digivideo3 memory issue/bumping us out of Observing.

As Corey mentioned, the DAQ stream from h1sush2b to DC0 had a checksum mismatch in one of its 1/16th second data blocks during the  05:56:09 PDT second. The data stream to DC1 did not see this issue.

This was a true data error for this data block, which meant that FW0 wrote a different frame than FW1 for the frame which encompassed this 1/16th of a second. We know that FW1's frame is correct, FW0's frame has bad h1sush2b data.

For the full frame file H-H1_R-1377176128-64.gwf:

Interestingly the correct frame is smaller than the bad frame, meaning the corrupt signal did not compress as well as the actual signal.

At 9:45am local time Chiller 1 at End Y went into alarm for "Evaporator Water Flow Lost". When I arrived at the EY chiller yard I observed that neither chiller was running but chilled water pump 1 was continuing to run. I noted the alarm and headed for the mezzanine above the AHU's to asses what the supply and return pressures were nearest the evaporator coil. 

Immediately I read 0 (or what I thought was 0) at the return line. This would generally indicate that there has been enough glycol loss within the system that makeup is necessary via the local tank (though I've never seen it get to 0). Until I read that both supply lines were at an alarming 140psi (normal operating pressures for all 4 supply and return float around 30). I immediately phoned Richard to have him command the chilled water pump 1 off to stop the oversupply of chilled water. For reasons not clear to me the disable command via FMCS Compass was not taken at the pump.

I went back to the chiller yard and observed that 1: the pump had not been disabled and 2: pressures at the pump were at around 100psi (normal operating for the current frequency is about 50).

Following that, I manually threw the pump off at the VFD to prevent further runaway of the system. Between the time of noting 140psi and manually throwing the pump off, the system pressure increased to 160psi.

After a thorough walk-down of the system, I elected not to utilize our designed redundancy in chiller 2 and chilled water pump 2 as I was still unaware what was causing the massive overpressure at all supply and return lines. It was also found that the return line was not actually at 0, but instead had made a full rotation and was pegged on the backside of the needle (all of these need replacement now).

Macdonald-Miller was called on site to help asses what the issue might be. Given that there was recent incursions to flow via R. Schofield the strainer was the primary point of concern. We flushed the strainer briefly at the valve and noted a large amount of debris. after a second flush, much less/next to none was noted. This alleviated the system pressure substantially. 

The exact cause of the fault and huge increase of pressure is still not clear. There are a number of flow switches at the chiller. Bryan with Mac-Miller suspects part of the issue may live there, and we are going to pursue this further during our next maintenance window. Work was also performed at the strainer within the chiller where rubber/latex-esque debris was found. Work on Chiller 1 to continue but for now the system and end station is happy on chiller2/CHWP2. Looking at the FMCS screen shows temp's have normalized as of the writing of this log.

T. Guidry B. Haithcox. R. Thompson C. Soike R. McCarthy

J. Kissel, for T. Guidry, R. McCarthy

Just wanted to get a clear separate aLOG in regarding what Corey mentioned in passing in his mid-shift status LHO:72423:

The EY HVAC Air Handler's chilled water pump 1 of 2 failed this morning 2023-08-25 at 9:45a PDT, and thus the EY HVAC system has been shut down for repair at 17:35 UTC (10:35 PDT). The YVEA temperature is therefore rising as it equilibrates with the outdoor temperature; thus far from 64 deg F to 67 deg F.

Tyler, Richard, and an HVAC contractor are on it, actively repairing the system, and I'm sure we'll get a full debrief later.

Note -- we did not stop our OBSERVATION INTENT until 2h 40m hours later 2023-08-25 20:18 UTC (13:18 PDT), when we've gone out to do some commissioning.

TITLE: 08/25 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 147Mpc
OUTGOING OPERATOR: Tony
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 14mph Gusts, 11mph 5min avg
    Primary useism: 0.05 μm/s
    Secondary useism: 0.07 μm/s
QUICK SUMMARY: H1 has been locked for 46 hours.

• All systems look good; a few earthquakes rolling through this morning but wind is low

Genevieve, Lance, Robert

To further understand the roughly 10Mpc lost to the HVAC (https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=72308), we made several focussed shutdowns today. These manipulations were made during observing (with times recorded) because such HVAC changes happen automatically during observing and also, we were reducing noise rather than increasing it. The times of these manipulations are given below.

One early outcome is that the peak at 52 Hz in DARM is produced by the chilled water pump at EX (see figure). We went out and looked to see if the vibration isolation was shorted, it was not, though there are design flaws (the water pipes arent isolated). We switched from CHWP-2 to CHWP-1 to see if the particular pump was extra noisy. CHWP-1 produced a similar peak in DARM at its own frequency. The peak in accelerometers is also similar in amplitude to the one from the water pump at EY. One possibility is that the coupling at EX is greater because of the undamped cryobaffle at EX.

Friday HVAC shutdowns; all times Aug. 18 UTC

15:26 CS SF1, 2, 3, 4 off

15:30:30 CS SF5 and 6 off

15:36 CS SF5 and 6 on

15:40 CS SF1, 2, 3, 4 back on

16:02 EY AH2 (only fan on) shut down

16:10 EY AH2 on

16:20 EY AH2 off

16:28 EY AH2 on

16:45 EY AH2 and chiller off

16:56:30 EY AH2 and chiller on

17:19:30 EX chiller only off, pump stays on

17:27 EXwater pump CHWP-2 goes off

17:32: EX CHWP-2 back on chiller back on right after

19:34:38 EX chiller off, CHWP-2 pump stays on for a while

19:45 EX chiller back on

20:20 EX started switch from chiller 2 to chiller 1 - slow going

21:00 EX Finally switched

21:03 EX Switched back to original, chiller 1 to chiller 2

Robert did an HVAC off test. Here is a comparison of GDS CALIB STRAIN NOLINES from earlier on in this lock and during the test. I picked both times off the range plot from a time with no glitches.

Improvement from removal of 120 Hz jitter peak, apparent reduction of 52 Hz peak, and broadband noise reduction at low frequency (scatter noise?).

I have attached a second plot showing the low frequency (1-10 Hz) spectrum of OMC DCPD SUM, showing no appreciable change in the low frequency portion of DARM from this test.