At 07:43 Wed 12feb2025 PST the psinject process on h1hwinj1 crashed and did not restart automatically. H1 was not locked at this time.

The failure to restart was tracked to an expired leap-seconds file on h1hwinj1. This is a Scientific Linux 7 machine, this OS is obsolete and updated tzdata packages are not available. As a work around to get psinject running again, I hand copied a debian12 version of the /usr/share/zoneinfo/[leapseconds, leap-seconds.list] files. At this point monit was able to successfully restart the psinject systemd process.

In conversation with Mike Thomas at LLO, he had seen this problem several weeks ago and implemented the same solution (hand copy of the leapsecond files). Both sites will schedule an upgrade their hwinj machines to Rocky Linux post O4.

Timeline is (all times PST):

07:43 psinject process fails, H1 is unlocked

08:41 H1 ready for observe, but blocked due to lack of hw-injections

09:34 psinject problem resolved, H1 able to transition to observe.

Lost observation time: 53 minutes.

We cannot definitely say why psinject failed today at LHO and several weeks ago at LLO. Mike suspects a local cache expired, causing the code to re-read the leapseconds file and discover that it had expired 28dec2024.

Post Script:

While investigating the loss of the h1calinj CW_EXC testpoint related to this failure, EJ found that the root file system on h1vmboot1 (primary boot server) was 100% full. We deleted some 2023 logs to quickly bring this down to 97% full. At this time we don't think this had anything to do with the psinject issue.

FAMIS27808

I added 140mL to both chillers, bringing TCS_X from 29.5 to 30.0, and bringing TCS_Y from 9.8 to 10.3. Both the filters were clean and there was no water in the dixie cup.

Wed Feb 12 10:05:23 2025 INFO: Fill completed in 5min 20secs

Very cold morning. TCmins [-61C, -60C] OAT (-8C, 17F) DeltaTempTime 10:05:24

Closes FAMIS#26030, last checked 82671

Everything looking good this week - many sensors that were elevated last week are now lower.

Because of the severe cold, the VEAs have been struggling to hold temperature. I increased the supply air set points of the air handlers so the spaces can hold onto more of the heat they're producing. This is what caused the increase on the trends. I returned the air handlers to auto set point so the temperatures will even out as the morning goes on.  

TITLE: 02/12 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Oli
CURRENT ENVIRONMENT:
   SEI_ENV state: CALM
   Wind: 3mph Gusts, 1mph 3min avg
   Primary useism: 0.02 μm/s
   Secondary useism: 0.34 μm/s
QUICK SUMMARY: Lost lock just before I got in. Looks like ALS has been been unable to stay locked even with good buildup, something we've seen over the last week or so. Running an initial alignment will generally fix this, and so far it has. VEA temperatures are not very stable, but Eric is looking into it.
 

TITLE: 02/12 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Oli
SHIFT SUMMARY:

IFO is in NLN and OBSERVING as of 05:58 UTC

Pretty quiet shift with one EQ caused Lockloss (alog bla bla)

We also dropped out of OBSERVING briefly since SQZ unlocked. The SQZ_OPO_LR was not reaching its count threshold of 80. Following this wiki page, I first attempted to change the SHG temperature in order to increase the SHG power, which was indeed reduced compared to Lock Acquisition time. This did not work, so I ended up reverting my -0.015C change and just set the threshold from 80 to 74 in SQZ params, which did end up working. After this, I retuned the opo temp, which prompted SQZ to automatically re-lock. After SQZ ANGLE guardian optimized the angle, we were able to get back into Lock. There were no SDFs to accept before going back to OBSERVING. Total troubleshooting/outage time was 40 mins, from 02:21 UTC to 03:01 UTC.

LOG:

None

Earthquake caused Lockloss (very local 4.6)

TITLE: 02/12 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Observing at 154Mpc
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 9mph Gusts, 4mph 3min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.42 μm/s
QUICK SUMMARY:

IFO is in NLN and OBSERVING as of 21:26 UTC

Gerardo, Jordan, Camilla 

We disassembled ASSY-D1700340-001 (installed in 39957, removed in 60667) with the plan of re-cleaning the VAC surface and reusing the metal parts with new DAR ZnSe in the upcoming vent. 

Sadly I chipped the 0.5" thick D1700340 window when removing it, photo, will add a note to the ICS record. The dust damaged the in-vac o-ring so we threw away the pair of o-rings but have spares in the LVEA TCS cabinets. 

D1700338 is at C&B, other parts are stored in the PCAL lab VAC storage and the ZnSe will be placed in the LVEA TCS cabinets.. 

Gerardo notified me earlier today he observed an abnormal noise and some wetness coming from the Y+ chilled water pump. When Eric and I arrived we more or less observed the same. It is most likely another failed pump seal which will have to be replaced. One will be ordered shortly assuming demand has lifted and there is available inventory. Pumps were swapped and no interruption to temperatures should be seen.

E. Otterman T. Guidry

Matthew, Camilla

Today we went to EX to try and measure some beam profiles of the HWS beam as well as the refl ALS beam.

Without analyzing the profiles too much, it seemed like (at least the HWS beam) matches previously taken data from Camilla and TJ.

The ALS beam still has the same behavior as reported a few years ago by Georgia and Keita (alog 52608), could not find a referencing alog) who saw two blobs instead of a nice gaussian, just as we see now

TITLE: 02/11 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 156Mpc
INCOMING OPERATOR: Ibrahim
SHIFT SUMMARY: The shift started with the IFO unlocked. I kept it down since we had a full 4 hours of maintenance planned for the day. Relocking presented a few odd ALS issues, but Sheila managed to figure it out (see below). Other than that it was smooth relocking and we have now been observing for and hour.
LOG:

• Relocking notes:
  • Couldn't get ALSX arm power much above 0.6, even though the camera spot looked nice and round. Sheila ended up moving ITMX a bit to get the power up higher, but we still aren't sure why the camera doesnt quite agree. The rest of initial alignment was smooth.
  • ALS WFS autocentering wasn't on when we were starting main locking, even though it was locked and the WFS were engaged. Turning it on brought the WFS to work properly and the arm power raised a bit.
  • Sheila went out to ISCT1 to asjust the POP_X alignment before we went to DC Readout
• 2126 Observing

I made two minor changes to these nodes today:

1. Certain states now check if the Beckhoff locking thinks we have unlocked and the Gaurdian node will then go back to LOCKING. This is a continuation from alog81870.
2. Changed all nds calls to use the call_with_timeout wrapper to avoid any potential hangs.

The POP X PZT was railed after yesterday's PR2 spot move, so Jennie and I went to the table and relieved it using the steering mirror right after the the pZT.

Elenna is looking for some information about the POP X path, so Jennie W and I took this photo and added optic labels to it.

The POP-X readout chain has been moved from 45.5MHz to 36.4MHz. All RF chains have been checked and they look fine.

H1:ASC-POP_X_RF_DEMOD_LONOM was changed from 24 to 23.

Sheila and I are continuing to check various PD calibrations (82260). Today we checked the POP A LF calibration.

Currently there is a filter labeled "to_uW" that is a gain of 4.015. After some searching, Sheila tracked this to an alog by Kiwamu, 13905, with [cnts/W] = 0.76 [A/W] x 200 [Ohm] x 2¹⁶ / 40 [cnts/V]. Invert this number and multiply by 1e6 to get uW/ct.

Trusting our recalibration of IM4 trans, we have 56.6 W incident on PRM. We trust our PRG is about 50 at this time, so 2.83 kW are in the PRC. PR2 transmission is 229 ppm (see galaxy optics page). Then, the HAM1 splitter is 5.4% to POP (see logs like 63523, 63625). So we expect 34 mW on POP. At this time, there was about 30.5 mW measured on POP according to Kiwamu's calibration.