Camilla, Erik, Dave:

h1hwsmsr (HWS ITMX and /data RAID) computer froze at 22:14 Thu 04 Jan 2024 PST. The EDC disconnect count went to 88 at this time.

Erik and Camilla have just viewed h1hwsmsr's console, which indicated a HWS driver issue at the time. They rebooted the computer to get the /data RAID NFS shared to h1hwsex and h1hwsmsr1. Currently the ITMX HWS code is not running, we will start it during this afternoon's commissioning break.

One theory of the recent instabilities is the camera_control code I started just before the break to ensure the HWS cameras are inactive (in extenal trigger mode) when H1 is locked. Every minute the camera_control code gets the status of the camera, which along with the status of H1 lets it decide if the camera needs to be turned ON or OFF. Perhaps with the main HWS code getting frames from the camera, and the control code getting the camera status, there is a possible collision risk.

To test, we turn the camera_control code off at noon. I will rework the code to minimize the number of camera operations to the bare minimum.

Ibrahim, Dave, Jonathan:

The CDS-GC NAT router (rtr-msr-cds) stopped running at 06:20 PST this morning. Symptoms are that all compuer networking between CDS and GC is down, but the control room VOIP phones still work.

Jonathan and Ibrahim power cycled rtr-msr-cds and all is working again now.

TITLE: 12/18 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at ??Mpc
OUTGOING OPERATOR: 
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: ??
    Primary useism: ??
    Secondary useism: ??

QUICK SUMMARY:

IFO is in NLN and OBSERVING as of 13:04UTC

Computers in Control Room are NOT working - here is a summary:

- I cannot sign into any of the computers - getting a "password incorrect" despite being able to sign in immediately on my laptop (to write ths).

- DARM FOM is frozen

- Picket fence is frozen

- Teamspeak is frozen

- Cameras are frozen

- No Range is being shown

This seems to be some CDS issue so will call the relevant people to notify and rectify

Other than this, it seems that microseism is on the rise.

Statues of H1: Relocking at CHECK_VIOLINS

11:04UTC H1 called for assistance as the NLN timer had expired with IA already been run. We were at PREP_ASC_FOR_FULL__IFO and I can see that the Violins look huge on DARM, H1 lost lock at MAX_POWER twice this morning/night so that's probably why (tagging SUS), most of the other LLs have been at ALS, presumably these ALS difficulties are from the increased ground motion. I'm going to have to spend some time damping violins it seems.

TITLE: 01/05 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY:

High Microseism (over the 95th percentile) is making acquisition tough.

Seems like best chances for locking is right after an alignment.

Have had mixed luck transitioning states for SEI_CONF (between USEISM & WINDY) for most of the shift.  Overall, I had more luck in WINDY, but still only made it to NLN once--sole lock of the evening of 2.5hrs.

Ran an alignment with non-trivial results in that some states would not complete.  Ended up spending about an hour on alignment.

Picket Fences have been flashing yellow quite a bit (don't recall this yesterday, although I wasn't looking at them as much).

End of shift consisted of frantically trying to get H1 to NLN (lots of quick/early locklosses only around LOCKING ALS) before the end of the shift to no avail.
LOG:

• start of shift:  H1 was down due to commissioning lockloss and was in the middle of an INITIAL ALIGNMENT.
• 0116 Due to consecutive locklosses, transitioned SEI_CONF back to WINDY (from microseism) via the code change for SEI_ENV.
  • 2:00 Back to OBSERVING
• 0423 Lockloss AFTER 2.5hr lock (marking as microseism)
  • ALSx needed a few rounds of INCREASE FLASHES
  • 0457 After 6 consecutive locklosses, switched SEI_CONF from WINDY to USEISM
  • 0521-0624 Initial Alignment
  • 0630 After a few locklosses switched SEI_CONF back to WINDY
  • Finally made it to DRMI, but it did not look well aligned.
  • PRMI eventually locked and was offloading, but then there was a lockloss after about 3min of offloading.
  • 0716 Lockloss at Max Power, 0728 Lockloss at Check AS Shutters
  • For the final 30min of the shift, H1 had trouble getting beyond LOCKING ALS, so I made one more switch to USEISM again.

H1 has been locked for 2hrs with a range just under 165Mpc.  This is with microseism above the 95th percentile just under 1um/s.

After Austin started an Initial Alignment, locking looked optimistic, but then several consecutive locklosses later, I ended up transitioning SEI_CONF node from USEISM --back to--> WINDY.  This required:

• Changing constants in the code for SEI_ENV, LOAD-ing, & INIT-ing

The picket fence has been pretty noisy and flashes yellow for various channels (don't recall seeing this earlier this week).

Violins are just under an order of magnitude higher than they were last night at 3e10^-17.

TITLE: 01/05 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Aligning
OUTGOING OPERATOR: Austin
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 5mph Gusts, 3mph 5min avg
    Primary useism: 0.15 μm/s
    Secondary useism: 0.55 μm/s
QUICK SUMMARY:

Arrived to my shift Mid-Initial Alignment for H1 per Austin's recovery from commissioning lockloss (and due to minor hardship for getting through early steps of ISC-LOCKing).  Status:  Alignment complete and H1 has just passed the DRMI locking steps.  NOTE:  Lockloss at LOW NOISE COIL DRIVERS, & then CHECK AS SHUTTERS.

Austin also mentioned the unique low-freq seismic activity--Microseism was starting a slow decrease 24hrs ago, but in the last 12hrs it started to turn around and is currently back up to where it was 24hrs ago (which is at/above the 95th percentile).  BUT--a unique observation is that the "secondary" microseism band has also increased around the 12hr turnaround.  So because of the odd microseism and rough locking in early states, SEI_CONF is at the USEISM state.  If we want to return SEI_CONF to WINDY, It's not a simple transition to WINDY. TJ shared how we make this transition:

• Open code for SEI_ENV node, and make change to the SEI_ENV.py file
• Comment/uncomment out constants for "sei_conf_nom" and "sei_conf_eq" from USEISM to WINDY
• Save the file.
• LOAD run on SEI_ENV guardian node.
• INIT on SEI_ENV---this will take SEI_CONF back to WINDY.

For much of the above---it most likely not be done tonight, unless the Earth becomes much more quieter seismically for us.

Dave, Erik, Camilla, Jonathan,

During a lock loss Jonathan went to EY and got the network interface configured.  This was enough to get the machine back on the network.  When the machine was replaced yesterday the camera was plugged into the wrong slot on the pci board, this was fixed as well.

Dave, Erik, and Camilla remotely activated and tested the camera.

Some notes:

 * During an iteration of work inside, go outside for cell phone reception something in the door mechanism to EY got stuck and it would not open despite being set to unlocked by the operator and by a key fob.  After many tries it eventually opened.
 * Since the re-keying there is no key in the lock box.

At 15:30:48 PST this afternoon h1hwsex crashed again. It had crashed Sat 23 Dec 2023 and was rebooted this Tue 02 Jan 2024. It ran for just about 2 days before crashing again.

Camilla says it is OK to leave the computer off overnight, Erik will replace this computer with a V1 during tomorrow's commissioning break (Fri 05 Jan 2024).

H1 was unlocked at the time of the crash, which means the HWS camera is acquiring images. Camilla confirmed it is OK to leave the camera in this state, it is the ITM cameras which have been causing noise issues.

To green up EDC and SDF, I started the HWS EX dummy IOC on cdsioc0. This is running as user 'ioc' in a tmux session.

TITLE: 01/04 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Lock Aquisition
INCOMING OPERATOR: Corey
SHIFT SUMMARY:

- 16:17 - GRB short E463149

- 16:51 - Superevent S240104b 

- 17:11 - 18:10 - Fire pump testing began - Tagging DetChar in case this shows up as noise on their end

• Back on and off again intermittenly @ 18:31 - 19:30

- EX saturation @ 22:02

- 22:35 - went into COMMISSIONING for opportunistic DARM measurements while LLO is down due to microseism

• Lockloss @ 22:57 - caused by commissioner measurement

- ISC went through MICH FRINGES a few times without being able to move up, which is my cue to begin running an IA, which is currrently ongoing

- Mystery rise and steady hold of primary microseism is still apparent, coupled with a now rising secondary microseism, I have moved the SEI CONF guardian to USEISM to help make locking less cumbersome

LOG:                                                                                                                                                                                                                                                                                                                                                                                                                                                 

Sheila, Louis

We tried transitioning to the new DARM configuration today slowly to debug the transition from NEW_DARM -> [nominal] DARM. We've had plenty of success transitioning into the new DARM state recently using the ETMY_NLN and NEW_DARM ISC_LOCK guardian states. However, today we encountered several errors when doing so. Some syntax errors cropped up from a few lines in ETMY_NLN (that neither of us recognized) that kept us from moving beyond the main() method. We ran the run() instructions by hand (with the GRD state back in NLN) and fixed the syntax errors in ETMY_NLN and then continued by hand to the NEW_DARM configuration (while the guardian state stayed in NLN) but lost lock when swapping the gains on SUS-ETM{Y,X}_L3_LOCK_L. We confirmed that all the expected DARM loop filters were installed and engaged as expected before moving to ETMX (NEW_DARM) from ETMY. It's not clear to us yet why we lost lock this time around after having virtually no issues switching to the new DARM state recently.

Lockloss @ 22:57 - caused by commissioner measurement.

Jenne, Naoki, Louis, Camilla, Sheila

Here is comparison of the DARM CLEAN spectrum with OM2 hot vs cold. The second screenshot shows a time series of OM2 cooling off.  The optical gain increased by 2%, as was seen in the past (for example 71087).  Thermistor 1 shows that the thermal transient takes much longer (12 + hours) than what thermistor 2 says (2 hours). 

Louis posted a comparison of the calibration between the two states, there are small differences in calibration ~1% (74913).  While the DARM spectrum is worse below 25Hz, it is similar at 70 Hz where we in the past thought that the sensitivity was worse with OM2 cold.  From 100-200 Hz the sensitivity seems slightly better with OM2 cold, some of the peaks are removed by Jenne's jitter subtraction (74879) but there also seems to be a lower level of noise between the peaks (which could be small enough to be a calibration issue).  At high frequency the cold OM2 noise seems worse, this could be because of the squeezing.  We plan to take data with some different squeezing angles tomorow and will check the squeezing angle as part of that.

So, it seems that this test gives us a different conculsion than the one we did in the spring/summer, and that now it seems that we should be able to run with OM2 cold to have better mode matching from the interferometer to the OMC.  We may have not had our feedforwards well tuned in the previous test, or perhaps some other changes in the noise mean that the result is different now.