I discovered this morning that my new HWS camera control code, which only talks to the camera when it needs to turn the camera on or off, was locking up in its caget of the IFO lock status. The last change was for the lock status function to return both the caget call status and the ifo_lock_state.

I reverted the code back to the previous version, and it is now running correctly on h1hwsex, h1hwsey and h1hwsmsr1 but for an unknown reason is still locking up on caget on h1hwsmsr. When I run the caget command on h1hwsmsr's command line it works. Investigation is continuing.

Note due to this bug HWS ITMX camera was mistakingly left ON overnight until I manually turned it OFF at 08:33 PST this morning.

TJ, Jonathan, Erik, Dave:

yesterday the camera client flashing-blue-screen for cameras whose servers are running on h1digivideo1 steadily got worse as time progressed. This was seen before on 21 Nov 2023, at that time reseating the ethernet cable which connects h1digivideo1 to the cameras fixed it.

We tried, with no success, the following:

• reboot h1digivideo1
• reseat the ethernet cable
• replace the ethernet cable
• move the ethernet cable to a different port on the switch (sw-msr-h1aux 1/1/16 -> 1/1/34)

In all cases the switch port utilization went close to 50%, packets were dropped and camera-clients flashed blue-screen

We then stopped the server processes of three cameras which were not being used often in the control room and were not on any FOM. The port utilization dropped to 36% (still higher than h1digivideo0 (28%) and h1digivideo2 (20%)) but it resolved the flashing.

Current status:

h1digivideo1 eth1 ethernet is temporarily connected to sw-msr-h1aux port 34. Should put this back to port 16.

h1digivideo1 monit is not running the camera service for three cameras:

1. cam17: SRM
2. cam18: SQZ SHG TRANS
3. cam20: SQZ IR TRANS

Thu Jan 11 10:15:36 2024 INFO: Fill completed in 15min 32secs

TITLE: 01/11 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 160Mpc
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
    SEI_ENV state: USEISM
    Wind: 18mph Gusts, 13mph 5min avg
    Primary useism: 0.05 μm/s
    Secondary useism: 0.40 μm/s
QUICK SUMMARY: Locked for 16 hours. useism still trending down, but the time is picking back up a bit.

TITLE: 01/11 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 158Mpc
INCOMING OPERATOR: Corey
SHIFT SUMMARY:

- EX saturations @ 1:21/3:32/4:35/4:46

- Quiet night otherwise, H1 has just reached an 8 hour lock

- Secondary microseism continues its steady decline, now just above 0.5 um/s velocity
LOG:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              

Uneventful night so far other than a couple glitches. The detector appears stable, microseism is still above the 95th percentile but is beginning to trend down.

TITLE: 01/11 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 155Mpc
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
    SEI_ENV state: USEISM
    Wind: 4mph Gusts, 3mph 5min avg
    Primary useism: 0.04 μm/s
    Secondary useism: 0.64 μm/s
QUICK SUMMARY:

- H1 just got back into observing (acquired @ 23:58)

- Secondary microseism is still high, but low enough where we can relock if H1 goes down

- CDS/DMs ok

TITLE: 01/10 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing
INCOMING OPERATOR: Austin
SHIFT SUMMARY: Two locks in the morning, then during commissioning time multiple locks and lock losses while testing a new DARM actuator. We are now back to observing, but violins are high and getting damped.
LOG:

• 1737 Observing
• 1838 Lock loss
• 1942 Observing
• 2008 Commissioning
• 2012 Lock loss
• 2358 Observing

Our new Electric Vehicle charge station was installed Tuesday 1/9/2024.  This is a stand alone solar/battery charger for the site vehicles.

There are two charger ports you can use simultaniously.  You can park under the solar panels on the steel plate but the charger cables will reach the parking spots around the unit.

Louis, Jenne, TJ, Sheila

Today we continued to try to transition to the new Darm configuration, which we had suceeded in doing in December but weren't able to repeat last week (75204). 

In our first attempt today we tried a faster ramp time, 0.1 seconds.  This caused immediate saturation of ETMX ESD.  We struggled to relock because of the environment. 

Because Elenna pointed out that the problem at roughly 3 Hz with the earlier transition attempts might have been the soft loops, we thought of trying to do the transition before the soft loops are engaged, after the other ASC is on.  We tried this first before transitioning to DC readout which wouldn't work because of the DARM filter changes.  Then we made a second attempt at DC readout.  We also lost lock due to a 2 Hz oscialltion, even without the soft loops on. 

Some gps times of transitions and attempt:

• 1387034100  succsesful transition from December
• 1388520329 attempt from last week with 5 second ramp
• 1388520329 attempt from last week to transition to the NEW DARM in TRANSITION_FROM_ETMX and LOWNOISE_ESD_ETMX with a 5 second ramp
• 1388952732 attempt from today to transition with a 0.1 second ramp, imediate saturation of ESD
• 1388961745 attempt at 2W with soft loops off (5 second ramp)

Erik, Dave, Patrick, TJ:

At 22:55 Tue PST the PR3 camera stopped updating. This morning I power cycled the camera and restarted the server process on h1digivideo1.

Soon after the AS AIR camera started blue-screen flashing on clients viewers. Each blue flash was short (less than a second),  the EPICS values did not show any flat-lining. The flash rate was random and about every 5 minutes.

We found nothing untoward on h1digivideo1.

For a first try at fixing this we restarted the server process on h1digivideo1 for this camer (h1cam16). This did not fix it.

For a second try we power cycled the camera via POE control and restarted the server process, again this did not fix it.

Then more cameras on h1digivideo1 started blue screen flashing, so at 13:37 we did a soft reboot of h1digivideo1.

There was concern regarding the temperatures of the areas around the control room. These spaces are supplied from AHU 3. The heaters which supply hot air for the hot deck operation were not running and have not been running for some time because the CFM set point of the AHU was below the duct pressure required to satisfy the flow switches. We increased the CFM from 12,000 to 14,000 CFM and this increased the duct pressure enough to satisfy the switches and enable the heaters. Bear in mind this will create a burning dust odor for a short time.

Commissioning caused lock loss, tripped ETMX HEPI and ISIs. Recovering and relocking now.

1388947149

No obvious cause. Ryan S noted that the LSC-DARM_IN1 channel seems to turn first like we have seen many times in the past.

We've been seeing SQZ angle not optimizing correctly (75245, 75151). At 20:05 Ibrahim took us into commissioning and I tried to change ADF frequency H1:SQZ-ADF_VCXO_FREQ_SET from 1300Hz to 200Hz. The ADF line didn't move from the 1300Hz region, just became noisy when I changed it. PLL didn't lock. Unsure why the ADF wouldn't move, I also tried 800Hz to no avail.  ADF frequency hasn't successfully been changed since Daniel adjusted the model in May 69453.