After a lockloss while attempting to relock this morning, I ran the dark offset script (/opt/rtcds/userapps/release/isc/h1/scripts/dark_offsets/dark_offsets_exe.py) after taking the IMC offline and shuttering ALS with the hope of fixing some of our ASC troubles.

I've attached a text file of the script output showing what all was changed, as well as screenshots of these being accepted in SDF safe.snap tables (TJ has more that he will add in a comment). These will likely need further accepting once we eventually reach NLN.

Closes FAMIS 37250, the first of this new task. All of the dust monitors look to be working as expected and are seeing counts, aside from LAB2 which is known.

TITLE: 03/05 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 119Mpc
OUTGOING OPERATOR: Oli
CURRENT ENVIRONMENT:
    SEI_ENV state: CALM
    Wind: 5mph Gusts, 3mph 3min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.31 μm/s
QUICK SUMMARY:

• In observing since 15:22 UTC
  • 2 short locks overnight
• Wind and microseism are relatively low
• 16:34 UTC lockloss

Timing things that I don't understand but accepted, ALS WFS turned off, which I reverted in the hopes that this will work again in the next lock,  ESD limiters as expected from 83158

TITLE: 03/05 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Preventive Maintenance
INCOMING OPERATOR: Oli
SHIFT SUMMARY:

IFO is in PREVENTATIVE_MAINTENANCE and LOCKING at DARM_TO_DC_READOUT

We have been unable to lock since conclusion of maintenance. The issue concerns PRMI's ASC not improvign POP counts sufficiently for DRMI to maintain lock.

While wind and microseism was the early culprit and assumption of locking issues, both have calmed down and revealed PRMI_ASC as the problem. Since ~20:30 PT, we have been in a cycle of PRMI and DRMI where PRMI is able to lock and stay locked to move to DRMI, but then DRMI is unable to lock. When we stay at PRMI_ASC, it can stay locked indefinitely.

Elenna, Sheila and I troubleshot this quite exentsively, by moving all relevant suspensions (PRM, BS, PR2). We also ran initial alignment now a total of 3 times without needing to change anything. At the end, Sheila suggested increasing the gain of of LSC_POPAIR_B_RF18 from G = 2 to G = 4. This allowed DRMI triggering to activate a tad earlier than it usually would, which (excitingly) locked DRMI.

Relevant alog: alog 83178. The gain change fix has been commented out. It is on line 564 and 565 on ISC_LOCK.py. The plan is that it will be reverted tomorrow given a solution to the underlying PRMI_ASC not improving.

Other:

• DOF2 P and Y had to be temporarily turned off because they were fighting the ALSY transmission signal, which would cause FIND_IR locklosses (due to slow signal degradation). Turning it off allowed me to get further.
• Wind speeds and microseism are down.

LOG:

None.

IFO is still LOCKING and not able to since MAINTENANCE ended.

At first, this was thought to be a combination of the microseism and wind speeds, but both have since gone down to since lockable states. We can also easily lock ALS and have gone through initial alignment fully automatically twice. What happens is that the PRMI_ASC alignment holds but is not high enough in counts, hovering around 50 and not improving. Then, when we attempt to lock DRMI, we lose lock every time. Elenna and I are troubleshooting the issue now. Certainly doesn't seem like the environment.

PCALY_STAT Guardian node was created, which is a very basic guardian node that just checks a handful of channels to be with a handfull of histoical threshholds to determine if the PCAL subsystem is functioning.
This is just a copy of PCALX_STAT which has been running well for a number of months.

TJ has also put PCALY_STAT on the GUARD_OVERVIEW screen for easy access.

Thank you Dave for fitting us into this last DAQ restart today.

WP12362 Upgrade h1susex to latest 5.30 RCG

To address a difference in the RCG5.3.0 running on h1omc0 and h1susex, all models on h1susex were built against the latest 5.3.0. h1susex was rebooted as part of this upgrade. No DAQ restart was needed.

WP12358 Move cameras to new AMD servers

All the cameras from h1digivideo0 and h1digivideo1 were added to the four already running on h1digivideo4. This brings the number of cameras on this server to 15. All cameras are running at 25 frames-per-second limits.

The cameras on h1digivideo3 (deb11) were moved to a new server, h1digivideo5.

The h1digivideo[0,1,3] machines were powered down. No changes were made to h1digivideo2.

A new camera MEDM was built showing the new layout.

See Jonathan's alog for details.

WP12356 Install independent 1PPS generator at EY

An independed 1PPS function generation was installed at EY and connected to the 5th port of the comparitor, next to the CNS-II clock. This will be used to invistigate when the CNS-II occassionally has a 500nS offset applied.

WP12355 Picket Fence Upgrade

Erik upgraded the picket fence system.

WP12332 BSC Temperature Monitors

Fil worked on the BSC1 (ITMY) temp monitor, it is now reading the correct temp.

DAQ Restart.

A EDC DAQ restart was needed for three changes:

H1EPICS_DIGVIDEO.ini: reflect the move of cameras from old to new software.

H1EPICS_CDSMON.ini: remove h1digivideo3's epics load mon channels, add those for h1digivideo5

H1EPICS_GRD.ini: add channels for new Guardian node PCALY_STAT

The DAQ restart followed our new EDC procedure, whereby the trend-writers were stopped before the edc was restarted with its new configuration.

No major surprises except FW1 sponteneously restarted itself after running for about seven minutes.

Jonathan, Erik, Dave, Patrick

As part of WP12358 we installed two new digivideo server h1digivideo5 and h1digivideo6.  These were racked in the aux rack.  To make room we removed two old conlog servers.  Both servers were installed with Debian 12 and are under puppet configuration and running the 0.1.16 version of the pylon-camera-server software.  They are connected to the new video1 switch.

We migrated cameras off of h1digivideo0, h1digivideo1, and h1digivideo3.  These machines have been powered off.

h1digivideo4 is running the following cameras

H1-VID-CAM-ALS-X
H1-VID-CAM-ALS-Y
H1-VID-CAM-FC-REFL
H1-VID-CAM-MC-REFL
H1-VID-CAM-MC-TRANS
H1-VID-CAM-MC1
H1-VID-CAM-MC2
H1-VID-CAM-MC3
H1-VID-CAM-PR2
H1-VID-CAM-PR3
H1-VID-CAM-PRM
H1-VID-CAM-ISCT6-AS-AIR
H1-VID-CAM-SRM
H1-VID-CAM-SQZT7-OPO-GR-TRANS
H1-VID-CAM-SQZT7-IR-TRANS

h1digivideo5 is running the following cameras

H1-VID-CAM-ETMX
H1-VID-CAM-ETMY
H1-VID-CAM-FC1
H1-VID-CAM-FC2
H1-VID-CAM-FCES-GREEN
H1-VID-CAM-FCES-IR-TRANS-B
H1-VID-CAM-SHG-TRANS
H1-VID-CAM15

We did a daqd restart to remove monitoring information regarding h1digivideo3 and to reflect changes in EPICS variables due to migrating from older versions of the camera software.

We still need to migrate cameras off of h1digivideo2.  That will likely happen next Tuesday.

FAMIS27777

I didn't find any major issues with the pipes themselves, no leaks were found and the supports that were added last year (alog76528) seem to be helping. The flex joints still seem good and the plastic wrap around it also look OK. The same issues reported in the last inspection are still present (alog80015): cuts/scratches in pipes and a blocked outlet under HAM4 that also stressed the pipes there a bit from people pushing them out of the way to use the outlet. After getting Richard's approval, I moved the outlet up on its support beam and on a conduit support further down to clear the TCS pipes and relieved some of the stress on the pipes there - Pic1  Pic2

TITLE: 03/04 Day Shift: 1530-0030 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Preventive Maintenance
INCOMING OPERATOR: Ibrahim
SHIFT SUMMARY: DRMI/PRMI has been giving us issues in the maintenance recovery, PRMI_ASC doesn't seem to actually improve the signal or camera. The wind has been increasing over the past 4 hours and the 2ndary microseism has been decreasing. We've run 1 full IA and a 2nd with only SRC. Some yellow PSL laser room alarms, probably from the wind.
LOG:                                                                                                                                                                      

Some of the maintance items completed today includes:

• Tumbleweed clearing with big green
• EndX compressor test and EndY compressor swap
• PCAL endX station measurement
• LVEA WAP work alog83159
• EX RCG upgrade alog83157
• EndY independent timing source alog83161
• Picket Fence updated and restarted alog83171
• REFL/POP cable delay measurement alog83155
• TCS line adjustment
• ISCT1 table alignment alog83172
• LY vac rack UPS replacement alog83151
• HAM1 SEI rack electronics installation alog83163
• EX chiller maintanance alog83154
• Digivideo upgrades
  • The new camera software has lost us the ability to take screenshots of the cameras in one of the ways we used to, using the channels H1:VID-CAM##_FILE.

TITLE: 03/04 Eve Shift: 0030-0600 UTC (1630-2200 PST), all times posted in UTC
STATE of H1: Preventive Maintenance
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
    SEI_ENV state: EARTHQUAKE
    Wind: 28mph Gusts, 16mph 3min avg
    Primary useism: 0.07 μm/s
    Secondary useism: 0.33 μm/s
QUICK SUMMARY:

IFO is LOCKING and having issues recovering from maintenance with the following being reported from DAY Ops (Ryan C):

• PRMI/DRMI Flashes are low
• Wind is causing ALS to drop out
• IFO was in EQ mode following Iceland EQ (that I don't think would have usually caused EQ mode). TJ just manualled to calm since the EQ was long past and low mag.
• Comissioners are troubleshooting ASC issues that involve lowering thresholds for DRMI lock.