Ryan C, Dave:

Today's reboot of opslogin0 gave us an opportunity to stop Ryans's test version which had been running on opslogin0, and start the production IOC on cdsioc0.

WP 11213
ECR E2200043

Network and power cables pulled from EY mechanical room to VEA. Part of upgrading HEPI pump controller PLC to Beckhoff.

Prior to start of science run, new weather station stagged in VEA for testing was powered down and unplugged. This was done at both end stations.

WP11210 h1hwinj1 configuration and reboot test

Keith R, Keith T, Erik, Dave:

We have put h1hwinj1 under puppet configuration control. Monit is being used to monitor the status of psinject (CW hardware injector), and will restart it if it is missing. Systemd is used to start/stop the processes, via scripts which ramp the GAIN of the CAL_INJ_CW_EXC filter module over 10 seconds. To test the system works when h1h1inj1 is rebooted, this morning I rebooted using "systemctl reboot". Systemd correctly ramped the CW gain to zero before shutting down psinject. After the reboot monit started psinject correctly.

WP11211 Replace failing ADC h1seih16

Fil, TJ, Jim, Erik, Dave:

FRS27187

Fil replaced the second ADC in h1seih16. It had been raising ADC errors with increasing frequency (lately several times per day).

Procedure was: Jim safed the system, Dave fenced from Dolphin, stopped the models and powered down the computer. Fil powered down the AI chassis, then removed the old ADC and installed the new. Fil powered the IO Chassis up, Dave powered the computer up, Fil then powered the AI chassis up.

WP11205 Restart all digital video camera servers via monit web page

Jonathan, Patrick, Dave:

To test recent monit configuration changes, all the cameras on h1digivideo[0,1,2] were restarted using their monit web page controls.

Procedure was: open the camera image, via monit press the RESTART button, verify camera image goes blue-screen, then returns to original image.

Test was successful for all cameras on h1digivideo[0,1,2]

The camera control web page for the new camera servers on h1digivideo3 was added to the MEDM (see attachment). It was noted that the camera order on the web page did not match that on the MEDM, and that the STOP button did not work.

Jonathan made several fixes and improvements: Camera order matches, STOP/START buttons work, web page shows camera uptime (similar to monit web page).

WP11208 h1daqframes-0 ZPOOL Scrub

Dave:

I started a ZPOOL SCRUB on h1daqframes-0 at the start of maintenance. So far, so good. It is scheduled to complete early Sat.

  scan: scrub in progress since Tue May 23 09:58:19 2023
        18.7T scanned at 1.46G/s, 6.88T issued at 550M/s, 184T total
        0B repaired, 3.75% done, 3 days 21:32:05 to go

WP11219 Replace Dolphin IX Adapter card h1cdsrfm

EJ, Erik, Jonathan, Dave, TJ, Jim, Jeff, Control Room:

Last night we had to reboot several corner station systems because h1cdsrfm glitched the corner Dolphin fabric when reporting a problem with the IX card installed in its local chassis (i.e. not in End Station Adnaco mini-chassis). This had happened before in August 2022. We elected to replace this card as preventative maintenance before O4.

Because the card's serial number had changed, the Dolphin network manager had to be restarted on h1vmboot1. Unfortunately, as seen before, this caused h1susex, h1susey, h1seiey and h1iscey to hard lockup, requiring IPMI resets to reboot them. These system had to be recovered from their reboots by CDS and OPS.

WP11193 Upgrade DC power supplies in CER

Marc, Fil, Dave:

Marc and Fil continued the DC power supply upgrades in the CER. This required the power down of h1omc0, h1asc0 and h1lsc0 front ends because their IO Chassis were powered down.

The procedure was: fence FE from Dolphin, stop all models, power down computer. 

Prior to this I had installed the new h1ascsqzifo and h1sqz models, so then ASC and LSC came back they were running the new code. 

WP11206 ADD ADS block to SQZ

Daniel, Vicky, Dave:

New h1ascsqzifo and h1sqz models were installed on h1asc0 and h1lsc0 respectively. DAQ restart was required to add slow channels.

DAQ Restart

Dave:

The DAQ was restarted because of the sqz model changes detailed above.

fw0 restarted itself after running 6 minutes. We think this was a "standard" fw0 restart and not because of the ongoing scrub.

Both gds0 and gds1 needed restarts, this appears to be modus operandi these days.

DQ Shifter: Brennan Hughey

Summary:

• 36% observing time over the week, with PEM week and other site activities the limiting factor Mon-Thu and earthquakes the limiting factor Fri-Sun.  While there was a lot going on this week, data quality within observing periods actually appears pretty clean for the start of a run, at least from a transient analysis perspective.
• GDS and front-end calibrations visibly match better in the range plots after O4 calibration is pushed, though not at the start of each lock.
• The mid-frequency 50-100 Hz glitching Derek identified last week shows up Monday and Tuesday in the omicron glitchgrams, but with reduced severity.  Didn't reappear the rest of the week.
• Some good things for DetChar to note: the 10 Mpc range increase when HVAC system is turned off, the lockloss due to parametric instability, the wind fence button. 
• Astrophysics:
  • Jenne fixed a squeezer issue and got LHO back into observing 3 minutes before detection of the NSBH candidate.  Thanks Jenne.
  • SN 2023ixf is the closest core-collapse supernova in over a decade.  While it was first identifed 17:27 UTC May 19 the relevant times for GW analysis are hours to days before that (and a current subject of discussion).  My point here is that the burst group is going to be serious about analyzing some of the engineering run data this week. 
• Full report contains notes on individual DQ shift monitoring tools, including some settings we should probably tweak going into O4.

The full report can be found here. 

At this point all maintenance activities have wrapped up. The last lingering tasks are:

• SEI issue at EY that Jim and Jeff are almost done solving
• CS and end station sweeps
• ISC table work after initial alignment

As soon as SEI gives the OK, we will start alignment.

With the update of opslogin0 I updated the nomachine (remote desktop) software on opslogin.

You will see a notice about a certificate fingerprint change when you log in.  This is a one-time thing.

Operating systems packages were updated on Opslogin0 and it was rebooted.

CDS and other LIGO packages including python libraries and control room tools that are distributed through conda were *not* updated.

J. Kissel, T. Shaffer, J. Warner

8 seconds after re-isolating the platform from today's Dolphin crash, H1ISIETMY ST2 WD Tripped.
Currently identifying and debugging the problem...

Using DTT with 0.001 Hz BW and 10 averages (.xml template attached), chi_XY (see LIGO-P2000113 for details) was calculated by hand during several quiet lock stretches of at least 2 hours during the past few days.

Our calculation of chi_XY in the front-end code needs some tuning of the demodulation and filtering parameters as can be seen in the attached plot.  This will get some attention soon; it is the focus of a planned summer SURF student investigation here at LHO.

chi_XY compares the calibrations of the fiducial Pcal displacements induced at both end stations.  Corrections factors have been applied at each end station (H1:CAL-PCALX_XY_COMPARE_CORR_FACT = 0.9988 and H1:CAL-PCALY_XY_COMPARE_CORR_FACT = 1.0015) such that chi_XY should be close to unity.  It appears that both Pcal systems are stable, operating as expected.

F. Mera, M. Pirello

Continuing with WP11193

7 - Installed segregated +24V run from VDC-C2 U34 RHS to SQZ-C1 for the OMC IO Chassis.  This conductively isolates the OMC IO Chassis +24V power from the Beckhoff SQZ +24V power.

8 - Replaced VDC-C2 U24 RHS -18V Kepco which supplies -18V to ISC-C1 & ISC-C2.

H1 VDC Rack Drawing D2300167

This concludes WP11193

J. Kissel, E. von Reis, T. Shaffer,
WP 11219

In prep for the suddenly needed replacement of the corner-station to end-station "CDS RFM" dolphin card, which has a non-zero risk of crashing all end-station models, we've 
    - Accepted the ETMX / TMSX, ETMY / TMSY alignment sliders in the SDF system
    - Brought the ETMX / TMSX, ETMY / TMSY SUS guardians to DAMPED
    - Brought the EX and EY SEI guardian managers to ISI_DAMPED_HEPI_OFFLINE

• Spend the first hour in EQ mode, struggling to lock Yarm from this
• Was unable to get past DRMI, some other small issues (Yarm struggling, COMM IR not being found)
  • Ended up running 2 initial alignments
  • Checked OM alignments, SR3, SRM, tried restoring SRM sliders to a previous lock
  • Ended up doing a manual initial alignment of the SRC and maually adjusted SRM ~300 microradians in P and ~150 in Yaw which fixed the AS AIR image
• Currently relocking at FIND_IR

TJ, Rahul, Austin

Today, we have decided to remove the MONITOR_PAUSE state in the VIOLIN_DAMPING guardian and combine the code into the main damping state - DAMP_VIOLINS_FULL_POWER. The motivation of this was to prevent the waiting in the monitor state and go directly into the damp violins state where the damping settings are set.