WP11210 h1hwinj1 configuration and reboot test

Keith R, Keith T, Erik, Dave:

We have put h1hwinj1 under puppet configuration control. Monit is being used to monitor the status of psinject (CW hardware injector), and will restart it if it is missing. Systemd is used to start/stop the processes, via scripts which ramp the GAIN of the CAL_INJ_CW_EXC filter module over 10 seconds. To test the system works when h1h1inj1 is rebooted, this morning I rebooted using "systemctl reboot". Systemd correctly ramped the CW gain to zero before shutting down psinject. After the reboot monit started psinject correctly.

WP11211 Replace failing ADC h1seih16

Fil, TJ, Jim, Erik, Dave:

FRS27187

Fil replaced the second ADC in h1seih16. It had been raising ADC errors with increasing frequency (lately several times per day).

Procedure was: Jim safed the system, Dave fenced from Dolphin, stopped the models and powered down the computer. Fil powered down the AI chassis, then removed the old ADC and installed the new. Fil powered the IO Chassis up, Dave powered the computer up, Fil then powered the AI chassis up.

WP11205 Restart all digital video camera servers via monit web page

Jonathan, Patrick, Dave:

To test recent monit configuration changes, all the cameras on h1digivideo[0,1,2] were restarted using their monit web page controls.

Procedure was: open the camera image, via monit press the RESTART button, verify camera image goes blue-screen, then returns to original image.

Test was successful for all cameras on h1digivideo[0,1,2]

The camera control web page for the new camera servers on h1digivideo3 was added to the MEDM (see attachment). It was noted that the camera order on the web page did not match that on the MEDM, and that the STOP button did not work.

Jonathan made several fixes and improvements: Camera order matches, STOP/START buttons work, web page shows camera uptime (similar to monit web page).

WP11208 h1daqframes-0 ZPOOL Scrub

Dave:

I started a ZPOOL SCRUB on h1daqframes-0 at the start of maintenance. So far, so good. It is scheduled to complete early Sat.

  scan: scrub in progress since Tue May 23 09:58:19 2023
        18.7T scanned at 1.46G/s, 6.88T issued at 550M/s, 184T total
        0B repaired, 3.75% done, 3 days 21:32:05 to go

WP11219 Replace Dolphin IX Adapter card h1cdsrfm

EJ, Erik, Jonathan, Dave, TJ, Jim, Jeff, Control Room:

Last night we had to reboot several corner station systems because h1cdsrfm glitched the corner Dolphin fabric when reporting a problem with the IX card installed in its local chassis (i.e. not in End Station Adnaco mini-chassis). This had happened before in August 2022. We elected to replace this card as preventative maintenance before O4.

Because the card's serial number had changed, the Dolphin network manager had to be restarted on h1vmboot1. Unfortunately, as seen before, this caused h1susex, h1susey, h1seiey and h1iscey to hard lockup, requiring IPMI resets to reboot them. These system had to be recovered from their reboots by CDS and OPS.

WP11193 Upgrade DC power supplies in CER

Marc, Fil, Dave:

Marc and Fil continued the DC power supply upgrades in the CER. This required the power down of h1omc0, h1asc0 and h1lsc0 front ends because their IO Chassis were powered down.

The procedure was: fence FE from Dolphin, stop all models, power down computer. 

Prior to this I had installed the new h1ascsqzifo and h1sqz models, so then ASC and LSC came back they were running the new code. 

WP11206 ADD ADS block to SQZ

Daniel, Vicky, Dave:

New h1ascsqzifo and h1sqz models were installed on h1asc0 and h1lsc0 respectively. DAQ restart was required to add slow channels.

DAQ Restart

Dave:

The DAQ was restarted because of the sqz model changes detailed above.

fw0 restarted itself after running 6 minutes. We think this was a "standard" fw0 restart and not because of the ongoing scrub.

Both gds0 and gds1 needed restarts, this appears to be modus operandi these days.