CDS Maintenance Summary: Tuesday 14th February 2023

H1 Upgrade to RCG5.1.1

Jonathan, Erik, EJ, Jim, TJ, Dave:

This morning we upgraded H1 from RCG5.0.5 to RCG5.1.1. The 5.0.5 boot server h1vmboot1 was retired, replaced with the 5.1.1 boot server h1vmboot0.

We had performed several test upgrades using a small number of non-Dolphined front ends. On Friday 10th we upgraded h1susaux[h2, h34, h56, ex, ey] and h1pemmx to RCG5.1.0 (as it was at the time). At the time we encountered several issues resulting from file ownership/permissions. A new RCG5.1.1 was release which is more relaxed file ownership rules was release.

Monday afternoon we upgraded the test front ends again from 5.1.0 to 5.1.1, this time not rebooting the computers, just restarting the models. We also did a full 5.1.1 install and rebooted h1build in preparation for the Tuesday main upgrade. Unfortunately we had several end station front end computers lock up around this time (h1susey, h1seiey, h1iscey, h1susex). To get these systems running again over Monday night we reverted their installs back to 5.0.5. While doing this we found that the DAQ INI file had changed for h1isietmy due model changes since its last restart. 

This morning, starting at 8am, we started the main upgrade. Jim undid the late 2022 changes to h1isiham4 and h1isietmy which we detected Monday evening. Boot and Dolphin management services were stopped on h1vmboot1 and started on h1vmboot0.

First we did a test reboot of h1build, no end station problems were seen. We then rebooted h1ecatmon0 and got the slow controls SDF online again. Because most susaux and pemmx had already been upgraded, these were left alone to feed data to the DAQ to keep that running.

We stopped all of H1 models using the "rcg stop --all" command.

First we rebooted h1susauxb123 because it is running the EDC, needed for vacuum trending.

Next EY was upgraded, using h1vmboot1 to log in and issue the reboot command after the Dolphin switch port was fenced.

After the EY Dolphin fabric checked out, EX was upgraded.

At this point the DAQ was restarted to read the new INI files. RCG5.1.1 added slow channels to every model for IPC diagnostics.

The corner station front ends were booted in the order PSL, SUS, SEI, ISC.

The last machine to be rebooted was h1cdsrfm.

h1iopseih16 would not start, this front end had lost contact with the second Adnaco backplane in its IO Chassis. h1seih16 front end computer and IO Chassis were power cycled, which resolved this problem.

At this point all of the front ends had been upgraded and we had a green oveview with the exeption of WD dackills.

A new H1EPICS_GRD.ini was generated following the creation of the CAMERA_SERVO node.

We did a second DAQ restart, with an EDC restart included.

At 10:40 we handed control of H1 over the the control room.