Reports until 15:29, Tuesday 10 December 2024
H1 CDS
david.barker@LIGO.ORG - posted 15:29, Tuesday 10 December 2024 - last comment - 15:43, Tuesday 10 December 2024(81738)
CDS Maintenance Summary: Tuesday 10th December 2024

WP12239 New VM server

Erik, Jonathan:

The old proxmox machine was taken out of service and repaired. In production it was replaced with a W2275 (aka oaf FE). Please see Jonathan's alog for details. After h0epics was moved, its iocs were started by hand as per wiki instructions:

dust monitors (lvea, lab, ey, ex, dr), weather stations (ex, ey, mx, my) and 3ifo dewpoint sensor.

WP12236 New RGC and Timing Card h1omc0

Jonathan, Erik, EJ, Marc, Daniel, Dave:

h1omc0 was upgraded to a custom RCG5.30 and a new LIGO Timing Card (ver1589). This will permit changing the Duotone frequency from 960,961Hz to 1920,1921Hz in conjection with LLO next week. Erik verified the duotone is currently unchanged at 960,961Hz.

Please see Erik's alog for details.

The RCG upgrade of h1iopomc0 added one slow channel to the DAQ INI (H1:FEC-179_TIMING_CARD_TEMP_DEG_C). A DAQ restart was required, which permitted an EDC change for several pending WPs (marked as *).

WP12243 Slow Controls Timing Card Version Check

Daniel

Daniel changed the slow-controls Beckhoff code to permit the following versions of the LIGO timing card in the FE IO-Chassis

Ver Desc
496 Standard first version timing card
1000 New LIGO DAC timing card
1589 Variable DuoTone frequency timing card

The code was restarted at 10:19. This fixed the persistant EX timing error due to h1susex TCver=1000 and preempted a corner station h1omc0 error with TCver=1589.

WP12164 Add new Guardian PCALX_STAT channels to DAQ*

Tony, TJ, Dave:

The new PCALX_STAT guardian node's DAQ channels were added to the EDC as part of the DAQ restart.

WP12195 Add missing CDSRFM slow channel to DAQ*

Dave:

A new H1EPICS_CDSRFM.ini was loaded, adding the missing channel H1:CDS-RFM_LRS_EX2CS_CHCNT

WP12185 add VACSTAT slow channels to DAQ*

Dave:

A new H1EPICS_VACSTAT.ini added the new state channels to the DAQ.

CDS Hardware Status IOC

Erik, Dave:

Following the upgrade of h1omc0's timing card the CDS HW stat reported problems with this card. We found that one unused bit in the LPTC_STATUS PV was unset in h1omc0 and set in all the other front ends, including h1susex.

All FE except h1omc0 LPTC_STATUS = 0x4fbf ee00 (bit 16 set)
h1omc0 LPTC_STATUS = 0x4fbe ee00 (bit 16 unset)

Now that Daniel has fixed the TCver error in EX, the temporary code in cds_status_ioc.py to verify the only EX error is the TCver error was removed.

DAQ Restart

Jonathan, Erik, EJ, Dave:

The DAQ was restarted primarily for the h1iopomc0 INI change due to its RCG upgrade. Several EDC changes which have been waiting for target-of-opportunity were also done.

This was a messy DAQ restart. Adding h1omc0 to the custom RCG boot server h1vmboot0 inadvertently changed the DAQ FE list to only include two frontends; h1susex and h1omc0. When the 0-leg was started initially DC0 looked good, but then we noticed that on the CDS overview only the DC0 EPICS channels for omc0 and susex were connecting, and Jonathan found GDS0 had lost a lot of channels. Around this time I also restarted the EDC on h1susauxb123 and also found that the expected number of channels was down by about 1000. At this time FW0 was writing tiny frames (only two frontends).

DC0s DAQ configuration was fixed and the 0-leg was restarted. At this point the EDC was out of sync, so it was restarted. This then generated a new H1EDC.ini which necessitated a third and final 0-leg restart (plus another GDS0).

The 1-leg restart was a single restart, with a GDS1 restart also needed.

Change of RCG reporting color on CDS Overview

Dave:

We now have two front end systems in production which are running with a custom version of the RCG. To highlight this, I have changed the non-standard-RCG indicator color from purple to black on the CDS Overview. Please see attached.

Images attached to this report
Comments related to this report
david.barker@LIGO.ORG - 15:36, Tuesday 10 December 2024 (81742)

Channels added to DAQ:

h1iopomc0:

< H1:FEC-179_TIMING_CARD_TEMP_DEG_C 4 16
 

h1edc (+26 chans, 57429 to 57455)

< H1:CDS-RFM_LRS_EX2CS_CHCNT 4 16
< H1:CDS-VAC_STAT_STATE 4 16
< H1:CDS-VAC_STAT_VERSION 4 16
< H1:GRD-PCALX_STAT_ACTIVE 4 16
< H1:GRD-PCALX_STAT_ARCHIVE_ID 4 16
< H1:GRD-PCALX_STAT_CONNECT 4 16
< H1:GRD-PCALX_STAT_ERROR 4 16
< H1:GRD-PCALX_STAT_EXECTIME 4 16
< H1:GRD-PCALX_STAT_INTENT 4 16
< H1:GRD-PCALX_STAT_LOAD_STATUS 4 16
< H1:GRD-PCALX_STAT_MODE 4 16
< H1:GRD-PCALX_STAT_NOMINAL_N 4 16
< H1:GRD-PCALX_STAT_NOTIFICATION 4 16
< H1:GRD-PCALX_STAT_OK 4 16
< H1:GRD-PCALX_STAT_OP 4 16
< H1:GRD-PCALX_STAT_PV_TOTAL 4 16
< H1:GRD-PCALX_STAT_READY 4 16
< H1:GRD-PCALX_STAT_REQUEST_N 4 16
< H1:GRD-PCALX_STAT_SPM_CHANGED 4 16
< H1:GRD-PCALX_STAT_SPM_MONITOR 4 16
< H1:GRD-PCALX_STAT_SPM_TOTAL 4 16
< H1:GRD-PCALX_STAT_STALLED 4 16
< H1:GRD-PCALX_STAT_STATE_N 4 16
< H1:GRD-PCALX_STAT_STATUS 4 16
< H1:GRD-PCALX_STAT_TARGET_N 4 16
< H1:GRD-PCALX_STAT_VERSION 4 16
 

david.barker@LIGO.ORG - 15:40, Tuesday 10 December 2024 (81743)

RESTARTS

Tue10Dec2024
LOC TIME HOSTNAME     MODEL/REBOOT
13:19:44 h1omc0       h1iopomc0   <<< reboot h1omc0
13:19:57 h1omc0       h1omc       
13:20:10 h1omc0       h1omcpi     


13:35:19 h1daqdc0     [DAQ] <<< First 0-leg restart with reduced DAQ config
13:35:33 h1daqfw0     [DAQ]
13:35:33 h1daqtw0     [DAQ]
13:35:34 h1daqnds0    [DAQ]
13:35:43 h1daqgds0    [DAQ]


13:36:49 h1susauxb123 h1edc[DAQ] <<< First EDC restart, also reduced config


13:42:33 h1daqdc0     [DAQ] <<< Second 0-leg restart, full FE config but incorrect EDC
13:42:38 h1daqfw0     [DAQ]
13:42:39 h1daqnds0    [DAQ]
13:42:39 h1daqtw0     [DAQ]
13:42:47 h1daqgds0    [DAQ]


13:44:25 h1susauxb123 h1edc[DAQ] <<< Second EDC restart, now has full config but disagrees with DC0


13:45:24 h1daqdc0     [DAQ] <<< Third 0-leg restart to sync up with EDC
13:45:35 h1daqfw0     [DAQ]
13:45:35 h1daqtw0     [DAQ]
13:45:36 h1daqnds0    [DAQ]
13:45:43 h1daqgds0    [DAQ]
13:46:23 h1daqgds0    [DAQ] <<< second GDS0 restart needed


13:51:20 h1daqdc1     [DAQ] <<< 1-leg restart, all configs good.
13:51:32 h1daqfw1     [DAQ]
13:51:33 h1daqtw1     [DAQ]
13:51:34 h1daqnds1    [DAQ]
13:51:41 h1daqgds1    [DAQ]
13:52:14 h1daqgds1    [DAQ] <<< second GDS1 restart needed
 

david.barker@LIGO.ORG - 15:43, Tuesday 10 December 2024 (81744)

Note that the last DAQ restart was on 15th October 2024, it had been running 56 days 4 hours with no CRC or retransmissions.