Reports until 14:41, Thursday 18 January 2024
H1 CDS
patrick.thomas@LIGO.ORG - posted 14:41, Thursday 18 January 2024 - last comment - 15:03, Thursday 18 January 2024(75464)
Updated the FMCS EPICS IOC code
WP 11626

Dave B., Jonathan H., Patrick T.

Jan. 17 - Jan. 18

The IOC reading out the FMCS data from the BACnet devices and putting it into EPICS has been migrated to the code from ORNL: https://controlssoftware.sns.ornl.gov/bacnet/index.html

As cdsadmin in /home/cdsadmin/github on fmcs-epics-cds I checked out https://github.com/ornl-epics/BACnet.git at this commit:

cdsadmin@fmcs-epics-cds:~/github/BACnet$ git log
commit b0ebd50612faae162a7b3b0dc898cd44c30b9ea3 (HEAD -> main, origin/main, origin/HEAD)
Author: kasemir 
Date:   Thu Aug 10 09:13:50 2023 -0400

    import


In /home/cdsadmin/github/BACnet/configure/ I copied RELEASE-template to RELEASE and in RELEASE changed EPICS_BASE=/home/8w4/epics/base/base-3-14-11 to EPICS_BASE=/usr/lib/epics. Jonathan installed the necessary build tools to compile the IOC and I did so by running make in /home/cdsadmin/github/BACnet.

As cdsadmin in /home/cdsadmin/gitlab I checked out https://git.ligo.org/patrick.thomas/fmcs-bacnet.git. This repository contains our custom FMCS EPICS database files and scripts and it should replace https://redoubt.ligo-wa.caltech.edu/viewvc/projects/trunk/epics/iocs/fmcs_bacnet/. In /home/cdsadmin/github/BACnet I created a db directory and made these soft links in it:

chiller_bacnet.db -> /home/cdsadmin/gitlab/fmcs-bacnet/chiller_bacnet.db
fces_bacnet.db -> /home/cdsadmin/gitlab/fmcs-bacnet/fces_bacnet.db
fmcs_bacnet_bi_to_ai.db -> /home/cdsadmin/gitlab/fmcs-bacnet/fmcs_bacnet_bi_to_ai.db
monitor.db -> /home/cdsadmin/gitlab/fmcs-bacnet/monitor.db

In /home/cdsadmin/github/BACnet/iocBoot/ioctestIoc/ I renamed the st.cmd file to st.cmd.original and created the soft link st.cmd -> /home/cdsadmin/gitlab/fmcs-bacnet/ornl_st.cmd.

The documentation for this BACnet driver code states that it only supports ai, ao, and stringin records. In order to continue to use our bi records that have ONAM and ZNAM strings defined, I created an additional ai record for each of these. Then instead of having the bi records read from the driver code I made the ai records read from it and send their values to the bi records.

In addition I added a pair of keep alive channels, H0:FMC-BACNET_COUNTER and H0:FMC-BACNET_HEARTBEAT. The former increments its value by 1 at 1 Hz. The latter toggles its value between 0 and 1 at 1 Hz.

This driver code also provides monitoring of the traffic to and from each BACnet device. Therefore for each device I created three channels, H0:FMC-BACNET_{device id}_TX, H0:FMC-BACNET_{device id}_RX, and H0:FMC-BACNET_{device id}_ER. These scan at 1 Hz.

This morning I manually started the code, and after correcting a couple of mistakes in the database files, it seemed to be running stably. It is interesting to note that although the scan rate of each of the records had not changed (the majority at .1 Hz), the values seemed to be updating much faster than before. I therefore reduced their scan rate to 1 Hz.

I have seen a number of timeout and other error messages on the console, but I am not certain how to troubleshoot them. The ER diagnostic channels that I looked at read 0, and the values seem to be updating. Since the old code also had error messages, I'm not sure if it is a problem.
Comments related to this report
david.barker@LIGO.ORG - 14:58, Thursday 18 January 2024 (75465)

fmcs-epics-cds puppet policy was updated to run the new IOC. The systemd config for the fmcs_ioc.service is now:
[Unit]
Description=FMCS IOC

[Service]
WorkingDirectory=/home/cdsadmin/BACnet_R0-8/iocBoot/e2b-ioc
# ExecStart=/usr/bin/procServ -f -q -c /home/cdsadmin/BACnet_R0-8/iocBoot/e2b-ioc -p /tmp/fmcs_ioc.pid -n fmcs_ioc --restrict -L /tmp/fmcs_ioc.log 1234 /home/cdsadmin/BACnet_R0-8/iocBoot/e2b-ioc/st.cmd
ExecStart=/usr/bin/procServ -f -q -c /home/cdsadmin/github/BACnet/iocBoot/ioctestIoc -p /tmp/fmcs_ioc.pid -n fmcs_ioc --restrict -L /tmp/fmcs_ioc.log 1234 /home/cdsadmin/github/BACnet/iocBoot/ioctestIoc/st.cmd
User=cdsadmin
Restart=always
RestartSec=60

[Install]
WantedBy=multi-user.target
 

 

david.barker@LIGO.ORG - 15:03, Thursday 18 January 2024 (75466)

After the new FMCS IOC was stable, I restarted the cell phone text alarms system (at 13:14)

The fmcs-ioc-restart code was also started in case of any flatline issues overnight. This code now also resets the epics record alarm fields after any IOC restart.