Beckhoff Slow Controls Work:

Daniel, Patrick, Fil, Dave:

The upgrade of EY Beckhoff was stopped following problems. The corner station PLC1 was upgraded to add BUSY signels for the Beam Diverters. A DAQ restart was needed.

h1build reboot

Dave:

I've been getting errors on my daily builds due to the /tmp file system getting full on h1build. This machine had been running for 105 days, so I rebooted it.

h1edc channel stats hosted by new IOC

Dave:

I found that h1edc has problems trending its own EPICS channels (e.g. connected and non-connected counters). I am running a new IOC on h1fescript0 (under a controls screen session) which samples these channels one a minute and reposts them. The new channels have the names H1:CDS-EDCU_CHAN_[CNT, CONN, NOCON, TIMESTAMP]. I removed the h1edc channels and replaced them with the new ones in H1EPICS_DAQ.ini. A DAQ restart was needed.

Attempted scrub of /opt/rtcds ZFS file system

Dave:

The /opt/rtcds ZFS pool on h1fs0 has not been scrubed recently (a file system cleanup, similar to fsck). I started one by hand at 21:23 UTC and immediately saw problems. First it was very difficult to log back into h1fs0 to monitor the status of the scrub. After 3 minutes, the continuous wave psinject stopped running, and the DAQ-CRC error count on h1edc started ramping up from 0 to 115 in 6 steps.

While this was happening I was checking file access times on my workstation, but I could not see any slow down.

At 21:46 UTC I cancelled the zpool scrub.

I cleared the DAQ-CRC error counters. I started the psinject process on h1hwinj1 (as user hinj):

[hinj@h1hwinj1 ~]$ /home/hinj/Details/bin/x_start_psinject david.barker "restart after possible /opt/rtcds slow down"
Entered psinject_common.tcl
Sourcing /ligo/cdscfg/stdenv.tcl
Sourced stdenv.tcl
Sourcing /home/hinj/Details/psinject_config.tcl
Entering psinject_config.tcl
Sourcing /ligo/cdscfg/stdenv.tcl
Sourced stdenv.tcl
Sourced psinject_config.tcl
Note: H1 psinject WAS running as process 23420, but has died
Starting H1 injection into H1:CAL-INJ_CW_EXC using RELEASE -> preO3_H1
Injection will start at t=1240093622 (60 seconds from now)...
Injection starting now
 

Note that h1fs0 has been running since 06 June 2016 (1051 days). We have started planning its reboot/replacement.

DAQ Restart:

Dave:

DAQ was restarted at 12:41 PDT for the ecatc1plc1 and h1edc changes. Again, after the restart all DAQ data from h1sush2a was marked as bad. I restarted mx_stream on this machine and the data became good.

Test of trending h1edc channels in DAQ

Daniel restarted the ecatc1 IOCs at 15:23 PDT, which meant that about 21,000 EPICS channels were taken away from h1edc. This gave me an opportunity to check that the EDCU channel counts were trending correctly in the DAQ using the new IOC. All looks good, plot attached.