We restarted H1DAQ to investigate "invisible" disconnected slow channels which the EDCU could not list by name. Restarting the DAQ caused h1sush2a (MC1,MC3) to go into a bad loop, constantly printing an error messge to the console. It was not ping'able or ssh'able. The error message was

nf_conntrack: table full, dropping packet

...ad infinitum

This is the network filter connection tracking system. We presume that the DAQ restart did something bad at the mx_stream FEDAQ network port on this machine. We had to power cycle to recover h1sush2a. Unfortunately h1sush2a also took down all MSR Dolphin machines, and all mx_streamers needed a restart.

At the end of this bootfest, the H1DAQ EDCU was now connected to all channels, so we could not reproduce the "invisible" disconnected channel issue. Patrick did some dataviewer tests for slow channels and did see some inconsistencies in minute trend data.

The IOChassis for H1 LSC and ASC were connected to the front ends and their timing was connected. We powered up h1asc0 first, and as expected it killed all user models on Dolphin connected computers and timing glitched their IOPs (a standard Dolphin glitch). Before we powered up h1lsc0 for the first time, we tried to disconnect its node from the dolphin fabric, but when it started we again dolphin glitched the MSR.

At the end of this bootfest, I copied the IOP models from L1 to run as h1ioplsc0 and h1iopasc0. There is an IRIG-B timing error on h1lsc0 which will look at later.