Following Tuesday's install of a new Solaris QFS-NFS gateway machine, the DAQ had became more unstable with both frame writers restarting regularly. Today we reverted the system back to its old configuration to see if we can get back to a more stable system for the long weekend and ER9.
h1ldasgw0 NFS was reconfigured to export ldasg-h1-frames to both nds machines
h1ldasgw1 NFS was reconfigured to export cds-h1-frames to both nds machines
h1nds0 processes were stopped, umounted h1ldasgw2 then mounted ldas-h1-frames from h1ldasgw0
h1nds1 processes were stopped, umounted h1ldasgw2 then mounted cds-h1-frames from h1ldasgw1
The new h0vacmx.ini file was incorporated into H0EDCU_VAC.ini.
11:06PDT full DAQ restart.
This was a messy restart, h1psl0 DAQ data was held in a bad state as long as mx_stream was running on h1asc0. This was tracked primarily to monit not running on h1asc0 (it was shutdown while the special asc awgtpman was installed this week). Our /etc/start_streamers.sh file was out of date, it should not attempt to start the mx_stream process itself, rather it should kill the process and allow monit to start it. This file was modified on h1boot accordingly. We are still unsure why we do not see two copies if mx_stream on the FECs that have had start_streamers.sh manually ran. I checked and all FECs now have one, correctly configured mx_stream process. This raised the question of if the data loading is balanced between the two data concentrator ports, as the port allocation is purely on the order of the models in the rtsystab file.
on the test stand x1psl0 machine I manually ran the incorrect /etc/start_streamers.sh and this system behaved as expected. The script started a mx_stream process missing many arguments, and monit attempted to run' /etc/init.d/mx_stream start' which failed because a process of that name already exists. This is not the behaviour we saw on h1psl0.
On x1boot I made the same change to start_streamers.sh to only kill the mx_stream process and leave the restart to monit.