Jamie, Sheila, Dave, Jonathan,
ISC_LOCK was not connecting to H1:ASC-POP_X_RF_Q3_GAIN. From h1guardian1 was was able to connect to this channel with no problems when running caget manually in a login shell. Sheila forced a reconnection of this node's EPICS channels and was able to proceed past this point.
We noticed that htop on h1guardian1 was looking very different to restart times. After a reboot all 40 cores were evenly loaded at about 50%. Now many are in the 70-90% range, and this shifts constantly over the cores.
Jonathan installed his system diagnostics EPICS IOC on the guardian machine this afternoon. We will monitor this via StripTool over the next few days, and add these channels to the DAQ next Tuesday. I would also like to reboot h1guardian1 next tuesday so we can monitor the core usage evolution.
htop core usage shown