Alexa, Nic, Jim, Dave
Alexa reported strange LSC-PD_DOF_MTRX input ramped mux matrix behavior last weekend. Symptoms were: new element values were taken immediately without loading, ramping continued indefinitely. The current theory is that a rogue script is setting H1:LSC-PD_DOF_MTRX_LOAD_MATRIX to 1 frequently. On monday this channel was added to conlog, and conlog reported that during monday afternoon and early evening this EPICS channel was being set to 1 several times a second. We also discovered that the DAQ trends were not showing this high frequency activity, this underreporting is under investigation (only seen in the LOAD channel, all other matrix channels trend correctly).
At 11/10 20:19PST monday evening the rogue loader stopped running and has not restarted since. Initial scans of guardian and home account directories have not found any script which would be loading the matrix frequently. Guardian logs its PV settings and these are seen at very low rates (max 100 per day). We may have to wait for the problem to reappear in order to track it down further. Investigation is continuing.
This reeks of bad guardian programming to me. I would put money on one of the ISC nodes writing a one to that LOAD_MATRIX channel every cycle. Look for that channel name in whatever guardian node is actuating on that matrix, or just grep for it in all ISC nodes.
Jamie, we had seen this behavior with all the ISC guardians set to "DOWN" and/or paused. I don't see any guardian that loads the matrix in the "DOWN" state.