Patrick, Dave
We have been scouring the houly conlog reports of EPICS PVs which change more than 500 times an hour for a reoccurrence of the rapid (many times a second) setting of the LSC ramped mux matrix which was observed last weekend. Unfortunately we have not seen anything yet, we will continue our vigilance.
FYI: here is the report from the last hour. (Remember that a 16Hz front end slow channel, if constanly changing, will give 16*3600 = 57,600 changes per hour.)
Process variables in conlog that have changed more than 500 time(s) in the last 3600 second(s):
H1:SUS-ETMY_L1_LOCK_L_RSET 57541
H1:SUS-ETMX_L1_LOCK_L_RSET 57541
H1:ALS-Y_CAM_ITM_SUM 2361
H1:ALS-Y_CAM_ITM_YAW_POS 2361
H1:ALS-Y_CAM_ITM_PIT_POS 2361
The L1 RSET frequency seems bad to me. I would interperet this to mean that the Guardian is resetting the filter history on that filter bank 16 times a second. Even if this filter bank is unused it indicates some rogue/bad Guardian script.
Rana is correct. Sheila, Alexa and Nic found the DOWN state of the ALS_DIFF guardian node was rapidly clearing the history of the ETM[X,Y]_L1_LOCK_L filters. This started at 8pm last Tuesday (11/11) and was ongoing until it was fixed a couple of hours ago. This highlighted a guardian logging issue with this node which Jamie is looking into.