jameson.rollins@LIGO.ORG - posted 14:10, Tuesday 05 May 2015 - last comment - 15:30, Wednesday 06 May 2015(18252)
minor guardian upgrade this morning
During maintenance this morning we pushed a fairly minor guardian upgrade. We are now running:
- guardian r1447 (+2 minor MEDM screen changes from r1445 that is running right now)
- cdsutils r474
Primary bugfixes or new features:
- Improve SPM checking. This should get rid of the "SPM flickering" that was being reported at LHO.
- MODE 'EXEC' is now called 'AUTO', to elliminate ambiguity with OP.
- Fix node supervision logging to properly time stamp all logs.
- Add TIME_INIT and TIME_UP channels that record when a node was initialized, and how long it's been up.
The last two are to help debug some of the intermittent issues we've been seeing with node processes hanging for unknown reasons.
The node status indicators on the main GUARD_OVERVIEW screen also now have a couple of extra indicators lights, for STALLED state of node, and if there are any SPM diffs:
Images attached to this report
Comments related to this report
Another new feature I forgot to mention is that guardian can now dump their internal setpoint table to a file.
This is triggered by writing a 1 ('True') to the SPM_SNAP channel for the node:
jameson.rollins@operator1:~ 0$ caput H1:GRD-SUS_ITMY_SPM_SNAP 1
Old : H1:GRD-SUS_ITMY_SPM_SNAP False
New : H1:GRD-SUS_ITMY_SPM_SNAP True
jameson.rollins@operator1:~ 0$ cat /ligo/cds/lho/h1/guardian/archive/SUS_ITMY/SUS_ITMY.spm
H1:SUS-ITMY_M0_TEST_P_SWSTAT IN,OT,DC
H1:SUS-ITMY_M0_TEST_Y_SWSTAT IN,OT,DC
H1:SUS-ITMY_R0_OPTICALIGN_Y_TRAMP 2.0
H1:SUS-ITMY_M0_OPTICALIGN_P_TRAMP 2.0
H1:SUS-ITMY_M0_TEST_P_TRAMP 2.0
H1:SUS-ITMY_M0_OPTICALIGN_Y_TRAMP 2.0
H1:SUS-ITMY_M0_TEST_Y_TRAMP 2.0
H1:SUS-ITMY_M0_OPTICALIGN_P_SWSTAT IN,OF,OT,DC
H1:SUS-ITMY_M0_OPTICALIGN_Y_SWSTAT IN,OF,OT,DC
H1:SUS-ITMY_R0_OPTICALIGN_P_TRAMP 2.0
jameson.rollins@operator1:~ 0$
A '.spm' file is written into the node archive directory (/ligo/cds/<site>/<ifo>/guardian/archive/<node>). The .spm file is a space-separated list of channel/setpoint pairs.
NOTE: the setpoints are accumulate over the running of the node. If the node has just been restarted (or the worker restarted after a STOP command has been issues) the setpoint table will be empty until the node passes through states where it actually writes to a channel. The full set of setpoints will only be fully known once the node runs through all system state code paths. Guardian has the ability to initialize the full set of setpoints for the node for a pre-defined set of channels, but we're not using that fascility yet.
The setpoint table is also written to the log when an SPM_SNAP is triggered, as is a full list of all PVs subscriptions currently active in the node:
2015-05-06T22:16:20.66686 SUS_ITMY W: PVs:
2015-05-06T22:16:20.66698 SUS_ITMY W: H1:SUS-ITMY_DACKILL_STATE = 1.0
2015-05-06T22:16:20.66705 SUS_ITMY W: H1:SUS-ITMY_L1_TEST_L_SW1R = 4.0
2015-05-06T22:16:20.66711 SUS_ITMY W: H1:SUS-ITMY_L1_TEST_P_SW1R = 4.0
2015-05-06T22:16:20.66717 SUS_ITMY W: H1:SUS-ITMY_L1_TEST_Y_SW1R = 4.0
2015-05-06T22:16:20.66723 SUS_ITMY W: H1:SUS-ITMY_L1_WDMON_STATE = 1.0
2015-05-06T22:16:20.66728 SUS_ITMY W: H1:SUS-ITMY_L2_TEST_L_SW1R = 4.0
2015-05-06T22:16:20.66734 SUS_ITMY W: H1:SUS-ITMY_L2_TEST_P_SW1R = 4.0
2015-05-06T22:16:20.66740 SUS_ITMY W: H1:SUS-ITMY_L2_TEST_Y_SW1R = 4.0
2015-05-06T22:16:20.66746 SUS_ITMY W: H1:SUS-ITMY_L2_WDMON_STATE = 1.0
2015-05-06T22:16:20.66751 SUS_ITMY W: H1:SUS-ITMY_L3_TEST_BIAS_SW1R = 4.0
2015-05-06T22:16:20.66757 SUS_ITMY W: H1:SUS-ITMY_L3_TEST_L_SW1R = 4.0
2015-05-06T22:16:20.66763 SUS_ITMY W: H1:SUS-ITMY_L3_TEST_P_SW1R = 4.0
2015-05-06T22:16:20.66768 SUS_ITMY W: H1:SUS-ITMY_L3_TEST_Y_SW1R = 4.0
2015-05-06T22:16:20.66775 SUS_ITMY W: H1:SUS-ITMY_M0_DAMP_L_SW2R = 1728.0
2015-05-06T22:16:20.66781 SUS_ITMY W: H1:SUS-ITMY_M0_DAMP_P_SW2R = 1728.0
2015-05-06T22:16:20.66787 SUS_ITMY W: H1:SUS-ITMY_M0_DAMP_R_SW2R = 1728.0
2015-05-06T22:16:20.66792 SUS_ITMY W: H1:SUS-ITMY_M0_DAMP_T_SW2R = 1728.0
2015-05-06T22:16:20.66798 SUS_ITMY W: H1:SUS-ITMY_M0_DAMP_V_SW2R = 1728.0
2015-05-06T22:16:20.66804 SUS_ITMY W: H1:SUS-ITMY_M0_DAMP_Y_SW2R = 1728.0
2015-05-06T22:16:20.66810 SUS_ITMY W: H1:SUS-ITMY_M0_OPTICALIGN_P_SW1 = 0.0
2015-05-06T22:16:20.66816 SUS_ITMY W: H1:SUS-ITMY_M0_OPTICALIGN_P_SW1R = 12.0
2015-05-06T22:16:20.66821 SUS_ITMY W: H1:SUS-ITMY_M0_OPTICALIGN_P_SW2 = 0.0
2015-05-06T22:16:20.66827 SUS_ITMY W: H1:SUS-ITMY_M0_OPTICALIGN_P_SW2R = 1536.0
2015-05-06T22:16:20.66833 SUS_ITMY W: H1:SUS-ITMY_M0_OPTICALIGN_P_SWSTAT = 302080.0
2015-05-06T22:16:20.66839 SUS_ITMY W: H1:SUS-ITMY_M0_OPTICALIGN_P_TRAMP = 2.0
2015-05-06T22:16:20.66845 SUS_ITMY W: H1:SUS-ITMY_M0_OPTICALIGN_Y_SW1 = 0.0
2015-05-06T22:16:20.66850 SUS_ITMY W: H1:SUS-ITMY_M0_OPTICALIGN_Y_SW1R = 12.0
2015-05-06T22:16:20.66858 SUS_ITMY W: H1:SUS-ITMY_M0_OPTICALIGN_Y_SW2 = 0.0
2015-05-06T22:16:20.66870 SUS_ITMY W: H1:SUS-ITMY_M0_OPTICALIGN_Y_SW2R = 1536.0
2015-05-06T22:16:20.66880 SUS_ITMY W: H1:SUS-ITMY_M0_OPTICALIGN_Y_SWSTAT = 302080.0
2015-05-06T22:16:20.66891 SUS_ITMY W: H1:SUS-ITMY_M0_OPTICALIGN_Y_TRAMP = 2.0
2015-05-06T22:16:20.66903 SUS_ITMY W: H1:SUS-ITMY_M0_TEST_L_SW1R = 4.0
2015-05-06T22:16:20.66913 SUS_ITMY W: H1:SUS-ITMY_M0_TEST_P_SW1 = 0.0
2015-05-06T22:16:20.66923 SUS_ITMY W: H1:SUS-ITMY_M0_TEST_P_SW1R = 4.0
2015-05-06T22:16:20.66933 SUS_ITMY W: H1:SUS-ITMY_M0_TEST_P_SW2 = 0.0
2015-05-06T22:16:20.66940 SUS_ITMY W: H1:SUS-ITMY_M0_TEST_P_SW2R = 1536.0
2015-05-06T22:16:20.66948 SUS_ITMY W: H1:SUS-ITMY_M0_TEST_P_SWSTAT = 300032.0
2015-05-06T22:16:20.66955 SUS_ITMY W: H1:SUS-ITMY_M0_TEST_P_TRAMP = 2.0
2015-05-06T22:16:20.66962 SUS_ITMY W: H1:SUS-ITMY_M0_TEST_R_SW1R = 4.0
2015-05-06T22:16:20.66969 SUS_ITMY W: H1:SUS-ITMY_M0_TEST_T_SW1R = 4.0
2015-05-06T22:16:20.66976 SUS_ITMY W: H1:SUS-ITMY_M0_TEST_V_SW1R = 4.0
2015-05-06T22:16:20.66983 SUS_ITMY W: H1:SUS-ITMY_M0_TEST_Y_SW1 = 0.0
2015-05-06T22:16:20.66995 SUS_ITMY W: H1:SUS-ITMY_M0_TEST_Y_SW1R = 4.0
2015-05-06T22:16:20.66998 SUS_ITMY W: H1:SUS-ITMY_M0_TEST_Y_SW2 = 0.0
2015-05-06T22:16:20.67005 SUS_ITMY W: H1:SUS-ITMY_M0_TEST_Y_SW2R = 1536.0
2015-05-06T22:16:20.67013 SUS_ITMY W: H1:SUS-ITMY_M0_TEST_Y_SWSTAT = 300032.0
2015-05-06T22:16:20.67023 SUS_ITMY W: H1:SUS-ITMY_M0_TEST_Y_TRAMP = 2.0
2015-05-06T22:16:20.67027 SUS_ITMY W: H1:SUS-ITMY_M0_WDMON_STATE = 1.0
2015-05-06T22:16:20.67037 SUS_ITMY W: H1:SUS-ITMY_MASTERSWITCH = 1
2015-05-06T22:16:20.67044 SUS_ITMY W: H1:SUS-ITMY_R0_DAMP_L_SW2R = 1728.0
2015-05-06T22:16:20.67048 SUS_ITMY W: H1:SUS-ITMY_R0_DAMP_P_SW2R = 1728.0
2015-05-06T22:16:20.67055 SUS_ITMY W: H1:SUS-ITMY_R0_DAMP_R_SW2R = 1728.0
2015-05-06T22:16:20.67062 SUS_ITMY W: H1:SUS-ITMY_R0_DAMP_T_SW2R = 1728.0
2015-05-06T22:16:20.67070 SUS_ITMY W: H1:SUS-ITMY_R0_DAMP_V_SW2R = 1728.0
2015-05-06T22:16:20.67077 SUS_ITMY W: H1:SUS-ITMY_R0_DAMP_Y_SW2R = 1728.0
2015-05-06T22:16:20.67088 SUS_ITMY W: H1:SUS-ITMY_R0_OPTICALIGN_P_SW1R = 12.0
2015-05-06T22:16:20.67172 SUS_ITMY W: H1:SUS-ITMY_R0_OPTICALIGN_P_TRAMP = 2.0
2015-05-06T22:16:20.67187 SUS_ITMY W: H1:SUS-ITMY_R0_OPTICALIGN_Y_SW1R = 12.0
2015-05-06T22:16:20.67198 SUS_ITMY W: H1:SUS-ITMY_R0_OPTICALIGN_Y_TRAMP = 2.0
2015-05-06T22:16:20.67210 SUS_ITMY W: H1:SUS-ITMY_R0_TEST_L_SW1R = 4.0
2015-05-06T22:16:20.67226 SUS_ITMY W: H1:SUS-ITMY_R0_TEST_P_SW1R = 4.0
2015-05-06T22:16:20.67229 SUS_ITMY W: H1:SUS-ITMY_R0_TEST_R_SW1R = 4.0
2015-05-06T22:16:20.67239 SUS_ITMY W: H1:SUS-ITMY_R0_TEST_T_SW1R = 4.0
2015-05-06T22:16:20.67250 SUS_ITMY W: H1:SUS-ITMY_R0_TEST_V_SW1R = 4.0
2015-05-06T22:16:20.67261 SUS_ITMY W: H1:SUS-ITMY_R0_TEST_Y_SW1R = 4.0
2015-05-06T22:16:20.67275 SUS_ITMY W: H1:SUS-ITMY_R0_WDMON_STATE = 1.0
2015-05-06T22:16:20.67287 SUS_ITMY W: SPMs:
2015-05-06T22:16:20.67316 SUS_ITMY W: H1:SUS-ITMY_M0_OPTICALIGN_P_SWSTAT = IN,OF,OT,DC
2015-05-06T22:16:20.67326 SUS_ITMY W: H1:SUS-ITMY_M0_OPTICALIGN_P_TRAMP = 2.0
2015-05-06T22:16:20.67353 SUS_ITMY W: H1:SUS-ITMY_M0_OPTICALIGN_Y_SWSTAT = IN,OF,OT,DC
2015-05-06T22:16:20.67363 SUS_ITMY W: H1:SUS-ITMY_M0_OPTICALIGN_Y_TRAMP = 2.0
2015-05-06T22:16:20.67393 SUS_ITMY W: H1:SUS-ITMY_M0_TEST_P_SWSTAT = IN,OT,DC
2015-05-06T22:16:20.67397 SUS_ITMY W: H1:SUS-ITMY_M0_TEST_P_TRAMP = 2.0
2015-05-06T22:16:20.67424 SUS_ITMY W: H1:SUS-ITMY_M0_TEST_Y_SWSTAT = IN,OT,DC
2015-05-06T22:16:20.67432 SUS_ITMY W: H1:SUS-ITMY_M0_TEST_Y_TRAMP = 2.0
2015-05-06T22:16:20.67436 SUS_ITMY W: H1:SUS-ITMY_R0_OPTICALIGN_P_TRAMP = 2.0
2015-05-06T22:16:20.67446 SUS_ITMY W: H1:SUS-ITMY_R0_OPTICALIGN_Y_TRAMP = 2.0
2015-05-06T22:16:20.67455 SUS_ITMY W: 66 PVs, 10 SPMs
2015-05-06T22:16:20.67828 SUS_ITMY W: SPM snapshot: /ligo/cds/lho/h1/guardian/archive/SUS_ITMY/SUS_ITMY.spm