An unnecessary trip of the ISI occurs everytime the complete platform is deisolated and then reisolated.
The model code keeps the T240 Saturations out of the watchdog bank for tripping the ISI when ever all the isolation gains are zero. But if the T240s are riled up the Saturations still accumulate. As soon as the T240 Monitor has alerted the Guardian that the T240 has settled and the Guardian then starts Isolating, the watchdog trips because the T240 saturations are too many. This only trips the ISI and so the T240 does not get upset again, and after the operator has untripped the watchdog (clearing the saturations,) the ISI isolates fine.
It seems we missed this loophole, if the HEPI does not trip, the T240s often don't get too upset so it isn't a problem. Otherwise, usually something is happening, EQ, platform restart, etc, and the operator (Jim & Me too) just untrip and chalk it up to whatever.
This should be fixed and I'm sure Jamie/Hugo will have some ideas but I suggest something like adding lines:
reset (push) H1:ISI-{platform}_SATCLEAR
wait 60+ seconds
after line 51 in .../guardian/isiguardianlib/isolation/states.py
ISSUEs: 1) The reset will clear all saturations, not just the T240s. 2) The wait is required because the Saturation bleed off code still has the bug of needing a bleed cycle to execute so any reset can take up to 60 seconds. Wasted time not locking/observing.
Integration Issue #1163 filed
JeffK HughR
Looking closer and studying, it looks like the model has logic to send a reset into the T240 WD when Isolation starts but it may have been fouled with the WD saturation bleed off upgrade done a couple months ago. Continuing.
I just checked and it looks like you have the latest models svn up'ed on your machines. We need to look into the models/code. My notes are attached.
Something that might be the issue: Your version of /opt/rtcds/userapps/release/isi/common/src/WD_SATCOUNT.c is out-dated (see below). It looks like there was a bug fix to the saturation counter code you did not receive. Updating is pretty invasive (recompile/restart all the ISI models). We need to make sure that this will solve all the issues you pointed out first.
controls@opsws2:src 0$ pwd
/opt/rtcds/userapps/release/isi/common/src
On the SVN:
controls@opsws2:src 0$ svn log -l 5 ^/trunk/isi/common/src/WD_SATCOUNT.c
------------------------------------------------------------------------
r11267 | brian.lantz@LIGO.ORG | 2015-08-11 16:36:13 -0700 (Tue, 11 Aug 2015) | 1 line
fixed the CLEAR SATURATIONS bug - cleanup of comments
------------------------------------------------------------------------
r11266 | brian.lantz@LIGO.ORG | 2015-08-11 16:32:19 -0700 (Tue, 11 Aug 2015) | 1 line
fixed the CLEAR SATURATIONS bug
------------------------------------------------------------------------
r11131 | hugo.paris@LIGO.ORG | 2015-07-30 18:37:24 -0700 (Thu, 30 Jul 2015) | 1 line
ISI update detailed in T1500206 part 2/2
------------------------------------------------------------------------
On the computers at LHO
controls@opsws2:src 0$ svn log -l 5 WD_SATCOUNT.c
------------------------------------------------------------------------
r11131 | hugo.paris@LIGO.ORG | 2015-07-30 18:37:24 -0700 (Thu, 30 Jul 2015) | 1 line
ISI update detailed in T1500206 part 2/2
------------------------------------------------------------------------
controls@opsws2:src 0$