Fabrice and I did more testing on the HAM4 guardian today. I've been lacking in posting, so I bit of recap. Last week we started testing the ISI_HAMX guardian code on the HAM4 ISI:
USERAPPS/isi/common/guardian/ISI_HAM4.py
This links against the isi/common/guardian/isiguardianlib that Charles Celerier wrote. A graph of the full system is included below.
The code has been behaving very well, but there have been a few discrepancies between the code and the desired mechanical behavior that we're ironing out. As of today we were able to cycle through all the states without tripping the watchdogs. Ultimately a couple of tweaks to the deisolation process did the trick.
It seemed that the shutting off of the ISO Boost_F filters was causing the isolation loops to saturate. We fixed this by adding a 5 second ramp to all ISO Boost_* filters for all degrees of freedom. Similarly, the ramp down of the isolation gains, which happens right after the boosts are shut off, was also happening too quickly. This was particularly true when the platform location was far from the equilibrium position. After a bit of trial and error we were able to survive the full deisolation process with 150um/urad offsets with a 20 second gain ramp to zero. Preferably we'll come up with a more systematic way to set these ramp times down the line, but these should do for the moment.
Some additional comments:
The cart bias offsets are set right before reaching full isolation. However, the guardian reaches its fully isolated state (*_ISOLATED) before the platform location has reached it's final position. I suggest the global guardian behavior would be more robust if the guardian does not transition to the full *_ISOLATED state until the platform reaches the desired location, within some tolerance. This might be accomplished by tuning the loops to ramp slower to the desired offset and eliminate some of the ringing after the ramp.
We're getting a bunch of guardian notifications about ISO filter states during _RESTORYING_CART_BIAS_OFFSETS states.
Another day of testing and I think we're ready to stand up guardian nodes for HAM2 and HAM3 ISIs.
A note about the ISI_HAMX state graph:
There are 36 total states, 7 of which are "requestable" (darker color blue and purple in the graph).
There are three potential isolation levels, reprented by the three states ROBUST_ISOLATED, MEDIUM_ISOLATED, HIGH_ISOLATED. There are 6 intermediate states for each isolation between the DAMPED state and the fully isolated states.
Here is a guardutil "print" of the module, which lists all code loaded from USERAPPS
controls@operator1:~ 0$ guardutil print ISI_HAM4 ifo: H1 name: ISI_HAM4 path: /opt/rtcds/userapps/release/isi/common/guardian/ISI_HAM4.py prefix: ISI-HAM4 usercode: /opt/rtcds/userapps/release/isi/common/guardian/isiguardianlib/const.py /opt/rtcds/userapps/release/isi/common/guardian/isiguardianlib/errormessage.py /opt/rtcds/userapps/release/isi/common/guardian/isiguardianlib/util.py /opt/rtcds/userapps/release/isi/common/guardian/isiguardianlib/masterswitch/const.py /opt/rtcds/userapps/release/isi/common/guardian/isiguardianlib/masterswitch/__init__.py /opt/rtcds/userapps/release/isi/common/guardian/isiguardianlib/isolation/const.py /opt/rtcds/userapps/release/isi/common/guardian/isiguardianlib/ligoblend.py /opt/rtcds/userapps/release/isi/common/guardian/isiguardianlib/isolation/__init__.py /opt/rtcds/userapps/release/isi/common/guardian/isiguardianlib/watchdog/const.py /opt/rtcds/userapps/release/isi/common/guardian/isiguardianlib/watchdog/__init__.py /opt/rtcds/userapps/release/isi/common/guardian/isiguardianlib/damping/const.py /opt/rtcds/userapps/release/isi/common/guardian/isiguardianlib/decorators.py /opt/rtcds/userapps/release/isi/common/guardian/isiguardianlib/watchdog/decorators.py /opt/rtcds/userapps/release/isi/common/guardian/isiguardianlib/damping/decorators.py /opt/rtcds/userapps/release/isi/common/guardian/isiguardianlib/isolation/decorators.py /opt/rtcds/userapps/release/isi/common/guardian/isiguardianlib/masterswitch/util.py /opt/rtcds/userapps/release/isi/common/guardian/isiguardianlib/masterswitch/states.py /opt/rtcds/userapps/release/isi/common/guardian/isiguardianlib/watchdog/util.py /opt/rtcds/userapps/release/isi/common/guardian/isiguardianlib/damping/__init__.py /opt/rtcds/userapps/release/isi/common/guardian/isiguardianlib/watchdog/states.py /opt/rtcds/userapps/release/isi/common/guardian/isiguardianlib/damping/util.py /opt/rtcds/userapps/release/isi/common/guardian/isiguardianlib/damping/states.py /opt/rtcds/userapps/release/isi/common/guardian/isiguardianlib/isolation/util.py /opt/rtcds/userapps/release/isi/common/guardian/isiguardianlib/cartbias.py /opt/rtcds/userapps/release/isi/common/guardian/isiguardianlib/isolation/states.py /opt/rtcds/userapps/release/isi/common/guardian/isiguardianlib/masterswitch/edges.py /opt/rtcds/userapps/release/isi/common/guardian/isiguardianlib/watchdog/edges.py /opt/rtcds/userapps/release/isi/common/guardian/isiguardianlib/damping/edges.py /opt/rtcds/userapps/release/isi/common/guardian/isiguardianlib/isolation/edges.py /opt/rtcds/userapps/release/isi/common/guardian/isiguardianlib/HAM/states.py states (*=requestable): 0 INIT * 1 ROBUST_ISOLATED * 2 OFFLINE * 3 READY * 4 MEDIUM_ISOLATED * 5 HIGH_ISOLATED * 6 DAMPED * 7 LOADING_CART_BIAS_FOR_ISOLATION 8 ENGAGING_FIRST_MEDIUM_ISOLATION_FILTERS 9 ROBUST_RESTORING_CART_BIAS_OFFSETS 10 RAMPING_DAMPING_FILTERS_UP 11 WATCHDOG_TRIPPED_DEISOLATING 12 DISENGAGING_DAMPING_LOOPS 13 ENGAGING_FIRST_ROBUST_ISOLATION_FILTERS 14 MEDIUM_RESTORING_CART_BIAS_OFFSETS 15 WATCHDOG_TRIPPED_FULL_SHUTDOWN 16 ENGAGING_SECOND_ROBUST_ISOLATION_FILTERS 17 RAMPING_FIRST_ROBUST_ISOLATION_FILTERS_UP 18 RAMPING_FIRST_HIGH_ISOLATION_FILTERS_UP 19 HIGH_DISENGAGING_BOOST 20 RAMPING_SECOND_HIGH_ISOLATION_FILTERS_UP 21 TURNING_ON 22 HIGH_RESTORING_CART_BIAS_OFFSETS 23 MEDIUM_DISENGAGING_BOOST 24 TURNING_OFF 25 ENGAGING_SECOND_HIGH_ISOLATION_FILTERS 26 RAMPING_SECOND_ROBUST_ISOLATION_FILTERS_UP 27 DEISOLATING 28 ENGAGING_FIRST_HIGH_ISOLATION_FILTERS 29 ENGAGING_SECOND_MEDIUM_ISOLATION_FILTERS 30 RAMPING_DAMPING_FILTERS_DOWN 31 ENGAGING_DAMPING_LOOPS 32 WATCHDOG_TRIPPED_DAMPING 33 RAMPING_SECOND_MEDIUM_ISOLATION_FILTERS_UP 34 ROBUST_DISENGAGING_BOOST 35 RAMPING_FIRST_MEDIUM_ISOLATION_FILTERS_UP