J. Kissel, J. McIver

After some fussing around with ISI command scripts vs. Watchdog vs. Guardian vs. my remote ssh/MEDM connection (see details below), we've changed the ISI from the guardian uncontrolled state of Isolated on Level 1 (see LHO aLOG 12706) to the guardian controlled state of ISOLATED DAMPED which is
- HEPI controlled with level 1 isolation filters, blended with "pos" position sensor only blends.
- ISI_ST1 control with level 3 isolation filters, blended with "TBetter" on all DOFs but RZ, which is on "TCrappy" (the current standard configuration)
- ISI_ST2 damped only.

The guardian finished the transistion to this state at
Jul 12 2014 20:43:29 UTC

Details
-------
We started this adventure trying to get the ISI-ETMY into both states controlled with *level 2* isolation on both stages, a state which guardian is incapable of creating. As such, I tried doing so using the command scripts. After running ctrlDown to clear out the level 1 isolation, I hit isolate lvl2. Unfortunately, because the SEI group's attention has been focused on guardian, the command scripts have gotten less attention: isolating level 2 consistently trips the watchdog. 

The trip is on the ST1 actuators (see example attached), and happens the instant when ST2 RX RY isolation filters begin their ramp, just after the blend filter switching -- yes, the command scripts (as opposed to the current guardian configuration) DOES still change the blend filters, switching them to "Start" filters. I suspect that its a case of forked processes not being synchronized and that the command scripts are still messing around with alignment biases. ST1 is fully isolated in Start filters (i.e. NOT using the T240s), ST2 alignment biases have just been ramped to some value, creating a large error signal in the isolation loops, the ST2 blend switching is not finished, and when the tilt isolation loops turn on with such a large error signal (even with a gain of 0.1 and the good spline ramping), it rocks ST1 enough that the actuators saturate. We're still tripping on a single sample of actuator saturation?

I tried this three times between Jul 12 2014 19:51:52 UTC and Jul 12 2014 20:08:20 UTC. After which, I gave up, and when on to the guardian controllable state of ISOLATED_DAMPED (as described above). This proved to be a pain for two reasons:
(1) After having been able to open many ISI screens over the double ssh -X connection, I went to the guardian overview screen to change the mode of the SEI_ETMY guardian and its subordinates back to MANAGED. But, for some unknown reason, by the time I got to the subordinates,  I could no longer open guard, and then subsequently any, medm screens. So, I tried from the command line, and that also failed:

jeffrey.kissel@opsws4:~$ guardmedm HPI_ETMY
medm -x -attach -noMsg -dg +0+0 -displayFont -misc-fixed-medium-r-normal--8-60-100-100-c-50-iso8859-1 -macro IFO=H1,SYSTEM=HPI_ETMY,MODULE_PATH=/opt/rtcds/userapps/release/isi/common/guardian/HPI_ETMY.py /ligo/apps/linux-x86_64/guardian-1021/lib/python2.7/site-packages/guardian/GUARD.adl

Could not open Display
jeffrey.kissel@opsws4:~$


I must have just stressed out the ssh connection too much, because only opening an entirely new ssh session remedied the problem. I was able to reproduce this problem, while writing this aLOG --> connection is open for a while, guardian screens cannot open but others can, then no screens can open.

Also of note, the command-line method of what I wanted to do from the MEDM screen, 
jeffrey.kissel@opsws4:~$ guardctrl mode MANAGED HPI_ETMY
doesn't do anything either.  

(2) Once I was able to use MEDM screens again, I switched the subordinates to managed. Soon after doing so, the ISI subordinates claim problems with "filters not in expected state. see log" which said "ISO filters needing to be cleared," even though all isolation filters, inputs and outputs were off, with the gain set to zero. In this state, the manager cannot change anyone's state to clear the problem. To solve the problem, I used the command screen to run ctrldown again. This cleared the warning message, and then I was able to request the manager to go ISOLATED_DAMPED. Note, that I had to manually switch the blend filters back to the above mention configuration before requesting ISOLATED_DAMPED, since the command scripts change them and guardian does not.