J. Kissel, K. Venkateswara, K. Izumi, J. Warner While Kiwamu was trying to lock the Michelson, several things went wrong, but it uncovered a flaw in the new gain switching of the GS13s that has been implemented in the Guardian (see LHO aLOG 15537. Here's what happened. (1) Kiwamu incorrectly brought up the BSC ISI in "fully isolated," which turns on stage 2 isolation loops, and switches the GS13s to high-gain mode. (2) As expected, while trying to acquire MICH lock, impulses sent to the SUS BS kick the cage as well, which is attached to the BS ISI ST2, and trips the ST2 on the GS13s watchdog. (3) We then reset the watchdog, and switched to "isolated damped," and this triggered the new guardian feature to *start* switching the GS13s back to low gain, but with MICH still trying to lock, impulses would still trip the watchdog before guardian had the chance to switch *all* of the GS13s gains. (4) This, trigger-happy watchdog resetting, and guardian half transitioning, caused a nasty loop of guardian sloshing the GS13 gains back and forth between high and low, which, with MICH impulses, continued to trip the watchdog. I attach a plot of one of the WD trip, which clearly shows that the V3 GS13 had failed to have its gain switched to low. We should (a) look the new code to make sure there isn't a bad loophole regarding the GS13 gain switching when transitioning from "fully isolated" to "isolated damped" (b) look for ways to increase the switching speed, or add a pause / check that the switch has occured on all GS13s before proceeding with the transition (c) remember that these are physical, several thousand pound systems -- if you have to reset watchdogs repeatedly something is wrong and you don't know why, don't just blindly continue to mash the reset button, figure out what's wrong, or do what Kiwamu did and ask an expert! #justwait
The GAIN and DWT filters' switching mode is set to zero-crossing, with a time-out of 2s (see attachement). Even though Guardian engages the filters properly, they don’t actually switch until a certain time, causing MICH to start acquiring lock before the ISI is ready for it.
This could be solved by selecting the immediately switch mode for the GS13 GAIN and DWT filters. But, after discussing it with Jeff yesterday it turned out that he recalled switching gains with the ISO on, which would be way less stable without zero-crossing.
I modified the SEI guardian to add a 3s wait at the end of the gain switch sequence to give the filters the time they need to switch with the current zero-crossing configuration, before allowing MICH to start acquiring lock.
This fix should be tried next time we start acquiring lock on MICH.
Jeff, Hugo,
The SEI guardian patch I made was tested today. Jeff locked MICH while SEI_BS was in the ISOLATED_DAMPED state (GS13 in low-gain). Once MICH was locked, we switched SEI_BS to FULLY_ISOLATED (GS13s in High Gian, and ST2 Iolation loops turned on). The ISI did not trip, and MICH remained locked.
In order to make sure that this was ra reliable fix, we went ahead and switched the state of the SEI_BS node back and forth between ISOLATED_DAMPED and FULLY_ISOLATED a couple times. Once again, the ISI did not trip, and MICH remained locked.
Time series of the state of the SEI_BS Guardian node, versus the MICH error signal, are attached
The updated code was commited under the SVN:
/opt/rtcds/userapps/release/isi/common/guardian/isiguardianlib/isolation/util.py -r9543
Note: Jamie gave me good feedback on how to improve this new code. The goal here was to make sure it works. I will optimize it once I am back at Stanford.
