Cheryl reported that IMC wouldn't lock and WFS was breaking the lock.
When I came into the control room the DC alignment of MC seemed OK. Disabled ASC output and manually enabled the PZT offset.
MC2 M3 watchdogs were found tripped, and assuming that this stage is necessary for locking I reset the watchdog.
After this IMC would lock but it lasted for 3 seconds that's incidentally the same as the ramp time for M2 and M3 lock filter. The lock was repeatedly broken by an oscillation at a few Hz that develops right after M2 and M3 lock filters were turned on.
I disabled the L-L element of the drive align matrix of M2 and M1, and the lock would last until the M3 coils rail.
Apparently something changed between this morning and afternoon, and now I had to make the M2 and M1 gain ridiculously small, otherwise a few Hz oscillation just kills the lock.
For the moment I changed the setting in Stefan's lock script such that H1:SUS-MC2_M2_LOCK_L_GAIN is 0.008 (VS the nominal 0.2) and M1_LOCK_L_GAIN is 0.3 (VS the nominal 1). I didn't fix anything, whatever the problem is it's still there, and the lock lasts for only a few minutes at most until M3 coils rail.
I haven't done anything to WFS, I don't know if it was doing something bad, but the DC alignment right now is OK.
Yesterday I was testing some Guardian IMC locking code around the time that Cheryl was trying to fix the alignment. While I don't see how this could have broken anything, it's well within the realm of possibility that it did, so I will look into it.
The guardian code that I was testing is designed to be identical in function to Stefan's locking script. I grep'd through the locking code for all channel access writes and this is what I found (all paths are relative to the userapps root /opt/rtcds/userapps/release):
H1:IMC-:
ioo/common/guardian/states/DOWN/run:ezca.write('REFL_SERVO_IN1GAIN', -10) ioo/common/guardian/states/DOWN/run:ezca.write('REFL_SERVO_COMBOOST', 0)ioo/common/guardian/states/BOOST/run:ezca.write('REFL_SERVO_IN1GAIN', 0) ioo/common/guardian/states/BOOST/run:ezca.write('REFL_SERVO_COMBOOST', 1)H1:SUS-MC2_:
sus/common/guardian/MC2/states/PRELOCK/run:ezca.write('M2_LOCK_L_GAIN', 0) sus/common/guardian/MC2/states/PRELOCK/run:ezca.switch('M2_LOCK_L', 'FM3', 'OFF') sus/common/guardian/MC2/states/PRELOCK/run:ezca.write('M1_LOCK_L_GAIN', 0) sus/common/guardian/MC2/states/PRELOCK/run:ezca.switch('M1_LOCK_L', 'FM1', 'OFF')sus/common/guardian/MC2/states/IMCLOCK/run:ezca.write('M2_LOCK_L_GAIN', 0.2) sus/common/guardian/MC2/states/IMCLOCK/run:ezca.switch('M2_LOCK_L', 'FM3', 'ON') sus/common/guardian/MC2/states/IMCLOCK/run:ezca.write('M1_LOCK_L_GAIN', 1) sus/common/guardian/MC2/states/IMCLOCK/run:ezca.switch('M1_LOCK_L', 'FM1', 'ON')In addition, to the above SUS-MC2 writes, there are additional SUS-MC2 writes performed by the SUS base guardian code (sus/common/guardian/common) via the sustools library (sus/common/guardian/sustools.py). It is my understanding the that sustools.py library has been previously, although maybe not on SUS-MC2. We should consult with the SUS team to confirm that their SUS library is indeed switching things as intended.
Those channels are really all the stuff that guardian would have touched (which I guess is potentially a lot).
I am investigating the remote possiblity that the ezca.switch() python function (part of the guardian ezca library guardian/python/lib/guardian/ezca.py) is writing something strange to the switch settings. I don't think this is happening, since it has been tested, but I'm looking into it just in case. I'll report back here either way.