It seems as though an oscialltion in the BS (visible on the stage 2 GS13) caused a DRMI lock loss. The oscillation seems to be in the ISI channels, but I don't see it in the control signals or on the OpLevs.
HAM3 also just tripped, we don't think there is anyone in the LVEA right now
Chris, Kiwamu, Suresh, Sheila
We have had several more of these odd HAM3 trips tonight. We chased it down to a problem in the IMC_LOCK guardian, where a huge impulse is sent to MC2 in the down state. We are strongly suspicious of a the lines where the gain of MC2_M2_LOCK_L and M1_LOCK_L are set to zero, and the integrator in M1 and boost in M2 are switched off. Both of these filters are switched with zero history, immediately. We changed the boost in M2 to have the output ramped off over 5 seconds. We also changed the code to nicely clear the integrator in M1:
ezca.switch('SUS-MC2_M1_LOCK_L', 'INPUT', 'OFF')
ezca.get_LIGOFilter('SUS-MC2_M1_LOCK_L').ramp_gain(0, 5)
ezca['SUS-MC2_M1_LOCK_L_RSET'] = 2
ezca.switch('SUS-MC2_M1_LOCK_L', 'FM1', 'OFF')
ezca['SUS-MC2_M1_LOCK_L_GAIN'] = -1
We haven't loaded this, since by the time we figured this out we had already moved onto locking DRMI.
One thing that is different tonight is that we started trying to use the ISC_LOCK guardian, which is managing the IMC_LOCK guardian. It's not clear why this would have caused these kicks to the IMC. There is some kind of bug in the way that the ISC_LOCK is managing its subordinates, so sometimes they hang up, and do not leave a state which has finished even though there is a path to the requested state.
Fixing this problem in the ISC guardian may help to prevent the HAM ISI trips that we have occasionally had on restarting beckhoff code.