(Georgia, Corey)
After running an INITIAL ALIGNMENT, H1 locked up with not too many issues. While at NLN, had some diffs (screenshot is attached):
1) H1OAF
This was known, and Sheila said I should ACCEPT these.
2) SUS-BS_M1_LOCK_P stuff!
For some reason this filter bank had its Input OFF, the gain at 0 (vs -1), and the TRAMP was different. There was also a TRAMP diff for M1_OPTICALIGN_P. Georgia figured out that the issue was an error with guaridan for a Fast EZCA (i.e. "fez" step). Basically:
- fez command ran a whole bunch of commands on a list of optics,
- Error occurs
- But guardian sort of continues but (maybe randomly) skips a handful of lines for H1:SUS-BS_M1_LOCK_P*.
- And then continues with the fez command.
(see log for ISC_DRMI node below)
2019-04-25_00:15:36.433915Z ISC_DRMI [OFFLOAD_DRMI_ASC.main] ezca: H1:SUS-IM4_M1_LOCK_Y_TRAMP => 5
2019-04-25_00:15:36.434484Z ISC_DRMI [OFFLOAD_DRMI_ASC.main] ezca: H1:SUS-IM4_M1_LOCK_Y_GAIN => 0
2019-04-25_00:15:36.435067Z ISC_DRMI [OFFLOAD_DRMI_ASC.main] ezca: H1:SUS-IM4_M1_OPTICALIGN_Y_TRAMP => 5
2019-04-25_00:15:36.435459Z ISC_DRMI [OFFLOAD_DRMI_ASC.main] waiting for ramps to finish...
2019-04-25_00:15:41.447834Z ISC_DRMI [OFFLOAD_DRMI_ASC.main] done waiting
2019-04-25_00:15:41.453307Z Traceback (most recent call last):
2019-04-25_00:15:41.453307Z File "_ctypes/callbacks.c", line 315, in 'calling callback function'
2019-04-25_00:15:41.466866Z File "/usr/lib/python2.7/dist-packages/epics/ca.py", line 583, in _onConnectionEvent
2019-04-25_00:15:41.467296Z if int(ichid) == int(args.chid):
2019-04-25_00:15:41.467296Z TypeError: int() argument must be a string or a number, not 'NoneType'
2019-04-25_00:15:41.492347Z ISC_DRMI [OFFLOAD_DRMI_ASC.main] ezca: H1:SUS-BS_M1_LOCK_Y_RSET => 2
2019-04-25_00:15:41.493127Z ISC_DRMI [OFFLOAD_DRMI_ASC.main] ezca: H1:SUS-BS_M1_LOCK_Y_SW1 => 4
2019-04-25_00:15:41.532661Z ISC_DRMI [OFFLOAD_DRMI_ASC.main] ezca: H1:SUS-PR2_M1_LOCK_P_RSET => 2
2019-04-25_00:15:41.533281Z ISC_DRMI [OFFLOAD_DRMI_ASC.main] ezca: H1:SUS-PR2_M1_LOCK_P_SW1 => 4
Tonight's Solution: For this lock, Georgia Turned ON the input, increased the TRAMP from 5 to 10, and then changed the gain from 0 to -1.0.
This cleared our SDF Diffs issue, and we then had a signal for H1:ASC-MICH_P_OUT16 (it had been 0 due to the gain of 0).
Conclusion: We are back to OBSERVING. But we are curious about whether this will occur for the next lock.
(If this happens again for BS_M1 Pitch, follow the steps for "Tonight's Solution" above until it can be fixed.)
Sheila weeded calls to fast ezca out of ISC_LOCK and ISC_DRMI, because of the problem Corey describes above. This instance snuck through as it is hidden in ISC_GEN_STATES, in the gen_OFFLOAD_ALIGNMENT_MANY state.
What happened today is there was a connection error to some channel, and the beamsplitter top mass LOCK filter never got switched back on. Amazingly, we still acquired lock.
I have updated ISC_GEN_STATES, removing the fez. I think we need to reload the ISC_DRMI and ALIGN_IFO guardians to implement the new guardian code. I only updated the lines about fez, but the code is untested and I'm not very confident in my python/guardian skills, so I'm attaching the code for ISC_GEN_STATES.py as it was before my changes. If guardian complains upon reloading or relocking, and there isn't an obvious typo my changes can be reverted by replacing
/opt/rtcds/userapps/release/isc/h1/guardian/ISC_GEN_STATES.py
with what I've attached here, and re-re-loading.
