J. Driggers, J. Kissel, Y. Lecoeuche T. Shaffer

We've been running in to a problem with the MICH_BRIGHT initial alignment step for the past few days (months?) in which the interaction between the automated initial alignment guardian manager (INIT_ALIGN) and its subordinate (ALIGN_IFO) does not make sufficient checks of the MICH system upon failure to lock before requesting to reacquire. As such, Jenne, TJ, and I set out to find a solution.

The conclusion: there are two problems -- 
    (1) During a successful first attempt at locking MICH BRIGHT, while the WFS loops are trying to converge MICH alignment, the M2 stage saturates (with a quite fast, but normal noise excursion). This saturation impulse kicks the ASC loops slewing the alignment to slowly bad, dropping the AS port QPD SUM which is used as a length-locking error signal, H1:ASC-AS_A_DC_SUM_OUT16, down below the threshold for a check whether MICH_BRIGHT is locked.
    See first attachment.

    (2) Once it's lost lock the first time, there's some screwy logic corner between the INIT_ALIGN manager, and the ALIGN_IFO subordinate which means the manager never realizes that the beam splitter is oscillating wildly, and at the same time trying to reaquire, which is blasting the SUS, pitching it more, and causing an AS port, Beam Splitter Dance Party / Laser Light Show.
    See second attachment.

TJ and I worked a bit on Problem 2, by changing some of the logic and clearing up the issues with the screwy logic corner this past Tuesday. These change were all inside the INIT_ALIGN guardian, in the states MICH_BRIGHT_ALIGNING: we
    - used the (IMHO) much more clear guardian call to gather the current state of a guardian (e.g. nodes['ALIGN_IFO'].STATE == 'DOWN', instead of just nodes['ALIGN_IFO'] == 'DOWN', which looks much like the request to *change* the state, nodes['ALIGN_IFO'] = 'DOWN', and only differs by one equal sign)
    - Instead INIT_ALIGN's MICH_BRIGHT_ALIGNING state making of two ambiguous checks that ALIGN_IFO has locked MICH BRIGHT before advancing to OFFLOADING_MICH_BRIGHT by only checking whether ALIGN_IFO has "arrived" in any state and that state is "done", we use more rigorous check that it has arrived and has completed the state "MICH_BRIGHT_ALIGN," i.e. the state that cooks the initial alignment.
    - Hoping that we've removed the logic flaw, we also reduced the "chill out" timer -- which is used after several attempts at locking MICH -- from 75 seconds to 45 seconds.

Since we've made the change (on Tuesday), we've NOT run an initial alignment beyond INPUT_ALIGN_OFFLOAD, so this change has not been tested.

Also -- while I *thought* that we'd added an additional check in the ALIGN_IFO state for ACQUIRE_MICH_BRIGHT to check that the max-min of the optical lever pitch signal is less than 0.5 urad, that test appears to have gotten lost in the fray. 

We have some more work to do to understand Problem (1), and to develop tests against it. [And today, as I was writing LHO aLOG 53711, I realize that we may have been having his problem since August 2019 because we have been in the wrong coil driver state!]