D. Barker, J. Kissel, T. Shaffer, J. Warner

Today we installed and tested a few new states in the the site-wide seismic configuration manager guardian node, "SEI_ENV,"
     /opt/rtcds/userapps/release/isi/h1/guardian/SEI_ENV.py
which executes all the changes to the SEI and SUS control systems that we believe put it in its most robust state. 

These changes are discussed at great length in the Standard Operating Procedure -- see M2100056 -- but are summarized by the following (paraphrasing from the SOP):

    (1) Brings the IMC to OFFLINE,
    (2) turns off the subtraction of differential motion between platforms from each platforms' feedback error signals which tie platforms together (aka CPS_DIFF),
    (3) Turns OFF the floor-feed-forward sensor correction (aka SENSCOR), 
    (4) Run SEI systems's feedback loops in their most robust configuration by bringing the guardian managers to DAMPED
        (a) Run HEPIs in position locking loops, position-sensor-only feedback (except HAM1, deemed too complicated to create a pos-only isolation design, so it?ll be turned OFF),
        (b) Run the ISIs in "damping only" narrow bandwidth loops that reduce motion on resonance using robust inertial sensors
    (5) Turn all switchable analog gains on all ISI sensors to their lowest setting, making it harder to saturate their ADCs.
    (6) Increasing the allowable number of actuator saturations to be as large as possible.
    (7) Make sure robust, suspensions' TOP mass damping remains on at all times.
        (a) Bypass all digital SUS watchdogs. 
             (i) Increase the time that the SUS IOP Software Watchdog can be bypassed to many many hours, then hit the bypass button.
             (ii) Increase the "user" SUS watchdog thresholds to be unachievably high
        (b) Ensure all top mass actuators are in their highest range state 
        (c)  Turn OFF alignment offsets to make sure damping loops have the full DAC range
        (d)  Turn OFF any and all optical lever damping (namely the SUS-BS)

There is a new state CONSTRUCTION that performs all of these steps, and two other states REVERTING_CONSTRUCTION and CONSTRUCTION_REVERTED that restore the above configuration changes to "normal."

We ran through the transition between MAINTENANCE (a tried and true state) and CONSTRUCTION, then back "down" to CONSTRUCTION_REVERTED several times, so we're confident it does every we need it to do. 

Save bypassing the SUS hardware watchdog, which will be done once for the duration of the construction period but has not been done yet, the control system is ready for well monitored A+ filter cavity road construction.

Some other details that made this possible:

Daves changes to the IOP watchdogs today (LHO aLOG 58670) allows us to bypass the IOP Software Watchdogs (step 7.a.i).

Also in passing, we found a ... bug? oversight? ... in the interactions and progression of new states that TJ and Jim added to the BSC SEI system managers a month ago (see LHO aLOG 58353) which was causing all the BSC ISI platforms to trip when going from FULLY_ISOLATED to DAMPED as requested. I'll leave that to TJ/Jim to give details in the comments, but the message is that it's now fixed.

    We also found intermittent problems with setting EPICs variables via the guardian's ezca interface, where when writing to a bunch of similar channels with a for loop, some times the guardian log would say that it got written, but the value never got written. We called the guardian help desk (i.e. Jamie), but he was as mystified as we were, given that the eacz python bindings should be throwing errors when a variable is not written. We exercised the CONSTRUCTION transition 3 times, and each of the three times we got a different result: (1) ITMY ST1 and ITMX ST2 actuator allowable counts were not set to max, (2) only ITMY ST1 was not set, and (3) it all worked as written. Each time in (1) and (2), the guardian log reports a change in variable, but MEDM and ndscope reveal that the value was never changed.   
    As with an intermittent problem that is not easily reproducible, this may be "one of those things that just comes and goes, never to be solved, and [hopefully] never seen again." But as is indicated in the SOP, this will be one of the things a human expert confirms with their eyeballs for the start of every day we use the CONSTRUCTION state.