Reports until 11:56, Wednesday 24 July 2019
H1 TCS (AOS, OpsInfo)
jeffrey.kissel@LIGO.ORG - posted 11:56, Wednesday 24 July 2019 - last comment - 11:56, Wednesday 24 July 2019(50777)
How to Restore Ring Heaters to Nominal Requested Heat Setting after Computer Failure when SDF is Poorly Maintained
C. Gray, J. Kissel, P. Thomas

I'm sure this is documented elsewhere, but in the heat (ha!) of recovery, it's always a challenge to remember where to look or whom to ask. So here's another bread crumb for future us (maybe LHO aLOG 47770 from Daniel "Danny" Vander-Hyde might have helped).

To restore ring heaters to their value in the event of a power outage:
(1) Trend the requested power and predictive kalman filter input for the past several days (assuming that the ring heaters have been in the desired state for the past several days. This two things should be the same. If they're not, then the actual requested power will be trending *very slowly* toward the kalman filter input. This is what we saw today. Note -- if we've recently increased the PSL input power, the nominal value may change for both, for example). 

Find the nominal value. 
      ndscope -k H1:TCS-ITMX_RH_SETUPPERPOWER H1:TCS-ITMX_RH_SETLOWERPOWER H1:TCS-ETMX_RH_SETUPPERPOWER H1:TCS-ETMX_RH_SETLOWERPOWER H1:TCS-ITMY_RH_SETUPPERPOWER H1:TCS-ITMY_RH_SETLOWERPOWER H1:TCS-ETMY_RH_SETUPPERPOWER H1:TCS-ETMY_RH_SETLOWERPOWER &

     As of 2019-07-24, the nominal values for a ~37 W IFO are 
              Upper [W]    Lower [W]
     ITMX     0.4907        0.4907
     ETMX     0.4500        0.4500
     ITMY     1.3930        1.3930
     ETMY     0.4024        0.4024

(2) Ring heater guardians are typically in "FILTER_RH_INPUT," which is taking "nominal" filter input, e.g. H1:TCS-ETMX_RH_INVERSE_FILTER_IN ,and if it's different from the actual requested powers, it'll slowly (predictively) push the actual request to the nominal via a predictive kalman filter (see LHO aLOG 46363). 

    Change which ever guardian is outside of its nominal, e.g. H1:GRD-TCS_ETMX_RH_PWR, from "FILTER_RH_INPUT" to "RESET."

(3) Change the actual requested power (e.g. H1:TCS-ETMX_RH_SETUPPERPOWER, H1:TCS-ETMX_RH_SETLOWERPOWER) to what it has been in the recent past.

(4) Change the guardian from "RESET" to "NOMINAL," this pushes the actual requested power (e.g. H1:TCS-ETMX_RH_SETUPPERPOWER) to the kalman filter input (e.g. H1:TCS-ETMX_RH_INVERSE_FILTER_IN).

(5) After confirming success (that the two channels in step (4) agree), then change the guardian back to "FILTER_RH_INPUT."
    EDIT: NOTE: See comment below -- we've decided to leave the guardians in NOMINAL, not FILTER_RH_INPUT. /EDIT

Today, we had ring heater settings restored by 2019-07-24 16:50 UTC (2019-07-24 09:50a PDT). 

ETMX and ETMY(upper and lower) had been wrong since 2019-07-24 06:27 UTC (2019-07-23 11:27p PDT; so about 10 hours out). 

We will work on updating the SDF system so that the ETM values are restored to the above mentioned nominal values upon computer restart.

Comments related to this report
jeffrey.kissel@LIGO.ORG - 11:31, Wednesday 24 July 2019 (50781)AOS, ISC, OpsInfo
The requested powers are stored in three different Beckhoff PLCs' and one Front-end's SDF system, which is why they've been poorly maintained:

Ring Heater    Name on SDF         Name of SDF            Front-End "Computer"
               Overview            EPICs server           Name Hosting EPICs       System Type
ITMX & ITMY    CS_ECAT_PLC3        H1SYSECATCPLC3SDF      H1:FEC-1026_SDF          Beckhoff

ETMX           EX_ECAT_PLC3        H1SYSECATXPLC3SDF      H1:FEC-1029_SDF          Beckhoff 

ETMY           EY_ECAT_PLC3        H1SYSECATY1PLC3SDF     H1:FEC-1032_SDF          Beckhoff

Filter Input
All RHs        H1TCSCS             H1TCSCS_SDF            H1:FEC-26_SDF          "Fast" Front-end

We've now updated ALL of these SDFs' safe.snap to be set at the nominal values described above.

FURTHER: We haven't / hadn't trended back far enough, but -- in short -- these ring heater guardians should be in the NOMINAL state at all times, not FILTER_RH_INPUT.
Likely, when we powered up to 37 W in May, we intentionally changed the ring heater settings, and *left* the guardian states in "FILTER_RH_INPUT" -- because using the predictive filtering takes time -- and thus, in the rush to get to observing we accepted this state, and it's been this way for the recent past that we've trended on *some* (namely the ETMs) guardians.

However, we're no longer intentionally changing the ring heater settings (for the forseeable future, at least -- the next attempt power up is likely "weeks" away) so we don't need to engage the predictive filtering system. Thus, we should have all ring heater guardians in NOMINAL.

ALSO, ALSO: Back in May, during the power up, and in the rush to get back to observe, we un-monitored these requested power and filter input channels. Why? Because when we're *using* the predicitive filtering system, the predictive filter is slowly but actively changing the requested power. Again, because we're no longer using the predicitive filtering, and the requested power should be stationary, we have now REMONITORED the following channels:
    H1TCSCS
        H1:TCS-ITMX_RH_INVERSE_FILTER_IN
        H1:TCS-ETMX_RH_INVERSE_FILTER_IN
        H1:TCS-ITMY_RH_INVERSE_FILTER_IN
        H1:TCS-ETMY_RH_INVERSE_FILTER_IN

    CS_ECAT_PLC3
        H1:TCS-ITMY_RH_SETUPPERPOWER
        H1:TCS-ITMY_RH_SETLOWERPOWER
        H1:TCS-ITMX_RH_SETUPPERPOWER
        H1:TCS-ITMX_RH_SETLOWERPOWER

    EX_ECAT_PLC3
        H1:TCS-ETMX_RH_SETUPPERPOWER
        H1:TCS-ETMX_RH_SETLOWERPOWER

    EY_ECAT_PLC3
        H1:TCS-ETMY_RH_SETUPPERPOWER
        H1:TCS-ETMY_RH_SETLOWERPOWER