C. Gray, J. Kissel, P. Thomas I'm sure this is documented elsewhere, but in the heat (ha!) of recovery, it's always a challenge to remember where to look or whom to ask. So here's another bread crumb for future us (maybe LHO aLOG 47770 from Daniel "Danny" Vander-Hyde might have helped). To restore ring heaters to their value in the event of a power outage: (1) Trend the requested power and predictive kalman filter input for the past several days (assuming that the ring heaters have been in the desired state for the past several days. This two things should be the same. If they're not, then the actual requested power will be trending *very slowly* toward the kalman filter input. This is what we saw today. Note -- if we've recently increased the PSL input power, the nominal value may change for both, for example). Find the nominal value. ndscope -k H1:TCS-ITMX_RH_SETUPPERPOWER H1:TCS-ITMX_RH_SETLOWERPOWER H1:TCS-ETMX_RH_SETUPPERPOWER H1:TCS-ETMX_RH_SETLOWERPOWER H1:TCS-ITMY_RH_SETUPPERPOWER H1:TCS-ITMY_RH_SETLOWERPOWER H1:TCS-ETMY_RH_SETUPPERPOWER H1:TCS-ETMY_RH_SETLOWERPOWER & As of 2019-07-24, the nominal values for a ~37 W IFO are Upper [W] Lower [W] ITMX 0.4907 0.4907 ETMX 0.4500 0.4500 ITMY 1.3930 1.3930 ETMY 0.4024 0.4024 (2) Ring heater guardians are typically in "FILTER_RH_INPUT," which is taking "nominal" filter input, e.g. H1:TCS-ETMX_RH_INVERSE_FILTER_IN ,and if it's different from the actual requested powers, it'll slowly (predictively) push the actual request to the nominal via a predictive kalman filter (see LHO aLOG 46363). Change which ever guardian is outside of its nominal, e.g. H1:GRD-TCS_ETMX_RH_PWR, from "FILTER_RH_INPUT" to "RESET." (3) Change the actual requested power (e.g. H1:TCS-ETMX_RH_SETUPPERPOWER, H1:TCS-ETMX_RH_SETLOWERPOWER) to what it has been in the recent past. (4) Change the guardian from "RESET" to "NOMINAL," this pushes the actual requested power (e.g. H1:TCS-ETMX_RH_SETUPPERPOWER) to the kalman filter input (e.g. H1:TCS-ETMX_RH_INVERSE_FILTER_IN).(5) After confirming success (that the two channels in step (4) agree), then change the guardian back to "FILTER_RH_INPUT."EDIT: NOTE: See comment below -- we've decided to leave the guardians in NOMINAL, not FILTER_RH_INPUT. /EDIT Today, we had ring heater settings restored by 2019-07-24 16:50 UTC (2019-07-24 09:50a PDT). ETMX and ETMY(upper and lower) had been wrong since 2019-07-24 06:27 UTC (2019-07-23 11:27p PDT; so about 10 hours out). We will work on updating the SDF system so that the ETM values are restored to the above mentioned nominal values upon computer restart.
The requested powers are stored in three different Beckhoff PLCs' and one Front-end's SDF system, which is why they've been poorly maintained: Ring Heater Name on SDF Name of SDF Front-End "Computer" Overview EPICs server Name Hosting EPICs System Type ITMX & ITMY CS_ECAT_PLC3 H1SYSECATCPLC3SDF H1:FEC-1026_SDF Beckhoff ETMX EX_ECAT_PLC3 H1SYSECATXPLC3SDF H1:FEC-1029_SDF Beckhoff ETMY EY_ECAT_PLC3 H1SYSECATY1PLC3SDF H1:FEC-1032_SDF Beckhoff Filter Input All RHs H1TCSCS H1TCSCS_SDF H1:FEC-26_SDF "Fast" Front-end We've now updated ALL of these SDFs' safe.snap to be set at the nominal values described above. FURTHER: We haven't / hadn't trended back far enough, but -- in short -- these ring heater guardians should be in the NOMINAL state at all times, not FILTER_RH_INPUT. Likely, when we powered up to 37 W in May, we intentionally changed the ring heater settings, and *left* the guardian states in "FILTER_RH_INPUT" -- because using the predictive filtering takes time -- and thus, in the rush to get to observing we accepted this state, and it's been this way for the recent past that we've trended on *some* (namely the ETMs) guardians. However, we're no longer intentionally changing the ring heater settings (for the forseeable future, at least -- the next attempt power up is likely "weeks" away) so we don't need to engage the predictive filtering system. Thus, we should have all ring heater guardians in NOMINAL. ALSO, ALSO: Back in May, during the power up, and in the rush to get back to observe, we un-monitored these requested power and filter input channels. Why? Because when we're *using* the predicitive filtering system, the predictive filter is slowly but actively changing the requested power. Again, because we're no longer using the predicitive filtering, and the requested power should be stationary, we have now REMONITORED the following channels: H1TCSCS H1:TCS-ITMX_RH_INVERSE_FILTER_IN H1:TCS-ETMX_RH_INVERSE_FILTER_IN H1:TCS-ITMY_RH_INVERSE_FILTER_IN H1:TCS-ETMY_RH_INVERSE_FILTER_IN CS_ECAT_PLC3 H1:TCS-ITMY_RH_SETUPPERPOWER H1:TCS-ITMY_RH_SETLOWERPOWER H1:TCS-ITMX_RH_SETUPPERPOWER H1:TCS-ITMX_RH_SETLOWERPOWER EX_ECAT_PLC3 H1:TCS-ETMX_RH_SETUPPERPOWER H1:TCS-ETMX_RH_SETLOWERPOWER EY_ECAT_PLC3 H1:TCS-ETMY_RH_SETUPPERPOWER H1:TCS-ETMY_RH_SETLOWERPOWER