13:29 took H1 back to Observing for tne next 80min before Maintenance.  (no intervention needed for the two lock attempts)

And since it's close to Maintenance, running the Revert script for my alerts.

1st locking attempt after Initial Alignment had lockloss at ENGAGE ASC FOR FULL IFO (looks like POP18 & PR Gain both drift down before lockloss; see attached screenshot).

NOTE:  There was a 5.7 Chilean EQ inbound, but don't really see any reason this was the cause for the lockloss.

On to 2nd lock...

I have accepted in SDF the differences that this caused for the digital and analog filters (see screenshots of sdf files).

I have also modified the ISC_LOCK and OMC_LOCK guardians such that when we next acquire lock, it should come back to this state of only 1 stage of analog filtering.

• ISC_LOCK I just removed the edges that were connected to the OMC_WHITENING state near the end of the acquisition sequence.  Lownoise_length_control now goes directly to Reduce_rf9_modulations_depth.  This prevents ISC_LOCK from asking the OMC guardian to add the 3rd stage of whitening.  Nothing else happened in this state.
• OMC_LOCK I commented out the lines in OMC_LSC_on that would have turned on the 2nd stage of filtering.  Note that in Prep_omc_scan all 3 stages of whitening are turned off, so the OMC should now only have 1 stage of analog filtering.

Calibration measurement suite sweeps are complete, and we are back in nominal low noise, but Robert has taken over the IFO for commissioning. 

We are remaining in this 1 stage of whitening only configuration.

We have resolved that it is not prudent to change the compensation filters, and/or update the calibration model parameter set, and we'll therefore just live with this increased systematic error in the low latency h(t). Eventually, after the systematic error is well quantified, we will create a new model parameter set, and re-calibrate the data -- starting at the next observation ready segment today.

I'll post the official time as to when we've gone back in to observation ready, so as to clearly define the first observation ready segment with this new change.

The first observation ready segment that includes the above described OMC whitening chassis filter configuration change started at Mar 16 2020 21:18:54 UTC (GPS time 1268428752)

Not that there is anything profound here, but I caught a glitch.  This is the first one that I've caught to grab a screenshot of after our change this morning of the OMC DCPD whitening filter configuration. 

Note to self: compare the _OUT_DQ of this glitch with other glitches in the old nominal config that I have screenshots of, to (try to) see if these glitches were attenuated due to the new analog configuration. 

The summary pages are reporting a significantly reduced range, but I'm not totally sure why. 

In the attachment, I've plotted ~an hour of data each for observing times this morning before we changed the whitening configuration, and for an observing segment after we made the change.  This is H1:GDS-CALIB_STRAIN, so has time dependent correction factors applied (although, as Jeff points out earlier in this thread those won't fully account for the small frequency dependent change that we've acquired).

Blue is an old nominal (3 analog filters) time, starting at 16 Mar 2020 16:24:40 UTC.  Orange is the new nominal (1 analog filter), starting at 16 Mar 2020 22:19:30 UTC.  The darker lines are the 50th %-ile, and the shaded regions are the 5th and 95th %-iles.  I can't tell much of a difference by eye.  Also, when I calculate the range from these median spectra, I get less than a 0.5 Mpc difference.  So, I'm not sure why the GDS range on the summary pages is so different now than it was this morning. 

I've used measurements of this OMC whitening chassis from (Sunday!) 2019-03-03 to predict the systematic error that transpired as a result of this configuration change. See further discussion in "PART II" of G2000527.

Making a *very* long story short, 
    (a) the systematic error incurred on the final DCS C01 calibration amounts to (a frequency dependent, but at maximum) 0.5% and 0.25 deg error at 100 Hz and 50 Hz, respectively.
    (b) the error arises from 
        - poor modeling of the super-Nyquist response of this chassis in the calibration pipeline, 
        - previously poor understanding of the circuit resulting in a poorly informed fit of the measurements, and finally 
        - the original data set only was taken down to 5 Hz, where the chassis (namely the first and third filters, which are the whitening stages) has response at 1 Hz and thus *any* fit (be it Stefan's original fit to update the compensation filters, or Lilli's fit for super-Nyquist poles) is fundamentally limited by the data.

Attached is a comparison of 
   - the ratio between the 2020-03-09 (pre-change) and 2020-03-16 (post-change) broad-band PCAL injections -- and thus the *measured* systematic error incurred by the change, and
   - the estimated systematic error derived from a re-fit to the original data.
Again, for further explanation of how this modeled error estimate was derived, see "PART II" of G2000527.

The script the accompanies the analysis for G2000527, from which all plots come, lives in 
    /ligo/svncommon/CalSVN/aligocalibration/trunk/Common/Electronics/H1/Scripts/
        fit_OMCDCPDWhiteningChassis_20190304_forG2000527.py

Great news! Well done SEI_ENV state!

Happened a third time: 2020-03-17 05:04:09 UTC. Still much reduced since the chiller swap.

Looked back at this after TJ suggested we should check that both chillers are keeping constant temperatures and flow rates. Temperatures in ITMX and ITMY cCO2 chillers seems to be stable. See plot over last 2 years here. As the laser hasn't ben on recently we cannot check how stable it is.

Flow rate reported by ITMX chiller seems to regularly drop from the nominal 3.8 to 2 GPM to up to 10 times an hour. However these drops only last one data point which doesn't effect the laser controller safety controls. See zoomed plot here. The chiller has been like for the past 3.5 years (as far back as I can easily see in EPICS).

2021/09/21 Tuesday 17:30UTC Both ITMX and ITMY chiller temperatures decreased from 22 degC to 19.5 degC. The set point is still at 22degC but the channel H1:TCS-{ITMY/ITMX}_CO2_CHILLER_OUT_GAIN_OUT16 dropped to zero Simulink for h1tcscs . Ndscope trace here. This apears to be the day the h1tcscs model was installed on h1oaf0 - see alog 60002. Tagging cds.

ECR for upgrading the CO2 controller is here: https://dcc.ligo.org/E1600312

Latest concept for redesigning CO2 controller is attached. Project is stalled right now due to lack of manpower.

Daniel fixed this setpoint problem by finding the filters had been removed (see alog 60660).

Once the filters were added back in ~19.15UTC 11/16/2021, the filters seems to be maxed out. Daniel turned off the laser power servo loop (sitemap > TCS > CO2{X/Y} > LASER > ON/OFF for PZT_SERVO_GAIN) and they behaved as expected. These values are not recorded in SDF. It doesn't make sence for the chillers to be tied to the laser when the laser is off so will investigate if the matrix changed. 

The PZT_SERVO_GAIN OUTPUT we turned OFF 2021/11/16 20UTC must have not being svn saved and turned back on on 2021/11/17 22UTC. See attached trend.

The design is to stabilize the laser PZT error signal by controlling the chiller temperature (E1300233p20). When the laser is off this seems to break down and change the have the setpoint given to the chiller (CHILLER_SETPT1) far from the ~20deg setpoint offset H1:TCS-ITM{}_CO2_CHILLER_SET_POINT_OFFSET (seen in TCS CO2 LASER screen). See table below.

TJ and I are looking into these settings. Maybe the matrix gain values or PZT gains are incorrect or the loop should be switched off when the laser is off.

*Documentation states it's important not to let the chillers be +/- 5deg from 20deg otherwise there may be condensation formed in the laser. The chillers have limits to shut off if they get outside this range but only compared to the CHILLER_SETPT1 value. 

The PZT_SERVO_GAIN OUTPUT is turned off in the CO2 Guardian's DOWN state (well done guardian), running DOWN fixed the problem.

Now the lasers think the set point given to the chillers is 20deg but the chillers are receiving set point request of 22degC. So another issue to solve.