Congratulations, all! 

This process went very smoothly, thanks to a lot of prep work by a lot of folks, and many teams working in parallel today.

The calibration update was pushed based on the second -175 [ct] SRCL offset sensing function data taken in LHO:70683.

Even though there were no new measurements of the actuators taken, these the N/ct actuation strength "free" parameters were also updated with, essentially, a new MCMC run on the last, most recent, old data from May 17 2023 (LHO:69684).

Here're the following "free parameter" values exported to foton:
   $ pydarm export
       searching for 'last' report...
       found report: 20230621T211522Z
       using model from report: /ligo/groups/cal/H1/reports/20230621T211522Z/pydarm_H1_site.ini
       filter file: /opt/rtcds/lho/h1/chans/H1CALCS.txt

          Hc: 3.4207e+06 :: 1/Hc: 2.9234e-07
          Fcc: 439.33 Hz

          Hau:  7.5083e-08 N/ct
          Hap:  6.2353e-10 N/ct
          Hat:  9.5026e-13 N/ct

    filters (filter:bank | name:design string):
       CS_DARM_ERR:10                O4_Gain:zpk([], [], 2.9234e-07)
       CS_DARM_CFTD_ERR:10           O4_Gain:zpk([], [], 2.9234e-07)
       CS_DARM_ERR:9                O4_NoD2N:zpk([439.32644887584786], [7000], 1.0000e+00)
       CS_DARM_ANALOG_ETMX_L1:4      Npct_O4:zpk([], [], 7.5083e-08)
       CS_DARM_ANALOG_ETMX_L2:4      Npct_O4:zpk([], [], 6.2353e-10)
       CS_DARM_ANALOG_ETMX_L3:4      Npct_O4:zpk([], [], 9.5026e-13)

The calibration report on the MCMC fitting for free parameters, as well as the GPR fit based on the two measurements at -175 [ct] (20230621T211522Z in LHO:70683 and 20230621T191615Z in LHO:70671) is attached below for convenience, but has been archived on the LDAS cluster under 
    H1_calibration_report_20230621T211522Z.pdf

For the primary metric of how the calibration's quality changed across the 75W to 60W, then change of SRCL offset change, the calibration push, see LHO:70705. I copy and past that attached image here for convenience.

Also repeating Louis:
The DTT template for this measurement is stored in  
    /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O4/H1/Measurements/FullIFOSensingTFs/
        20230621_systematic_error_deltal_external_gds_calib_strain.xml

we changed is_pro_spring to False in the pyDARM parameter model set. Commit: 353de502.

Attached are the raw the blow-by-blow notes I took during yesterday's calibration push that highlights all the command-line commands and actions we needed to take in order to update the calibration.

Recapping here with a little more procedural clarity: 
(Any command recalled without a path are/were/may be run any new, fresh terminal; we did not need to invoke any special conda environment thanks to the hard work done behind the scenes by the pydarm-cmd team):

    (0) If at all possible, understand what you expect to change in the calibration ahead of time. 
        If that is *limited* to something changing that can only be measured with the full IFO, 
        i.e. you expect *only* a change in the "free parameters" (overall sensing function gain, 
        DARM cavity pole frequency, or any of the three ETMX UIM, PUM, TST actuator strengths) 
        then you run through the process outlined below as we did yesterday. Other changes to the 
        DARM loop, like electronics changes or computational arrangement mean you have to do a 
        more in-depth characterization of that thing, update the DARM loop model parameter set, 
        *then* start at (1).

    (1) Measure something new about the IFO. In this case we *knew* that we expect a change in 
        the inteferometric response of the IFO because of the ring heater changes and input 
        power change, so we remeasured the sensing function; and expected only the optical gain 
        and the cavity pole to change.
        
        $ pydarm measure --run-headless bb sens pcal
        
        We, of course, should be out of observing, and the ISC_LOCK guardian should be in NLN_CAL_MEAS.
        When the measurement is complete, you can do steps (2) through (6) with the IFO *back* 
        in NOMINAL_LOW_NOISE, and you can even go back in to OBSERVING during that time.

    (2) Process that measurement, and create the folder of material that's required for that 
        processing, as though it were a part of the on-going "epoch" of measurements where 
        you expect nothing to have changed about the DARM loop other than the time-dependent 
        corrections to the free parameters. This gives you a "report" that shows the residuals 
        between the last installed model of the IFO and you current measurement compared to 
        the rest of the measurement/model residuals in the inventory for that last "epoch." 
        In this way, you can confirm or refute your expectations of what has changed.
        
        $ pydarm report
        
        which generates the folder in 
        /ligo/groups/cal/H1/reports/20230621T191615Z/

    (3) Looking at the first results, we were disappointed that occasionally the MCMC fit 
        would land on a parameter hyperspace island that had a large SRC detuning spring 
        frequencies, even through the lower frequency limit of the data fed into the fitter 
        was ~80 Hz. As such, we adjusted the *default* model parameter set,
         
        /ligo/groups/cal/H1/ifo/pydarm_H1.ini
        changing the following parameter,
        Line 15    is_pro_spring = False
        and re-ran the report,
        $ pydarm report --force
        in order to re-run the MCMC. This worked so we committed pydarm_H1.ini to the 
        ifo/H1/ repo as git hash 353de502.
 
    (4) After looking through the history of measurement/model residuals, you should 
        then have an understanding what you want to *tag* as "valid" and an understanding 
        of whether you new measurement *is* infact the boundary of a new epoch. This 
        may also be the time when you *don't* like what you see, so you modify the 
        controls settings of the IFO to change it further and go back to step (1). As you 
        can see from LHO aLOGs 70671, 70677, and 70683, 
        we were doing just that.

        In the end, we had *two* measurements in "the new epoch" that we liked, and one 
        measurement in the middle -- technically its *own* epoch -- that we didn't like. 

        So, after processing all the data and making no updates to the report tags so we 
        could see the whole history of the sensing function, we tagged the reports in the 
        following way

        $ cd /ligo/groups/cal/H1/reports/
        $ pydarm ls -r                  # before tagging
            20230620T234012Z valid          # Last valid 75W sensing function data set
            20230621T191615Z                # First new 60W data set, with SRCL offset -175 [ct]
            20230621T201733Z                # Second new 60W data set, with SRCL offset -165 [ct]
            20230621T211522Z                # Third new 60W data set, with SRCL offset -175 [ct]
        $ # validating First and Third 60W data sets, both with -175 [ct] SRCL offset the report so it shows up on the
        $ touch 20230621T191615Z/tags/valid 
        $ touch 20230621T191615Z/tags/epoch-sensing
        $ touch 20230621T211522Z/tags/valid  
        $ pydarm ls -r                  # after tagging
            20230620T234012Z valid
            20230621T191615Z valid epoch-sensing
            20230621T201733Z
            20230621T211522Z valid

    (5) Then, now that these tags are set up -- and specifically the epoch-sensing tag -- 
        only these new 60W data sets are included in the history, which means only 
        those data sets are stacked and fit to GPR. Thus, this report generation run will 
        be the "final" report that generates what we end up exporting out to the calibration 
        pipeline. Importantly, even though the epoch boundary is defined by the the *first* 
        20230621T191615Z measurement, the parameters that will be installed are defined by 
        the MCMC of the *latest* 20230621T211522Z measurement. This works because we're 
        assuming the IFO is the same in this entire boundary, so we should get equivalent 
        answers (within uncertainty, and modulo time-dependent correction factors) if we MCMC 
        any of the measurements in the epoch.
        
        $ pydarm report --force
        
        yields a good report, with "free parameters," foton exports, FIR filters, MCMC 
        posteriors, and GPR fits that are ready to export to the calibration pipeline.

        Also note that all of these re-runs of the report ("pydarm report --force") 
        are *over-writing* the contents of the report, so if you want to save any interim 
        products you must move them out of the way to a different location and/or different name.

    (6) We can validate what we are about to push out into the world with the dry run command,
        
        $ pydarm export
        
        where if you don't specify the report ID, then it exports the latest report. In this case, 
        the latest is 20230621T211522Z, so we do want to use this simplest use of this command. 
        That spits out text like what's shown in LHO:70699. 
   
        Another option is 
        
        $ pydarm status
        
        which spits out a comparison between what's *actually* installed in the front-end against 
        the latest report (which, in this case, is what we're *about* to install).

    (7) If you're happy with what you see, then it's time to shove "the calibration" out into the world.
        (a) Presumably, the IFO is still locked, in NOMINAL_LOW_NOISE, and maybe even in OBSERVING. 
            Warn folks that you're about to take the IFO out of OBSERVING, and the DARM_FOM on the 
            wall is about to go nuts, but the IFO is fine. Try to do steps (b) through (g) as quickly 
            but accurately/completely/carefully as possible.

        (b) push EPICs records to the front end and save new foton files.
            $ pydarm export --push

        (c) open up the CAL-CS GDS-TP screen, and look at the DIFF of filter coefficients. 
            Hit the LOAD_COEFFICIENTS button if you see what you expect from the DIFF.

        (d) on the same screen, open up the SDF OVERVIEW. Review the changes and accept if you see 
            what you expect.

        Now everything's updated in the front-end, so it's time to migrate stuff out to the cluster 
        so GDS and the uncertainty pipelines get updated. 

        (e) Add an additional tag to the report which you just pushed,
        
        $ touch /ligo/groups/cal/H1/reports/20230621T211522Z/tags/exported
        $ pydarm ls -r 
            20230620T234012Z valid
            20230621T191615Z valid epoch-sensing
            20230621T211522Z exported valid
        

        (f) Archive all the reports that are a part of this wonderful new epoch, which pushes the 
            whole folder so it includes the tags. Having the "exported" tag is particularly 
            important for the GDS pipeline. 
            
            /ligo/groups/cal/H1/reports$ arx commit 20230621T191615Z
            /ligo/groups/cal/H1/reports$ arx commit 20230621T211522Z 
            

        (g) Restart the gds pipeline, which picks up the .npz of filter coefficients from the latest 
            report marked with the "exported" tag. 
            
            $ pydarm gds restart
            

            This opens up prompts form both DMT machines, dmt1 and dmt2, to say "yes" to confirm that 
            you want to restart.

            After restarting the GDS pipeline, you can check the status of the machines as well,
            
            $ pydarm gds status
            

    (8) Once you're done with the GDS pipeline restart, then you've gotta wait ~2-5 minutes for 
        the pipeline to complete its restart. To check whether the pipeline is back up and running, 
        head to "grafana" calibration monitoring page. 
        Presumably, the IFO is still in NOMINAL_LOW_NOISE, so eventually the live measurement of 
        response function systematic error should begin to reappear, hopefully even closer to 
        1.0 mag, 0.0 deg phase. While you're waiting, you can also pull up trends of 
            - the front-end computed TDCFs, see if those move closer to 1.0 (or in the case of f_cc, closer to the MCMC value)
            - the front-end computed DELTAL_EXTERNAL / PCAL systematic error transfer function, see if those move closer to 1.0
        In addition, the newest, latest *modeled* systematic error budget only gets triggered once 
        every hour, so you just have to be patient on that one, and check in later.

    (9) Once everything is settled, take the ISC_LOCK guardian to NLN_CAL_MEAS, and take a broad-band 
        PCAL injection for final, high-density frequency resolution, post-install validation a la LHO:70705

Report saved in /ligo/groups/cal/H1/reports/20230621T211522Z/H1_calibration_report_20230621T211522Z.pdf and attached.

The less noisy pressure is consistent with what Hugh found way back in 2017 when they first started testing the EX Beckhoff unit.

There's also a 2-5 psi drop in the individual pressures sensors for EY, but the new levels line up well with the pressures we are seeing at EX.

Report saved in /ligo/groups/cal/H1/reports/20230621T201733Z/H1_calibration_report_20230621T201733Z.pdf and attached.

Jeff and Louis checked and saw no obvious change on the new sensing function so decided to revert back to -175 SRCL detuning.

Thermalization guardian commented out of ISC_LOCK for now.

Reverted the change to ITMY A2L gains from 69082

NOISE_CLEAN will not turn on any NonSENS cleaning.  This means that GDS-CALIB_STRAIN_CLEAN will be the same as GDS-CALIB_STRAIN_NOLINES.

I've turned *ON* the "thermalization" calibration lines, via the CAL_AWG_LINES guardian for this power up, in order to track the thermalization of the sensing function during a 60W power up (we did not turn on these lines until we were regularly at 75W, so we don't really have as clear of an analysis [e.g. LHO:69593] of the thermalization behavior during 60W)

Recall, the eight line frequencies (the highest at 24.5 Hz) are listed in LHO:69284.

They've been on and running since 2023-06-21 15:18:04 UTC.

Note, I have *not* recoded this up to be turned on automatically in ISC_LOCK, as I hope that we'll get a few thermalization runs during these next two 8 hour periods, and I'll be present for them.

As per discussion in LHO:70650, I've edited 
    /opt/rtcds/userapps/release/isc/h1/guardian/
        lscparams.py
in order to change the hard-coded value to which we set the SRCL offset, changing it from -265 [ct] that we've been using at 75/76W PSL input power, to -175 [ct] which we'd used at 60W PSL input power. This is line 526 (at the time I edited the params):

    offset = {'SRCL_MODEHOP':-800,
              'SRCL_RETUNE':-175  # updated 20230621
             }

While we're not confident this is the perfect value, it's certainly a fine place to start.

Reverted LSC FF filters, as noted by Elenna's config alog.

SRCLFF1 again uses FM2. MICHFF again uses FM6-9.  PRCLFF gain is commented out (so, should be left at zero from Down).

PRCL OLG measured after the loop changes in LOWNOISE_LENGTH_CONTROL

Sheila measured the PRCL OLG, having left the 'new' filters in place, and letting lownoise_length_control set PRCL1 gain to 6, and not using the thermalization guardian.

Our UGF is about 25 Hz, so a little lower than the target of 30 Hz, but stable and fine.  We'll re-check after a while of having been at full power.

Lowered CARM gain by hand by 6dB (lowered H1:LSC-REFL_SERVO_IN1GAIN and H1:LSC-REFL_SERVO_IN2GAIN by 1dB each, alternating, until I was down on each slider by 6dB).

When we just ran through LaserNoiseSuppression, we saw lots of excess noise, and Sheila measured the highest CARM UGF to be around 27 kHz, which is too high.  We the lowered the overall gain by 6dB.  Not yet in guardian.

EDIT: Lowering by 6dB brought our lowest UGF to ~12kHz, too low.  We re-increased by 3dB, so that in the end we've only reverted the 3dB increase that Elenna mentioned in the config alog.

EDIT2: this is now in guardian.

PRCL OLG remeasured longer into the lock, and the UGF was quite low.  I increased the PRCL1 gain to 10 (from the nominal, without-thermalization-guardian, 6), and the UGF is back to 30 Hz. 

We can likely afford to just put this gain of 10 into lownoise_length_control, but that would put our UGF at the beginning of the lock at 37 Hz with 24 deg of phase margin.  Probably fine, but much higher starts to be not fine.

I increased the gain of the SRCL FF (and measured that I did not need to change the gain of MICH FF).

Attached shows the reduction in coherence with SRCL when the gain of the SRCLFF1 filter bank is set to 2.1 (rather than 1.0). Blue is the old coherence, green is the updated coherence.  You can see that if we want to keep the coherence reduced at lower frequencies, we'll have to make a frequency-dependent change to the feedforward, but so far this at least helps.

I did this by injecting noise into SRCL (by just using the SRCL OLG measurement template's excitation, just set to exponential rather than fixed averages), and changed the feedforward gain until the noise in DARM seemed minimized above 30 Hz.  I did the same also for MICH, but found that the existing gain value of 0.97 was already the best.

This means that I incidentally got SRCL and MICH olg measurements, which are the second and third attachments.

Accepted FF-related SDFs.  Also accepted PRCL1 gain at 2 sec, since the thermalization guardian is off and won't set it to 30 sec.

Not shown, I also accepted the OAF-WHITENING gain at zero (which means that there's no NonSENS cleaning going forward).

Another PRCL measurement, UGF is just a bit under 30 Hz.

Sheila plugged in the SR785 to the ISS second loop chassis similar to the photo in alog 61721.

Keita confirmed that Err1Mon is equivalent to our digital filter banks' In1 (so, before the excitation), and Err2Mon is equivalent to In2 (so, after the excitation).  The excitation BNC is likely the one plugged into the port under Err1Mon on the photo.

And, since the two monitor points have different gains, the UGF of the loop should be read off of the TF at the -20dB line.

With the ISS second loop gain H1:PSL-ISS_SECONDLOOP_GAIN at the 75W value of -5 dB, Sheila measured that we had a UGF of about 17kHz.  With the gain increased to -2dB, we have a UGF of about 21.7kHz.  I've accepted the value of -2dB into SDF.

Attached is a photo of the SR785 with the IFO at 60W and the ISS second loop gain at -2dB.

We've changed the PRCL1 gain in lownoise_length_control to be 10.  Since this means that we don't need the Thermalization guardian, we'll just leave that in IDLE, and TJ has set it's nominal to be IDLE (see alog 70694).

Sheila has written a separate alog 70692 for what to do if this is too much gain for PRCL at the beginning of the lock.

I did a simple caget to find out what the values of H1:LSC-SRCLFF1_GAIN & H1:LSC-SRCLFF1_TRAMP and a caput to change the gain to 2.1. 
After the change I saw a noticable increase in SENSMON Range.

IFO Current Status : NOMINAL_LOW_NOISE & OBSERVING  with a range of 140.6 Mpc

This is a photo of the CARM OLG measurement refered to in 70662

Here are 2 plots when the interferometer is locked. The ISS second loop sensors see about 60mW of light, whereas for this time LSC-REFL_A/B see about 8mW each.

The first plot shows a large excess in REFL power fluctuations below ~200Hz. EVen the flat part 300Hz shows some excess. It should be about 70% of the 16mW measurements, but shows a very similar level. Looking at the coherence between REFL_A and B indicates that this is a real signal and not noise.

The second plot shows relative intensity noise. To get the curves calibrated correctly one should match the peak near 4.5kHz since this seems rela intensit noise from the laser. (There is a factor of 0.3 in the calibrations of the RIN of REFL_A/B to acocunt for the interferometer reflectivity at DC. This factor should be 1 when the interferometer isn;t locked.)

Here is comparison between early in the lock and after 4 hours.

The hump in the reflected power is clearly getting larger as time progresses, and is its coherence with PRCL. The input power as measured by the ISS second loop outer sensor doesn't have a large correlection with the reflected power (some is expected due to the shot noise of the inner sensor).

Q1: Why is PRCL coherent with the power in reflection? If theer is a couplinh, shouldn't it be at least second order?

Q2: What's the flat noise above 300Hz that we see in the reflection power?

Here is the power trend during this lock.

And these are the plots more than 7 hours into a 60W lock. The REFL PD now seems to be shot noise limited above 100Hz.

Here is a comparison between the noise measured in reflection at 75W and 60W and against the dark noise. Some observations:

1. The power in reflection reduced about 2.5, much more than the scaled input powers.
2. The noise also scaled down more than just shot noise, indicating that there was additional noise at 75W (also seen in the coherence).
3. The measured noise at 60W is only a factor of 1.8 above dark noise, somewhat marginal.
4. If we subtract the dark noise from the measured noise we are a factor of 1,5 above ADC noise.
5. Assuming this is all shot noise, the shot noise limited power becomes 2.8mW.

The outer loop RIN is always reported about 8% higher than the innner loop one. This is not real. In the PSL ISS model of the second loop ISS both detector values are divided by the DC value of the inner loop detector. Since the outer loop detector sees about 8% more light, the RIN in the outer loop detector is overestimated by this amount. To get a better value multiply by 0.922. With this correction both RIN spectra agree with each other.

A better calibration of the REFL/ISS PDs measured with 10W input and all TMs misaligned.

Here is a 60W trend for completness.

Well, it appears that the annulus ion pump continues to rail.

Attached is a trend of the pump behavior for the past 3 days.

And, at around Sunday midnight, it started up again! Anyways, we'll keep an eye on it.

deleted

Here are some plots of the cross correlation for this time.  

The first is a comparison of the pyDARM model of the OLG to the measured OLG. At 24Hz, the model was predicting 2% less gain than the measurement, so here I've scaled the model up by 2% and used that for the estiamtion of the correlated noise. 

The second plot is the DCPD sum ASD loop corrected compared to the cross correlation.  You can see that the cross correlation is above the DCPD sum by 7% at 24Hz, which is incorrect.  I thought about if this could be because of the imbalance of the DCPDs, I will attach a note here that explains how I attempted to handle this imbalance in estimating the cross correlation.  (The DCPD_A and B channels are recorded before mulitplying by the balance matrix, the sum channel is after that matrix.)  In the end this did not make a significant difference, the cross correlation is still nearly 7% overestimating DARM at 24Hz when I corrected for this. 

The third plot shows the cross correlation compared to an estimate of the correlated noise obtained by subtracting the calculated shot noise from the DCPD SUM asd in quadrature, this mostly agrees with the cross correlation except at low frequencies. 

These plots were made using the code https://git.ligo.org/sheila-dwyer/cross-corelation commit 3c60740b  I will attempt to make a comparison of this code with Craig's cross correlation code to see if this problem is still present there. 

Here is a note describing how I corrected for the DCPD imbalance.  Once the matrix was reset as above, things become simple.  I will add a diagram to this note if I have time.

Daniel raised the point that perhaps a phase difference between the two PDs could explain a discrepancy at low frequency, in the correction I did I assumed that the two paths were only imbalanced by a scalar gain. The attached png shows a transfer function between the two DCPD channels, taken at the time of one of yesterday's broadband pcal injections. The frequency dependence of this at first glance doesn't seem right to explain what we see, ie, the error in the cross correlation doesn't have a wiggle between 20-30 Hz, although the error does seem to happen around the frequency of the cross correlation problem.