Reports until 11:02, Monday 10 February 2020
H1 CAL
jeffrey.kissel@LIGO.ORG - posted 11:02, Monday 10 February 2020 - last comment - 11:34, Monday 10 February 2020(55007)
2020-01-03 Calibration Model Uncertainty Budget Update: Second Attempt Complete, Systematic Error Estimate Better without PCALSYSERR Data, But Now Too Uncertain to Understand Remaining Flaw 
J. Kissel

I've continued effort towards converging on an uncertainty estimate for the 2020-01-03 calibration loop model, and thus the uncertainty in h(t) that has used this model (i.e. GDS-CALIB_STRAIN data products after 2020-01-13, but we'll likely use this model to cover all of O3B when we produce DCS-CALIB_STRAIN products.)

We last left off with LHO aLOG 54907, where the broadband transfer functions between PCAL and GDS-CALIB_STRAIN are consistently outside of the estimated 68% confidence interval (see this plot from that aLOG).

The comment to that aLOG (LHO aLOG 54911) left several paths forward, one of which included a better understanding of how to account for the known 0.43% systematic error in PCAL displacement estimates prior to 2019-11-12 (LHO aLOG 53188).
However, in the interest of "seeing what the answer is if you just ignore the data you don't understand," here I show what the results look like when one only includes data *after* 2019-11-12.

See the attached plots,
    [5] The "final answer" uncertainty budget compared against the two versions of measurements which should be a cross-check and equivalent. These are PCAL transfer functions to the following response channels:
         (a) Against GDS-CALIB_STRAIN: 2020-02-07_H1_PCAL2GDS_BB_meas1262990871_on_2020-01-13_vs_model_uncertainty_for_2020-01-03.pdf
         (b) Against "corrected***" CAL-DELTAL_EXTERNAL 

I again note that for [5](b), the CAL-DELTAL_EXTERNAL transfer function data has been corrected for the expected time-independent differences between CAL-DELTAL_EXTERNAL and GDS-CALIB_STRAIN (see T1900169), but the data has *not* been corrected for time-dependence (i.e. no TDCFs for the data have been applied to properly compare the reference model time on 2019-12-26 against the measurement times).

One can see in [5](a) and [5](b), that -- while the uncertainty estimate and median systematic error agree much better with the data below 40 Hz, are are closer to the data above 125 Hz -- there is still some discrepancy between the median line and the data. However, given that I've excluded all data prior to 2019-11-12 (i.e. sensing function data between 2019-10-31 and 2019-11-11, and all actuation data from 2019-11-12 all the way back to 2019-03-29), the uncertainty on the estimate has grown larger.

As such, with this larger uncertainty, 
    (i) Yes, you can say that the data is now within the 68% confidence interval, but 
    (ii) there is still evidence for discrepancy right around 90-110 Hz.

So -- I'd like to 
    (I) continue to assess how to apply the PCAL systematic error in a way that makes us confident in using data prior to 2019-11-12,
    (II) Get more GDS-CALIB_STRAIN broad band data from Aaron, to make sure this one measurement is not a fluke (though the CAL-DELTAL_EXTERNAL data that I do have implies that it's not),
    (III) build in the functionality in to the pyDARM universe to correct the CAL-DELTAL_EXTERNAL data for both GDS / CALCS corrections *and* user defined TDCFs.

I'll add the script locations and further plots that went in to creating this data in the comments.
    

In doing so, the estimated 68% confidence interval
Non-image files attached to this report
2020-02-07_H1_PCAL2GDS_BB_meas1262990871_on_2020-01-13_vs_model_uncertainty_for_2020-01-03.pdf
2020-02-07_H1_model20200103_deltaL_over_pcal.pdf
Comments related to this report
jeffrey.kissel@LIGO.ORG - 11:34, Monday 10 February 2020 (55011)
Link 

As before, the most detailed notes of the process for creating these plots can be found in the notes posted to T2000006. However, I'll copy the highlights that one needs to reconstruct this data set here.

In order create this data, I *only* needed to modify the data fed in to the Gaussian Process Regression, so there is no new information on the reference model MCMC uncertainty. See LHO aLOG 54909 for those details.

Here, I ran the following uncertainty budget command: 
python3.5 /ligo/svncommon/CalSVN/aligocalibration/trunk/Common/pyDARM/RRNom.py --IFO=LHO --HDF5_A_MCMCresults=/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Results/Uncertainty/O3B_H1_A_MCMC_20200103Model_REF.hdf5 --HDF5_A_GPRresults=/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Results/Uncertainty/2020-02-07_GPR_Run_NOPCALSYSERR_meas20191204-20200127_collection_O3B_H1_A_model20200103_ALL_actuationMultiGPR_UIM_f7-250Hz_L0p1_PUM_f7-500Hz_L0p2_TST_f10-1000Hz_L0p5.hdf5 --HDF5_C_MCMCresults=/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Results/Uncertainty/O3_H1_C_MCMC_2019-12-26_forreference_fmin20Hz.hdf5 --HDF5_C_GPRresults=/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Results/Uncertainty/2020-02-07_GPR_Run_H1_C_allO3BMeas_collection_NOPCALSYSERR_nofsQcorr_MCMCfmin20Hz_GPRfmin20_lengthscaleZp35_withPCALXHFdata_model20200103_sensingFunction_GPR_posteriors.hdf5 --modelPath=/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/params/ --modelFilename=modelparams_H1_20200103 --IFOmodel=modelPars --sampleNumber=1000 --seed=1111 --version=ref --outDir=/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Results/Uncertainty/  --plot1SigmaUncs --saveSummaries --gpsTime=1261415486

using SVN rev 9311 of RRnom.py.

The careful reader will notice the only difference in this call are the two new GPR .hdf5 files, 
    /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Results/Uncertainty/
        2020-02-07_GPR_Run_NOPCALSYSERR_meas20191204-20200127_collection_O3B_H1_A_model20200103_ALL_actuationMultiGPR_UIM_f7-250Hz_L0p1_PUM_f7-500Hz_L0p2_TST_f10-1000Hz_L0p5.hdf5       rev 9338
        2020-02-07_GPR_Run_H1_C_allO3BMeas_collection_NOPCALSYSERR_nofsQcorr_MCMCfmin20Hz_GPRfmin20_lengthscaleZp35_withPCALXHFdata_model20200103_sensingFunction_GPR_posteriors.hdf5    rev 9343

These hdf5 files were created by
    /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Scripts/Uncertainty/
        process_allmeas_writeGPRHDF5_meas20191120-20200203_withHF_NOPCALSYSERR_model20200103-C.py       rev 9345
    /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Scripts/Uncertainty/
        process_allmeas_writeGPRHDF5_meas20191204-20200127_model20200103-A.py                           rev 9337
I'll note that -- just because I was there -- I added more recent data that we've taken that was not included in the 2020-02-04 estimate -- essentially all the data taken in January after the model installation on 2020-01-13.

Once the uncertainty budget,
    /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Results/Uncertainty
        2020-02-07_O3_LHO_GPSTime_1261415486_ref_RelativeResponse1SigmaUncertainty.png
        2020-02-07_O3_LHO_GPSTime_1261415486_ref_RelativeResponseUncertainty_FinalResults.txt
        2020-02-07_O3_LHO_GPSTime_1261415486_ref_RelativeResponseUncertainty_MinMax.txt
was committed to rev 9348,

I updated the processing scripts that compare that envelope to the measured broadband PCAL injections, which are
    /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Scripts/FullIFOSensingTFs/
        plot_GDS_BB_vs_Uncertainty_20200207.py                      rev 9351
    /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Scripts/CALCS_FE
        process_broadband_pcal2darmtf_collection_20200207.py        rev 9352.

OK. So lets attach some supporting plots. I follow the same "where in the process" labeling as in LHO aLOG 54907, but group the plots by sensing and actuation, instead.
    /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Results/Uncertainty/ 
    [2] (a)   2020-02-07_GPR_Run_H1_C_allO3BMeas_collection_NOPCALSYSERR_nofsQcorr_MCMCfmin20Hz_GPRfmin20_lengthscaleZp35_withPCALXHFdata_meas2020-02-03_sensingFunction_referenceModel_vs_allMeasurements.pdf
    [3] (a)   2020-02-07_GPR_Run_H1_C_allO3BMeas_collection_NOPCALSYSERR_nofsQcorr_MCMCfmin20Hz_GPRfmin20_lengthscaleZp35_withPCALXHFdata_model2020-01-03_sensingFunction_GPR.pdf

    [2] (b)   2020-01-27_H1_2020-02-07_GPR_Run_NOPCALSYSERR_meas20191204-20200127_collection_UIM_actuationMeasurement_allresiduals_MCMCInput.pdf
    [3] (b)   2020-02-07_GPR_Run_NOPCALSYSERR_meas20191204-20200127_collection_H1_A_model2020-01-03_lastmeas2020-01-27_UIM_actuationMultiGPR_f7-250Hz_L0p1.pdf

    [2] (b)   2020-01-27_H1_2020-02-07_GPR_Run_NOPCALSYSERR_meas20191204-20200127_collection_PUM_actuationMeasurement_allresiduals_MCMCInput.pdf
    [3] (b)   2020-02-07_GPR_Run_NOPCALSYSERR_meas20191204-20200127_collection_H1_A_model2020-01-03_lastmeas2020-01-27_PUM_actuationMultiGPR_f7-500Hz_L0p2.pdf

    [2] (b)   2020-01-27_H1_2020-02-07_GPR_Run_NOPCALSYSERR_meas20191204-20200127_collection_TST_actuationMeasurement_allresiduals_MCMCInput.pdf
    [3] (b)   2020-02-07_GPR_Run_NOPCALSYSERR_meas20191204-20200127_collection_H1_A_model2020-01-03_lastmeas2020-01-27_TST_actuationMultiGPR_f10-1000Hz_L0p5.pdf

With each of the [2]-type plots, that show in the inputs to the GPR, one can see, indeed, that the data is much more tightly clustered around a single frequency-dependent function. Further, the magnitude of those residuals -- especially for the PUM and TST stages of which we were suspicious before -- are closer to being consistent with "no" systematic error, as opposed to both having a 1% systematic error below unity.
However, the sensing function systematic error has a tighter distribution around a more well-defined wiggle, which may the source of some of the discrepancy.

Still more thinking required, unfortunately...
Non-image files attached to this comment