jeffrey.kissel@LIGO.ORG - posted 17:30, Tuesday 11 February 2020 - last comment - 15:20, Thursday 20 February 2020(55055)
2020-01-03 Calibration Model Uncertainty Update: Adding 2020-02-10 Data Sets Improve Uncertainty, But Don't Reveal Discrepancy Between Overall Systematic Error Prediction vs. Measurement
J. Kissel I've processed yesterday's measurement suite (see LHO aLOG 55018) to the point where I've been able to add it to the collection of measurements used to produce an estimate of the unknown, frequency-dependent (but time independent) systematic error in the model (i.e. the Gaussian Process Regression of the measurement collection's residual frequency dependence after the model has been divided out of the measurement, and the measurement has been compensated for time dependence). I've done this for four reasons: (1) To confirm that most of the response function uncertainty in the middle frequency range (20-300 Hz) is dominated by the uncertainty in this estimate of unknown systematic error (2) I've always been curious how much "one more data set" gathered improves the overall uncertainty (you would guess it should improve by sqrt(N), but N is different for each of the four model components -- C, A_UIM, A_PUM, and A_TST -- and the digital filtering on each of these components means that the contribute to the uncertainty in a sophisticated, frequency dependent way that is in no way easy to predict) (3) In the event that we *do* throw away the data prior to 2020-11-12's fix to the known PCAL systematic error (aka PCALSYSERR) because it's easier than figuring out how to salvage the data prior, as alluded to in LHO aLOG 55007, then I'd like to see as much PCALSYSERR-free data included in the estimate as possible (4) To see if adding "one more data set" would help us "resolve" the discrepancy we see between a processed GDS-CALIB_STRAIN in the presence of a broad-band PCALY injection. The results satisfy (1) through (3) positively, but they leave (4) unresolved. See the attached figure, which shows the two uncertainty envelopes from 2020-02-07 (i.e. LHO aLOG 55007) and 2020-02-11 (today, this aLOG) compare against the same 2020-01-13 pre-processed GDS transfer function data (from LHO aLOG 54565). We see improvement (frequency dependent, of course) with the addition of the 2020-02-10 measurement suite. This confirms (1), and demonstrates (2) that we're still in the "winning" region of 1/sqrt(N) for the actuator, given that the number of measurements, N, is 5 to 6 for each stage. Since we measure it every week, the sensing function measurement number 13, so we don't see "as much" improvement above ~100 Hz where the sensing function's contribution to the response function is dominant. Addressing (3) the improvement (again, it's frequency dependent -- look at the plot -- so in this sentence I'm just using "verbal" numbers to describe what I see) at ~50 Hz is about 0.5% / 0.25 deg -- and when you're talking about a budget that is achieving a 3-4% / 1-2 degree level, that's pretty substantial. So, for the next few calibration measurement days, I might start getting actuator data too (i.e., until N gets large, increase the frequency of measuring the actuator suite from ~twice a month to once a week). As far as (4) -- In the region that we're seeing the largest discrepancy between median predicted systematic error (plus uncertainty) and the measured systematic error between PCAL and the final product -- between 40 Hz and 150 Hz -- the uncertainty remains too large to "resolve" the discrepancy. In fact -- if we focus on exactly 100 Hz, I'm now pretty confident that we're limited by the overall PCAL uncertainty (a frequency independent 0.54%) and *not* the GPR uncertainty, so this discrepancy may never get resolved. I still would like to see - a different day's broad band injection process and put on this plot, and - all of the swept sine transfer functions for which we only have CAL-DELTAL_EXTERNAL, but properly corrected for time-dependence, such that the comparison with this "reference time" envelope is valid and we can see how the envelope compares to measurement across all frequencies from 5 to 5000 Hz, before declaring that "it won't get any better than this -- let's release this envelope!" Stay tuned for the results of that work. Also -- remember, this envelope will likely *not* be valid for the first 2 hours of every lock stretch. Analysis of showing the systematic error as the IFO thermalizes (based on the data from LHO aLOG 53951) is underway.
Non-image files attached to this report
Comments related to this report
Information about how the 2020-02-11 uncertainty envelope was created:
To produce the new GPR fits:
/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Scripts/Uncertainty
process_allmeas_writeGPRHDF5_20200211_A_meas20191204-20200210_model20200103_NOPCALSYSERR.py
process_allmeas_writeGPRHDF5_20200211_C_meas20191120-20200210_model20200103_NOPCALSYSERR_withHF.py
committed to rev 9412.
The new HDF5 files from the fit:
2020-02-11_GPR_Run_H1_A_O3B_meas20191204-20200210_collection_model20200103_NOPCALSYSERR_ALL_UIM_f7-250Hz_L0p1_PUM_f7-500Hz_L0p2_TST_f10-1000Hz_L0p5_posteriors.hdf5
2020-02-11_GPR_Run_H1_C_O3B_meas20191120-20200210_collection_model20200103_NOPCALSYSERR_nofsQcorr_MCMCfmin20Hz_GPRfmin20_lengthscale0p35_withPCALXHFdata_model20200103_posteriors.hdf5
committed to rev 9413.
The updated call to RRNom.py:
python3.5 /ligo/svncommon/CalSVN/aligocalibration/trunk/Common/pyDARM/RRNom.py --IFO=LHO --HDF5_A_MCMCresults=/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Results/Uncertainty/O3B_H1_A_MCMC_20200103Model_REF.hdf5 --HDF5_A_GPRresults=/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Results/Uncertainty/2020-02-11_GPR_Run_H1_A_O3B_meas20191204-20200210_collection_model20200103_NOPCALSYSERR_ALL_UIM_f7-250Hz_L0p1_PUM_f7-500Hz_L0p2_TST_f10-1000Hz_L0p5_posteriors.hdf5 --HDF5_C_MCMCresults=/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Results/Uncertainty/O3_H1_C_MCMC_2019-12-26_forreference_fmin20Hz.hdf5 --HDF5_C_GPRresults=/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Results/Uncertainty/2020-02-11_GPR_Run_H1_C_O3B_meas20191120-20200210_collection_model20200103_NOPCALSYSERR_nofsQcorr_MCMCfmin20Hz_GPRfmin20_lengthscale0p35_withPCALXHFdata_model20200103_posteriors.hdf5 --modelPath=/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/params/ --modelFilename=modelparams_H1_20200103 --IFOmodel=modelPars --sampleNumber=1000 --seed=1111 --version=ref --outDir=/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Results/Uncertainty/ --plot1SigmaUncs --saveSummaries --gpsTime=1261415486
The resulting uncertainty envelope:
/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Results/Uncertainty
2020-02-11_O3_LHO_GPSTime_1261415486_ref_RelativeResponse1SigmaUncertainty.png
2020-02-11_O3_LHO_GPSTime_1261415486_ref_RelativeResponseUncertainty_FinalResults.txt
2020-02-11_O3_LHO_GPSTime_1261415486_ref_RelativeResponseUncertainty_MinMax.txt
committed to rev 9414.
The script to produce the attached plot:
/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Scripts/FullIFOSensingTFs
plot_GDS_BB_vs_Uncertainty_20200211.py rev 9415
Just trying gather some further clues -- today, I thought "I know that the measured broadband injection shown in these plots has been corrected for TDCFs as per LHO aLOG 54676. That means the measurement has been 'propogated' back to the reference time and is directly comparable to the RRNom-produce uncertainty envelope, using the argument version=ref. But what if we correct the *envelope* for the TDCFs surrounding the reference time?" The results are the *incorrect* thing to do, but they do help in understanding how much impact the TDCFs can make on the estimate. I envoked RRnom with --version=C00, and set the --gpsTime=1261411218 (2019-12-26 16:00 UTC, just *before* the *reference* measurement was taken -- not when the broadband injection was taken), and the data gathering method to --GetData=GWPY, with otherwise everything else identical, python3.5 /ligo/svncommon/CalSVN/aligocalibration/trunk/Common/pyDARM/RRNom.py --IFO=LHO --HDF5_A_MCMCresults=/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Results/Uncertainty/O3B_H1_A_MCMC_20200103Model_REF.hdf5 --HDF5_A_GPRresults=/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Results/Uncertainty/2020-02-11_GPR_Run_H1_A_O3B_meas20191204-20200210_collection_model20200103_NOPCALSYSERR_ALL_UIM_f7-250Hz_L0p1_PUM_f7-500Hz_L0p2_TST_f10-1000Hz_L0p5_posteriors.hdf5 --HDF5_C_MCMCresults=/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Results/Uncertainty/O3_H1_C_MCMC_2019-12-26_forreference_fmin20Hz.hdf5 --HDF5_C_GPRresults=/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Results/Uncertainty/2020-02-11_GPR_Run_H1_C_O3B_meas20191120-20200210_collection_model20200103_NOPCALSYSERR_nofsQcorr_MCMCfmin20Hz_GPRfmin20_lengthscale0p35_withPCALXHFdata_model20200103_posteriors.hdf5 --modelPath=/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/params/ --modelFilename=modelparams_H1_20200103 --IFOmodel=modelPars --sampleNumber=1000 --seed=1111 --version=C00 --outDir=/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Results/Uncertainty/ --plot1SigmaUncs --saveSummaries --gpsTime=1261411218 --GetData=GWPY This plot is flawed, because at 2019-12-26 16:00 UTC, GDS was still using the 2019-09-09 model parameter set. That means the TDCFs were computed against the 2019-09-09 model, when we know that the interferometer was behaving more like the 2020-01-03 model parameter set. So, the uncertainty envelope has been corrected for by the *the wrong thing*. In order to truly make this comparison, we should find the uncertainty envelop for a time *after* the CAL-CS / GDS calibration was updated on the 2020-01-13. I compared the uncertainty envelopes against the data just to show what it looks like. We should run this comparison again once we have C01, where all the O3B data has been calibrated with the 2020-01-03 model.
Images attached to this comment
Non-image files attached to this comment
J. Kissel I've re-run the uncertainty estimate at 1263000618, Jan 14 2020 01:30:00 UTC, just after the broad band injection and compared *that* against the reference time envelope (i.e. one with no TDCFs applied). I also attach screenshots of the values of the TDCFs at *this* time, so we can compare them against what Aaron reported the GDS TDCFs that were used to correct the broadband injection in LHO aLOG 54676. One can see that the envelope disagrees with the data. HOWEVER -- an aLOG is still pending -- but Aaron believes he's uncovered a systematic error in the TST stage of the actuator. You can see a sneak peak of the preliminary investigation here. That plot shows the response function with (in blue) and without (in maroon) the systematic error present. You could convince yourself that the discrepancy between the GDS measurement (which would have this TST flaw) and the uncertainty envelope is *about* what you see as the blue curve in that preliminary plot. But, stay tuned for the analysis to be done right -- applying the correction in "the right direction" and putting these things on the same plot. Anyways, just wanted to show this data as well, given that this is actually a *legit* comparison as far as we know it -- and now I have a clue that points my finger back at GDS. It's still impressive how different the TDCF-applied envelope is from the reference envelope, but I suppose we shouldn't be surprised as the reference envelope is based on data from 2019-12-26 (for sensing) and 2019-04-03 (for the actuator). Stay tuned! The command to create this uncertainty envelope is python3.5 /ligo/svncommon/CalSVN/aligocalibration/trunk/Common/pyDARM/RRNom.py --IFO=LHO --HDF5_A_MCMCresults=/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Results/Uncertainty/O3B_H1_A_MCMC_20200103Model_REF.hdf5 --HDF5_A_GPRresults=/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Results/Uncertainty/2020-02-11_GPR_Run_H1_A_O3B_meas20191204-20200210_collection_model20200103_NOPCALSYSERR_ALL_UIM_f7-250Hz_L0p1_PUM_f7-500Hz_L0p2_TST_f10-1000Hz_L0p5_posteriors.hdf5 --HDF5_C_MCMCresults=/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Results/Uncertainty/O3_H1_C_MCMC_2019-12-26_forreference_fmin20Hz.hdf5 --HDF5_C_GPRresults=/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Results/Uncertainty/2020-02-11_GPR_Run_H1_C_O3B_meas20191120-20200210_collection_model20200103_NOPCALSYSERR_nofsQcorr_MCMCfmin20Hz_GPRfmin20_lengthscale0p35_withPCALXHFdata_model20200103_posteriors.hdf5 --modelPath=/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/params/ --modelFilename=modelparams_H1_20200103 --IFOmodel=modelPars --sampleNumber=1000 --seed=1111 --version=C00 --outDir=/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Results/Uncertainty/ --plot1SigmaUncs --saveSummaries --gpsTime=1263000618 --GetData=GWPY
Images attached to this comment
Non-image files attached to this comment