Starting from the process documented in https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=70735 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Prep Work leading up to today 2023-08-31. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% (0) We know we want to update the calibration for several reasons: - reduce \kappa_T - account for DARM2 FM8 boost - update for OM2 hot (1) We've measured the IFO anew on 2023-08-- https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=72573 Unlike in June, we're now using the simulines infrastructure a la https://cdswiki.ligo-wa.caltech.edu/wiki/TakingCalibrationMeasurements pydarm measure --run-headless bb gpstime;python /ligo/groups/cal/src/simulines/simulines/simuLines.py -i /ligo/groups/cal/src/simulines/simulines/settings_h1.ini;gpstime (2-5) Yesterday, Louis had done a TON of hard work bringing the inventory of measurements up to speed, and setting the epoch boundary tags accordingly, and generated new reports. (2-5) Also yesterday, we updated the static DARM loop model parameter set to account for the new DARM2 FM8 boost filter. https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=72569 and indeed, Louis re-ran the 20230830T213653Z report to make sure it was process with that new parameter file. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% STARTING Today 2023-08-31. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Now we begin the validation process. (6) For validating the report we want to push, first we have to make sure the new report has the valid tag. It didn't yet, so we did so: pydarm ls -r 20230830T213653Z report didn't appear in this list. touch /ligo/groups/cal/H1/reports/20230830T213653Z/tags/valid Now we can employ the two validation options: pydarm status -r 20230830T213653Z pydarm export -r 20230830T213653Z NOTE: New since June 2023 -- we can now *specify* the report name that we're checking the status against / exporting. We did a few sanity checks -- mostly that the TST stage actuation strength has changed. But also that the IFO measured values match what's printed in the report. NOW WE'RE READY TO EXECUTE STEP (7): (a) 2023-08-23 18:30 UTC we go out of observing. This is done by closing the beam diverter on the squeezer since they were going to do some commissioning on the SQZ system while we update the calibration. (b) Push the changes to the CAL-CS portion of the pipeline: pydarm export -r 20230830T213653Z --push (c) Looked at the CALCS GDS_TP screen's filter coefficient differences, we see what's expected. So, we LOAD COEFFICIENTS. AT THIS POINT, WE SEE THAT SYSTEMATIC ERROR IN THE CAL-CS PRODUCT, DELTAL_EXTERNAL, GETS MUCH SMALLER. ALSO WE SEE THE TDCFs GET CLOSE TO ONE. VICTORY IS OURS. (d) Louis captures screenshots and accepts SDF diffs. Remember, for the h1calcs model, OBSERVE.snap is the same as safe.snap, so you shouldn't need to flip back and forth to accept in both. We flipped back and forth between snaps just to double check and confirmed that this is true. (e) Added the "exported" tag to the report, touch /ligo/groups/cal/H1/reports/20230830T213653Z/tags/exported (f) make sure all of the reports that are included in your epoch for gpring are on the arx: pydarm ls -r 20230504T055052Z epoch valid epoch-sensing epoch-actuation 20230505T012609Z valid 20230505T174611Z valid 20230505T200419Z valid 20230506T182203Z valid epoch-sensing 20230508T180014Z valid 20230509T070754Z valid 20230510T062635Z exported valid 20230517T163625Z valid 20230616T161654Z valid 20230620T234012Z valid 20230621T191615Z valid epoch-sensing 20230621T211522Z exported valid 20230628T015112Z valid epoch-sensing 20230716T034950Z valid epoch-sensing 20230727T162112Z valid epoch-sensing 20230802T000812Z valid 20230817T214248Z valid epoch-actuation 20230823T213958Z valid 20230830T213653Z valid exported arx ls --depth 1 (plus is local, - is on the remote system) + 20230621T201733Z 20230621T211522Z 20230628T015112Z + 20230716T034950Z + 20230727T162112Z + 20230802T000812Z + 20230817T214248Z + 20230823T213958Z + 20230830T191430Z + 20230830T213653Z + archive - last-exported - latest commit what you want and has a plus: arx commit 20230716T034950Z 20230727T162112Z arx commit 20230802T000812Z 20230817T214248Z 20230823T213958Z 20230830T213653Z Double check that these are uploaded and in the system by seeing them appear on the web-accessible list: https://ldas-jobs.ligo-wa.caltech.edu/~cal/ they are! (8) Restart the GDS pipeline: (new a) Verify that the GDS pipeline's .npz hash against the cluster: In a terminal $ pydarm gds restart This holds until you type "yes" so hold while you verify. That command spits out a message waiting for you to say "yes" -- among this is the sha256 hash. b9903d41c49628daf1f92ffac24a8edd8d7474fe164a0ce3a9b2a7d739814484 Now look for this hash on the cluster's copy of the report: https://ldas-jobs.ligo-wa.caltech.edu/~cal/ Shows the "latest" report -- which coincidentally is the 20230830T213653Z report we're pushing, and lists the GDS filter sha256 hash: >> b9903d41c49628daf1f92ffac24a8edd8d7474fe164a0ce3a9b2a7d739814484 # check check Check the local copy in another terminal: $ cd /ligo/groups/cal/H1/reports/20230830T213653Z $ sha256sum gstlal_compute_strain_C00_filters_H1.npz >> b9903d41c49628daf1f92ffac24a8edd8d7474fe164a0ce3a9b2a7d739814484 #check check (new b) Now go back to the terminal where you were restarting the pipeline and type "yes" to restart the production DMT machine copy of the pipeline (new c) In that same terminal, you'll have to write "yes" again to restart the redundant DMT machine of the pipeline Last GDS DMT machine getting restarted on 12:09:13 PDT (new d) After a few minutes, check the status of the GDS pipeline. $ pydarm gds status INFO | ==================== dmt1 status ==================== ifo config git status: commit 3c57e5f9191b9069bd0554f165a8f7f61b06fafe (HEAD -> main, origin/main, origin/HEAD) Author: Jeff Kissel Date: Wed Aug 30 13:22:59 2023 -0700 Changed H1OMC fitler file to include new DARM2 FM8. See LHO:72562 ---------------------------------------- filter checksum: b9903d41c49628daf1f92ffac24a8edd8d7474fe164a0ce3a9b2a7d739814484 /home/dmtexec/calibration/gstlal_compute_strain_C00_filters_H1.npz ---------------------------------------- systemd service status: ● gstlal-calibration.service - Calibration pipeline DMT process Loaded: loaded (/home/dmtexec/.config/systemd/user/gstlal-calibration.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2023-08-31 12:08:05 PDT; 8min ago Main PID: 3414904 (gstlal_compute_) CGroup: /user.slice/user-1002.slice/user@1002.service/gstlal-calibration.service └─3414904 /usr/bin/python3.6 /usr/bin/gstlal_compute_strain --frame-duration=1 --frames-per-file=1 --config-file=/home/dmtexec/calibration/ifo/gstlal_compute_strain_C00_H1.ini --filters-file=/home/dmtexec/calibration/gstlal_compute_strain_C00_filters_H1.npz INFO | ======================================== INFO | ==================== dmt2 status ==================== ifo config git status: commit 3c57e5f9191b9069bd0554f165a8f7f61b06fafe (HEAD -> main, origin/main, origin/HEAD) Author: Jeff Kissel Date: Wed Aug 30 13:22:59 2023 -0700 Changed H1OMC fitler file to include new DARM2 FM8. See LHO:72562 ---------------------------------------- filter checksum: b9903d41c49628daf1f92ffac24a8edd8d7474fe164a0ce3a9b2a7d739814484 /home/dmtexec/calibration/gstlal_compute_strain_C00_filters_H1.npz ---------------------------------------- systemd service status: ● gstlal-calibration.service - Calibration pipeline DMT process Loaded: loaded (/home/dmtexec/.config/systemd/user/gstlal-calibration.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2023-08-31 12:09:13 PDT; 6min ago Main PID: 2228663 (gstlal_compute_) CGroup: /user.slice/user-1002.slice/user@1002.service/gstlal-calibration.service └─2228663 /usr/bin/python3.6 /usr/bin/gstlal_compute_strain --frame-duration=1 --frames-per-file=1 --config-file=/home/dmtexec/calibration/ifo/gstlal_compute_strain_C00_H1.ini --filters-file=/home/dmtexec/calibration/gstlal_compute_strain_C00_filters_H1.npz INFO | ======================================== jeffrey.kissel@cdsws02:/ligo/groups/cal/H1/reports$ SEEMS OK The total down time is from 12:07 to 12:23 PDT, i.e. 16 minutes, contrary to the "2-5" minutes mentioned in the Jun instructions. (new e) Ask the operator corps to restart the control room FOM that's displaying DELTAL_EXTERNAL and GDS-CALIB_STRAIN (new 9) Post the aLOG hollering that you've made an update to the calibration (and why). (new 10 was 9) Measure BB: We had the time -- SQZ folks weren't yet done -- so we grabbed both a BB as well as a full additional measurement. https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=72592 (11) An hour or more later, make sure to check back in that the *modeled* and *measured* systemtatic error agree -- (a) Look at the "grafana" pages: https://gstlal.ligo.caltech.edu/grafana/d/StZk6BPVz/calibration-monitoring Make sure the live *measured* systematic error is running, and is low. WE'VE FOUND THAT THIS IS BORKED TODAY. Poked Maddie. (b) Head to https://ldas-jobs.ligo-wa.caltech.edu/~cal/ and check that the *modeled* systematic error - is using the right report, - has been generated recently, - and the modeled vs. measured agree. WE'VE FOUND THAT THAT THIS WAS BORKED TODAY, AND EVEN SINCE YESTERDAY. Louis is confident this is a result of him reworking the report directory librarianship, and is working on fixing it.