The calibration has been updated at LHO using Cal report 20240330T211519Z.
The TDCF EPICS channel changes and the CALCS filter changes are attached.
In O4b, the Calibration group is using a slightly different scheme for keeping track of cal reports, the front end pipeline settings, the GDS pipeline, and the hourly online uncertainty budgets. The biggest change is that each cal report is now also a git repository with its own history. This will allow for better tracking in situations for which it is deemed necessary to regenerate / reprocess calibration measurements.
Additionally, there are now two additional channels available on the front end: H1:CAL-CALIB_REPORT_HASH_INT and H1:CAL-CALIB_REPORT_ID_INT. These new channels are populated by the pyDARM tools when someone in the control room runs 'pydarm export --push'.
New Channels and their purpose:
H1:CAL-CALIB_REPORT_HASH_INT: numeric representation of the git commit hash for the report that was used to generate the current calibration pipeline
H1:CAL-CALIB_REPORT_ID_INT: numeric representation of the report id (e.g. 20240330T211519Z) the current calibration pipeline is configured with.
Current channel values:
caget H1:CAL-CALIB_REPORT_HASH_INT H1:CAL-CALIB_REPORT_ID_INT
H1:CAL-CALIB_REPORT_HASH_INT 4.14858e+07
H1:CAL-CALIB_REPORT_ID_INT 1.39587e+09
End-to-end procedure for updating the Calibration pipeline:
0. Take new calibration measurements following the instructions in OpsWiki/TakingCalibrationMeasurements.
1. make sure that the current pyDARM deployment version is up-to-date
1.a) run pydarm -v and check that the returned version (e.g. 20240405.1) matches the latest 'production release' tag listed at https://git.ligo.org/Calibration/pydarm/-/tags.
1.b) if the tags do not match, have a member of the Calibration group deploy the latest pyDARM tools to the site and the ldas cluster. They should follow the instructions laid out here.
2. generate a new cal report
2.a) run pydarm report (if this measurement set should be considered an epoch in sensing or actuation then apply the appropriate command line options as listed in the pyDARM help menu (pydarm report -h)). Report generation will now populate the report directory at /ligo/groups/cal/H1/reports/<report id>/ with various 'export' products. These include dtt calibration files, inverse sensing foton exports, and TDCF EPICS records that would be updated if this report were to be exported to the front end.
Here is a quick list of the some of the products that get stored at this step:
pcal_calib_dtt.txt: Pcal calibration into meters of displacement
deltal_external_calib.txt: calibration of DELTAL_EXTERNAL into strain
pydarm_version: the pyDARM tag indicating the version of pyDARM used to generate the report
export_epics_records.txt: list of each EPICS channel name and the value it would get set to when the report is exported to the front end
gstlal_compute_strain_C00_filters_H1.npz: a set of GDS filters and meta data that is sent to the GDS pipeline when the report is exported.
3. inspect the plots in the cal report to make sure they're reasonable. Typically this done by a member of the calibration group that is well-acquainted with the IFO and the calibration pipeline.
3.a) if the cal report is valid, set the 'valid' tag in the cal report: touch /ligo/groups/cal/H1/reports/<report id>/tags/valid.
3.b) if the cal report was marked valid in 3.a), then 'commit' the report now that its contents have been changed: pydarm commit <report id>. If you have not done this before, you may see a message from git complaining about dubious ownership. If that happens, follow the instructions in the message and try committing again. If you continue to have trouble, reach out to me, Jamie Rollins, or another member of the Calibration group that is knowledgeable about the new infrastructure.
4. if the report is 'valid', export the new calibration to the front end.
4.a) to first compare the cal report against the currently installed calibration pipeline, run pydarm status
4.b) to have pyDARM list all of the changes it would make if exported, run pydarm export
4.c) once you are certain that you want to update the calibration, run pydarm export --push. This will write to all of the EPICS channels listed in export_epics_records.txt and perform various CAL-CS frontend foton filter changes.
4.d) reload the CAL-CS front end coefficients via the MEDM screen system to make sure the new changes are loaded into place.
4.e) add an 'export' tag to the current report (touch /ligo/groups/cal/H1/reports/<report id>/tags/exported) and commit it again (pydarm commit <report id>).
5. upload the newly exported report to the ldas cluster.
5.a) run pydarm upload <report id>
5.b) wait about 1-2 minutes after the upload to allow time for the systemd timers on the ldas cluster to recognize that the new report exists. You can confirm that the latest report is recognized by the ldas cluster by verifying that https://ldas-jobs.ligo-wa.caltech.edu/~cal/archive/H1/reports/latest/ points to the correct report.
6. restart the GDS pipeline
6.a) run pydarm gds restart to begin the process of restarting the GDS pipeline. This will show prompts from the DMT machines (DMT1 and DMT2) asking you to confirm that the hash for the GDS pipeline package (gstlal_compute_strain_C00_filters_H1.npz). The prompts will contain the following line:
b1c9f6cd1ba3c202a971c6b56c7a1774afb1931625a7344e9a24e6795f3837d7 gstlal_compute_strain_C00_filters_H1.npz
To confirm that the hash above is correct, run sha256sum /ligo/groups/cal/H1/reports/<report id>/gstlal_compute_strain_C00_filters_H1.npz and verify that the two hashes are identical. If they are the same then type 'yes' and continue with the GDS restart process. After performing this process for the second DMT machine, pyDARM will continue with the pipeline restart. The GDS pipeline currently takes about 12 minutes to fully reboot and begin producing data again. During this time, no GDS calibration data will be receivable.
6.b) If the two hashes are not the same and all of the above checks were done, then something is likely wrong with the pyDARM+GDS pipeline system and you cannot continue with the calibration push. Take the following steps to reset the calibration to its former state:
1. open the CAL-CS SDF table and revert all of the EPICS channel pushes listed in export_epics_records.txt.
2. reset the foton filters by reverting to the last installed h1calcs filter before you exported the calibration report.
3. remove the exported tag from the new report (rm /ligo/groups/cal/H1/reports/<report id>/tags/exported), commit it (pydarm commit <report id>), and re-upload it to the ldas cluster (pydarm upload <report id>).
Summary of pyDARM commands (for use in the control room):
pydarm report [<args ...> <report id>]: generate a calibration report based on the measurement set at <report id>. See output of pydarm report -h for additional customization.
pydarm status [<args ...> <report id>]: compare current calibration pipeline against what the pipeline would be if report <report id> were exported to the front end
pydarm commit [<args ...> <report id>]: make a new commit in the report <report id> git repository. this creates a new hash and should be done any time the report's contents are changed.
pydarm upload <report id>: upload/sync the report <report id> with the ldas cluster
pydarm gds restart: initiate a GDS pipeline restart
pydarm ls -r: list all reports
