This is a late entry to list everything we checked on Friday afternoon (which spilled to roughly midnight CT).

-- Current Status --
LHO is currently running the same cal configuration as in 20240330T211519Z. The error reported over the weekend using the PCAL monitoring lines suggests that LHO is well within the 10% magnitude & 10 degree error window after about 15 minutes into each lock (attached image). Here is a link to the 'monitoring line' grafana page that we often look at to keep track of the PCALY/GDS_CALIB_STRAIN response at the monitoring line frequencies. The link covers Saturday to today. This still suggests the presence of calibration error near 33Hz that is not currently understood but, as mentioned earlier, it's now well within the 10%/10deg window. We think that the SRCL offset adjustment in LHO:80334 and LHO:80331 accounts for the largest contribution to the improvement of the calibration at LHO in the past week. 

-- What's Been Tried --
No attempts by CAL (me, JoeB, Vlad) to correct the 33Hz error have been successful. Many things were tried throughout the week (mostly on Tuesday & Friday) that went without being properly logged. Here's a non-exhaustive list of checks we've tried (in no particular order):


  
We cross-checked every single front end filter and gain parameter in the pyDARM model file against what's installed at LHO. Other than non-impacting foton file changes (e.g. out-of-path module changes) the only mismatch we identified is that the bias voltage in pyDARM was set to 3.3 instead of the IFO's 3.25.
  
We quickly implemented SRC detuning by copying the LLO setup. I did this by making on-the-fly changes to pyDARM on a local copy of the repo. I'll have to follow up with what these exact adjustments entailed. This seemed to do what we expected but it did not significantly impact the large 33Hz error.
  
Some GDS pipeline-affecting parameters made their way into the pydarm ini at LHO. These changes were made around the time of the Cal F2F over the summer. I reverted them, regenerated GSTLAL filters and installed them in the GDS pipeline. This didn't help. The commits in question are d2bfe349, 8717a9f5, and dc7208e0.
  
We don't believe the delays being reported by the pyDARM fits of the actuation delay on the PUM stage. We made adjustments to the fitting parameters for this stage to increase the fit range to roughly 400Hz in the hopes that pyDARM would produce a somewhat believable number. More details on this item as time allows; we are going to tune our injections to reduce uncertainty on these measurements in particular. This tuning will need to be done in tandem with other tunings (e.g. reducing PCAL at low frequencies to avoid leakage from polluting simultaneous higher frequency measurements).
  
We made similar adjustments to the sensing fit parameters. We extended the low end of the fit range to better capture the detuning near 10Hz in the fit.
  
Once it approached midnight on Friday night we reverted everything and left the IFO back in the state we found it in. This process is also not straightforward and requires creating a new 'fake' pyDARM report each time due recently-discovered bugs in the pyDARM infrastructure. This bug is being tracked here.


--What Changed?--
We think the SRC change(s) on September 5 (LHO:79929, also discussed in LHO:79903) line up well with the time frame at which the 33Hz error issue popped up. Here is a screenshot showing that the 33Hz error showed up on September 5. I've also attached a shot of ndscope showing the SRCL offset change on 09/05 from -175cts to -290cts here. cal_better_after_srcl_changed.png shows the error tracked by the monitoring lines before and after the SRCL offset was changed back to -190cts. The last image is a scope shot of that change (here).