Arianna, Derek
We noticed that the calibration line subtraction pipeline is often not correctly subtracting the calibration lines at the start of lock segments.
A particularly egregious case is on July 13, where the calibration lines are still present in the NOLINES data for ~4 minutes into the observing mode segment but then quickly switch to being subtracted. This on-to-off behavior can be seen in this spectrogram of H1 NOLINES data from July 13, where red lines at the calibration frequencies disappear at 1:10 UTC. Comparing the H1 spectrum at 1:06 UTC and 1:16 UTC shows that the H1:GDS-CALIB_STRAIN spectrum is not changed, but the H1:GDS-CALIB_STRAIN_NOLINES spectrum has calibration lines in the 1:06 UTC spectrum and no lines in the 1:16 UTC spectrum. This demonstrates that the problem is with the subtraction rather than the actual calibration lines.
This problem is still ongoing for recent locks. The most recent lock on August 15 has the same problem. The calibration lines "turn off" at 4:46 UTC as seen in the attached spectrogram.
The on-to-off behavior of the subtraction is particularly problematic for data analysis pipelines as the quickly changing line amplitude can result in the data being over- or under-whitened.
A while back, I investigated this and found that the reason for occasional early lock subtraction problems is that, at the end of the previous lock, the lines were turned off, but the TFs were still being calculated. Then, at the beginning of the next lock, it takes several minutes (due to the 512-s running median) to update the TFs with accurate values. There were two problems that contributed to this issue. I added some more gating in the pipeline to use the line amplitude channels to gate the TF calculation. This fix was included included in gstlal-calibration-1.5.3, as well as the version currently in production (1.5.4). However, there have been some instances in which those channels were indicating that the lines were on when they were actually off at the end of the previous lock, which means this code change, by itself, does not fix the problem in all cases. The version that is currently in testing, gstlal-calibration-1.5.7, includes some other fixes for line subtraction, which may or may not improve this problem.
A more reliable way to solve this issue would be to ensure that the line amplitude channels we are using always carry accurate real-time information. Specifically, we would need to prevent the occurance of the lines turning off long before these channels indicate this has occurred. The names of these channels are:
{IFO}:SUS-ETMX_L{1,2,3}_CAL_LINE_CLKGAIN
{IFO}:CAL-PCAL{X,Y}_PCALOSC{1,2,3,4,5,6,7,8,9}_OSC_SINGAIN