I've done some testing to see how well the line subtraction algorithm in the gstlal calibration pipeline works when used to remove the roaming calibration line (> 1kHz, and changes periodically). Since the line subtraction has already been reviewed, I mainly wanted to investigate what happens right around the time that the line frequency changes. The gstlal calibration pipeline reads in the line frequency from the raw frames, so it is able to change frequency. Fortunately, the line frequency only changes when we are not in low noise, so there should be no concern about noise being injected at the moment of the change. The attached plots show data from around a time when the frequency changed from 2501.3 Hz to 2001.3 Hz, at GPS 1239997514 (Mon Apr 22 19:44:56 UTC 2019).
The first four plots are ASDs and ASD ratios from just before the change. Note that the 2501.3 Hz line is not subtracted very effectively.
The next four plots are ASDs and ASD ratios from just after the change. After the change, the 2001.3 Hz line is subtracted effectively, as expected.
The next (9th) plot is a time series of ratios of (cleaned data) / (uncleaned data) at the roaming line frequency, showing magnitude and phase. Note that the line subtraction of the 2501.3 Hz line stops working before the time of the change. It appears as though the algorithm is switching the frequency before the frequency actually changes. I will need to investigate this more to find out why this is happening. Hopefully, it can be fixed by a configuration change.
The last plot is similar to the previous one, except that it is for the regular calibration lines, so it provides a reference for roughly how effective the subtraction should be.
After some investigation, I found the problem causing the line frequency to be updated too early. When run offline (in DCS/C01 mode), the gstlal calibration pipeline has access to large amounts of data all at once, allowing computationally cheap processes to operate much faster than expensive ones. The reading in and updating of the calibration line frequencies is cheap, so it happens quickly, before the calibrated h(t) and subtraction algorithm are ready. The process of updating calibration lines involves changing a property of a gstreamer element, which is not necessarily timestamp-synced, unlike many other processes, such as adding two streams. The fix to the problem was to use the GstController, a tool which allows GObject properties to be updated over stream time. I've made this fix, it appears to have worked. The first attached plot is a time series of ratios of (cleaned data) / (uncleaned data) at the roaming line frequency, similar to the one in the above aLOG. The update occurs just before the 4-hour mark in the plot. The second plot is a similar plot showing all the other calibration lines, which do not change during this time.
