Unfotunately turning off the cosmic ray power supply did not remove the intermittent 4.05 Hz bumps in DARM, even though all the CS signals that showed a large peak at 4.06 Hz are now clean.
Looking again at all the signals that contain a peak around 4.05 Hz, we now have
H1:PEM-CS_MAG_LVEA_VERTEX_Z_DQ 4.050 Hz
H1:PEM-EX_ADC_0_08_OUT_DQ 4.040 Hz
H1:PEM-EX_ADC_0_09_OUT_DQ 4.040 Hz
H1:PEM-EX_LOWFMIC_VEA_FLOOR_DQ 4.040 Hz
H1:PEM-EX_MAG_EBAY_SUSRACK_Y_DQ 4.040 Hz
H1:PEM-EX_MAG_EBAY_SUSRACK_Z_DQ 4.040 Hz
H1:PEM-EX_MIC_EBAY_RACKS_DQ 4.040 Hz
H1:PEM-EX_TEMPERATURE_BSC9_ETMX_DQ 4.040 Hz
H1:PEM-EX_VMON_ETMX_ESDPOWER18_DQ 4.040 Hz
H1:PEM-EY_ADC_0_14_OUT_DQ 4.040 Hz
H1:PEM-EY_MAG_EBAY_SEIRACK_X_DQ 4.040 Hz
H1:PEM-EY_MAG_EBAY_SEIRACK_Y_DQ 4.040 Hz
H1:PEM-EY_MAG_EBAY_SEIRACK_Z_DQ 4.040 Hz
H1:PEM-EY_MAG_EBAY_SUSRACK_X_DQ 4.040 Hz
H1:PEM-EY_MAG_EBAY_SUSRACK_Y_DQ 4.040 Hz
Attached plots shows spectra and time series. Although the frequency of all peaks is very close, EX signals are only coherent among themselves, and not with EY signals, and the other way around too. So it looks like at both end stations there is a source of 4.04 Hz that is local. Maybe a clue? Still, we don't have evidence that those sources are related to the DARM noise.
To help track down the source of these lines, I've been investigating the time this feature appeared in these channels. The two channels that were the first to have this feature are H1:PEM-EX_LOWFMIC_VEA_FLOOR_DQ and H1:PEM-EY_ADC_0_14_OUT_DQ, which both have had a 4 Hz line present since October 2019. The exact time of appearance for each is:
The attached spectrograms show the lines appearing at the above times. I have also includes plots of the channel spectra (plot 3 and plot 4) that show the 4 Hz line before/after these times, as well as a recent time (2023-05-03 22:00:00).
Looking through the alogs posted on these days, there may be an association to install work for BRS heaters. The related EX work (alog 52494) was on 2019-10-15, and the EY work (alog 52542) was on 2019-10-23. The time on both days roughly lines up with when the lines appeared.
The full day shift reports for these days are alogs 52495 and 52656. SInce these days were during the O3a-O3b commissioning break, there are a number of other activities that could be related.
I checked several spectragras from the summary pages during the last month of O3. There was no sign of the 4 Hz bumps.
Looking back at spectrograms since the restart, the first evidence I could find of the 4 Hz bumps is on January 31, 2023. I could not find any sign of the 4 Hz bumps in any lock stretch before that time
Under Robert's suggestion that this might have resulted from some alignment change of the beam at EX (LHO:69578), the ISC_LOCK guardian archives show the following A2L gain changes in January 2023:
The issue with trying to blame the 13 Jan gain change is that there were subsequent long locks with high sensitivity around 30 Hz that did not show the bumps.
Since January, there was also a retuning of the IY P2L/Y2L (commit 463245ed, see LHO:69082).
Some more clues — seems like the ground motion at EX from 3 Hz to 30 Hz experienced a step increase on January 25 and has not settled back down since then. The Güralps and STSs agree on this point, but as for what the ground motion was like in the fall of 2022 they seem to diverge.
Looking at the summary pages in fall of 2022 it seems like the bumps were intermittently present in DARM, e.g.: https://ldas-jobs.ligo-wa.caltech.edu/~detchar/summary/day/20220902/#gallery-6
The increase in ground motion corresponds to turning on FAN1 at each of the end stations. Betsy points to alog 67055 indicating this work, and the change can be seen on the fan vibrometers (first attached plot). There is a peak at 22.5 Hz that pops up, and a broad increase in noise which includes the 4 Hz region, though not so strongly (attachment 2).