During the several hour lock over night, that I found the IFO in this morning there were some strange combs in the CARM signals.
The particularly strange thing is that I see these spikes in the LSC-MCL signals for that lock first thing this morning, but haven't really seen them in any later locks today. During the lock that the spikes are there, there are a few times that the spikes go away for 2-4 sec, but they always came back. In the next lock, the spikes were almost never there, but did come in 2 or 3 times for 1-2 sec, but then went away again. Between these 2 locks, I'm not aware that any configuration changes were made to the CARM system.
The SR785 was plugged in to the IFO common mode board, and the excitation input was enabled, so it's possible that the SR785 and whatever mysterious grounding problems we've been having are the culprit. But, since the SR785 wasn't unplugged until much later in the day, I'm not so sure. Anyhow, we're going to try to make a habit of unplugging any instruments from the ISC racks as soon as a measurement is complete, just in case.
It looks to me like there are 2 different combs, both very near 232 Hz, but they are drifting in time relative to one another. See attached DARM spectrum with the giant comb, as well as ndscope screenshots from 2 different times during the same lock. The ndscope screenshots also show that the IFO is in the same state (NomLowNoise) for both of these times. No gain sliders in the common mode or IMC servo boards were being moved either.
These spikes could be causing me to repeatedly lose lock when I try to transition DARM to RF (CARM is already on TR at this point). The spikes that I see in MCL are also showing up clearly in PRCL, and are definitely present in DARM.
UPDATE: The second figure is from a time when I'm transitioning CARM off of the ALS signals and onto the IR transmissions. During this time, the spikes begin to be impressed on AS45, which is the signal that we'd use in the next step for DARM. So, the AS signals don't have the spikes natively, but they do show up once the spiky signals are used for CARM.
Also, both the SR785 and AG4395 were plugged into the CARM and IMC boards respectively. I had ndscope open on the workstation by the racks, and was watching the spikes while the IFO was locked on ALS, with DRMI locked. When I wiggled the cable connections at the analyzers, I did not see any effect on the spikes. I then unplugged the IN2 cable from the IMC board (at the chassis), and saw no change. Next I unplugged the EXC cable from the IMC front panel, and the spikes immediately went away. This was repeatable, re-introducing the cable brought the spikes back, removing the cable removed the spikes. Note that the excitation point on the board was disabled, but somehow the Agilent is still causing noise. It didn't look like the Agilent was taking any measurement, it was just passively plugged in. I unplugged the 3rd cable from the IMC chassis, as well as all 3 cables from the CARM common mode board just for good measure. The SR785 and AG4395 are still on, but they are not cabled to any IFO electronics.
With the SR785 and AG4395 unplugged, I do not see regular spikes on MCL. However, when doing the CARM and DARM to IR transitions, I did see the spikes come back for a few seconds, but then they're gone again. So, there may still be something going on (does the Agilent need to be turned off entirely?), but it's much better now. I have not had any trouble locking since.
Moral of the story - we can't leave the AG4395 plugged in to the IMC common mode board, even if the excitation input is disabled. Needing to turn the RF analyzer off, or also forbidding the SR785 is still a possibility, but not yet proven.
Separately, while I was standing next to the ISC racks, I noticed that if I took a step (heavy step, but not stomping) I could see a buzz of noise in both the IMC signals and AS45, despite our still being locked entirely on ALS for the arms. I did not check how close / far I had to be from things to see this, but if we're getting a big burst of noise just from walking with anything but the most gentle of steps, perhaps that's a clue we can use to figure out why we are so sensitive lately.
