Reports until 12:25, Wednesday 22 February 2023
H1 ISC (CAL, CDS, DAQ, DetChar, ISC, SYS)
jeffrey.kissel@LIGO.ORG - posted 12:25, Wednesday 22 February 2023 - last comment - 12:07, Thursday 23 February 2023(67552)
Beating RF Oscillator Noise Polluting H1OMC0 ADC Channels including OMC DCPD Dark Noise ASD
J. Kissel, M. Pirello, R. Schofield

I've collated some data collected yesterday during Marc's re-configuring the h1omc0 IO chassis fans (LHO:67519), and took a few more data sets this morning, given that we were seeing evidence that 
    (a) No fan configuration was particularly great as there's tons of aliasing going on in each configuration, and
    (b) There is clear non-stationary, time-dependent noise swimming around in the OMC DCPD A Channel that is coherent with a shorted channel read out on the same ADC card.

EDIT: Robert & Marc blame the time-dependent noise on beating RF Oscillator noise, NOT on the fans. Even though this ADC card was segregated from the card-dense LSC0 IO chassis into its own OMC0 IO chassis there are still unsynchronized "clocks" or RF oscillators the ADC itself has one at least, and the Adnaco motherboard has a couple as well. They suspect that changing the fan characteristics played a role in changing the frequencies of the oscillators and thus changing the beat characteristics, rather that the fan's noise itself acoustically or electronically coupling directly.

The big picture data is shown in the attachment, where
    (1) I compare most recent data of the shorted CH16 against the two versions of the DCPDs (like in LHO:67530) to show that these features are coherent between the raw CH16 (neon green), the raw first-copy of the DCPDA channel (CH0, in red), and the average of all copies of the DCPDA channels (A0_IN1, in blue), and
    (2) The CH16 data from multiple times and configurations over the past few days showing how inconsistent the noise is.

The 8-trace upper plot already has too many traces, so I don't show the single raw copy, CH0 or the average of 4 copies,  A0_IN1 for each CH16 trace. However having watched the data go by, the identical features are present and at consistent amplitude relationship in each CH0 and A0_IN1 data for each CH16 time. I add a smattering of time's coherence in the lower panel to show this, that they are coherent at each time.

One might argue that having 0 fans on is the worst, but there's no *clear* evidence that 1 fan is better than 2 fans (though there may be *some* evidence, once I dig further into the details of this data).
We also see that a somewhere between ~10 minutes and a day's worth of IO chassis thermalization has an impact on frequency content, so it's unclear if our quick studies of "0 fan" and "2 fans" were a legitimate comparison.

We should again attack the fans to try to build up a story and try to mitigate what we can, and perhaps this is best tackled on a test stand -- unfortunately, the cross-section of me being present on site to take this live data and the IFO being locked on DC readout remains at 0.0 since last week. As such, I don't have this same data during nominal low noise to tell if these features will be / are problematic when there's the standard 10 [mA] of light on the PD... yet.

Further, and again, we're again limited by what's stored in the frames to confirm, e.g. DCPD B's channels see similar things. I'll slowly get better at displaying what I can
I'll also work on comparing this raw 524 kHz data against the 65 kHz and 16 kHz versions of the same channels to confirm that the *digital* anti-aliasing is doing a good job.
But -- if there's already aliasing down to the 10-1000 Hz range in this signals prior to the filtering -- that digital aliasing won't do any good.
Even worse, if the noise appears in the IO chassis / ADC card system, then putting an analog AA (rather than the existing pass-through filter) won't help.
Images attached to this report
Comments related to this report
jeffrey.kissel@LIGO.ORG - 12:07, Thursday 23 February 2023 (67579)
The previous version of this aLOG had suggested, via the title, that the IO chassis fans were polluting the ADC channels directly (I didn't make a guess as to the coupling mechanism, but I guess either acoustically or mechanically). Robert and Marc are confident that this isn't the case and blame unsynchronized oscillators. 

Robert provides further wisdom:
Most clocks are good to a PPM or a few, so, roughly, a MHz clock will drift around by 1 or a few Hz, and a 100 MHz clock will drift around by 100ish Hz. They move through these bands on thermal time scales, say seconds to tens of minutes depending on the thermal stability. I am a little skeptical that these are internal oscillators, just because we didn't see anything like this on the test stand. 

An easy check is to watch the spectrum and put an insulated finger on the clock chips to heat them and see if you can move the peaks around.  If not, I have cooled and heated nearby electronics chassis to see if they are the culprit (of course an educated guess helps) just removing the lid is often enough to change the temperature. I agree with Marc that more flow generally reduces temperature rise and thus fluctuations from ambient.