Ansel, Ben, Sheila, Daniel, Fil, Georgia
We went to End X to investigate the extra noise we are seeing in the SUSRACK and SEIRACK magnetometers (46999, 64930, 64526).
In the morning we tried turning off 4 Beckhoff chassis in the EE bay while monitoring the magnetomoeters as well as a PEM channel which monitors the ESD 18V power supply. We found that turning off the link chassis, the ISC common chassis, and the end station 2 and 3 chassis (first attachment) reduced the broadband noise seen in both magnetometers and the ESD power supply (second attachment) and squashed the 95Hz peak as seen by the SUSRACK magnetometer (but not SEIRACK?). The noise improvement seemed to be partially due to the ISC common and partially due to end station 2 - end station 3 didn't seem to contibute to the noise.
We went back with Fil in the afternoon and tried:
- Powering off suspicious chassis out at the field rack, next to the chamber, disconnecting various things from the Beckhoff 24V power supply which is closest to the ESD rack (this was mostly uneventful, powering off the baffle PD amplifier 24V power supply added extra noise seen by the ESD power supply monitor (weird)
- Pulling dodgey-looking electronics related to the buried PEM STS2, and some loose cables connected to the racks (attachments 3 and 4) (also didn't affect the noise)
- Powering off the Beckhoff chassis mentioned above (end station 2, end station 3, ISC common) one-by-one to try to pin down the noise contributors (ISC common powering off showed clear noise reduction)
- Floating the ground of the 24V power supply which powers the Beckhoff chassis (this also reduced the noise!)
We're still working on making some plots, but attached below are my full notes with local times.
At the chamber-side racks
15:08:45 unplug dodgey clip doodle from rack
15:15:45 24V illuminator power supply + chassis powered off (no change)
15:20:27 Hepi pump controller 24V unplugged
15:26:58 Hepi pump controller 24V plugged in again (sorry Jim)
15:30:10 shiney beckhoff hub distribution box 24V unplugged (no change)
15:35:00 24V fiber to copper converter beckhoff field rack (maybe some high frequency broadband noise in esd power supply monitor)
Considering pulling STS2 seismometer (when we touched the cables there was noise in the esd power supply monitors)
15:43:47 cps timing chassis off
15:45:00 pulling buried seismometer electronics and cables
15:52:04 baffle pd amplifier powered off (high noise in esd power supply monitor?)
15:53:45 unplugged field cable (from amplified to chamber) pulled for baffle PD (noise back to normal...)
Into the EE bay
16:01:00 Beckhoff ISC common fiber out unplugged
16:03:11 ISC common chassis powered off only
16:06:00 ISC common back on
16:07:00 end station 2 powered off
16:11:00 end station 2 powered back on (extra noise on power monitors for a little bit, settled after a min or 2)
16:12:43 end station 3 powered off
16:14:30 end station 3 powered back on (no excess noise this time, maybe not powered off for long enough)
16:15:20 end station 2 powered off
16:15:40 end station 2 powered on again
16:25:20 isc c1 beckhoff rack power supply ground floating (improved the noise on the magnetometer to same level as when beckhoff chassis powered off)
16:28:20 isc common powered off while power supply floating
16:29:10 end station 2 powered off while power supply floating
16:30:30 end station 3 powerwd off while power supply floating
16:30:55 end station link powered off while power supply floating
16:36 18V and 48V monitors unplugged
After this we plugged everything back in and reconnected the 24V ground to the rack as we found it.
Here are some times from the test we did yesterday (9/20/22) morning (sumarized in Georgia's plot).
18:10:22 UTC 4 beckhoff chassis powered off: End link, End station 2 and End station 3, ISC Common (noise in sus and seirack magnetometers reduced, and in ESD voltage monitor)
18:39:18 UTC, all chassis back on and functioning after some confusiong about why they didn't restart when we powered them back on. (magnetometers and power monitor back to original noisy state)
18:43:34 End Station 3 chassis powered off, no change in magnetometers of power monitor
18:45:47 End station 2 + 3 off, some reduction of noise in magnetometers and ESD supply monitor
18:48:55 End station 2,3 and ISC Common off more noise reduction
18:59:13 ISC Common powered back on, fiber out of that chassis to floor modules in facilties rack disconnected (turns off baffle PD modules, illuminator, ect).
The TwinCAT update rate over fiber/EtherCAT is 100Hz (proabably pretty sharp) as set in software.
Adding some plots from the afternoon tests described by Georgia above.
Blue and green lines show the effect of turning ISC common off (nothing else off) and floating the ground of the 24V power supply for the Beckhoff chassis (ISC common on). These two tests produced almost identical results. It looks like the 95 Hz peaks were shifted in both cases, too.
Pink curve shows that powering off the end station 2 chassis (nothing else off) has an effect, but not as much as ISC common.
Sheila recommended comparing these results to a previous time when the ESD electronics grounding to the beam tube was lifted (alog 64638), which is shown in the brown curve, but a similar noise reduction was not seen at that time.
I dug a little deeper trying to work out when exactly the noise in the SEIRACK ebay magnetometer got worse.
- The broadband noise in the megnetometer is a bit confusing and lumpy (first attachment). We know from Ben's investigation that the noise changed at some point between Feb 2020 and May 2021, and then again between May 2021 and August 2022. I think the first attached plot supports this and narrows down the first change to between November 2020 and Feb 2021.
- I've pinned down one change in the noise which happened in January 2022 (second attachment shows spectra for a few different times in late 2021, early 2022), and a line around 97 Hz appeard (not as broad as the current feature we're seeing). It seems like when the Baffle PD amplifier was switched on 12 Jan 2022 (alog 61271) was wen the noise changed (third attachment).
- I looked back at the times on Tuesday when we turned off the Baffle PD Ethercat chassis power supply and disconnected the chassis from the chamber and saw no change in the noise though!
- From Ansel's plots above turning off the ISC common Beckhoff chassis significantly reduces the 95Hz peak too, ISC common covers the ALS slow controls, and is unrelated to the baffle pd
I noticed that in a recent Bruco of Gabriele's there was also some weaker coherence at the 95Hz peak with the sus rack magnetometer at EY. The line at EY is narrower spectrally than EX, and shifted in frequency, but there is definietely something there (first attachment)
I made a spectrogram of the EX magnetometer while we were turning off the Beckhoff chassis one-by-one on Tuesday afternoon.
- between t=180 and t=360 in we powered off the ISC common chassis, which crushes the 95Hz peak and reduces the broadband noise
- around t=1500 Fil lifted the ground for the Beckhoff 24V power supply which improved the noise
- At t=1700 we turned off ISC common chassis while the ground remained lifted. This appeared to improve things a little.
- While ISC common was off, at t=1750 we powered off end station 2. At t~1800 we powered off end station 3, and at t~1830 we powered off the end station link.
I made a spectrogram of the EY magnetometer during this time too. Weirdly when we turned off EX ISC common there is a small but visible change in the EY magnetometer.
Adding a few more plots and info from this afternoon's investigations with a portable magnetometer. Note that the X axis of the magnetometer wasn't working, which is why only two axes are plotted here.
Summary: We confirmed that turning off ISC common has a large effect. Figs 1 and 2 show the on/off comparison for a magnetometer placed in the same rack (actually on top of the end station 3 chassis). Fig 3 shows the on/off comparison for a magnetometer placed near the ESD driver. Here the difference is less obvious, but the 95 and 100 Hz peaks are affected. After we returned from EX, I also plotted the ADC channels (same ones mentioned in previous comments) which are monitoring the ESD power supplies. Figs 3 and 4 use the same time stamps.
We went to EX again today and did some more tests with the magnetometer, hunting for the coupling from the ISC common Beckhoff chassis and the ESD power supply.
- We tried turning off End station 2 again, and found that for some reason today turning off that chassis increased the noise (broadband and 95Hz) in the magnetometer and in the ESD power supply (fig 1)
- We tried placing the magnetometer at a couple of points between the ebay racks and the racks on the floor and found that the 95Hz peak is not visible in the magnetometer between those two racks (fig 2), we the coupling is not radiative
- Ben found that the 95Hz peak is larger in the magnetometer when it placed on top of the PEM power distribution chassis (fig 3, fig 4, 5 for photos) which delivers 12V power to the PCAL, some seismometers, tempeterature sensors and something in the BRS
- We tried powering that chassis off. This cut the power to the magnetometer, but the peak was still visible in the power line monitors so is not the cause of the coupling
- With the magnetometer on the ESD chassis we disconnected the power monitor cable at the ADC in the ebay, but this did not change the peak in the magnetometer, so the monitor cable is not that cause of the coupling.
I went to EX and took more measurements with the magnetometer around the VEA. I've highlighted some interesting points in the power spectrum plots. The 1-900 Hz plot shows more broadband behavior of the magnetometer and the 50-150Hz plot is meant to focus on the 95 Hz peak we've been investigating.
First, the red curve serves as a reference for the general level of noise I saw in the VEA (in this particular time I was right next to the +X wall of the VEA).
The blue curve shows the magnetometer on top of the ESD drivers. We originally suspected this as a possible source of the 95 Hz peak in the VEA because we were seeing the peak throughout the whole electronics rack that the ESD driver is a part of.
The green curve shows when the magnetometer was placed on the DC Power Sequencer (pictured) which sits on top of the VEA electronics rack. This was the place I found to have the highest amount of magnetometer noise. Note that the 95Hz peak is highest in magnitude here.
Finally, the brown curve represents when the magnetometer was placed under the beam tube. This was the only place aside from the electronics rack that I could find elevated levels of magnetometer noise in the VEA. This noise is consistent along the beam tube, and it seems to be correlated with the cable tray that runs under the beam tube.
Early this afternoon Fil, Ansel and I went back to EX.
Fil fixed a power issue with the whitening chassis (acromag?) which was powered off 18V rather than the required 24V. We looked at the magnetometers while the arcomag was powered off and saw no change in the noise witnessed by the magnetometers and the ESD power monitor.
We tried disconnecting all the field cables from ISC common. this also didn't change the noise (95Hz peak or broadband noise) - interesting!
We tried powering off end station 2 and saw the noise slightly decrease. Fil unplugged all the field cable from end station 2 and this did not change anything.
Other stuff we tried which didn't affect the nosie
- We tried powering off the 18 and 48V supply to the ESD pass through chassis - this did not affect the noise
- Fil powered off the SUS BIO chassis in the ebay - this did not affect the noise
- Fil unplugged the SUS BIO from the pass through chassis - this did not affect the noise
- We tried powering off the supply to the Baffle PD amplifier at the field rack - this resulted in temporarily increased broadband noise in the magnetometer but not in the ESD power supply monitor
- We unplugged field cable from the baffle PD amplifier, noise back to normal levels
- We replugged in the field cable - noise is still at normal levels
- We turned the power supply back on, noise is still the same
Some times I noted (Ansel took better notes though)
20:01:00 acromag off - no change
20:06:00 isc common off
20:07:36 isc common on, field cables ALL disconnected (no change)
20:11:00 end station 2 off (everything still disconnected from ISC common) noise slightly lower
13:15:16 End station 2 off and everything disconnected (ISC common on, everything disconnected)
20:17:28 End station 2 on, everything disconnected - noise is back
Not coming through the field cables!
on top of rack with the ESD 10V and 48V is: 18V oplev, illuminator, PEM microphone, BRS, etc Maybe we should move that?
13:47:35 SUS BIO chassis off at the field rack - no change