Josh, Andy, David, Jess
Conclusion: We suspect that the 50Hz glitches in DARM are caused by EX Mains glitches that happen 400+/-70ms earlier (reliably matched) and may(?) couple through EX Seismic.
Throughout September there has been a persitent line of glitches at 50Hz in DARM with a pretty high rate. See the attached glitch-gram from today's summary pages. Based on the one hour plots on the summary pages, the rate of these 50Hz glitches in DARM is about once per minute.
We found (see e.g. the first round here) that these DARM glitches are correlated in time with glitches in EX channels, particularly PEM EX SEIS/ACC*, ISI ETMX ST1 BLND*, and ASC-X_TR*.
It looks like the cause of this may be a big glitch in EX Mains. Wew see the EX and DARM channels are all glitching with a 400+/-70ms delay (averaged five examples followed up by hand) after EX mainsmon. Attached are two examples where we lined up the glitch in EX mainsmon and showed that the glitches in EX seismic and DARM are 400ms later.
Notes:
- The EX Mains glitches are broadband, but EX Seis, ISI, and DARM are all only localized glitches at 50Hz.
- We checked and the EX microphones look clean.
- The glitches seen by EX mainsmon are also seen (sometimes) very weakly by the LVEA mainsmons (at the same time, i.e. earlier than in other EX channels and DARM).
We would be happy to follow up further leads, but wanted to report what we found so far.
PS. Should you wish to repeat this yourself, attached is a list of these glitches in DARM today (with nice GPS times) and a screenshot of the ldvw settings I used.
Note that the signal in the seismometer is a bit later than the mainsmon glitch, and that the seismometer glitch is narrowband around 50 Hz in contrast to the mainsmon. A guess is that there is a mechanical coupling -- maybe a motor running slow due to it hitting something, or a transformer core pulsing -- which induces the 50 Hz characteristic. Seems like it might be easy to hear. The delay suggests that it could be something a bit away -- maybe a chiller?
The chiller compressor is probably on the order of 100 hp (>100 amps at 480v) and is cycling with a period of ~1hour during most of the day. The period depends on the heat load into the building. During the hot time of day one chiller compressor will continue to run while a second compressor will cycle as needed. In the attached plot the peaks represent the start of the chiller compressor. The valleys represent the shutdown. Times are PDT.
After emails with Robert S and John W we did an extensive search in the "HVE-EX:*" and "H0:FMC-EX*" channels for anything that is switching on or off at the same time as the glitches reported above. John Worden suggested that the instrument air compressor (a ~5hp motor at EX) would switch on and off with about the right period. I attach a plot that shows that the glitches discussed above for EX seismic (left) and DARM (right) are correlated with the compressor turning ON at the bottom of the triangle waveforms.
