There are several calibration lines whose injection amplitudes are set to give a signal-to-noise ratio of 100 for a 10-second integration time. Thus in typical strain noise spectra, made with a frequency resolution of order 0.1 Hz, these calibration lines show up about 100x above the noise floor. I have been concerned whether there is any non-linear noise conversion with the calibration lines at these amplitudes. Certainly we see upconversion around the test mass bounce and roll modes when they are at a high amplitude. Any egregious upconversion would have been flagged before, but it wasn't clear whether smaller effects had been ruled out before.
To test this, a few days ago the calibration folks made a controlled test where the calibration lines were turned off and then on, while everything else in the interferometer remained (nominally) the same. These times are:
Sep-11-2015 1:45:25 UTC, calibration lines turned off, no excitations, fully locked at NOMINAL_LOWNOISE
Sep-11-2015 1:59:06 UTC, calibration lines turned back on
Sep-11-2015 2:50:03 UTC, unlocked
I made 0.05 Hz-resolution spectra of H1:CAL-DELTAL_EXTERNAL_DQ during these times; see the attached plots. I see no signs of any upconversion around the calibration lines in the 35-38 Hz band, nor the one near 332 Hz. I also looked around the first harmonics of the 35-38 Hz lines; there is no sign of twice the frequency of any of these lines showing up. So the calibration line amplitudes look ok from this standpoint. (Comment: I thought all 3 lines in the 35-38 Hz band were to have a high amplitude -- SNR ~ 100 -- but the 37.3 Hz line is smaller by a large factor.)
First figure: zoom around the 331.9 Hz line
Second figure: zoom around the 35-38 Hz band (upper) and twice that frequency (lower)
Note: the FFTs were performed with a Hanning window; the strong calibration lines in these spectra display the spectral leakage associated with the windowing/finite time series
The 37.3 Hz line is injected at the DARM CONTROL excitation point (See fig. 1 in LIGO-T1500377). Because of how Delta L_ext is generated (A*d_ctrl + d_err/C), the lines injected in DARM CONTROL (x_ctrl) don't appear in Delta L_ext.
So of the three calibration likes inear 36 Hz we only expect to see the x_tst line at 35.9 Hz and the Pcal line at 36.7 Hz.
Andy, Laura, Dr. Dan Hoak Something went wrong with the Pcal Y. It's not clear right now whether these were causing the glitches in ETMY, or whether there's a more general electrical failure at EY. The Beckhoff seems to have failed at 5:45 UTC (alog). At that time, we see the PCAL Y lines suddenly disappear, and this is obvious in DARM (first plot) as well as the PCAL Y photodiodes (plots 2 and 3). There are bursts of noise in the TX channel seen by the summary page spectrogram (plot 4), but shorter spectrograms of TX and RX don't show anything at times when we know DARM was glitching (plots 5 and 6). We'll continue to investigate.
Seems like pcal was gone somewhat after Beckhoff. A big DARM glitch came almost exactly at the same time as the Pcal power loss at 05:44:07.91, which was somewhat after beckhoff communication loss (1st attachment). Right after that glitch DARM became 2Hz-rich.
In the second plot, you can see that AOM drive, Pcal receiver and transmitter went away at the same time. Since there was no light, Pcal cannot be the cause of 2Hz thing. It's still possible that the loss of Beckhof was somehow responsible.
I checked mainsmon at EY though I don't know what exactly these are monitoring. I don't see mainsmon glitch when pcal was lost (2nd attachment middle), though it does glitch once in a while, e.g. t~440. This is through 60Hz and 180Hz band stop.
16:45 UTC Sheila, Filiberto, Vern, Nutsinee to end Y to investigate problem with Beckhoff. Looks like some of the channels froze around 09/15 5:45 UTC. May be coincident with start of 2 Hz comb in spectrum.
Does this have any interaction with the Pcal? We've seen something go very wrong there at the time near where the 2 Hz glitches started (we'll alog that shortly). There's also something wrong with some ETMY M0 OSEMs. Detchar is happy to have this investigated at the cost of locked time. This problem ruins the data and it's probably unanalyzable.
The PCAL team is investigating.
17:37 UTC Vern, Nutsinee, Filiberto and Sheila back. Brought Beckhoff chassis with them. Vern says they found a broken Beckhoff terminal.
18:11 UTC Filiberto, Sheila, Vern replacing end station 2 Ethercat chassis at end Y with spare 18:31 UTC Error remains in Beckhoff. Team at end Y is restarting Beckhoff computer. I put ISC_LOCK guardian to manual and down prior.
18:48 UTC Hugh burtrestored h1ecaty1plc1, h1ecaty1plc2, h1ecaty1plc3 to 09/14/22:10 as requested by team at end Y.
19:32 UTC Filiberto restarted Beckhoff vacuum gauges, going to turn back on high voltage for ESD
Strange DC level shift in EY microphone channels, especially EBAY racks (see ch3). But not low frequency mic.
As soon as the Beckhof froze, EY microphone signals DC level went down, and after Vern/Sheila/Elli/Fil restarted Beckhof the DC level came back close to original. Why? Is the ground level of ebay area pulled by something else controlled by Beckhof (or Beckhof chassis itself)?
Attached are 2 minutes of full data time series of the two ISI stages CPSs. Nothing 2hzish looking here. I zoomed in as well.
Also attached are 0.01hz bw spectra of the CPSs. Reference traces are from ~0400utc, before the comb started. Current traces are from 1200utc about 0500 local. There are some spikes around 2 hz but that are barely above the background and not only in the current traces. These don't look to be the source of this comb.
15:25 UTC Out of observing 15:34 UTC Fred R., David N., John W., Bubba G., Albert L., Peter K. to LVEA for safety audit 15:41 UTC LVEA camera shows group entering LVEA 16:25 UTC Group out of LVEA, going to end X 16:40 UTC Group entering end X 17:00 UTC Group leaving end X, going to end Y 17:26 UTC Group going into end Y 17:47 UTC Group out of end Y, coming back to corner station
STATE Of H1: Observing OUTGOING OPERATOR: Nutsinee QUICK SUMMARY: Comb in spectrum up to ~ 100 Hz. Range around 56 MPc. Lights on in LVEA. Lights appear off at mid and end stations. Lights appear off in PSL enclosure. Winds are around 5 mph. 0.03 - .1 Hz seismic is around 10^-2. 0.1 - 0.3 Hz seismic has been slowly increasing.
Title: 09/15 OWL Shift: 7:00-15:00UTC (00:00-8:00PDT), all times posted in UTC
State of H1: Observation Mode at 60 Mpc for the past 9 hours. The range has been dropping slowly.
Support: I called Jenne about the 2Hz comb. Thomas Abbott was helping me with the comb hunting.
Shift Summary: Besides the nasty looking 2Hz comb up to 100Hz, seismic activity was minimal. Wind speed <=10mph. The 2Hz comb shows up on EY SUSRACK magnetometer, ETMY top stage F1, F2, and SD, OMC DCPD, and ETMY ISI ST2 GS13.
Incoming Operator: Patrick
Activity Log:
12:00 UTC Christina and Karen on site.
15:00 Handing off to Patrick
There's an increase in seismic activity at 0.1-0.3 Hz band. Wind speed ~10 mph. The 2Hz comb is still present. Thomas Abbott has been helping me figuring out where it could have come from (alog21528)
Note: while investigating the 2Hz glitches we noticed 1Hz glitches at EX seirack magnetometer.
This 2 Hz comb is actually glitches every half a second. It seems like the period is not exactly half a second. They appear in DARM but not MICH, SRCL, or PRCL, so it's probably not a bad PR or SR mirror driver. Attached are a few Omega scans. The first is several seconds long and shows that these are very short and span about 40 to 100 Hz. The next three Omega scans are 4 seconds apart, showing that these don't sit right on the integer second and have a period that is not quite 0.5 seconds.
The comb shows up in the ETMY top stage F1, F2, and SD but none of the others. See attached plots.
Nutsinee, Thomas A., Ran omega scan for 1126336348 (Sep 15 2015 07:12:11 UTC), and found that the once-per-0.5s glitches repeating glitches are noticeable in: H1:OMC-DCPD_SUM_OUT_DQ H1:ISI-ETMY_ST2_BLND_RZ_GS13_CUR_IN1_DQ H1:ISI-ETMY_ST2_BLND_Z_GS13_CUR_IN1_DQ H1:PEM-EY_MAG_EBAY_SUSRACK_QUAD_SUM_DQ H1:PEM-EY_MAG_EBAY_SUSRACK_X_DQ H1:PEM-EY_MAG_EBAY_SUSRACK_Y_DQ H1:PEM-EY_MAG_EBAY_SUSRACK_Z_DQ Full omega scan: https://ldas-jobs.ligo-wa.caltech.edu/~tabbott/H12HzComb/
Title: 09/15 OWL Shift: 7:00-15:00UTC (00:00-8:00PDT), all times posted in UTC
State of H1: Observation Mode at 60Mpc for the last 2.5 hrs. Was at 70 Mpc before.
Outgoing Operator: Ed M.
Quick Summary: The 2Hz comb appeared on DARM 2.5 hours ago for no apparent reason. It shows up on EY SUS rack magnetometer. I called Jenne and she suggested I leave it there since the ifo is still locked. The light in the LVEA is on. No one else is on site. Wind speed below 10 mph. No seismic activity.