While the Beckhoff troubleshooting was going this morning, I ran a set of charge measurements on the unused ETMx SUS. After completion a few hours ago, OPS/commiss started relocking attempts which are ongoing.
Results of charge measurements to be posted later.
With all of the Beckhoff attempted repair work this morning, the ETMy ESD "railed" with all 5 channels reading ~-15k. I went down to EY and did the Push RED ON/OFF button, unplug far right DAQ cable, Push RED ON/OFF back ON button, replug DAQ cable in procedure. This worked - the DC bias channel is back to ~-32k while the other 4 channels are near zero at ~-200 as viewed on the lower right of the ETMY SUS screen.
There are several calibration lines whose injection amplitudes are set to give a signal-to-noise ratio of 100 for a 10-second integration time. Thus in typical strain noise spectra, made with a frequency resolution of order 0.1 Hz, these calibration lines show up about 100x above the noise floor. I have been concerned whether there is any non-linear noise conversion with the calibration lines at these amplitudes. Certainly we see upconversion around the test mass bounce and roll modes when they are at a high amplitude. Any egregious upconversion would have been flagged before, but it wasn't clear whether smaller effects had been ruled out before.
To test this, a few days ago the calibration folks made a controlled test where the calibration lines were turned off and then on, while everything else in the interferometer remained (nominally) the same. These times are:
Sep-11-2015 1:45:25 UTC, calibration lines turned off, no excitations, fully locked at NOMINAL_LOWNOISE
Sep-11-2015 1:59:06 UTC, calibration lines turned back on
Sep-11-2015 2:50:03 UTC, unlocked
I made 0.05 Hz-resolution spectra of H1:CAL-DELTAL_EXTERNAL_DQ during these times; see the attached plots. I see no signs of any upconversion around the calibration lines in the 35-38 Hz band, nor the one near 332 Hz. I also looked around the first harmonics of the 35-38 Hz lines; there is no sign of twice the frequency of any of these lines showing up. So the calibration line amplitudes look ok from this standpoint. (Comment: I thought all 3 lines in the 35-38 Hz band were to have a high amplitude -- SNR ~ 100 -- but the 37.3 Hz line is smaller by a large factor.)
First figure: zoom around the 331.9 Hz line
Second figure: zoom around the 35-38 Hz band (upper) and twice that frequency (lower)
Note: the FFTs were performed with a Hanning window; the strong calibration lines in these spectra display the spectral leakage associated with the windowing/finite time series
The 37.3 Hz line is injected at the DARM CONTROL excitation point (See fig. 1 in LIGO-T1500377). Because of how Delta L_ext is generated (A*d_ctrl + d_err/C), the lines injected in DARM CONTROL (x_ctrl) don't appear in Delta L_ext.
So of the three calibration likes inear 36 Hz we only expect to see the x_tst line at 35.9 Hz and the Pcal line at 36.7 Hz.
Andy, Laura, Dr. Dan Hoak Something went wrong with the Pcal Y. It's not clear right now whether these were causing the glitches in ETMY, or whether there's a more general electrical failure at EY. The Beckhoff seems to have failed at 5:45 UTC (alog). At that time, we see the PCAL Y lines suddenly disappear, and this is obvious in DARM (first plot) as well as the PCAL Y photodiodes (plots 2 and 3). There are bursts of noise in the TX channel seen by the summary page spectrogram (plot 4), but shorter spectrograms of TX and RX don't show anything at times when we know DARM was glitching (plots 5 and 6). We'll continue to investigate.
Seems like pcal was gone somewhat after Beckhoff. A big DARM glitch came almost exactly at the same time as the Pcal power loss at 05:44:07.91, which was somewhat after beckhoff communication loss (1st attachment). Right after that glitch DARM became 2Hz-rich.
In the second plot, you can see that AOM drive, Pcal receiver and transmitter went away at the same time. Since there was no light, Pcal cannot be the cause of 2Hz thing. It's still possible that the loss of Beckhof was somehow responsible.
I checked mainsmon at EY though I don't know what exactly these are monitoring. I don't see mainsmon glitch when pcal was lost (2nd attachment middle), though it does glitch once in a while, e.g. t~440. This is through 60Hz and 180Hz band stop.
16:45 UTC Sheila, Filiberto, Vern, Nutsinee to end Y to investigate problem with Beckhoff. Looks like some of the channels froze around 09/15 5:45 UTC. May be coincident with start of 2 Hz comb in spectrum.
Does this have any interaction with the Pcal? We've seen something go very wrong there at the time near where the 2 Hz glitches started (we'll alog that shortly). There's also something wrong with some ETMY M0 OSEMs. Detchar is happy to have this investigated at the cost of locked time. This problem ruins the data and it's probably unanalyzable.
The PCAL team is investigating.
17:37 UTC Vern, Nutsinee, Filiberto and Sheila back. Brought Beckhoff chassis with them. Vern says they found a broken Beckhoff terminal.
18:11 UTC Filiberto, Sheila, Vern replacing end station 2 Ethercat chassis at end Y with spare 18:31 UTC Error remains in Beckhoff. Team at end Y is restarting Beckhoff computer. I put ISC_LOCK guardian to manual and down prior.
18:48 UTC Hugh burtrestored h1ecaty1plc1, h1ecaty1plc2, h1ecaty1plc3 to 09/14/22:10 as requested by team at end Y.
19:32 UTC Filiberto restarted Beckhoff vacuum gauges, going to turn back on high voltage for ESD
Strange DC level shift in EY microphone channels, especially EBAY racks (see ch3). But not low frequency mic.
As soon as the Beckhof froze, EY microphone signals DC level went down, and after Vern/Sheila/Elli/Fil restarted Beckhof the DC level came back close to original. Why? Is the ground level of ebay area pulled by something else controlled by Beckhof (or Beckhof chassis itself)?
Attached are 2 minutes of full data time series of the two ISI stages CPSs. Nothing 2hzish looking here. I zoomed in as well.
Also attached are 0.01hz bw spectra of the CPSs. Reference traces are from ~0400utc, before the comb started. Current traces are from 1200utc about 0500 local. There are some spikes around 2 hz but that are barely above the background and not only in the current traces. These don't look to be the source of this comb.
15:25 UTC Out of observing 15:34 UTC Fred R., David N., John W., Bubba G., Albert L., Peter K. to LVEA for safety audit 15:41 UTC LVEA camera shows group entering LVEA 16:25 UTC Group out of LVEA, going to end X 16:40 UTC Group entering end X 17:00 UTC Group leaving end X, going to end Y 17:26 UTC Group going into end Y 17:47 UTC Group out of end Y, coming back to corner station
STATE Of H1: Observing OUTGOING OPERATOR: Nutsinee QUICK SUMMARY: Comb in spectrum up to ~ 100 Hz. Range around 56 MPc. Lights on in LVEA. Lights appear off at mid and end stations. Lights appear off in PSL enclosure. Winds are around 5 mph. 0.03 - .1 Hz seismic is around 10^-2. 0.1 - 0.3 Hz seismic has been slowly increasing.
Title: 09/15 OWL Shift: 7:00-15:00UTC (00:00-8:00PDT), all times posted in UTC
State of H1: Observation Mode at 60 Mpc for the past 9 hours. The range has been dropping slowly.
Support: I called Jenne about the 2Hz comb. Thomas Abbott was helping me with the comb hunting.
Shift Summary: Besides the nasty looking 2Hz comb up to 100Hz, seismic activity was minimal. Wind speed <=10mph. The 2Hz comb shows up on EY SUSRACK magnetometer, ETMY top stage F1, F2, and SD, OMC DCPD, and ETMY ISI ST2 GS13.
Incoming Operator: Patrick
Activity Log:
12:00 UTC Christina and Karen on site.
15:00 Handing off to Patrick
There's an increase in seismic activity at 0.1-0.3 Hz band. Wind speed ~10 mph. The 2Hz comb is still present. Thomas Abbott has been helping me figuring out where it could have come from (alog21528)
Note: while investigating the 2Hz glitches we noticed 1Hz glitches at EX seirack magnetometer.
This 2 Hz comb is actually glitches every half a second. It seems like the period is not exactly half a second. They appear in DARM but not MICH, SRCL, or PRCL, so it's probably not a bad PR or SR mirror driver. Attached are a few Omega scans. The first is several seconds long and shows that these are very short and span about 40 to 100 Hz. The next three Omega scans are 4 seconds apart, showing that these don't sit right on the integer second and have a period that is not quite 0.5 seconds.
The comb shows up in the ETMY top stage F1, F2, and SD but none of the others. See attached plots.
Nutsinee, Thomas A., Ran omega scan for 1126336348 (Sep 15 2015 07:12:11 UTC), and found that the once-per-0.5s glitches repeating glitches are noticeable in: H1:OMC-DCPD_SUM_OUT_DQ H1:ISI-ETMY_ST2_BLND_RZ_GS13_CUR_IN1_DQ H1:ISI-ETMY_ST2_BLND_Z_GS13_CUR_IN1_DQ H1:PEM-EY_MAG_EBAY_SUSRACK_QUAD_SUM_DQ H1:PEM-EY_MAG_EBAY_SUSRACK_X_DQ H1:PEM-EY_MAG_EBAY_SUSRACK_Y_DQ H1:PEM-EY_MAG_EBAY_SUSRACK_Z_DQ Full omega scan: https://ldas-jobs.ligo-wa.caltech.edu/~tabbott/H12HzComb/
Jeff, Darkhan, Sudarshan, Craig, Kiwamu,
(Verification measurement)
The above screen shot shows a measured transfer function from displacement estimated by Pcal Y to displacement estimated by CAL-CS. They agree within +/- 10 % in magnitude and +/- 5 deg in phase all across the frequency band we swept. Note that one data point at 10 Hz showed magnitude that is slightly above 10%, but this was not repeatable and therefore we don't think it is a reliable data point. We measured the same transfer function three times within the same lock stretch and saw the magnitude changing to a value between 0.85 and 1.1 at this particular frequency point. We are guessing that this is due to a bounce mode confusing our measurement.
Also, even though the coherence was high all across the frequency band, the data points below 30 Hz seemed to change in magnitude in every sweep. So we increased the integration time from 3 sec to 6 sec which seemed to improved the flatness.
The optical gain was adjusted by measuring the sensing function with a Pcal sweep within the same lock stretch. This gave me a 341 Hz cavity pole (which is the same as two nights ago, alog 21352) and an optical gain of 8.834e-7 meters/counts. Both the parameters are now loaded into the CALCS foton file and enabled.
(Phase correction)
Sudarshan will make a separate alog on this topic, but a trick to get this beautiful plot was to properly incorporate the know time delays. Based on our knowledge, we have included a 115 usec = (41 + 61 + 13 usec) time delay. If we did not remove the delay, the phase would have been off by 40 deg at 1 kHz.
(An extra measurement)
Independently of the calibration validation measurement, we did a simple measurement -- check the binary range with and without the calibration lines. Here is the relevant time stamps:
We will check the range later.
All the data are accessible at the following SVN locations:
DARM open loop measurements
aligocalibration/trunk/Runs/ER8/H1/Measurements/DARMOLGTFs/2015-09-10_H1_DARM_OLGTF_7to1200Hz.xml
aligocalibration/trunk/Runs/ER8/H1/Measurements/DARMOLGTFs/2015-09-10_H1_DARM_OLGTF_7to1200Hz_halfamp.xml
For the analysis, I have used the first measurement. The second measurement was meant to assess repeatability of the measurement by applying the half size of the usual excitation in DARM.
Pcal to DARM responses:
aligocalibration/trunk/Runs/ER8/H1/Measurements/PCAL/2015-09-10_PCALY2DARMTF_7to1200Hz.xml
aligocalibration/trunk/Runs/ER8/H1/Measurements/PCAL/2015-09-10_PCALY2DARMTF_7to1200Hz_v2.xml
aligocalibration/trunk/Runs/ER8/H1/Measurements/PCAL/2015-09-10_PCALY2DARMTF_7to1200Hz_v3.xml
The final plot, that I have posted above, is from the third measurement in which I have doubled the integration time in order to obtain better signal-to-noise ratio.
DARM paramter file (as reporeted in alog 21386):
aligocalibration/trunk/Runs/ER8/H1/Scripts/DARMOLGTFs/H1DARMparams_1125963332.m
On 2015-09-12 06:30:00, the gain from the DCPD sum to DARM IN1 was 3.477×10−7 ct/mA. Therefore, using Kiwamu's number of 8.834×10−7 m/ct, this gives the optical gain as 3.26 mA/pm. (One stage of DCPD whitening.)