While investigating the 2 Hz comb in DARM, Sheila and Patrick found communications errors with beckhoff at EY. The End Station 2 Ethercat chasssis was brought back to the lab for further troubleshooting. Spare unit was installed and Beckhoff computer was restarted.

While troubling shooting the unit found a bad EK1100 coupler on the third rail (left) of the chassis. Re-scanned unit and found 5 terminals all EL3104 (anaolog inputs) that were giving us errors. After re-scanning/checking internal cabling/power cycling, we were only able to reproduce 3 errors. We tried multiply times to try and reproduce all five errors, but could not. Eventually after a power cycle, all errors we had previously seen could not be reproduce. 

After multiple re-scans and power cycles with no errors showing up, we used a voltage calibrator and injected a 5V DC signal into the EL3104 terminals. It was later decided to reinstall unit back at EY.

I was trying to use the fast ODC channels (e.g. H1:SUS-ETMY_ODC_CHANNEL_OUT_DQ) to track down ETMY saturation causes, but ran into a number of bugs in our data access tools that made it impossible:

1) dataviewer can play the trend data of integer ODC channels, but not the full data.

2) both dtt (diaggui) and lockloss (pydv) can't access integer ODC channels

3) using NDS2, the ODC channel is down-sampled to 256 Hz, making it useless for my purpose.
(why do we do that? That channel compresses either way, and takes the same amount of disk space)

4) dtt (diaggui) can access that channel, but only with the "now" setting, and

  IT RETURNS DATA WITH A TIME STAMP IN THE FUTURE!!!!

(The attached plot has a UTC clock in the terminal, and dtt's repored time stamp...)


sigh... I give up for today.

TITLE: Sep 15 EVE Shift 23:00-07:00UTC (16:00-00:00 PDT), all times posted in UTC

STATE Of H1: Locking

OUTGOING OPERATOR: Patrick/Jeff B

QUICK SUMMARY:Full control room, mostly working on last night’s 2Hz comb problem. Patrick is in H2 examining the EtherCat chassis from EY, responsible for Ring Heaters, for baddness. Lights are on in the LVEA. Wind is below 20mph. Seismic activity looks a little rumbly perhaps from a quake 7 hours ago and subsequent smaller ones in various areas. IFO is locked again for the first time in 7 hours. We are currently going to make some compensatory Ring Heater adjustments

Hugh, Sheila, Jenne, filling in for Patrick

We are filling in for Patrick at the moment because he is helping Fil to use the beckhoff test stand in the H2 building.  We are trying to lock now after Patrick and Jenne did a full initial alignment.  

This morning, Fil, Vern, Elli, Nutsinee and I swapped out the End Station 2 Beckhoff Chassis, because all the modules on the right rail were in the state INIT NO_COMM, and we could not change their state by requesting different states (including safe) or clearing errors.  We swapped the chassis with a spare.  Hugh and Patrick burt restored plc1,2,3 to 10 pm last night.  

When we first arrived we saw that the high voltage power supplies for the ESD were off, Fil turned them on and restored the settings.  This may be because the vacuum gauges which are controlled by beckhoff would have tripped them off. The ESD driver was off, but we did not reset it believing that we could reset it remotely. Later Betsy reset it by driving back to EY after she could not reset it in the normal way remotely.

Elli quickly noticed that something was amiss with the ring heaters at end Y, which are controlled by the same beckhoff chassis.  We checked that the settings were all the same, it seems like there is some difference between the 2 chassis.  

Fil, with help from several people, has spent most of the afternoon with the chassis we removed.  We had trouble getting a test set up going that we could use to diagnose the problem, so Hugh, Jenne, and I are relieving Patrick for a few hours so he and Fil can work together on the removed chassis.  They think that they have found a problem with one of the modules, and are currently replacing that module.  

We are about to reach low noise (with the spare chassis, and incorrect power on the ring heaters).  We hope to see that the comb is gone, and if we see that we may break the lock to revert to the old chassis if possible.  

Patrick's notes about his shift:

10 am PDT safety tour changed from X end to Y end(this may have broken our noisy lock this morning)

18:11 UTC placed beckhoff chassis at End Y

18:20 UTC Rick retrieves (?) PCAL out of optics lab

18:30 UTC Dick in optics lab out at 12:41 PDT

18:31 UTC Fil restarts beckhoff computer (Guardian to down)

a little while later h1ecaty1 burt restores to 22:10 Sep 14

18:53 UTC Jeff B, Jason retrieve part from optics lab (?)

12:38 PDT UPS truck at LSB

20:37 start of initial alignment finished at 21:22

20:54 Sheila to H2 electronics back 20:56, Fil is still there

20:58 Betsy to EY to restart ESD back 14:31

While the Beckhoff troubleshooting was going this morning, I ran a set of charge measurements on the unused ETMx SUS.  After completion a few hours ago, OPS/commiss started relocking attempts which are ongoing.

Results of charge measurements to be posted later.

With all of the Beckhoff attempted repair work this morning, the ETMy ESD "railed" with all 5 channels reading ~-15k.   I went down to EY and did the Push RED ON/OFF button, unplug far right DAQ cable, Push RED ON/OFF back ON button, replug DAQ cable in procedure.  This worked - the DC bias channel is back to ~-32k while the other 4 channels are near zero at ~-200 as viewed on the lower right of the ETMY SUS screen.

There are several calibration lines whose injection amplitudes are set to give a signal-to-noise ratio of 100 for a 10-second integration time. Thus in typical strain noise spectra, made with a frequency resolution of order 0.1 Hz, these calibration lines show up about 100x above the noise floor. I have been concerned whether there is any non-linear noise conversion with the calibration lines at these amplitudes. Certainly we see upconversion around the test mass bounce and roll modes when they are at a high amplitude. Any egregious upconversion would have been flagged before, but it wasn't clear whether smaller effects had been ruled out before.

To test this, a few days ago the calibration folks made a controlled test where the calibration lines were turned off and then on, while everything else in the interferometer remained (nominally) the same. These times are:

• 		Sep-11-2015 1:45:25 UTC, calibration lines turned off, no excitations, fully locked at NOMINAL_LOWNOISE

• 		Sep-11-2015 1:59:06 UTC, calibration lines turned back on

• 		Sep-11-2015 2:50:03 UTC, unlocked

I made 0.05 Hz-resolution spectra of H1:CAL-DELTAL_EXTERNAL_DQ during these times; see the attached plots. I see no signs of any upconversion around the calibration lines in the 35-38 Hz band, nor the one near 332 Hz. I also looked around the first harmonics of the 35-38 Hz lines; there is no sign of twice the frequency of any of these lines showing up. So the calibration line amplitudes look ok from this standpoint. (Comment: I thought all 3 lines in the 35-38 Hz band were to have a high amplitude -- SNR ~ 100 -- but the 37.3 Hz line is smaller by a large factor.)

First figure: zoom around the 331.9 Hz line

Second figure: zoom around the 35-38 Hz band (upper) and twice that frequency (lower)

Note: the FFTs were performed with a Hanning window; the strong calibration lines in these spectra display the spectral leakage associated with the windowing/finite time series

Andy, Laura, Dr. Dan Hoak

Something went wrong with the Pcal Y. It's not clear right now whether these were causing the glitches in ETMY, or whether there's a more general electrical failure at EY. The Beckhoff seems to have failed at 5:45 UTC (alog). At that time, we see the PCAL Y lines suddenly disappear, and this is obvious in DARM (first plot) as well as the PCAL Y photodiodes (plots 2 and 3). There are bursts of noise in the TX channel seen by the summary page spectrogram (plot 4), but shorter spectrograms of TX and RX don't show anything at times when we know DARM was glitching (plots 5 and 6). We'll continue to investigate.

16:45 UTC Sheila, Filiberto, Vern, Nutsinee to end Y to investigate problem with Beckhoff. Looks like some of the channels froze around 09/15 5:45 UTC. May be coincident with start of 2 Hz comb in spectrum.

Jeff, Darkhan, Sudarshan, Craig, Kiwamu,

We have updated the calibration filters in CAL-CS with the latest filters.

With a Pcal sweep measurement, we have confirmed that the current calibration is accurate within 10 % in magnitude and 5 deg in phase.

Calibration on the front end side is done.

(Verification measurement)

The above screen shot shows a measured transfer function from displacement estimated by Pcal Y to displacement estimated by CAL-CS. They agree within +/- 10 % in magnitude and +/- 5 deg in phase all across the frequency band we swept. Note that one data point at 10 Hz showed magnitude that is slightly above 10%, but this was not repeatable and therefore we don't think it is a reliable data point. We measured the same transfer function three times within the same lock stretch and saw the magnitude changing to a value between 0.85 and 1.1 at this particular frequency point. We are guessing that this is due to a bounce mode confusing our measurement.

Also, even though the coherence was high all across the frequency band, the data points below 30 Hz seemed to change in magnitude in every sweep. So we increased the integration time from 3 sec to 6 sec which seemed to improved the flatness.

The optical gain was adjusted by measuring the sensing function with a Pcal sweep within the same lock stretch. This gave me a 341 Hz cavity pole (which is the same as two nights ago, alog 21352) and an optical gain of 8.834e-7 meters/counts. Both the parameters are now loaded into the CALCS foton file and enabled.

(Phase correction)

Sudarshan will make a separate alog on this topic, but a trick to get this beautiful plot was to properly incorporate the know time delays. Based on our knowledge, we have included a 115 usec = (41 + 61 + 13 usec) time delay. If we did not remove the delay, the phase would have been off by 40 deg at 1 kHz.

(An extra measurement)

Independently of the calibration validation measurement, we did a simple measurement -- check the binary range with and without the calibration lines. Here is the relevant time stamps:

• Sep-11-2015 1:45:25 UTC, calibration lines turned off, not exctiations, fully locked at NOMINAL_LOSWNOISE
• Sep-11-2015 1:59:06 UTC, calibration lines turned back on.
• Sep-11-2015 2:50:03 UTC, unlocked

We will check the range later.