I'm not sure this is common knowledge, but every time an awgstream runs it logs in a daily log file, for example

/ligo/logs/h1/injection_log/2017/06/27/injection.txt

...

1182627394.000000 H1:SUS-ETMY_L3_ESDOUTF_LL_EXC start zotws10 16523 1 awgstream ./test_sinewave_ETMY.txt 1
1182627418.059011 H1:SUS-ETMY_L3_ESDOUTF_LL_EXC stop  zotws10 16523 1
1182627430.000000 H1:SUS-ETMY_L3_ESDOUTF_UR_EXC start zotws10 16536 1 awgstream ./test_sinewave_ETMY.txt 1
1182627454.055341 H1:SUS-ETMY_L3_ESDOUTF_UR_EXC stop  zotws10 16536 1
1182627466.000000 H1:SUS-ETMY_L3_ESDOUTF_LR_EXC start zotws10 16548 1 awgstream ./test_sinewave_ETMY.txt 1
1182627490.054266 H1:SUS-ETMY_L3_ESDOUTF_LR_EXC stop  zotws10 16548 1
 

J. Kissel, F. Clara

Talking to Fil after he got back from this week, he has indeed fixed the FAST IMON channels that were showing the ~4300 [ct] from one of the ADC legs of the given channels being shorted, but it looks like upon reseating the cable, the connector then shorted some of the NOISE MON legs. 

As the original documentation points out (see LLO aLOG 1857), this a problem with the soft material on the female side of the densely packed SCSI connector. If one male pin gets bent, it scars the female connector in a way that just makes more pins bent upon re-insertion, spreading the flaw to other channels.

Fil will open a work permit to 
(a) reseat and bolt down the I/O chassis backplanes, and
(b) mayhaps replace the connectors if need be

I'll update the FRS ticket.

In this afternoon, after a lockloss at NOISE_TUNINGS, we updated the offset (H1:PSL-ISS_SECONDLOOP_AC_COUPLING_OFFSET) again following Kissel's instruction. We set it to -0.12 which had been +0.05. However, this tuning wasn't good enough -- it still caused a power variation as the DC coupling was engaged by 10% or so. In the end, we set it to -0.095 as a result of a few iterative trials.

Note that when the offset was at +0.05, this caused a reduction in the power sent to the interferometer as the DC coupling was engaged. When it was -0.12, it increased the interferometer power instead.

2nd loop AC coupling offset H1:PSL-ISS_SECONDLOOP_AC_COUPLING_OFFSET is fine as of now.

No need to adjust. The fact that the diffraction stays at about 4% when the 2nd loop is ON and AC coupling is ON means that the offset is good. (That offset should not have any impact on the kick to the ISS when you turn off AC coupling.)

The problem seems to be the timing you turn on the third loop.

When the third loop is on while the 2nd loop is AC coupled, AC component of the AC coupling output becomes huge. We're talking about 10-something% RMS in the equivalent RIN (see the first attachment, top left, beginning of the plot).

In that plot, when AC coupling is turned off, instead of settling on the average of 6000, it settled on 6400. Suddenly the DC increased by a factor of 1.07, and this is seen by the second loop PDs (Ch15).

The output is held at the exact time when the AC coupling is turned off (except that it still slowly changes due to power scaling). Since the AC component is so huge, each time AC coupling is turned off, there's a big chance that the output is held at some number that is totally off from the average.

Smooth DC coupling transition requires that the AC coupling output settles on the value close enough to the average. This is only possible when the AC component is small enough to start with. Enable 3rd loop after ISS is DC coupled. If that's impossible we need a fancier solution.

If you look at the 2nd attachment, which looks at the same channels for 7 days, you can see that the AC coupling output never got huge until the 3rd loop was put in place in the guardian. It seems like we got 2 failures out of 3 trials.

Guardian changed, no 3rd loop for now.

With a telephone support from Sheila, ISC_LOCK.py line 3184 was commented out so that the third loop is NOT turned on during INCREASE_POWER.

We aren't turning on the 3rd loop in the guardian. If you think that CSOFT instability comes back, for now follow the instruction in alog 37046 and manually turn on the 3rd loop. You'll risk kicking yourself out, but if the noise is terrible you might choose to gamble.

I forgot earlier on the phone, we commented out the line in the guardian which requests the third loop, but this isn't enough to make sure it doesn't come on. We also have to turn off the input.  The reason is that in Noise Tunnig the request for IMC_LOCK is set to ISS_DC_COUPLED, which will require it to pass through ISS_TR_CLOSED, which turns on the third loop unless the input is switched off.  

Next attempt got me through ACCEPTING the aforementioned ISS diffs only to be left with a waiting for IMC_LOCK node notification (see screenshot). When I attempted to request the nominal state for this node everything came loose.

12:20UTC I'm leaving H1 in NLN but not Observing. I don't know how to get the IMC_LOCK node into it's nominal state without breaking the lock that I have.

Keita and I looked into one of these locklosses from last night, and for one of them the cause seems to be related to the ESD turning off (before the lockloss). Ed got the error message about the ESD in the next lock acquisition and reset the ESD.  

The last couple of problems that Ed had are related to a problem with running the guardian by hand line by line.  In the code there are places where ISC_LOCK makes a request from other guardians (in NOISE_TUNNINGS there is a line nodes[IMC_LOCK] = 'ISS_DC_COUPLED') this doesn't work in the command line unless you have initialized the node manager.  The easiest way to do this is to request the state from the IMC guardian using the medm screen. In this case, the ISS has to be DC couped before the IMC boost comes on later in the state, doing these things in the opposite order breaks the lock. 

We will continue investigating the problem in the NOISE_TUNNINGS that caused Travis and Ed to do it by hand in the first place. 

Correction: Guardian was stopped at SRC_ASC_HIGH_POWER so he could step through NOISE_TUNINGS.

The PcalX is still fully functional and hardware injection can be carried out if needed. The WS_PD channel through which the excitation channel is routed is a spare channel that is only used during the end-station PD calibration.

Summary:

We have established that most of these down converted lines (the one that hurts us)  reported in alog 36959 arise from the aliasing of the harmonics of the injected high frequency calibration lines. The lines that have the possibility to show up in the DARM are the ones that lie between 10 Hz to few hundred Hz. We can avoid these lines by selecting the high frequency calibration lines such that  the aliased lines are not in the given low frequency region.

Details:

We ran into issues with low frequency lines (in 100 Hz range) showing up in DARM when we were running high frequency calibration lines in the region of 5.5 - 6 kHz. This was first noticed in this LHO alog 36959 and LLO alog 34346. To understand this problem we wanted to find where these lines were being produced, photon calibrator itself or the data acquisition system. We disconnected the cable that inputs the excitation to the Pcal electronics and used a DB9 breakout box to acquire the excitation signal so that we were getting these signal before it enters the Pcal system. We rerouted this signal through the H1:CAL-PCALX_WS_PD channel which was one of the unused Pcal channel. 

The first plot shows the spectrum with a 5036 Hz excitation line (in blue) and without an excitation (in red). Some of the extra lines seen in the  blue spectra are the result of aliasing of the  higher order harmonics  of the injected as well as the imaged lines (imaging happens when going from 16khz to 64 kHz). In this particular case the 68 Hz line is the aliasing of the 13th order harmonic of 5036 Hz injected line. The second plot shows the predicted aliased lines (based on harmonics and upsampling) that would be produced for 5036 Hz injected line. Some of the predicted lines show up in the actual spectrum but not all and not all line that are present in the spectra are predicted. However, the lines that show up above few hundred Hz will be well below the DARM signal as the actuation transfer function goes as 1/f^2. Shivaraj, at LLO has taken some additional measurement to see if we can find the source of all the aliased lines. Report to follow.

Tagging DetChar and CDS on this, so that the CW group can follow up with them to understand how to flag the data for this known issue.

RickS, SudarshanK,

The breakout box used to acquire the analog excitation signal on X-end Pcal has been removed and the Pcal configuration is now back to normal. During the visit, Rick repeated some measurement that reproduced the 86 Hz line when a Pcal line was injected at 5950 Hz with an excitation amplitude of 25000 cts. This measurement was made at the DAC (analog) output using SR785. The 86 Hz line is about 70dB below the injected line. This still does not explain why we don't see 86 Hz line in analog output in the measurement (LLO alog 34495) that Shivaraj made using Agilent signal analyzer. :(

On Jeff's request, I am putting down the empirical formula that predicts the lowest frequency down-converted lines for each high frequency excitation.

DCF =  2^16 - EF * n, where DCF is the down converted frequency, EF is the excitation frequency and n is the harmonic number. For frequencies around 4-6 kHz, the harmonic number that produces the line in the bucket are between 11 and 13.

I don't have a physical intuition on why this happens. If we were looking at the resampled 16 kHz signals, the presence of these lines would not be surprising because these would be produced because of aliasing but we see these low frequency lines  on the DAC output, measured using the analog spectrum analyzer. Plots showing the same are attached.

Was associated with FRS Ticket 8335.
Now associated with IIET Ticket 10318.