I didn't quite fully anticipate all of the affects of separating DOWN from the rest of the graph.  In particular, one really bad unanticipated effect was that after lockloss, when the ISC_LOCK jumps to the LOCKLOSS state, it doesn't find any paths from LOCKLOSS to the last requested state, which causes it to just stall out in LOCKLOSS, and not proceed to DOWN.  In other words, DOWN was not run after the lockloss this morning after last night's 10 hour lock.

When I came in this morning I therefore found a bit of a poo show that I then had to clean up.  None of the control signals had been shut off, multiple SUS and SEI systems were tripped, and bouce roll modes were rung up.  Evan and I eventually wrangled everything back under control, and we're now back to locking.

I have reconnected DOWN to the rest of the graph.  NOTE, however, that this problem is not inherent in the fact that DOWN was disconnected.  It's just that once you do something like that you remove the ability of guardian to find the right path for you, so you have to be careful to make sure you have all the appropriate jumps to get you where you need to be.  I'll rethink things.

Some notable issues:

• HAM3 had tripped, which somehow send IMC_LOCK into "christmas mode", where it was continually jumping between DOWN and MOVE_TO_OFFLINE.  This was due to some questionable logic in the IMC_LOCK guardian that sends it to MOVE_TO_OFFLINE when HAM2 or HAM3 SEI are not in their nominal states.  It's doing this even from the DOWN state, which makes no sense and was causing this problem.  I'll try to clean up the logic here.
• The "slow" LSC IMC feedback (IMC_MCL) was not shut off, even after we had recovered ISC_LOCK and IMC_LOCK into their respective DOWN states. This was causing signals to continually be sent to MC2, even though it was misaligned, which was causing HAM3 to continual trip during isolation, which therefore led to "christmas mode" above.

Lesson's learned:

• We need to be very careful when working on things that might affect the prompt running of the ISC_LOCK DOWN state.   DOWN needs to be run promptly after lockloss, to prevent ringing up the bounce/roll modes that take a while to damp out.  Both yesterday and today we had different issues that resulted in DOWN not being run after lockloss, both of which resulted in extra hours of recovery.
• We should consider adding triggering to the front end to shut off all the control signals after a lockloss, and not rely on something external to do it.  This is a natural extension of the loop engage triggering, but we hadn't quite gotten there yet.
• Seeminly inocuous guardian changes can sometime have unanticipated far reaching affects.  We should be particularly careful when adjust the graph logic, as I did yesterday.

It seems like the rate of epics freezes has increased today, I have seen more than 5 in the last 2 hours. 

Sheila, Evan, Jeff B, Corey

Both yesterday and this morning, we had extremly rung up bounce and roll modes (both times because the IFO lost lock and DOWN was not run, yesterday for the reasons explained in comments to alog 20103, today because of a different snafu).

When this happens, we need to damp bounce on ETMY while locked on ALS.  To do this, it seems that we need to use a phase that is +150 degrees compared to the phase we use in full lock.  This phase shift comes from the difference between the DARM loop here and in full lock.  When locked on RF DARM, we need to use +120 degrees compared to the normal settings.

We also had difficulty yesterday with rung up roll modes.  To damp roll we use AS WFS, so we need to get to RF DARM before we try to damp them this way. One difficulty we had was that the roll mode notches in the PUMs were not wide enough (Evan adds that the notch needs to be wide because of the Shapiro effect), so that DHARD could saturate because of the roll mode.  

Bringing these things down when they are verry rung up is very slow, because the actuation authority is small compared to the amount of energy in the mode.  Fortunately, we are normally in the regime where the mode is small and it only takes a few minutes to damp them. 

Evan, Lisa

This entry is to clarify the fact that the impact of  this excess of high frequency noise  is actually bigger than the coherence with the ASC channels suggests, as it can clearly be seen by comparing OMC NULL and SUM.

For example, around 2 kHz, the discrepancy in the noise floor between OMC SUM (total noise) and OMC NULL (shot + dark noise) is about 15%, so corresponding to a noise which is 0.6 times shot + dark.

The attachment shows OMC SUM/NULL in H1 at low noise (left) compared to L1 (right). 

So, the message is that we are looking for something quite big here..

After 17:30 UTC the interferometer was not undisturbed: I was making PEM injections.

 Measured temps of heated areas of RGA -> 95C < temps < 120C -> Made slight changes to variac settings -> Aux. cart @ 2.5 x 10-5 torr (seems high for this configuration)

Its been a week since the DAQ reconfiguration which reduced the NFS/QFS disk loading and both framewriters continue to be 100% stable. The attached plot shows the restarts of h1fw0 (red circles), h1fw1 (green circles) and the DAQ system as a whole (blue squares) for the month of July. The Magenta lines show when h1fw0 and h1fw1 were modified. In the past 7 days, the only restarts of the framewriters are associated with complete DAQ restarts.

Matt, Lisa, Evan

Tonight we looked at the coherences between the OMC DCPD channels and ASC AS C, this time at several different interferometer powers. In the attached plots, green is at 11 W, violet is at 17 W, and apricot is at 24 W.

Evidently, the appearance of excess high-frequency noise in OMC DCPD sum (and the coherence of OMC DCPD sum with ASC AS C) grows as the power is increased. We believe that this behavior rules out the possibility that this is excess noise is caused by RIN in the AS port carrier, assuming that any such RIN is independent of the DARM offset and of the PSL power. Since the DARM offset is adjusted during power-up to maintain a constant dc current on the DCPDs, RIN in the AS carrier should result in an optical power fluctuation whose ASD (in W/rtHz) does not vary during the power-up. This is the behavior that we see in the null stream, where the constant DCPD dc currents ensure that the shot-noise-induced power fluctuation is independent of the PSL power.

I modified the ISC_LOCK guardian to revert the DOWN state back to being a 'goto'.  This allows you to select the state directly, without having to go to MANUAL.

The reason it had been removed as a 'goto' was because occaissionally someone would accidentally request a lower state while the IFO is locked, which would cause the IFO to go back through DOWN to get to the errantly requested state.  To avoid this I implemented some graph shenanigans:  I disconnected DOWN from the rest of the graph, but told it to jump to a new READY state at the bottom of the main connected part of the graph once it's done:

This allows DOWN to be a goto, so it's always directly requestable, but prevents guardian from seeing a path through it to the rest of the graph.  Once DOWN is done, though, it jumps into the main part of the graph at which point guardian will pick up with the last request and move on up as expected.

J. Kissel
WP 5395
ECR E1500230
II 1054

I've removed all redundant IPC Error EPICs channels from the top-lvel models of all SUS this evening. This is in accordance with ECR E1500230. The models compile, and have been committed to the SVN. They will be installed this coming Tuesday.

Once installed, this closes out the ECR and Integration Issue for LHO.

Jeff, Sheila

We have had three locklosses in the last 2 days that were caused by the ETMX UIM coil driver rocker switch tripping.  The only solution is to drive to the end station, and flip the rocker switch back on.  Something is wrong with this coil driver (Jeff thinks it should just be replaced).

The only way to notice this is looking at the OSEM centering medm screen.  It would be great to add it to SYS DIAG (its not caught by the current noisemon check), and the ops overview screen, and the quad overview in a more obvious way.

attached is a file listing the channel differences between the L1 and H1 science frames.

Something happened at lock loss and the IFO did not reach the defined DOWN state.

Symptoms:

• SUS: ITMX, ITMY, ETMY - driven and rung up, watchdogs tripped, I had to adjust the RMS limit on ETMY from 8000 to 10000 to reset watchdog
• SEI: ITMY ISI tripped - Jim rest
• IO: MC WFS left pushing optics to a bad place and needed to be cleared by hand
• PSL: ISS and FSS in oscillation - outer loop on while IMC was unlock, throwing inner loop into oscillation
• GRD: ISC_LOCK - red for a while, cleared after INIT

Dave, Jaime, Sheila, operators, and others are investigating.

Summary

To decrease uncertainty in calculation of actuation function correction factor, kappa_A, sensing function correction factor, kappa_C, and CC pole frequency, f_c, we've recently increased calibration line amplitudes to give SNR of 100 with 10s FFT (see LHO alog #19792). Earlier Kiwamu posted his investigation of CC pole frequency over the last weekend in LHO alog comment #19988. In this alog we show kappa_A, kappa_C and f_c calculated according to the method described in T1500377-v3 for the same time interval (2015-07-25 00:00 UTC to 2015-07-27 UTC, when GRD-ISC_LOCK_STATE_N >= 501, 1 min FFTs).

Statistical uncertainties of kappa_A, kappa_C and f_c within 1.5 hours time interval highlighted with green are:

Xctrl(34.7) and PcalX(33.1), std(kappa_A) = +/- 0.45 % (1 sigma)
PcalX(325.1), std(kappa_C) = +/- 1.12 % (1 sigma); std(f_c) = +/- 5.20 Hz (1 sigma)
PcalY(331.9), std(kappa_C) = +/- 1.43 % (1 sigma); std(f_c) = +/- 5.55 Hz (1 sigma)
PcalX(534.7), std(kappa_C) = +/- 0.70 % (1 sigma); std(f_c) = +/- 2.08 Hz (1 sigma)
PcalY(540.7), std(kappa_C) = +/- 0.78 % (1 sigma); std(f_c) = +/- 2.68 Hz (1 sigma)

Notice that kappa_C and f_c on the left subplots were calculated from low SNR 325.1 Hz and 331.9 Hz Pcal lines set by Evan (see LHO alog comment #19823). Calculation of these parameters using higher SNR 534.7 Hz and 540.7 Hz Pcal lines (right subplots) gave less noisy results.

Details

C_0, D_0 and A_0 were taken from LHO ER7 DARM model.

To make kappa_C calcluations consistent between results from 4 Pcal lines, a manual correction to phases of Pcal lines that correspond to 130us of time advance was applied. On the plot we report only changes in f_c by subtracting mean value of about 300 Hz. In order to receive an absolute value of f_c using this method, we must take into account exact time delay/advance of PCAL RXPD channel w.r.t. DARM_ERR; possibly a frequency independent phase shift (however we do now know any reason for that); and the DARM model TFs at the reference time, C_0, D_0 and A_0.

Plots of 1 min FFT dewhitened calibration line amplitudes and phases are given below.

Calibration line uncertainties in DARM_ERR readout in a 1.5 hours interval (highlighted with green color) are as follows:

Xctrl( 34.7) = 2.2000e-01 (+/- 0.00 %); Derr( 34.7) = 2.9738e-10 (+/- 0.15 %)
PcalX( 33.1) = 2.4587e-02 (+/- 0.00 %); Derr( 33.1) = 3.0817e-10 (+/- 0.26 %)
PcalX(325.1) = 1.0724e-01 (+/- 0.00 %); Derr(325.1) = 2.0593e-10 (+/- 1.48 %)
PcalY(331.9) = 9.3791e-02 (+/- 0.01 %); Derr(331.9) = 2.0150e-10 (+/- 1.55 %)
PcalX(534.7) = 7.1100e-01 (+/- 0.00 %); Derr(534.7) = 5.8223e-10 (+/- 0.52 %)
PcalY(540.7) = 6.3845e-01 (+/- 0.01 %); Derr(540.7) = 5.8948e-10 (+/- 0.45 %)

P.S.

After today's calibration telecon we've changed calibration lines that will be used for estimation of kappa_A, kappa_C and f_c to (see LHO alog #20063):

• PCALY 36.7 Hz, SNR ~100;
• DARM_CTRL 37.3 Hz, SNR ~90;
• PCALY 331.9 Hz, SNR~100;
• PCALY 1083.7 Hz, SNR~40;

We are also planning to add an ESD line close to low frequency PCALY line and another high frequency low SNR PCALX line at 3001.3 Hz after completing power budget investigations of PCALX module.