Attached plot shows CDS overview after the crash (13:56 PDT). Things to note: h1susauxex has no problems (this is the only machine without a Dolphin card at EX), most user models stopped running except for h1susetmxpi, h1isietmx, IOP models are running but with DAC/DK errors.

At EX I noted:

No activity in the building around the time

No timing issues on any IO Chassis

Computers have been running for close to 8 days (since 16:00 Tue 18th Sept PDT)

All models not running reporting ADC timeouts in dmesg. Details below

I checked all front end computers could see their IO Chassis. I then restarted all the models, no reboots were necessary.

I've opened (and immediately closed) an FRS

https://services.ligo-la.caltech.edu/FRS/show_bug.cgi?id=11542

Maybe Jim will toss in a plot, but it sounds like this time may have captured some lockloss(es), so integrated in a time when we were kicking ETMX, which back-reacted against the ISI.  Seems like no further investigation needed here, unless it continues to show up.

Tried to check that the Xarm and Yarm WFS have similar sensing matrices.  All ASC loops on green arms open.  Step ETMX in PIT, see on both WFS A and WFS B.  Step TMSX in PIT, see on WFS A, very small on WFS B. 

Step ETMY in PIT, see on both WFS A and WFS B.  Step TMSY in PIT, see on WFS B, not at all on WFS A.  This is how things should be, with the current sensing matrix. 

For the Xarm, if WFS B is being used as the sensor for TMS, but the sensor doesn't respond to TMS motion, its not surprising that we're struggling to use the Xarm WFS.

We tried (as Hang writes) to make the Xarm analog gain and whitening settings the same as Yarm, but that did not help.  So, we left them as they were.  See screenshot for current settings for each arm.

Gabriele is measuring the sensing matrices for the green WFS more carefully.

Also, I plotted the control filters for both arms' ASC loops.  The shapes down near the UGFs (~0.1 Hz) are the same with some DC gain differences.  The lowpasses are very different from eachother though.  The Xarm filters have more aggressive lowpassing (6th order), but Hang had empirically found in the past that the Yarm couldn't handle that, so it has only 4th order lowpasses.  To make the phase okay near the control band, the cutoff frequencies for the lowpasses are different. 

We noticed that the low-pass filters (corner freq at 0.3 Hz; FM10) in the ALS_X/Y_WFS_A/B_I_PIT/YAW filter banks were turned on. However, the design should not have those filters on. This could explain in part the green WFS oscillation as an extra LP at 0.3 Hz would for sure degrade the phase margin at the designed UGF of ~ 0.1 Hz.

Now we turned all the LPs off.

In addition we noticed that the X arm green WFSs had two stages of whitening while the Y arm has no whitening. Nonetheless, this was consistent with the settings in the past when the green WFS worked properly. Thus for now we leave the whitening setting as it is.

Xarm FIBR trans is much noisier than Yarm. 

Attached is a time series showing a time from last night that Craig referenced (alog 44165) when Xarm starts clean, and gets glitchy.  Left panel is all things Xarm, right is all things Yarm.  Also attached is a spectrum.

In the Xarm time series, you can see that the FIBR_A_LF_OUT sees glitches at the same time as the ALS-C_TRX.  You can also see that the FIBR_CTRL is outputting glitches at the same time, in response.  The FIBR_A_LF is telling us about the phase difference between the light coming down the fiber from the PSL, and the end laser.  So, it's not telling us which of those things has a problem, but Daniel's hunch is that something is causing the fiber to vibrate or shake.  Note however that the FIBR_TRANS and FIBR_REJECTED signals are bad for X throughout the whole time series shown, not just the times that we're seeing glitches. 

The spectra show us that yes, the Xarm is more noisy than Y, but there isn't one particular characteristic frequency that we can blame.

Created corresponding ticket FRS Ticket 11535.

A few more notes / things I learned this morning:

ALS-X_FIBR_ERR_OUT_DQ is the error signal for the end station PLL.  When it hits its limit we see a glitch in FIBR_A_LF_OUT.  But, the ERR_OUT monitor channel has a factor of 10 on the board for the monitor, so the actual signal voltage used isn't that high.  But maybe it is still too much for something.

Increasing the gain on the FIBR_PLL board makes the glitches not as apparent in green trans.  But we still see them in ERR and A_LF.  Right now, we're using the boost in the FAST path of the Xarm FIBR_PLL common mode board to give us this extra gain.  Since we can suppress away the glitches by actuating on the laser, maybe that indicates that the problem is with the laser.  But, the glitches seem kind of similar to what we saw when we had the tag flicking in the AC breeze hitting the fibers in the MSR, which was an issue with the signal coming down the fiber. 

I am concerned that we are saturating something in the readout chain of the broadband PD.  According to the screen for ALS-X_FIBR_A_DC, we have 10x as much gain on this PD than on the Yarm.  I see this in that the FIBR_A_DC_TRANSIMPEDANCE channel on Xarm is 10x larger than the one on the Yarm.  Is this correct? 

For much of the day, since we were mostly able to stay locked and hand of the IFO to IR control, we ignored this.  Georgia and Craig are continuing investigations in parallel with locking. 

[Craig Keita Georgia]

As Craig mentioned yesterday the ALS X arm noisy periods also correspond to additional intensity noise from the Aux laser at the end station. I suspect this could be the laser diode or the noise eater.

Attached screenshot shows the laser power IR and green monitors (row 2), laser diode current monitors [Amps] (row 3), and the noise eater monitor (row 4), during the noisy period from last night. I'm not sure fiber noise could cause this, but rather the extra laser intensity noise could be the source of the problems.

When we get a chance I'd like to go to the end station and change the set point current by ~100 mA and see if that changes things; perhaps we're in a bad region for laser diode 1, where the extra noise is just visible.

We powered up again to 20 W after going through the CARM_TO_ANALOG state by hand.

We sat at 10 W for 30 mins and tried to run the A2L script, but suspect the gains were too low as we did not see the 20Hz lines in DARM (the ADS CLK_GAIN was 300, and we did turn the A2L inputs on at the drivealign matrix).

We sat at 16 W for ~30 mins and played around with the CO2 settings. Accidentally set CO2 X to 100 mW and CO2Y to 300 mW initially (instead of the other way around), this possibly made things a bit worse on RF18. Eventually set them both to 0W.

First attached screenshot is a grid of diagnostics (RF18, arm transmission, input power, CO2 power, HWS spherical power) for the first 20W power up Sheila talks about earlier this evening, second attachment is for our second power up. Note that RF18 at 10 W stabilised at a lower level with the second lock (suboptimal CO2 settings).

Changed ISC_LOCK guardian code to reflect the new early shuttering of ALS:


State           Old Number     New Number
PARK_ALS_VCO            451            306
SHUTTER_ALS             452            307
CARM_OFFSET_REDUCTION   306            308
CARM_5_PICOMETERS       307            309

From the simulation numbers on the TCS page, we expect about 17.3 udiopters in lensing for ITMY as seen on the Hartmann sensors= when powering up from 2 watts to 10 watts which starts to get there before turning down the CO2 lasers so maybe the estimate of the absorption isn't that far off.  What is interesting is that the Hartmann spherical power eventually reaches close to the original lens that was seen before any power up happened  but at this point, the CO2 lasers are off.  With the decay of pop18 as a function of power up, we might want to load a bit more power onto the ring heaters in order to reach 25 Watts without losing too much PRC gain.

During the power up, the ITMX Hartmann sensor got a very large kick which we're trying to hunt down now so the data is very hard to interpret.

Some may recall an issue we had back in July and August when we were testing the new green arms, where the out-of-loop X-arm frequency noise measurements we were making made little sense:

In-loop COMM Frequency Noise
IR Frequency Noise measurement using COMM as out-of-loop sensor
COMM Frequency Noise measurement using IR as out-of-loop sensor

Our out-of-loop witnesses of X-arm frequency noise disagreed by about a factor of 10: COMM frequency noise was ~2 green Hz RMS, but IR frequency noise was 12.4 red Hz RMS (~25 green Hz RMS).  They also did not show the same spectral features, leading us to believe that our out-of-loop IR measurements were not limited by X-arm noise.

After Stefan reduced the COMM and DIFF PLL gain from 26 to 6 dB, the in-loop COMM frequency noise plummeted, and we start to see some of the same spectral features as seen in alog 43214.  In-loop COMM now reports 6.7 green Hz RMS noise, which is still slightly higher than the ~2 Hz reported by our out-of-loop IR witness in August.  Could be worth it to try an reduce the gain further, then redo this measurement.

Also plotted is the ETMX coil master monitor now as opposed to this morning.  ETMX actuation is greatly reduced.

Questions remaining: It doesn't make sense that the in-loop sensor would see higher noise than the out-of-loop sensor, particularly when we have proof that the COMM PLL was actuating hard on ETMX.  Could be that COMM and DIFF PLL CTRL OUT are calibrated incorrectly, further investigation required.

Here's a plot of ALS COMM and ALS DIFF control signals (H1:ALS-C_COMM_PLL_CTRL_OUT_DQ and H1:ALS-C_DIFF_PLL_CTRL_OUT_DQ that come "pre-calibrated" into um) when they have the arm cavities under control, holding with a 200 Hz and 1.4 kHz detuning offsets in place, respectively, later in the day during IFO lock acquisition. 

Assuming a conversion from frequency to length using the green wavelength, that's an offset of 
   COMM
   200 Hz * L_arm* lambda_g / c =
   200 Hz * 4000 * 532e-9 / 3e8 = 1.42e-9 m = 1.4 nm.

   DIFF
   1481 ct * (0.11802 ct / Hz)^-1 = 12549 Hz
   12549 Hz * L_arm* lambda_g / c =
   12549 Hz * 4000 * 532e-9 / 3e8 =  8.9e-8 m = 90 nm.

The COMM offset comes from H1:ALS-C_COMM_VCO_CONTROLS_SETFREQUENCYOFFSET on the COMM VCO (pre-calibrated into Hz), and the DIFF offset comes from the H1:ALS-C_DIFF_PLL_CTRL_OFFSET, which is uncalibrated ADC counts.

EDIT as a result of Daniel's comment (LHO aLOG 44163) below
P.S. now that I've found Kiwamu's LHO aLOG 20629, and compared it against the calibration filters installed in the PLL_CTRL filter bank I'm skeptical that the filters are correct.
    - Kiwamu measures 0.11802 ct / Hz.
    - The filters have gains of "cnts2V" = 3.05176e-4 V/ct and "V2Hz" = 1.90146e4 Hz/V whose product is 5.8028 ct / Hz Hz / ct. Which makes the inverse 0.1723 ct / Hz.
Thus the "pre-calibrated" ASD of the performance as reported by the DIFF PLL CTRL may in fact be too high by a factor of 6.
Thus the percent difference is 
    100*abs(0.11802 - 0.1723)/0.11802 = 45.9%
So, to correct the calibration of the H1:ALS-C_DIFF_PLL_CTRL_OUT_DQ channel, one must multiply by
    0.11802 ct / Hz
    ---------------- = 0.68497 [Hz / "Hz"] (or [m / "m"] assuming the conversion from frequency to displacement is correct).
    0.1723 ct / "Hz"
The calibration is off by a "about a factor of 2" (not the factor of 6 as mentioned in the striken comment).

One area of suspicion that would resolve the remaining discrepancy: "cnts2V" is 3.05176e-4 V/ct which is equivalent to 20 V / 2^16 ct. 
The LIGO-standard differential input 16 bit ADCs have a calibration of 40 Vpp / 2^16 ct = 6.1035e-4 V/ct, as demonstrated in T1100538.
If "cnts2V" were changed to match that value, the product would be 6.1035e-4 V/ct * 1.90146e4 Hz/V = 11.606 Hz / ct, and the inverse = 0.086166 ct / Hz. 
That reduces the percent difference to 100*abs(0.11802 - 0.086166)/ 0.11802 = 26.9% and reduces the correction factor to 0.11802 / 0.086166 = 1.3697.

Isn't it the inverse?
"cnts2V" = 3.05176e-4 and "V2Hz" = 1.90146e4 whose product is 5.8028 Hz / ct => 0.17 ct/Hz.

A tuning of ~20kHz/V is about what we expect the VCO w/ FDD calibration to be, when we measure at the single-ended output of the CM board. However, the DAQ readback implements a differential driver that adds a gain of 2, therefore the 3.05176e-4 V/ct is relative to the CM board output. Does the VCO filter has a gain of 1 at DC?

Thanks Hugh! Would you like these new sensors to be added to the DetChar summary pages? Unfortunately the LHO cluster is down at the moment, so I can't find an example page for you, but if you would like the sensors to be added please let me know.