There have been a lot of things happening today, most of which is other logs, but I thought it would be useful to make a summary:

• DRMI locking:
  • This morning Hang undid the filter change that he made to the MICH low pass a month or so ago.  Hang made the plot which is the first attachment and shows the lock acquisition times before and after he made his change. The vertical red lines show percentiles of the locking time.  
• Beam dumps:
  • After locking was recovered, Jenne and I added beam dumps for the beams transmitted by the AS AIR and POP AIR beam diverters 45010  More work is needed to dump the popair beam well.
• ASC cut offs:
  • I asked Hang to look at why CHARD Y noise was higher in our noise budget than it was in O2, he changed the low pass so that CHARD Y is now much lower in our noise budget 45023
  • Jenne noticed that SRC2 P had high coherence with DARM, and did noise injections. 45026  Like AHRD Y, this was a significant contribution the the DARM noise budget up to 30 Hz until Hang added roll offs 45032
  • Hang also reduced the gain of SRC2Y and added a cut off there.  This is only engaged in the LOW_NOISE_ASC state because it causes instabilities if used earlier. 
  • We took new noise budget injections for SRC2 P, Y, and CHARD Y after these changes were made. 
• LSC noise:
  • Stefan checked the input matrix element from POP9I to SRCL that is used to cancel the PRCL signal in POP45I.  45034  As we suspected, this coupling seems to be changing from lock to lock. 
  • Stefan also lowered the SRCL gain and put this in the guardian. 45035, then we redid the SRCL injection for the noise budget. 
  • We redid the SRCL and HAM3 noise injections for the noise budget after this work was done. 
  • Craig made a comparison of measurements Jenne and I had from O2 with ones he took recently of the MICH to DARM coupling with and without feedforward, 45029 the main message is that our coupling to DARM is the same but our feedforward performance has been worse. 
  • Gabriele posted an interesting model of how the SRCL FF can be unstable, which is why we need to AC couple it which limits how accurate our feedforward can be.  45030
  • We moved the notches for the beam splitter bound and roll modes into the LSC filter bank, rather than the suspension filter bank.  This was to avoid having them in the feedforward path which makes fitting the feed forward more difficult. We set up the DRMI guardian to acquire with the notches still in the suspensions, so that there high Q filters can be after the triggering rather than before, then after DRMI has acquired we swtich it to the LSC bank.  The real motivation for doing this is that I want to be conservative about changes to DRMI acquisition.  
  • Craig has measured and done fitting to retune the MICH feedforward, sending the feedforward to the DARM actuator rather than the ITMs.  The first round resulted in a similar level of suppression as the existing feedforward, so Craig is now doing a recursive fitting.    Craig was able to reduce the coherence using a recursive fitting 45038
• Georgia found possible down conversion of violin damping 45037

A noise budget including all of these improvements is attached, there is a trace here for HAM3 coupling as well as SRCL and MICH, but there is a risk that we are double counting now because the HAM3 noise couples to DARM through MICH (and SRCL). The last attachment is a comparison of the DARM spectrum tonight to Friday night in grey.  The improvements today have brought us to the early O2 reference below 25 Hz.  

Peter, Hang

One brute-force way of reducing the LSC MICH noise is to reduce its loop gain, and then we can move the LP filters to lower freqs. To achieve so we need to know the LSC MICH noise budget at low freq (which dominates the rms) and see if it is possible to reduce the input noise to the loop.

In the first plot we show the noise projection to MICH ctrl output based on f-domain MISO coherence analysis. The low-freq rms is largely dominated by the BS local damping noise (mostly from L, with a non-negligible contribution from P and Y). As a comparison, the largest ISI contribution (from BS ISI itself) is almost a factor of 10 below the BOSEM noise.

Since LSC MICH is already controlled with a clean, interferometric signal, it might not be necessary to have high gains on the local damping loops anymore. Thus we might want to reduce the local damping gain and thus reduce the amount of input noise to MICH. This should allow us to further reduce MICH noise to DARM by lowering MICH BW.

We measured the SRCL OLG in nominal now-noise and found it at 20Hz, with plenty of phase margin. We lowered it by 3dB ( H1:LSC-SRCL1_GAIN = -8.8 ), to 15Hz, which seemed to help DARM at 30Hz.

We also looked at MICH - it currently has a 6Hz UGF. Attached is a OLG measurement, but I am not sure I trust the phase at 6Hz. Craig and SHJeila are now working on MICH FF, which currently is worse than O2, and limiting DARM.

We fine-tuned the LSC SRCL input matrix again today by driving PRCL length at 32Hz and 135Hz, and minimizing the lines from the SRCL error signal: 

POP_A_RF9_I ---> SRCL was 0.038 and was set to 0.024

This reduced  the PRCL to SRCL coupling by a factor of 14 to 30 initially, cleaning up the SRCL error signal nicely. However, we noticed that in the next lock the coupling again came up some, to a net reduction of about a factor of 5.
We added this settings to lscparams.py.

I made a rough but representative model of why the SRCL feed-forward causes instabilities. The model works quite well in predicting an instability at about 1 Hz when the SRCL feed-forward is engaged and DC-coupled.

The model is based on:

• only MICH and SRCL longitudinal loops are modeled
• measurement of the optical response of the MICH and SRCL error signals, as measured in 44941. To keep things simple I treated the error signal response as frequency independent and I included only diagonal terms
• the suspension response is taken from the QUAD model plus G1100968 and from 44941
• I assumed the geometrical coupling from MICH to SRCL due to the fact that we are actuating MICH only on BS, and thus creating a significant SRCL motion
• the control filters are taken from the real time model
• the SRCL feed-forward is taken to be zero (no feed-forward), or as a constant (DC-coupled), or a constant times an high pass at 10 Hz (AC-coupled)

With the feed-forward off, the model reproduces well the open loop transfer functions. The key thing to consider here are however the closed-loop responses of the two coupled loops. The plot below shows three traces: in blue (hidden by the green trace) the case without feed-forward, which is known to be stable (as confirmed by the open loop gain); in orange the case with the SRCL feed-forward on, but without the high pass: this trace shows a "wiggle" around 1 Hz; in green the case with feed-forward on and high pass at 10 Hz (this trace hides the blue trace).

So the model explains quite well that adding the current 10Hz high pass remove any perturbation to the closed loop gain at 1 Hz. Morevoer, we can take the ratio [closed loop gain with DC-coupled feed-forward] / [closed loop gain without DC-coupled feed-forward] and fit it. The fit contains indeed an unstable pole at about 1.03 Hz. So the model explains how the interaction of the SRCL feed-forward with the SRCL and MICH loop creates an instability.

Finally, if we reduce the geometrical coupling of BS to SRCL to 10% of the value we get by actuating only on the BS (in other words if we send the MICH control to SRM or SR2 or SR3 and cancel 90% of the coupling), the model predicts that the 1 Hz instability is gone, see the plot below for a zoom of the closed loop transfer function around 1 Hz. Blue trace: stable, no SRCL FF; orange unstable, just with DC couple FF and MICH driving to BS only; green stable, with DC coupled SRCL FF and MICH driving to both BS and SRC, with a precision of 10%.

So the model gives us two ways (at least) to fix this instability, by reducing the MICH / SRCL cross coupling:

• send the MICH control to BS and to SRM / SR2 or SR3 (whichever is practical)
• send the SRCL feed-forward to the ETMs, since this should largely reduce the direct coupling of SRCLFF output to MICH, and thus cut the spurious loop in a different place

Sheila, Craig

I gathered a bunch of times of MICH excitations with feedforward on and off for O2 and O3 to see how well we're doing now as opposed to the past.

We seem to be doing an order of magnitude worse now than we did in O2.

Template exists at /ligo/home/craig.cahillane/Git/IFO/LSC/data/MICH_to_DARM_excitations_O2.xml.

MICH Excitation Times:
O2 MICH FF off: July 1 2016, 9:14:00 UTC
O2 MICH FF on:  June 14, 2017, 00:20:05 UTC
O3 MICH FF off: Nov 4, 2018, 10:29:37 UTC
O3 MICH FF on:  Nov 3, 2018, 22:47:00 UTC

• 15:50 In clean area/LVEA JeffB)
• 16:25 LVEA cleaning (Vanessa)
• 16:52 HAM6 check (Bubba, Jeff J)
• 17:00 Impromptu morning meeting
• 17:07 Schneider elec for Richard
• 18:24 TVo and Daniel to turn off OpLev on SR2
• 18:36 Sheila and Jenne out to ISCT1 and ISCT6
• 18:45 Karen to LVEA to clean
• 19:20 Vanessa to EX to clean
• 19:25 Nutsinee to ISCT6
• 19:28 Karen to EY to clean
• 19:35 Jenne and Sheila out of the LVEA
• 19:44 Nutsinee out of LVEA
• 19:44 TVo and to LVEA
• 20:06 Karen leaving EY
• 22:37 Nutsinee to ISCT6
• 00:17 TVo and Daniel to LVEA ITMY Hartman

OpLev trends for a week - NOTE SR3 optical lever was turned off today for a test, and is still off,  so the change in it's signal is understood.  See TVo for more info.

Laser Status:
Front End Power is 33.26W (should be around 30 W)
70W Output Power is 75.6W
Front End Watch is GREEN
70W Watch is GREEN

PMC:
It has been locked 0 days, 6 hr 29 minutes (should be daysweeks)
Reflected power = 14.87Watts
Transmitted power = 43.86Watts
PowerSum = 58.73Watts.

FSS:
It has been locked for 0 days 3 hr and 17 min (should be dys/weeks)
TPD[V] = 2.939V (min 0.9V)

ISS:
The diffracted power is around 2.2%
Last saturation event was 0 days 3 hours and 17 minutes ag (should be days/weeks)

Possible Issues:

SRC2 is pretty coherent with DARM (attached .png, bottom right corner pink and cyan).  I did some injections, so that Sheila can stick these into the noise budget (.xml attachments).  But, we should roll this QPD loop off faster.

TVo, Dan B., Danny V.

      In the ITMX HWS contour/gradient plots we notice a beam introduced onto the Hartmann camera while the IFO powers up (please see attached images). We performed a couple of tests to make sure it was not a stray Hartmann beam or oplev beam and narrowed it down to what we think is some weak IR light. Currently working towards a solution to this. 

There's a very sharp peak at 1.13Hz in HAM3 GS13 and CPS, and it seems to be shaking IMC and PRC pretty hard. What's this? This is not present in HAM2.

Top left/right shows that the signal is present in pretty much all DOFs except RZ in GS13 (left) and CPS (right).

Bottom left shows that it affects all LSC signals, but most strongly affects REFL_A_RF9_I (CM, red) and POP_A_RF9_I (mostly PRCL, magenta). POP_A_RF9_I RMS is totally dominated by 1.1Hz.

Bottom right shows the OSEM signals for MC3, PR2, PRM and PR3. PR3 doesn't show anything because it's in HAM2 and is not controlled by LSC. PR2 is not controlled by LSC but is in HAM3, thus shows a strong peak.

We modified the CHARD_YAW LP filter (instead of using FM3 + FM9 we used FM2). The new CH_Y ctrl output is shown in the figure below. The blue trace is now and red is from the data on Sat.

Here are links to Sheila's recent noise budget, and the Evan's noise budget from Sept 2015:

alog 44998: 2018-11-03 (png).
alog 21162: 2015-09-12 (pdf).

Remarks:
 - LSC auxiliary couplings: they are about a factor of 2 higher now than in 2015 at 30Hz.
 - Also, in 2015, we used rolled off the signals faster, but rolled them back up again at higher frequencies to gain back phase. So above 60Hz, LSC AUX was completely out of the picture.
 - ASC noise: at 20Hz the ASC noise is now 10x (!) higher in DARM now compared to 2015. Hang is working on that now.

Attached is the DARM spectrum, DARM coherence with some auxiliary channels, and the LSC auxiliary error point spectra for a quiet stretch on Saturday:

1225344730 (Nov 4 2018 05:31:52 UTC).

Note that - compared to Aug 21 2015, PRCL has an excess noise bump up to 60Hz, and an increased high frequency flat noise level (I hesitate to call it shot noise).
That bleeds slightly over to SRCL. Hoverer, the linear noise coupling, at least below 30Hz, is dominated by SRCL.

Gabriele might also post a bruco for this stretch.
Also note that this is from the same lock as Shaila's noise budget (alog 44998)

   Inspected the chiller filters and plumbing. The diode chiller is clean and clear. The yellowed crystal chiller filter is scheduled for replacement after the PSL plumbing upgrades Monday or Tuesday of November 12th vent. Closing FAMIS task #8314. 

• HAM1 HEPI watchdog was tripped.
  • screen link to watchdog screen doesn't work.  Opened another way, and untripped.
  • Due to the kernal panick, the guardian code was in error (some line wanted an int, but was getting NoneType because it couldn't connect to the channel).  Stopped, re-Exec, code runs fine, back to Robust_isolated.
• ITMY ISI sensor correction was not in correct state.  Not sure what the difference was, but the red light on the ISI CONFIG screen for ITMY was red, and re-requesting ISI_ITMY_ST2_SC to CONFIG_FIR made things happy.  Now the green Yarm is much more stable.
• Ran through initial alignment
• Locking Xarm IR: POP45 dark offset very wrong.  Tried locking with AS45Q, didn't catch.  Eventually went back to POP45 and just set the Xarm input offset (rather than unlocking the IMC), and Xarm locked pretty quickly.
• Reset the POP45 dark offsets.
• Bounce mode very high.  Using the DAMP_BOUNCE state which only actuates on EY damps it well.  Why is it getting rung up each lock though?
• Apparently ITMY sensor correction still not right - TJ fixing.
• SOFT pitch loops unstable when -20dB FM1s get turned off.  Leaving on for this lock (no guardian change for now).
• Lost lock with CHARD_BLEND - why?
• OM1 not getting cleared at lockloss?
• CSOFT P can have FM1 off, DSOFT P not. 
  • The blend filters were not engaged for DSOFT P.  Guardian copy-paste error.
• Sheila and I went to the ASAIR and POPAIR paths, to look at where the light goes when the beam diverters are closed.  Recall that the POP and AS beam diverters are 90/10 splitters, not high reflectors, so 10% of the light still goes to the in-air table.  With the beam diverters closed, the beam transmits through the glass, so gets deflected a little bit.  This causes the 10% light to hit a mirror mount on the AS path, and a uniblitz shutter on the POP path. 
  • We put in a razor beam dump on the ASAIR path between the first 1" steering mirror and the beam splitter that picks off beam for the camera.  This dump catches the beam when the diverter is closed, but lets the ASAIR beam go to the camera and OSA when the diverter is open.
  • We put in a razor beam dump on the POPAIR path just after the bottom periscope mirror, just in front of the uniblitz shutter that is in that path.  Because the deflection is rather small, with the beam dump so close to the bottom periscope mirror we can't quite catch all of the closed beam while letting the beam-diverter-open beam go through.  So, the part of the beam that had been hitting the uniblitz is now dumped, but a small amount is still going through the uniblitz's aperture and is hitting a mirror mount later in the path.  Since I can't recall us using those shutters at all, I think we should remove this uniblitz shutter on the POP path, which would allow us to move the beam dump back, hopefully allowing us to catch all of the 10% light when the POP beam diverter is closed.
    • At a few frequencies around 40Hz, 55Hz and 110Hz, we seem to be a little bit better than the new gray reference trace on the front wall.  Since it's tricky to access those beam dumps, and place them correctly, we don't want to do an on/off test.

J. Kissel,

The verbal alarm began yelling about the OPO DACs were saturating this morning for no, as of yet, apparent reason. We traced it down to the *actual* OPOS saturating (because multiple suspensions are in the same front-end model could have been the OPOS, ZM1 or ZM2). Since the only control going to the OPOS DAC is the damping loops, I turned them off, and the saturations quickly stopped.

There's no report of anyone from the SQZ team being in or around HAM6 at the time, and there's no admitted cause. (see first attachment)
An ASD of the OSEMs, both before and after -- and out to several kHz shows nothing abnormal other than the extremely rung-up fundamental resonances. (see second attachment)
Closing the L, T, and V loops resulted in an even slower positive feedback run-away. (see third attachment)
Damping loop settings and filters have not changes since prior stable operations.

No idea what's going on. Further investigation needed.

For now, I've left the damping loops OFF.

I attach the .xml in case the future debuggers need a nice calibrated ASD template. Naturally, the date in the file name is wrong. Please ignore.