Sheila, Carl, Stefen

With the ISS 2nd loop patch (see alog 28076) we successfully reached the "lowish" nosie state again. The spectrum loked the same as in alog 27990.
.

Start: Jun 30 2016 05:01:36 UTC
Loss:Jun 30 2016 05:05:22 UTC

The loss was presumably cause by a 18.04kHz mode ringing up rapidly. Carl will add more details on that.

J. Kissel, E. Goetz, K. Izumi, S. Ballmer, P. Thomas, T. Shaffer, D. Tuyenbayev, C. Cahillane, P. King

We had a pretty rough day today as far as robust interferometry. Lots of problems resulted in no low-noise locks today (well, none that lasted more than a few minutes. Also "low noise" still only ~40 Mpc). Some problems were related to freshly commissioned things, some due to infrequently seen not-yet-fixed problems, some due to maintenance day recovery, and finally some due to honestly challenging things that are just not yet well-commissioned. The message: we were not able to perform any of the measurements needed for Calibration Day (i.e. measurements of the IFO's optical plant and of the ETMY Isolation Stage Actuation Strengths).

We'll discuss tomorrow if and when to reschedule. 

Details:
----------------

Why do you care? It's only ER9...
Most importantly, it doesn't matter if the run is 3 days or 3 months, we need to measure the interferometer so that we can *quantify* how well or how poor the calibration uncertainty is. Otherwise, when told "It's just ER9 ... you don't have to do better that ___% and ___ [deg]" (where you can pick your favorite large numbers for "___") we do not have any way to claim whether we are meeting that accuracy other than gut feel. Yes, we could go through the laundry list of things that have changed, try to model them, and make some sort of "how has the response function changed" model and infate the uncertainty accordingly, but that takes just as much, if not more, effort than measuring the IFO as is because those changes have not been quantitatively well-tracked over these past few months (which is totally understandable for a commissioning period).

There's been very little change between O1 and ER9, right? What's the big deal?
Here's a list of things that have changed between O1 and ER9 that all affect the most sensitive region of the interferometer at the several percent level:
- ETMY ESD Actuation Strength has changed due to charge accumulation / evolution. This hasn't been quantified in Longitudinal since the PostO1 calibration period, and we've since reduce the charge (as measured in angle) by manipulating the bias voltage sign. We need to confirm its current value. Especially since we've either not had calibration lines on for the past few months or we've not had a sensitive enough interferometer to resolve them.
- We want to identify the PUM / UIM Actuation Strength better. The Calibration Lines during O1 revealed a 2% systematic error on either the PUM or UIM. Because we only ran with a limited number of lines, we don't know to which stage we should associate this error. 
- Given all of the new alignment work over the past few months, and the recent woes on recycling gain, we want to get a new reference for the DARM coupled cavity pole frequency to make sure that 341 Hz is still a good reference.
- Now that we're running at 40 W, and it has recently been shown that signal recycling cavity detuning is affecting us as low as 10-20 W, we need to pay close attention to this, as the detuning might be high enough to start affecting the DARM actuation and sensing crossover at ~45 Hz.
- We've updated all of the coil driver and OMC sensing chain compensation to match the analog electronics "perfectly." All of these upgrade filters have a frequency dependent improvement on the few percent level. We need to make sure that all of those filters are indeed making the measurements and model comparison better.
- The new digital AA and AI (or IOP downsampling) filter has phase significantly different from the O1 filter, even down as low as a few hundred Hz. 

Other things that have changed that will affect high and low frequencies:
- A New OMC DCPD split whitening chassis has been installed to support PI tracking. Though the Gravitational Wave DCPD path is *nominally* the same, i.e. it has the same circuit design, its components are real and different, so they need to be remeasured. These will likely be differences in the high-frequency poles, so there affect will be likely be small and above ~1 kHz.
- Evan and Rana have changed the local QUAD damping loops. So we need to create a new SUS dynamical model and update everywhere its needed. This should primarily affect frequencies and dynamics below 10 Hz.

Why can't you get what you need next Tuesday, the day before the run?
*If* we get all the measurements we need, we need time to analyze and process those measurements, update all of the appropriate calibration filters (both in the front-end and in the GDS pipeline), make sure the calibrated pipeline agrees with our measurements and is producing sane results, run for a day or two's worth of lock stretches and confirm that it all works . Again, if we want to be quantitatively sure of our calibration, one really has to take all of the appropriate steps to well-calibrated data, regardless of the length of the run. Further, we've made drastic software changes to the GDS calibration pipeline, in order to make incremental progress toward O2. These need to be tested, and they'll likely only be ready by the end of the week.

Today's Lock Loss Break Down; Why we had so much trouble
There're 5 major lock stretches shown in the attached inspiral range time series. The lock losses and in between problems are described below:
Lock 1: Patrick and I'd gotten all the way to a 40 W (nice work commissioning team!), and was in the middle of switching coil drivers to their low-noise configuration (ISC_LOCK state "COIL DRIVERS") but got hit by some Izumi-Ballmer dP/dtheta instabilities. I found out later that this was because the ASC SOFT degrees of freedom had had their offsets commissioned last night, but they were left out of the Guardian because they weren't yet confident in their robustness. Apparently, with out them, the IFO is aligned to a place with reasonably large dP/dtheta, so I lost lock.

Between locks 1 and 2:
- We were having the same problems with the PSL FSS vs.ISS that Shiela found last night. We spent some time with Peter King trying to diagnose and debug.
- The OMC SUS had tripped (which is apparently a reasonably common occurance after lock-loss because of the suddenly bogus OMC input alignment signals)
- We'd lost lock in the middle SRM's bottom-stage matrix dance that facilitates coil driver state switching. However, the DOWN states did not restore these matrix values. Stefan spent some time fixing this. See LHO aLOG 28053.

Mini-lock 2: Intentionally broke the lock because PSL FSS / ISS / Noiseeater was oscillating, and because we were past DARM ASC signals being turned on we were blasting the PUM stage of test masses with junk, and we were worried about violin mode ring-ups and PUM stage coil driver RMS watchdog trips.

Big Lock 2: Lost lock after 40W power up because of ISS 2nd Loop Offset transient.

Between locks 2 and 3: 
- Continuing to have problems with PSL servos, Kiwmau and Peter found that settings had not been restored properly after maintenance day. These issues were completely resolved after we reverted some PMC locking servo gains using the correct SAFE.snap SDF file. See LHO aLOG 28058.

Lock 3: Unknown lock loss. These things happen! ¯\_(ツ)_/¯

Between locks 3 and 4
- OMC LSC Input Matrix ramping failed to scale the DARM gain correctly. Solution found in LHO aLOG 28074.
- Violin Modes on ITMY and ETMX rung up for unknown reasons, some PUM stage RMS watchdog trips, took a while to damp back down with some manual gain reduction and gradual re-increase.
- Needed to come up with no stages of OMC whitening to prevent saturations

Lock 4: Lost lock after 40W power up due to ISS 2nd Loop Offset transient.

Between locks 4 and 5
- TJ takes over, runs initial alignment.

Lock 5: Lost lock after 40W power up due to ISS 2nd Loop Offset transient.

As of this entry we're on Lock #6. And if we lose lock due to 2nd ISS offset problems again, Stefan's going to spend some time commissioning it. 

Note, that the 2nd loop closure problems (recall that the 3rd loop is ON during power up, and then once complete, we have to transition back to turning on the 2nd loop) have been newly difficult because there is just more intensity noise coming out of the IMC than there was during O1. Both the DC alignment into the mode cleaner has changed for the worse, increasing the natural jitter-to-intensity conversion of the IMC, and also the HPO just produces more intensity/jitter noise now that it's in use. Check out LHO aLOG 28076 for Stefan's report on today's attempts to improve it.

Jeff, Kiwamu, Stefan

We had trouble engaging the ISS 2nd loop at 40W. The core issue is that the digitized signal (H1:PSL-ISS_SECONDLOOP_SIGNAL_OUTPUT) swings rail to rail without the loop, making it difficult to properly pre-set the offset adjuster (H1:PSL-ISS_SECONDLOOP_REF_SIGNAL_ANA).
As a result, when engaged, the ISS 1st loop tended to swing into saturation.

I patched the Guardian as follows:
- Instead of trying to rely on a calculation for the offset adjuster (H1:PSL-ISS_SECONDLOOP_REF_SIGNAL_ANA), I hard-coded it to 0.987 V, which is where it likes to sit at 40W. (THis is obviously not a long-term solution.)
- As soon as the servo comes on, I ramp H1:PSL-ISS_SECONDLOOP_REF_SIGNAL_SERVO_GAIN to zero, clear the history, and re-engage it on the diffreation power.

This seemed to engage the 2nd loop without excessive transient - back to full locking.

PS: I might have already mentioned this: but the ISS SHOULD BE AC COUPLED!

Code:

In PREPARE_ISS:

        # read current servo signal value. If it is too far from the operating point, correct the offset.
        #second_loop_out = cdu.avg(1, 'PSL-ISS_SECONDLOOP_SIGNAL_OUT16')
        #if abs(second_loop_out) > 1.0:
        #    # measure current input PD value
        #    # Sets the ref. signal value to bring it to close to the operating point
        #    pd_avg = cdu.avg(1, 'PSL-ISS_SECONDLOOP_PD_14_SUM_INMON')
        #    ezca['PSL-ISS_SECONDLOOP_REF_SIGNAL_ANA'] = round(0.5*((pd_avg*2.29*.0006103515625)+0.101), 6)
        #    time.sleep(5)

        # above section commented out - instead command the value that worked  (Stefan Ballmer 20160629 for 40W)
        ezca['PSL-ISS_SECONDLOOP_REF_SIGNAL_ANA']=-0.9826934814453125
 

In CLOSE_ISS, once the loop is closed:

            # the loop is successfully locked, ramp off the loop (Stefan Ballmer 20160629 for 40W)
            ezca['PSL-ISS_SECONDLOOP_REF_SIGNAL_SERVO_GAIN'] = 0

and a little later:

            # clear and ramp on the loop (Stefan Ballmer 20160629 for 40W)
            ezca['PSL-ISS_SECONDLOOP_REF_SIGNAL_SERVO_RSET'] = 2
            time.sleep(0.1)
            ezca['PSL-ISS_SECONDLOOP_REF_SIGNAL_SERVO_GAIN'] = 1
 

After talking with Rana and Evan during the NSF review, I made some independent estimates of the thermal noise that
could come from the composite SRM which had a measured resonance frequency of 3.34KHz with a Q of 150. I get the same
thermal noise displacement spectra of the mirror surface as Evan in log 27488,27490,and 27490. Tend to favor the viscous case
as Peek is a thermoplastic as is Nylon which was measured to have a viscous spectrum. The model is a shear motion of Peek 8-32 screws 
clamping the 2 inch diameter 1 inch thick mirror into the metal carrier. The motion is a displacement of the mirror allowed by shearing
the 8-32 bolts in and out of the plane of the carrier - longitudinal modulation of the light beam. Thermo elastic damping is much too small
to explain the low Q. The time constant for thermal diffusion in the Peek screws is about 10 seconds. The thermo elastic delta is about 10^-2,
so that loss tangent phi varies as 1/f with a value of 5x 10^-8 at 3.4kHz and about 2 x 10-6 at 100Hz. My guess unfortunately is the
SRM is not responsible for the excess low frequency noise we are seeing. I think Evan and Rana had come to the same conclusion. 

The OMC would not lock as we entered DC_READOUT_TRANSITION, and the DARM spectrum had risen around 100Hz. Sheila tracked it down to H1:LSC-ARM_INPUT_MTRX_RAMPING_1_3 having a gain of 20 and not 8. She investigated further and realized that the matrix was not being loaded beore the actual value was in place, something that a sleep of 0.1s can cure. With her help. I added in these short sleeps or just loaded later.

J. Kissel

Several lock losses were caused by DC coupled ISS loops today, so in my desperation to make things better, I modified the PSL ISS 2ND Loop MEDM screen to include all the relevant PSL AOM Diffracted Power information on the same screen (formerly only available on the 1st loop ISS screen). Also, to facilitate the turn on dance, I've decreased the default increment on the slider used to tune the analog reference signal (H1:PSL-ISS_SECONDLOOP_REF_SIGNAL_ANA) and increased the precision which the reference signal value is displayed.

The updated screen has been committed to the user apps repo here:
/opt/rtcds/userapps/release/psl/common/medm/ISS/PSL_CUST_SECONDLOOP.adl

Sheila, Haocun

We designed some new filters for the low bandwith CHARD Yaw at 40W.

We plan to lower the gain for about 6dB, which will give us a UGF at 3.5Hz and use the new filters 6 ('40W') and 7 ('ctrlHBW') to stablize the loop (blue curve in the 2nd Figure). The new cutoff filter for 40W is filter 9 ('cutoff40W') (Figure 1). The pink curve in the second curve is what we expect from the simulation.

yesterday fw0 was more stable than fw1. In the evening the new QFS-NFS server was serving fw0's files to both nds0 and nds1. I also stopped fw1 from writing science frames to see if this would improve its stability. Overnight fw0 restarted several times, meaning that fw1 needs to write its science frames again (it had not become more stable). Again we are seeing both framewriters sometimes restart around the same time, a synchronisation we had not previous seen.

The configuration for fw1 was changed to write science frames again and I waited until 4pm for it to restart itself, but fw1 went into a period of stability. At 4pm I restarted fw1 and it became very unstable. After power cycling both h1fw1 and h1ldasgw1 it is now stable.

During the day Stefan needed nds1 to link to fw1's frames to see data for a time fw0 was not running, nds1 was reconfigured and restarted.

Support: Jeff K., Jenne, Kiwamu, Peter K., Sheila

At the beginning off the shift I changed the power_sepoint in lsc_params.py from 20 to 40 and loaded it.

After a lock loss from 40 W we started having FSS and ISS issues similar to what TJ (alog 28034) and Sheila (alog 28045) had reported last night. Kiwamu was able to find and address the problem (alog 28058).

A couple of times the guardian did not appear to recognize a lock loss and thus did not take the IFO to down. I had to do so manually.

There were occasional notifications that the POP X PZT had railed and that the IMC WFS were not centered. There were also notifications that the TCSX and TCSY CO2 power was off by more that 0.05 W.

Stefan fixed a bug in the DRMI down state (alog 28053).

I ended the shift having trouble locking DRMI and handed it over to TJ to run an initial alignment.

14:58 UTC Peter to LVEA PSL racks to reset noise eater
16:08 UTC Bubba and Nicole to mid X
16:57 UTC Karen and Christina to mid X and mid Y to clean
17:02 UTC Ed to CER to reset DBB AA/AI chassis (alog 28049)
17:52 UTC Karen leaving mid Y
18:07 UTC Christina leaving mid X
18:19 UTC Ryan patching alog
21:44 UTC Kyle refilling CP3 (alog 28062)
22:03 UTC Kyle back

biggest = prm pitch changing by 2.5urad

close second = im4 pitch changing by 1,15urad

plot and chart of top 16 optics/DOFs attached

1450 - 1505 hrs. local -> To and from Y-mid 

Opened exhaust check-valve bypass, opened LLCV bypass valve 1/2 turn -> LN2 at exhaust in 50 seconds -> Restored valves to as found state.  

Next CP3 over-fill to be Friday, July 1st.  

Kiwamu, Stefan

We were trying to nail down what is responsible for the Power Recycling Gain change during power-up, so we went through the whole sequence slowly, and gave the system time to settle in each state:

All times UTC of 20160629:

- 18:54:37 / Green dot: Power-up starts from 2Watt to 10Watt
- 19:00:32 / Red dot: Power-up restarts from 10Watt to 40Watt
- 19:06:10 / Black dot: : Power-up to 40W finishes, but additional SOFT offsets remain left off.
- 19:10:15 / Blue dot: Soft offsets in yaw start ramping on over 2 minutes
- 19:12:17: Soft offsets ramp in yaw is done
- 19:15:17: Soft offsets in pit start ramping on over 3 minutes
- 19:18:19: Soft offsets ramp in yaw is done
- 19:21:20: End of undisturbed period - guardian continued
 

In particular, we now there is a significan power recycling gain improvement with a soft yaw adjustment of the x-arm. Additionally, the beam splitter cannot affect the relative alignment of x-arm and power recycling cavity. SO attached are only yaw signals from x-arm and power recycling cavity.

Conclusions:

- During the very first step in power up (green dot) there is a momentary shift in PRM yaw and IM4 yaw, but it does not continue during the additional power increase.
- During the main power increase (between red dot and black dot). There is a consiostent signature in the ETMX control, oplev and IR camera signal, sggesting that we are slightly moving the x-arm with the ASC system. THe ITMX doesn't see much of this in control signal and oplev signal (its camera is broken and not plotted). However no significant motion is visible in any of the power recycling cavity mirrors (this includes PR2, which is not plotted).
- Once we start moving the soft loops with soft offsets, ETMX, ITMX, PR3 and BS clearly respond. Interestingly, the ETMX moves to the same direction as during the power increase.

• I have defined an earthquake state in the SEI_CONF guardian. This state currently just turns off all sensor correction to the the HAM ISIs (X,Y&Z) BSC-ISIs (X&Y), BSC HEPIs (Z) and HAM1 HEPI (X,Y&Z). Following the IFO surviving a small earthquake last night, I could believe that it might work as intended, at least during an otherwise quiet ground environment (low wind, low microseism).
• The dream would be to add automation or operator notifications. Terramon being down makes this harder. When are we getting an epics variable? Otherwise, TJ and I have talked about an environment guardian.
• Chris Buchanan has a study of O1 earthquake statistics (G1600348), it would be nice to get more detail about what IFO signals were impacted by earthquakes during O1, so we know what signals to look at. IMC-F and LSC-X/Y_TIDAL_MON seem like good starting points. Chris's DCC doc includes a link to an earthquake wiki with a lot of information.

Last night at about 23:30 UTC last night, we had an earthquake arrive at LHO. I don't know where it was or how big, because Terramon has been down for a while. The peak BLRMS displacement seen by the ITMY STS in the .03-.1hz region was .28 microns. The first thing I noticed was the IMC F and LSC Y TIDAL on the Tidal StripTool were showing large ~20 second oscillations. When Sheila switched to  the "Earthquake" SEI_CONF state, these oscilations went away, even though the earthquake band ground motion continued going up. My first plot shows the SEI_CONF state ( 40 is the windy state (SC on everywhere), 17 is the earthquake state) ITMY Z .03-.1hz BLRMS displacement in nm, IMC-F and LSC-Y_TIDAL_MON. You can see that at about the time the ground motion BLRMS gets to about 100nm, the IMC and LSC TIdal starts ringing up. As soon as the sensor correction is turned off, the IMC and Tidal settle down. This lock lasted in this configuration past the earthquake, until the commissioners tried to move to a higher guardian state.

The next two images are two times later in the evening, one when the SEI_CONF was still in the earthquake mode (6/29/2016 1:10 UTC), the other after the later lockloss when the commissioners had changed back to the "Windy" SEI_CONF state (6/29/2016 2:13 UTC). IMC and Tidal don't seem to be bothered by either state. The newt four images are various spectra comparing IFO control signals at these two times. Unfortunately it looks like there was maybe another small earthquake and higher winds when the SEI_CONF was still in "earthquake". All "REF" traces are with the earthquake configuration, the live traces are with the windy configuration. I think these spectra show that for now we can be in either state, but some of the suspension length signals show lower drives with the "windy" config. Additionally, the earthquake config will probably not be okay with higher microseism, we have been sitting at about 10th percentile levels for a while now. The last plot just shows the effect of turning of the (BRS subtracted, STS based) sensor correction on the ETMY ST1 T240s. The Y blend is the same for both blue and pink traces. The subtraction at the microseism doesn't look good right now, but I think that is because of the low microseism.

DBB:

.001 files= Lo Power

,002 files = Hi Power

As expected beam pointing and mode matching in the Hi Power path remain to be addressed. 

ISS_RPN:

It looks like AOM diffraction noise is high consistent with the higher-than-usual measured diffracted power.

PDB is "in-loop" and noise is ≈ 1 oreder of magnitude higher than reference.

Pointing noise in X-axis seems to be elevated in the regions between 12Hz & 80Hz. Also, there is some Y-Axis noise between 100Hz & 850Hz.

Marie, Sheila, Kiwamu, Lisa, Stefan

We tried a few things to improve the recycling gain tonight. We  believe that the problem is with something in our PRC.  We can see that the green power transmitted through the X arm stays stable as we power up, we think this means the optics are stable.  When we move the soft offsets to improve the recycling gain we change the X arm alignment to follow something in the vertex.

• We started with dithering IM1+IM3.  The IM1 dither was meant to test is we were clipping into the Faraday, we don't think that is happeneing because we could clearly see our dither in IM4 trans PIT and Yaw, but not in the sum. 
• We tried moving IM3 and PR2 offsets.  We saw taht IM4 trans pit moves as we power up, so we started by trying to correct this by moving IM3. We were repeatably able to improve the reycyling gain from 28 to 29 by moving IM3 by 3000 in pitch counts (we think this is calibrated in urad! Our inital starting point was 58495 on the IM3 pit slider, our best recycling gain was at 55515 urad).  This seems like too big of a move to be the real drift that we are trying to correct, and is much larger than the move needed to correct the position on IM4 trans. 
• We tried moving loop offsets in PRC2 (POPX-> PR3 loop) In pit we went +- 400 counts without seeing any real change, in yaw we saw a repeatable imprvement with a offset of +400 counts. 
• We switched off the loop, moved optics, and broke the lock. 

As TJ noted, since the computer problems earlier we are having trouble relocking the PSL after locklosses, with noise eater, PMC and FSS and ISS problems.  It seems like trying to lock the FSS causes the noise eater to oscillate again after it has been reset. 

Jenne, Lisa, Stefan,

Still running at 40W, for 2h.

Tried to damp some PIs and go to low-noise.

- We successfully damped 15009Hz (ETMY) and 15542Hz (ETMY) (damp settings picture attached.)

- 15541Hz (ETMX) was ringing up.So we tried switching to ETMY and transition the ETMX coil driver to low noise.

- We successfully switch the coil drivers, but...

- We switched to ETMY, low noise, held lock for a while, but saturated the ETMY ESD with 20Hz noise from the ASC, as well as a 532.77Hz line, that we don't know the origin off. (PI?) (picture)

===================================

Also, we realized that DTT is a bit too smart: The 64kHz channel H1:OMC-PI_DCPD_64KHZ_A_DQ can in principle look at PIs above the Nyquest through aliasing. However, since DTT only alows you to select a freqency about 10% below the Nyquist frequency, there is an effecive dead band where DTT cannot see PIs between ~29500Hz and ~36036Hz. The MATLAB script below produces a spectrum up to the Nyquist. Indeed, we found an elevated mode at 30018.3Hz, although it was not quite saturating.

MATLAB code to get spectrum up to Nyquist frequency:

addpath /ligo/svncommon/NbSVN/aligonoisebudget/trunk/Externals/SimulinkNb/Utils/
gwd = GWData();
gwd.make_kerberos_ready;
gwd.site_info(2).port = 31200;
gwd.site_info(2).server = 'nds.ligo-wa.caltech.edu';
H1.channels = {'H1:OMC-PI_DCPD_64KHZ_A_DQ'};
time_fetch = tconvert('26 Jun 2016 01:15:00');
[data, t, ~] = gwd.fetch(time_fetch, 1000, H1.channels);
Fs=4*16384;
pwelch(data,[],[],[],Fs)