Durring ER7 we had 36 locklosses in the final stages of CARM offset reduction.  Assuming that recovering from these locklosses takes about a half an hour (or more if we decide to redo inital alignment because of them),  these locklosses were responsible for at least 18 hours of down time durring ER7, or about 8% of the run.  Indirectly they probably caused even more down time than that. 

32 out of 36 locklosses are the same mechanism

The first thing I noticed is that the majority of locklosses from the states REFL_TRANS, RESONANCE, and ANALOG_CARM are all actually occuring durring the state REFL TRANS, where we transition from the sqrt(TRX+TRY) signal to the REFLAIRI/sqrt(TRY) signal. They are not recognized by the gaurdian because we had some long sleeps in the guardian durring these steps.  This is bad because we risk ringing up suspension modes we we loose lock but the guardian doesn't recognize it.  I've removed sleeps from REFL_TRANS and RESONANCE and replaced them with timers.  

33 of these locklosses all happen within a second or so of the end of the ramp from TR CARM to REFL 9/sqrt(TRY), although sometimes before and sometimes after.  The remaining 4 happen latter, I have not investigated them.

Alignment is a factor 

,

In this plot the build ups durring transitions that result in lockloss are in red, sucsessfull locks are in blue.  I took means of the build in the first 10 seconds of this data (before any of the locklosses happened), and made histograms of sucsefull vs unsucsesfull attempts at transisioning.  Here is an example for AS 90:

We do not suceed when POP18 was below 124 uW and AS90 is below 1275 counts; 12 of the locklosses fit into this category.  A completely different set of 10 locklosses occured when REFL DC was above 6.5 mW at the begining of the attempt.   

Bounce and Roll modes are not a factor

This plot shows that there is no apparent relationship between how rung up the bounce and roll modes are and locklosses, which had been one of our theories for why we were failing in this transition.  

Radiation Pressure causes PItch motion at quad resonance

What is happening durring this transition is that radiation pressure moves the test masses in pitch.  If our beams are not well centered on the test masses the light builiding up in the arms produces a radiation pressure torque on the masses.  Since we are not yet on resonance in the arms, the radiation pressure can vary linearly with cavity length, and if we are not well aligned it can vary linearly with alignment as well.   While we lock CARM on the sqrt(TRX+TRY) signal, the power in the arms is kept constant, so we have no instability.  As soon as we transition to REFL 9 the power in the arms can fluctuate more, causing the optics to swing at the main pitch resonances of the quads.  

This is not obvious if you are looking at locklosses, since it mostly appears that the pitch of the optics is a result of the lockloss.  Attached is a plot of 35 sucsesfull transitions, showing sqrt(TRX+TRY) and the pitch of the three test masses that had functional optical levers durring ER7. All of these signals are rescaled and the mean of the oplev signals from the first 10 seconds is removed.  From about 10 to 15 seconds the input matrix is ramping, and the carm offset is ramped down starting around 15 to 16 seconds.  You can see that the oplev motion is well correlated with fluctuations in TR CARM, and that the oplevs move up in pitch as the CARM offset is reduced to zero.  

  In April we starting using oplev damping loops to avoid this type of lockloss, that was in use durring ER7, but this isn't enough to prevent the locklosses reliably.  Right now I am inclined to think this is the main problem with this transition.  

A few ideas for improving things:

• Make the transition faster. We are curently using a 5 second ramp to change the error signals and a 5 second ramp to reduce the CARM offset to 0 after we transition, both of these could probably be much faster, allowing us to zoom through the transition without enough time to ring up the pitch mode.
• We could try moving the transition to a smaller CARM offset.  When we get closer to resonance, the transmitted arm power will vary less with arm length fluctuations. This means it will become a less reliable locking signal, but it might be that we could get away with going closer than we are now. 
• We could try engaging more ASC loops durring the CARM offset reduction, to get a consistent alignment when we arrive at this transition.  The ITM QPD loops could possibly be turned on earlier, we should be able to turn them on as soon as we move the CARM signal to sqrt(TRX+TRY), at this point we are still running all DRMI ASC loops.  
• we can have the gaurdian check AS 90 and REFL DC before doing the transition, this would have identified about 60% of the times when the transitions were not going to suceed durring ER7.  
• we should not be waiting for the bounce modes to be damped in earlier states of the CARM offset reduction.  While waiting for bounce to be damped the IFO can become misalinged making lockloss durring the transition more likely. We have sucsesfull transitioned with the bounce mode rung up to about 10^-11 m/rt Hz,  and we can transition to DC readout once it is below about 10^-13 m/rt Hz.  

I used Sheila's matlab script to plot PR3 Pitch as a function of IFO power.

The plots are from 12 days of data ending June 14th at 17:45UTC.  This includes locks at 17W and 24W.

Locked  = 'in lock, high power' 

Data are locks with input power between 17W and 18W, or locks at 24W or more.

Unlocked = 'low power, locked or unlocked'

Data are from input power less that 3W.

PR3 Pitch position is unchanged locked vs unlocked at 3W.

I removed other data points as transitional power states.

First plot: PR3_PIT_IFO17W.png

Locked pitch position is distributed around -2.8urad.

Unlocked pitch position is distributed around -1.4urad.

Difference = -1.4urad

Second plot: PR3_PIT_IFO24W.png

Locked pitch position is distributed around -3.2urad.

Unlocked pitch position is distributed around -1.4urad.

Difference = -1.8urad

Summary:

The PR3 Pitch position in-lock is changing with the change IFO power.

Nutsinee & Hugh

The Counter weights on top of the WHAM6 ISI Optical Table expected to interfer with Shroud work have been removed.  They are staged under the Support Tubes, on Stage0, and in the Spool at the Septum--Do be mindful of them and apologies for not spending more effort to get them out of the way.

The Five Cables attached to the OMC have been disconnected.  The Cables are mark with foil, 1 2 & 3 strips and 1 & 2 strips bottom to top for the 3 on the -X side and for the 2 on the +X side of the OMC.  They are coiled up, not that neatly, with the three together on the +Y side and the two together on the -Y side.  Note that one of the 3 attached on the -X side comes from the -Y side of the table, the other two attached to the -X side of the OMC come from the +Y side of the table.  The two cables attached on the +X side of the OMC come from the -Y side of the table. See first four attachments.

The M8 Mirror between the OMC and OM3 was removed, and bagged.  I scrounged a narrow dog clamp from the OM3 which was only finger tight anyway to mark the +X side of the M8 fork.  I then held M8 fork against this clamp and removed the one dog clamp holding M8 to the table and used it to mark the -Y side of the fork.  See the last two attachment for 1) before any clamps where moved and 2) after the 'marks' are in place.  Other than which side of M8 faces the OMC, I think this should serve pretty well.  I'm sure it is deterministic which side of the M8 faces M9/OMC.  There is a front and back side of the mount in which M8 is mounted.

Otherwise, the ISI is locked on A B & C Lockers and Robert is still making measurements.

Whilst performing the weekly DBB scans, Jason noticed that there was something amiss with the ISS.
Namely that the output of the ISS photodiodes was 0 and it had been that way since just after we
performed the Tuesday maintenance routines.  It was noticeable that on the ISS MEDM screen the
reported DC value for photodiodes A and B were significantly less than the usual 10V.  Oddly enough
all signs indicated that the loop was locked.

    The output of both photodiodes was checked at the ISS box.  They both indicated over 10V output
as they should.  We measured the same output at the end of the cables going to the ISS servo card.
We injected a DC voltage into the cable going to the AA filter chassis and the MEDM screen reported
no real change.

    Fil noticed that one of the power LEDs of the AA/AI boards was dim, so he power cycled the chassis.
Afterwards all the readings were as they should be.  The chassis was pulled and and frequency response
and noise tests were performed on the board that the photodiode signals are connected to.  The
measurements passed the test criteria.  The chassis was re-installed.

    The strange thing is that the system still thought the ISS was locked.  The only difference on the
display is that the maximum and minimum diffraction percentage was quite separated from the mean value.
There was no sign of an oscillation.  Both the PMC and ISS were happy throughout.



Fil, Peter, Jason

The projector0 computer ran out of memory, so it was rebooted.  The Seismic FOM has been restarted.

The guardian log files are being backed up every hour to   /ligo/backups/guardian  This directory is available to all CDS workstations, so there is no need to log into the h1guardian0 machine to view the full set of logs. Remember that recent logs can be viewed by running the guardlog command

Shiela showed me that PR3 alignment in Pitch is different for in vs out of lock, and it's about 1.5urad.

I've been looking at other optics, and today found that PR2 Pitch also has two distinct alignments of in vs out of lock.

Attached is a trend of PR2 during ER7, pitch on the bottom plot, that shows the two states.

The trend of PR3 pitch is essentially PR2 pitch inverted, however due to techincal dificulties, the plot is forthcoming.

Kyle, Gerardo

Finished replacing vibration cushions on Y-end and X-end Turbos -> Removed iLIGO RGA from BSC9 and replaced with 10" blank -> Finished torqueing flanges added yesterday -> Installed zero-length reducer and valve for aLIGO RGA -> Began pumping BSC10 and BSC9 annulus volumes 


Bubba 

Nearly completed 1" TMDS run at Y-end (still need one support) 


Danny S., Gary T. 

Hung door on BSC9 


END STATION STATUS -> Friday we will remove the 1st generation ESD pressure gauges from BSC9 and 10 and install 4.5" blanks in their place.  Next, we will install isolation valves and new style wide-range Inficon gauges on the domes of BSC 5 and 6 while leaving iLIGO gauges pairs as they are.  Then we will begin pumping the Y-end and X-end.  We will capture the pump down data using the existing gauge pairs while Richard M. arranges for the cabling of the new gauges into CDS. 

During ER7 there was some anecdotal evidence that the DARM gain was changing by a few percent between the handoff to DC readout and our high-power, low-noise state.  One place this could happen is in the power-up step, where we changed the DARM offset to maintain a roughly constant light level on the OMC DCPDs.  As the DARM offset changes the gain in the DCPD_Sum --> DARM path is scaled following a quadratic curve, but we know our approximation for the response doesn't include small static offsets, and maybe this has an effect on our gain scaling.  In this post I try to quantify this error - turns out it should be on the order of a few percent.

DARM Gain Scaling in OMC-READOUT Path

Stefan and I didn't explain this very well after we commissioned the OMC-READOUT path, so I wanted to document how the variables in this path are set by the OMC Guardian.  The carrier power at the AS port is assumed to follow a quadratic dependence on the DARM offset (see T0900023 for more math):

P_AS = (P0 / Pref) * (x / xf)**2          [Eq 1]

...where P0 is the input power, Pref is a power normalization factor, x is the current DARM offset*, and xf is an arbitrary constant we call the "fringe offset".

Step 1. When the OMC is locked on the carrier, Pref is set such that when the DARM offset is equal to the "fringe offset", x = xf, the output of the power normalization step in the OMC-READOUT calculation will be equal to P0:

Pref = (P0 / P_AS) * (x / xf)**2          [Eq 2]

Step 2. Small variations in DARM (dx) will generate small variations in the carrier power (dP) on the DCPDs, like so:

dP = (P0/ Pref) * (2*x / xf**2) * dx      [Eq 3]

The OMC-READOUT path calculates dx and sends the following to the LSC input matrix:

G * dx(x,dP) --> DARM_IN1                 [Eq 4]

..where dx is an explicit function of dP and the DARM offset, x.  The overall gain factor G converts dx into counts at the DARM error point.  G is calculated by taking the transfer function between dx and DARM_IN1.  It's set before the handoff to DC readout, and not changed.  After this point, the reponse of the light on the DCPDs to small length fluctuations (the slope, dP/dx) is assumed to be a linear function of the DARM offset, following Eq 3.

Static DARM Offset

When Stefan and I set up the OMC-READOUT path, Keita pointed out that a static DARM offset was making our lives more complicated.  Before ER7 I swept the DARM offset and measured the photocurrent on the DCPDs - the quadratic dependence has a clear offset from zero, see Figure 1.  The residuals for the simple quadratic fit (blue) have a linear dependence on the offset, while the residuals for the fit that includes a static offset (red) are well-centered around zero.  The best-fit value for x0 is -0.25 picometers, +/- 0.01 pm.  We're not sure where this comes from, whether it's changing, etc (on May 15 Stefan calculated 0.6pm).

Effect of DARM Offset on DARM Gain

If there's a static offset, x0, our model of the carrier power as a simple quadratic is incorrect, and Eq 1 needs to be rewritten as:

P_AS = (P0 / Pref) * (x + x0)**2 / xf**2          [Eq 5]

During the power-up step, we change the DARM offset to maintain ~20mA of photocurrent in DCPD_SUM, and we adjust the slope of dx(x,dP) accordingly, see Figure 2.  If we're using Eq 1 instead of Eq 5, this gain-scaling is incorrect, and our DARM gain will change.  For a negative static offset like we observe, the gain will be lower than it should be.

But, since the offset is very small, this effect is only 1-2%.  Without going into exhaustive detail, the ratio of the slopes for a simple quadratic, Ax^2, and a quadratic that is offset from zero, A(x+x0)^2, is close enough to 1 for small values of x0 that I don't think we need to worry about it, see Figure 3.  The error in the gain as we move from a DARM offset of ~42pm at 2.3W input power to ~16pm at 24W is 1% for x0=-0.25pm.  For x0=-0.75pm the effect is 3%.

Summary: We have a small static DARM offset, this means our approximation of the DARM gain as a function of DARM offset is a little wrong.  The effect should be small.  Eventually we will account for the DARM optical gain in the calibration (the 'gamma' parameter) and small changes in the DARM loop gain will be absorbed into the calculation of CAL-DELTAL_EXTERNAL.

Edit: On further reflection, this isn't the case.  We have a small static offset while locked on RF DARM, but after the handoff to DC readout this offset is absorbed into the power normalization (the Pref variable, Eq 2).  Once DARM is locked on DC, a zero offset will result in zero carrier light at the dark port, by definition.  A small offset might change the gain scaling at the handoff such that we observe a few percent difference from what we expect, but the mechanism is not what I describe here.  Will have to think about it more.

* In the OMC-READOUT path, the DARM offset is referred to as OMC-READOUT_X0_OFFSET.  Here I call it 'x', and refer to the static offset as 'x0'.  Sorry.

A while a go RichM gave me some new blend filters to try. Because we have no IFO, I've installed them on ITMY in the X and RY dof's. I'll leave the ISI like this overnight to get a good long quiet stretch to compare data with.So far, not much to say. I'll put more detail in tomorrow.

I used the sitemap to look at the code revision numbers of each timing device. All DuoTones, Master/Fanouts, and Comparators are programmed to the correct code revision (documented at https://awiki.ligo-wa.caltech.edu/aLIGO/TimingFpgaCode). The 3 issues are:

1) The C_MA_A IRIG-B in the MSR shows Rev. 0. This is a known issue and will be corrected by reprogramming the IRIG-B when possible. The IRIG-B seems to be producing the correct time code output, so we're going to test the effect of a temporary (< 15 minute) interruption to the IRIG-B signal on an IRIG-B device in the test stand to see whether we can reprogram the MSR IRIG-B without causing hiccups.

2) The other 2 IRIG-B in Y_FO_A and X_FO_A show Rev. 133. They should show Rev. 113. They are otherwise operating nominally. It is possible that the timing wiki needs to be updated, or that the revision number was entered incorrectly in the firmware source code.

3) All of the Oscillator Locking slaves in Y_FO_A, X_FO_A, and C_FO_A show Rev. 83, which is the deprecated code. They are operating nominally. Again, this could be an incorrect revision number.

We need to check whether 2) and 3) are meaningful problems that need correction.

Today we finished up the ETMx in-chamber activities adn closed the door.  Sequence of events:

Filiberto checked for TMS QPD cable grounds at the feedthru.  All ok there. (Actually did this yesterday afternoon.)

Kyle and Gerardo finished the port work (that was scheduled in the original vent plan).

Fil check the ESD pinout with aid from an "inside" guy multiple times in order to make clear the understanding of the pins on the air-side of the feedthru.

Jim unlocked the ISI.

Performed the discharge procedure on ETMx as per T1500101 - again, this took around 1 hour.

Swapped the vertical witness plates (1" and 4") that were mounted on the QUAD structure.  1"  optic vertical on floor is now: SN 242.

Removed all equipment and re-suspended lower masses.

Dust counts taken throughout.  Counts were nomnally 30-200 counts in all sizes of particles with few bursts up to 500 in th 0.3um size once in a while when 2 or more people were moving in chambelr.  Counts were zero or 30 (max when entering chamber after lunch or in morning.

Measured ETMx main and reaction chain P and V TFs.  All looked healthy with purge air on and soft cover over door.

Door being put on.

Checked EQ (earthquake) stops (Note this is the same as we found/set on ETMy):

Barrels of ERM ~2mm

Barrels of ETM ~1.5-2mm

12 o'clock stop ~3+mm

PenRe barrel ~1.5-2mm

PUM barrel ~2mm

All face stops 1-1.5mm

Summary: A seismometer was buried 40 m from EY to take advantage of strong attenuation of the tilt signal relative to the acceleration signal from distant sources. Seismometers located outside the buildings may be useful in reducing problems associated with tilt.

Our buildings are tilted by wind. Our seismometers do not discriminate between this tilt and acceleration (like a pendulum), so wind-induced tilt produces spurious control signals that can make it difficult to lock and maintain lock. A tilt sensor can be used to discriminate between tilt and acceleration, but we may also be able to discriminate between the two by taking advantage of source differences in the band below 0.5 Hz, where tilt generates the largest spurious acceleration signals. The tilt in this band is mainly generated locally by the wind pressure against the walls, while the acceleration signal is mainly generated by ocean waves and other distant sources.

While the seismometers are in the far field for most low frequency accelerations, they are in the near field for building-generated tilt (wavelengths at 0.1 Hz are about 50 km). To take advantage of the rapid attenuation with distance in the near-field, I buried an STS-2 seismometer in a meter deep hole I dug 40 m from EY (Figure 1).  Figures 2 and 3 show a comparison of the buried seismometer to the SEI seismometer on the ground inside the EY station. During the low wind period the spectra look virtually identical below 0.5 Hz and the coherence is high because the sources are mostly distant and the wavelengths are large. During the high wind period, the buried seismometer doesn’t change that much (there is some change in the microseismic peak because high/low wind were about 24 hours apart), but the building seismometer shows a huge signal from tilt.  The coherence in windy conditions is low because the tilt is highly attenuated 40 m from the building. Figure 4 shows spectra for higher wind, 15-35 MPH.

Of course the external seismometer would not detect real building accelerations due to the wind. But if the unwanted tilt signal dominates over the wanted wind acceleration signal, a seismometer outside the building may be useful, and I think that this is the case below about 0.5 Hz.

I note that I first tried this in the beam tube enclosure tens of meters from EY, but found that the wind tilts the beam tube enclosure almost as much as the building (but not coherently), supporting a hypothesis that aspect ratio is the important variable. Also, Jeff points out that, if we insulate the buried seismometer well (and I did not) we might even be able to do better than the building seismometers with their current insulation during even low-wind periods. Figure 5 is like Figure 2 except that the signal from the stage 1 Trilliums is included.

Kyle -> Soft-closed GV7 and GV5 -> Vented HAM6 

Bubba, Hugh, Jim W. -> Removed HAM6 South and East doors 

Kyle, Gerardo -> Installed NEG and TMDS reducer nipple+valve+elbow at X-end 

Danny S., Gary T., Gerardo, Kyle -> Hung North door on BSC10 

(NOTE: Noticed ~1 hour before hanging door on BSC10 that the purge-air drying tower had been left off since tying-in the 1" TMDS line on Monday afternoon -> Turned it back on and notified the ETMY discharge crew

Dan Hoak gave us a clue on page 19067 and then Bas Swinkels ran Excavator and found that the channel H1:SUS-OMC_M1_DAMP_L_IN1_DQ had the highest correlation with the fringes. Andy Lundgren pointed me to equation 3 from “Noise from scattered light in Virgo's second science run data” which predicts the fringe frequency from a moving scatterer as f_fringe = 2*v_sc/lambda. Using the time from Dan and the channel from excavator and the fringe prediction, I wrote the attached m-file. Fig 1 shows the motion, velocity, and predicted frequency from OMC M1 longitudinal. Figure 2 shows the predicted frequency overlayed with the fringes in DELTAL. Math works.

PS. Thanks to Jeff and the SUS team for calibrating these channels and letting me know they are in um. Thanks to Gabriele for pointing out an error in an earlier draft of the derivative calculation.

We had four known reasons for having difficulty locking today, one is an unsolved mystery that might be hurting us more often than we realize.

• The BS limiter was off durring DRMI lock acquisition, this is turned on and off by guardian.  I've attached a screenshot of the gaurdian log.  The code should have returned true imediately after having turned off the limiter, and proceeded onto the next state, however, it ran the LOCK_DRMI_1F run again.  The requested state at this time was DRMI_1F_OFFLOADED, which means that this guardian should have moved onto the next state.  Is there a way to search the Guardian logs to see if we have had other instances of this?
  • we have seen this behavoir in the past, and Jamie suggested that this could have been because the state LOCK_DRMI_1F was requested.  I know that was not the issue today, but I changed the state so that it is no longer a requestable state anyway, since it should never be requested.
• The roll mode damping for ETMY was switched off at the output.  All the roll mode damping settings are now enforced by the guardian, using ezca.get_LIGOFilter().only_on.  

I reloaded both ISC_DRMI and ISC_LOCK guardians today, to incorporate these changes.  

• We had difficulty transitioning to ETMY ESD, evan is writting an alog about what we think is the solution
• Now the winds are quite high, the forecast says they will die down around 1 am.  

Good luck TJ!