Durring ER7 we had 36 locklosses in the final stages of CARM offset reduction.  Assuming that recovering from these locklosses takes about a half an hour (or more if we decide to redo inital alignment because of them),  these locklosses were responsible for at least 18 hours of down time durring ER7, or about 8% of the run.  Indirectly they probably caused even more down time than that. 

32 out of 36 locklosses are the same mechanism

The first thing I noticed is that the majority of locklosses from the states REFL_TRANS, RESONANCE, and ANALOG_CARM are all actually occuring durring the state REFL TRANS, where we transition from the sqrt(TRX+TRY) signal to the REFLAIRI/sqrt(TRY) signal. They are not recognized by the gaurdian because we had some long sleeps in the guardian durring these steps.  This is bad because we risk ringing up suspension modes we we loose lock but the guardian doesn't recognize it.  I've removed sleeps from REFL_TRANS and RESONANCE and replaced them with timers.  

33 of these locklosses all happen within a second or so of the end of the ramp from TR CARM to REFL 9/sqrt(TRY), although sometimes before and sometimes after.  The remaining 4 happen latter, I have not investigated them.

Alignment is a factor 

,

In this plot the build ups durring transitions that result in lockloss are in red, sucsessfull locks are in blue.  I took means of the build in the first 10 seconds of this data (before any of the locklosses happened), and made histograms of sucsefull vs unsucsesfull attempts at transisioning.  Here is an example for AS 90:

We do not suceed when POP18 was below 124 uW and AS90 is below 1275 counts; 12 of the locklosses fit into this category.  A completely different set of 10 locklosses occured when REFL DC was above 6.5 mW at the begining of the attempt.   

Bounce and Roll modes are not a factor

This plot shows that there is no apparent relationship between how rung up the bounce and roll modes are and locklosses, which had been one of our theories for why we were failing in this transition.  

Radiation Pressure causes PItch motion at quad resonance

What is happening durring this transition is that radiation pressure moves the test masses in pitch.  If our beams are not well centered on the test masses the light builiding up in the arms produces a radiation pressure torque on the masses.  Since we are not yet on resonance in the arms, the radiation pressure can vary linearly with cavity length, and if we are not well aligned it can vary linearly with alignment as well.   While we lock CARM on the sqrt(TRX+TRY) signal, the power in the arms is kept constant, so we have no instability.  As soon as we transition to REFL 9 the power in the arms can fluctuate more, causing the optics to swing at the main pitch resonances of the quads.  

This is not obvious if you are looking at locklosses, since it mostly appears that the pitch of the optics is a result of the lockloss.  Attached is a plot of 35 sucsesfull transitions, showing sqrt(TRX+TRY) and the pitch of the three test masses that had functional optical levers durring ER7. All of these signals are rescaled and the mean of the oplev signals from the first 10 seconds is removed.  From about 10 to 15 seconds the input matrix is ramping, and the carm offset is ramped down starting around 15 to 16 seconds.  You can see that the oplev motion is well correlated with fluctuations in TR CARM, and that the oplevs move up in pitch as the CARM offset is reduced to zero.  

  In April we starting using oplev damping loops to avoid this type of lockloss, that was in use durring ER7, but this isn't enough to prevent the locklosses reliably.  Right now I am inclined to think this is the main problem with this transition.  

A few ideas for improving things:

• Make the transition faster. We are curently using a 5 second ramp to change the error signals and a 5 second ramp to reduce the CARM offset to 0 after we transition, both of these could probably be much faster, allowing us to zoom through the transition without enough time to ring up the pitch mode.
• We could try moving the transition to a smaller CARM offset.  When we get closer to resonance, the transmitted arm power will vary less with arm length fluctuations. This means it will become a less reliable locking signal, but it might be that we could get away with going closer than we are now. 
• We could try engaging more ASC loops durring the CARM offset reduction, to get a consistent alignment when we arrive at this transition.  The ITM QPD loops could possibly be turned on earlier, we should be able to turn them on as soon as we move the CARM signal to sqrt(TRX+TRY), at this point we are still running all DRMI ASC loops.  
• we can have the gaurdian check AS 90 and REFL DC before doing the transition, this would have identified about 60% of the times when the transitions were not going to suceed durring ER7.  
• we should not be waiting for the bounce modes to be damped in earlier states of the CARM offset reduction.  While waiting for bounce to be damped the IFO can become misalinged making lockloss durring the transition more likely. We have sucsesfull transitioned with the bounce mode rung up to about 10^-11 m/rt Hz,  and we can transition to DC readout once it is below about 10^-13 m/rt Hz.