[Jenne, Kiwamu, Sheila, JeffK]
Today, the interferometer decided that it didn't want to go from ASC_PART2 to ASC_PART3 without some hand-tuning each lock. However, after some investigation, it's still not entirely clear what the right thing to do is :(
The things we were playing with were the offsets in the CSOFT and DSOFT pitch filter banks (can only be tweaked *after* the loops are on, since there's an integrator always on in each of those filter banks) and the transmon qpds' ASC-[x,y]_TR_[a,b]_[pit,yaw]_OFFSETs. These things are kind of the same, but you need to adjust the transmon offsets if you want to do some adjusting before the loops are closed, and you need to adjust the CSOFT and DSOFT offsets if you want to do some adjusting after the loops are closed.
We've had locks where we didn't survive if we didn't re-zero the qpd offsets before going to part3, and we've had locks where we didn't survive part3 if we didn't adjust the CSOFT and DSOFT offsets. But, we've also had locks where the opposite was true.
Anyhow, at this point, I think our advice is to try going through the guardian like normal (no by-hand offsetting), and if that fails one or more times, call a commissioner (it's me for now, so please call). For example, the last lock, I re-set the qpd offsets during asc part 2, but forgot to engage the CSOFT and DSOFT offsets, and we seemed to be just fine. So, it's not totally clear yet what the final prescription should be.
Tonight we had one OK lock with the offsets as Jenne aloged above, which we lost because of an EQ (see Cheryl's shift summary). On relocking without changing the offsets or doing anything special, we lost lock due to the CSOFT pitch instability that has plauged us when we have changed the TMS QPD offsets in the past. (There was a small EQ that arrived on site shortly after this lockloss, but clearly we lost lock because of the pit instability before it arrived). After this, Cheryl and I let the IFO lock and engage ASC with the offsets that Jenne had found this afternoon, then once the soft loops were closed very slowly reverted to the old offsets (see Cheryl's alog) at 2 Watts. Our recycling gain was around 41 when we started, and although it dropped as we reverted the offsets it returned to 41-42 once they were all reverted. We powered up slowly and the recycling gain stayed above 40. After about 20 minutes I thought that things seemed better than they have in several days, so I suggested we offload the full lock ASC. Sadly this broke the lock.
Since this situation is very similar to what happened Dec 3rd -4th, I re-ran the script I used then to look at build ups (23959), for the past 10 days. The results are very similar, the drop off in the POP DC trace is very similar to the drop off in the arm transmissions (ratio of arm transmissions to POP DC is shown in the middle panel, and is changing by at worst 1% while the powers drop by nearly 15%. )
One explanation we considered in December was that the OMC was becoming misaligned, causing the DARM offset to change. The second attachment shows the OMC ASC control signals and the QPDs, which are out of loop. You can see that they move around even in locks where the recycling gain is stable and don't seem to be any worse in the last few days, so it seems like this is not our problem.
Jim has just relocked with a recycling gain of 41, which has been stable for nearly 20 minutes, so this is looking more promising. One thing that we did is edit lscparams so that the guardian will go to 10 Watts in the increase power state instead of the normal set point of 23 W.
Until this is reverted the steps to follow for power up are:
1) request INCREASE_POWER (it is important to change the PSL power only in this state, this state automatically takes care of gain scalings for you).
wait until the power reaches approximately 10 watts, and the recycling gain seems stable. We have been waiting a few minutes here to make sure it stays stable, that should not normally be necessary.
2) open the PSL rotation stage medm screen (from site map under IOO on the top right hand side, Rotation Stage).
3) type 23 in the requested power dialog box and hit go to power
4) only once the rotation stage has finished moving is it safe to leave the INCREASE_POWER state.
For the record, I give you another similar example that Nutsinee and I had experienced back in October (alog 22575 and comments). At that time, we conluded that it was due to subopitimal alignment in TMSY.
Opened FRS ticket
Fault Report 4186 - Trouble advancing past Guardian state ENGAGE_ASC_PART2
(This was done as "for-real" practice with R. Oram)