Ed, Cheryl, Corey, Sheila, Keita,
We are relocked. The keys to relocking today seemed to be: waiting for the wind to calm down, powering up to 10W before moving the spots to their high power positions, and fixing the FSS common gain which was accidentally low.
A general summary of the last ~38 hours:
My understanding of the ~38 hours of down time we've had: After the dolphin crash, some time was spent recovereing the IMC, I don't really know what was done, but it does seem that we now need rather different offsets on the IMC PZT than we did before the crash, and this was found. We had already been down for about 16 hours when Corey got in touch about lockosses at ENGAGE_ASC and MOVE spots last night. We tried a few rounds of initial alignment, including reverting changes to the IMC sus sliders, which didn't seem to make much of a difference. For initial alignment, we found that we had to reduce the trigger threshold for SRCL in SRY from 0.5 to 0.3, we didn't implement this in the gaurdian but it helped more than once so probably do want to make this change permanent. Corey had about 1 attempt at locking after we had exhusted the initial alignment possibiltes, and lost it with a similar fast lockloss while ADS was moving. Then the winds picked up, which made locking durring Ed's shift impossible and closed SR240 with tumbleweeds, gusty wind continued this morning on Cheryl's shift, so we could probably estimate that about 14 hours of this down time is environmental. The FSS gain was accidentally low for about 6 hours, which prevented power up. The remained of the down time which isn't due to the dolphin crash, the wind, or the FSS was due to the problem we had moving spots. It took us an long time to find a workaround for this because of the other problems.
More details about today's efforts:
Cheryl and I were able to get to DC readout by commenting out the lines in the gaurdian state MOVE_SPOTS which change the A2L gains and actually cause the spots to move. We tried made a few attempts to move the spots in smaller steps, or to move some of the spots first then others, and to try moving the spots part of the way then adjusting the SRC1P offset to improve the POP18 build up, none of which worked. Some of the locklosses that we had in these attempts could have been coincdent with wind gusts, but there was a pretty consistent pattern that we'd loose lock a few minutes after moving spots, without any apparent oscillations in ASC or LSC loops. The sideband build ups have always dropped when we do the spot move, and the drop in power before locklosses in the last day has been similar to the drops we had in sucsesfull locks before the computer crash.
We also made three attempts to power up without moving the spots, but lost lock very suddenly. Keita and I looked at FSS, which is a good suspect for sudden locklosses during the power increase. The alignment looks good, but the gain was reduced earlier today, durring a time when the FSS wouldn't lock. This is something that is unmonitored in the SDF, which is why Cheryl and I didn't catch this earlier when we checked SDF, but it isn't set by the guardian. (It used to be hard coded in the guardian but the PSL team sometimes changes the FSS gain when the reference cavity alignment drfits, so I edited the guardian a few years ago so that it just checks the gain setting, lowers the gain if the FSS is oscillating, and then puts it back that the setting it found). Keita set this to be monitored in SDF.
After fixing the FSS gain, we left the spot moves commented out of the guardian but left the SRC1P offset handled as it has been, used when ASC is engaged but turned off just before power up. We were able to increase the power to 10W using INCREASE_POWER, and then manually moved the spots by setting the ADS gains to half their full power settings, waiting for things to settle, and then setting them to the nominal gains. In October Craig added a guardian servo to the MOVE_SPOTs state that adjusts the TMS alignment to center on the TMS QPDs, we didn't use this doing things manually and the most off center QPD is around 0.6 now. After doing this manually Corey was able to relock to nominal low noise.
I put the A2L gain changes into increase power, and have left them commented out of MOVE_SPOTS for now, they should happen in one step at the same time as the DHARD loop shaping in increase power. I am leaving it this way for tonight, but I think that tomorow it would make sense to change the guardian to split INCREASE power into 2 states and move MOVE_SPOTS to 10W.
Things that still need follow up:
SRY triggering, moving the move spots gaurdian state to 10W.
FAC staff are aware of that water leak in outdoor pipe.