Sheila, Camilla
The operator team has been having a lot of issues with the SQZ FC locking recently, e.g. 86153. We hoped this was due to incorrect SEI settings 86120, but this didn't solve the issue.
The issue isn't with green locking as green locking can stay for ~minutes. It appears to be when we get to state FC_ASC_ON, although the ASC is not the issue (Oli had tried keeping it off). Comparing an unsuccessful to successful time, both the green and IR powers are lower when we are unsuccessful, as well as the FC_WFS_A_SUM Q_OUT channel, both the WFS and the IR signal get noisy with a growing ~15Hz oscillation before the lockloss plot, which happens before the ASC is turned on.
Sheila notes that this means that the FC IR LSC loop was probably unstable. We measured the crossover and could see it was very close to unstable at 15Hz, see plot. We increased the gain in at the FC LSC input matrix a factor of 1.3, H1:SQZ-FC_LSC_INMTRX_RAMPING_2_7 via sqzparams.py (fc_wfs_a_ir_gain) from -0.86 to -1.12. This brought the crossover back to the reference with 50deg of phase margin. Ideally we would compleatly redesign this loop but it was hard to design originally 66092.
To get FC to lock successfully, Sheila did had to trend and revert FC2 and ZM3 alignments. It seems that while the FC locking is struggling, the fC ASC can pull these away from nominal.
Interestingly the successful lock, as the ASC comes on, the green light decreases as the IR light increases, maybe this si a sign that green and IR aren't well co-aligned. There is picos in the green FC path, but we don't want to touch these as it's more likely that different OPO crystal voltages or spots change the alignment.
Other things that have changed over the past ~month:
- FC launch power is lower and correlated to the FC_TRANS green power being lower, plot. We were having FC issues at the end of June when this was low as well. We can't increase this without taking power from the green pump.
- FC IR power is degrading quickly throughout a lock, the operators adjusting the OPO temperature brings this up again, plot, but this only sometimes fixes issues with FC locking, e.g. 86153.
Hopefully this has fixed our issues, but if this happens again, steps operators should take are (added to wiki):
- Check OPO temperature via taking SQZ_OPO_LR guardian to SCAN_OPOTEMP (will take 3 minutes, can watch via !SQZ Scopes > OPO temp) once it's done you can take SQZ_OPO_LR back to LOCKED_CLF_DUAL and check the OPO temp plot that's in https://lhocds.ligo-wa.caltech.edu/exports/SQZ/GRD/OPOTEMP_SCAN/
- If H1:SQZ-FC_TRANS_C_LF_OUTPUT is a lot lower than 80, should check alignments and revert or optimize of FC2 and ZM3 (explicit instructions in the wiki (under "Issues with SQZ_FC Guardian").
