When we got to full lock today, there was excess low frequency noise around 30 Hz. The noise was from the LSC, and we quickly realized that the whitening gain for LSC POP A RF45 was set to 21 dB, but the antiwhitening gain was set to -15 dB. When going to full lock, those numbers should be 15 dB and -15 dB respectively. However, when performing initial alignment, those numbers should be 21 dB and -21 dB.
I trended the guardian state numbers, and this is what occurred:
- the IFO went into prep for locking (state 9) and held there for some time. During this time, the whitening gain changed from 15 dB to 21 dB
- the IFO then went to SDF revert (state 6) briefly
- the IFO moved to Ready (11) and then Locking Arms Green (12)
Tony (operator on shift) ran an initial alignment. He tells me he took ISC_LOCK to down (which holds in Prep for Locking) and selected the INIT ALIGN guardian by hand. Once initial alignment completed, he proceeded to move to locking.
This means that the initial alignment guardian correctly set the whitening and antiwhitening gains to 21 dB and -21 dB (lines 728-730 of ALIGN_IFO.py). However, the Prep for Locking state never reran, so the gains were not properly reset to 15 dB and -15 dB (lines 410-414 in ISC_LOCK.py). However, we did run through SDF Revert, which resets the anti-whitening gain to -15 dB, because that filter bank is SDFed.
My understanding of the "proper" way to run initial alignment is to take ISC_LOCK to the initial alignment state, which will automatically run the init align guardian. Then, the guardian state path forward will correctly move through the states like prep for locking and ensure this error does not occur.
As a reminder, we need the whitening gains to be 15 dB and -15 dB because of the higher SRCL offset we are running with.
We (Jenne and I) have also learned you cannot changing analog whitening gains in lock, as it causes an immediate lockloss (something that Daniel Sigg told us wouldn't work, right after we tried it).
I'm tagging Opsinfo and GRD so the relevant people know of this error mode.