Ryan S., Elenna
Thge MOVE_SPOTS state is taking 13 minutes (!) to complete, because the YAW3 ADS DOF is very far off and taking a significant time to converge. Both Jenne and I have found that bumping up the YAW3 gain (PRM yaw) slowly helps converge the loops much faster.
Ryan kindly helped me update the state code to slowly increase the gain if the convergence is taking too long. We added a new timer 'ADS', that waits for one minute after the new A2L gains are ramped (so an additional minute after the 2 minute ramp time of the A2L gains). If, after that first minute, there is still no convergence, then the YAW3 gain is doubled. After that, the 'ADS' timer waits 2 minutes, and again doubles the gain. This process can happen up to three times, which should increase the YAW3 gain to a maximum value of 8. Jenne and I have found that the gain can go as high as 10 in this state. The two minute waits give the other ASC, like SRC1 and INP1 Y time to converge as the ADS pulls the PRM in faster. Once the convergence checker returns true, the YAW3 gain is set back to 1.
We will monitor how this proceeds on this locking attempt. I updated the guardian notify statements so it states when the gain is increased.
This was a sucess- this run through took only 7 minutes. I am shortening the 2 minute wait before increasing the gain to 90 seconds. If that still works, maybe we can go to 60 seconds.
To be more specific, the first attempt as described above meant the state took 6 minutes, 50 seconds. I loaded the change to reduce the wait time from 120 to 90 seconds, which only shortened the state length to 6 minutes, 30 seconds. The gain was only ramped to 8 for a very short period of time. I still think we can make this shorter, which we can do by making that wait time 60 seconds, and maybe taking bigger steps in the gain each time. However, we are still in the RCG upgrade, so I will hold off on changes to the guardian for now.
YAW3 is still limiting the length of the state. In this morning's relock, YAW3 convergence took nearly an additional minute more than the other loops. Once we have caught YAW3 up to everything else, we could make the state even shorter by raising the gain of other ADS loops. Two minutes of the state are taken up in the ramp of the A2L gain, so it is taking an additional 4 minutes, 30 seconds to wait for loop convergence.
Now it seems that PIT3 is taking too much time to converge, so I updated the guardian to also increase the PIT3 gain in the same way.