Sheila told me that there were two kinds of mysterious events in IMC
- IMC unlocked during ALS diff
- MC1 and MC3 suspensions tripped for some reason
So I looked into these two kinds of incidents. Here is a summary of my understanding of what were happening, although they are not 100% understood yet.
(IMC unlocking events)
This was due to saturation in the longitudinal drive at the M3 stage of the MC2 suspension. I picked up two unlocking events from this after noon at around 22:24 and 22:53 UTC respectively. Both unlocking events were caused by saturation in the M3 stage of the MC2 suspension. I attach a screenshot of the lockloss analyzer for the 2nd event:
In the upper right panel of the attached lock loss, it is clearly seen that the M3 longitudinal drive hit the limiter (which had been set to be close to the DAC saturation counts). According to T1300079-v2, the range of the M3 stage is about 0.5 um in peak. Therefore, roughly speaking, the IMC cavity must have moved more than 0.5 um in order to saturate the DACs.
On the other hand, we have no idea of why the IMC length motion (or perhaps PSL frequency) moved by such amount on a time scal of 200 msec or so.
We switched the coil driver setting to "Acq on, LP off" to see if it helps, but we then had another incident which was also caused by the same issue. We then turn them back into the low noise mode i.e. state request=1.
(MC1 and 3 tripping)
At the beggining, I was imagining that the MC1 and MC3 suspensions were kicked by the ASC loops right after the IMC unlocked. However, carefully looking into them, I found that they tripped some times after a lock loss. In a case, they tripped ~40 seconds after the lock loss event at around 22:53 UTC. According to various IMC ASC data, it seems that DOF4 gave a huge impulse (~107 cnts ! ) to the MC1 and MC3 suspensions. This happened when the IMC LOCK guardian catched a fringe and hence a high build up. See the attached below:
It looks as if the DOF4 maintained a large number during they were off and then released it when the ASC was triggered. A strange point is that any of the IMC ASC loops, including this DOF4, should not maintain the past values as all the intergrators are cleared by the filter trigger in every lock loss. We double-checked the history handling of the integrators in foton, but they were set right -- no history maintained. Also, we reloaded the latest filters in order to be 100% sure that the filters in the front end are actually the ones in the latest foton file.
P.S.
We hoped that the issue would be gone by loading the latest foton file, but it seems to still persist. We had another incident where the MC1 and MC3 suspensions were tripped again when the IMC lost the lock. The lock loss was initated by saturation in the M3 stage of the MC2 suspension. Almost at the same time as the lock loss, the DOF4 kicked the MC1 and MC3 suspensions again. Hmmmm.... We need to further investigate this issue.