Tonight we are again having random, fast locklosses, in different configurations. We also are seeing some large glitches that don't knock us out of lock. Again they seem to correspond to times when there is something noisy in SR3 channels, while its not clear that the SR3 channels are seeing real optic motion, it is probably worth swapping some electronics as a test because these frequent locklosses are making commissioning very difficult.
See 27437 and Andy Lundgren's comments
The first attached plot shows that something about this channel changed on May 10th, and that there have been noisy periods since then. The next two are two more examples of sudden unexplained locklosses where something shows up in SR3.
SR3 is not the problem after all.
KIwamu and I unplugged the cables from the Sat amp to the chamber for both M2 and M3, and the locklosses and glitches still happened. The good news is that Kiwamu seems to have found a good clue about the real culprit.
Our current theory is that locklosses are due to the ISS which shuts itself off for some reason at random times at a rate of once in 10 minutes or so. This causes a glitch in the laser intensity. Before a lockloss, there was a fast glitch (~milliseconds) in PRCL, SRCL and CARM error signals. That made us think that the laser field may be glitching. Indeed, we then found that the ISS had gone off automatically at the same time as the glitch and seemingly had caused the subsequent lockloss. We then tested the stability of ISS in a simpler configuration where only IMC is locked. We saw glitches of the same type in this configuration too.
In order to localize the issue, we are leaving the ISS open overnight to see if some anomaly is there without the ISS loop.
Conclusion: it was the ISS which had a too low diffraction power.
According to the overnight test last night, I did not find a glitchy behavior in the laser intensity (I looked at IMC-MC2_TRANS_SUM). This means that the ISS first loop is the culprit. Looking at trend of the recent diffraction power, the diffraction power kept decreasing in the past few days from 12-ish to almost 10% (see the attached). As Keita studied before (alog 27277), a diffraction power of 10% is about the value where the loop can go unstable (or hit too low diffraction value to shut off the auto-locked loop). I increased the diffraction power to about 12% so that the variation in the diffraction power looks small to my eyes.
Note that there are two reasons that the diffracted power changes, i.e. intentional change of the set point (left top) and the HPO power drift (right bottom). When the latter goes down, ISS doesn't have to diffract as much power, so the diffraction goes lower.
In the attached, at the red vertical line somebody lowered the diffraction for whatever reason, and immediately the ISS got somewhat unhappy (you can see it by the number of ISS "saturation" in right middle panel).
Later at the blue vertical line (that's the same date when PSL air conditioning was left on), the diffraction was reduced again, but the HPO power went up, and for a while it was OK-ish.
After the PSL was shut down and came back, however, the power slowly degraded, the diffraction went lower and lower, and the number of saturation events sky-rocketed.
