Evan and Kiwamu,
We have been unable to lock the interferometer since this Tuesday (alog 25431). We finally identified and fixed the problem.
It was the cage servo of SR3 which for some reason had a too high control gain and therefore kept feeding sensing noise to the suspension up to 1 Hz. We are happy now.
[Symptoms]
The symptoms were described in Evan's original alog.
In addition, today we noticed that the lock loss in DRMI often happened with a mode hop, indicating that some kind of misalignment was involved. Since the mode hop can be easily triggered by the large optics (i.e. BS, ITMs, PR3 and SR3), we suspected the large optics. Then, we found that we could stably lock PRMI which pointed us to SR3.
[Something happened in this Tuesday]
Trend data showed increase in the pitch motion of SR3. See the attached. The increase of noises occurred at around 4 pm local time of this Tuesday. We have no idea what triggered this situation.
By the way, the UR OSEM seemed as if it had reduced noise, but it turned out that this was due to the 60 Hz power line for some reason decreased at the same time. We then switched off the cage servo to see if it improves the situation. Indeed it did. In fact DRMI was able to lock more stably.
We have checked the damped spectra of the top and bottom stages without the cage servo comparing against the past ones (alog 17781). We did not find any obvious issues with the suspension when the cage servo was off. At this point, we concluded that the problem is in the cage servo code rather than the suspension or its damping system.
[The fix]
We lowered the control gain in the guardian by a factor of 100 which may sound big to some readers, but this gave us a good control time scale of 3 minutes according to a coarse impulse response test.
The resulting spectra at the top and bottom stages seemed as quiet before. So this is good. The code is checked into SVN.
We still don't understand why this happened.
Later, Jeff B locked the interferometer all the way to low noise state where we did a simple test of switching the cage servo off and on to see if it has some effect on the cavity build ups. We did not see any obvvious effect. Good.
A followup study.
I checked the following items to investigate what could possiblly change the cage servo. Nonetheless, still no clue.
-
Conlog for *SR3* channels between 02/02/2016 23:00:00 UTC and 03/02/2016 00:00:00 UTC
- No change found out of 1461 EPICS variables. See the first attachment.
-
Checked whether the front end was restarted.
- No restart found. The last model restart was January 26th 2016 according to log.txt in traget.
-
Guardian code seemes to have used the same gain value of 1 in the past 4 days until I changed it to 0.01
- Comparison with the previous SVN version from Dec is shown in the second attachment.
-
Guardian was not reloaded in the past 4 days until I started fixiing it today.
- See the last attachment.
- This rules out the scenario that the guardian had a wrong by loading a temporary code.
-
I have confirmed that the gain of the system indeed changed by looking at the output of the cage servo at M1_DITHER_OUTPUT.
- See the last attachment.
- This is either explained by reduced actuators' efficiencies, or increased servo gain (or increased sensing gain of the OSEMs at the bottom stage but this scenario has been already ruled out by the fact that we still see similar spectral shapes). Plus, since reducing the servo gain fixed the issue, it is unlikely that the actuator efficiency dropped. This supports the hypothesis that the servo gain increased.
- By the way my reduction in the gain by a factor of 100 now provides a lower UGF than it was according to the thinckness of the trend curve for M1_DITHER_OUTPUT.
- The last update of cdsutils which provides this servo function in python was on Aug 6th 2015.
