yesterday we noticed that ITMX has been tripping, it seems to happen within a minute or so of becoming fully isolated. HEPI seems to have position loops running, there is no feedback from ISC to the suspension or HEPI. The last few trips have been just stage 2, actuator trips. It looks like something is ringing up.
The stage 2 blends were on Tbetter expect RZ, which was on Tcrappy for the two trips plotted in the first screenshot attached. I've tried setting stage 2 blends to start, this also caused a trip which is the second attached screenshot.
Now I have set the guardian to isolated_damped, so that stage 1 is isolated and stage 2 is damped. So far its not tripped.
For the last 20 minutes or so, stage 2 has been isolated level 1 with no problems.
It seems that the level 3 controllers for X are the problem. It was fine with stage 2 on level one controllers, and it stayed isolated with all DOFs other than X at level 3. It tripped even when I used no boost for X.
From looking at the trends it seems as though the ITMX ISI hasn't really been isolated since the first week of june. Jeff suggests that the problem could be that the plant has changed durring the vent, and now the controllers designed before the vent aren't stable (for X). Here is a screen shot of a spectrum just before one of these trips. The problem seems to be at 147 Hz. The filter is also plotted here, with the cursor at 147 Hz.
Hugh also reported a problem on August 8th, at 320 Hz. (alog 13294 )
It looks like this loop needs some TLC. For now I'm leaving the guardian at ISOLATED_DAMPED, but it can be semi isolated by going to the command scripts, and turning on the level three controller for all DOFs except X.
I worked on these loops a bit this morning. I ended up adding a notch in the ST2 X isolation loop at 147, and now everything turns on. I wouldn't say everything is happy, as the drives are still quite high, so it would be good to get an in-vac tf and re-visit the loop design. It's been running for the last 20 minutes or so. Will talk to commissioners about a convenient time to do further work on this.