I spent several hours in the optics lab to diagnose the FSS trouble that we've been experiencing lately. I've found two failure modes though there may be more. One was fixed and the other was not.

1) EO monitor slowly creeps up due to 14.4 kHz oscillation building up.

Fixed by rebalancing the PZT-EOM crossover. (Fast gain was 592 or something like that in the morning, now it is 815).

In this mode, FSS still appears to be locked and the REFCAV transmission looks OK, but if you lock the arm your arm transmission slowly degrades, and after a while it becomes impossible to lock the arm.

When you look at the EO RMS monitor and the PZT fast monitor on the oscilloscope, EO monitor creeps up very slowly though you wouldn't notice anything in the PZT, so you would think that this is some really high frequency stuff. 

However, it turns out that this is almost entirely due to 14.4kHz oscillation slowly building up. PZT signal has lower frequency crap so 14.4kHz is not that apparent on the oscillo.

Attached shows the spectrum of the common path test point in the FSS box in bad and normal state.

2) Fast glitch in PZT.

Not fixed.

This is distinctively different from the first one in that there is a big fast glitch observed in the PZT signal first. ("Fast" just means that it looks instantaneous on the oscilloscope that is set to its slowest setting, so it could be 100Hz or 100kHz.) 

Sometimes the PZT can take it and the signal goes back to normal after a while. Sometimes the PZT cannot, and the refcav loses lock. It looks like the refcav can relock itself in this mode even if the lock is lost.

EOM monitor doesn't show anything until refcav loses lock.

Even though the glitch itself seems to be fast, what seems like a transient response of the servo is very slow (slower than 1Hz).

I wanted to see the glitch on the analyzer in real time, but somehow it happens only when I'm not looking at the screen.  We need a good young scientist who can sit in front of the analyzer without blinking for an hour.