At around Sep 09 2018 07:10:53 UTC (GPS 1220512271), the IMC and FSS lost lock. I was unable to recover the FSS. The FSS is railed at about 11.9 volts. I can move this value around by turning off the autolocker and moving around the common and fast gain sliders, but as soon as I reengage the autolocker it rails again. The laser crystal temp at the time of the unlocking was 0.31 K. I tried manually setting the temperature there, but the PZT just is not responding, and every time I ask it for control it rails to 11 V. I pressed pretty much every button and shook every slider on the PSL_FSS medm screens. I don't know what else to do from the control room. My uninformed opinion is that somehow the PZT lost it's lower DC power rail, since it always rails positively. However, I can get the Fast monitor to read negatively if I disengage the autolocker, turn on the "mode loop" (no idea what that is), and turn up the fast gain (Picture 2 below), so I'm not sure what's actually going on.
There is no DC response from the FSS refl PD. The value on the REFL PD doesn't change when the PMC is locked or unlocked, however I can see that the diode did show a higher level of reflected light during a time when the FSS was unlocked last Tuesday.
In the second attached screenshot you can see that as I have locked and unlocked the RFPD over the last hour or so there has been no response on the FSS PD.
Another issue that is confusing is that the PMC sometimes reports that it is locked when it is not. The PMC medm screen has a misleading statement that the PMC is resonant if H1:PSL-PMC_MIN_TRIGGER < H1:PSL-PMC_TRIGGER_OUTPUT < H1:PSL-PMC_REF
Looking in the PMC model this is unrelated to the logic that goes into the flag H1:PSL-PMC_RESONANT, which is missing from the medm screen.
Corresponds to FRS Ticket 11427.
It's IMC VCO failure.
See the first attachment, at some point IMC VCO failed and eventually came back.
No VCO output = no diffracted light into RefCav.
That's the reason why there was no response from FSS DC diodes even though Sheila locked/unlocked PMC.
Reference input of IMC VCO is taken off of an RF amp and 8-way splitter (D1000124) that is also used for COMM and DIFF which were OK (indeed everything except IMC VCO was OK, see 2nd attachment), so this cannot be the amplifier problem, it should be VCO itself or the cable connecting the splitter and VCO.
It happened again at about 11:50 AM (or 18:50 UTC).
See the third plot. This time I went to the floor and wanted to power cycle VCO, but since it doesn't have a convenient power switch I disconnected the reference input cable (that's where VCO OUTPUTMON dropped from -10 to -34dB). After playing around with it, swapping cables with one of ALS VCOs and such, I couldn't find anything wrong, reconnected everything back, and it seems to be working OK for now.
I cannot tell if this is an electronics or a shoddy connection or shoddy cable/attenuator.
IMC VCO showed this behavior 5 times (counting today/yesterday as one) in the past 4 months where the output intermittently dropped to -10dBm.
I also plotted COMM VCO output, you might think something happened to COMM too for #3 and #4, but it's actually not the case (2nd and 3rd attached).
I've found the FSS auto locker parameters totally out of whack.
Oscillation threshold of 3 doesn't make sense, even when it's oscillating I don't think H1:PSL-FSS_PC_PP goes above 3. I manually set it back to 0.6.
State transition delays of [delay1, 2, 3]=[10, 10, 10] sec don't make sense either. The servo is first turned on with low gain after the refcav is brought close to resonance for delay1 sec. Then wait for delay2 and the common gain is ramped. Then wait for delay3 and the temperature loop is turned on. I manually changed them back to [0.1, 1, 0.1].
I pulled the PSL-VCO (S1200558) and performed some testing on it today. Power checks out, tuning checks out, monitors work, I was unable to compete the phase noise test before noon. I checked the attenuators and the REF cable, these passed as well. I changed the REF cable position on the RF Amplifier Splitter from position 5 to position 7.
Based on Keita's trending, REFERENCEMON drops out, this would cause OUTPUTMON to drop out as well and it does. DIVIDERMON does not drop out all the way, curiously. The power OK signal was good throughout these incidents, the intermittent fault is not likely power related.
Looking at the past 4 months behavior posted by Keita above, the glitching seems to be periodic, and when I zoomed in most of this glitching is happening on Tuesdays before noon.
5 = Tuesday June 12, 11 AM
4 = Tuesday June 24, 11 AM
3 = Tuesday July 31, 8 AM
2 = Thursday August 16, 9 AM
So it misbehaved when somebody worked near the rack. Maybe poor connection at the connector (too little or too much torque?) combined with poor strain relief.