J. Kissel, (R. Savage, S. Karki,  Y. Lecoeuche #PCALTeam), (J. Driggers, S. Dwyer #CommissioningTeam)

This afternoon, after the commissioning team hard-coded the information about the calibration lines into the Guardian such that they stick (47369), they found that the PCAL had gathered all sorts of nasty combs and harmonics, and was actually causing physical displacement in DELTAL_EXTERNAL.

Upon review by myself and the PCAL team (suspecting come terrible amount of clipping because the transmitted beam mount had become misaligned again; see LHO aLOG 47336), we tried
- turning OFF all PCALY calibration lines (all features disappeared, but returned when re-engaged)
- turning OFF all PCAL calibration lines, and turned off the Optical Follower Servo (this turns OFF the entire PCAL demodulation system so all light disappeared, but features returned after re-engaging the OFS and calibration lines)
- turned OFF 9.73 Hz line, leaving others on -- some harmonics of the nastiness disappeared.
This clued in Rick / Sudarshan to suspect that the optical follower servo's OFFSET that keeps the modulated output in the middle of the range of the AOM driver, had been set too high, or potentially, equivalently, the PCAL Y laser was dying (such that the existing offset was too high for the now lower power laser).
However, similarly, it could be that we simply were requesting too much drive level, equally saturating the linear region of the OFS / AOM.

So, I compared the current requested amplitudes, against what I want them to be (changed Sep 2018, but since then there has been no reason for them to change) -- see LHO aLOG 43964. This was the problem:
PCAL Line (Hz)       Should be (ct)     What it was (ct)      Channel that controls it
    7.93                 5000               5000                  H1:CAL-PCALY_PCALOSC4_OSC_SINGAIN
    36.7                 1000               1000                  H1:CAL-PCALY_PCALOSC1_OSC_SINGAIN
    331.9                20000              20000                 H1:CAL-PCALY_PCALOSC2_OSC_SINGAIN
    1083.7               5000               20000                 H1:CAL-PCALY_PCALOSC3_OSC_SINGAIN

I attach ASD of the RXPD in several configurations. 
Gray -- what we were seeing that caused everyone to launch this investigation. Lots of non-linear harmonics.
Green -- "what it should be" from above, but with the amplitude of 7.93 Hz line reduced by 5. No non-linear harmonics.
Red -- "what it should be" from above. Some non-linear harmonics, but none peaking above the noise floor.

We're now running in the Red configuration.

We all checked the SDF system which should have caught this mistake but it did not reveal that the amplitudes were in error, which means, somehow, the higher amplitude settings were accepted into the SDF system. Who knows. I've now accepted the "should be" amplitudes into both the safe.snap and OBSERVE.snaps.

Since there are some non-linear harmonics in this configuration (in which the 7.93 Hz line is at the higher, 5000 ct amplitude, which we need at this SNR of ~10 in order to resolve the SRC detuned spring frequency), we should probably see if the OFS OFFSET can be tuned a bit, so we're not skating along the non-linear rails.