J. Oberling, R. Short, M. Pirello, F. Clara
Several PSL items during yesterday's maintenance day:
- Take TFs of PSL stabilization systems to see where things stand before we start removing things
- Especially want to check if the apparent EOM resonance peak noted here in the FSS (~1.68MHz) moves with Common gain changes as we would expect
- Remove in-service TTFSS (SN LHO01) for characterization in the EE shop for preparing working spares and checking for not functional components
- Test spare TTFSS (SN LHO03) to see if it will lock the RefCav after Marc found several gain issues since the previous test
- Check ISS AOM RF input connection for tightenss after a suggestion from Tony based on observations seen in PCal AOMs
- Wire Amp2 power monitor PD to a higher data rate channel (was currently only going to PSL Beckhoff and read into EPICS at 16Hz)
Summary
- Took TFs, all looked good, FSS Common gain moves the 1.68MHz peak as expected
- Found ISS AOM RF cable loose, re-tightened it with an SMA torque wrench
- Wired Amp2 power monitor PD to a faster data rate channel, H1:PSL-PWR_HPL_DC_OUT_DQ
- Tested and characterized in-service TTFSS, SN LHO01, found and replaced a bad PA85 op amp
- Tested spare TTFSS, SN LHO03; would lock RefCav but would oscillate for a couple minutes before calming down; unit has less gain than LHO01
- Tuned notches in TTFSS, SN LHO01, reinstalled and locked RefCav
Details
To start we took PMC, ISS, and FSS transfer functions. The PMC and ISS (1st and 2nd attachments, respsectively) look as we had set them at the end of the NPRO swap. The FSS looked as left after lowering the Common gain to 13dB on Nov 1st, with the marker on the 1.68MHz peak (3rd attachment); we raised the Common gain to 15dB and saw the peak raise by 2dB, as we expect. I didn't get a picture of it, but we also lowered the Common gain by 2dB to 11dB and saw the 1.68MHz peak also lower by 2dB, also as we expect.
Next we wired the Amp2 power monitor PD to an EPICS channel with a faster data rate; we used the old High Power Laser (HPL) channel from the HPO days, which was still alive and well (and already had a fitting name). We ran a cable to the Lemo tee used to wire the PD to the PSL Sensor Interface Chassis, and plugged this cable into the HPL DC port of the PSL Monitoring Fieldbox. We then recalibrated this channel to match the power reading from PSL Beckhoff, this required a change in the channel gain from 0.0160515 to 0.047732. The full channel name is H1:PSL-PWR_HPL_DC_OUT_DQ and the data rate is 16kHz.
Next we checked the ISS AOM RF connection after a suggestion from Tony (he said they had seen weird AOM glitches in PCal in the past, and when this happened they found the RF cable had managed to work itself loose by upwards of half a turn). The ISS was turned OFF, the ISS AOM driver was turned OFF, the PMC unlocked, and the PSL external shutter closed. Using an SMA torque wrench I attempted to tighten the SMA connection to the ISS AOM; I found it loose by ~3/4 of a turn. I hadn't considered checking this connection before, but after this we will check all of our SMA connections regularly to make sure they are tight (PSL EOM, ISS AOM, LO and PD inputs on the TTFSS box). At this time I also noted new component locations on the PSL table as a result of the NPRO swap (FI replacement, new mode matching solution, etc.) so I can update the As-Built PSL table drawing. We also took the opportunity to re-center the Bullseye PD (uses the leakage beam from M10, the first mirror after the ISS AOM), as we had forgot to do this at the end of the NPRO swap.
We then removed the in-service TTFSS box, SN LHO01, for testing/characterization in the EE shop. Once Fil turned off the high voltage we removed the TTFSS box and took it to the lab, where Marc started testing the individual signal paths. Both the Common and PZT paths of the TTFSS looked as expected based on Marc's modeling of the TTFSS circuit, but the higher frequency portion of the EOM path looked completely wrong. Marc has all of this data that he can post once he has it in a readable format. The EOM path features 2 op amps in parallel that drive the EOM, an AD829 and a PA85. Marc checked the AD829 path and all looked good there, so it looked like the PA85 had gone dodgy. I recall in the past when the PA85 blew the FSS either went completely unstable with wall-to-wall oscillations in the PZT path that could not be cured (as noted the last time a PA85 blew in 2019) or would not lock the RefCav at all; this is completely different behavior than what we have been seeing, so we did not suspect the PA85 until Marc took a TF of the TTFSS EOM path. We then began a hunt for a spare PA85 and eventually found 3 of them. We swapped the dodgy PA85 with one of the spares (we now have 2 left) and Marc took the EOM path TF again; now all looked as expected based on the models, so Marc continued characterizing the in-service TTFSS.
While Marc continued characterizing TTFSS SN LHO01, we took the Rev B spare unit, SN LHO03, and installed it to see if it could lock the RefCav. Recall that the last time this unit was tested the RefCav would lock but had very large PZT oscillations that never went away; Marc subsequently found multiple resistors that were the wrong values (as per the circuit documentation), which would cause gain issues and likely instability in the control loop. Now that Marc had fixed these issues we wanted to test the unit again. With Fil helping with the high voltage we installed unit LHO03 and attempted to lock the RefCav. The cavity immediately locked, and immediately begain large oscillations in the PZT. However, unlike the last test, after about 2 minutes or so the oscillations quit and the RefCav seemed mostly happy (we had to pause the FSS guardian so it would stop yanking the gains around, and we set them at -10dB and waited). We did have one moment where the RefCav lost lock for no apparent reason, and once it relocked it had the PZT oscillations again. This time they did not clear on their own, so we unlocked the and relocked the cavity. The oscillations started again, but this time they did clear after about 2 minutes with the gains at -10dB. Now with a seemingly happily locked RefCav, we took a TF to set the UGF and measured the crossover. The 5th attachment shows the TF with a UGF of ~439kHz; to get this we needed to raise the Common gain from 13dB to 22dB, so this box apparently still has lower gain than SN LHO01. The 6th attachment shows the crossover measurement for setting the Fast gain, this picture taken with a Fast gain of 10dB. We let it sit and watched for several minutes and the RefCav stayed locked and the loop didn't go into oscillation again. So it seems we have a mostly-working spare unit now, with the caveat that the gains still aren't the same and the unit takes some time settle out once the RefCav is locked. The test mostly successful, we removed unit LHO03 and went back to the EE shop.
At this point Marc had finished characterizing unit LHO01, and indeed he found that the unit had more gain than LHO03. The PZT path was mostly the same, with a couple dB more gain in unit LHO01 vs LHO03, but the EOM path at lower frequencies was roughly 20dB different (with unit LHO01 having more gain than LHO03). This fit with our observations from our locking test with unit LHO03. Since we wanted to test if unit LHO01, with the now-fixed PA85, would be better behaved, we tuned the notches in the unit. The EOM notch was moved from ~1.77MHz to ~1.68MHz, and we moved the PZT notch from ~36kHz (where we found it) to ~32.7kHz (to hopefully squash the new peak we see at ~32.7kHz). This done we took unit LHO01 out to the enclosure and reinstalled it, with Fil once again helping out with the high voltage power supply. The RefCav locked almost immediately and had no PZT oscillations upon locking, so we took a TF to set the UGF and measured the crossover. The 7th attachment shows the TF with a UGF of ~458kHz and a phase margin of 65.4°. The peak previously seen at ~1.68MHz is now completely gone, so the EOM notch tuning was successful. There is still that feature around 750kHz, but that has been there and not problematic since April 2015, when notches targeting it were removed. The peakiness previously seen around 500kHz seemed less than observed back on Nov 1st, so we left the Common gain at 15dB (we can always lower it by a dB or two should we suspect the gain increase is causing instability, with the knowledge this will lower the UGF significantly (2dB causes the UGF to be around 320kHz)). Looking at the crossover (final attachment), we decided to leave it at 5dB as the hump around 20kHz looked better here. It should be noted, however, that we did not see much change in the hump with small, 1dB changes in the Fast gain; we didn't start seeing a clear peak at the crossover until the Fast gain got to around 7dB+. It should also be noted that the peak at ~32.7kHz is still present. This indicates either our notch isn't deep enough to remove the peak, or the peak is from something other than a PZT resonance. At this point I suspect it's something other than a PZT resonance, as we see a 2nd and 3rd harmonic of this peak in the IN1 spectrum (2nd harmonic at ~65.5kHz, 3rd at ~98kHz); I realized after we had left the LVEA that I didn't get a picture of the IN1 spectrum at the full span of the SR785 that shows these harmonics. This done, we cleaned up and left the enclosure.
Back in the Control Room we relocked the ISS and Tony relocked the IMC. There was some IMC instability seen, but we chalked it up to ground motion from a recent earthquake. Marc mentioned that moving the PZT notch lowered the overall gain in the PZT path slightly, so we raised the Fast gain to 6dB, since we had seen a 1dB gain change not have a large effect on the crossover. Eventually, once the environment had calmed down (Corey was on the Eve shift at this point), the relocking process began. The IMC lost lock during expected points in the initial alignment process, so we weren't too worried about it. However, there was one IMC lockloss during the relock (I think during Find IR?), and it took the IMC many minutes to relock. There were no PSL oscillations or glitches going on while we were waiting for the IMC to relock, so not sure what the issue was; it completely refused to grab lock, and when it would look like it was going to grab it the IMC Trans image swung away quickly. Elenna did question why MC2 was apparently swinging around so much (the IMC Trans camera image was moving all over the place, hinting at MC2 being pushed hard), so this behavior was not exactly normal. Eventually the IMC relocked and the locking process continued. At this point we called it a day, hoping that things would be more stable overnight. Turns out they were not, so we're still not sure where the problem is. Investigations will continue.
This closes WP12187.
This lockloss (1415595557, 2024-11-13 20:58:59 PT) looks related to IMC (/FSS /ISS / PSL)... 10-15 seconds before LL, FSS starts to see some glitches, then more glitches starting ~5 sec before LL. ~30 second trends here.
Then zooming in ~100ms before LL (plot), maybe a small glitch is seen on ISS second loop PDs, and then the IMC loses lock. At the same time that the IMC loses lock, then lockloss pulse and AS port and LSC_REFL see the lock loss. So it seems like an IMC lockloss with some FSS glitching beforehand.