Patrick told me that the ISS diffracted power was swinging around more than usual. Indeed the diffraction
minimum and diffraction maximum values were more than usual even with the second loop disabled. After a
quick survey of the ISS MEDM screen, I saw that the calculated diffracted power was inconsistent with the
AOM drive voltage. At one point I saw that the reported diffracted power was ~15% for a reported AOM
drive of less than 0.2 V, which I know to be false.
Some time around 7:17 - 7:18 UTC the output of the AOM driver suddenly dropped. I had a quick look
at some other signals but didn't find anything that matched
I am wondering if the ground reference voltage of the PSL rack has changed relative to the PSL table
surface? There does not appear to be any coincidence with the LVEA temperature, so I cannot think of a
reason why it would change all of a sudden.
For the meanwhile, it is probably worth trying to keep the AOM drive to be ~0.5 V. One does this by adjusting the first loop ISS reference signal (bottom left hand corner slider of the first loop ISS MEDM screen). The adjustment should be done with the second loop ISS off.
Plots of the ISS reference signal and its monitor are attached. The reference signal comes from a DAC. The monitor is read off the ISS board from an OP-27. One thing to note is that the monitor output is somewhat noisier than the reference signal. This might be related to the grounding issue (if there is indeed one).
Problem was tracked down to the loss of -18V in PSL ISS AA chassis. Power cycling seems to have fixed it.
I was called and came to the site at 6PM to work with Peter who was also called earlier.
We have found that some of the ISS 1st loop signals were much smaller than they should be, starting 23:20 local time yesterday. PDA readback was only 0.7V or so when it should be 3.2, PDB was also too small, and ISS reference voltage readback was only -0.1 when it was set to -0.48.
We were first suspicious about analog failure of the ISS chassis, but when we went to the floor, everything looked normal in analog land. We measured the OLTF of ISS 1st loop and the UGF was about 38kHz.
Voltages on one of the DB9 that goes to the AA looked correct.
Then we went to the PSL ISS AA and found that LEDs for negative voltage was off, and the positive voltage LED was dim (will attach picture later). Peter power cycled it and everything seems to be back to normal.
Things to do for Tuesday:
Check PSL ISS AA chassis power board.
Things unrelated to this but are troubling:
We also went to the power supply area in mezzanine, and things look OK except that MANY power supplies have faulty power indicator lamps, some of them were flickering on and off.
Also, oplev power supply has a label saying +18V, but the actual voltage is more like +10V.
The reason why the readback failure matters is because we are now using the 1st loop PDA and PDB values for the power normalization of 2nd loop digital AC coupling.
With the AA chassis failing, H1:PSL-ISS_SECONDLOOP_INPUT_NORM hit the limitter bottom and no power normalization was done. Withrout a proper power normalization this AC coupling is known to cause loss of lock.
Fisrt picture: LEDs of the front panel with negative power failue (top chassis).
Second picture: LEDs of the back panel with negative power failue.
Third picture: UGF of ISS 1st loop.
