[JimW, Jenne, Evan, DanH]
Relocking after we took the IFO down for some Cal measurements about 7 hours ago has been pretty painful. However, we're back up. We were back up, before DHARD Pit kicked us out.
Our biggest problem was with some glitches that we think are in one of the mode cleaner mirrors. They went away after a few hours though, so we aren't really clear on what the problem is. We see them in MC2 MasterOut channels, so the glitches do exist in the digital system. But, loops are closed everywhere, so it's tricky to tell where they originate. While these glitches were present, we couldn't lock ALS for more than a few minutes at a time. We would lose the lock before we were able to lock the DRMI, much less transition off of ALS. The 2nd and 3rd attachments show quiet and noisy times, respectively.
Later, we lost lock a few times at Switch_to_QPDs. Last month, Sheila and I kept adjusting the TRAMP on the reduction of the CARM offset that happens in that state, so I changed it again tonight. I still don't have any particular understanding of why the speed at which we reduce the CARM offset matters so much. For a while, we had been making it longer and longer (up to a Tramp of 60 sec), and then eventually reverted to the original 10 second ramp time. Tonight, I put in the guardian that it should do this ramp over 30 seconds, and it's worked a few times in a row.
Next up was a set of weird PRCL glitches. The attached lockloss screenshot shows them appearing in PRCL out, and the effect they have on the power recycling cavity's 9MHz buildup. Some time around here, although I don't remember the exact order of operations (Evan, do you remember?) Evan reverted the 9MHz RF source from the IFR (which he installed during the calibration break earlier in the evening) back to the OCXO. I don't know if we have a way to tell, other than someone's memory of the time, whether we were on the IFR or the OCXO at the time of this PRCL weirdness. Anyhow, I think this only caused one lockloss, and it doesn't seem to be there right now.
We have been seeing DHARD pitch oscillations during the CARM offset reduction for the past few days now. Sheila and I were going to do some loop measurements last night to see why the loop was so unstable, but last night had its own troubles. Anyhow, Evan commented out the turn-on of DHARD pitch FM1, "boostLTE". This boost increases the gain below 1 Hz, but eats about 70 degrees of phase at 1 Hz. With this commented out, we were finally able to get to DC Readout. After we were sitting on Nominal Low Noise for a while, DHARD Pit started oscillating. We saw it in the ASAIR camera, and in the ASC control signals. My current hypothesis (without doing any measurements...) is that the IFO isn't stable enough for us to handle this semi-delicate loop before the rest of the ASC comes on. So, I have added FM1 of DHARD Pit back to the guardian, but I put it at the very end of Engage_ASC_Part3 so that it comes on last. Since it didn't ring up for more than ~30 minutes, I think delaying the boost turn-on should be okay.
The last sticking point didn't cause a lockloss, but it did prevent the guardian from getting to Nominal low noise on its own: The IMC guardian got stuck trying to turn on the ISS. The ISS guardian has some logic that if the absolute value of the 1-second average of some number (PSL-ISS_SECONDLOOP_SIGNAL_OUT16) is above 2.5, don't turn on the ISS. If this happens, it returns to the Locked state, but the ISC_Lock guardian is still requesting ISS_ON, so I think it should go back to trying to preparing the ISS, and rechecking this number. Instead though, it just hangs at the Locked state. I tried hand-requesting ISS_ON a few times, with the same result. I don't know what this channel means, and I don't know how conservatively that threshold was set, but to try to get the lock to complete, I temporarily changed the threshold to be 3.5. (After we got past this state, I put it back to 2.5 and reloaded the guardian, so next lock it will be back to the original threshold). What is this signal (it's totally unclear from the screens, and I haven't opened up the model to see if I can figure it out)? What are the consequences of increasing this threshold? Why is the guardian getting stuck? I think we need to re-examine this guardian logic.
.
.
