Summary: A few odd mysteries, some very bad alignments, and a crashed ITMY camera were cause for a ~5 hour lock recovery. Ten minutes later we had anther lock loss, reason unknow as of yet.

#1

• Immediately after the lock loss the green arms had little to no light. This has been a recent trend lately that we are not sure of the reasons yet. Increase flashes did a great job getting it back to a good power, BUT there was an issue when it started to engage the WFS. Tracked it down to the PZT servo tiggering not turned on yet for the Y arm. I.F. should not have started if this was off, but there was no check for this in the FAULT state of the ALS_YARM node.
• Jeff found a new alignment after the triggering was back on and then the autocentering would turn off with the WFS engaged. It is still unknown as to why.
• The DIFF beatnote was low this entire time, and was suspect (it was around -25dBm). Sheila said that if an initial alignment doesn't fix it that would should pico it.
• Initial alignment:
  • Initial alignment for the green arms would pull itself away over time. We, eventually, tracked this down to the ITMY green digital camera being frozen. Restarting this via monit did not work. I had to ssh into the machine and run the restart command manually (located in /etc/init.d/  then {camera_name restart}). This camera had been dead since yesterday at ~19:45 UTC, but was not noticed until we were in Observing. The note to restart this was forgotten/lost/misinterpreted as it goes here sometimes (something we need to look into more).
  • PRC alignment was far off. Jeff and Sheila had to move PRM to to increase the flashes before it would lock. Sheila then had to increase the threshold for PRX in guardian because it was locking too soon.
  • During SRC align there was a lock loss and since we werw at 10W the IMC would not relock. This took a handful of minutes to remember and then bring the power back down and then back up. (Need to add a check in the IMC_LOCK node)
• Once initial alingment finished we didn't have any issues and ISC_LOCK brought us all the way back up without any intervention.

#2

• Because I was unable to look into and fix the issue before the lock loss, the same thing happened where the ALS alignment servo triggering was not on when increase flashes started. Sheila and I lowered the trigger threshold to allow the guardian to move on, but perhaps we should just leave it here. We will investgate why this is differenent as of late.
• A lock loss at find IR promted Sheila to adjust the DIFF beatnote.
• Made it all the way.

Some important action items from this:

• The FAULT state in the ALS nodes needs to check for the PZT servo triggering to be enabled before leaving FAULT.
• Why is the ALSY green QPD B sum so low for these last two locks?
• [done] Make a way to gracefully get out of Increase_Flashes
• Table alignment to get the DIFF beatnote up
• Need some alert if critical cameras are frozen (Dave and I are working on a solution)
• IMC_LOCK should warn if we are at higher power and trying to lock