Jamie, Sheila,
Each time we run an initial alignment, it seems that we loose lock in the early stages of the CARM offset reduction in the the next locking attempt, although these steps are otherwise quite reliable, so it seems as though some setting is not being set correctly after initial alignment is run until the IFO looses lock once.
After we run initial alignment, we do run the down state of ISC_LOCK, which requests 'DOWN' from several of its subordinate guardians, however, since some of these guardians are already in the DOWN state making the request doesn't cause them to re-run their main states, where the settings get reset. (For a similar situation and a different solution see 49520 and comments). Yesterday I tried to look manually at the ALS_COMM and IMC_LOCK down states, but catch the problem with my first look. Patrick also tried to use SDF to find the probelm setting yesterday, 51452 but that didn't show us anything obviously wrong either.
On the phone Jamie told me that when the guardian code was changed so that re-requesting a state doesn't cause the state to be re-run (which was a change that solved problems with having code that isn't idempotent), he added a feature that would allow us to choose the old behavoir for certain states, by adding an edge from that state to itself. In that case if the guardian is in DOWN and gets a request for DOWN it will re-run its main state.
So, I've added paths from DOWN to DOWN in ALS_COM, ALS_DIFF, ISC_DRMI, IMC_LOCK, FAST_SHUTTER, and OMC_LOCK. IMC_LOCK doesn't get sent to DOWN in the DOWN state of ISC_LOCK, so that if the mode cleaner is already locked after initial alignment runs we don't unlock it, and I haven't changed that, I simply added the edge from DOWN to DOWN.
So, the next time that we have a chance, (when we are out of observing) I'd like to load all of these guardians. Then we can see if this will solve the problem of loosing lock on the first attempt after initial alignment.
Patrick loaded these changes after a lockloss, and didn't have guardian problems when relocking.
Request for operators: I am hoping that this might solve the problem of the IFO lossing lock in the early CARM reduction steps in the first locking attempt after each time we run initial alignment. I would appreciate if you would report relocking attempts over the weekend, including if you run initial alignment, and if on the first relocking attempt you make it to the CARM offset reduction do you survive those steps?
As a reminder, there is a spreadsheet which could be used to help us keep track of problems encountered when relocking:
https://docs.google.com/spreadsheets/d/1byXR-s2ATegciJCEjORb1FEZhZTp77j6DS-fZuvEY-8/edit#gid=0
In the first locking attempt after Tuesday maintence, initial alignment was run, and the interferometer was relocked without loosing lock, on the first attempt https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=51561 (Also confirmed by trending the guardian state at the times Jeff aloged)
This means that this solution has worked, and we shouldn't have these failed locking attempts after each initial alignment.