[Sheila, Jenne, Kiwamu, Patrick]

We lost lock about 8 hours ago, and haven't been able to lock since.  Sheila suggested we look at the Yarm ALS PDH control signal, since that has caused trouble in the past.  Indeed, we saw glitches.

We were looking at H1:ALS-Y_REFL_CTRL_OUT_DQ, and saw very obvious glitches at about a 1Hz rep rate.  At the same time, we see dips in the ALS transmitted power (H1:ALS-Y_TR_A_LF_OUT) down to 0.5 or lower from a nominal locked value of 1.0.  We also see much smaller glitches in the Xarm, often at the same time as the Yarm glitches, but not always. The Xarm power dips are also much smaller than those for the Yarm. The first attachment shows the REFL CTRL signals.  We don't save the transmitted powers, so I don't have a good plot of those. 

We have certainly seen these glitches before, and there are rumors that it seems to happen more often when it's raining.  Not totally clear how that's related - is the extra humidity doing something to us??  Anyhow, in the past the solution has always been to wait it out, and it'll go away.  Colloquially the problem is usually gone by the time anyone drives down to the end station. 

We went down to the end station and tried several things while looking at the PLL beatnote, but none of them changed the glitches.  At a few times, Sheila noted that there were peaks around the beatnote, 8MHz above and below.  We're not totally sure what those are. 

• I changed the temperature of the crystal, and Sheila could see the beatnote moving around, but saw no evidence of mode hopping in the laser.
  • To do this:  On the ALS_Overview screen in the Yarm state machine, select "unlocked". 
  • On the Yarm PLL screen, top left corner, select "off" for the servo.
  • Change the value in H1:ALS-Y_LASER_HEAD_CRYSTALFREQUENCY - this will move the temperature.
  • To go back to regular locking: On the ALS_Overview screen in the Yarm state machine, select "PLL locked".
• We toggled the ALS laser's noise eater, then relocked the green arm.  No change in the glitch behavior.
  • To do this:  From the Yarm PLL screen, top left corner go to ALS Laser sub-screen.  Toggle noise eater button off, then on again.
• We turned off the ALS laser, then turned it back on and relocked the green arm.  Again no change in the glitch behavior.

Suspecting that the PSL was somehow involved since we often see glitches simultaneously in the X and Y arms, we closed the light pipe shutter for the PSL ALS beam that goes to ISCT1.  We were hoping that if there were some kind of feedback from the Yarm to the PSL that was then getting sent to the Xarm through the PLL, this would prevent that.  However, no change.  The light pipe's shutter was re-opened.

We then went back to the Yend to lock the green arm, but bypassing the PLL, thus eliminating the PSL light from the system.  We didn't see any tell-tale dips in the transmitted power while we did this, but when we put the system back to normal, the problem seemed to have already resolved itself.  So, it's not a 100% conclusive test. 

• To do this, we unplugged the PDH common mode board's input (lower of the 2 common mode boards) and plugged it into the PLL common mode board's input (upper of the 2 common mode boards). 
• On the PLL servo board, we changed the Input1 polarity to positive (it's usually negative), and increased the gain to above 10dB.  (16dB worked for a while, but then it seemed like it was too much and causing locklosses).  The arm locked in green very easily.  If it unlocked, sometimes we toggled the polarity, and sometimes it seemed to acquire better if we ramped the gain down then back up again.
• We were locked this way from about 06:42 UTC to 06:50 UTC 14Oct2016. 
• Since the refl control signal wasn't in use, we just monitored the transmitted power and said that we didn't see any dips in the power.
• Assuming that the glitches would have still been there in the normal configuration, we think that this absolves the ALS laser and PDH sensor and PLL servo board from blame. 
• However, when we put things back to normal, the glitches were basically gone.  So, maybe we didn't learn much of anything :/

We're going back to trying to lock for now.