Summary: 

After swapping a few electronics that we broke while investigating the ALS glitches at EY, the glitches are still there, but not nearly as bad as they were.  We have had these glitches go away on their own before when nothing was done, so that might be what is happening.  Now they are at End X, and about as bad as they were at End Y for most of the week.  ALS still will not lock for longer than 10 minutes.  It might be that this problem will go away on its own, but will probably come back again.  

• After seeing that the behavior of the VCO at end Y tune mon changed around 6:30 UTC June 2nd, and based on Keita's alog last night, I swapped it for a spare I found in the EE shop this morning.  The unit I removed is S1200566, the unit I replaced it with is S12000562.  The VCO is no longer railing and the tune mon is near zero when the tune input is terminated.  The tune monitor BNC however has about 900 mV output even with the tune input terminated.  After this the arm would lock and I was not getting any VCO errors
• I found that the Seismic configuration was left in Large earthquake mode since Friday, and changed it to windy.  I found this because the arm transmitted powers were fluctuating more than normal for both arms, and also the oplevs showed extra noise since Friday evening.  Now the arms are both locked and quiet.
• At 1:08-1:23 local time, I started 15 minutes of locking the Y arm in green without automation (the PLL automation is on, not the PDH automation).  At 1:32-1:47 I did another 15 minute test with the only difference being that the PDH CM board was controlled by the tester, not the automation. At 1:50 I locked the arm using the automation, in the normal way. During all of these tests, there are a few glitches, but not as many and not as large as we were seeing mid week and last weekend. 
• I attempted to lock ALS, which didn't hold for more than a few minutes.  I then noticed that we are having the type of glitches we had been struggling with in the Y arm in the X arm now. (1st attached screenshot shows the Y arm and X arm both locked in the normal configuration with automation).
• I went to end X, and put the cable for the PDH signal into IN1 of the PLL board.  The X arm locked for green with in 1 polarity negative, and -4 dB of gain.  (For Y arm, we used positive polarity and 6 dB of gain).  No glitches in this configuration. 
• I also swapped the cables to substitute the spare FET demodulator for the PFD.  I added a 20dB attentuator to the RF cable into the demod, with this in I got a ugf of 27 kHz for an in1 gain of 13 dB. The autolocker works fine to lock this, you just have to force it to ignore the error message from the PFD by hitting the "Force" button. The arm was locked this way for 15 minutes starting at 0:03:30 UTC June 5th for about 10 minutes, the glitches were just as bad as the normal configuration. 
• I used Keita's set up from End Y to use the CM board tester at End X.  The cavity was locked this way from 0:47 UTC to 0:54 UTC June 5th, glitches are just as bad as the normal configuration.  

Things that are not the cause of glitches at EX:

From PLL bypass test:

• PLL servo board
• PLL servo board controls
• Laser
• PDH sensing, including diode, demod, and LO
• PLL automation

Other things that are not the problem:

• PFD (glitches still seen when it was swapped for a mixer)
• PDH servo board control cables swapped (so PDH control cables and automation are not the problem)
• loop instabilities (X end PLL UGF is 24 KHz with 47 degrees of phase margin, PDH Ugf is 5.6kHz with 56 degrees of phase margin).
• The beatnote is a clean sine wave, there are 8MHz sidebands on it but they are 40dB smaller than the carrier. 

Things that we haven't checked:

• PDH servo board, but this is not likely to be the problem since it wasn't at End Y and swapping it has some risks.
• VCO, we could try swapping this for an IFR but I wasn't able to find one in the shop today. 
• BBPD
• A problem on the light coming out of the fiber. it still seems like glitches on the PSL are unlikely because the glitches are not at the same time in the two end stations. 

I recommend that the first thing in the morning operators take the ISC lock guardian to locking arms green, wait there for 10-20 minutes, and look at second trends of the arm transmissions.  You can compare them to the screenshot I've attached here, if the glitches are gone or much smaller it would be worth doing an initial alignment and trying to lock to DC readout, if the glitches are still there it is not worth trying. 

CP3 log file DOES NOT exist!
CP4 Fill completed in 258 seconds. Starting CP4 fill. LLCV enabled. LLCV set to manual control. LLCV set to 70% open. Fill completed in 258 seconds. TC A did not register fill. LLCV set back to 41.0% open.

1. When I came in, Y VCO "tune ofst" was railing at -5V ("tune mon" 8.3V or so). As such, the VCO frequency was higher than 80 MHz (set frequency is 78.92MHz) and the beat note somewhat above 40MHz. Manually resetting it, "frequency servo" brought it back. This is will PLL only, no PDH. I thought that the "tune mon" is the readback of the VCO "tune" input on the front panel, but it was 8.3V even when I terminated the BNC. Apparently I forgot these things.

2. At the end station, I reseated the two slow controls cables for PDH CM board at Beckhoff chassis end, and locked the arm green, no change in glitches.

3. Connected the CM tester to refl CM to lock the arm but failed. I think the tester setting was good but maybe some automation was doing something funny though I disabled the state machine in the ALS overview. Refl CM was put back on slow controls cables.

4. Switched the PFD channel, locked the arm,  glitches persist. Switched it back.

5. Time's up for the day.

Keita arrived in the control and noticed vacuum alarms relating to the Vacuum gauge at CP2, PT134B, and the Corner Station instrument air ??199.  PT134B is currently off and shows up as "RED" MEDM field on the vacuum screens.  This has been discussed in entries made previously and is being pursued by Gerardo M., Chandra R. and Richard M..  The instrument air alarm may be a stale alarm from when before Bubba replaced the GV8 instrument air supply pressure regulator.  If so, it could be acknowledged and should not re-occur.  There still may be a "low pressure" condition which would result in a new instrument air alarm.  

Control room folks - 

1. Ignore PT134B alarm, 
 
2. Acknowledge ??199 instrument air alarm and treat new occurrences of this alarm as valid.  

We are monitoring remotely periodically.  

Remote logged in to toggle PT-134, CC was off when I logged in.

I turned on and off the CC several time to try to get it to get on scale, but no luck.  Left the CC on, but it is not on scale, sometimes they take long to do so.

CP3 log file DOES NOT exist!
CP4 Fill completed in 298 seconds. Starting CP4 fill. LLCV enabled. LLCV set to manual control. LLCV set to 70% open. Fill completed in 298 seconds. TC A did not register fill. LLCV set back to 41.0% open.

Bubba, Kyle

Fixing Corner Station instrument air.  LVEA and chiller yard.  Will make comment to this entry when leaving site.

here is a minute trend (mean) plot of CP3's pump level and LLCV position over the past 24 hours showing good control of the LN2 level at the 92% set point. Reminder that only CP4 will autofill in 20 minutes time.

TITLE: 06/03 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Planned Engineering
INCOMING OPERATOR: None
SHIFT SUMMARY: I tried to lock ALS a few times after commissioners left to no success. Y would complain of VCO problems and would not hold the a lock longer than a few tens of seconds. No one relieving me, and no day operator tomorrow.
LOG:

• 02:22 Kyle to MY

1925 hrs. local 

I found the rotometer ball off of the stop about 1/2 the diameter of the ball (< 0.4 LPM the smallest graduation of the flow scale).  When the isolation valve at the plumbing manifold was closed, this small flow stopped, i.e. the ball rested.  This was repeatable.  This could be flow across the sensing line blockage or it could also be leakage flow out of the 4 or 5 pipe joints which can't be isolated.  Anyway, this wasn't occurring the last time I inspected it.  The UHP N2 bottle is at 1800 psi tonight, down 300 psi from 24 hrs. ago.  

While there, I repeated the exercise of switching between pumping the sensing line with a small vacuum pump then switching to pressurizing with the UHP N2.  I had made a few attempts at this yesterday? the day before? but could never establish any flow.  Tonight, I was able to achieve flow for 10s of seconds at a time. I would have like to have continued this as it is so encouraging but I don't want my children to grow up without knowing their daddy + plus I'm hungry plus .....

I am leaving the apparatus at 60 psi regulator output and adjusted for 2 LPM max.

Fil, Daniel, Jenne, Keita, Patrick, Richard, Sheila, Kiwamu,

The investigation continues. No solid conclusion or resolution yet.

• We tried locking the laser to the Y arm by only using the PDH servo board in order to test the PDH board.
  • The output of the PDH board was connected to the cable which is nominally connected to the output of the PLL board for this test operation.
  • The same filter settings as usual but with a lower gain (I had to lower the fast path gain as well).
  • This locked the laser somewhat but kept unlocking roughly every 10 sec. Note that this behavior is different from the glitches we have seen recently.
  • It is inconclusive to say whether the PDH board is to blame for the glitches.
• Regardless of whether the PDH board is the culprit, we decided to swap the board.
  • See 36622.
  • The first replacement unit turned out to be not working (non funzione in Italian) because it didn't responds to changes in the remote offset adjusters.
  • Fil then installed another spare unit with which we were able to control the electronic offset using the MEDM sliders.
• We then found that the FET demodulator gave us a big offset.
  • We drove to the end station and then determined that the demodulator unit was non funzione either. The I-signal was just flat 5 V.
  • We switched the system to the second half of the same demodulator unit.
• Now we are waiting for the Alaskan earthquake (LLO 34072) to settle so that we can test out the new PDH board.

TITLE: 06/02 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Planned Engineering
OUTGOING OPERATOR: Ed
CURRENT ENVIRONMENT:
    Wind: 7mph Gusts, 4mph 5min avg
    Primary useism: 3.58 μm/s
    Secondary useism: 0.87 μm/s
QUICK SUMMARY: ALS work continues. 6.9M earthquake off of Alaska will be hindering us for a bit.

TITLE: 06/02 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Planned Engineering
INCOMING OPERATOR: TJ
SHIFT SUMMARY:
LOG:

15:00 Sheila to EY to investigate ALSY PLL.

15:15 sensor correction turned of at EY

15:46 Jim doing TFs on HAM5

16:00 Sheila back

16:05 SC turned back on at EY

16:11 Granted Aiden remote access

16:14 Richard transitioning LVEA to LASER SAFE

16:31 SC back OFF on EY - Kiwamu heading back out

16:54 Betsy in LVEA to look through TCS cabinets

16:57 Ken into LVEA to look for a ladder

17:07 Ken out

17:10 Richard and Kiwamu restarting ISC EY computers

17:21 Betsy out. Heading to the MX in 15 minutes.

17:38 Richard back from EY

17:58 Bubba into the LVEA

18:38 Kiwamu back

18:44 Gerardo going into LVEA

18:54 Gerardo out

19:18 Fil going to EY to switch out a common mode servo chassis for ALS

20:02 Fil back

20:46 Fil goin back to end station. Newly installed chassis not responding

20:46 Peter into the optics lab

22:00 Crew back at EY working on ALS

22:30 6.9mag EQ in Alaska 30um/s

The faulty regulator which turned out to be only a bad o-ring in the regulator itself, has been repaired, reinstalled, and air is flowing again. I unlocked the valves with Chandra watching the vacuum screen in the control room.
While I was repairing the regulator, Kyle was ordering a couple of new regulators for spares. 

Chandra and I also replaced the check valves on the corner station instrument air compressor.

CP3's reservoir level is back on scale - reading around 97% full. I lowered the PID lower limit of LLCV to 15% open when it was ~99% full to measure percentage per hour to get a feel for how low the reservoir got over the weekend.

PID is programmed to hold the reservoir at 92% full. Once it drops to that level, PI will kick in.