TJ, Nutsinee

We went in and move top+bottom periscope to get ride of the reflection. Here's what it looks like now:

HWSX with ETM

HWSX without ETM

HWSX with ETM and ITM

Alignment looks good. How much did you guys have to move these mirrors?

I moved the top periscope at least one whole turn (probably something like 1 turn and a half) to get rid of the dim shape of the test mass. Then I moved the bottom periscope to bring back the return SLED beam.

The Hartmann plates are back on and the codes are running.

0900 hrs. local 
8.9% full @ 0.8 psi exhaust.  PT243 1.8 x 10-9 torr  

This is my last update for this aLOG entry - on my way to LHO

2302 hrs. local 
CP3's level indication is 4.7% full with 0.5 psi exhaust and no change to PT243. 

Midnight:  3.6% and 0.4 psi exhaust lressure. No change on PT-243 (2.0e-9 Torr). Exhaust temps are both 21C.

5/30/2017 0105 hrs. local 
5.2% full, 0.5 psi exhaust pt243 @ 1.98 x 10-9 torr

2 am:  5.9% and 0.5 psi exhaust pressure. PT-243 = 1.94e-9 Torr.

0300 hrs. local 
5.4% full @ 0.6 psi and 1.92 x 10-9 torr

4am
5.6% full @ 0.5 psi and 1.89 x 10-9 torr

0500 hrs. local 
6.1% full @ 0.8 psi exhaust.  PT243 1.87 x 10-9 torr

6am
7.1% full @ 0.7 psi exhaust.  PT243 1.84 x 10-9 torr

0700 hrs. local 
7.7% full @ 0.9 psi exhaust.  PT243 1.83 x 10-9 torr 

We are using a T-size GN2 cylinder which is typically filled with 300cubic*ft of GN2.  So, (300cubic*ft)(28L/cubic*ft)(min/2L)(hour/60min)(day/24hour) = 2-3 days of flow

Kyle and I are headed to CP3 to see if we can blow out the newly forming passage.

Here is a trend. Looks like it started coming on scale around midnight local time. Watching it live from control room, it's fluctuating between 0.1 and 7.2 from the little bit I watched it. Heading to mid-Y now to set up N2 bottle. We'll apply 100 psi to sensing line.

Based on this log entry from last June 24, it took 35 minutes to overfill CP4 until LN2 poured out the exhaust. This was before CP4 clog - we were experimenting with durations and flow rates to create a work-around for CP3.

https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=27950

Tonight's real life scenario has been my nightmare for the past 18 months (since ice plugs have required manual filling of CP3 and CP4).  It happened on my watch and I am responsible for it.  In my defense, this was not the result of inattention or a false sense of security on my part.  I had lowered CP3's manual LLCV %open value to 15% open, down from 17% open, in response to Friday's automated fill having only taken 17 seconds.  This would have been an appropriate response, perhaps, for springtime ambient temperature conditions but proved too much of a reduction/correction for this weekend's warmest-of-the-year weather.  

I look at the vacuum site overview screen multiple times on non-work days and am quite familiar with what the "normal" values are.  Today was no different. At around 07:30 pm local time, I looked and noticed that PT243 was 3.97 x 10-9 torr which is higher than normal and caught my attention.  I reasoned that this was probably hydrogen emitting from the BT steel on this "hot" day but was concerned enough that I resolved to check it again before going to bed.  At approx. 10:30 pm local time, I looked and saw that PT243 had fallen to 2.?? x 10-9 torr.  Minutes later, I checked my phone before going to bed and became aware of a text thread between Chandra R. and Gerardo M. which had been in progress for the previous 30 minutes.  So, the reduction in PT243 at 10:30 pm was the result of the fact that Chandra was already on site and had started filling CP3 manually via the opening the LLCV bypass valve.  

Had Sheila D. not contacted Chandra at approx. 09:50 pm and Chandra not responded by doing a manual fill, the pressure shown by PT243 10:30 pm would have been much higher than the previuosly "concerning" value seen at ~07:30 and I feel that I would have responded appropriately.  Still, this didn't have to happen.  As Chandra reminded me, pressure trends are available (new location) for remote viewing and, had I reviewed these in addition to the Vacuum Site Overview, I would have noticed that the Y-mid values were increasing independent of the rest of the site pressures.  This would have dispelled my "hydrogen" theory at 07:30 and I would have done a manual refill then.  

Kyle, you shouldn't feel responsible. I'm usually the one who manipulates the LLCV settings based on temperature fluctuations and fullness of Dewar and have a better feel for adjustments. Sorry I didn't explain that better before I left. We can start to think about the next level of automation on this system which would increase/decrease valve setting based on how long it took to fill the previous time. Folks should also recognize that the work we're proposing to do post O2 by either decommissioning and/or regenerating these CPs will eliminate these risks.

Also, I doubt PT-170 alarming was actually a crazy coincidence. I forgot to trend its pressure but am guessing it was also starting to increase due to loss of cryopump action at MY. And because its pressure was already high from vent, it alerted us before we had to wait for -8 torr range alarms in arm. Thank goodness Sheila was in the lab at the time to catch it!

Good save, I should have thought of this. The dominant boiloff load is (should be) blackbody radiation from the tube, which is at BTE ambient temperature. I will make a number for the fractional effect on liquid mass flow per degree, so we can add that % on to our "open-loop estimate". 

EDIT: see post 36496

Worth noting, though, the high ambient (BTE ) temp is raising the hydrogen diffusion flux, which doubles every 6 C or so (harmlessly, as long as we have ion pumps). So the pressure trend (and particularly, any attribution to the CP) has to be interpreted carefully.  

Even before today, the ice plugs gave me nightmares. We have to fix them, and stop any more from happening. 

Dave Barker suggested increasing to daily auto overfills rather than Mon-Wed-Fri. I like this idea. We'll discuss with vacuum group this week.

The cell phone alarm system currently monitors the vacuum gauge pairs at the ends of the 2km beam tube sections. For MY those are PT243 (closest to corner) and PT246 (closest to EX). I can certainly add all the other gauges in MY (PT244, PT245, PT210) to the system if needed.

I've created a remote access permit for the vacuum group, good through the end of the year.

No cell phone alarms were raised for this event, their upper alarm range is 5.0e-08 torr, an order of magnitude higher than what MY saw Sunday night (trend attached).

Gerardo was able to remotely log into CDS from home, he had a permit open. He was unable to directly log into the vacuum1 machine to make vacuum changes due to recent ssh cert changed. I recommend that every week the vacuum group test remote log into vacuum1 to verify this is possible.

More clues....or confusion. MidY IP9 current plotted with PT-210 pressure increase from Sunday evening. Strange behavior in IP.

Trended PT-170 over the weekend to understand the verbal alarm Sheila heard in control room. PT-170 pressure has steadily been falling since the vent, but at about 9:30pm local time on Sunday, it was crossing over the 7e-8 Torr alarm threshold causing the verbal alarm. What luck!

Outside temperature plotted over 30 days along with PT-243 at mid-Y.