Kyle and I are headed to CP3 to see if we can blow out the newly forming passage.

Here is a trend. Looks like it started coming on scale around midnight local time. Watching it live from control room, it's fluctuating between 0.1 and 7.2 from the little bit I watched it. Heading to mid-Y now to set up N2 bottle. We'll apply 100 psi to sensing line.

I wonder what's cheaper: LN2 at 35 cents a gallon or electricity at 6 cents a kWh.

According to PSI V049-1-033 (aka LIGO-C960964, attachment 4, p. 98 ff) the short CP in “iced” surface condition (thermal emissivity ~ 0.9) absorbs 250W thermal blackbody radiation from each end aperture, assuming 80K reservoir and 300K beamtube or chamber blackbody temperature. Another 102W are expected to enter through conduction, supply line loss, and local shell radiation [1]. 

With other factors fixed (i.e., midstation climate-controlled,  pipe insulation good, etc.) the boiloff rate vs. variable beamtube temperature can be approximated as 

mdot ~ P/C 
    ~ [352W  + 250W*(T_bt/300K)^4 ] / C 
    ~ (602W + 1000W*[(T_bt-300K)/300K])/C

= 2.6 g/s @ T_bt = 0C, 3.0 g/s @ 27C, and 3.2 g/s @ 40C    (32, 81 and 104 F respectively)

where mdot is mass evaporation rate (g/s), T_bt is the beamtube temperature, P is absorbed thermal power (W), and C is LN2 heat of vaporization (~ 200 J/g).  

In other words, short CP LN2 consumption should increase by about 0.6 % per degree C of beamtube temperature [2]. 

We should install BTE thermometers as done at LLO. Just one or two per module would probably be adequate. 

[1] The PSI calc assumes heat shielding and reflective liners built into the CP have very low emissivity (< .05), which may not be true any more; if the tube liner finish or cleanliness is compromised due to age, radiant heat load could be 1.5x greater.

[2] At LLO naked tube steel now tracks BTE enclosure air temperature closely [G1400433]. BTE air temp is a function of outside ambient, sun exposure and the insulating and heat storage properties of the BTE concrete, but we think both sites hit at least 49C peak in summer. Diurnal variation is about 20C p-p.

Increased CP3's LLCV %open value to 20%, up from 18%.  Chandra was awarre that I was on site and suggested this value.  

Made good progress towards re-configuring VBOC but there is still much more to do.  

1540 hrs. local -> Kyle leaving site now.   Leaving Kiwamu and Sheila to their thoughts.

Based on this log entry from last June 24, it took 35 minutes to overfill CP4 until LN2 poured out the exhaust. This was before CP4 clog - we were experimenting with durations and flow rates to create a work-around for CP3.

https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=27950

Tonight's real life scenario has been my nightmare for the past 18 months (since ice plugs have required manual filling of CP3 and CP4).  It happened on my watch and I am responsible for it.  In my defense, this was not the result of inattention or a false sense of security on my part.  I had lowered CP3's manual LLCV %open value to 15% open, down from 17% open, in response to Friday's automated fill having only taken 17 seconds.  This would have been an appropriate response, perhaps, for springtime ambient temperature conditions but proved too much of a reduction/correction for this weekend's warmest-of-the-year weather.  

I look at the vacuum site overview screen multiple times on non-work days and am quite familiar with what the "normal" values are.  Today was no different. At around 07:30 pm local time, I looked and noticed that PT243 was 3.97 x 10-9 torr which is higher than normal and caught my attention.  I reasoned that this was probably hydrogen emitting from the BT steel on this "hot" day but was concerned enough that I resolved to check it again before going to bed.  At approx. 10:30 pm local time, I looked and saw that PT243 had fallen to 2.?? x 10-9 torr.  Minutes later, I checked my phone before going to bed and became aware of a text thread between Chandra R. and Gerardo M. which had been in progress for the previous 30 minutes.  So, the reduction in PT243 at 10:30 pm was the result of the fact that Chandra was already on site and had started filling CP3 manually via the opening the LLCV bypass valve.  

Had Sheila D. not contacted Chandra at approx. 09:50 pm and Chandra not responded by doing a manual fill, the pressure shown by PT243 10:30 pm would have been much higher than the previuosly "concerning" value seen at ~07:30 and I feel that I would have responded appropriately.  Still, this didn't have to happen.  As Chandra reminded me, pressure trends are available (new location) for remote viewing and, had I reviewed these in addition to the Vacuum Site Overview, I would have noticed that the Y-mid values were increasing independent of the rest of the site pressures.  This would have dispelled my "hydrogen" theory at 07:30 and I would have done a manual refill then.  

Kyle, you shouldn't feel responsible. I'm usually the one who manipulates the LLCV settings based on temperature fluctuations and fullness of Dewar and have a better feel for adjustments. Sorry I didn't explain that better before I left. We can start to think about the next level of automation on this system which would increase/decrease valve setting based on how long it took to fill the previous time. Folks should also recognize that the work we're proposing to do post O2 by either decommissioning and/or regenerating these CPs will eliminate these risks.

Also, I doubt PT-170 alarming was actually a crazy coincidence. I forgot to trend its pressure but am guessing it was also starting to increase due to loss of cryopump action at MY. And because its pressure was already high from vent, it alerted us before we had to wait for -8 torr range alarms in arm. Thank goodness Sheila was in the lab at the time to catch it!

Good save, I should have thought of this. The dominant boiloff load is (should be) blackbody radiation from the tube, which is at BTE ambient temperature. I will make a number for the fractional effect on liquid mass flow per degree, so we can add that % on to our "open-loop estimate". 

EDIT: see post 36496

Worth noting, though, the high ambient (BTE ) temp is raising the hydrogen diffusion flux, which doubles every 6 C or so (harmlessly, as long as we have ion pumps). So the pressure trend (and particularly, any attribution to the CP) has to be interpreted carefully.  

Even before today, the ice plugs gave me nightmares. We have to fix them, and stop any more from happening. 

Dave Barker suggested increasing to daily auto overfills rather than Mon-Wed-Fri. I like this idea. We'll discuss with vacuum group this week.

The cell phone alarm system currently monitors the vacuum gauge pairs at the ends of the 2km beam tube sections. For MY those are PT243 (closest to corner) and PT246 (closest to EX). I can certainly add all the other gauges in MY (PT244, PT245, PT210) to the system if needed.

I've created a remote access permit for the vacuum group, good through the end of the year.

No cell phone alarms were raised for this event, their upper alarm range is 5.0e-08 torr, an order of magnitude higher than what MY saw Sunday night (trend attached).

Gerardo was able to remotely log into CDS from home, he had a permit open. He was unable to directly log into the vacuum1 machine to make vacuum changes due to recent ssh cert changed. I recommend that every week the vacuum group test remote log into vacuum1 to verify this is possible.

More clues....or confusion. MidY IP9 current plotted with PT-210 pressure increase from Sunday evening. Strange behavior in IP.

Trended PT-170 over the weekend to understand the verbal alarm Sheila heard in control room. PT-170 pressure has steadily been falling since the vent, but at about 9:30pm local time on Sunday, it was crossing over the 7e-8 Torr alarm threshold causing the verbal alarm. What luck!

Outside temperature plotted over 30 days along with PT-243 at mid-Y.

before I left last night, I drove to End X and power cycled the laser, at 4:59 UTC on May 29th. In the attached screenshot you can identify this by the time that the LASERDIODEPOWERMONITOR goes to 0.  ALthought the power out of the laser decreased by about 3% for IR ( but only 1% for green, which should depend on the IR power^2) the sudden power drops didn't happen for 10 hours after I power cycled it.  

I was able to lock ALS for about 6 minutes, but am going to head to the end station now to check for mode hopping in the ALS laser. 

I was able to move the laser temperature so that the beat note moved by +-500MHz, without seeing any mode hopping. 

Attached are two screenshots of two different kinds of power drops, one that only appears in green channels and one that only appears in IR channels. 

I also adjusted the temperature of the doubling crystal, which was at 33.51 when I started, to get about 3% more green power out.  I don't think this will fix the problem. 

To prepare for swapping the laser, I checked the green irises on the input beam right before the periscope, the beam is well centered on them.  I also added two irises in the IR path right after the laser before the steering mirror, using a card not a viewer.  I attempted to add an irises to the green path but did not have the right base plates. 

I also added and aligned 4 irises right after the laser, in the IR path the two irises are in the longish beam line before any steering mirrors, for green they are in the path after two steering mirrors before the EOM.  I also put 4 dog clamps marking the location of the laser base plate.  We can use the dog clamps and the irises in the IR path for placing the laser, and use the irises in the green path for the two steering mirrors right after the laser, to hopefully get us to roughly the right alignment. 

After Keita left I was able to lock twice before the ALS problems made locking impractical. 

In the first lock, I tried the soft offsets that Vaishali and I found last night at 2W input power.  I was able to get the recycling gain which had been about 31 at 2W to nearly 33 using an offset of 0.04 in both DSOFT and CSOFT yaw, with the offsets that Vaishali and I found the recycling gain was down to about 29.  As I powered up with these offsets in, the recycling gain stayed high and at 27W we had a PR gain of nearly 29. In the second lock I moved these offsets into the TMS QPD offsets, and have saved these in SDF. I also made a simple script for doing this, which is attached.  

After powering up I started executing the LOWNOISE_ASC state by hand, and started with the HARD loops because Keita was suspicous of a problem with DSOFT.  switching the HARD loops to their low noise configuration caused an instability that looked like DHARD Y which unlocked the IFO. The next lock I lost after sitting at DC readout for about a half hour, as soon as I reduced the RF45 Modulation depth there was an ASC instability (PIT this time).