We had a potentially scary situation tonight at mid-Y and through crazy coincidence managed to fix it before it became a serious problem. Sheila contacted me around 10 pm local time about a verbal pressure alarm that was going off in control room for BSC7 (gauge PT-170). I checked the MEDM screen from home and didn't see anything abnormal - except that the pressure is a bit high since the vent (7e-8 Torr). Most likely it's alarming because of set point setting.
This alarm made me look at our site pressure trend (48 hr trend here) and noticed that PT-210 at mid-Y was quickly drifting up starting around 7pm. Suspected CP3 and/or CP4 were warming up due to very hot temperatures we've had this weekend. Gerardo was unable to remotely log into CDS to initiate a remote overfill even though we were supposed to have permission until June 1. I drove out to the site to manually overfill both cryopumps at the skid by opening the bypass valve 1/2 turn (just like the good ole days). Filled CP3 first and observed an almost immediate drop in pressure. Took 50 min. to overfill (verified by watching LN2 pour out of exhaust). As soon as I started the fill, the exhaust flow increased to turbulent. CP4 didn't exhibit the same turbulent behavior, and took 30 minutes to overfill. Conclusion is CP3's valve actuator setting from Friday was too low at 15% open. I reset to 18%. Also increased CP4 from 37% to 39% open. Tomorrow is supposed to be 98F!
Need to learn what the current pressure alarms are set to; I propose we tighten these just for mid-Y so vacuum staff is alerted quickly when pressure starts to rise. I also suggest we try to maintain seconds vs. minutes of overfill time as we approach a hot summer.
Based on this log entry from last June 24, it took 35 minutes to overfill CP4 until LN2 poured out the exhaust. This was before CP4 clog - we were experimenting with durations and flow rates to create a work-around for CP3.
https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=27950
Tonight's real life scenario has been my nightmare for the past 18 months (since ice plugs have required manual filling of CP3 and CP4). It happened on my watch and I am responsible for it. In my defense, this was not the result of inattention or a false sense of security on my part. I had lowered CP3's manual LLCV %open value to 15% open, down from 17% open, in response to Friday's automated fill having only taken 17 seconds. This would have been an appropriate response, perhaps, for springtime ambient temperature conditions but proved too much of a reduction/correction for this weekend's warmest-of-the-year weather. I look at the vacuum site overview screen multiple times on non-work days and am quite familiar with what the "normal" values are. Today was no different. At around 07:30 pm local time, I looked and noticed that PT243 was 3.97 x 10-9 torr which is higher than normal and caught my attention. I reasoned that this was probably hydrogen emitting from the BT steel on this "hot" day but was concerned enough that I resolved to check it again before going to bed. At approx. 10:30 pm local time, I looked and saw that PT243 had fallen to 2.?? x 10-9 torr. Minutes later, I checked my phone before going to bed and became aware of a text thread between Chandra R. and Gerardo M. which had been in progress for the previous 30 minutes. So, the reduction in PT243 at 10:30 pm was the result of the fact that Chandra was already on site and had started filling CP3 manually via the opening the LLCV bypass valve. Had Sheila D. not contacted Chandra at approx. 09:50 pm and Chandra not responded by doing a manual fill, the pressure shown by PT243 10:30 pm would have been much higher than the previuosly "concerning" value seen at ~07:30 and I feel that I would have responded appropriately. Still, this didn't have to happen. As Chandra reminded me, pressure trends are available (new location) for remote viewing and, had I reviewed these in addition to the Vacuum Site Overview, I would have noticed that the Y-mid values were increasing independent of the rest of the site pressures. This would have dispelled my "hydrogen" theory at 07:30 and I would have done a manual refill then.
Kyle, you shouldn't feel responsible. I'm usually the one who manipulates the LLCV settings based on temperature fluctuations and fullness of Dewar and have a better feel for adjustments. Sorry I didn't explain that better before I left. We can start to think about the next level of automation on this system which would increase/decrease valve setting based on how long it took to fill the previous time. Folks should also recognize that the work we're proposing to do post O2 by either decommissioning and/or regenerating these CPs will eliminate these risks.
Also, I doubt PT-170 alarming was actually a crazy coincidence. I forgot to trend its pressure but am guessing it was also starting to increase due to loss of cryopump action at MY. And because its pressure was already high from vent, it alerted us before we had to wait for -8 torr range alarms in arm. Thank goodness Sheila was in the lab at the time to catch it!
Good save, I should have thought of this. The dominant boiloff load is (should be) blackbody radiation from the tube, which is at BTE ambient temperature. I will make a number for the fractional effect on liquid mass flow per degree, so we can add that % on to our "open-loop estimate".
EDIT: see post 36496
Worth noting, though, the high ambient (BTE ) temp is raising the hydrogen diffusion flux, which doubles every 6 C or so (harmlessly, as long as we have ion pumps). So the pressure trend (and particularly, any attribution to the CP) has to be interpreted carefully.
Even before today, the ice plugs gave me nightmares. We have to fix them, and stop any more from happening.
Dave Barker suggested increasing to daily auto overfills rather than Mon-Wed-Fri. I like this idea. We'll discuss with vacuum group this week.
The cell phone alarm system currently monitors the vacuum gauge pairs at the ends of the 2km beam tube sections. For MY those are PT243 (closest to corner) and PT246 (closest to EX). I can certainly add all the other gauges in MY (PT244, PT245, PT210) to the system if needed.
I've created a remote access permit for the vacuum group, good through the end of the year.
No cell phone alarms were raised for this event, their upper alarm range is 5.0e-08 torr, an order of magnitude higher than what MY saw Sunday night (trend attached).
Gerardo was able to remotely log into CDS from home, he had a permit open. He was unable to directly log into the vacuum1 machine to make vacuum changes due to recent ssh cert changed. I recommend that every week the vacuum group test remote log into vacuum1 to verify this is possible.
More clues....or confusion. MidY IP9 current plotted with PT-210 pressure increase from Sunday evening. Strange behavior in IP.
Trended PT-170 over the weekend to understand the verbal alarm Sheila heard in control room. PT-170 pressure has steadily been falling since the vent, but at about 9:30pm local time on Sunday, it was crossing over the 7e-8 Torr alarm threshold causing the verbal alarm. What luck!
Outside temperature plotted over 30 days along with PT-243 at mid-Y.
