Starting CP3 fill. LLCV enabled. LLCV set to manual control. LLCV set to 50% open. Fill completed in 26 seconds. LLCV set back to 16.0% open.
Starting CP4 fill. LLCV enabled. LLCV set to manual control. LLCV set to 70% open. Fill completed in 82 seconds. LLCV set back to 35.0% open.

As found, the shop light used as a local heat source for CP5's LLCV ("control valve" for John) was on, the valve stem pointer was 60% < pointer < 70% and the pump level was ~75% full as indicated by CDS and 30" WC as per the magnahelic diff pressure gauge (mechanical level) -> I opened the LLCV bypass valve 1/2 turn to manually fill CP5 while I troubleshot (reached > 100% full in 35 minutes).  I decided to fix the heat tape issue on this visit so we wouldn't need the light anymore.  The trace heat switch in the VEA was on but no 120VAC at the heat tape -> found tripped breaker (CKT #1 in panel VEAC-01A???).  I then confirmed 120VAC at the JCT box feeding the trace heat tapes - fixed (the breaker was likely tripped when the heat trace was installed a few years ago?)  Next, I removed the electric actuator cover and found a 1/2" of standing water in the bottom of the housing (not frozen).  One of the 24VDC wires had a crimp connector which was fully submerged.  Measured 3.9VDC at the terminal strip inside of the housing.  I noticed that the 1/2" plastic liquid-tight conduit elbow had been cross-threaded into the actuator housing such that it wasn't fully threaded and, as such, wasn't compressing the O-ring, i.e., was not making a water-tight seal -> I lifted the wires and removed and re-threaded this fitting (I should have tried to find a new one). I also followed Gerardo's lead and used electrical tape to seal all of the liquid-tight joints - Brrrrr.. cold fingers makes working with 18awg wires in tight spaces and electrical tape a humiliating task!  

Back in the "oh, so warm" VEA I found the blown fuse for XV300 terminal 212 and replaced it -> Confirmed 24VDC in actuator housing - fixed! I stroked the LLCV to 0% open via MANUAL mode in CDS (note, I had to come back to the Fuching Corner Station to do this because I couldn't navigate the Beckhoff screens!  Argh! I feel like a stranger in my own home.  No more changes/upgrades people!) and confirmed (back at the X-mid) that the valve stem pointer was at 0% etc.. 

Leaving in PID control

The pending leap second LED cleared at 16:00 PST indicating the leap second was applied correctly. Disappointingly I did not see the seconds indicator do the 58, 59, 60, 00, 01 sequence, it looks like it stayed on 00 for two seconds.

The computer h1susauxh34 died at 14:57 PST Saturday. It was powered up, but had a blank console and not reachable via network. Since this is not on the Dolphin fabric, I power cycled it at 15:53 PST and it came back correctly.

I'll stop in and fix after lunch, 90 minutes or so.

Started receiving text messages about PT343 CC  @ 9:13 PM (PST), it turns out that somehow ithe CC turned off, I logged in to turn it back on (I did a handfull of toggles (on->off->on->off....->on)) but CC did not comeback on scale, I will leave it ON.

As found, neither CP3 nor CP4 showing vapor at exhaust.  CP3 had frosted exhaust line (typical for environment, shaded).  CP4's exhaust line not frosted (typical for environment, in sunshine).  As previously mentioned by Dave B., the T/C wires fed into CP4's exhaust have an excessive loop which will get snagged by tumbleweeds eventually -> Manually overfilled CP4 by opening its LLCV bypass valve 1/2 turn -> LN2 at exhaust outlet in 15 seconds.  -> Manually overfilled CP3, by opening its LLCV bypass valve 1/2 turn -> LN2 at exhaust outlet in 15 seconds as well.  Leaving CP3 LLCV %open for "others" to fiddle with remotely.

Bottle pressure of UHP N2 bottle used to slightly pressurize CP4's clogged sensing line (20 psi) now at 1000 psi.  

~1340 - 1345 hours local -> Kyle in and out of LVEA

Rotating shaft vacuum pumps ("my girls") used for PT180 experiment are cool and quiet ("and happy - a man knows these things!")  No sign of gauge drift at this point.

1410 hrs. local -> Kyle leaving site now. 

(This is for yesterday during the Holiday Site Inspection for 12/29.)

Topped off the Crystal Chiller with 240mL of water.  (FYI:  It was lasted filled on 12/25 by Jason with 175mL)

No alarm for the Diode Chiller.

This closes FAMIS #6503.

Starting CP3 fill. TC A error. TC B error. Fill aborted.
Starting CP4 fill. LLCV enabled. LLCV set to manual control. LLCV set to 70% open. Fill completed in 90 seconds. TC A did not register fill. LLCV set back to 35.0% open.

CP3 auto fill did not start, here are the logs

Starting CP3 fill. TC A error. TC B error. Fill aborted.

Starting CP4 fill. LLCV enabled. LLCV set to manual control. LLCV set to 70% open. Fill completed in 90 seconds. TC A did not register fill. LLCV set back to 35.0% open.

The code looks at the TC-A and TC-B temps before starting, for CP3 if they are below -20C the script will abort.

Looking at CP3 StripTool (attached) it looks like the script ran when the temps were in their low spike.

VAC team are looking into the issue.

CP4 filled normally.

Apollo was on site Tuesday and Wednesday of this week to begin the HVAC Controls Upgrade Project,specifically at the LSB. They have ~ 90% of the hardware installed in that building and plan on trying to finish up there possibly next week. They have started balancing the system and are writing the programs for the new system. Huge difference already. 

Today as part of the end of the year maintenance I reboot the following servers and services:
DNS servers NS0 and NS1,
file servers CDSFS0 and CDSFS1,
All workstations but the NUCs
A list of servers that need reboot was generated and they will be reboot as part of the regular Tuesday maintenance

 

Summary: shutting down the main HVAC system increased the range by 2-3 Mpc. The shutdown produced a large change in DARM, 8-10 Hz, and ~ 3% change between 48-130 Hz. The feature in the 8-10 Hz band may be due to turbulence in the Y-end chilled water flow and would probably be reduced by lowering the frequency of the VFD at Y-end. It is not yet known if this will reduce the noise in the 48-130 Hz band.

Thursday, Dec. 22, before we shut down, I cycled the main HVAC multiple times in order to see if it was costing us range. In the blue “off” periods in the figures, the chiller pad chillers and water pumps were off, the turbines were off at EX, EY, and CS, and the OSB fan was off. Figure 1 shows that the range improved by 2 to 3 Mpc during the “off” periods (we were averaging about 70 Mpc).

Figure 2 shows that the DARM spectrum improved in the 6-18Hz region, by a large factor between 8 and 10 Hz, and by roughly 3% in the 48-130 Hz regions. It is not clear whether the noise produced by the HVAC in the 48-130 Hz band is produced by vibration at lower frequency or by direct coupling. Low coherence with vibration sensors (< 1% in most of the 48-130 Hz band), and the observation that there is little change in peaks in that band that are known to be driven by vibration, suggest that the increased noise in this band is produced by vibration at lower frequencies and not by linear coupling. However, a preliminary look at PEM injection suggest that it is not impossible that the noise in the 48-130 Hz band is produced by direct vibration coupling. We will investigate this with further analysis of data from PEM injections.

The biggest difference between “on” and “off” was in the 8 -10 Hz band (Figure 2). Figure 3 shows that Y-motion of ETMY ST2 is quite coherent with DARM, while X-motion at ETMX is not coherent with DARM. The CS is also not as coherent at these frequencies. The chilled water flow at EY has previously been shown to produce vibration in this band from turbulence (https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=11466), so reducing the setting of the variable frequency drive below the current 50Hz level (I recommend 35-45Hz in the linked log) might reduce the 8-10 Hz region of DARM. Further analysis of PEM injections and additional injections at 9 Hz at EY would help us determine if the 8-10 Hz drive at Y-end is also responsible for the noise in the 48-130 Hz band.

Shutdown times:

UTC Dec. 22

21:24:00 shutdown starts, 21:26:00 complete

21:34:00 startup starts,  21:36:40 complete

21:44:00 shutdown starts, 21:45:40 complete

21:54:00 startup starts, 21:55:40 complete

22:04:00 shutdown starts, 22:05:40 complete

22:14:00 startup starts, 22:15:40 complete

Found my truck battery was dead so am driving around to charge it up ;)

~1535 hrs. local -> Manually overfilled CP4 via opening LLCV bypass 1/2 turn.  LN2 at exhaust outlet in 90 seconds -> Increased manual LLCV %open to 35% from the as found value of 34%.  

CP3 still showing intermittent vapor from earlier automated overfill.  

UHP N2 bottle pressure now 1250 psi.  

Pumps running at BSC8 quiet and happy.