(This is for yesterday during the Holiday Site Inspection for 12/29.)
Topped off the Crystal Chiller with 240mL of water. (FYI: It was lasted filled on 12/25 by Jason with 175mL)
No alarm for the Diode Chiller.
This closes FAMIS #6503.
Starting CP3 fill. TC A error. TC B error. Fill aborted. Starting CP4 fill. LLCV enabled. LLCV set to manual control. LLCV set to 70% open. Fill completed in 90 seconds. TC A did not register fill. LLCV set back to 35.0% open.
I logged in remotely and changed the setting for CP3 to 16% from 18%.
CP3 auto fill did not start, here are the logs
Starting CP3 fill. TC A error. TC B error. Fill aborted.
Starting CP4 fill. LLCV enabled. LLCV set to manual control. LLCV set to 70% open. Fill completed in 90 seconds. TC A did not register fill. LLCV set back to 35.0% open.
The code looks at the TC-A and TC-B temps before starting, for CP3 if they are below -20C the script will abort.
Looking at CP3 StripTool (attached) it looks like the script ran when the temps were in their low spike.
VAC team are looking into the issue.
CP4 filled normally.
Apollo was on site Tuesday and Wednesday of this week to begin the HVAC Controls Upgrade Project,specifically at the LSB. They have ~ 90% of the hardware installed in that building and plan on trying to finish up there possibly next week. They have started balancing the system and are writing the programs for the new system. Huge difference already.
Today as part of the end of the year maintenance I reboot the following servers and services: DNS servers NS0 and NS1, file servers CDSFS0 and CDSFS1, All workstations but the NUCs A list of servers that need reboot was generated and they will be reboot as part of the regular Tuesday maintenance
Summary: shutting down the main HVAC system increased the range by 2-3 Mpc. The shutdown produced a large change in DARM, 8-10 Hz, and ~ 3% change between 48-130 Hz. The feature in the 8-10 Hz band may be due to turbulence in the Y-end chilled water flow and would probably be reduced by lowering the frequency of the VFD at Y-end. It is not yet known if this will reduce the noise in the 48-130 Hz band.
Thursday, Dec. 22, before we shut down, I cycled the main HVAC multiple times in order to see if it was costing us range. In the blue “off” periods in the figures, the chiller pad chillers and water pumps were off, the turbines were off at EX, EY, and CS, and the OSB fan was off. Figure 1 shows that the range improved by 2 to 3 Mpc during the “off” periods (we were averaging about 70 Mpc).
Figure 2 shows that the DARM spectrum improved in the 6-18Hz region, by a large factor between 8 and 10 Hz, and by roughly 3% in the 48-130 Hz regions. It is not clear whether the noise produced by the HVAC in the 48-130 Hz band is produced by vibration at lower frequency or by direct coupling. Low coherence with vibration sensors (< 1% in most of the 48-130 Hz band), and the observation that there is little change in peaks in that band that are known to be driven by vibration, suggest that the increased noise in this band is produced by vibration at lower frequencies and not by linear coupling. However, a preliminary look at PEM injection suggest that it is not impossible that the noise in the 48-130 Hz band is produced by direct vibration coupling. We will investigate this with further analysis of data from PEM injections.
The biggest difference between “on” and “off” was in the 8 -10 Hz band (Figure 2). Figure 3 shows that Y-motion of ETMY ST2 is quite coherent with DARM, while X-motion at ETMX is not coherent with DARM. The CS is also not as coherent at these frequencies. The chilled water flow at EY has previously been shown to produce vibration in this band from turbulence (https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=11466), so reducing the setting of the variable frequency drive below the current 50Hz level (I recommend 35-45Hz in the linked log) might reduce the 8-10 Hz region of DARM. Further analysis of PEM injections and additional injections at 9 Hz at EY would help us determine if the 8-10 Hz drive at Y-end is also responsible for the noise in the 48-130 Hz band.
Shutdown times:
UTC Dec. 22
21:24:00 shutdown starts, 21:26:00 complete
21:34:00 startup starts, 21:36:40 complete
21:44:00 shutdown starts, 21:45:40 complete
21:54:00 startup starts, 21:55:40 complete
22:04:00 shutdown starts, 22:05:40 complete
22:14:00 startup starts, 22:15:40 complete
Given the large coherence between EY GS13 and DARM in the 8-10Hz band, can this be used to subtract out the noise in the affected band? perhaps in the F?E or in the calibration pipeline? Or would it be easier to eliminate the noise at its source?
I was wondering if the coherence between DARM and the GS13s could be somehow caused by our feedback of DARM to the ETMY suspension, but I think that it is not and that the coherence is probably due to noise from EY coupling to DARM as Robert suggests.
I found a 4 hour period starting at 11:51 UTC on Dec 1st when we had the interferometer locked on ETMX. The coherence between DARM and the GS13s is pretty similar when we are locked on ETMX. I wasn't able to plot data from NDS2 and NDS1 on the same dtt template, but the noise was higher in the lock from Dec 1st, which explains the somewhat lower coherence. In the attached screenshot both plots are made with 30 averages, despite the DTT display that says the one taken today only has 2 averages.
The point is that the coherence is higher with ETMY GS13s than ETMX no matter which suspension we are feeding back to.
Found my truck battery was dead so am driving around to charge it up ;) ~1535 hrs. local -> Manually overfilled CP4 via opening LLCV bypass 1/2 turn. LN2 at exhaust outlet in 90 seconds -> Increased manual LLCV %open to 35% from the as found value of 34%. CP3 still showing intermittent vapor from earlier automated overfill. UHP N2 bottle pressure now 1250 psi. Pumps running at BSC8 quiet and happy.
Monday (Dec. 19) evening we replaced the L2 coil driver for ETMY in order to check if it was the source of glitches, as suggested by DetChar. By Tuesday morning there had been a run of a couple of hours with the new driver, and I found at least 3 glitches with similar spectrogram/omegagram signatures during the run with the substitute chassis. Figure 1 shows one of these glitches during the period when the substitute chassis was used. Since typical glitches were still common, we put the original chassis back in during maintenance on Tuesday to avoid re-calibration. When the interferometer came back, the glitch rate was lower. It is thus unlikely that the original chassis was the source of the glitches. One possible reason for the increase/decrease in glitch rate would be a decrease/increase in humidity as the outside temperature rose, similar to what was found by Paul S. and others for O1 (https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=28534). It would be interesting to check how well glitches of different SNRs correlate with humidity, and if the correlation is better for any of the "RELHUM" channels located at multiple sites in the 3 buildings.
The fire protection system at the LSB has been restored. The ruptured valve was replaced with a new valve and relocated to the attic area away from the exterior wall.
Temperature differences on the PSL table are of interest because of the potential for producing beam jitter through thermally driven air currents (https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=31190, https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=31263)
During PEM injections, Anamaria and I measured temperatures on the PSL table at LLO to compare to LHO temperatures, measured previously. The table below compares hot spots that were several degrees warmer than the PSL table, at least at one site, and close to the beam path. The PMC, for example, is not included because the outer surface of the tank was not several degrees warmer than the PSL table at either site.
Starting CP3 fill. LLCV enabled. LLCV set to manual control. LLCV set to 50% open. Fill completed in 23 seconds. LLCV set back to 18.0% open. Starting CP4 fill. LLCV enabled. LLCV set to manual control. LLCV set to 70% open. Fill completed in 369 seconds. LLCV set back to 34.0% open.
Since there looks to have been some EQs last night, I'm going to attempt remote login to reset things.
-Betsy (and JKissel consult)
Only HAM2 and HAM3 Watchdogs tripped, no SUSes, and no other ISI's. The remote login was extremely slow as expected, but I was able to reset these 2 watchdogs and everyone is back to green.
1120 - 1125 hrs. local -> Kyle in and out of the LVEA 1135 - 1140 hrs. local -> Kyle in and out of the LVEA Valved-in locally mounted turbo to PT180. Should know in a day or two if the indicated pressure value will "drift" up while the gauge is pumped by the turbo and not exposed to the site vacuum volume. Also, I witnessed the truck driver decouple and store the transfer line following the LN2 delivery to CP3. He installed a plug to, at least, one end of the transfer line before storing it - I assume the other end was also plugged. Cylinder pressure of UHP N2 bottle holding pressure to CP4's clogged sensing line is @ 1500 psi while the flow meter indicates no flow. The regulator output is set for 20 psi and the flow meter is limited to 0.4 LPM.
Starting CP3 fill. LLCV enabled. LLCV set to manual control. LLCV set to 50% open. Fill completed in 20 seconds. LLCV set back to 20.0% open. Starting CP4 fill. LLCV enabled. LLCV set to manual control. LLCV set to 70% open. Fill completed in 29 seconds. LLCV set back to 34.0% open.
1220 - 1230 hrs. local -> Kyle in and out of LVEA Will resume pumping the isolated PT180 gauge soon to continue our long-term "gauge drift" data collecting. Found LVEA lights on and, thus, turned them off when leaving for "sustainability" reasons ;) Also, manually confirmed that CP3 and CP4 had been filled via script -> they are full.
It appears as if CP6 LLCV valve stem is frozen again, I'm at the site looking into it.
I think this is a different problem, the actuator does not appear to be actuating anymore.
Arrived to the site and went straight to CP6, turned flood lamp on, and went back to the corner station to pick up tools. Arrived at CP6 for the second time, started using an EE "hair dryer", but the actuator did no show improvement. Then removed the cap from the actuator and discovered that the whole inside was flooded, removed the water by removing the plugs at the base of the actuator, the water is getting in via the conduit (see photo CP6_actuator_conduit). The water that was hard to remove was the one that pools by the power port via, see CP6_actuator. Used some paper towels to soak the water from there, the shape of this location creates a great little pool. I tried to manually actuate the valve up and down using the internal buttons but no movement was noted. To cover all bases I went inside the MX-VEA and checked for blown fuses but no blown fuse was found, and the actuator is humming (making noise as if is trying to move thus I assume it has power).
I cleaned the cap of the actuator and installed it, with both of the plugs removed, I used the "hairdryer" to blow hot air inside the actuator via the signal port, no change noted on the behaviour of the actuator. Then my flashlight batteries died, so I stopped work. So I don't know if the water inside the actuator shorted or seized the motor, but the actuator is not working like it should.
Flood light remains on near the actuator, and for some reason the fill volume remained somewhat steady while working on it.
FRS ticket filed, https://services.ligo-la.caltech.edu/FRS/show_bug.cgi?id=6998
Filaberto turned off the heat lamp on Friday during his site inspection walk through. The heat gun is either in my office or in the Mechanical shop. There really is a Santa Claus - only now he is based out of Grandview WA!
Done, removed and replaced the faulty actuator, as I removed the lock-nut of the valve stem, it became clear that the valve stem was free, since I was able to move it up and down very easy.
Zeroed, and set the span for the new actuator. Also extended the conduit a bit to fix the leak.
