1120 - 1125 hrs. local -> Kyle in and out of the LVEA 1135 - 1140 hrs. local -> Kyle in and out of the LVEA Valved-in locally mounted turbo to PT180. Should know in a day or two if the indicated pressure value will "drift" up while the gauge is pumped by the turbo and not exposed to the site vacuum volume. Also, I witnessed the truck driver decouple and store the transfer line following the LN2 delivery to CP3. He installed a plug to, at least, one end of the transfer line before storing it - I assume the other end was also plugged. Cylinder pressure of UHP N2 bottle holding pressure to CP4's clogged sensing line is @ 1500 psi while the flow meter indicates no flow. The regulator output is set for 20 psi and the flow meter is limited to 0.4 LPM.
Starting CP3 fill. LLCV enabled. LLCV set to manual control. LLCV set to 50% open. Fill completed in 20 seconds. LLCV set back to 20.0% open. Starting CP4 fill. LLCV enabled. LLCV set to manual control. LLCV set to 70% open. Fill completed in 29 seconds. LLCV set back to 34.0% open.
1220 - 1230 hrs. local -> Kyle in and out of LVEA Will resume pumping the isolated PT180 gauge soon to continue our long-term "gauge drift" data collecting. Found LVEA lights on and, thus, turned them off when leaving for "sustainability" reasons ;) Also, manually confirmed that CP3 and CP4 had been filled via script -> they are full.
hand-off run of the CP3,4 autofill scripts completed quickly. Now waiting for robo-alog report at 12:10
Starting CP3 fill. LLCV enabled. LLCV set to manual control. LLCV set to 50% open. Fill completed in 20 seconds. LLCV set back to 20.0% open.
Starting CP4 fill. LLCV enabled. LLCV set to manual control. LLCV set to 70% open. Fill completed in 29 seconds. LLCV set back to 34.0% open.
The PSL tripped off this morning, interlocks indicated a loss of flow in the laser head water circuit. After restarting the laser from home, I drove out to the site to check a chiller alarm (that turned out to be a low water level alarm), and ended up having to reset the noise eater as I recovered the PMC/FSS/ISS (as usual when recovering from a trip). The laser is now back up and running in the same state it was left in on Thursday.
The whole valve stem was covered with ice, see photo, so I moved the light from CP6 to CP5 and left it on.
h1tw0 (secondary minute trend writer) started having RAID issues again beginning at 05:35 PST this morning. Its been in a start-stop-restart cycle ever since. I've turned off monit on h1tw0 to stop it respawning. We'll rely on h1tw1 for the rest of the holidays.
The DAQ EDCU is now not green (purple) because of the loss of h1tw0. To confirm, you can run the command h1dc0_edcu_list_disconnected_epics_channels.py and verify all 57 chans are from H1:DAQ-TW0
david.barker@zotws2: h1dc0_edcu_list_disconnected_epics_channels.py
daqd>
33796 channels
33739 connected
57 disconnected
80237 connection events processed
2220990763 value change events processed
H1:DAQ-TW0_CYCLE
H1:DAQ-TW0_TOTAL_CHANS
H1:DAQ-TW0_DATA_RATE
H1:DAQ-TW0_EDCU_CHANS
H1:DAQ-TW0_EDCU_CONN_CHANS
H1:DAQ-TW0_UPTIME_SECONDS
H1:DAQ-TW0_LOOKBACK_RAM
H1:DAQ-TW0_LOOKBACK_FULL
H1:DAQ-TW0_LOOKBACK_DIR
H1:DAQ-TW0_LOOKBACK_STREND
H1:DAQ-TW0_LOOKBACK_STREND_DIR
H1:DAQ-TW0_LOOKBACK_MTREND
H1:DAQ-TW0_LOOKBACK_MTREND_DIR
H1:DAQ-TW0_FAST_DATA_CRC
H1:DAQ-TW0_FAULT
H1:DAQ-TW0_BCAST_RETR
H1:DAQ-TW0_BCAST_FAILED_RETR
H1:DAQ-TW0_GPS
H1:DAQ-TW0_CHANS_SAVED
H1:DAQ-TW0_FRAME_SIZE
H1:DAQ-TW0_SCIENCE_FRAME_SIZE
H1:DAQ-TW0_SCIENCE_TOTAL_CHANS
H1:DAQ-TW0_SCIENCE_CHANS_SAVED
H1:DAQ-TW0_FRAME_WRITE_SEC
H1:DAQ-TW0_SCIENCE_FRAME_WRITE_SEC
H1:DAQ-TW0_SECOND_FRAME_WRITE_SEC
H1:DAQ-TW0_MINUTE_FRAME_WRITE_SEC
H1:DAQ-TW0_SECOND_FRAME_SIZE
H1:DAQ-TW0_MINUTE_FRAME_SIZE
H1:DAQ-TW0_RETRANSMIT_TOTAL
H1:DAQ-TW0_PRDCR_TIME_FULL_MEAN_MS
H1:DAQ-TW0_PRDCR_TIME_FULL_MIN_MS
H1:DAQ-TW0_PRDCR_TIME_FULL_MAX_MS
H1:DAQ-TW0_PRDCR_TIME_RECV_MEAN_MS
H1:DAQ-TW0_PRDCR_TIME_RECV_MIN_MS
H1:DAQ-TW0_PRDCR_TIME_RECV_MAX_MS
H1:DAQ-TW0_PRDCR_CRC_TIME_FULL_MEAN_MS
H1:DAQ-TW0_PRDCR_CRC_TIME_FULL_MIN_MS
H1:DAQ-TW0_PRDCR_CRC_TIME_FULL_MAX_MS
H1:DAQ-TW0_PRDCR_CRC_TIME_CRC_MEAN_MS
H1:DAQ-TW0_PRDCR_CRC_TIME_CRC_MIN_MS
H1:DAQ-TW0_PRDCR_CRC_TIME_CRC_MAX_MS
H1:DAQ-TW0_PRDCR_CRC_TIME_XFER_MEAN_MS
H1:DAQ-TW0_PRDCR_CRC_TIME_XFER_MIN_MS
H1:DAQ-TW0_PRDCR_CRC_TIME_XFER_MAX_MS
H1:DAQ-TW0_PROFILER_FREE_SEGMENTS_MAIN_BUF
H1:DAQ-TW0_RAW_FW_STATE
H1:DAQ-TW0_RAW_FW_DATA_STATE
H1:DAQ-TW0_RAW_FW_DATA_SEC
H1:DAQ-TW0_SCIENCE_FW_STATE
H1:DAQ-TW0_SCIENCE_FW_DATA_STATE
H1:DAQ-TW0_SCIENCE_FW_DATA_SEC
H1:DAQ-TW0_STREND_FW_STATE
H1:DAQ-TW0_MTREND_FW_STATE
H1:DAQ-TW0_FRAME_CHECK_SUM_TRUNC
H1:DAQ-TW0_SECOND_FRAME_CHECK_SUM_TRUNC
H1:DAQ-TW0_MINUTE_FRAME_CHECK_SUM_TRUNC
Total 57 disconnected.
It appears as if CP6 LLCV valve stem is frozen again, I'm at the site looking into it.
I think this is a different problem, the actuator does not appear to be actuating anymore.
Arrived to the site and went straight to CP6, turned flood lamp on, and went back to the corner station to pick up tools. Arrived at CP6 for the second time, started using an EE "hair dryer", but the actuator did no show improvement. Then removed the cap from the actuator and discovered that the whole inside was flooded, removed the water by removing the plugs at the base of the actuator, the water is getting in via the conduit (see photo CP6_actuator_conduit). The water that was hard to remove was the one that pools by the power port via, see CP6_actuator. Used some paper towels to soak the water from there, the shape of this location creates a great little pool. I tried to manually actuate the valve up and down using the internal buttons but no movement was noted. To cover all bases I went inside the MX-VEA and checked for blown fuses but no blown fuse was found, and the actuator is humming (making noise as if is trying to move thus I assume it has power).
I cleaned the cap of the actuator and installed it, with both of the plugs removed, I used the "hairdryer" to blow hot air inside the actuator via the signal port, no change noted on the behaviour of the actuator. Then my flashlight batteries died, so I stopped work. So I don't know if the water inside the actuator shorted or seized the motor, but the actuator is not working like it should.
Flood light remains on near the actuator, and for some reason the fill volume remained somewhat steady while working on it.
FRS ticket filed, https://services.ligo-la.caltech.edu/FRS/show_bug.cgi?id=6998
Filaberto turned off the heat lamp on Friday during his site inspection walk through. The heat gun is either in my office or in the Mechanical shop. There really is a Santa Claus - only now he is based out of Grandview WA!
Done, removed and replaced the faulty actuator, as I removed the lock-nut of the valve stem, it became clear that the valve stem was free, since I was able to move it up and down very easy.
Zeroed, and set the span for the new actuator. Also extended the conduit a bit to fix the leak.
As noted previously, alternating between pressurization of the clogged line followed by applying a rough vacuum, results in gas generation on the atmospheric side (gauge side when connected to the gauge) of the obstruction results. My initial, optimistic, interpretation was that the ice plug was getting "drawn" slowly up the sensing tube and making contact with relatively warm surfaces, thus melting the leading portion of the plug. Considering the repeatable phenomenon of watching the gas pressure build up on the gauge only to have it fluctuate as if the accumulated gas was being released via some unknown path, I now have a new, more pessimistic, picture in my head (see attached sketch). Maybe the ice plug is irregular in shape and not it complete contact with the large diameter aluminum tube wall such that it gets rocked, cocked, tilted etc. by the pushing and pulling of applying pressurized UHP N2 followed by vacuum pumping. If this is happening then small amounts of LN2 can leak past and turn to gas and this is the source of the gas not the vanishing ice plug. Since the gauge end of the sensing tube is closed, the pressure build up of GN2 can then escape past the ice obstruction back to the LN2 side. This is consistent with the observations. Note in the sketch that water ice is denser that LN2 and would be expected to concentrate at the bottom of the pump. Also, once formed, it has no place to go. It is too big to enter into the small I.D. of the SS sensing tube or make it past the "hard" 90 degree aluminum tube elbow. My wife thinks I'm Christmas shopping!
Starting CP3 fill. LLCV enabled. LLCV set to manual control. LLCV set to 50% open. Fill completed in 21 seconds. TC B did not register fill. LLCV set back to 20.0% open. Starting CP4 fill. LLCV enabled. LLCV set to manual control. LLCV set to 70% open. Fill completed in 18 seconds. LLCV set back to 34.0% open.
sigh, looks like the png strip tool image wasn't available at the time the roboalog posted, here it is:
In order to quell the voices in my head and allow me to resume my "normal" activities, I had to manually overfill CP4 and CP3 after the automated routine ran. Sure enough both pumps were full!
I have noticed an excessive amount of flow exhausting out of CP4's dewar (a.k.a. vertical storage tank #8514372) this week (earlier too?). The indicated vapor pressure between the two mechanical gauges range between 5 < vapor pressure < 10 psi which is on the "low side" of normal for this tank -> By kicking the piping in order to liberate the accumulated frost and gain access to the economizer valve's pressure adjusting screw I caused the flow to stop. No adjustment was needed - valve was just stuck open. This used to happen frequently during freezing weather because these valves (actually function only as adjustable pressure regulating valves) were originally installed with the adjusting screw pointing up. Water would get inside the valve body via the screw's parallel threads and, when frozen, would lock the diaphragm and prevent it from deflecting. However, most of these (including this one) had long ago been rotated so as to point down. Don't know why this one was stuck.
autofill of CP3 and CP4 did not start correctly, I'm investigating...