With LHO closed over the holidays, and O3 still taking place 24/7 over the holidays, Site-Walkthrus have been schedule to be performed by operators during their shifts (DAY). I performed one this morning per instructions here. Since inspection of the LVEA, EX VEA, EY VEA are required, these incursions would require us to go out of OBSERVING. I chose to lump those three inspections together to minimize/optimize time we were out of OBSERVING (where I would take H1 out of OBSERVING, walk through the LVEA, and then drive out to EY & then EX consecutively, AND then finally return H1 to OBSERVING & finish remaining items of the Walk-Through.)
This morning I had H1 down for 35min to run calibration measurements, I was going to tack on VEA incursions to this down time, but we received an Earthquake Alert, so instead, I took H1 to OBSERVING & monitored the EQ in case we needed to transition SEI_CONF to the EARTHQUAKE state (this wasn't needed). Instead, it was observed the Calibration measurement prossibly affected Calibration Lines. Took H1 out of Observing, fixed this, and then began VEA incursions. Here is log:
Duration Total of Site Walk-Thru: ~90min (18:14 - 19:45utc)
Duration out of OBSERVING: ~51min
- 18:10 Out of OBSERVING (first to address PCal Lines & then start Walk-thru)
- Buddy notified
- 18:15-18:28 In LVEA (H1 was in NLN, so I tiptoed!)
- Heard a faint clicking noise while quietly walking around (appeared to come from the ceiling; I'm guessing this was HVAC-related.)
- 18:30-19:02 VPW, EX, & EY
- At End Station VEAs, I was reluctant to enter the VEAs since I was worried about disrupting/affecting the BRS systems. So I only entered and walked a few steps into the VEAs. Do not see the effects of me entering the VEAs on the BLRMs seismic traces.
- Noticed ice build-up on various parts of half of the site Chiller dewars.
- EX had a few dead bugs on floor.
- 19:02 Back to OBSERVING! (then continued with walk-thru of other buildings----I did not enter Mid Stations.)
- For RO Room at the Wood Shop building, heard water dripping, but this is from lines run to a floor drain.
- At LSB, noticed a white Nissan sedan parked in the LSB (person was in car, but no contact was made).
- Needed my FOB to swipe into the LDAS room of the VPW.
- Failed to check the janitor's closet in the Staging Bldg bottom floor.
- 19:45 Back in Control Room (texted buddy & notified LLO/Virgo Control Rooms)
Ice On Liquid Nitrogen Dewars
One item we were asked to check was the exhaust of the LN2 dewars as we drove the site. I wasn't totally sure where the exhaust was, but eventually figured it out.
As you inspect the dewars there is frost/ice build up at various points (i.e. input plumbing where Norco connects, output to VEAs, & output exhausts [see attached photos for examples]). I texted Chandra and she mentioned alot of this is normal, but she said posting photos is good. Found ice on exhausts at End Stations and LVEA. Notes:
- At EX I tried to figure out where the exhaust was and I eventually found it, there was a block of ice connected from the exhaust port all the way to the ground---I was able to break this fairly easily.
- CP2 exhaust had a big chunk of ice. I was able to break connection to the ground, but was not able to break off the ice frozen to the output pipe.
- While breaking off ice to EY's dewar exhaust, noticed that this output plumbing was not secured. So it would move like a big pendelum/spring if you touched it.
- DetChar if you see some seismic bumps around these times: 18:55, 19:30, 19:35, this is when I bonked ice chunks off the exhausts...one other instance I don't have time for.
~23:28 Notice the SEISMON Dead RED box is gone and DIAG MAIN for this is also gone. And Seismon has more recent events listed on its medm screen.
But I do not see the box flashing on the SEISMON medm screen (it was flashing when we had the issues though).
It looks like it's back up and running now. Not sure at all what happened. Will try to summarize here.
I let Dave and Corey try to recover following the instructions on the wiki (which are still accurate as far as I know) : https://cdswiki.ligo-wa.caltech.edu/wiki/seismon
I then tried following the same steps, but also deleted the old data for the USGS productclient (which took many minutes to complete, the last time this may have been done is July or maybe February).
I then told Dave it would be worth trying to restart the VM. After he did that, I expected the seismon processes to restore on their own, but they didn't after several minutes. So I follow the wiki procedure again.
Seismon screen and DIAG_MAIN still reported dead, which can be due to the seismon calculated GPS time not updating. I poked around in the epics code and added a couple lines to us gpstime to get the right gpstime, then just write that. Still didn't seem to work. Then a couple restarts later (from me messing with just the seismon epics client) it just started working. The only material change to the code was something like adding:
gt=gpstime.now.gpstime() -> whatever the right call is to get the current gps time, I don't remember
counter=int(gt)
caput(seismon_gps_epics_time_channel, gt)
I tried to get the epics code to print the "counter" value in STDOUT, so I could see it was updating in the terminal, but couldn't get python to be happy with doing that. I think that would be a useful indicator in the code. The only thing I can think of is that sometimes after deleting the USGS data, it can take a little while for the USGS code to get to recent enough earthquakes for the seismon code to be happy.
Also, when I try to plot the eq reponse plot from home, python complains about not being able to find the epics package. I don't remember seeing that before.
We are able to bring up the EQ Response plots in the Control Room.