I noticed that GWIstat screen is reporting that H1 is NOT OK. I think this may have something to do with ODC bits not being "happy"? I checked the ODC overview and noticed some flashing of bits 10 and 31 in the ASC_ODC screen that has the status flashing between Green and Red. Maybe this is an issue maybe not. In any case, the intent bit is set to Observe but GWI says we're NOT OK. 

After a visual inspection of the Kitchen, Bathrooms and some of the offices, THere was no sig of any new water leaks. All fans were running. That's about it.

State of H1: Observe

Activity:

• 04:51:14UTC - H1 lock loss
• adjusted IM1, IM2, and IM3 alignments - target Dec 8th, as shown on medm under IO, I think their still good
• relocked
• HAM6 ISI tripped while shutter was tested
• 07:05:04UTC - H1 in Observe
• 7:33:50UTC -GRB alert

Attachments:

• 11 days = IM alignment, IM4 Trans, and ISS2 - ISS2 yaw has changed, but look at the next plot of IMC
• 11 days - IMC optics - MC1 pitch and yaw have changed 20urad, and this is down from the peak, where the change was about 25urad

State of H1: in Observe

TITLE: 12/18 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: OBSERVING
INCOMING OPERATOR: Cheryl
SHIFT SUMMARY:

H1 continues to glitch.  Been up for 7hrs.  Fair amount of activity on a Sunday here due to burst pipe at the LSB.
LOG:

• EY Saturations:  17:18....many.
• 16:37 EY Instrument Air
• 17:07 OBSERVING
• 17:55 "EY temperature is high" Verbal Alarm
• 18:25 Bubba & John moving from EX to EY Air Handler Room (no VEA entrance).
• ~20:00 Sometime around noonPST, Hanford Fire Truck was driving around the LVEA.  They are here to work with Bubba on a -2nd pipe burst- at the LSB.
• Getting "IM1 P is out of range" DIAG_MAIN messages.

While OBSERVING with a glitchy H1 for the last 19hrs, there was mention of a possible culprit being noisy violin modes. 

Before taking any action, I consulted with Keita (Run Coordinator) to ask whether this would be acceptable (because L1 is currently down and we would not jeopordize double coincidence).  Keita pointed me to a note about diagnosing violin modes here, (alog 32615).  So, I went through is diagnostics for determining whether we have ADC or DAC saturations (which is a main symptom of big violin modes). 

Diagnostics for saturations due to violin modes:

1)  Make sure OMC DCPD is far from +/- 32k, and ESD is far from +/- 131k, and this can be checked via timeseries. 

2)  Make sure there are no large values for a a spectrum of these channels

Both of these measurements are attached, and I believe they look OK (please check me on this).  So I will not bother with an attempt of violin mode damping.

H1 has been locked for almost 4hrs.  It continues to be a bit glitchy.  L1 continues to be down for high seismic.

LSB Burst Pipe recovery work continues.

Talked with John & Bubba little bit about water pipe that burst yesterday.

Yesterday, a water pipe behind the dishwasher in the LSB kitchen burst (see attached photo).  First inkling of an issue was alluded to in Cheryl's alog about very low water pressure here in the OSB.  Apparently the pipe burst yesterday afternoon at some point, and within 2hrs all the water from the Potable water tank was evacuated into the LSB.  Bubba started initial clean-up work yesterday and arranged for a clean-up crew to work today (Sunday).  

While H1 was locked, I took a quick run down to the LSB to look at the damage.  Christina was down there & helping out the clean up along with the crew which has been on-site since early morning.  Christina walked me through the areas of damage.  The water travel extensively through the building---from the kitchen it went to Fred's office in one direction and into the auditorium in another direction.  Baseboards have been removed rom many walls, furniture has been moved, fans are running throughout.  

UPDATE:  A Second Pipe Burst?

Bubba just informed me a possible 2nd pipe may have burst.  The Hanford Fire Dept are onsite to investigate with Bubba.

Arctic Temperatures!

We have had sub-freezing temperatures for the last 11 days (since 12/7), and the last 2 days have we have not made it above 15degF(see attached).  Temperatures are due to warm up by Tues as we finally get above freezing.

Had FREEZING & busy shifts yesterday (Saturday)!  

• Magnitude 8.0 EQ during OWL shift (Travis)
• Beckhoff Failure for ~6hrs of DAY shift (Ed & a crew of many others who came in & helped remotely)
• Alignment Restoration Work for end of DAY shift & into EVE shift. (Cheryl)
• ---> And somewhere in here we had a pipe burst in the LSB! (temps have been in the single digits for days)<---

Kudos to the heroic work of the operators, all who came in & to the people who are still here cleaning up the LSB.

A Request for DetChar

The locks for the last 15hrs appear to be ratty/glitchy.  Wondering if the DetChar team could point us to any culprits here.  There's been a thought this could be alignment-related (there was the alignment recovery by Ed & Cheryl after the quake and Beckhoff failure yesterday, and then Travis & I performed another Initial Alignment today.)  We can try another alignment next time we drop out of lock, but wondering if there could be something else amiss causing our glitchy H1 after the flurry of Saturday action. 

I submitted an FRS (6965) for the downtime for this (~14hrs from when Beckhoff issue happened to when we were back to OBSERVING).

Have started getting this verbal alarm.  For first one I went to check TCSy Chiller and topped it off.

But after getting more alarms, I went to check again and it was still full.

Looking at Chiller screen, can see that the Flow Rate is RED & attached is a trend of the last 6-hrs.  Looks like there was a step down in flow rate.  It might be coming back up slowly.....

(I'm also not very familiar with how the TCSy Laser works.  Does it turn on/off normally during locks?)

Have been locked for an hour.  Continue with EY glitches (maybe a little less prevalent than overnight?). 

I went ahead and ACCEPTED the PZT offsets Travis changed due to his IMC locking issues earlier:

• H1:IMC-PZT_PIT_OFFSET:  -650 --> 235
• H1:IMC-PZT_YAW_OFFSET:  6500 --> 6680

TITLE: 12/18 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Travis
CURRENT ENVIRONMENT:
    Wind: 2mph Gusts, 1mph 5min avg
    Primary useism: 0.03 μm/s
    Secondary useism: 0.30 μm/s
QUICK SUMMARY:
Tagged in with Initial Alignment from Travis (he finished INPUT ALIGN).  Michelson required going to DOWN once.  SRC didn't look right, but taking it DOWN once did the trick.  Now locking up.

Saw renovation crew at LSB when I drove in.  Travis said Bubba/John were at EY per his alog.  

L1 continues to have high seismic noise (ours useism appears to be trending down & winds are calm).

TITLE: 12/18 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Aligning
INCOMING OPERATOR: Corey
SHIFT SUMMARY:  Up and down all night for various reasons.  After the last lockloss, IMC was having difficulties locking.  I tracked it down to PZT offsets.  Running IA now.  Lots of EY saturations tonight during the locks making for ratty looking range plots.  Hopefully a fresh IA will help with this.
LOG:

See previous aLogs.

15:13 Restoration crew working on mopping up LSB.

15:47 John and Bubba to EY to look at compressor. 
 

PI mode 9 rang up faster than I do anything about. 

Had to accept 2 SDF diffs for CS ECAT PLC2 to go to Observing.  See screenshot.

1215 hrs. local -> Kyle on site to check on 80K pumps

Local temperature are dipping into the "single digits" which means now is an opportune time to investigate if our old problem of over-filling or under-filling of 80K pumps will result from frozen LLCV valve stems is going to be a problem.  Recall that in the past this had occurred but could not be confidently attributed to water freezing in the old pneumatic actuators or from water freezing in the valve stem from having got past too loose packing nuts. Or, both for that matter.  Chandra R. and John W. had replaced the pneumatic actuators with electric (water tight) ones last spring so now is the first real world test.  Initially, I'm on site to verify that the recently elevated exhaust pressures on CP3 and CP4 weren't the result of continuous overfilling -> Inspection of the Y-mid's CP3 and CP4 showed neither was overfilling and both are fine for now (remnant standing LN2 in lines from yesterday's overfill?).  

By dumb luck, I found that CP6 was indicating a frozen valve stem while I was on site - "Go figure!" -> As found, the local actuator pointer was showing ~30% open and the actuator motor was making noise.  The PID output was "railed" at its programmed minimum of 20% open.  I had John W. (he and Bubba also on site for separate instrument air issue) switch CP6's level control to manual 50% open -> the motor then went quiet and it stroked open to 50% as per the pointer.  Next, I had him change the value to 0% open and the stem stroked closed smoothly until stopping at about the 30% open value (as indicated by the local pointer).  I then tested the valve stem packing nut and found that it was completely loose, i.e. no thread engagement.  So, even though the local heat tape is wrapped around most of the exposed valve stem, was energized and working, water which made it past the ineffective packing washer was able to freeze nonetheless.  This is because this area is about 2" away from the closest heat tape wrap ->  I heated up the frozen area with a heat gun and lightly engaged the packing nut.  I then had John W. alternate between 100% open and 0% open to demonstrate that the valve could stroke through its whole range smoothly etc..  I setup and am leaving a heat lamp to prevent this unit from refreezing (which it would do, now that water is past the packing washer) and had John W. reduce the Beckhoff PID minimum to 15% open down from its as found value of 20%.  

 1535 hrs. local -> Kyle leaving site