PIT:
YAW:
This CLOSES FAMIS #4706
TITLE: 12/19 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 66.0999Mpc
OUTGOING OPERATOR: Ed
CURRENT ENVIRONMENT:
Wind: 3mph Gusts, 2mph 5min avg
Primary useism: 0.04 μm/s
Secondary useism: 0.49 μm/s, with a slight trend up over last 12hrs
QUICK SUMMARY:
Locked for almost 10hrs, albeit glitchy. Temperatures a balmy 16degF this morning.
Andy, following work from TJ and Josh My best guess for the cause of the extreme glitchiness is an electronic problem in the ETMY L2 driver. I think TJ and Josh were thinking the same thing, but I don't want to speak for them. We've been seeing the glitchiness get worse and worse at LHO for at least a week. These are short glitches, and now they are loud and frequent (once every few minutes at least). The humidity has been dropping as the weather gets cold, and last winter Robert and others showed that was correlated with increases in these kinds of glitches. We also see that the only associated channels are the ETMY L2 noisemons. These do see the drive signal, so they will show glitches whenever DARM does. But these are very loud. And I quickly developed some code to measure a transfer function from a few weeks ago, and use that to subtract out the drive signal. The subtraction seems to work well, but the glitches get shorter, sharper, and louder. So my hypothesis would be that the low humidity is exacerbating an electronics problem in the ETMY L2 driver, maybe due to sparking. While LLO is down, would it be possible to replace the driver with a spare, or at least try to diagnose whether it is working properly?
As an alternative to swapping the coil driver, one could also try swapping the control back to ETMX.
The transmission through the reference cavity has deteriorated somewhat since the last touch up alignment
last week. Went through the usual signals to see if anything matched up. Nothing from the usual signals.
However this time there seems to be coincidence with the horizontal output of the quadrant photodetector
in the ISS box. That in turn implies that the pointing out of the pre-modecleaner changed.
The change in horizontal seems to coincide with the drive voltage to the pre-modecleaner heater. From
the cavity transmission camera, it is obvious that the alignment has drifted in both pitch and yaw.
I noticed that GWIstat screen is reporting that H1 is NOT OK. I think this may have something to do with ODC bits not being "happy"? I checked the ODC overview and noticed some flashing of bits 10 and 31 in the ASC_ODC screen that has the status flashing between Green and Red. Maybe this is an issue maybe not. In any case, the intent bit is set to Observe but GWI says we're NOT OK.
Also bit 4
I see that the UTC time on the GWIstat display is frozen at 6:46 UTC, a number of hours ago. It looks like the machine that it runs on, ldas-grid.ligo.caltech.edu, developed a problem. I am trying to ssh to that machine now but it's not responding. I'll contact the Caltech sysadmins.
After a visual inspection of the Kitchen, Bathrooms and some of the offices, THere was no sig of any new water leaks. All fans were running. That's about it.
State of H1: Observe
Activity:
Attachments:
State of H1: in Observe
TITLE: 12/18 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: OBSERVING
INCOMING OPERATOR: Cheryl
SHIFT SUMMARY:
H1 continues to glitch. Been up for 7hrs. Fair amount of activity on a Sunday here due to burst pipe at the LSB.
LOG:
While OBSERVING with a glitchy H1 for the last 19hrs, there was mention of a possible culprit being noisy violin modes.
Before taking any action, I consulted with Keita (Run Coordinator) to ask whether this would be acceptable (because L1 is currently down and we would not jeopordize double coincidence). Keita pointed me to a note about diagnosing violin modes here, (alog 32615). So, I went through is diagnostics for determining whether we have ADC or DAC saturations (which is a main symptom of big violin modes).
Diagnostics for saturations due to violin modes:
1) Make sure OMC DCPD is far from +/- 32k, and ESD is far from +/- 131k, and this can be checked via timeseries.
2) Make sure there are no large values for a a spectrum of these channels
Both of these measurements are attached, and I believe they look OK (please check me on this). So I will not bother with an attempt of violin mode damping.
The decision not to damp violin modes seems like a good one today, but I would add two more things to check, since we have seen glitches around the violins and harmonics even when we are not close to having saturations. (ie alog 32026 for the symptom, for a solution see 32050)
First, you can check the glitch plot on the wall or the summary page (under Lock> glitches)for a line of glitches around the violin modes or harmoincs (500Hz, 1kHz, 1.5kHz ect).
Second, you can look at the main spectrum on the wall and see if the violin modes are high compared to the reference.
H1 has been locked for almost 4hrs. It continues to be a bit glitchy. L1 continues to be down for high seismic.
LSB Burst Pipe recovery work continues.
Talked with John & Bubba little bit about water pipe that burst yesterday.
Yesterday, a water pipe behind the dishwasher in the LSB kitchen burst (see attached photo). First inkling of an issue was alluded to in Cheryl's alog about very low water pressure here in the OSB. Apparently the pipe burst yesterday afternoon at some point, and within 2hrs all the water from the Potable water tank was evacuated into the LSB. Bubba started initial clean-up work yesterday and arranged for a clean-up crew to work today (Sunday).
While H1 was locked, I took a quick run down to the LSB to look at the damage. Christina was down there & helping out the clean up along with the crew which has been on-site since early morning. Christina walked me through the areas of damage. The water travel extensively through the building---from the kitchen it went to Fred's office in one direction and into the auditorium in another direction. Baseboards have been removed rom many walls, furniture has been moved, fans are running throughout.
UPDATE: A Second Pipe Burst?
Bubba just informed me a possible 2nd pipe may have burst. The Hanford Fire Dept are onsite to investigate with Bubba.
Arctic Temperatures!
We have had sub-freezing temperatures for the last 11 days (since 12/7), and the last 2 days have we have not made it above 15degF(see attached). Temperatures are due to warm up by Tues as we finally get above freezing.
Had FREEZING & busy shifts yesterday (Saturday)!
Kudos to the heroic work of the operators, all who came in & to the people who are still here cleaning up the LSB.
A Request for DetChar
The locks for the last 15hrs appear to be ratty/glitchy. Wondering if the DetChar team could point us to any culprits here. There's been a thought this could be alignment-related (there was the alignment recovery by Ed & Cheryl after the quake and Beckhoff failure yesterday, and then Travis & I performed another Initial Alignment today.) We can try another alignment next time we drop out of lock, but wondering if there could be something else amiss causing our glitchy H1 after the flurry of Saturday action.
I submitted an FRS (6965) for the downtime for this (~14hrs from when Beckhoff issue happened to when we were back to OBSERVING).
Have started getting this verbal alarm. For first one I went to check TCSy Chiller and topped it off.
But after getting more alarms, I went to check again and it was still full.
Looking at Chiller screen, can see that the Flow Rate is RED & attached is a trend of the last 6-hrs. Looks like there was a step down in flow rate. It might be coming back up slowly.....
(I'm also not very familiar with how the TCSy Laser works. Does it turn on/off normally during locks?)
I just spoke to Corey by phone. This is the same problem we have been seeing with the flow sensor. Corey has checked the power output of the laser ( at the thermopile on the laser output, which has a channel name of LSR_HD_PD_OUTPUT) and the laser is running and giving approx 56W. So it has not tripped off.
Altering the power output to the CP using the rotation stage would cause a change in the power registered at the power meter ( PWM_MTR_OUTPUT ) so assuming the IFO is setup to alter power during locking then you would expect that change. Will double check with commisioners what the standard settings are for power output during lock.
I'll check with Betsy/Jason/Nutsinee when they're planning on doing the flow meter swap out. It would be good to do it sooner rather than later I think, otherwise it could start tripping off that laser and putting us out of observing mode.