What happened:

An accidental button push took H1 out of Observe at 19:03:46UTC, and H1 was returned to Observe at 19:25:31UTC after the PSL diode chiller was filled.

Timeline Details:

• The filter, ETMY_L3_DRIVEALIGN_L2L FM3, was briefly turned off and back on, and the glitch was visible in H1 DARM.
• The filter was in a non-standard configuration approximately between 1145364642 and 1135364645 GPS time (times taken from dataview plot).
• During that time I was talking to Peter King - he called to let me know that the PSL Diode Chilller was in alarm, low water level.
• Trend of 7 day history of this alarm is attached, and shows that although the chiller was filled on the 24th, it quickly went back into alarm for low water level.
• History is that when filled, the amount added has been 125ml.
• I added 250ml today.
• This alarm should be watched closely in the next 24 hours to insure it doesn't go back into low water level alarm.
• I finished filling the chiller by 19:24:00UTC.
• I returned to the Control Room and returned H1 to Observe at 19:25:13UTC.

Given these activites, after ETMY L2L filter back was restored and resulting glitch in DARM passed, the IFO was not disturbed again until I filled the chiller between 19:19:00UTC and 19:24:00UTC.

Timeline Summary:

• 19:03:46 H1 out of Observe
• approx. 19:03:50, ETMY filter bank was back in standard config.
• H1 undisturbed for approx. 15 minutes
• 19:19:31UTC I take a "before" snapshot of the chiller warning
• I fill the chiller
• 19:23:52UTC I take the "after" snapshot of the chilller and then exit the chiller room
• 19:25:13UTC H1 is returned to Observe

IFO Guardian Log Details:

2015-12-27T08:01:00.52831 IFO [ALL_NODES_OK.enter]
2015-12-28T19:03:46.23209 IFO [ALL_NODES_OK.run] USERMSG: node NOTIFICATION: DIAG_SDF
2015-12-28T19:03:46.35568 IFO state returned jump target: DROP_OK
2015-12-28T19:03:46.35588 IFO [ALL_NODES_OK.exit]
2015-12-28T19:03:46.41705 IFO JUMP: ALL_NODES_OK->DROP_OK
2015-12-28T19:03:46.41724 IFO calculating path: DROP_OK->ALL_NODES_OK
2015-12-28T19:03:46.41748 IFO new target: WAITING_FOR_NODES_OK
2015-12-28T19:03:46.41790 IFO executing state: DROP_OK (5)
2015-12-28T19:03:46.41802 IFO [DROP_OK.enter]
2015-12-28T19:03:46.60496 IFO EDGE: DROP_OK->WAITING_FOR_NODES_OK
2015-12-28T19:03:46.60534 IFO calculating path: WAITING_FOR_NODES_OK->ALL_NODES_OK
2015-12-28T19:03:46.60572 IFO new target: ALL_NODES_OK
2015-12-28T19:03:46.60645 IFO executing state: WAITING_FOR_NODES_OK (20)
2015-12-28T19:03:46.60664 IFO [WAITING_FOR_NODES_OK.enter]
2015-12-28T19:03:47.92328 IFO EDGE: WAITING_FOR_NODES_OK->ALL_NODES_OK
2015-12-28T19:03:47.92369 IFO calculating path: ALL_NODES_OK->ALL_NODES_OK
2015-12-28T19:03:47.92443 IFO executing state: ALL_NODES_OK (100)
2015-12-28T19:03:47.92461 IFO [ALL_NODES_OK.enter]
 

Plots attached:

1. ETMY filter (approx. 3 seconds)
2. PSL Diode Chiller Low Alarm (30 minutes)
3. Both channels plotted together, showing the time in between where H1 is undistrubed, but not in Observe (23 minutes total, approx. 15 minutes of "unofficial undisturbed" time)

A glitch at 19:03:46 UTC (Dec 28, 2015) was caused by me accidentally pressing a filter switch in aSUS-ETMY front-end model.

The filter name that was touched is H1:SUS-ETMY_L3_DRIVEALIGN_L2L, it was turned off and back on within about a second, because I double-clicked on a switch button.

O1 days 100,101

model restarts logged for Sun 27/Dec/2015
2015_12_27 15:32 h1nds1

daqd process on h1nds1 unexpectedly stopped, had to be manually restarted.

model restarts logged for Sat 26/Dec/2015 No restarts reported

Ops Day Shift: 16:00-23:59UTC, 08:00-16:00PT

State of H1: locked in Observe for 33 hours, range is 81.5Mpc

On Site: Darkhan and DickG

Commissioning Meeting at 8:30:

• MikeL called in
• MikeL expressed a preference to run through Maintenance tomorrow
• only known commissioning/calibratoin activity for tomorrow is EX Pcal - Darkhan said it is ok to delay until next week

Shift Details as of 17:20UTC

• four ETMY saturations, at 16:46:03, 16:48:46, 17:10:19, and 17:18:11UTC
• range dropped to approx. 5Mpc for the saturations at 16:48 and 17:10UTC
• no wind - 0mph across the site
• useism at the Corner Station in Z direction is at 0.5um/s and is increasing
• useism at EY and EX, at Y direction and X direction respectively, is between 0.2 and 0.3um/s, and looks pretty flat for the past 8 hours

ISI Blend Filters - snapshot of current configuration attached

TITLE: Dec 28 OWL Shift 08:00-16:00UTC (00:00-08:00 PDT), all times posted in UTC

STATE Of H1: Observing @ 82Mpc

SUPPORT: N/A

INCOMING OPERATOR: Cheryl

SHIFT SUMMARY: Great night! IFO locked for 32 hours at ≈81Mpc. One small earthquake cause things to wiggle a bit up to .1µm/s. µSei is still hovering ≤ 90th percentile. Winds are calm. Mike called to check in. It’s snowing. Handing off to Cheryl.

Mid-SHIFT SUMMARY: IFO locked for 28 hours at 81Mpc (82.4Mpc max). Environmental conditions are the same as shift transition.  One small glitch at ≈ 11:45UTC that caused a drop in range of ≈ 10Mpc. No saturations. LLO still down.

TITLE: Dec 28 OWL Shift 08:00-16:00UTC (00:00-08:00 PDT), all times posted in UTC

STATE Of H1: Observing

OUTGOING OPERATOR: Jim

QUICK SUMMARY: IFO has been locked in the 80Mpc range for ≈ 24.5 hours. Seismic conditions in the EQ bands are back to nominal after some small activity and µSei is ≤ 90th percentile. Wind is ≤5mph. ISI blends are all 45mHz in the corner and all 90mHz at the ends except for ETMY on axis. Livingston is down due to high µSeism.

Title: 12/27 Eve Shift 0:00-8:00 UTC

State of H1: Observing

Shift Summary: Quiet shift, range has been somewhat less steady than other times, but no obvious  reason for it

Activity log:

5:45 Kyle to MY to work on CP3, done 6:00

This was a very quiet and uneventful shift. Spectrum looked good all night except for a few saturations at EY. The range was a but drifty, going down to the low 70Mpc's at times and then returning for no apparent reason. Spectra of auxiliary DOFs and RF45 stayed right on the reference curves. Omicron recorded 2-4 loud triggers per hour. In last half hour of shift we had elevated seismic due to nearly concurrent smaller earthquakes in Martinique and Alaska. This seemed to excite Dhard a bit but no noticeable effect on the range.

2150 hrs. local 

Overfilled CP3 manually -> I prefer the new procedure of opening the LLCV bypass valve 1/2 turn ccw then partially opening the exhaust check valve bypass valve to minimize back pressure.  Once LN2 is observed flowing out of the exhaust (~2 minutes), closing the LLCV bypass valve then closing the exhaust check valve bypass valve - in that order.  This should reduce the maximum exhaust pressure.

NOTE:  I did not open the instrument air pressure regulator bowl drain to check for dryness tonight, i.e. change the air pressure at the CDS controlled solenoid valve but nevertheless, the CDS MEDM "START/enable STOP/disable" indicator went from green to red again and I had to re-enabled it - why?  Manually filling the 80K pump using the above method only changes the 80K pump exhaust pressure and should have no effect on the instrument air. 

Gerardo is scheduled to do the next CP3 overfill exercise sometime before 5:00 p.m. hrs. local tomorrow, Monday.

Another quiet shift. Microseism and wind are both low, RF45 has been quiet. Range had been drifting down to about 70 Mpc when I came in, but seems to have at least stabilized or even come back up a little. Lock has lasted about 21 hours.

Title: 12/27 Day Shift 16:00-24:00 UTC (8:00-16:00 PST).  All times in UTC.

State of H1: Observing

Shift Summary: In Observing for my entire shift and 16 hours total now.  No issues related to observation this shift.  Microseism ~0.4 um/s and wind ~7 mph.

Incoming operator: Jim

Activity log:

23:30 Contacted Dave regarding NDS1 issues.

22:37 Dave called back reporting he had restarted NDS1 and it should be back up and running. Did a quick trend of H1 range for 2 days as a check of access to old data.  Worked fine.

With only a few ETMY Saturations not associated with the RF45, the shift has relatively little to remark on in data quality.  The wind has lumped up a couple times but never more than 10 mph or so.  The useism after trending down for a day has swung back up and again maybe peaked, running at ~.3um/sec.  Barker the restarted the nds1 machine after it died.

the daqd process on h1nds1 stopped running at 14:51 PST, log file details shown below. Looking back I see many of these "NDS failed" messages from the 12/24 21:14 evening restart (following the computer freeze up). I restarted the daqd process and then stopped and started the nds process. If we are having any problems getting archived data, we could try rebooting the machine.

[Sun Dec 27 14:51:07 2015] read(seq_num) from NDS failed; errno=0

[Sun Dec 27 14:51:07 2015] connection closed on fd=186

[Sun Dec 27 14:51:09 2015] read(seq_num) from NDS failed; errno=0

[Sun Dec 27 14:51:09 2015] connection on fd 144 closed

[Sun Dec 27 14:51:09 2015] connection closed on fd=226

[Sun Dec 27 14:51:09 2015] connection on fd 110 closed

Observing for the past 12 hours.  Microseism and wind are both up a bit, but still at manageable levels.  There is a dusting of ~1/2 inch of snow on site.

I won't be able to come in during the afternoon today so I did a fill this morning (~0720 hrs. local) and will back tonight ~2200 hours or so.  


Pressure step (~0100 hrs. local) depicted in attached shows up on Y-arm gauges but no indication that IP9 is responsible.  

Title: 12/27 Owl Shift 08:00-16:00 UTC (00:00-08:00 PST).  All times in UTC.

State of H1: Observing at 80 Mpc for 8 hours.

Shift Summary: Another very quiet shift. Useism is trending up and reaching 90th percentile. Almost 0 wind.

Incoming operator: Travis

Activity:

15:14 Kyle called letting me know that he's driving down to EY to check on something.

15:38 Kyle called that he's back in the corner station