TITLE: 04/30 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC

STATE of H1: Observing at 64Mpc

INCOMING OPERATOR: Ed

SHIFT SUMMARY:

Nice quiet shift.  H1 locked for over 38.5hrs.  L1 was under tornado warning.

5.2 Magnitude EQ from New Zealand originating at 12:22utc.

• ~13:10utc:  Seen on USGS, Terramon, tidal & ISI CONFIG, & BLRMS (0.3um/s at 0.03-0.1Hz), DIAG_MAIN EQ message
• Seismon:  Nothing
• H1 remained locked.  L1 already down.

Nice steady range (with a few glitches). Currently in a wind lull. H1 locked for over 35.5 hrs.

(I think it's lunch time.)

TITLE: 04/30 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 65Mpc
OUTGOING OPERATOR: Nutsinee
CURRENT ENVIRONMENT:
    Wind: 5mph
    Primary useism: 0.01 μm/s
    Secondary useism: 0.10 μm/s (at 50 percentile)
QUICK SUMMARY:

H1 locked for 3.5hrs & useism took a tiny step up 8hrs ago.

TITLE: 04/30 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC

STATE of H1: Observing at 62Mpc

INCOMING OPERATOR: Corey

SHIFT SUMMARY: Quiet shift. Been locked and observing for 31.5 hours. No issue to report.

Been locked and observing for almost 27 hours. No issue to report. LLO is down due to >20mph wind (at least that's what I gather from their alog).

TITLE: 04/29 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Aligning
INCOMING OPERATOR: Nutsinee
SHIFT SUMMARY:

Basically quiet shift. There were a few tours through the control room. I ran the a2l script at the beginning ofthe shift while Livingston was down and the end of the shift when Livingston dropped out.
LOG:

15:30 big glitch

16:40 big glitch

19:16 I just noticed two guys with backpack sprayers, tending the grounds on the walkway between the Staging Building and the LSB.

22:55UTC YAW out by approx .75

• Contacted Livingston: IFO had been down due to artifacts from commissioning left un-reverted. They got back in it about 3 hrs ago
• 2 tour groups in before noon
• 2 pretty big glitches about 1 hr apart 15:30 and 16:40
• Keita called to check in
• winds rising to around 20mph
• range has been approx 65Mpc

19:11UTC LLO Operator contacted for alert verification. 1 hr stand-down time initiated by Guardian INJ_TRANS node.

There is a write up of the bug which is possibly causing our front end issues here

I confirmed that our 2.6.34 kernel source on h1boot does have the arch/x86/include/asm/timer.h file with the bug.

To summarize, there is a low level counter in the kernel which has a wrap around bug with kernel versions 2.6.32 - 2.6.38. The bug causes it to wrap around after 208.499 days, and many other kernel subsystems assume this could never happen. 

At 06:10 Thursday morning h1seiex partially locked-up. At 05:43 Friday morning h1susex partially locked-up. As of late Wednesday night, all H1 front ends have been running for 208.5 days. The error messages on h1seiex and h1susex consoles show timers which had been reset Wednesday night, within two hours of each other.

We are going on the assumption that the timer wrap-around has put the front end computers in a fragile state where lock-ups may happen. We don't know why only two computers at EX have seen the issue, or why around 6am, or why one day apart. Nothing happened around 6am this morning.

I am filing a work permit to reboot all H1 front end computers and DAQ computers which are running kernels with this bug.

15:03UTC

TITLE: 04/29 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 64Mpc
INCOMING OPERATOR: Ed
SHIFT SUMMARY:

Nice shift with H1 being locked for over 14.5hrs and only a trio of glitches.  Luckily no frontend issues (knock on wood).  L1 continues to be down due to possible alignment issues post-EQ from yesterday.  Chatted briefly with Doug L who said Adam M was inbound.
LOG:

• 8:17:24utc Glitch
• 11:31 Alaska EQ quickly rolled through
• 12:00 Surprised to see Karen arriving.  Here for post-meeting clean up.

While looking into possible sources of glitches (noticed one Evan mentioned (alog#35800) earlier this week about ITMx glitching every few seconds....is this acceptable?). 

I was using the Oplev Overview screen to grab the oplev sum channels and while doing that I noticed that the buttons on it for the SUS screens looked different from the main SUS screens one gets via the Sitemap.  The problem button links on the oplev sum screen were for the ITMx, BS, & ITMy.  The windows which I clicked open had INVALID/white areas and they just looked subtly different.  (perhaps this medm screen is calling out old SUS screens.

Attached is a screenshot showing the difference for the BS.  The Oplev Screen opens a screen called SUS_CUST_HLTX_OVERVEW.adl (vs the one from the sitemap which is SUS_CUST_BSFM_OVERVIEW.ADL).

Alaska 5.4magnitude EQ which originated from there at 11:15utc

• Timeline Of What Was Seen:
  • Backstory:  10:30 BLRMS slowly started increasing an hour before, but this could have been from Colombian quake, too.  
  • 11:25-11:35 BLRMS shows clear peak up to 0.5um/s
  • 11:27 Terramon
  • 11:31 seen on tidal signals, ISI CONFIG, ASC CONTROL SIGNALS
  • 11:33 USGS shows RED dot in Alaska
• No audible alarm from Seismon On VERBAL ALARM
• Results:  H1 remained LOCKED through it and no decrease in BNS Range (L1 was already down from EQ earlier in the day).

Fairly quiet.  Two glitches on H1.

See BLRMS 0.03-0.1Hz seismic elevating.

• Terramon shows an earthquake from Colombia with an R-wave arrival earlier (at 10:36utc), which might correspond with BLRMS activity.
• H1SEISMON has an ORANGE box for the R20-Wave with an arrival of (after tconvert-ing gps time...can't we make these UTC times??) 11:56utc.  So we'll see what happens.