TITLE: 04/30 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 51Mpc
INCOMING OPERATOR: Nutsinee
SHIFT SUMMARY:
LOG:
Wind's reaching 40 mph but the IFO is still locked.
Got this message, in RED, twice on Verbal Alarms computer. 21:02 & 22:35UTC
@ 20:19 the H1SUSAUX FE crashed
TITLE: 04/30 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 65Mpc
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
Wind: Calm
Primary useism: 0.09 μm/s
Secondary useism: 0.25 μm/s
QUICK SUMMARY: ops lazy script isn't doing transition (-t). It didn't work for me yesterday either. The End (-e) funcion worked for me yesterday.
I ran a2l after seeing the measurement that Corey took just before he left
TITLE: 04/30 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 64Mpc
INCOMING OPERATOR: Ed
SHIFT SUMMARY:
Nice quiet shift. H1 locked for over 38.5hrs. L1 was under tornado warning.
5.2 Magnitude EQ from New Zealand originating at 12:22utc.
Nice steady range (with a few glitches). Currently in a wind lull. H1 locked for over 35.5 hrs.
(I think it's lunch time.)
TITLE: 04/30 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 65Mpc
OUTGOING OPERATOR: Nutsinee
CURRENT ENVIRONMENT:
Wind: 5mph
Primary useism: 0.01 μm/s
Secondary useism: 0.10 μm/s (at 50 percentile)
QUICK SUMMARY:
H1 locked for 3.5hrs & useism took a tiny step up 8hrs ago.
TITLE: 04/30 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 62Mpc
INCOMING OPERATOR: Corey
SHIFT SUMMARY: Quiet shift. Been locked and observing for 31.5 hours. No issue to report.
Been locked and observing for almost 27 hours. No issue to report. LLO is down due to >20mph wind (at least that's what I gather from their alog).
TITLE: 04/29 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Aligning
INCOMING OPERATOR: Nutsinee
SHIFT SUMMARY:
Basically quiet shift. There were a few tours through the control room. I ran the a2l script at the beginning ofthe shift while Livingston was down and the end of the shift when Livingston dropped out.
LOG:
15:30 big glitch
16:40 big glitch
19:16 I just noticed two guys with backpack sprayers, tending the grounds on the walkway between the Staging Building and the LSB.
22:55UTC YAW out by approx .75
23:03 Intention Bit Undisturbed
19:11UTC LLO Operator contacted for alert verification. 1 hr stand-down time initiated by Guardian INJ_TRANS node.
There is a write up of the bug which is possibly causing our front end issues here
I confirmed that our 2.6.34 kernel source on h1boot does have the arch/x86/include/asm/timer.h file with the bug.
To summarize, there is a low level counter in the kernel which has a wrap around bug with kernel versions 2.6.32 - 2.6.38. The bug causes it to wrap around after 208.499 days, and many other kernel subsystems assume this could never happen.
At 06:10 Thursday morning h1seiex partially locked-up. At 05:43 Friday morning h1susex partially locked-up. As of late Wednesday night, all H1 front ends have been running for 208.5 days. The error messages on h1seiex and h1susex consoles show timers which had been reset Wednesday night, within two hours of each other.
We are going on the assumption that the timer wrap-around has put the front end computers in a fragile state where lock-ups may happen. We don't know why only two computers at EX have seen the issue, or why around 6am, or why one day apart. Nothing happened around 6am this morning.
I am filing a work permit to reboot all H1 front end computers and DAQ computers which are running kernels with this bug.
attached is a list of 2.6.3x kernel machine and their runtime. Looks like h1nds0 will wrap-around Sunday afternoon.
h1nds0's timer will wrap around to zero about 11pm Sunday local time.
13:40UTC Spoke to Dave. He had me reset the FE computer. Everything came back up rather quickly and I was able to get back to Observing by 13:45UTC
This was a good test for Tuesday's planned reboots. We don't need to power cycle the IO Chassis when rebooting the computer. Since the computer's general core is locked up, the only way to reboot it is to press the front panel Reset button (a power cycle of the computer is not needed).
Looking at the up-times, we only need to reboot those with times close to or beyond 208 days, which means we don't have to immediately reboot:
h1psl0 up 133 days,
h1susauxh2 just rebooted
h1susauxh34 up 120 days
h1oaf0 up 166 days
h1susex booted Friday
h1seiex booted Thursday
h1iscex up 179 days
We have the option of deferring the h1psl0 reboot and the others until the 5/8 vent.