FYI:

1. Note that the good news is I see H1 low latency hoft making it to CIT, and it is being aggregated. Also it seems the low-latency analysis can determine the lock segments from the hoft frames. 

Thus, the problems listed below are something we can investigate Monday morning. An email has been sent to DASWG and the relevant persons about this.

2. However, the redundant copy of hoft written by the DMT seems to have
stopped today at:

Apr 30 11:21 PDT.

Much earlier today, the DMT logs report Input/output error like this:

$ tail  logs/H1_rename_Frames-2017.04.25-09.43.log

‘/gds-h1/dmt/frames/hoft/H1/.temp/H-H1_DMT_C00-1177574268-4.gwf’ ->
‘/gds-h1/dmt/frames/hoft/H1//H-H1_DMT_C00-117757/H-H1_DMT_C00-1177574268-4.gwf’
mv: cannot move
‘/gds-h1/dmt/frames/hoft/H1/.temp/H-H1_DMT_C00-1177574268-4.gwf’ to
‘/gds-h1/dmt/frames/hoft/H1//H-H1_DMT_C00-117757/H-H1_DMT_C00-1177574268-4.gwf’:
Input/output error

3. Also, thus summary pages show dropout of the H1 lock segments

https://ldas-jobs.ligo-wa.caltech.edu/~detchar/summary/day/20170430/

and nagios reports errors like,

https://dashboard.ligo.org/cgi-bin/nagios3/status.cgi?host=all&servicestatustypes=28

"Known segments in frames that aren't in database"

Thus, low-latency data is making it to CIT, I think with the correct
locked segments, but these are not making into the segment database.

This is something we'll investigate Monday morning.

Been locked and Observe for almost 52 hours. The wind is still~30 mph.

Amplitude of 32761 Hz PI (Mode23 in LHO damping scheme) correlates with 3 - 30 Hz seismic motion. There is also a corresponding comb that rises in the PI OMC channel at 2*(Nyquist - PI frequency), where Nyquist = 32768 Hz and PI freq ~ 32761 Hz. 

Thanks to the summary pages, I'd noticed that the 28 - 32 kHz PI band somewhat correlated with SenseMon range. I narrowed this correlation down to 3 - 30 Hz seismic motion (which we now know correlates with range due to input beam scattering) and PI ETMY Mode 23 aka ~32761 Hz. Correlation seems to be only with this mode, ruling out electronics coupling since a mode sensed right next door in frequency world has no hardware difference - both are sensed with H1:OMC-PI_DCPD_64KHZ_AHF_DQ. First attachment is 32 hr stretch showing correlation between narrow OMC-PI band, seismic, and range. 

Kiwamu then did a drive/damp test of this mode and found narrow lines appeared in DARM while driving/damping well below saturation. Looking at the OMC channel, he actually saw just the first peak of a 14.5 Hz comb seen in OMC-PI, as well as 64, 78.5, 142.5 Hz lines. Second attachment shows comb while he was driving. This ~15 Hz comb is visible in OMC-PI anytime Mode23 is rung up enough (haven't quantified 'enough' yet) and first peak is always 2x the frequency difference of 32768 Hz (OMC-PI's Nyquist) and Mode23 frequency; I've confirmed this to within .01 Hz.  Third attachment shows presence of first peak in comb during a typical day's Mode23 non-driven ring up. I haven't seen the other peaks Kiwamu saw (64, etc.) any other time.

This is an actual mode ringing up (as opposed to just changing in sensing at OMC) as it's seen in the TransMon QPD's (and all feed forward is off for this mode). These have much lower SNR for PI so mode has to be unusually high to see. Fourth attachment shows a recent day with high 3 - 30 Hz seismic activity where amplitude correlations are visible everywhere. Fifth attachment shows spectrum during the peak just after 5 hours in previous attachment. Note that in TransMon channel, signal has been demoded with 32800 Hz, so two peaks are (32800 - 32761) = 39 Hz and (32800 + 32761 - 65536) = 25 Hz.

TITLE: 04/30 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 51Mpc
INCOMING OPERATOR: Nutsinee
SHIFT SUMMARY:
LOG:

Wind's reaching 40 mph but the IFO is still locked.

Got this message, in RED, twice on Verbal Alarms computer. 21:02 & 22:35UTC

@ 20:19 the H1SUSAUX FE crashed

TITLE: 04/30 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC

STATE of H1: Observing at 64Mpc

INCOMING OPERATOR: Ed

SHIFT SUMMARY:

Nice quiet shift.  H1 locked for over 38.5hrs.  L1 was under tornado warning.

5.2 Magnitude EQ from New Zealand originating at 12:22utc.

• ~13:10utc:  Seen on USGS, Terramon, tidal & ISI CONFIG, & BLRMS (0.3um/s at 0.03-0.1Hz), DIAG_MAIN EQ message
• Seismon:  Nothing
• H1 remained locked.  L1 already down.

Nice steady range (with a few glitches). Currently in a wind lull. H1 locked for over 35.5 hrs.

(I think it's lunch time.)

TITLE: 04/30 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 65Mpc
OUTGOING OPERATOR: Nutsinee
CURRENT ENVIRONMENT:
    Wind: 5mph
    Primary useism: 0.01 μm/s
    Secondary useism: 0.10 μm/s (at 50 percentile)
QUICK SUMMARY:

H1 locked for 3.5hrs & useism took a tiny step up 8hrs ago.

TITLE: 04/30 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC

STATE of H1: Observing at 62Mpc

INCOMING OPERATOR: Corey

SHIFT SUMMARY: Quiet shift. Been locked and observing for 31.5 hours. No issue to report.

Been locked and observing for almost 27 hours. No issue to report. LLO is down due to >20mph wind (at least that's what I gather from their alog).

TITLE: 04/29 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Aligning
INCOMING OPERATOR: Nutsinee
SHIFT SUMMARY:

Basically quiet shift. There were a few tours through the control room. I ran the a2l script at the beginning ofthe shift while Livingston was down and the end of the shift when Livingston dropped out.
LOG:

15:30 big glitch

16:40 big glitch

19:16 I just noticed two guys with backpack sprayers, tending the grounds on the walkway between the Staging Building and the LSB.

22:55UTC YAW out by approx .75

There is a write up of the bug which is possibly causing our front end issues here

I confirmed that our 2.6.34 kernel source on h1boot does have the arch/x86/include/asm/timer.h file with the bug.

To summarize, there is a low level counter in the kernel which has a wrap around bug with kernel versions 2.6.32 - 2.6.38. The bug causes it to wrap around after 208.499 days, and many other kernel subsystems assume this could never happen. 

At 06:10 Thursday morning h1seiex partially locked-up. At 05:43 Friday morning h1susex partially locked-up. As of late Wednesday night, all H1 front ends have been running for 208.5 days. The error messages on h1seiex and h1susex consoles show timers which had been reset Wednesday night, within two hours of each other.

We are going on the assumption that the timer wrap-around has put the front end computers in a fragile state where lock-ups may happen. We don't know why only two computers at EX have seen the issue, or why around 6am, or why one day apart. Nothing happened around 6am this morning.

I am filing a work permit to reboot all H1 front end computers and DAQ computers which are running kernels with this bug.