attached email shows central ntp server reporting a NTP Stratum Change Alarm around the year change. Maybe this is related to conlog's problem? Further investigation is needed.
TITLE: 1/1 DAY Shift: 16:00-00:00UTC (08:00-04:00PDT), all times posted in UTC
STATE of H1: DOWN
Outgoing Operator: TJ
Quick Summary:
Walked in to find H1 down, but TJ was taking it right back to locking....specifics on this in upcoming alog.
O1 days 104,105
No restarts reported for both days.
BTW: DAQ has now been running for 31 days continuously. Only h1broadcaster (reconfiguration) and h1nds1 (crash) has been restarted in this time period. Both frame writers and trend writers have been stable.
While we were down I walked to the onsite warehouse and visually inspected the DCS (LDAS) cluster. Monitoring of the computers and HVAC showed all things were fine, and I've now verified this visually. It was also good to see that the HVAC system is running fine in this cold (18 degree F) weather.
Title: 1/1 OWL Shift: 08:00-16:00UTC (00:00-8:00PDT), all times posted in UTC
State of H1: Relocking
Shift Summary: Almost locked my entire shift, dropped 10min befrore the shift change from an earthquake.
Incoming Operator: Corey
Activity Log:
Most likely due to:
Title: 1/1 OWL Shift: 08:00-16:00UTC (00:00-8:00PDT), all times posted in UTC
State of H1: Observing at 79Mpc for 2hours
Outgoing Operator: Jeff B
Quick Summary: Happy New Year! Wind <6mph, useism 0.3um/s, CW inj running.
Activity Log: All Times in UTC (PT) 00:00 (16:00) Take over from Corey 00:44 (16:44) GRB alert. Spoke to LLO. In one hour hold for collecting background statistical data 01:30 (17:30) Dave B. called to say Conlog was down due to UTC year switch 01:44 (17:44) End one hour GRB hold 01:50 (17:50) Power reset Video2 to free up hung FOMs 02:40 (18:40) Lockloss – Due to mag 6.3 EQ south of Australia 02:55 (18:55) Dave B. has restarted Conlog 03:07 (19:07) Seismic up to 0.3um/s – Put IFO into down until the earth settles down a bit 05:10 (21:10) GRB Alert – LHO & LLO down due to EQ – Ignored alert 06:23 (22:23) IFO relocked and in Observing mode 08:00 (00:00) Turn over to TJ End of Shift Summary: Title: 12/31/2015, Evening Shift 00:00 – 08:00 (16:00 – 00:00) All times in UTC (PT) Support: Mike, Incoming Operator: TJ Shift Detail Summary: Lost lock about 3 hours into the shift due to a 6.3 mag EQ. After the seismic noise quieted, did an Initial alignment, and relocked the IOF with relative little trouble. IFO is currently in Observing mode with 21.8W of power and 79Mpc of range. Environmental conditions are generally good. HAPPY 2016!
After seismic 0.03-0.1 band dropped below 0.1um/s ran through initial alignment.
NOTE: Had no problems with the Guardian INPUT_ALIGN process completing successfully. Was able to "fine tune" MICH_DARK by hand.
Relocked the IFO on the first try with no problems and was in Observing mode at 06:23 (10:23). Power is at 21.8W and range is at 82Mpc.
NOTE: This is the first relock I've done in the past couple of week that FIND_IR Diff completed under Guardian control. All previous locks I had to tune IR Diff by hand.
IFO is down due to the EQ reported in an earlier post. Seismic was dropping nicely and was just about to a point where it might be possible to relock, when it took another leg up and is back around 1.0um/s. Will give it some more time before doing an initial alignment.
Lockloss at 02:40 (18:40) due to mag 6.3 EQ south of Australia. Seismic rang up to 3.0um/s. No locking of ALS right now. Put IFO into Down state until things settle down.
The conlog process on h1conlog1-master failed soon after the UTC new year. I'm pretty sure it did the same last year but I could not find an alog confirming this. I followed the wiki instructions on restarting the master process. I did initially try to mysqlcheck the databases, but after 40 minutes I abandoned that. I started the conlog process on the master and configured it for the channel list. After a couple of minutes all the channels were connected and the queue size went down to zero. H1 was out of lock at the time due to an earthquake.
For next year's occurance, here is the log file error this time around
root@h1conlog1-master:~# grep conlog: /var/log/syslog
Dec 31 16:00:12 h1conlog1-master conlog: ../conlog.cpp: 301: process_cac_messages: MySQL Exception: Error: Duplicate entry 'H1:SUS-SR3_M1_DITHER_P_OFFSET-1451606400000453212-3' for key 'PRIMARY': Error code: 1062: SQLState: 23000: Exiting.
Dec 31 16:00:12 h1conlog1-master conlog: ../conlog.cpp: 331: process_cas: Exception: boost: mutex lock failed in pthread_mutex_lock: Invalid argument Exiting.
This indicates that it tried to write an entry for H1:SUS-SR3_M1_DITHER_P_OFFSET twice with the same Unix time stamp of 1451606400.000453212. This corresponds to Fri, 01 Jan 2016 00:00:00 GMT. I'm guessing there was a leap second applied.
of course there was no actual leap second scheduled for Dec 31 2015, so we need to take a closer look at what happened here.
The previous line before the error reports the application of a leap second. I'm not sure why, since you are right, none were scheduled. Dec 31 15:59:59 h1conlog1-master kernel: [14099669.303998] Clock: inserting leap second 23:59:60 UTC Dec 31 16:00:12 h1conlog1-master conlog: ../conlog.cpp: 301: process_cac_messages: MySQL Exception: Error: Duplicate entry 'H1:SUS-SR3_M1_DITHER_P_OFFSET-1451606400000453212-3' for key 'PRIMARY': Error code: 1062: SQLState: 23000: Exiting. Dec 31 16:00:12 h1conlog1-master conlog: ../conlog.cpp: 331: process_cas: Exception: boost: mutex lock failed in pthread_mutex_lock: Invalid argument Exiting.
Received GRB alert at 00:44 (16:44). IFO is up in Observing mode. Spoke with LLO. I one hour hold to accumulate background information.
Transition Summary: Title: 12/31/2015, Evening Shift 00:00 – 08:00 (16:00 – 00:00) All times in UTC (PT) State of H1: 00:00 (16:00), IFO in Observing mode for 7 hours. Power is 21.8W and range is 79Mpc. The wind at the site is a light to gentle breeze (4 - 12mph). Seismic 0.03-0.1Hz band showed a bit of excitation for about 4 hours in the End-Y channel, which has subsided. Microseism remains flat at 0.4um/s. Outgoing Operator: Corey
TITLE: 12/31 DAY Shift: 16:00-00:00UTC (08:00-04:00PDT), all times posted in UTC
STATE of H1: Locked for 6+hrs at or under 80Mpc.
Incoming Operator: Jeff B.
Support: Kiwamu called in for OWL & DAY
Quick Summary:
After Kiwamu fought the fight early this morning, H1 has been in Observing ever since (even with a few notable [red Terramon] rumblers). Will be curious to see how H1 fares during its next Initial Alignment (see note for possible work-around for Input Align portion of Initial Alignment from Kiwamu's alog earlier).
There was warning (albeit after-the-fact) of another EQ at 22:40 from Alaska with motion of 0.6um/s, but it had no effect here.
~1505 hrs. local
Looks like a 5.4 New Zealand quake is a few minutes (22:18utc/218pmPST) from approaching with 0.5um/s motion. We'll see how we fair.
Around 22:13 Tidal, ASC (huge in yaw), and POP_A_LF all started getting big oscilations, but we appear to have ridden out the 22:18:52 R-wave arrival time. Didn't really see anything obvious in the 0.03-0.1 seismic band. Range has trended down to around 76Mpc over the last few minutes.
And all of this as the winds slowly get above 10mph.
Go H1! #knockonwood