Patrick, Jim, Dave:

we investigated the conlog crash at 16:00 PDT Friday 31st Dec 2015. As Patrick logged, h1conlog1-master syslogged that the kernel applied a leap second at that time. Looking at other machines we noticed that all systems which are NTP clients to the CDS NTP server (a Symmetricom SyncServer S250 unit with GPS antenna) similarly logged the leap second being appied whereas machines not using this NTP server did not.

The details of the conlog crash are: UNIX TIME (seconds since the 01jan1970 epoch) repeats a second when a leap second is added, we had a noisy channel being monitored by conlog which changed every second around this time resulting in conlog attempting to update the database with two events one second apart but with the same time stamp which crashed conlog and alerted us to the problem.

We then verified that all the clocks and computer times within CDS have the correct time, and they are indeed correct.

We logged into the Symmetricom NTP server and viewed the logs (attached). It shows that it reported the leap second addtion 30th June 2015 (correct) and 31st December 2015 (incorrect).

We are assuming that all the computer clocks did leap by one second at 16:00:00 PST Friday 31st December 2015, but then skewed back to the correct time soon after.

We are planning on upgrading to a new NTP server soon and donate the old server to GC.