aLIGO LHO Logbook

H1 CDS

david.barker@LIGO.ORG - posted 19:00, Thursday 31 December 2015 - last comment - 11:33, Tuesday 05 January 2016(24595)

Conlog failed at UTC new year roll over

The conlog process on h1conlog1-master failed soon after the UTC new year. I'm pretty sure it did the same last year but I could not find an alog confirming this. I followed the wiki instructions on restarting the master process. I did initially try to mysqlcheck the databases, but after 40 minutes I abandoned that. I started the conlog process on the master and configured it for the channel list. After a couple of minutes all the channels were connected and the queue size went down to zero. H1 was out of lock at the time due to an earthquake.

For next year's occurance, here is the log file error this time around

root@h1conlog1-master:~# grep conlog: /var/log/syslog

Dec 31 16:00:12 h1conlog1-master conlog: ../conlog.cpp: 301: process_cac_messages: MySQL Exception: Error: Duplicate entry 'H1:SUS-SR3_M1_DITHER_P_OFFSET-1451606400000453212-3' for key 'PRIMARY': Error code: 1062: SQLState: 23000: Exiting.

Dec 31 16:00:12 h1conlog1-master conlog: ../conlog.cpp: 331: process_cas: Exception: boost: mutex lock failed in pthread_mutex_lock: Invalid argument Exiting.

Comments related to this report

patrick.thomas@LIGO.ORG - 19:53, Thursday 31 December 2015 (24597)

Link

This indicates that it tried to write an entry for H1:SUS-SR3_M1_DITHER_P_OFFSET twice with the same Unix time stamp of 1451606400.000453212. This corresponds to Fri, 01 Jan 2016 00:00:00 GMT. I'm guessing there was a leap second applied.

david.barker@LIGO.ORG - 09:42, Friday 01 January 2016 (24605)

Link

of course there was no actual leap second scheduled for Dec 31 2015, so we need to take a closer look at what happened here.

patrick.thomas@LIGO.ORG - 11:33, Tuesday 05 January 2016 (24708)

Link

The previous line before the error reports the application of a leap second. I'm not sure why, since you are right, none were scheduled.

Dec 31 15:59:59 h1conlog1-master kernel: [14099669.303998] Clock: inserting leap second 23:59:60 UTC
Dec 31 16:00:12 h1conlog1-master conlog: ../conlog.cpp: 301: process_cac_messages: MySQL Exception: Error: Duplicate entry 'H1:SUS-SR3_M1_DITHER_P_OFFSET-1451606400000453212-3' for key 'PRIMARY': Error code: 1062: SQLState: 23000: Exiting.
Dec 31 16:00:12 h1conlog1-master conlog: ../conlog.cpp: 331: process_cas: Exception: boost: mutex lock failed in pthread_mutex_lock: Invalid argument Exiting.