Lockloss at 02:40 (18:40) due to mag 6.3 EQ south of Australia. Seismic rang up to 3.0um/s. No locking of ALS right now. Put IFO into Down state until things settle down.
The conlog process on h1conlog1-master failed soon after the UTC new year. I'm pretty sure it did the same last year but I could not find an alog confirming this. I followed the wiki instructions on restarting the master process. I did initially try to mysqlcheck the databases, but after 40 minutes I abandoned that. I started the conlog process on the master and configured it for the channel list. After a couple of minutes all the channels were connected and the queue size went down to zero. H1 was out of lock at the time due to an earthquake.
For next year's occurance, here is the log file error this time around
root@h1conlog1-master:~# grep conlog: /var/log/syslog
Dec 31 16:00:12 h1conlog1-master conlog: ../conlog.cpp: 301: process_cac_messages: MySQL Exception: Error: Duplicate entry 'H1:SUS-SR3_M1_DITHER_P_OFFSET-1451606400000453212-3' for key 'PRIMARY': Error code: 1062: SQLState: 23000: Exiting.
Dec 31 16:00:12 h1conlog1-master conlog: ../conlog.cpp: 331: process_cas: Exception: boost: mutex lock failed in pthread_mutex_lock: Invalid argument Exiting.
Received GRB alert at 00:44 (16:44). IFO is up in Observing mode. Spoke with LLO. I one hour hold to accumulate background information.
Transition Summary: Title: 12/31/2015, Evening Shift 00:00 – 08:00 (16:00 – 00:00) All times in UTC (PT) State of H1: 00:00 (16:00), IFO in Observing mode for 7 hours. Power is 21.8W and range is 79Mpc. The wind at the site is a light to gentle breeze (4 - 12mph). Seismic 0.03-0.1Hz band showed a bit of excitation for about 4 hours in the End-Y channel, which has subsided. Microseism remains flat at 0.4um/s. Outgoing Operator: Corey
TITLE: 12/31 DAY Shift: 16:00-00:00UTC (08:00-04:00PDT), all times posted in UTC
STATE of H1: Locked for 6+hrs at or under 80Mpc.
Incoming Operator: Jeff B.
Support: Kiwamu called in for OWL & DAY
Quick Summary:
After Kiwamu fought the fight early this morning, H1 has been in Observing ever since (even with a few notable [red Terramon] rumblers). Will be curious to see how H1 fares during its next Initial Alignment (see note for possible work-around for Input Align portion of Initial Alignment from Kiwamu's alog earlier).
There was warning (albeit after-the-fact) of another EQ at 22:40 from Alaska with motion of 0.6um/s, but it had no effect here.
~1505 hrs. local
Looks like a 5.4 New Zealand quake is a few minutes (22:18utc/218pmPST) from approaching with 0.5um/s motion. We'll see how we fair.
Around 22:13 Tidal, ASC (huge in yaw), and POP_A_LF all started getting big oscilations, but we appear to have ridden out the 22:18:52 R-wave arrival time. Didn't really see anything obvious in the 0.03-0.1 seismic band. Range has trended down to around 76Mpc over the last few minutes.
And all of this as the winds slowly get above 10mph.
Go H1! #knockonwood
Noticed on our Seismic FOMs that some of the 1-3Hz seismic BLRMS trends took a step up at midnight last night (~8:03UTC today). I went ahead and looked at all the bands & axis of the EX/EY/LVEA seismometers and only found this jump on (3) trends for 1-3Hz: LVEA_X, LVEA_Z, & EY_Y.
Attached are trends of 1-3Hz for the last 24hrs.
(Ed, Kiwamu),
A workaroud (temporary, manual and painful):
What I tried (none of them worked):
I spent a couple of hours investigating the issue, but I could not identify or fix the issue. And I was never able to lock the laser the X arm. The below is a list of what I tried.
Per Operator Shift Checksheet activity for Thursday Task, I topped off the chiller by adding 175mL of water (this was before we took H1 to Observing).
TITLE: 12/31 DAY Shift: 16:00-00:00UTC (08:00-04:00PDT), all times posted in UTC
STATE of H1: Down & alignment/locking investigation by Kiwamu
Outgoing Operator: Ed
Quick Summary:
Kiwamu & Ed ran into issues with the INPUT ALIGN section of the Initial Alignment. Kiwamu mentioned it could be related to the AS_Air photodiode. (Even though we can't get to INPUT ALIGN, the AS Air spot looks nice [no higher order modes or jumps when flashing AND the IR Xarm flashes have been up to 1.0---so, it looks good...but no INPUT ALIGN). Current plan is to skip the INPUT ALIGN section of Alignment and try locking.
Additional Notes:
1) And if you are familiar with Initial Alignment issues of late, you will remember that lately we have been skipping the Dark Michelson portion of Initial Alignment....and this state also uses AS_Air. So this sounds suspicious....
BUT,
2) SRC state of Initial Alignment also uses the AS_Air PD, and we were able to get SRC with no problem. Suspicious indeed!
Currently waiting for DRMI to lock.
Back To Observing at 17:10UTC (9:10amPST)
Only SDF Diff was: Dark offset for AS_AIR was Accepted (from -75.0 to -80.0).
Notes on Guardian steps:
I have incremented the XEND heater HC3 from 13.0 to 13.5 ma. The VEA temperature had fallen a little overnight.
TITLE: Dec 30 OWL Shift 08:00-16:00UTC (00:00-08:00 PDT), all times posted in UTC
STATE Of H1: Uknown
SUPPORT: Kiwamu
INCOMING OPERATOR: Corey
SHIFT SUMMARY: After the quake recovery, which didn’t take long. The IFO was in need of some initial alignment. I never got past INPUT_ALIGN. THe X-Arm would not lock in IR. After trying many things and a long telephone call with Kiwamu, he decided to come in. The spot looked a little pitchy in the PR2 and IM4 dept. Also SR2 has moved 17µrad in Pitch causing the spot to be higher than usual on the AS_AIR camera. YAW on that same optic was restored. Kiwamu is still working on the problem.
11:27UTC
5.7
36.5 km
11:20UTC IFO locked at 68 Mpc. In other news....5.7 in Nicaraugua is rolling us which is why the range is down. 2.63µm R-Wave expected to roll through here right about now. Fingers crossed. Preparing the lockloss log. :/
of course there was no actual leap second scheduled for Dec 31 2015, so we need to take a closer look at what happened here.