Follow up from yesterday's work, following the last full rsync of /opt/rtcds from h1boot to h1boot-backup which completed at 2am Saturday morning local time, we have had no epics freeze events in the past 31 hours. For comparison, during the past week we have had an average of 6 events per day. I'm now working on splitting the rysnc into smaller chunks so hopefully each one has no impact on the front end EPICS communications but we still keep the h1boot-backup machine up todate as an emergency replacement of h1boot.

The DIAG_MAIN exec time normally runs around 6/7sec, and every few months it will jump up to ~50sec. It is not a gradual increase, it will just shoot up. December 3, around 8am PST was the most recent time increase, and last night around 20:15 PST it dropped back to ~7sec. I have checked a few other nodes (IMC_LOCK, ISC_LOCK, ISI_ITMX_ST1, and a few SUS nodes that I can't remember) and their exec times dont seem to show any coincidence with this. The DIAG_MAIN node is different than most of the other ones because it will make numerous ezca calls continuously to check on channels and evaluate what may be out of normal, so it might make sense that the other nodes are not showing the same signs.

I quickly logged into the h1guardian0 machine this morning, while LLO was down, to see if the usage has dropped from last time I checked on the 3rd. I wasn't smart enough then to take a screenshot, but from what I remember from the 'top' command screen the tasks were about the same and the CPU%'s were similar but maybe now slighty lower, but only maybe. [Login times to h1guardian0: 16:18,16:48 UTC for a few minutes each time.]

Nothing seemed to have happened around 20:15 last night, we have been locked and observing for the past 34 hours. The investigation continues...

TITLE:    Dec 20 Owl Shift 08:00-16:00UTC (00:00-08:00 PT), all times posted in UTC

STATE Of H1: Observing at ~80 Mpc

SUPPORT:

INCOMING OPERATOR: TJ

SHIFT SUMMARY: Low wind through out the night. Useism picked up and hit 90th percentile. Mysterious calls continues in the morning, three times in a row. LLO still down from high useism.

• TITLE: 12/20 Day Shift: 16:00-00:00UTC (08:00-16:00 PDT), all times posted in UTC"

• STATE Of H1: Observing at 78Mpc for 33hrs

• OUTGOING OPERATOR: Nutsinee

•  

• QUICK SUMMARY: Long lock! Wind <10mph, useism 0.5um/s, PSL 21.9W, CW inj running, LLO not locked.
  

Useism slight trend down and now just below 90th percentile.Wind picks up just a bit but still less than 10 mph. Very clean DMT Omega. LLO's down due to high useism.

Ed has mentioned the "External" phone call with no one at the end of the line. That thing just called again four times on three different phones within a few minutes, including the telecon phone on the desk that I've never heard it rung before.

TITLE:    Dec 19 EVE Shift 00:00-08:00UTC (16:00-00:00 PDT), all times posted in UTC

STATE Of H1: Observing

SUPPORT: Darkhan

INCOMING OPERATOR: Nutsinee

SHIFT SUMMARY: I’m happy to report that there’s nothing to report except µSei is trending down slightlyto ≈.8µm/s. EQ bands are ≈ .04µm/s and wind is calm. Handing of to Nutsinee locked for 25 hours at ≈80Mpc. There has been a number of external phone calls to both the control room desk phone and the  cr conference phone that have no one attached to the other side. Weird.

02:26UTC Livingston down. Ignored.

TITLE: Dec 19 EVE Shift 00:00-08:00UTC (16:00-00:00 PDT), all times posted in UTC

STATE Of H1: Observing

OUTGOING OPERATOR: TJ

QUICK SUMMARY: Everything looks fine in terms of the lock. IFO is still locked from when I relocked yesterday evening.  µSei is creeping up into 1µm/s land. EQ bands look quiet….for now. Winds are calm. Livingston fell out just over 4 hours ago. CW injections running. TINJ are not running.

• TITLE: 12/19 Day Shift: 16:00-00:00UTC (08:00-16:00 PDT), all times posted in UTC"

• STATE Of H1: Observing at 80Mpc for 17hrs

• SHIFT SUMMARY: Observing the whole time. LLO went out from an earthquake that we made it through about 4 hours ago and has not come back yet.

• INCOMING OPERATOR: Ed M

• ACTIVITY LOG: n/a

13.5hrs lock and Observing. We have made it through 2 earthquakes, wind is minimal, useism 0.8um/s. 

O1 days 90-92

No restarts reported for these 3 days.

Max, Chris

Summary:

We tested scheduling the burst injections. The purpose of this test was to test the burst pipeline hardware injection recovery as outlined in the O1 plan.

The initial aLog is here: aLog entry

Waveform:

The waveform parameters are here: https://daqsvn.ligo-la.caltech.edu/svn/injection/hwinj/Details/config/Burst/Waveform/WNB_burst_1134573920_1.info

The waveform file for H1 and L1 are: https://daqsvn.ligo-la.caltech.edu/svn/injection/hwinj/Details/config/Burst/Waveform/WNB_burst_1134573920_1_H1.txt https://daqsvn.ligo-la.caltech.edu/svn/injection/hwinj/Details/config/Burst/Waveform/WNB_burst_1134573920_1_L1.txt

First Test:

The first test was scheduled for 1134577813. The schedule file was given:
1134577813 2 1.0 WNB_burst_1134573920_1_

This injection failed.

It failed because the waveform files ended in .dat instead of .txt. tinj was updated but the documentation for performing hardware injections document was not!

Max and I checked the PINJX_TRANSIENT EXCMON channel to confirm that a signal did not go into the inteferometers.

Second Test:

The second test was scheduled for 1134581196. We renamed the waveform ASCII files and the schedule file was given:
1134581196 2 1.0 WNB_burst_1134573920_1_

This injection was successful.

cWB already picked up the injection. See gracedb link: gracedb entry

Action Items

  * The documentation for burst injections needs to be updated with the correct filename.

TITLE:  12/19 OWL Shift:  08:00-16:00UTC (00:00-08:00PST), all times posted in UTC     

STATE of H1:   Observing at ~80 Mpc.

Incoming Operator:  TJ

Support:  

Quick Summary:

The night was quiet. Except for the high useism and slow computer that was an issue for a while, I have nothing to complain.

Shift Activities:

15:32 Chris Biwer called control room regarding a hardware injection.

This morning's striptool flatline and computer slow down reports further strengthens the evidence that this problem is related to h1boot's NFS disk activity. Yesterday I stopped all running rsync backups of /opt/rtcds and started a single backup at 16:30 PST Friday. It took about 10 hours to complete, and finished around 02:00 this morning. This is around the time the EPICS data froze and Nutsinee reported workstation slow downs. I am monitoring EPICS freezes by looking at the Dolphin manager logs on h1boot, they shows freeze events around 16:30 yesterday and 02:00 this morning and none inbetween.

This rsync used to take only 20 minutes, I'll look into why it is now taking longer (a major file cleanup could be in order).

No linux workstation is reporting loss of NFS connection to this server at these times, looks like a general slow down which impacts diskless frontend computers more.

Investigation continues.

Long term fix is to install a new NFS server for /opt/rtcds post O1.

Here are the dolphin logs for this period of time (reporting when nodes come back)

Dec 18 2015 16:34:15 Fabric 0 status: All nodes are ok!

Dec 18 2015 16:34:28 Fabric 0 status: All nodes are ok!

Dec 18 2015 16:34:29 Fabric 0 status: All nodes are ok!

Dec 19 2015 01:51:09 Fabric 0 status: All nodes are ok!

Dec 19 2015 01:51:45 Fabric 0 status: All nodes are ok!

Dec 19 2015 01:51:46 Fabric 0 status: All nodes are ok!

Max, Chris

We preparing to do a burst hardware injection using tinj. I will update this aLog when we add the injection to the schedule.