Max, Chris

Summary:

We tested scheduling the burst injections. The purpose of this test was to test the burst pipeline hardware injection recovery as outlined in the O1 plan.

The initial aLog is here: aLog entry

Waveform:

The waveform parameters are here: https://daqsvn.ligo-la.caltech.edu/svn/injection/hwinj/Details/config/Burst/Waveform/WNB_burst_1134573920_1.info

The waveform file for H1 and L1 are: https://daqsvn.ligo-la.caltech.edu/svn/injection/hwinj/Details/config/Burst/Waveform/WNB_burst_1134573920_1_H1.txt https://daqsvn.ligo-la.caltech.edu/svn/injection/hwinj/Details/config/Burst/Waveform/WNB_burst_1134573920_1_L1.txt

First Test:

The first test was scheduled for 1134577813. The schedule file was given:
1134577813 2 1.0 WNB_burst_1134573920_1_

This injection failed.

It failed because the waveform files ended in .dat instead of .txt. tinj was updated but the documentation for performing hardware injections document was not!

Max and I checked the PINJX_TRANSIENT EXCMON channel to confirm that a signal did not go into the inteferometers.

Second Test:

The second test was scheduled for 1134581196. We renamed the waveform ASCII files and the schedule file was given:
1134581196 2 1.0 WNB_burst_1134573920_1_

This injection was successful.

cWB already picked up the injection. See gracedb link: gracedb entry

Action Items

  * The documentation for burst injections needs to be updated with the correct filename.

TITLE:  12/19 OWL Shift:  08:00-16:00UTC (00:00-08:00PST), all times posted in UTC     

STATE of H1:   Observing at ~80 Mpc.

Incoming Operator:  TJ

Support:  

Quick Summary:

The night was quiet. Except for the high useism and slow computer that was an issue for a while, I have nothing to complain.

Shift Activities:

15:32 Chris Biwer called control room regarding a hardware injection.

This morning's striptool flatline and computer slow down reports further strengthens the evidence that this problem is related to h1boot's NFS disk activity. Yesterday I stopped all running rsync backups of /opt/rtcds and started a single backup at 16:30 PST Friday. It took about 10 hours to complete, and finished around 02:00 this morning. This is around the time the EPICS data froze and Nutsinee reported workstation slow downs. I am monitoring EPICS freezes by looking at the Dolphin manager logs on h1boot, they shows freeze events around 16:30 yesterday and 02:00 this morning and none inbetween.

This rsync used to take only 20 minutes, I'll look into why it is now taking longer (a major file cleanup could be in order).

No linux workstation is reporting loss of NFS connection to this server at these times, looks like a general slow down which impacts diskless frontend computers more.

Investigation continues.

Long term fix is to install a new NFS server for /opt/rtcds post O1.

Here are the dolphin logs for this period of time (reporting when nodes come back)

Dec 18 2015 16:34:15 Fabric 0 status: All nodes are ok!

Dec 18 2015 16:34:28 Fabric 0 status: All nodes are ok!

Dec 18 2015 16:34:29 Fabric 0 status: All nodes are ok!

Dec 19 2015 01:51:09 Fabric 0 status: All nodes are ok!

Dec 19 2015 01:51:45 Fabric 0 status: All nodes are ok!

Dec 19 2015 01:51:46 Fabric 0 status: All nodes are ok!

• TITLE: 12/19 Day Shift: 16:00-00:00UTC (08:00-16:00 PDT), all times posted in UTC"

• STATE Of H1: Observing at 78Mpc for 9hrs

• OUTGOING OPERATOR: Nutsinee

• QUICK SUMMARY: useism is on the rise at 0.7 um/s right now, wind is minimal, PSL power at 21.9, blends are on the 45's everywhere so I will keep an eye on the EX ISI, possible earthquake about to hit us.
  

Max, Chris

We preparing to do a burst hardware injection using tinj. I will update this aLog when we add the injection to the schedule.

Still Observing. Very stable range. zero wind. Useism is trending slightly upward. The computers are no longer frozen. Violin modes are high but none seems to be ringing up. Seems like some of the modes get kicked in the wrong way everytime the ifo loses lock.

TITLE:    Dec 19 OWL Shift 08:00-16:00UTC (00:00-08:00 PST), all times posted in UTC

Outgoing Ops: Ed

Quick Summary: Useism is hanging at 90th percentile. Violin modes are relatively high (damping still turned off). Wind below 10 mph. Using 45mHz blen everywhere. Have StripTool running to monitor ETMs CPS sensors as Jim suggested (which Ed kindly passed on the information). The computer opposite to Ops station is very slow...

TITLE:    Dec 18 EVE Shift 00:00-08:00UTC (16:00-00:00 PDT), all times posted in UTC

STATE Of H1: Observing

SUPPORT: Jason

INCOMING OPERATOR: Nutsinee

SHIFT SUMMARY:

After repeated locklosses waiting at various stages of the locking sequence after I had changed the blends to QUITE_90 as recommended. I decided to switch back to the 45mHz blends at the end stations. I let the IFO “cook” on ENGAGE_ASC_PART3 for a good long while before advancing. I can’t say conclusively that this change in the ISI blends did the trick, but the IFO went all the way to NLN/78Mpc and it’s been there ever since. EQ bands are nominal. The mean uSeism is slightly above the 90th percentile. Wind is calm. Handing off Locked IFO to Nutsinee.

ACTIVITY LOG:

04:34 Lockloss while at ENGAGE_ASC_PART3

05:00 Switched end station blends to Quite_90 (off axis) to see if locking would improve.

05:33  still no satisfaction locking. Lock breaks at DC_READOUT_TRANSITION, repeatedly. I put a call in to Sheila.

05:32 OMC SUS watchdog tripped.

05:53  it appears that when DC_READOUT_TRANSITION occurs that OMC_LOCK reports OMC not ready as it may still be in TUNE_OFFSETS.

06:05 Waited at ENGAGE_ASC_PART3 for OMC to move to READY_FOR_HANDOFF. Lockloss occured, this time, at the very next step (REFL_IN_VACUO)

06:21 Locklos occurred sitting on ENGAGE_ASC_PART3 Grasping at straws at this point. Haven’t heard back from Sheila.

06:51 FINALLY! Locked at NLN. Switched back to 45mHz blends at Ends. Had to manually start ISS second Loop

MID-SHIFT SUMMARY:

After the lock loss that occurred at 02:54UTC caused by earthquake in Vanuatu and about a 1 hr stand down to permit ring down, re-locking is being delayed by locklosses at various stages in the sequence.

EQ bands have come back down below .1um/s. The mean uSeism is right about the 90th percentile. Wind is calm. Re-locking marches on.

ITLE: Dec 18 EVE Shift 00:00-08:00UTC (16:00-00:00 PDT), all times posted in UTC

STATE Of H1: Observing

OUTGOING OPERATOR: Jim

QUICK SUMMARY: IFO recently re-locked. µSeism @ 90thpercentile. EQ bands nominal. Winds ≤ 10mph.