Useism slight trend down and now just below 90th percentile.Wind picks up just a bit but still less than 10 mph. Very clean DMT Omega. LLO's down due to high useism.
Ed has mentioned the "External" phone call with no one at the end of the line. That thing just called again four times on three different phones within a few minutes, including the telecon phone on the desk that I've never heard it rung before.
TITLE: Dec 19 EVE Shift 00:00-08:00UTC (16:00-00:00 PDT), all times posted in UTC
STATE Of H1: Observing
SUPPORT: Darkhan
INCOMING OPERATOR: Nutsinee
SHIFT SUMMARY: I’m happy to report that there’s nothing to report except µSei is trending down slightlyto ≈.8µm/s. EQ bands are ≈ .04µm/s and wind is calm. Handing of to Nutsinee locked for 25 hours at ≈80Mpc. There has been a number of external phone calls to both the control room desk phone and the cr conference phone that have no one attached to the other side. Weird.
02:26UTC Livingston down. Ignored.
TITLE: Dec 19 EVE Shift 00:00-08:00UTC (16:00-00:00 PDT), all times posted in UTC
STATE Of H1: Observing
OUTGOING OPERATOR: TJ
QUICK SUMMARY: Everything looks fine in terms of the lock. IFO is still locked from when I relocked yesterday evening. µSei is creeping up into 1µm/s land. EQ bands look quiet….for now. Winds are calm. Livingston fell out just over 4 hours ago. CW injections running. TINJ are not running.
TITLE: 12/19 Day Shift: 16:00-00:00UTC (08:00-16:00 PDT), all times posted in UTC"
STATE Of H1: Observing at 80Mpc for 17hrs
SHIFT SUMMARY: Observing the whole time. LLO went out from an earthquake that we made it through about 4 hours ago and has not come back yet.
INCOMING OPERATOR: Ed M
ACTIVITY LOG: n/a
13.5hrs lock and Observing. We have made it through 2 earthquakes, wind is minimal, useism 0.8um/s.
O1 days 90-92
No restarts reported for these 3 days.
Max, Chris Summary: We tested scheduling the burst injections. The purpose of this test was to test the burst pipeline hardware injection recovery as outlined in the O1 plan. The initial aLog is here: aLog entry Waveform: The waveform parameters are here: https://daqsvn.ligo-la.caltech.edu/svn/injection/hwinj/Details/config/Burst/Waveform/WNB_burst_1134573920_1.info The waveform file for H1 and L1 are: https://daqsvn.ligo-la.caltech.edu/svn/injection/hwinj/Details/config/Burst/Waveform/WNB_burst_1134573920_1_H1.txt https://daqsvn.ligo-la.caltech.edu/svn/injection/hwinj/Details/config/Burst/Waveform/WNB_burst_1134573920_1_L1.txt First Test: The first test was scheduled for 1134577813. The schedule file was given: 1134577813 2 1.0 WNB_burst_1134573920_1_ This injection failed. It failed because the waveform files ended in .dat instead of .txt. tinj was updated but the documentation for performing hardware injections document was not! Max and I checked the PINJX_TRANSIENT EXCMON channel to confirm that a signal did not go into the inteferometers. Second Test: The second test was scheduled for 1134581196. We renamed the waveform ASCII files and the schedule file was given: 1134581196 2 1.0 WNB_burst_1134573920_1_ This injection was successful. cWB already picked up the injection. See gracedb link: gracedb entry Action Items * The documentation for burst injections needs to be updated with the correct filename.
I checked the segment database to make sure the injection was logged correctly. It was. Querying ODC-INJECTION_BURST:2 in both IFOs from 1134577800 I get a segment for [1134581196,1134581198).
The documentation (DCC link) has been updated to correctly say to use .txt extension for burst injection waveform files.
TITLE: 12/19 OWL Shift: 08:00-16:00UTC (00:00-08:00PST), all times posted in UTC
STATE of H1: Observing at ~80 Mpc.
Incoming Operator: TJ
Support:
Quick Summary:
The night was quiet. Except for the high useism and slow computer that was an issue for a while, I have nothing to complain.
Shift Activities:
15:32 Chris Biwer called control room regarding a hardware injection.
This morning's striptool flatline and computer slow down reports further strengthens the evidence that this problem is related to h1boot's NFS disk activity. Yesterday I stopped all running rsync backups of /opt/rtcds and started a single backup at 16:30 PST Friday. It took about 10 hours to complete, and finished around 02:00 this morning. This is around the time the EPICS data froze and Nutsinee reported workstation slow downs. I am monitoring EPICS freezes by looking at the Dolphin manager logs on h1boot, they shows freeze events around 16:30 yesterday and 02:00 this morning and none inbetween.
This rsync used to take only 20 minutes, I'll look into why it is now taking longer (a major file cleanup could be in order).
No linux workstation is reporting loss of NFS connection to this server at these times, looks like a general slow down which impacts diskless frontend computers more.
Investigation continues.
Long term fix is to install a new NFS server for /opt/rtcds post O1.
Here are the dolphin logs for this period of time (reporting when nodes come back)
Dec 18 2015 16:34:15 Fabric 0 status: All nodes are ok!
Dec 18 2015 16:34:28 Fabric 0 status: All nodes are ok!
Dec 18 2015 16:34:29 Fabric 0 status: All nodes are ok!
Dec 19 2015 01:51:09 Fabric 0 status: All nodes are ok!
Dec 19 2015 01:51:45 Fabric 0 status: All nodes are ok!
Dec 19 2015 01:51:46 Fabric 0 status: All nodes are ok!
TITLE: 12/19 Day Shift: 16:00-00:00UTC (08:00-16:00 PDT), all times posted in UTC"
STATE Of H1: Observing at 78Mpc for 9hrs
OUTGOING OPERATOR: Nutsinee
QUICK SUMMARY: useism is on the rise at 0.7 um/s right now, wind is minimal, PSL power at 21.9, blends are on the 45's everywhere so I will keep an eye on the EX ISI, possible earthquake about to hit us.
Max, Chris We preparing to do a burst hardware injection using tinj. I will update this aLog when we add the injection to the schedule.
tinj crashed. Max and I looked through the code and see:
% update on 9 June, 2015: burst injections use txt now
% injfile = [burstpath '/' filefuture{1} ifo '.dat'];
injfile = [burstpath '/' filefuture{1} ifo '.txt'];
THIS NEEDS TO BE UPDATED IN THE DOCUMENTATION
tinj was restarted at H1 at 11:54 EST.
tinj was restarted at L1 at 11:58 EST.
The schedule has been updated with this burst injection: 1134577813 2 1.0 WNB_burst_1134573920_1_
We checked that there was no excitation from the last injection. See plots below. Max and I will fix the filename and retry once more.
We are going to try one more time. The schedule file was updated with: 1134581196 2 1.0 WNB_burst_1134573920_1_
We have finish doing injections test. The second injection was logged as succesful. Max and I will make an aLog entry summarizing the tests.