edmond.merilh@LIGO.ORG - posted 18:56, Saturday 19 December 2015 (24333)
GRB Alert
02:26UTC Livingston down. Ignored.
02:26UTC Livingston down. Ignored.
TITLE: Dec 19 EVE Shift 00:00-08:00UTC (16:00-00:00 PDT), all times posted in UTC
STATE Of H1: Observing
OUTGOING OPERATOR: TJ
QUICK SUMMARY: Everything looks fine in terms of the lock. IFO is still locked from when I relocked yesterday evening. µSei is creeping up into 1µm/s land. EQ bands look quiet….for now. Winds are calm. Livingston fell out just over 4 hours ago. CW injections running. TINJ are not running.
TITLE: 12/19 Day Shift: 16:00-00:00UTC (08:00-16:00 PDT), all times posted in UTC"
STATE Of H1: Observing at 80Mpc for 17hrs
SHIFT SUMMARY: Observing the whole time. LLO went out from an earthquake that we made it through about 4 hours ago and has not come back yet.
INCOMING OPERATOR: Ed M
ACTIVITY LOG: n/a
13.5hrs lock and Observing. We have made it through 2 earthquakes, wind is minimal, useism 0.8um/s.
O1 days 90-92
No restarts reported for these 3 days.
Max, Chris Summary: We tested scheduling the burst injections. The purpose of this test was to test the burst pipeline hardware injection recovery as outlined in the O1 plan. The initial aLog is here: aLog entry Waveform: The waveform parameters are here: https://daqsvn.ligo-la.caltech.edu/svn/injection/hwinj/Details/config/Burst/Waveform/WNB_burst_1134573920_1.info The waveform file for H1 and L1 are: https://daqsvn.ligo-la.caltech.edu/svn/injection/hwinj/Details/config/Burst/Waveform/WNB_burst_1134573920_1_H1.txt https://daqsvn.ligo-la.caltech.edu/svn/injection/hwinj/Details/config/Burst/Waveform/WNB_burst_1134573920_1_L1.txt First Test: The first test was scheduled for 1134577813. The schedule file was given: 1134577813 2 1.0 WNB_burst_1134573920_1_ This injection failed. It failed because the waveform files ended in .dat instead of .txt. tinj was updated but the documentation for performing hardware injections document was not! Max and I checked the PINJX_TRANSIENT EXCMON channel to confirm that a signal did not go into the inteferometers. Second Test: The second test was scheduled for 1134581196. We renamed the waveform ASCII files and the schedule file was given: 1134581196 2 1.0 WNB_burst_1134573920_1_ This injection was successful. cWB already picked up the injection. See gracedb link: gracedb entry Action Items * The documentation for burst injections needs to be updated with the correct filename.
TITLE: 12/19 OWL Shift: 08:00-16:00UTC (00:00-08:00PST), all times posted in UTC
STATE of H1: Observing at ~80 Mpc.
Incoming Operator: TJ
Support:
Quick Summary:
The night was quiet. Except for the high useism and slow computer that was an issue for a while, I have nothing to complain.
Shift Activities:
15:32 Chris Biwer called control room regarding a hardware injection.
This morning's striptool flatline and computer slow down reports further strengthens the evidence that this problem is related to h1boot's NFS disk activity. Yesterday I stopped all running rsync backups of /opt/rtcds and started a single backup at 16:30 PST Friday. It took about 10 hours to complete, and finished around 02:00 this morning. This is around the time the EPICS data froze and Nutsinee reported workstation slow downs. I am monitoring EPICS freezes by looking at the Dolphin manager logs on h1boot, they shows freeze events around 16:30 yesterday and 02:00 this morning and none inbetween.
This rsync used to take only 20 minutes, I'll look into why it is now taking longer (a major file cleanup could be in order).
No linux workstation is reporting loss of NFS connection to this server at these times, looks like a general slow down which impacts diskless frontend computers more.
Investigation continues.
Long term fix is to install a new NFS server for /opt/rtcds post O1.
Here are the dolphin logs for this period of time (reporting when nodes come back)
Dec 18 2015 16:34:15 Fabric 0 status: All nodes are ok!
Dec 18 2015 16:34:28 Fabric 0 status: All nodes are ok!
Dec 18 2015 16:34:29 Fabric 0 status: All nodes are ok!
Dec 19 2015 01:51:09 Fabric 0 status: All nodes are ok!
Dec 19 2015 01:51:45 Fabric 0 status: All nodes are ok!
Dec 19 2015 01:51:46 Fabric 0 status: All nodes are ok!
TITLE: 12/19 Day Shift: 16:00-00:00UTC (08:00-16:00 PDT), all times posted in UTC"
STATE Of H1: Observing at 78Mpc for 9hrs
OUTGOING OPERATOR: Nutsinee
QUICK SUMMARY: useism is on the rise at 0.7 um/s right now, wind is minimal, PSL power at 21.9, blends are on the 45's everywhere so I will keep an eye on the EX ISI, possible earthquake about to hit us.
Max, Chris We preparing to do a burst hardware injection using tinj. I will update this aLog when we add the injection to the schedule.
tinj crashed. Max and I looked through the code and see:
% update on 9 June, 2015: burst injections use txt now
% injfile = [burstpath '/' filefuture{1} ifo '.dat'];
injfile = [burstpath '/' filefuture{1} ifo '.txt'];
THIS NEEDS TO BE UPDATED IN THE DOCUMENTATION
tinj was restarted at H1 at 11:54 EST.
tinj was restarted at L1 at 11:58 EST.
The schedule has been updated with this burst injection: 1134577813 2 1.0 WNB_burst_1134573920_1_
We checked that there was no excitation from the last injection. See plots below. Max and I will fix the filename and retry once more.
We are going to try one more time. The schedule file was updated with: 1134581196 2 1.0 WNB_burst_1134573920_1_
We have finish doing injections test. The second injection was logged as succesful. Max and I will make an aLog entry summarizing the tests.
Still Observing. Very stable range. zero wind. Useism is trending slightly upward. The computers are no longer frozen. Violin modes are high but none seems to be ringing up. Seems like some of the modes get kicked in the wrong way everytime the ifo loses lock.
TITLE: Dec 19 OWL Shift 08:00-16:00UTC (00:00-08:00 PST), all times posted in UTC
Outgoing Ops: Ed
Quick Summary: Useism is hanging at 90th percentile. Violin modes are relatively high (damping still turned off). Wind below 10 mph. Using 45mHz blen everywhere. Have StripTool running to monitor ETMs CPS sensors as Jim suggested (which Ed kindly passed on the information). The computer opposite to Ops station is very slow...
I saw the signal flatted out twice in a row. The lock wasn't broken. Same thing also happened twice during my shift last night. Is this problem becoming worse?
Also Ops computer and two other computers I've tried to logon to are equally VERY VERY SLOW. They're pretty much frozen. There's even a big gap when the verbal alarm was telling me time ("CURRENT TIME ------------- SOMETHING UTC") It took forever just to take a screenshot and attach it to this alog........ Let's hope the we don't lose lock because I don't know how to bring it back up with a frozen computer......
10:18 UTC I stepped out for a bit. Came back and saw a missed call. I don't know how to review the missed call number (sorry these wired phones are ancient technology to me. I will learn next time.). I'm back in the control room now.
The computers are better now.......