I simply logged in to the alog using my credential on Ops station. I opened up GraceDB webpage and noticed that the GraceDB querying is good again.

The ext_alert.py script which periodically views GraceDB had failed. I have just restarted it, instructions for restarting are in https://lhocds.ligo-wa.caltech.edu/wiki/ExternalAlertNotification

Getting this process to autostart is now on our high priority list (FRS3415).

here is the error message displayed before I did the restart.

 File "ext_alert.py", line 150, in query_gracedb

    return query_gracedb(start, end, connection=connection, test=test)

  File "ext_alert.py", line 150, in query_gracedb

    return query_gracedb(start, end, connection=connection, test=test)

  File "ext_alert.py", line 135, in query_gracedb

    external = log_query(connection, 'External %d .. %d' % (start, end))

  File "ext_alert.py", line 163, in log_query

    return list(connection.events(query))

  File "/usr/lib/python2.7/dist-packages/ligo/gracedb/rest.py", line 441, in events 

    uri = self.links['events']

  File "/usr/lib/python2.7/dist-packages/ligo/gracedb/rest.py", line 284, in links  

    return self.service_info.get('links')

  File "/usr/lib/python2.7/dist-packages/ligo/gracedb/rest.py", line 279, in service_info

    self._service_info = self.request("GET", self.service_url).json()

  File "/usr/lib/python2.7/dist-packages/ligo/gracedb/rest.py", line 325, in request

    return GsiRest.request(self, method, *args, **kwargs)

  File "/usr/lib/python2.7/dist-packages/ligo/gracedb/rest.py", line 201, in request

    response = conn.getresponse()

  File "/usr/lib/python2.7/httplib.py", line 1038, in getresponse

    response.begin()

  File "/usr/lib/python2.7/httplib.py", line 415, in begin

    version, status, reason = self._read_status()

  File "/usr/lib/python2.7/httplib.py", line 371, in _read_status

    line = self.fp.readline(_MAXLINE + 1)

  File "/usr/lib/python2.7/socket.py", line 476, in readline

    data = self._sock.recv(self._rbufsize)

  File "/usr/lib/python2.7/ssl.py", line 241, in recv

    return self.read(buflen)

  File "/usr/lib/python2.7/ssl.py", line 160, in read

    return self._sslobj.read(len)

ssl.SSLError: The read operation timed out

TITLE:  Sep 26 OWL Shift 23:00-07:00UTC (16:00-00:00 PDT), all times posted in UTC

STATE Of H1: Observing

LOCK DURATION: Entire Shift

SUPPORT: N/A

INCOMING OPERATOR: Nutsinee

Activity log:

09:13 SUS E_T_M_Y saturating (Sun Sep 27 9:13:21 UTC)

09:13 SUS E_T_M_Y saturating (Sun Sep 27 9:13:23 UTC) .6Mpc
10:19 SUS E_T_M_Y saturating (Sun Sep 27 10:19:5 UTC) 3Mpc

11:45 SUS E_T_M_Y saturating (Sun Sep 27 11:45:40 UTC) odd...no loss of range

12:21 Noticed the GraceDB quey failure.

14:15 Mike checked in by telephone. He suggested that I e-mail Duncan, Leo and Peter S about the GraceDB query failure that has existed for the duration of my shift.

Shift Summary: Smooth sailing with the IFO locked at 74Mpc. Wind speeds <10mph. 4 ETMY glitches. No obtrusive seismic or weather activity.  I e-mailed Duncan, Leo and Peter S about the GraceDB query failure that has existed for the duration of my shift. Handing off to Nutsinee.

Mid-Shift Summary: Smooth sailing so far with the IFO locked at 74Mpc. Wind speeds have decreased to <5mph. 3 ETMY glitches so far.

TITLE: Sep 27 OWL Shift 07:00-15:00UTC (00:00-08:00 PDT), all times posted in UTC

STATE Of H1: Observing

OUTGOING OPERATOR: Patrick

QUICK SUMMARY:. IFO Locked at 70Mpc/22.9W. No injections are running. All lights are off in LVEA, MID, END and PSL. Wind is below 15mph. Microseism seems to be riding steady @ ~.1µm/s. EQ seismic seems nominally steady @ .2µm/s. No noticable 45Mhz RFAM glitching noticible in triggers, in the last hour at least. Verbal Alarms is down on FOM,Video5.

TITLE: 09/26 [EVE Shift]: 23:00-07:00 UTC (16:00-00:00 PDT), all times posted in UTC
STATE Of H1: Observing, ~71 MPc
SHIFT SUMMARY: Remainder of shift quiet except for SUS ETMY saturations. Wind has come back below 20 mph. Seismic in 0.03 - 0.1 Hz is coming down at end X and end Y.
INCOMING OPERATOR: Ed
ACTIVITY LOG:

03:59 - 4:06 UTC Stepped out of control room

ETMY SUS Saturations:

SUS E_T_M_Y saturating (Sat Sep 27 0:2:19 UTC)
SUS E_T_M_Y saturating (Sat Sep 27 0:14:27 UTC)
SUS E_T_M_Y saturating (Sat Sep 27 0:57:42 UTC)
SUS E_T_M_Y saturating (Sat Sep 27 2:23:2 UTC)
SUS E_T_M_Y saturating (Sat Sep 27 2:23:5 UTC)
SUS E_T_M_Y saturating (Sat Sep 27 2:40:7 UTC)
SUS E_T_M_Y saturating (Sat Sep 27 2:41:41 UTC)
SUS E_T_M_Y saturating (Sat Sep 27 3:20:30 UTC)
SUS E_T_M_Y saturating (Sat Sep 27 4:0:59 UTC)
SUS E_T_M_Y saturating (Sat Sep 27 4:18:21 UTC)
SUS E_T_M_Y saturating (Sat Sep 27 5:0:57 UTC)
SUS E_T_M_Y saturating (Sat Sep 27 5:7:52 UTC)
SUS E_T_M_Y saturating (Sat Sep 27 6:20:14 UTC)

23:12 UTC Tour in control room
before 23:36 UTC Tour out of control room
23:37 - 23:39:10 UTC Stepped out of control room
23:55 UTC Tour in control room
00:13 UTC Tour out of control room
00:30 - 00:39 UTC Stepped out of control room

7 SUS ETMY saturations. Range is starting to trend down over last hour, now at ~ 65 MPc. Winds have ranged up to 30 mph. Seismic in 0.03 - 0.1 Hz band at end X and end Y has come up to ~ .1 um/s. Corner station is at ~ .01 um/s. Have remained in observation mode since beginning of shift.

Executive summary: 

 In regard to narrow lines, early O1 data resembles early ER8 data: a pervasive 16-Hz comb persists throughout the CW search band (below 2000 Hz); there is a notable 1-Hz comb below 100 Hz (0.5-Hz offset); and other sporadic combs persist.
 On the other hand, there are distinct improvements: noise floor is cleaner nearly everywhere and substantially lower in the 10-70 Hz band; non-linear upconversion around quad violin modes and harmonics is much reduced; some combs and isolated lines have disappeared; and the OMC alignment dither frequencies have moved out of the CW search band (hurray!).
 Oh the third hand, new artifacts have appeared or strengthened: a sporadic 8-Hz comb suspected before is confirmed; a 1-Hz comb (no offset) has emerged below 70 Hz; a broad bulge appears in the 1240-1270 Hz band.


Details: 

Using 104.5 hours of FScan-generated, Hann-windowed, 30-minute SFTs, I have gone through the first 2000 Hz of the DARM displacement spectrum (CW search band) to identify lines that could contaminate CW searches.
This study is very similar to prior studies of ER7 data and ER8 data, but since this is my first O1 report, I will repeat below some earlier findings.

Some sample displacement amplitude spectra are attached directly below, but more extensive sets of spectra are attached in zipped files. One set is for O1 sub-band spectra with labels (see code below), and one set is an overlay of early ER8 spectra (50 hours) and the early O1 spectra. As usual, the spectra look worse than they really are because single-bin lines (0.5 mHz wide) appear disproportionately wide in the graphics

A flat-file line list is attached with the same alphabetic coding as in the figures.

Findings:

 A 16-Hz comb pervades the entire 0-2000 Hz band (and well beyond, based on daily FScans)
 A typically much weaker and sporadic 8-Hz comb (odd harmonics) is also pervasive - previously suspected, now confirmed (all harmonics are labeled in figures, even when not visible)
 A 1-Hz comb with a 0.5-Hz offset is visible from 15.5 Hz to 78.5 Hz (slightly wider span than before)
 A new 1-Hz with zero offset is visible from 20.0 Hz to 68.0 Hz
 A 99.9989-Hz comb is visible to its 8th harmonic (was previously visible to its 13th harmonic)
 The 60-Hz power mains comb is visible to its 5th harmonic (was previously visible to its 9th harmonic)
 There is a sporadic comb-on-comb with 0.088425-Hz fine spacing that appears with limited spans in three places near harmonics of 77, 154 and 231 Hz (ambiguity in precise fundamental frequency)
 There is a 31.4149-Hz comb visible to its 2nd harmonic
 The OMC alignment dithers have been moved to above 2000 Hz (thanks!)
 Upconversion around the quad violin modes and their harmonics is much reduced, although the strengths of the higher harmonics themselves remain high. To be more specific, the fundamental and higher harmonics of the upconversion itself (integer harmonics) due to the fundamental violin harmonics are highly suppressed, while the higher harmonics of the violin modes themselves  (not integer harmonics)  remain high. 
 A number of previously seen combs are no longer apparent:  59.3155 Hz, 59.9392 Hz, 59.9954 Hz and 75.3 Hz
 A variety of single lines have disappeared, and new ones have appeared (see attached line list) 
 Compared to early ER8 data, the noise floor is slightly lower in most of the band and significantly lower in the 10-70 Hz band, but there is a significant new bulge in the 1240-1270 Hz band


Line label codes in figures:

b - Bounce mode (quad suspension)
r - Roll mode (quad suspension)
Q - Quad violin mode and harmonics
B - Beam splitter violin mode and harmonics
C - Calibration lines
M - Power mains (60 HZ)
s - 16-Hz comb
e - 8-Hz comb (odd harmonics)
O - 1-Hz comb (0.5-Hz offset)
o - 1-Hz comb (zero offset)
H - 99.9989-Hz comb
J - 31.4149-Hz comb
K - 0.088425-Hz comb
x - single line

Figure 1 - 0-2000 Hz (Early O1 data with line labels)
Figure 2 - 20-100 Hz sub-band (shows complexity of combs below ~70 Hz
Figure 3 - 1300-1400 Hz sub-band (shows how clean the noise floor is away from 8-Hz, 16-Hz lines at high frequencies
Figure 4 - 0-2000 Hz (Early ER8 and O1 data comparison, no labels)

Attachments:
* Zip file with miscellaneous sub-band spectra for early O1 data (with line labels)
* Zip file with sub-band spectra comparing early ER8 and early O1 data
* Flat-file list of lines marked on figures

LHO hosted a public event this afternoon.  Arrival time at LSB = 1:00 - 2:30 PM PM.  Departure time = 3:00 - 5:00 PM.  Group size = ~170 adults/children of all ages.  Vehicles at the LSB = ~50 passenger cars (very uncertain).  A series of three walking tours between 3:00 and 5:00 brought a total of ~120 people into the control room during this time span. 

TITLE: 09/26 [DAY Shift]: 15:00-23:00 UTC (08:00-16:00 PDT), all times posted in UTC

STATE Of H1: Observing at ~75 Mpc

SUPPORT: Mike

QUICK SUMMARY: Less 45MHz glitches during my shift. DHARD Y boost turned off (as MIke requested). Wind speed coming below 20 mph. Several small earthquakes throughout the shift.

INCOMING OPERATOR: Patrick

ACTIVITY LOG:

18:07 Jodi to high bay area looking for contamination control stuff. Still observing.

18:14 Jodi out

19:08 Switched the ifo to Commissioning to turn off the DHARD Y boost filter. 

19:09 Back to Observing

22:33 A big tour group starting to march in the control room. ~20 adults. There was also a 5.1M earthquake at Pacific-Antarctic Ridge arrving at about the same time so there were glitches in DARM during the time. Just in case anyone wants to look into outreach noise. 

22:46 Group leaving.

23:00 Handing off to Patrick.

TITLE: 09/26 [EVE Shift]: 23:00-07:00 UTC (16:00-00:00 PDT), all times posted in UTC
STATE Of H1: Observing at ~71 MPc
OUTGOING OPERATOR: Nutsinee
QUICK SUMMARY: Lights appear off in the LVEA, PSL enclosure, end X, end Y and mid X. I can not tell from the camera if they are off at mid Y. Seismic in 0.03 - 0.1 Hz band is coming down from a small peak to ~.7 um/s. Seismic in 0.1 - 0.3 Hz band is around .09 um/s. Wind speeds are less than 20mph, may be coming down.

The LIGO-Caltech systems folks have set up a dedicated account for GWIstat and the associated processes that Chad set up to feed it information about the LIGO detectors, running on the ldas-grid machine at Caltech.  The new URL for the GWIstat display is https://ldas-jobs.ligo.caltech.edu/~gwistat/, and it does not require ligo.org authentication any more.  The old URL now redirects to the new URL.  (At least I think so -- let me know if you run into any problems.)

Due to moving GWIstat to its new location, the durations of the current observing segments are not reported correctly.

Parameters:

GPS Start Time = 1127247535 # Beginning of time span, in GPS seconds, to search for injections
GPS End Time = 1127333935 # Ending of time span, in GPS seconds, to search for injections
Check Hanford IFO = True # Check for injections in the Hanford IFO frame files.
Check Livingston IFO = True # Check for injections in the Livingston IFO frame files.
IFO Coinc Time = 0.01 # Time window, in seconds, for coincidence between IFO injection events.
Check ODC_HOFT = True # Check ODC-MASTER_CHANNEL_OUT_DQ channel in HOFT frames.
Check ODC_RAW = True # Check ODC-MASTER_CHANNEL_OUT_DQ channel in RAW frames.
Check ODC_RDS = True # Check ODC-MASTER_CHANNEL_OUT_DQ channel in RDS frames.
Check GDS_HOFT = True # Check GDS-CALIB_STATE_VECTOR channel in HOFT frames.
Report Normal = True # Include normal (IFO-coincident, consistent, and scheduled for network injections and consistent and scheduled for IFO injections) injections in report
Report Anomalous = True # Include anomalous (non-IFO-coincident, inconsistent, or unscheduled) injections in report

All injections found were UNSCHEDULED CBC injections in H1. There were no injections found in L1. All injections were consistent.

Mike asked me to turn it off to maintain the configuration. So I went off observing for a bit to switch off the boost filter. I accepted the change in SDF and back to observing. All this took less than a minute.

TITLE: 09/26 [DAY Shift]: 15:00-23:00 UTC (08:00-16:00 PDT), all times posted in UTC

STATE Of H1: Observing. The range fluctuates between ~70-40Mpc most likely due to RF45 glitches.

OUTGOING OPERATOR: Ed M.

QUICK SUMMARY: RF 45MHz has been glitching. Wind speed seems to be increasing (now ~30 mph). Seismic activity in earthquake band increasing coresponding to the wind speed. Last earthquake report from Terramon was about an hour ago.

TITLE:  Sep 26 OWL Shift 23:00-07:00UTC (16:00-00:00 PDT), all times posted in UTC

STATE Of H1: Observing

LOCK DURATION: Entire Shift

SUPPORT: Mike Landry

INCOMING OPERATOR: Nutsinee

Activity log:

13:59 After tweaking, 45Mhz RFAM held stable for ~15 minutes. Still glitching but not as frequently.

14:05 looks like a quake is incoming. Engaging Sheila’s DHARD_Y boost.

14:25 Wind on the X arm is picking up to around 16 to 20mph.

14:28 accepted DHARD_Y boost change in SDF and going to go back to observing.

14:31 Intent bit set to Undisturbed

End-Shift Summary: Last half of my shift has been dominated by a continuing loss of range due to constant low frequency glitching in DARM. Mike visited me and we realized that it was the 45Mhz RFAM issue. We toggled the gain a bit as per Stefan’s aLog and then reverted it back. This seems to have lessened the occurrences although it will still happen sporadically. IFO was locked for the entire shift. ETMY glitched twice around 08:00UTC and didn't glitch again for about 4 hours. Then they came at the typical frequency. Mexican EQ showed on the graph so while we were out of Observing I switched on Sheila’s DHARD_Y boost and accepted the change to be able to return to Undisturbed. Neither Terramon nor the USGS site showed the quake before our graph did. nice. Handing off to Nustinee.

Now that the multi-processing version of HWInjReport is operational and returning results (multi-processing code is an absolute bear!), this is the first of daily reports of analysis of output from HWInjReport. Currently, HWInjReport still has to be run manually and requires one to checkout the current schedule file from the SVN (currently at https://svn.ligo.caltech.edu/svn/dac/hwinj/Details/tinj/schedule; if this has changed, please let me know so I can modify the run script appropriately). Automatic execution of HWInjReport is soon to come.  I've attached copies of the output report file and the schedule file used.  NOTE: the report file is very wide due to the number and size of the columns in the network injections tables of the report.  To examine the report, you will need to either zoom out or change the font size in your browser/text editor to 10pt or less (I'm looking into compressing the columns in the network injections tables in future updates).

The daily run performed with the following parameters:

GPS Start Time = 1127163296    # Beginning of time span, in GPS seconds, to search for injections
GPS End Time = 1127249696    # Ending of time span, in GPS seconds, to search for injections
Check Hanford IFO = True    # Check for injections in the Hanford IFO frame files.
Check Livingston IFO = True    # Check for injections in the Livingston IFO frame files.
IFO Coinc Time = 0.01    # Time window, in seconds, for coincidence between IFO injection events.
Check ODC_HOFT = True    # Check ODC-MASTER_CHANNEL_OUT_DQ channel in HOFT frames.
Check ODC_RAW = True    # Check ODC-MASTER_CHANNEL_OUT_DQ channel in RAW frames.
Check ODC_RDS = True    # Check ODC-MASTER_CHANNEL_OUT_DQ channel in RDS frames.
Check GDS_HOFT = True    # Check GDS-CALIB_STATE_VECTOR channel in HOFT frames.
Report Normal = True    # Include normal (coherent, consistent, and scheduled) injections in report
Report Anomalous = True    # Include anomalous (incoherent, inconsistent, or unscheduled) injections in report

NOTE: coherent -> coincident.  Missed changing that in the code where it outputs the report.

The schedule file only has injections spanning 1125280499 - 1126450499.  This is outside the range of times checked by HWInjReport, so there are no occurring or non-occurring scheduled injections reported.

No normal injections, as defined above for HWInjReport, were reported for the network injections.  All injections found were reported as UNSCHEDULED, and all injections occurring were reported as CBC injections.

Two H1-L1 coincident injections were found: CBC 1127175853.757(H1), 1127175853.764(L1) and CBC 1127179822.757(H1), 1127179822.764(L1).  Both injections were reported as UNSCHEDULED but, otherwise, had no other reported anomalies.

H1 had only 1 single-IFO injection, CBC 1127173426.757 (the report file shows 3, but only 1 is actually an H1-only injection.  There is apparently a bug due to the multi-processing code that is not propagating the association of the other 2 injections with their corresponding L1 injections. It's basically a problem of how memory works for a multi-processing environment.)

L1 had 5 single-IFO injections:

• CBC 1127162126.764
• CBC 1127162532.764
• CBC 1127162961.764
• CBC 1127181622.764
• CBC 1127183422.764

The first three injections have the anomaly that they occur in the ODC hoft and GDS hoft frames but not in the ODC raw or ODC rds frames. The remaining 2, other than being UNSCHEDULED, had no other anomalies reported.

ADDENDUM: I was able to successfully fix the data propagation bug.  I've attached a copy of the resulting "fixed" report that correctly shows the single-IFO injections for H1 and L1.

14:31UTC Intent bit set to Undisturbed

Ed, Mike L

13:50UTC addressing RF45 AM issue that has been ongoing for about 4.5 hours. Mike L and I decided to try a reduction in the LSC-MOD_RF_45_AM gain by 1dB to as per Stefan's aLog https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=21716

Mid-Shift Summary: IFO Locked @ 64Mpc (lower than normal due to funny business visible in DARM). No obtrusive Seismic or wind activity. 2 ETMY glitches. Some increasing activity in the lower DARM frequencies are causing drops in range. It just keeps bouncing up and down (see screenshots) Is this a combination of bounce and roll modes ringing up? This was the same activity that I noted in my transition summary. Is it from anthropogenic noise?