Richard, Gerardo, Jim, Dave:

While the IFO was in commissioning we went to Mid-Y to investigate the h1pemmy failure which happened at 03:53 PDT this morning. We found that the IO Chassis was not powered up, but the +24V power supply was running. We found that the fuse in the fuse-units at the top of the DC power supply rack had blown for the +24V line. We replaced the fuse, powered up the IO Chassis and noted the DC power supply was drawing at steady 3.5A as expected. It was then we noticed the AA-Chassis was not on. It has a fuse on the +18V line which had blown. Putting a new fuse in caused a hot electrical odor from the AA-Chassis. The fuse was removed to de-energize the AA.

Initially we left all the systems down (computer and IO Chassis) and returned to the corner station. But with h1pemmy not running, the DAQ EDCU and CONLOG were red on the overview screen. We remotely powered up the h1pemmy computer, which then started the h1ioppemmy and h1pemmy models. Wtith the EPICS IOCs running, this has appeased EDCU and CONLOG. For now the two models have FEC and DAQ errors as they are not actually running, only the epics IOC is running. We will remain in this state until the AA Chassis is runnng again.

A trend of a seismic signal and the front end status shows they both went down at 03:53 PDT Thu 01 oct 2015. We presume the failure of the AA-chassis and the overcurrent of the 18V glitched the 24V line which in turn blew the fuse.

Activity Log: All Times in UTC (PT)

15:00 (08:00) Take over from TJ
15:05 (08:05) GRB Alert – Switch to Observing mode
16:33 (09:33) Jodi – Delivering storage items to mechanical building
16:59 (09:59) Jodi – Finished with delivery
18:31 (11:31) GRB Alert – In Observing mode before notice
20:47 (13:47) Switch to Commissioning mode – Commissioning work while LLO is down
21:26 (14:26) Kyle & Gerardo – Going to Mid-Y to start rotating pump and leak detector 
21:35 (14:35) Dave & Jim – Going to Mid-Y to check on PEM problem
22:31 (15:31) Filiberto – Going to Mid-Y to work on bad AA chassis 
22:45 (15:45) Lockloss – Commissioning activities
23:00 (16:00) Turn over to Travis


End of Shift Summary:

Title: 10/01/2015, Day Shift 15:00 – 23:00 (08:00 – 16:00) All times in UTC (PT)

Support: Sheila, Mike, 

Incoming Operator: Travis

Shift Summary: 
- 15:00 IFO locked. Intent Bit = Commissioning Mode. Wind is calm, no seismic activity. All appears normal. Sheila doing some testing while LLO is relocking.
    
- 17:05 (08:05) GRB Alert. Switch to Observing mode
- 18:31 (11:31) GRB Alert. In Observing mode 
- 23:00 (16:00) Relocking  

Just in case you're wondering why LHO sees two noise bumps at 315 and 350Hz (attached, middle blue) but not at LLO, we don't fully understand either but here is the summary.

There are three things here, environmental noise level, PZT servo, and jitter coupling to DARM. Even though the former two explains a part of the LLO-LHO difference, they cannot explain all of it, and the coupling at LHO seems to be larger.

Reducing the PSL chiller flow will help but that's not a solution for the future.

Reimplementing PZT servo at LHO will help and this should be done. Squashing it all will be hard, though, as we are talking about the jitter between 300 and 370Hz and there's a resonance at 620Hz.

Reducing coupling is one area that was not well explored. Past attempts at LHO were on top of dubious IMC WFS quadrant gain imbalances.

1. Environmental difference

These bumps are supposed to be from the beam jitter caused by PSL periscope resonances (not from the PZT mirror resonances). In the attached you can see that the bumps in H1 (middle blue) correspond to the bumps in PSL periscope accelerometer (top blue). (Don't worry, we figured out which server we need to use for DTT to give us correct results.)

Because of the PSL chiller flow difference between LLO and LHO (LHO alog, couldn't find LLO alog but we have MattH's words), in general LLO periscope noise level is lower than LHO. However, the difference in the accelerometer signal is not enough to explain the difference in IFO.

For example, at 350Hz LHO PSL periscope is only a factor of 2 noisier than LLO. At 330Hz, LHO is quieter than LLO by more than a factor of 2. Yet we have a huge hump in DARM at LHO, it becomes larger and smaller in DARM but it never goes away, while LLO DARM is deat flat.

At LLO they do have a servo to supress noise at about 300Hz, but it shouldn't be doing much if any at 350Hz (see the next section).

So yes, it seems like environmental difference is one of the reasons why we have larger noise.

But the jitter to DARM coupling itself seems to be larger.

Turning down the chiller flow will help but that's not a solution for the future.

2. Servo difference

At LLO there's a servo to squash beam jitter in PIT at 300Hz. LHO used to have it but now it is disabled.

At LLO, IOOWFS_A_I_PIT signal is used to suppress PIT jitter targetting the 300Hz peak which was right on some mechanical resonance/notch structure in PZT PIT (which LHO also has), and the servo reduced the noise between about 270 and about 320Hz (LLO alog 19310).

Same servo was successfully copied to LHO with some modification, which also targeted 300Hz bump (except that YAW was more coherent than PIT and we used YAW signal), with somewhat less (but not much less) aggressive gain and bandwidth. At that time 300Hz bump was problematic together with 250Hz bump and 350Hz bump. Look at the plots from alog 20059 and 20093.

Somehow 250Hz and 300Hz subsided, and now LHO is suffering from 315Hz and 350Hz bumps (compare the attached with the above mentioned alog). Since we never had time to tune the servo filter to target either of the new bumps, and since turning the servo on without modification is going to make marginal improvement at 300Hz and will make 250Hz/350Hz somewhat worse due to gain peaking, it was disabled.

Reimplementing the servo to target 315 and 350Hz bumps will help.  But it's not going to be easy to make this servo wide band enough to squash everything because of 620Hz resonance, which is probably something in the PZT mirror itself (look at the above mentioned alog 20059 for open loop transfer function of the current servo, for example). In principle we can go even wider band, but we'll need more than 2kHz sampling rate for that. We could stiffen the mount if 620Hz is indeed the mount.

3. Coupling difference

As I wrote in the environment difference, from the accelerometer data and IFO signal, it seems as if the coupling is larger at LHO.

There are many jitter coupling measurements at LHO but the best one to look at is this one. We should be able to make a direct comparison with LLO but I haven't looked.

Anyway, it is known that the coupling depends on IMC alignment and OMC alignment (and probably the IFO alignment).

At LHO, IMC WFS has offsets in PIT and YAW in an attempt to minimize the coupling. This is on top of dubious imbalances in IMC WFS quadrant gains at LHO (see alog 20065, the minimum quadrant gain is a factor of 16  larger  smaller than the maximum). We should fix that before spending much time on studying the jitter coupling via alignment.

At LLO, there's no such imbalance and there's no such offset.

Chris Biwer and other members of the hardware injections team will likely be doing coherent hardware injections in the near future, and these will hopefully be detected successfully by one or more of the low-latency data analysis pipelines.  Currently, we are still testing the EM follow-up infrastructure, so the "Approval Processor" software is configured to treat hardware injections like regular triggers.  Therefore, these significant GW "event candidates" should cause audible alarms to sound in each control room, similar to a GRB alarm.  The operator at each site will be asked to "sign off" by going to the GraceDB page for the trigger and answering the question, "At the time of the event, was the operating status of the detector basically okay, or not?"  You can also enter a comment.

For the purpose of these tests, if you are the operator on shift, please:
  * Do not disqualify the trigger based on it being a hardware injection -- we know it is!  So, please sign off with "OKAY" if the detector was otherwise operating OK.
  * Pay attention to whether the audible alarm sounded.  In the past we had issues at one site or the other, so this is one of the things we want to test.
  * Feel free to enter a comment on the GraceDB page when you sign off, like maybe "this was a hardware injection and the audible alarm sounded".
  * You may get a phone call from a "follow-up advocate" who is on shift to remotely help check the trigger.

Note: in the future, once the EM follow-up project is "live", a known hardware injection will not cause the control-room alarms to sound (unless it is a blind injection).  You should not write anything in the alog about alarms from GW event candidates, because that is potentially sensitive information and the alogs are publicly readable.

IFO has been locked at NOMINAL_LOW_NOISE, 23.0W, 72Mpc for the past 5 hours. Wind and seismic activity are low. 4 ETM-Y saturation alarms. Received GRB alert at 18:31UTC (12:31PT) - LHO was in Observing mode during this event      

The attached plot shows the 2 day trend of the RF45 glitches. There were no glitches in the past day. The large glitches 24 hours ago were us. This is not inconsistent with a cable or connection problem. No one should be surprised, if the problem reappears.

Title:  10/01/2015, Day Shift 15:00 – 23:00 (08:00 – 16:00) All times in UTC (PT)

State of H1: At 15:00 (08:00) Locked at NOMINAL_LOW_NOISE, 23.0W, 72Mpc

Outgoing Operator: TJ

Quick Summary: Wind is calm, no seismic activity. All appears normal. Intent Bit at Commissioning while LLO was recovering from a lockloss.     

• Title: 10/1 OWL Shift: 7:00-15:00UTC (00:00-8:00PDT), all times posted in UTC
  

• State of H1: Locked but not Observing for inj
  

• Shift Summary: I had one lockloss, but it came back up with relative ease. The RF Noise wasn't bothering me.

• Incoming Operator: Jeff B

• Activity Log:

• 7:18 - ETMY sat
• 8:21 - GRB alarm
• 8:23 - ETMY sat
• 9:17 - ETMY sat
• 11:47 - GraceDB query failure
• 12:03 - Restarted ext_alert.py, this remedied the query failue
• 12:30 - GraceDB query failure on and off for about 20 min. The problem sorted itself out before I restarted anything again.
• 12:39 - ETMY sat
• 13:59 - Lockloss
• 14:03 - Richard using forklift near OSB
• 14:13 - Bubba into LVEA to look at storage containers
• 14:18 - Richard done
• 14:32 - Bubba out
• 14:38 - Observing
• 14:42 - Out of Observing for injection while LLO is down.

Relocked @ 14:38

Sheila wants to do a quick injection while LLO is down.

Lockloss @ 13:59 UTC

ITMX saturation, and it tripped SUS OMC WD. No obvious reason for lockloss.

H1IOPPEMMY and H1PEMMY both started to report errors for FE, ADC, and DDC on the CDS Overview around 12:55 UTC. There was a red around TDS, so I checked out the timing screen and there seems to be a problem with Port 13 "Invalid or no data".

Since this is only PEM at MidY, I have NOT taken us out of Observing.

Humming along @ 75Mpc. Have had a handful of glitches during my shift, but the RF noise seems to be in control for now.

J. Kissel, for the Calibration Team

I've updated the results from LHO aLOG 21825 and G1501223 with an ASD from the current lock stretch, such that I could display the computed time dependent correction factors, which have recently been cleared of systematics (LHO aLOG 22056), sign errors (LHO aLOG 21601), and bugs yesterday (22090). 

I'm happy to say, that not only does the ASD *without* time dependent corrections still fall happily within the required 10%, but if one eye-balls the time-dependent corrections and how they would be applied at each of the respective calibration line frequencies, they make sense.

To look at all relevant plots (probably only interesting to calibrators and their reviewers), look at the first pdf attachment. The second and third .pdfs are the money plots, and the text files are a raw ascii dump of respective curves so you can plot them however or whereever you like. All of these files are identical to what is in G1501223.

This analysis and plots have been made by
/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O1/H1/Scripts/produceofficialstrainasds_O1.m
which has been committed to the svn.

• Title: 10/1 OWL Shift: 7:00-15:00UTC (00:00-8:00PDT), all times posted in UTC
  

• State of H1: Observation Mode at 75Mpc for the last 10hrs
  

• Outgoing Operator: Travis S

• Quick Summary: Locked his whole shift.

Title: 9/30 Eve Shift 23:00-7:00 UTC (16:00-24:00 PST).  All times in UTC.

State of H1: Observing

Shift Summary: No news is good news.  Locked my entire shift in Observing.  Only 4 ETMy saturations.  Seismic and wind calm.  No RF45 issues.

Incoming operator: TJ

Activity log:

23:38 ETMy saturation

3:07 ETMy saturation

4:56 ETMy saturation

6:36 ETMy saturation

Following on from where Hugh left off in alog 21412, I have copied the OBSERVE.snap files from their target area to their appropriate userapps/...h1xxxxx_observe.snap area in prep for committing them to svn.  I have copied these files for: SUS, ALS, ISC, CALEX, CALEY.  Hugh had already done LSC LSCAUX ASC ASCIMC OMC OAF TCSCS CALCS, and all SEI.  The balance (AUX IOP ODC and PEM) have no observe.snap file since it is unneccessary.  The ones that I've just copied will be committed to svn tomorrow.

C. Biwer, J. Kissel

Taking advantange of single IFO time to run PCAL vs DARM hardware injections. More details later.

The ext_alert.py script which periodically views GraceDB had failed. I have just restarted it, instructions for restarting are in https://lhocds.ligo-wa.caltech.edu/wiki/ExternalAlertNotification

Getting this process to autostart is now on our high priority list (FRS3415).

here is the error message displayed before I did the restart.

 File "ext_alert.py", line 150, in query_gracedb

    return query_gracedb(start, end, connection=connection, test=test)

  File "ext_alert.py", line 150, in query_gracedb

    return query_gracedb(start, end, connection=connection, test=test)

  File "ext_alert.py", line 135, in query_gracedb

    external = log_query(connection, 'External %d .. %d' % (start, end))

  File "ext_alert.py", line 163, in log_query

    return list(connection.events(query))

  File "/usr/lib/python2.7/dist-packages/ligo/gracedb/rest.py", line 441, in events 

    uri = self.links['events']

  File "/usr/lib/python2.7/dist-packages/ligo/gracedb/rest.py", line 284, in links  

    return self.service_info.get('links')

  File "/usr/lib/python2.7/dist-packages/ligo/gracedb/rest.py", line 279, in service_info

    self._service_info = self.request("GET", self.service_url).json()

  File "/usr/lib/python2.7/dist-packages/ligo/gracedb/rest.py", line 325, in request

    return GsiRest.request(self, method, *args, **kwargs)

  File "/usr/lib/python2.7/dist-packages/ligo/gracedb/rest.py", line 201, in request

    response = conn.getresponse()

  File "/usr/lib/python2.7/httplib.py", line 1038, in getresponse

    response.begin()

  File "/usr/lib/python2.7/httplib.py", line 415, in begin

    version, status, reason = self._read_status()

  File "/usr/lib/python2.7/httplib.py", line 371, in _read_status

    line = self.fp.readline(_MAXLINE + 1)

  File "/usr/lib/python2.7/socket.py", line 476, in readline

    data = self._sock.recv(self._rbufsize)

  File "/usr/lib/python2.7/ssl.py", line 241, in recv

    return self.read(buflen)

  File "/usr/lib/python2.7/ssl.py", line 160, in read

    return self._sslobj.read(len)

ssl.SSLError: The read operation timed out