TITLE: 06/20 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Commissioning
INCOMING OPERATOR: Jeff
SHIFT SUMMARY: Sheila and Georgia currently taking IFO to observing without squeezing for noise budget measurements.
LOG:

16:25 UTC Sundae to optics lab
16:41 UTC Corey filling small LN2 dewar
17:06 UTC Lock loss
17:08 UTC Filiberto to vault
17:09 UTC Corey done
17:37 UTC Chandra, Kyle and Gerardo to mid Y
17:40 UTC Filiberto back
17:46 UTC Lockloss from ENGAGE_SOFT_LOOPS
18:08 UTC Dave to end X
18:10 UTC Karen to mid Y
18:17 UTC Dave at end X
18:28 UTC Dave back
18:31 UTC GRB alert (E336744) Unlocked, ignoring
18:45 UTC Observing
19:29 UTC Ace through gate to end X to fix portapotty
19:44 UTC Ace at end X
19:45 UTC Portapotty stood back up
19:46 UTC Portapotty being sprayed down
19:51 UTC Ace leaving end X
20:06 UTC Tour outside control room
20:11 UTC Ace leaving site
21:20 UTC Ken to mechanical warehouse
21:30 UTC Mike leading tour down x arm (in car)
21:31 UTC Ken done
21:46 UTC GRB alert (E336758) Confirmed with LLO. LLO says to stand down. Called Mike, to avoid driving down arm. Says he is at overpass and will return to the control room. EDIT: Just realized I had forgotten to set INJ_TRANS to INJECT_KILL. Too late at this point.
22:16 UTC Kyle done at mid Y
22:27 UTC Mike leading tour out of control room
22:56 UTC Sheila and Georgia taking IFO to observing without squeezing

Here are some MC_F spectra.

18:45 UTC

Accepted attached ASC SDF differences.

16:56 - 17:05 UTC: Vehicle near CP1 to fill small LN2 dewar for outreach.

I needed to fill our small LN2 dewar for an outreach event in a few hours & wanted to tag this activity under DetChar.

This activity is not on our Accepted/Not Accepted document (T1500425) for Observing Run activities, but when I asked the operator if it was OK, we had Robert nearby & were able to defer to him.  He said this is fine.  I mentioned driving my vehicle nearby and his only concern was a vehicle driving on the inner road (near H2 Electronics room) since this is closer to H1 than the standard Y-arm path.  I decided to avoid that path as much as possible and drove on the Y-arm road and backed in (photo attached) toward CP1 a little bit.  

Just to be mindful of rare activities such as this, Robert asked me to alog the times of this activity.

All plots in good nominal ranges

TITLE: 06/20 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 113Mpc
OUTGOING OPERATOR: Niko
CURRENT ENVIRONMENT:
    Wind: 19mph Gusts, 16mph 5min avg
    Primary useism: 0.04 μm/s
    Secondary useism: 0.07 μm/s 
QUICK SUMMARY:

No issues.

TITLE: 06/20 Owl Shift 07:00 – 15:00 (00:00-08:00), all times posted in UTC

STATE of H1: Observing

INCOMING OPERATOR: Patrick

SHIFT SUMMARY: Quiet shift with a few GRB’s, medium winds, and one EQ scare that turned out to be no big deal

LOG:

07:00 (00:00) Start of shift

08:14 (01:14) Switching from WINDY to EARTH_QUAKE mode for an incoming 5.7 from Niue

08:19 (01:19) Switching back to WINDY after lackluster EQ

12:10 (05:10) GRB (E336722). Trigger duration too long, can be ignored

13:53 (06:53) GRB (E336728). Trigger matches criteria, standing down for 15 minutes

14:18 (07:18) Peter to Optics Lab

15:00 (08:00) End of shift

DQ Shifter: Evan Goetz
DQ Shift Mentor: Beverly Berger
LSC Fellow(s): Laurence Datrier

DQ shift wiki page

Summary (highlights) of the shift:

   * LHO cluster problems crashed summary pages Friday and recovery is ongoing. This was due to the large number of files from summary pages/plots and other DetChar tools (~500 million combined). There are some missing or incomplete plots starting Thursday, but many more Friday/Saturday. Omicron and Hveto are incomplete or non-existant from Friday - Sunday.
   * Observing summary: (71.06 + 59.53 + 62.41+ 98.06 + 35.29 + 90.35 + 67.60) / 7 = 69.19%; more or less in line the mean observing time for H1 during O3 thus far.
   * Several earthquakes in the latter part of the week took away from observing time. Also, when H1 was locked or re-locked near earthquake times, the whistle glitches were much more problematic because of the increased ground motion.
   * Whistle glitches remain problematic for the early part of the week [Omicron problems prevented late week Hveto updates] as evidenced by round 1 or 2 channel winner LSC-REFL_A_LF_OUT_DQ. The other ongoing common round 1 or 2 winner remained LSC-POP_A_LF_OUT_DQ.
   * Low frequency narrow spectral artifacts in H1 remain problematic. They are quite dynamic (changing day by day) and very prevalent below 100 Hz compared with L1.
   * The 48 Hz feature is still present with varying amplitude, as reported by previous DQ shifts. I looked through some accelerometer spectra and I am unsure if I see correlation by eye. I will continue to look in more detail.
   * Several other features seem to have cropped up this weekend at the start of lock stretches, but then seem to reduce in amplutide during the course of a lock stretch (~550 Hz, ~850 Hz, possible violin harmonic near 1010 Hz) - see below.
   * PCALX showed some differences in the spectrum [likely not immediately problematic for h(t)], perhaps indicative of an unstable optical follower servo as resetting the offset has helped the stability.
   * Later in the week violin resonances seem more rung up at the start of locks than early in the week.

Note that I'll update the shift summary once some of the backfilled summary pages are updated.

Around 4:17 pm local ion pump #6 in corner station tripped. I restarted the power supply and it seems to have recovered normally down to 162 micro-amps and falling.

18:31:36-18:32:58 Out of OBSERVING

Once again H1 was dropped out of OBSERVING (for ~82 secs) due to the another case of the laser unlocking (lasted ~6hrs since last occurence)  as mentioned in Jim's shifts.  Saw same symptoms as Jim:

• Nothing on SDF:  This is not an issue due to SDF dropping us out, it is due to a Monitored Guardian Node NOT being in a NOMINAL state.
• "Monitored Nodes" list on the Observation Intent Bit screen listed 1 Guardian Node as "NOT OK":  the TCS_ITM_CO2 guardian node.
• Opened up TCS_ITM_CO2 Guardian Node it was actively trying to relock the laser all on its own (SDF was fine and green through this).
• Guardian Log looks similar to what Jim posted earlier today.

The Pcal reciever-side photodiode at EX is showing unwanted spikes at various frequencies. Jeff helpfully pointed out that these changes started after we performed an end station calibration during maintenance yesterday. Looking at the spectra of the transmitter-side photodiode and reciever-side photodiode before and after maintenance (pcalx_spec.pdf), we can see that there is indeed a significant difference, and the fact that we see it in both photodiodes tells us that these changes are happening inside the transmitter module.

Looking average time series of both photodiodes before and after maintenance (pcalx_trends.png) shows that the voltage output of these sensors is much less stable after we perform the end station calibration.

Given that the voltage on the photodiodes (and hence the laser power) appears to to be unstable and larger than before, I would hazard a guess that the issue is with the Optical Follower Servo and not with the laser dying. However, I'm not sure how our work yesterday would have caused this. We will continue to investigate.

~22:03 Seismon:  Jenne noticed DIAG_MAIN gave note about "Seismon system not updating"

• On Seismon medm, there was a SEISMON Is Dead red box, but along with this, the "Keep Alive" was flashing away.
  • The Sys Time (GPS) time was also frozen.
• Gave a call to LLO to check on their status (if they were good, was going to have them give us warnings for any EQs), but they (JoeH) had a similar issue.
• Brought this to Dave B's attention, but....
• At 22:14 SEISMON came back on its own.  (Looking at a recent email from Jim W, think this was a code issue due to a time hiccup and it came back on its own.)
• Will send an email to Jim for his notes.

Summary: Looking over O3 H1 data so far, spectral lines in 1800 s long Fourier transforms continue to be problematic and will be degrade astrophysical results of continuous gravitational wave searches. It would be helpful if commissioning/engineering teams can compare with what the LLO team has done in order to mitigate most spectral contamination.

I'll note that I'm getting greedy because O3 data is already much better for spectral content than O2 data for both detectors (and better broadband sensitivity!). Again, my thanks to the entire team on the progress over O2! When I see how clean of spectral features L1 data appears compared to H1, however, I want both detectors to have equally clean data. I hope some improvements to hardware configuration will mitigate the H1 spectral lines, but this needs attention from boots on the ground at LHO.

Details:
Keith Riles has a very nice week-by-week and running cumulative pages for looking at H1 and L1 spectral data computed from the standard set of 1800 s long, Tukey windowed Short Fourier Transforms--SFTs (see here for H1 and here for L1). 

I want to draw attention in particular to two problematic features that require attention:
1) Below 100 Hz, the prevalence of lines in H1 data is much greater than L1. See attached figure 1 (H1) compared with figure 2 (L1) showing both the most recent week of data and the cumulative O3 data up to now.

2) The behavior of the spectral features is quite dynamic from day to day and week to week. This can be seen, for example, in attached figure 3 showing the ratio of this week's H1 data divided by the cumulative H1 data. Spikes above the mean show an increase in line activity, while spikes dropping below the mean show a decrease in line activity. Every week shows a slightly different figure, meaning the lines and/or coupling is changing with time. Figure 4 shows the L1 data, which is much better behaved, probably because there are so few lines present in the data.

These features are likely going to limit some searches for long duration gravitational wave signals, and may obscure putative signals that could be detectable during O3. With lines present in the data, especially when the lines are dynamic as they appear to be in H1, will require changes to searches and increase the amount of time spent to deal with these artifacts in post-processing. With improvements (as demonstrated by L1), the searches will be more sensitive and can be published more timely.

Apparently this occurred at 20:20 UTC. I had muted Verbal Alarms while Amber was in the control room recording and interview with a TV reporter and forgot to UNmute. Apologies for that one. The SDF diff that occurred is still a bit of a mystery and perhap[s be UNmonitored?

20:30 Accepted diff and returned to OBSERVING

J. Kissel

After restarting the front wall Main DELTAL EXTERNAL ASD figure of merit, I saw that PCALX did not show the expected high-frequency roaming line. A quick look at the PCALX overview, and it was easy to see that the optical follower servo (OFS, the intensity stabilization and linearization servo for PCAL) was railed. The main time-series trend on the overview, of the OFS PD -- H1:CAL-PCALX_OFS_PD_OUTMON -- was steady at -7.7 V, with no sign of the expected ~kHz oscillation. 

I trended back to find out when the OFS had railed, and it appears coincident with the Beckhoff restarts this morning at 15:06:46 UTC (08:06:46 PDT). The OFS PD went to zero then, and then a few minutes later (at 15:13:44 UTC, 08:13:44 PDT), after bouncing around between OFF and RAILED, it settled on RAILED.

I fixed this by turning OFF, then turning ON the OFS loop, i.e. toggling the orange H1:CAL-PCALX_OPTICALFOLLOWERSERVOENABLE "loop enable" switch next to the diagram of the loop -- also on the PCALX overview. I did so at 23:11:55 UTC (16:11:55 UTC), before we resumed any observation ready segments, so this has no permanent impact on that quality of data.

We should consider adding "check on the PCALs!!" to the recovery checklist after any maintenance day in which a Beckhoff restart has taken place.
We should also add "Restart the wall FOMs!" to the recovery checklist after any maintenance day in which a DAQ restart has taken place.
Such activities should be considered akin to a physical VEA sweep done at the close maintenance.

Operators have been saying that they have difficulty adjusting the ALS polarizations lately.  I tried to adjust the Y fiber polarizaton today and saw the stange behavoir in the attached screenshot.  There are certain steps that are not like the others, and seem to make a large jump in the polarization. 

After the laser power was reduced, there is no longer enough power coming out of SHG to deliver 20 mW into the fiber. This problem was compounded a few days ago when the offset to the power servo was reset to the wrong sign. Compared to the past, we currently run with about 70% of the power into the OPO (or ~14 mW into the fiber). The SHG output power currently reads about 26 mW when it was up 50 mW in the past. HW9 on ISCT6 needs to be adjusted.

Following CPS are over threshold:

• BS_ST2_CPSINF_V1_I high freq noise is high!