Jeff B, Jim B, Carlos, Fil The EPICS IOC for the LVEA and PSL dust monitors was restarted this morning because it stopped working. Upon a lengthy investigation it was determined that someone unplugged and removed the power cord for the Comtrol unit which was under the beam tube behind the vacuum rack on the X arm. The cord appears to have been removed this morning based on trends of dust channels.
HWSX Code has been resumed. HWSY is not running.
I also want to add an interesting finding while doing this test. So I unplugged one fiber at a time to see which camera stopped streaming. Instead of a complete black out both camera seemed to be streaming off the other while being unplugged. This is exactly what happened as reported in alog32019.


I've restarted the primary and redundant h(t) pipelines at Hanford at GPS second 1165664237. This restart picks up a software update to gstlal-calibration-1.1.2-v1, which contains a couple of bug fixes:
(1) The h(t) data product should now be identical between the primary and redundant pipelines when time-dependent corrections are applied (i.e., the previously observed numerical roundoff error has been addressed);
(2) The pipeline will now run properly in offline mode, which had become a serious problem after the first bug fix referred to above.
The filters file has also been changed; see this aLOG: https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=32389
Note, time-dependent corrections to h(t) (the kappa factors) are still being applied in low latency at Hanford, with the exception of changes to the coupled cavity pole frequency.
The most recent bug fix that prevents the pipeline from hanging does so by adding a 20 second latency to the low-pass-filter used in demodulating the time-dependent calibration factors. This was necessary so the the timestamps on this stream are aligned with the timestamps of the coherence uncertainty channels. Otherwise, the gate element that receives these misaligned streams will lock up, causing the pipeline to hang. However, I have now realized that this introduces minor subtlety -- the coherence gating does not take this additional 20 seconds into account. This introduces the possibility that there could be undesirable jumps in the kappas in C01 frames produced by this version of the code that do not show up in C00 (Similar to what is discussed in https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=31911 , which has been addressed in a recent bug fix). I have a different fix to this, for which I will request another release for next Tuesday. Instead of adding a latency to the low-pass filter, I have added queues before the gate, which allow it to run properly. (Note that we never encountered this problem online; the reason is because there are queues that have this effect when running online that were not previously in all the right places for the offline version.)
J. Kissel WP 6368 (Now Closed) The PCALX roaming calibration line has been off since Dec 09 2016 06:41:24 UTC (1165300901, Dec 08 2016 22:41:24 PST) when Rick, Evan, and Travis had last turned it OFF for transient hardware injections (see LHO aLOG 32376). Since Sudarshan now has a satisfactory amount of data at 30 [W], and has processed the data (see LHO aLOG 32512), we will keep this line OFF as planned, until further notice.
WeeklyXtal - PSL_AMP_D3 and D4PWR are tracking humidity trends pretty coherently. D1&2 powers have trended upward slightly.
WeeklyLaser - Over the last 3 or 4 days the LRA position has walked it's way down from the Nominal 4500 to 4496. OSC_BOXHUM has dropped even lower.
WeeklyEnv - Humidities down, temperatures stable.
WeeklyChiller - OSC_PRESS1&2 are trending slightly downward. Headflows are also, very marginally, drifting downward.
WP 6396
Installed new LPC (+) voltage power supply for ion pump #12 at EX. Next Gerardo will install CDS wiring for MEDM read back.
FAMIS#7416
Laser Status:
SysStat is good
Front End Power is 34.36W (should be around 30 W)
Front End Watch is GREEN
HPO Watch is GREEN
PMC:
It has been locked 0.0 days, 0.0 hr 6.0 minutes (should be days/weeks)
Reflected power is 14.17Watts and PowerSum = 75.1Watts.
FSS:
It has been locked for 0.0 days 0.0 hr and 6.0 min (should be days/weeks)
TPD[V] = 2.18V (min 0.9V)
ISS:
The diffracted power is around 3.314% (should be 5-9%)
Last saturation event was 0.0 days 0.0 hours and 6.0 minutes ago (should be days/weeks)
Possible Issues:
PMC reflected power is high
wiki: https://wiki.ligo.org/DetChar/DataQuality/DQShiftLHO20161208 Below I summarise the main highlights of this shift: duty cycle 65.2%, the range is 70 - 75Mpc. on Thursday 8th lockloss due to multiple earthquakes keeping occuring to morning on Friday Software saturation - H1:LSC-X_ARM_CTRL [1] at ~11:00UTC on Thursday 8th 18kHz PI modes strong peak at 11:07UTC and 12:10UTC on Thursday 8th, at ~8:00UTC, ~ 9:00UTC, ~ 10:30UTC and at ~12:30 on Friday 9th. the planned stochastic injection at 00:10UTC Thursday 8th quiet on Saturday 10th and Sunday 11th
The PSL Enclosure phone has been switched off. A reminder that it needs to be re-enabled when work is being done in the Laser Area Enclosure. The power supply for it is in Rm 163.
Please note: The DetChar-noticed ringtone (see https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=32503) forwarded to a second nearby phone, with an audible speakerphone auto response, that must also be disabled. More generally, the VEAs must be re-checked for active telephones.
The "2nd phone" heard in the audio recording from DetChar is the phone in the PSL enclosure anteroom. It is important to note that there is no call forwarding involved, both phones always ring if you call the PSL enclosure extension. The anteroom phone always lags behind the laser room phone though, which is why it appears to be forwarding the call. It sounds so faint because of the acoustic damping of the laser room (these phones are quite loud). Switching off the PSL enclosure phone turns off both the laser room and anteroom phones. This was tested this morning, as we can hear the anteroom phone from outside of the PSL enclosure; when the enclosure extension was called, no ringing was heard. To my knowledge neither of the PSL enclosure phones have the capability to auto-answer, they will continue to ring until physically answered or the caller hangs up.
Tagging a few more subsystems so they're made digitally aware of this fix. This also closes FRS Ticket 6923.
TITLE: 12/13 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 73.1393Mpc
INCOMING OPERATOR: Jeff
SHIFT SUMMARY:
Operator-less for the first 90min, but rest of the shift saw me here watching a nice performing H1 on a quiet Earth.
LOG:
Ops Shift Transition:12/13/2016, Day Shift 16:00 – 00:00 (08:00 - 16:00) - UTC (PT)
Andy, Tom Dent, Josh Nutsinee (alog 32159) and Patrick (alog 32163) both reported a string of glitches during their shifts on Dec 4th. These showed up again in connection with an investigation into some PyCBC loudest events. At Tom's suggestion, we listened to the PSL microphone, which is one of many PEM channels seeing a huge disturbance at these times. I've attached the wav file of the PSL center mic. It seems like the PSL intercom (or some other phone) rings twice and afterwards forwards the call somewhere else. You can hear a faint third ring, and Josh has another file where talking can be heard after the faint ring. When this happened at LLO during an ER, (L1 alog 8451 the intercom would answer and then make a busy signal. It's not clear why we haven't seen this more often, or when the intercom could have gotten connected. This is in the middle of a long stretch of science mode. Edit: To be clear, the loud rings make something very strong in DARM. We're investigating to what extent the quieter stuff shows up.
Here's a spectrogram of the PSL Mic annotated. The second set of more distant rings followed by auto answer starts around second 19. Also attached is the STRAIN spectrogram. The coupling to STRAIN is clear for the PSL rings and it looks like there is faintly also the second set of rings seen. Also attached is the longer Wav file with the speakerphone and talking for the first call and the rings of the second call a minute later.
Opend FRS Ticket 6923
Both PSL Phones have been disabled, see LHO aLOG 32508. FRS Ticket 6923 closed. Tagging other relevant subsystems to envoke L-Mail.
Noticed L1 dropping out of lock at around 13:42 & around this time noticed elevated noise on our EQ BLRMS signal (also notice it on LLO EQ band screenshots). Nothing observed on Terramon or USGS website (and other websites). I did notice a 4.4 in Baja California here: http://www.emsc-csem.org/Earthquake/earthquake.php?id=552372
Noticed Terramon finally reported this earthquake. But it must have been 1-2hrs after we saw it on our seismometers.
Since the start of O2, we have had some pesky dropouts of OBSERVING due to SDF channel changes for some ALS channels from a Beckhoff computer at each of our End Stations (sysecatx1plc2 & sysecaty1plc2). Below, I wanted to list each of the occassions, the amount of time we were out of Science mode, channels involved, and their current state.
Should UNMONITOR (3) ALS-Y Channels
Above we can see all of the channels we noticed/caught/UNMONITORED. I would suggest we take the preemptive action of UNMONITORING (if not already done so) the following channels for sysecaty1plc2 next time we are out of OBSERVING:
Done.
Filed an FRS for this (#6932). I CLOSED it, but there is always the possibility that rogue channels which haven't been addressed can drop us out in the future.
Have remained locked at NLN since 21:44 UTC. Stood down for a GRB at 23:20 UTC. It appeared that the injections were not automatically blocked for the GRB, so I set the INJ_TRANS guardian node to INJECT_KILL manually. Requested INJECT_SUCCESS at 00:35 UTC. Damped PI modes 26 and 28. Ran a2l near beginning of lock.
About the GRB, it turns out that the guardian was doing the right thing.
The alert was received at 2016-12-12 23:20:22 UTC, but the actual GRB event was at 2016-12-12 15:38:59, hours before the alert.
Since the alert came after our standard window of one hour, the guardian didn't block the injeciton.
We just got another GRB alert at 07:05 UTC. This time the event time is around the same time as the created time. The INJ_TRANS guardian did not change again. Perhaps this is because LLO is not locked?
I and Patrick looked at the trend of H1:GRD-INJ_TRANS_STATE_N, and actually GRB made the guardian transition to state 30 (EXTTRIG_ALERT_ACTIVE) for an hour (attached).
The GRB event is this one: https://gracedb.ligo.org/events/view/E265603
J. Kissel for S. Karki WP #6368 I've moved the PCALX roaming line, its long duration sweep, from 4001.3 Hz to 4301.3 Hz. See schedule status below. Since it looks like we'll have the duty cycle to just finish the schedule within a week, I'm just going to continue to re-gathering the highest frequency data. As such, Sudarshan has something consistent and straightforward with which to work instead of data with a bunch details or qualifications. Frequency Planned Amplitude Planned Duration Actual Amplitude Start Time Stop Time Achieved Duration (Hz) (ct) (hh:mm) (ct) (UTC) (UTC) (hh:mm) --------------------------------------------------------------------------------------------------------------------------------------------------------- 1001.3 35k 02:00 39322.0 Nov 28 2016 17:20:44 UTC Nov 30 2016 17:16:00 UTC days @ 30 W 1501.3 35k 02:00 39322.0 Nov 30 2016 17:27:00 UTC Nov 30 2016 19:36:00 UTC 02:09 @ 30 W 2001.3 35k 02:00 39322.0 Nov 30 2016 19:36:00 UTC Nov 30 2016 22:07:00 UTC 02:31 @ 30 W 2501.3 35k 05:00 39322.0 Nov 30 2016 22:08:00 UTC Dec 02 2016 20:16:00 UTC days @ 30 W 3001.3 35k 05:00 39322.0 Dec 02 2016 20:17:00 UTC Dec 05 2016 16:58:57 UTC days @ 30 W 3501.3 35k 05:00 39322.0 Dec 05 2016 16:58:57 UTC Dec 06 2016 21:09:56 UTC ~15:00 @ 30 W 4001.3 40k 10:00 39322.0 Dec 06 2016 21:09:56 UTC Dec 07 2016 18:50:09 UTC ~20:00 @ 30 W 4301.3 40k 10:00 39322.0 Dec 07 2016 18:50:09 UTC 4501.3 40k 10:00 39322.0 4801.3 40k 10:00 39222.0 5001.3 40k 10:00 39222.0 Frequency Planned Amplitude Planned Duration Actual Amplitude Start Time Stop Time Achieved Duration (Hz) (ct) (hh:mm) (ct) (UTC) (UTC) (hh:mm) --------------------------------------------------------------------------------------------------------------------------------------------------------- 1001.3 35k 02:00 39322.0 Nov 11 2016 21:37:50 UTC Nov 12 2016 03:28:21 UTC ~several hours @ 25 W 1501.3 35k 02:00 39322.0 Oct 24 2016 15:26:57 UTC Oct 31 2016 15:44:29 UTC ~week @ 25 W 2001.3 35k 02:00 39322.0 Oct 17 2016 21:22:03 UTC Oct 24 2016 15:26:57 UTC several days (at both 50W and 25 W) 2501.3 35k 05:00 39322.0 Oct 12 2016 03:20:41 UTC Oct 17 2016 21:22:03 UTC days @ 50 W 3001.3 35k 05:00 39322.0 Oct 06 2016 18:39:26 UTC Oct 12 2016 03:20:41 UTC days @ 50 W 3501.3 35k 05:00 39322.0 Jul 06 2016 18:56:13 UTC Oct 06 2016 18:39:26 UTC months @ 50 W 4001.3 40k 10:00 39322.0 Nov 12 2016 03:28:21 UTC Nov 16 2016 22:17:29 UTC days @ 30 W (see LHO aLOG 31546 for caveats) 4301.3 40k 10:00 39322.0 Nov 16 2016 22:17:29 UTC Nov 18 2016 17:08:49 UTC days @ 30 W 4501.3 40k 10:00 39322.0 Nov 18 2016 17:08:49 UTC Nov 20 2016 16:54:32 UTC days @ 30 W (see LHO aLOG 31610 for caveats) 4801.3 40k 10:00 39222.0 Nov 20 2016 16:54:32 UTC Nov 22 2016 23:56:06 UTC days @ 30 W 5001.3 40k 10:00 39222.0 Nov 22 2016 23:56:06 UTC Nov 28 2016 17:20:44 UTC days @ 30 W (line was OFF and ON for Hardware INJ)
Note, PCALX was briefly turned OFF during this 4301.3 Hz stretch. See LHO aLOG 32323.
The Pcal to DARM ratio for these individual lines were calculated using SLM Tool and the result is tabulated below. The Cavity pole and the optical gain on the last column are eyeballed measurement from the CalMon Tool . These time dependent corrections havenot been applied to the sensing function analysis yet.
Freq Start Time End Time FFT Real Complex Coh CC Pole (O. Gain)
---------------------------------------------------------------------------------------------------------------------
1001.3 30-Nov-2016 09:16:44 30-Nov-2016 13:47:04 10 0.022149 -0.358814 0.999589 346.05(0.9863)
---------------------------------------------------------------------------------------------------------------------
1501.3 30-Nov-2016 17:28:11 30-Nov-2016 19:36:18 10 -0.017951 -0.109817 0.999943 346.05(0.9863)
---------------------------------------------------------------------------------------------------------------------
2001.3 30-Nov-2016 19:36:53 30-Nov-2016 21:01:35 10 -0.015679 -0.044511 0.999391 346.05(0.9863)
---------------------------------------------------------------------------------------------------------------------
2501.3 02-Dec-2016 07:52:51 02-Dec-2016 20:16:47 30 -0.011385 -0.020594 0.998321 347.69(0.9703)
---------------------------------------------------------------------------------------------------------------------
3001.3 04-Dec-2016 02:27:58 05-Dec-2016 00:00:17 30 -0.007719 -0.010563 0.999057 347.06(0.9614)
---------------------------------------------------------------------------------------------------------------------
3501.3 06-Dec-2016 02:27:26 06-Dec-2016 06:10:11 30 -0.005708 -0.005431 0.988492 350.70(0.9793)
3501.3 06-Dec-2016 10:36:18 06-Dec-2016 15:03:35 30 350.70(0.9793)
---------------------------------------------------------------------------------------------------------------------
4001.3 12-Nov-2016 17:20:18 12-Nov-2016 20:49:17 60 -0.004374 -0.003211 0.996350 350.00(0.9966)
4001.3 12-Nov-2016 09:48:25 12-Nov-2016 16:40:53 60 350.00(0.9966)
---------------------------------------------------------------------------------------------------------------------
4301.3 18-Nov-2016 05:20:00 18-Nov-2016 12:44:49 60 -0.003607 -0.002212 0.985922 345.04(0.9748)
4301.3 17-Nov-2016 14:26:40 17-Nov-2016 17:20:13 60 347.00(0.9730)
4301.3 17-Nov-2016 08:56:40 17-Nov-2016 11:07:35 60 347.00(0.9730)
---------------------------------------------------------------------------------------------------------------------
4501.3 20-Nov-2016 05:00:00 20-Nov-2016 13:24:30 60 -0.003366 -0.002162 0.990728 350.76(0.9985)
4501.3 19-Nov-2016 15:40:00 19-Nov-2016 23:48:16 60 350.21(1.0017)
---------------------------------------------------------------------------------------------------------------------
4801.3 22-Nov-2016 07:43:20 22-Nov-2016 16:01:44 60 -0.002883 -0.001369 0.959987 346.89(0.9890)
4801.3 21-Nov-2016 15:20:00 21-Nov-2016 18:22:20 60 346.71(0.9979)
4801.3 21-Nov-2016 04:36:40 21-Nov-2016 09:28:48 60 346.71(0.9979)
---------------------------------------------------------------------------------------------------------------------
5001.3 25-Nov-2016 07:43:17 25-Nov-2016 14:52:54 60 -0.002634 -0.000904 0.973928 348.94(0.9625)
5001.3 27-Nov-2016 11:44:40 27-Nov-2016 18:04:12 60 345.48(0.9724)
---------------------------------------------------------------------------------------------------------------------
More detailed Analysis.