Summary (highlights) of the DQ Shift : - IFO observed at ~110Mpc with avg of ~73% duty cycle. - There was more than usual BNS range variation since Thursday, range would drop to as low as 100Mpc. Sunday was better. alog 51177 - We had squeezer problem that resulted in a couple of broadband noises/ range drops / locklosses. alog 51053 alog 51099 alog 51158 alog 51190 - Hveto round 1 winner was H1:LSC-REFL_A_RF45_Q_ERR_DQ throughout the week, which vetoed most of the low-frequency glitches - 48Hz line is clearly visible most days (correlated with ASC-OMC and OMC-ASC channels) alog51080 - On tuesday, broadband noise is present at ~12:15hrs UTC, Hveto round 3 winner vetoed most of these glitches (problem in FSS loop or laser? ) alog 51053. - The band in band ratio seems comparatively broader on Tuesday and Thursday for both 4KHz and 100Hz - Elevated microseismic noise thursday through Sunday - PCal X and Y lines seems usual. New deltaL external curve added to the plot. PCal curves had a broader spread on Friday - There was a lockloss on sunday due to power glitch. alog 51197. - Thunderstorm/windy on Sunday, glitchier in general from 18:00hrs - 22:00hrs UTC Detailed DQ shift report can be found in: https://wiki.ligo.org/DetChar/DataQuality/DQShiftLHO20190805
The decay of the squeezer laser seems to have slowed down over the weekend. However, it looks like we are multi-moding again. The transmitted OPO power is at 80% of nominal.
Attached are the VPW desiccant plots with the data from June 2019 until August 2019. DB3 had a low battery in mid July. This wasn't noticed until today and was replaced. DB3 is recording again like normal. However, we are missing about 4 weeks of DB3 plots. All other readouts look normal. Nothing out of the ordinary shown.
TITLE: 08/12 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC STATE of H1: Observing at 112Mpc INCOMING OPERATOR: Jeff SHIFT SUMMARY: Remained locked and in observing entire shift. 'green power stabilization railed' message is back. No other issues.
Have remained locked and in observing. No issues.
TITLE: 08/12 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 116Mpc
OUTGOING OPERATOR: Niko
CURRENT ENVIRONMENT:
Wind: 7mph Gusts, 6mph 5min avg
Primary useism: 0.01 μm/s
Secondary useism: 0.05 μm/s
QUICK SUMMARY: No issues.
Ops Shift Transition: 08/11/2019, Eve Shift 23:00 – 07:00 (16:00-00:00) - UTC (PT)
State of H1: Locked
Intent Bit: Observing
Weather: 0-20mph wind
Primary 0.03 – 0.1Hz: 0.01 um/s
Secondary 0.1 – 0.3Hz: 0.1 um/s
Outgoing Operator: Jeff
Quick Summary: Range is more stable than it was in the previous lock. Locked and Observing 15 hours.
All good so far. The range is sitting around 112 to 115Mpc. The wind has come up rather sharply, now blowing 15 to 25mph. End-X and End-Y microseism is elevated as well.
TITLE: 08/11 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC STATE of H1: Observing at 116Mpc INCOMING OPERATOR: Jeff SHIFT SUMMARY: Knocked out of observing once by TCS_ITMY_CO2 node. Lightning storm continued. No other issues. LOG: 08:16 - 08:18 UTC Knocked out of observing by TCS_ITMY_CO2 node
Have remained locked and in observing. No issues.
TITLE: 08/11 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 114Mpc
OUTGOING OPERATOR: Niko
CURRENT ENVIRONMENT:
Wind: 6mph Gusts, 5mph 5min avg
Primary useism: 0.01 μm/s
Secondary useism: 0.05 μm/s
QUICK SUMMARY: Just relocked. Still seeing lightning flashes on the camera.
07:25 UTC Accepted attached SDF difference for H1:OMC-ASC_MASTERGAIN. Why is this different?
TITLE: 08/10 Eve Shift 23:00 – 07:00 (16:00-00:00), all times posted in UTC
STATE of H1: Locking
INCOMING OPERATOR: Patrick
SHIFT SUMMARY: Quiet shift until power glitch knocked us out of lock. Been having troubles around ENGAGE_ASC_FOR_FULL_IFO.
LOG:
03:30 (20:30) Lockloss, possibly from power glitch
03:39 (20:39) Starting an initial alignment
04:12 (21:12) Initial alignment complete, attempting to lock
05:15 (22:15) Lost lock from ENGAGE_ASC_FOR_FULL_IFO. Diag main log says OMC DCPD whitening not equal to anti-whitening and ESD X Driver OFF
05:36 (22:36) Lost lock from ENGAGE_ASC_FOR_FULL_IFO again, same issues. Starting initial alignment.
06:21 (23:21) Patrick to CER mezzanine -- check HV power supplies are on
06:26 (23:26) Initial alignment complete, attempting relock
The rocker switches on all of the HV power supplies were on.
While we were working on engaging ASC (see midday mini update alog 50947), we lost lock quite suddenly. I had my locking screen open, and saw that the LSC-DARM gains had been set to zero, and the yellow ramping triangles were active. I didn't really have time to figure out what to do, and then we lost lock suddenly (which makes sense if we stopped controlling the DARM length).
In the attached ndscope screenshot you can see that the POP DC has good buildups, then the DARM1 and DARM2 gains are set to zero (this EPICS channel doesn't capture the fact that they were ramping down, just that the setpoint went to zero). After that, we lost lock according to the buildups, and then ISC_LOCK noticed and started the DOWN sequence.
This is a problem. If this is the only time it's ever happened, fine. But, we should look to see if this is what has been causing our mysterious locklosses the past few weeks.
Several locklosses are NOT this problem, so maybe it's not systemic, and was some weird fluke this one last lockloss.
While investigating this lockloss, I noticed that there was a glitch in H1:LSC-DARM_OUT_DQ at very close to the same time as the DARM gains go to zero (see attached png). The last relevant ISC_LOCK log entries around this time were "Unstalling ALS_DIFF" and "Unstalling ALS_YARM", which both happened about a second before the glitch. These occurred in the run state of PREP_ASC_FOR_FULL_IFO, which doesn't do anything except check if the nodes are stalled. My guess is that the 'node.revive' function (in NodeManager) for ALS is what changed the DARM gains to zero.
ALS is usually shuttered at ENGAGE_ASC_FOR_FULL_IFO but Sheila suggested that, since we were skipping SHUTTER_ALS to reposition the optics around this time, we could have inadvertently run some code that is not supposed to happen in that ISC_LOCK state.