TITLE: 09/03 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 115Mpc
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
SEI_CONF state: WINDY
Wind: 8mph Gusts, 6mph 5min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.23 μm/s
QUICK SUMMARY:
TITLE: 09/03 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 119Mpc
INCOMING OPERATOR: Ed
SHIFT SUMMARY: A relatively quick recovery from maintenance, but then a bit of commissioning and odd couple of lock losses slowed us down from getting to Observe.
LOG:
M. Ball, S. Dwyer
Before lock was lost during maintenance this morning, we turned on our L2A rerouting as originally tested in LHO alog 51604. This time, we activated a boost (below 0.2Hz gain in the L1 L2L filter) and still maintained lock. With the rerouted L2A signals, we saw an increase in the actuator control signals between 3 and 5 Hz. With the boost on, we saw a decrease in these actuator signals below 0.3Hz.
We then tested a "new" (modified) L2 length to L3 pitch decoupling filter that included a cutoff above 2Hz to reduce the 3-5Hz signal increase. Little else was changed by this filter.
Attached are time series of the L1, L2, and L3 actuator control signals at the time the boost was activated. The L1 signal is largely unchanged, but some spikes appear larger. The L2 signal shows a net decrease in amplitude (this may help with locklosses triggered by saturation of these signals, such as this one from 9/3). The L3 signal remains largely unchanged.
Also attached are spectra of one of these signals before turning on the L2A rerouting and boost (dotted lines), between turning on the boost and loading the new filter (dashed lines), and after loading this filter (solid lines). Timestamps are as follows:
Rerouting turned on at 15:37:43 UTC.
Boost activated at 15:38:28 UTC.
L2P filter loaded at 15:51:30 UTC.
This has not been implemented yet, but we plan to do calibration measurements for it tomorrow.
Here's a plot of the design of the "muBoost" filter. Lives in FM2 of the H1 SUS ETMX L1 LOCK L bank.
Keita, Jeff K, Sheila, TJ
Some of the locklosses that Betsy points out here: 51706 (especially the ones where there is a glitch in DARM that causes the ADS signals to move the IFO) look similar to some of the locklosses over the weekend durring engage soft loops and increase power (eg 51677) In these locklosses the SRC1+SRC2 error signals aren't held to 0, and neither is INP1, and the PRC2 offloading which is slow is very behind. Yesterday I moved top mass integrators in SRM and SR2 so that we would have the option of adding gain upstream of the intergrators, which was implemented when Ed reloaded the ISC_DRMI guardian last night.
This morning we increased the gain of the top mass offloading for SR2 +SRM by 10 dB. (I caused one lockloss with a mistake while doing this.) We also moved the integrator for PR2, and made the appropriate changes in ALIGN_IFO so that we could add 20dB more offloading gain to the PRC2 P+Y loops. These changes are all in the ISC_DRMI guardian, and we have locked twice with them implemented as is. This seems to do a better job of offloading, and has spend up the DRMI ASC. It looks like we would benefit from increasing the SRC offloading gains by another 10dB, and from increasing the overall loop gain of INP1.
We lost lock once today because of a mistake I made, a few other times with symptoms simliar to the INCREASE power and soft loop locklosses over th weekend (before implementing these changes and with them partially implemented), and our last lockloss happened while the soft loops were steering the beam around the point absorpber, as we passed through a point where all the build ups get low. We decided to try in the next lock not to steer the beam around, but manually let the ADS converge with the spots centered, then moved directly to the final position. This worked and didn't bring the buildups as low as they were in the previous lock (see 1st screenshot, time cursors are at times when the spot position was moved, the first one was going around the test mass, the second was going directly to the final position). We've implemented this in the guardian, which should help to speed up the long wait for ADS convergence, and hopefully avoid some locklosses while the spots are moving around to places where build ups are bad.
Maintenance day recovery started around 1830UTC. Here's the main points:
Isolated cable trays from supports on the following SUS floor racks: SUS-R1 (Next to HAM2), SUS-R2 (Next to HAM3), SUS-R3 (Next to HAM4), and SUS-R4 (Next to HAM5).
Filiberto Clara
I've re-worked the startup scripts for the control room right wall FOM machines (nuc20-23) and the seismic BLRMS (nuc5). All these FOM displays start automatically. If there are any issues with these startups, please contact me.
The Chromium web brower, dmtviewer and diaggui required additional code to automatically press tabs and buttons to transition them into an operational state.
Scripts are documented in this wiki page
Checked on the replacement pump at EndX, (which was swapped in Thursday, see alog 51617,) this morning and it seems to be running fine with quieter operation and no leaks. Performance is also good with no change in output demand since startup Thursday.
[J. Betzwieser, T. Mistry, M. Wade, A. Viets]
As per LHO alog 51654, we have decided to adjust the configuration for the GDS calibration pipeline to *not* correct for the complex actuation time-dependent correction factors. Instead, we will only correct for the real part of the actuation time-dependent correction factors. I have made this configuration change in the GDS calibration pipeline. The new configuration file is in the calibration SVN
aligocalibration/trunk/Runs/O3/GDSFilters/H1GDS_1251572901.ini
The primary and redundant calibration pipelines were restarted with these configurations around GPS time 1251572901.
Attached below are the results for the monthly charge measurements performed on the ETMX and ETMY. Due to time constraints we were only able to perform 3 measurements (against 5).
ETMX: The effective bias voltage has reached 40V (linear rise over time), for the pitch mode of the 1st quadrant . In the other 3 quadrants the bias voltage is 30V or below. For the Yaw mode, the bias voltage is around 40-45V for the 1st and 2nd quadrant, however it drops down to below 30V for the 3rd and 4th quadrant.
ETMY: For the pitch mode the effective bias is within limit and lower than 40V (zero for the 2st and 3rd quadrant). For the Yaw mode, the 2nd quadrant is still sitting at 50V, for the other 2 quadrants it is below 40V.
SDF overview: The lock bias offset values for the ETMY (flipped sign) and ETMY (changed from zero to 10.0. however this is switched off) changed, please see the ndscope plot attached. The values were later restored (based on the past trends) and any other SDF differences removed.
I spent the morning mining the seemingly random locklosses from the weekend. Then Sheila pointed me towards the ndscope useful channels list that Jenne made - indeed the dither system (ADS) wanders off in many cases.
I don't know what they mean, but the lockloss tool shows the following:
39: 2019-09-01_12:13:59Z ISC_LOCK NOMINAL_LOW_NOISE -> LOCKLOSS
H1:ASC-CHARD_Y_OUT_DQ starts to really wander off t-2 sec before lockloss, SRC channels wander at -2 sec.
H1:OAF-RANGE_RPL chans all WANDER at -7 sec.
38: 2019-09-01_16:15:40Z ISC_LOCK NOMINAL_LOW_NOISE -> LOCKLOSS
H1:ASC-CHARD_Y_OUT_DQ starts to really wander off t-10 sec before lockloss, SRC channels wander at -1- sec.
No DARM glitch in the OAF RANGE RPL chans.
35: 2019-09-01_20:19:04Z ISC_LOCK NOMINAL_LOW_NOISE -> LOCKLOSS
~WINDY - ends at ~16-20MPH
H1:ASC-CHARD_Y_OUT_DQ starts to really wander off t-3 sec before lockloss, SRC channels wander at -2 sec.
ADS PIT and YAW DOFs and DEMOD chans WANDER 10 sec before LL.
H1:OAF-RANGE_RPL chans all WANDER at -5 sec.
30: 2019-09-02_03:19:04Z ISC_LOCK NOMINAL_LOW_NOISE -> LOCKLOSS
H1:ASC-CSOFT_Y_OUT_DQ starts to really wander off t-6 sec before lockloss, SRC channels wander at -5 sec.
ADS PIT and YAW DOFs WANDER 8 sec before LL.
H1:OAF-RANGE_RPL chans all WANDER at -7 sec.
24: 2019-09-02_13:10:16Z ISC_LOCK NOMINAL_LOW_NOISE -> LOCKLOSS
H1:ASC-CHARD_Y_OUT_DQ starts to really wander off t-5 sec before lockloss, SRC channels wander at -4 sec.
ADS PIT and YAW DOFs WANDER 8 sec before LL.
H1:OAF-RANGE_RPL chans all WANDER at -9 sec.
19: 2019-09-03_01:52:57Z ISC_LOCK NOMINAL_LOW_NOISE -> LOCKLOSS
H1:ASC-CSOFT_Y_OUT_DQ starts to really wander off t-8 sec before lockloss, SRC channels wander at -10 sec.
ADS PIT and YAW DOFs WANDER 12 sec before LL. H1:OAF-RANGE_RPL chans all WANDER at -12 sec.
13: 2019-09-03_10:17:41Z ISC_LOCK NOMINAL_LOW_NOISE -> LOCKLOSS
H1:SUS-ETMX_L3_MASTER_OUT - glitch 1 sec before LL, but other glitches of same size/shape earlier not an issue... SRC/SOFT/HARD loops loop nominal in seconds prior to LL.
ADS NOMINAL before LL.
No DARM glitch in the OAF RANGE RPL chans.
________________________
on command line:
userapps
lockloss -c ADS_3_4_5.yml select (to show ADS)
lockloss -c darm_blrms.yaml select (to show glitch info on DARM, not sure what math is happening but large wander from -10sec on H1:OAF-RANGE_RPL_1_OUTPUT and all other similar chans, seems there was a glitch in DARM which maybe saturated the Dither line in DARM so that ADS couldn't work correctly... or something like that in commissioning person words...)
So, is it the old story that a glitch causes havoc in with the ADS - and does it seem worse this weekend? Dunno...
Here are some plots of the locklosses caused by a DARM glitch sending a large signal to ADS, and the rest of the ASC loops not holding their error signals as the ADS moves the IFO around. This is the lockloss labeled 19 in Betsy's log, 1251510796
This seems like something that we could add a tag for in the lockloss tool, for example if we are in low noise and the ADS error signals go above some threshold we could tag it as an ADS glitch lockloss.
Today Keita and I increased the low frequency gains in PR2, SRC1+SRC2 P+Y loops, hopefully this will help the ASC signals stay closer to zero in future events like this 51715. It seems as though we may also need to increase the INP1 gain.
We could also avoid having these glitches impact the ADS by moving the dither lines to lower frequencies where they wouldn't be drowned out by the glitches. That caused excess noise in DARM when Georgia tried it, but Matthew Ball has been doing some modeling that suggests that a frequency dependent A2L decoupling might help allow us to move them down in frequency.
The last and probably best solution would be to get rid of the glitches, of course.
Bi-Monthly checked the dust monitor vacuum pumps. Both end stations were fine. Vacuum pressures were correct and the temperatures were between 100 and 130f, Found the CS vacuum pump shut down. Recycled the power switch and the pump restarted. There was a good blast of carbon dust out of the exhaust. This is the second time this pump has shutdown. Will continue to monitor the pump throughout the day. Plan is to take it off line and rebuild the pump section. I have the repair kit in hand. Closing FAMIS #12995
Referring to PeterK's log, 51658, while the Supermoon will change the Earth tides, these effects will be a few percent not a factor of four. Nor will these normal tidal forces produce a discontinuity as seen in JeffBs' plots in logs 51648 and 51650, and herein. Even if the IFO were to remain locked for an entire year allowing the tertiary affects (seasonal) to 'drift' the DC level of the tides, the total range on the system would be less than +- 400u and certainly not 700u in 30 hours. This world would be in a world of hurt if that were the case. This extreme excursion of demand is causally related to the PSL incursion on Thursday. And given that this caused a lock-loss, warrants an FRS. Possibly the operation of or procedure of running the AC in the PSL area needs adjusting. Alternatively, the locking of the RefCav with a 10-20% step in power may be the culprit.
I suspect that the temperature excursion of the PSL room while not getting to the PSL RefCav temperature sensor, the laser table is warmed up and this slowly flows into the reference cavity via its legs. Alternatively, I see that the PSL-FSS_TPD_DC output jumps 10+% up after the Thursday activity. I see the RFPD of the FSS also steps up so something upstream increased laser power. Could this increase of power just be warming and increasing the RefCav length--seems plausible. The attached plot shows the step and very slow trend back down of the FSS Slow loop (Ch3) which is putting a wavelength change on the 1u light.
So, I see the room temperature change and the increase RefCav power warming and increasing the length of the RefCav evidenced in the slow loop control. Could this be mitigated to reduce risk of lockloss? Maybe. Change AC operation, monitor & shunt 'excess' power going to RefCav to keep it heat load the same.
Attached plot shows 8 days of LASER room temp, 'tidal' drive to EX HEPI (EY similar,) FSS_NPRO_TEMP, and the FSS_TPD_DC.
FRS Ticket 13516.
Ed called about difficulty locking, since I don't see anything obvious I looked at BRS channels I foudn trended here: 51338. It looks like the low frequency BLRMS of BRS X has been growing exponentially.
Here's th BRS HEALTH ndscope image during ENGAGE_SOFT_LOOPS
I don't see anything problematic really in the time series. But I did notice from the Detchar summary pages that there was some weird behavior with the inside temperature for a few hours from 16 UTC yesterday (02/09/2019). It takes hours until the BRS itself responds to temperature changes so I'm just guessing but you might be seeing the effects of that. You might consider turning the sensor correction to NON BRS state and lock since the wind is low now anyways. Here is the plot of the temperature
?
I don't think there is anything wrong with BRSX. First attached trend are 7 weeks of the both BRS driftmons, which are kind of a measure of the DC position. BRSX is close to the level where I would want to recenter, but I hope to push that off until we try to install Eyal and Arnaud's heating pad. I don't think the "strangeness" Eyal notes is anything to be worried about. It's a tenth of a degree, from 8am local to about 4pm, probably the aircon.
Second plot are 4 days of trends for ISC_LOCK (top left), the sensor correction control signal generated from the tilt subtracted STS (bottom left) and the drift (top right) and velocity of BRSX (bottom right). I don't see any signals in the right 2 plots that would cause the spikes in the sensor correction (so I'm assuming they are earthquakes, but I'll keep looking), or would be any cause for concern.
I don't know why the lowest frequency BLRMS of the BRS seem to accumulate these large numbers but I don't think it's a problem. Or if it is, both BRS are busted. Third image is the BLRMS for both BRS for the last 2 weeks. Maybe this is some sort of computational weirdness? Some integrator in the BLRMS filter? Maybe we should periodically clear the histories of some of the filters.
I've looked at few non-brs blrms channels and all of the dc-30mhz blrms channels show this same behavior. The 30mhz low pass is just integrating non-sense. There is no easy way to clear the history of the filter that does the calculation, without restarting the model. Dave and I are talking about adding an epic variable to zero out the filter. We will try prototyping on the test stand.
can you inject in parallel a negative large DC to try and zero it out?
The 30mhz BLRMS filter shouldn't be integrating like this. Filed FRS https://services.ligo-la.caltech.edu/FRS/show_bug.cgi?id=13602
Pursuant to LIGO-T1900555 [1] and discussions on the DetChar and HW injections mailing list [2,3], I've scheduled DetChar safety injections to begin at 23:00:00 UTC (18:00:00 CDT, 16:00:00 PDT) 3 Sep 2019. These should last for 390 seconds and contain 75 injections spaced 5 sec apart. The scheduled injections have been added to GraceDb [4]. We will update the events in GraceDb to record whether these injections were successful post-hoc. **Please note**, we have schedule coincident (but not necessarily coherent) injections at LLO for the same time. We include the LLO aLOG describing those injections in a comment below. More details on how these injections were generated and added to the HW injection SVN repo are available via a README [5]. [1] https://dcc.ligo.org/LIGO-T1900555 [2] detchar@ligo.org [3] hw-injections@ligo.org [4] https://gracedb.ligo.org/search/?query=Test+INJ+gpstime%3A+1251586818+..+1251587400+instruments%3A+%22H1%22&query_type=E&results_format=S [5] https://ldas-jobs.ligo.caltech.edu/~detchar/hwinj/LIGO-T1900555/README
The coincident (but not necessarily coherent) injections at LLO are described here:
https://alog.ligo-la.caltech.edu/aLOG/index.php?callRep=48279
I svn uploaded the waveforms and schedule file for this injection, and reloaded the guardian, so if the IFO is observing then the injections should go in.
This injection failed (see alog 51720) with what looks like the same INJ TRANS guardian error we have been seeing at LLO (see llo alog 46132 and FRS ticket 12939)
GraceDb entries associated with these injections have been updated to reflect the fact that they did not go in correctly (labeled HWINJNO)
Pursuant to LIGO-T1900555 [1] and discussions on the DetChar and HW injections mailing list [2,3], I've scheduled DetChar safety injections to begin at 21:00:00 UTC (16:00:00 CDT, 14:00:00 PDT) 3 Sep 2019. These should last for 390 seconds and contain 75 injections spaced 5 sec apart. The scheduled injections have been added to GraceDb [4]. We will update the events in GraceDb to record whether these injections were successful post-hoc. **Please note**, we have schedule coincident (but not necessarily coherent) injections at LLO for the same time. We include the LLO aLOG describing those injections in a comment below. More details on how these injections were generated and added to the HW injection SVN repo are available via a README [5]. [1] https://dcc.ligo.org/LIGO-T1900555 [2] detchar@ligo.org [3] hw-injections@ligo.org [4] https://gracedb.ligo.org/search/?query=Test+INJ+gpstime%3A+1251579618+..+1251580100+instruments%3A+%22H1%22&query_type=E&results_format=S [5] https://ldas-jobs.ligo.caltech.edu/~detchar/hwinj/LIGO-T1900555/README
The coincident (but not necessarily coherent) injections at LLO are described here: https://alog.ligo-la.caltech.edu/aLOG/index.php?callRep=48277
Neither IFO was up by this time, so I removed this injection from the schedule file.
Injection failed. Guardian log below:
2019-09-03_22:37:31.316944Z INJ_TRANS RELOAD requested. reloading system data...
2019-09-03_22:37:31.360492Z INJ_TRANS module path: /opt/rtcds/userapps/release/cal/common/guardian/INJ_TRANS.py
2019-09-03_22:37:31.361022Z INJ_TRANS user code: /opt/rtcds/userapps/release/cal/common/guardian/injtools/__init__.py
2019-09-03_22:37:31.361022Z INJ_TRANS user code: /opt/rtcds/userapps/release/cal/common/guardian/injtools/inj_det.py
2019-09-03_22:37:31.361022Z INJ_TRANS user code: /opt/rtcds/userapps/release/cal/common/guardian/injtools/inj_io.py
2019-09-03_22:37:31.361022Z INJ_TRANS user code: /opt/rtcds/userapps/release/cal/common/guardian/injtools/inj_types.py
2019-09-03_22:37:35.800454Z INJ_TRANS RELOAD complete
2019-09-03_22:37:35.801803Z INJ_TRANS W: RELOADING @ WAIT_FOR_NEXT_INJECT.run
2019-09-03_22:55:00.027528Z INJ_TRANS [WAIT_FOR_NEXT_INJECT.run] ezca: H1:CAL-INJ_TINJ_START => 1251586518.03
2019-09-03_22:55:00.028168Z INJ_TRANS [WAIT_FOR_NEXT_INJECT.run] ezca: H1:CAL-INJ_TINJ_OUTCOME => 0
2019-09-03_22:55:00.148272Z INJ_TRANS EDGE: WAIT_FOR_NEXT_INJECT->CHECK_SCHEDULE_TIMES
2019-09-03_22:55:00.148896Z INJ_TRANS calculating path: CHECK_SCHEDULE_TIMES->INJECT_SUCCESS
2019-09-03_22:55:00.149298Z INJ_TRANS new target: CREATE_AWG_STREAM
2019-09-03_22:55:00.150149Z INJ_TRANS executing state: CHECK_SCHEDULE_TIMES (35)
2019-09-03_22:55:00.151587Z INJ_TRANS [CHECK_SCHEDULE_TIMES.main] USERMSG 0: INJECTION IMMINENT: 1251586818.000000
2019-09-03_22:55:00.152104Z INJ_TRANS [CHECK_SCHEDULE_TIMES.main] Skipping checking schedule times since its already been done.
2019-09-03_22:55:00.265266Z INJ_TRANS EDGE: CHECK_SCHEDULE_TIMES->CREATE_AWG_STREAM
2019-09-03_22:55:00.271061Z INJ_TRANS calculating path: CREATE_AWG_STREAM->INJECT_SUCCESS
2019-09-03_22:55:00.273132Z INJ_TRANS new target: READ_WAVEFORM
2019-09-03_22:55:00.274979Z INJ_TRANS executing state: CREATE_AWG_STREAM (50)
2019-09-03_22:55:00.275920Z INJ_TRANS [CREATE_AWG_STREAM.enter]
2019-09-03_22:55:00.277244Z INJ_TRANS [CREATE_AWG_STREAM.main] ezca: H1:CAL-INJ_TINJ_TYPE => 3
2019-09-03_22:55:00.391173Z INJ_TRANS EDGE: CREATE_AWG_STREAM->READ_WAVEFORM
2019-09-03_22:55:00.391774Z INJ_TRANS calculating path: READ_WAVEFORM->INJECT_SUCCESS
2019-09-03_22:55:00.391774Z INJ_TRANS new target: RAMP_GAIN_TO_1
2019-09-03_22:55:00.392868Z INJ_TRANS executing state: READ_WAVEFORM (60)
2019-09-03_22:55:00.395535Z INJ_TRANS [READ_WAVEFORM.main] Reading waveform data from /hwinj/Details/detchar/detchar-hwinj-schedule_LIGO-T1900555-PROPOSED_H1_INJECTIONS-1251586818-390.txt
2019-09-03_22:55:27.569259Z INJ_TRANS EDGE: READ_WAVEFORM->RAMP_GAIN_TO_1
2019-09-03_22:55:27.570014Z INJ_TRANS calculating path: RAMP_GAIN_TO_1->INJECT_SUCCESS
2019-09-03_22:55:27.570014Z INJ_TRANS new target: AWG_STREAM_OPEN_PREINJECT
2019-09-03_22:55:27.570974Z INJ_TRANS executing state: RAMP_GAIN_TO_1 (65)
2019-09-03_22:55:27.581834Z INJ_TRANS [RAMP_GAIN_TO_1.enter]
2019-09-03_22:55:27.583248Z INJ_TRANS [RAMP_GAIN_TO_1.main] ezca: H1:CAL-INJ_TRANSIENT_GAIN => 1.0
2019-09-03_22:55:29.829405Z INJ_TRANS EDGE: RAMP_GAIN_TO_1->AWG_STREAM_OPEN_PREINJECT
2019-09-03_22:55:29.830281Z INJ_TRANS calculating path: AWG_STREAM_OPEN_PREINJECT->INJECT_SUCCESS
2019-09-03_22:55:29.831068Z INJ_TRANS new target: INJECT_CBC_ACTIVE
2019-09-03_22:55:29.832679Z INJ_TRANS executing state: AWG_STREAM_OPEN_PREINJECT (70)
2019-09-03_22:55:29.838744Z INJ_TRANS [AWG_STREAM_OPEN_PREINJECT.main] USERMSG 0: INJECTION IMMINENT: 1251586818.000000
2019-09-03_22:59:40.085443Z INJ_TRANS JUMP target: INJECT_DETCHAR_ACTIVE
2019-09-03_22:59:40.086053Z INJ_TRANS [AWG_STREAM_OPEN_PREINJECT.exit]
2019-09-03_22:59:40.150495Z INJ_TRANS JUMP: AWG_STREAM_OPEN_PREINJECT->INJECT_DETCHAR_ACTIVE
2019-09-03_22:59:40.151051Z INJ_TRANS calculating path: INJECT_DETCHAR_ACTIVE->INJECT_SUCCESS
2019-09-03_22:59:40.151051Z INJ_TRANS new target: RAMP_GAIN_TO_0
2019-09-03_22:59:40.152161Z INJ_TRANS executing state: INJECT_DETCHAR_ACTIVE (104)
2019-09-03_22:59:40.153871Z INJ_TRANS [INJECT_DETCHAR_ACTIVE.main] USERMSG 0: INJECTION ACTIVE: 1251586818.000000
2019-09-03_22:59:40.154308Z awgSetChannel: awg_clnt[42][0] = NULL
2019-09-03_22:59:40.154308Z Error code from awgSetChannel: -5
2019-09-03_22:59:40.170270Z INJ_TRANS [INJECT_DETCHAR_ACTIVE.main] File "/opt/rtcds/userapps/release/cal/common/guardian/INJ_TRANS.py", line 572, in main
2019-09-03_22:59:40.170270Z self.hwinj.stream.send(self.hwinj.data)
2019-09-03_22:59:40.170928Z INJ_TRANS [INJECT_DETCHAR_ACTIVE.main] File "/usr/lib/python2.7/dist-packages/awg.py", line 621, in send
2019-09-03_22:59:40.170928Z self.append(data, scale=scale)
2019-09-03_22:59:40.170928Z INJ_TRANS [INJECT_DETCHAR_ACTIVE.main] File "/usr/lib/python2.7/dist-packages/awg.py", line 599, in append
2019-09-03_22:59:40.170928Z self.open()
2019-09-03_22:59:40.170928Z INJ_TRANS [INJECT_DETCHAR_ACTIVE.main] File "/usr/lib/python2.7/dist-packages/awg.py", line 584, in open
2019-09-03_22:59:40.170928Z + ": " + awgbase.SIStrErrorMsg(ret))
2019-09-03_22:59:40.170928Z INJ_TRANS [INJECT_DETCHAR_ACTIVE.main] <class 'awg.AWGStreamError'> can't open stream to H1:CAL-INJ_TRANSIENT_EXC: Error setting up an awg slot for the channel
2019-09-03_22:59:40.197166Z INJ_TRANS JUMP target: FAILURE_DURING_ACTIVE_INJECT
2019-09-03_22:59:40.197823Z INJ_TRANS [INJECT_DETCHAR_ACTIVE.exit]
2019-09-03_22:59:40.261389Z INJ_TRANS JUMP: INJECT_DETCHAR_ACTIVE->FAILURE_DURING_ACTIVE_INJECT
2019-09-03_22:59:40.262094Z INJ_TRANS calculating path: FAILURE_DURING_ACTIVE_INJECT->INJECT_SUCCESS
2019-09-03_22:59:40.262094Z INJ_TRANS new target: WAIT_FOR_NEXT_INJECT
2019-09-03_22:59:40.270095Z INJ_TRANS executing state: FAILURE_DURING_ACTIVE_INJECT (300)
2019-09-03_22:59:40.272312Z INJ_TRANS [FAILURE_DURING_ACTIVE_INJECT.enter]
2019-09-03_22:59:40.273597Z INJ_TRANS [FAILURE_DURING_ACTIVE_INJECT.main] ezca: H1:CAL-INJ_TRANSIENT_GAIN => 0.0
2019-09-03_22:59:42.405392Z INJ_TRANS [FAILURE_DURING_ACTIVE_INJECT.main] ezca: H1:CAL-INJ_TINJ_OUTCOME => -4
2019-09-03_22:59:42.405950Z INJ_TRANS [FAILURE_DURING_ACTIVE_INJECT.main] ezca: H1:CAL-INJ_TINJ_ENDED => 1251586800.41
2019-09-03_22:59:42.451041Z INJ_TRANS [FAILURE_DURING_ACTIVE_INJECT.run] USERMSG 0: ERROR
Was actually the 2300UTC injection from this alog (alog 51684) that failed. The 2100 UTC injection was removed from the schedule.
GraceDb entries associated with these injections have been updated to reflect the fact that they were not made successfully (labeled HWINJNO)