Referring to PeterK's log, 51658, while the Supermoon will change the Earth tides, these effects will be a few percent not a factor of four. Nor will these normal tidal forces produce a discontinuity as seen in JeffBs' plots in logs 51648 and 51650, and herein. Even if the IFO were to remain locked for an entire year allowing the tertiary affects (seasonal) to 'drift' the DC level of the tides, the total range on the system would be less than +- 400u and certainly not 700u in 30 hours. This world would be in a world of hurt if that were the case. This extreme excursion of demand is causally related to the PSL incursion on Thursday. And given that this caused a lock-loss, warrants an FRS. Possibly the operation of or procedure of running the AC in the PSL area needs adjusting. Alternatively, the locking of the RefCav with a 10-20% step in power may be the culprit.
I suspect that the temperature excursion of the PSL room while not getting to the PSL RefCav temperature sensor, the laser table is warmed up and this slowly flows into the reference cavity via its legs. Alternatively, I see that the PSL-FSS_TPD_DC output jumps 10+% up after the Thursday activity. I see the RFPD of the FSS also steps up so something upstream increased laser power. Could this increase of power just be warming and increasing the RefCav length--seems plausible. The attached plot shows the step and very slow trend back down of the FSS Slow loop (Ch3) which is putting a wavelength change on the 1u light.
So, I see the room temperature change and the increase RefCav power warming and increasing the length of the RefCav evidenced in the slow loop control. Could this be mitigated to reduce risk of lockloss? Maybe. Change AC operation, monitor & shunt 'excess' power going to RefCav to keep it heat load the same.
Attached plot shows 8 days of LASER room temp, 'tidal' drive to EX HEPI (EY similar,) FSS_NPRO_TEMP, and the FSS_TPD_DC.
FRS Ticket 13516.
I took some photos of the ITMY and X in the hope that the test mass earthquake stops would be visible, and usable as a reference to restore the cameras to their original positions should they get bumped again. These photos were taken when we were out of lock, and with the illuminators on, and using the green camera. The earthquake stops are not really visible, and the prospect of using these as references to return the cameras to their optimal positions is not promising.
Our connection was moved between switch ports at our primary ISP this morning at 9:00AM Pacific to allow maintenance of their equipment. This closes WP# 8336.
I reset both PSL power watchdogs at 15:47 UTC (8:47 PDT). This completes FAMIS 10726.
TITLE: 09/03 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Preventive Maintenance
OUTGOING OPERATOR: Travis
CURRENT ENVIRONMENT:
SEI_CONF state: WINDY_NO_BRSX
Wind: 3mph Gusts, 1mph 5min avg
Primary useism: 0.08 μm/s
Secondary useism: 0.21 μm/s
QUICK SUMMARY: Maintenance activities have started, but we holding lock for a bit. useism has leveled out.
TITLE: 09/03 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Preventive Maintenance
INCOMING OPERATOR: TJ
SHIFT SUMMARY: Rough shift for locking, much like the past several days.
LOG:
Ran Initial Alignment. After IA completed, would either lose lock at FIND_IR or it wouldn't be able to find COMM IR. I tried adjusting the COMM offset like usual, but could never find resonance. After a few attempts at this, I re-did IA thinking that perhaps it didn't finish correctly although I didn't notice anything out of the ordinary during the sequence. After second IA, IR locked and moved on.
After several attempts at locking and struggling to get past DRMI, H1 went all the way to NLN in a single attempt.
Looking at the last 3 relocking periods that are shown on the range plot on the wall, they are almost identically ~4 hours long. Interesting, but not necessarily in a good way.
Started script at 14:45 UTC.
After a long relock period, we are finally back to Observing.
No obvious cause.
TITLE: 09/03 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 114Mpc
OUTGOING OPERATOR: Ed
CURRENT ENVIRONMENT:
SEI_CONF state: WINDY_NO_BRSX
Wind: 6mph Gusts, 4mph 5min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.21 μm/s
QUICK SUMMARY: Observing for 1.75 hours. Microseism is climbing quickly, but otherwise environment is calm so hopefully no need to turn BRSx back on.
TITLE: 09/03 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 115Mpc
INCOMING OPERATOR: Travis
SHIFT SUMMARY:
LOG:
01:51UTC EX and H_T_T_S Verbal Alarms. Nothing environmentally obvious
02:31 unable to cover PRMI in a timely manner
2:32 IA
2:55 Re-Lock attempt - 1
03:16 Re-Lock attempt -2
03:39 Loaded new ISC_DRMI code
03:40 Re-Lock attempt - 3
04:09 Re-Lock attempt - 4
~ 4:30UTC I placed a call to Sheila
5:20UTC NLN
5:24UTC Observing
Ed called about difficulty locking, since I don't see anything obvious I looked at BRS channels I foudn trended here: 51338. It looks like the low frequency BLRMS of BRS X has been growing exponentially.
Here's th BRS HEALTH ndscope image during ENGAGE_SOFT_LOOPS
I don't see anything problematic really in the time series. But I did notice from the Detchar summary pages that there was some weird behavior with the inside temperature for a few hours from 16 UTC yesterday (02/09/2019). It takes hours until the BRS itself responds to temperature changes so I'm just guessing but you might be seeing the effects of that. You might consider turning the sensor correction to NON BRS state and lock since the wind is low now anyways. Here is the plot of the temperature
?
I don't think there is anything wrong with BRSX. First attached trend are 7 weeks of the both BRS driftmons, which are kind of a measure of the DC position. BRSX is close to the level where I would want to recenter, but I hope to push that off until we try to install Eyal and Arnaud's heating pad. I don't think the "strangeness" Eyal notes is anything to be worried about. It's a tenth of a degree, from 8am local to about 4pm, probably the aircon.
Second plot are 4 days of trends for ISC_LOCK (top left), the sensor correction control signal generated from the tilt subtracted STS (bottom left) and the drift (top right) and velocity of BRSX (bottom right). I don't see any signals in the right 2 plots that would cause the spikes in the sensor correction (so I'm assuming they are earthquakes, but I'll keep looking), or would be any cause for concern.
I don't know why the lowest frequency BLRMS of the BRS seem to accumulate these large numbers but I don't think it's a problem. Or if it is, both BRS are busted. Third image is the BLRMS for both BRS for the last 2 weeks. Maybe this is some sort of computational weirdness? Some integrator in the BLRMS filter? Maybe we should periodically clear the histories of some of the filters.
I've looked at few non-brs blrms channels and all of the dc-30mhz blrms channels show this same behavior. The 30mhz low pass is just integrating non-sense. There is no easy way to clear the history of the filter that does the calculation, without restarting the model. Dave and I are talking about adding an epic variable to zero out the filter. We will try prototyping on the test stand.
can you inject in parallel a negative large DC to try and zero it out?
The 30mhz BLRMS filter shouldn't be integrating like this. Filed FRS https://services.ligo-la.caltech.edu/FRS/show_bug.cgi?id=13602
Pursuant to LIGO-T1900555 [1] and discussions on the DetChar and HW injections mailing list [2,3], I've scheduled DetChar safety injections to begin at 23:00:00 UTC (18:00:00 CDT, 16:00:00 PDT) 3 Sep 2019. These should last for 390 seconds and contain 75 injections spaced 5 sec apart. The scheduled injections have been added to GraceDb [4]. We will update the events in GraceDb to record whether these injections were successful post-hoc. **Please note**, we have schedule coincident (but not necessarily coherent) injections at LLO for the same time. We include the LLO aLOG describing those injections in a comment below. More details on how these injections were generated and added to the HW injection SVN repo are available via a README [5]. [1] https://dcc.ligo.org/LIGO-T1900555 [2] detchar@ligo.org [3] hw-injections@ligo.org [4] https://gracedb.ligo.org/search/?query=Test+INJ+gpstime%3A+1251586818+..+1251587400+instruments%3A+%22H1%22&query_type=E&results_format=S [5] https://ldas-jobs.ligo.caltech.edu/~detchar/hwinj/LIGO-T1900555/README
The coincident (but not necessarily coherent) injections at LLO are described here:
https://alog.ligo-la.caltech.edu/aLOG/index.php?callRep=48279
I svn uploaded the waveforms and schedule file for this injection, and reloaded the guardian, so if the IFO is observing then the injections should go in.
This injection failed (see alog 51720) with what looks like the same INJ TRANS guardian error we have been seeing at LLO (see llo alog 46132 and FRS ticket 12939)
GraceDb entries associated with these injections have been updated to reflect the fact that they did not go in correctly (labeled HWINJNO)
Pursuant to LIGO-T1900555 [1] and discussions on the DetChar and HW injections mailing list [2,3], I've scheduled DetChar safety injections to begin at 21:00:00 UTC (16:00:00 CDT, 14:00:00 PDT) 3 Sep 2019. These should last for 390 seconds and contain 75 injections spaced 5 sec apart. The scheduled injections have been added to GraceDb [4]. We will update the events in GraceDb to record whether these injections were successful post-hoc. **Please note**, we have schedule coincident (but not necessarily coherent) injections at LLO for the same time. We include the LLO aLOG describing those injections in a comment below. More details on how these injections were generated and added to the HW injection SVN repo are available via a README [5]. [1] https://dcc.ligo.org/LIGO-T1900555 [2] detchar@ligo.org [3] hw-injections@ligo.org [4] https://gracedb.ligo.org/search/?query=Test+INJ+gpstime%3A+1251579618+..+1251580100+instruments%3A+%22H1%22&query_type=E&results_format=S [5] https://ldas-jobs.ligo.caltech.edu/~detchar/hwinj/LIGO-T1900555/README
The coincident (but not necessarily coherent) injections at LLO are described here: https://alog.ligo-la.caltech.edu/aLOG/index.php?callRep=48277
Neither IFO was up by this time, so I removed this injection from the schedule file.
Injection failed. Guardian log below:
2019-09-03_22:37:31.316944Z INJ_TRANS RELOAD requested. reloading system data...
2019-09-03_22:37:31.360492Z INJ_TRANS module path: /opt/rtcds/userapps/release/cal/common/guardian/INJ_TRANS.py
2019-09-03_22:37:31.361022Z INJ_TRANS user code: /opt/rtcds/userapps/release/cal/common/guardian/injtools/__init__.py
2019-09-03_22:37:31.361022Z INJ_TRANS user code: /opt/rtcds/userapps/release/cal/common/guardian/injtools/inj_det.py
2019-09-03_22:37:31.361022Z INJ_TRANS user code: /opt/rtcds/userapps/release/cal/common/guardian/injtools/inj_io.py
2019-09-03_22:37:31.361022Z INJ_TRANS user code: /opt/rtcds/userapps/release/cal/common/guardian/injtools/inj_types.py
2019-09-03_22:37:35.800454Z INJ_TRANS RELOAD complete
2019-09-03_22:37:35.801803Z INJ_TRANS W: RELOADING @ WAIT_FOR_NEXT_INJECT.run
2019-09-03_22:55:00.027528Z INJ_TRANS [WAIT_FOR_NEXT_INJECT.run] ezca: H1:CAL-INJ_TINJ_START => 1251586518.03
2019-09-03_22:55:00.028168Z INJ_TRANS [WAIT_FOR_NEXT_INJECT.run] ezca: H1:CAL-INJ_TINJ_OUTCOME => 0
2019-09-03_22:55:00.148272Z INJ_TRANS EDGE: WAIT_FOR_NEXT_INJECT->CHECK_SCHEDULE_TIMES
2019-09-03_22:55:00.148896Z INJ_TRANS calculating path: CHECK_SCHEDULE_TIMES->INJECT_SUCCESS
2019-09-03_22:55:00.149298Z INJ_TRANS new target: CREATE_AWG_STREAM
2019-09-03_22:55:00.150149Z INJ_TRANS executing state: CHECK_SCHEDULE_TIMES (35)
2019-09-03_22:55:00.151587Z INJ_TRANS [CHECK_SCHEDULE_TIMES.main] USERMSG 0: INJECTION IMMINENT: 1251586818.000000
2019-09-03_22:55:00.152104Z INJ_TRANS [CHECK_SCHEDULE_TIMES.main] Skipping checking schedule times since its already been done.
2019-09-03_22:55:00.265266Z INJ_TRANS EDGE: CHECK_SCHEDULE_TIMES->CREATE_AWG_STREAM
2019-09-03_22:55:00.271061Z INJ_TRANS calculating path: CREATE_AWG_STREAM->INJECT_SUCCESS
2019-09-03_22:55:00.273132Z INJ_TRANS new target: READ_WAVEFORM
2019-09-03_22:55:00.274979Z INJ_TRANS executing state: CREATE_AWG_STREAM (50)
2019-09-03_22:55:00.275920Z INJ_TRANS [CREATE_AWG_STREAM.enter]
2019-09-03_22:55:00.277244Z INJ_TRANS [CREATE_AWG_STREAM.main] ezca: H1:CAL-INJ_TINJ_TYPE => 3
2019-09-03_22:55:00.391173Z INJ_TRANS EDGE: CREATE_AWG_STREAM->READ_WAVEFORM
2019-09-03_22:55:00.391774Z INJ_TRANS calculating path: READ_WAVEFORM->INJECT_SUCCESS
2019-09-03_22:55:00.391774Z INJ_TRANS new target: RAMP_GAIN_TO_1
2019-09-03_22:55:00.392868Z INJ_TRANS executing state: READ_WAVEFORM (60)
2019-09-03_22:55:00.395535Z INJ_TRANS [READ_WAVEFORM.main] Reading waveform data from /hwinj/Details/detchar/detchar-hwinj-schedule_LIGO-T1900555-PROPOSED_H1_INJECTIONS-1251586818-390.txt
2019-09-03_22:55:27.569259Z INJ_TRANS EDGE: READ_WAVEFORM->RAMP_GAIN_TO_1
2019-09-03_22:55:27.570014Z INJ_TRANS calculating path: RAMP_GAIN_TO_1->INJECT_SUCCESS
2019-09-03_22:55:27.570014Z INJ_TRANS new target: AWG_STREAM_OPEN_PREINJECT
2019-09-03_22:55:27.570974Z INJ_TRANS executing state: RAMP_GAIN_TO_1 (65)
2019-09-03_22:55:27.581834Z INJ_TRANS [RAMP_GAIN_TO_1.enter]
2019-09-03_22:55:27.583248Z INJ_TRANS [RAMP_GAIN_TO_1.main] ezca: H1:CAL-INJ_TRANSIENT_GAIN => 1.0
2019-09-03_22:55:29.829405Z INJ_TRANS EDGE: RAMP_GAIN_TO_1->AWG_STREAM_OPEN_PREINJECT
2019-09-03_22:55:29.830281Z INJ_TRANS calculating path: AWG_STREAM_OPEN_PREINJECT->INJECT_SUCCESS
2019-09-03_22:55:29.831068Z INJ_TRANS new target: INJECT_CBC_ACTIVE
2019-09-03_22:55:29.832679Z INJ_TRANS executing state: AWG_STREAM_OPEN_PREINJECT (70)
2019-09-03_22:55:29.838744Z INJ_TRANS [AWG_STREAM_OPEN_PREINJECT.main] USERMSG 0: INJECTION IMMINENT: 1251586818.000000
2019-09-03_22:59:40.085443Z INJ_TRANS JUMP target: INJECT_DETCHAR_ACTIVE
2019-09-03_22:59:40.086053Z INJ_TRANS [AWG_STREAM_OPEN_PREINJECT.exit]
2019-09-03_22:59:40.150495Z INJ_TRANS JUMP: AWG_STREAM_OPEN_PREINJECT->INJECT_DETCHAR_ACTIVE
2019-09-03_22:59:40.151051Z INJ_TRANS calculating path: INJECT_DETCHAR_ACTIVE->INJECT_SUCCESS
2019-09-03_22:59:40.151051Z INJ_TRANS new target: RAMP_GAIN_TO_0
2019-09-03_22:59:40.152161Z INJ_TRANS executing state: INJECT_DETCHAR_ACTIVE (104)
2019-09-03_22:59:40.153871Z INJ_TRANS [INJECT_DETCHAR_ACTIVE.main] USERMSG 0: INJECTION ACTIVE: 1251586818.000000
2019-09-03_22:59:40.154308Z awgSetChannel: awg_clnt[42][0] = NULL
2019-09-03_22:59:40.154308Z Error code from awgSetChannel: -5
2019-09-03_22:59:40.170270Z INJ_TRANS [INJECT_DETCHAR_ACTIVE.main] File "/opt/rtcds/userapps/release/cal/common/guardian/INJ_TRANS.py", line 572, in main
2019-09-03_22:59:40.170270Z self.hwinj.stream.send(self.hwinj.data)
2019-09-03_22:59:40.170928Z INJ_TRANS [INJECT_DETCHAR_ACTIVE.main] File "/usr/lib/python2.7/dist-packages/awg.py", line 621, in send
2019-09-03_22:59:40.170928Z self.append(data, scale=scale)
2019-09-03_22:59:40.170928Z INJ_TRANS [INJECT_DETCHAR_ACTIVE.main] File "/usr/lib/python2.7/dist-packages/awg.py", line 599, in append
2019-09-03_22:59:40.170928Z self.open()
2019-09-03_22:59:40.170928Z INJ_TRANS [INJECT_DETCHAR_ACTIVE.main] File "/usr/lib/python2.7/dist-packages/awg.py", line 584, in open
2019-09-03_22:59:40.170928Z + ": " + awgbase.SIStrErrorMsg(ret))
2019-09-03_22:59:40.170928Z INJ_TRANS [INJECT_DETCHAR_ACTIVE.main] <class 'awg.AWGStreamError'> can't open stream to H1:CAL-INJ_TRANSIENT_EXC: Error setting up an awg slot for the channel
2019-09-03_22:59:40.197166Z INJ_TRANS JUMP target: FAILURE_DURING_ACTIVE_INJECT
2019-09-03_22:59:40.197823Z INJ_TRANS [INJECT_DETCHAR_ACTIVE.exit]
2019-09-03_22:59:40.261389Z INJ_TRANS JUMP: INJECT_DETCHAR_ACTIVE->FAILURE_DURING_ACTIVE_INJECT
2019-09-03_22:59:40.262094Z INJ_TRANS calculating path: FAILURE_DURING_ACTIVE_INJECT->INJECT_SUCCESS
2019-09-03_22:59:40.262094Z INJ_TRANS new target: WAIT_FOR_NEXT_INJECT
2019-09-03_22:59:40.270095Z INJ_TRANS executing state: FAILURE_DURING_ACTIVE_INJECT (300)
2019-09-03_22:59:40.272312Z INJ_TRANS [FAILURE_DURING_ACTIVE_INJECT.enter]
2019-09-03_22:59:40.273597Z INJ_TRANS [FAILURE_DURING_ACTIVE_INJECT.main] ezca: H1:CAL-INJ_TRANSIENT_GAIN => 0.0
2019-09-03_22:59:42.405392Z INJ_TRANS [FAILURE_DURING_ACTIVE_INJECT.main] ezca: H1:CAL-INJ_TINJ_OUTCOME => -4
2019-09-03_22:59:42.405950Z INJ_TRANS [FAILURE_DURING_ACTIVE_INJECT.main] ezca: H1:CAL-INJ_TINJ_ENDED => 1251586800.41
2019-09-03_22:59:42.451041Z INJ_TRANS [FAILURE_DURING_ACTIVE_INJECT.run] USERMSG 0: ERROR
Was actually the 2300UTC injection from this alog (alog 51684) that failed. The 2100 UTC injection was removed from the schedule.
GraceDb entries associated with these injections have been updated to reflect the fact that they were not made successfully (labeled HWINJNO)