Dropped out of observing for one minute for a Squeezer hiccup. Had one long GRB alert. Ignored due to duration trigger time. No other concerns at this time.
TITLE: 09/14 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 112Mpc
INCOMING OPERATOR: Jeff
SHIFT SUMMARY: My shift started with the IFO engaging soft loops, and Jeff and Sheila discussing the recent lock losses from this state. Made it up with no trouble this time and we have been locked since. After getting to low noise the squeezer manager reported "green power stabilization railed". Sheila tweaked a pico and all is well since then.
LOG:
Locked for 4.5 hours. No issues to report
Jeff K, Sheila, TJ, Jim
Jim had two more locklosses at engage ASC while trying to relock after this afternoon's lockloss. In one case, there aren't many clues except for the usual error signals deviating from 0: 1252450014
In the earlier one 1252449154, the lockloss happened after all the ASC was on and the first convergence checker had passed. At that time we increase the gain of some loops and turn off the offsets in AS36Q which are set in DRMI. I noticed that we have a 4 second wait for a 10 second ramp on the offsets, so I set the wait time to be the TRAMP for the offsets.
The attached ndscope for this lockloss shows both CHARD and INP1Y hitting the soft limiters in the 5 seconds before the lockloss. We use the soft limiters to allow us to turn on all the ASC loops at once, in principle we would like to not have soft limiters on our slowest loops, and only limit those which move fast so that the slow ones can keep up. We haven't taken the time to figure out which loops actually need the limiters, so we have them on all the loops that get engaged here. This means we are probably unnecessarily limiting the ability of our slower loops to keep up in some cases, so eventually we probably want to be more selective about which soft limiters we are using.
The soft limiter is only meant to help us when we are first engaging the loops. Once we have all the loops on and they have converged once, they shouldn't be on anymore since all they will do is make slow down the ASC system's ability to respond to disturbances. In this lockloss, INP1 + CHARD Y hit their limiters after everything was converged (I am not sure what caused a disturbance). I changed the guardian code so that the soft limiters will be turned off after the first convergence check.
Jeff is looking through some of our recent locklosses from ENGAGE_ASC to see if this could potentially have helped us avoid other similar locklosses.
I finished completing Jeff K's table below. Out of the lock losses from ENGAGE_ASC_FOR_FULL_IFO (STATE N = 430), 6 out of the 9 hit some soft limiters. Most consistently, it was INP1_Y and CHARD_Y. It's tough to say if there was correlation between lock losses before or after the PRC2/SRC1 gain increase at the end of that state.
Loops that get error signal limiters:
PRC2_
INP1_
CHARD_
SRC1_
SRC2_
Ones that don't DHARD, MICH, ADS3 (PRC1)
(1) need to wait for ramp off of offset on MICH
(2) after first convergence checker, we turn off the soft limiters. Used to be that we didn't turn off the soft limiteres until after the *second* converegnece checker ,which is after the PRC2/SRC1 boosts get turned on. So maybe the loops that were limited couldn't keep up with the new boosts.
Locklosses around ENGAGE_ASC_FOR_FULL_IFO (STATE N = 430)
search for guardian state 430
LLN GPS UTC What State? What Hits Soft Limits? Before or After PRC2/SRC1 Gain Increases?
1252450014 September 13 2019, 22:46:36 UTC 435 (no, survived) Well after, INP1, CHARD runaway
0 1252449154 September 13 2019, 22:32:16 UTC 430 INP1, CHARD After
1252421630 September 13 2019, 14:53:32 UTC 435 (no, survived) Well after, INP1, CHARD runaway
1252252912 September 11 2019, 16:01:34 UTC 429 INP1 is wildly oscillating Before (Sheila caused, increased INP1 gain too much )
1 1252249409 September 11 2019, 15:03:11 UTC 430 none After
1252145321 September 10 2019, 10:08:23 UTC 435 (no, survived) Well after, INP1, CHARD runaway
2 1252130742 September 10 2019, 06:05:24 UTC 430 CHARD
1252043449 September 09 2019, 05:50:31 UTC 430 INP1_Y, CHARD_Y After, INP1_P runaway
1251914435 September 07 2019, 18:00:17 UTC 435 none Well after, INP1_Y, CHARD_Y runaway
3 1251913373 September 07 2019, 17:42:35 UTC 430 CHARD
4 1251807482 September 06 2019, 12:17:44 UTC 430 CHARD, INP1
1251806452 September 06 2019, 12:00:34 UTC 430 none After
1231738298 September 05 2019, 17:04:40 UTC 435 none Well after, INP1_Y, CHARD_Y runaway
1251638577 September 05 2019, 13:22:39 UTC 430 CHARD_Y, INP1_Y After
1251636875 September 05 2019, 12:54:17 UTC 430 none After
Follow-up on this change: there have been several more lock-losses in/around the ENGAGE_ASC_FOR_FULL_IFO (see, e.g. operator report in LHO aLOG 51961). I'm not sure this guardian code change to move turning off the error signal ("smooth") limiters worked. One collective series during Travis' Sunday shift: GPS Time UTC Time State Error Limiters Hit Lock Loss Before / After PRC2/SRC1 Gain Increases Did Error Limiters Get Turned Off? 1252609254.84 Sep 15 2019, 19:00:36 UTC 430 none, but close After Yes (Made it to where they're turned off) 1252610293.71 Sep 15 2019, 19:17:55 UTC 430 CHARD_Y, INP1_Y After No 1252611608.05 Sep 15 2019, 19:39:50 UTC 430 none, but close After Yes (Made it to where they're turned off) 1252612537.75 Sep 15 2019, 19:55:19 UTC 430 CHARD_Y, INP1_Y After No 1252613443.62 Sep 15 2019, 20:10:25 UTC 430 CHARD_Y, INP1_Y After No 1252615042.75 Sep 15 2019, 20:37:04 UTC 430 CHARD_Y, INP1_Y After No 1252666222.31 Sep 16 2019, 10:50:04 UTC 430 CHARD_Y, INP1_Y After No For the record, so others can perform the same study: Command for lockloss tool used: lockloss -c /opt/rtcds/userapps/release/isc/h1/ndscope/asc_fullIFO.yml select - Looking at the error signals of the ndscope template (you'll likely have to zoom the y-axis out), look for "flat-lining" where the error signal appears to get large and goes to what looks like an artificially flat trace. Leading up to the above locklosses, this happens at ~ -6 for INPY1_Y and ~ -2 for CHARD_Y. - Looking through the guardian log, see where there are lines 2019-09-16_10:49:38.220815Z ISC_LOCK [ENGAGE_ASC_FOR_FULL_IFO.run] ezca: H1:ASC-PRC2_P_SW1 => 16 2019-09-16_10:49:38.474971Z ISC_LOCK [ENGAGE_ASC_FOR_FULL_IFO.run] ezca: H1:ASC-PRC2_P => OFF: FM1 2019-09-16_10:49:38.474971Z ISC_LOCK [ENGAGE_ASC_FOR_FULL_IFO.run] ezca: H1:ASC-PRC2_Y_SW1 => 16 2019-09-16_10:49:38.723825Z ISC_LOCK [ENGAGE_ASC_FOR_FULL_IFO.run] ezca: H1:ASC-PRC2_Y => OFF: FM1 2019-09-16_10:49:38.724816Z ISC_LOCK [ENGAGE_ASC_FOR_FULL_IFO.run] ezca: H1:ASC-SRC1_Y_SW1 => 16 2019-09-16_10:49:38.975953Z ISC_LOCK [ENGAGE_ASC_FOR_FULL_IFO.run] ezca: H1:ASC-SRC1_Y => OFF: FM1 2019-09-16_10:49:38.977005Z ISC_LOCK [ENGAGE_ASC_FOR_FULL_IFO.run] ezca: H1:ASC-SRC1_P_GAIN => 12 2019-09-16_10:49:38.977982Z ISC_LOCK [ENGAGE_ASC_FOR_FULL_IFO.run] ezca: H1:ASC-AS_A_RF36_Q_PIT_SW1 => 8 2019-09-16_10:49:39.229301Z ISC_LOCK [ENGAGE_ASC_FOR_FULL_IFO.run] ezca: H1:ASC-AS_A_RF36_Q_PIT => OFF: OFFSET 2019-09-16_10:49:39.230172Z ISC_LOCK [ENGAGE_ASC_FOR_FULL_IFO.run] ezca: H1:ASC-AS_A_RF36_Q_YAW_SW1 => 8 2019-09-16_10:49:39.481531Z ISC_LOCK [ENGAGE_ASC_FOR_FULL_IFO.run] ezca: H1:ASC-AS_A_RF36_Q_YAW => OFF: OFFSET 2019-09-16_10:49:39.482179Z ISC_LOCK [ENGAGE_ASC_FOR_FULL_IFO.run] ezca: H1:ASC-AS_A_RF36_Q_YAW_OFFSET => 0 2019-09-16_10:49:39.482503Z ISC_LOCK [ENGAGE_ASC_FOR_FULL_IFO.run] ezca: H1:ASC-AS_A_RF36_Q_PIT_OFFSET => 0 happen, in relation to lines like 2019-09-15_19:39:50.260416Z ISC_LOCK JUMP: ENGAGE_ASC_FOR_FULL_IFO->LOCKLOSS 2019-09-15_19:39:50.260416Z ISC_LOCK calculating path: LOCKLOSS->NOMINAL_LOW_NOISE 2019-09-15_19:39:50.267164Z ISC_LOCK new target: DOWN 2019-09-15_19:39:50.273348Z ISC_LOCK executing state: LOCKLOSS (2) 2019-09-15_19:39:50.274286Z ISC_LOCK [LOCKLOSS.enter] 2019-09-15_19:39:50.395063Z ISC_LOCK JUMP target: DOWN namely, if they're before or after lines like 2019-09-15_19:39:46.596598Z ISC_LOCK [ENGAGE_ASC_FOR_FULL_IFO.run] ezca: H1:ASC-INP1_P_SMOOTH_ENABLE => 0 2019-09-15_19:39:46.596943Z ISC_LOCK [ENGAGE_ASC_FOR_FULL_IFO.run] ezca: H1:ASC-INP1_Y_SMOOTH_ENABLE => 0 2019-09-15_19:39:46.597208Z ISC_LOCK [ENGAGE_ASC_FOR_FULL_IFO.run] ezca: H1:ASC-CHARD_P_SMOOTH_ENABLE => 0 2019-09-15_19:39:46.597478Z ISC_LOCK [ENGAGE_ASC_FOR_FULL_IFO.run] ezca: H1:ASC-CHARD_Y_SMOOTH_ENABLE => 0 2019-09-15_19:39:46.598132Z ISC_LOCK [ENGAGE_ASC_FOR_FULL_IFO.run] ezca: H1:ASC-PRC2_P_SMOOTH_ENABLE => 0 2019-09-15_19:39:46.598455Z ISC_LOCK [ENGAGE_ASC_FOR_FULL_IFO.run] ezca: H1:ASC-PRC2_Y_SMOOTH_ENABLE => 0 2019-09-15_19:39:46.598808Z ISC_LOCK [ENGAGE_ASC_FOR_FULL_IFO.run] ezca: H1:ASC-SRC1_P_SMOOTH_ENABLE => 0 2019-09-15_19:39:46.599082Z ISC_LOCK [ENGAGE_ASC_FOR_FULL_IFO.run] ezca: H1:ASC-SRC1_Y_SMOOTH_ENABLE => 0 2019-09-15_19:39:46.599343Z ISC_LOCK [ENGAGE_ASC_FOR_FULL_IFO.run] ezca: H1:ASC-SRC2_P_SMOOTH_ENABLE => 0 ... if they happen. As indicated above, most did not happen.
Sheila confirms, after discussing my review with her, that it was likely that code changes did not get loaded.
I've loaded the ISC_LOCK guardian code this afternoon, during the calibration measurement period.
2019-09-16_21:16:49.389046Z ISC_LOCK LOAD REQUEST
2019-09-16_21:16:49.390035Z ISC_LOCK RELOAD requested. reloading system data...
2019-09-16_21:16:49.423530Z /opt/rtcds/userapps/release/isc/h1/guardian/ISC_LOCK.py: inconsistent use of tabs and spaces in indentation
2019-09-16_21:16:50.217943Z /opt/rtcds/userapps/release/isc/h1/guardian/ISC_LOCK.py: inconsistent use of tabs and spaces in indentation
2019-09-16_21:16:50.274528Z ISC_LOCK module path: /opt/rtcds/userapps/release/isc/h1/guardian/ISC_LOCK.py
2019-09-16_21:16:50.275249Z ISC_LOCK user code: /opt/rtcds/userapps/release/isc/h1/guardian/switch_SDF_source_files.py
2019-09-16_21:16:50.275249Z ISC_LOCK user code: /opt/rtcds/userapps/release/sys/common/guardian/cdslib.py
2019-09-16_21:16:50.275249Z ISC_LOCK user code: /opt/rtcds/userapps/release/isc/h1/guardian/lscparams.py
2019-09-16_21:16:50.275249Z ISC_LOCK user code: /opt/rtcds/userapps/release/isc/h1/guardian/ISC_GEN_STATES.py
2019-09-16_21:16:50.275249Z ISC_LOCK user code: /opt/rtcds/userapps/release/isc/h1/guardian/ISC_library.py
2019-09-16_21:16:50.275249Z ISC_LOCK user code: /opt/rtcds/userapps/release/isc/h1/guardian/fast_ezca.py
2019-09-16_21:16:55.589754Z ISC_LOCK system archive: code changes detected and committed
2019-09-16_21:16:56.618081Z ISC_LOCK system archive: id: d347c69389743cbdc979e7dfbc2df8a33cd97f2a (221543529)
2019-09-16_21:16:56.618594Z ISC_LOCK RELOAD complete
2019-09-16_21:16:56.619981Z ISC_LOCK W: RELOADING @ NLN_CAL_MEAS.run
CAL CS with the only SDF diffs, Jeff K said that he would work on the rounding issue soon. I accepted all of these since they are all tiny differences.
Briefly out of Observing to fix a squeezer green power railed notification from the SQZ_MANAGER.
Out time: 2343 - 0002 UTC
TITLE: 09/13 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
OUTGOING OPERATOR: Jim
CURRENT ENVIRONMENT:
SEI_CONF state: WINDY
Wind: 16mph Gusts, 13mph 5min avg
Primary useism: 0.04 μm/s
Secondary useism: 0.13 μm/s
QUICK SUMMARY: Currently trying to understand why we are losing lock when we engage soft loops.
TITLE: 09/13 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: TJ
SHIFT SUMMARY:
LOG:
18:45 Kyle to My
20:00 Kyle to Mx
21:00 Lockloss, no cause, DRMI/PRMI seem hopess, so Initial alignment
23:00 ASC keeps blowing locks, leaving this for Sheila, TJ and Jeff
No CLF power jumps due to AOM1, since the hardware swap last Tuesday (alog 51856).
h1psl0 DAQ data had an issue last night at 11:00:20 PDT (06:00:20 UTC Fri 13). Attached plot shows that 0.5 seconds of data were repeated at this time. Next time we are out of observe I'll clear the CRC counters.
I've cleared the CRC counter.
CAL CS had 27 diffs on getting back to NLN.
Attached are the generic lock loss trends with a 5sec window and 500sec. No obvious cause for lock loss here.
First half of the shift was good. The IFO has been observing for 30 hours. Environmental conditions are favorable. The range is 115.8Mpc. No issues or concerns at this time.
TITLE: 09/11 Eve Shift 23:00 – 07:00 (16:00-00:00), all times posted in UTC
STATE of H1: Observing
INCOMING OPERATOR: Jeff
SHIFT SUMMARY: Quiet shift. Locked and Observing for 26 hours.
LOG:
23:00 (16:00) Start of shift
03:44 (20:44) GRB-Short (E350280) -- Standing down for 15
07:00 (00:00) End of shift
YES THIS WILL DROP THE FRONT-END COMPUTED BNS RANGE, because we're improving the accuracy of the real-time pipeline in order to better matrch the current IFO (more power than when it was last updated, no detuning, good spot positions, SRCL offset, accounting for new UIM boost, etc.) Should be done in an hour or three. When we come back in to OBSERVATION READY, the new model will be in place. See details of the model in LHO aLOG 51784, and references there in. The decision point to change the model is documented in LHO aLOG 51782, which has yet more references and explanation.
INSTALLATION COMPLETE. As predicted, front-end BNS range dropped by ~5 Mpc, and now in better agreement with GDS produced BNS range. First OBSERVATION READY SEGMENT with this new 2019-09-09 model installed started at GPS Time 1252088806 (Sep 09 2019 18:26:28 UTC; Sep 09 2019 11:26:28 PDT).
Reference Model values at calibration line frequencies (aka "EPICs records") for this model update have been generated and installed.
/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Results/CALCS_FE/
epicsrecords_model-H1_20190909_created-20190909.txt
In doing so, I've created the script
/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Scripts/CALCS_FE/
python3.5 createEPICS_for_20190909.py rev 8397.
and modified it and computeDARM.py to be able to write the EPICs records to file separately from actually pushing it to the front-end.
/ligo/svncommon/CalSVN/aligocalibration/trunk/Common/pyDARM/src/
computeDARM.py rev 8399
I've used the broadband injection made just after the model update to do a comparison of calibration accuracy with different levels of time-dependent corrections applied, using the new model. This is similar to the analysis done in LHO aLOG 51794. The attached plot has 4 versions of offline calibration, each with different TDCF corrections applied, as indicated in the plot legend. The 5th (red) curve is from the online GDS data (C00). The previous conclusions are still supported by this data: 1) We should not be correcting for SRC time dependence; and 2) We should not be applying the imaginary parts of the actuation kappas. Note the large errors caused by applying corrections for SRC time dependence. This may be due to the fact that the model value of f_s is zero, causing numerical instability when applying time-dependent corrections.
This plot, as well as text files with the data, are available in the calibration SVN here:
aligocalibration/trunk/Runs/O3/H1/Measurements/FullIFOSensingTFs/DCS_BB_plots
The values of the TDCFs used by the GDS and DCS calibration pipelines to produce the calibrated data during this broadband injection were:
For the C00 data (this can be seen on the summary page for that day, just after 18 UTC):
For the DCS data (very similar, but not quite identical):