This afternoon we had 2 more locklosses due to the instability at 1.88 Hz which we have struggled with for a few weeks. (31707)
This time Keita and I looked at the filters for the soft loops, and saw that we have some kind of plant inversion from a long time ago that has a complex pole at 1.83 Hz, which was giving us 15bB of gain there unnecessarily. (reminder, the soft loops generally have bandwidths of a few hundred mHz, except for CSOFT P which needs to have gain at 0.5Hz to supress our dPdtheta instability).
We turned off these invY filters, and saw that the noise at 1.88Hz in several of our YAW ASC loops was reduced. For the soft yaw loops, the peak at 1.88Hz was reduced by about 20dB (in the attachment, dashed lines are from last night's undisturbed lock stretch, the solid lines are from the last lock where the invY filters were off.) There is still clearly extra noise at 1.88 Hz, so something else may be nearly unstable.
I've scheduled a coherent bbh injection for 3:00 UTC as part of the end-to-end test. The schedule file (schedule_1163473217.txt) was modified and checked into the svn (https://daqsvn.ligo-la.caltech.edu/svn/injection/hwinj/Details/inj_trans/). I logged into each site's hardware injection machine (l1hwinj1 and h1hwinj1) and checked out the updated schedule file (/usr/local/home/hinj/Details/inj_trans). I then reloaded the INJ_TRANS guardian at each site. At LHO EvanG checked that the guardian was ok, he had to change the request to INJECT_SUCCESS (i forgot to check this). Here is the new line in the schedule file: 1163905217 H1L1 INJECT_CBC_ACTIVE 1 1.0 Inspiral/{ifo}/coherentbbh7_1126259455_{ifo}.out Each IFO will need to be in observation intent mode for the injection to go through.
The injections were injected. The section of the INJ_TRANS log that covers the injection is in the attached text file.
End-to-end test:
Hardware injection was successfully performed on H1 and L1 at the same time at around 03:00:07 UTC, analysis pipelines picked it up and we heard multiple audible alarms.
Operator went through site response (L1500117) and acknowledged alarms as they came in. I and fellows got on teamspeak EM channel with EM and detchar representatives.
Also see LLO alog: https://alog.ligo-la.caltech.edu/aLOG/index.php?callRep=29750
J. Kissel WP 6338 We've implemented the same bug fix to the CAL-CS front-end model that LLO has that connects the H1:CAL-CS_TDEP_DARM_LINE2_DEMOD_OSC into the sum for calibration line output of the CAL-CS model, shipped over to the OMC model to be added in just after the LSC-DARM banks. This oscillator is occasionally used for testing, so we need it hooked up. Check out LLO aLOG 29521 for screenshots of the model change. This change merely required an svn update of the /opt/rtcds/userapps/release/cal/common/models/ CAL_CS_MASTER.mdl to absorb the changes that Joe has already made, followed by an installation and restart of the h1calcs model. No DAQ restart was required, but the DAQ was restarted for various other changes made this morning.
J. Kissel, ECR E1600340 II 6665 WP 6338 I've installed the infrastructure that allows for control of the IFO's DARM DOF with either (or both) PCALs as designed by LLO (see LHO aLOG 29509). Unfortunately, because this was implemented quickly -- this is not decided in the LSC output matrix, but a combination of gains downstream. This change required an update to the common library part, /opt/rtcds/userapps/release/cal/common/models/PCAL_MASTER.mdl, modifications to both PCAL model's top-level /opt/rtcds/userapps/release/cal/h1/models h1calex.mdl h1caley.mdl, and a DAQ restart after installation and restarts of both end-station cal models. Also, I've updated the PCAL_END excitation screen, /opt/rtcds/userapps/release/cal/common/medm/PCAL_END_EXC.adl However, I discovered that the PCAL overview /opt/rtcds/userapps/release/cal/common/medm/PCAL_END.adl also need modification to pass down the correct macro variables from that it received from the SITEMAP. Otherwise the new path to the new PCAL_DARM filter bank was ill-defined and wouldn't open. I've committed the changes to the top level models, as well and the PCAL overview screen in there respective locations in the SVN. For now, to transition from using ETM DARM control to PCAL DARM control, one should leave the LSC output matrix pointing to the correct end station's QUAD, but zero the H1:SUS-ETM[X/Y]_L3_ISCINF_L_GAIN (i.e. the first longitudinal input to the SUS), and turn on that end-stations PCAL DARM control bank, H1:CAL-PCALY_DARM. Of course, you have to design the appropriate filtering to account for the difference between the ESD drive and PCAL drive, which I have not yet done.
Krishna
I recentered the c-BRS based on a visual inspection of the instrument. The two interferometer arm-lengths were mismatched on the order of ~1 mm and were accordingly adjusted. The differential tilt signal may show some improvement, assuming it is not limited by other noise.
I have also been investigating the excess low-frequency signals in c-BRS by comparing it to a host of nearby sensors. So far, the likely suspect appears to be air pressure, though I don't yet understand the mechanism for the effect. I'll fill in more details tomorrow.
I have restarted the DCS Disk2Disk processes writing to archive at LHO with, set CHECK_SOURCE_MD5SUM 1 in the Disk2Disk .conf files, so that the CDS .md5 files are now compared with the md5sum found by Disk2Disk before the file gets copied to the LDAS archive. (The file must also pass FrCheck to be copied, as before.)
Evan, Dave
the CW hardware injection has not been running since the restart of the h1calex model at lunch time. Via monit, I restarted this process and it looks like the injection is running again.
J. Kissel for S. Karki I've moved the roaming calibration line to its highest frequency we intend to go, and it's also the last super-long duration we need. We may run through the lower frequency points again, given that (a) they need much less data, and (b) those data points were taken at various input powers that will likely confuse/complicate the analysis. Below is the current schedule status. Current Schedule Status: Frequency Planned Amplitude Planned Duration Actual Amplitude Start Time Stop Time Achieved Duration (Hz) (ct) (hh:mm) (ct) (UTC) (UTC) (hh:mm) --------------------------------------------------------------------------------------------------------------------------------------------------------- 1001.3 35k 02:00 39322.0 Nov 11 2016 21:37:50 UTC Nov 12 2016 03:28:21 UTC ~several hours @ 25 W 1501.3 35k 02:00 39322.0 Oct 24 2016 15:26:57 UTC Oct 31 2016 15:44:29 UTC ~week @ 25 W 2001.3 35k 02:00 39322.0 Oct 17 2016 21:22:03 UTC Oct 24 2016 15:26:57 UTC several days (at both 50W and 25 W) 2501.3 35k 05:00 39322.0 Oct 12 2016 03:20:41 UTC Oct 17 2016 21:22:03 UTC days @ 50 W 3001.3 35k 05:00 39322.0 Oct 06 2016 18:39:26 UTC Oct 12 2016 03:20:41 UTC days @ 50 W 3501.3 35k 05:00 39322.0 Jul 06 2016 18:56:13 UTC Oct 06 2016 18:39:26 UTC months @ 50 W 4001.3 40k 10:00 39322.0 Nov 12 2016 03:28:21 UTC Nov 16 2016 22:17:29 UTC days @ 30 W (see LHO aLOG 31546 for caveats) 4301.3 40k 10:00 39322.0 Nov 16 2016 22:17:29 UTC Nov 18 2016 17:08:49 UTC days @ 30 W 4501.3 40k 10:00 39322.0 Nov 18 2016 17:08:49 UTC Nov 20 2016 16:54:32 UTC days @ 30 W (see LHO aLOG 31610 for caveats) 4801.3 40k 10:00 39222.0 Nov 20 2016 16:54:32 UTC Nov 22 2016 23:56:06 UTC days @ 30 W 5001.3 40k 10:00 39222.0 Nov 22 2016 23:56:06 UTC
Before the HW injection test, we turned off this line (before entering observation intent). I turned it back on at Nov 23 2016 03:25 UTC, but this did not drop us out of observation intent.
This line was again turned off at 4:12 Nov 23 2016 UTC so that DetChar safety study can be made late tonight.
The analysis of sensing function at frequency above 1 kHz obtained from the roaming lines listed in the alog above is attached. These lines were run at different times than the low frequency sweep (below 1 kHz) taken on Nov 18 and included in this plot. So, the lines above 1 kHz will need to be compensated for the time varying parameters to make accurate comparison and has not been done for this plot.
One way of compensating the changes are by applying kappas calculated using the SLM Tool (or GDS). The other way of doing it is comparing each individual line with 1083.7 Hz (which is always on) line at time t (time at which each line is running) and time t0 (time of low freqeuncy sweep).
Sensing Function [ct] /[m] = (DARM_ERRR/TxPD) f= hf, t * (TxPD/DARM_ERR) f = 1083.7, t * (DARM_ERRR/TxPD) f= 1083.7, t0
Both methods are essentially same but I will use the second method and the plot with the correct compensation applied to come soon.
16:00 Kyle is at EX as reported by Nutsinee
16:02 Joe D out to LVEA to do his weeklies.
16:02 Richard and Gerardo to EX to gt RGAs started WP
16:13 Joe out and He , Bubba amd Chris are headed to EY
16:30 Krishsna out to floor to re-center CS BRS WP6355
16:36 Jason and Peter out to floor to turn of DBB WP6317
16:37 Hugh headed to Ends to do HEPI fluid checks
16:39 Vending machine maintenance on site - Coca Cola
16:46 Lockloss due to maintenance incursions. Guardian set to down.
16:48 Jason and Peter out
16:49 Kyle back
16:49 TJ trying some new SEI code on HAM6 for pringle mode WP6341
16:56 noticed Fast Shutter check request from DIAG_MAIN. Set LOCKLOSS_SHUTTER_CHECK to LOW_ARM_POWER.
17:01 Chandra in LVEA woking on Ion pump HAM11. Gerardo out to join her.
17:03 H0:VAC-MY_CP4_LT250_PUMP_LEVEL_PCT alarm
17:04 EY HEPI DIFF pressure trip while Hugh working on it. Telephone call had me manually adjust output tweak on pressure servo
17:11 Alfredo to CS and out buildings to reinstal electronics for infrasound mics WP6343
17:15 reset HEPI pump pressure servo at EY a second time. Untripping watchdogs.
17:30 LN2 load on site
17:35 Corner station tripped. We think it may be that Alfredo settting the infrasound mic down in the Biergarten close to the STS did it. Expectin the same from the end stations when he gets to them.
17:44 put FAST_SHUTTER node into manual and brought to DOWN. Returned to AUTO and ran INIT in ISC_LOCK.
17:45 untripping corner
18:08 Duat monitor 6 in LVEA - Yellow alarm. Looks like it reached a threshold of 7920 cts
18:11 Karen back from EY and headed into LVEA.
18:16 Kyle turning on his noise sources at BSC8
18:20 Guardian reboot WP6337
18:28 Richard and Alfredo going to check on RGA network connection and then to the vault
18:37 Re dust alarms monitor 10 - .3=74020 .5=27820
18:39 BS HEPI WD trip - reason uncertain
18:41 dust monitor 6 RED alarm - .3=11270
18:48 Paradise water delivery on site
19:10 Joe back from End and heading into LVEA to complete weekly tasks
19:21 Joe out of LVEA
19:29 PSL Dust alarm .3=1520 .5=730
19:43 TJ restarting HAM6 guardian node to revert back to original config.
20:02 restarted Ops workstation at Carlos's request
20:20 BRS back on at end stations
20:20 begin initial alignment
20:36 Green arms aligned and offloaded. Waiting for Sheila to do INPUT_ALIGN as she has a new filter to try.
20:38 resuming IA
20:39 Jeff and Cheryl have re-aligned IMs
20:46 Richard to EY to put anemometer on roof.
21:22 Begin main locking sequence
22:55 OMC node is repaired(it seems) DC readout achieved. a2l script started and the stopped due to Sheila noticing something fishy in DARM.
00:06 running a2l
00:07 handing off to Corey
J. Kissel, C. Vorvick This morning, during repair of an infrasound microphone in the PEM stay-clear zone the corner station sensor correction STS got an impulse. Unfortunately sensor correction to the corner-station SEI was not turned off, because it was unclear to those present that these two things would be interrelated. As such, this impulse was sent to the SEI systems and tripped a large fraction of their watchdogs (see LHO aLOG 31735). This includes the HAM2 ISI, on which the IMs are suspended. Whenever the HAM2 ISI trips or goes offline, the IMs all move to a new position, regardless of their requested alignment slider values. This is merely another instance of the on-going issue with the IMs, as reported in Integration Issue 4698 (note that as of this aLOG I've cleaned up the long and confusing digital paper trail associated with this bug.) Sadly, we must wait until WHENVENT HAM2 before we can resolve this issue. Here's how much the IMs moved: dP [urad] dY [urad] IM1 0.2 0.8 IM2 31 5 IM3 6 0.5 IM4 0.5 1 I've attached a 24-hour trend of the position to show this as a function of time. The black vertical line on each plot indicates the time of HAM2 ISI trip. Notes: - From Cheryl's experience, a "large" jump that we need to correct for is 1 [urad] and 5 [urad] for IM1 and IM2/IM3 respectively, thus I've bolded the values in the table that are a large jump. I put "large" in quotes, because to-date, these numbers have just been empirically determined from commissioning experience. Cheryl is working on an analytical quantitative method for determining what is a large move based on metrics of the input faraday like the alignment's impact on the IFI's extinction ratio, etc. - IM2 and IM3 are those SUS that typically move. IM4 never moves appreciably (so its typically not reported), and indeed that has been a defining bit of evidence in the study as to why IM2/IM3 move. We've restored their alignment to the most recent out of lock positions as reported by their M1_DAMP_P/Y_IN1_DQ channels (or whatever EPICs versions of those channels you prefer). These must be restored to an out of lock alignment, because IM4 is globally controlled for alignment to follow the IFO when locked, which is decidedly the wrong alignment for lock acquisition.
at 17:35 UTC today, work on the corner station infrasound mic, in the biergarten, close to the ITMY STS caused a corner station trip that included all BSC ISI and HEPI as well as all HAM ISIs (NOT HAM HEPIS). Some suspensions were tripped as well. It is believed that the heavy piece of equipment, replaced onto the ground in the PEM "stay clear" zone is what cause the spike.
Tied the shunt trip main breakers to the thermal protection and E-Stop button in the DAQ room. Tested that the system trips with either an over temperature condition or pushing the E-Stop Button.
WP 6344 Jim B, Dave B, TJ Reverted nds2-client software to version 0.12.2 for Ubuntu 12, Ubuntu 14, and Debian8 workstations. nds2-client-0.13.1 software was causing problems with a python getdata request for 30 seconds of data from 40 seconds ago, and/or multiple channels. We will characterize this issue a bit more after the holiday to file a bug report.
John Z., Aaron V. John restarted the primary GDS pipeline at GPS time 1163860820 +- several minutes, and I restarted the redundant pipeline at GPS time 1163887460. The latency was ~5-10 seconds and the CPU usage is ~70%, both usual. This restart picked up gstlal-calibration-1.0.8, which includes these changes: - Addition of option --kappas-default-to-median to replace rejected kappas with last good median instead of default - Bug fix to prevent/reduce latencies due to bad dataValid flags - Addition of option --demodulation-filter-settle-time to allow filter to settle before accepting computed kappas. A new filters file is also being used. See this aLOG for information on the filters: https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=31712
(see WP6332) Also running turbo pump connected to PT180 hardware on dome of BSC8. Coordinating with PEM folks to see if the IFO commissioning community can live with this noise source for a few days. If so, we (Vacuum) would then bake the PT180 hardware while being valved-out from the YBM and while being pumped by these temporary pumps. Will be leaving these run past the official "Maintenance Day" period so as to see what these look like if/when the IFO is in low-noise. May need to do some "on/off" tests as per Robert S.
~1654 hrs. local -> De-energized all pumps (and fan) @ BSC8. Will restart tomorrow morning.
Jeff K, Kiwamu I, Darkhan T,
Overview
The CAL-CS EPICS records for tracking temporal variations of the DARM parameters have been updated at 2016-11-21 22:03:46 UTC. These values are identical to the ones in LHO alog 31677 from yesterday, i.e. D20161120_H1_CAL_EPICS_VALUES.m
and D20161121_H1_CAL_EPICS_VALUES.m
are identical.
New values have been accepted in SDF_OVERVIEW.
Details
Following DARM paramemter files were used to calculate these values:
${CalSVN}/Runs/ER10/Common/params/IFOindepParams.conf r3752
${CalSVN}/Runs/ER10/H1/params/H1params.conf r3826
${CalSVN}/Runs/ER10/H1/params/2016-11-12/H1params_2016-11-12.conf r3786
And the DARM model scripts from
${CalSVN}/Runs/O2/DARMmodel/* r3814
The *.m file with EP1-9 values and the verbose output are attached to this report. All of the files have been committed to CalSVN at
ER10/H1/Scripts/CAL_EPICS/
D20161121_H1_CAL_EPICS_VALUES.m
20161121_H1_CAL_EPICS_VALUES.txt
20161121_H1_CAL_EPICS_verbose.log
Shivaraj K, Jeffrey K, Aaron V, Darkhan T,
Updated DARM time-dependence EPICS
We have updated the EPICS values used for DARM time-dependent parameter calculations (DCC T1500377). We added a previously (Nov 21) missing delay in the DARM_ERR signal due to one computational cycle delay, comes from the fact that the signal is transferred from the OMC front-end model to CAL-CS. The *.m file with EP1-9 values and the verbose log are attached to this alog.
The new values were updated in the front-end and were accepted in SDF_OVERVIEW.
Location in the SVN:
${CalSVN}/Runs/ER10/H1/Scripts/CAL_EPICS/
D20161122_H1_CAL_EPICS_VALUES.m
20161122_H1_CAL_EPICS_verbose.log
20161122_H1_CAL_EPICS_VALUES.txt
Corrections to be applied to the channels in GDS and SLM (CalMon) when calculating "kappas"
All of the necessary corrections, except for Pcal RX PD channel corrections have been incorporated in the EPICS values, i.e. when calculating kappas with the method from T1500377, except for the Pcal Rx PD the rest of the channels must be used without applying any additional corrections. The Pcal RX PD channels from the frames must be corrected for the freq. dep. part of the free mass response (two poles at 1 Hz), analog AI, digital AI (IOP) and the time delay of the CAL-EY channels w.r.t. [V] from the PD (this piece was measured to be zero). One of the ways to get the correction TF is to extract it from the Matlab DARM model, an example of extracing this correction TF for Hanford can be found at (the resulting TF is attached):
${CalSVN}/Runs/ER10/H1/Scripts/PCAL/examplePcalCorrExtraction.m
LLO analog AA/AI models have been updated for ER10/O2, while at LHO they did not change since the O1 run. So it is important to calculate the site-specific Pcal corrections using the most up-to-date DARM reference-time parameters.
Corrections to be applied to the CAL-CS demodulators
Since "kappa" calculations in CAL-CS are not done using the Pcal Rx PD channels written into frames, but rather with the PCALY_RX_PD signal that gets transmitted from the CALEY model to CAL-CS, the Pcal Rx PD data seen by CAL-CS has an additional one computational cycle delay. This means that the phases and amplitudes in the front-end demodulators must be adjusted with PcallCorr * (one_cycle_advance_phase).
Pcal calibration line demodulator phases have been adjusted in the CAL-CS front-end model.
PCAL_LINE1 (36.7 Hz line) demodulator phase was set to -1.87 deg (old value was -2.40 deg)
PCAL_LINE2 (331.9 Hz line) demodulator phase was set to -16.95 deg (old value was -19.9 deg)
The phases were calculated according to the instructions given above and taking into account that two poles at 1 Hz (free-mass response) are applied separately in a Foton filter.
The script for calculating the phases was committed to
${CalSVN}/Runs/ER10/H1/Scripts/PCAL/getPcalCorrForCALCSdemod.m
The new values have been accepted in SDF_OVERVIEW.
After updating the phases we have got about 25 minutes of data before we lost the lock. κPU, κT, κC, in this interval were within 1% of their reference-time values (1.0) and fC was within 3% of its reference-time value (346.7 Hz).
I compared both of Pcal correction factors for LHO used by the GDS pipeline to the transfer function produced by the example script. These are computed at two line frequencies: 36.7 Hz and 331.9 Hz. As seen in the plot, they agree quite precisely. I also checked that the numerical values produced by this script at the line frequencies were identical to those used in the GDS pipeline.
The pcalcorrection factor used for SLM Tool and computed by the exapmple script are in good agreement as well.