State of H1: Nominal Low Noise
Activities that will contnue:
I did a few more things tonight
Now that we have the DBB plugged in again, we can relook at the coherences of the QPDs. The screenshot shows coherences for DARM and SRCL with for the 200W beam (opening the 35W shutter unlocks the IFO).
We have small coherences with DARM just below 1kHz and around 4.2kHz, but not otherwise. SRCL does have more coherence with one of the QPDs below 100Hz.
I think that the nominal setting of the CAL-PINJX_TRANSIENT filter bank gain is supposed to be zero. When a transient injection is imminent, then the gain is ramped to 1.0, the injection happens, and then the gain is ramped to zero. However, the SDF safe set point is 1.0. Therefore, I am setting the gain to 0 and accepting the change in the SDF safe file.
The existing code in the INJ_TRANS Guardian script does indeed do this.
if guardian is controlling the gain, perhaps sdf shouldn't be monitoring it.
Jeff B, Cheryl, Travis and Kiwamu,
In the past two days, Jeff, Cheryl and Travis performed a random walk on the CO2 settings for me (30920 and comments therein). I don't see significant change so far. In fact, it might have deteriorated the jitter peaks slightly.
I now ask the operators to perform differential scans instead (e.g. raising only one CO2 at a time).
The motivation was to see if we can exert any kind of effects on the jitter peaks in 200-1000Hz by changing the CO2 settings. Because TCS measurements usually take a long time, I have asked the operators to do some random walk on the TCS settings when possible. So far we have done a common scan (i.e. raising both CO2 powers simultaneously) and I don't see big change in the jitter peaks in 200-1000 Hz, in particular the ones at 285, 365 and 620 Hz. The attached shows DARM spectra with different CO2 settings.
These curves correspond to the following time.
As you can see, the ambient noise (most of the time appears to be shot noise) varies depending on the time because some of them overlapped with the active injection tests by Robert. But, this is not something I am looking for. Among the 6 noise curves, the best jitter noise was obtained from 27/10/2016 9:40:00 UTC which is actually before the series of CO2 tests started. So it is possible that the common CO2 may have deteriorated the jitter peaks slightly. We should do a differential scan next.
J. Kissel, B. Weaver Betsy grabbed charge measurements yesterday. I've processed them. The charge is still right around 0 [V] effective bias -- we're ready for regular bias flipping.
Summary- the sensing sign in the online calibration for SRCL has been wrong.
This has been causing overestimated noise in 10 - 100 Hz in the past years(!). My bad. This is now fixed.
Details- Daniel and Stefan a week or two ago told me that changing the shape of the digital filter in SRCL affected the calibrated SRCL displacement spectrum. This statement made me suspect that something was wrong in the online calibration or aka CALCS. Today, I have re-measured the SRCL open loop gain in nominal low noise with an input power of 25 W. A plot of the open loop is attached in this etnry. The absolute value of the SRCL sensing was found to be the same. However, the measurement indicated that the sensing gain should be a negative number (dP/dL < 0 or smaller counts as the SRC length expands). This contradicted with what we have had in CALCS where the sensing was set to positive. This is very simillar to what we had in the online DARM calibration (29860), but this time SRCL has been wrong for years.
Flipping the sensing gain in CALCS (so as to match the sign with the measurement) decreased the noise level in the online monitor by a factor of 2 at 60 Hz in 10 - 100 Hz. You can see the difference below.
The cyan is before the sign flip in CALCS, and the green is after. In order to double check the validity, I produced the calibrated spectrum only using SRCL_IN1 (blue) which agreed with the online calibration. There is small discrepancies between SRCL_IN1 and CAL-CS by a few % which, I believe, is due to the fact that we don't take the time delay of the system into account in CAL-CS. The sign flip is now implemented by adding a minus sign in FM1 of CAL-CS_SUM_SRCL_ERR (which is now a scaler value of -9.55e-6). I did not change the absolute value.
Additionally, I looked at some calibration codes that I made some time ago (18742) and confirmed that I mistakenly canceled the minus sign in the sensing gain of the model. Also, according to the guardian code ISC_LOCK and trend data, the sign of the SRCL servo gain in LSC or the relevant filters in the SUS SRM did not change at least in this past year. I am fairly sure that this calibration has been wrong for quite some time.
The relevant items can be found at the following locations.
Open loop measurement: /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/PreER10/H1/Measurements/LscDrmi/SRCL_oltf_25W_20161028.xml
Open loop analysis code: /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/PreER10/H1/Scripts/LscDrmi/H1SRCL_OLTFmodel_20161028.m
Plots for open loop: /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/PreER10/H1/Results/LscDrmi/2016-10-28_SRCL_openloop.pdf
/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/PreER10/H1/Results/LscDrmi/2016-10-28_SRCL_openloop.png
J. Kissel With hints of improvement last night from Sheila (see LHO aLOG 30947) on reducing the most egregious ~41 Hz peak in DARM (which is known to be the HSTS roll modes, a.k.a R3), I launched a campaign of adding similar ~1 Hz wide notches to all HSTS suspensions' local damping loops (M1_DAMP) at both the highest roll mode and highest vertical modes (a.k.a V3 at ~27.5 Hz). The filter designs are V3 notch: FM8 "SB27.5" ellip("BandStop",4,1,60,27.05,28.05)gain(1.12202) [Input -- Always On; Output -- Ramp; Ramp Time -- 2 sec] R3 notch: FM9 "SB40.9" ellip("BandStop",4,1,60,40.4,41.3)gain(1.12202) [Input -- Always On; Output -- Ramp; Ramp Time -- 2 sec] where I've made sure that the lowest and highest frequency V3 modes (coincidentally both IMC mirrors: MC1 @ 27.38 Hz, MC2 @ 27.74 Hz) faLL within the stop band of the notch (and I've confirmed that this is true with Sheila's R3 notch design as well). Further, the V3 notch causes a phase loss of only 2.2 [deg] at 10 Hz, so it will not impact any of the damping loop's phase margins. To confirm, I've spot checked SR2's local damping open loop gain tfs just to be sure. Indeed all DOFs are still quite stable (and quite poorly tuned, as expected). During the campaign I found that PRM, MC1, and MC3's "ellip50" standard low-pass filters were not engaged, so I engaged them. I've greened up all ODC status lights, and then accepted all of the changes in the SDF system -- V3 filters ON, R3 filters ON, ellip50 filters ON, and Correct Damp State. While haven't had an uber-long lock stretch since I've turned on all of these notches, I was at least able to grab 5 averages of a 5 [mHz] BW ASD and compare against the long data set from last night. The notches appear to have done their job -- a large fraction of the modes have been squashed and don't show up in DARM anymore. Success! The SR2 damping loop open loop gain TF templates have been committed here: /ligo/svncommon/SusSVN/sus/trunk/HSTS/H1/SR2/SAGM1/Data/ 2016-10-28_2223_H1SUSSR2_M1_openloop_L_WhiteNoise.xml 2016-10-28_2223_H1SUSSR2_M1_openloop_P_WhiteNoise.xml 2016-10-28_2223_H1SUSSR2_M1_openloop_R_WhiteNoise.xml 2016-10-28_2223_H1SUSSR2_M1_openloop_T_WhiteNoise.xml 2016-10-28_2223_H1SUSSR2_M1_openloop_V_WhiteNoise.xml 2016-10-28_2223_H1SUSSR2_M1_openloop_Y_WhiteNoise.xml
Following up with another extremely long lock stretch from over the weekend (new data starts at 2016-10-31 07:00 UTC), it looks like I've definitely killed several of the resonances, such that they don't substantially appear in the DARM noise. However, the MC2 V3 mode @ 27.7642 Hz and what remains of the PR2 R3 mode @ 40.935 Hz are still as bad as they were before. This likely means these modes are getting excited via non-local control. The ISC control is a likely culprit for MC2, because the V3 mode is abnormally high and may be out of range of a generic notch. Further the IMC cross-over is around 15 Hz, so notching these frequencies is more difficult (i.e. there's currently no notches for the M3 stage length control of the IMC). We'll continue to poke around looking for poorly notched loops.
model restarts logged for Thu 27/Oct/2016
2016_10_27 10:30 h1psliss
2016_10_27 10:32 h1broadcast0
2016_10_27 10:32 h1dc0
2016_10_27 10:32 h1fw0
2016_10_27 10:32 h1fw1
2016_10_27 10:32 h1fw2
2016_10_27 10:32 h1nds0
2016_10_27 10:32 h1nds1
2016_10_27 10:32 h1tw0
2016_10_27 10:32 h1tw1
Daniel's ISS model change with associated DAQ restart.
model restarts logged for Wed 26/Oct/2016
2016_10_26 00:26 h1fw2
2016_10_26 06:36 h1fw2
2016_10_26 08:21 h1fw2
2016_10_26 12:14 h1fw2
2016_10_26 12:32 h1fw2
2016_10_26 12:38 h1fw2
2016_10_26 13:44 h1fw2
2016_10_26 16:45 h1susetmx
Jonathan's daqd work on fw2. Jeff restarted susetmx as part of BIO investigation.
model restarts logged for Tue 25/Oct/2016
2016_10_25 08:23 h1sushtts
2016_10_25 08:38 h1susmc1
2016_10_25 08:38 h1susmc3
2016_10_25 08:38 h1susprm
2016_10_25 08:39 h1suspr3
2016_10_25 08:40 h1susim
2016_10_25 08:48 h1susmc2
2016_10_25 08:48 h1suspr2
2016_10_25 08:48 h1sussr2
2016_10_25 08:57 h1sussrm
2016_10_25 08:59 h1susomc
2016_10_25 08:59 h1sussr3
2016_10_25 09:10 h1susbs
2016_10_25 09:10 h1susitmx
2016_10_25 09:10 h1susitmy
2016_10_25 09:18 h1susauxasc0
2016_10_25 09:20 h1susauxh2
2016_10_25 09:20 h1susauxh34
2016_10_25 09:20 h1susauxh56
2016_10_25 09:22 h1susauxb123
2016_10_25 09:33 h1susetmx
2016_10_25 09:35 h1sustmsx
2016_10_25 09:37 h1susetmy
2016_10_25 09:37 h1sustmsy
2016_10_25 09:40 h1susauxex
2016_10_25 09:41 h1susauxey
2016_10_25 10:23 h1alsey
2016_10_25 10:23 h1caley
2016_10_25 10:23 h1iopiscey
2016_10_25 10:23 h1iscey
2016_10_25 10:23 h1pemey
2016_10_25 10:28 h1broadcast0
2016_10_25 10:28 h1dc0
2016_10_25 10:28 h1fw0
2016_10_25 10:28 h1fw1
2016_10_25 10:28 h1fw2
2016_10_25 10:28 h1nds0
2016_10_25 10:28 h1nds1
2016_10_25 10:28 h1tw1
2016_10_25 11:58 h1iopiscex
2016_10_25 11:58 h1pemex
2016_10_25 12:00 h1alsex
2016_10_25 12:00 h1calex
2016_10_25 12:00 h1iscex
2016_10_25 12:02 h1fw2
2016_10_25 12:10 h1fw2
2016_10_25 12:27 h1fw2
2016_10_25 12:28 h1fw2
2016_10_25 12:32 h1fw2
2016_10_25 13:52 h1fw2
2016_10_25 13:56 h1fw2
Tuesday maintenance. All SUS DAQ configurations changed. Unexpected restarts of h1iscey and h1iscex, related to powering of test equipment. Jonathan's daqd testing on fw2.
TITLE: 10/28 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Aligning
INCOMING OPERATOR: Jim
SHIFT SUMMARY: Walked in to a locked IFO. It broke lock shortly thereafter, possibly due to PEM injections. After a bit of a prolonged IA, we have been back at NLN for the last half of the day.
LOG:
15:13 Robert to LVEA
15:15 Chris to MX
15:57 Chandra to EX
16:15 Chandra back
16:19 Kiwamu to LVEA for ISS OLTF
16:20 Betsy, Bubba to LVEA
16:36 Betsy, Bubba out
16:37 Kiwamu done
17:30 Adjusted fiber polarization for both arms
18:48 Richard down the Xarm BTE
21:26 Chandra to MY
2:30 pm local Took 24 sec. to overfill CP3 by doubling LLCV to 36% open. TC plot attached.
Evan G., Keita K. Summary: We analyzed the measurements made at the end station (see LHO aLOG 30854). The measured delay of a 960 Hz digital sine wave excitation to the output of the analog AI is 205.7 usec. The expected delay is 203.6 usec. The difference of 2.1 usec could be the result of small differences in analog AA/AI electronics. The measurements also reveal the DAC delay = 63 usec (61 usec expected, again probably the difference is the small variability of the AA/AI electronics), ADC delay = approx. 1 usec (<1 usec expected). Additionally, digital excitations are found to be advanced by 61 usec. This is important because we need to independently measure the the timing of the analog signal relative to the witness GPS 1 PPS signal to determine the timing of the ADC. We also independently measure the timing of the DAC relative to the witness GPS 1 PPS signal by using the digital system to produce an excitation that is measured in the analog world. Details: The oscilloscope measurements are analyzed using the duotoneDelay.m script which directly computes the Fourier coefficients of a DuoTone time series with two sine wave signals at 960 and 961 Hz. The Fourier coefficients yield the amplitude and phase, where the phase delay is interpreted as a time delay. The time series input is synchronized to the 1 PPS signal from the witness GPS receiver located at each end station (time series is triggered on the rising edge of the 1 PPS). The oscilloscope is set to produce 1e6 samples per trace. We saved two different duration traces: 1 sec (1e6 samples/sec) and 2 sec (5e5 samples/sec). Oscilloscope ------------ 1 PPS --| CH1 | ==> Time series with pulse leading edge = 0 sec | | Pcal/DuoTone --| CH2 | ==> Fourier coefficient of time series with timestamps referenced to 1 PPS yields delay ------------ Analog measurement Delay (usec) ---------------------------------------- Pcal X excitation (1 sec) 145.2 Pcal X excitation (2 sec) 144.1 Pcal Y excitation (1 sec) 144.4 Pcal Y excitation (2 sec) 144.9 DuoTone X (1 sec) 257.9 DuoTone X (2 sec) 258.0 DuoTone Y (1 sec) 257.2 DuoTone Y (2 sec) 256.6 We saved a 2-second DTT time series of H1:CAL-PCAL*_DAC_FILT_DTONE_IN1_DQ and H1:CAL-PCAL*_EXC_SUM_DQ (where * = X or Y). The time series starts on an integer second. Using the same duotoneDelay.m script, we compute the delay of the digital Pcal excitation signals and the digital DuoTone signal. Digital measurement Delay (usec) ------------------------------------------------ H1:CAL_PCALX_EXC_SUM_DQ -61.0 H1:CAL_PCALY_EXC_SUM_DQ -61.0 H1:CAL-PCALX_DAC_FILT_DTONE_IN1_DQ 340.4 H1:CAL-PCALY_DAC_FILT_DTONE_IN1_DQ 339.8 (Note the 61 usec advance of the excitation signal) To determine the analog output delay of an excitation, we subtract the measured delay of the digital Pcal excitation from the measured analog delay and compute the mean value, yielding a delay of 205.7 usec. We expect a delay of 203.6 usec (61 usec delay from USER model to IOP model, 43.5 usec from phase effect in digital AI, 3 IOP DAC FIFO cycles, 0.5 IOP cycles from DAC zero-order-hold, 0.5 IOP cycles from DAC clock offset, and approx. 38 usec from phase effect in analog AI). The difference of 2.1 usec is likely due small differences in the analog AI filters. To determine the DAC delay, we can remove the analog and digital AI filtering, and the 61 usec delay from USER to IOP models. This yields a delay of 63 usec, compared to the expectation of 61 usec. The difference is, again, likely due to small differences in the analog AI filtering. The ADC delay can be determined using the difference between the measured delay of the DTONE_IN1_DQ channels and the delay measured in the analog DuoTone signals. We also need to remove the analog and digital AA filtering. This yields an ADC delay of approx 1 usec (we expect a delay of <1 usec). Small differences in the analog AA filtering may be the reason for this difference. Analysis script can be found at: /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/PreER10/H1/Scripts/timing/analyzeTimingMeasurements.m
Summary: new ISS 2nd loop gain = 17 dB.
Since we have changed the amount of light on the inner loop PD (PDA, alog30871) and also decided to lock with an input light power of 25 W, these let us re-tune the gain of the ISS second loop. The goal of this adjustment is to maintain the intended UGF in the second loop. This morning, I have measured the open loop and adjusted the gain. The measurements were done with the IMC locked at 25 W and the second loop locked with the DC-coupling enabled. The inner loop was closed with a gain of 18 dB and a diffracted power of 4% as usual (30871).
The attached image shows the measured open loop transfer function of the ISS second loop. The UGF was tuned to 19 kHz (which used to be 18 kHz back in this September alog29915) where the phase margin is almost the same value of 45 deg. The gain margin is found to be 7 dB. Everything seems good. Additionally, the raw data files are attached.
The PR2 Roll mode has been reduced in the DARM spectrum by adding 1Hz wide bandstops (centered at 40.9Hz) to all the top mass damping loops for PR2.
Once these were on the line in DARM at 40.934Hz is gone. There are other HSTS roll modes which now show up, 40.918 (MC2 or SR2 maybe) and something smaller at about 40.937Hz.
For the record, Sheila saved the modified filter file on Oct 27 2016 at 16:55 PDT, or Oct 27 2016 23:55:00 UTC. She says she hit load on the filter banks individually "pretty much right after."
I've now subsequently loaded the entire filter file from the GDS_TP screen on Fri Oct 28 2016 at 10:38:53 PDT (Oct 28 2016 17:38:53 UTC)
The filter design string is ellip("BandStop",4,1,60,40.4,41.3)gain(1.12202) and lives in FM9.
Also, I attach a comparison between calibrated DARM ASDs from last night's long lock stretch against one from several nights ago. The BW is a delightful 0.3 [mHz] -- hooray for IFO stability! The conclusion is a little different from Sheila's above conclusion: the PR2 R3 mode @ 40.93 Hz is not *gone*, but it has definitely been reduced by an order of magnitude. Interestingly, the 27.41 [Hz] V3 mode also is a reduced by a factor of 0.70.
Indeed, there are other R3 modes now exposed at 40.8687, 40.8775, 40.9171, in addition to the residual of the PR2 mode at 40.9356 [Hz]. That being said, it looks like given the density of modes around 40.9, it'll be fine to use the same notch design for all HSTS.
We'll successively notch the R3 mode out of every HSTS, and see what disappears.
By looking at my 2014 comparison between LLO, LHO, and a Low Noise damping loop design, G1401291, at first glance since the filter only distorts the phase at 20 Hz by 2 [deg], we should have plenty phase margin to notch out this mode in every suspension. But do not that LHO still has a complete mess of a design -- pretty much all the SUS have different gains and filter designs. So, we'll do so with caution. Even more so if we want to notch the V3 mode.
Kiwamu saked the ops to run some TCS laser noise measurements.
SETUP:
Started the run:
TSC X - Initial Power = 0.2W TSC Y - Initial Power = 0.0W
| Time | Power | Time | Power | |
| 03:00 | 0.3W | 03:00 | 0.1W | |
| 04:30 | 0.4W | 04:30 | 0.2W | |
At 05:08 Lost lock due to a Mag5.8 EQ in Alaska.
I only managed to get one more data point for both arms:
15:30 utc: TCSx at 0.5W for 90 minutes.
TCSy at 0.3W for 90 minutes.
Oct 28, 10:03UTC, TCSX power set to 0.6, TCSY power set to .4
oct 28, 11:32UTC, change X from 0.6 to 0.7, changed y from 0.3 to 0.4
oct 28, 13:27UTC, tcsx raised to 0.8, tcsy raised to 0.5
As range dropped and arm signals got more noisy I feared H1 was about to lose lock, and touched up TMSX and TMSY alignment. Hopefully this didn't invalidate the data for TCS analysis.
Tweaks by TMS:
Tweaks and TCS changes by timeline:
15:05 UTC: TCSx at 0.9W for 45 min.
TCSy at 0.6W for 45 min.
At 4:43 UTC today (10/30, or (10/29 still PST)), after Robert left, I changed TCSY to .2W, per Cheryl's suggestion left with Corey. TCSX is still at .4W.
I got suspicious about PMC length locking offset and changed H1:PSL-PMC_INOFFSET.
Increasing it by 1.7mV decreased the PMC length feedback by about a factor of 2, and 1st loop out of loop sensor by a factor of 4 or so, which doesn't make sense. In the attached, red and green are with nominal 3.1mV offset, blue and brown are with 4.8mV.
(After this measurement I noticed that Daniel increased the length gain from 16 to 28dB and forgot to bring it back. This measurement is with 28dB locking gain, but it still doesn't make sense.)
What's the nominal signal level for PMC demod? Is it tiny? When is the last time the PMC demod phase was optimized?
More strange stuff, when we look at the PDA and PDB photodetectors of the first loop in the ISS. In the attached plot, the current traces are with a PMC offset of 3.28mV, reference traces 1-15 are with a 3.58mV offset and reference traces 16-19 are with a 2.98mV offset. With a positive offset change we see a some degradation in PDA at high frequencies, whereas PDB sees significantly less noise. For a negative offset PDA gets a tad bit better and PDB gets worse. Overall PDB shows up to an order of magnitude change in its noise level, whereas PDA only shows up to a factor of 2, going the opposite way. The PMC gain was high and 28dB.
Here is the throughput as function of the offset with a Lorentzian as a fit. The parameters are 0.761, 3.28mV and 4.59mV for the amplitude, offset and HWHM, respectively. Looks like the demodulated signal is only ~9mV pk-pk.
(Keita writing as Sheila)
For those of you who are interested, Daniel's measurement doesn't mean that the noise behavior (in length locking and in intensity noise) makes sense.
(Now writing as myself.)
According to T0900577 (select ilspmc_servo3.pdf) the output of TUF-3 mixer is amplified by a DC gain of 4 and sent to a summation amplifier that has a gain of 10 for the demod and a gain of 1/100 for the offset.
The offset signal seems to be calibrated to represent the offset in the OUTPUT of the summation amplifier (i.e. +-100mV when the offset from DAC is +-10V). Update: I was deceived by HOPR and LOPR of he signal on MEDM being 100 and -100, but the calibration filter of this channel of this gain is just 3.2k, so the number should represent the equivalent offset after the gain of 4 but before the gain of 10.
So this 9mVpp is after the gain of 40 total, the demod right after the mixer should be ~9mV/4/10=230uVpp.
Update: The demod right after the mixer should be ~9mV/4=2.3mVpp.
If this is true this is excessively small and cannot be good, and I wonder if the demod phase is correct or if this is an expected signal level. If this is as designed, can't we increase the modulation depth upstream or something?
(The main document in the above DCC is so-called PDF Portfolio, which is just a document containing all pdfs listed in "other files". If you're on Linux workstations the pdf in the above DCC appears as if it's just a one-page document promoting Adobe product, but if you're using evince document viewer, change "thumbnails" on the left panel to "attachments", and you can select whichever file in the portfolio to view).
Looking at the RF chain:
Therefore, the drive to the modulator seems to be -10 dBm, or 71 mV rms. A standard New Focus 4004 EOM has a modulation coefficient of 25 mrad/V. So the estimated modulation depth is around 1 mrad.
The mixer readbacks are flawed and just see ADC noise. They could use a gain of 200 to get above the ADC noise. Proposed values:
We recorded AI output of PCAL injection as well as PCAL_DAC_FILT thing together with witness GPS 1pps using Tektronics MSO4043 (1 sec with 1Msample per channel, and 2 sec with 1Msample per channel).
We also recorded the digital output of the PCAL injection as well as digital input of the PCAL_DAC_FILT thing.
Evan will look at the timing comparator data.
These will be analyzed in the near future for further timing sanity check.
Data is stored at /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/PreER10/H1/Measurements/timing. Analysis forthcoming...
Analysis complete. See LHO aLOG 30965.
The results of cdsutils.avg() in guardian is sometimes giving us very weird values.
We use this function to measure the offset value of the trans QPDs in Prep_TR_CARM. At one point, the result of the average gave the same (wrong) value for both the X and Y QPDs, to within 9 decimal places (right side of screenshot, about halfway down). Obviously this isn't right, but the fact that the values are identical will hopefully help track down what happened.
The next lock, it correctly got a value for the TransX (left side of screenshot, about halfway down), but didn't write a value for the TransY QPD, which indicates that it was trying to write the exact same value that was already there (epics writes aren't logged if they don't change the value).
So, why did 3 different cdsutils averages all return a value of 751.242126465?
This isn't the first time that this has happened. Stefan recalls at least one time from over the weekend, and I know Cheryl and I found this sometime last week.
This is definitely a very strange behavior. I have no idea why that would happen.
As with most things guardian, it's good to try to get independent verification of the effect. If you make the same cdsutils avg calls from the command line do you get similarly strange results? Could the NDS server be getting into a weird state?
On the one hand, it works just fine right now in a guardian shell. On the other hand, it also worked fine for the latest acquisition. So, no conclusion at this time.
This happened again, but this time the numbers were not identical. I have added a check to the Prep_TR_CARM state that if the absolute value of the offsets are larger than 5 (normally they're around 0.2 and 0.3, and the bad values have all been above several hundred) then notify and don't move on.
Operators: If you see the notification Check Trans QPD offsets! then look at H1:LSC-TR_X_QPD_B_SUM_OFFSET and H1:LSC-TR_Y_QPD_B_SUM_OFFSET. If you do an ezca read on that number and it's giant, you can "cheat" and try +0.3 for X, and +0.2 for Y, then go back to trying to find IR.
This happened again to Jim, and Cheryl, today and caused multiple locklosses
I've commented out the averaging of the offsets in the guardian.
We used to not do this averaging, and jsut rely on the dark offsets not to change. Maybe we could go back to that.
For operators, until this is fixed you might need to set these by hand:
If you are having trouble with FIND IR, this is something to check. From the LSC overview sceen, click on the yellow TRX_A_LF TRY_A_LF button toward the middle oc the left part of the screen. Then click on the R INput button circled in the attachment, and from there check that both the X and Y arm QPD SUMs have reasonable offsets. (If there is not IR in the arms, the offset should be about -1*INMON)
Opened as high priority fault in FRS:
Ed, Sheila
Are ezca connection errors becoming more frequent? Ed has had two in the last hour or so, one of which contributed to a lockloss (ISC_DRMI).
The first one was from ISC_LOCK, the screenshot is attached.
Happened again but for a different channel H1:SUS-ITMX_L2_DAMP_MODE2_RMSLP_LOG10_OUTMON ( Sheila's post was for H1:LSC-PD_DOF_MTRX_7_4). I trended and found data for both of those channels at the connection error times, and during the second error I could also caget the channel while ISC_LOCK still could not connect. I'll keep trying to dig and see what I find.
Relevant ISC_LOCK log:
2016-10-25_00:25:57.034950Z ISC_LOCK [COIL_DRIVERS.enter]
2016-10-25_00:26:09.444680Z Traceback (most recent call last):
2016-10-25_00:26:09.444730Z File "_ctypes/callbacks.c", line 314, in 'calling callback function'
2016-10-25_00:26:12.128960Z ISC_LOCK [COIL_DRIVERS.main] USERMSG 0: EZCA CONNECTION ERROR: Could not connect to channel (timeout=2s): H1:SUS-ITMX_L2_DAMP_MODE2_RMSLP_LOG10_OUTMON
2016-10-25_00:26:12.129190Z File "/ligo/apps/linux-x86_64/epics-3.14.12.2_long-ubuntu12/pyext/pyepics/lib/python2.6/site-packages/epics/ca.py", line 465, in _onConnectionEvent
2016-10-25_00:26:12.131850Z if int(ichid) == int(args.chid):
2016-10-25_00:26:12.132700Z TypeError: int() argument must be a string or a number, not 'NoneType'
2016-10-25_00:26:12.162700Z ISC_LOCK EZCA CONNECTION ERROR. attempting to reestablish...
2016-10-25_00:26:12.175240Z ISC_LOCK CERROR: State method raised an EzcaConnectionError exception.
2016-10-25_00:26:12.175450Z ISC_LOCK CERROR: Current state method will be rerun until the connection error clears.
2016-10-25_00:26:12.175630Z ISC_LOCK CERROR: If CERROR does not clear, try setting OP:STOP to kill worker, followed by OP:EXEC to resume.
It happened again just now.
Opened FRS on this, marked a high priority fault.
J. Kissel, for the Calibration Team I've updated the results from LHO aLOG 21825 and G1501223 with an ASD from the current lock stretch, such that I could display the computed time dependent correction factors, which have recently been cleared of systematics (LHO aLOG 22056), sign errors (LHO aLOG 21601), and bugs yesterday (22090). I'm happy to say, that not only does the ASD *without* time dependent corrections still fall happily within the required 10%, but if one eye-balls the time-dependent corrections and how they would be applied at each of the respective calibration line frequencies, they make sense. To look at all relevant plots (probably only interesting to calibrators and their reviewers), look at the first pdf attachment. The second and third .pdfs are the money plots, and the text files are a raw ascii dump of respective curves so you can plot them however or whereever you like. All of these files are identical to what is in G1501223. This analysis and plots have been made by /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O1/H1/Scripts/produceofficialstrainasds_O1.m which has been committed to the svn.
Apparently, this script has been moved to a slightly different location. The script can be found at
/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O1/H1/Scripts/DARMASDs/produceofficialstrainasds_O1.m