Another event at 14:02 PST , proc status file for IOP model posted below.
If this is a timing error (and at this point we are not certain of this), our first change was to replace the fiber optics cable between the IO Chassis timing slave and the timing fanout chassis. A new cable was installed before the system was restarted.
Note, the existing fiber was found to be not cable tied to the bundle of fibers at the timing fan-out card, rather was floating by itself.
Sequence was:
* we did not do the TCS-AI power sequence this time and unfortunately this may have tripped the TCS chillers.
controls@h1oaf0 ~ 130$ cat /proc/h1iopoaf0/status
startGpsTime=1162838354
uptime=12257
cpuTimeEverMax=8
cpuTimeEverMaxWhen=1162838680
adcHoldTime=16
adcHoldTimeEverMax=91
adcHoldTimeEverMaxWhen=1162850544
adcHoldTimeMax=17
adcHoldTimeMin=13
adcHoldTimeAvg=14
usrTime=2
usrHoldTime=3
cycle=49476
gps=1162850611
buildDate=Nov 10 2016 08:33:32
cpuTimeMax(cur,past sec)=3,5
cpuTimeMaxCycle(cur,past sec)=21,21
cycleHist: 3=65183@17 4=350@65535 5=3@1
DAC #0 18-bit buf_size=40
DAC #1 16-bit fifo_status=0 (OK)
ADC #0 read time MAX=91 Current=14
ADC #1 read time MAX=0 Current=0
ADC #2 read time MAX=0 Current=0
ADC #3 read time MAX=0 Current=0
ADC #4 read time MAX=0 Current=0
ADC #5 read time MAX=0 Current=0
press DIAG_RESET
controls@h1oaf0 ~ 0$ cat /proc/h1iopoaf0/status
startGpsTime=1162838354
uptime=12333
cpuTimeEverMax=8
cpuTimeEverMaxWhen=1162838680
adcHoldTime=14
adcHoldTimeEverMax=91
adcHoldTimeEverMaxWhen=1162850544
adcHoldTimeMax=17
adcHoldTimeMin=13
adcHoldTimeAvg=14
usrTime=2
usrHoldTime=4
cycle=43808
gps=1162850687
buildDate=Nov 10 2016 08:33:32
cpuTimeMax(cur,past sec)=3,6
cpuTimeMaxCycle(cur,past sec)=21,1
cycleHist: 3=65201@17 4=333@65535 5=1@0 6=1@1
DAC #0 18-bit buf_size=40
DAC #1 16-bit fifo_status=0 (OK)
ADC #0 read time MAX=17 Current=15
ADC #1 read time MAX=0 Current=0
ADC #2 read time MAX=0 Current=0
ADC #3 read time MAX=0 Current=0
ADC #4 read time MAX=0 Current=0
ADC #5 read time MAX=0 Current=0
Here's the report of a BruCo scan for last night's lock
https://ldas-jobs.ligo.caltech.edu/~gabriele.vajente/bruco_lho_1162821617/
Some interesting results:
A question for commissioners, should I exclude channels like SUS-ETMY_L2_FASTIMON_* ??
Correlation between HPI-HAM1 and the REFL mirror RM2 has been seen before - See series of logs on Oct 24. https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=30790 Note that HPI-HAM1 is different than all the other HPI-HAMs (2-6) because HAM1 is the only chamber without an ISI.
at 10:30 PST the h1iopoaf0 model detected an ADC/DAC error and stopped driving the DAC outputs. Before restarting the models I pressed the DIAG_RESET button to verify the ADC error in the STATE_WORD cleared (it did) and the DAC error was latched on (it was). We also did a quick data dump of h1iopoaf0's proc status file and dmesg.
Here is the /proc/h1iopoaf0/status file before the restart
root@h1oaf0 /proc/h1iopoaf0 0# cat /proc/h1iopoaf0/status
startGpsTime=1162832563
uptime=5511
cpuTimeEverMax=8
cpuTimeEverMaxWhen=1162832990
adcHoldTime=15
adcHoldTimeEverMax=90
adcHoldTimeEverMaxWhen=1162837830
adcHoldTimeMax=17
adcHoldTimeMin=13
adcHoldTimeAvg=14
usrTime=2
usrHoldTime=3
cycle=25821
gps=1162838074
buildDate=Nov 10 2016 08:33:32
cpuTimeMax(cur,past sec)=3,5
cpuTimeMaxCycle(cur,past sec)=21,0
cycleHist: 3=65068@17 4=466@65535 5=2@1
DAC #0 18-bit buf_size=40
DAC #1 16-bit fifo_status=0 (OK)
ADC #0 read time MAX=17 Current=14
ADC #1 read time MAX=0 Current=0
ADC #2 read time MAX=0 Current=0
ADC #3 read time MAX=0 Current=0
ADC #4 read time MAX=0 Current=0
ADC #5 read time MAX=0 Current=0
The adcHoldTimeEverMax looks very high (we see it on other systems in the 60-70 range but not this high). The adcHoldTimeEverMaxWhen=1162837830 equates to Nov 10 2016 10:30:13 PST which is the time of the event.
On other systems running with 16bit DACs the FIFO_STATUS is always 2. After h1iopoaf0 was restarted, its status is now 2. Looking at the manual, it suggests
2 | FIFO in Low Quarter |
0 | FIFO in 2nd or 3rd Quarter |
I've setup monitors to report on ADC and FIFO status if this happens again.
This is a continuity of 31381.
For a bookkeeping purpose, I have extended the spot position thing to PRM and MC mirrors. Here is a summary. They all seem fine as expected.
p2l or y2l gain | alpha | spot position from the center [mm] | previous spot position record [mm] (18106, Apr 2015) | |
PRM PIT | +1.23 | +0.059 | -2.5 | N/A |
PRM YAW | + 1.05 | -0.050 | +2.1 | N/A |
MC1 PIT | -1.3 | -0.020 | +0.84 | +4 |
MC1 YAW | +1.6 | -0.076 | +3.2 | +1.9 |
MC2 PIT | -3.8 | -0.18 | +7.6 | N/A |
MC2 YAW | -0.4 | +0.019 | -0.80 | N/A |
MC3 PIT | -1.1 | -0.053 | +2.2 | -3.7 |
MC3 YAW | -2.6 | +0.12 | -5.0 | -2.4 |
The measurement was done with the interferometer fully locked at 25 W in nominal low noise. The measurement precision should be something like 10% or so.
I repeated this for the IMC when the IFO was unlocked, 2W into the vacuum, to see if things are drastically different hot vs. cold. Conclusion: the spots in the IMC are basically the same hot vs. cold, so this probably isn't what is changing our alignment (or something) and giving us the drift down in range as we thermalize after increasing the power into the vacuum.
P2L or Y2L gain | alpha for UR coil | spot position cold [mm] | spot position hot (from 31392) [mm] | abs(diff) hot vs. cold [mm] | |
MC1 P | -1.2 | -0.057 |
+2.4 |
+2.6 (mistyped there as +0.84) | 0.2 |
MC1 Y | +1.5 | 0.072 |
+3.0 |
+3.2 | 0.2 |
MC2 P | -3.79 | -0.18 |
+7.6 |
+7.6 | 0 |
MC2 Y | -0.42 | -0.02 |
-0.8 |
-0.80 | 0 |
MC3 P | -1.15 | -0.055 |
+2.3 |
+2.2 | 0.1 |
MC3 Y | -2.6 | -0.12 |
-5.1 |
-5.0 | 0.1 |
Some notes on the method, basically transcribing conversations with Kiwamu:
Since the ADS system does not actuate on the IMC mirrors, nor does it read in IMC_MCL, this was done by hand (also by hand by Kiwamu yesterday). I put a 100 count dither line at 20.125 Hz into H1:SUS-MC[1, 2, 3]_M3_DITHER_[P, Y]_EXC using awggui and minimized the appearance of that line in H1:IMC-MCL_OUT_DQ.
WP 6287 Removed the ADC card that was added on Tuesday (WP 6287) because of instability of the h1oaf IOP model. Several instances of ADC/DAC/DK bad status have been noted, the most recent at 06:53 PST this morning. We will put the ADC card in the DTS for long term testing. While I had the I/O chassis open, I checked for loose connections or anything else that might cause problems. I noted that about half the interface cards at the rear of the I/O chassis were loose and able to wiggle back and forth. I pushed them all down and tightened the screws on the brackets which were loose. I then looked at the ADC and DAC cards. All the metal brackets were tight, but the last ADC card was not secured to it's metal bracket. One screw was near falling out, the other was loose (picture attached), so the card was free to wiggle around in it's card connector. This was tightened and reinstalled. Not all of the cables plugged in to the back of the I/O chassis are able to be secured to their connectors. I pushed on all of them to make sure they were seated. I checked both the fans after power-up, they are both operational. The h1iopoaf0 model was changed back to it's previous version to reflect the lesser number of ADC cards, and the DAQ was restarted.
The IFO has been in NLN for a while now, but I couldn't take the IFO to OBSERVE because the LASER_PWR guardian had the nominal state set wrong and it took a while to figure out how to change it.
For other operators while we are still in a semi-commissioning phase, I fixed this by editing the LASER_PWR guardian. In the LASER_PWR there is a line that says:
nominal = 'POWER_{}W'.format(REQUEST_POWERS[X])
where X is some negative number. REQUEST_POWER is a vector defined above this line:
REQUEST_POWERS = [DEFAULT_POWER,10,25,35,50]
X is the index of the element that corresponds to the power, counting from the right (i.e for 50 W, X = -1, for 35 W X= -2...).
I had to change:
nominal = 'POWER_{}W'.format(REQUEST_POWERS[-2]) for 35 W to
nominal = 'POWER_{}W'.format(REQUEST_POWERS[-3]) for 25 W.
Save, then hit load on LASER_PWR.
The number in square brackets in e.g. "REQUEST_POWERS[-2]" is the index of the element in the REQUEST_POWER list. Negative numbers could from the right while positive numbers count from the left. So for:
REQUEST_POWERS = [DEFAULT_POWER,10,25,35,50]
then:
REQUEST_POWERS[-2] = REQUEST_POWERS[3] = 35
So you could also just specify the power level directly, e.g.:
nominal = 'POWER_{}W'.format(35)
Daniel, Sheila
We have not been using REFL WFS to control PR3 in the last week or so because with somewhat high microseism this was making the IFO unstable. Things got slightly better when we lowered the IMC WFS gain to avoid oscillating there.
Tonight we started to look at the REFL centering loops, and re did them. FIrst we started with diagonalizing them, the new matrix is in the screen shot. Then we reshaped the loops a bit. Before these changes the pitch loops were pretty close to unstable at 2.5 Hz with the ELF20s engaged, so we moved up the cut offs and moved the zeros down. We have had trouble occasionally with CHARD loops ringing at 2.5 Hz, I hope this will be fixed now.
Since the microseism is down to around 0,4um/second, we tried the refl WFS again but lost the lock.
The 260Hz peak was gone earlier tonight with an offset of 120 in DOF1P
spectra and rms
The results of the recent A2L measurements for PR2 and PR3 are analyzed. Here is a summary.
(coarsely) averaged Y2L or P2L gain [cnts] | alpha at UL osem | spot position from the center of optic | |
PR3 PIT | + 0.85 | + 0.088 | 4.61 mm too low |
PR3 YAW | + 1.8 | -0.19 | 18 mm toward West |
PR2 PIT | 0 | 0 | 0 |
PR2 YAW | -9.5 | 0.45 | 19 mm toward West |
Synopsys
The horizontal spot position on PR2 may sound too large, but I don't think this is bad.
Back in 2014 we adjusted the PR2 spot to be at 14 mm toward the West (14788) in order to improve the clipping situation on PR2 based on the study that Keita did (14640). Since the beam radius on PR2 is about 6 mm (see for example 30130), the spot position seems to have shifted by roughly the radius of the beam size further to the West comparing to the 2014 measurement. Note that the size of the PR2 optic is 150 mm in diameter (D1101377). According to these numbers, I don't think the measured spot position is too crazy.
Some details
Over the past week or so, the operators ran the a2l scripts for PR2 and PR3 (for example 31239). The scripts were the ones Jenne newly wrote specifically for checking the beam spot on these mirrors (31157). The attached shows trend of the a2l gains for the past two weeks. The values are then roughly averaged and used to estimate the typical spot positions as listed above. For the definition of alpha, see Koji's log entry in the 40m elog. For the conversion from alpha to the spot positions, see 14788.
There are some jittery peaks that may be doughnut jitter. How much are we going to gain if we squash them?
As a sensor for jittery things I chose IMC WFSA DC YAW. (Not because we believe that it's an angulay jitter, but because the coherence with DARM is high and therefore whatever polluts DARM is likely coming from the same place as the angular jitter measured by IMC WFSA DC YAW.)
Using DTT, I took 100 averages of CAL_DELTAL_EXTERNAL and IMC WFSA DC YAW from the night before (Nov/08 10:00:00 UTC) when there was this fabulously steady BNS inspiral range. As you can see from the first plot, IMC WFS is coherent with CAL_DELTAL_EXTERNAL.
I used the part where coherence is greater than 0.2 and did a dirty subtraction of the WFS signal from CAL_DELTA_EXTERNAL using a transfer function and the spectra measured. 60Hz and harmonics are excluded from the subtraction. The top panel shows both the raw and the post-subtraction spectrum.
The displacement was then divided by 4000 and was plugged into BNS inspiral range code (BNS_range.m) in calibration svn. Note that the range produced this way disagrees with the sensemon by about 10-% (my number overestimates), but it should still be useful for noise subtraction comparison.
In the second attachment, bottom shows the sqrt of the integrand (i.e. to obtain Mpc, you square the plot, add everything together, then sqrt).
The effect of the subtraction is small but not totally negligible, it seems like we'll gain something like 1.5 Mpc (or 1.3Mpc if you believe that the overestimation of the inspiral range is just a scale difference).
4pm local [Gerardo, Chandra] Took 3 min. to overfill CP3 by increasing LLCV to 50% open (from 20%).
The attached plots show a length noise injection into the PMC using TFIN. The first plot is calibrated in Volts for the HV_MON, whereas the second plot is calibrated in displacement. The coupling coefficient at 1 kHz is about 3.5 x 10-8 between the PZT and DARM length.
The reference traces 0 and 1 are without injection, whereas reference traces 2, 3 and 4 are from before the changes on the PSL table on Tuesday.
A measurement done at the very beginning of the run had about half the coupling—indicating that the coupling depends on thermal lensing.
The attached plot shows frequency and intensity noise measured by the IMC/REFL and second loop ISS sensors.
There is coherence above 1 kHz with intensity noise. Using the intensity noise coupling TF from alogs 30274 or 29926, one can conclude that it doesn't couple through intensity noise. The intensity noise is at 10-8 RIN/rtHz at 1 kHz. With a coupling of 10-13 m/RIN, we are far below the DARM noise.
The frequency noise produced by the PMC length noise is of the order of 1 Hz/rtHz at 1 kHz. It will be suppressed by the FSS, IMC and REFL. As such it is much too small to be responsible for the coupling directly. There is however, coherence with frequency noise as seen by REFL_A_RF9_I in the 1 kHz region. Assuming the REFL control signal is dominated by IMC sensing noise which seems to be around 2 x 10-4 Hz/rtHz at 1 kHz for 25 W input, see alog 31138, and using the noise coupling from alog 31176, we get a number around 10-19 m/rtHz at 1 kHz. So, it is conceivable that the PMC noise produces error point offsets in the REFL servo which in turn propagate to DARM as frequency noise.
We saw an increase of the PMC length noise coupling to DARM, when we misaligned SRM by 20 µrad. According T0900142 the requirement for SRM is 2.5 µrad which puts the required 3x10-6 /rtHz at 1 kHz with a safety factor of 10. So, for 20 µrad, a jitter of 7x10-6/rtHz at 1kHz is at the DARM noise level at full sensitivity. Our jitter was roughly 5x10-6/rtHz and it made a difference at the 1x10-19 level. Maybe somewhat higher than expected but close.
Looking at the IMC WFS signal we can clearly see the PMC length noise injection. If we take the 260 Hz periscope peak as a reference, this doesn't explain the coupling to DARM. What is surprising is that even without length noise injection (REF traces), the coherence between IMC WFS and PMC HVMon is large at frequencies between the jitter peaks.
This measurement was repeated with just the IMC locked at 25 W and ISS second loop enageged. This did not change the HV MON coherence with the IMC WFS nor the coherence with MC_F.
Sheila reported that Nutsinee switched the SEI_CONFs to BLEND_45mHz_SC_useism last night but looking this morning I see that it did not switch the Blend on ETMY. I manually switched the (X & Y dof) Blends only; the Sensor Correction did switch to not use the BRS. Otherwise all switching worked with Sensor Correction for all Test Masses and no BRS in use at the Ends.
by the way, I noticed this issue because I saw the SDF differences at the end stations were not the same. Since this is a guardian controlled feature, it might be argued that it would be Not_Monitored. Were that the case...
In case you are wondering what difference this can make...
The attached asd shows the ETMs 12 and 6 hours ago. 12 hours ago (ref traces) the ETMY was in 250mHz blends while the ETMX was in the 45mHz blend. 6 hours ago was after the ETMY was actually switched to 45mHz blends.
The thick green and brownish curves become much more like the others after the blend switching and moving the blend gain peaking away from the secondary microseismic peak.
The second attachment are trends of theX & Y ARM SUSPOINT Motions based on the ISI's onboard GS13s and the ETMY and ITMY SWSTAT showing when the ITMY blend switched but the ETMY did not and then this morning when the ETMY switched to the same blend as the ITMY. When the two ISIs are in different blends, the motion is clearly larger although based on these trends. Based on the XARM Motion though, not clearly a better blend overall.
ETMY_ST1_CONF wasn't switching properly because I had found an issue with the guardian when trying to write some different configurations. I forgot to revert the code to a properly working state. Sorry. I've fixed the configuration, and tested that it works on ETMY.
J. Kissel, D. Tuyenbayev Following preliminary results from Darkhan on the individual actuation strength of the UIM and PUM stages for H1SUSETMY (see, thus far LHO aLOG 31275), and the current delightfully long lock stretch with them in place, I'm bringing this study to a close. I've turned off the temporary L1 and L2 calibration lines at 33.7 and 34.7 Hz, respectively. We do not intend on turning on these lines again for the duration of the run. These lines were turned OFF at Nov 07 2016 21:21:49 UTC.
Summary
A refined analysis of the L1, L2 and L3 stange actuation strenghts was done using the data from last several days that include several low-noise lock stretches. Actuation strength factors are:
KU = 8.020-8 +/- 2.983-10 N/ct ( std(KU) / |KU| = 0.0037 )
KP = 6.482-10 +/- 2.748-12 N/ct ( std(KP) / |KP| = 0.0033 )
KT = 4.260-12 +/- 1.313-14 N/ct ( std(KT) / |KT| = 0.0031 )
Details
Following 4 lines were used to calculate the factors: UIM (L1) line at 33.7 Hz, PUM (L2) line at 34.7 Hz, TST (L3) line at 35.9 Hz and PcalY line at 36.7 Hz. The most recent DARM model parameters were used for this analysis. Also, values past Nov 5 were calculated with the updated DARM filters (see LHO alog 31201), not accounting for this would produce results biased by 1-2%.
Each data point is a quantity calculated from 10s FFTs. The outliers were removed in two steps:
- took the mean and the standard deviation of all data points in intervals when the IFO range was >=50 MPC, removed 3-sigma outliers;
- removed the 3-sigma outliers from the mean of the remaining data points.
The mean values and the standard devitaions noted above were taken from GPS time interval [1162369920 1162413500], ~11 hours of low-noise data (blue markers). Standard errors on the mean values, std(Ki) / sqrt(N), are orders of magnitude smaller compared to the Pcal and the DARM loop model uncertainties (number of data points in the seletected interval - N=4251).
For preliminary results from Nov 4 data and before see related reports: 31183, 31275.
Recall the ER8/O1 values for these coefficients were 'Optic' 'Weighted Mean' '1-sigma Uncertainty' '1-sigma Uncertainty' 'Stage' '[N/ct]' '[N/ct]' '%' 'ETMY L1' '8.17e-08' '3.2e-09' '3.9' 'ETMY L2' '6.82e-10' '5.2e-13' '0.076' 'ETMY L3' '4.24e-12' '4.1e-15' '0.096' from LHO aLOG 21280. Comparing against numbers above, KU = 8.020-8 +/- 2.983-10 N/ct ( std(KU) / |KU| = 0.0037 ) KP = 6.482-10 +/- 2.748-12 N/ct ( std(KP) / |KP| = 0.0033 ) KT = 4.260-12 +/- 1.313-14 N/ct ( std(KT) / |KT| = 0.0031 ) This means a change of (ER8 - ER10)/ER8 = ETMY L1 0.0183 ETMY L2 0.0495 ETMY L3 -0.0047 We will compare these numbers against those determined by frequency-dependent transfer functions, e.g. the to-be processed data from LHO aLOG 31303, and update the low-latency/ calibration accordingly next week. It will also be interesting to re-cast the L1 and L2 numbers into a combined actuation strength change from ER10/O1, and compare it against the constantly calculated kappa_PU and check consistency there.
Data points prior to DARM filter update mentioned in the report were analyzed with the help of following DARM model parameters:
ifoIndepFilename : ${CalSVN}/Runs/PreER10/Common/params/IFOindepParams.conf (r3519)
ifoDepFilename : ${CalSVN}/Runs/PreER10/H1/params/H1params.conf (r3640)
ifoMeasParams : ${CalSVN}/Runs/PreER10/H1/params/H1params_2016-10-13.conf (r3519)
and after the the DARM filters were updated (GPS 1162336667) the following configuration was used:
ifoIndepFilename : ${CalSVN}/Runs/PreER10/Common/params/IFOindepParams.conf (r3519)
ifoDepFilename : ${CalSVN}/Runs/PreER10/H1/params/H1params_since_1162336667.conf (r3640)
ifoMeasParams : ${CalSVN}/Runs/PreER10/H1/params/H1params_2016-10-13.conf (r3519)
Scripts were uploaded to CalSVN at
${CalSVN}/Runs/PreER10/H1/Scripts/Actuation/2016-11-08/
5 days SLM data (75 MB): ${CalSVN}/Runs/PreER10/H1/Measurements/Actuation/2016-11-08/
Plots: ${CalSVN}/Runs/PreER10/H1/Results/Actuation/2016-11-08_H1_UPT_act_strengths_*
We discovered that in the single-line analysis we had an incorrect sign for TST stage actuation (we incorrectly set the sign of the N/ct coefficient).
The updated results have been posted in LHO alog 31668.
From Dave's above alog: "* we did not do the TCS-AI power sequence this time and unfortunately this may have tripped the TCS chillers."
This unfortunately did trip the TCS chillers and therefore the TCS CO2 lasers. I reset them all around14:30 PST; everything came up without issue.