james.batch@LIGO.ORG - posted 14:17, Wednesday 20 January 2016 (25051)
Restart x1 test stand computers.
Restarted all computers and services on x1 test stand following power failure.
Restarted all computers and services on x1 test stand following power failure.
Peter, Jason, Jeff B, Nutsinee
The TCS CO2X laser was down due to the power outage this morning along with many other systems. Jeff Bartlett reported that the chiller was tripped and he restarted it. We later restarted the power supply on the mezzaine which then brought the system back up. However, the timeseries of the flow alarm didn't imply that the alarm was tripped when the chiller was shut off and the laser output continued to have non-zero reading for over an hour until the frontend was restarted. Moral of the story is, some of the channels would continue to record fault values after a power outage until the frontend gets restarted. Another moral of the story, maybe we should move TCS CO2 power supply to the CER.
I think the TCS power supplies were located in the mechanical room because they are switching supplies and we wanted them far away from other electronics. By the way, there is a second mechanism that shuts off the laser if the chiller goes off (not just the flow meter) which is that the chiller has a built-in relay that acts on the laser controller to turn off the controller. It was added so that we weren't reliant on low flow rate turning off the laser.
Here are some recovery actions I have performed:
Restarted weather station IOC's at EY, MY, MX, and EX following the power failure.
Notables:
HEPI CS Pump Controller required a couple power cycles before it would respond to ssh for restart.
HAM2 ISI tripped on CPSs during Isolation--Jim suggests we increase the DC Bias Ramp time.
All other platforms started and isolated without trouble.
All SDF snaps have been set to OBSERVE. Still some reds as Jim is setting up blends and sensor correction.
Kyle, Gerardo
Noticed that the pressure at X-End continued to climb after the power outage, we drove to X-End and found the controller on "standby mode", restarted the HV manually.
We then changed the controller settings for Auto HV Start to "Enabled".
We have begun recovery from an unexpected power outage that occurred at 7:44 PST and lasted for 3 minutes (as noted from the UPS).
FRS ticket 4245 opened.
These are the 60 day trends for the PSL crystal and diode chillers.
TITLE: Jan 19 EVE Shift 20:00-04:00UTC (12:00-20:00 PDT), all times posted in UTC
STATE Of H1: Locking
SUPPORT: Jenne and Sheila
INCOMING OPERATOR: N/A
SHIFT SUMMARY: Commissioning activities listed from previous log
ACTIVITY LOG:
00:33 Patrick cleared the HEPI accumulators
As per Sheila's request I have started to edit the IFO_ALIGN screen. This involves removing buttons that no longer work (ie. "Save Alignment" and "G"). She also expressed interest in have the border that illuminates while the slider values are ramping such as those in the "Optic Align" sub-windows from the individual SUS medm screens. i'm open to suggestions.
This afternoon I took measurements of the DHARD Yaw loop at different PSL powers. In addition to general characterization of the O1 IFO, I will use this data to verify the ASC loop model. Once we're confident in the loop model at powers that we can measure, we will use it to try to design ASC filters that we can use for high power operation in a few months.
In the first attached screenshot and .xml file, the measurement at 2 W is blue, the measurement at 10 W is orange, and the measurement at 20 W is red.
The 2 W measurement was taken at the DC_READOUT state, and only FM6 of the DHARD Yaw filter bank was engaged. This measurement was taken from a lock stretch earlier in the day, using 40 points at 3 avg each. In the xml file, the 2 W data is saved as references 0-4. In the second screenshot and .xml file, I include some higher resolution measurements of the peaks, with 5 avg for each point.
The 10 W measurement was taken at INCREASE_POWER, and both FM2 and FM6 were engaged. This measurement used 60 points at 5 avg each. In the xml file, the 10 W data is saved as references 5-9. I had modified the lscparams.py guardian code to stop the power increase at 10 W, but I have reverted that change, so everything should still be as normal.
The 20 W measurement was taken at INCREASE_POWER, and both FM2 and FM6 were engaged. This measurement used 60 points at 5 avg each. In the xml file, the 20 W data is the "live" traces.
The 10W and 20W measurements today are broadly consistent with the measurements from 31 July 2015 (alog 20084), which is good.
I have plotted yesterday's Dhard Yaw measurements against the ASC model that I have.
The ASC model seems to be missing some gain related to the laser power, since I need a different fudge factor for each input power to get the upper UGF of the model to match the measurement. This is probably a problem with the Optickle part of the model since that's the only thing that should change very significantly in overall gain as a function of power. The suspension model (which includes radiation pressure) shows the peaks from the lower stages moving to higher frequency with higher input power as expected.
In the individual plots (eg. 2W_only), I show the measurement (dark blue) with some error bars (light blue) derived from the measured coherence plotted against the model (black trace). The 10 W and 20 W measurements match the model pretty well (except for the gain fudge required), but the 2 W measurement doesn't match the model very well below a few Hz. I'm not yet sure why this is.
In the final plot attached, I show all 3 models (solid traces) and all 3 measurements (dotted traces), but without error bars to avoid clutter.
Robert, Sheila, Evan, Gabriele
I tried to look at one of Robert's injections from yesterday, and we noticed a dangerous bug, which had previously been reported by Annamaria and Robert 20410. This is also the subject of https://bugzilla.ligo-wa.caltech.edu/bugzilla3/show_bug.cgi?id=804
When we changed the Stop frequency on the template, without changing anything else, the noise in DARM changes.
This means we can't look at ISI, ASC, PEM, or SUS channels at the same time as DARM channels and get a proper representation of the DARM noise, which is what we need to be doing right now to improve our low frequency noise. Can we trust coherence measurements between channels that have different sampling rates?
This is not the same problem as reported by Robert and Keita alog 22094
people have looked at the DTT manual and speculate that this could be because of the aggressive whitening on this channel, and the fact that DTT downsmaples before taking the spectrum.
If there is no near term prospect for fixing the problem in DTT, then we would want to have less aggressive whitening for CAL_DELTA_L_EXTERNAL
I spent a little time looking into this and added some details to the bug report. As you said, it seems to be an issue of high frequency noise leaking through the downsampling filter in DTT.
Until this gets fixed, any reason you can't use DARM_IN1 instead of DELTAL_EXTERNAL as your DARM channel? It's better whitened, so it doesn't suffer from this problem.
The dynamic range issue in the whitened channel can be improved by switching to five zeros at 0.3 Hz and five poles at 30 Hz.
The current whitening settings (five zeros at 1 Hz, five poles at 100 Hz) produce more than 70 dB of variation from 10 Hz to 8 kHz, and 130 dB of variation from 0.05 Hz to 10 Hz.
The new whitening settings can give less than 30 dB of variation from 10 Hz to 8 kHz, and 90 dB of variation from 0.05 Hz to 10 Hz.
We could also use 6 zeros at 0.3 Hz and 6 poles at 30 Hz, which would give 30 dB of variation from 10 Hz to 8 kHz, and 66 dB of variation from 0.05 Hz to 10 Hz.
The 6x p/z solution was implemented: LHO#25778
At low frequency, DARM is coherent with many ASC and LSC channels. However, ASC channels are not uncorrelated to each other, and the same is true for DRMI LSC channels. So if one wants to use coherence to estimate how much of DARM is explanable with coupling from a set of auxliary channels, it is important to avoid over counting due to the cross coheence between the aux signals.
To cope with this, I wrote a multi-channel cohrence code, which takes into account the cohrence between the aux channels and estimates the total contribution of a set of correlated aux channels to a target channel. The attached script multicoherence.m does this in MATLAB.
I computed this multi-channel coherence (and the projection of noise into DARM) for DRMI LSC signals (plot 1) and all ASC signals (plot 2). In each of those plots, the top panel shows the total coherence of DARM with the set of channels being considered. This coherence is defined as (PSD_DARM - PSD_LINEAR_COMBINATION_AUX) / PSD_DARM and so gives an idea of how well the combination of aux channels can explain the noise we see in DARM. In each plot, the bottom panel shows a coherence-baed projection of the aux channels into DARM.
As expected, we see that the low frequency region of DARM (<20 Hz) is dominated by the sum of all ASC noises. Similarly, the LSC noises are not far from the measured sensitivity (as already known, we need a MICH FF retuning).
I attached a brief not describing how the multi-channel coherence is computed. This kind of computation is not new material, it's been around for a long time.
Reset counters in attached screenshot.
Evan, Kiwamu,
As some of us have already noticed, there is a broadband noise with a 1/f^{0.5} shape in frequency from 60 to 200 Hz. This noise is unidentified.
Do not believe any statements in this report until futher analaysis. Something is fishy with the claibration of the cross-spectrum.
We are planing to check how stable this noise level is over the course of the entire O1.
The below shows an example spectrum of DARM.
Blue curves are twenty spectra of DARM (aka C01 frame, converted into displacement), each of which is made by the Pwelch with Hanning, detrended, 50% overwrap for a 12 minutes time series. The data starts at a GPS time of 1134604817. Green curves are the square-root of twenty cross-power-spectra of DCPD A and B which are reconstructed from the sum and null streams of the DCPDs. The DARM suppression effect was removed from the sum signal. The cross-spectra are then calibrated to the displacement using the latest O1 DARM model of the calibration group. No time varying correction (i.e. kappas) is applied. Red line is a 1/f^{0.5} line to show how steep the slope of the green curves is. I also attach the fig file.
Gabriele, Evan, Kiwamu
There was a human-error in my code for calibrating the cross-spectrum. It was removing the loop suppression after the power spectrum of the null stream was subtracted from that of the sum stream. This was fixed such that the subtraction happens after the removal of the suppression in the sum spectrum. The below is the latest plot.
The plot shows the ampitude spectral desnsities of the calibrated darm displacement (aka C01) and the calibrated cross-spectrum. The cross-spectrum should represent noises which are coherent between two OMC DCPDs.
As a coarse verification, I have eye-ball-fitted the shot noise level with the fixed cavity pole frequency of 341 Hz (shown as a dotted line in cyan). Then I subtracted the shot noise component quadratically out from the actual displacement spectrum (in black). The residual (in blue) agrees with the estimation from the cross-spectrum. In order to check the slope of the cross-spectrum, I also drew a 1/f line. The cross-spectrum seems to follow 1/f from 50-ish Hz to 150 Hz.
The fig file is attached as well.
An update can be found in entry 25106
A higher resolution version is attached. The frequency resolution is set to 0.1 Hz, 50% overlap with Hanning for 1 hour data. No new findings.
The 1 Hz comb feature (see for example alog 24695) is becoming visible in 20-50 Hz. By the way, the legend in the plot is wrong.
Corey, Adam, We're just about to start a detchar safety injection.
We've finished this injection. I'll post a few more details shortly.
More Details: I injected the waveform from 'https://daqsvn.ligo-la.caltech.edu/svn/injection/hwinj/Details/detchar/detchar_03Oct2015_PCAL.txt'. The injection start time was 1136927627. The log file is checked into the svn - 'https://daqsvn.ligo-la.caltech.edu/svn/injection/hwinj/Details/detchar/O1/log_H1detcharinj_20160115.txt', although for some reason it only shows the start time.
The time noted in the alog entry is incorrect. The correct time from the log is 1136927267. The injections are visible at the corrected time.
Evan G., Jeff K.
Revisiting measurements Jeff made in the field [1],[2],[3] and new measurements with those I took in the EE lab, we compared with the UIM residuals measurements obtained using the Pcal and ALS DIFF measurements. Attached is a figure showing the electronics chain and comparing with the residuals obain. We find that the BOSEM electronics account for some of the residuals found in the UIM measurements, but not all. At this point, we have only clues, but no solid evidence for what remains of the residuals. We have three theories:
I set up in the EE shop a UIM driver, satellite box, and BOSEM to repeat Jeff's measurements and verify we observe the same effects. Indeed, I observed similar issues that Jeff had observed in his measurements from the floor. We put these measurements on top of the UIM actuation residuals measurement/model but, unfortunately, find that they are not completely accounted for by the electronics chain.
We started to think about what else could be going wrong with the residuals, but so far have come up with the only three theories above. To undertand this effect in more detail, Jeff is currently undertaking exploratory measurements of the UIM-->DARM and Pcal-->DARM to frequencies higher than 100 Hz. Hopefully these measurements will shine some light on this effect.
The quad model on the svn does not have UIM-PUM wire violin modes. I just drafted an update that does include these, which I used to generate the attached figures. I'll commit this update if it appears consistent with measurements.
The plot ViolinModes_12Jan2016.jpg compares the model UIM L to Test L transfer function with and without the UIM-PUM modes, but with the fiber modes in both cases. I guessed the UIM-PUM violin modes to be a Q of 100,000, but that could be off an oder of magnitude or two. The second figure plots the ratio of these 2 transfer functions.
According to this second figure, the UIM-PUM violin modes explain some, but not all of the discrepancy seen between the measurements and the model in the log above. So either the model is not correct, or there is still something in the feedback loops we are missing.
For the Bench measurements, the data is stored at:
/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/PostO1/H1/Measurements/Electronics/BenchUIMDriver/2016-01-12
Attached are plots showing the individual components of the coil driver electronics fitted with the vectfit program in Matlab and using LISO. I report the fitted LISO values below with respective uncertainties.
Dummy BOSEM connected, with output impedance network (see figure UIM_out_impedance.pdf):
Best parameter estimates: zero0:f = 84.1507169277 +- 1.627 (1.93%) pole0:f = 303.5726548020 +- 5.431 (1.79%) pole1:f = 127.6915337428k +- 3.535k (2.77%) factor = 2.2065872530m +- 17.45u (0.791%)
This fit shows the calibration of 2.2 mA/V, one zero at 84.15 Hz, and two poles at 303.57 Hz and 127.7 kHz.
BOSEM only (output impedance network divided out so only BOSEM transfer function remains, see figure UIM_bosem.pdf):
Best parameter estimates: zero0:f = 334.8526120460 +- 3.892 (1.16%) zero1:f = 1.2383234778k +- 43.39 (3.5%) zero2:f = 8.2602408615k +- 320.8 (3.88%) pole0:f = 747.0160319882 +- 27.6 (3.69%) pole1:f = 5.3613221192k +- 210.2 (3.92%) pole2:f = 25.8483289876k +- 310.1 (1.2%) pole3:f = 232.8627791989k +- 3.041k (1.31%) factor = 11.6075630096m +- 14.92u (0.129%)
For some reason, this transfer function is tricky to fit. These are the fewest number of zeros and poles I could put in LISO and still provide a good fit to the data. LISO does complain that strong correlation exists between pole1<-->zero2 and pole0<-->zero1. When I removed these pairs, the fit became much worse, so I left them in.
As a comparison with the full chain: digital AntiAcq x analog Acq (output impedance network) x BOSEM (see figure UIM_full.pdf). The model fits the measurement with 2% up in magnitude to 40 kHz, and within 1 degree in phase up to 50 kHz.
Finally, the previously shown plot in the original post divides out the full BOSEM measurement in the field ('field BOSEM'), but the model already takes care of the analog output impedance network, so this original plot was double-counting. I attach here a corrected version of the plot (see UIM_res_with_elec.pdf). This shows that the BOSEM does indeed correct for some of the excess residual, it is not the dominant contributor to the behaviour above ~60 Hz.
J. Kissel, R. Savage, LHO Operators Tallying up the progress so far on the schedule of PCALX excitations at high frequency (see plan in LHO aLOG 24802): Achieved Planned Frequency Amplitude Start Time Stop Time Duration Duration Success? (Hz) (ct) (mm-dd UTC) (mm-dd UTC) (hh:mm) (hh:mm) (Yes / No, reason if no) ------------------------------------------------------------------------------------------------------------------------------ 1001.3 35k 01-09 22:45 01-10 00:05 01:20 01:00 Yes 1501.3 35k 01-09 21:12 01-09 22:42 01:30 01:00 Yes 2001.3 35k 01-09 18:38 01-09 21:03 02:25 02:00 Yes 2501.3 40k 01-09 12:13 01-09 18:31 06:18 02:00 Yes 3001.3 35k 01-10 00:09 01-10 04:38 04:29 04:00 Yes 3501.3 35k 01-10 04:41 01-10 12:07 05:26 06:00 Good Enough! 4001.3 40k 01-09 04:11 01-09 12:04 07:55 08:00 Good Enough! 4501.3 40k 01-10 17:38 01-11 06:02 12:24 12:00 Yes 5001.3 40k 01-11 06:18 on-going (as long as we can get) Thanks to all of the operators who have been dilligently caring for these lines while we sleep! For the record, while these PCALX calibration lines are on, the majority (if not all) of the range is consumed, so we cannot perform PCALX hardware injections.
I used the high frequency calibration lines injected above to estimate the sensing function at those frequencies. For this analysis, SLM Tool was used to obtain the line amplitude and phase of these calibration lines at different relevant channels.
Sensing Function = DARM_ERR[ct] / PCAL_TXPD[m]
The DARM_ERR signal is dewhitened and the PCAL_TXPD is corrected to get metres using the scheme described in G1501518.
Furthermore, ratio of GDS/Pcal is calculated and is included in the attached plot.
A data quality flag has been created to capture times when these extra PCAL lines were in the data. It is H1:DCH-EXTRA_PCAL_LINES:1 and a description of this flag can be found on the detchar wiki.
