at 14:11 PST h1dc0 froze up with a kernel panic. We reset the computer by pressing the front panel RESET button, but it did not come back cleanly.
When h1dc0's daqd process first was restarted, it ran for only a short period and then died. It was logging GPS timing jumps of one to many seconds.
Monit restarted it and it is now running correctly. The other DAQ systems restarted at this time correctly, except h1broadcast0 took a long time to get going.
Frame gap due to this problem is
1139782208 to 1139782912
Feb 17 2016 22:09:51 UTC to Feb 17 2016 22:21:35 UTC
The TCS simulation model to estimate the lenses and ROC of the optics is now running. The following lists gives the nominal values and the transfer functions used for all the filters:
This screen is accessible from the TCS_MASTER > SIMULATION button.
The static lenses from the ITMs have been added also (see T1400602)
This morning while TJ was trying to relock, ETMX HEPI tripped again. Kiwamu, Sheila and I all poked around at this a bit and can't find an obvious culprit yet. It's definitely something to do with the X HEPI position (tidal or something else). First attached plot is the ETMX actuators time series at the trip. The little shark tooth just before the trip seems to be caused by something in tidal (a reaction to HEPI walking off, maybe? Or something causing the IMC to do something weird?). When I looked a the cartesian signals, the movement seems to be all in X. The second plot is the time series over the same 32 second window of a few other signals, but none of them are particularly enlightening. The dark blue trace on the middle left is the HEPI X location, which goes on some 30 micron excursion, which ultimately causes the lock loss. But the motion seems to be chickens/eggs. Did tidal get some other input that drove HEPI or did HEPI go crazy and drive off tidal? I don't know.
model restarts logged for Tue 16/Feb/2016
2016_02_16 13:58 h1tcscs
2016_02_16 14:27 h1isietmy
2016_02_16 14:29 h1dc0
2016_02_16 14:31 h1broadcast0
2016_02_16 14:31 h1nds0
2016_02_16 14:31 h1nds1
2016_02_16 14:31 h1tw1
Maintenance day. TCS and ISI-ETMY model changes. H0EDCU_VE.ini change. Related DAQ restart.
Wed 10/Feb/2016 - Mon 15/Feb/2016 No restarts reported.
TCS model change. WP5732
Dave, Aidan:
Aidan made model changes to h1tcscs.mdl
Buried EY Seismometer added to h1isietmy model. WP5733
Jeff, Robert, Dave:
h1isietmy was modifed to read the external seismometer, these channels were added to the science frame.
Adding all Vacuum Controls channels to the DAQ. WP5731
Dave:
All the vacuum controls channels were added to the DAQ. Increased the number of channels from 280 to 1777
DAQ restart
Jim, Dave:
DAQ was restarted to support all the above changes. After several hours h1fw1 became unsynchronized to h1fw0 (we had not seen this before). Restarting h1fw1 fixed this.
GLIBC patching
Dave:
Patched and restarted critical servers and about half of the control room workstations.
I filled the PSL diode chiller with 350ml of water. Attached is a 30 day plot showing where the diode chilller was filled on Feb 4th, and then the alarm level increasing starting around Feb 13th.
How does this alarm? Is it on Verbal? Audible alarm out in the Diode Room? Or is the red light the only way to tell if it's low. In general, the Crystal Chiller is the one which requires action every week. For the Diode Chiller all we have is the red LED to warn us of low levels (I've never seen it). We (operators) just need to remember to also keep checking the Diode Chiller when performing this task.
Conceivably, we check this every Thursday. If the alarm goes off on Friday, is it OK for this chiller to be in alarm until it's checked the next Thursday?
I wonder if we should include trending the alarm level on the FAMIS procedure for this as Cheryl did.
Unfortunately the red LED on the face of the diode chiller is the only alarm we have and checking this LED is part of the FAMIS procedure. Additionally, especially with Peter and I in Germany until mid-March, if anyone is in the diode room for any reason and sees this red LED lit, fill the diode chiller immediately until the light goes out and post in the alog that water was added and how much. If the water level gets too low, the chiller will shut off, which also shuts off the entire PSL. We need a better alarm (especially in the hypothetical situation Corey brings up, this could result in a laser shut down if not caught before the next scheduled check), but this is what we have for now.
I have increased the LLCV valve setting to 18% (from 15%) to try and compensate for changing conditions.
Factors that are changing are:
The supply pressure is falling as the liquid level in the storage dewar falls.
Heat leak to the liquid transfer line is increasing with outdoor temperature. This results in more vapor in the line and less liquid to the pump.
Radiation from the beam tube as the beam tube warms.
The attached plot shows 100 days of the control valve setting. The flat line at 15% is due to the failure of the differential pressure sensing- we have had to resort to manual setting of this valve.
For reference the liquid capacity of the pump reservoir is ~77 gallons when full.
I forgot about the heat into the pump due to radiation from the surrounding vacuum chamber in the mid station. Here is a plot of recent VEA temperatures at MID Y.
That is illuminating, thanks. We take feedback controls for granted around here; don't miss it till it's gone. Hope this sophisticated PIL* control algorithm will tide us over until we can regen & fix the level sensor.
*Primate In Loop
Here is a 45-day trend of HEPI Pump Pressures & Drives (thus closing out FAMIS # 4517).
A few note:
Attached is a time series of TMSX roll osems over the last year, which has been slowly drifting in the same direction the whole time. There was a went in June, wich is the larger jump, you can see when the sat amp started to oscillate a few weeks ago, and me moving roll over the weekend. This drift is mostly seen in F1, I don't know if it is real or not.
Rana has told us that at LLO the TMS QPD positions are sensitive to roll. Before closing the soft loops we moved roll by 250 urad (accroding to the osems) and saw that this had no impact on the error signals of the soft loops.
Here are optical lever 7-day trends; closing out FAMIS # 4392.
Masayuki
The ISS 2nd loop medm screen has been Improved. The digital path has been shown.
I have noticed that h1fw1 is lagging quite a ways behind h1fw0 (they are normally synced to the second for the fast data CRC and within a few seconds of writing frame files). h1fw1 was lagging 1 to 10 seconds on CRC and up to 20 seconds for frame writing.
17:47PST, restart h1fw1 daqd process. h1fw1 is now back in sync. The lack of CRC sync lead me to think this is a daqd problem and not a Solaris QFS slow raid.
Jeff, Darkhand, Kiwamu,
We have been meaning to do this, but kept missing a chance to do. Today we have fliiped the sign of the ESD bias voltage both on ETMX and ETMY.
This alog only describes what we did on the digital front end models. We will perform a set of confirmation measurement and verify the H1 DARM model later.
See alog 22135 for a concise summary of the bias flip.
[Bias flip on ETMY ]
[Bias flip on ETMX]
We took open loop and Pcal sweep measurements in nominal low noise after the biases were flipped. They reside at:
/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/PostO1/H1/Measurements/DARMOLGTFs/2016-02-16_H1_DARM_OLGTF_7to1200Hz.xml
/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/PostO1/H1/Measurements/PCAL/2016-02-16_PCALY2DARMTF_7to1200Hz.xml
Darkhan will process the data later.
P.S.
I have updated the SDF accordingly for SUS-ETMX and ETMY and CAL-CS when the interferometer was in nominal low noise.
Kiwamu, Darkhan,
Summary
Our analysis of a DARM OLG TF and a PCALY to DARM TF measurements taken after the ESD bias sign flip on Feb 16, 2016 showed that:
Disscussion (actuation function)
The ETM ESD drivers' electronics and front-end FOTON filters associated with the ESD responses were recently updated, see LHO alogs 25468, 25485. Since the front end filter's do not perfectly cancel the ESD response, in the Matlab parameter files we used a better approximated ZPK responses that are based on the old ESD electronics measurements. Since currently we do not have TF measurements for the new ESD driver electronics, we have removed all of the compensations (for the non-perfect front-end filters) from the Matlab analysis of the DARM and PCALY to DARM TF measurements.
Measurement files, scripts
DARM OLG TF and PCALY to DARM TF measurements have been committed to CalSVN:
Runs/PostO1/H1/Measurements/DARMOLGTFs/2016-02-16_H1_DARM_OLGTF_*.txt
Runs/PostO1/H1/Measurements/PCAL/2016-02-16_PCALY2DARMTF_*.txt
Up to date filter files have been copied to
Common/H1CalFilterArchive/h1omc/H1OMC_1125881488.txt
Common/H1CalFilterArchive/h1susetmy/H1SUSETMY_1126240802.txt
DARM parameter file and comparison script have been committed to
Runs/PostO1/H1/Scripts/DARMOLGTFs/CompareDARMOLGTFs_O1andPostO1.m
Runs/PostO1/H1/Scripts/DARMOLGTFs/H1DARMparams_1139722878.m
Results (plots) are in
Runs/PostO1/H1/Results/DARMOLGTFs/2016-02-16_*.pdf
Bubba and I were able to crane the Genie manlift over the Y beam manifold this morning in order to continue the crane rail work.
GV5 and GV7 were soft closed for the duration from UTC 16:15 to 16:53.
The crane behaved normally.
{Rana, Evan}
This evening we looked at the coupling of DCPD bias voltage into DARM.
Each DCPD is biased with +12 V from a linear regulator with a 1 Ω output resistor. We wanted to inject extra bias noise, so we exposed the bias lines with a breakout board at the DCPD chassy in the HAM6 rack (see D1300502). Then we summed in the output of an SR785 across an impedance of 10 kΩ in series with 20 µF (see diagram; green shows the normal DCPD electronics and red shows the addition). The 10 kΩ gives a 1:104 ratio of bias fluctuation to drive voltage, and the capacitor ac-couples the drive so that it does not pull on the bias at dc.
With the SR875 we drove a 1 Vpk line first at 187.3 Hz and then at 62.4 Hz (i.e., the bias fluctuation was 70 µV rms). We looked at the response in DARM (at 2 W dc readout, with DARM still controlled by EX). We had 10 mA dc on each PD, in the 400 Ω transimpedance configuration.
For 187.3 Hz (from 02:03:00 to 05:22:00 Z), the magnitude in DARM was 1.13×10−18 m rms, which implies a coupling of 1.6×10−14 m/V. The noise of the regulator at this frequency was 0.9 µV/Hz1/2, which implies a DARM noise of 1.4×10−20 m/Hz1/2.
For 62.4 Hz (from 05:33:00 to 05:56:30 Z), the magnitude in DARM was 1.4×10−18 m rms, which implies a coupling of 2.0×10−14 m/V. The noise of the regulator at this frequency was 1.9 µV/Hz1/2, which implies a DARM noise of 3.7×10−20 m/Hz1/2. Note that there is some room for error here because the DARM control configuration here is not exactly the same as the low-noise configuration. However, the ugf and phase margin should not be too different in the two configurations.
This is very close to the current low-noise DARM noise floor. However, if DARM is limited by voltage noise of the regulators, we should either expect that the DCPD null stream is equal in magnitude to the DCPD sum (it isn't), or the regulator noises are coherent (they aren't).
I am not sure if there are any contradictions in these numbers. DARM calibration at DC is C [W / m] = 4 * pi * G_arm / lambda * sqrt(G_prc * P_in * P_as / G_src) = 1.5e9 and 5e9 for input power 2W and 22W and 26mW at the AS port. This means that the response you measured is ~2e-5 A/V. If the input power is 22W, 1uV/sqHz projects to 4e-21 m/sqHz around 100Hz. This number is what you have in your noise budget plots.
The freerunning DARM channel may not be properly calibrated in the low-power state with EX control, so the numbers I gave originally could be too high.
We directly measured 1.2×10−6 mA rms in DCPD B at 187.3 Hz, which implies a coupling of 0.017 mA/V, which implies the bias noise shows up in DCPD B at 1.5×10−8 mA/Hz1/2. The shot noise with 10 mA on the PD is 5.7×10−8 mA/Hz1/2. So it is a factor of 4 or so below the DARM shot noise, assuming DCPD A is similar. That's close to the dark noise, but the dark noise is flat down to a few tens of hertz while the regulator noise has some negative slope.
Voltage noise of bias (pin #8 of the d-sub), measured with BNC clip doodle and SR785 (with AC coupling).
Sheila, Rana
We continued the investigation. Today we hooked up a DAC channel to the bias to make sweeps and noise injection. No surprises relative to yesterday. Still seems so close to DARM as to be unbelievable. Maybe the 1 Ohm resistor in the bias circuit is not stuffed?
We also injected random noise into PZT2 to look for upconversion there. We saw the usual kinds of quadratic coupling; noise at a level ~500x above the quiesscent HVmon noise level was able to be just visible in DARM in the baseband as well as by injecting noise close to the 4100 Hz dither frequency.
0554 UTC, leaving it in 'low noise mode'. The BS coil driver is in some mixed low noise state giving us increased DAC noise below 50 Hz, but the noise above there is as good as ever.
Can't get the photodiode D's off the DCC at the moment (outage???), but surely there must be a biga$$ cap (ideally several in ||) from the pd cathode to ground??
Yes, in the in-vacuum preamp, there is a 1 uF capacitor from the Bias to ground. At the LM317T (voltage regulator that sets the bias), there is a 10 uF cap to ground, on the regulator side of the 1 ohm resistor (it's not shown in Rana's sketch above). In any case, the projection given above relies on the 1e4:1 scaling of the injected signal - can't you just measure directly the drive on the bias line, rather than assuming the 1e4:1 ratio?
Bigger caps are unlikely to help much. The noise is due to the internal burried zener diode which serves as the voltage reference. The BW of the regulator is about 10kHz here. A bigger C may reduce this, but may also increase gain peaking. Just use a low noise supply such as the one here. Noise performance: T1000025.
Den, Rana
We directly measured the voltage noise on pin8 and confirmed that our injection through the 9.09k resistor (not 10k as indicated above) scales as expected. It seems that the 1 Ohm resistor is really 1 Ohm (+/- 20%).
From MZ, we have http://www.edn.com/electrical-engineer-community/industry-blog/4422750/2/Simple-circuits-reduce-regulator-noise-floor.
This indicates that some larger caps may reduce the regulator output noise at 100 Hz by a factor of 5 or so. Could be easy to try if we have a spare whitening chassis sitting around.
Rana, Sheila, Den We have measured the bias noise with a better resolution and found that this noise is almost flat below 60Hz. Then made a projection of this noise to DARM when the input power was 22W and power on each OMC PD was 10mAmps. For PD A the coupling is 1.5e-15 m/V while for PD B is 3e-15 m/V. Attached plot shows the projection of bias noise to DARM. It is factor of ~4 below the current sensitivity.
According to the noise budget and measurement of the dark noise, this noise is above the dark noise level. This means that the noise is present only when the DC photocurrent is present.
Suggests QE compression due to bias reduction. For higher P it may be wise to not only improve bias voltage noise, but to fix anode-cathode potential. Either a true transimpedance preamp, or a tracking supply at the rack.
We drove PD_B bias and measured the coupling of the bias noise to OMC NULL channel for different current levels. Plot is attached. The coupling is close to zero when there is no power. The coupling scales linear for small currents and stays almost constant for large currents (>10mA on each diode). We have also double checked that the optical signal scales linear with the current by driving an intensity line.
Kiwamu, Den We have digitized bias channels (LSC:EXTRA_AI_2 channel) and double checked coupling to DARM. Attached plots show coupling of PD_B bias to DARM. There is a calibration line at 62.4Hz of this coupling.