05:06 UTC Retrieved verbal notification of earthquake from Taiwan. Response script said to take ISI to large earthquake. I hit LARGE_EQ_NOBRSXY and it broke the lock. Leaving in DOWN until earthquake passes.
03:49 UTC Addressed attached SDF differences.
Sharan Banagiri, Philippe Nguyen, Robert Schofield
After the calibration sweep, we made acoustic and magnetic injections in the electronics bay and LVEA. We also made a couple of impulse injections.
I planned to note last night that for this week, Tuesday to Tuesday, there may be equipment that is on that is not normally on during the run. This includes the computer by the PSL and 3 amplifiers that we leave on so that we can inject from the control room. We sweep normally, with these exceptions, before we approve the return to observation mode.
Time: 21:08 UTC to 23:57 UTC
Green X arm not locking ALS X: Messages: Fiber polarization, PLL, PDH Set to DOWN, INIT ALS_CUST_LOWNOISEVCO frequency servo controls error, low limit reached, flashing Hit RESET on ALS_CUST_LOWNOISEVCO, Frequency Servo, cleared briefly, errors returned
I guess it was just the polarization. I lowered it from 30 to 27 and it was fixed.
J. Driggers, J. Kissel, L. Sun
Today, during the calibration measurement time, we've done the following:
(1) Tuned the calibration line heights such that the uncertainty of each line is now roughly 0.2%.
2019-04-17_TuningCalibrationLineAmplitudes_Uncertainty.png shows the time series of the uncertainty for all calibration lines over many hours before and after the line amplitude change -- the middle section is where they were turned off for the standard suite of calibration sweeps; before they were at their former value, after they were at their current value.
(2) We realized the discrepancy between GDS production of kappa_C (the relative optical gain correction) and f_cc (the cavity pole frequency) and CAL-CS was because in the heat of battle yesterday, we neglected to update the value of the necessary one clock-cycle delay to 410.3 Hz in the PCAL DEMOD -- i.e. it errantly remained as the delay value at the former calibration line frequency at ~331 Hz. Fixing this error returned the time-dependent correction factors to the expected values from the reference model: kappa_C = 1.00 +/- 0.01, and f_cc = 410.6 +/- a few Hz (as opposed to the brief period yesterday between after the calibration line frequencies and now, where the cavity pole was reporting in the 425 Hz region, and the optical gain correction was reporting 0.97).
2019-04-17_FixingPCALDEMODPhase_410Hz_Fixes_TDCFs.png shows a screenshot of the current values.
(3) We gather a collection of broad-band PCAL injections, as well as the standard sensing function sweeps.
Times of broad-band PCAL injections: 2019-04-17 20:13:20 UTC for ~2 minutes, and again at Apr 17 2019 20:15:31 UTC.
/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Measurements/FullIFOSensingTFs
2019-04-17_H1_DARM_OLGTF_SS_5to1100Hz_15min.xml
2019-04-17_H1_PCAL2DARMTF_BB.xml
2019-04-17_H1_PCAL2DARMTF_SS_5t1100Hz_10min.xml
The data remains remarkably consistent with the reference model (see 2019-04-17_H1_sensingFunction_mcmcModel_vs_measurement.pdf ).
We've stacked all the recent sensing function measurements without correcting for time dependence to form an estimate of the frequency dependent, time-independent systematic erro with a Gaussian process regression, and it shows the same consistency (see 20190416_ref_model_H1_sensingFunction_GPR_on_AllSensingMeasurements.pdf)
More details to come.
Here is a look at one of the excursion that happened in the front-end estimation of kappa_tst (first one seen in this summary page). The attached plot show various estimations of kappa_tst that happens in the front-end. It also show uncertainty estimation in the calculation of calibration line ratios. We see that when the uncertainty in the estimation of calibration line ratios exceeds set threshold of 0.02, it triggers the gating function. Since the glitch happened to be in the DARM_ERR (IFO) both PCAL_LINE1/DARM_ERR and TST/DARM_ERR ratios get uncertainties higher than 0.02 as in this case. The KAPPA_TST_GATE_UNC_INPUT which is finally used, according to model, supposed to be maximum of the two uncertainties but in this case it is the minimum of the two (KAPPA_TST_GATE_UNC_INPUT is on top of SUS_LINE3_UNCERTAINTY). Since it is still larger than 0.02, gating is triggered. However the gated kappa_tst values don't make any sense. It supposed to be avoid the excursion in the raw kappa_tst but it seems to change the level values of kappa_tst. This function need to checked (this is a user defined c function block). The signals after that make sense.
Similar plot for LLO which also show similar problem with gating function block.
Turns out the switch blocks used were defaulting to a logic comparison of >= 0 instead of > 0. Which means the first input was always being passed. I'll update this in the common CAL_CS_MASTER.mdl file and we should be able to push that next Tuesday. The gating issue is a bit trickier. As far as I can tell the gate is doing what its supposed to do. At the time the uncertainty rises above threshold, it freezes then and there as shown in Shivaraj's plots. The problem is, since the uncertainty calculation is basically on a cadence of 10 seconds, there's a 10 second gap between the demodulated low pass starting to respond to a large excursion and the I think this means we need to put 10 seconds (or maybe a touch longer) buffer between the kappa value input to the gate, so that the coherence corresponds represents the data coming in instead of data that came in during the last 10 seconds. A 163840 or so sample ring buffer is doable, but I'll have to modify a C code user block to do so.
TITLE: 04/17 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Commissioning
OUTGOING OPERATOR: Jeff
CURRENT ENVIRONMENT:
Wind: 12mph Gusts, 9mph 5min avg
Primary useism: 0.04 μm/s
Secondary useism: 0.31 μm/s
QUICK SUMMARY:
Locked at NLN. PEM injections ongoing.
A follow up on the occasional h1calcs CPU_MAX warnings when the model's processing time exceeds 63uS:
To confirm that when this happens the number of cycles which exceed this value is very small, I have analyzed the cpu histogram information available in the kernel /proc/model-name/status file.
Each model writes a /proc/model-name/status file on the front end computer, the last line of which is a histogram of the cycle-histogram giving cycle processing time and number of cycles with this processing time (only non-zero instances are shown). I have written a pyqtgraph python program to analyze one or more histogram lines and plot them as a graphical histogram. Attached image shows h1calcs (lower histogram, 29 data sets) and for comparison h1susitmx (upper histogram, 21 data sets). Y-axis is number of cycles with the processing time, X-axis is the processing time bins (1uS - 60uS)
The plot shows that h1calcs spends most of its cycles in the 25uS-26uS region, with only one or two cycles in each of the 40+ bins. When CPU_MAX exceeds 62uS we can confidently say that only one or two cycles have exceeded that value out of the 16,384 cycles from that second. This happens about once or twice a week now, my cronjob on h1fescript0 clears the error within a minute.
(Note that the y-axis is logarithmic which necessitated adding one to the count values)
IFO is locked at NLN and Observing for the past 13 hours. Range is around 107.0Mpc. The wind has come a bit since morning and is between 6 to 12mph, with microseism elevated in X and Y. No other issues or concerns at this time.
Maybe, there is some correlation between laser/diode room humidity levels and the FSS tpd movement?
I've attached a 10-day minute trend and a 2-day minute trend. On the 2-day trend, it can be seen that the PSL enclosure humidity begins rising at ~15:00 UTC (8:00 PDT) yesterday, 4/18/2019, with no real change in the FSS TPD. Once the humidity gets to ~30%, the FSS RefCav TPD begins to decrease. Conversely, near the start of the 10-day trend it can be seen that the FSS RefCav TPD begins to increase once the enclosure humidity drops below ~30%. Also on the 10-day trend, while the humidity varies between 20% and 30% the FSS RefCav TPD appears to follow the changes, but with some delay. We're continuing to monitor this.
Received permission from the run coordinator for Chris to test run the propane forklift parked just outside the OSB rollup door. At 16:32 (09:32) started the propane forklift. There was strong coupling into DARM around 20Hz to 80Hz, and a drop of about 5.5Mpc,while the forklift was running. The forklift was shutdown at 16:43 (09:43); DARM and the range returned to normal. The conclusion: NO idling or running the propane forklift during Observing. Will update the NOT Acceptable document to exclude operation of the propane forklift.
Tag aLOG for DetChar.
After commenting on the lock durations reported by the PSL weekly script the other day, this morning JeffB and I
happened to be looking at the PMC relock counter and noticed that it was slowly increasing whilst the PMC
seemingly remained locked (see image PMCRelockCounter.png).
I don't know why this would be the case.
Took a quick look at this. According to the model, the relock counter reads the channel H1:PSL-PMC_LOCKED, which is a logical AND between the channels H1:PSL-PMC_LOCK_ON, H1:PSL-PMC_RESONANT, and the result of a Greater Than operation between the channels H1:PSL-PMC_TRIGGER_OUTPUT and H1:PSL-PMC_TRIGGER_MIN (no channel exists for the output of this operation); see 1st attachment for a picture of the relevant portion of the model.
I fired up both available channels from the AND operation, in addition to H1:PSL-PMC_RELOCK_COUNT and H1:PSL-PMC_LOCKED, in ndscope and watched the real-time data for a bit. H1:PSL-PMC_RELOCK_COUNT was steadily increasing at a rate of approximately 1 increment every 3-10 seconds; there was no change in the other channels. I then added H1:PSL-PMC-TRIGGER-OUTPUT to the plot to see if the relock counter increments were coinciding with a momentary drop of the trigger output causing the Greater Than operation in the model to fail; upon looking at several of the increments, at no point did the trigger output fall below the minimum trigger set point (0.005V). Even when it got close to the min trigger, there was no corresponding increment in the relock counter.
Taking a more long-term look, I took ndscope out to approximately 30 days (see 2nd attachment), and it appears this increase in relock counts began sometime around 2019-03-20. This continued until sometime on 2019-04-02, where the counter increments stopped. The counter stayed relatively flat until 2019-04-12, when it began increasing again; the increments really took off sometime on 2019-04-15 and have been increasing steadily ever since.
So far I have found no reason for the counter to be increasing like this. It may be worth looking at the code that handles the PMC relock counter, PMC_LOCKCOUNTER.c. Investigation continues.
Filed FRS 12753 for this issue.
I took a quick cursory glance at the PMC_LOCKCOUNTER.c code and did not see anything obviously out of place,
although I had some questions which were subsequently answered by both Dave and Jonathan.
Dave later looked at the combination of the front end model and the code. From trend data it was concluded
that the code behaved properly. The question became why was it that the trigger level fell below the minimum
trigger level?
The trigger level is the DC output of the locking photodiode. The minimum trigger level is, typically,
set to be a little lower than the DC level when the PMC is resonant. Currently this is set to a level which
is close to the dark level output of the locking photodiode. This was done to allow changing the overall
loop gain on the PMC servo electronically, via the AD603 variable gain amplifier. The question remains why
is the output of the photodiode varying, and my guess is that the output offset voltage is varying due to
either fluctuations in temperature or scattered light level. A number of the changes seem to coincide with
an excursion into the PSL Enclosure. The remedy for this would be to increase the light level on the locking
photodiode and reduce the servo gain electronically.
DaveB / Peter
TITLE: 04/17 Owl Shift 07:00 – 15:00 (00:00 -08:00), all times posted in UTC
STATE of H1: Observing
INCOMING OPERATOR: Jeff
SHIFT SUMMARY: Locked at ~109 Mpc when I arrived, stayed locked throughout the night. Got a GRB alert at both sites at 11:08 UTC, aside from that it was a quiet night.
LOG:
07:00 (00:00) Start of shift
11:08 (04:08) GRB Alert E329931
15:00 (08:00) End of shift
TITLE: 04/16 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC STATE of H1: Observing at 106Mpc INCOMING OPERATOR: Niko SHIFT SUMMARY: Another rough night for locking. I could not get to PRC_ALIGN at all in initial alignment, despite being able to lock PRMI in the regular lock sequence with little trouble. I tried twice. I also had quite a lot of trouble locking DRMI, despite stopping at LOCK_PRMI and moving the BS and PRM. LOG: 23:04 UTC Lock loss 23:26 - 23:33 UTC Robert to LVEA to turn on amp 23:50 - 23:55 UTC Sheila to LVEA to check that phone is unplugged 23:51 UTC NLN PEM running injections 00:46 UTC PEM injections done 00:56 UTC Observing 02:14 UTC Lock loss PEM group moving equipment during relocking 03:13 UTC NLN Reverted SDF differences PEM group running excitations 03:38 UTC Lock loss Starting initial alignment 04:26 UTC Can't get PRC to lock. Giving up on rest of initial alignment. 04:46 UTC DRMI not locking, despite locking and improving PRMI. Trying initial alignment again 05:11 UTC Still can't get PRC to lock. 05:15 UTC Giving up on initial alignment again. 05:58 UTC Observing.
The reason that Patrick had trouble locking PRX (for PRC alignment) last night is that I calibrated the REFL A 9I signal inot volts thinking that it was not used anywhere for feedback, but it is used in PRX.
I've changed the gain for PRX in lscparams to compensate for the calibration change I made last night.
Apologies, and thank you to Patrick for following up.
J. Driggers, J. Kissel, A. Viets After a ton of work this morning (see LHO aLOGs 48534, 48530, 48499, 48546, and 48512) - we are now in observation with the newly moved calibration line frequencies, - the interferometer is stable, - the time dependent correction factors calculated from them are reasonable, - the line uncertainties is low, - the GDS-CALIB_STRAIN channel remains corrected for optical gain, cavity pole change, and TST stage actuation strength (but NOT for PUM, UIM, or any optical spring parameters) - the subtraction pipeline has been updated to subtract out these new frequencies. Attached are some relevant screenshots showing the success. - 2019-04-16_H1CALCS_CalibrationLineMove_Success_7to100Hz.png shows the actuator calibration lines at low frequency, now at 15.6 Hz (driven by UIM, L1), 16.4 Hz (driven by PUM, L2), 17.1 Hz (driven by PCALY), and 17.6 Hz (driven by TST, L3) - 2019-04-16_H1CALCS_CalibrationLineMove_Success_300to550Hz.png shows the sensing calibration line, now at 410.3 Hz (driven by PCAL) - 2019-04-16_H1CALCS_CalibrationLineMove_Success_FinalAnswers.png shows the values of the time-dependent correction factors (note that the spring parameters are still bogus) - 2019-04-16_H1CALCS_CalibrationLineMove_Success_Uncertainty.png shows the current levels of uncertainty in the calibration lines. Great work team, and at lightning speed! Later today, while out of observation mode and the PEM team is working on their injections, we intend to further reduce the amplitude of the lines to better compromise uncertainty and noise impact.
[M. Wade, J. Driggers, J. Kissel, A. Viets]
I've produced new filters for the GDS calibration pipeline based on the calibration model made earlier today (see LHO aLOG 48534). The main changes were:
The filters were produced in SVN revision 7252, and can be found here:
aligocalibration/trunk/Runs/O3/GDSFilters/H1GDS_1239476409.npz
They were produced using the script
/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Scripts/TDfilters/H1_run_td_filters_1239476409.sh
Bode plots of the correction filters for the inverse sensing path and actuation path are attached. They agree with the model to the expected level. More plots will be added soon, once we have some low-noise data after the update to work with.
Using data from the first lock stretch after the calibration model and line frequencies were updated, I have made several more diagnostic plots: