00:37 UTC Notification of EQ. 00:38 UTC Changed SEI to Earthquake
I've processed the actuator measurements taken on 0424, and produced new GPR results.
The data files are in ^/trunk/Runs/O3/H1/Measurements/FullIFOActuationTFs/
The script is ^/trunk/Runs/O3/H1/Scripts/Uncertainty/process_allmeas_writeGPRHDF5_20190502.py
I have compared the MCMC fitting results from the 0424 meas to the MAP values in the 0416 model file. They are very close:
MCMC UIM: Gain = 1.628 (N/A), model: 1.634, kappaU = 0.996328029375765
MCMC PUM: Gain = 0.02945 (N/A), model: 0.02947, KappaP = 0.9993213437393961
MCMC TST: Gain = 4.417e-11 (N/V**2), model: 4.427e-11, KappaT = 0.9977411339507568
The multi-meas GPR results without kappa correction (first pdf) are not different from the kappa corrected multi-meas GPR results (second pdf). So I decided not to correct kappas.
The full 4 sets of measurements vs the 0416 model plots are in the third pdf.
That further improves the uncertainty estimate. See the png file. Note: it does not include the 0502 sensing meas (48945)
TITLE: 05/03 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 110Mpc
OUTGOING OPERATOR: TJ
CURRENT ENVIRONMENT:
Wind: 6mph Gusts, 4mph 5min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.12 μm/s
QUICK SUMMARY:
Seismon has notifications of incoming earthquakes. The script that plots the appropriate response indicates to do nothing.
TITLE: 05/03 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 108Mpc
INCOMING OPERATOR: Patrick
SHIFT SUMMARY: 29? hr lock. No troubles or anything to note.
LOG:
1542 Vanessa to MX
1700 Kyle to mid stations
2031 Kyle to mids
Philippe, Sharon, Anamaria, Robert
We looked for stray light through the 5 viewports indicated on the first page of Figure 1. We saw stray light from the OFI, but, by far, the brightest light appeared to be from the septum window from the end-cap view. I could not see this bright spot from the viewports on HAM5 or the center viewport on the HAM6 endcap, only from the +X viewport on the HAM6 end cap. This view is not available at LLO. Figure 1a-c show photos, and detail the argument that the spot is forward-scattering, at 3-degrees, of light from the beam spot on the septum window. For example, the parallax photographs in Figure 1c are consistent with the origin being at the septum window. In addition, they show that the spot is bright over several inches at the viewport, and thus more likely to be diffuse scattering than a ghost beam.
We could not see all parts of the septum, but, except for the window, we did not see other spots on the septum (this was also the case for the parts of the septum we could see at LLO - https://alog.ligo-la.caltech.edu/aLOG/index.php?callRep=44985).
Impulse injections, along with shaker injections, suggest that the septum is the dominant scattering site, at least in the band above 20 Hz, at LHO and LLO: https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=48886. The noise produced by septum motion, estimated from PEM injection results, is consistent with the loss of range due the HVAC at LHO: https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=48912. The observation of bright scattering from the LHO window, but not from other examined regions of the septum at either site, is evidence that the septum window could be the scattering site on the septum.
Figure 2 shows a plot from LIGO-T0900269-v2 that is a reminder that we expect scattering from the septum window to be close to limiting. This, along with the visual observation evidence and the evidence from the shaker and impulse injections, all accumulate to suggest that it may be time to consider removing the septum windows at both sites.
On Tuesday 23rd April I turned on a ZFS scrub for the /opt/rtcds file system and we got a skew of h1edc CRC errors until I stopped the scrub.
To investigate this further, I wrote a script which creates a 1GB file on both the /ligo and /opt/rtcds file systems from a workstation. This runs every 5 minutes and writes the disk-access speed into two EPICS records (units = MB/s).
At 04:49 PDT this morning we got 4 CRC errors, which in this case correlated to a slow down of /opt/rtcds access speed of 30%.
Attached plot shows past 48 hour trend of h1edc CRC (upper ndscope plot) and file access speed (lower strip-tool) which have been aligned on their x-axis. The strip tool shows /opt/rtcds access speeds (blue line) and /ligo (green line).
Note:
Both /ligo and /opt/rtcds slowed down this morning between 2am and 9am, with /opt/rtcds significantly slowing around 5am.
The previous CRC error of 3 counts from almost 48 hours ago is not related to disk slow down.
/opt/rtcds/ also slowed between 4am and 6am Thursday morning.
Jonathan checked that network statistics between the hours of 2am and 9am this morning, nothing unusual was found (core switch showed a slight traffic increase between 2am and 4am).
Jonathan, Dave:
One way a NFS file system slow down could impact h1edc is during its periodic calculation of the file-checksum for the file H1EDC.ini. To further test this, Jonathan made a RCG-branch3.5 change on the DTS to take this checking out of x1edc (alog link below). This has been running for 90mins with no errors. Overnight last night x1edc raised 700+ CRC errors, so by Monday we should have a definitive result.
https://alog.ligo-la.caltech.edu/TST/index.php?callRep=12509
Dave pointed this out. It seems that the segment 1 of the EY ASC TR B QPD railed after 18hours of the current lock. I'm guessing that we aren't using this PD during the lock, but there is a clear drift to its rail/saturation point. Trending a bit back, there are a few small times that will saturate, but nothing for this amount of time (going on ~7 hrs at this point).
I attached a trend of this lock with all four quadrants. I guess the other odd bit is that SEG3 has drifted to near 0, so we must be moving off this PD.
Might be nothing, but worth noting here.
TITLE: 05/03 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 109Mpc
OUTGOING OPERATOR: Corey
CURRENT ENVIRONMENT:
Wind: 5mph Gusts, 3mph 5min avg
Primary useism: 0.01 μm/s
Secondary useism: 0.11 μm/s
QUICK SUMMARY: 23hr lock, rode through a nice earthquake last shift.
TITLE: 05/03 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 108Mpc
INCOMING OPERATOR: TJ
SHIFT SUMMARY:
Focused on getting through an EQ at beginning of shift. H1 has been locked for almost 20hrs (so no opportunity to address phase of AS_A_36) and observing for 16+hrs with a range around 110Mpc.
LOG:
h1 has been locked for almost 16hrs (observing for 12+) & this is with riding through 6.1 south pacific EQ (now gone from blrms but was elevated for 3.5hrs). Range hovering around 111Mpc as we run with Triple Coincidence.
Summary: Below is a rough summary of how things went down for H1 to ride it through an earthquake. The notification came just as Jim was leaving so we discussed scenarios before he left. (I was also distracted by a pair of coyotes on our cameras.)
Attached are screenshots of various tools to show a narrative of EQ tonight.
EQ Log:
TITLE: 05/03 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 108Mpc
OUTGOING OPERATOR: Jim
CURRENT ENVIRONMENT:
Wind: 14mph Gusts, 12mph 5min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.12 μm/s
Quiet seismically: Sub-50th percentile microseism & low winds.
QUICK SUMMARY:
Incoming Earthquake!
And a 6+ Solomon EQ is shaking the earth. Talked with Jim about this and monitioring.
Have transitioned to EARTHQUAKE state (when I saw action for ASC & tidal signals). Have not transitioned out of OBSERVING to take ISC_LOCK to the EARTHQUAKE state.
Making notes of what I'm observing and any actions.
TITLE: 05/03 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 104Mpc
INCOMING OPERATOR: Corey
SHIFT SUMMARY: Nothing happened, porcupines seem to be back.
LOG:
J. Driggers, J. Kissel, L. Sun
We've made several advances in knowledge with today's collection of bi-weekly sensing function measurements:
(1) I was able to successfully complete PCAL to DARM IN1 transfer functions out to 5 kHz with and extra template. It takes an extra 20 minutes. This data is still of poor quality, so it doesn't improve the estimate of response function systematic error *much* but it does a little. Pep and I continue to work on analyzing the roaming PCALX calibration line so we have better data quality up there and confirm our supposition that the error / uncertainty is not as large as is falsely reported by the Guassian Process Regression. Lilli will comment more on how much this high frequency data improves the uncertainty (spoiler: it's not much).
(2) We've been continuing to suffer from ~2 minute period oscillations in ASC that result in reported arm power fluctuations at the peak-to-peak of 2 kW (!!). A first attempt at re-phasing AS 38 (i.e. MICH ASC sensors) "fixed" it on Monday for the overnight long lock stretch (LHO aLOG 48857), but that phase change was reverted after failing at to re-acuire lock after Tuesday maintenance (LHO aLOG 48878). Jenne adjusted the phase just before I began measuring today, but (unknowingly at the time) was using the wrong metric. Thus, these sensing function measurements may be corrupted by the oddball phasing of AS36, because the oscillations would come and go during my sweeps.
Evidence points toward us needing to dynamically change the WFS phase, having one setting for lock-acquisition, and another after the IFO has thermalized. Jenne has written m,ore on this in LHO aLOG 48948.
(3) In that semi-stable, some-times oscillating configuration, with the oddball phasing, apparently, the DARM optical plant changed. The most significant change is that the SRC optical spring frequency changed from the reference 4.5 Hz to a lower 2.3 Hz. The DARM coupled cavity pole reduced from 410 to 405 Hz, and the optical gain increased by 1%. Jenne has re-rephased the AS36 WFS *after* I've taken this data, so I want to re-confirm with more measurements in this new-new AS36 WFS phase configuration before I set off the "we need to change the reference model" alarm bells.
(4) Unfortunately, even though it's dreadfully obvious that an arm power fluctuation at the 2 / 170 = 0.011 = 1% level should be reported in the time-dependent correction factor for the optical gain, kappa_C, we see no such reported fluctuation in these TDCFs. We also see no such fluctuation reported in the DARM coupled cavity pole, f_cc. This is likely because the fluctuation is faster than the integration / average time of the calibration lines (currently 13 averages of 10 sec FFTs, i.e. effectively only giving an update every 2 minutes, and thus this two minute period oscillation is averaged away)
Below are details about the data, and how we've processed it in order to come to these conclusions.
Attached are three plots:
(a) 2019-05-02_H1IFO_OpticalPlant_Fluctuations.png A trend showing the supposed time-dependent correction factors for the optical plant against arm powers and various ASC control / error signals that show the same behavior. (Remember, because the optical spring still has unphysical phase rotation at low frequency, the tracking of the optical spring parameters are bogus, and thus have been turned off in the real time system).
(b) 2019-05-02_H1_sensingFunction_noMCMCTDCFcorr.pdf A collection of plots of the processed sensing function *if the measurement is not corrected for any time dependence* where the time dependence is measured by the ratio of MCMC fit values (i.e. not from estimates of those values from calibration lines).
(c) 2019-05-02_H1_sensingFunction_withMCMCTDCFcorr.pdf The same collection of plots, but now correcting the measurement for the MCMC values from (b), and thus "propagating it back in time" to the reference values.
One can see from (b), and specifically page 5, that today's data set is definitely an outlier -- all other days have not been corrected for anything. Similarly, looking at pages 3 and 4, comparing the reference model against today's measurement leaves a large residual, and "screws up" the Gaussian Process Regression, or rather, the GPR is accurately reporting large systematic error.
Once we correct the measurement for the difference in MCMC parameters, the results in (c) show things clean up better.
Again, we should not create a new model, nor update any of the low-latency data stream yet. We should gather more data, as we continue to settle on a ASC configuration to combat this oscillation issue.
%%%%%%%%%%% Details %%%%%%%%%%%%
2019-04-04 Reference Model Parameters are from LHO aLOG 48378.
The detailed answers from the MCMC fit of the *uncorrected* data are as follows:
2019-04-04 2019-05-02
Reference New Values
Overall gain, H_c (ct/m) | 3.250e+06 3.281e+06 (+1490,-1382) or (+0.0454%,-0.04213%)
Cavity pole, f_cc (Hz) | 410.6 404.3 (+0.4895,-0.5211) or (+0.1211%,-0.1289%)
Detuned SRC spring frequency, f_s (Hz) | 4.468j 2.269 (+0.1046,-0.1181) or (+4.611%,-5.205%)
Detuned SRC spring quality factor, Q_s () | 52.14 94.15 (+902.2,-2156) or (+10.44%,-4.368%)
Residual time delay, tau_c (usec) | 1.0 0.9597 (+0.2495,-0.2655) or (+ 26%,-27.66%)
The resulting TDCFs from the comparing the above new MCMC results against the reference model:
kappa_c = 1.009538 (New Value / Reference Value)
f_cc = 405.1 (New Value)
f_s = 2.292j (New Value, entered as complex)
Q = 94.18 (New Value)
Because we've yet to figure out a good system for correcting sweep measurements for time dependence, these have been added to the list of hard-coded corrections in
^/trunk/Common/pyDARM/src/sensing.py
and we chose to either invoke them, or not invoke them in the following analysis scripts, used to process today's data:
^/trunk/Runs/O3/H1/Scripts/FullIFOSensingTFs/process_sensingmeas_20190502.py
^/trunk/Runs/O3/H1/Scripts/Uncertainty/process_allmeas_writeGPRHDF5_20190502.py
and those scripts have been committed to the repo.
The data from today lives here:
^/trunk/Runs/O3/H1/Measurements/FullIFOSensingTFs/
2019-05-02_H1_DARM_OLGTF_LF_SS_5to1100Hz_15min.xml
2019-05-02_H1_PCAL2DARMTF_BB_aftersweeps.xml
2019-05-02_H1_PCAL2DARMTF_BB.xml
2019-05-02_H1_PCAL2DARMTF_HF_SS_600to5000Hz_15min.xml
2019-05-02_H1_PCAL2DARMTF_LF_SS_5t1100Hz_10min.xml
There are broad band sweeps "in the can" before the measurements started (though after Jenne phased the AS36 WFS), and after the measurements were complete (but before Jenne re-re-phased the WFS with a better metric). They show no evidence time dependence during the measurement, so I've not processed them in any more detail.
You'll note there are LF and HF templates for the PCAL2DARMTF. That data was exported and *by hand* stitched together to create the data shown in the attached plots. Those stitched together .txt files are
^/trunk/Runs/O3/H1/Measurements/FullIFOSensingTFs/
2019-05-02_H1_DARM_OLGTF_Stitch_SS_A_DARMIN2_B_DARMEXC_coh.txt
2019-05-02_H1_DARM_OLGTF_Stitch_SS_A_DARMIN2_B_DARMEXC_tf.txt
2019-05-02_H1_PCAL2DARMTF_Stitch_SS_A_PCALYRX_B_DARMIN1_coh.txt
2019-05-02_H1_PCAL2DARMTF_Stitch_SS_A_PCALYRX_B_DARMIN1_tf.txt
We should figure out a better way to merge this data.
Including the high freq data points can improve the systematic error from 7.5% (second figure) to 5% (first figure) at ~2kHz.
This is obtained by using the multi-measurement GPR including the time dependence corrected 0502 measurement (page 6 in Jeff's 2nd attachment).
The dots on the first plot indicate the corrected sweep measurement taken today. Due to the questionable data quality, we have not included it in the latest uncertainty estimate. This plot here shows the slight improvement at high frequency with only a few poor data points. We expect further improvement when more measurements at high frequencies are available.
About an hour and a half after lock acquisition we were still getting the yaw ASC oscillations. I looked at moving MICH ASC to the AS 72 MHz signals (which are in use for SRC alignment), but there is very little signal there. I did get a single-frequency transfer coefficient by driving BS in pitch quite hard and tried blending the RF72 signal in with the RF36 signal (with the idea of trying to turn off the RF36), but the noise increase in MICH ASC was quite dramatic. I unblended, and left MICH ASC on the nominal AS_A_RF36_Q signals.
I then moved the phase of AS_A_36 and watched the ASC yaw oscillations and circulating power buildups, and ended up with an answer 5 degrees different than Georgia's (alog 48857) when she checked the phase in full lock. This is good, although not so great if we can't reacquire lock with this phasing.
I also tried moving the SRC1 offsets in both pitch and yaw to see if I could further improve any buildups or the cavity pole. In the end, having no offsets seemed to be pretty close to the best I could find.
I see that the noise between 20 Hz and 30 Hz is slightly elevated, very similar to what Andy pointed out in alog 48881 after Georgia initially rephased AS 36 on Monday. I ran A2L on ITMX just in case it was some slight change in spot position on that optic, but that didn't improve things. I looked at Andy's bruco, but it shows coherences with REFL WFS signals, which I don't really see right now when I look at a dtt version of coherences (see attached figure). I do see coherence with LSC-MCL, which is a little bit surprising. And, we still have the old friend DC2 Yaw.
z write H1:ASC-AS_A_RF36_SEG1_PHASE_R -175 H1:ASC-AS_A_RF36_SEG2_PHASE_R -190 H1:ASC-AS_A_RF36_SEG3_PHASE_R -170 H1:ASC-AS_A_RF36_SEG4_PHASE_R -175
Once we're locked, after about an hour and a half you'll likely see ASC yaw oscillations (most noticeably in the blue INP1 trace on nuc6). When this happens, copy the following into a terminal:
z step H1:ASC-AS_A_RF36_SEG1_PHASE_R +1,40 H1:ASC-AS_A_RF36_SEG2_PHASE_R +1,40 H1:ASC-AS_A_RF36_SEG3_PHASE_R +1,40 H1:ASC-AS_A_RF36_SEG4_PHASE_R +1,40 -s 1
This should step each segment's phase up by 1 degree, 40 times, with 1 second waits between each step. So, this will require about 1 minute out of observing time, but will help the IFO be more stable.
Can the RF36 phase change be deduced from the SUM channels, ie. atan (Q/I)?
After running another bruco analysis (https://ldas-jobs.ligo-wa.caltech.edu/~pep.covas/bruco/1240824092/) I found coherence around these two lines with several channels related to SRM and PRM.
I found these lines in the coil driver monitors of PRM and SRM, and also in some sensors like LSC-REFL_A_RF45_I_ERR_DQ and LSC-REFL_B_RF9_Q_ERR_DQ. I traced these signals back to the controls signals of the loop LSC-SRCL_OUT and LSC-PRCL_OUT, which also show increased coherence around these frequencies.
I checked more channels with DTT, and apparently these lines are there for LSC signals, but not ASC signals. Jeff has suggested that some notches around these frequencies could be applied to the LSC loops in order to suppress these lines.
More coherent lines found:
17.8 Hz, see first image
27.71, second image
WP 8188
I made I slight adjustment to the light level on the PMC locking PD to hopefully alleviate the constant increase in the relock counter we've been seeing lately (LHO alog 48568 & FRS 12753). I began by tweaking the alignment into the PMC locking PD, which did not change the PD voltage. I then slightly tweaked the TFP directly in front of the PD to increase the PD voltage; this had a small effect on the PD voltage. The changes are:
I did not change the PMC gain. I reset the PMC relock counter at 18:44 UTC (11:44 PDT) and will monitor this over the next few days before closing the work permit.
FAMIS 10708
Both PSL power watchdogs were reset at 18:35 UTC (11:35 PDT). This completes FAMIS 10708.
Other Work
I happened to notice that the ISS diffraction % was at ~1.7% with the just the 1st loop locked; the diffracted power should be between 2% and 3%. I changed the ISS Reference Signal from -2.08 V to -2.07 V; this brought the diffraction % to ~2.1%. The change was accepted in both the SAFE and OBSERVE SDF files.
