Locked for almost 16 hours and Observing my entire shift aside from a quick out to damp violin modes.
There is a reported earthquake on its way here, but it shouldn't kick us out.
The weekly Fscan is helping to reveal additional comb structure in the H1 data. Similar to the analysis for one day of data late last week (see LHO aLOG 48046), I used the weekly cumulative Fscan to try and find additional combs. I repeated the procedure, but this time used the weekly Fscan of March 30 (see here). Attached is the weekly Fscan showing marked known combs that are currently tracked 0.996795 Hz with 0 Hz offset, 1 Hz with 0 Hz offset, and 1 Hz with 0.5 Hz offset; the two combs I found late last week 5.048889 Hz with 3.365555 Hz offset and 11.394444 Hz with 0 Hz offset; and now one additional comb 0.985915 Hz with 0 Hz offset.
The violin mode Guardian briefly knocked us out of Observing to damp ITMX mode 11. To stop this from happening anymore, I went through the SDF for all the quads and unmonitored all of the gains that the guardian changes, as well as all the TRAMPS. Hopefully I didn't miss any.
Another thing that I found odd, is that when this brought us out of observing, it was because the mode was growing and got turned off. But then 13 seconds later it turned the gain bak on with "gain to the max". Seems odd, but perhaps it's doing what it should.
2019-04-01_16:55:14.636336Z VIOLIN_DAMPING [DAMPING_ON_DC.run] USERMSG 0: ITMX mode 11 is growing!! turning off gain
2019-04-01_16:55:14.638116Z VIOLIN_DAMPING [DAMPING_ON_DC.run] ezca: H1:SUS-ITMX_L2_DAMP_MODE11_GAIN => 0
2019-04-01_16:55:27.257471Z VIOLIN_DAMPING [DAMPING_ON_DC.run] setting MODE11 on ITMX gain to the max
2019-04-01_16:55:27.259129Z VIOLIN_DAMPING [DAMPING_ON_DC.run] ezca: H1:SUS-ITMX_L2_DAMP_MODE11_GAIN => 600
Jeff K, Lilli S,
After trying all the model/parameter options over the weekend, we could not find a model that matches both the interferometer and understandable physics. See alog 48102, 48083, 48080, 48057
The model file matching what is currently installed in CAL-CS is /aligocalibration/trunk/Runs/O3/H1/params/modelparams_H1_20190401_ER14_095A.py
The sensing parameters were not obtained from measurements in a correct way. And there's a gain factor manually added to tune the 3 stages of actuation, in order to reduce the overall systematic error. (See plot in 48102)
We need to continue investigating the problem offline, but this is the model that can be used in GDS, and it should generate consistent results to CAL-CS.
While looking at the FMCS screen during the morning meeting, I noticed AHU 3, Supply Fan 6 was not powered up. I reset the alarm and the fan restarted. Will continue to monitor.
O3 has started!
Tues. Maintenance Activity:
LVEA - Transition to Laser SAFE
Comm TBD Work
All trends look to be reasonably nominal
Richard, Peter, Betsy
This morning at ~7:30 we asked Patrick to take the IFO out of Observing in order to complete the official hardware walkthru. Since the checklist for the end stations from last week was already executed, we only performed the sweep in the LVEA. Tomorrow is a maintenance window so we will be able to check ends once again at noon tomorrow (local time). The checklist status from today:
DONE - Make sure no one is in the LVEA
2 CRANES will be moved tomorrow - Cranes in their "parking spots" & their lights are OFF
DONE - Monitors/work stations are turned OFF (except VAC computers)
DONE - Phones unplugged (wall-wars & RJ11 plugs) & batteries pulled from handsets (Phone locations here)
DONE - Confirm no mechanical shorts onto HEPI.
DONE - Cleanrooms OFF • PSL in Science Mode
DONE - ISC Table fans OFF
DONE - Confirm wifi access points are unplugged (instructions)
DONE - Electronics racks (i.e. make sure no test equipment connected to a rack, unless work permit for it.)
DONE - Forklift NOT connected to charger
DONE - MAY BE a few still there, but not plugged into anything - Unplug unused power supplies/extension cords
DONE - Lights OFF (for end stations check lights via webcams)
DONE - Unplug power supplies for Valcom Paging System and 48V DC H1PSL Phone in Communications Room 163.
DONE - ALOG the LVEA has been swept.
Things out of nominal:
- SQZ IFR signal generator and associated hardware used to drive the SQZ AOM are used in the SQZ rack near HAM6 - these will be replaced/moved in a few weeks.
- Cranes will be moved to parking spot when out of lock during a maint period, then deenergized again.
- The HAM4 cleanroom curtains have sagged since the attempt to stow them last week, and are sitting on the SR3 OPLEV housing. The cleanroom is off and this has been the state for many months (including PEM injections). Since no one has complained that this is affecting H1, we will not try to fix them again.
10:38am local - Turned off 1 of the 3 HEPA flow bench units in the optics lab (which is unused).
I cnfirmed that the 2 Fume Hoods were OFF.
TITLE: 04/01 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC STATE of H1: Observing at 107Mpc INCOMING OPERATOR: TJ SHIFT SUMMARY: Notifications that 'ETMX mode 9 is growing, turning off gain', but did not seem to be an issue. Locked entire shift. In observing entire shift, except for LVEA sweep at 14:13 UTC. LOG: 09:51 UTC GRB notification (GRB E328492) 14:13 UTC Took out of observing to allow Betsy, Peter, and Richard to sweep the LVEA 14:37 UTC Betsy, Peter, and Richard back from LVEA 14:39 UTC Back to observing
TITLE: 04/01 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 105Mpc
OUTGOING OPERATOR: Patrick
CURRENT ENVIRONMENT:
Wind: 9mph Gusts, 6mph 5min avg
Primary useism: 0.04 μm/s
Secondary useism: 0.21 μm/s
QUICK SUMMARY: Official start of O3! Currently locked at 105Mpc for almost 11hrs.
TITLE: 04/01 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 108Mpc
OUTGOING OPERATOR: Jim
CURRENT ENVIRONMENT:
Wind: 4mph Gusts, 3mph 5min avg
Primary useism: 0.04 μm/s
Secondary useism: 0.16 μm/s
QUICK SUMMARY:
Locked and in observing.
TITLE: 04/01 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 107Mpc
INCOMING OPERATOR: Patrick
SHIFT SUMMARY: speeling matrs
LOG:
3:18 Lockloss, re-locking wasn't any big deal, except for a typo in ISC_DRMI guardian during MICH locking, but it was an obvious typo. Fixed and moved on.
Lost lock, relocked with minimal touching, but on getting to NLN, there is an SDF diff. The OSC3 to AO3 element on the LSC outpu matrix is set to 1. Not sure what this means, OSC3 and AO3 aren't exactly descriptive. Have accepted and back to observing.
Hmmm. This should have been set to 0 by the AWG_LINES guardian. Anyhow, since we're out of Observing for sweeps, I have set it to zero and accepted this in SDF. There wasn't an excitation going, and the analog enable switch was off, so there's no problem with this having been accepted, but it's now set to zero.
TJ found an issue with the ext_alert.py code, he has fixed this and I restarted the service at 19:45 PDT.
J. Kissel, L. Sun Remaining confused by yesterday's realization that the IFO needs a right-half-plane complex pair of a poles to represent its low-frequency sensing function detuning (see LHO aLOG 48083), in the interest of forward progress toward the observing run, we decided to try to cut our losses and attempted to install a sensing function compensation filter that minimizes the magnitude and phase error between model and measurement above 10 Hz, regardless of whether it matches measurement below 10 Hz. This attempted failed, resulting in a broadband PCAL2DELTAL systematic error rolling up to 5% / 5 deg between 10 and 40 Hz -- much worse that the poorly-understood, but does-the-right-thing configuration we've been observing in since Friday (see LHO aLOG 48040), which *still* doesn't exceed 1% / 5 deg above 20 Hz (and the phase error cleans up to 2 degs by 40 Hz) -- see discussion of it's stability this morning (LHO aLOG48098). We've reverted to this "ER14" configuration with systematic error at the 1% level, and will continue to think on all this. The plan d'jour in light of this is to generate a pyDARM model that re-creates what's in the front-end regardless from whence it came (LHO aLOG 48012) and how many spring sign bugs there were in the code that generated it (see LHO aLOGs and 48080). This way -- even though we won't be able to reproduce the posterior distributions on these numbers and quantify an uncertainty -- we can at least create a self-consistent CAL-CS, pyDARM world, from which we can generate - reference model values (aka "EPICs records") to compute sensible time-dependent correction factors, and - generate FIR filters for GDS-CALIB_STRAIN. Many more details about our failure today below. DETAILS Picking up where the entry starts -- we want a inverse sensing function filter that reproduces the data above 10 Hz. So, to do this, we ran the *fixed* MCMC code inside, /ligo/svncommon/CalSVN/aligocalibration/trunk/Common/pyDARM/src/ sensing.py Revision: 7085 over both the 2019-03-29 data and the 2019-03-31 data, limiting the MCMC fit region to above 10 Hz. This results are the following: 2019-03-29 Data Optical gain, H_c (ct/m) | 3.245e+06 (+1287,-1276) or (+0.03966%,-0.03933%) Optical gain, H_c (mA/pm) | 4.333 (+0.001718,-0.001704) or (+0.03966%,-0.03933%) Cavity pole, f_cc (Hz) | 410.3 (+0.8639,-0.8516) or (+0.2105%,-0.2075%) Detuned SRC spring frequency, f_s (Hz) | 4.026 (+0.03837,-0.03817) or (+0.9529%,-0.9481%) Detuned SRC spring quality factor, Q_s | 92.61 (+910.3,-1710) or (+10.17%,-5.416%) Residual time delay, tau_c (usec) | -0.2354 (+0.6043,-0.5969) or (+-256.7%,--253.6%) processing script: /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Scripts/FullIFOSensingTFs/ process_sensingmeas_20190329.py 2019-03-31 Data Optical gain, H_c (ct/m) | 3.256e+06 (+1256,-1263) or (+0.03856%,-0.03879%) Optical gain, H_c (mA/pm) | 4.348 (+0.001676,-0.001686) or (+0.03856%,-0.03879%) Cavity pole, f_cc (Hz) | 419.5 (+1.261,-1.286) or (+0.3007%,-0.3066%) Detuned SRC spring frequency, f_s (Hz) | 3.561 (+0.01374,-0.01384) or (+0.3857%,-0.3887%) Detuned SRC spring quality factor, Q_s | 12.75 (+297.8,- 307) or (+4.282%,-4.153%) Residual time delay, tau_c (usec) | 2.031 (+0.8938,-0.8984) or (+44.01%,-44.24%) processing script: /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Scripts/FullIFOSensingTFs/ process_sensingmeas_20190331.py Notice that, as expected, the uncertainty on the Q is quite large. The first two .pdf attachments 2019-03-29_H1_sensingFunction_mcmcModel_vs_measurement.pdf 2019-03-31_H1_sensingFunction_mcmcModel_vs_measurement.pdf show the the goodness of fit for each data set against a model created with the maximum a posteriori values from the MCMC fit. With these two values, we created two model files, /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/params/ modelparams_H1_20190328springfix.py # this model is informed by the 2019-03-29 data modelparams_H1_20190331.py # this model is informed by the 2019-03-31 data and ran a Gaussian process regression on the stacked 2019-03-29 and 2019-03-31 data, given the two models, using /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Scripts/FullIFOSensingTFs/ process_sensingmeas_collection_GPR_model20190328springfix.py # this uses the model informed by the 2019-03-29 data process_sensingmeas_collection_GPR_model20190331.py # this uses the model informed by the 2019-03-31 data and we see, in the 3rd and 4th attachments, 2019-03-28-springfix_ref_model_H1_sensingFunction_GPR_on_AllSensingMeasurements.pdf 2019-03-31_ref_model_H1_sensingFunction_GPR_on_AllSensingMeasurements.pdf that the 2019-03-29 data produces a model with less systematic error (I mean, just barely) for both measurement sets. So -- we installed a filter that had the following parameters: Optical gain, H_c (ct/m) | 3.238e+06 Cavity pole, f_cc (Hz) | 412.448 Detuned SRC spring frequency, f_s (Hz) | 4.308 Detuned SRC spring quality factor, Q_s | 80.1019 Turned it on, increase the gain of the actuator stages from 0.95 to 1.00, and measured the standard PCAL2DELTAL transfer function hoping for resounding success. Instead we find that while (a) The high frequency asymptote is a little better, and the systematic error is nil all the way down 45 Hz, we saw that (b) The phase systematic was worse at all frequencies, and (c) There was now a much larger systematic error below 45 Hz, swelling up to 5% between 15 and 45 Hz. #facepalm So, in desperation, we began exploring the hand-tune-able parameter space: (A) Changing the overall actuator gain again -- see first image attachment, (B) Changing the relative gain of the PUM to other stages, thinking that maybe there could still be a systematic there -- see second image attachment, and (C) Even creating a few other sensing filter options with differing phase and magnitudes -- see third image attachment (D) For posterity, we checked the relative path delay -- this had no effect as before, no image shown While all of these are a wonderful exercise in understanding how each parameter impacts the DELTAL systematic error, we conclude that there is still something dreadfully wrong with the magnitude *and* phase of some or multiple paths, and this really needs a hard think. I conclude with the same plan as above, since we're clearly lost in our ability to translate measurement to filters, so I suggest we recreate what works online in the offline world so we can at least have a world that is self-consistent. This will wreak havoc on how we propagate statistical uncertainty, but that's a problem for another day, for now it is more important to have our systematic error small.
TITLE: 03/31 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 109Mpc
INCOMING OPERATOR: Jim
SHIFT SUMMARY: ~14hr lock with a few CAL measurements.
LOG:
I believe I have removed all of what was left of the fastezca. I only found used instances of it in ISC_DRMI, but there were a few gen states that still have it that aren't used.
If there is a problem during locking in the future, please svn revert (userapps)/isc/h1/guardian/ISC_DRMI.py
2232 Back to Observing
