State of H1: in Observe
Site Activities:
Current Weather Conditions:
model restarts logged for Tue 03/Jan/2017
2017_01_03 12:10 h1iopsusauxh34
2017_01_03 12:10 h1susauxh34
Restarted susauxh34 as part of testpoint problem investigation, also restarted h1asc's awgtpman process (which fixed the problem).
Note, DAQ has now been running for 29.9 days (h1tw0 down due to bad raid, h1fw0 uptime is lesser due to previous VPW power problems)
Frigid temperatures on site today with negative windchills
EX temp 13.9(degF) wind 13.0(mph) windchill -0.6(degF)
MX temp 13.4(degF) wind 12.0(mph) windchill -0.6(degF)
CS temp 15.3(degF) wind 07.0(mph) windchill 05.4(degF)
MY temp 11.7(degF) wind 08.0(mph) windchill 00.1(degF)
EY temp 14.0(degF) wind 15.0(mph) windchill -1.5(degF)
wind chill average over site 0.6 degF
TITLE: 01/04 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 71.8083Mpc
INCOMING OPERATOR: Cheryl
SHIFT SUMMARY: Not much happened. a2l dtt template shows high coherence with ETMX and ETMY both pitch and yaw. Been waiting for LLO to go down so I didn't run the script. Keep getting EY temperature alarm.
I just increased the heat by 1ma at E Y.
TITLE: 01/04 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 70.6717Mpc
OUTGOING OPERATOR: Ed
CURRENT ENVIRONMENT: Wind: 21mph Gusts, 15mph 5min avg Primary useism: 0.07 μm/s Secondary useism: 0.20 μm/s
QUICK SUMMARY: Been Locked and Observing when I arrived.
(08:40 - 08:46 UTC) After pausing the code during the holidays they are running again. HWSY is left alone until the glitch issue when running both cameras is solved. I also cleared 5% of space removing ETM data from November. Thining code still has to be rewritten for the new file format (.p).
As I recall, the state if this node had been in WAIT_FOR_NEXT_INJECT. Late in my shift I decided to have a look and noticed that the current state is NONE. Should this node be returned to WAIT_FOR NEXT INJECT?
00:33 Runnnig a2l. DTT ploit shows high incoherence in YAW. Python problems? Not running from the MEDM screen
00:52 Jeff K running calibration measurements
02:44 Checking a2l, post calibration. PIT is showing a pretty good deal of mis-alignment. Going to run the script one more time before setting the intention bit.
03:02 H1 in Observing
Also, PSL AOM diffracted power is running at ≈7%, which by my recollection is kind of high(ish).
He was informing me that they were going to go to Observing. I told him we had been there for a few hours already but he brought to my attention the fact that GWI stat is reporting us as NOT ok. Anyone?
Apologies. We've been at NLN for about that long. In Observation for only about 1 hour.
Seems like H1:DMT-CALIBRATED is 0 (zero) not 1. Is this related to the calibration task performed today?
Is this why GWI stat thinks that H1 is not OK?
Sent a message to Jeff Kissel, Aaron Viets and Alex Urban.
I tried a few things to see if I could figure out why the calibration flag wasn't set. 1) restarted the redundant calibration pipeline, This probably caused some of the backup frames to be lost but the primary and low latency frames would not be affected. The Science_RSegs_H1 process https://marble.ligo-wa.caltech.edu/dmt/monitor_reports/Science_RSegs_H1/Segment_List.html is generating segments from the output of the (restarted) redundant pipeline, but it is getting the same results. 2) Checked for dataValid errors in the channels in the broadcaster frames. dataValid would probably cause the pipeline to flush the h(t) data. No such errors were found 3) checked for subnormal/Nan data in the broadcaster frames. Another potential proble,m tha tmight cause the pipeline to flush the data. No problems of this type were found either. 4) checked pipeline log file - nothing unusual 5) Checked for frame errors or broadcaster restarts flagged by the broadcast receiver. Last restart was Dec 5! So, I can see no reason for the ht pipeline to not be running smoothly.
Alex U. on behalf of the GDS h(t) pipeline team
I've looked into why the H1:DMT-CALIBRATED flag is not being set, and TL;DR: it's because of the kappa_TST and kappa_PU factors.
Some detail: the H1:DMT-CALIBRATED flag can only be active if we are OBSERVATION_READY, h(t) is being produced, the filters have settled in, and, since we're tracking time-dependent corrections at LHO, the kappa factors (except f_CC) must each be within range -- outside of 10% their nominal value, the DMT-CALIBRATED flag will fail to be set. (See the documentation for this on our wiki page: https://wiki.ligo.org/viewauth/Calibration/TDCalibReviewO1#CALIB_STATE_VECTOR_definitions_during_ER10_47O2)
I attach below a timeseries plot of the real and imaginary parts of each kappa factor. (What's actually plotted is 1 + the imaginary part, to make them fit on the same axes.) As you can see, around half an hour or so in, the kappa_TST and kappa_PU factors go off the rails, straying 20-30% outside their nominal values. (kappa_C, which is a time-dependent gain on the sensing function, and f_CC both stay within range during this time period.)
Earlier today, Jeff reported on some work done with the L2/L3 actuation stages (https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=32933) which may in principle affect kappa_TST and kappa_PU. It's possible we will need a new set of time domain filters to absorb these changes into the GDS pipeline. (I also tried a test job from the DMT machine, but the problems with kappas were still present, meaning a simple restart won't solve the problem.)
GWIstat (also the similar display gwsnap) was reporting that H1 was down because of the h(t) production problem; it did not distinguish between that and a down state. I have now modified GWIstat (and gwsnap) to indicate if there is no good h(t) being produced but otherwise the detector is running.
The attached pdf shows that CALCS and GDS agree on the calculation of kappa_tst. I suspect we may need to calculate new EPICS. Jeff (or perhaps Evan or Darkhan) will need to confirm this based on the recent L2/L3 crossover changes that Alex pointed out.
Here is a comparison between h(t) computed in C00 frames (with kappas applied) and the "correct"-ish calibration, with no kappas applied. The first plot shows the spectra of the two from GPS time 1167559872 to 1167559936. The red line is C00, and the blue line has no kappas applied. The second plot is an ASD ratio (C00 / no-kappas-applied) during the same time period. The cache file that has the no-kappas-applied frames can be found in two locations: ldas-pcdev1.ligo-wa.caltech.edu:/home/aaron.viets/H1_hoft_GDS_frames.cache ldas-pcdev1.ligo-wa.caltech.edu:/home/aaron.viets/calibration/H1/gstreamer10_test/H1_hoft_GDS_frames.cache Also, the file ldas-pcdev1.ligo-wa.caltech.edu:/home/aaron.viets/H1_hoft_test_1167559680-320.txt is a text file that has only h(t) from GPS time 1167559680 to 1167600000.
Summary: Calibration measurements utilizing Pcal to DARM transfer functions can be impacted by non-unity gain when making corrections for the modeled frequency response of the AA filtering. At LHO, the ER10/O2 model had an analog AA transfer function with non-unity gain based on the LTI measurements from ER8 (at LLO, the LTI model for ER10/O2 was normalized to 1). The analog AA model at LHO has a gain of ~0.99. Below we detail the impact on sensing function and actuation coefficients. In summary, the sensing function gain is ~1% larger than originally modeled, and the actuation coefficients are ~1% smaller. This would imply that the inspiral range is ~1% higher than currently predicted. This new understanding means that the analysis code needs to be re-run for the optical response parameters and actuation coefficients. The front-end calibration will need to be updated, and, finally, the GDS pipeline needs new filters generated and installed. Details: The calibration of the Pcal channels (ex: H1:CAL-PCALX_RX_PD_DQ) determines the watts reflecting from the ETM per count of the channel at DC. Whatever gain of the analog AA, this is already accounted for in this calibration procedure. This gain is implicitly accounted for when the value of the calibration is installed in the front-end filter module. A Pcal to DARM transfer function was previously understood as follows (note that PD calib., susnorm, m/N coeff are taken care of in the front end and (1+G)*(1/f^2) are taken care of in analyzing the measurements): DARM (IFO opt. resp.) (OMC DCPD TF) (AA(a) freq. resp.) (AA(a) gain) (AA(d) TF) ---- = ---------------------------------------------------------------------------------------------- PCAL (PD calib.) (AA(a) freq. resp.) (AA(a) gain) (AA(d) TF) (susnorm) (m/N coeff.) (1 + G) (1/f^2) where AA(a) is the analog AA, AA(d) is the digital AA, and "freq. resp." means the normalized transfer function, and G is the open loop gain. In the above (incorrect) understanding, the AA(a) and AA(d) terms cancel. The reason this is incorrect is that the AA(a) gain has an inverse in the PD calibration factor. So the real (correct) Pcal to DARM transfer function is: DARM (IFO opt. resp.) (OMC DCPD TF) (AA(a) freq. resp.) (AA(a) gain) (AA(d) TF) ---- = --------------------------------------------------------------------------------- . PCAL (PD calib.) (AA(a) freq. resp.) (AA(d) TF) (susnorm) (m/N coeff.) (1 + G) (1/f^2) Thus, to isolate the IFO optical response, we need to divide out the modeled AA(a) gain. Since the gain is ~0.99, then the gain of the optical response should go up by ~1%. The above equations are laid out in a graphical subway map schematic in G1501518-v14. The actuation coefficients will also be impacted by this, although the coefficients will be multiplied by the AA(a) gain so that the overall DARM OLG remains unchanged. I have pushed the changes to the DARM model code and scripts that account for this. Specifically: computeSensing.m (r4025) create_partial_td_filters.m (r4026) create_full_td_filters.m (r4027) fitDataToC_20161116.m (r4028). Re-running analysis of optical response parameters requires re-running the fitDataToC_20161116.m script first (with printing data to file), then running the fitCTF_mcmc.m script. Re-running analysis of actuator coefficients requires re-running actuatorCoefficients_Npct.m (with printing data to file), then re-running fitActCoefs_Npct.m. Hopefully this takes care of everything. Unfortunately, I cannot verify these changes with Matlab because I have no way of running Matlab offsite (need a network license). :(
For reference to the full paths, these files live at:
{CALSVN}/trunk/Runs/O2/DARMmodel/src/computeSensing.m
{CALSVN}/trunk/Runs/O2/TDfilter/create_partial_td_filters.m
{CALSVN}/trunk/Runs/O2/TDfilter/create_full_td_filters.m
{CALSVN}/trunk/Runs/ER10/H1/Scripts/PCAL/fitDataToC_20161116.m
{CALSVN}/trunk/Runs/ER10/H1/Scripts/PCAL/fitCTF_mcmc.m
{CALSVN}/trunk/Runs/ER10/H1/Scripts/FullIFOActuatorTFs/actuatorCoefficients_Npct.m
{CALSVN}/trunk/Runs/ER10/H1/Scripts/FullIFOActuatorTFs/fitActCoefs_Npct.m