Ed, Jason, Dave:
It looks like the Beckhoff OPC computer in the diode room is not accessible over the network. This machine is called h1pslctrl0.
At first we wondered if the CER network switch had failed, but the digital cameras are working, only the diode room computer is down.
Jason is investigating remotely and Ed is going into the diode room to check on the computer.
Camilla, Sheila
We went back to ISCT6 during the commisioning time this afternoon. We repeated the measurement done here: 54566
We also changed the lens in the path to the temporary AS air 1811 diode. After realigning this we found that we could get a much larger signal out of the 1811. Camilla measured 147mV of DC power on the 1811, with a transimpedance gain of 1V/mA and responsivity of about 0.75A/W this is about 196uW, enough that we could be saturating the diode. Last time we measured the power in the path to this diode, 54636, we saw only 130uW, so there is a bit of an inconsistency there. Not suprisingly, the RF spectrum out of this 1811 is dominated by 45MHz and harmoincs, perhaps so much that we are also having RF saturations. After demodulating at 3.125MHz, we see a large noise signal at 270kHz.
At this point we decided to quit for today and go back to triple coincidence.
TITLE: 01/23 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 118Mpc
INCOMING OPERATOR: Ed
SHIFT SUMMARY:
There was COMMISSIONING time (1hr51min) today.
SEI_DIFF has been in the CORNER_DIFF_CPS state, but Jim noted that since microseism signals are below the 90th percentile, we could switch SEI_DIFF to FULL_DIFF_CPS...with low microseism, this would mainly help us ride through and acquire better with higher winds.
LOG:
S. Karki, J.Kissel, S. Dwyer, C. Compton
We made two set of sensing function measurements during a routine calibartion measurment time on Monday 2020-01-20. We made the first measurement in normal configuration and the second with A2L gain turned off. The measurements, analysis scripts and the results are saved in the appropriate location in the cal svn as listed below:
Measurements:
svn/aligocalibration/trunk/Runs/O3/H1/Measurements/FullIFOSensingTFs/2020-01-20_H1_DARM_OLGTF_LF_SS_5to500Hz_15min.xml
svn/aligocalibration/trunk/Runs/O3/H1/Measurements/FullIFOSensingTFs/2020-01-20_H1_PCALY2DARMTF_LF_SS_5to500Hz_10min.xml
svn/aligocalibration/trunk/Runs/O3/H1/Measurements/FullIFOSensingTFs/2020-01-20_H1_DARM_OLGTF_LF_SS_5to500Hz_15min_A2LGainOff.xml
svn/aligocalibration/trunk/Runs/O3/H1/Measurements/FullIFOSensingTFs/2020-01-20_H1_PCALY2DARMTF_LF_SS_5to500Hz_10min_A2LGainOff.xml
Analysis scripts: (First script collects all past sensing function measurements and compares against these new two measurmnets and the second script compares the new two measurements by themselves)
svn/aligocalibration/trunk/Runs/O3/H1/Scripts/FullIFOSensingTFs/process_sensingmeas_collection_20200120.py
svn/aligocalibration/trunk/Runs/O3/H1/Scripts/FullIFOSensingTFs/process_sensingmeas_collection_20200120_A2LGainOnvsOff.py
Detail results can be found inside the following folder (svn/aligocalibration/trunk/Runs/O3/H1/Results/FullIFOSensingTFs/) but the main plots are attached below.
The first plot contains the comparison between the two measurements made on 2010-01-20 and the second plot compares them against all past sensing function measurements made during O3B.
The goal was to see if the sensing function is different at low frequency between these two configurations and results show that they are. Next step is to create a working model that describe these results.
Managed to get through the tumble weeds to layout the wind sensors. Attached is 6 hours showing the direction sensor and the two speeds starting back up a few hours ago. Speed_4 is on top of the building near the PEM sensor and has continued to run while all the others were off line. I'll put in a comment with an updated location map but I'll say now that all the senors are 39' +Y of the Wind Fence line. Maybe not exactly free-stream but that was all the cable we have. I hope we can extend this to get further +Y and higher altitude. The direction sensor and speed2 are 17' above ground level (AGL) and speed3 is 12' AGL. They are about 21 to 29 feet -X of the Wind Fence X position center.
Here is a sketch of the wind fence anemometer locations wrt the fence and the EndX building.
Also there are a couple photos of the cable route up the slope through the tumbleweeds (they are over head height) and of the sensors with the fence.
Today Jeff K. pointed out that the results from the final position survey of the NCal at EX were never posted to the alog; this post rectifies that.
We surveyed the NCal in 2 ways: we measured the vertical position of the top, center, and bottom of the NCal rotor w.r.t. WBSC09=0.0; we also measured the X, Y, and Z coordinates of a reference punch mark that Timesh placed on one of the support struts for the NCal housing. This punch mark was roughly placed in line with the center of the NCal rotor. Please recall that the H1 ETMx sits at -80.0 mm w.r.t. WBSC09=0.0 (from E1400205). And now, the results:
Please keep in mind that the above coordinates are only for the reference punch mark, not for the center of the NCal itself. Timesh will have to provide those coordinates, as that is a calculation he made that I do not have access to.
I did a quick look at the DetChar pages to see if I could get a sense for the utility of the new wind fences. We do not have the free-stream anemometer running again at EndX, so for now I'm just using the roof-top PEM anemometers to get a qualitative sense.
The 'flip-book' style set of pages below shows that the fences work pretty well.
Before the fence, when the wind is blowing, the histograms of the roof-top sensors are all about the same.
After the fence, the wind speeds at the two ends are clearly lower that the speed at the corner station. By eye, the wind speed is not quite a factor of 2 smaller, but close. The slab tilts at the end are much smaller. (flip between pages 4 and 5). There is a lot to see in these plots, and hopefully they can be used to ask interesting questions that a REAL study can try to answer.
These are also in the DCC at G2000122.
Similar to this alog in Nov 2019 by TJ, noticed DIAG MAIN quickly flashing the message: "BRS_CHECK: BRS X C code has stopped".
Looking at the DIAG MAIN log, we've had a flurry of these errors over the last 24hrs (generally lasting about an hour). A symptom is that during these errors the H1:ISI-GND_BRS_ETMX_VEL channel drops to 0. Attached are three periods of these errors/drops over the last 24hrs. I notified Hugh and he mentioned this could mean a restart of the code would be needed.
Not sure what is going on here, but I don't think this is affecting the actual signals used to do the tilt subtraction for the sensor correction. Attached time series shows some of the other status bits around one of the c-bit drops. Most importantly the RY signal in the lower right doesn't show any drops or glitches during the c-bit drop. I've checked and this also doesn't show up in the tilt subtracted super-sensor (gnd STS - BRS). I think it's probably safe to wait until Tuesday to try restarting the code to try addressing this.
TITLE: 01/23 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 117Mpc
OUTGOING OPERATOR: Cheryl
CURRENT ENVIRONMENT:
SEI_CONF state: WINDY
Wind: 4mph Gusts, 3mph 5min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.28 μm/s
Microseism has continued its downward trend from yesterday (just above 50th percentile). Winds are calm.
QUICK SUMMARY:
TITLE: 01/23 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 117Mpc
INCOMING OPERATOR: Corey
SHIFT SUMMARY: relocked first try, one break in Observe, ETMY violin modes 12, 18, 20 damped
LOG:
New Violin Damping: ETMY mode 20 - needs to be monitored by OPS
With a new damping filter for ETMY mode 20, I have damped modes 12, 18, and 20, with a great deal of monitoring, so be aware that the settings are not tested, have only worked tonight, and not always well, so they're available, but not to be used unattended.
Current Gains for ETMY:
I added 4 new damping filters and one monitor filter into the violin mode filter banks for 5 modes, saved and loaded.
I took 10 minutes to try the new ETMY mode 20 filter, after reaching NLN, and it looked promissing, but I wanted the option to use either the old or new filter, so I unmonitored FM1 (old filter) and FM7 (new filter) in SDF. In order to put a new filter in FM7, I removed an ancient 1.5K violin mode damping filter.
To clear SDFs I accepted ETMY mode 7 changes, in the attached snapshot, and I unmonitored ETMY mode 20 filters FM1 and FM7.
Later, switching the phase on ETMY mode2 caused H1 to be kicked out of Observe, so I unmonitored the phase filter banks, FM2, FM3, FM4, and FM5.
Attachments:
H1 locked in Observe.
TITLE: 01/23 Owl Shift: 08:00-16:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Earthquake
OUTGOING OPERATOR: Ed
CURRENT ENVIRONMENT:
SEI_CONF state: EARTH_QUAKE
Wind: 3mph Gusts, 1mph 5min avg
Primary useism: 0.16 μm/s
Secondary useism: 0.29 μm/s
QUICK SUMMARY: unlocked due to EQ, relocking asap
I'm submitting an early summary as re-locking may not be possible before my shift is over
TITLE: 01/23 Eve Shift: 00:00-08:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Earthquake
INCOMING OPERATOR: Cheryl
SHIFT SUMMARY:
LOG:
07:23 began re-lock attempt
07:51 Failed at DRMI_1F
I am experimenting in trying to manually multiply the all the Kappas and set the cavity pole as at the time of measurement on 20200113, as is being discussed in the last two comments in this aLOG thread.
I used ndscope to acquire Kappas for: UIM, PUM, TST, Optical Gain, and Cavity Pole (not really Kappa, just altered the actual value to the one at time of measurement).
Attached here are two plots:
I am beginning to suspect that the estimate of the Kappa_PUM might be grossly exaggerated? Perhaps the frequency where it is being sensed is not a great choice?
Jeff is currently working hard to get a better figure for the actuation force coefficients. However if what I am suggesting above is true: if he improves the actuation force coefficient for the PUM stage, then the Kappa_PUM would change by the magnitude of the change he makes. At the end of the line, after applying Kappas there would actually be no change at all, and this problem would perpetuate.
[M. Wade, J. Kissel, A. Viets]
Maddie and I have produced new GDS filters for the calibration model update described in LHO aLOGs 54269 and 54473.
I restarted the primary, redundant, and testing calibration pipelines on the DMTs around GPS time 1262990593. Data seems to flowing normally.
The filters are found in revision 9137 of the calibration SVN here:
aligocalibration/trunk/Runs/O3/GDSFilters/H1GDS_1262900044_no_response_corr.npz
They were produced using the run script
aligocalibration/trunk/Runs/O3/H1/Scripts/TDfilters/H1_run_td_filters_1262900044_no_response_corr.sh
Plots of the frequency response of the filters are attached, comparing them to the frequency-domain model. Note the line at ~4kHz in the resudual corrections filter. This is actually in the control correction model, but it shows up the residual corrections filter plot because we normally apply control corrections above 1kHz in the residual path, since the control path is sampled at only 2kHz. It is surprising to something this large coming from the actuation at such a high frequency. Moreover, modeling this accurately would require a much longer filter sampled at at least 8 kHz, which we do not currently have the computational power to do. Given our skepticism, these filters do not model anything in the actuation above 1 kHz. We have an opportunity tomorrow to update the filters again should we decide that it is a good idea to model this the best we can.
Jeff did a Pcal broadband injection just after the pipelines got running, so once the C00 frames are available, I will add GDS results from that injection. I also plan to test these filters on real data to see how well the model the response function once enough data is available.
The first observation ready segment with the updated calibration model start just now at Jan 14 2020 00:47:59 UTC, or GPS 1262998097.
Attached are plots of GDS data during the broadband injection, as well as plots showing how well the filters represent the frequency-domain DARM model. The broadband injection (first plot) looks good, showing deviations no greater than ~2% from 20 Hz - 350 Hz. The last plot shows how well the low-latency (front-end + GDS) calibration pipeline applies the response function R(f). The "spike" seen at 4276.0 Hz is not modeled at all by the filters. The source of this in the model is in all 3 stages of the actuation (only TST and PUM contribute significantly to the response function), and I assume it is a violin mode. This is a very narrow freature in the model, no wider than 0.25 Hz. Models for TST, PUM, and UIM all rise 12 or 13 orders of magnitude at this very narrow feature. We can attempt to model this in the inverse sensing path (which has a high enough sample rate), but it won't be modeled very well if we try, since it is such a narrow feature. Moreover, this would also compromise accuracy in neighboring frequency bins. Most likely, there will still be a loud spectral line in h(t) at 4276 Hz, and the systematic error induced by using the current filters would be that this line appears 3 orders of magnitude lower than it actually is.
Here's a comparison between GDS-CALIB_STRAIN's response to a broadband PCAL Y injection before vs. after this model update. Assuming PCAL is a perfect reference, this should be equivalent to a direct measure of the systematic error in the response function and h(t). One can see that, while we've cleaned up the UIM feature at 153 Hz, and improved the response ratio below 30 Hz, we seemed to made the systematic error worse between 60 and 150 Hz. In the first attachment, I show each of the transfer functions on top of each other, to show the former vs. the current level of systematic error. In the second attachment, I show the ratio of the two transfer functions, to show the *change* in systematic error. This second attachment should correspond to what Vlad predicted in LHO aLOG 54523. It's close... but not quite right. Still investigating... The script used to make this plot can be found here: /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Scripts/FullIFOSensingTFs/ plot_GDS_BB_20200115.py and relies on data processed by Aaron and committed to /ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Results/GDS_BB_plots/ H1_C00_over_CAL-PCALY_RX_PD_OUT_DQ_1262638969-178.txt H1_C00_over_CAL-PCALY_RX_PD_OUT_DQ_1262990871-153.txt
I did some empirical probing into what could be giving this kind of responce between 20 and 300 Hz.
For this I created a new copy of the modelparams_H1_20200103.py file to play with.
The plot attached here, is where I have taken the ratio of my new version over the currently used version, but I have altered:
This is plotted against the very data in the above comment, for reference.
It looks like the current systematic error trend can be ?just about? completely explained by this correction! It seems response in this range is *extremely* sensitive to the value of ccOpticalGain in the model.
EDIT: spoke to Jeff - more convincing to reanalyse this w.r.t the Orange line in his figures in the previous comment, and see if it explains the complete error in response.
For reference, here are the values of the TDCFs that were applied to the data in the GDS pipeline during the broadband injections.
During the injection starting at 1262638969:
kappa_tst = 1.0052612
kappa_pum = 1.0198756
kappa_uim = 0.99628365
kappa_C = 0.99136031
f_cc = 411.13184 Hz
During the injection starting at 1262990871:
kappa_tst = 0.99716723
kappa_pum = 1.017796
kappa_uim = 0.99592042
kappa_C = 0.99556768
f_cc = 410.88696 Hz
We needed a better understanding of the impact of this systematic error at ~150 Hz. for the UIM, so I added a copy of ratio plot from LHO aLOG 54565 to the same script, ^/trunk/Runs/O3/H1/Scripts/FullIFOSensingTFs/plot_GDS_BB_20200115.py and zoomed in around the 100-200 Hz frequency region. Attached are the results. (1) We're, of course, limited by the frequency resolution and noise of the measurement, BUT, (2) We see what Vlad has told us all along: there are actually three features: highQ anti-resonance at 151 Hz, highQ anti-resonance at 153 Hz, and then a high Q resonance at 154 Hz (rounding to the nearest Hz). Note that this description is of the *ratio* of (fixed) / (not fixed), so take my description of whether the feature is a "resonance" vs. "anti-resonance" with a grain of salt. (3) Each highQ feature peaks at around a -2%, -3%, and +3%, BUT -- that includes influence from the "underlying" broad frequency dependent error "sweeping through" this region -- known to be a result of problems with the TST actuator model in this low-latency data.
opened FRS14121
Ed found h1pslctrl0 powered up with a system message on the console. It problem started at 18:11 PST (02:11 Friday UTC).
This is the running log I had started:
02:12 Intenion Bit Commissioning - 357 EPICS channels are lost
From Google translate:
"Twin CAT OPC Server has encountered a problem and needs to close
If you haven't saved your work yet, data may be lost.
Please also report this problem to Microsoft
A problem report that you can send us has been created. We will process this report confidentially and anonymously.
To see what data the report contains, click here"
The LASER_PWR node could not connect to a channel that I thought that we had removed back on Dec. 4th (Camilla alog53682), and I confirmed that it is not in the current userapps code. I can't tell at the moment, but perhaps this node just hasn't ever been reloaded since then to take in the new version of the code that no longer uses that channel for this exact reason.
I had Ed check that a reload happened tonight and we will see if this connection error comes back.
To confirm this does not look like a network switch problem. At this time all digital video cameras are working, h1pslctrl0 is responding to pings but its IOC is not responding to channel connection requests. It would appear the problem is with the twincat/IOC software on h1pslctrl0 and is intermittent.
Earlier this evening in my first attempt to get guardian to ignore the LASER_PWR errors, I added this node to the exceptions list in sys/h1/guardian/IFO_NODE_LIST.py and reloaded the IFO node. Now that we think we have correctly deleted the link between guardian and h1pslctrl0, I have undone my change by reverting IFO_NODE_LIST.py and reloading the IFO node.
Tomorrow we will check that the LASER_PWR node had its code changed to remove the AMP_PWR4 channel but this was waiting for a reload.
While we are having connection problems with h1pslctrl0, the EDC on the CDS overview will continue to show 357 disconnected channels. If this number is larger then something else has failed.
R. Savage, J. Oberling, T. Shaffer, D. Barker, E. Merilh
The PSL Beckhoff PC (h1pslctrl0) is running normally and the laser is running normally; the laser control program is operating as usual and I can manipulate it (change menus, turn watchdogs off/on). It looks like a failure of the OPC IOC Shell that sends the PSL Beckhoff channels to Epics. We cleared the error message and shut down the OPC server; at this point it appears the computer became pingable (it wasn't previously) and we can log into it remotely (which we couldn't previously). We need to restart the OPC server, but noticed that the keyboard on the Beckhoff PC is frozen; it's an old PS/2 connected keyboard so we can't simply unplug it and plug it back in (PS/2 connections aren't hot swappable), it would require a restart of the PC. At this point we decided to wait until the next target of opportunity (lockloss, earthquake, etc) to attempt to restart the OPC IOC shell (if the restart causes further problems, we would rather be fresh to deal with it than tired). Due to this, we will have no Epics monitoring or trending of the laser-based PSL channels until this restart takes place; PMC, ISS, and FSS are all run directly through Epics and not the PSL Beckhoff PC, so are available.
The issue stopping the IFO from going into Observe had to do with a failure of the Laser_Pwr guardian node due to the failure of the OPC IOC shell; the node was complaining that it lost connection to H1:PSL-AMP_PWR4, which comes into Epics via the PSL OPC IOC shell (it used to look at this channel to confirm if the laser was running). TJ, live from Pasadena, looked through the Laser_Pwr node and saw no reference to this channel in the code. At this point our theory is that the old version of the Laser_Pwr guardian node, looking at H1:PSL-AMP_PWR4, was running in this node. Back in December Camilla had edited the node to read H1:PSL-PWR_HPL_DC_LP_OUTPUT (which is not read through the PSL Beckhoff PC); this was done to avoid just this situation. To the best we can tell, it appears the guardian node wasn't reloaded at the next available opportunity and was therefore still running the old code. Once it was running the new code, the Laser_Pwr node worked without problem and the IFO had no issues returning to NLN.