corey.gray@LIGO.ORG - posted 02:12, Friday 07 June 2019 (49726)
PSL Chiller Water Level Top-Off
Added 125mL to the Crystal Chiller. Diode Chiller was good. Filters both looked OK.
Added 125mL to the Crystal Chiller. Diode Chiller was good. Filters both looked OK.
TITLE: 06/07 Owl Shift: 07:00-15:00 UTC (00:00-08:00 PST), all times posted in UTC
STATE of H1: Observing at 111Mpc
OUTGOING OPERATOR: Cheryl
CURRENT ENVIRONMENT:
Wind: 7mph Gusts, 5mph 5min avg
Primary useism: 0.01 μm/s
Secondary useism: 0.13 μm/s
Microseism below 50th percentile (woo woo!) & currently looks like the wind has died down.
QUICK SUMMARY:
TITLE: 06/07 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 113Mpc
INCOMING OPERATOR: Corey
SHIFT SUMMARY: locked all shift, calibrations from 5-7PM local (00:00-02:00 UTC)
LOG:
J. Kissel
Since it's been a while since we've gather a complete set of calibration measurements, I spent a bit of extra time gathering all measurements for both sensing and each stage of the ETMX actuator. The full suite took XXX hours (where the longest template is the PUM stage excitation at 22 minutes; the rest are each 10-15 minutes).
The data lives here:
Sensing
/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Measurements/FullIFOSensingTFs/
2019-06-06_H1_DARM_OLGTF_LF_SS_5to1100Hz_15min.xml
2019-06-06_H1_PCAL2DARMTF_LF_SS_5t1100Hz_10min.xml
Actuation
/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Measurements/FullIFOActuationTFs/
2019-06-06_H1SUSETMX_L1_iEXC2DARM_10min.xml
2019-06-06_H1SUSETMX_L1_PCAL2DARM_8min.xml
2019-06-06_H1SUSETMX_L2_iEXC2DARM_22min.xml
2019-06-06_H1SUSETMX_L2_PCAL2DARM_8min.xml
2019-06-06_H1SUSETMX_L3_iEXC2DARM_8min.xml
2019-06-06_H1SUSETMX_L3_PCAL2DARM_8min.xml
Things even went so well that I grabbed some bonus data -- a repeat of the sensing function measurement set, but during which I've gathered all ASC Arm DOF error signals and the ETMX optical lever as additional response channels. This was motivated by the suspicion that there are L to A to L cross-couplings that are fooling us into thinking that the SRC is acausally detuned. That data is here:
/ligo/svncommon/CalSVN/aligocalibration/trunk/Runs/O3/H1/Measurements/FullIFOSensingTFs/
2019-06-07_H1_DARM_OLGTF_LF_SS_wASC_5to100Hz_10min.xml
2019-06-07_H1_PCAL2DARMTF_LF_SS_wASC_5to100Hz_8min.xml
And I attach screenshots of the transfer functions and coherence between these new angular channels and each respective DARM (L) and PCAL (L) excitation [First and second attachment, respectively].
Angle is definitely coherent, especially in DHARD P -- and strangely the coherence increases with frequency for the PCAL excitation, yet decreases with frequency for the DARM excitation. Fascinating clues...
(Many sincere apologies, but I do not have a calibration for any of the ASC error signals. The optical lever calibration is at least known -- and displayed in urads in the plot.)
All data has been exported and committed to the repo in the above locations with corresponding names.
Data will be processed further in due time.
A line at 333.324 Hz has been present since the start of the run. I just recently realised that this line is part of a comb in the spectrum with that spacing (see first attachment), and the first six lines of this comb can be seen with SFTs of 1800 s averaged over ~one week of data. The amplitudes of these lines seem to be quite constant over the run.
I found that the line at 333.325 Hz is coherent with some ETMX channels: L1_NOISEMON, L1_MASTER, L3_MASTER, L3_LVESD... These lines are also at the LSC-DARM_OUT, OMC-DCPD_A and OMC-DCPD_B channels, as can be seen in the second attachment.
It is a little bit strange that the second stage L2 is not showing the line, but it could happen if the noise monitors for L2 have a higher noise floor. The line does not appear at any ETMY channels. Since the lines appear in all these channels I couldn't pinpoint where it is coming from, although given its very high Q it seems to be a digital feature.
TITLE: 06/06 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 114Mpc
OUTGOING OPERATOR: Ed
CURRENT ENVIRONMENT:
Wind: 18mph Gusts, 15mph 5min avg
Primary useism: 0.05 μm/s
Secondary useism: 0.12 μm/s
QUICK SUMMARY:
TITLE: 06/06 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 115Mpc
INCOMING OPERATOR: Cheryl
SHIFT SUMMARY:
Quiet shift: There were a couple of brief periods out of Observing to correct some oscillations due to too much ADS gain and the strange DBB intrusion as posted in my previous log.
LOG:
18:16 Gerardo to MX
14:55 Chandra to MY
Summary: Looking over O3 H1 data so far, spectral lines in 1800 s long Fourier transforms continue to be problematic and will be degrade astrophysical results of continuous gravitational wave searches. It would be helpful if commissioning/engineering teams can compare with what the LLO team has done in order to mitigate most spectral contamination. I'll note that I'm getting greedy because O3 data is already much better for spectral content than O2 data for both detectors (and better broadband sensitivity!). Again, my thanks to the entire team on the progress over O2! When I see how clean of spectral features L1 data appears compared to H1, however, I want both detectors to have equally clean data. I hope some improvements to hardware configuration will mitigate the H1 spectral lines, but this needs attention from boots on the ground at LHO. Details: Keith Riles has a very nice week-by-week and running cumulative pages for looking at H1 and L1 spectral data computed from the standard set of 1800 s long, Tukey windowed Short Fourier Transforms--SFTs (see here for H1 and here for L1). I want to draw attention in particular to two problematic features that require attention: 1) Below 100 Hz, the prevalence of lines in H1 data is much greater than L1. See attached figure 1 (H1) compared with figure 2 (L1) showing both the most recent week of data and the cumulative O3 data up to now. 2) The behavior of the spectral features is quite dynamic from day to day and week to week. This can be seen, for example, in attached figure 3 showing the ratio of this week's H1 data divided by the cumulative H1 data. Spikes above the mean show an increase in line activity, while spikes dropping below the mean show a decrease in line activity. Every week shows a slightly different figure, meaning the lines and/or coupling is changing with time. Figure 4 shows the L1 data, which is much better behaved, probably because there are so few lines present in the data. These features are likely going to limit some searches for long duration gravitational wave signals, and may obscure putative signals that could be detectable during O3. With lines present in the data, especially when the lines are dynamic as they appear to be in H1, will require changes to searches and increase the amount of time spent to deal with these artifacts in post-processing. With improvements (as demonstrated by L1), the searches will be more sensitive and can be published more timely.
FRS ticket 13115 created, link given below,
https://services.ligo-la.caltech.edu/FRS/show_bug.cgi?id=13115
For other reasons, Marie made a comparison of HAM6 motion between L1 and H1, which shows many more lines at LHO than at LLO. Are there lines at the same frequencies as the lines in DARM? (I made this comment on a different alog of Evan's, but since this log is the one which is in FRS I am repeating it here.) https://alog.ligo-la.caltech.edu/aLOG/index.php?callRep=46603
Apparently this occurred at 20:20 UTC. I had muted Verbal Alarms while Amber was in the control room recording and interview with a TV reporter and forgot to UNmute. Apologies for that one. The SDF diff that occurred is still a bit of a mystery and perhap[s be UNmonitored?
20:30 Accepted diff and returned to OBSERVING
The DBB is offline, not to mention physically no longer on the table.
Since the DBB is not a part of H1 operations, should we UNMONITOR? (Why did this change during OBSERVING?)
Yes, I think the DBB should be unmonitored. I don't know why it would have changed while we were in observe though.
Entered FRS ticket 13107 for this.
First saw it on the YAW Control strip tool in front of me (1st attachment), but then saw it in the 40min Strip tool as well. It is mostly gone, but can still faintly see it.
W went out of observing very briefly to remove some gain from the ADS yaw loops, who's gain we had increased yesterday. It looked as though this oscillation could be a cross coupling between the ADS and SRC2 loops. Several times yesterday we noted that the SRC2 loops are bringing their error signals to 0 very slowly, so we might need to increase these gains.
Since the gain reduction seems to have gotten rid of the oscillation, I've edited the guardian to keeps the ADS gains lower for now.
The next time the interferometer looses lock, loading ISC_LOCK should make these changes happen.
LOAD was executed at 10:58utc on June 7th.
Noticed a new message (other than Violin Damping) which says: ALS_YARM: Stolen (by: User)
I reckon this must have been me when I had to touch up the Y-arm when I was trying to relock. Not really sure what I could have done differently to steal that node.
"Stolen by USER" will show up on a manager node when one of its subordinates has a new manager. In this case, it is USER. User means that someone had manually clicked on the medm to manage that node. I will change this node to be managed by ISC_LOCK as ALS_XARM is, otherwise we may run into some stalling when there is a lock loss.
I wonder when/how this happened. I don't think I changed the MODE (from ALS_YARM being MANAGED). I did select UNLOCKED for it....would that have been the reason? Hmmmm.
That shouldn't have been the reason. My guess is an accidental clicking of the MANAGE button from the "all states" screen is what happened.
Not sure if this is real (it looked real on DARM spectrum!), but H1 made it up to about 176.7Mpc. And it was in this higher-range state for about 10min. (attached is the last 6hrs of H1 running with first lock on left at 37 W and the last two locks at 35W).
There were multiple ADC overflows in the DARM loop flagged in the database during this period, triggering automated h(t) gates. The DQ times flagged (integer second discreteness) were 1243843959 1243843960 1243843967 1243843969 1243843975 1243843977 1243843981 1243843984 1243843985 1243843988 1243843989 1243843990 1243843991 1243843994 1243843995 1243843996 1243843997 1243843999 1243844000 1243844004 which span 08:12:21 UTC - 08:13:06 UTC.
Andy, TJ, Laura This looks like it was a PI mode ringing up. The broadband monitor (plot 1) sees a nice exponential ringup. The high-bandwidth DCPD channel shows that the line is about 10,430 Hz (plot 2). As Keith posted, the DCPD began to saturate due to the huge line, and plot 3 shows that this looked like a decrease in shot noise on the DCPD which is why the range looked unrealistically high.
From mode identification before the change in test masses https://dcc.ligo.org/LIGO-T1600080 it looks like this could be one of the ~10420Hz mode group. The mechanical eigen-frequency model from this dcc implies this is likely the drumhead mode. No measured mode is within 1Hz of this frequency indicating its probably one of the new test masses. This mode would best be actuated on with the ESD with all four quadrants driven with the same phase. There is also a more complicated mode with a similar frequency, see the 3 and 4th attached figures. And there is also the possibility that this mode is aliased (real frequency 55106) though this is unlikely given there are AMDs.
It looks like this mode also rang up slightly 5th june when the power was increased. It appears that ring heater were not adjusted for this power up. I guess applying some more ring heater will likely avoid this instability.
Lockloss at 8:16 (lockloss file)
Reverted to 35W per Sheila's alog and noted in my earlier "Back to Observing" alog. Here is the "Operator Play By Play" for locking.
NOTE: This is after two new states:
Play By Play Notes:
Additional NOTE: I wasn't sure whether to mark downtime as "locking" or "commissioning". I did a little bit of both, but mostly "commissioning", because I had to undo some high power stuff and then had to deal with new procedure for ISC_LOCK from Tues work.
10:04 Back to OBSERVING
In regards to the ALS_X/YARM in UNLOCKED, on the main guardian screen for a node, there is a Request area below State and Target. This area has 2 fields, an EPICS readout and a state selector. The state that the state selector reads is not an accurate way to judge what state we are in, hence the EPICS readout right above it. There are only so many states that can fit on the state selector, so if a state is requested from a manager that is not in the selector, it will not be able to show it there. Recent changes have added new states to this node, most likely pushing Unlocked off of the state selector.
Hopefully this clears it up a bit.
Thanks for the explanation, TJ!
I really wanted to note/document what I saw because it was different from what I had been used to & so it had me scratching my head a little. But I do remember there were changes made on Tuesday. Anyway, I've had experience with the new way it works now and think I have it down. THANKS!
As mentioned in my transition summary, noticed that the ~48Hz line happened to be noticeably higher than normal as I walked in this corresponded to a noisier DARM seen via DARM BLRMS (also noticed by Cheryl). Apparently, it looked like 48Hz line would get rung up by certain types of glitches and then have effects on DARM...which looked negative during Cheryl's shift and part of mine.
Then at 7:55:50utc, I noticed a BIG glitch. For a few minutes I watched the DARM spectra with an eye on the big 48Hz line just so I could get some screenshots of that fundamental and its 2nd harmonic. Attached are a couple of them (one where the fundamental/2nd harmonic are "fat" & the other where they are higher in amplitude). See first two attachments.
While looking at these two noisy lines, all of a sudden noticed a big change in DARM: The Bucket Dropped!
- From about 80-200Hz noticed that DARM was well below reference and the range went up to something like 120-140Mpc! (see 3rd attachment)
- Then a few minutes later, higher frequencies followed suit and DARM was below reference from 70Hz well above 7kHz! (I glanced up at the range and saw saw us hit over 170Mpc at one point!). (see 4th attachment)
- All of this weirdness corresponds with a drop in DARM blrms in the 100-450Hz band taking a notable drop (see 5th attachement)
- 6th Attachment is a screenshot of the BNS Range plot where H1 goes off-scale for a few minutes.
It should be noted for the high range observed, there were also many more lines/harmonics appearing in DARM from 200Hz and above. Lasted a few minutes at these high ranges, but then started getting "DCPD" Verbals as well as low frequency (10-40Hz band) bump/glitches. Then H1 just dropped out of lock after a few minutes of this weird behavior.
We should have a look at what the calibration metrics are doing during this time. Was there a hiccup in calcs? The calibration lines don't line up properly in the 2 spectra that have higher reported range.
Though I haven't yet checked calibration metrics, this appears to have been a result of the OMC DCPD's saturating from a PI mode (and converting the entire upper-end of the spectrum into a featureless white noise) -- see LHO aLOG 49704. The investigation continues...
Laurence Datrier, Sheila Dwyer
A repeat of the analysis in 49542 along with a look at the impact of wind on BNS range.
We used the maximum wind speed of the three max. 1min and 30min trends for the H1 EX, EY, CS wind channels for O3 up to 2 June 2019.
The first histograms and trend line show duty cycle vs wind speed percentiles, for wind speed vs lock status and previous wind speed (max. of previous 1 or 30 min) vs lock status. Looking at the previous wind speed shows more down time due to high winds. Percentiles and wind speed values are (in mph):
| percentile | 5 | 10 | 20 | 30 | 40 | 50 | 60 | 70 | 80 | 90 | 95 | 99 |
| 1min trend wind speed [mph] | 4 | 5 | 9 | 8 | 9 | 11 | 12 | 14 | 17 | 21 | 25 | 35 |
| 30min trend wind speed [mph] | 6 | 7 | 9 | 11 | 13 | 15 | 17 | 19 | 22 | 27 | 31 | 43 |
The following histograms are the normalised distributions for wind speed at:
Compared to the normalised distribution at all times (blue).
The following plots look at the BNS range as a function of wind speed while the interferometer is locked. Again, the maximum wind speed of the three max. 1min trends for EX, EY and CS wind channels are used.
Thanks for looking into the effect of wind on lock loss and BNS range in the first two months of O3. I've used the BNS Range vs Wind Speed reported above, convolved with the yearly wind speed PDF (based on ~8 yr of data) to calculate the effect on yearly average BNS range and BNS detection rate (volume) in the attached notebook. I calculate ~1.2 MPc BNS range increase (or ~1.5 MPc for the windy season). This amounts to a ~3% increase (~4% in the windy season) in the detection rate (volume x time) for BNS.
The DetChar group has been asked to look into the effect of wind on BBH range and any glitches which might negatively effect searches such as Bursts.
A long time ago I came up with some blend filters to try to improve the 1-10 hz HAM RX/RY motion, to try to reduce SRCL. We ran the filter for about a week last October, but I've been having trouble trying to find good data to compare to the current configuration. Sam Cooper has a document comparing the differences and some of the motivation for finding a better filter at https://dcc.ligo.org/DocDB/0159/T1900107/001/main.pdf. I've switched HAMs 4&5 RX&RY blends to this filter to get some data overnight (or weekend). I made the transition while Corey was locking the IFO, and it's running fine for now. I've accepted the differences in SDF, so this shouldn't prevent us from going to OBSERVE.
The new blends are called "between_350" on the HAM ISI blend screens. If there is some issue where SRCL motion is bad, the old filters are called "notch_rolloff". If there are any questions or anyone needs help reverting over the weekend, give me a call.
Some initial data, looks like the blend filters do improve SRCL. Attached plot shows SRCL err from a lock after switching the blends (blue) and a segment from a NLN segment last night before the switch (red). Microseism hasn't changed much, but it is windier now than it was last night, but still the SRCL rms is about half of what it was. The magnitude units are wrong because I'm not sure of the dewhitening filter gain but the filter is shaped with 2 poles at 1 hz and 2 zeros at 100hz. Also the blue traces where taken around DC readout, while the red are from the IFO in NLN, so the blue is missing some high frequency cut-offs making the >10hz noise worse on the blue trace.
I'm attaching some plots comparing two NLN SRCL asds, first plot is the SRCL ERR, blue is with the new blends, red is the old blends. The new blends are about a factor of 2 better rms between 1 and 10 hz for the SRCL error. I would like to try adding some notches to the blend to see if we can do any better, but this seems like a solid improvement.
The second plot is SRCL suspoint, which uses projections of the ISI GS13s to each relevant suspensions top mass to estimate the SRCL cavity length. This channel combines the suspoint motions of the BSCs and both HAMs, so is not purely deteremined by the improved motion on HAMs 4&5. Again, red is from a lock with the old blends, blue is with the new blends. This also shows a good improvement over the 1-20hz band, with some worse performance below 1hz, but not too bad.
Attached is a comparison of the RX HAM4 blend changes at LHO during May vs the nominal L1 blend filter. The current filter for LHO is the purple one.
We see that LHO has now more isolation than LLO between 1.5-8Hz, but less roll-off above 8Hz. It would be interesting to understand how this difference in the filtering affects the suspoint motion at both sites, as a follow up from 46407.
