B. Gateley, T. Guidry The well pump was running both Wednesday (8/2) and Thursday (8/3) from approximately 8am to 12pm to replenish the fire water tank.
TITLE: 08/03 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Commissioning
SHIFT SUMMARY:
Arrived to H1 almost done with NOMINAL LOW NOISE and finishing out COMMISSIOING time by Louis.
Random lockloss 2hrs into the shift.
Relocked with no issues, exept the SUSETMx had calibration line related diffs (see earlier alogs).
LOG:
We've only had one short (2hr) lock stretch since it was loaded, but it looks like Gabriele's adjustment to the MICH FF filter (LHO:71934) improved the noise in the UIM Kappa (screenshot attached).
L. Dartez I reduced the amplitudes of the SUS-ETMX calibration lines between 15-17.6Hz. These line amplitudes hadn't yet been adjusted for the reduced noise level at low frequencies (LHO:71200). As such, they were significantly higher than necessary to maintain an uncertainty of 0.5%. N.B. The threshold itself is rather arbitrary. This is discussed further in LHO:69555. Moreover, there was evidence that the high SNR lines were causing excess noise (LHO:71149) and repeated on/off tests with and without the calibration lines active showed an improvement when the lines were off (LHO:71296). The new values, as compared to their old settings set in LHO:69555, are shown below. Channel Old Gain New Gain Reduction Factor ================= ========= =========== =========== H1:SUS-ETMX_L1_CAL_LINE_CLKGAIN 35 6.6 5.3 H1:SUS-ETMX_L2_CAL_LINE_CLKGAIN 50 9 5.55 H1:SUS-ETMX_L3_CAL_LINE_CLKGAIN 0.3 0.085 3.529 The third attachment shows a short trend of the uncertainty calculations after changing the line amplitudes. They are all hovering between 0.4% and 0.5%. It's worth noting that the IFO was thermalizing during this activity; we had just relocked when I got started. I didn't have time to visit reducing the amplitude of the PCALY line at 17.1Hz before the end of the allotted commissioning time.
While I was adjusting the line heights, Jeff let me know about the following tip:
Stepping the calibration height for one of the ETM{X,Y} SUS lines consists of changing all of the following channels such that they're all reflecting the same amplitude gain value.
Using the TST stage as a reference:
H1:SUS-ETMX_L3_CAL_LINE_CLKGAIN
H1:SUS-ETMX_L3_CAL_LINE_SINGAIN
H1:SUS-ETMX_L3_CAL_LINE_COSGAIN
H1:CAL-CS_TDEP_SUS_LINE3_COMPARISON_OSC_CLKGAIN
H1:CAL-CS_TDEP_SUS_LINE3_COMPARISON_OSC_SINGAIN
H1:CAL-CS_TDEP_SUS_LINE3_COMPARISON_OSC_COSGAIN
For example, here's the command that I used to change the TST line to 0.085:
val=0.085 && caput H1:SUS-ETMX_L3_CAL_LINE_CLKGAIN $val && caput H1:SUS-ETMX_L3_CAL_LINE_SINGAIN $val && caput H1:SUS-ETMX_L3_CAL_LINE_COSGAIN $val && caput H1:CAL-CS_TDEP_SUS_LINE3_COMPARISON_OSC_CLKGAIN $val && caput H1:CAL-CS_TDEP_SUS_LINE3_COMPARISON_OSC_SINGAIN $val && caput H1:CAL-CS_TDEP_SUS_LINE3_COMPARISON_OSC_COSGAIN $val
Although I saw Louis ACCEPT the SDF Diffs for SUSETMx, I think with the safe.snap stuff afterward, maybe it didn't take?
Anyway, We had SUSETMX diffs again. So I ACCEPTED the 9-diffs (attached). Hopefully good for next lock.
Something happened that reverted my changes for CAL-ETMX back to the old settings. Maybe something happened with SDF? Corey brought us out of OBSERVE for me to restore the new amplitude heights. Here's the command I used:caput H1:SUS-ETMX_L3_CAL_LINE_CLKGAIN 0.085 && caput H1:SUS-ETMX_L3_CAL_LINE_SINGAIN 0.085 && caput H1:SUS-ETMX_L3_CAL_LINE_COSGAIN 0.085 && caput H1:SUS-ETMX_L2_CAL_LINE_CLKGAIN 9 && caput H1:SUS-ETMX_L2_CAL_LINE_COSGAIN 9 && caput H1:SUS-ETMX_L2_CAL_LINE_SINGAIN 9 && caput H1:SUS-ETMX_L1_CAL_LINE_CLKGAIN 6.6 && caput H1:SUS-ETMX_L1_CAL_LINE_COSGAIN 6.6 && caput H1:SUS-ETMX_L1_CAL_LINE_SINGAIN 6.6output:Old : H1:SUS-ETMX_L3_CAL_LINE_CLKGAIN 0.3 New : H1:SUS-ETMX_L3_CAL_LINE_CLKGAIN 0.085 Old : H1:SUS-ETMX_L3_CAL_LINE_SINGAIN 0.3 New : H1:SUS-ETMX_L3_CAL_LINE_SINGAIN 0.085 Old : H1:SUS-ETMX_L3_CAL_LINE_COSGAIN 0.3 New : H1:SUS-ETMX_L3_CAL_LINE_COSGAIN 0.085 Old : H1:SUS-ETMX_L2_CAL_LINE_CLKGAIN 50 New : H1:SUS-ETMX_L2_CAL_LINE_CLKGAIN 9 Old : H1:SUS-ETMX_L2_CAL_LINE_COSGAIN 50 New : H1:SUS-ETMX_L2_CAL_LINE_COSGAIN 9 Old : H1:SUS-ETMX_L2_CAL_LINE_SINGAIN 50 New : H1:SUS-ETMX_L2_CAL_LINE_SINGAIN 9 Old : H1:SUS-ETMX_L1_CAL_LINE_CLKGAIN 35 New : H1:SUS-ETMX_L1_CAL_LINE_CLKGAIN 6.6 Old : H1:SUS-ETMX_L1_CAL_LINE_COSGAIN 35 New : H1:SUS-ETMX_L1_CAL_LINE_COSGAIN 6.6 Old : H1:SUS-ETMX_L1_CAL_LINE_SINGAIN 35 New : H1:SUS-ETMX_L1_CAL_LINE_SINGAIN 6.6
OK, for this current lock, we had SDF Diffs for SUSETMX (I'm assuming for the previous lock, we did not save the appropriate .snap file (SAFE.snap or OBSERVE.snap?)).
So for this current lock, the OBSERVE segment from 0222-0238 was with the WRONG (9) SUSETMX CAL_LINE channels & I ACCEPTED these diffs to get to OBSERVE.
Then Louis noticed my alog, and saw that we had the OLD values for these (9) channels! So we immediately ended that OBSERVE segment and worked on correcting SUSETMX. I'm hoping I am doing this right, but this is what we did for the SUSETMX SDF:
I think that covers everything. I was able to go back to OBSERVE. Fingers crossed that for the next lock, SUSETMX has all the CORRECT values for these pesky (9) CAL_LINE channels! Sorry for not being more diligent on ACCEPTING diffs! I should have caught the values---I'm glad Louis was online and noticed this from my earlier alog comment!
The reason why Louis' CAL-ETMX gain changes "didn't take" is because they are set by the ISC_LOCK guardian in several places during lock acquisition, including when we reach Nominal Low Noise. This was discovered somewhat recently by Jeff; see his alog 71710 for the full breakdown of the tug-of-war that guardian plays with these gains. Corey and Louis had correctly accepted the new gain values in both the SAFE and OBSERVE SDF tables, but this caused SDF diffs to appear once we reached NLN, preventing us from going into observation.
Since I don't have an answer for how these gains should be set during full lock acquisition, I've just updated the gain values to Louis' new ones for when we reach NOMINAL_LOW_NOISE, not in PREP_FOR_LOCKING or anywhere else. This is set in the gen_NOMINAL_LOW_NOISE() function within ISC_GEN_STATES.py
EDIT: ISC_LOCK has been loaded.
TITLE: 08/03 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Commissioning
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 8mph Gusts, 5mph 5min avg
Primary useism: 0.01 μm/s
Secondary useism: 0.04 μm/s
QUICK SUMMARY:
H1 just completed getting to NLN (after ADS converged)--post power glitch (thanks for quick recovery, RC!). During this time, Louis has been working on "shrinking cal lines". Once he's done, Sheila has a 20min ESD bias test. Then we can go for OBSERVING.
TITLE: 08/03 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Lock Acquisition
SHIFT SUMMARY: Some commissioning was done until we lost lock from a power glitch at 20:54. We relocked and have resumed commissioning as of 23:00UTC. Hnading off to COrey
Lock#1:
Lock#2:
LOG:
| Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
|---|---|---|---|---|---|---|
| 15:24 | FAC | Cindy | MidY | N | Tech clean | 16:58 |
| 15:25 | FAC | Karen | Vac prep, optics | N | Tech clean | 15:36 |
| 17:25 | FAC | Mitch | Ends | N | FAMIS task | 20:04 |
| 17:38 | FAC | Cindy | H2 | N | Tech clean | 17:51 |
| 20:04 | Jeff | Mech room | N | Quick check | 20:09 | |
| 20:18 | Sheila | CR | N | ESD bias test | 21:03 | |
| 21:03 | SEI | Jim | Ends | N | Reset VFD panels | 21:43 |
| 21:11 | EE | Fil | MidY | N | Drop off parts/tools | 21:40 |
| 21:43 | VAC | Gerardo | FCES | N | Checks, look for a VAC gatevalve | 22:26 |
I started to change the ESD bias and adjust the gain to compensate it. The goal of this is to see if there is any change to the DARM noise, bicoherence, or coherence with EX power monitors for the full bias, and we were also planning to revisit the excess noise seen at -125V ESD bias back in Feb.
We lost lock due to a site power glitch, so I'm just recording here the settings to be used to change the ESD bias and keep the DARM OLG and calibration the same. The second set of three references (positive biases) are settings found today, while the negative biases are settings found in Jan (66751), apparently the DARM loop has changed in the meantime.
To do the test we intended today (get 20 minutes of quiet time at full positive bias), we would do:
ezca['SUS-ETMX_L3_DRIVEALIGN_L2L_TRAMP'] = 60
ezca['SUS-ETMX_L3_LOCK_BIAS_TRAMP'] = 60
ezca['SUS-ETMX_L3_DRIVEALIGN_L2L_GAIN'] = 47.22
ezca['SUS-ETMX_L3_LOCK_BIAS_GAIN'] = 2.85
ezca['SUS-ETMX_L3_DRIVEALIGN_L2L_GAIN'] = 184.65
ezca['SUS-ETMX_L3_LOCK_BIAS_GAIN'] = 1
this will ramp the bias back to it's normal 125V value over 60seconds. Then we can clear the SDF for the TRAMP:
ezca['SUS-ETMX_L3_DRIVEALIGN_L2L_TRAMP'] = 10
then we should be able to go back to observe.
The lights flickered in the control room then we lost lock, and the ETMs HEPI and ISI watchdogs dripped
MSR UPS Reports:
Date: 08/03/2023
Time: 13:54:33
Warning - UPS: On battery power in response to an input power problem.
Date: 08/03/2023
Time: 13:54:36
Informational - UPS: No longer on battery power.
Mainsmon from h1pemcs at time of glitch:
0.125 Hz bandwidth: https://ldas-jobs.ligo-wa.caltech.edu/~gabriele.vajente/bruco_STRAIN_1375096943/
0.031 Hz bandwidth: https://ldas-jobs.ligo-wa.caltech.edu/~gabriele.vajente/bruco_STRAIN_1375096943_highres/
Some highlights:
In summary, the low frequency noise is improved and more stationary with the new CHARD_Y higher gain controller and reduced test mass damping gain in yaw. Also, the glitch rate is largely reduced.
A new controller was designed based on the previous measurement of the CHARD_Y plant. This controller is implemented in FM9 (as an additional module that can be switched on with the other modeles). This can be engaged with the nominal gain of 60, and then the gain can be increased up to a factor 3 to 180. After a few tests I found that a gain of 100 seems ok, higher gain increase the amplification of a 3.4 Hz narrow peak. The amplification is expected due to a broad gain peaking of the new controller. However, the peak itself is already there and not related to the CHARD_Y controller. If we manage to track down the origin of this peak and improve it, we could increase the CHARD_Y gain even more. As it is now, more gain doesn't improve much the CHARD_Y RMS because of this 3.4 Hz peak.
With this new configuration, the CHARD_Y RMS is significantly reduced around 1 Hz (no more gain peaking) and 2-3 Hz (more suppression).
With the new controller in place, I repeated the old test of reducing the test mass damping gain in yaw. I realized that the 1 Hz oscillation I saw in my previous attempt was due to the CHARD_Y gain peaking at 1 Hz getting worse. the new controller solved this problem. Indeed, I could reduce all test mass damping gains from -1.0 to -0.5 without problems. This improves a bit the CHARD_Y motion, and improves significantly the DHARD_Y motion. So it's a good change to have.
I also tried to reduce the PR3 M1 yaw damping gain, since CHARD_Y was coherent with PR3 damping, but I did not see much of an improvement.
Those changes were left overnight, and the noise was much better. On average, the DARM noise below a few tens of Hz is lower. The main difference is the disappearence of most of the non-stationay noise, as visible in a spectrogram. Comparing the omicron glitch plots from before and after the change, the number of glitches in the 20-50 Hz region is much lower.
I suggest we leave the new CHARD_Y controller with a gain of 100 as the default configuration. I also suggest that we run with reduced test mass yaw damping. I would leave the PR3 damping at -1. I would also keep the current value of -1.50 for ITMY Y2L gain.
Next things to check:
Right before the lock loss yesterday, I managed to snap a quick CHARD_Y plant measurement. It's not good enough to fit a new plant model, but it hints at small changes in the plant due to the reduced test mass damping.
It's worth doing again a careful measurement of both CHARD_Y and DHARD_Y with the reduced test mass dampng, and fine tune filters accordingly.
Here's a better plot showing how the coupling from CHARD_Y to DARM changes with the ITMY Y2L gain.
For now it doesn't look like CHARD_Y is limiting DARM, but we should investigate this behavior more if we need to further reduce the coupling to increse the DHARD and CHARD gains
I did a quick check into the Omicron glitches before and after these changes to see the difference. I grabbed Omicron triggers with SNR > 7.5 and frequency between 10-1024 Hz on August 2nd 0-14 UTC and August 3rd 0-14 UTC. The first plot attached shows a comparison of the glitch rate per hour. During the period on the 2nd, the glitch rate per hour was around 51, whereas on the 3rd after these changes it's at about 7 per hour. The second plot shows the glitch rate as a function of SNR. Below an SNR of 50, we see a large reduction in the glitch rate, suggesting a large reduction of this low frequency noise.
The new configuration has been implemented in the ISC_LOCK guardian, in the LOWNOISE_ASC state:
class LOWNOISE_ASC(GuardState):
[...]
def run(self):
[...]
elif self.counter == 2 and self.timer['LoopShapeRamp']:
ezca.get_LIGOFilter('ASC-CHARD_Y').ramp_gain(100, ramp_time=10, wait=False) # increased to 100 again on August 3, see below
ezca.switch('ASC-CHARD_Y', 'FM3', 'FM9', 'ON') # new chard Y high power controller with LP built in, should be engaged with low power controller FM2, more useism with FM3
# FM9 is the controller to improve low frequnecy noise (August 2, see alog 71927)
self.counter += 1
self.timer['LoopShapeRamp'] = 10
[...]
elif self.counter== 9 and self.timer['LoopShapeRamp']:
# reduce test masses M0 DAMP yaw gains
ezca['SUS-ITMX_M0_DAMP_Y_TRAMP'] = 10
ezca['SUS-ETMX_M0_DAMP_Y_TRAMP'] = 10
ezca['SUS-ITMY_M0_DAMP_Y_TRAMP'] = 10
ezca['SUS-ETMY_M0_DAMP_Y_TRAMP'] = 10
ezca['SUS-ITMX_M0_DAMP_Y_GAIN'] = -0.5
ezca['SUS-ETMX_M0_DAMP_Y_GAIN'] = -0.5
ezca['SUS-ITMY_M0_DAMP_Y_GAIN'] = -0.5
ezca['SUS-ETMY_M0_DAMP_Y_GAIN'] = -0.5
self.counter +=1
self.timer['LoopShapeRamp'] = 10
elif self.counter==10 and self.timer['LoopShapeRamp']:
return True
This gave the expected SDF diff in the PR3 damping gain. I have accepted it, as attached.
Kappa_UIM has seen a huge increase in noise starting ~8 days ago (screenshot attached). It's not clear to me what is causing this yet. This is also shown on the summary pages: https://ldas-jobs.ligo-wa.caltech.edu/~detchar/summary/day/20230728/cal/time_varying_factors/ The UIM Kappa line value started to show noisy activity at GPS 1373905226.2881477.
There's a feature just above the corresponding 15.6 Hz calibration line which appears at the identified time. Fig 1 shows this feature in high resolution. It peaks around 15.605 and is present consistently (in Fscan daily data) since the date of the noise change.
I also computed some shorter-duration spectra with gwpy to double check that its appearance corresponds to the gps time Louis posted. I needed about 500s fft length to resolve the relevant features, and I wanted a couple of averages so I ended up looking at 1000s time periods. Apologies for the messy overlay of figures with not-precisely-matching y-axes! I think the shape difference is clear regardless of the scale. Fig 2 shows some samples right before the change. The change occurs near the end of an observing segment but low-noise data remains available right after, so I looked at both the immediate time after the change (fig 3) and the next observing segment (fig 4). Indeed, it looks like a good match.
This is caused by a mistake in the MICHFF filter. It turns out that the filter I retuned on July 20 has a sharp feature at 15.6 Hz that I did not nottice before. This is injecting MICH noise in DARM at 15.6 Hz. This can be fixed by either tweaking the current filter, or retuning the MICHFF again (being more careful with narrow feartures in the fit!)
I modified the current filter to remove the 15.6 Hz feature and saved it. We should reload the MICHFF filters at the first opportunity
This filter was reloaded since we lost lock. We should see an improvement in our next lock.