WP 6432
Carlos, Jim
Investigation of failed power supply in network switch at EY. Checked power cords for the network switches at EX, EY, found that the AC main supply at EY had been unplugged from the receptacle in the rack. The cord was still attached to the switch, it was just dangling loose in the rack. We are investigating the system logs to try to get a better idea of when this happened.
Starting CP3 fill. LLCV enabled. LLCV set to manual control. LLCV set to 50% open. Fill completed in 16 seconds. LLCV set back to 14.0% open. Starting CP4 fill. TC A error. LLCV enabled. LLCV set to manual control. LLCV set to 70% open. Fill completed in 32 seconds. LLCV set back to 35.0% open.
Lowered CP4 LLCV from 35% to 34% open because TCs are reading around -30degC.
CP3 filled fast but TCs are reading temps as expected.
As noted here, the SR3 oplev began glitching at around the same time H1 experienced a large range drop. While the glitching is unlikely to be the cause of the range drop (no oplev damping on SR3), and Keita's request I have attempted to address the glitching. As soon as H1 lost lock, I increased the laser power; the voltage that monitors the laser diode current increased from 0.934 mV to 0.940 mV. This is the max for this laser, so if this does not alleviate the glitching I will try lowering the laser power. If this does not work then a laser swap will be necessary. I will leave WP 6433 open until it is known whether or not the laser power increase has alleviated the laser glitching.
After a couple 3.5 hours it seems as though the power adjustment cleared the glitching. Here is a link the SR3 summary pages; I've also attached the 2 relevant graphs to this alog. The first attachment is a time series of the oplev SUM, where the glitching is clearly visible, as is my adjustment and the following cessation of the glitching. The second attachment is a spectrogram of the SUM signal, normalized relative to the median. Here it the start and end of the glitching can be clearly seen. I will continue to monitor this over the weekend, but initially it looks like the laser power adjustment cleared the glitching for now.
J. Kissel, S. Dwyer, E. Goetz, K. Venkateswara We're lamenting last night's severe decay in range, and the gross long term trend toward zero since the restart after the break. Our most said theory (which doesn't make it the best or the right theory) is temperature adversely affecting the global alignment system's operating point, which has steered the IFO into a place with much worse scattering coupling. As such the anthropogenic noise (3-10, 10-30 Hz BLRMS of ground motion) is now adversely affecting the IFO much more than in the past. We'd like DetChar's help to confirm: - We've lost our fellow whom has run BruCo for us. Does detchar in general know how to run BruCo, or does Gabriele need to teach new people again? We suggest that no coherence with any particular channel would confirm the non-linear process of scattering. - We see summary page spectrograms showing excess noise, but they're too long of a time scale to tell if the features are scattering arches. Note that Bubba had started plowing at the X-End this morning, so data from ~17:00 UTC will show elevated 3-10 and 10-30 Hz BLRMS seismic noise there. So we understand that ground motion is much higher then, and because of the greater sensitivity to it (again, claiming the alignment operating point increases scattered light coupling theory) -- the problem is overnight when there appears to be no elevation in seismic noise, but the range is still terrible. Any other ideas are welcome. -------- Also because Bubba was going down to the X-end, we turned off sensor correction at 16:52 UTC (LHO aLOG 33221). This looks to coincide with the 1080 Hz glitching getting worse on the summary pages. This is consistent with the OMC length noise increase investigations we've done over the past few days (LHO aLOG 33104 and LHO aLOG 33037).
Andy ran Bruco on this already: https://ldas-jobs.ligo.caltech.edu/~lundgren/aDQ/bruco/H1_1168346418_OMC_DCPD/
Josh, TJ, Jess, Alex
This looks like PI ringing up. In the spectrogram posted above there is a line at 4734.75Hz that grows around the time of the noise. Here are some plots:
Fig 1: the line at 4734.75
Fig2: a BLRMS of this line vs time it grows alot.
Fig3: 8.8 hours of the line growing from onset of noise to really ratty time.
We will work harder to tie this to the noise. We've started looking back at earlier pages and seeing this line rising around times that noise rises but have work to do to tie them together.
There are three things that we can try to improve or investigate the drop in range as Patrick relocks.
Josh Smith, Jess, TJ and I have investigated a bit and found that there is a line around 4700 Hz (Fig. 1) that rings up around the same time as a scratchy forest of lines in mid-range frequencies, right in the bucket. At Josh's suggestion I took a comparative look at the beginnings and ends of three lock stretches on Jan 6 where this pattern rings up (the circled regions in Fig. 1). The last three figures show 10-second averaged spectra of 100 seconds of data gathered at the start and end of these lock stretches (GPS times given in the plots). "Good" periods, where there is no 4.7 kHz line, are shown in blue; "bad" periods where the line shows up are plotted in red. You can clearly see the ASD of H1:GDS-CALIB_STRAIN is systematically worse between 70-500 Hz when the high-frequency line is rung up, compared to when it wasn't. (We also note this behavior on Friday, 13 Jan, and on a couple of other days back in december, all coincident with a wandering downward trend in BNS range.)
Right now, Bubba is at end X plowing snow, and the equipment he is using is enough to have a significant impact on the seismic platform, so we've had to turn a few things off. The BRS-X is pretty unhappy right now, as is the STS, so Patrick took the ETMX_ST1_CONF node to BLEND_Quite_250_None, which we can use because of the low microseism. This doesn't turn off all of the feed forward from the ground STS to the chamber though, so I had to ramp the HEPI Z sensor correction match gain to 0 by hand. None of this is defined in any state in SEI_CONF (it could be, but a few people are already unhappy with my profusion of states there...). Re-requesting WINDY on SEI_CONF should revert all of this to the nominal state. The HEPI gain is monitored (it is controlled in some states by SEI_CONF, so maybe it shouldn't be?), so that took us out of observe, but LLO had just gone down.
A minute or two after Patrick took us back to observe, Krishna touched a filter on the ETMX BRS which dropped us out of observe again. We aren't using that sensor right now so it wasn't a "real" change to the configuration, and he reverted the filter back pretty quickly.
Tagging DetChar.
TITLE: 01/13 Day Shift: 16:00-00:00 UTC (08:00-16:00 PST), all times posted in UTC STATE of H1: Observing at 51.3756Mpc OUTGOING OPERATOR: Ed CURRENT ENVIRONMENT: Wind: 1mph Gusts, 0mph 5min avg Primary useism: 0.02 μm/s Secondary useism: 0.16 μm/s QUICK SUMMARY: In standdown for GRB until 16:30 UTC. Range is down to around 50 MPc. Dust alarms in the PSL enclosure.
08:23 Operator checklist completed. only notable item is PSL dust alarms for monitor 102/.3u
09:43 the card reader alarm system is showing a standing 'door ajar' issue wit the LDAS door.
10:39 EY dust monitor 1 .3u alarm
Bubba is out plowing wit the frontloader prior to the GRB alert. His activity clearly shos in the 3-10Hz anthropogenic BLRMS.
15:29:32UTC
nsection Guardian is, and has been,set to INJECT_KILL.
H1 is in Observe Intent
1 hour stand down time is being initiated.
The noise reported earlier is still present and the H1 range remains in the same diminished state.
The SR3 oplev has a big increase in noise in sum, pitch, and yaw starting at 12:30 UTC, the same time the range decreased. There's no excess of noise in SRCL (or any other aux degrees of freedom), so it's not clear how it is affecting the range, but it seems like a clue. Or maybe the oplev noise is caused by whatever caused the range loss. We'll keep looking into it. Here's the SR3 oplev summary page.
The SR3 oplev is unlikely to be a cause as we do not use oplev damping for this optic, but the glitching still needs to be addressed. I have a WP filed to address this as soon as H1 loses lock.
Temps were falling in the LVEA. I brought on 1 stage of heat at Zone 3A. I will monitor this change.
I'm making an attempt to rectify my deteriorating range. Attached is the latest, passive, DTT coherence measurement. I'm grasping at straws but it's all I've got at the present.
This morning we sat in nominal low noise without going to observing from 19:21 to 19:51 UTC (Jan 9th) in a configuration that should be much better for the 1084Hz glitches. (WP6420)
On Friday we noticed that the 1084Hz feature is due to OMC length fluctuations, and that the glitch problem started on Oct 11th when the dither line amplitude was decreased (alog 30380 ). This morning I noticed that the digital gain change described in alog 30380 that was intended to compensate for the reduced dither amplitude didn't make it into any guardian, so that we have had a UGF that was a factor of 8 lower than what I used when projecting OMC length noise to DARM: 30510 The first attachment shows open loop gain measurements from the 3 configurations: before oct 11th (high dither amplitude), after october 11th (lower dither amplitude, uncompensated) and the test configuration (lower dither amplitude, compensated).
We ran with the servo gain set to 24 (to give us the nominal 6Hz ugf) and the lowered dither line amplitude from 19:21 UTC to 19:51 UTC Jan 9th. You can see the spectrum durring this stretch in the second attached screenshot, in the test configuration the peak around 1083Hz is gone, with just the pcal line visible, and the OMC length dither at 4100Hz is reduced by more than an order of magnitude. You can also compare the glitches from this lock stretch with one from yesterday to see that the glitches at 1084 Hz seem to be gone. This is probably the configuration we would like to run with for now, but we may try one more test with increased dither line amplitude.
Other notes because we don't have an operator today due to weather:
This morning all 4 test mass ISIs were tripped probably from the Earthquake last night that brought the EQ BLRMS to 10 um/second around midnight UTC. ITMY tripped again while it was re-isolating, no problem on the second try.
Richard topped added 400mL to the TCSY chiller around 10:15 or 10:30 local time, since we were getting low flow alarms. The flow alarms came back a few minutes before 11am local time.
I went through inital alingment witout problems and got to DC_readout transition. Then I measured the UGF of the OMC length loop in preparation for increasing the dither line height From that measurement and trends it became clear that when the OMC dither amplitude was reduced, the compensation of the OMC digital gain described in didn't make it into the guardian. This means we have been operating with a UGF in the OMC length loop that was a factor of 8 too low since mid october.
We arrived in low noise at 19:21 UTC with the OMC ugf increased to 6Hz. After about a half hour PI modes 27 and 28 rang up, and I wasn't fast enough to get them under control so we lost lock.
Here's a graphical version of what Sheila wrote, showing the time on Oct 11 when the 1083 Hz glitches started. The dither amplitude was reduced at 3:20 UTC, but the servo gain was increased to compensate. There are no 1083 Hz glitches at this time. Severe RF45 noise starts an hour later and lasts until the end of the lock. The 1083 Hz glitches are evident from the beginning of the next lock, and persist in every lock until the recent fix. The dither amplitude stayed low in the second lock, but the servo gain was reset back to its low value. Apparently, both need to be low to produce the glitches.
Keita tells me that people are concerned about making this change because of the increased noise below 25 Hz in the screenshot attached to the original post. We did not run the A2L decoupling durring this lock strech, and it was not well tuned. The shape of the HARD loop cut off at 25Hz is visible in the spectrum, which is one way of identifying bad ASC noise. The high coherence between CHARD P and DARM at this time is another way of seeing that this is angular noise (attachment).
So I think that this is unrelated to the OMC gain change and not really a problem.
1080Hz removal OMC gain/line conditions, does it make more low frequency noise?
Josh, Andy, TJ, Beverly
Conclusion: For two on/off times each for the two OMC gain tests (total 8 times) it looks like the high gain / low line configuration that takes away 1080 Hz (and also takes away some bumps around 6280Hz) coincides with a bit more noise below 25Hz.
Request: We hope this connection with noise below 25Hz is chance (It might have just been drift and we chose times unluckily) and we would like debunk/confirm it. We could do that with a couple cycles of on/off, (e.g. 5 minutes each, with the current configuration vs high gain / low dither configuration).
See the attached PDF. The pages are:
Also: There is no coherence above 10Hz between STRAIN and OMC LSC SERVO/I for any of these test times. So coupling must be non-linear.
Also: When the 1080Hz bumps disappear we also see a bump around 6280Hz disappear (attached image, sorry no x-axis label but its 6280Hz)
Our post crossed with Sheila's. If possible, we'd still like to see a quick on/off test with the A2L tuned. Could we have five minutes with the gain high and then ramp it down? Maybe with one repeat. Since this is a non-linear effect, we'd like to make sure there's no funny coupling with the CHARD noise. We're not too worried by excess noise below 25 Hz now, but it might be important when we're able to push lower.
While LLO was down I attempted to do a test by increasing the OMC length gain while in lock, which unlocked the IFO. So on/off tests aren't possible. Edit: I broke the lock by changing the gain after the integrator (which had been OK when not on DC readout), we can change the gain upstream instead without unlocking.
For now I put the new gain into the guardian so the next lock will be with the increased gain, and hopefully see that the low frequency noise is fine.
Now we have relocked, Patrick ran a2l, and Jeff, Evan, Krishna and I did an on off test by ramping H1:OMC-LSC_I_GAIN:
The attached screen shot using the same color scheme as in the presentation above shows that there is not a difference at low frequency between high gain and low gain.
We are back in observing in the low gain configuration, but the gain is set in an unusual way (and accepted in SDF so that we can go to obsevering). Keita would like us to hear confirmation from detchar before making this permanent.
Thank you Sheila, this looks really good. No change at low frequency. 1080Hz gone. The 6280Hz just varies on its own timescale. From our end we're happy with the configuration change since it only does good. Sorry for the red herring about low frequencies.