The X1 DTS has been restarted with the exception of the frame writer and NDS. The LDAS gateway computer seems reluctant to boot, and is required for the frame writer and NDS.
no restarts reported for both days.
There are small glitches very close to each second boundary in the ETMY drive signal and the ETMY SUS ad SEI rack magnetometers. In order to investigate the half-Hz combs in DARM (see alog 20790), I took an hour of data and folded it with a four-second period. If there is a repeated glitch at any multiple of this period, it should become far more visible. The result is that in several channels, there are glitches very near the boundary of each GPS second. The peak time of these glitches seems to be about 10 milliseconds after the start of the second. The glitch does not repeat identically every second. There is one shape in the first second, then one with an opposite polarity in the next second. The first two attached plots are for the SUS and SEI racks, which are shown with a 40-Hz zero-phase lowpass. The SUS has a narrow spike, and the negative spike is larger than the positive one. The SEI signal is more complicated. The ETMY L3 MASTER signal, which is the DARM output to the ESD, is shown with a 10 to 50 Hz bandpass. These glitches are more like sine-Gaussians, but the even and odd seconds still seem to have opposite polarities. There are more channels with similar glitches. We can make a more thorough investigation, and use more data and more times, to try to track down the origin of these glitches. Hopefully these glitches are responsible for the 0.5 Hz combs such that removing them will improve those.
J. Kissel
Tagging CDS in this entry.
I'd recently taken a look at the requested output of the ETMY L1 stage,
${IFO}:SUS-ETMY_L1_MASTER_OUT_*_DQ
and was interested to find ~1 [Hz] combs in the requested output. Though this isn't the 0.5 [Hz] combs that Andy mentions above, I think it's an excellent place to take the investigation further in a more focused manner -- with the point being that even the SUS's *requested* signal has a comb.
Attached is a 100-sec FFT ASD, of a typical, 1000 [sec] stretch of observation-ready data during the run (2016-01-04 04:00 UTC). Here, to give a feel for the physical amplitude of these signals: at 30 [Hz] the noise amplitude of one of these comb peaks is roughly 1e-3 [ct/rtHz] of requested DAC output, which corresponds to
1e-3 [ct/rtHz] * (20.0 [V] /2**18 [ct]) = 7.6e-8 [V/rtHz] @ 30 Hz ( * sqrt(2 * 0.01) = 1.1e-08 [V_pk])
Potentially verifiable / refutable Crack-pot Theories / Wild guesses:
- Perhaps there is some of this glitching in the inter-process communication (IPC) on the reflected memory (RFM) data transfer from the corner to the end station, that's only exposed for requested drives that have such a huge dynamic range? For whatever channels in the signal chain that are stored, can you reproduce the same combs by filtering those channels offline?
- Recall that the power supplies for the Hartmann Wavefront Sensor (HWS) were replaced some time ago, see Integration Issue 1062. Has anyone made a before-and-after comparison on this searched other sources for such combs in auxiliary channels? Perhaps forming a BruCo-like search where this UIM / L1 stage control signal is the response instead of DARM?
- Keith has already done a long-term study of the analog-to-digital converters (ADCs), looking for combs: see, eg. pg 44-49 (yeah!) of G1300997. He found no-such combs. Perhaps we should do a similar study on the digital-to-analog convert (DAC) side of things? I could also imagine a similar set up for a set of RFM channels that make the 4km journey along the arms.
In response to Jeff: The SUS-ETMY MASTER signal is just a filtered version of DARM, so if DARM has the comb so do the drive signals. I don't think that tells us where in the loop they originate. But you're right, this could be a digital problem or an electronics one involving something synched to GPS. Keith T. and Annamaria both suggested that the power supply of the timing fanout might be involved. That of course can be perfectly synched to the GPS second. Annamaria showed me an ADC in the L1 corner station being used as a temporary monitor of one of these power supplies. That signal (attached) jumps downward one second and upward the next, matching what we see in the magnetometers and DARM. Could we check if there's such an effect at the H1 Y-end?
Note that the small glitches in Andy's post are exactly synchronized to GPS; this makes coupling to many power supply glitches (HWS, or trickle chargers for magnetometers, etc.) an unlikely source.
An electromagnetic shaker on the blue cross beams of the BS chamber produced greater upconversion than was produced by shaking of any other chamber or the PSL table (ITMY wasn’t shaken) (https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=24194). I unsuccessfully tried to reproduce this upconversion using stage 2 ISI injections in X, Y and Z (I did not try RX, RY or RZ). Next I investigated the upconversion by injecting single lines with the EM shaker on the BS cross beams in order to find the most sensitive frequency band. Figure 1 shows the line injections that produced the most upconversion; they are located between 40 and 65 Hz. Lines injected with amplitudes between 1 and 2 orders of magnitude greater than normal produced broad features around the injection frequency and at harmonics, with amplitudes that were several times background in DARM. The next step will be to inject uniform bands in order to estimate the contribution of the 40-65 Hz vibration background to the DARM noise in the 80-200 Hz band.
I spent some time looking at how we can reduce the DARM residual.
In the attached plot (template lives in evan.hall/Public/Templates/LSC/DarmResReduce_2015-01-23.xml), yellow shows the nominal DARM residual and estimated freerunning displacement.
Blue shows the residual after the addition of a 3 Hz resonant gain. I put this in LSC-DARM FM10, so the freerunning estimate should still be correct. [However, since the DARM filter bank is full, I had to overwrite some other filter. After this test was over, I reverted my changes and placed this RG filter in the LSC-OMC_DC filter bank for safekeeping.]
Red shows the residual after the addition of the 3 Hz resonant gain, as well as a microseism boost, which I put in LSC-OMC_DC. This affects the freerunning estimate, but the amplitude and phase changes at and above 10 Hz are minimal (less than 2 ° of phase change at 10 Hz).
Here is a plot similar to the one in 25053.
It shows the DCPD spectra from last saturday when I had a 6 Hz line injected, with the predicted upconversion from the DC readout quadratic term explaining the line at 12 Hz as well as the noise in the DCPDs at around 10 Hz. The red trace is the spectrum evan calls nominal above, where there were no changes to the DARM loop but the feed forward was retuned, as you can see the predicted upconversion is reduced by nearly 2 orders of magnitude at 10 Hz, and the upconversion around the calibration lines is reduced.
The black trace is once Evan had added both a boost and a resonant gain to the DARM loop, so we expect the DCPD specrtum to be reduced at low frqeuency simply because of the change in the DARM loop shape. You can see that there is also a reduction in the predicted upconversion around the calibration lines as well as at 10 Hz.
Incidentally I was looking at the residual spectra of LLO and LHO (from Jan 1 2016 00:00:00 UTC) for some other reason. I post the plot for the record.
I ran BruCo on one hour of data from last night. The report is available here:
https://ldas-jobs.ligo.caltech.edu/~gabriele.vajente/bruco_1137640217/
At low frequency, SRCL and DHARD_P are the largest contributions. SRCL is still a factor of few below the measured noise, while DHARD is very close around 14 Hz.
There is coherence with magnetometers (still far from the measured noise)
No significant broadband coherence in the mistery noise region.
A 7.1 earthquake in Alaska at 10:30 UTC tripped many watchdogs. I reset all and went back to nominal working conditions for all seismic isolation at about 17:30 UTC.
Aborting a DTT measurement caused a lockloss - no good, although not a new phenomenon.
Since there are so many SDF diffs, I'm wouldn't put the IFO in Observe anyway, so I'm going to just leave it Down for the night.
Reported as FRS Ticket 4274, so we begin to quantify how often and how painfully this remaining DTT abort problem affects us.
Evan, Gabriele
There are too many SDF to accept, so we're just leaving the IFO undisturbed in low noise, starting 3:10 UTC
I just got here to do some ASC stuff, so the IFO will no longer be undisturbed, as of 4:10 UTC.
We noticed a small bump in DARM at about 360 Hz, coherent with PRCL. We added a 270-430 Hz elliptic band-stop. DARM improved and the coherence is almost gone. We are losing 9 degrees of phase at 50 Hz (PRCL UGF).
The filter is implemented in the SUS filter bank.
The bandstop filter is now engaged by the guardian in the NOISE_TUNING step.
1220 - 1235 hrs. local -> To and from Y-mid Next manual overfill expected Monday, Jan 25th before 4:00 pm
First a bit of history: h1boot-backup is an identical machine to h1boot and can be used to replace h1boot if it were to fail. The /opt filesystem on h1boot used to be rsynced every hour to keep h1boot-backup current. Over the course of summer 2015 the rsync took longer, and became linked to the EPICS freeze ups of the front end computers. We suspect the main reason is that the NDS jobs directory became filled with many small files. Over Christmas I was doing select rsyncs manually, skipping these huge directories of NDS logs.
Over the last week I have reduced the number of files in these directories to a manageable number. This took some time because when a directory has 6.5 million files in it standard commands like find or ls wont work. I used customized C code to scan the files.
We now have NDS logs which look back over the past 7 days. The full rsync now only takes 15 minutes (it was taking either 10 hours or would not complete).
Because there is a non zero chance these fast rsyncs could freeze EPICS, I have scheduled the backups 3 times daily at 8am, noon and 6pm. If anyone sees epics freeze up at these times this would be the reason.
no restarts reported for both days.
Apologies for no transition entry. I was tasked with locking as soon as I walked in.
TITLE: Jan 22 EVE Shift 00:00-07:11UTC (16:00-23:11:00 PDT), all times posted in UTC
STATE Of H1: Down
SUPPORT: Sheila,Evan, Gabriele
INCOMING OPERATOR: N/A
ACTIVITY LOG:
00:28 HFD on site as a matter of protocol for a faulty fire alarm panel
00:30 another fire unit on site
00:42 reloaded ISC_LOCK Guardian
00:45 HFD Off site. Three firetrucks drove past corner station during lock
00:46 Dave B called and informed me of an AC failure. He’s contacting the proper folks.
01:13 Dick out of optics lab
01:32 Bubba back on site to handle AC issue in H2 building
01:37 BS ISI St2 watchdog trip
01:38 cleared HAM6 ACT watchdog accumulation counter
02:18 GRB alert
02:51 Bubba has reported there’s nothing he can do about the AC in the H2 building tonight so he’s leaving it off. He also went into the CER to check on the AC in there. He didn’t think anyone had been in there to check since the power outage. It seemed ok.
04:xx Relocked at NLN. Gabriele has turned me loose into the world!
We have a cooling problem with the H2 DTS building. Bubba came back in this evening to take a look at this, in the mean time we have powered down all DTS computers and DC power supplies for the weekend.
After an extensive search for the meaning of the error code displayed on both thermostats indicating a low pressure fault, which in and of itself is odd for both systems to have the exact same fault, I believe I have narrowed the problem down to faulty thermistors. This may have been caused by the recent power outage or may be simply coincidence. Parts will need to be ordered next week. I will follow up with a FRS.
I retuned the MICH feed-forward path. I injected noise in SRCL and measured the transfer function to DARM: TF_SRCL_DARM. Then I injected noise at the feed-forward input, with the FF shaping filters off, but with SBVio1 and SBVio2 on. I measured the transfer function from SRCLFF_IN2 to DARM: TF_SRCLFF_DARM. The optimal feed-forward filter should then be -TF_SRCL_DARM/TF_SRCL_FF. The first plot shows the measurement and the fit. Again, I decided not to fit the sharp features due to the bounce and roll filters.
I implemented the new filter in the bank 'newFF' and engaged it. The performance improved, as shown by the reduced coherence. The peak at 3 Hz also reduced a bit. See the second plot: green no feed-forward; blue old feed-forward, red new filter. The last plot compares the old (blue) and new (red) filters. With the new filter, I also switched on an AC coupling (double zero, double pole at 0.1 Hz).
Guardian code modified to load the new MICH and SRCL feedforward filters.
