Dave created a dummy target area where I stuck an autoBurt.req file which will start capturing all of the SUS alignment offsets on the auto-hourly basis.  This will aid in realignment after a Tuesday set of reboots.  (Recall, the reboots usually preceed a burt restore which often houses old alignment offset values.  - And the age of each restore file is different so the alignment settings are from a variety of alignments, not a matched set.  Therefore, we have been having to perform major realignments after Tues maintance, such as today.)

In the future - find and restore the latest locked good alignment from the hourly backups in:

/ligo/cds/lho/h1/burt/2015

under the appropriate date /h1ifoalignepics.snap

For ASC modelers, here are screenshots of our matrices in full lock.  

Nutsinee, Pat Meyers, Marissa

In the recent LHO locks, we've noticed a line in DARM that wanders around between ~410-420 Hz, consistently appearing in every lock from May 13th to the present. For example, here is a spectrogram of 2 hours from May 15.

There is a peak around that frequency in the Rayleigh plot (This plot shows the standard deviation of the PSD divided by the mean in each frequency bin. For a random Poisson-distributed signal, this ratio should be one.) For example, the Rayleigh plot from May 15 shows this peak, and going back through the summary pages, this non-stationarity in the spectrum appears to have been there since May 13.

Looking further back, we saw that there was a wandering line around the same frequency in the March 27th lock, but its frequency drifted much more slowly; see this 2 hour spectrogram.  

Pat has used STAMP-PEM to look for coherence with other channels but hasn't found any obvious culprit. We've also made spectrograms of many other channels, including MICH, PRCL, and SRCL, which don't show the same line.

Nutsinee also did some follow-up to see if this was the same as the "seismic bumbling line" from ETMX (see alog 17681), but found that it is quite different. For example, compare the DARM spectrogram with this 2 hour ETMX spectrogram from May 15 (at the same time as the spectrogram shown above for DARM). At this time, the seismic line wanders between 200 and 240 Hz, with no obvious correlations with the DARM wandering line.

J. Kissel, J. Betzwieser

Joe had uncovered a particularly nasty bug in the wiring of the CAL-CS front end model's reproduction of the digital actuation chain in the SUS (see LHO aLOG 18465). He's committed a bug fix, I've svn up'd and recompiled, reinstalled, restarted, and restored the front end code with the new fix.

Details:
--------
As you know, in the suspensions, one has the ability to choose between a distributed and offloaded hierarchy for the actuation. In the CAL-CS model, a stripped-down, longitudinal-only version of this hierarchy option has been installed such that we can exactly replicate what's been installed in the actual SUS hierarchy for the DARM / DELTA L calibration. As Joe says,
"At some point the tag coming from the LOCK_L3 output (shown in magenta) and the L3_ISCINF output (shown in orange) had been switched"
which causes indescribably confusing things to happen with the hierarchical control (at least not without a serious exercise in loop math). The two attachments show the before and after.

Yikes! Good catch Joe!

To run timing slaves in the EE shop, a fiber was pulled from port 8 on the MSR timing master to EE.

WP 5219

Pulled excess multimode fiber cable from computer user's room (CUR) to MSR to allow them to be connected to the patch panel in the MSR. The fibers had previously been pulled but did not reach as far as the patch panel, with the excess being coiled up in the CUR.

Connected DAQ test stand network switch to MSR using multimode fiber, then to CUR to extend the DTS network allowing a DTS workstation to be used in the CUR.  Also pulled one multimode fiber between the CUR and the timing master fanout rack to allow extending timing to the EE shop for timing component testing. Note that this fiber will not be connected to the master fanout until it is needed.

Wasn't able to test set point interlock functionality (maybe next maintenance day?) 

Several front ends had partially loaded filter modules. To prepare H1 for ER7 data taking, I ran a script which pressed all LOAD_NEW_COEFF buttons on every model. I'll periodically "load all filters" if we encounter any partially loaded files during ER7.

Daniel and I looked at three of the locklosses from Travis's shift last night, from 14:40, 14:02 and 11:33 UTC.  The earlier two both seem to be related to an alignment drift over 2-3 minutes before the lockloss, which shows up clearly in SR3 PIT.  (there is currently not feedback to SR3 PIT)  According to the witness sensors, this drift is only seen on M3.  No optics saturated until after the lockloss.  The DC4 centering loop, as well as both of the SRC alignment loops respond to the drift.  

Its unclear what causes the drift to accelerate in the minutes before the lockloss.  There is also a drfit of SR3 when we power up, as we noted yesterday, but this happens on a slower timescale than the dirfts that preceed a lockloss (3rd screenshot).  Also, there is a longer, slow drift that happens whenever we are locked.  

With Patrick and Cheryl I have engaged a DC coupled optical lever for SR3 PIT, we will see if this helps.  The last screen shot attached shows the MEDM screen used to turn this on or off.  

If the operators need to disable this (due to an earthquake, a trip, or if the optic becomes misalinged for any other reason) you can get to this screen from SR3, M2 OLDAMP.  

Turning off:  

turn off FM1 (labeled DC), then the input

Turning it back on:

Once the optic has settled and the beam is back on the oplev QPD, turn on the damping loop (with FM1 still off).  Average INMON (in a  command line tdsavg 10 H1:SUS-SR3_M2_OLDAMP_P_INMON), type -1 times the average into the offset, make sure the offset is engaged, and finally turn on FM1 to make the loop DC coupled.  

Since this is just a trial, Jeff is not including these changes in his current SDF cleanup campaign. 

Attached is a pre-ER7 list of narrow lines seen above 5 Hz in recent H1 DARM data, along with spectra containing labels for the lines.

The spectra used for line-hunting are from 18 hours of DC-readout conditions on May 17. Most of the lines were also seen
in the early-May mini-run data, but are more exposed in the more sensitive May 17 data (see figure 1)

Notable combs / lines:

 Exact integers: 16.0000 Hz, 64.0000 Hz (distinctly louder than other 16-Hz lines), 81.0000 Hz, 1150.0000 Hz, 1672.0000 Hz, 1704.0000 Hz, 1880.0000 Hz, 1896.0000 Hz
 Nearly exact-integer combs: 

 3.9994 Hz (offset by ~2 Hz from zero, first visible harmonic at 13.9977 Hz -- these seem to correlate with a near 4-Hz comb starting at 2 Hz in EX magnetometers (and to a lesser degree in EY magnetometers), where the contamination in DARM at 10 Hz and below is presumably too faint to be seen in the rapidly rising DARM noise. A strong comb is apparent in PEM FScans.
 99.9989 Hz (correlated with magnetometers in EX - see NoEMi and PEM FScans)

 Quad suspension violin modes with fundamentals near 500 Hz are not quite truly harmonic, but some of their upconversion is (see figure 2)
 Especially pervasive combs: 3.9994 Hz (34 harmonics), 16.0000 Hz (124 harmonics), 60.0 Hz (9 harmonics), 64.0000 Hz (31 harmonics), 36.9733 (54 harmonics), 59.9954 (6 harmonics), 99.9989 (13 harmonics)
 There are tentative identifications here of lines near 300 Hz and their harmonics as due to beam splitter violin modes, but their frequencies shifted by several mHz between the mini-run and the May 17 lock stretches, unlike the quad suspension violin modes which hardly moved at all (attributable to temperature?)
 Designated calibration line frequencies are shown on the spectra even when there is no apparent line (some not yet enabled?)

Notes:

 Because the binning in the spectra is 0.5 mHz (based on 30-minute FScan Hann-windowed SFTs), but the line widths in the plots are much larger, the spectra shown here look awful (but aren't). Dewhitening applies five poles at 1 Hz and five zeroes at 100 Hz.
 I didn't bother labeling the forest of bounce/roll-mode sidebands on the quad suspensions and their harmonics
 Although the 1150-Hz line seems well centered on the integer value, its energy is spread over several tenths of a Hz, unlike other integer-Hz lines
 Many line frequencies are give to four digits after the decimal point, but their statistical uncertainties are typically no better than a few tenths of a mHz, and based on changes between early and mid May, some lines have systematic drifts of O(mHz).
 With one exception, the quad violin mode fundamental frequencies were determined from the mini-run data, where they were more excited than on May 17. Those frequencies agree (independently) to within 2 mHz (in most cases, to better than 1 mHz) with the frequencies measured for individual test masses here. The one exception was that I had originally marked 504.1492 Hz as a mode in the mini-run, while the earlier study had found a mode instead at 501.254 Hz. Since the earlier measurements are guided by test-stand data, I am deferring to them and tagging 501.2544 Hz here, since I do see a line there in the mini-run data, albeit weaker than other modes.


Key to spectra labels:

b = Bounce modes
r = Roll modes
C = Calibration lines
L = Lock-in oscillator
M = Power Mains comb
N = Near to power mains comb
S = 64 Hz comb
s = 16 Hz comb
F = Near to 4 Hz (3.9994 Hz) comb
H = Near to 100 Hz (99.9989 Hz) comb
D =  59.3155 Hz comb
E = 36.9733 Hz comb
G = 75.23 Hz comb
Q = Quad violin modes
B = Beam splitter violin modes
x = Miscellaneous singlets

Figures:
1 - Superposition of 0-2000 Hz spectra for minirun (29.5 hours) and May 17 data (18 hours)
2 - Quad violin mode regions for first three harmonics (actual modes are non-harmonic, but some upconversion of fundamental is propagation harmonically)
3 - May 17 spectrum with labeled lines

Other attachments:
1 - Ascii list of narrow lines / combs found in 0-2000 Hz
2 - zipped tar file of 27 sub-band spectra (mostly 100-Hz wide)

J. Kissel, D. Barker,

Receiveing word from DetChar on yesterday's run meeting call that they're seeing major carry transition glitches in the IMC, Dave and I have restarted the h1iopsush34 front end model on the h1sush34 front-end, which runs a calibration routine on the 18-bit DACs in the corresponding I/O chassis. The reboot is now complete, and the IMC is up and running. We've confirmed that all DAC cards had a successful auto-calkbiration.


controls@h1sush34 ~ 0$ uptime
 11:52:19 up 13 days, 33 min,  0 users,  load average: 0.01, 0.02, 0.00
controls@h1sush34 ~ 0$ dmesg | grep AUTOCAL
# After the I/O chassis Power Supply, Timing Slave Firmware, and 18 bit DAC card upgrades had finished on May 21 2015:
[   49.512048] h1iopsush34: DAC AUTOCAL SUCCESS in 5341 milliseconds 
[   54.875289] h1iopsush34: DAC AUTOCAL SUCCESS in 5344 milliseconds 
[   60.668130] h1iopsush34: DAC AUTOCAL SUCCESS in 5345 milliseconds 
[   66.030689] h1iopsush34: DAC AUTOCAL SUCCESS in 5341 milliseconds 
[   71.823494] h1iopsush34: DAC AUTOCAL SUCCESS in 5344 milliseconds 
[   77.186841] h1iopsush34: DAC AUTOCAL SUCCESS in 5345 milliseconds 
# This restart:
[1121136.381304] h1iopsush34: DAC AUTOCAL SUCCESS in 5345 milliseconds 
[1121141.741653] h1iopsush34: DAC AUTOCAL SUCCESS in 5344 milliseconds 
[1121147.529535] h1iopsush34: DAC AUTOCAL SUCCESS in 5344 milliseconds 
[1121152.889081] h1iopsush34: DAC AUTOCAL SUCCESS in 5340 milliseconds 
[1121158.677788] h1iopsush34: DAC AUTOCAL SUCCESS in 5345 milliseconds 
[1121164.037291] h1iopsush34: DAC AUTOCAL SUCCESS in 5340 milliseconds


@ DetChar -- we are VERY interested to track how we these DAC calibrations behave over time, to find out how often we need to do these auto cal routines. Please make sure to look for glitches *every day* during ER7, and report back to us
- Did *this* autocal fix the problem you've seen that made you request it?
- If you still see the problem, did the autocal at least *reduce* the glitches?
- How quickly, if at all, do glitches come back? (a plot of glitch amplitude vs time over ER7)
- You'd mentioned you can't see it in DARM -- confirm that this is true for the entire run
- Because we've have so many of these DAC cards fail during the May 21 upgrade, we were forced to *not* upgrade the cards in the h1sush56 I/O chassis. This means that H1SUSSRM, H1SUSSR3 and H1SUSOMC *have not* had their DAC cards upgraded. Can you tell a difference? 
- I would expect H1SUSSRM and H1SUSSR3 to have great influence on DARM, given the know SRCL to DARM coupling. Is there evidence of this?
- We send control ASC control to both SR2 and SRM, and LSC control to SRM. SR2 has new 18 bit DACs and SRM does not. If you can see glitching in SRCL -- can you tell if it's more SRM / SR3 than SR2? 

Thanks ahead of time!

Greg, Dan, Jim, Dave:

Dan has the new LDAS SFP to install on the single mode link between MSR and LSB which was reporting errors. For now he has inserted the new SFPs in the QLogic switches but for now these ports are still disabled. We will enable them and remove the link between the MSR switches at the end of ER7.

To clear the accumulated processor load of 40 on h1guardian0, I rebooted this machine at 11:41PDT. All guardian nodes came back online with no problems. The load average is now back at its starting value of 7.

Jeff, Dave

We restarted the h1iopsush34 model, which performed a calibration on all six 18bit DAC cards (including MC2's M3 DAC which had reported zero crossing glitching Link). All autocals completed successfully. 

Andrew, could you verify if the recalibration has made any improvement in MC2 M3? The calibration could deteriorate over time.

[Ryan F, Duncan M, etc.]

I have committed a new version of the CAL_INJ_MASTER common library model used to control hardware injections in the front-end system. This change reimplements the logic for the ODC state vector, making the state reporting more robust, and implementing logging for the Stochastic injections. The change has no impact outside of the /INJ/ODC block excepting input connections to that block.

If this model could be pulled onto the production system at LHO, and the h1calcs front-end rebuilt and restarted, that would be spiffing. At LLO this change did not require a restart of daqd, which is nice.

Once that change is made, I can remotely modify the related MEDM screens and EPICS variables to ensure that hwinj reporting is configured correctly before data-taking restarts this afternoon.

J. Kissel, E. Hall

See Evan's entry here: LHO aLOG 18770.

We still need to double check it, and I'm sure we've made a mistake or two, but we think we've installed as much as we can based on the results of the DARM Open Loop Gain transfer functions we have compared against a model (see LHO aLOG 18769) and of the actuation coefficient measurements (see LHO aLOG 18767)

For lack of better quantitative understanding of the DARM OLGTFs we have, we should still consider this calibration at an accuracy of 50% and 20 [deg]. (At least it's better than the factor of two promised :-/ ).

Note that we have NOT yet updated the inverse actuation function in the hardware injection path. Sorry -- but that'll have to wait until the morning.

We've still got plenty more to do and understand, but thanks to all who have helped over the past 1.5 weeks. You help has been invaluable, and much appreciated!!

P.S. IF NEED BE -- one can revert to the old CAL-CS calibration by switching back to ETMX, then reverting to the former sensing function via the filter archive.