There was a 6.6 magnitude earthquake in the Pacific Ocean 500 miles from us. For reference, the Montana earthquake was 5.8 and 340 miles from us. Seismic motion is off the charts here in the control room. Everything tripped straight away.
After the last lockloss, ETMY Mode 5 was rung up bad enough to prevent locking on the OMC. I turned on ENGAGE_DAMPING_ALL in the violin mode guardian while sitting on RF DARM. This helped with many of the modes, but ETMY Mode 5 was the most rung up, and the guardian did not turn on this damping (gain was at 0). I turned it on and it worked very quickly with a gain setting of 2 (gain of 20 was too much).
I lost lock at Oct 22 2018 01:51:16 UTC today, and afterwards the guardian asked that I "Check that fast shutter triggered on last lockloss". I'm pretty sure it did. (see plot 2). The fast shutter trigger goes off before the arm power decreases, according to the slow channels, and the OMC-DCPD SUM fast channel never exceeds 30 mA. I took a quick OMC scan to make sure, and things seem okay.Investigation
Prior to the lockloss, the FAST_SHUTTER guardian was in READY (state 21), but the LOCKLOSS_SHUTTER_CHECK guardian was in SHUTTER_FAIL (state 30). Because LOCKLOSS_SHUTTER_CHECK was not in state LOW_ARM_POWER (aka state 1), ISC_LOCK told me to manually check the fast shutter. LOCKLOSS_SHUTTER_CHECK entered SHUTTER_FAIL at GPS 1224202947, while ISC_LOCK was in ADJUST_POWER (state 506). I had entered the ADJUST_POWER state to lower the power to 5 W because the violin modes had rung up too much, and INCREASE_POWER had caused OMCDCPD saturations at around 10 W. I then moved on to NOMINAL_LOW_NOISE after the violins had calmed down, not noticing any notifications about the SHUTTER_FAIL state being entered. In the CHECK_SHUTTER state, LOCKLOSS_SHUTTER_CHECK looks at the HAM6 GS13 channel and if the value is too low, returns SHUTTER_FAIL. Here is an excerpt from the LOCKLOSS_SHUTTER_CHECK guardian log when it entered SHUTTER_FAIL:2018-10-22_00:22:04.082186Z LOCKLOSS_SHUTTER_CHECK [CHECK_SHUTTER.run] USERMSG 0: run shutter check, then manually take to low power 2018-10-22_00:22:09.044349Z LOCKLOSS_SHUTTER_CHECK [CHECK_SHUTTER.run] timer['datawait'] done 2018-10-22_00:22:09.120162Z fetching 1224202932... ... 2018-10-22_00:22:09.122571Z fetching 1224202943... 2018-10-22_00:22:09.237937Z LOCKLOSS_SHUTTER_CHECK [CHECK_SHUTTER.run] No kick, but should have. Peak GS13 signal = 46.335 2018-10-22_00:22:09.302082Z LOCKLOSS_SHUTTER_CHECK JUMP target: SHUTTER_FAILThis whole system check may have been messed up by the violin mode ringup and the increase and decrease in power. EDIT: We reached NOMINAL LOW NOISE again, all seems well.
I'll be back and forth between the VPW and my office. I expect to be here for an hour or so and will make a comment to this log entry when I leave.
Bake of VOPO assembly (Lisa B.) in VBOC is OK.
HOT bake of VBOD (empty, background clean up) not so much.... I found that the RGA was cold due to a blown fuse in the output circuit of the Variac supplying its heat tapes. I replaced the fuse (variac setting for "HOT" bake-out is close to fuse rating) but now wonder how long this fuse has been open. It would be an easy thing to overlook as the power lamp on the variac remains illuminated - you wouldn't expect that anything was wrong if you were just looking at it. Hmmm...
1322 hrs. local -> Leaving site now.
We lowered the HARD YAW loop gains at the LOWNOISE_ASC state. Please see the attached plots for the comparison between new (red traces) /old (blue traces) OLTFs.
For CHARD YAW (first plot), it seemed that we lost some phase in it and our original loop had only 10 deg of phase margin... I lowered the gain from 0.75 to 0.3 so that we have a UGF of 3 Hz and 30 deg phase margin. As we now have digital SS torque compensation to keep the plant power-independent and ISIFF to reduce the microseismic motion, it seemed that 3 Hz (or even lower) UGF should be sufficient. On the other hand, the phase at > 3 Hz needs some further investigation. With such a small phase margin we cannot adding extra low-pass to reduce its noise further.
For DHARD YAW the phase behavior is much better, but still for noise consideration we still lowered its gain from -60 to -40 to get a roughly 3 Hz UGF. From the measurement it suggested that we could engage FM2 in DHARD_Y to further reduce its noise, but before we can test it a fast lockloss happened.
I put the new gain settings into the guardian for now (without engaging the extra LP in DHARD_Y FM2). Before the fast lockloss happened I didn't see ASC oscillations due to the reduction in the ctrl BW, so it should be fine. After I relocked the IFO, I saw the violin modes got rung up again... The ETMY mode 1 and 6 were high (RMSLP_LOG10_OUTMON reaching ~ 5.5 or so), and the ITMX mode 1 reached RMSLP_LOG10_OUTMON value of almost 7...
I took the damping params from lscparams.py which seemed to be doing their work. However the damping was extremely slow, especially for the ITMX mode 1. For now I parked the IFO at the ENGAGE_DC_VIOLINS state (I also ran once the REMOVE_WHITENING in the OMC guardian to avoid DCPD saturation) and decided to go home. The new LSC MICH setup (LHO:44703) seemed fine for both grabbing lock and keeping IFO locked. I have not yet tested to go to NLN with the new, low-bandwidth C/D H yaw loops due to the high violin mode rms now.
I tried engaging DHARD Y FM2, and immediately lost lock. I retook the DHARD Y OLG using Hang's script at the end of the current LOWNOISE ASC state (which works now). UGF = 4 Hz. I used Jenne's TF predictor to see what the loop looks like after engaging FM2. The phase crossover is at 4.5 Hz, but the gain is > 1, so the loop goes unstable when engaging FM2. Probably, some gain reduction or less severe cutoff is required.
With the new front-end channels added to the GDS frame broadcaster channel list, I just restarted the testing calibration pipeline on DMT3. The restart occurred around GPS second 1224110900. The only feature we are unable to test at the moment is the calibration line subtraction, due to a few front-end channels that are still unavailable:
CAL-CS_TDEP_SUS_LINE1_REF_A_UIM_NOLOCK_REAL CAL-CS_TDEP_SUS_LINE1_REF_A_UIM_NOLOCK_IMAG CAL-CS_TDEP_SUS_LINE2_REF_A_PUM_NOLOCK_REAL CAL-CS_TDEP_SUS_LINE2_REF_A_PUM_NOLOCK_IMAG CAL-CS_TDEP_SUS_LINE3_REF_A_TST_NOLOCK_REAL CAL-CS_TDEP_SUS_LINE3_REF_A_TST_NOLOCK_IMAG
[Dave Barker, John Zweizig, Aaron Viets]
I restarted the calibration pipeline on the testing machine again to include calibration line subtraction, after Dave added the needed channels to the frame broadcaster. The restart occurred around GPS second 1224470390. John Z also restarted the DMTDQ process earlier today.
After successfully subtracting the Sidles-Sigg torque for the yaw HARD modes (LHO:44349), we are now able to subtract the pitch modes as well.
To compensate for the HARD modes, we need to digitally send in a torque signal corresponding to the soft mode motion. This means if we over-subtract the SS hard mode, we will create a digital soft mode which de-stabilize the system. To avoid such a situation, we should deliberately under-subtract the hard mode so that small errors in the radiation compensation path won't de-stabilize the plant. Consequently, instead of aiming to recover the free (0 W) pendulum, we only compensate the suspension back to the plant corresponding to 10 W input power (~ 65 kW arm circulating power).
Please see the attached plots for the results. The first one is for CHARD and second for DHARD.
In the plots the left panels correspond to the OLTF (IN2/IN1), and the right ones proportional to the suspension plant (OUT/IN1).
We use the following color convention: blue for 2 W input power, pink for 10 W input power (no digital compensation), green for 23 W input power (no digital compensation), and cyan for the 23 W input power together with digital subtraction of the SS torque. As said above, we intentionally subtract the plant back to the 10 W one instead of the 2 W one so that the subtraction is robust against both fluctuations in arm power & the sensing (i.e. the optical) gain.
In the future to use the RPC, we should put in a gain corresponds to
RPC gain = sign(ASC ctrl) * (P_in - 10 W) / (10 W),
where sign(ASC ctrl) is the sign of the DC gain in the regular ASC ctrl filter bank. E.g., currently the DHARD ASC ctrl filters have negative gains, then the sign of the RPC for the DHARD P/Y should be negative.
The LOWNOISE_ASC state still had filters engaging for pitch (already had commented them out for yaw) that move the frequencies of the ASC control filters, to compensate for the increased radiation pressure. Since the radiation pressure compensation is keeping the ASC plants looking like they are 10W, we don't need to do this. Having these filters on caused a lockloss, so Sheila and I have commented them out.
In the morning I saw the violin modes got rung up, might be due to that the IFO kept trying to go to NLN without letting the violin modes to settle down sufficiently first. This kept causing OMC DCPD saturation, which then led to locklosses, and rang up the violin modes further. To avoid such situations in the future we might add a checking state in the guardian (maybe at the ENGAGE_DC_VIOLINS state) to monitor the violin modes and if their rms contribution is too high, the guardian should stop keep going further.
======================================================================
While waiting the violin modes to be damped, we tried to restore the O2 longitudinal MICH setup to reduce its rms value (see LHO:44693).
Specifically, we turned on FM1 in MICH1 and FM3 in MICH2 to get more gain at < 4 Hz. The boosts ate some phase at the UGF~10 Hz, therefore we also modified the lp50 filter in MICH2 to a higher cutoff freq ~ 80 Hz to get more phase. I left FM2 in MICH2 unchanged but put the new LP filter to FM5 in MICH1 (originally a 2nd order butter lp at 80 Hz).
For the attached figures:
Fig. 1 is the comparison between the MICH error signal between the new (red; O2-like) and old (blue; past few months) filter setups.
Fig. 2 shows the original MICH2 FM2 (lp50) in blue and new MICH1 FM5 (lp80) in red. The new, higher cutoff filter should give about 10 more deg of phase compared to the original one.
Fig. 3 is the OLTF of the MICH loop. The blue trace is the original OLTF used for the past few months and the red is the new one. Both are measured at 2W at the ENGAGE_DC_VIOLINS state. For the new setup we have 10 Hz BW and a phase margin of 28 deg.
Fig. 4 is the filter bank setup for the new, high-LF-gain config.
Keita and I have been thinking about the 100 Hz noise in DARM that appeared when we had low CARM gain. Tonight I reproduced the 100 Hz DARM noise in NOMINAL_LOW_NOISE. I did this by lowering LSC-REFL_SERVO_IN1GAIN by -12 dB. The second plot is DARM before and after I reduced the CARM gain by -12 dB. We can see the return of the ~ kHz noise, as well as an increase in 100 Hz and below. The first is DARM and CARM coherence before and after. This is particularly interesting: DARM vs CARM coherence is completely unchanged, even at higher frequencies. The pdf is the CARM OLG I was taking when I lost lock. CARM UGF was ~ 4.5 kHz. Unclear what could be causing this nonlinear coupling between CARM and DARM. But it is good that we can turn a knob and control this noise. It is extremely suspicious that the 100 Hz and lower noise is not coherent with anything. Further investigations when we lock again. EDIT2: I was able to relock and get another spectrum with -6 dB of CARM gain (plot 2). Unfortunately I lost lock when I reduced the IMC gain by 6 dB to try and get a DARM spectrum there. The IFO doesn't like when you take away its laser frequency control...
This plot shows IMC-F, MICH, PRCL, and SRCL coherence with DARM at a time of high CARM gain (green), and low CARM gain (red). It seems that PRCL and SRCL become a lot more coherent with DARM at high frequencies when we have low CARM gain. But there are no broad features at or below 100 Hz. EDIT: Added the -6 dB LSC signals. Green is normal CARM gain, blue is -6 dB, purple is -12 dB CARM gain.
"DARM vs CARM coherence is completely unchanged, even at higher frequencies"
IMC board output is not CARM (Common ARM) in that it's not what the IFO sees. It's totally dominated by the frequency noise of PSL. No change in the coherence (which is mostly f>6.5kHz) is because the residual frequency noise coming from PSL is much considerably larger than the shot noise at that frequency.
If you look at REFL CM board output you'll see that the coherence changes with CM gain.
I and Daniel went to the floor on Friday while CM was at the nominal gain and the power was 20-something W to observe REFL_A_RF9 RF monitor.
On the scope it was about 10mV pp, the monitor has 23dB attenuation so the PD output should have been about 140mVpp. The largest component was 9MHz, closely followed by 72MHz, everything else was small.
RF power monitor in the demod board sometimes shows -5dBm or so but mostly -10dBm-ish.
From INMON channels it's clear most of the remaining RF was in Q phase (note the analog whitening gain of 12dB).
Nothing stood out.
- Had to rerequest HAM2 HPI ROBUST ISOLATED state to get HAM2 ISI to ISOLATED. Then was able to lock ALS. - Got back up to NOMINAL LOW NOISE at 23 watts. - Fast lockloss - Struggled to relock ALS, something to do with X arm fiber polarization being bad. I enabled the polarization controller, which seems to spin wildly until it's shut off. Eventually it found a better polarization for the X arm. - Lockloss as soon as I entered LOW NOISE COIL DRIVERS (directly after LOWNOISE ASC). - Again, lockloss as soon as I entered LOW NOISE COIL DRIVERS (directly after LOWNOISE ASC). - Stepped through the LOWNOISE ASC state by hand and made it with no problems. Added some wait times to the state, otherwise did not change. - For a third time, lockloss at the end of LOWNOISE ASC. This time the state didn't move onto LOW NOISE COIL DRIVERS because of the waits I added at the end of the state. The problem seems to be with H1:ASC-CHARD_Y FM3 (a 40 Hz cutoff filter) being turned on, but that could just be because that's the last thing that happens in LOWNOISE ASC. Could also be because LOWNOISE ASC is happening when the IFO is thermalizing, so optical gains could be changing under our feet. Adding more, longer waits... - Fourth time lockloss at the end of LOWNOISE ASC. Could see a clear pitch oscillation on the AS camera, turns out it was a DHARD P oscillation at 6 Hz (see plot 1). Probably a loop stability issue. - In LOWNOISE ASC at 23 watts, DHARD P FM7 (some LF boost/angular suspension resonance shaper thing) gets turned on, gain goes from -50 to -40, and FM1 (a 20 Hz cutoff filter) gets turned on. - Fifth time lockloss during LOWNOISE ASC, this time while I was going through slowly by hand. The last thing I did was step 4, engaging the arm ASC cutoff filters. (I did all four at once like an idiot.) I had left DHARD P at -50 gain. No 6 Hz oscillation this time, just a quick death. - Made it through LOWNOISE ASC by turning on the cutoff filters slowly and one by one. Made guardian so that it will do it that way too. - Leaving the IFO trying to lock DRMI with NOMINAL LOW NOISE requested. Range was 60 Mpc at NLN, absolutely CRUSHED by Livingston, they are literally off the charts here at Hanford.
About the polarization controller:
Immediately after turning on the polarization controller, click the restore button. 43559 Maybe we can edit the medm screen to make this more obvious. The problem is that the controller restores the paddles to the last position that they were manually set to (by turning knobs). Patrick created the restore button so that we can use it to reset the paddles to the last position that they were set to using the remote interface.
I roughly calibrated an IMC spectrum taken during one of our longer high power locks into frequency noise. This spectrum was taken with high CARM gain at 23 watts of input power. It was taken at IMC Servo Board Test1. Well informed guesses about the laser frequency control hierarchy during this measurement:MCL Crossover ~ 30 Hz CARM UGF ~ 10 kHz IMC UGF ~ 75 kHz FSS UGF ~ 200 kHzI relied on the calibration of the IMC VCO of 268302 Hz/V, times two for the doublepass AOM. There's also a VCO zero/pole of 40/1.6, the IMC Servo Board fast path signal coloring (Fast Option 200kHz pole, Fast HPF 70kHz/140kHz, and Fast Gain), and the FSS CLG. The calibration looks like this:where
is the measured IMC OLG, and
is the measured FSS OLG. (This FSS measurement is a little old, things could have changed, but the gain settings are the same: 20 dB Common and 9 dB Fast gain) I cut off the spectrum at 30 Hz because that's the MCL crossover freq and I haven't included the slow path in the model. The low frequency stuff greatly increased my RMS, but I doubt this is really the case since the suppression of MCL is insane. Frequency RMS from 30 Hz to 2 MHz: 1.1 mHz
I am baffled. If you measure at MC_F, you should see mainly frequency noise from the laser. The frequency noise of the VCO corresponds to ~1mHz/rtHz alone.
The last time I measured the FSS transfer function was back on October 9th. The gain slider settings were the same as before so I would not expect too much impact on your F(f).
Hang, Sheila, Dave, everyone
We had a repeat of this morning's crash of HAM2,3,4,5. 44678 44671
Dave walked us through the reset over the phone:
log into h1seih23 and h1seih45 as controls (ssh controls@h1seih23) (check secrets.ligo.org) run the command /etc/startWorld.sh to restart all the models on those machines. After they come back you will need to reset watchdogs and dackill.
It seems that the sensor correction gains already on this time.
If this continues to happen overnight people can do the restart this way. If it happens tomorrow during the day Dave might want to try shutting things down and starting them again, so we can give him a call.
As the proc files show, the DAC FIFOs are getting exhausted. This typically happens if the IOP cycle time is too long (so it doesn't get the DAC FIFO filled before it gets clocked out every cycle). We see a long cycle time on each (500 mu-sec) but unsure when it occurs. A common factor between seih23 and seih45 is the DACKILL IPC from IOP model on sush34, or it could be a DC power supply. I would definitely recommend I/O chassis power-cycle (or at least a front-end computer reboot). As I understand it, the SEI HAM front-ends have newer (2012 production) PCIe expansion fibers, so we hope they are not degrading.
Attached image show the state_word for h1iopseih23 (red) and h1iopseih45 (blue) as a second trend around the time of the problem.
The sequence is:
h1iopseih23 goes to DAC+DK for 1 second, then goes into DAC+DK+OVR for 7 seconds
h1iopseih23 returns to DAC+DK and stays there
7 seconds later, h1iopseih45 follows the same sequence.
Attached plot shows situation during this morning's 03:29 crash, same general sequence but the IOP order is reversed.
There is a comb in h(t) at 56.84044 Hz and multiples, identified in the Oct 11 low-noise data, with extreme amplitude modulation. In alog 44495 and follow up, it is found to be in many EX channels, including unused ADCs in the PEM system. The time when this line began has now been tracked to August 28 15:20 UTC. The attached spectrogram shows the lines appearing. The most likely cause that I can see in the alog is an update of the TwinCAT system manager on h1ecatx1. This is logged as starting at 14:54 UTC (alog 43706, and alog 43697 seems to indicate that the restart happened at 15:20 UTC.
I've found that there are sounds in the EX MINUSX mic related to the 56.84 Hz line beginning. There are a number of clicks, which sound like a door being opened or electronics boxes being opened. At each one, the 56.84 Hz either appears or disappears or there is a broadband glitch in the electronics. So it seems that the reboot is not the cause, rather something was jostled. Attached are a spectrogram, and an MP3 of the microphone, in case anyone recognises the sounds. I didn't see anything in the other PEM mic signals at EX.
Andrew, what a nice observation. It sounds like an electrical relay contact opening and closing to me. I don't know where this microphone is physically located in proximity to the air conditioning equipment for the EX VEA, but a component of that system is the first thing that comes to mind. Perhaps the heaters that are part of the dehumidification and temperature control scheme? Are you able to see correlation of the glitches iin the magnetometer?
There's a map and a picture of the location on pem.ligo.org. It looks like it's under the beam tube, near the floor. Since Aug 28, I haven't found any time where the line goes away in the ADC. It wasn't there before those noises, and it also hasn't changed since. We could check this in more detail if needed (I've just chosen a few dozen times at random). Was there anyone at End-X at this time (Aug 28 15:20 UTC), or was all of the work remote?
FYI, 56.8Hz and its harmonics were present in DCPDs last night even when there was no light, though the amplitude when dark was much much smaller than in-lock.
Red and blue are when IFO was locked with 20W, green and brown are when IFO was left unlocked last night and no meaningful light was coming to AS port.
Maybe this is a potential problem for all stations, just that it got much worse at EX on Aug 24?
Nice catch, Keita! I think this may be the cause of the amplitude modulation. The EX comb is at 56.84044 Hz, and the CS has a comb at 56.84078 Hz. DARM, in lock, seems to be a mixture of the two. I've found this comb in a few of the apparently unused ADCs for the CS PEM. The frequency is the same as seen in the DCPD when it's dark, as pointed out by Keita. The line is very stable and can be well-separated from the frequency in EX, especially when looking at the higher harmonics. The first plot shows these two lines in the ADCs. EX is marked by the red vertical line and CS by green. The second plot compares the DCPD when dark and in lock. Only the CS line is evident when dark. When in lock, it's harder to resolve the lines since there's not as much data. But it is clear that the CS line disappears. But now there are symmetric sidebands around where it was: one matches the EX line, and there's a new line marked blue which is mirrored across the CS line. The separation of these matches the amplitude modulation seen in DARM with a 50-minute total period. It's hard to understand the exact mechanism here, and why the central line should disappear. It does seem that both CS and EX have combs but they can be distinguished by the precise frequency (needs 0.1 mHz resolution).
During the low noise lock at 4:50 UTC today (Oct 11), there is a comb of lines in h(t) at 56.84 and multiples. They are quite narrow lines. The most interesting characteristic is that they have extreme amplitude modulation. The first attachment is the h(t) spectrum with the lines marked. The second shows narrow BLRMS around each of the lines. The fundamental at 56.84 Hz has a half-period of about 24 minutes. The amplitude goes very close to zero, and looks very periodic. Each multiple n has zeros in the same place, but the period is n times shorter. Maybe this could be all generated from the fundamental with the right kind of nonlinearity. I'm not sure what would give such a slow but extreme modulation - maybe some centering servo? The BRUCO results show a lot of channels at End-X coherent. Accelerometers on BSC9 seem to see it, but without amplitude modulation. PCal X sees it in the TX and RX PDs, but stronger in RX; but it doesn't seem anywhere near the level to get into h(t).
The 56.84 Hz comb is a blast from the past. Here is an entry from March 2013 concerning H2 one-arm data suggesting that the comb could be purely DAQ-related: https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=5806
This one is close enough to the 57Hz ETM HWS that we've seen in the past that I thought I should mention it here. However, we've recently overhauled that system and I've not confirmed what the new frame-rate is yet.
Here's the old analysis: https://alog.ligo-la.caltech.edu/aLOG/index.php?callRep=14778
I Looked at an empty EX ADC channel (H1:PEM-EX_ADC_0_08_OUT_DQ) and found that the 58.64 Hz comb is there too, indicating that the comb comes from some source independent of the interferometer (i.e. in the DAQ or RF pick-up). I have attached ASDs of the empty ADC in 0.1mHz bins both for the entire spectrum (0-1024Hz) and near to the 14th 56.84 harmonic (~795.76Hz). I was slightly surprised that a second nearby line (that would explain the amplitude modulation) was not seen even with the 0.1mHz resolution.
Yes - the ETM HWS are running at 57Hz. This doesn't mean that this is the issue since it appears to be DAQ related, buuuut .....
aidan.brooks@zotws11:~$ caget H1:TCS-ETMX_HWS_SYNC_FREQUENCY
H1:TCS-ETMX_HWS_SYNC_FREQUENCY 57 Hz
aidan.brooks@zotws11:~$ caget H1:TCS-ETMY_HWS_SYNC_FREQUENCY
H1:TCS-ETMY_HWS_SYNC_FREQUENCY 57 Hz
I've searched EX channels and found some channels where amplitude modulation is seen: H1:SUS-ETMX_L3_DRIVEALIGN_L_OUT_DQ H1:SUS-ETMX_L3_ISCINF_L_IN1_DQ H1:SUS-ETMX_L3_LVESDAMON_LL_OUT_DQ H1:SUS-ETMX_L3_LVESDAMON_LR_OUT_DQ H1:SUS-ETMX_L3_LVESDAMON_UL_OUT_DQ H1:SUS-ETMX_L3_LVESDAMON_UR_OUT_DQ H1:SUS-ETMX_L3_MASTER_OUT_LL_DQ H1:SUS-ETMX_L3_MASTER_OUT_LR_DQ H1:SUS-ETMX_L3_MASTER_OUT_UL_DQ H1:SUS-ETMX_L3_MASTER_OUT_UR_DQ Attached plots are BLRMS and zoomed spectrum of H1:SUS-ETMX_L3_DRIVEALIGN_L_OUT_DQ as an example. The vertical line (orange, dotted) in the zoomed spectrum is at 568.404 Hz (10th harmonics).
NO
we moved the Hartmann sync frequency and the lines moved accordingly - its Hartmann, not DAQ.
Why does the Hartmann sensor so strongly couple to DARM?
With regards to the coupling, the ETM HWS are served by the same power supply as the ring heaters. It's a long shot but it might be worthwhile to try disconnecting the ring heaters from the driver while the to see if the coupling of the 57Hz to DARM is changed.
https://dcc.ligo.org/LIGO-E1100891
Recovery started a few minutes ago. So far, resetting all SEI watchdogs and then requesting "recover EQ" seems to be bringing everything back online.
Suspensions reset okay, and I also brought the SEI_Config node to windy_nobrsx, so we've got sensor correction going as well.