Investigating a dolphin crash of several corner station front ends.
I first verified for the IOPs with DACKILLs that their IO Chassis could be seen, which was the case.
I restarted all the models (to clear IOP DACKILLS) on h1oaf0, h1lsc0, h1susb123, h1sush2a, h1sush34, h1sush56. I put all the corresponding SEI IOP SWWD into bypass mode before restarting the SUS IOPs.
All the models restarted with no issues, no reboots (with their associated Dolphin fencing) were needed.
In the morning I'll see if I can find which front end caused the Dolphin crash.
The EDC is slow in reconnecting to susmc3 and susprm channels, but it looks like it will get there in time.
Sat21Oct2023
LOC TIME HOSTNAME MODEL/REBOOT
01:48:20 h1oaf0 h1iopoaf0
01:48:34 h1oaf0 h1pemcs
01:48:48 h1oaf0 h1tcscs
01:49:02 h1oaf0 h1susprocpi
01:49:16 h1oaf0 h1seiproc
01:49:30 h1oaf0 h1oaf
01:49:44 h1oaf0 h1calcs
01:49:58 h1oaf0 h1susproc
01:50:12 h1oaf0 h1calinj
01:50:26 h1oaf0 h1bos
01:52:02 h1lsc0 h1ioplsc0
01:52:16 h1lsc0 h1lsc
01:52:30 h1lsc0 h1lscaux
01:52:44 h1lsc0 h1sqz
01:52:58 h1lsc0 h1ascsqzfc
01:55:25 h1susb123 h1iopsusb123
01:55:39 h1susb123 h1susitmy
01:55:53 h1susb123 h1susbs
01:56:07 h1susb123 h1susitmx
01:56:21 h1susb123 h1susitmpi
01:58:57 h1sush2a h1iopsush2a
01:59:11 h1sush2a h1susmc1
01:59:25 h1sush2a h1susmc3
01:59:39 h1sush2a h1susprm
01:59:53 h1sush2a h1suspr3
02:01:57 h1sush34 h1iopsush34
02:02:11 h1sush34 h1susmc2
02:02:25 h1sush34 h1suspr2
02:02:39 h1sush34 h1sussr2
02:04:13 h1sush56 h1iopsush56
02:04:27 h1sush56 h1sussrm
02:04:41 h1sush56 h1sussr3
02:04:55 h1sush56 h1susifoout
02:05:09 h1sush56 h1sussqzout
TITLE: 10/21 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Observing at 162Mpc
INCOMING OPERATOR: Camilla (OWL)
SHIFT SUMMARY: Observing and Locked for 35hrs 45mins.
LOG:
23:00UTC Detector in Commissioning and has been Locked for 28hours
23:02 Back to Observing
23:26 Earthquake mode activated due to incoming earthquake from Guatemala
23:36 Back to CALM
Observing at 164Mpc and have been Locked for 32 hours.
Nothing to report
Gabriele, Jim, Arnaud
We have been looking for the 1.23Hz line seen largely in DARM and other loops (73622). We think this is coming from the ETMX ISI, which shows a strong line at the same frequency mostly in the Y and RZ dofs. The summary pages from today show the evidence of this problem, as well as the ndscope attached in fig1. Trending back the signal, this issue started around October 01 at 05am PDT, see bottom plot on fig2. Spectrum of all individual sensors (in loop) before and after are attached. Some outliers in the stage 1 sensors (H1 L4C, Y1, Y2 T240 ...) but hard to tell what's the problem without taking the ISI down. More to follow at next opportunity (maintenance, or ifo down).
TITLE: 10/20 Day Shift: 15:00-23:00 UTC (08:00-16:00 PST), all times posted in UTC
STATE of H1: Observing at 165Mpc
INCOMING OPERATOR: Oli
SHIFT SUMMARY:
| Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
|---|---|---|---|---|---|---|
| 18:38 | FAC | Tyler | EndY | N | Rodent deterant | 19:38 |
| 20:00 | ISC | Gabriele | REMOTE | N | SRCL FF retune | 20:17 |
| 20:08 | SEI | Jim | Office | N | Peakmon filter | 21:21 |
| 20:17 | SQZ | Sheila, Naoki | CR | M | SQZ tests | 22:49 |
| 21:12 | ISC | Gabriele | REMOTE | N | SRCL FF retune | 21:20 |
| 21:32 | VAC | Gerardo | FCES | N | Grab tool to access parts for measurement, left encl at 22:10 | 22:16 |
| 22:16 | ISC | Gabriele | REMOTE | N | Noise reduction test | 22:26 |
TITLE: 10/20 Eve Shift: 23:00-07:00 UTC (16:00-00:00 PST), all times posted in UTC
STATE of H1: Commissioning
OUTGOING OPERATOR: Ryan C
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 7mph Gusts, 5mph 5min avg
Primary useism: 0.03 μm/s
Secondary useism: 0.31 μm/s
QUICK SUMMARY:
We're Commissioning and are going on 28hours Locked.
(Randy P., Chris S., Travis S., Gerardo M.)
After flipping the septum carefully with the forklift to get to the non-grooved side, the new door for HAM7 +X side got new threads on what it used to be the annulus thru hole. Now, the door is in queue to get cleaned. We used this aLOG as a guide, as well as the following drawing taken from the DCC D1900512. Only IPA alcohol was used to cut the threads.
Naoki, Sheila
Summary: We have repeated the test from 73400, with our higher level of squeezing after the crystal move, and with the ASC off for squeezing angles other than squeezing. This seems to generally reproduce the results in that alog, convincing us that most of the rotation we are seeing is from the mode matching change not an alignment shift.
We have saved references in userapps/sqz/h1/Templates/dtt/DARM/PSAMS_tests_Oct202023.xml. This is a copy of PSAMS_tests_Oct112023.xml, with the references saved there and shown in alog 73400 The times written below are minimum times, for many of these references there is actually more time.
times (PSAMs 200/200):
set sqz angle back to sqz, turned AS42 and FC ASC, set PSAMs to 120/120 with 30 second ramp
times (PSAMS 120/120):
For PSAMS 120 vs. 200, adding in subtracted squeezing dBs data for this 20 Oct 2023 dataset (new crystal spot with more sqz, better controlled alignment), and from our initial dataset LHO:73400 12 Oct 2023 (before the crystal move so less sqz, ASC running the whole time). A model of unsqueezed quantum noise is used as the reference for the subtraction; relevant GWINC IFO parameters are in the plot title.
Now working on fitting the mode-mismatch amplitudes + phases to a GWINC quantum noise model.
RyanC, Camilla
I wrote some code to plot 5 months of ESD actuation force on ETMX based on equation 10 from T1700446 using the coefficients derived by the inlock charge measurements, I wasn't sure the values to use for δVs so the exact numbers for force are incorrect but the trend should be the same. We can see that the ESD actuation force for ETMX has been increasing since June.
Regarding Kappa_TST there was a step down at the end of August which was from the CAL team updating the calibration to bring Kappa closer to 1 alog72594, but we can see that it has been slowly increasing since June. KAPPA_TST increasing means that for the same amount of digital counts, the ESD is applying more force which matches what the figures are showing.
The code is located at /ligo/home/ryan.crouch/Desktop/Charge_measurements/Force_calc_from_inlock.py and it needs to be run in the aligoNB conda environment.
Following up on the observtion that the SRCL FF is increasing the DARM low frequency (2-3 Hz) motion (73561) and that this motion might be the origin of non-stationary noise (73546), I designed a new high pass filter for SRCLFF and retuned the FF.
The new high pass gives us about 4 times less reinjection between 2 and 4 Hz, at the price of a phase rotation of 20 degrees at 10 Hz. Therefore the old SRCL FF was not performing well anymore. I did a noise inection and refit the FF filter. Fitting this filter is tricky and I don't think I got an optimal solution. But it looks like the new highpass with the new FF performs fine, is at least as effective as the old highpass and the old FF, by looking at the coherence between DARM and SRCL.
DARM RMS is improved a little between 3 and 4 Hz, so maybe we'll see some improvement in the non-stationary noise. We need to figure out what else is making DARM noise at 3.4 Hz. There is also an enormous line at 1.23 Hz that is the cause of most of DARM RMS. I think the bicoherence shows that the 1.23 Hz line is a source of noise modulation too.
Noticed the old SRCLFF1 HP filters were being turned on in sdf revert so updated the safe.snap for the current highpass and filter, attached.
Fri Oct 20 10:11:27 2023 INFO: Fill completed in 11min 23secs
Gerardo confirmed a good fill curbside.
After Gabriele's 73546, Jenne and I looked LSC-DARN_IN1 before and after the feedforward was added. You can see the new FF (in particular SRCL) has more noise 5 to 9Hz, plot attached of only old vs new FF. The larger peaks at 0.05 and 0.1Hz are caused by increased microseism, they are not present on Oct 14th. No other noise down to 1e-4Hz.
We turned off the MICH FF only 20:07UTC - 20:21UTC and then the SRCL FF only 20:22UTC - 20:35UTC. Turned off both FF from 20:35UTC until 20:42UTC where we had a lockloss. Plot attached of LSC-DARN_IN1 in these configurations.
As part of thinking about why a little bit of excess motion around a few Hz can cause nonstationarity in the GW band, I had a cursory look at the filters in place for the ETM R0 tracking.
I have not yet re-looked to see why we made different choices for the angular vs length loops, but the top frame of the attached plot shows the magnitude of the control filters for ETMY's R0 tracking for pitch and length. The bottom frame shows the rough magnitude of the open loop gain of the length loop, using a filter that was in the filter bank called 'plant', and including the gain of 20 that is in the filter bank.
So, it looks like the angular loops (I'm assuming the yaw filters, which are named the same, are actually the same as the pitch filters) emphasize this region where we've got a bit of excess motion in DARM due to the LSC FF, but the length loop does not. We should give this some thought, but consider adding some emphasis to the length tracking filter to have the length tracking work up to ~5 Hz or so.
A parallel path forward is that we should also consider is changing the highpass in the SRCL FF filter bank to be more severe (thus sacrificing some phase and efficacy around 10 Hz, but hopefully an overall win above 20 Hz), however this implies doing the full remeasurement and refitting of the SRCL FF. We can work on this later this week, or whenever we next have some commissioning time.
Here's a design for a more aggressive high pass filter, at the price of 20 degrees of phase rotation at 10 Hz. I think we might need a retuned of the SRCLFF after implementing it. The new high pass is saved into FM10 of SRCLFF1 (not yet loaded).
The noise reinjection between 2 and 4 Hz should be about 4 times lower if we manage to retuned the SRCLFF filter without increaasing the low frequency gain.
Using two periods of quiet time during the last couple of days (1381575618 + 3600s, 1381550418 + 3600s) I computed the usual coherences:
https://ldas-jobs.ligo-wa.caltech.edu/~gabriele.vajente/bruco_STRAIN_1381550418/
https://ldas-jobs.ligo-wa.caltech.edu/~gabriele.vajente/bruco_STRAIN_1381575618/
The most interesting observation is that, for the first time as far as I can remember, there is no coherence above threshold with any channels for wide bands in the low frequency range, notably between 20 and 30 Hz, and also for many bands above 50 Hz. I'll assume for now that most of the noise above ~50 Hz is explained by thermal noise and quantum noise, and focus on the low frequency range (<50 Hz).
Looking at the PSDs for the two hour-long times, the noise belowe 50 Hz seems to be quite repeatable, and follows closely a 1/f^4 slope. Looking at a spectrogram (especially when whitened with the median), one can see that there is still some non-stationary noise, although not very large. So it seems to me that the noise below ~50 Hz is made up o some stationary 1/f^4 unknown noise (not coherent with any of the 4000+ auxiliary channels we record) and some non-stationary noise. This is not hard evidence, but an interesting observation.
Concerning the non-stationary noise, I think there is evidence that it's correlated with the DARM low frequency RMS. I computed the GDS-CALIB RMS between 20 and 50 Hz (whitened to the median to weight equally the frequency bins even though the PSD has a steep slope), and the LSC_DARM_IN1 RMS between 2.5 and 3.5 Hz (I tried a few different bands and this is the best). There is a clear correlation between the two RMS, as shown in a scatter plot, where every dot is the RMS computed over 5 seconds of data, using a spectrogram.
DARM low frequency (< 4 Hz) is highly coherent with ETMX M0 and R0 L damping signals. This might just be recoil from the LSC drive, but it might be worth trying to reduce the L damping gain and see if DARM RMS improves
Bicoherence is also showing that the noise between 15 and 30 Hz is modulated according to the main peaks visible in DARM at low frequency.
We might be circling back to the point where we need to reconsider/remeasure our DAC noise. Linking two different (and disagreeing) projections from the last time we thought about this, it has the correct slope. However, Craig's projection and the noisemon measurement did not agree, something we never resolved.
Projection from Craig: https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=68489
Measurement from noisemons: https://alog.ligo-wa.caltech.edu/aLOG/uploads/68382_20230403203223_lho_pum_dac_noisebudget.pdf
I updated the noisemon projections for PUM DAC noise, and fixed an error in their calibration for the noise budget. They now agree reasonably well with the estimates Craig made by switching coil driver states. From this we can conclude that PUM DAC noise is not close to being a limiting noise in DARM at present.
To Chris' point above -- we note that the PUMs are using 20-bit DACs, and we are NOT using and "DAC Dither" (see aLOGs motivating why we do *not* use them in LHO:68428, and LHO:65807, namely that [in the little testing that we've done] we've seen no improvement, so we decided they weren't worth the extra complexity and maintenance.)
If at some point there’s a need to test DAC dithers again, please look at either (1) noisemon coherence with the DAC request signal, or (2) noisemon spectra with a bandstop in the DAC request to reveal the DAC noise floor. Without one of those measures, the noisemons are usually not informative, because the DAC noise is buried under the DAC request.
Attached is a revised PUM DAC noisemon projection, with one more calibration fix that increases the noise estimate below 20 Hz (although it remains below DARM).
Summary: To lock green arms I put ITMs, BS, PR2 and PR3 back to Oplev positions before trip/last time we were aligning. Also touched ETMs and TMSs as usual in green arms aligning. During intal alignment I touched H1:LSC-XARM_GAIN (and reverted), PR2, PRM, SRM.
Documented in FRS 29441. Ryan C got us back to NLN at 20:16UTC by restoring all the sliders to when we were locking before the previous lock and then running another initial alignment alog 73637.
Restoring sliders with (sitemap > ASC > Init Align Compat > ! Restore) should have been the first thing to do as during the computer restart the sliders are reset to their safe.snap files, which are not (currently) recent values. Tagging OpsInfo and updated Recovery from Dolphin/Computer Crash wiki.