ryan.crouch@LIGO.ORG - posted 18:50, Friday 31 May 2024 - last comment - 20:24, Friday 31 May 2024(78170)
Lockloss from Earthquake
01:49 UTC Another EQ lockloss, this time a 5.6 from Tibet
01:49 UTC Another EQ lockloss, this time a 5.6 from Tibet
Out buildings look fine
CS fans shows work on fans 5 and 6
Closes FAMIS 26443
The beginning of the BRS X plot shows work being done on the pico alog76888
BRS Y looks good
J. Kissel, F. Clara At my wits' end with measuring the analog response of the OMC DCPD transimpedance amplifiers via the OMC DCPD whitening chassis's preamp monitor interface (LHO:78090), the only thing left to question was the SR785 we'd been using that had been out by the SQZ and HAM6 racks -- a trusty race horse that had been in service out on the floor for a long time. Fil and I have discovered that indeed, it was time to put this race horse out to pasture. We pulled the unit from the floor, grabbed a random spare electronics chassis we had lying around (Fil chose a D0902783-style anti-aliasing chassis), and measured it's transfer function using a suspected good, and the suspected bad SR785 from the floor. Our suspicions were proved correct: the floor SR785 that we'd been using for months (if not years) is broken in a nasty, subtle way. Check out the attached comparison of transfer functions of the same electronics channel, but between - The "bad" floor SR785, where the CH1 and CH2 inputs are in differential, (A-B) mode :: the configuration that's needed for the measurements I was doing for months, - The "bad" floor SR785, where the CH1 and CH2 inputs are in singe-ended, (A only) mode :: totally fine - The "good" EE shop SR785, where the CH1 and CH2 inputs are in differential :: totally fine - The "good" EE shop SR785, where the CH1 and CH2 inputs are in singe-ended, (A only) mode :: totally fine If you compare the ratio of the "bad" floor SR785 in differential mode to the "good" SR785 in differential mode against the "measurement setup" with pages 3,4, and 5 of the plots from LHO:78090, then you see exactly the same "gain is 2 [V/V], and has a frequency dependent hump above 1 [kHz]" that drove me insane this past Tuesday. GUH. It's been a rough couple of months when you find out you've been using a calculator that computes 2 + 2 = 5... The unit is now boxed up and off for "re-calibration," which (a) it hadn't had in a really long time, and (b) usually means that we'll get it back fully-functional.
TITLE: 05/31 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing
INCOMING OPERATOR: Ryan C
SHIFT SUMMARY: Pretty easy shift with one lockloss due to an earthquake, but relocking was automated besides needing to take the ifo out of initial alignment due to not being able to get past SRC_ALIGN and having SRM M2 and M3 watchdogs trip, and then having to run SRC in manual IA state by state. There were no issue when going state by state.
LOG:
14:30 Detector Observing and Locked for almost 15 hours
16:19 Earthquake mode activated from a couple large earthquakes
16:32 Out of Observing and into Commissioning to try and get more range running SQZ and A2L script 78155
16:45 Lockloss due to earthquake
- SRM was saturating before the LL
- Holding in DOWN until earthquakes pass
17:49 Back to CALM
17:50 Started Initial Alignment
18:14 Couldn''t get past SRC_ALIGN
- SRM M2 M3 watchdogs tripped, I took us out of initial alignment
18:18 Untripped SRM WDs, went to MANUAL_INITIAL_ALIGNMENT
- Ran SRC alignment states one at a time
- No issue at all getting through SRC
18:23 Back to DOWN and ready to start relocking
19:09 NOMINAL_LOW_NOISE
19:13 Observing
23:32 Kicked out of Observing by SQZer losing lock
23:36 Back into Observing
| Start Time | System | Name | Location | Lazer_Haz | Task | Time End |
|---|---|---|---|---|---|---|
| 16:31 | Terry | Optics Lab | y(local) | Working on SHG | 18:59 | |
| 18:20 | PCAL | Tony | PCAL Lab | y(local) | Setting up for measurements | 18:40 |
| 20:29 | Terry | Optics Lab | y(local) | Continuing work on SHG | 23:20 | |
| 22:22 | PCAL | Rick | PCAL Lab | y(local) | picking up stuff | 23:35 |
TITLE: 05/31 Eve Shift: 2300-0800 UTC (1600-0100 PST), all times posted in UTC
STATE of H1: Observing at 152Mpc
OUTGOING OPERATOR: Oli
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 7mph Gusts, 4mph 5min avg
Primary useism: 0.02 μm/s
Secondary useism: 0.06 μm/s
QUICK SUMMARY:
Fri May 31 10:09:24 2024 INFO: Fill completed in 9min 20secs
Jordan confirmed a good fill curbside.
Lockloss @ 05/31 16:45 UTC due to earthquake
We were out of Observing to touch up SQZ and A2L when we lost lock (78155). We were able to finish adjusting SQZ and running A2L for ETMX.
19:13 UTC Back to Observing
Out of Observing at 16:32 UTC to try and tune up A2L and SQZ. We might be losing lock from earthquakes in a few minutes.
Our range hovered around 150 Mpc overnight, here's a screenshot of the darm coherence checks from the operator LowRangeChecks
This shows that some of our noise around 20 Hz is due to ASC, so we want to run A2L to see if we can improve that.
Want to change gain from 3.19 to 3.25, rounded to 2 decimal places. St.Div is 0.56. Change of -0.061
H1:SUS-ETMX_L2_DRIVEALIGN_P2L_SPOT_GAIN => 3.25
Want to change gain from 4.89 to 4.9, rounded to 2 decimal places. St.Div is 0.99. Change of -0.011
H1:SUS-ETMX_L2_DRIVEALIGN_Y2L_SPOT_GAIN => 4.9
by the time we started ETMY the earth was shaking so much that the measurement wasn't good. I've put these ETMX values into the guardian.
TITLE: 05/31 Day Shift: 1430-2330 UTC (0730-1630 PST), all times posted in UTC
STATE of H1: Observing at 149Mpc
OUTGOING OPERATOR: Tony
CURRENT ENVIRONMENT:
SEI_ENV state: CALM
Wind: 7mph Gusts, 5mph 5min avg
Primary useism: 0.01 μm/s
Secondary useism: 0.06 μm/s
QUICK SUMMARY:
We're Observing and have been Locked for almost 15 hours. Everything looks nominal
TITLE: 05/30 Eve Shift: 2300-0800 UTC (1600-0100 PST), all times posted in UTC
STATE of H1: Lock Acquisition
INCOMING OPERATOR: Tony
SHIFT SUMMARY:
After locking H1 at the beginning of the shift, there was some Squeezer work for about an hour and then H1's been Observing for most of the rest of the shift (except for a quick drop noted below). And in the final minutes of the shift was capped off with a gravitational wave candidate.
LOG:
At around 0528utc, noticed a slight step down in range. The telltale tells of this instance showed a slight increase in DARM from 18-30Hz resulting in a drop in range of about 5Mpc. Took this opportunity to drop H1 from Observing to Turn OFF Squeezing, run a DARM spectrum, and then return to Observing.
Plot #1:
Plot #2: Range over the last 6hrs with the drop in range noted.
This dtt measurement is located at: /ligo/home/corey.gray/Templates/dtt/DARM_05312024.xml
H1's been locked for 4hrs (& observing for ~3hrs). Range is hovering just under 150Mpc. Mindful of any drops in range so that I can drop out of Observing, turn OFF squeezing and then run a measurement for Sheila---just to see whether we can determine if Squeezing is responsible for range drops....but I've not had the opportunity so far tonight.
LSC's SDF Diff for MICH Feed Foward gain change (0.97 to 1.00) was confirmed by Sheila and accepted. (1st attachement)
LSCAUX's SDF Diff for LSC LOCKIN_1_Freq (& TRamp) were a mystery to Sheila, but since the amplitude was 0 for this, they were ACCEPTED. (2nd attachment)
Continuing the effort I started in alog77692, I've created histograms showing the various frequencies used in the FIND_IR main locking state for both COMM and DIFF. For COMM I used the actual VCO frequency (since the offset is zeroed early on) and for DIFF I used the offset value. The data starts from the beginning of O4.
It became clear as I gathered the data for these plots that for both COMM and DIFF there are two main groups where the frequency likes to be for IR locking. There were also occasional outliers which look to be from a person finding the IR resonance by-hand moving the COMM or DIFF offset.
COMM:
Lower group offsets: {78893888.0, 78893760.0, 78893664.0, 78893568.0, 78893824.0, 78893856.0, 78893832.0, 78893864.0, 78893512.0, 78893712.0, 78893616.0, 78893880.0, 78893840.0, 78893808.0, 78893912.0, 78893816.0}
Upper group offsets: {78931328.0, 78931232.0, 78931432.0, 78931080.0, 78931368.0, 78931376.0, 78931184.0, 78931440.0, 78931280.0, 78931128.0, 78931032.0}
DIFF:
Lower group offsets: {385.0, 388.0, 390.0, 391.0, 394.0, 394.5, 395.0, 397.0, 398.3, 399.0, 400.0, 402.0, 403.0, 405.0, 406.0, 408.0, 409.0, 410.0, 411.0, 412.0, 413.0, 414.0, 415.0, 417.0, 418.0, 420.0, 421.0, 423.0, 424.0, 425.625, 426.0, 425.0, 427.0, 429.0, 430.0, 432.0, 435.0, 364.0, 496.0, 379.0, 381.0}
Upper group offsets: {3206.5, 3232.0, 3232.9, 3212.5, 3212.0, 3215.5, 3215.0, 3218.5, 3218.0, 3220.0, 3221.0, 3221.5, 3223.0, 3224.0, 3224.5, 3225.0, 3227.0, 3227.5, 3226.0, 3230.0, 3230.5, 3231.0, 3233.0, 3233.5, 3229.0, 3236.0, 3236.5, 3238.0, 3239.0, 3239.5, 3241.0, 3242.0, 3242.5, 3244.0, 3245.0, 3245.5, 3247.0, 3248.0, 3248.5, 3250.0, 3251.0, 3251.5, 3249.0, 3254.0, 3254.5, 3256.0, 3257.0, 3257.5, 3259.0, 3260.0, 3261.0, 3263.0, 3265.0, 3266.0, 3268.0, 3272.0, 3401.0, 3275.0, 3277.0, 3221.2, 3278.0, 3281.0, 3222.0, 3249.2, 3362.5, 3252.0, 3436.0, 3235.0, 3191.0}
J. Kissel, [emotional support from L. Dartez remotely] Executive Summary Contrary to the grand plan, I instead unplugged and re-plugged in some cables, and lost 4 hours of my life. The OMC DCPD electronics chain remains as it did before today. I'm confused yet again. Full Summary After we last left the saga of "the in-vacuum OMC DCPD Transimpedance Amplifier (TIA) frequency response has changed a little since the OMC Swap; let's re-measure it in analog" (LHO:77735), the 2024-May-13 plan was to swap the apparently broken S2300003, D2200215-style OMC DCPD Whitening Chassis, for a spare (I chose S2300002), in hopes that we'd be able to make a new analog measurement of the TIA without the non-linearity in the whitening chassis' measurement setup. Today, the installed-in-the-racks measurement setup using S2300002 got even more weird :: very different from the exact same setup of the same chassis in the EE shop. "Weird" is :: 2 [V/V] rather than 1 [V/V], and :: a very obvious, 10 [%] / 3 [deg] wiggle, frequency response above 1 kHz, rather than "subtle, slightly different nonlinear roll off." Indeed, to confirm my insanity, I quickly switched over to CH B of S2300003 while there with everything else the same, and it's similarly weird -- and different from the previous in-rack measurements taken only a few weeks ago where I'd probed its subtle non-linearity around 1 [V/V]. Just maddening. That being said, after dividing the maddening response of the measurement setup out of the data we really want -- the analog measurement of the in-vacuum TIA -- we see a good clean measurement, with finally with the same DC gain as the good old 2023-03-10 measurements. BUT -- DCPD A's response shows change from the best 2023-03-10 measurement data set -- but at *high* frequency rather than "below 25 Hz." I don't think this is real. More on this in a follow-up aLOG. In the end, after four hours in front of the SR785, Louis and I decided to revert the signal chain back to using S2300003 as had and has been the case for all of O4. I'll hang my head, eat my hat, put foot in mouth, and head back to the EE shop. Attached is a collection of plots the shows today's results. :: Pages 1 & 2, for DCPDA and DCPDB, respectively, the TIA response, with the maddening measurement setup divided out. One can see that the for DCPDB (page 2) that today's 2024-05-28 measurement divided by the currently installed compensation (from 2023-03-10) agrees very well with the 2023-03-10 measurement over compensation. However, for DCPDA (page 1), we see a diverence from the model and old measurement above 1 kHz. :: Pages 3 & 4, for DCPDA and DCPDB, again show the obscene difference between today's measurement setup's response -- the 2 [V/V], with 10 % / 3 deg wiggle above 1 kHz -- and the previous "bad" result from 2024-04-23, and the originally "I'm happy with that" 2023-03-10 result, both of which are 1 [V/V] with "a little bit of phase loss, but that's it." :: Page 5 shows the ratio of all of today's measurement setup responses, showing that even from channel to channel and chassis to chassis, I get this exact same response. I even swapped out the SR785 accessory box for another to see if this was the issue, and I got the same response. The only piece of electronics left as suspect is the SR785. But -- (1) Out of the utmost caution, I run a factory reset on any SR785 I touch before I begin configuring it for measurement, and (2) This is a *differential* transfer function measurement with the SR785. That means the ratio CH2 (A-B) / CH1 (A-B) is changing. I find it quite hard to believe that this could be happening. And in case you're suspicious that "Oh, you must have blown the input electronics by driving too hard into it," We're careful to keep the input voltage for each A and B spigot of CH1 and CH2 below the specified 5 [V_pk], and we're watching for its ADC saturations at all times. So, ya, maddening. More details in the comments for posterity.
First attachment: pictures of the momentary installation of S2300002 into ISC-R5, both where I landed it in U26, and then followed up with pictures of the measurement while measuring it. Second attachment: A copy forward of the diagrams of the measurement setup that I followed during the above measurements. (The pictures from above show it while I was measuring the TIA; I did not take pictures of the measurement setup, but I *promise* you it matches the diagram. I checked 15 times once things started to go pear-shaped with the frequency-dependent, 2 [V/V], business.) Third attachment: A post-mortem collection of photos proving that the system has been completely reverted, leaving only S2300003 in U24, with all cables reconnected.
The data and script to quickly plot it from today's work lives in
/ligo/svncommon/CalSVN/aligocalibration/trunk/
Common/Electronics/H1/DCPDTransimpedanceAmp/OMCA/S2100832_SN02/20240528/Data/
I attach the measurement notes.
It was the SR785 that has been slowly failing in a subtle way. That's why the "measurement setup" measurements have been continually changing, and in this aLOG look totally bonkers with the 2 [V/V] and a frequency-dependent wiggle above 1 kHz. See LHO:78165.
Back to Observing at 03:24UTC