J. Kissel [with support from L. Dartez, F. Llamas, and D. Schaetzl]

Executive Summary
In Feb 2024, we identified that both the DCPDA and DCPDB channels of the in-vacuum OMC DCPD transimpedance amplifier (TIA) frequency response dropped by 0.3% below ~25 Hz (LHO:75986) and that the frequency-dependent balance of the DCPDs changed in a different way at the 0.2 dB or 10^(0.2/20) = 2.5% level (LHO:76232). Since then, with what limited person power we had, we've been scrapping together ~2 hours of time during maintenance days to try to re-characterize the response of the TIA in order to fit and re-compensate the response, and then re-re-balance the DCPDs. We've also done a lot of thinking in between. This aLOG covers the saga since then, and executive summary is that we don't have good enough measurements of the TIA response yet, we have broken the functionality of the TIA monitor path in the S2300003 D2200215 whitening chassis, and it needs to be replaced in order to move forward with characterizing the in-vacuum TIA.

Timeline
    Editorial note: You'll find that the story is fraught with missing institutional knowledge due to lack of person power and interferometer time. It would pay for us to have had a spare TIA system *out* of vacuum to test our characterization methods and train new staff.

    . 2024-02-26: In LHO:75986 after measuring twice to ensure we weren't being fooled by vacuum pressure and thermalization, we use the remote, DAC driven characterization setup -- measuring the product of the TIA, the whitening, the whitening compensation, and the TIA compensation -- we conclude that the response of the TIAs has changed, and is no longer as flat as it was the last time we measured it, which was Jul 2023 (LHO:71225).
  
    . 2024-02-26: That same day, we measured the TIA response "alone" using the same remote, DAC driven measurement, but instead turning OFF the analog whitening, and the compensation for both -- as the first attempt to "just get the data we need quickly in order to get the job done." We never processed this data because (a) we didn't have the person power, and (b) we later remember that it's not good enough to use for updating the compensation because this measurement doesn't contain the TIA response "alone," but instead also 
        - several computational delays,
        - the anti-imaging chassis from the DAC,
and it's limited by the 16 kHz sampling frequency of the DAC system so we can't characterize everything we need for the TIA, given that there are 
        - 2x 10kHz poles in the TIA itself, and
        - 1x 44kHz pole in the whitening chassis, even with the z:p = 1:10 Hz whitening filter turned OFF.

    . From 2024-02-26 thru 2024-04-29: I worked offline with Dean to try to understand the electrical grounding paths in the OMC DCPD signal chain, because it's the only think I can imagine that could cause a *common* change in frequency response to both channels. I now have a much better understanding of the electrical grounding in the OMC signal chain, have produced G2400755, and Dean has made an update to the O4 OMC DCPD wiring diagram accordingly D2100716-v3. However, I conclude that any potential change in the in-vacuum electrical grounding during the OMC swap just is not possible, and the DCPDs are tied to earth ground by and at the in-air whitening chassis, so this is likely NOT the source of the frequency dependent change.

    . 2024-04-02: Louis and Francisco finally muster up the person power to go out to the HAM6 ISC-R5 rack to repeat the long, arduous, measurement needed. However, they misinterpret my diagrams from the 2023-03-06 and 2023-03-10 measurements (LHO:67801 and LHO:68167) against the reality of the D2200215-style whitening chassis, and drive 1 [V_pk] source voltage backwards against the monitor path's buffer amplifiers. The first attachment shows
        - page 1: how one should characterize the TIA response, buffer amplifiers, and measurement setup
        - page 2: how one should characterize the measurement setup and buffer amplifiers in order to divide it out of the page 1 measurement, and
        - page 3: how the measurement setup and buffer amplifier characterization was misinterpreted on 2024-04-02.
      Further, as is standard practice, they took SR785 setup notes from "the most recent good measurement available in the SVN," which was the 2023-03-10 data set where Hsiang-yu and I were exploring options. Thus, they used some settings that were meant for testing rather than those which get the best results out of the measurement. More on this later.

    . 2024-04-09: With Louis and Francisco away for the Solar Eclipse, I head out to the floor to take another crack at it. While I get the measurement setup right, I don't know yet, but I still use bad SR785 settings from the 2023-03-10 date. As such, I get results that are too noisy below 50 Hz to make fits. In the second attachment, I show a quad bode-plot of the DCPDA (page 1) and DCPDB (page 2) where I divide 
       - the Last Best Good measurement -- the 2023-03-06 data and
       - the 2024-04-09 data
      against the model fit from that data (see poles and zeros listed in LHO:67809). While one can see hints of the ~0.3% change, it's just too noisy to be confident that we can get a good fit from it.

    . from 2024-04-09 thru 2024-04-23: With review from Louis, we think hard about why the measurement is noisy. I get a hunch about the SR785 parameters' settle time and settle cycles vs. step (or impulse) response of the TIA response itself, insisting that the settle time of 100 [msec] we've been using since 2022 is not long enough for what the impulse response of the TIA circuit is (because we're perpetually trying cut corners and speed up this measurement). I make a bunch of plots that show the step response of the TIA compared with the bad 2023-03-10 settle settings, but while looking back through my notes I also discover that a difference between the 2023-03-06 and 2023-03-10 measurements is that I had decreased the SR785 source voltage from 4 V_pk to 1 V_pk. The third attachment shows plots that show the good vs. bad SR785 settings and why.

       - page 1: this compares the settle parameters, visually showing the crossover frequency where, as the swept sine measurement sweeps down in frequency, the SR785 switches over from using settle time to settle cycles, given that it takes the larger of (settle time) vs. (settle cycles / frequency). The dashed lines are the *bad* too little settle time or cycle parameters, and the solid lines are the good, just long enough, settle parameters. Turns out the cross-over frequency isn't that different, but it's important to understand *where* it is, so that you understand *where* in the frequency response this switch happens so you know which parameter to increase when you see noise in the TIA response at a certain frequency.

       - page 2: Now thinking about the *amplitude* of the excitation vs. the step response of the circuit, I show the magnitude of the modeled response of the TIA with a few points highlighted. Note that, rather than displaying the TIA response magnitude in its "fundamental" units of [Volts output / Amps of current from DCPDs] = [V/A], I display it in the units of this characterization measurement, which is through the 100e3 [V/A] series resistance of the "test" input. Given the intellegent design of the circuit, that means than this measurement of the TIA -- which has transimpedance of 100e3 [V/A] in the gravitational wave band -- has a magnitude of 1 [V/V] in the gravitational wave band. 
            :: the blue "x" and yellow "o" are the cross-over frequencies of the "too little" BAD and the "just enough" GOOD settle params. Good to know that these are *not* in the middle of the 25 Hz resonance feature. But, these cross-over frequency that are the lower bound of a region where we must compare the "desired response" amplitude with the "step response" amplitude after the chosen amount of settle time -- if the "step response" amplitude is still a large fraction of the "desired response" amplitude after (settle cycles/frequency) amount of time, then the data is going to be incoherent (since the circuit is still "ringing" from the incoherent "excitation" that is the step-change in frequency).
            :: the red square, at 102.4 kHz represents the lowest magnitude point in the "settle time" regime. This is where, again we'll need to make sure that amplitude of the "desired response" amplitude at that frequency is larger than the step response amplitude after the total settle time is complete.
            :: the black triangle, at "DC," in this case 0.1 Hz, shows the magnitude of the TIA response at low frequency, at 0.002 [V/V]. This is relevant because the step response will asymptote to this value. Also it's the lowest magnitude point of the whole transfer function, but it still needs to be extremely well resolved -- just as much as the ~1.2 [V/V] region around 25 Hz. So one needs to be conscious of the excitation amplitude (be it 1V, or 4V), filtered through this circuit (thus 0.002 [V_pk] or 0.008 [V_pk]) with respect to the ADC noise of the SR785 at the given input range. Note, the input "range" (which really the amount of attenuation of the input) for the A and B inputs of CH2 has been auto-ranged in the past with 4 [V_pk] source (and locked as static during the measurement once found) to be +16 [dBVpk] == 6.3 [V_pk]. The noise floor at the range setting, at this low of frequency is ~0.1 [mV] (see e.g. figure B3 of Hoffman 2009), so a non-negligible fraction of 0.002 [V_pk] response to 1 [V_pk] source, which is why increasing the source amplitude by a factor of 4 is necessary.

        - Page 3: This shows the full step response of the TIA circuit as a function of time. One can see, as expected, since the maximum gain of the circuit is ~ 1 [V/V], the maximum amplitude of the step response roughly hits the value of the source input within ~100 [usec] of the step. However, we can see that the step response doesn't appreciable attenuate until 0.1 [sec], or 100 [msec]!

        - Page 4: Here, I zoom in on the amplitude and time of the step response, and compare the BAD SR785 settings with the good SR785 settings. Namely, at the two interesting frequency points from page 2 -- the settle time/settle cycle crossover, and the lowest magnitude point in the settle time region. With 1 V input (page 4), at the bad 101 [msec] settle time, and (1/crossover freq) = 101 [msec] crossover frequency cycle duration, the "desired response" amplitude of circuit is comparable to step response at 100 [msec]. This is why the ~25 Hz region of the measurement has been "noisy" all these years we've been measuring the TIA, and why we need to increase the settle time to 250 [msec]. The 102.4 kHz region was still OK, because there's upwards of 10e3 cycles in 100 [msec], oodles of averages to get good SNR. 

        - Page 5: This shows the same plot, but with 4 [V_pk] source voltage step input and settle time of 250 [msec] and 2 settle cycles. Here, we see that it's not an awesome amount of margin between the step response and the desired circuit response at the interesting frequencies, but it's *definitely* better than 100 [msec] and 1 cycle.

     . 2024-04-23: So -- we're good right? We take the hit in patience / IFO time, and increase the settle time from 101 [msec] to 250 [ms], settle cycles from 1 to 2, and increase the excitation amplitude from 1 to 4 [V_pk], and we'll get the bet measurement ever, right?  No. The fourth attachment shows my utter dismay in that -- yes, while the measurement is much less noisy after having improved the SR785 settings -- BUT the magnitude and frequency response of the 2024-04-23 is subtlety different than the 2023-04-09 and 2023-03-06 measurements at the 0.5 [%] / 1 [deg] level. And that one can trace that difference down to that the measurement setup characterization has a non-linear response to excitation voltage.
        - Pages 1 and 3 show the quad bode the same quad bode plot, but now it shows the 2023-03-06, 2024-04-09, and the 2024-04-23 measurement all together. See the ~1% magnitude discrepancy?
        - Pages 2 and 4 show a regular bode plot showing the measurement setup response for all three 2023-03-06, 2024-04-09, and the 2024-04-23 measurements. See the overall magnitude discrepancy, the high-frequency response discrepancy? Maddening.

     As long as we're going insane, Louis and I convince ourselves (via the data sheet) that the AD8672 op amps in this monitor path on the front-interface board, D2200043 of the D2200215 whitening chassis assembly (which are the same used in the main gravitational wave path on the primary functional board, D2200044) that "there's no way this op-amp can drive +/-4V with an input range of +/- 2.5V and an output drive of +/-3.7V!" ... but we were looking at the table for +/-5V operational state, rather than the +/-15 V operational state, which can happily receive +/-12V and drive +/-13V. But that's what drove me to the EE lab...

     . 2024-04-24 and 2024-04-25: Since we have spares of these OMC Whitening Chassis in the EE lab, I repeated this "measurement setup" characterization on both channels of spare chassis serial numbers S2300002 and S2300004. 
         - The fifth attachment shows the results the results of these 2x chassis, each with 2x channels, measurements as a function of input voltage. The spare chassis show no evidence of non-linearity in scale, or frequency-dependent magnitude or phase from 0.5 [V_pk] to 5 [V_pk], to the level of the ratio of the 0.5 [V_pk] to all other amplitude excitations being noisy around the 1.0 +/- 0.0001 (or 0.01 [%] or 1 [HOP]) magnitude and 0 +/- 0.02 [deg] in phase across a 10 Hz to 102.4 kHz span.
         - The sixth attachment compares across all four of these channels at 4.0 [V_pk] source amplitude. All four instantiations of this monitor path in spare chassis show no difference from each other at the same level, using S2300002 CHA as the reference, the ratio of the other channels to it is within 1 +/- 0.0001 in magnitude and 0.1 [deg] in phase.

     . 2024-04-30: To put the nail in the coffin, I go out and measure the S2300003 chassis' monitor path as a function of input voltage. 
         - The seventh attachment shows the damning evidence -- that the frequency response and scale of this monitor path on the S2300003 chassis changes with input voltage, where the other two instantiations do not.
         - The eigth attachment shows that the response to 1 [V_pk] source has changed over time, as I compare the 2023-03-10 measurement setup characterization to the 2024-04-30 results.

And, while I can't definitely say that the incorrect measurement setup from 2024-04-02 borked the op-amps in the monitor path, it's the only smoking gun I have.

"OK, Jeff. You're lost. Why do we care so much about measuring 1.0 from the monitor path of this whitening Chassis?"
I hear you, I feel lost. But, in the world where we're trying to update the TIA compensation at the 0.3% level, and the scale and frequency dependence of the measurement tools you're using are non-linear and frequency dependent at the 0.5 [%] / 1.0 [deg] level in the 1 to 100 kHz region, and you're trying to fit for poles at ~10 kHz, then you're gunna get the wrong answer.

As such, to restate the executive summary we need to replace the S2300003 D2200215 OMC whitening chassis with one of the spares, in order to get a re-characterization of the OMC TIA after the OMC swap.

*sigh*