Following up from Cheryl's find (alog57124) that the T2 coil test on the MC3 M1 was reading inconsistent. Keita and I ran some different tests to diagnose where the problem actually is.
With the P_DAMP bank off we first checked for a response in the noisemon and fastmon outputs by placing an offset in the TEST banks for T1, T2, T3. Once we verified a response, we moved on to looking at transfer functions similar to the MC1 diagnoses yesterday (we started with the same template but swapped channels for MC3). These were done with the coil driver in state 2.
The results are attached below, and Keita can comment more on the specifics of what was done, but it is looking to be an issue in analog. Fil, Rahul, Sheila, and Keita came up with a game plan for tomorrow to investigate further.
Two things that these measurements show are:
1. BIO screen problem for M3 T2 TEST/AI switching Cheryl noted is just a readback problem.
All of TJ's traces have decent coherence. Apparently the DAC output reaches T2 coil. The relay to connect the DAC output to T2 is working fine. However, readback problem is still a problem and Fil will have a look tomorrow.
2. Driver problem for MC3 T3 coil where the voltage and current are 6dB lower than others.
TJ's 1st plot shows TF from COILOUTF input to FASTIMON for MC3 T3, MC3 T2 and MC1 T2 coils. Clearly MC3 T3 (blue) is 6db lower.
His second plot used all three readbacks (FASTIMON, NOISEMON, VOLTMON), MC3 T3 is again consistently 6db smaller than MC3 T1 and T2. Sheila checked digital gains and filters and found nothing weird.
This sounds like a driver problem to me, maybe one opamp in either negative or positive chain was busted? But this cannot be a digital problem. We're worried if this was triggered by the power supply failure today, and if so, similar problem might be in other coils in HAM2.
FYI the circuit diagram is https://dcc.ligo.org/DocDB/0007/D0902747/005/D0902747_v5.pdf.
Because of these, we asked Rahul to measure top-to-top damping loop TF for MC1, MC3, PRM and PR3 to compare with measurements in the past. Hopefully the problem of MC3 M1 T3 coil shows up as some difference in loop shape (and if it does, we know that the problem developed between now and back then).
Rahul will talk to Fil before Fil goes to LVEA.
Note that the problem 2 cannot be solved by swapping the driver for problem 1, as MC3 T3 coil is on the last of the three driver chassis while MC3 T2 is on another one. (https://dcc.ligo.org/DocDB/0007/D0902810/008/SUS-C4.pdf and https://dcc.ligo.org/D1000581).
Also note that this could be DAC/AA problem in that the positive or negative path for T3 coil path is busted upstream of the driver.
If there's no difference between the damping loop shape now VS back then, probably the problem was there for a long time. However, if that's the case, we need to make REALLY sure that there's a driving problem. To do this,
- Turn off damping as well as alignment offset of MC3 and MC1.
- Put the driver to state 2.
- Somebody goes to the floor with at least 3 DB9 and 3 DB25 breakout boards. (It might be a good idea to bring out DB37 and DB15 too just in case.)
- Connect DB9 breakout board to the "Drive Input" cable on the driver front panel. There are three relevant drivers.
- Enable 5000 counts offset for all of individual coils of MC1 and MC3 top coils e.g. H1:SUS-MC3_M1_COILOUTF_T1_OFFSET.
- Using a multimeter, record the voltage between pin 5 (ground) and other pins.
- If all pins for all three cables show the same absolute value, DAC side is fine. At that point we can tell that the problem is in the driver. We'll confirm what we already know by measuring voltmon and imon in analog. Circuit diagram: https://dcc.ligo.org/DocDB/0158/D1900052/002/noisemon.pdf
- Disable offsets.
- Connect DB25 breakout boards to the "V, I and RMS monitor" cable on the rear panel.
- Enable offsets.
- Read the voltage between pin 25 (GND) and four voltmon pins, i.e. pin 3, 6, 9, 12.
- We hope that the problem is only in MC2 T3, but we don't know for sure. Anyway, if that's the case the absolute value for voltmon pins are all about the same except pin 3 for the driver that is used for MC3 T3, LF, RT and SD.
- We'll need to swap the driver, but not before discussing with SUS experts because we may need to change the damping loop.
- If DAC side is found to be broken, apparently we won't swap the driver.
The Top to Top TF measurements results (see Keita's alog 57145 above) are posted in the following alog 57155 .
Corresponding FRS ticket 16116 for the issue: "MC3 top T3 coil output is about 6dB weaker than others".