[Sheila, Keita, Jenne, with input and thoughts from several others]
We have looked at a few things today, trying to understand why the apparent signal level on AS_C changes with the state of the fast shutter. The size of the jump when the shutter is open vs. closed is similar to the expected signal size for true light on the PD. So, with the single-bounce beam, it looks like the amount of light on AS_C doubles when the shutter is opened. There shouldn't be any actual light change on AS_C as a result of the fast shutter state, since AS_C is before the shutter (and in fact used as the trigger for the shutter).
This jump in signal happens with the same amplitude whether or not the IMC is locked. So, opening the shutter makes AS_C look like there is an extra ~5mW on the QPD, even if there is no actual IR light in HAM6. This makes us suspicious of an electronics problem, rather than a mechanical interference with the fast shutter wires, or some optical thing like scattered light. Also, the signal is not affected if we move either SRM, OM1, OM2, or OM3 when there is light in HAM6, which is further evidence that it's not a real beam hitting AS_C.
The attached screenshot shows the jump in AS_C_NSUM as well as AS_C_SEG1 (there is a minus sign between them), when we close the shutter. These are the purple and green traces in the strip tool, and the shutter was activated around -7.5 minutes. At -5.2 minutes, we unplugged the input of the AS_C transimpedance amplifier, and plugged the output of the PD to the OMC QPD tranimpedance amp at -4 minutes. At -2.5 minutes I closed and re-opened the shutter. The scale of the NSUM signals are all the same as each other, and the scale of the SEG1 signals are all the same as each other. The whitening gain and filter settings of the OMC QPDs were set to be the same as the AS_C settings at the -7.5 minute time (no whitening filters, gain of 36 dB). So, if the problem were entirely on the PD, we would expect the same size jump when the PD was plugged in to the OMC QPD chassis. Since we don't see nearly the same size jump, it seems like at least some of the problem is in the transimpedance amplifier. Also, the inmons of OMC_A seemed much less noisy (when plugged in to the AS_C PD) than when the AS_C PD is actually plugged in to its regular chassis.
The fact that we do still see a jump, and in both OMC_QPD_A (which should actually have the AS_C signal) and OMC_QPD_B (which should be nothing) makes it seem like there is still potentially some funny business going on.
With no input to the whitening board, we didn't see the jump behavior. So, it's not by itself doing this jumping, but it isn't fully exonerated from being involved in some ground loop craziness. We did observe that the whitening board seems to be switching filters and gains as expected, although there was a weird time at 16:22:15 UTC where 2 of the 4 AS_C segments were totally weird. But, that was not reproduce-able. Sheila has a DTT template saved with that data, where you can see a big difference even below 5 Hz. But, interestingly, when I go back in time and check the INMONs (which should be the same, just slower data rate as the IN1 channels), I don't see any of that discrepancy. Confusing. And, also not reproduce-able thus far.
A good test will be to unplug the OMC DCPDs from the whitening board (which is on the same chassis as AS_C), since that shared connection ties the AS_C ground to chamber ground, and see if AS_C still jumps in a weird way.
There seems to be some extra noise on the analog AS_C sum channel that is used for the fast shutter trigger, when that signal is connected to the trigger controller. We had it T-ed to a 'scope, and then removed the connection to the trigger controller, and saw that the RMS got smaller.
tl;dr: We're still confused by AS_C. Maybe it's a problem with the transimpedance amplifier? We'll have to modify one of the spares, or see if one of the other Lab locations has a modified one, if we're going to switch it out, since it needs the sum channel to come out in analog for the fast shutter trigger.
About the weird time that Jenne mentioned:
Attached is a dtt template and screenshot, as well as dataviewer traces from the time. This was measured with the IMC unlocked, and no whitening filters on. We attempted to reproduce this several times by changing the whitening settings and setting them back, but weren't able to. When the IMC_LOCK guardian state is 80 it is offline, and when the whitening filter settings are 0 they are off.
About amount of light on AS_C compared to these spurious signals:
This diode is calibrated into W arriving in HAM6, (see 29078 and links in comments, the calibration was turned off in Jenne's screenshot above), between the beam arriving in HAM6 and the QPD there is OM1 with 800ppm transmission and a 48% beam splitter. So 5mW measured on the SUM channel in single bounce means about 2uW on the QPD.
Looking at the screenshot linked above, it looks like the change in the sum channel when the fast shutter closes is about 2.5mW into HAM6, or about 1 uW, the second attached screenshot shows that this is about the same as the change due to the shutter when the mode cleaner is unlocked.
The 1610 counts of dark current that we see with the shutter closed, 36dB of whitening gain and no whitening filters on is equivalent to 26mW into HAM6 or 10uW on the diode.
While fully locked in O2 we had about 0.4W into HAM6, meaning 150 uW on the QPD.
TL, DR: Our dark currents right now are about 7% of the total signal we will get in full lock, about 5 times larger than the signal size in single bounce (using this diode in single bounce is part of our initial alignment procedure), and about 10 times larger than the spurious signal from the fast shutter.
[Fil, Jenne]
After a long list of tests to isolate where the problem was, we have found and fixed the problem with AS_C! Hooray!
It turns out that we have an in-chamber short between pins 3 and 4 of the cabling that goes to the input of the transimpedance amplifier. Recall that this transimpedance amplifier chassis can serve 2 DC QPDs, although we are only using one. Pin 3 of the input from chamber is the "sense" input channel that is used to do some analog noise cancellation. Pin 4 is the anode from one of the segments of the 2nd QPD. So, we were taking any pickup noise from this unused QPD channel and feeding it into the input of the noise cancellation, thus actually making it a noise injector.
Fil has extracted pin 4 from the cable connector that goes into the transimpedance amplifier. After doing so, and plugging everything back in, the large rms noise was gone, the large dark current was gone, and there was no longer a jump when we actuate the shutter - it all looks good.
Fil noted that the whitening chassis that is currently installed is a spare, that does not have the new mods for the OMC DCPD half that prevent saturations when the violin modes are high. So, we should put the original whitening chassis back in place, now that we know it is not the problem. Sheila also noted that the dark offset of AS_C seems surprisingly large if she turns on the 3rd stage of whitening, so maybe there is a large DC offset on the 3rd whitening stage in this spare chassis - hopefully swapping back to the original will solve this also.
When we do in-chamber short checks, we typically only check pins against ground, not pins against other pins. We should add these noise sense pins on the DC QPDs to a list of pins that ought to be checked for shorts against any other pin (and similar, if there are similar noise cancellation circuits in other PD chassis).
Opened (and closed) FRS Ticket 10800 in order to capture this (and all previous aLOG's) history of the issue and relevant drawings. Opened (and marked as whenvent) IIET Ticket 10802 which indicates that the above solution is a temporary fix to the in-vacuum problem that pins are shorted in chamber. Note, we believe the short was created on Apr 26 2018 when searching for *other* "ground loops" (i.e. shorts to ground) (see LHO aLOG 41709).