I'm putting a summary of the things we tried with BRSX today here. It might be fixed, but we need to monitor it overnight.
First thing, Patrick and I remotely restarted the BRSX beckhoff computer. This didn't resolve the glitching.
Next, Hugh, Fil and I went to the end station. We checked nothing was hitting the box, the inside looked undisturbed. Fil and Hugh power cycled the 24V power to the beckhoff (not sure what all that involved) and I restarted the beckhoff computer again. Still glitching.
Later, I found that there were fluctuations on the reference spots on the CCD, as Jeff K put in the DCC earlier.
Hugh and I went to EX again, this time we pulled the power on the light source, then plugged it back in. While we were there, we also opened the BRS enclosure again and pulled the CCD of the top of the light pipe. After looking down into the light pipe, we noticed two pretty good sized, dead, bugs sitting on the viewport at the bottom of the light pipe. I couldn't get my phone to focus on them through the beamsplitter. Since we couldn't reach them and didn't want to take the light pipe off, we put the CCD back on.
The BRS is more or less back in it's nominal state and I've been watching it for the last hour. It hasn't glitched since being restored after this last incursion, but it's still pretty rung up. Looking at trends from earlier, it seems possible that the glitches don't appear when the BRS is moving above some threshold. Could also be that power cycling the light source fixed it.
We should still monitor the BRS for at least overnight, before trying to run with it again.
C. Hagedorn, M. Ross
After watching the video Jeff posted (G1901492), scanning through some of the glitches, and the fact that dead bugs were found, we believe that a live bug on the lens or vacuum window is the most likely culprit for the glitches. This will require intervention before this BRS can be trusted again.
It is quite unlikely that it is a light-source problem. Both patterns are illuminated from the same source and we were able to find transients that affected only one of the two patterns. Additionally the video shows asymmetric transient intensity variations which is very unlikely to come from the light source.
We believe that there's still a live insect inside the autocollimator in addition to the dead insect. Dead insects won't give frequent transients in both patterns. The transients in the Ref and Main traces and the distortions in the video are all consistent with a bug on the condensing lens, beamsplitter, objective lens, or window. This causes smaller and wider intensity dips than the bug that previously caused problems (40396) which was directly on the CCD.
With some help from TJ, I've added a simple test to DIAG_MAIN to help diagnose this in the future. It just looks at the 300mhz - 1 hz blrms for the BRS and if that channel goes above 20, DIAG_MAIN will report the BRS as noisy, see the last two lines here:
@SYSDIAG.register_test
def BRS_CHECK():
"""Check the two end station Beam Roation Sensors to make sure that
they are not in FAULT (as read from their Guardian status node state.
Also checks the status of the CBIT, to see if C code is live
"""
for end in ['X', 'Y']:
if ezca['ISI-GND_BRS_ETM{}_CBIT'.format(end)] == 0:
yield 'BRS {} C code has stopped'.format(end)
elif ezca['GRD-BRS{}_STAT_STATE'.format(end)] == 'FAULT':
yield 'BRS {} is in FAULT'.format(end)
if ezca['ISI-GND_BRS_ETM{}_R{}_BLRMS_300M_1'.format(end, 'X' if end == 'Y' else 'Y')] > 20:
yield 'BRS{} is noisy'.format(end)
On the attached trend, neither BRS goes above this value, or even close, in the last 30 days except BRSXwhen it went "buggy".
