J. Kissel, for the people actually doing the work: J. Warner, F. Clara We've lost lock because HAM5 ISI capacitive position sensors (CPS) have suddenly started going terribly glitchy and saturating. Jim and Fil are investigating, but this'll likely involve some electronics power cycling, if not swapping. Stay tuned for further details.
Some detail on finding and fixing the problem.
HAM5 suddenly tripped, presumably causing the lockloss, when I went to the overview several of the HAM5 CPSs were saturating nearly constantly, but the GS-13s didn't seem very perturbed, so I didn't believe the CPS were reading real motion. I couldn't reset the watchdog because of this, and when I bypassed the CPS to force a reset, the ISI rang up and tripped immediately.
While Jeff did the paperwork, Fil and I went first to the CER to try power cycling the AA and CPS power. Several rounds of this showed no improvement (several CPS were still glitching over 20k counts), so we then went out to the chamber. There we reseated the power and data cables, disconnected and reconnected the probes and cards in the Corner 1 & 2 satellite rack, but saw no improvement.
We then went to the corner 3 rack, and as soon as Fil grabbed the power cable to try reseating it (first thing he was going to try), the CPS settled down. Doing some "tap" tests on the ground showed believable motion on the live CPS signals in dataviewer (i.e. stomping on the ground showed "real" motion, mostly in Z). We were then able to reset the watchdog and the ISI was able to isolate. We went out again to make sure all of the cables were tight, but while we were there the CPS started glitching again. We then did another round of checking cables on the Corner 3 box and this time when Fil unscrewed the retaining screws on the data cable, the CPSs started behaving again. He carefully reseated the screws and we gently walked away. It's not clear which tweak to what cable fixed the problem, and we didn't find anything obviously wrong before touching stuff. I did find one of the data cables a the CER rack loose, but didn't fix anything by securing it.
This whole process was kind of hampered by not being able to open the sitemap on the mac book we were using. I could get dataviewer, but I could not get the ISI overview to see when we tripped the platform or reset the watchdogs. We were forced to use the workstation in the CER for MEDM.
We'll monitor this for the next couple of days. This does not seem to be exactly the same failure that Hugh found on HAM3 a few weeks ago (in alogs 31564, 32076, 32079). The attached watchdog plot from HAM5 doesn't seem to show the single glithches that Hugh saw on HAM3. Instead the noise on V2, H3 and V3 seem to just suddenly get worse. Kind of worrying.
Second CPS unit (slave) for HAM5 needs to be investigated further. Possible grounding or connector issue, as glitchy/saturation in channels comes and goes with moving of cables for this unit.