J. Kissel, J. Warner More details to come, but after trending around this morning, Jim found that Ryan's issues with lock losses last night were coincident with H1:ISI-ITMY_ST1 and ST2 capacitive position sensor (CPS) glitching as loud as ~4 micron jumps in 65-100 Hz band limited RMS. Attached are trends of the new-ish band-limited RMS channels installed in April 2023 (see CPS BLRMS ECR installation mentioned in LHO aLOG 68798, and design in SEI aLOGs 1849 and 1867) compared against the ISC_LOCK_STATE_N, where 600 in NOMINAL_LOW_NOISE. The 2nd row of both plots show the ST1 and ST2, H2 and V2 CPS BLRMS trends for both the 65-100 and 130-200 Hz bands. One can see the loudest of glitching in these sensors, where the y-axis is in nano meters. These 4 sensors are all in the corner two "rack," but each sensor has its own readout card. The loud glitches correspond to times of ITMY watchdog trips too, which is what Ryan saw in LHO aLOGs 72671 and 72666. Jim is swapping out some of the cards as I type, and will aLOG accordingly.
For future investigations, I call out these channels for ITMY explicitly:
H1:ISI-ITMY_ST1_CPSINF_H1_BLRMS_65_100
H1:ISI-ITMY_ST1_CPSINF_H2_BLRMS_65_100
H1:ISI-ITMY_ST1_CPSINF_H3_BLRMS_65_100
H1:ISI-ITMY_ST1_CPSINF_V1_BLRMS_65_100
H1:ISI-ITMY_ST1_CPSINF_V2_BLRMS_65_100
H1:ISI-ITMY_ST1_CPSINF_V3_BLRMS_65_100
H1:ISI-ITMY_ST1_CPSINF_H1_BLRMS_130_200
H1:ISI-ITMY_ST1_CPSINF_H2_BLRMS_130_200
H1:ISI-ITMY_ST1_CPSINF_H3_BLRMS_130_200
H1:ISI-ITMY_ST1_CPSINF_V1_BLRMS_130_200
H1:ISI-ITMY_ST1_CPSINF_V2_BLRMS_130_200
H1:ISI-ITMY_ST1_CPSINF_V3_BLRMS_130_200
H1:ISI-ITMY_ST2_CPSINF_H1_BLRMS_65_100
H1:ISI-ITMY_ST2_CPSINF_H2_BLRMS_65_100
H1:ISI-ITMY_ST2_CPSINF_H3_BLRMS_65_100
H1:ISI-ITMY_ST2_CPSINF_V1_BLRMS_65_100
H1:ISI-ITMY_ST2_CPSINF_V2_BLRMS_65_100
H1:ISI-ITMY_ST2_CPSINF_V3_BLRMS_65_100
H1:ISI-ITMY_ST2_CPSINF_H1_BLRMS_130_200
H1:ISI-ITMY_ST2_CPSINF_H2_BLRMS_130_200
H1:ISI-ITMY_ST2_CPSINF_H3_BLRMS_130_200
H1:ISI-ITMY_ST2_CPSINF_V1_BLRMS_130_200
H1:ISI-ITMY_ST2_CPSINF_V2_BLRMS_130_200
H1:ISI-ITMY_ST2_CPSINF_V3_BLRMS_130_200
but the equivalent of these BLRMS channels are now installed on ALL ISIs with CPS have them, so for BSC-ISIs, replace ITMY with ITMX, BS, ETMX, or ETMY.
For HAMs, take out the "_ST#" stage reference, and use "HAM2" through "HAM8".
Templates for the above attached trends live here:
/ligo/home/jeffrey.kissel/Templates/NDScope/
ISI_ITMY_ST1_CPS_65to200Hz_BLRMS.yaml
ISI_ITMY_ST2_CPS_65to200Hz_BLRMS.yaml
So far I have replaced the St1 H2 and St2 H2 sensors. As per usual with these, there's no clear indicator of which sensor is misbehaving. The first sensor I replaced was St1 H2, because it seemed from looking at trends for the last couple days for all corner 2 sensors that St1 H2 glitches were on average louder than the other sensors. It worked for a while, but then St2 tripped a bit after we got to NLN. Looking at trends for the raw CPSINF channels, the St1 H2 sensor didn't even see this glitch and it seemed loudest on St2 H2. I've now replaced the St2 H2 sensor, things seem quiet so far.
I had hoped that the 65-100hz or 130-200hz blrms for the CPS would help find a glitching sensor, but so that doesn't seem to be the case. But, neither has looking at asds of the sensors or trends of the sensor readouts. For now, I don't have any clever ideas to tell which sensor is misbehaving beyond guessing at what sensor to replace and seeing when the ISI stops tripping.