david.barker@LIGO.ORG - posted 09:43, Sunday 04 June 2023 - last comment - 14:17, Monday 05 June 2023(70117)
ISI Ham7 ADC signals for the second ADC, channels 8-19 are not reading the correct signal
Austin, Jim, Dave:
Starting at 06:56 PDT the h1seih7 ADC1 (2nd) channels 08-19 started misbehaving. All the other channels on this ADC are unaffected.
Austin confirmed that the AA chassis for this ADC looks correct on the mezzanine.
Jim confirmed that this block of channels are associated with a single ISI interface chassis which feeds the AA.
The six broken signals are:
H1:ISI-HAM7_L4CINF_[V2,H2]_MON
H1:ISI-HAM7_GS13INF_[V2,H2]_MON
H1:ISI-HAM7_CPSINF_[V3,H3]_MON
The asds of the CPS, GS13 and L4C in this chassis all look identically bad, the other sensors seem fine. All ISI controls are off, loops can't be re-engaged at this point and we probably can't get the squeezer back on. I don't think we can get to Observe without fixing this.
Attached ADC Monitor MEDMS for h1iopseih7 and h1isiham7 highlight the broken channels. The signals have two modes, the are either somewhat static with incorrect low values, or they are in a flashing mode, where they transition between the low value to a very high value (tens of thousands) with a flash frequency of about 1Hz, they then go back to the quiet mode for 10-20 seconds.
For about 30 minutes after the 06:58 failure these signals were very noisy all the time, and then they settled to the static/flash-modes we are seeing now.
Marc, Austin and I pulled a spare ISI interface chassis from the staging building test stand and replace the corner 2 unit. So far signals all look normal now and the ISI was able to re-isolate. Marc opened the chassis and it looks like a cap or something failed on the power board.
We also noticed at the mezzanine racks that all of the chassises there are quite hot, I think Marc said he saw 110 F on some part of a chassis. It's warm out there, maybe we should space those chassises out a bit on the rack.
We replaced S20000568 with S1201328. This spare interface is an ALIGO spare and the last one we have.
IFO stayed locked this whole time, but squeezer was down. Vicki has now relocked and we are back observing.
C5 blown on power regulator.
Asds of sensors after chassis swap.
Nice recovery, well done all.