Reports until 09:43, Sunday 04 June 2023
H1 CDS
david.barker@LIGO.ORG - posted 09:43, Sunday 04 June 2023 - last comment - 14:17, Monday 05 June 2023(70117)
ISI Ham7 ADC signals for the second ADC, channels 8-19 are not reading the correct signal

Austin, Jim, Dave:

Starting at 06:56 PDT the h1seih7 ADC1 (2nd) channels 08-19 started misbehaving. All the other channels on this ADC are unaffected.

Austin confirmed that the AA chassis for this ADC looks correct on the mezzanine.

Jim confirmed that this block of channels are associated with a single ISI interface chassis which feeds the AA.

The six broken signals are:

H1:ISI-HAM7_L4CINF_[V2,H2]_MON

H1:ISI-HAM7_GS13INF_[V2,H2]_MON

H1:ISI-HAM7_CPSINF_[V3,H3]_MON

Comments related to this report
jim.warner@LIGO.ORG - 09:52, Sunday 04 June 2023 (70120)

The asds of the CPS, GS13 and L4C in this chassis all look identically bad, the other sensors seem fine. All ISI controls are off, loops can't be re-engaged at this point and we probably can't get the squeezer back on. I don't think we can get to Observe without fixing this.

 

Images attached to this comment
david.barker@LIGO.ORG - 09:58, Sunday 04 June 2023 (70121)

Attached ADC Monitor MEDMS for h1iopseih7 and h1isiham7 highlight the broken channels. The signals have two modes, the are either somewhat static with incorrect low values, or they are in a flashing mode, where they transition between the low value to a very high value (tens of thousands) with a flash frequency of about 1Hz, they then go back to the quiet mode for 10-20 seconds.

For about 30 minutes after the 06:58 failure these signals were very noisy all the time, and then they settled to the static/flash-modes we are seeing now.

Images attached to this comment
jim.warner@LIGO.ORG - 11:59, Sunday 04 June 2023 (70122)SEI

Marc, Austin and I pulled a spare ISI interface chassis from the staging building test stand and replace the corner 2 unit.  So far signals all look normal now and the ISI was able to re-isolate. Marc opened the chassis and it looks like a cap or something failed on the power board.

We also noticed at the mezzanine racks that all of the chassises there are quite hot, I think Marc said he saw 110 F on some part of a chassis. It's warm out there, maybe we should space those chassises out a bit on the rack. 

We replaced S20000568 with S1201328. This spare interface is an ALIGO spare and the last one we have.

IFO stayed  locked this whole time, but squeezer was down. Vicki has now relocked and we are back observing.

marc.pirello@LIGO.ORG - 12:03, Sunday 04 June 2023 (70123)

C5 blown on power regulator.

Images attached to this comment
jim.warner@LIGO.ORG - 12:10, Sunday 04 June 2023 (70125)

Asds of sensors after chassis swap.

Images attached to this comment
michael.landry@LIGO.ORG - 15:47, Sunday 04 June 2023 (70126)

Nice recovery, well done all.

jeffrey.kissel@LIGO.ORG - 14:17, Monday 05 June 2023 (70147)CDS, FRS, OpsInfo, SEI
Tagging SEI, OpsInfo, and FRS.

Opened and closed FRS Ticket 28196.

Observation time lost is ~5 hours, see attached screenshot.
Images attached to this comment