Reports until 13:32, Wednesday 10 May 2023
H1 CDS
david.barker@LIGO.ORG - posted 13:32, Wednesday 10 May 2023 - last comment - 13:57, Wednesday 10 May 2023(69479)
h1seih45 crash with ADC error

EJ, Erik, TJ, Dave, Fil

At 12:38 PDT all models on h1seih45 stopped running.

There was an ADC issue, so both the front-end computer and its IO Chassis were power cycled.

When the system came back up, the 3rd ADC was showing it had failed its auto-cal, we decided to replace this ADC.

Comments related to this report
ezekiel.dohmen@LIGO.ORG - 13:45, Wednesday 10 May 2023 (69480)
Link 

Original crash was from models timing out on an ADC read. 

First solution attempt was to try and restart the models, AUTOCAL passed on the first two ADCs, but I observed a null pointer dereference in the real time model for the third. 'lspci' command showed that one of the ADCs was missing, probably causing the model restart failure. 

Front-end/I/O chassis was power cycled, and models were able to start. AUTOCAL was bad for the third ADC, and with the evidence from above suggest replacing the third ADC.
david.barker@LIGO.ORG - 13:55, Wednesday 10 May 2023 (69481)
Link

david.barker@LIGO.ORG - 13:57, Wednesday 10 May 2023 (69483)
Link

Fil replaced the 3rd ADC, which EJ confirmed was the one missing from the PCI bus following the crash.

 

old card (removed) new card (installed)
110204-08 110506-47

 

david.barker@LIGO.ORG - 13:57, Wednesday 10 May 2023 (69484)
Link

New card has no autocal issues. Closing ticket as fixed by hardware replacement.