all the models on h1oaf0 have stopped running. At time of writing it looks like a possible timing glitch caused by in-rack work in the CER. We will hold off on restarts until the rack wiring work is completed because there are ongoing alignment measurements.
Soon after we lost h1seih16 models
Initially I thought these were just timing glitches, but logging onto the front ends and scanning the IO Chassis cards using lspci shows that many cards are missing. This suggests +24V DC power issues.
For example, h1seih16 is showing the first Adnaco backplane (timing_card, ADC0, DAC0) but is missing the second Adnaco (ADC1, ADC2, ADC3, DAC1). The third is empty, the fourth has the Contec card and is seen.
Plan is to let Marc finish the accelerometer work he was doing in the CER (WP11653: replacing accelerometer power conditioners) at the time of these model crashes before starting any model-recovery.