At approximately 23:55 Friday 15 August 2025 PDT all models on h1susex failed with ADC timeout.

lspci showed that Adnaco2 slot4 looked not quite correct. This is the 3rd 18bit-DAC which is only used by the h1sustmsx model and was replaced recently during lock-loss investigations.

I power cycled h1susex following the procedure:

fence from Dolphin, stop all models, power down via terminal, wait, power up via IPMI.

Now lspci is not reporting any card in A2-S4, indicating this 18bit DAC has failed.