At 19:39:56 PDT all the models on h1seiex stopped running after h1iopseiex detected an ADC timeout on the first ADC.

PCI bus scans can see all of the IO Cards.

Uptime of this crash was only 3hr 26mins suggesting a restart at 16:16 this afternoon. Trending the IOP CPU usage also shows an anomaly between 14:15 and 14:29

We will diagnose the issue and restart first thing in the morning.

2026-09-24T19:39:58-07:00 h1seiex.cds.ligo-wa.caltech.edu kernel: h1iopseiex: ERROR - An ADC timeout error has been detected on ADC 0, waiting for an exit signal.
2026-09-24T19:39:58-07:00 h1seiex.cds.ligo-wa.caltech.edu kernel: ligo_roce_mcast: ERROR - ib_poll_cq() - returned 128 results suggesting the buffer is not large enough.
2026-09-24T19:39:58-07:00 h1seiex.cds.ligo-wa.caltech.edu kernel: h1isietmx: ERROR - Waiting for ADC cycle 1242, live read is 62682, mm: 0, ioMemCtr: 1242
2026-09-24T19:39:58-07:00 h1seiex.cds.ligo-wa.caltech.edu kernel: h1hpietmx: ERROR - Waiting for ADC cycle 1242, live read is 62682, mm: 3, ioMemCtr: 1242
2026-09-24T19:39:58-07:00 h1seiex.cds.ligo-wa.caltech.edu kernel: h1isietmx: ERROR - An ADC timeout error has been detected, waiting for an exit signal.
2026-09-24T19:39:58-07:00 h1seiex.cds.ligo-wa.caltech.edu kernel: h1hpietmx: ERROR - An ADC timeout error has been detected, waiting for an exit signal.
2026-09-24T19:39:59-07:00 h1seiex.cds.ligo-wa.caltech.edu rts_awgtpman_exec[11050]: H1:HPI-ETMXIOP cycle timeout
2026-09-24T19:39:59-07:00 h1seiex.cds.ligo-wa.caltech.edu rts_awgtpman_exec[11207]: H1:ISI-ETMX_CDMON_ST2_V2_I_IIOP cycle timeout
2026-09-24T19:39:59-07:00 h1seiex.cds.ligo-wa.caltech.edu rts_awgtpman_exec[10903]: CONIOP cycle timeout
2026-09-24T19:39:59-07:00 h1seiex.cds.ligo-wa.caltech.edu systemd[1]: rts-awgtpman@h1hpietmx.service: Main process exited, code=exited, status=1/FAILURE
2026-09-24T19:39:59-07:00 h1seiex.cds.ligo-wa.caltech.edu systemd[1]: rts-awgtpman@h1hpietmx.service: Failed with result 'exit-code'.
2026-09-24T19:39:59-07:00 h1seiex.cds.ligo-wa.caltech.edu systemd[1]: rts-awgtpman@h1hpietmx.service: Consumed 11.418s CPU time, 42.4M memory peak.
2026-09-24T19:39:59-07:00 h1seiex.cds.ligo-wa.caltech.edu systemd[1]: rts-awgtpman@h1iopseiex.service: Main process exited, code=exited, status=1/FAILURE
2026-09-24T19:39:59-07:00 h1seiex.cds.ligo-wa.caltech.edu systemd[1]: rts-awgtpman@h1iopseiex.service: Failed with result 'exit-code'.
2026-09-24T19:39:59-07:00 h1seiex.cds.ligo-wa.caltech.edu systemd[1]: rts-awgtpman@h1iopseiex.service: Consumed 10.982s CPU time, 76.6M memory peak.
2026-09-24T19:39:59-07:00 h1seiex.cds.ligo-wa.caltech.edu systemd[1]: rts-awgtpman@h1isietmx.service: Main process exited, code=exited, status=1/FAILURE
2026-09-24T19:39:59-07:00 h1seiex.cds.ligo-wa.caltech.edu systemd[1]: rts-awgtpman@h1isietmx.service: Failed with result 'exit-code'.
2026-09-24T19:39:59-07:00 h1seiex.cds.ligo-wa.caltech.edu systemd[1]: rts-awgtpman@h1isietmx.service: Consumed 11.149s CPU time, 44.2M memory peak.
2026-09-24T19:39:59-07:00 h1seiex.cds.ligo-wa.caltech.edu cps_xmit[10899]: Closing client
2026-09-24T19:39:59-07:00 h1seiex.cds.ligo-wa.caltech.edu cps_xmit[10899]: message repeated 2 times: [ Closing client]
2026-09-24T19:39:59-07:00 h1seiex.cds.ligo-wa.caltech.edu cps_xmit[10899]: Closing out OpenMX and exiting
2026-09-24T19:39:59-07:00 h1seiex.cds.ligo-wa.caltech.edu systemd[1]: rts-transport@cps_xmit.service: Main process exited, code=exited, status=1/FAILURE
2026-09-24T19:39:59-07:00 h1seiex.cds.ligo-wa.caltech.edu systemd[1]: rts-transport@cps_xmit.service: Failed with result 'exit-code'.
2026-09-24T19:39:59-07:00 h1seiex.cds.ligo-wa.caltech.edu systemd[1]: rts-transport@cps_xmit.service: Consumed 4min 46.006s CPU time, 28.1M memory peak.
2026-09-24T19:39:59-07:00 h1seiex.cds.ligo-wa.caltech.edu systemd[1]: rts-awgtpman@h1hpietmx.service: Scheduled restart job, restart counter is at 1.
2026-09-24T19:39:59-07:00 h1seiex.cds.ligo-wa.caltech.edu systemd[1]: rts-awgtpman@h1iopseiex.service: Scheduled restart job, restart counter is at 1.
2026-09-24T19:39:59-07:00 h1seiex.cds.ligo-wa.caltech.edu systemd[1]: rts-awgtpman@h1isietmx.service: Scheduled restart job, restart counter is at 1.
2026-09-24T19:39:59-07:00 h1seiex.cds.ligo-wa.caltech.edu systemd[1]: Started rts-awgtpman@h1hpietmx.service - Advanced LIGO RTS awgtpman: h1hpietmx.