Attached plot shows CDS overview after the crash (13:56 PDT). Things to note: h1susauxex has no problems (this is the only machine without a Dolphin card at EX), most user models stopped running except for h1susetmxpi, h1isietmx, IOP models are running but with DAC/DK errors.
At EX I noted:
No activity in the building around the time
No timing issues on any IO Chassis
Computers have been running for close to 8 days (since 16:00 Tue 18th Sept PDT)
All models not running reporting ADC timeouts in dmesg. Details below
computer
model
adc timeout parameters
h1seiex
h1hpietmx
ADC TIMEOUT 3 40582 5 40646
h1susex
h1susetmx
ADC TIMEOUT 0 39380 20 39444
ditto
h1sustmsx
ditto
h1iscex
h1iscex
ADC TIMEOUT 1 39571 19 39635
ditto
h1alsex
ditto
ditto
h1calex
ADC TIMEOUT 0 39571 19 39635
ditto
h1pemex
ditto
I checked all front end computers could see their IO Chassis. I then restarted all the models, no reboots were necessary.
Images attached to this comment
david.barker@LIGO.ORG - 14:19, Wednesday 26 September 2018 (44188)
Attached plot shows CDS overview after the crash (13:56 PDT). Things to note: h1susauxex has no problems (this is the only machine without a Dolphin card at EX), most user models stopped running except for h1susetmxpi, h1isietmx, IOP models are running but with DAC/DK errors.
At EX I noted:
No activity in the building around the time
No timing issues on any IO Chassis
Computers have been running for close to 8 days (since 16:00 Tue 18th Sept PDT)
All models not running reporting ADC timeouts in dmesg. Details below
I checked all front end computers could see their IO Chassis. I then restarted all the models, no reboots were necessary.
I've opened (and immediately closed) an FRS
https://services.ligo-la.caltech.edu/FRS/show_bug.cgi?id=11542