Austin, Dave:
At 16:15PDT h1iopseih45 went into DACKILL mode.
There were no dmesg logs corresponding to this time (see below) and no IOP processing time overrun.
Austin put HAM4,5 suspensions into a safe state and I restarted the models, which has cleared the problem.
Mia Culpa, I mistakenly restarted h1seih16 models first before realizing I was on the wrong front end. Austin is recovering HAM6 supsensions as well, apologies.
h1seih45 recent dmesg logs:
Fri May 26 15:16:58 2023] rts_cpu_isolator: entering ligo_play_dead, cpu handler = 00000000b41de7f7
[Fri May 26 15:16:58 2023] rts_cpu_isolator: calling LIGO code
[Fri May 26 15:16:58 2023] h1isiham5: INFO - CPY2=1 CPY2OFFSET=1500 CPY1SZ=1500 CPY2SZ=536 DBLOFFSET=2036
[Fri May 26 15:16:59 2023] h1isiham5: INFO - Controller initialization complete, starting front end control loop
[Fri May 26 15:17:04 2023] smpboot: CPU 5 didn't die...
[Fri May 26 15:18:11 2023] h1isiham4: ERROR - FE_ERROR_FPU - FPU detected an division by 0 math operation.
[Fri May 26 15:18:16 2023] h1isiham5: ERROR - FE_ERROR_FPU - FPU detected an division by 0 math operation.
[Tue Jun 13 12:30:25 2023] h1isiham4: ERROR - FE_ERROR_FPU - FPU detected an division by 0 math operation.
[Sat Jun 24 16:23:32 2023] h1hpiham4: ERROR - FE_ERROR_FPU - FPU detected an division by 0 math operation.
[Sat Jun 24 16:24:21 2023] h1hpiham5: ERROR - FE_ERROR_FPU - FPU detected an division by 0 math operation.
Sat24Jun2023
LOC TIME HOSTNAME MODEL/REBOOT
16:58:43 h1seih16 h1iopseih16 <<< Mistaken restart of h1seih16 models
17:02:51 h1seih16 h1iopseih16
17:03:05 h1seih16 h1hpiham1
17:03:19 h1seih16 h1hpiham6
17:03:33 h1seih16 h1isiham6
17:05:21 h1seih45 h1iopseih45 <<< Restart of h1seih45 models, no reboot of computer
17:06:17 h1seih45 h1hpiham4
17:06:34 h1seih45 h1hpiham5
17:06:54 h1seih45 h1isiham4
17:07:10 h1seih45 h1isiham5