Rebooted the machine at 18:54 UTC (11:54 PST). All nodes naturally recovered except SUS_IM4 and SUS_ETMX. These nodes did not time out like we have seen in previous reboots, due to the timeout duration being extended (alog44092). Instead, the logs show that the nodes started and then stopped after "terminate called after throwing an instance of 'int'". I'm not sure where this error is coming from yet. A restart of the nodes recovered both.
Screen shots of the status logs and grd logs of the two failed nodes are attached.
Also, zotws20 cannot run many of the guardctrl commands. I ssh'd into zotws6 and "list -s" ran just fine. They are both running Debian 9.5. but the kernal release and version are different:
zotws20 = 4.9.0-7-amd64 | #1 SMP Debian 4.9.110-3+deb9u1 (2018-08-03)
zotws6 = 4.9.0-8-amd64 | #1 SMP Debian 4.9.110-3+deb9u4 (2018-08-21)
Attached a screen shot of the error.
This has nothing to do with the kernel. I'm not sure if it's related, but I note that the site environment variables are not being set appropriately on this machine. In particular the GUARDCTRL_HOST variables is not being set properly to point to "h1guardian". This might be a red herring, but after setting the variable correctly I can see the listing fine:
jameson.rollins@zotws20:~ 0$ GUARDCTRL_HOST=h1guardian guardctrl list
ALIGN_IFO
ALS_COMM
ALS_DIFF
ALS_XARM
ALS_YARM
BRSX_STAT
BRSY_STAT
DIAG_CRIT
DIAG_EXC
DIAG_MAIN
DIAG_SDF
FAST_SHUTTER
HIGH_FREQ_LINES
HPI_BS
HPI_ETMX
....