The h1guardian1 machine had been up for 125 days (May9th), so we tried a reboot today during maintenance. The few issues that came up:
- Most of the ISI, HPI, and a few SUS nodes would not start up due to systemd watchdog timeouts. This is a known issue and will be addressed soon. Doing a restart one or two nodes at a time fixed it.
- SEI_HAM2 would not come up due to a lock file in the .git directory that node's archive. This lock file has to be removed by the guardian user or a sysadmin. This is also a known issue, will be resolved soon.