jameson.rollins@LIGO.ORG - posted 14:12, Friday 21 September 2018 (44092)
increased guardian systemd watchdog timeouts
I have adjusted the two systemd watchdog timeouts for the guardian process supervision on h1guardian1.
The "TimeoutStartSec" and "WatchdogSec" settings were increased in the hope that we can avoid the timeout issues we've been seeing on boot and reload of select systems (usually SEI). The new settings are:
- TimeoutStartSec=4min (increased from 3min)
- WatchdogSec=30s (increased from 20s)
The TimeoutStartSec adjustment will be tested at next h1guardian1 reboot (likely next Tuesday). Hopefully this will prevent nodes from being killed during the startup process. The WatchdogSec setting can be tested the next time anyone has an opportunity to reload any of the nodes that were being killed on reload (ISI_* I think).
Full settings:
guardian@h1guardian1:~ 0$ systemctl --user cat guardian@SUS_ETMX.service
# /usr/lib/systemd/user/guardian@.service
[Unit]
Description=Advanced LIGO Guardian service: %i
After=network-online.target
[Unit]
Description=Advanced LIGO Guardian service: %i
After=network-online.target
[Service]
Type=notify
WatchdogSec=3
Slice=guardian.slice
EnvironmentFile=/etc/guardian/local-env
Environment=GUARD_LOG_NOTIME=true
ExecStartPre=/usr/bin/guardian -p %i
ExecStart=/usr/bin/guardian %i
Type=notify
WatchdogSec=3
Slice=guardian.slice
EnvironmentFile=/etc/guardian/local-env
Environment=GUARD_LOG_NOTIME=true
ExecStartPre=/usr/bin/guardian -p %i
ExecStart=/usr/bin/guardian %i
[Install]
WantedBy=guardian.target
WantedBy=guardian.target
# /home/guardian/.config/systemd/user/guardian@.service.d/timeout.conf
[Service]
TimeoutStartSec=4min
WatchdogSec=30s
[Service]
TimeoutStartSec=4min
WatchdogSec=30s