Gracedb had some issues last night, details in Keith's alog Link
Our ext_alert program on h1fescript0 had given up attempting to reconnect due to the long duration of the server outage. This morning I tried restarting ext_alert via a monit restart, but this did not work and I ended up starting it by hand. It should be stable now.
Are operators supposed to restart this? I did not receive an alarm last night or tonight (only way I knew of a "GraceDB quiery failure" was a red box appearing on the Ops Overview.
There used to be instructions to re-starting this on a wiki, but those instructions have been removed from this page:
https://lhocds.ligo-wa.caltech.edu/wiki/ExternalAlertNotification
So not sure if I'm supposed to use the old instructions to start this or have someone else restart.