Jonathan, Dave:
The alarms service on cdslogin stopped reporting around 1am this morning. Symptoms are status file was not being updated (caused alarm block on CDS Overview MEDM to turn PURPLE) and the report file was not being updated. Presumably no alarms would have been sent from this time onwards.
At 08:10 I restarted the alarms.service on cdslogin. A new report file was created but not written to, the /tmp/alarm_status.txt file was not changed (still frozen at 01:00) but I did get a startup text. Then 14 minutes later the files started being written. I raised a test alarm and got a text, but no email.
At 09:38 after not getting a keepalive email at 09:00 or any SSH login emails I rebooted cdslogin. Same behavior as 08:10; report file created not written, tmp file not created, startup text sent successfully. After 14 minutes alarms starts running, writes to file system, test alarms are texted but no emails at all.
Jonathan is going to check on bepex.
Jonathan rebooted bepex which has fixed the no-email problem with alarms and alerts. I raised a test alarm and alert to myself and got both texts and emails.
FRS32853
Alarms got stuck again around noon today, presumably due to a reoccurring bepex issue. I have edited the code to skip trying to use bepex and only use twilio for texts. alarms.service was restarted on cdslogin at 16:48 PST.