The outbuilding weather station IOCs stopped updating again at 04:02 this morning (they stopped at 04:27 Mon morning).
As part of the investigation I found that I am still running a 04:01 cronjob which restarts any stuck weather stations. This should only restart an IOC if its weather station PVs are in an invalid state. The cronjob was installed on 17th Jan 2023 following a spate of lock ups which were occuring between 03:18 - 03:28.
I have disabled this crontab for now.
Instead of disabling the crontab, we have decided to reschedule it to run at 05:01 each morning:
01 05 * * * /ligo/home/controls/restart_weather > /dev/null