Recovery of the front end systems was hampered by a slow /opt/rtcds file system, which still persists. There are no logs on h1fs0 indicating the problem, and the ZFS file system itself appears to be fully functional. Hourly backups by h1fs1 continue normally. At this point we cannot rule out a network issue.

The alarms system is running but does not seem to be operational, again no errors are being logged and channel access appears to be working.

Jim reported a bad model initialization with its safe.snap, most probably caused by a slow /opt/rtcds. I had to restart many models several times to get through a "burt restore" timeout for similar reasons.

We may need to restart h1fs0 in the morning and remount /opt/rtcds on all the NFS clients