I restarted the DAQ at 12:05 PDT. About 90 minutes later h1fw0 crashed and was auto-restarted by monit. This was followed quickly with another restart. At this point I started monitoring it carefully and noticed it was very slow in writing its frame files, to the extent that it was sometimes over 64 seconds behind. Most of the time it managed to almost catch up.

I verified that h1fw0's E18 RAID was not reporting any errors. On h1fw0 I noticed that looking at the /trend mount froze the terminal. So as a first try, I rebooted h1fw0 (leaving h1ldasgw0 running). It came back up correctly and the problem appears to have been resolved. The /trend issue was an old mount in /etc/fstab trying to mount h1tw0, a machine which was turned off several weeks ago. I have corrected the fstab file, but for the current run it was still in the file and /trend is resolving as an empty mount point (as it should be) and not freezing the terminal.

Attached plot show 30 mins of second trend, showing the full file size for h1fw0 (red), h1fw1 (green) and h1fw2 (blue). fw2 and fw3 are in lock step, fw0 is one cycle behind initially and catches up later.