This morning's striptool flatline and computer slow down reports further strengthens the evidence that this problem is related to h1boot's NFS disk activity. Yesterday I stopped all running rsync backups of /opt/rtcds and started a single backup at 16:30 PST Friday. It took about 10 hours to complete, and finished around 02:00 this morning. This is around the time the EPICS data froze and Nutsinee reported workstation slow downs. I am monitoring EPICS freezes by looking at the Dolphin manager logs on h1boot, they shows freeze events around 16:30 yesterday and 02:00 this morning and none inbetween.

This rsync used to take only 20 minutes, I'll look into why it is now taking longer (a major file cleanup could be in order).

No linux workstation is reporting loss of NFS connection to this server at these times, looks like a general slow down which impacts diskless frontend computers more.

Investigation continues.

Long term fix is to install a new NFS server for /opt/rtcds post O1.

Here are the dolphin logs for this period of time (reporting when nodes come back)

Dec 18 2015 16:34:15 Fabric 0 status: All nodes are ok!

Dec 18 2015 16:34:28 Fabric 0 status: All nodes are ok!

Dec 18 2015 16:34:29 Fabric 0 status: All nodes are ok!

Dec 19 2015 01:51:09 Fabric 0 status: All nodes are ok!

Dec 19 2015 01:51:45 Fabric 0 status: All nodes are ok!

Dec 19 2015 01:51:46 Fabric 0 status: All nodes are ok!