Hugh asked why there were 'dot nfs' files in his directory and why he could not remove them, but later when he exited matlab they were removed. Here is a part of his directory listing when the files were there

-rw-rw-r-- 1 hugh.radkins controls 73076 Feb 18  2015 .nfs000000000d840bb80000002f
-rw-rw-r-- 1 hugh.radkins controls 76376 Feb 18  2015 .nfs000000000d840bb70000002e
-rw-rw-r-- 1 hugh.radkins controls  5974 Feb 18  2015 .nfs000000000d840bb60000003e
-rw-rw-r-- 1 hugh.radkins controls  5974 Feb 18  2015 .nfs000000000d840b9a0000002d
-rw-rw-r-- 1 hugh.radkins controls 73076 Feb 18  2015 .nfs000000000d840b2c0000002b
-rw-rw-r-- 1 hugh.radkins controls 73076 Feb 18  2015 .nfs000000000d840b210000002a
-rw-rw-r-- 1 hugh.radkins controls 73076 Feb 18  2015 .nfs000000000d840b1200000029
-rw-rw-r-- 1 hugh.radkins controls 73076 Feb 18  2015 .nfs000000000d84096200000028
-rw-rw-r-- 1 hugh.radkins controls 73076 Feb 18  2015 .nfs000000000d84095700000027
-rw-rw-r-- 1 hugh.radkins controls 73076 Feb 18  2015 .nfs000000000d84094c00000026
-rw-rw-r-- 1 hugh.radkins controls 76376 Feb 18  2015 .nfs000000000d84093400000025

-rw-rw-r-- 1 hugh.radkins controls 89335 Feb  3 16:15 .nfs000000000d840bc600000046
-rw-rw-r-- 1 hugh.radkins controls 89335 Feb  3 16:15 .nfs000000000d840bbe0000003c
-rw-rw-r-- 1 hugh.radkins controls 77540 Feb  3 16:15 .nfs000000000d840b4c00000035
-rw-rw-r-- 1 hugh.radkins controls 74240 Feb  3 16:15 .nfs000000000d840b9900000036
-rw-rw-r-- 1 hugh.radkins controls 74240 Feb  3 16:15 .nfs000000000d840b9c00000037
-rw-rw-r-- 1 hugh.radkins controls 74240 Feb  3 16:15 .nfs000000000d840ba200000038
-rw-rw-r-- 1 hugh.radkins controls 74240 Feb  3 16:15 .nfs000000000d840ba300000039
-rw-rw-r-- 1 hugh.radkins controls 74240 Feb  3 16:15 .nfs000000000d840ba40000003a
-rw-rw-r-- 1 hugh.radkins controls 74240 Feb  3 16:15 .nfs000000000d840ba50000003b
-rw-rw-r-- 1 hugh.radkins controls 77540 Feb  3 16:19 BSC9_H1_Valve_Check.xml
...
 

In Linux is is permitted for a program to open a file and then to unlink it. On a local file system this creates a file with no name (wont show up in 'ls' for example) but the inodes still exist for the file and the program can still read and write to the phantom file. Only when the program stops running does the OS clear the inodes for reuse. This is great for temporary files which should disappear when the program stops, even if it crashes and does not cleanly close the files.

The situation Hugh encountered is when the file is on an NFS mounted file system. In this case when the file is unlinked, the NFS client renames the file to a dot nfs file with a random name. If a user on the same NFS client machine tries to delete the file, they get a "cannot remove, Devicee or resource busy" error and the file is not deleted.

Interestingly we found that on a different NFS client it is possible to delete the file since the controlling process is not on that machine.

So we have potentially two problems. One is that dot nfs files get stuck open (like the ones from last Feb 18th in Hugh's listing, we presume the NFS client was abruptly shutdown). Second is that someone deletes a file on one NFS client which is actually being used by another NSF client.'s program.