Ryan, Carlos, Jim, Dave:

The new ZFS file system on h1fs0 which is NFS exported as /opt/rtcds became 97% full recently. This is a fast (SSD) but relatively small (1TB) file system.  The reason for being fast is to prevent front end epics freez-ups which we saw last year when the disk was busy syncing to its backup machine.

With this being a snapshotted ZFS file system, freeing up space is not as easy as removing and/or compressing files. First all the snapshots must be destroyed to free up the file blocks and allow file modifications.

Process on both h1fs0 (primary) and h1fs1 (backup) was:

 1 destroy all snapshots

 2 remove all old adl files in the rcg generated dirs /opt/rtcds/lho/h1/medm/model-name/

 3 tar and compress old build areas in /opt/rtcds/lho/h1/rtbuild*

 4 delete all non-rcg generated target_archive dirs (mostly from 2011-2012)

 5 delete all rcg generated target_archive dirs with the exception of the most recent four dirs

2&3  freed up only a modest amount of disk space, they were mainly to clean up clogged directories.

4&5: prior to doing this we verified these old target archives were backed up to tape. Stop 5 freed the lion's share of disk space, about 600GB.

The end result is the amount of free space went from 53GB to 685GB, disk usage went down from 97% to 28%.

At the end of the procedure, a full ZFS-snapshot sync of h1fs0 to h1fs1 was done manually, and the hourly sync was turned back on.