The following tools are used to transfer data between /home, /scratch, /fastscratch and the archive storage device.
The archive storage device is based on DMF (Data Migration Facility). Data you copy to the archive device will be written to tape as if the tape was a regular disk. However, everytime you archive a new version of a file, which you had already archived, a new copy is made on tape without deleting the earlier version. But if you delete a file in the archive then the file will be lost for good.
The data transfer tools work on the login nodes and interact with the archive storage device using the "rsync" command. So if you archive an entire directory twice only the files which have been modified will be updated.
The tools can only be run interactively (no background process), each user can only have one archiving command active at a time, and a single user at a time can retrieve files from the archive (you will see a message appear while you wait for your turn). No directory can be archived if it contains more than 500 files - you should "tar" such directories first. Finally, the path to the archive must be absolute - eg /archive/myid/path.
Here are the commands to use the archive storage device:
"dmfls" : list the contents of the archive.
Options : same as "ls".
"dmfdu" : show the storage space consumed.
Options : "-h" for human readable output, "-c" for total usage.
"dmfrm" : remove files / directories from the archive.
Options : "-r" to remove directories recursively.
"dmfput": copy data to archive.
Options : none.
"dmfget": retrieve data from archive.
Options : none.
`dmfls -l /archive/zidane`
total 16
drwxrwxr-x 112 apps apps 4096 2015-04-30 08:25 (REG) builds
-rw-rw-r-- 1 apps apps 693985 2015-06-22 12:49 (OFL) eigen-intel14.tar.gz
-rw-rw-r-- 1 apps apps 10546302 2015-06-22 12:46 (OFL) eigensoft-intel14.tar.gz
-rw-rw-r-- 1 apps apps 2438932 2015-06-22 12:40 (OFL) fftw2-intel14.tar.gz
-rw-rw-r-- 1 apps apps 7921210 2015-06-22 12:40 (OFL) fftw3-intel14.tar.gz
-rw-rw-r-- 1 apps apps 2730147 2015-06-22 12:43 (OFL) gsl-intel14.tar.gz
-rw-rw-r-- 1 apps apps 25386040 2015-06-22 12:42 (OFL) hdf5-intel14.tar.gz
drwxrwxr-x 3 apps apps 26 2015-02-08 08:39 (REG) jags
`dmfdu -ch /archive/ronaldo`
27G /archive/ronaldinho/WIEN2K
27G /archive/donaldinho
27G total
`dmfrm /archive/beckham/file1`
`dmfrm -f /archive/beckham/dir1`
`dmfput /scratch/neymar/test-sep2017/dir1 /archive/neymar/test-sep2017/dir1`
`dmfget /archive/messi/version3.2/sept16/20Km/grid1/indianocean.nc /scratch/messi`
NYUAD HPC Apps Team:
--------------------
- Benoit Marchand
- Guowei He
- Jorge Naranjo