Logo Cineca Logo SCAI
MARCONI status
GALILEO status
Marconi100 status

You are here

Data Transfer

CINECA supports different protocols, including scp/sftp, rsync, and GridFTP for data transfer:

As a rule of thumb, calling your PC local and an HPC machine somewhere remote, you are strongly encouraged to use GridFTP (in a server-server configuration) for your remote<->remote transfer and rsync (in a client-server configuration) for your local<->remote transfer. The main advantage of rsync with respect to scp is that it can limit itself to the changed/unmoved files so resulting restartable, that is more flexible in scripting. Anyway globus-url-copy is faster and more reliable of both of them.

File-dimension limits

Interactive transfer comparison

Interactive transfers are limited by the Operating System:

$ulimit -t 600 

This limit applies to each transfer which is issued by an unprivileged user, but not to forked processes (such as thirdy party globus-url-copy).

The maximum dimension of a single file transfer depends on several hardware parameters (such as network bandwidth, network TTL and disk access-rate) and software parameters; for a local_memory->remote_disk transfer, using the loopback interface on SP6, some average values follow:

- scp                 : max file dimension = 20GB;

- rsync              : max file dimension = 20GB;

 -globus-url-copy: max file dimension =150 GB.

N.B. The globus-url-copy limits on the file dimension are present only with ssh authentication because the process is issued by the user after authentication. The GSI autentication, wich involves root forking the process, does not present such limits.

Batch transfer comparison

In a client-server configuration (the only one available for scp and rsync) the batch transfers capacity (as well as the intercative ones) is limited by the required Wall Clock Time: the process duration multiplied by the transfer rate gives you the maximum size of the file you can transfer.

For example, in an interative tranfer, which is limited to 10', assuming an average transfer rate of 50MB/s you can tranfer only file smaller than: 600s*50MB/s=30GB   

Comparative table of transfer tools

  local <-> CINECA CINECA <-> CINECA CINECA <-> HPC-machine tape
cp/mv   V    
scp/sftp V   V  
rsync VV VV V  
globus-url-copy V   VVV  
cart(uccia)       VVV

Where the V simbol shows, inside each column, the relative usefulnes of the various tools.