datatransferchoose

Choosing the Right Data Transfer Method

Although each data transfer may be unique, following a few rules-of-thumb will greatly increase efficiency of transfer:

• GridFTP page for syntax on using GridFTP Clients,
• Where to Initiate File Transfers for information on GridFTP servers,
• Data Storage for information on choosing fast file systems, and the
• Examples parameter optimization syntax page.

There are three GridFTP clients available on XSEDE for data transfer, uberftp, tgcp, and globus-url-copy. Both uberftp and tgcp utilize and improve on globus-url-copy. Each offers different features; in general, tgcp is recommended for batch jobs and uberftp is an interactive client; that is, it provides step-by-step instructions for the transfer. Please view the table below for a comparison, then click on the name of the client for a detailed description and examples of how to use the client.

GridFTP Client Recommendations

table needed from https://www.teragrid.org/web/user-support/transfer_method

The following guidelines are recommended for each client:

1. Use a gridFTP client: tgcp, globus-url-copy, or uberFTP.
2. Utilize dedicated gridFTP servers wherever possible using a third-party transfer.
3. Use optimization parameters: tcp buffer size=8 MB and parallel=1.
4. Plan ahead and send results of your jobs to fast file systems, where available.
5. If you are transferring multiple small files, create a tarball.

   While some basic file copy commands may be used for very small files, in general they are markedly less efficient than utilizing gridFTP.

   Because XSEDE is constantly evolving, users should check back for changes in data transfer recommendations. For individualized help with file transfers, please contact the XSEDE Help Desk.