« Back

datatransferexamples

Data Transfer Examples

This page provides examples of several GridFTP transfer operations using different clients. See Where to Initiate Large File Transfers for an overview of GridFTP on XSEDE. See the GridFTP Deployment table to find connection info for all XSEDE sites. You can also find RFT Deployment info on the same page.

Third Party Transfers

Here is an example showing use of third-party file transfer method.

Need graphic from https://www.teragrid.org/web/user-support/transfer_examples

  • Transfer file from current directory to home directory at another site using default values: tg-login1> globus-url-copy file:'pwd'/ gsiftp://tg-login..teragrid.org/~/>filename<
  • Copy a file to Mass Storage at NCSA; disable data channel authorization, use 3rd party transfer, and use optimized transfer parameters: globus-url-copy -nodcau -tcp-bs 8388608 gsiftp://tg-gridftp..teragrid.org/~/MyBigFile gsiftp://mss.ncsa.teragrid.org/~/MyBigFile

General Performance Recommendations

  • 8MB is an optimum TCP buffer size
  • reasonable settings for TG cross-site transfers are:
    • -p=1 and -tcp_bs=8388608 (most transfers)
    • -p=2 and -tcp_bs=4194304 (high network traffic or files > x GB)

Using uberftp to transfer files

Example: Copy 3 files (each named "OneMBfile-Ex") from Mass Storage System at NCSA to UC/ANL via third party transfer using gridFTP servers and optimized transfer parameters. (Note that mss.ncsa.teragrid.org is a gridFTP server.)

  1. Start uberftp and set transfer parameters: tg-login1> uberftp uberftp> parallel 1 uberftp> tcpbuf 8388608
  2. Open connection to Mass Storage System: uberftp> open mss.ncsa.teragrid.org
  3. Establish "local" connection to UC: uberftp> lopen tg-gridftp.uc.teragrid.org 220 tg-grid1.uc.teragrid.org GridFTP Server … ready. 230 User xxx logged in.
  4. Get multiple files from MSS to the "local" (UC) (UC) site: uberftp> mget OneMBfile* dst> 500 ‘SBUF 8388608': command not understood dst> 500 `WIND 8388608': command not understood src> 150 Opening BINARY connection(s) for OneMBfile-Ex1 (1048576 bytes). dst> 150 Opening BINARY mode data connection. src> 226 Transfer complete. dst> 226 Transfer complete. . . . src> 150 Opening BINARY connection(s) for OneMBfile-Ex2 (1048576 bytes). . . . src> 150 Opening BINARY connection(s) for OneMBfile-Ex3 (1048576 bytes). dst> 150 Opening BINARY mode data connection. src> 226 Transfer complete. dst> 226 Transfer complete. UberFTP documentation (NCSA) UberFTP documentation 2 (NCSA)

Using tgcp to transfer files

The following examples show four methods of using tgcp to transfer files. Please be advised that tgcp, although currently a required component of the CTSS Application and Development Runtime Kit, is deprecated, and therefore not guaranteed to work correctly on all XSEDE resources. It will soon become an optional component and disappear from some resources. The recommended GridFTP copy clients to use for most resources are UberFTP and globus-url-copy.

Transfer of files larger than 100 MB using the "-big" option

The following example describes a transfer of the file "bigfile" from the local site to the site specified in the command (in this case, SDSC). The option "-big" designates a striped transfer and should always be used to transfer files larger than 100 MB. tgcp -big FILENAME tg-login.sdsc.teragrid.org:/users/USERNAME/FILENAME.dest

Using Globus RFT (Reliable File Transfer)

RFT is a service that performs third party transfers on behalf of the requester. The benefits to using RFT are performing third party gridFTP transfers for the user on a separate server, transferring entire directories, performing multiple transfers in parallel, and automatically retrying individual transfers if they fail. This last feature ensures that the TGCP operation completes regardless of whether the user waits for it to complete or not. tgcp -rft FILENAME /disks/scratchgpfs1/USERNAME/FILENAME.onscratch

Transfer of a small file

The following example describes a transfer of the file "smallfile" from the local site to the site specified in the command (in this case, SDSC): tgcp FILENAME tg-login.sdsc.teragrid.org:/users/USERNAME/FILENAME.dest When transferring large files the '-big' option should be used as a hint that tgcp should use high-performance file transfer methods. Since high-performance file transfer methods may have a higher startup cost, this option should not be used for small transfers.

Local home to local scratch transfer

The following example describes a transfer of the file "bigfile" from the local site to local scratch. The option "-stripe" designates a striped transfer: tgcp -stripe FILENAME /disks/scratchgpfs1/USERNAME/FILENAME.onscratch

The current method for improving performance when '-big' is specified is using striped gridFTP, which is equivalent to the '-striped' option in globus-url-copy. tgcp supports the '-stripe' option for users who are familiar with globus-url-copy.