datatransfer

The data transfer section provides information on the two types of transfers made using XSEDE:

• Moving data between locations on XSEDE: The Globus Online File Transfer service can be used to move data between all XSEDE sites, which are accessible as transfer endpoints in the service. Globus Online simplifies and automates file transfer without the need to install or interact with GridFTP.
• Moving data to XSEDE from a local machine: This might include observational data from sensors, surveys, etc., that will be analyzed on XSEDE computing resources. Globus Connect is a feature of Globus Online that makes it possible to create a transfer endpoint on any machine (including campus servers and home laptops) with just a few clicks and without the typical difficulties of a GridFTP install. Unix commands like scp, ftp and rsync can also be used to transfer your data. These commands are installed on most XSEDE login hosts. However, they do not provide the performance and ease of use of Globus Online.
• Moving data between two non-XSEDE machines: If you need to move data from a campus to another campus, computing facility, or personal machine, then Globus Connect can help you do that -- even if the machine is behind a firewall or NAT.

About Globus Online

Globus Online is a fast, reliable service for high performance file transfer. This hosted service provides a solution to data movement challenges by providing a robust, secure, and highly monitored environment for file transfers that has powerful yet easy-to-use interfaces. Globus Online simplifies secure data movement without requiring construction of custom end-to-end systems. Users have reported achieving up to 4-5 Gb/sec moving terabytes worth of data among DOE and other sites. For details about the service, visit https://www.globusonline.org/about. To sign up, visit https://www.globusonline.org/SignUp.

Data Transfer Performance

What impacts transfer rates?

• Disk speed
• Connectivity of disk to node
• Node characteristics & load
• Connectivity of node to WAN
• For all networks:
  • Bandwidth
  • Latency
  • Buffer Size
  • Protocol
  • Load
  • Encryption

Many of these factors are intrinsic to the system and not in the control of the user. However, by using the recommendations above, choices made by the user can significantly influence the speed of data transfers, in some cases resulting in 50 times the transfer speed that results from default settings and non-recommended commands. Because of the factors above, even with the best optimization, don't expect 40 Gb/sec; performance is usually limited by end node connectivity, not WAN bandwidth.