Data Transfer Tools

This article lists the data transfer tools available on JASMIN and provides links to articles that describe them in more detail.

Data Transfer tools

The are many tools that you can use for transferring data to/from JASMIN. See the articles linked below for more details.

Tool Info
scp
A basic transfer tool that works over the SSH protocol. Similar to "cp" but copies between remote servers.
rsync (over SSH)
Like scp but slightly more sophisticated. Allows synchronisation between remote directory trees.
sftp
SSH FTP - works over SSH.
bbcp
A command-line tool that allows the user to specify parallel transfer over multiple streams, using SSH authentication.
GridFTP (over SSH)   A comprehensive data transfer tool. Highly configurable and able to transfer over multiple parallel streams. Used over SSH in this case.
GridFTP (certificate-based)
As above, but using certificate-based authentication instead of SSH credentials.  
Globus Online
Managed service for GridFTP, includes capability to orchestrate transfers between third-party hosts and receive notifications of job status. Has web and command-line interfaces.
FTP
File Transfer Protocol. An aged transfer protocol suitable for small file transfers but limited.
LFTP
Parallel-capable FTP client.
wget & curl Download tools for accessing resources over HTTP and HTTPS primarily.
Python transfer tools Methods of managing/scripting data transfer tasks using Python.
synda A command-line tool developed for downloading and replicating data from ESGF nodes. Able to use HTTP and GridFTP transfer methods.
MASS-client (Met Office)
A specific command-line tool installed on the mass-cli1.ceda.ac.uk server on JASMIN. Enables extractions from the Met Office MASS Archive directly on to
OPeNDAP A transfer protocol for extracting subsets of files from a remote server (over
bcopy
A parallel transfer tool (specific to JASMIN) that uses the LOTUS batch cluster to copy data efficiently between disk areas within JASMIN.

Still need help? Contact Us Contact Us