Data Transfer Tools: Copy service

This article provides information about the bcopy service, including:

  • Where you can use it
  • How to use it
  • Further advice

bcopy command

The bcopy command provides a service which uses the LOTUS batch cluster to copy data efficiently between certain disk areas within JASMIN: those which use Panasas storage hardware. For most users, this means only:

  • home directory*
  • group workspaces

It is the recommended method to be used for copying large amounts of data between group workspaces. * Home directories are unlikely to contain large enough quantities of data to make it worthwhile copying data this way, but this may be useful for testing purposes.

The service is available on the scientific analysis servers jasmin-sci[123] and cems-sci[12] as the command /usr/local/bin/bcopy. These servers are able to submit jobs to LOTUS.

It is not installed on the data transfer servers, as these have no access to LOTUS.

The service uses 8 CPU cores as workers and makes use of a special queue called copy

How to use it

$ bcopy -s SourceDirectory -d DestinationDirectory [-c]  [--retry]
where:
-c is optional for checking if the source is readable by the current user
--retry is only used when the system reports an error and asks the user to re-run the job with the --retry switch appended and the end of the command
Note that both source and destination need to be directories. It is not possible to specify particular files within a directory to be copied.

Example : 

$ bcopy -s /group_workspaces/jasmin/gws1/mydir -d /group_workspaces/jasmin/gws2/dest

Once the command has been issued, you should receive confirmation of job submission:

PANFS source directory: /group_workspaces/jasmin/gws1/mydur PANFS destination directory: /group_workspaces/jasmin/gws2/dest
Job <8063810> is submitted to queue <copy>.
Done

Once the job has completed, you should see files names JOBID.log JOBID.err where JOBID is the job number as given in the submission confirmation above. You should inspect these text files to check any output given by the service.

Further advice

  • You are strongly recommended to use the bcopy service in combination with a synchronisation tool like rsync to ensure that all data has been copied. For example, you could use the bcopy service to do the main bulk of the copying, then run a subsequent rsync command to "mop up" any remaining files which may have been missed. In this case, a data transfer server should be used for the rsync operation, in case the remaining transfer is large. See rsync.
  • You should cease any further changes in the source directory once the initial bcopy command has been issued.
  • For full verification, you should consider generating a manifest including checksums of each file before and after the copy operation. The 2 manifests can then be compared to ensure that all the data has transferred and its integrity preserved. See Tips for data transfer workflows.
  • Note that, as with other transfers (including rsync) invoked by a user with normal system privileges, file and directory ownership are NOT preserved when moving data which contains items belonging to multiple users. Files and directories created in the destination will take on the ownership of the user who invoked the transfer. If you need to preserve ownership metadata, please contact the helpdesk support@ceda.ac.uk.

Still need help? Contact Us Contact Us