Depositing Data at CEDA: FTP and RSYNC

There are 3 ways you can send data to CEDA (HTTP, FTP and RSYNC). This help document will take you through steps to upload your data via FTP and RSYNC recommend for large amounts of data. To upload data via HTTP please refer to Depositing Data at CEDA: A step by step guide

FTP

Once you have collected/produced your atmospheric or earth observation data; processed, calibrated, validated and quality checked it and prepared it for archiving using correctly applied standard data formats (for example, NetCDF should adhere to the CF conventions and have good quality metadata including global attributes) and meaningful filenames, you are now ready to upload it to the CEDA archive, using the following steps. A video tutorial can be found here.

1. Register for a  CEDA account.

2. Go to arrivals.ceda.ac.uk and login with your CEDA username and password.

3. Click begin. You need to agree to the deposit agreement the first time you use the service.  

4. Click browse deliveries.

5. Choose an existing delivery directory or create a new delivery directory by clicking on 'new directory'. (Add a suitable title e.g. the project, instrument name, model name or what the data contains, etc).

7. Click on 'other upload methods' and take note of the URL string

8. Download FileZilla https://filezilla-project.org/

9. Paste the URL connection string, e.g.  <a href="ftp://username/">ftp://username</a>:<a href="mailto:password@arrivals.ceda.ac.uk">password@arrivals.ceda.ac.uk</a>/mydata into the "Host" field and hit the "Quickconnect" button (the other fields will be filled in automatically, or you can fill them in yourself)

10.Browse and upload files using the explorer at the bottom of the screen

This will be similar for any desktop FTP client
With the  ftp command bundled with many Linux/UNIX operating systems and Windows:
  1. Type ftp <a href="http://arrivals.ceda.ac.uk/">arrivals.ceda.ac.uk</a>` to connect directly to the arrivals server
  2. Enter your FTP username and password as found on your data upload page
  3. When connected, you will be in your Arrivals home area, use cd mydata navigate to the subdirectory with the same name as your data delivery
  4. Start uploading data with the put command

11. Create and upload a metadata.yaml text file along with the data.  The metadata.yaml file needs to contain the details of the dataset, project and instrument or model. Adding the information this way helps keep the record and the data together. To create the metadata file use this utility or edit one of the examples below.

station-data_metadata_example.yaml

instrument_metadata_example.yaml

model_metadata_example.yaml

12. Click check delivery button on arrivals service.

13. Confirm Submission. It s really important to confirm delivery in the arrivals service so that CEDA  can be notified.

RSYNC

Once you have collected/produced your atmospheric or earth observation data; processed, calibrated, validated and quality checked it and prepared it for archiving using correctly applied standard data formats (for example, NetCDF should adhere to the CF conventions and have good quality metadata including global attributes) and meaningful filenames, you are now ready to upload it to the CEDA archive, using the following steps. A video tutorial can be found here.

1. Register for a  CEDA account.

2. Go to arrivals.ceda.ac.uk and login with your CEDA username and password.

3. Click begin. You need to agree to the deposit agreement the first time you use the service.  

4. Click browse deliveries.

5. Choose an existing delivery directory or create a new delivery directory by clicking on 'new directory'. (Add a suitable title e.g. the project, instrument name, model name or what the data contains, etc).

7. Click on 'other upload methods' and take note of the command to edit

8. You need to have the  rsync command installed (which comes by default on Linux/UNIX systems, Windows users will have to install it themselves)

9.Edit the command given on the Rsync instructions of the upload page (e.g.  rsync -av /path/to/source <a href="mailto:username@arrivals.ceda.ac.uk">username@arrivals.ceda.ac.uk</a>::username/mydata) and change "/path/to/source" to the location of your data files

10. Run the command, enter your Rsync password when prompted and your data will be synced with the upload directory

As mentioned on the upload page, after generating a new password for Rsync you may need to wait up to an hour before you can use it

11. Create and upload a metadata.yaml text file along with the data.  The metadata.yaml file needs to contain the details of the dataset, project and instrument or model. Adding the information this way helps keep the record and the data together. To create the metadata file use this utility or edit one of the examples below.

station-data_metadata_example.yaml

instrument_metadata_example.yaml

model_metadata_example.yaml

12. Click check delivery button on arrivals service.

13. Confirm Submission. It s really important to confirm delivery in the arrivals service so that CEDA  can be notified.

Did this answer your question? Thanks for the feedback There was a problem submitting your feedback. Please try again later.