CEDA OPeNDAP: scripted interactions

As the CEDA Archive has various data access controls in place users wishing to interact with the CEDA OPeNDAP service will need to do so using security certificates. The following instructions will guide you through how to:

Further how-to guides will follow in due course covering the steps needed for Windows users and additional functions such as parameter sub-setting and including these calls within common scripting languages such as Python, IDL and MatLab.

Getting Started

  • First create a directory where you will download the scripts to generate the security certificates you'll be using later
mkdir ~/ceda_pydap_cert_code
  • Switch to that directory
cd ~/ceda_pydap_cert_code
  • Next you'll need to download the code from the following Git Hub repository:
git clone https://github.com/cedadev/online_ca_client
  • Then navigate to the scripts directory:
cd online_ca_client/contrail/security/onlineca/client/sh/
  • Finally, run the following command to get a folder full of trustroots - these will be needed later when we generate your temporary certificate:
./onlineca-get-trustroots-wget.sh -U https://slcs.ceda.ac.uk/onlineca/trustroots/ -c ~/trustroots -b

Getting a Security Certificate.

Having obtained the necessary code from the git hub repository you now need a certificate associated with your myCEDA account in order to access restricted datasets

  • Run the following command to get your temporary (3 day) certificate (remember to replace your ceda username below!:
./onlineca-get-cert-wget.sh -U  https://slcs.ceda.ac.uk/onlineca/certificate/ -c ~/trustroots -l <ceda username> -o $PWD/creds.pem
  • NOTE - You'll need to input your password after you submit the command
  • This will pipe the output (your certificate) into a file of your choosing (above it will be a local file: $PWD/creds.pem), you can alter this if you wish e.g.  ~/tmp/temp_cert.pem.

Downloading a file with cURL

Finally, you can use your new certificate to download data using any tool that supports this feature. It should be noted that, due to an issue with our service, you may have trouble downloading data using Wget. For this reason, we are using cURL in our example.

  • Use your certificate as a parameter to cURL to retrieve something from PyDAP:
curl --cert $PWD/creds.pem -L -c /dev/null http://dap.ceda.ac.uk/thredds/fileServer/badc/ukmo-midas-open/data/uk-daily-temperature-obs/dataset-version-201901/aberdeenshire/00145_cairnwell/qc-version-1/midas-open_uk-daily-temperature-obs_dv-201901_aberdeenshire_00145_cairnwell_qcv-1_1994.csv
The -L option is required for following links and the -c option tells cURL to store cookies. On Windows, you will need to use NULL rather than /dev/null  for the -c option input.

Note that older versions of cURL may require an absolute path to your certificate file, as above.

  • The example above downloads a text file and displays it on the console.  To download to file specify the -o option, for example:
curl --cert $PWD/creds.pem -L -c /dev/null http://dap.ceda.ac.uk/thredds/fileServer/badc/ukmo-hadobs/data/insitu/MOHC/HadOBS/HadUK-Grid/v1.0.0.0/1km/tas/ann/v20181126/tas_hadukgrid_uk_1km_ann_188901-188912.nc -o tas_hadukgrid_uk_1km_ann_188901-188912.nc
  • If you are seeing problems with SSL certificates such as:
curl: (60) Peer certificate cannot be authenticated with known CA certificates

then you can try adding the --insecure option to the curl command:

curl --insecure --cert $PWD/creds.pem -L -c /dev/null ...

Finding the download URL

To discover the URL to download files from:

  • first navigate to the dataset you wish to download in the CEDA archive browser (for example, UKMO-midas-open).  
  • Down the right hand side of the list there is a download icon for each file in the dataset.  
  • Right click this download icon and select "copy link address".  
  • Now you can paste that link address into your script, or onto the command line.  The links will start:
http://dap.ceda.ac.uk/

Connecting to a file in a Python script using the OpenDAP service

Please see the example help page on Reading a NetCDF file from a Python Script using OpenDAP.

Still need help? Contact Us Contact Us