CEDA OPeNDAP: scripted interactions

As the CEDA Archive has various data access controls in place users wishing to interact with the CEDA OPeNDAP service will need to do so using security certificates. The following instructions will guide you through how to:

• Find a download URL
• Download data from the CEDA Archive via cURL
  • Getting started
  • Get a security certificate
  • Running the cURL command
• Download a file with Python
  • Simple download script example
  • Connecting to a file in a Python script using the OpenDAP service
• Known Issues

Find a download URL

To discover the URL to download files from:

• first navigate to the dataset you wish to download in the CEDA archive browser (for example, UKMO-midas-open).  
• Down the right hand side of the list there is a download icon for each file in the dataset.  
• Right click this download icon and select "copy link address".  
• Now you can paste that link address into your script, or onto the command line.  The links will start:

http://dap.ceda.ac.uk/

Download a file with cURL

Getting started

• First create a directory where you will download the scripts to generate the security certificates you'll be using later

mkdir ~/ceda_pydap_cert_code

• Switch to that directory

cd ~/ceda_pydap_cert_code

• Next you'll need to download the code from the following Git Hub repository:

git clone https://github.com/cedadev/online_ca_client

• Then navigate to the scripts directory:

cd online_ca_client/contrail/security/onlineca/client/sh/

• Finally, run the following command to get a folder full of trustroots - these will be needed later when we generate your temporary certificate:

./onlineca-get-trustroots-wget.sh -U https://slcs.ceda.ac.uk/onlineca/trustroots/ -c ~/trustroots -b

Getting a Security Certificate.

Having obtained the necessary code from the git hub repository you now need a certificate associated with your myCEDA account in order to access restricted datasets

• Run the following command to get your temporary (3 day) certificate (remember to replace your ceda username below!:

./onlineca-get-cert-wget.sh -U  https://slcs.ceda.ac.uk/onlineca/certificate/ -c ~/trustroots -l <ceda username> -o $PWD/creds.pem

• NOTE - You'll need to input your password after you submit the command
• This will pipe the output (your certificate) into a file of your choosing (above it will be a local file: $PWD/creds.pem), you can alter this if you wish e.g.  ~/tmp/temp_cert.pem.

Running the cURL command

Finally, you can use your new certificate to download data using any tool that supports this feature. In our example we will use curl, but you can also use wget with the --certificate option.

• Use your certificate as a parameter to cURL to retrieve something from PyDAP:

curl --cert $PWD/creds.pem -L -c /dev/null http://dap.ceda.ac.uk/thredds/fileServer/badc/ukmo-midas-open/data/uk-daily-temperature-obs/dataset-version-201901/aberdeenshire/00145_cairnwell/qc-version-1/midas-open_uk-daily-temperature-obs_dv-201901_aberdeenshire_00145_cairnwell_qcv-1_1994.csv

The -L option is required for following links and the -c option tells cURL to store cookies. On Windows, you will need to use NULL rather than /dev/null   for the -c option input.

Note that older versions of cURL may require an absolute path to your certificate file, as above.

• The example above downloads a text file and displays it on the console.  To download to file specify the -o option, for example:

curl --cert $PWD/creds.pem -L -c /dev/null http://dap.ceda.ac.uk/thredds/fileServer/badc/ukmo-hadobs/data/insitu/MOHC/HadOBS/HadUK-Grid/v1.0.0.0/1km/tas/ann/v20181126/tas_hadukgrid_uk_1km_ann_188901-188912.nc -o tas_hadukgrid_uk_1km_ann_188901-188912.nc

• If you are seeing problems with SSL certificates such as:

curl: (60) Peer certificate cannot be authenticated with known CA certificates

then you can try adding the --insecure option to the curl command:

curl --insecure --cert $PWD/creds.pem -L -c /dev/null ...

Download a file with Python

Simple download script

We have prepared an example Python script which shows how to use the ContrailOnlineCAClient and requests Python libraries to download a file from the archive.

The script was tested with Python 2.7 and Python 3.8 on both Windows 10 and Linux.

Connecting to a file in a Python script using the OpenDAP service

Please see the example help page on Reading a NetCDF file from a Python Script using OpenDAP.

Known Issues

Presently CEDA's DAP implementation has the following known issue. Whilst updates could be made to the service to address them, these are not being taken at this time as this will lead to loss of other service functionality. We hope that these will be addressed in future iterations of the services and underpinning protocols:

• int64 data type - files containing data with this type are presently unsupported via the DAP protocol deployed within the CEDA Opendap service (via Thredds) and thus will lead to a 403: message = "NcDDS Variable data type = long"; error message.