Synda : Synchronise Data

Synda

Synda stands for Synchronise Data it is a command line utility that can manage the downloading of data through the Earth System Grid Federation (ESGF). Synda is composed of two components a transfer module (sdt) and a post-processing module (sdp). The process for installing on JASMIN the transfer module is outlined here. Full documentation on usage can be found in the main Synda documentation.

The Synda installation instructions can be found here. However, some useful additional installation instructions for the transfer module are provided below.

Before downloading:

Make a directory where you want the software installed

$ mkdir /synda

You may wish to have multiple versions (3.9 is the current latest version) if so, make an "sdt" directory (the default) and symlink this to your version, from your Synda directory:

$ mkdir sdt_3.9
$ ln -s sdt_3.9 sdt

Create a `setup_paths.sh` file in the synda directory with the following content (uses a general version of ) :

# Custom installation of Python
# define a deps dir that contains a custom build of Python path below is a current working version 
deps_dir=/group_workspaces/jasmin3/esgf_repl/synda/deps/
python_dir=$deps_dir/deps/Python

# Specify the top_dir as a location of synda once built
top_dir=/group_workspaces/jasmin2/cp4cds1/synda
export LD_LIBRARY_PATH=$python_dir/lib:$LD_LIBRARY_PATH
export PATH=$python_dir/bin:$PATH
export ST_HOME=$top_dir/sdt
export PATH=$ST_HOME/bin:$PATH

# If using the Synda post processing module
#export SP_HOME=$top_dir/sdp
#export PATH=$SP_HOME/bin:$PATH

unset deps_dir
unset top_dir
unset python_dir

Before running synda this file must be executed every time

$ chmod u+x ./setup_paths.sh 
$ source setup_paths.sh

Download

Now following the online Synda instructions:

wget the install script, and make it executable.
run the install script:

$ ./install.sh -d <version-number> -t <directory path to where sdt is>

Configurations

(1) In the sdt configurations file

sdt/conf/sdt.conf:

Set:

Default directory path for selection files: selection_path=<> default=synda/sdt/selection
Default directory path for default selection files: default_path=<> default=synda/sdt/selection
Default data directory path: data_path=<> default=synda/sdt/data; likely want to change this
Default database directory: db_path=<> default=synda/sdt/db/
Change default index to be CEDA by commenting out DKRZ and uncommenting CEDA

(2) In the sdt credentials file

sdt/conf/credentials.conf

Supply a valid openid and username