Using the command line for CREPP
The CREPP CEDA Receive to publish pipeline was developed by Alan Iwi and Ag Stephens to ingest CMIP6 data and then subsequently publish this to ESGF. Data for ingest will be from either the Met Office MASS archive, already on JASMIN, or replicated international data.
Managing CREPP transfers
CREPP dataset status' can be seen through the CREPP application web interface at ppln.ceda.ac.uk most commonly using the dataset view.
Debugging
Logs are on the two machines used by CREPP some logs are only available on appropriate machine ( confirm with Alan?)
ingest3 ~/software/crepp/crepp/logs
esgf-pub /usr/local/crepp/logs
If a dataset is found to be in error if you have admin rights the dataset can be retried using the web interface, however this is only feasible for a very small number of datasets.
For bulk dataset retries use the dataset_actions.sh script located in the scripts dir, note this script only resets flags and so it is irrelevant which server this is initiated from:
On this can be done from the scripts/ directory using
dataset_actions.sh -h
usage: dataset_actions.py [-h] (-R | -S | -P <priority> | --clear-priority | -p | -r | -u | -q | -W | --republish) [-v] config [datasets [datasets ...]] positional arguments: config CREPP configuration name (e.g. 'esgf-prod') datasets dataset ID(s); if none, they are read from stdin optional arguments: -h, --help show this help message and exit -R, --retry retry failed step -S, --skip skip failed step -P <priority>, --set-priority <priority> set dataset priority --clear-priority clear dataset priority -p, --pause pause dataset -r, --release release dataset pause -u, --update update processing status and last event -q, --query query status -W, --withdraw withdraw dataset --republish republish withdrawn dataset -v, --verbose more verbose output
Removing from the archive - (esgdrs errors)
If a dataset needs removing from the archive to clear an esgdrs.out error use or other reason then on:
ingest3 ~/software/crepp/crepp/scripts/remove-from-archive.sh < <dataset_list>
then retry the ingest step.