00 TEMPLATE - more detailed ingestion workflows

NOTE - DO NOT PUBLISH THESE ARTICLES

just click on the tick to save them

What are the data

Collection Title/description:
MOLES collection page:
DMP link:

Who is involved

CEDA officer	Graham Parton
Data provider contact	See DMP links

Ingestion process

Data arrivals route

Data are prepared by ### staff which they push to their arrivals areas into various streams. Arrivals areas are:

- .

Arrivals stream name(s):

cfarr-radar : for all radar related products

Data pre-ingestion handling

These use file-processor as automatic processes on the crontab.

config details in: /home/badc/software/datasets//_fileProcessor.cfg

steam: [cfarr-cloud-camera-unpacker-2]

description: unpacks cloud-camera-2 images ready for ingestion

script: unpacker -c /home/badc/software/datasets/chilbolton/chilbolton_fileProcessor.cfg -s cfarr-cloud-camera-unpacker

Data Ingestion

Ingestion processes have to map between the filename components and directory locations. To do this the ingestion is done via calls to the cfarr_ingester.py script. This script has a class that inherits from the standard Ingester class. cfarr_ingester.py is then called for each ingest stream as defined in the /home/badc/software/datasets/chilbolton/chilbolton_fromDeliveries.cfg config file

Runs on crontab.

Post Ingestion work

None

QC checks:

Files are only checked by CFARR staff. No other checking done at this stage. Format of files was confirmed to be CF-compliant ahead of setting up automatic processes.

Common problems:

Standard issues around ingest only.

Any other useful information need to know:

None.