00 TEMPLATE - more detailed ingestion workflows

NOTE - DO NOT PUBLISH THESE ARTICLES

just click on the tick to save them  

What are the data

Collection Title/description:
MOLES collection page:
DMP link:

Who is involved

CEDA officer   Graham Parton
Data provider contact See DMP links

Ingestion process

Data arrivals route

Data are prepared by ### staff which they push to their arrivals areas into various streams. Arrivals areas are:

 - . 

- . 

Arrivals stream name(s): 

  • cfarr-radar : for all radar related products

Data pre-ingestion handling

These use file-processor as automatic processes on the crontab.
config details in:  /home/badc/software/datasets//_fileProcessor.cfg
steam: [cfarr-cloud-camera-unpacker-2]
description: unpacks cloud-camera-2 images ready for ingestion
script: unpacker -c /home/badc/software/datasets/chilbolton/chilbolton_fileProcessor.cfg -s cfarr-cloud-camera-unpacker

Data Ingestion

Ingestion processes have to map between the filename components and directory locations. To do this the ingestion is done via calls to the  cfarr_ingester.py script. This script has a class that inherits from the standard Ingester class. cfarr_ingester.py is then called for each ingest stream as defined in the /home/badc/software/datasets/chilbolton/chilbolton_fromDeliveries.cfg config file

Runs on crontab.

Post Ingestion work

None

QC checks:

Files are only checked by CFARR staff. No other checking done at this stage. Format of files was confirmed to be CF-compliant ahead of setting up automatic processes.

Common problems:

Standard issues around ingest only. 

Any other useful information need to know:

None.

Did this answer your question? Thanks for the feedback There was a problem submitting your feedback. Please try again later.

Still need help? Contact Us Contact Us