Elasticsearch at CEDA

Elasticsearch at CEDA: Making data more discoverable

CEDA maintains an Elasticsearch cluster to index information which enables us to improve the searchability of our data holdings. This is both for us to improve the services we provide and allows external users to query our data holdings and build services using the response.

What is available?

Name Description
ceda-fbi Main index containing file level metadata about all files in the CEDA archive
ceda-dirs Index of all the directories in the CEDA archive. Includes information about the CEDA catalogue and 00README if there is one.
ceda-eo Earth observation index containing metadata about satellite scenes from Sentinel 1,2,3 and Landsat 5,7 and 8.
faam Metadata from flights flown by the Facility for Airborne Atmospheric Research
eufar Metadata from flights flown by the European Facility for Airborne Research
arsf Metadata from flights flown by the Airborne Research and Survey Facility
bas-masin Metadata from flights flown by the British Antarctic Survey – Meteorological Airborne Science Instrumentation

You can access the search and mappings API to get information.

401 Unauthorised

Access to our Elasticsearch cluster is restricted. Unauthenticated requests will only allow you to perform search and get mapping operations on the indices specified above. 

How to search the indexes?

Python

There is a very useful python client which will enable you to interact easily with elasticsearch at CEDA.

It can be installed using:

pip install python-elasticsearch

A basic example script which will return some results from the ceda-fbi index:

from elasticsearch import Elasticsearch

query = {
	"query": {"match_all": {}}
}

es = Elasticsearch(["https://jasmin-es1.ceda.ac.uk"])
es.search(index="ceda-fbi", body=query)
es.indices.get_mapping(index="ceda-fbi")

Access via command-line tools: wget and curl

Here is a link to the URI search documentation from elasticsearch which describes all the allowed keywords.

https://www.elastic.co/guide/en/elasticsearch/reference/current/search-uri-request.html

Some examples using the wget and curl command-line tools:

wget https://jasmin-es1.ceda.ac.uk/ceda-fbi/_search?size=1
wget https://jasmin-es1.ceda.ac.uk/ceda-fbi/_mapping
curl https://jasmin-es1.ceda.ac.uk/ceda-fbi/_search?size=1
curl https://jasmin-es1.ceda.ac.uk/ceda-fbi/_mapping

Curl with body

curl https://jasmin-es1.ceda.ac.uk/ceda-fbi/_search -H 'Content-Type: application/json' -d '{"query": {"match_all": {}}}'

Still need help? Contact Us Contact Us