Elasticsearch at CEDA
Elasticsearch at CEDA: Making data more discoverable
CEDA maintains an Elasticsearch cluster to index information which enables us to improve the searchability of our data holdings. This is both for us to improve the services we provide and allows external users to query our data holdings and build services using the response.
What is available?
| Name | Description |
|---|---|
| ceda-fbi | Main index containing file level metadata about all files in the CEDA archive |
| ceda-dirs | Index of all the directories in the CEDA archive. Includes information about the CEDA catalogue and 00README if there is one. |
| ceda-eo | Earth observation index containing metadata about satellite scenes from Sentinel 1,2,3, Sentinel ARD and Landsat 5,7 and 8. |
| faam | Metadata from flights flown by the Facility for Airborne Atmospheric Research |
| eufar | Metadata from flights flown by the European Facility for Airborne Research |
| arsf | Metadata from flights flown by the Airborne Research and Survey Facility |
| bas-masin | Metadata from flights flown by the British Antarctic Survey – Meteorological Airborne Science Instrumentation |
You can access the search and mappings API to get information.
The Elasticsearch API is extensively and thoroughly documented at: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-your-data.html
401 Unauthorised
Access to our Elasticsearch cluster is restricted. Unauthenticated requests will only allow you to perform search and get mapping operations on the indices specified above.
How to search the indexes?
Python
There is a very useful python client that will enable you to interact easily with elasticsearch at CEDA.
It can be installed using:
pip install python-elasticsearch
A basic example script which will return some results from the ceda-fbi index:
from elasticsearch import Elasticsearch
query = {
"query": {"match_all": {}}
}
es = Elasticsearch(["https://elasticsearch.ceda.ac.uk"])
es.search(index="ceda-fbi", body=query)
es.indices.get_mapping(index="ceda-fbi")
Access via command-line tools: wget and curl
Here is a link to the URI search documentation from elasticsearch which describes all the allowed keywords.
https://www.elastic.co/guide/en/elasticsearch/reference/current/search-uri-request.html
Some examples using the wget and curl command-line tools:
wget https://elasticsearch.ceda.ac.uk/ceda-fbi/_search?size=1 wget https://elasticsearch.ceda.ac.uk/ceda-fbi/_mapping
curl https://elasticsearch.ceda.ac.uk/ceda-fbi/_search?size=1 curl https://elasticsearch.ceda.ac.uk/ceda-fbi/_mapping
Curl with body
curl https://elasticsearch.ceda.ac.uk/ceda-fbi/_search -H 'Content-Type: application/json' -d '{"query": {"match_all": {}}}'