Elasticsearch at CEDA
Elasticsearch at CEDA: Making data more discoverable
CEDA maintains an Elasticsearch cluster to index information which enables us to improve the searchability of our data holdings. This is both for us to improve the services we provide and allows external users to query our data holdings and build services using the response.
What is available?
| Name | Description |
|---|---|
| fbi-2022 | Main index containing file and directory level metadata about all files in the CEDA archive |
| cedamoles-haystack-prod-alias | The CEDA data catalogue search index. |
| ceda-eo | Earth observation index containing metadata about satellite scenes from Sentinel 1,2,3, Sentinel ARD and Landsat 5,7 and 8. |
| faam | Metadata from flights flown by the Facility for Airborne Atmospheric Research |
| eufar | Metadata from flights flown by the European Facility for Airborne Research |
| arsf | Metadata from flights flown by the Airborne Research and Survey Facility |
| bas-masin | Metadata from flights flown by the British Antarctic Survey – Meteorological Airborne Science Instrumentation |
You can access the search API to get information.
The Elasticsearch API is extensively and thoroughly documented at: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-your-data.html
401 Unauthorised
Access to our Elasticsearch cluster is restricted. Unauthenticated requests will only allow you to perform search operations on the indices specified above.
How to search the indexes?
Python
There is a very useful python client that will enable you to interact easily with elasticsearch at CEDA.
It can be installed using:
pip install elasticsearch==8
A basic example script which will return some results from the ceda-fbi index:
from elasticsearch import Elasticsearch
es = Elasticsearch(["https://elasticsearch.ceda.ac.uk"])
es.search(index="fbi-2022", query={"match_all": {}})
Access via command-line tools: wget and curl
Here is a link to the URI search documentation from elasticsearch which describes all the allowed keywords.
https://www.elastic.co/guide/en/elasticsearch/reference/current/search-uri-request.html
Some examples using the wget and curl command-line tools:
wget https://elasticsearch.ceda.ac.uk/fbi-2022/_search?size=1 curl https://elasticsearch.ceda.ac.uk/fbi-2022/_search?size=1
Curl with body
curl https://elasticsearch.ceda.ac.uk/fbi-2022/_search -H 'Content-Type: application/json' -d '{"query": {"match_all": {}}}'