MIDAS Meteorological Data: FAQ

The Met Office MIDAS dataset collection is one of the most popular dataset collections in CEDA's archives and often causes a number of questions for users.

Whilst answers to most questions can be found in the MIDAS User Guide, the following Frequently Asked Questions (FAQs) have been put together to hopefully answer some common issues:

1. When are station data updates available?
2. Am I using the right station ID?
3. Why doesn't this station have X type of data?
4. My station says it was operational for my time period, but the data aren't available!
5. Looks like my station should report the data I want, but I can't find them!
6. Why does a site only report daily/monthly rain values and not both?
7. Can I get a complete station listing?
8. Where else can I get station data?
9. How to access the latest MIDAS Open Data

1. When are station data updates available?

MIDAS data are collected by the Met Office as part of their ongoing long term database. Some of the data messages they receive in the MIDAS system may arrive sometime after they were originally observed due to the mechanism the information may be submitted and/or digitised. Such delays may mean that data may be a month or two behind, depending on the practices being followed.

In addition, CEDA only obtains a copy of the data from the MIDAS system once a month and usually covering the previous 6 months to pick up older data. Thus, in some cases there may be upto 2 months delay before the bulk of missing data are within the MIDAS collection at CEDA .

2.  Am I using the right station ID?

A meteorological station may operate a suite of instruments each with their own instrument identifiers and even reporting within different networks. As such a site will typically have more than one identifier associated with it.

For example, Camborne has a MIDAS src_id '1395' which helps to "join" all the data from this station together in the MIDAS collection. However, for it also has a range of station codes for the wide range of message types that it issues, as shown in this sample below:

3. Why doesn't this station have X type of data?

Different station covered by the MIDAS collection have differing operational uses - some are part of networks of rain gauges used by water companies to monitor rainfall levels, others are at airfields whilst others are part of meteorological agencies' monitoring networks. As such, the types of data that they collect and, therefore, report via "messages" they issue will vary. In the Camborne example above many message types can be seen, from "UATMP" which indicates that the site issues TEMP messages as part of an upper air network to "MODLERAD" which indicates that it also collects and reports irradiance data at the surface. As such attention to the different message types is also needed to ensure that the station is likely to be reporting the required data that you want. (e.g. if it only reports "WADRAIN" - i.e. daily rain - then don't expect to find wind data from the site).

4. My station says it is operational for my time period, but the data aren't available!

Though a station's start and end date may be a good indicator when data may be available, a station's reporting characteristics will often vary over time, so it's important that you also check the individual start and end dates for the message types related to the data that you are after for the station. For example, in the Camborne example above you can see that SREW, SYNOP messages are still ongoing, but CLM71 and HWND6910 messages have ceased. In other cases the site itself may have been operational for some time, but there may not be data available from that site until a later time.

Earlier data may not yet have been added to MIDAS, but could be available via the National Meteorological Library and Archive operated by the Met Office. See FAQ 8: Where else can I get station data? for more information.

Additionally, 2020 data delivery may have been affected due to COVID-19 restrictions at some sites and will either be delivered at a later date as the backlog of data are added to the MIDAS system or will remain unavailable.

5. Looks like my station should report the data I want, but I can't find them!

Sometimes the station information would suggest that you should be able to get the data that you want from the station, but it still doesn't appear to be available within MIDAS. This could be due to a variety of reasons such as: 

• they are fairly recent data and they haven't been added to the MIDAS collection yet
• they are recent data that have arrived at the Met Office in a paper form, but haven't been digitised yet
• they are old data, but no digitisation has happened (see below)
• the station information may not be correct - these are hand crafted by a team at the Met Office, based on available information and sometimes this doesn't reflect the actual MIDAS holdings

For all the above cases CEDA have very limited ability to get any missing data, except for recent data that is still in the process of being digitised (for these, CEDA still have to wait for that process to be complete and have no way of determining when those data will be available).  Where we are aware of station information being incorrect, though, we do try and let the Met Office MIDAS team know about these so get this information checked and updated in due course.

Earlier data may not yet have been added to MIDAS, but could be available via the National Meteorological Library and Archive operated by the Met Office. See FAQ 8: Where else can I get station data? for more information.

6. Why does a site only report daily/monthly rain values and not both?

Within MIDAS all data from a given site can be found though search via its "src_id" - a unique identifier that the Met Office gives for the site itself. A site may, however, operate one or more raingauges - for example, it may operate a raingauge to record daily values or one to record monthly values, or one for both types of record. Each rain gauge will also have its own identifier (id, as opposed to src_id in the MIDAS data) and the Met Office stores the data all gauges for the site. The gauge IDs are listed on the station details page available through the MIDAS Station Search tool. Thus, by looking by "src_id" you will get data from all gauges on site, and specific rain gauge data can be further identified by its specific "id". However, whilst daily data may be recorded at a given site without an additional monthly rain gauge the Met Office does not derive monthly values from daily values reported from that site within MIDAS. 

7. Can I get a complete station listing?

The MIDAS Station Search tool is a great way to find specific stations and access their metadata, but if you need access to station metadata from many sites (e.g. all the UK sites) to pull into a database, for example, then you can access the site station details a series of "metadata" files in the CEDA archive. You can access these files at: https://data.ceda.ac.uk/badc/ukmo-midas/metadata/. Here you will find a readme file detailing the contents of each sub-folder and the metadata files they contain. These provide details such as a station's name, location and river basin, as well as specific notes ("remarks") given for data from the sites and details on the geographic locations (e.g. county names). Note, these metadata files are updated periodically when CEDA obtains a new version of them from the Met Office, roughly monthly, so these files may change from time to time.

8. Where else can I get station data?

The National Meteorological Library and Archive, operated by the Met Office, have large stores of old weather reports stretching back to the 19th century. Whilst some of these records are in the process of being digitised and values converted into computer files for eventual inclusion in MIDAS, there are may records yet to go through this process. The NMLA archives include items such as:

• ship logs
• weather report from stations operated in the UK by both the Met Office and, prior to that, various meteorological societies or other organisations
• weather charts

These may be held purely in paper archives or may also have been digitised to be viewable online, but not yet 'keyed' (the process to convert the written text visible in the digital image of the original form to values stored in a database/file.

To explore what is available the National Meteorological Library and Archive visit the NMLA catalogue at: https://library.metoffice.gov.uk/Portal/Default/en-GB/Search/SimpleSearch.

9. How to access the latest MIDAS Open Data

The MIDAS Open data are release annually where the latest version holds data up to the end of the previous year. Whilst more recent data are available in the full MIDAS collection for users, up to the end of the previous month when CEDA gets an update, these aren't available to all users. However, limited extractions for recent data are available via the Met Offce's National Meteorological Library and Archive enquiry service. For details of the levels of enquiry they are able to support see the NMLA data policy.