Centre for Environmental Data Analysis
Centre for Environmental Data Analysis Digital RepositoryNot a member yet
1250 research outputs found
Sort by
MIDAS Data User Guide for UK Land Observations, v20210705
Full text linkBackground information on using data from the Met Office climate database, MIDAS. This version has minor update to add units for cloud height and visibility, decametres
NCAS CAO NLC-Camera Time-Lapse Video starting at 2020-06-21 20:00 UTC
No full textA time-lapse video showing Noctilucent Clouds (NLCs) seen from southern England (51.15°N,-1.44°E) during the night of 21st/22nd June 2020. NLCs are a seasonal wonder of the natural world. They can only be seen from upper-middle and high latitudes during the mid-summer months (between mid May and mid August in the northern hemisphere). They are the result of ice crystals forming at the extraordinarily high altitude of around 82 km. This is 70 km higher than virtually all other clouds seen at these latitudes and qualifies as being at the edge of space (the atmospheric density and pressure are approximately 100,000th of their values at sea level). NLCs can only be seen during twilight hours, hence the name noctilucent, which means night-shining. In this video there is a mild display of NLCs during the dusk followed by a much more impressive display during the dawn. Note that British Summer Time (BST) is one hour ahead of Coordinated Universal Time (UTC). The solar elevation angles do not take account of atmospheric refraction, which is only noticeable when the sun is close to the horizon
MIDAS Data User Guide for UK Land Observations, v20200921
Full text linkBackground information on using data from the Met Office climate database, MIDAS. This version has minor updates cover the user of the 'Z' prefix on station IDs
MIDAS Data User Guide for UK Land Observations, v1.1
Full text linkBackground information on using data from the Met Office climate database, MIDAS. This version has minor updates to sections 2 and 3
CEDA Annual Report 2019 - 2020
Full text linkThis annual report presents key statistics for the year past (2019 - 2020) as well as a series of snapshots of activity, expressed as short highlights and short reports
MIDAS Data User Guide for UK Land Observations
Full text linkBackground information on using data from the Met Office climate database, MIDAS
NCAS CAO NLC-Camera Time-Lapse Video starting at 2020-07-11 00:00 UTC
No full textA time-lapse video showing Noctilucent Clouds (NLCs) and a Comet seen from southern England (51.15°N,-1.44°E) during the dawn of 11th July 2020. NLCs are a seasonal wonder of the natural world. They can only be seen from upper-middle and high latitudes during the mid-summer months (between mid May and mid August in the northern hemisphere). They are the result of ice crystals forming at the extraordinarily high altitude of around 82 km. This is 70 km higher than virtually all other clouds seen at these latitudes and qualifies as being at the edge of space (the atmospheric density and pressure are approximately 100,000th of their values at sea level). NLCs can only be seen during twilight hours, hence the name noctilucent, which means night-shining. Note that British Summer Time (BST) is one hour ahead of Coordinated Universal Time (UTC). The solar elevation angles do not take account of atmospheric refraction, which is only noticeable when the sun is close to the horizon. The brightest star seen in the video is Capella, which is in the constellation Auriga. It starts near the bottom-centre and moves in an arc towards the right and upwards. The comet NEOWISE can be seen following a similar path from approximately 01:20 UTC
Long-term Archive Challenges: Enhancing Data Discovery via Multilevel Metadata Aggregations At Scale
Full text linkData archives require accurate, content-rich data catalogues that are fit-for-purpose to support meaningful data discovery. However, sourcing suitable high-quality metadata to populate catalogues at scale can be problematic when manual workflows are no longer able to cope. One solution is automated metadata harvesting directly from archive contents, drawing on technical solutions to Big Data challenges faced by rapidly evolving, petabyte-scale, heterogeneous archives. Yet other issues quickly arise, including: changes in, or lack of, metadata standards over time; missing or incorrect metadata; diversity of, and lack of interoperability between, formats and metadata conventions; and, changes in data availability over time. These are further compounded when dealing with historical archives and legacy systems stretching back decades before comprehensive end-to-end metadata harvesting workflows were envisioned.
The Centre for Environmental Data Analysis (CEDA) Archive has long-term archiving responsibility for the UK atmospheric, climate change and Earth observation communities. With highly heterogeneous deposits of both historic and rapidly growing fresh data holdings, spanning over 5,500 datasets, 200 million files and over 5 Pb of online storage, the CEDA Archive is no newcomer to the 4Vs of the Big Data challenge: Volume, Velocity, Variety and Veracity. Recently CEDA’s combined use of parallel archive and processing systems with noSQL Elasticsearch indexes has addressed many of these challenges. This has permitted file-level metadata such as parameter, spatial and temporal information to be indexed and then aggregated to populate most dataset records in CEDA’s ISO-driven data catalogue. However, there remain significant shortcomings in the quality and coverage of the metadata harvested via this pipeline (less than 50% of files return parameter information for example) that require additional, complementary approaches.
The key remaining challenges are how to address the following issues: where file-based metadata is incomplete or missing (e.g. unscannable file-formats; incomplete metadata); coping with incorrect file-based metadata (e.g. incorrect geo-temporal information; mapping between coordinate systems); dealing with removed data files; and, coverage for offline/external content. To address this a complementary YAML-driven ’Manual Metadata Store’ has been set up operating at the dataset records level in the CEDA data catalogue. This enables a manually maintainable, traceable, versioned alternate metadata source and splicing rules to determine how the information should be used in conjunction with automatically harvested metadata such as CEDA’s file-level index. Splicing options include: partial or complete replacement; appending of information; and, setting default information. Whilst this has allowed CEDA to begin addressing issues with automated metadata harvesting, it has also raised additional questions, especially as CEDA seeks to allow file-level faceted searches in the future, where file-derived metadata is known to be incorrect or missing. At what level should amendments to metadata be applied? The dataset record, file-level index or the original metadata themselves? And how to manage the conflicting pressures of limited resources, preserving original data integrity and the desire to deliver quality information at all levels and at scale. Essentially, the next issue to face can be summed up with the questions: ‘What is Truth? And at what level to convey it
FIDUCEO project: Product user guide – Upper Tropsheric CDRs from microwave sounders
Full text linkThis document describes the Upper Tropospheric Humidity (UTH) CDR(version 1.2) data files uploaded to CEMS in February 2019. The released data record is based on the Microwave “easy” FCDR version 4.1 (see Microwave FCDR PUG). The instantaneous observations from the FCDR are used to derive a spatio-temporal averaged data record, which contains monthly mean UTH and brightness temperature mapped to a regular latitude/longitude grid covering the tropical region with a spatial resolution of 1° x1°. It covers all mission years of SSMT2 on F11, F12, F14, F15, AMSU-B on NOAA15-17 and MHS missions (NOAA18, NOAA19, MetopA, Metop-B). This product user guide gives: 1. An overview of the specifications of the data record; 2. A description of the implementation of the retrieval processing chain; 3. Information on limitations of this current version of the data record; 4. Technical details on the format and on how to access the dat
FIDUCEO project: Product user guide – Microwave FCDR release 4.1
Full text linkThis document describes the Microwave “easy” FCDR data files of version 4.1 uploaded to CEDA in January 2019. The released data record contains all mission years of SSMT2 on F11, F12, F14, F15, AMSU-B on NOAA15, NOAA16 and NOAA17 and MHS missions (NOAA18, NOAA19, MetopA,-B), i.e. a data record long enough to generate climate data records (CDRs) for climate research. The presented FCDR is a long data record of increased consistency among the instruments compared to the operational data record. The improvements are based on the strict application of the measurement equation as well as dedicated corrections and improvements within the calibration process. The data record is uncertainty quantified, respecting the correlation behaviour of underlying effects. This product user guide gives: 1. An overview of the specifications of the data record; 2. Scientific records on the generation, definition, and algorithms of the data record; 3. Information on limitations of this version of the data record; 4. Technical details on the format and on how to access the dat