84 research outputs found
Active provenance for data intensive research
No full textThe role of provenance information in data-intensive research is a significant topic of
discussion among technical experts and scientists. Typical use cases addressing traceability,
versioning and reproducibility of the research findings are extended with more
interactive scenarios in support, for instance, of computational steering and results
management. In this thesis we investigate the impact that lineage records can have on
the early phases of the analysis, for instance performed through near-real-time systems
and Virtual Research Environments (VREs) tailored to the requirements of a specific
community. By positioning provenance at the centre of the computational research
cycle, we highlight the importance of having mechanisms at the data-scientists’ side
that, by integrating with the abstractions offered by the processing technologies, such
as scientific workflows and data-intensive tools, facilitate the experts’ contribution to
the lineage at runtime. Ultimately, by encouraging tuning and use of provenance for
rapid feedback, the thesis aims at improving the synergy between different user groups
to increase productivity and understanding of their processes.
We present a model of provenance, called S-PROV, that uses and further extends
PROV and ProvONE. The relationships and properties characterising the workflow’s
abstractions and their concrete executions are re-elaborated to include aspects related
to delegation, distribution and steering of stateful streaming operators. The model is
supported by the Active framework for tuneable and actionable lineage ensuring the
user’s engagement by fostering rapid exploitation. Here, concepts such as provenance
types, configuration and explicit state management allow users to capture complex
provenance scenarios and activate selective controls based on domain and user-defined
metadata. We outline how the traces are recorded in a new comprehensive system,
called S-ProvFlow, enabling different classes of consumers to explore the provenance
data with services and tools for monitoring, in-depth validation and comprehensive
visual-analytics. The work of this thesis will be discussed in the context of an existing
computational framework and the experience matured in implementing provenance-aware
tools for seismology and climate VREs. It will continue to evolve through
newly funded projects, thereby providing generic and user-centred solutions for data-intensive
research
D3.4 Data Lineage Services II
No full textThis report provides an overview of the second and final release of the DARE Lineage Services
icclim: Calculating Climate Indices and Indicators Made Easy
No full textResearchers and end users using climate data face a challenge when they analyze the data they need. Data volumes are increasing very rapidly, and the ability to download all needed data is often no longer possible. Most of the climate analysis tools for research and application needs must use very large datasets, often distributed among several data centres and into a large quantity of files. This is especially true when they are stored in a federated architecture like the ESGF.
One of these tools is icclim (https://github.com/cerfacs-globc/icclim ), a flexible python software package to calculate climate indices and indicators. This tool adhere as much as possible to metadata conventions such as CF, implementing also provenance information. It also aims at providing increasing support for all FAIR aspects. It is designed with performance and optimisation in mind, because the goal is to provide on-demand calculations for users. It provides the implementation of most of the international standard climate indices such as ECAD, ETCCDI, ET-SCI, including the correct methodology for calculating percentile indices using the bootstrapping method. It has been validated against R.Climdex as well (https://cran.r-project.org/web/packages/climdex.pcic/index.html ). The new 5.x version of icclim is now based on functions from the xclim python library, which was inspired by earlier versions of icclim, but using xarray and dask for data access and processing. icclim is also a candidate as the software to calculate climate indices for the C3S toolbox (https://cds.climate.copernicus.eu/cdsapp#!/toolbox ).
icclim is integrated in the IS-ENES C4I 2.0 platform (https://climate4impact.eu/ ), using a Jupyter notebook collection in a SWIRRL environment (Software for Interactive Reproducible Research Labs https://gitlab.com/KNMI-OSS/swirrl ). Having access to this type of analysis tool is very useful, and seamless integration with front-ends like C4I enable the use of those tools by a larger number of researchers and end users.
This project (IS-ENES3) has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement N°824084
<i>Active</i> provenance for Data-Intensive workflows: engaging users and developers
No full textWe present a practical approach for provenance capturing in Data-Intensive workflow systems. It provides contextualisation by recording injected domain metadata with the provenance stream. It offers control over lineage precision, combining automation with specified adaptations. We address provenance tasks such as extraction of domain metadata, injection of custom annotations, accuracy and integration of records from multiple independent workflows running in distributed contexts. To allow such flexibility, we introduce the concepts of programmable Provenance Types and Provenance Configuration. Provenance Types handle domain contextualisation and allow developers to model lineage patterns by re-defining API methods, composing easy-to-use extensions. Provenance Configuration, instead, enables users of a Data-Intensive workflow execution to prepare it for provenance capture, by configuring the attribution of Provenance Types to components and by specifying grouping into semantic clusters. This enables better searches over the lineage records. Provenance Types and Provenance Configuration are demonstrated in a system being used by computational seismologists. It is based on an extended provenance model, S-PROV
S-ProvFlow. Storing and Exploring Lineage Data as a Service
No full textWe present a set of configurable Web service and interactive tools, s-ProvFlow, for managing and exploiting records tracking data lineage during workflow runs. It facilitates detailed analysis of single executions. It helps users manage complex tasks by exposing the relationships between data, people, equipment and workflow runs intended to combine productively. Its logical model extends the PROV standard to precisely record parallel data-streaming applications. Its metadata handling encourages users to capture the application context by specifying how application attributes, often using standard vocabularies, should be added. These metadata records immediately help productivity as the interactive tools support their use in selection and bulk operations. Users rapidly appreciate the power of the encoded semantics as they reap the benefits. This improves the quality of provenance for users and management. Which in turn facilitates analysis of collections of runs, enabling users to manage results and validate procedures. It fosters reuse of data and methods and facilitates diagnostic investigations and optimisations. We present S-ProvFlow’s use by scientists, research engineers and managers as part of the DARE hyper-platform as they create, validate and use their data-driven scientific workflows.Published226-2423T. Fisica dei terremoti e Sorgente Sismica3IT. Calcolo scientificoJCR Journa
Provenance for seismological processing pipelines in a distributed streaming workflow
No full textHarvesting provenance for streaming workflows presents challenges related to the high rate of the updates and a large distribution of the execution, which can be spread across several institutional infrastructures. Moreover, the typically large volume of data produced by each transformation step can not be always stored and preserved efficiently. This can represent an obstacle for the evaluation of the results, for instance, in real-time, suggesting the importance of customisable metadata extraction procedures. In this paper we present our approach to the aforementioned provenance challenges within a use-case driven scenario in the field of seismology, which requires the execution of processing pipelines over a large datastream. In particular, we will discuss the current implementation and the upcoming challenges for an in-worfklow programmatic approach to provenance tracing, building on composite functions, selective recording and domain specific metadata production
Better Tailoring of Climate Information for End Users using Targeted Interfaces and Tools
No full textPresentation delivered at EGU 2022 (Vienna, Austria):
End users of climate change information are relying on climate services and tools in order to produce meaningful information for specific applications. Data volumes as well as the number of datasets are increasing very rapidly, and the ability to select, process and download all needed data is getting complex, technical and very time-consuming. It is especially true since those datasets are often distributed among several data centres and into a large quantity of files.
Several platform are being developed to hide this complexity to users and provide a seamless access to climate data, as well as to provide on-demand data analysis capabilities. We can cite, for example, the Copernicus Data Store (CDS https://cds.climate.copernicus.eu), along with its toolbox to perform online data analysis. Another platform is developed within the H2020 IS-ENES3 project, called climate4impact (C4I 2.0 https://dev.climate4impact.eu ). It is using an enhanced Jupyter-Lab environment called SWIRRL (Software for Interactive Reproducible Research Labs https://gitlab.com/KNMI-OSS/swirrl ) along with a collection of Jupyter notebooks (https://gitlab.com/is-enes-cdi-c4i/notebooks) as useful set of example on how to use the data.
Finally, the portal provides interactive pages for the evaluation of climate models (using ESMValTool) to guide users on selecting climate datasets.
The notebooks that can be executed in C4I, are developed using a very convenient software library, which is made available via SWIRRL, to calculate climate indices and indicators called icclim (v5.0 https://github.com/cerfacs-globc/icclim ). This library, which is also in the process of being integrated into the C3S, is a flexible python software package to calculate climate indices and indicators. This tool adhere as much as possible to metadata conventions such as the Climate & Forecasting Conventions (CF-1.x) as well as the clix-meta (https://github.com/clix-meta) work that is being done in IS-ENES3. Proper provenance information still needs to be added. The ultimate goal is to be as close as possible to all FAIR aspects. icclim is designed with performance and optimisation in mind, because the goal is to provide on-demand calculations for users. It provides the implementation of most of the international standard climate indices such as ECAD, ETCCDI, ET-SCI, including the correct methodology for calculating percentile indices using the bootstrapping method. It has been validated against R.Climdex as well (https://cran.r-project.org/web/packages/climdex.pcic/index.html). This new 5.x version of icclim is based on functions from the xclim (https://github.com/Ouranosinc/xclim) python library, which was inspired by earlier versions of icclim, but using xarray and dask for data access and processing.
In this presentation, the climate4impact 2.0 platform will be described along with the icclim climate indices tool. Important metadata aspects will also be discussed (clix-meta). A few examples using the jupyter notebook collection will be shown
- …
