42 research outputs found
RevOnt: Reverse engineering of competency questions from knowledge graphs via language models
No full textThe process of developing ontologies – a formal, explicit specification of a shared conceptualisation – is addressed by well-known methodologies. As for any engineering development, its fundamental basis is the collection of requirements, which includes the elicitation of competency questions. Competency questions are defined through interacting with domain and application experts or by investigating existing datasets that may be used to populate the ontology i.e. its knowledge graph. The rise in popularity and accessibility of knowledge graphs provides an opportunity to support this phase with automatic tools. In this work, we explore the possibility of extracting competency questions from a knowledge graph. This reverses the traditional workflow in which knowledge graphs are built from ontologies, which in turn are engineered from competency questions. We describe in detail RevOnt, an approach that extracts and abstracts triples from a knowledge graph, generates questions based on triple verbalisations, and filters the resulting questions to yield a meaningful set of competency questions; the WDV dataset. This approach is implemented utilising the Wikidata knowledge graph as a use case, and contributes a set of core competency questions from 20 domains present in the WDV dataset. To evaluate RevOnt, we contribute a new dataset of manually-annotated high-quality competency questions, and compare the extracted competency questions by calculating their BLEU score against the human references. The results for the abstraction and question generation components of the approach show good to high quality. Meanwhile, the accuracy of the filtering component is above 86%, which is comparable to the state-of-the-art classifications
Refining Statistical Data on the Web
No full textHarmelen, F.A.H. van [Promotor]Schlobach, K.S. [Copromotor]Scharnhorst, A. [Copromotor
Towards Explainable Automated Knowledge Engineering with Human-in-the-loop
No full textKnowledge graphs are important in human-centered AI as they provide large labeled machine learning datasets, enhance retrieval-augmented generation, and generate explanations. However, knowledge graph construction has evolved into a complex, semi-automatic process that increasingly relies on black-box deep learning models and heterogeneous data sources to scale. The knowledge graph lifecycle is not transparent, accountability is limited, and there are no accounts of, or indeed methods to determine, how fair a knowledge graph is in downstream applications. Knowledge graphs are thus at odds with AI regulation, for instance, the EU's AI Act, and with ongoing efforts elsewhere in AI to audit and debias data and algorithms. This paper reports on work towards designing explainable (XAI) knowledge graph construction pipelines with humans in-the-loop and discusses research topics in this area. Our work is based on a systematic literature review, in which we study tasks in knowledge graph construction that are often automated, as well as common methods to explain how they work and their outcomes, and an interview study with 13 experts from the knowledge engineering community. To analyze the related literature, we introduce use cases, their related goals for XAI methods in knowledge graph construction, and the gaps in each use case. To gain an understanding of the role of XAI models in practical scenarios, and reveal the requirements for improving the current XAI methods, we designed interview questions covering broad transparency and explainability topics, along with example discussion sessions using examples from the literature review. From practical knowledge engineering experience, we collect requirements for designing XAI methods, propose design blueprints, and outline directions for future research: (i) tasks in knowledge graph construction where manual input remains essential and where AI assistance could be beneficial; (ii) integrating XAI methods into established knowledge engineering practices to improve stakeholder experience; (iii) the need to evaluate how effective explanations genuinely are making human-machine collaboration in knowledge graph construction more trustworthy; (iv) adapting explanations for multiple use cases; and (v) verifying and applying the XAI design blueprint in practical settings
Towards Explainable Automated Knowledge Engineering with Human-in-the-loop
No full textKnowledge graphs are important in human-centered AI as they provide large labeled machine learning datasets, enhance retrieval-augmented generation, and generate explanations. However, knowledge graph construction has evolved into a complex, semi-automatic process that increasingly relies on black-box deep learning models and heterogeneous data sources to scale. The knowledge graph lifecycle is not transparent, accountability is limited, and there are no accounts of, or indeed methods to determine, how fair a knowledge graph is in downstream applications. Knowledge graphs are thus at odds with AI regulation, for instance, the EU's AI Act, and with ongoing efforts elsewhere in AI to audit and debias data and algorithms. This paper reports on work towards designing explainable (XAI) knowledge graph construction pipelines with humans in-the-loop and discusses research topics in this area. Our work is based on a systematic literature review, in which we study tasks in knowledge graph construction that are often automated, as well as common methods to explain how they work and their outcomes, and an interview study with 13 experts from the knowledge engineering community. To analyze the related literature, we introduce use cases, their related goals for XAI methods in knowledge graph construction, and the gaps in each use case. To gain an understanding of the role of XAI models in practical scenarios, and reveal the requirements for improving the current XAI methods, we designed interview questions covering broad transparency and explainability topics, along with example discussion sessions using examples from the literature review. From practical knowledge engineering experience, we collect requirements for designing XAI methods, propose design blueprints, and outline directions for future research: (i) tasks in knowledge graph construction where manual input remains essential and where AI assistance could be beneficial; (ii) integrating XAI methods into established knowledge engineering practices to improve stakeholder experience; (iii) the need to evaluate how effective explanations genuinely are making human-machine collaboration in knowledge graph construction more trustworthy; (iv) adapting explanations for multiple use cases; and (v) verifying and applying the XAI design blueprint in practical settings
OntoScope: Using a Divergent-Convergent Interaction Framework to Support LLM-based Ontology Scoping
No full textAn ontology is a formal, explicit specification of a shared conceptualization that, with problem‑solving and reasoning methods, supports efficient semantic technology development. In ontology engineering, Competency Questions (CQs) capture functional requirements that define an ontology's application domain. Auditing this domain scope with CQs is challenging because in nature, there are no clear domain boundaries, and ontology engineers must then decide which subdomains to cover (horizontal coverage) and how much detail to model (vertical granularity) in an ontology. LLM‑based systems can generate many candidate CQs to guide these decisions, but current tools underuse this potential: they lack support for users' divergent (lateral) and convergent (vertical) thinking in a visualized CQs space organized by coverage and granularity. As a result, users struggle to systematically decide which CQs to adopt, discard, or refine. We propose an interaction framework that fills this gap, demonstrated through OntoScope, an LLM‑based interactive system, and a user study with 15 ontology engineers. To our knowledge, this is the first validated interaction framework with an LLM‑based system that helps ontology engineers audit domain boundaries and unifies fragmented, expert‑driven ontology scoping practices into a coherent, accessible approach. More broadly, it shows how LLM‑based systems can transparently and accountably support a wider range of knowledge‑intensive tasks
OntoScope: Using a Divergent-Convergent Interaction Framework to Support LLM-based Ontology Scoping
No full textAn ontology is a formal, explicit specification of a shared conceptualization that, with problem‑solving and reasoning methods, supports efficient semantic technology development. In ontology engineering, Competency Questions (CQs) capture functional requirements that define an ontology's application domain. Auditing this domain scope with CQs is challenging because in nature, there are no clear domain boundaries, and ontology engineers must then decide which subdomains to cover (horizontal coverage) and how much detail to model (vertical granularity) in an ontology. LLM‑based systems can generate many candidate CQs to guide these decisions, but current tools underuse this potential: they lack support for users' divergent (lateral) and convergent (vertical) thinking in a visualized CQs space organized by coverage and granularity. As a result, users struggle to systematically decide which CQs to adopt, discard, or refine. We propose an interaction framework that fills this gap, demonstrated through OntoScope, an LLM‑based interactive system, and a user study with 15 ontology engineers. To our knowledge, this is the first validated interaction framework with an LLM‑based system that helps ontology engineers audit domain boundaries and unifies fragmented, expert‑driven ontology scoping practices into a coherent, accessible approach. More broadly, it shows how LLM‑based systems can transparently and accountably support a wider range of knowledge‑intensive tasks
Linking Dutch civil certificates
No full textFinding and linking different appearances of the same entity in an open Web setting is one of the primary challenges of the Semantic Web. In social and economic history, record linkage has dealt with this problem for a long time, linking historical individual records at a local database level. With the advent of semantic technologies, Knowledge Graphs containing these records have been published, raising the need for large-scale linking techniques that consider the particularities of historical individual linking. In this paper we focus on our current investigation of such techniques to link the Dutch civil certificates in the LINKS/CLARIAH project. We describe the production of the LINKS Knowledge Graph, and we show its potential at answering domain research questions through its large number of owl:sameAs links.</p
Explanation of Link Predictions on Knowledge Graphs via Levelwise Filtering and Graph Summarization
No full textLink Prediction methods aim at predicting missing facts in Knowledge Graphs (KGs) as they are inherently incomplete. Several methods rely on Knowledge Graph Embeddings, which are numerical representations of elements in the Knowledge Graph. Embeddings are effective and scalable for large KGs; however, they lack explainability.Kelpie is a recent and versatile framework that provides post-hoc explanations for predictions based on embeddings by revealing the facts that enabled them. Problems have been recognized, however, with filtering potential explanations and dealing with an overload of candidates. We aim at enhancing Kelpie by targeting three goals: reducing the number of candidates, producing explanations at different levels of detail, and improving the effectiveness of the explanations. To accomplish them, we adopt a semantic similarity measure to enhance the filtering of potential explanations, and we focus on a condensed representation of the search space in the form of a quotient graph based on entity types. Three quotient formulations of different granularity are considered to reduce the risk of losing valuable information. We conduct a quantitative and qualitative experimental evaluation of the proposed solutions, using Kelpie as a baseline
