1,720,968 research outputs found
Automated multimodal sensemaking: Ontology-based integration of linguistic frames and visual data
No full textFrame evocation from visual data is an essential process for multimodal sensemaking, due to the multimodal abstraction provided by frame semantics. However, there is a scarcity of data-driven approaches and tools to automate it. We propose a novel approach for explainable automated multimodal sensemaking by linking linguistic frames to their physical visual occurrences, using ontology-based knowledge engineering techniques. We pair the evocation of linguistic frames from text to visual data as “framal visual manifestations”. We present a deep ontological analysis of the implicit data model of the Visual Genome image dataset, and its formalization in the novel Visual Sense Ontology (VSO). To enhance the multimodal data from this dataset, we introduce a framal knowledge expansion pipeline that extracts and connects linguistic frames – including values and emotions – to images, using multiple linguistic resources for disambiguation. It then introduces the Visual Sense Knowledge Graph (VSKG), a novel resource. VSKG is a queryable knowledge graph that enhances the accessibility and comprehensibility of Visual Genome's multimodal data, based on SPARQL queries. VSKG includes frame visual evocation data, enabling more advanced forms of explicit reasoning, analysis and sensemaking. Our work represents a significant advancement in the automation of frame evocation and multimodal sense-making, performed in a fully interpretable and transparent way, with potential applications in various fields, including the fields of knowledge representation, computer vision, and natural language processing
Recommended from our members
Reconciliation in Homo sapiens : Behavioral Perspectives on the Human Post-Conflict Period
No full textEvolutionarily, social living can confer both fitness benefits and costs. In order to reap the benefits—such as access to valued resources and protection—social animals often have to overcome the disruptive costs of conflict, aggression, and competition by maintaining social bonds. Within primatology, the label “reconciliation” as a conflict-resolution tactic was first employed in 1979 when de Waal and Roosmalen noticed that, in chimpanzees, former opponents were more likely to interact peacefully in the minutes that followed conflicts than at other times. Since then, systematic study of post-conflict affiliation compared to control periods has taken place in a wide range of species, but hardly at all in human adults. In this study, same-sex dyads of young adult friends participated in a standardized conflict procedure that included relaxation periods before and after an intense competition. From the procedure, video data was coded to quantify the duration of selected behaviors. Data for seven behaviors was analyzed to investigate the effect of condition (pre- or post-conflict), gender, and post-conflict status (winner or loser) on behavior durations. Results show an increase in anxiety-related behavior and in human affiliative behaviors—such as talking, laughing, and looking—in the post-conflict period. The results in this study signal that, immediately after conflicts, human adults naturally behave in ways that could smooth interaction and increase affiliation. This study provides concrete evidence for behavioral continuity between humans and other primates regarding an increase in both anxiety markers and species-specific forms of affiliation in the post-conflict context
Going Beyond Counting First Authors in Author Co-citation Analysis
Get PDFThe present study examines one of the fundamental aspects of author co-citation analysis (ACA) - the way co-citation
counts are defined. Co-citation counting provides the data on which all subsequent statistical analyses and mappings
are based, and we compare ACA results based on two different types of co-citation counting - the traditional type that
only counts the first one among a cited work's authors on the one hand and a non-traditional type that takes into
account the first 5 authors of a cited work on the other hand. Results indicate that the picture produced through this non-traditional author co-citation counting contains more coherent author groups and is therefore considerably clearer. However, this picture represents fewer specialties in the research field being studied than that produced through the traditional first-author co-citation counting when the same number of top-ranked authors is selected and analyzed. Reasons for these effects are discussed
Variations on the Author
Get PDF“Variations on the Author” discusses two of Eduardo Coutinho’s recent films (Um Dia na Vida, from 2010, and Últimas Conversas, posthumously released in 2015) and their contribution to the general question of documentary authorship. The director’s filmography is characterized by a consistent yet self-effacing form of authorial self-inscription: Coutinho often features as an interviewer that rather than express opinions propels discourses; an interviewer that is good at listening. This mode of self-inscription characterizes him as an author who is not expressive but who is nonetheless markedly present on the screen. In Um Dia na Vida, however, Coutinho is completely absent form the image, while Últimas Conversas, on the contrary, includes a confessional prologue that moves the director from the margins to the center of his films. This article examines the ways in which these works stand out in the filmography of a director who offers new insights into the notion of cinematic authorship
Situated Ground Truths: Enhancing Bias-Aware AI by Situating Data Labels with SituAnnotate
Get PDFIn the contemporary world of AI and data-driven applications, supervised
machines often derive their understanding, which they mimic and reproduce,
through annotations--typically conveyed in the form of words or labels.
However, such annotations are often divorced from or lack contextual
information, and as such hold the potential to inadvertently introduce biases
when subsequently used for training. This paper introduces SituAnnotate, a
novel ontology explicitly crafted for 'situated grounding,' aiming to anchor
the ground truth data employed in training AI systems within the contextual and
culturally-bound situations from which those ground truths emerge. SituAnnotate
offers an ontology-based approach to structured and context-aware data
annotation, addressing potential bias issues associated with isolated
annotations. Its representational power encompasses situational context,
including annotator details, timing, location, remuneration schemes, annotation
roles, and more, ensuring semantic richness. Aligned with the foundational
Dolce Ultralight ontology, it provides a robust and consistent framework for
knowledge representation. As a method to create, query, and compare label-based
datasets, SituAnnotate empowers downstream AI systems to undergo training with
explicit consideration of context and cultural bias, laying the groundwork for
enhanced system interpretability and adaptability, and enabling AI models to
align with a multitude of cultural contexts and viewpoints.Comment: Author preprin
Appropriate Similarity Measures for Author Cocitation Analysis
Get PDFWe provide a number of new insights into the methodological discussion about author cocitation analysis. We first argue that the use of the Pearson correlation for measuring the similarity between authors’ cocitation profiles is not very satisfactory. We then discuss what kind of similarity measures may be used as an alternative to the Pearson correlation. We consider three similarity measures in particular. One is the well-known cosine. The other two similarity measures have not been used before in the bibliometric literature. Finally, we show by means of an example that our findings have a high practical relevance.information science;Pearson correlation;cosine;similarity measure;author cocitation analysis
Mind the gaps: cognitive-inspired AI for high-level visual sensemaking. Towards abstract concept image classification
Get PDFThe abundance of visual data and the push for robust AI are driving the need for automated visual sensemaking. Computer Vision (CV) faces growing demand for models that can discern not only what images "represent," but also what they "evoke." This is a demand for tools mimicking human perception at a high semantic level, categorizing images based on concepts like freedom, danger, or safety. However, automating this process is challenging due to entropy, scarcity, subjectivity, and ethical considerations. These challenges not only impact performance but also underscore the critical need for interoperability. This dissertation focuses on abstract concept-based (AC) image classification, guided by three technical principles: situated grounding, performance enhancement, and interpretability. We introduce ART-stract, a novel dataset of cultural images annotated with ACs, serving as the foundation for a series of experiments across four key domains: assessing the effectiveness of the end-to-end DL paradigm, exploring cognitive-inspired semantic intermediaries, incorporating cultural and commonsense aspects, and neuro-symbolic integration of sensory-perceptual data with cognitive-based knowledge. Our results demonstrate that integrating CV approaches with semantic technologies yields methods that surpass the current state of the art in AC image classification, outperforming the end-to-end deep vision paradigm. The results emphasize the role semantic technologies can play in developing both effective and interpretable systems, through the capturing, situating, and reasoning over knowledge related to visual data. Furthermore, this dissertation explores the complex interplay between technical and socio-technical factors. By merging technical expertise with an understanding of human and societal aspects, we advocate for responsible labeling and training practices in visual media. These insights and techniques not only advance efforts in CV and explainable artificial intelligence but also propel us toward an era of AI development that harmonizes technical prowess with deep awareness of its human and societal implications
BEYOND STATIC COLORS: AN INTERACTIVE PARTICIPATORY DESIGN PERSPECTIVE ON COLOR-CENTRIC EXPERIENCES
No full textBy combining cognitive, sensorial, historical, and artistic aspects into one experience, digital interactive technologies have afforded new ways to perceive, preserve, curate, exhibit, and access cultural objects. However, there is a critical lack of frameworks and tools designed specifically for colored cultural artifacts–cultural items for which color is a key means of conveying the creative message. While colored artifacts are a priority for Conservation Science, due to their fragility and to the complexity of recreating original appearances, in this article we argue that the conservation of colored artifacts is not merely a matter of scientific studies, analysis and static preservation. Instead, we argue for holistic conservation including the valorization of the social dimensions of color, including for civic engagement. The work first investigates the types of data and knowledge that Conservation Science produces regarding colored collections which specifically consider the social dimension of color. We then research the relational ties between humans and colored cultural artifacts, proposing ways that caring attitudes can be triggered and maintained. We finally survey previous color-centric approaches to such artifacts with digital technologies in an interactive media participatory design perspective. We conclude with lessons learned and further directions, including novel research questions and ideas for future user experiences.Pandiani, Delfina Sol Martinez, and Sofia Pescarin. "BEYOND STATIC COLORS: AN INTERACTIVE PARTICIPATORY DESIGN PERSPECTIVE ON COLOR-CENTRIC EXPERIENCES." International Journal of Conservation Science 13 (2022): 1691-1706
- …
