1,720,989 research outputs found
Ontology Based Semantic Image Interpretation
No full textSemantic image interpretation (SII) leverages Semantic Web ontologies for generating a mathematical structure that describes the content of images. SII algorithms consider the ontologies only in a late phase of the SII process to enrich these structures. In this research proposal we study a well-founded framework that combines logical knowledge with low-level image features in the early phase of SII. The image content is represented with a partial model of an ontology. Each element of the partial model is grounded to a set of segments of the image. Moreover, we propose an approximate algorithm that searches for the most plausible partial model. The comparison of our method with a knowledge-blind baseline shows that the use of ontologies significantly improves the results
Semantic Image Interpretation - Integration of Numerical Data and Logical Knowledge for Cognitive Vision
Get PDFSemantic Image Interpretation (SII) is the process of generating a structured description of the content of an input image. This description is encoded as a labelled direct graph where nodes correspond to objects in the image and edges to semantic relations between objects. Such a detailed structure allows a more accurate searching and retrieval of images. In this thesis, we propose two well-founded methods for SII. Both methods exploit background knowledge, in the form of logical constraints of a knowledge base, about the domain of the images. The first method formalizes the SII as the extraction of a partial model of a knowledge base. Partial models are built with a clustering and reasoning algorithm that considers both low-level and semantic features of images. The second method uses the framework Logic Tensor Networks to build the labelled direct graph of an image. This framework is able to learn from data in presence of the logical constraints of the knowledge base. Therefore, the graph construction is performed by predicting the labels of the nodes and the relations according to the logical constraints and the features of the objects in the image. These methods improve the state-of-the-art by introducing two well-founded methodologies that integrate low-level and semantic features of images with logical knowledge. Indeed, other methods, do not deal with low-level features or use only statistical knowledge coming from training sets or corpora. Moreover, the second method overcomes the performance of the state-of-the-art on the standard task of visual relationship detection
Explainable AI meets persuasiveness: Translating reasoning results into behavioral change advice
Get PDFExplainable AI aims at building intelligent systems that are able to provide a clear, and human understandable, justification of their decisions. This holds for both rule-based and data-driven methods. In management of chronic diseases, the users of such systems are patients that follow strict dietary rules to manage such diseases. After receiving the input of the intake food, the system performs reasoning to understand whether the users follow an unhealthy behavior. Successively, the system has to communicate the results in a clear and effective way, that is, the output message has to persuade users to follow the right dietary rules. In this paper, we address the main challenges to build such systems: (i) the Natural Language Generation of messages that explain the reasoner inconsistency; and, (ii) the effectiveness of such messages at persuading the users. Results prove that the persuasive explanations are able to reduce the unhealthy users’ behaviors
An End-to-End Semantic Platform for Nutritional Diseases Management
Get PDFThe self-management of nutritional diseases requires a system that combines food tracking with the potential risks of food categories on people’s health based on their personal health records (PHRs). The challenges range from the design of an effective food image classification strategy to the development of a full-fledged knowledge-based system. This maps the results of the classification strategy into semantic information that can be exploited for reasoning. However, current works mainly address the single challenges separately without their integration into a whole pipeline. In this paper, we propose a new end-to-end semantic platform where: (i) the classification strategy aims to extract food categories from food pictures; (ii) an ontology is used for detecting the risk factors of food categories for specific diseases; (iii) the Linked Open Data (LOD) Cloud is queried for extracting information concerning related diseases and comorbidities; and, (iv) information from the users’ PHRs are exploited for generating proper personal feedback. Experiments are conducted on a new publicly released dataset. Quantitative and qualitative evaluations, from two living labs, demonstrate the effectiveness and the suitability of the proposed approach
Ontology-Driven Food Category Classification in Images
Get PDFThe self-management of chronic diseases related to dietary habits includes the necessity of tracking what people eat. Most of the approaches proposed in the literature classify food pictures by labels describing the whole recipe. The main drawback of this kind of strategy is that a wrong prediction of the recipe leads to a wrong prediction of any ingredient of such a recipe. In this paper we present a multi-label food classification approach, exploiting deep neural networks, where each food picture is classified with labels describing the food categories of the ingredients in each recipe. The aim of our approach is to support the detection of food categories in order to detect which one might be dangerous for a user affected by chronic disease. Our approach relies on background knowledge where recipes, food categories, and their relatedness with chronic diseases are modeled within a state-of-the-art ontology. Experiments conducted on a new publicly released dataset demonstrated the effectiveness of the proposed approach with respect to state-of-the-art classification strategies
Integration of numeric and symbolic information for semantic image interpretation
No full textSemantic image interpretation (SII) is the process of generating meaningful descriptions of the content of images. Background knowledge (BK), in the form of logical theories, is extremely useful for SII. State-of-the-art algorithms for SII mainly adopt a bottom-up approach, which generates semantic interpretations of images starting from their low-level features. In these approaches BK is used only at a late stage for both enriching the semantic descriptions and improving image retrieval. In this paper, we show how BK plays an important role also during the early phase of SII. To this aim, we propose: (i) a reference framework where a semantic image description is a partial model of the BK. The elements of the partial model are grounded (linked) to a (set of) image segment(s). (ii) A loss function that evaluates how well this partial model fits the picture; (iii) a clustering-based optimization process that searches the partial model that better fits a picture. BK is used to prune branches of the search space that correspond to partial models which are inconsistent with BK. To evaluate our approach, we built a gold standard dataset of 203 pictures annotated with complex objects and their parts. We also evaluated our method on a reference dataset in Computer Vision, namely, the PASCAL-Part dataset. The results are positive. The evaluation assumes a perfect detection of parts. To understand the impact of a realistic (and noisy) part detection on our algorithm, we did a preliminary evaluation by implementing the entire SII pipeline. Part detection is performed by a recent deep learning architecture trained for detecting parts. From a qualitative analysis, it emerges that recognizing complex objects starting from parts in some cases gets better results than detecting complex objects directly
Compensating Supervision Incompleteness with Prior Knowledge in Semantic Image Interpretation
No full textSemantic Image Interpretation is the task of extracting a structured semantic description from images. This requires the detection of visual relationships: triples 〈subject, relation, object〉 describing a semantic relation between a subject and an object. A pure supervised approach to visual relationship detection requires a complete and balanced training set for all the possible combinations of 〈subject, relation, object〉. However, such training sets are not available and would require a prohibitive human effort. This implies the ability of predicting triples which do not appear in the training set. This problem is called zero-shot learning. State-of-the-art approaches to zero-shot learning exploit similarities among relationships in the training set or external linguistic knowledge. In this paper, we perform zero-shot learning by using Logic Tensor Networks, a novel Statistical Relational Learning framework that exploits both the similarities with other seen relationships and background knowledge, expressed with logical constraints between subjects, relations and objects. The experiments on the Visual Relationship Dataset show that the use of logical constraints outperforms the current methods. This implies that background knowledge can be used to alleviate the incompleteness of training sets
Explaining reasoning algorithms with persuasiveness
Get PDFExplainable AI aims at building intelligent systems that are able to provide a clear, and human understandable, justification of their decisions. This holds for both rule-based and data-driven methods. In management of chronic diseases, the users of such systems are patients that follow strict dietary rules to manage such diseases. After receiving the input of the intake food, the system performs reasoning to understand whether the users follow an unhealthy behaviour. Successively, the system has to communicate the results in a clear and effective way, that is, the output message has to persuade users to follow the right dietary rules. In this paper, we address the main challenges to build such systems: i) the natural language generation of messages that explain the reasoner inconsistency; ii) the effectiveness of such messages at persuading the users. Results prove that the persuasive explanations are able to reduce the unhealthy users' behaviours
Learning and Reasoning in Logic Tensor Networks: Theory and Application to Semantic Image Interpretation
No full textThis paper presents a revision of Real Logic and its implementation with Logic Tensor Networks to address the problem of Semantic Image Interpretation. Real Logic is a framework where learning from numerical data and logical reasoning are integrated using first order logic syntax. The symbols of the signature of Real Logic are interpreted in the data-space, i.e, on the domain of real numbers. The integration of learning and reasoning obtained in Real Logic allows us to formalize learning as approximate satisfiability in the presence of logical constraints, and to perform inference on symbolic and numerical data. After introducing a refined version of the formalism, we describe its implementation into Logic Tensor Networks which uses deep learning within Google's Tensorflow. We evaluate LTN on the task of classifying objects and their parts in images, where we combine state-of-the-art-object detectors with a part-of ontology. LTN outperforms the state-of-the-art on object classification, and improves the performances on part-of relation detection with respect to a rule-based baseline
Mixing Low-Level and Semantic Features for Image Interpretation
No full textSemantic Content-Based Image Retrieval (SCBIR) allows users to retrieve images via complex expressions of some ontological language describing a domain of interest. SCBIR adds some flexibility to the state-of-the-art methods for image retrieval, which support query either by keywords or by image examples. The price for this additional flexibility is the generation of a semantically rich description of the image content reflecting the ontology constraints. Generating these semantic interpretations is an open research problem. This paper contributes to this research line by proposing an approach for SCBIR based on the somehow natural idea that the interpretation of a picture is an (onto)logical model of an ontology that describes the domain of the picture. We implement this idea in an unsupervised method that jointly exploits the ontological constraints and the low-level features of the image. The preliminary evaluation, presented in the paper, shows promising results
- …
