1,720,968 research outputs found
GraphCLIP: Image-graph contrastive learning for multimodal artwork classification
No full textWe present GraphCLIP, a novel contrastive learning framework for multimodal artwork classification that integrates visual and contextual information to improve predictive accuracy and interpretability. Traditional computer vision methods often fall short in visual arts, where context is crucial. GraphCLIP leverages image data and a Knowledge Graph to extract features from both perspectives. Evaluated on the ArtGraph dataset, with over 100,000 artworks in 32 styles and 18 genres, GraphCLIP outperforms existing models in single-task (up to +8% in F1-score) and multi-task settings (up to +6%), demonstrating robustness even with unseen classes. Additionally, visual and contextual qualitative explanations enhance model transparency. The versatility of GraphCLIP extends beyond art classification: its methodology can be adapted to other domains where integrating diverse data types is essential. (The code is publicly available at: https://github.com/CILAB-ArtGraph/graphclip.git.
Recognizing the Style, Genre, and Emotion of a Work of Art Through Visual and Knowledge Graph Embeddings
No full textRecognizing attributes of unknown artworks relies on more than visual information: prior knowledge and emotional context can play a crucial role. Building an AI system mimicking this perception requires a multi-modal model integrating computer vision and contextual factors. In this paper, we propose a new model that uses vision transformers and graph attention networks to learn new artworks’ visual and contextual features and predict their style, genre, and emotion. Contextual features are acquired from an extended version of our ArtGraph knowledge graph, enriched with emotion information from the ArtEmis dataset. Our inductive end-to-end multi-task architecture enables real-time execution and resilience to graph evolutions. Combining computer vision and knowledge graphs could facilitate a deeper understanding of the fine arts, bridging the gap between computer science and the humanities (The new version of the graph is available at https://doi.org/10.5281/zenodo.8172374, while the code is available at https://github.com/CILAB-ArtGraph/multi-modal-end-to-end-art-classifier)
Multimodal Artwork Topic Modeling via Fine-Tuned CLIP and Knowledge-Driven Prompts
No full textWe propose a novel multimodal topic modeling framework to extract and explain latent themes in extensive collections of digitized artworks. Our approach leverages CLIP's contrastive pre-training to encode images and textual metadata into a shared semantic space. We fine-tune CLIP on a domain-specific dataset built from ArtGraph, an art-domain knowledge graph containing over 100k artworks enriched with curated metadata. Using the resulting multimodal embeddings, we perform clustering to uncover latent visual topics and associate each cluster with descriptive terms via cosine similarity with templated textual prompts. Finally, to further interpret the discovered topics, we employ LLaVA to generate textual summaries based on representative images. Our framework demonstrates promising performance in terms of topic coherence and diversity, evaluated through both visual and textual metrics. The method is unsupervised, easily adaptable, and provides interpretable outputs, making it suitable for applications in digital humanities, cultural heritage analysis, and content-based art retrieval
Unveiling Visual Features in Artwork Classification: Towards Explainable Vision Transformers in the Arts
No full textRecent advances in deep learning have enabled accurate artwork classification using models such as Vision Transformers (ViTs). However, interpreting the internal mechanisms behind such decisions remains challenging, especially in the abstract and symbolic domain of visual arts. We propose an interpretability framework that combines feature visualization via activation maximization with natural language grounding through a Multimodal Large Language Model. Our method extracts class-specific visual patterns learned by ViTs, synthesizes prototype images that activate key features, and generates human-readable descriptions. Applied to a large-scale art dataset, the approach reveals that ViTs attend to subtle and abstract cues—such as texture, shape, and composition—differently from natural image tasks. The resulting visual and textual explanations offer valuable insight into model behavior and move toward more transparent, human-aligned AI systems for art analysis
Going Beyond Counting First Authors in Author Co-citation Analysis
Full text linkThe present study examines one of the fundamental aspects of author co-citation analysis (ACA) - the way co-citation
counts are defined. Co-citation counting provides the data on which all subsequent statistical analyses and mappings
are based, and we compare ACA results based on two different types of co-citation counting - the traditional type that
only counts the first one among a cited work's authors on the one hand and a non-traditional type that takes into
account the first 5 authors of a cited work on the other hand. Results indicate that the picture produced through this non-traditional author co-citation counting contains more coherent author groups and is therefore considerably clearer. However, this picture represents fewer specialties in the research field being studied than that produced through the traditional first-author co-citation counting when the same number of top-ranked authors is selected and analyzed. Reasons for these effects are discussed
Variations on the Author
Full text link“Variations on the Author” discusses two of Eduardo Coutinho’s recent films (Um Dia na Vida, from 2010, and Últimas Conversas, posthumously released in 2015) and their contribution to the general question of documentary authorship. The director’s filmography is characterized by a consistent yet self-effacing form of authorial self-inscription: Coutinho often features as an interviewer that rather than express opinions propels discourses; an interviewer that is good at listening. This mode of self-inscription characterizes him as an author who is not expressive but who is nonetheless markedly present on the screen. In Um Dia na Vida, however, Coutinho is completely absent form the image, while Últimas Conversas, on the contrary, includes a confessional prologue that moves the director from the margins to the center of his films. This article examines the ways in which these works stand out in the filmography of a director who offers new insights into the notion of cinematic authorship
Appropriate Similarity Measures for Author Cocitation Analysis
Full text linkWe provide a number of new insights into the methodological discussion about author cocitation analysis. We first argue that the use of the Pearson correlation for measuring the similarity between authors’ cocitation profiles is not very satisfactory. We then discuss what kind of similarity measures may be used as an alternative to the Pearson correlation. We consider three similarity measures in particular. One is the well-known cosine. The other two similarity measures have not been used before in the bibliometric literature. Finally, we show by means of an example that our findings have a high practical relevance.information science;Pearson correlation;cosine;similarity measure;author cocitation analysis
Dispelling the Myths Behind First-author Citation Counts
Full text linkWe conducted a full-scale evaluative citation analysis study of scholars in the XML research field to explore just how different from each other author rankings resulting from different citation counting methods actually are, and to demonstrate the capability of emerging data and tools on the Web in supporting more realistic citation counting methods. Our results contest some common arguments for the continued
use of first-author citation counts in the evaluation of scholars, such as high correlations between author rankings by first-author citation counts and other citation
counting methods, and high costs of using more realistic citation counting methods that are not well-supported by the ISI databases. It is argued that increasingly available digital full text research papers make it possible for citation analysis studies to go beyond what the ISI databases have directly supported and to employ more
sophisticated methods
- …
