4,799 research outputs found
Segmentación semántica multiclase en la digitalización del patrimonio mueble utilizando técnicas de aprendizaje profundo
No full text[EN] Digitisation processes of movable heritage are becoming increasingly popular to document the artworks stored in our museums. An increasing number of strategies for the three-dimensional (3D) acquisition and modelling of these invaluable assets have been developed in the last few years, to efficiently respond to this documentation need and contribute to deepening the knowledge of the masterpieces investigated constantly by researchers operating in many fieldworks. Nowadays, one of the most effective solutions is represented by the development of image-based techniques, usually connected to a Structure-from-Motion (SfM) photogrammetric approach. However, while the acquisition of the images is relatively rapid, it is the processes connected to the data processing that are very time-consuming and require substantial manual involvement of the operator. The development of deep learning-based strategies can be an effective solution to enhance the level of automatism. In the case of the current research, which has been carried out in the framework of the digitisation of a collection of wooden maquettes stored in the ‘Museo Egizio di Torino’ using a photogrammetric approach, an automatic masking strategy using deep learning techniques is proposed, to increase the level of automatism and therefore, optimise the photogrammetric pipeline. Starting from a manually annotated dataset a neural network has been trained to automatically perform a semantic classification with the aim to isolate the maquettes from the background. The proposed methodology has allowed obtaining automatically segmented masks with a high degree of accuracy. The followed workflow is described (as regards acquisition strategies, dataset processing, and neural network training), and the accuracy of the results is evaluated and discussed. In addition, the possibility of performing a multiclass segmentation on the digital images to recognise different categories of objects in the images and define a semantic hierarchy is proposed to perform automatic classification of different elements in the acquired images.[ES] Los procesos de digitalización del patrimonio mueble son cada vez más populares para documentar las obras de arte almacenadas en nuestros museos. En los últimos años se han desarrollado un número creciente de estrategias de adquisición y modelado tridimensional (3D) de estos activos de valor incalculable, que responden de manera eficiente a esta necesidad de documentación y contribuyen a profundizar en el conocimiento de las obras maestras investigadas constantemente por investigadores que operan en muchos trabajos de campo. Hoy en día, una de las soluciones más efectivas está relacionada con el desarrollo de técnicas basadas en imágenes, generalmente conectadas a un enfoque fotogramétrico de estructura-y-movimiento (SfM). Sin embargo, si bien la adquisición de las imágenes es relativamente rápida, son los procesos relacionados con el procesamiento de los datos los que consumen mucho tiempo y requieren una participación manual sustancial del operador. El desarrollo de estrategias basadas en el aprendizaje profundo puede ser una solución eficaz para mejorar el nivel de automatismo. En el caso de la presente investigación, que se ha llevado a cabo en el marco de la digitalización de una colección de maquetas de madera almacenadas en el 'Museo Egizio di Torino' mediante un enfoque fotogramétrico, se propone una estrategia de enmascaramiento automático mediante técnicas de aprendizaje profundo, que incrementa el nivel de automatismo y por tanto optimiza el flujo fotogramétrico. A partir de un conjunto de datos anotados manualmente, se ha entrenado una red neuronal que realiza automáticamente una clasificación semántica con el objetivo de aislar las maquetas del fondo. La metodología propuesta ha permitido obtener más caras segmentadas automáticamente con alto grado de precisión. Se describe el flujo de trabajo seguido (en cuanto a estrategias de toma, procesamiento del conjuntos de datos y entrenamiento de las redes neuronales), y se evalúa y discute la precisión de los resultados. Además, se propone la posibilidad de realizar una segmentación multiclase sobre las imágenes digitales que permitan reconocer diferentes categorías de objetos en las imágenes y definir una jerarquía semántica que clasifique automáticamente diferentes elementos en la toma de las imágenes.The authors thank Volta® A.I. (and in particular Silvio Revelli) for the contribution to this work and for providing high-end hardware for neural network training.
In addition, they would like to thank Alessia Fassone of Museo Egizio di Torino and all the people involved in the B.A.C.K. TO T.H.E. F.U.T.U.RE. project (in particular, Fulvio Rinaudo, who coordinated the Geomatic team).
Finally, they wish to express their gratitude to Nannina Spanò and Filiberto Chiabrando for the helpful confrontation during the presented research.Patrucco, G.; Setragno, F. (2021). Multiclass semantic segmentation for digitisation of movable heritage using deep learning techniques. Virtual Archaeology Review. 12(25):85-98. https://doi.org/10.4995/var.2021.15329OJS85981225Adami, A., Balletti, C., Fassi, F., Fregonese, L., Guerra, F., Taffurelli, L., Vernier, P. (2015). The bust of Francesco II Gonzaga: From digital documentation to 3D printing. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, II-5/W3, 9-15. https://doi.org/10.5194/isprsannals-II-5-W3-9-2015Badrinarayanan, V., Kendall, A., Cipolla, R. (2017). Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(12), 2481-2495. https://doi.org/10.1109/TPAMI.2016.2644615Balletti, C., Ballarin, M., & Guerra, F. (2017). 3D printing: state of the art and future perspectives. Journal of Cultural Heritage, 26,172-182. https://doi.org/10.1016/j.culher.2017.02.010Balletti, C., & Ballarin, M. (2019). An application of integrated 3D technologies for replicas in Cultural Heritage. International Journal of Geo-Information, 8(6), 285. https://doi.org/10.3390/ijgi8060285Barbieri, L., Bruno, F., & Muzzupappa, M. (2018). User-centered design of a virtual reality exhibit for archaeological museums. International Journal on Interactive Design and Manufacturing (IJIDeM), 12, 561-571. https://doi.org/10.1007/s12008-017-0414-zCaruana, R., Lawrence, S., & Giles, C. L. (2001). Overfitting in neural nets: Backpropagation, conjugate gradient, and early stopping. Advances in Neural Information Processing Systems (pp. 402-408). https://doi.org/10.1109/IJCNN.2000.857823Cermelli, F., Mancini, M., Bulo, S. R., Ricci, E., & Caputo, B. (2020). Modeling the background for incremental learning in semantic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 9233-9242. https://doi.org/10.1109/CVPR42600.2020.00925Condorelli, F., Rinaudo, F., Salvadore, F., & Tagliaventi, S. (2020). A neural network approach to detecting lost heritage in historical video. International Journal of Geo-Information, 9(5), 297. https://doi.org/10.3390/ijgi9050297Chiabrando, F., Sammartano, G., Spanò, A., & Spreafico, A. (2019). Hybrid 3D models: When Geomatics innovations meet extensive built heritage complexes. International Journal of Geo-Information, 8(3), 124. https://doi.org/10.3390/ijgi8030124Dall'Asta, E., Bruno, N., Bigliardi, G., Zerbi, A., & Roncella, R.(2016). Photogrammetric techniques for promotion of archaeological Heritage: the Archaeological Museum of Parma (Italy). International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLI-B5, 243-250. https://doi.org/10.5194/isprs-archives-XLI-B5-243-2016Felicetti, A., Paolanti, M., Zingaretti, P., Pierdicca, R., & Malinverni, E. S. (2020). Mo.Se.: Mosaic image segmentation based on deep cascading learning. Virtual Archaeology Review, 12(24), 25-38. https://doi.org/10.4995/var.2021.14179Fiorucci, M., Khoroshiltseva, M., Pontil, M., Traviglia, A., Del Bue, A., & James, S. (2020). Machine Learning for Cultural Heritage: A Survey. Pattern Recognition Letters, 133, 102-108. https://doi.org/10.1016/j.patrec.2020.02.017Garcia-Garcia, A., Orts-Escolano, S., Oprea, S., Villena-Martinez, V., & Garcia-Rodriguez, J. (2017). A survey on deep learning techniques for image and video semantic segmentation. Applied Soft Computing, 70, 41-65. https://doi.org/10.1016/j.asoc.2018.05.018George, D., Xie, X., & Tam, G. K. (2018). 3D mesh segmentation via multi-branch 1D convolutional neural networks. Graphical Models, 96, 1-10. https://doi.org/10.1016/j.gmod.2018.01.001Giuffrida, D., Mollica Nardo, V., Giacobello, F., Adinolfi, O., Mastelloni, M. A., Toscano, G., & Ponterio, R. S. (2019). Combined 3D surveying and Raman Spectroscopy Techniques on artifacts preserved at Archaeological Musem of Lipari. Heritage, 2(3), 2017-2027. https://doi.org/10.3390/heritage2030121Grilli, E., Farella, E. M., Torresani, A., & Remondino, F. (2019). Geometric features analysis for the classification of Cultural Heritage point clouds. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLII-2/W15, 541-548. https://doi.org/10.5194/isprs-archives-XLII-2-W15-541-2019Grilli, E., Özdemir, E., & Remondino, F. (2019). Application of machine and deep learning strategies for the classification of Heritage point clouds. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLII-4/W18, 447-454. https://doi.org/10.5194/isprs-archives-XLII-4-W18-447-2019Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press.Gu, J., Wang, Z., Kuen, J., Ma., L., Shahroudy, A., Shuai, B., & Chen., T. (2018). Recent advances in convolutional neural networks. Pattern Recognition, 77, 354-377. https://doi.org/10.1016/j.patcog.2017.10.013Guidi, G., Malik, U. S., Frischer, B., Barandoni, C., & Paolucci, F. (2017). The Indiana University-Uffizi project: Metrological challenges and workflow for massive 3D digitization of sculptures. 23rd International Conference on Virtual System & Multimedia (VSMM), 1-8. https://doi.org/10.1109/VSMM.2017.8346268He, T., Shen, C., Tian, Z., Gong, D., Sun, C., & Yan, Y. (2019). Knowledge adaptation for efficient semantic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 578-587. https://doi.org/10.1109/CVPR.2019.00067Jégou, S., Drozdzal, M., Vazquez, D., Romero, A., & Bengio, Y. (2017). The one hundred layers tiramisu: Fully convolutional densenets for semantic segmentation. Proceedings of the IEEE conference on computer vision and pattern recognition workshops (pp. 11-19). https://doi.org/10.1109/CVPRW.2017.156Kersten, T. P., Tschirschwitz, F., & Deggim, S. (2017). Development of a virtual museum including a 4D presentation of building history in Virtual Reality. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLII-2/W3, 361-367. https://doi.org/10.5194/isprs-archives-XLII-2-W3-361-2017Knyaz, A. V., Kniaz, V. V., Remondino, F., Zheltov, S. Y., & Gruen, A. (2020). 3D reconstruction of a complex grid structure combining UAS images and deep learning. Remote Sensing, 12(19), 3128. https://doi.org/10.3390/rs12193128Lin, P., Sun, P., Cheng, G., Xie, S., Li, X., & Shi, J. (2020). Graph-guided architecture search for real-time semantic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 4203-4212. https://doi.org/10.1109/CVPR42600.2020.00426Llamas, J., Lerones, P. M., Medina, R., Zalama, E., & Gómez-García-Bermejo, J. (2017). Classification of architectural heritage images using deep learning techniques. Applied Science, 7(10), 992. https://doi.org/10.3390/app7100992Lo Turco, M., Piumatti, P., Rinaudo, F., Tamborrino, R., & González-Aguilera, D., (2018). B.A.C.K. TO T.H.E. F.U.T.U.RE. − BIM acquisition as cultural key to transfer heritage of ancient Egypt for many uses to many users replayed. In S. Bertocci (Ed.), Programmi Multidisciplinari Per L'internazionalizzazione Della Ricerca. Patrimonio Culturale, Architettura e Paesaggio (pp. 107-109). DIDA Press.Lo Turco, M., Piumatti, P., Rinaudo, F., Calvano, M., Spreafico, A., & Patrucco, G. (2018). The digitisation of museum collections for research, management and enhancement of tangible and intangible heritage. 3rd Digital Heritage International Congress (DigitalHERITAGE) held jointly with 24th International Conference on Virtual Systems & Multimedia (VSMM 2018), San Francisco, CA, USA. https://doi.org/10.1109/DigitalHeritage.2018.8810128Mafrici, N., & Giovannini, E. C. (2020). Digitalizing data: From the historical research to data modelling for a (digital) collection documentation. In M. Lo Turco, E. C. Giovannini, , & N. Mafrici (Eds.), Digital & Documentation. Digital Strategies for Cultural Heritage (Vol. 2, pp. 38-51). Pavia University Press. https://doi.org/10.5194/isprs-archives-XLII-2-W15-519-2019Malik, U. S., Guidi, G. (2018). Massive 3D digitization of sculptures: Methodological approaches for improving efficiency. IOP Conference Series: Material Science and Engineering, 364. https://doi.org/10.1088/1757-899X/364/1/012015Minto, S., & Remondino, F. (2014). Online access and sharing of reality-based 3D models. SCIRES-IT-SCIentific RESearch and Information Technology, 4(2), 17-28. http://doi.org/10.2423/i22394303v4n2p17Patrucco, G., Chiabrando, F., Dondi, P, & Malagodi, M. (2018). Image and range-based 3D acquisition and modeling of popular musical instruments. Proceedings from the Document Academy, 5(2), 9. https://doi.org/10.35492/docam/5/2/9Patrucco, G., Rinaudo, F., & Spreafico, A. (2019). A new handheld scanner for 3D survey of small artifacts: The Stonex F6. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLII-2/W15, 895-901. https://doi.org/10.5194/isprs-archives-XLII-2-W15-895-2019Pierdicca, R., Paolanti, M., Matrone, F., Martini, M., Morbidoni, C., Malinverni, E. S., Frontoni, E., & Lingua, A. M. (2020). Point cloud semantic segmentation using a deep learning framework for Cultural Heritage. Remote Sensing, 12(6), 1005. https://doi.org/10.3390/rs12061005Salvador-García, E., Viñals, M. J., & García-Valldecabres, J. L. (2020). Potential of HBIM to improve the efficiency of visitor flow management in Heritage sites. Towards smart heritage management. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLIV-M-1-2020, 451-456. https://doi.org/10.5194/isprs-archives-XLIV-M-1-2020-451-2020Shorten, C., & Khoshgoftaar, T. M. (2019). A survey on image data augmentation for deep learning. Journal of Big Data, 6(1), 1-48. https://doi.org/10.1186/s40537-019-0197-0Stathopoulou, E. K., & Remondino, F. (2019). Semantic photogrammetry: Boosting image-based 3D reconstruction with semantic labeling. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLII-2/W9, 685-690. https://doi.org/10.5194/isprs-archives-XLII-2-W9-685-2019UNESCO. (1979). Recommendation for the Protection of Movable Cultural Property, Records of the General Conference, 20th Session, I: Resolutions. Paris: UNESCO.Vargas, R., Mosavi, A., & Ruiz, R. (2018). Deep learning: A review. Advances in Intelligent Systems and Computing, 29(8), 232-244. https://doi.org/10.20944/PREPRINTS201810.0218.V1Yazan, E., & Talu, M. F. (2017). Comparison of the stochastic gradient descent based optimization techniques. 2017 International Artificial Intelligence and Data Processing Symposium (IDAP), 1-5. https://doi.org/10.1109/IDAP.2017.809029
AUTOMATIC MASKS GENERATION FOR THE DIGITIZATION OF A WOODEN COLLECTION OF MAQUETTES USING DEEP LEARNING
No full textENHANCING AUTOMATION OF HERITAGE PROCESSES: GENERATION OF ARTIFICIAL TRAINING DATASETS FROM PHOTOGRAMMETRIC 3D MODELS
No full textNowadays, many efficient technologies have been developed with the aim of collecting digital images and other metric data, greatly optimising the acquisition procedures and techniques. However, processing this data can be onerous and time-consuming, and increasingly often, there is a need to develop new strategies to enhance the level of automation of these processes. Using artificial intelligence, and particularly Convolutional Neural Networks, it is possible to automate processing tasks such as classification and segmentation. However, a significant challenge is represented by the necessity of obtaining sufficient training data to properly train a deep learning model. These datasets are composed of a significant amount of data and need to be annotated, which sometimes represents an onerous and challenging task. Synthetic data can represent an effective solution to this problem, significantly reducing the time and effort required to manually create annotated datasets and can be particularly useful when studying objects characterised by specific features and high complexity, requiring tailored solutions and ad hoc training. The presented research explores the opportunity of using synthetic datasets – generated from photogrammetric 3D models – for deep-learning-based heritage digitisation applications. The use of synthetic data generated from textured 3D models derived from SfM photogrammetric processes is proposed, with the aim of enhancing automatic procedures in the framework of heritage processes
Synthetic Training Datasets for Architectural Conservation: A Deep Learning Approach for Decay Detection
No full textArchitectural heritage conservation increasingly relies on innovative tools for detecting and monitoring degradation. The study presented in the current paper explores the use of synthetic datasets—namely, rendered images derived from photogrammetric models—to train convolutional neural networks (CNNs) for the automated detection of deterioration in historical reinforced concrete structures. The primary objective is to assess the effectiveness of synthetic images for deep learning training, comparing their performance with models trained on traditional datasets. The research focuses on a significant case study: the parabolic concrete arch of Morano sul Po. Two classification scenarios were tested: a single-class model for structure recognition and a multi-class model for identifying degradation patterns, such as exposed reinforcement bars. The findings indicate that synthetic datasets can effectively support structure identification, achieving results comparable to those obtained with real-world imagery. However, challenges arise in multi-class classification, particularly in distinguishing fine-grained degradation features. This study highlights the potential of artificial datasets in overcoming the limitations of annotated data availability in heritage conservation. The proposed approach represents a promising step toward automating documentation and damage assessment, ultimately contributing to more efficient and scalable heritage monitoring strategies
GeoAI tools for urban spaces and their built heritage
No full textAs happened in many disciplines, also in the Geomatic field, the use of Artificial Intelligence (AI) methods such as Machine Learning (ML) and Deep Learning (DL) had a rapid and pioneering development, substantially overturning the entire science of geoinformation.
These innovations contribute to optimising consolidated methods, opening new research avenues and enabling new solutions, among which those relating to urban environments and their current needs in relation to the continuous transformations they undergo will be addressed and discussed. However, this progress also increasingly raises significant challenges, including ethical considerations and questions on AI implementation.
The present contribution analyses the relevant operational advancements in the context of urban space and its building heritage, highlighting the research directions that influence the development of the tools for better urban planning control and addressing challenges affecting its evolution.
A complex set of research directions emerges in the realm of 3D city models, which aim to integrate cognitive systems that allow the combination of geometric and semantic information. These efforts focus on making model creation more time-efficient and sustainable in terms of effectiveness, usability and application potential.
Classification and subsequent segmentation of image and range-based products, such as point clouds, DTMs, and other photogrammetric data, are extensively investigated using both ML and DL methods at different scales, ranging from the urban to the architectural scale. Both levels of analysis are crucial components of three-dimensional urban databases, and the paper will present a study focused on and supporting the urban and architectural scale, developed in the framework of the Geomatic Lab of Politecnico di Torino activities
The Violin Ontology
No full textBowed musical instruments have been the subject of scientific investigations for centuries. Yet, the physical phenomena that are behind their timbral quality are still far from being fully understood. This is one of the reasons why the art of violin making is still so strongly tied to tradition. This manuscript describes early results in a study of the relations that exist between timbral and acoustic characteristics of such instruments and their high-level descriptors. In particular, we propose a suitable ontology for a timbral characterization of violins, where every resource is connected and provided with formally defined semantics. Semantic web technologies have taught us how ontologies can become a powerful tool for gathering and managing knowledge in specific areas of interest, where resources are connected and described with formally defined semantics. This, in fact, represents a crucial step for building applications that reason over Web data. In this paper we present an ontology for knowledge representation of violins, as part of a wider ontology of bowed instruments. With this ontology we capture timbral and acoustic aspects of violins as well as violin making and properties of the materials used for their production. We collected and organized semantic descriptors used by numerous violin makers (from natural language) to describe sound proprieties of musical instruments. We also developed an initial model of the relation between semantic descriptors and low-level audio features. The ontology that we present in this study formalizes the semantics of the high-level descriptors and investigates the relation with low- level features. The terminology has been collected through a series of interviews with violin makers in the city of Cremona (Italy), world heritage site for the practice of violin makers. Through listening tests and a feature extraction we study the correlation between high-level descriptors and objective properties of sound
An unsupervised approach to the semantic description of the sound quality of violins
No full textIn this study we propose a set of semantic musical descriptors that can be used for describing the timbre of violins. The pro-posed semantic model follows a dimensional approach, which allows us to express the degree of intensity of each descrip-tor. A set of recordings of a number of violins (among them, Stradivari, Amati and Guarnieri instruments) were annotated with the descriptors through questionnaires. The recordings are processed with deep learning techniques, to learn salient features from the audio signal in an unsupervised fashion. In this study we propose an automatic annotation procedure based on a set of regression functions that model each seman-tic descriptor using the learned set of features. Index Terms — High-level music descriptor, violin, tim-bre, sound qualit
Training-based semantic descriptors modeling for violin quality sound characterization
No full textViolin makers and musicians describe the timbral qualities of violins using semantic terms coming from natural language. In this study we use regression techniques of machine intelligence and audio features to model in a training-based fashion a set of high-level (semantic) descriptors for the automatic annotation of musical instruments. The most relevant semantic descriptors are collected through interviews to violin makers. These descriptors are then correlated with objective features extracted from a set of violins from the historical and contemporary collections of the Museo del Violino and of the International School of Luthiery both in Cremona. As sound description can vary throughout a performance, our approach also enables the modelling of time-varying (evolutive) semantic annotation
- …
