48 research outputs found
Interpretable deep-learning models for sound event detection and classification
No full textDeep-learning models have revolutionized state-of-the-art technologies in many research areas, but their black-box structure makes it difficult to understand their inner workings and the rationale behind their predictions. This may lead to unintended effects, such as being susceptible to adversarial attacks or the reinforcement of biases. As a consequence, there has been an increasing interest in developing deep-learning models that provide explanations of their decisions, a field known as interpretable deep learning. On the other hand, in the past few years, there has been a surge in developing technologies for environmental sound recognition motivated by its applications in healthcare, smart homes, or urban planning. However, most of the systems used for these applications are deep-learning-based black boxes and, therefore, can not be inspected, so the rationale behind their decisions is obscure. Despite recent advances, there is still a lack of research in interpretable machine learning in the audio domain. This thesis aims to reduce this gap by proposing several interpretable deep-learning models for automatic sound classification and event detection. We start by describing an open-source software tool for reproducible research in the sound recognition field, which was used to implement the models and run experiments presented in this document. We then propose an interpretable front-end based on domain knowledge to tailor the feature-extraction layers of an end-to-end network for sound event detection. We then present a novel interpretable deep-learning model for automatic sound classification, which explains its predictions based on the similarity of the input to a set of learned prototypes in a latent space. We leverage domain knowledge by designing a frequency-dependent similarity measure. The proposed model achieves results comparable to state-of-the-art methods. In addition, we present two automatic methods to prune the proposed model that exploits its interpretability. This model is accompanied by a web application for the manual editing of the model, which allows for a human-in-the-loop debugging approach. Finally, we propose an extension of this model that works for a polyphonic setting, such as the sound event detection task. To provide interpretability, we leverage the prototype network approach and attention mechanisms. The tools for reproducible research and the interpretable deep-learning models, such as those proposed in this thesis, can contribute to developing a more responsible and trustworthy Artificial Intelligence in the audio domain.Els models de deep learning han revolucionat les tecnologies d’última generació en moltes àrees de recerca, però la seva estructura black-box fa difícil entendre el seu funcionament intern i la lògica darrere de les seves prediccions. Això pot conduir a efectes no desitjats, com ara ser susceptible a atacs adversos o el reforç de biaixos. Com a conseqüència, hi ha hagut un interès creixent en el desenvolupament de models de deep learning que proporcionen explicacions de les seves decisions, un camp conegut com a deep learning interpretable. D’altra banda, en els últims anys, s’ha produït un augment en el desenvolupament de les tecnologies per al reconeixement de so ambiental motivat per les seves aplicacions en l’assistència sanitària, les llars intel·ligents o la planificació urbana. No obstant això, la majoria dels sistemes utilitzats per a aquestes aplicacions són black-boxes basades en el deep learning i, per tant, no poden ser inspeccionades, de manera que la raó de les seves decisions és confusa. Malgrat els avenços recents, encara hi ha una manca d’investigació en el deep learning interpretable en el domini d’àudio. Aquesta tesi té com a objectiu reduir aquest buit proposant diversos models de deep learning per a la classificació automàtica del so i la detecció d’esdeveniments. Comencem descrivint una eina de programari de codi obert per a la investigació reproduïble en el camp del reconeixement de so, que es va utilitzar per implementar els models i executar experiments presentats en aquest document. A continuació, proposem un front-end interpretable basat en el coneixement del domini per adaptar les capes d’extracció de característiques d’una xarxa d’extrem a extrem per a la detecció d’esdeveniments sonors. Llavors presentem un nou model interpretable de deep learning per a la classificació automàtica del so, que explica les seves prediccions basades en la similitud de l’entrada a un conjunt de prototips apresos en un espai latent. Aprofitem el coneixement del domini dissenyant una mesura de similitud dependent de la freqüència. El model proposat aconsegueix resultats comparables als mètodes més moderns. A més, presentem dos mètodes automàtics per a reduir el model proposat que explota la seva interpretabilitat. Aquest model està acompanyat per una aplicació web per a l’edició manual del model, que permet una formulació de depuració human-in-the-loop. Finalment, proposem una extensió d’aquest model que funcioni per a un entorn polifònic, com la tasca de detecció d’esdeveniments sonors. Per proporcionar interpretabilitat, aprofitem l’formulació de la xarxa prototip i els mecanismes d’atenció. Les eines per a la investigació reproduïble i els models interpretables de deeplearning, com els proposats en aquesta tesi, poden contribuir al desenvolupament d’una intel·ligència artificial més responsable i fiable en l’àmbit de l’àudio.Programa de Doctorat en Tecnologies de la Informació i les Comunicacion
Freesound explorer: make music while discovering freesound!
No full textComunicació presentada a: Web Audio Conference WAC 2017, celebrada a Londres del 21 al 23 d'agost.Freesound Explorer is a visual interface for exploring Freesound content in a two-dimensional space and creating music by linking content in that space. Freesound Explorer is implemented as a web application which takes advantage of modern web technologies including the Web Audio API and the Web MIDI API. This extended abstract describes Freesound Explorer's features and provides some technical details about its implementation
Freesound explorer: make music while discovering freesound!
No full textComunicació presentada a: Web Audio Conference WAC 2017, celebrada a Londres del 21 al 23 d'agost.Freesound Explorer is a visual interface for exploring
Freesound content in a two-dimensional space and creating
music by linking content in that space. Freesound Explorer
is implemented as a web application which takes advantage
of modern web technologies including the Web Audio API
and the Web MIDI API. This extended abstract describes
Freesound Explorer's features and provides some technical
details about its implementation
Tag recommendation using folksonomy information for online sound sharing platforms
No full textLes plataformes d'intercanvi de recursos multimèdia contenen grans quantitats de contingut
creat pels seus usuaris. Habitualment, aquest contingut no està ben anotat. En aquesta tesi,
proposem mètodes per ajudar els usuaris a anotar el seu contingut d'una manera més
completa i uniforme. Concretament, proposem mètodes per recomanar etiquetes - tags -
basats en informació sobre anotacions prèvies. Avaluem aquesta tasca utilitzant diverses
metodologies i en el context d'una plataforma d'intercanvi de sons. A part de testar el
funcionament de diferents mètodes, també analitzem l'impacte d'un d'aquests mètodes en el
sistema de tagging d'aquesta plataforma. A més a més, també explorem un nou enfocament
per als sistemes de recomanació de tags que incorpora una ontologia amb infromació
específica del camp del so. En general, aquesta tesi contribueix a l'avenç de l'estat de l'art
dels sistemes de tagging i de recomanació de tags basats en folksonomies, i explora
direccions interessants per continuar investigant.Online sharing platforms host a vast amount of multimedia content generated by its own
users. However, such content is typically not uniformly annotated. In this thesis, we propose
methods for helping users to annotate their resources in a more comprehensive and uniform
way. Specifically, we propose methods for tag recommendation which are based on
information gathered from previous resource annotations. Tag recommendation is evaluated
using several methodologies and in the context of a large-scale sound sharing platform.
Besides studying the performance of several methods in this scenario, we analyse the impact
of one of our proposed methods on that platform. In addition, we explore a new perspective
for tag recommendation which employs a sound-specific ontology. Overall, this thesis
contributes to the advancement of the state of the art in tagging systems and folksonomybased
tag recommendation and explores interesting directions for future research.Programa de doctorat en Tecnologies de la Informació i les Comunicacion
Tag recommendation using folksonomy information for online sound sharing platforms
No full textLes plataformes d'intercanvi de recursos multimèdia contenen grans quantitats de contingut
creat pels seus usuaris. Habitualment, aquest contingut no està ben anotat. En aquesta tesi,
proposem mètodes per ajudar els usuaris a anotar el seu contingut d'una manera més
completa i uniforme. Concretament, proposem mètodes per recomanar etiquetes - tags -
basats en informació sobre anotacions prèvies. Avaluem aquesta tasca utilitzant diverses
metodologies i en el context d'una plataforma d'intercanvi de sons. A part de testar el
funcionament de diferents mètodes, també analitzem l'impacte d'un d'aquests mètodes en el
sistema de tagging d'aquesta plataforma. A més a més, també explorem un nou enfocament
per als sistemes de recomanació de tags que incorpora una ontologia amb infromació
específica del camp del so. En general, aquesta tesi contribueix a l'avenç de l'estat de l'art
dels sistemes de tagging i de recomanació de tags basats en folksonomies, i explora
direccions interessants per continuar investigant.Online sharing platforms host a vast amount of multimedia content generated by its own
users. However, such content is typically not uniformly annotated. In this thesis, we propose
methods for helping users to annotate their resources in a more comprehensive and uniform
way. Specifically, we propose methods for tag recommendation which are based on
information gathered from previous resource annotations. Tag recommendation is evaluated
using several methodologies and in the context of a large-scale sound sharing platform.
Besides studying the performance of several methods in this scenario, we analyse the impact
of one of our proposed methods on that platform. In addition, we explore a new perspective
for tag recommendation which employs a sound-specific ontology. Overall, this thesis
contributes to the advancement of the state of the art in tagging systems and folksonomybased
tag recommendation and explores interesting directions for future research.Programa de doctorat en Tecnologies de la Informació i les Comunicacion
An Interpretable deep learning model for automatic sound classification
No full textDeep learning models have improved cutting-edge technologies in many research areas, but their black-box structure makes it difficult to understand their inner workings and the rationale behind their predictions. This may lead to unintended effects, such as being susceptible to adversarial attacks or the reinforcement of biases. There is still a lack of research in the audio domain, despite the increasing interest in developing deep learning models that provide explanations of their decisions. To reduce this gap, we propose a novel interpretable deep learning model for automatic sound classification, which explains its predictions based on the similarity of the input to a set of learned prototypes in a latent space. We leverage domain knowledge by designing a frequency-dependent similarity measure and by considering different time-frequency resolutions in the feature space. The proposed model achieves results that are comparable to that of the state-of-the-art methods in three different sound classification tasks involving speech, music, and environmental audio. In addition, we present two automatic methods to prune the proposed model that exploit its interpretability. Our system is open source and it is accompanied by a web application for the manual editing of the model, which allows for a human-in-the-loop debugging approach
Tempo estimation for music loops and a simple confidence measure
No full textComunicació presentada a la 17th International Society for Music Information Retrieval Conference (ISMIR 2016), celebrada els dies 7 a 11 d'agost de 2016 a Nova York, EUA.Tempo estimation is a common task within the music information retrieval community, but existing works are rarely evaluated with datasets of music loops and the algorithms are not tailored to this particular type of content. In addition to this, existing works on tempo estimation do not put an emphasis on providing a confidence value that indicates how reliable their tempo estimations are. In current music creation contexts, it is common for users to search for and use loops shared in online repositories. These loops are typically not produced by professionals and lack annotations. Hence, the existence of reliable tempo estimation algorithms becomes necessary to enhance the reusability of loops shared in such repositories. In this paper, we test six existing tempo estimation algorithms against four music loop datasets containing more than 35k loops. We also propose a simple and computationally cheap confidence measure that can be applied to any existing algorithm to estimate the reliability of their tempo predictions when applied to music loops. We analyse the accuracy of the algorithms in combination with our proposed confidence measure, and see that we can significantly improve the algorithms' performance when only considering music loops with high estimated confidence.This work has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 688382
Analysis of the folksonomy of freesound
No full textComunicació presentada al 2nd CompMusic Workshop, celebrat els dies 12 i 13 de juliol de 2012 a Istanbul (Turquia), organitzat per CompMusic.User generated content shared in online communities is often described using collaborative tagging systems where users assign labels to content resources. As a result, a folksonomy emerges that relates a number of tags with the resources they label and the users that have used them. In this paper we analyze the folksonomy of Freesound, an online audio clip sharing site which contains more than two million users and 150,000 user-contributed sound samples/ncovering a wide variety of sounds. By following methodologies taken from similar studies, we compute some metrics that characterize the folksonomy both at the global/nlevel and at the tag level. In this manner, we are able to better/nunderstand the behavior of the folksonomy as a whole, and also obtain some indicators that can be used as metadata for describing tags themselves. We expect that such a methodology for characterizing folksonomies can be useful to support processes such as tag recommendation or automatic annotation of online resources.This work is partially supported by the European Research Council under the European Unions Seventh Framework Program, as part of the CompMusic project (ERC grant agreement 267583), and by the Spanish Ministry of Science and Innovation under the BES-2010-037309 FPI grant for the TIN2009-14247-C02-01 DRIMS project
Extending sound sample descriptions through the extraction of community knowledge
No full textComunicació presentada a la 19th International Conference (UMAP) que va tenir lloc de l'11 al 15 de juliol a Girona.Sound and music online services driven by communities of users are filled with large amounts of user-created content that has to be properly described. In these services, typical sound and music modeling is performed using either content-based or context-based strategies, but no special emphasis is given to the extraction of knowledge from the community. We outline a research plan in the context of Freesound.org and propose ideas about how audio clip sharing sites could adapt and take advantage of particular user communities to improve the descriptions of their content.This work is partially supported under BES-2010-037309
FPI grant from the Spanish Ministry of Science and Innovation for the TIN2009-
14247-C02-01 DRIMS project
Tempo estimation for music loops and a simple confidence measure
No full textComunicació presentada a la 17th International Society for Music Information Retrieval Conference (ISMIR 2016), celebrada els dies 7 a 11 d'agost de 2016 a Nova York, EUA.Tempo estimation is a common task within the music information retrieval community, but existing works are rarely evaluated with datasets of music loops and the algorithms are not tailored to this particular type of content. In addition to this, existing works on tempo estimation do not put an emphasis on providing a confidence value that indicates how reliable their tempo estimations are. In current music creation contexts, it is common for users to search for and use loops shared in online repositories. These loops are typically not produced by professionals and lack annotations. Hence, the existence of reliable tempo estimation algorithms becomes necessary to enhance the reusability of loops shared in such repositories. In this paper, we test six existing tempo estimation algorithms against four music loop datasets containing more than 35k loops. We also propose a simple and computationally cheap confidence measure that can be applied to any existing algorithm to estimate the reliability of their tempo predictions when applied to music loops. We analyse the accuracy of the algorithms in combination with our proposed confidence measure, and see that we can significantly improve the algorithms' performance when only considering music loops with high estimated confidence.This work has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 688382
