40 research outputs found
Listenable Maps for Audio Classifiers
No full textDespite the impressive performance of deep learning models across diverse tasks, their complexity poses challenges for interpretation. This challenge is particularly evident for audio signals, where conveying interpretations becomes inherently difficult. To address this issue, we introduce Listenable Maps for Audio Classifiers (L-MAC), a posthoc interpretation method that generates faithful and listenable interpretations. L-MAC utilizes a decoder on top of a pretrained classifier to generate binary masks that highlight relevant portions of the input audio. We train the decoder with a loss function that maximizes the confidence of the classifier decision on the masked-in portion of the audio while minimizing the probability of model output for the masked-out portion. Quantitative evaluations on both in-domain and out-of-domain data demonstrate that L-MAC consistently produces more faithful interpretations than several gradient and masking-based methodologies. Furthermore, a user study confirms that, on average, users prefer the interpretations generated by the proposed technique
LMAC-TD: Producing Time Domain Explanations for Audio Classifiers
No full textNeural networks are typically black-boxes that remain opaque with regards to their decision mechanisms. Several works in the literature have proposed post-hoc explanation methods to alleviate this issue. This paper proposes LMAC-TD, a post-hoc explanation method that trains a decoder to produce explanations directly in the time domain. This methodology builds upon the foundation of L-MAC, Listenable Maps for Audio Classifiers, a method that produces faithful and listenable explanations. We incorporate SepFormer, a popular transformer-based time-domain source separation architecture. We show through a user study that LMAC-TD significantly improves the audio quality of the produced explanations while not sacrificing from faithfulness
Generative modeling of sequential data
No full textIn this thesis, we investigate various approaches for generative modeling, with a special emphasis on sequential data. Namely, we develop methodologies to deal with issues regarding representation (modeling choices), learning paradigm (e.g. maximum likelihood, method of moments, adversarial training), and optimization.
For the representation aspect, we make the following contributions:
-We argue that using a multi-modal latent representation (unlike popular methods such as variational autoencoders or generative adversarial networks) significantly enhances the generative model learning performance, as evidenced by the experiments we conduct on handwritten digit dataset (MNIST) and celebrity faces dataset (CELEB-A).
-We prove that the standard factorial Hidden Markov model defined in the literature is not statistically identifiable. We propose two alternative identifiable models, and show their validity on unsupervised source separation examples.
-We experimentally show that using a convolutional neural network architecture provides performance boost over time agnostic methods such as non-negative matrix factorization, and auto-encoders.
-We experimentally show that using a recurrent neural network with a diagonal recurrent matrix increases the convergence speed and final accuracy of the model in most cases in a symbolic music modeling task.
For the learning paradigm aspect, we make the following contributions:
-We propose a method of moment based parameter learning framework for Hidden Markov Models (HMMs) with special transition structures such as mixture of HMMs, switching HMMs and HMMs with mixture emissions.
-We propose a new generative model learning method which does approximate maximum likelihood parameter estimation for implicit generative models.
-We argue that using an implicit generative model for audio source separation increases the performance over models which specify a cost function, such as NMF or autoencoders trained via maximum likelihood. We show performance improvement in speech mixtures created from the TIMIT dataset.
For the optimization aspect, we make the following contributions:
-We show that using the method of moment framework we propose in this thesis boosts the model performance when used as an initialization scheme for the expectation maximization algorithm.
-We propose new optimization algorithms for identifiable alternatives to Factorial HMM.
-We propose a two-step optimization algorithm for learning implicit generative models which efficiently learns multi-modal latent representations.Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2018-08-31 without embargo termsThe student, Y. Cem Subakan, accepted the attached license on 2018-04-13 at 14:35.The student, Y. Cem Subakan, submitted this Dissertation for approval on 2018-04-13 at 14:42.This Dissertation was approved for publication on 2018-04-13 at 17:13.DSpace SAF Submission Ingestion Package generated from Vireo submission #12239 on 2018-08-31 at 17:12:17Made available in DSpace on 2018-09-04T20:27:04Z (GMT). No. of bitstreams: 3
SUBAKAN-DISSERTATION-2018.pdf: 7033977 bytes, checksum: 3b7a01a31117fe88901da54ac825d61f (MD5)
LICENSE.txt: 4208 bytes, checksum: e591479980e55468b70bd521b9e1dbd1 (MD5)
PROQUEST_LICENSE.txt: 4554 bytes, checksum: ac882a6086af6ebc6a6acd8b92b3acf3 (MD5)
Previous issue date: 2018-04-1
Posthoc Interpretation via Quantization
No full textIn this paper, we introduce a new approach, called Posthoc Interpretation via
Quantization (PIQ), for interpreting decisions made by trained classifiers. Our
method utilizes vector quantization to transform the representations of a
classifier into a discrete, class-specific latent space. The class-specific
codebooks act as a bottleneck that forces the interpreter to focus on the parts
of the input data deemed relevant by the classifier for making a prediction.
Our model formulation also enables learning concepts by incorporating the
supervision of pretrained annotation models such as state-of-the-art image
segmentation models. We evaluated our method through quantitative and
qualitative studies involving black-and-white images, color images, and audio.
As a result of these studies we found that PIQ generates interpretations that
are more easily understood by participants to our user studies when compared to
several other interpretation methods in the literature.Comment: Francesco Paissan and Cem Subakan contributed equall
Audio Editing with Non-Rigid Text Prompts
No full textIn this paper, we explore audio-editing with non-rigid text edits. We show that the proposed editing pipeline is able to create audio edits that remain faithful to the input audio. We explore text prompts that perform addition, style transfer, and in-painting. We quantitatively and qualitatively show that the edits are able to obtain results which outperform Audio-LDM, a recently released text-prompted audio generation model. Qualitative inspection of the results points out that the edits given by our approach remain more faithful to the input audio in terms of keeping the original onsets and offsets of the audio events.Proceedings of Interspeech. Please refer to the reference available at https://www.isca-archive.org/interspeech_2024/paissan24b_interspeech.htm
Listenable Maps for Audio Classifiers
No full textDespite the impressive performance of deep learning models across diverse tasks, their complexity poses challenges for interpretation. This challenge is particularly evident for audio signals, where conveying interpretations becomes inherently difficult. To address this issue, we introduce Listenable Maps for Audio Classifiers (L-MAC), a posthoc interpretation method that generates faithful and listenable interpretations. L-MAC utilizes a decoder on top of a pretrained classifier to generate binary masks that highlight relevant portions of the input audio. We train the decoder with a loss function that maximizes the confidence of the classifier decision on the masked-in portion of the audio while minimizing the probability of model output for the masked-out portion. Quantitative evaluations on both in-domain and out-of-domain data demonstrate that L-MAC consistently produces more faithful interpretations than several gradient and masking-based methodologies. Furthermore, a user study confirms that, on average, users prefer the interpretations generated by the proposed technique.Accepted to ICML 2024 (Oral
