268 research outputs found
On feature representations for marmoset vocal communication analysis
No full textThe acoustic analysis of marmoset (Callithrix jacchus) vocalizations is often
used to understand the evolutionary origins of human language. Currently, the
analysis is largely carried out in a manual or semi-manual manner. Thus, there
is a need to develop automatic call analysis methods. In that direction,
research has been limited to the development of analysis methods with small
amounts of data or for specific scenarios. Furthermore, there is lack of prior
knowledge about what type of information is relevant for different call
analysis tasks. To address these issues, as a first step, this paper explores
different feature representation methods, namely, HCTSA-based hand-crafted
features Catch22, pre-trained self supervised learning (SSL) based features
extracted from neural networks trained on human speech and end-to-end acoustic
modeling for call-type classification, caller identification and caller sex
identification. Through an investigation on three different marmoset call
datasets, we demonstrate that SSL-based feature representations and end-to-end
acoustic modeling tend to lead to better systems than Catch22 features for
call-type and caller classification. Furthermore, we also highlight the impact
of signal bandwidth on the obtained task performances
SMILE Swiss German Sign Language Dataset
No full textDescription
The SMILE Swiss German Sign Language Dataset consists of videos, joint coordinates and annotations of 100 isolated signs of a Swiss German Sign Language (Deutschschweizerische Gebärdensprache, DSGS) vocabulary production test. All items were produced multiple times by 16 adult L1 signers and 22 adult L2 learners of DSGS. Associated linguistic transcriptions and annotations are available for second path data of 10 adult L1 signers and 18 adult L2 learners of DSGS.
The dataset has been created in the context of developing an assessment system for lexical signs of DSGS in the SNSF project SMILE.
More precisely, for each participant, the following files are available:
Kinect color video (.mp4); 1920x1080 Pixels @ 30 FPS;
Kinect Pose Information (.csv); 25 Joints; 3D Joint Coordinates and Angles;
OpenPose output (.json); 2D Joint Coordinates and Confidences;
iLex annotation files (.xml); linguistic annotations.
Reference
If you use this database, please cite the following publication:
Sarah Ebling, Necati Cihan Camgöz, Penny Boyes Braem, Katja Tissi, Sandra Sidler-Miserez, Stephanie Stoll, Simon Hadfield, Tobias Haug, Richard Bowden, Sandrine Tornay, Marzieh Razavi, and Mathew Magimai-Doss. SMILE Swiss German Sign Language Dataset. In Proceedings of the 11th Language Resources and Evaluation Conference (LREC 2018), pages 4221–4229, 2018
On Learning Grapheme-to-Phoneme Relationships through the Acoustic Speech Signal
No full textAutomatic speech recognition (ASR) systems, through use of the phoneme as an intermediary unit representation, split the problem of modeling the relationship between the written form, i.e., the text and the acoustic speech signal into two disjoint processes. The first process deals with modeling of the relationship between the written form and phonemes through development of a pronunciation dictionary using prior knowledge about grapheme-to-phoneme relationships. Given the pronunciation lexicon and the transcribed speech data, the second process then deals with modeling of the relationship between the phonemes and the acoustic speech signal using statistical sequence processing techniques, such as hidden Markov models. As a consequence of the two disjoint processes, development of an ASR system heavily relies on the availability of well-developed acoustic and lexical resources in the target language. This paper presents an approach where the relationship between graphemes and phonemes is learned through acoustic data, more precisely, through phoneme posterior probabilities estimated from the speech signal. In doing so, the approach tightly couples the above mentioned two processes and leads to a framework where, existing acoustic and lexical resources from different domains and languages can be effectively exploited to build ASR systems without development of a pronunciation lexicon and to develop lexical resources for resource scarce domains and languages. We demonstrate these capabilities of the proposed approach through cross domain studies in English, where the grapheme-to-phoneme relationship is deep.LIDIA
Trustworthy speaker recognition with minimal prior knowledge using neural networks
No full textThe performance of speaker recognition systems has considerably improved in the last decade. This is mainly due to the development of Gaussian mixture model-based systems and in particular to the use of i-vectors. These systems handle relatively well noise and channel mismatches and yield a low error rate when confronted with zero-effort impostors, i.e. impostors using their own voice but claiming to be someone else. However, speaker verification systems are vulnerable to more sophisticated attacks, called presentation or spoofing attacks. In that case, the impostors present a fake sample to the system, which can either be generated with a speech synthesis or voice conversion algorithm or can be a previous recording of the target speaker. One way to make speaker recognition systems robust to this type of attack is to integrate a presentation attack detection system.
Current methods for speaker recognition and presentation attack detection are largely based on short-term spectral processing. This has certain limitations. For instance, state-of-the-art speaker verification systems use cepstral features, which mainly capture vocal tract system characteristics, although voice source characteristics are also speaker discriminative. In the case of presentation attack detection, there is little prior knowledge that can guide us to differentiate bona fide samples from presentation attacks, as they are both speech signals that carry the same high level information, such as message, speaker identity and information about environment.
This thesis focuses on developing speaker verification and presentation attack detection systems that rely on minimal assumptions. Towards that, inspired by recent advances in deep learning, we first develop speaker verification approaches where speaker discriminative information is learned from raw waveforms using convolutional neural networks (CNNs). We show that such approaches are capable of learning both voice source related and vocal tract system related speaker discriminative information and yield performance competitive to state of the art systems, namely i-vectors and x-vectors-based systems. We then develop two high performing approaches for presentation attack detection: one based on long-term spectral statistics and the other based on raw speech modeling using CNNs. We show that these two approaches are complementary and make the speaker verification systems robust to presentation attacks. Finally, we develop a visualization method inspired from the computer vision community to gain insight about the task-specific information captured by the CNNs from the raw speech signals.LIDIA
Grapheme-based Automatic Speech Recognition using Probabilistic Lexical Modeling
No full textAutomatic speech recognition (ASR) systems incorporate expert knowledge of language or the linguistic expertise through the use of phone pronunciation lexicon (or dictionary) where each word is associated with a sequence of phones. The creation of phone pronunciation lexicon for a new language or domain is costly as it requires linguistic expertise, and includes time and money. In this thesis, we focus on effective building of ASR systems in the absence of linguistic expertise for a new domain or language. Particularly, we consider graphemes as alternate subword units for speech recognition. In a grapheme lexicon, pronunciation of a word is derived from its orthography. However, modeling graphemes for speech recognition is a challenging task for two reasons. Firstly, grapheme-to-phoneme (G2P) relationship can be ambiguous as languages continue to evolve after their spelling has been standardized. Secondly, as elucidated in this thesis, typically ASR systems directly model the relationship between graphemes and acoustic features; and the acoustic features depict the envelope of speech, which is related to phones. In this thesis, a grapheme-based ASR approach is proposed where the modeling of the relationship between graphemes and acoustic features is factored through a latent variable into two models, namely, acoustic model and lexical model. In the acoustic model the relationship between latent variables and acoustic features is modeled, while in the lexical model a probabilistic relationship between latent variables and graphemes is modeled. We refer to the proposed approach as probabilistic lexical modeling based ASR. In the thesis we show that the latent variables can be phones or multilingual phones or clustered context-dependent subword units; and an acoustic model can be trained on domain-independent or language-independent resources. The lexical model is trained on transcribed speech data from the target domain or language. In doing so, the parameters of the lexical model capture a probabilistic relationship between graphemes and phones. In the proposed grapheme-based ASR approach, lexicon learning is implicitly integrated as a phase in ASR system training as opposed to the conventional approach where first phone pronunciation lexicon is developed and then a phone-based ASR system is trained. The potential and the efficacy of the proposed approach is demonstrated through experiments and comparisons with other standard approaches on ASR for resource rich languages, nonnative and accented speech, under-resourced languages, and minority languages. The studies revealed that the proposed framework is particularly suitable when the task is challenged by the lack of both linguistic expertise and transcribed data. Furthermore, our investigations also showed that standard ASR approaches in which the lexical model is deterministic are more suitable for phones than graphemes, while probabilistic lexical model based ASR approach is suitable for both. Finally, we show that the captured grapheme-to-phoneme relationship can be exploited to perform acoustic data-driven G2P conversion.LIDIA
On matching data and model in LF-MMI-based dysarthric speech recognition
No full textIn light of steady progress in machine learning, automatic speech recognition (ASR) is entering more and more areas of our daily life, but people with dysarthria and other speech pathologies are left behind. Their voices are underrepresented in the training data and so different from typical speech that ASR systems fail to recognise them. This thesis aims to adapt both acoustic models and training data of ASR systems in order to better handle dysarthric speech. We first build state-of-the-art acoustic models based on sequence-discriminative lattice-free maximum mutual information (LF-MMI) training that serve as baselines for the following experiments. We propose the dynamic combination of models trained on either control, dysarthric, or both groups of speakers to address the acoustic variability of dysarthric speech. Furthermore, we combine models trained with either phoneme or grapheme acoustic units in order to implicitly handle pronunciation variants. Second, we develop a framework to analyse the acoustic space of ASR training data and its discriminability. We observe that these discriminability measures are strongly linked to subjective intelligibility ratings of dysarthric speakers and ASR performance. Finally, we compare a range of data augmentation methods, including voice conversion and speech synthesis, for creating artificial dysarthric training data for ASR systems. With our analysis framework, we find that these methods reproduce characteristics of natural dysarthric speech
On Joint Modelling of Grapheme and Phoneme Information using KL-HMM for ASR
No full textIn this paper, we propose a simple approach to jointly model both grapheme and phoneme information using Kullback-Leibler divergence based HMM (KL-HMM) system. More specifically, graphemes are used as subword units and phoneme posterior probabilities estimated at output of multilayer perceptron are used as observation feature vector. Through preliminary studies on DARPA Resource Management corpus it is shown that although the proposed approach yield lower performance compared to KL-HMM system using phoneme as subword units, this gap in the performance can be bridged via temporal modelling at the observation feature vector level and contextual modelling of early tagged contextual graphemes.LIDIA
Novel Methods for Incorporating Prior Knowledge for Automatic Speech Assessment
No full textSpeech signal conveys several kinds of information such as a message, speaker identity, emotional state of the speaker and social state of the speaker. Automatic speech assessment is a broad area that refers to using automatic methods to predict human judgements regarding different kinds of information conveyed in speech, such as intelligibility of the spoken message, dialect and fluency of the speaker. Unlike other speech technology areas, such as automatic speech recognition, text-to-speech synthesis and automatic speaker recognition, automatic speech assessment is an emerging direction of research. One of the challenges in this field is that there is no single method or framework that scales across diverse speech assessment tasks. Thus, this thesis takes a broader outlook and focuses on prior knowledge incorporation for diverse data-driven speech assessment problems.
First, we focus on the development of end-to-end acoustic modelling methods for non-verbal cue-based speech assessment. More precisely, we develop neural network-based methods that can integrate prior knowledge about speech production to learn to assess speech from raw waveform. We validate the developed methods through investigations on several speech assessment tasks, viz. dialect identification, depression detection and speech fluency rating prediction.
Second, we focus on advancing a recently proposed phone posterior feature-based intelligibility estimation technique. Specifically, to enhance phone posterior probability estimation, we propose two novel approaches to incorporate linguistic segment level knowledge during the training of neural networks through estimation of confidence measures. We validate the two proposed approaches through automatic speech recognition and dysarthric speech intelligibility assessment studies.
Finally, in the context of privacy preservation, we develop a signal processing-based speech pseudonymization approach that alters voice source information and vocal tract system information based on prior knowledge to obfuscate the speaker identity, while retaining intelligibility, i.e. the phones and words remain recognizable. We validate the proposed pseudonymization approach through listening experiments and automatic evaluations.LIDIA
On Modeling the Synergy Between Acoustic and Lexical Information for Pronunciation Lexicon Development
No full textState-of-the-art automatic speech recognition (ASR) and text-to-speech systems require a pronunciation lexicon that maps each word to a sequence of phones. Manual development of lexicons is costly as it needs linguistic knowledge and human expertise. To facilitate this process, grapheme-to-phone (G2P) conversion approaches are used, in which given a seed lexicon provided by linguistic experts, the G2P relationship is learned by applying statistical techniques. Despite advances in these approaches, there are two challenges remaining: (1) the seed lexicon development through linguistic expertise incorporates limited acoustic information, which may not necessarily cover all natural phonological variations, and (2) the linguistic expertise required for the development of the seed lexicon may not be available for all languages, particularly under-resourced languages. The goal of this thesis is to address these challenges by developing a framework that effectively integrates linguistic information and acoustic data for pronunciation lexicon development. To achieve that goal, we first study the problem of matching a word hypothesis to the acoustic signal, and show that the hidden Markov model-based ASR approach achieves that match via a latent symbol set. Building on that understanding, we develop a data-driven G2P conversion approach in which a probabilistic G2P relationship is learned by matching the acoustic signal with the word hypothesis represented by graphemes, using phones as the latent symbols. Through a theoretical development, we show that this acoustic G2P conversion approach is a particular case of an abstract posterior-based G2P conversion formalism, which requires estimation of phone class conditional probabilities. Through studies on two languages, we show that the acoustic G2P conversion approach yields lexicons that can perform comparable to state-of-the-art G2P conversion methods at the ASR level, despite performing relatively poorly at pronunciation level. We build on the posterior-based formalism to show that different G2P conversion approaches in the literature can be regarded as different estimators of phone class conditional probabilities, and can be combined in a multi-stream fashion to yield better lexicons. We also demonstrate that the multi-stream formulation can be further extended to unify acoustic-to-phone conversion and G2P conversion. We validate the proposed multi-stream formulation on two challenging tasks on English. Finally, we address the issue of developing lexical resources for under-resourced languages by proposing an acoustic subword unit (ASWU)-based lexicon development approach. In this approach, ASWU derivation is cast as the problem of determining a latent symbol space given the word hypothesis and acoustics, and the pronunciations are generated using the proposed acoustic G2P conversion approach. Through experimental studies and analysis on well-resourced and under-resourced languages, we show that the derived ASWUs are "phone-like" , and the ASWU-based lexicons yield better ASR systems compared to the alternative grapheme-based lexicons.LIDIA
Use voice conversion for pseudonymisation?
No full textSharing speech data is needed for progress in speech science&technology. Privacy is a concern, especially for pathological speech. The question is discussed whether pseudonymization is possible such that the identity of the speaker is obscured but linguistic & para-linguistic features of the speech are retained. Possible applications are demonstrations for live audience, speech corpora for study, generating fully Open Data, and protection during speech processing in the cloud.The ZIP file contains the presentation PDF, LaTeX Beamer code, figures, and Audio files. Opening the presentation PDF in Acrobat Reader, Skim, or Evince will allow to play the audio during the presentation. Other PDF readers were not tested
- …
