1,721,008 research outputs found
Representation learning for computational sociopragmatics
No full textNatural Language Processing (NLP) emerges as a critical solution for analyzing, manipulating, and understanding human language automatically and computationally, enabling the processing of vast amounts of language data swiftly. Computational NLP systems utilize numerical matrices or vectors as inputs, necessitating the conversion of discrete language symbols into a continuous representation space. The efficacy of these continuous representations is pivotal for developing successful NLP systems. With the advent of attention mechanisms, attention-based models have been adopted to learn contextual language representations by pre-training with language modeling (LM) objectives on extensive textual corpora. Despite the proven effectiveness of attention-based pre-trained language models (PLMs) in learning sequence-level representations for various NLP tasks, the integration of social aspects into representation learning remains unexplored. Recent efforts have applied PLMs to derive user-level representations, aiming to enhance content-based recommendation systems' transferability and precision. However, challenges persist in encoding lengthy user engagement histories, capturing users' diverse interests, and generating precomputable user-level representations. This dissertation focuses on advancing language representation learning for sequence-level sociopragmatic meaning (SM) comprehension and user-level content-based recommendation. For sequence-level SM, we introduce a novel weakly supervised method for pretraining and fine-tuning language models (Chapter 2). To enhance representation quality further, we propose a new contrastive learning framework for pretraining LMs (Chapter 3). Our approach is extended to the multilingual domain, presenting a unified, massively multilingual evaluation benchmark for SM (Chapter 4), alongside a comprehensive evaluation of state-of-the-art large language models for SM understanding. Addressing the challenges in learning user-level representations for recommendation systems, Chapter 5 introduces a novel framework that incorporates multiple poly-attention layers and sparse attention mechanisms. This framework hierarchically fuses token-level embeddings of session-based user history texts using PLM, tackling the intricacies of recommendation systems.Information, School ofGraduat
Representation learning for Arabic dialect identification
No full textArabic dialect identification (ADI) is an important aspect of the Arabic speech processing pipeline, and in particular dialectal Arabic automatic speech recognition (ASR) models. In this work, we present an overview of corpora and methods applicable to both ADI and dialectal Arabic ASR, then we benchmark two approaches to using pre-trained speech representation models for ADI. Namely, we first employ direct fine-tuning, and then use fixed-representations extracted from pre-trained models as an intermediate step in the ADI process. We train and evaluate our models on the granular ADI-17 Arabic dialect corpus (92% F1 for our fine-tuned HuBERT model), and further probe generalization by evaluating our trained models on coarse-grained ADI-5, (80% F1 for fine-tuned HuBERT).Arts, Faculty ofiSchool (Library, Archival and Information Studies)Graduat
Towards Afrocentric natural language processing
No full textThis dissertation centers on Natural Language Processing (NLP) for African languages, endeavoring to unravel the progress, challenges, and future prospects within this linguistic context. The research encompasses language identification and Natural Language Understanding (NLU), Natural Language Generation (NLG), and culminates in a comprehensive case study on machine translation.
The first chapter introduces the problem statement, articulates the motivation for addressing the
issue, and presents the innovative solutions developed throughout this research. Chapter two discusses intricate details of African languages, offering insights into the genealogical classification, linguistic landscape, and the challenges of multilingual NLP. Building upon this foundation, the third chapter advocates for an Afrocentric approach to technology development, emphasizing the significance of aligning technology with the cultural values and linguistic diversity of African communities. It addresses challenges such as data scarcity and representation bias, spotlighting community-driven initiatives aimed at advancing NLP in the region.
The fourth chapter unveils AfroLID, a neural language identification tool designed for 517 African
languages and language varieties, establishing itself as the new state-of-the-art solution for African
language identification.
Chapter five introduces SERENGETI, a massively multilingual language model tailored to support
517 African languages and language varieties. Evaluation on AfroNLU, an extensive benchmark for
African NLP, showcases SERENGETI’s superior performance, thereby paving the way for transformative research and development across a diverse linguistic landscape.
The sixth chapter addresses NLG challenges in African languages, presenting Cheetah, a language model designed for 517 African languages. Comprehensive evaluations underscore Cheetah’s capacity to generate contextually relevant text across various African languages.
The seventh chapter presents a case study on machine translation, focusing on Bare Nouns (BNs)
translation from Yorùbá to English. This study illuminates the challenges posed by information
asymmetry in machine translation and provides insights into the linguistic capabilities of Statistical
Machine Translation (SMT) and Neural Machine Translation (NMT) systems. Emphasizing the
importance of fine-grained linguistic considerations, the study encourages further research in addressing translation challenges faced by languages with BNs, analytic languages, and low-resource languages.
In chapter eight, I conclude and discuss possible directions for future work.Arts, Faculty ofLinguistics, Department ofGraduat
Multi-Task Sequence Prediction for Tunisian Arabizi Multi-Level Annotation
No full textIn this paper we propose a multi-task sequence prediction system, based on recurrent neural networks and used to annotate on multiple levels an Arabizi Tunisian corpus. The annotation performed are text classification, tokenization, PoS tagging and encoding of Tunisian Arabizi into CODA* Arabic orthography. The system is learned to predict all the annotation levels in cascade, starting from Arabizi input. We evaluate the system on the TIGER German corpus, suitably converting data to have a multi-task problem, in order to show the effectiveness of our neural architecture. We show also how we used the system in order to annotate a Tunisian Arabizi corpus, which has been afterwards manually corrected and used to further evaluate sequence models on Tunisian data. Our system is developed for the Fairseq framework, which allows for a fast and easy use for any other sequence prediction problem
Improving language models with novel contrastive learning objectives
No full textContrastive learning (CL) has recently emerged as an effective technique in natural language processing, especially in the important area of language modeling. In this work, we offer novel methods for deploying CL in both pretraining and finetuning of language models. First, we present PACT (Pretraining with Adversarial Contrastive Learning for Text Classification), a novel self-supervised framework for text classification. Instead of contrasting against in-batch negatives, a popular approach in the literature, PACT mines negatives closer to the anchor representation. PACT operates by endowing the standard pretraining mechanisms of BERT with adversarial contrastive learning objectives, allowing for effective joint optimization of token- and sentence-level pretraining of the BERT model. Our experiments on 13 diverse datasets including token-level, single-sentence, and sentence-pair text classification tasks show that PACT achieves consistent improvements over SOTA baselines. We further show that PACT regularizes both token-level and sentence-level embedding spaces into more uniform representations, thereby alleviating the undesirable anisotropic phenomenon of language models. Subsequently, in the context of finetuning, we apply CL in tackling cross-platform abusive language detection. The prevalence of abusive language on different online platforms has been a major concern that raises the need for automated cross-platform abusive language detection. However, prior works focus on concatenating data from multiple platforms, inherently adopting Empirical Risk Minimization (ERM) method. In our work, we address this challenge from the perspective of domain generalization objective. We design SCL-Fish, a supervised contrastive learning integrated meta-learning algorithm to detect abusive language on unseen platforms. Our experimental analysis shows that SCL-Fish achieves better performance over ERM and the existing state-of-the-art models. We also show that SCL-Fish is data-efficient and achieves comparable performance with the large-scale pretrained models upon finetuning for the abusive language detection task.Science, Faculty ofComputer Science, Department ofGraduat
Methods for design of efficient on-device natural language processing architectures
No full textDeep learning based models often achieve state-of-the-art performance in a wide range of natural language processing (NLP) tasks, which include open-ended tasks (e.g., story generation, brainstorming, and chat) and closed-ended tasks (e.g., summarization, question answering, and rewriting). To further enhance quality, there is a growing interest in scaling the model size and the amount of data used for training. These research efforts often overlook the impact of footprint metrics, such as high latency, high memory usage, and high energy consumption, on these deep learning models. A high footprint makes these models significantly inefficient for deployment on servers and devices such as tablets, handhelds, and wearables. Methods for improving model efficiency often come at the cost of degrading model quality.
In this dissertation, we address the central question: how can we push the envelope in improving the efficiency-quality tradeoff of deep learning models for on-device NLP tasks? To this end, we propose methods that take on-device efficiency constraints (e.g., ≤ 16 MB memory or ≤ 200 ms latency) to inform the design of the model architecture. We propose methods for the manual design of architecture for the auto-completion task (generate continuations for user-written prompts) that enjoy a better memory-accuracy tradeoff than existing auto-completion models (Chapter 2). Additionally, we introduce methods that can directly take efficiency constraints to automatically search for efficient sparsely activated architectures for machine translation tasks (Chapter 3) and efficient pretrained (task-agnostic) language modeling architectures (Chapter 4). Finally, in Chapter 5, we explore a novel use case of employing large language models to speed up architecture search, while maintaining the efficiency and quality of state-of-the-art neural architecture search algorithms.Science, Faculty ofComputer Science, Department ofGraduat
Going Beyond Counting First Authors in Author Co-citation Analysis
Get PDFThe present study examines one of the fundamental aspects of author co-citation analysis (ACA) - the way co-citation
counts are defined. Co-citation counting provides the data on which all subsequent statistical analyses and mappings
are based, and we compare ACA results based on two different types of co-citation counting - the traditional type that
only counts the first one among a cited work's authors on the one hand and a non-traditional type that takes into
account the first 5 authors of a cited work on the other hand. Results indicate that the picture produced through this non-traditional author co-citation counting contains more coherent author groups and is therefore considerably clearer. However, this picture represents fewer specialties in the research field being studied than that produced through the traditional first-author co-citation counting when the same number of top-ranked authors is selected and analyzed. Reasons for these effects are discussed
- …
