1,720,971 research outputs found
Recommended from our members
Analyzing Easy Data Augmentation Techniques for Text Classification
No full textIn natural language processing, text classification is the task of assigning a category to a given text example. Text classification has a variety of applications ranging from automated processing of customer reviews to spam detection. Current state-of-the-art approaches for text classification tasks use neural language models. These models are resource-intensive, requiring large amounts of labeled training data. However, training data may not always be available in large quantities, especially for low-resource languages, and labeled data is often laborious to obtain. Consequently, it is desirable to understand the factors contributing to text classification models' performance. I address several questions about which factors contribute to the high performance achieved by the current state-of-the-art neural models. To do so, I analyze traditional and neural methods for a diverse range of text classification tasks. I study various properties such as model assumptions and word vector representations to determine the effect of each of these features on text classification performance. On the best performing models from these understandings, I evaluate existing data augmentation techniques for text classification proposed by Wei and Zou (2019), which are methods that perform simple text editing operations to generate new training examples. However, such existing data augmentation techniques require external datasets or knowledge about the semantic properties of words. To this end, I propose and assess a novel length-based method that does not require external linguistic knowledge. This method replaces words with other words of similar length, as word length closely reflects the average information content and conceptual complexity of words in English (Piantadosi, Tily, and Gibson, 2011; Lewis and Frank, 2016). I demonstrate that this length-based technique adds consistent gains for several of the evaluated text classification tasks
Recommended from our members
Causal Mediation Analysis Reveals Syntactic Agreement Mechanisms in Neural Language Models
No full textTargeted syntactic evaluations have demonstrated the ability of language models to perform subject-verb agreement given difficult contexts. Although this is well established, the mechanisms by which neural language models achieve syntactic agreement are still not well understood. As a remedy, this thesis applies causal mediation analysis to pre-trained neural language models to locate model components and discover mechanisms responsible for predicting correctly inflected verbs. In particular, we investigate the magnitude of models’ grammatical inflections preferences, as well as compare which neurons process subject-verb agreement across sentences with different syntactic structures. In our results, we uncover both similarities and differences across architectures and model sizes, and get a glimpse at the within-model mechanisms that produce number agreement. Notably, we learn that larger models do not necessarily learn stronger preferences, we observe two distinct mechanisms for producing subject- verb agreement depending on the syntactic structure of the input sentence, and we find that language models rely on similar sets of neurons when given sentences with similar syntactic structure
Recommended from our members
Inferring Speaker Affect in Spoken Natural Language Communication
Get PDFThe field of spoken language processing is concerned with creating computer programs that can understand human speech and produce human-like speech. Regarding the problem of understanding human speech, there is currently growing interest in moving beyond speech recognition (the task of transcribing the words in an audio stream) and towards machine listening—interpreting the full spectrum of information in an audio stream. One part of machine listening, the problem that this thesis focuses on, is the task of using information in the speech signal to infer a person’s emotional or mental state. In this dissertation, our approach is to assess the utility of prosody, or manner of speaking, in classifying speaker affect. Prosody refers to the acoustic features of natural speech: rhythm, stress, intonation, and energy. Affect refers to a person’s emotions and attitudes such as happiness, frustration, or uncertainty. We focus on one specific dimension of affect: level of certainty. Our goal is to automatically infer whether a person is confident or uncertain based on the prosody of his or her speech. Potential applications include conversational dialogue systems (e.g., in educational technology) and voice search (e.g., smartphone personal assistants). There are three main contributions of this thesis. The first contribution is a method for eliciting uncertain speech that binds a speaker’s uncertainty to a single phrase within the larger utterance, allowing us to compare the utility of contextually-based prosodic features. Second, we devise a technique for computing prosodic features from utterance segments that both improves uncertainty classification and can be used to determine which phrase a speaker is uncertain about. The level of certainty classifier achieves an accuracy of 75%. Third, we examine the differences between perceived, self-reported, and internal level of certainty, concluding that perceived certainty is aligned with internal certainty for some but not all speakers and that self-reports are a good proxy for internal certainty.Engineering and Applied Science
Recommended from our members
A Comparison of Natural Language Models to Subtype Ischemic Stroke from Electronic Health Records
No full textIschemic stroke is a leading cause of disability and death worldwide. While the prevalence of ischemic stroke varies across race and ethnicity, it is particularly pronounced in low-resource medical institutions, where patients may experience higher rates of post-stroke complications and mortality. Despite concerted attempts to implement established interventions and explore novel treatment modalities, the coarse classification of ischemic stroke obscures the underlying heterogeneity in both pathophysiological mechanisms and clinical manifestations, rendering these efforts insufficient. As such, there is huge value in investigating the underlying etiology of ischemic stroke in large, diverse cohorts of patients that could power refined subtype discovery. While electronic health record (EHR) data represent a valuable resource for stroke subtyping, there are few studies that have utilized EHR data for building these cohorts due to the significant human effort required to label features and cases.
To this goal, the rapid progress of Natural Language Processing methods has opened the door to fully automated stroke subtyping. This study investigates the performance of two Natural Language Processing approaches, namely Logistic Regression, a statistical model, and Clinical Longformer, a pre-trained transformer model, in subtyping ischemic stroke directly from the EHR. The models were trained and tested on EHR data of about 3000 stroke patients adjudicated by board-certified neurologists. Both models achieve commendable performance, with the transformer model slightly outperforming the statistical model with recall and precision of 0.83 and 0.74 respectively. This finding highlights that Natural Language Processing offers a more consistent and scalable approach to subtyping ischemic stroke from EHR, which could significantly enhance the statistical power and facilitate large-scale stroke research. In particular, this outcome has enabled us to infer toast subtypes across a sizable cohort of 30,000 coded strokes at Massachusetts General Hospital, thereby opening up novel avenues for investigating stroke risk prediction and genetic underpinnings.Computer Scienc
Recommended from our members
How to Get Transformers to Process in Steps
No full textAs pointed out by Daniel Kahneman, there are at least two qualitatively distinct modes of cognition that take place in the human brain — fast and automatic thinking, labeled “System 1”, and slow and methodical thinking, labeled “System 2”. Progress in Artificial Intelligence requires approaches for modeling both kinds of cognition with computers. Classical programming is very effective for solving many tasks from the do- main of System 2, but is not practical for solving tasks from the domain of System 1. Machine learning is very effective for solving many tasks from the domain of System 1, but has not yet been shown to work robustly on tasks from the domain of System 2. In this work, we investigate if and how machine learning could be successfully applied for algorithmic tasks which are solved via System 2 by humans.
We argue that the standard way of framing machine learning problems that is suit- able for System 1 tasks is inadequate for System 2 tasks, propose an alternative, and demonstrate its effectiveness. Specifically, while learning a direct mapping from inputs to outputs is feasible for System 1 tasks, we argue that algorithmic System 2 tasks can only be solved by learning a mapping from inputs to outputs through a series of inter- mediate steps. We first show that by using enough intermediate steps a 1-layer Trans- former can in principle compute any finite function. We then show empirically that a 1-layer Transformer cannot learn to compute the sum of binary numbers directly from the inputs, but is able to compute the sum when trained to first generate a series of in- termediate results. This demonstrates, at a small scale, how a fixed-size neural network can lack the expressivity to encode the direct input-output mapping for an algorithmic task and yet be fully capable of computing the outputs through intermediate steps. Fi- nally, we show that a Frozen Pretrained Transformer is able to learn binary addition when trained to compute the carry bits before the sum, while it fails to learn the task without using the intermediates. These results support our hypothesis that the use of intermediate computations is necessary for tackling algorithmic tasks from the domain of System 2 via machine learning
Recommended from our members
Impacts of Predictive Text on Writing Content
No full textPeople are increasingly communicating using digital technologies, especially using written language. Many modern text entry systems include features that offer predictive suggestions of words and phrases. These features have been designed and evaluated for the goal of making text entry more efficient. However, when these predictions are shown to writers to use, they also serve as suggestions of what content to write. This dissertation addresses two main questions: what effects do current predictive text system designs have on writing content, and what are some challenges and opportunities to designing the effects that predictive systems can have on writing content?
The human-subjects studies conducted found evidence that the text that people write using predictive text entry systems reflects bias of these systems. Specifically, the presence of word suggestions decreased the degree to which writers chose the sorts of words that the system would not have expected in an image captioning task. Phrase suggestions resulted in even more predictable writing than word suggestions. Finally, phrase suggestions that were generated with a bias towards positive sentiment resulted in people writing more positive content, which implies that biases in training data could lead to biases in the content people create.
Several pilot studies towards designing suggestions with different effects failed because writers viewed the experimental suggestions as irrelevant, so the dissertation presents work along two approaches towards enabling alternative suggestion designs. First, it discusses how to design suggestion systems that can adjust content attributes without overly compromising suggestion acceptability by applying an approach from reinforcement learning. Then it discusses how to design suggestions to guide writers about what topics to include, including a study comparing the perceived relevance of two designs.Engineering and Applied Sciences - Computer Scienc
Going Beyond Counting First Authors in Author Co-citation Analysis
Get PDFThe present study examines one of the fundamental aspects of author co-citation analysis (ACA) - the way co-citation
counts are defined. Co-citation counting provides the data on which all subsequent statistical analyses and mappings
are based, and we compare ACA results based on two different types of co-citation counting - the traditional type that
only counts the first one among a cited work's authors on the one hand and a non-traditional type that takes into
account the first 5 authors of a cited work on the other hand. Results indicate that the picture produced through this non-traditional author co-citation counting contains more coherent author groups and is therefore considerably clearer. However, this picture represents fewer specialties in the research field being studied than that produced through the traditional first-author co-citation counting when the same number of top-ranked authors is selected and analyzed. Reasons for these effects are discussed
Recommended from our members
Deep Latent Variable Models of Natural Language
No full textUnderstanding natural language involves complex underlying processes by which meaning is extracted from surface form. One approach to operationalizing such phenomena in computational models of natural language is through probabilistic latent variable models, which can encode structural dependencies among observed and unobserved variables of interest within a probabilistic framework. Deep learning, on the other hand, offers an alternative computational approach to modeling natural language through end-to-end learning of expressive, global models, where any phenomena necessary for the task are captured implicitly within the hidden layers of a neural network. This thesis explores a synthesis of deep learning and latent variable modeling for natural language processing applications. We study a class of models called deep latent variable models, which parameterize components of probabilistic latent variable models with neural networks, thereby retaining the modularity of latent variable models while at the same time exploiting rich parameterizations enabled by recent advances in deep learning. We experiment with different families of deep latent variable models to target a wide range of language phenomena---from word alignment to parse trees---and apply them to core natural language processing tasks including language modeling, machine translation, and unsupervised parsing.
We also investigate key challenges in learning and inference that arise when working with deep latent variable models for language applications. A standard approach for learning such models is through amortized variational inference, in which a global inference network is trained to perform approximate posterior inference over the latent variables. However, a straightforward application of amortized variational inference is often insufficient for many applications of interest, and we consider several extensions to the standard approach that lead to improved learning and inference. In summary, each chapter presents a deep latent variable model tailored for modeling a particular aspect of language, and develops an extension of amortized variational inference for addressing the particular challenges brought on by the latent variable model being considered. We anticipate that these techniques will be broadly applicable to other domains of interest.Engineering and Applied Sciences - Computer Scienc
Recommended from our members
Structured Neural Models for Coreference and Generation
No full textNatural Language Processing (NLP) has recently entered a period marked by impressive empirical performance on a wide variety of natural language tasks, with much of this empirical success due to the use of deep learning techniques. Deep learning has, in particular, offered a simple approach for learning expressive, global models of linguistic data. While incredibly powerful, this style of modeling poses a problem for NLP tasks that require structured prediction, the prediction of outputs with combinatorial structure, such as sequences, trees, and graphs. Indeed, whereas the standard approach to tackling structured prediction problems in NLP involves predicting the structure with the highest score under the learned model, it is often intractable to find the highest scoring structure under the sort of global model common in deep learning approaches to NLP. In this thesis we argue that search-based structured prediction, where a model is trained to search incrementally for a structure, is a particularly natural choice for doing structured prediction with deep models. Specifically, we argue that recurrent neural networks make it simple and convenient to compactly represent the history of incremental predictions made during search, which allows for the learning of powerful search-based structured predictors.
We first investigate this approach to deep structured prediction in the context of the NLP task of coreference resolution. In particular we first discuss a baseline, neural coreference resolver, which was sufficient for state-of-the-art performance on its introduction, and we then show that a search-based, structured approach improves even over this. We then discuss an approach to training the celebrated sequence-to-sequence model as a search-based structured predictor, and we show that this leads to improvements on word ordering, dependency parsing, and machine translation tasks.
In the final chapter of this thesis we discuss the structured prediction problem of long-form text generation, and database-to-text generation in particular, which is not well handled by the techniques introduced in the preceding chapters. We introduce a new dataset for studying the challenges posed by this structured prediction problem, suggest new automatic approaches to evaluating performance on this problem, and use these automatic approaches to analyze the performance of various state-of-the-art generation models.Engineering and Applied Sciences - Computer Sciencenatural language processing; deep learning; structured predictio
- …
