636 research outputs found
Stochastic gradient descent with random label noises: doubly stochastic models and inference stabilizer
No full textRandom label noises (or observational noises) widely exist in practical machine learning settings. While previous studies primarily focus on the affects of label noises to the performance of learning, our work intends to investigate the implicit regularization effects of the label noises, under mini-batch sampling settings of stochastic gradient descent (SGD), with assumptions that label noises are unbiased. Specifically, we analyze the learning dynamics of SGD over the quadratic loss with unbiased label noises, where we model the dynamics of SGD as a stochastic differentiable equation (SDE) with two diffusion terms (namely a Doubly Stochastic Model ). While the first diffusion term is caused by mini-batch sampling over the (labelnoiseless) loss gradients, as in many other works on SGD [1, 2], our model investigates the second noise term of SGD dynamics, which is caused by mini-batch sampling over the label noises, as an implicit regularizer. Our theoretical analysis finds such implicit regularizer would favor some convergence points that could stabilize model outputs against perturbation of parameters (namely inference stability). Though similar phenomenon have been investigated by Blanc et al. [3], our work doesn't assume SGD as an Ornstein-Uhlenbeck like process and achieve a more generalizable result with convergence of approximation proved. To validate our analysis, we design two sets of empirical studies to analyze the implicit regularizer of SGD with unbiased random label noises for deep neural networks training and linear regression. Our first experiment studies the noisy self-distillation tricks for deep learning, where student networks are trained using the outputs from well-trained teachers with additive unbiased random label noises. Our experiment shows that the implicit regularizer caused by the label noises tend to select models with improved inference stability
Dou: distributivity and beyond
No full textThis dissertation investigates the semantic properties of the particle dou in Chinese. The standard view of it is that it is a particle that accompanies plural noun phrases and has a semantics somewhat similar (not identical) to the floated all in English. In this dissertation, I will explore in some depth several phenomena where dou seems to play a role that goes beyond distributivity.
Chapter 1 introduces the standard view of dou as a distributive operator as proposed in Lin (1998) and the topics of the thesis. In so doing, the similarities and differences between dou and English all are highlighted.
Chapters 2 and 3 are devoted to two topics that are not covered in Lin's original work and that seem to pose problems for his analysis. Chapter 2 discusses what I call the dou-(dis)harmony phenomenon: dou's (in)compatibility with quantifier phrases. This challenges the standard semantics of dou in that all of the quantifier noun phrases, dou-compatible or not, are presumably plural and thus should be compatible with dou. In this chapter, I first argue that previous approaches that characterize the (dis)harmony effect in terms of categories of NPs are not correct. Then I claim that this has to do with a presupposition that accompanies dou. In particular, I argue that dou is has a presupposition about expectations and I propose to build this aspect of meaning into the semantics of dou. Chapter 3 investigates dou in a structure where plurality is not needed to license dou. Instead, focus is the crucial licensing factor. This is traditionally assumed to involve the lian...dou/ye 'dou/also' structure where it has a scalar reading similar to the meaning even has in English. Researchers disagree as to whether this dou should be assimilated to distributive dou or should be treated separately. Through careful investigations into some rarely addressed properties of dou in this structure, I conclude in favor of the ambiguity view of dou. In addition, I propose to link this dou to distributive dou through context sensitivity as I developed in chapter 2. Finally, I provide a compositional semantics for lian...dou/ye based on the semantics of each individual particle.
Chapter 4 extends the discussion to dou in free choice structures: dou co-occurring with renhe-NPs 'any' or wh-NPs yields a FC reading, similar to the corresponding English sentences with FC any. In this chapter, I explore the two FC structures from the perspective of English FC any and whatever on the one hand and from that of our prior discussions of dou on the other. We argue that renhe...dou is like universal any but wh...dou is neither like universal any nor definite whatever. It is suggested that dou in the two FC structures, renhe...dou and wh...dou, is related to distributive dou and scalar dou respectively, in support of our claim that there are two related but distinct dou's.
Chapter 5 closes this thesis and provides some initial exploration of the interactions between dou and bare NPs. Chinese bare NPs are, basically, like English bare plurals displaying various readings in various contexts. This chapter examines the behavior of bare NPs in various contexts from the perspective of the two-dou account developed in this dissertation. This investigation, though preliminary, provides further support for our claim that dou has a presupposition about the prior expectations on the part of the speaker and that the two dou's need to be separated.Ph.D.Includes bibliographical references (p. 188-193)
Prediction of ICD-9 Code Assignment Using Attention-Based Convolutional Neural Networks
No full textIn intensive care units, most patients are usually in critical conditions which require physicians to make immediate diagnosis and treatments. However, not every patient could get the best treatment because it highly related to the physician’s expertise. With the development of the machine learning, many studies have started trying to develop models that can learn the representations in Electronic Health Records (EHR) and make accurate predictions on clinical tasks. On code assignment tasks, models based on convolutional neural networks (CNN) or Recurrent Neural Networks (RNN) have shown promising results but their performances are still insufficient to be applied on real-world applications due to (1) the large number of codes and (2) the length of the document. Here, we propose a Convolutional Neural Network with Multi-label attention mechanism (Multi-Label AT-CNN) model that predict ICD-9 code assignments by learning the base representations of the clinical notes from EHRs.2021-04-3
Knowledge Base Refinement and Knowledge Translation with Markov Logic Networks
No full textMachine learning and data mining have provided plenty of tools for extracting knowledge from data. Yet, such knowledge may not be directly applicable to target applications and might need further manipulation: The knowledge might contain too much noise, or the target application may use a different representation or terminology.
In this dissertation, we study three problems related to knowledge management and manipulation. First, given a knowledge base (KB) automatically extracted from the text, we explore how to refine it based on the dependencies among the possible KB instances and their confidence values. Second, when the target application to which we want to apply our knowledge uses a different schema, we explore how to translate the knowledge based on the mapping between the schemas. Sometimes, the mapping between two schemas can be discovered automatically, so the third problem we consider is whether we can find the mapping more accurately using the corresponding knowledge contained in the two schemas.
We notice that a large fraction of data and knowledge can be represented in relational models, which can be formalized with first-order logic. Moreover, uncertainty is a common feature existing in these problems, e.g., the confidence values associated with the KB instances, the probabilistic knowledge rules to be translated, or the schemas not perfectly aligned with each other. Therefore, we adopt statistical relational learning, which combines first-order logic with probabilistic models, to resolve these problems. In particular, we use Markov logic networks (MLNs), which consist of sets of weighted first-order formulas. MLNs are a powerful and flexible language for representing hard and soft constraints of relational domains.
We develop the MLN formulations for each of these problems, and we use the representation, inference and learning approaches in the literature with certain
adaptations to solve them. The experiment results show that MLNs successfully provide solutions to these problems or achieve better performances than the existing methods.
This dissertation includes previously published and unpublished coauthored material
Understanding Perceived Sense of Movement in Static Visuals Using Deep Learning
No full textThis thesis introduces the problem of learning the representation and the classification of the perceived sense of movement, defined as dynamism in static visuals. To solve the said problem, we study the definition, degree, and real-world implications of dynamism within the field of consumer psychology. We employ Deep Convolutional Neural Networks (DCNN) as a method to learn and predict dynamism in images. The novelty of the task, lead us to collect a dataset which we synthetically augmented for spatial invariance, using image processing techniques. We study the methods of transfer learning to transfer knowledge from another domain, as the size of our dataset was deemed to be inadequate. Our dataset is trained across different network architectures, and transfer learning techniques to find an optimal method for the task at hand. To show a real-world application of our work, we observe the correlation between the two visual stimuli, dynamism and emotions
Sampling sparse representations with randomized measurement langevin dynamics
No full textStochastic Gradient Langevin Dynamics (SGLD) have been widely used for Bayesian sampling from certain probability distributions, incorporating derivatives of the log-posterior. With the derivative evaluation of the log-posterior distribution, SGLD methods generate samples from the distribution through performing as a thermostats dynamics that traverses over gradient flows of the log-posterior with certainly controllable perturbation. Even when the density is not known, existing solutions still can first learn the kernel density models from the given datasets, then produce new samples using the SGLD over the kernel density derivatives. In this work, instead of exploring new samples from kernel spaces, a novel SGLD sampler, namely, Randomized Measurement Langevin Dynamics (RMLD) is proposed to sample the high-dimensional sparse representations from the spectral domain of a given dataset.Specifically, given a random measurement matrix for sparse coding, RMLD first derives a novel likelihood evaluator of the probability distribution from the loss function of LASSO, then samples from the high-dimensional distribution using stochastic Langevin dynamics with derivatives of the logarithm likelihood and Metropolis–Hastings sampling. In addition, new samples in low-dimensional measuring spaces can be regenerated using the sampled high-dimensional vectors and the measurement matrix. The algorithm analysis shows that RMLD indeed projects a given dataset into a high-dimensional Gaussian distribution with Laplacian prior, then draw new sparse representation from the dataset through performing SGLD over the distribution. Extensive experiments have been conducted to evaluate the proposed algorithm using real-world datasets. The performance comparisons on three real-world applications demonstrate the superior performance of RMLD beyond baseline methods
A Hybrid Approach for Ontology-based Information Extraction
No full textInformation extraction (IE) is the process of automatically transforming written natural language (i.e., text) into structured information, such as a knowledge base. However, because natural language is inherently ambiguous, this transformation process is highly complex. On the other hand, as Information Extraction moves from the analysis of scientific documents to the analysis of Internet textual content, we cannot rely completely on the assumption that the content of the text is correct. Indeed, in contrast to scientific documents, which are peer reviewed, Internet content is not verified for the quality and correctness.
Thus, two main issues that affect the IE process are the complexity of the extraction process and the quality of the data.
In this dissertation, we propose an improved ontology-based IE (OBIE) by providing solutions to these issues of accuracy and content quality. Based on a hybrid strategy that combines aspects of IE that are usually considered as opposite to each other, or that are not even considered, we intend to improve IE by developing a more accurate extraction and new functionality (semantic error detection). Our approach is based on OBIE, a sub-area of IE, which reduces extraction complexity by including domain knowledge, in the form of concepts and relationships of the domain, to guide the extraction process.
We address the complexity of extraction by combining information extractors that have different implementations. By integrating different types of implementation into one extraction system, we can produce a more accurate extraction. For each concept or relationship in the ontology, we can select the best implementation for extraction, or we can combine both implementations under an ensemble learning schema. In tandem, we address the quality of information by determining its semantic correctness with regard to domain knowledge. We define two methods for semantic error detection: by predefining the types of errors expected in the text or by applying logic reasoning to the text.
This dissertation includes both published and unpublished coauthored material
Physical co-presence intensity: Measuring dynamic face-to-face interaction potential in public space using social media check-in records
No full textUrban public spaces facilitate social interactions between people, reflecting the shifting functionality of spaces. There is no commonly-held consensus on the quantification methods for the dynamic interplay between spatial geometry, urban movement, and face-to-face encounters. Using anonymized social media check-in records from Shanghai, China, this study proposes pipelines for quantifying physical face-to-face encounter potential patterns through public space networks between local and non-local residents sensed by social media over time from space to space, in which social difference, cognitive cost, and time remoteness are integrated as the physical co-presence intensity index. This illustrates the spatiotemporally different ways in which the built environment binds various groups of space users configurationally via urban streets. The variation in face-to-face interaction patterns captures the fine-resolution patterns of urban flows and a new definition of street hierarchy, illustrating how urban public space systems deliver physical meeting opportunities and shape the spatial rhythms of human behavior from the public to the private. The shifting encounter potentials through streets are recognized as reflections of urban centrality structures with social interactions that are spatiotemporally varying, projected in the configurations of urban forms and functions. The results indicate that the occurrence probability of face-to-face encounters is more geometrically scaled than predicted based on the co-location probability of two people using metric distance alone. By adding temporal and social dimensions to urban morphology studies, and the field of space syntax research in particular, we suggest a new approach of analyzing the temporal urban centrality structures of the physical interaction potentials based on trajectory data, which is sensitive to the transformation of the spatial grid. It sheds light on how to adopt urban design as a social instrument to facilitate the dynamically changing social interaction potential in the new data environment, thereby enhancing spatial functionality and the social well-being
Semantic Oppositeness for Inconsistency and Disagreement Detection in Natural Language
No full textSemantic oppositeness is the natural counterpart of the rather more popular natural language processing concept, semantic similarity. Much like how semantic similarity is a measure of the degree to which two concepts are similar, semantic oppositeness yields the degree to which two concepts would oppose each other. This complementary nature has resulted in most applications and studies incorrectly assuming semantic oppositeness to be the inverse of semantic similarity. In other trivializations, "semantic oppositeness" is used interchangeably with "antonymy," which is as inaccurate as replacing semantic similarity with simple synonymy. These erroneous assumptions and over-simplifications exist due, mainly, to either a lack of information, or the computational complexity of calculation of semantic oppositeness. This dissertation considers the following question: How can we convert the linguistic concept of semantic oppositeness to the computing domain? To answer this question, we follow the linguistic definition of oppositeness and develop a novel methodology based on antonymy as well as similarity. We also propose a novel method to embed the obtained semantic oppositeness in a vector space for increased generalization and efficiency. We then consider two realms of applications: inconsistency and disagreements. The inconsistency application helped us track changes in a medical research domain. The disagreement application accentuated the ability to detect rumours in the social media domain. Finally, we extract the commonalities and patterns in these methodologies to provide a comprehensive summary and a set of recommendations and future work. This dissertation is a culmination of previously published, co-authored material
A Graph-based Approach for Semantic Data Mining
No full textData mining is the nontrivial extraction of implicit, previously unknown, and potentially useful information from data. It is widely acknowledged that the role of domain knowledge in the discovery process is essential. However, the synergy between domain knowledge and data mining is still at a rudimentary level. This motivates me to develop a framework for explicit incorporation of domain knowledge in a data mining system so that insights can be drawn from both data and domain knowledge. I call such technology "semantic data mining."
Recent research in knowledge representation has led to mature standards such as the Web Ontology Language (OWL) by the W3C's Semantic Web initiative. Semantic Web ontologies have become a key technology for knowledge representation and processing. The OWL ontology language is built on the W3C's Resource Description Framework (RDF) that provides a simple model to describe information resources as a graph. On the other hand, there has been a surge of interest in tackling data mining problems where objects of interest can be best described as a graph of interrelated nodes. I notice that the interface between domain knowledge and data mining can be achieved by using graph representations. Therefore I explore a graph-based approach for modeling both knowledge and data and for analyzing the combined information source from which insight can be drawn systematically.
In summary, I make three main contributions in this dissertation to achieve semantic data mining. First, I develop an information integration solution based on metaheuristic optimization when data mining task require accessing heterogeneous data sources. Second, I describe how a graph interface for both domain knowledge and data can be structured by employing the RDF model and its graph representations. Finally, I describe several graph theoretic analysis approaches for mining the combined information source. I showcase the utility of the proposed methods on finding semantically associated itemsets, a particular case of the frequent pattern mining. I believe these contributions in semantic data mining can provide a novel and useful way to incorporate domain knowledge.
This dissertation includes published and unpublished coauthored material
- …
