327 research outputs found
Utilizing Worker Groups And Task Dependencies in Crowdsourcing
No full textCrowdsourcing has emerged as a convenient mechanism to collect human judgments on a variety of tasks, ranging from document and image classification to scientific experimentation. However, in recent times crowdsourcing has evolved from solving simpler tasks, like recognizing objects in images, to more complex tasks such as collaborative journalism, language translation, product designing etc. Unlike simpler micro-tasks performed by a single worker, these complex tasks require a group of workers and greater resources. In such scenarios, where groups of participants are the atomic units, it is a non-trivial task to distinguish workers (who contribute positively) from idlers (who do not contribute to group task) among the participants using only group's performance. The first part of this thesis studies the problem of distinguishing workers from idlers, without assuming any prior knowledge of individual skills and considers \groups" as the smallest observable unit for evaluation. We draw upon literature from group-testing and give bounds over minimum number of groups required to identify quality of subsets of individuals with high confidence. We validate our theory experimentally and report insights for the number of workers and idlers that can be identified for a given number of group-tasks with significant probability.
In most crowdsourcing applications, there exist dependencies among the pool of Human Intelligence Tasks (HITs) and often in practical scenarios there are far too many HITs available than what can realistically be covered by limited available budget. Estimating the accuracy of automatically constructed Knowledge Graphs (KG) is one such important application. Automatic construction of large knowledge graphs has gained wide popularity in recent times. These KGs, such as NELL, Google Knowledge Vault, etc., consist of thousands of predicate-relations (e.g., is Person, is Mayor Of) and millions of their instances (e.g., (Bill de Blasio, is Mayor Of, New York City)). Estimating accuracy of such KGs is a challenging problem due to their size and diversity. In the second part of this study, we show that standard single-task crowdsourc- ing is sub-optimal and very expensive as it ignores dependencies among various predicates and instances. We propose Relational Crowdsourcing (RelCrowd) to overcome this challenge, where the tasks are created while taking dependencies among predicates and instances into account. We apply this framework in the context of large-scale Knowledge Graph Evaluation (KGEval) and demonstrate its effectiveness through extensive experiments on real-world datasets
Methods for Improving Data-efficiency and Trustworthiness using Natural Language Supervision
No full textTraditional strategies to build machine learning based classification systems employ discrete labels as targets. This limits the usefulness of such systems in two ways. First, the generalizability of these systems is limited to labels present and well represented in the training data. Second, with increasingly larger neural network models gaining acceptability, supervision with discrete labels alone does not lead to a straightforward interface for generating explanations for the decisions taken by such systems. Natural Language (NL) Supervision (NLS), in the form of task descriptions, examples, label descriptions and explanations for labelling decisions, provides a way to overcome these bottlenecks. Working in this paradigm, we propose novel methods for improving data-efficiency and trustworthiness:
(1) [Data Efficiency using NLS] Word Sense Disambiguation (WSD) using Sense Definition Embeddings:
WSD, a long-standing open problem in Natural Language Processing (NLP), typically presents itself with small training corpora with long tails of label distributions. Existing supervised methods didn’t generalize well to rare or unseen classes while NL supervision based systems did worse on overall (standard) evaluation benchmarks. We propose Extended WSD Incorporating Sense Embeddings (EWISE), a supervised model to perform WSD by predicting over a continuous sense embedding space as opposed to a discrete label space. This allows EWISE to generalize over both seen and unseen senses, thus achieving generalized zero-shot learning. To obtain target sense embeddings, EWISE utilizes NL sense definitions along with external knowledge in WordNet relations. EWISE achieved new state-of-the-art WSD performance at the time of publication, specifically by improving on zero-shot and few-shot learning.
(2) [Trustworthiness using NLS] Natural Language Inference (NLI) with Faithful NL Explanations:
Generated NL explanations are expected to be faithful, i.e., they should correlate well with the model’s internal decision making. In this work, we focus on the task of NLI and address the following question: can we build NLI systems which produce labels with high accuracy, while also generating faithful explanations of its decisions? We propose Natural-language Inference over Label-specific Explanations (NILE), a novel NLI method which utilizes auto-generated label-specific NL explanations to produce a label along with its faithful explanation. Our evaluation of NILE also supports the claim that accurate systems capable of providing testable explanations of their decisions can be designed.
(3) [Improving the NLS interface of Large Language Models (LLM)]
LLMs, pre-trained on unsupervised corpora, have proven to be successful as zero-shot and few-shot learners on downstream tasks using only a textual interface. This enables a promising NLS interface. A typical usage involves augmenting an input example along with some priming text comprising of task descriptions and training examples and processing the output probabilities to make predictions. In this work, we further explore priming-based few-shot learning and make the following contributions:
(a) Reordering Examples Helps during Priming-based Few-Shot Learning: We show that presenting training examples in the right order is key for generalization. We introduce PERO (Prompting with Examples in the Right Order), where we formulate few-shot learning as search over the set of permutations of the training examples. We demonstrate the effectiveness of the proposed method on the tasks of sentiment classification, natural language inference and fact retrieval. We show that PERO can learn to generalize efficiently using as few as 10 examples, in contrast to existing approaches.
(b) Answer-level Calibration (ALC) Helps Free-form Multiple Choice Question Answering (QA): We consider the QA format, where we need to choose from a set of free-form textual choices of unspecified lengths, given a context. We present ALC, where our main suggestion is to model context-independent biases in terms of the probability of a choice without the associated context and to subsequently remove these biases using an unsupervised estimate of similarity with the full context. ALC improves zero-shot and few-shot performance on several benchmarks while also providing a more reliable estimate of performance
Analysis and Methods for Knowledge Graph Embeddings
No full textKnowledge Graphs (KGs) are multi-relational graphs where nodes represent entities, and typed edges represent relationships among entities. These graphs store real-world facts such as (Lionel Messi, plays-for-team, Barcelona) as edges, called triples. KGs such as NELL, YAGO, Freebase, and WikiData have been very popular and support many applications such as Web Search, Query Recommendation, and Question Answering. Although popular, these KGs suffer from incompleteness. Learning Knowledge Graph Embeddings (KGE) is a common approach for predicting missing edges (i.e., link prediction) and representing entities and relations in downstream tasks. While numerous KGE methods have been proposed in the past decade, our understanding and analysis of such embeddings have been limited. Further, such methods only work well with ontological KGs. In this thesis, we address these gaps.
Firstly, we study various KGE methods and present an extensive analysis of these methods, resulting in many insights. Next, we address an under-explored problem of link prediction in Open Knowledge Graphs (OpenKGs) and present a novel approach that improves the type compatibility of predicted edges. Lastly, we present an adaptive interaction framework for learning KG embeddings that generalizes many existing methods.
In the first part, we present a macro and a micro analysis of embeddings learned by various KGE methods. Despite the popularity and effectiveness of KG embeddings, their geometric understanding (i.e., arrangement of entity and relation vectors in vector space) is unexplored. We initiate a study to analyze the geometry of KG embeddings and correlate it with task performance and other hyper-parameters. Firstly, we present a set of metrics (e.g., Conicity, ATM) to analyze the geometry of a group of vectors. Using these metrics, we find sharp differences between the geometry of embeddings learned by different classes of KGE methods. The vectors learned by a multiplicative model lie in a narrow cone, unlike additive models where the vectors are spread out in the space. This behavior of multiplicative models is amplified by increasing the number of negative samples used for training. Further, a very high Conicity value is negatively correlated with the performance on the link prediction task. We also study the problem of understanding KG embeddings’ semantics and propose an approach to learn more coherent dimensions. A dimension is coherent if the top entities have similar types (e.g., person). In this work, we formalize the notion of coherence using entity co-occurrence statistics and propose a regularizer term that maximizes coherence while learning KG embeddings. The proposed approach significantly improves coherence while having a comparable performance with baseline in the link prediction and triple classification tasks. Further, based on the human evaluation, we demonstrate that the proposed approach learns more coherent dimensions than the baseline.
In the second part, we address the problem of learning KG embeddings for Open Knowledge Graphs (OpenKGs), focusing on improving link prediction. An OpenKG refers to a set of (head noun phrase, relation phrase, tail noun phrase) triples such as (tesla, return to, new york) extracted from a text corpus using OpenIE tools. While OpenKGs are easy to bootstrap for a domain, they are very sparse. Therefore, link prediction becomes an important step while using these graphs in downstream tasks. Learning OpenKG embeddings is one approach for link prediction that has received some attention lately. However, on careful examination, we find that current algorithms often predict noun phrases (NPs) with incompatible types for given noun and relation phrases. We address this problem and propose OKGIT that improves OpenKG link prediction using novel type compatibility score and type regularization. With extensive experiments on multiple datasets, we show that the proposed method achieves state-of-the-art performance while producing type compatible NPs in the link prediction task.
In the third part, we address the problem of improving the KGE models. Firstly, we show that the performance of existing approaches vary across different datasets, and a simple neural network-based method can consistently achieve better performance on these datasets. Upon analysis, we find that KGE models depend on fixed sets of interactions among the dimensions of entity and relation vectors. Therefore, we investigate ways to learn such interactions automatically during training. We propose an adaptive interaction framework for learning KG embeddings, which can learn appropriate interactions while training. We show that some of the existing models could be seen as special cases of the proposed framework. Based on this framework, we also present two new models, which outperform the baseline models on the link prediction task. Further analysis demonstrates that the proposed approach can adapt to different datasets by learning appropriate interactions
Evaluating Quadratic Weighted Kappa as the Standard Performance Metric for Automated Essay Scoring
No full textAutomated Essay Scoring (AES) tools aim to improve the efficiency and consistency of essay scoring by using machine learning algorithms. In the existing research work on this topic, most researchers agree that human-automated score agreement remains the benchmark for assessing the accuracy of machine-generated scores. To measure the performance of AES models, the Quadratic Weighted Kappa (QWK) is commonly used as the evaluation metric. However, we have identified several limitations of using QWK as the sole metric for evaluating AES model performance. These limitations include its sensitivity to the rating scale, the potential for the so-called kappa paradox to occur, the impact of prevalence, the impact of the position of agreements in the diagonal agreement matrix, and its limitation in handling a large number of raters. Our findings suggest that relying solely on QWK as the evaluation metric for AES performance may not be sufficient. We further discuss insights into additional metrics to comprehensively evaluate the performance and accuracy of AES models
Leveraging KG Embeddings for Knowledge Graph Question Answering
No full textKnowledge graphs (KG) are multi-relational graphs consisting of entities as nodes and relations
among them as typed edges. The goal of knowledge graph question answering (KGQA) is to
answer natural language queries posed over the KG. These could be simple factoid questions
such as “What is the currency of USA? ” or it could be a more complex query such as “Who
was the president of USA after World War II? ”. Multiple systems have been proposed in the
literature to perform KGQA, include question decomposition, semantic parsing and even graph
neural network-based methods.
In a separate line of research, KG embedding methods (KGEs) have been proposed to
embed the entities and relations in the KG in low-dimensional vector space. These methods
aim to learn representations that can be then utilized by various scoring functions to predict the
plausibility of triples (facts) in the KG. Applications of KG embeddings include link prediction
and KG completion. Such KG embedding methods, even though highly relevant, have not been
explored for KGQA so far.
In this work, we focus on 2 aspects of KGQA: (i) Temporal reasoning, and (ii) KG incompleteness. Here, we leverage recent advances in KG embeddings to improve model reasoning in
the temporal domain, as well as use the robustness of embeddings to KG sparsity to improve
incomplete KG question answering performance. We do this through the following contributions:
Improving Multi-Hop KGQA using KG Embeddings
We first tackle a subset of KGQA queries – multi-hop KGQA. We propose EmbedKGQA, a
method which uses ComplEx embeddings and scoring function to answer these queries. We find
that EmbedKGQA is particularly effective at KGQA over sparse KGs, while it also relaxes the
requirement of answer selection from a pre-specified local neighborhood, an undesirable constraint imposed by GNN-based for this task. Experiments show that EmbedKGQA is superior
to several GNN-based methods on incomplete KGs across a variety of dataset scales.
Question Answering over Temporal Knowledge Graphs We then extend our method to temporal knowledge graphs (TKG), where each edge in the KG
is accompanied by a time scope (i.e. start and end times). Here, instead of KGEs, we make
use of temporal KGEs (TKGE) to enable the model to make use of these time annotations and
perform temporal reasoning. We also propose a new dataset - CronQuestions - which is one of
the largest publicly available temporal KGQA dataset with over 400k template-based temporal
reasoning questions. Through extensive experiments we show the superiority of our method,
CronKGQA, over several language-model baselines on the challenging task of temporal KGQA
on CronQuestions.
Sequence-to-Sequence Knowledge Graph Completion and Question Answering
So far, integrating KGE into the KGQA pipeline had required separate training of the KGE
and KGQA modules. In this work, we show that an off-the-shelf encoder-decoder Transformer
model can serve as a scalable and versatile KGE model obtaining state-of-the-art results for
KG link prediction and incomplete KG question answering. We achieve this by posing KG link
prediction as a sequence-to-sequence task and exchange the triple scoring approach taken by
prior KGE methods with autoregressive decoding. Such a simple but powerful method reduces
the model size up to 98% compared to conventional KGE models while keeping inference time
tractable. It also allows us to answer a variety of KGQA queries, not being restricted by query
type
Deep Learning over Hypergraphs
No full textGraphs have been extensively used for modelling real-world network datasets, however, they
are restricted to pairwise relationships, i.e., each edge connects exactly two vertices. Hypergraphs
relax the notion of edges to connect arbitrary numbers of vertices. Hypergraphs
provide a mathematical foundation for understanding and learning from large amounts of
real-world data. State-of-the-art techniques for learning vertex representations from graph
data with pairwise relationships use graph-based deep models such as graph neural networks.
A prominent observation that inspires this thesis is that neural networks are still underexplored
for hypergraph data with group-wise relationships. The main challenges involved are
(a) handling the relational nature of hypergraph data and (b) dealing with group relations
where a group contains an arbitrary number of vertices rather than a fixed number. In this
work, we tackle these challenges and fill important research gaps through the following contributions.
Deep Learning for Hypergraph Vertex-Level Predictions
We explore connections between graph neural networks (GNNs) and spectral hypergraph
theory and also connections between GNNs and optimal transport. These connections lead to
novel vertex representation learning methods over hypergraphs. We demonstrate the effectiveness
of the proposed methods on vertex property prediction.
Deep Learning for Hypergraph Link Prediction
We propose novel hypergraph scoring functions for link prediction. In contrast to existing
methods, our proposed methods can be applied for predicting missing links in real-world
hypergraphs in which hyperedges need not represent similarity.
Deep Learning for Multi-Relational and Recursive Hypergraphs
We unify various methods for message passing on different structures (e.g., hypergraphs,
heterogeneous graphs, etc.) into a single framework. We next propose novel extensions of
these methods and demonstrate the effectiveness for reasoning over knowledge base
Inducing Constraints in Paraphrase Generation and Consistency in Paraphrase Detection
No full textDeep learning models typically require a large volume of data. Manual curation of datasets is time-consuming and limited by imagination. As a result, natural language generation (NLG) has been employed to automate the process. However, in their vanilla formulation, NLG model are prone to producing degenerate, uninteresting, and often hallucinated outputs. Constrained generation aims to overcome these shortcomings by providing additional information to the generation process. Training data thus generated can help improve the robustness of deep learning models. Therefore, the central research question of the thesis is:
“How can we constrain generation models, especially in NLP, to produce meaningful
outputs and utilize them for building better classification models?”
To demonstrate how generation models can be constrained, we present two approaches for paraphrase generation. Paraphrase generation involves the generation of text that conveys the same meaning as a reference text. We propose two strategies for paraphrase generation:
(1) DiPS (Diversity in Paraphrases using Submodularity): The first approach deals with constraining paraphrase generation to ensure diversity, i.e., ensuring that generated text(s) are sufficiently different from each other. We propose a decoding algorithm for obtaining diverse texts. We provide a novel formulation of the problem in terms of monotone submodular function maximization, specifically targeted toward the task of paraphrase generation. We demonstrate the effectiveness of our method for data augmentation on multiple tasks such as intent classification and paraphrase recognition.
(2) SGCP (Syntax Guided Controlled Paraphraser): The second approach deals with constraining paraphrase generation to ensure syntacticality, i.e., ensuring that the generated text is syntactically coherent with an exemplar sentence. We propose Syntax Guided Controlled Paraphraser (SGCP), an end-to-end framework for syntactic paraphrase generation without compromising relevance (fidelity). Through a battery of automated metrics and comprehensive human evaluation, we verify that this approach does better than prior works that utilize only limited syntactic information in the parse tree.
The second part (meaningful outputs) of the research question pertains to ensuring that the generated output is meaningful. Towards this, we present an approach for paraphrase detection to ascertain that the generated output is semantically coherent with the reference text. Paraphrase Detection is the task of detecting whether or not the two input natural language statements are paraphrases of each other. Fine-tuning pre-trained models such as BERT and RoBERTa on paraphrastic datasets have become the go-to approaches for such tasks. However, tasks like paraphrase detection are symmetric - they require the output to be invariant of the order of the inputs. In the traditional fine-tuned approach for paraphrase classification, inconsistency is often observed in the predicted labels or confidence scores based on the order of the inputs. We validate this shortcoming and apply a consistency loss function to alleviate inconsistency in symmetric classification. Our results show an improved consistency in predictions for three paraphrase detection datasets without a significant drop in the accuracy scores.
While these works address the research question via paraphrase generation and detection, the approaches presented here apply broadly to NLP-based deep learning models that require imposing constraints and ensuring consistency. The work on paraphrase generation can be extended to impose new kinds of constraints (for example, sentiment coherence) on generation, while paraphrase detection can be applied to ensure consistency in other symmetric classification tasks (for example, sarcasm interpretation) that use deep learning models
Relating Representations in Deep Learning and the Brain
No full textDeep Neural Networks (DNN) inspired by the human brain have redefined the state-of-the-art performance in AI during the past decade. Much of the research is still trying to understand and explain the function of these networks. In this thesis, we leverage knowledge from the neuroscience literature to evaluate the representations learned in state-of-the-art language models. We use sentences with simple syntax and semantics (e.g., “The bone was eaten by the dog.”), and train multiple neural networks to predict the part of speech, next word. We present other sentences of this same simple form, word-by-word to humans in a magnetoencephalography (MEG) scanner for silent reading and comprehension. We then train a linear regression model to predict observed brain recording from the hidden layers of the trained neural networks and popular pre-trained networks like BERT and ELMo.
We find that the middle layers of these networks are the most predictive of the recorded brain activity. But, a more fine-grained evaluation shows that various types of stimuli (determiner, adjective, noun, verb) are represented more dominantly in different layers of the language model. Further, we test the semantic composition capabilities of these networks with respect to the human brain. Semantic composition is defined as the rule-based combination of the parts that constitutes the meaning of the whole. We collect new data and develop a new framework to perform this evaluation incrementally as each word in the sentence is processed in the brain and DNN. As a result, we are able to analyze the effect of the composition function in representing the same word as more of the sentence context becomes available. Our experiments show that DNN models are effective in encoding the sentence being read and are able to predict the word which occurred earlier in the sentence, indicating good composition. We find that in these tests, the right frontal and right temporal brain regions are predicted with best accuracy. Previous research has suggested that these brain regions are responsible for executive and memory function.
As an additional contribution, we propose a new dynamic time warping based distance metric to evaluate alignment between the predicted brain activity versus the observed brain activity. The new metric helps tackle the variability observed in a single subject’s recorded brain activity.Ministry of Human Resource and development India (MHRD), Pratiksha Trust, and CMU BrainHu
Learning through Wikipedia and Generative AI Technologies
No full textThis tutorial will examine the use of Wikipedia and generative AI technologies in asynchronous learning environments. Participants will learn about the research on accountable talk and its impact on student learning, as well as the challenges of implementing the learning principles using Wikipedia in an asynchronous settings. The tutorial will also showcase the potential of generative AI technologies, such as chatbots and language models, to facilitate accountable talk and support student-led discussions in asynchronous learning environments. By the end of the tutorial, participants will have a solid understanding of the potential of generative AI technologies to enhance student learning and scale accountable talk in asynchronous learning environments. Praveen Garimella Director, Digital Learning ISB Hyderabad Vasudeva Varma Professor IIIT Hyderaba
A Data Mining Approach for Detecting Collusion in Unproctored Online Exams
No full textDue to the precautionary measures during the COVID-19 pandemic many
universities offered unproctored take-home exams. We propose methods to detect
potential collusion between students and apply our approach on event log data
from take-home exams during the pandemic. We find groups of students with
suspiciously similar exams. In addition, we compare our findings to a proctored
control group. By this, we establish a rule of thumb for evaluating which cases
are "outstandingly similar", i.e., suspicious cases
- …
