14 research outputs found
Embeddings for word sense disambiguation: an evaluation study
No full textRecent years have seen a dramatic growth in the popularity of word embeddings mainly owing to their ability to capture semantic information from massive amounts of textual content. As a result, many tasks in Natural Language Processing have tried to take advantage of the potential of these distributional models. In this work, we study how word embeddings can be used in Word Sense Disambiguation, one of the oldest tasks in Natural Language Processing and Artificial Intelligence. We propose different methods through which word embeddings can be leveraged in a state-of-the-art supervised WSD system architecture, and perform a deep analysis of how different parameters affect performance. We show how a WSD system that makes use of word embeddings alone, if designed properly, can provide significant performance improvement over a state-of-the-art WSD system that incorporates several standard WSD features
Neural-grounded semantic representations and word sense disambiguation: a mutually beneficial relationship
No full textLanguage, in both the written and the oral forms, is the ground basis of living in society. The same basic kinds of rules and representations are shared across all the languages. Understand those rules is the objective of Natural Language Processing (NLP), the computerized discipline responsible to analyze and generate language. Building complex computational systems that mimic the human language and are capable to interact and collaborate with us is the holy grail of Natural Language Processing. Semantic representations are the rock-solid foundation on which many successful applications of NLP depend. Their main purpose is to extract and highlight the most important semantic features of textual data. Whereas over the years different approaches have been presented, lately, embeddings have become the dominant paradigm on vectorial representation of items. Currently, many outstanding NLP tasks rely on embeddings to achieve their performance. Embeddings are semantic spaces that carry valuable syntactic and semantic information. The name groups a set of feature learning techniques based on neural networks. Concretely, these techniques are capable to learn semantic spaces that effectively represent words as low-dimensional continuous vectors. They also maintain the structure of language by representing diverse lexical and semantic relations by a relation-specific vector offset. With the increasing amount of available text, as well as the increased computing power, techniques which take advantage of large volumes of unstructured data, as word embeddings, have become the prevailing approach of semantic representation of natural language. However, despite their enormous success, common word-embeddings approaches came with two inherent flaws: these representations are incapable to handle ambiguity, as senses of polysemous words are aggregated into single vectors. In addition, most word embeddings rely only on statistical information of word occurrences, leaving aside existing rich knowledge of structured data. To tackle the problem of polysemy, a fundamental task of Natural Language Processing (NLP), Word Sense Disambiguation (WSD), seems particularly suitable. The task, an open problem in the discipline, aims at identifying the correct meaning of word based given its context. Concretely, it links each word occurrence to a sense from a predefined inventory. Most successful approaches for WSD combine the use of unstructured data, manually annotated datasets and semantic resources. In the present thesis we address the issue of of ambiguity in semantic representations from a multimodal perspective. Firstly, we introduce and investigate new neural-based approaches to build better word and sense embeddings relying on both statistical data and prior semantic knowledge. We employ diverse techniques of WSD for linking word occurrences to their correct meaning on large amounts of raw corpora. Then, we use the resulting data as training input for learning the embeddings. We show the quality of these representations by evaluating them on standard semantic similarity frameworks reporting state-of-the-art performance on multiple datasets. Secondly, we show how these representations are capable to create better WSD systems. We introduce a new way to leverage word representations which outperforms current WSD approaches in both supervised and unsupervised configurations. We show that our WSD framework, based solely on embeddings, is capable to surpass WSD approaches based on standard features. Thirdly, we propose two new technique for leveraging semantic-annotated data. We incorporate more semantic features resulting in an increment in the performance compared with our initial approaches. We close the loop by showing that our semantic representations enhanced with WSD are also suitable for improving the task of WSD itself
Embedding Words and Senses Together via Joint Knowledge-Enhanced Training
No full textWord embeddings are widely used in Nat-ural Language Processing, mainly due totheir success in capturing semantic infor-mation from massive corpora. However,their creation process does not allow thedifferent meanings of a word to be auto-matically separated, as it conflates theminto a single vector. We address this issueby proposing a new model which learnsword and sense embeddings jointly. Ourmodel exploits large corpora and knowl-edge from semantic networks in order toproduce a unified vector space of wordand sense embeddings. We evaluate themain features of our approach both qual-itatively and quantitatively in a variety oftasks, highlighting the advantages of theproposed method in comparison to state-of-the-art word- and sense-based models
SensEmbed: Learning sense embeddings for word and relational similarity
No full textWord embeddings have recently gained considerable popularity for modeling words in different Natural Language Processing (NLP) tasks including semantic similarity measurement. However, notwithstanding their success, word embeddings are by their very nature unable to capture polysemy, as different meanings of a word are conflated into a single representation. In addition, their learning process usually relies on massive corpora only, preventing them from taking advantage of structured knowledge. We address both issues by proposing a multifaceted approach that transforms word embeddings to the sense level and leverages knowledge from a large semantic network for effective semantic similarity measurement. We evaluate our approach on word similarity and relational similarity frameworks, reporting state-of-the-art performance on multiple datasets
Improving Commonsense Causal Reasoning by Adversarial Training and Data Augmentation
No full textDetermining the plausibility of causal relations between clauses is a commonsense reasoning task that requires complex inference ability. The general approach to this task is to train a large pretrained language model on a specific dataset. However, the available training data for the task is often scarce, which leads to instability of model training or reliance on the shallow features of the dataset. This paper presents a number of techniques for making models more robust in the domain of causal reasoning. Firstly, we perform adversarial training by generating perturbed inputs through synonym substitution. Secondly, based on a linguistic theory of discourse connectives, we perform data augmentation using a discourse parser for detecting causally linked clauses in large text, and a generative language model for generating distractors. Both methods boost model performance on the Choice of Plausible Alternatives (COPA) dataset, as well as on a Balanced COPA dataset, which is a modified version of the original data that has been developed to avoid superficial cues, leading to a more challenging benchmark. We show a statistically significant improvement in performance and robustness on both datasets, even with only a small number of additionally generated data points
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
No full textTo alleviate the problem of structured databases' limited coverage, recent
task-oriented dialogue systems incorporate external unstructured knowledge to
guide the generation of system responses. However, these usually use word or
sentence level similarities to detect the relevant knowledge context, which
only partially capture the topical level relevance. In this paper, we examine
how to better integrate topical information in knowledge grounded task-oriented
dialogue and propose ``Topic-Aware Response Generation'' (TARG), an end-to-end
response generation model. TARG incorporates multiple topic-aware attention
mechanisms to derive the importance weighting scheme over dialogue utterances
and external knowledge sources towards a better understanding of the dialogue
history. Experimental results indicate that TARG achieves state-of-the-art
performance in knowledge selection and response generation, outperforming
previous state-of-the-art by 3.2, 3.6, and 4.2 points in EM, F1 and BLEU-4
respectively on Doc2Dial, and performing comparably with previous work on
DSTC9; both being knowledge-grounded task-oriented dialogue datasets.Comment: Findings of EMNLP 202
Enhancing Transformers with Gradient Boosted Decision Trees for NLI Fine-Tuning
No full textTransfer learning has become the dominant paradigm for many natural language
processing tasks. In addition to models being pretrained on large datasets,
they can be further trained on intermediate (supervised) tasks that are similar
to the target task. For small Natural Language Inference (NLI) datasets,
language modelling is typically followed by pretraining on a large (labelled)
NLI dataset before fine-tuning with each NLI subtask. In this work, we explore
Gradient Boosted Decision Trees (GBDTs) as an alternative to the commonly used
Multi-Layer Perceptron (MLP) classification head. GBDTs have desirable
properties such as good performance on dense, numerical features and are
effective where the ratio of the number of samples w.r.t the number of features
is low. We then introduce FreeGBDT, a method of fitting a GBDT head on the
features computed during fine-tuning to increase performance without additional
computation by the neural network. We demonstrate the effectiveness of our
method on several NLI datasets using a strong baseline model (RoBERTa-large
with MNLI pretraining). The FreeGBDT shows a consistent improvement over the
MLP classification head.Comment: Findings of ACL 202
Conversation Graph: Data Augmentation, Training, and Evaluation for Non-Deterministic Dialogue Management
No full textHumanRankEval: Automatic Evaluation of LMs as Conversational Assistants
No full textLanguage models (LMs) as conversational assistants recently became popular
tools that help people accomplish a variety of tasks. These typically result
from adapting LMs pretrained on general domain text sequences through further
instruction-tuning and possibly preference optimisation methods. The evaluation
of such LMs would ideally be performed using human judgement, however, this is
not scalable. On the other hand, automatic evaluation featuring auxiliary LMs
as judges and/or knowledge-based tasks is scalable but struggles with assessing
conversational ability and adherence to instructions. To help accelerate the
development of LMs as conversational assistants, we propose a novel automatic
evaluation task: HumanRankEval (HRE). It consists of a large-scale, diverse and
high-quality set of questions, each with several answers authored and scored by
humans. To perform evaluation, HRE ranks these answers based on their
log-likelihood under the LM's distribution, and subsequently calculates their
correlation with the corresponding human rankings. We support HRE's efficacy by
investigating how efficiently it separates pretrained and instruction-tuned LMs
of various sizes. We show that HRE correlates well with human judgements and is
particularly responsive to model changes following instruction-tuning.Comment: Accepted to NACCL 2024 main conferenc
