539 research outputs found
Sort by
Exploring Benefits of Transfer Learning in Neural Machine Translation
No full textNeural machine translation is known to require large numbers of parallel training sentences, which generally prevent it from excelling on low-resource language pairs. This thesis explores the use of cross-lingual transfer learning on
neural networks as a way of solving the problem with the lack of resources. We
propose several transfer learning approaches to reuse a model pretrained on a
high-resource language pair. We pay particular attention to the simplicity of
the techniques. We study two scenarios: (a) when we reuse the high-resource
model without any prior modifications to its training process and (b) when we
can prepare the first-stage high-resource model for transfer learning in advance.
For the former scenario, we present a proof-of-concept method by reusing a
model trained by other researchers. In the latter scenario, we present a method
which reaches even larger improvements in translation performance. Apart
from proposed techniques, we focus on an in-depth analysis of transfer learning
techniques and try to shed some light on transfer learning improvements. We
show how our techniques address specific problems of low-resource languages
and are suitable even in high-resource transfer learning. We evaluate the potential drawbacks and behavior by studying transfer learning in various situations,
for example, under artificially damaged training corpora, or with fixed various
model parts
Keyphrase Generation: A Multi-Aspect Survey
No full textExtractive keyphrase generation research has been around since the nineties, but the more advanced abstractive approach based on the encoder-decoder framework and sequence-to-sequence learning has been explored only recently. In fact, more than a dozen of abstractive methods have been proposed in the last three years, producing meaningful keyphrases and achieving state-of-the-art scores. In this survey, we examine various aspects of the extractive keyphrase generation methods and focus mostly on the more recent abstractive methods that are based on neural networks. We pay particular attention to the mechanisms that have driven the perfection of the later. A huge collection of scientific article metadata and the corresponding keyphrases is created and released for the research community. We also present various keyphrase generation and text summarization research patterns and trends of the last two decades
Biases and perils of MT evaluation
No full textVirtually all MT evaluation metrics and techniques, both automatic and manual, have their own perils.
Some of the perils can be considered biases if there are MT systems which can (unfairly) benefit from a given evaluation aspects. In my presentation, I focus on the following aspects of evaluations: sentence-level vs. document-level vs.
document-aware, source-based vs. reference-based, direct assessment vs. comparison-based, fluency-biased vs. adequacy-biased. I also discuss the aspect of translationese and native target/source-language translators and evaluators
Efficiency Metrics for Data-Driven Models: A Text Summarization Case Study
No full textUsing data-driven models for solving text summarization or similar tasks has become very common in the last years. Yet most of the studies report basic accuracy scores only, and nothing is known about the ability of the proposed models to improve when trained on more data. In this paper, we define and propose three data efficiency metrics: data score efficiency, data time deficiency and overall data efficiency. We also propose a simple scheme that uses those metrics and apply it for a more comprehensive evaluation of popular methods on text summarization and title generation tasks. For the latter task, we process and release a huge collection of 35 million abstract-title pairs from scientific articles. Our results reveal that among the tested models, the Transformer is the most efficient on both tasks
Replacing Linguists with Dummies: A Serious Need for Trivial Baselinesin Multi-Task Neural Machine Translation
No full textRecent developments in machine translation experiment with the idea that a model can
improve the translation quality by performing multiple tasks, e.g., translating from source to
target and also labeling each source word with syntactic information. The intuition is that the
network would generalize knowledge over the multiple tasks, improving the translation performance, especially in low resource conditions. We devised an experiment that casts doubt on
this intuition. We perform similar experiments in both multi-decoder and interleaving setups
that label each target word either with a syntactic tag or a completely random tag. Surprisingly, we show that the model performs nearly as well on uncorrelated random tags as on true
syntactic tags. We hint some possible explanations of this behavior.
The main message from our article is that experimental results with deep neural networks
should always be complemented with trivial baselines to document that the observed gain is
not due to some unrelated properties of the system or training effects. True confidence in where
the gains come from will probably remain problematic anyway
Findings of the 2019 Conference on Machine Translation (WMT19)
No full textThis paper presents the results of the premier
shared task organized alongside the Conference on Machine Translation (WMT) 2019.
Participants were asked to build machine
translation systems for any of 18 language
pairs, to be evaluated on a test set of news
stories. The main metric for this task is human judgment of translation quality. The task
was also opened up to additional test suites to
probe specific aspects of translation
Towards Automatic Minuting of Meetings
No full textMany meetings of different kinds will potentially benefit from technological support like automatic creation of meeting minutes. To prepare a reasonable automation, we need to have a detailed understanding of common types of meetings, of the linguistic properties and commonalities in the structure of meeting minutes, as well as of methods for their automation.
In this paper, we summarize the quality criteria and linguistic properties of meeting minutes, describe the available meeting corpora and meeting datasets and propose a classification of meetings and minutes types. Furthermore, we analyze the methods and tools for automatic minuting with respect to their use with existing types of datasets. We summarize the obtained knowledge with respect to our goal of designing automatic minuting and present our first steps in this direction
Defining Verbal Synonyms: between Syntax and Semantics
No full textWhile studying verbal synonymy, we have investigated the relation between syntax and semantics
in hope that the exploration of this relationship will help us to get more insight into the question
of synonymy as the relationship relating (similar) meanings between different lexemes. Most
synonym lexicons (Wordnets and similar thesauri) are based on an intuition about the similarity
of word meanings, or on notions like “semantic roles.” In some cases, syntax is also taken into
account, but we have found no annotation and/or evaluation experiment to see how strongly
can syntax contribute to synonym specification. We have prepared an annotation experiment
for which we have used two treebanks (Czech and English) from the Universal Dependencies
(UD) set of parallel corpora (PUDs) in order to see how strong correlation exists between syntax
and the assignment of verbs in context to pre-determined (bilingual) classes of synonyms. The
resulting statistics confirmed that while syntax does support decisions about synonymy, such
support is not strong enough and that more semantic criteria are indeed necessary. The results of
the annotation will also help to further improve rules and specifications for creating synonymous
classes. Moreover, we have collected evidence that the annotation setup that we have used
can identify synonym classes to be merged, and the resulting data (which we plan to publish
openly) can possibly serve for the evaluation of automatic methods used in this area
CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies
No full textEvery year, the Conference on Computational Natural Language Learning (CoNLL) features a shared task, in which participants train and test their learning systems on the same data sets. In 2018, one of two tasks was devoted to learning dependency parsers for a large number of languages, in a real-world setting without any gold-standard annotation on test input
Improving a Neural-based Tagger for Multiword Expression Identification
No full textIn this paper, we present a set of improvements introduced to MUMULS, a tagger for the automatic detection of verbal multiword expressions. Our tagger participated in the PARSEME shared task and it was the only one based on neural networks. We show that character-level embeddings can improve the performance, mainly by reducing the out-of-vocabulary rate. Furthermore, replacing the softmax layer in the decoder by a conditional random field classifier brings additional improvements. Finally, we compare different context-aware feature representations of input tokens using various encoder architectures. The experiments on Czech show that the combination of character-level embeddings using a convolutional network, self-attentive encoding layer over the word representations and an output conditional random field classifier yields the best empirical results