539 research outputs found
Sort by
ELITR Non-Native Speech Translation at IWSLT 2020
No full textThis paper is an ELITR system submission for the non-native speech translation task at IWSLT 2020. We describe systems for offline ASR, real-time ASR, and our cascaded approach to offline SLT and real-time SLT. We select our primary candidates from a pool of pre-existing systems, develop a new end-to-end general ASR system, and a hybrid ASR trained on non-native speech. The provided small validation set prevents us from carrying out a complex validation, but we submit all the unselected candidates for contrastive evaluation on the test set
Automating Text Naturalness Evaluation of NLG Systems
No full textAutomatic methods and metrics that assess various quality criteria of automatically generated texts are important for developing NLG systems because they produce repeatable results and allow for a fast development cycle. We present here an attempt to automate the evaluation of text naturalness which is a very important characteristic of natural language generation methods. Instead of relying on human participants for scoring or labeling the text samples, we propose to automate the process by using a human likeliness metric we define and a discrimination procedure based on large pretrained language models with their probability distributions. We analyze the text probability fractions and observe how they are influenced by the size of the generative and discriminative models involved in the process. Based on our results, bigger generators and larger pretrained discriminators are more appropriate for a better evaluation of text naturalness. A comprehensive validation procedure with human participants is required as follow up to check how well this automatic evaluation scheme correlates with human judgments
Universal Dependency Treebanks for Low-Resource Indian Languages: The Case of Bhojpuri
No full textThis paper presents the first dependency treebank for Bhojpuri, an Indo-Aryan language. Bhojpuri is one of the resource-poor Indian languages. The objective of the Bhojpuri Treebank (BHTB) project is to provide a substantial, syntactically annotated treebank for Bhojpuri which helps in building language technological tools. This project will also help in cross-lingual learning and typological research. Currently, the treebank consists of 4,881 tokens using the annotation scheme of Universal Dependencies (UD). We develop a Bhojpuri tagger and parser using the machine learning approach. The accuracy of the model is 57.49% UAS, 45.50% LAS, 79.69% UPOS accuracy and 77.64% XPOS accuracy. Finally, we discuss linguistic analysis and annotation process of the Bhojpuri UD treebank
WMT20 Document-Level Markable Error Exploration
No full textEven though sentence-centric metrics are
used widely in machine translation evaluation,
document-level performance is at least equally
important for professional usage. In this paper,
we bring attention to detailed document-level
evaluation focused on markables (expressions
bearing most of the document meaning) and
the negative impact of various markable error
phenomena on the translation.
For an annotation experiment of two phases,
we chose Czech and English documents translated
by systems submitted to WMT20 News
Translation Task. These documents are from
the News, Audit and Lease domains. We show
that the quality and also the kind of errors
varies significantly among the domains. This
systematic variance is in contrast to the automatic
evaluation results.
We inspect which specific markables are problematic
for MT systems and conclude with an
analysis of the effect of markable error types
on the MT performance measured by humans
and automatic evaluation tools
Proceedings of the 1st International Workshop on Language Technology Platforms (IWLTP 2020, co-located with LREC 2020)
No full textWith the increasing number of platforms, grids and infrastructures in the wider area of Language
Technologies (LT), NLP, NLU, speech, interaction and language-centric AI, there is also a growing need
for sharing experiences, approaches and best practices to learn and benefit from the work of others and
also, practically, to start a collaboration towards platform interoperability.
The 1st InternationalWorkshop for Language Technology Platforms (IWLTP 2020) addresses all smaller
and larger language grids, language-related infrastructures, platform initiatives as well as collaborative
research projects that touch upon LT platforms, especially platform interoperability and related topics,
both in Europe and world-wide. Its objective is to exchange and discuss observations, experiences,
solutions, best practices as well as current and future challenges. The workshop also addresses the issue
of fragmentation in the Language Technology landscape. Instead of “platform islands” that simply exist
side by side, possibly even competing with each other, initiatives should discuss how their platforms
can be made interoperable and how they can interact with one another to create synergies towards a
productive LT platform ecosystem.
The EU project European Language Grid (ELG; 2019-2021) is creating a platform that will provide
thousands of data sets and hundreds of LT services. ELG aims to promote technologies tailored to all
European languages and cultures, adapted to their social and economic needs. At the same time, there
are several established platforms or infrastructure-related initiatives as well as emerging new ones, both
on the European but also on the national level as well as on other continents. Some of the initiatives are
more language-related and have a strong industry focus, others are mainly research-oriented. Moreover,
there are digital public service initiatives, and platforms, in which language is only one aspect of many.
With all these established and emerging initiatives, there is a risk of even stronger fragmentation in the
Language Technology field, which is already highly fragmented, at least in Europe. Our approach is to
bring these initiatives together to discuss ways not only of preventing further fragmentation but, crucially,
of reversing it. This will only be possible if interoperability and mutual data exchange is ensured and if
metadata formats and technical requirements are compatible, among others.
A total of 30 papers were submitted to IWLTP 2020, 17 of which were accepted (acceptance rate: 56.7%).
The organisers would like to thank all contributors for their valuable submissions and all members of the
Programme Committee for reviewing the submitted papers. Due to the ongoing SARSCoV-2 pandemic,
the workshop cannot be held as originally foreseen. Together with the organisers of LREC 2020 we will
explore if we can organise the workshop at a later point in time or if we can organise it as a virtual event.
G. Rehm, K. Bontcheva, K. Choukri, J. Hajic, S. Piperidis, A. Vasil,jevs May 2020
ii
Costra 1.1: An Inquiry into Geometric Properties of Sentence Spaces
No full textIn this paper, we present a new dataset for testing geometric properties of sentence embeddings spaces. In particular, we concentrate on examining how well sentence embeddings capture complex phenomena such paraphrases, tense or generalization.
The dataset is a direct expansion of Costra 1.0, which we extended with more sentences and sentence comparisons.
We show that available off-the-shelf embeddings do not possess essential attributes such as having synonymous sentences embedded closer to each other than similar sentences with a significantly different meaning.
On the other hand, some embeddings appear to capture the linear order of sentence aspects such as style (formality and simplicity of the language) or time (past to future)
FINDINGS OF THE IWSLT 2020 EVALUATION CAMPAIGN
No full textThe evaluation campaign of the International Conference on Spoken Language Translation (IWSLT 2020) featured this year six challenge tracks: (i) Simultaneous speech translation, (ii) Video speech translation, (iii) Offline speech translation, (iv) Conversational speech translation, (v) Open domain translation, and (vi) Non-native speech translation. A total of teams participated in at least one of the tracks. This paper introduces each track’s goal, data and evaluation metrics, and reports the results of the received submissions
ELITR: European Live Translator
No full textELITR (European Live Translator) project
aims to create a speech translation system
for simultaneous subtitling of conferences
and online meetings targetting up to 43
languages. The technology is tested by
the Supreme Audit Office of the Czech Republic and by alfaview®, a German online
conferencing system. Other project goals
are to advance document-level and multilingual machine translation, automatic
speech recognition, and meeting summarization
Cross-Lingual Information Retrieval DEMO
No full textThe Cross-Lingual Information Retrieval (CLIR) tool allows you to search in documents in various languages, using your own language both to enter the search query as well as to display the search results, thanks to automated machine translation.
In the demo, you can search in audits and other documents published by the Czech and Belgian Supreme Audit Institutions. The demo works in English, German, French and Czech
Transforming machine translation: a deep learning system reaches news translation quality comparable to human professionals
No full textThe quality of human translation was long thought to be unattainable for computer translation systems. In this study, we present a deep-learning system, CUBBITT, which challenges this view. In a context-aware blind evaluation by human judges, CUBBITT significantly outperformed professional-agency English-to-Czech news translation in preserving text meaning (translation adequacy). While human translation is still rated as more fluent, CUBBITT is shown to be substantially more fluent than previous state-of-the-art systems. Moreover, most participants of a Translation Turing test struggle to distinguish CUBBITT translations from human translations. This work approaches the quality of human translation and even surpasses it in adequacy in certain circumstances. This suggests that deep learning may have the potential to replace humans in applications where conservation of meaning is the primary aim