1,721,204 research outputs found
A Framework for Diagnostic Evaluation of Machine Translation Based on Linguistic Checkpoints
No full textThis paper describes an approach to the diagnostic
evaluation of machine translation (MT) based on linguistic checkpoints, which can provide valuable information both to the developers and to the end-users of MT systems.
We present a flexible framework and a new tool, DELiC4MT, for fine-grained diagnostic MT evaluation which can be extended to any
language pair and applied to any evaluation target, once the phenomena of interest are
covered by the linguistic analysis. As a case study, we evaluate the CoSyne1 MT software against four leading web-based MT systems across a set of linguistic phenomena for three language pairs (from German, Italian and Dutch into English)
A Comparative Evaluation of Research vs. Online Machine Translation Systems
No full textThis paper reports MT evaluation experiments
that were conducted at the end
of year 1 of the EU-funded CoSyne1 project for three language combinations,
considering translations from German, Italian and Dutch into English. We present a comparative evaluation of the MT software developed within the project against four of the leading free web-based
MT systems across a range of
state-of-the-art automatic evaluation metrics. The data sets from the news domain
that were created and used for training purposes and also for this evaluation exercise,
which are available to the research community, are also described. The evaluation results for the news domain
are very encouraging: the CoSyne MT software consistently beats the rule-based MT systems, and for translations from Italian and Dutch into English in particular the scores given by some of the standard
automatic evaluation metrics are not too distant from those obtained by wellestablished
statistical online MT systems
Digital language equality: definition, metric, dashboard
Full text linkThis chapter presents the concept of Digital Language Equality (DLE) that
was at the heart of the European Language Equality (ELE) initiative, and describes
the DLE Metric, which includes technological factors (TFs) and contextual factors
(CFs): the former concern the availability of Language Resources and Technologies
(LRTs) for the languages of Europe, based on the data included in the European
Language Grid (ELG) catalogue, while the latter reflect the broader socio-economic
contexts and ecosystems of the languages, as these determine the potential for LRT
development. The chapter discusses related work, presents the DLE definition and
describes how it was implemented through the DLE Metric, explaining how the TFs
and CFs were quantified. The resulting scores of the DLE Metric for Europe’s languages
can be visualised and compared through the interactive DLE dashboard, to
monitor the progress towards DLE in Europe
Relating Translation Quality Barriers to Source-Text Properties
No full textThis paper aims to automatically identify which linguistic phenomena represent barriers to better MT quality. We focus on the
translation of news data for two bidirectional language pairs: EN↔ES and EN↔DE. Using the diagnostic MT evaluation toolkit
DELiC4MT and a set of human reference translations, we relate translation quality barriers to a selection of 9 source-side PoS-based
linguistic checkpoints. Using output from the winning SMT, RbMT, and hybrid systems of the WMT 2013 shared task, translation
quality barriers are investigated (in relation to the selected linguistic checkpoints) according to two main variables: (i) the type of the
MT approach, i.e. statistical, rule-based or hybrid, and (ii) the human evaluation of MT output, ranked into three quality groups
corresponding to good, near miss and poor. We show that the combination of manual quality ranking and automatic diagnostic
evaluation on a set of PoS-based linguistic checkpoints is able to identify the specific quality barriers of different MT system types
across the four translation directions under consideration
Example-based controlled translation
Full text linkThe first research on integrating controlled language data in an Example-Based Machine Translation (EBMT) system was published in [Gough & Way, 2003]. We improve on their sub-sentential alignment algorithm to populate the system’s databases with more than six times as many potentially useful fragments. Together with two simple novel improvements—correcting mistranslations in the lexicon, and allowing multiple translations in the lexicon—translation quality improves considerably when target language
translations are constrained. We also develop the first EBMT system which attempts to filter the source language data using controlled language specifications. We provide
detailed automatic and human evaluations of a number of experiments carried out to test the quality of the system. We observe that our system outperforms Logomedia in a number of tests. Finally, despite conflicting results from different automatic evaluation metrics, we observe a preference for controlling the source data rather than the target translations
Controlled generation in example-based machine translation
Full text linkThe theme of controlled translation is currently in vogue in the area of MT. Recent research (Sch¨aler et al., 2003;
Carl, 2003) hypothesises that EBMT systems are perhaps best suited to this challenging task. In this paper, we present
an EBMT system where the generation of the target string is filtered by data written according to controlled language
specifications. As far as we are aware, this is the only research available on this topic. In the field of controlled language applications, it is more usual to constrain the source language in this way rather than the target. We translate a small corpus of controlled English into French using the on-line MT system Logomedia, and seed the memories of our EBMT system with a set of automatically induced lexical resources using the Marker Hypothesis as a segmentation tool. We test our system on a large set of sentences extracted from a Sun Translation Memory, and provide both an automatic and a human evaluation. For comparative purposes, we also provide results for Logomedia itself
An example-based approach to translating sign language
Full text linkUsers of sign languages are often forced to use a language in which they have reduced competence simply because documentation in their preferred format is not available. While some research exists on translating between natural and sign languages, we present here what we believe to be the first attempt to tackle this problem using an example-based (EBMT) approach.
Having obtained a set of English–Dutch Sign Language examples, we employ an approach to EBMT using the ‘Marker Hypothesis’ (Green, 1979), analogous to the successful system of (Way & Gough, 2003), (Gough & Way, 2004a) and (Gough & Way, 2004b). In a set of experiments, we show that
encouragingly good translation quality may be obtained using such an approach
- …
