131 research outputs found
USFD’s phrase-level quality estimation systems
No full text© 2016 The Authors. Published by Association for Computational Linguistics. This is an open access article available under a Creative Commons licence.
The published version can be accessed at the following link on the publisher’s website: http://dx.doi.org/10.18653/v1/W16-2386Logacheva, V., Blain, F. and Specia, L. (2016) USFD’s phrase-level quality estimation systems. In, Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, Bojar, O., Buck, C., Chatterjee, R., Federmann, C. et al. (eds.) Stroudsburg, PA: Association for Computational Linguistics, pp. 800-805.This work was supported by the EXPERT (EU FP7 Marie Curie ITN No. 317471, Varvara Logacheva) and the QT21 (H2020 No. 645452, Lucia Specia, Fred´ eric Blain) projects
Phrase level segmentation and labelling of machine translation errors
No full text© 2016 The Authors. Published by European Language Resources Association (ELRA). This is an open access article available under a Creative Commons licence.
The published version can be accessed at the following link on the publisher’s website: https://www.aclweb.org/anthology/L16-1356/This paper presents our work towards a novel approach for Quality Estimation (QE) of machine translation based on sequences of adjacent words, the so-called phrases. This new level of QE aims to provide a natural balance between QE at word and sentence-level, which are either too fine grained or too coarse levels for some applications. However, phrase-level QE implies an intrinsic challenge: how to segment a machine translation into sequence of words (contiguous or not) that represent an error. We discuss three possible segmentation strategies to automatically extract erroneous phrases. We evaluate these strategies against annotations at phrase-level produced by humans, using a new dataset collected for this purpose.The authors would like to thanks all the annotators who helped to create the first version of gold-standard annotations at phrase-level. This work was supported by the QT21 (H2020 No. 645452, Lucia Specia, Fred´ eric Blain) and EX-PERT (EU FP7 Marie Curie ITN No. 317471, Varvara Logacheva) projects
Varvara Stepanova: Incomplete Design History Podcast
No full textVarvara Stepanova was a leader of the constructivist movement and co-author of the constructivist manifesto. Described as “a frenzied artist,” she designed books, magazines, posters, advertisements, as well as textiles, clothing, and costumes. On top of that, she was an author and poet. Her design style and aesthetic was avant garde, modern, and often characterized by simplicity and geometric forms and patterns, but Varvara was never content with stasis. She constantly evolved her style and worked to develop new concepts and ideas. As a co-founder of the Constructivist movement, her work typifies the aesthetic and philosophies of Constructivism. While Varavara worked on many of her own projects, she also did many together with her husband, Alexander Rodchenko, another well-known Constructivist designer. Her work often gets linked to Rodchenko’s, and it’s Rodchenko who gets mentioned in histories of graphic design and credit for designs that should be attributed to Stepanova as well. At a time when women were largely still expected to be no more than wives and mothers, Stepanova was a woman who made an unmistakable mark on art and design.NoUniversity of Central Oklahoma. School of Design
Human Feedback in Statistical Machine Translation
No full textThe thesis addresses the challenge of improving Statistical Machine Translation (SMT) systems via feedback given by humans on translation quality.
The amount of human feedback available to systems is inherently low due to cost and time limitations. One of our goals is to simulate such information by automatically generating pseudo-human feedback.
This is performed using Quality Estimation (QE) models. QE is a technique for predicting the quality of automatic translations without comparing them to oracle (human) translations, traditionally at the sentence or word levels.
QE models are trained on a small collection of automatic translations manually labelled for quality, and then can predict the quality of any number of unseen translations.
We propose a number of improvements for QE models in order to increase the reliability of pseudo-human feedback.
These include strategies to artificially generate instances for settings where QE training data is scarce.
We also introduce a new level of granularity for QE: the level of phrases. This level aims to improve the quality of QE predictions by better modelling inter-dependencies among errors at word level, and in ways that are tailored to phrase-based SMT, where the basic unit of translation is a phrase. This can thus facilitate work on incorporating human feedback during the translation process.
Finally, we introduce approaches to incorporate pseudo-human feedback in the form of QE predictions in SMT systems. More specifically, we use quality predictions to select the best translation from a number of alternative suggestions produced by SMT systems, and integrate QE predictions into an SMT system decoder in order to guide the translation generation process
Conversational Intelligence Challenge: Accelerating Research with Crowd Science and Open Source
No full textWMT17 Quality Estimation Shared Test Data
No full textTest data for the WMT17 QE task. Train data can be downloaded from http://hdl.handle.net/11372/LRT-1974
This shared task will build on its previous five editions to further examine automatic methods for estimating the quality of machine translation output at run-time, without relying on reference translations. We include word-level, phrase-level and sentence-level estimation. All tasks will make use of a large dataset produced from post-editions by professional translators. The data will be domain-specific (IT and Pharmaceutical domains) and substantially larger than in previous years. In addition to advancing the state of the art at all prediction levels, our goals include:
- To test the effectiveness of larger (domain-specific and professionally annotated) datasets. We will do so by increasing the size of one of last year's training sets.
- To study the effect of language direction and domain. We will do so by providing two datasets created in similar ways, but for different domains and language directions.
- To investigate the utility of detailed information logged during post-editing. We will do so by providing post-editing time, keystrokes, and actual edits.
This year's shared task provides new training and test datasets for all tasks, and allows participants to explore any additional data and resources deemed relevant. A in-house MT system was used to produce translations for all tasks. MT system-dependent information can be made available under request. The data is publicly available but since it has been provided by our industry partners it is subject to specific terms and conditions. However, these have no practical implications on the use of this data for research purposes
WMT17 Quality Estimation Shared Task Training and Development Data
No full textTraining and development data for the WMT17 QE task. Test data will be published as a separate item.
This shared task will build on its previous five editions to further examine automatic methods for estimating the quality of machine translation output at run-time, without relying on reference translations. We include word-level, phrase-level and sentence-level estimation. All tasks will make use of a large dataset produced from post-editions by professional translators. The data will be domain-specific (IT and Pharmaceutical domains) and substantially larger than in previous years. In addition to advancing the state of the art at all prediction levels, our goals include:
- To test the effectiveness of larger (domain-specific and professionally annotated) datasets. We will do so by increasing the size of one of last year's training sets.
- To study the effect of language direction and domain. We will do so by providing two datasets created in similar ways, but for different domains and language directions.
- To investigate the utility of detailed information logged during post-editing. We will do so by providing post-editing time, keystrokes, and actual edits.
This year's shared task provides new training and test datasets for all tasks, and allows participants to explore any additional data and resources deemed relevant. A in-house MT system was used to produce translations for all tasks. MT system-dependent information can be made available under request. The data is publicly available but since it has been provided by our industry partners it is subject to specific terms and conditions. However, these have no practical implications on the use of this data for research purposes
- …
