1,721,126 research outputs found
Automatic evaluation of generation and parsing for machine translation with automatically acquired transfer rules
Full text linkThis paper presents a new method of evaluation for generation and parsing components of transfer-based MT systems where the transfer rules have been automatically
acquired from parsed sentence-aligned bitext corpora. The method provides a means of quantifying the upper bound imposed on the MT system by the quality of the parsing
and generation technologies for the target language. We include experiments to calculate this upper bound for both handcrafted and automatically induced parsing and generation technologies currently in use by transfer-based MT systems
Robust PCFG-based generation using automatically acquired LFG approximations
No full textWe present a novel PCFG-based architecture for robust probabilistic generation based on wide-coverage LFG approximations (Cahill et al., 2004) automatically
extracted from treebanks, maximising the probability of a tree given an f-structure. We evaluate our approach using string-based evaluation. We currently achieve coverage of 95.26%, a BLEU score of 0.7227 and string accuracy of 0.7476 on the Penn-II WSJ Section 23 sentences of length ≤20
Relating Translation Quality Barriers to Source-Text Properties
No full textThis paper aims to automatically identify which linguistic phenomena represent barriers to better MT quality. We focus on the
translation of news data for two bidirectional language pairs: EN↔ES and EN↔DE. Using the diagnostic MT evaluation toolkit
DELiC4MT and a set of human reference translations, we relate translation quality barriers to a selection of 9 source-side PoS-based
linguistic checkpoints. Using output from the winning SMT, RbMT, and hybrid systems of the WMT 2013 shared task, translation
quality barriers are investigated (in relation to the selected linguistic checkpoints) according to two main variables: (i) the type of the
MT approach, i.e. statistical, rule-based or hybrid, and (ii) the human evaluation of MT output, ranked into three quality groups
corresponding to good, near miss and poor. We show that the combination of manual quality ranking and automatic diagnostic
evaluation on a set of PoS-based linguistic checkpoints is able to identify the specific quality barriers of different MT system types
across the four translation directions under consideration
Parser-based retraining for domain adaptation of probabilistic generators
Full text linkWhile the effect of domain variation on Penn-treebank-
trained probabilistic parsers has been investigated in previous work, we study its effect on a Penn-Treebank-trained probabilistic generator. We show that applying the generator to data from the British National Corpus
results in a performance drop (from a BLEU score of 0.66 on the standard WSJ test set to a BLEU score of 0.54 on our BNC test set). We develop a generator retraining method where the domain-specific training data is automatically
produced using state-of-the-art parser output. The retraining method recovers a substantial portion of the performance drop, resulting in a generator which achieves a BLEU score of 0.61 on our BNC test data
Josef van Genabith
No full textWe report findings from a user study with professional post-editors using a translation recommendation framework (He et al., 2010) to integrate Statistical Machine Translation (SMT) output with Translation Memory (TM) systems. The framework recommends SMT outputs to a TM user when it predicts that SMT outputs are more suitable for postediting than the hits provided by the TM. We analyze the effectiveness of the model as well as the reaction of potential users. Based on the performance statistics and the users’ comments, we find that translation recommendation can reduce the workload of professional post-editors and improve the acceptance of MT in the localization industry.
Accurate and robust LFG-based generation for Chinese
Full text linkWe describe three PCFG-based models for Chinese sentence realisation from Lexical-Functional Grammar (LFG) f-structures. Both the lexicalised model and the history-based model improve on the accuracy of a simple
wide-coverage PCFG model by adding lexical and contextual information to weaken inappropriate independence assumptions implicit in the PCFG models. In addition, we provide techniques for lexical smoothing and rule smoothing to increase the generation coverage. Trained on 15,663 automatically LFG fstructure annotated sentences of the Penn Chinese treebank and tested on 500 sentences randomly
selected from the treebank test set, the lexicalised model achieves a BLEU score of 0.7265 at 100% coverage, while the historybased model achieves a BLEU score of 0.7245
also at 100% coverage
Adapting a WSJ-trained parser to grammatically noisy text
Full text linkWe present a robust parser which is trained on a treebank of ungrammatical sentences. The treebank is created automatically by modifying Penn treebank sentences so that they contain one or more syntactic errors. We evaluate
an existing Penn-treebank-trained parser on the ungrammatical treebank to see how it reacts to noise in the form of grammatical errors. We re-train this parser on the training section of the ungrammatical treebank, leading
to an significantly improved performance on the ungrammatical test sets. We show how a classifier can be used to prevent performance degradation on the original grammatical data
Exploiting multi-word units in history-based probabilistic generation
Full text linkWe present a simple history-based model for sentence generation from LFG f-structures, which improves on the accuracy of previous models by breaking down PCFG independence assumptions so that more f-structure conditioning context is used in the prediction of grammar rule expansions. In addition, we present work on experiments with named entities and other multi-word units,
showing a statistically significant improvement of generation accuracy. Tested on section 23 of the PennWall Street Journal Treebank, the techniques described in this paper improve BLEU scores from 66.52 to 68.82, and coverage from 98.18% to 99.96%
Large-scale induction and evaluation of lexical resources from the Penn-II treebank
Full text linkIn this paper we present a methodology for extracting
subcategorisation frames based on an automatic LFG f-structure annotation algorithm for the Penn-II Treebank. We extract abstract syntactic function-based subcategorisation frames (LFG semantic forms), traditional CFG categorybased subcategorisation frames as well as mixed
function/category-based frames, with or without preposition information for obliques and particle information for particle verbs. Our approach does not predefine frames, associates probabilities with frames conditional on the lemma, distinguishes between active and passive frames, and fully reflects the effects of long-distance dependencies in the source data structures. We extract 3586 verb lemmas,
14348 semantic form types (an average of 4 per lemma) with 577 frame types. We present a large-scale evaluation of the complete set of forms extracted against the full COMLEX resource
Dependency-based automatic evaluation for machine translation
Full text linkWe present a novel method for evaluating the output of Machine Translation (MT), based on comparing the dependency
structures of the translation and reference rather than their surface string forms. Our method uses a treebank-based, wide coverage, probabilistic Lexical-Functional
Grammar (LFG) parser to produce a set of structural dependencies for each translation-reference sentence pair, and then calculates the precision and recall for these dependencies. Our dependency-based evaluation, in contrast to most popular string-based evaluation metrics, will not unfairly penalize perfectly valid syntactic variations in the translation. In addition to allowing for legitimate
syntactic differences, we use paraphrases in the evaluation process to account for lexical variation. In comparison with
other metrics on 16,800 sentences of Chinese-English newswire text, our method reaches high correlation with
human scores. An experiment with two translations of 4,000 sentences from Spanish-English Europarl shows that, in
contrast to most other metrics, our method does not display a high bias towards statistical models of translation
- …
