539 research outputs found
Sort by
Robust Data-to-text Generation with Pretrained Language Models
No full textThe task of data-to-text generation amounts to describing structured data, such as RDF triples, in fluent natural language sentences. The state-of-the-art approach in research systems today is finetuning pretrained language models (PLMs). This often leads to overfitting the data and may produce hallucinations, i.e. situations where the PLM generates outputs that are not grounded in the input, typically replicating (or amplifying) training data noise. Rather than applying a PLM as black box for the whole data-to-text task, we aim at using
PLMs for simple individual subtasks, aiming to achieve broad generalization and minimize hallucination.
First, we use a pipeline approach where the PLMs only work as text "editors", rather than generators, taking advantage of their high output fluency. The data is converted into text in an initial preprocessing step, where we use simple handcrafted templates (one per RDF relation). This results in very short sentences recounting the individual input facts, corresponding to RDF triples. The PLMs then order the individual facts and fuse them into fluent sentences. This helps us generate without in-domain training data (except the simple
templates) and achieve good fluency and accuracy.
We further examine the capability of PLMs to produce accurate descriptions of individual RDF triples, in order to remove the last handcrafted step. Using a specially collected dataset of varied RDF triple descriptions, we show that PLMs finetuned with a variety of relations are very robust in verbalizing novel, unseen relations. The key to PLMs' usability here is providing clear and meaningful relation
labels
The Problem of Coherence in Natural Language Explanations of Recommendations
No full textProviding natural language explanations for recommendations is particularly useful from the perspective of a non-expert user. Although several methods for providing such explanations have recently been proposed, we argue that an important aspect of explanation quality has been overlooked in their experimental evaluation. Specifically, the coherence between generated text and predicted rating, which is a necessary condition for an explanation to be useful, is not properly captured by currently used evaluation measures. In this paper, we highlight the issue of explanation and prediction coherence by 1) presenting results from a manual verification of explanations generated by one of the state-of-the-art approaches 2) proposing a method of automatic coherence evaluation 3) introducing a new transformer-based method that aims to produce more coherent explanations than the state-of-the-art approaches 4) performing an experimental evaluation which demonstrates that this method significantly improves the explanation coherence without affecting the other aspects of recommendation performance
Three Ways of Using Large Language Models to Evaluate Chat
No full textThis paper describes the systems submitted by team6 for ChatEval, the DSTC 11 Track 4 competition. We present three different approaches to predicting turn-level qualities of chatbot responses based on large language models (LLMs). We report improvement over the baseline using dynamic few-shot examples from a vector store for the prompts for ChatGPT. We also analyze the performance of the other two approaches and report needed improvements for future work. We developed the three systems over just two weeks, showing the potential of LLMs for this task. An ablation study conducted after the challenge deadline shows that the new Llama 2 models are closing the performance gap between ChatGPT and open-source LLMs. However, we find that the Llama 2 models do not benefit from few-shot examples in the same way as ChatGPT
Negative Lexical Constraints in Neural Machine Translation
No full textThis paper explores negative lexical constraining in English to Czech neural machine translation. Negative lexical constraining is used to prohibit certain words or expressions in the translation produced by the neural translation model. We compared various methods based on modifying either the decoding process or the training data. The comparison was performed on two tasks: paraphrasing and feedback-based translation refinement. We also studied to which extent these methods “evade" the constraints presented to the model (usually in the dictionary form) by generating a different surface form of a given constraint.We propose a way to mitigate the issue through training with stemmed negative constraints to counter the model’s ability to induce a variety of the surface forms of a word that can result in bypassing the constraint. We demonstrate that our method improves the constraining, although the problem still persists in many cases
Data-to-text Generation with Neural Language Models
No full textWe introduce the problems of data-to-text generation and the current state of the art, i.e. pretrained language models. We further detail our experiments with using pretrained language models in a pipeline setup, thus increasing explainability and accuracy of the outputs. We also discuss our experiment in collecting a special dataset for verbalizing individual facts and evaluating pretrained language models, including ChatGPT, on this data
Critic-Driven Decoding for Mitigating Hallucinations in Data-to-text Generation
No full textHallucination of text ungrounded in the input is a well-known problem in neural data-to-text generation. Many methods have been proposed to mitigate it, but they typically require altering model architecture or collecting additional data, and thus cannot be easily applied to an existing model. In this paper, we explore a new way to mitigate hallucinations by combining the probabilistic output of a generator language model (LM) with the output of a special “text critic” classifier, which guides the generation by assessing the match between the input data and the text generated so far. Our method does not need any changes to the underlying LM’s architecture or training procedure and can thus be combined with any model and decoding operating on word probabilities. The critic does not need any additional training data, using the base LM’s training data and synthetic negative examples. Our experimental results show that our method improves over the baseline on the WebNLG and OpenDialKG benchmark
Skipping Chit-chat with ChatGPT: Large Language Models and Structured Outputs
No full textAn introduction into LLM workings and problems as well as an overview of recent experiments with using LLMs to model and evaluate dialogue
Text Detoxification as Style Transfer in English and Hindi
No full textThis paper focuses on text detoxification, i.e., automatically converting toxic text into nontoxic text. This task contributes to safer and more respectful online communication and can be considered a Text Style Transfer (TST) task, where the text’s style changes while its content is preserved. We present three approaches: (i) knowledge transfer from a similar task (ii) multi-task learning approach, combining sequence-to-sequence modeling with various toxicity classification tasks, and (iii) delete and reconstruct approach. To support our research, we utilize a dataset provided by Dementieva et al. (2021), which contains multiple versions of detoxified texts corresponding to toxic texts. In our experiments, we selected the best variants through expert human annotators, creating a dataset where each toxic sentence is paired with a single, appropriate detoxified version. Additionally, we introduced a small Hindi parallel dataset, aligning with a part of the English dataset, suitable for evaluation purposes. Our results demonstrate that our approach effectively balances text detoxification while preserving the actual content and maintaining fluency
Dialogue Systems (introduction)
No full textThis lecture presents an introduction into the state of the art in current dialogue systems, mostly focusing on neural architectures, end-to-end models, and pretrained language models
CLARIN in Training and Education
No full textTo help realise its potential as the research infrastructure for language as social and cultural data,
CLARIN is supporting the training of students and scholars in using its language data, tools and
services. Lecturers and teachers in the CLARIN network have integrated CLARIN language
resources into higher education programmes and other training activities. This paper showcases
some recent courses and training initiatives, along with inventories and new learning materials,
partly developed in EU-funded projects, which are accessible through the CLARIN Learning
Hub. Each section briefly describes the motivation behind the initiative, the authors’ experience,
related efforts in the field, and future perspectives