1,721,023 research outputs found
Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answering
No full textLarge Language Models (LLMs) are capable of performing zero-shot closed-book question answering tasks, based on their internal knowledge stored in parameters during pre-training. However, such internalized knowledge might be insufficient and incorrect, which could lead LLMs to generate factually wrong answers. Furthermore, fine-tuning LLMs to update their knowledge is expensive. To this end, we propose to augment the knowledge directly in the input of LLMs. Specifically, we first retrieve the relevant facts to the input question from the knowledge graph based on semantic similarities between the question and its associated facts. After that, we prepend the retrieved facts to the input question in the form of the prompt, which is then forwarded to LLMs to generate the answer. Our framework, Knowledge-Augmented language model PromptING (KAPING), requires no model training, thus completely zero-shot. We validate the performance of our KAPING framework on the knowledge graph question answering task, that aims to answer the user's question based on facts over a knowledge graph, on which ours outperforms relevant zero-shot baselines by up to 48% in average, across multiple LLMs of various sizes
Direct Fact Retrieval from Knowledge Graphs without Entity Linking
No full textThere has been a surge of interest in utilizing Knowledge Graphs (KGs) for various natural language processing/understanding tasks. The conventional mechanism to retrieve facts in KGs usually involves three steps: entity span detection, entity disambiguation, and relation classification. However, this approach requires additional labels for training each of the three subcomponents in addition to pairs of input texts and facts, and also may accumulate errors propagated from failures in previous steps. To tackle these limitations, we propose a simple knowledge retrieval framework, which directly retrieves facts from the KGs given the input text based on their representational similarities, which we refer to as Direct Fact Retrieval (DiFaR). Specifically, we first embed all facts in KGs onto a dense embedding space by using a language model trained by only pairs of input texts and facts, and then provide the nearest facts in response to the input text. Since the fact, consisting of only two entities and one relation, has little context to encode, we propose to further refine ranks of top-k retrieved facts with a reranker that contextualizes the input text and the fact jointly. We validate our DiFaR framework on multiple fact retrieval tasks, showing that it significantly outperforms relevant baselines that use the three-step approach
Approximating neural machine translation for efficiency
Full text linkNeural machine translation (NMT) has been shown to outperform statistical machine
translation. However, NMT models typically require a large number of parameters
and are expensive to train and deploy. Moreover, its large model size makes parallel
training inefficient due to costly network communication. Likewise, distributing and
locally running the model for a client-based NMT model such as a web browser or
mobile device remains challenging. This thesis investigates ways to approximately
train an NMT system by compressing either the gradients or the parameters for faster
communication or reduced memory consumption. We propose a gradient compression
technique that exchanges only the top 1% of the most significant gradient values while
delaying the rest to be considered for the next iteration. This method reduces the
network communication cost by 50-fold but causes noisy gradient updates. We also
find that Transformer–the current state-of-the-art NMT architecture–is highly sensitive
to noisy gradients. Therefore, we extend the compression technique by restoring the
compressed gradient with locally-computed gradients. We obtained a linear scale-up
in parallel training without sacrificing model performance. We also explore transfer
learning as a better method of initialising the training. With transfer learning, the model
converges faster and can be trained with more aggressive hyperparameters. Lastly, we
propose a log-based quantisation method to compress the model size. Models are
quantised to 4-bit precision with no noticeable quality degradation after re-training
combined with reserving the quantisation errors as feedback
Daisy-TTS: Simulating Wider Spectrum of Emotions via Prosody Embedding Decomposition
Full text linkWe often verbally express emotions in a multifaceted manner, they may vary in their intensities and may be expressed not just as a single but as a mixture of emotions. This wide spectrum of emotions is well-studied in the structural model of emotions, which represents variety of emotions as derivative products of primary emotions with varying degrees of intensity. In this paper, we propose an emotional text-to-speech design to simulate a wider spectrum of emotions grounded on the structural model. Our proposed design, Daisy-TTS, incorporates a prosody encoder to learn emotionally-separable prosody embedding as a proxy for emotion. This emotion representation allows the model to simulate: (1) Primary emotions, as learned from the training samples, (2) Secondary emotions, as a mixture of primary emotions, (3) Intensity-level, by scaling the emotion embedding, and (4) Emotions polarity, by negating the emotion embedding. Through a series of perceptual evaluations, Daisy-TTS demonstrated overall higher emotional speech naturalness and emotion perceiveability compared to the baseline.Project Page: https://rendchevi.github.io/daisy-tts; Updates: (1) Fixed typos, missing references, and layout, (2) Revise explanation on emotion classifier or discriminato
Style Over Substance: Evaluation Biases for Large Language Models
Full text linkAs large language models (LLMs) continue to advance, accurately and
comprehensively evaluating their performance becomes increasingly challenging.
Ranking the relative performance of LLMs based on Elo ratings, according to
human judgment, is gaining more popularity. However, the extent to which humans
and LLMs are capable evaluators remains uncertain. This study investigates the
behavior of crowd-sourced and expert annotators, as well as LLMs, when
comparing outputs from different models. To achieve this, we curate a dataset
of intentionally flawed machine-generated answers. Our findings reveal a
concerning bias in the evaluation process, as answers with factual errors are
rated more favorably than answers that are too short or contained grammatical
errors. To address this issue, we propose independently evaluating
machine-generated text across multiple dimensions, rather than merging all the
evaluation aspects into a single score. We instantiate this idea with the Elo
rating system, resulting in the Multi-Elo Rating System (MERS). Empirical
results from our study reveal that this proposed approach significantly
enhances the quality of LLM-based evaluations, particularly in terms of factual
accuracy. However, there is no significant improvement in crowd-sourced-based
evaluations, indicating the need for further investigation.Comment: Work in progress, 15 pages, 5 tables, 12 figure
Mintaka: A Complex, Natural, and Multilingual Dataset for End-to-End Question Answering
Full text linkWe introduce Mintaka, a complex, natural, and multilingual dataset designed
for experimenting with end-to-end question-answering models. Mintaka is
composed of 20,000 question-answer pairs collected in English, annotated with
Wikidata entities, and translated into Arabic, French, German, Hindi, Italian,
Japanese, Portuguese, and Spanish for a total of 180,000 samples. Mintaka
includes 8 types of complex questions, including superlative, intersection, and
multi-hop questions, which were naturally elicited from crowd workers. We run
baselines over Mintaka, the best of which achieves 38% hits@1 in English and
31% hits@1 multilingually, showing that existing models have room for
improvement. We release Mintaka at https://github.com/amazon-research/mintaka.Comment: Accepted at COLING 202
Beyond Probabilities: Unveiling the Misalignment in Evaluating Large Language Models
Full text linkLarge Language Models (LLMs) have demonstrated remarkable capabilities across various applications, fundamentally reshaping the landscape of natural language processing (NLP) research. However, recent evaluation frameworks often rely on the output probabilities of LLMs for predictions, primarily due to computational constraints, diverging from real-world LLM usage scenarios. While widely employed, the efficacy of these probability-based evaluation strategies remains an open research question. This study aims to scrutinize the validity of such probability-based evaluation methods within the context of using LLMs for Multiple Choice Questions (MCQs), highlighting their inherent limitations. Our empirical investigation reveals that the prevalent probability-based evaluation method inadequately aligns with generation-based prediction. Furthermore, current evaluation frameworks typically assess LLMs through predictive tasks based on output probabilities rather than directly generating responses, owing to computational limitations. We illustrate that these probability-based approaches do not effectively correspond with generative predictions. The outcomes of our study can enhance the understanding of LLM evaluation methodologies and provide insights for future research in this domain.Accepted to KnowledgeableLMs @ ACL 202
LLM-powered Data Augmentation for Enhanced Cross-lingual Performance
Full text linkThis paper explores the potential of leveraging Large Language Models (LLMs)
for data augmentation in multilingual commonsense reasoning datasets where the
available training data is extremely limited. To achieve this, we utilise
several LLMs, namely Dolly-v2, StableVicuna, ChatGPT, and GPT-4, to augment
three datasets: XCOPA, XWinograd, and XStoryCloze. Subsequently, we evaluate
the effectiveness of fine-tuning smaller multilingual models, mBERT and XLMR,
using the synthesised data. We compare the performance of training with data
generated in English and target languages, as well as translated
English-generated data, revealing the overall advantages of incorporating data
generated by LLMs, e.g. a notable 13.4 accuracy score improvement for the best
case. Furthermore, we conduct a human evaluation by asking native speakers to
assess the naturalness and logical coherence of the generated examples across
different languages. The results of the evaluation indicate that LLMs such as
ChatGPT and GPT-4 excel at producing natural and coherent text in most
languages, however, they struggle to generate meaningful text in certain
languages like Tamil. We also observe that ChatGPT falls short in generating
plausible alternatives compared to the original dataset, whereas examples from
GPT-4 exhibit competitive logical consistency.Comment: EMNLP 2023 Main Conferenc
Going Beyond Counting First Authors in Author Co-citation Analysis
Full text linkThe present study examines one of the fundamental aspects of author co-citation analysis (ACA) - the way co-citation
counts are defined. Co-citation counting provides the data on which all subsequent statistical analyses and mappings
are based, and we compare ACA results based on two different types of co-citation counting - the traditional type that
only counts the first one among a cited work's authors on the one hand and a non-traditional type that takes into
account the first 5 authors of a cited work on the other hand. Results indicate that the picture produced through this non-traditional author co-citation counting contains more coherent author groups and is therefore considerably clearer. However, this picture represents fewer specialties in the research field being studied than that produced through the traditional first-author co-citation counting when the same number of top-ranked authors is selected and analyzed. Reasons for these effects are discussed
- …
