10 research outputs found
HANSEN: Human and AI Spoken Text Benchmark for Authorship Analysis
No full textAuthorship Analysis, also known as stylometry, has been an essential aspect of Natural Language Processing (NLP) for a long time. Likewise, the recent advancement of Large Language Models (LLMs) has made authorship analysis increasingly crucial for distinguishing between human-written and AI-generated texts. However, these authorship analysis tasks have primarily been focused on written texts, not considering spoken texts. Thus, we introduce the largest benchmark for spoken texts - HANSEN (Human ANd ai Spoken tExt beNchmark).
HANSEN encompasses meticulous curation of existing speech datasets accompanied by transcripts, alongside the creation of novel AI-generated spoken text datasets. Together, it comprises 17 human datasets, and AI-generated spoken texts created using 3 prominent LLMs: ChatGPT, PaLM2, and Vicuna13B. To evaluate and demonstrate the utility of HANSEN, we perform Authorship Attribution (AA) & Author Verification (AV) on human-spoken datasets and conducted Human vs. AI spoken text detection using state-of-the-art (SOTA) models. While SOTA methods, such as, character ngram or Transformer-based model, exhibit similar AA & AV performance in human-spoken datasets compared to written ones, there is much room for improvement in AI-generated spoken text detection. The HANSEN benchmark is available at: https://huggingface.co/datasets/HANSEN-REPO/HANSEN
The Word2vec Graph Model for Author Attribution and Genre Detection in Literary Analysis
No full textAnalyzing the writing styles of authors and articles is a key to supporting
various literary analyses such as author attribution and genre detection. Over
the years, rich sets of features that include stylometry, bag-of-words, n-grams
have been widely used to perform such analysis. However, the effectiveness of
these features largely depends on the linguistic aspects of a particular
language and datasets specific characteristics. Consequently, techniques based
on these feature sets cannot give desired results across domains. In this
paper, we propose a novel Word2vec graph based modeling of a document that can
rightly capture both context and style of the document. By using these Word2vec
graph based features, we perform classification to perform author attribution
and genre detection tasks. Our detailed experimental study with a comprehensive
set of literary writings shows the effectiveness of this method over
traditional feature based approaches. Our code and data are publicly available
at https://cutt.ly/svLjSgkComment: 12 pages, 6 figure
Understanding Social Structures from Contemporary Literary Fiction using Character Interaction Graph -- Half Century Chronology of Influential Bengali Writers
No full textSocial structures and real-world incidents often influence contemporary
literary fiction. Existing research in literary fiction analysis explains these
real-world phenomena through the manual critical analysis of stories.
Conventional Natural Language Processing (NLP) methodologies, including
sentiment analysis, narrative summarization, and topic modeling, have
demonstrated substantial efficacy in analyzing and identifying similarities
within fictional works. However, the intricate dynamics of character
interactions within fiction necessitate a more nuanced approach that
incorporates visualization techniques. Character interaction graphs (or
networks) emerge as a highly suitable means for visualization and information
retrieval from the realm of fiction. Therefore, we leverage character
interaction graphs with NLP-derived features to explore a diverse spectrum of
societal inquiries about contemporary culture's impact on the landscape of
literary fiction. Our study involves constructing character interaction graphs
from fiction, extracting relevant graph features, and exploiting these features
to resolve various real-life queries. Experimental evaluation of influential
Bengali fiction over half a century demonstrates that character interaction
graphs can be highly effective in specific assessments and information
retrieval from literary fiction. Our data and codebase are available at
https://cutt.ly/fbMgGEMComment: 8 pages, 11 figures, 6 pages appendi
Evaluation of classification and forecasting methods on time series gene expression data.
No full textTime series gene expression data is widely used to study different dynamic biological processes. Although gene expression datasets share many of the characteristics of time series data from other domains, most of the analyses in this field do not fully leverage the time-ordered nature of the data and focus on clustering the genes based on their expression values. Other domains, such as financial stock and weather prediction, utilize time series data for forecasting purposes. Moreover, many studies have been conducted to classify generic time series data based on trend, seasonality, and other patterns. Therefore, an assessment of these approaches on gene expression data would be of great interest to evaluate their adequacy in this domain. Here, we perform a comprehensive evaluation of different traditional unsupervised and supervised machine learning approaches as well as deep learning based techniques for time series gene expression classification and forecasting on five real datasets. In addition, we propose deep learning based methods for both classification and forecasting, and compare their performances with the state-of-the-art methods. We find that deep learning based methods generally outperform traditional approaches for time series classification. Experiments also suggest that supervised classification on gene expression is more effective than clustering when labels are available. In time series gene expression forecasting, we observe that an autoregressive statistical approach has the best performance for short term forecasting, whereas deep learning based methods are better suited for long term forecasting
HANSEN: Human and AI Spoken Text Benchmark for Authorship Analysis
No full textAuthorship Analysis, also known as stylometry, has been an essential aspect
of Natural Language Processing (NLP) for a long time. Likewise, the recent
advancement of Large Language Models (LLMs) has made authorship analysis
increasingly crucial for distinguishing between human-written and AI-generated
texts. However, these authorship analysis tasks have primarily been focused on
written texts, not considering spoken texts. Thus, we introduce the largest
benchmark for spoken texts - HANSEN (Human ANd ai Spoken tExt beNchmark).
HANSEN encompasses meticulous curation of existing speech datasets accompanied
by transcripts, alongside the creation of novel AI-generated spoken text
datasets. Together, it comprises 17 human datasets, and AI-generated spoken
texts created using 3 prominent LLMs: ChatGPT, PaLM2, and Vicuna13B. To
evaluate and demonstrate the utility of HANSEN, we perform Authorship
Attribution (AA) & Author Verification (AV) on human-spoken datasets and
conducted Human vs. AI spoken text detection using state-of-the-art (SOTA)
models. While SOTA methods, such as, character ngram or Transformer-based
model, exhibit similar AA & AV performance in human-spoken datasets compared to
written ones, there is much room for improvement in AI-generated spoken text
detection. The HANSEN benchmark is available at:
https://huggingface.co/datasets/HANSEN-REPO/HANSEN.Comment: 9 pages, EMNLP-23 findings, 5 pages appendix, 6 figures, 17 table
RAG based Question-Answering for Contextual Response Prediction System
No full textLarge Language Models (LLMs) have shown versatility in various Natural Language Processing (NLP) tasks, including their potential as effective question-answering systems. However, to provide precise and relevant information in response to specific customer queries in industry settings, LLMs require access to a comprehensive knowledge base to avoid hallucinations. Retrieval Augmented Generation (RAG) emerges as a promising technique to address this challenge. Yet, developing an accurate question-answering framework for real-world applications using RAG entails several challenges: 1) data availability issues, 2) evaluating the quality of generated content, and 3) the costly nature of human evaluation. In this paper, we introduce an end-to-end framework that employs LLMs with RAG capabilities for industry use cases. Given a customer query, the proposed system retrieves relevant knowledge documents and leverages them, along with previous chat history, to generate response suggestions for customer service agents in the contact centers of a major retail company. Through comprehensive automated and human evaluations, we show that this solution outperforms the current BERT-based algorithms in accuracy and relevance. Our findings suggest that RAG-based LLMs can be an excellent support to human customer service representatives by lightening their workload.Accepted at the 1st Workshop on GenAI and RAG Systems for Enterprise, CIKM\u2724. 6 page
AdolescentBot: Understanding Opportunities for Chatbots in Combating Adolescent Sexual and Reproductive Health Problems in Bangladesh
No full textA Ship of Theseus: Curious Cases of Paraphrasing in LLM-Generated Texts
No full textIn the realm of text manipulation and linguistic transformation, the question
of authorship has been a subject of fascination and philosophical inquiry. Much
like the Ship of Theseus paradox, which ponders whether a ship remains the same
when each of its original planks is replaced, our research delves into an
intriguing question: Does a text retain its original authorship when it
undergoes numerous paraphrasing iterations? Specifically, since Large Language
Models (LLMs) have demonstrated remarkable proficiency in both the generation
of original content and the modification of human-authored texts, a pivotal
question emerges concerning the determination of authorship in instances where
LLMs or similar paraphrasing tools are employed to rephrase the text--i.e.,
whether authorship should be attributed to the original human author or the
AI-powered tool. Therefore, we embark on a philosophical voyage through the
seas of language and authorship to unravel this intricate puzzle. Using a
computational approach, we discover that the diminishing performance in text
classification models, with each successive paraphrasing iteration, is closely
associated with the extent of deviation from the original author's style, thus
provoking a reconsideration of the current notion of authorship.Comment: To appear in Association for Computational Linguistics (ACL 2024
