108495 research outputs found
Sort by
What's Wrong With This Translation? Simplifying Error Annotation For Crowd Evaluation
Get PDFMachine translation (MT) for Faroese faces challenges due to limited expert annotators and a lack of robust evaluation metrics. This study addresses these challenges by developing an MQM-inspired expert annotation framework to identify key error types and a simplified crowd evaluation scheme to enable broader participation. Our findings based on an analysis of 200 sentences translated by three models demonstrate that simplified crowd evaluations align with expert assessments, paving the way for improved accessibility and democratization of MT evaluation
The Application of Corpus-Based Language Distance Measurement to the Diatopic Variation Study (on the Material of the Old Novgorodian Birchbark Letters)
Get PDFThe paper presents a computer-assisted exploration of a set of texts, where qualitative analysis complements the linguistically-aware vector-based language distance measurements, interpreting them through close reading and thus proving or disproving their conclusions. It proposes using a method designed for small raw corpora to explore the individual, chronological, and gender-based differences within an extinct single territorial lect, known only by a scarce collection of documents. The material under consideration is the Novgorodian birchbark letters, a set of rather small manuscripts (not a single one is more than 1000 tokens) that are witnesses of the Old Novgorodian lect, spoken on the territories of modern Novgorod and Staraya Russa at the first half of the first millennium CE. The study shows the existence of chronological variation, a mild degree of individual variation, and almost absent gender-based differences. Possible prospects of the study include its application to the newly discovered birchbark letters and using an outgroup for more precise measurements.https://aclanthology.org/2025.resourceful-1.0
Using the political settlement framework to analyze external promoter's anti-corruption efforts: the case of RoLAC in Nigeria (2017-2023)
No full textCorruption undermines ethical universalism, and institutional capacity, eroding democracy and exacerbating governance challenges, especially in developing countries. Multilateral efforts, such as the United Nations Convention Against Corruption (UNCAC) have been employed to tackle this menace in international fora. Such efforts emphasize capacity building, particularly in countries like Nigeria, where governance issues persist despite significant resources. Aligned with these efforts of institution capacity building, the EU-funded RoLAC program between 2017 to 2023 sought to strengthen governance and combat corruption in Nigeria. However, the country’s Control of Corruption Index stagnated at -1.1 during this period, reflecting limited progress. To explain such trend, scholars have increasingly moved away from traditional approaches like donor coordination, which offer limited insights, toward examining the domestic realities of target countries, particularly within the European Neighborhood Policy (Börzel et al., 2010; Börzel and Pamuk 2011; Börzel and van Hüllen 2014; Kralikova, 2022). However, this perspective remains relatively underexplored in Sub-Saharan Africa, especially in Nigeria. This gap is significant, given that Nigeria is among the largest recipients of EU governance funding outside Europe (Hackenesch, 2016, p.26). Informed by political settlement scholarship which matches the target country’s political and institutional dynamics with good-fit instruments (Levy, 2014; Muhhina, 2023; Roy, 2017), this study examines the impact of policy design elements, precision of targeting, involvement of actors, and degree of coerciveness, on the RoLAC program. Using qualitative coding of documents and expert interviews, the findings reveal that reliance on comprehensive, government-driven, and highly coercive policy instruments was ineffective in Nigeria's fragmented political landscape and weak institutional context, hindering RoLAC's success. These findings emphasize the practical necessity of aligning policy designs and tools with the unique political and institutional dynamics of target countries. External promoters like the EU should adopt tailored policy instruments to achieve meaningful governance improvements, ensuring compatibility with the local context for sustainable reform outcomes.https://www.ester.ee/record=b5733928*es
Analyzing the Online Communication of Environmental Movement Organizations: NLP Approaches to Topics, Sentiment, and Emotions
Get PDFThis project employs state-of-the-art Natural Language Processing (NLP) techniques to analyze the online communication of international Environmental Movement Organizations (EMOs). First, we introduce our overall EMO dataset and describe it through topic modeling. Second, we evaluate current sentiment and emotion classification models for our specific dataset. Third, as we are currently in our annotation process, we evaluate our current progress and issues to determine the most effective approach for creating a high-quality annotated dataset that captures the nuances of EMO communication. Finally, we emphasize the need for domain-specific datasets and tailored NLP tools and suggest refinements for our annotation process moving forward
Mitmikagendilise arhitektuuri rakendamine nimeolemite märgendamiseks
No full textNamed Entity Recognition(NER) traditionally requires extensive domain-specific training data to achieve satisfactory performance for a given domain. Recent advancements in large language models have enabled the development of NER systems without supervised training, though this approach still requires careful prompt engineering and may need external knowledge augmentation during inference. This thesis introduces a novel domain-agnostic NER framework based on a collaborative multi-agent architecture that can adapt to any domain given only entity definitions and their descriptions. The framework consists of 4 high-level components: a team of agents, a metaprompter, a chat supervisor and a grounding engine. The system requires no training data or prompt engineering for new domains, operating as a few-shot solution for NER tasks. The framework's performance is evaluated across 4 distinct domains using standard NER benchmark datasets. Our evaluation shows that the multi-agent approach outperforms the baseline of few-shot NER with single LLM call in 3 out of 4 benchmarks, suggesting a promising direction for domain-agnostic NER. Ablation studies demonstrate varying effectiveness of each component on the system's performance depending on the domain, with the combination of three specialized agents and grounding engine proving generally most effective in all tested domains.Nimiolemite märgendamine(NER) nõuab mingis valdkonnas rahuldava tulemuse saamiseks tüüpiliselt massiivset valdkonnapõhist treeningandmestikku. Viimased edusammud suurte keelmudelite vallas on võimaldanud välja töötada NER-süsteeme juhendatud õppimiseta. See lähenemine nõuab aga hoolikat viipade loomist (prompt engineering) ja võib vajada lisaks domeenieksperdi sisendit. Käesoleva lõputöö raames valmis uus domeenist sõltumatu NER-raamistik, mis põhineb mitme agendi koostöö arhitektuuril ning suudab kohaneda erinevate valdkondadega, vajades vaid olemite definitsioone ja kirjeldusi. Raamistik koosneb neljast kõrgetasemelisest komponendist: agentide meeskonnast, metaviipajast, vestluse juhendajast ja tõendajast. Süsteem ei vaja uutes valdkondades olemeid tuvastades treeningandmeid ega eraldi viiba kohandamist, talitades nõnda väheste näidete lahendusena (few-shot solution) NER ülesannete raames. Raamistiku jõudlust hinnatakse neljas erinevas valdkonnas, kasutades standardseid NER-süsteemide hindamisandmeid. Saadud tulemused toetavad domeenist sõltumatu NER-süsteemi loomise võimalikkust, saavutades väheste näidetega üksikust viipamisest parema tulemuse kolmes valdkonnas neljast. Abaltsiooniuuringud näitavad, et iga komponendi efektiivsus varieerub valdkonnati, kusjuures kolme agendi ja tõendaja kombinatsioon osutus valdkondade üleselt üldiselt kõige tõhusamaks
Comparing Human and Machine Translations of Generative Language Model Evaluation Datasets
No full textThe evaluation of Large Language Models (LLMs) is one of the crucial current challenges in the field of Natural Language Processing (NLP) and becomes even more challenging in the multilingual setting. Since the majority of the community's benchmarks exist only in English, test sets are now being machine translated at scale into dozens of languages. This work explores the feasibility of that approach, comparing a Finnish machine translation (MT) of ARC-Challenge with a new human translated version. Our findings suggest that since absolute scores are fairly close and model size rankings are preserved, machine translation is adequate in this case. Surprisingly, however, the datasets reverse the order of base models compared to their chat-finetuned counterparts
Modeling Multilayered Complexity in Literary Texts
No full textWe explore the relationship between stylistic and sentimental complexity in literary texts, analyzing how they interact and affect overall complexity. Using a dataset of over 9,000 English novels (19th-20th century), we find that complexity at the stylistic/syntactic and sentiment levels tend to show a linear association. Finally, using dedicated datasets, we show that both stylistic/syntactic features – particularly those relating to information density – as well as sentiment features are related to text difficulty rank as well as average processing time
A Collection of Question Answering Datasets for Norwegian
No full textThis paper introduces a new suite of question answering datasets for Norwegian; NorOpenBookQA, NorCommonSenseQA, NorTruthfulQA, and NRK-Quiz-QA. The data covers a wide range of skills and knowledge domains, including world knowledge, commonsense reasoning, truthfulness, and knowledge about Norway. Covering both of the written standards of Norwegian – Bokmål and Nynorsk – our datasets comprise over 10k question-answer pairs, created by native speakers. We detail our dataset creation approach and present the results of evaluating 11 language models (LMs) in zero- and few-shot regimes. Most LMs perform better in Bokmål than Nynorsk, struggle most with commonsense reasoning, and are often untruthful in generating answers to questions. All our datasets and annotation materials are publicly available
Universal Dependencies Treebank for Uzbek
Get PDFWe present the first Universal Dependencies treebank for Uzbek, a low-resource language from the Turkic family. The treebank contains 500 sentences (5850 tokens) sourced from the news and fiction genres and it is annotated for lemmas, part-of-speech (POS) tags, morphological features, and dependency relations. We describe our methodology for building the treebank, which consists of a mix of manual and automatic annotation and discuss some constructions of the Uzbek language that pose challenges to the UD framework.https://aclanthology.org/2025.resourceful-1.0