NII Repository (National Institute of Informatics)
Not a member yet
2035 research outputs found
Sort by
Best-Case Retrieval Evaluation: Improving the Sensitivity of Reciprocal Rank with Lexicographic Precision
Get PDFAcross a variety of ranking tasks, researchers use reciprocal rank to measure the effectiveness for users interested in exactly one relevant item. Despite its widespread use, evidence suggests that reciprocal rank is brittle when discriminating between systems. This brittleness, in turn, is compounded in modern evaluation settings where current, high-precision systems may be difficult to distinguish. We study the scenario where there is more than one relevant item and address the lack of sensitivity of reciprocal rank by introducing and connecting it to the concept of best-case retrieval, an evaluation method focusing on assessing the quality of a ranking for the most satisfied possible user across possible recall requirements. This perspective allows us to generalize reciprocal rank and define a new preference-based evaluation we call lexicographic precision or lexiprecision. By mathematical construction, we ensure that lexiprecision preserves differences detected by reciprocal rank, while empirically improving sensitivity and robustness across a broad set of retrieval and recommendation tasks.conference pape
Evaluating Systems that Generate Content
Get PDFThe astounding emergence of ChatGPT and other AI systems that generate content, and their apparently incredible performance, are an inspiration to the research community. The performance of these LLMs is so impressive it is widely supposed that we can use them to measure their own effectiveness! We have had evaluation methods for generated content, including question answering, summarization, and translation, and in this talk I dust them off and present both a historical view and how we might approach those methods today. tl;dr, we have a lot of work to do.conference pape
Using Language Models for Relevance Labelling
Get PDFRelevance labels – annotations that say whether a result is relevant to a given search – are key to evaluating the quality of a search engine. Standard practice to date has been to ask in-house or crowd workers to label results, but recently-developed language models are able to produce labels at greatly reduced cost. At Bing we have been using GPT-4, with human oversight, for relevance labelling at web scale. We find that models produce better labels than third-party or even in-house workers, for a fraction of the cost, and these labels let us train notably better rankers. In this talk I'll report on our experiences with GPT-4, including experiments with in-house data and with TREC-Robust. We see accuracy as good as human labellers, and similar capability to pick "interesting" cases, as well as variation due to details of prompt wording. High accuracy makes it hard to improve, and I'll also discuss our work on high-quality "gold" labels and on metrics for the labels themselves.conference pape
Responsible Information Access: Fairness, Harmlessness, Sustainability, and More
Get PDFconference pape
SPARC Japan Seminar 2022 "Current Status and Challenges of Open Access Affected by E-Journal Transformative Agreement and the APC Issues" The History of SPARC and the Transformation of Open Access Document
Get PDFSPARC Japan Seminar 2022 "Current Status and Challenges of Open Access Affected by E-Journal Transformative Agreement and the APC Issues"
Place：Online
Date&Time：February 17, 2023 / 13:00－17:00conference objec
NACSIS-CAT/ILL Newsletter No.52
Get PDF1．これからの学術情報システム構築検討委員会の活動について[p.1]
2．2022 年度 新NACSIS-CAT/ILLの変更点（2022 年12 月末時点）[p.2]
3．新NACSIS-CAT/ILL移行にかかるシステム停止（2023 年1 月）[p.2]
4．新NACSIS-CAT/ILL 公開後の書誌・所蔵データ一括登録の取扱い[p.4]
5．新NACSIS-CAT/ILL Q&A DB の公開について[p.5]
6．電子リソースデータ共有サービス「ライセンス（JUSTICE）」正式公開について[p.6]
7．CAT/ILLリプレイス＆電子リソースデータ共有サービス説明会 開催報告[p.9]
8．NACSIS-CAT/ILLウェブサイトリニューアルのお知らせ[p.10]
9．2021年度NACSIS-CAT/ILL 業務分析表の公開[p.11]
10．NIIでの目録品質管理（19）[p.12]
11．ILL文献複写等料金相殺サービス処理報告（2022年度第2四半期）[p.14]
12．目録システム書誌作成研修について[p.14]articl
SPARC Japan セミナー2022「電子ジャーナルの転換契約とAPC問題で変わるオープンアクセスの現状と課題」オープンアクセスの実現手段としての機関リポジトリ ドキュメント
Get PDFSPARC Japan セミナー2022「電子ジャーナルの転換契約とAPC問題で変わるオープンアクセスの現状と課題」
開催場所：オンライン開催
日時：2023年2月17日（金）13:00-17:00conference objec
SPARC Japan セミナー2022 「電子ジャーナルの転換契約とAPC問題で変わるオープンアクセスの現状と課題」 生命科学系研究におけるAPCの事例紹介 発表資料
Get PDFSPARC Japan セミナー2022「電子ジャーナルの転換契約とAPC問題で変わるオープンアクセスの現状と課題」
開催場所：オンライン開催
日時：2023年2月17日（金）13:00-17:00conference objec
FA Team at the NTCIR-17 UFO Task
Get PDFThe FA team participated in the Table Data Extraction (TDE) and Text-to-Table Relationship Extraction (TTRE) task of the NTCIR-17 Understanding of Non-Financial Objects in Financial Reports (UFO). This paper reports our approach to solving the problem and discusses the official results. We successfully utilized various enhancement techniques based on the ELECTRA language model to extract valuable data from tables. Our efforts resulted in an impressive TDE accuracy rate of 93.43\%, positioning us in second place on the Leaderboard rankings. This outstanding achievement is a testament to our proposed approach's effectiveness. In the TTRE task, we proposed the rule-based method to extract meaningful relationships between the text and tables task and confirmed the performance.conference pape
HPIDHC at NTCIR-17 MedNLP-SC: Data Augmentation and Ensemble Learning for Multilingual Adverse Drug Event Detection
Get PDFThe Social Media Adverse Drug Event Detection (SM-ADE) track of the NTCIR-17 MedNLP-SC shared task aims to identify adverse drug events (ADE) in Japanese, English, French, and German social media texts.In this paper, we describe selected details of our contribution addressing the shared task. As a base model, we fine-tune RoBERTa models for the different language subtasks. In addition, we apply ensemble learning and data augmentation techniques. By leveraging data augmentation, we successfully elevate the resulting micro-averaged F1 scores on the German dataset by 5pp compared to the baseline. The application of ensemble learning yields a remarkable improvement of 7pp. Through combining RoBERTa with these methods, we achieve promising results in the challenge. Our best runs accomplish exact accuracy scores between 0.84 and 0.87 and per-class F1 scores between 0.77 and 0.82, consistently achieving the second-best results across all languages.conference pape