1,720,986 research outputs found
OPEN SCIENCE POLICIES AND CORE OPEN SCIENCE PRACTICES IN MAJOR MEDICAL JOURNALS: DEVELOPMENT OF AN AUTOMATIZED TOOL BASED ON A LLM
No full textOpen science policies and core open science practices in major medical journals: development of an automatized tool based on a LLM
GUILLAUME FREYERMUTH / CONSTANT VINATIER / GAUTHIER LE-BARTZ-LYAN / SARAH BUET / PERRINE LUNEL / NICOLAS ROBILLARD / GWENAEL DUMONT / FLORIAN NAUDET / MATHIEU ACHER
BACKGROUND
Monitoring the use and application of Open Science practices is key to fostering Open Science and to understanding barriers and facilitators of open science practices. Monitoring also aids in measuring the impact of open science policies and can serve as a possible intervention to improve transparency and reproducibility. For instance, the Hong Kong principles support the assessment of researchers on the use of Open Science practices. Open Science Monitoring is a key priority for organizations such as the UNESCO [and the G7. The scientific community is also beginning to agree on key indicators to measure. For instance, in biomedicine, a core set of 19 open science practices has been developed to monitor various open science practices such as preregistration, open data, open access, open code, conflict of interest statements etc.
In recent years, biomedical journals have implemented some policies that promote open science practices. Most notably, the influential International Committee of Medical Journal Editors (ICMJE) made prospective registration a requirement to publish clinical trial results in 2005. In 2018, it required a specific data-sharing statement to be included in each newly submitted paper (and prespecified in study registration). The impact of these policies have been studied previously showing an major influence on trial registration practices, and mixed results regarding actual data sharing.
Automatized options are being developed to track change within the biomedical literature. It is therefore useful to assess the impact of the various open science policies adopted by biomedical journals using those tools.
OBJECTIVES
This aim of this study, targeting major medical journals, is three-fold:
1. to describe open science policies in these journals,
2. to develop and validate automatic tools that can be used to monitor key transparency practices
3. to describe retrospectively changes on open science and transparency practices for articles published in those journals.
METHODS
Selection criteria
We will include in our study a sample of 10 major medical journals.
Population: We will include research studies published in 2020 or 2021 in journals of interest:
- Annals of internal medicine
- BioMed Central Medicine
- The British Medical Journal
- Canadian Medical Association Journal
- Journal of the American Medical Association
- JAMA Network Open
- Lancet
- Nature Medicine
- Public Library Of Science
Study design: All types of research articles will be included with a proportional number of 1/RCTs, 2/ meta-analyses and 3/ other research articles.
Data sources:
All articles and supplementary materials will be extracted from Pubmed and from the journal website. If an article isn’t open access, academic access will be used to get it.
Automatized tool
Here we want to develop a tool using machine learning and to explore the feasibility of a tool based on a large language model (LLM) such as Mixtral or llama to extract the presence and absence of open science practices inside. Ideally this tool should be automatized, open source (or failing that, open weights). and used locally to enforce reproducibility. It should be able to identify journal articles (either in XML or pdf format) and all supplementary material (in pdf format) and to extract core Open Science practices. The task of extracting content out of a scholar article with an LLM involves optional innovations to ensure accuracy, robustness, and efficiency, such as embedding of raw data into vector stores, refining the method of prompting the LLM, and employing techniques like retrieval augmented generation (RAG) or fine-tuning.
Creation of the validation database:
Data collection occurred from February 25th to March 18th 2024, following a standardized approach. For each journal, a stratification about study type was realized to select 10 RCT, 10 meta research and 10 other research articles per journal. To perform article selection, 1/ the algorithm of research from PubMed was used as described in table 1. Then, in each journal/category, we randomly selected 10 articles. As it, the database is composed of 30 articles from each journal with an equal representation of the different type of study designs for a total of 300 articles. In case the required number of articles of a given type wasn’t available in a given journal, the total number of articles was completed by articles with the same study design published in other journals (that were each time from a different journal). This was done by a first group of master 1 students (SB, PL, NR).
Search strategy and article selection process
All articles were classified by the publication date, the latest will get the number 1 and the oldest the max. After assigning these numbers, for each journal and type of study, each was given a random number and eligibility of articles will be assessed – for each journal and type of study- using this random number and included if those are in line with our selection criteria (data and code available on the Open Science Framework). As soon as the target number of included articles was reached, the same selection process was applied for the next journal and type of study.
Table 1: research algorithm for each study type:
Study type Research algorithm
RCT (2020:2021[Date - Publication] AND "hasabstract" AND randomizedcontrolledtrial[Filter] AND ("Annals of Internal Medicine"[Journal] OR "BMC Medicine"[Journal] OR "BMJ"[Journal] OR "CMAJ"[Journal] OR "JAMA"[Journal] OR "JAMA Netw Open"[Journal] OR "Lancet"[Journal] OR "Nat Med"[Journal] OR "N Engl J Med"[Journal] OR "PLOS Med"[Journal]))
Meta research (2020:2021[Date - Publication] AND "hasabstract" AND meta-analysis[Filter] AND ("Annals of Internal Medicine"[Journal] OR "BMC Medicine"[Journal] OR "BMJ"[Journal] OR "CMAJ"[Journal] OR "JAMA"[Journal] OR "JAMA Netw Open"[Journal] OR "Lancet"[Journal] OR "Nat Med"[Journal] OR "N Engl J Med"[Journal] OR "PLOS Med"[Journal]))
Other research article ((2020:2021[Date - Publication]) AND "hasabstract" NOT ("comment"[Publication Type]) NOT ("letter"[Publication Type]) NOT ("editorial"[Publication Type]) NOT ("published erratum"[Publication Type]) NOT ("news"[Publication Type]) NOT ("introductory journal article"[Publication Type]) NOT ("biography"[Publication Type]) NOT ("portrait"[Publication Type]) NOT ("congress"[Publication Type]) NOT ("interview"[Publication Type]) NOT ("retraction of publication"[Publication Type]) NOT ("personal narrative"[Publication Type]) NOT ("retracted publication"[Publication Type]) NOT ("patient education handout"[Publication Type]) NOT ("lecture"[Publication Type]) NOT ("autobiography"[Publication Type]) NOT ("clinical conference"[Publication Type]) NOT ("classical article"[Publication Type]) NOT ("address"[Publication Type]) NOT ("legal case"[Publication Type]) NOT ("expression of concern"[Publication Type]) NOT ("festschrift"[Publication Type]) NOT ("overall"[Publication Type]) NOT ("bibliography"[Publication Type]) NOT ("corrected and republished article"[Publication Type]) NOT ("interactive tutorial"[Publication Type]) NOT ("duplicate publication"[Publication Type]) NOT ("directory"[Publication Type]) NOT ("newspaper article"[Publication Type]) NOT ("periodical index"[Publication Type]) NOT ("dictionary"[Publication Type]) NOT ("meta analysis"[Publication Type]) NOT ("randomized controlled trial"[Publication Type])) AND ("Annals of Internal Medicine"[Journal] OR "BMC Medicine"[Journal] OR "BMJ"[Journal] OR "CMAJ"[Journal] OR "JAMA"[Journal] OR "JAMA Netw Open"[Journal] OR "Lancet"[Journal] OR "Nat Med"[Journal] OR "N Engl J Med"[Journal] OR "PLOS Med"[Journal])
Each research algorithm was validated according to the PRESS guideline (1) by an information specialist (XCH).
Description of tools’ performance
Development will use the validation database of 300 papers selected from articles published in the journals of interest during the year 2020-2021 (30 per journal).
Statistical analysis:
Tool’s performance will be determined by the computation of metrics: sensitivity, specificity, PPV, NPC and the F1 score and the plotting of the receiver operating characteristic curve (ROC).
As highlighted in the literature (2), any error may also come from human. For each disagreement, a manual verification of the result by an expert (CV or FN) will be done to determine if the LLM is truly false or not.
Outcomes to be extracted both for training and validation
- Type of study (Randomized Control Trial, meta-analyses, network meta-analysis, Observational study)
- Pre-registration (Yes/No)
- Data sharing (fully open access, on request, not available)
- Open Access Publishing (Yes/No)
- Code Sharing (Yes/No)
- Publication of results within 1 year of study completion (RCT) (Yes/No)
- Materials sharing (STAP, Protocol) (Yes/No)
- Utilization of reporting guideline (Yes/No) + which reporting guideline is use
- Reporting the existence of a preprint (Yes/No)
- Authors contribution (Yes/No)
- COI statement (Yes/No)
- Reporting ORCID identifiers (Yes/No)
- Funding statement (Yes/No)
Implications
We aim to develop automated tools to monitor open science practices at major generalist biomedical journals. This work will inform the development of a broader open science observatory of biomedical journals. The leading biomedical journals in this study will hence provide important information about these journals that are considered as trusted sources by the medical community. Last, by tracking the open science practices, we will be able to describe the impact of various ICJME policy changes on selected open science practices (clinical trial data sharing and registration). This project is part of OSIRIS (Open Science to Improve Reproducibility in Science), a European project.
More information : [email protected]
Acknowledgment:
We would like to thanks Xavier Chard-Hutchinson (XCH) from the library of the medical university of Rennes for validating the research algorithm.
Research lab: Inserm IRSET, team 11, Rennes, France
Florian Naudet (Professor, Inserm IRSET) will be in charge of overseeing the biomedical and open science aspects.
Mathieu Acher (Professor, University of Rennes) will be in charge of overseeing the machine learning aspects.
Constant Vinatier (PhD student, Inserm IRSET) will be in charge to oversee the biomedical and open science aspects and to establish connections with the OSIRIS consortium.
References
1. McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C. PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement. J Clin Epidemiol. 2016 Jul 1;75:40–6.
2. Issaiy M, Ghanaati H, Kolahi S, Shakiba M, Jalali AH, Zarei D, et al. Methodological insights into ChatGPT’s screening performance in systematic reviews. BMC Med Res Methodol. 2024 Mar 27;24(1):78
Going Beyond Counting First Authors in Author Co-citation Analysis
Get PDFThe present study examines one of the fundamental aspects of author co-citation analysis (ACA) - the way co-citation
counts are defined. Co-citation counting provides the data on which all subsequent statistical analyses and mappings
are based, and we compare ACA results based on two different types of co-citation counting - the traditional type that
only counts the first one among a cited work's authors on the one hand and a non-traditional type that takes into
account the first 5 authors of a cited work on the other hand. Results indicate that the picture produced through this non-traditional author co-citation counting contains more coherent author groups and is therefore considerably clearer. However, this picture represents fewer specialties in the research field being studied than that produced through the traditional first-author co-citation counting when the same number of top-ranked authors is selected and analyzed. Reasons for these effects are discussed
Variations on the Author
Get PDF“Variations on the Author” discusses two of Eduardo Coutinho’s recent films (Um Dia na Vida, from 2010, and Últimas Conversas, posthumously released in 2015) and their contribution to the general question of documentary authorship. The director’s filmography is characterized by a consistent yet self-effacing form of authorial self-inscription: Coutinho often features as an interviewer that rather than express opinions propels discourses; an interviewer that is good at listening. This mode of self-inscription characterizes him as an author who is not expressive but who is nonetheless markedly present on the screen. In Um Dia na Vida, however, Coutinho is completely absent form the image, while Últimas Conversas, on the contrary, includes a confessional prologue that moves the director from the margins to the center of his films. This article examines the ways in which these works stand out in the filmography of a director who offers new insights into the notion of cinematic authorship
Appropriate Similarity Measures for Author Cocitation Analysis
Get PDFWe provide a number of new insights into the methodological discussion about author cocitation analysis. We first argue that the use of the Pearson correlation for measuring the similarity between authors’ cocitation profiles is not very satisfactory. We then discuss what kind of similarity measures may be used as an alternative to the Pearson correlation. We consider three similarity measures in particular. One is the well-known cosine. The other two similarity measures have not been used before in the bibliometric literature. Finally, we show by means of an example that our findings have a high practical relevance.information science;Pearson correlation;cosine;similarity measure;author cocitation analysis
Dispelling the Myths Behind First-author Citation Counts
Get PDFWe conducted a full-scale evaluative citation analysis study of scholars in the XML research field to explore just how different from each other author rankings resulting from different citation counting methods actually are, and to demonstrate the capability of emerging data and tools on the Web in supporting more realistic citation counting methods. Our results contest some common arguments for the continued
use of first-author citation counts in the evaluation of scholars, such as high correlations between author rankings by first-author citation counts and other citation
counting methods, and high costs of using more realistic citation counting methods that are not well-supported by the ISI databases. It is argued that increasingly available digital full text research papers make it possible for citation analysis studies to go beyond what the ISI databases have directly supported and to employ more
sophisticated methods
koamabayili/VECTRON-author-checklist: VECTRON author checklist
No full textWe have done our best to complete the author checklist relating to the use of animals in the hut study. Note that the objective for the hut study was to evaluate the IRS treatment applications for residual efficacy against Anopheles mosquitoes, including the local An. coluzzii mosquito population. Cows were only used to attract mosquitoes into the huts and no tests were carried out directly on the cows. The author checklist is intended for use with studies where experiments are carried out on animals, which is why we have had such difficulty in completing this for the hut study, as many of the questions do not relate to how the cows were used
Author-wise bibliometric analysis based on entropy.
No full textAuthor-wise bibliometric analysis based on entropy.</p
- …
