1,720,959 research outputs found
Meta-research in environmental science and economics
No full textSelective reporting, the practice of publishing favorable results while omitting unfavorable
ones, poses a significant challenge to the reliability of scientific research. Studies with
statistically significant or positive outcomes are more likely to be published than those with
null or negative results. Such bias distorts the overall body of scientific literature, potentially
leading to a false understanding of phenomena. This can have far‑reaching implications in the
fields of environmental science and economics since policies crafted without a comprehensive
understanding of the available evidence may fail to address societal issues effectively or may
even exacerbate them.
In Chapter 2 and 3, this dissertation introduces a new method used to detect and quantify the
extent of selective reporting. To illustrate the practicality and relevance of the new approach,
two large data sets collected from published and unpublished meta‑analyses in the fields of
economics and environmental science are presented. Moreover, we use a negative binomial
regression model to explore potential driving factors associated with selective reporting.
In economics, we based our analysis on 70,399 effect sizes collectedfrom 192 meta‑analyses. We
empirically estimate the extent of selective reporting by comparing the distribution of observed
p‑values with the counterfactual distribution of p‑values generated under the assumption of
no biases. Our study shows that about 58–72% fewer significant p‑values should have been
published in the absence of any biases. Subsample analysis suggests that the extent of selective
reporting is reduced in research fields that use experimental designs, analyze microeconomics
research questions, and have at least some adequately powered studies. A possible explanation
for this might be researchers are less likely to feel pressure to engage in questionable practices
like p‑hacking to achieve statistical significance when a study is adequately powered to detect
an effect.
To quantify the extent of selective reporting in environmental science, we analyzed over 60,000
statistical tests gathered from 705 meta‑analyses. By employing the new method, we find that
30–53% fewer significant p‑values should have been published in the absence of any biases in
the research and publication process. Moreover, our study indicates that the median statistical power in environmental science is only 8–13%, and only 9% of tests are adequately powered at
the conventional threshold of 80% or more. This raises concern over the reliability of published
findings in the field. Exploratory regressions suggest that the extent of selective reporting
decreases with increasing statistical power. The greater the power of the study, the lower the
likelihood of Type II errors (false‑negatives) and, therefore, the greater the chance of finding true
effects. As a result, researchers will be less inclined to p‑hack to obtain significant results.
We also employ Bayesian model‑averaging to identify the presence or absence of selective
reporting and obtain the bias‑corrected average to evaluate its impact in four scientific
disciplines, namely, medicine, psychology, economics, and environmental science. By
analyzing a large data set from more than 68,000 meta‑analyses, we find that economics
meta‑analyses are most affected by selective reporting, followed by environmental science and
psychology. In contrast, meta‑analyses in medicine are the least affected.
Finally, five tests used to detect selective reporting due to the omission of control variables are
evaluated (caliper test, discontinuity test, monotonicity test (CS1), joint test of monotonicity
and bounds (CS2B), and concavity test (LCM)). The tests are compared on the basis of their
statistical power and ability to control Type I errors using exact, rounded, and derounded data
sets simulated through the Monte Carlo procedure. We find that the discontinuity test applied
to the z‑value distribution seems to perform well, followed by the CS2B test, regardless of the
data type.
Overall, this dissertation highlights the importance of addressing selective reporting in evidence
synthesis, mainly by considering the statistical power of studies. Underpowered studies are less
reliable and could jeopardize scientific progress and knowledge accumulation, which further
undermines evidence‑based decision and policy making. In addition, this work may encourage
further meta‑research studies and suggests fostering the culture of open science practices could
help mitigate selective reporting in the synthesis of research across scientific disciplines
Meta-research in environmental science and economics
No full textSelective reporting, the practice of publishing favorable results while omitting unfavorable
ones, poses a significant challenge to the reliability of scientific research. Studies with
statistically significant or positive outcomes are more likely to be published than those with
null or negative results. Such bias distorts the overall body of scientific literature, potentially
leading to a false understanding of phenomena. This can have far‑reaching implications in the
fields of environmental science and economics since policies crafted without a comprehensive
understanding of the available evidence may fail to address societal issues effectively or may
even exacerbate them.
In Chapter 2 and 3, this dissertation introduces a new method used to detect and quantify the
extent of selective reporting. To illustrate the practicality and relevance of the new approach,
two large data sets collected from published and unpublished meta‑analyses in the fields of
economics and environmental science are presented. Moreover, we use a negative binomial
regression model to explore potential driving factors associated with selective reporting.
In economics, we based our analysis on 70,399 effect sizes collectedfrom 192 meta‑analyses. We
empirically estimate the extent of selective reporting by comparing the distribution of observed
p‑values with the counterfactual distribution of p‑values generated under the assumption of
no biases. Our study shows that about 58–72% fewer significant p‑values should have been
published in the absence of any biases. Subsample analysis suggests that the extent of selective
reporting is reduced in research fields that use experimental designs, analyze microeconomics
research questions, and have at least some adequately powered studies. A possible explanation
for this might be researchers are less likely to feel pressure to engage in questionable practices
like p‑hacking to achieve statistical significance when a study is adequately powered to detect
an effect.
To quantify the extent of selective reporting in environmental science, we analyzed over 60,000
statistical tests gathered from 705 meta‑analyses. By employing the new method, we find that
30–53% fewer significant p‑values should have been published in the absence of any biases in
the research and publication process. Moreover, our study indicates that the median statistical power in environmental science is only 8–13%, and only 9% of tests are adequately powered at
the conventional threshold of 80% or more. This raises concern over the reliability of published
findings in the field. Exploratory regressions suggest that the extent of selective reporting
decreases with increasing statistical power. The greater the power of the study, the lower the
likelihood of Type II errors (false‑negatives) and, therefore, the greater the chance of finding true
effects. As a result, researchers will be less inclined to p‑hack to obtain significant results.
We also employ Bayesian model‑averaging to identify the presence or absence of selective
reporting and obtain the bias‑corrected average to evaluate its impact in four scientific
disciplines, namely, medicine, psychology, economics, and environmental science. By
analyzing a large data set from more than 68,000 meta‑analyses, we find that economics
meta‑analyses are most affected by selective reporting, followed by environmental science and
psychology. In contrast, meta‑analyses in medicine are the least affected.
Finally, five tests used to detect selective reporting due to the omission of control variables are
evaluated (caliper test, discontinuity test, monotonicity test (CS1), joint test of monotonicity
and bounds (CS2B), and concavity test (LCM)). The tests are compared on the basis of their
statistical power and ability to control Type I errors using exact, rounded, and derounded data
sets simulated through the Monte Carlo procedure. We find that the discontinuity test applied
to the z‑value distribution seems to perform well, followed by the CS2B test, regardless of the
data type.
Overall, this dissertation highlights the importance of addressing selective reporting in evidence
synthesis, mainly by considering the statistical power of studies. Underpowered studies are less
reliable and could jeopardize scientific progress and knowledge accumulation, which further
undermines evidence‑based decision and policy making. In addition, this work may encourage
further meta‑research studies and suggests fostering the culture of open science practices could
help mitigate selective reporting in the synthesis of research across scientific disciplines
Estimating the extent of selective reporting: An application to economics
No full textAbstract
Using a sample of 70,399 published
p
‐values from 192 meta‐analyses, we empirically estimate the counterfactual distribution of
p
‐values in the absence of any biases. Comparing observed
p
‐values with counterfactually expected
p
‐values allows us to estimate how many
p
‐values are published as being statistically significant when they should have been published as non‐significant. We estimate the extent of selectively reported
p
‐values to range between 57.7% and 71.9% of the significant
p
‐values. The counterfactual
p
‐value distribution also allows us to assess shifts of
p
‐values along the entire distribution of published
p
‐values, revealing that particularly very small
p
‐values (
p
< 0.001) are unexpectedly abundant in the published literature. Subsample analysis suggests that the extent of selective reporting is reduced in research fields that use experimental designs, analyze microeconomics research questions, and have at least some adequately powered studies.Deutsche Forschungsgemeinschaft https://doi.org/10.13039/501100001659Bijzonder Onderzoeksfonds UGent https://doi.org/10.13039/50110000722
Going Beyond Counting First Authors in Author Co-citation Analysis
Full text linkThe present study examines one of the fundamental aspects of author co-citation analysis (ACA) - the way co-citation
counts are defined. Co-citation counting provides the data on which all subsequent statistical analyses and mappings
are based, and we compare ACA results based on two different types of co-citation counting - the traditional type that
only counts the first one among a cited work's authors on the one hand and a non-traditional type that takes into
account the first 5 authors of a cited work on the other hand. Results indicate that the picture produced through this non-traditional author co-citation counting contains more coherent author groups and is therefore considerably clearer. However, this picture represents fewer specialties in the research field being studied than that produced through the traditional first-author co-citation counting when the same number of top-ranked authors is selected and analyzed. Reasons for these effects are discussed
Variations on the Author
Full text link“Variations on the Author” discusses two of Eduardo Coutinho’s recent films (Um Dia na Vida, from 2010, and Últimas Conversas, posthumously released in 2015) and their contribution to the general question of documentary authorship. The director’s filmography is characterized by a consistent yet self-effacing form of authorial self-inscription: Coutinho often features as an interviewer that rather than express opinions propels discourses; an interviewer that is good at listening. This mode of self-inscription characterizes him as an author who is not expressive but who is nonetheless markedly present on the screen. In Um Dia na Vida, however, Coutinho is completely absent form the image, while Últimas Conversas, on the contrary, includes a confessional prologue that moves the director from the margins to the center of his films. This article examines the ways in which these works stand out in the filmography of a director who offers new insights into the notion of cinematic authorship
Appropriate Similarity Measures for Author Cocitation Analysis
Full text linkWe provide a number of new insights into the methodological discussion about author cocitation analysis. We first argue that the use of the Pearson correlation for measuring the similarity between authors’ cocitation profiles is not very satisfactory. We then discuss what kind of similarity measures may be used as an alternative to the Pearson correlation. We consider three similarity measures in particular. One is the well-known cosine. The other two similarity measures have not been used before in the bibliometric literature. Finally, we show by means of an example that our findings have a high practical relevance.information science;Pearson correlation;cosine;similarity measure;author cocitation analysis
- …
