LnuOpen (Linnaeus University)
Not a member yet
3576 research outputs found
Sort by
Bayesian Evaluation of Replication Studies
No full textIn this paper a method is proposed to determine whether the result from an original study is corroborated in a replication study. The paper is illustrated using two replication studies and the corresponding original studies from the Reproducibility Project: Psychology by the Open Science Collaboration. This method emphasizes the need to determine what one wants to replicate from the original paper. This can be done by translating the research hypotheses formulated in the introduction into informative hypotheses, or, by translating the results into interval hypotheses. The Bayes factor will be used to determine whether the hypotheses resulting from the original study are corroborated by the replication study. Our method to assess the successfulness of replication will better fit the needs and desires of researchers in fields that use replication studies
The Effect of Variety on Perceived Quantity: Failures to Replicate Redden and Hoch (2009)
No full textRedden and Hoch (2009) found that variety in a set of items robustly decreased the perceived quantity of the sum of these items across multiple studies. For example, a set of multicolored M&M’s was estimated to contain fewer M&M’s than an equally large set of single-colored M&M’s (e.g., Redden & Hoch, 2009, Study 3). We conducted six close replication studies of the studies reported by Redden and Hoch and did not find this effect in any of them. A meta-analysis of the four original studies and 6 replication studies (N = 1,383) revealed no evidence for the phenomenon that variety reduces perceived quantity
The Devil is Mainly in the Nuisance Parameters: Performance of Structural Fit Indices Under Misspecified Structural Models in SEM
No full textTo provide researchers with a means of assessing the fit of the structural component of structural equation models, structural fit indices- modifications of the composite fit indices, RMSEA, SRMR, and CFI- have recently been developed. We investigated the performance of four of these structural fit indices- RMSEA-P, RMSEAs, SRMRs, and CFIs-, when paired with widely accepted cutoff values, in the service of detecting structural misspecification. In particular, by way of simulation study, for each of seven fit indices- 3 composite and 4 structural-, and the traditional chi-square test of perfect composite fit, we estimated the following rates: a) Type I error rate (i.e., the probability of (incorrect) rejection of a correctly specified structural component), under each of four degrees of misspecification in the measurement component; and b) Power (i.e., the probability of (correct) rejection of an incorrectly specified structural model), under each condition formed of the pairing of one of three degrees of structural misspecification with one of four degrees of measurement component misspecification. In addition to sample size, the impacts of two model features, incidental to model misspecification- number of manifest variables per latent variable and magnitude of factor loading- were investigated. The results suggested that, although the structural fit indices performed relatively better than the composite fit indices, none of the goodness-of-fit index with a fixed cutoff value pairings was capable of delivering an entirely satisfactory Type I error rate/Power balance, [RMSEAs, .05] failing entirely in this regard. Of the remaining pairings; a) RMSEA-P and CFIs suffered from a severely inflated Type I error rate; b) despite the fact that they were designed to pick up on structural features of candidate models, all pairings- and especially, RMSEA-P and CFIs-manifested sensitivities to model features, incidental to structural misspecification; and c) although, in the main, behaving in a sensible fashion, SRMRs was only sensitive to structural misspecification when it occurred at a relatively high degree
Knowing What We're Talking About: Facilitating Decentralized, Unequivocal Publication of and Reference to Psychological Construct Definitions and Instructions
No full textA theory crisis and measurement crisis have been argued to be root causes of psychology's replication crisis. In both, the lack of conceptual clarification and the jingle-jangle jungle at the construct definition level as well the measurement level play a central role. We introduce a conceptual tool that can address these issues: Decentralized Construct Taxonomy specifications (DCTs). These consist of comprehensive specifications of construct definitions, corresponding instructions for quantitative and qualitative research, and unique identifiers. We discuss how researchers can develop DCT specifications as well as how DCT specifications can be used in research, practice, and theory development. Finally, we discuss the implications and potential for future developments to answer the call for conceptual clarification and epistemic iteration. This contributes to the move towards a psychological science that progresses in a cumulative fashion through discussion and comparison
Responsible Research Assessment Should Prioritize Theory Development and Testing Over Ticking Open Science Boxes
No full textWe appreciate the initiative to seek for ways to improve academic assessment by broadening the range of relevant research contributions and by considering a candidate’s scientific rigor. Evaluating a candidate's ability to contribute to science is a complex process that cannot be captured through one metric alone. While the proposed changes have some advantages, such as an increased focus on quality over quantity, the proposal's focus on adherence to open science practices is not sufficient, as it undervalues theory building and formal modelling: A narrow focus on open science conventions is neither a sufficient nor valid indicator for a “good scientist” and may even encourage researchers to choose easy, pre-registerable studies rather than engage in time-intensive theory building. Further, when in a first step only a minimum standard for following easily achievable open science goals is set, most applicants will soon pass this threshold. At this point, one may ask if the additional benefit of such a low bar outweighs the potential costs of such an endeavour. We conclude that a reformed assessment system should put at least equal emphasis on theory building and adherence to open science principles and should not completely disregard traditional performance metrices
Valuing Preprints Must be Part of Responsible Research Assessment
No full textComments on papers by Schönbrodt et al. (2022) and Gärtner et al. (2022) proposing reforms to the research assessment process. Given the prominent role of preprints in contemporary scientific practice, they must be an accepted and central component of research assessment
Response to responsible research assessment I and II from the perspective of the DGPs working group on open science in clinical psychology
No full textWe comment on the papers by Schönbrodt et al. (2022) and Gärtner et al. (2022) on responsible research assessment from the perspective of clinical psychology and psychotherapy research. Schönbrodt et al. (2022) propose four principles to guide hiring and promotion in psychology: (1) In addition to publications in scientific journals, data sets and the development of research software should be considered. (2) Quantitative metrics can be useful, but they should be valid and applied responsibly. (3) Methodological rigor, research impact, and work quantity should be considered as three separate dimensions for evaluating research contributions. (4) The quality of work should be prioritized over the number of citations or the quantity of research output. From the perspective of clinical psychology, we endorse the initiative to update current practice by establishing a matrix for comprehensive, transparent and fair evaluation criteria. In the following, we will both comment on and complement these criteria from a clinical-psychological perspective
Responsible research assessment in the area of quantitative methods research: A comment on Gärtner et al.
No full textIn this commentary, we discuss the proposed criteria in Gärtner et al. (2022) for hiring or promoting quantitative methods researchers. We argue that the criteria do not reflect aspects that are relevant to quantitative methods researchers and typical publications they produce. We introduce a new set of criteria that can be used to evaluate the performance of quantitative methods researchers in a more valid fashion. We discuss the necessity to balance scientific expertise and open science commitment in such ranking schemes
