25 research outputs found
pedermisager.org
No full textFiles related to personal website of Peder Mortvedt Isager (pedermisager.org
Quantifying Replication Value: A formula-based approach to study selection in replication research.
No full textBackground: The concept of replication is a central value of empirical science. At the same time scientists do not regard every replication as equally valuable. Even though replications are a cornerstone of empirical science (Bertamini & Munafò, 2012; Falk, 1998; Jasny, Chin, Chong, & Vignieri, 2011; Koole & Lakens, 2012; Moonesinghe, Khoury, & Janssens, 2007; Rosenthal, 1990; Schmidt, 2009), most researchers will agree that conducting 20 direct replications of the classic and extremely robust Stroop color-naming task (Stroop, 1935) would not be the best way to spend one’s grant money. This raises an important question: when is a replication of an empirical finding of sufficient valuable to the scientific community that it should be be performed? Given limited resources, one could also ask: which among currently published findings are the most valuable to replicate? Some discussion of the circumstances under which replication efforts are more or less beneficial has already occurred in the wake of increased replication efforts in psychology (Brandt et al., 2014; Coles, Tiokhin, Scheel, Isager, & Lakens, 2018), and recently suggestions have been put forward for how to select target studies for replication (Field, Hoekstra, Bringmann, & van Ravenzwaaij, 2018; Kuehberger & Schulte-Mecklenbeck, 2018). A comprehensive evaluation of the factors that could be used to quantify the replication value of a study is currently lacking, which is becoming increasingly important now that more replication studies are funded, performed, and published. Objectives: We propose a quantitative approach to help researchers, editors and funders evaluate and compare the replication value of original findings. Our approach rests on two fundamental assumptions: (1) That close replication (LeBel, Berger, Campbell, & Loving, 2017; LeBel, McCarthy, Earp, Elson, & Vanpaemel, 2018) is in principle a worthwhile endeavor, and (2) that there are more original observations worth replicating than we currently have the resources to replicate. In order to help researchers determine which among many findings might be the most promising candidates for replication, we outline a formula-based approach that can be relatively easily used to quantify the replication value of original findings. We propose that two components determine the replication value of empirical findings: (i) the impact of the effect, and (ii) the corroboration of the effect. Impact indicates the influence that the effect has had on scientific theory, research activities, or in society. All else being equal, findings that have had more impact are more important to replicate than findings that have had less impact. Corroboration indicates the empirical observations bearing on the finding, as well as the quality of these observations. As the corroboration of a particular finding increase, it becomes less important to replicate this finding, relative to a finding with little corroboration. The purpose of a replication value formula is to clarify how one intends to weigh the factors one considers important against one another, and to standardize parts of the study selection procedure. Because these formulas can be calculated quickly (and sometimes even automatically), they can be powerful tools for exploring a large set of studies to discover original findings that are particularly replication-worthy, assuming that the input to the formula is meaningful. Their ultimate goal is to make sure that resources spent on replication are efficiently utilized and that all relevant options for study choice can be considered when a replication effort is initialized. Research Questions: 1) What factors are considered important for determining the replication value of a particular finding or result? 2) Can we create a formula that is able to yield meaningful quantitative estimates of the relative replication value of empirical findings, based on metrics related to the impact and the corroboration of the finding? Approach & Preliminary results: To assess whether our conceptualization of replication value is in line with evaluations of replication value in the broader community of researchers, and to better understand how replicating researchers justify decisions of study choice, we conducted a literature review of justifications of study selection in 85 replication reports. The literature review suggests that researchers use many different information sources to assess replication value that could be subsumed under the categories of impact and corroboration (e.g. citation impact, theoretical importance, imprecise estimates, lack of prior replication). However, it is also clear that some types of information cannot easily be quantified (e.g. theoretical importance), and it is clear that factors other than the value of replication matter for the evaluation process as well (e.g. feasibility). We are currently in the process of constructing one version of a replication value formula that captures the impact and corroboration of a finding. Once a formula has been derived, we aim to evaluate whether the candidate studies returned by the formula track researchers’ qualitative judgements of relative replication value. We will pursue this through two lines of inquiry. First, we will calculate replication value for a large number of studies in the psychological literature and evaluate the face-validity of the recommendations produced, as well as inspect formula recommendations for examples where the true replication is known to be very high or very low (e.g. Stroop, 1935). Second, we will design a study to assess whether formula-based replication value estimate is able to predict researchers’ intuitive evaluation of replication value. Preliminary assessment of face-validity for data in the Curate Science database suggests that the formula yields sensible estimates of replication value for cases where true replication value is known. At the time of the conference, we expect to have completed a comprehensive evaluation of formula performance for both the Curate Science database and a large sample of published studies from the psychological literature. In addition, we will be able to present the planned experimental design for the study that will compare formula recommendations to researchers’ intuitive judgements. References: Bertamini, M., & Munafò, M. R. (2012). Bite-Size Science and Its Undesired Side Effects. Perspectives on Psychological Science, 7(1), 67–71. https://doi.org/10.1177/1745691611429353
Brandt, M. J., IJzerman, H., Dijksterhuis, A., Farach, F. J., Geller, J., Giner-Sorolla, R., … Van’t Veer, A. (2014). The replication recipe: What makes for a convincing replication? Journal of Experimental Social Psychology, 50, 217–224.
Coles, N., Tiokhin, L., Scheel, A., Isager, P., & Lakens, D. (2018). The Costs and Benefits of Replication Studies. https://doi.org/10.17605/osf.io/c8akj
Falk, R. (1998). Replication-A Step in the Right Direction: Commentary on Sohn. Theory & Psychology, 8(3), 313–321. https://doi.org/10.1177/0959354398083002
Field, S., Hoekstra, R., Bringmann, L., & van Ravenzwaaij, D. (2018). When and Why to Replicate: As Easy as 1, 2, 3? Open Science Framework. https://doi.org/10.17605/osf.io/3rf8b
Jasny, B. R., Chin, G., Chong, L., & Vignieri, S. (2011). Again, and Again, and Again ... Science, 334(6060), 1225–1225. https://doi.org/10.1126/science.334.6060.1225
Koole, S. L., & Lakens, D. (2012). Rewarding Replications: A Sure and Simple Way to Improve Psychological Science. Perspectives on Psychological Science, 7(6), 608–614. https://doi.org/10.1177/1745691612462586
Kuehberger, A., & Schulte-Mecklenbeck, M. (2018). Selecting target papers for replication. Behavioral and Brain Sciences, 41. https://doi.org/10.1017/S0140525X18000742
LeBel, E. P., Berger, D., Campbell, L., & Loving, T. J. (2017). Falsifiability is not optional. Journal of Personality and Social Psychology, 113(2), 254–261. https://doi.org/10.1037/pspi0000106
LeBel, E. P., McCarthy, R. J., Earp, B. D., Elson, M., & Vanpaemel, W. (2018). A Unified Framework to Quantify the Credibility of Scientific Findings. Advances in Methods and Practices in Psychological Science, 1(3), 389–402. https://doi.org/10.1177/2515245918787489
Moonesinghe, R., Khoury, M. J., & Janssens, A. C. J. W. (2007). Most Published Research Findings Are False—But a Little Replication Goes a Long Way. PLoS Medicine, 4(2), e28. https://doi.org/10.1371/journal.pmed.0040028
Rosenthal, R. (1990). Replication in behavioral research. Journal of Social Behavior & Personality, 5(4), 1–30.
Schmidt, S. (2009). Shall we really do it again? The powerful concept of replication is neglected in the social sciences. Review of General Psychology, 13(2), 90–100. https://doi.org/10.1037/a0015108
Stroop, J. R. (1935). Studies of interference in serial verbal reactions. Journal of Experimental Psychology, 18(6), 643–662. https://doi.org/10.1037/h005465
Ontologies, metascience, and me
No full textSlides from presentation on the widespread potential for ontologies as a tool in psychological science and meta-science. The talk included brief notes on replication value, research mapping, and the Psychological Science Accelerator. </p
μ-Opioid Modulation of Wanting of Palatable Food Images
No full textThe endogenous μ-opioid receptor (MOR) system in the brain is central to reward behaviors across species, and brain areas implicated in reward are dense with μ-opioid receptors. The MOR system has received the most interest through its involvement in pleasure mediation (‘liking’), but there is much evidence to suggest a role for the MOR system in motivated ‘wanting’ as well. Nevertheless, we still know very little about the mechanisms of MOR modulation in reward motivation in healthy humans. Further, it is unclear to what extent the animal research on MOR modulation of reward-processing in the brain can be extended to humans, as very few studies have explored this relationship directly in the human brain. We examined the effects of a low dose (10mg) of per oral morphine (a μ-opioid agonist) on reported food wanting, and of applying a cognitive regulation task to downregulate this wanting, in healthy human participants. We also measured neural activity as approximated by functional magnetic resonance imaging. The study was designed to minimize the risk of potential confound effects of the drug manipulation. In a within-subject, counterbalanced, placebo-controlled, double-blind design, 63 participants (31 male, mean age 27 ±5) were tested in a morphine and placebo session on two separate days. In line with our expectations, morphine did not significantly affect subjective mood or state, respiration- or heart rate, or motor coordination. Morphine also did not appear to alter global BOLD, measured by a simple visual control task. The food wanting task elicited significant activation in reward related regions compared to baseline, and cognitive regulation produced the expected decrease in food wanting, together with increased activity in ventral prefrontal regions. Activation in extrastriate occipital regions was observed across tasks. Preliminary analyses confirmed our hypothesis that MOR agonism would increase food wanting, but did not confirm our hypothesis of associated activity increase in the striatum and medial prefrontal areas. Instead, increased activity in regulation-related regions may be required for successful downregulation of wanting after morphine treatment. In summary, we have now validated the paradigm and task design of this study. Thus, a complete analysis of the drug effects of interest can be conducted and the results interpreted to draw meaningful conclusions regarding the effects of MOR stimulation with morphine on BOLD signals relating to ‘wanting’ for palatable food images
Replication value as a function of citation impact and sample size:response to commentaries
No full textThe primary goal of our target article (Isager et al., 2025) is to give the research community an example of what a well-justified replication value metric could look like, and to encourage discussion of how replication value could be quantified in practice. Furthermore, in the target article we discuss practical hurdles to quantification and possible practical applications for RVCn and other metrics. As that article proposes a method for how to do research–in this case a method to select which claims in the literature need replication most–it is important to receive criticism, feedback, and viewpoints from a diverse range of authors interested in this topic. We are delighted to read the many thoughtful yet critical commentaries, several of which proposing adjustments or alternatives to the equations we have proposed in the target article. This is very encouraging to see, as our aim with initiating this call in Meta-Psychology was to create an open dialogue in the scientific record. RVCn is an efficient but limited metric. Its limitations should be laid bare, and we fully expect that improved metrics and selection procedures can be created in the future. We hope our target article and these commentaries together will inspire readers to continue the discussion of how to efficiently and transparently select studies for replication. In this rejoinder we will summarize what we see as the major themes touched on in the commentaries, and we will reply to some of the specific proposals and criticisms brought up by different commentary authors
Morphine reduced perceived anger from neutral and implicit emotional expressions
Full text linkThe μ-opioid system modulates responses to pain and psychosocial stress and mediates non-social and social reward. In humans, the μ-opioid agonist morphine can increase overt attention to the eye-region and visual exploration of faces with neutral expressions. However, little is known about how the human μ-opioid system influences sensitivity to and appraisal of subtle and explicit cues of social threats and reward. Here, we examined the effects of selective μ-opioid stimulation on perception of anger and happiness in faces with explicit, neutral or implicit emotion expressions. Sixty-three healthy adults (32 females) attended two sessions where they received either placebo or 10 mg per oral morphine in randomised order under double-blind conditions. Based on the known μ-opioid reduction of pain and discomfort, as well as reports suggesting that the non-specific partial agonist buprenorphine or the non-specific antagonist naltrexone affect appraisal of social emotional stimuli, we hypothesised that morphine would reduce threat sensitivity and enhance perception of happy facial expressions. While overall perception of others’ happiness was unaffected by morphine treatment, morphine reduced perception of anger in stimuli with neutral and implicit expressions without affecting perception of explicit anger. This effect was statistically unrelated to gender, subjective drug effects, mood and autism trait measures. The finding that a low dose of μ-agonist reduced the propensity to perceive anger in photos with subtle facial expressions is consistent with the notion that μ-opioids mediate social confidence and reduce sensitivity to threat cues
Deciding what to replicate: A decision model for replication study selection under resource and knowledge constraints.
Full text linkRobust scientific knowledge is contingent upon replication of original findings. However, replicating researchers are constrained by resources, and will almost always have to choose one replication effort to focus on from a set of potential candidates. To select a candidate efficiently in these cases, we need methods for deciding which out of all candidates considered would be the most useful to replicate, given some overall goal researchers wish to achieve. In this article we assume that the overall goal researchers wish to achieve is to maximize the utility gained by conducting the replication study. We then propose a general rule for study selection in replication research based on the replication value of the set of claims considered for replication. The replication value of a claim is defined as the maximum expected utility we could gain by conducting a replication of the claim, and is a function of (a) the value of being certain about the claim, and (b) uncertainty about the claim based on current evidence. We formalize this definition in terms of a causal decision model, utilizing concepts from decision theory and causal graph modeling. We discuss the validity of using replication value as a measure of expected utility gain, and we suggest approaches for deriving quantitative estimates of replication value. Our goal in this article is not to define concrete guidelines for study selection, but to provide the necessary theoretical foundations on which such concrete guidelines could be built.Translational Abstract Replication-redoing a study using the same procedures-is an important part of checking the robustness of claims in the psychological literature. The practice of replicating original studies has been woefully devalued for many years, but this is now changing. Recent calls for improving the quality of research in psychology has generated a surge of interest in funding, conducting, and publishing replication studies. Because many studies have never been replicated, and researchers have limited time and money to perform replication studies, researchers must decide which studies are the most important to replicate. This way scientists learn the most, given limited resources. In this article, we lay out what it means to think about what is the most important thing to replicate, and we propose a general decision rule for picking a study to replicate. That rule depends on a concept we call replication value. Replication value is a function of the importance of the study, and how uncertain we are about the findings. In this article we explain how researchers can think precisely about the value of replication studies. We then discuss when and how it makes sense to use replication value as a measure of how valuable a replication study would be, and we discuss factors that funders, journals, or scientists could consider when determining how valuable a replication study is.</p
