242 research outputs found
pedermisager.org
No full textFiles related to personal website of Peder Mortvedt Isager (pedermisager.org
Open Practices Disclosure, LakensOpenPracticesDisclosure – Equivalence Testing for Psychological Research: A Tutorial
No full textOpen Practices Disclosure, LakensOpenPracticesDisclosure for Equivalence Testing for Psychological Research: A Tutorial by Daniël Lakens, Anne M. Scheel, and Peder M. Isager in Advances in Methods and Practices in Psychological Science</p
Three-Sided Testing to Establish Practical Significance
No full textContains supplementary materials for the article "Three-Sided Testing to Establish Practical Significance: A tutorial" by Peder M Isager and Jack Fitzgerald.
Article abstract: Researchers may want to know whether an observed statistical relationship is either meaningfully negative, meaningfully positive, or small enough to be considered practically equivalent to zero. Such a question can not be addressed with standard null hypothesis significance testing, nor with standard equivalence testing. Three-sided testing (TST) is a procedure to address such questions, by simultaneously testing whether
an estimated relationship is significantly below, within, or above predetermined smallest effect sizes of interest. TST is a natural extension of the standard two one-sided test for equivalence (TOST). TST offers a more comprehensive decision framework than TOST with no penalty to error rates or statistical power. In this paper, we give a non-technical introduction to TST, provide commands for conducting TST in both R
and Jamovi, and provide a Shiny app for easy implementation. Whenever a meaningful smallest effect size of interest can be specified, TST should be combined with null hypothesis significance testing as the default frequentist testing procedure
Peder M. Isager's Quick Files
No full textThe Quick Files feature was discontinued and it’s files were migrated into this Project on March 11, 2022. The file URL’s will still resolve properly, and the Quick Files logs are available in the Project’s Recent Activity
Test validity defined as d-connection between target and measured attribute: Expanding the causal definition of Borsboom et al. (2004)
Full text linkThis article suggests a modification to the conception of test validity put forward by Borsboom, Mellenberghand van Heerden (2004). According to the original definition, a test is only valid if test outcomes are causedby variation in the target attribute. According to the d-connection definition of test validity, a test is validfor measuring an attribute if (a) the attribute exists, and (b) variation in the attribute is d-connected tovariation in the measurement outcomes. In other words, a test is valid whenever test outcomes inform useither about whathashappened to the target attribute in the past, or about whatwillhappen to the targetattribute in the future. Thus, the d-connection definition expands the number of scenarios in which a test canbe considered valid. Defining test validity as d-connection between target and measured attribute situatesthe validity concept squarely within the structural causal modeling framework of Pearl (2009)
Quantifying Replication Value: A formula-based approach to study selection in replication research.
No full textBackground: The concept of replication is a central value of empirical science. At the same time scientists do not regard every replication as equally valuable. Even though replications are a cornerstone of empirical science (Bertamini & Munafò, 2012; Falk, 1998; Jasny, Chin, Chong, & Vignieri, 2011; Koole & Lakens, 2012; Moonesinghe, Khoury, & Janssens, 2007; Rosenthal, 1990; Schmidt, 2009), most researchers will agree that conducting 20 direct replications of the classic and extremely robust Stroop color-naming task (Stroop, 1935) would not be the best way to spend one’s grant money. This raises an important question: when is a replication of an empirical finding of sufficient valuable to the scientific community that it should be be performed? Given limited resources, one could also ask: which among currently published findings are the most valuable to replicate? Some discussion of the circumstances under which replication efforts are more or less beneficial has already occurred in the wake of increased replication efforts in psychology (Brandt et al., 2014; Coles, Tiokhin, Scheel, Isager, & Lakens, 2018), and recently suggestions have been put forward for how to select target studies for replication (Field, Hoekstra, Bringmann, & van Ravenzwaaij, 2018; Kuehberger & Schulte-Mecklenbeck, 2018). A comprehensive evaluation of the factors that could be used to quantify the replication value of a study is currently lacking, which is becoming increasingly important now that more replication studies are funded, performed, and published. Objectives: We propose a quantitative approach to help researchers, editors and funders evaluate and compare the replication value of original findings. Our approach rests on two fundamental assumptions: (1) That close replication (LeBel, Berger, Campbell, & Loving, 2017; LeBel, McCarthy, Earp, Elson, & Vanpaemel, 2018) is in principle a worthwhile endeavor, and (2) that there are more original observations worth replicating than we currently have the resources to replicate. In order to help researchers determine which among many findings might be the most promising candidates for replication, we outline a formula-based approach that can be relatively easily used to quantify the replication value of original findings. We propose that two components determine the replication value of empirical findings: (i) the impact of the effect, and (ii) the corroboration of the effect. Impact indicates the influence that the effect has had on scientific theory, research activities, or in society. All else being equal, findings that have had more impact are more important to replicate than findings that have had less impact. Corroboration indicates the empirical observations bearing on the finding, as well as the quality of these observations. As the corroboration of a particular finding increase, it becomes less important to replicate this finding, relative to a finding with little corroboration. The purpose of a replication value formula is to clarify how one intends to weigh the factors one considers important against one another, and to standardize parts of the study selection procedure. Because these formulas can be calculated quickly (and sometimes even automatically), they can be powerful tools for exploring a large set of studies to discover original findings that are particularly replication-worthy, assuming that the input to the formula is meaningful. Their ultimate goal is to make sure that resources spent on replication are efficiently utilized and that all relevant options for study choice can be considered when a replication effort is initialized. Research Questions: 1) What factors are considered important for determining the replication value of a particular finding or result? 2) Can we create a formula that is able to yield meaningful quantitative estimates of the relative replication value of empirical findings, based on metrics related to the impact and the corroboration of the finding? Approach & Preliminary results: To assess whether our conceptualization of replication value is in line with evaluations of replication value in the broader community of researchers, and to better understand how replicating researchers justify decisions of study choice, we conducted a literature review of justifications of study selection in 85 replication reports. The literature review suggests that researchers use many different information sources to assess replication value that could be subsumed under the categories of impact and corroboration (e.g. citation impact, theoretical importance, imprecise estimates, lack of prior replication). However, it is also clear that some types of information cannot easily be quantified (e.g. theoretical importance), and it is clear that factors other than the value of replication matter for the evaluation process as well (e.g. feasibility). We are currently in the process of constructing one version of a replication value formula that captures the impact and corroboration of a finding. Once a formula has been derived, we aim to evaluate whether the candidate studies returned by the formula track researchers’ qualitative judgements of relative replication value. We will pursue this through two lines of inquiry. First, we will calculate replication value for a large number of studies in the psychological literature and evaluate the face-validity of the recommendations produced, as well as inspect formula recommendations for examples where the true replication is known to be very high or very low (e.g. Stroop, 1935). Second, we will design a study to assess whether formula-based replication value estimate is able to predict researchers’ intuitive evaluation of replication value. Preliminary assessment of face-validity for data in the Curate Science database suggests that the formula yields sensible estimates of replication value for cases where true replication value is known. At the time of the conference, we expect to have completed a comprehensive evaluation of formula performance for both the Curate Science database and a large sample of published studies from the psychological literature. In addition, we will be able to present the planned experimental design for the study that will compare formula recommendations to researchers’ intuitive judgements. References: Bertamini, M., & Munafò, M. R. (2012). Bite-Size Science and Its Undesired Side Effects. Perspectives on Psychological Science, 7(1), 67–71. https://doi.org/10.1177/1745691611429353
Brandt, M. J., IJzerman, H., Dijksterhuis, A., Farach, F. J., Geller, J., Giner-Sorolla, R., … Van’t Veer, A. (2014). The replication recipe: What makes for a convincing replication? Journal of Experimental Social Psychology, 50, 217–224.
Coles, N., Tiokhin, L., Scheel, A., Isager, P., & Lakens, D. (2018). The Costs and Benefits of Replication Studies. https://doi.org/10.17605/osf.io/c8akj
Falk, R. (1998). Replication-A Step in the Right Direction: Commentary on Sohn. Theory & Psychology, 8(3), 313–321. https://doi.org/10.1177/0959354398083002
Field, S., Hoekstra, R., Bringmann, L., & van Ravenzwaaij, D. (2018). When and Why to Replicate: As Easy as 1, 2, 3? Open Science Framework. https://doi.org/10.17605/osf.io/3rf8b
Jasny, B. R., Chin, G., Chong, L., & Vignieri, S. (2011). Again, and Again, and Again ... Science, 334(6060), 1225–1225. https://doi.org/10.1126/science.334.6060.1225
Koole, S. L., & Lakens, D. (2012). Rewarding Replications: A Sure and Simple Way to Improve Psychological Science. Perspectives on Psychological Science, 7(6), 608–614. https://doi.org/10.1177/1745691612462586
Kuehberger, A., & Schulte-Mecklenbeck, M. (2018). Selecting target papers for replication. Behavioral and Brain Sciences, 41. https://doi.org/10.1017/S0140525X18000742
LeBel, E. P., Berger, D., Campbell, L., & Loving, T. J. (2017). Falsifiability is not optional. Journal of Personality and Social Psychology, 113(2), 254–261. https://doi.org/10.1037/pspi0000106
LeBel, E. P., McCarthy, R. J., Earp, B. D., Elson, M., & Vanpaemel, W. (2018). A Unified Framework to Quantify the Credibility of Scientific Findings. Advances in Methods and Practices in Psychological Science, 1(3), 389–402. https://doi.org/10.1177/2515245918787489
Moonesinghe, R., Khoury, M. J., & Janssens, A. C. J. W. (2007). Most Published Research Findings Are False—But a Little Replication Goes a Long Way. PLoS Medicine, 4(2), e28. https://doi.org/10.1371/journal.pmed.0040028
Rosenthal, R. (1990). Replication in behavioral research. Journal of Social Behavior & Personality, 5(4), 1–30.
Schmidt, S. (2009). Shall we really do it again? The powerful concept of replication is neglected in the social sciences. Review of General Psychology, 13(2), 90–100. https://doi.org/10.1037/a0015108
Stroop, J. R. (1935). Studies of interference in serial verbal reactions. Journal of Experimental Psychology, 18(6), 643–662. https://doi.org/10.1037/h005465
Replication value as a function of citation impact and sample size
No full textResearchers seeking to replicate original research often need to decide which of several relevant candidates to select for replication. Several strategies for study selection have been proposed, utilizing a variety of observed indicators as criteria for selection. However, few strategies clearly specify the goal of study selection and how that goal is related to the indicators that are utilized. We have previously formalized a decision model of replication study selection in which the goal of study selection is to maximize the expected utility gain of the replication effort. We further define the concept of replication value as a proxy for expected utility gain (Isager et al., 2023). In this article, we propose a quantitative operationalization of replication value. We first discuss how value and uncertainty - the two concepts used to determine replication value – could be estimated via information about citation count and sample size. Second, we propose an equation for combining these indicators into an overall estimate of replication value, which we denote RVCn. Third, we suggest how RVCn could be implemented as part of a broader study selection procedure. Finally, we provide preliminary data suggesting that studies that were in fact selected for replication tend to have relatively high RVCn estimates. The goal of this article is to explain how RVCn is intended to work and, in doing so, demonstrate the many assumptions that should be explicit in any replication study selection strategy
Ontologies, metascience, and me
No full textSlides from presentation on the widespread potential for ontologies as a tool in psychological science and meta-science. The talk included brief notes on replication value, research mapping, and the Psychological Science Accelerator. </p
Exploring a Formal Approach to Selecting Studies for Replication: a Feasibility Study in Social Neuroscience
No full textThis exploratory report (accepted for publication in Cortex, preprint here: https://osf.io/preprints/metaarxiv/3u52w/) tests, amongst other things, the feasibility of implementing Replication Value (RVcn), see: Isager, P.M., van 't Veer, A.E., & Lakens, D. (2021). Replication value as a function of citation impact and sample size. https://doi.org/10.31222/osf.io/knjea. The accompanying Github repository for this project can be found here: https://github.com/pederisager/NeuroRep_RV, however, the files on this osf project should be stand-alone, see Wiki for File contents description.
Abstract:
Replication of published results is crucial for ensuring the robustness and self- correction of research, yet replications are scarce in many fields. Replicating researchers will therefore often have to decide which of several relevant candidates to target for replication. Formal strategies for efficient study selection have been proposed, but none have been explored for practical feasibility–a prerequisite for validation. Here we move one step closer to efficient replication study selection by exploring the feasibility of a particular selection strategy that estimates replication value as a function of citation impact and sample size (Isager, van ’t Veer, & Lakens, 2021). We tested our strategy on a sample of fMRI studies in social neuroscience. We first report our efforts to generate a representative candidate set of replication targets. We then explore the feasibility and reliability of estimating replication value for the targets in our set, resulting in a dataset of 1358 studies ranked on their value of prioritising them for replication. In addition, we carefully examine possible measures, test auxiliary assumptions, and identify boundary conditions of measuring value and uncertainty. We end our report by discussing how future validation studies might be designed. Our study demonstrates the importance of investigating how to implement study selection strategies in practice. Our sample and study design can be extended to explore the feasibility of other formal study selection strategies that have been proposed
μ-Opioid Modulation of Reported Wanting of Palatable Food Images:A pharmacological fMRI study in healthy humans
No full textThe endogenous μ-opioid receptor (MOR) system in the brain is central to reward behaviors across species, and brain areas implicated in reward are dense with μ-opioid receptors. The MOR system has received the most interest through its involvement in pleasure mediation (‘liking’), but there is much evidence to suggest a role for the MOR system in motivated ‘wanting’ as well. Nevertheless, we still know very little about the mechanisms of MOR modulation in reward motivation in healthy humans. Further, it is unclear to what extent the animal research on MOR modulation of reward-processing in the brain can be extended to humans, as very few studies have explored this relationship directly in the human brain. We examined the effects of a low dose (10mg) of per oral morphine (a μ-opioid agonist) on reported food wanting, and of applying a cognitive regulation task to downregulate this wanting, in healthy human participants. We also measured neural activity as approximated by functional magnetic resonance imaging. The study was designed to minimize the risk of potential confound effects of the drug manipulation. In a within-subject, counterbalanced, placebo-controlled, double-blind design, 63 participants (31 male, mean age 27 ±5) were tested in a morphine and placebo session on two separate days. In line with our expectations, morphine did not significantly affect subjective mood or state, respiration- or heart rate, or motor coordination. Morphine also did not appear to alter global BOLD, measured by a simple visual control task. The food wanting task elicited significant activation in reward related regions compared to baseline, and cognitive regulation produced the expected decrease in food wanting, together with increased activity in ventral prefrontal regions. Activation in extrastriate occipital regions was observed across tasks. Preliminary analyses confirmed our hypothesis that MOR agonism would increase food wanting, but did not confirm our hypothesis of associated activity increase in the striatum and medial prefrontal areas. Instead, increased activity in regulation-related regions may be required for successful downregulation of wanting after morphine treatment. In summary, we have now validated the paradigm and task design of this study. Thus, a complete analysis of the drug effects of interest can be conducted and the results interpreted to draw meaningful conclusions regarding the effects of MOR stimulation with morphine on BOLD signals relating to ‘wanting’ for palatable food images
- …
