49 research outputs found
Formalizzazione delle Ipotesi di Ricerca in Psicologia: Design Analysis e Model Comparison
No full textLa valutazione di ipotesi definite in accordo con le aspettative dei ricercatori o di prospettive teoriche è uno degli obiettivi principali della ricerca empirica. Quando viene condotto uno studio, infatti, i ricercatori di solito vogliono valutare la plausibilità delle loro ipotesi sulla base dei dati osservati. Per fare ciò, sono stati sviluppati diversi approcci statistici come, ad esempio, il Null Hypothesis Significance Testing (NHST).
In psicologia, il NHST è l'approccio statistico dominante per valutare le ipotesi di ricerca. In realtà, tuttavia, l'approccio NHST non consente ai ricercatori di rispondere alla domanda a cui di solito sono interessati. Infatti, l'approccio NHST non quantifica l'evidenza a favore di un'ipotesi, ma quantifica solo l'evidenza contro l'ipotesi nulla. Ciò può facilmente portare a un'errata interpretazione dei risultati che, insieme all'applicazione meccanica ad insensata dell'approccio NHST, è considerata una delle cause dell'attuale crisi di replicabilità.
Nella prima parte della tesi, introduciamo il framework della Design Analysis che ci permette di valutare i rischi inferenziali legati alla stima della dimensione dell'effetto quando si seleziona per la significatività. Nel caso di studi con campioni ridotti che valutano fenomeni complessi e con grande variabilità nei dati (tutte condizioni molto comuni in psicologia), la selezione per significatività può facilmente portare a risultati fuorvianti ed inaffidabili. Questo aspetto è spesso trascurato nella Power Analysis tradizionale. La Design Analysis, invece, mette in evidenza questo importante problema.
Nella seconda parte della tesi, ci spostiamo dal NHST verso l'approccio del Model Comparison. Il Model Comparison ci consente di valutare correttamente l'evidenza relativa a favore di un'ipotesi in base ai dati. In primo luogo, le ipotesi di ricerca vengono formalizzate sotto forma di diversi modelli statistici. Successivamente, queste vengono valutate secondo diversi possibili criteri come, ad esempio, gli Information Criteria e il Bayes Factor con encompassing prior. Gli Information Criteria valutano la capacità predittiva dei modelli penalizzando per la complessità del modello. Il Bayes Factor con encompassing prior, invece, consente ai ricercatori di valutare facilmente ipotesi informative con vincoli di uguaglianza e disuguaglianza sui parametri del modello.The evaluation of research and theoretical hypotheses is one of the principal goals of empirical research. In fact, when conducting a study, researchers usually have expectations based on hypotheses or theoretical perspectives they want to evaluate according to the observed data. To do that, different statistical approaches have been developed, for example, the Null Hypothesis Significance Testing (NHST).
In psychology, the NHST is the dominant statistical approach to evaluate research hypotheses. In reality, however, the NHST approach does not allow researchers to answer the question they usually are interested in. In fact, the NHST approach does not quantify the evidence in favour of a hypothesis, but it only quantifies the evidence against the null hypothesis. This can easily lead to the misinterpretation of the results that, together with a mindless and mechanical application of the NHST approach, is considered as one of the causes of the ongoing replicability crisis.
In the first part of the thesis, we introduce the Design Analysis framework that allows us to evaluate the inferential risks related to effect size estimation when selecting for significance. In the case of underpowered studies evaluating complex multivariate phenomena with noisy data (all very common conditions in psychology), selecting for significance can easily lead to misleading and unreliable results. This aspect is often neglected in traditional power Analysis. Design analysis, instead, highlights this relevant issue.
In the second part of the thesis, we move away from the NHST towards the model comparison approach. Model comparison allows us to properly evaluate the relative evidence in favour of one hypothesis according to the data. First, research hypotheses are formalized into different statistical models, subsequently, these are evaluated according to different possible criteria. We consider the information criteria and the Bayes Factor with encompassing prior. Information criteria assess models predictive ability penalizing for model complexity. Bayes Factor with encompassing prior, instead, allows researchers to easily evaluate informative hypotheses with equality and inequality constraints on the model parameters
Designing Studies and Evaluating Research Results: Type M and Type S Errors for Pearson Correlation Coefficient
No full textIt is widely appreciated that many studies in psychological science suffer from low statistical power. One of the consequences of analyzing underpowered studies with thresholds of statistical significance is a high risk of finding exaggerated effect size estimates, in the right or the wrong direction. These inferential risks can be directly quantified in terms of Type M (magnitude) error and Type S (sign) error, which directly communicate the consequences of design choices on effect size estimation. Given a study design, Type M error is the factor by which a statistically significant effect is on average exaggerated. Type S error is the probability to find a statistically significant result in the opposite direction to the plausible one. Ideally, these errors should be considered during a prospective design analysis in the design phase of a study to determine the appropriate sample size. However, they can also be considered when evaluating studies’ results in a retrospective design analysis. In the present contribution, we aim to facilitate the considerations of these errors in the research practice in psychology. For this reason, we illustrate how to consider Type M and Type S errors in a design analysis using one of the most common effect size measures in psychology: Pearson correlation coefficient. We provide various examples and make the R functions freely available to enable researchers to perform design analysis for their research projects
Stereotype Threat Effects on Italian Girls' Mathematics Performance: A Failure to Replicate
No full textMany studies have found that males, on average, perform better than females in mathematics, although the size of this gender gap is small and varies considerably across countries. Stereotype threat has been proposed as a principal cause of this gender gap. From this perspective, females’ performance is affected by fear of confirming a negative stereotype about females’ mathematical ability and this stereotype can be activated by an experimental manipulation that reminds females of the stereotype. Yet, evidence of a stereotype threat effect on mathematics performance in childhood and adolescence has been mixed. The present study replicated a highly cited study of stereotype threat among Italian adolescents with a much larger sample of Italian ninth grade (89 male, 75 female, mean age = 14.2) and eleventh grade (84 male, 80 female, mean age = 16.2) public high school students. Performance in tests administered both before and after the experimental manipulations were analyzed with a series of logistic mixed-effects models. Model comparisons confirmed that males performed better than females, but the probability of a stereotype threat effect was infinitesimal. We conclude that Italian adolescent gender differences in mathematics may not be explained by stereotype threat effect
Effects of digital games on student motivation in mathematics: A meta-analysis in K-12
No full textBackground: Motivation is an important factor in the learning process and supporting
students' motivation in mathematics is a significant challenge for educators.
Educational technologies, such as digital games, offer potential for engagement in
mathematics learning activities.
Objectives: To contrast the general decrement in student motivation in mathematics,
a multilevel meta-analysis was carried out to synthesize the results of studies
concerning the impact of digital games on K-12 student motivation in mathematics.
Methods: Standardized measure of effect size (dppc2) for pre- post-control group
designs was used, and different sources of dependency among the effects were
taken into account. Moreover, through meta-regressions, we examined whether specific
characteristics of the participants, interventions and outcomes were associated
with effect size differences throughout the studies.
Results and Conclusions: A total of 20 primary studies (43 effect sizes) meeting eligibility
criteria was included. Results showed a significant overall effect (dppc2 = 0.27;
95%CI = [0.14; 0.41]) and a great heterogeneity between studies. Moderator analyses
showed differences in effect size associated to the duration of intervention and
motivational construct in terms of expectancy and value.
Implications: Overall, the findings indicate that digital games are effective tools compared
to conventional teaching practices. The results are promising and could be useful
for the design of digital educational interventions aimed at promoting motivation
in mathematics
Incorporating Expert Knowledge in Structural Equation Models: Applications in Psychological Research
No full textStructural Equation Modeling (SEM) is used in psychology to model complex structures of data. However, sample sizes often cannot be as large as ideal forSEM, leading to a problem of insufficient power. Bayesian estimation with informed priors can be beneficial in this context. Our simulation study examines this issue over a real case of a mediation model. Parameter recovery, power and coverage were considered. The advantage of a Bayesian approach was evident for the smallest effects. The correct formalization of the theoretical expectations is crucial, and it allows for increased collaboration among researchers in Psychology and Statistics
PRDA: Conduct a Prospective or Retrospective Design Analysis
No full textPRDA is an R package performing prospective or retrospective design analysis to evaluate inferential risks (i.e., power, Type M error, and Type S error) in a study considering Pearson’s correlation between two variables or mean comparisons (one-sample, paired, two-sample, and Welch’st-test). Prospective Design Analysis is performed in the planning stage of a study to define the required sample size to obtain a given level of power. Retrospective Design Analysis, instead, is performed when the data have already been collected to evaluate the inferential risks associated with the study
Effectiveness of digital-based interventions for children with mathematical learning difficulties: A meta-analysis
No full textThe purpose of this work was to meta-analyze empirical evidence about the effectiveness of digital-based interventions for students with mathematical learning difficulties. Furthermore, we investigated whether the school level of the participants and the software instructional approach were decisive modulated factors. A systematic search of randomized controlled studies published between 2003 and 2019 was conducted. A total of 15 studies with 1073 participants met the study selection criterion. A random effects meta-analysis indicated that digital-based interventions generally improved mathematical performance (mean ES = 0.55), though there was a significant heterogeneity across studies. There was no evidence that videogames offer additional advantages with respect to digital-based drilling and tutoring approaches. Moreover, effect size was not moderated when interventions were delivered in primary school or in preschool
Enhancing Statistical Inference in Psychological Research via Prospective and Retrospective Design Analysis
No full textIn the past two decades, psychological science has experienced an unprecedented replicability crisis, which has uncovered several issues. Among others, the use and misuse of statistical inference plays a key role in this crisis. Indeed, statistical inference is too often viewed as an isolated procedure limited to the analysis of data that have already been collected. Instead, statistical reasoning is necessary both at the planning stage and when interpreting the results of a research project. Based on these considerations, we build on and further develop an idea proposed by Gelman and Carlin (2014) termed “prospective and retrospective design analysis.” Rather than focusing only on the statistical significance of a result and on the classical control of type I and type II errors, a comprehensive design analysis involves reasoning about what can be considered a plausible effect size. Furthermore, it introduces two relevant inferential risks: the exaggeration ratio or Type M error (i.e., the predictable average overestimation of an effect that emerges as statistically significant) and the sign error or Type S error (i.e., the risk that a statistically significant effect is estimated in the wrong direction). Another important aspect of design analysis is that it can be usefully carried out both in the planning phase of a study and for the evaluation of studies that have already been conducted, thus increasing researchers' awareness during all phases of a research project. To illustrate the benefits of a design analysis to the widest possible audience, we use a familiar example in psychology where the researcher is interested in analyzing the differences between two independent groups considering Cohen's d as an effect size measure. We examine the case in which the plausible effect size is formalized as a single value, and we propose a method in which uncertainty concerning the magnitude of the effect is formalized via probability distributions. Through several examples and an application to a real case study, we show that, even though a design analysis requires significant effort, it has the potential to contribute to planning more robust and replicable studies. Finally, future developments in the Bayesian framework are discussed
The role of vision and proprioception in self-motion encoding: An immersive virtual reality study
No full textPast research on the advantages of multisensory input for remembering spatial information has mainly focused on memory for objects or surrounding environments. Less is known about the role of cue combination in memory for own body location in space. In a previous study, we investigated participants' accuracy in reproducing a rotation angle in a self-rotation task. Here, we focus on the memory aspect of the task. Participants had to rotate themselves back to a specified starting position in three different sensory conditions: a blind condition, a condition with disrupted proprioception, and a condition where both vision and proprioception were reliably available. To investigate the difference between encoding and storage phases of remembering proprioceptive information, rotation amplitude and recall delay were manipulated. The task was completed in a real testing room and in immersive virtual reality (IVR) simulations of the same environment. We found that proprioceptive accuracy is lower when vision is not available and that performance is generally less accurate in IVR. In reality conditions, the degree of rotation affected accuracy only in the blind condition, whereas in IVR, it caused more errors in both the blind condition and to a lesser degree when proprioception was disrupted. These results indicate an improvement in encoding own body location when vision and proprioception are optimally integrated. No reliable effect of delay was found
Proprioceptive accuracy in Immersive Virtual Reality: A developmental perspective
No full textProprioceptive development relies on a variety of sensory inputs, among which vision is hugely dominant. Focusing on the developmental trajectory underpinning the integration of vision and proprioception, the present research explores how this integration is involved in interactions with Immersive Virtual Reality (IVR) by examining how proprioceptive accuracy is affected by Age, Perception, and Environment. Individuals from 4 to 43 years old completed a self-turning task which asked them to manually return to a previous location with different sensory modalities available in both IVR and reality. Results were interpreted from an exploratory perspective using Bayesian model comparison analysis, which allows the phenomena to be described using probabilistic statements rather than simplified reject/not-reject decisions. The most plausible model showed that 4–8-year-old children can generally be expected to make more proprioceptive errors than older children and adults. Across age groups, proprioceptive accuracy is higher when vision is available, and is disrupted in the visual environment provided by the IVR headset. We can conclude that proprioceptive accuracy mostly develops during the first eight years of life and that it relies largely on vision. Moreover, our findings indicate that this proprioceptive accuracy can be disrupted by the use of an IVR headset
