11 research outputs found
An Introduction to the DA-T Gibbs Sampler for the Two-Parameter Logistic (2PL) Model and Beyond
No full textThe DA-T Gibbs sampler is proposed by Maris and Maris (2002) as a
 Bayesian estimation method for a wide variety of Item Response Theory
 (IRT) models. The present paper provides an expository account of the DAT
 Gibbs sampler for the 2PL model. However, the scope is not limited to
 the 2PL model. It is demonstrated how the DA-T Gibbs sampler for the 2PL
 may be used to build, quite easily, Gibbs samplers for other IRT models.
 Furthermore, the paper contains a novel, intuitive derivation of the Gibbs
 sampler and could be read for a graduate course on sampling
A limited dependent variable model for heritability estimation with non-random ascertained samples
No full textIn a questionnaire study, a random sample of Dutch families was asked whether they suffered from asthma and related symptoms. From these families, a selected sample was invited to come to the hospital for further phenotyping. Families were selected if at least one family member reported a history of asthma and the twins were 18 years of age or older. Not all families that were thus selected volunteered, leaving us with a fraction of the original sample. The aim of this paper is to describe a limited dependent variable model that can be used in such situations in order to obtain estimates that are representative of the population from which the sample was originally drawn. The model is a linear (DeFries-Fulker) regression model corrected for sample selection. This correction is possible when (some of) the characteristics that determine whether subjects volunteer (or not) are known for all subjects, including those that did not volunteer. The questionnaire study is of interest by itself but serves mainly to provide a concrete illustration of our method. The present model is used to analyze the data and the results are compared to those obtained with other methods: raw (or direct) likelihood estimation, multiple imputation, and sample weighting. Throughout, Rubin's general theory of inference with missing data serves as an integrating framework
What can we learn from Plausible Values?
No full textIn this paper, we show that the marginal distribution of plausible values is a consistent estimator of the true latent variable distribution, and, furthermore, that convergence is monotone in an embedding in which the number of items tends to infinity. We use this result to clarify some of the misconceptions that exist about plausible values, and also show how they can be used in the analyses of educational surveys
Detecting halo effects in performance-based examinations
No full textThe main purpose of this article is to demonstrate how halo effects may be detected and quantified using two independent ratings of the same person. A practical illustration is given to show how halo effects can be avoided
Turning simulation into estimation: Generalized exchange algorithms for exponential family models
No full textThe Single Variable Exchange algorithm is based on a simple idea; any model that can be simulated can be estimated by producing draws from the posterior distribution. We build on this simple idea by framing the Exchange algorithm as a mixture of Metropolis transition kernels and propose strategies that automatically select the more efficient transition kernels. In this manner we achieve significant improvements in convergence rate and autocorrelation of the Markov chain without relying on more than being able to simulate from the model. Our focus will be on statistical models in the Exponential Family and use two simple models from educational measurement to illustrate the contribution
Composition Algorithms For Conditional Distributions
No full textThis chapter is about two recently published algorithms that can be used to sample from conditional distributions. We show how the efficiency of the algorithms can be improved when a sample is required from many conditional distributions.
Using real-data examples from educational measurement, we show how the algorithms can be used to sample from intractable full-conditional distributions of the person and item parameters in an application of the Gibbs sampler
A mixing distribution for SVE with oversampling.
No full textThe distribution of transition kernels, i.e., , for sampling from the posterior π(θ ∣ x+ = 9) when choosing the best one out of m = 5 generated proposals (left panel) and m = 20 generated proposals (right panel). In this example the acceptance rate was equal to 75% when generating m = 5 proposals and equal to 95% when generating m = 20 proposals.</p
Bayesian inference for low-rank Ising networks
No full textEstimating the structure of Ising networks is a notoriously difficult problem. We demonstrate that using a latent variable representation of the Ising network, we can employ a full-data-information approach to uncover the network structure. Thereby, only ignoring information encoded in the prior distribution (of the latent variables). The full-data-information approach avoids having to compute the partition function and is thus computationally feasible, even for networks with many nodes. We illustrate the full-data-information approach with the estimation of dense network
The validity of international surveys of reading literacy: the case of the IEA reading literacy study
No full textAn Item Response Theory Approach to the Maintenance of Standards in Public Examinations in England
No full textAbstract
Every year outcomes from public examinations in the UK rise: politicians congratulate pupils on their hard earned achievement; the media questions whether this achievement is real; those responsible for administrating the examinations defend their standards; various subject councils and employers decry the failings of candidates with high grades; admissions officers from the elite universities report their struggle with the decrease in discrimination in grades achieved; and academics debate what it means to compare standards from one year to the next. The debate cannot be easily resolved because examination results are put to many purposes some of which are more suited to certain definitions of comparability than others. In procedural terms, however, it should be relatively straightforward to evaluate the strength of the evidence that is put forward on the comparability of standards against various definitions.
Broadly, solely in terms of discrimination, the statistical evidence in the maintenance of standards over time and between qualifications can be evaluated by reference to measures such as model fit, significance and effect size. An evaluation of the literature suggests that predictive statistical models, where employed in the maintenance of standards to meet definitions of cohort referencing, tend to be robust. Beyond discrimination, measures of performance standards are required to support inferences drawn from grades on what candidates can actually do. These are, and have been for many years, underpinned by processes reliant on human judgement. An evaluation of the literature suggests that judgement provides very weak evidence and is subject to unknown bias. The combination of statistical and judgemental evidence is poorly specified, has no theoretical basis and is therefore impossible to evaluate. If anything more than pure cohort referencing is required from public examinations in the UK there is clearly a need to explore models with a sound theoretical basis whose evidence can be evaluated in terms of model fit, significance and effect size.
The task of maintaining a performance standard can essentially be reduced under test theory to making comparisons between persons that are independent of the items on the basis of which these comparisons are made. Test theory however has been sparingly applied to comparability issues in UK public examinations. This study considers which test theory model would be most suited to the examinations in use in the UK, examines issues of model fit under frequentist and Bayesian frameworks, compares the results from different test equating methods and the practical issues of implementing a test equating design under the given constraints of the UK examination system.
To begin with the Rasch model and the One Parameter Logistic Model were fitted to operational data gathered from examinations in a range of subject domains where marking reliability would not be considered as a potential confound. In this framework the Rasch model requirement of a single discrimination parameter across items appeared overly restrictive. Further, potential issues with model fit were highlighted related to dimensionality, guessing and weak local independence. More complex models were therefore pursued under a Bayesian framework. The Posterior Predictive Model Checking Procedures and Deviance Information Criterion confirmed that a model which allowed discrimination to vary across items, such as the two-parameter Item Response Theory model, would produce better model predictions. Use of the Multi-Class Mixture Rasch Model suggested that multidimensionality due to a confounding speededness factor could result in misleading inferences being drawn from unidimensional models. The Testlet Response Theory model showed enhanced predictions where weak local independence was correctly specified; however it proved difficult to specify where this weak local independence was expected. When tests from one of the examinations particularly affected by speededness were equated OPLM proved more robust to the confounding speededness factor than the Rasch model.
A Post-equating Non-Equivalent Groups Design was then set up as an experiment using a set of relatively simple Science examinations and candidates at a later stage in their programme of study than those who would take the live examinations in order to understand some of the practical issues involved in equating designs. The study found that item parameters were not stable across samples due to context effects, school effects and maturity effects. These results were partly due to the scale of study, which, though small, still produced reasonably sensible outcomes. It is suggested that more care paid to the context of linking items, their underlying construct, and the sampling of schools would yield more robust results. Finally, a qualitative exploration of views related to test equating designs suggested that teachers, pupils and examiners would not reject the possibility of embedding equating items into live tests.
For examinations where marking reliability is not considered an issue the results reported here suggest that the use of test theory could provide a unified theoretical framework for the maintenance of standards in UK public examinations which would allow the strength of the evidence presented to be evaluated. This would represent a substantial improvement over the current situation in which no comprehensive or coherent evaluation can be made. The time and investment required, however, to introduce such a framework is also substantial. A suitable technical infrastructure is required as well as psychometric expertise. The alternative is to revert to an examinations system that is essentially cohort referenced and focuses on discrimination between candidates in any one year rather than attempting to quality assure, as it cannot do, performance standards from one year to the next
