586 research outputs found
On Selecting and Conditioning in Multiple Testing and Selective Inference
No full textWe investigate a class of methods for selective inference that condition on a
selection event. Such methods follow a two-stage process. First, a data-driven
(sub)collection of hypotheses is chosen from some large universe of hypotheses.
Subsequently, inference takes place within this data-driven collection,
conditioned on the information that was used for the selection. Examples of
such methods include basic data splitting, as well as modern data carving
methods and post-selection inference methods for lasso coefficients based on
the polyhedral lemma. In this paper, we adopt a holistic view on such methods,
considering the selection, conditioning, and final error control steps together
as a single method. From this perspective, we demonstrate that multiple testing
methods defined directly on the full universe of hypotheses are always at least
as powerful as selective inference methods based on selection and conditioning.
This result holds true even when the universe is potentially infinite and only
implicitly defined, such as in the case of data splitting. We provide a
comprehensive theoretical framework, along with insights, and delve into
several case studies to illustrate instances where a shift to a non-selective
or unconditional perspective can yield a power gain
Valid double-dipping via permutation-based closed testing
No full textFunctional Magnetic Resonance Imaging (fMRI) cluster analysis is widely popular for finding neural activation associated with some stimulus. However, it suffers from the spatial specificity paradox, and making follow-up inference inside clusters is not allowed. Valid double-dipping can be performed by closed testing, which determines lower confidence bounds for the number of active voxels, simultaneously over all regions. Moreover, a permutation framework adapts to the unknown joint distribution of the data. In the fMRI context, we evaluate two methods that rely on closed testing and permutations: permutation-based true discovery guarantee by sum tests, and permutation-based All-Resolutions Inference
Multiple Testing for Exploratory Research
No full textDevelopment and application of statistical models for medical scientific researc
The Inheritance Procedure: Multiple Testing of Tree-structured Hypotheses
No full textDevelopment and application of statistical models for medical scientific researc
Rotation-based Multiple Testing in the Multivariate Linear Model.
No full textPermutation methods are very useful in several scientic elds. They have the advantage of making fewer assumptions about the data and of providing more reliable inferential results. They are also particularly useful in case of high-dimensional problems since they easily account for dependence between tests, thereby allowing for more powerful multiplicity control procedures. Indeed, Westfall and Young's min-p procedure often improves on the Holm procedure by providing more rejections. The advantage of being able to make fewer assumptions about the process generating the data unfortunately involves an inherent limitation in the way a process can be modeled (e.g. through multiple linear models). In this work, we propose a permutation (and rotation) method which allows the inference in the multivariate linear model even in the presence of covariates (i.e. nuisance parameters, i.e. confounders). Also, the method allows for the immediate application of the min-p procedure. We make clear how permutations are a particular case of rotations of the data. Permutation tests are exact, while rotation tests retain exactness under multiple-multivariate linear model with normal errors. When errors are not normal, the rotation tests are weakly exchangeable (i.e. approximated and asymptotically exact). A real application to genetic data is presented and discussed
Permutation-Based True Discovery Guarantee by Sum Tests
No full textSum-based global tests are highly popular in multiple hypothesis testing. In
this paper we propose a general closed testing procedure for sum tests, which
provides lower confidence bounds for the proportion of true discoveries (TDP),
simultaneously over all subsets of hypotheses. These simultaneous inferences
come for free, i.e., without any adjustment of the alpha-level, whenever a
global test is used. Our method allows for an exploratory approach, as
simultaneity ensures control of the TDP even when the subset of interest is
selected post hoc. It adapts to the unknown joint distribution of the data
through permutation testing. Any sum test may be employed, depending on the
desired power properties. We present an iterative shortcut for the closed
testing procedure, based on the branch and bound algorithm, which converges to
the full closed testing results, often after few iterations; even if it is
stopped early, it controls the TDP. We compare the properties of different
choices for the sum test through simulations, then we illustrate the
feasibility of the method for high dimensional data on brain imaging and
genomics data.Comment: Main: 27 pages, 3 figures. Appendices: 19 pages, 7 figure
Comparing three groups
No full textFor multiple comparisons in analysis of variance, the practitioners' handbooks generally advocate standard methods such as Bonferroni, or an F-test followed by Tukey's honest significant difference method. These methods are known to be suboptimal compared to closed testing procedures, but improved methods can be complex in the general multigroup set-up. In this note, we argue that the case of three-groups is special: with three groups, closed testing procedures are powerful and easy to use. We describe four different closed testing procedures specifically for the three-group set-up. The choice of method should be determined by assessing which of the comparisons are considered primary and which are secondary, as dictated by subject-matter considerations. We describe how all four methods can be used with any standard software.Development and application of statistical models for medical scientific researc
Closed testing with Globaltest, with application in metabolomics
No full textThe Globaltest is a powerful test for the global null hypothesis that there is no association between a group of features and a response of interest, which is popular in pathway testing in metabolomics. Evaluating multiple feature sets, however, requires multiple testing correction. In this paper, we propose a multiple testing method, based on closed testing, specifically designed for the Globaltest. The proposed method controls the familywise error rate simultaneously over all possible feature sets, and therefore allows post hoc inference, that is, the researcher may choose feature sets of interest after seeing the data without jeopardizing error control. To circumvent the exponential computation time of closed testing, we derive a novel shortcut that allows exact closed testing to be performed on the scale of metabolomics data. An R package ctgt is available on comprehensive R archive network for the implementation of the shortcut procedure, with applications on several real metabolomics data examples
Flexible control of the median of the false discovery proportion
No full textWe introduce a multiple testing procedure that controls the median of the proportion of false discoveries in a flexible way. The procedure only requires a vector of p-values as input and is comparable to the Benjamini–Hochberg method, which controls the mean of the proportion of false discoveries. Our method allows free choice of one or several values of alpha after seeing the data, unlike the Benjamini–Hochberg procedure, which can be very anti-conservative when alpha is chosen post hoc. We prove these claims and illustrate them with simulations. Our procedure is inspired by a popular estimator of the total number of true hypotheses. We adapt this estimator to provide simultaneously median unbiased estimators of the proportion of false discoveries, valid for finite samples. This simultaneity allows for the claimed flexibility. Our approach does not assume independence. The time complexity of our method is linear in the number of hypotheses, after sorting the p-values
- …
