62 research outputs found
Resampling from the past to improve on MCMC algorithms
No full textWe introduce the idea that resampling from past observations in a Markov Chain Monte Carlo sampler can fasten convergence. We prove that proper resampling from the past does not disturb the limit distribution of the algorithm. We illustrate the method with two examples. The first on a Bayesian analysis of stochastic volatility models and the other on Bayesian phylogeny reconstruction.Monte Carlo methods, Resampling, Stochastic volatility models, Bayesian phylogeny reconstruction.
Topics in sparse Bayesian machine learning
No full text2023This dissertation is devoted to addressing several challenging problems in machine learning via the Bayesian approach. One popular approach to Bayesian deep learning is to use Monte Carlo methods, such as Markov Chain Monte Carlo (MCMC), to approximate the posterior distribution. These methods generate a set of samples from the posterior, which can be used to quantify the uncertainty in the parameters and make probabilistic predictions. Bayesian methods in deep learning provide a framework for incorporating uncertainty into the learning process and can lead to more robust models with improved performance on unseen data. They have been applied to a wide range of problems, including image classification, reinforcement learning, and generative models, among others. This dissertation is organized as follows. First chapter is fast asynchronous sampler in sparse bayesian learning. In this chapter, We propose a very fast approximate Markov Chain Monte Carlo(MCMC) sampling framework that is applicable to a large class of sparse Bayesian inference problems, where the computational cost per iteration in several regression models is of order O(n(s + J)), where n is the sample size, s the underlying sparsity of the model, and J is the size of a randomly selected subset of regressors. This cost can be further reduced by data sub-sampling when stochastic gradient Langevin dynamics are employed. The algorithm is an extension of the asynchronous Gibbs sampler of Johnson et al. (2013), but can be viewed from a statistical perspective as a form of Bayesian iterated sure independent screening (Fan et al. (2009)). We show that in high-dimensional linear regression problems, the Markov chain generated by the proposed algorithm admits an invariant distribution that recovers correctly the main signal with high probability under some statistical assumptions. Furthermore we show that its mixing time is at most linear in the number of regressors. We illustrate the algorithm with several models. Second chapter is A one-step Laplace Approximation for high-dimensional variable selection. In this chapter, we introduce a rapid one-step Laplace approximation method, referred to as OLAP, which effectively tackles the computational burden of variable selection in high dimensions. Our findings demonstrate that this approximation offers a consistent variable selection procedure under reasonable assumptions. Additionally, we establish that the mixing time of the Gibbs sampler, employed for sampling from the posterior distribution of OLAP, scales linearly with the dimension p. Through comprehensive simulations, we validate the efficiency and accuracy of our proposed sampler, highlighting its potential to significantly enhance variable selection processes. Third chapter is Sparse(Cyclical) MCMC in Deep Neural Networks. In this chapter, we propose a general cyclical MCMC framework for a class of Bayesian inference problem, aiming to generate samples from one single mode in each cycle andhave mode swapping among different cycles to capture multimodality. We provide extensive results on the performance of prediction, multimodality of different cyclical MCMC methods on high-dimensional gaussian mixture models. We then introduce the sparse cyclical MCMC sampler in deep neural networks and present promising simulation results from the perspective of uncertainty estimation and calibration
Topics in sparse Bayesian machine learning
No full text2023This dissertation is devoted to addressing several challenging problems in machine learning via the Bayesian approach. These problems frequently arise in diverse fields, such as epidemiology, biomedicine, robust statistics and imaging science, and are usually high-dimensional and have certain sparsity assumptions. In this dissertation, we will focus on three important problems, which are sparse canonical correlation analysis, minimum distance estimation and inverse problems. For each problem, we will develop a new method from the Bayesian perspective to solve it effectively and efficiently, with statistical guarantees and numerical evidence
An Adaptive Version for the Metropolis Adjusted Langevin Algorithm with a Truncated Drift
No full textThis paper proposes an adaptive version for the Metropolis adjusted Langevin algorithm with a truncated drift (T-MALA). The scale parameter and the covariance matrix of the proposal kernel of the algorithm are simultaneously and recursively updated in order to reach the optimal acceptance rate of 0:574 (see Roberts and Rosenthal (2001)) and to estimate and use the correlation structure of the target distribution. We develop some convergence results for the algorithm. A simulation example is presented.Markov Chain Monte Carlo, Stochastic approximation algorithms, Metropolis Adjusted Langevin algorithm, geometric rate of convergence.
Contributions to the Analysis of Multistate and Degradation Data.
No full textTraditional methods in survival, reliability, actuarial science, risk, and other event-history applications are based on the analysis of time-to-occurrence of some event of interest, generically called failure. In the presence of high-degrees of censoring, however, it is difficult to make inference about the underlying failure distribution using failure time data. Moreover, such data are not very useful in predicting failures of specific systems, a problem of interest when dealing with expensive or critical systems. As an alternative, there is an increasing trend towards collecting and analyzing richer types of data related to the states and performance of systems or subjects under study. These include data on multistate and degradation processes. This dissertation makes several contributions to the analysis of multistate and degradation data. The first part of the dissertation deals with parametric inference for multistate processes with panel data. These include interval, right, and left censoring, which arise naturally as the processes are not observed continuously. Most of the literature in this area deal with Markov models, for which inference with censored data can be handled without too much difficulty. The dissertation considers progressive semi-Markov models and develops methods and algorithms for general parametric inference. A combination of Markov Chain Monte Carlo techniques and stochastic approximation methods are used. A second topic deals with the comparison of the traditional method and the process method for inference about the time-to-failure distribution in the presence of multistate data. Here, time-to-failure is the time when the process enters an absorbing state. There is limited literature in this area. The gains in both estimation and prediction efficiency are quantified for various parametric models of interest. The second part of the dissertation deals with the analysis of data on continuous measures of performance and degradation with missing data. In this case, time-to-failure is the time at which the degradation measure exceeds a certain threshold or performance level goes below some threshold. Inference problems about the mean and variance of the degradation and the imputation of the missing are studied under different settings.PhDPure SciencesStatisticsUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/127197/2/3476829.pd
Statistical Inference and Computational Methods for Large High-Dimensional Data with Network Structure.
No full textNew technological advancements have allowed collection of datasets of large volume and different levels of complexity. Many of these datasets have an underlying network structure. Networks are capable of capturing dependence relationship among a group of entities and hence analyzing these datasets unearth the underlying structural dependence among the individuals. Examples include gene regulatory networks, understanding stock markets, protein-protein interaction within the cell, online social networks etc. The thesis addresses two important aspects of large high-dimensional data with network structure. The first one focuses on a high-dimensional data with network structure that evolves over time. Examples of such data sets include time course gene expression data, voting records of legislative bodies etc. The main task is to estimate the change-point as well as the network structures prior and post it. The network structures are obtained by penalized optimization method and we establish a finite sample estimation error bound for the change-point in the high-dimensional regime. The other aspect that we examine is about parameter estimation in large heterogeneous data with network structure. Our primary goal is to develop efficient computational techniques based on random subsampling and parallelization to estimate the parameters. We provide an analysis of rate of decay of bias and variance of our parallel implementation with a single round of communication after every iteration. We further show two applications of our methodology in the case of Gaussian Mixture Model (GMM) and Stochastic Block Model (SBM).The emphasis is placed on developing new theoretical techniques and computational tools for network problems and applying the corresponding methodology in many fields, including biomedical and social science research, where network modeling and analysis plays an exceedingly important role.PhDStatisticsUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/113602/1/sandipan_1.pd
Discussion
No full textPeer Reviewedhttp://deepblue.lib.umich.edu/bitstream/2027.42/106905/1/insr12033.pd
Minimax Quasi-Bayesian estimation in sparse canonical correlation analysis via a Rayleigh quotient function
No full textCanonical correlation analysis (CCA) is a popular statistical technique for
exploring relationships between datasets. In recent years, the estimation of
sparse canonical vectors has emerged as an important but challenging variant of
the CCA problem, with widespread applications. Unfortunately, existing
rate-optimal estimators for sparse canonical vectors have high computational
cost. We propose a quasi-Bayesian estimation procedure that not only achieves
the minimax estimation rate, but also is easy to compute by Markov Chain Monte
Carlo (MCMC). The method builds on Tan et al. (2018) and uses a re-scaled
Rayleigh quotient function as the quasi-log-likelihood. However, unlike Tan et
al. (2018), we adopt a Bayesian framework that combines this
quasi-log-likelihood with a spike-and-slab prior to regularize the inference
and promote sparsity. We investigate the empirical behavior of the proposed
method on both continuous and truncated data, and we demonstrate that it
outperforms several state-of-the-art methods. As an application, we use the
proposed methodology to maximally correlate clinical variables and proteomic
data for better understanding the Covid-19 disease
Large-Scale Quasi-Bayesian Inference with Spike-and-Slab Priors
No full textThis dissertation studies a general framework using spike-and-slab prior distributions to facilitate the development of high-dimensional Bayesian inference. Our framework allows inference with a general quasi-likelihood function to address scenarios where likelihood based inference are infeasible or the underlying optimization problems are not the same as the data generating mechanisms. We show that highly efficient and scalable Markov Chain Monte Carlo (MCMC) algorithms can be easily constructed to sample from the resulting quasi-posterior distributions. We study the large scale behavior of the resulting quasi-posterior distributions as the dimension of the parameter space grows, and we establish several convergence results. In large-scale applications where computational speed is important, variational approximation
methods are often used to approximate posterior distributions. We show that the contraction behaviors of the quasi-posterior distributions can be exploited to provide theoretical guarantees for their variational approximations. We illustrate the theory with several examples. Finally we develop a quasi-likelihood based algorithm for estimation of Ising/Potts models that incorporates inbuilt mechanism for parallel computation. We illustrate the usability of the method by analyzing 16 Personality Factors data under the setup of Five-level Potts Model. The data analysis recovers known clusters of personality traits and also indicates plausible novel clusters.PhDStatisticsUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/163007/1/anwebha_1.pd
Limit theorems for some adaptive MCMC algorithms with subgeometric kernels (II
No full textAbstract. This paper deals with the ergodicity (convergence of the marginals) and the law of large numbers for adaptive MCMC algorithms built from transition kernels that are not necessarily geometrically ergodic. We develop a number of results that broaden significantly the class of adaptive MCMC algorithms for which rigorous analysis is now possible. As an example, we give a detailed analysis of the Adaptive Metropolis Algorithm of Haario et al. (2001) when the target distribution is sub-exponential in the tails. 1
- …
