625 research outputs found
State-Space Inference and Learning with Gaussian Processes
No full text18.10.13 KB. Ok to add author version to spiral, authors hold copyright.State-space inference and learning with Gaussian processes (GPs) is an unsolved problem. We propose a new, general methodology for inference and learning in nonlinear state-space models that are described probabilistically by non-parametric GP models. We apply the expectation maximization algorithm to iterate between inference in the latent state-space and learning the parameters of the underlying GP dynamics model. Copyright 2010 by the authors
Direct numerical simulation of turbulent Couette-Poiseuille flow with zero skin friction
No full textThe near-wall scaling of mean velocity U(y) is addressed for the case of zero skin friction on one wall of a fully turbulent channel flow. The present DNS results can be added to the evidence in support of the conjecture that U is proportional to √yw in the region just above the wall at which the mean shear dU/dy = 0
Forward and Reverse Gradient-Based Hyperparameter Optimization
No full textWe study two procedures (reverse-mode and
forward-mode) for computing the gradient of the
validation error with respect to the hyperparameters
of any iterative learning algorithm such as
stochastic gradient descent. These procedures
mirror two methods of computing gradients for
recurrent neural networks and have different
trade-offs in terms of running time and space requirements. Our formulation of the reverse-mode
procedure is linked to previous work by Maclaurin
et al. (2015) but does not require reversible
dynamics. The forward-mode procedure is suitable
for real-time hyperparameter updates, which
may significantly speed up hyperparameter optimization
on large datasets. We present experiments
on data cleaning and on learning task interactions.
We also present one large-scale experiment
where the use of previous gradient-based
methods would be prohibitive
Practical Gauss-Newton Optimisation for Deep Learning
No full textWe present an efficient block-diagonal approximation to the Gauss-Newton matrix for feedforward neural networks. Our resulting algorithm is competitive against state-of-the-art first-order optimisation methods, with sometimes significant improvement in optimisation performance. Unlike first-order methods, for which hyperparameter tuning of the optimisation parameters is often a laborious process, our approach can provide good performance even when used with default settings. A side result of our work is that for piecewise linear transfer functions, the network objective function can have no differentiable local maxima, which may partially explain why such transfer functions facilitate effective optimisation
Bayesian learning via stochastic gradient langevin dynamics
No full textIn this paper we propose a new framework for learning from large scale datasets based on iterative learning from small mini-batches. By adding the right amount of noise to a standard stochastic gradient optimization algorithm we show that the iterates will converge to samples from the true posterior distribution as we anneal the stepsize. This seamless transition between optimization and Bayesian posterior sampling provides an inbuilt protection against overfitting. We also propose a practical method for Monte Carlo estimates of posterior statistics which monitors a "sampling threshold" and collects samples after it has been surpassed. We apply the method to three models: a mixture of Gaussians, logistic regression and ICA with natural gradients. Copyright 2011 by the author(s)/owner(s)
A fast and simple algorithm for training neural probabilistic language models
No full textIn spite of their superior performance, neural probabilistic language models (NPLMs) remain far less widely used than n-gram models due to their notoriously long training times, which are measured in weeks even for moderately-sized datasets. Training NPLMs is computationally expensive because they are explicitly normalized, which leads to having to consider all words in the vocabulary when computing the log-likelihood gradients. We propose a fast and simple algorithm for training NPLMs based on noise-contrastive estimation, a newly introduced procedure for estimating unnormalized continuous distributions. We investigate the behaviour of the algorithm on the Penn Treebank corpus and show that it reduces the training times by more than an order of magnitude without affecting the quality of the resulting models. The algorithm is also more efficient and much more stable than importance sampling because it requires far fewer noise samples to perform well. We demonstrate the scalability of the proposed approach by training several neural language models on a 47M-word corpus with a 80K-word vocabulary, obtaining state-of-the-art results on the Microsoft Research Sentence Completion Challenge dataset. Copyright 2012 by the author(s)/owner(s)
The Mondrian Kernel
No full textWe introduce the Mondrian kernel, a fast approximation to the Laplace kernel. It is suitable for both batch and online learning, and admits a fast kernel-width-selection procedure as the random features can be re-used efficiently for all kernel widths. The features are constructed by sampling trees via a Mondrian process [Roy and Teh, 2009], and we highlight the connection to Mondrian forests [Lakshminarayanan et al., 2014], where trees are also sampled via a Mondrian process, but fit independently. This link provides a new insight into the relationship between kernel methods and random forests
A Hierarchical Bayesian Language Model based on Pitman-Yor Processes
No full textWe propose a new hierarchical Bayesian n-gram model of natural languages. Our model makes use of a generalization of the commonly used Dirichlet distributions called Pitman-Yor processes which produce power-law distributions more closely resembling those in natural languages. We show that an approximation to the hierarchical Pitman-Yor language model recovers the exact formulation of interpolated Kneser-Ney, one of the best smoothing methods for n-gram language models. Experiments verify that our model gives cross entropy results superior to interpolated Kneser-Ney and comparable to modified Kneser-Ney. © 2006 Association for Computational Linguistics
Combination Of Cfd And Csd Packages For Fluid-Structure Interaction
No full textIn this article the UDF script file in the Fluent software was rewritten as the "connecting file" for the Fluent and the ANSYS/ABAQUS in order that the joined file can be used to do aero-elastic computations. In this way the fluid field is computed by solving the Navier-Stokes equations and the structure movement is integrated by the dynamics directly. An analysis of the computed results shows that this coupled method designed for simulating aero-elastic systems is workable and can be used for the other fluid-structure interaction problems
- …
