1,720,995 research outputs found
Computing Complex Visual Features with Retinal Spike Times
Full text linkNeurons in sensory systems can represent information not only by their firing rate, but also by the precise timing of individual spikes. For example, certain retinal ganglion cells, first identified in the salamander, encode the spatial structure of a new image by their first-spike latencies. Here we explore how this temporal code can be used by downstream neural circuits for computing complex features of the image that are not available from the signals of individual ganglion cells. To this end, we feed the experimentally observed spike trains from a population of retinal ganglion cells to an integrate-and-fire model of post-synaptic integration. The synaptic weights of this integration are tuned according to the recently introduced tempotron learning rule. We find that this model neuron can perform complex visual detection tasks in a single synaptic stage that would require multiple stages for neurons operating instead on neural spike counts. Furthermore, the model computes rapidly, using only a single spike per afferent, and can signal its decision in turn by just a single spike. Extending these analyses to large ensembles of simulated retinal signals, we show that the model can detect the orientation of a visual pattern independent of its phase, an operation thought to be one of the primitives in early visual processing. We analyze how these computations work and compare the performance of this model to other schemes for reading out spike-timing information. These results demonstrate that the retina formats spatial information into temporal spike sequences in a way that favors computation in the time domain. Moreover, complex image analysis can be achieved already by a simple integrate-and-fire model neuron, emphasizing the power and plausibility of rapid neural computing with spike times.Molecular and Cellular BiologyVersion of Recor
Recommended from our members
Statistical mechanics of Bayesian inference and learning in neural networks
No full textThis thesis collects a few of my essays towards understanding representation learning and generalization in neural networks. I focus on the model setting of Bayesian learning and inference, where the problem of deep learning is naturally viewed through the lens of statistical mechanics. First, I consider properties of freshly-initialized deep networks, with all parameters drawn according to Gaussian priors. I provide exact solutions for the marginal prior predictive of networks with isotropic priors and linear or rectified-linear activation functions. I then study the effect of introducing structure to the priors of linear networks from the perspective of random matrix theory. Turning to memorization, I consider how the choice of nonlinear activation function affects the storage capacity of treelike neural networks. Then, we come at last to representation learning. I study the structure of learned representations in Bayesian neural networks at large but finite width, which are amenable to perturbative treatment. I then show how the ability of these networks to generalize when presented with unseen data is affected by representational flexibility, through precise comparison to models with frozen, random representations. In the final portion of this thesis, I bring a geometric perspective to bear on the structure of neural network representations. I first consider how the demand of fast inference shapes optimal representations in recurrent networks. Then, I consider the geometry of representations in deep object classification networks from a Riemannian perspective. In total, this thesis begins to elucidate the structure and function of optimally distributed neural codes in artificial neural networks
Recommended from our members
Scaling and Renormalization in Statistical Learning
No full textThis thesis develops a theoretical framework for understanding the scaling properties of information processing systems in the regime of large data, large model size, and large computational resources. The goal is to develop an understanding of the impressive performance that deep neural networks have exhibited.
The first part of this thesis examines models linear in their parameters but nonlinear in their inputs. This includes linear regression, kernel regression, and random feature models. Utilizing random matrix theory and free probability, I provide precise characterizations of their training dynamics, generalization capabilities, and out-of-distribution performance, alongside a detailed analysis of sources of variance. A variety of scaling laws observed in state-of-the-art large language and vision models are already present in this simple setting.
The second part of this thesis focuses on representation learning. Leveraging insights from models linear in inputs but nonlinear in parameters, I present a theory of early-stage representation learning where a network with small weight initialization can learn features without altering the loss. This phenomenon, termed silent alignment, is empirically validated across various architectures and datasets. The idea of starting at small initialization leads naturally to the "maximal update parameterization", μP, that allows for feature learning at infinite width. I present empirical studies showing that practical networks can approach their theoretical infinite-width feature learning limits. Finally, I consider down-scaling the output of a neural network by a fixed constant. When this constant is small, the network behaves as a linear model in parameters; when large, it induces silent alignment. I present theoretical and empirical results of the influence of this hyperparameter on feature learning, performance, and dynamics
Going Beyond Counting First Authors in Author Co-citation Analysis
Full text linkThe present study examines one of the fundamental aspects of author co-citation analysis (ACA) - the way co-citation
counts are defined. Co-citation counting provides the data on which all subsequent statistical analyses and mappings
are based, and we compare ACA results based on two different types of co-citation counting - the traditional type that
only counts the first one among a cited work's authors on the one hand and a non-traditional type that takes into
account the first 5 authors of a cited work on the other hand. Results indicate that the picture produced through this non-traditional author co-citation counting contains more coherent author groups and is therefore considerably clearer. However, this picture represents fewer specialties in the research field being studied than that produced through the traditional first-author co-citation counting when the same number of top-ranked authors is selected and analyzed. Reasons for these effects are discussed
Variations on the Author
Full text link“Variations on the Author” discusses two of Eduardo Coutinho’s recent films (Um Dia na Vida, from 2010, and Últimas Conversas, posthumously released in 2015) and their contribution to the general question of documentary authorship. The director’s filmography is characterized by a consistent yet self-effacing form of authorial self-inscription: Coutinho often features as an interviewer that rather than express opinions propels discourses; an interviewer that is good at listening. This mode of self-inscription characterizes him as an author who is not expressive but who is nonetheless markedly present on the screen. In Um Dia na Vida, however, Coutinho is completely absent form the image, while Últimas Conversas, on the contrary, includes a confessional prologue that moves the director from the margins to the center of his films. This article examines the ways in which these works stand out in the filmography of a director who offers new insights into the notion of cinematic authorship
Recommended from our members
Statistical Mechanics of Neural Processing of Object Manifolds
No full textInvariant object recognition is one of the most fundamental cognitive tasks performed by the brain. In the neural state space, different objects with stimulus variabilities are represented as different manifolds. In this geometrical perspective, object recognition becomes the problem of linearly separating different object manifolds. In feedforward visual hierarchy, it has been suggested that the object manifold representations are reformatted across the layers, to become more linearly separable. Thus, a complete theory of perception requires characterizing the ability of linear readout networks to classify object manifolds from variable neural responses.
A theoretical understanding of the perceptron of isolated points was pioneered by Elizabeth Gardner who formulated it as a statistical mechanics problem and analyzed it using replica theory. In this thesis, we generalize the statistical mechanical analysis and establish a theory of linear classification of manifolds synthesizing statistical and geometric properties of high dimensional signals.
First, we study the theory of linear classification of simple spherical manifolds, such as line segments, L2 balls, or L1 balls. We provide analytical formula for classification capacity of balls, as a function of dimension, radius, and margin. We also find that the notion of support vectors needs to be generalized, and identify different support configurations of the manifolds, which has implications in generalization error.
Next, we present a Maximum Margin Manifold Machine (M4), an efficient iterative algorithm that can find a maximum margin linear binary classifier for manifolds with an uncountable set of training samples per each manifold. We provide a convergence proof with a polynomial bound on the convergence time. We further generalize M4 for non-separable manifolds with slack variables. We report that the number of training examples required to achieve the same generalization error is much smaller for M4, compared with traditional support vector machines.
Next, we generalize our theory further to linear classification of random general manifolds. We start with classification capacity of random ellipsoids, and generalize to classification capacity of general smooth and non-smooth manifolds. We identify that the capacity of a manifold is determined that effective radius, R_M, and effective dimension, D_M.
Finally, we show extensions to directions relevant for applications to real data. We have extended our general manifold classification theory to incorporate correlated manifolds, mixtures of manifold geometries, sparse labels and nonlinear classifications. Then, we analyze how object-based manifolds reformat in a conventional deep network (GoogLeNet). We find that the deep network indeed changes the manifolds in the direction that the capacity is increased.
This thesis lays the groundwork for a computational theory of neuronal processing of objects, providing quantitative measures for linear separability of object manifolds. We hope that our theory will provide new insights into the computational principles underlying processing of sensory representations in the brain. As manifold representations of the sensory world are ubiquitous in both biological and artificial neural systems, exciting future work lies ahead.Engineering and Applied Sciences - Applied Physic
- …
