1,720,964 research outputs found
Deep Stable neural networks: large-width asymptotics and convergence rates
Full text linkIn modern deep learning, there is a recent and growing literature on the interplay between large-width asymptotic
properties of deep Gaussian neural networks (NNs), i.e. deep NNs with Gaussian-distributed weights, and Gaussian stochastic processes (SPs). Motivated by empirical analyses that show the potential of replacing Gaussian
distributions with Stable distributions for the NN’s weights, in this paper we present a rigorous analysis of the
large-width asymptotic behaviour of (fully connected) feed-forward deep Stable NNs, i.e. deep NNs with Stabledistributed
weights. We show that as the width goes to infinity jointly over the NN’s layers, i.e. the “joint growth”
setting, a rescaled deep Stable NN converges weakly to a Stable SP whose distribution is characterized recursively
through the NN’s layers. Because of the non-triangular structure of the NN, this is a non-standard asymptotic
problem, to which we propose an inductive approach of independent interest. Then, we establish sup-norm convergence rates of the rescaled deep Stable NN to the Stable SP, under the “joint growth” and a “sequential growth” of the width over the NN’s layers. Such a result provides the difference between the “joint growth” and the “sequential growth” settings, showing that the former leads to a slower rate than the latter, depending on the depth of the layer and the number of inputs of the NN. Our work extends some recent results on infinitely wide limits for deep Gaussian NNs to the more general deep Stable NNs, providing the first result on convergence rates in the “joint growth” setting
Learning-augmented count-min sketches via Bayesian nonparametrics
No full textThe count-min sketch (CMS) is a time and memory efficient randomized data
structure that provides estimates of tokens' frequencies in a data stream of
tokens, i.e. point queries, based on random hashed data. A learning-augmented
version of the CMS, referred to as CMS-DP, has been proposed by Cai,
Mitzenmacher and Adams (\textit{NeurIPS} 2018), and it relies on Bayesian
nonparametric (BNP) modeling of the data stream of tokens via a Dirichlet
process (DP) prior, with estimates of a point query being obtained as suitable
mean functionals of the posterior distribution of the point query, given the
hashed data. While the CMS-DP has proved to improve on some aspects of CMS, it
has the major drawback of arising from a ``constructive" proof that builds upon
arguments tailored to the DP prior, namely arguments that are not usable for
other nonparametric priors. In this paper, we present a ``Bayesian" proof of
the CMS-DP that has the main advantage of building upon arguments that are
usable, in principle, within a broad class of nonparametric priors arising from
normalized completely random measures. This result leads to develop a novel
learning-augmented CMS under power-law data streams, referred to as CMS-PYP,
which relies on BNP modeling of the data stream of tokens via a Pitman-Yor
process (PYP) prior. Under this more general framework, we apply the arguments
of the ``Bayesian" proof of the CMS-DP, suitably adapted to the PYP prior, in
order to compute the posterior distribution of a point query, given the hashed
data. Applications to synthetic data and real textual data show that the
CMS-PYP outperforms the CMS and the CMS-DP in estimating low-frequency tokens,
which are known to be of critical interest in textual data, and it is
competitive with respect to a variation of the CMS designed for low-frequency
tokens. An extension of our BNP approach to more general queries is also
discussed.Comment: 47 page
Large-width asymptotics and training dynamics of alpha-Stable Re{LU} neural networks
No full textLarge-width asymptotic properties of neural networks (NNs) with Gaussian distributed weights have been extensively investigated in the literature, with major results characterizing their large-width asymptotic behavior in terms of Gaussian processes and their large-width training dynamics in terms of the neural tangent kernel (NTK). In this paper, we study large-width asymptotics and training dynamics of α-Stable ReLU-NNs, namely NNs with ReLU activation function and α-Stable distributed weights, with α ∈ (0, 2). For α ∈ (0, 2], α-Stable distributions form a broad class of heavy tails distributions, with the special case α = 2 corresponding to the Gaussian distribution. Firstly, we show that if the NN’s width goes to infinity, then a rescaled α-Stable ReLU-NN converges weakly (in distribution) to an α-Stable process, which generalizes the Gaussian process. As a difference with respect to the Gaussian setting, our result shows that the activation function affects the scaling of the α-Stable NN; more precisely, in order to achieve the infinite-width α-Stable process, the ReLU activation requires an additional logarithmic term in the scaling with respect to sub-linear activations. Secondly, we characterize the large-width training dynamics of α-Stable ReLU-NNs in terms an infinite-width random kernel, which is referred to as the α-Stable NTK, and we show that the gradient descent achieves zero training error at linear rate, for a sufficiently large width, with high probability. Differently from the NTK arising in the Gaussian setting, the α-Stable NTK is a random kernel; more precisely, the randomness of the α-Stable ReLU-NN at initialization does not vanish in the large-width training dynamics
Large-width functional asymptotics for deep Gaussian neural networks
Full text linkIn this paper, we consider fully connected feed-forward deep neural networks where weights and biases are independent and identically distributed according to Gaussian distributions. Extending previous results (Matthews et al., 2018a;b; Yang, 2019) we adopt a function-space perspective, i.e. we look at neural networks as infinite-dimensional random elements on the input space . Under suitable assumptions on the activation function we show that: i) a network defines a continuous Gaussian process on the input space ; ii) a network with re-scaled weights converges weakly to a continuous Gaussian process in the large-width limit; iii) the limiting Gaussian process has almost surely locally -Hölder continuous paths, for . Our results contribute to recent theoretical studies on the interplay between infinitely wide deep neural networks and Gaussian processes by establishing weak convergence in function-space with respect to a stronger metric
Going Beyond Counting First Authors in Author Co-citation Analysis
Full text linkThe present study examines one of the fundamental aspects of author co-citation analysis (ACA) - the way co-citation
counts are defined. Co-citation counting provides the data on which all subsequent statistical analyses and mappings
are based, and we compare ACA results based on two different types of co-citation counting - the traditional type that
only counts the first one among a cited work's authors on the one hand and a non-traditional type that takes into
account the first 5 authors of a cited work on the other hand. Results indicate that the picture produced through this non-traditional author co-citation counting contains more coherent author groups and is therefore considerably clearer. However, this picture represents fewer specialties in the research field being studied than that produced through the traditional first-author co-citation counting when the same number of top-ranked authors is selected and analyzed. Reasons for these effects are discussed
Variations on the Author
Full text link“Variations on the Author” discusses two of Eduardo Coutinho’s recent films (Um Dia na Vida, from 2010, and Últimas Conversas, posthumously released in 2015) and their contribution to the general question of documentary authorship. The director’s filmography is characterized by a consistent yet self-effacing form of authorial self-inscription: Coutinho often features as an interviewer that rather than express opinions propels discourses; an interviewer that is good at listening. This mode of self-inscription characterizes him as an author who is not expressive but who is nonetheless markedly present on the screen. In Um Dia na Vida, however, Coutinho is completely absent form the image, while Últimas Conversas, on the contrary, includes a confessional prologue that moves the director from the margins to the center of his films. This article examines the ways in which these works stand out in the filmography of a director who offers new insights into the notion of cinematic authorship
Appropriate Similarity Measures for Author Cocitation Analysis
Full text linkWe provide a number of new insights into the methodological discussion about author cocitation analysis. We first argue that the use of the Pearson correlation for measuring the similarity between authors’ cocitation profiles is not very satisfactory. We then discuss what kind of similarity measures may be used as an alternative to the Pearson correlation. We consider three similarity measures in particular. One is the well-known cosine. The other two similarity measures have not been used before in the bibliometric literature. Finally, we show by means of an example that our findings have a high practical relevance.information science;Pearson correlation;cosine;similarity measure;author cocitation analysis
Dispelling the Myths Behind First-author Citation Counts
Full text linkWe conducted a full-scale evaluative citation analysis study of scholars in the XML research field to explore just how different from each other author rankings resulting from different citation counting methods actually are, and to demonstrate the capability of emerging data and tools on the Web in supporting more realistic citation counting methods. Our results contest some common arguments for the continued
use of first-author citation counts in the evaluation of scholars, such as high correlations between author rankings by first-author citation counts and other citation
counting methods, and high costs of using more realistic citation counting methods that are not well-supported by the ISI databases. It is argued that increasingly available digital full text research papers make it possible for citation analysis studies to go beyond what the ISI databases have directly supported and to employ more
sophisticated methods
- …
