1,720,966 research outputs found
An unsupervised tour through the hidden pathways of deep neural networks
No full textThe goal of this thesis is to improve our understanding of the internal mechanisms by which deep architectures build meaningful representations and are able to generalize.
We focus on the challenge of characterizing the semantic content of the hidden representation with unsupervised learning tools, partially developed by us and described in this thesis which allow harnessing the low-dimensional structure of the data. Indeed, real-world data are typically hosted in manifolds that can be topologically complex, but that are typically low-dimensional.
Chapter 2 introduces Gride, a method that allows estimating the intrinsic dimension of the data as an explicit function of the scale without performing any decimation of the data set. Our method is simple and computationally efficient since it relies only on the distances among data points.
In chapter 3 we study the evolution of the probability density across the hidden layers in some state-of-the-art deep neural networks. We find that the initial layers generate a unimodal probability density getting rid of any structure irrelevant to classification. In subsequent layers, density peaks arise in a hierarchical fashion that mirrors the semantic hierarchy of the concepts. This process leaves a footprint in the probability density of the output layer where the topography of the peaks allows reconstructing the semantic relationships of the categories.
In chapter 4 we then study the problem of generalization in deep neural networks: adding parameters to a network that interpolates its training data will typically improve its generalization performance, at odds with the classical bias-variance trade-off. We show that over-parametrized neural networks learn redundant representations instead of overfitting to spurious correlation and that redundant neurons appear only if the network is regularized and the training error is zero
Redundant representations help generalization in wide neural networks
Full text linkDeep neural networks (DNNs) defy the classical bias-variance trade-off:
adding parameters to a DNN that interpolates its training data will typically
improve its generalization performance. Explaining the mechanism behind this
``benign overfitting'' in deep networks remains an outstanding challenge. Here,
we study the last hidden layer representations of various state-of-the-art
convolutional neural networks and find that if the last hidden representation
is wide enough, its neurons tend to split into groups that carry identical
information, and differ from each other only by statistically independent
noise. The number of such groups increases linearly with the width of the
layer, but only if the width is above a critical value. We show that redundant
neurons appear only when the training process reaches interpolation and the
training error is zero
The generalized ratios intrinsic dimension estimator
Full text linkModern datasets are characterized by numerous features related by complex dependency structures. To deal with these data, dimensionality reduction techniques are essential. Many of these techniques rely on the concept of intrinsic dimension (id), a measure of the complexity of the dataset. However, the estimation of this quantity is not trivial: often, the id depends rather dramatically on the scale of the distances among data points. At short distances, the id can be grossly overestimated due to the presence of noise, becoming smaller and approximately scale-independent only at large distances. An immediate approach to examining the scale dependence consists in decimating the dataset, which unavoidably induces non-negligible statistical errors at large scale. This article introduces a novel statistical method, Gride, that allows estimating the id as an explicit function of the scale without performing any decimation. Our approach is based on rigorous distributional results that enable the quantification of uncertainty of the estimates. Moreover, our method is simple and computationally efficient since it relies only on the distances among data points. Through simulation studies, we show that Gride is asymptotically unbiased, provides comparable estimates to other state-of-the-art methods, and is more robust to short-scale noise than other likelihood-based approaches
How to choose the right transfer learning protocol? A qualitative analysis in a controlled set-up
Full text linkTransfer learning is a powerful technique that enables model training with limited amounts
of data, making it crucial in many data-scarce real-world applications. Typically, transfer
learning protocols require first to transfer all the feature-extractor layers of a network pretrained on a data-rich source task, and then to adapt only the task-specific readout layers
to a data-poor target task. This workflow is based on two main assumptions: first, the
feature maps of the pre-trained model are qualitatively similar to the ones that would have
been learned with enough data on the target task; second, the source representations of
the last hidden layers are always the most expressive. In this work, we demonstrate that
this is not always the case and that the largest performance gain may be achieved when
smaller portions of the pre-trained network are transferred. In particular, we perform a set
of numerical experiments in a controlled setting, showing how the optimal transfer depth
depends non-trivially on the amount of available training data and on the degree of sourcetarget task similarity, and it is often convenient to transfer only the first layers. We then
propose a strategy to detect the most promising source task among the available candidates.
This approach compares the internal representations of a network trained entirely from
scratch on the target task with those of the networks pre-trained on the potential source
tasks
Going Beyond Counting First Authors in Author Co-citation Analysis
Full text linkThe present study examines one of the fundamental aspects of author co-citation analysis (ACA) - the way co-citation
counts are defined. Co-citation counting provides the data on which all subsequent statistical analyses and mappings
are based, and we compare ACA results based on two different types of co-citation counting - the traditional type that
only counts the first one among a cited work's authors on the one hand and a non-traditional type that takes into
account the first 5 authors of a cited work on the other hand. Results indicate that the picture produced through this non-traditional author co-citation counting contains more coherent author groups and is therefore considerably clearer. However, this picture represents fewer specialties in the research field being studied than that produced through the traditional first-author co-citation counting when the same number of top-ranked authors is selected and analyzed. Reasons for these effects are discussed
Variations on the Author
Full text link“Variations on the Author” discusses two of Eduardo Coutinho’s recent films (Um Dia na Vida, from 2010, and Últimas Conversas, posthumously released in 2015) and their contribution to the general question of documentary authorship. The director’s filmography is characterized by a consistent yet self-effacing form of authorial self-inscription: Coutinho often features as an interviewer that rather than express opinions propels discourses; an interviewer that is good at listening. This mode of self-inscription characterizes him as an author who is not expressive but who is nonetheless markedly present on the screen. In Um Dia na Vida, however, Coutinho is completely absent form the image, while Últimas Conversas, on the contrary, includes a confessional prologue that moves the director from the margins to the center of his films. This article examines the ways in which these works stand out in the filmography of a director who offers new insights into the notion of cinematic authorship
Appropriate Similarity Measures for Author Cocitation Analysis
Full text linkWe provide a number of new insights into the methodological discussion about author cocitation analysis. We first argue that the use of the Pearson correlation for measuring the similarity between authors’ cocitation profiles is not very satisfactory. We then discuss what kind of similarity measures may be used as an alternative to the Pearson correlation. We consider three similarity measures in particular. One is the well-known cosine. The other two similarity measures have not been used before in the bibliometric literature. Finally, we show by means of an example that our findings have a high practical relevance.information science;Pearson correlation;cosine;similarity measure;author cocitation analysis
Dispelling the Myths Behind First-author Citation Counts
Full text linkWe conducted a full-scale evaluative citation analysis study of scholars in the XML research field to explore just how different from each other author rankings resulting from different citation counting methods actually are, and to demonstrate the capability of emerging data and tools on the Web in supporting more realistic citation counting methods. Our results contest some common arguments for the continued
use of first-author citation counts in the evaluation of scholars, such as high correlations between author rankings by first-author citation counts and other citation
counting methods, and high costs of using more realistic citation counting methods that are not well-supported by the ISI databases. It is argued that increasingly available digital full text research papers make it possible for citation analysis studies to go beyond what the ISI databases have directly supported and to employ more
sophisticated methods
- …
