1,721,036 research outputs found
An Analytical Theory of Curriculum Learning in Teacher-Student Networks
No full textNo abstract availabl
Analysing the dynamics of online learning in over-parameterised two-layer neural networks
No full textTilting the odds at the lottery: the interplay of overparameterisation and curricula in neural networks
Full text linkAn Analytical Theory of Curriculum Learning in Teacher-Student Networks
Full text linkIn humans and animals, curriculum learning -- presenting data in a curated
order - is critical to rapid learning and effective pedagogy. Yet in machine
learning, curricula are not widely used and empirically often yield only
moderate benefits. This stark difference in the importance of curriculum raises
a fundamental theoretical question: when and why does curriculum learning help?
In this work, we analyse a prototypical neural network model of curriculum
learning in the high-dimensional limit, employing statistical physics methods.
Curricula could in principle change both the learning speed and asymptotic
performance of a model. To study the former, we provide an exact description of
the online learning setting, confirming the long-standing experimental
observation that curricula can modestly speed up learning. To study the latter,
we derive performance in a batch learning setting, in which a network trains to
convergence in successive phases of learning on dataset slices of varying
difficulty. With standard training losses, curriculum does not provide
generalisation benefit, in line with empirical observations. However, we show
that by connecting different learning phases through simple Gaussian priors,
curriculum can yield a large improvement in test performance. Taken together,
our reduced analytical descriptions help reconcile apparently conflicting
empirical results and trace regimes where curriculum learning yields the
largest gains. More broadly, our results suggest that fully exploiting a
curriculum may require explicit changes to the loss function at curriculum
boundaries.Comment: Accepted to NeurIPS 202
Probing transfer learning with a model of synthetic correlated datasets
Full text linkTransfer learning can significantly improve the sample efficiency of neural
networks, by exploiting the relatedness between a data-scarce target task and a
data-abundant source task. Despite years of successful applications, transfer
learning practice often relies on ad-hoc solutions, while theoretical
understanding of these procedures is still limited. In the present work, we
re-think a solvable model of synthetic data as a framework for modeling
correlation between data-sets. This setup allows for an analytic
characterization of the generalization performance obtained when transferring
the learned feature map from the source to the target task. Focusing on the
problem of training two-layer networks in a binary classification setting, we
show that our model can capture a range of salient features of transfer
learning with real data. Moreover, by exploiting parametric control over the
correlation between the two data-sets, we systematically investigate under
which conditions the transfer of features is beneficial for generalization
Continual Learning in the Teacher-Student Setup: Impact of Task Similarity
Full text linkContinual learning—the ability to learn many tasks in sequence—is critical for artificial learning systems. Yet standard training methods for deep networks often suffer from catastrophic forgetting, where learning new tasks erases knowledge of the earlier tasks. While catastrophic forgetting labels the problem, the theoretical reasons for interference between tasks remain unclear. Here, we attempt to narrow this gap between theory and practice by studying continual learning in the teacher-student setup. We extend previous analytical work on two-layer networks in the teacher-student setup to multiple teachers. Using each teacher to represent a different task, we investigate how the relationship between teachers affects the amount of forgetting and transfer exhibited by the student when the task switches. In line with recent work, we find that when tasks depend on similar features, intermediate task similarity leads to greatest forgetting. However, feature similarity is only one way in which tasks may be related. The teacher-student approach allows us to disentangle task similarity at the level of \textbackslashemphreadouts (hidden-to-output weights) as well as \textbackslashemphfeatures (input-to-hidden weights). We find a complex interplay between both types of similarity, initial transfer/forgetting rates, maximum transfer/forgetting, and the long-time (post-switch) amount of transfer/forgetting. Together, these results help illuminate the diverse factors contributing to catastrophic forgetting
Going Beyond Counting First Authors in Author Co-citation Analysis
Full text linkThe present study examines one of the fundamental aspects of author co-citation analysis (ACA) - the way co-citation
counts are defined. Co-citation counting provides the data on which all subsequent statistical analyses and mappings
are based, and we compare ACA results based on two different types of co-citation counting - the traditional type that
only counts the first one among a cited work's authors on the one hand and a non-traditional type that takes into
account the first 5 authors of a cited work on the other hand. Results indicate that the picture produced through this non-traditional author co-citation counting contains more coherent author groups and is therefore considerably clearer. However, this picture represents fewer specialties in the research field being studied than that produced through the traditional first-author co-citation counting when the same number of top-ranked authors is selected and analyzed. Reasons for these effects are discussed
Variations on the Author
Full text link“Variations on the Author” discusses two of Eduardo Coutinho’s recent films (Um Dia na Vida, from 2010, and Últimas Conversas, posthumously released in 2015) and their contribution to the general question of documentary authorship. The director’s filmography is characterized by a consistent yet self-effacing form of authorial self-inscription: Coutinho often features as an interviewer that rather than express opinions propels discourses; an interviewer that is good at listening. This mode of self-inscription characterizes him as an author who is not expressive but who is nonetheless markedly present on the screen. In Um Dia na Vida, however, Coutinho is completely absent form the image, while Últimas Conversas, on the contrary, includes a confessional prologue that moves the director from the margins to the center of his films. This article examines the ways in which these works stand out in the filmography of a director who offers new insights into the notion of cinematic authorship
Appropriate Similarity Measures for Author Cocitation Analysis
Full text linkWe provide a number of new insights into the methodological discussion about author cocitation analysis. We first argue that the use of the Pearson correlation for measuring the similarity between authors’ cocitation profiles is not very satisfactory. We then discuss what kind of similarity measures may be used as an alternative to the Pearson correlation. We consider three similarity measures in particular. One is the well-known cosine. The other two similarity measures have not been used before in the bibliometric literature. Finally, we show by means of an example that our findings have a high practical relevance.information science;Pearson correlation;cosine;similarity measure;author cocitation analysis
- …
