5 research outputs found
Science of Deep Learning: From Initialization to Emergent Structures
No full textAs artificial intelligence (AI) systems grow increasingly powerful and permeate every aspect of our lives, their impact on both individuals and society is an urgent concern. Questions of safety and robustness in AI stem largely from our limited understanding of deep learning. Research in this domain has traditionally followed two parallel paths: an empirical approach that prioritizes practical advancements and a theoretical approach that seeks a mathematical understanding from first principles. Despite notable progress, a significant gap remains between deep learning practice and its theoretical underpinnings. This dissertation advocates for a phenomenological approach to understanding AI systems -- one that integrates empirical observations with theoretical model-building. This methodology has been instrumental in the physical sciences, and it holds similar promise for advancing the science of deep learning. Over two broad parts, this work demonstrates the effectiveness of this approach in characterizing model architectures and their emergent capabilities.
In the first part, we explore how signal propagation analysis in large-N limits can inform the design and initialization of model architectures. We develop a diagnostic observable that distinguishes between ordered and chaotic behaviors in neural networks, guiding optimal parameter initialization for training. Our analysis establishes the theoretical soundness of this observable in simple networks and confirms its empirical utility in state-of-the-art architectures. The findings reveal an architecture design paradigm that eliminates the need for careful initialization, shedding light on widely used heuristic practices. Additionally, we introduce an algorithm that automates initialization across diverse model architectures, enhancing their trainability.
In the second part, we highlight the importance of the systems identification approach for characterizing AI systems. We explore several stylized setups where model capabilities emerge as a function of compute, data quantity, and data diversity. Using arithmetic and cryptographic tasks as examples, we demonstrate that emergent abilities such as grokking and in-context learning arise alongside the formation of interpretable structures within the model’s parameters, hidden representations, and outputs. Through targeted experiments, we identify these structures using (i) black-box probing, which examines model responses to characteristic inputs, and (ii) open-box analysis, which leverages curated task-specific observables and metrics to study internal model states.
This dissertation promotes a paradigm for understanding deep learning that complements both heuristic-driven and hypothesis-driven approaches. By integrating experimental methodologies and analytical tools from established scientific disciplines, this framework has the potential to steer the field toward safer, more robust, and more efficient AI systems
Critical Initialization of Wide and Deep Neural Networks through Partial Jacobians: General Theory and Applications
No full textDeep neural networks are notorious for defying theoretical treatment.
However, when the number of parameters in each layer tends to infinity, the
network function is a Gaussian process (GP) and quantitatively predictive
description is possible. Gaussian approximation allows one to formulate
criteria for selecting hyperparameters, such as variances of weights and
biases, as well as the learning rate. These criteria rely on the notion of
criticality defined for deep neural networks. In this work we describe a new
practical way to diagnose criticality. We introduce \emph{partial Jacobians} of
a network, defined as derivatives of preactivations in layer with respect
to preactivations in layer . We derive recurrence relations for the
norms of partial Jacobians and utilize these relations to analyze criticality
of deep fully connected neural networks with LayerNorm and/or residual
connections. We derive and implement a simple and cheap numerical test that
allows one to select optimal initialization for a broad class of deep neural
networks; containing fully connected, convolutional and normalization layers.
Using these tools we show quantitatively that proper stacking of the LayerNorm
(applied to preactivations) and residual connections leads to an architecture
that is critical for any initialization. Finally, we apply our methods to
analyze ResNet and MLP-Mixer architectures; demonstrating the
everywhere-critical regime.Comment: Accepted (spotlight) at NeurIPS2023. Additional ResNet results. 42
pages, 12 figure
Grokking Modular Polynomials
No full textNeural networks readily learn a subset of the modular arithmetic tasks, while
failing to generalize on the rest. This limitation remains unmoved by the
choice of architecture and training strategies. On the other hand, an
analytical solution for the weights of Multi-layer Perceptron (MLP) networks
that generalize on the modular addition task is known in the literature. In
this work, we (i) extend the class of analytical solutions to include modular
multiplication as well as modular addition with many terms. Additionally, we
show that real networks trained on these datasets learn similar solutions upon
generalization (grokking). (ii) We combine these "expert" solutions to
construct networks that generalize on arbitrary modular polynomials. (iii) We
hypothesize a classification of modular polynomials into learnable and
non-learnable via neural networks training; and provide experimental evidence
supporting our claims.Comment: 7+4 pages, 3 figures, 2 table
Energy Aware Scheduling and Routing of Periodic Lightpath Demands in Optical Grid Networks
No full textAbstractOptical grid networks provide an ideal infrastructure to support large-scale data intensive applications and interconnection of data centers. The power consumption of communications equipment for such networks has been increasing steadily over the past decade and energy efficient routing schemes and traffic models can be utilized to reduce the energy consumption. In many applications it is possible to select the destination node from a set of possible destinations, which have the required computing/storage resources. This is known as anycasting. We propose a novel formulation that exploits knowledge of demand holding times and the flexibility of anycast routing to optimally schedule demands (in time) and route them in order to minimize overall network energy consumption. Our simulation results demonstrate that the proposed approach can lead to significant reductions in energy consumption, compared to traditional routing schemes
Budhan Stories S1E5: The village of Dead
No full textEpisode 5 of Season 1 contains a 15 minutes long play that was performed by Budhan Theatre actors during the first wave of Corona. They performed by maintaining social distance in one dark room and they adapted famous writer Dharmveer Bharti's book "Murdo Ka Gaanv' in the Corona context. The play talks about the conditions of Corona patients and the labourers who walked thousands of kilometers. The episode contains interviews of real life victims of Corona.Directed (Author) by: Budhan Theatre Team. Participants: Dakxin Chhara, Atish Indrekar, Ruchika Kodekar, Chetna Rathod, Kushal Batunge, Keyur Bajrange, Anish Garange, Siddharth Garange, Murdo Ka gaanv, Dharmveer Bharti, Mitthuben, Bharat Tamaychi, Darshil Tamaychi, Akshay Khanna, Alice TilcheSupporting materials include poster, subtitles, short clips and stills. </p
