14 research outputs found
Duality for Neural Networks through Reproducing Kernel Banach Spaces
No full textReproducing Kernel Hilbert spaces (RKHS) have been a very successful tool in various areas of machine learning. Recently, Barron spaces have been used to prove bounds on the generalisation error for neural networks. Unfortunately, Barron spaces cannot be understood in terms of RKHS due to the strong nonlinear coupling of the weights. We show that this can be solved by using the more general Reproducing Kernel Banach spaces (RKBS). This class of integral RKBS can be understood as an infinite union of RKHS spaces. As the RKBS is not a Hilbert space, it is not its own dual space. However, we show that its dual space is again an RKBS where the roles of the data and parameters are interchanged, forming an adjoint pair of RKBSs including a reproducing property in the dual space. This allows us to construct the saddle point problem for neural networks, which can be used in the whole field of primal-dual optimisation
Sparsifying dimensionality reduction of PDE solution data with Bregman learning
No full textClassical model reduction techniques project the governing equations onto a linear subspace of the original state space. More recent data-driven techniques use neural networks to enable nonlinear projections. While those often enable stronger compression, they may have redundant parameters and lead to suboptimal latent dimensionality. To overcome these issues, we propose a multistep algorithm that induces sparsity in the encoder-decoder networks for effective reduction in the number of parameters and additional compression of the latent space. This algorithm starts with sparsely initializing a network and training it using linearized Bregman iterations. These iterations have been very successful in computer vision and compressed sensing tasks, but have not yet been used for reduced-order modeling. After the training, we further compress the latent space dimensionality by using a form of proper orthogonal decomposition. Last, we use a bias propagation technique to change the induced sparsity into an effective reduction of parameters. We apply this algorithm to three representative PDE models: 1D diffusion, 1D advection, and 2D reaction-diffusion. Compared to conventional training methods like Adam, the proposed method achieves similar accuracy with 30\% fewer parameters and a significantly smaller latent space
Evoking Imagination: Exploring the role of imagination in the perception of objects
No full textThe research in this thesis is done to discover what elements play a role in evoking imagination.A framework is constituted with the elements familiarity and instructions that are believed to play a role in evoking imagination. The elements are discussed through a literature research and the exploration of objects fitting in the framework. Eventually via a design experiment the influence of familiarity and instructions on imagination is tested.The outcome of the design experiment exposed that familiarity of objects is evoking imagination in their understanding of their purpose. And the ‘script’, instructions 'inscribed' in objects is providing or limiting space for multiple interpretations. Creating a space for possibilities.The understanding in how to evoke imagination is helpful to interaction design, because it can challenge our conceptions of how to challenge the user in their interaction with object and the constitution of meaning.Design for Interactio
Learning a Sparse Representation of Barron Functions with the Inverse Scale Space Flow
Get PDFThis paper presents a method for finding a sparse representation of Barron
functions. Specifically, given an function , the inverse scale space
flow is used to find a sparse measure minimising the loss between
the Barron function associated to the measure and the function . The
convergence properties of this method are analysed in an ideal setting and in
the cases of measurement noise and sampling bias. In an ideal setting the
objective decreases strictly monotone in time to a minimizer with
, and in the case of measurement noise or sampling bias the
optimum is achieved up to a multiplicative or additive constant. This
convergence is preserved on discretization of the parameter space, and the
minimizers on increasingly fine discretizations converge to the optimum on the
full parameter space.Comment: 30 pages, 0 figure
Deep Networks are Reproducing Kernel Chains
Get PDFIdentifying an appropriate function space for deep neural networks remains a key open question. While shallow neural networks are naturally associated with Reproducing Kernel Banach Spaces (RKBS), deep networks present unique challenges. In this work, we extend RKBS to chain RKBS (cRKBS), a new framework that composes kernels rather than functions, preserving the desirable properties of RKBS. We prove that any deep neural network function is a neural cRKBS function, and conversely, any neural cRKBS function defined on a finite dataset corresponds to a deep neural network. This approach provides a sparse solution to the empirical risk minimization problem, requiring no more than neurons per layer, where is the number of data points
Sparsifying dimensionality reduction of PDE solution data with Bregman learning
Get PDFClassical model reduction techniques project the governing equations onto a linear subspace of the original state space. More recent data-driven techniques use neural networks to enable nonlinear projections. Whilst those often enable stronger compression, they may have redundant parameters and lead to suboptimal latent dimensionality. To overcome these, we propose a multistep algorithm that induces sparsity in the encoder-decoder networks for effective reduction in the number of parameters and additional compression of the latent space. This algorithm starts with sparsely initialized a network and training it using linearized Bregman iterations. These iterations have been very successful in computer vision and compressed sensing tasks, but have not yet been used for reduced-order modelling. After the training, we further compress the latent space dimensionality by using a form of proper orthogonal decomposition. Last, we use a bias propagation technique to change the induced sparsity into an effective reduction of parameters. We apply this algorithm to three representative PDE models: 1D diffusion, 1D advection, and 2D reaction-diffusion. Compared to conventional training methods like Adam, the proposed method achieves similar accuracy with 30% less parameters and a significantly smaller latent space
Duality for Neural Networks through Reproducing Kernel Banach Spaces
Get PDFReproducing Kernel Hilbert spaces (RKHS) have been a very successful tool in
various areas of machine learning. Recently, Barron spaces have been used to
prove bounds on the generalisation error for neural networks. Unfortunately,
Barron spaces cannot be understood in terms of RKHS due to the strong nonlinear
coupling of the weights. This can be solved by using the more general
Reproducing Kernel Banach spaces (RKBS). We show that these Barron spaces
belong to a class of integral RKBS. This class can also be understood as an
infinite union of RKHS spaces. Furthermore, we show that the dual space of such
RKBSs, is again an RKBS where the roles of the data and parameters are
interchanged, forming an adjoint pair of RKBSs including a reproducing kernel.
This allows us to construct the saddle point problem for neural networks, which
can be used in the whole field of primal-dual optimisation
Duality for neural networks through Reproducing Kernel Banach Spaces
Get PDFReproducing Kernel Hilbert spaces (RKHS) have been a very successful tool in various areas of machine learning. Recently, Barron spaces have been used to prove bounds on the generalisation error for neural networks. Unfortunately, Barron spaces cannot be understood in terms of RKHS due to the strong nonlinear coupling of the weights. This can be solved by using the more general Reproducing Kernel Banach spaces (RKBS). We show that these Barron spaces belong to a class of integral RKBS. This class can also be understood as an infinite union of RKHS spaces. Furthermore, we show that the dual space of such RKBSs, is again an RKBS where the roles of the data and parameters are interchanged, forming an adjoint pair of RKBSs including a reproducing kernel. This allows us to construct the saddle point problem for neural networks, which can be used in the whole field of primal-dual optimisation.</p
Learning a Sparse Representation of Barron Functions with the Inverse Scale Space Flow
Get PDFThis paper presents a method for finding a sparse representation of Barron functions. Specifically, given an function , the inverse scale space flow is used to find a sparse measure minimising the loss between the Barron function associated to the measure and the function . The convergence properties of this method are analysed in an ideal setting and in the cases of measurement noise and sampling bias. In an ideal setting the objective decreases strictly monotone in time to a minimizer with , and in the case of measurement noise or sampling bias the optimum is achieved up to a multiplicative or additive constant. This convergence is preserved on discretization of the parameter space, and the minimizers on increasingly fine discretizations converge to the optimum on the full parameter space
Embeddings between Barron spaces with higher order activation functions
Get PDFThe approximation properties of infinitely wide shallow neural networks heavily depend on the choice of the activation function. To understand this influence, we study embeddings between Barron spaces with different activation functions. These embeddings are proven by providing push-forward maps on the measures used to represent functions . An activation function of particular interest is the rectified power unit () given by . For many commonly used activation functions, the well-known Taylor remainder theorem can be used to construct a push-forward map, which allows us to prove the embedding of the associated Barron space into a Barron space with a as activation function. Moreover, the Barron spaces associated with the have a hierarchical structure similar to the Sobolev spaces
