1,721,101 research outputs found
Neurally plausible mechanisms for learning selective and invariant representations
Get PDFAbstract
Coding for visual stimuli in the ventral stream is known to be invariant to object identity preserving nuisance transformations. Indeed, much recent theoretical and experimental work suggests that the main challenge for the visual cortex is to build up such nuisance invariant representations. Recently, artificial convolutional networks have succeeded in both learning such invariant properties and, surprisingly, predicting cortical responses in macaque and mouse visual cortex with unprecedented accuracy. However, some of the key ingredients that enable such success—supervised learning and the backpropagation algorithm—are neurally implausible. This makes it difficult to relate advances in understanding convolutional networks to the brain. In contrast, many of the existing neurally plausible theories of invariant representations in the brain involve unsupervised learning, and have been strongly tied to specific plasticity rules. To close this gap, we study an instantiation of simple-complex cell model and show, for a broad class of unsupervised learning rules (including Hebbian learning), that we can learn object representations that are invariant to nuisance transformations belonging to a finite orthogonal group. These findings may have implications for developing neurally plausible theories and models of how the visual cortex or artificial neural networks build selectivity for discriminating objects and invariance to real-world nuisance transformations
Understanding Robustness and Generalization of Artificial Neural Networks Through Fourier Masks
Get PDFDespite the enormous success of artificial neural networks (ANNs) in many disciplines, the characterization of their computations and the origin of key properties such as generalization and robustness remain open questions. Recent literature suggests that robust networks with good generalization properties tend to be biased toward processing low frequencies in images. To explore the frequency bias hypothesis further, we develop an algorithm that allows us to learn modulatory masks highlighting the essential input frequencies needed for preserving a trained network's performance. We achieve this by imposing invariance in the loss with respect to such modulations in the input frequencies. We first use our method to test the low-frequency preference hypothesis of adversarially trained or data-augmented networks. Our results suggest that adversarially robust networks indeed exhibit a low-frequency bias but we find this bias is also dependent on directions in frequency space. However, this is not necessarily true for other types of data augmentation. Our results also indicate that the essential frequencies in question are effectively the ones used to achieve generalization in the first place. Surprisingly, images seen through these modulatory masks are not recognizable and resemble texture-like patterns
Translational Symmetry in Convolutions With Localized Kernels Causes an Implicit Bias Toward High Frequency Adversarial Examples
Get PDFAdversarial attacks are still a significant challenge for neural networks. Recent efforts have shown that adversarial perturbations typically contain high-frequency features, but the root cause of this phenomenon remains unknown. Inspired by theoretical work on linear convolutional models, we hypothesize that translational symmetry in convolutional operations together with localized kernels implicitly bias the learning of high-frequency features, and that this is one of the main causes of high frequency adversarial examples. To test this hypothesis, we analyzed the impact of different choices of linear and non-linear architectures on the implicit bias of the learned features and adversarial perturbations, in spatial and frequency domains. We find that, independently of the training dataset, convolutional operations have higher frequency adversarial attacks compared to other architectural parameterizations, and that this phenomenon is exacerbated with stronger locality of the kernel (kernel size) end depth of the model. The explanation for the kernel size dependence involves the Fourier Uncertainty Principle: a spatially-limited filter (local kernel in the space domain) cannot also be frequency-limited (local in the frequency domain). Using larger convolution kernel sizes or avoiding convolutions (e.g., by using Vision Transformers or MLP-style architectures) significantly reduces this high-frequency bias. Looking forward, our work strongly suggests that understanding and controlling the implicit bias of architectures will be essential for achieving adversarial robustness
Going Beyond Counting First Authors in Author Co-citation Analysis
Get PDFThe present study examines one of the fundamental aspects of author co-citation analysis (ACA) - the way co-citation
counts are defined. Co-citation counting provides the data on which all subsequent statistical analyses and mappings
are based, and we compare ACA results based on two different types of co-citation counting - the traditional type that
only counts the first one among a cited work's authors on the one hand and a non-traditional type that takes into
account the first 5 authors of a cited work on the other hand. Results indicate that the picture produced through this non-traditional author co-citation counting contains more coherent author groups and is therefore considerably clearer. However, this picture represents fewer specialties in the research field being studied than that produced through the traditional first-author co-citation counting when the same number of top-ranked authors is selected and analyzed. Reasons for these effects are discussed
User-Centered Policy Interventions for Future Cities
No full textRising societal demands are a direct result of the exponential rise in urbanization and technological development. Addressing these needs in a way that is sustainable, inclusive, and user-friendly is a problem for policymakers. Citizens' participation at every stage of a policy's development is critical to the user-centricity concept's success in policymaking. By involving the citizens in this collaborative effort, we can make sure that policy interventions are specific to each community's needs, which will increase their acceptability, relevance, and longevity. In the long run, this can make cities better places to live by making them more resilient, citizen-centric, and focused on the needs of ambitious inhabitants. Overall, this paper presents a range of user-centric policy initiatives that might help policymakers better understand the varied requirements, preferences, and urban experiences of city dwellers
Interpreting Deep Neural Networks and Beyond: Visualization, Learning Dynamics, and Disentanglement
No full textDespite their great success, deep neural networks are always considered black boxes. In various domains such as self-driving cars and tumor diagnosis, it is crucial to know the reasoning behind a decision made by a neural network. In addition, a better understanding of current deep learning methods will inspire the development of more principled computational approaches with better robustness and interpretability. In this thesis, we focus on interpreting and improving deep neural networks from the following three perspectives: visualization, learning dynamics, and disentanglement.
First, the demand for human interpretable explanations for model decisions has driven the development of visualization techniques. In particular, a class of backpropagation-based visualizations has attracted much attention recently, including Saliency Map, DeconvNet, and Guided Backprop (GBP). However, we find that there exist some perplexing behaviors: DeconvNet and GBP are more human-interpretable but less class-sensitive than Saliency Map. Motivated by this, we develop a theory that shows that GBP and DeconvNet are essentially doing image reconstruction, which is unrelated to network decisions. This analysis, together with various experiments, implies that those human-interpretable visualizations do not always reveal the inner working mechanisms of deep neural networks.
Second, although generative adversarial networks (GANs) have been one of the most powerful deep generative models, they are notoriously difficult to train and the reasons underlying their (non-)convergence behaviors are still not completely understood. To this end, we conduct a non-asymptotic analysis of local convergence in GAN training dynamics by evaluating the eigenvalues of its Jacobian near the equilibrium. The analysis reveals that to ensure a good convergence rate, two factors should be avoided: (i) Phase Factor, i.e., the Jacobian has complex eigenvalues with a large imaginary-to-real ratio, and (ii) Conditioning Factor, i.e., the Jacobian is ill-conditioned. Thus, we propose a new regularization method called JARE that addresses both factors by construction.
Third, disentanglement learning aims to make representations in neural networks more disentangled and human interpretable. However, we find that current disentanglement methods have several limitations: 1) difficulty with high-resolution images, 2) neglecting the existence of a trade-off between learning disentangled representations and controllable generation, and 3) non-identifiability due to the unsupervised setting. To overcome these limitations, we propose new losses and network architectures based on StyleGAN [karras et al., 2019] for semi-supervised high-resolution disentanglement learning. Experimental results show that using very limited supervision significantly improves disentanglement quality and that the proposed method can generalize well to unseen images in the tasks of semantic fine-grained image editing.
Looking forward, with more efforts and meaningful interactions in these three directions, we believe that we can improve our understanding of both the successes and failures of current deep learning methods, and develop more robust, interpretable and flexible AI systems
Variations on the Author
Get PDF“Variations on the Author” discusses two of Eduardo Coutinho’s recent films (Um Dia na Vida, from 2010, and Últimas Conversas, posthumously released in 2015) and their contribution to the general question of documentary authorship. The director’s filmography is characterized by a consistent yet self-effacing form of authorial self-inscription: Coutinho often features as an interviewer that rather than express opinions propels discourses; an interviewer that is good at listening. This mode of self-inscription characterizes him as an author who is not expressive but who is nonetheless markedly present on the screen. In Um Dia na Vida, however, Coutinho is completely absent form the image, while Últimas Conversas, on the contrary, includes a confessional prologue that moves the director from the margins to the center of his films. This article examines the ways in which these works stand out in the filmography of a director who offers new insights into the notion of cinematic authorship
Appropriate Similarity Measures for Author Cocitation Analysis
Get PDFWe provide a number of new insights into the methodological discussion about author cocitation analysis. We first argue that the use of the Pearson correlation for measuring the similarity between authors’ cocitation profiles is not very satisfactory. We then discuss what kind of similarity measures may be used as an alternative to the Pearson correlation. We consider three similarity measures in particular. One is the well-known cosine. The other two similarity measures have not been used before in the bibliometric literature. Finally, we show by means of an example that our findings have a high practical relevance.information science;Pearson correlation;cosine;similarity measure;author cocitation analysis
- …
