59 research outputs found
Stability Analysis of Neural Language Models and Brains
No full textIn this paper, we define and apply representational stability analysis (ReStA), an intuitive way of analyzing neural language models. ReStA is a variant of the popular representational similarity analysis (RSA) in cognitive neuroscience. While RSA can be used to compare representations in models, model components, and human brains, ReStA compares instances of the same model, while systematically varying single model parameter. Using ReStA, we study four recent and successful neural language models, and evaluate how sensitive their internal representations are to the amount of prior context. Using RSA, we perform a systematic study of how similar the representational spaces in the first and second (or higher) layers of these models are to each other and to patterns of activation in the human brain. Our results reveal surprisingly strong differences between language models, and give insights into where the deep linguistic processing, that integrates information over multiple sentences, is happening in these models. The combination of ReStA and RSA on models and brains allows us to start addressing the important question of what kind of linguistic processes we can hope to observe in fMRI brain imaging data. In particular, our results suggest that the data on story reading from Wehbe et al./ (2014) contains a signal of shallow linguistic processing, but show no evidence on the more interesting deep linguistic processing.<br/
Quantifiers in a Multimodal World: Hallucinating Vision with Language and Sound
No full textInspired by the literature on multisensory integration, we develop a computational model to ground quantifiers in perception. The model learns to pick, out of nine quantifiers (‘few’, ‘many’, ‘all’, etc.), the one that is more likely to describe the percent of animals in a visual-auditory input containing both animals and artifacts. We show that relying on concurrent sensory inputs increases model performance on the quantification task. Moreover, we evaluate the model in a situation in which only the auditory modality is given, while the visual one is ‘hallucinanted’ either from the auditory input itself or from a linguistic caption describing the quantity of entities in the auditory input. This way, the model exploits prior associations between modalities. We show that the model profits from the prior knowledge and outperforms the auditory-only setting
In spoken word recognition, the future predicts the past
No full textSpeech is an inherently noisy and ambiguous signal. To fluently derive meaning, a listener must integrate contextual information to guide interpretations of the sensory input. Although many studies have demonstrated the influence of prior context on speech perception, the neural mechanisms supporting the integration of subsequent context remain unknown. Using MEG to record from human auditory cortex, we analyzed responses to spoken words with a varyingly ambiguous onset phoneme, the identity of which is later disambiguated at the lexical uniqueness point. Fifty participants (both male and female) were recruited across two MEG experiments. Our findings suggest that primary auditory cortex is sensitive to phonological ambiguity very early during processing at just 50 ms after onset. Subphonemic detail is preserved in auditory cortex over long timescales and re-evoked at subsequent phoneme positions. Commitments to phonological categories occur in parallel, resolving on the shorter timescale of ∼450 ms. These findings provide evidence that future input determines the perception of earlier speech sounds by maintaining sensory features until they can be integrated with top-down lexical information
ChapGTP, ILLC’s Attempt at Raising a BabyLM: Improving Data Efficiency by Automatic Task Formation
No full textWe present the submission of the ILLC at the University of Amsterdam to the BabyLM challenge (Warstadt et al., 2023), in the strict-small track. Our final model, ChapGTP, is a masked language model that was trained for 200 epochs, aided by a novel data augmentation technique called Automatic Task Formation. We discuss in detail the performance of this model on the three evaluation suites: BLiMP, (Super)GLUE, and MSGS. Furthermore, we present a wide range of methods that were ultimately not included in the model, but may serve as inspiration for training LMs in low-resource settings
Quantized current steps due to the a.c. coherent quantum phase-slip effect
No full text| openaire: EC/H2020/862660/EU//QUANTUM E-LEAPS Funding Information: This work was supported by European Union’s Horizon 2020 Research and Innovation Programme under grant agreement no. 862660/QUANTUM E-LEAPS and Engineering and Physical Sciences Research Council (EPSRC) grant no. EP/T004088/1. Publisher Copyright: © 2022, The Author(s), under exclusive licence to Springer Nature Limited.The a.c. Josephson effect predicted in 19621 and observed experimentally in 19632 as quantized ‘voltage steps’ (the Shapiro steps) from photon-assisted tunnelling of Cooper pairs is among the most fundamental phenomena of quantum mechanics and is vital for metrological quantum voltage standards. The physically dual effect, the a.c. coherent quantum phase slip (CQPS), photon-assisted tunnelling of magnetic fluxes through a superconducting nanowire, is envisaged to reveal itself as quantized ‘current steps’3,4. The basic physical significance of the a.c. CQPS is also complemented by practical importance in future current standards, a missing element for closing the quantum metrology triangle5,6. In 2012, the CQPS was demonstrated as superposition of magnetic flux quanta in superconducting nanowires 7. However, the direct flat current steps in superconductors, the only unavailable basic effect of superconductivity to date, was unattainable due to lack of appropriate materials and challenges in circuit engineering. Here we report the direct observation of the dual Shapiro steps in a superconducting nanowire. The sharp steps are clear up to 26 GHz frequency with current values 8.3 nA and limited by the present set-up bandwidth. The current steps were theoretically predicted in small Josephson junctions 30 years ago5. However, unavoidable broadening in Josephson junctions prevents their direct experimental observation8,9. We solve this problem by placing a thin NbN nanowire in an inductive environment.Peer reviewe
Recommended from our members
Universal linguistic inductive biases via meta-learning
No full textHow do learners acquire languages from the limited data avail-able to them? This process must involve some inductivebiases—factors that affect how a learner generalizes—but it isunclear which inductive biases can explain observed patternsin language acquisition. To facilitate computational model-ing aimed at addressing this question, we introduce a frame-work for giving particular linguistic inductive biases to a neu-ral network model; such a model can then be used to em-pirically explore the effects of those inductive biases. Thisframework disentangles universal inductive biases, which areencoded in the initial values of a neural network’s param-eters, from non-universal factors, which the neural networkmust learn from data in a given language. The initial statethat encodes the inductive biases is found with meta-learning,a technique through which a model discovers how to acquirenew languages more easily via exposure to many possible lan-guages. By controlling the properties of the languages that areused during meta-learning, we can control the inductive biasesthat meta-learning imparts. We demonstrate this frameworkwith a case study based on syllable structure. First, we specifythe inductive biases that we intend to give our model, and thenwe translate those inductive biases into a space of languagesfrom which a model can meta-learn. Finally, using existinganalysis techniques, we verify that our approach has impartedthe linguistic inductive biases that it was intended to impart
- …
