42 research outputs found
Specific Pavlovian-instrumental transfer: relationship with instrumental reward probabilities
No full textAppetitive pavlovian-instrumental transfer: a review
No full textReward-related cues are an important part of our daily life as they often influence and guide our actions. This paper reviews one of the experimental paradigms used to study the effects of cues, the Pavlovian to Instrumental Transfer paradigm. In this paradigm, cues associated with rewards through Pavlovian conditioning alter motivation and choice of instrumental actions. The first transfer experiments date back to the 1940s, but only in the last decade has it been fully recognised that there are two types of transfer, specific and general. This paper presents a systematic review of both the neural substrates and the behavioral factors affecting both types of transfer. It also examines the recent application of the paradigm to study the effect of cues on human participants, both in normal and pathological conditions, and the interactions of transfer with drugs of abuse. Finally, the paper analyses the theoretical aspects of transfer to build an overall picture of the phenomenon, from early theories to recent hierarchical accounts
The three principles of action: a Pavlovian-instrumental transfer hypothesis
No full textPavlovian conditioned stimuli can influence instrumental responding, an effect called Pavlovian-instrumental transfer (PIT).During the last decade, PIT has been subdivided into two types: specific PIT and general PIT, each having its own neural substrates.Specific PIT happens when a conditioned stimulus (CS) associated with a reward enhances an instrumental response directed to the same reward.Under general PIT instead, the CS enhances a response directed to a different reward.While important progress has been made into identifying the neural substrates, the function of specific and general PIT and how they interact with instrumental responses, are still not clear.In the experimental paradigm that distinguishes specific and general PIT an effect of PIT inhibition has also been observed and is waiting for an explanation.Here we propose an hypothesis that links these three PIT effects (specific PIT, general PIT and PIT inhibition) to three aspects of action evaluation.These three aspects, which we call "principles of action" are: context, efficacy, and utility.In goal-directed behavior, an agent has to evaluate if the context is suitable to accomplish the goal, the efficacy of his action in getting the goal and the utility of the goal itself:we suggest that each of the three PIT effects is related to one of these aspects of action evaluation.In particular, we link specific PIT with the estimation of efficacy, general PIT with the evaluation of utility and PIT inhibition with the adequacy of context.We also provide a latent cause Bayesian computational model that exemplifies this hypothesis.This hypothesis and the model provide a new framework and new predictions to advance knowledge about PIT functioning and its role in animal adaptation
A Bayesian model for a Pavlovian-instrumental transfer hypothesis
No full textA Pavlovian conditioned stimulus (CS) associated with a reward can enhance an instrumental response directed to the same or other rewards. This effect is called Pavlovian-instrumental transfer (PIT). In recent years, lesion studies using rats have gained insight into its neural substrates dissociating between specific PIT (where CS and instrumental response share the same reward) and general PIT (where they do not) (Corbit and Balleine, 2005, 2011). Despite these advances, the functional differences between specific and general PIT and how Pavlovian cues interact with instrumental response are still not clear. Here we try to explain Pavlovian-instrumental transfer effects by using a latent causes Bayesian model. Previous work in the Pavlovian conditioning literature (Courville et al., 2005) suggests that during Pavlovian conditioning rats do not simply learn associations between two events (CS and reward); instead, they actually try to figure out the real hidden causes behind them by constructing a latent cause model. We expanded that view to include instrumental actions and so explain the interactions between Pavlovian conditioning and instrumental conditioning. Our model correctly reproduces both the presence of specific and general PIT and the absence of general PIT when the CS is associated to the reward of another instrumental action. By framing the PIT effects explanation in Bayesian terms, our model offers a new integrated view on their functional mechanisms and new testable predictions
Interplay of prefrontal cortex and amygdala during extinction of drug seeking
No full textExtinction of Pavlovian conditioning is a complex process that involves brain regions such as the medial prefrontal cortex (mPFC), the amygdala and the locus coeruleus. In particular, noradrenaline (NA) coming from the locus coeruleus has been recently shown to play a different role in two subregions of the mPFC, the prelimbic (PL) and the infralimbic (IL) regions. How these regions interact in conditioning and subsequent extinction is an open issue. We studied these processes using two approaches: computational modelling and NA manipulation in a conditioned place preference paradigm (CPP) in mice. In the computational model, NA in PL and IL causes inputs arriving to these regions to be amplified, thus allowing them to modulate learning processes in amygdala. The model reproduces results from studies involving depletion of NA from PL, IL, or both in CPP. In addition, we simulated new experiments of NA manipulations in mPFC, making predictions on the possible results. We searched the parameters of the model and tested the robustness of the predictions by performing a sensitivity analysis. We also present an empirical experiment where, in accord with the model, a double depletion of NA from both PL and IL in CPP with amphetamine impairs extinction. Overall the proposed model, supported by anatomical, physiological, and behavioural data, explains the differential role of NA in PL and IL and opens up the possibility to understand extinction mechanisms more in depth and hence to aid the development of treatments for disorders such as addiction
The relationship between specific Pavlovian instrumental transfer and instrumental reward probability
No full textGoal-directed behavior is influenced by environmental cues: in particular, cues associated with a reward can bias action choice toward actions directed to that same reward. This effect is studied experimentally as specific Pavlovian-instrumental transfer (specific PIT). We have investigated the hypothesis that cues associated to an outcome elicit specific PIT by rising the estimates of reward probability of actions associated to that same outcome. In other words, cues reduce the uncertainty on the efficacy of instrumental actions. We used a human PIT experimental paradigm to test the effects of two different instrumental contingencies: one group of participants had a 33% chance of being rewarded for each button press, while another had a 100% chance. The group trained with 33% reward probability showed a stronger PIT effect than the 100% group, in line with the hypothesis that Pavlovian cues linked to an outcome work by reducing the uncertainty of receiving it. The 100% group also showed a significant specific PIT effect, highlighting additional factors that could contribute to specific PIT beyond the instrumental training contingency. We hypothesize that the uncertainty about reward delivery due to testing in extinction might be one of these factors. These results add knowledge on how goal-directed behavior is influenced by the presence of environmental cues associated with a reward: such influence depends on the probability that we have to reach a reward, namely when there is less chance of getting a reward we are more influenced by cues associated with it, and vice versa
Integrating unsupervised and reinforcement learning in human categorical perception: A computational model
No full textCategorical perception identifies a tuning of human perceptual systems that can occur during the execution of a categorisation task. Despite the fact that experimental studies and computational models suggest that this tuning is influenced by task-independent effects (e.g., based on Hebbian and unsupervised learning, UL) and task-dependent effects (e.g., based on reward signals and reinforcement learning, RL), no model studies the UL/RL interaction during the emergence of categorical perception. Here we have investigated the effects of this interaction, proposing a system-level neuro-inspired computational architecture in which a perceptual component integrates UL and RL processes. The model has been tested with a categorisation task and the results show that a balanced mix of unsupervised and reinforcement learning leads to the emergence of a suitable categorical perception and the best performance in the task. Indeed, an excessive unsupervised learning contribution tends to not identify task-relevant features while an excessive reinforcement learning contribution tends to initially learn slowly and then to reach sub-optimal performance. These results are consistent with the experimental evidence regarding categorical activations of extrastriate cortices in healthy conditions. Finally, the results produced by the two extreme cases of our model can explain the existence of several factors that may lead to sensory alterations in autistic people
Pavlovian Instrumental transfer un modello computazionale bayesiano con cause latenti
No full textNo abstract availabl
Pavlovian-Instrumental transfer: computational models and function
No full textReward-related cues are an important part of our daily life as they often influence and guide our actions. This thesis focuses on one of the experimental paradigms used to study the effects of cues, the Pavlovian to Instrumental Transfer paradigm (PIT). In this paradigm, cues associated with rewards through Pavlovian conditioning alter motivation and choice of instrumental actions. During the last decade, the PIT effect - the influence of Pavlovian stimuli over instrumental actions - has been subdivided into two types: specifc PIT and general PIT, each having its own neural substrates. Specifc PIT happens when a conditioned stimulus (CS) associated with a reward enhances an instrumental response directed to the same reward. Under general PIT instead, the CS enhances a response directed to a different reward as well. While important progress has been made into identifying the neural substrates, the function of specifc and general PIT and how they interact with instrumental responses, are still not clear. In the experimental paradigm that distinguishes specifc and general PIT an effect of PIT inhibition has also been observed and is waiting for an explanation. In this thesis we propose an hypothesis that links these three PIT effects (specifc PIT, general PIT and PIT inhibition) to three aspects of action evaluation. These three aspects, which we call principles of action" are: context, efficacy, and utility. In goaldirected behaviour, an agent has to evaluate if the context is suitable to accomplish the goal, the efficacy of his action in getting the goal and the utility of the goal itself: we suggest that each of the three PIT effects is related to one of these aspects of action evaluation. In particular, we link specific PIT with the estimation of efficacy, general PIT with the evaluation of utility and PIT inhibition with the adequacy of context. We then provide a first computational model that exemplifies this hypothesis. The model is a Bayesian generative model with latent variables, based on a Bayesian understanding of conditioning that has been gaining grounds in the latest years. The underlying hypothesis is that animals learn hidden (latent) causes that jointly explain the co-occurrences of several observables (namely, sounds, levers, foods) { as opposed to learning simple associations between these events as more commonly assumed in the animal learning literature. In this scheme, PIT depends on Bayesian inference on the presence or absence of such hidden causes. We have then tested one part of our hypothesis and its predictions in a human behavioral experiment. In particular, we investigated the hypothesis that cues associated to an outcome elicit specific PIT by rising the estimates of reward probability of actions associated to that same outcome. In other words, cues reduce the uncertainty on the efficacy of instrumental actions. We used a human PIT experimental paradigm to test the effects of two different instrumental contingencies: one group of participants had a 33% chance of being rewarded for each button press, while another had a 100% chance. The group trained with 33% reward probability showed a stronger PIT effect than the 100% group, in line with the hypothesis that Pavlovian cues linked to an outcome work by reducing the uncertainty of receiving it. However, contrary to our prediction, the 100% group also showed a significant specific PIT effect, highlighting additional factors that could contribute to specifc PIT beyond the instrumental training contingency. In the last chapter, we developed a second Bayesian computational model on transfer, to account for the above experimental results. Compared to the previous model, this second model explicitly models Pavlovian and instrumental conditioning into two different components, arranged in a hierarchical fashion. We posited that the key link between the two components is the prediction of food availability by the Pavlovian process, which is then used by the instrumental process to determine which instrumental context is active and subsequently determine the best course of action. The model correctly reproduces the qualitative pattern of the behavioral experiment, albeit it is so far limited to specific transfer only
A generative spiking neural-network model of goal-directed behaviour and one-step planning
No full textInternational audienceIdea of the model, specification of the model and tests, implementation of the model, tests, data analysis, analysis of results, writing-up. ‡Idea of the model, specification of the model and tests, analysis of results, writing-up. ¤Specification of the model and tests, analysis of results, writing-up. * Abstract In mammals, goal-directed and planning processes support flexible behaviour usable to face new situations or changed conditions that cannot be tackled through more efficient but rigid habitual behaviours. Within the Bayesian modelling approach of brain and behaviour, probabilistic models have been proposed to perform planning as a probabilistic inference. Recently, some models have started to face the important challenge met by this approach: grounding such processes on the computations implemented by brain spiking networks. Here we propose a model of goal-directed behaviour that has a probabilistic interpretation and is centred on a recurrent spiking neural network representing the world model. The model, building on previous proposals on spiking neurons and plasticity rules having a probabilistic interpretation, presents these novelties at the system level: (a) the world model is learnt in parallel with its use for planning, and an arbitration mechanism decides when to exploit the world-model knowledge with planning, or to explore, on the basis of an entropy-based confidence on the world model knowledge; (b) the world model is a hidden Markov model (HMM) able to simulate sequences of states and actions, thus planning selects actions through the same neural generative process used to predict states; (c) the world model learns the hidden causes of observations, and their temporal dependencies, through a biologically plausible unsupervised learning mechanism. The model is tested with a visuomotor learning task and validated by comparing its behaviour with the performance and reaction times of human participants solving the same task. The model represents a further step towards the construction of an autonomous architecture bridging goal-directed behaviour as probabilistic inference to brain-like computations. Author summary Goal-directed behaviour relies on brain processes supporting planning of actions based on the prediction of their consequences before performing them in the environment. An important computational modelling approach of these processes sees the brain as a probabilistic machine implementing goal-directed processes relying on probability distributions and operations on them. An important challenge for this approach is to explain how these distributions and operations might be grounded on the brain spiking doi: bioRxiv preprint neurons and learning processes. Here we propose a hypothesis of how this might happen by presenting a computational model of goal-directed processes based on artificial spiking neural networks. The model presents three main novelties. First, it can plan even while it is still learning the consequences of actions by deciding if planning or exploring the environment based on how confident it is on its predictions. Second, it is able to 'think' alternative possible actions, and their consequences, by relying on the low-level stochasticity of neurons. Third, it can learn to anticipate the consequences of actions in an autonomous fashion based on experience. Overall, the model represents a novel hypothesis on how goal-directed behaviour might rely on the stochastic spiking processes and plasticity mechanisms of the brain neurons
