Different Learning Strategies Used During Pavlovian Conditioning

Post by Shireen Parimoo

What's the science?

In Pavlovian conditioning, people form associations between a neutral stimulus (e.g. a bell) and an upcoming unconditioned stimulus (e.g. food). The neutral stimulus later becomes the conditioned stimulus because it elicits the same response as the unconditioned stimulus. People can learn these associations using a value-based or an uncertainty-based strategy. In value-based learning, learning occurs based on the difference between the expected reward and the actual reward received, which is the reward prediction error. In uncertainty-based learning, people learn the probability that a conditioned stimulus will elicit a specific unconditioned stimulus, which generates the state prediction error. There are individual differences in whether people pay more attention to the conditioned stimulus (sign-trackers) or the unconditioned stimulus (goal-trackers). The neural basis of these learning strategies is not yet well understood. This week in Nature Human Behavior, Schad and colleagues used eye-tracking and functional magnetic resonance imaging (fMRI) techniques to investigate the neural substrates of learning strategies used by sign-trackers and goal-trackers.

How did they do it?

Participants were 129 male adults who completed a Pavlovian conditioning task in the fMRI scanner while their eye movements were recorded. They learned associations between visual-auditory cues that predicted monetary reward (appetitive conditioned stimulus; $1, $2), no reward (neutral conditioned stimulus: $0), or loss (aversive conditioned stimulus; -$1, -$2). The authors computed a gaze index to categorize participants as sign-trackers or goal-trackers. The gaze index is the difference between the proportion of fixations made to the unconditioned stimulus and the proportion of fixations made to the conditioned stimulus. A value of 0 indicates that participants made an equal proportion of fixations to both conditioned and unconditioned stimuli, whereas positive and negative values indicate that they made more fixations to the conditioned and the unconditioned stimulus, respectively. To identify sign-trackers, the authors examined the relationship between gaze index and the value of the conditioned stimulus. The top third of the participants who looked more frequently at the conditioned stimulus predicting monetary rewards than at the conditioned stimulus predicting losses were deemed to be sign-trackers. A similar analysis was conducted with the value of the unconditioned stimulus to identify goal-trackers. Eye movement behavior during the conditioning task, including pupil dilation and the number of fixations, was compared across the two groups for the different stimuli.

The authors used computational modeling to determine whether the eye movement patterns of sign-trackers and goal-trackers during the conditioning task reflected value-based or uncertainty-based learning strategies. Value-based learning was assessed in a reinforcement learning model that computes a reward-prediction error value. On the other hand, uncertainty-based learning was assessed in a model that produced a state prediction error value. Finally, the authors examined the neural substrates of the different learning strategies. They used a reinforcement learning model to compute reward prediction error signals in reward-processing regions like the nucleus accumbens in response to the stimuli. Uncertainty-based learning was investigated by computing the state prediction error signal at the onset of the unconditioned stimulus in regions associated with the state prediction error effect, such as the intraparietal sulcus and the lateral prefrontal cortex. The reward prediction and state prediction error effects in the brain were compared between sign-trackers and goal-trackers.

What did they find?

Sign-trackers made more fixations to the appetitive conditioned stimulus associated with a monetary reward than to the aversive conditioned stimulus associated with monetary loss, which is in line with value-based learning. Conversely, goal-trackers made more fixations to the appetitive unconditioned stimulus more than the aversive unconditioned stimulus, but over time, they looked away from the conditioned stimuli and more at the unconditioned stimuli, which is in line with uncertainty-based learning. Pupil dilation in response to both conditioned and unconditioned stimuli also differed between sign-trackers and goal-trackers. Pupil size decreased over the course of learning in goal-trackers but did not change in response to the appetitive and aversive conditioned stimuli. Among sign-trackers, there was no change in pupil size over time, but the pupils dilated in response to appetitive conditioned stimuli compared to the aversive conditioned stimuli. Computational modeling indicated that the value-based model captured pupillary changes for sign-trackers, whereas the uncertainty-based model best explained the pupil dilation in goal-trackers. Overall, these results suggest that eye movement behavior tracks the value of the stimulus among sign-trackers and the upcoming expectation state among goal-trackers.

Distinct patterns of brain activity were associated with learning strategies used by sign-trackers and goal-trackers. In reward-processing brain regions such as the nucleus accumbens, ventromedial prefrontal cortex, and amygdala, the value-based model explained more variance in brain activity for the sign-trackers than for the goal-trackers. In contrast, the uncertainty-based model better explained activity in the intraparietal sulcus of the goal-trackers than the sign-trackers. In sum, the eye-tracking and neural data indicate that sign-trackers used value-based learning strategies while goal-trackers relied more on an uncertainty-based strategy.

What's the impact?

This study is the first to demonstrate the distinct behavioral and neural profiles of learning in sign-trackers, who primarily use value-based learning strategies, and goal-trackers, who rely on uncertainty-based learning. By providing a deeper understanding of the different learning systems in humans, these findings have important implications for the treatment of disorders that involve aberrant reward learning, like addiction.

Schad et al. Dissociating neural learning signals in human sign- and goal-trackers. Nature Human Behavior (2019). Access the original scientific publication here.

Two Neural Features Related to Information Encoding and Behavior

Post by Stephanie Williams

What's the science?

To understand how neurons encode information related to the external world, such as information from the environment, we need to understand which statistical features of individual neurons contain that information. In the past, research groups have suggested the mean firing rate of individual neurons or groups of neurons may represent information about stimuli. Other research has looked at the amount of noise that populations of neurons share (correlated noise), however, the specific statistical properties of neurons related to the information encoding are not well understood. This week in The Journal of Neuroscience, Noguiera and colleagues identify neural features that explain most of the variance in information encoding and behavior. The authors specifically address two questions 1) what features are related to information encoding 2) do those features affect behavioral performance? 

How did they do it?                             

This study involved both experimental data collection and theoretical modelling. For the experimental arm of the study, four monkeys were trained to perform three different tasks. Two of the three tasks were direction discrimination tasks (one coarse discrimination and one fine discrimination), and the third was a spatial attention task, in which two of the monkeys had to detect a change in the orientation of some lines displayed in a circle (i.e. a Gabor patch). The authors recorded neural activity while the monkeys performed a given task from two brain regions: the middle temporal area (MT) and area 8a in the lateral prefrontal cortex. Performance on the tasks was quantified as the number of correct reports of motion for the direction tasks, and as the mean reaction time for the attention task. To test which neural features were related to 1) information encoding and 2) behavioral performance, the authors isolated features of interest by iterating through each extracted feature, and changing the values of one set of features while holding all of the other features constant. By using a statistical technique, called bootstrapping, they could select the bootstrap iterations that produced feature values that were in a narrow range around the median for that particular feature. They used these bootstrapped samples to generate the “fluctuations” in the feature that they were changing during their iterations (they call this method “conditioned bootstrapping”). The authors then trained a binary classifier to predict which task the monkeys were performing and the specific behavior performed in the task (i.e. which Gabor patches were attended to or left vs. right motion stimulus).

For the theoretical arm of the study, the authors defined decoding performance, mathematically simulated experimental data, and tested decoding performance on the simulated data. To define decoding performance, the authors first derived a mathematical expression that described the theoretical optimal performance of a linear classifier. They defined two terms representing the two categories of features of interest 1) the population signal feature, which is a measure of how the overall modulation of activity of the measured neuronal population changes as a function of the stimulus condition, and 2) the projected precision feature which is related to the trial-by-trial variability. To simulate population activity, the authors built a neural population activity model with a large ensemble than was recorded experimentally (N = 1000 model neurons), with each neuron’s activity modeled as a function of stimuli (2 stimuli) of different strengths (3 different strengths). They also incorporated a mathematical term that represented noise correlations (corresponding changes in trial-to-trial variability) between neurons. To model behavioral performance, the authors used an optimal linear classifier to make predictions from simulated neural activity. They then compared the performance of their theoretical decoder with the performance of the decoder trained on their experimental data. 

What did they find?

The authors found that two of the features they examined were important features that affected how much information was encoded, and were the strongest predictors of behavioral performance. They found that changing both the 1) population signal feature and the 2) projected precision feature (one at a time, while holding all other features constant), significantly affected the amount of encoded information and also predicted changes in behavioral performance.

The first feature related to population tuning was specifically a length metric that joined the mean population responses across different experimental conditions. The second feature, related to the amount of trial-by-trial variability was calculated as the inverse of the population covariability projected onto the direction of the population signal. Importantly, the authors did not find that other features (such as global activity and mean pairwise correlations), which had previously been suggested by other research, were related to the amount of information encoding when they controlled for the two features they identified. However, it is worth noting that changing the global activity and correlation features did change the amount of information encoded in population activity when the two features that authors identified were not controlled for.  

What's the impact?

The authors show for the first time that two features, population signal and projected precision, modulate the amount of information encoded by finite neuronal populations and predict changes in behavior. They also show that two other features did not modulate the amount of encoded information or behavioral performance. These findings shed light on the specific properties of neurons involved in encoding information.

Nogueira et al. The effects of population tuning and trial-by-trial availability on information encoding and behavior. J. Neurosci (2019). Access the original scientific publication here.

The Role of Evolution on Brain Connectivity in Schizophrenia

Post by Elisa Guma

What's the science?

Schizophrenia is a debilitating psychiatric disorder characterized by hallucinations, delusions, and cognitive dysfunction, often associated with impaired brain connectivity. The genetic origin, human-specific traits, and similar prevalence observed across societies (1% of the population is affected globally) have led to the idea that human brain evolution may have played a role in the development of the disorder. This week in Brain, van den Heuvel and colleagues aim to investigate schizophrenia-related changes in brain connectivity in the context of evolutionary changes in the human brain by comparing humans and chimpanzees.

How did they do it?

To measure brain connectivity, the authors studied diffusion-weighted imaging data (sensitive to the integrity of the brain’s white matter tracts and myelin levels) from individuals with schizophrenia and gender-matched healthy control subjects, as well as from chimpanzees. Connectome maps were created by 1) subdividing the cerebral cortex into 114 subdivisions based on a commonly used brain atlas, and then 2) calculating cortico-cortical connectivity between every subdivision or brain region and every other brain region. The measure of cortico-cortical connectivity was based on whether each pair of brain regions were interconnected by common diffusion-weighted streamlines (measuring common white matter tracts). The authors compared the connectome maps of individuals with schizophrenia to those of controls to examine differences in brain connectivity. In order to confirm whether the results they observed were schizophrenia specific, they performed the same analysis for a variety of other neuropsychiatric and neurological disorders including major depressive disorder, bipolar disorder, obsessive-compulsive disorder, autism spectrum disorder, a behavioural variant of frontotemporal dementia, and mild cognitive impairment. The authors also performed their analysis in a separate cohort of schizophrenia and control subjects to ensure their results were not biased by the data acquisition of their original sample.

To determine the human-specific connections in the brain, the authors compared the connectome maps between humans and chimpanzees. Connections that were present in at least 60% of the human subjects and 0% of the chimpanzees were classified as human-specific connections (and vice-versa for chimpanzee-specific connections). Human-chimpanzee-shared connections were classified as those present in at least 60% of humans and 60% of chimpanzees. To investigate the evolutionary nature of dysconnectivity (i.e. reduced connectivity) in schizophrenia, the authors compared human-specific connection patterns to brain dysconnectivity patterns due to schizophrenia. The authors extended their cross-species comparison to include rhesus macaques, a more distantly related primate species, to further investigate the evolutionary specialization of the human-specific connections they identified. Finally, the authors validated their findings by repeating the analysis using a different brain atlas that is designed to map homologous regions across humans and chimpanzees and subdivides cortical areas into 38 rather than 114 regions.

What did they find?

First, the authors show that many brain regions involved in higher-order processing had greater dysconnectivity in patients with schizophrenia. These findings were consistent with previous studies. Next, the authors found a 94% overlap between the human and chimpanzee cortico-cortical networks, with 3.5% human-specific connections, and 1.1% chimpanzee-specific connections. Many of the brain regions identified to be human-specific are involved in language networks and important for semantic comprehension. The authors also identified human-specific connections in regions implicated in cognitive control, social cognition, emotional processing, and bonding behaviour. Interestingly, these behaviours are often impaired in individuals with schizophrenia. In fact, the authors found high similarity between patterns of schizophrenia dysconnectivity and human-specific connections in comparison to connections shared by both humans and chimpanzees. This held true when the authors extended their cross-species comparison to include macaques, and when the data were analyzed with the alternative brain atlas. Furthermore, the human-specific connections were not associated with brain dysconnectivity patterns for any other disorders. There were trend level associations with bipolar disorder, which shares a genetic background with schizophrenia, as well as some symptoms like psychosis.

What's the impact?

These findings provide compelling evidence for the hypothesis that human brain evolution may have played a role in the development of schizophrenia. The authors show that human-specific features of cortical connectivity are associated with patterns of cortical dysconnectivity in individuals with schizophrenia, suggesting that evolutionary pressure to develop higher-order functions may have rendered the brain vulnerable to dysfunction. In future work, it could be interesting to see how these patterns extend to other measures of brain function.

Van den Heuvel et al. Evolutionary modifications in human brain connectivity associated with schizophrenia. Brain (2019). Access the original scientific publication here.