Experience Replay is Associated with Efficient Non-local Learning

Post by Andrew Vo

What's the science?

When we make decisions, we often rely on previously learned relationships between actions and their outcomes. Although it is relatively easy to assign a value to an action when they are experienced close together in time and space (local), it becomes more challenging to do so when they are further apart (non-local). It has been hypothesized that non-local learning is achieved via a model-based approach such as ‘experience replay’, in which a learned map of the environment is used to link local rewards to non-local actions. Direct evidence for such a mechanism in humans has yet to be established, however. This week in Science, Liu and colleagues used a novel decision-making task and non-invasive brain imaging to test how experience replay helps achieve non-local learning.

How did they do it?

To test their hypothesis, the authors designed a decision-making task that separated local and non-local learning. On each trial, the participant was presented with one of three starting arms. Each arm contained two paths, between which the participant would choose. Each path was composed of a sequence of three stimuli followed by a reward outcome. Critically, the two possible outcomes that were reachable in each arm were also shared across all starting arms. In this way, participants could use the learned outcome in the current arm (local) to inform and update their choices when encountering the other two starting arms (non-local), in a model-based approach.

To measure neural replay, the authors used magnetoencephalography (MEG) to record fast whole-brain activity as participants performed the task. Replay was defined as the reactivation of a sequence at the time of reward receipt, which could occur in both forward (i.e., from the beginning of a path sequence to the eventual outcome) and backward (i.e., from the outcome at the end of a path sequence towards the beginning) directions. They also examined how neural replay was prioritized for non-local experiences based on their utility for future decisions. This utility was determined by the gain (i.e., how informative is this current reward for improving my choice at this arm) and the need (i.e., how frequently will this arm be visited in the future) of each experience. These two task features were manipulated by changing the reward and arm probabilities, respectively, across trials.

What did they find?

When participants encountered the same starting arm as before, they were more likely to favor the path that was previously rewarded, indicative of direct, model-free learning. This local learning was found to transfer to non-local experiences, as participants would favor the path leading to a previously rewarded outcome even when it was presented in a different starting arm. Using computational modeling, the authors found that values of non-local paths were updated to a similar extent as those of local paths, and learning rates were higher for non-local paths with greater priority (higher gain and need).

Examining MEG recordings, the authors observed two types of neural replay occurring after reward receipt: (1) a fast, forward replay that peaked at 30-ms lag, and (2) a slower, backward replay that peaked at 160-ms lag. The forward replay was associated with local actions and an increase in ripple frequency power, whereas the backward replay preferentially encoded non-local actions. Computational modeling revealed that only the backward replay was associated with efficient non-local learning. Similarly, only the backward replay was related to the utility (higher gain and need) of non-local experiences.

What's the impact?

In summary, this study revealed that backward replay serves as a neural mechanism for non-local learning and is prioritized based on utility for future decisions. The results here build upon model-based reinforcement learning theories largely tested in rodents and extend them to human behavior. They contribute to our understanding of how the brain might bridge the gap between direct and indirect experiences to guide our decisions.

Liu et al. Experience replay is associated with efficient nonlocal learning. Science (2021). Access the original scientific publication here.

How Neural Activity is Organized During Sleep

Post by Lani Cupo

What's the science?

During sleep, the hippocampus is relied upon to consolidate experiences into long-term memories by synchronizing subcortical and cortical activity through different patterns of neuronal firing. By examining patterns of brain activity during sleep, researchers can investigate how communication occurs between distant regions, coordinating wide-spread activity. This week in PNAS, Skelin and colleagues probe the synchronization of activity between the hippocampus, subcortical structures, and cortex during sleep in human participants, examining the role of the hippocampus in memory consolidation.

How did they do it?

In order to assess brain activity during sleep, data from electrodes implanted into the brains of twelve participants with epilepsy for the purpose of seizure evaluation was obtained. While the participants slept, the electrodes recorded local brain activity (intracranial electroencephalography, iEEG), in the hippocampus, and its target regions, including the amygdala, temporal lobe, and frontal cortex. Two precise patterns of activity were identified within the overnight recordings. These included hippocampal sharp-wave ripples (SWRs), which are characterized by large, fast waves of activity (80-150 Hz), and high frequency activity (HFA; 70-200 Hz) in the target regions. HFA was previously shown to reflect the spiking activity in the electrode vicinity.

First, the researchers paired time series from each electrode in the hippocampus that contained at least 100 SWRs overnight with target sites acquired outside of the hippocampus. Each target site was identified as HFA+ if the HFA level was modulated during SWR or HFA- if it was not. This allowed the researchers to assess whether SWRs positively or negatively modulated HFA in target regions. Second, the researchers expected that SWRs would interact with slow-wave activity (SWA) or sleep spindles in target regions to modulate HFA, a hypothesis that they investigated by calculating synchrony between hippocampal SWRs and regional SWA, before correlating synchrony with the strength of HFA modulation. Finally, the researchers predicted that correlations between HFA amplitude across targets would indicate that modules of brain regions were functionally connected.

What did they find?

The authors first found that the most common modulations that occur simultaneously with SWRs are positive-modulations ipsilateral (in the same hemisphere) to the hippocampal activity in the temporal and amygdala regions of interest. This implies that SWRs in the hippocampus may play a role in stimulating neuronal activity predominantly in the amygdala and temporal lobe, especially in the same hemisphere of the brain.

The researchers also found that there was a consistent relationship between hippocampal SWRs phase-locking to SWA or sleep spindles in subcortical and cortical structures and HFA modulation in the same structures. These findings imply that SWA/spindles may play an important role in the SWRs modulation of HFA, and are involved in consolidating new memories. Interestingly, SWR-SWA coupling is present bilaterally (in both hemispheres), while the SWR-spindle coupling is present only in the brain hemisphere ipsilateral to SWR origin.

Regarding their final hypothesis, the findings suggest that slow waves are synchronized across brain regions that are anatomically distinct, providing a possible mechanism for the functional association of distributed memory traces.

What's the impact?

This study found an interaction between hippocampal SWRs and subcortical/cortical slow waves plays a role in modulating HFA of specific regions—especially the amygdala and temporal lobe. Simply, the activity of hippocampal neurons during sleep acts in concert with distant populations of neurons to coordinate the consolidation of memories. The uncommon opportunity to study this human population with implanted electrodes lends deeper insight into how distributed memory traces are formed during sleep.

Skelin et al. Coupling between slow-waves and sharp-wave ripples organizes distributed neural activity during sleep in humans. PNAS (2021). Access the original scientific publication here.

Attention in the Age of Social Media

Post by Elisa Guma

The advent of the internet

The Internet is the most widespread and rapidly adopted technology in the history of humankind. With the advent of broadband Wi-Fi and smartphone technologies, we have constant access to the internet. This has rapidly changed the way we work, search for and access information, consume media and entertainment, and engage socially. Indeed, we currently live in a media-saturated world, using it not only for entertainment purposes such as listening to music or watching movies but also for communicating with peers. Connecting with family and friends across the globe can help people feel more connected in times of isolation, such as in the current global pandemic. However, access to this endless stream of communication and connection may be changing the way we think and absorb information and may also impact our mental health.

Attention and the brain

Attention is the behavioural and cognitive process by which we selectively concentrate on a discrete aspect of information while ignoring other information. Focusing our attention recruits brain regions such as the prefrontal and visual cortices, thalamic and midbrain nuclei. It can alternatively be thought of as an allocation of limited cognitive processing resources to a particular topic or task. The ability to achieve selective and sustained attention, free from distractions, is critical to our ability to complete tasks, learn new information, and engage socially with others. Once attention is engaged, we remain focused until some external environmental or internal state change occurs that triggers a shift. The constant flow of information and notifications the internet brings may interfere with our ability to maintain sustained concentration on other tasks. Social media is designed to be highly engaging in an attempt to keep us browsing for as long as possible. Furthermore, content that fails to gain our attention is quickly drowned out in a sea of incoming information, while information that does capture our attention is amplified or proliferated.

How does social media impact our attention?

One of the first studies investigating the effect of social media on attention found that heavy social media use may increase people’s susceptibility to distraction from irrelevant stimuli. Neuroimaging studies have shown that those who engage in heavy media multitasking perform poorer in distracted attention tasks while exhibiting greater activity in prefrontal regions during those tasks. These findings suggest that these individuals may require higher cognitive effort to maintain concentration when faced with distractor stimuli. Similarly, heavy internet usage and multitasking have been associated with decreased grey matter volume in brain regions involved in decision-making. Comorbidities between internet use disorders and attention deficit hyperactivity disorders have also been reported, suggesting that there may be strong links between excessive media usage and disorders of inattention.

Although research in this area is growing, the findings are still mixed. Some studies have confirmed these negative effects on attention, whereas others report that increased media multitasking may even be linked to increased performance in some aspects of cognition. It is possible that the internet allows for “cognitive off-loading” of certain cognitively demanding tasks, such as semantic memory retrieval, which may free up our cognitive resources for use in other tasks. It is difficult to disentangle whether heavy social media use leads to higher distractibility, or whether pre-existing differences in neural activity make some individuals more susceptible to distraction. What we do know, is that social media and technology offer easy-to-reach distractions, which may interfere with our ability to focus.

Social media use and mental health

Engaging with social media apps taps into more than just our brain’s attention network. It requires social reward processing, emotion-based processing, regulation, and thinking about the thoughts and feelings of others. Numerous studies have reported that positive attention on social media in the form of likes on Instagram, Twitter, and Facebook may cause our brains to release dopamine and activate reward circuits in the brain. Furthermore, reduced grey matter volume in regions involved in emotional regulation and social cognition, such as the amygdala and ventral striatum have also been associated with excessive social media use. Given the tight link between social media use and the brain’s reward system, there is potential for abuse or dependence.

Heavy social media use may also have important implications for psychological well-being. While social media use may provide an opportunity for social integration with similar interest groups, access to support groups, and motivation for a healthy lifestyle, it may also have more toxic effects on users’ mental health. Increased feelings of depression, anxiety, poor body image, and loneliness have all been reported following social media use.

Why are adolescents more susceptible?

Adolescence is a developmental stage in which the brain is undergoing extensive structural and functional remodeling. Impulse and cognitive control, as well as social reward and emotional processing, are not yet developed. This can lead adolescents to engage in more reward-seeking or risk-taking behaviours, and be more susceptible to distracting highly engaging social media content. As discussed above though, it is unclear whether social media use may influence our long-term ability to sustain attention, or whether it is merely a source of temporary distraction. Of greater importance for this age group may be the effects of social media on mental health. Adolescence is a sensitive developmental window in which neuropsychiatric disorders are most likely to emerge. Parental influence decreases, while the influence of peers and the need for peer acceptance increases. Managing social media use may be one helpful way to avoid overuse and some potential negative outcomes. Setting boundaries with social media use, such as reducing time spent on social networks, and establishing some no-phone zones in the home, or no-phone times (e.g. before bed) can be an effective way to prevent overuse. Gaining a better understanding of how adolescents process media content and peers’ feedback will be of critical importance for understanding how best to avoid negative impacts on mental health. 

What’s the takeaway?

With social media becoming a more and more prominent part of our everyday lives, there are many risks to be aware of, including social media overuse. Furthermore, heavy social media use may have an impact on how our brain functions. Although the extent to which social media use impacts our cognition and attention is still unclear, it certainly provides an additional source of distraction. Of greater concern, however, are the effects it may have on our mental health, particularly in more vulnerable age groups, such as adolescents. More research will be needed to better understand the impact that social media has in our lives, and how we can navigate its use in the future. 

References

Crone EA, Konijn EA. Media use and brain development during adolescence. Nature Communications (2018) 9(588). https://doi.org/10.1038/s41467-018-03126-x

Frith JA, Torous J, Frith J. Exploring the impact of internet use on memory and attention processes. International Journal of Environmental Research and Public Health (2020). 17 (9481); doi:10.3390/ijerph17249481

Baumgartner SE, van der Schuur WE, Lemmens JS, & Poel F.  The Relationship Between Media Multitasking and Attention Problems in Adolescents: Results of Two Longitudinal Studies. Human Communication Research (2017). 44 (1), 3-30. https://academic.oup.com/hcr/article-abstract/44/1/3/4760433

Ra CK, Cho J, Stone MD, De La Cedra J, Goldenson NI, Moroney E, Tung I, Lee SS, Leventhal AM. Association of Digital Media Use With Subsequent Symptoms of Attention-Deficit/Hyperactivity Disorder Among Adolescents. JAMA (2018). 320(3):255-263. doi:10.1001/jama.2018.8931

Cohen R.A. (2014) Neural Mechanisms of Attention. In: The Neuropsychology of Attention. Springer, Boston, MA. https://doi.org/10.1007/978-0-387-72639-7_10