“Circuit Motifs” Underlying Short-Term Memory

Post by Lani Cupo

What's the science?

Neurons generally fire briefly in response to stimuli that activate them. However, the neurons that underlie short-term memory are arranged in “circuit motifs,” or interconnected groups of neurons that continue to activate one another. This pattern of activation allows neurons to maintain a signal after the initial stimulus has ended, forming the basis of short-term memory. Circuits can take many forms in terms of the way they are connected and the strength of their connections. It is still unclear what role these different forms of motifs play in short-term memory. This week in Nature Neuroscience, Daie and colleagues investigated circuit motifs of short-term memory by using lasers (photostimulation) to stimulate neurons in the anterolateral motor cortex, an area of the brain that stores short-term memories for upcoming planned movements.

How did they do it?

The authors used data from adult mice that expressed genes allowing the authors to activate neurons with light and record activation as fluorescence. They imaged neuron activation while the animals behaved freely using two-photon microscopy. The mice were trained to distinguish between two different auditory stimuli, responding either “right” or “left” for a reward. By imaging neuron activity while they performed this task, the authors could identify which specific neurons were selective for left and right movement directions. 

Next, in order to better understand circuit motifs, the authors directly stimulated entire groups of neurons at the same time. They then measured the activity in neurons that were greater than 30 micrometers away from the directly targeted neurons. By activating these groups of neurons within a network, the authors could test whether other nearby neurons would also respond to this activation, or in other words, whether they were ‘coupled’ with this circuit. Using the activation patterns, they were able to calculate the connection strength between stimulated and unstimulated neurons in the same circuit, as well as the duration of activity in the circuit. Furthermore, they examined the activity of neurons selective for “left” and “right” responses in the behavioral task following stimulation. 

What did they find?

The authors found that stimulating small groups of neurons altered activity in other nearby neurons, demonstrating that they were ‘coupled’ with that circuit. These ‘coupled’ neurons that were indirectly activated also showed persistent activity lasting well beyond the duration of the light stimulation. This finding demonstrates evidence for circuits composed of strongly-connected subnetworks that produce persistent activity which may underlie short-term memory. The authors also found that neurons with similar directional selectivity (i.e. ‘right’ or ‘left’ in the behavioural task) were more likely to be coupled. When the authors stimulated neurons that were selective for the ‘right’ direction or ‘left’ direction in the behavioural task, they found that this stimulation reliably biased behaviour in the task to a greater degree than would be expected by chance. However, the direction of movement did not necessarily correspond with the selectivity of the neuron (e.g., right activation did not always result in a rightwards movement). The authors concluded that the activation of a small group of neurons reliably predicted neural activity and behaviour (i.e. movement direction).

What's the impact?

This study found that brief stimulation of groups of neurons resulted in long-lasting, persistent activity in a network of neurons. Further, the authors demonstrated that the stimulation indirectly activated nearby ‘coupled’ neurons, suggesting that these circuits are composed of subnetworks or modules. The persistent activity in these networks was directly related to short-term behavioural outcomes. These findings provide insight into the mechanisms of short-term memory at the circuit level. Future research is needed to further investigate the structure and activity of these modular networks and how they impact short-term memory and behaviour.

Daie et al. Targeted photostimulation uncovers circuit motifs supporting short-term memory. Nature Neuroscience (2021). Access the original scientific publication here.

Reinforcement Learning Models Capture Human Decision-Making Processes

Post by Shireen Parimoo

What's the science?

How do people flexibly plan their actions in service of novel goals? According to reinforcement learning (RL) models of human behaviour, actions are chosen to maximize reward in the long run. In standard RL algorithms, actions are guided by knowledge of the environment, where the outcomes achieved are either known (model-based) or learned when they occur without much prior knowledge (model-free). These algorithms can require a lot of resources and also run the risk of under- or over-generalizing to new tasks. 

Two new algorithms have been proposed to model human cross-task generalization. One approach extracts similarities across tasks to inform future actions using universal value function approximators (UVFAs). Let’s consider this example: you are both tired and hungry, and want to make a decision between going to a Burger Shop (known for food), a coffee shop (known for coffee) or a diner (known for both). You know the Burger Shop is good when hungry and the coffee shop is good when tired, so you select an action using UVFA: you look for a place similar to both these places when you are both hungry and tired (the diner). In other words, you use previous values to extrapolate and predict a new value. Another approach is to keep track of actions associated with commonly encountered tasks using a generalized policy improvement algorithm (GPI): this predicts the outcome of an action from learned experience. Here’s an example of selecting an action using GPI: You are hungry and looking for a place to eat. You have been frequenting the diner lately and you have a ‘policy’ of going there when hungry. Now, you’re tired and would like to get coffee. You can also envision the outcome of going to the diner - there is coffee available, and people tend to drink coffee there. Therefore, you might choose to apply this same policy of going to the diner, in your decision to get coffee. In other words, you are generalizing the policy to a new task. This is the distinction between UVFA and GPI: UVFA uses previously learned values to approximate a solution, while GPI evaluates a previously learned policy (the relationship between an action and an end state or solution), and applies this policy to a new situation.

This week in Nature Human Behavior, Tomov and colleagues tested the generalizability of standard and new RL algorithms across tasks and compared their performance to human behavior.

How did they do it?

In a set of four online experiments, over 1100 participants played a resource-trading game set in a castle. Before each trial began, participants saw the “daily market price” for the resources – the amount of money they could expect to either receive or pay for each resource (wood, stone, and iron). For example, they might see “Wood: $1, Stone, $2, and Iron, $0”, indicating that they would receive $1 for each wood and pay $2 for each stone they had at the end of the trial. Each trial consisted of a two-step decision-making process: Participants were instructed to choose a) between three doors to enter one of three rooms, and then b) choose between another three doors per room to enter a final room containing resources. Importantly, each final door always led to the same amount of resources in the corresponding final room (e.g., door 2 in room 3 always contained 100 wood, 40 stone, and 0 iron across all trials). The amount of money received or paid, however, would change from trial to trial because the ‘daily market value’ changed each trial. In our example, participants might receive $1 x 100 wood ($100), -$2 x 40 stone (-$80), and $0 x 0 iron = $20 total.

The researchers carefully selected a few sets of daily market prices for the resources for each experiment in order to manipulate the computational demands required and to vary the degree of difficulty in successfully mapping actions to outcomes across the four experiments. In the training phase of each of the four experiments, participants completed 100 trials, each of which was randomly assigned one of the pre-selected sets of daily market prices. For example, the profit from finding wood might double earnings on one trial but cost participants money on the next trial. Thus, participants had to learn which final door would ensure maximum profit. 

The goal of the experiments was to determine which RL model the participants likely had used (model-free, model-based, UVFA, or GPI). The experiments were designed in such a way that the door participants chose on a test trial, completed after the 100 training trials, would be likely to reflect the underlying RL algorithm that best modelled how they chose actions to maximize reward. For example, the third door in the second room in one experiment would in fact result in the highest profit in the final test trial, but it would only be selected by a model-based learner who had successfully learned the entire structure of the environment. 

What did they find?

In the first experiment, participants were most likely to select actions leading to the final doors predicted by the model-based and the GPI algorithms. In more difficult experiments, however, participants were by far more likely to choose the final door predicted by GPI. Interestingly, the final door chosen by the model-based and UVFA algorithms would have been the most rewarding, yet participants did not choose those actions more frequently than would be expected by chance. In comparing the different algorithms, the authors found that participants learned to select the final door predicted by GPI faster than that predicted by the model-based algorithm, which is consistent with the fact that model-based algorithms tend to require more resources. Finally, as GPI makes predictions based on learned experience, the authors compared participants’ choice history during the training phase to their actions on the test trial. Here, learned experience did indeed drive choice at test time; participants’ tendency to choose the same door during training predicted the probability that they would select that door in the test trial. This indicates that participants kept track of the different situations they encountered during the training phase, along with the associated action-state mapping, which informed their behavior during the test.

What's the impact?

People use their knowledge of frequently encountered experiences to make predictions about future outcomes and inform their decisions. One of the interesting outcomes of this study is that people do not necessarily make the most rewarding decisions, but rather they tend to map previously used policies onto new scenarios. This finding provides exciting new insight into how reinforcement learning captures human decision-making processes in complex and changing environments.

Tomov et al. Multi-task reinforcement learning in humans. Nature Human Behavior (2021). Access the original scientific publication here.

How Sleep Helps Us Remember and Forget

Post by Amanda McFarlan

What’s the deal with sleep?

Humans spend approximately one third of their lives sleeping, so it is no surprise that we’re curious about it! Sleep has a wide variety of benefits, like repairing and regenerating tissues in the body, improving cognitive and physical performance, and consolidating memories. On the other hand, a chronic lack of sleep can put us at risk of developing health problems like cardiovascular disease, high blood pressure, diabetes, and depression. So, what happens when we sleep? Every night, when our heads hit the pillow, we enter into the first stage of ‘non-Rapid Eye Movement’ (non-REM) sleep. Non-REM sleep consists of 4 stages, with Stage 1 being the lightest sleep stage and Stage 4 being the deepest. Your body moves through the 4 stages of non-REM sleep and finally through REM sleep in a cycle that takes approximately 90 minutes, and this cycle is repeated throughout the night. Non-REM and REM sleep are characterized by different brain activity patterns, with non-REM sleep creating slow waves in its deepest stages, called ‘slow-wave sleep’, and REM sleep generating activity patterns that resemble wakefulness. The role of non-REM and REM sleep in the transfer and long-term storage of memories, known as memory consolidation, has been studied for many years. Here, we will discuss how sleep helps us remember or forget, as well as what goes wrong when we don’t sleep.

How does sleep help us remember?

Evidence strongly suggests that sleep is integral to memory consolidation. For example, a behavioural study, in which participants performed a visual task, a motor sequence task, and a motor adaptation task, found that participants’ performance was greatly improved if they had a full night’s sleep compared to those that did not sleep. The degree of performance improvement for each type of task was dependent on improved sleep in different stages in the sleep cycle. These findings suggest that non-REM and REM sleep both play an important role in memory consolidation. In line with this, other studies have shown that intensive learning of a new task is followed by increased time spent in REM sleep, resulting in subsequent task improvement, as well as the amplification of slow waves during non-REM sleep. Sleep results in a reactivation of cells in the hippocampus, which subsequently reactivate representations of memory in the cortex, also known as an engram. Over time, after many reactivations, these memories become distributed and consolidated within the cortex. 

Interestingly, research has shown that while we’re sleeping there is increased activity in the same hippocampal place cells (neurons that are activated when moving through specific locations in the environment) that were active throughout the day. This reactivation of hippocampal place cells during REM sleep follows a theta frequency band pattern of firing, hypothesized to be critical for memory consolidation. This hippocampal activity is mediated by neurons that release the neurotransmitter acetylcholine in the hippocampus. Acetylcholine, which plays a major role in altering the strength of synaptic connections, crucial for memory, is known to be elevated during REM sleep. REM sleep has also been associated with the upregulation of the expression of several calcium-dependent genes that are thought to be involved in synaptic plasticity and memory consolidation. 

Compared to REM sleep, the conditions in non-REM sleep are less ideal for promoting synaptic plasticity. For example, acetylcholine and calcium-dependent genes are expressed at low levels or are absent altogether during non-REM sleep. However, researchers have proposed that non-REM sleep might be important for the later stages of memory consolidation, rather than the initial conversion of short-term memories to long-term memories. In support of this, protein synthesis, which is required for long-term but not short-term potentiation (strengthening) of synapses, is increased during non-REM sleep. Therefore, the induction of protein synthesis during non-REM sleep may act to strengthen the synapses that were sufficiently potentiated during wakefulness. 

Although the majority of research on sleep and memory focuses on the role of the hippocampus in memory consolidation, a recent study has provided evidence that the thalamus might also play a role in memory consolidation during sleep. In this study, memory encoding (when memories are initially stored) during a visual task was shown to increase the activity of sensory relay nuclei of the thalamus in mice. Following a night of sleep, the primary visual cortex also showed evidence of a potentiated response to the visual task. Together, these findings suggest that task-related information may be passed from the thalamus to the primary visual cortex, resulting in the formation of a corresponding memory during sleep.

How does sleep help us forget?

Sleep research is centered around how we remember. However, sleep arguably plays just as important a role in the process of forgetting memories. The hippocampus serves as a temporary storage area for newly formed memories until they can be consolidated and integrated into long-term memory storage in the cortex. As a result, the hippocampus must be able to unlearn memories that have already been consolidated or memories that are not pertinent in order to store new memories. Research has shown that in addition to helping with memory consolidation, sleep is also important for unlearning memories. Studies in rats have shown that following sleep, there are widespread reductions in dendritic spines (protrusions on the dendrite that form synapses with nearby neurons) in the cortex as well as a reduction in receptors on glutamatergic neurons that are critical for memory and learning.

Norepinephrine and serotonin are two neurotransmitters in the brain that are associated with the enhancement of synaptic plasticity. During REM sleep, however, norepinephrine and serotonin signaling is suppressed, suggesting that REM sleep may allow for the depotentiation — or weakening — of synapses.  

What happens when we don’t sleep?

We all know how difficult it is to get through the day after a sleepless night. Suddenly, concentrating on what was previously a trivial task can become very challenging. Neuroimaging data has shown that sleep deprived individuals recruit more brain areas while performing the same cognitive task compared to individuals who slept normally. Moreover, brain imaging studies have revealed that hippocampal function is greatly reduced following one night of sleep deprivation, which suggests that losing sleep may actually disrupt our ability to learn new things. Sleep deprivation studies in rats have demonstrated the importance of REM sleep for learning as well as the induction and maintenance of long-term potentiation of synapses during learning. Additionally, REM sleep deprivation was shown to impair learning-dependent neurogenesis (the formation of new neurons) in the hippocampal dentate gyrus, which can impact future learning. The role of REM sleep for learning and memory is particularly relevant for individuals who are treated for depression with antidepressants, since these medications can greatly reduce the amount of time spent in REM sleep and may potentially have consequences on the efficacy of memory consolidation.

How can we get a good night’s sleep?

Given what we know about the role of sleep for learning and memory, it’s important to ensure that we get a good night’s sleep. However, with the challenges of daily life, this is not always an easy feat. First, it is important to establish a regular sleep schedule where you go to sleep and wake up around the same time each day, even when traveling or on the weekends. This habit can reinforce your body’s circadian rhythms, which helps your body to prepare for sleep and wakefulness more efficiently. Second, it is important to avoid using electronic devices before bed, like watching television or using your phone or tablet. The blue light that is emitted by these devices tricks our bodies into thinking it is daylight, and, as a result, our bodies produce lower levels of the hormone melatonin which promotes sleep. Third, use what you know about the science of sleep cycles to your advantage by timing your sleep in 90-minute intervals. For example, by setting your alarm for 7.5 hours of sleep (5 sleep cycles x 90 minutes each) you may actually feel more refreshed than if you slept for 8.5 hours and were awakened during the middle of a deep stage of sleep. Finally, avoiding caffeine and naps late in the afternoon or evening, as well as avoiding large meals or exercise right before bed may help to promote better sleep. 

Now, time to consolidate all of this learning with a good night’s sleep!

Feld, G.B., & Born, J. Sculpting memory during sleep: concurrent consolidation and forgetting. Current opinion in neurobiology, 44, 20–27 (2017). https://doi.org/10.1016/j.conb.2017.02.012

Klinzing, J.G., Niethard, N. & Born, J. Mechanisms of systems memory consolidation during sleep. Nat Neurosci 22, 1598–1610 (2019). https://doi.org/10.1038/s41593-019-0467-3

Poe, G. R., Walsh, C. M., & Bjorness, T. E. Cognitive neuroscience of sleep. Progress in brain research, 185, 1–19 (2010). https://doi.org/10.1016/B978-0-444-53702-7.00001-4

Stickgold, R. Sleep-dependent memory consolidation. Nature 437, 1272–1278 (2005). https://doi.org/10.1038/nature04286