How Much Effort Does it Take to Just Listen?

Post by Anastasia Sares

What is listening effort?

We don’t often think about how much of our mental space we reserve for listening to speech—for many people, it feels effortless. However, there are often obstacles: background noises, distracting conversations, or age-related hearing loss, to name a few. According to some models, we have a limited amount of mental resources, and the more we spend trying to decipher speech, the less we have left over for critical thinking, memory, and other high-level processing. However, a nebulous concept such as “effort” isn’t easy to quantify, and scientists have tried a number of approaches, from self-report questionnaires to full-sized brain scanners. Here’s a run-down of all the techniques used to measure listening effort.

Using self-report measures

One way to measure effort on a task is to ask people about it directly. This is the simplest method, but it can get tricky because people may have different interpretations of what “effort” means. To be more precise, some recommend breaking down effort into sub-components, like mental effort, physical effort, time pressure, or frustration. The NASA task load index is one such breakdown. However, a recent study suggests that we should ask about tiredness, a question that is not present in the NASA task load index. In that study, people’s tiredness ratings during speech listening were shown to correlate with the next method of measuring effort: pupil size.

Using the size of the pupil (pupillometry)

Our pupils dilate under states of mental arousal or effort, and specifically when listening conditions are worsened. Using video recordings of the eye or special glasses with infra-red cameras, we can measure the size of a person’s pupil as they hear and respond to sounds—this is called pupillometry. It is less subjective than self-report, and we can evaluate the effort someone expends on a task without forcing them to stop and reflect. However, pupillometry currently needs specific lighting conditions, and sometimes the pupil response can plateau in complex tasks. So, there are still some challenges to using this method.

Using brain activity (EEG, fMRI, fNIRS)

Electroencephalography, or EEG, measures electrical activity in the brain and can be used as another way to tap into listening effort. Among other EEG measures, the N100 response to sound is one index of this effort. This automatic response happens 100 milliseconds after the onset of a sound, and it becomes bigger when the speech is made less intelligible. Another EEG measure of effort is alpha power. If we take the activity in the alpha range and sum up its power over the course of an experiment, we can see when more effort is being expended.

Blood flow to different brain areas has long been used as a proxy for brain activity in those areas. In particular, blood flow to the left inferior frontal region of the brain (close to the temples) and the superior temporal gyrus (just above the ears) can give us a hint about how much effort is being exerted. This can be done in a magnetic scanner that detects the magnetic properties of blood (fMRI) or using a cap with small infra-red lights pointed at the scalp (fNIRS). These blood flow methods are a little slower than EEG, but fMRI, in particular, can pinpoint the location of activity in the brain with better accuracy, and fNIRS is advantageous because it doesn’t interfere with hearing aids or other devices.

What’s the bottom line?

Hearing is a crucial aspect of health: hearing loss has a large societal burden and may contribute to the risk of dementia later in life. Armed with multiple tools to measure listening effort, we can study how it varies in different conditions and populations, and better understand the link between hearing and cognition.

References

Pichora-Fuller, M. K. et al. Hearing Impairment and Cognitive Energy. Ear Hear. 37, 5S-27S (2016).

Hart, S. G. & Staveland, L. E. Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research. Adv. Psychol. 52, 139–183 (1988).

McGarrigle, R., Rakusen, L. & Mattys, S. Effortful listening under the microscope: Examining relations between pupillometric and subjective markers of effort and tiredness from listening. Psychophysiology 58, 1–22 (2021).

Zekveld, A. A., Kramer, S. E. & Festen, J. M. Pupil response as an indication of effortful listening: The influence of sentence intelligibility. Ear Hear. 31, 480–490 (2010).

Koo, M. et al. Effects of noise and serial position on free recall of spoken words and pupil dilation during encoding in normal-hearing adults. Brain Sci. 11, 1–14 (2021).

Zhang, Y., Lehmann, A. & Deroche, M. Disentangling listening effort and memory load beyond behavioural evidence: Pupillary response to listening effort during a concurrent memory task. (2020). doi:10.1101/2020.05.04.076588

Obleser, J. & Kotz, S. A. Multiple brain signatures of integration in the comprehension of degraded speech. Neuroimage 55, 713–723 (2011).

Obleser, J., Wöstmann, M., Hellbernd, N., Wilsch, A. & Maess, B. Adverse listening conditions and memory load drive a common alpha oscillatory network. J. Neurosci. 32, 12376–12383 (2012).

Wild, C. J. et al. Effortful listening: The processing of degraded speech depends critically on attention. J. Neurosci. 32, 14010–14021 (2012).

Kousaie, S. et al. Language learning experience and mastering the challenges of perceiving speech in noise. Brain Lang. 196, 104645 (2019).

Zekveld, A. A., Heslenfeld, D. J., Johnsrude, I. S., Versfeld, N. J. & Kramer, S. E. The eye as a window to the listening brain: Neural correlates of pupil size as a measure of cognitive listening load. Neuroimage 101, 76–86 (2014).

Rovetti, J., Goy, H., Pichora-Fuller, M. K. & Russo, F. A. Functional Near-Infrared Spectroscopy as a Measure of Listening Effort in Older Adults Who Use Hearing Aids. Trends Hear. 23, 233121651988672 (2019).

Ford, A. H. et al. Hearing loss and the risk of dementia in later life. Maturitas. 112, 1–11 (2018).

Neural Mechanisms Involved in the Extinction of Long-Term Trauma

Post by Lina Teichmann

What's the science?

Traumatic experiences often result in enduring memories of fear. Exposure therapy is a common treatment to overcome trauma by exposing patients to the context of fear-inducing memories in a safe environment. However, it is known that exposure therapy is less successful if the traumatic experience occurred a long time ago. This week in Nature Neuroscience, Silva and colleagues used a fear extinction paradigm with mice to test how neural mechanisms involved in overcoming fear differ depending on the age of the traumatic memory.

How did they do it?

To mimic the trauma, the authors placed mice in a conditioning chamber, where they received electric shocks to their paws. After either 1 day or 30 days, the mice were re-exposed to the same chamber without receiving shocks (recall) with subsequent re-exposing events over several days (fear extinction). When the animals were re-exposed to the traumatic context, the fear response was behaviorally quantified by examining freezing responses (prolonged absences of motion). Viral tracing, neuronal activity mapping, fiber photometry, and chemo- and optogenetics were used to examine the effect of long-term fear extinction on neural circuitry. In particular, the authors examined the functional responses to long- and short-term fear extinction in infralimbic cortex to basolateral amygdala and thalamic nucleus reuniens to basolateral amygdala pathways.

What did they find?

To overcome trauma, fear-evoking contexts have to be newly associated with safety. The results show that the neural mechanisms underlying this type of fear extinction depend on whether the fear-evoking experience occurred recently or a long time ago. While direct connections from the infralimbic cortex to the basolateral amygdala are critical for recent fear extinction, long-term fear extinction requires the recruitment of an additional pathway. In particular, when overcoming long-term fear, fear-related information is sent upstream from the infralimbic cortex via the thalamic nucleus reuniens to the basolateral amygdala. The behavioral expression of fear – freezing – was modulated by the activity in the thalamic nucleus reuniens. The activity in the nucleus reuniens peaked just before the freezing response ended and the freezing length could be manipulated by artificially increasing or decreasing activity in this area. This finding suggests that activity in the thalamic nucleus reuniens plays a role in learning to associate safety with a context that initially evoked fear.

What's the impact?

Traumatic memories are often long-lasting and can lead to mental disorders such as post-traumatic stress disorder. Silva and colleagues show that the time that has passed since a fear-evoking event modulates neural mechanisms involved in overcoming trauma. These findings improve our understanding of long-lasting traumatic memories and set the stage for future research into how we can weaken traumatic associations.   

Silva et al. A thalamo-amygdalar circuit underlying the extinction of remote fear memories. Nature Neuroscience (2021).Access the original scientific publication here.

Experience Replay is Associated with Efficient Non-local Learning

Post by Andrew Vo

What's the science?

When we make decisions, we often rely on previously learned relationships between actions and their outcomes. Although it is relatively easy to assign a value to an action when they are experienced close together in time and space (local), it becomes more challenging to do so when they are further apart (non-local). It has been hypothesized that non-local learning is achieved via a model-based approach such as ‘experience replay’, in which a learned map of the environment is used to link local rewards to non-local actions. Direct evidence for such a mechanism in humans has yet to be established, however. This week in Science, Liu and colleagues used a novel decision-making task and non-invasive brain imaging to test how experience replay helps achieve non-local learning.

How did they do it?

To test their hypothesis, the authors designed a decision-making task that separated local and non-local learning. On each trial, the participant was presented with one of three starting arms. Each arm contained two paths, between which the participant would choose. Each path was composed of a sequence of three stimuli followed by a reward outcome. Critically, the two possible outcomes that were reachable in each arm were also shared across all starting arms. In this way, participants could use the learned outcome in the current arm (local) to inform and update their choices when encountering the other two starting arms (non-local), in a model-based approach.

To measure neural replay, the authors used magnetoencephalography (MEG) to record fast whole-brain activity as participants performed the task. Replay was defined as the reactivation of a sequence at the time of reward receipt, which could occur in both forward (i.e., from the beginning of a path sequence to the eventual outcome) and backward (i.e., from the outcome at the end of a path sequence towards the beginning) directions. They also examined how neural replay was prioritized for non-local experiences based on their utility for future decisions. This utility was determined by the gain (i.e., how informative is this current reward for improving my choice at this arm) and the need (i.e., how frequently will this arm be visited in the future) of each experience. These two task features were manipulated by changing the reward and arm probabilities, respectively, across trials.

What did they find?

When participants encountered the same starting arm as before, they were more likely to favor the path that was previously rewarded, indicative of direct, model-free learning. This local learning was found to transfer to non-local experiences, as participants would favor the path leading to a previously rewarded outcome even when it was presented in a different starting arm. Using computational modeling, the authors found that values of non-local paths were updated to a similar extent as those of local paths, and learning rates were higher for non-local paths with greater priority (higher gain and need).

Examining MEG recordings, the authors observed two types of neural replay occurring after reward receipt: (1) a fast, forward replay that peaked at 30-ms lag, and (2) a slower, backward replay that peaked at 160-ms lag. The forward replay was associated with local actions and an increase in ripple frequency power, whereas the backward replay preferentially encoded non-local actions. Computational modeling revealed that only the backward replay was associated with efficient non-local learning. Similarly, only the backward replay was related to the utility (higher gain and need) of non-local experiences.

What's the impact?

In summary, this study revealed that backward replay serves as a neural mechanism for non-local learning and is prioritized based on utility for future decisions. The results here build upon model-based reinforcement learning theories largely tested in rodents and extend them to human behavior. They contribute to our understanding of how the brain might bridge the gap between direct and indirect experiences to guide our decisions.

Liu et al. Experience replay is associated with efficient nonlocal learning. Science (2021). Access the original scientific publication here.