Summary: Using the Dr. Seuss classic, The Lorax, researchers shed new light on how the brain engages during complex audiovisual speech perception. The findings reveal how the brain utilizes a complex network of brain regions involved in sensory processing, multisensory integration, and cognitive functions to comprehend a story’s context.
Source: University of Rochester
Researchers at the Del Monte Institute for Neuroscience at the University of Rochester have expanded the understanding of how the brain is engaged during complex audiovisual speech perception.Â
The study now out in NeuroImage, describes how listening and watching a narrator tell a story activates an extensive network of brain regions involved in sensory processing, multisensory integration, and cognitive functions associated with the comprehension of the story content.
Understanding the involvement of this larger network has the potential to give researchers new ways to investigate neurodevelopmental disorders.
“Multisensory integration is an important function of our nervous system as it can substantially enhance our ability to detect and identify objects in our environment,” said Lars Ross, Ph.D., research assistant professor of Imaging Sciences and Neuroscience and first author of the study.
“A failure of this function may lead to a sensory environment that is perceived as overwhelming and can cause a person to have difficulty adapting to their surroundings, a problem we believe underlies symptoms of some neurodevelopmental disorders such as autism.”
Using fMRI, researchers examined the brain activity of 53 participants as they watched a video recording of a speaker reading “The Lorax.” How the story was presented would change randomly in one of four ways – audio only, visual only, synchronized audiovisual, or unsynchronized audiovisual. Researchers also monitored the participants’ eye movements.
They found that along with the previously identified sites of multisensory integration, viewing the speaker’s facial movements also enhanced brain activity in the broader semantic network and extralinguistic regions not usually associated with multisensory integration, such as the amygdala and primary visual cortex.
Researchers also found activity in thalamic brain regions which are known to be very early stages at which sensory information from our eyes and ears interact.
“This suggests many regions beyond multisensory integration play a role in how the brain processes complex multisensory speech – including those associated with extralinguistic perceptual and cognitive processing,” said Ross.
Researchers designed this experiment with children in mind, according to the investigators who have already begun working with both children and adults on the autism spectrum in an effort to gain insight into how their ability to process audiovisual speech develops over time.
“Our lab is profoundly interested in this network because it goes awry in a number of neurodevelopmental disorders,” said John Foxe, Ph.D., lead author of this study.
“Now that we have designed this detailed map of the multisensory speech integration network, we can ask much more pointed questions about multisensory speech in neurodevelopmental disorders, like autism and dyslexia, and get at the specific brain circuits that might be impacted.”
Additional co-authors include Sophie Molholm, Ph.D., and Victor Bene of Albert Einstein College of Medicine, and John Butler, Ph.D., of Technological University Dublin.
This research was a collaboration of two Intellectual and Developmental Disability Research Centers (IDDRC), which are supported by the National Institute of Child Health and Human Development (NICHD). In 2020, the University of Rochester was designated as an IDDRC by the NICHD.Â
The award recognized the Medical Center’s national leadership in research for conditions such as autism, Batten disease, and Rett syndrome. Molholm is the co-director of the Rose F. Kennedy IDDRC at Einstein.
About this speech processing research news
Author: Kelsie Smith Hayduk
Source: University of Rochester
Contact: Kelsie Smith Hayduk – University of Rochester
Image: The image is in the public domain
Original Research: Open access.
“Neural correlates of multisensory enhancement in audiovisual narrative speech perception: A fMRI investigation” by Lars Ross et al. NeuroImage
Abstract
Neural correlates of multisensory enhancement in audiovisual narrative speech perception: A fMRI investigation
This fMRI study investigated the effect of seeing articulatory movements of a speaker while listening to a naturalistic narrative stimulus. It had the goal to identify regions of the language network showing multisensory enhancement under synchronous audiovisual conditions.
We expected this enhancement to emerge in regions known to underlie the integration of auditory and visual information such as the posterior superior temporal gyrus as well as parts of the broader language network, including the semantic system.
To this end we presented 53 participants with a continuous narration of a story in auditory alone, visual alone, and both synchronous and asynchronous audiovisual speech conditions while recording brain activity using BOLD fMRI.
We found multisensory enhancement in an extensive network of regions underlying multisensory integration and parts of the semantic network as well as extralinguistic regions not usually associated with multisensory integration, namely the primary visual cortex and the bilateral amygdala.
Analysis also revealed involvement of thalamic brain regions along the visual and auditory pathways more commonly associated with early sensory processing.
We conclude that under natural listening conditions, multisensory enhancement not only involves sites of multisensory integration but many regions of the wider semantic network and includes regions associated with extralinguistic sensory, perceptual and cognitive processing.
