Reconstructing spoken words as processed in nonhuman primate brains

Summary: Using artificial intelligence and brain-computer interface technology, researchers reconstructed English words from neural signals recorded from the brains of non-human primates.

Source: Brown University

A team of Brown University researchers has used a brain-computer interface to reconstruct English words from neural signals recorded in the brains of nonhuman primates. The research, published in the journal Nature Communications Biology, could be a step toward developing brain implants that may help people with hearing loss, the researchers say.

“What we’ve done is to record the complex patterns of neural excitation in the secondary auditory cortex associated with primates’ hearing specific words,” said Arto Nurmikko, a professor in Brown’s School of Engineering, a research associate in Brown’s Carney Institute for Brain Science and senior author of the study. “We then use that neural data to reconstruct the sound of those words with high fidelity.

“The overarching goal is to better understand how sound is processed in the primate brain,” Nurmikko added, “which could ultimately lead to new types of neural prosthetics.”

The brain systems involved in the initial processing of sound are similar in humans and non-human primates. The first level of processing, which happens in what’s called the primary auditory cortex, sorts sounds according to attributes like pitch or tone. The signal then moves to the secondary auditory cortex, where it’s processed further. When someone is listening to spoken words, for example, this is where the sounds are classified by phonemes — the simplest features that enable us to distinguish one word from another. After that, the information is sent to other parts of the brain for the processing that enables human comprehension of speech.

But because that early-stage processing of sound is similar in humans and non-human primates, learning how primates process the words they hear is useful, even though they likely don’t understand what those words mean.

For the study, two pea-sized implants with 96-channel microelectrode arrays recorded the activity of neurons while rhesus macaques listened to recordings of individual English words and macaque calls. In this case, the macaques heard fairly simple one- or two-syllable words — “tree,” “good,” “north,” “cricket” and “program.”

The researchers processed the neural recordings using computer algorithms specifically developed to recognize neural patterns associated with particular words. From there, the neural data could be translated back into computer-generated speech. Finally, the team used several metrics to evaluate how closely the reconstructed speech matched the original spoken word that the macaque heard. The research showed the recorded neural data produced high-fidelity reconstructions that were clear to a human listener.

The use of multielectrode arrays to record such complex auditory information was a first, the researchers say.

“Previously, work had gathered data from the secondary auditory cortex with single electrodes, but as far as we know this is the first multielectrode recording from this part of the brain,” Nurmikko said. “Essentially we have nearly 200 microscopic listening posts that can give us the richness and higher resolution of data which is required.”

One of the goals of the study, for which doctoral student Jihun Lee led the experiments, was to test whether any particular decoding model algorithm performed better than others. The research, in collaboration with Wilson Truccolo, a computational neuroscience expert, showed that recurrent neural networks (RNNs) — a type of machine learning algorithm often used in computerized language translation — produced the highest-fidelity reconstructions. The RNNs substantially outperformed more traditional algorithms that have been shown to be effective in decoding neural data from other parts of the brain.

This is a print out of the recordings
Using a brain-computer interface, a team of researchers has reconstructed English words from the brain activity of rhesus macaques that listened as the words were spoken. The image shows spectral analysis of the sound of the words the primate heard, as well as the corresponding neuronal spikes associated with each word. The image is credited to Nurmikko Lab.

Christopher Heelan, a research associate at Brown and co-lead author of the study, thinks the success of the RNNs comes from their flexibility, which is important in decoding complex auditory information.

“More traditional algorithms used for neural decoding make strong assumptions about how the brain encodes information, and that limits the ability of those algorithms to model the neural data,” said Heelan, who developed the computational toolkit for the study. “Neural networks make weaker assumptions and have more parameters allowing them to learn complicated relationships between the neural data and the experimental task.”

Ultimately, the researchers hope, this kind of research could aid in developing neural implants the may aid in restoring peoples’ hearing.

“The aspirational scenario is that we develop systems that bypass much of the auditory apparatus and go directly into the brain,” Nurmikko said. “The same microelectrodes we used to record neural activity in this study may one day be used to deliver small amounts of electrical current in patterns that give people the perception of having heard specific sounds.”

Funding: The research was supported by the U.S. Defense Advanced Research Projects Agency (N66001-17-C-4013) and a private gift to Brown. Other coauthors on the paper were Ronan O’Shea, Laurie Lynch and David Brandman.

About this neuroscience research article

Source:
Brown University
Media Contacts:
Kevin Stacey – Brown University
Image Source:
The image is credited to Nurmikko Lab.

Original Research: Open access
“Decoding speech from spike-based neural population recordings in secondary auditory cortex of non-human primates”. Christopher Heelan, Jihun Lee, Ronan O’Shea, Laurie Lynch, David M. Brandman, Wilson Truccolo & Arto V. Nurmikko.
Communications Biology doi:10.1038/s42003-019-0707-9.

Abstract

Decoding speech from spike-based neural population recordings in secondary auditory cortex of non-human primates

Direct electronic communication with sensory areas of the neocortex is a challenging ambition for brain-computer interfaces. Here, we report the first successful neural decoding of English words with high intelligibility from intracortical spike-based neural population activity recorded from the secondary auditory cortex of macaques. We acquired 96-channel full-broadband population recordings using intracortical microelectrode arrays in the rostral and caudal parabelt regions of the superior temporal gyrus (STG). We leveraged a new neural processing toolkit to investigate the choice of decoding algorithm, neural preprocessing, audio representation, channel count, and array location on neural decoding performance. The presented spike-based machine learning neural decoding approach may further be useful in informing future encoding strategies to deliver direct auditory percepts to the brain as specific patterns of microstimulation.

Feel free to share this Neuroscience News.
Join our Newsletter
I agree to have my personal information transferred to AWeber for Neuroscience Newsletter ( more information )
Sign up to receive our recent neuroscience headlines and summaries sent to your email once a day, totally free.
We hate spam and only use your email to contact you about newsletters. You can cancel your subscription any time.