Scientists See What People Picture in Their Mind’s Eye

Summary: Using electrocorticogram technology to capture brain waves, researchers found the meaning of what people imagine can be determined from brain wave patterns, even if the image differs from what a person is looking at.

Source: Osaka University

They say a picture is worth a thousand words. Now, researchers from Japan have found that even a mental picture can communicate volumes.

In a study published this month in Communications Biology, researchers from Osaka University have revealed that the meaning of what a person is imagining can be determined from their brain wave pattern, even if the image differs from what the person is looking at.

When we see images in real life, whether we are talking to a friend, watching a movie, or watching a beautiful sunset, our brains take in this visual information in a way that can be detected by a technique called electrocorticogram, which detects patterns of electrical activity in the brain. These patterns are not set in stone, however; they can be changed by what we are paying attention to or imagining at the time.

“Attention is known to modulate neural representations of perceived images,” says lead author of the study Ryohei Fukuma. “However, we didn’t know whether imagining a different image could also change these representations.”

To test this, the researchers developed a new technology by working with patients with epilepsy who already had electrodes implanted in their brains to record and display electrocorticogram readouts of images that they were imagining. The patients were shown an image of the real-time readout and instructed to mentally picture a different image representing a “landscape,” “human face,” or “word” (for example, thinking of a human face while looking at various types of images) to control the readout.

This shows a diagram from the study with a person sitting at a desk looking at a picture of a monkey and the output from the ECoG system
Electrocorticogram (ECoG) recordings were taken from 17 patients with epilepsy who had implanted subdural cortical electrodes related to visual perception. A decoder was trained to estimate the semantic meaning of the images that the patients were viewing from the intracranial ECoG recording using visual-semantic space. Based on the real-time inferred semantic information with the decoder, an image was displayed on a monitor placed in front of the patient. The patient then tried to display an image with instructed meaning by imagining it. Credit: The researchers

“The results clarified the relationship between brain activities when people look at images versus when they imagine them,” explains Takufumi Yanagisawa, senior author. “The electrocorticogram readouts of the imagined images were distinct from those provoked by the actual images viewed by the patients. They could also be modified to be even more distinct when the patients received real-time feedback.”

The time needed to generate a very clear distinction between the imagined image and the viewed image was different for imagining a “word” and a “landscape,” which could have something to do with the different parts of the brain involved in imagining these two concepts.

“Our findings suggest that a readout image controlled by the subject’s imagery can be inferred by an observer using this technology,” says Fukuma.

Given the accuracy with which this new technology displays images that exist within the subject’s mind, a similar approach could be used to develop a communication device for severely paralyzed patients, such as those with amyotrophic lateral sclerosis. Similar devices already used by some patients with this condition rely on motor control, which degenerates more quickly than visual cortical activity, so an imagery-based device could be highly valuable.

About this neuroscience research news

Author: Press Office
Source: Osaka University
Contact: Press Office – Osaka University
Image: The image is credited to the researchers

Original Research: Open access.
Voluntary control of semantic neural representations by imagery with conflicting visual stimulation” by Ryohei Fukuma et al. Communications Biology


Voluntary control of semantic neural representations by imagery with conflicting visual stimulation

Neural representations of visual perception are affected by mental imagery and attention. Although attention is known to modulate neural representations, it is unknown how imagery changes neural representations when imagined and perceived images semantically conflict.

We hypothesized that imagining an image would activate a neural representation during its perception even while watching a conflicting image. To test this hypothesis, we developed a closed-loop system to show images inferred from electrocorticograms using a visual semantic space.

The successful control of the feedback images demonstrated that the semantic vector inferred from electrocorticograms became closer to the vector of the imagined category, even while watching images from different categories. Moreover, modulation of the inferred vectors by mental imagery depended asymmetrically on the perceived and imagined categories.

Shared neural representation between mental imagery and perception was still activated by the imagery under semantically conflicting perceptions depending on the semantic category.

Join our Newsletter
I agree to have my personal information transferred to AWeber for Neuroscience Newsletter ( more information )
Sign up to receive our recent neuroscience headlines and summaries sent to your email once a day, totally free.
We hate spam and only use your email to contact you about newsletters. You can cancel your subscription any time.