Summary: What does a mouse actually see? For the first time, researchers have successfully reconstructed 10-second video clips based purely on the neural activity of mice. By recording the firing of individual neurons in the visual cortex while mice watched videos, the team developed an algorithm that “translates” brain signals back into moving images.
This breakthrough uses single-cell recordings—a much more precise method than traditional fMRI—to show that our brains don’t just act like a camera; they warp and skew reality based on how the mind interprets sensory data.
Key Facts
- Neural Decoding: Researchers used microscopic imaging to detect calcium level boosts in individual brain cells, identifying exactly which neurons were firing in response to visual stimuli.
- Pixel-by-Pixel Reconstruction: The algorithm started with a blank screen and gradually updated pixels based on the “difference” between predicted and actual neural activity until a video emerged.
- 10-Second Clips: The team achieved high-quality reconstructions of 10-second movies that the mice had never seen during the model’s training phase.
- Precision Matters: The accuracy of the “reconstructed movie” significantly improved as data from more individual neurons was added, proving the value of high-resolution cellular data.
- “The Feature, Not the Bug”: Lead author Dr. Joel Bauer notes that the deviations between the reconstructed video and reality aren’t errors; they represent the brain’s unique “skew” and interpretation of the world.
Source: UCL
Scientists have successfully reconstructed videos purely from the brain activity of mice, showing what the mice were seeing, in a new study led by University College London (UCL) researchers.
The findings, published in eLife, could help shed light on the intricate workings of how the brain processes visual information and open new avenues for exploring how different species perceive the world.
Over recent years, there has been a growing interest in understanding exactly how the human brain interprets signals from the eye. Images and movies have been played to people in fMRI machines and researchers around the world have tried to decode the brain’s representations of visual information on a pixel level.
The new study builds on this approach but instead used single-cell recordings in mice, which offer the potential to provide a more precise measure of the brain’s representations. This technique has enabled the team to create high-quality reconstructions of videos played to mice, based solely on the neural activity in the visual cortex.
Lead author Dr Joel Bauer (Sainsbury Wellcome Centre at UCL) said: “We wanted to have a better way of investigating how the brain interprets what we see. The current methods of understanding what specific groups of neurons are representing are not very generalisable to situations which haven’t been specifically tested for. And so, we wanted to develop a method that can capture what is being represented in the brain and compare that to reality.”
By looking for deviations between brain representations and reality, the new method could help researchers understand how specific visual cues shape neural representations.
Dr Bauer and colleagues used a dynamic neural encoding model, developed by another team for the 2023 Sensorium Competition, which predicts activity of individual neurons (brain cells) based on movies mice were being shown, while also accounting for the mouse’s own movements and pupil diameter.
Using the same dataset, the UCL team further refined this model by calculating the difference between the predicted activity of neurons if a mouse had seen a blank screen, and the actual activity of the neurons (measured using a microscopic imaging technique that detects which individual brain cells are firing based on localised boosts in calcium levels).
This allowed the scientists to gradually update the pixels of the blank movie through an algorithm and refine the model so that the output video closely resembled the video presented to the mouse.
Once the model was sufficiently trained, the researchers were then able to construct a 10-second movie based on a mouse’s neural activity alone, measured as the mouse watched a video that had not been used previously in training the model.
Dr Bauer added: “Using this approach, we were able to achieve high-quality reconstructions of 10-second video clips. The accuracy of the reconstructions improved with the inclusion of data from more individual neurons, demonstrating the importance of comprehensive neural data.”
To quantify the reliability of the reconstructions, the team used pixel correlation – correlating each pixel of the movie between the original version and the reconstructed version.
They found minimal differences in the timing of the two videos, but they plan to focus on improving the resolution and coverage of visual reconstructions. This will involve getting data that can give higher resolution reconstructions and larger coverage of the visual scene.
Next, the team plan to use the technique to uncover new insights into the brain’s visual processing capabilities. Specifically, they are interested in understanding how visual representations in the brain can deviate from what’s actually in front of our eyes.
Dr Bauer concluded: “We don’t have a perfect representation of the world in our heads. The visual processing pipeline skews and warps our representation in a way that modifies information.
“This deviation between reality and representations in the brain is not necessarily an error but a feature, reflecting how our minds interpret and augment sensory information. We want to explore how this happens in the brain.”
Key Questions Answered:
A: We’re not there yet, but this is a massive step in that direction! While this study focused on what mice see in real-time, the technology decodes how the brain represents visual information. Theoretically, the same principles could one day be used to visualize mental imagery, memories, or even dreams.
A: Mostly, but with a twist. The researchers found that the brain’s representation of a video is a bit “warped” compared to the actual pixels. Our brains (and a mouse’s) don’t just mirror the world; they highlight certain cues and ignore others. This “warping” is actually how the brain prioritizes important information.
A: Surprisingly accurate! By using pixel correlation (comparing the original video to the brain’s version), the team found the timing was nearly perfect. The more neurons they recorded, the clearer the picture became. It’s essentially a “neural livestream” of the mouse’s visual experience.
Editorial Notes:
- This article was edited by a Neuroscience News editor.
- Journal paper reviewed in full.
- Additional context added by our staff.
About this neuroscience and neurotech research news
Author: Chris Lane
Source: UCL
Contact: Chris Lane – UCL
Image: The image is credited to Neuroscience News
Original Research: Open access.
“Movie reconstruction from mouse visual cortex activity” by Joel Bauer, Troy W. Margrie, and Claudia Clopath. eLife
DOI:10.7554/eLife.105081.3
Abstract
Movie reconstruction from mouse visual cortex activity
The ability to reconstruct images represented by the brain has the potential to give us an intuitive understanding of what the brain sees. Reconstruction of visual input from human fMRI data has garnered significant attention in recent years.
Comparatively less focus has been directed towards vision reconstruction from single-cell recordings, despite its potential to provide a more direct measure of the information represented by the brain.
Here, we achieve high-quality reconstructions of natural movies presented to mice, from the activity of neurons in their visual cortex for the first time.
Using our method of video optimization via backpropagation through a state-of-the-art dynamic neural encoding model, we reliably reconstruct 10 s movies at 30 Hz from two-photon calcium imaging data.
We achieve a pixel-level correlation of 0.57 between ground-truth movies and single-trial reconstructions. Previous reconstructions based on awake mouse V1 neuronal responses to static images achieved a pixel-level correlation of 0.24 over a similar retinotopic area.
We find that critical for high-quality reconstructions are the number of neurons in the dataset and the use of model ensembling. This paves the way for movie reconstruction to be used as a tool to investigate a variety of visual processing phenomena.
