Summary: A new study implicates the dorsolateral prefrontal cortex in our ability to assign credit for whatever action leads to a desired outcome. While the study does not rule out the involvement of other brain areas to credit assignment, it does show the dlPFC is a key player in how we assess causality.
Source: Brown University.
When you do something right, you can’t learn anything from your success without a system in the brain for assigning credit to whatever action led to the desired outcome.
Say, for instance, you’ve forgotten which of your 10 usual passwords logs you into a favorite website. When you finally enter the right one, your brain should have a mechanism for noting which one led to that success. You have such a mechanism, of course, and now a study in non-human primates is the first to directly pinpoint it at work in the dorsolateral prefrontal cortex.
The new findings in the Journal of Neuroscience not only add insight into how the brain works, but also could lead to improvements in the care of patients who suffer traumatic brain injuries that affect the area, said lead author Dr. Wael Asaad, an assistant professor of neurosurgery at the Warren Alpert Medical School of Brown University and a surgeon at Rhode Island Hospital.
Positioned on the brain’s surface behind the top of the forehead, the dorsolateral prefrontal cortex (dPFC) is in a vulnerable place. The long-term impairments that can result from the traumatic brain injuries he treats in the operating room are what motivate Asaad’s research, he says.
“The frontal lobes are sites where you often find traumatic hemorrhages that lead to all kinds of problems in these people’s lives in the future, and many of those problems have to do with poor decision making,” said Asaad, who is affiliated with the Brown Institute for Brain Science. “Our hypothesis was that this part of the brain is responsible for trying to figure out why things happen the way they did and linking causes and effects so that you can make better choices in the future.”
A simple game
The research team gathered its data more than six years ago at Massachusetts General Hospital. Two rhesus macaques performed a simple experimental game: figure out which image among four presented on a screen was the “correct” one to earn a rewarding sip of juice. The task was an exercise in credit assignment because to consistently get the juice, the subjects needed to recognize which image was the one that had led to the reward and would continue to in the future. In a control experiment, the researchers rewarded the subjects not for choosing a correct image, but instead for choosing a correct corner of the grid.
As the macaques played the simple game thousands of times (the correct image or location would change every so often to produce the need for a new credit assignment), the researchers recorded the electrical spiking activity of hundreds of their individual neurons in the dPFC, which neuroscientists have debated as perhaps playing a role in credit assignment.
If some dPFC neurons have that function, Asaad and his colleagues reasoned, then they should have to behave in four specific ways during the task.
“If you lay out a list of criteria that the neural activity would need to conform to in order to solve credit assignment, these neurons fulfill all those criteria,” Asaad said.
First, for neurons to associate a cause (the correct image) with a desired effect (juice reward and green circle on the screen), their spiking activity should simultaneously represent the image and the successful outcome at the time the positive feedback was delivered. Using software called a decoder, researchers were able to isolate the unique neural spiking patterns associated with each image and with positive or negative feedback, the latter being simply the absence of juice and a red X symbol on the screen. Using this technique, they were able to see that the representation of the chosen image persisted in many neurons through the time when those same neurons also responded to the positive feedback, producing the required simultaneous representation.
Another requirement was that neurons represent the correct image consistently over time, specifically when the picture was visible and when the outcome was revealed, even though by that time the picture had disappeared to make way for the green circle or the red X. Indeed, neurons that represented the images (via unique spiking patterns) did so consistently throughout the task, meaning the same neural code used to represent that picture could link cause and effect.
Because credit assignment is a learning process, Asaad noted, there should be a greater degree and fidelity of neural activity across time when the learning was occurring than when it was well established and merely being reapplied. Again, the researchers saw just such a pattern. Early in each new credit assignment exercise, neural fidelity associated with the correct image and rewarding result was greater than it was when the macaques had become accustomed to selecting same correct image simply by habit.
Finally, if the neurons were truly assigning credit for the reward to the correct cause, then in the control experiment where location was the cause rather than image, the representation of the now-irrelevant image shouldn’t still appear at the time of feedback. Again that proved to be the case. While some neurons still showed activity when images were first presented, that increased activity did not reoccur simultaneously with reward when the right answer was grid position rather than image.
“Together, these results are consistent with the notion that neurons in the dPFC provide the necessary, selective and stable representation of relevant features at the time of feedback to enable credit assignment,” the researchers wrote in the journal.
The study does not rule out that other brain regions are also involved, Asaad noted. Instead, credit assignment almost certainly does involve other regions. But the new evidence shows that the dPFC is a key player.
In addition to Asaad, the paper’s other authors are Peter Lauro, Janos Perge and Emad Eskandar.
Funding: Funding for the study came from the National Institutes of Health (P20GM103645, 1R01EY017658, 1R01NS063249), the National Science Foundation (IOB0645886), a Tosteson Fellowship, a Neurosurgery Research and Education Foundation Young Clinician Investigator Award, and the Howard Hughes Medical Institute.
Source: David Orenstein – Brown University
Image Source: NeuroscienceNews.com image is credited to Asaad et. al.
Original Research: Abstract for “Prefrontal neurons encode a solution to the credit assignment problem” by Wael F. Asaad, Peter M. Lauro, János A. Perge and Emad N. Eskandar in Journal of Neuroscience. Published online June 20 2017 doi:10.1523/JNEUROSCI.3311-16.2017
Prefrontal neurons encode a solution to the credit assignment problem
To adapt successfully to our environments, we must use the outcomes of our choices to guide future behavior. Critically, we must be able to correctly assign credit for any particular outcome to the causal features which preceded it. In some cases, the causal features may be immediately evident, whereas in others they may be separated in time or intermingled with irrelevant environmental stimuli, creating a potentially nontrivial credit assignment problem. We examined the neuronal representation of information relevant for credit assignment in the dorsolateral prefrontal cortex (dPFC) of two male rhesus macaques performing a task that elicited key aspects of this problem. We found that neurons conveyed the information necessary for credit assignment. Specifically, neuronal activity reflected both the relevant cues and outcomes at the time of feedback and did so in a manner that was stable over time, in contrast to prior reports of representational instability in the dPFC. Furthermore, these representations were most stable early in learning, when credit assignment was most needed. When the same features were not needed for credit assignment, these neuronal representations were much weaker or absent. These results demonstrate that the activity of dPFC neurons conforms to the basic requirements of a system that performs credit assignment, and that spiking activity can serve as a stable mechanism that links causes and effects.
Credit assignment is the process by which we infer the causes of our successes and failures. We found that neuronal activity in the dorsolateral prefrontal cortex conveyed the necessary information for performing credit assignment. Importantly, while there are various potential mechanisms to retain a “trace” of the causal events over time, we observed that spiking activity was sufficiently stable to act as the link between causes and effects, in contrast to prior reports that suggested spiking representations were unstable over time. In addition, we observed that this stability varied as a function of learning, such that the neural code was more reliable over time during early learning, when it was most needed.
“Prefrontal neurons encode a solution to the credit assignment problem” by Wael F. Asaad, Peter M. Lauro, János A. Perge and Emad N. Eskandar in Journal of Neuroscience. Published online June 20 2017 doi:10.1523/JNEUROSCI.3311-16.2017