The researchers have shown mathematically how a network of neurons, when connected appropriately, can identify the best decision in a given situation and its future cumulative reward. Image is adapted from University of Cambridge press release. Credit: Seung Lab.
Mathematically Modelling How The Brain Makes Complex Decisions
Researchers have built the first biologically realistic mathematical model of how the brain plans and learns when faced with a complex decision-making process.
Researchers have constructed the first comprehensive model of how neurons in the brain behave when faced with a complex decision-making process, and how they adapt and learn from mistakes.
The mathematical model, developed by researchers from the University of Cambridge, is the first biologically realistic account of the process, and is able to predict not only behaviour, but also neural activity. The results, reported in The Journal of Neuroscience, could aid in the understanding of conditions from obsessive compulsive disorder and addiction to Parkinson’s disease.
The model was compared to experimental data for a wide-ranging set of tasks, from simple binary choices to multistep sequential decision making. It accurately captures behavioural choice probabilities and predicts choice reversal in an experiment, a hallmark of complex decision making.
Our decisions may provide immediate gratification, but they can also have far-reaching consequences, which in turn depend on several other actions we have already made or will make in the future. The trouble that most of us have is how to take the potential long-term effects of a particular decision into account, so that we make the best choice.
There are two main types of decisions: habit-based and goal-based. An example of a habit-based decision would be a daily commute, which is generally the same every day. Just as certain websites are cached on a computer so that they load faster the next time they are visited, habits are formed by ‘caching’ certain behaviours so that they become virtually automatic.
An example of a goal-based decision would be a traffic accident or road closure on that same commute, forcing the adoption of a different route.
“A goal-based decision is much more complicated from a neurological point of view, because there are so many more variables – it involves exploring a branching set of possible future situations,” said the paper’s first author Dr Johannes Friedrich of Columbia University, who conducted the work while a postdoctoral researcher in Cambridge’s Department of Engineering. “If you think about a detour on your daily commute, you need to make a separate decision each time you reach an intersection.”
Habit-based decisions have been thoroughly studied by neuroscientists and are fairly well-understood in terms of how they work at a neural level. The mechanisms behind goal-based decisions however, remain elusive.
Now, Friedrich and Dr Máté Lengyel, also from Cambridge’s Department of Engineering, have built a biologically realistic solution to this computational problem. The researchers have shown mathematically how a network of neurons, when connected appropriately, can identify the best decision in a given situation and its future cumulative reward.
“Constructing these sorts of models is difficult because the model has to plan for all possible decisions at any given point in the process, and computations have to be performed in a biologically plausible manner,” said Friedrich. “But it’s an important part of figuring out how the brain works, since the ability to make decisions is such a core competence for both humans and animals.”
The researchers also found that for making a goal-based decision, the synapses which connect the neurons together need to ‘embed’ the knowledge of how situations follow on from each other, depending on the actions that are chosen, and how they result in immediate reward.
Crucially, they were also able to show in the same model how synapses can adapt and re-shape themselves depending on what did or didn’t work previously, in the same way that it has been observed in human and animal subjects.
“By combining planning and learning into one coherent model, we’ve made what is probably the most comprehensive model of complex decision-making to date,” said Friedrich. “What I also find exciting is that figuring out how the brain may be doing it has already suggested us new algorithms that could be used in computers to solve similar tasks,” added Lengyel.
The model could be used to aid in the understanding of a range of conditions. For instance, there is evidence for selective impairment in goal-directed behavioural control in patients with obsessive compulsive disorder, which forces them to rely instead on habits. Deep understanding of the underlying neural processes is important as impaired decision making has also been linked to suicide attempts, addiction and Parkinson’s disease.
Goal-Directed Decision Making with Spiking Neurons
Behavioral and neuroscientific data on reward-based decision making point to a fundamental distinction between habitual and goal-directed action selection. The formation of habits, which requires simple updating of cached values, has been studied in great detail, and the reward prediction error theory of dopamine function has enjoyed prominent success in accounting for its neural bases. In contrast, the neural circuit mechanisms of goal-directed decision making, requiring extended iterative computations to estimate values online, are still unknown. Here we present a spiking neural network that provably solves the difficult online value estimation problem underlying goal-directed decision making in a near-optimal way and reproduces behavioral as well as neurophysiological experimental data on tasks ranging from simple binary choice to sequential decision making. Our model uses local plasticity rules to learn the synaptic weights of a simple neural network to achieve optimal performance and solves one-step decision-making tasks, commonly considered in neuroeconomics, as well as more challenging sequential decision-making tasks within 1 s. These decision times, and their parametric dependence on task parameters, as well as the final choice probabilities match behavioral data, whereas the evolution of neural activities in the network closely mimics neural responses recorded in frontal cortices during the execution of such tasks. Our theory provides a principled framework to understand the neural underpinning of goal-directed decision making and makes novel predictions for sequential decision-making tasks with multiple rewards.
SIGNIFICANCE STATEMENT Goal-directed actions requiring prospective planning pervade decision making, but their circuit-level mechanisms remain elusive. We show how a model circuit of biologically realistic spiking neurons can solve this computationally challenging problem in a novel way. The synaptic weights of our network can be learned using local plasticity rules such that its dynamics devise a near-optimal plan of action. By systematically comparing our model results to experimental data, we show that it reproduces behavioral decision times and choice probabilities as well as neural responses in a rich set of tasks. Our results thus offer the first biologically realistic account for complex goal-directed decision making at a computational, algorithmic, and implementational level.
“Goal-Directed Decision Making with Spiking Neurons” by Johannes Friedrich and Máté Lengyel in Journal of Neuroscience. Published online February 3 2016 doi:10.1523/JNEUROSCI.2854-15.2016