June 18, 2004
Insight into the Brain's Machinery of Value
New research with monkeys has yielded insight into the neural machinery that animals, and possibly humans, use to mentally represent the value of one action over another. Scientists at the Howard Hughes Medical Institute (HHMI) have determined the strategy by which the brain calculates the value of potential behavioral choices as those values change over time with new experience.
Their studies, said the cognitive neuroscientists who conducted the research, help in understanding the mysterious process by which animals and humans process sensory input to decide on actions that yield the greatest advantage.
“These monkeys had to make accurate probability estimates, and we found that the bits of math going on in their brains are quite impressive.”
William T. Newsome
The researchers, led by HHMI investigator William Newsome at the Stanford University School of Medicine, published their findings in the June 18, 2004, issue of the journal
“Researchers who study behavior regard decision-making as a critical link between the classic fields of the study of perception and the study of motor output,” said Newsome. “Sensory information comes into the brain, and somewhere that information is evaluated and decisions get made about what's out there. For example: Is it a predator? Is it a prey? Is it food or another object?”
In their studies, Newsome and co-authors Leo Sugrue and Greg Corrado set out to explore the machinery of value by asking rhesus monkeys to perform an eye-tracking task to get a fruit juice reward. Newsome said the researchers designed the task to probe deeply into the complex mechanism the brain uses to encode the value of choices.
The monkeys were given a choice of looking at either a red or green target to get a sip of juice. For each target, the experimenters changed the odds of receiving the reward at unannounced times. The monkey could receive the most sips of juice by adjusting its choices over time to match the changing frequency of rewards. Importantly, the most effective strategy required the monkeys to periodically `check out' the target with the lower odds to detect the unannounced changes in the reward rates. By analyzing the statistics of the choices over time, the researchers gained insight into how the monkeys tracked the targets' changing values, said Newsome.
“Psychologists and economists have known for a long time that decision-making is far more complex than just receiving sensory information from the environment and basing an action on it,” said Newsome. “You build up a history over time of what the likely outcomes of different actions are and build up these internal representations of value. These internal estimates of value—or the likelihood of acquiring your reward—influence decision-making as much as sensory evidence, and sometimes more so.”
For example, said Newsome, a fly fisherman will decide where to fish along his favorite stream based on a decision-making process that includes a complex judgment of immediate sensory input—like the weather and the water conditions—as well as a multitude of past experiences in fishing the stream.
After analyzing many hundreds of trials with the monkeys, Newsome and his colleagues found that the animals were, indeed, making sophisticated, flexible judgments about where to look to get the reward. They adapted to even subtle changes in the statistical likelihood that looking at a particular target would yield a sip of juice.
Not only did the monkeys rapidly adjust the frequency with which they looked at one target or the other as their values changed, but they “hedged their bets” by glancing at the less rewarding target in case it was becoming more rewarding, said Newsome.
“These monkeys had to make accurate probability estimates, and we found that the bits of math going on in their brains are quite impressive,” said Newsome. “It's not math the way humans would write equations. This is math that is built into the brain through evolutionary history, presumably from calculating the odds of finding food or water or a mate at different places in the wild.”
One surprise, said Newsome, was that the monkeys relied on information that was more `local' than studies by other researchers had suggested. Rather than adding a new experience to the entire history of experience with the task, the monkeys tended to give more value to the last seven or eight trials, with the influence of earlier trials dropping off rapidly beyond that.
Such a local valuation reveals an important tradeoff, said Newsome. If the monkeys used a longer history, they might be more accurate, but they would respond sluggishly to changes. On the other hand, if they used only very recent information, they would respond more quickly to change, but would make more mistakes when the choice system was stable.
The researchers found the same localization phenomenon when they developed and ran a mathematical simulation of the monkeys' choice behavior.
Newsome and his colleagues also used recording electrodes during the behavioral trials to explore whether such decision-making was taking place in the area of the brain called the parietal cortex, which contributes to decisions about where to move the eyes. Their electrophysiological studies revealed patterns of neuronal activity corresponding to the value estimates the monkeys indicated by their target choices. This finding provided an initial clue about how and where value is represented in the brain, and how value signals might influence decision-making.
However, said Newsome, analysis of parietal cortex activity in the brief interval between behavioral trials suggested that the brain did not store value information in that area. Rather, the value appeared to be “downloaded” at the start of each trial from some other part of the brain. Newsome and his colleagues plan further studies, including magnetic resonance imaging studies, to pinpoint these regions.
Such a combination of behavioral and neurophysiological studies and mathematical modeling offers an important new route to insight into the neural machinery of valuation and decision-making, concluded Newsome. Also, he emphasized, such studies depend on the decades of studies of both sensory and motor systems in the brain.
“This whole area of decision-making studies is exploding right now, largely because for the first time in history we know enough about the sensory and the motor ends of the system to put ourselves in position to ask more reasonable questions about the key intermediate stage of decision-making,” he said.