Home › Research › Neural Mechanisms Underlying Visually Based Cognition

Our Scientists

Neural Mechanisms Underlying Visually Based Cognition

Research Summary

William Newsome's research focuses on the neural mechanisms underlying visually based decision making and related issues in cognitive neuroscience. He seeks to understand how higher mammals acquire sensory information about the world, how that information is processed within the brain, and how behavioral responses to that information are organized. New experimental and conceptual tools are creating a paradigm shift in the field. The analysis of single-neuron activity remains critical, but single-neuron activity is most comprehensible in the context of collective neural states that are revealed through dynamical systems analysis.

My lab is interested in fundamental questions concerning the neural basis of cognition: How is information encoded in the streams of action potentials emitted by single neurons? How does the activity of large populations of neurons represent the external world and internal states of the organism? How do the high-level behavioral goals of an organism influence the low-level processing of sensory information? What are the neural mechanisms underlying the formation of decisions, the key cognitive processes that link perception to action?

As a field, we are in the early stages of a paradigm shift in how such questions are addressed. For decades we have faced two fundamental barriers to progress. First, although it is a truism that cognitive processes must depend upon the coordinated activity of large populations of neurons, the traditional neurophysiological approach is to record from one cell at a time during repeated behavioral sessions, hoping to stitch together post hoc a picture of population activity. This simple approach has been more successful than we had any right to expect, but it is fundamentally incapable of capturing complex interactions among large populations of cells in real time. Second, single neurons are noisy signaling devices. Thus, neurophysiologists typically average electrophysiological data across multiple trials, time locked to some measurable external event (onset of a stimulus or execution of a movement) to reveal the "signals" carried by a particular neuron. Most cognitive states, however, are not precisely time locked to external events, nor are they repeated with sufficient frequency and reliability to be readily detected by averaging techniques. To get to the next level, therefore, we must devise new experimental and analytic approaches that allow detection and dynamic tracking of neural population states during spontaneous, internally generated cognitive processes.

Exciting new developments, both technical and conceptual, are now enabling us to make novel advances in detecting and analyzing higher organizational states of the nervous system. We and other labs are using multi-electrode arrays to obtain simultaneous recordings from hundreds of neurons in behaving animals, providing the essential rich substrate for dynamical analysis of neural populations. Critically, we are at the same time devising new data analysis techniques whose main thrust is to discover low-dimensional axes of variation in neural population activity that are relevant to the ongoing behavior of the animal. Finally, we are analyzing the resulting low-dimensional data in the conceptual framework of dynamical systems.

Figure 1: State-space representation of neural population activity measured while a rhesus monkey performs a contextually sensitive decision-making task. The monkey views random dot patterns (presented on a video monitor) that have two properties, motion and color, that are varied parametrically in strength and independently of each other across trials. The animal discriminates the direction of motion and ignores the color in one behavioral context, and does the opposite in the other context. The animal reveals its choice at the end of each trial via an eye movement.

The responses of 213 prefrontal cortex (PFC) neurons were measured, which creates a 213-dimensional space for the state of the neural system considered as a whole. Dimensionality reduction techniques reveal, however, that the population response varies primarily along a small number of axes in the 213-dimensional space that code behavioral variables that are significant to the animal. Movement of the system in this low-dimensional space is illustrated in the six trajectories illustrated in the figure, one trajectory for each of three motion strengths in two directions—leftward and rightward. In all six conditions, the neural population begins initially at neutral state (on) but as the monkey examines the visual stimulus, the neural population migrates through a series of states (individual data points) toward one of two final states corresponding to the two possible decisions (choice left or choice right).

These neural trajectories reveal two variables that are coded at the population level. Movement of the system along the choice axis (red axis, lower right) represents the slow accumulation of visual information toward one or the other decision. In contrast, the pronounced arcs away from the choice axis reflect movement of the neural population along the motion axis (black axis, lower right), which reveals the existence of a momentary motion signal that is not integrated toward a choice. That this signal reflects motion strength is evident from the systematic ordering of the trajectories with respect to the strength and direction of the motion signal (shades of gray). Different views of the state space (not shown) reveal color and context signals in the same neural population. These signals, though profoundly mixed at the single-neuron level, are nicely separable at the population level, as illustrated in the state-space plot.

Image: Valerio Mante, former HHMI research associate in the Newsome lab.

The central idea underlying dynamical systems analysis is that the behavior of complex systems is best described and understood in terms of high-level states and transitions between states (dynamics) whose relationship to the behavior of individual components may be very indirect. Critically, the high-level states of the system are not directly observable, but are latent and must therefore be inferred from the coordinated activity of the components. For example, to reverse engineer a digital computer, one would of necessity measure dynamic patterns of signals across many transistors. Ultimate success, however, would involve inference of the succinct, unobserved rules—the software—that govern the behavior of the system. Physical scientists and engineers have long employed the tools of dynamical systems analysis to study the behavior of complex physical systems, and a few pioneers have applied this approach to the nervous system as well (Gilles Laurent [Max Planck Institute for Brain Research] and Krishna Shenoy [Stanford University] have been inspirational to me personally). But systematic analysis has not been possible without new experimental techniques for population recording and new analytic techniques for effective dimensionality reduction.

As a first step toward dynamical systems analysis of cognition, we have tackled the problem of contextually sensitive decision making. If I am in my house and the phone rings, I answer it; if the phone rings in your house, however, I do not answer it. The same stimulus enters the nervous system, but different decisions and motor outputs ensue. The key variable that "gates" the flow of information from input to output is behavioral context.

To investigate the neural mechanisms underlying context sensitivity and gating of information flow within the cortex, we devised a context-sensitive decision-making task that is performed by animals in the laboratory. Frontal cortex recordings during this task reveal a bewildering mixture of signals at the single-neuron level: signals related to the animal's choice, the behavioral context, and different sensory inputs are expressed in nearly random proportions from neuron to neuron. Remarkably, however, orderly representation emerges at the level of the neural population when analyzed within a dynamical systems framework. Furthermore, a new computational model reproduces key aspects of the neural data and suggests a novel dynamical mechanism for gating and integration of information within the cortex. We are formulating predictions of the model that can be tested in future experiments.

This type of research program is a new and promising way forward for systems and cognitive neuroscience. Using dynamical systems analysis, we can detect high-level states and system variables that govern the behavior of the entire system, thus rendering the activity of individual components of the system comprehensible. As a traditional single-neuron physiologist, I find myself in a curious but most interesting situation. The physiological responses of single neurons remain critical—single-neuron activity is the necessary substrate on which dynamical systems analysis is based. Yet the meaning of single-neuron activity is revealed most beautifully within the collective context of the population and the high-level rules that drive population activity. Though it is too early to know for sure, I suspect that this general approach to understanding the neural mechanisms of cognition (pioneered by others) will be extraordinarily productive in the future—sufficiently so to comprise a paradigm shift for the field.

As of March 26, 2013

Scientist Profile

Investigator
William T. Newsome, PhD
Stanford University
Neuroscience

Related Links

The Newsome Lab
The Newsome Faculty Page
View Research Papers on PubMed