illustration by Ping Zhu

JAABA: Automating the Human Observer

This software can learn to ID animal behavior.

Even the simplest experiments must be reproducible to be convincing. And as new variables are introduced—different genes, conditions, or time points to be tested—a study’s size can become unwieldy. Automated technologies can take over some of lab work’s more tedious tasks, but when it comes time to make sense of the data, a scientist’s intuition is irreplaceable. Right?

Not always, says computer scientist Kristin Branson, a lab head at HHMI’s Janelia Farm Research Campus. Branson agrees that intuition will probably always be critical in the quest for scientific truth. But for certain tasks, she says, computers can be trained to re-create researchers’ gut instincts. Her new machine learning software, which she calls JAABA for Janelia Automatic Animal Behavior Annotator, is a tool for identifying characteristic behaviors captured on video. Reviewing hours of recorded activity, the software can zero in on, for example, each instance of one fruit fly chasing another.

Biologists study behavior in model organisms to answer many questions—to help reveal, for example, how the brain works, how genes drive behavior, or how learning changes it. Behavioral studies can offer insights into developmental biology or clues about the evolutionary past. Analyzing animal behavior requires careful observation, and while many behavior patterns are well characterized and easy to recognize, the task can become daunting when a study involves hundreds or thousands of individuals. Searching for an organism that exhibits subtle changes in behavior—such as a fly that walks 20 percent slower than others because of a genetic mutation—increases the challenge.

Technology advances have made large-scale studies more common, and even in smaller labs behavioral scientists commonly collect video containing more data than they could ever hope to analyze, Branson says. “It’s cheap and easy to collect a lot of video. It’s not that easy to say anything quantitative about it,” she points out.

When Branson moved to Janelia in 2010, she brought along the machine vision software she had developed as a postdoctoral researcher at the California Institute of Technology. That software identifies and tracks the movements of fruit flies on video, churning out data about animals’ speed and position. Branson had planned to work with Janelia biologists to refine the program and make it more useful for their studies. “But Janelia is a special place. People here love technology,” she says. “When I got here, there were already four different tracking programs in use.”

So Branson switched her focus. Instead of extracting the simple quantitative information that tracking programs derive from videos, her next program went further. With JAABA, she says, “our goal is to automate the human.”

Even when experienced biologists can readily recognize a behavior, the precise qualities that define that behavior might be difficult to articulate. How fast does a fruit fly move when it gives chase? How close does it have to be to another fly for it to be considered chasing? At what point does walking transition to chasing? Biologists may not have explicit answers to these questions, but somehow, Branson says, experimenters know what they’re looking for. To automate behavior identification, biologists must transfer that knowledge to the computer.

Scientists teach JAABA to recognize behaviors through short training sessions with rapid feedback. For each frame of a training video, the user tells the computer whether a particular behavior, such as walking or wing grooming, is taking place. JAABA computes key features from the labeled frames—for example, an animal’s speed and its distance from other animals—and combines that information with context from surrounding frames to develop its own criteria for detecting the behavior. The user can test JAABA’s accuracy at any time during the training process.

Watch as a user begins to train JAABA.

That immediate feedback helps users understand how the software is performing and recognize areas in which JAABA requires additional training, Branson says. The software can be trained to recognize behaviors in several animals, including adult fruit flies, fruit fly larvae, and mice—even by a user with no background in computer science, according to a paper her team published in Nature Methods in January 2013. They’ve made JAABA freely available for download at, and she hopes researchers will use it to free themselves from manual analysis.

Eventually, Branson says, JAABA might do something more ambitious. As they train the software to recognize more behaviors, her team may be able to mine the data they collect to go beyond human capabilities and find measurable features of behavior that are not apparent to the human eye.