Rather than scrutinizing hours of video, scientists can quickly teach the software how to recognize key behaviors.

Using new software developed at HHMI’s Janelia Farm Research Campus, a computer can be trained to recognize characteristic animal behaviors like an experienced biologist. Rather than scrutinizing hours of video to catalog how often fruit flies chase one another, for example, scientists can quickly teach the software what to look for, and let it convert the video into useful statistics.

With affordable technology for capturing video and new research tools that enable large-scale studies, some behavioral scientists are finding themselves with more data than they could ever hope to analyze. “We need automated techniques to keep up with the technology,” says Janelia fellow Kristin Branson, who led the development of the new software. Branson’s team developed the Janelia Automatic Animal Behavior Annotator, or JAABA, to automate and standardize the essential but time-consuming task of classifying animal behavior.

Branson’s team described JAABA in the December 2, 2012, issue of the journal Nature Methods. They have shown the software can be trained to recognize behaviors in a variety of animals, including adult fruit flies, fruit fly larvae, and mice. The software is freely available for download at and can be used by biologists with no computer science or machine learning expertise, Branson says.

This video shows a screencast of a user beginning to train a fly wing-grooming classifier. In this video, the user labels one bout of frames in which the fly is wing grooming and one bout of frames in which the fly is not wing grooming. The user leaves a gap between the end of one of these bouts and the beginning of the next, as it is difficult to consistently label these frames. The user then trains an initial classifier from this first set of labels, a process that takes a few seconds. The user examines the resulting classifier's predictions on frames not in the training data. This initial classifier makes several mistakes, and the user labels a third bout to correct some mistakes. The user then retrains a new classifier and again examines the predictions. After just three bouts of training labels, the classifier is making reasonable predictions on data outside the training set.

Branson says her own lab at Janelia collects about 100 hours of fruit fly videos each week, with 20 flies moving about in each video. The flies in the videos have been genetically modified so that researchers can switch on small subsets of neurons, and Branson and her colleagues would like to determine how that genetic tweak affects the flies’ behavior. But the detailed quantitative analysis necessary to detect subtle differences in behavior are daunting.

“It’s cheap and easy to collect a lot of video,” she says. “It’s not that easy to say anything quantitative about it.” Branson and many of her colleagues use computer vision software she and others have developed to track animals’ movements in the videos, generating information about the animals’ trajectories and positions in each frame. “This gives us some very simple information, like speed or how long an animal spends in a certain part of its environment,” she says. “But we knew there was much richer information there.”

To make that information more accessible, Branson set to work writing a program that could be trained to recognize characteristic behaviors the same way experienced scientists do. The result was JAABA.

To use JAABA, scientists begin by training the software. For each frame of a training video, the user tells the computer whether a particular behavior—such as walking or wing grooming—is taking place. Using features that can be computed from just a few frames, such as an animal’s speed or distance to other animals, combined with context gleaned from surrounding frames of the video, JAABA develops criteria for detecting the behavior.

Branson says the software learns quickly, producing behavior detectors based on training data in just 15 to 45 seconds. That means users can test its ability to predict a behavior at any time during the training process. The immediate feedback can help users understand how the software is performing and recognize areas in which it requires additional training, she says. “Most biologists don’t have a background in machine learning, so having an interactive framework where you can see what the classifier can and can’t do gives them some comfort interacting with these complicated algorithms,” she explains.

Branson says her team made the first prototype of JAABA within a few weeks. Then they spent the next two years ensuring the program would work consistently on diverse data, and converting their basic algorithms into software that would be accessible to biologists without computer science expertise. “The basic idea was quite simple,” she says, “but what took a lot of time was making it very robust and usable.”

Once Branson and her colleagues had made the program more intuitive, they gathered 12 fruit fly researchers into a conference room, loaded JAABA onto their laptops, and gave them a 15-minute introduction to the software. The users were able to train JAABA to identify chasing behavior in fruit flies with greater than 97 percent accuracy.

JAABA not only frees scientists from manually identifying behaviors, it also standardizes the definition of those behaviors. The same video might be analyzed slightly differently by two different scientists, Branson explains, especially at points where an animal is transitioning between behaviors, or with individuals who behave less stereotypically than others. Automating the analysis helps ensure consistency, she says.

Branson adds that while certain behavioral patterns—such as those associated with courtship and locomotion—are well defined in model organisms like mice and fruit flies, there are many unstudied behaviors that could be informative to scientists. Eventually, she hopes that machine learning programs like JAABA will help scientists define what constitutes a behavior and discover new ones.

Scientist Profiles

For More Information

Jim Keeley 301.215.8858