Janelia’s IT team, from left to right: Vijay Samalam, Spartaco Cicerchia, and Goran Ceric
Computing Cluster Attains Superpower Status
For most of us, upgrading from a Porsche to a Ferrari would be an act of pure decadence. But for the information technology (IT) professionals at HHMI’s Janelia Farm Research Campus in Ashburn, VA, a computing cluster upgrade—from powerful to state of the art—became sheer necessity as more and more scientists joined the neuroscience hub, stressing the existing infrastructure.
The rebuilt computing cluster, installed this spring, is an IT high-performance vehicle—blazingly fast and ultra-responsive. And even better than a Ferrari, it offers plenty of storage.
Over the four years since the campus opened, Janelia researchers have increasingly come to better understand their IT needs, says Vijay Samalam, Janelia’s director of scientific computing and information technology. “The scientists have had time to establish their labs, their instruments have come online, and the various computer usage scenarios have become very clear,” he says. “We felt this year was the perfect time to perform a refresh,” says Samalam. “We now know what our users want and need.”
Filling 14 tall floor-to-ceiling racks deep inside Janelia, the upgraded supercomputer contains more than 4,000 processors—each equivalent to a fast desktop computer. The processors can sync up for huge jobs or work independently on smaller tasks. Capable of 36 trillion calculations per second, Janelia’s cluster in 2010 was ranked number 208 on a list of the world’s 500 fastest supercomputers. Samalam says it’s one of the fastest devoted to biology.
The new cluster is also much more energy efficient than its predecessor, requiring less electricity to run and less air conditioning to keep cool.
Along with clocking lightning speeds, the cluster is also hair-trigger responsive. The previous cluster operated with an impressive lag time of 60 microseconds—that’s how long it took each processor to communicate with another processor. With the new networking regime, big processing jobs hum along even faster because the lag time has dropped to 10 microseconds—a sixfold improvement.
Samalam and colleagues Spartaco Cicerchia and Goran Ceric designed the cluster with off-the-shelf hardware from Intel, Dell, and Arista Networks. The team decided to link the humble components together in a unique way, that is, via fast, 10-gigabit Ethernet rather than the more expensive, finicky networking protocol that supercomputers around the world typically run.
“We went out on a limb a little when we chose this system,” Samalam says of the Ethernet backbone. “The secret sauce for any supercomputer is how quickly all the processors can work together without getting in each other’s way. Making the right networking choice was crucial.”
The results have been spectacular, Samalam says. The Ethernet backbone delivers high reliability with low maintenance needs at just a fraction of the cost of a traditional supercomputer network.
“It’s a working-class supercomputer,” says Janelia researcher Sean Eddy. He would know: Eddy’s genomic data-mining software puts the cluster through its paces every day. Software developed by Eddy and his colleagues sifts through billions of letters of DNA across various species in search of similar sequences.
Samalam says his team designed the cluster to excel at such data mining. It is optimized, too, for a second critical application: image processing and storage. As a neuroscience hub, Janelia’s laboratories are packed with all manner of optical and electron microscopes and other imaging equipment.
One laboratory, headed by Anthony Leonardo, even boasts a high-speed camera capable of snapping a million frames per second. The flood of imaging data from these instruments pours through Janelia’s 450 miles of optical fiber cable into the computing cluster and then onto the center’s huge storage rack, which houses stacks of hard drives. Total storage capacity is the equivalent of the content in about 50 Libraries of Congress.
As terabytes of imaging data arrive at the cluster, the processors begin the taxing job of analyzing them. Electron microscopes produce high-density images that need to be analyzed, sorted, and linked to other images to build colorful 3-D structures out of grayscale 2-D layers.
For instance, a Janelia initiative to map every neuron in the brain of the fruit fly Drosophila melanogaster relies heavily on the cluster. The entire map contains some 4 million individual images. Aligning all of those images, so that neighboring neurons in the fly’s brain appear as neighboring neurons on a computer monitor, would take a desktop computer about four years. Janelia’s cluster finished the job in a few hours.
“The feedback from users has been uniformly extremely high,” says Samalam. “I can say with confidence that the new computing cluster is increasing the pace of research.