When not performing weapons simulation duties,
Roadrunner will be tapped for unclassified research inastronomy, energy, human genomics, nanoelectronics, and climatology.
An IBM application team hasalready achieved a petaflop (using single precision floating point) on a simulation code for the human brain.Some of the initial testing planned for early 2009 at Los Alamos will involve a number of open science codes.These include applications that simulate the molecular breakdown of cellulose for biofuels, supernova lightcurves, 3D magnetic reconnection in plasma physics, and time evolution of metallic nanowires.
Roadrunner represents a unique architecture
that combines AMD dual-core Opteron processors with thenew souped-up IBM Cell (PowerXCell 8i) processors. It is the Cell processors that are doing most of the heavylifting though. The 6000+ Opterons in the compute blades contribute only 44 teraflops, while the 12,000+ Cellchips contribute 1,332 teraflops. Both numbers reflect peak performance. The sustained Linpack result is 1,026teraflops, or just over one petaflop.Drilling down a little, Roadrunner is made up of 17 "connect units (CUs)," each of which is a collection of 180compute nodes. Each CU sports a 288-port InfiniBand DDR switch that routes 55 miles of optical interconnectsthroughout the system. A compute node is a "TriBlade," consisting of a single 2-socket dual-core Opteron LS21 blade connected to two dual-socket QS22 Cell blades. Internally, each Opteron core is connected to one Cellchip over a dedicated PCIe link. While the node-to-node communication for the compute units is all InfiniBand,the machine employs 10GbE to talk to 2 petabytes of external storage, which is supplied by Panasas.Because most of the compute power relies on the high-performance Cell processor, the system is quite energyefficient. According to IBM, Roadrunner draws 3.9 megawatts, and delivers 376 megaflops/watt, besting eventhe PowerPC-based Blue Gene/P metric of around 350 megaflops/watt. For comparison, the most energy-efficient Xeon-based supercomputer clusters deliver only about 150 megaflops/watt.Despite the exotic hardware design, a lot of the effort for the project went into getting all the software in placeto make application porting and development easy. Chief IBM Roadrunner engineer, Don Grice believes thatmulticore/manycore and heterogeneous computing is "the wave of the future," at least for the next 10 years or so. But, he says, the key to unleashing this power will be developing software that is able to tap into all this processing performance.IBM uses its internally-developed SDK as well as open source software for the Roadrunner application platform. The software model is based on standard MPI, where each MPI task makes use of one Opteron coreand a Cell processor. Custom MPI implementations could presumably change that mix, depending upon theneeds of specific workloads.
IBM's SDK DaCS library
provides the low-level glue between the Cell and theOpteron pieces, while at the outer level,
Red Hat Linux and xCAT cluster management
supplies theapplication's operating environment.The ability to optimize memory flow across the system will be the critical factor in unleashing the performancefrom these hybrid machines. "This feels very similar to the change we made when we went from sharedmemory to distributed memory...," observed Grice. "Now we have to figure out how to get around this memory bandwidth wall and heterogeneous cores."Grice admits that the software model they have constructed is just a start for making hybrid systems, likeRoadrunner, easily programmable. When you combine multiple computing technologies (i.e., heterogeneousinstruction sets, multicore processors, vector SIMD units, local memory stores, explicit DMA, on-chipCPU/memory networks, remote accelerators and cluster computing) the developer is going to need a framework that provides some level of hardware independence. For the first cut at this, IBM decided to go the library routeas a relatively easy way to glue together the different binaries and help take the complexity out of theheterogeneity. Later versions could involve new programming languages and compiler/runtime technologies.