5/24/2013

InfiniBand & Firewire

Assignment 4

Saud ul Khalid
ROLL # 256

Contents
................................................................................................................................................ 2 ......................................................................................................................................... 2 ......................................................................................................................................... 2 ................................................................................................................................ 3 ................................................................................................................................................ 3 ............................................................................................................................................. 4 ..................................................................................................................... 4 ............................................................................................................. 5 .................................................................................................................................................... 7 ......................................................................................................................................... 7 ............................................................................................................................. 8 1934 standard requirements: ......................................................................................................... 8 ....................................................................................................................................... 8 .................................................................................................................... 9 ........................................................................................................................................... 9 .................................................................................................................................... 9 ............................................................................................................................... 10

InfiniBand is a type of communications link for data flow between processors and I/O devices that offers throughput of up to 2.5 gigabytes per second and support for up to 64,000 addressable devices.

The InfiniBand architecture specification defines a connection between processor nodes and high performance I/O nodes such as storage devices.

Because it is also scalable and supports quality of service (QoS) and failover, InfiniBand is often used as a server connect in high-performance computing (HPC) environments.

Figure 1-infiniband 4x card

The internal data flow system in most PCs and server systems is inflexible and relatively slow. As the amount of data coming into and flowing between components in the computer increases, the existing bus system becomes a bottleneck. Instead of sending data in parallel (typically 32 bits at a time, but in some computers 64 bits) across the backplane bus, InfiniBand specifies a serial (bit-at-a-time) bus. Fewer pins and other electrical connections are required, saving manufacturing cost and improving reliability. The serial bus can carry multiple channels of data at the same time in a multiplexing signal. InfiniBand also supports multiple memory areas, each of which can addressed by both processors and storage devices.

With InfiniBand, data is transmitted in packets that together form a communication called a message. A message can be a Remote direct memory access (RDMA) read or write operation A channel send or receive message A reversible transaction-based operation or a multicast transmission.

Like the channel model many mainframe users are familiar with, all transmission begins or ends with a channel adapter. Each processor (your PC or a data center server, for example) has what is called a host channel adapter (HCA) and each peripheral device has a target channel adapter (TCA). These adapters can potentially exchange information that ensures security or work with a given Quality of Service level.

Figure 2-Infiniband Architecture Model

The single data rate switch chips have a latency of 200 nanoseconds, DDR switch chips have a latency of 140 nanoseconds and QDR switch chips have a latency of 100 nanoseconds. The end-to-end latency range spans from 1.07 microseconds MPI latency (Mellanox ConnectX QDR HCAs) to 1.29

microseconds MPI latency (Qlogic InfiniPath HCAs) to 2.6 microseconds (Mellanox InfiniHost DDR III HCAs). InfiniBand also provides RDMA capabilities for low CPU overhead. The latency for RDMA operations is less than 1 microsecond.

Figure 3-infiband connectivity

InfiniBand uses a switched fabric topology, as opposed to a hierarchical switched network like traditional Ethernet architectures. All transmissions begin or end at a "channel adapter." Each processor contains a host channel adapter (HCA) and each peripheral has a target channel adapter (TCA). These adapters can also exchange information for security or quality of service (QoS).

InfiniBand uses copper CX4 cable for SDR and DDR rates also commonly used to connect SAS (Serial Attached SCSI) HBAs to external (SAS) disk arrays. With SAS, this is known as an SFF-8470 connector, and is referred to as an "InfiniBand-style" Connector.

The latest connectors used with QDR and FDR are QSFP (Quad SFP) and can be copper or fiber, depending on the length required.

A few years back, picking the protocol to link your computers together into a network was a nobrainer. The servers in a mid-sized data center were wired together using Ethernet. And if you wanted to connect many nodes into a single high performance computer (HPC), you went with InfiniBand. But nowadays, the choice is blurrier. The two protocols are encroaching on each other's turf, engaging in showdowns for the honor of networking the larger data centers. In November 2012, InfiniBand served as the primary interconnect for 226 of the top 500 systems. Gigabit Ethernet was used on 188 systems. So Infiniband was having the lead. InfiniBand offers advantages such as a flatter topology, less intrusion on the server processor and lower latency. Unlike Gigabit Ethernet's hierarchical topology, InfiniBand is a flat fabric, topologically speaking, meaning each node has a direct connection to all the others. InfiniBand's special sauce is RDMA (Remote Direct Memory Access), which allows the network card to write and read data on a server, eliminating the need for the server processor to conduct this work itself. InfiniBand quickly gained favor in HPC systems, the technology is now creeping into the enterprise. Oracle, for instance, uses InfiniBand as a performance edge for its Exadata and Exalogic data analysis appliances. Microsoft added direct support for RDMA to its newly released Windows Server 2012. One enterprise user of InfiniBand is the Department of Veterans Affairs. The U.S. federal agency's informatics operation runs on about 200 servers, which communicate via InfiniBand. "We do a lot of data transfer," says Augie Turano, a solutions architect at the VA. Databases are moved around quite a bit among the servers so they can be analyzed by different applications. "Being able to move the data at InfiniBand speeds from server to server has been a big boost for us," Turano says. While Ethernet was first used to connect small numbers of computers, successive versions of the specification were tailored for larger jobs, such as serving as the backplane for entire data centers, a job for which it quickly became a dominant player. In this same way, a number of technologies -- such as iWarp and RoCE (RDMA over Converged Ethernet) -- have been started so Gigabit Ethernet can compete directly with InfiniBand by reducing latency and processor usage.

Figure 4-Infiniband vs. 10GB Ethernet

Figure 5-Infiniband card

FireWire is Apple Computer's version of a standard, IEEE 1394, High Performance Serial Bus, for connecting devices to your personal computer. FireWire provides a single plug-and-socket connection on which up to 63 devices can be attached with data transfer speeds up to 400 Mbps (megabits per second). The standard describes a serial bus or pathway between one or more peripheral devices and your computer's microprocessor. Many peripheral devices now come equipped to meet IEEE 1394. FireWire and other IEEE 1394 implementations provide: A simple common plug-in serial connector on the back of your computer and on many different types of peripheral devices A thin serial cable rather than the thicker parallel cable you now use to your printer, for example A very high-speed rate of data transfer that will accommodate multimedia applications (100 and 200 megabits per second today; with much higher rates later) Hot-plug and plug and play capability without disrupting your computer The ability to chain devices together in a number of different ways without terminators or complicated set-up requirements

Figure 6-Firewire port

There are two levels of interface in IEEE 1394 One for the backplane bus within the computer Another for the point-to-point interface between device and computer on the serial cable.

A simple bridge connects the two environments. The backplane bus supports 12.5, 25, or 50 megabits per second data transfer. The cable interface supports 100, 200, or 400 megabits per second. Each of these interfaces can handle any of the possible data rates and change from one to another as needed. The serial bus functions as though devices were in slots within the computer sharing a common memory space. A 64-bit device address allows a great deal of flexibility in configuring devices in chains and trees from a single socket. IEEE 1394 provides two types of data transfer: Asynchronous Isochronous.

Asynchronous is for traditional load-and-store applications where data transfer can be initiated and an application interrupted as a given length of data arrives in a buffer. Isochronous data transfer ensures that data flows at a pre-set rate so that an application can handle it in a timed way. For multimedia applications, this kind of data transfer reduces the need for buffering and helps ensure a continuous presentation for the viewer.

1934 standard requirements:

The 1394 standard requires that a device be within 4.5 meters of the bus socket. Up to 16 devices can be connected in a single chain, each with the 4.5 meter maximum (before signal attenuation begins to occur) so theoretically you could have a device as far away as 72 meters from the computer. Another new approach to connecting devices, the Universal Serial Bus (USB), provides the same "hot plug" capability as the 1394 standard. It's a less expensive technology but data transfer is limited to 12 Mbps (million bits per second). Small Computer System Interface offers a high data transfer rate (up to 40 megabytes per second) but requires address pre-assignment and a device terminator on the last device in a chain. FireWire can work with the latest internal computer bus standard, Peripheral Component Interconnect (PCI), but higher data transfer rates may require special design considerations to minimize undesired buffering for transfer rate mismatches.

Figure 7-firewire cable

The designers of FireWire had several particular goals in mind when they created the standard: Fast transfer of data Ability to put lots of devices on the bus Ease of use Hot-pluggable ability Provision of power through the cable Plug-and-play performance Low cabling cost Low implementation cost

The FireWire device is designed according to the FireWire standard, the device can be connected and disconnected, while both the device and the Macintosh as well the system with which it is to be connected are turned on. Fire Wire can work with up to 63 devices on a bus. The number of bus-powered devices you can connect depends on the amount of power available from the computer, and the amount of power required by each device.

The process of the bus deciding which node gets to transmit data at what time is known as arbitration. Each arbitration round lasts about 125 microseconds. During the round, the root node (device nearest the processor) sends a cycle start packet. All nodes requiring data transfer respond, with the closest node winning. After the node is finished, the remaining nodes take turns in order. This repeats until all the devices have used their portion of the 125 microseconds, with isochronous transfers having priority. Up to 80% of the time can be given to isochronous nodes.

The key difference between FireWire and USB is that FireWire is intended for devices working with a lot more data -- things like camcorders, DVD players and digital audio equipment. FireWire and USB share a number of characteristics but differ in some important ways. Implementing FireWire costs a little more than USB, which led to the adoption of USB as the standard for connecting most peripherals that do not require a high-speed bus.

Speed aside, the big difference between FireWire and USB 2.0 is that USB 2.0 is host-based, meaning that devices must connect to a computer in order to communicate. FireWire is peer-to-peer, meaning that two FireWire cameras can talk to each other without going through a computer.

FireWire can be used for ad-hoc (terminals only, no routers except where a FireWire hub is used) computer networks. Specifically, RFC 2734 specifies how to run IPv4 over the FireWire interface, and RFC 3146 specifies how to run IPv6. Mac OS X, Linux, and FreeBSD include support for networking over FireWire. Windows Me, Windows XP and Windows Server 2003 include native support for IEEE 1394 networking. Windows 2000 does not have native support but may work with third party drivers. A network can be set up between two computers using a single standard FireWire cable, or by multiple computers through use of a hub. This is similar to Ethernet networks with the major differences being transfer speed, conductor length, and the fact that standard FireWire cables can be used for point-to-point communication. The PlayStation 2 console had an i.LINK-branded 1394 connector. This was used for networking until the release of an Ethernet adapter late in the console's lifespan, but very few software titles supported the feature.

http://searchstorage.techtarget.com/definition/InfiniBand http://en.wikipedia.org/wiki/InfiniBand http://www.networkworld.com/news/2012/120312-argument-infiniband-gigabit264530.html?page=2 https://en.wikipedia.org/wiki/IEEE_1394 http://computer.howstuffworks.com/firewire3.htm http://www.ehow.com/about_5091205_firewire-cable-used.html http://support.apple.com/kb/TA26476 http://searchnetworking.techtarget.com/definition/FireWire http://www.howstuffworks.com/firewire.htm