Cisco InfiniBand Switching

for High Performance Applications

Defining a Unified Compute Fabric for The
Enterprise

Amir Sharif, Product Manager

InfiniBand Infrastructure

June 2006

Session Number
Presentation_ID © 2005 Cisco Systems, Inc. All rights reserved. 1
Agenda

• Unified Compute Fabric and Distributed Systems

• Cisco leadership: Open fabrics & Open MPI
• SFS-OS 2.7 update & CiscoWorks Integration

© 2005 Cisco Systems, Inc. All rights reserved. 2

 InfiniBand Adoption Across Customer Markets
Example 2005 Cisco SFS Deployments
InfiniBand-
Top 500
Vertical attached
Ranking
servers

Sandia National Labs Government/Research 4400 5

Bio-Infomatics Cluster* Research 1066
Tier 1 Systems Company* Service Provider 1024
University of Sherbrooke (Canada) Education/Research 576 51
NCSA Education/Commercial 512 59
University of Oklahoma Education 512 67
University* Education 386
Wall Street Bank* Financial/Grid 370
Japanese Car Company* Automotive/CAE 300
MCNC Government/Education 284
SARA (Netherlands) Government/Education 272 277

Tier 1 Systems Company* Hosted Services 256

Wall Street Bank* Finance/Grid 256

TACC Education/Commercial 200 301

Arizona State Education 200 328

* Non-Referenceable Customer Name

© 2005 Cisco Systems, Inc. All rights reserved. 3

 Cisco Data Center Network Framework
Defining a Unified Compute Fabric
PLM CRM ERP Instant Unified Meeting
Business Messaging Collaboration
Messaging Place

HCM Applications
Procurement SCM IPCC Applications
IP Phone Video
Delivery
Traditional Architecture / Service Oriented Architecture

Advanced Analytics and Decision Support

Services Management

ANALYTICS & ADAPTIVE

Application Networking Services
WAAS, App Acceleration,
Application Delivery Services
INTERACTIVE

Optimization, Security and Server Offload

SERVICES
LAYER

POLICY
Infrastructure Enhancing Services
Firewalls, Intrusion Protection,
Security
Security Services
Agents

RDMA, Virtual I/O, Virtualization, Replication,

Compute
Low LatencyServices
Clustering Storage Fabric Services
Virtual Fabrics
Infrastructure Management
INFRASTRUCTURE

Compute Network Storage Network

NETWORKED

Server Server Storage Data Center

LAYER

Fabric Switching Switching Interconnect

Modular Director DWDM,
Infiniband Rack SONET,
Switching Fabric
Blade SDH, FCIP
SFS Family Catalyst Family MDS Family ONS Family

© 2005 Cisco Systems, Inc. All rights reserved. 4

 Distributed Applications & Networking
Distributed High- High-Performance
Systems Performance Database and
Computing Storage

I/O Nodes
Head Node

TIBco, Wombat, 29West, etc Fluent, Ansys, Charmm, Oracle, IBM DB2
Amber, LS-Dyna, etc Luster, IBM GPFS, Panasas,

© 2005 Cisco Systems, Inc. All rights reserved. 5

Are feeds and Speeds enough?

© 2005 Cisco Systems, Inc. All rights reserved. 6

 Fragmented Landscape Inhibits Distributed
Systems Adoption

Ethernet InfiniBand

4 5
2
3 6
7
• Multiple MPI implementations – no standards to build
1 8 off of, more options to certify
4 5 4 5
3 6 • Extensive Configuration and • Limited Configuration and 3 6
2 7
management tools management tools 2 7
1 8 1 8

4 5 • Loosely defined protocol 4 5

3 6 • Broad support for standard stacks – limited 3 6
2 7
protocol stacks - Interoperable interoperability
2 7
1 8 1 8

3
4 5
6 • Simple “Plug & Play” Operation • Do-it-yourself testing, 4 5
3 6
2 7 integration and certification 2 7
1
• 10/100/1000 + 10GE
8 • 4X 10Gbps Ports 1 8

Multiple Fabric Options – No uniform integration of Capabilities

© 2005 Cisco Systems, Inc. All rights reserved. 7
What is a Unified Compute Fabric?

A unified compute fabric integrates network

management and troubleshooting, application
protocols and APIs for Ethernet and InfiniBand
fabrics, offering IT cluster administrators a single
operational model for greater ease-of-use, rapid
deployment, higher reliability and integrated
security

© 2005 Cisco Systems, Inc. All rights reserved. 8

Driving Open Standards New
Open Fabric and Open MPI

IBM
MPICH MPI
Open Source RDS
SRP
iSER
LAM/MPI SCALI
Open MPI SDP MVAPICH iWARP Open Fabric
MPI

• Problem: Proprietary protocol stacks multiply options for application

development and complicate certifications
• Solution: Cisco will move strategic focus to Open Fabrics and Open MPI
• Benefits:
Single stack, fully interoperable between vendors
Eliminates fragmented infrastructure that reduce performance
Accelerates application development for the ISV and deployment and certification for
the end-user
Consistent operation across Ethernet & InfiniBand

© 2005 Cisco Systems, Inc. All rights reserved. 9

 Bringing Ethernet-Like Maturity to Infiniband

Ethernet Infiniband
Standard CiscoWorks LMS
Manageability Integration and TopSpin
OS 2.7
Standard MIBs and
Common platform for
Manageability
multi-fabric management
integrated into
Unifying and new management
Ethernet
Ethernet and and security capability
Standard Drivers and Infiniband for
Application Interface Open Fabrics and
High Open MPI
IEEE 802.3, 802.1, Performance Standard
TCP/IP have unified Applications development platform
the world around a for drivers and app
common capability integration

Evolutionary Double-Data Rate

Performance (DDR)
Incremental perform- Higher bandwidth
ance capabilities provide option for incremental
right-sized price per port bandwidth
requirements

© 2005 Cisco Systems, Inc. All rights reserved. 10

 Delivering the Unified Compute Fabric

Ethernet InfiniBand

4 5
3 6 • Drive commercial Open MPI middleware to deliver standard features and
2 7 perfoprmance, and reduce implementation and support overhead
1 8

4 5
3 6 • InfiniBand Integration into CiscoWorks
• Management tools
Limited Configuration and
2 7
management tools
1 8

3
4 5
6 • Drive standardization of Protocols within Open Fabrics Consortium to deliver
2 7 standard features and reduce time to deploy
1 8

4 5
3 6 • Extend Ethernet “Plug & Play” functionality to InfiniBand
2 7
1 8 • Deliver Next-generation technologies – InfiniBand dual speed DDR/SDR

Uniform Cross-Fabric Capabilities

© 2005 Cisco Systems, Inc. All rights reserved. 11
 Cisco Open Fabric Strategy
• The Enabler for true Enterprise
High-performance Computing
UserAccess
HPC/GRID Network
Applications Systems
Application
• Only vendor to offer complete Open MPI
multifabric solution
Open Fabrics LAN
• Multi-fabric Management GE/10GE RNIC Switching
Common tools for Ethernet &
InfiniBand HPC
Comprehensive APIs for 3rd Network
Party Integration
Storage Fabric Server
• Fabric Agnostic Protocols Fabric Switching
Unify MPI across Ethernet & Services
InfiniBand with Open MPI
Unify RDMA across Ethernet & Application
InfiniBand with Open Fabrics.
Open MPI
• Cisco ISV Certification Open Fabrics
InfiniBand & Ethernet IB HCA
Pooled
Certification for InfiniBand & Storage
Pooled
Fiber Channel Storage Resources
Compute
Resources

© 2005 Cisco Systems, Inc. All rights reserved. 12

Cisco Delivering Unification to Distributed
Computing

Cisco Technology
Developer Program &
Open source integration
Application Integration and Certification

Extended Network
Awareness
Control and Monitoring
Cluster Configuration,

Message Passing
Interface (MPI) Standards based
Management,

functionality

Open Fabrics (OFED) Unify Ethernet &

Infiniband Capabilities

Storage and Develop Next Generation

File Ethernet Infiniband InfiniBand & Ethernet
Systems
Unify Network
Cisco CLI, Common syntax, File and Image management
Management, SSH, SNMPv3, Radius, etc

© 2005 Cisco Systems, Inc. All rights reserved. 13

 Cisco SFS-OS 2.7 New
Consistent Configuration & Management

• Common CLI across all products

Command Syntax, scripting, etc.
• Consistent Security model
TACACS and RADIUS for Centralized
Auth/ACS Integration
SSH/SSL/SNMPv3 for full management
security
Multiple authorization levels
• File & Image Management
System image and configuration file
libraries.
• Consistent Management Notification
Full SNMP v1/v2/v3 Support across all
fabrics
Streaming Syslog: Integrates with
Syslog Analyzer

© 2005 Cisco Systems, Inc. All rights reserved. 14

 CiscoWorks LMS Support for InfinibandNew
• Single network management
application for Ethernet and
InfiniBand networks
• Delivers Centralized Device,
Software and Configuration
Inventory Manager
• Diagnostic Tools and Syslog
Analyzer
• Centralized Reporting
• Device level fault analysis for
network fabric, including high
availability monitoring,
pager/email/trap notification
• Benefit: Eliminates
administrative and usage
barriers; identify and fix Available: CQ3, 2006
problems -> increased
performance
© 2005 Cisco Systems, Inc. All rights reserved. 15
 High Performance Subnet Manager
• Overall Subnet Manager and
framework tested on Sandia
Thunderbird -- World’s largest
standard HPC server cluster (4500 IB-
attached servers)
• SM brings multi-thousand server
cluster up in less than one minute
• Includes database synchronization
between redundant SM’s for HA
• Includes performance and statistics
tools capable of reporting on tens of
thousands of network ports in half a
minute
• Multi-Vendor support

© 2005 Cisco Systems, Inc. All rights reserved. 16

Cisco InfiniBand Hardware Roadmap

New

Cisco PCI-X & PCI-EX

New
IBM Bladecenter 1*4X HCA
InfiniBand 1X
Switch & HCA IBM Bladecenter H
InfiniBand 4X
Switch & HCA

Dell 1855 Bladecenter SFS-3012P

InfiniBand 4X
Module & HCA
SFS-7024P New
SFS-3012

SFS-3012 SFS-7008P
SFS-7000D
Gateways Family
SFS-7008 SFS-7012P

SFS-7000P New
SFS-7000
4X DDR
PCI-Ex HCA
Cisco PCI-X & PCI-Ex
2*4X HCA

© 2005 Cisco Systems, Inc. All rights reserved. 17

 Cisco InfiniBand Software Roadmap

OpenFabrics
iWARP integration

HPSM 1.3

OpenFabricsNew
Enterprise
Distribution v2.0 – New
RDS and iSER SFS-OS
support Resource Manager Advanced QoS (SL to VL
Essentials SFS mappings)
High-Performance integration InfiniBand port mirroring
InfiniBand Subnet
Manager New
InfiniBand packet sniffing
Cluster Readiness Tool Kit
TopspinOS Switch Management v3.2 SFS-OS v2.7.0 CiscoWorks Device fault
OpenFabrics Enterprise New Manager
Distribution - Advanced Fabric
(IPoIB, SDP, SRP, OpenMPI, Management Tools
High-Performance
MVAPICH)
InfiniBand Subnet - Additional topology-
Manager 1.2 aware logic/variables
Embedded Subnet Manager <1100 nodes
(multiple vendors)

© 2005 Cisco Systems, Inc. All rights reserved. 18

 Q and A

© 2005 Cisco Systems, Inc. All rights reserved. 19

© 2005 Cisco Systems, Inc. All rights reserved. 20