BRKSAN-2821

14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 1

I/O Consolidation
in the Data Center

BRKSAN-2821

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 2

© 2006, Cisco Systems, Inc. All rights reserved. 1

Presentation_ID.scr
 Agenda

ƒ Section 1: What is I/O Consolidation

ƒ Section 2: Enabling Technologies
ƒ Section 3: FCoE (Fibre Channel over Ethernet)
ƒ Section 4: I/O Consolidation Use Cases
ƒ Challenges
ƒ Closing Remarks

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 3

Section 1
What Is I/O
Consolidation

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 4

© 2006, Cisco Systems, Inc. All rights reserved. 2

Presentation_ID.scr
 What Is I/O Consolidation

ƒ IT organizations operate multiple parallel networks

IP and other LAN protocols over an Ethernet network
SAN over a Fibre Channel network
HPC/IPC over an InfiniBand network

ƒ I/O consolidation supports all three types of traffic onto

a single network
ƒ Servers have a common interface adapter that supports
all three types of traffic

IPC: Inter-Process Communication

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 5

I/O Consolidation in the Network

Processor Processor

Memory Memory

I/O I/O I/O I/O Subsystem

Storage

Storage
LAN
IPC

LAN
IPC

IPC: Inter-Process Communication

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 6

© 2006, Cisco Systems, Inc. All rights reserved. 3

Presentation_ID.scr
 I/O Consolidation in the Host

ƒ Fewer CNAs (Converged Network Adapters) instead

of NICs, HBAs, and HCAs
ƒ Limited number of interfaces for Blade Servers

FC HBA FC Traffic
FC HBA FC Traffic
CNA All Traffic
NIC Enet Traffic Goes over
10 GE
NIC Enet Traffic CNA

NIC Enet Traffic

HCA IPC Traffic
HCA IPC Traffic
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 7

Cabling and I/O Consolidation

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 8

© 2006, Cisco Systems, Inc. All rights reserved. 4

Presentation_ID.scr
 I/O Consolidation: Benefits to Customers

FC Traffic
FC Traffic FCoE
FCoE SAN A
FCoE

Enet Traffic FCoE SAN B

Enet Traffic

FCoE SAN

Fewer CNAs and Cables Storage Keeps the Same

Management Model as Native FC

Display
FCoE
Adapter

FC Storage FC Switch FCoE Server

Switch

No Storage Gateway Less Power and Cooling

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 9

Merging the Requirements

IPC
LAN/IP Storage (Inter-Process
Communication)

ƒ Must be Ethernet ƒ Must follow the ƒ Doesn’t care of

Too much
Fibre Channel the underlying
investment model network, provided
that:
Too many ƒ Losing frames is
applications that not an option It is cheap
assume Ethernet It is low latency
It supports APIs
like OFED, RDS,
MPI, sockets

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 10

© 2006, Cisco Systems, Inc. All rights reserved. 5

Presentation_ID.scr
 Why Has It Not Succeeded Yet?

ƒ Previous attempts
Fibre Channel
Never credible
InfiniBand
Not Ethernet
iSCSI
Not Fibre Channel

ƒ Before PCI-Express there was not enough I/O

bandwidth in the servers
ƒ It needs to be Ethernet, but…
1 GE didn’t have enough bandwidth
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 11

PCI-Express

ƒ PCI Express (PCI-E or PCIe)

A computer expansion card interface format designed to
replace PCI, PCI-X, and AGP

ƒ PCIe 1.1
Serial links at 2.5 Gbps (2 Gbps at the Datalink)
Speeds from 2 Gbps (1x) to 32 Gbps (16x)
8x is required for 10 GE

ƒ PCIe 2.0 (aka PCIe Gen 2)

Doubles the bandwidth per serial link from 2 Gbit/s to 4 Gbit/s
Spec available since January 2007
Products are making their way into the market
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 12

© 2006, Cisco Systems, Inc. All rights reserved. 6

Presentation_ID.scr
 10 GE

ƒ 2008 will be the year of 10GE

ƒ 10 GE has enough bandwidth
ƒ Merging example
2 x 1 GE Ethernet NIC
1 x 4 Gbps FC (really 3.2 Gbps)
Total 5.2 Gbps over a 10 Gbps link

ƒ CNAs will all be dual-ported for HA

20 Gbps usable bandwidth per server with a single CNA

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 13

Evolution of Ethernet Physical Media

Role of Transport in Enabling These Technologies

Mid-1980s Mid-1990s Early-2000s Late-2000s

SFP+ to SFP+
10 Mb 100 Mb 1SFP+
Gb Cu 10 Gb

UTP Cat 5 UTP Cat 5

SFP Fiber

Power Transceiver
Technology Cable Distance (Each Side) Latency (Link)

SFP+ CU
Twinax 10 m 0.1W 0.1 μs
Copper

SFP+ USR MM OM2 10 m

1W 0
Ultra Short Reach MM OM3 100 m

SFP+ SR MM 62.5 μm 82 m
Short Reach MM 50 μm 300 m 1W 0

Cat6 55 m 8W 2.5 μs
10GBASE-T Cat6a/7 100 m 8W 2.5 μs
Cat6a/7 30 m 4W 1.5 μs

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 14

© 2006, Cisco Systems, Inc. All rights reserved. 7

Presentation_ID.scr
 Twin-ax Copper Cable
ƒ Low power consumption SAN A LAN SAN B

ƒ Low cable cost

ƒ Low transceiver latency
Application Server

ƒ Low error rate (10–17) Application Server

Application Server

ƒ Thinner cable with higher Application Server

16x10 GE Cables

16x10 GE Cables
Application Server

bend radius Application Server

Application Server

ƒ Easier to manage cabling Application Server

Application Server
solution reduces Application Server

deployment time Application Server

Application Server

ƒ All copper cables are Application Server

Application Server

contained within rack Application Server

Application Server

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 15

Drivers for 10GE to the Servers

Multicore CPU Architectures Allowing Bigger and
Multiple Workloads on the Same Machine

Server Virtualization Driving the Need for More

Bandwidth per Server Due to Server Consolidation

Growing Need for Network Storage Driving the

Demand for Higher Network Bandwidth to the Server

Multicore CPUs and Server Virtualization Driving the

Demand for Higher Bandwidth Network Connections

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 16

© 2006, Cisco Systems, Inc. All rights reserved. 8

Presentation_ID.scr
 Section 1
Enabling Technologies

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 17

Three Challenges + One

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 18

© 2006, Cisco Systems, Inc. All rights reserved. 9

Presentation_ID.scr
 Why Are Frames Lost?

Transmission
Collision Congestion
Error

ƒ No longer present ƒ Very rare in the ƒ Most common

in full duplex data center cause
Ethernet
ƒ Congestion is a
switch issue, not a
link issue
A full duplex IEEE
802.3 link does not
lose frames
ƒ It must be dealt
with in the
bridge/switch
By IEEE 802.1

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 19

Can Ethernet Be Lossless?

ƒ Yes, with Ethernet PAUSE Frame

Ethernet Link

STOP PAUSE Queue Full

Switch A Switch B

ƒ Defined in IEEE 802.3—Annex 31B

The PAUSE operation is used to inhibit transmission of
data frames for a specified period of time

ƒ Ethernet PAUSE transforms Ethernet into a

lossless fabric
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 20

© 2006, Cisco Systems, Inc. All rights reserved. 10

Presentation_ID.scr
 How PAUSE Works

Threshold

A B
PAUSE
Frame
Start
Stop Sending
Frames
FramesforAgain
This
Interval of Time
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 21

Let’s Compare PAUSE with

FC Buffer to Buffer Credit

ƒ Eight credits preagreed

A B

R_RDY

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 22

© 2006, Cisco Systems, Inc. All rights reserved. 11

Presentation_ID.scr
 How PAUSE Propagates

Threshold

PAUSE PAUSE

S1 S2 S3

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 23

PAUSE Frame Format

PAUSE Frame
ƒ A standard Ethernet frame, not
01:80:C2:00:00:01 tagged

Source Station MAC

ƒ EtherType = 0x8808 means MAC
Control Frame
EtherType = 0x8808
Opcode = 0x0001
Pause_Time
ƒ Opcode = 0x0101 means PAUSE
ƒ Pause_Time is the time the link
needs to remain paused in
Pad
Pause Quanta (512-bits time)
42 Bytes
ƒ There is a single Pause_Time for
the whole link
…
CRC

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 24

© 2006, Cisco Systems, Inc. All rights reserved. 12

Presentation_ID.scr
 Why Is PAUSE Not Widely Deployed?

ƒ Inconsistent implementations
Standard allows for asymmetric implementations
Easy to fix

ƒ PAUSE applies to the whole links

Single mechanism for all traffic classes

ƒ This may cause “traffic interference”

e.g., Storage traffic paused due to a congestion on IP traffic

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 25

Priority Flow Control (PFC)

ƒ a.k.a. PPP (Per Priority Pause)
ƒ PFC enables PAUSE functionality per Ethernet priority
IEEE 802.1Q defines eight priorities
Traffic classes are mapped to different priorities:
No traffic interference
IP traffic may be paused while storage traffic is being forwarded
Or, vice versa
Requires independent resources per priority (buffers)
ƒ High level of industry support
Cisco distributed proposal
Standard track in IEEE 802.1Qb

16 3 1 12 Bits
EtherType = IEEE 802.1Q Priority CFI VLAN ID

IEEE 802.1Q Tag

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 26

© 2006, Cisco Systems, Inc. All rights reserved. 13

Presentation_ID.scr
 Priority Flow Control in Action

Transmit Queues Receive Queues

Ethernet Link
One One
Two Two
Three Three
Eight
Four Four Priorities
Five Five
Six STOP PAUSE Six
Seven Seven
Eight Eight

Switch A Switch B

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 27

PFC Frame Format

Priority Flow Control
ƒ Similar to the PAUSE frame
01:80:C2:00:00:01
ƒ Opcode = 0x0101 is used to
Source Station MAC
distinguish PFC from PAUSE
EtherType = 0x8808 ƒ Class vector indicates for which
Opcode = 0x0101
Class Enable Vector priorities the frame carries valid
Time (Class 0)
Time (Class 1)
Pause information
Time (Class 2)
Time (Class 3) ƒ There are eight Time fields,
Time (Class 4)
Time (Class 5)
one per priority
Time (Class 6)
Time (class 7)
Pad
28 Bytes
…
CRC

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 28

© 2006, Cisco Systems, Inc. All rights reserved. 14

Presentation_ID.scr
 Is Lossless Better?

ƒ Frames are ƒ TCP relies on ƒ Congestion

not dropped losses spreading and
head of line
ƒ FC over lossless ƒ We can run it on
blocking
Ethernet works a priority where
well we do not enable
Pause

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 29

Is Anything Else Required?

In Order to Build a Deployable I/O Consolidation

Solution, the Following Additional Components
Are Required:

ƒ Discovery protocol (DCBX)

ƒ Bandwidth manager
ƒ Congestion management

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 30

© 2006, Cisco Systems, Inc. All rights reserved. 15

Presentation_ID.scr
 Discovery Protocol

DCBX:
Data Center
Bridging eXchange
Data Center
Ethernet Links

DCBX Data Center Ethernet

Links with Partial
Enhancements

DCBX

Legacy Ethernet
Links
Data Center
Ethernet Cloud

XP
Legacy Ethernet
Network

BC
DC

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 31

DCBX

ƒ Hop-by-hop negotiation for:

Priority Flow Control (PFC)
Bandwidth management
Congestion management (BCN/QCN)
Applications
Logical link-down

ƒ Based on LLDP (Link Level Discovery Protocol)

Added reliable transport

ƒ Allows either full configuration or configuration checking

Link partners can choose supported features and willingness to
accept configuration from peer
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 32

© 2006, Cisco Systems, Inc. All rights reserved. 16

Presentation_ID.scr
 Bandwidth Management

ƒ IEEE 802.1Q defines priorities, but not a simple,

effective, and consistent scheduling mechanism
ƒ Products typically implement some form of Deficit
Weighted Round Robin (DWRR)
Configuration and interworking is problematic

ƒ Proposal for HW-efficient, two-level DWRR with strict

priority support
Consistent behavior and configuration across network elements

ƒ Standard track in IEEE 802.1Qaz

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 33

Priority Groups
Priority Groups
Are Then
Scheduled
LAN

Priorities Are
Assigned to
Individual SAN
Traffic Classes

IPC Final Link

Behavior
Priority
Groups

First Level of Scheduling

Inside Each Group
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 34

© 2006, Cisco Systems, Inc. All rights reserved. 17

Presentation_ID.scr
 Goals
ƒ BW assignment for each ƒ Cannot compromise low-
“Priority Group” latency application due
Example: 40% LAN, to convergence
40% SAN, 20% IPC Allow strict, high priority
scheduling of IPC (and
ƒ Should allow multiple traffic equivalent) traffic
classes within a “Priority
Group” ƒ Should provide
management infrastructure
Allow these traffic classes to
share BW without hard (MIBs)
configuration Defining scheduling algorithms
Example: VoIP and bulk traffic is too restrictive and not
to share 40% LAN BW necessary
Interoperability for
management is important

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 35

Example of Link Bandwidth Allocation

Offered Traffic 10 GE Link Realized Traffic Utilization

3 Gbs 3 Gbs 2 Gbs (30%) HPC Traffic (20%)

(30%)

LAN Traffic
(30%) (50%)
3 Gbs 4 Gbs 6 Gbs (40%)

Storage Traffic
(30%) (30%)
3 Gbs 3 Gbs 3 Gbs (30%)

T1 T2 T3 T1 T2 T3

HPC Traffic—Priority Class “High”—20% Guaranteed Bandwidth

LAN Traffic—Priority Class “Medium”—50% Guaranteed Bandwidth
Storage Traffic—Priority Class “Medium-High”—30% Default Bandwidth
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 36

© 2006, Cisco Systems, Inc. All rights reserved. 18

Presentation_ID.scr
 Congestion Management

ƒ Layer 2, end-to-end congestion management

ƒ Standards track in IEEE 802.1Qau
ƒ a.k.a. BCN (Backward Congestion Notification) or
QCN (Quantized Congestion Notification)
Switch Switch

Switch

Transmit Queue
Thro
ttle Receive Buffer
Switch Switch

e
ottl
Thr

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 37

Congestion Management Principles

ƒ Move congestion to network edges to avoid congestion
spreading
ƒ Use rate-limiters at the edge to shape flows causing
congestion
Tune rate-limiter parameters based on feedback coming from
congestion points
ƒ Inspired by:
TCP
AIMD (Additive Increase, Multiplicative Decrease) rate control
TCP window increases linearly in absence of congestion
Decreases exponentially (gets halved) at every congestion
indication (either implicit or explicit)
FCC (Fibre Channel Congestion Control)
A feature on Cisco MDS switches
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 38

© 2006, Cisco Systems, Inc. All rights reserved. 19

Presentation_ID.scr
 Congestion Point and Reaction Point

ƒ Roles and responsibilities

Reaction Points (RP) shape
traffic entering the network
RP
Congestion Points (CP)
indicate congested state
of queuing points
CP
RP

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 39

DCB: Data Center Bridging

ƒ Industry consensus term to indicate

Priority flow control
Bandwidth management
Congestion management
Discovery (DCBX)

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 40

© 2006, Cisco Systems, Inc. All rights reserved. 20

Presentation_ID.scr
 DCE: Data Center Ethernet

ƒ Cisco term used to indicate Cisco switches that

implement the DCB features, plus
Layer 2 multipathing
Fibre Channel over Ethernet

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 41

Layer 2 Multipathing

ƒ Increase bandwidth
of L2 networks via
multiple active links

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 42

© 2006, Cisco Systems, Inc. All rights reserved. 21

Presentation_ID.scr
 L2 Multipathing

ƒ Multiple paths are used, reclaiming network bandwidth

ƒ L3 multipathing is common in IP networks
ƒ Important when there is limited or no differentiation in
speed between access links and backbone links
ƒ Reduces latency
ƒ L2 multipathing
Eliminates Spanning Tree from the backbone
No packet flooding
Small forwarding tables

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 43

Layer 2 Multipathing

ƒ Cisco DCE is:

A precursor of TRILL, an IETF project for Layer 2 multipath
Inspired to FSPF (Fibre Channel Shortest Path First)

ƒ Cisco DCE
9 Computes topology and forwarding via IS-IS
9 Provides optimal pair-wise unicast forwarding
9 Provides multipathing for unicast and multicast frames
9 Provides seamless interoperability with existing devices

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 44

© 2006, Cisco Systems, Inc. All rights reserved. 22

Presentation_ID.scr
 FCoE: Fibre Channel over Ethernet

ƒ FCoE is the protocol

used to carry Fibre
SAN LAN SAN
Channel over CEE/DCE
Allows storage
I/O consolidation
FCoE

ƒ It’s in an advanced state

of definition in INCITS
T11 FC-BB-5 WG

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 45

Delayed Drop

ƒ Delayed Drop is a mechanism that:

Allows a switch buffer to virtually extend to previous hop
This reduces packet drop for transient congestions
Is enabled per priority

ƒ It is implemented by asserting PFC on the priority for

a short time
After that time, traffic can flow again or can be dropped

Delayed Drop Is a Means of Using PFC to Mitigate

the Effects of Short-Term Traffic Bursts While
Maintaining Packet Drop for Long-Term Congestion

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 46

© 2006, Cisco Systems, Inc. All rights reserved. 23

Presentation_ID.scr
 Delay Drop and Proxy Queue
Actual Queue
Traffic
Flow

PAUSE

Proxy Queue

UNPAUSE

ƒ During short-term congestion, both queues drain fast enough that

the actual queue releases the PAUSE on its own
ƒ During long-term congestion, the proxy queue fills to its high-water
mark, and it releases the PAUSE; the actual queue begins
dropping packets, and the congestion is managed through higher-
level protocols
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 47

Section 3
FCoE: Fibre Channel
over Ethernet

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 48

© 2006, Cisco Systems, Inc. All rights reserved. 24

Presentation_ID.scr
 FCoE: FC over Ethernet

ƒ FCoE is I/O consolidation of FC storage traffic

over Ethernet
FC traffic shares Ethernet links with other traffics
Requires a lossless Ethernet fabric

Ethernet

Fibre Channel
Traffic

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 49

FCoE Protocol Stack

ƒ From a Fibre Channel
standpoint, its FC connectivity
over a new type of cable
called an Ethernet cloud
ƒ From an Ethernet standpoint,
it’s yet another ULP (Upper
Layer Protocol) to be
transported

1, 10…Gbps

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 50

© 2006, Cisco Systems, Inc. All rights reserved. 25

Presentation_ID.scr
 FCoE Benefits

ƒ FCoE benefits are the same of any I/O consolidation

solution
Fewer cables
Both block I/O and Ethernet traffic coexist on same cable
Fewer adapters needed
Overall less power

ƒ Plus additional advantages of being FC

Seamless integration with existing FC SANs
No gateway

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 51

FCoE Benefits
LAN SAN-A SAN-B LAN SAN-A SAN-B
4 4

2
2

Nearly Half
the Cables

16 Servers Enet FC Total 16 Servers Enet FC Total

Adapters 16 16 32 Adapters 16 0 16
Switches 2 2 4 Switches 2 0 2
Cables 36 36 72 Cables 36 4 40
Mgmt Pts 2 2 4 Mgmt Pts 2 0 2
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 52

© 2006, Cisco Systems, Inc. All rights reserved. 26

Presentation_ID.scr
 FCoE Is Fibre Channel
FCoE Is Fibre Channel at the Host and Switch Level

Easy to Completely Based

Understand on the FC Model
Same Host-to-Switch and
Same
Operational Model Switch-to-Switch Behavior
of FC
Same Techniques of e.g., in Order Delivery or
Traffic Management FSPF Load Balancing

Same Management WWNs, FC-IDs, Hard/Soft

and Security Models Zoning, DNS, RSCN

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 53

Protocol Organization

FCoE Itself FIP

(FCoE Initialization Protocol)
ƒ Is the data plane protocol ƒ It is the control plane protocol
ƒ It is used to carry most of the ƒ It is used to discover the FC
FC frames and all the SCSI entities connected to an
traffic Ethernet cloud
ƒ It is also used to login to and
logout from the FC fabric

The Two Protocols Have:

ƒ Two different EtherTypes
ƒ Two different frame formats
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 54

© 2006, Cisco Systems, Inc. All rights reserved. 27

Presentation_ID.scr
 FCoE Frame Size
Ethernet
Header 12 Bytes (MAC Addresses) +
4 Bytes (802.1Q Tag)
FCoE
Header
16 Bytes
FC
Header

24 Bytes
Tota
l: 2180
FC Payload

Byte
Up to 2112 Bytes s

4 Bytes

1 Byte (EOF) + 3 Bytes (Padding)

CRC
EOF
FCS
4 Bytes
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 55

FCoE Frame Format

Destination MAC Address

Source MAC Address

IEEE 802.1Q Tag
ET = FCoE Ver Reserved
Reserved
Reserved
Reserved SOF

Encapsulated FC Frame
(Including FC-CRC)

EOF Reserved
FCS
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 56

© 2006, Cisco Systems, Inc. All rights reserved. 28

Presentation_ID.scr
 ENode: Simplified Model

ƒ ENode (FCoE Node): a Fibre Channel HBA

implemented within an Ethernet NIC
a.k.a. CNA (Converged Network Adapter)

FC Node

FCoE FCoE

Enet Enet
Port Port

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 57

FCoE Switch: Simplified Model

ƒ FCF (Fibre Channel Forwarder), the forwarding entity

inside an FCoE switch

FC
Port
FCoE Switch
FC
FCF Port
FCoE
FC
Port

Ethernet Bridge FC
Port

Eth Eth Eth Eth Eth Eth Eth Eth

Port Port Port Port Port Port Port Port

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 58

© 2006, Cisco Systems, Inc. All rights reserved. 29

Presentation_ID.scr
 FC-BB-5 Terminology

ƒ Unchanged from previous FC standard

VN_Port: Virtual N_Port
VF_Port: Virtual F_Port
VE_Port: Virtual E_Port

ƒ Added to support FCoE

FCoE_LEP (FCoE link endpoint): The data forwarding
component that handles FC frame encapsulation/decapsulation,
and transmission/reception of FCoE frames
FCoE Controller: the entity that implement the FIP protocol

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 59

ENode: Complete Model

FC-3/FC-4s FC-3/FC-4s FC-3/FC-4s FC-3/FC-4s

FC FC FC FC
VN_Port VN_Port VN_Port VN_Port
Entity Entity Entity Entity

FCoE FCoE FCoE FCoE

FCoE_LEP FCoE_LEP FCoE_LEP FCoE_LEP
Entity Entity Entity Entity
FCoE FCoE
Controller Controller

Lossless Ethernet MAC Ethernet_Port Lossless Ethernet MAC Ethernet_Port

DISC
F FLOGI
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 60

© 2006, Cisco Systems, Inc. All rights reserved. 30

Presentation_ID.scr
 Means
FCoE Switch: Complete Model Optional

FC Fabric Interface E_Port E_Port E_Port F_Port F_Port F_Port

FC Switching Element

FC FC FC FC
VE_Port Entity VF_Port Entity VE_Port Entity VF_Port Entity

FCoE FCoE_LEP FCoE FCoE FCoE_LEP FCoE

FCoE_LEP Entity FCoE_LEP Entity FCoE_LEP Entity FCoE_LEP Entity
FCoE_LEP FCoE_LEP
FCoE FCoE
Controller Controller

Lossless Ethernet MAC Ethernet_Port Lossless Ethernet MAC Ethernet_Port

Ethernet_Port Ethernet_Port
Lossless Ethernet Lossless Ethernet
Bridging Element Bridging Element
Ethernet_Port Ethernet_Port Ethernet_Port Ethernet_Port Ethernet_Port Ethernet_Port Ethernet_Port Ethernet_Port

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 61

FCoE: Initial Deployment

SAN A SAN B
10 GE
Backbone

VF_Ports

VN_Ports

10 GE
4/8 Gbps FC

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 62

© 2006, Cisco Systems, Inc. All rights reserved. 31

Presentation_ID.scr
 FCoE: Adding Blade Servers
SAN A SAN B
10 GE
Backbone

VF_Ports

VN_Ports
10 GE
4/8 Gbps FC

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 63

FCoE: Adding Native FCoE Storage

SAN A SAN B
10 GE
Backbone

VN_Ports

VF_Ports

VN_Ports
10 GE
4/8 Gbps FC

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 64

© 2006, Cisco Systems, Inc. All rights reserved. 32

Presentation_ID.scr
 FCoE: Adding VE_Ports
SAN A SAN B
10 GE
Backbone

VE_Ports

VF_Ports

VN_Ports
10 GE
4/8 Gbps FC

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 65

FCoE Addressing and Forwarding

ƒ FCoE frames have:

MAC addresses (hop-by-hop)
FC addresses (end-to-end)

FC Storage FCID 1.1.1

FC Domain 3 FC Domain 1 MAC C
FCID 7.1.1 FC Domain 7 MAC A MAC B
Ethernet Ethernet
Fabric Fabric
FC Fabric FC Fabric

Dest. = MAC B Dest. = MAC C

D_ID = FC-ID (1.1.1)
S_ID = FC-ID (7.1.1)
FC Frame
D_ID = FC-ID (1.1.1) Srce. = MAC A Srce. = MAC B
S_ID = FC-ID (7.1.1) FCoE
D_ID = FC-ID (1.1.1) D_ID = FC-ID (1.1.1)
S_ID = FC-ID (7.1.1) S_ID = FC-ID (7.1.1) Frame
FC Frame

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 66

© 2006, Cisco Systems, Inc. All rights reserved. 33

Presentation_ID.scr
 FCoE MAC Addresses

ƒ VE_Ports and VF_Ports always use MAC addresses

derived from the switch pool
ƒ VN_Ports may use two types of MAC addresses
SPMA (Server Provided MAC Addresses)
FPMA (Fabric Provided MAC Addresses)

ƒ MAC Addresses are negotiated in FIP

ƒ Initial deployment will use FPMA only

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 67

The Mapped MAC Addresses

ƒ A dedicated MAC address for each FC-ID

Assigned by the FC fabrics
Consistent with the FC model
OUIs with U/L = 1 (Local addressing), called FC-MAPs
Multiple FC-MAPs may be supported (one per FC fabric)

24 Bits 24 Bits
FC-MAP FC-ID
(ex 02-12-34) 7.8.9

MAC FC-MAP FC-ID

Address (ex 02-12-34) 7.8.9
48 Bits
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 68

© 2006, Cisco Systems, Inc. All rights reserved. 34

Presentation_ID.scr
 Initial Login Flow Ladder
ENode FCoE Switch
Solicitati
on
Discovery Discovery
Advertisement
FIP:
FCoE
Initialization
Protocol
FLOGI/FDISC FLOGI/FDISC Accept

FC Command FCOE
FC Command Responses Protocol

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 69

FIP Frame: Contains FIP Operation

Destination MAC Address

Source MAC Address

IEEE 802.1Q Tag

ET = FIP Ver Reserved

Encapsulated FIP Operation

(Self-Describing Length)

PAD to Minimum Length or Mini-Jumbo Length

Ethernet FCS

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 70

© 2006, Cisco Systems, Inc. All rights reserved. 35

Presentation_ID.scr
 FIP Descriptors (1)
Type = 1 Len = 4 Reserved Priority

Type = 2 Len = 8
MAC Address

Type = 3 Len = 8 Reserved

FC-MAP

Type = 4 Len = 12 Reserved

Switch_Name

Type = 5 Len = 12 Reserved

Fabric_Name

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 71

FIP Descriptors (2)

Type = 6 Len = 12 Reserved

Port_Name

Type = 7 Len = XX Reserved

FLOGI Request, FLOGI LS_ACC/LS_RJT

NPIV FDISC Request, FDISC LS_ACC/LS_RJT
Fabric LOGO Request, LOGO LS_ACC/LS_RJT
(No SOF/EOF / FC-CRC?)

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 72

© 2006, Cisco Systems, Inc. All rights reserved. 36

Presentation_ID.scr
 FIP
Multicast Solicitation from H2
MAC (H1) FCF-MAC (A)

H1 FCF A
Lossless
Ethernet FC
Bridge Fabric
H2
FCF A

MAC (H2) FCF-MAC (B)

All-MACs
ƒ Solicitation identifies VF_Port capable FCF-MACs
MAC(H2)
with compatible addressing capabilities
Solicitation (FIP)
ƒ Other parameters may include ENode’s
[F=0, S=0, MAC(H2),
Capability, Other]
Port_Name for optional duplicate MAC
address detection
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 73

FIP
Unicast Advertisements from A and B
MAC (H1) FCF-MAC (A)

H1 FCF A
Lossless
Ethernet FC
Bridge Fabric
H2
FCF A

MAC (H2) FCF-MAC (B)

MAC(H2) MAC(H2)

FCF-MAC(A) FCF-MAC(B)
H2’s FCF list: ƒ FCF not meeting
Mini-jumbo Mini-jumbo
Advertisement (FIP) Advertisement (FIP) FCF-MAC(A) [J] capability of ENode
[S=1, F=1, Priority, FC-MAP, [S=1, F=1, Priority, FC-MAP, FCF-MAC(B) [J] does not reply
FCF-MAC(A), Switch_Name, FCF-MAC(B), Switch_Name,
Fabric_Name, Capability, Fabric_Name, Capability,
Other] Other]
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 74

© 2006, Cisco Systems, Inc. All rights reserved. 37

Presentation_ID.scr
 FIP
FLOGI Request
MAC (H1) FCF-MAC (A)

H1 FCF A
Lossless
Ethernet FC
Bridge Fabric
H2
FCF A

MAC (H2) FCF-MAC (B)

FCF-MAC(A) FCF-MAC(B)

MAC (H2) MAC(H2)

ƒ Capability agreed
FLOGI Request (FIP) FLOGI Request (FIP)
during discovery
[FC Header, FLOGI data, [FC Header, FLOGI data,
Proposed MAC’(H2)] Proposed MAC’’(H2)]

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 75

FIP
FLOGI LS_ACC
MAC (H1) FCF-MAC (A)

H1 FCF A
Lossless
Ethernet FC
Bridge Fabric
H2
FCF A

MAC (H2) FCF-MAC (B)

MAC (H2) MAC(H2)

FCF-MAC(A) FCF-MAC(B)
ƒ ENode uses MAC address
in FIP FLOGI LS_ACC as the
FLOGI LS-ACC (FIP) FLOGI LS-ACC (FIP) VN_Port MAC address for the
[FC Header, LS_ACC data, [FC Header, LS_ACC data, FC-ID contained in the FLOGI data
Approved MAC(H2)’] Approved MAC(H2)’’] for subsequent FCoE frames

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 76

© 2006, Cisco Systems, Inc. All rights reserved. 38

Presentation_ID.scr
 FCoE
Data Transfer
MAC (H1) FCF-MAC (A)

H1 FCF A
Lossless
Ethernet FC
Bridge Fabric
H2
FCF A

MAC (H2)
MAC (H2)’ FCF-MAC (B)
MAC (H2)’’

FCF-MAC(A) FCF-MAC(B)
ƒ All subsequent FCoE frames use
MAC(H2)’ MAC(H2)’’ granted MAC address and assigned
Fibre Channel Frame Fibre Channel Frame FC-ID
(FCoE) (FCoE) FIP frames continue to use MAC(H2)
For SPMA, MAC(H2)’ = MAC(H2)’’ = MAC(H2)
[FC SOF, FC Header, FC [FC SOF, FC Header, FC For FPMA, MAC(H2)’ and MAC(H2)’’ use
Data, FC CRC FC EOF] Data, FC CRC, FC EOF] FC-IDs as low order 24 bits
and FC-MAP for upper 24 bits
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 77

The Most Asked Question:

Is FCoE Routable?

FCoE

1, 2, 4, (8), 10 Gbps 1, 10 . . . Gbps 10, 20 Gbps

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 78

© 2006, Cisco Systems, Inc. All rights reserved. 39

Presentation_ID.scr
 Is FCoE Routable?
ƒ Most folks mean “Is FCoE IP-routable”
The answer is NO, there is no IP layer in FCoE
This was a design goal to keep FCoE simple
FC-BB-5 contains FCIP that is “IP-routable”

ƒ FCoE is FC-routable
FCoE switches may forward FC frames across different
Ethernet clouds
FCoE switches may forward FC frames over the Internet using FCIP

FCoE FCIP FCIP FCoE

IP Cloud

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 79

CNA: Converged Network Adapter

LAN HBA CNA
10GbEE

10GbEE
10GbE
10GbE

HBA

HBA

Link Link Link

Fibre Channel

Fibre Channel
Ethernet

Ethernet

Ethernet

PCIe PCIe PCIe

Ethernet Fibre Channel Ethernet Fibre Channel

Drivers Drivers Drivers Drivers

Operating System Operating System

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 80

© 2006, Cisco Systems, Inc. All rights reserved. 40

Presentation_ID.scr
 View from Operating System
ƒ Standard drivers
ƒ Same management
ƒ Operating system sees:
Dual-port, 10 Gigabit
Ethernet adapter
Dual-port, 4 Gbps Fibre
Channel HBAs

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 81

Open-FCoE Software
HBA FCoE

HBA Mgmt Plane FCoE Mgmt Plane

Linux Kernel Linux Kernel

File System layers File System layers

SCSI Layer
SCSI Layer

OpenFC Layer

HBA Driver FCoE Layer

Net Device
HBA Ethernet Driver
Ethernet

Fibre Server Ethernet Server

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 82

© 2006, Cisco Systems, Inc. All rights reserved. 41

Presentation_ID.scr
 Open-FCoE Software: How to Get It

ƒ Open source project

ƒ Open-FCoE.org—source Git trees
ƒ Or Open-FCoE source package—TBD
ƒ Install a Linux Red Hat EL5, Fedora Core 7,
or SuSE 10 distribution
ƒ Update kernel to 2.6.23 or later
ƒ Install: see Quick Start Guide at open-fcoe.org
ƒ Use switch, soft-target, or gateway

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 83

Wireshark

ƒ Once known as Ethereal

ƒ Captures and displays network traffic
ƒ Available from: http://wireshark.org/
ƒ Sample trace file
/common/openfc/traces/fcoe-t11.cap

ƒ Use tcpdump to capture

tcpdump –i eth0 –s 0 –w /tmp/fcoe.cap

ƒ Screenshots/demo

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 84

© 2006, Cisco Systems, Inc. All rights reserved. 42

Presentation_ID.scr
 Wireshark Screenshot

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 85

Section 4
Case Studies

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 86

© 2006, Cisco Systems, Inc. All rights reserved. 43

Presentation_ID.scr
 Current Data Center Environment

LAN Core
SAN-A SAN-B

Distribution
MDS 9500
4
Cisco Catalyst® 6509

4 x 4G FC 1 x 10 GE STP BLK

POD 1 POD N Access

MDS 9500
Cisco Catalyst 6509

NIC Teaming
Active/Standby

Discrete
1 GE NICs
and
FC HBA
Server Cabinet 1 Server Cabinet N Server Cabinet 1 Server Cabinet N

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 87

Top-of-Rack Consolidated I/O

I/O Consolidation at Access

LAN Core
SAN-A SAN-B
8
Distribution
8 MDS 9500
Cisco Catalyst 6509

STP BLK

POD 1 POD N Access

Nexus 5000

Adapter: CNA
Converged
Network
Adapter
10 GE/FCoE

Server Cabinet Pair 1 Server Cabinet Pair N Server Cabinet Pair 1 Server Cabinet Pair N

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 88

© 2006, Cisco Systems, Inc. All rights reserved. 44

Presentation_ID.scr
 Top-of-Rack Consolidated I/O
Ethernet Host Virtualizer

LAN Core
SAN-A SAN-B
8
Distribution
8 MDS 9500
Cisco Catalyst 6509

Ethernet Host Virtualizer

Active/Active

POD 1 POD N Access

Nexus 5000

Adapter: CNA
Converged
Network
Adapter
10 GE/FCoE

Server Cabinet Pair 1 Server Cabinet Pair N Server Cabinet Pair 1 Server Cabinet Pair N

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 89

Top-of-Rack Consolidated I/O

VSS Support at Aggregation

LAN Core
SAN-A SAN-B
8
VSS Distribution
MDS 9500
8
Cisco Catalyst 6509

VSS Support
at Aggregation

POD 1 POD N Access

Nexus 5000

Adapter: CNA
Converged
Network
Adapter
10 GE/FCoE

Server Cabinet Pair 1 Server Cabinet Pair N Server Cabinet Pair 1 Server Cabinet Pair N

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 90

© 2006, Cisco Systems, Inc. All rights reserved. 45

Presentation_ID.scr
 Physical Topology—40 Servers, ToR
LAN SAN-A SAN-B

4x10 GE
Ports

4x10 GE 6x4 GFC

Ports 6x4 GFC

Nexus 5000

40x10 GE Ports
per Switch

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 91

Physical Topology—200 Servers

40 4GFC
SAN-A SAN-B
LAN

4x10 GE
Ports 6x4 GFC

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 92

© 2006, Cisco Systems, Inc. All rights reserved. 46

Presentation_ID.scr
 Blade Servers with
Copper Pass-Through
DC Core
MDS MDS
SAN-Core SAN-Core
8

Distribution
MDS 9500
8 Cisco Catalyst 6509

POD 1 POD N Access

Nexus Family

Blade Server
Copper
Pass-Through

Adapter: CNA
Converged
Network
Adapter
10 GE/FCoE
Blade Server 1 Blade Server N Blade Server 1 Blade Server N
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 93

Blade Servers with

Ethernet Switches
DC Core
MDS MDS
SAN-Core SAN-Core
8

Distribution
MDS 9500
8 Cisco Catalyst 6509

POD 1 POD N Access

Nexus Family

Blade Server
Ethernet-Only
Switch

Adapter: CNA
Converged
Network
Adapter
10 GE/FCoE
Blade Server 1 Blade Server 1 Blade Server 1 Blade Server 1
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 94

© 2006, Cisco Systems, Inc. All rights reserved. 47

Presentation_ID.scr
 Physical Topology—
20 Blade Server, ToR
LAN SAN-A SAN-B

2x10 GE
Ports

2x10 GE 4x4 GFC

Ports 4x4 GFC

Nexus 5000

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 95

Topology—100 Blade Servers, ToR

LAN SAN-A SAN-B

4x10 GE
Ports

8x4 GFC
8x4 GFC
4x10 GE
Ports

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 96

© 2006, Cisco Systems, Inc. All rights reserved. 48

Presentation_ID.scr
 Access Nexus 5000
Distribution Nexus 7000
LAN Core
SAN-A SAN-B
8

Distribution
MDS 9500
8 Nexus 7000

Layer 2 Multipath

POD 1 POD N Access

Nexus 5000

Adapter: CNA
Converged
Network
Adapter
10 GE/FCoE

Server Cabinet Pair 1 Server Cabinet Pair N Server Cabinet Pair 1 Server Cabinet Pair N

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 97

Consolidation in the Distribution Layer

LAN Core
SAN-A SAN-B
8

Distribution
MDS 9500
8 Nexus 7000

Layer 2 Multipath

POD 1 POD N Access

Nexus 5000

Adapter: CNA
Converged
Network
Adapter
10 GE/FCoE

Server Cabinet Pair 1 Server Cabinet Pair N Server Cabinet Pair 1 Server Cabinet Pair N

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 98

© 2006, Cisco Systems, Inc. All rights reserved. 49

Presentation_ID.scr
 Conclusions

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 99

Challenges

ƒ FCoE redefines consolidated scenarios

Ethernet switch manufacturers will try to enter the FC
switching market
FC switch manufacturers will try to enter the Ethernet
switching market
HBA manufacturers will try to enter the NIC market
NIC manufacturers will try to enter the HBA market

ƒ Deep integration with virtualization will take some time

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 100

© 2006, Cisco Systems, Inc. All rights reserved. 50

Presentation_ID.scr
 Web Pointers
ƒ PCI Express
http://en.wikipedia.org/wiki/Pci_express

ƒ IEEE 802.3
http://standards.ieee.org/getieee802/802.3.html

ƒ Improvements to Ethernet
http://www.nuovasystems.com/EthernetEnhancements-Final.pdf

ƒ IEEE 802.1 activities

http://www.ieee802.org/1/files/public/docs2007/new-cm-barrass-pause-proposal.pdf
http://www.ieee802.org/1/files/public/docs2007/new-cm-pelissier-enabling-block-storage-0705-v01.pdf
http://www.ieee802.org/1/files/public/docs2007/au-ko-fabric-convergence-0507.pdf
http://www.ieee802.org/1/pages/802.1au.html
http://www.ieee802.org/1/files/public/docs2008/az-wadekar-dcbcxp-overview-rev0.2.pdf

ƒ FCoE
http://www.fcoe.com/
http://www.t11.org/
http://www.open-fcoe.org/
http://www.fibrechannel.org/OVERVIEW/FCIA_SNW_FCoE_WP_Final.pdf

ƒ TRILL
http://www.ietf.org/html.charters/trill-charter.html
BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 101

Thank You

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 102

© 2006, Cisco Systems, Inc. All rights reserved. 51

Presentation_ID.scr
 Recommended Reading

ƒ Continue your Cisco Live

learning experience with further
reading from Cisco Press
ƒ Check the Recommended
Reading flyer for suggested
books

Available Onsite at the Cisco Company Store

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 103

Complete Your Online

Session Evaluation
ƒ Give us your feedback and you could win Don’t forget to activate
fabulous prizes. Winners announced daily. your Cisco Live virtual
account for access to
ƒ Receive 20 Passport points for each session all session material
evaluation you complete. on-demand and return
for our live virtual event
ƒ Complete your session evaluation online now in October 2008.
(open a browser through our wireless network Go to the Collaboration
to access our portal) or visit one of the Internet Zone in World of
stations throughout the Convention Center. Solutions or visit
www.cisco-live.com.

BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 104

© 2006, Cisco Systems, Inc. All rights reserved. 52

Presentation_ID.scr
 BRKSAN-2821
14572_05_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 105

© 2006, Cisco Systems, Inc. All rights reserved. 53

Presentation_ID.scr