Operating System View

C97-485980-00 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Confidential 1
 Evolution of 10G Ethernet Physical Media
Role of Transport in Enabling these Technologies!
Mid 1980’s Mid 1990’s Early 2000’s Late 2000’s

10Mb 100Mb 1Gb 10Gb

UTP Cat 3 UTP Cat 5 UTP Cat 5 X2

SFP Fiber SFP+ Cu (BER better than 10 -18 )
SFP+ Fiber
Cat 6/7
Technology Cable Distance Power Transceiver
(each side) Latency (link)
SFP+ CU Twinax 7m ~0.1W ~0.1μs
Copper
SFP+ USR MM OM2 10m 1W ~0.1μs
Ultra short reach MM OM3 100m
SFP+ SR MM 62.5 μm 26-33m 1W ~0.1μs
Short reach MM 50 μm 66-300m
SFP+ LR SMF G.652 10km 0.5W
Long range
10GBASE-T Cat6 55m ~8W 2.5μs
Cat6a/7 100m ~8W 2.5μs
Cat6a/7 30m ~4W 1.5μs
C97-485980-00 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Confidential 2
Introduction to FCoE

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential (please do not distribute) 3
 What is Fibre Channel over Ethernet?

 From a Fibre Channel standpoint it’s

FC connectivity over a new type of cable called… an Ethernet
cloud

 From an Ethernet standpoints it’s

Yet another ULP (Upper Layer Protocol) to be transported

FCoE is an extension of Fibre Channel

onto a Lossless Ethernet fabric

C97-485980-00 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Confidential 4
 Unified Fabric Overview
Fibre Channel over Ethernet (FCoE)

FCoE Benefits
•• Mapping
Mapping of of FC
FC Frames
Frames over
over • Fewer Cables
Ethernet
Ethernet •• Both
Both block
block I/O
I/O &
& Ethernet
Ethernet
•• Enables
Enables FC
FC to
to Run
Run traffic
traffic co-exist
co-exist on
on same
same
on
on aa Lossless
Lossless cable
cable
Ethernet
Ethernet Network
Network
• Fewer adapters needed
• Overall less power
Ethernet • Interoperates with
existing SAN’s
Fibre •• Management
Management SAN’s
SAN’s
Channel remains
remains constant
constant
Traffic
• No Gateway

05/26/20
C97-485980-00 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Confidential 5 5
 FCoE Enablers

 10Gbps Ethernet
 Lossless Ethernet
Matches the lossless behavior guaranteed in FC by B2B credits
 Ethernet jumbo frames

Normal ethernet frame, ethertype = FCoE

Same as a physical FC frame
Ethernet

Header
Header
Header
FCoE

CRC
EOF
FC Payload

FCS
FC

Control information: version, ordered sets (SOF,

EOF)
C97-485980-00 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Confidential 6
 Unified I/O
Fibre Channel over Ethernet (FCoE)
FCoE is managed like FC at initiator, target, and switch level

Easy to Completely based on the

Understand FC model

Same Same host-to-switch and switch-to-

Operational Model switch behavior as FC
FCoE is
FibreTechniques
Same Channel of e.g. in order delivery,
Traffic Management FSPF load balancing

Same Management WWNs, FC-IDs, hard/soft

and Security Models zoning, DNS, RSCN

C97-485980-00 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Confidential 7
 Network Stack Comparison

SCSI SCSI SCSI SCSI SCSI

iSCSI FCP FCP FCP

FC FC FC

FCIP Less Overhead

than FCIP, iSCSI
TCP TCP

IP IP FCoE

Ethernet Ethernet Ethernet

PHYSICAL WIRE
SCSI iSCSI FCIP FCoE FC
C97-485980-00 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Confidential 8
FCoE Frame Format

Destination MAC Address

Source MAC Address

(IEEE 802.1Q Tag)

ET = FCoE Ver Reserved

Reserved

Reserved

Reserved SOF

Encapsulated FC Frame (with CRC)

EOF Reserved

FCS

C97-485980-00 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Confidential 9
FCoE Standards
Defined

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential (please do not distribute) 11
 A larger picture
 IEEE 802
• Evolution of Ethernet (10 GE, 40 GE, 100 GE, copper and fiber)
• Evolution of switching (Priority Flow Control, Enhanced Transmission,
Congestion Management, Data Center Bridging eXchange)

 INCITS/T11
• Evolution of Fibre Channel (FC-BB-5)
• FCoE (Fibre Channel over Ethernet)

 IETF
• Layer 2 Multi-Path
•TRILL (Transparent Interconnection of Lots of Links)

C97-485980-00 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Confidential 12
 DCE versus DCB
 DCE is an old Cisco marketing term
 Cisco is now using the term DCB
The term IEEE uses

 Cisco supports the DCB standard activity

By implementing products that are DCB compliant

 CIN-DCBX – Cisco, Intel, Nuova Data Center Bridging Exchange protocol, pre-
standard
 CEE-DCBX – Converged Enhanced Ethernet Data Center Bridging Exchange
protocol, which is standards base

C97-485980-00 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Confidential 13
 What’s FC-BB-5

 FC-BB-5 covers the majority of the FC features, using

Ethernet
 From an Ethernet perspective, FC-BB-5 is
Ethernet control plane referred to as FIP (Fibre Channel over
Ethernet Initiation Protocol)
discover and build virtual paths between end points
Ethernet data plane providing FCoE forwarding
including both FC control plane and FC data plane (FCF)

C97-485980-00 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Confidential 14
 FC-BB-6

 It is an active working group of T11 that will discuss the

future of FCoE or FCoE v2.0
 It is just started, 18 months to have a standard
Approximate target spring 2011

 You can track it on

http://www.fcoe.com

C97-485980-00 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Confidential 15
 Protocol Organization

FCoE itself … FIP (FCoE initiation

 Is the data plane protocol
 It is used to carry most of the
FC frames and all the SCSI traffic
protocol)
 It is the control plane protocol
 It is used to discover the FC entities
connected to an Ethernet cloud
 It is used to login to and logout from the
FC fabric

The two protocols have:

• Two different Ethertypes
• Two different frame formats
C97-485980-00 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Confidential 16
 What’s NOT FC-BB-5

 FC-BB-5 doesn’t deal with how lossless is realized in

Ethernet
no Priority Flow Control, Bandwidth Management, etc.

 FC-BB-5 doesn’t deal with management functions

C97-485980-00 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Confidential 17
 IEEE DCB standards status
DCB technologies allow Ethernet to be lossless and to
manage bandwidth allocation of SAN and LAN flows
Feature / Standard Standards Status
IEEE 802.1Qbb
PAR approved
Priority Flow Control (PFC)
Enable multiple traffic types to share a common 1.0 published
Ethernet link without interfering with each other

IEEE 802.1Qaz
PAR approved
Bandwidth Management (ETS)
Enable consistent management of QoS at the 1.0 published
network level by providing consistent scheduling

Data Center Bridging Exchange

Protocol (DCBX)
This is part of IEEE 802.1Qaz
Management protocol for enhanced Ethernet
capabilities

C97-485980-00 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Confidential 18
Data Center Ethernet:
PFC & Bandwidth Management

CoS based
Priority Flow Control Bandwidth Management
Transmit Queues Receive Buffers
Ethernet Link Offered Traffic 10 GE Realized Traffic Utilization
Zero Zero
Zero
3G/s 3G/s 2G/s 3G/s HPC Traffic 2G/s
One One 3G/s
Two Two
3G/s Storage Traffic 3G/s
Three STOP PAUSE Three Eight 3G/s 3G/s 3G/s 3G/s
Virtual
Four Four
Four Lanes
Five Five 3G/s LAN Traffic 5G/s
3G/s 4G/s 6G/s
Six Six
Six 4G/s

Seven
Seven Seven
t1 t2 t3 t1 t2 t3

• Enables lossless behavior • Enables Intelligent sharing of

for each class of service bandwidth between traffic classes
• PAUSE sent per virtual lane control of bandwidth
when buffers limit exceeded • 802.1Qaz Enhanced Transmission

05/26/20
C97-485980-00 © 2008 Cisco Systems, Inc. All rights reserved.
Nuova Systems Inc.
Cisco Confidential
19 19
DCBX Overview
Auto-negotiation of capability and configuration
Priority Flow Control capability and associated CoS values

Allows one link peer to push config to other link peer

Link partners can choose supported features and willingness to accept

Discovers FCoE Capabilities

Responsible for Logical Link Up/Down signaling of
Ethernet and FC
DCBX negotiation failures will result in:
vfc not coming up
Per-priority-pause not enabled on CoS values with PFC configuration

http://download.intel.com/technology/eedc/dcb_cep_spec.pdf
http://www.ieee802.org/1/files/public/docs2008/
C97-485980-00 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Confidential 20
 FCoE control plane

C97-485980-00 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Confidential 21
 FIP: FCoE Initialization Protocol
 FCoE VLAN discovery
Automatic discovery of FCoE VLANs
 Device discovery
ENodes discover VF_Port capable FCF-MACs for VN_Port to VF_Port Virtual
Links
VE_Port capable FCF-MACs discover other VE_Port capable FCF-MACs for
VE_Port to VE_Port Virtual Links
The protocol verifies the Lossless Ethernet network supports the required Max
FCoE Size
 Virtual Link instantiation
Builds on the existing Fibre Channel Login process, adding the Negotiation of the
MAC address to use
Fabric Provided MAC Address (FPMA), or
Server Provided MAC Address (SPMA)
 Virtual Links maintenance
Timer based
C97-485980-00 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Confidential 22
Server Provided MAC Fabric Provided MAC
Addresses Addresses
Adapter uses burned-in or configured MAC address assigned for each FC_ID:
MAC address: Consistent with the Fibre Channel model
Multiple FC-MAPs may be supported
Consistent with the Ethernet
model One per SAN
No table needed for Encapsulation
FCF needs a table to map between
MAC addresses and FC_IDs Multiple MACs may be needed for NPIV

FC-MAP FC-ID
MAC (0E-FC-00) 7.8.9
Address 24 24
bits bits
Burned in or Configured FC-MAP FC-ID
(0E-FC-00) 7.8.9
48 48
bits bits
Cisco Nexus 5000 uses FPMA
C97-485980-00 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Confidential 23
 Initial Login Flow ladder
ENode FCoE Switch

VLAN VLAN
Discovery Discovery

Solicitat
ion FIP:
FCF FCF FCoE
Discovery Advertisement Discovery Initialization
Protocol

FLOGI/FDISC FLOGI/FDISC Accept

FC Command FCOE
FC Command responses Protocol

C97-485980-00 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Confidential 24