DC Nexus Design

Data Center Access Design
with Nexus Switches

1K
Cisco Nexus
x86
Christian Hasse
chhasse@cisco.com
25.03.2010
Presentation_ID © 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 1
Agenda
Elements of the Evolving Data Centre

Access
Nexus Layer 2 Foundations
Virtualized Access Layer Design
vPC Design Principles
Extending vPC from access to aggregation
Nexus 2000 Fabric Extender – Physical
Virtualization
Nexus 1000v – Embedded Virtual Switching
Nexus 4000 – Unified I/O
Summary
Introduction
Session Objectives
At the end of the session, the participants should be able to:

Talk to your customers about the evolving Data Center Access
Architecture
Be familiar with Nexus-based DC design best practices with
particular focus on the Access Layer part of the design
Easily implement vPC-based LAN switching designs (including
Nexus 7000, Nexus 5000, Nexus 2000) for POC or
Design/Deployment
Understand how to integrate Unified IO and Nexus1000v into the
Evolving Access Layer Architecture
Understand the integration of UCS into the Evolving Access Layer
Architecture
Agenda

Access
Virtualization
Summary
Elements of the Evolving Data Centre Access
Evolving Access Layer
The Data Centre Access Layer is Core

evolving
Embedded Blade Switches
Embedded Virtual Switches
Aggregation
Distribution of the Access
Layer
Challenges this brings
Increase in the size of the
access topology (STP growth)
Need to understand the High
Availability design Distribution of the
requirements Blade Chassis
Access Layer
1G/10G Migration
Increase in the number of Embedded
Switching Embedded
management points Switching in the
Server (vSwitch)
Policy Boundaries
Increased Complexity and
Scope of the Access Layer
Assembling the New Data Centre Edge
Core/Aggregation Layer
Core
MDS
Nexus 7000
Nexus 5000/7000
Access Layer
10 GigE/FCOE
FCOE
Nexus 2000
Virtual Access Layer

blade1
slot 1 blade1
slot 1 blade1 blade1
slot 1 slot 1
NEXUS
blade2 blade2 blade2 blade2
NEXUS NEXUS
slot 2 slot 2 slot 2 slot 2
blade3 blade3 blade3
VM VM
slot 3 slot 3 blade3
VM VM VM VM
slot 3 slot 3
blade4
slot 4 blade4
slot 4 blade4
slot 4 blade4
slot 4
NEXUS
VM VM
blade6
slot 6 blade6
slot 6 blade6
slot 6 blade6
slot 6
1000v
blade7
1000v 1000v
blade7
slot 7 blade7
slot 7
VM VM
blade8
VM VM VM VM
blade8
slot 8 blade8
slot 8
1000v
VM VM
VM VM
blade1
slot 1
VM VM
blade2
slot 2
blade3
slot 3
blade1
slot 1
blade2
slot 2
blade3
slot 3
blade1
slot 1
VM VM
blade2
slot 2
blade3
slot 3
blade1
slot 1
blade2
slot 2
blade3
slot 3
VM VM
blade4
slot 4 blade4
slot 4 blade4
slot 4 blade4
slot 4
blade6
slot 6 blade6
slot 6 blade6
slot 6 blade6
slot 6
blade7
slot 7 blade7
slot 7 blade7
slot 7 blade7
slot 7
blade8
slot 8 blade8
slot 8 blade8
slot 8 blade8
slot 8

blade2
slot 2 blade2
slot 2 blade2
slot 2 blade2
slot 2
blade4
slot 4 blade4
slot 4 blade4
slot 4 blade4
slot 4
blade5
slot 5 blade5
slot 5 blade5
slot 5 blade5
slot 5
blade6
slot 6 blade6
slot 6 blade6
slot 6 blade6
slot 6
blade7
slot 7 blade7
slot 7 blade7
slot 7 blade7
slot 7
blade8
slot 8 blade8
slot 8 blade8
slot 8 blade8
slot 8
1G and 10GE Rack 1G and 10GE Blade N4K - DCB Blade 10GE Blade
Mount Servers Servers (HP) UCS Compute UCS Compute
Switch Pod Pod
Pass-Thru IBM/Dell
HP/IBM/Dell
Nexus Data Center Access Technologies
VM VM VM VM
Nexus
1000V
VEM
vSphere
Virtual Port-Channel Nexus 2000/FEX Nexus 1000V + VN-Link
Network interconnect
concepts
Network Protocols
part of the design
Enhanced Spanning Tree FCoE
Provisioning Ethernet
Fibre
Channel
Agenda
Access
Virtualization
Summary
A Closer Look at Layer 2
Do’s:
Use MST for large scale Layer2 environments:
With MST CPU load is relatively low even with large number of trunks/VLAN
Check documented VLAN/STP scale numbers for target releases prior to make
your design decision:
Nexus 7000 (Release 4.2):

http://www.cisco.com/en/US/docs/switches/datacenter/sw/4_2/nx-
os/layer2/configuration/guide/Cisco_Nexus_7000_Series_NX-
OS_Layer_2_Switching_Configuration_Guide_Release_4.2_appendix1.html#concept_1A55D144F3504CC49B22DD500
E766A4E
Nexus 5000 (Release 4.1(3)):

http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/configuration/nxos/Cisco_Nexus_5000_Se
ries_NX-OS_Software_Configuration_Guide_appendix1.html
Use newest STP enhancements like Bridge Assurance

Avoid STP blocked ports by using vPC
Don’ts:
Do NOT disable Spanning Tree as vPC still relies on STP as failsafe back-up
Spanning Tree Design Considerations
NX-OS STP modes

Rapid-PVST+ (Default mode)
N N
MST (Supported)
NR N R
Network Ports
PVST (Not supported, but interoperable) All Send BPDUs
NX-OS always uses Extended System ID N N
NX-OS uses a fixed STP link cost for

Etherchannel links (based on number of links Edge Ports E E E
No BPDUs
configured, not number active as in IOS)
Understand the three port modes
“Edge” port type replaces spanning-tree
portfast Root port E Edge port
“Network” port type for bridge-to-bridge Alternate port N Network port
links Designated port R Root Guard
“Normal” for generic links in spanning
Interface Ethernet 1/5
tree switchport mode access
switchport access vlan 100
spanning-tree port type edge
Dispute Mechanism
BPDUs in Rapid-PVST carry information

about STP port roles
N N
Only one “designated” port can exist per

segment without creating a loop N R N R
Dispute Mechanism performs

consistency check validating path to
root against port role
N N
Requires no configuration, enabled

automatically
In all versions of NX-OS, available in Root port E Edge port
IOS on the Catalyst 6500 beginning Alternate port N Network port
12.2(33) SXI Designated port R Root Guard
%STP-2-DISPUTE_DETECTED: Dispute detected on port Ethernet1/2 on VLAN0700.
Nexus# sh spanning vlan 700 | in BLK

Eth1/2 Desg BLK 2000 128.130 Network P2p
Bridge Assurance
Specifies transmission of BPDUs
on all ports of type “network”. Stopped receiving
BPDUs!
Requires configuration, best Root
Malfunctioning
practice is to set global default to BPDUs switch
type “network”, default is
“normal” Network Network
Protects against unidirectional BA Inconsistent
links and peer switch software Network

BPDUs
Network
issues
BPDUs
In summary BA is the concept of Blocked
BA Inconsistent
IGPs hellos applied to STP Network Network
Stopped receiving
BPDUs!
Edge Edge
interface port-channel200
switchport mode trunk
switchport trunk allowed vlan 200-202
spanning-tree port type network
Root Guard Prevents Unwanted Changes to STP Topology
Secondary Root
Enable Root Guard on links Root Bridge
Bridge
connecting to access layer to
protect from edge switches
becoming root and causing sub-
N N
optimal traffic flow N R N R
Forces Layer 2 LAN interface to

be a designated port. If port
receives a superior BPDU, Root Should never
N N Should never
Guard puts the interface into the receive a superior receive a superior
BPDU
BPDU
root-inconsistent (blocked) state
Channel the trunk between
Root port Edge port
Distribution Switches so failure
E
Alternate port Network port

doesn’t break topology
N
Designated port R Root Guard
interface Ethernet1/32
description dc10-5020-4
switchport trunk allowed vlan 15,98,180-183
spanning-tree guard root
Spanning Tree Design Considerations – VLANs and MST
In Catalyst IOS the range of reserved VLANs is:

0, 4095, 1002-1005 reserved by the cat6k
http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SX/con
figuration/guide/vlans.html#wp1032562
In NX-OS the range of reserved VLANs is:
VLANs 3968 to 4047 and 4094 are reserved for internal use in each VDC
The software allocates a group of 80 VLAN numbers for those features,
like multicast and diagnostics, that need to use internal VLANs for their
operation.
http://www.cisco.com/en/US/partner/docs/switches/datacenter/sw/4_2/nx-
os/layer2/configuration/guide/Cisco_Nexus_7000_Series_NX-
OS_Layer_2_Switching_Configuration_Guide_Release_4.2_chapter3.html
#con_1273370
The Nexus 5000 supports up to 507 concurrent VLANs whose number
can be picked from any in the the above range.
Be mindful of this when creating MST regions encompassing
Catalyst and Nexus as you have to create region mapping for the
least common denominator
Spanning Tree Design
N Network port
E Edge or portfast port type
Data Center Core
- Normal port type
B BPDUguard
R Rootguard
L Loopguard
HSRP UDLD
HSRP Layer 3
ACTIVE STANDBY
Aggregation
N N Secondary
Primary
Root
Root
-
Layer 2 (STP + BA +UDLD)
N N N N N N -
R
R R R R R R R
Layer 2 (STP + BA + Root guard + UDLD)
N N
Access
N N N L
N L
E E E E E
B B B B B
Layer 2 (STP + BPDUguard)
vPC Design principles
Feature Overview
Allow a single device to use a

port channel across two upstream
switches
Eliminate STP blocked ports
Uses all available uplink
bandwidth Logical Topology without vPC
Dual-homed server operate in

active-active mode
Provide fast convergence upon
link/device failure
Reduce CAPEX and OPEX
Available on current and future
hardware (i.e. M1 and D1
generation cards on Nexus 7000)
Logical Topology with vPC
Agenda

Access
Virtualization
Summary
vPC and VSS Comparison
vPC VSS
(Virtual Port Channels) (Virtual Switching System)
Multi-Chassis Port Channel Yes Yes

Loop-free Topology (no blocking ports) Yes Yes
STP as a “fail-safe” protocol only Yes Yes
Two Independent Nodes,

Control Plane Single Logical Node
both active
Switch Redundancy (sup failover) Nexus 7000 with dual sups:
Inter-Chassis
Intra-chassis
Control Plane Protocols Instances per Node Single Instance
Independent Configs
Switch Configuration Single Configuration
(w/ consistency checker)
Maximum Physical Nodes 2 2
Yes on the Nexus 7000,
ISSU Support Disruptive
non-disruptive ISSU
10GE interfaces Sup720-10G, 6708, 6716

Inter-switch Link Hardware
Current Hardware (PFC3C mode)
vPC Terminology (common to Nexus 5000/7000)*
vPC peer – a vPC switch, one of a pair
vPC member port – one of a set of ports
vPC peer-keepalive vPC peer-link
(port channels) that form a vPC
link
vPC – the combined port channel between
CFS protocol
the vPC peers and the downstream device
vPC peer-link – Link used to synchronize
vPC peer state between vPC peer devices, must be
10GbE
vPC
vPC
vPC vPC peer-keepalive link – the keepalive
member
member link between vPC peer devices, i.e., backup
port
port to the vPC peer-link
vPC VLAN – one of the VLANs carried
over the peer-link and used to
vPC communicate via vPC with a peer device.
non-vPC non-vPC VLAN – One of the STP VLANs
device
not carried over the peer-link
CFS – Cisco Fabric Services protocol, used
*vPC is within the context of a VDC (applies only to N7k)
for state synchronization and configuration
validation between vPC peer devices
vPC Roles Primary (but Secondary
may be (but may be
Two Nexus running vPC appear as a Operational Operational
Secondary) Primary)
single STP entity
vPC Role defines which of the two
vPC peers processes BPDUs
7k01 7k02
Role matters for the behavior with
peer-link failures!
Role is defined under the domain
configuration
5k01
Lower priority wins if not, lower 5k02
system mac wins
Role is non-preemptive so
Operational Role is what matters
Operational Role may different from
the priorities configured under the
domain Primary (but Secondary
may be (but may be
Operational Operational
Secondary) Primary) 20
Presentation_ID © 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential
Building a vPC domain - Deployment Steps
Following steps are needed to build a vPC (Order does Matter!)”
1. Configure globally a vPC domain on both vPC devices
2. Configure a Peer-Keepalive link on both vPC peer switches (make sure is operational)
NOTE: When a vPC domain is configured the keepalive must be operational to allow a
vPC domain to successfully form.
3. Configure (or reuse) an interconnecting port-channel between the vPC peer switches
4. Configure the inter-switch channel as Peer-link on both vPC devices (make sure is
operational)
5. Configure (or reuse) Port-channels to dual-attached devices
6. Configure a unique logical vPC and join port-channels across different vPC peers
vPC peer- vPC peer-link

keepalive link
vPC peer
Standalone
Port-channel vPC vPC member port
Building a vPC domain – Peer-Keepalive (1 of 2)
Definition: vPC peer-keepalive link
Heartbeat between vPC peers

Active/Active (no Peer-Link) detection
Messages sent on 2 second interval
3 second hold timeout on peer-link loss
Fault Tolerant terminology is specific to VSS and deprecated in
vPC.
Packet Structure:
UDP message on port 3200, 96 bytes long (32 byte payload),
includes version, time stamp, local and remote IPs, and domain ID.
Recommendations:
Should be a dedicated link (1Gb is adequate)
Should NOT be routed over the Peer-Link
Can optionally use the mgmt0 interface (along with management
traffic)
As last resort, can be routed over L3 infrastructure
Building a vPC domain – Peer-Keepalive (2 of 2)
Cautions/Additional Recommendations:
On the Nexus 7000 when using supervisor management interfaces to carry the
vPC peer-keepalive, do not connect them back to back between the two
switches.
Use the management interface only if you have an out-of-band management
network (management switch in between).
Management Standby Management

Management Network Interface
Network Active Management
Interface
vPC_PK vPC_PK
vPC_PL
vPC_PL
Building a vPC domain – Peer-link requirements
Definition: vPC peer-link
Standard 802.1Q Trunk : Can Carry vPC and non vPC VLANs*
Carries Cisco Fabric Services messages
Carries flooded and/or orphan port traffic from attached devices
Carries STP BPDUs, HSRP Hellos, IGMP updates, etc.
Requirements:
Member ports must be 10GE interfaces
Peer-link are point-to-point. No other device should be inserted
between the vPC peers.
Recommendations (strong ones!)
Minimum 2x 10GbE ports on separate cards for best resiliency.
On Nexus 7000 is recommended a Dedicated 10GbE ports (not
shared mode ports)
*It is Best Practice to split vPC and non-vPC
VLANs on different Inter-switch Port-Channels.
5020 (config)# interface port-channel 10
5020 (config-if)# switchport mode trunk
5020 (config-if)# switchport trunk allowed <BETTER TO ALLOW ALL VLANS>
5020 (config-if)# vpc peer-link
5020 (config-if)# spanning-tree port type network
vPC Forwarding Rule
To achieve the “optimal usage of peer-link goal, learning on the Peer Link is
such that the MAC addresses learned on a peer’s vPC port are learned on
“local” vPC ports instead of Peer-link
This is done by disabling learning in the hardware on Peer-link ports. The
MAC addresses learned on a switch’s non vPC ports are notified to the peer
switch via CFSoE in address update messages indicating the interface on
which it is learned.
When the update message is received from the peer, the local switch will
update the local MAC table with the destination of
The local vPC port if it was learned on an vPC port on the peer switch
The Peer-link port if it was learned on a non vPC port (Orphan Port)
Example of Forwarding
CFSoE MAC table

update message MAC
=> vPC1
5k01 5k02 Packet(s) blocked on vPC

Packet member ports
Flooding
vPC1 vPC2
MACA MACB
vPC Control Plane – Consistency Check
Both switches in the vPC Domain maintain distinct
control planes
CFS provides for protocol state synchronization
between both peers (MAC Address table, IGMP
state, …)
System configuration must also be kept in sync
Currently a manual process with an automated
consistency check to ensure correct network Consistency
behavior Checks
Two types of consistency checks
Type 1 – Will put interfaces into suspend NX01 NX02
state to prevent invalid forwarding of
packets
Type 2 – Error messages to indicate
potential for undesired forwarding
behavior
Indicates that the behavior

described equally applies to
7k or 5k
Type 1 Consistency Check
Type 1 Consistency Checks are intended to prevent network
failures NX01 NX02
Incorrectly forwarding of traffic
Physical network incompatibilities
vPC will be suspended
interface port-channel51 interface port-channel51

switchport mode trunk switchport mode trunk
switchport trunk allowed vlan 100-105 switchport trunk allowed vlan 100-105
vpc 51 vpc 51
spanning-tree port type network spanning-tree port type network
spanning-tree guard root
5020# show vpc brief

Legend:
(*) - local vPC is down, forwarding via vPC peer-link
vPC status
----------------------------------------------------------------------------
id Port Status Consistency Reason Active vlans
------ ----------- ------ ----------- -------------------------- -----------
51 Po51 up failed vPC type-1 configuration -
incompatible - STP
interface port guard -
Root or loop guard
inconsistent
Type 2 Consistency Check
Type 2 Consistency Checks are intended to
prevent undesired forwarding NX01 NX02
vpc 51
vpc 51
5020# show vpc brief

vPC status
----------------------------------------------------------------------------
id Port Status Consistency Reason Active vlans
------ ----------- ------ ----------- -------------------------- -----------
51 Po51 up success success 100-104
2009 May 17 21:56:28 dc11-5020-1 %ETHPORT-5-IF_ERROR_VLANS_SUSPENDED: VLANs 105 on Interface port-

channel51 are being suspended. (Reason: Vlan is not configured on remote vPC interface)
vPC consistency
jumbo frames
VLAN Mapping
Be mindful of Global Type-1 inconsistencies
You need to change MST on both NX01 and NX02
Solution: define MST region mappings from the very beginning

of the deployment, for ALL VLANs, the ones that exist as well as the
ones that have not yet been created. Defining a region mapping
is orthogonal to creating a VLAN or not. mst region
vlans 1-5, 10
mst region mgmt0 mgmt0

vlans 1-5, 12
NX01 NX02
Indicates that the vPC vPC vPC

behavior described
equally applies to 7k
or 5k
Attaching to a vPC Domain - STP Interoperability
STP Uses:
• Loop detection (failsafe to vPC)
• Non-vPC attached device
• Loop management on vPC addition/removal
Requirements:
• Needs to remain enabled, but doesn’t dictate vPC
member port state
• Logical ports still count, need to be aware of number of
VLANs/port-channels deployed!
Best Practices:
• Not recommended to enable Bridge Assurance feature
on vPC channels (i.e. no STP “network” port type).
Tracked by CSCsz76892. vPC
STP
vPC is running to manage
• Make sure all switches in you layer 2 domain are loops outside of vPC’s
running with Rapid-PVST or MST (IOS default is non- direct domain, or before
rapid PVST+), to avoid slow STP convergence (30+
secs) initial vPC configuration
• Remember to configure portfast (edge port-type) on
host facing interfaces to avoid slow STP convergence
(30+ secs)
Attaching to a vPC Domain - ”My device can’t be dual attached!”
Recommendations (in order of preference):
1. ALWAYS try to dual attach devices using vPC (not applicable for routed links).
PROS: Ensures minimal disruption in case of peer-link failover and consistent behavior with vPC dual-
active scenarios. Ensures full redundant active/active paths through vPC.
CONS: None
2. If (1) is not an option – connect the device via a vPC attached access switch (could use VDC to create a
“virtual access switch”).
PROS: Ensures minimal disruption in case of peer-link failover and consistent behavior with vPC dual-
active scenarios. Availability limited by the access switch failure.
CONS: Need for an additional access switch or need to use one of the available VDCs. Additional
administrative burden to configure/manage the physical/Virtual Device
3. If (2) is not an option – connect device directly to (primary) vPC peer in a non-vPC VLAN* and provide
for a separate interconnecting port-channel between the two vPC peers.
PROS: Traffic diverted on a secondary path in case of peer-link failover
CONS: Need to configure and manage additional ports (i.e. port-channel) between the Nexus 7000
devices.
4. If (3) is not an option – connect device directly to (primary) vPC peer in a vPC VLAN
PROS: Easy deployment
CONS: VERY BAD. Bound to vPC roles (no role preemption in vPC) , Full Isolation on peer-link failure
when attached vPC toggles to a secondary vPC role.
* VLAN that is NOT part of any vPC and not present on vPC peer-link
Attaching to a vPC Domain - vPC and non-vPC VLANs (i.e. single attached .. )
P S P S
1. Dual Attached 2. Attached via VDC/Secondary Switch
Orphan
Ports
P S
P S
P Primary vPC
S Secondary vPC
3. Secondary ISL Port-Channel 4. Single Attached to vPC Device

Attaching to a vPC Domain - ”My device only does STP!”
Recommendations (in order of preference):
1. ALWAYS try dual attach devices using vPC
PROS: Ensures minimal disruption in case of peer-link failover and consistent behavior with
vPC dual-active scenarios. Ensures full redundant active/active paths through vPC.
CONS: None
2. If (1) is not an option – connect the device via two independent links using STP. Use non-
vPC VLANs ONLY on the STP switch.*
PROS: Ensures minimal disruption in case of peer-link failover and consistent behavior with
vPC dual-active scenarios. Ensures full redundant Active/Active paths on vPC VLANs.
CONS: Requires an additional STP port-channel between the vPC devices. Operational
burden in provisioning and configuring separate STP and vPC VLAN domains. Only
Active/Standby paths on STP VLANs.
3. If (2) is not an option – connect the device via two independent links using STP. (Use vPC
VLANs on this switch)
PROS: Simplify VLAN provisioning and does not require allocation of an additional 10GE
port-channel.
CONS: STP and vPC devices may not be able to communicate each other in certain failure
scenarios (i.e. when STP Root and vPC primary device do not overlap). All VLANs carried
over the peer-link may suspend until the two adjacency forms and vPC is fully
synchronized".
* Run the same STP mode as the vPC domain. Enable portfast/port type edge on host facing ports
Attaching to a vPC Domain - vPC and non-vPC VLANs (STP/vPC Hybrid)
Non vPC port-channel
P S SR PR
P S
1. All devices Dual Attached via vPC 2. Separate vPC and STP VLANs
SR PR
P S
P Primary vPC
S Secondary vPC
PR Primary STP Root
SR Secondary STP Root
3. Overlapping vPC and STP VLANs

STP Configuration Overview N Network port
E Edge or portfast port type
- Normal port type
B BPDUguard
Rootguard
Data Center Core R
L Loopguard
UDLD
Primary Secondary
vPC vPC
vPC
HSRP Domain HSRP Layer 3
ACTIVE STANDBY
Aggregation
N N Secondary
Primary
Root Root
- - - - - - - - Layer 2 (STP + Rootguard)
R R R R R R R R
- -
Access
- - - - L L
E E E E E
B B B B B
Layer 2 (STP + BPDUguard)
Attaching to a vPC Domain - 16-way Port-Channel (1 of 2)
Multi-Layer vPC can join 8 active

ports port-channels in a unique 16-
way port-channel*
vPC peer side load-balancing is Nexus 7000
LOCAL to the peer
Each vPC peer has only 8 active 16-port
port-channel
links, but the pair has 16 active
load balanced links Nexus 5000
Nexus 7000
* Possible with any device supporting

vPC/MCEC and 8-way active port-channels
Attaching to a vPC Domain - 16-way Port-Channel (2 of 2)
16 active ports between 8

active port-channel devices
and 16 active port-channel
devices?
Nexus 7000
vPC peer side load-balancing
is LOCAL to the peer
16-port
Each vPC peer has only 8 port-channel
active links, but the pair has 16 Nexus 5000
active load balanced links to
the downstream device
supporting 16 active ports
D-series N7000 line cards will
also support 16 way active
port-channel load balancing, Nexus 5000 16-port port-channel
providing for a potential 32 support introduced in 4.1(3)N1(1a)
way vPC port channel! release
Agenda
Access
Virtualization
Summary
Double-sided vPC between N7K and N5K
DESIGN 1
Make sure to leverage LACP vPC on the N7k
domain-id needs to differ N7k01 N7k02

between the N7k vPC and the
N5k vPC
Spanning-Tree root is defined Max 16 Ports
on one of the 2 N7ks 1 2 3 4
N5k priorities are unmodified

vPC on the N5k
N5k01 N5k02
1 2 3
Double-sided vPC Nexus 7000/5000/2000 A/A
DESIGN 3
DESIGN 2
N7k01 N7k02
N7k01 N7k02
1 2 3 4
1 2 3 4
N5k01 N5k02 N5k01 N5k02
5 6 7 8
N2k01 N2k02 N2k01 N2k02
1 2 3 1 3
domain-id same MAC (7k / 5k have same algorithm)
LAGID = [System-id for switch 1, Administrative-key-switch1 (channel-group-id), 0,
System-id for switch 2, Administrative-key-switch2 (channel-group-id), 0]
system-id of “switch 1”=

[system priority, MAC address of the “switch”]
N7k01 N7k02
derived from domain-id
or
must have same “MAC” of switch system-mac command
must have same “MAC” of switch
system-id of “switch 2” =
[system priority, MAC address of the “switch”] N5k01 N5k02
derived from domain-id

or
system-mac command
1 2 3
Loop Free access
Clear access VLANs to Clear access VLANs to
create a Loop Free Topology create a Loop Free Topology
Root Secondary Root

Primary Root Secondary Root
HSRP primary HSRP secondary
HSRP primary HSRP secondary
SW01 SW02
SW01 SW02
2/9 2/10 2/9 2/9 2/10 2/9 2/10

2/10
2/1 2/2 2/1 2/2 F F

Po51 Po10
logical equivalent
N5k01 N5k02
Peer Link
primary secondary
regular STP priority This link is almost unutilized
Loop Free access Clear access VLANs to
create a Loop Free Topology
Primary Root Secondary Root

Traffic flows are symmetric from HSRP primary HSRP secondary
access to aggregation
vPC is still useful to optimize
traffic flows from access to SW01 SW02
aggregation
2/9 2/10 2/9
All traffic flows through the active 2/10
HSRP switch, in this case SW01
Peer-link is almost unutilized 2/1 2/2 2/1 2/2
Po51 Po10
For more information and a
complete analysis please refer N5k01
Peer Link
N5k02
to:
http://wwwin-
eng.cisco.com/Eng/SAVBU/TechMkt
g/vpc-access-layer-wo-vpc-agg.ppt
Multi-Layer vPC Logic equivalent
root
Root
vPC on the N7k
N7k01 N7k02 logical equivalent
2/10 2/9 2/10

2/9
2/1 2/2 2/1 2/2

Po51 Po10
N5k01 N5k02
Peer Link
primary secondary
regular STP priority
Incorrect Configuration vPC at the Aggregation Layer
vPC at the Access Layer
root
Two Separate vPCs
vPC on the N7k One of the vPCs is blocking
Root
N7k01 N7k02
2/9 2/10 2/9 2/10 logical equivalent
2/1 2/2 2/1 2/2

Po51 Po10 Po51
N5k01 N5k02
Peer Link
primary secondary
regular STP priority
Spanning-Tree Secondary Root Primary Root
bridge id 80c2 bridge id ac32
vPC vPC
Operational Operational
secondary Peer-link primary
RP DP
vPC
N7k01 N7k02
DP DP
up to up to
8 links 8 links
RP RP
Peer-link
root bridge is
vPC
RP DP
N5k01 N5k02 ac32
vPC vPC
Presentation_ID
operational
© 2006 Cisco Systems, Inc. All rights reserved. secondary
Cisco Confidential
operational primary 48
Nexus 7000 End of Row Access-Layer design
Design Benefits
Large VLAN Scale: 4K VLAN (per VDC)
Core
L2/L3 boundary at access layer

Active-active host connectivity with vPC
VDC or VRF-lite for customer
Aggregation
segmentation
Flexibility of connectivity options –
1/10GE, STP Network Port
High-density access with Nexus 7010
(even higher with 7018).
Carrier Class availability
Hardware High availability: dual
Access
supervisor, internal redundancy)

Software High availability: Stateful
Switchovers, non disruptive ISSU and
process restart.
VM VM
VM VM
VM VM
Nexus 7000
Positioning
1/10GE Blade Servers
Environments with existing structured
End of Row cabling (EoR/MoR)
Nexus vPC Latest Enhancements
Several recent enhancements to vPC in 4.2
Some of these are specific to the L2/L3 boundary, for more
details:
4.2 Release Notes

http://www.cisco.com/en/US/docs/switches/datacenter/sw/4_2/nx-os/release/notes/42_nx-
os_release_note.html#wp218085
Summary Checklist for vPC Design (1)
Choose between MST and Rapid PVST+
With MST be aware of the NXOS VLAN
range and of the Global Type-1
Inconsistencies, hence configure VLAN-to- N7k01 N7k02
region mappings from day 1
Connect the N7ks with redundant peer-links
across linecards
Connect the N5ks with redundant peer-links 1 2 3 4
Make sure that peer-keepalive connectivity

is routed and “out-of-band”
Assign roots/secondary roots as usual
(regardless of primary/secondary roles) 5 6 7 8
Leverage 4.2(1) code on the N7k in order to

exclude non-vPC SVIs from autostate N2k01 N2k02
1 2 3
Summary Checklist for vPC Design (2)
Create a single Port-channel leveraging
LACP between Aggregation and Access
Ensure domain-id or system-mac
differs between Agg pair and Access N7k01 N7k02
pair
On the Nexus 5000 layer calculate and
check VLAN utilization keeping FCoE
and VSANs into account
1 2 3 4
Trim VLANs that are used for VSANs
from the uplinks going to a Nexus 7000
When available leverage LACP for N5k01 N5k02
teaming between servers and FEX/5k 5 6 7 8
Do not forget that putting a VLAN on a
vPC requires that that VLAN be on the N2k01 N2k02
Peer-link too
Make sure the configuration is not
causing Type-1 Inconsistencies
1 2 3
Nexus 5000/7000 Scalability Numbers
Release Supported Scalability
Nexus 7000 System wide 16,384 VLANs (4096 per VDC)

4.2(1) Fabric Extender support to come in Q2CY10
256 vPC’s (4-port) with the following
260 VLANs*
200 SVI/HSRP Groups
40k MACs & 40K ARPs
10K (S,G) w. 66 OIFs (L3 sources)
3K (S,G) w. 64 OIFs (L2 sources)
Nexus 5000 System wide 507 VLANs (512 minus number of VSANs)
4.1(3)N1(1) 12 Fabric Extenders
16 Hardware Ethernet port channels
Up to 480 “Host vPCs” (via FEX)
* NOTE: Supported numbers of VLANs on vPCs are NOT related to an hardware or software limit but reflect
what has been currently validated by our QA. The BUs are planning to continuously increase these numbers
as soon as new data-points become available.
Agenda
Access
Virtualization
Summary
Nexus Fabric Extender
Fabric Extended Terminology
Parent Switch: Acts as the Nexus 5000
combined Supervisor and
Switching Fabric for the virtual
switch
Fabric Links: Extends the
Switching Fabric to the remote
line card (Connects Nexus 5000 FEX100 FEX101
to Fabric Extender)
Host Interfaces (HIF)
Fabric connectivity between
Nexus 5000 and Nexus 2000
(FEX) can leverage either pinning
or port-channels
dc11-5020-1# show interface fex-fabric

Fabric Fabric Fex FEX
Fex Port Port State Uplink Model Serial
---------------------------------------------------------------
100 Eth1/17 Active 1 N2K-C2148T-1GE JAF1311AFLL
101 Eth1/21 Active 1 N2K-C2148T-1GE JAF1311AFMT
101 Eth1/22 Active 2 N2K-C2148T-1GE JAF1311AFMT
Fabric Extender & vPC
Peer Keepalive
Peer Link/ MCT

Terminology and Components vPC Member Port
Nexus 2000 Single-homed vPC Nexus 2000 active/active

(or dual homed)
mgmt network mgmt network
FT link (can be routed)
mgmt0 mgmt0
mgmt0 mgmt0
Peer-link
primary secondary
Peer-link
primary secondary
5k01 5k02 5k01 5k02
“fabric links” “fabric links”
vPC 1 vPC 2
FEX100 FEX120
HIF 2 ports HIF
FEX100 FEX120
HIF HIF
vPC
LACP is supported
Server 802.3ad not
2-GigE ports host port channel
supported in this topology
Fabric Extender
BPDU “filtering” + guard (not an oxymoron)
Peer Keepalive
Peer Link/ MCT
mgmt network vPC Member Port
mgmt0 mgmt0 BPDU filtering + Guard

Peer-link
primary secondary
5k01 5k02
With BPDU filtering, FEX still sends out
10 BPDUs after link up which prevents
the introduction of an unwanted loop.
FEX100 FEX101 BPDU filtering is used in conjunction with

HIF HIF BPDU guard.
Errdisable
Fabric Extender
Port Type Edge (Portfast or Trunkfast)
Peer Keepalive
Peer Link/ MCT

vPC Member Port
mgmt network
Spanning-Tree Port Type Edge

mgmt0 mgmt0 or
Spanning-Tree Port Type Edge Trunk
Peer-link
primary secondary
5k01 5k02
FEX100 FEX101
HIF HIF
Fabric Extender vPC HIF Ports
Do not Forget putting HIF VLANs on the peer-link
Peer Keepalive
Peer Link/ MCT

vPC Member Port
mgmt network
Peer-link must carry all the VLANs that are

mgmt0 mgmt0 used on the HIFs
Peer-link
primary secondary
The peer-link is always forwarding
5k01 5k02
HIF configuration must match (see

FEX100 FEX101
HIF HIF
consistency check)
Fabric Extender Mixed Topology
vPC is a per line card (FEX) behaviour
Management Network
mgmt0 mgmt0
5k01 5k02
primary secondary
FEX100 FEX120
FEX101 FEX121
2-GigE ports host port channel single attached servers and/or A/S
Fabric Extender Scaling
Nexus 2000 Single Homed (aka Straight Through)
Typical Redundant Deployment as of 4.0(1a)
Nexus 2000 Straight-through deployment
n5k01 n5k01 n5k02

max 4 “fabric links”
FEX100 FEX102 FEX120 FEX122

FEX101 FEX121
max 12 x 2 = 576 ports x 2
FEX100 FEX102
FEX101
max 12 = 576 ports

Active/Standby
http://www.cisco.com/en/US/partner/products/ps9670/products_installation_and_configuration_guides_list.html
Port-Channels & vPC
Consumes 1 HW Port- does NOT consume HW does NOT consume

channel of the 16 resources HW resources
available
2-Ports vPCs
As many as the number of ports on the 5k
4+ Ports vPCs mgmt0 mgmt0
Peer-link
primary secondary
5k01 5k02
5k01 5k02 5k01 5k02
“fabric links”
Max 16 HW-Port Channel
FEX100 FEX120
HIF HIF
eth2/1,2/2 eth2/3,2/4 eth2/1 eth2/2 2 ports
vPC
vPC
Scalability for “Host” vPC Nexus 2000 straight-through
n5k01 n5k02
max 24 FEXes = 1152
max 480 vPCs (each vPC has 2 ports)
Nexus 2000 dual-homed
5k01 5k02
vPC Secondary
vPC Primary
Po10
max 12 FEXes
For more detailed information on N2K design please see: http://bock-

bock.cisco.com/wiki/N5K:tecRes:desGui
Agenda
Access
Virtualization
Summary
Cisco Nexus 1000V
Components
Cisco VSMs
vCentre Server
Virtual Ethernet Module(VEM) Virtual Supervisor Module(VSM)

Replaces Vmware’s virtual switch CLI interface into the Nexus 1000V
Enables advanced switching Leverages NX-OS 4.04a
capability on the hypervisor
Controls multiple VEMs as a single
Provides each VM with dedicated network device
“switch ports”
VM VM VM VM VM VM VM VM VM
#1 #2 #3 #4 #5 #6 #7 #8 #9
Connecting the Virtual Access Layers
Connecting Nexus 1000V to 2148T without vPC
In a Four NIC implementation

Access switch configured with
Trunk ports (no Etherchannel)
VEM Configured with SRC based
hashing
Trunk Edge Port
N1KV Port Channel 1 (vPC-HM) supporting only
the VM VLANs
VM Data
N1KV Port Channel 2 (vPC-HM)

Service Console, VM Kernel, VEM SG0 SG1 SG0 SG1
Control and Packet
VM traffic SC and VMK

carried on a traffic carried on
second vPC- one upstream
HM uplink VM VMK SC vPC-HM uplink
bundle bundle
Connecting Nexus 1000V to 2148T
4.0(4)SV1(1) release of N1KV supports

the MAC pinning with up to 8 links in the
uplink bundle
Allows for up to 8 Independent ports
spread across multiple switches in the
physical access layer
If a failover occurs, all the traffic pinned to Edge Ports
an interface will be migrated to the other
interfaces.
Independent
links on the
upstream
switches
Nexus1000(config)#port-profile sys-uplink
Nexus1000(config-port-prof)#no shut
Nexus1000(config-port-prof)#capability uplink
Nexus1000(config-port-prof)#channel-group auto mode mac-pinning Traffic load
Nexus1000(config-port-prof)#switchport mode trunk shared across
Nexus1000(config-port-prof)#switchport trunk allowed vlan 10-25 up to 8 links
VM VMK SC using a MAC
Nexus1000(config-port-prof)#state enabled
Nexus1000(config-port-prof)#vmware port-group hash
Connecting Nexus 1000V to 2148T
vPC between a pair of N5K/N2K
allows symmetrical Etherchannels
One of the benefits of using port-
channels for connectivity is the
reduction in the amount of flooding /
broadcast that the software switch
has to drop
vPC MCEC
17 hashing algorithms available Bundles
Selected either system wide or per

module
Default is source MAC
N1KV Port Channel 1 VM traffic

carried on a SC and VMK
VM Data second uplink traffic carried on
bundle one upstream
N1KV Port Channel 2 VM VMK SC uplink bundle
Service Console, VM Kernel, VEM Control
and Packet
Connecting Nexus 1000V
Nexus 1000V offers multiple modes to

connect upstream to the first physical
upstream switch
Host Mode vPC provides a mechanism
to support upstream devices that are
not port channel capable VM VMK SC
When combined both MCEC vPC

(virtual access switch) and vPC-HM
the sub-groups must be manually
configured on the N1KV side
Going beyond 4 NICs you should be
migrating to 10GE connectivity to
simplify the overall solutions design
For more detailed information on installing and

configuring N1KV please see:
VM VMK SC
http://bock-bock.cisco.com/wiki/N1KV:Technical
Agenda
Access
Virtualization
Summary
Unified Fabric
Where we started - Segregated Fabrics
Unified Fabric with FCoE
Phase 1 – Where we have been
Servers with Converged Network Adapters

(CNAs) provide consolidated LAN/SAN
Direct-attach connectivity to the Nexus 5000.
The Nexus 5000 can operate in FCF mode or

NPV mode. It can then split off Ethernet traffic
and Native Fibre Channel traffic
Nexus 5000s as FCF or

In NPV mode
FCoE Direct Attach CNAs
Phase 2 – Where we are today (FIP support)
Servers with Converged Network Adapters

(CNAs) provide consolidated LAN/SAN
connectivity to the Nexus 5000.
FIP on the Nexus 5000s now provides discovery

of direct and indirect attached FCoE initiators
Supported as of NX-OS 4.1(3)N1 on Nexus 5000
VPC Peer link carrying Ethernet Traffic

only maintaining SAN A and SAN B isolation
Nexus 5000s as FCF or

In NPV mode
FIP on Nexus 5000 allows for direct attach

And indirect attach CNAs
Phase 2 – Adding the Nexus 4000 (Q4CY09)
Mezzanine CNAs can be installed within Blade
Enclosures, connecting up to Nexus 4000 Blade
Switches.
Nexus 4000 will provide FIP Snooping

Capabilities, providing lossless servicing and
Segmentation.
FIP on the Nexus 5000s now provides discovery

of indirect attached FCoE initiators
FIP on Nexus 5000 allows for direct attach

And indirect attach CNAs
Nexus 4000 Blade Switches providing FIP

Snooping capabilities
Phase 2 – Adding the Nexus 4000 (Q4CY09)
Nexus 4001I Switch Module
14 x 10G downlinks & 6 x10G uplink
ports
Dual-mode downlink ports (1G / 10G
to server)
RJ-45 Management interface
RS-232 Console port
Goes in high-speed slots in IBM
BCH or BCH-T
10G uplink ports Console
Max of Four 4001I per chassis
Support CX1 SFP+, SR, LSR optics Management Ejector Handles
Integration with Consolidate Scale Bandwidth

Server Infrastructure w/ with 10G & Multi-
Multi-
Virtualization Unified IO pathing
FIP – FCoE Initialization Protocol
FIP is “FCoE Initialization ENode FCoE Switch
Protocol”
Enables FCoE adapter to VLAN VLAN
discover FCoE FCFs on a

Discovery Discovery
VLAN and establish a

VNVF link with one of the
FCFs
FIP:
FCF FCF FCoE
Discovery Discovery Initialization
Separate Ethertype makes Protocol
FIP messages easier to

intercept for FIP snooping
by traditional Ethernet
bridges as opposed to FLOGI/FDI
SC
FLOGI/FDISC
Accept
standard FCoE login
frames.
Building foundation for
future multi-hop FCoE
FCOE
FC FC Command Protocol
topologies
Command responses
FIP Snooping
FIP Capable
FIP Snooping – Nexus 4000
Topology SAN
Security (Protection from MAC
Address spoofing of FCoE end
devices “ENode”)
Spoofed MAC
Fibre Channel links are Point-to-Point 0E.FC.00.DD.EE.FF
Ethernet bridges can utilize ACLs to
provide the equivalent path control FCF
(equivalent of point-to-point) FCF MAC
FIP protocol allows for efficient 0E.FC.00.DD.EE.FF
automatic configuration of the ACLs
necessary to lock down the forwarding FIP
path (FIP Snooping) Snooping
Ethernet-NPV (E-NPV) - Future

On the control plane (FIP ethertype), ENode MAC
an "Ethernet NPV bridge" improves 0E.FC.00.07.08.09
over a "FIP snooping bridge" by ENode
intelligently proxying FIP functions
between a CNA and an FCF
Data Center HA Design and FCoE
Ethernet, a historical perspective
Ethernet ? ? ?
The L2 network is a ?
?
communication pipe
Amorphous pipe, ? Switch Switch ?
amorphous end device
relationships
East-west vs. north-south Switch
?
traffic ratios are undefined ?
Maximum flexibility ?
?
? ?
Network designs fill the
void
Give shape to device roles,
client/server relationships, ? ?
availability semantics Client/server
relationships
?
Data Center HA Design and FCoE
Fibre Channel, a historical perspective
Fibre Channel T0 T1 T2
The L2 network embeds most
services and provides end
device connectivity DNS FSPF
Switch
FSPF
Switch
Zone
Well defined end device Zone

RSCN DNS RSCN
relationships (initiators and DNS
targets)
FSPF
Switch
Zone
RSCN I5
Only north-south traffic, east- I0
west traffic mostly irrelevant I1
I4
Tailored to fit one function I2 I3
(limited flexibility)
Network designs build scale
and enhance availability I(c)
Everything else is Client/server
T(s)
predetermined relationships
I(c)
Data Center Access Architecture
FCoE High Availability Design Considerations
In a Unified I/O configuration (FCoE) Direct Attach Topology
we have two distinct topologies
LAN SAN A SAN B
Isolated access switches - SAN
‘A’ and SAN ‘B’
Combined access switches –
vPC supporting MCEC
FCF FCF
To ensure correct forwarding
behavior ‘vfc’ interface can only be Nexus 5000
associated with a vPC etherchannel
(only one physical interface per
switch)
Works with Gen-2 FIP enabled CNAs
ONLY
While the Port-channel is the same
on 5k1 and 5k2, the FCoE VLANs type edge trunk
are different.
FCoE VLANs are NOT carried on the
peer-link
Direct Attach Topology
A VLAN is dedicated for every VSAN
in the fabric. SAN A SAN B
LAN
The VLAN is signaled to the hosts
over FIP VLAN 10 ONLY
HERE!
The FCoE controller in the host tags
all subsequent FIP login and FCoE FCF FCF
frames with the signaled FCoE VLAN
VLAN 10,20
Nexus 5000
This does not require trunking to be
enabled at the host driver as tagging
is performed by the FCoE controller VLAN 10,30
on the host.
All ports in the FCoE network would
have to be enabled for trunking to be
able to carry VLAN tagged frames. !VLAN 20 is dedicated to carry
traffic for VSAN 2
(config)# vlan 20
(config-vlan)# fcoe vsan 2
Direct Attach Topology
Single initiator dual homed via a
Port-Channel to a single Nexus LAN SAN A SAN B
5000
No ability to isolate SAN ‘A’ and
SAN ‘B’
This is an unsupported
configuration FCF FCF
Not consistent with Fibre
Nexus 5000 Nexus 5000
Channel High Availability
design requirements
4G FC
Neither Direct Attach nor Multi-Hop
10G Ethernet
Capable CNAs currently supported
10G Unified I/O
FIP Capable Topology
Initial vPC is supported as it is SAN A SAN B
LAN
possible to isolate the SAN ‘A’
and SAN ‘B’ traffic between the
CNAs and first hop switches
No ability to isolate SAN ‘A’ and
SAN ‘B’ between the first and FcoE frames
FCF
second tier of switches load balanced
over the
This is an unsupported Etherchannel
NO SAN ‘A’
configuration and SAN ‘B’
Not consistent with Fibre FIP
Snooping
isolation
Channel High Availability

design requirements
4G FC
10G Ethernet Multi-Hop Capable
10G Unified I/O
FCoE and Nexus 4000 Design Considerations
Nexus 4000 enabled for Multi- FIP Capable Topology

Hop FCoE leverages
Uplinks carrying FCoE VLAN LAN SAN A SAN B
must only be connected to a

single upstream N5K (FCF)
switch
On Nexus 5000, enable “feature N5K-1 N5K-2
fcoe” FCF FCF
On Nexus 4000, enable “feature

fip-snooping”
N4K-1 N4K-2
From an FCoE perspective this FIP FIP
design required to preserve the Snooping Snooping
dual Fabric environment
4G FC Blade Chassis
10G Unified I/O
Nexus 4000 – Ethernet Only
Ethernet Only Capable Topology

In Ethernet ONLY topologies
standard Ethernet HA design N7K-2
rules can be utilized N7K-1
Dual NIC servers with standard

NIC teaming supported between
N4K N5K-2
MCEC enabled uplinks from the N5K-1
N4K to the first aggregation tier

is supported since no
requirement for LAN ‘A’ and LAN N4K-1 N4K-2
‘B’ topologies
No server side vPC is currently
supported
10G Ethernet
Multi-Hop Unified Fabric Configuration
FCoE Multi-Hop Configuration
Nexus 5000 FIP Capable Topology
• Switching mode for FC traffic
• VSAN 1 Mapped to VLAN 30 for FCoE LAN SAN A SAN B
• VFC interface will bind to MAC Address of
Blade Server
• Port-Channel downlink to N4K will be “mode
trunk” (all vlans will be allowed)
N5K-1 N5K-2
Nexus 4000
FCF FCF
• VLAN 30 & 31 in FIP-Snooping mode

• Port-Channel uplink to N5K will be “mode
trunk” N4K-1 N4K-2
• Qlogic CNA in Windows Blade Server FIP FIP
• CNA configured as .1q trunk (for IP and FCoE Snooping Snooping
traffic)
4G FC Blade Chassis
10G Unified I/O
N5K1 – Enabled FCoE and vPC
n5k-1(config)# feature lacp
n5k-1(config)# interface port-channel 1
n5k-1(config-if)#switchport mode trunk
n5k-1(config-if)#switchport trunk allow vlan except 30-31
n5k-1(config-if)# interface ethernet 1/17 - 18
n5k-1(config-if-range)#switchport mode trunk N5K-1 N5K-2
n5k-1(config-if-range)#switchport trunk allow vlan except 30-31
n5k-1(config-if-range)# channel group 1 mode active
n5k-1(config-if-range)#switchport mode trunk
n5k-1(config)# interface ethernet 1/9 - 10
n5k-1(config-if-range)# show port-channel summary

Flags: D - Down P - Up in port-channel (members)
I - Individual H - Hot-standby (LACP only)
s - Suspended r - Module-removed N4K-1 N4K-2
S - Switched R - Routed
U - Up (port-channel)
--------------------------------------------------------------------------
Group Port- Type Protocol Member Ports
Channel
--------------------------------------------------------------------------
1 Po1(SU) Eth LACP Eth1/17(I) Eth1/18(I)
300 Po300(SU) Eth LACP Eth1/9(I) Eth1/10(I)
Blade Center
n5k-1(config)# feature fcoe
n5k-1(config)# vlan 30
n5k-1(config-vlan)#fcoe vsan 1
n5k-1(config-vlan)# show vlan fcoe

VLAN VSAN Status
-------- -------- --------
30 1 Operational
N4K1 – Enabled Uplinks and FIP Snooping
n4k-1# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
n4k-1(config)# feature lacp
n4k-1(config)# feature fip-snooping
n4k-1(config-if)#switchport mode trunk N5K-1 N5K-2
n4k-1(config-if)#fip-snooping port-mode fcf
n4k-1(config-if)# interface ethernet 1/15 - 16
n4k-1(config-if-range)#fip-snooping port-mode fcf
n4k-1(config)# vlan 30
n4k-1(config-vlan)# fip-snooping
n4k-1(config-vlan)# show running-config vlan 30
version 4.1(2)E1(1)
vlan 30
fip-snooping enable N4K-1 N4K-2
n4k-1(config-vlan)# interface ethernet 1/4
n4k-1(config-if)#switchport mode trunk
n4k-1(config-if)#spanning-tree port type edge trunk
n4k-1(config-if)# show fip-snooping vlan-discovery
Legend:
-------------------------------------------------------------------------------
Interface VLAN FIP MAC
Blade Center
-------------------------------------------------------------------------------
Eth1/4 1 00:c0:dd:04:0d:11
This MAC-Address will be used to bind

the “vfc” from the Nexus 5000 switch
Binding of “vfc” on N5K1 to CNA on N4K1
This vfc number is arbitrary

n5k-1(config)# interface vfc 104
n5k-1(config-if)# bind mac-address 00:c0:dd:04:0d:11
n5k-1(config-if)# no shut
N5K-1 N5K-2
n5k-1(config-if)# show interface vfc104
vfc104 is up
Bound MAC is 00:c0:dd:04:0d:11 This MAC-Address was
FCF priority is 128 shown from the Nexus 4000
Hardware is Virtual Fibre Channel
Port WWN is 20:67:00:0d:ec:b1:1f:ff command
Admin port mode is F, trunk mode is on
snmp link state traps are enabled
Port mode is F, FCID is 0x160001
Port vsan is 1
1 minute input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec N4K-1 N4K-2
1 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
0 frames input, 0 bytes
0 discards, 0 errors
0 frames output, 0 bytes
0 discards, 0 errors
Interface last changed at Mon Sep 7 05:06:24 2009
Blade Center
For more detailed information on the Nexus 4000 please

see: http://bock-bock.cisco.com/wiki/N4K
Agenda
Access
Extending vPC from access to
aggregation
Virtualization
Nexus 1000v – Embedded Virtual
Switching
Summary
Summary
Assembling the New Data Centre Edge
Core/Aggregation Layer
Core
MDS
Nexus 7000
Nexus 5000/7000
Access Layer
10 GigE/FCOE
FCOE
Nexus 2000
Virtual Access Layer

blade1
slot 1 blade1
slot 1 blade1 blade1
slot 1 slot 1
NEXUS
NEXUS NEXUS
blade3 blade3 blade3
VM VM
VM VM VM VM
slot 3 slot 3
blade4
slot 4 blade4
slot 4 blade4
slot 4 blade4
slot 4
NEXUS
VM VM
blade6
slot 6 blade6
slot 6 blade6
slot 6 blade6
slot 6
1000v
blade7
1000v 1000v
blade7
slot 7 blade7
slot 7
VM VM
blade8
VM VM VM VM
blade8
slot 8 blade8
slot 8
1000v
VM VM
VM VM
blade1
slot 1
VM VM
blade2
slot 2
blade3
slot 3
blade1
slot 1
blade2
slot 2
blade3
slot 3
blade1
slot 1
VM VM
blade2
slot 2
blade3
slot 3
blade1
slot 1
blade2
slot 2
blade3
slot 3
VM VM
blade4
slot 4 blade4
slot 4 blade4
slot 4 blade4
slot 4
blade6
slot 6 blade6
slot 6 blade6
slot 6 blade6
slot 6
blade7
slot 7 blade7
slot 7 blade7
slot 7 blade7
slot 7
blade8
slot 8 blade8
slot 8 blade8
slot 8 blade8
slot 8

blade2
slot 2 blade2
slot 2 blade2
slot 2 blade2
slot 2
blade4
slot 4 blade4
slot 4 blade4
slot 4 blade4
slot 4
blade5
slot 5 blade5
slot 5 blade5
slot 5 blade5
slot 5
blade6
slot 6 blade6
slot 6 blade6
slot 6 blade6
slot 6
blade7
slot 7 blade7
slot 7 blade7
slot 7 blade7
slot 7
blade8
slot 8 blade8
slot 8 blade8
slot 8 blade8
slot 8
1G and 10GE Rack 1G and 10GE Blade N4K - DCB Blade 10GE Blade
Mount Servers Servers (HP) UCS Compute UCS Compute
Switch Pod Pod
Pass-Thru IBM/Dell
HP/IBM/Dell
Additional Resources
Public References
Enterprise Solutions Engineering:

http://www.cisco.com/en/US/docs/solutions/Enterprise/Data_Center/DC_3_0/DC-3_0_IPInfra.html
White paper on vPC:

http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9402/white_paper_c11-
516396.html
Key Takeaways
Nexus-based DC LAN designs are continuously evolving to

enable 10 Gigabit to the server adoption, better bandwidth
utilization in the DC and evolution towards Unified IO
This allows cost effective 10 Gigabit to the server for
VMWARE or single OS deployment, and for Unified IO
End-to-end vPC-based designs are now possible and yield
much better utilization of network infrastructure
Continuous improvements on vPC/FEX convergence and
usability

DC Nexus Design

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

DC Nexus Design

Uploaded by

Copyright:

Available Formats

Data Center Access Design

with Nexus Switches

Elements of the Evolving Data Centre

At the end of the session, the participants should be able to:

Elements of the Evolving Data Centre

The Data Centre Access Layer is Core

Virtual Access Layer

blade1 blade1 blade1 blade1

Virtual Port-Channel Nexus 2000/FEX Nexus 1000V + VN-Link

Nexus 7000 (Release 4.2):

Nexus 5000 (Release 4.1(3)):

Use newest STP enhancements like Bridge Assurance

NX-OS STP modes

NX-OS uses a fixed STP link cost for

BPDUs in Rapid-PVST carry information

Only one “designated” port can exist per

Dispute Mechanism performs

Requires no configuration, enabled

%STP-2-DISPUTE_DETECTED: Dispute detected on port Ethernet1/2 on VLAN0700.

Nexus# sh spanning vlan 700 | in BLK

Protects against unidirectional BA Inconsistent

links and peer switch software Network

optimal traffic flow N R N R

Forces Layer 2 LAN interface to

Alternate port Network port

Designated port R Root Guard

In Catalyst IOS the range of reserved VLANs is:

Allow a single device to use a

Dual-homed server operate in

Elements of the Evolving Data Centre

Multi-Chassis Port Channel Yes Yes

Two Independent Nodes,

10GE interfaces Sup720-10G, 6708, 6716

vPC peer- vPC peer-link

Heartbeat between vPC peers

Management Standby Management

CFSoE MAC table

5k01 5k02 Packet(s) blocked on vPC

Indicates that the behavior

interface port-channel51 interface port-channel51

5020# show vpc brief

5020# show vpc brief

2009 May 17 21:56:28 dc11-5020-1 %ETHPORT-5-IF_ERROR_VLANS_SUSPENDED: VLANs 105 on Interface port-

Solution: define MST region mappings from the very beginning

mst region mgmt0 mgmt0

Indicates that the vPC vPC vPC

1. Dual Attached 2. Attached via VDC/Secondary Switch

3. Secondary ISL Port-Channel 4. Single Attached to vPC Device

PR Primary STP Root

SR Secondary STP Root

3. Overlapping vPC and STP VLANs

Multi-Layer vPC can join 8 active

* Possible with any device supporting

16 active ports between 8

Make sure to leverage LACP vPC on the N7k

domain-id needs to differ N7k01 N7k02

N5k priorities are unmodified

N5k01 N5k02 N5k01 N5k02

N2k01 N2k02 N2k01 N2k02

system-id of “switch 1”=

must have same “MAC” of switch

derived from domain-id

Root Secondary Root