Best Practices come from

YOU
BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 2
 Apple iphone4 launched in June 2010

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
‘Antennagate’

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
IPHONE4 Best Practices from CUSTOMERS

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
vPC Best Practices and Design on
NXOS
BRKDCT-2378

Nazim Khan, CCIE#39502 (DC/SP)

Technical Marketing Engineer, Data Center Group
Session Goals
• Best Practices and Designs for vPC – virtual
port-channel
• Nexus 2000 (FEX) will only be addressed from
vPC standpoint
• Fabricpath Overview
• VPC+ Overview
• vPC with VXLAN based networks

vPC : Get it Right the very First time

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
Session Non-Goals
• vPC troubleshooting
• Details of vPC+
• Details of Fabricpath and VXLAN
• ACI with or without vPC
• FCoE

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
Agenda
• Feature Overview
• Configuration Best Practices
• Design Best Practices
• Fabricpath / vPC+
• VxLAN
• Scalability
• Reference Material

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
 MPLS, OTV,

Data Center Technology Evolution MPLS, OTV,

LISP

LISP

ACI

VXLAN

FabricPath with vPC+

FEX with vPC

VPC
2014-2015
STP
2013-2014

2010

2010
2009
2008

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
 Why vPC in 2015 ?

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
vPC is Foundation

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
Role of vPC in the Evolution of Data Center
• vPC launched in 2009

• Deployed by almost 95% of Cisco customers

Unified Fabric
• Used to redundantly connect network entities at the
edge of the Fabric
– Dual-homed servers (bare metal, blades, etc.)
– Network services (Firewalls, Load Balancers, etc.)

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
Agenda
• Feature Overview
− Concepts and Benefits
− Terminology
• Configuration Best Practices
• Design Best Practices
• Fabricpath / vPC+
• VxLAN
• Scalability
• Reference Material

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
vPC Feature Overview
vPC Concept & Benefits

S1 S2

S3
STP vPC Physical Topology vPC Logical Topology

• No Blocked Ports, More Usable Bandwidth, Load Sharing

• Fast Convergence

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
Feature Overview
vPC Terminology

Layer 3 Cloud
vPC vPC Domain vPC Peer
Peer Keepalive Link
Peer-Link

Orphan
CFS S2
Port S1

vPC Member
vPC Port

Orphan
Device S3

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
vPC Failure Scenario
vPC Peer-Keepalive Link up & vPC Peer-Link down
vPC peer-link failure (link loss):
vPC Peer-keepalive
• VPC system checks active status of the remote vPC peer P S

via peer-keepalive link (heartbeat)

• If both peers are active, then Secondary vPC
peer will disable all vPCs to avoid Dual-Active vPC_PLink

• Data will automatically forward down remaining Suspend secondary

vPC Member Ports
active port channel ports
• Failover gated on CFS message failure, vPC1 vPC2

or UDLD/Link state detection

• Orphan devices connected to secondary peer will be SW3 SW4

isolated
Keepalive Heartbeat

P Primary vPC

S Secondary vPC

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public
Agenda
•
• vPC Configuration Best Practices
− Building a vPC domain
− Domain-ID
− Peer-Link
− Peer-Keepalive Link
− Spanning-Tree
− Peer-switch
− Auto-recovery
− Object tracking
− vPC shutdown
− Maintenance Mode
•
•
•
•
•

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
vPC Configuration Best Practices
Building a vPC domain – Configuration Steps

1. Define domains S1 S2
2. Establish Peer Keepalive connectivity

3. Create a Peer link CFS

4. Create vPCs

5. Make Sure Configurations are Consistent

(Order does Matter!)

S3

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
 vPC Configuration Best Practices
vPC Domain-ID vPC Domain 10

• The vPC peer devices use the vPC domain ID to S1 S2

automatically assign a unique vPC system MAC
address
• You MUST use unique Domain id’s for all vPC
pairs defined in a contiguous layer 2 domain vPC Domain 20

S3 S4
! Configure the vPC Domain ID – It should be unique within the layer 2
domain
NX-1(config)# vpc domain 20

! Check the vPC system MAC address

NX-1# show vpc role
<snip>
vPC system-mac : 00:23:04:ee:be:14
S5

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
vPC Configuration Best Practices
vPC Peer-Link

S1 S2 S1 S2

S3
S3

• vPC Peer-link should be a point-to-point connection

• Peer-Link member ports can be 10/40/100GE interfaces
• Peer-Link bandwidth should be designed as per the vPC
• vPC imposes the rule that peer-link should never be blocking
BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
vPC Configuration Best Practices
vPC Peer-Keepalive link

Preference Nexus 7X00 / 9X00 Nexus 6000 /

series 5X00 / 3X00
series
Recommendations 1 Dedicated link(s) mgmt0 interface
(in order of (1GE/10GE LC)
preference):
2 mgmt0 interface Dedicated link(s)
(1GE/10GE LC)

3 L3 infrastructure L3 infrastructure

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
vPC Configuration Best Practices For Your
Reference
vPC Peer-Keepalive link – Dual Supervisors
Management Switch Management
Network

• When using dual supervisors and mgmt0 interfaces vPC_PKL

vPC_PKL

to carry the vPC peer-keepalive, DO NOT connect

them back to back between the two switches
vPC_PL

• Only one management port will be active a given point

vPC1 vPC2
in time and a supervisor switchover may break keep-
alive connectivity

• Use the management interface when you have an out-

of-band management network (management switch in Standby Management Interface
between)
Active Management Interface

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
 vPC Configuration Best Practices
Spanning Tree (STP)

STP is running to manage S1 S2

loops outside of vPC domain,
or before initial vPC
configuration !

S4
S3

S5

• All switches in Layer 2 domain should run either Rapid-PVST+ or MST

• Do not disable spanning-tree protocol for any VLAN
• Always define the vPC domain as STP root for all VLAN in that domain
BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
vPC Configuration Best Practices
vPC Peer-Gateway

• Allows a vPC switch to act as the active S1 S2

gateway for packets addressed to the peer
router MAC
• Keeps forwarding of traffic local to the vPC
node and avoids use of the peer-link
• Allows Interoperability with features of some S3 S4
NAS or load-balancer devices

N7k(config-vpc-domain)# peer-gateway

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
vPC Configuration Best Practices
vPC Peer-switch Primary Secondary
vPC vPC

Without Peer-switch
BPDUs
• STP for vPCs controlled by vPC primary.
• vPC primary send BPDU’s on STP designated ports
• vPC secondary device proxies BPDU’s to primary

Primary Secondary
With Peer-switch vPC vPC

• Peer-Switch makes the vPC peer devices to appear

as a single STP root
• BPDUs processed by the logical STP root formed by
the 2 vPC peer devices
N7k(config-vpc-domain)# peer-switch
BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
 vPC Configuration Best Practices
vPC auto-recovery Operational
Primary
P S P S
P

S1 S2 S1 S2 S1 S2

S3 S3 S3

1. vPC peer-link down : S2 - secondary shuts all its vPC member ports
2. S1 down : vPC peer-keepalive link down : S2 receives no keepalives
P vPC Primary
3. After 3 keepalive timeouts, S2 changes role and brings up its vPC
S vPC Secondary

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
vPC Configuration Best Practices For Your
Reference
vPC auto-recovery
Auto-recovery addresses two cases of single switch behavior
• Peer-link fails and after a while primary switch (or keepalive link) fails
• Both VPC peers are reloaded and only one comes back up

How it works
• If Peer-link is down on secondary switch, 3 consecutive missing peer-keepalives will
trigger auto-recovery
• After reload (role is ‘none established’) auto-recovery timer (240 sec) expires while
peer-link and peer-keepalive still down, autorecovery kicks in
• Switch assumes primary role
• VPCs are brought up bypassing consistency checks

Nexus(config)# vpc domain 1

Nexus(config-vpc-domain)# auto-recovery

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
vPC Configuration Best Practices
Why Object-Tracking ?
S4 S5
• Modules hosting peer-link and uplink
fail on the vPC primary
Primary Secondary
• Peer-Link is down and vPC
Secondary shut all its vPC
S1 S2
• Auto-Recovery does not kick in as
peer-keepalive link is active

• Traffic is black holed

S3

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
 vPC Configuration Best Practices
Object-tracking
• vPC object tracking, tracks both peer-link and S4 S5
uplinks in a list of Boolean OR
• Object Tracking triggered when the track object
goes down
• Suspends the vPCs on the impaired device.
• Traffic forwarded over the remaining vPC peer.
! Track the vpc peer link
track 1 interface port-channel11 line-protocol
! Track
track 2
the uplinks
interface Ethernet1/1 line-protocol
S1 S2
track 3 interface Ethernet1/2 line-protocol

! Combine all tracked objects into one.

! “OR” means if ALL objects are down, this object will go down
track 10 list boolean OR
object 1
object 2
object 3
S3
! If object 10 goes down on the primary vPC peer,
! system will switch over to other vPC peer and disable all local vPCs
vpc domain 1
track 10
BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
 vPC Configuration Best Practices
vPC Shutdown
• Isolates a switch from the vPC complex Primary Secondary

• Isolated switch can be debugged, reloaded, or vPC

even removed physically, without affecting the
vPC traffic going through the non-isolated switch
S1 S2

switch# configure terminal

switch(config)# vpc domain 100
switch(config-vpc)# shutdown

S3

This Feature is currently supported only on Nexus 5X00 and 600X series

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
vPC Configuration Best Practices
Maintenance Mode

Primary 3 Secondary
1. vPC Primary enters maintenance mode via CLI
vPC
1 2. Running configuration is saved, key show
5
command output is collected and saved
2
4 3. Change priority to highest value (65635)
4. Admin down all vPCs and vPC peer-link
5. Advertise state as “self-isolated” over peer
keepalive link

This Feature is currently supported only on Nexus 5X00 and 600X series

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public
 Agenda
•
•
• vPC Design Best Practices
− Mixed Hardware across vPC Peers
− Dynamic Routing over VPC
− vPC and Multicast
− vPC as Data Center Interconnect (DCI)
− FHRP with vPC
− Hybrid topology (vPC and non-vPC)
− vPC and Network Services
− vPC Fex Supported Topologies
•
•
•
•

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
Design Best Practices
Mixed Hardware across vPC Peers : Line Cards

Always use identical line cards on either sides of the peer link and VPC legs !

Examples

vPC Primary vPC Secondary vPC Primary vPC Secondary

vPC Peer-link vPC Peer-link

S2
S1 S1 S2
N7700
N7000 F2E F2E M2
M1
F3 F3

vPC vPC

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
Design Best Practices
Mixed Hardware across vPC Peers : Chassis & Supervisors
• N7000 and N7700 in same vPC Construct -Supported
• VDC type should match on both peer device
• vPC peers can have mixed SUP version* (SUP1, SUP2, SUP2E)
• N5X00 and N600X in same vPC Construct –Not Supported

vPC Primary vPC Secondary vPC Primary vPC Secondary

vPC Peer-link vPC Peer-link
S2 S1 S2
S1
N7700 N5X00 N600X
N7000

*Recommended only for short period such as migration

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
Design Best Practices
Dynamic Routing over VPC

• Don’t attach routers to VPC domain via L2 port-channel

• Common workarounds:
A. Individual L3 links for routed traffic
B. Static route to FHRP VIP

A B

SVI 1 SVI 1 SVI 1 SVI 1 SVI 1 SVI 1

IP Y IP Z IP Y IP Z IP Y IP Z
VIP A VIP A VIP A VIP A VIP A VIP A

L3 ECMP S2 S1 S2
S1 S2 S1

Router SVI 2 Router

SVI 2 IP X
SVI 2 Router IP X Static Route to VIP A
IP X
BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
Design Best Practices
vPC and Multicast
Source

• vPC supports PIM-SM only

• vPC uses CFS to sync IGMP state
• Sources in vPC domain
− both vPC peers are forwarders
− Duplicates avoided via vPC loop-avoidance logic

S1 S2 • Sources in Layer 3 cloud

− Active forwarder elected on unicast metric
− vPC Primary elected active forwarder in case metric are equal

Source Receivers

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
vPC - Data Center Interconnect(DCI)
Multi-layer vPC for Aggregation
DC 1 and DCI DC 2
N Network port
vPC domain 11 vPC domain 21 E Edge or portfast
Long Distance
Dark Fiber - Normal port type

CORE
CORE

B BPDUguard
E F F E
- - F BPDUfilter
N N R Rootguard
802.1AE (Optional)
N N

- E F F E -
R
R -
- - R R
AGGR

AGGR
N N N N

- - vPC domain 10 vPC domain 20 - -

R R
R R
ACCESS

ACCESS
- -

E
E
B
B

Server Cluster Server Cluster

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
Design Best Practices
vPC as Data Center Interconnect (DCI)

PROS
• vPC is easy to configure and it provides robust and resilient interconnect
solution

CONS
• Maximum of only two Data Centers can be interconnected
• Layer 3 peering between Data Centers cannot be done through vPC and
separate links are required

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
Design Best Practices
vPC -Data Center Interconnect (DCI)

• vPC Domain id for vPC layers should be UNIQUE

• BPDU Filter on the edge devices to avoid BPDU propagation

• STP Edge Mode to provide fast Failover times

• No Loop must exist outside the vPC domain

• No L3 peering between Nexus 7000 devices (i.e. pure layer 2)

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
FHRP with vPC
HSRP / VRRP/ GLBP Active/Active
FHRP FHRP
“Active”: “Standby”:
Active for Active for
• FHRP in Active/Active mode with vPC shared L3 MAC shared L3 MAC

L3
• No requirement for aggressive FHRP timers
L2
S1 S2

• Best Practice : Use default FHRP timers

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
 Use one transit vlan to establish L3 routing
backup path over the vPC peerlink in case L3
FHRP with vPC uplinks were to fail, all other SVIs can use
passive-interfaces
Backup Routing Path
• Point-to-point dynamic routing protocol
adjacency between the vPC peers to establish a
L3 backup path to the core through PL in case of S3 S4
uplinks failure P P

OSPF/EIGRP
• Define SVIs associated with FHRP as routing L3
passive-interfaces in order to avoid routing
adjacencies over vPC peer-link
L2
P
VLAN 99
• A single point-to-point VLAN/SVI (aka transit
P

vlan) will suffice to establish a L3 neighbor OSPF/EIGRP

Primary Secondary
• Alternatively, use an L3 point-to-point link S1 vPC vPC S2
between the vPC peers to establish a L3 backup
path

P Routing Protocol Peer

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
 Hybrid topology (vPC and non-vPC)
STP Root STP Root STP Root
VLAN 1 VLAN 1 VLAN 2
Bridge Priority VLAN 2 Bridge Priority
VLAN 1  4K VLAN 1  8K
VLAN 2  8K vPC Primary vPC Secondary VLAN 2  4K
vPC Peer-link
S1 S2
peer-switch
VLAN 1
vPC1 (blocked)
S3 S4
VLAN 2
(blocked)
• supports hybrid topology where vPC and non-vPC are connected to the same vPC domain
• Need additional configuration parameters : spanning-tree pseudo-information
• If previously configured global spanning tree parameters and subsequently configure spanning
tree pseudo information parameters, then pseudo information parameters take precedence over
the global parameters.
BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
vPC and Network Services
Services Chassis w. Services VDC Sandwich
Two Nexus 7000 Virtual Device Contexts to “sandwich” services
between virtual switching layers
• Layer-2 switching in Services Chassis with transparent
services
Agg Agg

•
Layer Layer
vPC running in both VDC pairs to provide portchannel for both
inside and outside interfaces to Services Chassis
Design considerations:

• Access switches requiring services are connected to sub- Sub-Agg Sub-Agg

aggregation VDC Layer Layer

• Access switches not requiring services be connected to

aggregation VDC
• If Peering at Layer 3 is required between vPC layers an
alternative design should be explored (i.e. using STP rather
than vPC to attach service chassis) or using static routing
BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
Nexus 2000 (FEX) Straight-Through Deployment with VPC

• Port-channel connectivity from the server

• Two Nexus switches bundled into a vPC S1 S2
pair
Fabric Links
• Suited for servers with Dual NIC and
capable of running Port-Channel

Fex 100 Fex 101

HIF HIF

* This design is currently not supported on N9500 series VPC

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
Nexus 2000 (FEX)Active-Active Deployment with VPC

• Fabric Extender connected to two Nexus S1 S2

Nexus 6000 / 5000

5X00 / 6000
• Suited for servers with Single NIC or Fabric Extender dual homed to
Dual NIC not having port-channel redundant Nexus switches
Fabric Links
capability.
• Scale implications of less FEX per
system and less VPC Fex 100 Fex
101
HIF HIF
* This design is currently not supported on N7000 / N7700 and
N9X00

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
 Nexus 2000 (FEX) - Enhanced VPC
• Port-channel connectivity to dual-homed S1 S2
FEXs Nexus 6000 / 5000
• From the server perspective a single
access switch with port-channel support – Fabric Extender dual homed to
each line card supported by redundant redundant Nexus 5000
Fabric Links
supervisors
• Ideal design for a combination of single
NIC and Dual NIC servers with port- Fex Fex
100 101
channel capability HIF HIF

• Scale implications of less FEX per

system and less VPC

* This design is currently not supported on N7000 / N7700 and

N9X00

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
 vPC : Get it Right the very First time

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
Agenda
• Feature Overview
• vPC Configuration Best Practices
• vPC Design Best Practices
• Fabricpath / vPC+
• vPC in VxLAN network
• Scalability
• Reference Material

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
FabricPath: an Ethernet Fabric
Shipping on Nexus 7x00, Nexus 600x and Nexus 5x00

FabricPath

• Eliminates Spanning tree limitations

• High resiliency, fast network re-convergence
• Any VLAN, Anywhere in the Fabric
• Connect a group of switches using an arbitrary topology
• With a simple CLI, aggregate them into a Fabric
N7K(config)# interface ethernet 1/1
N7K(config-if)# switchport mode fabricpath

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
VPC vs VPC+
Architecture of vPC and FabricPath with vPC+
CE FP

CE Port FP Port

CE VLAN’s FP VLAN’s

vPC vPC+
• Physical architecture of vPC and vPC+ is the same from the access edge
• Functionality/Concepts of vPC and vPC+ are the same
• Key differences are addition of Virtual Switch ID and Peer Link is a FP Core Port
• vPC+ is not supported on Nexus 9X00 & Nexus 3X00 Series
BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
VPC+ Virtual Switch ID

S10 S20 S30 S40

• Mac address flapping on S300

• Single path to A
AB S100  S300
S200

S100 S200
FabricPath
S300

1/1 1/2
S300: CE MAC
Address Table
MAC IF

B 1/2
A A S100
S200
S100 B

Classical Ethernet
BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 55 55
VPC+ Virtual Switch

S10 S20 S30 S40

• A consistently associated to S1
• Multipathing to A
AB S1  S300

S100 S200 FabricPath S300

1/2
S300: CE MAC
Address Table
MAC IF
S1 B 1/2
virtual A S1
A B

Classical Ethernet
Refer BRKDCT-2081 – Cisco Fabric Path Technology & Design
BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 56 56
Dynamic Routing over vPC+
• Layer 3 devices can form routing adjacencies with both
the vPC+ peers over vPC
Fabricpath Core
• The peer link ports and VLAN are configured in
FabricPath mode.

• N55xx, N56xx, N6000 support this design with vPC

IPv4/IPv6 unicast and PIM-SM multicast

• This design is not supported on N7X00 P P

N55xx, N56xx,
N6000
Router/ Firewall
Fabricpath Link
Dynamic Peering Relationship
P Routing Protocol Peer P

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
Agenda
• Feature Overview
• vPC Configuration Best Practices
• vPC Design Best Practices
• Fabricpath / vPC+
• VxLAN
• Scalability
• Reference Material

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
Why VXLAN ?
 Problems being addressed:
• VLAN scale – VXLAN extends the L2 segment ID field to 24-bits,
potentially allowing for up to 16 million unique L2 segments over the
same network
• Layer 2 segment elasticity over Layer 3 boundary – VXLAN
encapsulates L2 frame in IP-UDP header
 High Level Technology Overview:
• MAC-in-UDP encapsulation.
• Leverages multicast in the transport network to simulate flooding
behavior for broadcast, unknown unicast and multicast in the same
segment
• Leverage ECMP to achieve optimal path usage over the transport
network
BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 59
 For Your
VXLAN Packet Format Reference

Outer Outer UDP Header VXLAN

Original
FCS L2 Frame FCS
Mac Header IP Header Header

14 Bytes
(4 bytes optional) 20 Bytes 8 Bytes 8 Bytes

VXLAN Port

UDP Length

Reserved
RRRR1RRR
IP Header
MAC Addr.

Misc Data

Reserved
MAC Addr.

Checksum
VLAN Type

Ether Type

Checksum
Src. Port
Protocol

Dst. IP
VLAN ID

Outer
Header
0x8100

Src. IP

0x0000
0x0800

Outer
0x11

VNID
VXLAN
UDP
Dst.

Src.

Tag

48 48 16 16 16 72 8 16 32 32 16 16 16 16 8 24 24 8

• VXLAN is a Layer 2 overlay scheme over a Layer 3 network.

• VXLAN uses Ethernet in UDP encapsulation
• VXLAN uses a 24-bit VXLAN Segment ID (VNI) to identify Layer-2 segments

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
VXLAN Terminology
VTEP – Virtual Tunnel End Point

Transport IP Network

VTEP VTEP
IP Interface IP Interface

Local LAN Segment Local LAN Segment

End System End System End System End System

• VXLAN terminates its tunnels on VTEPs (Virtual Tunnel End Point).

• VTEP has two interfaces :
1. Bridging functionality for local hosts
2. IP identification in the core network for VXLAN encapsulation / de-encapsulation.

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 61
vPC VTEP
• When vPC is enabled an ‘anycast’ VTEP
address is programmed on both vPC
peers
• Symmetrical forwarding behavior on both
peers provides
• Multicast topology prevents BUM traffic VXLAN
being sent to the same IP address across vPC VTEP vPC VTEP
the L3 network (prevents duplication of
flooded packets)
VLAN
• vPC peer-gateway feature must be
enabled on both peers
• VXLAN header is ‘not’ carried on the vPC
Peer link

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 62
VXLAN & VPC
VPC Configuration
VTEP1
vlan 10
vn-segment 10000
Map VNI to VLAN
interface loopback 0
ip address <VTEP individual IP – orphan)
ip address <VTEP anycast IP – per VPC domain> secondary
Source Interface !
individual IP is used for single attached Hosts interface nve1
anycast IP is used for VPC attached Hosts source-interface loopback0
member vni 10000 mcast-group 235.1.1.1
vtep vtep vtep vtep
1 2 3 4
VXLAN Tunnel Interface
VTEP2
vlan 10
vn-segment 10000

interface loopback 0
ip address <VTEP individual IP - orphan>
ip address <VTEP anycast IP – per VPC domain> secondary
!
interface nve1
source-interface loopback0
member vni 10000 mcast-group 235.1.1.1
H1 H2
10.10.10.10 10.10.10.20
VLAN 10 VLAN 10
(vpc) (vpc)

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
 VXLAN & VPC For Your
Reference
VPC Configuration
VTEP1 VTEP3
vlan 10 vlan 10
vn-segment 10000 vn-segment 10000

interface loopback 0 interface loopback 0

ip address 1.1.1.1/32 ip address 1.1.1.3/32
ip address 1.1.1.201/32 secondary ip address 1.1.1.202/32 secondary
! !
Interface nve1 Interface nve1
source-interface loopback0 source-interface loopback0
member vni 10000 mcast-group 235.1.1.1 member vni 10000 mcast-group 235.1.1.1
vtep vtep vtep vtep
1 2 3 4
VTEP2 VTEP4
vlan 10 vlan 10
vn-segment 10000 vn-segment 10000

interface loopback 0 interface loopback 0

ip address 1.1.1.2/32 ip address 1.1.1.4/32
ip address 1.1.1.201/32 secondary ip address 1.1.1.202/32 secondary
! !
Interface nve1 Interface nve1
source-interface loopback0 source-interface loopback0
member vni 10000 mcast-group 235.1.1.1 member vni 10000 mcast-group 235.1.1.1
H1 H2
10.10.10.10 10.10.10.20
VLAN 10 VLAN 10
(vpc) (vpc)

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
VXLAN & VPC
Dual attached Host to dual attached Host (Layer-2)

• Host 1 (H1) and Host 2 (H2) are

dual connected to a VPC domain

• As H1 is behind a VPC interface, the vtep vtep vtep vtep

1 2 3 4
anycast VTEP IP is the source for
the the VXLAN encapsulation

• As H2 is behind a VPC interface, the

anycast VTEP IP is the target
H1 H2
10.10.10.10 10.10.10.20
VLAN 10 VLAN 10
(vpc) (vpc)

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 65
Agenda
• Feature Overview
• vPC Configuration Best Practices
• vPC Design Best Practices
• Fabricpath / vPC+
• VxLAN
• Scalability
• Reference Material

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 66
vPC Scalability
For Latest Scalability numbers please refer to the scalability limits pages for the platform

Nexus 7X00
http://www.cisco.com/en/US/docs/switches/datacenter/sw/verified_scalability/b_Cisco_Nexus_7000_Series_NX-OS_Verified_Scalability_Guide.html

Nexus 5X00
http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus5600/sw/verified_scalability/701N11/b_N5600_Verified_Scalability_701N11/b_N6000_Verified_
Scalability_700N11_chapter_01.html

Nexus 600X
http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus6000/sw/verified_scalability/602N21/b_N6000_Verified_Scalability_602N21/b_N6000_Verified_
Scalability_602N12_chapter_01.html

Nexus 3000
http://www.cisco.com/en/US/docs/switches/datacenter/nexus3000/sw/configuration_limits/503_u5_1/b_Nexus3k_Verified_Scalability_503U51.html

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 67
Agenda
• Feature Overview
• vPC Configuration Best Practices
• vPC Design Best Practices
• Fabricpath / vPC+
• VxLAN
• Scalability
• Reference Material

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 68
Reference Material For Your
Reference

• vPC Best Practices Design Guide:

http://www.cisco.com/en/US/docs/switches/datacenter/sw/design/vPC_design/vPC_best_practices_design_guide.pdf
• vPC design guides:
http://www.cisco.com/en/US/partner/products/ps9670/products_implementation_design_guides_list.html
• vPC and VSS Interoperability white Paper:
http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps708/white_paper_c11_589890.html
• VXLAN Overview :
http://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/white-paper-c11-729383.html

• Fabrcipath whitepaper :
http://www.cisco.com/c/en/us/products/collateral/switches/nexus-7000-series-switches/white_paper_c11-687554.html

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 69
Key Take-Aways
vPC in 2015 vPC Benefits
• No Blocked Ports
VXLAN, ACI, Fabricpath • High availability
• Fast Convergence
vPC Design & Best Practices
Optimal vPC performance with
recommended deployment
techniques
Fabricpath VXLAN
• Eliminates Spanning-Tree *
• High resiliency
• L2 segment scalability
• vPC+ for legacy switches, • VTEP redundancy with
servers, hosts vPC

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 70
Related Cisco Live Milan 2015 Events
Technical Breakout Sessions

Session-ID Session Name

BRKDCT-2404 VXLAN deployment models - A practical perspective

BRKDCT-2334 Real World Data Center Deployments and Best Practice

Building simplified, automated and scalable DataCenter

BRKDCT-3378
network with Overlays (VXLAN/FabricPath)

BRKAPP-9000 Introduction to Application Centric Infrastructure

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 71
Call to Action
• Visit the World of Solutions for
– Cisco Campus – Data Center and Cloud
– Walk in Labs
– Technical Solution Clinics
• Meet the Engineer (Right after this session, until 30th Jan 2015)
• Lunch time Table Topics
• DevNet zone related labs and sessions
• Recommended Reading: for reading material and further resources for this
session, please visit www.pearson-books.com/CLMilan2015

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 72
Complete Your Online Session Evaluation
• Please complete your online session
evaluations after each session.
Complete 4 session evaluations
& the Overall Conference Evaluation
(available from Thursday)
to receive your Cisco Live T-shirt.

• All surveys can be completed via

the Cisco Live Mobile App or the
Communication Stations

BRKDCT-2378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 73