BN-Switching-1 Virtual Port Channel

Cisco Tech-Know Day
Frankfurt 2009
Nexus Family
Virtual Port Channel
Dieter Hadwiger
Systems Engineer Team Finance Germany
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 1

Agenda
Nexus 7000 vPC Feature Overview & Terminology
Nexus 7000 vPC Design Guidance & Best Practices
Building a vPC domain
Attaching to a vPC domain
Layer 3 and vPC
Spanning Tree Recommendations
Data Center Interconnect (& Encryption)
HSRP with vPC
vPC and Services
vPC latest enhancements
ISSU
Nexus 7000 vPC Convergence and Scalability

Nexus 7000 vPC Roadmap and Reference Material
Nexus 5000 / 2000 vPC design considerations

vPC Feature Overview &
Terminology

Feature Overview & Terminology
vPC Definition
Allow a single device to use a

port channel across two upstream
switches
Eliminate STP blocked ports
Uses all available uplink
bandwidth Logical Topology without vPC
Dual-homed server operate in

active-active mode
Provide fast convergence upon
link/device failure
Reduce CAPEX and OPEX
Available on current and future
hardware for M1 and D1
generation cards.
Logical Topology with vPC
vPC Terminology vPC peer – a vPC switch, one of a pair
vPC member port – one of a set of ports
(port channels) that form a vPC
vPC peer-keepalive vPC peer-link
link vPC – the combined port channel between
the vPC peers and the downstream device
CFS protocol
vPC peer-link – Link used to synchronize
state between vPC peer devices, must be
vPC peer 10GbE
vPC peer-keepalive link – the keepalive
vPC
vPC
vPC link between vPC peer devices, i.e., backup
member
member to the vPC peer-link
port
port
vPC VLAN – one of the VLANs carried
over the peer-link and used to
communicate via vPC with a peer device.
vPC
non-vPC VLAN – One of the STP VLANs
non-vPC
device
not carried over the peer-link
CFS – Cisco Fabric Services protocol, used
for state synchronization and configuration
validation between vPC peer devices
vPC Design Guidance &
Best Practices

Agenda
Layer 3 and vPC
HSRP with vPC
vPC and Services
ISSU


Building a vPC Domain
Configuration Steps
Following steps are needed to build a vPC (Order does Matter!)
1. Configure globally a vPC domain on both vPC devices
2. Configure a Peer-keepalive link on both vPC peer switches (make sure is operational)
NOTE: When a vPC domain is configured the keepalive must be operational to allow a
vPC domain to successfully form.
3. Configure (or reuse) an interconnecting port-channel between the vPC peer switches
4. Configure the inter-switch channel as Peer-link on both vPC devices (make sure is
operational)
5. Configure (or reuse) Port-channels to dual-attached devices
6. Configure a unique logical vPC and join port-channels across different vPC peers
vPC peer- vPC peer-link
keepalive link
vPC peer
Standalone
Port-channel vPC vPC member port
vPC Configuration Commands
configure vPC, and start the peer-keepalive link on

both peers:
(config)# feature vpc
(config)# vpc domain 1
(config-vpc-domain)# peer-keepalive destination x.x.x.x source
y.y.y.y vrf management
(conifg)# int port-channel 10
(config-int)# vpc peer-link
Move any port-channels into appropriate vPC groups

(config)# int port-channel 20
(config-int)# vpc 20

Peer Link
Definition:
Standard 802.1Q Trunk
vPC peer-link
Can Carry vPC and non vPC VLANs*

Carries Cisco Fabric Services messages (tagged as CoS=4 for
reliable communication)
Carries flooded traffic from a vPC peer
Carries STP BPDUs, HSRP Hellos, IGMP updates, etc.
Requirements:
Member ports must be 10GE interfaces one of the N7K-
M132XP-12 modules
Peer-link are point-to-point. No other device should be inserted
between the vPC peers.
Recommendations (strong ones!)
Minimum 2x 10GbE ports on separate cards for best resiliency.
Dedicated 10GbE ports (not shared mode ports)
use udld on vpc peer links
*It is Best Practice to split vPC and non-vPC
VLANs on different Inter-switch Port-Channels.
Peer Link with Single 10G Module
Common Nexus 7000 configuration:

1x 10G, 7x 1G cards
vPC recommendation is 2 10G cards
Potential problem occurs if Nexus 7000 is L3 boundary with
single 10G card
Use Object Tracking Feature available in 4.2
More information on CCO:
http://www.cisco.com/en/US/docs/switches/datacenter/sw/4_2/nx-
os/interfaces/configuration/guide/if_vPC.html#wp1529488

Peer Link with Single 10G Module – Object Tracking
Scenario:
vPC deployments with a single N7K-
M132XP-12 card, where core and peer-
link interfaces are localized on the same
card.
e1/… e1/… e1/… e1/…
This scenario is vulnerable to access-
L3
e1/… vPC PL e1/…
layer isolation if the 10GE card fails on
the primary vPC.
e1/… e1/…
L2 vPC PKL
vPC Object Tracking Solution: vPC

e2/… e2/… vPC
Primary Secondary
Leverages object tracking capability in
vPC (new CLI commands are added).
Peer-link and Core interfaces are
tracked as a list of boolean objects.
vPC object tracking suspends vPCs on
the impaired device, so traffic can get
diverted over the remaining vPC peer. rhs-7k-1(config-vpc-domain)# track <object>

Cisco Fabric Services (CFS)
Definition/Uses:
Configuration validation/comparison CFS Messaging
MAC member port synchronization
vPC member port status
STP Management
HSRP and IGMP snooping synchronization
vPC status
Characteristics:
Transparently enabled with vPC features
CFS messages encapsulated in standard Ethernet
frames delivered between peers exclusively on the
peer-link
Cisco Fabric Services messages are tagged as
CoS=4 for reliable communication.
Based on CFS from MDS product development
Many years in service, robust protocol

Peer-Keepalive (1 of 2)
Definition:
Heartbeat between vPC peers vPC peer-
Active/Active (no Peer-Link) detection keepalive link
Messages sent on 2 second interval

3 second hold timeout on peer-link loss
Fault Tolerant terminology is specific to VSS and deprecated in vPC.
Packet Structure:
UDP message on port 3200, 96 bytes long (32 byte payload), includes
version, time stamp, local and remote IPs, and domain ID.
Keepalive messages can be captured and displayed using the onboard
Wireshark Toolkit.
Recommendations:
Should be a dedicated VRF and link (1Gb is adequate)
Should NOT be routed over the Peer-Link
Can optionally use the mgmt0 interface (along with management traffic)
As last resort, can be routed over L3 infrastructure

Peer-Keepalive (2 of 2)
Cautions/Additional Recommendations:
When using supervisor management interfaces to carry the vPC peer-
keepalive, do not connect them back to back between the two switches.
Only one management port will be active a given point in time and a
supervisor switchover may break keep-alive connectivity
Use the management interface only if you have an out-of-band
management network (management switch in between).
Management Standby Management
Management Network Interface
Switch Active Management
vPC_PK Interface
vPC_PK
vPC_PL
vPC1 vPC2

vPC Member Port
Definition:
Port-channel member of a vPC peer.
Requirements:
Configuration needs to match other vPC
peer’s member port config.
In case of inconsistency a VLAN or the
entire port-channel may suspend (i.e.
MTU mismatch, inconsistent set of Vlans,
values and config).
Number of member ports on both vPC
vPC
member
peers is not required to match. port
Up to 8 active ports between both vPC

peers (16-way port-channel can be build
with multi-layer vPC)
VDC Interaction
vPC works seamlessly in any VDC based environment.
One vPC domain per VDC is supported, up to the maximum number of
VDCs supported in the system.
It is still necessary to have a separate vPC peer-link and vPC Peer-
Keepalive Link infrastructure for each VDC deployed.
Can vPC run between VDCs on the same switch?
This scenario should technically work, but it is NOT officially supported
and has not been extensively tested by our QA team.
Could be useful for Demo or hands on, but It is NOT recommended for
production environments. Will consolidate redundant points on the same
box with VDCs (e.g. whole aggregation layer on a box) and introduce a
single point of failure.
ISSU will NOT work in this configuration, because the vPC devices can
NOT be independently upgraded.

Agenda
Layer 3 and vPC
HSRP with vPC
vPC and Services
ISSU


The One and Only Rule…
ALWAYS
dual attach
devices to a vPC
Domain!!!
Attaching to a vPC Domain
IEEE 802.3ad and LACP
Definition:
Port-channel for devices for devices dual-attached to
the vPC pair.
Provides local load balancing for port-channel
members
STANDARD 802.3ad port channel
Access Device Requirements
STANDARD 802.3ad capability
LACP Optional
vPC
Recommendations:
vPC
Regular
Use LACP when available for better failover and mis- member
Port-
port
channel
configuration protection (config consistency check) port

”My device can’t be dual attached!”
Recommendations (in order of preference):
1. ALWAYS try to dual attach devices using vPC (not applicable for routed links).
PROS: Ensures minimal disruption in case of peer-link failover and consistent behavior with vPC dual-
active scenarios. Ensures full redundant active/active paths through vPC.
CONS: None
2. If (1) is not an option – connect the device via a vPC attached access switch (could use VDC to create a
“virtual access switch”).
PROS: Ensures minimal disruption in case of peer-link failover and consistent behavior with vPC dual-
active scenarios. Availability limited by the access switch failure.
CONS: Need for an additional access switch or need to use one of the available VDCs. Additional
administrative burden to configure/manage the physical/Virtual Device
3. If (2) is not an option – connect device directly to (primary) vPC peer in a non-vPC VLAN* and provide
for a separate interconnecting port-channel between the two vPC peers.
PROS: Traffic diverted on a secondary path in case of peer-link failover
CONS: Need to configure and manage additional ports (i.e. port-channel) between the Nexus 7000
devices.
4. If (3) is not an option – connect device directly to (primary) vPC peer in a vPC VLAN
PROS: Easy deployment
CONS: VERY BAD. Bound to vPC roles (no role preemption in vPC) , Full Isolation on peer-link failure
when attached vPC toggles to a secondary vPC role.
* VLAN that is NOT part of any vPC and not present on vPC peer-link
vPC and non-vPC VLANs (i.e. single attached .. )
P S P S
1. Dual Attached 2. Attached via VDC/Secondary Switch
Orphan
Ports
P S
P S
P Primary vPC
S Secondary vPC
3. Secondary ISL Port-Channel 4. Single Attached to vPC Device

”My device only does STP!”
Recommendations (in order of preference):
1. ALWAYS try dual attach devices using vPC
PROS: Ensures minimal disruption in case of peer-link failover and consistent behavior with
vPC dual-active scenarios. Ensures full redundant active/active paths through vPC.
CONS: None
2. If (1) is not an option – connect the device via two independent links using STP. Use non-
vPC VLANs ONLY on the STP switch.*
PROS: Ensures minimal disruption in case of peer-link failover and consistent behavior with
vPC dual-active scenarios. Ensures full redundant Active/Active paths on vPC VLANs.
CONS: Requires an additional STP port-channel between the vPC devices. Operational
burden in provisioning and configuring separate STP and vPC VLAN domains. Only
Active/Standby paths on STP VLANs.
3. If (2) is not an option – connect the device via two independent links using STP. (Use vPC
VLANs on this switch)
PROS: Simplify VLAN provisioning and does not require allocation of an additional 10GE
port-channel.
CONS: STP and vPC devices may not be able to communicate each other in certain failure
scenarios (i.e. when STP Root and vPC primary device do not overlap). All VLANs carried
over the peer-link may suspend until the two adjacency forms and vPC is fully
synchronized".
* Run the same STP mode as the vPC domain. Enable portfast/port type edge on host facing ports
vPC and non-vPC VLANs (STP/vPC Hybrid) Non vPC port-
channel
P S SR PR
P S
1. All devices Dual Attached via vPC 2. Separate vPC and STP VLANs
SR PR
P S
P Primary vPC
S Secondary vPC
PR Primary STP Root
SR Secondary STP Root
3. Overlapping vPC and STP VLANs

16-way Port-Channel (1 of 2)
Multi-Layer vPC can join 8 active

ports port-channels in a unique 16-
way port-channel*
vPC peer side load-balancing is Nexus
LOCAL to the peer 7000
Each vPC peer has only 8 active 16-way port

channel
links, but the pair has 16 active load
balanced links Nexus
5000
* Possible with any device supporting

vPC/MCEC and 8-way active port-channels

16-way Port-Channel (2 of 2)
16 active ports between 8

active port-channel devices
and 16 active port-channel
devices?
Nexus
vPC peer side load-balancing 7000
is LOCAL to the peer
Each vPC peer has only 8 16-port port-channel
active links, but the pair has 16 Nexus
active load balanced links to 5000
the downstream device
supporting 16 active ports
D-series N7000 line cards will
also support 16 way active
port-channel load balancing, Nexus 5000 16-port port-channel
providing for a potential 32 support introduced in 4.1(3)N1(1a)
way vPC port channel! release

Agenda
Layer 3 and vPC
HSRP with vPC
vPC and Services
ISSU


Layer 3 and vPC
Recommendations
Use separate L3 links to hook up routers to a vPC domain is still standing.
Don’t use L2 port channel to attach routers to a vPC domain unless you can
statically route to HSRP address
If both, routed and bridged traffic is required, use individual L3 links for routed
traffic and L2 port-channel for bridged traffic
Switch Switch
Po2 Po2
7k1 7k2
L3 ECMP
Po1
Router Router
Layer 3 and vPC
What can happen… (1 of 3)
vPC view Layer 2 topology Layer 3 topology
7k vPC
7k1 7k2 7k1 7k2
R
R
R
R could be any router, Port-channel looks like Layer 3 will use ECMP
L3 switch or VSS a single L2 pipe. for northbound traffic
building a port-channel Hashing will decide
which link to chose

Layer 3 and vPC
1) Packet arrives at R
S
2) R does lookup in routing table and sees 2
Po2
equal paths going north (to 7k1 & 7k2)
3) Assume it chooses 7k1 (ECMP decision)
4) R now has rewrite information to which
router it needs to go (router MAC 7k1 or
7k2)
5) L2 lookup happens and outgoing 7k1 7k2
interface is port-channel 1
Po1
6) Hashing determines which port-channel
member is chosen (say to 7k2)
7) Packet is sent to 7k2
R
8) 7k2 sees that it needs to send it over the
peer-link to 7k1 based on MAC address

Layer 3 and vPC
9) 7k1 performs lookup and sees that it

needs to send to S S
Po2
10) 7k1 performs check if the frame came
over peer link & is going out on a vPC.
11) Frame will only be forwarded if outgoing
interface is NOT a vPC or if outgoing
vPC doesn’t have active interface on
other vPC peer (in our example 7k2)
7k1 7k2
Po1

Agenda
Layer 3 and vPC
HSRP with vPC
vPC and Services
ISSU


Overview – STP Interoperability
STP Uses:
• Loop detection (failsafe to vPC)
• Non-vPC attached device
• Loop management on vPC addition/removal
Requirements:
• Needs to remain enabled, but doesn’t dictate vPC
member port state
• Logical ports still count, need to be aware of number of
VLANs/port-channels deployed!
Best Practices:
• Not recommended to enable Bridge Assurance feature
on vPC channels (i.e. no STP “network” port type) vPC
STP
vPC is running to manage
• Make sure all switches in you layer 2 domain are loops outside of vPC’s
running with Rapid-PVST or MST (IOS default is non- direct domain, or before
rapid PVST+), to avoid slow STP convergence (30+
secs) initial vPC configuration
• Remember to configure portfast (edge port-type) on
host facing interfaces to avoid slow STP convergence
(30+ secs)
Port Configuration Overview N Network port
E Edge or portfast port type
- Normal port type
B BPDUguard
Rootguard
Data Center Core R
L Loopguard
Primary Secondary
vPC vPC
vPC
HSRP Domain HSRP Layer 3
ACTIVE STANDBY
Aggregation
N N Secondary
Primary
Root Root
Layer 2 (STP + Rootguard)
- - - - - - - -
R R R R R R R R
-
Access
- - L
E E E E E
B B B B B
Layer 2 (STP + BPDUguard)

STP interaction on double failure
On a peer-link and peer-keepalive

symultaneous failure, Active/Active mode
may occur
Both vPC peers forward BPDUs with same
bridge IDs (NEW as of 4.2(x)), this resolves
the need to disable the etherchannel guard
feature on downstream devices
Before 4.2(x) BPDUs are beeing sent due to
dual active from both N7k with different
Bridge ID which results in legacy Ethernet
Guard feature (enabled by default) to kick in
and disabling the portchannel -> you would
be needed to disable portchannel guard
feature

Agenda
Layer 3 and vPC
HSRP with vPC
vPC and Services
ISSU


Data Center Interconnect
N Network port
E Edge or portfast port type
Multi-layer vPC for Agg and DCI

- Normal port type
B BPDUguard
F BPDUfilter
R Rootguard
DC 1 vPC domain 11
Long Distance
vPC domain 21 DC 2
CORE
CORE
- F F -
- -
N N
N N
- - F F - -
R R
- R -
- R -
AGGR
AGGR
N N N N
- - vPC domain 10 vPC domain 20 - -

R R
R R
Key Recommendations
ACCESS
vPC Domain id for facing vPC layers should be different

- -
ACCESS
E No Bridge Assurance on interconnecting vPCs E
B BPDU Filter on the edge devices to avoid BPDU propagation B
No L3 peering between DCs (i.e. L3 over vPC)
Server Cluster Server Cluster

Encrypted Interconnect
DC-1 DC-2
Nexus 7010 Nexus 7010
vPC vPC
CTS Manual Mode
(802.1AE 10GE line-rate
encryption)
No ACS is required

References
Validated TrustSec between Nexus

7000 connected back to back.
Validated TrustSec across EoMPLS
cloud with ASR 1000 routers and DCI Dark Fiber
Catalyst 6500s terminating EoMPLS.

Agenda
Layer 3 and vPC
HSRP with vPC
vPC and Services
ISSU


HSRP with vPC
FHRP Active/Active
Support for all FHRP protocols
in Active/Active mode with vPC
No additional configuration HSRP/VRRP
“Active”:
HSRP/VRRP
“Standby”:
required Active for Active for
shared L3 MAC shared L3 MAC
Standby device communicates L3

with vPC manager to determine
L2
if vPC peer is “Active”
HSRP/VRRP peer
General HSRP best practices
still applies.
When running active/active
aggressive timers can be
relaxed (i.e. 2-router vPC case)

HSRP with vPC
Do NOT use Object Tracking
Cautions:
Not recommended using HSRP link tracking in a vPC configuration
Reason: vPC will not forward a packet back on a vPC once it has
crossed the peer-link, except in the case of a remote member port
failure
L3 CORE
ACTIVE HSRP STANDBY HSRP

GW
GW VLAN 100, 200 GW L2/L3
Aggregation
VLAN 200
VLAN 100 VLAN 200
VLAN 100

HSRP with vPC
L3 Backup Routing
Use an OSPF point-to-point adjacency (or equivalent L3 protocol)
between the vPC peers to establish a L3 backup path to the Core
through in case of uplinks failure
A single point-to-point VLAN/SVI will suffice to establish a L3
neighborship.
OSPF
OSPF
VLAN 99
L3 OSPF
L2
Primary Secondary
vPC vPC

HSRP with vPC
Dual L2/L3 Pod Interconnect
Scenario:
Provide L2/L3 interconnect between
L2 Pods, or between L2 attached
Datacenters (i.e. sharing the same
HSRP group).
A vPC domain without an active
HSRP instance in a group would not
be able to forward traffic. Active Standby Listen Listen
Multi-layer vPC with single HSRP:
L3 on the N7K supports
Active/Active on one pair, and still
allows normal HSRP behavior on
other pair (even across different vPC
domains we support all in one HSRP
group)
L3 traffic will run across Intra-pod
link for non Active/Active L3 pair

Agenda
Layer 3 and vPC
HSRP with vPC
vPC and Services
ISSU


vPC and Services
vPC Services Integration
Services deployed as part of
Catalyst 6500 Service chassis
Investigation ongoing with
standalone services (ASA, ACE)
Appliance based services that
do not support port-channel may L3
require additional peer-link L2
connections to deal with the
additional traffic forced across
the peer-link
More information will be posted
as soon as more scenario are
verified – keep in touch w/ your
responsible Cisco SE
vPC and Services
Catalyst 6500 Services Chassis w. Services VDC Sandwich
Two Nexus 7000 Virtual Device Contexts used to “sandwich”
services between virtual switching layers
• Layer-2 switching in Services Chassis with transparent
services
• Services Chassis provides Etherchannel capabilities for
interaction with vPC
• vPC running in both VDC pairs to provide Etherchannel for
both inside and outside interfaces to Services Chassis
Design considerations:
• Access switches requiring services are connected to sub-
aggregation VDC
• Access switches not requiring services may be connected to
aggregation VDC
• May be extended to support multiple virtualized service
contexts by using multiple VRF instances in the sub-
aggregation VDC
Design Cautions:
• Be aware of the Layer 3 over vPC design caveat. If Peering at
Layer 3 is required across the two vPC layers an alternative
solution should be explored (i.e. using STP rather than vPC to
attach service chassis)
Agenda
Layer 3 and vPC
HSRP with vPC
vPC and Services
ISSU


vPC Latest Enhancements
Summary
Several enhancements to vPC:
vPC Object Tracking
vPC Peer-Gateway
vPC Delay Restore
Multi-layer vPC with single HSRP group
vPC unicast ARP handling
vPC Exclude Interface-VLAN
vPC single attached device Listing
vPC Convergence and Scalability
For more details:

4.2 Release Notes
http://www.cisco.com/en/US/docs/switches/datacenter/sw/4_2/nx-os/release/notes/42_nx-
os_release_note.html#wp218085

vPC Latest Enhancements
vPC Peer-Gateway for NAS interoperability
Local Routing for peer
Scenario: router –mac Traffic
Interoperability with non RFC

compliant features of some NAS devices
(i.e. NETAPP Fast-Path or EMC IP-
Reflect)
vPC PL
L3
NAS device may reply to traffic using L2
vPC PKL
the MAC address of the sender device
rather than the HSRP gateway.
Packet reaching vPC for the non local
Router MAC address are sent across the
peer-link and can be dropped if the final
destination is behind another vPC.
vPC Peer-Gateway Solution:
Allows a vPC switch to act as the
active gateway for packets addressed
to the peer router MAC (Non disruptive N7k(config-vpc-domain)# peer-gateway
CLI command added in the vPC global
config)
4.2(1) vPC Enhancements
vPC Delay Restore convergence improvement
Problem/Impact:
After a vPC device reloads and come
back up routing protocol needs time to
reconverge. vPCs may blackhole
routed traffic from access to core until
OSPF
layer 3 connectivity is reestablished
vPC Delay restore solution:
L3
vPC PL
Delays vPCs bringup after a vPC L2

device reload (SVI bring-up timing is vPC PKL
unchanged), vPC vPC
Primary Secondary
Allows for Layer 3 routing protocols to
converge for a more graceful
restoration.
Enabled by default with a vPC
restoration default timer of 30 seconds
Timer can be tuned according to a
specific layer 3 convergence baseline.
vPC unicast ARP handling
Problem/Impact:
Lack of interoperability with BigIP (F5
devices) using Unicast ARP requests
to monitor gateway liveness Active Standby
HSRP HSRP
Due to the hashing mechanism the vPC PL
unicast ARP requests for the HSRP L3
virtual IP may reach the secondary L2
HSRP device. If that is the case they
vPC PKL
vPC vPC
get punted to the Sup and dropped – Primary Secondary
due to NOT the active control plane
vPC unicast ARP handling solution:
4.2(1) achieve interoperability
forwarding unicast ARP requests via
the peer-link to the active HSRP
instance.
No additional configuration Is
required to enable the functionality.
vPC Exclude Interface-VLAN
Problem/Impact:
When a dual active condition is detected SVIs
and vPC ports on the secondary vPC peer are
suspended and therefore Single homed
devices on secondary peer suffer due to loss
of gateway SVI
vPC PL
Only the primary vPC peer continues data L3
plane and control plane functionalities L2
vPC PKL
vPC exclude interface-VLAN solution: vPC vPC
Primary Secondary
The vPC exclude interface-VLAN feature
ensures that a configurable list of SVIs are not
suspended on the secondary vPC peer
Consequently Layer 3 connectivity is
maintained even in a dual active condition for
a restricted selection of interfaces
Other option : configure separate VLAN(s) for
single attached devices (recommended)
N7K (config-vpc-domain)# dual-active exclude interface-vlan ?
<1-3967,4048-4093> Set allowed interface vlans
vPC single attached device Listing
Problem/Impact:
Single attached devices that are not
connected via a vPC but still carry
vPC VLANs are also known as Port #1 Port #2
orphan ports. L3
vPC PL
In case of a peer-link shut or L2

restoration, an orphan port's vPC PKL
connectivity may be bound to the

vPC vPC
Primary Secondary
vPC failure or restoration process.
vPC single attached device listing:
For this reason, NX-OS Release
4.2(1) introduces a show command
to check and list single attached
devices in the system along with
impacted VLANs.
N7K (config-vpc-domain)# show vpc orphan-ports

Agenda
Layer 3 and vPC
HSRP with vPC
vPC and Services
ISSU


In-Service Software Upgrade (ISSU)
vPC System Upgrade/Downgrade
4.1(3)
4.2(1) 4.2(1)
4.1(3)
ISSU is still the recommended system upgrade in a
multi-device vPC environment
vPC system can be independently upgraded with no
disruption to traffic.
4.1(3)
4.2(1)
Upgrade is serialized and must be run one at a time
(i.e. config lock will prevent synchronous upgrades)
Configuration is locked on “other” vPC peer during
ISSU.
No card reloads or port flaps, even different releases
during interim condition
Begin End Caveats

4.1(x) 4.2(x) None
4.2(x) 4.1(x) None

vPC Convergence &
Scalability

Convergence Topology 20 flows @1000 pps
OSPF L3 Core
Nexus 7000
N7K-1 N7K-2 L2/L3

OSPF Aggregation
Po10
Nexus 7000 vPC
16-way port-channel 4-way port-channel

Po160 Po20
L2 Access
Nexus 5000
vPC Peer Link LACP

Channel (2x10 GigE)
vPC Peer-Keepalive (GigE) 20 flows @1000 pps 20 flows @1000 pps

vPC on Nexus 7000
Convergence Numbers- Disclaimer: without engagement
Failover case Failure Topology Convergence Time
Failure Restoration
Failure of 4.1(4) 4.1(4)
secondary vPC
P S North-Bound: ~700 ms North-Bound: ~3 sec
peer*
South-Bound: ~2.5 sec South-Bound: ~3.4 sec
4.2(1) 4.2(1)
North-Bound: ~50 ms. North-Bound: 100 – 900 ms
South-Bound: ~100 ms South-Bound: 1.2 -2 s
Failure of a 4.1(4) 4.1(4)
primary vPC peer*
P S North-Bound: ~150 ms North-Bound:~4.5 secs
South-Bound: ~3 sec South-Bound: ~5 secs
4.2(1) 4.2(1)
North-Bound: ~50 ms North-Bound: ~400 ms-1.5 s
South-Bound: ~100 ms South-Bound: ~1.5 s
Failover of the 4.1(4) 4.1(4)
vPC Peer Link
P S North-Bound: ~1.3 s North-Bound: ~900 ms
South-Bound: ~1.8 s South-Bound: up to 10+ s (CSCsz88998)
4.2(1) 4.2(1)
North-Bound: 100-300 ms North-Bound: 150 - 900 ms
South-Bound: 50-500 ms South-Bound: ~ 900 ms–1.5 s
NOTE: Convergence numbers may vary depending on the specific configuration (i.e. scaled
number of VLANs/SVIs or HSRP groups) and traffic patterns (i.e. L2 vs L3 flows).
vPC on Nexus 7000
Scalability Number Improvements
Release Supported Scalability
4.1(5) 192 vPC’s (2-port) with the following,

200 VLANs
200 HSRP Groups
40K MACs & 40K ARPs
10K (S,G) w. 66 OIFs (L3 sources)
Latest 256 vPC’s (4-port) with the following,

Ankara 260 VLANs
4.2(2a) 200 SVI/HSRP Groups
40k MACs & 40K ARPs
NOTE: Supported numbers of VLANs/vPCs are NOT related to an hardware or software limit but reflect what
has been currently validated by our QA (data-points). The N7k BU is planning to continuously increase these
numbers as soon as new data-points become available. Please contact your responsible Cisco team if you
have particular VPC scaling requirements.

vPC Roadmap and
Reference Material

Roadmap and Reference Material
vPC Plan of Action
Disclaimer: without engagement – subject to correction
Bogota Cairo Delhi Future
• vPC scalability, • vPC scalability, • vPC scalability, • vPC scalability,
new data-point new data-point new data-point new data-point
targets: 50 targets: 768 vPC- targets: 2000 targets: 3072
vPCs-2Ports and 2ports and 300 FEX hosts- FEX hosts-
1000 VLANs VLANs 2ports and 300 2ports and 200
VLANs VLANs
300 vPCs-4ports • vPC over D1
and 300 VLANs ports • PVLANs over
• 16-port vPC on vPC
• Enhanced vPC D1 modules with • Config sync for
dual Active N5K downstream vPC
support • Port-Security • vPC for FEX
over vPC Host Ports
CCd and ECd Not CCd Not CCd Not CCd
1HCY’10 2HCY’10 1HCY’11

vPC/VSS Interop Test Details
Physical Logical
L3 Core
N7K-1 N7K-2
L2/L3 Aggregation
Nexus 7000 vPC

Po10
E1/26 E1/25
Po100 Po100
Te1/2/1 Te2/2/1
6K-1 6K-2 L2 Access vPC Peer Link LACP

Channel (2x10 GigE)
6500 VSS
vPC Peer-
Keepalive (GigE)
VSS VSL Channel

(2x10 GigE)

vPC/VSS Interop Test Details
The following scenarios were tested:

• VSS and vPC member failover and convergence
• Dual active scenarios and behavior
• Best practice guidelines for STP, L3 (NSF), Multicast
Catalyst 6500/Nexus 7000 interoperability:

Enterprise Solutions Engineering:
http://www.cisco.com/en/US/docs/solutions/Enterprise/Data_Center/DC_3_0/DC-3_0_IPInfra.html
Please refer to CCO for more detailed information or refer to your CiscoSE

Datacenter designs with
Nexus 5000/2000

NX-OS 4.1(3)N1(1)
Support for 512 HW (but SW allows

507 maximum) VLANs (minus
number of VSANs) Fabric Ports
Supports 12 Fabric Extenders

Supports 16 Hardware Ethernet
port channels (12 Ethernet and 4
Fiber Channel supported
concurrently as well as just 16
Ethernet Portchannels and zero FC 5020 = 52 Fabric Ports & 16 Port Channels
port-channels)
Supports the use of the GEM 5010 = 26 Fabric Ports & 16 Port Channels
Supports vPC

Nexus 2000 Fabric Extender
Network Topology – Physical vs. Logical
Physical Topology Logical Topology
Core Core
Layer Layer
VSS VSS
L3 L3
L2 L2
4x10G uplinks
FE
from each rack

Nexus 5020
Nexus 5020
FEX FEX FEX FEX FEX FEX 12 FEX 12 FEX
Servers Rack-1 Rack-N Rack-1 Rack-N

Servers
Rack-1 Rack-2 Rack-3 Rack-4 Rack-5 Rack-12

Fabric Extended Terminology
Fabric Links: connect n5k01
Nexus 5000 to Fabric
Extender (switchport mode
fex-fabric)
Host Interfaces (HIF)
FEX100 FEX102
FEX connectivity between
Nexus 5000 and Nexus FEX101
2000 (FEX) can leverage

either (static) pinning or
port-channels
FEX: N2148T-1GE
(48x1GE + 4x10GE)

Port-Channeling
With Static Pinning if a fabric A
uplink port fails, the associated
HIFs are beeing shut down
With Port-Channeling if a N5k01
fabric uplink fails then HIFs
use the remaining fabric 1,2,3,4
uplinks Fabric Ports
Port-channeling is the
Host Ports N2k01
recommended design method

1-48
pinning max-links 1

What is Nexus 2000 Single Homed
(aka Straight Through)
Typical Redundant straight-through
deployment as of 4.0(1a)
Nexus 2000 Straight-through deployment
n5k01 n5k01 n5k02

max 4 “fabric links”
FEX100 FEX102 FEX120 FEX122

FEX101 FEX121
max 12 x 2 = 576 ports x 2
FEX100 FEX102
FEX101
max 12 = 576 ports

Active/Standby
http://www.cisco.com/en/US/partner/products/ps9670/products_installation_and_configuration_guides_list.html

vPC Terminology (NX-OS 4.1(3))
Fault Tolerant vPC peer link

or peer keepalive
link
mgmt0 mgmt0
mgmt0 vrf
nexus5k01 nexus5k02
vPC member port
vPC
Peer Keepalive
Peer Link/ MCT
vPC Member Port

Virtual Port-Channel
Terminology vPC peer – a vPC switch, one of a pair
vPC peer vPC peer link
vPC member port – one of a set of ports
keepalive link (port channels) that form a vPC
vPC – the combined port channel
between the vPC peers and the
5k01 5k02 downstream device
vPC peer vPC peer link – Link used to synchronize
state between vPC peer devices, must
vPC
be 10GbE. Also carries
multicast/broadcast/flooding traffic and
data traffic in case of vpc member port
vPC
member failure
port
vPC peer keepalive link – the peer
keepalive link between vPC peer
switches. It is used to carry heartbeat
packets
CFS – Cisco Fabric Services protocol,
Orphan used for state synchronization and
Port
configuration validation between vPC
peer devices
Virtual Port-Channel
vPC Peer Link
Peer Link carries both vPC data and

control traffic between peer switches
Carries any flooded and/or orphan vPC Peer
port traffic Link
Carries STP BPDUs IGMP updates, 5k01 5k02

etc.
Carries Cisco Fabric Services
messages (vPC control traffic)
Minimum 2 x 10GbE ports
5020 (config)# interface port-channel 10

5020 (config-if)# switchport mode trunk
5020 (config-if)# switchport trunk allowed <BETTER TO ALLOW ALL VLANS>
5020 (config-if)# vpc peer-link
5020 (config-if)# spanning-tree port type network

STP implementation
Virtual Port-Channel vPC Roles
Two Nexus 5000s running vPC

appear as a single STP entity
vPC Role defines which of the
two vPC peers processes
BPDUs
Role matters for the behavior 5k01 5k02
with peer-link failures!
Role is defined under the
domain configuration
Lower priority wins - if not, lower
system MAC wins
Primary Secondary
Role Role

vPC on the Nexus 5000
2-Ports vPCs
4+ Ports vPCs
5k01 5k02
5k01 5k02
Max 16 HW-Port Channel As many as the number of ports on the 5k
eth2/1 eth2/2
eth2/1,2/2 eth2/3,2/4
vPC vPC
access
Peer Keepalive
Peer Link
vPC Member Port

Peer Keepalive
vPC with FEX

Peer Link/ MCT
vPC Member Port
Nexus 2000 Single-homed vPC Nexus 2000 active/active

(or dual homed)
mgmt network mgmt network
FT link (can be routed)
FT link (can be routed)
mgmt0 mgmt0
mgmt0 mgmt0
Peer-link
primary secondary
Peer-link
primary secondary
5k01 5k02 5k01 5k02
“fabric links” “fabric links”
vPC 1 vPC 2
FEX100 FEX120
HIF HIF
FEX100 FEX120
2 ports
HIF HIF
vPC
2-GigE ports host port channel

Nexus 2000 straight-through with vPC
n5k01 n5k02
max 24 FEXes = 1152 (24 x 48GE ports)
max 480 vPCs (each vPC has 2 ports)

Nexus 2000 dual-homed
5k01 5k02
vPC Primary vPC Secondary
Po10
max 12 FEXes

Host and Switch Port-channels to 5k
The 4.1(3)N1 release enables the
configuration of virtual port-channels from
switches connected to Nexus 5000 Mgmt network
It also enables port-channels from
servers connected redundantly to the 5k01 5k02
Nexus 5000
It enables both 2-ports port-channels and primary secondary
4+ ports port-channels
Maximum 16 4+ ports portchannels are
possible (minus the number of FC port- vPC member ports
channels)
Any of the 52 ports of the 5020 or the
26 ports of the 5010 can be utilized (i.e.
can also use the GEM modules)
vPC
2-ports 4+ ports
vPC member port host host
2-ports port channel port channel
Peer Keepalive or FT link switch 4+ ports
port channel switch
vPC Peer Link aka MCT
port channel

vPC Mixed Topology equally work
Management Network
mgmt0 mgmt0
5k01 5k02
primary secondary
FEX100 FEX120
FEX101 FEX121
2-GigE ports host port channel single attached servers and/or A/S

Double-sided vPC between Nexus 7000 and Nexus 5000
DESIGN 1 DESIGN 2
vPC on the N7k
N7k01 N7k02 N7k01 N7k02
Max 16 Ports
1 2 3 4 1 2 3 4
vPC on the N5k

N5k01 N5k02 N5k01 N5k02
1 2 3 1 2 3

Double-sided vPC between Nexus 7000 and Nexus5000 and Nexus 2000
DESIGN 3 DESIGN 4
vPC on the N7k
N7k01 N7k02 N7k01 N7k02
Max 16 Ports
1 2 3 4 1 2 3 4
vPC on the N5k

N5k01 N5k02 N5k01 N5k02
5 6 7 8 5 6 7 8
N2k01 N2k02 N2k01 N2k02
1 2 3 1 2 3
Double-sided vPC between Nexus 7000 and Nexus5000 and FEX A/A
DESIGN 5 DESIGN 6
vPC on the N7k
N7k01 N7k02 N7k01 N7k02
Max 16 Ports
1 2 3 4 1 2 3 4
vPC on the N5k

N5k01 N5k02 N5k01 N5k02
N2k01 N2k02 N2k01 N2k02
1 3 1 3
16-ports Port-Channel
Each vPC peer has

only 8 active links,
but the pair has 16
active load balanced
links
16 x 10 GigE ports

You can still use TLB with FEX A/A
you cannot just use 802.3ad or static port-channel with FEX A/A
5k01 5k02
Peer-link
primary secondary
“fabric links”
vPC 1 vPC 2
FEX100 FEX120
HIF HIF

How Many Paths?
In a typical vPC deployment, e.g.

in FEX A/A you want to tune the
traffic to use all the available
paths.
Remember that there are 3 5k01 5k02
components involved:
vPC
vPC
Nexus 5k which can load balance Primary
Secondary
based on L2/L3/L4 information Po10
FEX (which can only load balance

based on L2/L3 information)
Teaming software (which can be
configured for various load
balancing options e.g. tcp
connections)
TLB

vPC Forwarding Behavior
core1 core2
core1 core2
vPC peer link almost

unutilized
5k01 5k02
5k01 5k02
acc1 acc2 acc3 acc1 acc2 acc3

Summary Checklist
Ensure MST region is configured for
the NXOS VLAN range
Use pathcost method long N7k01 N7k02
Assign roots/secondary roots as

usual (regardless of
primary/secondary roles)
1 2 3 4
Create a single Port-channel
leveraging LACP
Trim VLANs that are used for VSANs N5k01 N5k02
Do not forget that putting a VLAN on

a vPC requires that that VLAN be on
the Peer-link too N2k01 N2k02
Make sure the configuration is not

causing Type-1 Inconsistencies
1 3
Intelligent L2 Domains POD Evolution
OTV Inter-POD Connectivity across L3
Failure Boundary Preservation
Failure
IP Cloud Boundary
Core
L3
L3 Aggregation
L2 vPC
L2MP
Access
L2
vPC vPC
Servers
STP+ vPC/VSS Cisco L2MP

STP Enhancements NIC Teaming 16x ECMP
Bridge Assurance Simplified loop-free trees Low Latency / Lossless
2x Multi-pathing MAC Scaling
Operational Flexibility
Networkers at Cisco Live 2010 - Barcelona
Registrieren Sie sich hier: www.cisco.com/go/networkersregister


BN-Switching-1 Virtual Port Channel

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

BN-Switching-1 Virtual Port Channel

Uploaded by

Copyright:

Available Formats

Cisco Tech-Know Day

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 1

Nexus 7000 vPC Convergence and Scalability

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 2

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 3

Allow a single device to use a

Dual-homed server operate in

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 6

Nexus 7000 vPC Convergence and Scalability

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 7

configure vPC, and start the peer-keepalive link on

Move any port-channels into appropriate vPC groups

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 9

Can Carry vPC and non vPC VLANs*

Common Nexus 7000 configuration:

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 11

vPC Object Tracking Solution: vPC

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 12

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 13

Messages sent on 2 second interval

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 14

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 15

Up to 8 active ports between both vPC

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 17

Nexus 7000 vPC Convergence and Scalability

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 18

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 20

1. Dual Attached 2. Attached via VDC/Secondary Switch

3. Secondary ISL Port-Channel 4. Single Attached to vPC Device

PR Primary STP Root

SR Secondary STP Root

3. Overlapping vPC and STP VLANs

Multi-Layer vPC can join 8 active

Each vPC peer has only 8 active 16-way port

* Possible with any device supporting

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 25

16 active ports between 8

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 26

Nexus 7000 vPC Convergence and Scalability

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 27

vPC view Layer 2 topology Layer 3 topology

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 29

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 30

9) 7k1 performs lookup and sees that it

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 31

Nexus 7000 vPC Convergence and Scalability

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 32

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 34

On a peer-link and peer-keepalive

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 35

Nexus 7000 vPC Convergence and Scalability

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 36

Multi-layer vPC for Agg and DCI

- - vPC domain 10 vPC domain 20 - -

vPC Domain id for facing vPC layers should be different

Server Cluster Server Cluster

Nexus 7010 Nexus 7010

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 38

Validated TrustSec between Nexus

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 39

Nexus 7000 vPC Convergence and Scalability

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 40

Standby device communicates L3