You are on page 1of 90

vPC Best Practices with Nexus

SAVBU TME Team

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

Nexus 7000
5.0(3) 5.1(2) 5.1(3) 5.2

Complete Sync

Partial Sync
Nexus 5000
5.0(3)N1
5.0(3)N2 5.1(3)N1 5.2N1 Nexus 7000

E-Rocks Nexus 3000


5.0(3)U1

5.0(3)U2

Andaman

5.1(3)U1

Complete sync done at major releases


Architectural changes Major enhancements Major new features
2010 Cisco and/or its affiliates. All rights reserved.

Partial sync done at minor releases


Critical flaws/bugs Minor new features Minor enhancements
Cisco Confidential 2

vPC basic components Hardware Specific Considerations vPC enhancements L3 and vPC Adding FEX Summary designs

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

vPC is a Port-channeling concept extending link

aggregation to two separate physical switches


Allows the creation of resilient L2 topologies based

on Link Aggregation.
Eliminates the need for STP in the access-distribution

Provides increased bandwidth


All links are actively forwarding

Physical Topology

Logical Topology

Virtual Port Channel


vPC maintains independent control planes vPC switches are joined together to form a domain
Si

L2
Si

vPC domain
Non-vPC vPC Increased BW with vPC
2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 5

vPC peer keepalive link

vPC peer link

vPC peer a vPC switch, one of a pair vPC member port one of a set of ports (port

channels) that form a vPC


Primary Secondary
vPC the combined port channel between the

vPC peers and the downstream device vPC peer


vPC peer link Link used to synchronize state

vPC

between vPC peer devices, must be 10GbE. Also carry multicast/broadcast/flooding traffic and data traffic in case of vpc member port failure

vPC member port

vPC peer keepalive link the peer keepalive

link between vPC peer switches. It is used to carry heartbeat packets


CFS Cisco Fabric Services protocol, used

for state synchronization and configuration validation between vPC peer devices
Orphan portNon-vPC member port

Orphan Port

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

Graceful consistency check:


On the N7k: NXOS 5.2 On the N5k: NXOS 5.0(2)N2(1)

IGMP bulk sync: On N7k: to be verified On N5k: starting from NXOS 5.0(3)N1(1a)

Per VLAN consistency check:


On the N7k: NXOS 5.2 On the N5k: 5.0(2)N2(1)

Multicast Optimization on Peer-link: On N7k: hidden comand as of NXOS 5.1(3) (but not supported) On N5k: starting from NXOS 5.0(3)N1(1a)

Autorecovery:
On the N7k: NXOS 5.2 On the N5k: NXOS 5.0(2)N2(1)

ARP synchronization: On N7k: NX-OS 4.2(6) and 5.0(2) (Bogota), fixed in 5.1(1) (Cairo) On N5k: under investigation for Goldcoast

vPC peer-switch: On N7k: 4.2(6), 5.x On N5k: under investigation for Goldcoast

Config-sync:
On the N7k: Freetown On the N5k: NXOS 5.0(2)N2(1)

FEX preprovisioning: On N7k: Freetown On N5k: NXOS 5.0(2)N1(1)

vPC on FEX
On the N5k: NXOS 4.2(1)N1(1) On the N7k: NXOS 5.2

Dual Layer vPC: On N7k: TBD On N5k: Fairhaven

Orphan Ports shutdown:


On N7k: NXOS 5.2 On N5k: E-Rocks+

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

vPC allows a single device to use a port

channel across two neighbor switches (vPC peers)


Eliminate STP blocked ports
Layer 2 port channel only Provide fast convergence upon

vPC Peers

Portchannel vPC Peers

link/device failure
Port channel

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

Peer Link carries both vPC data and control

traffic between peer switches


Carries any flooded and/or orphan port traffic Carries STP BPDUs, IGMP updates, etc. Carries Cisco Fabric Services messages (vPC
vPC Peer Link 5k01 5k02

control traffic)
Carries multicast traffic (more details follow)

Minimum 2 x 10GbE ports ALL VLANS used on vPC PORTS MUST BE

PRESENT ON THE PEER-LINK


5020 5020 5020 5020 5020 (config)# interface port-channel 10 (config-if)# switchport mode trunk (config-if)# switchport trunk allowed <BETTER TO ALLOW ALL VLANS> (config-if)# vpc peer-link (config-if)# spanning-tree port type network

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

Peer Keep-alive provides and out of band heartbeat

between vPC peers

Purpose is to detect and resolve roles if a Split Brain

(Dual Active) occurs timeout

Messages sent on 1 second interval with 5 second


3 second hold timeout on peer-link loss before

Peer Keepalive can be carried over the OOB management network int mgmt 0

triggering recovery

Should not be carried over the Peer-Link Use the mgmt0 interface in the management VRF Can optionally be a dedicated link, 1Gb is adequate

(first 16 ports on 5020 are 1/10GE ports)

3rd option, use a routed inband connection over L3

infrastructure (using SVIs in the default VRF)

dc11-5020-1(config)# vpc domain 20 dc11-5020-1(config-vpc-domain)# peer-keepalive destination 172.26.161.201 source 172.26.161.200 vrf management Note: --------:: Management VRF will be used as the default VRF ::--------

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

10

Peer keep-alive is a routable protocol (both N5K

Standby Management Interface Active Management Interface

and N7K)
Primary design requirement is to have a physically

different path than all other vPC traffic


In all cases do not carry the peer-keepalive

communication over the vPC peer-link On Nexus 7000 when possible use dedicated VRF and front panel ports for peer-keepalive link (1G is more than adequate). 2nd best is to use the management interfaces 3rd option is to use an upstream L3 network for peer-keepalive
If using mgmt 0 interfaces do not connect the

supervisor management interfaces back to back

In a dual supervisor configuration only one management port will be active at a given point in time!
Connect both mgmt 0 ports to the OOB network
2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 11

vPC basic components Hardware Specific Considerations

vPC forwarding rules


vPC enhancements L3 and vPC

Adding FEX
Summary designs

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

12

Cisco Nexus 5000 Series


Peer keepalive:
1st option management port. 2nd option dedicated front panel port in dedicated VLAN. 3rd option upstream L3 network

Cisco Nexus 7000 Series vPC works on all existing I/O modules Peer keepalive:

1st option dedicated front panel port in dedicated VRF.

2nd option is management interface.


3rd option upstream L3 network

M1/F1 cards can be used for vPC Peer-link requires 10 GigE cards Peer-link should not span M1 and F1,

peer-link should be made on either all F1 cards or all M1 cards

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

13

NEXUS 7000 I/O modules


Part number N7K-M132XP-12 N7K-M132XP-12L Model vPC Peer-link (10 GE Only) VPC Member Port

N7K-M148GT-11 N7K-M148GT-11L N7K-M148GS-11 N7K-M148GS-11L

N7K-M108X2-12L

N7K-F132XP-15

Cisco Confidential 14

2010 Cisco and/or its affiliates. All rights reserved.

M-Series Mode
M

Mixed Chassis Mode


M

F-Series Mode vPC Peer-link on F-Series Modules

Mixed Chassis Mode vPC Peer-link on F-Series Modules (*)

vPC Peer-link on M-Series Modules

vPC Peer-link on M-Series Modules

Recommendation : for mixed chassis mode (F1/M1) with vPC peer-link on F1 ports, use at least 2 M1 LC. This will provide resiliency for L3 features (FHRP, SVI). (*) : command peer-gateway exclude-vlan <vlan list> needed for backup routing path over vPC peer-link
2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 15

NX-OS 5.1.3 introduces new behavior for handling vPC peer-gateway in mixed chassis

mode (M1/F1) : Topology with M1 peer-link : IP/ARP packets destined to the remote Active IP/MAC get routed locally Topology with F1 peer-link : IP/ARP packets destined to the remote Active IP/MAC use the tunneling mechanism

M-Series Mode

Mixed Chassis Mode

F-Series Mode

Mixed Chassis Mode

Knob Not Required Classic behavior of peer-gateway

Knob not Required Peer Gateway not required

Knob Required for transit path/VLAN IP/ARP Tunneling over Peer link

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

16

vPC Primary S1
F1

vPC Secondary S2
F1

vPC Primary
S1
M1

vPC Secondary S2
M1

vPC Peer-link

vPC Peer-link

vPC Primary S1
M1

vPC Secondary S2
F1

vPC Peer-link

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

17

vPC basic components Hardware Specific Considerations

vPC forwarding rules


vPC enhancements L3 and vPC

Adding FEX
Summary designs

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

18

With dual-active scenarios

5k01

5k02

MAC address synchronization is interrupted


IGMP synchronization is interrupted
There is a 50% likelihood that
3 IGMP report for G1

unicast traffic is flooded and that multicast traffic is dropped

4 igmp sync lost

2 - Host subscribes to G1

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

19

There will be 2 primary switches sending independent BPDUs

VPC Port-channels on upstream/downstream switches will be error-disabled by EtherChannel Misconfiguration Guard after ~90 seconds
http://www.cisco.com/en/US/tech/tk389/tk213/technologies_tech_note09186a008009448d.shtml

If Nexus 7000/5000 is on the other end of VPC no action from

STP as 7000/5000 do not support EtherChannel Guard

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

20

When the peer-link is

disconnected
vPC secondary detects primary

5k01 vPC Primary Po10

5k02

switch is alive through peer keepalive link


The secondary vpc peer switch

vPC Secondary

suspends all its vpc member ports in order to avoid traffic drop
KEEP PEER KEEPALIVE

AND PEER-LINKS SEPARATE

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

21

dca-n7k2-vdc2

vPC supports standard 802.3ad port channels from upstream and or downstream devices Recommended to enable LACP channel-group 201 mode active
dca-n7k2-vdc2# sh run interface port-channel 201 version 4.1(5) interface port-channel201 switchport mode trunk switchport trunk allowed vlan 100-105

dc11-5020-1

dc11-5020-2

dc11-5020-1# show running int port-channel 201 version 4.1(3)N1(1) interface port-channel201 switchport mode trunk switchport trunk native vlan 100 switchport trunk allowed vlan 100-105 vpc 201

dc11-5020-2# show running int port-channel 201 version 4.1(3)N1(1) interface port-channel201 switchport mode trunk switchport trunk native vlan 100 switchport trunk allowed vlan 100-105 vpc 201

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

22

vPC forwards only on locally connected

dca-n7k2-vdc2

members of the port channel if any exist (same principle as VSS)


Multiple topology choices Square Full Mesh
dca-n7k2-vdc2# sh run interface port-channel 201 version 4.1(5) interface port-channel201 switchport mode trunk switchport trunk native vlan 100 switchport trunk allowed vlan 100-105 vpc 201

dc11-5020-1

dc11-5020-2

dc11-5020-1# show running int port-channel 201 version 4.1(3)N1(1) interface port-channel201 switchport mode trunk switchport trunk native vlan 100 switchport trunk allowed vlan 100-105 vpc 201

dc11-5020-2# show running int port-channel 201 version 4.1(3)N1(1) interface port-channel201 switchport mode trunk switchport trunk native vlan 100 switchport trunk allowed vlan 100-105 vpc 201

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

23

MAC_C

vPC maintains layer 2 topology synchronization

5
CFS

via CFS
Copies of flooded frames are sent across the

vPC-Link in case any single homed devices are attached

56 3
N5K-1 N5K-2

Frames received on the vPC-Link are not forwarded out vPC ports
1. 2. 3. Host MAC_A send packet to MAC_C FEX runs hash algorithm to select one fabric uplink N5K-1 learns MAC_A and flood packets to all ports (in that VLAN). A copy of the packet is sent across the peer link N5K-2 floods the packet to any port in the VLAN except the vPC member ports to prevent duplicated packets N7K-1 and N7K-2 repeat the same forwarding logic N5K-1 updates the the MAC address learned on the vPC port on N5K-2 via CFS

CFS

4.

1
MAC_A

5. 6.

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

24

MAC_C

Traffic is forwarded if destination address is

known (both switches MAC address tables populated)


Always forward via a locally attached member of

a vPC if it exists 1. Host MAC_C send packet to MAC_A 2. N7K-2 forwards frame based on learned MAC address 3. N5K-2 forwards frame based on learned MAC address
N5K-1# sh mac-address-table vlan 101 VLAN MAC Address Type Age Port ---------+-----------------+-------+---------+----101 001b.0cdd.387f dynamic 0 Po30 101 0023.ac64.dda5 dynamic 30 Po201 Total MAC Addresses: 4

2
N5K-1 N5K-2

3
N5K-2# sh mac-address-table vlan 101 VLAN MAC Address Type Age Port ---------+-----------------+-------+---------+----101 001b.0cdd.387f dynamic 0 Po30 101 0023.ac64.dda5 dynamic 30 Po201 Total MAC Addresses: 4

MAC_A

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

25

MAC_C

On loss of all of the locally attached

members of the vPC MAC address table is updated to forward frames for the vPC across the vPC Peer Link
Note: Po20 is the vpc peer-link
2
N5K-1 N5K-2

N5K-1# sh mac-address-table vlan 101 VLAN MAC Address Type Age Port ---------+-----------------+-------+---------+----101 001b.0cdd.387f dynamic 0 Po30 101 0023.ac64.dda5 dynamic 30 Po201 Total MAC Addresses: 4

N5K-2# sh mac-address-table vlan 101 VLAN MAC Address Type Age Port ---------+-----------------+-------+---------+----101 001b.0cdd.387f dynamic 0 Po20 101 0023.ac64.dda5 dynamic 30 Po201 Total MAC Addresses: 4

3
MAC_A

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

26


2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential 27

Both switches in the vPC Domain maintain

distinct control planes


CFS provides for protocol state synchronization

between both peers (MAC Address table, IGMP state, )


System configuration must also be kept in sync
Currently there are 2 options to keep

configuration consistent: a manual process with an automated consistency check to ensure correct network behavior config-sync Two types of interface consistency checks
Type 1 Will put interfaces into suspend state to prevent invalid forwarding of packets

Type 2 Error messages to indicate potential for undesired forwarding behavior

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

28

Type 1 Consistency Checks are intended to

prevent network failures


Incorrectly forwarding of traffic Physical network incompatibilities vPC will be suspended
dc11-5020-1# sh run int po 201 interface port-channel201 switchport mode trunk switchport trunk native vlan 100 switchport trunk allowed vlan 100-105 vpc 201 dc11-5020-2# sh run int po 201 interface port-channel201 switchport mode trunk switchport trunk native vlan 100 switchport trunk allowed vlan 100-105 vpc 201 spanning-tree guard root

dc11-5020-2# show vpc brief Legend: (*) - local vPC is down, forwarding via vPC peer-link <snip> vPC status ---------------------------------------------------------------------------id Port Status Consistency Reason Active vlans ------ ----------- ------ ----------- -------------------------- ----------201 Po201 down failed vPC type-1 configuration incompatible - STP interface port guard Root or loop guard 2010 Cisco and/or its affiliates. All rights reserved. inconsistent

Cisco Confidential

29

Type 2 Consistency Checks are intended to

prevent undesired forwarding


vPC will be modified in certain cases (e.g.

VLAN mismatch)
dc11-5020-1# sh run int po 201 version 4.1(3)N1(1) interface port-channel201 switchport mode trunk switchport trunk native vlan 100 switchport trunk allowed vlan 100-105 vpc 201 dc11-5020-2# sh run int po 201 version 4.1(3)N1(1) interface port-channel201 switchport mode trunk switchport trunk native vlan 105 switchport trunk allowed vlan 100-104 vpc 201

dc11-5020-1# show vpc brief vpc 201 vPC status ---------------------------------------------------------------------------id Port Status Consistency Reason Active vlans ------ ----------- ------ ----------- -------------------------- ----------201 Po201 up success success 100-104 2009 May 17 21:56:28 dc11-5020-1 %ETHPORT-5-IF_ERROR_VLANS_SUSPENDED: VLANs 105 on Interface portchannel201 are being suspended. (Reason: Vlan is not configured on remote vPC interface)

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

30

c-nexus5010-1# show vpc consistency-parameters global Legend: Type 1 : vPC will be suspended in case of mismatch Name ------------QoS Network QoS (MTU) Network Qos (Pause) Type ---2 2 2 Local Value Peer Value

Global Parameters

---------------------- ----------------------([], [3], [], [], [], (1538, 2240, 0, 0, 0, (F, T, F, F, F, F) ([], [3], [], [], [], (1538, 2240, 0, 0, 0, (F, T, F, F, F, F)

STP Mode
STP Disabled STP MST Region Name STP MST Region Revision STP MST Region Instance to VLAN Mapping STP Loopguard

1
1 1 1 1

Rapid-PVST
None "" 0

Rapid-PVST
None "" 0

Global QoS Parameters need to be consistent

Disabled

Disabled

STP Bridge Assurance


STP Port Type, Edge Allowed VLANs Local suspended VLANs

1
1 -

Enabled
Normal, Disabled, 1,50 50

Enabled
Normal, Disabled, 1 -

Global Spanning Tree Parameters need to be consistent

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

31

Dont forget to keep global configuration in

sync
Any configuration that could cause an error in

forwarding (e.g. loop) will disable all affected interfaces


As an example if you make a change to an

MST region you must make it on both peers


Solution: define MST region mappings from

mst region vlans 1-5, 12

mst region vlans 1-5, 10

the very beginning of the deployment, for ALL VLANs, the ones that exist as well as the ones that have not yet been created
Defining a region mapping is orthogonal to
vPC vPC vPC

creating a VLAN

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

33

vPC basic components Hardware Specific Considerations

vPC forwarding rules


vPC enhancements L3 and vPC

Adding FEX
Summary designs

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

34

Inconsistency VLAN to MST Region mapping mismatch STP global settings (BA, Loop Guard, Root Guard)

Type

Impact

Recommendation Pre-provision and MAP all VLANs on the MST region Perform STP operations per port Operate change during maintenance window Leverage graceful conflict resolution Operate change during maintenance window and/or leverage graceful conflict resolution

New Enhancements

Global

1
Spanning-tree per interface settings, switchport type (trunk/versus access) Port-channel mode

Config Sync (5.0(2)N1(1) on N5K, Freetown for N7K) & Graceful Conflict Resolution (CSCtf84865,N7K 4.2(8)& 5.2, N5K 5.0(2)N2(1))

Per-vPC

Quality of Service Configuration 2 VLANs configured on vPC

Global
Minimum disruption Per-vPC

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

35

tc-nexus5010-1# show vpc consistency-parameters global Name Type Local Value Peer Value

------------QoS

---2

---------------------- ----------------------([], [3], [], [], [], []) ([], [3], [], [], [], []) (1538, 2240, 0, 0, 0, 0) (F, T, F, F, F, F) (50, 50, 0, 0, 0, 0) (F, F, F, F, F, F)

Network QoS (MTU)

(1538, 2240, 0, 0, 0, 0)

Network Qos (Pause) Input Queuing (Bandwidth) Input Queuing (Absolute Priority) Output Queuing (Bandwidth) Output Queuing (Absolute

2 2 2

(F, T, F, F, F, F) (50, 50, 0, 0, 0, 0) (F, F, F, F, F, F)

2 2

(50, 50, 0, 0, 0, 0) (F, F, F, F, F, F)

(50, 50, 0, 0, 0, 0) (F, F, F, F, F, F)

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

36

With Graceful Resolution only ports on the

vPC secondary are suspended if a Type-1 global inconsistency occurs


This limits the impact of configuration

changes.
switch(config)# vpc domain 10 switch(config-vpc-domain)# [no] graceful
mst region vlans 1-5, 12 mst region vlans 1-5, 10

consistency-check
Requires 5.0(2)N2(1) on the Nexus 5k Requires 5.2 on the Nexus 7k
vPC vPC vPC

vPC primary

vPC secondary

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

38

5.2

5.0(2)N2(1)

Check whether STP is enabled or disabled on per-VLAN basis. VLANs that have mismatched status will be suspended on both switches

Disable STP on VLAN 5

Rest of VLANs wont be affected Prior to this change all VLANs are affected

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

39

Config-sync allows administrators to make configuration changes on one switch and have the system automatically synchronize to its peers. This eliminates any user prone errors & reduces the administrative overhead of having to configure both vPC members simultaneously. Config-sync and Graceful conflict resolution are complementary features Config-sync traffic is carried over the peer keepalive link
vPC vPC

+ vlan 12

mst region vlans 1-5, 12

mst region vlans 1-5

vPC

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

40

Global Configurations:

Which configurations are not

VLANs ACLs STP configurations QOS


Interface Level Configurations:

synchronized?

Enabling Feature
vPC domain configuration FCoE configuration

Ethernet Interfaces Port Channel Interfaces vPC Interfaces

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

41

N5000-1# feature vpc vpc domain 10 peer-keepalive destination 10.29.170.8 N5000-1#sh run switch-profile Switch-profile Apple sync-peers destination 10.29.170.8 N5000-1(config-if)# config sync N5000-1(config-sync)# switch-profile Apple N5000-1(config-sync-sp)# int ethernet 100/1/3 N5000-1(config-sync-sp-if)# switch mode trunk N5000-1(config-sync-sp-if)# verify Verify Successful N5000-1(config-if)# config sync N5000-1(config-sync)# switch-profile Apple N5000-1(config-sync-sp)# commit Commit Successful

N5000-2# feature vpc vpc domain 10 peer-keepalive destination 10.29.170.7 N5000-2#sh run switch-profile Switch-profile Apple sync-peers destination 10.29.170.7

NOTE: Verify does not push the config to peer, user must issue commit for sync to take place If sync fails, then the config is in the BUFFER

N5000-1#sh run switch-profile interface ethernet 100/1/3 switchport mode trunk


2010 Cisco and/or its affiliates. All rights reserved.

N5000-2#sh run switch-profile interface ethernet 100/1/3 switchport mode trunk


Cisco Confidential 42

Configuration is stored in a buffer until commit is applied. User can add/delete/move configuration. Once the config has been pushed via commit, it will no longer show up in buffer (it will show up in show runningconfig switch-profile X) If the commit fails due to mutex check or other reasons, the failed configuration still shows in the buffer, you have to explicitly remove it to continue

N5K-1(config-sync-sp-if)# sh switch-profile A buffer ----------------------------------------------------Seq-no Command ----------------------------------------------------1 interface Ethernet100/1/9 1.1 switchport mode trunk 1.2 switchport trunk allowed vlan 5-10 2 interface Ethernet100/1/10 2.1 switchport mode access
N5K-1(config-sync-sp)# ? buffer-delete Delete buffered command(s) buffer-move Move buffered command(s) N5K-1(config-sync-sp)# buffer-delete 1 N5K-1(config-sync-sp)# sh switch-profile A buffer ----------------------------------------------------Seq-no Command ----------------------------------------------------2 interface Ethernet100/1/10 2.1 switchport mode access

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

43

Interface Ethernet1/11

config-t area This portion is not synchronized

fex associate 100 switchport mode fex-fabric channel-group 100

Interface Ethernet1/11

switch-profile area

shut/no shut
This portion is synchronized

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

44

A port-channel may consist of port ethernet 1/1 on n5k01 And erthernet 1/2 on n5k02 FEX A/A has the same FEX configured to both N5ks, so Preprovisioning has to be configured identically

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

45

If one vPC peer needs to

be disconnected completely from the vPC domain you can still operate the remaining one For this you need to leverage the commands reload restore and autorecovery

Reload restore deals

with the split brain scenario allowing a vPC peer to bring up new vPC ports even after a reload Autorecovery deals with the sequential loss of peer-link first, and peer-keepalive second, allowing the vPC secondary to bring up the vPC ports (which were down previously)

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

46

VPC needs to be able to talk to the peer (over peer-link) before

bringing up VPC port-channels


Negotiate LACP/STP operating roles for the chassis

Wait for per-port peer parameters and handshake to bring up vPC ports

Performs peer parameters consistency check on each VPC bringup Only after VPC port-channels are brought up. What if after a full DC outage (both Nexus down), only one switch is

coming up ?
Will not bring up VPCs if after a datacenter outage, only one VPC

peer comes back up

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

47

Switch1

Switch2

Switch1

Switch1

Switch3

Switch3

Switch3

Existing vPCs are brought up When adding a new vPC member port, the port goes up
2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 48

1 Switch1 Switch2

Switch1

Switch2

Switch3

Switch1

Switch2

Switch3

Switch3
2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 49

Keepalive S1 -Primary vPC peer-link S2-Secondary S1 -Primary

Keepalive S2-Secondary vPC peer-link

vPC 1
po1

vPC 1
po1

Peerlink down and keepalive working Secondary shuts vPCs

Primary fails Po1 is completely shut


Cisco Confidential 50

2010 Cisco and/or its affiliates. All rights reserved.

Keepalive vPC Primary vPC peer-link vPC Secondary

vPC Operational Primary Keepalive

vPC peer-link

vPC 1
po1
po1

vPC 1

After 3 consecutive keepalive timeouts

Peerlink down and keepalive down

Secondary changes role and brings up vPCs

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

51

STP for vPCs is controlled by

Primary vPC

Secondary vPC

the vPC operationally primary switch and only such device sends out BPDUs on STP designated ports.
This happens irrespectively of where

BPDUs

the designated STP Root is located


The vPC operationally secondary

device proxies STP BPDU messages from access switches toward the primary vPC

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

52

vPC peer-link is a regular STP port


ECMP
Primary vPC

ECMP
Secondary vPC

SW1
Primary Root

L3 L2

vPC PK-Link vPC_PL

SW2
Secondary Root

vPC Primary Switch Source and controls STP for vPCs


vPC1 vPC2

The secondary vPC device does NOT source STP BPDUs on symmetrical vPCs

SW3 MAC_A

SW4 MAC_B

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

53

Assume the following topology with

vPC enabled on the vPC


If the Primary fails over, the

Primary / Root

Secondary becomes primary and root

secondary needs to start sending BPDUs


If the Primary was also the STP root,

the secondary also has to overtake the role as a root


If this process lasts too long, the

7k01

7k02

uplink port on 5k02 may go into BA_Inconsistent state


Better not use Bridge Assurance

BPDUs prior to the failure


5k01 5k02

BA Inconsistent

with vPC
Bridge Assurance on peer-link is

fine (and is the default)

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

54

Primary
left# sh span vlan 101 VLAN0101 Spanning tree enabled protocol rstp Root ID Priority 8293 Address 0023.04ee.be01 This bridge is the root ... Bridge ID ... Interface ---------------Po1 Po100 Role ---Desg Root Sts --FWD FWD Cost --------1 2 Prio.Nbr -------128.4096 128.4195 Type --------------(vPC) P2p (vPC peer-link) Priority Address 8293 (priority 8192) 0023.04ee.be01

Secondary ROOT

ROOT

left# sh vpc role | i mac vPC system-mac vPC local system-mac

: 00:23:04:ee:be:01 : 00:1b:54:c2:42:43

right# sh span vlan 101 VLAN0101 Spanning tree enabled protocol rstp Root ID Priority 8293 Address 0023.04ee.be01 This bridge is the root ... Bridge ID Priority Address Role ---Desg Desg 8293 (priority 8192) 0023.04ee.be01 Sts --FWD FWD Cost --------1 2 Prio.Nbr -------128.4096 128.4195 Type --------------(vPC) P2p (vPC peer-link)
Cisco Confidential 55

In Peer-Switch mode bridge-ID comes from system-mac as opposed to local mac in normal mode

... Interface ---------------Po1 Po100

2010 Cisco and/or its affiliates. All rights reserved.

BA is default enabled on Peer-Link (and recommended to remain enable), not

recommended for VPCs unless Peer-Switch feature is used


Without Peer-switch BA should be kept only on Peer-Link (no BA/Loop guard on VPCs)

Dispute is default enabled (for both RSTP and MST on VPC) UDLD [normal mode] is recommended to take out bad links from channels BA + UDLD + Dispute (on all interswitch links when using Peer-switch) when all

switches support this (nexus7000/5000)

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

56

By default on the Nexus 5x00 series, LACP sets a port to I state if it

does not receive an LACP PDU from the peer. This behavior is different on the Nexus 7000 series where the default is to suspend a port if it doesnt receive LACP PDUs.
For server facing port-channels it is better to allow LACP ports to revert

to I-state if the server doesnt send LACP PDUs. By doing this the I-state port can operate like a regular Spanning-Tree port. Also this allows immediate server connectivity when it boots up before the full LACP negotiation has taken place.
For network facing ports, allowing ports to revert to I-state creates

additional Spanning-tree state without any real benefit.


This behavior can be configured on a per Port-Channel basis with the

configuration [no] lacp suspend-individual (which is the equivalent of the Catalyst IOS command port-channel standalone-disable.
57

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

IGMP snooping shares the snooped reports with the peer vPC

switch to help with multicast forwarding


Forwarding of IGMP protocol packets is tweaked so that IGMP

reports received on one vPC switch is also forwarded to the vPC peer. Thus multicast forwarding state remains in sync on both the vPC switches.
Do NOT DISABLE IGMP Snooping!
If you need to support Firewalls / Clusters: Use static IGMP entries OR Create an IGMP querier!

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

58

vPC maintains dual active control

planes and STP still runs on both switches


IGMP join/leave messages received

vPC Primary

vPC Secondary

on one peer is forwarded to another peer via peer link


IP muticast packets are sent to host

through local port


Non-IP multicast and broadcast

vPC Primary

vPC Secondary

packets are flooded


IGMP join/leave

IGMP join/leave

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

59

So is the multicast traffic going

to the peer link?


Yes, but duplicates are avoided by using the vPC loop prevention technique, which should rather be called duplicate prevention
N7k01 N7k02

And how about orphan ports?


Orphan ports receive traffic because the multicast traffic is always sent over the peer-link
N5k01

N5k02

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

60

Assuming that there are no orphan ports it is possible to remove

multicast traffic from crossing the peer-link with the command


no ip igmp snooping mrouter vpc-peer-link (Nexus 5k) ip igmp snooping vpc peer-link-exclude (hidden command on the

Nexus 7k, not supported)

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

61

VPC peer-link is considered as mrouter

port. Therefore all multicast traffic is flooded over peer-link


A CLI was introduced in 5.0(3)N1(1) to

N5k-1

N5k-2

avoid that. With the CLI multicast traffic is sent to vPC peer-link only when it is necessary, such as, there is singly connected host
Improving multicast convergence time

IGMP Group sync

with peer-link down/up and switch reload


The CLI is not supported for FEX dual-

home topology in 5.0(3)N1(1). The limitation will be removed in upcoming release-5.0(3)N2(1)

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

62

If the peer-link is lost the vPC

vPC on the N7k N7k01 N7k02

secondary is going to shut down the vPC member ports


For single attached hosts, pls

see
CSCtc49559
1 2 3 4

and Orphan ports suspend

feature

vPC on the N5k

N5k01

N5k02

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

63

S1 -Primary

Keepalive

S2-Secondary

Intended for devices that do not support port-channel. Other devices should be dually connected by vPCs (Orphanport CLI is available only on physical ports, not on portchannels) Configure single attached devices (like FW or LB) port as orphan-port When vPC peer-link goes down, vPC secondary peer device shuts all its vPC member ports as well as orphan ports

vPC peer-link
Active or Standby Active or Standby

vPC 1
po1 CE-1 Orphan port

S1(config)# int eth 1/1 S1(config-if)# vpc orphan-ports suspend S2(config)# int eth 1/1 S2(config-if)# vpc orphan-ports suspend

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

64

vPC basic components Hardware Specific Considerations

vPC forwarding rules


vPC enhancements L3 and vPC

Adding FEX
Summary designs

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

66

vPC maintains dual active control

HW Programmed to forward frames sent to the FHRP MAC address on BOTH Switches
HSRP Active HSRP Standby

planes and STP still runs on both switches


HSRP active process communicates

the active MAC to its neighbour


Only the HSRP active process

responds to ARP requests


HSRP active MAC is populated into

the L3 hardware forwarding tables, creating a local forwarding capability on the HSRP standby device
Consistent behavior for HSRP, VRRP

and GLBP
No need to configure aggressive

FHRP hello timers as both switches are active

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

67

It recommended to not use HSRP link tracking in a vPC configuration Reason: vPC will not forward a packet back on a vPC once it has crossed the peer-link, except in the case of a remote member port failure Use an L3 point-to-point link between the vPC peers to establish a L3 backup path to the Core in case of uplinks failure A single point-to-point VLAN/SVI will suffice to establish a L3 neighbor
SVI VLAN 300
VLAN 100, 200,300

SVI VLAN 300

VLAN 100

VLAN 200

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

68

Non-RFC compliant end hosts Device required to send packets to the MAC address returned in ARP response (HSRP virtual MAC) Some non-compliant devices use the MAC address of the sender device (Switch physical MAC) NAS devices (i.e. NETAPP Fast-Path or EMC IP-Reflect) have been found to do this vPC Peer Gateway - NX-OS 4.2(1) Allows a vPC peer to respond both the the HSRP virtual and the real MAC address of both itself and its peer

peer-gateway command tells the vPC to respond to the physical MAC address of its peer

L3 L2

VLAN 200

VLAN 100

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

69

Not enabled by default After the peer-link comes up perform an ARP bulk sync over CFSoE to the peer switch Improve Convergence for Layer 3 flows
ARP TABLE
IP1 IP2 MAC1 MAC2 VLAN 100 VLAN 200 IP1 IP2

ARP TABLE
MAC1 MAC2 VLAN 100 VLAN 200

SVIs
P
S

S1(config-vpc-domain)# ip arp synchronize


P S

S2(config-vpc-domain)# ip arp synchronize


Primary vPC Secondary vPC

IP1

MAC1

IP2

MAC2

Note: CSCti06907 has been fixed

ARP Synchronization Process


2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 70

Feature
VPC interaction with FHRP Peer-gateway vPC delay restore vPC exclude VLAN ARP synchronization PIM pre-built-SPT PIM dual DR

Function
Both active and standby peer function as gateway L3 forwarding when the DMAC is peers MAC Delay bringing up vPC ports CLI to specify SVI interfaces wont be suspended when peer-link fails Synchronize ARP between two peer switches Both N5k joins source tree as PIM last hop router Both N5k can be DR when it is first hop router

Availability
HSRP VRRP

Roadmap

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

71

vPC basic components Hardware Specific Considerations

vPC forwarding rules


vPC enhancements L3 and vPC

Adding FEX
Summary designs

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

72

2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

FEX 2248, 2232, 2224 from 4.2(1)N1(1)


73

FEX2148T starting from 4.1(3)N1(1) FEX 2248, 2232 from 4.2(1)N1(1)

Fairhaven

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

74

N7K NXOS 5.1(1)

N7K NXOS 5.2

Future

active

active

active

active

active

active

active

active

active

active

active

active

N
2010 Cisco and/or its affiliates. All rights reserved.

radar

Cisco Confidential

75

Nexus 5000 Topologies (Nexus 2248TP & 2232PP)


Straight Through Dual Homed

vPC Supported with up to 2 x 8 links

FCoE Adapters supported on 10G N2K interfaces

Local Etherchannel with up to 8 links

Redundancy model Dual Switch with redundant fabric

Provides isolation for Storage topologies (SAN A and B)


Port Channel and Pinning supported for Fabric Link
2010 Cisco and/or its affiliates. All rights reserved.

Redundancy model Single switch with dual supervisor for fabric, data control & management planes No SAN A and B isolation (VSAN isolation sufficient in the future?)

Cisco Confidential

76

Nexus 7000 Topologies (Nexus 2248TP & 2232PP)


NXOS 5.2 Nexus 2248TP & 2232PP

NIC Teaming: TLB/ALB

Local Etherchannel with up to 8 links

Fabric links supported on N7K-M132XP-12 & N7K-M132XP-12L Port Channel only supported for Fabric Links
2010 Cisco and/or its affiliates. All rights reserved.

Local port channel support on 2248 & 2232

No support for DCB and FCoE (parent switch fabric ports not DCB capable yet)
Cisco Confidential 77

Nexus 7000 - vPC NXOS 5.2

Nexus 5000 Fairhaven

MCEC Etherchannel with up to 16 links

MCEC Etherchannel with up to 16 links

Redundancy model Dual Switch (each switch supports redundant supervisors)

Redundancy model Single switch with dual supervisor, fabric, line card, data control & management planes

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

78

24 FEX
Nexus 2000 Straight-through deployment

n5k01

n5k01

max 4/8 fabric links

n5k02

FEX100 FEX101

FEX102

FEX120
FEX121

FEX122

FEX100 FEX101

FEX102

max 24 x 2

max 24 with Nexus 5500 = 768 ports

Active/Standby

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

79

Peer Keepalive

FEX 2248
Cisco Nexus 2000 Series Straight-Through vPC Cisco Nexus 2000 Active-Active

Peer Link vPC Member Port

vPC Primary

vPC Secondary

vPC Primary

vPC Secondary

up to 4 ports

up to 4 ports

Fabric Links

Fabric Links
up to 4 ports up to 4 ports

FEX100
HIF
up to 8 ports

FEX120
HIF
up to 8 ports up to 24 PC per FEX up to 24 PC per FEX

vPC 1

vPC 2

FEX100
HIF

FEX120
HIF
up to 8 ports

up to 8 ports

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

80

Peer Keepalive

FEX 2232
Cisco Nexus 2000 Series Straight-Through vPC Cisco Nexus 2000 Active-Active

Peer Link vPC Member Port

vPC Primary

vPC Secondary

vPC Primary

vPC Secondary

up to 8 ports

up to 8 ports

Fabric Links

Fabric Links
up to 8 ports up to 8 ports

FEX100
HIF
up to 8 ports

FEX120
HIF
up to 8 ports up to 16 PC per FEX up to 16 PC per FEX

vPC 1

vPC 2

FEX100
HIF

FEX120
HIF
up to 8 ports

up to 8 ports

Compatible with FCoE IF server uses 2 uplinks


2010 Cisco and/or its affiliates. All rights reserved.

Doesnt support FCoE, today


Cisco Confidential 81

SAN A

SAN B

In a Dual Tier vPC configuration

FCoE traffic will NOT be load shared across both sets of fabric links
SAN A and B isolation is

LAN traffic

maintained
This may result in un-even sharing

of traffic across the multiple fabric links


FCoE + LAN on one set of fabric links LAN only on the other set of fabric links

Need to plan for the aggregate

LAN & SAN traffic

traffic capacity

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

82

vPC basic components Hardware Specific Considerations

vPC forwarding rules


vPC enhancements L3 and vPC

Adding FEX
Summary designs

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

83

root
vPC on the N7k Root

N7k01 2/9 2/10 2/9

N7k02 2/10

logical equivalent

2/1 Po51,2 N5k01

2/2

2/1

2/2

N5k02

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

84

root
vPC on the N7k Root

N7k01 2/9 2/10 2/9

N7k02 2/10

logical equivalent

2/1 Po51 N5k01

2/2 Po10

2/1

2/2

Peer Link

N5k02

primary

secondary

regular STP priority


2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 85

16 port HW Etherchan nel

vPC peer link

7010s with F1linecards


x8 32 PORTS

x8

x8

x8

16 port HW Etherchan nel

vPC peer link Running vPC only for server attach ports

5500 or 50x0

...

...

2248TPs

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

86

Clear access VLANs to create a Loop Free Topology

logical equivalent

Root HSRP primary

Secondary Root HSRP secondary

Root HSRP primary

Secondary Root HSRP secondary

SW01 2/9 2/10 2/9

SW02 SW01 2/10 2/9

Clear access VLANs to create a Loop Free Topology

SW02 2/10 2/9 2/10

2/1 Po51 N5k01

2/2 Po10 Peer Link

2/1

2/2

N5k02

primary

secondary

regular STP priority


2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 87

Clear access VLANs to create a Loop Free Topology

Traffic flows are symmetric from access

to aggregation
vPC is still useful to optimize traffic

Root HSRP primary

Secondary Root HSRP secondary

flows from access to aggregation


All traffic flows through the active HSRP

SW01 2/9 2/10 2/9

SW02

switch, in this case SW01


Client-to-Server traffic uses both SW01

2/10

and SW02
Peer-link is almost unutilized
Po51

2/1

2/2
Po10

2/1

2/2

N5k01

Peer Link

N5k02

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

88

Following steps are needed to build a vPC

Define domains
Establish Peer Keepalive connectivity Create a Peer link Create vPCs Make Sure Configurations are

N7k01

N7k02

Consistent / leverage config-sync / configure graceful conflict resolution

5 6 N2k01

7 8 N2k02

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

89

Ensure domain-id or system-mac differs

N7k01

N7k02

between Agg pair and Access pair


Connect the N7ks with redundant peer-links

across linecards
Connect the N5ks with redundant peer-links Create a single Port-channel leveraging LACP
1

LACP
2 3 4

between Aggregation and Access


Do not forget that putting a VLAN on a vPC
N5k01 5 6 N2k01 N5k02 7 8 N2k02

requires that that VLAN be on the Peer-link too


If you foresee significant multicast traffic, or

there is a high percentage of single attached devices, you may want to size the peer-link to match the uplink bandwidth utilization

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

90

If you use the peer switch functionality, then define Identical Priorities on the Aggregation Layer switches, to make them the root Do not use Bridge Assurance Keep the default STP priorities on the access layer switches IF using MST, Make Sure that VLAN range Configurations are Consistent With MST be aware of the NXOS VLAN range and of the Global Type-1 Inconsistencies, hence configure VLAN-to-region mappings from day 1 Use pathcost method long Configure STP port type edge or port type edge trunk

N7k01

N7k02

5 6 N2k01

7 8 N2k02

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

91

Configure HSRP priorities as usual, both

peers forward L3 traffic


N7k01 N7k02

Configure vPC delay restore to avoid L3

traffic loss upon reboot


Create a L3 backup link between the N7k Configure peer-gateway for firewalls, load

balancers, filers
Configure regular L3 ECMP from the core to

the aggregation layer

N5k01 5 6 N2k01

N5k02 7 8 N2k02

1
2010 Cisco and/or its affiliates. All rights reserved.

3
Cisco Confidential 92

Make sure to leverage Reload

N7k01 2/9 2/10 2/9

N7k02 2/10

Restore and auto-recovery


Make sure to have mgmt0

connectivity for config-sync to work (you may want to use the same mgmt0 for vPC peer keepalive)
FEX A/A provides redundancy and

2/1

2/2

2/1

2/2

each HIF
Config-sync also helps with regular

N5k01

N5k02

port channels
FEX pre-provisioning is highly

recommended

2010 Cisco and/or its affiliates. All rights reserved.

Cisco Confidential

93

Thank You