You are on page 1of 124

VPC & VSS: Operation and Troubleshooting

BRKCRS-1930

VSS and VPC


enable us to build EtherChannel to 2 separate
switches and transform network building block
from this

to this

or, logically

No blocked ports, More usable bandwidth, Load-sharing


Distribution or link failure != network reconvergence
BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Goals
Understand general
concepts of VPC on Nexus
7000 and VSS on Catalyst
6500
Study the impact of VPC and
VSS on bridging and routing
Learn how to troubleshoot
VPC and VSS

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Spirit of this session

Simple description on how things work


Special cases
Troubleshooting

More on the topic


Cisco Catalyst Virtual Switching System
(BRKCRS-3468)
Advanced Enterprise Campus Design: Virtual Switching System
(BRKCRS-3035)
Deploying Virtual Port Channel in NXOS
(BRKDCT-2048)

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

VSS

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

VSS Agenda
Initialization
Internal redundancy considerations
Spanning Tree

1st hop redundancy


Traffic forwarding

Multicast considerations

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

VSS
1 active redundant control plane
single config
single point of management

VSS domain

2 active data planes

Active

Standby

Active
Control Plane

Standby switch is essentially a


set of additional linecards
Control messages and Data
frames flow between active and
standby via VSL

Standby

VSL

Control Plane

Active

Active

Data Plane

Data Plane

Dual-Active
detection link

MEC

(can be seen as backplane


extension)

Special encapsulation on VSL


frames to carry additional
information
BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

VSS initialization
Before the Virtual Switch domain can become active, the Virtual Switch Link
(VSL) must be brought online to determine Active and Standby roles. The
initialization process essentially consists of 3 steps:
1

Link Bringup to establish connectivity with remote chassis

Link Management Protocol (LMP) used to track and reject Unidirectional Links,
Exchange Chassis ID and other information between the 2 switches

LMP

LMP

RRP

RRP

Role Resolution Protocol (RRP) used to determine compatible Hardware and


Software versions to form the VSL as well as determine which switch becomes
Active and Hot Standby from a control plane perspective

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Troubleshooting VSS: quick sanity check


vss# sh switch virtual
Switch mode
:
Virtual switch domain number :
Local switch number
:
Local switch operational role:
Peer switch number
:

Virtual Switch
111
1
Virtual Switch Active
2

vss# sh switch virtual link


VSL Status : UP
VSL Uptime : 18 hours, 38 minutes
VSL SCP Ping : Pass
VSL ICC Ping : Pass
VSL Control Link : Te1/6/1

In VSS mode?
Domain# unique for each VSS?
Role of this switch
Peer-switch visible?
VSL is up?
Link used to carry control plane
messages (ICC, IPC, SCP)
VSL member-links state
Redundancy mode SSO?

vss# sh switch virtual link port


LMP summary
Link info:
Configured: 2
Operational: 1
Peer Peer
Peer
Peer
Timer(s)running
Interface Flag State
Flag MAC
Switch Interface (Time remaining)
-------------------------------------------------------------------------------Te1/5/4
v
link_down
Te1/6/1
vfs operational vfs 0007.0d72.4800 2
Te2/6/1
T4(960ms)
T5(29.98s)
...
vss# sh redundancy states
my state = 13 -ACTIVE
peer state = 4 -STANDBY COLD
Mode = Duplex
...
BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Aside from packet/bit rate this is


one-stop-shop command for VSL
packet and error counters
Always take 2-3 samples
All errors should be at or near zero
and most importantly not
incrementing (giants are ok)

Troubleshooting VSL:
counters

vss# sh switch virtual link counters


Port
Po10
Te1/6/4
Te1/6/5

InOctets
3084500343
523470151
2814244020

InUcastPkts
31059
139662
11346

InMcastPkts
7382085
1323349
6883221

InBcastPkts
1046088
1045940
258

Port
Po10
Te1/6/4
Te1/6/5
...

OutOctets
1457635126
363835687
1214900160

OutUcastPkts
1467466
264788
1202788

OutMcastPkts
9890548
2732502
8103037

OutBcastPkts
0
0
0

Port
Po10
Te1/6/4
Te1/6/5
Port
Po10
Te1/6/4
Te1/6/5
Port
Po10
Te1/6/4
Te1/6/5
BRKCRS-1930

Align-Err
0
0
0
Single-Col
0
0
0
SQETest-Err
0
0
0
2011 Cisco and/or its affiliates. All rights reserved.

FCS-Err
0
0
0
Multi-Col
0
0
0
Deferred-Tx
0
0
0
Cisco Public

Xmit-Err
0
0
0
Late-Col
0
0
0
IntMacTx-Err
0
0
0

...
...
...
...
...
...
...
...
...
...
...
...
10

Complete information about LMP


layer of VSLP
At least 1 link should be operational
vss# sh switch virtual link detail
Should see a neighbor
...
LMP summary
Should not see any events except
...
t4_exp (hello tx timer expiry)
LMP neighbors
Non-zero (low number) error
Peer Group info:
# Groups: 1
(* => Preferred
PG) are acceptable as long as
counters
PG #
MAC
Switch Ctrl Interface Interfaces they do not increment (take 2-3
snapshots)
---------------------------------------------------------------

Troubleshooting VSL: LMP

*1
0004.9bbe.ac00
...
LMP hello timer
...
LMP FSM info

Te1/6/4

Te1/6/4, Te1/6/5

sm(vslp_lmp 6/4), running yes, state operational


Last transition recorded: (hello)-> operational (t4_exp)-> operational (hello)->
operational (hello)-> operational (t4_exp)-> operational (hello)-> operational
...
LMP counters
Tx
Rx
Interface
OK
Fail
Bidir
Uni
Fail
Bad
-------------------------------------------------------------------Te1/6/4
805969 0
806270 7
0
0
Te1/6/5
640674 0
640726 3
0
0
Rx error details
Interface My info
My info
Bad MAC
Bad switch Domain id
Peer info
mismatch
absent
Address
id
mismatch
mismatch
------------------------------------------------------------------------------Te1/6/4
0
7
0
0
0
0
Te1/6/5
0
3 its affiliates. All rights 0reserved.
0
0
BRKCRS-1930
Cisco0
Public
2011 Cisco and/or

11

Troubleshooting VSL: LMP


vss# sh switch virtual link port
LMP summary
Link info:

Configured: 2

Compared to previous command


this one provides details of the
previous failure (if there was any) of
VSL links
Rest of the information is identical

Operational: 2

Peer Peer
Peer
Peer
Timer(s)running
Interface Flag State
Flag MAC
Switch Interface (Time remaining)
-------------------------------------------------------------------------------Te1/6/4
vfsp operational vfsp 0004.9bbe.ac00 2
Te2/6/4
T4(756ms)
T5(29.98s)
Te1/6/5
vfsp operational vfsp 0004.9bbe.ac00 2
Te2/6/5
T4(756ms)
T5(29.92s)
Flags:

v - Valid flag set


s - Negotiation flag set

Timers: T4 - Hello Tx Timer

f - Bi-directional flag set


p - Peer detected flag set

T5 - Hello Rx Timer

LMP Status
Last operational
Current packet
Last Diag
Time since
Interface Failure state
State
Result
Last Diag
------------------------------------------------------------------------------Te1/6/4
Link down
Hello bidir
Never ran
-Te1/6/5
Link down
Hello bidir
Never ran
-LMP hello timer
Hello Tx (T4) ms
Hello Rx (T5*) ms
Interface
State
Cfg
Cur
Rem
Cfg
Cur
Rem
------------------------------------------------------------------------Te1/6/4
operational 1000
756
30000
29896
Te1/6/5
operational 1000
756
30000
29228
BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

12

One of the switches must be


standby. If both are active it means
VSS has recovered from dualactive condition, but new standby
has not been reloaded, most likely
due to unsaved config
This only refers to local switch

Troubleshooting VSL:
RRP

vss# sh switch virtual role detail


Switch

Switch Status Preempt


Priority Role
Session ID
Number
Oper(Conf) Oper(Conf)
Local Remote
-----------------------------------------------------------------LOCAL
1
UP
FALSE(N )
100(100) ACTIVE
0
0
REMOTE
2
UP
FALSE(N )
100(100) STANDBY 6480
9910
RRP Counters:
-------------------------------------------------------------------Inst. Peer Direction Req
Acc
Est
Rsugg
Racc
---------------------------------------------------------------------1
1
Tx
0
1
0
1
3
1
1
Rx
2
0
1
0
3
RRP FSM info:
-------------------------------------------------------------------sm(vslp_rrp RRP SM information for Instance 1, Peer 1), running yes, state role_res
Last transition recorded: (lmac)-> lstart (req)-> hold (srt_exp)-> hold (req)-> hold
(est)-> role_neg (srt_exp)-> role_neg (racc)-> role_res (racc)-> role_res (srt_exp)> role_res (racc)-> role_res (srt_exp)-> role_res (srt_exp)-> role_res
In dual-active recovery mode: No

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

13

All ports on both sides of VSL


should be in bundled (P) state
Verify reliability of each individual
VSL link output interface specifies
egress link (one of the VSL
interfaces). VSLP ping should work
when VSL is up, even if remote is in
RPR mode etc

Troubleshooting VSL

vss# sh switch virtual link port-channel


Flags: D - down
P - bundled in port-channel
I - stand-alone s - suspended
H - Hot-standby (LACP only)
R - Layer3
S - Layer2
U - in use
N - not in use, no aggregation
w - waiting to be aggregated
Group Port-channel Protocol
Ports
------+-------------+-----------+------------------10
Po10(RU)
Te1/6/4(P)
Te1/6/5(P)
20
Po20(RU)
Te2/6/4(P)
Te2/6/5(P)
vss# ping vslp output interface t1/6/4 count 100 size 1388
Type escape sequence to abort.
Sending 100, 1388-byte VSLP ping to peer-sup via output port 1/6/4, timeout is 2
seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (100/100), round-trip min/avg/max = 12/12/28 ms

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

14

Troubleshooting VSL:
what information to collect
Note: with VSS many commands use switch <#> module <#>
notation instead of just module <#>

In case of issues with VSL or VSS bring up, collect the following
information
sh tech
(if VSS is split, collect from both sides)
remote command switch sh monitor event vslp all detail
(if VSS is split, collect from both sides)

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

15

VSS Agenda
Initialization
Internal redundancy considerations
Spanning Tree

1st hop redundancy


Traffic forwarding

Multicast considerations

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

16

High Availability
Redundancy Mechanisms
The default redundancy mechanism between the 2 VSS chassis and their associated
supervisors is NSF/SSO, allowing state information and configuration to be
synchronized. Additionally, only in NSF/SSO mode does the Standby supervisor PFC,
Switch Fabric, modules and their associated DFCs become active
Switch 2
SSO Standby

Switch 1
Active

VSL

Should a mismatch of information occur between the Active and Standby Chassis, the
Standby Chassis will revert to RPR mode, where only configuration is synchronized, but
PFC, Switch Fabric and modules will not be brought up
Switch 2
12.2(33)SXH2
RPR Standby

Switch 1
12.2(33)SXI3
Active
VSL
BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

17

Troubleshooting redundancy:
why standby is not in SSO mode
In case of certain mismatches standby will only boot to RPR mode
(fabric, PFC & modules will be down)
vss# show switch virtual redundancy
My Switch Id = 1
Peer Switch Id = 2
Last switchover reason = none
Configured Redundancy Mode = sso
Operating Redundancy Mode = rpr
...
vss# show switch virtual redundancy mismatch
Startup Config Mismatch:
Mismatch in config file between local Switch 1 and peer Switch 2:
ACTIVE : Interface TenGigabitEthernet1/6/5 shutdown
STANDBY : Interface TenGigabitEthernet1/6/5 not shut

Other possibilities
IOS version mismatch
Other VSL-related config mismatch
Non-SSO redundancy mode is configured
Forwarding engine (PFC) mismatch
BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

18

VSS with 4 supervisors


Initially in-chassis redundant
supervisors were kept in rommon
not used

Pre-12.2(33)SXI4
VSL

As of 12.2(33)SXI4 in-chassis
redundant supervisors function as
a linecard ports are useable

Active

SSO

rommon>

rommon>

Si

Si

Before switching to linecard mode


supervisors will boot to RPR-warm
mode meaning they will have their
configuration synchronized
If active supervisor fails entire
chassis is reloaded 2nd chassis
takes over same model as with
2 sups

VSL

If supervisor fails completely


(doesnt boot) or removed, the inchassis redundant supevisor will
boot as active supervisor no
need to follow procedure for
supervisor replacement

Active

SSO

RPR-warm

RPR-warm

Si

Si

12.2(33)SXI4 and later

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

19

What is Dual-Active?
If VSL goes down standby needs
to know if it was just VSL or the
active switch that failed

Si

For faster failovers assumption is


that active switch fails Old
standby becomes Active a.s.a.p.
If old Active is still there however
we will have 2 devices with
identical config on the network
IGP adjacencies will start to flap
or will go down

Layer3-MEC

Standby
Active

Active

Si

L2 MEC will be error-disabled


after ~1 minute by EtherChannel
misconfig guard (because of
receiving 2 different BPDUs)

VSL

Si

Layer2-MEC

Dual-active, if not detected will cause severe network outage


Configure robust dual-active detection
BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

20

Dual-Active Detection options


Enhanced PAGP

Switch 1

Active

Switch 2

Hot Standby

Requires PAGP+ capable neighbor with


3750
12.2(46)SE
4500
12.2(44)SE
6500
12.2(33)SXH

VSLP Fast Hello

Switch 1

Active

Switch 2

VSLP

VSLP

IP-BFD

Switch 1

Switch 2

BFD

Hot Standby

BFD

Active

L2 Heart Beat Link

L3 Heart Beat Link

Software-12.2(33)SXI

Software -12.2(33)SXH1

Hot Standby

Enhanced subsecond detection in


12.2(33)SXI3

Software -12.2(33)SXH1

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

21

Dual Active Recovery


Switch 1 detects that switch 2 is now also active triggering dual active
condition thus switch 1 brings down all the local interfaces to avoid network
instability. Until VSL link restoration occurs, switch 1 is isolated from the
network;
Once the VSL link comes up, the role negotiation determines that switch 1
needs to come up in STAND_BY mode hence it reboots itself; finally, all
interface on switch 1 are brought on line and switch 1 assumes STAND_BY
role

OLD
ACTIVE

New
ACTIVE

Switch 1
Reboot and
Comes Up in
STAND_BY
Mode

Switch 2 in
ACTIVE
Mode

Switch 1
All
Interfaces
Down

VSS Restoration

Dual Active Recovery


BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

22

Dual-active recovery,
If configuration was changed but has not been saved the would-bestandby switch will not be reloaded following VSL recovery
Save the config & reload standby
19:54:59: %VSLP-SW2_SP-5-RRP_MSG: Role change from Active to Standby and hence need
to reload
19:54:59: %VSLP-SW2_SP-5-RRP_UNSAVED_CONFIG: Ignoring system reload since there are
unsaved configurations. Please save the relevant configurations
19:54:59: %VSLP-SW2_SP-5-RRP_MSG: Use 'redundancy reload shelf' to bring this switch
to its preferred STANDBY role

Reload from active switch will not correct this


After reloading it might happen that config between Active and Standby
is not consistent Standby will come up in RPR mode
Save the config once again and reload standby again (redundancy
reload peer)

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

23

Virtual Switching System


Which Dual Active Recovery Method Should I Use?

Since dual-active detection is important


redundancy is highly recommended

Si

Si

ePAgP

Use Fast-hello + e-PAgP


In case of all-LACP deployment, use Fasthello over port-channel
Only case where BFD had advantage was in
pre-SXI3 release with routed ECMP uplinks
and OSPF

Redundant
VSL Fiber

VSLP Fast-Hello
or BFD

ePAgP

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

24

VSS Agenda
Initialization
Internal redundancy considerations
Spanning Tree

1st hop redundancy


Traffic forwarding

Multicast considerations

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

25

Spanning Tree and VSS


Physical
Active

Logical
Standby

4
1

STP process

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

VSS domain behaves as a single bridge


STP runs only on SP of active switch

VSL is not part on STP and will not be blocked

BPDUs will travel across single link of the MEC

STP will be blocking ports is there are redundant


links Keep STP enabled
Cisco Public

26

Troubleshooting STP
vss#sh spanning-tree interface po201 detail
Port 5767 (Port-channel201) of VLAN0001 is designated forwarding
Port path cost 3, Port priority 128, Port Identifier 128.5767.
Designated root has priority 0, address 001e.4963.7b94
Designated bridge has priority 32768, address 0008.e3ff.fdbd
Designated port id is 128.5767, designated path cost 16
Timers: message age 0, forward delay 0, hold 0

Number of transitions to forwarding state: 1


Link type is point-to-point by default
BPDU: sent 4447, received 12

...

STP state, role and BPDU counters


for given port
All debugging for STP is on active
SP
Limit debugs to port in question
Abbreviated BPDU debug
vss# remote login switch
Detailed BPDU debug (when
vss-sp# debug interface po201
enabled together with abbreviated
Condition 1 set
one)
vss-sp# debug spanning-tree switch tx
Observe normal precautions
Spanning Tree Switch Shim transmit bpdu debugging is on
regarding debugs
Dec 6 14:59:22.594: SW1_SP: STP SW: FAST TX: VLAN 555 Port-channel201: bpdu size 116, refcnt 1
Dec 6 14:59:23.502: SW1_SP: STP SW: FAST TX: VLAN 1 Port-channel201: bpdu size 112, refcnt 1
Dec 6 14:59:23.502: SW1_SP: STP SW: FAST TX: VLAN 1 Port-channel201: bpdu size 116, refcnt 1
Dec 6 14:59:24.594: SW1_SP: STP SW: FAST TX: VLAN 555 Port-channel201: bpdu size 116, refcnt 1
vss-sp# debug spanning-tree switch tx decode
Spanning Tree Switch Shim decode transmitted packets debugging is on
Dec 6 14:59:43.510: SW1_SP: STP SW: FAST TX: 0180.c200.0000<-0015.6301.26f8 type/len 0026
Dec 6 14:59:43.510: SW1_SP:
encap SAP linktype ieee-st vlan 1 len 112 on v1 Po201
Dec 6 14:59:43.510: SW1_SP:
42 42 03 SPAN
Dec 6 14:59:43.510: SW1_SP:
CFG P:0000 V:00 T:00 F:00 R:0000 001e.4963.7b94 00000010
Dec 6 14:59:43.510: SW1_SP:
B:8000 0008.e3ff.fdbd 96.87 A:0400 M:1400 H:0200 F:0F00
...
vss-sp# undebug all
All possible debugging has been turned off
BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

27

Spanning Tree stability features recap


Feature

UDLD

Bridge
Assurance
(BA)

Dispute

Loop
Guard

Condition
Detects if link becomes
unidirectional
I.e. link cannot carry BPDUs
both ways causes loops
Expects to receive a BPDU
every hello_time from the
peer.
I.e. cases of dead control
plane on the remote side,
also BPDU loss
Checks the remote port role
in the received BPDU, role
should not be designated in
BPDU received on
designated port
Cases of unidirectional
communication
Doesnt allow port to take
designated role if it stopped
receiving BPDUs
Unidirectional
communication, control plane
issues on remote

BRKCRS-1930

Works on

Effect

Note

Physical
port

Error-disables
unidirectional
links

Useful on port-channels to
take out broken links,
alternative fast-timers
PAGP/LACP

Blocks port at
STP level
(BAinconsistent
state)

Main protection mechanism


where supported, alternative
is Loop Guard

Logical
port

Blocks port at
STP level
(Disputed
state)

Complements BA, on by
default. Somewhat overlaps
with UDLD, but not as
effective on port-channels.
Only works with RSTP/MST
BPDUs

Logical
port

Blocks port at
STP level
(Loopinconsistent)

Superseded by BA + Dispute,
use with PVST+ or when BA
is not supported

Logical
port

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

28

Bridge assurance, Dispute & UDLD

Preferred combination is Bridge Assurance + UDLD normal


mode + Dispute (on all interswitch links) when both sides
support it
UDLD is needed to take out bad links from port-channels
(otherwise BA or Dispute will keep whole port-channel
blocked). PAgP/LACP will take out bad links, but will take
longer (~105sec vs ~20sec for UDLD with 7 sec timer)
If preferred config is not supported use Loop Guard + UDLD
(supported by all Cisco switches)

Defaults: BA/UDLD disabled, Dispute - enabled

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

29

VSS Agenda
Initialization

Internal redundancy considerations


Spanning Tree
1st hop redundancy
Traffic forwarding
Multicast considerations

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

30

Asymmetric Routing
Alternating HSRP Active between
distribution switches can be used
for upstream load balancing,
however downstream traffic hits
both distribution block switches
This can cause a problem
with unicast flooding
ARP entries age in 4 hours while
L2 entries age in 5 minutes
ARP entry with no matching L2
entry unicast flooding
In many cases when the HSRP
standby needs to forward a frame
it will have to unicast flood the
frame since its CAM table is
empty

Switch 1: Active
HSRP and Root
Bridge VLAN 3
CAM Table
Empty for
VLAN 2

Switch 2: Active
HSRP and Root
Bridge VLAN 2

Si

CAM Table
Empty for
VLAN 3

Si

VLAN 3

VLAN 2

VLAN 3

VLAN 2

With VSS there is single logical router thus no asymmetric routing


BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

31

1st hop redundancy with VSS


MAC_B Router MAC
IP B IP A

PC B

Router MAC
0001.0002.0003

VSS acts as 1 router there is 1 router MAC


address, both switches will L3 switch packets
destined to that MAC address

Router MAC
0001.0002.0003

Once either switch learns dynamic MAC address,


other switch will also learn no unicast floods
due to asymmetry of traffic between switches
In case of failover router MAC address does not
change Inherrent 1st hop redundancy

PC A

MAC_A Router MAC


IP A IP B
BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

32

VSS mac-address
By default VSS will use Router mac-address from active switch backplane
Router mac-address is maintained across switchovers no 1st hop redundancy
protocol is needed
If entire VSS system is brought down and then up again and switch 2 ends up
being active router mac-address might change (this will only have impact on
devices that ignore gratuitous ARPs)
To avoid such change, use mac-address use-virtual with this command VSS will
use special mac-address reserved for VSS
vss(config)#switch virtual domain 111
vss(config-vs-domain)#mac-address use-virtual
Configured Router mac address is different from operational value. Change will take
effect after config is saved and the entire Virtual Switching System (Active and
Standby) is reloaded.

Virtual mac is based on 0008.e3ff.fc00


Alternatively router-mac maybe statically configured with mac-address
<address> in the domain config context
BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

33

Troubleshooting Router-MAC
When VSS receives a packet destined to Router-MAC it will try to L3 switch
(route in hardware) the packet, else the packet will be bridged
vss# sh interface vlan 226
What is router MAC for given
Vlan226 is up, line protocol is up
Hardware is EtherSVI, address is 0008.e3ff.fdbc (bia 0008.e3ff.fdbc) interface
Internet address is 192.168.222.18/30
It should be pointing to the Router
...
vss# sh mac-address-table address 0008.e3ff.fdbc vlan 226 all
Legend: * - primary entry
age - seconds since last seen
n/a - not available

Actual hardware L2 entry must


have non-zero Xtag in order for
forwarding engine to consider such
packets for L3 switching

vlan
mac address
type
learn
age
ports
------+----------------+--------+-----+----------+-------------------------Supervisor switch 1 Module 6
* 226 0008.e3ff.fdbc
static No
Router
Supervisor switch 2 Module 6
* 226 0008.e3ff.fdbc
static No
Router
vss# sh mac-address-table address 0008.e3ff.fdbc vlan 226 detail switch 2 module 6
MAC Table shown in details
========================================
PI_E RM RMA Type Alw-Lrn Trap Modified Notify Capture Flood
Mac Address Age Pvlan SWbits Index XTag
----+---+---+----+-------+----+--------+------+-------+------+--------------+----+------+------+------+---Supervisor switch 2 Module 6
Yes No
No ST
No
No
No
No
No
No
0008.e3ff.fdbc 0xE8 226
0
0x380 1
BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

34

MAC address learning with VSS

MAC A is learned on lower MEC, triggering the


frame to be sent to every forwarding engine
(DFC/PFC) Flood to Fabric mechanism (HW)

Internal frame header (carried over VSL) includes


source index which identifies source port and
hence the MAC is learned on lower MEC although
the frame is received on VSL

PC B

Depending on how traffic is flowing through VSS


some forwarding engines might not see the
packets from A after initial flood to fabric which
might lead to aging of address and flooding
MAC synchronization feature keeps address from
expiring as long as traffic from that address is
seen anywhere in the system

PC A

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

35

MAC address synchronization


Initial new learns are syncronized between switch 1 and switch 2
However if only switch 1 or switch 2 sees the traffic for given address L2 entry might age out
in one of the switches (this behavior is per forwarding engine: PFC/DFC)
In order to reduce chance of unicast flooding we need to keep L2 entries consistent access
both switches
mac-address-table synchronize feature will keep L2 tables synchronized
Enabled by default when WS-X6708 linecard is present in the chassis
Enabled by default in VSS as of 12.2(33)SXI4
Recommended in all cases
Make sure there is at least 2x aging intervals in synchonization interval
(i.e. for sync interval 160, L2 aging is >320 seconds, 480 recommended)
vss(config)# mac-address-table synchronize
% Current OOB activity time is [160] seconds
% Recommended aging time for all vlans is atleast three times the activity interval
and global aging time will be changed automatically if required

When troubleshooting unicast flooding, 2 items are very important

What module traffic arrives to (use commands to check ether-channel load-balancing)

Whether the module in question has the mac-address learned


(use sh mac-address address <mac> all)
BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

36

VSS Agenda
Initialization
Internal redundancy considerations
Spanning Tree

1st hop redundancy


Traffic forwarding

Multicast considerations

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

37

Ingress forwarding model


Distributed architecture. Ingress forwarding engine makes
forwarding, ingress *and* egress ACL/QOS decisions
IMPORTANT: If the linecard where packet is received has DFC
entries on that linecard need to be looked at when troubleshooting.
Otherwise look at active supervisors forwarding entries
i.e. sh mls cef <prefix> module <mod#>
or sh mls cef <prefix>

DFC
Ingress

Fabric

DFC

Egress

Traffic flow
BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

38

Traffic locality
Main concept for traffic forwarding is locality
Only local ports are used to send traffic out
except when there are no local ports, this is when traffic will cross
VSL/Peer-link

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

39

Traffic locality for ECMP routes


ECMP follows a similar behavior, local
links are preferred and all traffic is
forwarded out of a locally attached link
Hardware FIB inserts entries for ECMP
routes using locally attached links
If all local links fail the FIB is programmed
to forward across the VSL link

Si

Si

Te1/2/1

Te1/2/2

SW1

vss# sh ip route 10.121.0.0 255.255.128.0 longer-prefixes


D
10.121.0.0/17
[90/3328] via 10.122.0.33, 2d10h, TenGigabitEthernet2/2/1
[90/3328] via 10.122.0.27, 2d10h, TenGigabitEthernet1/2/1
[90/3328] via 10.122.0.22, 2d10h, TenGigabitEthernet2/2/2
[90/3328] via 10.122.0.20, 2d10h, TenGigabitEthernet1/2/2

Four ECMP
Entries

vss# sh mls cef 10.121.0.0 17 switch 1


Codes: decap - Decapsulation, + - Push Label
Index Prefix
Adjacency
102400 10.121.0.0/17
Te1/2/2
, 0012.da67.7e40 (Hash: 0001)
Te1/2/1
, 0018.b966.e988 (Hash: 0002)
BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Two FIB
Entries
40

VSS L2/L3 Forwarding (Data Plane)


VSS Data Plane Troubleshooting L2 MEC
VSS specific commands
augmented with switch id

Verify the load-balance algorithm used


vss# show etherchannel load-balance switch 2 module 2
EtherChannel Load-Balancing Configuration:
src-dst-ip vlan included
mpls label-ip
EtherChannel Load-Balancing Addresses Used Per-Protocol:
Non-IP: Source XOR Destination MAC address
IPv4: Source XOR Destination IP address
IPv6: Source XOR Destination IP address
MPLS: Label or IP

Important:: Only use parameters


consistent with the configured
load-balancing algorithm.
Command uses all the specified
arguments to calculate the hash.

Identify the physical path for flow from host 2 host 1 (out of Port-channel 2)
vss# show etherchannel load-balance hash-result interface Port-channel 2 switch 1
ip 9.0.1.2 vlan 705 8.0.1.1
Packet coming in on switch 1, needing to go
Computed RBH: 0x6
out on Po2 will select Gi1/6/2
Would select Gi1/6/2 of Po2
vss# show etherchannel load-balance hash-result interface Port-channel 2 switch 2
ip 9.0.1.2 vlan 705 8.0.1.1
Computed RBH: 0x6
Packet coming in on switch id 2, needing to
Would select Gi2/9/15 of Po2
go out on Po2 will select Gi2/9/15

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

41

VSS L2/L3 Forwarding (Data Plane)


VSS Data Plane Troubleshooting ECMP: Host 1 Host 2
Routing table shows two Equal Cost Paths to 9.0.0.0/8
vss# show ip route 9.0.0.0 | i via
Known via "eigrp 101", distance 90, metric 3072, type internal
Redistributing via eigrp 101
7.7.1.2, from 7.7.1.2, 1d00h ago, via TenGigabitEthernet2/2/7
* 7.6.1.2, from 7.6.1.2, 1d00h ago, via TenGigabitEthernet1/3/2

Looking at the HW table shows next hop directly attached to local switch
is preferred
vss# show mls cef lookup 9.0.1.0 switch 1 mod 3

Packet coming in on switch 1 module 3, for 9.0.0.0/8


prefers next hop attached to local switch id 1

Codes: decap - Decapsulation, + - Push Label


Index Prefix
Adjacency
108775 9.0.0.0/8
Te1/3/2
, 000f.35ed.7c00
vss# show mls cef lookup 9.0.1.0 switch 2 mod 2

Packet coming in on switch 2 module 2, for 9.0.0.0/8


Codes: decap - Decapsulation, + - Push Label
prefers next hop attached to local switch id 2
Index Prefix
Adjacency
108775 9.0.0.0/8
Te2/2/7
, 000f.35ed.7c00
DUT# show mls cef exact-route 8.0.1.1 0 9.0.1.2 0 switch 1 mod 3
Interface: Te1/3/2, Next Hop: 7.6.1.2, Vlan: 4064, Destination Mac: 000f.35ed.7c00
DUT# show mls cef exact-route 8.0.1.1 0 9.0.1.2 0 switch 2 mod 2
Interface: Te2/2/7, Next Hop: 7.7.1.2, Vlan: 4056, Destination Mac: 000f.35ed.7c00

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

42

1/1/33

Will the
packet cross
VSL link?

1/1/15

Po4
2/4/33

Po3

VSS

2/6/3

vss# sh mac-address-table address 0005.9a3b.6c80 vlan 226


...
vlan
mac address
type
learn
age
ports
------+----------------+--------+-----+----------+-------------------------Supervisor switch 1 Module 6
* 226 0005.9a3b.6c80
dynamic Yes
10
Po3
Supervisor switch 2 Module 6
What is the port
* 226 0005.9a3b.6c80
dynamic Yes
10
Po3

0005.9a3b.6c80

for this mac

address
What are physical ports of portchannel

vss# sh etherchannel 3 summary


...
Group Port-channel Protocol
Ports
------+-------------+-----------+----------------------------------------------3
Po3(SU)
PAgP
Gi1/1/15(D)
Gi2/6/3(P)

All ports on switch1 side are


down
If packet will arrive to switch1 to
be switched to po3, packet will
cross VSL

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

43

1/1/33

Will the
packet cross
VSL link?

1/1/15

Po4
2/4/33

Po3

VSS

2/6/3

0005.9a3b.6c80

vss# sh mac-address-table address 0005.9a3b.6c80 vlan 226 detail switch 1 module 6


MAC Table shown in details
========================================
PI_E RM RMA Type Alw-Lrn Trap Modified Notify Flood
Mac Address Age Pvlan Index XTag
----+---+---+----+-------+----+--------+------+------+--------------+----+------+------+---Supervisor switch 1 Module 6
Yes No
No DY
No
No
Yes
No
No
0005.9a3b.6c80 0x86 226
0xB40 0
vss# remote command switch test switch virtual ltl index 0xB40
...
Unmapped index: 0xB40
------+---------------------------------------SW view
Index | Ports
------+---------------------------------------0x0B40 Po3[Gi2/6/3],Po10[Te1/6/4]
...
------+---------------------------------------HW view
Index | Ports
------+---------------------------------------0x0B40 Te1/6/4,Gi2/6/3
...
vss# sh switch virtual link port-channel | i Po
Group Port-channel Protocol
Ports
10
Po10(RU)
Te1/6/4(P)
20
Po20(RU)
Te2/6/4(P)

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Find the index for given mac


address on ingress forwarding
engine
Find what ports on the local
switch (1) this index includes
Index should include VSL ports

How to verify if the packet from


switch 1 will cross VSL in order to
reach next-hop mac-address?
Cisco Public

44

VSS forwarding troubleshooting


summary
Unless the traffic is crossing VSL,
troubleshooting VSS packet forwarding is
exactly the same as troubleshooting
standalone cat6500
When traffic crosses VSL, verify
L3 entries on the ingress forwarding
engine (PFC or DFC)
L2 entries (for next hop destination mac)
on forwarding engine servicing the VSL on
the 2nd chassis (strictly speaking L2 entries
need to be checked on all DFCs along the
packet path)

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

45

Special case for flooding

MAC_B

3
1

MAC B is not known flood the frame

Internal frame header (carried over VSL) includes


destination index which is remapped by egress
switch to another index that does not include any
MEC that has operational ports on ingress switch

Frame is flooded to devices that are single


connected to egress switch (on the right)

MAC_A

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

46

EtherChannel Adaptive Hash


Each flow is assigned to 1 of 8 buckets
Each port in port-channel transmits traffic for some buckets (i.e. 4 for 2-port channel, 2
for 4-port etc)
When ports are joining/leaving channel the buckets are redistributed among operational
ports in deterministic fashion
Flows that remain on operational ports might be disturbed while ASICs are being
programmed
Member 1

Member 2

buckets that must move

New member
joins

Member 1

Member 2

Member 3

buckets moving between


operational ports

With adaptive hash option, only buckets that must move are reprogrammed
Member 1

Member 2

BRKCRS-1930

buckets that must move

New member
joins

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Member 1

Member 2

Member 3

47

EtherChannel Adaptive Hash


Adaptive hash is enabled by default on VSL link
If there is 1 link / chassis / MEC adaptive hash on MEC will not make any difference

If the network consists of several adjacent VSS systems, adaptive hash was enhanced
to avoid traffic polarization (as of 12.2(33)SXI)
Configured per port-channel

vss(config)#int port-channel200
vss(config-if)#port-channel port hash-distribution adaptive

With adaptive hash less flows should be impacted when ports join or leave portchannels
This is mostly evident when control-plane is busy (i.e. when many changes are
happening at the same time during failovers etc)

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

48

SPAN
When SPANed traffic is crossing VSL it is transmitted
over single link this might cause oversubscription of
VSL link if amount of SPANed traffic is significant
Use MEC as SPAN destination to prevent SPANed
traffic from crossing VSL
If one side of the MEC goes down SPANed traffic will
cross VSL
Provision enough bandwidth on VSL
Use port-channel min-links LACP feature on SPAN
destination MEC to bring down MEC if link is down on one
side
Use EEM script to shut down MEC or SPAN session when
one side of SPAN destination MEC goes down

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

49

VSS Agenda
Initialization
Internal redundancy considerations
Spanning Tree

1st hop redundancy


Traffic forwarding

Multicast considerations

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

50

Multicast forwarding
Layer 2 access has two multicast routers on the access subnet, RPF
checks and split roles between high and low IP address routers
VSS has a single multicast router which simplifies multicast topology
The multicast forwarder is selected based on which member of VSS
link receives multicast traffic
IGMP Querier
(Low IP address)

Si

Non-DR Has to
Drop All
Non-RPF Traffic

BRKCRS-1930

Single Logical Multicast


Designated Router and IGMP Querier

Si

Designated
Router
(High IP Address)

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

51

MEC behavior upon


VSS recovery after SSO switchover
3
2
1

Following SSO switchover left switch comes up


after reload

MEC link from left switch is brought up and joins


the bundle

Top switch starts sending a share of traffic to the left


switch, but the left switch might still be converging
(loading FIB tables, programming ASICs etc), so it
might not be fully ready to correctly forward the this
traffic
this might cause part of traffic to be lost for
some time after the switch recovery

To prevent this issue, configure port-channel load-defer feature on upstream switch


Upstream switch will delay sending traffic to newly bundled port for configured duration
vss(config)#port-channel load-defer 120
vss(config)#int po200
vss(config-if)#port-channel port load-defer
This will enable the load share deferral feature on this port-channel.
The port-channel should connect to a Virtual Switch (VSS).
Do you wish to proceed? [yes/no]: y
BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

52

Multicast fast-redirect
When a member of egress
Layer2 port-channel (MEC or
DEC) is unbundled/bundled On
VSS replicating multicast traffic in
egress mode it might take
noticeable time to reprogram
hardware to send traffic via
remaining links (local or across
VSL)

Sources
MEC

Si

Fast-redirect feature shortens


reprogramming time by
preprogramming most of the
needed changes

Si

MEC

Receivers

vss(config)#interface port-channel 40
vss(config-if)#mls ip multicast egress fast-redirect

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

53

VSS: summary
1 active redundant control plane
single config
single point of management

VSS domain

2 active data planes

Active

Standby

Active
Control Plane

Standby switch is essentially a


set of additional linecards
Control messages and Data
frames flow between active and
standby via VSL

Standby

VSL

Control Plane

Active

Active

Data Plane

Data Plane

Dual-Active
detection link

MEC

(can be seen as backplane


extension)

Special encapsulation on VSL


frames to carry additional
information
BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

54

V PC

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Both VPC and VSS


simplify logical Layer 2 topology
use Traffic Locality for efficient shortest path
forwarding

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

56

VPC Agenda
Initialization
Redundancy considerations
Spanning Tree

Traffic forwarding
1st hop redundancy

Multicast considerations

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

57

VPC Virtual Port channel


2 active control planes
2 configs
2 points of management
2 active data planes

VPC domain

Primary-Secondary notion for some


aspects of operation

Primary

Secondary

Active
Control Plane

Control messages and Data frames


flow between active and standby via
Peer-Link
Peer-Link is L2 trunk with plain 802.1q
encapsulation
Control messages are carried by CFS
over Peer Link

Active

Peer-Link

Control Plane

Active

Active

Data Plane

Data Plane

Peer
Keepalive link

VPC

Peer keepalive link to detect dualactive condition


We call VPC the MCEC between VPC
domain and access switches
BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

58

VPC initialization
VPC init is largely independent of NXOS boot each
switch boots on its own
VPC feature starts

Keep-alive linkup / peer communication is established


Peer-link linkup / CFS communication is established
Primary/Secondary role is resolved

Consistency is checked via CFS and applications synced


Peer-Link brought UP for data
VPCs brought UP

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

59

Cisco Fabric Services


CFS

CFS messaging

Uses
Configuration validation

MAC member port synchronization


vPC member port status
IGMP snooping synchronization
vPC status

For VPC CFS messages are encapsulated in Ethernet frames


delivered between peers on the peer-link
Nexus# sh cfs application
---------------------------------------------Application
Enabled
Scope
---------------------------------------------arp
Yes
Physical-eth
stp
Yes
Physical-eth
vpc
Yes
Physical-eth
igmp
Yes
Physical-eth
l2fm
Yes
Physical-eth
...
BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

60

VPC Configuration consistency


VPC has distributed management plane. Configurations of both
switches are managed separately
Some configurations inconsistencies could lead to undesirable
forwarding implications (packet duplication, blackholing etc). VPC
takes different action depending on the type of inconsistency
Type 1: VPC will not come up

Type 2: VPC will come up, but undesirable forwarding implications


might occur, syslog will be printed upon detected inconsistency
Nexus#
Nexus# sh
sh vpc
vpc consistency-parameters
consistency-parameters global
interface port-channel 1
Name
Name
Type
Type Local
Local Value
Value
Peer
Peer Value
Value
------------------------------- ------------------------------------------- --------------------------------------------STP
lag-id
Mode
11
Rapid-PVST
[(7f9b,
Rapid-PVST
[(7f9b,
STP
... Disabled
1
None
None
STP
modeMST Region Name
11
""
active
""
active
STP
STP MST
PortRegion
Type Revision
11
0Default
0Default
STP
STP MST
PortRegion
Guard Instance to 11
None
None
STP
VLANMST
Mapping
Simulate PVST
1
Default
Default
STP
Native
Loopguard
Vlan
11
Disabled
1
Disabled
1
STP
PortBridge
Mode Assurance
11
Enabled
trunk
Enabled
trunk
STP
MTU Port Type, Edge
11
Normal,
1500
Disabled,
Normal,
1500
Disabled,
BPDUFilter,
Duplex
Edge BPDUGuard 1
Disabled
full
Disabled
full
STP
Speed
MST Simulate PVST
11
Enabled
10 Gb/s
Enabled
10 Gb/s
Interface-vlan
Allowed VLANs admin up
2101
101
101
101
Interface-vlan routing
2
1,101
1,101
BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

61

Troubleshooting VPC initialization


Use sh vpc to check the feature status
vpc1# show feature | i vpc
vpc
1

enabled

vpc1# sh vpc
Legend:
(*) - local vPC is down, forwarding via vPC peer-link
vPC domain id

: 1

Peer status

: peer adjacency formed ok

vPC keep-alive status

: peer is alive

Configuration consistency status: success


Type-2 consistency reason

: Consistency Check Not Performed

vPC role

: primary

Number of vPCs configured

: 1

Peer Gateway

: Disabled

Dual-active excluded VLANs

: -

vPC Peer-link status


--------------------------------------------------------------------id

Port

Status Active vlans

--

----

------ --------------------------------------------------

Po100

up

CFS can communicate with the


peer
We hear peer-alives
Configs are compatible
Master/Slave for certain apps
Peer-Link will come up after CFS +
Peer-Keepalive + Config check are
ok

1,101

vPC status
---------------------------------------------------------------------id

Port

Status Consistency Reason

Active vlans

--

----

------ ----------- ------

------------

Po1

up

101

BRKCRS-1930

success

success

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

62

Troubleshooting VPC initialization


Stable, not expecting issues here
Set VPC logging level to 5 (default) to see more verbose messaging during the
VPC bringup
vpc1(config)# logging level vpc 5
08:18:47 %ETHPORT-5-SPEED: Interface port-channel100, operational speed changed to 10 Gbps Peer-Link comes up
08:18:51 %VPC-3-PEER_UNREACHABLE: Remote Switch Unreachable
08:18:51 %VPC-3-VPC_PEER_LINK_BRINGUP_FAILED: vPC peer-link bringup failed (vPC peer is not reachable over cfs)
08:18:51 %ETHPORT-3-IF_ERROR_VLANS_SUSPENDED: VLANs 1,100-101 on Interface port-channel100 are being suspended.
(Reason: vPC peer is not reachable over cfs)
08:18:51 %ETHPORT-5-IF_UP: Interface port-channel100 is up in mode trunk
08:18:58 %VPC-4-VPC_ROLE_CHANGE: In domain 1, VPC role status has changed to primary
08:18:58 %ETHPORT-3-IF_ERROR_VLANS_REMOVED: VLANs 1,100-101 on Interface port-channel100 are removed from
suspended state.
08:18:58 %VPC-5-VPC_DELAY_SVI_BUP_TIMER_START: vPC restore, delay interface-vlan bringup timer started
08:19:08 %VPC-5-VPC_DELAY_SVI_BUP_TIMER_EXPIRED: vPC restore, delay interface-vlan bringup timer expired,
reiniting interface-vlans
08:19:08 %VPC-5-VPC_RESTORE_TIMER_START: vPC restore timer started to reinit vPCs
08:19:38 %VPC-5-VPC_RESTORE_TIMER_EXPIRED: vPC restore timer expired, reiniting vPCs

In case process does not go beyond certain stage, one should look at
communication between the peers (CFS)
BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

63

VPC config remarks


Check config consistency using sh vpc consistency-parameters
Complete list of parameters which should be consistent is quite
extensive: physical port config, QOS, security, STP, routing
protocols etc
check config guide for specific NXOS version

Domain id must be unique for each domain reachable adjacent


at Layer 2
VPC domain 100

Domain id MUST be
different
(cant be 100 on both
Pair)

VPC

BRKCRS-1930

2011 Cisco and/or its affiliates. AllVPC


rights reserved.
domain

200

Cisco Public

64

VPC: CFS troubleshooting


Cisco Fabric Services Transport of control messages between VPC peers
Nexus# show cfs status
Distribution : Enabled
Distribution over IP : Disabled
IPv4 multicast address : 239.255.70.83
IPv6 multicast address : ff15::efff:4653
Distribution over Ethernet : Enabled
Nexus# show cfs peers
Physical Fabric
--------------------------------------------Switch WWN
IP Address
--------------------------------------------20:00:00:1b:54:c2:42:41 10.48.73.222 [Local]
Nexus
20:00:00:1b:54:c2:42:44 0.0.0.0
Total number of entries = 2

Nexus# show cfs internal ethernet-peer statistics


| i Trans|Rece
Number of Segments Transmitted
: 218
Number of Acks Transmitted
: 223
Maximum Segment Size Transmitted
: 0
Number of Transmission Timeouts
: 0
Number of segments in Transmit Queue
: 0
Number of segments in Re-Transmit Queue
: 0
Total Number of Segments Received
: 441
Number of Acks Received
: 217
Number of Duplicate Messages Received
: 0
Number of Unexpected Segments Received
: 0
Number of fragmented segments Received
: 2
Number of duplicate fragments Received
: 0
Number of unfragmented segments Received
: 210
Number of Received Segments Dropped
: 0
TX/RX
counters
should move
when
Number of Unreliable
segments
Transmitted
: 1
Number of UnreliableVPC
segments
Received
is active
or coming up : 1

Nexus# sh cfs internal notification log name vpc


Sun Nov 14 15:27:22 2010: Peer add 20:00:00:1b:54:c2:42:44
Sun Nov 14 19:05:25 2010: Peer gone 20:00:00:1b:54:c2:42:44
Sun Nov 14 19:08:03 2010: Peer add 20:00:00:1b:54:c2:42:44
BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Remote peer should be seen


Shows timestamps for when CFS
communication for VPC was
interrupted (peer-reload, peer-link
issues etc)
65

More information
sh tech
(collect for offline analysis, takes ~5 min when redirected to file)

sh tech vpc
(collect when there is no time for big sh tech)
debug vpc peer
(peer events, useful for indepth vpc troubleshooting)
debug vpc peer-link
(peer-link events, for indepth vpc bringup troubleshooting)
debug cfs event ethernet
(cfs event peer communication)

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

66

VPC Agenda
Initialization
Redundancy considerations
Spanning Tree

Traffic forwarding
1st hop redundancy

Multicast considerations

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

67

VPC redundancy model


Process restartability
Supervisor redundancy
VPC redundancy

Processes checkpoint their runtime state


Crashing process is restarted statefully by
system manager
VPC Domain

Switch 1
HA-policy will trigger
supervisor switchover
in response to
excessive process
crashing, software,
hardware or
diagnostic failure

BRKCRS-1930

Active

Switch 2
Process 1

Process 1

Active

Process 2

Process 2

Process X

Process X

Standby(SSO)

Standby(SSO)

Devices dual-attached to VPC domain are protected against


single switch failureCisco
(power,
hardware, maintenance etc)
Public
2011 Cisco and/or its affiliates. All rights reserved.

68

Peer-link failure handling


(similar to dual-active detection in VSS)
Primary is alive

VPC peer-link failure

I am primary

Primary is gone
Receiving
Keepalives*

2ndary

no

Become primary

yes

primary

Bring down all VPC ports

Done
VPC peers do not require reload following
peer-link failure or recovery
BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

69

Keepalive link
Peer Keepalives

Heartbeat between vPC peers to prevent dual-active scenario


Keepalives are sent every second by default on UDP port 3200

3 second hold timeout on peer-link loss (ignore keepalive to leave


time for convergence before taking action)
5 seconds keepalive timeout (starts after hold timeout after peer-link
down) if no keepalive received during this timeout dual active
detection seconday bring down VPC
Use dedicated link, though NXOS does not enforce this just IP
connectivity is verified
Mgmt interface can be used as keepalive link, but do not connect the
managemet interfaces together directly (only active supervisor
management interface is up)
vpc1# debug vpc
13:10:54.257099
your_context(0)
13:10:54.257126
13:10:55.257442
your_context(0)
13:10:55.257469
13:10:56.257324
your_context(0)
13:10:56.257351
BRKCRS-1930

peer-keepalive
vpc: received new OOB packet, version(0) flags(0) my_context(0)
my_epoch(604049) your_epoch(604104) my_ip(1.1.1.2)
vpc: your_ip(1.1.1.1) domainId(1)
vpc: received new OOB packet, version(0) flags(0) my_context(0)
my_epoch(604050) your_epoch(604105) my_ip(1.1.1.2)
vpc: your_ip(1.1.1.1) domainId(1)
vpc: received new OOB packet, version(0) flags(0) my_context(0)
my_epoch(604051) your_epoch(604106) my_ip(1.1.1.2)
vpc: your_ip(1.1.1.1) domainId(1)
2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

70

Troubleshooting VPC peer-keepalives


Nexus# show vpc peer-keepalive
vPC keep-alive status
--Send status
--Last send at
--Sent on interface
--Receive status
--Last receive at
--Received on interface
--Last update from peer

:
:
:
:
:
:
:
:

peer is alive
Success
2009.06.19 00:41:15 589 ms
Eth2/35
Success
2009.06.19 00:41:14 580 ms
Eth2/35
(1) seconds, (9) msec

vPC Keep-alive parameters


--Destination
: 7.7.7.77
--Keepalive interval
: 1000 msec
--Keepalive timeout
: 5 seconds
--Keepalive hold timeout
: 3 seconds
--Keepalive vrf
: v1
--Keepalive udp port
: 3200
--Keepalive tos
: 192
Nexus# show vpc statistics peer-keepalive

Peer-keepalive is only essential at


the time when peer-link goes down
At any other time peer-keepalive
failure will only trigger syslog
Peer-keepalives might be affected
by extreme control plane load
(check CPU utilization & COPP)
Number of keepalive state
transitions, closer to 0 - better

vPC keep-alive status


: peer is alive
vPC keep-alive statistics
---------------------------------------------------peer-keepalive tx count:
9773
peer-keepalive rx count:
8985
average interval for peer rx:
991
Count of peer state changes:
0
BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

71

VPC behavior at initialization


(default)
VPC needs to be able to talk to the
peer (over peer-link) before bringing
up VPC port-channels
Negotiate LACP/STP operating roles for
the chassis
Wait for per-port peer parameters and
handshake to bring up vPC ports

Performs peer parameters consistency


check on each VPC bringup

Only after VPC port-channels are


brought up.
What if after a full DC outage (both
Nexus down), only one switch is coming
up ?
Will not bring up VPCs if after a
datacenter outage, only one VPC peer
comes back up
BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

73

VPC Reload Restore


Allows to bring up VPCs after timeout
if peer is presumed dead

Default timeout 240 sec


Assumes primary role for STP and
LACP
Nexus(config)# vpc domain 1
Nexus(config-vpc-domain)# reload restore ?
<CR>
delay Duration to wait before assuming
peer dead and restoring vpcs
Nexus(config-vpc-domain)# reload restore delay ?
<240-3600> Time-out for restoring vPC links
(in seconds)

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

74

ARP synchronization
PC B

ARP
Ip B ???
Needs to be
Resolved ?

ARP
Ip B Mac B

PC A

BRKCRS-1930

When traffic pattern


changes (due to VPC links
going up/down, due to
failover etc) the peer that
handles the traffic might
need to resolve ARP before
being able to forward
packets
This might introduce
additional delay to traffic
recovery
ARP sync feature is
supported as of 4.2(6), and
allows VPC peers to
synchronize their ARP
tables over CFS

vpc(config)# vpc domain 1


vpc(config-vpc-domain)# ip arp synchronize

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

75

More information
sh log last <x>
(review sequence of events)
show file logflash://sup-standby/log/messages
(in case other supervisor was active when everything started)
sh process log
(which processes have crashed when)
sh redundancy status
(status of supervisor redundancy & last switchover data)
sh system reset-reason
(last reset/switchover reason per module)
sh logging onboard internal reset-reason
(reset reason from different components point of view useful
for complex cases)
sh tech /from main VDC/
(collects most of the above for offline analysis)
BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

76

VPC Agenda
Initialization
Redundancy considerations
Spanning Tree

Traffic forwarding
1st hop redundancy

Multicast considerations

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

77

Handling of Spanning Tree: VPC


1

Primary

Secondary

STP process

STP runs on both switches (2 active control


planes) but only primary switch controls VPCs.
(even if root is secondary , then Primary will send
bpdu with root info being secondary)
VPC port states changes are communicated to
secondary via CFS messages.
For non-VPC ports domain appears as 2 bridges

STP process

Peer-link is part of STP. BPDU handling is


modified such that Peer-link will never be blocked
(similar to MST implementation of IST)
Non-VPC ports are managed independently by
local STP process on each switch

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

78

STP troubleshooting
Left-Root# sh spanning vlan 35
VLAN0035
Spanning tree enabled protocol rstp
Root ID
Priority
24611
Address
001b.54c2.4241
This bridge is the root
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Bridge ID Priority
24611 (priority 24576 sys-id-ext 35)
Address
001b.54c2.4241
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Interface
Role Sts Cost
Prio.Nbr Type
---------------- ---- --- --------- -------- -------------------------------Po1
Desg FWD 1
128.4096 (vPC) P2p
Po100
Desg FWD 2
128.4195 (vPC peer-link) Network P2p

Peer link is running STP


Right# sh spanning-tree vl 35 detail | i "^ Port|BPDU"
Port 4096 (port-channel1, vPC) of VLAN0035 is designated forwarding
BPDU: sent 0, received 0
Port 4195 (port-channel100, vPC Peer-link) of VLAN0035 is root forwarding
BPDU: sent 3754, received 3755

On the other end of peer-link po1 is designated despite not sending or


receiving single BPDU
BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

79

STP troubleshooting
Looking at BPDUs
Left-Root# debug spanning-tree bpdu_tx tree 101
14:20:37.556707 stp: RSTP(101): transmitting RSTP BPDU
14:20:37.556750 stp: vb_vlan_shim_send_bpdu(1933): VDC
channel100 enc_type 1 len 42
14:20:37.556834 stp: RSTP(101): transmitting RSTP BPDU
14:20:37.556863 stp: vb_vlan_shim_send_bpdu(1933): VDC
enc_type 2 len 36

This output can be easily limited to


necessary Vlan/Interface, but it
doesnt dump the BPDU
Very chatty use debug logfile
<file> to redirect output to a file
on port-channel100
4 Vlan 101 port port-

on port-channel1
4 Vlan 101 port port-channel1

Left-Root# debug spanning-tree all


14:22:23.560147 stp: RSTP(1): transmitting RSTP BPDU on port-channel100
14:22:23.560169 stp: vb_vlan_shim_send_bpdu(1933): VDC 4 Vlan 1 port port-channel100
enc_type 2 len 36
14:22:23.560219 stp: BPDU TX: vb 1 vlan 1 port port-channel100 len 36 ->0180c2000000
CFG P:0000 V:02 T:02 F:78 R:80:01:00:1b:54:c2:42:43 00000002
B:80:01:00:1b:54:c2:42:44 9063 A:0000 M:0014 H:0002 F:000f

Looking at past events


Left-Root# sh spanning-tree internal event-history tree 0 interface port-channel 50
VDC02 MST0000 <port-channel50>
0) Transition at 497772 usecs after Tue Oct 20 17:42:01 2009
State: FWD Role: Root Age: 5 Inc: no [STP_PORT_STATE_CHANGE]
1) Transition at 661395 usecs after Tue Oct 20 17:42:01 2009
State: FWD Role: Root Age: 4 Inc: no [STP_PORT_ROLE_CHANGE]
2) Transition at 17741 usecs after Tue Oct 20 17:42:03 2009
State: BLK Role: Root Age: 5 Inc: no [STP_PORT_STATE_CHANGE]
...
BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

80

STP inconsistencies
When STP detects certain abnormal situations it may
mark ports as inconsistent and block them to prevent
forwarding loops
Root Root Guard feature detected inconsistency
(unwanted bridge tries to become root)
Loop Loop Guard feature detected inconsistency
(port becomes designated because no BPDUs are being
received)
Bridge Assurance (BA)
(no BPDUs are received from remote side)

VPC Peer-link
(any of above inconsistencies happened on VPC peer-link)
%STP-2-VPC_PEER_LINK_INCONSIST_BLOCK: vPC peer-link detected BPDU receive timeout
blocking port-channel11 VLAN0121.

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

81

Primary

BRKCRS-1930

inconsistency

Handling Peer-Link STP inconsistencies


on Primary switch

1
Secondary

2011 Cisco and/or its affiliates. All rights reserved.

When peer-link STP inconsistency is detected on


primary switch the link will be put in inconsistent
STP state (effectively blocking state)
BPDUs are not sent on peer-link when it is
inconsistent. This is to allow secondary switch to
detect inconsistency and react

Cisco Public

82

inconsistency

Primary

inconsistency

Handling Peer-Link STP inconsistencies


on Secondary switch

2Secondary 1

1
2

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

When peer-link STP inconsistency is detected on


secondary switch the peer link will be put in
inconsistent STP state (effectively blocking
state)
Respective vlans or MST instances are also
blocked on all VPCs

Cisco Public

83

Bridge assurance, Dispute & UDLD


BA is default enabled on Peer-Link (and recommended to remain
enable), not recommended for VPCs unless Peer-Switch feature is
used
Dispute is default enabled (for both RSTP and MST on VPC)
UDLD [normal mode] is recommended to take out bad links from
channels (otherwise LACP takes ~100sec vs ~20 with UDLD)
Recommendation
Preferred BA + UDLD + Dispute (on all interswitch links when using
Peer-switch) when all switches support this (nexus7000/5000 and
cat6500/VSS do support)
Without Peer-switch BA should be kept only on Peer-Link (no
BA/Loop guard on VPCs)
If preferred config is not supported use Loop Guard + UDLD
(supported by all Cisco switches)

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

85

STP behavior upon VPC primary failure

Primary

ROOT

OP-Primary
Secondary
Backup

ROOT
ROOT

Primary switch (STP root) fails

Secondary switch becomes operational primary


and STP root

STP root port doesnt change for access switch


nor any STP port states for VPCs, forwarding
continues
Depending on control plane load it might take few
seconds for Op-primary to start sending BPDUs.
This might cause STP reconvergence on
connected switches hence increasing hello time
or peer-switch feature might be considered in
large deployments

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

86

STP behavior upon VPC primary recovery

OP-Secondary

ROOT

2
SYNC

OP-Primary
Secondary
Backup

Left switch comes back up

Peer-Link comes back up

VPC role is resolved as Operational-secondary

Left switch has better STP priority becomes


STP root

STP root port of right switch will change and that


will trigger SYNC: all non-edge STP ports will be
temporarily blocked

ROOT
ROOT

Once sync is complete ports will resume


forwarding

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

87

VPC Peer-Switch feature


Both VPC switches originate BPDUs with preconfigured information. This
allows to keep the same BPDU when primary fails/recovers no extra
SYNC required avoid short interruption in forwarding described on
previous slide is avoided

Both left and right switches consider themselves root


Both left and right switches send BPDUs all the time no need to raise
hello time

Available 4.2(6) 5.x software


Primary

Secondary

ROOT

ROOT

spanning-tree vlan 1-1000 priority 8192


vpc domain 1
peer-switch

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

spanning-tree vlan 1-1000 priority 8192


vpc domain 1
peer-switch

Cisco Public

88

VPC Peer-Switch feature


Primary

Secondary

ROOT

ROOT

left# sh span vlan 101

VLAN0101
Spanning tree enabled protocol rstp
Root ID
Priority
8293
Address
0023.04ee.be01
This bridge is the root
...
Bridge ID

Priority
Address

8293
(priority 8192)
0023.04ee.be01

...
Interface
---------------Po1
Po100

Role
---Desg
Root

Sts
--FWD
FWD

left# sh vpc role | i mac


vPC system-mac
vPC local system-mac

Cost
--------1
2

Prio.Nbr
-------128.4096
128.4195

Type
--------------(vPC) P2p
(vPC peer-link)

: 00:23:04:ee:be:01
: 00:1b:54:c2:42:43

In Peer-Switch mode bridge-ID


comes from system-mac as
opposed to local mac in normal
mode

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

right# sh span vlan 101


VLAN0101
Spanning tree enabled protocol rstp
Root ID
Priority
8293
Address
0023.04ee.be01
This bridge is the root
...
Bridge ID

Priority
Address

...
Interface
---------------Po1
Po100
Cisco Public

Role
---Desg
Desg

8293
(priority 8192)
0023.04ee.be01
Sts
--FWD
FWD

Cost
--------1
2

Prio.Nbr
-------128.4096
128.4195

Type
--------------(vPC) P2p
(vPC peer-link)
89

More information
show spanning-tree internal event-history all
(allows to look back at past STP events, not included in sh tech)

sh tech stp
(from both sides of VPC)
sh tech
(from both sides of VPC, this will include in it sh tech stp, in case
VPC is is non-default VDC collect also sh tech from VDC 1)

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

90

VPC Agenda
Initialization
Redundancy considerations
Spanning Tree

Traffic forwarding
1st hop redundancy

Multicast considerations

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

91

Special case for forwarding


x

PC B

PC A ends a packet to PC B

MAC B is not known by left switch flood

MAC B is not known by right switch flood

B receives duplicate frames

MAC A will be learned on wrong port on the lower


access switch blackholing traffic to A

PC A

5
Frames received on Peer-Link may not be flooded
out of VPCs

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

92

Special case for forwarding:


VPC implementation
PC B

MAC B is not known by left switch flood

Frames received from Peer-Link are never sent


out of VPC (except those without operational
ports on ingress switch)
Egress port ASICs will drop the frame

Frame is still flooded to devices that are solely


connected to egress switch

2
PC A

This rule (called VPC check) stands for all traffic


(L2, L3, unicast, multicast, broadcast, flooded etc)
BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

93

Summary: VPC traffic forwarding

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

X
Cisco Public

94

VPC forwarding and L3 implication


vPC view

Layer 2 topology

Layer 3 topology

7k vPC
7k1

7k1

7k2

7k2

R
R

R could be any router,


L3 switch or VSS
building a port-channel

Port-channel looks like


a single L2 pipe.
Hashing will decide
which link to chose

Layer 3 will use ECMP


for northbound traffic

R can Decide to send to 7k1 at L3 (next-hop = 7k1 if Po) and


uses link to 7k2 at L2 level !!!
Path is R 7k2 7k1 DROPPED (per VPC check) as
incoming on peer-link if it must be routed to another VPC

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

95

Layer 3 and vPC Design update


Use L3 links to hook up routers and peer with a vPC domain
Dont use L2 port channel to attach routers to a vPC domain unless you statically route to
HSRP address
If both, routed and bridged traffic is required, use individual L3 links for routed traffic and L2
port-channel for bridged traffic
Use of peer-gateway does NOT change above recommendations
Switch

Switch
Po2

Po2

7k1

7k2
Po1

L3 ECMP
P

Routing Protocol Peer


Dynamic Peering Relationship

P
BRKCRS-1930

Router

Router
2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

96

Layer 3 and VPC consideration


Best : use Routed links from VPC pair to routers
Alternative : VPC in a pure L2 VDC and routing in a
separate VDC
Do not make L3 routing protocol peering between
VPC pair of switches on a VPC vlan.
May lead to routing frame towards Peer-link leading to drop
per VPC-Check
If peering between VPC devices is needed, must be done
outside of the peer link

Keep SVI interface administrative status in sync


(both up or both down) This is a type 2
consistency check

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

97

Special case for L2 learning

PC B

2
1

MAC A is learned on lower VPC

MAC A is learned on Peer-Link

Frame destined to A arriving to right switch will be


sent to Peer-Link

A
A

PC A

BRKCRS-1930

Traffic should prefer local links when available


(traffic locality rule)

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

99

L2 learning: VPC implementation

PC B

1
2
A

CFS message

MAC A is learned on lower VPC


MAC addresses are never learned from traffic on
Peer-Link

Left switch sends a CFS message to right switch


telling about MAC A learned on lower VPC. Right
switch updates MAC address table

Frame destined to A arriving to right switch will be


sent out of lower VPC

PC A

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

100

Po50
Vlan 50

Troubleshooting
Layer 2

Po22
Vlan 20

VPC

91.0.0.10
0013.1908.e246

20.1.2.3

nexus# sh mac address-table address 0013.1908.e246 vlan 50


VLAN
MAC Address
Type
age
Secure NTFY
Ports
---------+-----------------+--------+---------+------+----+-----------------* 50
0013.1908.e246
dynamic
0
F
F Po50
nexus# sh spanning-tree vlan 50 interface port-channel 50
Mst Instance
Role Sts Cost
Prio.Nbr Type
---------------- ---- --- --------- -------- -------------------------------MST0002
Desg FWD 200
128.4145 (vPC) P2p
nexus# sh hardware mac address-table 2 address 0013.1908.e246 vlan 50

MAC addresses should point


to expected ports in expected
vlans (path towards source)
nexus# sh system internal pixm info ltl 0x00a36 | i Eth.*,
0x0a36
Eth2/36,
The ports should be in STP
nexus# sh mac address-table address 0021.55e0.66c2 vlan 20
forwarding mode
VLAN
MAC Address
Type
age
Secure NTFY
Ports
Hardware MAC address
---------+-----------------+--------+---------+------+----+------------------
* 20
0021.55e0.66c2
dynamic
660
F
F Po22
table should be consistent
nexus# sh spanning-tree vlan 20 interface port-channel 22
with software table
Mst Instance
Role Sts Cost
Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Finding port# for given index
MST0000
Desg FWD 200
128.4117 (vPC) Network P2p
Valid| PI | BD
|
MAC
| Index | Stat| SW | Modi| Age | Tmr |
|
|
|
|
| ic |
| fied| Byte| Sel |
-----+----+-------+---------------+--------+-----+----+-----+-----+-----+
1
1
161
0013.1908.e246 0x00a36
0
3
0
141
1

nexus# sh hardware mac address-table 1 address 0021.55e0.66c2 vlan 20


Valid| PI | BD
|
MAC
| Index | Stat| SW | Modi| Age | Tmr |
|
|
|
|
| ic |
| fied| Byte| Sel |
-----+----+-------+---------------+--------+-----+----+-----+-----+-----+
1
1
18
0021.55e0.66c2 0x00a32
0
2
0
103
1
nexus# sh system internal pixm info ltl 0x00a32 | i Eth.*,
0x0a32
Eth1/13, Eth1/14,
BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

102

Po50
Vlan 50

Troubleshooting
Layer 3

Po22
Vlan 20

VPC

91.0.0.10
0013.1908.e246

20.1.2.3

nexus# sh routing ip 20.1.2.3


...
20.1.2.3/32, ubest/mbest: 1/0
*via 20.1.1.240, Vlan20, [1/0], 03:48:59, static
nexus# sh ip arp 20.1.1.240
Address
Age
MAC Address
20.1.1.240
00:02:17 0021.55e0.66c2

Interface
Vlan20

nexus# sh forwarding ip route 20.1.2.3 module 2


...
------------------+------------------+--------------------Prefix
| Next-hop
| Interface
------------------+------------------+--------------------20.1.2.3/32
20.1.1.240
Vlan20
nexus# sh forwarding adjacency 20.1.1.240 module 2
IPv4 adjacency information
next-hop
rewrite info
interface
-------------- --------------- ------------20.1.1.240
0021.55e0.66c2 Vlan20
nexus# sh int vl 20 | i address
Hardware is EtherSVI, address is

Is there route to
destination
Is the next hop resolved
Looking at module 2
because this is where
packets in question
should be received
Is adjacency consistent
with ARP
Router MAC must have
Gateway flag in order for
packet to be L3 switched

0023.ac66.1a42

nexus# sh mac address-table address 0023.ac66.1a42 vlan 20


VLAN
MAC Address
Type
age
Secure NTFY
Ports
---------+-----------------+--------+---------+------+----+-----------------G 20
0023.ac66.1a42
static
F
F sup-eth1(R)
BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

105

Where given packet will be load-balanced


For equal-cost routes
nexus# sh routing hash 91.0.0.10 20.1.2.3
Load-share parameters used for software forwarding:
load-share mode: address source-destination port source-destination
Universal-id seed: 0xcdb5769f
Hash for VRF "default"
Hashing to path *20.1.1.3 (hash: 0x2a), for route:
20.1.2.3/32, ubest/mbest: 2/0
*via 20.1.1.3, Vlan20, [1/0], 00:01:37, static
*via 20.1.1.240, Vlan20, [1/0], 16:32:42, static

Load-balancing is configurable
under ip load-sharing address in
default VDC and affects all VDCs

For port-channels
nexus# sh port-channel load-balance forwarding-path interface port-channel 22 dst-ip
20.1.2.3 src-ip 91.0.0.10 vlan 20 module 2
Load-balancing is configurable
Missing params will be substituted by 0's.
under port-channel load-balance
Module 2: Load-balance Algorithm: source-dest-ip-vlan
RBH: 0 Outgoing port id: Ethernet1/14
in default VDC and affects all VDCs

Use sh port-channel rbh-distribution to see which link sends traffic for


which of 8 available load-balancing buckets

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

106

Hardware path packet drops

#1 command to look for hardware


packet drops
Not every drop listed here is actual
data packet drop
|------------------------------------------------------------------------|
| Device:R2D2
Role:MAC
| times to see if any
Run several
|------------------------------------------------------------------------|
Instance:7
counters increase at rate similar to
ID
Name
Value
Ports
traffic loss
------------28688 aric_no_port_select_error
0000000000000002
To clear1,3,5,7
counters,I2use
...
|------------------------------------------------------------------------|
clear statistics
module-all device all
| Device:Ashburton
Role:MAC
Mod: 1
|
nexus# sh hardware internal errors all
---------------------------------------Hardware errors as reported in module 1
----------------------------------------

|------------------------------------------------------------------------|
Instance:0
3629 Egress Port-1 VSL Dropped Packet Count
0000000853635833
5 3630 Egress Port-2 VSL Dropped Packet Count
0000000857893046
3 ...
|------------------------------------------------------------------------|
| Device:Naxos
Role:MAC SECURITY
|
|------------------------------------------------------------------------|
Instance:0
ID
Name
Value
Ports
------------106 m1_fab_p25_txq_tc0_drop_count
00000000000012af
2 ...
|------------------------------------------------------------------------|
| Device:Metropolis
Role:REWR
|
|------------------------------------------------------------------------|
Instance:1
ID
Name
Value
Ports
------------70
Krypton input controller zero portsel cnt
0000000000000038
18,20,22,24,26,28,30,32
|------------------------------------------------------------------------|
| Device:Lamira
Role:L3
|
|------------------------------------------------------------------------|
Instance:0
ID
Name
Value
Ports
------------93
CL2 Invalid Pkt count
00000008759cb9cb
1-32 I1
...
BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

109

VPC Agenda
Initialization
Redundancy considerations
Spanning Tree

Traffic forwarding
1st hop redundancy

Multicast considerations

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

112

1st hop redundancy with VPC


MAC_B vMAC
IP B IP A

PC B

Each of VPC peers will L3 forward packets


destined to its respective Router MAC address
HSRP/VRRP/GLBP used for 1st hop redundancy
Router MAC1
0001.0002.0003
Virtual MAC
0000.0c07.ac00

HSRP

Router MAC2
0005.0006.0007
Virtual MAC
0000.0c07.ac00

Both switches will L3 switch packets to vMAC


address as long as one of them is HSRP active or
HSRP standby.
If both switches are HSRP listening, they will not
L3 switch packets to vMAC

PC A

MAC_A vMAC
IP A IP B
BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

113

First hop redundancy troubleshooting


standby
Interface Vlan1
ip address 1.1.1.252/24
hsrp 1
ip 1.1.1.254

Left# sh hsrp brief


Interface
Grp Prio P State
Vlan1
1
100
Standby

active
Interface Vlan1
ip address 1.1.1.253/24
hsrp 1
ip 1.1.1.254

HSRP

Active addr
1.1.1.253

Standby addr Group addr


local
1.1.1.254

Left# sh mac address-table address 0000.0c07.ac01


VLAN
MAC Address
Type
age Secure NTFY
Ports
---------+-----------------+--------+-----+------+------+----------G 1
0000.0c07.ac01
static
False False sup-eth1(R)
Right# sh hsrp brief
Interface
Grp Prio P State
Vlan1
1
100
Active

Active addr
local

Standby addr Group addr


1.1.1.252
1.1.1.254

Right# sh mac address-table address 0000.0c07.ac01


VLAN
MAC Address
Type
age Secure NTFY
Ports
---------+-----------------+--------+-----+------+------+----------G 1
0000.0c07.ac01
static
False False sup-eth1(R)
BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Both peers will L3


forward packets destined
to vMac address as long as
either peer in VPC domain
is in active or standby
state for corresponding
group
Virtual mac address (vMac)
will be installed in both
peers
G (gateway) flag must be
present on any MAC
address for which the
nexus is expected to L3
forward packets
Only active will respond
to ARP for VIP

114

1st hop issue with some devices


MAC_B Router MAC1
IP B IP A

Server B
Router MAC1 MAC_B

MAC_B Router MAC1

IP A IP B

IP B IP A

4
2

Router MAC1
0001.0002.0003
Virtual MAC
0000.0c07.ac00

Router MAC2
0005.0006.0007
Virtual MAC
0000.0c07.ac00

PC A sends a packet to Server B

Left VPC switch will receive the packet and


forward it to Server B, note Source MAC of
outgoing packet will be that of Router1

Server B responding to PC A will populate


destination MAC from source MAC of received
frame (this is wrong, it should use ARP)

If frame from BA will be load-balanced to right


switch the MAC address of Router1 will point to
Peer-Link and this is where the frame will be sent

Left switch will receive the frame from Peer-Link


and drop it

5
PC A

MAC_A vMAC
IP A IP B
BRKCRS-1930

1
2011 Cisco and/or its affiliates. All rights reserved.

Why? Frames received from Peer-Link are never


sent out of VPC except those without operational
ports on ingress switch
(egress port ASICs will drop the frame)
Cisco Public

115

Peer-Gateway : the workaround


MAC_B Router MAC1
IP B IP A

With peer-gateway both peers will install router


MACs of each other in L2 table which will allow
them to L3 forward traffic destined to either
Router MAC

Server B
MAC_B Router MAC1
IP B IP A

2
Router MAC1
0001.0002.0003
Virtual MAC
Router
MAC2
0000.0c07.ac00
0005.0006.0007
Virtual MAC
0000.0c07.ac00

Router MAC2
0005.0006.0007
Virtual MAC
Router
MAC1
0000.0c07.ac00
0001.0002.0003
Virtual MAC
0000.0c07.ac00

Server B responding to PC A will populate


destination MAC from source MAC of received
frame (this is wrong, it should use ARP)

Right switch will forward packet towards


destination

PC A

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

116

Peer-Gateway : the implications

MAC_B Router MAC1


IP TOP IP LEFT, TTL 1
Router MAC1
0001.0002.0003
Router MAC2
0005.0006.0007
Virtual MAC
0000.0c07.ac00

Router MAC2
0005.0006.0007
Router MAC1
0001.0002.0003
Virtual MAC
0000.0c07.ac00

Top device attempts to establish OSPF adjacency


with the left switch

If peer-gateway is enabled in VPC domain and


OSPF unicast packet will be load-balanced to the
right switch, this packet will be dropped
Why? Right switch will try to L3-switch the
unicast packet (because RouterMAC1 is marked
as gateway MAC and destination IP is not local)
As packet has TTL==1 it will be dropped
Same applies to any other protocol that uses
unicast packets with TTL==1 entering right switch
but destined to left switch (or vise versa)

Routing protocol peering with devices attached to


VPC domain via SVI interface is not supported
Routed interface should be used in this case
BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

117

More information
sh mac address-table <address>
(L2 entry for given MAC )

sh hardware mac address-table <mod> address <address>


(hardware L2 entry for given MAC should be consistent with
above)
sh system internal l2fm l2dbg macdb address <addr>
(history of changes for given mac address)
sh tech hsrp
(from both sides of VPC)

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

118

VPC Agenda
Initialization
Redundancy considerations
Spanning Tree

Traffic forwarding
1st hop redundancy

Multicast considerations

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

119

IP Multicast with VPC


Receiver sends IGMP report (join)
RP

Access switch sends join to right VPC peer


Right VPC peer creates (*,G) adds VPC to OIF (as
proxy-DR)

Source S1

IGMP is encapsulated in CFS and sent to left peer


Left peer (DR) creates (*,G) adding VPC to OIF

(*,G)VPC

(*,G)VPC

Primary

2ndary

CFS:IGMP

(S1,G)VPC
DR

(S1,G)null
Proxy-DR

DR (left peer) sends PIM Join to RP


Once (S1,G) traffic starts arriving, VPC peers will
resolve which one will be forwarder for that (S,G):
peer with best metric to source or primary in a tie
(this mechanism is specific to PIM in VPC mode,
normally PIM would use assert)
Only forwarder will have OIFs populated in (S,G)
the non-forwarder wont have VPC SVIs in OIF list

Receiver

Forwarder will send a copy of frame to the peerlink for receivers single-connected to other peer

IGMP join

Goal is to allow peer that 1st sees source traffic to forward it to receivers behind VPC
BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

120

IP Multicast with VPC


Prebuilt-SPT
RP

In case of DR failure proxy-DR becomes DR and


posts OIF-list from (*,G) to (S,G), but it will also
need to pull traffic from RP/source which delays
recovery

Source S1

(*,G)VPC

(*,G)VPC

Primary

(S1,G)VPC

2ndary

With ip pim pre-build-spt proxy-DR will also send


a PIM Join to source/RP to draw the traffic

(S1,G)null

Traffic pulled by proxy-DR will be dropped until it


becomes DR provision uplink accordingly (if
pre-build-spt is used)

DR

Receiver

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

121

IP Multicast with VPC


source behind VPC
When Source is behind VPC both DR and ProxyDR will add OIFs for the group to (S,G)

RP

This is because either peer can receive source


traffic and need to be able to send it to receivers
behind VPCs without crossing peer-link (to keep
traffic locality and to avoid dropping the traffic by
VPC check)
(*,G)VPC2

(*,G)VPC2

Primary

Going to Left switch from Source

2ndary

(S1,G)VPC2

(S1,G)VPC2

Or going to Right switch from Source

Source S1

BRKCRS-1930

VPC1

VPC2

Receiver

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

122

Which of VPC peers will be forwarder


Peers do metrics exchange over CFS for each new source
Peer that has better metric to source or primary will be forwarder
VPC1# sh ip pim internal vpc rpf
Source: 10.0.1.1
Pref/Metric: 110/21
Source role: primary
Forwarding state: Win (forwarding)

For sources behind VPC both peers will forward as they have no control on which
one will get the traffic
VPC1# sh ip pim internal vpc rpf
Source: 1.1.1.1
Pref/Metric: 0/0
Source role: primary
Forwarding state: Win-force (forwarding)

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

123

VPC mcast: following packet flow


Nexus# show ip mroute 239.1.2.3
(*, 239.1.2.3/32), uptime: 06:46:05, igmp pim ip static
Incoming interface: Vlan36, RPF nbr: 36.0.0.3
Outgoing interface list: (count: 2)
Ethernet2/43, uptime: 03:01:36, static
Vlan37, uptime: 06:46:05, igmp

control plane state for this group


where information came from
stable?
RPF interface

where are receivers on this vlan?

Is traffic being switched for this group?


counters updated once ~1 minute
packets forwarded in software
average packet size

(33.0.0.33/32, 239.1.2.3/32), uptime: 06:46:05, ip pim mrib


Incoming interface: Vlan36, RPF nbr: 36.0.0.3
Outgoing interface list: (count: 2)
Ethernet2/43, uptime: 03:01:36, mrib
Vlan37, uptime: 06:46:04, mrib
Nexus# show ip igmp snooping groups vlan 37
Type: S - Static, D - Dynamic, R - Router port
Vlan
37
37

Group Address
*/*
239.1.2.3

Ver
v2

Type
R
D

Port list
Vlan37
Eth2/8

Are packets being switched by this entry?


Nexus# show ip mroute 239.1.2.3 summary software-forwarded
Total
Total
Total
Total
Group

number
number
number
number
count:

of
of
of
of
1,

routes: 3
(*,G) routes: 1
(S,G) routes: 1
(*,G-prefix) routes: 1
rough average sources per group: 1.0

Group: 239.1.2.3/32, Source count: 1


Source
packets
bytes
(*,G)
0
0
sw-pkts: 0
33.0.0.33
5046908
252345396
sw-pkts: 1
BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

aps
0

pps
0

bit-rate
0.000
bps

49

200

80.053

Cisco Public

oifs
2

kbps 2
125

Following the flow: forwarding information


Nexus# show forwarding multicast route group 239.1.2.3
slot 1
=======
(*, 239.1.2.3/32), RPF Interface: Vlan36, flags: G
Received Packets: 0 Bytes: 0
Number of Outgoing Interfaces: 2
Outgoing Interface List Index: 4
Vlan37 Outgoing Packets:0 Bytes:0
Ethernet2/43 Outgoing Packets:N/A Bytes:N/A

(33.0.0.33/32, 239.1.2.3/32), RPF Interface: Vlan36, flags:


Received Packets: 5723369 Bytes: 366295616
Number of Outgoing Interfaces: 2
This is platform independent forwarding
Outgoing Interface List Index: 4
Vlan37 Outgoing Packets:0 Bytes:0
information
Ethernet2/43 Outgoing Packets:N/A Bytes:N/A

Ingress linecard entry


Egress linecard entry
Counters are updated once per ~1minute
Counters between ingress/egress do not have to
Received Packets: 0 Bytes: 0
match, as information is collected not at the same
Number of Outgoing Interfaces: 2
Outgoing Interface List Index: 4
exact time, receiver might join after the entry was
Vlan37 Outgoing Packets:5725816 Bytes:366452224
created etc
Ethernet2/43 Outgoing Packets:3032294 Bytes:194066816

(*, 239.1.2.3/32), RPF Interface: Vlan36, flags: G

slot 2
=======

(33.0.0.33/32, 239.1.2.3/32), RPF Interface: Vlan36, flags:


Received Packets: 0 Bytes: 0
Number of Outgoing Interfaces: 2
Outgoing Interface List Index: 4
Vlan37 Outgoing Packets:5725816 Bytes:366452224
Ethernet2/43 Outgoing Packets:3032294 Bytes:194066816

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

126

When traffic arrives via VPC


How to find which slot receives the
S,G flow when ingress interface is
port-channel scattered across
several modules?

VPC domain 100

VPC

show forwarding multicast route


group <g> source <s>

Nexus# show forwarding multicast route group 239.1.1.1 source 1.0.1.2 | i Received|slot
slot 1
Received Packets: 0 Bytes: 0
slot 2
Received Packets: 727203 Bytes: 487290999

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

127

Following the flow: hardware entries


Nexus# show system internal forwarding ipv4 multicast route group 239.1.2.3 source 33.0.0.33 detail
slot

(33.0.0.33/32, 239.1.2.3/32), Flags: *S


Lamira: 1, HWIndex: 0x2200, VPN: 1
RPF Interface: Vlan36, LIF: 0x45, PD oiflist index: 0x2
ML3 Adj Idx: 0xa016, MD: 0x2003, MET0: 0x2004, MET1: 0x2004, MTU Idx: 0x1
Metro Instance: 0
Dev: 1 Index: 0xa019
Type: MDT
elif: 0xc0002
dest idx: 0x7fe7
recirc-dti: 0xe20000
Ingress forwarding engine (FE)
Metro Instance: 1
replicates packets to receivers on that
Dev: 1 Index: 0xa019
Type: MDT
elif: 0xc0002
dest idx: 0x7fe7
recirc-dti: 0xe20000
linecard and creates distribution copy
Metro Instance: 2
of the packet for other linecards
Dev: 1 Index: 0xa019
Type: MDT
elif: 0xc0002
dest idx: 0x7fe7
recirc-dti: 0xe20000
MET pointers (MD + MET0)
Metro Instance: 3
RPF interface read from entry
Dev: 1 Index: 0xa019
Type: MDT
elif: 0xc0002
dest idx: 0x7fe7
recirc-dti: 0xe20000
TCAM Entry

Decoded MET chain (on ingress there


is only MD copy created)
(33.0.0.33/32, 239.1.2.3/32), Flags: *S
Egress linecard will receive distribution
Lamira: 1, HWIndex: 0x2200, VPN: 1
copy and replicate it to receivers (using
RPF Interface: Vlan36, LIF: 0x45, PD oiflist index: 0x2
ML3 Adj Idx: 0xa026, MD: 0x2003, MET0: 0x2004, MET1: 0x2004,
MTUpointer)
Idx: connected
0x1
MET1
to the card
Metro Instance: 0
MET1 on egress linecard points to
Dev: 1 Index: 0xa029
Type: MDT
elif: 0xc0002
dest idx: 0x7fe7
recirc-dti: 0xe20000
receivers on vlan37 and e2/43
slot

Dev: 1 Index: 0x6046


Metro Instance: 1
Dev: 1 Index: 0xa029

Dev: 1 Index: 0xa028


BRKCRS-1930

Type: OIF
dest idx: 0x0

elif: 0x80046
Vlan37
smac: 001b.54c2.4241

Type: MDT
elif: 0xc0002
dest idx: 0x7fe7
recirc-dti: 0xe20000
Type: OIF
elif: 0x84029
Ethernet2/43
dest idx: 0x44c
smac: 001b.54c2.4241

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

128

Are there drops in forwarding path?


Start looking from Ingress module
Nexus# show hardware internal errors module 1
---------------------------------------Hardware errors as reported in module 1
---------------------------------------...
|------------------------------------------------------------------------|
| Device:Lamira
Role:L3
Mod: 1
|
| Last cleared @ Thu Apr 8 12:57:37 2010
| Device Statistics Category :: ERROR
|------------------------------------------------------------------------|
Instance:0
ID
Name
Value
Ports
------------259 L3 Fib Miss Pkt ctr
0000000000000007
1-32 I1
262 L3 Non-Rpf Drop Pkt ctr
0000000000125617
1-32 I1
319 NF2 V4 IPMAC Lkup Error
0000000000272277
1-32 I1
455 Exception cause: DROP (Unicast)
0000000000025510
1-32 I1
465 Exception cause: DROP (Multicast)
0000000000226148
1-32 I1

Always take several snapshots and look for drops that grow coherently with
[suspected] multicast traffic drops
There are always some drops shown by above command this doesnt always
mean the actual network packets are dropped. Some of these are diag packets,
some are packets that are dropped on blocked ports, extra floods etc
BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

129

Wrapping UP

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

VPC compared to VSS


VPC

VSS

Control Plane
SSO
HSRP/VRRP

Distributed
InTRAchassis (w/2 sups)
2 routers, each forwards
traffic

Traffic locality
Failover time
Configuration
synchronization

Yes
Subsecond
Separate configs, key
parameters checked via
CFS
via the Peer-Keepalive link via L2 hellos and
PAgP+

Dual active
detection

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Redundant Centralized
InTERchassis
Inherent 1st hop
redundnancy, no need
for HSRP
Yes
Subsecond
Using IOS redundancy
framework

132

VPC/VSS: summary
Remember about the implications of 2 control planes
and 2 data planes active at the same time

Pay special attention to configuration and operational


consistency, not only to what is enforced, but also L3
interfaces including their operational state, FHRP
config, ACL config, queueing config
Troubleshoot like a standalone switch 1st, then dive into
VPC/VSS specifics: main one being traffic locality

Both VPC and VSS


simplify logical Layer 2 topology
use Traffic Locality for efficient shortest path
forwarding
BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

133

BRKCRS-1930
Recommended Reading

Also browse on-site Cisco Store for suitable reading

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

134

Please complete your Session Survey

We value your feedback - don't forget to complete your online


session evaluations after each session. Complete 4 session
evaluations & the Overall Conference Evaluation (available from
Thursday) to receive your Cisco Networkers 20th Anniversary t-shirt.

All surveys can be found on our onsite portal and mobile website:
www.ciscoliveeurope.com/connect/mobi/login.ww

You can also access our mobile site and


complete your evaluation from your mobile
phone:
1. Scan the Access Code
(See http://tinyurl.com/qrmelist for software,
alternatively type in the access URL)

2. Login
3. Complete and Submit the evaluation

BRKCRS-1930

2011 Cisco and/or its affiliates. All rights reserved.

Cisco Public

135