You are on page 1of 199

Nexus9000(Standalone)

Architecture And
Troubleshooting
Shridhar V. Dhodapkar Technical Leader (Services)
CCIE 6367 (Routing & Switching)
BRKDCT-3101

Session Abstract
This session presents briefly the architecture of the latest generation
of Nexus 9000 Series Modular switches. Topics include supervisors,
fabrics, I/O modules, forwarding engines, and physical design elements, as
well as the Top of the Rack Nexus9300 Switches.
The session will also cover how to monitor the health of the system.
We will walk you through in depth troubleshooting Tools and Techniques.

Session Goal

To provide an overall understanding of the Nexus 9000 switching


architecture, supervisor, fabric, and I/O module design, packet flows, and
key forwarding engine functions

This session will introduce System Telemetry, Troubleshooting tool Kits


and troubleshooting case scenarios

This session will not examine NX-OS software architecture or other Nexus
platform architectures

Related Sessions
BRKARC-2222 - Cisco Nexus 9000 architecture
BRKARC-3471 - Cisco NX-OS Software Architecture

BRKDCT-3144 - Advanced - Troubleshooting Cisco Nexus 7000 Series Switches

Agenda

Introduction

Architecture

System Health check Telemetry

Troubleshooting Toolkit

Nexus 9000 Troubleshooting

Common Link Layer Issues-L1


Fabric Connectivity and
In band
L2/L3 Packet Forwarding
vPC

Nexus9000 Specific Limitation and Goodies

Introduction

Introduction-What is Nexus9000 Family ?

Nexus 9500 Series Switches


Nexus9504/Nexus9508/Nexus9516

Nexus 9300 Series Switches


N9K-C9332PQ N9K-C9372PX N9K-C9372TX N9K-C9396

Architecture

9500 Field Upgradeable Units (FRU)


9500 has the following modular components which can upgraded or
replaced in the field
Nexus 9508 Front View
Nexus 9508 Rear View

Supervisor
Fabric Module
Line Card
System Controller
Fan Tray
Power Supply

The Supervisor, System controller ,Fabric Module and LC have OBFL


(On-Board Failure Logging) for failure analysis

Nexus 9500 Platform FRU


Supervisor Module-What it is Role

Redundant Half-width supervisor engine


Common for 4-, 8-, and 16- slot chassis
External Clock Input (PTP)
Responsible for control-plane functions

System Controller-What it is Role

Offload supervisor from internal device management tasks


Central Point of Chassis Control
EOBC Switch (Ethernet Out of Band Channel)
EPC Switch (Ethernet Protocol Channel)
Power Supplies via SMB (System Management Bus)
Fan Trays

Nexus 9500 Platform Line Card


Line Cards

FM6

I/O module with Merchant and


Merchant+ ASIC

FM4

FM5

HG
MUX1

HG
MUX2

HG
MUX4

Have Various Forwarding Tables


L2 Mac Table And L3 Host Table
ACL and Buffers for Queuing

45

01
23

FM3

FM2
HG
MUX3

HG
MUX5

89
10
11

7
MN6Port

01
23

Northstar 1

ASIC Name

MF Port

0
2

NFE=Network Forwarding Engine-Trident 2(T2)


ALE=Application Leaf Engine-North Star(NS)

7
5

-Donner

3
5

68

91
1

2
62
4

1Warpcor
0
2
e 9
T2

45
7
MN6 Port

HG
MUX6

89
10
11

Northstar 2
MF Port

0
2

HG

7
5

N9K-X9564PQ

91
1
2
62
4

T2

Note: Internal ports are called as Hi-Gig/HG ports


F
F F F F F F F F F
P
PPPPPPPPP
1
1 2 3 4 5 6 7 8 9
0

FM1

40G
QSFP
F
P
1
1

F
P
1
2

10G SFP+ Ports


F
P
1
3

F
P
1
4

F
P
1
5

F
P
1
6

F
P
1
7

F
P
1
8

F
P
1
9

F
P
2
0

F
P
2
1

F
P
2
2

F
P
2
3

F
P
2
4

F
P
2
5

F
P
2
6

F
P
2
7

F
P
2
8

F
P
2
9

F
P
3
0

F
P
3
1

F
P
3
2

F
P
3
3

F
P
3
4

F
P
3
5

F
P
3
6

F
P
3
7

F
P
3
8

F
P
3
9

F
P
4
0

F
P
4
1

F
P
4
2

F
P
4
3

F
P
4
4

F
P
4
5

F
P
4
6

F
P
4
7

F
P
4
8

FP FP FP FP
49 50 51 52

Nexus 9500 Fabric Module


Fabrics Modules

Interconnect Line Card slots


Installed at the rear of the chassis
Leverages Broadcom Trident II ASICs
Max 1.92 Tbps per line card slot (6 Fabric Cards)
960 Gbps per line card slot (3 Fabric Cards)
All Fabric Cards are active and carry traffic
Fan Tray requires Fabric Card to be present in even slot
Trident II
ASIC-NFE

32 x 40G
Hi-Gig2

Trident II
ASIC-NFE

32 x 40G
Hi-Gig2

Nexus 9500 Fabric Module


Data Plane Scaling for 8-Slot Chassis

An 8-Slot chassis fabric module can provide up to 320Gbps to each Line Card slot
With 6 fabric modules, each Line Card slot can have up to 1.92Tbps duplex
forwarding bandwidth
T2

Fabric 1
T2

320 Gbps
(8x 40Gbps)

Fabric 2
T2
T2

Fabric 3
T2
T2

Fabric 4
T2
T2

Fabric 5
T2
T2

Fabric 6
T2
T2

320 Gbps
(8x 40Gbps)

320 Gbps
(8x 40Gbps)

320 Gbps
(8x 40Gbps)

320 Gbps
(8x 40Gbps)

320 Gbps
(8x 40Gbps)

320 Gbps
640 Gbps
960 Gbps

Line Card Slot


1.28 Tbps
1.60 Tbps
1.92 Tbps

Distributed Data Plane of Nexus 9500 Series Switches


Fabric 1

Fabric 2

Fabric 3

Fabric 4

Fabric 5

Fabric 6

Nx NFE

Nx NFE

Nx NFE

Nx NFE

Nx NFE

Nx NFE

N = 1 for N9504
N = 2 for N9508
N = 4 for N9516
2 x 42 Gbps

2 x 42 Gbps
ALE

ALE

12 x 42 Gbps

12 x 42 Gbps

NFE

NFE

ALE

ALE

12 x 42 Gbps

12 x 42 Gbps

NFE

NFE

Note: Internal ports are called as Hi-Gig/HG ports

Nexus9500 Series Line Card Summary


Information

X9600 Series Line


Cards

X9500 Series Line


Cards

X9400 Series

ASIC Technology

Merchant only
N9K-X9636PQ

Merchant+
N9K-X9564PX
N9K-X9564TX
N9K-X9536PQ

Merchant only
N9K-X9432PQ
N9K-X9464PX
N9K-X9464TX

Number of ASIC

3 T2

2 T2 + 2 NS
2 T2 + 2 NS
2 T2 + 2 NS

2 T2 40 gig 32 Ports
1 T2 48 1/10 gig , 4
QSFP

Non Blocking

Non Blocking

Line rate > 200 byte


packet

36 MB

104 MB

12 MB with one T2
24 MB with two T2

Buffer Size

High Level Block Diagram-N9500

All PSU, SC, SUP, FM, and


LC plug into the same
Power Supply Interface

N9K-C9300 Series

Fixed Chassis

Port QSFP+ Uplink Module

1 RU or 2RU or 3RU

AC/DC Power Supply

Front-to-Back & Back-to-Front Airflow

Latency: 1-2 usec

Wire-Speed L2/L3 Forwarding

Switch will not boot up without GEM

Expansion Module

Nexus 9300 Series Switch Summary


N9396TX/PX

N93128TX

N9372TX

N9372TX

N9372PX

NFE (BCM T2)

ALE ( NorthStar)/GEM

GEM-1 NS

GEM-1 NS

No GEM-1
Donner

No GEM -1
Donner

No GEM- 1
Donner

Oversubscribed

No

1.5:1

No

No

No

Line Rate

Yes

Yes (packets >

Yes

Yes

194-Bytes)

QoS Classes

Buffer (MB)

36 (12*3)

104 (12*2+40*2)

24 (12*2)

104 (12*2+40*2)

104 (12*2+40*2)

High Level Block Diagram-N9300


GEM 4x 40GE QSFP+ Uplinks

12
x 40G
(12+12)x12
Hi-Gig2

CPU
2C 1.5GHz

(12+12)x12

Trident II
ASIC
BRCM Trident2
12 x 40G
48 10G
Ethernet

PCIe

DDR3
DIMM2
16GB Total

NorthStar
Northstar ASIC 1
Ingress
Egress

x 12 40G

Network Interfaces

Front Panel 48x 1GE/10GE Ports

The last 2/3 numbers stand for


total bandwidth in Gigabits

1000BaseT
Mgmt Port
2 USB
Ports
eUSB
Boot Flash
12C

93128 128G (96 x 10G + 8 x 40G)

9396 96G (48 x 10G + 12 x 40G)

9372 72G ( 48 x 10G + 8 x 40 G)

Nexus9500 Unicast Packet Flow


Fabric Module
Performs L3 LPM
lookup and resolves
Egress port and
next-hop

Fabric Module
L3 LPM Lookup & Forwarding
Ingress Line Card

Classify traffic
based on 802.1q
COS, IP Pres,
DSCP &ACL
Remark if needed

L2/L3 Lookup in
MAC Table and IP
Host Table

Parse the first 128


Byte and extract
header info

Egress Line Card

EoQ

ALE-NS
Ingress
Accounting &
Policing
Traffic
Classification
& Remarking
I-ACL
L2/L3
Lookup &
forwarding

T2-NFE
Output Q
& Shaping
E-ACL
Packet
Modification

Parser
Network Interface

EoQ

ALE-NS
OOBFC
Signaling

Ingress
Accounting &
Policing
Traffic
Classification
& Remarking
I-ACL
L2/L3
Lookup &
forwarding

T2-NFE

OOBFC
Signaling

Output Q
& Shaping
E-ACL
Packet
Modification

Parser
Network Interface

Additional buffer is
available for
extended out put
Ques EoQ

Class-based output
queues. Support 6
classes including
control traffic class

Egress Line card


sends packet to
egress port based
on DMOD/DPORT

N9K-C9300 High Level Block Diagram

(16 x 10G) x 3 =
480G FP Bandwidth

(12 x 40G) = 480G


Bandwidth to GEM
Module

HiGiG2 Interface on T2
MACF ports on the GEM and to MACN ports

(12 x 40G) = 480G FP


Bandwidth Uplink Ports
MACN ports.

Main Features of Trident2 1280Gbps Switch ASIC


Features

DCB Engine
Content aware Engine

L2
MAC

L2/L3
Processing

Dynamic Memory
Manager

Packet
Buffer

128 Integrated SerDes


128 SERDES@10Gbps
OR
32 SERDES@40Gbps

L2/L3
Multicast
L3
Route

Information

Maximum IO and Core bandwidth

1280G

MAC(L2) Entries

32K min -288K max

L3 Hosts

IPv4:16K min-112Kmax
IPv6:8K min-56 max

L3 Multicast Group

8K

Virtual Ports

16K

Maximum number of Physical ports

104

Counters
Host IF

North Star
Features

Information

Support Mixed Speed but in Fixed


configuration.

Network Interface:12 Ports Fabric


Interface: 12 40 Gig

Forwarding

720Mpps lookup rate on Ingress


Datapath
720Mpps lookup rate at Egress
Datapath

Shared Memory Subsystem


Ingress Path Buffer
Egress Path Buffer

10 Mbytes
30 Mbytes

Maximum number of Physical ports

24

Broadcom Unified Forwarding Table


T2 has the following Unified Forwarding Table:
SUPPORTED COMBINATIONS
Mode

L2

L3 Host

LPM

288K

16K

16K

224K

56K

16K

160K

90K

16K

98K

122K

16K

32K

16K

128K

Routing Mode for Nexus9300


LPM Routing Mode

Broadcom T2
Mode

Default system routing mode

ALPM Routing mode

CLI Command

System routing max-mode l3

N93K#show system routing mode


Configured System Routing Mode: Hierarchical
Applied System Routing Mode: Hierarchical (Default)
N93K#show hardware internal forwarding table utilization module 1
Max Host Route Entries (shared v4/v6): 124928
Max LPM Table Entries : 16384

Routing Mode for Nexus9500


LPM Routing Mode

Broadcom T2 Mode

Default System routing mode

3 (For Line card)


4 (For Fabric Module)
2--Line Card- V6 in LPM
3--For Fabric Module
3--For Line Card
4--With max-l3-mode option
For Line card
No Routes on Fabric Module

Max-host routing mode


Nonhierarchical routing mode

64-bit ALPM routing mode

Sub mode of mod 4 for


Fabric modules

show hardware internal forwarding table utilization mod 1


Max Host Route Entries (shared v4/v6):16384
Max LPM Table Entries : 131072
show hardware internal forwarding table utilization mod 21
Max Host Route Entries (shared v4/v6): 0
Max LPM Table Entries :0

Cli Command

System routing max-mode host


System routing non-hierarchical
Option [max-l3-mode]
System routing mode hierarchical
64b-alpm

Non hierarchical
routing mod

ACL TCAM TABLE


Characteristic
Ingress ACL: 4K TCAM entries - 4x 512 banks + 8x 256 banks
Egress ACL: 1K TCAM entries - 4x 256 banks
Each ACL type needs its own dedicated bank/banks

IPv4, IPv6 or MAC each needs dedicated bank/banks


MAC-ACL IPv6 & any QOS needs double-width entries, which means needs at least 2 banks
VACL is programmed symmetrically in both egress and ingress ACL
Interface
Type

Ingress ACL

Egress ACL

SVI

TCAM Shared

TCAM Not shared

L3

TCAM Shared

TCAM Shared

ACL Characteristics

Atomic/hitless update of existing applied ACL while modified

Temporary label swap (no use of default-result)

Two acl copies in tcam, if there is no enough space, process fails

ACL TCAM banks chaining not supported

L4OPs/LOUs only used for expansion beyond 5 lines, configurable

10 L4op per acl limit

Specific applications (dhcp, bfd) may install their own ACLs which must merge
with user configured racl, vacl, pacl

TCAM Carving for Nexus 9000


TCAM Region-N9500

Size Per Region

IPV4 RACL

1536

IPv4 L3 QOS

256

Ingress System

256

SPAN

256

Ingress CoPP

256

Redirect

256

vPC Convergence 512


Egress IPv4
RACL

768

Egress System

256

Ingress

Ingress

TCAM Region-N9300

Size Per Region

IPv4 PACL

512

512

IPV4 VACL

512

512

IPV4 RACL

512

512

256

IPv4 Port QOS

256

256

256

Ingress System

256

256

256

SPAN

256

256

Ingress CoPP

256

256

Redirect

256

256

vPC Convergence

512

512

Egress IPv4 RACL

256

Egress IPv4 VACL

512

256
512

Egress System

256

256

3X512
256

256
512

Egress
3X256
256

Egress

ACL TCAM Default Region and Carving

TCAM Banks will first get assigned to Feature which has largest region.

Next TCAM Bank will get assigned to Feature which need double Width.

TCAM Carving requires Line Card/TOR reload to take effect

To read current TCAM allocation


N9K#Show system internal access-lists global

To reconfigure TCAM Region


N9K(config)hardware access-list tcam <feature name> <size>

Buffer And Queuing-T2

T2 has 12 Mbytes of
Buffer shared by all
ports for all Traffic

Shared buffer divided Into


Control and default service
pool if module is T2 only

Shared buffer divided


into Control, default and
OOBFC service Pool if
Module is T2 and NS
based

Control
Shared
Buffer
12 MB

Control

Default

Module with T2 only

OOBFC: Out of band flow control unicast service pool

Shared
Buffer
12 MB

OOBFC
Default

Module with T2 And NS

Buffer And Queuing-North Star


GEM 4x 40GE QSFP+ Uplinks

10 MB
Buffer

NorthStar
ASIC 1

20 MB
Buffer

10 MB
Buffer

North Star has 40 Mbytes of Buffer


Divided in to Three Pool
Control , SPAN , Default

12 x 40G
Hi-Gig2

Control
Trident II
ASIC
12 x 40G
Ethernet

Front Panel 48x 1GE/10GE Ports

Shared
Buffer

SPAN
Default

Buffer Boost Function with T2 and NS


Fabric Module

ALE-NS
10 MB
Buffer

NFE
T2

10 MB
Buffer

20 MB
Buffer

12 MB Buffer Shared by all


ports

Network Interface

1/10GE

1/10GE

1/10GE

1/10GE

Buffer boost is function which allow T2 to use extra


buffer of NS
When Buffer boost is enabled on a port , T2 Local switch
traffic is Sent to NS for extra buffer space When Buffer boost is disabled on a port, T2 local traffic
to this port remains local on this NFE
Buffer Boost is enabled by default and can be disabled
on a per port basis

System Health check Telemetry

Most Common System Health Check

What is the Best Recommended NX-OS Release


CPU & Memory usage
Inter Process Messaging usage-MTS
Traffic Stats/Drop To CPU
CoPP/Hardware Rate Limiter Drops
Interface Errors for STP/Error disable
Ethernet Out of Band Drops/Error
Inter ASIC Utilization
Instant Buffer usage Stats
Hardware Capacity Check
FATAL System Errors
Consistency Checkers Various Tables
GOLD Diagnostic Checks
Sev1/2 Syslog

General Recommendation for New and Existing


Deployments
Software Recommendation
Platform Series

Minimum Release

Recommended Release

Cisco Nexus 9500

6.1(2)I2(2b)

6.1(2)I3(4a)

Cisco Nexus 9300

6.1(2)I2(2b)

6.1(2)I3(4a)

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/recommend
ed_releaseb_Minimum_and_Recommended_Cisco_NXOS_Releases_for_Cisco_Nex
us_9000_Series_Switches.html

Verified Scale limits for different features and protocol for each release
http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/6x/scalability/guide_34/b_Cisco_Nexus_9000_Series_NXOS_Verified_Scalability_Guide_61
2I34/b_Cisco_Nexus_9000_Series_NXOS_Verified_Scalability_Guide_612I34_chapter_01.
html

CPU & Memory Usage


N9K#show system resources
Load average: 1 minute: 0.00 5 minutes: 0.03 15 minutes: 0.05
Processes
:
432 total, 1 running
CPU states :
2.76% user, 0.75% kernel, 96.48% idle
CPU0 states : 0.00% user, 0.00% kernel, 100.00% idle
CPU1 states : 0.00% user, 1.01% kernel, 98.98% idle
CPU2 states : 0.00% user, 2.94% kernel, 97.05% idle
CPU3 states : 10.89% user, 1.98% kernel, 87.12% idle
Memory usage:
16402328K total,3443588K used, 12958740K free
Current memory status: OK

N9K#show system internal memory-usage-per-module in-KB


Slot 01:Used:1647420 Kbytes,Free:425680 Kbytes,Total:2073100 Kbytes
Slot 02:Used:1627524 Kbytes,Free:445576 Kbytes,Total:2073100 Kbytes
Slot 04:Used:1647560 Kbytes,Free:425540 Kbytes,Total:2073100 Kbytes

N9K#show system internal memory-alerts-log

Make sure log is clean

CPU

D
R
A
M

D
R
A
M

CPU & Memory Usage


Provides top process using CPU cycle
show processes cpu sort | head lines
PID
Runtime(ms) Invoked
uSecs
----- ----------- -------- ----3357
220
3100
7099
5853
31655
10181
3109
5859
9489
52308
181
3477
672
3107
216
3478
268
175
1535

12
1Sec
-----45.50%
0.50%
2.00%
0.50%
0.50%

Process
----------adjmgr
ipqosmgr
diag_port_lb
netstack
ospf

Possibly ARP Table Churn

Top Command-display top CPU processes


top provides an ongoing look at processor activity in real time
N9K#run bash
Auto update
bash-4.2$ top
top - 11:13:32 up 9 days, 3:34, 4 users, load average: 0.11, 0.11, 0.08
Tasks: 226 total, 1 running, 220 sleeping, 0 stopped, 5 zombie
Cpu(s): 0.8%us, 0.2%sy, 0.0%ni, 98.5%id, 0.0%wa, 0.1%hi, 0.3%si, 0.0%st
Mem: 16402328k total, 3445044k used, 12957284k free, 72676k buffers
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root
20 0 2152 620 556 S
0 0.0 0:08.05 init
2 root

20

0S

0 0.0

0:00.00 kthreadd

3 root

20

0S

0 0.0

0:00.58 ksoftirqd/0

Inter Process Messaging Usage


N9K#sh sockets client detail | inc pim|drops|Errors
select drops:
Errors:
select drops:
Errors:
select drops:
Errors:

10
Make Sure Drops/Errors not incrementing
0

Message and transaction service-MTS

N9K#sh system internal mts buffers sum | diff


node
sapno
recv_q pers_q npers_q log_q
sup
320
0
0
4592
0
sup
284
0
19
0
0
sup
250
2
0
0
0

For SAP 320 own


by OSPF npers_q
increasing

Inband Driver Statistics-CPU Drops


N9K#show hardware internal cpu-mac inband stats
eth2 stats:
RMON counters
Rx
Tx
total packets
601163425
318962431
Per Queue Stats
Queue Idx
Packet Count Bytes Drops Csum Errors Allocation Failure
Queue 0
17677525
111822449180 0
0
0
- - - - - - - - - - - - - - -SNIP- - - - - - - - - - - - - - - - - - - - - - - Queue 7
17677525
111822449180 0
0
CRC 0errors/Collisions/late
Collisions
Alignment errors
Interrupt Counters
Symbol errors
Rx overrun
0
Sequence errors/Rx jabbers
Error counters
RX errors/Rx length errors

Rate statistics
Rx packet rate (current/peak) 717 / 80695 pps
Tx packet rate (current/peak) 360 / 1338 pps

Traffic Stats/Drops to CPU (Contd)


N9K# show system internal frame traffic | in drops
Global input drops:

bad-interface 0, bad-encap 0, failed-decap 0,

Global output drops:

Drops From PKTmgr


eth_output_err 0, gre_err 0 otv_err 0 span_drop_en: 0 span_drops: 0
Crossbar down drops : 0

Flood_to_core LTL: Hits: 0 Misses: 0

N9K# show system inband queuing statistics | in drop


bpdu: recv 68, drop 0, congested 0 rcvbuf 2097152, sndbuf 262142 no drop 0
(q0): recv 1249377,

drop 0, congested 0

rcvbuf 2097152, sndbuf 262142 no drop

(q1): recv 4138154,

drop 0, congested 0

rcvbuf 2097152, sndbuf 262142 no drop

Instant Buffer Utilization For CPU Port


show hardware internal buffer info pkt-stats cpu
[Q00-07]
0
0
0
0
0
0
0
0
[Q08-15]
0
0
0
0
0
0
0
0
[Q16-23]
0
0
0
0
0
0
0
0
[Q24-31]
0
0
0
0
0
0
0
0
[Q32-39]
0
0
0
0
0
0
0
0
[Q40-47]
0
0
0
0
0
0
0
0

Total 48 Queues

Each Line Display Cell utilized


for 8 queues

One Cell represent approximately


208 Bytes

Congestion encountered if Counters


keep incrementing

Ethernet Out Of Band Drops/Errors


N9K#show hardware internal eobc stats | inc dropped
RX packets:248308217 errors:0 dropped:0 overruns:0 frame:0
TX packets:71554006 errors:0 dropped:0 overruns:0 carrier:0
N9K# show system internal emon stats
EMON MOD ONLINE BMP: 37f00067
FSM ID: 0 EOBCMON
=======================================
HB tx_req
186396
Provides Stats for all Modules
module 1:
including Fabric module
rx_req
176410
rx_resp
176426
Heart bit miss
rx_miss
7
tx_resp
176410

Instant Buffer Usage Stats


Buffer polling interval for 7.0
N9K#show hardware internal buffer info pkt-stats mod 1

Release is 500msecs

INSTANCE: 0
---------------------------------------------------------Output Shared Service Pool Buffer Utilization (in cells)
SP-0
SP-1
SP-2
SP-3
----------------------------------------------------------Total Instant Usage

4474

89

2939

Remaining Instant
Usage

25466

14255

3405

Peak/Max Cells Used

4821

327

3060

29940

14344

6344

Switch Cell Count

---------------------------------------------------------show hardware internal ns buffer info pkt-stats

Instant Buffer utilization per queue


per port
One cell represents 208 bytes
Show hardware internal buffer info pktstats input mod 1
SP-3-Dedicted resource for Control
Traffic
SP-0-Resource for Locally Switched
Unicast ,Multicast and SPAN

SP-2 Extended Output queue for


Unicast using buffers from North
Star

Instant Buffer Usage Stats - With Buffer Usage


N9K#show hardware internal buffer info pkt-stats mod 1
INSTANCE: 0
Output Shared Service Pool Buffer Utilization (in cells)
SP-0

SP-1

SP-2

SP-3

------------------------------------------------------------------------Total Instant Usage


Remaining Instant Usage

4474

89

2939

25466

14255

3405

SP-3 Started filling


the Queue

-------------------------------------------------------------------------

ASIC Port
[13]

Q3

Q2

Q1

Q0

Port 13 onwards are Front Panel Port

UC(OOBFC)->0

CPU

SPAN

Only printed if there is congestion

UC->

1249

332

MC->

3247

1996

CPU buffer filling


up

CoPP Drops
N9K# show policy-map interface control-plane mod 1 | in dropped
dropped 0 packets;
dropped 0 packets;

dropped 0 packets;
dropped 0 packets;
dropped 7800 packets;

Drops Seen for Default-Class at minimal rate is normal

We recommend that you use the strict default CoPP policy initially and then later modify the CoPP
policies based on the data center and application requirements.
Parameters

Default

Default policy

Strict

Default Policy

9 policy entries

CoPP Drops-Exception drops


class-map copp-system-p-class-l3uc-data (match-any)
match exception glean
class-map copp-system-p-class-redirect (match-any)
match access-group name copp-system-p-acl-ptp
class-map copp-system-p-class-exception (match-any)
Goal is to Classify all Traffic Using CoPP
match exception ip option
match exception ip icmp unreachable
match exception ipv6 option
match exception ipv6 icmp unreachable
class-map copp-system-p-class-exception-diag (match-any)
match exception ttl-failure
match exception mtu-failure

Hardware Rate Limiter


N9K# show hardware rate-limiter mod 1
Units for Config: packets per second
Allowed, Dropped & Total: aggregated since last clear counters
Module: 1
R-L Class

Config

Allowed

Dropped

Total

+----------+-----+------------+------------+-------------+
L3 glean

100

L3 mcast loc-grp

3000

access-list-log

100

bfd

10000

1352890

fex

3000

span

50

1352890

FATAL System Errors


N9K#show logging onboard mod 1 exception-log | incl FATAL prev 15
-----------------------------------------------------------------------Date (mm/dd/yy)=01/15/15

Time (hs:mn:sec): 00:16:58

OBFL Exception log data for THIS SUP Module:0

********* Exception info for module 0 ********


exception information --- exception instance 1 ---Device Name

: System Manager

Device Errorcode

: 0x0000023a

ErrNum (devInfo)

: 58 (0x3a)

System Errorcode
Error Type

: 0x401e0089 Service in VDC has had a hap-reset


: FATAL error

Common Interface Error counters and Status


N9K# show interface counters errors mod 4
Port

Align-Err

FCS-Err

Xmit-Err

Rcv-Err

UnderSize OutDiscards

-------------------------------------------------------------------------Eth4/1

100

581

N9K# show interface status err-disabled


Port

Name

Status

Reason

--------------------------------------------------------------------------

Eth4/1

err-disable

link-flap

Interface Queuing Stats


N9K#show queuing interface 4/18

Egress Queuing for Ethernet4/18 [System]


QoS-Group# Bandwidth% PrioLevel
3

Shape

Min

Max

Qlimit

Units
-

6(D)

-------------------------SNIP--------------------------

100

6(D)

---------------------------------------------------QOS GROUP 0
Unicast
Dropped Pkts |

0|

| OOBFC Unicast

0|

Multicast
0|

-----------------------------------------------------------QOS GROUP 7
Unicast
Dropped Pkts |

0|

| OOBFC Unicast
0|

Multicast
0|

Inter ASIC Utilization-HG Ports


T2 #0

T2 #1

HG00

HG00

Fabric Module

T2 #0

T2 #1

T2 #2

N9K#show system internal interface counters mod 1

Line Card

Internal Port Counters (150 secs rate) for Slot: 1


====================================================
Interface

ASIC

ASIC

BCM

Port

Inst

Port

TxBitRate(BwUtil) TxPktRate
(bps)

(pps)

RxBitRate(BwUtil) RxPktRate
(bps)

(pps)

----------------------------------------------------------------------------------------ii1/1/1

HG0

170512 (0.00)

0(0.00)

-------------------------------------------Snip-----------------------------------------ii1/1/14

HG1

ii1/1/25

HG0

0( 0.00)
1790648

(0.00)

1129882872(2.51)

960753

1043

22864(0.00)

20

Verify Consistency Between Software and Hardware


Table
Table

CLI

Physical Interface

show consistency-checker link-state

Port-Channel
Membership
Mac Address Table

show consistency-checker membership port-channels

Vlan Membership

show consistency-checker membership vlan

L3 interface-LIF
programming
For RIB and FIB

L3 interface-LIF programming Logical Interface for Routing

show consistency-checker l2

show consistency-checker forwarding ipv4 unicast

Consistency Checkers-Link and STP state


N9K#show consistency-checker link-state mod 1
Link State Checks: Link state only

Consistency Check: PASSED


No inconsistencies found for:
Ethernet1/1
2015 Mar 24 03:23:27 N9508a-SJ %$ VDC-1 %$ vshd: CC_LINK_STATE: Consistency
Check: PASSED

N9K# show consistency-checker stp-state


Checks: Spanning tree state
Consistency Check: PASSED

vlan 18

2015 Mar 24 03:25:21 N9508a-SJ %$ VDC-1 %$ vshd: CC_VLAN_STP_STATE:


Consistency Check: PASSED

Consistency Checkers-Port Channel-Vlan Membership


N9K#show consistency-checker membership port-channels
Checks: Trunk group and trunk membership table.
Consistency Check: Failed
Inconsistency found for port-channel1:
Module:1, Unit:

['Ethernet3/49', 'Ethernet2/49']

Module:26, Unit: ['Ethernet3/49', 'Ethernet2/49]


N9K# show consistency-checker membership vlan

18

Checks: Port membership of Vlan in vlan and egr_vlan table


Ports configured as "switchport monitor

will be skipped

Consistency Check: PASSED


Vlan:18, Hardware state consistent for:
Ethernet2/49
2015 Mar 24 03:28:31 N95a%$ VDC-1 %$ vshd: CC_VLAN_MEMBERSHIP: Consistency
Check: PASSED

Consistency Checkers-Mac address Table


N9K# show consistency-checker l2 module 1
Consistency check: PASSED
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen, + - primary entry using vPC Peer-Link,
(T) - True, (F) - False
Missing entries in the HW MAC Table
VLAN

MAC Address

Type

age

Secure NTFY

Ports

---------+-----------------+--------+---------+------+----+-----------------Extra and Discrepant entries in the HW MAC Table


VLAN

MAC Address

Type

age

Secure NTFY

Ports

---------+-----------------+--------+---------+------+----+------------------

Consistency Checkers-L3 Interface


N9K# show consistency-checker l3 mod 1
L3 LIF Checks: L3 Vlan, CML Flags, IPv4 Enable
Consistency Check: PASSED

No inconsistencies found for:


Ethernet1/1
Ethernet1/2
Ethernet1/3

2015 Mar 24 04:07:27 N9508a-SJ %$ VDC-1 %$ vshd: CC_L3_LIF: Consistency Check:


PASSED

Consistency Checker Unicast Forwarding


N9K#test consistency-checker forwarding

Consistency check started.


N9K# show consistency-checker

forwarding

ipv4 unicast module 1

IPV4 Consistency check (in progress): table_id(0x1) slot(1)


Elapsed time : 8257 ms

N9K# show consistency-checker

forwarding

ipv4 unicast

IPV4 Consistency check : table_id(0x1) slot(1)


Execution time : 13244 ms ()
No inconsistent adjacencies.

No inconsistent routes.
Consistency-Checker: PASS for 1

module 1

Gold Diagnostic Checks


N9K# show diagnostic result mod 2

On Demand Diagnostic can be executed

Module 2: 48x1/10G-T 4x40G Ethernet Module

Test results:(.=Pass, F=Fail,I=Incomplete,U=Untested,A=Abort,E=Error disabled)


1) ASICRegisterCheck------------>

2) PrimaryBootROM--------------->

3) SecondaryBootROM------------->

4) OBFL------------------------->

6) BootFlash-------------------->

7) AsicMemory------------------->

8) FpgaRegTest---------------- ->

9) PortLoopback:--------------- >

Port

RewriteEngineLoopback

9 10 11 12 13 14 15 16

----------------------------------------------------U

Sev1/2 Syslog
show logging logfile | incl -1-|-22015 Feb 25 10:30:17 N9508a-SJ %PLATFORM-2-MOD_PWRUP: Module 26 powered up
(Serial number SAL1738D37W)
2015 Feb 25 10:32:37 N9508a-SJ %XBAR-2-XBAR_HGLINK_NOT_UP: fabric link 1 on
module 2 unit 0 connected to fabric module 26 unit:0 is not up during module
bring up
2015 Feb 25 10:32:39 N9508a-SJ %MODULE-2-MOD_FAIL: Initialization of module 26
(Serial number: SAL1738D37W) failed
2015 Feb 25 10:32:39 N9508a-SJ %PLATFORM-2-MOD_PWRDN: Module 26 powered down
(Serial number SAL1738D37W)

Troubleshooting Toolkit

Troubleshooting Toolkit
Ethanalyzer

TCP Dump
ELAM
Packet Tracer
Flex Counter
ERSPAN

Consistency Checkers

Ethanalyzer-When To Use it
To Analyze the traffic sent and received by CPU
It uses wiresharks code (an open source software)

Netstack

Pseudo Inband

Troubleshooting High CPU

Troubleshoot Control Plane issues Ex. OSPF , PIM , STP

SUP
NIC-ETH2

Flap.

Note: Ethanalyzer does not allow capturing of hardware switched traffic between data
ports of the switch

Ethanalyzer-CLI
N9K# ethanalyzer local interface inband capture-filter "pim

detail

Capturing on inband
Frame 1 (60 bytes on wire, 60 bytes captured)
Arrival Time: Mar 24, 2015 10:01:10.018889000
-------Snip-----------------[Protocols in frame: eth:ip:pim]
N9K#ethanalyzer local interface inband display-filter "ospf detail
Capturing on inband
Frame 1 (82 bytes on wire, 82 bytes captured)
Arrival Time: Mar 24, 2015 10:04:11.425523000
-------------------Snip-------------------[Frame is marked: False]
[Protocols in frame: eth:ip:ospf]

Some Available Options


autostop

:Autostop

decode-internal
header decoding

:Internal

limit-captured-frames :Maximum
number of

TCP Dump
Tcpdump command works on most flavors of Linux operating system
Helps to prints out a description of the contents of packets on a network interface
Tcpdump will, if not run with the -c flag, continue capturing packets until it is
interrupted by a SIGINT signal CTRL-C
Tcpdump output can be saved to file for further reference
More info at http://www.tcpdump.org/

Tcpdump -syntax
N9K# show feature | in bash

Feature Name Instance


bash-shell

State

tcpdump version 4.1.1

enabled

libpcap version 1.2.1

N9K# run bash


bash-4.2#

Syntax: tcpdump -h

sudo su

Usage: tcpdump [-aAbdDefIKlLnNOpqRStuUvxX] [ -B size ] [ -c count]


[ -C file_size ] [ -E algo:secret ] [ -F file ] [ -G seconds]

Password:******

[ -i interface ] [ -M secret ] [ -r file ]

bash-4.2# whoami

[ -s snaplen ] [ -T type ] [ -w file ] [ -W filecount ]

root

[ -y datalinktype ] [ -z command ] [ -Z user ]

bash-4.2# tcpdump c 10 I ps-inb

[ expression ]
bash-4.2#

Tcpdump-Examplesbash-4.2# tcpdump -c 100 -w tcpdump.pcap -vvvv -i ps-inb

Capturing 100 packets And


tcpdump: WARNING: ps-inb: no IPv4 address assigned
writing to file
tcpdump: listening on ps-inb, link-type EN10MB (Ethernet), capture size 65535
bytes
100 packets captured
102 packets received by filter
bash-4.2#cd /bootflash

bash-4.2# tcpdump -tttt -r tcpdump.pcap | more


reading from file tcpdump.pcap, link-type EN10MB (Ethernet)

Reading captured file

2015-04-26 03:21:31.309350 00:0e:ee:01:1b:01 (oui Unknown) > 00:00:00:ff:ff:01


(oui Ethernet), ethertype Unknown (0x8833), length 160:

0x0000:

0000 fc08 0b00 0000 0000 0800 0000 0ffd

...............

-------------------------------------more---------------------------------

tshark
bash-4.2$ tshark -i ps-inb
Capturing on inband
0.000000 00:0e:ee:01:1b:01 -> 00:00:00:ff:ff:01 0x8833 Ethernet II
12.328377 00:0e:ee:01:1b:01 -> 00:00:00:ff:ff:01 0x8833 Ethernet II
^C2 packets captured
bash-4.2$

Elam-Embedded Logic Analyzer Module-NS

Elam Allows to capture single packet based on Trigger


Triggers are configured using Packet information
Only Supported on North Star Based(ALE) Line Cards and GEMs
Use with TAC Supervision
Help to Answer following Questions
Was the Packet indeed Received by device on given Line card?
How did the Packet Look like?
How was the packet rewritten based on forwarding Decision made by T2?
Was the Packet correctly forwarded or Dropped?

ELAM Configuration

Init Initialize the ELAM select the Asic instance, pipeline and
select lines
module-1# debug platform internal ns elam asic
module-1(NS-elam)# trigger init ingress in-select 3 out-select 5

2. Config

Config Configure the trigger based on different fields in the packet


module-1(NS-elam-insel3)# set outer ipv4 src_ip 13.13.13.10

3. Arm
Trigger

Arm Arm the trigger by setting the fields to match in hardware


module-1(NS-elam-insel3)# start

Read Once the trigger is triggered, read the report


module-1(NS-elam-insel3)# report

Reset Once the process is complete, reset the trigger to restart


the process
module-1(NS-elam-insel3)# reset

1. Init

4. Read
5. Reset

Elam Ingress & Egress Direction-TOR


Traffic entering GEM ports which has NS and
exiting T2 is Egress Pipeline

GEM 4x 40GE QSFP+ Uplinks

Egress

Ex. trigger init egress in-select 3 out-select 5


set outer ipv4 dst_ip 13.13.13.10

Ingress
NorthStar
ASIC 1

12 x 40G
Hi-Gig2

Traffic Entering T2 and Exiting GEM ports is


Ingress Pipeline

Trident II
ASIC
12 x 40G
Ethernet

Network Interfaces

Ex. trigger init ingress in-select 3 out-select 5


set outer ipv4 src_ip 13.13.13.10

Front Panel 48x 1GE/10GE Ports

IP.Add=13.13.13.10

Elam Ingress & Egress Direction-EOR


Traffic entering from Fabric Module in to NS of
Line Card is Egress Pipeline

Fabric 3
Fabric 1

Ex. trigger init egress in-select 3 out-select 5


set outer ipv4 dst_ip 13.13.13.10

Egress

N
FE

N
FE

Ingress

Line Card
North Star ASIC

Traffic Entering NS and exiting towards Fabric


Module is Ingress Pipeline

12 x 40G
Hi-Gig2

12 x 40G
Ethernet

Ex. trigger init ingress in-select 3 out-select 5


set outer ipv4 src_ip 13.13.13.10

Trident II
ASIC
Network Interfaces

Front Panel 48x 1GE/10GE Ports


13.13.13.10

ELAM Sample Configuration & Key Info


Nexus9508 with N9K-X9564TX

13.13.13.1/30

Eth6/52

Eth5/1

N9K-X9564TX

13.13.13.10/30

4 40Gig Port On NS

40 1/10 Gig On T2

N9K# attach mod 6


module-6# debug platform internal ns elam asic 1
module-6(NS-elam)# trigger init egress in-select 3 out-select 5
module-6(NS-elam-insel3)# set outer ipv4 dst_ip 13.13.13.10
module-6(NS-elam-insel3)#

start

module-6(NS-elam-insel3)#

status

module-6(NS-elam-insel3)#

report

If Packet Captured
Status: Triggered

Important ELAM Fields


GBL_C++ [INFO] hg2_srcmod: 0E
GBL_C++ [INFO] hg2_srcpid: 0D
GBL_C++ [INFO] hg2_dstmod: 11

Information
is in Hex
Convert to
Dec.

GBL_C++ [INFO] hg2_dstpid: 0A


GBL_C++ [INFO] ip_da: 000000000000D0D0D0A
GBL_C++ [INFO] ip_sa: 000000000000D0D0D01

GBL_C++:
GBL_C++:

[MSG] - sideband is complete


[INFO]
ovector: 000FFF

N9K# show interface hardware-mappings


---------------------------------------------------------------------Name Ifindex Smod Unit HPort FPort NPort VPort

-----------------------------------------Eth5/2 1a280000 14 0 13 255 0


-1
Eth6/52 1a286600 17 1 10 255 51
-1

Sideband is the result where


packet will be sprayed.
Should never be 0

Packet Tracer-T2
FM Mod
Helps to Trace the packet inside Switch.
Only packets in the direction of the flow are traced

Trident II
ASIC

Two Acls are installed for each filter on each Line card
One ACL for Front Panel Port Group
Second ACL for traffic exiting Fabric Module and ingressing Line
card

Network Interfaces

Packet Tracer Configuration


test packet-tracer dst-ip 13.13.13.10 detail-fp

Configure Filter

test packet-tracer dst-ip 13.13.13.10 detail-hg


13.13.13.10/30

Start Tracer

test packet-tracer start

rt
Stop Tracer

Check Counter
Filter
Clear/Remove-all

test packet-tracer stop

test packet-tracer show

test packet-tracer clear remove

Sample Configuration & Identify Front Port-LC


N9K#test packet-tracer dst-ip 13.13.13.10 src-ip 13.13.13.1 detail-fp
N9K#test packet-tracer show
Module 6:
Filter 1 installed: src-ip
Module 21:
Filter 1 installed: src-ip
Module 26:
Filter 1 installed: src-ip

filter 1 non-zero Packet-tracer stats


13.13.13.10/30

13.13.13.1 dst-ip 13.13.13.10 detail-fp


13.13.13.1 dst-ip 13.13.13.10 detail-fp

13.13.13.1 dst-ip 13.13.13.10 detail-fp

Nexus9508 with N9K-X9564TX


Eth6/1

13.13.13.1/30

Eth6/52

13.13.13.10/30

Packet Tracer Sample Configuration & Key Info


N9K# test packet-tracer start filter 1

N9K# test packet-tracer show filter 1

13.13.13.10/30
Nexus9508 with N9K-X9564TX

mod 6 non-zero

Packet-tracer stats
Eth6/1

Module 6:
Filter1 installed: src-ip 13.13.13.1 dst-ip
13.13.13.10 detail-fp

Eth6/52

13.13.13.1/30

13.13.13.10/30

ASIC instance 0:
Entry 1: id = 7426, count = 5, active, fp, port 13

N9K# show interface hardware-mappings | grep 6/1


Name

Ifindex

Smod

Unit

Eth6/1

1a280000

16

Hport
13

FPort

Nport

255

VPort
-1

Sample Configuration Identify Fabric Port LC From FM


N9K# test packet-tracer dst-ip 13.13.13.10

src-ip 13.13.13.1 detail-hg

N9K# test packet-tracer start filter 1


N9K# test packet-tracer show

mod 6 non-zero

Module 6:
Filter 1 installed:

src-ip 13.13.13.1 dst-ip 13.13.13.10 detail-hg

ASIC instance 0:
Entry 0: id = 7425, count = 68, stopped, fp,
ASIC instance 1:

Eth8/1

Eth6/52

13.13.13.1/30

Entry 1: id = 7426, count = 13, stopped, hg, port 1


Entry 2: id = 7427, count = 11, stopped, hg, port 2

13.13.13.10/30

13.13.13.10/30
Nexus9508 with N9K-X9564TX

Flex Counters Adjacency Statistics


Flex counters used to count Next hop Adjacency stats
One can attach Stats to multiple Adjacency at same time
One Stat Counter per adjacency
Total Flex Counters are 16K per Switch

How To Configure Flex Counters


Nexus9508 with N9K-X9564TX

Eth6/1

Eth6/52

13.13.13.1/30

13.13.13.10/30

13.13.13.10/30

N9K# sh ip route 13.13.13.10


IP Route Table for VRF "default"
'%<string>' in via output denotes VRF <string>
13.13.13.8/30, ubest/mbest: 1/0
*via 13.13.13.6, Eth6/52, [110/41], 00:33:14, ospf-10, intra
N9K# test hardware internal adjacency statistics nexthop ipv4 13.13.13.6
interface ethernet 6/52 (enable |disable | show)

Sample Configuration
N9K# test hardware internal adjacency statistics nexthop ipv4 13.13.13.6
interface ethernet 6/52 show
13.13.13.10/30

Module:21 Unit:0
-----------------Adjacency counters for nhip 13.13.13.6 if Ethernet6/52:
Nexus9508 with N9K-X9564TX
Ucast: Packets 738 Bytes 90036
Mcast: Packets 0 Bytes 0
Eth6/1

Module:22 Unit:1
13.13.13.1/30
-----------------Adjacency counters for nhip 13.13.13.6 if Ethernet6/52:
Ucast: Packets 946 Bytes 115412
Mcast: Packets 0 Bytes 0

Eth6/52

13.13.13.10/30

SPAN & ERSPAN

Switch Port Analyzer


Provides efficient, high-performance traffic monitoring service
Duplicates network traffic to one or more monitor interfaces
Types Of SPAN
Local SPAN
Encapsulated Remote SPAN(ERSPAN)
Applications:
Troubleshooting connectivity issues
Base lining network utilization/performance
Detecting anomalous traffic flows
On Nexus9000 Span Traffic uses dedicated queue
Queue carrying SPAN traffic has low Priority over other queues
during congestion

SPAN QOS Queue


N9K# show queuing interface ethernet 4/18 | begin SPAN
|

SPAN QOS GROUP

+-----------------------------------------------------------------+
|

Unicast

| OOBFC Unicast

Multicast |

+------------------------------------------------------------------+
|

Tx Pkts |

0|

0|

0|

Tx Byts |

0|

0|

0|

Dropped Pkts |

0|

0|

0|

Dropped Byts |

0|

0|

0|

Q Depth Byts |

0|

0|

0|

SPAN Configuration

N9K(config)# monitor
N9K(config-monitor)#
N9K(config-monitor)#
N9K(config-monitor)#
N9K(config-monitor)#
N9K(config-monitor)#

Sup-eth

session 1
source interface sup-eth 0 both
source interface ethernet 6/1
e6/1
destination interface ethernet 6/2
Local SPAN
No Shut
N9K(config)#int et 6/2
show monitor

Session State
Reason
--- ---------------1
up The session is up

Local
e6/2

N9K(config-if)# switchport monitor

Description
-------------------Local SPAN Session

ERSPAN Configuration
Only Supports Source ERSPAN
N9K(config)# monitor erspan origin ip-address 13.13.13.2
global
Type-3 Header 32-bit Timestamp
N9K(config)# monitor session 1 type erspan-source
Supports on Nexus9300 only
N9K(config-erspan-src)# header-type 3
Layer 3
N9K(config-erspan-src)# source interface ethernet 6/1
N9K(config-erspan-src)# erspan-id 1
L3
N9K(config-erspan-src)# ip ttl 16
N9K(config-erspan-src)# vrf default
e6/2
e6/1
N9K(config-erspan-src)# destination ip 9.1.1.2
ERSPAN
N9K(config-erspan-src)# marker-packet-2
Marker packet carry original UTC time
N9K(config-erspan-src)# no shut
stamp to over come 32-bit wrapper
issue

Consistency Checkers-Summary

Show consistency-checker
Show consistency-checker
Show consistency-checker
Show consistency-checker
Show consistency-checker
Show consistency-checker
Show consistency-checker
Show consistency-checker

stp-state vlan
link-state
membership vlan
membership port-channels
membership port-channels
l2
l3
forwarding ipv4 unicast

Nexus 9000
Troubleshooting

Understanding T2 interfaces-Xe0/hg
N9K# bcm-shell mod 1 "show unit"
Unit 0 chip BCM56852_A2 (current)
Unit 1 chip BCM56852_A2
hg0

N9K#bcm-shell mod 1 0:ps


ena/ speed/ link auto
port link duplex scan neg?

STP
state

hg11

hg0

lrn
inter max
T2
pause discrd Instance
ops
face frame
0

hg11

loop
T2
back
Instance 1

hg0 up
42G FD HW No
Forward
None
FAF F FXGMII
16360
F F F F F F F
F F F F FQSPF
F F F F
F F F F FQSPF
hg2 up
42G FD HW No
Forward
None
16360
P P P P P P P
P P P P PPorts
PPorts
P P PXGMII
P P P P
P P P P FA
13 14 15 16 17 18 19 20 21 22 23 24
01 02 03 04 05 06 07 08 09 10 11 12
--------------------------------Snip---------------------------------Hg11 up
42G FD HW No
Forward
None FA XGMII 16360
Xe0
Xe11
Xe0
Xe11
Xe0 !ena
40G FD HW No
Disable
None
FA XGMII
1582
Eth1/13
Eth1/24
Eth1/1
xe1 up
40G FD HW No
Disable
None
FA Eth1/12
XGMII 1582
--------------------------------Snip---------------------------------Xe11 !ena
40G FD HW No
Disable
None FA XGMII 1582

Hg=Internal Ports
Xe=Front Panel Port

Layer -1 Issues- Transceiver Not Recognized


N9K# show interface ethernet 4/18 transceiver details
Ethernet4/18
transceiver is not present
module-4# show hardware internal bcm-usd event-history xcvr 18
1) Event:E_STRING, length:135, at 220346 usecs after Thu Apr 16 20:50:17 2015
bcm_usd_xcvr_fcot_notify_default(941): [unit=0 nxosport=18 bcmport=30]
fcot_state:0x2 fcot_type:0 sent MTS_OPC_FCOT_EVENT_INFO, rc 0x0
2) Event:E_STRING, length:93, at 647132 usecs after Thu Apr 16 20:50:14 2015
bcm_usd_xcvr_fcot_scan_sfp(3003): [unit=0 nxosport=18 bcmport=30]
FCOT not supported err=-1

Interface MTU/Speed/Flow Control Verification


N9K# show interface Ethernet 4/18

Ethernet4/18 is up
admin state is up, Dedicated Interface Belongs to Po10
Hardware: 10000/40000 Ethernet, address: 7c69.f66e.d860 (bia 7c69.f66e.d860)

MTU 9216 bytes, BW 40000000 Kbit, DLY 10 usec


N9K# bcm-shell module 4 1: ps Xe17"
ena/

speed/

port

link

duplex

xe17

up

40G

FD

link

auto

scan

neg?

HW

No

STP
state
Disable

lrn inter
pause

discrd

ops

None

FA

max

face frame
SR4

9298

loop
back

Interface Flow Control Check


N9K#Show interface ethernet 1/1 flowcontrol
Port

Send

FlowControl

admin

Receive

oper

FlowControl
admin

RxPause

TxPause

oper

-----------------------------------------------------------------------------

Eth1/1
0

off

off

off

off

N9K#bcm-shell module 1 "ps"

port
xe0

Wrong programming

ena/

speed/

link

auto

link

duplex

scan

neg? state

up

10G

FD

HW

No

TP
pause

Disable

discrd

TX RX

None

lrn

inter

ops

face

FA

SFI

max

loop

frame back
9298

Interface Input Drops


Ethernet1/30 is up
Hardware: 1000/10000 Ethernet, address: 7426.acea.ceb9 (bia 7426.acea.ceb9)
EtherType is 0x8100
0 input with dribble 1316 input discard
N9K#bcm-shell mod1 cstat xe29
+------------------Programmable Statistics Counters[Port xe29]------+
| Type | No. |
Value
|
Enabled For
|
+----------------------------------------------------------------- -+
| RX | 0(R)|
19163028| RIPD4 RIPD6 RDISC RPORTD
|
|
|
|
| PDISC VLANDR
|
|
| 1(R)|
28744286| IMBP
|
|
| 4
|
993820| RPORTD FcmPortClass3RxDiscards
|
|
| 6
|
19163407| RFILDR FcmPortClass2RxDiscards
|
|
| 7
|
19163048| RDROP
|
|
| 8
|
18169208| VLANDR
| | gre VLANDR
bcm-shell mod 6 "cstat info"
+-------------------------------------------------------------------+
VLANDR
Rx VLAN drops
|
| 3(R)|
14704| TPKTD
|
|
| 4(R)|
968303| TGIP4 TGIP6 FcmPortClass3TxFrames|
|
| 6
|
968303| TGIP4 FcmPortClass3TxFrames
|
+-------------------------------------------------------------------+

Fabric Connectivity and Troubleshooting


T2

T2

T2

T2

T2

T2

T2

N9K-C9508-FM-4

N9K-C9508-FM-8

N9K-C9516-FM-16

In an 4-slot chassis N9K-C9504-FM has 1 T2 per module


In an 8-slot chassis N9K-C9508-FM has 2 T2 per module
In an 16-slot chassis N9K-C9516-FM has 4 T2 per module
FMs provides redundancy for internal data flow, the loss of FMs just increases
the oversubscription factor.

Full-Rate Mode(FRM) V/S Oversubscribed Mode(OSM)


Each T2 have 32 40Gigport with total capacity of 1.2Tbps with 2 switching
mode
OSM(Default) - Uses all 32 40 Gig ports Line Rate achieved for packets > 200 Bytes
FRM - Uses only 24 40 Gig ports Line rate achieved for > 64 Bytes

Configuration Knob to Change the mode.


N9K(config)# system fabric-mode full-rate
Configuration effective after Reboot
N9K#show system fabric-mode
Applied System Fabric Mode:Full rate mode
Use FRM mode to achieve line rate for 64 byte packets on 9636PQ , 9564PQ ,
9564TX cards
All other 94xx line cards will not be powered up in this mode

RTAG7 and DLB


Two Packet Hashing algorithm available from LC to FM
RTAG7-To Select HG Port use Packet Header.
For a flow same HG Link is used

FM1

FM6

FM-2

DLB-Dynamic Load Balancing- Default algorithm


Initial Hash same as RTAG7

HGPorts

Based on Link Quality pick up optimum HG Port

HGPorts
LC1

Better utilization of all HG links

N9K(config)# port-channel load-balance internal [dlb/rtg7]

N9K# show port-channel load-balance internal algorithm

HighGig port-channel load balance algorithm: dlb

LC2

Higig Link Failures Fabric Module Policy


For any single Higig link failure between FM and LC
Bring down the FM, if there is more than one FM
Else bring down LC
Multiple Higig links failures for a Single LC going to Multiple FM - Bring down
the LC module.

Multiple Higig links failures on LC to one of the FM - Bring down the LC module

4/8 slot Chassis Fabric Connectivity

9500/9600 Series Line Cards T2


have connectivity to all 6 Fabric
Modules T2

9400 series Line cards connects to


all T2 but use only 4 Fabric Modules
-No Connection to Slot 21 & 25

Traffic between 9500/9600 Series


Line Card and 9400 Line card will
use subset Hi Gig links .

T
2

T
2

T
2

T
2

T
2

T
2

T
2

T
2

T
2

T
2

T
2

T
2

40 Gig Link
T2

T2

N9K-X9536PQ

16 slot Chassis Fabric Connectivity

9500 Series Line Cards T2 will have


connection to all 6 Fabric Module but to
only 2 T2s from each Fabric Module

9500 series Line Cards T2 will have


connection to all 4 T2s of Fabric module if
there are only 3 Fabric module present

9400 series cards connects to all T2 but


use only 4 FM-No Connection to Slot 21 &
25

N9K-C9516-FM
T
2

T
2

T
2

T
2

T
2

T
2

T
2

T
2

T
2

T
2

T
2

T
2

T
2

T
2

T
2

T
2

T
2

T
2

T
2

T
2

T
2

T
2

T
2

T
2

40 Gig Link

Traffic between 9500 Series Line Card and


9400 will use subset Hi gig links.
N9K-X9636PQ line card module is not
supported in 16 slot chassis

T2

T2

N9K-X9536PQ

16 slot Chassis Fabric Connectivity

With 3 FM configuration All 4 T2 units in


each FM are connected to 9500 series LC
modules' T2 units

T
2

T
2

T
2

T
2

T
2

T
2

T
2

T
2

Each blue line represents one 40 Gig link

T2

T2

N9K-X9536PQ

T
2

T
2

T
2

T
2

Line Cards With Mux to FM


FM25

FM26

HG
MUX1

FM24

HG
MUX2

HG
MUX4

012
3

456
MN7Port

75

35

6-8

911

2-

31-

2624

0
29
Warpcore

T2

012
3

02

Standby Mux Link

456
MN7Port

HG
MUX6

MF Port
911

75

2624

T2

Fabric Modules

Line Card

Line cards N9K-X9464PX/TX ,


N9K-X9564PQ/TX have Mux

Mux used for connecting HiG Link


from Line Cards to multiple Fabric
Module

Mux available only for Half of the


HiG interface of LC

By Default Mux Link Active to Odd


number of Fabric Module

8 9 10
11

Northstar 2

Active Mux Link

MF Port

FM21

HG
MUX3

HG
MUX5

8 9 10
11

Northstar 1
02

FM22

FM23

FM Connectivity For N9K-X9564PX With MUX


show system internal fabric connectivity mod 5 | in B

HiGIG Link-info Linecard slot:5

LC-Slot LC-Unit
5
5

LC-HGLink

MUX

FM-Slot

FM-Unit FM-HGLink

HG02

1B

25

HG12

HG03

1B

25

HG12

Line cards N9K-X9464PX/TX ,


N9K-X9564PQ/TX have Mux
Mux used for connecting HiG Link
from Line Cards to multiple Fabric
Module
Mux available only for Half of the
HiG interface of LC
By Default Mux Link Active to Odd
number of Fabric Module

show system internal fabric connectivity mod 5

HiGIG Link-info Fabriccard slot:5


LC-Slot LC-Unit LC-HGLink

MUX

With FM from Slot 25 Down

FM-Slot FM-Unit FM-HGLink

HG02

1A

26

HG14

HG03

1A

26

HG14

FM-25
T2-0 T2-1

FM-26
T2-0 T2-1

HG012

HG012

B
HG02

HG014

HG03

MUX
LC
T2-0 T2-1

HG014

FM Connectivity For N9K-X9564PX With MUX


show system internal fabric connectivity mod 5 | in B
HiGIG Link-info Linecard slot:5
LC-Slot LC-Unit
5
5

LC-HGLink

MUX

FM-Slot

FM-Unit FM-HGLink

HG02

1B

25

HG12

HG03

1B

25

HG12

FM-26
T2-0 T2-1

FM-25
T2-0 T2-1
HG012

HG012

HG014

HG014

HG03

HG02

MUX
LC
T2-0 T2-1

show system internal fabric connectivity mod 5

HiGIG Link-info Fabriccard slot:5


LC-Slot LC-Unit LC-HGLink

MUX

With FM from Slot 25 Down

FM-Slot FM-Unit FM-HGLink

HG012

HG012

HG02

1A

26

HG14

HG03

1A

26

HG14

FM-26
T2-0 T2-1

FM-25
T2-0 T2-1
B
HG02

HG014

A
HG03

MUX
LC
T2-0 T2-1

HG014

Fabric Troubleshooting commands


Fabric Module Slot-21

show system internal fabric connectivity mod 1


HiGIG Link-info Linecard slot:1
LC-Slot LC-Unit LC-HGLink MUX FM-Slot FM-Unit FM-HGLink

HG00

21

HG00

HG01

21

HG00

T2
#1

T2
#0
HG00

HG00

show system internal fabric connectivity mod 21


HiGIG Link-info Fabriccard slot:21
FM-Slot FM-Unit FM-HGLink

LC-Slot LC-Unit LC-HGLink

21

HG00

HG00

21

HG00

HG01

MUX

T2
#0

T2
#1

Line Card Slot-1

T2
#2

Fabric Port Drops and Link Status


N9K# show hardware internal fabric interface asic counters mod 21
Counters for Fabric Ports:
FabricInterface Forward
Forward Error
Pkt Error
Pkt
QOS Rx
RxDrops
TxDrops
RxDrops
TxDrops
Drops
0 / 1 / HG0
0
0
0
0
0
1 / 1 / HG0
0
0
1 0
0
0

QOS Tx
Drops
0
0

N9K# bcm-shell mod 21 "ps | inc hg0


ena/
port
hg0

speed/
link
up

link
duplex

scan

auto
neg?

42 FD

HW

No

STP
state

pause discrd

Forward None

lrn
ops

inter max loop


face frame back

FA

XGMII 16360

Fabric Port STP State HW point of View


N9K# sh vlan id 100
VLAN Name
Status
Ports
---- ------------------ --------- 100
VLAN0100 active
Po1, Eth1/1

N9K# bcm mod 21 " stg show


STG 5: contains 1 VLAN (100)
Forward: hg

show sys internal xbar event-history {trace|errors|msgs|sw}


show sys internal xbar-client event-history {trace|errors|msgs|sw}
show tech-support xbar

Path of the Packet -Inband


CPU

Netstack

System Controller-SC1

Mod29

Fabric Module

Mod23

Fabric Module

Fabric Module

Mod21

NIC-Eth3

NIC-Eth2

Traffic from all ingress Line Card


to Supervisor will hash to one
Fabric module
Traffic from Supervisor Card to
Egress Line cad will hash on one
FM. May not be same
CoPP is operational on all LC.
However aggregate CoPP is on
FM

Mod26

Eth6/1
Line Card

OSPF Hello

Check for Drops/Errors-Line Card

Line Card
North Star ASIC
Trident II
ASIC

Network Interfaces
N9K#show hardware internal interface ethernet 6/1 asic counters
Important Counters/Drops
--------------- --------- --------- --------- --------- --------- --------Interface Name
Forward
Forward Error Pkt Error Pkt
QOS Rx
QOS Tx
RxDrops
TxDrops
RxDrops
TxDrops
Drops
Drops
--------------- --------- --------- --------- --------- --------- --------Ethernet6/1
870
0
100
0
0
0

--------------- --------- --------- --------- --------- --------- --------Forward Rx Drops = [ RDBGC0 RDBGC4 RDBGC6 RDBGC7 RDBGC8 ]
Forward Tx Drops = [ TDBGC1 TDBGC3 TDBGC5 (excludes expected Multicast drops)]
ErrorPkt Rx Drops= [ IUNHGI IUNKOPC RFCS RALN RFLR RERPKT RJBR RSCHCRC RUND RMTUE]
ErrorPkt Tx Drops= [ TJBR TFCS TRPKT RMTUE TUFL TPCE ]
QOS Rx Drops
= [ RDISC DROP_PKT_ING DROP_PKT_IMTR DROP_PKT_YEL DROP_PKT_RED ]
QOS Tx Drops
= [ MCQ_DROP_PKT(0) MCQ_DROP_PKT(1) MCQ_DROP_PKT(2)
RDBGC0
Use slot <#> show hardware internal interface indiscard-stats instance <#>
N9K#bcm-shell mod 6 "listreg RALN"| grep Description
Description: Receive Alignment Error Frame Counter

Instant Buffer Usage Stats-With Buffer Usage


N9K#show hardware internal buffer info pkt-stats mod 6
INSTANCE: 0

Output Shared Service Pool Buffer Utilization (in cells)


SP-0

SP-1

SP-2

SP-3

------------------------------------------------------------------------ SP-3 Started filling


Total Instant Usage
4474
0
89
2939
the Queue
Remaining Instant Usage 25466
0
14255
3405
-----------------------------------------------------------------------ASIC Port

Q3

Q2

Q1

Q0

CPU

SPAN

[13]
UC(OOBFC)->

Only printed if there is congestion

UC->

1249

332

MC->

3247

1996

CPU buffer filling


up

CoPP Drops on Line Card


Line Card
North Star ASIC
Trident II
ASIC

Network Interfaces

N9K# show policy-map interface control-plane mod 6 class copp-system-p-classcritical | in ospf|trans|dropped


match access-group name copp-system-p-acl-ospf
transmitted 21898 packets;
dropped 0 packets;

Identify FM -Check CoPP Drops


N9K# show hardware internal cpu-mac inband active-fm traffic-to-sup
Active FM Module for traffic to sup:
0x00000015
Fabric Module in Slot 21 carry all traffic to Sup

N9K# show policy-map interface control-plane mod 21 class copp-system-p-classcritical | in ospf|trans|dropped


match access-group name copp-system-p-acl-ospf
match access-group name copp-system-p-acl-ospf6
transmitted 21898 packets;
dropped 0 packets;

Check for Drops/Errors-Fabric Module


Identify HG Port on LC and FM

N9K# show system internal fabric connectivity mod 6 | grep 21

LC-Slot
6

LC-Unit

LC-HGLink

HG10

MUX

FM-Slot

3B

FM-Unit

21

FM-HGLink

HG15

N9K# sh hardware internal fabric interface asic counters module 6 instance 0 asic-port 11

Important Counters/Drops
FabricInterface

0 / 11 / HG10

Verify Drops/Error on HG port on LC

Forward

Forward

Error Pkt

Error Pkt

QOS Rx

QOS Tx

RxDrops

TxDrops

RxDrops

TxDrops

Drops

Drops

N9K# sh hardware internal fabric interface asic counters mod 21 in 0

RxDrops
0 / 11 / HG15

TxDrops RxDrops
0

TxDrops
0

Drops
0

asic-port 16

Drops
0

Verify Drops Between FM and SC

System Controller
MVDXN-SW

module-21# show mvdxn internal port-status


Switch type: Marvell 98DXN11 - 10 port switch

Port
3

Descr

Enable Status ANeg

SC1EPCswitch Yes

UP

Fabric Module in Slot 21

Speed

Mode

No

MVDXN-SW
FABRIC CARD

InByte

OutByte

109548011

117051401

InPkts

OutPkts

274144

587285

10 port switch on System


controller and Fabric
module connect SC to FM

module-29# show mvdxn internal port-status


Switch type: Marvell 98DXN11 - 10 port switch System Controller in Slot 29

Port
7

Descr

FM1EPCswitch

Enable Status ANeg


Yes

UP

No

Speed
2

Mode
6

InByte

OutByte

InPkts OutPkts

746159513

60543666

620863

269592

Drops/Errors On Supervisor
N9K#show hardware internal cpu-mac inband counters
inb|dro

eth2

Link encap:Ethernet

in eth|ps-

HWaddr 00:00:00:01:1b:01

RX packets:2922013 errors:0 dropped:0 overruns:2 frame:0


TX packets:1652929 errors:0 dropped:0 overruns:0 carrier:0
eth3

Link encap:Ethernet

HWaddr 00:00:00:01:1b:01

RX packets:0 errors:0 dropped:0 overruns:0 frame:0


TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
ps-inb

Link encap:Ethernet

Netstack

HWaddr 00:00:00:01:1b:01

RX packets:54204 errors:0 dropped:3579 overruns:0 frame:0


TX packets:50626 errors:0 dropped:0 overruns:0 carrier:0

Pseudo Inband
NIC-Eth2

NIC-Eth3

Supervisor Card

Drops/Errors On Supervisor-Cont.
N9K#show hardware internal cpu-mac inband stats | in
Queue Idx

Packet Count

Bytes

Drops

errors|rate|Queue

Csum Errors

Allocation Failure

Queue 0

65429

580195964

Queue 7

65429

580195964

CRC errors ...................... 0


Alignment errors ................ 0
Symbol errors ................... 0
Carrier extension errors .........0
Rx packet rate (current/peak)

812 / 1097 pps

Tx packet rate (current/peak)

454 / 741 pps

Related show tech(s)


Nexus9500# sh tech-support inband
counters
Nexus9500# show tech-support pktmgr
Nexus9500# show tech-support <service>

L2 Mac And Vlan Table Verification


N9K# sh mac address-table dynamic vlan 100
Legend:

age - seconds since last seen,+ - primary entry using vPC Peer-Link,
True, (F) - False
VLAN
*

100

Eth6/1

* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC

MAC Address

Type

547f.ee1c.06fc

age

dynamic

Mac=547f:ee1c.06fc

interface Ethernet6/1

Secure NTFY Ports

(T) -

Eth6/1

switchport
switchport access vlan 100

N9K# bcm-shell mod 6 " l2 show"

| in Hit

mac=54:7f:ee:1c:06:fc vlan=100 GPORT=0x800800d modid=16 port=13/xe0 Hit

N9K# bcm-shell mod 6 "vlan show 100


vlan 100

ports xe0,hg ....... untagged xe0

no shutdown

Spanning Tree Verification


N9K# sh spanning-tree interface ethernet 6/1

Vlan
VLAN0100

Role Sts
Desg FWD

Cost
128.1537

Prio.Nbr

Type

P2p

VLAN.ipipe0[100]: <VP_GROUP_BITMAP=0x00000STG=0X67
FID_ID=0x64

FID_ID=Vlan ID.

Block: xe1-xe47
Forward: xe0,hg

no shutdown

N9K# Dec 0x67=103


STG= STP Group ID

N9K# bcm-shell mod 6 "stg stp 103


STG 103:

switchport
switchport access vlan 100

N9K# bcm-shell mod 6 "dump vlan 100

N9K# Dec 0x64=100

interface Ethernet6/1

Eth6/1
Mac=547f:ee1c.06fc

Unicast L3 Forwarding
T2 has combination of dedicated TCAM table space and shared hash table
memory known as Unified Forwarding Table (UFT)
The UFT is partitioned into three forwarding tables
MAC Address Table
IP Host Table
Longest Prefix Match-LPM Table
To maximize the system-wide forwarding scalability UFT tables on line
cards and fabric modules for different forwarding lookup functions
Feature

Scale

L3 LPM
Table

128K

FM

Feature

Scale
120K

LC

L3 Host Table
And L2/L3
Multicast

L2 Mac Table

96K

Unicast L3 Forwarding- Component Information


Theory of Operation
Software/Hardware Programming
OSPF communicates with uRIB to build the
routing table
AM builds the next-hop adjacency entry
uFDM distributes the information to the line
cards
IP FIB (running on the line cards) programs the
ASIC components with the forwarding and
adjacency information.

Remember: Software forwarding by the SUP is only


used for control and exception packets

OSPF

AM

ARP

uRIB

uFDM
Supervisor
Hardware-T2
FIB Manager

Forwarding Hardware

L3 Unicast Troubleshooting Flow


Next-Hop
Check the routing table

ARP/MAC
Check the ARP Table

Checking Route on
RIB And FIB.
Check Forwarding Route

Show ip route [ipv4] [<prefix>]


Show ip arp [ipv4]
show ip adjacency (Ipv4]
show forwarding adjacency platform [ipv4]
module <mod>

show forwarding [ipv4] route module <mod>

HW Programming
On LC/FM
Use BCM commands

bcm-shell mod 22 "l3 defip show"

Unicast L3 Forwarding- Two Possible Scenarios


Case 1: If incoming packet hit /32 host route on LC, forwarding decision is made on LC
Case 2: If incoming packet miss /32 host route on LC. Now for Longest Prefix
match (LPM) the packet get forwarded to FM
Install a default route 0.0.0.0/0 on Line Cards using the virtual MOD ID for Fabric Module
as the DMOD to force Line Cards to forward LPM packets to Fabric Modules
Fabric Modules perform LPM lookup and forward packets to the resolved Destination
MOD/Destination PORT
Also will verify How to Check ECMP Route

Network Diagram-Problem Definition


13.13.13.12/30
.13
.14

13.13.13.0/30
.1

.2

13.13.13.8/30
.9

.18
.17
13.13.13.16/30

Nexus3064Q-ESC#
N9K#

.10

N9508c-SJ#

Nexus3064Q-ESC# ping 13.13.13.10


PING 13.13.13.10 (13.13.13.10): 56 data bytes
Request 0 timed out
Nexus3064Q-ESC# traceroute 13.13.13.10
traceroute to 13.13.13.10 (13.13.13.10), 30 hops max, 40 byte
packets

1
2

13.13.13.2 (13.13.13.2)
* * *

1.124 ms

0.911 ms

0.752 ms

N9508d-SJ#

Router MAC Programming Check


Router Mac address must be programmed in Hardware
N9K1#show interface ethernet 6/1 | grep address
Hardware: 100/1000/10000 Ethernet, address: 003a.99fc.dd7f
N9K1# bcm-shell mod 6 "0:d chg my_station_tcam" | grep dd7f
MY_STATION_TCAM.ipipe0[0]: <VALID=1,------snip----MAC_ADDR=0x003a99fcdd7f,

Verify /32 Host Route on Line card-Case 1


N9K1#show ip route 13.13.13.14
/32 Host Entry
13.13.13.14/32, ubest/mbest: 1/0, attached
*via 13.13.13.14, Eth6/33, [250/0], 00:37:24, am
N9K1#bcm-shell mod 6 "0:l3 l3table show" | grep 13.13.13.14
Entry VRF IP address
Mac Address
INTF
MOD PORT CLASS HIT
10
1
13.13.13.14 00:00:00:00:00:00 100010
0
0
0
y

N9K1#bcm-shell mod 6 "0:l3 egress show"| grep 100010


Entry Mac
Vlan INTF
PORT MOD MPLS_LABEL
100010 88:f0:31:bf:ad:17 4095 4432
45
16
-1

ToCpu
no

Drop
no

N9K1#show system internal ethpm info interface ethernet 6/33 | grep -i STATIC
IF_STATIC_INFO: port_name=Ethernet6/33,if_index:0x1a284000,ltl=40875,slot=5,
nxos_port=32,dmod=16,dpid=45,

Next Hop Reached via L3-Port Channel


N9K1#show ip route 10.164.112.22
/32 Host Entry
10.164.112.22/32, ubest/mbest: 1/0
*via 13.13.13.14, Po200, [110/3], 00:09:33, ospf-10, intra
N9K1#bcm-shell mod 6 "0:l3 l3table show" | grep 10.164.112.22
Entry VRF IP address
Mac Address
INTF
MOD PORT CLASS HIT
175660 1 10.164.112.22 00:00:00:00:00:00 100012
0
0
0
y

N9K1#bcm-shell mod 6 "0:l3 egress show"| grep 100012


Entry Mac
Vlan INTF
PORT MOD MPLS_LABEL
100010 88:f0:31:bf:ad:17 665
4761
3t
0
-1

ToCpu
no

Drop
no

N9K1#show system internal ethpm info interface port-channel 200 |grep I STATIC
IF_STATIC_INFO: port_name=port-channel200,if_index:0x160000c7,ltl=2597,slot=0,
nxos_port=02,dmod=0,dpid=3,

Verify HW-Programming on LC or FM ? Case 2


N9K# show ip route 13.13.13.10

This is not /32 host Route.

IP Route Table for VRF "default

Packet forwarding decision


responsibility is of the Fabric
Module

13.13.13.8/30, ubest/mbest: 1/0


*via 13.13.13.6, Eth6/52, [110/41],
00:22:29, ospf-10, intra

ALL FM will be programmed

N9K# show forwarding route 13.13.13.10 module 21

with this Route

IPv4 routes for table default/base


Prefix

| Next-hop

13.13.13.8/30

Interface

13.13.13.6

| Labels

Ethernet6/52

Line Card Punting Packets to Fabric For LPM ?


N9K# show hardware internal forwarding adjacency statistics default-route mod 6
Module:6 Unit:0
Traffic matched adjacency for default route (destined to FM):
Unicast: Packets 148 Bytes 13382
N9K# bcm-shell mod 6 "0:l3 defip show"
Unit 0, Total Number of DEFIP entries: 12288
#
VRF
Net addr Next Hop Mac
INTF
VLAN
3072Override 0.0.0.0/0 00:00:00:00:00:00
149149 0

MODID PORT PRIO CLASS HIT


0

Mod 100 is assign to Fabric Module


N9K# bcm-shell mod 6 "l3 egress show" | inc 149149
Entry
Mac
Vlan INTF PORT MOD MPLS_LABEL ToCpu Drop
149149 00:12:12:12:12:12 4095 8189 1
100
-1
no
no

Longest Prefix Match on Fabric Module


N9K# bcm-shell mod 22 "l3 defip show" | grep 13.13.13.8
#

VRF Net addr

196620 1 13.13.13.8/30

Next Hop Mac

INTF

00:00:00:00:00:00 100008

MODID PORT PRIO CLASS HIT VLAN


0

N9K# bcm-shell mod 22 "l3 egress show" | grep 100008


Entry
100008

Mac
88:f0:31:bf:ad:17

Vlan

INTF

4095

4520

PORT MOD MPLS_LABEL ToCpu Drop


10

17

-1

no

no

Mac add used for rewrite


N9K# show system internal ethpm

info interface eth 6/52 | grep dmod

IF_STATIC_INFO:
port_name=Ethernet6/52,if_index0x1a286600,ltl=40856,slot=5,nxos_port=51,
dmod=17,dpid=10,unit=1,

ECMP Route Validation


N9K#show ip route 10.164.112.22
10.164.112.22/32, ubest/mbest: 2/0

Multi-Path

*via 13.13.13.14, Eth6/33, [110/5], 01:11:55, ospf-10, intra

*via 13.13.13.18, Eth6/34, [110/5], 01:11:55, ospf-10, intra


N9K#sh routing hash

13.13.13.2 10.164.112.22 mod 6

Hashing to path *13.13.13.18

N9K#bcm-shell mod 6 "l3


multipath show"
Multipath Egress Object 200256

Out Interface: Eth6/34

Interfaces: 100008 100010

N9K#bcm-shell mod 6 "0:l3 l3table show" | grep 10.164.112.22


Entry VRF IP address

Mac Address

INTF

17

00:00:00:00:00:00

200256

10.164.112.22

MOD PORT
0

CLASS HIT
0

n (ECMP)

Follow same steps demonstrated for /32 Host entry to learn about Interface in multipath show cli

Use Tools From Toolkit


ELAM- IF Line Card has North Star
module-6# debug platform internal ns elam asic 1
module-6(NS-elam)# trigger init egress in-select 3 out-select 5
module-6(NS-elam-insel3)# set outer ipv4 dst_ip 13.13.13.10

Packet Tracer- For All FM and LC having T2


N9K# test packet-tracer dst-ip 13.13.13.10 src-ip 13.13.13.1 detail-fp

Flex Counter- Check Adjacency hit counter


N9K# test hardware internal adjacency statistics nexthop ipv4 13.13.13.6
interface ethernet 6/52 enable
show tech-support forwarding l3 unicast

Consistency Checker
show consistency-checker

show tech-support adjmgr


forwarding

ipv4 unicast show tech routing unicast

Virtual Port-Channel-vPC

Allow a single device to use a port channel across


two upstream switches

Eliminate STP blocked ports

Dual-homed server operate in active-active mode

HSRP-Both active and standby peers forward


packets-ARP response by Active

Configuration steps Same as other Nexus


Products

Logical Topology with vPC

Case:1 All vPC Leg UP


Scenario: Traffic of a Host in Vlan 10 connected to Switch-A hash to N9K1 to reach Host in Vlan 20
connected to Switch-B
N9k1

vPC Peer Link =Eth1/1,4/1

N9k2

Keep Alive
PC1-PeerLink
SVI10
SVI-Mac 78da.6e71.9a3f
Standby 10.10.10.3

SVI10

MCT-1/1, 4/1
Eth6/20
Eth4/18

10.10.10.1/24

10.10.10.2/24

Eth6/20

Eth4/18

HSRP-Mac 0000.0c07.ac0a

Standby 10.10.10.3
HSRP-Mac 0000.0c07.ac0a

SVI20

vPC10

SVI-mac 78da.6e71.9a3f

vPC20

SVI20
SVI-mac 003a.99fc.dd7f

10.10.20.1/24
Standby 10.10.20.3

SVI-mac 003a.99fc.dd7f

Switch-A

Switch-B

10.10.20.2/24
Standby 10.10.20.3

HSRP-Mac 0000.0c07.ac14

HOST-A

Vlan-10

10.10.10.x/24

HOST-B

Vlan-20 HSRP-Mac 0000.0c07.ac14

20.20.20.x/24

vPC-Router MAC Programming Check


Both Active and Standby Peer responsible for L3 switching
Virtual Mac address must be programmed in Hardware on Both peers
Interface
Vlan10

Grp Prio P
10
100

State Active addr


Active 10.10.10.2

Standby addr
local

Group addr
10.10.10.3

N9K1# bcm-shell mod 4 "0:d chg my_station_tcam" | grep


VLAN_ID=0xa
VLAN_ID=0xa,VALID=1, MAC_ADDR=0xc07ac0a,

Interface
Vlan10

Grp Prio P State Active addr


10
100
Standby 10.10.10.2

Standby addr
local

Group addr
10.10.10.3

N9K2# bcm-shell mod 4 "0:d chg my_station_tcam" | grep


VLAN_ID=0xa
VLAN_ID=0xa,VALID=1, MAC_ADDR=0xc07ac0a,

vPC Peer Gateway Programming Check


Are N9Ks Configured with Peer-Gateway
N9K1-SJ# show mac address-table vlan 10 | in G
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
G
10 0000.0c07.ac0a static
F
F
vPC Peer-Link(R)
G 10 003a.99fc.dd7f static
F F
sup-eth1(R)
N9K2 SVI MAC
G
10 78da.6e71.9a3f static
F
F
vPC Peer-Link

N9K# bcm-shell mod 4 "0:d chg my_station_tcam" | egrep 0x003a99fcdd7f


MY_STATION_TCAM.ipipe0[0]:
<VALID=1,MAC_ADDR_MASK=0xffffffffffff,MAC_ADDR=0x003a99fcdd7f,KEY=0x00000000003
a99fcdd7f,IPV6_TERMINATION_ALLOWED=1,IPV4_TERMINATION_ALLOWED=1,DATA=0x38,ARP_R
ARP_TERMINATION_ALLOWED=1>

vPC Check For Traffic Ingressing Peer Link


Egress Block Mask
vPC Check-Traffic from Peer Link should Not L2/L3 Switch with local and remote
Legs up
N9K1# show vpc brief | grep Po
id Port
Status
Active vlans
1
Po1
up
10-20
id Port Status Consistency Reason Activevlans
10 Po10
up
success
success
10-20
20 Po20
up
success
success
10-20
N9K2# show vpc brief | grep Po
id Port
Status
Active vlans
1
Po1
up
10-20
id Port Status Consistency Reason Activevlans
10 Po10
up
success
success
10-20
20 Po20
up
success
success
10-20

N9k1

N9k2

Keep Alive
PC1-PeerLink
MCT-1/1, 4/1

Eth6/20

Eth4/18

Eth4/18

Eth6/20
vPC10
Switch-A

vPC20
Switch-B

vPC Check for Traffic Ingressing Peer Link (Contd)


N9K1#show port-ch summary |
Group Port- Type Protocol
1
Po1(SU) Eth
LACP
10
Po10(SU)Eth
LACP
20
Po20(SU)Eth
LACP

in Po
Member Ports
Eth1/1(P)Eth4/1(P)
Eth4/18(P)
Eth6/20(P)

N9K1# show system internal vpcm info mask


module 6
Masked ports for Module 6, Unit 0:
[Src Port None]: Eth6/20
[Src Port Eth1/1]: Eth6/20
[Src Port Eth4/1]: Eth6/20
Masked ports for Module 6, Unit 1:

N9k1

N9k2

Keep Alive
PC1-PeerLink
MCT-1/1, 4/1

Eth6/20
Eth4/18

Eth4/18
Eth6/20

vPC10
Switch-A

vPC20
Switch-B

Traffic Ingressing on Eth1/1 and


Eth4/1 will not exit Eth 6/20

ACL redirect logic for routed packets-vPC Leg Down

Redirect ACL installed to redirect routed packets for the


vPC for which local interface goes down

N9k1

Mac address learned from vPC points virtual port

PC1-PeerLink

MCT-1/1, 4/1

N9K1# show hardware access-list tcam region | grep vpc


VPC Convergence [vpc-convergence] size =

512

N9K1# sh mac address-table address30f7.0d9b.d401


VLAN MAC Address
20

Type

30f7.0d9b.d401 dynamic

age Secure NTFY Ports


0

vPC Peer-Link

N9k2

Keep Alive

Eth6/20
Eth4/18

Link Down

Eth4/18

Eth6/20

vPC10

Switch-A

vPC20

Switch-B

ACL redirect logic for routed packets-vPC Leg Down

On N9K1 traffic entering Eth6/20 after L3 switch


should egress Peer Link

N9k1

PC1-PeerLink

N9K2 Should not drop traffic entering Peer link and


forward traffic out to Eth 4/8

MCT-1/1, 4/1

Eth6/20

N9K# bcm-shell module 6 "fp show group 57


InPorts->L3Routable

N9k2

Keep Alive

Eth4/18

Eth4/18

Ln Down

Eth6/20

vPC10

vPC20

DstTrunk
Switch-A

Offset: 213 Width: 16


DATA=0x00008003

Trunk-id of 3 Down vPC

action={act=RedirectTrunk, param0=1(0x1)

Trunk-id of vPC Peerlink

Switch-B

ACL redirect logic for routed packets-Verify TrunkID


N9Ka# show system internal ethpm info int port-channel1 | grep dpid
IF_STATIC_INFO: port_name=port-channel1,if_index:0x16000000,ltl=2595,slot=95
dpid=1,unit=0,queue=0,xbar_unitbmp=0x0 ns_pid=0
N9508a-SJ# show system internal ethpm info int port-channel10 | grep dpid
IF_STATIC_INFO: port_name=port-channel10,if_index:0x16000000,ltl=2595,slot=95
dpid=3,unit=0,queue=0,xbar_unitbmp=0x0 ns_pid=0

show tech-support vPC


show tech-support cfs
show tech-support port-channel

Some important info to capture

ACL redirect logic for routed packets-Verify TrunkID


NX-OS -7.0(3)I1(2)
N9508a-SJ# show system internal access-list vpc-convergence mod 6
-----------------------------------------------------------VPC Convergence Entries
-----------------------------------------------------------Instance: 0
Trunk-id of 3 Down vPC
==========
Ingress:
Trunk-id of vPC Peerlink
---------Entry-ID DstTrunk-GID RedirectTrunk-GID Packet-Count
-----------------------------------------------------------------------1539
3
1
6082015

Nexus9000 Specific
Limitation and Goodies

Email from Nexus9000 To Cisco SR

Commands output directly sent to email address

Information from Nexus9000 Can be directly attached to Service Request.

Information is sent as body to email- not as attachment


N9K(config)# email
N9K(config-email)# smtp
N9K(config-email)# smtp-host 173.37.37.37
N9K(config-email)# from N9508a-sj@cisco.com
N9K(config-email)# smtp-port 25
show run | email subject <SR-number> attach@cisco.com

Bash Support !!!!


Goes beyond what standard CLI can provide
Customers demand more capabilities/freedom Creativity
Feature: bash-shell
User Role: dev-ops or network-admin or vdc-admin*
Strongly recommended: Some experience with shell/Linux-Use with

extreme care

Broadcom ASIC shell access on the Nexus 9000 !!!

The Nexus 9000 is based largely on the Broadcom Trident II ASIC-Known as T2

The modular unit Fabric Modules (FM) and Line Cards (LC) each contain multiple
instances of the T2 ASIC, as well as the TOR (top of rack) units

Access is provided to each and every instance of the T2 ASIC

No additional license is required to access the bcm-shell

Permitted by default role network-admin

Role based access control (RBAC) can be used to limit user access

Accounting log available for BCM activity

BCM Access some Examples

hg0

T2
Instance

N9K# bcm-shell mod 6 "show unit"

Unit 1 chip BCM56852_A2

Xe0

N9K# bcm-shell mod 6 "ps" | in 19


xe19 up
1G FD
SW Yes Disable None

N9K# show accounting log

Eth1/1

FA

XGMII

hg11

T2
Instance 1

F F F F F F F
F F F F FQSPF
P P P P P P P
P P P P PPorts
01 02 03 04 05 06 07 08 09 10 11 12

Unit 0 chip BCM56852_A2 (current)

hg0

hg11

Xe11
Eth1/12

F F F F F F F
F F F F FQSPF
P P P P P P P
P P P P PPorts
13 14 15 16 17 18 19 20 21 22 23 24

Xe0
Eth1/13

1582

| last 2

Mon Apr 20 08:31:52 2015:type=update:id=console0:user=admin:cmd=bcm-shell


module 6 "show unit" (SUCCESS)
Mon Apr 20 08:32:14 2015:type=update:id=console0:user=admin:cmd=bcm-shell
module 6 "ps" | in 19 (SUCCESS)

Xe11
Eth1/24

BCM Access some Examples (Contd)


N9K# bcm-shell mod 21 "config show l3"
l3_alpm_enable=2
l3_max_ecmp_mode=1
l3_mem_entries=16384
N9K# bcm-shell mod 4 "config show l2
l2xmsg_hostbuf_size=16384
l2_mem_entries=98304

Python !!!!

Python is - Established, Modern and Powerful, Clean, lots of libraries, liberal


license
Perl is available in gdb images only not available in final images
Tcl is there but no one uses it in NX-OS
The license that Python has (GPL-Like with very few restrictions on modification,
distribution and commercial use) make it very attractive to embed and distribute

On the box applications that can currently use Python scripts

Embedded Event Manager


Power On Auto Provisioning (POAP)
Create your own scripts that are like Super commands
Create your own command modifiers the things that act on commands applied with a
pipe |

Python-Continued

There are two Python environments on the N9000


One executed from VSH
One executed from Bash

Both run in their own forked process

The main differences comes from the environment that they get initialized into

These differences between them should be minimal

There is a sandbox that should primarily contain lower privileged users

Network-admin users get basically a pure 2.7.5 python environment

That sandbox mostly applies to lower privileged users, they may be prevented from doing certain things
in python

Also prevents file operations on files outside of bootflash

Python-Example
N9K# python
Python 2.7.5 (default, Oct

8 2013, 23:59:43)

switching between VSH and Python

[GCC 4.6.3] on linux2

Type "help", "copyright", "credits" or


"license" for more information.
>>>

switch between VSH and the

N9K# run bash python

Interpreter (Bash 1)

Python 2.7.5 (default, Oct

8 2013, 23:59:43)

[GCC 4.6.3] on linux2


Type "help", "copyright", "credits" or
"license" for more information.
>>>

Python Script Example

Why Patching?
Many customers spend extensive time and effort to test and qualify software prior to deployment. In todays
environments, if a defect is found, effectively root-caused, and integrated, since it is rolled out through a
maintenance release, customers would need to restart their qualification cycle, wasting time, and pushing out
deployment dates
Bug Found, Diagnose, Root
Cause

Begin Code Test &


Qualification Cycle

Maint. Released
Restart Qual Cycle

Defect Resolved, integrated


into Maint.

Actual Deployment

Target Deployment

6 Months

10 Months

NX-OS Image Patching


The Nexus9000 Standalone platforms introduces new patching capabilities that allows specific defects to be
rolled out in an independent package that can be applied to existing base software binaries. This will help
reduce customer code certification times, leading to greater customer satisfaction.
Bug Found, Diagnose, Root
Cause

Begin Code Test &


Qualification Cycle

Continue Qual
With additional tests

Defect Resolved, Patch


Released

6 Months

7 Months

Actual Deployment

Target Deployment

Patching Overview

NXOS platforms release major versions when introducing new features and engineering
special builds to provide bug fixes.

The new goal will be to allow customers to deploy patches for specific fixes only without
affecting the data plane of the device.

The patching architecture comes from IOS XR (SMU Software Maintenance Upgrade)
used to deliver Quick, Effective and Focused patches for specific sections of code.

Both binaries and libraries can be patched.

Supervisors and Line Card services can be patched.

Software patching will leverage process restart/reload or ISSU

Patch Uninstall Workflow - Detailed

User invokes install deactivate <patch_name>

System manager gracefully shuts down each impacted process

Softlinks are changed from active SMU to one in backup folder (if present).

Relevant SMU is removed from the /var/installer/activated/SMU directory.

System Manager triggers restart of impacted processes

(Optional) install remove deletes the patch from the local repository

CLI Commands Patch Install


Command

Syntax

Function

Notes

Install add

install add <uri> [activate]

Download patch from URI and add


patch to repository.

Only one patch can be added at


a time. Optionally can activate
patch in this step.

Install remove

install remove [<package> |


inactive]

User can remove only nonactivated patches

Confirmation y/n will be prompted

Install activate

install activate <package> [test]

Installs a patch from the local


repository. If not present, an error
will be returned.

Only one patch can be activated


at a time. No show commands
permitted during operation.

Install deactivate

install deactivate <package>

Uninstall patch and move it to nonactivated repository

Only one patch can be


deactivated at a time. **Patches
must have no other patch
dependencies

Install commit

install commit

Preserves all activated patches


across reloads.

Activated patches are committed


to a list kept in the patch
repository

CLI Commands Show Commands


Command

Function

Sample

show install request

Shows current install operation along


with time stamp, package name,
initiating user and % complete.

Fri May 10 09:06:55.921 UTC


Install operation 13 '(admin) install activate n9000-dk.6.0.2.U1.1.CSCuf08219.bin
Started by user 'cisco' via CLI at 09:06:48 UTC Fri May 10 2013
The operation is 10% complete

show install log [id | detail


| from | last | reverse]

Shows user information on previous


installation operations. Optional [detail]
command for verbose information.

Install operation 1 by user admin at Tue Sep 28 01:37:02 2004:


install commit
Operation completed successfully
Install operation 2 by user admin at Mon Oct 18 17:26:36 2004:
install add tftp://10.52.241.252/bcarter/n3000-uk9.6.0.2.U1.1.CSCuf08219.bin
Operation completed successfully
Install operation 7 by user lab at Mon Oct 18 17:31:13 2004:
install activate n3000-uk9.6.0.2.U1.1.CSCuf08219
Operation failed because service failed to come up.

show install active [onreload]

Displays boot images and active or


committed patches

switch# show install active


Boot Images:
Kickstart Image: bootflash:/n9000-dk.6.1.234.gbin
System Image: package:/isanboot/bin/images/sys

Active Packages:
n9000-dk.6.1.1.CSCui56298.bin

CLI Commands Show Commands (Contd)


Command

Function

Sample

show install inactive [onreload]

Shows patches in the repository not


yet activated

switch# show install inactive


Boot Images:
Image: bootflash:/inseor.6.1.1.234.gbin
System Image: package:/isanboot/bin/images/sys
Inactive Packages:
switch#

show install pkg-info


<package>

Shows details of a specific patch.


Requires that patch has been added
using install add first.

switch# show install pkg-info n9000-dk.6.1.1.CSCui56298.bin


Contents of Package file 'n9000-dk.6.1.1.CSCui56298.bin':
Expiry date : Jan 19, 2015 02:55:56 UTC
Uncompressed size : 17892613
Vendor : Cisco Systems
Desc : Bug Fix for CDET: CSCui56298
Build : Built on Wed May 10 08:04:58 UTC 2013
Source : By n9k-infra-bld
Platform: Nexus-9000.
Supersedes: n9000-uk9.6.1.1.U1.1.CSCuf09119, n9000-uk9.6.1.1.U1.1.CSCuf02229
Pre-requisite: n9000-uk9.6.1.1.U1.1.CSCuf09219
Restart information: BGP process restart.

Sample Patch Install Copy Patch to Switch


N9K# copy
scp://sdn@172.18.217.42/home/sdn/n9k/inseor_CSCuxP1fix.6.1.2.I1.2.CSCab00001.gbin
bootflash:
Enter vrf (If no input, current vrf 'default' is considered): management
sdn@172.18.217.42's password:
inseor_CSCuxP1fix.6.1.2.I1.2.CSCab00001.gbin
100% 233KB
232.7KB/s
00:01
Copy complete, now saving to disk (please wait)...
N9508#

N9508# dir | grep .gbin


238230
Jan 15 10:52:31 2014inseor_CSCuxP1fix.6.1.2.I1.2.CSCab00001.gbin
N9508#

Sample Patch Install Add patch to repository & verify


N9K# install add bootflash:inseor_CSCuxP1fix.6.1.2.I1.2.CSCab00001.gbin
Install operation 19 completed successfully at Wed Jan 15 10:55:14 2014
N9508#

N9K# show install packages


----------------------------------------------------------inseor_CSCuxP1fix.6.1.2.I1.2.CSCab00001.gbin inactive-commit
Modules
Module #27: inactive-commit
Module #28: inactive-commit
----------------------------------------------------------N9K# show install inactive
Inactive Packages:
inseor_CSCuxP1fix.6.1.2.I1.2.CSCab00001.gbin
N9K#

Important Limitations

For every Feature please review Guidelines and Limitations

Cisco Nexus 9000 Series NX-OS Verified Scalability Guide

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/6x/scalability/guide_34/b_Cisco_Nexus_9000_Series_NX-OS_Verified_Scalability_Guide_612I34.html

Only one software image (called nx-os) is required to load the Cisco NX-OS operating system.

EPLD Upgrade are recommended but are not mandatory

User Configured MAC address for SVI- Packets will not be flooded if Layer 2 Adjacency is missing

Diagnostic-The Port Loop back and Boot up Port Loop back tests are not supported

ASIC Memory-NS test is applicable only for the N9K-X9564PX and N9K-X9564TX line cards.

Priority flow control (PFC) is supported on Cisco Nexus 9500 Series switches with the N9KX9636PQ line card.

FEX is supported only on the Cisco Nexus 9372PX and 9396PX switches.

Cisco Nexus 9500 Series Switch can run in 8-queue mode only if all of its line cards are capable of
running 8-queue mode.

Participate in the My Favorite Speaker Contest


Promote Your Favorite Speaker and You Could Be a Winner

Promote your favorite speaker through Twitter and you could win $200 of Cisco
Press products (@CiscoPress)

Send a tweet and include


Your favorite speakers Twitter handle <Speakerenter your Twitter handle here>
Two hashtags: #CLUS #MyFavoriteSpeaker

You can submit an entry for more than one of your favorite speakers

Dont forget to follow @CiscoLive and @CiscoPress

View the official rules at http://bit.ly/CLUSwin

Complete Your Online Session Evaluation

Give us your feedback to be


entered into a Daily Survey
Drawing. A daily winner
will receive a $750 Amazon
gift card.

Complete your session surveys


though the Cisco Live mobile
app or your computer on
Cisco Live Connect.
Dont forget: Cisco Live sessions will be available
for viewing on-demand after the event at
CiscoLive.com/Online

Continue Your Education

Demos in the Cisco campus

Walk-in Self-Paced Labs

Table Topics

Meet the Engineer 1:1 meetings

Related sessions

Thank you

Backup Slides

Backup Slides !!!!

Fabric Module
Fabric Module for Nexus 9504
NFE

Fabric Module for Nexus 9508


NFE

NFE

Fabric Module for Nexus 9516

Chassis Type
NFEs per Fabric Module

Nexus 9504 Nexus 9508 Nexus 9516


1

NFE

NFE

NFE

NFE

Nexus 9500 Platform FRU- Line Card


Connect to Fabric Modules
12 x 42 Gbps

Fabric 2

Fabric 1
N
F
E

N
F
E

N
F
E

N
F
E

Fabric 3
N
F
E

Fabric 4
N
F
E

N
F
E

N
F
E

Fabric 5
N
F
E

Fabric 6
N
F
E

N
F
E

N
F
E

ALE 1
12 x 42
Gbps

1 x 42
Gbps

1 x 42
Gbps

NFE 1
18x 40
Gbps
Ethern
et

NFE

NFE

Network
Interfaces

NFE
18x 40Gbps

12 x 40 Gbps
12 x 40 Gbps
12 x 40 Gbps

Connect to Hosts or
Network

N9K-X9636PQ
FM1

FM2

HG Ports

FM3

FM4

HG Ports

FM5

FM6

HG Ports

T2

T2

T2

Instance 0

Instance 1

Instance 2

QSPF Ports

QSPF Ports

QSPF Ports

FP FP FP FP FP FP FP FP FP FP FP FP
01 02 03 04 05 06 07 08 09 10 11 12

FP FP FP FP FP FP FP FP FP FP FP FP
13 14 15 16 17 18 19 20 21 22 23 24

FP FP FP FP FP FP FP FP FP FP FP FP
25 26 27 28 29 30 31 32 33 34 35 36

N9K-X9464PX
FM2

FM3

FM4

MUX1-2

FM6
MUX3-4

HG Ports

T2

HG Ports

10G SFP+ Ports


FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

40G QSFP
FP FP FP FP
49 50 51 52

N9K-X9464TX
FM2

FM3

FM4

MUX1-2

FM6
MUX3-4

HG Ports

HG Ports

T2

100/1000/10000 T Ports
10G
PHY

10G
PHY

10G
PHY

10G
PHY

10G
PHY

10G
PHY

10G
PHY

10G
PHY

40G QSFP
10G
PHY

10G
PHY

10G
PHY

10G
PHY

FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

FP FP FP FP
49 50 51 52

N9K-X9432PQ
FM2

FM3

FM4

FM6

HG Ports

FP
01

FP
02

FP
03

FP
04

FP
05

FP
06

HG Ports

T2

T2

Instance 0

Instance 2

QSPF Ports

QSPF Ports

FP
07

FP
08

FP
09

FP
10

FP
11

FP
12

FP
13

FP
14

FP
15

FP
16

FP
17

FP
18

FP
19

FP
20

FP
21

FP
22

FP
23

FP
24

FP
25

FP
26

FP
27

FP
28

FP
29

FP
30

FP
31

FP
32

N9K-X9564PQ
FM6

FM4

FM5

HG MUX1

HG MUX2

HG MUX4

4567

0123

8 9 10 11

FM3

FM2
HG MUX3

HG MUX5

0123

7-5

4567

HG MUX6

8 9 10 11

MN Port

MN Port

Northstar 1

Northstar 2

MF Port
0-2

FM1

MF Port

3-5

6-8

9-11

0-2

9-11

2-0

31-29

26-24

7-5

26-24

Warpcore

T2

T2
40G QSFP

10G SFP+ Ports


FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

FP FP FP FP
49 50 51 52

N9K-X9564TX
FM6

FM4

FM5

HG MUX1

HG MUX2

HG MUX4

4567

0123

FM3

FM2
HG MUX3

HG MUX5

8 9 10 11

4567

0123

MN Port

Northstar 1

Northstar 2
MF Port

0-2

3-5

6-8

9-11

0-2

9-11

7-5

2-0

31-29

26-24

7-5

26-24

T2

T2
40G QSFP

100/1000/10000 T Ports
10G
PHY

10G
PHY

10G
PHY

10G
PHY

HG MUX6

8 9 10 11

MN Port
MF Port

10G
PHY

FM1

10G
PHY

10G
PHY

10G
PHY

10G
PHY

10G
PHY

10G
PHY

10G
PHY

FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

FP FP FP FP
49 50 51 52

Multicast L3 Forwarding

Before hardware can forward any Multicast packets,


forwarding information has to propagate from Sup to the LC
Several layers are to be verified:
MRIB (control-plane is created here)
MFDM PI /PD (platform independent & forwarding
information)
MFIB-IPFIB
IP FIB process programs hardware:
FIB Table contains (*,G) and (S,G) forwarding entries and RPF
information
GROUP table contains forwarding and pointers replication
information (pointers to MC VLAN)
MC VLAN tables contain replication information (~OIF lists)
Hardware (packets are forwarded here) & SDK

PIM

IGMP

Supervisor
MSDP

MRIB
MF DM

IP FIB

T2

Line Card

FIB Table MC VLAN Table


IPMC_GR

L2/L3 Multicast Packet Walk


Fabric Module

Lookup to resolve egr.


modules;
Sends one copy to each
egr. module;

Trident II
Lkup in Host Table
& L2 Table

L2/L3 mcast lookup;


Replicate pckts to local
receiving ports;
Send 1 copy to fabric
module;

Trident II

Trident II
IACL
Traffic
Classification&
Remarking

Egress Q

EACL

Parser

Network Interfaces
10GE

Egress Q
EACL

L2/L3
Lookup &
Pkt rewrite
Parser

L2/L3
Lookup &
pkt rewrite

Examines ingress
packet. Get packet
headers for
processing.

IACL
Traffic
Classification
& Remarking

Network Interfaces

40GE

10GE

40GE

Lookup for local


receiving ports;
replicate pkts onto
those ports.

Multicast L3 Forwarding-MRIB
N9K# show ip mroute 239.10.10.10 shared-tree

Supervisor

IP Multicast Routing Table for VRF "default

(*, 239.10.10.10/32), uptime: 00:23:32, ip pim


Incoming interface: Ethernet6/1, RPF nbr:
13.13.13.1
Outgoing interface list: (count: 1)

Ethernet6/52, uptime: 00:22:42, pim

PIM

IGMP

MRIB
IP FIB

MF DM

MSDP

Multicast L3 Forwarding-mFDM PI-Supervisor


N9K# show forwarding distribution ip multicast route group 239.10.10.10 source
13.13.13.14 | in 13|Index
(13.13.13.14/32, 239.10.10.10/32), RPF Interface: Ethernet6/1, flags:
Outgoing Interface List Index: 1
N9K# show forwarding distribution multicast outgoinginterface-list l3 1
Outgoing Interface List Index: 1
Reference Count: 4

Supervisor
PIM

IGMP

MSDP

Platform Index: 0xb00001

MRIB
MF DM

Number of Outgoing Interfaces: 1 t6/52

IP FIB

Multicast L3 Forwarding IPFIB-Line card


N9K# show forwarding ip multicast route group 239.10.10.10 source
13.13.13.14 mod 6 | inc 239|Eth
(13.13.13.14/32, 239.10.10.10/32), RPF Interface: Ethernet6/1, flags:
Outgoing Interface List Index: 1
Outgoing Interface List Index: 0x1
Ethernet6/52

T2

Line Card

FIB Table MC VLAN Table


IPMC_GR

Mod 6 is N9K-X9564TX
To reach Ethernet 6/52 which is on NS from front port of T2,Packets need to
cross Fabric module

Multicast L3 Forwarding Entries on LC BCM Shell


N9K# bcm-shell mod 6 "ipmc table show"
SRC IP ADDRESS

MC IP ADDRESS

MC GROUP

VID VRF COS

13.13.13.14

239.10.10.10

0x2000007

75680

no

0.0.0.0

239.10.10.10

0x2000007

86578

no

N9K#bcm-shell module 6 "mc show group=0x2000007"


Group 0x2000007 (L3)

HWIDX CLASS HIT

T2

Line Card

FIB Table MC VLAN


Table IPMC_GR

port hg0, encap id -1


-------snip-----------port hg11, encap id -1

Traffic spared to Hig towards Fabric

Multicast L3 Forwarding Entries on LC BCM Shell


N9K# bcm-shell mod 6 " search l3_entry_ipv4_multicast group_ip_addr=0xef0a0a0a
source_ip_addr=0x0d0d0d0e
L3_ENTRY_IPV4_MULTICAST.ipipe0[75680]:
SOURCE_IP_ADDR=0xd0d0d0e,
GROUP_IP_ADDR=0xef0a0a0a,

L3MC_INDEX=7

T2

Line Card

FIB Table MC VLAN Table


IPMC_GR

N9K# bcm-shell mod 6 " dum chg l3_entry_ipv4_multicast 75680


show tech-support multicast`
IPV4MC:EXPECTED_L3_IIF=0x112e,
show tech-support forwarding multicast
N9K# show system internal eltm info interface ethernet 6/1 | in LIF
cr_flags = INTF LIF , LIF = 4398 (0x112e), LTL = 40959 (0x9fff) (S 0x0, P 0x0)

IGM Snooping
Forwarding programming in vPC Scenario

IGMP Process Provides both Layer 3 IGMP Processing , and Layer 2 IGMP snooping functionality

Receivers use IGMP (Internet Group Management Protocol) to report their multicast group
Membership to router

Layer 2 IGMP Snooping functions of IGMP process include processing snooped multicast router
Packets Including IGMP reports and leaves sent by receiver

Once the group membership is learned , the Supervisor Engine informs I/O modules , which

program Hardware

This will Constrain data-plane multicast packets to only those ports with multicast routeror interested
receivers in HW

IGMP Snooping continued

BCM on FM are in Mode 4. This will have L2 Table size of 32K & L3 Host Table 16K

L3 Host table will be used to program (*,G) /(S,G) entry. This will will accommodate
maximum of 8K entry.

MFDM sends two OIF List information to MFIB. One for LC (S,G) OIF List and other for
FM ( Mac, Group) OIF List in PIM disable Vlan.

MFIB will use (S,G) OIF list to program LC and Mac Group to Program FM in 32K L2 Table.

If PIM is enable FM can accommodate 8K(VRF, S,G) and will program Hardware.

Address aliasing is possible because on FM we use L2 table to program Mac Group information

IGMP Snooping (Contd)

With vPC IGMP will have knowledge of multi chassis Ether Channel trunk (MCT) interface.

When one of the vPC peer receives IGMP join , it will sync up this with peer over MCT link
using cFS-Cisco Fabric Services over Ethernet .

Duplication of traffic crossing MCT is avoided using Port block Mask

VPC Support PIM-SM Only

For source in VPC domain dual Forwarders are used

For Source in Layer 3 Cloud , Unicast best metric determines active forwarder

VPC Operational Primary in case of tie. CFS used to negotiate active Forwarder role

Configuration-IGMP Snooping enable by default


Nexus9508-13# sh ip igm snooping vlan 103
IGMP Snooping information for vlan 103
IGMP snooping enabled
Lookup mode: IP
Optimised Multicast Flood (OMF) enabled
IGMP querier present, address: 10.10.103.5, version: 2, i/f Po30
Nexus9508-13# sh ip igm snooping vlan 100
IGMP Snooping information for vlan 100

IGMP snooping enabled


Lookup mode: IP
Optimised Multicast Flood (OMF) enabled
IGMP querier present, address: 192.168.100.2, version: 2, i/f Vlan100

Querier interval: 125 secs


Querier last member query interval: 1 secs

Reference Topology for Troubleshooting


vPC Keep Alive

N9508-12

vPC Peer Link PO-10


Eth 3/1-2

N9508-13
Eth 3/1-2
Eth 6/9/1-4

Eth1/3/1-4

vPC30
Eth 1/17-18 ,Eth 1/33-34

N93k
Eth 1/48

Ixia 10/2-Source

vPC 35
Eth 1/17,Eth 1/19 , Eth 1/33-34

N35K
Eth1/48

Ixia 10/1-Receiver

IGMP Snooping Troubleshooting


Stream will enter one of the VPC-Peer , Which will get forwarded across Peer link to other VPC Peer
Both boxes will have (S ,G)
Upon Creation of (S,G) , VPC Peers negotiate best metric
Both realize source is VPC-Connected

Install Entry as Win-Force


If either peer gets a PIM/IGMP Join for the given source , they both add Interface to OIF
Nexus9508-12(config)# sh ip pim internal vpc rpf-source

Nexus9508-13# sh ip pim internal vpc rpf-source

PIM vPC RPF-Source Cache for Context "default" - Chassis


Role Primary

PIM vPC RPF-Source Cache for Context "default" - Chassis


Role Secondary

Source: 192.168.100.10

Source: 192.168.100.10

Pref/Metric: 0/0

Pref/Metric: 0/0

Source role: primary

Source role: secondary

Forwarding state: Win-force (forwarding)

Forwarding state: Win-force (forwarding)

MRIB Forwarding state: forwarding

MRIB Forwarding state: forwarding

vPC Peer receiving Join

IGMP Join from one of the receiver enter one of the VPC Pee.
This Peer encapsulates IGMP in CFS , sends to other Peer
Both Peer have identical State
Both Peer install OIF
Data traffic flows down to Receiver, also forwarded to other Peer on Peer Link
Other Peer drop the packet either by PORT BLOCK MASK blocking or no OIF

Nexus9508-ESC-12# sh ip mroute 239.10.10.10 192.168.100.10

Nexus9508-ESC-13# sh ip mroute 239.10.10.10 192.168.100.10

IP Multicast Routing Table for VRF "default"

IP Multicast Routing Table for VRF "default"

(192.168.100.10/32, 239.10.10.10/32), uptime: 01:00:09, ip pim


mrib

(192.168.100.10/32, 239.10.10.10/32), uptime: 04:25:36, ip pim


mrib

Incoming interface: Vlan100, RPF nbr: 192.168.100.10, uptime:


01:00:09, internal

Incoming interface: Vlan100, RPF nbr: 192.168.100.10, uptime:


04:25:36

Outgoing interface list: (count: 1)

Outgoing interface list: (count: 1)

Vlan101, uptime: 00:59:40, mrib

Vlan101, uptime: 02:04:41, mrib

Nexus9508-ESC-12#

Nexus9508-ESC-13#

Step to verify PI On Supervisor. Verify on Both Peers


Nexus9508-ESC-12# sh ip igmp groups 239.10.10.10
IGMP Connected Group Membership for VRF "default" - matching Group "239.10.10.10"
Type: S - Static, D - Dynamic, L - Local, T - SSM Translated
Group Address
239.10.10.10

Type Interface
D

Vlan101

Uptime

Expires Last Reporter

00:01:23 00:02:56 192.168.101.13

Nexus9508-ESC-12#
Nexus9508-ESC-13# sh ip igmp groups 239.10.10.10
IGMP Connected Group Membership for VRF "default" - matching Group "239.10.10.1
0"
Type: S - Static, D - Dynamic, L - Local, T - SSM Translated
Group Address
239.10.10.10

Type Interface
D

Nexus9508-ESC-13#

Vlan101

Uptime

Expires Last Reporter

00:01:18 00:03:01 192.168.101.13

CFS Provide info


Nexus9508-ESC-12# sh ip igmp snooping groups vlan 101 detail
IGMP Snooping group membership for vlan 101
Group addr: 239.10.10.10

Nexus9508-ESC-13# sh ip igm snooping groups vlan


Group ver: v2 [old-host-timer: not running] 101 det
Last reporter: 192.168.101.10
IGMP Snooping group membership for vlan 101
IGMPv2 member ports:
IGMPv1/v2 memb ports:
Po35 [1 GQ missed], cfs:false, native:true

Group addr: 239.10.10.10


Group ver: v2 [old-host-timer: not running]
Last reporter: 192.168.101.10

vPC grp peer-link flag: exclude

IGMPv2 member ports:

M2RIB vPC grp peer-link flag: exclude

IGMPv1/v2 memb ports:

Nexus9508-ESC-12#

Po35 [0 GQ missed], cfs:true, native:false


vPC grp peer-link flag: exclude
M2RIB vPC grp peer-link flag: exclude
Nexus9508-ESC-13#

Verifying Multicast forwarding Distribution Module


Platform Independent On Supervisor
Nexus9508-ESC-12# sh forwarding distribution multicast route group 239.10.10.10
source 192.168.100.10
(192.168.100.10/32, 239.10.10.10/32), RPF Interface: Vlan100, flags:
Received Packets: 1073 Bytes: 36977
Number of Outgoing Interfaces: 2
Outgoing Interface List Index: 10
Vlan100
( Mem L2 Ports: port-channel10 )

Not showing PC 10 for Vlan 101 because of


exclude flag seen while checking igmp
snooping stats.

Vlan101
( Mem L2 Ports: port-channel35 )
Note: On shutting down local vpc only, igmp does not send update to mfdm/ipfib to update the mroute state.
That is why you did not see mfdm/ipfib removing local vpc. So if local leg of vPC is down we will still PC in the above output.

Verifying Multicast forwarding Distribution Module


Platform Independent On Supervisor-(Contd)
Nexus9508-12# sh forwarding multicast route group 239.10.10.10 source 192.168.100.10 mod 1

(192.168.100.10/32, 239.10.10.10/32), RPF Interface: Vlan100, flags:


Received Packets: 1111 Bytes: 72215
Outgoing Interface List Index: 9
Number of next hops: 2

Outgoing Interface List Index: 9


Vlan: 101
port-channel35
bridged Vlan

port-channel10
Hardware Outgoing Interface List Index: 33554443

Verifying Multicast forwarding Distribution Module


Platform Independent On Supervisor-IGMP-Snooping
Nexus9508-12# sh forwarding distribution ip igmp snooping vlan 101 group 239.10.10.10 det
Vlan: 101, Group: 239.10.10.10, Source: 0.0.0.0
Outgoing Interface List Index: 4
Reference Count: 1

Nexus9508-13# sh forwarding distribution ip igmp snooping vlan 101 group

Platform Index: 0xa00004

239.10.10.10 det

Vpc peer link exclude flag set

Vlan: 101, Group: 239.10.10.10, Source: 0.0.0.0

Number of Outgoing Interfaces: 2

Outgoing Interface List Index: 5

port-channel10

Reference Count: 1

port-channel35

Platform Index: 0xa00005

Vpc peer link exclude flag set


Number of Outgoing Interfaces: 2
port-channel10
port-channel35

Verifying Multicast Forwarding Distribution Module


Platform Independent On Supervisor-Snooping Group.
Nexus9508-12# sh forwarding distribution l2 multicast mac-based vlan
101
Vlan: 101, Group: 0100.5e0a.0a0a, Source: 0000.0000.0000
Outgoing Interface List Index: 3
Reference Count: 1

Platform Index: 0xa00003

Nexus9508-13# sh forwarding distribution l2 multicast mac-based vlan 101

Vpc peer link exclude flag set

Vlan: 101, Group: 0100.5e0a.0a0a, Source: 0000.0000.0000

Number of Outgoing Interfaces: 2

Outgoing Interface List Index: 8

port-channel10

Reference Count: 1

port-channel35

Platform Index: 0xa00008


Vpc peer link exclude flag set
Number of Outgoing Interfaces: 2
port-channel10
port-channel35

IPFIB on LC for IGMP Snooping programming.


Nexus9508--12# sh forwarding multicast route group 239.10.10.10 source 192.168.100.10 mod 1
(192.168.100.10/32, 239.10.10.10/32), RPF Interface: Vlan100, flags:
Received Packets: 5708 Bytes: 371020
Outgoing Interface List Index: 5
Number of next hops: 2
Outgoing Interface List Index: 5
port-channel30 (Vlan: 101)
port-channel10 (bridged)
Hardware Outgoing Interface List Index: 33554441

Nexus9508-13# sh forwarding multicast route group 239.10.10.10 source 192.168.100.10 mod 6


(192.168.100.10/32, 239.10.10.10/32), RPF Interface: Vlan100, flags:
Received Packets: 6798 Bytes: 441870
Outgoing Interface List Index: 19
Number of next hops: 2
Outgoing Interface List Index: 19
port-channel30 (Vlan: 101)
port-channel10 (bridged)
Hardware Outgoing Interface List Index: 33554437

Verifying Hardware Programming


Nexus9508--12# bcm-shell mod 1 "mc show group=33554441"

If we see encap id a positive #

Executing mc show group=33554441 on bcm shell on module 1

then it is LIF

Group 0x2000009 (L3)


port hg0, encap id 400005

If we see encap id = -1 then it is

port hg1, encap id 400005

L2 bridge copy.

port xe10, encap id 21


port xe11, encap id 21
Nexus9508-12# bcm-shell mod 3 "mc show group=33554441"
Executing mc show group=33554441 on bcm shell on module 3
Group 0x2000009 (L3)
port hg0, encap id 400005
port xe0, encap id -1
port xe1, encap id -1

Nexus9508-12# sh system internal eltm info interface vlan 101 | in LIF


cr_flags = INTF VLAN , LIF = 21 (0x15), LTL = -1 (0xffffffff) (S 0x0, P 0x0)
Nexus9508-ESC-12#

From BCM to check what is HW index for given Group


Static entry of Mcast group
Hit Bit indicate flow is present
Mcast Index is where the traffic need to bridge
Nexus9508-12# bcm-shell module 1 "l2 show" | in MCast
mac=01:00:5e:0a:0a:0a vlan=101 GPORT=0x0 modid=0 port=0 Static Hit MCast=33554435
mac=01:00:5e:0a:0a:14 vlan=100 GPORT=0x0 modid=0 port=0 Static MCast=33554435
Nexus9508-12# sh ip igmp gr vlan 100

show tech-support ip igmp snooping


show tech-support ip multicast