BRKDCT 3101 PDF

Nexus9000(Standalone)
Architecture And
Troubleshooting
Shridhar V. Dhodapkar –Technical Leader (Services)
CCIE 6367 (Routing & Switching)
BRKDCT-3101
Session Abstract
This session presents briefly the architecture of the latest generation

of Nexus 9000 Series Modular switches. Topics include supervisors,
fabrics, I/O modules, forwarding engines, and physical design elements, as
well as the Top of the Rack Nexus9300 Switches.
The session will also cover how to monitor the health of the system.
We will walk you through in depth troubleshooting Tools and Techniques.
3
Session Goal
• To provide an overall understanding of the Nexus 9000 switching
architecture, supervisor, fabric, and I/O module design, packet flows, and
key forwarding engine functions
• This session will introduce System Telemetry, Troubleshooting tool Kits

and troubleshooting case scenarios
• This session will not examine NX-OS software architecture or other Nexus
platform architectures
4
Related Sessions
BRKARC-2222 - Cisco Nexus 9000 architecture
BRKARC-3471 - Cisco NX-OS Software Architecture
BRKDCT-3144 - Advanced - Troubleshooting Cisco Nexus

7000 Series Switches
5
Agenda
• Introduction
• Architecture
• System Health check Telemetry
• Troubleshooting Toolkit
• Nexus 9000 Troubleshooting
• Common Link Layer Issues-L1
• Fabric Connectivity and
• In band
• L2/L3 Packet Forwarding
• vPC
• Nexus9000 Specific Limitation and Goodies
6
Introduction
7
Introduction-What is Nexus9000 Family ?
Nexus 9500 Series Switches Nexus 9300 Series Switches
Nexus9504/Nexus9508/Nexus9516 N9K-C9332PQ N9K-C9372PX N9K-C9372TX N9K-C9396
8
Architecture
9
9500 Field Upgradeable Units (FRU)
• 9500 has the following modular components which can upgraded or
replaced in the field Nexus® 9508 Front View Nexus® 9508 Rear View
• Supervisor
• Fabric Module
• Line Card
• System Controller
• Fan Tray
• Power Supply
• The Supervisor, System controller ,Fabric Module and LC have OBFL

(On-Board Failure Logging) for failure analysis
10
Nexus 9500 Platform FRU
Supervisor Module-What it is Role
• Redundant Half-width supervisor engine
• Common for 4-, 8-, and 16- slot chassis
• External Clock Input (PTP)
• Responsible for control-plane functions
System Controller-What it is Role

• Offload supervisor from internal device management tasks
• Central Point of Chassis Control
• EOBC Switch (Ethernet Out of Band Channel)
• EPC Switch (Ethernet Protocol Channel)
• Power Supplies via SMB (System Management Bus)
• Fan Trays
11
Nexus 9500 Platform Line Card
Line Cards FM6 FM5 FM4 FM3 FM2 FM1
• I/O module with Merchant and
Merchant+ ASIC HG
MUX1
HG
MUX4
HG
MUX2
HG
MUX5
HG
MUX3
HG
MUX6
• Have Various Forwarding Tables

• L2 Mac Table And L3 Host Table
• ACL and Buffers for Queuing 01
23
45
MN6Port
7
89
10
01
23
45
MN6 Port
7
89
10
Northstar 1 11
Northstar 2 11
ASIC Name 0
MF Port
3 6- 9- 0
MF Port
9-
NFE=Network Forwarding Engine-Trident 2(T2)

-
2
-
5
8 1
1 HG -
2
1
1
ALE=Application Leaf Engine-North Star(NS) 7

-
2
Warpcor
- 1-
3 2
6-
7
-
2
6-
-Donner
5 0
T2 2
e 9
2
4
N9K-X9564PQ 5
T2 2
4
Note: Internal ports are called as Hi-Gig/HG ports 40G

QSFP
10G SFP+ Ports FP FP FP FP
F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F 49 50 51 52
P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 2425 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
12
Nexus 9500 Fabric Module
Fabrics Modules
• Interconnect Line Card slots
• Installed at the rear of the chassis
• Leverages Broadcom Trident II ASICs
• Max 1.92 Tbps per line card slot (6 Fabric Cards)
• 960 Gbps per line card slot (3 Fabric Cards)
• All Fabric Cards are active and carry traffic
• Fan Tray requires Fabric Card to be present in even slot
Trident II Trident II
ASIC-NFE ASIC-NFE
32 x 40G 32 x 40G
Hi-Gig2 Hi-Gig2
13
Nexus 9500 Fabric Module
Data Plane Scaling for 8-Slot Chassis
• An 8-Slot chassis fabric module can provide up to 320Gbps to each Line Card slot
• With 6 fabric modules, each Line Card slot can have up to 1.92Tbps duplex
forwarding bandwidth
Fabric 1 Fabric 2 Fabric 3 Fabric 4 Fabric 5 Fabric 6
T2 T2 T2 T2 T2 T2 T2 T2 T2 T2 T2 T2
320 Gbps 320 Gbps 320 Gbps 320 Gbps 320 Gbps 320 Gbps
(8x 40Gbps) (8x 40Gbps) (8x 40Gbps) (8x 40Gbps) (8x 40Gbps) (8x 40Gbps)
320 Gbps
640 Gbps Line Card Slot
960 Gbps
1.28 Tbps
1.60 Tbps
1.92 Tbps
14
Distributed Data Plane of Nexus 9500 Series Switches

Nx NFE Nx NFE Nx NFE Nx NFE Nx NFE Nx NFE
N = 1 for N9504
N = 2 for N9508
2 x 42 Gbps N = 4 for N9516

2 x 42 Gbps
ALE ALE ALE ALE
12 x 42 Gbps 12 x 42 Gbps 12 x 42 Gbps 12 x 42 Gbps
NFE NFE NFE NFE
Note: Internal ports are called as Hi-Gig/HG ports
15
Nexus9500 Series Line Card Summary
Information X9600 Series Line X9500 Series Line X9400 Series
Cards Cards
ASIC Technology Merchant only Merchant+ Merchant only
N9K-X9636PQ N9K-X9564PX •N9K-X9432PQ
N9K-X9564TX •N9K-X9464PX
N9K-X9536PQ •N9K-X9464TX
Number of ASIC 3 T2 2 T2 + 2 NS 2 T2 40 gig 32 Ports

2 T2 + 2 NS 1 T2 48 1/10 gig , 4
2 T2 + 2 NS QSFP
Non Blocking Non Blocking Line rate > 200 byte

packet
Buffer Size 36 MB 104 MB 12 MB with one T2
24 MB with two T2
16
High Level Block Diagram-N9500
All PSU, SC, SUP, FM, and

LC plug into the same
Power Supply Interface
17
N9K-C9300 Series
• Fixed Chassis
• Port QSFP+ Uplink Module
• 1 RU or 2RU or 3RU
• AC/DC Power Supply
• Front-to-Back & Back-to-Front Airflow

Expansion Module
• Latency: 1-2 usec
• Wire-Speed L2/L3 Forwarding
• Switch will not boot up without GEM
18
Nexus 9300 Series Switch Summary
N9396TX/PX N93128TX N9372TX N9372TX N9372PX
NFE (BCM T2) 1 1 2 1 1
ALE ( NorthStar)/GEM GEM-1 NS GEM-1 NS No GEM-1 No GEM -1 No GEM- 1
Donner Donner Donner
Oversubscribed No 1.5:1 No No No
Line Rate Yes Yes (packets > Yes Yes
194-Bytes)
QoS Classes 8 4 8 4 4
Buffer (MB) 36 (12*3) 104 (12*2+40*2) 24 (12*2) 104 (12*2+40*2) 104 (12*2+40*2)
19
High Level Block Diagram-N9300
GEM 4x 40GE QSFP+ Uplinks
NorthStar
16GB Total
DIMM2
PCIe
DDR3
Northstar ASIC 1 CPU
2C 1.5GHz
Egress Ingress
• The last 2/3 numbers stand for
12 x 40G
(12+12)x12 (12+12)x12
Hi-Gig2 total bandwidth in Gigabits
1000BaseT • 93128 – 128G (96 x 10G + 8 x 40G)
Trident II Mgmt Port
ASIC • 9396 – 96G (48 x 10G + 12 x 40G)
2 USB
BRCM Trident2 Ports • 9372 – 72G ( 48 x 10G + 8 x 40 G)
12 x 40G
48 10G
Ethernet x 12 40G eUSB
Boot Flash
Network Interfaces 12C
Front Panel 48x 1GE/10GE Ports
20
Nexus9500 Unicast Packet Flow
Fabric Module
Fabric Module
Performs L3 LPM
L3 LPM Lookup & Forwarding
lookup and resolves
Egress port and
Ingress Line Card Egress Line Card
next-hop
ALE-NS EoQ ALE-NS EoQ
Additional buffer is
Classify traffic T2-NFE OOBFC T2-NFE OOBFC available for
Ingress Signaling Ingress
based on 802.1q Signaling extended out put
Accounting & Accounting &
COS, IP Pres, Output Q Ques EoQ
Policing Policing Output Q
DSCP &ACL & Shaping
Remark if needed Traffic Traffic & Shaping
Classification Classification
E-ACL
& Remarking & Remarking E-ACL Class-based output
L2/L3 Lookup in queues. Support 6
I-ACL I-ACL classes including
MAC Table and IP Packet Packet
Host Table L2/L3 Modification L2/L3 Modification control traffic class
Lookup & Lookup &
forwarding forwarding
Parse the first 128 Egress Line card
Byte and extract Parser sends packet to
Parser
header info egress port based
on DMOD/DPORT
Network Interface Network Interface
21
N9K-C9300 High Level Block Diagram
(12 x 40G) = 480G

Bandwidth to GEM
(16 x 10G) x 3 =
Module
480G FP Bandwidth
HiGiG2 Interface on T2
MACF ports on the GEM and to MACN ports
(12 x 40G) = 480G FP

Bandwidth Uplink Ports
MACN ports.
22
Main Features of Trident2 1280Gbps Switch ASIC
Features Information
DCB Engine
Maximum IO and Core bandwidth 1280G
L2/L3
Content aware Engine Multicast MAC(L2) Entries 32K min -288K max
L2 L2/L3 L3 Hosts IPv4:16K min-112Kmax

L3
IPv6:8K min-56 max
MAC Processing Route
L3 Multicast Group 8K
Dynamic Memory Packet
Counters
Manager Buffer
Virtual Ports 16K
128 Integrated SerDes Host IF Maximum number of Physical ports 104
128 SERDES@10Gbps
OR
32 SERDES@40Gbps
23
North Star
Features Information
Support Mixed Speed but in Fixed Network Interface:12 Ports Fabric
configuration. Interface: 12 40 Gig
Forwarding 720Mpps lookup rate on Ingress

Datapath
720Mpps lookup rate at Egress
Datapath
Shared Memory Subsystem
Ingress Path Buffer 10 Mbytes
Egress Path Buffer 30 Mbytes
Maximum number of Physical ports 24
24
Broadcom Unified Forwarding Table
T2 has the following Unified Forwarding Table:
SUPPORTED COMBINATIONS
Mode L2 L3 Host LPM
0 288K 16K 16K
1 224K 56K 16K
2 160K 90K 16K
3 98K 122K 16K
4 32K 16K 128K
25
Routing Mode for Nexus9300
LPM Routing Mode Broadcom T2 CLI Command
Mode
Default system routing mode 3
ALPM Routing mode 4 System routing max-mode l3
N93K#show system routing mode

Configured System Routing Mode: Hierarchical
Applied System Routing Mode: Hierarchical (Default)
N93K#show hardware internal forwarding table utilization module 1
Max Host Route Entries (shared v4/v6): 124928
Max LPM Table Entries : 16384
26
Routing Mode for Nexus9500
LPM Routing Mode Broadcom T2 Mode Cli Command
Default System routing mode 3 (For Line card)
4 (For Fabric Module)
Max-host routing mode 2--Line Card- V6 in LPM System routing max-mode host
3--For Fabric Module
Nonhierarchical routing mode 3--For Line Card System routing non-hierarchical
4--With max-l3-mode option Option [max-l3-mode]
For Line card
No Routes on Fabric Module
64-bit ALPM routing mode Sub mode of mod 4 for System routing mode hierarchical
Fabric modules 64b-alpm
show hardware internal forwarding table utilization mod 1

Max Host Route Entries (shared v4/v6):16384 Non hierarchical
Max LPM Table Entries : 131072 routing mod
show hardware internal forwarding table utilization mod 21
Max Host Route Entries (shared v4/v6): 0
Max LPM Table Entries :0
27
ACL TCAM TABLE
Characteristic
• Ingress ACL: 4K TCAM entries - 4x 512 banks + 8x 256 banks
• Egress ACL: 1K TCAM entries - 4x 256 banks
• Each ACL type needs its own dedicated bank/banks
• IPv4, IPv6 or MAC each needs dedicated bank/banks
• MAC-ACL IPv6 & any QOS needs double-width entries, which means needs at least 2 banks
• VACL is programmed symmetrically in both egress and ingress ACL

Interface Ingress ACL Egress ACL
Type
SVI TCAM Shared TCAM Not shared
L3 TCAM Shared TCAM Shared

28
ACL Characteristics
• Atomic/hitless update of existing applied ACL while modified
• Temporary label swap (no use of default-result)
• Two acl copies in tcam, if there is no enough space, process fails
• ACL TCAM banks chaining not supported
• L4OPs/LOUs only used for expansion beyond 5 lines, configurable
• 10 L4op per acl limit
• Specific applications (dhcp, bfd) may install their own ACLs which must merge
with user configured racl, vacl, pacl
29
TCAM Carving for Nexus 9000
TCAM Region-N9500 Size Per Region
Ingress TCAM Region-N9300 Size Per Region Ingress
IPV4 RACL 1536 IPv4 PACL 512 512
3X512
IPV4 VACL 512 512
IPv4 L3 QOS 256
256
IPV4 RACL 512 512
Ingress System 256
256 IPv4 Port QOS 256 256
SPAN 256 256 Ingress System 256 256
Ingress CoPP 256
256 SPAN 256 256
Redirect 256 Ingress CoPP 256 256
256
Redirect 256 256
vPC Convergence 512
512 512
vPC Convergence 512
Egress IPv4 768 Egress Egress IPv4 RACL 256 Egress
RACL 256
3X256 Egress IPv4 VACL 512 512
Egress System 256 Egress System 256
256 256
30
ACL TCAM Default Region and Carving
• TCAM Banks will first get assigned to Feature which has largest region.
• Next TCAM Bank will get assigned to Feature which need double Width.
• TCAM Carving requires Line Card/TOR reload to take effect
• To read current TCAM allocation

N9K#Show system internal access-lists global
• To reconfigure TCAM Region

N9K(config)hardware access-list tcam <feature name> <size>
31
Buffer And Queuing-T2
• T2 has 12 Mbytes of
Buffer shared by all
ports for all Traffic
• Shared buffer divided Into Control
Control and default service
pool if module is T2 only Control
Shared Shared OOBFC
• Shared buffer divided Buffer Buffer
into Control, default and 12 MB Default 12 MB
Default
OOBFC service Pool if
Module is T2 and NS
based
Module with T2 only Module with T2 And NS
OOBFC: Out of band flow control unicast service pool
32
Buffer And Queuing-North Star
• North Star has 40 Mbytes of Buffe
GEM 4x 40GE QSFP+ Uplinks
10 MB
Buffer
NorthStar
20 MB
Buffer • Divided in to Three Pool
ASIC 1
10 MB
Buffer • Control , SPAN , Default
12 x 40G
Hi-Gig2
Control
Trident II SPAN
ASIC Shared
Buffer
12 x 40G Default
Ethernet
Front Panel 48x 1GE/10GE Ports
33
Buffer Boost Function with T2 and NS
Fabric Module
• Buffer boost is function which allow T2 to use extra
buffer of NS
ALE-NS
10 MB 20 MB • When Buffer boost is enabled on a port , T2 Local switch
Buffer 10 MB Buffer
Buffer traffic is Sent to NS for extra buffer space-
• When Buffer boost is disabled on a port, T2 local traffic
NFE 12 MB Buffer Shared by all to this port remains local on this NFE
T2 ports
• Buffer Boost is enabled by default and can be disabled
on a per port basis
Network Interface
1/10GE 1/10GE 1/10GE 1/10GE
34
System Health check Telemetry
35
Most Common System Health Check
• What is the Best Recommended NX-OS Release
• CPU & Memory usage
• Inter Process Messaging usage-MTS
• Traffic Stats/Drop To CPU
• CoPP/Hardware Rate Limiter Drops
• Ethernet Out of Band Drops/Error • Interface Errors for STP/Error disable
• Instant Buffer usage Stats • Inter ASIC Utilization
• FATAL System Errors • Hardware Capacity Check
• Consistency Checkers –Various Tables
• GOLD Diagnostic Checks
• Sev1/2 Syslog
36
General Recommendation for New and Existing
Deployments
• Software Recommendation
Platform Series Recommended Release
Cisco Nexus 9500 6.1(2)I3(5), 7.0(3)I1(3), or 7.0(3)I2(2a)*
Cisco Nexus 9300 6.1(2)I3(5), 7.0(3)I1(3), or 7.0(3)I2(2a)*
Cisco NX-OS 7.0(3)I2(x) is the long-lived release train for the Cisco Nexus 9000 Series
switches.
http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/recommended_releaseb_Minimum_and_Recommended_
Cisco_NXOS_Releases_for_Cisco_Nexus_9000_Series_Switches.html
• Verified Scale limits for different features and protocol for each release
http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/6-
x/scalability/guide_34/b_Cisco_Nexus_9000_Series_NXOS_Verified_Scalability_Guide_612I34/b_Cisco_Nexus_9000_Series_NXO
S_Verified_Scalability_Guide_612I34_chapter_01.html
37
CPU & Memory Usage
N9K#show system resources
Load average: 1 minute: 0.00 5 minutes: 0.03 15 minutes: 0.05
Processes : 432 total, 1 running D
CPU states : 2.76% user, 0.75% kernel, 96.48% idle R
CPU0 states : 0.00% user, 0.00% kernel, 100.00% idle CPU
A
M
CPU1 states : 0.00% user, 1.01% kernel, 98.98% idle
Memory usage: 16402328K total,3443588K used, 12958740K free
Current memory status: OK
D
R
N9K#show system internal memory-usage-per-module in-KB A
Slot 01:Used:1647420 Kbytes,Free:425680 Kbytes,Total:2073100 Kbytes M
Slot 02:Used:1627524 Kbytes,Free:445576 Kbytes,Total:2073100 Kbytes
Slot 04:Used:1647560 Kbytes,Free:425540 Kbytes,Total:2073100 Kbytes
N9K#show system internal memory-alerts-log Make sure log is clean
38
CPU & Memory Usage
Provides top process using CPU cycle

show processes cpu sort | head lines 12
PID Runtime(ms) Invoked uSecs 1Sec Process
----- ----------- -------- ----- ------ -----------
3357 220 3100 7099 45.50% adjmgr Possibly ARP Table Churn
5853 31655 10181 3109 0.50% ipqosmgr
5859 9489 52308 181 2.00% diag_port_lb
3477 672 3107 216 0.50% netstack
3478 268 175 1535 0.50% ospf
39
Top Command-display top CPU processes
“top” provides an ongoing look at processor activity in real time
N9K#run bash Auto update

bash-4.2$ top
top - 11:13:32 up 9 days, 3:34, 4 users, load average: 0.11, 0.11,
0.08
Tasks: 226 total, 1 running, 220 sleeping, 0 stopped, 5 zombie
Cpu(s): 0.8%us, 0.2%sy, 0.0%ni, 98.5%id, 0.0%wa, 0.1%hi,
0.3%si, 0.0%st
Mem: 16402328k total, 3445044k used, 12957284k free, 72676k buffers
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

1 root 20 0 2152 620 556 S 0 0.0 0:08.05 init
2 root 20 0 0 0 0 S 0 0.0 0:00.00 kthreadd
3 root 20 0 0 0 0 S 0 0.0 0:00.58 ksoftirqd/0

40
Inter Process Messaging Usage
N9K#sh sockets client detail | inc pim|drops|Errors

select drops: 10 Make Sure Drops/Errors not incrementing
Errors:
select drops: 0
Errors:
select drops: 0
Errors:
Message and transaction service-MTS
N9K#sh system internal mts buffers sum | diff
For SAP 320 own
node sapno recv_q pers_q npers_q log_q
by “OSPF” npers_q
sup 320 0 0 4592 0
sup 284 0 19 0 0 increasing
sup 250 2 0 0 0
41
Inband Driver Statistics-CPU Drops
N9K#show hardware internal cpu-mac inband stats
eth2 stats:
RMON counters Rx Tx
total packets 601163425 318962431
Per Queue Stats
Queue Idx Packet Count Bytes Drops Csum Errors Allocation Failure
Queue 0 17677525 111822449180 0 0 0
- - - - - - - - - - - - - - -SNIP- - - - - - - - - - - - - - - - - - - - - - - -
Queue 7 17677525 111822449180 0 CRC 0errors/Collisions/late
0 Collisions
Interrupt Counters Alignment errors
Rx overrun 0 Symbol errors
Sequence errors/Rx jabbers
Error counters
RX errors/Rx length errors
Rate statistics
Rx packet rate (current/peak) 717 / 80695 pps
Tx packet rate (current/peak) 360 / 1338 pps
42
Traffic Stats/Drops to CPU— (Cont’d)
N9K# show system internal frame traffic | in drops
Global input drops: bad-interface 0, bad-encap 0, failed-decap 0,
Global output drops: Drops From PKTmgr
eth_output_err 0, gre_err 0 otv_err 0 span_drop_en: 0 span_drops: 0
Crossbar down drops : 0 Flood_to_core LTL: Hits: 0 Misses: 0
N9K# show system inband queuing statistics | in drop

bpdu: recv 68, drop 0, congested 0 rcvbuf 2097152, sndbuf 262142 no drop 0
(q0): recv 1249377, drop 0, congested 0 rcvbuf 2097152, sndbuf 262142 no drop
(q1): recv 4138154, drop 0, congested 0 rcvbuf 2097152, sndbuf 262142 no drop
43
Instant Buffer Utilization For CPU Port
show hardware internal buffer info pkt-stats cpu • Total 48 Queues

[Q00-07] 0 0 0 0 0 0 0 0 • Each Line Display Cell utilized
[Q08-15] 0 0 0 0 0 0 0 0 for 8 queues
[Q16-23] 0 0 0 0 0 0 0 0 • One Cell represent approximately
[Q24-31] 0 0 0 0 0 0 0 0 208 Bytes
[Q32-39] 0 0 0 0 0 0 0 0 Congestion encountered if Counters
[Q40-47] 0 0 0 0 0 0 0 0 keep incrementing
44
Ethernet Out Of Band Drops/Errors
N9K#show hardware internal eobc stats | inc dropped
RX packets:248308217 errors:0 dropped:0 overruns:0 frame:0
TX packets:71554006 errors:0 dropped:0 overruns:0 carrier:0
N9K# show system internal emon stats

EMON MOD ONLINE BMP: 37f00067
FSM ID: 0 EOBCMON
=======================================
HB tx_req 186396
module 1: Provides Stats for all Modules
rx_req 176410 including Fabric module
rx_resp 176426
rx_miss 7 Heart bit miss
tx_resp 176410
45
Instant Buffer Usage Stats
Buffer polling interval for 7.0
N9K#show hardware internal buffer info pkt-stats mod 1 Release is 500msecs
INSTANCE: 0
---------------------------------------------------------- • Instant Buffer utilization per queue
per port
Output Shared Service Pool Buffer Utilization (in cells)
• One cell represents 208 bytes
SP-0 SP-1 SP-2 SP-3
----------------------------------------------------------- Show hardware internal buffer info pkt-
stats input mod 1
Total Instant Usage 4474 0 89 2939
Remaining Instant • SP-3-Dedicted resource for Control
Usage 25466 0 14255 3405 Traffic
Peak/Max Cells Used 4821 0 327 3060 • SP-0-Resource for Locally Switched
Switch Cell Count 29940 0 14344 6344 Unicast ,Multicast and SPAN
---------------------------------------------------------- • SP-2 Extended Output queue for
show hardware internal ns buffer info pkt-stats Unicast using buffers from North
Star
46
Instant Buffer Usage Stats - With Buffer Usage
N9K#show hardware internal buffer info pkt-stats mod 1
INSTANCE: 0
SP-0 SP-1 SP-2 SP-3
-------------------------------------------------------------------------
Total Instant Usage 4474 0 89 2939 • SP-3 Started filling
Remaining Instant Usage 25466 0 14255 3405 the Queue
-------------------------------------------------------------------------
ASIC Port Q3 Q2 Q1 Q0 CPU SPAN
[13] Port 13 onwards are Front Panel Port Only printed if there is congestion
UC(OOBFC)->0 0 0 0
UC-> 0 0 0 1249 332 0 • CPU buffer filling
MC-> 0 0 0 3247 1996 0 up
47
CoPP Drops
N9K# show policy-map interface control-plane mod 1 | in dropped
dropped 0 packets;
dropped 0 packets;
dropped 0 packets;
dropped 0 packets;
dropped 7800 packets; Drops Seen for Default-Class at minimal rate is normal
We recommend that you use the strict default CoPP policy initially and then later modify the CoPP
policies based on the data center and application requirements.
Parameters Default
Default policy Strict
Default Policy 9 policy entries
48
CoPP Drops-Exception drops
class-map copp-system-p-class-l3uc-data (match-any)
match exception glean
class-map copp-system-p-class-redirect (match-any)
match access-group name copp-system-p-acl-ptp
class-map copp-system-p-class-exception (match-any)
match exception ip option Goal is to Classify all Traffic Using CoPP
match exception ip icmp unreachable
match exception ipv6 option
match exception ipv6 icmp unreachable
class-map copp-system-p-class-exception-diag (match-any)
match exception ttl-failure
match exception mtu-failure
49
Hardware Rate Limiter
N9K# show hardware rate-limiter mod 1
Units for Config: packets per second
Allowed, Dropped & Total: aggregated since last clear counters
Module: 1
R-L Class Config Allowed Dropped Total
+----------+-----+------------+------------+-------------+
L3 glean 100 0 0 0
L3 mcast loc-grp 3000 0 0 0
access-list-log 100 0 0 0
bfd 10000 1352890 0 1352890
fex 3000 0 0 0
span 50 0 0 0
50
FATAL System Errors
N9K#show logging onboard mod 1 exception-log | incl FATAL prev 15
------------------------------------------------------------------------
Date (mm/dd/yy)=01/15/15 Time (hs:mn:sec): 00:16:58
OBFL Exception log data for THIS SUP Module:0
********* Exception info for module 0 ********
exception information --- exception instance 1 ----
Device Name : System Manager
Device Errorcode : 0x0000023a
ErrNum (devInfo) : 58 (0x3a)
System Errorcode : 0x401e0089 Service in VDC has had a hap-reset
Error Type : FATAL error
51
Common Interface Error counters and Status
N9K# show interface counters errors mod 4
Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize OutDiscards
--------------------------------------------------------------------------
Eth4/1 0 100 0 581 0 0
N9K# show interface status err-disabled
Port Name Status Reason
--------------------------------------------------------------------------
Eth4/1 err-disable link-flap
52
Interface Queuing Stats
N9K#show queuing interface 4/18
Egress Queuing for Ethernet4/18 [System]
QoS-Group# Bandwidth% PrioLevel Shape Qlimit
Min Max Units
3 1 - - - 6(D)
-------------------------SNIP--------------------------
0 100 - - 6(D)
----------------------------------------------------
QOS GROUP 0
Unicast | OOBFC Unicast | Multicast
Dropped Pkts | 0| 0| 0|
------------------------------------------------------------
QOS GROUP 7
Unicast | OOBFC Unicast | Multicast
Dropped Pkts | 0| 0| 0|
53
Inter ASIC Utilization-HG Ports
T2 #0 T2 #1
HG00 HG00
Fabric Module
T2 #0 T2 #1 T2 #2
N9K#show system internal interface counters mod 1 Line Card
Internal Port Counters (150 secs rate) for Slot: 1

====================================================
Interface ASIC ASIC BCM TxBitRate(BwUtil) TxPktRate RxBitRate(BwUtil) RxPktRate
Port Inst Port (bps) (pps) (bps) (pps)
-----------------------------------------------------------------------------------------
ii1/1/1 HG0 0 1 170512 (0.00) 0 0(0.00) 0
-------------------------------------------Snip------------------------------------------
ii1/1/14 HG1 1 2 0( 0.00) 0 1129882872(2.51) 960753
ii1/1/25 HG0 1 1 1790648 (0.00) 1043 22864(0.00) 20
54
Verify Consistency Between Software and Hardware
Table
Table CLI
Physical Interface show consistency-checker link-state
Port-Channel show consistency-checker membership port-channels

Membership
Mac Address Table show consistency-checker l2
Vlan Membership show consistency-checker membership vlan
L3 interface-LIF L3 interface-LIF programming –Logical Interface for Routing

programming
For RIB and FIB show consistency-checker forwarding ipv4 unicast
55
Consistency Checkers-Link and STP state
N9K#show consistency-checker link-state mod 1
Link State Checks: Link state only
Consistency Check: PASSED
No inconsistencies found for:
Ethernet1/1
2015 Mar 24 03:23:27 N9508a-SJ %$ VDC-1 %$ vshd: CC_LINK_STATE: Consistency
Check: PASSED
N9K# show consistency-checker stp-state vlan 18

Checks: Spanning tree state
2015 Mar 24 03:25:21 N9508a-SJ %$ VDC-1 %$ vshd: CC_VLAN_STP_STATE:
56
Consistency Checkers-Port Channel-Vlan Membership
N9K#show consistency-checker membership port-channels
Checks: Trunk group and trunk membership table.
Consistency Check: Failed
Inconsistency found for port-channel1:
Module:1, Unit: ['Ethernet3/49', 'Ethernet2/49']
Module:26, Unit: ['Ethernet3/49', 'Ethernet2/49’]
N9K# show consistency-checker membership vlan 18
Checks: Port membership of Vlan in vlan and egr_vlan table
Ports configured as "switchport monitor” will be skipped
Vlan:18, Hardware state consistent for:
Ethernet2/49
2015 Mar 24 03:28:31 N95a%$ VDC-1 %$ vshd: CC_VLAN_MEMBERSHIP: Consistency
Check: PASSED
57
Consistency Checkers-Mac address Table
N9K# show consistency-checker l2 module 1
Consistency check: PASSED
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen, + - primary entry using vPC Peer-Link,
(T) - True, (F) - False
Missing entries in the HW MAC Table
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
Extra and Discrepant entries in the HW MAC Table
---------+-----------------+--------+---------+------+----+------------------
58
Consistency Checkers-L3 Interface
N9K# show consistency-checker l3 mod 1
L3 LIF Checks: L3 Vlan, CML Flags, IPv4 Enable

No inconsistencies found for:
Ethernet1/1
Ethernet1/2
Ethernet1/3
2015 Mar 24 04:07:27 N9508a-SJ %$ VDC-1 %$ vshd: CC_L3_LIF: Consistency Check:
PASSED
59
Consistency Checker –Unicast Forwarding
N9K#test consistency-checker forwarding
Consistency check started.
N9K# show consistency-checker forwarding ipv4 unicast module 1
IPV4 Consistency check (in progress): table_id(0x1) slot(1)
Elapsed time : 8257 ms
N9K# show consistency-checker forwarding ipv4 unicast module 1
IPV4 Consistency check : table_id(0x1) slot(1)
Execution time : 13244 ms ()
No inconsistent adjacencies.
No inconsistent routes.
Consistency-Checker: PASS for 1
60
Gold Diagnostic Checks
N9K# show diagnostic result mod 2
On Demand Diagnostic can be executed
Module 2: 48x1/10G-T 4x40G Ethernet Module
Test results:(.=Pass, F=Fail,I=Incomplete,U=Untested,A=Abort,E=Error disabled)
1) ASICRegisterCheck------------> .
2) PrimaryBootROM---------------> .
3) SecondaryBootROM-------------> .
4) OBFL-------------------------> .
6) BootFlash--------------------> . RewriteEngineLoopback
7) AsicMemory-------------------> .
8) FpgaRegTest---------------- -> .
9) PortLoopback:--------------- > .
Port 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
-----------------------------------------------------
U U U . U U U U . . U U U . U U
61
Sev1/2 Syslog
show logging logfile | incl -1-|-2-
2015 Feb 25 10:30:17 N9508a-SJ %PLATFORM-2-MOD_PWRUP: Module 26 powered up
(Serial number SAL1738D37W)
2015 Feb 25 10:32:37 N9508a-SJ %XBAR-2-XBAR_HGLINK_NOT_UP: fabric link 1 on
module 2 unit 0 connected to fabric module 26 unit:0 is not up during module
bring up
2015 Feb 25 10:32:39 N9508a-SJ %MODULE-2-MOD_FAIL: Initialization of module 26
(Serial number: SAL1738D37W) failed
2015 Feb 25 10:32:39 N9508a-SJ %PLATFORM-2-MOD_PWRDN: Module 26 powered down
(Serial number SAL1738D37W)
62
Troubleshooting Toolkit
63
Troubleshooting Toolkit
• Ethanalyzer
• TCP Dump
• ELAM
• Packet Tracer
• Flex Counter
• ERSPAN
• Consistency Checkers
64
Ethanalyzer-When To Use it
• To Analyze the traffic sent and received by CPU Netstack
• It uses wireshark’s code (an open source software)

Pseudo Inband
• Troubleshooting High CPU SUP

NIC-ETH2
• Troubleshoot Control Plane issues Ex. OSPF , PIM , STP
Flap.
Note: Ethanalyzer does not allow capturing of hardware switched traffic between data
ports of the switch
65
Ethanalyzer-CLI
N9K# ethanalyzer local interface inband capture-filter "pim” detail
Capturing on inband
Frame 1 (60 bytes on wire, 60 bytes captured)
Arrival Time: Mar 24, 2015 10:01:10.018889000
-------Snip------------------
[Protocols in frame: eth:ip:pim]
N9K#ethanalyzer local interface inband display-filter "ospf” detail

Capturing on inband
Some Available Options
Frame 1 (82 bytes on wire, 82 bytes captured)
Arrival Time: Mar 24, 2015 10:04:11.425523000 autostop :Autostop
-------------------Snip-------------------- decode-internal :Internal
header decoding
[Frame is marked: False]
limit-captured-frames :Maximum
[Protocols in frame: eth:ip:ospf] number of
66
TCP Dump
• Tcpdump command works on most flavors of Linux operating system
• Helps to prints out a description of the contents of packets on a network interface
• Tcpdump will, if not run with the -c flag, continue capturing packets until it is
interrupted by a SIGINT signal –CTRL-C
• Tcpdump output can be saved to file for further reference
• More info at http://www.tcpdump.org/
67
Tcpdump -syntax
N9K# show feature | in bash Syntax: tcpdump -h

Feature Name Instance State tcpdump version 4.1.1
bash-shell 1 enabled libpcap version 1.2.1
N9K# run bash Usage: tcpdump [-aAbdDefIKlLnNOpqRStuUvxX] [ -B size ] [ -c count]
bash-4.2# sudo su [ -C file_size ] [ -E algo:secret ] [ -F file ] [ -G seconds]
Password:****** [ -i interface ] [ -M secret ] [ -r file ]
bash-4.2# whoami [ -s snaplen ] [ -T type ] [ -w file ] [ -W filecount ]
root [ -y datalinktype ] [ -z command ] [ -Z user ]
bash-4.2# tcpdump –c 10 –I ps-inb [ expression ]
bash-4.2#
68
Tcpdump-Examples-
bash-4.2# tcpdump -c 100 -w tcpdump.pcap -vvvv -i ps-inb Capturing 100 packets And
tcpdump: WARNING: ps-inb: no IPv4 address assigned writing to file
tcpdump: listening on ps-inb, link-type EN10MB (Ethernet), capture size 65535
bytes
100 packets captured
102 packets received by filter
bash-4.2#cd /bootflash
bash-4.2# tcpdump -tttt -r tcpdump.pcap | more
reading from file tcpdump.pcap, link-type EN10MB (Ethernet) Reading captured file
2015-04-26 03:21:31.309350 00:0e:ee:01:1b:01 (oui Unknown) > 00:00:00:ff:ff:01
(oui Ethernet), ethertype Unknown (0x8833), length 160:
0x0000: 0000 fc08 0b00 0000 0000 0800 0000 0ffd ...............
-------------------------------------more---------------------------------
69
tshark
bash-4.2$ tshark -i ps-inb
Capturing on inband
0.000000 00:0e:ee:01:1b:01 -> 00:00:00:ff:ff:01 0x8833 Ethernet II
12.328377 00:0e:ee:01:1b:01 -> 00:00:00:ff:ff:01 0x8833 Ethernet II
^C2 packets captured
bash-4.2$
70
Elam-Embedded Logic Analyzer Module-NS
• Elam Allows to capture single packet based on Trigger
• Triggers are configured using Packet information
• Only Supported on North Star Based(ALE) Line Cards and GEMs
• Use with TAC Supervision
• Help to Answer following Questions
• Was the Packet indeed Received by device on given Line card?
• How did the Packet Look like?
• How was the packet rewritten based on forwarding Decision made by T2?
• Was the Packet correctly forwarded or Dropped?
71
ELAM Configuration
• Init – Initialize the ELAM – select the Asic instance, pipeline and
1. Init select lines
module-1# debug platform internal ns elam asic
module-1(NS-elam)# trigger init ingress in-select 3 out-select 5
2. Config • Config – Configure the trigger based on different fields in the packet
module-1(NS-elam-insel3)# set outer ipv4 src_ip 13.13.13.10
• Arm – Arm the trigger by setting the fields to match in hardware

3. Arm module-1(NS-elam-insel3)# start
Trigger
• Read – Once the trigger is triggered, read the report
4. Read module-1(NS-elam-insel3)# report
• Reset – Once the process is complete, reset the trigger to restart

the process
5. Reset module-1(NS-elam-insel3)# reset
72
Elam Ingress & Egress Direction-TOR
• Traffic entering GEM ports which has NS and GEM 4x 40GE QSFP+ Uplinks
exiting T2 is Egress Pipeline
Egress Ingress
Ex. trigger init egress in-select 3 out-select 5 NorthStar

ASIC 1
set outer ipv4 dst_ip 13.13.13.10
12 x 40G
Hi-Gig2
• Traffic Entering T2 and Exiting GEM ports is Trident II

ASIC
Ingress Pipeline
12 x 40G
Ethernet
Network Interfaces
Ex. trigger init ingress in-select 3 out-select 5
set outer ipv4 src_ip 13.13.13.10 Front Panel 48x 1GE/10GE Ports
IP.Add=13.13.13.10
73
Elam Ingress & Egress Direction-EOR
• Traffic entering from Fabric Module in to NS of
Line Card is Egress Pipeline Fabric 3
Fabric 1
N N
Ex. trigger init egress in-select 3 out-select 5 Egress FE FE
Ingress
set outer ipv4 dst_ip 13.13.13.10 Line Card
North Star ASIC

• Traffic Entering NS and exiting towards Fabric 12 x 40G
Hi-Gig2
Module is Ingress Pipeline Trident II
ASIC
12 x 40G
Ethernet
Network Interfaces
Ex. trigger init ingress in-select 3 out-select 5
set outer ipv4 src_ip 13.13.13.10 Front Panel 48x 1GE/10GE Ports
13.13.13.10
74
ELAM Sample Configuration & Key Info
Nexus9508 with N9K-X9564TX
13.13.13.10/30
13.13.13.1/30
Eth5/1 Eth6/52
N9K-X9564TX 4 40Gig Port On NS 40 1/10 Gig On T2
N9K# attach mod 6

module-6# debug platform internal ns elam asic 1
module-6(NS-elam)# trigger init egress in-select 3 out-select 5
module-6(NS-elam-insel3)# set outer ipv4 dst_ip 13.13.13.10
module-6(NS-elam-insel3)# start
If Packet Captured
module-6(NS-elam-insel3)# status
Status: Triggered
module-6(NS-elam-insel3)# report
75
Important ELAM Fields
GBL_C++ [INFO] hg2_srcmod: 0E Information

GBL_C++ [INFO] hg2_srcpid: 0D
is in Hex N9K# show interface hardware-mappings
Convert to -------------------------------------------
GBL_C++ [INFO] hg2_dstmod: 11 Dec. ----------------------------
Name Ifindex Smod Unit HPort FPort NPort VPort
GBL_C++ [INFO] hg2_dstpid: 0A
------------------------------------------
GBL_C++ [INFO] ip_da: 000000000000D0D0D0A Eth5/2 1a280000 14 0 13 255 0 -1
GBL_C++ [INFO] ip_sa: 000000000000D0D0D01 Eth6/52 1a286600 17 1 10 255 51 -1
GBL_C++: [MSG] - sideband is complete Sideband is the result where

GBL_C++: [INFO] ovector: 000FFF packet will be sprayed.
Should never be “0”
76
Packet Tracer-T2
FM Mod
• Helps to Trace the packet inside Switch.
• Only packets in the direction of the flow are traced Trident II

ASIC
• Two Acls are installed for each filter on each Line card
• One ACL for Front Panel Port Group

Network Interfaces
• Second ACL for traffic exiting Fabric Module and ingressing Line
card
77
Packet Tracer Configuration
test packet-tracer dst-ip 13.13.13.10 detail-fp
Configure Filter
test packet-tracer dst-ip 13.13.13.10 detail-hg
13.13.13.10/30
Start Tracer test packet-tracer start

rt
Stop Tracer test packet-tracer stop
Check Counter test packet-tracer show

Filter
Clear/Remove-all test packet-tracer clear remove
78
Sample Configuration & Identify Front Port-LC
N9K#test packet-tracer dst-ip 13.13.13.10 src-ip 13.13.13.1 detail-fp
N9K#test packet-tracer show filter 1 non-zero Packet-tracer stats

13.13.13.10/30
Module 6:
Filter 1 installed: src-ip 13.13.13.1 dst-ip 13.13.13.10 detail-fp
Module 21:
Module 26:
Eth6/1 Eth6/52
13.13.13.1/30 13.13.13.10/30
79
Packet Tracer Sample Configuration & Key Info
N9K# test packet-tracer start filter 1 13.13.13.10/30

N9K# test packet-tracer show filter 1 mod 6 non-zero Nexus9508 with N9K-X9564TX
Packet-tracer stats
Eth6/1 Eth6/52
Module 6:
Filter1 installed: src-ip 13.13.13.1 dst-ip 13.13.13.1/30
13.13.13.10 detail-fp 13.13.13.10/30
ASIC instance 0:
Entry 1: id = 7426, count = 5, active, fp, port 13
N9K# show interface hardware-mappings | grep 6/1

Name Ifindex Smod Unit Hport FPort Nport VPort
Eth6/1 1a280000 16 0 13 255 0 -1
80
Sample Configuration Identify Fabric Port LC From FM
N9K# test packet-tracer dst-ip 13.13.13.10 src-ip 13.13.13.1 detail-hg

N9K# test packet-tracer start filter 1
N9K# test packet-tracer show mod 6 non-zero
Module 6:
Filter 1 installed: src-ip 13.13.13.1 dst-ip 13.13.13.10 detail-hg
ASIC instance 0:
Eth8/1 Eth6/52 13.13.13.10/30
Entry 0: id = 7425, count = 68, stopped, fp,
ASIC instance 1: 13.13.13.1/30 13.13.13.10/30
Entry 1: id = 7426, count = 13, stopped, hg, port 1 Nexus9508 with N9K-X9564TX
Entry 2: id = 7427, count = 11, stopped, hg, port 2
81
Flex Counters –Adjacency Statistics
• Flex counters used to count Next hop Adjacency stats
• One can attach Stats to multiple Adjacency at same time
• One Stat Counter per adjacency
• Total Flex Counters are 16K per Switch
82
How To Configure Flex Counters
13.13.13.10/30
Eth6/1 Eth6/52
13.13.13.1/30 13.13.13.10/30
N9K# sh ip route 13.13.13.10

IP Route Table for VRF "default"
‘'%<string>' in via output denotes VRF <string>
13.13.13.8/30, ubest/mbest: 1/0

*via 13.13.13.6, Eth6/52, [110/41], 00:33:14, ospf-10, intra
N9K# test hardware internal adjacency statistics nexthop ipv4 13.13.13.6
interface ethernet 6/52 (enable |disable | show)
83
Sample Configuration
interface ethernet 6/52 show
13.13.13.10/30
Module:21 Unit:0
------------------
Adjacency counters for nhip 13.13.13.6 if Ethernet6/52:
Ucast: Packets 738 Bytes 90036 Nexus9508 with N9K-X9564TX
Mcast: Packets 0 Bytes 0

Eth6/1 Eth6/52
Module:22 Unit:1
------------------ 13.13.13.1/30 13.13.13.10/30
Adjacency counters for nhip 13.13.13.6 if Ethernet6/52:
Ucast: Packets 946 Bytes 115412
Mcast: Packets 0 Bytes 0
84
SPAN & ERSPAN
• Switch Port Analyzer”
• Provides efficient, high-performance traffic monitoring service
• Duplicates network traffic to one or more monitor interfaces
• Types Of SPAN
• Local SPAN
• Encapsulated Remote SPAN(ERSPAN)
• Applications:
• Troubleshooting connectivity issues
• Base lining network utilization/performance
• Detecting anomalous traffic flows
• On Nexus9000 Span Traffic uses dedicated queue
• Queue carrying SPAN traffic has low Priority over other queue’s
during congestion
85
SPAN QOS Queue
N9K# show queuing interface ethernet 4/18 | begin SPAN
| SPAN QOS GROUP |
+-----------------------------------------------------------------+
| | Unicast | OOBFC Unicast | Multicast |
+------------------------------------------------------------------+
| Tx Pkts | 0| 0| 0|
| Tx Byts | 0| 0| 0|
| Dropped Pkts | 0| 0| 0|
| Dropped Byts | 0| 0| 0|
| Q Depth Byts | 0| 0| 0|
86
SPAN Configuration
Sup-eth Local
N9K(config)# monitor session 1
N9K(config-monitor)# source interface sup-eth 0 both
N9K(config-monitor)# source interface ethernet 6/1 e6/1 e6/2
N9K(config-monitor)# destination interface ethernet 6/2 Local SPAN
N9K(config-monitor)# No Shut
N9K(config)#int et 6/2
N9K(config-monitor)# show monitor
N9K(config-if)# switchport monitor
Session State Reason Description
--- ----- ------------ --------------------
1 up The session is up Local SPAN Session
87
ERSPAN Configuration
Only Supports Source ERSPAN
N9K(config)# monitor erspan origin ip-address 13.13.13.2
global Type-3 Header 32-bit Timestamp
N9K(config)# monitor session 1 type erspan-source Supports on Nexus9300 only
N9K(config-erspan-src)# header-type 3
N9K(config-erspan-src)# source interface ethernet 6/1 Layer 3
N9K(config-erspan-src)# erspan-id 1 L3
N9K(config-erspan-src)# ip ttl 16
N9K(config-erspan-src)# vrf default e6/1 e6/2
N9K(config-erspan-src)# destination ip 9.1.1.2
ERSPAN
N9K(config-erspan-src)# marker-packet-2
Marker packet carry original UTC time
N9K(config-erspan-src)# no shut
stamp to over come 32-bit wrapper
issue
88
Consistency Checkers-Summary
• Show consistency-checker stp-state vlan
• Show consistency-checker link-state
• Show consistency-checker membership vlan
• Show consistency-checker membership port-channels
• Show consistency-checker membership port-channels
• Show consistency-checker l2
• Show consistency-checker l3
• Show consistency-checker forwarding ipv4 unicast
89
Nexus 9000
Troubleshooting
90
Understanding T2 interfaces-Xe0/hg
N9K# bcm-shell mod 1 "show unit"
Unit 0 chip BCM56852_A2 (current)
Unit 1 chip BCM56852_A2
hg0 hg11 hg0 hg11
N9K#bcm-shell mod 1 “0:ps”

ena/ speed/ link auto STP lrnT2
inter max loop
T2
port link duplex scan neg? state pause discrd Instance
ops 0
face frame back
Instance 1
hg0 up 42G FD HW No Forward None F FA FQSPF F F

XGMII F
16360 F F F F FQSPF F F F F F F
hg2 up 42G FD HW No Forward None
FPFP FP FP
01 02
P
04
FA
PPorts
06
P PXGMII
FPFP FP
09 10
P
12
16360 P P P P PPorts P
FP
19
P P P P P
--------------------------------Snip----------------------------------
03 05 07 08 11 13 14 15 16 17 18 20 21 22 23 24
Hg11 up 42G FD HW No Forward None FA XGMII 16360 Xe0

Xe0 !ena 40G FD HW No Disable None
Xe0
FA XGMII Xe11
1582 Xe11
xe1 up 40G FD HW No Disable None

Eth1/1 FA Eth1/12XGMII 1582 Eth1/13 Eth1/24
--------------------------------Snip----------------------------------
Xe11 !ena 40G FD HW No Disable None FA XGMII 1582
Hg=Internal Ports
Xe=Front Panel Port
91
Layer -1 Issues- Transceiver Not Recognized
N9K# show interface ethernet 4/18 transceiver details

Ethernet4/18
transceiver is not present
module-4# show hardware internal bcm-usd event-history xcvr 18
1) Event:E_STRING, length:135, at 220346 usecs after Thu Apr 16 20:50:17 2015
bcm_usd_xcvr_fcot_notify_default(941): [unit=0 nxosport=18 bcmport=30]
fcot_state:0x2 fcot_type:0 sent MTS_OPC_FCOT_EVENT_INFO, rc 0x0
2) Event:E_STRING, length:93, at 647132 usecs after Thu Apr 16 20:50:14 2015
bcm_usd_xcvr_fcot_scan_sfp(3003): [unit=0 nxosport=18 bcmport=30]
FCOT not supported err=-1
92
Interface MTU/Speed/Flow Control Verification
N9K# show interface Ethernet 4/18
Ethernet4/18 is up
admin state is up, Dedicated Interface Belongs to Po10
Hardware: 10000/40000 Ethernet, address: 7c69.f66e.d860 (bia 7c69.f66e.d860)
MTU 9216 bytes, BW 40000000 Kbit, DLY 10 usec
N9K# bcm-shell module 4 ” 1: ps Xe17"
ena/ speed/ link auto STP lrn inter max loop
port link duplex scan neg? state pause discrd ops face frame back
xe17 up 40G FD HW No Disable None FA SR4 9298
93
Interface Flow Control Check
N9K#Show interface ethernet 1/1 flowcontrol
Port Send FlowControl Receive FlowControl RxPause TxPause
admin oper admin oper
-----------------------------------------------------------------------------
Eth1/1 off off off off
0 0
N9K#bcm-shell module 1 "ps" Wrong programming
ena/ speed/ link auto TP lrn inter max loop
xe0 up 10G FD HW No Disable TX RX None FA SFI 9298
94
Interface Input Drops
Ethernet1/30 is up
Hardware: 1000/10000 Ethernet, address: 7426.acea.ceb9 (bia 7426.acea.ceb9)
EtherType is 0x8100
0 input with dribble 1316 input discard
N9K#bcm-shell mod1 “ cstat xe29”
+------------------Programmable Statistics Counters[Port xe29]------+
| Type | No. | Value | Enabled For |
+----------------------------------------------------------------- -+
| RX | 0(R)| 19163028| RIPD4 RIPD6 RDISC RPORTD |
| | | | PDISC VLANDR |
| | 1(R)| 28744286| IMBP |
| | 4 | 993820| RPORTD FcmPortClass3RxDiscards |
| | 6 | 19163407| RFILDR FcmPortClass2RxDiscards |
| | 7 | 19163048| RDROP |
| | 8 | 18169208| VLANDR | | gre VLANDR
bcm-shell mod 6 "cstat info"
+-------------------------------------------------------------------+
| | 3(R)| 14704| TPKTD VLANDR Rx VLAN drops
|
| | 4(R)| 968303| TGIP4 TGIP6 FcmPortClass3TxFrames|
| | 6 | 968303| TGIP4 FcmPortClass3TxFrames |
+-------------------------------------------------------------------+
95
Fabric Connectivity and Troubleshooting
T2 T2 T2
T2
T2 T2 T2
N9K-C9508-FM-4 N9K-C9508-FM-8 N9K-C9516-FM-16
• In an 4-slot chassis N9K-C9504-FM has 1 T2 per module

• FMs provides redundancy for internal data flow, the loss of FMs just increases
the oversubscription factor.
96
Full-Rate Mode(FRM) V/S Oversubscribed Mode(OSM)
• Each T2 have 32 40Gigport with total capacity of 1.2Tbps with “2” switching
mode
OSM(Default) - Uses all 32 40 Gig ports Line Rate achieved for packets > 200 Bytes
FRM - Uses only 24 40 Gig ports Line rate achieved for > 64 Bytes
Configuration Knob to Change the mode.
N9K(config)# system fabric-mode full-rate
Configuration effective after Reboot
N9K#show system fabric-mode
Applied System Fabric Mode:Full rate mode
Use FRM mode to achieve line rate for 64 byte packets on 9636PQ , 9564PQ ,
9564TX cards
All other 94xx line cards will not be powered up in this mode
97
RTAG7 and DLB
• Two Packet Hashing algorithm available from LC to FM
• RTAG7-To Select HG Port use Packet Header.
• For a flow same HG Link is used FM1 FM-2
FM6
• DLB-Dynamic Load Balancing- Default algorithm

HG- HG-
• Initial Hash same as RTAG7 Ports Ports
• Based on Link Quality pick up optimum HG Port LC1 LC2
• Better utilization of all HG links

• N9K(config)# port-channel load-balance internal [dlb/rtg7]
• N9K# show port-channel load-balance internal algorithm

• HighGig port-channel load balance algorithm: dlb
98
Higig Link Failures – Fabric Module Policy
• For any single Higig link failure between FM and LC
Bring down the FM, if there is more than one FM
Else bring down LC
• Multiple Higig links failures for a Single LC going to Multiple FM - Bring down
the LC module.
• Multiple Higig links failures on LC to one of the FM - Bring down the LC module
99
4/8 slot Chassis – Fabric Connectivity
• 9500/9600 Series Line Card’s T2
T T T T T T T T T T T T
have connectivity to all 6 Fabric 2 2 2 2 2 2 2 2 2 2 2 2
Module’s T2
• 9400 series Line cards connects to

all T2 but use only 4 Fabric Modules
-No Connection to Slot 21 & 25
• Traffic between 9500/9600 Series 40 Gig Link

Line Card and 9400 Line card will T2 T2
use subset Hi Gig links .

N9K-X9536PQ
100
16 slot Chassis – Fabric Connectivity
N9K-C9516-FM
• 9500 Series Line Card’s T2 will have connection 2 2 2 2 2 2 2 2 2 2 2 2
to all 6 Fabric Module but to only 2 T2’s from T T T T T T
T T T T T T
each Fabric Module 2 2 2 2 2 2 2 2 2 2 2
2
• 9500 series Line Card’s T2 will have connection

to all 4 T2’s of Fabric module if there are only 3
Fabric module present
• 9400 series cards connects to all T2 but use only

4 FM-No Connection to Slot 21 & 25
• Traffic between 9500 Series Line Card and 9400 40 Gig Link
will use subset Hi gig links.
• N9K-X9636PQ line card module is not supported T2 T2

in 16 slot chassis
N9K-X9536PQ
101
16 slot Chassis – Fabric Connectivity
• With 3 FM configuration All 4 T2 units in
each FM are connected to 9500 series LC 2 2 2 2 2 2 2 2 2 2 2 2
modules' T2 units
• Each blue line represents one 40 Gig link
T2 T2
N9K-X9536PQ
102
Line Cards With Mux to FM
FM26 FM25 FM24 FM23 FM22 FM21 Fabric Module’s
HG HG HG HG HG HG
MUX1 MUX4 MUX2 MUX5 MUX3 MUX6
Line Card
• Line cards N9K-X9464PX/TX ,

012 456 8 9 10 012 456 8 9 10
3 MN7Port 11 3 MN7Port 11 N9K-X9564PQ/TX have Mux
Northstar 1 Northstar 2
0-
MF Port
3- 6-8 9-
Active Mux Link
0-
MF Port
9-
• Mux used for connecting HiG Link
2 5 11 2 11 from Line Cards to multiple Fabric
Module
7- 2- 31- 26- Standby Mux Link 7- 26-
5 Warpcore
0 29 24 5 24 • Mux available only for Half of the
T2 T2 HiG interface of LC
• By Default Mux Link Active to Odd
number of Fabric Module
103
FM Connectivity For N9K-X9564PX –With MUX
show system internal fabric connectivity mod 5 | in B • Line cards N9K-X9464PX/TX ,
N9K-X9564PQ/TX have Mux
HiGIG Link-info Linecard slot:5 • Mux used for connecting HiG Link
from Line Cards to multiple Fabric
LC-Slot LC-Unit LC-HGLink MUX FM-Slot FM-Unit FM-HGLink Module
5 0 HG02 1B 25 0 HG12 • Mux available only for Half of the
HiG interface of LC
5 0 HG03 1B 25 1 HG12 • By Default Mux Link Active to Odd
number of Fabric Module
show system internal fabric connectivity mod 5
HiGIG Link-info Fabriccard slot:5 With FM from Slot 25 Down FM-25 FM-26
T2-0 T2-1 T2-0 T2-1
LC-Slot LC-Unit LC-HGLink MUX FM-Slot FM-Unit FM-HGLink HG012 HG012 HG014
5 0 HG02 1A 26 0 HG14 B A HG014
HG02 HG03
5 0 HG03 1A 26 1 HG14 MUX
LC
T2-0 T2-1
104
FM Connectivity For N9K-X9564PX –With MUX
show system internal fabric connectivity mod 5 | in B
FM-25 FM-26
HiGIG Link-info Linecard slot:5 T2-0 T2-1 T2-0 T2-1
HG012 HG012 HG014 HG014
LC-Slot LC-Unit LC-HGLink MUX FM-Slot FM-Unit FM-HGLink A
B
HG02 HG03
5 0 HG02 1B 25 0 HG12
MUX
LC
5 0 HG03 1B 25 1 HG12
T2-0 T2-1
HiGIG Link-info Fabriccard slot:5 With FM from Slot 25 Down FM-25 FM-26
LC-Slot LC-Unit LC-HGLink MUX FM-Slot FM-Unit FM-HGLink T2-0 T2-1 T2-0 T2-1
HG012 HG012 HG014 HG014
5 0 HG02 1A 26 0 HG14 B A
HG02 HG03
5 0 HG03 1A 26 1 HG14
MUX
LC
T2-0 T2-1
105
Fabric Troubleshooting commands
Fabric Module Slot-21
HiGIG Link-info Linecard slot:1 T2 T2
LC-Slot LC-Unit LC-HGLink MUX FM-Slot FM-Unit FM-HGLink #0 #1
HG00
HG00
1 0 HG00 - 21 0 HG00
1 0 HG01 - 21 1 HG00
HiGIG Link-info Fabriccard slot:21
T2 T2 T2
FM-Slot FM-Unit FM-HGLink LC-Slot LC-Unit LC-HGLink MUX
#0 #1 #2
21 0 HG00 1 0 HG00
21 1 HG00 1 0 HG01
Line Card Slot-1
106
Fabric Port Drops and Link Status
N9K# show hardware internal fabric interface asic counters mod 21
Counters for Fabric Ports:
FabricInterface Forward Forward Error Pkt Error Pkt QOS Rx QOS Tx
RxDrops TxDrops RxDrops TxDrops Drops Drops
0 / 1 / HG0 0 0 0 0 0 0
1 / 1 / HG0 0 0 1 0 0 0 0
N9K# bcm-shell mod 21 "ps” | inc hg0
ena/ speed/ link auto STP lrn inter max loop

hg0 up 42 FD HW No Forward None FA XGMII 16360
107
Fabric Port STP State HW point of View
N9K# sh vlan id 100 N9K# bcm mod 21 " stg show”

VLAN Name Status Ports
---- ------------------ --------- 100 STG 5: contains 1 VLAN (100)
VLAN0100 active Po1, Eth1/1 Forward: hg
show sys internal xbar event-history {trace|errors|msgs|sw}

show sys internal xbar-client event-history {trace|errors|msgs|sw}
show tech-support xbar
108
Path of the Packet -Inband
CPU
• Traffic from all ingress Line Card
to Supervisor will hash to one
Netstack
Fabric module
• Traffic from Supervisor Card to
NIC-Eth2 NIC-Eth3 Egress Line cad will hash on one
FM. May not be same
Mod29 System Controller-SC1 • CoPP is operational on all LC.
However aggregate CoPP is on
FM
Fabric Module
Fabric Module
Fabric Module
Mod26
Mod21 Mod23
Eth6/1
Line Card OSPF Hello
109
Line Card
North Star ASIC
Check for Drops/Errors-Line Card Trident II
ASIC
Network Interfaces
N9K#show hardware internal interface ethernet 6/1 asic counters
Important Counters/Drops
--------------- --------- --------- --------- --------- --------- ---------
Interface Name Forward Forward Error Pkt Error Pkt QOS Rx QOS Tx
--------------- --------- --------- --------- --------- --------- ---------
Ethernet6/1 870 0 100 0 0 0
--------------- --------- --------- --------- --------- --------- ---------
Forward Rx Drops = [ RDBGC0 RDBGC4 RDBGC6 RDBGC7 RDBGC8 ]
Forward Tx Drops = [ TDBGC1 TDBGC3 TDBGC5 (excludes expected Multicast drops)]
ErrorPkt Rx Drops= [ IUNHGI IUNKOPC RFCS RALN RFLR RERPKT RJBR RSCHCRC RUND RMTUE]
ErrorPkt Tx Drops= [ TJBR TFCS TRPKT RMTUE TUFL TPCE ]
QOS Rx Drops = [ RDISC DROP_PKT_ING DROP_PKT_IMTR DROP_PKT_YEL DROP_PKT_RED ]
QOS Tx Drops = [ MCQ_DROP_PKT(0) MCQ_DROP_PKT(1) MCQ_DROP_PKT(2)
Use slot <#> show hardware internal interface indiscard-stats instance <#> RDBGC0
N9K#bcm-shell mod 6 "listreg RALN"| grep Description
Description: Receive Alignment Error Frame Counter
110
Instant Buffer Usage Stats-With Buffer Usage
N9K#show hardware internal buffer info pkt-stats mod 6
INSTANCE: 0

SP-0 SP-1 SP-2 SP-3
-------------------------------------------------------------------------
Total Instant Usage 4474 0 89 2939 • SP-3 Started filling
Remaining Instant Usage 25466 0 14255 3405 the Queue
------------------------------------------------------------------------
ASIC Port Q3 Q2 Q1 Q0 CPU SPAN
[13]
Only printed if there is congestion
UC(OOBFC)-> 0 0 0 0
UC-> 0 0 0 1249 332 0 • CPU buffer filling
MC-> 0 0 0 3247 1996 0 up
111
CoPP Drops on Line Card
Line Card
North Star ASIC
Trident II
ASIC
Network Interfaces
N9K# show policy-map interface control-plane mod 6 class copp-system-p-class-

critical | in ospf|trans|dropped
match access-group name copp-system-p-acl-ospf
transmitted 21898 packets;
dropped 0 packets;
112
Identify FM -Check CoPP Drops
N9K# show hardware internal cpu-mac inband active-fm traffic-to-sup

Active FM Module for traffic to sup:
0x00000015 Fabric Module in Slot 21 carry all traffic to Sup
N9K# show policy-map interface control-plane mod 21 class copp-system-p-class-

critical | in ospf|trans|dropped
match access-group name copp-system-p-acl-ospf
match access-group name copp-system-p-acl-ospf6
transmitted 21898 packets;
dropped 0 packets;
113
Check for Drops/Errors-Fabric Module
N9K# show system internal fabric connectivity mod 6 | grep 21 Identify HG Port on LC and FM
LC-Slot LC-Unit LC-HGLink MUX FM-Slot FM-Unit FM-HGLink
6 0 HG10 3B 21 0 HG15
N9K# sh hardware internal fabric interface asic counters module 6 instance 0 asic-port 11
Important Counters/Drops Verify Drops/Error on HG port on LC

FabricInterface Forward Forward Error Pkt Error Pkt QOS Rx QOS Tx
0 / 11 / HG10 0 0 0 0 0 0
N9K# sh hardware internal fabric interface asic counters mod 21 in 0 asic-port 16

0 / 11 / HG15 0 0 0 0 0 0
114
Verify Drops Between FM and SC System Controller
MVDXN-SW
module-21# show mvdxn internal port-status MVDXN-SW

Switch type: Marvell 98DXN11 - 10 port switch Fabric Module in Slot 21 FABRIC CARD
Port Descr Enable Status ANeg Speed Mode InByte OutByte InPkts OutPkts
3 SC1EPCswitch Yes UP No 2 6 109548011 117051401 274144 587285
10 port switch on System
controller and Fabric
module-29# show mvdxn internal port-status module connect SC to FM
Switch type: Marvell 98DXN11 - 10 port switch System Controller in Slot 29
Port Descr Enable Status ANeg Speed Mode InByte OutByte InPkts OutPkts
7 FM1EPCswitch Yes UP No 2 6 746159513 60543666 620863 269592
115
Drops/Errors On Supervisor
N9K#show hardware internal cpu-mac inband counters in eth|ps-
inb|dro Netstack
eth2 Link encap:Ethernet HWaddr 00:00:00:01:1b:01
Pseudo Inband
TX packets:1652929 errors:0 dropped:0 overruns:0 carrier:0 NIC-Eth2 NIC-Eth3
eth3 Link encap:Ethernet HWaddr 00:00:00:01:1b:01 Supervisor Card
ps-inb Link encap:Ethernet HWaddr 00:00:00:01:1b:01
116
Drops/Errors On Supervisor-Cont.
N9K#show hardware internal cpu-mac inband stats | in errors|rate|Queue
Queue Idx Packet Count Bytes Drops Csum Errors Allocation Failure
Queue 0 65429 580195964 2 0 0
Queue 7 65429 580195964 0 0 0
CRC errors ...................... 0
Alignment errors ................ 0
Symbol errors ................... 0
Carrier extension errors .........0 Related show tech(s)

Nexus9500# sh tech-support inband
Rx packet rate (current/peak) 812 / 1097 pps
counters
Tx packet rate (current/peak) 454 / 741 pps Nexus9500# show tech-support pktmgr
Nexus9500# show tech-support <service>
117
L2 Mac And Vlan Table Verification
N9K# sh mac address-table dynamic vlan 100
Legend: * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC Eth6/1
age - seconds since last seen,+ - primary entry using vPC Peer-Link, (T) -
True, (F) - False Mac=547f:ee1c.06fc
VLAN MAC Address Type age Secure NTFY Ports interface Ethernet6/1
switchport
* 100 547f.ee1c.06fc dynamic 0 F F Eth6/1
switchport access vlan 100
N9K# bcm-shell mod 6 " l2 show" | in Hit
no shutdown
mac=54:7f:ee:1c:06:fc vlan=100 GPORT=0x800800d modid=16 port=13/xe0 Hit
N9K# bcm-shell mod 6 "vlan show 100”

vlan 100 ports xe0,hg ....... untagged xe0
118
Spanning Tree Verification
N9K# sh spanning-tree interface ethernet 6/1
Vlan Role Sts Cost Prio.Nbr Type
interface Ethernet6/1
VLAN0100 Desg FWD 4 128.1537 P2p switchport
N9K# bcm-shell mod 6 "dump vlan 100” switchport access vlan 100
no shutdown
VLAN.ipipe0[100]: <VP_GROUP_BITMAP=0x00000……STG=0X67
FID_ID=0x64
N9K# Dec 0x67=103
N9K# Dec 0x64=100
STG= STP Group ID Eth6/1
FID_ID=Vlan ID.
Mac=547f:ee1c.06fc
N9K# bcm-shell mod 6 "stg stp 103”
STG 103:
Block: xe1-xe47
Forward: xe0,hg
119
Unicast L3 Forwarding
• T2 has combination of dedicated TCAM table space and shared hash table
memory known as Unified Forwarding Table (UFT)
• The UFT is partitioned into three forwarding tables
• MAC Address Table
• IP Host Table
• Longest Prefix Match-LPM Table
• To maximize the system-wide forwarding scalability UFT tables on line
cards and fabric modules for different forwarding lookup functions
Feature Scale
FM Feature Scale
L3 LPM Table 128K
L3 Host Table 120K
LC And L2/L3
Multicast
L2 Mac Table 96K
120
Unicast L3 Forwarding- Component Information
Theory of Operation OSPF AM ARP
Software/Hardware Programming
• OSPF communicates with uRIB to build the uRIB
routing table
• AM builds the next-hop adjacency entry
• uFDM distributes the information to the line uFDM
cards Supervisor
• IP FIB (running on the line cards) programs the Hardware-T2
ASIC components with the forwarding and
adjacency information. FIB Manager
Remember: Software forwarding by the SUP is only

used for control and exception packets Forwarding Hardware
121
L3 Unicast Troubleshooting Flow
Next-Hop
Check the routing table Show ip route [ipv4] [<prefix>]
ARP/MAC
Show ip arp [ipv4]
show ip adjacency (Ipv4]
Check the ARP Table
show forwarding adjacency platform [ipv4] module
<mod>
Checking Route on
RIB And FIB.
Check Forwarding Route show forwarding [ipv4] route module <mod>
HW Programming
On LC/FM
bcm-shell mod 22 "l3 defip show"
Use BCM commands
122
Unicast L3 Forwarding- Two Possible Scenarios
Case 1: If incoming packet hit /32 host route on LC, forwarding decision is made on LC
Case 2: If incoming packet miss /32 host route on LC. Now for Longest Prefix
match (LPM) the packet get forwarded to FM
• Install a default route 0.0.0.0/0 on Line Cards using the virtual MOD ID for Fabric Module
as the DMOD to force Line Cards to forward LPM packets to Fabric Modules
• Fabric Modules perform LPM lookup and forward packets to the resolved Destination
MOD/Destination PORT
Also will verify How to Check ECMP Route
123
Network Diagram-Problem Definition
13.13.13.12/30
.13 .14
13.13.13.0/30 13.13.13.8/30
.1 .2 .17 .18 .9 .10 N9508d-SJ#
Nexus3064Q-ESC# 13.13.13.16/30
N9K# N9508c-SJ#
Nexus3064Q-ESC# ping 13.13.13.10
PING 13.13.13.10 (13.13.13.10): 56 data bytes
Request 0 timed out
Nexus3064Q-ESC# traceroute 13.13.13.10

traceroute to 13.13.13.10 (13.13.13.10), 30 hops max, 40 byte
packets
1 13.13.13.2 (13.13.13.2) 1.124 ms 0.911 ms 0.752 ms
2 * * *
124
Router MAC Programming Check
• Router Mac address must be programmed in Hardware
N9K1#show interface ethernet 6/1 | grep address
Hardware: 100/1000/10000 Ethernet, address: 003a.99fc.dd7f
N9K1# bcm-shell mod 6 "0:d chg my_station_tcam" | grep dd7f
MY_STATION_TCAM.ipipe0[0]: <VALID=1,------snip----
MAC_ADDR=0x003a99fcdd7f,
125
Verify /32 Host Route on Line card-Case 1
N9K1#show ip route 13.13.13.14
13.13.13.14/32, ubest/mbest: 1/0, attached /32 Host Entry
*via 13.13.13.14, Eth6/33, [250/0], 00:37:24, am
N9K1#bcm-shell mod 6 "0:l3 l3table show" | grep 13.13.13.14
Entry VRF IP address Mac Address INTF MOD PORT CLASS HIT
10 1 13.13.13.14 00:00:00:00:00:00 100010 0 0 0 y
N9K1#bcm-shell mod 6 "0:l3 egress show"| grep 100010

Entry Mac Vlan INTF PORT MOD MPLS_LABEL ToCpu Drop
100010 88:f0:31:bf:ad:17 4095 4432 45 16 -1 no no
N9K1#show system internal ethpm info interface ethernet 6/33 | grep -i STATIC
IF_STATIC_INFO: port_name=Ethernet6/33,if_index:0x1a284000,ltl=40875,slot=5,
nxos_port=32,dmod=16,dpid=45,
126
Next Hop Reached via L3-Port Channel
N9K1#show ip route 10.164.112.22
10.164.112.22/32, ubest/mbest: 1/0 /32 Host Entry
*via 13.13.13.14, Po200, [110/3], 00:09:33, ospf-10, intra
N9K1#bcm-shell mod 6 "0:l3 l3table show" | grep 10.164.112.22
175660 1 10.164.112.22 00:00:00:00:00:00 100012 0 0 0 y
N9K1#bcm-shell mod 6 "0:l3 egress show"| grep 100012

100010 88:f0:31:bf:ad:17 665 4761 3t 0 -1 no no
N9K1#show system internal ethpm info interface port-channel 200 |grep –I STATIC
IF_STATIC_INFO: port_name=port-channel200,if_index:0x160000c7,ltl=2597,slot=0,
nxos_port=02,dmod=0,dpid=3,
127
Verify HW-Programming on LC or FM ? Case 2
N9K# show ip route 13.13.13.10
This is not /32 host Route.
IP Route Table for VRF "default” Packet forwarding decision
responsibility is of the Fabric
13.13.13.8/30, ubest/mbest: 1/0 Module
*via 13.13.13.6, Eth6/52, [110/41],
00:22:29, ospf-10, intra ALL FM will be programmed
N9K# show forwarding route 13.13.13.10 module 21 with this Route
IPv4 routes for table default/base

Prefix | Next-hop Interface | Labels
13.13.13.8/30 13.13.13.6 Ethernet6/52
128
Line Card Punting Packets to Fabric For LPM ?
N9K# show hardware internal forwarding adjacency statistics default-route mod 6
Module:6 Unit:0
Traffic matched adjacency for default route (destined to FM):
Unicast: Packets 148 Bytes 13382
N9K# bcm-shell mod 6 "0:l3 defip show"

Unit 0, Total Number of DEFIP entries: 12288
# VRF Net addr Next Hop Mac INTF MODID PORT PRIO CLASS HIT
VLAN
3072Override 0.0.0.0/0 00:00:00:00:00:00 149149 0 0 0 0 y
N9K# bcm-shell mod 6 "l3 egress show" | inc 149149 Mod 100 is assign to Fabric Module
149149 00:12:12:12:12:12 4095 8189 1 100 -1 no no
129
Longest Prefix Match on Fabric Module
N9K# bcm-shell mod 22 "l3 defip show" | grep 13.13.13.8
# VRF Net addr Next Hop Mac INTF MODID PORT PRIO CLASS HIT VLAN
196620 1 13.13.13.8/30 00:00:00:00:00:00 100008 0 0 0 0 n
N9K# bcm-shell mod 22 "l3 egress show" | grep 100008
100008 88:f0:31:bf:ad:17 4095 4520 10 17 -1 no no
Mac add used for rewrite

N9K# show system internal ethpm info interface eth 6/52 | grep dmod
IF_STATIC_INFO:
port_name=Ethernet6/52,if_index0x1a286600,ltl=40856,slot=5,nxos_port=51,
dmod=17,dpid=10,unit=1,
130
ECMP Route Validation
N9K#show ip route 10.164.112.22
10.164.112.22/32, ubest/mbest: 2/0
Multi-Path
N9K#sh routing hash 13.13.13.2 10.164.112.22 mod 6 N9K#bcm-shell mod 6 "l3
multipath show"
Hashing to path *13.13.13.18
Multipath Egress Object 200256
Out Interface: Eth6/34
Interfaces: 100008 100010
N9K#bcm-shell mod 6 "0:l3 l3table show" | grep 10.164.112.22
17 1 10.164.112.22 00:00:00:00:00:00 200256 0 0 0 n (ECMP)
Follow same steps demonstrated for /32 Host entry to learn about Interface in multipath show cli
131
Use Tools From Toolkit
• ELAM- IF Line Card has North Star
module-6# debug platform internal ns elam asic 1
module-6(NS-elam)# trigger init egress in-select 3 out-select 5
module-6(NS-elam-insel3)# set outer ipv4 dst_ip 13.13.13.10
• Packet Tracer- For All FM and LC having T2

N9K# test packet-tracer dst-ip 13.13.13.10 src-ip 13.13.13.1 detail-fp
• Flex Counter- Check Adjacency hit counter

interface ethernet 6/52 enable show tech-support forwarding l3 unicast
show tech-support adjmgr
• Consistency Checker show tech routing unicast
show consistency-checker forwarding ipv4 unicast
132
Virtual Port-Channel-vPC
• Allow a single device to use a port channel across
two upstream switches
• Eliminate STP blocked ports
• Dual-homed server operate in active-active mode
• HSRP-Both active and standby peers forward
packets-ARP response by Active
• Configuration steps Same as other Nexus Logical Topology with vPC
Products
133
Case:1 All vPC Leg UP
Scenario: Traffic of a Host in Vlan 10 connected to Switch-A hash to N9K1 to reach Host in Vlan 20
connected to Switch-B
N9k1 N9k2 vPC Peer Link =Eth1/1,4/1
Keep Alive
PC1-PeerLink
SVI10
SVI10
10.10.10.1/24 MCT-1/1, 4/1 10.10.10.2/24
SVI-Mac 78da.6e71.9a3f Eth6/20 Eth4/18
SVI-mac 003a.99fc.dd7f
Standby 10.10.10.3 Eth4/18 Eth6/20
Standby 10.10.10.3
HSRP-Mac 0000.0c07.ac0a
HSRP-Mac 0000.0c07.ac0a
SVI20
vPC10 vPC20 SVI20
SVI-mac 78da.6e71.9a3f
SVI-mac 003a.99fc.dd7f
10.10.20.1/24
Switch-A Switch-B 10.10.20.2/24
Standby 10.10.20.3
Standby 10.10.20.3
HSRP-Mac 0000.0c07.ac14
HOST-A Vlan-10 HOST-B Vlan-20 HSRP-Mac 0000.0c07.ac14
10.10.10.x/24 20.20.20.x/24
134
vPC-Router MAC Programming Check
• Both Active and Standby Peer responsible for L3 switching
• Virtual Mac address must be programmed in Hardware on Both peers
Interface Grp Prio P State Active addr Standby addr Group addr
Vlan10 10 100 Active 10.10.10.2 local 10.10.10.3
N9K1# bcm-shell mod 4 "0:d chg my_station_tcam" | grep
VLAN_ID=0xa
VLAN_ID=0xa,VALID=1, MAC_ADDR=0xc07ac0a,
Interface Grp Prio P State Active addr Standby addr Group addr
Vlan10 10 100 Standby 10.10.10.2 local 10.10.10.3
N9K2# bcm-shell mod 4 "0:d chg my_station_tcam" | grep
VLAN_ID=0xa
VLAN_ID=0xa,VALID=1, MAC_ADDR=0xc07ac0a,
135
vPC Peer Gateway Programming Check
• Are N9K’s Configured with Peer-Gateway
N9K1-SJ# show mac address-table vlan 10 | in G
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
G 10 0000.0c07.ac0a static - F F vPC Peer-Link(R)
G 10 003a.99fc.dd7f static - F F sup-eth1(R) N9K2 SVI MAC
G 10 78da.6e71.9a3f static - F F vPC Peer-Link®
N9K# bcm-shell mod 4 "0:d chg my_station_tcam" | egrep 0x003a99fcdd7f

MY_STATION_TCAM.ipipe0[0]:
<VALID=1,MAC_ADDR_MASK=0xffffffffffff,MAC_ADDR=0x003a99fcdd7f,KEY=0x00000000003
a99fcdd7f,IPV6_TERMINATION_ALLOWED=1,IPV4_TERMINATION_ALLOWED=1,DATA=0x38,ARP_R
ARP_TERMINATION_ALLOWED=1>
136
vPC Check For Traffic Ingressing Peer Link
Egress Block Mask
• vPC Check-Traffic from Peer Link should Not L2/L3 Switch with local and remote
Legs up
N9K1# show vpc brief | grep Po
id Port Status Active vlans N9k1 N9k2
1 Po1 up 10-20 Keep Alive
id Port Status Consistency Reason Activevlans PC1-PeerLink
10 Po10 up success success 10-20 MCT-1/1, 4/1

20 Po20 up success success 10-20 Eth6/20 Eth4/18
Eth4/18 Eth6/20
N9K2# show vpc brief | grep Po
id Port Status Active vlans vPC10 vPC20
1 Po1 up 10-20
id Port Status Consistency Reason Activevlans Switch-A Switch-B
10 Po10 up success success 10-20
20 Po20 up success success 10-20
137
vPC Check for Traffic Ingressing Peer Link (Cont’d)
N9K1#show port-ch summary | in Po N9k1 Keep Alive N9k2

Group Port- Type Protocol Member Ports PC1-PeerLink
1 Po1(SU) Eth LACP Eth1/1(P)Eth4/1(P) MCT-1/1, 4/1
10 Po10(SU)Eth LACP Eth4/18(P) Eth6/20 Eth4/18
20 Po20(SU)Eth LACP Eth6/20(P) Eth4/18 Eth6/20
vPC10 vPC20
N9K1# show system internal vpcm info mask
module 6 Switch-A Switch-B
Masked ports for Module 6, Unit 0:
[Src Port None]: Eth6/20 Traffic Ingressing on Eth1/1 and
[Src Port Eth1/1]: Eth6/20
[Src Port Eth4/1]: Eth6/20 Eth4/1 will not exit Eth 6/20
Masked ports for Module 6, Unit 1:
138
ACL redirect logic for routed packets-vPC Leg Down
• Redirect ACL installed to redirect routed packets for the
vPC for which local interface goes down N9k1 N9k2
Keep Alive
• Mac address learned from vPC points virtual port PC1-PeerLink
MCT-1/1, 4/1
N9K1# show hardware access-list tcam region | grep vpc Eth6/20 Eth4/18
VPC Convergence [vpc-convergence] size = 512 Eth4/18 Eth6/20
N9K1# sh mac address-table address30f7.0d9b.d401

Link Down
vPC10 vPC20
20 30f7.0d9b.d401 dynamic 0 F F vPC Peer-Link

Switch-A Switch-B
139
ACL redirect logic for routed packets-vPC Leg Down
• On N9K1 traffic entering Eth6/20 after L3 switch
N9k1 N9k2
should egress Peer Link Keep Alive
PC1-PeerLink
• N9K2 Should not drop traffic entering Peer link and
forward traffic out to Eth 4/8 MCT-1/1, 4/1
Eth6/20 Eth4/18
Eth4/18 Eth6/20
N9K# bcm-shell module 6 "fp show group 57”
InPorts->L3Routable Ln Down vPC10 vPC20
DstTrunk
Switch-A Switch-B
Offset: 213 Width: 16
DATA=0x00008003 Trunk-id of “3” Down vPC
action={act=RedirectTrunk, param0=1(0x1) Trunk-id of vPC Peerlink
140
ACL redirect logic for routed packets-Verify TrunkID
N9Ka# show system internal ethpm info int port-channel1 | grep dpid
IF_STATIC_INFO: port_name=port-channel1,if_index:0x16000000,ltl=2595,slot=95
dpid=1,unit=0,queue=0,xbar_unitbmp=0x0 ns_pid=0
N9508a-SJ# show system internal ethpm info int port-channel10 | grep dpid
IF_STATIC_INFO: port_name=port-channel10,if_index:0x16000000,ltl=2595,slot=95
dpid=3,unit=0,queue=0,xbar_unitbmp=0x0 ns_pid=0
show tech-support vPC

show tech-support cfs Some important info to capture
show tech-support port-channel
141
ACL redirect logic for routed packets-Verify TrunkID
NX-OS -7.0(3)I1(2)
N9508a-SJ# show system internal access-list vpc-convergence mod 6
------------------------------------------------------------
VPC Convergence Entries
------------------------------------------------------------
Instance: 0
========== Trunk-id of “3” Down vPC
Ingress:
Trunk-id of vPC Peerlink
----------
Entry-ID DstTrunk-GID RedirectTrunk-GID Packet-Count
------------------------------------------------------------------------
1539 3 1 6082015
142
Nexus9000 Specific
Limitation and Goodies
143
Email from Nexus9000 To Cisco SR
• Commands output directly sent to email address
• Information from Nexus9000 Can be directly attached to Service Request.
• Information is sent as body to email- not as attachment
N9K(config)# email
N9K(config-email)# smtp
N9K(config-email)# smtp-host 173.37.37.37
N9K(config-email)# from N9508a-sj@cisco.com
N9K(config-email)# smtp-port 25
show run | email subject <SR-number> attach@cisco.com
144
Bash Support !!!!
• Goes beyond what standard CLI can provide
• Customers demand more capabilities/freedom Creativity
• Feature: bash-shell
• User Role: dev-ops or network-admin or vdc-admin*
• Strongly recommended: Some experience with shell/Linux-
Use with extreme care
145
Broadcom ASIC shell access on the Nexus 9000 !!!
• The Nexus 9000 is based largely on the Broadcom Trident II ASIC-Known as T2
• The modular unit Fabric Modules (FM) and Line Cards (LC) each contain multiple
instances of the T2 ASIC, as well as the TOR (top of rack) units
• Access is provided to each and every instance of the T2 ASIC
• No additional license is required to access the bcm-shell
• Permitted by default role network-admin
• Role based access control (RBAC) can be used to limit user access
• Accounting log available for BCM activity
146
BCM Access some Examples hg0 hg11 hg0 hg11
T2 T2
Instance 0 Instance 1
N9K# bcm-shell mod 6 "show unit"

F F F F FQSPF F F F F F F F F F F QSPFF F F F F
FP FP FPFP
P P P P PPorts P P P P P P P P P P Ports
P P P P P
Unit 0 chip BCM56852_A2 (current) 01 02 03 04 05 06 07 08 09 10 11 12 13 14
15
16
17
18
19 20
21 22 23 24
Unit 1 chip BCM56852_A2 Xe0 Xe11 Xe0 Xe11
Eth1/1 Eth1/12 Eth1/13 Eth1/24

N9K# bcm-shell mod 6 "ps" | in 19
xe19 up 1G FD SW Yes Disable None FA XGMII 1582
N9K# show accounting log | last 2
Mon Apr 20 08:31:52 2015:type=update:id=console0:user=admin:cmd=bcm-shell
module 6 "show unit" (SUCCESS)
Mon Apr 20 08:32:14 2015:type=update:id=console0:user=admin:cmd=bcm-shell
module 6 "ps" | in 19 (SUCCESS)
147
BCM Access some Examples (Cont’d)
N9K# bcm-shell mod 21 "config show l3"
l3_alpm_enable=2
l3_max_ecmp_mode=1
l3_mem_entries=16384
N9K# bcm-shell mod 4 "config show l2 ”
l2xmsg_hostbuf_size=16384
l2_mem_entries=98304
148
Python !!!!
• Python is - Established, Modern and Powerful, Clean, lots of libraries, liberal
license
• Perl is available in gdb images only – not available in final images
• Tcl is there but no one uses it in NX-OS
• The license that Python has (GPL-Like with very few restrictions on modification,
distribution and commercial use) make it very attractive to embed and distribute
• On the box applications that can currently use Python scripts

• Embedded Event Manager
• Power On Auto Provisioning (POAP)
• Create your own scripts that are like “Super commands”
• Create your own command modifiers – the things that act on commands applied with a
pipe “|”
149
Python-Continued
• There are two Python environments on the N9000
• One executed from VSH
• One executed from Bash
• Both run in their own forked process

• The main differences comes from the environment that they get initialized into
• These differences between them should be minimal
• There is a sandbox that should primarily contain lower privileged users
• Network-admin users get basically a “pure” 2.7.5 python environment
• That sandbox mostly applies to lower privileged users, they may be prevented from doing certain things
in python
• Also prevents file operations on files outside of bootflash
150
Python-Example
N9K# python
switching between VSH and Python
Python 2.7.5 (default, Oct 8 2013, 23:59:43)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or
"license" for more information.
>>> switch between VSH and the
N9K# run bash python Interpreter (Bash 1)
Python 2.7.5 (default, Oct 8 2013, 23:59:43)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or
"license" for more information.
>>>
151
Python Script Example
152
Why Patching?
Many customers spend extensive time and effort to test and qualify software prior to deployment. In today’s
environments, if a defect is found, effectively root-caused, and integrated, since it is rolled out through a
maintenance release, customers would need to restart their qualification cycle, wasting time, and pushing out
deployment dates…
Bug Found, Diagnose, Root Maint. Released

Cause Restart Qual Cycle Actual Deployment
Begin Code Test & Defect Resolved, integrated Target Deployment

Qualification Cycle into Maint.
6 Months
153
10 Months
NX-OS Image Patching
The Nexus9000 Standalone platforms introduces new patching capabilities that allows specific defects to be
rolled out in an independent package that can be applied to existing base software binaries. This will help
reduce customer code certification times, leading to greater customer satisfaction.
Bug Found, Diagnose, Root Continue Qual

Cause With additional tests Actual Deployment
Begin Code Test & Defect Resolved, Patch Target Deployment

Qualification Cycle Released
6 Months
7 Months
154
Patching Overview
• NXOS platforms release major versions when introducing new features and engineering
special builds to provide bug fixes.
• The new goal will be to allow customers to deploy patches for specific fixes only without
affecting the data plane of the device.
• The patching architecture comes from IOS XR (SMU – Software Maintenance Upgrade)
used to deliver Quick, Effective and Focused patches for specific sections of code.
• Both binaries and libraries can be patched.
• Supervisors and Line Card services can be patched.
• Software patching will leverage process restart/reload or ISSU
155
Patch Uninstall Workflow - Detailed
• User invokes “install deactivate <patch_name>”
• System manager gracefully shuts down each impacted process
• Softlinks are changed from active SMU to one in backup folder (if present).
• Relevant SMU is removed from the /var/installer/activated/SMU directory.
• System Manager triggers restart of impacted processes
• (Optional) “install remove” deletes the patch from the local repository
156
CLI Commands – Patch Install
Command Syntax Function Notes
Install add install add <uri> [activate] Download patch from URI and add Only one patch can be added at
patch to repository. a time. Optionally can activate
patch in this step.
Install remove install remove [<package> | User can remove only non- Confirmation y/n will be prompted
inactive] activated patches
Install activate install activate <package> [test] Installs a patch from the local Only one patch can be activated
repository. If not present, an error at a time. No show commands
will be returned. permitted during operation.
Install deactivate install deactivate <package> Uninstall patch and move it to non- Only one patch can be
activated repository deactivated at a time. **Patches
must have no other patch
dependencies
Install commit install commit Preserves all activated patches Activated patches are committed
across reloads. to a list kept in the patch
repository
157
CLI Commands – Show Commands
Command Function Sample
show install request Shows current install operation along Fri May 10 09:06:55.921 UTC
Install operation 13 '(admin) ‘install activate n9000-dk.6.0.2.U1.1.CSCuf08219.bin’
with time stamp, package name, Started by user 'cisco' via CLI at 09:06:48 UTC Fri May 10 2013
The operation is 10% complete
initiating user and % complete.
show install log [id | detail Shows user information on previous Install operation 1 by user ‘admin’ at Tue Sep 28 01:37:02 2004:
install commit
| from | last | reverse] installation operations. Optional [detail] Operation completed successfully
command for verbose information. Install operation 2 by user ‘admin’ at Mon Oct 18 17:26:36 2004:
install add tftp://10.52.241.252/bcarter/n3000-uk9.6.0.2.U1.1.CSCuf08219.bin
Operation completed successfully
Install operation 7 by user ‘lab’ at Mon Oct 18 17:31:13 2004:
install activate n3000-uk9.6.0.2.U1.1.CSCuf08219’
Operation failed because service failed to come up.
show install active [on- Displays boot images and active or switch# show install active
Boot Images:
reload] committed patches Kickstart Image: bootflash:/n9000-dk.6.1.234.gbin
System Image: package:/isanboot/bin/images/sys
Active Packages:
n9000-dk.6.1.1.CSCui56298.bin
158
CLI Commands – Show Commands (Cont’d)
Command Function Sample
show install inactive [on- Shows patches in the repository not switch# show install inactive
Boot Images:
reload] yet activated Image: bootflash:/inseor.6.1.1.234.gbin
System Image: package:/isanboot/bin/images/sys
Inactive Packages:
switch#
show install pkg-info Shows details of a specific patch. switch# show install pkg-info n9000-dk.6.1.1.CSCui56298.bin
<package> Requires that patch has been added Contents of Package file 'n9000-dk.6.1.1.CSCui56298.bin':
using ‘install add’ first. Expiry date : Jan 19, 2015 02:55:56 UTC
Uncompressed size : 17892613
Vendor : Cisco Systems
Desc : Bug Fix for CDET: CSCui56298
Build : Built on Wed May 10 08:04:58 UTC 2013
Source : By n9k-infra-bld
Platform: Nexus-9000.
Supersedes: n9000-uk9.6.1.1.U1.1.CSCuf09119, n9000-uk9.6.1.1.U1.1.CSCuf02229
Pre-requisite: n9000-uk9.6.1.1.U1.1.CSCuf09219
Restart information: BGP process restart.
159
Sample Patch Install – Copy Patch to Switch
N9K# copy
scp://sdn@172.18.217.42/home/sdn/n9k/inseor_CSCuxP1fix.6.1.2.I1.2.CSCab00001.gbin
bootflash:
Enter vrf (If no input, current vrf 'default' is considered): management
sdn@172.18.217.42's password:
inseor_CSCuxP1fix.6.1.2.I1.2.CSCab00001.gbin 100% 233KB
232.7KB/s 00:01
Copy complete, now saving to disk (please wait)...
N9508#
N9508# dir | grep .gbin

238230 Jan 15 10:52:31 2014inseor_CSCuxP1fix.6.1.2.I1.2.CSCab00001.gbin
N9508#
160
Sample Patch Install – Add patch to repository & verify
N9K# install add bootflash:inseor_CSCuxP1fix.6.1.2.I1.2.CSCab00001.gbin
Install operation 19 completed successfully at Wed Jan 15 10:55:14 2014
N9508#
N9K# show install packages

-----------------------------------------------------------
inseor_CSCuxP1fix.6.1.2.I1.2.CSCab00001.gbin inactive-commit
Modules
Module #27: inactive-commit
Module #28: inactive-commit
-----------------------------------------------------------
N9K# show install inactive
Inactive Packages:
inseor_CSCuxP1fix.6.1.2.I1.2.CSCab00001.gbin
N9K#
161
Important Limitations
• For every Feature please review Guidelines and Limitations
• Cisco Nexus 9000 Series NX-OS Verified Scalability Guide
• http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/7-
x/scalability/guide_703I21/b_Cisco_Nexus_9000_Series_NX-OS_Verified_Scalability_Guide_703I21.html
• EPLD Upgrade are recommended but are not mandatory
• User Configured MAC address for SVI- Packets will not be flooded if Layer 2 Adjacency is missing
• ASIC Memory-NS test is applicable only for the N9K-X9564PX and N9K-X9564TX line cards.
• Priority flow control (PFC) is supported on Cisco Nexus 9500 Series switches with the N9K-X9636PQ line card.
• Cisco Nexus 9500 Series Switch can run in 8-queue mode only if all of its line cards are capable of running 8-queue
mode.
162
Recap
• Introduction
• Architecture
• System Health check Telemetry
• Troubleshooting Toolkit
• Nexus 9000 Troubleshooting
• Common Link Layer Issues-L1
• Fabric Connectivity and
• In band
• L2/L3 Packet Forwarding
• vPC
• Nexus9000 Specific Limitation and Goodies
163
Core Message
• Technical
Services Helps to solve
complex networking issues
164
Continue Your Education
• Demos in the Cisco campus
• Walk-in Self-Paced Labs
• Table Topics
• Meet the Engineer 1:1 meetings
• Related sessions
165
Complete Your Online Session Evaluation
• Please complete your online session
evaluations after each session.
Complete 4 session evaluations
& the Overall Conference Evaluation
(available from Thursday)
to receive your Cisco Live T-shirt.
• All surveys can be completed via

the Cisco Live Mobile App or the
Communication Stations
Thank you
167
Backup Slides
Backup Slides !!!!
169
Fabric Module
Fabric Module for Nexus 9504
NFE
NFE NFE
Chassis Type Nexus 9504 Nexus 9508 Nexus 9516

NFE NFE NFE NFE
NFEs per Fabric Module 1 2 4
170
Nexus 9500 Platform FRU- Line Card
Connect to Fabric Modules
12 x 42 Gbps

N N N N N N N N N N N N
F F F F F F F F F F F F
E E E E E E ALE 1
E E E E E E
12 x 42
Gbps
1 x 42 1 x 42 NFE 1
Gbps Gbps
18x 40
Gbps
Ethernet
Network
Interfaces
NFE NFE NFE
18x 40Gbps
12 x 40 Gbps 12 x 40 Gbps 12 x 40 Gbps
Connect to Hosts or
Network
171
N9K-X9636PQ
FM1 FM2 FM3 FM4 FM5 FM6
HG Ports HG Ports HG Ports
T2 T2 T2
Instance 0 Instance 1 Instance 2
QSPF Ports QSPF Ports QSPF Ports

FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP
01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36
172
N9K-X9464PX
FM2 FM3 FM4 FM6
MUX1-2 MUX3-4
HG Ports HG Ports
T2
10G SFP+ Ports 40G QSFP

FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52
173
N9K-X9464TX
FM2 FM3 FM4 FM6
MUX1-2 MUX3-4
HG Ports HG Ports
T2
100/1000/10000 T Ports 40G QSFP

10G 10G 10G 10G 10G 10G 10G 10G 10G 10G 10G 10G FP FP FP FP
PHY PHY PHY PHY PHY PHY PHY PHY PHY PHY PHY PHY
49 50 51 52
FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
174
N9K-X9432PQ
FM2 FM3 FM4 FM6
HG Ports HG Ports
T2 T2
Instance 0 Instance 2
QSPF Ports QSPF Ports

FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP
01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32
175
N9K-X9564PQ
HG MUX1 HG MUX4 HG MUX2 HG MUX5 HG MUX3 HG MUX6
0123 4567 8 9 10 11 0123 4567 8 9 10 11

MN Port MN Port
MF Port MF Port
0-2 3-5 6-8 9-11 0-2 9-11
7-5 2-0 31-29 26-24 7-5 26-24

Warpcore
T2 T2
40G QSFP
10G SFP+ Ports FP FP FP FP

49 50 51 52
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
176
N9K-X9564TX
HG MUX1 HG MUX4 HG MUX2 HG MUX5 HG MUX3 HG MUX6
0123 4567 8 9 10 11 0123 4567 8 9 10 11

MN Port MN Port
MF Port MF Port
0-2 3-5 6-8 9-11 0-2 9-11
7-5 2-0 31-29 26-24 7-5 26-24
T2 T2
40G QSFP
100/1000/10000 T Ports
10G 10G 10G 10G 10G 10G 10G 10G 10G 10G 10G 10G
PHY PHY PHY PHY PHY PHY PHY PHY PHY PHY PHY PHY
FP FP FP FP
49 50 51 52
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
177
Multicast L3 Forwarding
• Before hardware can forward any Multicast packets,
forwarding information has to propagate from Sup to the LC Supervisor
• Several layers are to be verified: PIM IGMP MSDP
MRIB (control-plane is created here)
MRIB
MFDM PI /PD (platform independent & forwarding MF DM
information)
IP FIB
• MFIB-IPFIB
• IP FIB process programs hardware:
FIB Table contains (*,G) and (S,G) forwarding entries and RPF T2 Line Card
information
GROUP table contains forwarding and pointers replication FIB Table MC VLAN Table
information (pointers to MC VLAN) IPMC_GR
MC VLAN tables contain replication information (~OIF lists)
Hardware (packets are forwarded here) & SDK
178
L2/L3 Multicast Packet Walk Lookup to resolve egr.
modules;
Sends one copy to each
Fabric Module egr. module;
Trident II
Lkup in Host Table
& L2 Table
Lookup for local

receiving ports;
Trident II Trident II replicate pkts onto
L2/L3 mcast lookup; IACL those ports.
Replicate pckts to local IACL Egress Q Egress Q
Traffic Traffic
receiving ports; Classification&
Send 1 copy to fabric Classification
Remarking
module; EACL & Remarking EACL
L2/L3 L2/L3
Lookup & Lookup &
pkt rewrite
Pkt rewrite
Examines ingress Parser Parser
packet. Get packet
headers for Network Interfaces Network Interfaces
processing.
10GE 40GE 10GE 40GE
179
Multicast L3 Forwarding-MRIB
N9K# show ip mroute 239.10.10.10 shared-tree

Supervisor
IP Multicast Routing Table for VRF "default”
PIM IGMP MSDP
(*, 239.10.10.10/32), uptime: 00:23:32, ip pim
Incoming interface: Ethernet6/1, RPF nbr: MRIB
13.13.13.1
Outgoing interface list: (count: 1) IP FIB
Ethernet6/52, uptime: 00:22:42, pim
MF DM
180
Multicast L3 Forwarding-mFDM PI-Supervisor
N9K# show forwarding distribution ip multicast route group 239.10.10.10 source
13.13.13.14 | in 13|Index
(13.13.13.14/32, 239.10.10.10/32), RPF Interface: Ethernet6/1, flags:
Outgoing Interface List Index: 1
N9K# show forwarding distribution multicast outgoing-

interface-list l3 1
Supervisor
Outgoing Interface List Index: 1 IGMP MSDP
PIM
Reference Count: 4 MRIB

Platform Index: 0xb00001 MF DM
Number of Outgoing Interfaces: 1 t6/52 IP FIB
181
Multicast L3 Forwarding IPFIB-Line card
N9K# show forwarding ip multicast route group 239.10.10.10 source
13.13.13.14 mod 6 | inc 239|Eth
(13.13.13.14/32, 239.10.10.10/32), RPF Interface: Ethernet6/1, flags:
Outgoing Interface List Index: 1 T2 Line Card
Outgoing Interface List Index: 0x1 FIB Table MC VLAN Table
IPMC_GR
Ethernet6/52
Mod 6 is N9K-X9564TX
To reach Ethernet 6/52 which is on NS from front port of T2,Packets
need to cross Fabric module
182
Multicast L3 Forwarding Entries on LC –BCM Shell
N9K# bcm-shell mod 6 "ipmc table show"
SRC IP ADDRESS MC IP ADDRESS MC GROUP VID VRF COS HWIDX CLASS HIT
13.13.13.14 239.10.10.10 0x2000007 0 1 0 75680 1 no
0.0.0.0 239.10.10.10 0x2000007 0 1 0 86578 2 no
N9K#bcm-shell module 6 "mc show group=0x2000007" T2 Line Card
FIB Table MC VLAN
Group 0x2000007 (L3)
Table IPMC_GR
port hg0, encap id -1
-------snip------------
port hg11, encap id -1 Traffic spared to Hig towards Fabric
183
Multicast L3 Forwarding Entries on LC –BCM Shell
N9K# bcm-shell mod 6 " search l3_entry_ipv4_multicast group_ip_addr=0xef0a0a0a
source_ip_addr=0x0d0d0d0e”
L3_ENTRY_IPV4_MULTICAST.ipipe0[75680]:
SOURCE_IP_ADDR=0xd0d0d0e, T2 Line Card
GROUP_IP_ADDR=0xef0a0a0a, FIB Table MC VLAN Table
IPMC_GR
L3MC_INDEX=7
N9K# bcm-shell mod 6 " dum chg l3_entry_ipv4_multicast 75680”
show tech-support multicast`
IPV4MC:EXPECTED_L3_IIF=0x112e,
show tech-support forwarding multicast
N9K# show system internal eltm info interface ethernet 6/1 | in LIF
cr_flags = INTF LIF , LIF = 4398 (0x112e), LTL = 40959 (0x9fff) (S 0x0, P 0x0)
184
IGM Snooping
Forwarding programming in vPC Scenario
• IGMP Process Provides both Layer 3 IGMP Processing , and Layer 2 IGMP snooping functionality
• Receivers use IGMP (Internet Group Management Protocol) to report their multicast group
Membership to router
• Layer 2 IGMP Snooping functions of IGMP process include processing snooped multicast router
Packets Including IGMP reports and leaves sent by receiver
• Once the group membership is learned , the Supervisor Engine informs I/O modules , which
program Hardware
• This will Constrain data-plane multicast packets to only those ports with multicast routeror interested
receivers in HW
185
IGMP Snooping continued…
• BCM on FM are in Mode 4. This will have L2 Table size of 32K & L3 Host Table 16K
• L3 Host table will be used to program (*,G) /(S,G) entry. This will will accommodate
maximum of 8K entry.
• MFDM sends two OIF List information to MFIB. One for LC (S,G) OIF List and other for
FM ( Mac, Group) OIF List in PIM disable Vlan.
• MFIB will use (S,G) OIF list to program LC and Mac Group to Program FM in 32K L2 Table.
• If PIM is enable FM can accommodate 8K(VRF, S,G) and will program Hardware.
• Address aliasing is possible because on FM we use L2 table to program Mac Group information
186
IGMP Snooping (Cont’d)
• With vPC IGMP will have knowledge of multi chassis Ether Channel trunk (MCT) interface.
• When one of the vPC peer receives IGMP join , it will sync up this with peer over MCT link
using cFS-Cisco Fabric Services over Ethernet .
• Duplication of traffic crossing MCT is avoided using Port block Mask
• VPC Support PIM-SM Only
• For source in VPC domain – dual Forwarders are used
• For Source in Layer 3 Cloud , Unicast best metric determines active forwarder
• VPC Operational Primary in case of tie. CFS used to negotiate active Forwarder role
187
Configuration-IGMP Snooping enable by default
Nexus9508-13# sh ip igm snooping vlan 103
IGMP Snooping information for vlan 103
IGMP snooping enabled
Lookup mode: IP
Optimised Multicast Flood (OMF) enabled
IGMP querier present, address: 10.10.103.5, version: 2, i/f Po30
Nexus9508-13# sh ip igm snooping vlan 100
IGMP Snooping information for vlan 100
IGMP snooping enabled
Lookup mode: IP
Optimised Multicast Flood (OMF) enabled
IGMP querier present, address: 192.168.100.2, version: 2, i/f Vlan100
Querier interval: 125 secs
Querier last member query interval: 1 secs
188
Reference Topology for Troubleshooting
vPC Keep Alive
vPC Peer Link PO-10

N9508-12 N9508-13
Eth 3/1-2 Eth 3/1-2
Eth 6/9/1-4
Eth1/3/1-4
vPC30 vPC 35
Eth 1/17-18 ,Eth 1/33-34 Eth 1/17,Eth 1/19 , Eth 1/33-34

N93k N35K
Eth1/48
Eth 1/48
Ixia 10/2-Source Ixia 10/1-Receiver
189
IGMP Snooping Troubleshooting
• Stream will enter one of the VPC-Peer , Which will get forwarded across Peer link to other VPC Peer
• Both boxes will have (S ,G)
• Upon Creation of (S,G) , VPC Peers negotiate best metric
• Both realize source is VPC-Connected
• Install Entry as Win-Force
• If either peer gets a PIM/IGMP Join for the given source , they both add Interface to OIF
Nexus9508-12(config)# sh ip pim internal vpc rpf-source Nexus9508-13# sh ip pim internal vpc rpf-source
PIM vPC RPF-Source Cache for Context "default" - Chassis PIM vPC RPF-Source Cache for Context "default" - Chassis
Role Primary Role Secondary
Source: 192.168.100.10 Source: 192.168.100.10
Pref/Metric: 0/0 Pref/Metric: 0/0
Source role: primary Source role: secondary
Forwarding state: Win-force (forwarding) Forwarding state: Win-force (forwarding)
MRIB Forwarding state: forwarding MRIB Forwarding state: forwarding
190
vPC Peer receiving Join
• IGMP Join from one of the receiver enter one of the VPC Pee.
• This Peer encapsulates IGMP in CFS , sends to other Peer
• Both Peer have identical State
• Both Peer install OIF
• Data traffic flows down to Receiver, also forwarded to other Peer on Peer Link
• Other Peer drop the packet either by PORT BLOCK MASK blocking or no OIF
Nexus9508-ESC-12# sh ip mroute 239.10.10.10 192.168.100.10 Nexus9508-ESC-13# sh ip mroute 239.10.10.10 192.168.100.10
IP Multicast Routing Table for VRF "default" IP Multicast Routing Table for VRF "default"
(192.168.100.10/32, 239.10.10.10/32), uptime: 01:00:09, ip pim (192.168.100.10/32, 239.10.10.10/32), uptime: 04:25:36, ip pim
mrib mrib
Incoming interface: Vlan100, RPF nbr: 192.168.100.10, uptime: Incoming interface: Vlan100, RPF nbr: 192.168.100.10, uptime:
01:00:09, internal 04:25:36
Outgoing interface list: (count: 1) Outgoing interface list: (count: 1)
Vlan101, uptime: 00:59:40, mrib Vlan101, uptime: 02:04:41, mrib
Nexus9508-ESC-12# Nexus9508-ESC-13#
191
Step to verify PI On Supervisor. Verify on Both Peers
Nexus9508-ESC-12# sh ip igmp groups 239.10.10.10
IGMP Connected Group Membership for VRF "default" - matching Group "239.10.10.10"
Type: S - Static, D - Dynamic, L - Local, T - SSM Translated
Group Address Type Interface Uptime Expires Last Reporter
239.10.10.10 D Vlan101 00:01:23 00:02:56 192.168.101.13
Nexus9508-ESC-12#
Nexus9508-ESC-13# sh ip igmp groups 239.10.10.10
IGMP Connected Group Membership for VRF "default" - matching Group "239.10.10.1
0"
Type: S - Static, D - Dynamic, L - Local, T - SSM Translated
Group Address Type Interface Uptime Expires Last Reporter
239.10.10.10 D Vlan101 00:01:18 00:03:01 192.168.101.13
Nexus9508-ESC-13#
192
CFS Provide info
Nexus9508-ESC-12# sh ip igmp snooping groups vlan 101 detail
IGMP Snooping group membership for vlan 101
Group addr: 239.10.10.10
Nexus9508-ESC-13# sh ip igm snooping groups vlan
Group ver: v2 [old-host-timer: not running] 101 det
Last reporter: 192.168.101.10 IGMP Snooping group membership for vlan 101
IGMPv2 member ports: Group addr: 239.10.10.10
IGMPv1/v2 memb ports: Group ver: v2 [old-host-timer: not running]
Po35 [1 GQ missed], cfs:false, native:true Last reporter: 192.168.101.10
vPC grp peer-link flag: exclude IGMPv2 member ports:
M2RIB vPC grp peer-link flag: exclude IGMPv1/v2 memb ports:
Nexus9508-ESC-12# Po35 [0 GQ missed], cfs:true, native:false
vPC grp peer-link flag: exclude
M2RIB vPC grp peer-link flag: exclude
Nexus9508-ESC-13#
193
Verifying Multicast forwarding Distribution Module
Platform Independent On Supervisor
Nexus9508-ESC-12# sh forwarding distribution multicast route group 239.10.10.10
source 192.168.100.10
(192.168.100.10/32, 239.10.10.10/32), RPF Interface: Vlan100, flags:

Received Packets: 1073 Bytes: 36977
Number of Outgoing Interfaces: 2
Outgoing Interface List Index: 10 Not showing PC 10 for Vlan 101 because of
Vlan100 exclude flag seen while checking igmp
( Mem L2 Ports: port-channel10 ) snooping stats.
Vlan101
( Mem L2 Ports: port-channel35 )
Note: On shutting down local vpc only, igmp does not send update to mfdm/ipfib to update the mroute state.
That is why you did not see mfdm/ipfib removing local vpc. So if local leg of vPC is down we will still PC in the above output.
194
Platform Independent On Supervisor-(Cont’d)
Nexus9508-12# sh forwarding multicast route group 239.10.10.10 source 192.168.100.10 mod 1
Number of next hops: 2
Vlan: 101
port-channel35
bridged Vlan
port-channel10
Hardware Outgoing Interface List Index: 33554443
195
Platform Independent On Supervisor-IGMP-Snooping
Nexus9508-12# sh forwarding distribution ip igmp snooping vlan 101 group 239.10.10.10 det
Vlan: 101, Group: 239.10.10.10, Source: 0.0.0.0
Reference Count: 1 Nexus9508-13# sh forwarding distribution ip igmp snooping vlan 101 group
Platform Index: 0xa00004 239.10.10.10 det
Vpc peer link exclude flag set Vlan: 101, Group: 239.10.10.10, Source: 0.0.0.0
Number of Outgoing Interfaces: 2 Outgoing Interface List Index: 5
port-channel10 Reference Count: 1
port-channel35 Platform Index: 0xa00005
Vpc peer link exclude flag set
port-channel10
port-channel35
196
Verifying Multicast Forwarding Distribution Module
Platform Independent On Supervisor-Snooping Group.
Nexus9508-12# sh forwarding distribution l2 multicast mac-based vlan
101
Vlan: 101, Group: 0100.5e0a.0a0a, Source: 0000.0000.0000
Reference Count: 1
Platform Index: 0xa00003 Nexus9508-13# sh forwarding distribution l2 multicast mac-based vlan 101
Vpc peer link exclude flag set Vlan: 101, Group: 0100.5e0a.0a0a, Source: 0000.0000.0000
Number of Outgoing Interfaces: 2 Outgoing Interface List Index: 8
port-channel10 Reference Count: 1
port-channel35 Platform Index: 0xa00008
Vpc peer link exclude flag set
port-channel10
port-channel35
197
IPFIB on LC for IGMP Snooping programming.
Nexus9508--12# sh forwarding multicast route group 239.10.10.10 source 192.168.100.10 mod 1

Nexus9508-13# sh forwarding multicast route group 239.10.10.10 source 192.168.100.10 mod 6
port-channel30 (Vlan: 101)
port-channel10 (bridged)
port-channel30 (Vlan: 101)
port-channel10 (bridged)
198
Verifying Hardware Programming
Nexus9508--12# bcm-shell mod 1 "mc show group=33554441" If we see encap id a positive #
Executing mc show group=33554441 on bcm shell on module 1 then it is LIF
Group 0x2000009 (L3)
port hg0, encap id 400005 If we see encap id = -1 then it is
port hg1, encap id 400005 L2 bridge copy.
port xe10, encap id 21
port xe11, encap id 21
Nexus9508-12# bcm-shell mod 3 "mc show group=33554441"
Executing mc show group=33554441 on bcm shell on module 3
Group 0x2000009 (L3)
port hg0, encap id 400005 Nexus9508-12# sh system internal eltm info interface vlan 101 | in LIF
port xe0, encap id -1 cr_flags = INTF VLAN , LIF = 21 (0x15), LTL = -1 (0xffffffff) (S 0x0, P 0x0)
port xe1, encap id -1 Nexus9508-ESC-12#
199
From BCM to check what is HW index for given Group
• Static entry of Mcast group
• Hit Bit indicate flow is present
• Mcast Index is where the traffic need to bridge
Nexus9508-12# bcm-shell module 1 "l2 show" | in MCast
mac=01:00:5e:0a:0a:0a vlan=101 GPORT=0x0 modid=0 port=0 Static Hit MCast=33554435
mac=01:00:5e:0a:0a:14 vlan=100 GPORT=0x0 modid=0 port=0 Static MCast=33554435
Nexus9508-12# sh ip igmp gr vlan 100
show tech-support ip igmp snooping

show tech-support ip multicast
200
Expert Suggestions on Creating Slides
• Make sure every slide is assigned to a layout from the new template.
• Reset slides to the correct layout using Home/Layout (both PC and Mac).
• Reset a slide back to the correct formatting use:
• Home/Reset (PC)
• Home/Layout/Reset Layout to Default Settings (Mac)
• Resetting a slide to the proper layout can resolve issues like disappearing
titles or misplaced bullets.
• If page numbers are not formatting correctly after the slides have been moved to
the new template and connected to the correct layout, then turn the slide
numbers off and then back on.
• Home/Replace (PC) Format/Replace Fonts (Mac) allows you to replace
fonts globally.
201
Thank you
202

BRKDCT 3101 PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

BRKDCT 3101 PDF

Uploaded by

Copyright:

Available Formats

Nexus9000(Standalone)

This session presents briefly the architecture of the latest generation

We will walk you through in depth troubleshooting Tools and Techniques.

• This session will introduce System Telemetry, Troubleshooting tool Kits

BRKARC-2222 - Cisco Nexus 9000 architecture

BRKARC-3471 - Cisco NX-OS Software Architecture

BRKDCT-3144 - Advanced - Troubleshooting Cisco Nexus

Nexus 9500 Series Switches Nexus 9300 Series Switches

Nexus9504/Nexus9508/Nexus9516 N9K-C9332PQ N9K-C9372PX N9K-C9372TX N9K-C9396

• The Supervisor, System controller ,Fabric Module and LC have OBFL

System Controller-What it is Role

• Have Various Forwarding Tables

NFE=Network Forwarding Engine-Trident 2(T2)

ALE=Application Leaf Engine-North Star(NS) 7

Note: Internal ports are called as Hi-Gig/HG ports 40G

Fabric 1 Fabric 2 Fabric 3 Fabric 4 Fabric 5 Fabric 6

2 x 42 Gbps N = 4 for N9516

NFE NFE NFE NFE

Note: Internal ports are called as Hi-Gig/HG ports

Number of ASIC 3 T2 2 T2 + 2 NS 2 T2 40 gig 32 Ports

Non Blocking Non Blocking Line rate > 200 byte

All PSU, SC, SUP, FM, and

• Port QSFP+ Uplink Module

• AC/DC Power Supply

• Front-to-Back & Back-to-Front Airflow

• Wire-Speed L2/L3 Forwarding

• Switch will not boot up without GEM

Front Panel 48x 1GE/10GE Ports

(12 x 40G) = 480G

(12 x 40G) = 480G FP

L2 L2/L3 L3 Hosts IPv4:16K min-112Kmax

128 Integrated SerDes Host IF Maximum number of Physical ports 104

Forwarding 720Mpps lookup rate on Ingress

Maximum number of Physical ports 24

0 288K 16K 16K

1 224K 56K 16K

2 160K 90K 16K

3 98K 122K 16K

4 32K 16K 128K

N93K#show system routing mode

show hardware internal forwarding table utilization mod 1

• Egress ACL: 1K TCAM entries - 4x 256 banks

• Each ACL type needs its own dedicated bank/banks

• IPv4, IPv6 or MAC each needs dedicated bank/banks

• VACL is programmed symmetrically in both egress and ingress ACL

L3 TCAM Shared TCAM Shared

• TCAM Carving requires Line Card/TOR reload to take effect

• To read current TCAM allocation

• To reconfigure TCAM Region

OOBFC: Out of band flow control unicast service pool

Front Panel 48x 1GE/10GE Ports

1/10GE 1/10GE 1/10GE 1/10GE

N9K#show system internal memory-alerts-log Make sure log is clean

Provides top process using CPU cycle

N9K#run bash Auto update

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

2 root 20 0 0 0 0 S 0 0.0 0:00.00 kthreadd

3 root 20 0 0 0 0 S 0 0.0 0:00.58 ksoftirqd/0

N9K#sh sockets client detail | inc pim|drops|Errors

N9K# show system inband queuing statistics | in drop

show hardware internal buffer info pkt-stats cpu • Total 48 Queues

N9K# show system internal emon stats

Default Policy 9 policy entries