Advanced Troubleshooting of The ASR1k and ISR (Including SDWAN Edge) Made Easy

Advanced troubleshooting
of the ASR1k and ISR

(including SDWAN Edge)
made easy
Frederic Detienne, Distinguished Engineer

Olivier Pelerin, Technical Leader
BRKCRS-3147
#CLUS
About us
#CLUS BRKARC-3147 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 2
About us
Agenda
• Platform and Hardware Architecture
• Software Architecture
• Resource troubleshooting
• Data plane troubleshooting
• Wrapping up...
Cisco Webex Teams
Questions?
Use Cisco Webex Teams to chat
with the speaker after the session
How
1 Find this session in the Cisco Live Mobile App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space
Webex Teams will be moderated cs.co/ciscolivebot#BRKARC-3147

by the speaker until June 16, 2019.
#CLUS © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
Debugging Strategies to Date
Top Down
IOS Control Plane
• ACL + show access-list,…

• show interface / ip route / bgp …
Rock bottom
Platform Control Plane
• ESP “stuff” Let’s

• e.g. show platform … hard to remember
change
that!!
Data Plane
• ESP “stuff”
• More arcane show platform …
Everyday situations
IPsec ZBF NAT
WAAS SNMP
OTV
First Routing
Which feature went wrong ?
What went wrong in the feature ?

Memory
Config Performance Ordering
Bug
Traffic Ambiguity
issue
Everyday situations Second

Memory
Bug
Traffic Ambiguity
Pattern
ASR Series
Hardware
Architecture
ASR 1000 Series
New Chassis
ASR 1009-X
ASR 1006-X
ASR 1013
ASR 1006
ASR 1004
ASR 1002 ASR 1002-X
ASR 1001-X
SHARED PORT ADAPTERS

RP-3
Interface Flexibility
RP-2
RP-1
End of Sales
‘
200 G
100G
40G
20G
10G
2.5-5G
ASR1K Building Blocks
ESP RP RP ESP
FECP CPU CPU FECP
Active
Stby
Active
Stby
QFP interconn. GE switch interconn. GE switch QFP
Crypto Crypto
QFP QFP
Assist. PPE BQS Assist. PPE BQS
Route Processor
interconn. Handles control plane traffic interconn.
Manages system
Embedded Service Processor

Handles forwarding plane Midplane
traffic
SIP interconn. SIP interconn. SIP interconn.
SPA SPA SPA

IOCP IOCP IOCP
Aggreg. Aggreg. Aggreg.
SPA SPA SPA SPA SPA SPA
SPA Interface Processor

Houses SPA’s
Queues packets in & out (FIFO)
System Architecture Control Plane
ESP RP RP ESP
FECP CPU CPU FECP
Active
Stby
Active
Stby
Crypto Crypto
QFP QFP
interconn. interconn.
Midplane
Ethernet Out of Band Channel
(aka EOBC)
Inter Integrated Circuit (I2C) Bus 1Gbps Ethernet bus
SIP interconn. SIP interconn. SIP interconn. Used by RP to program system
Slow (few kbps)
Used for system monitoring Used by system to notify RP
(temp., OIR, fan speed,…)
SPA SPA SPA
IOCP IOCP IOCP
System Architecture Forwarding Plane
ESP RP RP ESP
FECP CPU CPU FECP
Active
Stby
Active
Stby
Crypto Crypto
Embedded Service Interconnect
aka ESI Bus
interconn. 11.2 – 200 Gbps Forwarding Bus interconn.
Centralized Architecture Midplane

All traffic flows through ESP
SIP interconn. SIP interconn. SIP interconn.
SPA SPA SPA

IOCP IOCP IOCP
RP
CPU
Route Processor Architecture interconn. GE switch
Highly Scalable Control Plane Processor

Route Processor System
Manages all chassis functions Not a traffic interface! Logging
Runs IOS Management only Core Dumps
Mgmt Console BITS

USB 2.5’’
Ethernet & Aux (input & output)
Hard disk
Card Infrastructure
Runs IOS, Linux OS
Manages boards and
chassis 33MB
IOS Memory: RIB, FIB & NVRAM
other processes RP1: 1GB
Determines BGP routing CPU CPU Bootdisk RP2: 2GB
table size
RP1: 4GB Memory Multi-Core
RP2: 8&16GB Stratum-3 Network
clock circuit GE, 1Gbps
I2C Chassis I2C
Management Bus ESI EOBC SPA Control
Interconnect Gig Eth Switch Output Input SPA Bus
clocks clocks
ESI, 11.2-40 Gbps
SPA-SPI, 11.2Gbps
Hypertransport, 10Gbps
Other
SIPs ESPs RP Misc ESPs SIPs ESPs RP SIPs SIPs RP
Ctrl
ESP
FECP
ESP20 Block Diagram Crypto

Assist. PPE
QFP
BQS
intercon.
Reset / Pwr Ctrl Packet Buffer

TCAM Resource DRAM Part Len / BW
DRAM
(10Mbit) (512MB) SRAM
Temp Sensor (128MB)
EEPROM
QFP
DDRAM Packet Processor Engine BQS
Boot Flash
(OBFL,…) FECP E-CSR
PPE1 PPE2 PPE3 PPE4 PPE5
JTAG Ctrl PPE6 PPE7 PPE8 … PPE40
PCI* E-RP*
Dispatcher
Global Packet GE, 1Gbps
Packet Buffer Memory
Two cases were I2C
SPA Control
Crypto escalated to us this year SPA Bus
(Nitrox-II CN2430)
SPI Mux
Reset / Pwr Ctrl ESI, 11.2Gbps
SA table SPA-SPI, 11.2Gbps
DRAM Interconnect Interconnect Hypertransport, 10Gbps
Other
RPs RPs ESP RPs SIPs

ESP
FECP
ESP20 Block Diagram (comments) Crypto

Assist. PPE
QFP
BQS
intercon.
Forwarding Engine Control
Quantum Flow Processor
Processor
Reset / Pwr Ctrl OverallDRAM Packet Buffer
packet forwarding
TCAM Resource Part Len / BW
Manages board (10Mbit) (512MB)
DRAM
SRAM
(128MB)
ProgramsTemp
QBS,Sensor
PPE, Crypto
Linux Kernel
EEPROM
QFP
Buffering Queuing & Scheduling
Executes complex QoS scheduling
(shapers, LLQ’s,…)
Queues and schedules packets in
Boot Flash
due time
PCI* E-RP* Packet Processor Engine

Multicore CPU
Dispatcher Routes and applies features to packets
Two cases were I2C
SPA Control
Crypto escalated to us this year SPA Bus
(Nitrox-II CN2430)
SPI Mux
Other

Embedded Services Processor – The Real Thing
Interconnect ASIC
SPI MUX
TCAM Crypto
Engine
FECP
QFP Subsystem CPU
PPE + BQS
FECP
DRAM
PPE BQS
DRAM Packet
DRAM
Cisco “Quantum Flow Processor”
• Packet Processing Engine (QFP-PPE)
– 40-64 Packet Processors with 4 threads per core
– Up to 1.2GHz Tensilica ISA processors + DRAM packet
memory
– Single TCAM4 I/F; can cascade 1-4 devices
– C-language for feature development; extensive development
Multi-Core (40) Packet Processor
support tools
– HW assist for flow-locks, look-ups, stats, WRED, policers,
range lookup, crypto, CRC
• Buffer/queue subsystem (QFP-BQS)
– HW hierarchical 3-parameter (min, max & excess) scheduler
– Fully configurable # of layers based on HQF
– Priority propagation through the multiple layers
Traffic Manager (BQS)

ESP200 Block Diagram
Packet Buffer Packet Buffer
TCAM Resource DRAM Resource DRAM
DRAM DRAM
(80Mbit) (2GB) (2GB)
(512MB) (512MB)
Reset / Pwr Ctrl QFP QFP

Packet Processor Engine BQS Packet Processor Engine BQS
Temp Sensor PPE PPE PPE PPE PPE PPE PPE PPE PPE PPE
1 2 3 4 5 1 2 3 4 5
EEPROM PPE PPE PPE

6 7 8
… PPE
64
PPE PPE PPE
6 7 8
… PPE
64
Dispatcher Dispatcher
DDRAM Packet Buffer Packet Buffer
Boot Flash Packet Buffer Packet Buffer GE, 1Gbps

(OBFL,…) FECP TCAM
(80Mbit)
Resource DRAM
(2GB)
DRAM
(512MB)
Resource DRAM
(2GB)
DRAM
(512MB)
I2C
SPA Control
SPA Bus
JTAG Ctrl QFP QFP
Packet Processor Engine BQS Packet Processor Engine BQS ESI Bus 23Gbps
SPA-SPI, 11.2Gbps
PPE PPE PPE PPE PPE PPE PPE PPE PPE PPE
1 2 3 4 5 1 2 3 4 5 Hypertransport, 10Gbps
Other
PPE PPE PPE
6 7 8
… PPE
64
PPE PPE PPE
6 7 8
… PPE
64
Dispatcher Dispatcher
Packet Buffer Packet Buffer
Reset / Pwr Ctrl

Memory Crypto Pkt Re-
Dispatcher Interconnect order Logic
Memory Crypto
RPs RPs SIPs
ESP RPs
SIP intercon.
SPA
IOCP
Aggreg.
SIP40 Block Diagram SPA SPA
ESPs RPs RPs
Reset / Pwr Ctrl

EV-RP
Interconnect
Temp Sensor EV-FC
In ref clocks
EEPROM
DDRAM Egress
Ingress
Buffer
Boot Flash IOCP Scheduler Status
(OBFL,…) (SC854x SOC)
JTAG Ctrl
SPA Aggregation Network
clock
ASIC (Marmot) … distribution
…
Ingress buffers Egress buffers

(per port) (per port)
Network
Ingress clocks
Reset / Pwr Ctrl SPA Agg.

Classifier C2W
ESI, 23Gbps GE, 1Gbps

RPs RPs SPA-SPI, 11.2Gbps
4 SPAs 4 SPAs I2C 4 SPAs 4 SPAs 4 SPAs
Hypertransport, 10Gbps SPA Control
Other SPA Bus
SIP intercon.
SPA
IOCP
Aggreg.
SIP40 Block Diagram (comments) SPA SPA
ESPs RPs RPs
Reset / Pwr Ctrl

EV-RP
Interconnect
Temp Sensor EV-FC
In ref clocks
EEPROM
DDRAM Egress
Ingress
Buffer
Boot Flash IOCP Scheduler Status SPA Aggregation
(OBFL,…) (SC854x SOC) Queues packets in & out
Uses Ingress and Egress buffers
JTAG Ctrl
IO Control Processor ASIC (Marmot) clock
Manages SPA OIR & drivers … distribution
…
Linux Kernel Egress Packet Buffers
Ingress buffers Egress buffers Holds packets if SPA backpressures
(per port) (per port) (e.g. Pause frames)
Network
Ingress clocks
Ingress Packet Buffers SPA Agg.
Reset / Pwr Ctrl Classifier C2W
Holds packets to ESP
Hi & Lo priority queues (1K only)
Other SPA Bus
ESI Capacity by ESP-xxx and SIP-xxx
• Enhanced SerDes Interconnect (ESI) links over midplane carry
• packets between ESP and other cards (SIPs, RP & other ESP)
QFP Complex • network traffic to/from SPA SIP’s
• punt/inject traffic to/from RP
11.2Gbps 11.2Gbps 25.6Gbps 40+G I/L
SPI4.2 SPI4.2 eSPI • state synchronization to/from standby
ESP-10G Interc. ESP-10G Interc.
• Additional full set of ESI links to/from standby ESP (not shown)
ESP-20G Interconnect • CRC protection of packet contents
• ESP-10G: 1x11.5G ESI to each SIP slot

ESP-40 G Interconnect
• ESP-20G: 2x11.5G ESI to two SIP slots; 1x11.5G to third SIP slot
• ESP-40G:
• 2x23G ESI* to all three SIP slots
• could also support a 6-SIP chassis with 1 ESI to each (e.g. voice
application)
• also 23G between two ESP-40G’s
Other RP1 RP0 SIP0 SIP1 SIP2 • SIP-10G: supports 1x11.5G mode only
ESP ASR1004 ASR1006 • SIP-40G: supports 1x11.5G, 2x11.5G, 2x23G
ISR Series
Hardware
Architecture
Tighter Hardware – Same Software Architecture
4451
4331
ISR 4451-X Hardware Diagram
DDR3 Control Plane 4xPCIe Data Plane 4xSGMI

DRAM (4 cores) (10 core) FPGE
Ctrl SVC1 PPE1 PPE2 PPE3 PPE4 PPE5
DDR3
DRAM
SVC2 SVC3 PPE6 PPE7 PPE8 PPE9 PPE10
10 Gbps XAUI
System 1xSGMI
FPGA DSP
Mgmt Ethernet Multi Gigabit
Fabric
Console / Aux 10 Gbps/slot
Peripheral SM-X
Interconnect SM-X
USB
2Gb/slot
Flash
NIM
NIM
NIM
ISR 4451-X Hardware Diagram (Comments)
10 Cores, 1 thread / core Inline Cryptography
5 fwd cores by default No Crypto Assist chip
4 remaining cores license Crypto “locks” core
activated True run-to-completion
DDR3 Control Plane 4xPCIe Data Plane 4xSGMI

BQS onFPGE
a core
DRAM (4 cores) (10 core)
One Core dedicated to BQS
1 Control Plane Core Ctrl SVC1 PPE1 PPE2 PPE3 PPE4 PPE5 Always active
RP and FECP-like roles DDR3(5+1 or 9+1 cores)
DRAM
SVC2 SVC3 PPE6 PPE7 PPE8 PPE9 PPE10
3 Services Core
No hardware TCAM
10 Gbps XAUI
System 1xSGMI
FPGA DSP
Mgmt Ethernet Multi Gigabit
Fabric
Console / Aux 10 Gbps/slot
Peripheral SM-X
Interconnect SM-X
USB
2Gb/slot
Flash
NIM
NIM
NIM
ISR 44xx System Layout (2RU Platform)
Dataplane DIMM (left)
and
Controlplane DIMM (2x right)
6 or 10 core
Dataplane 4 Cores Control and
Services Plane
Compact Flash
Multi Gig
Ethernet
Fabric
1 SW-NIM or Dual HDD
Configurable Slot
Integrated
(@ factory only)
Services Card
(e.g. DSP)
4431 & 4451
Front panel PoE

power
The ISR 43xx Series
4351 Hardware Diagram
Rangeley CPU mSATA

DRAM
(MO-300)
PPE1 PPE2 PPE3 PPE4 PPE5 Console, Aux & USB

Mgmt Ethernet
Front Panel Ethernet System Glue Logic Console
Front Panel Ethernet
Front Panel Ethernet FPGA
PPE6 PPE7 PPE8
I2C to Modules
SPI Flash GE Switch
USB Host Ports PCIe Switch
eMMc
USB-to-SD
NIM Slots x 2 NGSM Slots x 2
NIM Slots x 2 NGSM slots x 2
4351 Hardware Diagram (Comments)
8 Cores @ 2.4 Ghz / 1 thread per core
1 core for RP/IOSd
1 core acting for Crypto & QoS
4 cores @ 1 thread/core for features
2 service cores
1 core
Rangeley as
CPU RP hosting IOSd mSATA
DRAM
(MO-300)
PPE1 PPE2 PPE3 PPE4 PPE5 2 service cores

Mgmt Ethernet Console, Aux & USB
PPE6 PPE7 PPE8
I2C to Modules
SPI Flash 1 core as Crypto and BQS GE Switch
2 cores QFP
2 cores QFP license activated
eMMc
USB-to-SD
NIM Slots x 2 NGSM Slots x 2 4331 and 4321 are similar; just less
cores and expansion slots
Rangeley CPU mSATA

DRAM
(MO-300)
PPE1 PPE2 PPE3 PPE4 PPE5 Console, Aux & USB

Mgmt Ethernet
PPE6 PPE7 PPE8
I2C to Modules
SPI Flash GE Switch
eMMc
USB-to-SD
8 Cores @ 2 Ghz / 1 thread per core
1 core for RP/IOSd
2 service cores
1 core
Rangeley as
CPU RP hosting IOSd mSATA
DRAM
(MO-300)
PPE1 PPE2 PPE3 PPE4 PPE5 2 service cores

Mgmt Ethernet Console, Aux & USB
PPE6 PPE7 PPE8
I2C to Modules
2 cores QFP
2 cores QFP license activated
eMMc
USB-to-SD
4 GB DRAM
Rangeley CPU mSATA
(MO-300)
4 GB Optional
PPE1 PPE2 Console, Aux & USB
Mgmt Ethernet
PPE3 PPE4
I2C to Modules
SPI Flash GE Switch
eMMc
USB-to-SD
NIM Slots
NIM Slots
4 Cores @ 2 Ghz / 1 thread per core
1 core for RP/IOSd
4 GB DRAM 1 core as RP hosting IOSd

Rangeley CPU mSATA
(MO-300)
4 GB Optional
PPE1 PPE2 Console, Aux & USB
Mgmt Ethernet
PPE3 PPE4
I2C to Modules
eMMc
USB-to-SD
NIM Slots
NIM Slots
ESP
FECP
Generic ESP Block Diagram Crypto

Assist. PPE
QFP
BQS
intercon.
Reset / Pwr Ctrl Packet Buffer Part Len / BW

TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
QFP Complex
Boot Flash
(OBFL,…) FECP PPE1 PPE2 PPE3 PPE4 PPE5
JTAG Ctrl PPE6 PPE7 PPE8 … PPEN
Dispatcher
GE, 1Gbps
Packet Buffer
I2C
SPA Control
Crypto SPA Bus
SPI Mux
DRAM Interconnect Hypertransport, 10Gbps
Other

Acronyms
• RP – Route Processor
• FP – Forwarding Processor = ESP (Embedded Service Processor)
• CPP – Cisco Packet Processor Compex= QFP (Quantum Flow Processor)
• PPE – Packet Processing Engine
• IOCP – I/O Control Processor
• FECP – Forwarding Engine Control Processor
• SPA – Shared Port Adapter
• SIP – SPA Interface Processor
• IOSd – IOS image that runs as a process on the RP
• FMAN – Forwarding manager (FMAN-RP, FMAN-FP)
• Scbac – FW Session Control Block
• EOBC = Ethernet Out of Band Channels – Packet Interface for Card to Card Control Traffic
• IOS-XE (BinOS) = Linux Based Software Infrastructure That Executes on MCP
Software
Architecture
ASR1K Software Architecture
RP
CPU
RP
Chass. Man. SDWAN
IOS CPU
ESI (10-40 Gbps)

Forwarding Manager
interconn.Linux GE switch
Kernel
ESP ESP
FECP Chassis Manager
EOBC (1 Gbps) FECP
Drivers Forwarding Manager
Drivers
I2C Drivers
Linux Kernel
Crypto
QFP
Assist.
QFP
µ µµ Crypto
µ BQS
µ µ Assist.
interconn.
ESI (10-40 Gbps)
SIP SIP
IOCP interconn.
SPA Driver Chassis
SPA Driver
SPA Driver Manager
SPA
IOCP
Aggreg.
Linux Kernel
SPA SPA SPA SPA SPA
Chassis Manager (CM)
RP
CPU
RP • CM on RP communicates with CM processes on
Chass. Man.
CPU
SDWAN ESP and SIP
IOS
• Distributed function
ESI (10-40 Gbps)
Forwarding Manager
Kernel • Initializes hardware and boots other processes
• CM on SIP queries SPA type and load SPA drivers
ESP ESP
FECP Chassis Manager • Manages hardware components
FECP
• Manages EOBC on RP
EOBC (1 Gbps)

Drivers
Drivers • Manages ESI links on RP/ESP/SIP
I2C
Linux Kernel
Crypto
QFP • Manages timing circuitry on RP
Assist.
QFP
µ µµ Crypto • Reset and power-down on RP/ESP/SIP
µ BQS
µ µ Assist.
interconn.
• Communicates IOS hardware components
ESI (10-40 Gbps)
SIP SIP
• Static & OIR
IOCP interconn.
SPA Driver Chassis
SPA Driver
SPA Driver Manager • Monitors environmental variables and alarms
SPA
IOCP
Aggreg.
Linux Kernel • Selects active/standby RP or ESP
SPA SPA SPA SPA SPA
• Coordinates switchover in case of failure or
operator command
Forwarding Manager (FMAN)
RP
CPU
RP
Chass. Man. SDWAN FMAN-RP
• FMAN on RP communicates with FMAN
IOS CPU
process on ESP
ESI (10-40 Gbps)
Forwarding Manager
Kernel
• Distributed function
FMAN-FP • Propagates control plane ops. to ESP
ESP ESP
FECP ESP aka Forwarding Plane
Chassis Manager
• CEF tables, ACL’s, NAT, SA’s,…
FECP
EOBC (1 Gbps)

Drivers
Drivers • FMAN-FP communicates information back
I2C
Linux Kernel
Crypto
QFP
to FMAN-RP
Assist.
QFP
µ µµ
µ BQS
Crypto • e.g. statistics
µ µ Assist.
interconn.
• FMAN-RP pushes info back to IOS
ESI (10-40 Gbps)
SIP SIP
IOCP interconn.
SPA Driver
SPA Driver
Chassis • FMAN on active RP maintains state for
Manager
SPA Driver
SPA
IOCP
both active & standby ESP’s
Aggreg.
Linux Kernel
• Facilitates NSF after re-start with bulk
SPA SPA SPA SPA SPA download of state information
SDWAN Daemons
RP
CPU
RP
Chass. Man. SDWAN • Terminate controllers DTLS connections
IOS CPU
• Propagate OMP routing updates

ESI (10-40 Gbps)
Forwarding Manager
Kernel
• Maintain BFD/Tunnel
ESP ESP
FECP Chassis Manager • Handles SDWAN edge configuration
FECP
EOBC (1 Gbps)

Drivers
Drivers
• Netconf/sshd daemon is a linux process
I2C
Linux Kernel
Crypto
QFP
Assist.
QFP
µ µµ Crypto
µ BQS
µ µ Assist.
interconn.
ESI (10-40 Gbps)
SIP SIP
IOCP interconn.
SPA Driver Chassis
SPA Driver
SPA Driver Manager
SPA
IOCP
Aggreg.
Linux Kernel
SPA SPA SPA SPA SPA
PPE microcode
RP
CPU
RP
Chass. Man. SDWAN
• Written in C
CPU
IOS
• proper features, no hack
ESI (10-40 Gbps)
Forwarding Manager
Kernel • Runs on each thread of the PPE
ESP ESP
FECP
• Processes packets
Chassis Manager
FECP • run to completion
EOBC (1 Gbps)

PPE Microcode runs here
Drivers
Drivers • assisted by various memories
I2C
Linux Kernel
Crypto
QFP • TCAM, DRAM,… various speeds
Assist.
QFP
µ
Packet Processor Engine BQS
µ
• Features applied via FIA
Crypto
PPE PPE PPE PPE PPE
µ µ …µ BQS
1 2 3 4 5
PPE PPE PPE PPE N
6
µ
7 8
Assist.
Dispatcher
Packet Buffer
interconn.
• Feature Invocation Array
ESI (10-40 Gbps)
SIP SIP
IOCP interconn.
SPA Driver
SPA Driver
SPA Driver
Chassis
Manager
• FIA per interface
SPA
Aggreg.
IOCP
Linux Kernel
• input FIA, output FIA
• drop FIA (Null interface)
SPA SPA SPA SPA SPA
Resource
Monitoring
The vital signs…
RP
CPU
RP Chassis
SDWAN
Manager
CPU
IOS
Forwarding Manager
Control Plane CPU’s
Kernel
ESP ESP
FECP
Where does it hurt ?

Chassis Manager
FECP
Drivers
Drivers
Linux Kernel
Crypto
QFP
Assist.
QFP
Data Plane CPU’s µ µµ Crypto
µ BQS
µ µ Assist.
interconn.
SIP SIP
IOCP interconn.
SPA Driver Chassis
SPA Driver
SPA Driver Manager
SPA
IOCP
Aggreg.
Linux Kernel
SPA SPA SPA SPA SPA
Example: IOS Memory Usage vs IOSd RP
Utilization
asr-1k#show mem stat
Load for five secs: 6%/1%; one minute: 5%; five minutes: 3% RP
Time source is NTP, 22:18:08.111 EDT Sat Apr 19 2014 CPU
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b) Chassis Manager SDWAN
Processor 300AE008 1713127140 564269356 1148857784 1066242316 992444168 IOS
lsmpi_io 963791D0 6295088 6294120 968 968 968
Forwarding Manager
asr-1k#show process mem | inc BGP
523 0 2333028 51368 389076 313 313 BGP Router
Linux Kernel
asr-1k#show process cpu
…
Complex CLI, platform specific.
Additional information require connecting to the Linux shell

asr-1k#sh platform software process list RP active summary
…
Architecture : ppc
Memory (kB)
Physical : 4127744
Total : 3874516
Used : 2095636
Free : 1778880
asr-1k#sh platform software process list RP active | inc fman

fman_rp 29015 27992 29015 S 20 136847360
asr-1k#show platform hardware qfp active infrastructure exmem statistics
QFP Memory Utilization

QFP exmem statistics
Type: Name: DRAM, QFP: 0

Total: 1073741824
It is getting worse… InUse: 219466752

Free: 854275072
Lowest free water mark: 854005760
Type: Name: IRAM, QFP: 0
Total: 134217728
InUse: 8728576
Free: 125489152
Type: Name: SRAM, QFP: 0
Total: 32768
InUse: 15088
Free: 17680
asr-1k#show platform hardware qfp active infrastructure exmem statistics user

…
10 279092 284672 CEF
40 36441494 36458496 NAT
ESP FECP Chassis Manager

Drivers
asr-1k#show platform hardware qfp active tcam resource-manager usage Drivers
Load for five secs: 0%/0%; one minute: 1%; five minutes: 1% Linux Kernel
Time source is NTP, 09:43:55.075 EDT Fri Apr 25 2014
QFP TCAM Usage Information

QFP
µ µ µ
<snip>
Crypto
µ BQS
µ µ Assist.
Total TCAM Cell Usage Information
----------------------------------
Name : TCAM #0 on CPP #0
Total number of regions :3
Total tcam used cell entries : 28 TCAM DRAM DRAM
Total tcam free cell entries : 524260
Threshold status : below critical limit
IOS 3.14
Resources: simplified view

RP
asr-1k# show platform resources CPU
Resource Usage Max Warning Critical State RP Chassis
SDWAN
RP0 (active) H Manager
CPU
IOS
Memory 1814MB 3783MB 90% 95% H
Forwarding Manager
CPU 5.80% 100% 90% 95% H
FP0 (active) H interconn.Linux GE switch
Memory 683MB 1962MB 90% 95% H Kernel
CPU 19.89% 100% 90% 95% H
QFP H
DRAM 76244KB 524288KB 80% 90% H ESP ESP
IRAM 8817KB 131072KB 80% 90% H
SRAM 14KB 32KB 80% 90% H FECP
TCAM 28cells 131072cells 80% 90% H Drivers Forwarding Manager
CPU Utilization 10.00% 100% 90% 95% H Drivers
Drivers
Linux Kernel
FP1 (standby) H Crypto
QFP
Memory 683MB 1962MB 90% 95% H Assist.
QFP
CPU 19.89% 100% 90% 95% H µ µµ Crypto
µ BQS
QFP H
µ µ Assist.
DRAM 76244KB 524288KB 80% 90% H interconn.
IRAM 8817KB 131072KB 80% 90% H
SRAM 14KB 32KB 80% 90% H SIP SIP
IOCP interconn.
TCAM 28cells 131072cells 80% 90% H SPA Driver Chassis
SIP0 (active) H SPA Driver
SPA Driver Manager
Memory 307MB 460MB 90% 95% H
SPA
CPU 4.10% 100% 90% 95% H IOCP
Aggreg.
Linux Kernel
SIP1 (standby) H
Memory 160MB 460MB 90% 95% H
CPU 1.10% 100% 90% 95% H SPA SPA SPA SPA SPA
**State Acronym: H - Healthy, W - Warning, C – Critical
Memory
troubleshooting
Effects of BGP
download of 800k
routes
Adding 800.000 routes to BGP
RP
CPU
RP
Chassis Manager SDWAN
IOS CPU
ESI (10-40 Gbps)

Forwarding Manager
Kernel
ESP ESP
EOBC (1 Gbps) FECP
Drivers
I2C Drivers
Linux Kernel
Crypto
QFP
Assist.
QFP
µ µµ Crypto
µ BQS
µ µ Assist.
interconn.
ESI (10-40 Gbps)
SIP SIP
IOCP interconn.
SPA Driver Chassis
SPA Driver
SPA Driver Manager
SPA
IOCP
Aggreg.
Linux Kernel
SPA SPA SPA SPA SPA
Effect on IOSd
Before injecting routes: After injecting routes:
ASR1001X#show proc mem sorted ASR1001X #show proc mem sorted
Processor Pool Total: 3941568560 Used: 398742584 Free: 3542825976 Processor Pool Total: 3941299768 Used: 1558483376 Free: 2382816392
lsmpi_io Pool Total: 6295128 Used: 6294296 Free: 832 lsmpi_io Pool Total: 6295128 Used: 6294296 Free: 832
<…removed….>
PID TTY Allocated Freed Holding Getbufs Retbufs Process PID TTY Allocated Freed Holding Getbufs Retbufs Process
693 0 274480 760 46104 0 0 BGP Router 693 0 935265552 40167544 934971640 0 0 BGP Router
ASR1001X show ip route summary bsns-asr1001x-3#show ip route summary

IP routing table name is default (0x0) IP routing table name is default (0x0)
IP routing table maximum-paths is 32 IP routing table maximum-paths is 32
Route Source Networks Subnets Replicates Overhead Memory Route Source Networks Subnets Replicates Overhead Memory (bytes
(bytes) connected 0 2 0 192 624
connected 0 4 0 384 1248 static 0 0 0 0 0
static 1 0 0 96 312 application 0 0 0 0 0
application 0 0 0 0 0 bgp 10 0 800000 0 76800000 249600000
bgp 10 0 0 0 0 0 External: 0 Internal: 800000 Local: 0
External: 0 Internal: 0 Local: 0 internal 14 64004648
internal 2 1264 Total 14 800002 0 76800192 313605272
Total 3 4 0 480 2824
bsns-asr1001x-3#
Adding 800.000 routes… Platform resources
Before injecting 800k routes via BGP:
ASR10001X#show platform resources
**State Acronym: H - Healthy, W - Warning, C - Critical
Resource Usage Max Warning Critical State
Only 1 control-
---------------------------------------------------------------------------------------------------- processor on
RP0 (ok, active) H ASR1001X
Control Processor 0.47% 100% 80% 90% H
DRAM 2536MB(32%) 7861MB 88% 93% H
ESP0(ok, active) H
QFP H
TCAM 21192cells(16%) 131072cells 65% 85% H
DRAM 435947KB(11%) 3670016KB 85% 95% H
IRAM 11477KB(8%) 131072KB 85% 95% H
CPU Utilization 4.00% 100% 90% 95% H
After injecting 800k routes via BGP:

----------------------------------------------------------------------------------------------------
RP0 (ok, active) H
Control Processor 1.57% 100% 80% 90% H
DRAM 5428MB(69%) 7861MB 88% 93% H
ESP0(ok, active) H
QFP H
DRAM 528747KB(14%) 3670016KB 85% 95% H
IRAM 11477KB(8%) 131072KB 85% 95% H
Zooming out – RP level comsumption
RP
CPU
RP
IOS CPU
ESI (10-40 Gbps)

Forwarding Manager
Kernel
ESP ESP
EOBC (1 Gbps) FECP
Drivers
I2C Drivers
Linux Kernel
Crypto
QFP
Assist.
QFP
µ µµ Crypto
µ BQS
µ µ Assist.
interconn.
ESI (10-40 Gbps)
SIP SIP
IOCP interconn.
SPA Driver Chassis
SPA Driver
SPA Driver Manager
SPA
IOCP
Aggreg.
Linux Kernel
SPA SPA SPA SPA SPA
Effect on RP
Before:
Only on ASR1K at the moment
ASR1001X#show platform resources rp active memory
System memory: 8049828K total, 2550496K used, 5499332K free,
Lowest: 5450288K
Pid Text Data Stack Dynamic RSS Name
----------------------------------------------------------------------
<removed>
17731 7576 126756 136 2532 126756 fman_rp
14402 243203 949008 136 284 949008 linux_iosd-imag
<removed>
After:
ASR1001X#show platform resources rp active memory
Lowest: 5450288K
----------------------------------------------------------------------
<removed>
17731 7576
24195 243501
126756
2067436
136
136
2532
284
126756
2067436
fman_rp
linux_iosd-imag
Only IOSd impacted
<removed> by the routes
Programming the QFP through FMAN-FP
RP
CPU
RP
IOS CPU
ESI (10-40 Gbps)

Forwarding Manager
Kernel
ESP ESP
EOBC (1 Gbps) FECP
Drivers
I2C Drivers
Linux Kernel
Crypto
QFP
Assist.
QFP
µ µµ Crypto
µ BQS
µ µ Assist.
interconn.
ESI (10-40 Gbps)
SIP SIP
IOCP interconn.
SPA Driver Chassis
SPA Driver
SPA Driver Manager
SPA
IOCP
Aggreg.
Linux Kernel
SPA SPA SPA SPA SPA
Effect on FMAN-FP
Before: Only on ASR1K at the moment
ASR1001X#show platform resources fp active memory
Lowest: 5450288K
----------------------------------------------------------------------
<removed>
24195 13188 223168 136 43544 223168 fman_fp_image
<removed>
After:
ASR1001X#show platform resources fp active memory
Lowest: 5450288K
----------------------------------------------------------------------
<removed>
24195 13196 1266748 136 864060 1266748 fman_fp_image
All routes now on
<removed>
FMAN-FP
Deeper into the FMAN-FP process
ASR1001X#show platform software memory forwarding-manager fp active brief

module allocated requested allocs frees
------------------------------------------------------------------------------
Summary 792036430 726160030 8007075 3889800
AOM object 155527414 155526070 84 0
route-subblock 121602128 96001680 1603442 3414
route-pfx 118402300 105602028 817576 17559
<removed>
ASR1001X#show platform software ip fp active cef summary

Forwarding Table Summary
Name VRF id Table id Protocol Prefixes State

------------------------------------------------------------------------------------------------
Default 0 0 IPv4 800013 hw: 0x558de85c3478 (created)
Prefix count
Monitoring
IOSd low memory
condition
IOSd low memory condition
RP
CPU
RP
IOS CPU
ESI (10-40 Gbps)

Forwarding Manager
Kernel
ESP ESP
EOBC (1 Gbps) FECP
Drivers
I2C Drivers
Linux Kernel
Crypto
QFP
Assist.
QFP
µ µµ Crypto
µ BQS
µ µ Assist.
interconn.
ESI (10-40 Gbps)
SIP SIP
IOCP interconn.
SPA Driver Chassis
SPA Driver
SPA Driver Manager
SPA
IOCP
Aggreg.
Linux Kernel
SPA SPA SPA SPA SPA
16.11
Low memory condition
Simulating low memory condition by artificially increasing logging buffer
ASR1001X (config)#logging buffered 2147483647

ASR1001X#
This command forces the logging buffer to eat up most of the memory
IOSd will produce the following syslog (rate limited)

*Jan 18 07:23:39.090: %SYS-4-FREEMEMLOW: Free Memory has dropped below low watermark. Pool: Processor
Free: 235388376 Threshold: 394129408 Top Allocator Name: BGP net chunk, PC:
iosd_shr_m_uk9_ROUTING_crb:7FF3B55F3000+907D18, Size: 212582048, Count: 3239 Largest block: 426644
A detailed memory output will be saved to bootflash (for TAC):

Directory of bootflash:/
66 -rw- 589091 Jan 18 2019 07:23:43 +00:00 threshold_lowmem_info_20190118-072340-UTC
TCAM exhaustion
How it started…
An administrator would like to augment his QoS policy by

applying a new ACL in order to mark some traffic as AF43.
The change looked benign, however…
*Jan 17 09:59:50.527: %CPP_FM-3-CPP_FM_TCAM_WARNING: R0/0: cpp_sp_svr: TCAM limit exceeded: HW

TCAM cannot hold class group [cce:7571632] mark. TCAM platform dependent limit exceeded. Try to
allocate 74680 TCAM cell entries. Free TCAM cell: 75700 Total TCAM cell: 131072. Use SW TCAM
instead.
ESP
FECP
TCAM is near the QFP (data plane memory) Crypto

Assist. PPE
QFP
BQS
intercon.

DRAM
Temp Sensor (128MB)
EEPROM
QFP
Boot Flash
PCI* E-RP*
Dispatcher
Packet Buffer Memory I2C
SPA Control
Crypto SPA Bus
(Nitrox-II CN2430)
SPI Mux
Other

Before the change
• Prior to changing the policy, the TCAM 16% consumed.

----------------------------------------------------------------------------------------------------
<removed>
ESP0(ok, active) H
QFP H
DRAM 1129516KB(30%) 3670016KB 85% 95% H
IRAM 11477KB(8%) 131072KB 85% 95% H
Deeper inside the TCAM
Only on ASR1K
ASR1001X#show platform resources tcam sorted

TCAM Usage Information
Total cells in TCAM: 131072

Free cells in TCAM: 109880
CG-Id Name Client 160bitVMR 320bitVMR Total Cell Total% Label

---------------------------------------------------------------------------------------------------------------
cce:7571632 mark QOS 0 2252 9008 6 44
cce:8719920 in2out FW 3003 0 6006 4 42
acl:2 infra_acl ACL 3001 0 6002 4 43
nat:1001 --- NAT 84 0 168 0 41
Internal QFP handles Policy name in IOS 4 features using TCAM
TCAM usage vs Configuration
• NOT a 1-1 mapping
• Ports range will consume more TCAM entries
• Interleaved Deny’s & Permit’s will consume more TCAM entries
QoS ACL’s: TCAM usage:

642 entries 2252 cells
Modifying a TCAM-based object
New Class- Clean up old

Service-policy If TCAM is service policy &
group ID is
TCAM class- overflown by the class-group
programmed to
group ID is new policy,
reflect the new New class-group
duplicated fallback to DRAM
configuration activated
CACE – TCAM offload
• When falling back to DRAM, the classification is done by CACE
• CACE = Classification Engine
• A slower, software version of the TCAM
• When a policy is moved to the CACE, it will not fall back to TCAM
• Supplemental classification engine:
• Uses DRAM resources
• Costs QFP resources when classifying packets
• May or may not be perceptible…
• Entries with port ranges, mix of deny & permit consume more TCAM
ESP
FECP
TCAM is near the QFP (data plane memory) Crypto

Assist. PPE
QFP
BQS
intercon.

DRAM
Temp Sensor (128MB)
EEPROM
QFP
DDRAM Packet Processor Engine
CACE memoryBQS
Boot Flash
PCI* E-RP*
Dispatcher
SPA Control
Crypto SPA Bus
(Nitrox-II CN2430)
SPI Mux
Other

If you missed the warning, check out the exmem
• Supplemental classification memory footprint can be checked with

the following platform resource command
ASR1001X# show platform resources exmem | i CACE

Allocations Bytes-Alloc Bytes-Total User-Name
-------------------------------------------------------------------------------
1 4969578 4970496 CACE
QFP DRAM
exhaustion
What happened…
• After configuring zone-based firewall service-policies, the admin decided to apply the zones to a few interfaces…
ASR1001X(config)# configure terminal
Interface gig0/0/0
Zone security inside
Interface gig0/0/1
Zone security outside Looks like some
DRAM issue
• After configuring, the router reaches the warning threshold and issues the following logs…
Jan 17 06:19:45.849: %QFPOOR-4-LOWRSRC_PERCENT_WARN: R0/0: cpp_ha_top_level_server: QFP 0 DRAM(EXMEM)

at 85 percent, exceeds warning level 85
*Jan 17 06:19:45.849: %QFPOOR-4-TOP_EXMEM_USER: R0/0: cpp_ha_top_level_server: EXMEM_USER: NAT,
Allocations: 371, Bytes-Alloc: -1917190944, Bytes-Total: 1916886016
*Jan 17 06:19:45.849: %QFPOOR-4-TOP_EXMEM_USER: R0/0: cpp_ha_top_level_server: EXMEM_USER: CVLA,
Allocations: 93, Bytes-Alloc: 336530288, Bytes-Total: 336591872
ESP
FECP
FYI “EXMEM” is part of

DRAM
Crypto
Assist. PPE
QFP
BQS
intercon.

DRAM
Temp Sensor (128MB)
EEPROM
QFP
Boot Flash
PCI* E-RP*
Dispatcher
SPA Control
Crypto SPA Bus
(Nitrox-II CN2430)
SPI Mux
Other

Looking at DRAM

----------------------------------------------------------------------------------------------------
<removed>
ESP0(ok, active) W
QFP W
DRAM 3379218KB(92%) 3670016KB 85% 95% W
IRAM 11477KB(8%) 131072KB 85% 95% H
EXMEM main consumers
Only on ASR1K
ASR1001X#sh platform resources exmem

<removed>
Type: Name: GLOBAL, QFP: 0
-------------------------------------------------------------------------------
<removed>
9 123657280 123665408 CFT (Common Flow Table)
98 357012368 357076992 CVLA (FNF/NBAR)
10 20058072 20064256 FNF
<removed>
21 10842832 10855424 STILE (NBAR)
<removed>
378 2406564608 2406874112 NAT
131 322867920 322941952 FW
EXMEM main consumers on non-ASR1k
ISR1111X# show platform hardware qfp active infrastructure exmem statistics user
<removed>
Type: Name: GLOBAL, QFP: 0
-------------------------------------------------------------------------------
<removed>
9 123657280 123665408 CFT (Common Flow Table)
98 357012368 357076992 CVLA (FNF/NBAR)
10 20058072 20064256 FNF
<removed>
21 10842832 10855424 STILE (NBAR)
<removed>
378 2406564608 2406874112 NAT
131 322867920 322941952 FW
Root cause analysis
• The router was managing 700k sessions while using FNF, NAT, NBAR
• Applying ZBF pushed DRAM consumption above the Warning mark.
ASR1001X#sh platform resources

----------------------------------------------------------------------------------------------------
<removed>
ESP0(ok, active) H
QFP H
DRAM 3117246KB(84%) 3670016KB 85% 95% H
IRAM 11477KB(8%) 131072KB 85% 95% H
DRAM was on the

brink before ZBF#CLUS BRKARC-3147 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 76
The forwarding
plane
SIP intercon.
SPA
IOCP
Aggreg.
Ingress Packet Through SIP SPA SPA
ESPs RPs RPs
Reset / Pwr Ctrl Interconnect

Temp Sensor
In ref clocks
EEPROM
DDRAM
Ingress Egress
Buffer
JTAG Ctrl
clock
…

Network
Ingress clocks

Classifier C2W

Other SPA Bus
ESP
FECP
Ingress Packet Through ESP Crypto

Assist. PPE
QFP
BQS
intercon.

TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
QFP Complex
Boot Flash
Dispatcher If full…
Global Packet Queueing strategy: fifo GE, 1Gbps
Output queue: 0/40 (size/max) I2C
5 minute input rate 852000 bits/sec, 544 packets/sec SPA Control
Crypto 5 minute output rate 132000 bits/sec, 137 packets/sec SPA Bus
SPI Mux 285328 packets input, 53247038 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
Reset / Pwr Ctrl 5 input errors, 0 CRC, 0 frame, 5 overrun, 0 ignoredESI, 11.2Gbps
DRAM Interconnect 0 watchdog, 242 multicast, 0 pause input
Hypertransport, 10Gbps
Other

ESP
FECP
Packet Dispatched to PPE Thread Crypto

Assist. PPE
QFP
BQS
intercon.

TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
QFP Complex
DDRAM
PPE2 Packet Processor Engine BQS
Boot Flash
FECP PPE1 PPE2 PPE3 PPE4 PPE5
Thread 1
Thread 2
Thread 3
Thread 4
(OBFL,…)
JTAG Ctrl PPE6 PPE7 PPE8 PPEN
Dispatcher
GE, 1Gbps
Packet Buffer
I2C
SPA Control
Crypto SPA Bus
SPI Mux
Other

ESP
FECP
Packet Dispatched to PPE Thread Crypto

Assist. PPE
QFP
BQS
intercon.

TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
QFP Complex
DDRAM
Boot Flash
Thread 1
Thread 2
Thread 3
Thread 4
(OBFL,…)
Dispatcher
Packet Buffer
Crypto
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect

ESP
FECP
FIA’s Applied on Packet by PPE Thread Crypto

Assist. PPE
QFP
BQS
intercon.
X-ConnectReset / Pwr L2
Ctrl Switch IPv4 IPv6 MPLS
Packet Buffer Part Len / BW
TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
QFP Complex
DDRAM
Netflow
Input ACL
PPE2 Packet Processor Engine
Netflow BQS
NAT
NBAR Classify
Boot Flash
NBAR Classify
Thread 1
Thread 2
Thread 3
Thread 4
(OBFL,…)
MQC Classify
JTAG Ctrl … IP Unicast

PPE 6 PPE7 PPE8 … …
PPEN
NAT IP Multicast MQC Policing
PBR MAC Accounting

Packet ForDispatcher
Us
Dialer IDLE Rst Packet Buffer
Output ACL
URD
Crypto
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect

ESP
FECP
FIA’s Applied on Packet by PPE Thread Crypto

Assist. PPE
QFP
BQS
intercon.
TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
QFP Complex
DDRAM
Netflow
Input ACL
Netflow BQS
NAT
NBAR Classify
Boot Flash
NBAR Classify
Thread 1
Thread 2
Thread 3
Thread 4
(OBFL,…)
MQC Classify

PPEN
PBR MAC Accounting

Us
Output ACL
URD
Crypto
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect

ESP
FECP
Leaving the PPE Thread Crypto

Assist. PPE
QFP
BQS
intercon.
Ctrl Switch IPv4 IPv6 Packet Buffer
MPLSPart Len / BW
TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
QFP Complex
DDRAM
Netflow
Input ACL
PPE2 Packet Processor Engine Netflow BQS
NAT
NBAR Classify
Boot Flash
Thread 1
Thread 2
Thread 3
Thread 4
(OBFL,…) NBAR Classify
MQC Classify
PPEN
PBR MAC Accounting

Us
Output ACL
Crypto
URD
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect

ESP
FECP
Packet proceeding to BQS then SIP Crypto

Assist. PPE
QFP
BQS
intercon.

TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
QFP Complex
Boot Flash
Dispatcher
SPA Control
Crypto SPA Bus
SPI Mux
Other

SIP intercon.
SPA
IOCP
Aggreg.
Egress Packet Through SIP SPA SPA
ESPs RPs RPs
Reset / Pwr Ctrl

EV-RP
Interconnect
Temp Sensor EV-FC
In ref clocks
EEPROM
DDRAM
Ingress Egress
Buffer
JTAG Ctrl
clock
…

Network
Ingress clocks

Classifier C2W

Other SPA Bus
Debugging
strategies
Everyday situations
Traffic did not reach its target !

What happened to that packet ?
Why did that happen ?
Everyday situations
IPsec ZBF NAT
WAAS SNMP
OTV
First Routing
Which feature went wrong ?

Memory
Bug
Traffic Ambiguity
issue
Everyday situations Second

Memory
Bug
Traffic Ambiguity
Pattern
Using statistics for troubleshooting packet drops
Not easy… not very practical either.
• ESP Let’s dig deeper before making it simpler
• SPA
• show platform hardware slot {f0|f1} serdes statistics
• show interfaces <interface-name>
• show platform hardware slot {f0|f1} serdes statistics internal
• show interfaces <interface-name> accounting
• show platform hardware qfp active bqs 0 ipm mapping
• show interfaces <interface-name> stats • show platform hardware qfp active bqs 0 ipm statistics channel all
• SIP • show platform hardware qfp active bqs 0 opm mapping
• show platform hardware qfp active bqs 0 opm statistics channel all
• show platform hardware port <slot/card/port> plim statistics
• show platform hardware qfp active statistics drop [detail]
• show platform hardware subslot {slot/card} plim statistics
• show platform hardware qfp active interface if-name <Interface-name> statistics
• show platform hardware slot {slot} plim statistics • show platform hardware qfp active infrastructure punt statistics type per-cause | exclude
_0_
• show platform hardware slot {0|1|2} plim status internal
• show platform hardware qfp active infrastructure punt statistics type punt-drop | exclude
_0_
• show platform hardware slot {0|1|2} serdes statistics
• show platform hardware qfp active infrastructure punt statistics type inject-drop |
exclude _0_
• RP
• show platform hardware qfp active infrastructure punt statistics type global-drop |
• show platform hardware slot {r0|r1} serdes statistics exclude _0_
• show platform hardware qfp active infrastructure bqs queue output default all
• show platform software infrastructure lsmpi
• show platform hardware qfp active infrastructure bqs queue output recycle all
The Road to
Simplification:
The Packet Tracer
PROBLEM!! Slow SCP Transfer
192.168.111.254 192.168.114.254
192.168.111.x 192.168.114.x
192.168.11.x R1 R2 R3 R4 192.168.14.x
R5 R6
A glimpse into the
future
Network Wide
Troubleshooting
QoS Packet
drops!
IOS 3.7
The Embedded Packet Capture

One way of capturing packets…
Device# monitor capture mycap start
Device# monitor capture mycap access-list v4acl
Device# monitor capture mycap limit duration 1000
Device# monitor capture mycap interface GigabitEthernet 0/0/1 both
Device# monitor capture mycap buffer circular size 10
Device# monitor capture mycap start Shows whether packets have been received or sent
Device# monitor capture mycap export tftp://10.1.88.9/mycap.pcap
Shows what packets look like
Device# monitor capture mycap stop
Requires hex dump analysis or export to decoder (sniffer)
Does not tell us what happened to the packet
Device# show monitor capture mycap buffer dump
0
0000: 01005E00 00020000 0C07AC1D 080045C0 ..^...........E.
0010: 00300000 00000111 CFDC091D 0002E000 .0..............
0020: 000207C1 07C1001C 802A0000 10030AFA .........*......
0030: 1D006369 73636F00 0000091D 0001 ..example....... Excellent tool but insufficient in many cases
1
0000: 01005E00 0002001B 2BF69280 080046C0 ..^.....+.....F.
0010: 00200000 00000102 44170000 0000E000 . ......D.......
0020: 00019404 00001700 E8FF0000 0000 .............. http://www.cisco.com/en/US/docs/ios-
xml/ios/epc/configuration/xe-3s/asr1000/nm-packet-
2 capture-xe.html
0000: 01005E00 0002001B 2BF68680 080045C0 ..^.....+.....E.
0010: 00300000 00000111 CFDB091D 0003E000 .0..............
0020: 000207C1 07C1001C 88B50000 08030A6E ...............n
0030: 1D006369 73636F00 0000091D 0001 ..example.......za
The Packet Tracer and FIA Debugger IOS 3.10
TCAM Resource DRAM
Packet Buffer Part Len / BW Packet # 16
Condition determines
Temp Sensor
DRAM SRAM
packets to be traced Output FIA

Input ACL
Input FIA
EEPROM
QFP ComplexOptionally match MQC Classify
DDRAM
Pak Match ?
on the egress FIA
BQS
Output ACL NAT

Input ACL
Boot Flash
PBR
Thread 1
Thread 2
Thread 3
Thread 4
(OBFL,…)
MQC Classify NAT
Encaps Output ACL
NAT
IP Unicast
Statistics and final action will be NAT
PBR Dispatcher Crypto
collected (matched packets
Packet Buffer dropped, punted to RP, forwarded to
PPE Encaps
output interface …) 2
Crypto Thread 3 Crypto
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect
Optionally, FIA actions can logged per packet
System can capture several packets flows
RPs RPs ESP RPs Packet flows can be reviewed in show
SIPs commands
Conditionally Matching Packets
Identifying Interesting Packets
asr-1k# debug platform condition ?
debug platform condition ?
both Simultaneous ingress and egress debug
egress Egress only debug
…
ingress Ingress only debug
interface Set interface for conditional debug
ipv4 Debug IPv4 conditions
ipv6 Debug IPv6 conditions
mpls Debug MPLS conditions
…
Match all ingress packets

asr-1k#debug platform condition ingress
Match all ingress packets on interface gig0/0/3
asr-1k#debug platform condition interface gig0/0/3 ingress
asr-1k#debug platform condition ipv4 10.0.0.1/32 both Match in & out packets with source
asr-1k#debug platform condition ipv4 access-list 100 egress or destination 10.0.0.1
asr-1k#debug platform condition mpls 10 1 ingress
Match egress packets passing
Match MPLS packets with access-list 100
top ingress label 10
Activating the Packet Tracer
Following packets through IOS-XE – Basic Statistics
asr-1k# debug platform packet-trace ?
copy Copy packet data
The packet tracer follows
drop Trace drops only a set of packets in details
enable Enable packet trace through the FIA
packet Packet count
asr-1k# debug platform condition interface gig0/0/0 ingress

asr-1k# debug platform condition start
asr-1k# debug platform packet-trace enable
asr-1k# … !send traffic Extraneous command -
asr-1k# show platform packet-trace statistics was suppressed in 16.3
Packets Summary
Matched 102
102 packets were matched
Traced 0 by the condition
Packets Received
Ingress 12
Inject 90
Count Code Cause
90 9 QFP ICMP generated packet
Packets Processed
Forward 12 12 packets were forwarded
Punt 0
Drop 90 90 packets were dropped
Count Code Cause
13 92 Ipv4Null0 13 packets were dropped
17 47 FirewallInvalidZone due to no route
60 184 FirewallL4
Consume 0
17 packets were dropped due to
60 packets dropped by L4
absence of zone pair
inspection (e.g. receiving window)
Packet Tracer – Tracing Packets…
The fate of 16 packets
asr-1k# debug platform condition start Automatically stops tracing
asr-1k# debug platform packet-trace packet 16 after 16 packets
asr-1k# debug platform packet-trace enable Extraneous command -
asr-1k# … !send traffic was suppressed in 16.3
asr-1k# show platform packet-trace summary
Pkt Input Output State Reason 16 packets were traced; we
0 Gi0/0/2 internal0/0/rp:0 PUNT 55 (For-us control) can zoom in
1 Gi0/0/2 internal0/0/rp:0 PUNT 55 (For-us control)
4 INJ.7 Gi0/0/2 FWD
5 INJ.7 Gi0/0/2 FWD
7 INJ.7 Gi0/0/2 FWD INJ.7: Packet injected by the RP
internal0/0/rp:0: Packet punted to the RP
8 …
The fate of an individual packet
asr-1k# show platform packet-trace packet 1
Packet: 1 CBUG ID: 109056985
Summary Zooming on packet 1
Input : GigabitEthernet0/0/2
Output : internal0/0/rp:0
State : PUNT 55 (For-us control)
Timestamp
Start : 334771580191282 ns (04/29/2014 08:01:38.017738 UTC)
Stop : 334771580487612 ns (04/29/2014 08:01:38.018035 UTC)
Path Trace
Feature: IPV4
Source : 17.0.0.196
Feature specific details are
Destination : 172.18.0.1 displayed
Protocol : 50 (ESP)
Feature: IPSec
Only major features
Action : DECRYPT
are shown
SA Handle : 753
SPI : 0x30ba5940
Peer Addr : 17.0.0.196
Local Addr: 172.18.0.1
Packet Tracer – Tracing Packets
... even keeping a copy of the packet if necessary
asr-1k# debug platform condition interface gig0/0/0 ingress Keep a copy of the packet in
asr-1k# debug platform condition start ingress and egress of the ESP
asr-1k# debug platform packet-trace packet 16 (before and after the FIA)
asr-1k# debug platform packet-trace copy packet both [l2 | l3 | l4]
asr-1k# debug platform packet-trace enable
asr-1k# … !send traffic
Can store L2, L3 or L4…
asr-1k# show platform packet-trace packet 1 pick-a-choose
Summary
Display the stored packet copy
Input : GigabitEthernet0/0/2
Path Trace
Feature: IPV4
Feature: IPSec
Packet Copy In
45c00088 c5ee0000 ff32346f 11000313 ac120001 d4b46317 0000017c 68a60265
0ef58135 650e2341 15cf6e81 dd434455 b42efef8 c6cf5ab1 44ad3f98 b165c3d5
Packet Copy Out
45c0003c 00000000 015804f4 c0ab1301 e000000a 0205efc8 00000000 00000000
00000000 0000000a 0001000c 01000100 0000000f 00040008 0a000200
The fate of a single packet… even more more more details
asr-1k# show platform packet-trace packet 1 decode
Summary
Input : GigabitEthernet0/0/2 Decode the stored packet copy
Path Trace
Feature: IPV4
Feature: IPSec
Packet Copy In
45c00088 c5ee0000 ff32346f 11000313 ac120001 d4b46317 0000017c 68a60265
0ef58135 650e2341 15cf6e81 dd434455 b42efef8 c6cf5ab1 44ad3f98 b165c3d5
IPv4
Version :4
Header Length :5
ToS : 0xc0
Total Length : 136
Identifier : 0xc5ee
IP Flags : 0x0
Frag Offset :0 Here showing the input copy
TTL : 255 (output copy follows)
Protocol : 50 (ESP)
Header Checksum : 0x346f
Source Address : 17.0.3.19
Destination Address : 172.18.0.1
ESP
SPI : 0xd4b46317
Sequence Number : 0x0000017c
...
Packet Tracer – Focus on Drops
Dropped packets – nothing else For drops, condition is optional…
asr-1k# debug platform condition start
asr-1k# debug platform packet-trace packet 16 Only save dropped packets
asr-1k# debug platform packet-trace drop [code <dropcode>]
asr-1k# debug platform packet-trace enable Focus on specific drop codes
asr-1k# … !send traffic
(find codes in packet-trace statistics)
asr-1k# debug platform condition stop Stop tracing before dumping the
asr-1k# show platform packet-trace summary summary (code limitation)
Pkt Input Output State Reason
0 Gi0/0/2 Gi0/0/2 DROP 53 (IpsecInput)
Admire dropped packets… real close
1 Gi0/0/2 Gi0/0/2 DROP 53 (IpsecInput) asr-1k#show platform packet-trace packet 1
2 Gi0/0/2 Gi0/0/2 DROP 53 (IpsecInput) Packet: 1 CBUG ID: 148787639
Summary
3 Gi0/0/2 Gi0/0/2 DROP 53 (IpsecInput) Input : GigabitEthernet0/0/2
Output : GigabitEthernet0/0/2
4 Gi0/0/2 Gi0/0/2 DROP 53 (IpsecInput) State : DROP 53 (IpsecInput)
Timestamp
5 Gi0/0/2 Gi0/0/2 DROP 53 (IpsecInput) Start : 361426338620013 ns (04/29/2014 15:25:52.785406 UTC)
6 Gi0/0/2 Gi0/0/2 DROP 53 (IpsecInput) Stop : 361426338684993 ns (04/29/2014 15:25:52.785471 UTC)
Path Trace
7 Gi0/0/2 Gi0/0/2 DROP 53 (IpsecInput) Feature: IPV4
Source : 17.0.1.34
8 … Destination : 172.18.0.1
Protocol : 50 (ESP)
Network Topology
192.168.111.254 192.168.114.254
192.168.111.x 192.168.114.x
192.168.11.x R1 R2 R3 R4 192.168.14.x
Packet drops seen

R5 R6
Packet Tracer
Demonstration
Low performance flow – Searching for clues
Asr1k#debug platform condition ipv4 192.168.114.254/32 both
Asr1k#debug platform condition start Enabling condition and packet-trace
Asr1k#debug platform packet-trace packet 256
Start flow and recreate the problem:

Displaying high level statistics
Asr1k#show platform packet-trace statistics
Packets Summary Number of packets matching the condition
Matched 138585
Traced 256
Packets Received
Number of packets traced within QFP
Ingress 138585 [up to 256 per configuration above]
Inject 0
Packets Processed
Forward 132660 Packet processing accounting
Punt 0 [ Forward / Punt / Drop ]
Drop 5925
Count Code Cause
373 23 TailDrop Drop accounting per drop cause
5552 24 Wred Consume 0
Low performance flow – Searching for clues (2)
Going deeper in the trace
Asr1k#show platform packet-trace summary Packet forwarded

<removed>
412 Gi2 Gi3 FWD TailDrop: Packet dropped due to queue being
413 Gi2 Gi3 DROP 23 (TailDrop) full
414 Gi2 Gi3 DROP 23 (TailDrop)
415 Gi2 Gi3 FWD
<removed>
434 Gi2 Gi3 FWD
435 Gi2 Gi3 FWD
436 Gi2 Gi3 DROP 24 (Wred)
439 Gi2 Gi3 FWD
<removed>
443 Gi2 Gi3 FWD
444 Gi2 Gi3 FWD
[Weighted random early detection(WRED ) drops
445 Gi2 Gi3 DROP 24 (Wred) Congestion avoidance mechanism that drop
446 Gi2 Gi3 FWD packets from lower important queues
447 Gi2 Gi3 FWD
Low performance flow – Packet level analysis (I)
csr1000v-6# show platform packet-trace packet 413 Input and Output interfaces
Summary
Input : GigabitEthernet2 Packet State + Drop reason in case of drops
Output : GigabitEthernet3
State : DROP 23 (TailDrop)
Timestamp
Packet Processing time
Start : 239465601952079 ns (01/28/2018 08:56:53.893725 UTC)
Stop : 239465601970437 ns (01/28/2018 08:56:53.893743 UTC)
Path Trace
Feature: IPV4(Input)
Input : GigabitEthernet2
Output : <unknown> Layer3 flow information
Source : 192.168.111.254
Destination : 192.168.114.254
Protocol : 6 (TCP)
SrcPort : 53446 Layer 4 ports [Here SCP]
DstPort : 22
Low performance flow – Packet level analysis (II)
Packet matched by the condition
Feature: DEBUG_COND_INPUT_PKT
Entry : Input - 0x8166b1e0
Input FIA of input interface [ GigabitEthernet2]
Output : <unknown>
Before forwarding lookup, destination interface is
Lapsed time : 1840 ns unknown
<removed>
Function performing forwarding
lookup
Feature: IPV4_INPUT_LOOKUP_PROCESS
Entry : Input - 0x8166b1f8
Output : GigabitEthernet3 Output interface is now set as GigabitEthernet3
Lapsed time : 2746 ns
Low performance flow – Packet level analysis (III)
<removed> The next feature will be executed on the egress interface

Feature: IPV4_INPUT_GOTO_OUTPUT_FEATURE
Entry : Input - 0x8166b2ec
Feature is now executed on the output FIA of the
Feature: CBUG_OUTPUT_FIA
egress interface GigabitEthernet3
Entry : Output - 0x8166b1e8
<removed>
Low performance flow – Packet level analysis (IV)
<removed>
Feature: QOS QoS dropped the traffic
Direction : Egress
Action : DROP
Drop Cause : TailDrop TailDrop: Packet dropped due to queue being
Policy : WRED Precedence full
Pak Priority : FALSE
Priority : FALSE
Queue ID : 115 (0x73) Queue ID where the drop occurred. We will use it for the
PAL Queue ID : 1073741831 (0x40000007)next lookup. Let’s keep the Queue ID 115 in mind
Queue Limit : 16
WRED enabled : TRUE Class-map queue
Inst Queue len: 16 length
Avg Queue len : 8
Feature: OUTPUT_DROP_EXT
Entry : Output - 0x81670e28
Feature: IPV4_OUTPUT_DROP_POLICY
Entry : Output - 0x8166b2d0
Low performance flow – QoS policies
ASR1k#show platform hardware qfp active feature qos interface GigabitEthernet3 output hierarchy
Interface: GigabitEthernet3, QFP interface: 8
Direction: Output Policy “Parent” is the top shaper configured in
Hierarchy level: 0 the class-map ”class-default”
Policy name: parent
Class name: class-default, Policy name: parent
Hierarchy level: 1
Policy name: child
Policy “child” is nested under the class-map
Parent class name: class-default
“class-default” from the “parent” policy-map
Class name: voice, Policy name: child
<removed>
Class name: class-default, Policy name: child That’s the Queue ID that we were looking for.
Queue: QID: 115 (0x73) Now we can remap that Queue ID to a class-map
bandwidth (cfg) : 150000 , bandwidth (hw) : 150000 “class-default” under policy “child”
shape (cfg) :0 , shape (hw) :0
prio level (cfg) : 0 , prio level (hw) : n/a
limit (pkts ) : 16
drop policy : wred precedence
Queue full at the time where the command has
Statistics: been issued [ real time data]
depth (pkts ) : 16
tail drops (bytes): 233223 , (packets) :122
total enqs (bytes): 0 , (packets) :0 Traffic actually dropped at the time when
licensed throughput oversubscription drops: the command has been issued
(bytes): 0 , (packets) :0
Low performance flow – Back to the configuration (I)
Policy-map parent Parent QoS Shaper

class class-default
shape average 15000000
service-policy child Nested child QoS policy under the shaper
policy-map child
class voice QoS policy definition
priority 300
class data
bandwidth percent 30
queue-limit 1024 packets
class urgent
queue-limit 512 packets Set of different
class cs6
bandwidth percent 5
traffic-classes
class class-default based on DSCP
random-detect
bandwidth percent 1
Low performance flow – Back to the configuration (II)
Policy-map parent
class class-default
shape average 15000000
service-policy child
policy-map child
class voice
priority 300
class data
class urgent
class cs6
bandwidth percent 5
class class-default
random-detect Traffic is handled by this class
bandwidth percent 1
Another example – Mapping drop statistics back
to flows
Statistics clear on read
ASR1k#show platform hardware qfp active statistics drop clear
-------------------------------------------------------------------------
Global Drop Stats Packets Octets
-------------------------------------------------------------------------
Disabled 177 17574
IpTtlExceeded 1 114
Ipv4NoRoute 245281 15711182
QosPolicing 2542680 508536000
TailDrop 18388 25860416
UnconfiguredIpv6Fia 650 46452
Wred 102711 145557564
ASR1k#show platform hardware qfp active statistics drop After waiting a few seconds – Collecting a new
set of statistics
-------------------------------------------------------------------------
Global Drop Stats Packets Octets
-------------------------------------------------------------------------
Ipv4NoRoute 5 320 Unexplained drops – Some flows are dropped due to a missing
QosPolicing 3648 729600
route
TailDrop 13 18272
Wred 160 226988
QoS related
drops
Another example – Enable packet-trace
ASR1k#show platform packet-trace code drop | i Ipv4NoRoute Drop Code lookup – Ipv4NoRoute drop code is 19
19 Ipv4NoRoute
ASR1k#clear platform condition all Clearing previous tracing and conditions
ASR1k#debug platform condition start Condition started without a filter since we
ASR1k#debug platform packet-trace packet 16 fia-trace don’t know the src/dst addresses
ASR1k#debug platform packet-trace copy packet both size 1500
ASR1k#debug platform packet-trace drop code 19
The drop subsystem is instrumented to trace any
<Waiting before collecting data>
packet dropped by code “19” [Ipv4NoRoute]
ASR1k#show platform packet-trace summary
0 Gi2 Gi2 DROP 19 (Ipv4NoRoute)
3 Gi2 Gi2 DROP 19 (Ipv4NoRoute) Output interface Gi2 does not always means the
4 Gi2 Gi2 DROP 19 (Ipv4NoRoute) traffic has been bouncing back on the same
5 Gi2 Gi2 DROP 19 (Ipv4NoRoute) interface but simply we display the “last” interface
6 Gi2 Gi2 DROP 19 (Ipv4NoRoute) where the traffic has been seen
Another example – Display the result
ASR1k#show platform packet-trace packet 0
Current packet-trace configuration requires that conditions are stopped
prior to showing packet details. When tracing drop / inject / punt, the condition must
Please execute 'debug platform condition stop' and try again. be turned off in order to access the trace
ASR1k #debug platform condition stop
ASR1k #show platform packet-trace packet 0 decode
Summary
State : DROP 19 (Ipv4NoRoute)
<removed>
Path Trace
Output : <unknown>
Source : 9.9.9.9
Destination : 6.6.6.6
<removed> Layer3,Layer4,ingress interface are recorded – We
Feature: STILE_LEGACY_DROP_EXT have identified the flow that is dropped
Entry : Input - 0x816b54c8
Output : <unknown>
<removed>
Another example – Display the packet
Packet Copy In
52540056 f5805254 00c61056 08004500 00320000 00003911 639e0909 09090606
Optionally, the packet has been copied so we can even
0606fde8 1770001e 5dc20001 02030405 06070809 0a0b0c0d 0e0f1011 12131415 discover mac-addresses
ARPA
Destination MAC : 5254.0056.f580
Source MAC : 5254.00c6.1056
Type : 0x0800 (IPV4)
IPv4
Version :4
Header Length :5
ToS : 0x00
Total Length : 50 IPv4 Layer 3 decoded
Identifier : 0x0000 fields
IP Flags : 0x0
Frag Offset :0
TTL : 57
Protocol : 17 (UDP)
Header Checksum : 0x639e
Destination Address : 6.6.6.6
UDP
Source Port : 65000
Destination Port : 6000
Length : 30 IPv4 Layer 4 decoded fields
Checksum : 0x5dc2
<removed>
show platform hardware qfp active
interface if-name GigabitEthernet1
General interface information Protocol 0 - ipv4_input Protocol 8 - layer2_input
Interface Name: GigabitEthernet1 FIA handle - CP:0x2fccfe0 DP:0xe73998c0 FIA handle - CP:0x2fcd100 DP:0xe73976c0
Interface state: VALID IPV4_INPUT_DST_LOOKUP_ISSUE (M) LAYER2_INPUT_SIA (M)
Platform interface handle: 7 IPV4_INPUT_ARL_SANITY (M) CBUG_INPUT_FIA
QFP interface handle: 6 CBUG_INPUT_FIA DEBUG_COND_INPUT_PKT
Rx uidb: 1023 DEBUG_COND_INPUT_PKT LAYER2_INPUT_ARL (D)
Tx uidb: 65530 IPV4_INPUT_DST_LOOKUP_CONSUME (M) LAYER2_INPUT_QOS
Channel: 30 IPV4_INPUT_ACL LAYER2_INPUT_LOOKUP_PROCESS (M)
Interface Relationships IPV4_INPUT_FOR_US_MARTIAN (M) LAYER2_INPUT_GOTO_OUTPUT_FEATURE (M)
IPV4_INPUT_STILE_LEGACY Protocol 9 - layer2_output
BGPPA/QPPB interface configuration information IPV4_INPUT_QOS FIA handle - CP:0x2fcd460 DP:0xe73910c0
Ingress: BGPPA/QPPB not configured. flags: 0000 IPV4_INPUT_VFR LAYER2_OUTPUT_ARL (D)
Egress : BGPPA not configured. flags: 0000 IPV4_NAT_INPUT_FIA LAYER2_OUTPUT_QOS
IPV4_INPUT_LOOKUP_PROCESS (M) LAYER2_OUTPUT_DROP_POLICY (M)
ipv4_input enabled. IPV4_INPUT_IPOPTIONS_PROCESS (M) MARMOT_SPA_D_TRANSMIT_PKT
ipv4_output enabled. IPV4_INPUT_GOTO_OUTPUT_FEATURE (M) DEF_IF_DROP_FIA (M)
layer2_input enabled. Protocol 1 - ipv4_output Protocol 14 - ess_ac_input
layer2_output enabled. FIA handle - CP:0x2fcd4a8 DP:0xe7390840 FIA handle - CP:0x2fcd190 DP:0xe73965c0
ess_ac_input enabled. IPV4_OUTPUT_VFR CBUG_INPUT_FIA
IPV4_OUTPUT_INSPECT PPPOE_GET_SESSION
Features Bound to Interface: IPV4_NAT_OUTPUT_FIA ESS_ENTER_SWITCHING
2 GIC FIA state IPV4_OUTPUT_THREAT_DEFENSE PPPOE_HANDLE_UNCLASSIFIED_SESSION
57 PUNT INJECT DB IPV4_VFR_REFRAG (M) DEF_IF_DROP_FIA (M)
46 ethernet IPV4_OUTPUT_L2_REWRITE (M)
44 VNIC Path IPV4_OUTPUT_STILE_LEGACY QfpEth Physical Information
1 IFM IPV4_OUTPUT_QOS DPS Addr: 0x00000000038b7e48
[…] IPV4_OUTPUT_FRAG (M) Submap Table Addr: 0x00000000
IPV4_OUTPUT_DROP_POLICY (M) VLAN Ethertype: 0x8100
MARMOT_SPA_D_TRANSMIT_PKT QOS Mode: Per Link
DEF_IF_DROP_FIA (M) VLAN AutoSense: No
Packet Tracing – Basic and FIA-TRACE
Features Pack Tracer Pack Tracer w/ FIA-TRACE
asr1000# show platform hardware qfp active interface if-name asr1000#show platform packet-trace packet 1 asr1000#show platform packet-trace packet 0
gig1 Packet: 1 CBUG ID: 518 Packet: 0 CBUG ID: 655
General interface information Summary Summary
Interface Name: GigabitEthernet1 Input : GigabitEthernet1 Input : GigabitEthernet1
Interface state: VALID Output : GigabitEthernet2 Output : GigabitEthernet2
Platform interface handle: 7 State : FWD State : FWD
QFP interface handle: 6 Timestamp Timestamp
… Start : 5331698002827 ns (07/11/2016 23:28:23.187027 UTC) Start : 5456699323393 ns (07/11/2016 23:30:28.244810 UTC)
Protocol 0 - ipv4_input Stop : 5331698159842 ns (07/11/2016 23:28:23.187184 UTC) Stop : 5456699556099 ns (07/11/2016 23:30:28.245043 UTC)
FIA handle - CP:0x2fccfe0 DP:0xe73998c0 Path Trace Path Trace
[…]
IPV4_INPUT_DST_LOOKUP_ISSUE (M)
IPV4_INPUT_ARL_SANITY (M)
CBUG_INPUT_FIA
Feature: IPV4 Feature: IPV4

DEBUG_COND_INPUT_PKT Input : GigabitEthernet1 Input : GigabitEthernet1
Output : <unknown> Output : <unknown>
Source : 192.168.3.1 Source : 192.168.3.1
Destination : 192.168.255.167 Destination : 192.168.255.167
Protocol : 50 (ESP) Protocol : 50 (ESP)
Feature: FIA_TRACE
Output : <unknown>
Entry : 0x8139f260 - DEBUG_COND_INPUT_PKT
Packet Tracing – Basic and FIA-TRACE (II)
IPV4_INPUT_DST_LOOKUP_CONSUME (M) Feature: IPV4_INPUT_DST_LOOKUP_CONSUME

Entry : Input - 0x816999a8
Output : <unknown>
IPV4_INPUT_ACL
Feature: IPV4_INPUT_ACL
Output : <unknown>
IPV4_INPUT_FOR_US_MARTIAN (M)
Feature: IPV4_INPUT_FOR_US_MARTIAN
Output : <unknown>
Packet Tracing – Basic and FIA-TRACE (III)
IPV4_INPUT_STILE_LEGACY Feature: CFT Feature: CFT

API : cft_handle_pkt API : cft_handle_pkt
packet capabilities : 0x0000008c packet capabilities : 0x0000008c
input vrf_idx :0 input vrf_idx :0
calling feature : STILE calling feature : STILE
direction : Input direction : Input
triplet.vrf_idx :0 triplet.vrf_idx :0
triplet.network_start : 0x00000000 triplet.network_start : 0x00000000
triplet.triplet_flags : 0x00000000 triplet.triplet_flags : 0x00000000
triplet.counter :0 triplet.counter :0
cft_bucket_number : 2120447 cft_bucket_number : 2120447
cft_l3_payload_size : 100 cft_l3_payload_size : 100
cft_pkt_ind_flags : 0x00000000 cft_pkt_ind_flags : 0x00000000
cft_pkt_ind_valid : 0x00000935 cft_pkt_ind_valid : 0x00000935
tuple.src_ip : 192.168.3.1 tuple.src_ip : 192.168.3.1
tuple.dst_ip : 192.168.255.167 tuple.dst_ip : 192.168.255.167
[…] […]
Feature: NBAR Feature: NBAR
Packet number in flow: N/A Packet number in flow: N/A
Classification state: Final Classification state: Final
Classification name: ipsec Classification name: ipsec
[…] Classification ID: [CANA-L7:9]
Number of matched sub-classifications: 0
Number of extracted fields: 0
Is PA (split) packet: False
TPH-MQC bitmask value: 0x0
Feature: IPV4_INPUT_STILE_LEGACY
Entry : Input - 0x80fa0f88
Output : <unknown>
Packet Tracing – Basic and FIA-TRACE (IV)
IPV4_INPUT_QOS Feature: QOS Feature: QOS

Direction : Ingress Direction : Ingress
Action : SET Action : SET
Fields : DSCP Fields : DSCP
Feature: IPV4_INPUT_QOS
Output : <unknown>
IPV4_INPUT_VFR Feature: IPV4_INPUT_VFR

Output : <unknown>
IPV4_NAT_INPUT_FIA Feature: IPV4_NAT_INPUT_FIA

Entry : Input - 0x816999r
Output : <unknown>
IPV4_INPUT_LOOKUP_PROCESS (M) Entry : Input - 0x816999a8
This is where we decide the output Input : GigabitEthernet1
interface will be GigabitEthernet 2
Packet Tracing – Basic and FIA-TRACE (V)
IPV4_INPUT_IPOPTIONS_PROCESS (M) Feature: IPV4_INPUT_IPOPTIONS_PROCESS

Entry : Input - 0x8166b2ec
IPV4_INPUT_GOTO_OUTPUT_FEATURE (M) Input : GigabitEthernet2
At this juncture, we switch from the Lapsed time : 453 ns
CBUG_OUTPUT_FIA
input FIA of GigabitEthernet1 to the Feature: CBUG_OUTPUT_FIA
output FIA of GigabitEthernet2 Entry : Output - 0x8166b1e8
GigabitEthernet3 output FIA Output : GigabitEthernet3
IPV4_VFR_REFRAG (M)
Feature: IPV4_VFR_REFRAG
Entry : Output - 0x8166b354
IPV4_OUTPUT_L2_REWRITE (M)
Feature: IPV4_OUTPUT_L2_REWRITE
Entry : Output - 0x8166ad94
Packet Tracing – Basic and FIA-TRACE (VI)
IPV4_OUTPUT_QOS Feature: IPV4_OUTPUT_QOS

Entry : Output - 0x8166b2cc
IPV4_OUTPUT_FRAG (M) Feature: IPV4_OUTPUT_FRAG

Entry : Output - 0x8166b33c
IPV4_OUTPUT_DROP_POLICY (M) Feature: IPV4_OUTPUT_DROP_POLICY

Entry : Output - 0x8166b2d0
DEBUG_COND_OUTPUT_PKT Feature: DEBUG_COND_OUTPUT_PKT

Entry : Output - 0x8166b1dc
MARMOT_SPA_D_TRANSMIT_PKT Feature: MARMOT_SPA_D_TRANSMIT_PKT

DEF_IF_DROP_FIA (M)
Tunnels and IPsec
Packet Forwarding
Basic Packet Forwarding
Layer 5+ IKE AAA BGP
Layer 4
Layer 3 Routing
Layer 2
Output
features
Input
features Encapsulation
Packet Forwarding – Tunnels & Features
Layer 5+ IKE AAA BGP
Layer 4
Post-encapsulation
(Tunnel Protection)
Layer 3 Routing Routing

Layer 2
Input Output features Encapsulation Output features

features (applied to clear (applied to
Encapsulation encrypted packet)
text packet)
ESP
FECP
Egress IPsec Packet Flow (I) Lookup SA Handler by class-group ID

Crypto
Assist. PPE
QFP
BQS
Look up IPsec proxy-identities

Obtain Crypto SA ctx ID intercon.
Obtain class-group ID
TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
QFP Complex
Boot Flash
Uses Crypto Context identified

by Context ID Dispatcher
SPA Control
Crypto SPA Bus
SPI Mux
Other

ESP
FECP
Egress IPsec Packet Flow (II) Crypto

Assist. PPE
QFP
BQS
intercon.

TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
QFP Complex
PPE may be different but
packet processing continues
where it stopped (right after
Boot Flash
(OBFL,…) FECP crypto) PPE1 PPE2 PPE3 PPE4 PPE5
Uses Crypto Context identified

by Context ID Dispatcher
SPA Control
Crypto SPA Bus
SPI Mux
Other

Simplifying the IPsec show commands
One show command to rule them all
interface: Virtual-Access1002 ------------------ show platform software ipsec fp active flow identifier 34130 ------------------
Crypto map tag: Virtual-Access1002-head-0, local addr 172.18.0.1
…
protected vrf: (none) ------------------ show platform hardware qfp active feature ipsec sa 1427 ------------------
local ident (addr/mask/prot/port): (172.18.0.1/255.255.255.255/47/0)
remote ident (addr/mask/prot/port): (17.0.0.26/255.255.255.255/47/0) …
current_peer 17.0.0.26 port 500
PERMIT, flags={origin_is_acl,}
------------------ show platform software ipsec fp active encryption-processor context 6502aa4f ------
#pkts encaps: 25227, #pkts encrypt: 25227, #pkts digest: 25227 ------------
#pkts decaps: 25237, #pkts decrypt: 25237, #pkts verify: 25237
#pkts compressed: 0, #pkts decompressed: 0 …
#pkts not compressed: 0, #pkts compr. failed: 0 ------------------ show platform software ipsec fp active flow identifier 34129 ------------------
#pkts not decompressed: 0, #pkts decompress failed: 0
#send errors 0, #recv errors 0 …
local crypto endpt.: 172.18.0.1, remote crypto endpt.: 17.0.0.26
------------------ show platform hardware qfp active feature ipsec sa 1867 ------------------
plaintext mtu 1458, path mtu 1500, ip mtu 1500, ip mtu idb GigabitEthernet0/0/2 …
current outbound spi: 0xA7B61FE5(2813730789)
PFS (Y/N): N, DH group: none ------------------ show platform software ipsec fp active encryption-processor context 2e02aa4e ------
-----------
inbound esp sas:
spi: 0xA222F391(2720199569) …
transform: esp-aes esp-sha-hmac ,
in use settings ={Transport, }
conn id: 36130, flow_id: HW:34130, sibling_flags FFFFFFFF80000008, crypto map: Virtual- Access1002-
head-0
sa timing: remaining key lifetime (k/sec): (4607974/2137)
IV size: 16 bytes
replay detection support: Y replay window size: 512
Status: ACTIVE(ACTIVE)
…
show crypto ipsec sa interface virtual-access 1002 platform
outbound esp sas:
spi: 0xA7B61FE5(2813730789)
transform: esp-aes esp-sha-hmac ,
or
in use settings ={Transport, }
conn id: 36129, flow_id: HW:34129, sibling_flags FFFFFFFF80000008, crypto map: Virtual- Access1002-
head-0
show crypto ipsec sa peer 17.0.0.26 platform
sa timing: remaining key lifetime (k/sec): (4607974/2137)
IV size: 16 bytes
replay detection support: Y replay window size: 512
Status: ACTIVE(ACTIVE)
…
Understanding and
Extracting ESP
Logs
ESP Tracing aka Logging
TEMP RAM FS
RP RP logs are first written
CPU
RP here (efficiency)
Chassis Manager
IOS CPU
NFS Shared Disk
ESI (10-40 Gbps)

Forwarding Manager
Hard disk is really here
Kernel
ESP ESP
FECP
TEMP RAM FS
Chassis Manager ESP logs are first written
FECP
EOBC (1 Gbps)
here (efficiency)
Drivers
I2C Drivers
Linux Kernel
Crypto
QFP
Assist.
QFP
Mounted NFS
µ µµ
Crypto
PPE PPE PPE PPE PPE
BQS ESP logs are committed

1 2 3 4 5
µ
PPE …
PPE PPE PPE N
Assist.
µ µ
6 7 8
interconn.
here at regular intervals
Dispatcher
Packet Buffer
ESI (10-40 Gbps)
SIP SIP
IOCP interconn.
SPA Driver Chassis
SPA Driver
SPA Driver Manager
SPA
IOCP
Aggreg.
Linux Kernel
SPA SPA SPA SPA SPA
Important logs
RP
CPU
RP
Chassis Manager
fman_rp_R[0|1]-0.log
IOS CPU
Under /harddisk/tracelogs
ESI (10-40 Gbps)

Forwarding Manager
interconn.Linux GE switch fman_rp_R[0|1]-0.log.<timestamp>

Kernel
fman-fp_R0.log.<timestamp>
cpp_cp_F[0|1]-0.log.<timestamp>
ESP ESP
EOBC (1 Gbps) FECP fman_fp_F[0|1]-0.log
Drivers
Drivers
Forwarding Manager cpp_cp_F[0|1]-0.log
I2C Drivers
Linux Kernel
Crypto
QFP
Assist.
QFP Under /harddisk/tracelogs/
µ µµ
Crypto fman-fp_R0.log.<timestamp>
PPE PPE PPE PPE PPE
BQS
1 2 3 4 5
µ
PPE …
PPE PPE PPE N
Assist.
µ µ
6 7 8
Dispatcher
Packet Buffer
interconn. cpp_cp_F[0|1]-0.log.<timestamp>
ESI (10-40 Gbps)
SIP SIP
IOCP interconn.
SPA Driver Chassis
SPA Driver
SPA Driver Manager
SPA
IOCP
Aggreg.
Linux Kernel
SPA SPA SPA SPA SPA
What log files are important?
• Important log files to get for security issues:
• fman_rp_R[0|1].log (under /tmp/rp/trace directory on RP)
• fman-fp_F[0|1]-0.log (under /tmp/fp/trace directory on ESP
• cpp_cp_F[0|1]-0.log (under /tmp/fp/trace directory on ESP)
• All these logs get rotated and are copied to /harddisk/tracelogs

directory on active RP.
• Look for the relevant log files depending on the time of the failure
• By default, all ERR messages are logged  should be the first
things to look for
Example log files
The timestamp…
My-ASR1000-2#dir harddisk:/tracelogs/cpp_cp_F0*
Directory of harddisk:/tracelogs/cpp_cp_F0*
Directory of harddisk:/tracelogs/
3768365 -rwx 1048934 Jan 6 2014 18:20:16 +00:00 cpp_cp_F0-0.log.7133.20140106182015
3768330 -rwx 551643 Jan 7 2014 09:27:51 +00:00 cpp_cp_F0-0.log.7133.20140107092751
3768335 -rwx 1048901 Jan 7 2014 08:56:44 +00:00 cpp_cp_F0-0.log.7133.2014010708564339313059840 bytes total
(30680653824 bytes free)
Rotating the log files
My-ASR1000-2#dir harddisk:/tracelogs/cpp_cp_F0*
Directory of harddisk:/tracelogs/cpp_cp_F0*
Directory of harddisk:/traceMy-ASR1000-2#test platform software trace slot rp active forwarding-manager rotate
Rotated file from: /tmp/rp/trace/stage/fman_rp_R0-0.log.13836.20140107094754, Bytes: 0, Messages: 6535
My-ASR1000-2#test platform software trace slot FP active cpp-control-process rotate
Rotated file from: /tmp/fp/trace/stage/cpp_cp_F0-0.log.7133.20140107093650, Bytes: 154027, Messages: 786
My-ASR1000-2#test platform software trace slot FP active forwarding-manager rotate
Rotated file from: /tmp/fp/trace/stage/fman-fp_F0-0.log.8247.20140107093738, Bytes: 20170, Messages: 210
OR use
My-ASR1000-2#request platform software trace rotate all Does not show the rotated file names w/
time stamp  have to hunt them down
Since IOS-XE Release 16.7
To see the logs from the end, backward
show logging process cpp_cp internal reverse

or
show logging process cpp_internal start timestamp “2018/01/29
13:44:55.499"
To see the logs from a given timestamp,

onward & forward
Conditional
Feature Debugging
IOS 3.10
The Packet Tracer and Conditional Debugger

TCAM Resource DRAM
Packet Buffer Part Len / BW Packet # 16
Condition determines
Temp Sensor
DRAM SRAM
packets to be traced Output FIA Input ACL

Input FIA
EEPROM
Ingress Match ? QFP Complex
DDRAM
MQC Classify
Output ACL
Output ACL NAT
Input ACL
Boot Flash
Thread 1
Thread 2
Thread 3
Thread 4
(OBFL,…)
NAT
NAT PBR
MQC Classify
IP Unicast Encaps Output ACL
NAT Encaps
NAT
PBR Dispatcher Crypto
Crypto
If feature conditional debugger
Packet Buffer
is activated, these blocks will be Encaps
debugged
Crypto Crypto
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect
The packet tracer collects statistics
and final action (matched packets
Our focus now dropped, punted to RP, forwarded to
RPs RPs ESP RPs SIPs output interface …)
Network Topology – Crypto, Tunnels and Feature
Debugging
192.168.111.x 192.168.114.x
192.168.11.x R1 R2 R3 R4 192.168.14.x
FlexVPN
192.168.15.x R5 R6 192.168.16.x
IPsec Tunneling,
Encapsulation &
Conditional
Debugger
Demonstration
IPSEC packet-trace output (I)
ASR1K #debug platform condition ipv4 192.168.15.254/32 ingress

ASR1K #debug platform condition start
ASR1K #debug platform packet-trace packet 16 fia-trace
ASR1K #debug platform packet-trace copy packet both size 1500
Packet copy size rounded up from 1500 to 2048
ASR1K#show platform packet-trace summary

Pkt Input Output State Reason Clear text Packet to be encrypted
0 Gi4 Gi3 FWD
1 Tu0 Gi4 FWD Packet traced after decryption since the
condition here is not the crypto peer IP
but the cleartext packet. All the
processing happening before the
decryption is not recorded due to the
condition in use!
IPSEC packet-trace output (II)
ASR1K#show platform packet-trace packet 0 decode
<removed>
Path Trace
Output : <unknown>
Source : 192.168.15.254
Destination : 192.168.16.254
Packet to be encrypted
Protocol : 1 (ICMP)
<removed>
Entry : Input - 0x8166b1f8 Forwarding decision is happening on the ingress FIA of
Input : GigabitEthernet4 the ingress interface [ GigabitEthernet4]. This packet will
be switched to Tunnel0
Output : Tunnel0
<removed>
IPSEC packet-trace output (III)
Feature: IPV4_INPUT_GOTO_OUTPUT_FEATURE Last feature on the input FIA. Moving to

Entry : Input - 0x8166b2ec the output FIA of Tunnel0 interface at the
next feature
Output : Tunnel0
Feature: MC_OUTPUT_GEN_RECYCLE
Entry : Output - 0x816a1ecc
Output : Tunnel0 Starting executing features on the output FIA of tunnel0
IPSEC packet-trace output (IV)
Feature: IPV4_TUNNEL_OUTPUT_FNF_AOR_RELEASE
Entry : Output - 0x81699a30
Output : Tunnel0
Lapsed time : 2840 ns Input interface is switched to Tunnel0 –
Feature: IPV4_TUNNEL_OUTPUT_FINAL GRE encapsulation is completed
Input : Tunnel0
Output : Tunnel0
Feature: IPV4_OUTPUT_TUNNEL_PROTECTION_ENCRYPT
Input : Tunnel0
Output : Tunnel0
Feature: IPSec Encrypting the packet. As of this stage, the packet
Result : IPSEC_RESULT_SA source and destination addresses are now local crypto
Action : ENCRYPT address and the crypto peer address
SA Handle : 71
Peer Addr : 172.18.1.6
IPSEC packet-trace output (V)
<removed>
Feature: IPV4_TUNNEL_ENCAP_FOR_US_EXT
Input : Tunnel0
Output : Tunnel0
Feature: IPV4_INPUT_LOOKUP_PROCESS_EXT
Entry : Output - 0x8166b1f8 New lookup in order to find the egress
Input : Tunnel0 interface. The encrypted packet will leave
Output : GigabitEthernet3 the router via GigabitEthernet3
<removed>
Feature: MARMOT_SPA_D_TRANSMIT_PKT
Input : Tunnel0 Packet is dispatched to the SPA in order to be
transmitted
IPSEC packet-trace output (VI)
Clear text packet before encryption:
Protocol : 1 (ICMP)
Destination Address : 192.168.16.254 From the packet-tracer decode, the traffic traced is an ICMP
ICMP packet from 192.168.15.254 to 192.168.16.254
Type : 8 (Echo)
Code : 0 (No Code)
Encrypted text packet after encryption

IPv4
Protocol : 50 (ESP)
Destination Address : 172.18.1.6 After encryption, the packet traced on the egress FIA is
ESP now an ESP packet from 172.18.1.5 to 172.18.1.6
SPI : 0xb370f09d
Sequence Number : 0x00000005
IPSEC feature debugging (I)
To debug encryption, the condition
matches the clear text traffic to be
encrypted.
To trace decryption, the condition would

ASR1K #debug platform condition ipv4 192.168.15.254/32 ingress have to match the tunnel destination IP
ASR1K #debug platform condition start address
ASR1K#debug platform condition feature ipsec dataplane submode all level info
ASR1k
IPSEC feature debugging (II)
Clear text packet is getting classified for
encryption.
ASR1k#show logging process cpp_cp

<removed>
2018/02/01 08:33:33.632 {cpp_cp_F0-0}{1}: [cpp-dp-ipsec] [19505]: (info): QFP:0.0 Thread:000
TS:00000180756943472618 :FEATURE:[172.18.1.5] 8 => [172.18.1.6] 0 1 (2): Performing classification of outgoing
packet to see if it needs IPSEC protection
2018/02/01 08:33:33.832 {cpp_cp_F0-0}{1}: [cpp-dp-ipsec] [19505]: (info): QFP:0.0 Thread:000
TS:00000180756943476593 :FEATURE:[172.18.1.5] 8 => [172.18.1.6] 0 1 (2): IPSec classification for outgoing
packet hit a valid SA (hdl 71, SPI = 0xb370f09d SNS idx:0)
Selecting SPI and SA handle in order to

effectively encrypt the flow
Breaking,
Multiplying and
Gluing Packets
Patterns of Interest
Multicast Replication
Input Packet Packet Copy 1

Packet Copy 2
Packet Copy 3
0 Gi1 <none> CONS Packet Consumed Silently

1 Gi1 Gi2 FWD
2 Gi1 Gi3 FWD
3 Gi1 Gi4 FWD
Fragmentation
Input Packet Fragment 1 Fragment 2

1 Gi1 Gi2 FWD
2 Gi1 Gi2 FWD
ICMP Echo Request & Reply
Packet Packet Packet
Consumed Recycled Injected
ICMP Echo Request
ICMP Echo Reply
0 Gi3 Gi3 CONS Packet Consumed

1 Gi3 internal0/0/recycle:0 PUNT 26 (QFP ICMP generated packet)
2 INJ.9 Gi3 FWD
Punt to recycle path; not to RP.
Debug ip icmp will be mute
Reassembly of For-Us Packets
(e.g. large ICMP Echo Request-Reply)
Packet Packet Packet
Consumed Recycled Injected
ICMP Echo Request

Input Fragment 2 Input Fragment 1
Output Fragment 2 Output Fragment 1

ICMP Echo Reply
0 Gi3 Gi3 CONS Packet Consumed Collect &

Reassemble
2 INJ.9 <none> CONS Packet Consumed Silently
Emit (Inject) ICMP… too big 
3 internal0/0/recycle:0 Gi3 FWD Forward
consume it for fragmentation
4 internal0/0/recycle:0 Gi3 FWD Fragments
Virtual Reassembly of Pass-Thru Packets
(e.g. with NAT)
Input Fragment 1
Input Fragment 2 Input Fragment 1 Output Fragment 2 Output Fragment 1
Start : 82194827793981 ns (01/26/2018 13:09:49.929627 UTC)

Stop : 82194827909191 ns (01/26/2018 13:09:49.929742 UTC)
0 Gi3 Gi4 FWD Total system time = 115260
1 Gi3 Gi4 FWD VFR Lapsed time : 743813 ns
Start : 82194827911554 ns (01/26/2018 13:09:49.929745 UTC)

Stop : 82194827947614 ns (01/26/2018 13:09:49.929781 UTC)
Total system time = 36060
VFR Lapsed time : 298093 ns
Fragment 1 enters and is processed until VFR. Then the packet freezes.
Fragment 2 enters until VFR at which point Fragment 1 is released and processing continues.
Reassembly of GET-VPN Packets (I)
Input Fragment 1
Input Fragment 1
Feature: IPV4_INPUT_DST_LOOKUP_CONSUME 0 Gi3 (no conclusion yet)
Feature: IPSec
Action : DECRYPT
SA Handle : 5
SPI : 0xc4770522
Peer Addr : 192.168.0.0
Local Addr: 192.168.0.0
Feature: IPV4_INPUT_IPSEC_CLASSIFY
Entry : Input - 0x816a0e3c
Output : <unknown>
Reassembly of GET-VPN Packets (II)
Reassembled Packet
Input Fragment 2
Feature: IPV4_INPUT_DST_LOOKUP_CONSUME
Feature: IPV4_INPUT_FOR_US_MARTIAN 1 Gi3 <none> CONS Packet Consumed Silently
Feature: IPV4_INPUT_IPSEC_CLASSIFY
Output : <unknown>
Reassembly of GET-VPN Packets (III)
Reassembled Packet

Crypto
Engine
Feature: IPV4_INPUT_IPSEC_INLINE_PROCESS
Entry : Input - 0x8166b250 0 Gi3 (no conclusion yet)
Output : <unknown>
Reassembly of GET-VPN Packets (IV)
Reassembled Packet

Crypto
Engine
Feature: IPV4_INPUT_IPSEC_RERUN_JUMP
Feature: IPV4_INPUT_SANITY_EXT Output Packet (decrypted reassembled)
Feature: IPV4_INPUT_ARL_EXT
Feature: IPV4_INPUT_IPSEC_POST_PROCESS_EXT
Feature: IPV4_INPUT_DST_LOOKUP_ISSUE_EXT
Feature: IPV4_INPUT_SRC_LOOKUP_ISSUE_EXT
Feature: IPV4_INPUT_IPSEC_DOUBLE_ACL_EXT
Feature: IPV4_INPUT_DST_LOOKUP_CONSUME_EXT 0 Gi3 Gi5 FWD
Feature: IPV4_INPUT_SRC_LOOKUP_CONSUME_EXT
Feature: IPV4_INPUT_FOR_US_EXT
Feature: IPV4_IPSEC_FEATURE_RETURN_EXT
Feature: IPV4_INPUT_IPOPTIONS_PROCESS
Feature: CBUG_OUTPUT_FIA
Feature: IPV4_OUTPUT_FRAG
Feature: DEBUG_COND_OUTPUT_PKT
Reassembly of Overlay VPN Packets (I) –
e.g. FlexVPN
Input Fragment 1
Input Fragment 1
Feature: IPV4_INPUT_DST_LOOKUP_CONSUME 0 Gi3 (no conclusion yet)
Feature: IPV4_INPUT_IPSEC_TUNNEL_FORUS_EXT
Reassembly of Overlay VPN Packets (II)
Reassembled Packet
Input Fragment 2
Feature: DEBUG_COND_INPUT_PKT 1 Gi3 <none> CONS Packet Consumed Silently
Reassembly of Overlay VPN Packets (III)
Reassembled Packet

Crypto
Engine
IPV4_INPUT_IPSEC_TUNNEL_FORUS_EXT
Feature: IPSec 0 Gi3 (no conclusion yet)
Action : DECRYPT
SA Handle : 7
SPI : 0x209cd024
Peer Addr : 172.18.1.6
Feature: IPV4_INPUT_IPSEC_CLASSIFY_EXT
Input : Tunnel0
Output : <unknown>
Feature: IPV4_INPUT_IPSEC_INLINE_PROCESS_EXT
Reassembly of Overlay VPN Packets (IV)
Reassembled Packet

Crypto
Engine
Feature: IPV4_INPUT_IPSEC_TUNNEL_RERUN_JUMP_EXT
Feature: IPV4_INPUT_SANITY_EXT
Feature: IPV4_INPUT_ARL_EXT
Feature: IPV4_INPUT_IPSEC_POST_PROCESS_EXT Output Packet (decrypted reassembled)
Feature: IPV4_INPUT_DST_LOOKUP_ISSUE_EXT
Feature: IPV4_INPUT_SRC_LOOKUP_ISSUE_EXT
Feature: IPV4_INPUT_IPSEC_DOUBLE_ACL_EXT
Feature: IPV4_INPUT_DST_LOOKUP_CONSUME_EXT
Feature: IPV4_INPUT_SRC_LOOKUP_CONSUME_EXT
Feature: IPV4_INPUT_FOR_US_EXT 0 Gi3 Gi5 FWD
Feature: IPV4_IPSEC_FEATURE_RETURN_EXT
Feature: IPV4_INPUT_TUNNEL_IPSEC_DECAP_EXT
Feature: IPV4_TUNNEL_PROTECT_GOTO_INPUT_TUNNEL_EXT
Feature: IPV4_INPUT_DST_LOOKUP_ISSUE
Feature: IPV4_INPUT_ARL_SANITY
Feature: CBUG_INPUT_FIA
Feature: IPV4_OUTPUT_FRAG
Network Topology – Fragmentation and multicast
192.168.111.x 192.168.114.x
192.168.11.x R1 R2 R3 R4 192.168.14.x
FlexVPN
R5 R6
Fragmentation &
Multicast demo
Multicast packet-trace output (I)
ASR1K #debug platform condition start Multicast destination address
ASR1K #debug platform packet-trace copy packet both size 1500
Packet copy size rounded up from 1500 to 2048
Multicast packet is consumed by the
feature multicast in order to replicate.
ASR1K#show platform packet-trace summary Packets #2 and #3 are replicated packets
Pkt Input Output State Reason that are transmitted to the network.
<removed>
2 Gi3 Gi2 FWD
3 Gi3 Gi5 FWD
Multicast packet-trace output (II)
csr1000v-4#show platform packet-trace packet 1
<removed>
Path Trace
Input : GigabitEthernet3 Multicast destination address
Output : <unknown>
Source : 192.168.14.254
Destination : 239.239.239.239
<removed>
Multicast feature handling the packet
Lapsed time : 9220 ns replication
Feature: IPV4_MC_INPUT_REPLICATION_MODULE
Entry : Input - 0x816a19c0
Output : <unknown>
Multicast packet-trace output (III)
<removed>
Path Trace
Feature: PACKET_COPY Parent packet is traced packet #1
Original packet number: 1
Output : <unknown>
Source : 192.168.14.254
Destination : 239.239.239.239
<removed>
Output : <unknown>
Feature: IPV4_MC_INPUT_POST_REPLICATION_PROCESSING
Entry : Input - 0x816a1a50 MFIB lookup – Output interface is
Input : GigabitEthernet3 GigabitEthernet2
<removed>
Multicast packet-trace output (IV)
<removed>
Path Trace
Feature: PACKET_COPY Parent packet is traced packet #1
Output : <unknown>
Source : 192.168.14.254
Destination : 239.239.239.239
<removed>
Output : <unknown>
Feature: IPV4_MC_INPUT_POST_REPLICATION_PROCESSING
Entry : Input - 0x816a1a50 MFIB lookup – Output interface is
Input : GigabitEthernet3 GigabitEthernet5
<removed>
Fragmentation packet-trace output (I)
ASR1K#clear platform condition all

ASR1K #debug platform condition start
ASR1K #show platform packet-trace summary Parent packet is consumed in order to be
fragmented
1 Gi4 Gi2 FWD
2 Gi4 Gi2 FWD
Fragmented packets are transmitted on
the egress interface
Fragmentation packet-trace output (II)
ASR1K#show platform packet-trace packet 0
Summary
Output : <none>
State : CONS Packet Consumed Silently
<removed>
Feature: IPV4_FRAG
Fragments created: 2 IPV4_OUTPUT_FRAG wil consume this
Feature: IPV4_OUTPUT_FRAG packet
Output : Tunnel0
Fragmentation packet-trace output (III)
<removed>
Path Trace This is the first IPv4 fragment of traced packet #0.
Feature: PACKET_COPY
<removed>
<removed>
Feature: IPSec
Result : IPSEC_RESULT_SA
Action : ENCRYPT
SA Handle : 61
Peer Addr : 172.18.1.5
<removed>
Fragmentation packet-trace output (III)
<removed>
Path Trace This is the second IPv4 fragment of traced packet
Feature: PACKET_COPY #0.
<removed>
<removed>
Feature: IPSec
Result : IPSEC_RESULT_SA
Action : ENCRYPT
SA Handle : 61
Peer Addr : 172.18.1.5
<removed>
SDWAN - cEdge
SDWAN cEdge: Disruption after SDWAN migration
cEdge7
MPLS
192.168.12.254
cEdge2
Internet
cEdge8
192.168.23.254
Network wide tracing by Truetrace
Client to server :
Server to client (reversed view):
SDWAN – cEdge
In depth datapath
debugging
Packet-trace output
<removed>
Output : Tunnel1
State : DROP 191 (FirewallNotInitiator)
<removed>
Path Trace
Output : <unknown>
Source : 192.168.23.254
Destination : 192.168.12.254
Protocol : 1 (ICMP) Echo-reply dropped by the second edge
Feature: ZBFW
Action : Drop
Reason : Not a session initiator
Zone-pair name : ZP_1_1_Zone1security
Class-map name : Zone1security-seq-1-cm_
Input interface : GigabitEthernet2
Egress interface : Tunnel1
AVC Classification ID : 0
AVC Classification name: N/A
SDWAN datapath: Packet-trace to the rescue
Blue: Service interface [ VPN 1 ]
Black: MPLS interface [ VPN 0 ]
Red: Biz-internet interface [ VPN 0 ]
cedge4#show platform packet-trace p
0 Gi2 Gi1 FWD
1 Gi2 Gi1 FWD
2 Tu1
3 Gi2
Gi2
Gi1
FWD
FWD
Why traffic flows to/from Gig1 at the
4 Gi2 Gi3 FWD beginning and then switch to Gig3 ?
5 Tu3 Gi2 FWD
6 Tu3 Gi2 FWD
cedge4#cedge4#show platform packet-trace packet 0

Summary
State : FWD
<removed>
Feature: NBAR NBAR application recognition does not have enough contextual
Packet number in flow: 1 information to recognize the application
Classification state: Not final
<removed>
Feature: SDWAN App Route Policy SDWAN App route policy will consider every available tunnels
VRF :1
Seq :1
SLA : __all_tunnels__ (0)
Policy Flags : 0x0
SLA Strict : No
Preferred Color : 0x0 none
cedge4#cedge4#show platform packet-trace packet 0
Summary
State : FWD
<removed>
Feature: NBAR
Packet number in flow: 5 NBAR application recognition has discovered the application as SSH
Classification state: Final
Classification name: ssh
Classification ID: [IANA-L4:22]
<removed>
Feature: SDWAN App Route Policy SDWAN App route policy will optimize the flow towards biz-internet
VRF :1 tunnel
Seq :0
SLA : TEST1 (1)
Policy Flags : 0x1
SLA Strict : Yes
Preferred Color : 0x10 biz-internet
Tunnel Match Reason : MATCHED_SLA_AND_PREF_COLOR
Network Address
Translation
NAT Quick Recap
Sample Config
interface GigabitEthernet0/0/2
ip nat inside Apply role to interfaces
interface GigabitEthernet0/0/3
ip nat outside
Static NAT configuration
ip nat inside source static 172.16.89.32 10.0.0.1
Dynamic NAT
configuration
ip nat inside source list pat interface GigabitEthernet0/0/3 overload (NAT Overload aka PAT)
ip access-list extended pat Match traffic to PAT

permit ip 172.18.25.0 0.0.0.255 any
ESP
FIA’s Applied on Packet by PPE thread FECP
QFP
Crypto
Assist. PPE BQS
intercon.
TCAM Resource DRAM
DRAM SRAM
Temp Sensor
Input FIA Output FIA

EEPROM
QFP Complex
DDRAM Netflow PPE2 Packet Processor Engine ... BQS
Input ACL OUTPUT_NAT

Boot Flash
FECP
INPUT_VFR
Thread 1
Thread 2
Thread 3
Thread 4
(OBFL,…)
JTAG Ctrl
MQC Classify
PPE6 PPE7 PPE8 … PPEN
…
IP Unicast
PBR
Dispatcher
URD Packet Buffer
PPE2
Crypto Thread 3
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect

NAT In  Out – OUTPUT_NAT
ESP
FECP
QFP
Crypto
Assist. PPE BQS
intercon.
TCAM Resource DRAM
DRAM SRAM
Temp Sensor Miss Hit
Input In  Out Check Session Lookup Child Session Lookup
EEPROM Hit
QFP Complex Miss
DDRAM
Netflow
Input ACL
PPE2 Packet Processor Engine ... Static NATBQS
Lookup
Hit
OUTPUT_NAT Miss
NBAR Classify Session DB PPE Door
PPE DB
Boot Flash
FECP 1 PPE2 PPE3 PPE
4 5
Thread 1
Thread 2
Thread 3
Thread 4
(OBFL,…)
MQC ClassifyGatekeeper cache Classification ACL
JTAG Ctrl … IP
PPE6 …PPE7 PPE8 PPEN then Route-Map
NAT
Unicas Bind DB
PBR
t Packet
Dispatcher
Buffer
Allocate Addr Drop
Dialer IDLE Rst

PPE2
URD
Output Crypto
L7 Translation Alg
SPI Mux L3/L4 Translation Session Thread
Create 3
Reset / Pwr Ctrl
SA table
Interconnect
Untranslated IPV4 OUTPUT
DRAM
NAT
ESP
NAT In  Out – OUTPUT_NAT FECP
QFP
Crypto
Assist. PPE BQS
Using Session DB Half Open (embryonic) intercon.

packet from NAT inside  outside ? L7 Child Sessions
Ctrl Switch IPv4in DRAM IPv6 MPLS Connection checks
TCAM Resource DRAM
DRAM SRAM
Temp Sensor Miss Hit
Input In  Out Check Session Lookup Child Session Lookup
EEPROM Hit
QFP Complex Miss
DDRAM
Netflow
Input ACL
PPE2 Packet Processor Engine ... Static NATBQS
Lookup
Hit
OUTPUT_NAT Miss
NBAR Classify Session DB PPE Door
PPE DB
Boot Flash
FECP 1 PPE2 PPE3 PPE
4 5
Use config  use CGM
Thread 1
Thread 2
Thread 3
Thread 4
(OBFL,…)
MQC ClassifyGatekeeper cache Classification ACL and TCAM to match
JTAG Ctrl … IP
PPE6 …PPE7 PPE8 PPEN then Route-Map packet against configured
Unicas Bind DB
rule
NAT Descriptor modification
SIP Media
+ RTP pinhole creation, FTP data
PBR
session fixup,… t Packet
Dispatcher
Buffer
Allocate Addr Drop
Dialer IDLE Rst

PPE2
URD
Output Crypto
L7 Translation Alg
SPI Mux L3/L4 Translation Session Thread
Create 3
Reset / Pwr Ctrl
SA table
Interconnect
Untranslated
Gatekeeper session – IPV4 OUTPUT
DRAM
Important for performances!! NAT
Simplified NAT commands
show ip nat pool platform
show ip nat statistics platform
show ip nat map
NAT Database Dumps
show platform hardware qfp active feature nat datapath sess-dump
id 0x8da733b0 io 172.18.25.66 oo 213.94.72.66 io 60973 oo 1967 it 213.94.72.254 ot 213.94.72.66 it 5266 ot 1967 pro 17 vrf 0 tableid 0 bck 524
id 0x8da8d070 io 172.18.25.66 oo 213.94.72.66 io 54970 oo 1967 it 213.94.72.254 ot 213.94.72.66 it 5355 ot 1967 pro 17 vrf 0 tableid 0 bck 845
…
Session DB
show platform hardware qfp active feature nat datapath bind

Bind longest chain 1 avg non-zero bucket len 1 non-zero bkts 4
bind 0x8e721660 oaddr 172.18.25.2 taddr 213.94.72.254 oport 23 tport 24 vrfid 0 tableid 0 proto 6 domain 0 create time 34701 refcnt 1 mask 0x0 flags 0 timeout 0 ifhandle 1020 wlan_info 0x0 flags
0x1000 mapping 0x8de2f090 cp_mapping_id 0 limit_type 0 last_use_ts 34739 mibp 0x0 rg 0nak_retry 0
Bind DB bind 0x8e7215c0 oaddr 172.18.25.2 taddr 1.1.1.1 oport 0 tport 0 vrfid 0 tableid 0 proto 0 domain 0 create time 34164 refcnt 1 mask 0x0 flags 0 timeout 0 ifhandle 0 wlan_info 0x0 flags 0x8900 mapping
0x0 cp_mapping_id 1 limit_type 0 last_use_ts 34739 mibp 0x0 rg 0nak_retry 0
sh platform hardware qfp active feature nat datapath door

DOOR global stats: door_count 23 door_limit_fail_count 0
Door 0x4bb86a60 IL:10.200.5.2 IG:5.10.5.1 OL:0.0.0.0 OG:0.0.0.0 ILP:10001 IGP:1025 OLP:0 OGP:0 proto 17 flags 0x502 vrf_id 0 idx 129
Door 0x4bb87060 IL:10.200.5.2 IG:5.10.5.2 OL:0.0.0.0 OG:0.0.0.0 ILP:10001 IGP:1025 OLP:0 OGP:0 proto 17 flags 0x502 vrf_id 0 idx 175
Door DB …
ESP
FIA’s Applied on Packet by PPE thread FECP
QFP
Crypto
Assist. PPE BQS
intercon.
TCAM Resource DRAM
DRAM SRAM
Temp Sensor

EEPROM
QFP Complex
DDRAM Netflow PPE2 Packet Processor Engine ... BQS
Input ACL OUTPUT_NAT

Boot Flash
FECP
INPUT_VFR
…
Thread 1
Thread 2
Thread 3
Thread 4
(OBFL,…)
VFR_REFRAG
JTAG Ctrl
MQC Classify
… L2_REWRITE
IP Unicast
PBR
Dispatcher
URD Packet Buffer
PPE2
Crypto Thread 3
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect

Network Topology – NAT troubleshooting
Extranet SAP
160.0.0.254
192.168.14.x
192.168.11.x R1 R2 R3 R4
R5 R6
Demo NAT
NAT troubleshooting (I)
ASR1k#debug platform condition ipv4 192.168.14.254/32 both

ASR1k#debug platform condition start
ASR1k#debug platform packet-trace packet 16 fia-trace
ccsr1000v-3#show platform packet-trace summary
ASR1K#show platform packet-trace summary
0 Gi3 Gi5 FWD
1 Gi3 Gi5 FWD
2 Gi3 Gi5 FWD
3 Gi3 Gi5 FWD
NAT troubleshooting (II)
ASR1k# csr1000v-3#show platform packet-trace packet 0
Summary
State : FWD
<removed>
Feature: NAT
Nat rule handle that will allow us to map back
Direction : IN to OUT to the configuration in step 3
Action : Translate Source
Steps : SESS-FD
Pre-NAT address
Match id : 1
Old Address : 192.168.14.254
New Address : 12.0.0.1 Post-NAT address
<removed>
NAT troubleshooting (II)
ASR1k# csr1000v-3#show platform packet-trace packet 0
Summary
State : FWD
<removed>
Feature: NAT
Nat rule handle that will allow us to map back
Direction : IN to OUT to the configuration in step 3
Action : Translate Source
Steps : SESS-FD
Pre-NAT address
Match id : 1
Old Address : 192.168.14.254
New Address : 12.0.0.1 Post-NAT address
<removed>
NAT troubleshooting (III)
NAT configuration entry

ASR1k# csr1000v-3#show ip nat map matching that flow
<removed>
1:ip nat inside source list test pool test overload
<removed> ACL that matched the flow
ASR1k# csr1000v-3#show access-list test

Extended IP access list test
10 permit ip 192.168.0.0 0.0.15.255 any Pool used by this flow
ASR1k# csr1000v-3#show run | i pool test

ip nat pool test 12.0.0.1 12.0.0.254 netmask 255.255.255.0
NAT troubleshooting (IV)
• The troubleshooter knows what NAT configuration has been matched

• The troubleshooter can ask the following questions?
• Is it the right one?
• Why do we see overlapping ACL?
Demo High CPU
Network Topology – High CPU (QFP) on Router2
192.168.111.x 192.168.114.x
192.168.11.x R1 R2 R3 R4 192.168.14.x
R5 R6
High CPU on Router 2 – Python/bash guestshell
bootstrap
• Leverage builtin linux container running CentOS
1- Create an interface in order for the container to be able to download python scripts
interface VirtualPortGroup0
ip address 10.10.10.1 255.255.255.0
• [Optional] Create a NAT rule to allow the container to reach the rest of the network
without advertising the 10.10.10.0/24 network
2- Configure NAT
ip access-list extended NATFORGUEST
permit ip 10.10.10.0 0.0.0.255 any
ip nat inside source list NATFORGUEST interface GigabitEthernet1 overload
interface VirtualPortGroup0
ip nat inside
Interface Gigabitethernet1
ip nat outside
bootstrap (2)
• Enable IOX
1- Enable IOX
ASR1K(config)#iox
• [Optional] Configure and start the guestshell

ASR1K(config)# ip http server
ASR1K(config)# app-hosting appid guestshell
ASR1K(config -app-hosting)# vnic gateway1 virtualportgroup 0 guest-interface 0 guest-ipaddress
10.10.10.2 netmask 255.255.255.0 gateway 10.10.10.1 name-server 173.38.200.100 default
ASR1K(config -app-hosting)#end
ASR1K(config)#exit
enable
• Enable guestshell
ASR1K#guestshell enable
Interface will be selected if configured in app-hosting
Please wait for completion
Current state is: DEPLOYED
guestshell activated successfully
Current state is: ACTIVATED
guestshell started successfully
Current state is: RUNNING
Guestshell enabled successfully
• Run bash!
ASR1K#guestshell run bash
[guestshell@guestshell ~]$
High CPU on Router 2 – Installing CPU profiler
• Download script from Cisco Devnet Github
[guestshell@guestshell ~]$sudo pip install csr_aws_guestshell
• Run the script!

Guestshell@guestshell~]$csr_aws_guestshell –seconds 5
High CPU on Router 2 – profiling QFP CPU
• CPU profiling
[guestshell@guestshell ~]$ guestshell run bash
[guestshell@guestshell ~]$ measure-packet-trace.py --seconds 10
executing CLI...
9 secs
Retrieving CLI...
Retrieved 384 packets
Parsing data...
Sorting data...
Min time is packet 45, value 1653
Max time is packet 161, value 12303
Storing list...
High CPU on Router 2 – profiling QFP CPU (2)
• CPU profiling
Packets are staying a long time in the NAT

process
High CPU on Router 2 – Config checks –
Correction - Profiling
• Activated ip nat service gatekeeper
-50% less time spent on NAT
IOS 3.14
Other show commands improved too

• Improves interaction with TAC RP
CPU
RP
Chassis Manager
IOS CPU
Forwarding Manager
Control Plane CPU’s
Kernel
ESP ESP
FECP
Drivers
Drivers
Linux Kernel
Crypto
QFP
Assist.
QFP
Data Plane CPU’s µ µµ Crypto
µ BQS
µ µ Assist.
interconn.
show process cpu or show memory or show process memory SIP SIP
IOCP interconn.
SPA Driver Chassis
SPA Driver
SPA Driver Manager
SPA
This command only shows processes inside the IOS daemon. IOCP
Aggreg.
Linux Kernel
Please use 'show <something> platform'
to show processes from the underlying operating system. SPA SPA SPA SPA SPA
Wrapping up…
New Debugging Strategy
IOS Control Plane

• show interface, show ip route, show bgp …
• Feature debugging
Platform Control Plane

• Unified show commands
• Platform show commands
• Future: control plane conditional debugging
Data Plane
• Packet Tracer
• Forwarding plane conditional debugging
• Embedded Packet Capture
Serviceability
North Star
Diagnostic tools are Pain Point
impacting, incomplete, complex
Difficult
Deployment Timeline Challenge
Re-educate
Monitoring Support Staff
Instrumentation
Educate Educate
Design Deploy Operate
Design staff Support Staff
Diagnostic tools are Pain Point
impacting, incomplete, complex
Difficult
Issue Timeline Challenge
CUSTOMER CISCO
Frustration Level
Problem Ticket DDTS Solution
SR Opened Fix applied
starts Opened Opened Identified
Monitoring Config change

User feedback Upgrade…
Capture Capture Capture Recreate
Analyze Analyze Recreate Analyze
Case Volume
So what is next… and why are we doing this ?
RP/0/RSP0/CPU0:router#show packet-trace result
0/1/CPU0 NP6: Packet 2:
Input interface Te0/1/0/0 : 1000 packets marked Input ACL my_input_acl passed at ACE 50
Packet 1: Input service-policy in_policy matched class-map iptv: policer allowed packet, qos-group
Input ACL my_input_acl passed at ACE 50 marked to 5
Input service-policy in_policy matched class-map iptv: policer allowed packet, qos-group RPF result points at Te0/1/0/0 so pass
marked to 5 CEF lookup result: Te/2/0/1 with label push 17000, fabric destination address 0x...
RPF result points at Te0/1/0/0 so pass Dump of packet:
CEF lookup result: Te/2/0/1 fabric destination address 0x... 0000: 6c 9c ed 7b ca b8 40 55 39 6e 5f 44 08 00 45 00 l.m{J8@U9n_D..E.
Dump of packet: 0010: 00 64 0c 64 00 00 ff 01 71 32 0a 00 15 01 0a 00 .d.d....q2......
0000: 6c 9c ed 7b ca b8 40 55 39 6e 5f 44 08 00 45 00 l.m{J8@U9n_D..E. 0020: 15 02 00 00 69 13 61 9c 0c 64 ab cd ab cd ab cd ....i.a..d+M+M+M
0010: 00 64 0c 64 00 00 ff 01 71 32 0a 00 15 01 0a 00 .d.d....q2...... 0030: ab cd ab cd ab cd ab cd ab cd ab cd ab cd ab cd +M+M+M+M+M+M+M+M
0020: 15 02 00 00 69 13 61 9c 0c 64 ab cd ab cd ab cd ....i.a..d+M+M+M 0040: ab cd ab cd ab cd ab cd ab cd ab cd ab cd ab cd +M+M+M+M+M+M+M+M
0030: ab cd ab cd ab cd ab cd ab cd ab cd ab cd ab cd +M+M+M+M+M+M+M+M 0050: ab cd ab cd ab cd ab cd ab cd ab cd ab cd ab cd +M+M+M+M+M+M+M+M
0040: ab cd ab cd ab cd ab cd ab cd ab cd ab cd ab cd +M+M+M+M+M+M+M+M 0060: ab cd ab cd ab cd ab cd ab cd ab cd ab cd ab cd +M+M+M+M+M+M+M+M
0050: ab cd ab cd ab cd ab cd ab cd ab cd ab cd ab cd +M+M+M+M+M+M+M+M 0070: ab cd +M
0060: ab cd ab cd ab cd ab cd ab cd ab cd ab cd ab cd +M+M+M+M+M+M+M+M Decode of packet:
0070: ab cd +M Destination MAC: Cisco_7b:ca:b8 (6c:9c:ed:7b:ca:b8)
Decode of packet: Source MAC: Cisco_6e:5f:44 (40:55:39:6e:5f:44)
Destination MAC: Cisco_7b:ca:b8 (6c:9c:ed:7b:ca:b8) Type: IP (0x0800)
Source MAC: Cisco_6e:5f:44 (40:55:39:6e:5f:44) IP Version: 4 (20 bytes)
Type: IP (0x0800) IP Header Length: 5
IP Version: 4 (20 bytes) IP Total Length: 100
IP Header Length: 5 IP Fragment offset: 0
IP Total Length: 100 IP Time to live: 255
IP Fragment offset: 0 IP Protocol: ICMP (1)
IP Time to live: 255 IP Header checksum: 0x7132 [correct]
IP Protocol: ICMP (1) IP Source: 10.0.21.1
IP Header checksum: 0x7132 [correct] IP Destination: 10.0.21.2
IP Source: 10.0.21.1 Internet Control Message Protocol
IP Destination: 10.0.21.2 Type: 0 (Echo (ping) reply)
Internet Control Message Protocol Code: 0
Type: 0 (Echo (ping) reply)
Code: 0
Serviceability - What are we looking at ? (few examples)
• Control plane debugging (Radioactive Tracing)

• end-to-end control flow debugging
• Unified logging and tracing
• human and machine readable
• Packet tracer on XR, NxOS and XE Switches
• Including Network Wide Packet Tracing
• Controller based debug orchestration and analytics
• two prongs strategy: device/human + controller/analytics
WISP Labs of Interest
• CLI Analyzer: LABRST-2405
• Packet Capturing Tools in Routing Environments: LABRST-2400
Cisco Webex Teams
Questions?
Use Cisco Webex Teams to chat
with the speaker after the session
How
1 Find this session in the Cisco Live Mobile App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space
Webex Teams will be moderated cs.co/ciscolivebot#BRKARC-3147

by the speaker until June 16, 2019.
#CLUS © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 217
Complete your
online session • Please complete your session survey
evaluation after each session. Your feedback
is very important.
• Complete a minimum of 4 session
surveys and the Overall Conference
survey (starting on Thursday) to
receive your Cisco Live water bottle.
• All surveys can be taken in the Cisco Live
Mobile App or by logging in to the Session
Catalog on ciscolive.cisco.com/us.
Cisco Live sessions will be available for viewing
on demand after the event at ciscolive.cisco.com.
Continue your education
Demos in the
Walk-in labs
Cisco campus
Meet the engineer

Related sessions
1:1 meetings
NDA Roadmap Sessions at Cisco Live
Customer Connection Member Exclusive
Join Cisco’s online user group to …
Connect online with 29,000 peer and Cisco NETWORKING ROADMAPS SESSION ID DAY / TIME
experts in private community forums
Roadmap: SD-WAN and Routing CCP-1200 Mon 8:30 – 10:00
Roadmap: Machine Learning and

CCP-1201 Tues 3:30 – 5:00
Learn from experts and stay informed Artificial Intelligence
about product roadmaps Roadmap: Wireless and Mobility CCP-1202 Thurs 10:30 – 12:00
 Roadmap sessions at Cisco Live
 Monthly NDA briefings
Give feedback to Cisco product teams Join at the Customer Connection Booth
(in the Cisco Showcase)
 Product enhancement ideas
 Early adopter trials Member Perks at Cisco Live
 User experience insights • Attend NDA Roadmap Sessions
• Customer Connection Jacket
Join online: www.cisco.com/go/ccp • Member Lounge
Thank you
#CLUS
#CLUS
Backup slides
Zone Based Firewall
ZBF Quick Recap
Sample Config
zone security inside Zone Definition ip access-list extended ipacl
zone security outside deny ip 10.0.0.0 0.0.0.255 172.16.0.0 0.0.0.127
deny ip 10.0.0.0 0.0.0.255 172.16.0.128
interface GigabitEthernet0/0/2 0.0.0.127
ip address 172.18.25.254 255.255.255.0 permit tcp any any
Apply zone to interfaces deny udp 10.0.0.0 0.0.0.255 172.16.0.0
zone-member security inside
0.0.0.127
interface GigabitEthernet0/0/3 permit udp any any
ip address 172.19.25.254 255.255.255.0 permit icmp any any
zone-member security outside
Class map to match traffic
class-map type inspect match-all ipv4acl
match access-group name ipacl
Policy map to determine
policy-map type inspect in2out action on matched traffic
class type inspect ipv4acl
inspect
class class-default
Apply policy between two
zone-pair security in2out source inside destination outside zones
service-policy type inspect in2out
Simplifying the ZBF show commands Collects
everything but
Three Commands for ZBF under the sky TCAM
show policy-firewall config platform
--show platform software firewall FP active bindings--

--show platform software firewall RP active bindings--
--show platform software firewall FP active pairs--
--show platform software firewall RP active pairs--
--show platform software firewall FP active parameter-maps--
--show platform software firewall RP active parameter-maps--
--show platform software firewall FP active zones--
--show platform software firewall RP active zones--
show policy-firewall sessions platform
--show platform hardware qfp active feature firewall datapath scb any any any any any all any show tech firewall
--
show policy-firewall stats platform
--show platform software firewall FP active statistics--

--show platform software firewall RP active statistics--
--show platform hardware qfp active feature firewall runtime--
--show platform hardware qfp active feature firewall memory--
--show platform hardware qfp active feature firewall drop--
--show platform hardware qfp active feature firewall client statistics--
ESP
FECP
ZBF Packet Flow Crypto

Assist. PPE
QFP
BQS
intercon.

TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
QFP Complex
Boot Flash
Dispatcher
Packet Buffer
Crypto
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect

ESP
FIA’s Applied on Packet by PPE thread

FECP
QFP
Crypto
Assist. PPE BQS
intercon.
TCAM Resource DRAM
DRAM SRAM
Temp Sensor

EEPROM
QFP Complex
DDRAM
Netflow
Input ACL
PPE2 Packet Processor Engine ... BQS
OUTPUT_INSPECT
NBAR Classify
Boot Flash
Thread 1
Thread 2
Thread 3
Thread 4
(OBFL,…)
MQC Classify
JTAG Ctrl … PPE6 PPE7 PPE8 … PPEN
NAT IP Unicast
PBR
Dispatcher
PPE2
URD
Crypto
SPI Mux Thread 3
Reset / Pwr Ctrl
SA table
DRAM Interconnect

ESP
Inside Output Threat Inspect

FECP
QFP
Crypto
Assist. PPE BQS
intercon.
TCAM Resource DRAM TCAM
DRAM SRAM
Input
Temp Sensor
Policy Selection Session Lookup Miss
Classify Traffic
(precise + imprecise)
EEPROM Hit
QFP Complex Pass
DDRAM
Netflow
Input ACL
PPE2 Packet Processor Engine ... BQS Drop Inspect
OUTPUT_INSPECT
NBAR Classify Create Session
Boot Flash
FECP Session DB PPE1 PPE2 PPE3 PPE4 PPE5
Thread 1
Thread 2
Thread 3
Thread 4
(OBFL,…)
MQC Classify Drop
JTAG Ctrl … IP
PPE6 …PPE7 PPE8 PPEN
L4 Inspection
NAT
Unicas
PBR
t Packet
Dispatcher
Pass
Buffer L7 Parse
Dialer IDLE Rst
PPE2
URD
Crypto Imprecise Channel Thread 3
SPI Mux L7 Inspection
Creation
Reset / Pwr Ctrl
Output SA table
Interconnect
IPV4 OUTPUT
DRAM
INSPECT
ESP

FECP
QFP
Crypto
Assist. PPE BQS
Using Session DB in DRAM

Imprecise lookup only for Match each class-map in intercon.
µIDB input+output  Zone Pair  Policy
X-ConnectReset / Pwr L2 initial packets (syn…)IPv6 policy (ACL’s in TCAM)
Ctrl Switch IPv4
Packet Buffer
MPLS
Part Len / BW
DRAM SRAM
Input
Temp Sensor
Classify Traffic
EEPROM Hit
QFP Complex Pass
DDRAM
Netflow
Input ACL
If Action = Inspect, create
session flow in DB
OUTPUT_INSPECT
Boot Flash
Thread 1
Thread 2
Thread 3
Thread 4
(OBFL,…)
MQC Classify Drop
JTAG Ctrl … IP
PPE6 …PPE7 PPE8 PPEN
L4 Inspection
NAT
Unicas PDU reassembly, parsing
PBR
t Packet
Dispatcher
Pass
Buffer L7 Parse
(HTTP GET, POST,…)
Dialer IDLE Rst
PPE2
URD
Creation
Reset / Pwr Ctrl
Output SA table IPV4 OUTPUT
DRAM Interconnect Action Mapping
INSPECT
Child session creation (data flow
RPs RPs ESP RPs from FTP, RTP flow from SIP,…)
SIPs
ESP

FECP
QFP
Crypto
Assist. PPE BQS
intercon.
DRAM SRAM
Input
Temp Sensor
Classify Traffic
EEPROM Hit
QFP Complex Pass
DDRAM
Netflow
Input ACL
OUTPUT_INSPECT
Boot Flash
Thread 1
Thread 2
Thread 3
Thread 4
(OBFL,…)
MQC Classify Drop
JTAG Ctrl …
show policy-firewall IP
session platform
L4 Inspection
--show platform Unicas
NAT hardware qfp active feature firewall datapath scb any any any any any all any --
[s=session i=imprecise channel c=control channel d=data channel]
PBR58513 10.0.0.1 1967 proto 6 (0:0)[sc]
172.18.25.66
Pass
172.18.25.66 59869 10.0.0.1 1967 proto 17 (0:0)[sc]
tDispatcher
Packet Buffer L7 Parse
Dialer IDLE Rst
172.18.25.66 59824 10.2.6.254 1967 proto 6 (0:0)[sc] PPE2
172.18.25.66 56338 10.11.32.15 6665 proto 17 (0:0)[sd]
…
URD
Creation
Reset / Pwr Ctrl
Output SA table
Interconnect
IPV4 OUTPUT
DRAM
INSPECT
ESP

FECP
QFP
Crypto
Assist. PPE BQS
intercon.
DRAM SRAM
Input
Temp Sensor
Classify Traffic
EEPROM Hit
QFP Complex Pass
DDRAM
Netflow
Input ACL
OUTPUT_INSPECT
Boot Flash
Thread 1
Thread 2
Thread 3
Thread 4
(OBFL,…)
MQC Classify Drop
JTAG Ctrl
show policy-firewall
… IP
sessionPPEplatform
PPE PPE
tcp…destination-port
PPE 6
80 detail 7 8 N
L4 Inspection
--show platform
Unicas
NAT hardware qfp active feature firewall datapath scb any any any 80 6 all any detail--
[s=session i=imprecise channel c=control channel d=data channel]
172.18.25.66
…
PBR53471 213.94.72.66 80 proto 6 (0:0)[sc]
Pass t Dispatcher
L7 Parse
nxt_timeout: 100, refcnt: 1, ha nak cnt: 0, rg: 0, sess id: 32584 Buffer
Dialer IDLE Rst Packet
…
PPE2
URD
ingress/egressCrypto Imprecise
intf: GigabitEthernet0/0/2 (1021), GigabitEthernet0/0/3 (65526)Channel Thread 3
current time 1384744571498 create SPI tstamp:
Mux 1384690046997 last access: 1384690179236 L7 Inspection
…
Creation
Reset / Pwr Ctrl
Output syncookie fixup: 0x0
SA table
Interconnect
IPV4 OUTPUT
… DRAM
INSPECT
ESP

FECP
QFP
Crypto
Assist. PPE BQS
intercon.
DRAM SRAM
Input
Temp Sensor
Classify Traffic
EEPROM Hit
QFP Complex Pass
DDRAM
Netflow
Input ACL
OUTPUT_INSPECT
Boot Flash
Thread 1
Thread 2
Thread 3
Thread 4
(OBFL,…)
MQC Classify Drop
show policy-firewall statistics platform
# of sessions freed
JTAG Ctrl … … IP…PPE6 PPE7 PPE8 PPEN
through the lifetime
==FW memory info== # of sessions active L4 Inspection
NAT …
Unicas of the FW
PBR
t Dispatcher
------------Total History----------
Pass # of memory
L7 failures…
Parse
DialerChunk-Pool
IDLE Rst Inuse |Allocated Freed Packet
Alloc_Fail|
Buffer allocation
FW Sessions ------------------------------------------------------------ PPE2
URD scb 33 32851 32818 0
Imprecise Channel
Synflood protect Crypto
hostdb 0 Thread 3
L7 Inspection
SPI11747
Mux 11747 0
ICMP Error 0 0 0 0 Creation
Reset / Pwr Ctrl dst pool 0 0 0 0
Output SA table
… Interconnect IPV4 OUTPUT
DRAM # of sessions
allocated through the INSPECT
lifetime of the FW
ESP
FECP
Post ZBF FIA Continuation Crypto

Assist. PPE
QFP
BQS
intercon.
Packet Buffer Part Len / BW TCAM
TCAM Resource DRAM
DRAM SRAM
Input Policy
Temp Selection
Sensor Session Lookup Miss
Classify Traffic
EEPROM Hit
QFP Complex Pass
DDRAM
Netflow
Input ACL
Drop
Inspect
OUTPUT_INSPECT Create Session

NBAR Classify Session DB
Boot Flash
FECP PPE1 PPE2 PPE3 PPE4
Drop
PPE5
Thread 1
Thread 2
Thread 3
Thread 4
(OBFL,…)
MQC Classify

L4 Inspection
NAT IP Unicast
PBR Pass
Dispatcher L7 Parse
Imprecise Channel PPE2
URD
Crypto L7 Inspection
Thread 3
SPI Mux Creation
Reset / PwrOutput
Ctrl
SA table
Interconnect
IPV4 OUTPUT
DRAM
INSPECT
ESP
Post ZBF FIA Continuation

FECP
QFP
Crypto
Assist. PPE BQS
intercon.
TCAM Resource DRAM
DRAM SRAM
Temp Sensor

EEPROM
QFP Complex
DDRAM
Netflow
Input ACL
OUTPUT_INSPECT
NBAR Classify
Boot Flash
…
Thread 1
Thread 2
Thread 3
Thread 4
(OBFL,…)
MQC Classify
VFR_REFRAG
L2_REWRITE
NAT IP Unicast
PBR
Dispatcher
PPE2
URD
Crypto
SPI Mux Thread 3
Reset / Pwr Ctrl
SA table
DRAM Interconnect

ESP
FECP
ZBF Packet Flow (cont.) Crypto

Assist. PPE
QFP
BQS
intercon.

TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
QFP Complex
Boot Flash
Dispatcher
Packet Buffer
Crypto
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect

#CLUS

Advanced Troubleshooting of The ASR1k and ISR (Including SDWAN Edge) Made Easy

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Advanced Troubleshooting of The ASR1k and ISR (Including SDWAN Edge) Made Easy

Uploaded by

Copyright:

Available Formats

Advanced troubleshooting

of the ASR1k and ISR

Frederic Detienne, Distinguished Engineer

Webex Teams will be moderated cs.co/ciscolivebot#BRKARC-3147

• ACL + show access-list,…

Platform Control Plane

• ESP “stuff” Let’s

Which feature went wrong ?

What went wrong in the feature ?

What went wrong in the feature ?

SHARED PORT ADAPTERS

Embedded Service Processor

SIP interconn. SIP interconn. SIP interconn.

SPA SPA SPA

SPA SPA SPA SPA SPA SPA

SPA Interface Processor

SPA SPA SPA SPA SPA SPA

Centralized Architecture Midplane

SIP interconn. SIP interconn. SIP interconn.

SPA SPA SPA

SPA SPA SPA SPA SPA SPA

Route Processor Architecture interconn. GE switch

Highly Scalable Control Plane Processor

Mgmt Console BITS

ESP20 Block Diagram Crypto

Reset / Pwr Ctrl Packet Buffer

JTAG Ctrl PPE6 PPE7 PPE8 … PPE40

RPs RPs ESP RPs SIPs

ESP20 Block Diagram (comments) Crypto

JTAG Ctrl PPE6 PPE7 PPE8 … PPE40

PCI* E-RP* Packet Processor Engine

RPs RPs ESP RPs SIPs

Traffic Manager (BQS)

Reset / Pwr Ctrl QFP QFP

EEPROM PPE PPE PPE

Boot Flash Packet Buffer Packet Buffer GE, 1Gbps

Reset / Pwr Ctrl

SIP40 Block Diagram SPA SPA

ESPs RPs RPs

Reset / Pwr Ctrl

Ingress buffers Egress buffers

Reset / Pwr Ctrl SPA Agg.

ESI, 23Gbps GE, 1Gbps

SIP40 Block Diagram (comments) SPA SPA

ESPs RPs RPs

Reset / Pwr Ctrl

ESP-20G Interconnect • CRC protection of packet contents

• ESP-10G: 1x11.5G ESI to each SIP slot

DDR3 Control Plane 4xPCIe Data Plane 4xSGMI

Ctrl SVC1 PPE1 PPE2 PPE3 PPE4 PPE5

DDR3 Control Plane 4xPCIe Data Plane 4xSGMI

Front panel PoE

Rangeley CPU mSATA

PPE1 PPE2 PPE3 PPE4 PPE5 Console, Aux & USB

USB Host Ports PCIe Switch

PPE1 PPE2 PPE3 PPE4 PPE5 2 service cores

Rangeley CPU mSATA

PPE1 PPE2 PPE3 PPE4 PPE5 Console, Aux & USB

USB Host Ports PCIe Switch

PPE1 PPE2 PPE3 PPE4 PPE5 2 service cores

USB Host Ports PCIe Switch

4 GB DRAM 1 core as RP hosting IOSd

USB Host Ports PCIe Switch