Ltrarc 2003

IOS-XE Hands on
Troubleshooting Lab
Rama Darbha, Solutions Architect, CCIE #28006
Michael Robertson, Network Engineer, CCIE #33990
LTRARC-2003
Agenda
• IOS-XE platform functionality
• Components
• Interaction with features
• Troubleshooting tools specific to IOS-XE
• Resource monitoring
• Embedded packet capture
• Packet tracing
• Feature specific tools
• Feature specific configuration
• Conditional debugging
• Firewall sessions and drops
• Monitoring traffic using Flexible Netflow
Troubleshooting: What does it mean?
Thank you for joining Company Corporation
• We’re a new organization trying to build and sell widgets
• The widget industry has intense competition, so we hire only the best
• You’re our newest network engineer
• We’re growing so quickly that we needed you to help out with the increasing
demands of network functionality and uptime
IOS-XE Overview
See Appendix for details diagrams
IOS-XE Overview
• Multi-core CPUs
• Symmetric multiprocessing (SMP)
• Each process uses a different core
• IOS runs as a daemon (IOSd) which utilizes multi-core infrastructure
• Data plane and Control plane Separation

• Forward and Feature Manager (FFM) provides set of API to control plane processes
• Forwarding Engine Driver (FED) programs data plane and maintains forwarding state
• Platform Abstraction
• Platform Independent (PI) focused IOS process
• Platform Dependent (PD) code/drivers abstracted from core IOS process
BRKCRS-3147 – Advanced Troubleshooting of ASR1K and ISR (IOS-XE) Made Easy

Day in the Life
of a Normal Packet
Packet Flow RP
CPU
RP
CPU
• Traditional IOS troubleshooting commands aren’t effective IOS
interconn. GE switch
ESP ESP
QFP
FECP
Crypto
QFP
Assist.
interconn.
SIP SIP interconn.
SPA
IOCP
Aggreg.
SPA SPA SPA

Packet Flow RP
CPU
RP
CPU
IOS
X-Connect L2 Switch IPv4 IPv6 MPLS
interconn. GE switch
ESP ESP
Netflow QFP
Netflow
FECP
NAT
Input ACL
NBAR Classify
NBAR Classify IP Unicast Crypto
QFP
… Assist.
MQC Classify
MQC Policing
… IP Multicast interconn.
MAC Accounting
NAT IOS Firewall
SIP SIP
Packet For interconn.
PBR Us Output ACL
SPA
IOCP
Aggreg.
SPA SPA SPA

Lab Topology
Take time now to log into your pods!
Lab Topology
X = PodNumber
Flag Server
172.16.1.200 172.16.1.X 10.1.X.1
192.168.255.X
Web Server
10.1.X.100
Laptop
192.168.1.Y
Day 1 – Overview
• It’s your first day on the job
• We’re a company that only runs IOS-XE
• We were told that it’s better than IOS!
• Your boss tells you to review the health of the routers in your network
• Check them for:
• CPU
• Memory
• Resource allocation
Resource Management
Resource Monitoring Commands
• Initial troubleshooting should be executed using fundamental resource
monitoring commands
Platform Outputs IOS Outputs
show platform resources
show processes platform show processes
show processes memory platform show processes memory
show processes cpu platform show processes cpu
• IOS-XE runs IOS as a daemon on top of Platform OS

• Necessary to look at Platform resource usage
• platform keyword ensures visibility for entire platform
Day 1 – Task 1 Objectives
• Issue the commands you learned to verify the health of network devices
• Use the lab guide to identify IOSd versus Platform based resources
• Answer the fundamental question: Is the router healthy?
• Bonus activity:
• Understand the scope of each command
• Identify the processes that utilize the most memory and CPU
• Trace the PIDs that launched IOSd
Time limit: 5 minutes

Resource Monitoring Outputs
• Look at a high level diagnostic output
• Memory utilization and allocation to RP and ESP
CSR# show platform resources

**State Acronym: H - Healthy, W - Warning, C - Critical
Resource Usage Max Warning Critical State
----------------------------------------------------------------------------------------------------
RP0 (ok, active) H
Control Processor 1.82% 100% 90% 95% H
DRAM 2807MB(72%) 3894MB 90% 95% H
ESP0(ok, active) H
QFP H
DRAM 62776KB(23%) 262144KB 80% 90% H
IRAM 213KB(10%) 2048KB 80% 90% H
Memory used Thresholds
Total Memory
Resources Monitoring – Process Memory IOS
• Memory utilization by processes within IOS
2.2Gb total available IOS

CSR# show processes memory sorted
Processor Pool Total: 2202708736 Used: 274613728 Free: 1928095008
lsmpi_io Pool Total: 6295128 Used: 6294296 Free: 832
PID TTY Allocated Freed Holding Getbufs Retbufs Process

0 0 2798596344 2567772584 212099784 0 0 *Init*
142 2 57289176 35556464 21821304 0 0 SSH Process
72 0 35627680 25075256 5193208 0 0 IOSD ipc task
0 0 14211776 10334328 4907888 9024279 0 *Dead*
136 0 6649376 18536 4676512 0 0 CWAN OIR Handler
418 0 3904392 5216 3941040 849828 0 EEM ED Syslog
467 0 1937856 0 1967720 0 0 qos_mon_periodic
441 0 1446360 27792 1448432 0 0 EEM Server
Resource Monitoring – Process Memory platform
• Memory allocated by Linux to processes
4Gb allocated to entire platform at boot
CSR# show processes memory platform sorted

System memory: 3988376K total, 3402116K used, 586260K free ~3.7Gb max
Lowest: 1112920K allocated to IOSd
Pid Text Data Stack Dynamic RSS Total Name
---------------------------------------------------------------------------
14187 257149 631344 212 56 631732 3706552 linux_iosd...
18537 10868 163836 192 18800 163840 1934784 fman_fp_image
17459 77 169964 120 19244 169968 1505980 cpp_cp_svr
13292 6122 80256 176 2020 80256 1031512 fman_rp
18224 62 91684 84 2276 91688 942132 cpp_sp_svr
17939 319 81264 84 3120 81268 916476 cpp_ha_top...
17696 142 97040 84 2484 97048 900476 cpp_driver
18974 17136 487768 84 268 708296 784232 qfp-ucode-csr
Resource Monitoring – CPU IOS
• CPU usage of processes within IOS
1% CPU usage by IOSd
CSR# show processes cpu sorted

CPU utilization for five seconds: 1%/0%; one minute: 0%; five minutes: 0%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
8 152318 24188 6297 1.19% 0.15% 0.11% 0 Check heaps
380 64231 2479332 25 0.07% 0.04% 0.04% 0 MMA DB TIMER
122 33317 158732 209 0.07% 0.03% 0.00% 0 Per-Second Jobs
169 64148 2479329 25 0.07% 0.03% 0.03% 0 VRRS Main thread
4 0 1 0 0.00% 0.00% 0.00% 0 Retransmission o
3 0 1 0 0.00% 0.00% 0.00% 0 XOS async sync X
5 0 1 0 0.00% 0.00% 0.00% 0 IPC ISSU Dispatc
Resource Monitoring – CPU platform
• CPU processes on Linux sorted by usage 1% CPU
• Equal to the Linux top command used by IOS
CSR# show processes cpu platform sorted

CPU utilization for five seconds: 1%, one minute: 1%, five minutes: 1%
Pid PPid 5Sec 1Min 5Min Status Size Name
--------------------------------------------------------------------------------
18974 18553 4% 4% 4% S 803053568 qfp-ucode-csr
14187 13897 1% 1% 1% S 3795505152 linux_iosd-imag
18537 18027 0% 0% 0% S 1981214720 fman_fp_image
18224 17769 0% 0% 0% S 964739072 cpp_sp_svr
17939 17505 0% 0% 0% S 938467328 cpp_ha_top_leve
17696 17159 0% 0% 0% S 922083328 cpp_driver
17459 16811 0% 0% 0% S 1542119424 cpp_cp_svr
17117 16549 0% 0% 0% S 408887296 cman_fp
~3.8Gb allocated
to IOSd
Day 1 – Summary
• Resources can be allocated to IOS as a daemon or Linux as a platform
• Run the correct command to understand where the resource limitation is occurring
• IOS resources are subset of platform resources
• Processes within IOS are invisible to the platform
• IOSd runs with ~3.8Gb of memory

End of Day 1
Day 2 – Overview
• It’s your second day on the job
• Your boss was impressed with your knowledge of IOS-XE!
• The company is rolling out a new external facing web server to sell widgets
• Application team has built the server and verified it’s up
• Boss needs you to verify that traffic to the web server is properly working
Embedded Packet
Capture
Embedded Packet Capture (EPC)
• Captures are performed on the router on receive and transmit of packet
• Tool for verifying connectivity at network layer
• Captures are performed on the interface

• Seen in capture = seen by router
• Basic captures do not require any configuration commands

• Low impact troubleshooting step
EPC Setup
• Step 1: Identify interface to apply capture
• Capture can be applied inbound, outbound or bidirectional on interface
CSR# monitor capture CAPTURE interface GigabitEthernet X [both|in|out]
• Step 2: Create capture filter

• Match based on IP type and TCP protocol
• Match condition is unidirectional
• Be cautious matching on TCP port and capturing bidirectional traffic
CSR# monitor capture CAPTURE match [any|ipv4|ipv6] [protocol tcp|udp] any any [eq PORT]
• ACL can be used for more granular bidirectional capture

CSR# monitor capture CAPTURE access-list CAPTURE_ACL
EPC Verification
• Step 3: Verify capture configuration
CSR# show monitor capture
• Step 4: Start capture

• Capture needs to be manually started to capture traffic
CSR# monitor capture CAPTURE start
• Step 5: Test traffic

• Initiate connection that is failing
• Wait for automated traffic to fail
EPC View Captured Data
• Step 6: Verify traffic was captured in buffer
CSR# show monitor capture CAPTURE buffer
• Step 7: Stop capture

CSR# monitor capture CAPTURE stop
• Step 8: Look at packet headers in buffer

CSR# show monitor capture CAPTURE buffer brief
• Step 9: Look at packet data in buffer

CSR# show monitor capture CAPTURE buffer detail
CSR# show monitor capture CAPTURE buffer dump
EPC Clear the Capture
• Need to clear the current capture buffer?
CSR# monitor capture CAPTURE clear
• Need to change the capture configuration?

CSR# monitor capture CAPTURE stop
CSR# monitor capture CAPTURE ...
CSR# monitor capture CAPTURE start

• Use the embedded packet capture tool
• Verify traffic is arriving on the external facing interface of the router
• Verify traffic is leaving on the internal facing interface of the router
• Verify bidirectional traffic and communication
• Bonus:
• Look at packet contents to validate:
• TCP flags
• TCP Data
• HTTP headers
• HTTP content
• Set up captures to only capture header data

Embedded Packet Capture Locations
Interface: GigabitEthernet1 Interface: GigabitEthernet2

Capture name: ______________ Capture name: ______________
Embedded Packet Capture Locations

Capture name: CAPOUT Capture name: CAPIN
Embedded Packet Capture Configuration
• Step 1: Configure ACL to capture HTTP traffic to Webserver
CSR(conf)# ip access-list extended CAPTURE_ACL
CSR(conf-ext-nacl)# permit ip host 192.168.1.X host 10.1.X.100
CSR(conf-ext-nacl)# permit ip host 10.1.X.100 host 192.168.1.X
• Step 2: Set up capture on external and internal interface

CSR# monitor capture CAPOUT interface GigabitEthernet1 both
CSR# monitor capture CAPOUT access-list CAPTURE_ACL
CSR# monitor capture CAPOUT start
CSR# monitor capture CAPIN interface GigabitEthernet2 both

CSR# monitor capture CAPIN access-list CAPTURE_ACL
CSR# monitor capture CAPIN start
Embedded Packet Capture Verify Configuration
• Step 3: Look at current CSR# show monitor capture

EPC configuration
Status Information for Capture CAPOUT
Target Type:
Interface and direction Interface: GigabitEthernet1, Direction: both
Status : Active
Filter Details:
ACL used to match traffic Access-list: CAPTURE_ACL
Buffer Details:
Buffer Type: LINEAR (default)
Buffer Size (in MB): 10
Limit Details:
Number of Packets to capture: 0 (no limit)
Packet Capture duration: 0 (no limit)
No capture limits Packet Size to capture: 0 (no limit)
Maximum number of packets to capture per second: 1000
Packet sampling rate: 0 (no sampling)
EPC Buffer Details TCP: SYN
TCP: SYN + ACK
TCP: ACK
• Step 4: Test traffic
HTTP: GET
TCP: ACK
• Step 5: Check packets in
HTTP: Data 1
capture buffer TCP: ACK
HTTP: 200OK
CSR# show monitor capture CAPOUT buffer
buffer size (KB) : 10240
buffer used (KB) : 128
packets in buf : 59
packets dropped : 0 TCP: ACK
packets per sec : 14 TCP: FIN
TCP: FIN
TCP: ACK
EPC Packet Header
• Summary of all packets
CSR#show monitor capture CAPOUT buffer brief
-------------------------------------------------------------
# size timestamp source destination protocol
-------------------------------------------------------------
0 74 0.000000 192.168.1.X -> 10.1.X.100 TCP
1 74 0.000000 10.1.X.100 -> 192.168.1.X TCP
2 66 0.000991 192.168.1.X -> 10.1.X.100 TCP Look at packet size to
3 241 0.000991 192.168.1.X -> 10.1.X.100 TCP assume importance
4 66 0.001999 10.1.X.100 -> 192.168.1.X TCP
5 1514 0.003997 10.1.X.100 -> 192.168.1.X TCP (ie. HTTP GET)
6 1097 0.003997 10.1.X.100 -> 192.168.1.X TCP
7 66 0.003997 10.1.X.100 -> 192.168.1.X TCP
8 66 0.003997 192.168.1.X -> 10.1.X.100 TCP
9 66 0.119033 192.168.1.X -> 10.1.X.100 TCP
10 66 0.119033 192.168.1.X -> 10.1.X.100 TCP < 80 byte packets
11 66 0.205043 10.1.X.100 -> 192.168.1.X TCP normally just signaling
12 66 0.210032 10.1.X.100 -> 192.168.1.X TCP
13 66 0.216028 192.168.1.X -> 10.1.X.100 TCP (ie. ACK, FIN, RST)
EPC Packet Data Full
CSR# show monitor capture CAPOUT buffer dump
0
0000: FA163E3B 994CFA16 3E3F93C6 08004500 ..>;.L..>?....E.
0010: 003C1DCC 40004006 61B3AC10 01C80A01 .<..@.@.a.......
0020: 0364AEC3 0050CBBD 48D00000 0000A002 .d...P..H....... • Full dump of packet
0030: 3908E9BA 00000204 05B40402 080A5142 9.............QB
0040: 551C0000 00000103 0307 U.........
• ASCII output from packet
1
0000: FA163E3F 93C6FA16 3E3B994C 08004500 ..>?....>;.L..E.
0010: 003C0000 40003F06 807F0A01 0364AC10 .<..@.?......d.. 3
0020: 01C80050 AEC3BF8F 96E8CBBD 48D1A012 ...P........H... 0000: FA163E8A ADE0FA16 3ED62901 08004500 ..>.....>.)...E.
0030: 3890A290 00000204 05B40402 080A082C 8.............., 0010: 016F0218 40007F06 2946C0A8 011E0A01 .o..@...)F......
0040: E8EC5142 551C0103 0307 ..QBU..... 0020: 0364D914 00503902 EF53C605 06065018 .d...P9..S....P.
0030: 0100F309 00004745 54202F20 48545450 ......GET / HTTP
2 0040: 2F312E31 0D0A486F 73743A20 31302E31 /1.1..Host: 10.1
0000: FA163E3B 994CFA16 3E3F93C6 08004500 ..>;.L..>?....E. 0050: 2EXX2E31 30300D0A 55736572 2D416765 .X.100..User-Age
0010: 00341DCD 40004006 61BAAC10 01C80A01 .4..@.@.a....... 0060: 6E743A20 4D6F7A69 6C6C612F 352E3020 nt: Mozilla/5.0
0020: 0364AEC3 0050CBBD 48D1BF8F 96E98010 .d...P..H....... 0070: 2857696E 646F7773 204E5420 362E313B (Windows NT 6.1;
0030: 00730976 00000101 080A5142 5520082C .s.v......QBU ., 0080: 20574F57 36343B20 72763A33 372E3029 WOW64; rv:37.0)
0040: E8EC .. 0090: 20476563 6B6F2F32 30313030 31303120 Gecko/20100101
00A0: 46697265 666F782F 33372E30 0D0A4163 Firefox/37.0..Ac
00B0: 63657074 3A207465 78742F68 746D6C2C cept: text/html,
Byte offset Raw HEX ASCII

dump conversion HTTP GET information in ASCII format
Day 2 – Overview Continued
• You verified that all packets are going through the router
• Your boss still isn’t satisfied!

• So the packets are going through the router, but how can we be sure they’re being
processed properly?
• EPC only verify packets arrive/leave an interface

Packet Tracing
The All-in-One Tool
Packet Tracing
• EPC shows you packets, but not how they are processed
• We know packet arrives on external interface but never gets sent to internal
• Packet Tracing is a Swiss army knife

• Singular tool, but provides many different outputs to help isolating the problem
Packet Tracing Explained
• Remember this diagram:
X-Connect L2 Switch IPv4 IPv6 MPLS
Netflow Netflow
Input ACL NAT
NBAR Classify
NBAR Classify
…
MQC Classify IP Unicast
MQC Policing
… IP Multicast
MAC Accounting
NAT Packet For Us
IOS Firewall
PBR Output ACL
• Packet tracing prints each module/process that the packet moves through
Packet Tracing Overview
• Two forms of packet tracing
1. Initial Packet Diagnostics
2. Feature Invocation Array (FIA) Tracing
• Initial Packet Diagnostics
• Verify packets are traced and passed
• FIA Tracing
• Perform detailed step by step analysis
• Understand every process that is touching the packet
• Isolate process that could potentially cause issue
Packet Tracing Locations
• Identify direction of initial packet
• Select receiving interface to apply packet trace
TCP: SYN

Packet direction: receive/transmit Packet direction: receive/transmit
Packet Tracing – Initial Packet Diagnostics
• Step 1: Identify traffic to be packet traced using ACL
• Ensure bidirectional communication
CSR(config)# ip access-list extended PACKET_TRACE
CSR(config-ext-nacl)#10 permit ip host 192.168.1.X host 10.1.X.100
CSR(config-ext-nacl)#20 permit ip host 10.1.X.100 host 192.168.1.X
• Step 2: Set up conditional match condition

• Match criteria should be as specific as possible
CSR# debug platform condition interface [GigabitEthernet X] ipv4 access-list PACKET_TRACE both
CSR# debug platform condition start
Pick receive interface of packet
• Step 3: Enable packet tracing

• Output reminds that conditional matching is required
CSR# debug platform packet-trace enable
Packet Tracing – Process tracing
• (Optional) Step 4: Copy the packet to the local buffer to look at contents
• Copy packets bidirectional, ingress or egress direction
• Copy starting from L2 (MAC), L3 (IP) or L4 (TCP/UDP) header
• Packet size captured up to size of 2048 bytes
• Most IPv4 packets have a maximum size of 1500 bytes
CSR# debug platform packet-trace copy packet [both|input|output] [l2|l3|l4] size [16-2048]
• Step 5: Identify the number of packets to be traced

• Number of packets to be copied and traced
• Must be power of 2
• Fia-trace keyword ensures more detailed packet tracing
CSR# debug platform packet-trace packet [16-8192] fia-trace
Packet Tracing – Validate output
• Step 6: Check configuration of process tracing settings
CSR# show debug CSR# show platform condition
• Step 7: Validate packets were traced and look at summary of packets

CSR# show platform packet-trace statistics CSR# show platform packet-trace summary
• Step 8: Trace the packet through the processing path

• Select packet number based on relevance
• Packet 0 is normally TCP SYN
• Packet 1 and 2 are SYN+ACK and ACK
• Packet 3 is normally first relevant protocol packet
• HTTP GET, FTP PORT/PASV, HTTPS Certificate, SMTP 220 banner
CSR# show platform packet-trace [packet X] Packet number to be displayed

Packet Trace – Disable and Clear
• Step 9: Clear packet tracing data
CSR# debug platform condition stop CSR# show platform packet-trace statistics
Packets Summary
CSR# clear platform packet-trace statistics Matched 0
Traced 0
Packets Received
Ingress 0
Inject 0
Packets Processed
CSR# debug platform condition start Forward 0
Punt 0
Drop 0
Consume 0
• (Optional) Step 10: Clear all conditional matching and packet trace
CSR# clear platform condition all
• Use packet tracing to validate that all packets are properly forwarded
1. Create ACL to match traffic
2. Bind ACL and interface to conditional matching
3. Enable packet tracing functionality
4. Copy packets by selecting correct:
• Starting header point
• Total size of packet
5. Enable FIA-trace by selecting correct:
• Total number of packets captured
• Total size of packets captured
6. Send test traffic
7. View packet tracer statistics
8. View FIA trace of packet
Packet Tracing – Initial Diagnostic Analysis
• Step 1: Identify traffic to be packet traced using ACL
• Ensure bidirectional communication
• ACL is created to match traffic (EPC ACL can be reused)
CSR(config-ext-nacl)#10 permit ip host 192.168.1.X host 10.1.X.100
CSR(config-ext-nacl)#20 permit ip host 10.1.X.100 host 192.168.1.X
• Step 2: Set up conditional match condition

CSR# debug platform condition interface GigabitEthernet 1 ipv4 access-list PACKET_TRACE both
Interface receiving initial packet

• Step 3: Enable packet tracing
Packet Tracing – Process Tracing
• Enable packet copying to trace through each step of packet processing
• Step 4: Copy the packet to the local buffer
CSR# debug platform packet-trace copy packet both l2 size 2048
Copy packets bidirectionally 1500 bytes is rounded up to 2048
Including L2 header to validate

MAC addresses

CSR# debug platform packet-trace packet 16 fia-trace
Tracing first 16 packets because

only validating traffic path
Packet Tracing – Process Tracing
• Step 6: Check configuration of process tracing settings
CSR# show debug
...
IOSXE Conditional Debug Configs:
Conditional Debug Global State: Start
Conditions Direction
----------------------------------------------------------------------------------------------|---------
GigabitEthernet1 & IPV4 ACL [PACKET_TRACE] both
...
IOSXE Packet Tracing Configs: Packet tracing
debug platform packet-trace enable
debug platform packet-trace packet 16 fia-trace data-size 2048 commands
debug platform packet-trace copy packet both size 2048 L2
FIA-trace commands
Packet Tracing – Tracing the packets
• Step 7: Packet has been been matched and traced
CSR# show platform packet-trace statistics
Packets Summary Packet matches conditional criteria
Matched 59
Traced 16
Packets Received
Validate that packets are traced
Ingress 59 in addition to being matched
Inject 0
Packets Processed Packet is received
Forward 59
Punt 0
Drop 0
Packet is forwarded
• Step 8: View traced packet

Packet Tracing – View the traced packet
Header information
• First section of output is header information
CSR# show platform packet-trace packet 0
Packet: 0 CBUG ID: 38
Summary
Input : GigabitEthernet1
Output : GigabitEthernet2
State : FWD
Timestamp
Start : 1121527837046543 ns (03/30/2015 18:23:24.919149 UTC)
Stop : 1121527837482250 ns (03/30/2015 18:23:24.919585 UTC)
Path Trace
Feature: IPV4
Source : 192.168.1.X
Destination : 10.1.X.100
Protocol : 6 (TCP)
SrcPort : 51671
DstPort : 80
...
Trace information
• Second section is FIA trace
...
Feature: FIA_TRACE
Entry : 0x81239458 - IPV4_INPUT_DST_LOOKUP_CONSUME
Lapsed time: 44180 ns
Feature: FIA_TRACE
Entry : 0x80e2ab20 - IPV4_INPUT_ACL
...
Feature: FIA_TRACE
Entry : 0x81269e60 - IPV4_INPUT_LOOKUP_PROCESS
Feature: FIA_TRACE
Entry : 0x8123948c - IPV4_INPUT_IPOPTIONS_PROCESS
...
Feature: FIA_TRACE
Entry : 0x812394b4 - IPV4_VFR_REFRAG
Feature: FIA_TRACE
Entry : 0x81229204 - IPV4_OUTPUT_L2_REWRITE
Feature: FIA_TRACE
Entry : 0x81239494 - IPV4_OUTPUT_FRAG
Feature: FIA_TRACE
Entry : 0x81239498 - IPV4_OUTPUT_DROP_POLICY
Packet data
• Third section is full packet dump in hex format
• Only included if packet is copied
• Includes packet headers and data
• Packet is captured input and output of router
• L2/L3/L4 information in header may changed based on features configured on router
• Data contents of packet should remain the same
• Unless L7 inspection applied
...
Packet Copy In
fa163ef3 b8cafa16 3e9ef62e 08004500 003c7e50 40004006 0389ac10 01c80a01
010ac9d7 00505223 dac10000 0000a002 39087cb3 00000204 05b40402 080a3dc3
a4910000 00000103 0307
Packet Copy Out
fa163eeb f48dfa16 3e590373 08004500 003c7e50 40003f06 0489ac10 01c80a01
010ac9d7 00505223 dac10000 0000a002 39087cb3 00000204 05b40402 080a3dc3
a4910000 00000103 0307
Zero Impact Troubleshooting
• Remove all configurations
• EPC
CSR# no monitor capture CAPOUT
CSR# no monitor capture CAPIN
• Packet Tracing
• Capture ACL
CSR(config)# no ip access-list extended CAPTURE_ACL
CSR(config)# no ip access-list extended PACKET_TRACE
Day 2 – Summary
• Your boss is satisfied!
• Go home and have a beer and relax knowing you did a great job!
• Summary of activities
• Embedded Packet Capture (EPC)
• Capture packet headers and packet data
• Traffic is captured on interface
• Packet tracing with FIA-trace
• Feature specific tracing of packets
• Features build on each other
End of Day 2
Day 3 – Task 1 Troubleshooting
• You come in on your third day of work, feeling good about yesterday’s validation
and verification
• Waiting for you at your desk is your boss, something happened in the middle of
the night and traffic to the webserver is no longer working!
• You tell your boss, “No worry chief! I got this covered.”
• Troubleshoot the issue using the steps you learned:

1. Run the EPC to verify that traffic is properly passing through the router
2. Enable Packet Tracing to identify the cause of the issue
3. Fix the issue with the least impacting solution
Embedded Packet Capture Troubleshooting

CSR# monitor capture CAPOUT access-group CAPTURE_ACL

CSR# monitor capture CAPIN access-group CAPTURE_ACL
• Step 3: Test the traffic

EPC Buffer
TCP: SYN
• Check packets in capture buffer • Check packet flow

CSR# show monitor capture CAPOUT buffer Pod1#show monitor capture CAPOUT buffer brief
buffer size (KB) : 10240 -------------------------------------------------------------
buffer used (KB) : 128 # size timestamp source destination protocol
packets in buf : 1 -------------------------------------------------------------
packets dropped : 0 0 74 0.000000 172.16.1.200 -> 10.1.1.10 TCP
packets per sec : 0
CSR# show monitor capture CAPIN buffer Pod1#show monitor capture CAPIN buffer brief
buffer size (KB) : 10240 -------------------------------------------------------------
packets in buf : 0 -------------------------------------------------------------
packets dropped : 0
packets per sec : 0
Packet Trace - Configuration
• Step 1: Set up ACL to capture traffic
CSR(config-ext-nacl)#10 permit tcp host 192.168.1.X host 10.1.X.100 eq 80
CSR(config-ext-nacl)#20 permit tcp host 10.1.X.100 eq 80 host 192.168.1.X
• Step 2: Configure conditional match and set up packet-tracing

TCP: SYN
• Enable FIA tracing without doing diagnostics

• Copy packets bi-directionally, including L2 header, up to a size of 2048

• Trace 16 packets
Packet Tracing
Packets Summary
Matched 1
Packet is matched and traced
Traced 1 by packet tracing
Packets Received
Ingress 1
Inject 0
Packets Processed
Packet is dropped instead of
Forward 0 forwarded
Punt 0
Drop 1
Count Code Cause Packet is dropped by uRPF
1 297 Ipv4uRpfStrictFailed
Consume 0
CSR#show platform packet-trace summary

Pkt Input Output State Reason
0 Gi1 Gi1 DROP 297 (Ipv4uRpfStrictFailed)
Header information
• Output header verifies the output from the packet trace summary
Summary Packet is dropped by uRPF
State : DROP 297 (Ipv4uRpfStrictFailed)
Timestamp
Start : 1235118556797707 ns (04/01/2015 01:56:35.729811 UTC)
Stop : 1235118557159275 ns (04/01/2015 01:56:35.730172 UTC)
Path Trace
Feature: IPV4
Source : 192.168.1.X
Protocol : 6 (TCP)
SrcPort : 51804
DstPort : 80
...
...
Trace information Feature: FIA_TRACE
Entry : 0x8122ecdc - DEBUG_COND_INPUT_PKT
• The last step in this Feature: FIA_TRACE
Entry : 0x80e31528 - EPC_INGRESS_FEATURE_ENABLE
process is the IPv4 input Lapsed time: 2046540 ns
Feature: FIA_TRACE
strict uRPF check Entry : 0x81239478 - IPV4_INPUT_SRC_LOOKUP_ISSUE
Feature: FIA_TRACE
• Can conclude that the uRPF Entry : 0x81239458 - IPV4_INPUT_DST_LOOKUP_CONSUME
feature is dropping the Lapsed time: 22400 ns
Feature: FIA_TRACE
packet Entry : 0x80e2ab20 - IPV4_INPUT_ACL
Feature: FIA_TRACE
Entry : 0x8123947c - IPV4_INPUT_SRC_LOOKUP_CONSUME
Feature: FIA_TRACE
Feature: FIA_TRACE
Entry Entry : 0x80e5f284 - IPV4_INPUT_RPF_STRICT
: 0x80e4d534 - INPUT_FNF_DROP_EXT
Lapsed
Feature: FIA_TRACE time: 3594886 ns
Packet is dropped by Entry : 0x80e4d60c - INPUT_FNF_AOR_RELEASE_EXT
RPF check Feature: FIA_TRACE
Entry : 0x81229214 - INPUT_DROP_EXT
Feature: FIA_TRACE
Entry : 0x80e5f284 - IPV4_INPUT_RPF_STRICT
...
Packet data
• Packet data isn’t relevant for uRPF drops
• uRPF only looks like source address from IP header
...
Packet Copy In
fa163ef3 b8cafa16 3e9ef62e 08004500 003c1836 40004006 69a3ac10 01c80a01
010aca5c 005088f7 1f650000 0000a002 3908972f 00000204 05b40402 080a4489
07530000 00000103 0307
Packet Copy Out
fa163ef3 b8cafa16 3e9ef62e 08004500 003c1836 40003f06 6aa3ac10 01c80a01
010aca5c 005088f7 1f650000 0000a002 3908972f 00000204 05b40402 080a4489
07530000 00000103 0307
Packet Tracing
• uRPF is set up to drop traffic
interface GigabitEthernet1
ip address 172.16.1.X 255.255.255.0
no ip proxy-arp
ip verify unicast source reachable-via rx allow-default 100
uRPF is enabled in
ip policy route-map REQUIRED_APPLICATIONS strict mode
ip access-group INTERNET_BLACKLIST in
ip route 0.0.0.0 0.0.0.0 172.16.1.203

ip route 172.16.1.200 255.255.255.255 Null0
Null0 route causes
ip route 192.168.0.0 255.255.0.0 Null0 uRPF to drop it
• Fix it by removed the Null0 route
• Removing uRPF is a heavy handed step that won’t fix the problem
• Adding the host to ACL 100 won’t work because return traffic will still be dropped
Day 3 – Troubleshooting 2
• Phew! Fixed the problem just in time for lunch
• Incorrect routes entered into the router was causing uRPF to drop the traffic
• Removing the routes fixed the problem
• Celebrating you go out for a well deserved lunch but when come back from
lunch and the website is down again!
• You tell your boss, “don’t worry captain, I’m on it!”




EPC Buffer
TCP: SYN

CSR# show monitor capture CAPOUT buffer Pod1#show monitor capture CAPOUT buffer brief
buffer size (KB) : 10240 -------------------------------------------------------------
packets in buf : 2 -------------------------------------------------------------
packets dropped : 0 0 74 0.000000 192.168.1.X -> 10.1.X.100 TCP
packets per sec : 0 1 74 0.000101 192.168.1.X -> 10.1.X.100 TCP
CSR# show monitor capture CAPIN buffer Pod1#show monitor capture CAPIN buffer brief
buffer size (KB) : 10240 -------------------------------------------------------------
packets in buf : 0 -------------------------------------------------------------
packets dropped : 0
packets per sec : 0
Packet Trace – Configuration

TCP: SYN
• Enable FIA tracing directly
• We’re confident enough in our troubleshooting ability to go directly to FIA tracing


Packet Tracing
CSR#show platform packet-trace statistics
Packets Summary
Matched 1
Packets Received
Ingress 1
Inject 0
Packets Processed Packet is forwarded which
Forward 1 initially looks correct
Punt 0
Drop 0
Consume 0
CSR#show platform packet-trace summary

Packet is forwarded,
Pkt Input Output State Reason but output interface is
0 Gi1 Gi1 FWD
incorrect
Header information
• Output header verifies the output from the packet trace summary
Packet: 0 CBUG ID: 257 Packet is forwarded,
Summary
Input : GigabitEthernet1 but output interface is
State : FWD incorrect
Timestamp
Start : 1237636474452628 ns (04/01/2015 02:38:33.546242 UTC)
Stop
Path Trace
: 1237636474715741 ns (04/01/2015 02:38:33.546506 UTC) Packet is forwarded
Feature: IPV4
Source : 192.168.1.X
Protocol : 6 (TCP)
SrcPort : 52239
DstPort : 80
...
...
Entry : 0x81239458 - IPV4_INPUT_DST_LOOKUP_CONSUME
• All output looks correct Feature: FIA_TRACE
• Packet passes all input Feature: FIA_TRACE
Entry : 0x80e5e470 - IPV4_INPUT_PBR
checks Lapsed time: 410413 ns
Feature: FIA_TRACE
• Packet passes all output Entry : 0x81269e60 - IPV4_INPUT_LOOKUP_PROCESS
rewrites Feature: FIA_TRACE
Entry : 0x8123948c - IPV4_INPUT_IPOPTIONS_PROCESS
LapsedFeature: FIA_TRACE
time: 6906 ns
• Look for processes that can Feature: FIA_TRACE
Entry Entry : 0x80e5e470 - IPV4_INPUT_PBR
: 0x81269e64 - IPV4_INPUT_GOTO_OUTPUT_FEATURE
change destination Lapsed
Lapsed time: 1220306 time:
ns 410413 ns
Feature: FIA_TRACE
interface Entry : 0x812394b4 - IPV4_VFR_REFRAG
Feature: FIA_TRACE
Packet is being Entry : 0x81229204 - IPV4_OUTPUT_L2_REWRITE
processed by PBR Feature: FIA_TRACE
Entry : 0x81239494 - IPV4_OUTPUT_FRAG
Feature: FIA_TRACE
Entry : 0x81239498 - IPV4_OUTPUT_DROP_POLICY
Feature: FIA_TRACE
Entry : 0x81269e78 - MARMOT_SPA_D_TRANSMIT_PKT
Lapsed time: 304393 ns ...
Packet data
• Packet data is not relevant
• PBR uses header information to route traffic independent of routing table
...
Packet Copy In
fa163ef3 b8cafa16 3e9ef62e 08004500 003c0012 40004006 81c7ac10 01c80a01
010acc0f 00506924 73b90000 0000a002 3908f47c 00000204 05b40402 080a44af
73ab0000 00000103 0307
Packet Copy Out
fa163e5c 0db9fa16 3ef3b8ca 08004500 003c0012 40003f06 82c7ac10 01c80a01
010acc0f 00506924 73b90000 0000a002 3908f47c 00000204 05b40402 080a44af
73ab0000 00000103 0307
Packet Tracing
ip address 172.16.1.X 255.255.255.0
no ip proxy-arp
ip verify unicast source reachable-via rx allow-default 100 PBR configured
ip policy route-map REQUIRED_APPLICATIONS to send traffic to
different next hop
route-map REQUIRED_APPLICATIONS permit 25 Match condition for
match ip address 99
set ip next-hop 172.16.1.203 PBR is incorrectly
! configured
access-list 99 permit host 172.16.1.200
access-list 99 permit 192.168.1.0 0.0.0.255
• PBR is configure to sent traffic to the wrong destination interface

• Fix it by correcting the match condition
Day 3 – Summary
• What a big day! We solved a lot of problems using a universal troubleshooting
tool
• EPC + Packet Tracing
• Thanks to EPC and Packet Tracing, we were able to identify all the problems
• uRPF dropping traffic due to incorrectly configured route
• PBR is sending traffic out wrong interface due to incorrect matching statement
• One last thing to do before we leave for the day...

Zero Impact Troubleshooting
• Remove all troubleshooting configurations
• EPC
CSR# no monitor capture CAPOUT
CSR# no monitor capture CAPIN
• Packet Tracing
• Capture ACL
CSR(config)# no ip access-list extended CAPTURE_ACL
CSR(config)# no ip access-list extended PACKET_TRACE
Use the time now to remove all implemented

troubleshooting commands.
End of Day 3
Day 4 – Overview
• Your boss let you come in late today since you were here late working on fixing
all the problems in the network yesterday
• Now that the network is stable, the boss wants a firewall implemented to protect
the webserver
• A contractor was paid to deploy it this morning
• When you come in, you have some work waiting for you
• You need to validate the firewall configuration implemented by the contractor
Firewall Configuration and
Session Monitoring
Firewall Configuration Validation
• Step 1: Validate firewall only configuration
CSR# show tech-support firewall
• Step 2: Examine sessions through firewall

CSR# show policy-firewall sessions platform
• Sessions can be filtered on IP address or port information

CSR# show policy-firewall sessions platform v4-source-address <IP_ADDR> detail
CSR# show policy-firewall sessions platform v4-destination-address <IP_ADDR> detail
CSR# show policy-firewall sessions platform destination-port <0-65535>
CSR# show policy-firewall sessions platform [tcp|udp]
Drop Counters - Platform and Feature
• Step 3: View drop statistics
• View packets dropped in general
Includes zero drop counters
CSR# show platform hardware qfp active statistics drop
CSR# show platform hardware qfp active statistics drop all
CSR# show platform hardware qfp active statistics drop clear Clears counters after output
Use this command to
initialize output
• View packets dropped by firewall feature
CSR# show platform hardware qfp active feature firewall drop
CSR# show platform hardware qfp active feature firewall drop all Includes zero drop
counters
• Use the commands provided in the previous slides to validate the firewall
configuration
• Examine the sessions on the device to ensure traffic is flowing properly
• Bonus:
• Look at detailed connection logs to examine:
• Egress/Ingress interfaces
• Connection up time

Firewall Configuration Validation
• Validate firewall only configuration
CSR# show tech-support firewall
• Output is grouped by firewall specific outputs

...
------------------ show version ------------------
------------------ show running-config ------------------
...
------------------ show policy-map type inspect zone-pair ------------------
------------------ show class-map type inspect ------------------
------------------ show zone security ------------------
------------------ show zone-pair security ------------------
------------------ show policy-firewall stats global ------------------
...
Firewall Sessions
• View current firewall sessions
Pod1# show policy-firewall sessions platform
--show platform hardware qfp active feature firewall datapath scb any any any any any all any --
[s=session i=imprecise channel c=control channel d=data channel]
Session ID:0x00000000 192.168.1.X 44414 10.1.1.100 80 proto 6 (0:0) [sc]
IP address and IP address and Protocol Data sent by source

port of source port of destination ICMP – 1 and destination
TCP – 6
UDP – 17
• On platforms with many sessions, the output can be filtered down
• Command may cause platform to hang when not using filter
Firewall Drop Counters
• Clear the drop counters first to initialize the output
Global Drop Stats Packets Octets
----------------------------------------------------------------
The Global drop stats were all zero

-------------------------------------------------------------------------------
Drop Reason Packets
-------------------------------------------------------------------------------
• All the output is blank indicating the firewall isn’t dropping anything
• This is a good thing in this scenario
Day 4 – Overview continued
• Your manager wants you to provide more detail information proving that the
traffic is being mapped to the correct zone security policy
• Conditional debugging can provide more detailed information regarding:

• Packet processing through firewall feature
• State of connection
• Zone security policy interaction
Conditional Debugging
Firewall
Conditional Debugging
• Step 1: Enable fundamentals of conditional matching
CSR# debug platform condition interface GigabitEthernet X ipv4 access-list PACKET_TRACE both
• Step 2: Enable Firewall feature specific conditional debugging

CSR# debug platform condition feature fw dataplane submode all level [error|info|verbose|warning]
• Rotate debug file

CSR# cd bootflash:/tracelogs
CSR# test platform software trace slot f0 cpp-control-process rotate
• Use the conditional debugging commands to verify the zone-pair and connection
is built
• Bonus:
• Use the Lab Guide to try to understand the relevant messages
• Use Lab Guide to track connection build states

Firewall Debugs
04/15 19:13:19.311 [cpp-dp-fw]: (info): QFP:0.0 Thread:000 TS:00000056710241760054 :FW_DEBUG_FLG_POLICY:[172.16.1.200] 44419 => [10.1.1.100] 80 6 (0): Check Box SYN flood limit.
per_box_limit_n_cnt: ffffffff00000000 limit ffffffff curr_cnt 0
04/15 19:13:19.311 [cpp-dp-fw]: (info): QFP:0.0 Thread:000 TS:00000056710241763425 :FW_DEBUG_FLG_POLICY:[172.16.1.200] 44419 => [10.1.1.100] 80 6 (0): Check SYN flood limit. limit_n_cnt:
ffffffff00000000 limit ffffffff curr_cnt 0
04/15 19:13:19.311 [cpp-dp-fw]: (verbose): QFP:0.0 Thread:000 TS:00000056710241768271 :FW_DEBUG_L4:[172.16.1.200] 44419 => [10.1.1.100] 80 6 (0): FW: Classify 172.16.1.200:44419 =>
10.1.1.100:80 6 zone pair OUTSIDE->INSIDE_ZP
04/15 19:13:19.311 [cpp-dp]: (verbose): QFP:0.0 Thread:000 TS:00000056710241772794 ###################################################################################################

04/15 19:13:19.311 [cpp-dp]: (verbose): QFP:0.0 Thread:000 TS:00000056710241773658 CCE IPV4 PKT (src:172.16.1.200,dst:10.1.1.100,sprt:ad83,dprt:0050,prot:06,tos:00,len:0014,ttl:3f) ,
intf:3ff
04/15 19:13:19.311 [cpp-dp-fw]: (info): QFP:0.0 Thread:000 TS:00000056710241820789 :FW_DEBUG_EVENT:[172.16.1.200] 44419 => [10.1.1.100] 80 6 (0): Classify Actions: Action Inspect (8)
Protocol http (12) Match_index 4
04/15 19:13:19.312 [cpp-dp-fw]: (info): QFP:0.0 Thread:000 TS:00000056710241824855 :FW_DEBUG_EVENT:[172.16.1.200] 44419 => [10.1.1.100] 80 6 (0): Classify Action: Rule[7361776.10422817.1]
ACE(10)
04/15 19:13:19.312 [cpp-dp-fw]: (info): QFP:0.0 Thread:000 TS:00000056710241914895 :FW_DEBUG_L4:[172.16.1.200] 44419 => [10.1.1.100] 80 6 (0): Session ID:0x00000005, TCP: detect SYN. state
was LISTEN -> SYNSENT seq# 2171264973 ack# 0 window 14600 zone-pair name OUTSIDE->INSIDE_ZP
04/15 19:13:19.312 [cpp-dp-fw]: (info): QFP:0.0 Thread:000 TS:00000056710241918324 :FW_DEBUG_L4:[172.16.1.200] 44419 => [10.1.1.100] 80 6 (0): Session ID:0x00000005, TCP: start SYNWAIT timer
- 30000 msec
04/15 19:13:19.312 [cpp-dp-fw]: (info): QFP:0.0 Thread:000 TS:00000056710243938614 :FW_DEBUG_L4:[10.1.1.100] 80 => [172.16.1.200] 44419 6 (0): Session ID:0x00000005, TCP: state was SYNSEND -
> SYNRCVD seq# 2663126407 ack# 2171264974 window 14480 zone-pair name OUTSIDE->INSIDE_ZP
04/15 19:13:19.314 [cpp-dp-fw]: (info): QFP:0.0 Thread:000 TS:00000056710245500749 :FW_DEBUG_L4:[172.16.1.200] 44419 => [10.1.1.100] 80 6 (0): Session ID:0x00000005, TCP: state was SYNRCVD -
> ESTAB seq# 2171264974 ack# 2663126408 window 115 zone-pair name OUTSIDE->INSIDE_ZP
04/15 19:13:19.314 [cpp-dp-fw]: (info): QFP:0.0 Thread:000 TS:00000056710245507922 :FW_DEBUG_L4:[172.16.1.200] 44419 => [10.1.1.100] 80 6 (0): Session ID:0x00000005, TCP: start TCP idle
timer - 3600000 msec
Day 4 – Summary
• Your boss is happy and lets you go home, knowing that the FW is working
properly
• Summary of Activities
• Firewall feature specific command outputs
• Outputs to validate firewall sessions
• Monitor drops caused by firewall feature
End of Day 4
Day 5 – Troubleshooting 1
• You come in the next day and, what would you know, looks like traffic is broken
again!
• Use the troubleshooting tools we’ve learned so far to validate the behaviour
• EPC
• Packet tracing
• Firewall session and drops
• Use the Lab Guide to follow the proper troubleshooting steps

Firewall - EPC



Firewall - EPC Buffer
TCP: SYN

CSR# show monitor capture CAPOUT buffer CSR# show monitor capture CAPOUT buffer brief
buffer size (KB) : 10240 -------------------------------------------------------------
packets in buf : 1 -------------------------------------------------------------
packets dropped : 0 0 74 0.000000 192.168.1.X -> 10.1.X.100 TCP
packets per sec : 0
CSR# show monitor capture CAPIN buffer CSR# show monitor capture CAPIN buffer brief
buffer size (KB) : 10240 -------------------------------------------------------------
packets in buf : 2 -------------------------------------------------------------
packets dropped : 0 0 74 0.000000 192.168.1.X -> 10.1.1.100 TCP
packets per sec : 0 1 74 0.003997 10.1.X.100 -> 192.168.1.X TCP
Firewall - Packet Tracing

Enable packet tracing on INSIDE interface

as that is the one dropping packets
TCP: SYN
Firewall – Process Tracing


Firewall – Packet Trace Summary
Packets Summary
Matched 12 Packet is matched and traced
Packets Received
Ingress 12
Inject 6
Count Code Cause
6 6 QFP Fwall generated packet
Packets Processed Half the packets are dropped
Forward 12
Punt 0
Drop 6 Return packet is
Count Code Cause dropped by firewall
6 185 FirewallL4
Consume 0
CSR# show platform packet-trace summary

0 Gi1 Gi2 FWD
1 Gi2 Gi1 DROP 185 (FirewallL4)
Firewall – Sessions and Drop counters
• Step 1: Current connection count shows no connections through firewall
CSR# show policy-firewall sessions platform v4-destination-address 10.1.X.100
--show platform hardware qfp active feature firewall datapath scb any any any any any all any --
• Step 2: Drop counters indicates that packets are being dropped due:
• Firewall
• Invalid TCP initiator
-------------------------------------------------------------------------
Global Drop Stats Packets Octets
-------------------------------------------------------------------------
FirewallL4 6 444

-------------------------------------------------------------------------------
Drop Reason Packets
-------------------------------------------------------------------------------
Invalid TCP initiator 6
Firewall – Conditional Debugging
• Step 3: Use conditional debugging to understand why firewall is dropping
packets
CSR# debug platform condition feature fw dataplane submode all level verbose
• Same as packet tracing condition

• GigabitEthernet2 as interface because that is interface dropping packet
Rotated file from: /tmp/fp/trace/stage/cpp_cp_F0-0.log.17643.20150415195840, Bytes: 22584, Messages: 109
CSR# more cpp_cp_F0-0.log.17643.20150415195840

Firewall Debugs
• Incoming packet debugs
04/15 19:58:36.030 [cpp-dp-fw]: (info): QFP:0.0 Thread:000 TS:00000059426961978584 :FW_DEBUG_L4:[172.16.1.200] 44429 => [10.1.1.100] 80 6 (0): Session miss:- SI packet
04/15 19:58:36.030 [cpp-dp-fw]: (info): QFP:0.0 Thread:000 TS:00000059426961986317 :FW_DEBUG_L4:[172.16.1.200] 44429 => [10.1.1.100] 80 6 (0): Searching for imprecise channel
10.1.1.100:80 172.16.1.200:44429 proto TCP (0:0)
04/15 19:58:36.030 [cpp-dp-fw]: (info): QFP:0.0 Thread:000 TS:00000059426961994721 :FW_DEBUG_L4:[172.16.1.200] 44429 => [10.1.1.100] 80 6 (0): Could not find imprecise channel -
ip_id:41667 seq #3822981330 window:14600
04/15 19:58:36.030 [cpp-dp-fw]: (info): QFP:0.0 Thread:000 TS:00000059426962001751 :FW_DEBUG_FLG_POLICY:[172.16.1.200] 44429 => [10.1.1.100] 80 6 (0): Check Box SYN flood limit.
per_box_limit_n_cnt: ffffffff00000000 limit ffffffff curr_cnt 0
04/15 19:58:36.030 [cpp-dp-fw]: (info): QFP:0.0 Thread:000 TS:00000059426962007744 :FW_DEBUG_FLG_POLICY:[172.16.1.200] 44429 => [10.1.1.100] 80 6 (0): Check SYN flood limit.
04/15 19:58:36.030 [cpp-dp-fw]: (info): QFP:0.0 Thread:000

limit_n_cnt: ffffffff00000000 limit ffffffff curr_cnt 0
10.1.1.100:80 6 zone pair OUTSIDE->INSIDE_ZP
TS:00000059426962078964 :FW_DEBUG_L4 FW_DEBUG_FLG_POLICY :[172.16.1.200] 44429
04/15 19:58:36.030 [cpp-dp]: (verbose): QFP:0.0 Thread:000 TS:00000059426962019777
###################################################################################################
=> [10.1.1.100] 80 6 (0): Policy Action Pass: Class CUSTOM_APPLICATIONS zone
04/15 19:58:36.030 [cpp-dp]: (verbose): QFP:0.0 Thread:000 TS:00000059426962021351 CCE IPV4 PKT (src:172.16.1.200,dst:10.1.1.100,sprt:ad8d,dprt:0050,prot:06,tos:00,len:0014,ttl:3f)
, intf:3ff
pair OUTSIDE->INSIDE_ZP
###################################################################################################
04/15 19:58:36.030 [cpp-dp]: (verbose): QFP:0.0 Thread:000 TS:00000059426962026797 CCE IPV4 UIDB_INFO W0:ea0cd910, W1:20000000, tcam_region_index:d910, key_index:00, cmd:00000000
04/15 19:58:36.030 [cpp-dp]: (verbose): QFP:0.0 Thread:000 TS:00000059426962053461 cce_sw_tcam_top:: root:0xea0cd910 top:0xea0cd934 pType:0000000000
04/15 19:58:36.030 [cpp-dp]: (verbose): QFP:0.0 Thread:000 TS:00000059426962057167 cce_sw_tcam_dump_key:: 0xac1001c8 :0x06002fff :0x0050ad8d :0x0a010164
:0xd9100001
04/15 19:58:36.030 [cpp-dp-fw]: (info): QFP:0.0 Thread:000 TS:00000059426962066407 :FW_DEBUG_EVENT:[172.16.1.200] 44429 => [10.1.1.100] 80 6 (0): Classify Actions: Action Unknown
Classify Action (82) Protocol unknown_proto (0) Match_index 0
04/15 19:58:36.030 [cpp-dp-fw]: (info): QFP:0.0 Thread:000 TS:00000059426962072141 :FW_DEBUG_EVENT:[172.16.1.200] 44429 => [10.1.1.100] 80 6 (0): Classify Action:
Rule[7361776.2547953.1] ACE(10)
04/15 19:58:36.030 [cpp-dp-fw]: (info): QFP:0.0 Thread:000 TS:00000059426962078964 :FW_DEBUG_L4 FW_DEBUG_FLG_POLICY :[172.16.1.200] 44429 => [10.1.1.100] 80 6 (0): Policy Action
Pass: Class CUSTOM_APPLICATIONS zone pair OUTSIDE->INSIDE_ZP
04/15 19:58:36.032 [cpp-dp-fw]: (verbose): QFP:0.0 Thread:000 TS:00000059426962085371 :FW_DEBUG_EVENT:[172.16.1.200] 44429 => [10.1.1.100] 80 6 (0): fw_invoke passed packet
04/15 19:58:36.032 [cpp-dp-fw]: (verbose): QFP:0.0 Thread:000 TS:00000059426964090102 :FW_DEBUG_EVENT:[10.1.1.100] 80 => [172.16.1.200] 44429 6 (0): insp pkt: in GigabitEthernet2
out GigabitEthernet1 pkt_sb_flags 0x80000000 pkt_sb_flags2 0x0 izone 2 ozone 1 vrf is not enabled. nvi is not enabled.
04/15 19:58:36.032 [cpp-dp-fw]: (verbose): QFP:0.0 Thread:000 TS:00000059426964099949 :FW_DEBUG_EVENT:[10.1.1.100] 80 => [172.16.1.200] 44429 6 (0): fw_ntuple: key1-ipaddr1
10.1.1.100 key1-ipaddr2 172.16.1.200 proto 6 key-flags 0 key2-port1 80 key2-port2 44429 ip6addr1 - :: ip6addr2 - :: ip6port1 - 80 ip6port2 - 44429
Firewall Debugs
• Response packet debugs
04/15 19:58:36.032 [cpp-dp-fw]: (info): QFP:0.0 Thread:000
TS:00000059426964201819 :FW_DEBUG_EVENT:[10.1.1.100] 80 => [172.16.1.200]
44429
04/15 19:58:36.0326[cpp-dp-fw]:
(0): (info):
Classify Action:
QFP:0.0 Thread:000 Rule[7683424.2459297.1]
TS:00000059426964107459 :FW_DEBUG_L4:[10.1.1.100] 80 => [172.16.1.200]ACE(20)
44429 6 (0): Session miss:- SI packet
04/15 19:58:36.235 [cpp-dp-fw]:
172.16.1.200:44429 6 zone pair INSIDE->OUTSIDE_ZP (warn): QFP:0.0 Thread:000
TS:00000059426964260225 :FW_DEBUG_FLG_DROP:[10.1.1.100] 80 => [172.16.1.200]
###################################################################################################
04/15 19:58:36.032 [cpp-dp]: (verbose): QFP:0.0 Thread:000 TS:00000059426964117685 CCE IPV4 PKT (src:10.1.1.100,dst:172.16.1.200,sprt:0050,dprt:ad8d,prot:06,tos:00,len:0014,ttl:3f)
44429
19:58:36.0326[cpp-dp]:
, intf:3fe
04/15 (0): Proto
(verbose): QFP:0.0 TCP (6)
Thread:000 not initiator Dropping packet reason 37:Invalid TCP
TS:00000059426964120769
###################################################################################################
initiator
04/15 19:58:36.032 [cpp-dp]: (verbose): QFP:0.0 Thread:000 TS:00000059426964121889 CCE IPV4 UIDB_INFO W0:ea0bea80, W1:20000000, tcam_region_index:ea80, key_index:00, cmd:00000000
04/15 19:58:36.032 [cpp-dp]: (verbose): QFP:0.0 Thread:000 TS:00000059426964125369 cce_sw_tcam_top:: root:0xea0bea80 top:0xea0beaa4 pType:0000000000
04/15 19:58:36.032 [cpp-dp]: (verbose): QFP:0.0 Thread:000 TS:00000059426964127925 cce_sw_tcam_dump_key:: 0x0a010164 :0x06012fff :0xad8d0050 :0xac1001c8
:0xea800001
04/15 19:58:36.032 [cpp-dp-fw]: (info): QFP:0.0 Thread:000 TS:00000059426964194635 :FW_DEBUG_ALG_INSPECT:[10.1.1.100] 80 => [172.16.1.200] 44429 6 (0): L7 App type calculated l7
appl 1
04/15 19:58:36.032 [cpp-dp-fw]: (info): QFP:0.0 Thread:000 TS:00000059426964197889 :FW_DEBUG_EVENT:[10.1.1.100] 80 => [172.16.1.200] 44429 6 (0): Classify Actions: Action Inspect
(8) Protocol unknown_proto (0) Match_index 1
04/15 19:58:36.032 [cpp-dp-fw]: (info): QFP:0.0 Thread:000 TS:00000059426964201819 :FW_DEBUG_EVENT:[10.1.1.100] 80 => [172.16.1.200] 44429 6 (0): Classify Action:
Rule[7683424.2459297.1] ACE(20)
04/15 19:58:36.235 [cpp-dp-fw]: (warn): QFP:0.0 Thread:000 TS:00000059426964260225 :FW_DEBUG_FLG_DROP:[10.1.1.100] 80 => [172.16.1.200] 44429 6 (0): Proto TCP (6) not initiator
Dropping packet reason 37:Invalid TCP initiator
04/15 19:58:38.030 [cpp-dp-fw]: (verbose): QFP:0.0 Thread:000 TS:00000059428961952609 :FW_DEBUG_EVENT:[172.16.1.200] 44429 => [10.1.1.100] 80 6 (0): insp pkt: in GigabitEthernet1
out GigabitEthernet2 pkt_sb_flags 0x80000000 pkt_sb_flags2 0x0 izone 1 ozone 2 vrf is not enabled. nvi is not enabled.
04/15 19:58:38.030 [cpp-dp-fw]: (verbose): QFP:0.0 Thread:000 TS:00000059428961969506 :FW_DEBUG_EVENT:[172.16.1.200] 44429 => [10.1.1.100] 80 6 (0): fw_ntuple: key1-ipaddr1
10.1.1.100 key1-ipaddr2 172.16.1.200 proto 6 key-flags 0 key2-port1 80 key2-port2 44429 ip6addr1 - :: ip6addr2 - :: ip6port1 - 80 ip6port2 - 44429
Firewall – Dropped Packet Analysis
• Initiating packet is let through on pass action by firewall
[172.16.1.200] 44429 => [10.1.1.100] 80 6 (0): Policy Action Pass: Class CUSTOM_APPLICATIONS zone pair
OUTSIDE->INSIDE_ZP
• Return packets are dropped by invalid TCP initiator by firewall

[10.1.1.100] 80 => [172.16.1.200] 44429 6 (0): Proto TCP (6) not initiator Dropping packet reason 37:Invalid
TCP initiator
• Invalid initiator is anything that is not a TCP SYN packet

• Including TCP SYN+ACK
• Look at ZBFW configuration to validate policy

Firewall – Configuration
• Traffic is being passed in one direction and being inspected in the other
policy-map type inspect OUTSIDE->INSIDE_PMAP policy-map type inspect INSIDE->OUTSIDE_PMAP
class type inspect CUSTOM_APPLICATIONS class type inspect TRUSTED_APPLICATIONS
pass inspect
class type inspect BLACKLIST_BOGONS class type inspect CUSTOM_APPLICATIONS
drop log pass
class type inspect ALLOWED_APPLICATIONS class class-default
inspect drop
class class-default
drop
• There is no existing connection when the SYN+ACK returns and is dropped by

Firewall security checks
• Re-order the Firewall policy to fix the issue
Firewall – Fixed Configuration
• Pass action is prioritized in both directions
policy-map type inspect OUTSIDE->INSIDE_PMAP policy-map type inspect INSIDE->OUTSIDE_PMAP
class type inspect CUSTOM_APPLICATIONS class type inspect CUSTOM_APPLICATIONS
pass pass
class type inspect BLACKLIST_BOGONS class type inspect TRUSTED_APPLICATIONS
drop log inspect
class type inspect ALLOWED_APPLICATIONS class class-default
inspect drop
class class-default
drop
Day 5 – Summary
• Wow, we’ve learned so much in just 5 days
• Iterative troubleshooting helps drill down to the heart of the problem

• Conditional debugging showed the incorrect policies being applied
End of Day 5
Day 6 – Task 1
• With all the mis-steps that you’ve had to fix recently, your manager wants you to
set up some proactive logging
• Monitor traffic through the CSR using flexible netflow (FNF)
• Benefits of monitoring traffic

• Post mortem analysis of application failure
• Trail for malicious activity
• Correlation of data using monitoring tool
Monitoring Traffic
Monitoring Traffic using Flexible Netflow (FNF)
• Set up a flow exporter flow exporter FLOWEXPORTER
destination 10.1.X.100
source GigabitEthernet2
ttl 5
transport udp 5500
template data timeout 5 Send netflow data
every 5 seconds
• Bind flow exported to flow monitor flow monitor FLOWMONITOR

exporter FLOWEXPORTER Simple netflow data
record netflow-original Can be customized for
more detail
• Apply flow monitor to interface interface GigabitEthernet1

ip flow monitor FLOWMONITOR input
View netflow data on server
• Visit http://10.1.X.100/netflow.php to view netflow data
Day 6 – Summary
• With all the mis-steps that you’ve had to fix recently, your manager wants you to
set up some proactive logging
• Monitor traffic through the CSR via Flexible Netflow
Summary
• Resource monitoring of IOSd vs Platform
• Initial troubleshooting should always leverage EPC
• Validate traffic is received and forwarded by router
• Use packet tracing as your Swiss army knife
• It’s the same 5 commands repeated for every problem experienced
• Output will be specific to the issue
• Implement traffic monitoring

• Set up Flexible Netflow to monitor traffic
Participate in the “My Favorite Speaker” Contest
Promote Your Favorite Speaker and You Could Be a Winner
• Promote your favorite speaker through Twitter and you could win $200 of Cisco
Press products (@CiscoPress)
• Send a tweet and include
• Your favorite speaker’s Twitter handle @Radar_Bot @mjr9804
• Two hashtags: #CLUS #MyFavoriteSpeaker
• You can submit an entry for more than one of your “favorite” speakers
• Don’t forget to follow @CiscoLive and @CiscoPress
• View the official rules at http://bit.ly/CLUSwin
Complete Your Online Session Evaluation
• Give us your feedback to be
entered into a Daily Survey
Drawing. A daily winner
will receive a $750 Amazon
gift card.
• Complete your session surveys
though the Cisco Live mobile
app or your computer on
Cisco Live Connect.
Don’t forget: Cisco Live sessions will be available
for viewing on-demand after the event at
CiscoLive.com/Online
Continue Your Education
• Demos in the Cisco campus
• Walk-in Self-Paced Labs
• Table Topics
• Meet the Engineer 1:1 meetings
• Related sessions
Thank you
Internet of Things (IoT) Cisco Education Offerings
Course Description Cisco Certification
NEW! CCNA Industrial An associate level instructor led training course designed to prepare you CCNA® Industrial
for the CCNA Industrial certification
Managing Industrial Networks with This curriculum addresses foundational skills needed to manage and Cisco Industrial
Cisco Networking Technologies (IMINS) administer networked industrial control systems. It provides plant Networking Specialist
administrators, control system engineers and traditional network engineers
with an understanding of the networking technologies needed in today's
connected plants and enterprises
Control Systems Fundamentals For IT and Network Engineers, covers basic concepts in Industrial Control
for Industrial Networking (ICINS) systems including an introduction to automation industry verticals,
automation environment and an overview of industrial control networks
Networking Fundamentals For Industrial Engineers and Control System Technicians, covers basic IP
for Industrial Control Systems (INICS) and networking concepts, and introductory overview of Automation
industry Protocols.
For more details, please visit: http://learningnetwork.cisco.com

Questions? Visit the Learning@Cisco Booth or contact ask-edu-pm-dcv@cisco.com
Business Transformation Cisco Education Offerings
For IT and Network Professionals:
Building Business Specialist Skills • Builds non-technical skills key to ensure business impact and influence. Cisco Enterprise IT
Topics include: business analysis, finance, technology adoption and Business Specialist
effective communications.
• Bridges IT and business impacts of mature and emerging solutions

including cloud plus Internet of Everything
For Technology Sellers:
Applying Cisco Specialized Business Value Builds skills to discover and address technology needs using a business- Cisco Business Value Specialist
Analysis Skills focused, consultative sales approach
Executing Advanced Cisco Business Value Enables customer transformation through business architecture and Cisco Certified Business
Analysis and Design Techniques solution selling expertise Value Practitioner
Performing Cisco Business-Focused Provides skills and an approach to build a strategic roadmap of IT Cisco Transformative
Transformative Architecture Engagements initiatives, aligned to business priorities Architecture Specialist
Security Cisco Education Offerings
Implementing Cisco IOS Network Security (IINS) Focuses on the design, implementation, and monitoring of a comprehensive CCNA® Security
security policy, using Cisco IOS security features
Implementing Cisco Edge Network Security Solutions
(SENSS) Configure Cisco perimeter edge security solutions utilizing Cisco Switches, Cisco
Routers, and Cisco Adaptive Security Appliance (ASA) Firewalls
Implementing Cisco Threat Control Solutions (SITCS)
Deploy Cisco’s Next Generation Firewall (NGFW) as well as Web Security, Email
Implementing Cisco Secure Access Solutions (SISAS) Security and Cloud Web Security
Implementing Cisco Secure Mobility Solutions Deploy Cisco’s Identity Services Engine and 802.1X secure network access
(SIMOS)
Protect data traversing a public or shared infrastructure such as the Internet by
implementing and maintaining Cisco VPN solutions
Securing Cisco Networks with Threat Detection and Designed for professional security analysts, the course covers essential areas of Cisco Cybersecurity Specialist
Analysis (SCYBER) competency including event monitoring, security event/alarm/traffic analysis, and
incident response

Network Security
Questions? VisitProduct and Solutions Training
the Learning@Cisco For official
Booth or contact product training on Cisco’s latest security products, including Adaptive
ask-edu-pm-dcv@cisco.com
Security Appliances, NGIPS, Advanced Malware Protection, Identity Services
Engine, Email and Web Security Appliances see
www.cisco.com/go/securitytraining
R&S Related Cisco Education Offerings
CCIE R&S Advanced Workshops (CIERS-1 & Expert level trainings including: instructor led workshops, self CCIE® Routing & Switching
CIERS-2) plus assessments, practice labs and CCIE Lab Builder to prepare candidates
Self Assessments, Workbooks & Labs for the CCIE R&S practical exam.
• Implementing Cisco IP Routing v2.0 Professional level instructor led trainings to prepare candidates for the CCNP® Routing & Switching
• Implementing Cisco IP Switched CCNP R&S exams (ROUTE, SWITCH and TSHOOT). Also available in
Networks V2.0 self study eLearning formats with Cisco Learning Labs.
• Troubleshooting and Maintaining
Cisco IP Networks v2.0
Interconnecting Cisco Networking Devices: Configure, implement and troubleshoot local and wide-area IPv4 and IPv6 CCNA® Routing & Switching
Part 2 (or combined) networks. Also available in self study eLearning format with Cisco Learning
Lab.
Interconnecting Cisco Networking Devices: Installation, configuration, and basic support of a branch network. Also CCENT® Routing & Switching
Part 1 available in self study eLearning format with Cisco Learning Lab.

Wireless Cisco Education Offerings
• Conducting Cisco Unified Wireless Site Survey Professional level instructor led trainings to prepare candidates to conduct CCNP® Wireless
• Implementing Cisco Unified Wireless Voice site surveys, implement, configure and support APs and controllers in
Networks converged Enterprise networks. Focused on 802.11 and related
• Implementing Cisco Unified Wireless Mobility technologies to deploy voice networks, mobility services, and wireless
Services security.
• Implementing Cisco Unified Wireless Security
Services
Implementing Cisco Unified Wireless Network Prepares candidates to design, install, configure, monitor and conduct CCNA® Wireless
Essential basic troubleshooting tasks of a Cisco WLAN in Enterprise installations.

Design Cisco Education Offerings
Designing Cisco Network Service Architectures Provides learner with the ability to perform conceptual, intermediate, and CCDP® (Design Professional)
(ARCH) detailed design of a network infrastructure that supports desired capacity,
performance, availability required for converged Enterprise network
services and applications.
Designing for Cisco Internetwork Solutions Instructor led training focused on fundamental design methodologies used CCDA® (Design Associate)
(DESGN) to determine requirements for network performance, security, voice, and
wireless solutions. Prepares candidates for the CCDA certification exam.

Service Provider Cisco Education Offerings
Deploying Cisco Service Provider Network Routing SPROUTE covers the implementation of routing protocols (OSPF, IS-IS, BGP), CCNP Service Provider®
(SPROUTE) & Advanced (SPADVROUTE) route manipulations, and HA routing features; SPADVROUTE covers advanced
routing topics in BGP, multicast services including PIM-SM, and IPv6;
Implementing Cisco Service Provider Next-Generation
Core Network Services (SPCORE) SPCORE covers network services, including MPLS-LDP, MPLS traffic engineering,
QoS mechanisms, and transport technologies;
Edge Network Services (SPEDGE) SPEDGE covers network services, including MPLS Layer 3 VPNs, Layer 2 VPNs,
and Carrier Ethernet services; all within SP IP NGN environments.
Building Cisco Service Provider Next-Generation The two courses introduce networking technologies and solutions, including OSI CCNA Service Provider®
Networks, Part 1&2 (SPNGN1), (SPNGN2) and TCP/IP models, IPv4/v6, switching, routing, transport types, security, network
management, and Cisco OS (IOS and IOS XR).
Implementing Cisco Service Provider Mobility UMTS The three courses (SPUMTS, SPCDMA, SPLTE) cover knowledge and skills Cisco Service Provider Mobility
Networks (SPUMTS); required to understand products, technologies, and architectures that are found in CDMA to LTE Specialist;
Implementing Cisco Service Provider Mobility CDMA Universal Mobile Telecommunications Systems (UMTS) and Code Division Multiple Cisco Service Provider Mobility UMTS
Networks (SPCDMA); Access (CDMA) packet core networks, plus their migration to Long-Term Evolution to LTE Specialist
Implementing Cisco Service Provider Mobility LTE (LTE) Evolved Packet Systems (EPS), including Evolved Packet Core (EPC) and
Networks (SPLTE) Radio Access Networks (RANs).
Implementing and Maintaining Cisco Technologies Service Provider/Enterprise engineers to implement, verification-test, and optimize Cisco IOS XR Specialist
Using IOS XR (IMTXR) core/edge technologies in a Cisco IOS XR environment.
Collaboration Cisco Education Offerings
CCIE Collaboration Advanced Workshop (CIEC) Gain expert-level skills to integrate, configure, and troubleshoot complex CCIE® Collaboration
collaboration networks
Implementing Cisco Collaboration Applications Understand how to implement the full suite of Cisco collaboration CCNP® Collaboration
(CAPPS) applications including Jabber, Cisco Unified IM and Presence, and Cisco
Unity Connection.
Implementing Cisco IP Telephony and Video Learn how to implement Cisco Unified Communications Manager, CUBE, CCNP® Collaboration
Part 1 (CIPTV1) and audio and videoconferences in a single-site voice and video network.
Implementing Cisco IP Telephony and Video Obtain the skills to implement Cisco Unified Communications Manager in a
Part 2 (CIPTV2) modern, multisite collaboration environment.
Troubleshooting Cisco IP Telephony and Video Troubleshoot complex integrated voice and video infrastructures
(CTCOLLAB)
Implementing Cisco Collaboration Devices Acquire a basic understanding of collaboration technologies like Cisco Call CCNA® Collaboration
(CICD) Manager and Cisco Unified Communications Manager.
For more details,

Implementing Ciscoplease visit: http://learningnetwork.cisco.com
Video Network Devices Learn how to evaluate requirements for video deployments, and implement
Questions?
(CIVND) Visit the Learning@Cisco Booth or contact ask-edu-pm-dcv@cisco.com
Cisco Collaboration endpoints in converged Cisco infrastructures.
Data Center / Virtualization Cisco Education Offerings
Cisco Data Center CCIE Unified Fabric Prepare for your CCIE Data Center practical exam with hands on lab CCIE® Data Center
Workshop (DCXUF); exercises running on a dedicated comprehensive topology
Cisco Data Center CCIE Unified Computing
Workshop (DCXUC)
Implementing Cisco Data Center Unified Fabric Obtain the skills to deploy complex virtualized Data Center Fabric and CCNP® Data Center
(DCUFI); Computing environments with Nexus and Cisco UCS.
Implementing Cisco Data Center Unified
Computing (DCUCI)
Introducing Cisco Data Center Networking Learn basic data center technologies and how to build a data center CCNA® Data Center
(DCICN); Introducing Cisco Data Center infrastructure.
Technologies (DCICT)
Product Training Portfolio: DCAC9k, DCINX9k, Get a deep understanding of the Cisco data center product line including
DCMDS, DCUCS, DCNX1K, DCNX5K, DCNX7K the Cisco Nexus9K in ACI and NexusOS modes

Network Programmability Cisco Education Offerings
Integrating Business Applications with Network Learn networking concepts, and how to deploy and troubleshoot Cisco Business Application
Programmability (NIPBA); programmable network architectures with these self-paced courses. Engineer Specialist Certification
Integrating Business Applications with Network
Programmability for Cisco ACI (NPIBAACI)
Developing with Cisco Network Programmability Learn how to build applications for network environments and effectively Cisco Network Programmability
(NPDEV); bridge the gap between IT professionals and software developers. Developer Specialist Certification
Developing with Cisco Network Programmability
for Cisco ACI (NPDEVACI)
Designing with Cisco Network Programmability Learn how to expand your skill set from traditional IT infrastructure to Cisco Network Programmability
(NPDES); application integration through programmability. Design Specialist Certification
Designing with Cisco Network Programmability
for Cisco ACI (NPDESACI)
Implementing Cisco Network Programmability Learn how to implement and troubleshoot open IT infrastructure Cisco Network Programmability
(NPENG); technologies. Engineer Specialist Certification
For more details,
Implementing Ciscoplease visit:
Network http://learningnetwork.cisco.com
Programmability
Questions?
for Cisco ACIVisit the Learning@Cisco Booth or contact ask-edu-pm-dcv@cisco.com
(NPENGACI)
Cloud Cisco Education Offerings
Designing the FlexPod Solution (FPDESIGN); Learn how to design, implement and administer FlexPod solutions FlexPod Design Specialist;
Implementing and Administering the FlexPod FlexPod Implementation &
Solution (FPIMPADM) Administration Specialist
UCS Director (UCSDF) Learn how to manage physical and virtual infrastructure using
orchestration and automation functions of UCS Director.
Cisco Prime Service Catalog Learn how to deliver data center, workplace, and application services in an
on-demand, automated, and repeatable method.
Cisco Intercloud Fabric Learn how to implement end-to-end hybrid clouds with Intercloud Fabric
for Business and Intercloud Fabric for Providers.
Cisco Intelligent Automation for Cloud Learn how to implement and manage cloud deployments with Cisco
Intelligent Automation for Cloud
Appendix
ASR1K Software Architecture
RP
CPU
RP
Chassis Manager
IOS CPU
ESI (10-40 Gbps)

Forwarding Manager
interconn.Linux GE switch
Kernel
ESP ESP
FECP Chassis Manager
EOBC (1 Gbps) FECP
Drivers Forwarding Manager
Drivers
I2C Drivers
Linux Kernel
Crypto
QFP
Assist.
QFP
µ µµ Crypto
µ BQS
µ µ Assist.
interconn.
ESI (10-40 Gbps)
SIP SIP
IOCP interconn.
SPA Driver Chassis
SPA Driver
SPA Driver Manager
SPA
IOCP
Aggreg.
Linux Kernel
SPA SPA SPA SPA SPA

Chassis Manager (CM)
RP
CPU
Chassis Manager
IOS
CM on RP communicates with CM processes on
ESI (10-40 Gbps)

Forwarding Manager
ESP and SIP
• Distributed function Linux Kernel
Initializes hardware and boots other processes

• CM on SIP queries SPA type and load SPA drivers ESP FECP Chassis Manager
Manages hardware components
EOBC (1 Gbps)
• Manages EOBC on RP Drivers
Drivers
I2C
• Manages ESI links on RP/ESP/SIP Linux Kernel
• Manages timing circuitry on RP
• Reset and power-down on RP/ESP/SIP QFP
µ µµ Crypto
µ BQS
µ µ Assist.
Communicates IOS hardware components
ESI (10-40 Gbps)

• Static & OIR
SIP
IOCP
Monitors environmental variables and alarms SPA Driver Chassis
SPA Driver
SPA Driver Manager
Selects active/standby RP or ESP
• Coordinates switchover in case of failure or operator command Linux Kernel
SPA SPA SPA

Forwarding Manager (FMAN)
RP
CPU FMAN-RP
Chassis Manager
IOS
ESI (10-40 Gbps)

FMAN on RP communicates with Forwarding Manager
FMAN process on ESP Linux Kernel

• Distributed function
FMAN-FP
Propagates control plane ops. to ESP ESP FECP ESP Manager
Chassis aka Forwarding Plane
• CEF tables, ACL’s, NAT, SA’s,…
EOBC (1 Gbps)
Drivers
FMAN-FP communicates information Drivers
I2C
Linux Kernel
back to FMAN-RP
QFP
• e.g. statistics µ µµ Crypto
µ BQS
• FMAN-RP pushes info back to IOS µ µ Assist.
ESI (10-40 Gbps)

FMAN on active RP maintains state for SIP
both active & standby ESP’s IOCP
SPA Driver
SPA Driver
Chassis
SPA Driver Manager
• Facilitates NSF after re-start with bulk download of
state information
Linux Kernel
SPA SPA SPA

PPE Microcode
RP
CPU
Written in C IOS
Chassis Manager
ESI (10-40 Gbps)

• proper features, no hack Forwarding Manager
Runs on each thread of the PPE Linux Kernel
Processes packets ESP FECP Chassis Manager

• run to completion
EOBC (1 Gbps)
• assisted by various memories Drivers
TCAM, DRAM,… various speeds PPE Microcode runs here
Drivers
I2C
• Linux Kernel
Features applied via FIA µ µµ

QFP
QFP
Packet Processor Engine
PPE PPE PPE PPE PPE
1 2 3 4 5
BQS
BQS
Crypto
µ …
PPE PPE PPE PPE
• Feature Invocation Array 6
µ µ
7 8
Dispatcher
N
Assist.
Packet Buffer
ESI (10-40 Gbps)

FIA per interface SIP
IOCP
SPA Driver Chassis
• input FIA, output FIA SPA Driver
SPA Driver Manager
• drop FIA (Null interface)
Linux Kernel
SPA SPA SPA

ASR1K Software Architecture
RP
CPU
RP
Chassis Manager
IOS CPU
ESI (10-40 Gbps)

Forwarding Manager
interconn.Linux GE switch
Kernel
ESP ESP
FECP Chassis Manager
EOBC (1 Gbps) FECP
Drivers
I2C Drivers
Linux Kernel
Crypto
QFP
Assist.
QFP
µ µµ Crypto
µ BQS
µ µ Assist.
interconn.
ESI (10-40 Gbps)
SIP SIP
IOCP interconn.
SPA Driver Chassis
SPA Driver
SPA Driver Manager
SPA
IOCP
Aggreg.
Linux Kernel
SPA SPA SPA SPA SPA

Chassis Manager (CM)
RP
CPU
Chassis Manager
IOS
CM on RP communicates with CM processes on
ESI (10-40 Gbps)

Forwarding Manager
ESP and SIP
• Distributed function Linux Kernel
Initializes hardware and boots other processes

• CM on SIP queries SPA type and load SPA drivers ESP FECP Chassis Manager
Manages hardware components
EOBC (1 Gbps)
• Manages EOBC on RP Drivers
Drivers
I2C
• Manages ESI links on RP/ESP/SIP Linux Kernel
• Manages timing circuitry on RP
• Reset and power-down on RP/ESP/SIP QFP
µ µµ Crypto
µ BQS
µ µ Assist.
Communicates IOS hardware components
ESI (10-40 Gbps)

• Static & OIR
SIP
IOCP
Monitors environmental variables and alarms SPA Driver Chassis
SPA Driver
SPA Driver Manager
Selects active/standby RP or ESP
• Coordinates switchover in case of failure or operator command Linux Kernel
SPA SPA SPA

Forwarding Manager (FMAN)
RP
CPU FMAN-RP
Chassis Manager
IOS
ESI (10-40 Gbps)

FMAN on RP communicates with Forwarding Manager
FMAN process on ESP Linux Kernel

• Distributed function
FMAN-FP
Propagates control plane ops. to ESP ESP FECP ESP Manager
Chassis aka Forwarding Plane
• CEF tables, ACL’s, NAT, SA’s,…
EOBC (1 Gbps)
Drivers
FMAN-FP communicates information Drivers
I2C
Linux Kernel
back to FMAN-RP
QFP
• e.g. statistics µ µµ Crypto
µ BQS
• FMAN-RP pushes info back to IOS µ µ Assist.
ESI (10-40 Gbps)

FMAN on active RP maintains state for SIP
both active & standby ESP’s IOCP
SPA Driver
SPA Driver
Chassis
SPA Driver Manager
• Facilitates NSF after re-start with bulk download of
state information
Linux Kernel
SPA SPA SPA

PPE Microcode
RP
CPU
Written in C IOS
Chassis Manager
ESI (10-40 Gbps)

• proper features, no hack Forwarding Manager
Runs on each thread of the PPE Linux Kernel
Processes packets ESP FECP Chassis Manager

• run to completion
EOBC (1 Gbps)
• assisted by various memories Drivers
TCAM, DRAM,… various speeds PPE Microcode runs here
Drivers
I2C
• Linux Kernel
Features applied via FIA µ µµ

QFP
QFP
Packet Processor Engine
PPE PPE PPE PPE PPE
1 2 3 4 5
BQS
BQS
Crypto
µ …
PPE PPE PPE PPE
• Feature Invocation Array 6
µ µ
7 8
Dispatcher
N
Assist.
Packet Buffer
ESI (10-40 Gbps)

FIA per interface SIP
IOCP
SPA Driver Chassis
• input FIA, output FIA SPA Driver
SPA Driver Manager
• drop FIA (Null interface)
Linux Kernel
SPA SPA SPA

SIP intercon.
SPA
IOCP
Aggreg.
Ingress Packet Through SIP ESPs

SPA SPA
Reset / Pwr Ctrl Interconnect

EV-RP
Temp Sensor EV-FC In ref
clocks
EEPROM
DDRAM Egress
Ingress
Buffer
Boot Flash IOCP Scheduler
Status
(OBFL,…) (SC854x SOC)
…
JTAG Ctrl
SPA Aggregation Network
clock
ASIC (Marmot) … distribution
…
Ingress buffers Egress buffers

(per port) (per port)
Network
clocks
Ingress Classifier
Reset / Pwr Ctrl
SPA C2W
Agg.
SPA
ESP
FECP
QFP
Ingress Packet Through ESP Crypto

Assist. PPE BQS
intercon.
Reset / Pwr Ctrl Packet Buffer Part Len / BW

TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
QFP
DDRAM Packet Processor EngineComplex BQS
Boot Flash PPE1 PPE2 PPE3 PPE4 PPE5

(OBFL,…) FECP
JTAG Ctrl PPE6 PPE7 PPE8 … PPEN
Dispatcher
Packet Buffer
Crypto
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect
RPs RPs ESP RPs SIPs

ESP
FECP
QFP
Packet Dispatched to PPE Core Crypto

Assist. PPE BQS
intercon.

TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
PPE2 QFP

FECP
Thread 1
Thread 2
Thread 3
Thread 4
(OBFL,…)
Dispatcher
Packet Buffer
Crypto
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect

ESP
FECP
QFP
Packet Dispatched to PPE Thread Crypto

Assist. PPE BQS
intercon.

TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
PPE2 QFP

FECP
Thread 1
Thread 2
Thread 3
Thread 4
(OBFL,…)
Dispatcher
Packet Buffer
Crypto
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect

ESP
FECP
FIA’s Applied on Packet by PPE Thread Crypto QFP

Assist. PPE BQS
intercon.
X-ConnectReset / Pwr Ctrl

L2 Switch IPv4 IPv6 MPLS
Packet Buffer Part Len / BW
TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
Input FIA Output FIA
PPE2 QFP
Netflow
DDRAM
Input ACL
Packet Processor EngineNetflow Complex BQS
NAT
Boot Flash NBAR Classify PPE1 PPE2 PPE3 PPE4 PPE5
FECP
Thread 1
Thread 2
Thread 3
Thread 4
(OBFL,…) NBAR Classify
JTAG Ctrl … PPE6 PPE7 PPE8 … … N
PPE
NAT IP Multicast MQC Policing

PBR MAC Accounting
Dispatcher
Dialer IDLE Rst Packet For
Packet Buffer Output ACL
Us PPE2
URD
Crypto Thread 3
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect

ESP
FECP
FIA’s Applied on Packet by PPE Thread Crypto QFP

Assist. PPE BQS
intercon.

TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
PPE2 QFP
Netflow
DDRAM
Input ACL
Packet Processor EngineNetflow Complex BQS
NAT
FECP
Thread 1
Thread 2
Thread 3
Thread 4
(OBFL,…) NBAR Classify
JTAG Ctrl … PPE6 PPE7 PPE8 … … N
PPE

PBR MAC Accounting
Dispatcher
Packet Buffer Output ACL
Us PPE2
URD
Crypto Thread 3
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect

ESP
FECP
QFP
Leaving the PPE Thread Crypto

Assist. PPE BQS
intercon.

TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
PPE2 QFP
Netflow Netflow
BGP Accounting
NAT
FECP NBAR Classify
Thread 1
Thread 2
Thread 3
Thread 4
(OBFL,…)
JTAG Ctrl … PPE6 PPE7 PPE8 … …N
PPE

PBR
Dispatcher WRED
Packet Buffer
Us Output ACL PPE2
URD
Crypto Thread 3
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect

ESP
FECP
QFP
Packet proceeding to BQS then SIP Crypto

Assist. PPE BQS
intercon.

TCAM Resource DRAM
DRAM SRAM
Temp Sensor
EEPROM
QFP

(OBFL,…) FECP
Dispatcher
Packet Buffer
Crypto
SPI Mux
Reset / Pwr Ctrl
SA table
DRAM Interconnect

SIP intercon.
SPA
IOCP
Aggreg.
Egress Packet Through SIP ESPs

SPA SPA
Reset / Pwr Ctrl Interconnect

EV-RP
Temp Sensor EV-FC In ref
clocks
EEPROM
DDRAM Egress
Ingress
Buffer
Boot Flash IOCP Scheduler
Status
(OBFL,…) (SC854x SOC)
JTAG Ctrl
SPA Aggregation Network
clock
ASIC (Marmot) … distribution
…
Ingress buffers Egress buffers

(per port) (per port)
Network
clocks
Ingress Classifier
Reset / Pwr Ctrl
SPA C2W
Agg.
SPA
Resource Monitoring – Processes IOS
• Processes used by IOS
CSR# show processes
CPU utilization for five seconds: 0%/0%; one minute: 0%; five minutes: 0%
PID QTy PC Runtime (ms) Invoked uSecs Stacks TTY Process
1 Cwe 7F645CF5AB33 2 13 15322656/24000 0 Chunk Manager
8 Lst 7F645CF88ECD 152689 24234 630022704/24000 0 Check heaps
9 Cwe 7F645CF7DBCD 286 2651 10722784/24000 0 Pool Manager
13 Cwe 7F64586985DE 1864 79491 2323192/24000 0 IOSXE heartbeat
37 Mwe 7F6459F5EF9D 2511 5257 47721376/24000 0 ARP Input
38 Mwe 7F6459F5936E 3896 165846 2321584/24000 0 ARP Background
41 Lwe 7F645A4969F2 0 1 023408/24000 0 CEF MIB API
49 Hwe 7F6458699A71 0 1 022512/24000 0 IOSXE signals IO
67 Lwe 7F645A7BFEC0 1 83 1246152/48000 0 Logger
68 Mwe 7F645A66B6CF 4067 158960 2522960/24000 0 TTY Background
This command only shows processes inside the IOS daemon. Use platform keyword to
Please use 'show processes platform' see processes for Linux
to show processes for the underlying operating system.
Resource Monitoring – Processes platform
• Processes running on platform
CSR# show processes platform
CPU utilization for five seconds: 0%, one minute: 1%, five minutes: 1%
Pid PPid Status Size Name
--------------------------------------------------------
1 0 S 1863680 init
13292 12762 S 1056264192 fman_rp
11915 1 S 4292608 pvp.sh
13897 11915 S 4227072 pman.sh
14187 13897 S 3795505152 linux_iosd-imag Process name
16251 15821 S 192000000 vman
18537 18027 S 1981214720 fman_fp_image
18974 18553 S 803053568 qfp-ucode-csr
Parent PID
Current PID
Resource Monitoring Outputs
Pod1# show processes platform detailed name linux_iosd-imag
Name: linux_iosd-imag
Process id : 14312
Parent process id: 14036
Current process
Group id : 14312
Status : S
Session id : 12385 Parent process
User time : 1017598
Kernel time : 314621
Priority : 20
Virtual bytes
Resident pages
: 3795484672
: 149681
~3.8 Gb
Resident limit : 4294967295
Minor page faults: 155255
Major page faults: 1245
Resource Monitoring – Process Memory platform
• Memory allocated by Linux to processes
4Gb allocated to entire platform at boot
CSR# show processes memory platform sorted

System memory: 3988376K total, 3402116K used, 586260K free ~3.7Gb max
Lowest: 1112920K allocated to IOSd
Pid Text Data Stack Dynamic RSS Total Name
---------------------------------------------------------------------------
14187 257149 631344 212 56 631732 3706552 linux_iosd...
18537 10868 163836 192 18800 163840 1934784 fman_fp_image
17459 77 169964 120 19244 169968 1505980 cpp_cp_svr
13292 6122 80256 176 2020 80256 1031512 fman_rp
18224 62 91684 84 2276 91688 942132 cpp_sp_svr
17939 319 81264 84 3120 81268 916476 cpp_ha_top...
17696 142 97040 84 2484 97048 900476 cpp_driver
18974 17136 487768 84 268 708296 784232 qfp-ucode-csr
~600Mb currently
used by IOSd
Resource Monitoring – Memory IOS
CSR# show memory
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 7F63DA92A010 2202708736 274602132 1928106604 1928057400 1297759876
lsmpi_io 7F6386C281A8 6295128 6294304 824 824 412
Processor memory
Address Bytes Prev Next Ref PrevF NextF Alloc PC what

7F63DA92A2A0 0000032776 7F63DA92A010 7F63DA932300 001 -------- -------- 7F645CF5B8C4 Managed Chunk Queue Elements
7F63DA9355F0 0000020008 7F63DA932390 7F63DA93A470 001 -------- -------- 7F645CF3158B List Elements
7F63DA93A470 0000010008 7F63DA9355F0 7F63DA93CBE0 001 -------- -------- 7F645CF315CD List Headers
7F63DA93CBE0 0000032776 7F63DA93A470 7F63DA944C40 001 -------- -------- 7F645868DA5B IOSXE Process Structure Chunks
7F63DA944C40 0000032776 7F63DA93CBE0 7F63DA94CCA0 001 -------- -------- 7F645868DA95 IOSXE Queue Producer Structure Chunks
lsmpi_io memory
Address Bytes Prev Next Ref PrevF NextF Alloc PC what

7F6386C281A8 0004195336 00000000 7F6387028608 001 -------- -------- 7F6459ACDC8F lsmpi_rx
7F6387028608 0000000336 7F6386C281A8 7F63870287B0 001 -------- -------- 7F6458678195 Init
7F63870287B0 0000000416 7F6387028608 7F63870289A8 000 7F646390A668 7F6387228E08 7F6458678195 (fragment)
7F63870289A8 0002098184 7F63870287B0 7F6387228E08 001 -------- -------- 7F6459ACDC8F lsmpi_tx
7F6387228E08 0000000416 7F63870289A8 00000000 000 7F63870287B0 0 7F645CF73184 (fragment)
This command only shows memory for the IOS daemon.

Please use 'show memory platform'
to show memory for the underlying operating system.
Resource Monitoring – Memory platform
CSR# show memory platform
Virtual memory : 13315932160
Pages resident : 526287
Major page faults: 2720
Minor page faults: 2585103
Architecture : x86_64 4Gb allocated to

Memory (kB) entire VM at boot
Physical : 3988376
Total : 3988376
Used : 2809180
Free : 1179196
Active : 1169448
...
Buffers (kB) : 162260
Load Average
1-Min : 0.00
5-Min : 0.00
15-Min : 0.00
EPC Capture Filter
• Filters can be replaced
CSR# monitor capture CAPOUT access-list CAPTURE_ACL

A filter is already attached to the capture. Replace with new access-list?[confirm]y
EPC Export capture
• Capture can be pulled off in PCAP format
CSR# monitor capture buffer CAPTURE export tftp://<SERVER_IP>/CAPTURE.pcap
EPC Advanced Settings
• Set packet length size
• Only IPv4 headers packets
CSR# monitor capture CAPTURE limit packet-len 64
• Set capture time duration

• 60 minutes from start of capture
CSR# monitor capture CAPTURE limit duration 3600
• Set number of packets captured

• Capture only 25 packets
CSR# monitor capture CAPTURE limit packets 25
• Set total size of capture

• 10Mb capture
CSR# monitor capture CAPTURE buffer 10
Embedded Packet Capture Verify Configuration
CSR# show monitor capture
• Using match statement Status Information for Capture CAPOUT

Target Type:
instead of ACL Interface: GigabitEthernet1, Direction: both
Status : Active
Filter Details:
IPv4
Capture settings Source IP: any
Destination IP: any
Protocol: tcp
Buffer Details:
Buffer Type: LINEAR (default)
Buffer Size (in MB): 10
Limit Details:
Number of Packets to capture: 0 (no limit)
Packet Capture duration: 0 (no limit)
Packet Size to capture: 0 (no limit)
Maximum number of packets to capture per second: 1000
Packet sampling rate: 0 (no sampling)
EPC Packet Data Condensed
CSR# show monitor capture CAPOUT buffer detail
-------------------------------------------------------------
# size timestamp source destination protocol
-------------------------------------------------------------
• Shows basic header output of 0 74 0.000000 192.168.1.X -> 10.1.1.10 TCP
0000: FA163EF3 B8CAFA16 3E9EF62E 08004500 ..>.....>.....E.
packet 0010: 003CE96C 40004006 986CAC10 01C80A01
0020: 010AC9C4 00505A08 DFCA0000 0000A002
.<.l@.@..l......
.....PZ.........
0030: 3908E899 00000204 05B40402 080A32ED 9.............2.
• Header dump of packet 1 74 0.005004 10.1.1.10 -> 192.168.1.X TCP

0000: FA163E9E F62EFA16 3EF3B8CA 08004500 ..>.....>.....E.
0010: 003C0000 40003F06 82D90A01 010AAC10 .<..@.?.........
• Always shows first 64 bytes 0020: 01C80050 C9C4AD3A 88A45A08 DFCBA012 ...P...:..Z.....
0030: 38905A7B 00000204 05B40402 080A32E2 8.Z{..........2.
• Used for diagnostic analysis 2 66 0.005996 192.168.1.X

0000: FA163EF3 B8CAFA16 3E9EF62E
-> 10.1.1.10
08004500
TCP
..>.....>.....E.
0010: 0034E96D 40004006 9873AC10 01C80A01 .4.m@.@..s......
0020: 010AC9C4 00505A08 DFCBAD3A 88A58010 .....PZ....:....
0030: 0073C156 00000101 080A32ED 36B432E2 .s.V......2.6.2.
3 240 0.005996 192.168.1.X -> 10.1.1.10 TCP

0000: FA163EF3 B8CAFA16 3E9EF62E 08004500 ..>.....>.....E.
0010: 00E2E96E 40004006 97C4AC10 01C80A01 ...n@.@.........
0020: 010AC9C4 00505A08 DFCBAD3A 88A58018 .....PZ....:....
0030: 00733CFD 00000101 080A32ED 36B432E2 .s<.......2.6.2.
Packet Tracing – Initial Packet Diagnostics
• Start with light weight output command
• If all packets are forwarded, then further troubleshooting may not be necessary
• Packet tracing requires conditional matching

• Use ACL to identify traffic to be traced
Packet Tracing Locations
• Identify direction of initial packet
• Select receiving interface to apply packet trace
TCP: SYN

Packet direction: receive/transmit Packet direction: receive/transmit
Packet Tracing – Initial Diagnostic Analysis
• Output reminds that conditional matching is required
Please remember to turn on 'debug platform condition start' for packet-trace to work
Day 3 – Task 1 Troubleshooting
• You come in on your third day of work, feeling good about yesterday’s validation
and verification
• Waiting for you at your desk is your boss, something happened in the middle of
the night and traffic to the webserver is no longer working!
• You tell your boss, “No worry chief! I got this covered.”

3. Fix the issue using the least impacting solution



EPC Buffer
TCP: SYN

CSR# show monitor capture CAPOUT buffer CSR# show monitor capture CAPOUT buffer brief
buffer size (KB) : 10240 -------------------------------------------------------------
packets in buf : 1 -------------------------------------------------------------
packets dropped : 0 0 74 0.000000 172.16.1.200 -> 10.1.1.10 TCP
packets per sec : 0
CSR# show monitor capture CAPIN buffer CSR# show monitor capture CAPIN buffer brief
buffer size (KB) : 10240 -------------------------------------------------------------
packets in buf : 0 -------------------------------------------------------------
packets dropped : 0
packets per sec : 0
Packet Trace - Configuration
CSR(config-ext-nacl)#10 permit tcp host 172.16.1.200 host 10.1.1.10 eq 80
CSR(config-ext-nacl)#20 permit tcp host 10.1.1.10 eq 80 host 172.16.1.20

TCP: SYN


Packet Tracing – View packet trace results
CSR# show platform packet-trace
statistics
Packets Summary
Matched 1
Packets Received
Ingress 1
Inject 0
Packets Processed
Packet is dropped instead of
Forward 0 forwarded
Punt 0
Drop 1
Count Code Cause Packet is dropped by ACL
1 8 Ipv4Acl
Consume 0
CSR# show platform packet-trace summary

0 Gi1 Gi1 DROP 8 (Ipv4Acl)
Packet Tracing – View traced packet
Header information
• Drop reason is verified in packet trace
Summary Packet is dropped by ACL
State : DROP 8 (Ipv4Acl)
Timestamp
Start : 1122422779503245 ns (03/30/2015 18:38:19.485261 UTC)
Stop : 1122422779752948 ns (03/30/2015 18:38:19.485510 UTC)
Path Trace
Feature: IPV4
Source : 172.16.1.200
Destination : 10.1.1.10
Protocol : 6 (TCP)
SrcPort : 51673
DstPort : 80
...
Packet Tracing – View traced packet
...
Entry : 0x8122ecdc - DEBUG_COND_INPUT_PKT
• The last step in this Feature: FIA_TRACE
Entry : 0x80e31528 - EPC_INGRESS_FEATURE_ENABLE
process is the IPv4 input Lapsed time: 1011673 ns
Feature: FIA_TRACE
ACL Entry : 0x81239458 - IPV4_INPUT_DST_LOOKUP_CONSUME
Feature: FIA_TRACE
• Can conclude that an input Entry : 0x80e63054 - STILE_LEGACY_DROP_EXT
ACL is dropping this packet Lapsed time: 110133 ns
Feature: FIA_TRACE
Entry : 0x80e59c74 - INGRESS_MMA_LOOKUP_DROP_EXT
Feature: FIA_TRACE
Entry : 0x80e4da14 - INPUT_FNF_AOR_DROP_EXT
Feature: FIA_TRACE
Feature: FIA_TRACE
Entry Entry - INPUT_FNF_DROP_EXT
: 0x80e4d534 : 0x80e2ab20 - IPV4_INPUT_ACL
Feature: FIA_TRACE Lapsed time: 3041106 ns
Packet is dropped by Entry : 0x80e4d60c - INPUT_FNF_AOR_RELEASE_EXT
interface ACL Feature: FIA_TRACE
Entry : 0x81229214 - INPUT_DROP_EXT
Feature: FIA_TRACE
...
Packet Tracing –the traced packet
Packet data
• Third section
• Displays full packet data
• No relevant information here this time because packet was dropped by ACL
• ACL drops packet based on header information
...
Packet Copy In
fa163ef3 b8cafa16 3e9ef62e 08004500 003ce490 40004006 9d48ac10 01c80a01
010ac9d9 00502be2 a8230000 0000a002 39082d61 00000204 05b40402 080a3dd1
4cb30000 00000103 0307
Packet Copy Out
fa163ef3 b8cafa16 3e9ef62e 08004500 003ce490 40003f06 9e48ac10 01c80a01
010ac9d9 00502be2 a8230000 0000a002 39082d61 00000204 05b40402 080a3dd1
4cb30000 00000103 0307
Packet Tracing
• IPv4_INPUT_ACL indicates that interface ACL is dropping the packet
• Check contents of ACL and make necessary corrections
ip address 172.16.1.X 255.255.255.0
no ip proxy-arp
ip verify unicast source reachable-via rx allow-default 100
ip policy route-map REQUIRED_APPLICATIONS
ACL is set up to
drop traffic
• Looks like the ACL has an incorrect entry

• Edit the ACL to fix the problem
CSR# show ip access-list INTERNET_BLACKLIST
Extended IP access list INTERNET_BLACKLIST
10 deny tcp any any eq 80(20 matches)
20 permit ip any any (6399 matches)
Firewall Sessions – Detailed connection information
CSR# show policy-firewall sessions platform v4-destination-address 10.1.X.100 detail
--show platform hardware qfp active feature firewall datapath scb any any 10.1.X.100 any any all any detail--
Session ID:0x0000045C 192.168.1.X 44686 10.1.X.100 80 proto 6 (0:0) [sc]
pscb : 0xe8e9ad40, bucket : 17121, fw_flags: 0x204 0x20419541, key1_flags: 0x0
scb state: active, scb debug: 0
nxt_timeout: 360000, refcnt: 1, ha nak cnt: 0, rg: 0, sess id: 1116
hostdb: 0x0, L7: 0x0, stats: 0xea08b740, child: 0x0
l4blk0: fca98680 l4blk1: ffcbd34a l4blk2: fca98681 l4blk3: 730107
l4blk4: ffcbd349 l4blk5: fca98681 l4blk6: ffcbd34a l4blk7: 38900107
l4blk8: 100 l4blk9: 18
root scb: 0x0 act_blk: 0xea082f80
ingress/egress intf: GigabitEthernet1 (1023), GigabitEthernet2 (65529)
current time 36241272590863 create tstamp: 36209333363434 last access: 36209337740504
nat_out_local_addr:port: 0.0.0.0:0 nat_in_global_addr:port: 0.0.0.0:0
syncookie fixup: 0x0
halfopen linkage: 0x0 0x0
cxsc_cft_fid: 0x0
tw timer: 0x0 0x0 0x37263 0x3700101
Number of simultaneous packet per session allowed: 25
bucket 14080 flags 1 func 1 idx 0 wheel 0xe91c5420
Debug Log Management
• Debugs are written to file on Linux platform
• Logs must be rotated in order to be viewed in IOS
Rotated file from: /tmp/fp/trace/stage/cpp_cp_F0-0.log.17459.20150417042100, Bytes: 501006, Messages: 2093
• Rotate logs messages before gathering information to initialize

• Rotate logs again after testing traffic to create filtered log file
• Log files have maximum size and write to consecutive files

• Multiple log files can be merged to make single log file
• Merges all files in RP
request platform software trace slot rp active merge target bootflash:MERGED_OUTPUT

Netflow Records CSR# show flow record netflow-original
flow record netflow-original:
Description: Traditional IPv4 input NetFlow with origin ASs
• Netflow record data fields No. of users: 1
Total field space: 53 bytes
can be viewed Fields:
match ipv4 tos
match ipv4 protocol
match ipv4 source address
match ipv4 destination address
match transport source-port
match transport destination-port
match interface input
match flow sampler
collect routing source as
collect routing destination as
collect routing next-hop address ipv4
collect ipv4 source mask
collect ipv4 destination mask
collect transport tcp flags
collect interface output
collect counter bytes
collect counter packets
collect timestamp sys-uptime first
collect timestamp sys-uptime last
Firewall High Speed Logging
• Remote high speed logging for firewall feature
• Syslog generation is offloaded to QFP and sent to Netflow collector
parameter-map type inspect-global
log flow-export template timeout-rate 1
log flow-export v9 udp destination 10.1.X.100 5500
• Configure packet logging at three places to capture all relevant data

parameter-map type inspect-global
Logs globally dropped packets log dropped-packets
parameter-map type inspect TRACK_CONNECTION

log dropped-packets
Logs packets dropped by security violation policy-map type inspect OUTSIDE->INSIDE_PMAP
class type inspect ALLOWED_APPLICATIONS
inspect TRACK_CONNECTION
policy-map type inspect OUTSIDE->INSIDE_PMAP

class type inspect BLOCKED_APPLICATIONS
Logs packets dropped by security policy drop log
class class-default
drop log
04/15 19:13:19.311 [cpp-dp-fw]: (info): QFP:0.0 Thread:000 TS:00000056710241740464 :FW_DEBUG_L4:[172.16.1.200] 44419 => [10.1.1.100] 80 6 (0): Session miss:- SI packet
04/15 19:13:19.311 [cpp-dp-fw]: (info): QFP:0.0 Thread:000 TS:00000056710241745289 :FW_DEBUG_L4:[172.16.1.200] 44419 => [10.1.1.100] 80 6 (0): Searching for imprecise channel 10.1.1.100:80 172.16.1.200:44419 proto TCP (0:0)
04/15 19:13:19.311 [cpp-dp-fw]: (info): QFP:0.0 Thread:000 TS:00000056710241755068 :FW_DEBUG_L4:[172.16.1.200] 44419 => [10.1.1.100] 80 6 (0): Could not find imprecise channel -ip_id:40654 seq #2171264973 window:14600
04/15 19:13:19.311 [cpp-dp-fw]: (info): QFP:0.0 Thread:000 TS:00000056710241760054 :FW_DEBUG_FLG_POLICY:[172.16.1.200] 44419 => [10.1.1.100] 80 6 (0): Check Box SYN flood limit. per_box_limit_n_cnt: ffffffff00000000 limit ffffffff
curr_cnt 0
04/15 19:13:19.311 [cpp-dp-fw]: (info): QFP:0.0 Thread:000 TS:00000056710241763425 :FW_DEBUG_FLG_POLICY:[172.16.1.200] 44419 => [10.1.1.100] 80 6 (0): Check SYN flood limit. limit_n_cnt: ffffffff00000000 limit ffffffff curr_cnt 0
04/15 19:13:19.311 [cpp-dp-fw]: (verbose): QFP:0.0 Thread:000 TS:00000056710241768271 :FW_DEBUG_L4:[172.16.1.200] 44419 => [10.1.1.100] 80 6 (0): FW: Classify 172.16.1.200:44419 => 10.1.1.100:80 6 zone pair OUTSIDE->INSIDE_ZP
04/15 19:13:19.311 [cpp-dp]: (verbose): QFP:0.0 Thread:000 TS:00000056710241773658 CCE IPV4 PKT (src:172.16.1.200,dst:10.1.1.100,sprt:ad83,dprt:0050,prot:06,tos:00,len:0014,ttl:3f) , intf:3ff
04/15 19:13:19.311 [cpp-dp]: (verbose): QFP:0.0 Thread:000 TS:00000056710241777311 CCE IPV4 UIDB_INFO W0:ea0be3e0, W1:20000000, tcam_region_index:e3e0, key_index:00, cmd:00000000
04/15 19:13:19.311 [cpp-dp]: (verbose): QFP:0.0 Thread:000 TS:00000056710241781148 cce_sw_tcam_top:: root:0xea0be3e0 top:0xea0be404 pType:0000000000
04/15 19:13:19.311 [cpp-dp]: (verbose): QFP:0.0 Thread:000 TS:00000056710241783505 cce_sw_tcam_dump_key:: 0xac1001c8 :0x06002fff :0x0050ad83 :0x0a010164 :0xe3e00001
04/15 19:13:19.311 [cpp-dp]: (verbose): QFP:0.0 Thread:000 TS:00000056710241791225 cce_sw_tcam_cuts:: cuts:0x00000004
04/15 19:13:19.311 [cpp-dp]: (verbose): QFP:0.0 Thread:000 TS:00000056710241792009 cce_sw_tcam_lookup:: leaf:0xea0be56e partN:0x00000000 partA:0xea0be40c
04/15 19:13:19.311 [cpp-dp]: (verbose): QFP:0.0 Thread:000 TS:00000056710241793189 cce_sw_tcam_cuts:: cuts:0x00000000
04/15 19:13:19.311 [cpp-dp]: (verbose): QFP:0.0 Thread:000 TS:00000056710241793616 cce_sw_tcam_lookup:: leaf:0xea0beacc partN:0x00000001 partA:0xea0be414
04/15 19:13:19.311 [cpp-dp]: (verbose): QFP:0.0 Thread:000 TS:00000056710241795935 cce_sw_dump_vm160::Mask0:0x00000000 :0xff000000 :0xffff0000 :0xffff0000 :0x00000009
04/15 19:13:19.311 [cpp-dp]: (verbose): QFP:0.0 Thread:000 TS:00000056710241797864 cce_sw_dump_vm160::Valu0:0x00000000 :0x06000000 :0x00500000 :0x0a010000 :0x00000001
04/15 19:13:19.311 [cpp-dp-fw]: (info): QFP:0.0 Thread:000 TS:00000056710241820789 :FW_DEBUG_EVENT:[172.16.1.200] 44419 => [10.1.1.100] 80 6 (0): Classify Actions: Action Inspect (8) Protocol http (12) Match_index 4
04/15 19:13:19.312 [cpp-dp-fw]: (info): QFP:0.0 Thread:000 TS:00000056710241824855 :FW_DEBUG_EVENT:[172.16.1.200] 44419 => [10.1.1.100] 80 6 (0): Classify Action: Rule[7361776.10422817.1] ACE(10)
04/15 19:13:19.312 [cpp-dp-fw]: (verbose): QFP:0.0 Thread:000 TS:00000056710241845969 :FW_DEBUG_DETAILED FW_DEBUG_L4 :[172.16.1.200] 44419 => [10.1.1.100] 80 6 (0): Created session for inspection, Session ID:0x00000005
04/15 19:13:19.312 [cpp-dp-fw]: (verbose): QFP:0.0 Thread:000 TS:00000056710241850299 :FW_DEBUG_L4:[172.16.1.200] 44419 => [10.1.1.100] 80 6 (0): Session ID:0x00000005, TCP: scb tcp state 1 dir 1 ip pkt hl 20, tl 60, fo 0, prot 6,
src 172.16.1.200, dst 10.1.1.100 id 9ece tcp sp 44419, dp 80, seq 2171264973, ack 0, do 10, flag 0x2, window 14600
04/15 19:13:19.312 [cpp-dp-fw]: (info): QFP:0.0 Thread:000 TS:00000056710241855928 :FW_DEBUG_L4:[172.16.1.200] 44419 => [10.1.1.100] 80 6 (0): Session ID:0x00000005, TCP Seq#: 2171264973, ack#: 0, Window: 14600.
04/15 19:13:19.312 [cpp-dp-fw]: (verbose): QFP:0.0 Thread:000 TS:00000056710241861268 :FW_DEBUG_L4:[172.16.1.200] 44419 => [10.1.1.100] 80 6 (0): dos mit pscb 0xe8e6c760 host 10.1.1.100 phostdb 0x0
04/15 19:13:19.312 [cpp-dp-fw]: (info): QFP:0.0 Thread:000 TS:00000056710241914895 :FW_DEBUG_L4:[172.16.1.200] 44419 => [10.1.1.100] 80 6 (0): Session ID:0x00000005, TCP: detect SYN. state was LISTEN -> SYNSENT seq# 2171264973 ack#
0 window 14600 zone-pair name OUTSIDE->INSIDE_ZP
04/15 19:13:19.312 [cpp-dp-fw]: (info): QFP:0.0 Thread:000 TS:00000056710241918324 :FW_DEBUG_L4:[172.16.1.200] 44419 => [10.1.1.100] 80 6 (0): Session ID:0x00000005, TCP: start SYNWAIT timer - 30000 msec
04/15 19:13:19.312 [cpp-dp-fw]: (verbose): QFP:0.0 Thread:000 TS:00000056710241923729 :FW_DEBUG_EVENT:[172.16.1.200] 44419 => [10.1.1.100] 80 6 (0): Session ID:0x00000005 Starting timer 3164+63 ageout 0 pscb 0xe8e6c760 10.1.1.100:80
172.16.1.200:44419 proto TCP (0:0) bucket 0 timer_info 0x1 cur 1 max 3 csec now 56710241923008 sysup 56697539 ms scb 0xe8e6c760
04/15 19:13:19.312 [cpp-dp-fw]: (verbose): QFP:0.0 Thread:000 TS:00000056710241935945 :FW_DEBUG_EVENT:[172.16.1.200] 44419 => [10.1.1.100] 80 6 (0): jitter 6 wheel 0xe91e5020 : 0x116dc fw_timer 0x0 0x1231b101
04/15 19:13:19.312 [cpp-dp-fw]: (info): QFP:0.0 Thread:000 TS:00000056710241944682 :FW_DEBUG_L4:[172.16.1.200] 44419 => [10.1.1.100] 80 6 (0): Created new session Session ID:0x00000005
04/15 19:13:19.312 [cpp-dp-fw]: (verbose): QFP:0.0 Thread:000 TS:00000056710241946362 :FW_DEBUG_L4:[172.16.1.200] 44419 => [10.1.1.100] 80 6 (0): rg 0 flgs 0x24405541 app 18
04/15 19:13:19.312 [cpp-dp-fw]: (verbose): QFP:0.0 Thread:000 TS:00000056710243893381 :FW_DEBUG_EVENT:[10.1.1.100] 80 => [172.16.1.200] 44419 6 (0): insp pkt: in GigabitEthernet2 out GigabitEthernet1 pkt_sb_flags 0x80000000
pkt_sb_flags2 0x0 izone 2 ozone 1 vrf is not enabled. nvi is not enabled.
04/15 19:13:19.312 [cpp-dp-fw]: (verbose): QFP:0.0 Thread:000 TS:00000056710243905158 :FW_DEBUG_EVENT:[10.1.1.100] 80 => [172.16.1.200] 44419 6 (0): fw_ntuple: key1-ipaddr1 10.1.1.100 key1-ipaddr2 172.16.1.200 proto 6 key-flags 0
key2-port1 80 key2-port2 44419 ip6addr1 - :: ip6addr2 - :: ip6port1 - 80 ip6port2 - 44419
04/15 19:13:19.312 [cpp-dp-fw]: (info): QFP:0.0 Thread:000 TS:00000056710243917554 :FW_DEBUG_L4:[10.1.1.100] 80 => [172.16.1.200] 44419 6 (0): Session ID:0x00000005 Sessiondb hit:-
src 10.1.1.100, dst 172.16.1.200 id 0 tcp sp 80, dp 44419, seq 2663126407, ack 2171264974, do 10, flag 0x12, window 14480
04/15 19:13:19.312 [cpp-dp-fw]: (info): QFP:0.0 Thread:000 TS:00000056710243938614 :FW_DEBUG_L4:[10.1.1.100] 80 => [172.16.1.200] 44419 6 (0): Session ID:0x00000005, TCP: state was SYNSEND -> SYNRCVD seq# 2663126407 ack# 2171264974
window 14480 zone-pair name OUTSIDE->INSIDE_ZP
04/15 19:13:19.314 [cpp-dp-fw]: (verbose): QFP:0.0 Thread:000 TS:00000056710245406482 :FW_DEBUG_EVENT:[172.16.1.200] 44419 => [10.1.1.100] 80 6 (0): insp pkt: in GigabitEthernet1 out GigabitEthernet2 pkt_sb_flags 0x80000000
pkt_sb_flags2 0x0 izone 1 ozone 2 vrf is not enabled. nvi is not enabled.
04/15 19:13:19.314 [cpp-dp-fw]: (verbose): QFP:0.0 Thread:000 TS:00000056710245446436 :FW_DEBUG_EVENT:[172.16.1.200] 44419 => [10.1.1.100] 80 6 (0): fw_ntuple: key1-ipaddr1 10.1.1.100 key1-ipaddr2 172.16.1.200 proto 6 key-flags 0
key2-port1 80 key2-port2 44419 ip6addr1 - :: ip6addr2 - :: ip6port1 - 80 ip6port2 – 44419
04/15 19:13:19.314 [cpp-dp-fw]: (info): QFP:0.0 Thread:000 TS:00000056710245458199 :FW_DEBUG_L4:[172.16.1.200] 44419 => [10.1.1.100] 80 6 (0): Session ID:0x00000005 Sessiondb hit:-
src 172.16.1.200, dst 10.1.1.100 id 9ecf tcp sp 44419, dp 80, seq 2171264974, ack 2663126408, do 8, flag 0x10, window 115
04/15 19:13:19.314 [cpp-dp-fw]: (info): QFP:0.0 Thread:000 TS:00000056710245500749 :FW_DEBUG_L4:[172.16.1.200] 44419 => [10.1.1.100] 80 6 (0): Session ID:0x00000005, TCP: state was SYNRCVD -> ESTAB seq# 2171264974 ack# 2663126408
window 115 zone-pair name OUTSIDE->INSIDE_ZP
04/15 19:13:19.314 [cpp-dp-fw]: (info): QFP:0.0 Thread:000 TS:00000056710245507922 :FW_DEBUG_L4:[172.16.1.200] 44419 => [10.1.1.100] 80 6 (0): Session ID:0x00000005, TCP: start TCP idle timer - 3600000 msec

Ltrarc 2003

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Ltrarc 2003

Uploaded by

Copyright:

Available Formats

IOS-XE Hands on

• Data plane and Control plane Separation

BRKCRS-3147 – Advanced Troubleshooting of ASR1K and ISR (IOS-XE) Made Easy

• Traditional IOS troubleshooting commands aren’t effective IOS

SIP SIP interconn.

SPA SPA SPA

SPA SPA SPA

show processes platform show processes

show processes memory platform show processes memory

show processes cpu platform show processes cpu

• IOS-XE runs IOS as a daemon on top of Platform OS

• Answer the fundamental question: Is the router healthy?

Time limit: 5 minutes

CSR# show platform resources

Memory used Thresholds

2.2Gb total available IOS

PID TTY Allocated Freed Holding Getbufs Retbufs Process

CSR# show processes memory platform sorted

CSR# show processes cpu sorted

CSR# show processes cpu platform sorted

• IOS resources are subset of platform resources

• Processes within IOS are invisible to the platform

• IOSd runs with ~3.8Gb of memory

• Captures are performed on the interface

• Basic captures do not require any configuration commands

• Step 2: Create capture filter

• ACL can be used for more granular bidirectional capture

• Step 4: Start capture

• Step 5: Test traffic

• Step 7: Stop capture

• Step 8: Look at packet headers in buffer

• Step 9: Look at packet data in buffer

• Need to change the capture configuration?

CSR# monitor capture CAPTURE ...

CSR# monitor capture CAPTURE start

Time limit: 10 minutes

Interface: GigabitEthernet1 Interface: GigabitEthernet2

Interface: GigabitEthernet1 Interface: GigabitEthernet2

• Step 2: Set up capture on external and internal interface

CSR# monitor capture CAPIN interface GigabitEthernet2 both

• Step 3: Look at current CSR# show monitor capture

Byte offset Raw HEX ASCII

• Your boss still isn’t satisfied!

• EPC only verify packets arrive/leave an interface

• Packet Tracing is a Swiss army knife

Input ACL NAT

Interface: GigabitEthernet1 Interface: GigabitEthernet2

• Step 2: Set up conditional match condition

Pick receive interface of packet

• Step 3: Enable packet tracing

• Step 5: Identify the number of packets to be traced

• Step 7: Validate packets were traced and look at summary of packets

• Step 8: Trace the packet through the processing path

CSR# show platform packet-trace [packet X] Packet number to be displayed

• Step 2: Set up conditional match condition

Interface receiving initial packet

Copy packets bidirectionally 1500 bytes is rounded up to 2048

Including L2 header to validate

• Step 5: Identify the number of packets to be traced

Tracing first 16 packets because

Conditional Debug Global State: Start

• Step 8: View traced packet

• Troubleshoot the issue using the steps you learned: