Professional Documents
Culture Documents
BRKSPG-3612
Agenda
Silver bullets
<QC> Quick Command
Troubleshooting Tools
Hardware
Software
New features
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 2
IOS-XR: Troubleshooting Tools
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
IOS-XR: Troubleshooting Tools
1. Logging
2. Trace (L-Trace and P-Trace)
3. Debug
4. Core dump / crash files
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
What is/has happening/ed?
<QC>CRS#show logging | i 5-
%ROUTING-BGP-5-ADJCHANGE
%ROUTING-OSPF-5-ADJCHG
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
Trace command
Always-on debug
Reduces the need to enable debugging and wait for the
event to re-occur
Need to be part of the “cisco-support” group
Logs are cyclical and overwrite older events.
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
L-Trace command
Lightweight Trace
OSPF Troubleshooting
Quick command
<QC>CRS#sh ospf trace all 10
Will list the last 10 outputs of all the OSPF debugs
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
P-Trace command
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
Trace command
BGP Troubleshooting
<QC>CRS#sh bgp trace reverse
–Will list the BGP debugs in reverse order
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
Debug commands
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
Core Dumps
<QC>CRS#show exception
–To see if writing a core dump failed
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
New Feature: Service Console – ASR9k
<QC>ASR9K(admin)#service console
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
IOS-XR: Hardware
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
IOS-XR Hardware 12000 / GSR
Multi-Service Edge
(also runs IOS 12.0S)
ASR 9000
Aggregation Services
CRS
Core & MSE
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
CRS Logical Architecture – Packet Flow
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
IOS-XR Hardware CRS
<QC>CRS(admin)#show platform
–This will show you what you have and is it running
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
Cisco CRS-1 Line Card and commands
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
Cisco CRS-1 Line Card and commands
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
Cisco CRS-1 Line Card and commands
MODULAR SERVICES CARD PLIM
MIDPLANE
Squid CPU Optics
(FABRIC)
GW
Show controllers plim asic statistics summary OC192
Framer
location R/S/M and
Optics
From TX PSE Egress
Fabric L3 Engine Queuing OC192
ASIC Framer
and
Optics
From Fabric Egress Packet Flow
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
PLIM Packet Drops
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
PLIM Packet Drops
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
PLIM Packet Drops
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
PLIM Packet Drops
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
Cisco CRS-1 Line Card and commands
MODULAR SERVICES CARD PLIM
MIDPLANE
Show
Squidcontroller pse tcam summary
CPU Optics
(FABRIC)
GW
OC192
Framer
Show controller pse util and
Optics
FromTX PSE Egress
Show controller
Fabric
ASIC
pse stats
L3 Engine Queuing OC192
Framer
and
Optics
From Fabric Egress Packet Flow
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
Cisco CRS-1 Line Card and commands
MODULAR SERVICES CARD PLIM
MIDPLANE
Squid Optics
Show
GW controllers ingressq statistics
CPU
(FABRIC)
OC192
Framer
Show controllers ingressq queue all and
Optics
From TX PSE Egress
Fabric
Show controllers ingressq block dps
ASIC
L3 Engine Queuing OC192
Framer
and
Optics
From Fabric Egress Packet Flow
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
Ingress Queue Packet Drops
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
Ingress Queue Packet Drops
In a normal scenario, none of the "Ingressq Drops" should be
incrementing.
In the above ouput "discard drops" will be incremented when
CRAB is oversubscribed , this is for only Unicast traffic.
"tail drops " These are the threshold drops in the destination
queues of the SEAL for Unicast traffic.
For multicast traffic check for "mcl tail drops " for low priority and
"mch tail drops" for high priority traffic.
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
Cisco CRS-1 Line Card and commands
MODULAR SERVICES CARD PLIM
MIDPLANE
Squid
Show
GW controllers fabricq statistics
CPU Optics
(FABRIC)
OC192
Framer
Show controllers fabricq queues and
Optics
From TX PSE Egress
Fabric L3 Engine Queuing OC192
ASIC Framer
and
Optics
From Fabric Egress Packet Flow
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
Fabric Queue Packet Drops
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
Cisco CRS-1 Line Card and commands
MODULAR SERVICES CARD PLIM
MIDPLANE
Squid CPU Optics
(FABRIC)
GW
OC192
Show controllers egressq statistics location R/S/M
Framer
and
Optics
From
8
TX PSE Egress
Fabric L3 Engine Queuing OC192
ASIC 6 7 Framer
and
5 Optics
From Fabric Egress Packet Flow
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
Egress PSE Drops
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
Egress Queue Drops
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
Egress PLIM Drops
Verify that ingress plim "To PSE : Total Good Packets" are almost
equal to egress plim "From EgressQ: Total Good Packets".
If BP count is incrementing "From PSE: Total BP Count", make
sure that ipogo is not oversubscribed.
BP packets are sent from ipogo to inform the Beluga to slowdown
the traffic.
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
Egress PLIM Drops
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
IOS-XR: Software
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
Router OS Evolution
SSH
Forwarding Plane Applications
Control Plane Data Plane Management Plane
SSH
Drivers
Management Plane
Routin
NetFlo
Interfa
Per.fM
SNMP
Alarm
Service
OSPF
LPTS
IGMP
BGB
SSH
HA Infrastructure
QoS
ACL
XML
ISIS
gmt
PIM
RIB
FIB
RIP
PFI
CLI
Host
L2
ce
Network Stack
w
g
Applications
SMU Package
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
Bundles
Security Manageability
MPLS Multicast
Optional MPLS, UCP PIM, MFIB, IGMP
IPSec, Encryption, ORB, XML,
Decryption Alarms management
Core Routing:
[Mandatory] RIB, BGP, ISIS, OSPF, RPL
Base
Admin
Interface manager,
Resource Management:
System database, checkpoint services
Rack, Fabric, LR management
Configuration management, etc.
OS:
Kernel, file system, memory management, and other slow changing core
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
IOS XR Modular Packaged Software
RP DRP LC
Manage- Manage- Multi-cast Opt’l
Security Security GMPLS
ability ability
Opt’l Opt’l
GMPLS Multi-cast GMPLS Multi-cast
Line Card
Forwarding
RPL BGP RPL BGP Mand
OSPF OSPF Base
ISIS ISIS
Forwarding Forwarding OS Mand
Mand Mand
Base Base
SC
Admin OS
Base
OS
Admin Mand
OS
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
PIE
PIE - Package Installation Envelopes. A delivery
mechanism for packages.
Non-bootable files.
Contain a single package or a set of packages, called a
composite package or bundle.
Being non-bootable, they are used to add software
package files to a running router.
PIE files have a pie extension.
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
SMU - Software Maintenance Upgrades
When a PIE file contains software for a specific bug fix, it is
called a SMU.
A composite SMU is a PIE file that contains SMUs for more
than one package.
SMU versions are synchronised to the package major,
minor, and maintenance versions they upgrade. Eg. of a
SMU Name: hfr-rout-3.6.1.CSCsr70080
Not an alternative to maintenance releases. All bugs fixed
by SMUs are integrated into the maintenance releases.
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
IOS-XR Software Naming
comp - The “comp” prefix indicates that the file is a
composite of multiple packages.
platform - Platform for which the software package is
designed. The platform designation is “hfr.”
composite_name - Identifies a specific composite package.
Only composite PIE file is named “mini” and includes
packages described Unicast Routing Core Bundle.
package_type – Type of package the file supports. Applies
only to single-package PIEs. Example – fwdg
[Forwarding package] , k9sec [Security package].
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
IOS-XR Software Naming (cont.)
major - Identifies the major release of this package.
–All packages on the router must be at the same major release level.
May require a router reboot
minor - Contains one new features and/ or bug fixes.
–Need not be identical for all packages on the router but has to be
compatible. A minor release may require a router reboot.
maintenance - Contains a collection of bug fixes for a
package.
–Need not be identical for all packages on the router but the major
and minor versions of the maintenance release must match. Does
not usually require a router reboot.
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
IOS-XR Software Naming (cont.)
Software Delivery
Filename Example
Type
comp-platform-
Composite (Bundle) PIE composite_name.pie- comp-hfr-mini.pie-3.7.0
major.minor.maintenance
platform-package_type.-p.pie- hfr-mgbl-p.pie-3.7.0
Single package PIE
major.minor.maintenance hfr-mpls-p.pie-3.2.0
comp-platform-
Composite SMU
composite_name.ddts.pie
comp-hfr-001.CSCec98xxx.pie
platform-package_type-
Single package SMU
major.minor.maintenance.ddts.pie
hfr-fwdg-3.7.0.CSCei4xxx.pie
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
IOS-XR Upgrade
Installing XR software from ROM Monitor mode required
“vm files”.
PIEs cannot be used to install XR software from ROM
Mon.
VM files are used for router recovery when the router
cannot boot the Cisco IOS XR software.
VM files contain the same packages as the Unicast Core
bundle pie. The format of a vm file is:
comp-platform-package_name.vm-major.minor.maintenance
Example - comp-hfr-mini.vm-3.9.0
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
IOS-XR Upgrade (cont.)
The tar files available on CCO contain both PIE files and
VM files. The format of a tar file is:
platform-bundle_name-major.minor.maintenance.tar
Example – CRS-1-iosxr-3.8.0.tar
bundle_name - 2 bundles available:
-- IOS XR - Unicast Routing Core Bundle, Management,
MPLS, and Multicast packages.
-- IOS XR-k9 – IOS XR bundle file plus the Security
package.
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
IOS-XR Upgrade Summary
Initial Installation is always from ROMMON
Sub-sequent upgrades can be done from CLI or ROMMON
In ROMMON, Use “.vm” file
From CLI, Use “.pie” file
SMU’s and security packages can only be added from CLI
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
IOS-XR Process
Copy the package file or files to a local storage device or
file server.
Add the package or packages on one or more SDRs using
the command install add.
Activate the package or packages on one or more SDRs
using the install activate command.
Commit the current set of packages using the install
commit command.
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
IOS-XR Upgrade Process
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
IOS-XR Rollback
Each time a package is activated or deactivated, a
rollback point is created.
This point defines the package set that was active at that
point.
A rollback point is also there for the last committed
package set.
<QC>#show install rollback to [id]
<QC>#show install request
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
IOS-XR Always Check
Check the recommended upgrading procedure on CCO:
http://www.cisco.com/web/Cisco_IOS_XR_Software/index.
html
Make sure you have the correct rommon version on all line
cards and RP’s
<QC>#(admin)#show hw-module fpd location all
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
Pre-emptive multi-tasking, multi-threaded
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
Multi-threading on CRS-1
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
Multi-threading on the CRS-1
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 55
Multi-tasking on CRS-1
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 56
Troubleshooting High CPU Utilisation
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
Virtual Memory, protected address space
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
Virtual Memory on CRS-1
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 59
Blocked Processes
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 61
Mutual Exclusion block
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 62
Send/Reply/Receive block
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
Send/Reply/Receive block
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
Troubleshooting
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 65
Troubleshooting Blocked Processes
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 66
“In routing, the control plane is the part of
architecture that is concerned with drawing the
network map, or more simply the information that
defines what to do with incoming packets.”
Definition: Control Plane
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 67
Control Plane Troubleshooting
In-band
Contention
ASIC
QoS
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 68
Control Plane interactions
Routing Routing
RIB Updates/
Process
Control Adjacencies
Plane
MPLS Label Binding
LIB Updates/
Process
Adjacencies
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 69
QNX
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 70
Troubleshooting BGP
RIB
BGP Table
BGP Best Path
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 71
The Usual Suspects
Configs;
Layer 1; IGP; Design;
CPU; MTU; Memory
Input Queues
Bug
……….
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 72
BGP Flowchart
www.cisco.com/en/US/tech/tk365/technologies_tech_note09
186a008009478a.shtml#main
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 73
Configs
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 75
Trace command
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 76
Debug commands
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 77
Process Debugging
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 78
Memory
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 79
CPU
<QC>CRS#top
303 processes; 1425 threads;
CPU states: 97.7% idle, 1.4% user, 0.7% kernel
Memory: 4096M total, 2123M avail, page size 4K
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 80
Blocked Process
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 81
Blocked Process
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 82
Collecting information
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 83
What Next
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 84
Q&A
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 85
Complete Your Online Session Evaluation
Complete your session evaluation:
Directly from your mobile device by visiting
www.ciscoliveaustralia.com/mobile and login
by entering your badge ID (located on the
front of your badge)
BRKSPG-3612 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 86