Professional Documents
Culture Documents
Cisco IOS XR Troubleshooting Guide For The Cisco ASR 9000 Aggregation Services Router
Cisco IOS XR Troubleshooting Guide For The Cisco ASR 9000 Aggregation Services Router
Americas Headquarters
Cisco Systems, Inc.
170 West Tasman Drive
San Jose, CA 95134-1706
USA
http://www.cisco.com
Tel: 408 526-4000
800 553-NETS (6387)
Fax: 408 527-0883
THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT
SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE
OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.
The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB’s public
domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.
NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS” WITH
ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT
LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF
DEALING, USAGE, OR TRADE PRACTICE.
IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING,
WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO
OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
Cisco and the Cisco Logo are trademarks of Cisco Systems, Inc. and/or its affiliates in the U.S. and other countries. A listing of Cisco's trademarks can be found at
www.cisco.com/go/trademarks. Third party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership
relationship between Cisco and any other company. (1005R)
Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display
output, network topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in
illustrative content is unintentional and coincidental.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
© 2010 Cisco Systems, Inc. All rights reserved.
CONTENTS
Preface xi
Validating and Troubleshooting Installation of the Cisco IOS XR Software Package 1-7
Verifying the Software Version 1-8
Validating the Installation 1-10
Validating and Troubleshooting Cisco IOS XR Software Configuration 1-16
Local and Global Configurations 1-16
Collecting Configuration Information 1-19
Verifying the Running Configuration 1-20
Using the show configuration failed Command 1-24
Verifying the System 1-26
Troubleshooting the Backplane Ethernet Control System 1-41
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 iii
Contents
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
iv OL-23591-02
Contents
Troubleshooting Routing and CEF Issues Related to Bundles and Load Balancing 5-115
Verifying Routing Table Entries for Parallel Links 5-115
Verifying the CEF Database and Measuring Flows 5-117
Troubleshooting Problems with Link Bundles 5-118
Bundle Does Not Come Up 5-118
Bundle Member Not Distributing 5-119
Bundle Not Using MAC-Address From Backplane 5-119
Layer 3 Data Traffic Not Flowing 5-120
Ping Failed over Bundle 5-120
Layer 3 Packets Not Synching Over Bundle 5-121
Layer 2 Traffic Not Flowing 5-121
Bundle Statistics 5-122
Troubleshooting Layer 2 Bundles and Load Balancing 5-122
Verifying the Bundle Status, IGP Route, and CEF Database 5-122
Viewing the Expected Paths and Measuring the Flows 5-123
Troubleshooting Layer 3 Bundles and Load Balancing 5-124
fib_mgr Does Not Come Up During LC Reload or After Multiple Process Restarts 6-134
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 v
Contents
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
vi OL-23591-02
Contents
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 vii
Contents
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
viii OL-23591-02
Contents
INDEX
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 ix
Contents
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
x OL-23591-02
Preface
This guide describes how to troubleshooting a router using the Cisco IOS XR software.
This preface contains the following sections:
• Changes to This Document, page xi
• Obtaining Documentation and Submitting a Service Request, page xii
continued ...
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 xi
Preface
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
xii OL-23591-02
CH A P T E R 1
General Troubleshooting Procedures
This chapter describes general troubleshooting techniques you can use to troubleshoot the
Cisco ASR 9000 Aggregation Series Router. It includes the following sections:
• Prerequisite Documentation for Troubleshooting, page 1-1
• Verifying and Troubleshooting CLI Access, page 1-2
• Validating and Troubleshooting Installation of the Cisco IOS XR Software Package, page 1-7
• Validating and Troubleshooting Cisco IOS XR Software Configuration, page 1-16
• Verifying the System, page 1-26
• Troubleshooting the Backplane Ethernet Control System, page 1-41
• Basic Cisco IOS XR Verification and Troubleshooting Commands, page 1-46
• Displaying ASIC Errors, page 1-54
• Using Trace Commands, page 1-56
• MIB Location, page 1-57
• Gathering Information Before You Call Cisco TAC, page 1-58
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 1-1
Chapter 1 General Troubleshooting Procedures
Verifying and Troubleshooting CLI Access
– Established baseline for your network, that is, the normal network behavior and performance at
different times of the day so that you can compare any problems with a baseline
– Name of the device that is the spanning tree root bridge for the system control plane Ethernet
network
• Captured output of all commands
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-2 OL-23591-02
Chapter 1 General Troubleshooting Procedures
Verifying and Troubleshooting CLI Access
Cisco-support Task ID
Many of the troubleshooting commands can be performed only by users who are assigned to a user group
that includes the cisco-support task ID. Users without the cisco-support task ID receive a “This
command is not authorized” response if they attempt to use those commands. The cisco-support
commands are normally reserved for use by Cisco Technical Support personnel, because there is some
risk that they may cause performance or other issues.
Caution These Cisco support commands are normally reserved for use by Cisco Technical Support personnel
only. There is some risk that they may cause performance or other issues that impact products without
proper usage, and we highly recommend that you contact Cisco Technical Support prior to using any of
these commands. See the “Obtaining Documentation and Submitting a Service Request” section on
page xii for information on contacting Cisco TAC.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 1-3
Chapter 1 General Troubleshooting Procedures
Verifying and Troubleshooting CLI Access
Step 8 Ensure that you have the transport input telnet command under the line for the target device. The target
device is the device that is connected to the terminal server.
Step 9 Use a PC/dumb terminal to connect directly to the console of the target router. The target router is the
device connected to the terminal server. This step helps you identify the presence of a port issue.
Step 10 If you are disconnected, check timeouts. You can remove or adjust timeouts.
Note If you encounter authentication failures, remember that the terminal server performs the first
authentication (if configured), while the device to which you try to connect performs the second
authentication (if configured). Verify whether AAA is configured correctly on both the terminal server
and the connecting device.
Step 11 Contact Cisco Technical Support. See the “Obtaining Documentation and Submitting a Service Request”
section on page xii for Cisco Technical Support contact information.
SUMMARY STEPS
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-4 OL-23591-02
Chapter 1 General Troubleshooting Procedures
Verifying and Troubleshooting CLI Access
DETAILED STEPS
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 1-5
Chapter 1 General Troubleshooting Procedures
Verifying and Troubleshooting CLI Access
Examples
The output from the show interfaces MgmtEth command displays the status of the management
Ethernet interface. In the following example, the management Ethernet interface is up, and there are 20
input errors and 8 output errors.
RP/0/RSP0/CPU0:router# show interface MgmtEth 0/RSP0/CPU0/0
Tue Sep 14 14:21:07.496 DST
MgmtEth0/RSP0/CPU0/0 is up, line protocol is up
Interface state transitions: 1
Hardware is Management Ethernet, address is 001b.53ff.4a62 (bia 001b.53ff.4a62)
Description: Connected to Lab LAN
Internet address is 172.29.52.137/24
MTU 1514 bytes, BW 100000 Kbit (Max: 100000 Kbit)
reliability 73/255, txload 0/255, rxload 0/255
Encapsulation ARPA,
Half-duplex, 100Mb/s, THD, link type is autonegotiation
output flow control is off, input flow control is off
loopback not set,
ARP type ARPA, ARP timeout 04:00:00
Last input 00:00:00, output 00:00:00
Last clearing of "show interface" counters never
5 minute input rate 2000 bits/sec, 3 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
373082 packets input, 51028824 bytes, 239105 total input drops
62028 drops for unrecognized upper-level protocol
Received 2601 broadcast packets, 194653 multicast packets
10 runts, 0 giants, 0 throttles, 0 parity
20 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
45232 packets output, 3042775 bytes, 0 total output drops
Output 24 broadcast packets, 0 multicast packets
8 output errors, 0 underruns, 0 applique, 0 resets
0 output buffer failures, 0 output buffers swapped out
1 carrier transitions
The output from the show arp MgmtEth 0/RSP0/CPU0/0 command displays the ARP table for the
management Ethernet interface. Use the output from this command to verify that there are dynamic ARP
addresses in the table and that ARP is functioning over the interface. The output shows that ARP is
functioning over the management Ethernet interface 0/RSP0/CPU0/0.
RP/0/RSP0/CPU0:router# show arp MgmtEth 0/RSP0/CPU0/0
Tue Sep 14 14:24:03.962 DST
-------------------------------------------------------------------------------
0/RSP0/CPU0
-------------------------------------------------------------------------------
Address Age Hardware Addr State Type Interface
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-6 OL-23591-02
Chapter 1 General Troubleshooting Procedures
Validating and Troubleshooting Installation of the Cisco IOS XR Software Package
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 1-7
Chapter 1 General Troubleshooting Procedures
Validating and Troubleshooting Installation of the Cisco IOS XR Software Package
SUMMARY STEPS
1. show version
2. show install
DETAILED STEPS
The following example shows that the Cisco IOS XR software and active packages are version 4.0.0.
RP/0/RSP0/CPU0:router# show version
2 Management Ethernet
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-8 OL-23591-02
Chapter 1 General Troubleshooting Procedures
Validating and Troubleshooting Installation of the Cisco IOS XR Software Package
12 DWDM controller(s)
12 TenGigE
40 GigabitEthernet
2 SONET/SDH
2 Packet over SONET/SDH
219k bytes of non-volatile configuration memory.
975M bytes of compact flash card.
33994M bytes of hard disk.
1605616k bytes of disk0: (Sector size 512 bytes).
1605616k bytes of disk1: (Sector size 512 bytes).
The following example shows that the Cisco IOS XR software and active packages are version 4.0.0. If
there is an expected package missing or an active package is not an expected package, install and activate
the missing package or upgrade the unexpected package to the appropriate package. See
Cisco ASR 9000 Aggregation Services Router Router Getting Started Guide for details on installing,
activating, and upgrading software packages.
RP/0/RSP0/CPU0:router# show install
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 1-9
Chapter 1 General Troubleshooting Procedures
Validating and Troubleshooting Installation of the Cisco IOS XR Software Package
disk0:asr9k-optic-4.0.0
disk0:asr9k-video-p-4.0.0
disk0:asr9k-mpls-p-4.0.0
disk0:asr9k-mcast-p-4.0.0
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-10 OL-23591-02
Chapter 1 General Troubleshooting Procedures
Validating and Troubleshooting Installation of the Cisco IOS XR Software Package
Note The install verify command can take up to two minutes per package to process.
Note The install verify command ignores secure domain router (SDR) boundaries and performs the operation
in global scope.
The following example shows the output of the install verify command. The output is used to verify the
consistency of a previously installed software set with the package file from which it originated.
RP/0/RSP0/CPU0:router(admin)# install verify
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 1-11
Chapter 1 General Troubleshooting Procedures
Validating and Troubleshooting Installation of the Cisco IOS XR Software Package
Info: Successful.
Info: /install/iosxr-routing-4.0.0: [SUCCESS] Verification
Info: Successful.
Info: /install/iosxr-infra-4.0.0: [SUCCESS] Verification
Info: Successful.
Info: /install/iosxr-fwding-4.0.0: [SUCCESS] Verification
Info: Successful.
Info: /install/iosxr-diags-4.0.0: [SUCCESS] Verification
Info: Successful.
Info: /install/asr9k-adv-video-supp-4.0.0: [SUCCESS] Verification
Info: Successful.
Info: /install/asr9k-diags-supp-4.0.0: [SUCCESS] Verification
Info: Successful.
Info: /install/asr9k-mcast-supp-4.0.0: [SUCCESS] Verification
Info: Successful.
Info: /install/asr9k-base-4.0.0: [SUCCESS] Verification Successful.
Info: 0/5/CPU0 [LC] [SDR: Owner]
.
.
.
Info: Verification Summary:
Info: 0/0/CPU0: SUCCESSFUL. No anomalies found.
Info: 0/6/CPU0: SUCCESSFUL. No anomalies found.
Info: 0/5/CPU0: SUCCESSFUL. No anomalies found.
Info: 0/7/CPU0: SUCCESSFUL. No anomalies found.
Info: 0/1/CPU0: SUCCESSFUL. No anomalies found.
Info: 0/4/CPU0: SUCCESSFUL. No anomalies found.
Info: 0/2/CPU0: SUCCESSFUL. No anomalies found.
Info: 0/RSP0/CPU0: SUCCESSFUL. No anomalies found.
Info: The system needs no repair.
Install operation 3 completed successfully at 08:19:48 DST Sat Sep 25 2010.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-12 OL-23591-02
Chapter 1 General Troubleshooting Procedures
Validating and Troubleshooting Installation of the Cisco IOS XR Software Package
Committed Packages:
disk0:asr9k-mini-p-3.9.1
disk0:asr9k-optic-3.9.1
disk0:asr9k-doc-p-3.9.1
disk0:asr9k-k9sec-p-3.9.1
disk0:asr9k-video-p-3.9.1
disk0:asr9k-mpls-p-3.9.1
disk0:asr9k-mgbl-p-3.9.1
disk0:asr9k-mcast-p-3.9.1
If the expected active software packages are not displayed, install the packages (if required) and activate
the packages. See Cisco ASR 9000 Aggregation Services Router Router Getting Started Guide for
information on installing and activating Cisco IOS XR software packages. The following example
output shows the active packages for all cards in a router. The output displays the disk on which each
package is located.
RP/0/RSP0/CPU0:router# show install active
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 1-13
Chapter 1 General Troubleshooting Procedures
Validating and Troubleshooting Installation of the Cisco IOS XR Software Package
The output shows the name of the disk on which the packages are located. In the above example, the
active packages for each node are on disk0, and for all nodes, the composite package asr9k-os-mbi-4.0.0
is active. Additional packages shown are optional packages that have been activated after the initial
loading of the Cisco ASR 9000 Aggregation Series Router Unicast Routing Core Bundle.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-14 OL-23591-02
Chapter 1 General Troubleshooting Procedures
Validating and Troubleshooting Installation of the Cisco IOS XR Software Package
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 1-15
Chapter 1 General Troubleshooting Procedures
Validating and Troubleshooting Cisco IOS XR Software Configuration
The output shows the name of the disk on which the packages are located. In the above example, the
committed packages for each node are on disk0, and for all nodes, the composite package
asr9k-os-mbi-4.0.0 is committed. Additional packages shown are optional packages that have been
committed after the initial loading of the Cisco ASR 9000 Aggregation Series Router Unicast Routing
Core Bundle.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-16 OL-23591-02
Chapter 1 General Troubleshooting Procedures
Validating and Troubleshooting Cisco IOS XR Software Configuration
interface Bundle-Ether16
description Connect to router-S Port-Ch 16
mtu 9216
bundle maximum-active links 1
!
interface Bundle-Ether16.160 l2transport
description Connect to router-S Port-Ch 16 Service Instance 160
encapsulation dot1q 160
!
.
.
.
!
interface Loopback0
ipv4 address 10.144.144.144 255.255.255.255
!
interface tunnel-ip10
!
interface tunnel-te44190
description Primary GE Tunnel from router-S to router-T
ipv4 unnumbered Loopback0
priority 0 0
autoroute announce
signalled-bandwidth 100000
destination 10.19.19.19
fast-reroute
record-route
path-option 1 explicit name Primary_GE_Path_to_router-T ospf 100 area 0
!
.
.
.
interface MgmtEth0/RSP0/CPU0/0
description Connected to LAN
ipv4 address 172.29.52.137 255.255.255.0
!
interface MgmtEth0/RSP0/CPU0/1
shutdown
!
interface GigabitEthernet0/0/0/0
shutdown
!
interface GigabitEthernet0/0/0/1
shutdown
!
interface GigabitEthernet0/0/0/2
shutdown
!
.
.
.
interface TenGigE0/7/0/1
shutdown
!
interface TenGigE0/7/0/2
shutdown
!
interface TenGigE0/7/0/3
shutdown
!
interface POS0/2/0/0
description Connected to PE_router-2 POS 0/2/0/0
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 1-17
Chapter 1 General Troubleshooting Procedures
Validating and Troubleshooting Cisco IOS XR Software Configuration
!
interface POS0/2/0/1
description Connected to PE_router-3 POS 0/2/0/1
!
controller SONET0/2/0/0
clock source internal
!
controller SONET0/2/0/1
clock source internal
!
Use the show sysdb trace commands to display the contents of the system database after a configuration
change. The trace information includes a history of any changes to the running configuration. You can
specify either a local node or the shared plane.
The following example output shows the contents of the local database, that is, for a specific location
(node):
RP/0/RSP0/CPU0:router# show sysdb trace verification location 0/5/cpu0 reverse
.
.
.
Timestamp nid reqid jid tid reg_hndl connid action
path
432 wrapping entries (8192 possible, 158 filtered, 590 total)
Sep 23 04:35:39.969 0/RSP0/CPU0 8168 354 1 94 4483 apply
reply '--'
Sep 23 04:35:39.960 0/RSP0/CPU0 8168 354 1 94 4505 Apply
called 'cfg/if/act/tunnel-ip10/v'
Sep 23 04:35:39.960 0/RSP0/CPU0 8168 354 1 94 4505 verify
reply: accept '--'
Sep 23 04:35:39.685 0/RSP0/CPU0 8168 354 1 94 4505 Verify
called 'cfg/if/act/tunnel-ip10/v'
Sep 23 04:35:39.678 0/RSP0/CPU0 0 354 1 94 4505
register 'cfg/if/act/tunnel-ip[0-9]*/mtu/tunnel-ip'
Sep 23 04:35:39.678 0/RSP0/CPU0 0 354 1 94 4505
register 'cfg/if/act/tunnel-ip[0-9]*/im/bw'
Sep 23 04:35:39.678 0/RSP0/CPU0 0 354 1 94 4505
register 'cfg/if/act/tunnel-ip[0-9]*/tunl_gre/keepalive'
Sep 23 04:35:39.678 0/RSP0/CPU0 0 354 1 94 4505
register 'cfg/if/act/tunnel-ip[0-9]*/tunl_gre/dfbit_disable'
Sep 23 04:35:39.678 0/RSP0/CPU0 0 354 1 94 4505
register 'cfg/if/act/tunnel-ip[0-9]*/tunl_gre/ttl'
Sep 23 04:35:39.678 0/RSP0/CPU0 0 354 1 94 4505
register 'cfg/if/act/tunnel-ip[0-9]*/tunl_gre/tos'
Sep 23 04:35:39.678 0/RSP0/CPU0 0 354 1 94 4505
register 'cfg/if/act/tunnel-ip[0-9]*/tunl_gre/mode'
.
.
.
The following example output shows the contents of the shared database, that is, the configuration data
that is shared with all LC and RP in the router:
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-18 OL-23591-02
Chapter 1 General Troubleshooting Procedures
Validating and Troubleshooting Cisco IOS XR Software Configuration
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 1-19
Chapter 1 General Troubleshooting Procedures
Validating and Troubleshooting Cisco IOS XR Software Configuration
• show configuration commit list [number-of-commits] [detail]—the command output displays a list
of the commit IDs (up to 100) available for rollback.
RP/0/RSP0/CPU0:router# show configuration commit list
• commit confirmed minutes (executed from config mode)—This command commits the
configuration on a trial basis for a minimum of 30 seconds and a maximum of 300 seconds (5
minutes). During the trial configuration period, enter commit to confirm the configuration. If
commit is not entered, then the system will revert to the previous configuration when the trial time
period expires.
SUMMARY STEPS
1. configure
2. show running-config
3. describe hostname hostname
4. end
5. show sysdb trace verification shared-plane | include path
6. show sysdb trace verification location node-id
7. show cfgmgr trace
8. show configuration history commit
9. show configuration commit changes {last | since | commit-id}
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-20 OL-23591-02
Chapter 1 General Troubleshooting Procedures
Validating and Troubleshooting Cisco IOS XR Software Configuration
DETAILED STEPS
Example:
RP/0/RSP0/CPU0:router# configure
Step 2 show running-config Displays the contents of the running configuration.
Verify that the running configuration is as expected.
Example:
RP/0/RSP0/CPU0:router(config)# show
running-config
Step 3 describe hostname hostname Determines the path.
Example:
RP/0/RSP0/CPU0:router(config)# describe
hostname router_A
Step 4 end Saves configuration changes.
• When you issue the end command, the system prompts
Example: you to commit changes:
RP/0/RSP0/CPU0:router(config)# end
Uncommitted changes found, commit them before
exiting(yes/no/cancel)?
[cancel]:
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 1-21
Chapter 1 General Troubleshooting Procedures
Validating and Troubleshooting Cisco IOS XR Software Configuration
Example:
RP/0/RSP0/CPU0:router# show cfgmgr trace
Step 8 show configuration history commit Displays a list of historical changes to the configuration.
Verify that the timeline of changes is as expected.
Example:
RP/0/RSP0/CPU0:router# show configuration
history commit
Step 9 show configuration commit changes {last | since Displays detailed committed configuration history
| commit-id} information.
Verify that the history information is as expected.
Example:
RP/0/RSP0/CPU0:router# show configuration
commit changes last 15
Step 10 show configuration failed startup Displays information on any configurations that failed
during startup.
Example:
RP/0/RSP0/CPU0:router# show configuration
failed startup
Step 11 cfs check Checks the current configuration to see if there are any
missing configurations.
Example:
RP/0/RSP0/CPU0:router# cfs check
Examples
The following example shows the output of the show running-config command:
RP/0/RSP0/CPU0:router# show running-config
Building configuration...
!! IOS XR Configuration 4.0.1.10I
!! Last configuration change at Thu Sep 23 04:35:38 2010 by user_A
!
hostname router
.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-22 OL-23591-02
Chapter 1 General Troubleshooting Procedures
Validating and Troubleshooting Cisco IOS XR Software Configuration
.
.
logging suppress duplicates
telnet vrf default ipv4 server max-servers 100
domain name cisco.com
domain lookup disable
taskgroup default
!
.
.
.
interface preconfigure GigabitEthernet0/3/0/7
shutdown
!
interface preconfigure GigabitEthernet0/3/0/8
shutdown
!
interface preconfigure GigabitEthernet0/3/0/9
shutdown
!
interface preconfigure GigabitEthernet0/3/0/10
shutdown
!
.
.
.
Package:
iosxr-infra
iosxr-infra V4.0.0 IOS-XR Infra Package Definition
Vendor : Cisco Systems
Desc : IOS-XR Infra Package Definition
Build : Built on Wed Sep 8 16:07:48 DST 2010
Source : By sc-g-01 in /auto/ioxbuild8/production/4.0.0/asr9k/workspace for pie
Card(s): RP, NP24-4x10GE, NP24-40x1GE, NP40-40x1GE, NP40-4x10GE, NP40-8x10GE,
NP40-2_20_COMBO, NP80-8x10GE, NP80-16x10GE, A9K-SIP-700, A9K-SIP-500
Restart information:
Default:
parallel impacted processes restart
Size Compressed/Uncompressed: 38MB/85MB (44%)
Component:
shellutil V[ci-401/7] Common shell utility applications
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 1-23
Chapter 1 General Troubleshooting Procedures
Validating and Troubleshooting Cisco IOS XR Software Configuration
Note You may browse startup failed configurations for up to the previous four router reloads.
Use the show configuration failed startup command and the load configuration failed startup
command to browse and reapply any failed configuration. The load configuration failed startup
command can be used in configuration mode to load the failed startup configuration into the target
configuration session, then the configuration can be modified and committed. See
Cisco ASR 9000 Aggregation Services Router Router Getting Started Guide for information on
committing a configuration.
RP/0/RSP0/CPU0:router# show configuration failed startup
RP/0/RSP0/CPU0:router# config
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-24 OL-23591-02
Chapter 1 General Troubleshooting Procedures
Validating and Troubleshooting Cisco IOS XR Software Configuration
The failed configuration is loaded into the target configuration, minus the errors that caused the startup
configuration to fail.
RP/0/RSP0/CPU0:router(config-bgp)# commit
Use the show configuration failed command to display failed items in the last configuration commit,
including reasons for the error.
In any mode, the configuration failures from the most recent commit operation are displayed.
The show configuration failed command can be used in EXEC mode and configuration mode. The
command is used in EXEC mode when the configuration does not load during startup. The command is
used in configuration mode to display information when a commit fails.
The following example shows the show configuration failed command.
RP/0/RSP0/CPU0:router(config)# interface pos 0/6/0/4
RP/0/RSP0/CPU0:router(config-if)# no vrf
RP/0/RSP0/CPU0:router(config-if)# commit
% Failed to commit one or more configuration items during an atomic operation, no changes
have been made. Please use 'show configuration failed' to view the errors
RP/0/RSP0/CPU0:router(config-if)# exit
RP/0/RSP0/CPU0:router(config)# show configuration failed
Note The show configuration failed command in configuration mode only exists as long as the configuration
session is active. Once you exit configuration mode, the command cannot be used to display the failed
configuration.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 1-25
Chapter 1 General Troubleshooting Procedures
Verifying the System
If a configuration commit fails, do not exit configuration mode (return to EXEC mode) as you will not
be able to view the failed configuration.
RP/0/RSP0/CPU0:router# configure
RP/0/RSP0/CPU0:router(config)# taskgroup bgp
RP/0/RSP0/CPU0:router(config-tg)# end
Uncommitted changes found, commit them before exiting(yes/no/cancel)? [cancel]:y
RP/0/RSP0/CPU0:router(config)# exit
Uncommitted changes found, commit them before exiting(yes/no/cancel)? [cancel]:n
RP/0/RSP0/CPU0:router# show configuration failed
RP/0/RSP0/CPU0:router#
SUMMARY STEPS
1. admin
2. show platform [node-id]
3. show version
4. show running-config
5. show logging
6. show environment
7. show context
8. exit
9. show context
10. show memory summary detail location all
11. show memory heap summary {job-id | all}
12. top processes
13. show running-config
14. show system verify start
show system verify report
15. show {ipv4 | ipv6} interface brief
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-26 OL-23591-02
Chapter 1 General Troubleshooting Procedures
Verifying the System
DETAILED STEPS
Example:
RP/0/RSP0/CPU0:router# admin
Step 2 show platform [node-id] Displays information about the status of cards and modules
installed in the router.
Example: • Some cards support a CPU module and service
RP/0/RSP0/CPU0:router(admin)# show platform processor (SP) module. Other cards support only a
single module.
• A card module is also called a node. When all nodes are
working properly, the status of each node displayed in
the State column is IOS-XR RUN.
• If you run the command without a node-id (show
platform as shown in the example), the output will
include all nodes in the system.
• Type the show platform node-id command to display
information for a specific node. Replace node-id with a
node name from the show platform command Node
column.
Step 3 show version Displays information about the router, including image
names, uptime, and other system information.
Example: Verify that the expected software version and images are
RP/0/RSP0/CPU0:router(admin)# show version installed.
Step 4 show running-config Displays all of the nondefault commands currently running,
including hardware module power status, secure domain
router (SDR) configuration, and fabric configuration. The
Example:
RP/0/RSP0/CPU0:router(admin)# show
output also displays the users defined in administration
running-config mode with root-system access.
Verify that the serial numbers for the nodes in the current
running configuration are what you expected. The expected
rack numbers and serial numbers should be listed in the
current system documentation. See the “Prerequisite
Documentation for Troubleshooting” section on page 1-1.
Also verify that the hardware module power status is as
expected and the SDR and fabric configurations are as
expected.
Step 5 show logging Displays all syslog messages stored in the buffer. The
command output displays the device operation history from
a system perspective.
Example:
RP/0/RSP0/CPU0:router(admin)# show logging Analyze the logged events and their order of happening.
Check for anything out of the ordinary such as errors,
tracebacks, or crashes. Also check for any Severity 1 or
Severity 2 errors.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 1-27
Chapter 1 General Troubleshooting Procedures
Verifying the System
Example:
RP/0/RSP0/CPU0:router(admin)# exit
Step 9 show context Displays core dump context information on CPUs
responsible for routing and Cisco Express Forwarding
(CEF). See the “show context Command” section on
Example:
RP/0/RSP0/CPU0:router# show context
page 1-50 for more information on the show context
command output.
Step 10 show memory summary detail location all Displays information about the memory available on the
router after the system image decompresses and loads.
Example: Verify that the expected memory is available or installed.
RP/0/RSP0/CPU0:router# show memory summary Ensure that all memory regions have adequate free space
detail location all available.
Step 11 show memory heap summary {job-id | all} Displays a summary of the information about the heap
space. The output displays each process and the amount of
memory allocated for each process.
Example:
RP/0/RSP0/CPU0:router# show memory heap Note The job-id is the output of the show processes
summary all command.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-28 OL-23591-02
Chapter 1 General Troubleshooting Procedures
Verifying the System
Examples
The output from the show platform command indicates that all expected nodes are in the run state. If
all nodes in the system are active, the cards should be in the IOS XR RUN and the SPAs should be in the
OK state. The example output shows that all expected nodes are in the run state.
RP/0/RSP0/CPU0:router(admin)# show platform
The output from the show version command indicates the version of software being run on the nodes
and from which location (disk or network). Check that the expected software version and images are
installed. The example output shows that the Cisco IOS XR software version is 4.0.0 and that the
installed pie versions are also 4.0.0.
RP/0/RSP0/CPU0:router(admin)# show version
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 1-29
Chapter 1 General Troubleshooting Procedures
Verifying the System
2 Management Ethernet
12 DWDM controller(s)
12 TenGigE
40 GigabitEthernet
2 SONET/SDH
2 Packet over SONET/SDH
219k bytes of non-volatile configuration memory.
975M bytes of compact flash card.
33994M bytes of hard disk.
1605616k bytes of disk0: (Sector size 512 bytes).
1605616k bytes of disk1: (Sector size 512 bytes).
The output from the show running-config command displays the current running configuration, that is,
all of the nondefault commands currently active. Verify that the contents of the current running
configuration are as expected.
Tip The output of this command in exec mode is different from the output in admin mode. You should run
the command from each of these modes to locate all of the configuration information.
Building configuration...
!! Last configuration change at 18:56:31 UTC Tue Feb 28 2006 by user_A
!
hostname router
clock timezone PST 8.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-30 OL-23591-02
Chapter 1 General Troubleshooting Procedures
Verifying the System
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 1-31
Chapter 1 General Troubleshooting Procedures
Verifying the System
!
interface preconfigure GigabitEthernet0/3/0/19
shutdown
!
interface preconfigure TenGigE0/3/0/0
shutdown
!
interface preconfigure TenGigE0/3/0/1
shutdown
!
router static
address-family ipv4 unicast
0.0.0.0/0 172.29.52.1 200
!
!
router isis 100
is-type level-2-only
net 49.0001.0000.0000.0044.00
nsf cisco
address-family ipv4 unicast
metric-style wide
mpls traffic-eng level-2-only
mpls traffic-eng router-id Loopback0
!
interface Loopback0
passive
address-family ipv4 unicast
!
!
interface TenGigE0/4/0/0
bfd minimum-interval 50
bfd multiplier 3
bfd fast-detect ipv4
address-family ipv4 unicast
!
!
!
router ospf 100
nsr
router-id 10.144.144.144
bfd minimum-interval 50
bfd multiplier 3
mpls ldp sync
nsf cisco
area 0
mpls ldp sync-igp-shortcuts
mpls traffic-eng
interface Loopback0
passive enable
!
.
.
.
http server
ssh server vrf default
igmp snooping profile default
system-ip-address 10.144.144.144
minimum-version 2
internal-querier
tcn query solicit
ttl-check disable
router-alert-check disable
!
igmp snooping profile mrouter
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-32 OL-23591-02
Chapter 1 General Troubleshooting Procedures
Verifying the System
router-guard
mrouter
!
router pim
address-family ipv4
mofrr mofrr-acl
rp-address 10.11.11.11
rp-address 10.144.144.144 bidir-acl bidir
spt-threshold infinity
!
!
end
RP/0/RSP0/CPU0:router# admin
Wed Oct 27 14:52:07.000 DST
The output from the show logging command displays the contents of the logging buffer. The output
displays details on syslog historical events. Analyze the logged events and the order in which they
happened. Check for anything out of the ordinary such as errors, tracebacks, or crashes. Also check for
any Severity 1 or Severity 2 errors.
RP/0/RSP0/CPU0:router(admin)# show logging
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 1-33
Chapter 1 General Troubleshooting Procedures
Verifying the System
--More--
The output from the show environment command displays environmental monitor parameters for the
system. Verify that the environment parameters are as expected. Environment parameter anomalies are
logged in the syslog, so if an environment parameter displayed in the show environment command
output is not as expected, check the syslog using the show logging command. The syslog provides details
on any logged problems.
RP/0/RSP0/CPU0:router(admin)# show environment
Temperature Information
---------------------------------------------
0/1/*
host Inlet0 36.4
host Hotspot0 46.7
0/2/*
spa0 InletTemp 35.5
spa0 Hotspot 35.5
0/3/*
host Inlet0 31.1
host Hotspot0 32.5
0/RSP0/*
host Inlet0 31.3
host Hotspot0 42.0
0/4/*
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-34 OL-23591-02
Chapter 1 General Troubleshooting Procedures
Verifying the System
0/6/*
host Inlet0 38.0
host Hotspot0 48.9
0/FT0/*
host Inlet0 31.9
host Hotspot0 33.5
0/FT1/*
host Inlet0 32.3
host Hotspot0 35.3
Voltage Information
---------------------------------------------
0/1/*
host IBV 10592 n/a
host 5.0V 4925 n/a
host VP3P3_CAN 3289 n/a
host 3.3V 3302 n/a
host 2.5V 2516 n/a
host 1.8VB 1812 n/a
host 1.2VB 1193 n/a
host 1.8VA 1806 n/a
host 0.9VB 886 n/a
host 1.2V_LDO_BRG0 1193 n/a
host 1.2V_LDO_BRG1 1195 n/a
host 1.8VC 1811 n/a
host 1.5VB 1505 n/a
host 1.5VA 1503 n/a
host 1.1V(1.05V_CPU) 1053 n/a
host 0.75VA 752 n/a
host 0.75VB_0.75VC 754 n/a
host 1.1VB 1103 n/a
host 1.2V_TCAM0 1003 n/a
host 1.2V_TCAM1 1000 n/a
host 1.0V_Bridge_LDO 999 n/a
host 1.0VB 1042 n/a
host 0.75VD_and_0.75VE 752 n/a
host 1.2V_TCAM2 1006 n/a
host 1.2V_TCAM3 1002 n/a
host 1.5VC 1504 n/a
host 1.8VD 1804 n/a
host 1.1VC 1100 n/a
host ZARLINK_3.3V 3272 n/a
host ZARLINK_1.8V 1807 n/a
host 1.2V_DB 1195 n/a
host 3.3V_DB 3318 n/a
host 2.5V_DB 2535 n/a
host 1.5V_DB 1509 n/a
.
.
.
LED Information
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 1-35
Chapter 1 General Troubleshooting Procedures
Verifying the System
---------------------------------------------
Fan Information
---------------------------------------------
0/FT0/*
3510 3480 3510 3570 3540 3510 3510 3480 3540 3540
3480 3480
0/FT1/*
3510 3510 3510 3540 3480 3510 3390 3510 3510 3540
3450 3480
Power Supply Information
---------------------------------------------
0/PM1/*
host PM 3000 Ok
0/PM2/*
host PM 3000 Ok
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-36 OL-23591-02
Chapter 1 General Troubleshooting Procedures
Verifying the System
0/FT1/SP 600
The output from the show context command displays core dump context information. See the “show
context Command” section on page 1-50 for more information on the show context command output.
RP/0/RSP0/CPU0:router# show context
node: node0_1_CPU0
------------------------------------------------------------------
Stack Trace
#0 0xfc1d3fa0
#1 0xfc1c6340
#2 0xfc1c5364
#3 0xfc1c542c
#4 0x48210930
#5 0x482110b8
#6 0x48212ba4
#7 0x48203dd8
#8 0x4820c61c
#9 0xfc1557ec
#10 0xfc15573c
#11 0xfc152fb8
#12 0x4820d140
Registers info
r0 r1 r2 r3
R0 00000000 481ff7b0 4824a55c 00000000
r4 r5 r6 r7
R4 0000f054 00000001 00000006 00000000
r8 r9 r10 r11
R8 00000000 fc220000 481fffc0 00000000
r12 r13 r14 r15
R12 4823be90 4824a4a0 48230000 00000000
r16 r17 r18 r19
R16 00000048 00000001 00000019 48256520
r20 r21 r22 r23
R20 00000000 00000000 00000003 00000045
r24 r25 r26 r27
R24 00000003 00000000 00000003 4825dc34
r28 r29 r30 r31
R28 00000006 0000f054 48254064 481ff810
cnt lr msr pc
R32 00000000 fc1c6340 0000d932 fc1d3fa0
cnd xer
R36 28004024 00000008
DLL Info
DLL path Text addr. Text size Data addr. Data size Version
/hfr-os-3.3.90/lib/libinfra.dll 0xfc142000 0x00034200 0xfc1343b8 0x00000bbc
0
/lib/libc.dll 0xfc1a8000 0x00079dd8 0xfc222000 0x00002000 0
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 1-37
Chapter 1 General Troubleshooting Procedures
Verifying the System
node: node0_6_CPU0
------------------------------------------------------------------
node: node0_RSP0_CPU0
------------------------------------------------------------------
node: node0_RSP1_CPU0
------------------------------------------------------------------
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-38 OL-23591-02
Chapter 1 General Troubleshooting Procedures
Verifying the System
The show system verify start command starts the system verification process and the show system
verify report generates the output from the system verification process. The output allows you to verify
that the system parameters are as expected.
RP/0/RSP0/CPU0:router# show system verify start
- Verifying LC Status
- Verified LC Status : [OK]
- Verifying QNET Status
Unable to get current LC status info
- Verified QNET Status : [FAIL]
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 1-39
Chapter 1 General Troubleshooting Procedures
Verifying the System
The show interface brief command displays the usability status of the configured interfaces. Verify that
all expected interfaces are listed. For an interface to be usable, both the interface hardware (Status) and
line protocol must be up. The protocol is Up if the interface can provide two-way communication. The
example output displays IP addresses, status, and protocol status for each interface. The output shows
that all assigned interfaces (interfaces that are configured with IP addresses) have an interface hardware
status and line protocol status of Up.
RP/0/RSP0/CPU0:router# show ipv4 interface brief
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-40 OL-23591-02
Chapter 1 General Troubleshooting Procedures
Troubleshooting the Backplane Ethernet Control System
GigabitEthernet0/1/0/8 10.146.4.44 Up Up
GigabitEthernet0/1/0/9 unassigned Shutdown Down
GigabitEthernet0/1/0/10 unassigned Shutdown Down
GigabitEthernet0/1/0/11 unassigned Shutdown Down
GigabitEthernet0/1/0/12 10.194.16.44 Up Up
GigabitEthernet0/1/0/13 unassigned Shutdown Down
GigabitEthernet0/1/0/14 unassigned Shutdown Down
GigabitEthernet0/1/0/15 unassigned Shutdown Down
GigabitEthernet0/1/0/16 unassigned Up Up
GigabitEthernet0/1/0/17 unassigned Up Up
GigabitEthernet0/1/0/18 10.194.4.44 Up Up
GigabitEthernet0/1/0/19 unassigned Up Up
GigabitEthernet0/1/0/19.2127 unassigned Up Up
GigabitEthernet0/1/0/19.2130 unassigned Up Up
GigabitEthernet0/1/0/20 unassigned Up Up
GigabitEthernet0/1/0/20.2125 unassigned Up Up
GigabitEthernet0/1/0/21 unassigned Shutdown Down
GigabitEthernet0/1/0/22 unassigned Shutdown Down
GigabitEthernet0/1/0/23 10.114.4.44 Up Up
GigabitEthernet0/1/0/24 unassigned Shutdown Down
GigabitEthernet0/1/0/25 unassigned Shutdown Down
GigabitEthernet0/1/0/26 unassigned Shutdown Down
GigabitEthernet0/1/0/27 10.145.4.44 Up Up
GigabitEthernet0/1/0/28 unassigned Shutdown Down
GigabitEthernet0/1/0/29 unassigned Shutdown Down
GigabitEthernet0/1/0/30 unassigned Up Up
GigabitEthernet0/1/0/30.215 unassigned Up Up
GigabitEthernet0/1/0/31 unassigned Up Up
GigabitEthernet0/1/0/32 unassigned Shutdown Down
GigabitEthernet0/1/0/33 unassigned Shutdown Down
GigabitEthernet0/1/0/34 unassigned Shutdown Down
GigabitEthernet0/1/0/35 unassigned Shutdown Down
GigabitEthernet0/1/0/36 unassigned Shutdown Down
GigabitEthernet0/1/0/37 unassigned Shutdown Down
GigabitEthernet0/1/0/38 unassigned Shutdown Down
GigabitEthernet0/1/0/39 unassigned Shutdown Down
POS0/2/0/0 unassigned Up Up
POS0/2/0/1 unassigned Up Up
TenGigE0/4/0/0 10.114.8.44 Up Up
TenGigE0/4/0/1 unassigned Shutdown Down
TenGigE0/4/0/2 unassigned Shutdown Down
TenGigE0/4/0/3 unassigned Shutdown Down
TenGigE0/4/0/4 unassigned Shutdown Down
TenGigE0/4/0/5 unassigned Shutdown Down
TenGigE0/4/0/6 unassigned Shutdown Down
TenGigE0/4/0/7 unassigned Shutdown Down
TenGigE0/6/0/0 unassigned Shutdown Down
TenGigE0/6/0/1 unassigned Shutdown Down
TenGigE0/6/0/2 unassigned Shutdown Down
TenGigE0/6/0/3 unassigned Shutdown Down
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 1-41
Chapter 1 General Troubleshooting Procedures
Troubleshooting the Backplane Ethernet Control System
All devices in a system using Cisco IOS XR software connect to the system control plane Ethernet
network, also called the Ethernet over backplane channel (EOBC). The control plane is provided using
Gigabit Ethernet (GE) links between nodes. The GE links are internal to the chassis and cannot be
removed.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-42 OL-23591-02
Chapter 1 General Troubleshooting Procedures
Troubleshooting the Backplane Ethernet Control System
Figure 1-1 shows the control plane Ethernet network (the dotted line in the drawing).
Figure 1-1 Cisco ASR 9000 Aggregation Services Router Control Ethernet Topology
Backplane
40x1GE Fabric GE PHY 8x10GE 2:1 Fabric GE PHY 4x10GE Fabric GE PHY
Line Card Interface Oversubscribed Interface Line Card Interface
Chip Line Card Chip Chip
CPU CPU CPU
NPU NPU NPU NPU NPU NPU NPU NPU NPU NPU NPU NPU
10 10 10 10 10 10 10 10 10 10 10 10
GE GE GE GE GE GE GE GE GE GE GE GE
X X X X X X X X X X X X
10 x 10 x 10 x 10 x F F F F F F F F F F F F
SFP SFP SFP SFP P P P P P P P P P P P P
Backplane
NPU NPU NPU NPU NPU NPU NPU NPU GE NPU NPU NPU NPU GE
SW SW
10 10 10 10 10 10 10 10 To 10x 10x 10 10 To
GE GE GE GE GE GE GE GE NPUs GE GE NPUs
S S
X X X X X X X X X X
To F F To
F F F F F F F F F F
247272
FPGAs P P FPGAs
P P P P P P P P P P
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 1-43
Chapter 1 General Troubleshooting Procedures
Troubleshooting the Backplane Ethernet Control System
To verify and troubleshoot booting of the system control plane Ethernet network, perform the following
procedure.
SUMMARY STEPS
1. show platform
2. show controllers backplane ethernet clients all location node-id
3. show controllers backplane ethernet clients 18 statistics location node-id
4. Contact Cisco Technical Support if the problem is not resolved.
DETAILED STEPS
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-44 OL-23591-02
Chapter 1 General Troubleshooting Procedures
Troubleshooting the Backplane Ethernet Control System
Examples
The output from the show platform command indicates that all expected nodes are in the run state. If
all nodes in the system are active, the cards should be in the IOS XR RUN and the SPAs should be in the
OK state. The example output shows that all expected nodes are in the run state.
RP/0/RSP0/CPU0:router# show platform
The following example shows the current state of each Ethernet server client.
RP/0/RSP0/CPU0:router# show controllers backplane ethernet clients all location
0/RSP0/CPU0
The following example shows that there are 18 nodes in the run state, which means that 12 boot requests
have been received by eth_server and 12 replies have been sent:
RP/0/RSP0/CPU0:router# show controllers backplane ethernet clients 18 statistics location
0/RSP1/CPU0
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 1-45
Chapter 1 General Troubleshooting Procedures
Basic Cisco IOS XR Verification and Troubleshooting Commands
man Command
The man command provides online help for standard Cisco IOS XR CLI commands using manual (man)
pages. The command is used to display the manual pages for a specific command based on the command
name, a feature, or a keyword. Each man page contains the command name, syntax, command mode,
usage, examples, and related commands.
Note To run the man command, you must have the Cisco IOS XR Documentation Package,
“asr9k-doc.pie-4.0.0, .man pages for Cisco IOS XR software on the Cisco ASR 9000 Series Router
chassis,” loaded. If you are running a release later than 4.0.0, the package installation envelope (PIE)
name might be different. For the appropriate PIE name and an explanation of PIE installation, see the
“Upgrading Cisco IOS XR Software” section of the Release Notes document for the IOS XR version you
are running
The following example shows the output from the man command show users command.
RP/0/RSP0/CPU0:router# man command show users
Tue Sep 14 14:39:16.409 DST
Building index table...
Total Number of Command Entries:2726
[OK]
DESCRIPTION
To display information about the active lines on the router, use the show users
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-46 OL-23591-02
Chapter 1 General Troubleshooting Procedures
Basic Cisco IOS XR Verification and Troubleshooting Commands
show users
SYNTAX DESCRIPTION
COMMAND DEFAULT
COMMAND MODES
EXEC
COMMAND HISTORY
Release
Modification
Release 3.7.2
USAGE GUIDELINES
To use this command, you must be in a user group associated with a task group
that includes the proper task IDs. If you suspect user group assignment is
preventing you from using a command, contact your AAA administrator for
assistance.
Use the show users command to display the line number, connection name, idle
time, hosts, and terminal location. An asterisk (*) indicates the current
terminal session.
===================================================================================
Note:
To display all user groups and task IDs associated with the currently logged-in
user, use the show user command in EXEC mode. See the Authentication,
Authorization, and Accounting Commands on Cisco IOS XR Software module in
Cisco^B^`ASR^B^`9000 Series Aggregation Services Router System Security Command
Reference.
===================================================================================
TASK ID
Task ID
Operations
tty-access
read
EXAMPLES
The following example shows sample output identifying an active vty terminal
session:
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 1-47
Chapter 1 General Troubleshooting Procedures
Basic Cisco IOS XR Verification and Troubleshooting Commands
* * * * * * * * * * * * * * START OF LISTING * * * * * * * * * * * * * *
* * * * * * * * * * * * * * END OF LISTING * * * * * * * * * * * * * * *
Field
Description
Line
User
Service
Conns
Idle
Location
RELATED COMMANDS
Command
Description
show user
Displays all user groups and task IDs associated with the currently
logged-in user.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-48 OL-23591-02
Chapter 1 General Troubleshooting Procedures
Basic Cisco IOS XR Verification and Troubleshooting Commands
describe Command
The describe command provides a preview of a command without actually implementing it. This
command lists information about the package, component, and task ID for a specific command. You
must be in the appropriate configuration mode for the specific command. For example, to display the
package, component, and task ID information for the router bgp 1 command, you must be in global
configuration mode.
The following example shows the output from the describe router bgp 1 command.
RP/0/RSP0/CPU0:router(config)#describe router bgp 1
Package:
iosxr-routing
iosxr-routing V4.0.0[DT_IMAGE] IOS-XR Routing Package Definition
Vendor : Cisco Systems
Desc : IOS-XR Routing Package Definition
Build : Built on Wed Sep 8 16:10:14 DST 2010
Source : By router-021 in /files/4.0.0.DT_IMAGE/asr9k/workspace fo8
Card(s): RP, NP24-4x10GE, NP24-40x1GE, NP40-40x1GE, NP40-4x10GE, NP40-8x10GE,
NP40-2_20_COMBO, NP80-8x10GE, NP80-16x10GE, A9K-SIP-700, A9K-SIP-500
Restart information:
Default:
parallel impacted processes restart
Size Compressed/Uncompressed: 8556KB/22MB (37%)
Component:
ipv4-bgp V[ci-401/18] IPv4 Border Gateway Protocol (BGP)
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 1-49
Chapter 1 General Troubleshooting Procedures
Basic Cisco IOS XR Verification and Troubleshooting Commands
top Command
The top command is used to monitor CPU usage on the system through interactive process statistics.
The following example show the output from the top command.
RP/0/RSP0/CPU0:router# top
Computing times...
224 processes; 803 threads;
CPU states: 93.6% idle, 3.6% user, 2.7% kernel
Memory: 4096M total, 3504M avail, page size 4K
node: node0_1_CPU0
------------------------------------------------------------------
node: node0_6_CPU0
------------------------------------------------------------------
node: node0_RSP0_CPU0
------------------------------------------------------------------
node: node0_RSP1_CPU0
------------------------------------------------------------------
The following example shows the output from the show context command where there is a crashed
process.
RP/0/RSP1/CPU0:router# show context
node: node0_1_CPU0
------------------------------------------------------------------
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-50 OL-23591-02
Chapter 1 General Troubleshooting Procedures
Basic Cisco IOS XR Verification and Troubleshooting Commands
Stack Trace
#0 0xfc1d3fa0
#1 0xfc1c6340
#2 0xfc1c5364
#3 0xfc1c542c
#4 0x48210930
#5 0x482110b8
#6 0x48212ba4
#7 0x48203dd8
#8 0x4820c61c
#9 0xfc1557ec
#10 0xfc15573c
#11 0xfc152fb8
#12 0x4820d140
Registers info
r0 r1 r2 r3
R0 00000000 481ff7b0 4824a55c 00000000
r4 r5 r6 r7
R4 0000f054 00000001 00000006 00000000
r8 r9 r10 r11
R8 00000000 fc220000 481fffc0 00000000
r12 r13 r14 r15
R12 4823be90 4824a4a0 48230000 00000000
r16 r17 r18 r19
R16 00000048 00000001 00000019 48256520
r20 r21 r22 r23
R20 00000000 00000000 00000003 00000045
r24 r25 r26 r27
R24 00000003 00000000 00000003 4825dc34
r28 r29 r30 r31
R28 00000006 0000f054 48254064 481ff810
cnt lr msr pc
R32 00000000 fc1c6340 0000d932 fc1d3fa0
cnd xer
R36 28004024 00000008
DLL Info
DLL path Text addr. Text size Data addr. Data size Version
/hfr-os-3.3.90/lib/libinfra.dll 0xfc142000 0x00034200 0xfc1343b8 0x00000bbc
0
/lib/libc.dll 0xfc1a8000 0x00079dd8 0xfc222000 0x00002000 0
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 1-51
Chapter 1 General Troubleshooting Procedures
Basic Cisco IOS XR Verification and Troubleshooting Commands
node: node0_6_CPU0
------------------------------------------------------------------
node: node0_RSP0_CPU0
------------------------------------------------------------------
node: node0_RSP1_CPU0
------------------------------------------------------------------
Use the show context command to locate the core dump file path. For example, the core dump file path
shown in the command output is: harddisk:/dumper/first.tcam_mgr.abort.node0_1_CPU0.ppc.Z. The
command output shows a crash on a node. The process is pkg/bin/tcam_mgr.
Collect the following information and send it to Cisco Technical Support. For Cisco Technical Support
contact information, see the “Obtaining Documentation and Submitting a Service Request” section on
page xii.
• ppc.Z file—This file contains the binary core dump information. Use the path listed in the command
output to copy the contents of the ppc.Z file. The path shown in the command output is:
harddisk:/dumper/first.tcam_mgr.abort.node0_1_CPU0.ppc.Z
• ppc.txt file—This file contains content on the core dump similar to the show context command
output. Use the path listed in the command output to copy the contents of the ppc.txt file. The path
shown in the command output is: harddisk:/dumper/first.tcam_mgr.abort.node0_1_CPU0.ppc.txt
• Collect the show version or show install active command output.
Examples
RP/0/RSP0/CPU0:router# show history
RP/0/RSP0/CPU0:router(admin)# show history
RP/0/RSP0/CPU0:router(config)# show history
The following example shows the output from the show history command in EXEC mode:
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-52 OL-23591-02
Chapter 1 General Troubleshooting Procedures
Basic Cisco IOS XR Verification and Troubleshooting Commands
Examples
RP/0/RSP0/CPU0:router# show configuration
RP/0/RSP0/CPU0:router# show configuration running-config
RP/0/RSP0/CPU0:router(admin)# show configuration running
RP/0/RSP0/CPU0:router(config)# show configuration
RP/0/RSP0/CPU0:router(config)# show configuration running
RP/0/RSP0/CPU0:router(config)# show configuration merge
In this example, the show configuration command displays uncommitted changes made during a
configuration session:
RP/0/RSP0/CPU0:router# configure
RP/0/RSP0/CPU0:router(config)# interface tengige0/3/0/3
RP/0/RSP0/CPU0:router(config-if)# description faq
RP/0/RSP0/CPU0:router(config-if)# ipv4 address 10.10.11.20 255.0.0.0
RP/0/RSP0/CPU0:router(config-if)# show configuration
Building configuration...
interface TenGigE0/3/0/3
description faq
ipv4 address 10.10.11.20 255.0.0.0
end
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 1-53
Chapter 1 General Troubleshooting Procedures
Displaying ASIC Errors
The following example shows sample output from the show configuration command with the optional
merge keyword. The command is entered during a configuration session. The output displays the result
of merging the target and running configuration, without committing the changes.
RP/0/RSP0/CPU0:router# configure
RP/0/RSP0/CPU0:router(config)# interface tengige0/3/0/3
RP/0/RSP0/CPU0:router(config-if)# description faq
RP/0/RSP0/CPU0:router(config-if)# ipv4 address 10.10.11.20 255.0.0.0
RP/0/RSP0/CPU0:router(config-if)# show configuration merge
Building configuration...
hostname router
interface TenGigE0/0/0/0
ipv4 address 10.2.3.4 255.0.0.0
exit
interface TenGigE0/3/0/3
description faq
ipv4 address 10.1.1.1 255.0.0.0
shutdown
end
************************************************************
* Mace ASIC Error Summary *
************************************************************
Instance : 0
Number of nodes : 0
SBE error count : 0
MBE error count : 0
Parity error count : 0
CRC error count : 0
Generic error count : 0
Reset error count : 0
--------------------
************************************************************
* Prm_np ASIC Error Summary *
************************************************************
Instance : 0
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-54 OL-23591-02
Chapter 1 General Troubleshooting Procedures
Displaying ASIC Errors
Number of nodes : 3
SBE error count : 0
MBE error count : 0
Parity error count : 0
CRC error count : 0
Generic error count : 0
Reset error count : 0
--------------------
Instance : 1
Number of nodes : 3
SBE error count : 0
MBE error count : 0
Parity error count : 0
CRC error count : 0
Generic error count : 0
Reset error count : 0
--------------------
Instance : 2
Number of nodes : 3
SBE error count : 0
MBE error count : 0
Parity error count : 0
CRC error count : 0
Generic error count : 0
Reset error count : 0
--------------------
Instance : 3
Number of nodes : 3
SBE error count : 0
MBE error count : 0
Parity error count : 0
CRC error count : 0
Generic error count : 0
Reset error count : 0
--------------------
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 1-55
Chapter 1 General Troubleshooting Procedures
Using Trace Commands
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-56 OL-23591-02
Chapter 1 General Troubleshooting Procedures
MIB Location
The following example shows the last 20 events in the address resolution protocol (ARP) table.
RP/0/RSP0/CPU0:router# show arp trace tailf last 20
MIB Location
To locate and download MIBs, use the Cisco MIB Locator found at the following URL and choose a
platform under the Cisco Access Products menu:
http://cisco.com/public/sw-center/netmgmt/cmtk/mibs.shtml.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 1-57
Chapter 1 General Troubleshooting Procedures
Gathering Information Before You Call Cisco TAC
Caution We strongy recommend that, if possible, you gather the information described in this section before you
reset any cards. If you reset cards before you gather information, the system erases the information and
it will be more difficult to diagnose and repair the problem.
Timesaver Before contacting Cisco Technical Support, review the information provided at the following URL:
http://www.cisco.com/web/services/ts/access/index.html.
For information on contacting Cisco Technical Support, see the “Obtaining Documentation and
Submitting a Service Request” section on page xii.
Capturing Logs
See the “Prerequisite Documentation for Troubleshooting” section on page 1-1 in Chapter 1, “General
Troubleshooting Procedures,” for information on collecting current system information.
Collect system information using the following commands:
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-58 OL-23591-02
Chapter 1 General Troubleshooting Procedures
Gathering Information Before You Call Cisco TAC
• show tech-support—Displays system information for Cisco Technical Support and includes a
traditional dump of the configuration and show command outputs.
Note Some tech-support commands require the user to be assigned the cisco-support task ID. For a mapping
of commands to task IDs and allowed operations, see Cisco IOS XR Task ID Reference Guide.
Note On the Cisco ASR 9000 Aggregation Services Router, only online diagnostics are supported.
To run a specified on-demand diagnostic test or series of tests, use the diagnostic start location
command.
Example:
RP/0/RSP0/CPU0:router(admin)# diagnostic start location 0/RSP1/CPU0 test 1
For details on the diagnostic commands, see Cisco ASR 9000 Series Aggregation Services Router
Interface and Hardware Component Command Reference.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 1-59
Chapter 1 General Troubleshooting Procedures
Gathering Information Before You Call Cisco TAC
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-60 OL-23591-02
CH A P T E R 2
Verifying and Troubleshooting Interface Status
This chapter describes how to verify that interfaces are up (operational), and how to troubleshoot
problems on interfaces. It includes the following sections:
• Verifying and Troubleshooting Gigabit Ethernet Interfaces, page 2-61
• Verifying and Troubleshooting Pluggable Optical Line Card Interfaces, page 2-68
SUMMARY STEPS
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 2-61
Chapter 2 Verifying and Troubleshooting Interface Status
Verifying and Troubleshooting Gigabit Ethernet Interfaces
DETAILED STEPS
Example:
RP/0/RSP0/CPU0:router# show hw-module subslot
0/0/0 counters
or
RP/0/RSP0/CPU0:router# show hw-module subslot
counters framer
Step 5 If your system has pluggable optical line cards, Check for interface drops on the pluggable optical line
perform the additional troubleshooting steps cards.
applicable to these cards. See “Verifying and
Troubleshooting Pluggable Optical Line Card
Interfaces” section on page 2-68.
Step 6 Contact Cisco Technical Support. For Cisco Technical Support contact information, see the
“Obtaining Documentation and Submitting a Service
Request” section on page xii in the Preface.
The following example shows POS 0/0/1/0 with no input drop counters.
RP/0/RSP0/CPU0:router# show interfaces pos 0/0/1/0
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
2-62 OL-23591-02
Chapter 2 Verifying and Troubleshooting Interface Status
Verifying and Troubleshooting Gigabit Ethernet Interfaces
Encapsulation HDLC, crc 32, controller loopback not set, keepalive set (10 sec)
Last clearing of "show interface" counters never
5 minute input rate 3000 bits/sec, 0 packets/sec
5 minute output rate 3000 bits/sec, 0 packets/sec
199794 packets input, 222359750 bytes, 0 total input drops
0 drops for unrecognized upper-level protocol
0 runts, 0 giants, 0 throttles, 0 parity
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
89911 packets output, 213413210 bytes, 0 total output drops
0 output errors, 0 underruns, 0 applique, 0 resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions
Ingress:
Input total bytes = 0
Input good bytes = 0
Egress:
Output total bytes = 0
Output good bytes = 0
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 2-63
Chapter 2 Verifying and Troubleshooting Interface Status
Verifying and Troubleshooting Gigabit Ethernet Interfaces
Chains
--------------------
Base decap chain:
hdlc <14> <0xfd6a0a74, 0x00000000> < 0, 0>
Protocol chains:
---------------
<Protocol number> (name) Stats
Type Chain_node <caps num> <function, context> <drop pkts, drop bytes>
<9> (chdlc) Stats IN: 48466 pkts, 3559516 bytes; OUT: 41378 pkts, 910312 bytes
Encap:
l2_adj_rewrite <86> <0xfceada88, 0x482c390c> < 0, 0>
queue_fifo <56> <0xfcedea68, 0x482ddc30> < 0, 0>
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
2-64 OL-23591-02
Chapter 2 Verifying and Troubleshooting Interface Status
Verifying and Troubleshooting Gigabit Ethernet Interfaces
The following example shows counters implemented for Multiprotocol Label Switching (MPLS)
packets. The following output under Protocol Chains in the show netio idb command shows the MPLS
packets incrementing:
mpls <25> <0xfcc7b2b8, 0x00000000> < 152, 17328>
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 2-65
Chapter 2 Verifying and Troubleshooting Interface Status
Verifying and Troubleshooting Gigabit Ethernet Interfaces
Chains
--------------------
Base decap chain:
ether <30> <0xfd018cd8, 0x482c736c> < 0, 0>
Protocol chains:
---------------
<Protocol number> (name) Stats
Type Chain_node <caps num> <function, context> <drop pkts, drop bytes>
<7> (arp) Stats IN: 0 pkts, 0 bytes; OUT: 0 pkts, 0 bytes
Encap:
l2_adj_rewrite <86> <0xfcaa997c, 0x4831a33c> < 0, 0>
pcn_output <54> <0xfd054bfc, 0x48319f04> < 0, 0>
q_fq <43> <0xfd05f4b8, 0x48320fec> < 0, 0>
txm_nopull <60> <0xfcadba38, 0x4824c0fc> < 0, 0>
Decap:
pcn_input <55> <0xfd054bfc, 0x4830ba8c> < 0, 0>
q_fq_input <96> <0xfd05f330, 0x48312c7c> < 0, 0>
arp <24> <0xfcbfc2cc, 0x00000000> < 0, 0>
Fixup:
l2_adj_rewrite <86> <0xfcaa945c, 0x00000000> < 0, 0>
pcn_output <54> <0xfd054bfc, 0x48319f04> < 0, 0>
q_fq <43> <0xfd05f4b8, 0x48320fec> < 0, 0>
txm_nopull <60> <0xfcadba38, 0x4824c0fc> < 0, 0>
<10> (clns) Stats IN: 0 pkts, 0 bytes; OUT: 1861623 pkts, 2062483853 bytes
Encap:
clns <15> <0xfcbe2c80, 0x00000000> < 0, 0>
ether <30> <0xfd0189b4, 0x482c736c> < 0, 0>
l2_adj_rewrite <86> <0xfcaa997c, 0x482d8660> < 0, 0>
pcn_output <54> <0xfd054bfc, 0x48319f04> < 0, 0>
q_fq <43> <0xfd05f4b8, 0x48320fec> < 0, 0>
txm_nopull <60> <0xfcadba38, 0x4824c0fc> < 0, 0>
Decap:
pcn_input <55> <0xfd054bfc, 0x4830ba8c> < 0, 0>
q_fq_input <96> <0xfd05f330, 0x48312c7c> < 0, 0>
clns <15> <0xfcbe2444, 0x00000000> < 0, 0>
Fixup:
l2_adj_rewrite <86> <0xfcaa945c, 0x00000000> < 0, 0>
pcn_output <54> <0xfd054bfc, 0x48319f04> < 0, 0>
q_fq <43> <0xfd05f4b8, 0x48320fec> < 0, 0>
txm_nopull <60> <0xfcadba38, 0x4824c0fc> < 0, 0>
<12> (ipv4) Stats IN: 0 pkts, 0 bytes; OUT: 759095 pkts, 57691220 bytes
Encap:
ipv4 <26> <0xfcc03dfc, 0x482e0414> < 0, 0>
ether <30> <0xfd0189b4, 0x482c736c> < 0, 0>
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
2-66 OL-23591-02
Chapter 2 Verifying and Troubleshooting Interface Status
Verifying and Troubleshooting Gigabit Ethernet Interfaces
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 2-67
Chapter 2 Verifying and Troubleshooting Interface Status
Verifying and Troubleshooting Pluggable Optical Line Card Interfaces
The following Packet-over-SONET/SDH (POS) port example shows the SPA counters. The output
displays any drop counters or error counters incrementing for the interface.
RP/0/RSP0/CPU0:router# show hw-module subslot counters framer
SUMMARY STEPS
1. show controllers plim asic SPAQFPBridgeCtrl counters instance <0 – 3> all location node-id
2. show hw-module subslot brief pluggable-optics
3. show hw-module subslot address status pluggable-optics
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
2-68 OL-23591-02
Chapter 2 Verifying and Troubleshooting Interface Status
Verifying and Troubleshooting Pluggable Optical Line Card Interfaces
DETAILED STEPS
Example:
RP/0/RSP0/CPU0:router# show controllers plim
asic SPAQFPBridgeCtrl counters instance 0 all
location 0/4/CPU0
Step 1 show hw-module subslot brief pluggable-optics Displays a brief summary of the pluggable optics status for
all line card nodes, including optics type, vendor, and state.
Check that the state is enabled for the node that you are
Example: troubleshooting.
RP/0/RSP0/CPU0:router# show hw-module subslot
brief pluggable-optics
Step 2 show hw-module subslot address status Displays the status of the pluggable optics for the specified
pluggable-optics line card node, including faults and environmental data.
Check that the state and the transceiver are enabled. Check
for any warnings or alarms.
Example:
RP/0/RSP0/CPU0:router# show hw-module subslot
0/2/0 status pluggable-optics
Step 3 show hw-module subslot address errors Displays any errors that are present on the node. Note if
pluggable-optics there are any errors.
Verify that Phased Initialization displays Phase Reached: 4.
Example: Verify that Socket Verification displays “passed” for both
RP/0/RSP0/CPU0:router# show hw-module subslot Compatibility and Security.
0/2/0 errors pluggable-optics
Step 4 show hw-module subslot address registers Displays all available information on the optics including
pluggable-optics the IDPROM contents.
Example:
RP/0/RSP0/CPU0:router# show hw-module subslot
0/2/0 registers pluggable-optics
Step 5 Contact Cisco Technical Support. For Cisco Technical Support contact information, see the
“Obtaining Documentation and Submitting a Service
Request” section on page xii in the Preface.
Examples
The following example displays the drop counters and error countersincrementing on the SPA in the line
card.
RP/0/RSP0/CPU0:router# show controllers plim asic SPAQFPBridgeCtrl counters instance 0 all
location 0/4/CPU0
Wed Nov 3 08:58:07.062 EDT
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 2-69
Chapter 2 Verifying and Troubleshooting Interface Status
Verifying and Troubleshooting Pluggable Optical Line Card Interfaces
SPAQFPBridgeCtrl 0:
=======================================
POS0/2/0/0:
ID: SFP
Extended ID: 4
Xcvr Type: OC48 SR/STM16 I-16 (44)
Connector: LC
Vendor name: CISCO-FINISAR
Vendor part number: FTRJ1321P1BTL-C4
State: Enabled
POS0/2/0/1:
ID: SFP
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
2-70 OL-23591-02
Chapter 2 Verifying and Troubleshooting Interface Status
Verifying and Troubleshooting Pluggable Optical Line Card Interfaces
Extended ID: 4
Xcvr Type: OC48 SR/STM16 I-16 (44)
Connector: LC
Vendor name: CISCO-FINISAR
Vendor part number: FTRJ1321P1BTL-C4
State: Enabled
POS0/2/0/0:
State: Enabled
Environmental Information - raw values
Temperature: 48.136 C
Supply voltage: 32970 in units of 100uVolt
Tx bias: 12678 in units of 2uAmp
Tx power: -4 dBm (2848 in units of 0.1 uW)
Rx power: -7 dBm (1554 in units of 0.1 uW)
Transceiver: Enabled
SW TX Fault: None
SW LOS: Active
No active alarms
No active warnings
Version Identifier (VID): V01
Product Identifier (PID): SFP-OC48-SR
Part Number (PN): 10-1961-01
CLEI: WM1T2TTAAA
POS0/2/0/1:
State: Enabled
Environmental Information - raw values
Temperature: 46.172 C
Supply voltage: 32914 in units of 100uVolt
Tx bias: 11166 in units of 2uAmp
Tx power: -5 dBm (2282 in units of 0.1 uW)
Rx power: -5 dBm (2411 in units of 0.1 uW)
Transceiver: Enabled
SW TX Fault: unavailable
SW LOS: unavailable
No active alarms
No active warnings
Version Identifier (VID): V01
Product Identifier (PID): SFP-OC48-SR
Part Number (PN): 10-1961-01
CLEI: WM1T2TTAAA
POS0/2/0/0:
Phased Initialization
Phase Reached: 4
Phase Exit Code: Success 0
Phase Read Offset: 256
Socket Verification
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 2-71
Chapter 2 Verifying and Troubleshooting Interface Status
Verifying and Troubleshooting Pluggable Optical Line Card Interfaces
POS0/2/0/1:
Phased Initialization
Phase Reached: 4
Phase Exit Code: Success 0
Phase Read Offset: 256
Socket Verification
Compatibility: Compatibility passed
Security: Security passed
POS0/2/0/0:
ID: SFP
Extended ID: 4
Xcvr Type: OC48 SR/STM16 I-16 (44)
Connector: LC
Encoding: reserved
Bit Rate: 2500 Mbps
Single mode fiber supported length: 2 km
Single mode fiber supported length: 20 m
Upper bit rate limit: not specified
Lower bit rate limit: not specified
Date code (yy/mm/dd): 05/12/13
Vendor name: CISCO
Vendor OUI: 36965
Vendor Part Number (PN): FTRJ1321P1BTL-C4
Vendor Rev: B
Vendor SN (SN): FNS0951J0VN
Options implemented:
LOS Signal
TX Fault Signal
TX Disable Signal
Enhanced options implemented:
Alarm/Warning Flags
Diagnostic monitoring implemented:
Exernally Calibrated
Digital Diagnostic Monitoring
Idprom contents (hex):
0x00: 03 04 07 00 01 00 00 12 00 01 05 05 19 00 02 14
0x10: 00 00 00 00 43 49 53 43 4F 2D 46 49 4E 49 53 41
0x20: 52 20 20 20 00 00 90 65 46 54 52 4A 31 33 32 31
0x30: 50 31 42 54 4C 2D 43 34 42 20 20 20 05 1E 00 23
0x40: 00 1A 00 00 46 4E 53 30 39 35 31 4A 30 56 4E 20
0x50: 20 20 20 20 30 35 31 32 31 33 20 20 58 80 01 D3
0x60: 00 00 02 4C D2 86 D7 04 F8 8D 92 6D 3C 8B D2 2D
0x70: 60 F3 BD 00 00 00 00 00 00 00 00 00 62 56 5D CD
0x80: FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
0x90: FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
Status/Control Register: 10f8
Alarm Status: 0000
Warning Status: 0000
THRESHOLDS
high alarm high warning low warning low alarm
Temperature C +110.000 +093.000 -30.000 -40.000
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
2-72 OL-23591-02
Chapter 2 Verifying and Troubleshooting Interface Status
Verifying and Troubleshooting Pluggable Optical Line Card Interfaces
POS0/2/0/1:
ID: SFP
Extended ID: 4
Xcvr Type: OC48 SR/STM16 I-16 (44)
Connector: LC
Encoding: reserved
Bit Rate: 2500 Mbps
Single mode fiber supported length: 2 km
Single mode fiber supported length: 20 m
Upper bit rate limit: not specified
Lower bit rate limit: not specified
Date code (yy/mm/dd): 06/04/25
Vendor name: CISCO
Vendor OUI: 36965
Vendor Part Number (PN): FTRJ1321P1BTL-C4
Vendor Rev: B
Vendor SN (SN): FNS1017R0BV
Options implemented:
LOS Signal
TX Fault Signal
TX Disable Signal
Enhanced options implemented:
Alarm/Warning Flags
Diagnostic monitoring implemented:
Exernally Calibrated
Digital Diagnostic Monitoring
Idprom contents (hex):
0x00: 03 04 07 00 01 00 00 12 00 01 05 05 19 00 02 14
0x10: 00 00 00 00 43 49 53 43 4F 2D 46 49 4E 49 53 41
0x20: 52 20 20 20 00 00 90 65 46 54 52 4A 31 33 32 31
0x30: 50 31 42 54 4C 2D 43 34 42 20 20 20 05 1E 00 23
0x40: 00 1A 00 00 46 4E 53 31 30 31 37 52 30 42 56 20
0x50: 20 20 20 20 30 36 30 34 32 35 20 20 58 80 01 CE
0x60: 00 00 02 91 44 1B BF 32 33 64 55 09 CB 9B FD A3
0x70: 39 11 A4 00 00 00 00 00 00 00 00 00 88 17 A0 7B
0x80: FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
0x90: FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
Status/Control Register: 10f8
Alarm Status: 0000
Warning Status: 0000
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 2-73
Chapter 2 Verifying and Troubleshooting Interface Status
Verifying and Troubleshooting Pluggable Optical Line Card Interfaces
THRESHOLDS
high alarm high warning low warning low alarm
Temperature C +110.000 +093.000 -30.000 -40.000
Voltage V 003.9000 003.7000 002.9000 002.7000
Bias Current mA 080.0000 070.0000 004.0000 002.0000
Transmit power mW 003.5276 002.2272 000.2022 000.1291
Receive power mW 005.0147 003.3353 000.0407 000.0155
Diagnostics contents (hex):
0x00: 6E 00 D8 00 5D 00 E2 00 98 58 69 78 90 88 71 48
0x10: 9C 40 03 E8 88 B8 07 D0 89 CC 05 0B 57 00 07 E6
0x20: C3 E3 00 9B 82 49 01 97 00 00 00 00 00 00 00 00
0x30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x40: 00 00 00 00 3E 73 C5 30 40 46 23 7C 01 00 00 00
0x50: 00 5B FF F2 01 00 00 00 01 00 00 00 10 00 00 7C
0x60: 21 D8 80 B8 00 54 00 22 2A 02 00 00 00 00 10 F8
0x70: 00 00 00 80 00 00 00 00 00 00 00 00 00 00 00 01
0x80: 57 4D 31 54 32 54 54 41 41 41 31 30 2D 31 39 36
0x90: 31 2D 30 31 56 30 31 20 8A FB 55 00 00 00 00 64
0xA0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0xB0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 AA AA
0xC0: 53 46 50 2D 4F 43 34 38 2D 53 52 20 20 20 20 20
0xD0: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 66
0xE0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0xF0: 00 00 00 00 00 00 00 00 00 40 00 40 00 00 00 00
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
2-74 OL-23591-02
CH A P T E R 3
Troubleshooting Interface Connectivity
This chapter explains how to troubleshoot problems with connectivity between interfaces on the
ASR 9000 and interfaces on remote devices. It includes the following topics:
• Troubleshooting Ping and ARP Connectivity, page 3-75
• Troubleshooting Bidirectional Forwarding Detection, page 3-81
• Troubleshooting Ethernet CFM, page 3-85
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 3-75
Chapter 3 Troubleshooting Interface Connectivity
Troubleshooting Ping and ARP Connectivity
Yes Is this
No
an Ethernet
interface?
Is the No
IGP protocol
running?
Yes
Is the No Try
IGP protocol
ping
up?
Yes
Was an Yes
ARP entry
created?
No
Debug
ARP
Try
ping
Yes
279971
Exit
Follow these steps to troubleshoot ping and ARP connectivity issues. See Figure 3-1 to help you locate
the steps that apply to your network scenario.
Step 1 Ping the remote end and check for a response. If there is no response, continue with this procedure to
determine why the ping was unsuccessful and to connect successfully to the remote end.
Tip Use a systematic process to isolate the location of the failure. Ping the local interface first. If that is
successful, ping the directly connected neighbor (single hop). If that is successful, ping the next hop, and
so forth.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
3-76 OL-23591-02
Chapter 3 Troubleshooting Interface Connectivity
Troubleshooting Ping and ARP Connectivity
Step 3 Check the routing table to make sure the IP address you are trying to ping has a route in the Routing
Information Base (RIB) table.
RP/0/RSP0/CPU0:router# show route
Step 4 Verify that the IGP protocol is running and the connection to the neighbor is up. The following example
assumes OSPF as the IGP.
RP/0/RSP0/CPU0:router# show protocols ospf
Step 5 Verify that packets are coming in and going out on the Ethernet interface.
Be aware of the following ARP behaviors when you are reviewing the display from the show commands
in this step:
• A normal ping will send an ARP packet out followed by the actual ICMP echo packets. ARP must
work before ICMP echo can work. If the system is receiving zero packets back, then there was no
ARP reply. Even a single packet back means there was an ARP reply. (The system sends the ARP
packet only if there is no ARP entry. Otherwise, it skips the ARP and proceeds with the ICMP echo.)
• By default, the system attempts to ping the remote router five times. If the remote router was recently
connected to the network, the first ping will fail because the system needs time to resolve the ARP
packet.
RP/0/RSP0/CPU0:router# show interfaces location node-id
or
RP/0/RSP0/CPU0:router# show interfaces type
Step 6 Display the ARP information. The IP address that you attempted to ping should be in the output.
RP/0/RSP0/CPU0:router# show arp
Step 7 If this port was previously attached to another device, or some othe major change has taken place on the
remote end, use the clear arp-cache command to build a new entry. Verify that the MAC address in the
ARP table is correct (see the MAC address in the Hardware Addr column in the example in Step 6).
RP/0/RSP0/CPU0:router# clear arp-cache
Step 8 Determine whether an ARP entry exists for the destination IP.
RP/0/RSP0/CPU0:router# show arp location node-id
a. If an ARP entry does not exist or is incomplete, add a static ARP entry. Ensure that the Tx adjacency
points to ‘COMPLETE’.
RP/0/RSP0/CPU0:router# show cef {ipv4} prefix hardware egress detail location node-id
Caution After you finish using the static ARP entry for troubleshooting purposes, you must remove it. If you do
not remove the static ARP entry, it will cause traffic to be misdirected.
b. If the ARP entry points to ‘COMPLETE’, it means that the ARP entry is not being updated.
Troubleshooting should now focus on why the ARP entry is not getting added (this includes steps
such as show arp, show arp idb, show adjacency gig node-id detail location node-id, and show
arp trace).
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 3-77
Chapter 3 Troubleshooting Interface Connectivity
Troubleshooting Ping and ARP Connectivity
c. If the Tx adjacency still points to ‘PUNT’, it means ARP is adding the entry in the database, but
fib_mgr fails to mark the adjacency as ‘COMPLETE’.
d. This could be a fib_mgr, ARP, or AIB problem. Delete and reconfigure the static ARP entry with
AIB and CEF debugs on. The debugs show if ARP is adding the entry inside the AIB and if the AIB
is informing fib_mgr.
Step 9 Send a burst of traffic to help troubleshoot whether traffic is getting through. This can help in scenarios
such as a disconnected cable, intermittent drops at unknown locations, and so forth.
a. Configure a static ARP entry, then send a large number of ping packets (for example, 100 or 1,000
packets) with a zero timeout. This sends out a burst of traffic from the router. When the ping fails,
the system displays a dot instead of an exclamation point. If the ping is not possible, the system
displays a ‘U’ instead of a dot.
Example:
b. There are parameters to the ping command that can be used to change the time delay for the reply.
Try lowering the delay, changing parameters to send 100 ping requests, and so forth.
c. There is a mode that allows you to suppress the ARP request and send out only the ICMP echo
packets. See if the pings are failing intermittently or all the time.
Step 10 If ARP connectivity fails, perform the following steps to find out why.
a. Remove the static ARP entry.
b. Local ping—Ping your own interface (the interface your router uses to send out the pings).
c. Determine whether the local ping was successful. If the local ping failed (no response), pings out of
that interface will also fail.
d. If you have an ARP entry, verify that there are outgoing/incoming ICMP echo/reply packets.
Note A ping is represented by a dot, exclam point, or capital U. The RSP allows a specific number of
seconds for the ping to complete. The ARP is hidden inside this event.
Step 11 If you have to dig deeper into the issue, use the following commands to dump ping packets.
Example
RP/0/RSP0/CPU0:router# show route
C 172.21.116.0/24 is directly connected, 2d19h, MgmtEth0/RSP0/CPU0/0
is directly connected, 2d19h, MgmtEth0/RSP1/CPU0/0
L 172.21.116.10/32 is directly connected, 2d20h, MgmtEth0/RSP0/CPU0/0
L 172.21.116.11/32 is directly connected, 2d19h, MgmtEth0/RSP1/CPU0/0
L 172.21.116.12/32 [0/0] via 172.21.116.12, 2d19h, MgmtEth0/RSP0/CPU0/0
O 192.168.12.0/24 [110/2] via 192.168.111.11, 2d19h, GigabitEthernet0/2/0/1
[110/2] via 192.168.121.12, 2d19h, GigabitEthernet0/2/0/2
O 192.168.21.0/24 [110/2] via 192.168.111.11, 2d19h, GigabitEthernet0/2/0/1
[110/2] via 192.168.121.12, 2d19h, GigabitEthernet0/2/0/2
C 192.168.111.0/24 is directly connected, 2d20h, GigabitEthernet0/2/0/1
L 192.168.111.1/32 is directly connected, 2d20h, GigabitEthernet0/2/0/1
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
3-78 OL-23591-02
Chapter 3 Troubleshooting Interface Connectivity
Troubleshooting Ping and ARP Connectivity
The following example shows the packet counts for a line card. Note that there are packets being input
and output.
RP/0/RSP0/CPU0:router# show interfaces location 0/4/CPU0
Wed Sep 1 09:22:03.427 DST
TenGigE0/4/0/0 is up, line protocol is up
Interface state transitions: 1
Hardware is TenGigE, address is 001b.53ff.a780 (bia 001b.53ff.a780)
Layer 1 Transport Mode is LAN
Description: Connected to P11_CRS-4 10GE 0/2/5/0
Internet address is 10.114.8.44/24
MTU 9100 bytes, BW 10000000 Kbit (Max: 10000000 Kbit)
reliability 255/255, txload 0/255, rxload 0/255
Encapsulation ARPA,
Full-duplex, 10000Mb/s, LR, link type is force-up
output flow control is off, input flow control is off
loopback not set,
ARP type ARPA, ARP timeout 04:00:00
Last input 00:00:00, output 00:00:00
Last clearing of "show interface" counters never
5 minute input rate 28000 bits/sec, 39 packets/sec
5 minute output rate 45000 bits/sec, 39 packets/sec
2356786692 packets input, 151622450429 bytes, 26 total input drops
0 drops for unrecognized upper-level protocol
Received 2 broadcast packets, 2327063140 multicast packets
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 3-79
Chapter 3 Troubleshooting Interface Connectivity
Troubleshooting Ping and ARP Connectivity
-------------------------------------------------------------------------------
0/4/CPU0
-------------------------------------------------------------------------------
Address Age Hardware Addr State Type Interface
10.114.8.11 02:40:44 001b.0c63.67ff Dynamic ARPA TenGigE0/4/0/0
10.114.8.44 - 001b.53ff.a780 Interface ARPA TenGigE0/4/0/0
-------------------------------------------------------------------------------
0/1/CPU0
-------------------------------------------------------------------------------
Address Age Hardware Addr State Type Interface
10.114.4.11 00:15:22 001b.0c63.67e7 Dynamic ARPA GigabitEthernet0/1/0/23
10.114.4.44 - 001b.53ff.87f7 Interface ARPA GigabitEthernet0/1/0/23
10.145.4.38 01:43:50 001e.f77d.5219 Dynamic ARPA GigabitEthernet0/1/0/27
10.145.4.44 - 001b.53ff.87fb Interface ARPA GigabitEthernet0/1/0/27
10.146.4.44 - 001b.53ff.87e8 Interface ARPA GigabitEthernet0/1/0/8
10.146.4.66 02:56:39 0022.0d26.3bc4 Dynamic ARPA GigabitEthernet0/1/0/8
10.147.4.44 - 001b.53ff.87e2 Interface ARPA GigabitEthernet0/1/0/2
10.147.4.64 00:33:21 0022.0d26.36c4 Dynamic ARPA GigabitEthernet0/1/0/2
10.194.4.19 03:16:59 001a.3029.d400 Dynamic ARPA GigabitEthernet0/1/0/18
10.194.4.44 - 001b.53ff.87f2 Interface ARPA GigabitEthernet0/1/0/18
10.194.8.44 - 001b.53ff.87f0 Interface ARPA Bundle-Ether16.162
10.194.12.44 - 001b.53ff.87f0 Interface ARPA Bundle-Ether16.163
10.194.16.44 - 001b.53ff.87ec Interface ARPA GigabitEthernet0/1/0/12
-------------------------------------------------------------------------------
0/RSP0/CPU0
-------------------------------------------------------------------------------
Address Age Hardware Addr State Type Interface
172.29.52.1 01:51:49 001e.f77d.2a19 Dynamic ARPA MgmtEth0/RSP0/CPU0/0
172.29.52.13 03:29:42 0010.79e9.6038 Dynamic ARPA MgmtEth0/RSP0/CPU0/0
172.29.52.21 00:50:04 0022.0d5a.a6c4 Dynamic ARPA MgmtEth0/RSP0/CPU0/0
172.29.52.22 02:44:58 0001.6443.1678 Dynamic ARPA MgmtEth0/RSP0/CPU0/0
172.29.52.27 02:36:46 0012.7fd6.ba08 Dynamic ARPA MgmtEth0/RSP0/CPU0/0
172.29.52.28 03:04:37 0012.7fd6.ba09 Dynamic ARPA MgmtEth0/RSP0/CPU0/0
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
3-80 OL-23591-02
Chapter 3 Troubleshooting Interface Connectivity
Troubleshooting Bidirectional Forwarding Detection
path-idx 1
next hop 10.114.8.11
local adjacency
local label 16021 labels imposed {16031}
.
.
.
TX Adjacency
Raw data for tx adj struct:
Raw result1: 0x03000100 0x01000000 0x7e23001b 0x0c6367ff
Raw result2: 0x95650300 0x00000000 0x00000000 0x00000000
---------------------------------------------
Search Ctrl Flags:
------------------
match : 1 valid : 1
gre_adj : 0 null_route : 0
tx_punt : 0 tx_drop : 0
next_hop_down : 0 adj_complete : 0
punt_ifib : 0 nhop_down : 0
stop : 0 match_all_bit: 0
default_action: 1
uidb_index : 0x0001
l3_mtu : 9086
dest mac : 001b.0c63.67ff
prefix_adj_cnt_index: 0x95650300
RX Adjacency
Raw data for rx adj struct:
Raw result1: 0x13000100 0x00001300 0x0c000280 0x00000000
---------------------------------------------
Search Ctrl Flags:
------------------
rx_punt : 0 rx_drop : 0
rx_adj_SFP : 1 rp_destined : 0
rp_drop : 0 match : 1
valid : 1 rx_LAG_adj : 0
match_all_bit : 0 pri_adj_down : 0
default_action: 1
rx_adj_field : 0x0013
egress_ifh : 0xc000280
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 3-81
Chapter 3 Troubleshooting Interface Connectivity
Troubleshooting Bidirectional Forwarding Detection
DETAILED STEPS
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
3-82 OL-23591-02
Chapter 3 Troubleshooting Interface Connectivity
Troubleshooting Bidirectional Forwarding Detection
Example:
RP/0/RSP0/CPU0:router# show tech-support
routing bfd location 0/1/CPU0
Step 2 Ensure that the router and remote device are configured with the following parameters:
• Number of BFD sessions they can support
• Timers to support the police rates
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 3-83
Chapter 3 Troubleshooting Interface Connectivity
Troubleshooting Bidirectional Forwarding Detection
Note SPP means software packet processing, but is more commonly referred to as vector path
processing (VPP).
If the following message appears, the BFD flap is a result of the application flap.
bfd_agent[104]: %BFD-6-SESSION_REMOVED : BFD session to neighbor 192.168.1.1 on interface
Gi0/5/0/0 has been remove
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
3-84 OL-23591-02
Chapter 3 Troubleshooting Interface Connectivity
Troubleshooting Ethernet CFM
Examine the SPP process on the LC CPU to determine the delay encountered by BFD echo packets:
RP/0/RSP0/CPU0:router# show bfd trace performance reverse location
Rule out BFD echo packet loss: show bfd counters packet location
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 3-85
Chapter 3 Troubleshooting Interface Connectivity
Troubleshooting Ethernet CFM
Operator Domains
Operator 1 Operator 2
CE 1 PE 1 PE 2 PE 3 PE 4 CE 2
207581
Level 3
Tip For an extensive discussion of CFM usage and CFM command examples, see Cisco ASR 9000 Series
Aggregation Services Router Interface and Hardware Component Configuration Guide.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
3-86 OL-23591-02
Chapter 3 Troubleshooting Interface Connectivity
Troubleshooting Ethernet CFM
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 3-87
Chapter 3 Troubleshooting Interface Connectivity
Troubleshooting Ethernet CFM
Note To clear the spp counters, run the command clear spp {client | interface | node-counters} location
node-id. This command clears client statistics, interface statistics, and per-node counters, depending on
the keyword you use.
Step 1 Display information about errors that might be preventing configured CFM operations from becoming
active, as well as any warnings that have occurred.
show ethernet cfm configuration-errors
Step 2 Display a list of local maintenance points that have been created. Verify that the list contains the
expected nodes.
show ethernet cfm local maintenance-points
Step 3 Display operational states of local MEPs. Verify that the states are as expected.
show ethernet cfm local meps
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
3-88 OL-23591-02
Chapter 3 Troubleshooting Interface Connectivity
Troubleshooting Ethernet CFM
Step 1 Display information about errors that might be preventing configured CFM operations from becoming
active, as well as any warnings that have occurred.
show ethernet cfm configuration-errors
Step 2 Display a list of local maintenance points that have been created. Verify that the list contains the
expected nodes. Check for MEPs configured on the interface, and for MIPs enabled on a service at a
lower level.
show ethernet cfm local maintenance-points
Step 3 If MIP creation is not functioning, verify that the bridge domain or xconnect is configured correctly. To
verify or troubleshoot these bridge domain and xconnect configurations, see Chapter 9,
“Troubleshooting L2VPN and Ethernet Services.”
No CCMs are Received at the MEP or Peer MEPs Are Not Seen
This section explains how to troubleshoot the following conditions:
• Continuity check messages (CCMs) are not seen at one or more maintenance end points (MEPs).
• Peer MEPs are not seen.
CFM MEPs exchange CCMs periodically according to parameters configured on the system. These
CCMs are multicast to all other MEPs in the service at the same level. When the local MEP receives a
CCM, it creates an entry in the peer MEP table. If CCMs are not being exchanged correctly, perform the
following steps.
Step 1 Verify that CCM is enabled and there is a supported encapsulation on the interface.
RP/0/RSP0/CPU0:router# show running-config
Step 3 Display operational states of local and peer MEPs. Verify that CCM is enabled and the states are as
expected.
RP/0/RSP0/CPU0:router# show ethernet cfm local meps
RP/0/RSP0/CPU0:router# show ethernet cfm local meps verbose
RP/0/RSP0/CPU0:router# show ethernet cfm peer meps
RP/0/RSP0/CPU0:router# show ethernet cfm peer meps detail
Step 4 View packets seen by the CFM PI. Enable all of the options. The output shows if packets are dropped,
forwarded, or processed.
RP/0/RSP0/CPU0:router# debug ethernet cfm packets packet-type ccm
Step 5 View remote MEPs shown by the specific LC CFM instance. If CCMs are not received, the peer does not
display.
Step 6 View CFM SID statistics seen by the SPP. This displays any CFM traffic that is injected and punted.
RP/0/RSP0/CPU0:router# show spp sid stats
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 3-89
Chapter 3 Troubleshooting Interface Connectivity
Troubleshooting Ethernet CFM
Step 9 Check for dropped PDUs as described in the “Dropped CFM PDUs” section on page 3-92.
Step 1 Run the following commands to obtain the output you will need for detailed troubleshooting.
RP/0/RSP0/CPU0:router# show ethernet cfm local meps
RP/0/RSP0/CPU0:router# show ethernet cfm local meps verbose
RP/0/RSP0/CPU0:router# show ethernet cfm peer meps
RP/0/RSP0/CPU0:router# show ethernet cfm peer meps detail
Step 2 Wrong level—Check the output of the commands in Step 1 to see if CCMs are being received at a lower
level than the level of the local MEP. This indicates a misconfiguration, for example:
• The domain level is configured incorrectly on the local device or the peer device.
• An intended MEP at the lower level has not been configured, and as a result the CCMs it would
consume are reaching the local MEP.
• The forwarding path within the network has been misconfigured, such that CCM packets are being
received from an unintended source.
Step 3 Cross-connect (wrong MAID)—Check the output of the commands in Step 1 to see if CCMs are being
received with an maintenance association identifier (MAID) that does not match the MAID configured
locally for the service. The MAID is formed from the maintenance domain identifier (MDID) and the
short maintenance association name (SMAN). By default, the MDID is set to the name of the domain
and the SMAN is set to the name of the service. A crossconnect error indicates a misconfiguration, for
example:
• The domain name or ID is configured incorrectly on the local device or on the peer device.
• The service name or ID is configured incorrectly on the local device or on the peer device.
• The forwarding path within the network has been misconfigured, such that CCM packets are being
received from an unintended source.
Step 4 Wrong interval—Check the output of the commands in Step 1 to see if CCMs are being received with a
CCM interval that does not match the locally configured CCM interval. This indicates that the interval
is configured incorrectly on either the local device or the peer device. For a given service, the same CCM
interval must be configured on all devices.
Step 5 Loop (local MAC address received)—Check the output of the commands in Step 1 to see if CCMs are
being received with the source MAC equal to the MAC address of the interface for the local MEP. This
indicates that there is a loop in the network such that the local device is receiving its own packets, or that
two devices in the network are configured with the same MAC address.
Step 6 Configuration (local MEP ID received)—Check the output of the commands in Step 1 to see if CCMs
are being received from a peer MEP with the same MEP ID as the local MEP. This defect indicates that
two MEPs are configured with the same MEP ID. Across the entire network, each MEP in the service
must be configured with a different MEP ID.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
3-90 OL-23591-02
Chapter 3 Troubleshooting Interface Connectivity
Troubleshooting Ethernet CFM
Step 7 Peer interface down—Check the output of the commands in Step 1 to see if CCMs are being received
that indicate the interface on the peer MEP is down, or that the interface on every peer MEP is STP
blocked. This indicates a problem with the operational state of the network.
Step 8 Missing (crosscheck)—Check the output of the commands in Step 1. If crosscheck is configured
specifying this peer MEP, but no CCMs are being received, the peer MEP is missing. This might indicate
a failure in the network.
Step 9 Unexpected (crosscheck)—Check the output of the commands in Step 1. If crosscheck is configured and
CCMs are being received from a peer MEP that is not specified, these CCMs are unexpected. This may
indicate a misconfiguration or that CCMs are being received from an unintended source.
Step 10 Remote defect received—Check the output of the commands in Step 1. If received CCMs indicate that
the peer MEP has detected a defect, take the action recommended in the “Remote Defect Indication
Received” section on page 3-91.
Step 1 Run the following commands to obtain the output you will need for troubleshooting RDIs.
RP/0/RSP0/CPU0:router# show ethernet cfm peer meps
RP/0/RSP0/CPU0:router# show ethernet cfm peer meps detail
Step 2 Determine the peer MEP from which the RDI is being received.
Step 3 Log into the peer device and follow the steps in the “Peer MEP Defects and Mismatches Are Seen”
section on page 3-90.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 3-91
Chapter 3 Troubleshooting Interface Connectivity
Troubleshooting Ethernet CFM
Note Note that the CFM packet rate limit (16,000 CFM packets per second per line card) includes all
CFM packet types, including linktrace (traceroute) and loopback (ping) packets, as well as
CCMs and Ethernet SLA probes. Normally, the number of linktrace or loopback packets is low;
however, the use of “continuity-check auto-traceroute” can cause a high number of linktrace
packets to be sent, if a number of peer MEPs time out in quick succession.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
3-92 OL-23591-02
Chapter 3 Troubleshooting Interface Connectivity
Troubleshooting Ethernet CFM
To display information for troubleshooting dropped CFM PDUs, perform the following steps. Take
corrective actions based on the outputs of the commands in these steps.
Step 1 Enable packet debugging to determine whether forwarded packets are being received at the MIP.
RP/0/RSP0/CPU0:router# debug ethernet cfm packets [received dropped interface
gigabitEthernet node-id]
Step 2 Display the statistics of the CFM PDUs per interface. Look for any drops ted to packets that are
improperly formed, invalid, wrong level, or unknown type.
RP/0/RSP0/CPU0:router# show ethernet cfm interfaces statistics
Step 3 Display the local MEPs and look for discarded CCMs. Discarded CCMs might indicate the the
configured maximum MEPs limit (default 100 MEPs per service) is reached.
RP/0/RSP0/CPU0:router# show ethernet cfm local meps verbose
Step 5 Check the STP status on the interfaces with MEPs or MIPs. CFM PDUs originating at MEPs on a STP
block port get forwarded, however, PDUs forwarded on a MIP are subject to the STP port state. This
means that if MIP is on a port that is STP blocked, then CFM PDUs will be dropped at the MIP.
Step 6 View STP state and CFM peer MEP status.
RP/0/RSP0/CPU0:router# show spanning-tree mst mstp
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 3-93
Chapter 3 Troubleshooting Interface Connectivity
Troubleshooting Ethernet CFM
• If there is another MEP on the interface at a higher level, and in the same direction, the AIS messages
are sent internally from the lower level MEP to the next highest level MEP. In this case, no actual
PDUs are transmitted.
• Otherwise, if there is a MIP on the interface then AIS PDUs are transmitted at the level of the MIP.
If there is no MIP on the interface, no AIS messages are transmitted.
Use the following steps for troubleshooting.
Step 3 Display the information published in the interface AIS table, including a record of the AIS
transmissions. Determine whether AIS messages are actually being sent.
RP/0/RSP0/CPU0:router# show ethernet cfm interfaces ais
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
3-94 OL-23591-02
CH A P T E R 4
Troubleshooting Packet Forwarding
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 4-95
Chapter 4 Troubleshooting Packet Forwarding
Troubleshooting IPv4 CEF
Figure 4-1 shows the components that contribute information to the CEF process, including
autosynchronization of the RIB with the FIB.
Note In this document, the FIB is also referred to as the CEF table.
LDP RP LC-CPU HW
RSVP
AIB IFMGR
LSD
BGP
BCDL
OSPF FIB
GSP Process
RIB
ISIS
Sw Hw
FIB FIB
STATIC
ROUTES
208808
Netio
SUMMARY STEPS
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
4-96 OL-23591-02
Chapter 4 Troubleshooting Packet Forwarding
Troubleshooting IPv4 CEF
DETAILED STEPS
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 4-97
Chapter 4 Troubleshooting Packet Forwarding
Troubleshooting IPv4 CEF
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
4-98 OL-23591-02
Chapter 4 Troubleshooting Packet Forwarding
Troubleshooting IPv4 CEF
Step 10 show cef platform trace ipv4 all location Displays CEF IPv4 hardware status and configuration trace
node-id table information.
Use this command for the ingress and egress interfaces for
Example: the local line card.
RP/0/RSP0/CPU0:router# show cef platform trace
ipv4 all location 0/3/cpu0
Step 11 show controllers pse qfp feature forward client (For SIP-700 line cards only) Displays trace files that
ltrace unicast error location node-id contain information on any engine error (if any) that
occurred in the unicast hardware structure programming.
Example:
RP/0/RSP0/CPU0:router# show contro pse qfp
feature forward client ltrace unicast error
location node-id
Step 12 Contact Cisco Technical Support. If the problem is not resolved, contact Cisco Technical
Support. For Cisco Technical Support contact information,
see the “Obtaining Documentation and Submitting a
Service Request” section on page xii in the Preface.
Examples
The following examples show routes to two networks, one that is directly connected and one that is
learned. In the first example, the route was installed about 19 days ago, which might be as expected.
However, in the second example, the route was installed only 54 seconds ago, so it appears to be
flapping:
RP/0/RSP0/CPU0:router# show route ipv4 10.114.4.11
Tue Jul 13 09:25:47.754 DST
Routing entry for 10.114.4.0/24
Known via "connected", distance 0, metric 0 (connected)
Installed Jul 12 14:18:06.668 for 19:07:41 <<< This route appears to be stable
Routing Descriptor Blocks
directly connected, via GigabitEthernet0/1/0/23
Route metric is 0
Redist Advertisers:
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 4-99
Chapter 4 Troubleshooting Packet Forwarding
Troubleshooting IPv4 CEF
ospf 100
The following example shows the CEF summary. Use this display to check the VRF names, route update
drops, and adjacencies:
RP/0/RSP0/CPU0:router# show cef ipv4 summary location 0/1/CPU0
Tue Jul 13 12:50:48.259 DST
Router ID is 10.144.144.144
IP CEF with switching (Table Version 552) for node0_1_CPU0
Load balancing: L4
Tableid 0xe0000000 (0xa4a6ddb0), Vrfid 0x60000000, Vrid 0x20000000, Flags 0x301
Vrfname default, Refcount 251
163 routes, 0 reresolve, 0 unresolved (0 old, 0 new), 13040 bytes
60 load sharing elements, 129968 bytes, 342 references
8 shared load sharing elements, 8564 bytes
52 exclusive load sharing elements, 121404 bytes
0 CEF route update drops, 0 CEF rcc update drops
176 revisions of existing leaves
Resolution Timer: 15s
0 prefixes modified in place
0 deleted stale prefixes
99 prefixes with label imposition, 111 prefixes with label information
23 next hops
0 incomplete next hops
0 PD backwalks on LDIs with backup path
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
4-100 OL-23591-02
Chapter 4 Troubleshooting Packet Forwarding
Troubleshooting Adjacency Information
SUMMARY STEPS
DETAILED STEPS
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 4-101
Chapter 4 Troubleshooting Packet Forwarding
Troubleshooting Adjacency Information
Example: • Check that the table look-up (TLU) pointers match the
RP/0/RSP0/CPU0:router# show adjacency pos TLU pointers in the show cef ipv4 prefix mask
0/13/0/2 remote detail hardware location hardware ingress detail location node-id command.
0/14/cpu0 For example:
Step 5 show adjacency ipv4 nexthop ipv4-address detail Displays adjacencies on an egress line card with a broadcast
location node-id interface that are destined to the specified IPv4 next hop.
When an egress interface is broadcast, use the show
Example: adjacency ipv4 nexthop command to display the adjacency
RP/0/RSP0/CPU0:router# show adjacency ipv4 information.
nexthop 192.168.2.0 detail location 0/12/cpu0
Compare the mac layer rewrite information that shows the
destination L2 address in the first part followed by the
source L2 address, and the Ethernet value with the output
from the show arp location node-id command.
Step 6 show adjacency interface-type Displays CEF adjacency table information for an egress line
interface-instance detail location node-id card with a point to point interface.
There should be two IPv4 entries in the command output.
Example: Ensure both entries exist.
RP/0/RSP0/CPU0:router# show adjacency pos
0/13/0/2 detail location 0/13/cpu0 • The SRC MAC only entry is used for multicast
switching
• The point to point entry is used for unicast switching.
On broadcast interfaces you will have a SRC MAC only and
one for each nexthop IP address. Please note the MTU is for
the IPv4 minus the Layer 2 header. Use the show im chains
command to display MTU details.
Step 7 show adjacency ipv4 nexthop ipv4-address detail Displays the hardware programming associated with the
hardware location node-id adjacency. Verify that the packets are being switched in the
hardware.
Example:
RP/0/RSP0/CPU0:router# show adjacency ipv4
nexthop 192.168.2.0 detail hardware location
0/12/cpu0
Step 8 show adjacency interface-type Displays the hardware programming information for a
interface-instance detail hardware location point-to-point interface such as the
node-id
Packet-over-SONET/SDH (POS) interface. The rewrite
information is slightly different because there is no MAC
Example: rewrite string as there is in Ethernet.
RP/0/RSP0/CPU0:router# show adjacency pos
Verify that the rewrite is appropriate for the encapsulation
0/13/0/2 detail hardware location 0/13/cpu0
on the interface. Compare the CEF hardware output and
verify that the pointer matches the egress adjacency.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
4-102 OL-23591-02
Chapter 4 Troubleshooting Packet Forwarding
Troubleshooting Adjacency Information
Example:
show cef adjacency tunnel-te 1 hardware egress
location 0/13/CPU
Step 13 Contact Cisco Technical Support. If the problem is not resolved, contact Cisco Technical
Support. For Cisco Technical Support contact information,
see the “Obtaining Documentation and Submitting a
Service Request” section on page xii in the Preface.
Examples
RP/0/RSP0/CPU0:router# show adjacency pos 0/2/0/1 remote detail hardware location 0/0/CPU0
Wed Nov 3 13:16:32.119 DST
Interface Address Version Refcount Protocol
PO0/2/0/1 (remote) 15 1( 0) fint_n2n
040001c0
flags 1 0 2
0 packets, 0 bytes
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 4-103
Chapter 4 Troubleshooting Packet Forwarding
Troubleshooting Adjacency Information
path-idx 1
next hop 10.114.8.11
remote adjacency
local label 16018 labels imposed {16012}
IP Leaf Data:
as:0 prefix_len:32
for_us:0x0 dft_route:0x0
real_intf:0x1 free1: 0x0
hw_use_only: 0x0
lspa_ptr: 0x0 oce_chain_p: 0x88f96320
extre_fib_data_ptr: 0x88fccc60
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
4-104 OL-23591-02
Chapter 4 Troubleshooting Packet Forwarding
Troubleshooting Adjacency Information
The following example shows that the address information matches. The addresses are indicated in bold.
RP/0/RSP0/CPU0:router# show arp location 0/1/cpu0
BE16.162 special 2
Interface: BE16.162 Type: glean
Interface Type: 0x19, Base Flags: 0x4400 (0x9e4e9bb0)
Nhinfo PT: 0x9e4e9bb0, Idb PT: 0x9e3591d8, If Handle: 0x80001a0
Dependent adj type: remote (0x9f8af79c)
Dependent adj intf: BE16.162
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 4-105
Chapter 4 Troubleshooting Packet Forwarding
Troubleshooting Transient Traffic Drop
BE16.163 special 2
Interface: BE16.163 Type: glean
Interface Type: 0x19, Base Flags: 0x4400 (0x9e4e9d1c)
Nhinfo PT: 0x9e4e9d1c, Idb PT: 0x9e359218, If Handle: 0x80001e0
Dependent adj type: remote (0x9f8b033c)
Dependent adj intf: BE16.163
Ancestor If Handle: 0x0
SUMMARY STEPS
DETAILED STEPS
Example:
show interface gigabitEthernet 0/0/0/0
Step 2 show platform Determines the line card type. This is necessary because the
next steps depend on whether you are troubleshooting an
Ethernet or SIP-700 line card (LC).
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
4-106 OL-23591-02
Chapter 4 Troubleshooting Packet Forwarding
Troubleshooting Transient Traffic Drop
Example:
show controllers pse qfp stat drop location
0/6/CPU0
Example
RP/0/RSP0/CPU0:router# show interface gigabitEthernet 0/0/0/0
Tue Oct 26 21:04:12.805 UTC
GigabitEthernet0/0/0/0 is up, line protocol is up
Interface state transitions: 5
Hardware is GigabitEthernet, address is 001b.53ff.a018 (bia 001b.53ff.a018)
Internet address is 45.1.1.1/24
MTU 2014 bytes, BW 1000000 Kbit
reliability 255/255, txload 0/255, rxload 0/255
Encapsulation ARPA,
Full-duplex, 1000Mb/s, SXFD, link type is force-up
output flow control is off, input flow control is off
loopback not set,
ARP type ARPA, ARP timeout 04:00:00
Last input 00:00:00, output 00:00:00
Last clearing of "show interface" counters 1w4d
5 minute input rate 4000 bits/sec, 0 packets/sec
5 minute output rate 11000 bits/sec, 0 packets/sec
1590651 packets input, 551036131 bytes, 0 total input drops <<< drops by framer or HW
97206 drops for unrecognized upper-level protocol <<< drops
Received 0 broadcast packets, 332301 multicast packets
0 runts, 0 giants, 0 throttles, 0 parity
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort <<< drops
1536152 packets output, 1427163508 bytes, 0 total output drops <<< sum of all output
drops, including drops from buffer, qos, or HW.
Output 0 broadcast packets, 339069 multicast packets
0 output errors, 0 underruns, 0 applique, 0 resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 4-107
Chapter 4 Troubleshooting Packet Forwarding
Troubleshooting Transient Traffic Drop
Node: 0/0/CPU0:
----------------------------------------------------------------
Node: 0/0/CPU0:
----------------------------------------------------------------
Note For a description of how to interpret NP counter information, see the “Displaying Traffic Status in Line
Cards and RSP Cards” section on page 7-147.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
4-108 OL-23591-02
Chapter 4 Troubleshooting Packet Forwarding
Troubleshooting Packet Drop in the Fabric
AttnInvalidSpid 0 0
BadAdj 0 0
BadBhdr 0 0
SUMMARY STEPS
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 4-109
Chapter 4 Troubleshooting Packet Forwarding
Troubleshooting Control Plane Information
DETAILED STEPS
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
4-110 OL-23591-02
Chapter 4 Troubleshooting Packet Forwarding
Troubleshooting Control Plane Information
Examples
The following example displays the control plane information for the software switching path. Check
for any errors or drops.
RP/0/RSP0/CPU0:router# show netio idb tenGigE 0/1/1/0 location 0/1/cpu0
Chains
--------------------
Base decap chain:
ether <30> <0xfd7aef88, 0x48302824> < 0, 0>
Protocol chains:
---------------
<Protocol number> (name) Stats
Type Chain_node <caps num> <function, context> <drop pkts, drop bytes>
<7> (arp) Stats IN: 0 pkts, 0 bytes; OUT: 0 pkts, 0 bytes
Encap:
l2_adj_rewrite <86> <0xfcec7a88, 0x4834efec> < 0, 0>
queue_fifo <56> <0xfcedda68, 0x482dbee4> < 0, 0>
txm_nopull <60> <0xfcea2a5c, 0x482dc11c> < 0, 0>
Decap:
queue_fifo <56> <0xfcedda4c, 0x482dbee4> < 0, 0>
arp <24> <0xfd1082cc, 0x00000000> < 0, 0>
Fixup:
l2_adj_rewrite <86> <0xfcec745c, 0x00000000> < 0, 0>
queue_fifo <56> <0xfcedda68, 0x482dbee4> < 0, 0>
txm_nopull <60> <0xfcea2a5c, 0x482dc11c> < 0, 0>
<12> (ipv4) Stats IN: 0 pkts, 0 bytes; OUT: 0 pkts, 0 bytes
Encap:
ipv4 <26> <0xfd10f41c, 0x482d7724> < 0, 0>
ether <30> <0xfd7aeb44, 0x48302824> < 0, 0>
l2_adj_rewrite <86> <0xfcec7a88, 0x4834f104> < 0, 0>
queue_fifo <56> <0xfcedda68, 0x482dbee4> < 0, 0>
txm_nopull <60> <0xfcea2a5c, 0x482dc11c> < 0, 0>
Decap:
queue_fifo <56> <0xfcedda4c, 0x482dbee4> < 0, 0>
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 4-111
Chapter 4 Troubleshooting Packet Forwarding
Troubleshooting Control Plane Information
The following example shows that the micro-idb index value is 12.
RP/0/RSP0/CPU0:router# show uidb index tengige1/3/0/6.30 location 1/3/cpu0
------------------------------------------------------------------------------
Location Interface-name Interface-Type Ingress-index Egress-index
---------------------------------------------------------------------------
1/3/CPU0 TenGigE1_3_0_6.30 Sub-interface 20 12
Comparing the IDB index value of 12 in the show uidb index command to the uidb index value in the
following command output shows that the values are the same.
RP/0/RSP0/CPU0:router# show adjacency ipv4 tengige1/3/0/6.30 detail hardware location
1/3/cpu0
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
4-112 OL-23591-02
Chapter 4 Troubleshooting Packet Forwarding
Troubleshooting Control Plane Information
The following example displays, from a software perspective, features that are enabled on a selected
interface. Compare the output to the configuration of the interface and expected features. Verify that the
configured features are correctly enabled.
RP/0/RSP0/CPU0:router# show uidb data location 0/6/cpu0
--------------------------------------------------------------------------
Location = 0/6/CPU0
Index = 0
Pse direction = INGRESS
General 16 bytes:
-----------------
IFHANDLE: 0x168002
STATUS: 0
IPV4 ENABLE: 0
IPV6 ENABLE: 0
MPLS ENABLE: 0
STATS POINTER: 0x2c400
SPRAYER QUEUE: 32
IPV4 MULTICAST: 0
IPV6 MULTICAST: 0
USE TABLE ID IPV4: 0
USE TABLE ID IPV6: 0
USE TABLE ID MPLS: 0
TABLE ID: 0
QOS ENABLE: 0
QOS ID: 0
NETFLOW SAMPLING PERIOD: 0
L2 PKT DROP: 0
L2 QOS ENABLE: 0
SRC FWDING: 0
*BUNDLE IFHANDLE: 0
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 4-113
Chapter 4 Troubleshooting Packet Forwarding
Troubleshooting Control Plane Information
*TUNNEL IFHANDLE: 0
*L2 ENCAP: 3
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
4-114 OL-23591-02
CH A P T E R 5
Troubleshooting Bundles and Load Balancing
This chapter explains the procedures for troubleshooting link bundles and load balancing on the
Cisco ASR 9000 Aggregation Services Router.
A link bundle is a group of ports that are bundled together and act as a single link. The advantages of
link bundles are:
• Multiple links can span several LCs to form a single interface; thus, the failure of a single link does
not cause a loss of connectivity.
• Bundled interfaces increase bandwidth availability, because traffic is forwarded over all available
members of the bundle. Therefore, traffic can move onto another link if one of the links within a
bundle fails. This allows you to add or remove bandwidth without interrupting packet flow.
This chapter contains the following sections:
• Troubleshooting Routing and CEF Issues Related to Bundles and Load Balancing, page 5-115
• Troubleshooting Problems with Link Bundles, page 5-118
• Troubleshooting Layer 2 Bundles and Load Balancing, page 5-122
• Troubleshooting Layer 3 Bundles and Load Balancing, page 5-124
SUMMARY STEPS
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 5-115
Chapter 5 Troubleshooting Bundles and Load Balancing
Troubleshooting Routing and CEF Issues Related to Bundles and Load Balancing
2. configure
3. router ospf process
4. maximum paths number
5. end
6. show route destination-address
7. show ospf process interface brief
8. show running-config router ospf process
Example:
RP/0/RSP0/CPU0:router# configure
Step 3 router ospf process Enters configuration mode for the OSPF process.
Example:
RP/0/RSP0/CPU0:router# router ospf 200
Step 4 maximum paths number Configures the maximum number of paths over which to
load balance. By default, OSPF balances up to 4 equal-cost
paths.
Example:
RP/0/RSP0/CPU0:router# maximum paths 3
Step 5 end Ends the configuration process. Enter yes at the prompt to
commit the changes.
Example:
RP/0/RSP0/CPU0:router# end
Step 6 show route destination-address Displays the routes to a destination address.
Verify that number of routes in the routing table equals the
Example: number of parallel links. If you have fewer routes than
RP/0/RSP0/CPU0:router# show route 10.1.2.1 expected, continue with this procedure.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
5-116 OL-23591-02
Chapter 5 Troubleshooting Bundles and Load Balancing
Troubleshooting Routing and CEF Issues Related to Bundles and Load Balancing
SUMMARY STEPS
1. show cef ipv4 [prefix [mask]] | interface-type interface-path-id] [detail] [location node-id]
2. show cef [ipv4 | ipv6] exact-route source-address destination address [protocol type]
[source-port source-port] [destination-port destination-port] [ingress-interface type
interface-path-id] [policy-class value] [detail | location node-id]
3. show interfaces [type interface-path-id | all | local | location node-id] [accounting | brief | detail |
summary]
DETAILED STEPS
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 5-117
Chapter 5 Troubleshooting Bundles and Load Balancing
Troubleshooting Problems with Link Bundles
Example:
RP/0/RSP0/CPU0:router# show cef exact-route
192.168.254.1 10.1.2.1 protocol ospf
source-port 5500 destination-port 80
ingress-interface gi0/6/5/4
Step 3 show interfaces [type interface-path-id | all | Displays the traffic rates by interface. Use this command to
local | location node-id] [accounting | brief | verify that the simulated traffic takes the expected egress
detail | summary]
interface.
Example:
RP/0/RSP0/CPU0:router# show interfaces
accounting rates
Step 2 If running Link Aggregation Control Protocol (LACP), ensure that LACP packets are able to send and
receive accordingly. If LACP packets are not able to send and receive accordingly, check interface
counters to identify at what stage packets are dropped.
RP/0/RSP0/CPU0:router# show lacp counters
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
5-118 OL-23591-02
Chapter 5 Troubleshooting Bundles and Load Balancing
Troubleshooting Problems with Link Bundles
Step 4 Ensure that the other side of the link is up (bundle and members).
RP/0/RSP0/CPU0:router# show bundle
c. If the bundle members have different characteristics, make them all the same.
d. Ensure that LACP packets are transmitted and received.
RP/0/RSP0/CPU0:router# debug bundlemgr local packets port node-id
Workaround
If the bundle with LACP cannot come up, use one side of the bundle in passive mode and the other in
active mode. At least one side must be active.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 5-119
Chapter 5 Troubleshooting Bundles and Load Balancing
Troubleshooting Problems with Link Bundles
Step 2 Verify that the lag table is programmed properly in the hardware.
Subinterface
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
5-120 OL-23591-02
Chapter 5 Troubleshooting Bundles and Load Balancing
Troubleshooting Problems with Link Bundles
Step 10 Ensure that the MAC address in the ARP table of the other side corresponds to that on the router.
Step 11 Ensure that the MAC address of the bundle is valid.
Step 12 Ensure that the routing and hardware routing table has an entry to the next hop.
Step 13 Check the interface counters to see if ping packets are transmitted and being received on the router
member port of the bundle.
Step 14 Check the ucode counters to see where packets are dropped on the incoming or outgoing member of the
bundle.
Step 15 Make sure that the table lookup (TLU) entries are allocated and bundle adjacency information is properly
programmed.
show cef adajacency bundle-type bundle-number hardware egress detail location location-id
show cef adajacency bundle-type bundle-number hardware ingress remote detail location
location-id
Workaround
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 5-121
Chapter 5 Troubleshooting Bundles and Load Balancing
Troubleshooting Layer 2 Bundles and Load Balancing
VPWS
Bundle Statistics
Layer 2 statistics are not supported in the show interface accounting command for bundle interfaces in
the current release.
SUMMARY STEPS
1. show running-config
2. show bundle {Bundle-Ether | Bundle-POS} interface-path-id
3. show interface {Bundle-Ether | Bundle-POS} bundle-id
4. show arm router-id
5. show controllers bundle {Bundle-Ether | Bundle-POS} bundle-id location node-id
6. show route destination-address
7. show cef ipv4 prefix
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
5-122 OL-23591-02
Chapter 5 Troubleshooting Bundles and Load Balancing
Troubleshooting Layer 2 Bundles and Load Balancing
DETAILED STEPS
Example:
RP/0/RSP0/CPU0:router# show interface
bundle-ether 12
Step 4 show arm router-id —
Example:
RP/0/RSP0/CPU0:router# show arm router-id
Step 5 show controllers bundle {Bundle-Ether | —
Bundle-POS} bundle-id location node-id
Example:
RP/0/RSP0/CPU0:router# show controllers bundle
bundle-ether 12 location 0/4/CPU0
Step 6 show route destination-address Displays the routes to a destination address. Use a
destination address on another host that is reachable
through the bundle.
Example:
RP/0/RSP0/CPU0:router# show route 10.1.2.1 Verify that the route to the desalination address includes the
bundle interface. If not, make sure that the bundle interface
is included in the IGP process configuration.
Step 7 show cef ipv4 prefix Displays the CEF forwarding table. Verify that it contains
the same bundle interface that the routing table has for this
subnet prefix.
Example:
RP/0/RSP0/CPU0:router# show cef ipv4 10.1.2.1
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 5-123
Chapter 5 Troubleshooting Bundles and Load Balancing
Troubleshooting Layer 3 Bundles and Load Balancing
SUMMARY STEPS
DETAILED STEPS
Example:
RP/0/RSP0/CPU0:router# bundle-hash
bundle-ether 12
Step 2 show interfaces [type interface-path-id | all | Displays interface information, which includes the traffic
local | location node-id] [accounting | brief | rates. Use this command for each link in the bundle to verify
detail | summary]
that the simulated traffic takes the expected link. Use clear
counters to make it easier to view the traffic allocation.
Example:
RP/0/RSP0/CPU0:router# show interfaces tenGigE
0/5/0/1
Step 7 View each member of the bundle to see which member is actually carrying the traffic out.
Step 8 Display the exact route, including the egress interface for a specific source and destination IP. Use this
command for several flows to verify that they are distributed equally over the parallel interfaces.
show cef [ipv4 | ipv6] exact-route source-address destination address [protocol type]
[source-port source-port] [destination-port destination-port] [ingress-interface type
interface-path-id] [policy-class value] [detail | location node-id]
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
5-124 OL-23591-02
CH A P T E R 6
Troubleshooting Layer 3 Connectivity
This section explains how to troubleshoot Layer 3 routing problems. If a ping to a remote site fails, the
cause could be in an interface or in the Layer 3 routing. The overall approach for troubleshooting a failed
ping should be to troubleshoot the interface failures and interface connectivity first, then proceed to
troubleshooting Layer 3 routing if necessary.
Note For interface troubleshooting, perform the procedures listed in Chapter 2, “Verifying and
Troubleshooting Interface Status” and Chapter 3, “Troubleshooting Interface Connectivity.”
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 6-125
Chapter 6 Troubleshooting Layer 3 Connectivity
Using show and debug Commands
DETAILED STEPS
Example:
RP/0/RSP0/CPU0:router# show cef ipv4
192.168.1.1/32 location 0/2/CPU0
Step 3 show bgp summary View Border Gateway Protocol (BGP) neighbors without an
inbound and outbound policy for each active address family.
Note Use this when there are many routes.
Step 4 show bgp [{ipv4 | all} {unicast | multicast | View which routes have dampening enabled.
all}] dampened-paths
Example:
RP/0/RSP0/CPU0:router# show bgp dampened-paths
Step 5 show bgp flap-statistics [ip-address[/mask]] View BGP flap statistics.
Note Use this for routes that have had dampening
enabled.
Example:
RP/0/RSP0/CPU0:router# show bgp flap-statistics If you do not specify arguments or keywords, all routes for
the address family are displayed.
If you enter an IP address without mask or prefix length, the
longest matching prefix is displayed.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
6-126 OL-23591-02
Chapter 6 Troubleshooting Layer 3 Connectivity
Using show and debug Commands
Example:
RP/0/RSP0/CPU0:router# show interface
accounting location 0/4/CPU0
Step 8 show cef ipv4 {prefix/mask} hardware {ingress | View IPv4 prefix/route in the hardware of an LC.
egress} location node-id
This information helps determine if the destination IP or
prefix action is COMPLETE, PUNT or DROP.
Example:
RP/0/RSP0/CPU0:router# show cef ipv4
38.1.1.2/32 hardware egress location 0/4/CPU0
Step 9 show cef platform trace common [all | errors | View common Dynamic Link Library (DLL) code traces.
events | info] [location node-id]
Example:
RP/0/RSP0/CPU0:router# show cef platform trace
common all errors location 0/4/CPU0
Step 10 show cef vrf [vrfname] [prefix] Verify that the L3 MTU value, encapsulation string value,
byte count, and packet count are as expected. (See the
example below.)
Example:
RP/0/RSP0/CPU0:router# show cef vrf 0xx
Example
RP/0/RSP0/CPU0:router# show cef vrf vrf1 192.168.1.2 hardware egress location 0/1/CPU0
192.168.1.2/32, version 0, internal 0x40800001 (ptr 0xaac1c468) [1], 0x0 (0xaab8c7b0), 0x0
(0x0)
Updated Oct 1 21:29:37.684
local adjacency 130.130.1.2
Prefix Len 32, traffic index 0, Adjacency-prefix, precedence routine (0)
via 130.130.1.2, GigabitEthernet0/1/0/0, 3 dependencies, weight 0, class 0 [flags 0x0]
path-idx 0
next hop 130.130.1.2
local adjacency
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 6-127
Chapter 6 Troubleshooting Layer 3 Connectivity
Traffic Loss
IP Leaf Data:
as:0 prefix_len:32
for_us:0x0 dft_route:0x0
real_intf:0x0 free1: 0x0
hw_use_only: 0x0
lspa_ptr: 0x0 oce_chain_p: 0x8d8274d0
extre_fib_data_ptr: 0x8d8dc0d0
Traffic Loss
This section provides steps for troubleshooting traffic loss.
Step 1 Check for packet loss by examining transmitted packets on the local router and the receive packets on
the destination router.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
6-128 OL-23591-02
Chapter 6 Troubleshooting Layer 3 Connectivity
Packets Are Punted and Switched in Software
Step 2 View the hardware data structures involved with the prefix (destination-ip)/(mask). Verify that the RIB
table is consistent with the information that the IGP learned from neighbors. that the CEF tables are
consistent with the RIB. For routes that are learned (not directly connected), the CEF table in the RSP
should be the same as the CEF table in the LC.
show cef {ipv4} [destination ip | destination-ip/mask] hardware egress detail location
node-id
a. If an ARP entry does not exist or is incomplete, add a static ARP entry. Ensure that the Tx adjacency
points to ‘COMPLETE’.
b. If so, then it means the issue is that of ARP entry not getting updated. Troubleshooting should now
focus on why the ARP entry is not getting added (this includes steps like show arp, show arp idb,
show adjacency gig node-id detail location node-id, show arp trace, and so forth).
c. If the Tx adjacency still points to ‘PUNT’, it means ARP is adding the entry in its database, but
fib_mgr fails to mark the adjacency as ‘COMPLETE’.
d. This could be a fib_mgr, ARP, or AIB problem. Delete and reconfigure the static ARP entry with
AIB and CEF debugs on. The debugs show if ARP is adding the entry inside the AIB and if the AIB
is informing fib_mgr.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 6-129
Chapter 6 Troubleshooting Layer 3 Connectivity
Traceroute Fails
Step 3 Packets could be dropped in the fabric. To verify this, view the fabric counters.
Workaround
Step 1 Use the shut command (followed by commit) and the no shut command (followed by commit) on the
outgoing interface.
Step 2 Add a static ARP entry for the destination IP.
Traceroute Fails
Use traceroute to verify the connectivity to a destination. When traceroute fails to a destination, use
the following commands:
• show cef {ipv4} {destination_ip}/(mask} hardware egress detail location node-id—View
the hardware data structures involved with the prefix.
• show interface location {outgoing_interface} accounting—View input and output packets
from the outgoing interface.
Step 1 Check if the destination IP address has the proper transmit adjacency. See the ‘Tx Adjacency’ state (it
should be ‘COMPLETE’).
RP/0/RSP0/CPU0:router# show cef {ipv4} prefix hardware egress detail location node-id
Step 2 If the transmit adjacency is not complete, there is an issue. If it is pointing to ‘PUNT’, that means
probably the mac-address corresponding to the destination IP has not been learned. Try adding a ‘static
arp’ entry and see if transmit adjacency moves to ‘COMPLETE’. If the destination IP is advertised by a
routing protocol such as OSPF, then the transmit adjacency should never show as ‘PUNT’.
If the transmit adjacency is shown as ‘DROP’, that means there is a static route to the destination IP
explicitly pointing the route to a DROP.
If the transmit adjacency is shown as ‘COMPLETE’, it means there is no problem in the hardware chains
that are set up. You should see the counters.
Step 3 See if the output packets are equal to the traceroute packets sent.
Workaround
Step 1 Use the shut command (followed by commit) and the no shut command (followed by commit) on the
outgoing interface.
Step 2 Add a static ARP entry for the destination IP.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
6-130 OL-23591-02
Chapter 6 Troubleshooting Layer 3 Connectivity
Adding Routes Fails
Note The sample commands in this section are applicable to Ethernet LCs, not SIP-700 LCs.
Step 1 Determine if any resources are experiencing problems. View the state of various data structures. Ideally
the state should be GREEN. If it is either YELLOW or RED, it indicates an OOR condition.
RP/0/RSP0/CPU0:router# show cef resource location node-id
Step 2 Determine which hardware table is OOR. Compare ‘max entries’ and ‘used entries’ too see which of the
data structures is using the entries close to the max limit.
RP/0/RSP0/CPU0:router# show cef platform resource location node-id
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 6-131
Chapter 6 Troubleshooting Layer 3 Connectivity
Adding Routes Fails
Step 3 After determining which data structure is OOR, verify if it is expected or unexpected. Usually, for each
LEAF (either IPv4), it requires four entries of NR_LDI structure. So if you find the NR_LDI structure
going OOR, see if you have appropriate number of IP LEAFs to take this NR_LDI number to such a
limit.
Step 4 If show cef resource location node-id shows the state in GREEN, it means that the problem is not
caused by an OOR condition. The reason for not being able to add further routes is some thing else.
Enable the following debugs to observe what is happening:
• RP/0/RSP0/CPU0:router# debug cef errors location node-id
• RP/0/RSP0/CPU0:router# debug cef {ipv4} error location node-id
• If you observe any tracebacks, decode the tracebacks by using SBT tool.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
6-132 OL-23591-02
Chapter 6 Troubleshooting Layer 3 Connectivity
Continuous Tracebacks
Step 5 View platform ltrace errors for protocols IPv4—show cef platform trace {ipv4} error reverse
location node-id.
Step 6 View platform ltrace common errors for all protocols—show cef platform trace common error
reverse location node-id.
Workaround
Continuous Tracebacks
When tracebacks appear continuously on the console (typically every 15 seconds), programming of the
entry inside the hardware is not successful. This causes the software to try repeatedly after every 15
seconds. It is possible that the layer just above the hardware or the hardware itself is not up and running.
Step 1 View all platform ltrace common messages. Verify that both CPPs are in ACTIVE_SOLO state.
Example
show controllers pse qfp system state location 0/1/CPU0
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 6-133
Chapter 6 Troubleshooting Layer 3 Connectivity
fib_mgr Does Not Come Up During LC Reload or After Multiple Process Restarts
Step 2 View all platform ltrace protocol messages for IPv4 or IPv6.
show cef platform trace {ipv4 | ipv6} all reverse location node-id
Step 3 Check that the NP provisioning layer (or PRM) is up. PRM is a layer just above hardware. If PRM is
down, no entry is programmed in hardware, indicating that NP may have had a problem during
initialization.
Workaround
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
6-134 OL-23591-02
Chapter 6 Troubleshooting Layer 3 Connectivity
CEF Entries Out of Sync
• show cef platform trace common all reverse location node-id—View platform ltrace common
messages.
• show cef platform trace common event reverse location node-id—View platform ltrace
common events.
• show cef platform trace {ipv4 | ipv6 | mpls} error reverse location node-id—View
platform ltrace error messages recorded for protocols IPv4, IPv6, or MPLS.
• show cef trace all reverse location node-id—View all CEF ltrace messages.
Step 1 Use the show controllers NP summary location and show controllers NP drvlog location commands
to determine if either the PRM or the underlying NP has a problem. If so, the fib_mgr will not come up.
Troubleshoot at the PRM layer or NP layer.
Step 2 If both CPPs are in ACTIVE_SOLO state, the problem is likely a software bug. In this case, collect the
core file and decode the tracebacks using the SBT tool. From root of the workspace, use ./util/bin/sbt -p
(process_name) -f (log_file).
Workaround
Step 1 Look for a default route 0.0.0.0/0 configured to go out through the management interface.
Step 2 Look for a static ARP configured for the prefix in question. It is possible that ARP is installing two
entries through both the management interface and also through the LC interface (because the prefix is
reachable by both routes).
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 6-135
Chapter 6 Troubleshooting Layer 3 Connectivity
fib_mgr Crashes
Step 3 If the above is not the case, use the show arp command to see if an ARP entry is advertising through the
management interface. If this is the case, clear the ARP and verify the cef entries again.
Workaround
• Use the shut command (followed by commit) and the no shut command (followed by commit) on
the management interface.
• Use the clear arp-cache command.
• Reboot the LC.
fib_mgr Crashes
• show cef platform trace common all reverse location node-id—View CEF platform common
traces.
• show cef platform trace common event reverse location node-id—View CEF platform
common event traces.
• show cef platform trace common error reverse location node-id—View CEF platform
common error traces.
• show cef platform trace {ipv4 | ipv6 | mpls} error reverse location node-id—View CEF
platform protocol traces for IPv4 or MPLS.
Workaround
Tracebacks Appearing
In this scenario, a few error tracebacks appear on the console because of some trigger (such as interface
shut/no shut, or any other similar trigger).
• show cef trace event location node-id—View CEF traces for major events.
• show cef trace errors location node-id—View CEF traces for major errors.
• show cef platform trace common errors location node-id—View CEF platform traces for
common errors across all protocols.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
6-136 OL-23591-02
Chapter 6 Troubleshooting Layer 3 Connectivity
Traffic Loss Because of Changing encap on a Subinterface
• show cef platform trace {ipv4 | ipv6 | mpls} errors location node-id—View CEF platform
traces for errors in protocols IPv4 or MPLS.
• show logging
Step 1 Decode the tracebacks using the SBT tool. From root of the workspace, use ./util/bin/sbt -p
(process_name_ -f (log_file).
Step 2 Save core files.
Workaround
Step 1 Restart the fib_mgr process and check whether that reduces the tracebacks.
Step 2 If the tracebacks continue, reboot the LC.
When encapsulation changes from dot1q vlan 300 to dot1q vlan 200 on the subinterface, fib_mgr
deletes all prefixes corresponding to this interface and creates them again. It takes 15 seconds to add all
prefixes; traffic does not get forwarded for that time. For example, there is an interface with address
192.0.2.0/8. There is a static ARP entry for 192.0.2.5.
RP/0/RSP0/CPU0:router# show run | inc arp
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 6-137
Chapter 6 Troubleshooting Layer 3 Connectivity
Traffic Loss during RSP Failover
The delay is less likely to happen with regular adjacency (not the static ARP).
When VLAN color changes, the following occurs:
• Adjacency is deleted, and the adjacency route 192.0.2.5 is deleted.
• Connected route is deleted.
• Adjacency is added before the connected route is added. The FIB treats adding an adjacency without
a covering connected route as an error, so the route 192.0.2.5 is placed in retry.
• Connected route 192.0.2.0/8 is added.
• Because the FIB retry timer is 15 seconds, the adjacency route 192.0.2.5 is added after 15 seconds.
Workaround
Workaround
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
6-138 OL-23591-02
Chapter 6 Troubleshooting Layer 3 Connectivity
Troubleshooting Virtual Router Redundancy Protocol
DETAILED STEPS
Example:
RP/0/0/CPU0:# show vrrp brief
Step 2 show vrrp interface type interface-id detail View detailed information of VRRP groups.
Example:
RP/0/0/CPU0:# show vrrp gigabitEthernet 0/1/0/1
detail
Step 3 show vrrp [interface {type interface-id}] View VRRP statistics.
statistics [all]
Example:
RP/0/0/CPU0:# show vrrp statistics
Step 4 show controllers type interface-id View the VRRP group MAC addresses as part of unicast
filter list.
Example:
RP/0/0/CPU0:# show controllers gigabitEthernet
0/3/0/9
Step 5 debug vrrp [ all | edm | events | packets | Debug the VRRP.
packets ]
Example:
RP/0/0/CPU0:# debug vrrp packets tengige
0/3/0/9
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 6-139
Chapter 6 Troubleshooting Layer 3 Connectivity
Troubleshooting Virtual Router Redundancy Protocol
Misconfiguration
If preemption is enabled and this router has higher operational priority than the other router, this router
remains in the Active state. Configured priority or the decrement for tracked interfaces needs to be
configured appropriately such that the state transition takes place. If the IP address is the same as the
interface IP address, the router does not change to the Standby state.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
6-140 OL-23591-02
Chapter 6 Troubleshooting Layer 3 Connectivity
Troubleshooting Virtual Router Redundancy Protocol
Check timestamps to determine whether there is a delay in sending or receiving packets. Check the CPU
usage to see if some process is hogging the system resources.
Step 3 RP/0/RSP0/CPU0:router# show spp node-counters location interface-running-vrrp
Step 2 Check timestamps to determine whether there is a delay in sending or receiving packets.
Check the CPU usage to see if a process is overusing resources.
Step 3 Enter the debug command for VRRP packets on the peer.
Check for lines similar to: RP/0/RSP0/CPU0:Sep 8 14:16:39.217 : vrrp[357]: Gi0/5/0/0: VR1:
Pkt: ADVER: IN: pri 100 src 192.0.2.11. This means advertisement packets are being received by
VRRP. If these are absent, no packets are being received and VRRP becomes active.
Look for lines similar to: RP/0/RSP0/CPU0:Sep 8 14:18:47.876 : vrrp[357]: Gi0/5/0/0: VR1: Pkt:
ADVER: Out: pri 100 src 192.0.2.11. This means the peer is sending VRRP packets.
Step 4 Check the output of the show spp node-counters location interface-running-vrrp on both routers, and
look for packet drops.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 6-141
Chapter 6 Troubleshooting Layer 3 Connectivity
Additional Information On Routing Configuration Commands
RP/0/RSP0/CPU0:Sep 8 14:16:39.217 : vrrp[357]: Gi0/5/0/0: VR1: Pkt: ADVER: IN: pri 100
src 192.0.2.11.
Note the time lag between the no shut and the first such message seen. For that amount of time, there is
traffic loss between two routers.
Step 7 If there is no traffic flowing between two routers after a no shut event, check the STP configuration on
the Cisco ASR 9000 Series Router. Lowering the fwd delay timer might help in reducing the traffic loss.
Step 8 For preemption disabled case, if the groups still preempt after reducing the fwd delay timer, repeat Step 1
through Step 4, and find the time period of traffic loss between the two routers. The preemption can be
avoided by configuring the minimum delay to be higher than the time period of traffic loss. Minimum
delay can be configured as follows:
RP/0/RSP0/CPU0:router(config)# router vrrp interface gigabitEthernet 0/2/0/10 vrrp delay
minimum 10 reload 5
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
6-142 OL-23591-02
CH A P T E R 7
Troubleshooting Router Switch Fabric and Data
Path
This chapter describes techniques to troubleshoot router switch fabric and data path. It includes the
following sections:
• Understanding Switch Fabric Architecture, page 7-143
• Getting Started with Fabric Troubleshooting, page 7-145
• Troubleshooting Packet Drops, page 7-146
• Troubleshooting RSP and LC Crashes, page 7-165
• Troubleshooting Complete Loss of Traffic, page 7-168
• Gathering Fabric Information Before Calling TAC, page 7-172
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 7-143
Chapter 7 Troubleshooting Router Switch Fabric and Data Path
Understanding Switch Fabric Architecture
Active Fabric
Switch
Fabric 0
Switch
Fabric 1
Arbiter
RSP0
Active RP
Fabric I/O Fabric I/O
(LC) (LC)
Active Fabric
Switch
Fabric 0
Switch
Fabric 1
23G fabric channels
Fabric requests
Arbiter
281342
RSP1
Standby RP
As shown in Figure 7-1, there are two fabric interface ASIC on each RSP. Each fabric interface ASIC
provides 40 GB of throughput. If one RSP is lost, the shelf can still operate at full capacity without loss
of bandwidth.
Each line card (LC) has four 23 GB fabric channels on which to send traffic to the fabric ASICs. The
switch fabric is in an active/active relationship. All four fabric ASICs are active, even though the RSP
cards are in an active/standby relationship. The system performs load balancing on unicast traffic across
these four channels.
The arbiters are in an active/standby relationship (the arbiter on the active RSP card is the active arbiter).
Both the active and standby arbiters receive requests for switch fabric access from the LCs. If there is a
switchover of the active RSP, the standby RSP arbiter has a current copy of switch fabric requests, which
helps to speed up the switchover.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
7-144 OL-23591-02
Chapter 7 Troubleshooting Router Switch Fabric and Data Path
Getting Started with Fabric Troubleshooting
Figure 7-2 shows the data path from ingress to egress. (Several types of LCs are shown in this example.)
RSP0 RSP1
Fabric I/O CPU Fabric I/O CPU0
Fabric Fabric
System Fabric Fabric GE System Fabric Fabric GE
Timing arbiter Switch Timing arbiter Switch
Backplane
NPU NPU NPU NPU NPU NPU NPU NPU NPU NPU NPU NPU
10 GE XFP
10 GE XFP
10 GE XFP
10 GE XFP
10 GE XFP
10 GE XFP
10 GE XFP
10 GE XFP
10 GE XFP
10 GE XFP
10 GE XFP
10 GE XFP
10x 10x 10x 10x
280888
SFP SFP SFP SFP
As shown in the drawing, the path travelled by each data packet is:
Incoming interface on LC--> NP mapped to incoming interface on LC --> Bridge3 on LC --> FIA
on LC --> Crossbar switch on RSP --> FIA on LC ---> Bridge3 on LC ---> NP mapped to outgoing
interface ---> Outgoing Interface
Note In this document, the network processor ASICs are referred to either as network processors (NPs) or
network processor units (NPUs).
Step 1 Look for active platform fault manager (PFM) alarms on the LCs and RSPs.
Step 2 Check that you have the appropriate version of the bridge field-programmable gate arrays (FPGAs) in
your RSP card.
Step 3 Check that you have the correct software version, board, and FPGA and ASIC versions.
RP/0/RSP0/CPU0:router# show version
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 7-145
Chapter 7 Troubleshooting Router Switch Fabric and Data Path
Troubleshooting Packet Drops
Step 4 Check if there are any errors detected by the system diagnostics.
RP/0/RSP0/CPU0:router# show diag
Step 5 Check that you have the appropriate version of the NPs in your RSP cards.
RP/0/RSP0/CPU0:router# show controllers np summary all
Node: 0/1/CPU0:
----------------------------------------------------------------
[total 4 NP] Driver - Version 10.26a Build 9 ( Dec 13 2008, 20:47:03 )
NP 0 : Hardware rev v2 A1
: Ucode - Version: 255.255 Build Date: ( Dec 12 2008, 2:13:00 )
NP 1 : Hardware rev v2 A1
: Ucode - Version: 255.255 Build Date: ( Dec 12 2008, 2:13:00 )
NP 2 : Hardware rev v2 A1
: Ucode - Version: 255.255 Build Date: ( Dec 12 2008, 2:13:00 )
NP 3 : Hardware rev v2 A1
: Ucode - Version: 255.255 Build Date: ( Dec 12 2008, 2:13:00 )
Node: 0/2/CPU0: <-- [ LC built with A0 NPU that has known issue ]
----------------------------------------------------------------
[total 4 NP] Driver - Version 10.26a Build 9 ( Dec 13 2008, 20:47:03 )
NP 0 : Hardware rev v2 A0
: Ucode - Version: 255.255 Build Date: ( Dec 12 2008, 2:13:00 )
NP 1 : Hardware rev v2 A0
: Ucode - Version: 255.255 Build Date: ( Dec 12 2008, 2:13:00 )
NP 2 : Hardware rev v2 A0
: Ucode - Version: 255.255 Build Date: ( Dec 12 2008, 2:13:00 )
NP 3 : Hardware rev v2 A0
: Ucode - Version: 255.255 Build Date: ( Dec 12 2008, 2:13:00 )
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
7-146 OL-23591-02
Chapter 7 Troubleshooting Router Switch Fabric and Data Path
Troubleshooting Packet Drops
PHY NPU-0
Bridge
FPGA 0
PHY NPU-1
Fabric I/O To RSP
PHY NPU-2 fabric I/O
Bridge
FPGA 1
PHY NPU-3
Line Card
281343
show controllers fabric fia drops <ingress | egress> location <...>
show controllers fabric fia errors <ingress | egress> location <...>
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 7-147
Chapter 7 Troubleshooting Router Switch Fabric and Data Path
Troubleshooting Packet Drops
Figure 7-4 shows the traffic path on the RSP and the corresponding CLI commands you use to display
information at each point in the path.
Fabric CPU
arbiter
Fabric
XBAR 0 Fabric I/O FPGA
Fabric
XBAR 1
RSP-0
show controllers fabric crossbar serdes instance <0 or 1> location <...>
show controllers fabric crossbar statistics instance <0 or 1> location <...>
show controllers fabric Itrace crossbar all location <...>
281344
show controllers fabric fia drops <ingress | egress> location <...>
show controllers fabric fia errors <ingress | egress> location <...>
SUMMARY STEPS
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
7-148 OL-23591-02
Chapter 7 Troubleshooting Router Switch Fabric and Data Path
Troubleshooting Packet Drops
Note For the procedure to troubleshoot drops of punted packets, see the Locating Drops of Punted Packets,
page 7-155.
DETAILED STEPS
Step 4 Start the traffic pattern that caused the packet drop.
Step 5 Run the following command to display the NP-to-interface mapping.
RP/0/RSP0/CPU0:router# show controllers np ports all
Step 7 Check the NP counters to verify that traffic is flowing in NP counters along the data path.
RP/0/RSP0/CPU0:router# show controllers np counters {np0|np1|np2|np3|all} location node-id
{| include DROP}
The show controllers np command displays information about counters that helps you troubleshoot
drops in the LCs. The names of the internal NP counters have the general format
STAGE_DIRECTION_ACTION, for example, PARSE_FABRIC_RECEIVE_CNT,
RESOLVE_EGRESS_DROP_CNT, and MODIFY_FRAMES_PADDED_CNT.
The values of stage, directon, and action are as follows:
• There are five stages in the NP:
– Parse
– Search-I
– Modify
– Search-II
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 7-149
Chapter 7 Troubleshooting Router Switch Fabric and Data Path
Troubleshooting Packet Drops
– Resolve
• Examples of the direction are:
– Ingress
– Egress
– Next_hop
• Examples of the action are:
– Drop_count
– Down
There are additional counters, such as DROP, PUNT, and DIAGS, that provide important information
but are not associated with a specific internal NP stage. Drop and punt counters are kept as an aggregate
total per stage.
Example
RP/0/RSP0/CPU0:router# show controllers np ports all
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
7-150 OL-23591-02
Chapter 7 Troubleshooting Router Switch Fabric and Data Path
Troubleshooting Packet Drops
In the following example, there were some ingress and egress drops in the RESOLVE stage. All of these
drops in the ingress (9 drops) and egress (6 drops) were caused by the next hop being unreachable (a
total of 15 drops for IPv4 next hop down).
RP/0/RSP0/CPU0:router# show controllers np counters np3 location 0/0/CPU0 | include DROP
Mon Nov 15 12:18:35.289 EST
30 RESOLVE_INGRESS_DROP_CNT 9 0
31 RESOLVE_EGRESS_DROP_CNT 6 0
295 DROP_IPV4_NEXT_HOP_DOWN 15 0
The following example shows a typical output from the same command, but without the modifier
| include DROP.
RP/0/RSP0/CPU0:router# show controllers np counters np3
Mon Nov 15 12:20:35.289 EST
Node: 0/0/CPU0:
----------------------------------------------------------------
Show global stats counters for NP3, revision v3
Check the NP-bridge rx/tx counters for each NP on the LC. View the packet sent and received counts,
bytes transferred, packet counters categorized by packet size, and so forth. The fields of interest are:
xaui_a_t_transmited_packets_cnt: The number of packets sent by the NP to the bridge
xaui_a_r_received_packets_cnt: The number of packets sent by the bridge to the NP
Step 9 Check the bridge counters
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 7-151
Chapter 7 Troubleshooting Router Switch Fabric and Data Path
Troubleshooting Packet Drops
Examples
RP/0/RSP0/CPU0:router# show controllers fabric fia bridge stats location 0/RSP0/CPU0
Mon Nov 22 14:14:48.010 PST
Device Rx Interface Packet Error Threshold
Count Drops Drops
--------------------------------------------------------------------------------
Bridge0 From-Fabric(DDR) 492283 0 0
From CPU 492283 0 0
UC - Unicast , MC - Multicast
LP - LowPriority , HP - HighPriority
--------------------------------------------------------------------------------
FIA 0
******
Cast/ Packet Packet Error Threshold
Prio Direction Count Drops Drops
--------------------------------------------------------------------------------
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
7-152 OL-23591-02
Chapter 7 Troubleshooting Router Switch Fabric and Data Path
Troubleshooting Packet Drops
UC LP NP-2 to Fabric 0
UC HP NP-3 to Fabric 70329
UC LP NP-3 to Fabric 0
--------------------------------------------------
UC Total Ingress 281316
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 7-153
Chapter 7 Troubleshooting Router Switch Fabric and Data Path
Troubleshooting Packet Drops
Step 11 Check the crossbar counters to make sure there are no dropped packets.
RP/0/RSP0/CPU0:router# show controllers fabric crossbar statistics instance [0|1] location
location
Example:
RP/0/RSP0/CPU0:router# show controllers fabric crossbar statistics instance 0 location
0/RSP0/CPU0
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
7-154 OL-23591-02
Chapter 7 Troubleshooting Router Switch Fabric and Data Path
Troubleshooting Packet Drops
Asic Instance: 0
Fabric info for node 0/RSP0/CPU0 (physical slot: 4)
SUMMARY STEPS
DETAILED STEPS
Step 1 Clear all packet counters as described in the “Locating Packet Drops by Examining Counters” section
on page 7-148.
Step 2 Start traffic.
Step 3 Check traffic counters at each component in the punted packet path. Use a procedure similar to the one
described in the “Locating Packet Drops by Examining Counters” section on page 7-148. However, for
punted packets, the data path is:
Incoming Interface --> NP --> LC CPU --> NP --> Bridge3 --> LC FIA --> RSP Crossbar--> Punt
FPGA on RSP --> RSP CPU --> RSP FIA --> RSP Crossbar --> LC FIA --> LC CPU --> NP0 --->
LC FIA ---> Crossbar ---> RSP FIA ---> RSP CPU
Step 4 Check the NP counters for NP mapping to interface, and check NP0 for the inject packet count. The
following fields provide information on the NP counters:
801 PARSE_FABRIC_RECEIVE_CNT
820 PARSE_LC_INJECT_TO_FAB_CNT
872 RESOLVE_INGRESS_L2_PUNT_CNT
970 MODIFY_FABRIC_TRANSMIT_CNT
822 PARSE_FAB_INJECT_IPV4_CNT
Step 5 Check the fabric-related counters for any packet drops.
RP/0/RSP0/CPU0:router# show controllers fabric crossbar statistics instance 0 location
0/RSP0/CPU0
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 7-155
Chapter 7 Troubleshooting Router Switch Fabric and Data Path
Troubleshooting Packet Drops
Example:
RP/0/RSP0/CPU0:router# show controllers fabric fia stats location 0/5/CPU0
Examples:
RP/0/RSP0/CPU0:router# show controllers fabric fia bridge stats location 0/RSP0/CPU0
UC - Unicast , MC - Multicast
LP - LowPriority , HP - HighPriority
--------------------------------------------------------------------------------
FIA 0
******
Cast/ Packet Packet Error Threshold
Prio Direction Count Drops Drops
--------------------------------------------------------------------------------
--More--
Step 6 To check for packets punted to and injected from the LC or RP CPU, run the following commands.
RP/0/RSP0/CPU0:router# show spp interface location node-id
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
7-156 OL-23591-02
Chapter 7 Troubleshooting Router Switch Fabric and Data Path
Troubleshooting Packet Drops
Note To clear the spp counters, run the command clear spp {client | interface | node-counters}
location node-id. This command clears client statistics, interface statistics, and per-node
counters, depending on the keyword you use.
Step 7 To query the punt switch for the statistics on the LC CPU, run the following command.
RP/0/RSP0/CPU0:router# show controllers punt-switch switch-stats location node-id
Follow this procedure to locate the problem. After you locate the problem, take corrective action based
on your findings. Corrective action might include, for example, configuration updates or
hardware/software version upgrades.
Step 1 If not already done, perform the procedures in the “Getting Started with Fabric Troubleshooting” section
on page 7-145 to verify that you have the correct versions of the hardware and software.
Step 2 Collect the sync status of fabric on the LC.
show controllers fabric fia link-status location <0/1/CPU0>
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 7-157
Chapter 7 Troubleshooting Router Switch Fabric and Data Path
Troubleshooting Packet Drops
Step 5 Collect the fabric I/O/Bridge counters on both source and destination card.
show interfaces
Where to Go Next
If you have not been able to locate or correct the problem, you might be able to clear it by performing
the following steps. However, these steps might delete information that would help you perform
additional troubleshooting with Cisco Technical Support. Some of the steps involve stopping or reducing
traffic streams, which might not be appropriate on a deployed system. Consult with your network
administrator before you perform any of these steps.
Caution Before you follow these next steps, consider contacting Cisco Technical Support. Some of these steps
can cause loss of data that would be useful for future analysis and troubleshooting, or could cause loss
of traffic.
Step 1 Perform ‘reset –h’ at LC ROMMON and reboot the LC again to see if this clears the problem.
Step 2 Pull out the LC and reinsert it to see if it can boot up.
Step 3 Stop other streams of traffic to see if this failed stream can go through.
Step 4 Reduce the rate of the traffic to see if the drop continues.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
7-158 OL-23591-02
Chapter 7 Troubleshooting Router Switch Fabric and Data Path
Troubleshooting Packet Drops
Follow this procedure to locate the problem. After you locate the problem, take corrective action based
on your findings. Corrective action might include, for example, configuration updates or
hardware/software version upgrades.
Step 1 If not already done, perform the procedures in the “Getting Started with Fabric Troubleshooting” section
on page 7-145 to verify that you have the correct versions of the hardware and software.
Step 2 Collect the sync status of fabric on the linecard.
show controllers fabric fia link-status location <0/1/CPU0>
Step 5 Collect the fabric I/O/bridge counters on both RSP and LC.
show interfaces
Where to Go Next
If you have not been able to locate or correct the problem, you might be able to clear it by performing
the following steps. However, these steps might delete information that would help you perform
additional troubleshooting with Cisco Technical Support. Some of the steps involve stopping or reducing
traffic streams, which might not be appropriate on a deployed system. Consult with your network
administrator before you perform any of these steps.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 7-159
Chapter 7 Troubleshooting Router Switch Fabric and Data Path
Troubleshooting Packet Drops
Caution Before you follow these next steps, consider contacting Cisco Technical Support. Some of these steps
can cause loss of data that would be useful for future analysis and troubleshooting, or could cause loss
of traffic.
Step 1 Perform ‘reset –h’ at LC ROMMON and reboot the LC again to see if this clears the problem.
Step 2 Pull out the LC and reinsert it to see if it can boot up.
Step 3 Stop other streams of traffic to see if this failed stream can go through.
Step 4 Determine whether the drop is a single burst in the beginning or is continuous.
Step 5 Determine if the drop is associated with particular packet size.
Follow this procedure to locate the problem. After you locate the problem, take corrective action based
on your findings. Corrective action might include, for example, configuration updates or
hardware/software version upgrades.
Step 1 Perform the procedures in the “Getting Started with Fabric Troubleshooting” section on page 7-145 to
verify that you have the correct versions of the hardware and software.
Step 2 Collect the sync status of fabric on the linecard.
show controllers fabric fia link-status location <0/1/CPU0>
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
7-160 OL-23591-02
Chapter 7 Troubleshooting Router Switch Fabric and Data Path
Troubleshooting Packet Drops
show run
Step 5 Collect the fabric I/O/bridge counters on both the RSP and LC.
show interfaces
Where to Go Next
If you have not been able to locate or correct the problem, you might be able to clear it by performing
the following steps. However, these steps might delete information that would help you perform
additional troubleshooting with Cisco Technical Support. Some of the steps involve stopping or reducing
traffic streams, which might not be appropriate on a deployed system. Consult with your network
administrator before you perform any of these steps.
Caution Before you follow these next steps, consider contacting Cisco Technical Support. Some of these steps
can cause loss of data that would be useful for future analysis and troubleshooting, or could cause loss
of traffic.
Step 1 Perform ‘reset –h’ at LC ROMMON and reboot the LC again to see if this clears the problem.
Step 2 Pull out the LC and reinsert it to see if it can boot up.
Step 3 Stop other streams of traffic to see if this failed stream can go through.
Step 4 Repeat Step 1 through Step 3 to determine whether the results are reproducible.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 7-161
Chapter 7 Troubleshooting Router Switch Fabric and Data Path
Troubleshooting Packet Drops
Follow this procedure to locate the problem. After you locate the problem, take corrective action based
on your findings. Corrective action might include, for example, configuration updates or
hardware/software version upgrades.
Step 1 Perform the procedures in the “Getting Started with Fabric Troubleshooting” section on page 7-145 to
verify that you have the correct versions of the hardware and software.
Step 2 Collect the sync status of fabric on the linecard before and after the switchover.
show controllers fabric fia link-status location <0/1/CPU0>
Step 5 Collect the fabric I/O/bridge counters on both the RSP and LC.
show interfaces
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
7-162 OL-23591-02
Chapter 7 Troubleshooting Router Switch Fabric and Data Path
Troubleshooting Packet Drops
Where to Go Next
If you have not been able to locate or correct the problem, you might be able to clear it by performing
the following steps. However, these steps might delete information that would help you perform
additional troubleshooting with Cisco Technical Support. Some of the steps involve stopping or reducing
traffic streams, which might not be appropriate on a deployed system. Consult with your network
administrator before you perform any of these steps.
Caution Before you follow these next steps, consider contacting Cisco Technical Support. Some of these steps
can cause loss of data that would be useful for future analysis and troubleshooting, or could cause loss
of traffic.
Step 1 Stop other streams of traffic to see if this failed stream can go through again.
Step 2 Repeat Step 1 several times to determine if the result is reproducible.
Step 3 Perfom a switchover back to the other side to determine whether both directions are having the same
traffic problems.
Step 4 After obtaining the necessary approvals from your network and system administrators (because this step
will stop all traffic on this unit), reboot the entire system and check to see if it recovers.
Follow this procedure to locate the problem. After you locate the problem, take corrective action based
on your findings. Corrective action might include, for example, configuration updates or
hardware/software version upgrades.
Step 1 Perform the procedures in the “Getting Started with Fabric Troubleshooting” section on page 7-145 to
verify that you have the correct versions of the hardware and software.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 7-163
Chapter 7 Troubleshooting Router Switch Fabric and Data Path
Troubleshooting Packet Drops
Step 2 Collect the sync status of fabric on the linecard before and after the switchover.
show controllers fabric fia link-status location <0/1/CPU0>
Step 4 Collect the fabric I/O/bridge counters on both the RSP and LC.
show interfaces
Step 6 Check for drops on the the fabric I/O interface (FIA drop counters) on the LC in both the ingress (to
fabric) and egress (from fabric) directions.
show controllers fabric fia drops egress location
show controllers fabric fia drops ingress location
show controllers fabric fia error egress location
show controllers fabric fia error ingress location
Step 7 Check for drops on the bridge. Counters are a combination of high priority (HP), low priority (LP),
unicast, multicast, DDR, and DDR-threshold packets. They are furthur segregated into critical and
informational based on their severity. All Ethernet linecards have 2 bridges. Use the following command
to obtain this information.
show controllers fabric fia bridge stats location <linecard location>
Where to Go Next
If you have not been able to locate or correct the problem, you might be able to clear it by performing
the following steps. However, these steps might delete information that would help you perform
additional troubleshooting with Cisco Technical Support. Some of the steps involve stopping or reducing
traffic streams, which might not be appropriate on a deployed system. Consult with your network
administrator before you perform any of these steps.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
7-164 OL-23591-02
Chapter 7 Troubleshooting Router Switch Fabric and Data Path
Troubleshooting RSP and LC Crashes
Caution Before you follow these next steps, consider contacting Cisco Technical Support. Some of these steps
can cause loss of data that would be useful for future analysis and troubleshooting, or could cause loss
of traffic.
Step 1 Stop other streams of traffic to see if this failed stream can go through again.
Step 2 Reboot the LCs one at a time and check if the traffic recovers.
Step 3 After obtaining the necessary approvals from your network and system administrators (because this step
will stop all traffic on this unit), reboot the entire system and check to see if it recovers.
Step 4 Reconfigure the system to see if it recovers.
Follow this procedure to locate the problem. After you locate the problem, take corrective action based
on your findings. Corrective action might include, for example, configuration updates or
hardware/software version upgrades.
Step 1 Perform the procedures in the “Getting Started with Fabric Troubleshooting” section on page 7-145 to
verify that you have the correct versions of the hardware and software.
Step 2 Collect the sync status of fabric on the RSP card.
show controllers fabric fia link-status location <0/RSP0/CPU0>
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 7-165
Chapter 7 Troubleshooting Router Switch Fabric and Data Path
Troubleshooting RSP and LC Crashes
Where to Go Next
If you have not been able to locate or correct the problem, you might be able to clear it by performing
the following steps. However, these steps might delete information that would help you perform
additional troubleshooting with Cisco Technical Support. Some of the steps involve stopping or reducing
traffic streams, which might not be appropriate on a deployed system. Consult with your network
administrator before you perform any of these steps.
Caution Before you follow these next steps, consider contacting Cisco Technical Support. Some of these steps
can cause loss of data that would be useful for future analysis and troubleshooting, or could cause loss
of traffic.
Step 1 Perform ‘reset –h’ at LC ROMMON and reboot the RSP again to see if this clears the problem.
Step 2 Pull out the RSP and reinsert it to see if it can boot up.
Step 3 Swap the slot (put the RSP card into the other RSP slot) and see if it can boot up properly.
Follow this procedure to locate the problem. After you locate the problem, take corrective action based
on your findings. Corrective action might include, for example, configuration updates or
hardware/software version upgrades.
Step 1 If not already done, perform the procedures in the “Getting Started with Fabric Troubleshooting” section
on page 7-145 to verify that you have the correct versions of the hardware and software.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
7-166 OL-23591-02
Chapter 7 Troubleshooting Router Switch Fabric and Data Path
Troubleshooting RSP and LC Crashes
Where to Go Next
If you have not been able to locate or correct the problem, you might be able to clear it by performing
the following steps. However, these steps might delete information that would help you perform
additional troubleshooting with Cisco Technical Support. Some of the steps involve stopping or reducing
traffic streams, which might not be appropriate on a deployed system. Consult with your network
administrator before you perform any of these steps.
Caution Before you follow these next steps, consider contacting Cisco Technical Support. Some of these steps
can cause loss of data that would be useful for future analysis and troubleshooting, or could cause loss
of traffic.
Step 1 Perform ‘reset –h’ at the ROMMON and reboot the standby RSP again to see if this clears the problem.
Step 2 Pull out the RSP and reinsert it to see if it can boot up.
Step 3 Swap the slot (put the RSP card into the other RSP slot) and see if it can boot up properly.
LC Is Crashing
In this scenario, a LC keeps crashing and the RSP console shows that fia_lc (the fabric I/O process)
terminates repeatedly.
The possible causes are:
• Initialization of the LC fabric I/O fails for some reason
• Fabric self-test on the LC fails
• The synchronization between the fabric I/O and the fabric NP or fabric arbiter NP has a problem
• Communication between the LC and the RSP is not working properly
• There is a sync problem between the fabric I/O and the bridge
• Unknown failures
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 7-167
Chapter 7 Troubleshooting Router Switch Fabric and Data Path
Troubleshooting Complete Loss of Traffic
Follow this procedure to locate the problem. After you locate the problem, take corrective action based
on your findings. Corrective action might include, for example, configuration updates or
hardware/software version upgrades.
Step 1 If not already done, perform the procedures in the “Getting Started with Fabric Troubleshooting” section
on page 7-145 to verify that you have the correct versions of the hardware and software.
Step 2 Collect the sync status of the fabric on the LC.
show controllers fabric fia link-status location <0/1/CPU0>
Where to Go Next
If you have not been able to locate or correct the problem, you might be able to clear it by performing
the following steps. However, these steps might delete information that would help you perform
additional troubleshooting with Cisco Technical Support. Some of the steps involve stopping or reducing
traffic streams, which might not be appropriate on a deployed system. Consult with your network
administrator before you perform any of these steps.
Caution Before you follow these next steps, consider contacting Cisco Technical Support. Some of these steps
can cause loss of data that would be useful for future analysis and troubleshooting, or could cause loss
of traffic.
Step 1 Perform ‘reset –h’ at the LC ROMMON and reboot the LC again to see if this clears the problem.
Step 2 Pull out the LC and reinsert it to see if it can boot up.
Step 3 Swap the slot (pull out the LC and insert it into another LC slot) and see if it can boot up properly.
Step 4 Put a different LC of same type to see if that card can booting up properly.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
7-168 OL-23591-02
Chapter 7 Troubleshooting Router Switch Fabric and Data Path
Troubleshooting Complete Loss of Traffic
No Traffic from LC to LC
In this scenario, you have configured the system and the RSP and LC have come up and are stable, but
no LC-to-LC traffic is going through.
The possible causes are:
• Traffic dropped at the interface
• Traffic dropped at NP3
• Traffic dropped at the bridge
• Traffic dropped at the fabric I/O
• Sync between the fabric I/O and the fabric NP or fabric arbiter NP has a problem
• Traffic has wrong vqi
• Unknown failures
Follow this procedure to locate the problem. After you locate the problem, take corrective action based
on your findings. Corrective action might include, for example, configuration updates or
hardware/software version upgrades.
Step 1 Perform the procedures in the “Getting Started with Fabric Troubleshooting” section on page 7-145 to
verify that you have the correct versions of the hardware and software.
Step 2 Collect the sync status of fabric on the LC.
show controllers fabric fia link-status location <0/1/CPU0>
Step 5 Collect the fabric I/O/bridge counters on both the source and destination cards.
show interfaces
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 7-169
Chapter 7 Troubleshooting Router Switch Fabric and Data Path
Troubleshooting Complete Loss of Traffic
Where to Go Next
If you have not been able to locate or correct the problem, you might be able to clear it by performing
the following steps. However, these steps might delete information that would help you perform
additional troubleshooting with Cisco Technical Support. Some of the steps involve stopping or reducing
traffic streams, which might not be appropriate on a deployed system. Consult with your network
administrator before you perform any of these steps.
Caution Before you follow these next steps, consider contacting Cisco Technical Support. Some of these steps
can cause loss of data that would be useful for future analysis and troubleshooting, or could cause loss
of traffic.
Step 1 Perform ‘reset –h’ at the LC ROMMON and reboot the LC again to see if this clears the problem.
Step 2 Pull out the LC and reinsert it to see if it can boot up and carry traffic.
Step 3 Stop other streams of traffic to see if this failed stream can go through.
Step 4 Run online diagnostics to locate errors in the system. For additional information on diagnostics, see the
“Using Diagnostic Commands” section on page 1-59.
Follow this procedure to locate the problem. After you locate the problem, take corrective action based
on your findings. Corrective action might include, for example, configuration updates or
hardware/software version upgrades.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
7-170 OL-23591-02
Chapter 7 Troubleshooting Router Switch Fabric and Data Path
Troubleshooting Complete Loss of Traffic
Step 1 If not already done, perform the procedures in the “Getting Started with Fabric Troubleshooting” section
on page 7-145 to verify that you have the correct versions of the hardware and software.
Step 2 Collect the sync status of fabric on the LC.
show controllers fabric fia link-status location <0/1/CPU0>
Step 5 Collect the fabric I/O/bridge counters on both the RSP and LC.
show interfaces
Where to Go Next
If you have not been able to locate or correct the problem, you might be able to clear it by performing
the following steps. However, these steps might delete information that would help you perform
additional troubleshooting with Cisco Technical Support. Some of the steps involve stopping or reducing
traffic streams, which might not be appropriate on a deployed system. Consult with your network
administrator before you perform any of these steps.
Caution Before you follow these next steps, consider contacting Cisco Technical Support. Some of these steps
can cause loss of data that would be useful for future analysis and troubleshooting, or could cause loss
of traffic.
Step 1 Perform ‘reset –h’ at the LC ROMMON and reboot the LC again to see if this clears the problem.
Step 2 Pull out the LC and reinsert it to see if it can boot up and carry traffic.
Step 3 Pull out the RSP card and reinsert it to see if it can boot up and carry traffic.
Step 4 Stop other streams of traffic to see if this failed stream can go through.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 7-171
Chapter 7 Troubleshooting Router Switch Fabric and Data Path
Gathering Fabric Information Before Calling TAC
Step 5 Run online diagnostics to locate errors in the system. For additional information on diagnostics, see the
“Using Diagnostic Commands” section on page 1-59.
• Platform-related information
show platform
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
7-172 OL-23591-02
CH A P T E R 8
Troubleshooting MPLS Services
This chapter describes techniques to troubleshoot MultiProtocol Label Switching (MPLS) services.
MPLS carries different kinds of traffic, such as IP packets and Ethernet frames. The general flow of
packets in a Cisco ASR 9000 Aggregation Series Router is as follows:
Incoming interface => Ingress NP => Switch fabric => Egress NP => Outgoing interface.
This chapter contains the following subsections:
• Verifying MPLS PIE Activation and MPLS Configuration, page 8-173
• Troubleshooting Connectivity Over MPLS, page 8-174
• Using show and debug Commands, page 8-174
• IP Packets Not Forwarded to LSP, page 8-175
• IP Packets Not Forwarded to MPLS TE Tunnel, page 8-176
• MPLS Packets Not Forwarded to MPLS TE Tunnel, page 8-176
• MPLS TE Tunnels Do Not Come Up, page 8-176
• FRR-Protected Tunnel Goes Down After Triggering FRR, page 8-177
• MPLS TE FRR Database Not Built, page 8-178
• MPLS FRR Switch Time Debugging, page 8-178
Caution Verify that the MPLS PIE is committed before you configure MPLS. Otherwise all of your
MPLS configuration data will be lost if the image is reloaded.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 8-173
Chapter 8 Troubleshooting MPLS Services
Troubleshooting Connectivity Over MPLS
Step 1 Ping the opposite interface (on the remote router) on the MPLS interface. Verify that the ping is
successful.
Step 2 Verify that the remote interface shows up as an ospf neighbor.
show ospf neigbor
Step 3 Verify that the remote router ID (typically the remote router loopback) is in the routing table.
show route ipv4
Step 4 Ping the IP address of the remote router (the same IP address that was displayed in Step 3). Verify that
the ping is successful.
Step 5 Verify that label distribution protocol (LDP) is up between the local and remote routers.
show mpls ldp neighbor
Step 6 Verify that you can find the ID of the remote router in an MPLS command. In the case of a PW, this ID
will be theIPv4 address for the PW.
Step 7 Verify that the BGP neighbor is up.
Step 8 If you are using PW in the core, verify that the PWs are properly configured on both PEs.
Step 9 Check that configurations are correct on all peers in the VPLS domain. This includes, for example,
loopbacks, IGP (OSPF or ISIS), LDP, BGP, and L2VPN.
Note L2VPN services rely on Layer 3 connectivity from the PE through the core. If you need to reconfigure
any routing parameters, use the procedures shown in Cisco ASR 9000 Series Aggregation Services
Router Routing Configuration Guide, Release 4.0.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
8-174 OL-23591-02
Chapter 8 Troubleshooting MPLS Services
IP Packets Not Forwarded to LSP
DETAILED STEPS
Step 3 Find out whether the ARP resolves for the next hop prefix.
RP/0/RSP0/CPU0:router# show arp prefix location node-id
Workaround
If ARP is not resolved, ping the destination and run the show arp command again to see if the ARP
resolves. If it does not resolve, it means there is no path to reach the destination, or the destination is not
operational.
If ARP is resolved, use the clear arp location node-id command to clear the ARP information on the
node. This command clears the current ARP table entries, and the system will refill the ARP entries with
the latest ARP information. This might help if there are stale and incorrect entries in the ARP table.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 8-175
Chapter 8 Troubleshooting MPLS Services
IP Packets Not Forwarded to MPLS TE Tunnel
Step 3 Check the hardware TE label FIB by running the following command on the ingress LC for the unicast
traffic.
RP/0/RSP0/CPU0:router# show cef adjacency tunnel-te tunnel-id hardware ingress location
node-id
Workaround:
Enter the shut command (followed by commit) and the no shut command (followed by commit) on the
tunnel interface to reprogram the hardware.
RP/0/RSP0/CPU0:router# show mpls forwarding labels label-id hardware egress location node-id
Step 2 Ensure that hardware tunnel adjacency is complete by running the following command on the ingress LC
for the unicast traffic.
RP/0/RSP0/CPU0:router# show cef mpls adjacency tunnel-te te-id hardware egress location
node-id
Workaround
Perform the shut command (followed by commit) and the no shut command (followed by commit) of
the tunnel interface to reprogram the hardware.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
8-176 OL-23591-02
Chapter 8 Troubleshooting MPLS Services
FRR-Protected Tunnel Goes Down After Triggering FRR
Step 2 Verify that the tunnel egress interface is up, for example:
RP/0/RSP0/CPU0:router# show interface GigabitEthernet0/4/0/8
Step 2 Ensure that the MP address is reachable. Check forwarding over the backup tunnel is working.
Step 4 Check RSVP traces to find out why the tunnel went down.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 8-177
Chapter 8 Troubleshooting MPLS Services
MPLS TE FRR Database Not Built
Step 3 Ensure that backup has enough backup bandwidth (if configured).
Step 4 Ensure that backup and protected tunnels have a merge point (check hop information).
Step 5 Ensure that protected and backup tunnels are in Up, Up state.
Note Protected tunnels with 0 signaled bandwidth cannot be protected by limited backup-bw tunnels.
Step 7 Shut and no shut the backup tunnel and/or protected tunnel (if possible). This resets backup tunnel
assignments.
Step 8 Ensure that the fast-reroute option is not configured on the backup tunnel.
Step 2 Upon FRR triggered, ensure that FRR is in the active state.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
8-178 OL-23591-02
Chapter 8 Troubleshooting MPLS Services
MPLS FRR Switch Time Debugging
Step 3 Check FRR switch time of LC that the primary tunnel is failed.
Step 4 Ensure that both primary and backup tunnels on the LC received the FRR trigger.
RP/0/RSP0/CPU0:router# show cef platform trace te all location node-id
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 8-179
Chapter 8 Troubleshooting MPLS Services
MPLS FRR Switch Time Debugging
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
8-180 OL-23591-02
CH A P T E R 9
Troubleshooting L2VPN and Ethernet Services
This chapter describes techniques to troubleshoot Layer 2 virtual private network (L2VPN) features. In
this document, L2VPN refers to a family of Layer 2 functions and Ethernet services provided by the
Cisco ASR 9000 Aggregation Series Router.
If you are experiencing a problem with L2VPN traffic, the source of the problem could be caused by any
of the following conditions:
• Interfaces in the customer edge (CE) router down or configured incorrectly.
• Interfaces in the provider edge (PE) router down or configured incorrectly.
• MAC address updates not functioning correctly.
• Bridge domain not configured correctly.
• Routing in the core network down or not configured correctly.
This chapter contains the following sections that explain how to troubleshoot these conditions:
• Troubleshooting VLAN Traffic and L2 TCAM Classification, page 9-181
• Troubleshooting Multipoint Layer 2 Services, page 9-190
• Troubleshooting Point-to-Point Layer 2 Services, page 9-206
• Troubleshooting Specific Outage Scenarios In Layer 2 Services, page 9-214
• Troubleshooting Dynamic Host Configuration Protocol Snooping, page 9-227
• Troubleshooting Multiple Spanning Tree, page 9-230
• Additional References—Command Reference and Configuration Guides, page 9-232
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 9-181
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting VLAN Traffic and L2 TCAM Classification
• When you create a subinterface for the AC (in interface config mode):
– You must include the l2transport keyword on the same command line
– You must configure an encapsulation statement
Example:
interface GigabitEthernet0/2/0/2.2 l2transport
encapsulation dot1q 100
• Review your running configuration to verify that it is complete and the necessary interfaces are up.
(show running-config).
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
9-182 OL-23591-02
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting VLAN Traffic and L2 TCAM Classification
• Ensure that the interfaces and subinterfaces for the ACs are actually up. View the up/down status of
the bridge domain, ACs, and PWs (if present) by means of the show l2vpn bridge-domain
summary command. Verify that the counts are incrementing, which means that the ACs are up.
• Make sure that bridge ports (for example, ACs and PWs) are assigned to the bridge domains.
• Verify that a unique main or subinterface is assigned to each AC in the bridge domain.
Step 1 Display the main interface state and subinterface state. (The main interface is also called the trunk
interface, and it is identified as trunk in some of the CLI commands.)
RP/0/RSP0/CPU0:router# show interface
Step 3 Display the Ethernet tags and check for any errors or mismatches. This command gives tag information
in a very concise format, if you want to check the encapsulation on multiple subinterfaces.
RP/0/RSP0/CPU0:router# show ethernet tags
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 9-183
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting VLAN Traffic and L2 TCAM Classification
Step 4 Verify that the subinterface matching order is as expected. The match-order option lists the subinterfaces
in the order that they match traffic. If the traffic is being classified to a different interface than you
expect, this command can help you determine why.
RP/0/RSP0/CPU0:router# show ethernet tags match-order
Step 5 Display the interface debug counters for each network processor unit. The following example shows the
NP counters.
RP/0/RSP0/CPU0:router# show controllers np counters {all | np0 | np1 | np2 | np3}
Step 6 If the output of the command in Step 5 shows that the UIDB_TCAM_MISS_AGG_DROP counter is
incrementing, it is possible that the physical port is receiving tagged traffic that does not match the
encapsulation statement of any subinterface. The parent/main interface is an untagged Layer 3 interface,
and rejects any tagged traffic that fails classification against any of its subinterfaces/children.
RP/0/RSP0/CPU0:router# clear controllers np counters all location node-id
RP/0/RSP0/CPU0:router# show controllers np counters {all | np0 | np1 | np2 | np3}
a. Verify that there is incoming tagged traffic that does not match the encapsulation statement of any
subinterface, and that this traffic is not needed (that is, you do not intend to configure a subinterface
to receive and forward this traffic).
– Encapsulation not matched but the traffic is needed—Create the necessary subinterface or
correct the encapsulation statement on the applicable existing subinterface.
– Encapsulation not matched, traffic not needed, and no encapsulation default currently
configured—Go to Substep b.
– Encapsulation not matched, traffic not needed, and there is an encapsulation default currently
configured—Go to Substep c.
b. Add an encapsulation default subinterface to receive all of the tagged traffic with unwanted
encapsulation statements. Check whether the UIDB_TCAM_MISS_AGG_DROP goes to zero, and
the default subinterface counters start going up. This process shifts the incrementing of counters
away from the main interface and isolates it on the default subinterface.
c. Verify that the Layer 2 encapsulation default subinterface is properly configured.
Note See the example below with the CLI statement encapsulation default.
Example
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
9-184 OL-23591-02
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting VLAN Traffic and L2 TCAM Classification
In this example, Bundle-Ether16 is the main interface (also referred to as the trunk interface or Layer 3
interface), and Bundle-Ether16.160 and Bundle-Ether16.161 are subinterfaces.
RP/0/RSP0/CPU0:router# show interfaces
Bundle-Ether16 is up, line protocol is up <<< The main interface is up
Interface state transitions: 1
Hardware is Aggregated Ethernet interface(s), address is 001b.53ff.87f0
Description: Connect to P19_C7609-S Port-Ch 16
Internet address is Unknown
MTU 9216 bytes, BW 1000000 Kbit (Max: 1000000 Kbit) reliability 255/255, txload 0/255,
rxload 0/255
Encapsulation ARPA, loopback not set,
ARP type ARPA, ARP timeout 04:00:00
No. of members in this bundle: 2
GigabitEthernet0/1/0/16 Full-duplex 1000Mb/s Active
GigabitEthernet0/1/0/17 Full-duplex 1000Mb/s Standby
Last input 00:00:00, output 00:00:00
Last clearing of "show interface" counters never
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 1000 bits/sec, 0 packets/sec
202037 packets input, 18079605 bytes, 1 total input drops <<< Includes the sum of
packets on all the subinterfaces in addition to the packets on the main interface.
5964 drops for unrecognized upper-level protocol
Received 0 broadcast packets, 202037 multicast packets
0 runts, 0 giants, 0 throttles, 0 parity
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
490241 packets output, 53719536 bytes, 0 total output drops
Output 3 broadcast packets, 490238 multicast packets
0 output errors, 0 underruns, 0 applique, 0 resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 9-185
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting VLAN Traffic and L2 TCAM Classification
interface Bundle-Ether16
description Connect to C7609-S Port-Ch 16
mtu 9216
bundle maximum-active links 1
!
interface Bundle-Ether16.160 l2transport
description Connect to C7609-S Port-Ch 16 Service Instance 160
encapsulation dot1q 160 <<< Encapsulation is correct
!
interface Bundle-Ether16.161 l2transport
description Connect to C7609-S Port-Ch 16 Service Instance 161
encapsulation dot1q 161
!
interface Bundle-Ether16.162
description Connect to C7609-S Port-Ch 16.162
ipv4 address 192.0.2.44 255.255.255.0
encapsulation dot1q 162
!
interface Bundle-Ether16.163
description Connect to C7609-S Port-Ch 16.163
ipv4 address 192.0.2.44 255.255.255.0
encapsulation dot1q 163
!
interface Loopback0
--More--
Summary 19 17 2 19 0 0
The following example shows the NP counters. For a description of how to interpret NP counter
information, see the “Displaying Traffic Status in Line Cards and RSP Cards” section on page 7-147.
Note If you want to clear counters at any time during this procedure (to make it easier to see which counters
are incrementing), use the command clear controllers np counters all location node-id.
Node: 0/0/CPU0:
----------------------------------------------------------------
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
9-186 OL-23591-02
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting VLAN Traffic and L2 TCAM Classification
This example shows that L2VPN packets are being forwarded on the interface and subinterface (if
applicable).
RP/0/RSP0/CPU0:router# show running-config l2vpn
l2vpn
bridge group BG
bridge-domain BD1
interface TenGigE0/1/0/0.0
!
interface TenGigE0/1/0/3.0
!
interface TenGigE0/1/0/4.0
!
neighbor 10.100.1.1 pw-id 2
!
!
!
!
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 9-187
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting VLAN Traffic and L2 TCAM Classification
This example shows the configuration and query of the Ethernet tags.
RP/0/RSP0/CPU0:router# show run interface gig0/0/0/0.1
Thu Oct 14 08:57:16.831 EDT
interface GigabitEthernet0/0/0/0.1 l2transport
encapsulation dot1q 1
!
In this example, 0.2 is listed before 0.1. Any traffic with outer VLAN .1Q 10, and inner tag .1Q 20 would
match Gi0/0/0/0.2.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
9-188 OL-23591-02
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting VLAN Traffic and L2 TCAM Classification
Bridge group: 189, bridge-domain: 189, id: 0, state: up, ShgId: 0, MSTi: 0
MAC learning: enabled
MAC withdraw: enabled
MAC withdraw for Access PW: enabled
Flooding:
Broadcast & Multicast: enabled
Unknown unicast: enabled
MAC aging time: 300 s, Type: inactivity
MAC limit: 4000, Action: none, Notification: syslog
MAC limit reached: no
MAC port down flush: enabled
MAC Secure: disabled, Logging: disabled
Split Horizon Group: none
Dynamic ARP Inspection: disabled, Logging: disabled
IP Source Guard: disabled, Logging: disabled
DHCPv4 snooping: disabled
IGMP Snooping profile: none
Bridge MTU: 9000
MIB cvplsConfigIndex: 1
Filter MAC addresses:
Create time: 22/09/2010 04:16:14 (2w4d ago)
No status change since creation
ACs: 2 (2 up), VFIs: 0, PWs: 0 (0 up), PBBs: 0 (0 up)
List of ACs:
AC: GigabitEthernet0/1/0/3.189, state is up
.
.
.
List of VFIs:
VFI 190
PW: neighbor 10.19.19.19, PW ID 190, state is up ( established )
PW class Use_Tu-44190, XC ID 0xfffc0003
Encapsulation MPLS, protocol LDP
PW type Ethernet, control word disabled, interworking none
PW backup disable delay 0 sec
Sequencing not set
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 9-189
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting Multipoint Layer 2 Services
This example shows how to set up an encapsulation default subinterface. in this scenario, you expect
incoming traffic on gig0/1/0/1 to be all single-tagged dot1q 100. However, you see some occasional
traffic with other encapsulations being dropped. These drops could be due to a few stray packets (for
example dot1q 200), and they are dropped without being processed on gig0/1/0/1; the
UIDB_TCAM_MISS_AGG_DROP counter is incremented. You can configure one default subinterface
to catch all the stray packets. Then the drops appear as counters on this isolated default interface, not as
UIDB_TCAM_MISS_AGG_DROP on the main interface.
interface gig0/1/0/1
mtu 1500
!
interface gig0/1/0/1.1 l2transport
encapsulation dot1q 100
!
interface gig0/1/0/1.2 l2transport
encapsulation default <=== encapsulation default
!
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
9-190 OL-23591-02
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting Multipoint Layer 2 Services
bundle-ether1
Bridge port 1
gig0/1/0/1
Bridge port 2
gig0/2/0/2.2 EFPs
Bridge domain “mybd”
EFPs
255023
The configuration for Figure 9-1 is as follows.
interface GigabitEthernet0/1/0/1
l2transport
!
interface GigabitEthernet0/2/0/2
!
interface GigabitEthernet0/2/0/2.2 l2transport
encapsulation dot1q 100
!
interface GigabitEthernet0/5/0/8
bundle id 1 mode active
!
interface GigabitEthernet0/5/0/9
bundle id 1 mode active
!
interface Bundle-Ether1
!
interface Bundle-Ether1.1 l2transport
encapsulation dot1q 100
!
l2vpn
bridge group bg_example
bridge-domain mybd
interface GigabitEthernet0/1/0/1
!
interface GigabitEthernet0/2/0/2.2
!
interface Bundle-ether1.1
!
!
!
!
Step 1 Verify that bundle members Gig0/5/0/8 and Gig0/5/0/9 are both Active, that is, that Link Aggregation
Control Protocol (LACP) indicates that they are connected with their adjacent neighbors.
RP/0/RSP0/CPU0:router# show bundle bundle-ether1
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 9-191
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting Multipoint Layer 2 Services
Note For additional information on troubleshooting bundles and LACP, see the “Troubleshooting Problems
with Link Bundles” section on page 5-118.
Step 2 Follow the steps in the “Troubleshooting VLAN Traffic and L2 TCAM Classification” section on
page 9-181 for the ACs—Gig0/1/0/1, Gig0/2/0/2, and Bundle-ether1.1.
Step 3 Display the bridge domain running configuration and ensure that it contains the appropriate commands
for your network.
RP/0/RSP0/CPU0:router# show run l2vpn bridge group bg_example
Step 4 Verify that the bridge domain, bridge ports, and ACs are all in Up state.
RP/0/RSP0/CPU0:router# show l2vpn bridge-domain bd-name mybd
Step 5 View additional details of the bridge domain, such as the feature settings and verify they are as expected.
RP/0/RSP0/CPU0:router# show l2vpn bridge-domain bd-name mybd detail
Step 1 Display the MAC address table for the bridge domain. Verify that MAC addresses are being learned and
resynced. Include the specific bridge domain and MAC address of interest, so the output will display the
specific bridge-port (AC or PW) on which the specific MAC address was learned.
RP/0/RSP0/CPU0:router# show l2vpn forwarding bridge-domain bridge-group:bridge-domain
mac-address mac-address-id location node-id
If the MAC address was learned on a PW, the output shows the IP address of the neighbor. Otherwise it
shows the MAC address of the AC.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
9-192 OL-23591-02
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting Multipoint Layer 2 Services
A bridge domain is an entity that exists on multiple LCs. However, the show command singles out one
LC. If the MAC address was learned on a bridge-port on a different LC, the display output reports the
LC on which it was learned—not the actual bridge-port. To get the bridge-port data, rerun the command
on the actual LC on which it was learned.
Step 2 (Optional) As an alternative to the procedure in Step 1, you can run a more general command without
specifying a specific bridge domain or MAC address. However, the output could flood your terminal
screen.
Caution Before you run this command without specifying a particular bridge domain and MAC address, take
steps to limit the amount of data that can be output on your terminal screen. Otherwise the amount of
output could be extremely large.
This command displays all the MAC addresses learned on all bridge domains. As a safety mechanism,
before you enter this command, set your terminal length, for example:
RP/0/RSP0/CPU0:router# term length 20
If you need the full display, direct the output to a file, for example:
RP/0/RSP0/CPU0:router# loc 0/6/cpu0 | file disk0:bdoutput.txt
Step 3 Display the MAC table for the bridge domain and verify that the MAC address has been learned. Notice
the bridge port (the same as the attachment circuit [AC]) from which the MAC address was learned, and
whether it was learned through a pseudowire (PW).
Caution Before you run this command without specifying a MAC address ID, take steps to limit the amount of
data that can be output on your terminal screen. Otherwise the amount of output could be extremely
large.
This command displays all the MAC addresses learned on a bridge domain. As a safety mechanism,
before you enter this command, set your terminal length, for example:
RP/0/RSP0/CPU0:router# term length 20
If you need the full display, direct the output to a file, for example:
RP/0/RSP0/CPU0:router# loc 0/6/cpu0 | file disk0:bdoutput.txt
One other approach to limit the output is to run the command with a pipe filter and CTRL-C after you
see the output you want.
RP/0/RSP0/CPU0:router# show l2vpn forwarding bridge-domain bridge-group:bridge-domain
mac-address detail location node-id [ | begin GigabitEthernet interface-id ]
Step 4 Use the following command to display the data for a specific bridge domain and MAC address.
RP/0/RSP0/CPU0:router# show l2vpn forwarding bridge-domain bridge-group:bridge-domain
mac-address mac-address detail location node-id
Example
RP/0/RSP0/CPU0:router# loc 0/6/cpu0 | file disk0:bdoutput.txt
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 9-193
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting Multipoint Layer 2 Services
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
9-194 OL-23591-02
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting Multipoint Layer 2 Services
Ethernet Ethernet
(VLAN/Port/EFP) Full Mesh PWs + LDP (VLAN/Port/EFP)
208684
The VPLS network requires the creation of a bridge domain (Layer 2 broadcast domain) on each of the
PE routers. The VPLS PE device holds all the VPLS forwarding MAC tables and bridge domain
information. In addition, it is responsible for all flooding broadcast frames and multicast replications.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 9-195
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting Multipoint Layer 2 Services
Caution Verify that the MPLS PIE is committed before you configure MPLS. Otherwise all of your MPLS
configuration data will be lost if the image is reloaded.
PWs operate over the MPLS network, therefore, MPLS connectivity is a prerequite for bringing up a PW.
To verify MPLS connectivity, see the “Troubleshooting Connectivity Over MPLS” section on
page 8-174.
Step 1 Check for the following underlying problems, which can cause failure of the multipoint Layer 2 services.
• The bridge domain uses an attachment circuit (AC) for which the interfaces have not been created.
• The AC interface for the bridge domain is operationally down.
• The AC interface for the bridge domain is administratively down.
• The AC is not configured as Layer 2 (the l2transport keyword is missing from the configuration
command).
• The traffic on the AC interface is not classified properly (wrong encapsulation statement).
• There is an MTU mismatch between the local and remote routers.
Step 2 Verify that you can ping the opposite interface (on the remote router) from the MPLS interface.
Step 3 Verify that the remote interface shows up as an ospf neighbor.
show ospf neighbor
Step 4 Verify that the remote router ID, typically the remote router loopback, is in the routing table.
show route ipv4
Step 5 Ping the remote router with the same IP address that is used for the PW (ping x.x.x.x).
Step 6 Verify that you can find the remote router ID in an MPLS command. It should be the ipv4 address for
the PW.
Step 7 Verify that the BGP neighbor is up. (This step is necessary only if BGP autodiscovery has been
configured.)
show bgp neighbors
Step 8 Verify that the VFI is advertized in both PEs, and that PWs are established.
show l2vpn bridge-domain [brief | detail]
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
9-196 OL-23591-02
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting Multipoint Layer 2 Services
Step 10 Verify that PWs are bound in the Layer 2 forwarding information base (L2FIB) with the proper
cross-connect ID.
show l2vpn forwarding detail location
Example
The following example shows that autodiscovery is on, the PW is up, and NLRIs have been received from
the peer router. Check the cross-connect ID. Check the local and remote label and compare with the label
binding in the MPLS label switching database (LSD) by means of the show mpls forwarding command.
In this example, the local MPLS label ID is 16005.
RP/0/RSP0/CPU0:router# show l2vpn bridge-domain detail
Bridge group: bg1, bridge-domain: bg1_bd1, id: 0, state: up, ShgId: 0, MSTi: 0
MAC learning: enabled
MAC withdraw: enabled
Flooding:
Broadcast & Multicast: enabled
Unknown unicast: enabled
MAC aging time: 300 s, Type: inactivity
MAC limit: 4000, Action: none, Notification: syslog
MAC limit reached: no
MAC port down flush: enabled
Security: disabled
Split Horizon Group: none
DHCPv4 snooping: disabled
IGMP Snooping profile: none
Bridge MTU: 1500
ACs: 1 (1 up), VFIs: 1, PWs: 2 (2 up), PBBs: 0 (0 up)
List of ACs:
AC: GigabitEthernet0/6/0/1.1, state is up
Type VLAN; Num Ranges: 1
VLAN ranges: [2, 2]
MTU 1504; XC ID 0x2040001; interworking none
MAC learning: enabled
Flooding:
Broadcast & Multicast: enabled
Unknown unicast: enabled
MAC aging time: 300 s, Type: inactivity
MAC limit: 4000, Action: none, Notification: syslog
MAC limit reached: no
MAC port down flush: enabled
Security: disabled
Split Horizon Group: none
DHCPv4 snooping: disabled
IGMP Snooping profile: none
Storm Control: disabled
Static MAC addresses:
Statistics:
packets: received 5650000, sent 5650000
bytes: received 429400000, sent 429400000
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 9-197
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting Multipoint Layer 2 Services
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
9-198 OL-23591-02
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting Multipoint Layer 2 Services
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 9-199
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting Multipoint Layer 2 Services
The following example shows that BGP is connected and active, and that there are VPNs and NLRIs on
the bridge domain.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
9-200 OL-23591-02
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting Multipoint Layer 2 Services
The following example shows that the local router ID is advertised and that NLRIs are recieved from the
remote peers.
RP/0/RSP0/CPU0:router# show l2vpn discovery
Service Type: VPLS, Connected
List of VPNs (1 VPNs):
Example
####Sample Configuration from WEST:
####CONFIGURE LOOPBACKs and Links
Interface loopback0
Ipv4 address 10.10.10.10 255.255.255.255
!
Interface gig0/6/0/1.1 l2transport
Description Attachment Circuit connected to Customer site
Encapsulation dot1q 2
!
Interface gig0/6/0/21
Description Connected to EAST Node
Ipv4 address 10.0.0.1 255.255.255.0
!
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 9-201
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting Multipoint Layer 2 Services
Interface gig0/6/0/3
Description Connected to CENTRAL Node
Ipv4 address 192.0.2.1 255.255.255.0
!
####CONFIGURE IGP
Router ospf 1
Router-id 10.10.10.10
Nsr
Nsf cisco
Area 0
interface loopback0
interface gig0/6/0/3
interface gig0/6/0/21
####CONFIGURE BGP
Router bgp 1
bgp router-id 10.10.10.10
bgp graceful-restart
address-family ipv4 unicast
address-family l2vpn vpls-vpws <<< This shows you have configured this family in BGP so
it will be able to handle the discovery of the neighbor.
!
neighbor 192.0.2.20
remote-as 1
update-source loopback0
address-family ipv4 unicast
address-family l2vpn vpls-vpws
neighbor 172.30.30.30
remote-as 1
update-source loopback0
address-family ipv4 unicast
address-family l2vpn vpls-vpws
####CONFIGURE L2VPN
l2vpn
bridge group bg1
bridge-domain bg1_bd1
interface gig0/6/0/1.1
!
vfi bg1_bd1_vfi
vpn-id 101
autodiscovery bgp
rd 101:1
route-target 101:1
signaling-protocol ldp
vpls-id 1:101
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
9-202 OL-23591-02
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting Multipoint Layer 2 Services
Example
RP/0/RSP0/CPU0:router# show l2vpn atom-db
Wed Apr 14 23:28:41.905 EDT
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 9-203
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting Multipoint Layer 2 Services
AS Number: 1
VPLS-ID: 1:101
L2VPN Router ID: 10.10.10.10
PW: neighbor 192.0.2.20, PW ID 1:101, state is up ( established ) <<< PW is up
PW class not set, XC ID 0xfffc0001
Encapsulation MPLS, Auto-discovered (BGP), protocol LDP
PW type Ethernet, control word disabled, interworking none
PW backup disable delay 0 sec
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
9-204 OL-23591-02
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting Multipoint Layer 2 Services
Step 3 Verify the configuration of MPLS forwarding and Label Switching Database (LSD) parameters.
a. show mpls forwarding
b. show mpls lsd forwarding
Example
RP/0/RSP0/CPU0:router# show mpls forwarding
Wed Apr 14 23:41:49.325 EDT
Local Outgoing Prefix Outgoing Next Hop Bytes
Label Label or ID Interface Switched
------ ----------- ------------------ ------------ --------------- ------------
16000 Pop 192.0.2.20/32 Gi0/6/0/21 10.0.0.2 226000292
16001 Pop 172.30.30.30/32 Gi0/6/0/3 192.0.2.2 0
16002 Pop 172.16.0/24 Gi0/6/0/3 192.0.2.2 0
16003 16003 192.168.40.40/32 Gi0/6/0/3 192.0.2.2 226000620
16004 Unlabelled 10.0.1.253/32 Mg0/RSP0/CPU0/0 10.2.0.4 0
16005 Pop PW(192.0.2.20:2814754062073957) \ <<< PW has label and traffic is
running
BD=0 point2point 214700000
16006 Pop PW(192.168.40.40:2814754062073957) \ <<< PW has label and traffic is
running
BD=0 point2point 214700000
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 9-205
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting Point-to-Point Layer 2 Services
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
9-206 OL-23591-02
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting Point-to-Point Layer 2 Services
Figure 9-3 Example of Deployment with Bridge Domain and XConnect Joined by Pseudowire
Pseudowire
gig0/1/0/1 gig0/2/0/1
gig0/1/0/2
gig0/1/0/3 gig0/2/0/2
Bridge port #2
gig0/1/0/2.6
Bridge domain
“mybd” gig0/3/0/1 gig0/3/0/2
Router3
Router1 Router2
281922
MPLS/OSPF router ID MPLS/OSPF router ID
(loopback) 10.1.1.1 (loopback) 10.2.2.2
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 9-207
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting Point-to-Point Layer 2 Services
Router2
interface GigabitEthernet0/2/0/1
!
interface GigabitEthernet0/2/0/1.7 l2transport
encapsulation dot1q 100
!
interface GigabitEthernet0/2/0/2
ipv4 address 10.0.23.1 255.255.255.0
!
interface Loopback0
ipv4 address 10.2.2.2 255.255.255.255
!
router ospf 1
log adjacency changes
router-id 10.2.2.2
area 0
interface GigabitEthernet0/2/0/2
!
interface Loopback0
!
!
!
mpls ldp
router-id 10.2.2.2
log
neighbor
!
interface GigabitEthernet0/2/0/2
!
!
l2vpn
xconnect group examples
p2p myxc
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
9-208 OL-23591-02
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting Point-to-Point Layer 2 Services
interface GigabitEthernet0/2/0/1.7
!
neighbor 10.1.1.1 pw-id 1
!
!
!
Router 3
interface GigabitEthernet0/3/0/1
ipv4 address 10.0.13.2 255.255.255.0
!
interface GigabitEthernet0/3/0/2
ipv4 address 10.0.23.2 255.255.255.0
!
interface Loopback0
ipv4 address 10.3.3.3 255.255.255.255
!
router ospf 1
log adjacency changes
router-id 10.3.3.3
area 0
interface GigabitEthernet0/3/0/1
!
interface GigabitEthernet0/3/0/2
!
interface Loopback0
!
!
!
mpls ldp
router-id 10.3.3.3
log
neighbor
!
interface GigabitEthernet0/3/0/1
!
interface GigabitEthernet0/3/0/2
!
!
Use the following procedure to locate any problems with traffic flow in this network. The IP addresses
are based on the sample configurations for Routers 1, 2, and 3 (above).
Step 3 Verify that the Router1 routing table contains the loopback address of Router2 (10.2.2.2). Also verify
that the Router2 routing table contains the loopback address of Router1 (10.1.1.1).
RP/0/RSP0/CPU0:router# show route ipv4
Step 4 Verify that Router1 can ping the Router2 loopback address, and Router2 can ping the Router1 loopback
address.
• From Router1—ping 10.2.2.2
• From Router2—ping 10.1.1.1
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 9-209
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting Point-to-Point Layer 2 Services
Step 5 Verify that MPLS neighbors are established in the links (the same links listed in Step 1).
RP/0/RSP0/CPU0:router# show mpls ldp neighbor
Step 6 Verify that Router1 has an MPLS label to reach the Router2 loopback address. Also verify that Router2
has an MPLS label to reach the Router1 loopback address.
Note The output of this command contains one additional MPLS label. This additional label
represents the pseudowire between Router1 and Router2.
Step 7 Verify that the status of the Router1 bridge domain is UP, and that all all ACs are up.
RP/0/RSP0/CPU0:router# show l2vpn bridge-domain mybd
Step 9 Verify that the Router2 xconnect is UP, and all ACs are UP.
RP/0/RSP0/CPU0:router# show l2vpn xconnect group examples
1. show l2vpn xconnect [detail | group | interface | neighbor | state | summary | type | state
unresolved]
2. show l2vpn forwarding {detail | hardware | interface | location | message | resource | summary
| unresolved} location node-id
3. show mpls forwarding [detail | {label label number} | interface interface-id | labels value |
location | prefix [network/mask | length] | summary | tunnels tunnel-id]
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
9-210 OL-23591-02
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting Point-to-Point Layer 2 Services
DETAILED STEPS
Example:
RP/0/RSP0/CPU0:router# show l2vpn forwarding
location 0/2/cpu0
Step 3 show mpls forwarding [detail | {label label View the MPLS Label Forwarding Information Base (LFIB)
number} | interface interface-id | labels value entries with a local labels range.
| location | prefix [network/mask | length] |
summary | tunnels tunnel-id]
AC Is Down
Step 1 View the interface state.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 9-211
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting Point-to-Point Layer 2 Services
Pseudowire Is Down
Step 1 View the pseudowire state.
RP/0/RSP0/CPU0:router# show l2vpn xconnect neighbor
Step 2 On the MPLS-enabled interface that connects to the router at the remote end of the PW, view MPLS LDP
neighbor information. Check these conditions:
a. Ensure that, if the MPLS router-id uses a loopback interface (it usually does), the loopback interface
is present in the OSPF configuration, so that a route to its address is advertised for the other router
to reach.
b. Ensure that an LDP session is established with the PE peer.
RP/0/RSP0/CPU0:router# show mpls ldp neighbor neighbor
Step 3 Ensure that the MPLS infrastructure has allocated a label for the mpls-id IP address on the opposite
router, and an additional label for the PW tunnel itself.
RP/0/RSP0/CPU0:router# show mpls forwarding
Step 4 (Perform this step if the MPLS LSP does not come up.) On the MPLS-enabled interface that connects to
the router at the remote end of the PW, view OSPF neighbor information. Verify that the IP address of
the MPLS router ID is reachable:
a. Ensure that this IP address appears in the routing table.
b. Ping this IP address and verify that it replies successfully.
c. Ensure that the PW ID (keyword "pw-id" in the configuration syntax) is identical on both ends of
the PW.
RP/0/RSP0/CPU0:router# show ospf neighbor
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
9-212 OL-23591-02
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting Point-to-Point Layer 2 Services
Step 2 View the bridge information about Broadcast, Multicast and Unknown Unicast.
Step 3 Ensure that the MAC limit has not been exceeded.
Traffic Loss
Step 1 View the bridge domain state.
Step 3 Ensure that the bandwidth rates match between the CEs.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 9-213
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting Specific Outage Scenarios In Layer 2 Services
Step 1 One thing to check immediately is if the next hop router also experienced an FO mechanism (Similar to
what is done on this router). If so, the OSPF may go down.
Step 2 If not, verify that ‘nsf cisco’ is configured under the OSPF. If ‘nsf cisco’ is configured, see if the next
hop is reachable during FO. If not, there may be a reachability issue like a link going down or negotiation
problems.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
9-214 OL-23591-02
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting Specific Outage Scenarios In Layer 2 Services
DETAILED STEPS
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 9-215
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting Specific Outage Scenarios In Layer 2 Services
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
9-216 OL-23591-02
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting Specific Outage Scenarios In Layer 2 Services
Example
These examples show the output from the show bgp commands.
RP/0/RSP0/CPU0:router# show bgp l2vpn vpls
Status codes: s suppressed, d damped, h history, * valid, > best
i - internal, r RIB-failure, S stale
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Rcvd Label Local Label
Route Distinguisher: 101:1 (default for vrf bg1:bg1_bd1)
*> 10.10.10.10/32 0.0.0.0 nolabel nolabel
*>i192.0.2.20/32 192.0.2.20 nolabel nolabel
*>i192.168.40.40/32 192.168.40.40 nolabel nolabel
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 9-217
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting Specific Outage Scenarios In Layer 2 Services
This example shows the output from the show l2vpn discovery command.
RP/0/RSP0/CPU0:router# show l2vpn discovery private
Service Type: VPLS, Connected
List of VPNs (1 VPNs):
VPLS-ID: 1:101
Local L2 router id: 10.10.10.10
List of Remote NLRI (2 NLRIs):
Local Addr Remote Addr Remote L2 RID Time Created
--------------- --------------- --------------- -------------------
10.10.10.10 192.0.2.20 192.0.2.20 04/14/2010 23:10:51
10.10.10.10 192.168.40.40 192.168.40.40 04/14/2010 23:19:06
AC Is Down
Step 1 RP/0/RSP0/CPU0:router# show interface
Step 2 RP/0/RSP0/CPU0:router# show l2vpn bridge interface detail
Step 3 Ensure that the AC interface has l2transport configured.
Step 4 Ensure that the AC interface is up.
Step 5 Ensure that the MTUs match.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
9-218 OL-23591-02
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting Specific Outage Scenarios In Layer 2 Services
Pseudowire Is Down
A pseudowire (PW) is both an L2VPN component and an MPLS component. If a PW is down in one
node, it could be caused by a problem in the local or remote node. Follow these steps to troubleshoot a
problem with a PW in an L2VPN network.
Note For PW troubleshooting in point-to-point networks, see the “Troubleshooting Point-to-Point Layer 2
Services” section on page 9-206.
Step 1 Check the configuration is valid (show run l2vpn, show run bgp, show run mpls ldp).
Step 2 Verify that L2VPN discovery shows the received NLRI (show l2vpn discovery). If the NLRI is not
received, follow the procedure in the “L2VPN Discovery Not Working” section on page 9-217.
Step 3 View the local and remote labels in the bridge-domain (show l2vpn bridge-domain detail) and compare
these labels with the label binding in LSD (show mpls lsd forwarding labels). See the example below.
Step 4 View OSPF neighbor information.
RP/0/RSP0/CPU0:router# show ospf neighbor
Step 7 If PWs are involved, be sure they are properly configured on both PEs. See the “Troubleshooting
Point-to-Point Layer 2 Services” section on page 9-206.
Step 8 Ensure that the MPLS package is installed.
Step 9 Ensure that the core interface is up.
Step 10 Ensure that an IGP (for example OSPF) is up.
Step 11 Ensure that an LDP session is established with the PE peer.
Step 12 Ensure that the MTUs match.
RP/0/RSP0/CPU0:router# show l2vpn bridge-domain detail
Example
These commands allow you to view the local and remote labels in the bridge-domain and compare them
with the label binding in LSD.
RP/0/RSP0/CPU0:router# show l2vpn bridge-domain detail
Signaling protocol: LDP
AS Number: 1
VPLS-ID: 1:101
L2VPN Router ID: 10.10.10.10
PW: neighbor 192.0.2.20, PW ID 1:101, state is up ( established )
PW class not set, XC ID 0xfffc0001
Encapsulation MPLS, Auto-discovered (BGP), protocol LDP
PW type Ethernet, control word disabled, interworking none
PW backup disable delay 0 sec
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 9-219
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting Specific Outage Scenarios In Layer 2 Services
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
9-220 OL-23591-02
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting Specific Outage Scenarios In Layer 2 Services
Caution Before you run this command without specifying a MAC address ID, take steps to limit the amount of
data that can be output on your terminal screen. Otherwise the amount of output could be extremely
large.
This command displays all the MAC addresses learned on a bridge domain. As a safety mechanism,
before you enter this command, set your terminal length, for example:
RP/0/RSP0/CPU0:router# term length 20
If you need the full display, direct the output to a file, for example:
RP/0/RSP0/CPU0:router# loc 0/6/cpu0 | file disk0:bdoutput.txt
One other approach to limit the output is to run the command with a pipe filter and CTRL-C after you
see the output you want.
RP/0/RSP0/CPU0:router# show l2vpn forwarding bridge-domain bridge-group:bridge-domain
mac-address detail location node-id [ | begin GigabitEthernet interface-id ]
Step 6 View the NP counters. Capture this output for both ingress and egress line cards. For a description of
how to interpret NP counter information, see the “Displaying Traffic Status in Line Cards and RSP
Cards” section on page 7-147.
RP/0/RSP0/CPU0:router# show controllers np counters all location
Step 9 If PWs are involved, be sure they are properly configured on both PEs. See the “Troubleshooting
Point-to-Point Layer 2 Services” section on page 9-206.
Step 10 Ensure that the MPLS package is installed.
Step 11 Ensure that the core interface is up.
Step 12 Ensure that OSPF is the routing protocol.
Step 13 Ensure that an LDP session is established with the PE peer.
Step 14 Ensure that the MTUs match.
Example
These commands allow you to view the local and remote labels in the bridge-domain and compare them
with the label binding in LSD.
RP/0/RSP0/CPU0:router# show l2vpn bridge-domain detail
Signaling protocol: LDP
AS Number: 1
VPLS-ID: 1:101
L2VPN Router ID: 10.10.10.10
PW: neighbor 192.0.2.20, PW ID 1:101, state is up ( established )
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 9-221
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting Specific Outage Scenarios In Layer 2 Services
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
9-222 OL-23591-02
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting Specific Outage Scenarios In Layer 2 Services
Statistics:
packets: received 0, sent 0
bytes: received 0, sent 0
Storm control drop counters:
packets: broadcast 0, multicast 0, unknown unicast 0
bytes: broadcast 0, multicast 0, unknown unicast 0
Dynamic arp inspection drop counters:
packets: 0, bytes: 0
IP source guard drop counters:
packets: 0, bytes: 0
.
.
.
Node: 0/0/CPU0:
----------------------------------------------------------------
Show global stats counters for NP0, revision v3
The following command allows you to view the bridge domain forwarding data.
RP/0/RSP0/CPU0:router# show l2vpn forwarding bridge-domain location 0/6/CPU0
Thu Apr 15 00:15:29.581 EDT
Bridge MAC
Bridge-Domain Name ID Ports addr Flooding Learning State
-------------------------------- ------ ----- ------ -------- -------- ---------
bg1:bg1_bd1 0 3 4 Enabled Enabled UP
The following command allows you to view the bridge domain MAC details. The output from this
command can be very large, so you should limit the terminal screen output or send the data to a file.
RP/0/RSP0/CPU0:router# loc 0/6/cpu0 | file disk0:bdoutput.txt
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 9-223
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting Specific Outage Scenarios In Layer 2 Services
Step 4 View bridge information about Broadcast, Multicast and Unknown Unicast.
Step 5 Ensure that the MAC limit has not been exceeded.
Step 7 Ensure that the pseudowires (as applicable) and AC are up.
Step 8 Verify the hardware is programmed for both ACs.
Step 9 RP/0/RSP0/CPU0:router# show l2vpn forwarding interface GigabitEtherne0/5/0/2 hardware
ingress detail location node-id
Step 3 View bridge information about Broadcast, Multicast and Unknown Unicast.
Step 4 Ensure that the MAC limit has not been exceeded.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
9-224 OL-23591-02
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting Specific Outage Scenarios In Layer 2 Services
Step 6 Ensure that the pseudowires (as applicable) and AC are up.
Step 7 Verify that the hardware is programmed for both ACs.
RP/0/RSP0/CPU0:router# show l2vpn forwarding interface GigabitEtherne0/5/0/2 hardware
ingress detail location node-id
Step 2 Ensure that the hardware is programmed for both AC and PW (as applicable).
Step 3 Ensure that the destination MAC entry is programmed for the LC’s destination interface.
RP/0/RSP0/CPU0:router# show l2vpn forwarding bridge-domain mac-address location node-id
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 9-225
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting Specific Outage Scenarios In Layer 2 Services
Traffic Loss
Step 1 Determine where the packets are being dropped, view the xconnect AC interface counters and the
L2VPN counters for the PW. For information on ping procedures, see the “Troubleshooting Ping and
ARP Connectivity” section on page 3-75.
Step 2 View the bridge domain state.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
9-226 OL-23591-02
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting Dynamic Host Configuration Protocol Snooping
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 9-227
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting Dynamic Host Configuration Protocol Snooping
Show Commands
The DHCP application runs on the RSP. It has several EXEC mode CLI show commands that present the
application's configuration state, DHCP client state, and DHCP packet statistics.
• show dhcp ipv4 snoop binding—View the state of DHCP clients in a table.
• show dhcp ipv4 snoop binding mac-address macaddress—View detailed state of DHCP Clients
with the specified MAC Address.
• show dhcp ipv4 snoop binding summary—View the total number of DHCP Clients.
• show dhcp ipv4 snoop profile—View a list of DHCP snoop profiles.
• show dhcp ipv4 snoop profile name name—View details of a specific DHCP snoop profile.
• show dhcp ipv4 snoop statistics—View aggregate DHCP snoop Rx, Tx, and drop packets for
each bridge domain.
• show dhcp ipv4 snoop statistics bridge-domain name—View detailed DHCP snoop Rx, Tx,
and drop packets for each message type in a bridge domain.
Trace Commands
The DHCP application has over 1200 Trace logs. The Trace logs record significant events that occur in
the application. Trace logs that are associated with a specific DHCP client will contain the client MAC
address.
• show dhcp ipv4 trace errors—View error traces.
• show dhcp ipv4 trace events—View event traces.
• show dhcp ipv4 trace packets—View packet processing traces.
• show dhcp ipv4 trace snoop errors—View error traces for DHCP snoop feature.
• show dhcp ipv4 trace snoop events—View event traces for the DHCP snoop feature.
• show dhcp ipv4 trace snoop internal—View internal debug traces for the DHCP snoop feature.
Syslog Commands
The DHCP application has over 1600 syslog logs. These logs record events that occur in the application.
• debug dhcp ipv4 errors—View error logs.
• debug dhcp ipv4 events—View event logs.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
9-228 OL-23591-02
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting Dynamic Host Configuration Protocol Snooping
Tech-support Commands
The DHCP application has four tech-support commands that call groups of DHCP CLI commands. Use
tech-support commands for information about the DHCP application for debugging.
• show tech-support dhcp ipv4 snoop file filename
• show tech-support dhcp ipv4 snoop bridge-domain-name bridge-domain-id file
filename—View information for the specified bridge domain.
• show tech-support dhcp ipv4 snoop profile-name profilename file filename—View
information for the specified profile.
Action Commands
Use the following CLI commands to clear DHCP snoop binding states:
• clear dhcp ipv4 snoop binding—Clears all DHCP snoop client bindings.
• clear dhcp ipv4 snoop binding bridge-domain bridge-domain-name—Clears all DHCP snoop
client bindings in the specified bridge domain.
• clear dhcp ipv4 snoop binding mac-address macaddress—Clears the DHCP snoop client
bindings with the specified MAC address.
L2VPN Commands
DHCP snoop is enabled on L2VPN ACs by attaching a DHCP snoop profile to a bridge domain or AC.
The DHCP snoop trusted attribute is configured on an AC according to the value of the trusted attribute
in the DHCP snoop profile. L2VPN CLI commands are used to display the status of DHCP snoop
attributes on L2VPN bridge domains and ACs.
• show l2vpn bridge-domain bd-name bridgename detail—View the L2VPN DHCP snoop
configuration for the specified bridge domain.
• show l2vpn forwarding interface interface detail location location—View the L2VPN
DHCP snoop configuration for a specific interface.
L2Snoop Commands
L2Snoop receives and transmits DHCP snoop packets between NETIO and the DHCP snoop application
on the RSP.
show l2snoop statistics pcb all—View the L2SNOOP DHCP packet Rx/Tx statistics to and from
the DHCP snoop application on the RSP.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 9-229
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting Multiple Spanning Tree
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
9-230 OL-23591-02
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Troubleshooting Multiple Spanning Tree
Look for topology change notifications. Run the following command and look for TC 1:
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 9-231
Chapter 9 Troubleshooting L2VPN and Ethernet Services
Additional References—Command Reference and Configuration Guides
Step 1 To view the BPDUs being sent by MSTAG, run the following command.
RP/0/RSP0/CPU0:router# show spanning-tree mstag protocol-instance-id bpdu interface
interface-name
Step 2 Ensure that the forwarding state is set as it was programmed in the hardware.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
9-232 OL-23591-02
CH A P T E R 10
Troubleshooting Quality of Service and Access
Control Lists
This chapter describes techniques for troubleshooting quality of service (QoS) and access control list
(ACL) features.
The system supports the following QoS features:
• Multilevel priority scheduling for voice and video applications with minimal jitter, latency and
packet loss.
• Priority propagation to ensure service integrity for voice and video throughout all hierarchy layers,
even at peak hours with high traffic load.
• Differentiated Service Code Point (DSCP), MPLS experimental bit (EXP) and IEEE 802.1p IP
Precedence bit classification with marking, policing and scheduling, ingress and egress.
This chapter includes the following sections:
• Using show and debug Commands, page 10-234
• Service-Policy Configuration Is Rejected, page 10-235
• Packets are Incorrectly Classified, page 10-235
• Packets in Wrong Queue, page 10-236
• Packets Incorrectly Marked, page 10-236
• Packets Incorrectly Policed, page 10-237
• Shaping Incorrect, page 10-237
• Weighted Random Early Detection Incorrect, page 10-237
• Bandwidth Not Guaranteed, page 10-238
• Bandwidth Ratio Not Working, page 10-238
• Non-zero Queue(conform) and Queue(exceed) Counters In show policy-map Commands,
page 10-239
• Unable to Modify or Delete policy-map or class-map, page 10-240
• Unable to Modify or Delete class-map ACL, page 10-240
• Unable to Delete service-policy, page 10-240
• After QoS EA Restarts, show policy-map interface Fails, page 10-240
• After QoS EA Restarts, service-policy config Fails, page 10-241
• show policy-map interface Output Error, page 10-241
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 10-233
Chapter 10 Troubleshooting Quality of Service and Access Control Lists
Using show and debug Commands
DETAILED STEPS
Example:
RP/0/RSP0/CPU0:router# show run policy-map
l1-all
Step 2 show run classmap View class-map configuration with name.
Example:
RP/0/RSP0/CPU0:router# show run class-map c2
Step 3 show run interface View the service-policy binding for a given
port/subinterface.
Example:
RP/0/RSP0/CPU0:router# show run interface
g0/2/0/0
Step 4 show policy-map interface type interface-name View all the statistics, queue IDs and class information.
[output|input]
Example:
RP/0/RSP0/CPU0:router# show policy-map
interface g0/2/0/0
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
10-234 OL-23591-02
Chapter 10 Troubleshooting Quality of Service and Access Control Lists
Service-Policy Configuration Is Rejected
Example:
RP/0/RSP0/CPU0:router# show qos int g0/2/0/0
out
Step 6 show qos-ea interface type interface-name View all the class information structures.
[output|input]
Example:
RP/0/RSP0/CPU0:router# show qos-ea int g0/2/0/0
out
Step 7 show qos-ea km View the key manager (TCAM key manager) related fields
associated to a policy-map/interface binding.
Example:
RP/0/RSP0/CPU0:router# show qos-ea km policy
l2-all vmr interface g0/2/0/0 sw
Step 8 debug qos-ea ? —
Example:
RP/0/RSP0/CPU0:router# debug qos-ea ?
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 10-235
Chapter 10 Troubleshooting Quality of Service and Access Control Lists
Packets in Wrong Queue
Step 6 Verify which class the packets are actually matching. If packet fields should match different class, then
NP Microcode needs to debug this further.
Step 7 Verify if in ingress QoS lookup occurs before Layer 2 ingress rewrite and that in egress Layer 2 rewrite
occurs before QoS lookup.
Step 4 Verify the hash key for the class and hash result of the class has correct Queue ID.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
10-236 OL-23591-02
Chapter 10 Troubleshooting Quality of Service and Access Control Lists
Packets Incorrectly Policed
Step 3 Get the token bucket and police node index of the class.
Shaping Incorrect
Step 1 Ensure that packets are correctly classified.
Step 2 Verify whether shaper CIR/CBS/PIR/PBS are set correctly as per configured service-policy. Get the
shape profile ID and entity handle information (np, tm, level, index, offset).
Step 3 Verify the shaper profiles in hardware if they are correctly configured.
Step 4 RP/0/RSP0/CPU0:router# show policy-map interface filename {input | output} [member
filename]
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 10-237
Chapter 10 Troubleshooting Quality of Service and Access Control Lists
Bandwidth Not Guaranteed
Step 3 Get the WRED profile ID and entity handle information (np, tm, level, index, offset).
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
10-238 OL-23591-02
Chapter 10 Troubleshooting Quality of Service and Access Control Lists
Non-zero Queue(conform) and Queue(exceed) Counters In show policy-map Commands
In this command, CIR is the police rate value and PIR is the police peak-rate value.
Example
RP/0/RSP0/CPU0:router# show policy-map location 0/7/0/30
Class class-default
Classification statistics (packets/bytes) (rate - kbps)
Matched : 4167078/4179237900 79024
Transmitted : 2224974/2229365484 42017
Total Dropped : 1942095/1949863380 36801
Policy SHAPE-OUT Class BFD-OUT
Classification statistics (packets/bytes) (rate - kbps)
Matched : 4786/296732 5
Transmitted : 4786/296732 5
Total Dropped : 0/0 0
Policing statistics (packets/bytes) (rate - kbps)
Policed(conform) : 4786/296732 5
Policed(exceed) : 0/0 0
Policed(violate) : 0/0 0
Policed and dropped : 0/0
Policed and dropped(parent policer) : Un-determined
Queueing statistics
Queue ID : 8
High watermark (Unknown)
Inst-queue-len (packets) : 0
Avg-queue-len (Unknown)
Taildropped(packets/bytes) : 0/0
Queue(conform) : 0/0 0
Queue(exceed) : 4786/296732 5
RED random drops(packets/bytes) : 0/0
A non-zero value displayed for Queue(exceed) does not mean that there is a packet drop, but rather the
number of packets above the configured (or system selected) CIR rate on that queue. Although you could
change the Queue(exceed) behavior by explicitly configuring a bandwidth and/or a shape rate on each
queue, it is not necessary to do so. You can treat these counters as informational or simply ignore them.
In the police rate command, if you do not explicitly configure a value for the police rate (the CIR), the
system automatically assigns one. The Queue(conform) counter in the show policy-map command is
the number of packets/bytes that were transmitted within this CIR value, and the Queue(exceed) value
is the number of packets/bytes that were transmitted within the PIR value. The Queue(exceed) counter
is based on whether the parent bandwidth is exceed or conform. If there is no parent bandwidth, all
traffic is counted as excess.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 10-239
Chapter 10 Troubleshooting Quality of Service and Access Control Lists
Unable to Modify or Delete policy-map or class-map
• show running-config
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
10-240 OL-23591-02
Chapter 10 Troubleshooting Quality of Service and Access Control Lists
After QoS EA Restarts, service-policy config Fails
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 10-241
Chapter 10 Troubleshooting Quality of Service and Access Control Lists
Troubleshooting Access Control Lists
DETAILED STEPS
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
10-242 OL-23591-02
Chapter 10 Troubleshooting Quality of Service and Access Control Lists
Troubleshooting Access Control Lists
Workaround
If an entry with the fragment flag is not present, remove the access-list from all the interfaces and reapply
it.
Note Fragmented packets are not matched against the deny ACE without fragment keyword. Add the explicit
fragment keyword in the ACE to deny the fragment packet. See the workaround commands in the
“Fragmented Packets Being Accepted” section on page 10-243.
Workaround
Fragmented packets are not matched against the deny ACE without the fragment keyword. If there is
not an entry with the fragment flag, perform the following procedure.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 10-243
Chapter 10 Troubleshooting Quality of Service and Access Control Lists
Troubleshooting Access Control Lists
Step 2 Add the explicit fragment keyword in the ACE to deny the fragment packet.
Step 3 Reapply the ACL to all interfaces.
Step 2 View ARP table entries. Look for the next hop.
Workaround
Step 1 If the route is missing or the ARP is incomplete, use the no shut command to recover.
Step 2 If the UIDB table or TCAM entry is incorrect, remove the ACL from all of the interfaces and reapply it.
Workaround
If the error is related to TCAM space, remove the ACEs from the ACL. There is a limit of 64 TCAM
entries per ACL.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
10-244 OL-23591-02
Chapter 10 Troubleshooting Quality of Service and Access Control Lists
Troubleshooting Access Control Lists
Workaround
Step 1 If a field in the ACL is not supported, remove it from the ACE.
Step 2 If the TCAM is out of space, reduce the ACEs in the ACL.
Step 3 Reduce the ranges in the ACL.
Workaround
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 10-245
Chapter 10 Troubleshooting Quality of Service and Access Control Lists
Troubleshooting Access Control Lists
No Statistics Counters
Statistics counters are not supported in the current release.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
10-246 OL-23591-02
CH A P T E R 11
Troubleshooting Multicast Services
This chapter describes techniques that you can use to troubleshoot multicast services. It includes the
following sections:
• Troubleshooting IGMP Snooping (Layer 2 Multicast), page 11-247
• Troubleshooting Native Multicast Routing (Layer 3), page 11-256
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 11-247
Chapter 11 Troubleshooting Multicast Services
Troubleshooting IGMP Snooping (Layer 2 Multicast)
DETAILED STEPS
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
11-248 OL-23591-02
Chapter 11 Troubleshooting Multicast Services
Troubleshooting IGMP Snooping (Layer 2 Multicast)
Note These commands might cause a large amount of information to be displayed on your output terminal.
This command gathers information about the IGMP control packets in the system, for example, whether
JOIN and QUERY packets are being received and transmitted.
debug igmp snooping {all | error | event | init | management | packet | packet-error | proto | topo}
• all—All debugging
• error—Error debugging
• event—Event debugging
• init—Init debugging
• management—Management debugging
• packet—Packet debugging
• packet-error—Packet error debugging
• proto—Proto debugging
• topo—Topology debugging
If IGMP control packets are not being received and transmitted as expected, use this command to help
locate the cause of the problem.
debug l2snoop {call | error | events | init | packet}
• call—L2snoop function call related debugging
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 11-249
Chapter 11 Troubleshooting Multicast Services
Troubleshooting IGMP Snooping (Layer 2 Multicast)
Step 1 Check the L2VPN configuration to verify that the IGMP snooping profile is configured on the bridge
domain and optionally on one or more bridge ports. The snooping profile must be present in the bridge
domain for IGMP snooping to be enabled.
RP/0/RSP0/CPU0:router# show run l2vpn
Step 2 Verify that IGMP snooping is enabled, that is, attached to the bridge-domain and optionally to one or
more bridge ports.
RP/0/RSP0/CPU0:router# show l2vpn bridge-domain detail
Step 3 Verify that Layer 2 multicast routes (mroutes) are present in the bridge-domain.
RP/0/RSP0/CPU0:router# show l2vpn forwarding bridge-domain [bridge-domain-id] mroute ipv4
summary location node-id
Step 4 Verify the status of flood forwarding. Confirm that the forwarding state matches the snooping state.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
11-250 OL-23591-02
Chapter 11 Troubleshooting Multicast Services
Troubleshooting IGMP Snooping (Layer 2 Multicast)
Step 5 Verify that the querier is enabled in the snooped domain. Without a querier, the system drops all IGMP
reports with reason No Querier.
RP/0/RSP0/CPU0:router# show igmp snooping summary statistics
include-zeroes | include Reports No
Thu Jan 6 12:03:10.715 MEZ
Reports No Querier: 0
V3 Reports No Querier: 0
V3 Reports No Sources: 0
Tip Use variations of this command (for example, omitting the include modifier) to show other types
of errors, such as time to live errors (TTL not 1), which are useful in troubleshooting.
Step 6 Verify that the bridge domain, number of mrouters, ports, and IP addresses are as expected. Run the
snooping statistics several times to notice any trends and the corresponding locations.
RP/0/RSP0/CPU0:router# show igmp snooping bridge-domain [bridge-domain-id] detail
statistics
Step 7 Verify that packets are being received and transmitted as expected, and that there are no failed packets.
Clear the counters and rerun them several times to notice any trends and the corresponding locations.
RP/0/RSP0/CPU0:router# show l2snoop statistics pcb all location active-RSP-location
The output of this command is an aggregate of all Layer 2 snoop counters, therefore, further
investigation is needed to determine if any observed failures are related to IGMP snooping. (In this
command, PCB = protocol control block.)
Step 8 Check whether the IGMP_SNOOP packet counters are incrementing on the network processors (NPs).
If so, this indicates that the packets are being punted. Clear the counters and rerun them several times to
notice any trends and the corresponding locations.
RP/0/RSP0/CPU0:router# show controller np counters <np number or all> location <LC
location where punts are expected>
Examples
1. This example shows that the IGMP snooping profile is configured in the bridge domain.
RP/0/RSP0/CPU0:router# show run l2vpn
Tue Jan 4 09:59:49.849 PST
l2vpn
router-id 10.144.144.144
pw-class CW_enable
encapsulation mpls
control-word
!
!
xconnect group g1
p2p p1
neighbor 10.1.1.1 pw-id 100
pw-class c1
!
!
!
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 11-251
Chapter 11 Troubleshooting Multicast Services
Troubleshooting IGMP Snooping (Layer 2 Multicast)
2. This example shows that IGMP snooping profile is present in the bridge domain, therefore IGMP
snooping is enabled.
RP/0/RSP0/CPU0:router# show l2vpn bridge-domain detail
.
.
.
Bridge group: 215, bridge-domain: 215, id: 4, state: up, ShgId: 0, MSTi: 0
MAC learning: enabled
MAC withdraw: enabled
MAC withdraw for Access PW: enabled
Flooding:
Broadcast & Multicast: enabled
Unknown unicast: enabled
MAC aging time: 300 s, Type: inactivity
MAC limit: 4000, Action: none, Notification: syslog
MAC limit reached: no
MAC port down flush: enabled
MAC Secure: disabled, Logging: disabled
Split Horizon Group: none
Dynamic ARP Inspection: disabled, Logging: disabled
IP Source Guard: disabled, Logging: disabled
DHCPv4 snooping: disabled
IGMP Snooping profile: default
Bridge MTU: 9000
MIB cvplsConfigIndex: 5
Filter MAC addresses:
Create time: 24/11/2010 15:47:48 (5w5d ago)
No status change since creation
ACs: 3 (3 up), VFIs: 1, PWs: 1 (1 up), PBBs: 0 (0 up)
List of ACs:
.
.
.
3. This example shows that Layer 2 multicast routes (mroutes) are present in the bridge-domain.
RP/0/RSP0/CPU0: router# show l2vpn forwarding bridge-domain 215:215 mroute ipv4 summary
location 0/1/CPU0
Tue Jan 4 10:20:57.264 PST
Global Mroute Stats:
-----------------------------------------------------------------------------------
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
11-252 OL-23591-02
Chapter 11 Troubleshooting Multicast Services
Troubleshooting IGMP Snooping (Layer 2 Multicast)
4. This example shows that IGMP snooping is enabled and flooding is disabled.
RP/0/RSP0/CPU0:router# show l2vpn forwarding bridge-domain 215:215 detail location
0/1/CPU0
Mon Jan 3 14:49:50.332 PST
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 11-253
Chapter 11 Troubleshooting Multicast Services
Troubleshooting IGMP Snooping (Layer 2 Multicast)
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
11-254 OL-23591-02
Chapter 11 Troubleshooting Multicast Services
Troubleshooting IGMP Snooping (Layer 2 Multicast)
6. This example shows that packets are being received and transmitted, and that there are no failed
packets.
RP/0/RSP0/CPU0:router# show l2snoop statistics pcb all location 0/RSP0/CPU0
Tue Jan 4 10:48:17.423 PST
7. This example shows the IGMP_SNOOP control packet counters on the network processors (NPs).
RP/0/RSP0/CPU0:router# show controllers np counters np2 location 0/1/CPU0
Mon Jan 3 12:10:40.215 PST
Node: 0/1/CPU0:
----------------------------------------------------------------
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 11-255
Chapter 11 Troubleshooting Multicast Services
Troubleshooting Native Multicast Routing (Layer 3)
29 MODIFY_FABRIC_TRANSMIT_CNT 75222687 22
30 MODIFY_ENET_TRANSMIT_CNT 146294912 42
31 PARSE_INGRESS_DROP_CNT 17654881 1
33 RESOLVE_INGRESS_DROP_CNT 44 0
34 RESOLVE_EGRESS_DROP_CNT 407520 0
112 DIAGS 56853 0
134 IGMP_SNOOP 3 0 <<< snoop
148 IPV4MC_DO_ALL 5860 0
149 IPV4MC_DO_ALL_EXCD 13590 0
170 PUNT_IFIB 69828282 20
172 PUNT_ADJ 2 0
224 PUNT_STATISTICS 5667050 2
225 PUNT_STATISTICS_EXCD 1 0
226 PUNT_DIAGS_RSP_ACT 57337 0
230 NETIO_RP_TO_LC_CPU_PUNT 1011 0
313 BFD_NOT_ENABLED 24 0
.
.
.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
11-256 OL-23591-02
Chapter 11 Troubleshooting Multicast Services
Troubleshooting Native Multicast Routing (Layer 3)
IGMP PIM
RSP-0
NP
282276
Line-Card
SUMMARY STEPS
1. show igmp {global-interface | groups | | interface | nsf | old-output | snooping | ssm | summary
| traffic | vrf name}
2. show pim [vrf vrf-name] {bgp-safi | bsr | context | df | global | group-map | interface | ipv4 | ipv6
| join-prune | ma | mdt | mstatic | multicast | neighbor | nsf | old-output | range-list | rpf | safi-all
| summary | table-context | topology | traffic | tunnel | unicast}
3. show mrib [vrf vrf-name] {client | ipv4 | ipv6 | label-table-info | mdt-interface | mpls | nsf |
old-output | platform | route | route-collapse | table-info |tlc }
4. show mfib [vrf vrf-name] {bundle-hash | connections | counter | encap-info | hardware |
interface | ipv4 | ipv6 | lsm | mdt | nsf | route | svd | table-info}
5. show mfib hardware {adjacency | connection | interface | ltrace | resource-counters | route |
table} location node-id
6. show mfib hardware route { accept-bitmap | internal | mofrr | olist | statistics | summary } {* |
A.B.C.D | A.B.C.D/length | detail | hex-dump} location node-id
7. show mfib hardware route summary location node-id
8. debug mrib errors
9. debug mrib events
10. debug mfib warning
11. debug mfib errors
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 11-257
Chapter 11 Troubleshooting Multicast Services
Troubleshooting Native Multicast Routing (Layer 3)
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
11-258 OL-23591-02
Chapter 11 Troubleshooting Multicast Services
Troubleshooting Native Multicast Routing (Layer 3)
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 11-259
Chapter 11 Troubleshooting Multicast Services
Troubleshooting Native Multicast Routing (Layer 3)
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
11-260 OL-23591-02
Chapter 11 Troubleshooting Multicast Services
Troubleshooting Native Multicast Routing (Layer 3)
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 11-261
Chapter 11 Troubleshooting Multicast Services
Troubleshooting Native Multicast Routing (Layer 3)
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
11-262 OL-23591-02
Chapter 11 Troubleshooting Multicast Services
Troubleshooting Native Multicast Routing (Layer 3)
Step 2 Ensure that the PIE file name is correct and reissue the command.
Step 3 Ensure that the location of the PIE file is correct and reissue the command
Step 4 Ensure that the PIE file has proper permissions (755) and reissue the command.
Step 5 If you are loading from the TFTP directory, ensure that the following are true:
a. Router has network connectivity.
b. TFTP address is properly configured.
c. TFTP server has connectivity.
Step 1 Verify that IGMP packets are being received by the IGMP process on the RSP.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 11-263
Chapter 11 Troubleshooting Multicast Services
Troubleshooting Native Multicast Routing (Layer 3)
IP statistics:
Rcvd: 294930104 total, 3052259 local destination
0 format errors, 0 bad hop count
593296 unknown protocol, 0 not a gateway
0 security failures, 0 bad source, 369473 bad header
1453418 with options, 0 bad, 0 unknown
Opts: 0 end, 0 nop, 0 basic security, 0 extended security
0 strict source rt, 0 loose source rt, 0 record rt
0 stream ID, 0 timestamp, 1453418 alert, 0 cipso
Frags: 0 reassembled, 0 timeouts, 0 couldn't reassemble, 0 fragments received
0 fragmented, 0 fragment count, 0 fragment max drop
Bcast: 0 sent, 0 received
Mcast: 9878922 sent, 291744628 received
Drop: 0 encapsulation failed, 3 no route, 0 too big
Sent: 13116263 total
.
.
.
Received Sent
Valid IGMP Packets 491447 777437
Queries 368364 246191
Reports 123083 531246
Leaves 0 0
Mtrace packets 0 0
DVMRP packets 0 0
PIM packets 0 0
Errors:
Malformed Packets 0
Bad Checksums 0
Socket Errors 0
Bad Scope Errors 0
Auxiliary Data Len Errors 0
Packets dropped due to invalid socket 0
Packets which couldn't be accessed 0
Packet allocation failure 0
Other packets drops 0
Step 2 Verify that IGMP has the specified group/source on the interface.
RP/0/RSP0/CPU0:router# show igmp interface gigabitEthernet 0/1/0/2
Thu Jan 6 10:34:30.664 PST
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
11-264 OL-23591-02
Chapter 11 Troubleshooting Multicast Services
Troubleshooting Native Multicast Routing (Layer 3)
Step 3 Verify that the multicast routing information base (MRIB) has the expected route and outgoing interface
list (OLIST).
RP/0/RSP0/CPU0:router# show mrib route
Thu Jan 6 10:38:42.800 PST
Step 4 Verify that the multicast forwarding information base (MFIB) on the LC has the interface as an OLIST
member.
RP/0/RSP0/CPU0:router# show mfib route location 0/1/CPU0
Thu Jan 6 10:47:06.989 PST
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 11-265
Chapter 11 Troubleshooting Multicast Services
Troubleshooting Native Multicast Routing (Layer 3)
(*,10.0.0.0/4), Flags: C
Up: 6w0d
Last Used: never
SW Forwarding Counts: 0/0/0
SW Replication Counts: 0/0/0
SW Failure Counts: 0/0/0/0/0
(*,10.66.66.66), Flags: C
Up: 6w0d
Last Used: never
SW Forwarding Counts: 0/0/0
SW Replication Counts: 0/0/0
SW Failure Counts: 0/0/0/0/0
TenGigE0/4/0/0 Flags: A, Up:6w0d
GigabitEthernet0/1/0/8 Flags: NS EG, Up:6w0d
Step 5 Verify that the MFIB in the hardware has the interface as an OLIST member.
RP/0/RSP0/CPU0:router# show mfib hardware interface gigabitEthernet 0/1/0/27 location
0/1/CPU0
Thu Jan 6 10:55:27.723 PST
LC Type: A9K-40GE-L
--------------------------------------------------------------------
Interface Handle RefCnt TTL Routes uIDB Enbld Comment
--------------------------------------------------------------------
Gi0/1/0/27 0x2000740 8 0 3 55 True success
--------------------------------------------------------------------
ROUTE INFORMATION:
Legend:
S: Source, G: Group, P: Prefix length, PI: Packets cn, PO: packets out,
RF: RPF failures, TF: TTL failures, OF: OLIST failures, F: Other failures
Route flags - (Ingress)
C: Chip ID, IC: BACL check, IP: Punt this packet to LC CPU,
ID: Directly connected, IS: RPF interface signal, IU: Punt copy to RP,
IF: Punt to LC CPU if forwarded, IM: Result match, IV: Valid entry,
IR: RPF IF, IA: Fabric slotmask, IG: Mulicast group ID
Route flags - (Egress)
ET: Table ID to be used for OLIST lookup, EO: OLIST count bit,
ER: Route MGID to be used for OLIST/NRPF lookup, EM: Result match,
EV: Valid entry, EC: Count of OLIST members on this chip,
BS: Base of the statistics pointer
Interface: Gi0/1/0/27
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
11-266 OL-23591-02
Chapter 11 Troubleshooting Multicast Services
Troubleshooting Native Multicast Routing (Layer 3)
Step 6 One possible cause could be that the IGMP group is timing out. One way to check is to create a static
route for the (*,G) and see if traffic is now received. If it is, it means that the groups are timing out.
Step 7 To confirm the result from Step 6, remove the static route and decrease the query interval (resulting in
more queries per minute) to make it clearer:
RP/0/RSP0/CPU0:router# conf
RP/0/RSP0/CPU0:router(config)# router igmp
RP/0/RSP0/CPU0:router(config-igmp)# query-interval 1
RP/0/RSP0/CPU0:router(config-igmp)# commit
Step 8 Ensure that packets are going out of interface at the interval set.
Step 9 Check that the tester responds with an IGMP membership report. If the packets are received at the tester,
the result from Step 6 is confirmed. Use the workaround.
Workaround
Step 1 Ensure that packets are going from the ingress network processor (NP) through the fabric to the egress
NP.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 11-267
Chapter 11 Troubleshooting Multicast Services
Troubleshooting Native Multicast Routing (Layer 3)
RP/0/RSP0/CPU0:router# show mfib hardware route olist location {ingress node-id | egress
node-id }
Step 3 View the statistics for the specific route and source.
RP/0/RSP0/CPU0:router# show mfib hardware route stat [src ip addr] location {ingress
node-id | egress node-id }
RP/0/RSP0/CPU0:router# show mfib hardware route olist location {ingress node-id | egress
node-id }
Step 3 Ensure that packets are transmitted out of ingress NP to fabric and received by egress NP from the fabric.
RP/0/RSP0/CPU0:router# show mfib hardware route statistics location {ingress node-id |
egress node-id }
Step 1 Ensure that packets are going from the ingress NP through the fabric to the egress NP.
RP/0/RSP0/CPU0:router# show mfib hardware route statistics location {ingress node-id |
egress node-id}
Step 2 The preceding command checks if if packets are punted to the RP. If so, check if the source of that
channel is setting some IP options or not.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
11-268 OL-23591-02
Chapter 11 Troubleshooting Multicast Services
Troubleshooting Native Multicast Routing (Layer 3)
Workaround
Workaround
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 11-269
Chapter 11 Troubleshooting Multicast Services
Troubleshooting Native Multicast Routing (Layer 3)
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
11-270 OL-23591-02
INDEX
A D
B E
C
F
CCM 3-89
CEF troubleshooting 4-95 fabric 4-109, 7-143
CFM 3-85 forwarding information base (FIB) 4-95
cfs check command 1-22
cisco-support task ID 1-3
G
Cisco Technical Support 1-58
CLI access 1-2 gathering information 1-58
commit confirmed command 1-20
connectivity fault management (CFM) 3-85
Continuity check messages 3-89
I
control plane 4-109 IGMP reports dropped 11-251
control plane Ethernet network IGMP snooping 11-247
overview 1-41 IGMP snooping querier 11-251
counters, NP 7-148 installation, software 1-7, 1-10
crashes, RSP and LC 7-165 install verify command 1-10
crosscheck for MEPs 3-92 interfaces 2-61
interfaces, connectivity 3-75
interfaces, optical 2-68
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 IN-1
Index
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
IN-2 OL-23591-02
Index
show cef platform trace ipv4 all command 4-99 show tech-support command 1-59
show cfgmgr trace command 1-22 show uidb data command 4-110
show configuration commit changes command 1-19, 1-22 show uidb index command 4-110
show configuration commit list command 1-20 show users command 1-52
show configuration failed command 1-24 show version command 1-8, 1-27
show configuration failed startup command 1-22 software installation 1-7, 1-10
show configuration history commit command 1-22 software version 1-8
show context command 1-28, 1-50 sohw hw-module subslot counters framer command 2-62
show controllers backplane ethernet clients all startup failed 1-24
command 1-44
show controllers backplane ethernet local clients statistics
command 1-44 T
show controllers stats command 2-62
TAC
show environment command 1-28
gathering information 1-58
show ethernet cfm configuration-errors command 3-87
TCAM classification 9-182
show history command 1-52, 1-53
top command 1-50
show hw-module subslot address status
pluggable-optics 2-68 top processes command 1-28
show hw-module subslot command 2-62 trace 1-56
show imds interface brief 4-110 tracebacks 6-133, 6-136
show install active command 1-12 trace commands 1-56
show install command 1-8 traceroute 6-130
show install committed command 1-12, 1-14 traffic drop, transient 4-106
show interface brief command 1-29 traffic engineering tunnel 8-176
show interface command 2-62 traffic loss 7-168
show interfaces brief command 1-3 transport input telnet command 1-4
show interfaces command 1-5 tunnel, TE 8-176
show ipv4 interface command 1-5
show logging command 1-27, 1-59
U
show memory 1-38
show memory heap command 1-28 user access 1-2
show memory summary command 1-28
show netio idb command 2-62, 4-110
show platform command 1-27, 1-44, 1-49
V
show route ipv4 command 4-97 version 1-8
show running-config 1-21 VLAN traffic 9-182
show running-config command 1-16, 1-27, 1-28 VPLS 9-195
show sysdb trace command 1-18 VPWS deployment 9-206
show sysdb trace verification command 1-22 VRRP 6-138
show system verify command 1-29, 1-59
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02 IN-3
Index
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
IN-4 OL-23591-02