Troubleshooting Cisco Catalyst

6500 / 6800 Series Switches

Shashank Singh, Technical Leader, Cisco Services
shashasi@cisco.com

BRKCRS-3143
Session Abstract
• Session covers platform specific troubleshooting methods and commands for the Cisco Catalyst
6500 and 6800 Series Switches.

• Session starts with an introduction to basic architecture of Sup2T/PFC4.

• Further into the session, commands and methods required to troubleshoot IPv4 unicast and
multicast packet forwarding are discussed.

• Towards the end, session provides an introduction to Catalyst 6800IA solution followed by
commands and techniques to troubleshoot the Catalyst 6800IA solution.

• Target audience – network engineers and technicians who work with the Cisco Catalyst 6500
and 6800 Series or who would like to gain familiarity with troubleshooting of the platform.

• First-hand experience with Catalyst 6500 and 6800 platforms is expected.

3
Agenda
• Architecture: Sup720 vs Sup2T
• Troubleshooting Unicast Forwarding
• Troubleshooting Multicast Forwarding
• Introduction to Instant Access (IA) Solution
• Troubleshooting 6800ia Solution

4
Goal of this Session …..
Teach commonly used techniques and commands to troubleshoot Cisco
Catalyst 6500/6800 switches and …... make it less of a BLACK BOX !!

5
Why this session covers both Catalyst 6500 and
6800 ? Catalyst 6800 Series
Catalyst 6807-XL

• Catalyst 6800 is the foundation of Instant

Access (IA) solution.
Catalyst 6880-X
• Catalyst 6800 switch and modules are
built on Sup2T/PFC4 architecture
Catalyst 6840-X

Catalyst 6800-IA
Take Away:
Whatever we learn in this session is
Catalyst 6500 Series
applicable to Catalyst 6500 Sup2T
standalone, VSS and Catalyst 6800 Instant
Access Solution.

6
Architecture: Sup720 vs. Sup2T

7
Acronyms Legend
PFC: Policy Feature Card
DFC: Distributed Forwarding Card
FE: Forwarding Engine  Reference slide
CAM: Content Addressable Memory
TCAM: Ternary or Tertiary CAM
FIB: Forwarding Information Base
ACL: Access Control List
ACE: Access Control Entry
EOBC: Ethernet Out-of-Band Channel
BD: Bridge Domain
LIF: Logical Interface
CoPP: Control Plane Policing
FPOE: Fabric Port of Exit

8
Supervisor 720/PFC3 Architecture
Layer2 Control-plane
E.g., LACP, BPDU and Replication engine L3/4 forwarding
Layer3 Control-plane
hardware programming E.g., Multicast, SPAN
E.g., OSPF, BGP, SNMP SFP /
SFP GETX

ACE

Flash
MSFC 3 1 Gbps
QoS Adj FIB ACL
NetFlow TCAM TCAM TCAM
Counter

RP TCAM
DRAM CPU Port ASIC
1 Gbps
Flash SP L3/4 Engine
DRAM CPU MET

L2 forwarding
Fabric Interface
Switch Fabric L2 Engine
20 Gbps and
18 x 20G Traces Replication Engine L2 CAM (64K)
PFC3

DBUS
RBUS 16 Gbps Bus
EOBC
Traces # 1 to 16
Integrated Switch Fabric
9
Supervisor 2T/PFC4 Architecture
MSFC5 Complex contains single
Replication engine
dual-core CPU for both Layer 2 and L3/4 forwarding
Layer 3 control-plane protocols and 1GE / 10GE E.g., Multicast, SPAN
hardware programming Uplinks

MSFC 5
NetFlow CL1 ADJ FIB CL2
Central 2 Gbps TCAM TCAM TCAM TCAM
Management Port ASIC
Processor
CPU LIF RPF
MET MAP L3/4 Engine Table
DRAM Flash

Fabric Interface LIF Table

ACE
Counter

Switch Fabric & LIF Stats

L2 Engine
40 Gbps
26 x 40G Traces Replication Engine L2 CAM (128K)
PFC4

DBUS L2 forwarding
EOBC RBUS
Traces # 1 to 24

Integrated Switch Fabric 10

Troubleshooting Unicast Forwarding

12
Troubleshooting Unicast Forwarding
Agenda
• L2 Topology and Packet Flow
• L2 Packet Flow Troubleshooting
L2 CAM, Interface counters/errors, Switch Fabric

• L3 Topology and Packet Flow

• L3 Packet Flow Troubleshooting
FIB and Adjacency TCAM

13
Troubleshooting Unicast Forwarding
Agenda
• L2 Topology and Packet Flow
• L2 Packet Flow Troubleshooting
L2 CAM, Interface counters/errors, Switch Fabric

• L3 Topology and Packet Flow

• L3 Packet Flow Troubleshooting
FIB and Adjacency TCAM

14
L2 Unicast Traffic
Topology
Po11 Po11 Po12 Po12
Ten1/4 Ten1/1 Ten1/5 Ten1/3
Ten1/8 Ten1/2 Ten1/6 Ten1/4
Ten1/5 Ten2/1 Ten2/5 Ten1/7
Host 1 Host 2
Ten1/7 Ten2/2 Ten2/6 Ten1/8
192.168.0.2
(0006.5bbc.81a2) 192.168.0.3
R1 DUT R2 (0006.5bbc.7acb)

Vlan 10
• DUT is the Device Under Test we are troubleshooting
• DUT is a 6509-E with Supervisor 2T
• Four TenGigabitEthernet L2 Etherchannel (R1  DUT)
• Four TenGigabitEthernet L2 Etherchannel (DUT  R2)

15
L2 Unicast Traffic
Where are the MAC Addresses Learned?
Sup2T# show mac address-table address 0006.5bbc.81a2
Host 1 Legend: * - primary entry
age - seconds since last seen
n/a - not available
S - secure entry
R - router's gateway mac address entry
D - Duplicate mac address entry
Displaying entries from DFC linecard [1]:
vlan mac address type learn age ports
----+----+---------------+-------+-----+----------+--------
* 10 0006.5bbc.81a2 dynamic Yes 5 Po11
Displaying entries from DFC linecard [2]:
vlan mac address type learn age ports
Host 2 ----+----+---------------+-------+-----+----------+--------
* 10 0006.5bbc.81a2 dynamic Yes 90 Po11

Sup2T# show mac address-table address 0006.5bbc.7acb

Displaying entries from DFC linecard [1]:
vlan mac address type learn age ports Both MAC addresses are
----+----+---------------+-------+-----+----------+-------- learned on Port-Channels;
* 10 0006.5bbc.7acb dynamic Yes 0 Po12
Displaying entries from DFC linecard [2]:
Which physical link in the
vlan mac address type learn age ports channel Is actually receiving
----+----+---------------+-------+-----+----------+-------- the packets?
* 10 0006.5bbc.7acb dynamic Yes 110 Po12

16
L2 Unicast Traffic
Which Link in the EtherChannel Is Being Used?
Po11 Po11 Po12 Po12
Gig4/1 Ten1/4 Ten1/1 Ten1/5 Ten1/3
Ten1/8 Ten1/2 Ten1/6 Ten1/4
Ten1/5 Ten2/1 Ten2/5 Ten1/7
Host 1 Host 2
Ten1/7 Ten2/2 Ten2/6 Ten1/8
192.168.0.2 192.168.0.3
R1 DUT R2
R1#show etherchannel load-balance module 4 Check load balancing configuration
EtherChannel Load-Balancing Configuration: Use ingress Module number in command
in case per-module load-balancing is
src-dst-ip vlan included configured (SXH images and later)
mpls label-ip
EtherChannel Load-Balancing Addresses Used Per-Protocol:
Mode is “src-dst-ip”. Only use src and dest
Non-IP: Source XOR Destination MAC address
IP as argument. Prior to 12.2(33)SXH, use
IPv4: Source XOR Destination IP address test etherchannel load-balance …(same
arguments) on the SP, for Sup720 engines.
IPv6: Source XOR Destination IP address
MPLS: Label or IP
R1# show etherhannel load-balance interface po11 ip 192.168.0.2 192.168.0.3
Computed RBH: 0x3
Link selected is Ten1/8 in Po11 of R1 for traffic to 192.168.0.3
Would select Te1/8 of Po11

17 17
L2 Unicast Traffic
Network Path Verification: Result

Po11 Po11 Po12 Po12

Gig4/1 Ten1/4 Ten1/1 Ten1/5 Ten1/3
Ten1/8 Ten1/2 Ten1/6 Ten1/4
Ten1/5 Ten2/1 Ten2/5 Ten1/7
Host 1 Host 2
Ten1/7 Ten2/2 Ten2/6 Ten1/8
192.168.0.2 192.168.0.3

R1
DUT R2
Po11 Po11 Po12 Po12
Gig4/1 Ten1/4 Ten1/1 Ten1/5 Ten1/3
Ten1/8 Ten1/2 Ten1/6 Ten1/4
Ten1/5 Ten2/1 Ten2/5 Ten1/7
Host 1 Host 2
Ten1/7 Ten2/2 Ten2/6 Ten1/8
192.168.0.2 192.168.0.3

R1
DUT R2

Each direction can use different links in the bundles !

18 18
Troubleshooting Unicast Forwarding
Agenda

• L2 Topology and Packet Flow

• L2 Packet Flow Troubleshooting
L2 CAM, Interface counters/errors, Switch Fabric

• L3 Topology and Packet Flow

• L3 Packet Flow Troubleshooting
FIB and Adjacency TCAM

19
Layer 2 Learning and Forwarding
• In Sup720 engines, Layer 2 forwarding is based on {VLAN, MAC} pairs. In
Sup2T engines, {port_index, MAC} is used by Bridge Domain(BD) for
Bridging and by Logical Interface (LIF) for routing.

• MAC learning is done per PFC or DFC

• Each PFC/DFC maintains separate L2 CAM table

• PFC and DFCs age entries independently

• Refreshing of entries based on “seeing” traffic from specific host
• New learns on one forwarding engine communicated to other engines via MAC-Sync process (which
occurs over EOBC)

20
Detailed L2 Packet Flow Troubleshooting
Are We Learning MAC Addresses?
Sup2T# show mac address-table address 0006.5bbc.7acb vlan 10 [all] By default, Sup2T prints entries
Legend: * - primary entry from all DFCs. In Sup720, use
all keyword to see entry from all
age - seconds since last seen DFCs in the system.
n/a - not available
S - secure entry
R - router's gateway mac address entry
D - Duplicate mac address entry * Denotes the primary
Displaying entries from DFC linecard [1]: forwarding entry. This
vlan mac address type learn age ports entry is owned by ingress
forwarding engine for
----+----+---------------+-------+-----+----------+--------- frames sourced from that
* 10 0006.5bbc.7acb dynamic Yes 0 Po12 ethernet address.
Displaying entries from DFC linecard [2]:
vlan mac address type learn age ports
----+----+---------------+-------+-----+----------+---------- Flooding can occur if
* 10 0006.5bbc.7acb dynamic Yes 185 Po12 MACs are not known
Displaying entries from active supervisor: by ALL FEs in the
vlan mac address type learn age ports system
----+----+---------------+-------+-----+----------+----------
10 0006.5bbc.7acb dynamic Yes 205 Po12

21
Detailed L2 Packet Flow Troubleshooting
Verify L2 Tables: MAC Sync Feature
Sup2T# show mac address-table synchronize statistics
Out-of-Band (OOB) MAC-Sync feature is enabled by
MAC Entry Out-of-band Synchronization Feature Statistics: default in Sup2T. By default, it is disabled in Sup720.
--------------------------------------------------------- Flooding can occur when L2 CAM tables are not in sync.
Module [1] Enable this feature with “mac-address-table
synchronize” command (under “config t”) in Sup720.
-----------
Module Status:
Off by default in Sup720.
Statistics collected from module : 1 When WS-X6708 is
Global Status: present, it is on by default,
Status of feature enabled on the switch : on and set the mac aging timer
Default activity time : 160 to 480 sec. Why 480 ?
Configured current activity time : 160
Statistics from ASIC 0 when last activity timer expired: Default value is 160
Age value in seconds from age byte register : 0x4C seconds; normal aging
timer should be at least
<snip>
3x activity interval … so
Number of entries created new : 377 with default of 160 sec,
Number of entries create failed : 0 change mac aging timer
Module [2] to 480 sec or more
-----------
Module Status:
Number of entries that were
Statistics collected from module : 2 synced by SW sync feature
Global Status:
Status of feature enabled on the switch : on
22
Detailed L2 Packet
Host2
Flow Troubleshooting
Ten2/5 Ten2/6
Port Port
ASIC ASIC
WS-X6908
Fabric Layer 2 Look at the interface
Module 2
Interface & Engine counters and errors
MET Replication
Engine for the ingress and
Layer 3/4
DFC4 Engine egress interfaces

Check the L2
Switch Fabric forwarding engine
counters
WS-X6908
Fabric Layer 2
Interface &
Module 1
Engine
MET Replication Verify the fabric
Engine Layer 3/4 channels used in the
Engine
Port Port DFC4 flow
ASIC ASIC
Ten1/1 Ten1/2

Host1 23
Detailed L2 Packet Flow Troubleshooting
Verify L2 Counters: Interface Counters
Sup2T# show interface ten 1/2 counters
Port InOctets InUcastPkts InMcastPkts InBcastPkts
Te1/2 249784 2000 8 40
Port OutOctets OutUcastPkts OutMcastPkts OutBcastPkts Did a ping (2000 packets/100
bytes per packet) from
Te1/2 83246 18 6 0 192.168.0.2 to 192.168.0.3.
Sup2T#show interface ten 1/1 counters verify interface counters relevant
Port InOctets InUcastPkts InMcastPkts InBcastPkts to the path did move sufficiently !!

Te1/1 10590 18 28 0
Port OutOctets OutUcastPkts OutMcastPkts OutBcastPkts
Te1/1 246412 2008 10 0
Sup2T#show interface ten 2/5 counters
Port InOctets InUcastPkts InMcastPkts InBcastPkts
Te2/5 2890 2890 0 0
Port OutOctets OutUcastPkts OutMcastPkts OutBcastPkts
Te2/5 273441 2032 11 0

And, similarly on Ten2/5

24
Detailed L2 Packet Flow Troubleshooting
Verify L2 Counters: Interface Counters
shows
Sup2T# show interface ten1/1 counter error interface level
Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize OutDiscards packet counts
Te1/1 0 0 0 0 0 0 and errors
since last time
Port Single-Col Multi-Col Late-Col Excess-Col Carri-Sen Runts Giants clear counters
Te1/1 0 0 0 0 0 0 0 was issued.

Port SQETest-Err Deferred-Tx IntMacTx-Err IntMacRx-Err Symbol-Err

Te1/1 0 0 0 0 0

Sup2T# clear counters

Sup2T# show counter interface te1/1
<snip> Sup2T# show counter interface te1/1 delta
64 bit counters: Time since last clear
0. rxHCTotalPkts = 13021673 ---------------------
1. txHCTotalPkts = 3090200 00:00:02 shows traffic statistics since last clear
2. rxHCUnicastPkts = 2684645
64 bit counters:
3. txHCUnicastPkts = 2684649
0. rxHCTotalPkts = 1
<snip>
1. txHCTotalPkts = 3
.... nearly 140 counters ... 2. rxHCUnicastPkts = 0
3. txHCUnicastPkts = 0
Hardware counters. Not cleared by <snip>
“clear counters” command.
25
Detailed L2 Packet Flow Troubleshooting
Verify L2 Counters: L2 Forwarding Engine VLAN Count
Sup2T #show vlan id 10 counters
* L2 counters include multicast and broadcast packets VLAN is bidirectional, so counts
both directions of the flow
(192.168.0.2  192.168.0.3)
Vlan Id : 10
L2 Unicast Packets : 4012
L2 Unicast Octets : 401868
L3 Input Unicast Packets : 0
L3 Input Unicast Octets : 0
L3 Output Unicast Packets : 0
L3 Output Unicast Octets : 0
L3 Output Multicast Packets : 0
L3 Output Multicast Octets : 0
L3 Input Multicast Packets : 0
L3 Input Multicast Octets : 0
L2 Multicast Packets : 0
L2 Multicast Octets : 0

26
Detailed L2 Packet Flow Troubleshooting
Identifying the Fabric Channels
Sup2T# sh fabric fpoe interface ten1/1 Host2
fpoe for TenGigabitEthernet1/1 is 1 Ten2/5 Ten2/6
For each ingress and Fabric Interface ASIC
Sup2T# sh fabric fpoe interface ten1/2
egress interface: find Port Port
fpoe for TenGigabitEthernet1/2 is 1 mapping between
Sup2T# sh fabric fpoe interface ten2/5 ASIC ASIC
interface and Fabric WS-X6908
fpoe for TenGigabitEthernet2/5 is 34 Port Of Exit (FPOE). Fabric Layer 2
Sup2T# sh fabric fpoe interface ten2/6 Interface &
Module 2
Engine
fpoe for TenGigabitEthernet2/6 is 34 MET Replication
Find mapping between Engine Layer 3/4
FPOE and Slot/Channel
DFC4 Engine Fabric ASIC
(requires service internal
config under config t)
FPOE 34
Sup2T# show fabric fpoe map
FPOE 1 Switch Fabric
slot channel logical fpoe physical fpoe
1 0 0 5 WS-X6908
1 1 1 5 Fabric Layer 2 Module 1
Interface &
1 2 32 6 MET
Engine
Replication
1 3 33 6 Engine Layer 3/4
2 0 2 11 Engine
Port Port DFC4
2 1 3 11
ASIC ASIC
2 2 34 7
2 3 35 7 Ten1/1 Ten1/2
<snip>
Host1
27
Detailed L2 Packet Flow Troubleshooting
Verify L2 Counters: Switching Fabric Utilization
Sup2T# show fabric status
slot channel speed module fabric hotStandby Standby Standby
status status support module fabric
1 0 40G OK OK Y(not-hot)
1 1 40G OK OK Y(not-hot) Check status of fabric channels is
OK. An example for misbehaving
2 0 40G OK OK Y(not-hot) module or fabric channel: Module
2 1 40G OK OK Y(not-hot) status is reported as “DDR Sync”
5 0 20G OK OK N/A
5 1 20G OK OK N/A
6 0 20G OK OK N/A
6 1 20G OK OK N/A
Check utilization (current and last peak
Sup2T# show fabric utilization detail value) for relevant fabric channels … did
Fabric utilization: Ingress Egress any peak coincide with moment of drops?
Module Chanl Speed rate peak rate peak
1 0 40G 0% 0% 0% 0%
1 1 40G 0% 1% @15:47 21Feb12 0% 0%
2 0 40G 0% 0% 0% 0%
2 1 40G 0% 0% 0% 1% @02:34
22Feb12
5 0 20G 0% 0% 0% 0%
5 1 20G 0% 0% 0% 0%
<snip>
28
Detailed L2 Packet Flow Troubleshooting
Verify L2 Counters: Relevant Fabric Channel Counters
unable to send packets from fabric to line
Sup2T# show fabric channel-counters 1
card: Check traffic levels, line card OK ?
slot channel rxErrors txErrors txDrops lbusDrops
1 0 0 0 0 0 fabric interface unable to send
packets from local bus to fabric
1 1 0 0 0 0 (Supervisor and 65XX modules only,
67XX and above report Overruns in
Sup2T# show fabric channel-counters 2 “show interface” results. check traffic
slot channel rxErrors txErrors txDrops lbusDrops levels, signs of congestion ?

2 0 0 0 0 0 fabric serial link bit errors (8 serial links in

each fabric channel), reported as soon as 2
2 1 0 0 0 0 fabric serial link interrupts within 100ms; can
result in rxErrors / txErrors; check card
inserted OK ?
Sup2T# show fabric errors 1
line card fabric ASIC reports bad
Module errors:
packets: card inserted properly ? A
few incrementing ‘rxErrors', which is slot channel crc hbeat sync DDR sync
not correlated to any network events, 1 0 0 0 0 0
is OK & acceptable. 1 1 0 0 0 0
Fabric errors:
slot channel sync buffer timeout fabric ASIC unable to send traffic
1 0 0 0 0 to the fabric enabled module for
1 1 0 0 0 last +3 seconds

29
Troubleshooting Unicast Forwarding
Agenda

• L2 Topology and Packet Flow

• L2 Packet Flow Troubleshooting
L2 CAM, Interface counters/errors, Switch Fabric

• L3 Topology and Packet Flow

• L3 Packet Flow Troubleshooting
FIB and Adjacency TCAM

30
L3 Unicast Traffic Network Configuration
Host1 Po11 Po11 Host2
Ten1/4 Ten1/1 Ten1/5 Ten1/3
Ten1/8 Ten1/2 Ten1/6 Ten1/4
Ten1/5 Ten2/1 Ten2/5 Ten1/7
100.100.100.1 Ten1/7 Ten2/2 Ten2/6 Ten1/8 200.200.200.1
L3 Links
R1 VLANS 10,20,30 and 40 DUT R2

• DUT is the Device Under Test we are troubleshooting

• DUT is a 6509-E with Supervisor 2T
• Four TenGigabitEthernet L2 Etherchannel Trunk (R1  DUT)
Vlan 10, 20, 30 and 40 are assigned with 192.168.10.0/24, 192.168.20.0/24,
192.168.30.0/24 and 192.168.40.0/24 subnets respectively.
• Four L3 Links (DUT  R2)
Four links are assigned with 172.16.10.0/24, 172.16.20.0/24, 172.16.30.0/24 and
172.16.40.0/24 subnets respectively.

31
L3 Unicast Traffic
Different Switching Paths for L3 Traffic in Catalyst 6500/6800

Process Switching Path

Software-based CEF Switching Path

Hardware-based CEF switching Path

Host1 Host2
DUT
This slide is just a logical representation of
different switching paths (also known as
Switching Vectors) in Catalyst 6500/6800.

32
L3 Unicast Traffic
Host 1 Host 2: Which L3 Next Hop / L2 Link from R1?
R1# show ip route 200.200.200.1 SW
Routing entry for 200.200.200.1/32
Known via "ospf 100", distance 110, metric 3, type intra area Equal Cost Routes to the
Last update from 192.168.40.1 on Vlan40, 00:10:12 ago destination prefix
Routing Descriptor Blocks:
192.168.40.1, from 192.168.0.2, 00:10:12 ago, via Vlan40
* denotes the path it takes for the next process-
Route metric is 3, traffic share count is 1
switched traffic. It moves in a round-robin fashion,
192.168.30.1, from 192.168.0.2, 00:10:12 ago, via Vlan30
Route metric is 3, traffic share count is 1
* 192.168.20.1, from 192.168.0.2, 00:10:12 ago, via Vlan20 Next hop used for SW based
Route metric is 3, traffic share count is 1 CEF (SW forwarding data path)
192.168.10.1, from 192.168.0.2, 00:10:12 ago, via Vlan10
Route metric is 3, traffic share count is 1 Next hop used for HW based CEF
(HW forwarding path). Note: “0” is
R1# show ip cef exact-route 100.100.100.1 200.200.200.1 used for both src and dest L4 port
100.100.100.1 -> 200.200.200.1 => IP adj out of Vlan40, addr 192.168.40.1 numbers as test flow was ICMP echo

R1# show mls cef exact-route 100.100.100.1 0 200.200.200.1 0

Interface: Vl10, Next Hop: 192.168.20.1, Vlan: 10, Destination Mac: b414.8961.3780
Note: R1 is a
Cat6500 with R1# show etherchannel load-bal int port-ch 11 ip 100.100.100.1 200.200.200.1
Sup720, which Computed RBH: 0x7
supports “mls” Would select Te1/8 of Po11 Check which link between R1 and HW
commands. DUT is chosen.

33
L3 Unicast Traffic
Host 1  Host 2: Which L3 Next Hop from DUT?
Sup2T# show ip route 200.200.200.1 SW
Routing entry for 200.200.200.1/32
Known via "ospf 100", distance 110, metric 2, type intra area Equal Cost Routes to
Last update from 172.16.20.2 on TenGigabitEthernet1/6, 00:36:01 ago the destination prefix
Routing Descriptor Blocks:
172.16.40.2, from 192.168.0.2, 00:36:01 ago, via TenGigabitEthernet2/6
Route metric is 2, traffic share count is 1
172.16.30.2, from 192.168.0.2, 00:36:01 ago, via TenGigabitEthernet2/5
Route metric is 2, traffic share count is 1
172.16.20.2, from 192.168.0.2, 00:36:01 ago, via TenGigabitEthernet1/6
Route metric is 2, traffic share count is 1
* 172.16.10.2, from 192.168.0.2, 00:36:01 ago, via TenGigabitEthernet1/5
Route metric is 2, traffic share count is 1 Next hop used for SW based
CEF (SW forwarding data path)
Sup2T# show ip cef exact-route 100.100.100.1 200.200.200.1
100.100.100.1 -> 200.200.200.1 => IP adj out of TenGigabitEthernet1/6, addr 172.16.20.2

Sup2T# show plat hardware cef exact-route 100.100.100.1 0 200.200.200.1 0 HW

Interface: Te2/6, Next Hop: 172.16.40.2, ifnum: 0x12, Destination Mac: f866.f2d2.fa80
LIF: 0x20004013

Next hop used for HW based CEF (HW forwarding

path). Note: “0” is used for both src and dest L4
port numbers as test flow was ICMP echo

34
L3 Unicast Traffic
Network Path Verification: Result
Po11 Po11
Ten1/4 Ten1/1 Ten1/5 Ten1/3
Ten1/8 Ten1/2 Ten1/6 Ten1/4
Ten1/5 Ten2/1 Ten2/5 Ten1/7
Host 1 Host 2
Ten1/7 Ten2/2 Ten2/6 Ten1/8
100.100.100.1 200.200.200.1
R1 DUT R2

Po11 Po11
Ten1/4 Ten1/1 Ten1/5 Ten1/3
Ten1/8 Ten1/2 Ten1/6 Ten1/4
Ten1/5 Ten2/1 Ten2/5 Ten1/7
Host 1 Ten1/7 Ten2/2 Ten2/6 Ten1/8 Host 2
100.100.100.1 200.200.200.1
DUT R2
R1

Each direction can use different links in the bundles !

35 35
What Did We Get from Path Verification?
• The physical links the specific traffic flow should come in and leave the DUT.
• Helps us to isolate if there is any faulty or oversubscribed interface.
• Caveats:
• Flapping links in port channel, can change the bundle hash mapping, and change
physical path of traffic
• Clearing routes can as well change the order in which the L3 adjacencies get re-
programmed, and in case of ECMP hence change the physical path of the traffic

• Any of these happen, you need to re-verify the path

36
Troubleshooting Unicast Forwarding
Agenda

• L2 Topology and Packet Flow

• L2 Packet Flow Troubleshooting
L2 CAM, Interface counters/errors, Switch Fabric

• L3 Topology and Packet Flow

• L3 Packet Flow Troubleshooting
FIB and Adjacency TCAM

37
Detailed L3 Packet Flow Troubleshooting
L3 FIB Table Programming Flow
Host2
Ten2/5 Ten2/6
Port Port
ASIC ASIC
WS-X6908
Fabric Layer 2 Module 2
Interface & Engine
MET Replication
Engine Layer 3/4
DFC4 Engine

Check the L3 / L4
forwarding engine Switch Fabric
WS-X6908
Fabric Layer 2
Interface &
Module 1
Engine
MET Replication
Engine Layer 3/4
DFC4 Engine
Port Port
ASIC ASIC
Ten1/1 Ten1/2
Host1
38 38
Detailed L3 Packet Flow Troubleshooting
L3/4 Engine in Detail: Counters and Tables
• L3 forwarding tables get programmed by SW: copy of SW forwarding tables in HW

• EOBC is used for communication between modules and RP, and program L3 tables

PFC4
NetFlow CL1 ADJ FIB CL2
TCAM TCAM TCAM TCAM

LIF RPF
MAP L3/4 Engine Table

ACE
LIF Table Counter

LIF Stats
L2 Engine
L2 CAM (128K)

DBUS
EOBC
RBUS

39
FIB / Adjacency Tables
L3 FIB Table Programming Flow in Sup2T

Verify Layer 3 rewrite Verify Layer 2 rewrite

show ip route (RIB) show ip arp

IOS® Routing Table (RP) IOS ARP Cache Table (RP)
show ip cef
IOS FIB Table (RP) IOS Adjacency Table (RP) show ip cef
adjacency

IOS FIB Table (PFC/DFC) IOS Adjacency Table (PFC/DFC)

remote command
module <mod> FIB Table (PFC/DFC) Adjacency Table (PFC/DFC) remote command
show ip cef module <mod> show
adjacency detail

show plat hard cef lookup show plat hard cef

<ip address> <mod> adjacency entry

40
Detailed L3 Packet Flow Troubleshooting
Verify IP Routing Table
Sup2T# show ip route 100.100.100.1 Host 1 SW
Routing entry for 100.100.100.1/32
Known via "ospf 100", distance 110, metric 2, type intra area
Last update from 192.168.40.2 on Vlan40, 00:00:19 ago
Routing Descriptor Blocks:
192.168.40.2, from 192.168.252.10, 00:00:19 ago, via Vlan40
Route metric is 2, traffic share count is 1
192.168.30.2, from 192.168.252.10, 00:00:19 ago, via Vlan30
Route metric is 2, traffic share count is 1
192.168.20.2, from 192.168.252.10, 00:00:19 ago, via Vlan20
Route metric is 2, traffic share count is 1
* 192.168.10.2, from 192.168.252.10, 00:00:29 ago, via Vlan10
Route metric is 2, traffic share count is 1

Sup2T# show ip route 200.200.200.1 Host 2 SW

Routing entry for 200.200.200.1/32
Known via "ospf 100", distance 110, metric 2, type intra area
Last update from 172.16.30.2 on TenGigabitEthernet2/5, 00:01:00 ago
Routing Descriptor Blocks:
* 172.16.40.2, from 192.168.0.2, 00:01:10 ago, via TenGigabitEthernet2/6
Route metric is 2, traffic share count is 1
172.16.30.2, from 192.168.0.2, 00:01:00 ago, via TenGigabitEthernet2/5
Route metric is 2, traffic share count is 1
172.16.20.2, from 192.168.0.2, 00:01:00 ago, via TenGigabitEthernet1/6
Route metric is 2, traffic share count is 1
172.16.10.2, from 192.168.0.2, 00:01:00 ago, via TenGigabitEthernet1/5
Route metric is 2, traffic share count is 1

41
Detailed L3 Packet Flow Troubleshooting
L3 FIB Table and Counters

Sup2T# show ip cef 200.200.200.1 SW

IP CEF entries for destination IP addr
200.200.200.1/32
nexthop 172.16.10.2 TenGigabitEthernet1/5
nexthop 172.16.20.2 TenGigabitEthernet1/6
nexthop 172.16.30.2 TenGigabitEthernet2/5
nexthop 172.16.40.2 TenGigabitEthernet2/6

Sup2T# show ip cef exact-route 100.100.100.1 src-port 0 200.200.200.1 dest-port 0

100.100.100.1 -> 200.200.200.1 => IP adj out of TenGigabitEthernet1/6, addr 172.16.20.2

IP CEF entries for destination IP address

Sup2T# show ip cef adjacency tengig 1/6 172.16.20.2
172.16.20.2/32
attached to TenGigabitEthernet1/6
200.200.200.1/32 IP CEF Adjacency entries for next-hop IP address
nexthop 172.16.20.2 TenGigabitEthernet1/6

42
Detailed L3 Packet Flow Troubleshooting
L3 FIB Table and Counters

Sup2T# show platform hardware cef lookup 200.200.200.1 HW

Codes: decap - Decapsulation, + - Push Label No more MLS for Sup2T engines. For
Index Prefix Adjacency Sup720, use “mls” instead of “platform
hardware”.
8080 200.200.200.1/32 Te1/5 ,f866.f2d2.fa80 (Hash: 0001)
Te1/6 ,f866.f2d2.fa80 (Hash: 0002)
Te2/5 ,f866.f2d2.fa80 (Hash: 0004)
Te2/6 ,f866.f2d2.fa80 (Hash: 0008)

Sup2T# show platform hardware cef exact-route 100.100.100.1 0 200.200.200.1 0

Interface: Te2/6, Next Hop: 172.16.40.2, ifnum: 0x12, Destination Mac:
f866.f2d2.fa80 LIF: 0x20004013

43
Detailed L3 Packet Flow Troubleshooting
L3 FIB Table and Counters
Sup2T# show adjacency TenGigabitEthernet1/6 172.16.20.2 detail SW
Protocol Interface Address
IP TenGigabitEthernet1/6 172.16.20.2(14)
0 packets, 0 bytes
Rewrite information epoch 0
(Dmac|Smac|0800): verify it
sourced in sev-epoch 0
is conform with next hop
rewrite info Encap length 14
F866F2D2FA80B414896137800800
L2 destination address byte offset 0
L2 destination address byte length 6
Link-type after encap: ip
ARP

Sup2T# show platform hardware cef ip 200.200.200.1 detail module 1 HW

Codes: M - mask entry, V - value entry, A - adjacency index, NR- no_route bit
LS - load sharing count, RI - router_ip bit, DF: default bit
Ingress module, for
CP - copy_to_cpu bit, AS: dest_AS_number, DGTv - dgt_valid bit the specific flow
DGT: dgt/others value
Format:IPV4 (valid class vpn prefix)
Start adjacency pointer is 311296
M(8080 ): 1 F 3FFF 255.255.255.255
V(8080 ): 1 0 0 200.200.200.1
(A:311296, LS:3, NR:0, RI:0, DF:0 CP:0 DGTv:1, DGT

44
Detailed L3 Packet Flow Troubleshooting
L3 FIB Table and Counters
Sup2T# show platform hardware cef adjacency entr311296 detail module 1 HW
Index: 311296 -- Valid entry (valid = 1) –
Adjacency fields:
___________________________________________________
Checking the entry in
|adj_stats = EN | fwd_stats = EN | trig = 0 the ingress module
|_________________|__________________|______________
|l3_enable = ON (classify as Layer3) | age = 3
|_________________|__________________|______________
|format = IP | rdt = ON | ignr_emut = 0
|_________________|__________________|______________
|vpn = 0x3FFF | elif = 0x400C | ri = 3
|_________________|__________________|______________
|top_sel = 0 | zone_enf = OFF | fltr_en = OFF
|_________________|__________________|______________
|frr_te = OFF | idx_sel = 0 | tnl_encap = 0
|_________________|__________________|______________
|rw_hint = 0 | ttl_control = 4 |
|_________________|__________________|______________

Format of the
packet sent out on
the wire ...

45
Detailed L3 Packet Flow Troubleshooting
L3 FIB Table and Counters
Output Continued ….
RIT fields: The entry has a Layer2 Format HW
_________________________________________________________
|decr_ttl = YES | pipe_ttl = 0 | utos = 0
|_________________|__________________|____________________
|l2_fwd = 0 | rmac = 0 | ccc = L3_REWRITE
|_________________|__________________|____________________
|rm_null_lbl = YES| rm_last_lbl = YES| pv = 0
|_________________|__________________|____________________
|add_shim_hdr= NO | rec_findex = N/A | rec_shim_op = N/A
|_________________|__________________|____________________
|rec_dti_type = N/A | rec_data = N/A
|____________________________________|____________________
|modify_smac = YES| modify_dmac = YES| egress_mcast = NO
|____________________________________|____________________
|ip_to_mac = NO
|_________________________________________________________
|dest_mac = f866.f2d2.fa80 | src_mac = b414.8961.3780 Rewrite MAC info
|___________________________|_____________________________
|
Statistics: Packets = 0 Increases in the ingress DFC/PFC. Counters
Bytes = 0 will be cleared when adjacency is read.

46
Troubleshooting Unicast Forwarding
Summary
 Determine path-of-the-packet through a
L2 and L3 network
 L2 Forwarding
‒ Check MAC Learning
‒ L2 MAC tables are in sync (flooding)
‒ Interface Errors and Statistics
‒ Switch fabric path
 L3 Forwarding
‒ SW and HW FIB entries
‒ Adjacency / Rewrite info
Take Away Points
 It is very critical to determine the flow
experiencing packet loss and find path-
of-the-packet through the network.
 Knowledge of switch hardware and
software architecture expedites the
troubleshooting, and helps for timely
resolution of the problem.
 L2 and L3 forwarding troubleshooting
for Catalyst 6800 is same as for Sup2T-
based Catalyst 6500.
49
Troubleshooting Multicast Forwarding

51
Multicast Troubleshooting
Agenda
• Terminology
• Multicast Replication and Modes
• Multicast Forwarding Troubleshooting

52
Multicast Troubleshooting
Agenda
• Terminology
• Multicast Replication and Modes
• Multicast Forwarding Troubleshooting

53
Terminology
• OIF: Outgoing Interface
• OIL: Outgoing Interface List
• IGMP: Internet Group Management Protocol
• Multicast FIB: Contains the (*,G) and (S,G) entries as well as RPF-VLAN
• Adjacency Table: Contains the rewrite information and MET index
• LTL: Local Target Logic - Forwarding logic for the Catalyst® 6500 / 6800
• MET: Multicast Expansion Table - Hardware table that contains the OIFs
for the (*,G) and (S,G) entries

54
Local Target Logic (LTL)
• Every valid packet that ingresses the Catalyst 6500/6800 will be sent to a
forwarding engine (FE) within the system (DFC or the PFC on the supervisor)
• The FE makes the decision about where to forward the packet or to drop the
packet
• Part of the result of the forwarding decision is a destination LTL index (or
destination index)
• The destination index is used to select the physical port(s) that will forward
the packet
• For multicast, another important part of the forwarding decision is the MET
index

55
Multicast Expansion Table (MET)
maps to port or set of ports

• The MET is memory where the list of OIFs for

the multicast entries are stored
• MET block contains the list of OIFs and the
corresponding destination LTL index for each
• Each replication engine has a separate MET
• MET index from the CEF adjacency can be
used to read the table
• MET tables are independent of the DFC. In
other words, even CFC modules have MET
tables

56
Multicast Troubleshooting
Agenda
• Terminology
• Multicast Replication and Modes
• Multicast Forwarding Troubleshooting

57
Multicast Replication

• Replication: Process of creating copies of packets

• L2 Replication: Creating copies of a packet within a single VLAN (e.g.,

Forwarding a single broadcast packet out all ports within a VLAN)
• Does not require a replication engine

• L3 Replication: Creating copies of a multicast packet for forwarding out each

of the interfaces in an OIL
• Requires a replication engine

• For this multicast discussion, the term Replication will mean L3 Replication

58
 Ingress Replication Mode
Three Packets
Cross Fabric
• Replication engine on ingress module
performs replication for all OIFs
2
• One copy of the original packet is forwarded RE
across the fabric for each of the OIFs
1
• Input and replicated packets get lookup on 3
PFC or ingress DFC Switch
RE
Fabric
• Default to ingress mode when at least one RE
module not capable of egress mode is 4
present in the system
RE
• MET’s on all replication engines are symmetric
or synchronized

RE = Replication Engine

59
 Egress Replication Mode
• Input packets get lookup on ingress DFC, One Packet
replicated packets get lookup on egress DFC Crosses Fabric

• For OIFs on ingress module, local engine performs 2

the replication 1 RE
• For OIFs on other modules, ingress engine
replicates a single copy of packet over fabric to all RE
Switch RE
egress modules Fabric
3
• Engine on egress module performs replication for RE
local OIFs
4
• MET tables on different modules can be
asymmetric
RE = Replication Engine

60
Multicast Troubleshooting
Agenda
• Terminology
• Multicast Replication and Modes
• Multicast Forwarding Troubleshooting

61
Diagram for Troubleshooting Example
Source: 172.16.10.1
Group: 225.1.1.1

Router
Layer 3 Gi1/1 Gi4/1 Receiver
Network 10.10.30.3
VLAN 10 VLAN 30

Gi1/2
Gi4/2
Receiver VLAN 20
L3 Link Receiver
10.10.20.3
10.10.40.3
DUT

• DUT is a Catalyst 6500 with a Sup2T engine

• Module 1 and 4 are WS-X6824-SFP with DFC4-A.
• Server sending 225.1.1.1 stream, received on Gig1/1 in Vlan 10
• Receivers are connected to module 1 and 4, and in vlan 20, 30 and across an L3 link
62
 Multicast Replication Modes
• In classic system (all modules are non-DFC), replication always occurs on the active supervisor engine
• In a fully fabric-enabled system, there are two possible replication modes:
 Ingress replication mode
Use show mls ip multicast
 Egress replication mode capability in older versions

Sup2T#show platform hardware capacity multicast

L3 Multicast Resources Shows that the mode
Replication mode: egress for the system is
Bi-directional PIM Designated Forwarder Table Capacity: 8 Per Vrf
Egress
Bi-directional PIM Designated Forwarder Table usage:
Vrf IPV4 used IPV6 used Total used
Replication capability: Module Capability
1 egress Capabilities of each module in the system. One
4 egress card in the chassis only capable of ingress mode
6 egress cause the mode to move to ingress
MET table Entries: Module Total Used %Used
1 65518 4 1%
4 65518 6 1%
6 32752 2 1%
Multicast LTL Resources
Usage: 38848 Total, 581 Used

63
IGMP Snooping
Membership Reports and L2 Forwarding Table
Use show ip igmp groups [group] to verify that the receivers’ membership reports are
received by the switch If a specific vlan is not shows ONLY
listed, then there is an the last reporter
Sup2T#sh ip igmp groups 225.1.1.1 issue with IGMP SW
IGMP Connected Group Membership
Group Address Interface Uptime Expires Last Reporter Group Accounted
225.1.1.1 Vlan30 01:44:42 00:02:24 10.10.30.5
225.1.1.1 Vlan20 01:44:42 00:02:17 10.10.20.5 shows the
225.1.1.1 GigabitEthernet4/2 01:48:46 00:02:13 10.10.40.3 receivers in the
VLANs and L3
Interfaces
Use show mac-address-table multicast igmp-snooping to display the
IGMP Snooping L2 forwarding table
Sup2T#sh mac address-table multicast igmp-snooping SW
vlan mac/ip address LTL ports
+----+-----------------------------------------+------+-------------- Gig1/2 and Gig4/1 are receivers
20 ( *,225.1.1.1) 0x912 Router Gi1/2 in vlan 20 and 30 respectively
30 ( *,225.1.1.1) 0x914 Router Gi4/1
10 IPv4 OMF 0x90C Router
20 IPv4 OMF 0x90C Router “Router” port indicates that
30 IPv4 OMF 0x90C Router the CPU is an mrouter port

64
Multicast Forwarding
(S,G) Entry in SW
Sup2T#show ip mroute 225.1.1.1 SW
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,
<snip>
Outgoing interface flags: H - Hardware switched, A - Assert winner
Timers: Uptime/Expires
Interface state: Interface, Next-Hop or VCD, State/Mode
(*, 225.1.1.1), 02:02:40/stopped, RP 192.168.100.1, flags: SJC
Incoming interface: Null, RPF nbr 10.10.10.5
Outgoing interface list: RPF neighbor
Vlan30, Forward/Sparse, 01:50:46/00:02:14
(S,G)
Vlan20, Forward/Sparse, 01:50:46/00:02:15
GigabitEthernet4/2, Forward/Sparse, 01:54:50/00:02:11
(172.16.10.1, 225.1.1.1), 01:32:44/00:02:09, flags: JT
Incoming interface: Vlan10, RPF nbr 10.10.10.5
Outgoing interface list: RPF VLAN
OIL GigabitEthernet4/2, Forward/Sparse, 01:32:44/00:02:11
Vlan20, Forward/Sparse, 01:32:44/00:02:15
Vlan30, Forward/Sparse, 01:32:44/00:02:14

65
Multicast Forwarding
Forwarded Multicast Packets
Sup2T#show ip mroute 225.1.1.1 count Make sure that drops are not SW
<snip> incrementing. If RPF drops are
seen, do show ip rpf <src-ip-
Forwarding Counts: Pkt Count/Pkts per second/Avg Pkt Size/Kilobits per second addr> to verify the RPF
Other counts: Total/RPF failed/Other drops(OIF-null, rate-limit etc) information. Also, do show ip
route <src-ip-addr> to verify the
RPF interface for that multicast
Group: 225.1.1.1, Source count: 1, Packets forwarded: 720, Packets received: 720 stream
RP-tree: Forwarding: 3/0/100/0, Other: 3/0/0
Source: 172.16.10.1/32, Forwarding: 717/0/100/0, Other: 717/0/0
Make sure that forwarding packet counts are
Sup2T#show ip mfib 225.1.1.1 count incrementing (updated every 10 seconds)
Forwarding Counts: Pkt Count/Pkts per second/Avg Pkt Size/Kilobits per second
Other counts: Total/RPF failed/Other drops(OIF-null, rate-limit etc)
<snip>
Group: 225.1.1.1 This command is recommended for faster
RP-tree, response, in large-scale deployments.
SW Forwarding: 0/0/0/0, Other: 0/0/0
HW Forwarding: 3/0/100/0, Other: 0/0/0
Source: 172.16.10.1, Packets forwarded in
SW Forwarding: 0/0/0/0, Other: 0/0/0 hardware vs. software
HW Forwarding: 878/0/100/0, Other: 0/0/0
Totals - Source count: 1, Packet count: 881
66
Multicast Forwarding Entry Egress Mode
Primary and Secondary Entries
• The primary entry is used by the ingress forwarding engine for:
• Forwarding to all receivers and mrouters in the ingress VLAN
• Forwarding to all “local” receivers and mrouters on all OIFs in the OIL
• Forwarding a copy of the packet across the switching fabric to egress module(s)

• The secondary (or non-primary) entry is used by the egress forwarding

engines for:
• Forwarding to all “local” receivers and mrouters on all OIFs in the OIL

67
 Multicast Forwarding Entry Egress Mode
Access the DFC using “remote
Closer Look at the Primary Entry login module X” command.

Sup2T-dfc1#sh platform hardware multicast routing ip group 225.1.1.1 detail HW

IPv4 Multicast CEF Entries for VPN#0
<snip> (S,G) RPF VLAN
(172.16.10.1, 225.1.1.1/32)
FIBAddr: 0x40 IOSVPN: 0 RpfType: SglRpfChk SrcRpf: Vl10
CPx: 0 s_star_pri: 1 non-rpf drop: 0
met3: MET index used to retrieve the LTL
PIAdjPtr: 0x38001 Format: IP rdt: off elif: 0xC5409 indices for receivers and mrouters local to
the ingress replication engine.
fltr_en: off idx_sel/bndl_en: 0 dec_ttl: on mtu_idx: 2(1518) met2: MET index used to retrieve the LTL
PV: 1 rwtype: MCAST_L3_RWT_L2_EXPS index used to forward a single copy of the
met3: 0x34 met2: 0x28 multicast packet across the switching fabric.
Packets: 1393 Bytes: 139300
Number of packets/bytes
Primary Entry (PI) Non-primary Entry (NPI) forwarded using this entry.

NPIAdjPtr: 0x38002 Format: IP rdt: on elif: 0xC5409

fltr_en: off idx_sel/bndl_en: 0 dec_ttl: off
PV: 0 rwtype: MCAST_L3_REWRITE
met3: 0x34 met2: 0x0 DestNdx: 0x7FF3
Packets: 0 Bytes:

68
Multicast Forwarding Entry Egress Mode
Closer Look at the Primary Entry (continued)
Continued …… HW

NPIAdjPtr: 0x38002 Format: IP rdt: on elif: 0xC5409

fltr_en: off idx_sel/bndl_en: 0 dec_ttl: off
PV: 0 rwtype: MCAST_L3_REWRITE
met3: 0x34 met2: 0x0 DestNdx: 0x7FF3
Packets: 0 Bytes:
met3 Index (from primary entry)
MET offset: 0x34
OIF AdjPtr Elif CR
+------+----------+--------+---+ Vlan 20 (receiver connected to Gig1/2)
Vl20 0x8014 0x14 1T1
MET offset: 0x28 met2 Index (from primary entry)
OIF AdjPtr Elif CR
+---------+-------+---------+----+ For copy of the packet sent
EDT-34001 0x34001 0x8400A 1T1 via the switching fabric

69
 Multicast Forwarding Entry Egress Mode
Closer Look at the Secondary Entry Access the DFC using “remote
login module X” command.

Sup2T-dfc4#sh platform hardware multicast routing ip group 225.1.1.1 detail HW

IPv4 Multicast CEF Entries for VPN#0
<snip> RPF VLAN
(S,G)
(10.10.10.5, 225.1.1.1/32)
FIBAddr: 0x0A IOSVPN: 0 RpfType: SglRpfChk SrcRpf: Vl10
CPx: 0 s_star_pri: 1 non-rpf drop: 0

PIAdjPtr: 0x54000 Format: IP rdt: off elif: 0xC5409

fltr_en: off idx_sel/bndl_en: 0 dec_ttl: on mtu_idx: 2(1518)
PV: 1 rwtype: MCAST_L3_RWT_L2_EXPS
met3: MET index used to retrieve the LTL
met3: 0xA met2: 0x8 indices for receivers and mrouters local to
Packets: 0 Bytes: 0 the egress replication engine.
met2: MET index used to retrieve the LTL
Primary Entry (PI) index used to forward a single copy of the
Non-primary Entry (NPI)
multicast packet across the switching fabric.
NPIAdjPtr: 0x54001 Format: IP rdt: on elif: 0xC5409 Here, met2 = 0, because egress module will
fltr_en: off idx_sel/bndl_en: 0 dec_ttl: off NOT send anything back to fabric for this
specific (S,G) flow.
PV: 0 rwtype: MCAST_L3_REWRITE
met3: 0xA met2: 0x0 DestNdx: 0x7FF3
Number of packets/bytes
Packets: 1393 Bytes: 139300 forwarded using this entry.

70
Multicast Forwarding Entry Egress Mode
Closer Look at the Secondary Entry (continued)
Continued …… HW

NPIAdjPtr: 0x54001 Format: IP rdt: on elif: 0xC5409

fltr_en: off idx_sel/bndl_en: 0 dec_ttl: off
PV: 0 rwtype: MCAST_L3_REWRITE
met3: 0xA met2: 0x0 DestNdx: 0x7FF3
Packets: 1393 Bytes: 139300

met3 Index
MET offset: 0xA
OIF AdjPtr Elif CR
+-------------+----------+-----------+------------+
Receiver connected to Gig4/2
Gig4/2 0x800C 0x408F 4/T1
across an L3 interface, and a
Vl30 0x801E 0x1E 4/T1 receiver on Gig4/1 in vlan 30
MET offset: 0x8
OIF AdjPtr Elif CR
+-------------+----------+-----------+------------+
EDT-34005 0x5C000 0x840A 4/T
Found 2 entries.

71
 Introduction to
Instant Access(IA) Solution

72
Catalyst Instant Access(IA) Solution
Agenda
• Introduction and Evolution
• Instant Access (FEX) Discovery
• Instant Access (FEX) Verification
• Forwarding on Instant Access Solution

73
 Catalyst Instant Access Evolution
INSTANT ACCESS
VSS
STANDALONE
Si Si

Si Si
VSL
LACP / VSL
PAGP
LACP /
PAGP

ACCESS ACCESS ACCESS INSTANT INSTANT

SWITCH SWITCH SWITCH ACCESS ACCESS
ACCESS
CLIENT CLIENT
SWITCH

74
Traditional Campus with stacking at access

Core 34 Total Devices

Management
(image and
Si Si

configuration)

Si Si Si Si Si Si Si Si
48 Access
Trunks/Port-
Channels

4032 User Ports

Building 1 Building 2 Building 3 Building 4

75
VSS Campus with Stacking

Core 29 Total Devices

for Image and
Configuration
Management

48 Access
Trunks/Port-
Channels

Building 1 Building 2 Building 3 Building 4032

4 User Ports
Catalyst Instant Access
5 Total Devices for
Core Image and
Configuration
Management

Automated Trunk
Configuration
4032 User Ports
Catalyst Instant Access

Core

NO Trunks to Configure from Access to

Distribution
NO Routing Protocols or Spanning-Tree
configuration between Access and Distribution
NO Configuration or Image Management at
Access
Troubleshooting 6800IA Solution

79
Terminologies
• FEX – Fabric Extender
• IA Parent – Instant Access Parent / Controller Switch
• IA Client – Instant Access Client / Cat6k Remote Line Card
• SDP – Switch Discovery Protocol
• SRP – Switch Role Protocol
• SCP – Switch Configuration Protocol
• RSL – Remote Satellite Link (fabric link interconnecting IA Parent with IA Client)
• VIF – Virtual Interface (logical representation of FEX physical ports)
• RPF – Route Path Forwarding
• VNTAG – Virtual Native Tagging
80
Troubleshooting 6800IA solution
Instant Access Components

81
Instant Access (FEX) Discovery
FEX Configuration commands
1. Create a Layer 2 Port-channel 100
VSS# config t
VSS(config)# interface port-channel 100
VSS(config-if)# switchport 2. Configure its mode as fex-fabric

VSS(config-if)# switchport mode fex-fabric

VSS(config-if)# fex associate 110
3. Associate it with a fex-id 110
VSS(config-if)# no shut
VSS(config-if)# exit
VSS(config)# interface range TenGig 1/5/4, TenGig 2/5/4 4. Select the Fabric physical links
VSS(config-if)# switchport
VSS(config-if)# channel-group 100 mode on 5. Bundle them into using mode on
VSS(config-if)# no shut

Repeat the configuration for configuring second FEX Client (with FEX ID 120)

82
Catalyst Instant Access(IA) Solution
Agenda
• Introduction and Evolution
• Instant Access (FEX) Discovery
• Instant Access (FEX) Verification
• Forwarding on Instant Access Solution

83
Instant Access (FEX) Discovery
Verify FEX state
VSS# show fex Current FEX state
FEX FEX FEX FEX FEX Serial
Number Description State Model Serial Numbers

---------------------------------------------------------------------------
FEX ids 110 and 120
110 FEX0110 online C6800IA-48TD FOC1736W1A8

120 FEX0120 online C6800IA-48FPD FOC1736W197

FEX model
FEX States
Init - First SDP exchange
Connect - Control VLAN and CVIF exchange
Registration - FIN (FEX internal Network) allocation, IDPROM, URI Path
Image Download/Version Mismatch – Happens when version mismatch occurs
Registered - FCP Ready
Online – Once Parent receives FCP Ready from FEX Client, it moves to online state
Offline – Seen when FEX Client is removed or disabled
84
 Instant Access (FEX) Discovery
Verify detailed FEX status
VSS# show fex detail Contd..
FEX: 110 Description: FEX0110 state: online FEX: 120 Description: FEX0120 state: online
FEX version: 15.0(2)EX4 FEX version: 15.0(2)EX4
FEX is online Extender Model: C6800IA-48FPD, Extender Serial:
Extender Model: C6800IA-48TD, Extender Serial:
FCW1901A4B2 FOC1736W197
FCP ready: yes FCP ready: yes
Image Version Check: enforced Image Version Check: enforced
Fabric Portchannel Ports: 1 Fabric Portchannel Ports: 2
Fabric port for control traffic: Te2/5/4 Fabric port for control traffic: Te2/5/3
Fabric interface state: Fabric interface state:
Po100 - Interface Up. Po200 - Interface Up.
Te1/5/4 - Interface Up. state: bound Te1/5/3 - Interface Up. state: bound
Te2/5/4 - Interface Up. state: bound Te2/5/3 - Interface Up. state: bound
Contd..
RSL members are bound and up

85
Instant Access (FEX) Discovery
Verify individual FEX members and environment status
VSS#show module fex
Switch Number: 110 Role: FEX
---------------------- -----------------------------
Fex stack number
Mod Ports Card Type Model Serial No.
--- ----- -------------------------------------- ------------------ -----------
1 48 C6800IA 48GE C6800IA-48TD FCW1901A4B2
2 48 C6800IA 48GE C6800IA-48TD FCW1901A49F
3 48 C6800IA 48GE C6800IA-48TD FCW1901A4AY
4 48 C6800IA 48GE C6800IA-48TD FCW1901A496
<snip>

VSS#show environment status fex all FEXswitch model number

Fex 110 Module 1
Power-supply 1: Type : Built-in
Status: on
Fex 110 Module 2
Power-supply 1: Type : Built-in FEX environment status
Status: on
Fex 110 Module 3
Power-supply 1: Type : Built-in
Status: on
<snip>
Fex 110 module 4:
FEX 110 module 4 cooling requirement: 84 cfm
Fex 110 module 1 inlet-1 temperature: 31C
Fex 110 module 2 inlet-1 temperature: 31C
Fex 110 module 3 inlet-1 temperature: 31C
86
Instant Access (FEX) Discovery
Verify virtual slot-map
VSS#show switch virtual slot-map
Virtual Slot to Remote Switch/Physical Slot Mapping Table:

Virtual Remote Physical Module

Slot No Switch No Slot No Uptime
---------+-----------+----------+---------- Switch 1 in FEX 110 stack
17 1 1 - is mapped to vslot 50
18 1 2 -
<snip>
50 110 1 00:13:12
51 110 2 00:13:12
52 110 3 00:13:12
53 110 4 00:13:07
54 120 1 00:12:46 Switch uptime
55 120 2 00:12:46
56 120 3 00:12:46
57 120 5 00:12:46
<snip>

VSS#test scp ping 50

pinging addr 50(0x32)
assigned sap 0x25
addr 50(0x32) is alive Testing if Switch 1 in FEX 110 is
alive

87
 Instant Access (FEX) Discovery
Control Plane – Behind the scene
Switch Discovery Protocol (SDP)
• Fabric Link Discovery
• Switch Discovery
• EtherChannel Link Aggregation
Switch Registration Protocol (SRP)
• Exchange Compatibility information
• IA Client Registration
• IA Client image management
• IA Client OIR
• Stack Member identification & mgmt
Switch Configuration Protocol
• Configuration
• Status
• Statistics
Inter Card Communication (ICC) for
Syslog, QoS, Remote login.
VSS#show platform fex-debug fex 110 sdp
SDP Debug information for FEX : 110 SDP state
---------------------------------------------
Apr 29 2015 17:26:01 SDP is UP on int TenGigabitEthernet2/5/4
Event trace logs for troubleshooting
VSS#show monitor event-trace fex clock 17:26 FEX discovery
*Apr 29 17:26:00.711: general SDP tx params on intf Te2/5/4
updated with vntag 2049
*Apr 29 17:26:11.923: chassis event FEX 110 ready: state registration
*Apr 29 17:26:11.923: SRP response controller: type:1, ID:110,
FCP:0x2/0x5, IP:192.168.1.1/24, MTS:0x0
*Apr 29 17:26:11.923: SRP response FEX: ID:110, FCP:0x4/0x32, IP:192.16
8.1.110/24, MTS:0x0, image ok:Y
*Apr 29 17:26:12.063: chassis event FEX 110 FCP ready: state registered
*Apr 29 17:26:12.087: chassis event FEX 110 ready: state ready
*Apr 29 17:26:12.087: chassis event FEX 110 online: state online
*Apr 29 17:26:12.091: FCP message FCP ID: FEX 110, num:1, switch:0x4,
slot 50
*Apr 29 17:26:12.095: FCP message FCP ID: FEX 110, num:2, switch:0x4,
slot 51
*Apr 29 17:26:12.107: FCP message FCP ID: FEX 110, num:3, switch:0x4,
slot 52
*Apr 29 17:26:12.127: FCP message FCP ID: FEX 110, num:4, switch:0x4,
slot 53
*Apr 29 17:26:17.487: chassis event FEX 110 FCP ready: state online
88
Catalyst Instant Access(IA) Solution
Agenda
• Introduction and Evolution
• Instant Access (FEX) Discovery
• Instant Access (FEX) Verification
• Forwarding on Instant Access Solution

89
 Catalyst
Instant Instant
Access Access
(FEX): Interface Naming
Host Port: Interface Naming Convention

<Interface-type>/<fex-id>/<module>/<submode>/<port>

Si Si
FEX ID
Stack Sub Module FEX Port
101-199

interface GigabitEthernet 110/1/0/1

interface GigabitEthernet 110/2/0/1

90
Instant Access (FEX): Console Access
VSS#attach fex 110
Attach FEX:110 ip:192.168.1.110
Trying 192.168.1.110 ... Open
ˇ˚ˇ˚ˇ˝ˇ˝
FEX-110> Remote into a FEX. Password is cisco.
FEX-110>en
Password: cisco
FEX-110#

FEX-110#show int tenGigabitEthernet 1/0/1

Show commands can be issued from FEX CLI
TenGigabitEthernet1/0/1 is up, line protocol is up (connected)
Hardware is Ten Gigabit Ethernet, address is 0022.bdf4.6633 (bia 0022.bdf4.6633)
MTU 9198 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 10Gb/s, link type is auto, media type is SFP-10GBase-SR
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
<snip>
Configuration mode is not available from FEX CLI.
FEX-110#conf t All FEX related configuration needs to be done
^ from parent 6800 configuration mode.
% Invalid input detected at '^' marker. 91

00
Instant Access (FEX): Platform resource usage
Contd…
VSS#show fex system platform usage Total FEX ports supported is
FEX id usage details FEX ports usage details 2016 in current releases.
Fex-ids inuse: 110, 120 Total FEX ID’s configurable FEX-id Switch-id Ports
Fex-ids online: 110, 120 is 12 in release 15.1(2)SY1 ------ --------- -----
Total Used Free 110 5 192
----- ---- ---- FEX 110 is a stacked FEX 120 6 192
42 2 40 with 4 switches bundled into Total Used Free
Total FEX switches that can
one FEX id ----- ---- ----
be stacked per stack is 5.
FEX slot usage details 2016 384 1632
FEX-id Switch-id Vslot Pslot Status
------ --------- ----- ----- ------ Stack members usage details
110 5 50 1 In-use FEX-id Switch-id Used Free
110 5 51 2 In-use ------ --------- ---- ----
110 5 52 3 In-use 110 5 4 1
110 5 53 4 In-use 120 6 4 1
120 6 54 1 In-use
120 6 55 2 In-use VNTAG MGR Usage
120 6 56 3 In-use -----------------------
120 6 57 5 In-use Max unicast VIFs available 2048
Total Used Reserved Temp-Use/Free Free Total unicast VIFs used 384
----- ---- -------- ------------- ---- Max non-mdest VIFs available 1019
47 8 0 0/5 39 Total non-mdest VIFs used 2
Max mdest VIFs available 16380
Current Temp vslot allowed FEXs: Each FEX module will Total mdest VIFs used 0
consume one vslot id
Contd…

92
Instant Access (FEX): Adding member to stack
VSS#show module fex 110 Stack before adding 4th member
Switch Number: 110 Role: FEX
---------------------- -----------------------------
Mod Ports Card Type Model Serial No.
--- ----- -------------------------------------- ------------------ -----------
1 48 C6800IA 48GE C6800IA-48TD FCW1901A4B2
2 48 C6800IA 48GE C6800IA-48TD FCW1901A49F
3 48 C6800IA 48GE C6800IA-48TD FCW1901A4AY

%STACKMGR-4-SWITCH_ADDED: Switch 4 has been ADDED to the stack (FEX-110)

%SATMGR-SW1-5-FEX_MODULE_ONLINE: FEX 110, module 4 online
%OIR-SW1-6-INSREM: Switch 110 Physical Slot 4 - Module Type LINE_CARD inserted
%STACKMGR-5-SWITCH_READY: Switch 4 is READY (FEX-110)
%DIAG-SW1-6-RUN_MINIMUM: Fex 110 Module 4: Running Minimal Diagnostics...
%DIAG-SW1-6-DIAG_OK: Fex 110 Module 4: Passed Online Diagnostics
%OIR-SW1-6-SP_INSCARD: Card inserted in Switch_number = 110, physical slot 4,
interfaces are now online

VSS#sh module fex 110 Stack Member automatically

Switch Number: 110 Role: FEX Discovered and associated to
---------------------- ----------------------------- FEX like a Line Card
Mod Ports Card Type New member added to stack Model Serial No.
--- ----- -------------------------------------- ------------------ -----------
1 48 C6800IA 48GE C6800IA-48TD FCW1901A4B2
2 48 C6800IA 48GE C6800IA-48TD FCW1901A49F
3 48 C6800IA 48GE C6800IA-48TD FCW1901A4AY
4 48 C6800IA 48GE C6800IA-48TD FCW1901A496
93
Instant Access (FEX): Helpful commands
show fex <fex-id> detail
show switch virtual slot-map
test scp ping <vslot id of fex>
show module fex <fex-id>
show platform fex-debug fex <fex-id> sdp
show monitor event-trace fex clock <hh:mm>
show fex image bundle version
debug fex [error|sdp|srp|init]<fex-id>*
debug switch virtual fexmgr[error|event|packet]<fex-id>*
94
Check in which state FEX is stuck
Get vslot id of fex in question
Check if FEX Internal Network is Up
Check FEX diagnostics have passed
Check if SDP handshake was
successful
Event trace utility to store all FEX
related events at different stages
Check the version and the stack
member details (run with caution)
Turn on common FEX debugs and
restart FEX (run with caution)
Catalyst Instant Access(IA) Solution
Agenda
• Introduction and Evolution
• Instant Access (FEX) Discovery
• Instant Access (FEX) Verification
• Forwarding on Instant Access Solution

95
Catalyst Instant Access Unicast D=1 Unicast to FEX Host Port

VNTAG Multicast P=1 Pointer to Multicast Table on

FEX Client
802.1Qbh
DA[6] SA[6] VNTAG[6] 802.1Q[4] Frame Payload …. CRC[4]

VNTAG ETHER TYPE

D[1] P[1] DVIF [14] L[1] R[1] R[1] R[1] SVIF[12]
(0X8926)

Destination VIF Source VIF

Destination Pointer bit
bit
(multicast) Loopback bit Reserved
96

96
 Catalyst Instant Access
Local Processing of Remote Ports, How?

Ingress Mapping
TAG IA Client Interface
VIF1 IF1
IA Parent
VIF2 IF2 (VIF1) (VIF2)

• Automatically assigned
• One VIF to each host port F101
Switch-3
IF2
IA Client
IF1
• One VIF to each Etherchannel
• One VIF to FEX CPU for Control Channel
• IA Parent VIF = 0 Host-1 Host-2
• Multicast/Broadcast: Pointer to Replication
Table in IA Client Hosts
TAG: Virtual NIC Tag VIF – Virtual Interface
97
Catalyst Instant Access
Packet Walk (IA Client Host Port to IA Parent)

MAC + Payload

IA
VNTAG
SVIF = VIF1 DVIF = 0 Parent
MAC + Payload
VNTAG
SVIF = VIF1 DVIF = 0
F101
MAC + Payload IF1
Switch-3
IA Client
(VIF1)

Host-1

MAC + Payload

VNTAG: Virtual NIC Tag VIF – Virtual Interface

98
Catalyst Instant Access
Packet Walk (IA Parent to IA Client Host Port)

MAC + Payload

VNTAG
SVIF = 0, DVIF = VIF1
IA Parent
MAC + Payload
VNTAG
SVIF = 0, DVIF = VIF1

MAC + Payload F101

Switch-3 IA Client
IF1
(VIF1)

MAC + Payload Host-1

VNTAG: Virtual NIC Tag VIF – Virtual Interface

99
Catalyst Instant Access
Packet Walk (Host 1 to Host 2)

SA=MAC1, DA=MAC2+ Payload SA=MAC1, DA=MAC2+ Payload

VNTAG VNTAG
SVIF = VIF1 DVIF = 0 SVIF = 0, DVIF = VIF2
SA=MAC1, DA=MAC2+ Payload SA=MAC1, DA=MAC2+ Payload

VNTAG VNTAG
SVIF = VIF1 DVIF = 0 SVIF = 0, DVIF = VIF2
F101
Switch-3
SA=MAC1,
SA=MAC1,DA=MAC2+
DA=MAC2+Payload
Payload IF1 IF2 SA=MAC1, DA=MAC2+ Payload
(VIF1) (VIF2)

Host-1 Host-2
MAC1 MAC2
SA=MAC1, DA=MAC2+ Payload
SA=MAC1, DA=MAC2+ Payload

VNTAG: Virtual NIC Tag

100
 Catalyst Instant Access
Packet Walk - Multicast / Broadcast

192.168.1.100, 224.0.255.1
Incoming Interface: FortyGig 5/1 RPF Neighbor 210.20.37.33
Outgoing interface list:
Gigabitethernet 101/1/0/1, Forward/Dense, 0:57:31/0:02:52
Gigabitethernet 101/1/0/2, Forward/Dense, 0:56:55/0:01:28 IA Parent
MAC + Payload

Group VIF
VNTAG, P=1 Outgoing Interface
SVIF =0, DVIF = Group VIF

MAC + Payload
F101
Switch-3
IF2
IA Client
IF1, IF2
IF1
(VIF1) (VIF2)

Host-1 Host-2
MAC + Payload MAC + Payload

Hosts
VNTAG: Virtual NIC Tag VIF – Virtual Interface
101
Catalyst Instant Access
Verifying LTL to VIF map – from VSS
VSS#show run interface g110/1/0/48

interface GigabitEthernet110/1/0/48 FEX port connecting host

switchport
switchport trunk allowed vlan 1
switchport mode access

VSS#show mac address-table interface g110/1/0/48 all detail

Detail option is hidden
Displaying entries from active supervisor:

b line pg: bd mac-address st gm pi cap sec rm rma m nf tr al ag t s fl index

-+----+--+-----+--------------+--+--+--+---+---+--+---+-+--+--+--+--+-+-+--+------- LTL index associated
0 D94 0 1 0000.0200.0300 0 0 0 0 0 0 0 1 0 0 0 49 0 0 0 0x206F with MAC address

Displaying entries from DFC switch [1] linecard [2]:

b line pg: bd mac-address st gm pi cap sec rm rma m nf tr al ag t s fl index

-+----+--+-----+--------------+--+--+--+---+---+--+---+-+--+--+--+--+-+-+--+-------
0 D94 0 1 0000.0200.0300 0 0 0 0 0 0 0 1 0 0 0 45 0 0 0 0x206F

VSS#test platform software switch virtual vntag_mgr vif-map LTL 0x206F detail 0  Status Unavailable
VIF INFO: 1 programming pending
VIF# 112
Type UNICAST VIF VIF used to send packet to FEX 2  programming success
LTL# 206F 4 programming failure
OperStatus# 2 8  MCAST CB Pending
VIF is in hardware 16  Delete pending

102
Catalyst Instant Access
Verifying LTL to VIF map- from 6800IA
VSS#attach fex 110
FEX-101>en

FEX-110#show platform fex ucast-entries | include 112

vif sw_idb portname GPN handle res_index
==== ========== ====================== ==== ========= =========
112 0x7710E04 GigabitEthernet1/0/48 48 0x34 0x3980000
FEX-110#
Host port mapped to VIF 122

FEX-110#sh platform pm if-numbers | include 1/0/48

interface gid gpn lpn port slot unit slun port-type lpn-idb gpn-idb
----------------------------------------------------------------------
Gi1/0/48 48 48 48 1/25 1 48 48 local Yes Yes

6800IA port-asic mapped to host port

FEX-110#sh platform port-asic stats drop port 25 asic 1
Port-asic Port Drop Statistics - Summary
========================================
Port 0 TxQueue Drop Stats: 0
Port 1 TxQueue Drop Stats: 0
<snip>
Queue 0 Check hardware level drops on port
Weight 0 Frames 0 25 on asic 1 on 6800IA
Weight 1 Frames 0
<snip>

103
Troubleshooting Catalyst 6500/6800 Switches
Final Message

Please practice and get familiar with the troubleshooting techniques.

If you don’t use it, you lose it.

Catalyst 6500/6800 is thriving ….. and …..

Innovation Continues !!!

104
Call to Action
• Visit the World of Solutions for
• Cisco Campus
• Walk in Labs
• Technical Solution Clinics

• Meet the Engineer

• Lunch and Learn Topics
• DevNet zone related sessions
Complete Your Online Session Evaluation
• Please complete your online session
evaluations after each session.
Complete 4 session evaluations
& the Overall Conference Evaluation
(available from Thursday)
to receive your Cisco Live T-shirt.

• All surveys can be completed via

the Cisco Live Mobile App or the
Communication Stations
Recommended Sessions # CiscoLive 2016
BRKCRS-3465 – Cisco Catalyst 6800 Switch Architectures

BRKCRS-3035 – Advanced Enterprise Campus Design: Virtual Switching System (VSS)

LTRCRS-2004 - Cisco Catalyst Instant Access - Virtual Switching System (IA - VSS) Lab

BRKARC-2011 - Overview of Packet Capturing Tools in Cisco Switches and Routers

BRKCRS-2501- Campus QoS Design-Simplified

107
Thank you

108