Cisco Catalyst 6500 Switch Architecture

Cisco Catalyst 6500 Switch
Architecture
RST-3465
RST-3465
12523_04_2006_c2 © 2006 Cisco Systems, Inc. All rights reserved. Cisco Public 1
Session Goal
To provide you with a thorough
understanding of the Catalyst® 6500
switching architecture, packet flow, and
key forwarding engine functions
RST-3465
12523_04_2006_c1 © 2006 Cisco Systems, Inc. All rights reserved. Cisco Public 2 2
Agenda
• Chassis Architecture
• Supervisor Engine and Switch Fabric Architecture
• Switching Module Architecture
• Layer 2 Forwarding
• IPv4 Forwarding
• IPv4 Multicast Forwarding
• Security and Feature ACLs
• QoS
• NetFlow
RST-3465
Chassis Architecture
RST-3465
Catalyst 6500 Chassis Architecture
• Modular chassis in variety of form factors

3, 4, 6, 9, and 13
- slot versions
• Enhanced (“E”) chassis offer higher system power capacity and better signal
integrity
3, 4, 6, and -9 slot versions
• Classic switching bus traces/connectors
• Crossbar fabric traces/connectors
• Redundant power supplies
• Fan tray for system cooling
6509 - NEB - A chassis offers
redundant fan trays and air
filtration
• Redundant voltage
termination (VTT)/clock
modules
• Redundant MAC address
EEPROMs
RST-3465
Catalyst 6503/6503E and 6504E
• Slots 1 and 2—Supervisor
engine, or switching module
Power Power
• Other slots—Any switching module Supply Supply
• 2 fabric channels per slot
VTT/Clock Modules EEPROMs
• Power supplies in rear
6503/6503E—Power entry modules (PEMs) in Dual Channels Slot 1
Fan Tray
front of chassis provides power connection
Dual Channels Slot 2
• 950W AC/DC and 1400W AC power Dual Channels Slot 3
supplies for 6503/6503E
• 2700W AC/DC power supplies for
6504E Crossbar Shared Bus
5 RU
4 RU
Note: CEF720 modules
not supported in
Catalyst 6503 (non-E) chassis
RST-3465
Catalyst 6506/6509 and 6506E/6509E
• Slots 1 and 2—Supervisor Engine 2, or VTT/Clock Modules EEPROMs
switching module
• Slots 5 and 6—Supervisor Engine 32/720, or Dual Channels Slot 1
switching module Dual Channels Slot 2
• Other slots—Any switching module
• 2 fabric channels per slot
Fan Tray
• Wide variety of power supplies, from legacy
1000W to new 6000W—E chassis requires at Dual Channels Slot 5
least 2500W PS Dual Channels Slot 6
• NEB-A chassis has vertical slot
alignment, dual fan trays, front-to-back air flow, Dual Channels Slot 7
air filtration system Dual Channels Slot 8
21 RU Crossbar Shared Bus
Power Power
Supply Supply
12 RU
15 RU
RST-3465
Catalyst 6513
VTT/Clock Modules EEPROMs
• Slots 1 and 2—Supervisor Single Channel Slot 1
Engine 2, or switching module
Single Channel Slot 2
• Slots 7 and 8—Supervisor Single Channel Slot 3
Engine 32/720, or switching
module Single Channel Slot 4
• Wide variety of Single Channel Slot 5

power supplies,
Fan Tray
from 2500W to
new 6000W
• 1 fabric channel
slots 1–8 Dual Channels Slot 9
Dual-fabric modules Dual Channels Slot 10
not supported in
slots 1–8! Dual Channels Slot 11
• 2 fabric 19 RU Dual Channels Slot 12
channels Dual Channels Slot 13
slots 9–13 Crossbar Shared Bus
Any switching module
Power Power
RST-3465 Supply Supply
Cisco Public
12523_04_2006_c1 © 2006 Cisco Systems, Inc. All rights reserved. 8
Agenda
• IPv4 Forwarding
• QoS
• NetFlow
RST-3465
Supervisor Engine and Switch Fabric
Architecture
RST-3465
Supervisor 2
• PFC2 forwarding engine • Internal RP and SP bootflash
daughter card (32MB each)
• Switch Processor CPU (300MHz • External PCMCIA flash slot
R7000)
• Supports optional Switch
• Optional MSFC2 daughter card Fabric Module (SFM)/SFM2
with Route Processor CPU
• 2 x 1GE GBIC uplink ports
(300MHz R7000)
• 256MB/256MB (Sup2) or
256MB/512MB (Sup2U) DRAM
Supported from Cisco IOS 12.1(5c)EX and Catalyst OS 6.1(1)/12.1(3a)E1

RST-3465
Supervisor 2 / PFC2 Architecture
QoS TCAM FIB contains IPv4 RP CPU runs L3 SP CPU runs L2
contains prefix entries protocols and protocols and
QoS ACL maintains control manages hardware
entries plane state
Supervisor 2 Baseboard GbE
ACL TCAM MSFC2 Daughter Card Uplinks
ADJ contains
contains rewrite info
security RP (MSFC2)
DRAM 1 Gbps
and feature NetFlow table for CPU
ACL entries stats and features
Port ASIC
QoS
FIB SP (NMP)
TCAM DRAM 1 Gbps
TCAM CPU
ACL
TCAM L2/L4 Layer 3 ADJ
LCDBUS
Engine Engine LCRBUS
NetFlow
L2
CAM Replication
PFC2 Fabric Interface Engine
Bus
Daughter Replication
Interface
Card engine for
8 Gbps
MET
multicast/
SPAN
L2 CAM DBUS
contains 16 Gbps
RBUS To SFM/SFM2 Interface to
Bus fabric and bus
MAC entries
RST-3465 EOBC
Supervisor 720
• 720Gbps crossbar fabric • 512/512MB (3A/B) or 1/1GB
(3BXL) DRAM
• PFC3 forwarding engine
daughter card • Internal RP and SP bootflash
(64MB each)
• Integrated RP/SP CPUs on
MSFC3 daughter card (600MHz • Optional 512MB CF bootflash
MIPS) upgrade for SP (WS-CF-UPG=)
• Dual external compact flash
slots
• 2 x GbE uplink ports—
2 x SFP <or>
1 x SFP and 1 x 10/100/1000
Supported from Cisco IOS 12.2(14)SX and Catalyst OS 8.1(1)/12.2(14)SX2

RST-3465
Supervisor 720 / PFC3 Architecture
RP and SP both
Fabric interface sit on MSFC3
Addition of ACL and QoS
and replication CPU daughter
ACL TCAM classification move to
engine combined card
counters L3/4 engine
GbE Uplinks
Supervisor 720 Baseboard CPU Daughter Card
Counter QoS FIB 1 Gbps RP (MSFC3) DRAM
FPGA ADJ
TCAM TCAM Port ASIC CPU
(B/BXL Only)
1 Gbps SP (NMP) DRAM
ACL L3/4 NetFlow CPU
TCAM Engine MET
Fabric Integrated
L2 Engine PFC3
20 Gbps 720 Gbps
L2 Daughter Interface/
Replication Switch Fabric
CAM Card
Engine
17 x 20 Gbps
Fabric
…
Channels
L2 CAM moved Crossbar switch
DBUS
on-chip for RBUS fabric integrated
higher 16 Gbps EOBC on supervisor
Bus baseboard
RST-3465
performance
Supervisor 32
• Classic supervisor—no fabric, uses
16Gig bus only
• PFC3B forwarding engine daughter
card
• SP CPU (400MHz Sibyte) 2 10GE Xenpak +
1 10/100/1000 RJ-45 uplink ports
• MSFC2a routing engine
• 256MB/256MB DRAM (512MB/512MB
with non-$0 feature set)
• Internal CF bootdisk (256MB) and
MSFC2A bootflash (64MB)
• External CF slot
• Uplink options:
8 1GE SFP +
8 SFP + 1 10/100/1000 1 10/100/1000 RJ-45
uplink ports
2 10GE + 1 10/100/1000
Supported from Cisco IOS 12.2(18)SXF and Catalyst OS 8.4(1)/12.2(17)SXB7

RST-3465
Supervisor 32-GE / PFC3 Architecture
PFC3 exactly the
same as on
Supervisor 720
Supervisor Engine 32 Baseboard

GbE Uplinks
Counter QoS FIB
ADJ
FPGA TCAM TCAM SP CPU DRAM
1 Gbps
Port ASIC
L3/4 1 Gbps DRAM
ACL NetFlow RP CPU
TCAM Engine
MSFC2a Daughter Card
L2 Engine PFC3
L2 Daughter Replication
CAM Card Engine
MET WS-SUP32-GE-3B
DBUS
16 Gbps
RBUS Bus attached only;
EOBC
Bus no fabric support
RST-3465
Supervisor 32-10GE / PFC3 Architecture
Dual port ASICs to
support two 10GE
interfaces
Supervisor Engine 32 Baseboard

10GE Uplinks
Counter QoS FIB
ADJ
FPGA TCAM TCAM SP CPU DRAM
Port Port 1 Gbps
ASIC ASIC
L3/4 1 Gbps DRAM
ACL NetFlow RP CPU
TCAM Engine
FPGA
MSFC2a Daughter Card
MUX
L2 Engine PFC3
L2 Daughter Replication
CAM Card Engine
MET WS-SUP32-10GE-3B
DBUS
RBUS
16 Gbps EOBC
Bus
RST-3465
Supervisor Chassis Requirements
Supervisor 720 and Supervisor 32 require:
• Catalyst 6500 or 6500-E chassis
• High speed fan tray (FAN2/E-FAN)
• 2500W power supply (AC or DC) or greater
3000W supply recommended for new deployments
• Specific chassis slots:

Slot 1 or 2 in 3/4 slot
Slot 5 or 6 in 6/9 slot
Slot 7 or 8 in 13 slot
RST-3465
Crossbar Switch Fabric
• Provides multiple conflict-free paths between switching
modules
Dedicated bandwidth per slot
Compare to system bus which is shared by all bus-attached modules
• 18 fabric channels in total

• Two fabric channels per slot in 6503/6504/6506/6509
• In 6513:
One fabric channel slots 1–8
Two fabric channels slots 9–13
“Dual-fabric channel” modules not supported in slots 1–8 of 6513
RST-3465
Switch Fabric Module and SFM2
• 256 Gbps crossbar switch fabric
• Works with Supervisor 2 and CEF256/dCEF256 modules
• Fabric channels run at 8 Gbps full duplex
8 Gbps in/8 Gbps out per channel
• Fabric module occupies a full slot

6506/6509—Slots 5 and 6
6513—Slots 7 and 8
• SFM—Supports 6506 and 6509

(and E-versions)
• SFM2—Supports 6506, 6509,
and 6513 (and E-versions)
• Not supported in 6503/6504
RST-3465
Supervisor 720 Switch Fabric
• 720 Gbps crossbar switch fabric
• Integrated on Supervisor 720 baseboard
• Fabric channels run at 20 Gbps
full duplex
20 Gbps in/20 Gbps out per channel
• Works with all fabric-capable modules
Fabric channels auto-sync speed on
per-slot basis (8 Gbps or 20Gbps)
RST-3465
Monitoring Fabric Status and Utilization
• Cisco IOS: show fabric [active | channel-counters |

errors | fpoe | medusa | status | switching-mode
| utilization]
• Catalyst OS: show fabric {channel {counters |
switchmode | utilization} | status}
6506#show fabric utilization

slot channel speed Ingress % Egress %
1 0 8G 22 23
2 0 8G 4 9
3 0 20G 0 1
3 1 20G 11 12
4 0 20G 0 1
4 1 20G 10 13
6 0 20G 0 1
6506#
RST-3465
Monitoring System Bus Utilization
• Monitor the traditional Catalyst 6500 bus when

using:
Classic modules
Centralized forwarding with a fabric
• Cisco IOS: show catalyst6000 traffic-meter

• Catalyst OS: show traffic
6506#show catalyst6000 traffic-meter
traffic meter = 7% Never cleared

peak = 46% reached at 08:07:50 PST Fri Dec 30 2005
6506#
RST-3465
Policy Feature Cards
• Mandatory daughter card for supervisor engine
• Provides the key components enabling high-performance
hardware packet processing
• Supervisor 2 supports PFC2
• Supervisor 32 supports PFC3B
• Supervisor 720 supports:
PFC3A
PFC3B
PFC3BXL
RST-3465
Policy Feature Cards (Cont.)
Key hardware-enabled features:
• Layer 2 switching
• IPv4 unicast forwarding
• IPv4 multicast forwarding
• Security ACLs
• QoS/policing
• NetFlow accounting
PFC3 also supports:
• IPv6, MPLS*/VRF-lite, Bidir PIM, NAT/PAT, GRE/v6
tunnels, CoPP
RST-3465
* MPLS on 3B/3BXL only
High-Level Forwarding Engine Logic
Frame Input Layer 2 Ingress
received lookup FIB TCAM Forwarding
Engine
Layer 2 Table FIB lookup
ACL TCAM
Router Yes Input QoS Input ACL
MAC? lookup lookup
No QoS TCAM NetFlow Table
NetFlow lookup
Input QoS
lookup
QoS TCAM
Input ACL Output QoS Output ACL Output Layer 2

lookup lookup* lookup lookup
ACL TCAM QoS TCAM ACL TCAM Layer 2 Table
Output QoS Output ACL Transmit

Bridged NetFlow
lookup* lookup frame
RST-3465
NetFlow Table QoS TCAM ACL TCAM *PFC3 only
PFC TCAM Technology
• TCAM—Ternary Content Value 1
Addressable Memory Value 2
Mask 1 Value 3
• Leveraged heavily in Catalyst
Value 4
6500
Value 5
FIB, ACL, QoS, NetFlow all utilize Value 6
TCAM memory
Value 7
• All entries accessed in Value 8
parallel—fixed performance
Value 1
independent of number of
Value 2
entries
Mask 2 Value 3
• Memory consists of groups of Value 4
values and associated masks
Value 5
8:1 ratio of values to masks
Value 6
• Masks are used to “wildcard” Value 7
some portion of values
Value 8
Masks Values
RST-3465
Generic TCAM Lookup Logic
1 3 Compare
Generate
Packet Fields Lookup
Key
01101010
011010xx
0110xxxx 110110xx 1
Lookup Key 000111xx 2
2
11111100 101101xx 3
100111xx 4
000000xx 5
1. Relevant fields read from 010010xx 6
contents of packet 1=“Compare”
111111xx 7
0=“Mask”
2. Lookup key created 001100xx 8
3. As lookup key compared to value 0111xxxx 1

entries, associated mask applied 1011xxxx 2
4. Longest match returns result 1101xxxx 3

11110000
HIT! 0110xxxx 4 Result
Result format varies depending on lookup type
1110xxxx 45
0011xxxx 6
0000xxxx 7
1000xxxx 8
RST-3465
12523_04_2006_c1 © 2006 Cisco Systems, Inc. All rights reserved.
Masks Values Cisco Public 28
Agenda
• IPv4 Forwarding
• QoS
• NetFlow
RST-3465
Switching Module Architecture
RST-3465
Classic Module
DBUS
Example: WS-X6416-GBIC
RBUS
Classic Module
Port Port Port Port

ASIC ASIC ASIC ASIC
4xGE 4xGE 4xGE 4xGE
DBUS
Port ASICs for physical RBUS
connectivity, buffering,
and queueing Classic Module
Port
ASIC
48x10/100
Example: WS-X6148A-RJ-45
RST-3465
CEF256 Module
Example: WS-X6516-GBIC
DBUS 8Gbps Fabric

RBUS Channel
CEF256
Module
Fabric
Interface
Fabric interface LCDBUS

to interface with LCRBUS
fabric and bus
Replication
MET
Engine Port Port Port Port
ASIC ASIC ASIC ASIC
4xGE 4xGE 4xGE 4xGE
Replication engine for

local SPAN/multicast Local linecard
replication bus for ASIC
interconnection
RST-3465
CEF256 Module with DFC
Example: WS-X6516-GBIC with WS-F6K-DFC
Layer 2/4 Engine for
L2 and ACL/QoS
lookups
8Gbps Fabric
Channel
Layer 3
Engine for
CEF256 FIB/Adj and
Layer 2/4 L3 Module NetFlow
Fabric lookups
Engine Engine with DFC
Interface
DFC
LCDBUS
LCRBUS
Replication
MET
Engine Port Port Port Port
ASIC ASIC ASIC ASIC
4xGE 4xGE 4xGE 4xGE
RST-3465
CEF720 Module
Example: WS-X6748-SFP
DBUS
20Gbps Fabric 20Gbps Fabric
RBUS
Channel Channel
Complex A Complex B CEF720

Module
Fabric Bus Bus Fabric
Interface & Interface Interface Interface &
MET Replication Replication MET
Engine CFC Engine
Port Port Port Port

ASIC ASIC ASIC ASIC
12xGE 12xGE 12xGE 12xGE
Combined fabric
interface and
replication engine
Transparent bus
interface
Bus interface for control data only!!

RST-3465
CEF720 Module with DFC3
Example: WS-X6748-SFP with WS-F6700-DFC3B
20Gbps Fabric 20Gbps Fabric

Channel Channel
Complex A Complex B CEF720

Module
Fabric Layer 2 Layer 2 Fabric
with DFC3
Interface & Engine Engine Interface &
MET Replication Replication MET
Engine L3/4 Engine

Engine
Port Port DFC3 Port Port
ASIC ASIC ASIC ASIC
12xGE 12xGE 12xGE 12xGE
Layer3/4
Engine for
FIB/Adj, ACL, Layer 2
QoS and Engine
NetFlow for L2
lookups lookups
RST-3465
Distributed Forwarding
• One or more modules have local forwarding engine
(DFC—Distributed Forwarding Card)
• Central engine and distributed engines perform different lookups
independently and simultaneously
• Implementation is fully distributed
All hardware from PFC is present on the DFC
Full Layer 2, Layer 3, ACL/QoS information downloaded from Supervisor
Ingress DFC performs all lookups locally
• Deterministic, highly scalable—Not flow-based

• NOT just for local switching—
destination interface irrelevant
• DFCs always require Cisco IOS
software and a switch fabric
RST-3465
Distributed Forwarding Cards
• DFCs work in conjunction with specific
supervisor
DFC works with PFC2 on Supervisor 2
DFC3A/3B/3BXL works with PFC3 on Supervisor 720
• PFC/DFC “major” module version must be identical
PFC/DFC “minor” module version mismatch supported in lowest common denominator mode
Example: System with PFC3B and DFC3As runs in PFC3A mode
• DFC is optional daughter card for CEF256 modules
• DFC3 is optional daughter card for CEF256/CEF720 modules
Several flavors and form factors available
• WS-X6816-GBIC module REQUIRES either DFC or DFC3
• Local CPU for managing hardware tables
• Use remote login module command to access DFC console
Commands available on DFC console for troubleshooting use, under direction from Cisco
TAC/escalation
RST-3465
Centralized Forwarding
L3/4 Supervisor Red

Engine Engine 32 D
Port Port4
Classic
L2 Engine ASIC ASIC
Module B
2
3 PFC3
DBUS
RBUS
Source S
Classic Port Port Destination D
Module A ASIC ASIC Blue VLAN
1
Red VLAN
Blue S
Entire Packet
Packet Header
RST-3465
Centralized Forwarding with Fabric
Red
D
Supervisor Port Port

L3/4
Engine 720 ASIC ASIC
Engine
LCRBUS
LCDBUS
L2 Engine 720Gbps
Switch 6
Fabric
3 8Gbps CEF256
PFC3 Fabric Interface
Module B
DBUS
RBUS
2
Source S
Fabric 5 8Gbps CEF256 Destination D
Interface
4 Module A
Blue VLAN
LCDBUS Red VLAN
LCRBUS
Entire Packet
Port Port
ASIC ASIC Packet Header
1
Blue S
RST-3465
Distributed Forwarding
Red
D
Port Port
CEF720
ASIC ASIC DFC3
Supervisor Engine 720 L3/4 Module B
Engine w/DFC3
5
720Gbps Fabric Interface/
PFC3 Switch 20Gbps Replication Layer 2
Fabric Engine Engine
20Gbps
CEF720 Source S
4 Module A
Fabric Interface/ 2 Layer 2 Destination D
w/DFC3
Replication 3Engine Blue VLAN
Engine Red VLAN
L3/4
Port Port Engine
DFC3 Entire Packet
ASIC ASIC
Packet Header
1
Blue S
RST-3465
Agenda
• IPv4 Forwarding
• QoS
• NetFlow
RST-3465
Layer 2 Forwarding
RST-3465
Layer 2 Lookups
Engine
ACL TCAM
MAC? lookup lookup
NetFlow lookup
Input QoS
lookup
QoS TCAM

lookup lookup lookup lookup

Bridged NetFlow
lookup lookup frame
RST-3465
NetFlow Table QoS TCAM ACL TCAM
Layer 2 Forwarding
• Layer 2 forwarding based on {VLAN, MAC} pairs
Same MAC can be learned in multiple VLANs
• MAC learning fully hardware based

CPU not involved in learning
• PFC and DFCs have copies of MAC table

Refreshing of entries based on “seeing” traffic—forwarding engines age
entries independently
New learns on one forwarding engine communicated to other engines
• MAC table size:

128K entries on PFC2 (32K effective)
64K entries on PFC3 (32K effective)
RST-3465
Layer 2 Forwarding Logic
Frame received
Layer 2 Table Layer 2 Table

SMAC lookup DMAC lookup
Learn New MAC? Router MAC? L3 forwarding

Yes Yes
Layer 2 Table No No
Update entry Known MAC? L2 forwarding

Yes
Layer 2 Table
No
L2 flooding
RST-3465
Layer 2 Forwarding Table Design
PFC2
PFC3
16 pages 4096 rows
16384 rows
MAC Table
8 pages
RST-3465
MAC Table 16K*8=128K entries 4K*16=64K entries
PFC2 Layer 2 Lookup
Frame
1
16384 rows
Lookup Key
2
VLAN MAC Address
10 | 0000.aaaa.aaaa
Destination
5 4000 interface(s)
40| 20
3233.1111.3333
| 0000.1111.2222
| 0000.cccc.cccc
Compare 111 |
3999 9000.8000.7000
| 9090.9090.9090
100 DMAC lookup
3 10| 0000.1111.1111
| 0000.bbbb.bbbb
2101 |
44444 4334.5445.6556
| 6666.6666.6666
| 0100.5e01.0101
10 | 0000.aaaa.aaaa 6
Hash Function HIT!
44 | 2468.ace0.2468
30 | 0000.dddd.dddd SMAC lookup
Update
Starting Page Entry
and Row 4
8 pages MAC Table

RST-3465
PFC3 Layer 2 Lookup
Frame
1
Lookup Key
2
VLAN MAC Address
10 | 0000.aaaa.aaaa
DMAC lookup
Destination
5 interface(s)
3 Compare
Hash Function
20 | 0000.cccc.cccc 4096 rows

MAC Table 4 10 | 0000.bbbb.bbbb
Row 10 | 0000.aaaa.aaaa 6
HIT!
30 | 0000.dddd.dddd
16 pages
MAC Table
Update
Entry
SMAC lookup
RST-3465
Displaying the Layer 2 Table
• Cisco IOS: show mac-address-table
• Catalyst OS: show cam
6509#show mac-address-table dynamic vlan 30

Codes: * - primary entry
vlan mac address type learn qos ports

------+----------------+--------+-----+---+-----------------------
* 30 0003.a088.c408 dynamic Yes -- Fa3/18
* 30 0012.d949.04d2 dynamic Yes -- Gi5/1
* 30 0003.a08a.15f3 dynamic Yes -- Fa3/24
* 30 0090.a400.1850 dynamic Yes -- Fa3/14
* 30 0003.a08a.15f9 dynamic Yes -- Fa3/25
<…>
6509#
RST-3465
Agenda
• IPv4 Forwarding
• QoS
• NetFlow
RST-3465
IPv4 Forwarding
RST-3465
IPv4 Lookups
Engine
ACL TCAM
MAC? lookup lookup
NetFlow lookup
Input QoS
lookup
QoS TCAM


Bridged NetFlow
lookup lookup frame
RST-3465
Hardware-Based CEF
• Catalyst 6500 leverages existing software Cisco Express
Forwarding (CEF) model
• Supervisor 2, Supervisor 32, Supervisor 720 extend CEF to
hardware
• What is CEF, in a nutshell?
Boil down the routing table = FIB table
Boil down the ARP table = adjacency table
• FIB table contains IP prefixes

• Adjacency table contains next-hop
information
RST-3465
Hardware-Based CEF (Cont.)
• Decouples control plane and data plane
Forwarding tables built on control plane
Tables downloaded to hardware for data plane forwarding
• Hardware CEF process:

FIB lookup based on destination prefix (longest-match)
FIB “hit” returns adjacency, adjacency contains rewrite
information (next-hop)
ACL, QoS, and NetFlow lookups occur in parallel and affect final
result
RST-3465
FIB TCAM and Adjacency Entries
FIB:
• IPv4 entries logically arranged from 172.20.45.1
most to least specific
10.1.1.100
• 0/0 default entry terminates unicast MASK (/32)
FIB entries … IF, MACs, MTU
• Overall FIB hardware shared by 10.1.3.0
IF, MACs, MTU
IPv4 unicast 10.1.2.0
IPv4 multicast MASK (/24) IF, MACs, MTU
IPv6 unicast …
IF, MACs, MTU
IPv6 multicast 10.1.0.0
MPLS 172.16.0.0
Adjacency table: MASK (/16)
…
• Hardware adjacency table also Adjacency Table
shared among protocols 0.0.0.0
MASK (/0)
• Actual adjacency table entries are
NOT shared FIB TCAM
RST-3465
IPv4 FIB TCAM Lookup
Compare 3
Generate
Lookup
Key
10.1.1.10
10.1.1.xx
2 10.1.1.2 1
Lookup Key
DIP 10.1.1.3 2
1 10.1.1.10 FFFFFFFF 10.1.1.4 3 Flow Data
Packet 10.10.0.10 4
10.10.0.100 5
10.10.0.33 6 IF, MACs, MTU
/32 entries 10.100.1.1 Load-Sharing
7
(compare all Hash IF, MACs, MTU
bits) 10.100.1.2 8 Offset
5
6 IF, MACs, MTU
10.1.2.xx 1
10.1.3.xx 2 IF, MACs, MTU
FFFFFF00 10.10.100.xx 3
HIT! 10.1.1.xx 4 Adj Index
10.100.1.xx 45
/24 entries
(mask last Result
10.10.0.xx 6 Adjacency Table
octet)
10.100.1.xx 7
8
RST-3465
12523_04_2006_c1 Masks FIB TCAM
© 2006 Cisco Systems, Inc. All rights reserved. Values Cisco Public 56
Displaying IPv4 Forwarding Summary
Information
• Cisco IOS: 6509-neb#show mls cef summary

show mls cef summary
Total routes: 8309
show mls cef statistics
IPv4 unicast routes: 5948
show mls statistics
IPv4 Multicast routes: 2359
show mls cef hardware
MPLS routes: 0
• Catalyst OS: IPv6 unicast routes: 0
show mls cef IPv6 multicast routes: 0
show mls EoM routes: 0
6509-neb#
RST-3465
Displaying Hardware IPv4 Prefix Entries
6509-neb#show mls cef
Codes: decap - Decapsulation, + - Push Label
Index Prefix Adjacency
64 127.0.0.51/32 receive
• Cisco IOS: show mls
65 127.0.0.0/32 receive
66 127.255.255.255/32 receive
cef
67 0.0.0.0/32 receive • Catalyst OS: show mls
68 255.255.255.255/32 receive entry cef ip
75 10.10.1.1/32 receive
76 10.10.1.0/32 receive
77 10.10.1.255/32 receive
78 10.10.1.2/32 Gi1/1, 0030.f272.31fe
3200 224.0.0.0/24 receive
3201 10.10.1.0/24 glean
3202 10.100.0.0/24 Gi1/1, 0030.f272.31fe
3203 10.100.1.0/24 Gi1/1, 0030.f272.31fe
3204 10.100.2.0/24 Gi1/1, 0030.f272.31fe
3205 10.100.3.0/24 Gi1/1, 0030.f272.31fe
<…>
RST-3465
Displaying Detailed Hardware Entries
• Cisco IOS:
show mls cef <prefix> [detail]
show mls cef adjacency [entry <entry> [detail]]
• Catalyst OS:
show mls entry cef ip <prefix/mask> [adjacency]
6509-neb#show mls cef 10.100.20.0 detail

<…>
M(3222 ): E | 1 FFF 0 0 0 0 255.255.255.0
V(3222 ): 8 | 1 0 0 0 0 0 10.100.20.0 (A:98304 ,P:1,D:0,m:0 ,B:0 )
6509-neb#show mls cef adjacency entry 98304

Index: 98304 smac: 000f.2340.5dc0, dmac: 0030.f272.31fe
mtu: 1518, vlan: 1019, dindex: 0x0, l3rw_vld: 1
packets: 4203, bytes: 268992
6509-neb#
RST-3465
Finding the Longest-Match Prefix Entry
• Cisco IOS: show mls cef lookup <ip_address> [detail]
6509-neb#show mls cef 10.101.1.0

6509-neb#show mls cef lookup 10.101.1.0

3203 10.101.0.0/16 Gi2/12, 0007.b30a.8bfc
6509-neb#
RST-3465
IPv4 CEF Load Sharing
• Up to 8 hardware load-sharing paths per
prefix
• Use maximum-paths command in routing
protocols to control number of load-sharing 10.10.0.0/16
via Rtr-A
paths via Rtr-B
• IPv4 CEF load-sharing is per-IP flow
• Per-packet load-balancing NOT supported
• Load-sharing based on Source and
Destination IP addresses by default A B
“Unique ID” in PFC3 prevents polarization
• Configuration option supports inclusion of L4
ports in the hash 10.10.0.0/16
mls ip cef load-sharing full
• Unique ID not included in hash in “full” mode
RST-3465
Load-Sharing Prefix Entry Example
• show mls cef

• show mls cef lookup
6509-neb#show mls cef lookup 10.100.20.1

3222 10.100.20.0/24 Gi1/1, 0030.f272.31fe
Gi1/2, 0008.7ca8.484c
Gi2/1, 000e.382d.0b90
Gi2/2, 000d.6550.a8ea
6509-neb#
RST-3465
Identifying the Load-Sharing Path
show mls cef exact-route
6509-neb#show mls cef exact-route 10.77.17.8 10.100.20.199
Interface: Gi1/1, Next Hop: 10.10.1.2, Vlan: 1019, Destination Mac: 0030.f272.31fe
6509-neb#show mls cef exact-route 10.44.91.111 10.100.20.199
Interface: Gi2/2, Next Hop: 10.40.1.2, Vlan: 1018, Destination Mac: 000d.6550.a8ea
6509-neb#
RST-3465
IPv4 Unicast RPF Check
6500 Routing Table
Prefix Next Hop Interface
10.255.0.0/16 10.10.1.1 gig 1/1
g1/1 10.20.1.1 gig 1/2
10.30.1.1 gig 2/1
10.40.1.1 gig 2/2
10.20.0.0/16 10.20.1.1 gig 6/3
g1/2
10.255.0.0/16 g2/1 10.20.0.0/16

gig 6/3
g2/2
Supervisor 2: Supervisor 720/Supervisor 32:

• One RPF interface per prefix in hardware • Up to 6 RPF interfaces per prefix in hardware
• Enabling uRPF check halves available FIB • Enabling does not affect available FIB entries
TCAM (128K entries) • Two reverse-path interfaces for all prefixes
• Four user-configurable “multipath interface
groups” to define additional interfaces for
uRPF
Gotcha: System supports only a global uRPF mode—strict or loose—last configured mode overrides
Gotcha: uRPF with exception ACL not recommended due to software processing
RST-3465
Verifying uRPF Check Configuration
• show mls cef ip rpf [<prefix>]

(PFC3 only)
6509#show mls cef ip rpf Global uRPF check
RPF global mode: strict mode
RPF mpath mode: punt
Index Interfaces
Global uRPF
-------+---------------------------------------- multipath mode
0
1
2 uRPF interface groups
3 (not configured)
6509#show mls cef ip rpf 192.168.1.0
RPF information for prefix 192.168.1.0
uRPF check performed in the hardware for interfaces:
Vlan776
Vlan777 uRPF details for
uRPF check punted to software for interfaces: specific IP prefix
uRPF check disabled for interfaces:
6509#
RST-3465
Agenda
• IPv4 Forwarding
• QoS
• NetFlow
RST-3465
IPv4 Multicast Forwarding
RST-3465
IPv4 Multicast Lookups
Engine
ACL TCAM
MAC? lookup lookup
NetFlow lookup
Input QoS
lookup
QoS TCAM


Bridged NetFlow
lookup lookup frame
RST-3465
IPv4 Multicast Forwarding
• Central and distributed IPv4 multicast
hardware forwarding
• Distributed multicast replication with
appropriate switching modules†
• PIM-SSM and PIM-SM forwarding in
hardware
• BiDir-PIM forwarding in hardware‡
• Off-loads majority of forwarding tasks from
RP CPU
† Supervisor 2/SFM and Supervisor 720 only, with fabric-enabled modules

‡ Supervisor 32 and Supervisor 720 only
RST-3465
Multicast Forwarding Tables
• RP CPU derives 3 key data structures from
multicast routing table
Multicast FIB—Consists of (S,G) and (*,G) entries, and RPF VLAN
Adjacency table—Contains rewrite MAC and MET index
Multicast Expansion Table (MET)—Contains output interface lists
(OILs), i.e., lists of interfaces requiring replication
• RP CPU downloads tables to SP CPU

• SP CPU installs tables in the appropriate
hardware
Multicast FIB and adjacency tables installed in PFC/DFC
hardware
MET installed in replication engines
• SP CPU also maintains L2 table for IGMP

snooping
RST-3465
Multicast Hardware Entries
• FIB
MAC, MET Index
IPv4 multicast entries arranged 172.21.4.19, 225.3.3.3
logically from most to least specific 10.1.44.199, 240.9.8.1 MAC, MET Index
10.1.1.1, 239.1.1.1
• Adjacency table MAC, MET Index
…
Different format than unicast MAC, MET Index
MASK (S,G) /32
Key piece of data is MET index …
*, 234.0.1.1
• MET
MASK BiDir Entries
Contains OILs for multicast routes
… Adjacency Table
Memory resident on replication 10.1.1.0, 224.0.0.0
engines (not PFC/DFC) OIL #1
MASK IF 224/4 Entries
… OIL #2
OIL #3
*, 229.0.1.1
MASK PIM-SM (*,G) /32 OIL #4
FIB TCAM
RST-3465
MET
Multicast FIB TCAM Lookup
Compare 3
Generate
Lookup
Key
10.1.1.10, 239.1.1.1
10.1.1.10, 239.1.1.1 1
Lookup Key 2
10.1.1.10, 239.1.1.1 2
FFFFFFFF FFFFFFFF 10.1.1.10, 239.1.1.1 3

S,G
1 10.1.1.10, 239.1.1.1 10.1.1.10, 239.1.1.1 4
Result
Multicast 10.1.1.10, 239.1.1.1 5
Packet 4 Adj Index
HIT! 10.1.1.10, 239.1.1.1 6
RPF VLAN
10.1.1.10, 239.1.1.1 7
10.1.1.10, 239.1.1.1 8
S,G compares
all bits in SIP Masks FIB TCAM Values
and GIP
Replication Engine(s)
MAC, MET Index
OIL #1
OIL #2 MAC, MET Index
6 5
OIL #3 MAC, MET Index
OIL #4
MAC, MET Index
MET
RST-3465 Adjacency Table Cisco Public 72
Displaying Summary Hardware Multicast
Information
• Cisco IOS: show mls ip 6506#show mls ip multicast summary

multicast summary 21210 MMLS entries using 3394656 bytes of memory
• show mls ip multicast Number of partial hardware-switched flows: 0
statistics Number of complete hardware-switched flows: 21210
• Catalyst OS: show mls
multicast
Directly connected subnet entry install is enabled
Hardware shortcuts for mvpn mroutes supported
Current mode of replication is Ingress
Auto-detection of replication mode is enabled
Consistency checker is enabled
Bidir gm-scan-interval: 10
6506#
RST-3465
Displaying Hardware Multicast
Forwarding Entries
• Cisco IOS: show mls ip multicast

• Catalyst OS: show mls multicast entry
6506#show mls ip multicast

Multicast hardware switched flows:
(10.3.1.100, 239.1.1.100) Incoming interface: Gi3/1, Packets switched: 720396460
Hardware switched outgoing interfaces:
Gi3/2 Vlan100 Vlan150 Gi4/1 Gi4/2 Vlan200
RPF-MFD installed
(10.3.1.103, 230.100.1.1) Incoming interface: Gi3/1, Packets switched: 443201

Hardware switched outgoing interfaces:
Gi3/2 Gi4/1
RPF-MFD installed
<…>
For more details, attend:
“RST-3262: Catalyst 6500 IP Multicast Architecture and Troubleshooting”
RST-3465
Agenda
• IPv4 Forwarding
• QoS
• NetFlow
RST-3465
Security and Feature ACLs
RST-3465
ACL Lookups
Engine
ACL TCAM
MAC? lookup lookup
NetFlow lookup
Input QoS
lookup
QoS TCAM


Bridged NetFlow
lookup lookup frame
RST-3465
Security ACLs
• Enforce security policies based on Layer 2,
Layer 3, and Layer 4 information
• Dedicated ACL TCAM ensures security ACLs
do not affect system performance
• Router ACL (RACL)—Enforced for all traffic
crossing a Layer 3 interface in a specified
direction
IPv4, IPX†, IPv6‡ RACLs supported
• VLAN ACLs (VACLs)—Enforced for all traffic in

the VLAN
IPv4, IPX†, MAC VACLs supported
• Port ACLs (PACLs)††—Enforced for all traffic

input on a Layer 2 interface
IPv4, MAC PACLs supported
† IPX ACLs in Supervisor 2 only

‡ IPv6 ACLs on Supervisor 720 and Supervisor 32 only
†† PACLs in Supervisor 720 and Supervisor 32 in CatOS only
RST-3465
Feature ACLs
• Classify traffic that requires additional or
special handling
Policy-Based Routing (PBR)
Reflexive ACLs
Network Address Translation (NAT/PAT)
WCCP redirection
• Programmed in ACL TCAM to preserve

performance
• Override FIB forwarding decision to allow
alternative processing
• Typically paired with NetFlow table and/or
Adjacency table
RST-3465
ACL Merge
• Sophisticated feature merge algorithm allows multiple security and
feature ACLs to be applied to a single interface/VLAN
• What is merging?
PFC/DFC hardware supports limited number of ACL lookups on a single packet
May need two or more ACL features on a single interface (e.g., RACL and PBR)
Merge produces ACEs that return correct result in a single lookup
• Downside: Can cause TCAM blowup

ACE intersection/interrelations can require lots of TCAM entries
• Two algorithms: ODM and BDD (Supervisor 2 only)

• If using Supervisor 2, USE ODM! (mls aclmerge algorithm odm)
• PFC3 dual-bank TCAM architecture can avoid merge entirely
White Paper on ACL Merge Algorithms and ACL Hardware Resources:

http://www.cisco.com/warp/public/cc/pd/si/casi/ca6000/tech/65acl_wp.pdf
RST-3465
ACL TCAM Entry Population
Protocol
xxxxxxxx 10.1.2.100 xx xxxx xxxx 1 Permit

Dest IP Source IP xxxxxxxx 10.1.68.101 xx xxxx xxxx 2 Deny
00000000 FFFFFFFF 00 0000 0000
xxxxxxxx 10.33.2.25 xx xxxx xxxx 3 Deny
Dest Port Source Port 4
5
6
1=“Compare” 7
0=“Mask” 8
ip access-list extended example xxxxxxxx xxxxxxxx 06 xxxx 0016 1 Permit

permit ip any host 10.1.2.100 xxxxxxxx xxxxxxxx 06 xxxx 0017 2 Deny
deny ip any host 10.1.68.101 xxxxxxxx xxxxxxxx 11 xxxx 0202 3 Deny
00000000 00000000 FF 0000 FFFF
deny ip any host 10.33.2.25 xxxxxxxx xxxxxxxx 06 xxxx 0080 4 Permit
permit tcp any any eq 22
xxxxxxxx xxxxxxxx 11 xxxx 00A1 5 Permit
deny tcp any any eq 23
deny udp any any eq 514
6
permit tcp any any eq 80 7
permit udp any any eq 161 8
RST-3465 Masks Values Cisco Public 81
ACL TCAM Lookup
Generate Compare 3
Lookup
Key 10.1.1.10
xxxxxxxx
xxxxxxxx| 10.1.2.11
xxxxxxxx
10.1.2.11| 06
xx
06|xxxx
xxxx
84C8xxxx
0050
| 0050 xxxxxxxx 10.1.2.100 xx xxxx xxxx 1
Lookup Key 2 xxxxxxxx 10.1.68.101 xx xxxx xxxx 2
00000000 FFFFFFFF 00 0000 0000

xxxxxxxx 10.33.2.25 xx xxxx xxxx 3
SIP=10.1.1.10
DIP=10.1.2.11 4
Protocol=TCP (6)
5
SPORT=33992
1 DPORT=80 6
Entries
matching only 7
Packet
destination IP 8
ip access-list extended example xxxxxxxx xxxxxxxx 06 xxxx 0016 1

permit ip any host 10.1.2.100 xxxxxxxx xxxxxxxx 06 xxxx 0017 2
deny ip any host 10.1.68.101 xxxxxxxx xxxxxxxx 11 xxxx 0202 3
00000000 00000000 FF 0000 FFFF
deny ip any host 10.33.2.25 HIT! xxxxxxxx xxxxxxxx 06 xxxx 0050 4 Permit
permit tcp any any eq 22 45
xxxxxxxx xxxxxxxx 11 xxxx 00A1
deny tcp any any eq 23 Entries matching Result
6
deny udp any any eq 514 only protocol and
permit tcp any any eq 80 destination port 7
permit udp any any eq 161 8
RST-3465 Masks Values Cisco Public 82
Monitoring ACL TCAM Utilization
• Cisco IOS: show
6509
- neb#show tcam counts tcam counts
Used Free Percent Used Reserved
- - -- - --- - - ---------- - - ------ • Catalyst OS: show
Labels: 23 4073 0 security acl
resource-usage
ACL_TCAM
--------
Masks: 2902 1194 70 72
Entries: 15261 17507 46 576
QOS_TCAM
--------
Masks: 7 4089 0 18
Entries: 32 32736 0 144
LOU: 47 81 36
ANDOR: 1 15 6
ORAND: 0 16 0
ADJ: 0 2048 0
6509
- neb#
RST-3465
Verifying Hardware ACL Enforcement
• show fm summary
6509-neb#show fm summary
Interface: Vlan199 is up
TCAM screening for features: ACTIVE inbound
TCAM screening for features: ACTIVE outbound
TCAM screening for features: ACTIVE outbound
TCAM screening for features: INACTIVE inbound
6509-neb#
fm = “Feature Manager”
ACTIVE = ACL policy is installed in hardware
INACTIVE = ACL policy is NOT installed in hardware
RST-3465
Displaying Hardware ACL “Hit Counters”
Cisco IOS: show tcam interface <interface> acl {in | out} ip
6509-neb#show tcam interface vlan199 acl in ip
<…>
permit udp any 10.89.210.0 0.0.0.255 (234265 matches)
deny ip any any (448691555 matches)
6509-neb#
Global or per-ACL entry
ACL Hit Counters Supported on PFC3B/BXL Only! (use [no] mls acl tcam
share-global to toggle)
RST-3465
Agenda
• IPv4 Forwarding
• QoS
• NetFlow
RST-3465
QoS
RST-3465
Catalyst 6500 QoS Model
Receive
Interface
Input Output
Ingress Egress Congestion
Queue Classify Mark Queue
Police Police Avoidance
Schedule Schedule
QoS Actions QoS Actions QoS Actions Transmit

at Ingress at PFC/DFC at Egress Interface
Port ASIC Port ASIC
RST-3465
QoS Lookups
Engine
ACL TCAM
MAC? lookup lookup
NetFlow lookup
Input QoS
lookup
QoS TCAM

lookup lookup* lookup lookup

Bridged NetFlow
lookup* lookup frame
RST-3465
NetFlow Table QoS TCAM ACL TCAM *PFC3 only
Classification
• Selects traffic for further

QoS processing
Marking
Policing
• Based on—
Port trust
QoS ACLs
RST-3465
QoS ACLs
• Support standard and extended IPv4,
IPv6,† and MAC ACLs for classification
• Use QoS TCAM to classify traffic for
marking and policing
• Leverage dedicated QoS TCAM
32K entries/4K masks
• Share other resources (LOUs and

labels) with security ACLs
† PFC3 only
RST-3465
QoS ACL Lookup Results
• QoS TCAM lookups behave exactly the same as ACL TCAM
lookups
• But, returned result differs:
Index into Aggregate table (identifies aggregate policer to use)
Index into Microflow table (identifies microflow policer to use)
Remarked DSCP/IP precedence value
RST-3465
Marking
• Untrusted port—Set a default QoS
value
• Trusted port—Use the marking
(COS, precedence, DSCP)
provided by upstream device
• QoS ACLs / service-policies—Set
QoS values based on standard or
extended ACL match
RST-3465
Policing
• Enforces a policy on a port or VLAN for traffic
matching classification policy
Markdown
Police (drop)
• Two types of policers:

Aggregate
Microflow
• Based on a classic token bucket scheme

Add tokens to bucket at constant rate (equivalent to policed rate)
Packets are “in profile” if enough tokens exist in the bucket to
transmit the packet
Packets without adequate tokens are dropped or marked down
• Note! PFC2 uses Layer 3 packet size, PFC3

uses Layer 2 frame size, when determining
rate
RST-3465
Aggregate Policing
• Bandwidth limit applied cumulatively to all flows that match the
associated class
Example—All FTP flows in a VLAN limited in aggregate to configured rate
• Ingress policing performed on per-switchport, per-Layer 3 interface,

or per-VLAN basis
PFC2 and PFC3 both support ingress policing
• Egress policing on a performed on per-Layer 3 interface or per-
VLAN basis
NOT possible on a per-switchport basis
PFC3 support only
• Dual-rate policers allow for combined markdown and drop policies
Normal rate and excess rate are configurable
PFC3 support only
RST-3465
Microflow Policing
• Bandwidth limit applied separately to each individual flow that
matches the associated class
Every individual FTP flow limited to configured rate
• User-based rate limiting using source-only and destination-only

flow masks
All FTP from a given source IP limited to configured rate
PFC3 only
• Leverages NetFlow table

• Microflow policing performed on ingress only
RST-3465
Remarking Traffic with Policers
• Policing action may remark certain traffic
For example, transmit with marked-down DSCP
• Dual-rate aggregate policer can mark-down traffic exceeding

the normal rate and drop traffic exceeding the excess rate
• Use markdown maps to configure marked-down DSCP values
mls qos map policed-dscp (Cisco IOS) or set qos policed-dscp-map (CatOS)
RST-3465
Monitoring Service Policies
(Marking and Policing)
6506#show policy- map interface vlan 100 • Cisco IOS: show

Service
- policy input: VLAN
- 100 policy-map
class
- map: NET
- 44
- TCP (match- all) interface*
Match: access
- group name POL
- 44
- TCP
• Catalyst OS: show
police :
qos statistics
100000000 bps 100000 limit 100000 extended limit
{aggregate-
Policed Earl in slot 6 :
policer |
Class 2940073472 bytes
l3stats}
5 minute offered rate 358172704 bps
aggregate
- forwarded 608631808 bytes action: transmit
exceeded 2331441664 bytes action: drop
aggregate
- forward 100352000 bps exceed 384495616 bps
class
- map: NET
- 55 (match
- all)
Match: access
- group name MARK
- 55
set precedence 5:
Marked Earl in slot 6 :
Class
2940069888 bytes
5 minute offered rate 358172616 bps
aggregate
- forwarded 2940069888 bytes
RST-3465 6506# * Shows aggregate policer stats only;
12523_04_2006_c1 © 2006 Cisco Systems, Inc. All rights reserved. Cisco Publicpolicing
use NetFlow table to monitor microflow 98
Agenda
• IPv4 Forwarding
• QoS
• NetFlow
RST-3465
NetFlow
RST-3465
NetFlow Lookups
Engine
ACL TCAM
MAC? lookup lookup
NetFlow lookup
Input QoS
lookup
QoS TCAM


Bridged NetFlow
lookup lookup frame
RST-3465
IPv4 NetFlow
• Tracks statistics for traffic flows through the system
• Entries created in NetFlow table when new flows
start
Flow mask determines format of entries
• Entries removed when flows expire

Timer and session based expiration
• Full collection by default when NetFlow enabled

Also support time- and packet-based NetFlow sampling
• Flow statistics can be exported using NetFlow Data

Export (NDE)
Supported export formats include NetFlow v5 and v7
NetFlow v9 export format supported in Supervisor 720 and
Supervisor 32
RST-3465
Displaying NetFlow Statistics Entries
• Cisco IOS: show mls netflow ip Which fields are
populated depends on
• Catalyst OS: show mls statistics entry the configured flow mask
6506#show mls netflow ip

Displaying Netflow entries in Supervisor Earl
DstIP SrcIP Prot:SrcPort:DstPort Src i/f :AdjPtr
---------------------------------------------------------------------------
Pkts Bytes Age LastSeen Attributes
---------------------------------------------------
10.102.130.213 10.214.39.79 tcp :46528 :www Vl39 :0x0
7 3766 17 15:47:37 L3- Dynamic
10.230.215.148 10.155.22.221 tcp :51813 :45912 Vl22 :0x0
25 21329 47 15:47:39 L3- Dynamic
10.97.36.200 10.17.64.177 tcp :65211 :www Vl144 :0x0
9 7664 17 15:47:38 L3- Dynamic
10.90.33.185 10.46.13.211 tcp :27077 :60425 Vl13 :0x0
2569654 1269409076 17 15:47:38 L3- Dynamic
<…>
RST-3465
NetFlow Table Utilization
• PFC2
NetFlow table contains 128K entries
Hash ~25% efficient (32K entries)
Probability of collision increases after 32K
entries
• PFC3
NetFlow table size varies
• PFC3A/B—128K entries
• PFC3BXL—256K entries
Hash ~50–90% efficient (64/96/230K entries for
PFC3A/B/BXL)
Probability of collision increases after
64K/96K/230K entries
Alias CAM handles hash collision cases
RST-3465
PFC2 NetFlow Table Architecture
Packet
Flow Key
2 16K rows
SIP 10.10.20.1
DIP | 10.20.2.2
Proto | 6SPort
| 1044 | 80
DPort
172.16.8.2 | 192.168.1.2 | 6 | 1025 | 80
5 172.16.1.1
10.1.1.1 | 10.1.1.2 | 172.16.2.2
| 6 | 1030 || 80 17 | 2334 | 23
10.10.10.1
10.1.1.1 | 10.1.1.2| |10.20.1.1
6 | 1030 || 80 6 | 2334 | 80
Compare 10.1.1.1 | 10.1.1.2 | 6 | 1030 | 80
10.1.1.2 |192.168.1.1
10.1.1.1 | 10.1.1.2
10.1.1.1
10.4.4.4 | 6 || 10.1.1.2
6 | 1030| 1
|80
| 172.16.8.8
|0|0
| 80
10.1.1.1
10.1.1.2 | 10.1.1.2
| 10.1.1.1 | 6 | |806 | |1030
17 | 1025
| 80 | 514
10.1.1.1 | 10.1.1.2
10.1.1.2 | 10.1.1.1 | 6 | 80 | 6 | 1030 | 80
3 10.1.1.1
10.1.1.2 | 239.1.1.1
| 10.1.1.1 | 6 | 17
80 | 5000
1030 | 5000 Update
10.10.20.1
10.1.1.2 | 10.20.2.2
| 10.1.1.1 | 6 | 80| 6 | 1044 | 80
| 1030
Hash Function
10.1.1.2 | 10.1.1.1 | 6 | 80 | 1030
HIT! 10.1.1.2 | 10.1.1.1 | 6 | 22 | 3245 6 Statistics
10.99.1.1 | 10.99.100.1 | 6 | 4444 | 25
10.99.100.1 | 10.4.5.6 | 6 | 25 | 1080
Starting Page 4
and Row
8 pages
RST-3465
12523_04_2006_c1 © 2006 Cisco Systems, Inc. All rights reserved. NetFlow Table Cisco Public 105
PFC3 NetFlow Lookups
Key Flow Data
Packet Key Flow Data
1 Key Flow Data
Key 6 Flow Data HIT!
2 Mask Compare
Key Flow Data Statistics
7
Flow
Flow Key
Key Key Flow Data
Key Flow Data
Key Flow Data
NetFlow
HIT! Key Flow Data
Table Index
3 Key 5 Flow Data
Result
Hash Function Key Flow Data
Flow Data
Mask
4
Hash Key
Hash Key 128K/256K 128K/256K
Compare
entries rows
Netflow TCAM Netflow Table
Key
128 entries
RST-3465
12523_04_2006_c1 Alias CAM
© 2006 Cisco Systems, Inc. All rights reserved. Cisco Public 106
Monitoring NetFlow Table Usage
• Cisco IOS: show mls netflow table-contention
• Catalyst OS: show mls debug
6506#show mls netflow table
- contention detailed
Earl in Module 6
Detailed Netflow CAM (TCAM and ICAM) Utilization
================================================
TCAM Utilization : 100%
ICAM Utilization : 82%
Current utilization
Netflow TCAM count : 131072
Netflow ICAM count : 105
Netflow Creation Failures : 3432605
Clear on read
Netflow CAM aliases : 8
6506#show mls netflow table
- contention aggregate
Earl in Module 6
Aggregate Netflow CAM Contention Information
=============================================
Netflow Creation Failures : 222917949
Cumulative
Netflow Hash Aliases : 834
RST-3465 6506# Cisco Public 107
NetFlow Aging
• Process of removing stale NetFlow
entries
• Types of aging
Normal—Fixed idle time for flows
Fast—Threshold-based aging of flows
Long—Maximum lifetime for flows
Session-based—Based on TCP FIN/RST flags
• Default timers are conservative

Tuning is recommended!
Start with more aggressive normal aging timer—
Reduce until no creation failures seen or CPU is at
threshold
Enable fast aging to remove short-lived flows—Adjust
until creation failures cease or CPU is at threshold
RST-3465
Changing and Viewing the
NetFlow Aging Configuration
• Cisco IOS:
mls aging {normal | fast | long}
show mls netflow aging
• Catalyst OS:
set mls agingtime [fast | long-duration]
show mls
6506#show mls netflow aging

enable timeout packet threshold
------ ------- ----------------
normal aging true 300 N/A
fast aging false 32 100
long aging true 1920 N/A
6506#
RST-3465
Conclusion
• You should now have a thorough
understanding of the Catalyst 6500
switching architecture, packet flow, and
key forwarding engine functions…
ANY QUESTIONS?
RST-3465
Related Networkers Sessions
• RST-3262: IP Multicast Architecture and Troubleshooting for
the Cisco Catalyst 6500 Series
• RST-3143: Troubleshooting Catalyst 6500 Series Switches
• RST-2031: Multilayer Campus Architectures and
Design Principles
• RST-3466: Cisco IOS Software Modularity—Architecture and
Deployment
• TECRST-3101: Troubleshooting Cisco Catalyst Switches
• TECRST-2001: Enterprise High Availability
• BoF-06: Enterprise Switching
RST-3465
Q and A
RST-3465
Recommended Reading
• Continue your Cisco Networkers
learning experience with further
reading from Cisco Press
• Check the Recommended Reading
flyer for suggested books
Available Onsite at the

Cisco Company Store
RST-3465
Complete Your Online Session Evaluation
• Win fabulous prizes; Give us your feedback
• Receive ten Passport Points for each
session evaluation you complete
• Go to the Internet stations located
throughout the Convention Center to complete
your session evaluation
• Drawings will be held in the
World of Solutions
Tuesday, June 20 at 12:15 p.m.
Wednesday, June 21 at 12:15 p.m.
Thursday, June 22 at 12:15 p.m. and 2:00 p.m.
RST-3465
RST-4501
11366_06_2005_x © 2005 Cisco Systems, Inc. All rights reserved. 115

Cisco Catalyst 6500 Switch Architecture

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Cisco Catalyst 6500 Switch Architecture

Uploaded by

Copyright:

Available Formats

Cisco Catalyst 6500 Switch

• Modular chassis in variety of form factors

• Wide variety of Single Channel Slot 5

Supported from Cisco IOS 12.1(5c)EX and Catalyst OS 6.1(1)/12.1(3a)E1

Supported from Cisco IOS 12.2(14)SX and Catalyst OS 8.1(1)/12.2(14)SX2

Supported from Cisco IOS 12.2(18)SXF and Catalyst OS 8.4(1)/12.2(17)SXB7

Supervisor Engine 32 Baseboard

Supervisor Engine 32 Baseboard

• Specific chassis slots:

• 18 fabric channels in total

• Fabric module occupies a full slot

• SFM—Supports 6506 and 6509

• Cisco IOS: show fabric [active | channel-counters |

6506#show fabric utilization

• Monitor the traditional Catalyst 6500 bus when

• Cisco IOS: show catalyst6000 traffic-meter

6506#show catalyst6000 traffic-meter

traffic meter = 7% Never cleared

No QoS TCAM NetFlow Table

Input ACL Output QoS Output ACL Output Layer 2

ACL TCAM QoS TCAM ACL TCAM Layer 2 Table

Output QoS Output ACL Transmit

3. As lookup key compared to value 0111xxxx 1

4. Longest match returns result 1101xxxx 3

Port Port Port Port

DBUS 8Gbps Fabric

Fabric interface LCDBUS

Replication engine for

Complex A Complex B CEF720

Engine CFC Engine

Port Port Port Port

Bus interface for control data only!!

20Gbps Fabric 20Gbps Fabric

Complex A Complex B CEF720

Engine L3/4 Engine

• Deterministic, highly scalable—Not flow-based

L3/4 Supervisor Red

Supervisor Port Port

No QoS TCAM NetFlow Table

Input ACL Output QoS Output ACL Output Layer 2

ACL TCAM QoS TCAM ACL TCAM Layer 2 Table

Output QoS Output ACL Transmit

• MAC learning fully hardware based

• PFC and DFCs have copies of MAC table

• MAC table size:

Layer 2 Table Layer 2 Table

Learn New MAC? Router MAC? L3 forwarding

Update entry Known MAC? L2 forwarding

16 pages 4096 rows

8 pages MAC Table

20 | 0000.cccc.cccc 4096 rows

6509#show mac-address-table dynamic vlan 30

vlan mac address type learn qos ports

No QoS TCAM NetFlow Table

Input ACL Output QoS Output ACL Output Layer 2

ACL TCAM QoS TCAM ACL TCAM Layer 2 Table

Output QoS Output ACL Transmit

• FIB table contains IP prefixes

• Hardware CEF process:

• Cisco IOS: 6509-neb#show mls cef summary

6509-neb#show mls cef 10.100.20.0 detail

6509-neb#show mls cef adjacency entry 98304

• Cisco IOS: show mls cef lookup <ip_address> [detail]