Brkarc 2003

Cisco ASR 9000 System
Architecture
Yongzhong Peng
Manager, Technical Marketing
SP Routing Infrastructure
BRKARC-2003
Cisco Webex Teams
Questions?
Use Cisco Webex Teams to chat
with the speaker after the session
How
1 Find this session in the Cisco Events Mobile App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space
BRKARC-2003 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Agenda
• ASR9000 Products Introduction

• ASR9000 System Hardware Architecture
• ASR9000 Distributed Control Plane
• ASR9000 Data Packet Processing
• ASR9000 QoS Architecture & TCAM Usage
• Conclusion
ASR 9000 Products Introduction
Cisco ASR 9000 System Portfolio
Compact & Powerful High Density Service Edge
Flexible Service Edge
Access/Aggregation and Core
• Small footprint with full IOS-XR • Optimized for MSE with high M-D scale for • Scalable, ultra high density
for distributed environments medium to large sites service routers for large,
high-growth sites
One Platform, One OS, One Family
ASR 9922
nV Satellites
ASR 9000v2, NCS5000 ASR 9912
ASR 9010 ASR 9910

ASR 9901
ASR 9906
ASR 9006
456Gbps
ASR 9904
ASR 9001
Fixed 2RU 2 LC/6RU 4 LC/10RU 4 LC/14RU 8 LC/21RU 8 LC/21RU 10 LC/30RU 20 LC/44RU

240 Gbps 16 Tbps 16 Tbps 32 Tbps 32 Tbps 64 Tbps 80 Tbps 160 Tbps
MSE DCI Peering P/PE CE Mobility Broadband
ASR 9000 Modular Systems
Generation 2
Generation 1
ASR 9006 ASR 9010 ASR 9904 ASR 9906 ASR 9910 ASR 9912 ASR 9922
• Up to 2T/slot • Up to 4T/slot
• A9K 5-Fabric Card Support • A9K 5-Fabric & A99 7-Fabric Card Support
ASR 9000 Hardware Components – Control Module
ASR 9922
RSP: Route Switch Processor
ASR 9910
ASR 9010
ASR 9912
ASR 9006
ASR 9904
ASR 9906
RP: Route Processor
RSP5/RP3 Front Panel Configuration
RSP5
RP3
• 2x BITS ports on RJ-45 • Alarm output serial port

• 100Mbps, 1588 port – RJ-45 • USB
• TOD – RJ-45 • 2x Management ports on RJ-45
• 10Mhz on SMA • AUX & Console on RJ-45 connectors
• 1x CMP • LED’s for major/critical and normal oper alarms or
states
• 1PPS on SMA
ASR 9000 Hardware Components – Linecards
ASR 9922
ASR 9910
ASR 9010
ASR 9912
ASR 9006
ASR 9904
ASR 9906
ASR 9900 - Switch Fabric Cards
ASR 9906 ASR 9922
✓ Common for ASR 9912 and ASR 9922
✓ SFC-S for ASR 9910
✓ 7 Fabric Card Slots
✓ 5 Fabric Card Slots for ASR 9910 (2 on RSPs => total 7)
✓ Decoupled, multi-stage switch fabric hardware
ASR 9912
✓ True HW separation between control and data plane
✓ Add bandwidth per slot easily & independently
ASR 9910
SFC2/SFC-S SFC3
Fabric Capacity per 230G 600G
SFC
Fabric Capacity 1.38T N+1 3.6T N+1
Per Line Card Slot 1.61T N+0 4.2T N+0
Fabric Redundancy N+1 N+1
LC Support Typhoon Tomahawk

Tomahawk LightSpeed
LightSpeed Plus
ASR 9000 Systems Switch Fabric Overview
Dedicated Fabric Cards

Integrated Fabric on RSP
ASR 9904 ASR 9006 ASR 9010

Hybrid Systems
ASR 9912 ASR 9922
Integrated Fabric/RP/LC
ASR 9001 ASR 9901
ASR 9906 ASR 9910
4th Generation Hardware Commons PIDs
• ASR-9922 • ASR-9906
• A99-RP3-SE/TR • A9K-RSP5-SE/TR
• A99-SFC3 • A99-SFC3-T
• ASR-9922-FAN-V3
• ASR-9904 / ASR-9010 / ASR-9006
• ASR-9912 • A9K-RSP5-SE/TR
• A99-RP3-SE/TR
A99-SFC3
• A99-SFC3
• ASR-9910
A99-SFC3-S
• A9K-RSP5-SE/TR
• A99-SFC3-S
A99-SFC3-T
Cisco ASR 9906 Overview
Feature Description
Total Capacity >32T
Capacity per
>4T
Slot
Slots 6 slots - 4 Line Cards and 2 RSPs
Rack size 14RU, front-to-back airflow. 2 Fan Trays
1 Power Tray
Power 4.4 KW DC supplies (max 4) 4 Line Card
6.0 KW AC supplies (max 3) Slots
Front-to-Back Airflow
Fan 2 RSPs
2 Fan Trays, FRU
RSPs Integrated Fabric, 1+1 Redundancy 5 Fabric
Cards
5 Fabric Cards + 2 RSP Integrated
Fabric
Fabric Cards 1 Power
Total 7 Fabric for 6+1 Redundancy
230G per FC2 or 600G per FC3 Tray
Tomahawk
LightSpeed
Line cards
LightSpeed Plus
Service Card: VSM with 32-bit XR
ASR 9000 Silicon Evolution
1st Gen
Trident
120G Trident Octopus Santa Cruz PowerPC
90nm,15 Gbps 130nm,60 Gbps 130nm,90 Gbps Dual Core,1.2 Ghz
2nd Gen
Typhoon
360G Typhoon Skytrain Sacramento PowerPC
55nm,60 Gbps 65nm,60 Gbps 65nm,220 Gbps Quad Core,1.5 Ghz
3rd Gen
Tomahawk
1.2T Tomahawk Tigershark SM15 X86
28nm,240 Gbps 28nm,200 Gbps 28nm,1.2 Tbps 6 Core, 2 Ghz
4th Gen
LightSpeed
3.2T LightSpeed SKB X86
16nm,400 Gbps 16nm,3.6 Tbps 8 Core, 2.2 Ghz
5th Gen
LightSpeed+
4T LightSpeed+ SKB
SKB X86
7nm,400 Gbps 16nm,3.6
16nm,3.6Tbps
Tbps 8 Core, 2.2 Ghz
ASR 9000 3rd Generation Line Card Overview
MPAs MPA
20x1GE 32x1GE
MPAs
2x10GE 1x100GE
4x10GE 2x100GE
1x40GE 20x10GE
2x40GE
3rd gen LC Tomahawk -TR, -SE

NPU: 240Gbps,
A9K-24X10GE-1G
~150Mpps
4x100GE & 8x100G CPAK OTN Line Card A9K-48X10GE-1G
MOD400/MOD200
4x100GE & 12x100G QSFP28 LAN Line Card

A9K-400G-DWDM
ASR 9000 4th Generation Line cards
1.6T-3.2T per slot
32 x 100GE TR line card
9922
9912
9910
9906
9904
16 x 100GE SE line card
9010
9006
ASR 9000 5th Generation Line cards
A9K-8HG-FLEX-SE/TR line card
9922
9912
9910
9906
9904
A99-32X100GE-X-SE/TR line card
9010
9006
ASR 9000 5th Generation 3.2T Card
A99-32X100GE-X-SE/TR line card
• 32 Ports 100G support

• Full Feature Parity with Tomahawk Except for
MACsec
• IPv6 Forwarding Optimized
ASR 9000 5th Generation 2T Combo Card
5 Slices:
400G/200G/100G 1x 400G/200G/100G
Multi-rate Ports: multi-rate port
0/7/8/12/19 + 3x100G ports per Slice
400G Ready
• 10G/25G/40G/100G/200G/400G Support
Each Slice Independently Configured as:
• 1x400G
• 1x200G + 2x100G
• 4x100G
Each 100G Breakout into 4x25G or 4x10G
ASR 9000 5th Generation 800G Combo Card
400G/200G/100G
Multi-rate Ports: 2 Slices:
0/7 1x 400G/200G/100G
multi-rate port
+ 3x100G ports per Slice

400G Ready
• 10G/25G/40G/100G/200G/400G Support
Each Slice Independently Configured as:
• 1x400G
• 1x200G + 2x100G
• 4x100G
Each 100G Breakout into 4x25G or 4x10G
Pop Quiz ????
What different interface rates can ASR9000 LightSpeed Plus 2T Combo

Linecard Support?
• 400G &100G
• 400G, 200G & 100G
• 400G, 100G & 40G
• 400G, 100G, 25G & 10G
• 400G, 200G, 100G, 40G, 25G & 10G
ASR 9000
Hardware
Architecture
ASR 9000 System Architecture “At-a-Glance”
Control plane split among Network Processor
RSP/RP
RSP/RP and LC CPU
CPU
RSP FIA
CPU CPU
NP Bay
Network Processor CP
BITS/DTI
U
Switch SerDes
FIA Fabric XBAR
FIC
CPU
CPAK
CPAK
PHY
NP FIA FIA NP Bay
CPAK
CPAK
PHY
NP FIA
Switch
CPAK
Fabric Line Card
CPAK
PHY
NP FIA
CPAK
CPAK
PHY
NP FIA Fully Distributed Architecture for
High Performance and High Multi-
Data forwarding is fully Switch Fabric Dimensional Control Plane Scale
distributed across NPs
A99-32X100GE-X-SE/TR (7-fabric) LC Architecture
(in 9922, 9912, 9910 & 9906) 600G/fab x 7 fab
= 4.2T
8x Slices
QSFP28 0
CPU
Retimer
Retimer
PHY LightSpeedPlus NP0 FIA
Switch
Fabric 0 x8
QSFP28 4 Retimer (SKB)
Retimer
QSFP28 8 Retimer
Retimer
QSFP28 12 Retimer
Retimer PHY LightSpeedPlus NP3 FIA
QSFP28 16 Retimer
Retimer
Switch
QSFP28 20 Retimer
Fabric 1
Retimer
PHY LightSpeedPlus NP5 FIA (SKB)
QSFP28 24 Retimer
Retimer
QSFP28 28 Retimer
Retimer
A99-32X100GE-X-SE/TR LC Architecture
(in 9904) 1.8T/RSP x 2RSP
8x Slices = 3.6T
QSFP28 0
CPU
Retimer
Retimer
Switch
Fabric 0 x8
QSFP28 4 Retimer (SKB)
PHY LightSpeedPlus NP1 FIA x8
Retimer
x8
RSP0
QSFP28 8 Retimer x8
Retimer
x8
x8
QSFP28 12 Retimer
Retimer PHY LightSpeedPlus NP3 FIA
QSFP28 16 Retimer
Retimer
Switch x8
QSFP28 20 Retimer
Fabric 1 x8
Retimer
PHY LightSpeedPlus NP5 FIA (SKB)
x8
RSP1
x8
QSFP28 24 Retimer
Retimer
x8
QSFP28 28 Retimer
Retimer
A9K-20HG-FLEX-SE/TR (5-fabric) LC Architecture
(in 9922, 9912, 9910 & 9906)
600G/fab x 5 fab
5x Slices
CPU = 3.0T (actual)
QSFP-DD
QSFP28 PHY LightSpeedPlus NP0 FIA
QSFP28
QSFP28
QSFP-DD 2x8
QSFP28 Switch
QSFP28 2x8
Fabric 0
(SKB)
QSFP-DD 2x8
QSFP28
QSFP28 2x8
QSFP-DD 2x8
QSFP28
QSFP28
QSFP-DD
QSFP28
QSFP28
(in 9010 & 9006)
5x Slices
CPU
QSFP-DD
QSFP28 PHY LightSpeedPlus NP0 FIA 1.1T/RSP x 2RSP
QSFP28 = 2.2T (actual)
QSFP28
QSFP-DD
QSFP28 PHY LightSpeedPlus NP1 FIA x8
QSFP28 Switch x8
QSFP28 RSP0
Fabric 0 x8
(SKB)
QSFP-DD x8
QSFP28 x8
QSFP28
x8
RSP1
QSFP-DD x8
QSFP28
QSFP28
QSFP28
QSFP-DD
QSFP28
QSFP28
(in 9904)
5x Slices
CPU
QSFP-DD
QSFP28 PHY LightSpeedPlus NP0 FIA 1.4T/RSP x 2RSP
QSFP28 = 2.8T (actual)
QSFP28
QSFP-DD
QSFP28
QSFP28 Switch x8
QSFP28 x8 RSP0
Fabric 0
(SKB) x8
QSFP-DD x8
QSFP28 x8
QSFP28
x8
x8 RSP1
QSFP-DD x8
QSFP28
QSFP28
QSFP-DD
QSFP28
QSFP28
(in 9922, 9912, 9910 & 9906)
600G/fab x 5 fab
2x Slices
CPU = 3.0T (actual)
QSFP-DD 2x8
QSFP28 Switch
QSFP28 2x8
Fabric 0
(SKB)
2x8
2x8
QSFP-DD 2x8
QSFP28
QSFP28
(in 9010 & 9006)
2x Slices
CPU
1.1T/RSP x 2RSP
= 2.2T (actual)
QSFP-DD
QSFP28 Switch x8
QSFP28 RSP0
Fabric 0 x8
(SKB)
x8
x8
x8
RSP1
QSFP-DD x8
QSFP28
QSFP28
QSFP28
(in 9904)
2x Slices
CPU
1.4T/RSP x 2RSP
= 2.8T (actual)
QSFP-DD
QSFP28
QSFP28 Switch x8
QSFP28 x8 RSP0
Fabric 0
(SKB) x8
x8
x8
x8
x8 RSP1
QSFP-DD x8
QSFP28
QSFP28
ASR90xx – RSP5 and Mixed LC
Stage 1 Stage 2 Stage 3
Active-Active
Fabric
Fabric
Fabric
Arbiter
FIA
FIA FIA
FIA FIA
Fabric
RSP5
FIA
SM15
Tomahawk Line
LightSpeed Plus Card
Linecard
Fabric
Ingress Linecard Egress Linecard
LightSpeed Plus LC Fabric bandwidth: Arbiter
Tomahawk LC Fabric bandwidth:
dual RSP: 2.2T/slot
RSP5 Dual RSP: 860Gbps/slot
single RSP: 1.1T/slot
Single RSP: 430Gbps/slot
ASR9922/12 System – LightSpeed Plus &
Tomahawk
ARBITER
RP3
Tomahawk LCs interoperate at 3.2T LC at line-rate with
full throughput w/ fabric fabric redundancy
redundancy
Tomahawk LCs LightSpeed Plus

600G LC
(7-fab)
SM15 (7-fab)
A99-12X100GE
A99-8X100GE-SE/TR A99-32X100GE-X-SE/TR
RP2 and RP3 can support SFC3

LightSpeed Plus 1.5T with 7 Fabrics 4.2T w/ 7 Fabrics
20x100G & 8x100G 1.29T w/ 6 Fabrics 3.6T w/ 6 Fabrics
linecards.
32x100G requires RP3
ASR9910/06 System - LightSpeed Plus &
Tomahawk
RSP5
Tomahawk LCs interoperate at 3.2T LC at line-rate with
full throughput w/ fabric fabric redundancy
redundancy
RSP5
Tomahawk LCs LightSpeed Plus

600G LC
(7-fab)
SM15 (7-fab)
A99-12X100GE
A99-8X100GE-SE/TR A99-32X100GE-X-SE/TR
SFC3
1.5T with 7 Fabrics 4.2T w/ 7 Fabrics
1.29T w/ 6 Fabrics 3.6T w/ 6 Fabrics
ASR99xx Mix Mode
1.075T with 5 fabrics

All Tomahawk LCs interoperate at full
throughput w/ fabric redundancy; except for 860G w/ 4 fabrics
12x100GE LC, which will be at 1.075T 3T with 5-fabrics
215G Prevents “high priority” packet drop
due to fabric congestion
215G
Tomahawk LC
215G
(5-fabric)
SM15 600G 3.2T LS+ LC
(7-fab)
System automatically rate-limits each

2.0T LC at line-rate LS+ LC
slice to 375G (93.75% linerate per
w/ fabric redundancy
(5-fab) SFC3 slice; single 100G port can reach
800G LC at line-rate SKB 100% linerate)
w/ fabric redundancy
For 100G linerate on all ports:
3T w/ 5 fabrics • Shut down one slice
2.4T w/ 4 fabrics BRKARC-2003 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
Egress FIA Rate-Limiter FIA
NP
375G NP->FIA Per NPU
rate-limiter.
600Gx5fab/8NPU=375G
NP per NPU Bandwidth
• Triggered by Switch Fabric shared by 4x100G
Degradation: Not enough NP
interfaces per NPU
switch fabric bandwidth to NP
support linerate NP
• Rate-Limiter is applied in NP
the direction NP -> FIA 89G NP->FIA Per
NP
Interface rate-limiter.
NP 215Gx5fab/6NPU/
2interface=89G per
LS+ 3.2T LC Interface Bandwidth.
(7-fab) 2x100G interfaces
per NPU
FIA NP
LS+ LC
FIA NP
(5-fab) SFC3
SKB FIA NP
FIA NP
SM15
FIA NP
• Tomahawk and before: applied per interface
• LightSpeed Plus: applied per NPU. Interfaces FIA NP
on the same NP share the Bandwidth
Tomahawk1.2T LC
(7-fab)
How do you know Rate-Limiter Triggered?
Console Message:
LC/0/3/CPU0: pfm_node_lc[261]: %FABRIC-FIA-1-RATE_LIMITER_ON : Set|fialc[4795]|0x108a000|Insufficient fabric capacity for card types in use - FIA egress rate limiter applied
LC/0/5/CPU0:pfm_node_lc[207]: %FABRIC-FIA-1-RATE_LIMITER_ON : Set|fialc[4798]|0x108a000|Insufficient fabric capacity for card types in use - FIA egress rate limiter applied
Command to check if Rate-limiter has been applied

“show pfm location all”
Slice Level Management
• CLI to shut down slice
• hw-module location ? slice ? power-savings
• Benefits:
• Ports on remaining slices are not affected
• Power Saving
• Switch Fabric Bandwidth Re-allocation
• Example: 32x100G LC mix with 5-fab cards
• Total switch fabric bandwidth = 5x600G =3T
• Rate-limiter triggered: 3000G/8 = 375G per NPU
• One Slice Shut: 3000G/7 > 400G per NPU, Linerate
Guaranteed
• Two Slice Shut: (3000G-600G)/6 = 400G per NPU.
N+1 Switch Fabric Redundancy
Runs distributed control plane
Line Card Components protocols for increased scale
BFD, CFM, ARP
Main forwarding engine L2 and L3 lookups Receive FIB table from RP and
program hardware forwarding
Multicast replication toward Optics table
User level QoS and Security features CPU
NPU P1
PHY P1
FIA
P2
P3 Switch
TM
P2
P3
BE
Fabric
BE ASIC
Dedicated queue ASIC – TM (traffic Provides data connection to switch fabric

manager) per NPU for QoS functions
Manage VoQ, Superframe and loadbalancing
User Configurable Queue on TM. data traffic across switch fabric
Default Port Queue Always Created. Mcast replication table for replication toward
NPs
Network Processor Architecture Details
TR and SE has different
TR and SE has same
memory size
memory size NPU Complex
STATS MEMORY
FIB MAC
LOOKUP Forwarding chip (multi core) FRAME MEMORY
MEMORY
- TCAM
• TCAM: VLAN tag, QoS and ACL classification
• Stats memory: interface statistics, forwarding statistics etc
• Frame memory: buffer, Queues
• Lookup Memory: forwarding tables, FIB, MAC, ADJ
• TR/SE
• Different TCAM/frame/stats memory size for different per-LC QoS, ACL, logical interface scale
• Same lookup memory for same system wide scale mixing different variation of LCs doesn’t impact
system wide scale
-TR: transport optimized, -SE: Service edge optimized
Pop Quiz ????
Can LightSpeed Plus A99-32X100GE-X-SE/TR be used in

ASR-9010 or ASR9006 Chassis?
• Yes
• No
ASR 9000 Distributed
Control Plane
IOS-XR 6.X: A New Software Infrastructure
• 64-bit OpenEmbedded Linux support.

• Processes containerization.
Classic XR XR 64 Bit
• Standard Linux toolchain.
• Third-Party applications.
System
• NCS5500, NCS5000 and NCS1002 System Control Admin
support 64-bit Linux. Control Admin
• ASR 9000 supports 64-bit Linux starting

with 6.1.1. 32 bit QNX 64 bit Linux
• Will still have 32-bit QNX support. NPU X86 Hardware
IOS XR 32-Bit and 64-Bit Differences
• IOS XR Exists in two flavors Separate Admin

Plane
IOS-XR
• 32-bit in XR12k, CRS, ASR9000 System
IOS-XR
Admin
Control
• QNX-based Control
Plane Plane
RP
System
• No Virtualization Admin Linux Linux 64-bit IOS XR
• No ISSU QNX Linux
• 64-bit in ASR9000, NCS 5500,

NCS 5000, NCS 1000 and IOS XR IOS-XR
Line Card
NCS 6000 Admin
Plane
Linux VM
• Linux based LC-CPU

LC-CPUs
• Larger addressable memory Linux Linux

64 bit Linux
• Separation Networking OS and Admin Plane QNX Linux Kernel
• Virtualization: VM or Container
“Classic” IOS XR Linux-based Virtualized
• ASR9000 Running with VMs
IOS XR 64 Bit
ASR9000 Fully Distributed Control Plane
CPU
LPTS (local packet transport service):
control plane policing
RP
CPU Punt
CPU FPGA FIA
Switch Fabric Switch Fabric

LC
Control
Punt Switch CPU
CPAK
packet
PHY
NP
LPTS FIA
CPAK RP CPU: Routing, MPLS, IGMP, PIM,
CPAK HSRP/VRRP, etc
CPAK
PHY
NP FIA
Switch
CPAK
Fabric
LC CPU: ARP, ICMP, BFD, NetFlow,
PHY
NP FIA
CPAK OAM/CFM, L2 Protocols etc
CPAK
CPAK
PHY
NP FIA
NP Offloading: BFD
L3 Control Plane Architecture
ISIS/OSPF BGP
Static BGP OSPF
LDP RSVP-TE
Static ISIS EIGRP
LSD RIB RSP/RP CPU

RSP/RP
LC
ARP/NDP
SW FIB HW FIB Adjacency
LC NP
AIB
AIB: Adjacency Information Base
RIB: Routing Information Base
LC CPU
FIB: Forwarding Information Base
LSD: Label Switch Database
Distributed ARP Processing
Incomplete ADJ Packets RP CPU RP

Incoming ARP Packets
Outgoing ARP Packets
ARP FIA
Switch Fabric
Line card ARP LC
CPU
spio netio
Switch Fabric
SPP
Tsec Driver
Punt Switch
PHY
LPTS
NP FIA
MAC Learning and Sync
Hardware based MAC learning: ~4Mpps/NP
1 NP learn MAC address in hardware (around
4M pps)
RP
2 NP flood MAC notification (data plane) Punt
CPU
message to all other NPs in the system to FPGA FIA
sync up the MAC address system-wide. MAC
notification and MAC sync are all done in
hardware
Switch Fabric Switch Fabric
CPU LC1
LC2
CPU
Data 1NP 2
packet NP
FIA NP
FIA
NP
NP
2
Fabric ASIC
Switch
FIA NP
Fabric ASIC
Switch
NP
FIA
NP
NP
FIA
NP
NP FIA
NP
NP
FIA NP
NP FIA
NP
Distributed Netflow Architecture
RP
Netflow Netflow show/clear
• Aggregate 200Kpps • Manage flow
netflow policer rate
Configure Command
table
• Evenly Divided among • Export flow
netflow-enabled NPs information
LC-CPU EXPORT
netIO
NPU:
• Traffic Filtering 1M Record Cache
• Traffic Sampling
• Extract flow
header
information
• LPTS Policer
LC
NPU NPU NPU NPU
Pop Quiz ????
When A9K-8HG-FLEX-SE/TR is used in the following chassis with

RSP5, which one(s) provide full fabric redundancy?
• ASR 9904 (1.4T/RSPx2=2.8T)
• ASR 9006 (1.1T/RSPx2=2.2T)
• ASR 9010 (1.1T/RSPx2=2.2T)
• All of above
ASR 9000 Data Packet
Processing
Distributed Two-Stage Packet Processing
• Ingress lookup yields packet egress port and applies ingress features
• Egress lookup performs packet-rewrite and applies egress features
CPU CPU
CPAK 0
1 CPAK 0
PHY NP FIA FIA NP PHY
CPAK 1
CPAK 1
CPAK 2
Switch CPAK 2
Fabric
PHY NP FIA FIA 2
NP PHY
CPAK 3 CPAK 3
Switch Switch
…
…
Fabric Fabric
CPAK 4 CPAK 4
FIA
(SM15) (SM15)
PHY NP FIA NP PHY
CPAK 5
CPAK 5 Up to
Up to
14x120G 14x120
CPAK 6 1 Switch G CPAK 6
PHY NP 2 FIA Fabric FIA NP PHY
CPAK 7 CPAK 7
Uniform packet flow for simplicity and predictable performance
ASR9000 Life of a Packet
2 4 • Buffering packet from 6 • Egress L2/L3 FIB lookup, 8
• Ingress L2/L3 FIB lookup, NP • MACSEC Emcryption
ACL/QoS lookup ACL/QoS lookup
• Requesting fabric credit • Egress PBR/ABR, ACL, uRPF • G.709/OTN/WAN-
• Ingress PBR/ABR, ACL, uRPF • Manage superframe PHY/LAN-PHY
• Ingress QoS: classification, • Egress QoS: classification,
and load-balancing marking, policing, shaping • Line Clocking
marking, policing packet across fabric
• Packet Punting • Incomplete Adj Packet
• Manage system VoQ Punting
• Ingress ECMP/LAG hashing
• Egress ECMP/LAG hashing
Ingress side of LC CPU CPU 6 8

NP PHY
5 7 TM
CPAK 0
1
2
4
FIA
CPAK 1
PHY NP
3 TM
FIA Switch
Fabric
Egress side of LC
1 3 5
7
• MACSEC Decryption • Ingress Queuing • Re-assembling packets • Egress Queuing
• G.709/OTN/WAN- Processing from superframe Processing
PHY/LAN-PHY • Bypassed in case no • Send packet to
• Line Clocking ingress queuing support corresponding NP
• Release buffer and
fabric credit
Switch Fabric Arbitration
5: credit return
Fabric
1: Fabric Request ASIC
Fabric
ASIC
Arbitration
Fabric ASIC Fabric ASIC
and VOQ SFC or RSP0 2: Arbitration and VOQ
Fabric
ASIC
3: Fabric Grant
Fabric
ASIC
Arbitration
4: load-balanced
SFC or RSP1
transmission
across fabric links
Fabric Load Balancing – Unicast
Fabric
ASIC
Fabric
ASIC
Arbitration Fabric ASIC
Fabric ASIC
and VOQ SFC or RSP0 and VOQ 4 3 2 1
Fabric
ASIC
Fabric
ASIC
Arbitration
SFC or RSP1
• Unicast traffic sent across first available fabric link to destination (maximizes efficiency)
• Each frame (or super frame) contains sequencing information
• All destination fabric ASIC have re-sequencing logic
• Additional re-sequencing latency is measured in nanoseconds
Distributed Multi-Stage Multicast Replication
7. MGID lookup for NPU to 4. 22-bit FGID for slot bitmask
egress OIF interface based unique fabric LC 6. MGID in FIA DI table lookup
replication replications and FPOE lookup 5. MGID lookup into FPOE table for unique NPU replication
(256k entries) for unique FIA Flow based load balance
replications
7 SFP+
6 SFP+
FIA NP SFP+
SFP+
SFP+
SFP+
SFP+
CPAK 0 5 FIA NP SFP+
SFP+
SFP+
7 6FIA
PHY NP Fabric 6 7 SFP+
CPAK 1 SM15 FIA NP SFP+
SFP+
SFP+
SFP+
SFP+
CPAK 2 7
5
4
FIA NP SFP+
SFP+
SFP+
NP 6FIA SFP+
PHY RSP5 Tomahawk Line Card
CPAK 3
Switch SFP+
Fabric SFP+
CPAK 4 (SM15) FIA NP SFP+
SFP+
SFP+
PHY NP FIA SFP+
CPAK 5 3 FIA NP
SFP+
SFP+
SFP+
SFP+
Fabric SFP+
CPAK 6 2 SM15 FIA NP SFP+
SFP+
SFP+
PHY NP 1 FIA SFP+
SFP+
CPAK 7 RSP5 FIA NP SFP+
SFP+
SFP+
SFP+
Tomahawk Line Card
1. 512k MGID/FGID lookup
Flow based (RBH key calc) 2 & 3. Flow based (using RBH)
multi link load balance multi link load balance
Fabric Load Balancing – Multicast
Fabric
ASIC
Fabric
ASIC
Fabric ASIC Arbitration Fabric ASIC

C1 B2 A3 B1 A2 A1
and VOQ SFC or RSP0 and VOQ
Fabric
Flows exit in-order
ASIC
Fabric
ASIC
Arbitration
SFC or RSP1
• Multicast traffic hashed based on (S,G) info to maintain flow integrity

• Very large set of multicast destinations preclude re-sequencing
• Multicast traffic is non arbitrated – sent across a different fabric plane
58
ASR 9000 QoS Architecture
& TCAM usage
ASR9000 Priority-Based QoS Architecture
• Dedicated Traffic Manager(TM) for Traffic Queuing
• User Configurable QoS Policy on Ingress/Egress NP
• End-to-End priority propagation → Guarantee bandwidth, low latency for high priority traffic
• Unicast VOQ and back pressure
Ingress side of LC Egress side of LC
CPU CPU 4
P1
P2
NP0 PHY
P3
BE 3 P1
P1 P2
P3 NP1 PHY
2
P2
CPAK 0 TM P3
BE FIA BE
CPAK 1
PHY NP FIA Switch NP2 PHY
1 Fabric P1
NP3
TM
P2
P3 PHY
BE
2 3
1 4
• Ingress (sub-)interface • 4xVOQ per VQI 4x Egress Destination Qs per VQI,
QoS Queues aggregated at egress port rate
Egress (sub-)interface
• Up to 8K VOQs per TSK FIA (vs
• User Configurable 4k per SKT FIA) QoS Queues
Ingress QoS Policy User-configuration
with Egress MQC
User-configuration Implicit Configuration
with Ingress MQC Not User-controllable
Default Implicit Trust Model L2 IF: trust outer Cos
L3 IF: trust DSCP
L3 MPLS: trust outer EXP
IPP=5 DSCP=44
802.1p = 1 L2 Bridging internal
IPP=5 DSCP=44 cos = 1 802.1p = 1 *
IPP=5 DSCP=44
Carried in internal buffer header, ASR 9000 would never modify

by default, internal cos is used for impositioned fields only, packet DSCP/IP without
For example, added vlan tag, impositioned MPLS label,
a policy-map configured
It doesn’t include VLAN tag translation or MPLS label swap
IPP=5 DSCP=44
Untagged L2 Bridging
internal 802.1p = 0 *
IPP=5 DSCP=44 IPP=5 DSCP=44
cos = 0
Ingress line card Egress line card

3-Layer Hierarchical QoS (H-QoS)
policy-map child
class Pr1
L0 L1 L2 L3 L4 police rate 64 kbps
Port Group Port scheduler Grand-Parent Parent Policy Child(leaf) policy priority level 1
scheduler not not Policy class Pr2
configurable configurable Class-default & Class-default & police rate 10 mbps
Class-default user-defined user-defined priority level 2
only classes classes
class Cl3
bandwidth 3 mbps
class Cl4
set precedence routine
bandwidth 1 mbps
!
policy-map parent
class parent1
shape average 100 mbps
service-policy child
class parent2
•N/A •N/A •Shape •Priority Level 1 •Priority Level 1 shape average 25 mbps
•Bandwidth •Priority Level 2 •Priority Level 2 service-policy child
remaining •Shape •Shape
class class-default
•1R2C policer •bandwidth or •bandwidth or
bandwidth bandwidth
!
remaining remaining policy-map grand-parent
•(W)RED •(W)RED class class-default
•Police •Police shape average 500 mbps
•Set (marking) •Set (marking) service-policy parent
H-QoS – Supported Classification/Policy
Policy-map hierarchy level Classification support Policy Support
• Shape Average
• Bandwidth remaining
Grand-parent Only class-default
• 1R2C policer with only drop/transimit
action(no set/mark)
• Priority/WRED Queue and Queue-limit on

User defined classes with
Leaf only
Parent restrictions based on
• Policer/Shaper/Marking/non-Priority
format/interface types.
Queue/Bandwidth/Bandwidth Remaining
• Priority/WRED Queue and Queue-limit on

User defined classes with
Leaf only
Child restrictions based on
• Policer/Shaper/Marking/non-Priority
format/interface types.
Queue/Bandwidth/Bandwidth Remaining
Default Interface Queues
P1 P2 P3 L
Level 4 Queues
Level 3 Schedulers
Level 2 Schedulers
Level 1 Schedulers
MQC Hierarchy in Queuing ASIC
Port default queues MQC queues
policy-map child
P1 P2 P3 L c1 c2 cd-c class c1
priority level 1
police rate 640 kbps
class c2
bandwidth 20 mbps
class class-default cd-c
bandwidth 1 mbps
L4 !
policy-map parent
class class-default cd-p
L3 cd-p !
interface GigabitEthernet0/0/0/0
service-policy output parent
L2
Inactive entity
Active entity
L1
65
MQC Hierarchy in Queuing ASIC
policy-map child
Port default queues G0/0/0/0.1 G0/0/0/0.2 class c1
P1 P2 P3 L c1 c2 cd-c c1 c2 cd-c priority level 1
police rate 640 kbps
class c2
bandwidth 20 mbps
class class-default cd-c
bandwidth 1 mbps
!
policy-map parent
L4 class class-default cd-p
!
L3 cd-p cd-p interface GigabitEthernet0/0/0/0.1
!
interface GigabitEthernet0/0/0/0.2
L2
Inactive entity
L1 Active entity
TCAM Used for Traffic Classification
TCAM
Ingress linecard
From wire I/F ACL QOS Fwd
classification classification classification lookup
QoS ACL IFIB
IFIB action L2 rewrite
action action lookup
To fabric From fabric
egress linecard
ACL QOS ACL Fwd
L2 rewrite
action classification classification lookup
QoS To wire TCAM
action
4th/5th Generation LC TCAM Bank Sets
Bank3 Bank2 Bank1 Bank0

• TCAM organized into bank sets, with
20B Entry 4 arrays in each bank set.
• Each array composed of 512x160b

80B Entry
entries or 128x640b entries
512 • Entries can be arranged as 20B/80B

lines wide and can span multiple
lines/blocks
• LPTS on 4th and 5th Generation LC

uses hash table instead of iTCAM
which saves iTCAM usage
20B 20B 20B 20B
3rd and 4th Generation LC TCAM Partitions
TCAM Regions 3rd Generation LC 4th Generation LC
L2 Partition Physical Ports, Bundles Reserved Partition for each N/A, L2 interface
type classification does not
Encap Default
require TCAM resource
Encap Untagged, Encap
Any
Single VLAN, PWHE
Double VLAN, BVI
ODS2 (160 bit ODS2 iFIB Reserved Partition for iFIB N/A, LPTS does not require
entries) (IPv4 LPTS) TCAM resource
Common Reserved Partition for all Reserved Partition for all
ODS2 features ODS2 features
ODS8(640 bit ODS8 iFIB Reserved Partition for iFIB N/A, LPTS does not require
entries) (IPv6 LPTS) TCAM resource
Common Reserved Partition for all Reserved Partition for all
ODS8 features ODS8 features
4th Generation LC TCAM Feature Lookup Region
TCAM Region Features sharing resources Search mode

160-ING L2-ACL 160 bits
IPV4-ACL
IPv4-QOS
PBR-IPV4
PBR-MPLS
PBR-L2
IPV4-LI
160-EGR L2-ACL 160 bits
IPV4-ACL
IPv4-QOS
640-ING IPV6-ACL 640 bits
IPv6-QOS
PBR-IPV6
IPV6-LI
EDPL
BGP Flowspec
640-EGR IPV6-ACL 640 bits
IPv6-QOS
TCAM Partition Example –3rd Generation LC
Note: “show controllers rm tcam summary all all np X location X/X/X” for 4th Generation LC
TCAM Partition Example – 4th Generation LC
160-ING Region
TCAM Partition Example – 4th Generation LC
160-ING Region
5th Gen LC Default Dynamic TCAM Carving
Static Carving
160-ING 160-EGR
Customer
Customer
Customer
Customer • LSP -SE by default with dynamic 64K TCAM allocation
Service Service
➢ TCAM usage on-demand, TCAM allocation at minimum block size 4K
➢ TCAM has no ingress/egress, or IPv4/IPv6 boundary at LC bootup
640-ING 640-EGR
Customer Customer
➢ LC reload required when switching to static carving
Customer Customer
Service Service
• LSP -TR by default with dynamic 40K TCAM allocation

➢ LSP -TR has 40K TCAM space at LC bootup with block size 4K
5th Gen LC Dynamic Carving ➢ LSP -TR re-carving TCAM into 16 blocks at minimum block size 2.5K
➢ LC reload required when switching to static carving
TCAM chunk based dynamic

allocation to meet the service
utilization scales • Per NPU Control
Common Shared Pool
TCAM Usage Summary
• 5th generation LC Supports Dynamic TCAM Carving/Allocation for Flexible MD-

Scales
• No TCAM resource required for L2 Interface Classification and LPTS on 4th and 5th
Generation LCs.
• IPv4-ACL/QoS search mode: 160bits(20Bs) Partition
• IPv6-ACL/QoS search mode: 640bits(80Bs) Partition
• BGP FlowSpec (both v4 and v6): 640bits(80Bs) Partition. Ingress Direction Only
• PBR/Lawful Interception(LI): Ingress Direction Only
Pop Quiz ????
For an incoming untagged layer 2 frame, what is the priority value used
when the frame is processed inside ASR9000?
• 2
• COS bit
• 802.1p Value
• 0
Pop Quiz ????
As we discussed, most of IPv4 related features use ODS2 partition and most of
IPv6 related features use ODS8 partition. When BGP flowspec is configured for
IPv4 traffic and rules has been pushed, in which TCAM region are the rules
programed?
• L2 Partition
• ODS8 Common
• ODS2 Common
• ODS2 iFIB
Conclusion
• ASR9000 - Truly Carrier-Class Edge Router Provides:

• Rich Features, Flexible Service Capability
• Variety of Hardware to Meet Different Capacity Requirements
• Fully Distributed Architecture for High Performance and

Massive Scalability
• Uniform, Open and Modularized Software Architecture
Complete your
online session
survey • Please complete your session survey
after each session. Your feedback
is very important.
• Complete a minimum of 4 session
surveys and the Overall Conference
survey (starting on Thursday) to
receive your Cisco Live t-shirt.
• All surveys can be taken in the Cisco Events
Mobile App or by logging in to the Content
Catalog on ciscolive.com/emea.
Cisco Live sessions will be available for viewing on

demand after the event at ciscolive.com.
Continue your education
Demos in the
Walk-in labs
Cisco campus
Meet the engineer

Related sessions
1:1 meetings
Thank you

Brkarc 2003

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Brkarc 2003

Uploaded by

Copyright:

Available Formats

Cisco ASR 9000 System

• ASR9000 Products Introduction

ASR 9010 ASR 9910

Fixed 2RU 2 LC/6RU 4 LC/10RU 4 LC/14RU 8 LC/21RU 8 LC/21RU 10 LC/30RU 20 LC/44RU

MSE DCI Peering P/PE CE Mobility Broadband

• 2x BITS ports on RJ-45 • Alarm output serial port

LC Support Typhoon Tomahawk

Dedicated Fabric Cards

ASR 9904 ASR 9006 ASR 9010

ASR 9912 ASR 9922

ASR 9001 ASR 9901

ASR 9906 ASR 9910

3rd gen LC Tomahawk -TR, -SE

4x100GE & 8x100G CPAK OTN Line Card A9K-48X10GE-1G

4x100GE & 12x100G QSFP28 LAN Line Card

16 x 100GE SE line card

A9K-20HG-FLEX-SE/TR line card

A99-32X100GE-X-SE/TR line card

• 32 Ports 100G support

A9K-20HG-FLEX-SE/TR line card

A9K-8HG-FLEX-SE/TR line card

What different interface rates can ASR9000 LightSpeed Plus 2T Combo

Tomahawk LCs LightSpeed Plus

RP2 and RP3 can support SFC3

Tomahawk LCs LightSpeed Plus

1.075T with 5 fabrics

System automatically rate-limits each

Command to check if Rate-limiter has been applied

Dedicated queue ASIC – TM (traffic Provides data connection to switch fabric

• TCAM: VLAN tag, QoS and ACL classification

• Stats memory: interface statistics, forwarding statistics etc

• Frame memory: buffer, Queues

• Lookup Memory: forwarding tables, FIB, MAC, ADJ

Can LightSpeed Plus A99-32X100GE-X-SE/TR be used in

• 64-bit OpenEmbedded Linux support.

• ASR 9000 supports 64-bit Linux starting

• IOS XR Exists in two flavors Separate Admin

• No ISSU QNX Linux

• 64-bit in ASR9000, NCS 5500,

• Linux based LC-CPU

• Larger addressable memory Linux Linux

Switch Fabric Switch Fabric

LSD RIB RSP/RP CPU

Incomplete ADJ Packets RP CPU RP

When A9K-8HG-FLEX-SE/TR is used in the following chassis with

• Egress lookup performs packet-rewrite and applies egress features

Uniform packet flow for simplicity and predictable performance

Ingress side of LC CPU CPU 6 8

Fabric ASIC Arbitration Fabric ASIC

• Multicast traffic hashed based on (S,G) info to maintain flow integrity

Ingress side of LC Egress side of LC

Carried in internal buffer header, ASR 9000 would never modify

Ingress line card Egress line card

• Priority/WRED Queue and Queue-limit on

• Priority/WRED Queue and Queue-limit on

To fabric From fabric

Bank3 Bank2 Bank1 Bank0

• Each array composed of 512x160b

512 • Entries can be arranged as 20B/80B

• LPTS on 4th and 5th Generation LC

20B 20B 20B 20B

TCAM Region Features sharing resources Search mode

• LSP -TR by default with dynamic 40K TCAM allocation