Anatomy of Core Network Elements Josef Ungerman

Dubrovnik, Croatia, South East Europe
20-22 May, 2013
Anatomy
of Internet Routers
Josef Ungerman
Cisco, CCIE #6167
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Connect 1
Agenda
On the Origin of Species

• Router Evolution
• Router Anatomy Basics
Packet Processors
• Lookup, Memories, ASIC, NP, TM, parallelism
• Examples, evolution trends
Switching Fabrics
• Interconnects and Crossbars
• Arbitration, Replication, QoS, Speedup, Resiliency
Router Anatomy
• Past, Present, Future – CRS, ASR9000
• 1Tbps per slot?
BRKSPG-2772 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 2
Hardware Router
Control Plane vs. Data Plane
Peripherals CPU Route DRAM Routing and Forwarding Processor

- Cisco 10000 PRE
- Cisco 7300 NSE
- Cisco ASR903 RSP
Control NPU headers TM Packet DRAM - Cisco ASR1001/1002 (fixed)
Memories
IC
Port Adapters, SIP/SPA

Interconnect
interfaces interfaces
BRKSPG-2772 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public
Hardware Router
Centralized Architecture
Peripherals CPU Route DRAM RP (Route Processor)

- Cisco ASR1000 RP
- Cisco 7600 (MSFC board)
Control NPU headers TM Packet DRAM FP (Forwarding Processor)

Memories DRAM
- Cisco ASR1000 ESP
- Cisco 7600 (PFC board, TM on LC)
FP
IC CPU

Interconnect
Scaling the Forwarding Plane
NP Clustering
DRAM
RP
CPU
Centralized Router with ASIC Cluster
- ME3600X (fixed)
- ASR903 (modular with RSP)
- ASR1006 (modular with FP and RP)
Ctrl Mem Cluster Interconnect Ctrl Mem
NP TM NP DRAM
Pkt Mem IC, TM Pkt Mem
FP
Distributor
CPU

Interconnect
Switching Fabric
DRAM
RP
CPU
Centralized Router with Switching Fabric
- Cisco 7600 without DFC (TM on LC)
- Cisco ASR9001
Switching
Ctrl Mem Fabric Ctrl Mem
NP TM NP DRAM
Pkt Mem TM Pkt Mem
FP
CPU
Port Adapters – MPA

Interconnect
Distributed Architecture
Distributed Router
DRAM
- Cisco 7600 with DFC (ES+)
RP
CPU
- Cisco 12000
- Cisco CRS
- Cisco ASR9000
Ctrl Mem Ctrl Mem

DRAM NP TM NP DRAM
interfaces FP
Pkt Mem TM Pkt Mem
FP interfaces
CPU CPU
Switching
Fabric
Packet Processors
Packet Processing Trade-offs
Performance vs. Flexibility
CPU (Central Processing Unit)
• multi-purpose processors
CPU • high s/w flexibility [weeks], but low performance [1’s of Mpps]
• high power, low cost
• usage example: access routers (ISR’s)
ASIC (Application Specific Integrated Circuit)

• mono-purpose hard-wired functionality
ASIC
• complex design process [years]
• high performance [100’s of Mpps]
• high development cost (but cheap production)
• usage example: switches (Catalysts)
Input Demux Feedback

NP (Network Processor) = “something in between”
0
IM
4
IM
8
IM
12
IM
• performance [10’s of Mpps] + programmability [months]

NP
Mem.
Column
• cost vs. performance vs. flexibility vs. latency vs. power

IM IM IM IM
1 5 9 13
Mem.
Column
IM IM IM IM
• high development cost

2 6 10 14
Mem.
Column
IM IM IM IM
3 7 11 15
• usage example: coreedge, aggregation routers

Mem.
Column
Output Mux
“It is always something
(corollary). Good, Fast, Cheap:
Pick any two (you can’t have all three).”
RFC 1925
“The Twelve Networking Truths”
Hardware Routing Terminology
BGP OSPF
LDP RSVP-TE
Static
ISIS EIGRP
LSD RIB RP
ARP
FIB Adjacency
SW FIB
AIB LC NPU
AIB: Adjacency Information Base
RIB: Routing Information Base
LC CPU FIB: Forwarding Information Base
LSD: Label Switch Database
FIB Memory & Forwarding Chain
TLU/PLU
memories storing Trie Data (today typically RLDRAM)
Typically multiple channels for parallel/pipelined lookup
•PLU (Packet Lookup Unit) – L3 lookup data (FIB itself)

•TLU (Table Lookup Unit) – L2 adjacencies data (hierarchy, load-sharing)
PLU TLU
M-Trie or TBM
Root L3 Load L2 Load
L3 Table L2 Balancing
Balancing
Table
32-way 64-way
PLU Leaf TLU-0 TLU-1 TLU-2 TLU-3

for 10.1.1.0/24
Pointer chain allowing In-place Modify (PIC)

CAM
Content (Value) Result
. . .
CAM (Content Addressable Memory) 0008.7c75.f431 01
“Associative Memory” 0008.7c75.c401 02
0008.7c75.f405 03
SRAM with a Comparator at each cell . . .
Stable O(1) lookup performance

Is expensive & power-hungry
0008.7c75.f401 02
PORT ID
usage: L2 switching (MAC addresses) Query Result
L2 Switching (also VPLS)

Destination MAC address lookup  Find the egress port (Forwarding)
• Read @ Line-rate = Wire-speed Switching
Source MAC address lookup  Find the ingress port (Learning)
• Write @ Line-rate = Wire-speed Learning
TCAM
Content (Value/Mask) Result
. . .
TCAM (Ternary CAM) 192.168.100.xxx 801
192.168.200.xxx 802
“CAM with a wildcard” (VMR) 192.168.300.xxx 803
CAM with a Selector at some cells . . .
Stable O(1) lookup performance

3rd state – Don’t Care bit (mask) 192.168.200.111 802
usage: IP lookup (addr/mask) Query Result 
pointer ACL PERMIT/DENY
IP Lookup Applications
L3 Switching (Dst Lookup) & RPF (Src Lookup) TCAM Evolution
CAM2 – 180nm, 80Msps, 4Mb, 72/144/288b wide
Netflow Implementation (flow lookup)
CAM3 – 130nm, 125 Msps, 18Mb, 72/144/288b wide
ACL Implementation (Filters, QoS, Policers…) CAM4 – 90nm, 250Msps, 40Mb, 80/160/320b wide
various other lookups

4 Mpps, 3 Gbps
Pipelining Programmable ASIC u-programmable stages
2 per LC (Rx, Tx)
2002: Engine3 (ISE) – Cisco 12000
Parallelism Principle #1
Pipeline
• Systolic Array, 1-D Array
PLU/TLU Scale (Pipeline Depth):
DRAM SDRAM TCAM3 SRAMs • multiplies instruction cycle budget
• allows faster execution (MHz)
• non-linear gain with pipeline depth
ACL • typically not more than 8-10 stages
u-code L3 fwd CAM L2 fwd u-code QOS u-code
NF
DRAM
headers only
TM ASIC
DMA Logic
- 2K queues
- 2L shaping
16 Mpps, 10 Gbps
SMP Pipelining Programmable ASIC 130/90nm, u-programmable
2 per LC (Rx, Tx)
2004: Engine5 (SIP) – Cisco 12000 240W/10G = 24 W/Gbps
FIB
Parallelism Principle #2:
DRAM TCAM3 RLDRAM TCAM3 SRAMs SMP (Symmetric Multiprocessing)
• Multi-Core, Divide & Conquer
Scale (# of cores)
ACL
u-code u-code L3 fwd CAM L2 fwd u-code QOS u-code • Instruction/Thread/Process/App
NF
ACL granularity (CPU’s: 2-8 core)
u-code u-code L3 fwd CAM L2 fwd u-code QOS u-code
NF • IP: tiny repeating tasks (typically up
ACL
u-code u-code L3 fwd CAM L2 fwd u-code QOS u-code
to hundreds of cores)
NF
ACL
u-code u-code L3 fwd CAM L2 fwd u-code QOS u-code DRAM
NF
headers only
TM ASIC
DMA Logic
- 8K queues
- 2L shaping
SMP NPU 2004 QFA (SPP)
80 Mpps, 40 Gbps
QFA (Quantum Flow Array) – CRS 130nm, 188 cores
185M transistors
PLU/TLU Stats/Police ACL
2 per LC (Rx, Tx),~9 W/Gbps
DRAM RLDRAM
FCRAM SRAM TCAM4
TCAM3
Resources & Memory Interconnect 2010 QFA

125 Mpps, 140 Gbps
65nm, more cores, faster MHz
Bigger, Faster Memories (RLDRAM, TCAM4)
64K queues TM
2 per LC (Rx, Tx), ~4.2 W/Gbps
Processing Pool
256 Engines
188 Future
40nm version
400G
Pkt DRAM More cores, more MHz
Integrated TM
headers only Faster TCAM etc.
TM ASIC
64Kqueues
- 8K queues
Distribute & Gather Logic - 3L shaping
“If you were plowing a field, which would you rather use:
Two strong oxen or 1024 chickens?”
Seymour Cray
“What would Cinderella pick

to separate peas from ashes?”
Unknown IP Engineer
“Good multiprocessors are built

from good uniprocessors”
Steve Krueger
PPE (Packet Processing Elements)
Generic CPU’s or COT?
• NP does not need many generic CPU features
‒ floating point ops, BCD or DSP arithmetic
‒ complex instructions that compiler does not use
(vector, graphics, etc.)
‒ privilege/protection/hypervisor
‒ Large caches QFP:
• Custom improvements  >1.3B transistors
 >100 engineers
‒ H/W assists (TCAM, PLU, HMR…)
 >5 years of development
‒ Faster memories  >40 patents
‒ Low power
‒ C language programmable! (portable, code reuse)
Intel Core 2
Cisco QFP Sun Ultrasparc T2
Mobile U7600 Packaging Examples:
Total number processes
160 64 2  ESP5 = 20 PPEs @ 900MHz
(cores x threads)
 ESP10 = 40 PPEs @ 900MHz
Power per process 0.51W 1.01W 5W
 ESP20 = 40 PPEs @ 1200MHz
Scalable traffic management 128k queues None None  etc.
SMP NPU (full packets processing)
QFP (Quantum Flow Processor) – ASR1000 (ESP), ASR9000 (SIP)
2008 QFP
16 Mpps, 20 Gbps
90nm, C-programmable
sees full packet bodies
Central or distributed
Clustering XC
2012 QFP
Processing Pool 32 Mpps, 60 Gbps
Fast Memory Access 45nm, C-programmable
160 Engines
256 Clustering capabilities
(64 PPEs x 4 threads)
(40 SOC, Integrated TM
on-chip RLDRAM2 7
sees full packet bodies
resources SRAM TCAM4 Central engine (ASR1K)
RLDRAM2 0
complete packets
Resources & Memory Interconnect
complete packets
Distribute & Gather Logic TM ASIC

- 128K queues
- 5L shaping TM ASIC
Pkt DRAM
- 128K queues
Pkt DRAM
- 5L shaping
Pipelining SMP NPU
Cisco 7600 ES+, ASR9000
Pkt
Pkt DRAM
DRAM
DRAM TCAM SRAM
TM ASIC
Resources & Memory Interconnect - 32K queues
- 4L shaping
PARSE SEARCH1 RESOLVE SEARCH2 MODIFY
Processing Processing Proc. Proc. Proc. TM ASIC

Pool 1 Pool 2 Pool 3 Pool 4 Pool 5
INPUT -TM ASIC
256K queues
& -- 5L
32Kshaping
queues
PRE- - 4L shaping
PARSE headers only
Distribute & Gather Logic

learn TM ASIC
feedback - 32K queues
LEARN
feedback - 4L shaping
bypass
2008 NP [Trident]: 2011 NP [Typhoon]:

28 Mpps, 30 Gbps 90 Mpps, 120 Gbps Future
90nm, 70+ cores 55nm, lot more cores 40nm version
3 on-board TM chips (2 Tx) Integrated TM and CPU 200+G
2,4 or 8 per LC 2,4 or 8 per LC
565W/120G = 4.7 W/Gbps 800W/240G = 3.3 W/Gbps
7600 ES+, ASR9K –L/-B/-E ASR9K –TR/-SE
Switching Fabrics
Interconnects Technology Primer
Capacity vs. Complexity
arbiter
Bus
• half-duplex, shared media
• standard examples: PCI [800Mbps], PCIe [Nx 2.5Gbps], LDT/HT [25Gbs]
• simple and cheap
IC
Serial Interconnect
• full-duplex, point-to-point media
• standard examples: SPI [10Gbps], Interlaken [100Gbps]
• Ethernet interfaces are very common (SGMII, XAUI,…)
arbiter
Switching Fabric (cross-bar)
• full-duplex, any-to-any media
• proprietary systems [up to multiple Tbps]
• often uses double-counting (Rx+Tx) to express the capacity
Q: What’s the capacity? 4 fabric ports @ 10Gbps
Switching Fabric A: (ENGINEERING) 4 * 10 = 40Gbps full-duplex
A: (MARKETING) 4 * 10 * 2 = 80Gbps
SWITCHING FABRIC Egress

Fabric Port
IP/MPLS unaware part Linecards
• FPOE (Fabric Point of Exit)
speaks cells/frames • addressable entity
IP/MPLS IP/MPL IP/MPLS IP/MPLS • single duplex pipe
NETWORK PROCESSOR ASIC S ASIC ASIC
IP/MPLS aware part ASIC
RX RX RX RX
speaks packets
1 2 3 4
Ingress
IP/MPLS 1
Linecards ASIC
TX
2
IP/MPLS
TX
ASIC
MULTICAST
to slots 2,3,4
IP/MPLS
3
TX
ASIC
UNICAST
to slot 3 Type B (1960)
Western Electric
IP/MPLS 4
TX 100-point 6-wire
ASIC
FIA (Fabric crossbar switch
Interface ASIC)
Fabric Port engineering – examples
CRS-3/16 – 4.48Tbps (7+1)
8x 5Gbps links
5x 5Gbps links
• 16 linecards, 2 RP
• linecard up to 140G (next 400G)
• backwards compatible 14x 10G TX RX 100G
per-cell loadsharing
8 planes = 200Gbps
- 8/10 code = 160Gbps
- cell tax = ~141Gbps
Active RSP440
16x 7.5Gbps links
TX ASR9010 – 7.04Tbps (1+1)

RX
36x 10G Active RSP440 • 8 linecards, 2 RSP
2x 100G
• linecard up to 360G (next 800G)
per-frame loadsharing
• backwards compatible
4 planes = 480Gbps
- 24/26 code = ~440 Gbps
“Non-Blocking” voodoo
RFC1925: It is more complicated than you think.
Ingress Egress
Linecards Linecard
TX
10G 10G sRX Non-blocking!
• zero packet loss
TX
10G 10G RX • port-to-port traffic profile
• certain packet size
10G 10G RX
Blocking (same fabric)..?
TX
• packet loss, high jitter

TX
10G 10G RX
• added meshed traffic profile
• added Multicast
• added Voice/Video
Example: 16x Multicast Replication
Egress Replication
Ingress RX
Linecards
Good:
Egress Replication
TX RX
• Cisco CRS, 12000
RX
• Cisco ASR9K, 7600
TX
10Gbps of multicast
RX
eats 10Gbps fabric bw!
What if the fabric can’t replicate multicast?
Ingress Replication Flavors
Egress Linecards
Ingress RX Bad:
Linecards Ingress Replication
TX RX • central replication or
encapsulation engines
*) of course, this is used in centralized routers
TX RX
10Gbps of multicast
RX eats 160Gbps fabric bw!
63 64 (10G multicast impossible)
16 17 18 19 20 21 22 23 24 25 26 27 29 30 31 32
15
07
14
03
RX
Ingress Good-enough/Not-bad-enough:
13
06
12
RX
01
33 34 36 38 40 42 44 46 48
Linecards
11
Binary Ingress Replication

05
10
02
TX RX
09
• dumb switching fabric

04
08
TX RX
• non-Cisco
TX RX
TX RX
10Gbps of multicast
RX
eats 80Gbps fabric bw!
RX
(10G multicast impossible)
Cell dip explained
cell format cell cell payload
Fixed overhead [cell header, ~10%]
example:
hdr [48B] Relative overhead [fabric header]
[5B]
Variable overhead [padding]
cell buffer Good efficiency

IP Packet
40B IP hdr hdr [40B]
1Mpps = 1Mcps
[5B] [8B] 1Gb/s  1.33Gb/s
Packet:
cell buffer cell Poor efficiency
IP Packet empty
41B IP hdr hdr [first 40B] hdr [47B padding]
1Mpps = 2Mcps
[5B] [8B] [5B] 1Gb/s  2.6Gb/s
Packet: IP Packet [last 1B]
cell buffer IP Packet cell Fair efficiency
empty
64B IP hdr hdr [first 40B] hdr [24B padding] 1Mpps = 2Mcps
[5B] [8B] [5B] 1Gb/s  1.7Gb/s
Packet: IP Packet [last 24B]
Cell Tax effect on traffic: saw-tooth curve super-cell or super-frame (packet packing)
% of Linerate
cell buffer buffer

IP Packet 1 hdr
IP Packet 2
hdr hdr
cell buffer
hdr hdr
beginning of IP Packet 3
cell buffer
rest of IP Packet 3 hdr
IP Packet 4
hdr
40
100
200
300
400
500
600
700
800
900
100
0
110
0
120
0
130
0
140
0
150
0
BRKSPG-2772 L3 Packet Size
© 2012 Cisco [B]
and/or its affiliates. All rights reserved. Cisco Public
Q: Is this SF non-blocking?
Cell dip A: (MARKETING) Yes Yes Yes !
A: (ENGINEERING) Non-blocking for unicast
packet sizes above 53B.
The lower ingress speedup, the more cell

dips.
1N0%
Percentage of Linerate
100%
80%
60%
Cell dip – per vendor @ 41B, 46B, 53B, 103B,...
40%
20%
0%
1400
40
100
200
300
400
500
600
700
800
900
1000
1100
1200
1300
1500
L3 Packet Size [B]
Cell dip gets bad – too low speedup (non-Cisco)
MARKETING: this is non-blocking fabric*
*) because we can find at least one packet size that does not block
1N0%
100%
80%
60%
40%
20%
0%
1400
40
100
200
300
400
500
600
700
800
900
1000
1100
1200
1300
1500
L3 Packet Size [B]
Cell dip gets worse – multicast added (non-Cisco)
*) because we can still find at least one packet size that does not block, and your network does not have that much multicast anyway
1N0%
100%
80%
60%
40%
20%
0%
1400
40
100
200
300
400
500
600
700
800
900
1000
1100
1200
1300
1500
L3 Packet Size [B]
Router is blocking – 1 fabric cards fails (non-Cisco)
*) because nobody said the non-blocking capacity is “protected”
1N0%
100%
80%
60%
40%
20%
0%
1400
40
100
200
300
400
500
600
700
800
900
1000
1100
1200
1300
1500
L3 Packet Size [B]
What is “Protected Non-Blocking”
8 Fabric Cards
CRS-1: 56G  49G  42G 2x egress speedup
CRS-3: 141G  123G  106G is always kept
CRS-1
40G non-blocking even with
1 or 2 failed fabric cards 40G
40G TX RX
100G 100G
CRS-3
100G eth. non-blocking with
1 or 2 failed fabric cards
X
X
failed RSP: 440G  220G Active RSP
ASR9000
TX
Active RSP
RX 200G non-blocking even with a
2x 100G 2x 100G failed RSP
X
HoLB (Head of Line Blocking) problem
Solution 1: Solution 2:
Traffic Lanes per direction Enough Room 
wait
(= Virtual Output Queues) (= Speedup)
grrr -
blocked
go go
wait wait
go
Red Light, or
Traffic Lights “Traffic Jam
(= Arbiter) Ahead“ message Highway Radio
(= Backpressure) (= Flow Control)
Good HoLB Solutions
Fabric Scheduling + Backpressure + QoS
Arbitrated: Implicit backpressure
Ingress Egress
arbiter + Virtual Output Queues (VOQ)
Linecards Linecard
10G 10G sRX
• Cisco 12000 (also ASR9000)
TX
• per-destination slot queues VOQ
IP ASIC  fab ASIC
no Grant = implicit backpressure (Virtual Output Queues)
8q 10G 10G
8q
TX RX
• GSR: 16 slots + 2 RP’s * 8 CoS
+ 8 Multicast CoS =152 queues per LC
Virtual Output Queues (IP)
-8 per destination (IPP/EXP)
-Voice: strict scheduling
-Multicast: separate queues
Speedup Queues (packets)
explicit -u’cast: strict EF, AF, BE
backpressure -m’cast: strict Hi, Lo Output Buffered: Explicit backpressure
141G 226G RX
+ Speedup & Fabric Queues
TX
• Cisco CRS (1296 slots!)

TX
141G 226G RX
• 6144 destination queues
S1 S2 S3 • 512 speedup queues
Input Q’s (IP) Destination Queues (packets) Fabric Queues (cells)
-configurable -u’cast: strict Hi, Lo -u’cast: strict Hi, Lo • 4 queues at each point
-shaped -m’cast: strict Hi, Lo -m’cast: strict Hi, Lo (Hi/Lo UC/MC) + vital bit
rnxm rmxn
Multi-stage Switching Fabrics crossbars
1
1
crossbars
1
n 1 2 1 n
…
Multi-stage Switching Fabric 2 2
• constructing large switching fabric out of smaller SF elements … … …
N r … r N
m
50’s: Ch. Clos – general theory of multi-stage telephony switch Unidirectional Communication N=rxn
60’s: V. Beneš – special case of rearrangeably non-blocking Clos (n = m = 2)
8 planes CRS-1 – Benes

TX
TX RX
RX • Multi-chassis capabilities (2+0, N+2,…)
• Massive scalability: up to 1296 slots !!!
TX
TX RX
RX • Output-buffered, speedup, backpressure
S1 S2 S3
2-7 planes ASR9000 – Clos

RX
TX
• Single-chassis so far
• Scales to 22 slots today
TX RX
• Arbitrated VOQ’s
S1 S2 S3
Virtual Output Queuing
• VOQ on ingress modules represents fabric capacity on egress modules
• VOQ is “virtual” because it represents egress capacity but resides on ingress modules,
however it is still physical buffer where packets are stored
• VOQ is not equivalent to ingress or egress fabric channel buffers/queues
• VOQ is not equivalent to ingress or egress NP/TM queues
IP lookup,
ASR9000
IP lookup,
set VOQ Backup/Shadow
Egress EFP • Multi-stage Fabric
Primary Queuing
Arbiter Arbiter • Granular Central VOQ arbiter
VOQ • VOQ set per destination
FIA Q
NP Fab Q Fab Q NP Destination is the NP, not just slot
1NP 1NP TM
2
4 per 10G, 8 per 40G, 16 per 100G
2
NP
3NP
NP • 4 VOQ’s per set
3NP
4
NP 4 4 VOQ’s per destination, strict priority
NP
5NP 5NP Up to 4K VOQ’s per ingress FIA
6
NP
6
NP • Example (ASR9922):
7NP 7NP
8
20 LC’s * 8 10G NP’s * 4 VOQ’s
FIA 8
FIA = up to 640 VOQ’s per ingress FIA
Router Anatomy
2004: Cisco CRS – 40G+ per slot
SPP (Silicon Packet Processor)

- 40 Gbps, 80 Mpps [u-programmable] RP (active) RP (standby)
- one for Rx, one for Tx processing
CPU CPU CPU CPU
SIP- MSC-40 – 40G FP-40 – 40G PLIM

800
49G rx
56G Switch Fabric Cards
112G tx
98G tx (8 planes active)
NP NP
midplane 40G
midplane 40G
SPA TM TM
TM TM TM TM
NP NP
SPA CPU CPU
4, 8 or 16 Linecard slots + 2 RP slots
2010: Cisco CRS – 100G+ per slot
Next: 400G+ per slot (4x 100GE)
same backward-compatible architecture, same upgrade process
RP (active) RP (standby)
CPU CPU CPU CPU

56G
49G rx Switch Fabric Cards
800 112G tx
98G tx
(8 planes active)
NP NP
midplane 40G
midplane 40G
SPA TM TM
TM TM TM TM
NP NP
SPA CPU CPU
FP-140 – 140G MSC-140– 140G
100GE NP NP
midplane 140G
midplane 140G
TM TM
CMOS
DSP/ADC
TM TM TM TM 14x 10GE
NP NP
CPU CPU
141G
123G rx
4, 8 or 16 Linecard slots + 2 RP slots 226G
197G tx
QFA (Quantum Flow Array)
- 140 Gbps, 125 Mpps [programmable]
- one for Rx, one for Tx processing
CRS Multi-Chassis
CPU CPU CPU CPU

800 Switch Fabric Cards
(8 planes active)
NP NP
midplane 40G
midplane 40G
SPA TM S2 TM
S1 S3
TM TM TM TM
NP NP
SPA CPU CPU
FP-140 – 140G MSC-140– 140G
100GE NP NP
midplane 140G
midplane 140G
TM TM
CMOS
DSP/ADC
NP NP
CPU CPU
CRS Multi-Chassis (Back-to-Back, 2+0)
CPU CPU CPU CPU

800
NP NP
midplane 40G
midplane 40G
SPA TM
S13 S13
TM
TM TM TM TM
NP NP
SPA CPU CPU
FP-140 – 140G MSC-140– 140G
100GE NP NP
midplane 140G
midplane 140G
TM TM
CMOS
DSP/ADC
NP NP
CPU CPU

Switch Fabric Cards
BRKSPG-2772
(8 planes active)
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public
CRS Multi-Chassis (N+1, N+2, N+4)
Fabric Chassis
Shelf
RP (active) Controller RP (standby)
CPU CPU CPU CPU

800
NP NP
midplane 40G
S2
midplane 40G
SPA TM
S1 S3
TM
TM TM TM TM
NP NP
SPA CPU CPU
FP-140 – 140G MSC-140– 140G
100GE NP NP
midplane 140G
midplane 140G
TM TM
CMOS
DSP/ADC
NP NP
CPU CPU

Switch Fabric Cards
BRKSPG-2772
(8 planes active)
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public
2009: Cisco ASR9000 – 80G+ per slot
Trident Network Processor RSP (Route/Switch Processor)

- 30 Gbps, 28 Mpps [programmable] • CPU + Switch Fabric
- shared for Rx and Tx processing • active/active fabric elements
- one per 10GE (up to 8 per card)
RSP2 (active)
Core or Edge LC – 8x 10GE CPU CPU Core or Edge LC – 4x 10GE

TM NP
92G
46G
NP NP TM
TM
TM NP
NP NP TM
TM
TM NP CPU 92G
46G CPU
NP NP TM
TM NP TM
NP NP TM
RSP2 (fab. active)
CPU CPU
4 or 8 Linecard slots
(4 planes active)
2011: Cisco ASR9000 – 200G+ per slot
Next: 800G+ per slot
new RSP, faster fabric, faster NPU’s, backwards compatible
RSP (Route/Switch Processor)
• CPU + Switch Fabric
RSP440 (active) • active/active fabric elements
Core or Edge LC – 8x 10GE CPU CPU Core or Edge LC – 4x 10GE

92G
46G
TM NP
NP NP TM
TM
TM NP
NP NP TM
TM
TM NP CPU 92G
46G CPU
NP NP TM
TM NP TM
NP NP TM
Core or Edge LC – 24x 10GE RSP440 (fab. active) Core or Edge LC – 2x 100GE
TM NP’
NP NP’ TM
TM CPU CPU TM
TM NP’
NP NP’ TM
TM NP’ CPU CPU
NP NP’ TM
TM TM
TM NP’
NP NP’ TM
440G
220G
4 or 8 Linecard slots
Typhoon Network Processor
(4 planes active) - 120 Gbps, 90 Mpps [programmable]
BRKSPG-2772
- shared or dedicated
© 2012 Cisco and/or its affiliates. All rights reserved.
for Rx and Tx
Cisco Public
2012: Cisco ASR9922 – 500+G per slot
Next: 1.5T+ per slot
7 fabric cards, faster traces, faster NPU’s, backwards compatible
CPU CPU CPU CPU
36x TGE 2x 100GE

CPU
550G
Switch Fabric Cards NP’ TM
TM NP’ TM TM
NP’ TM (5 planes active) NP’ TM
TM NP’ TM CPU
NP’ TM NP’ TM
TM
TM NP’ TM
NP’ TM
NP’ TM
24x TGE 24x TGE

TM NP’ NP’ TM
NP TM NP
TM
TM NP’ NP’ TM
NP NP
TM NP’ CPU CPU NP’ TM
NP TM TM NP
TM NP’ NP’ TM
NP NP
10 or 20 Linecards + 7 SF’s + 2 RP’s

Typhoon Network Processor
- 120 Gbps, 90 Mpps [programmable]
BRKSPG-2772
- shared or dedicated
© 2012 Cisco and/or its affiliates. All rights reserved.
for Rx and Tx
Cisco Public
Entering the 100GE world
Router port cost break-down
10G 40G 100G
DWDM
TDM + Packet Optics
Switching &
Routing
DWDM
Commons
Core Routing Example

• 130nm (2004)  65nm (2009): 3.5x more capacity, 60% less Watt/Gbps, ~8x less $/Gbps
• 40nm (2013): up to 1Tbps per slot, adequate Watt/Gbps reduction…
Cisco puts 13% revenue
Silicon keeps following Moore’s Law
Optics is fundamentally an analog problem ! (almost 6B$ annually) to R&D
cca 20,000 engineers
Terabit per slot… CFP
CMOS Photonics
What is CMOS Photonics?
• Silicon is semi-transparent for SM wavelengths
• Use case: Externally modulated lasers
LASER CPAK
color 1
LASER Simple Laser Specifications
color 2
LASER Low Power with Silicon

color n based drive & modulation
System
Signaling Optical
MUX
10x 100GBase-LR ports per slot

• 70% size and power reduction!
• <7.5W per port (compare with existing CFP at 24W)
• 10x 10GE breakout cable (100x 10GE LR ports per slot)
NPU Model (40nm)
Terabit per-slot… Resources & Memory Interconnect
How to make it practically useful?

Processing Pool
Silicon Magic @ 40nm
• Power zones inside the NPU – low power mode
• Duplicate processing elements – in-service u-code upgrade
Distribute & Gather Logic

Optical Magic @ 100G
• Single-carrier DP-QPSK modulators for 100GE (>3000km)
• CMOS Photonics
• ROADM
Data Plane Magic – Multi-Chassis Architectures
• nV Satellite
• DWDM and OTN shelves
Control Plane Magic – SDN (Software Defined Networks)

• nLight: IP+Optical Integration
• MLR – Multi-Layer Restoration
• Optimal Path & Orchestration (DWDM, OTN, IP, MPLS)
Evolution: Keeping up with Moore’s Law
higher density = less W/Gbps, less $/Gbps
Switching Fabrics
• Faster, smaller, less power hungry
• Elastic – multi-stage, extensible
• Integrated – VOQ systems, arbiter systems, multi-functional fabric elements
Packet Processors
• 45nm, 40nm, 28nm, 20nm process
• Integrated functions – TM, TCAM, CPU, RLDRAM, OTN,…
• ASIC slices – Firmware ISSU, Low-power mode, Partial Power-off
Router Anatomy
• Control plane enhancements – SDN, IP+Optical
• DWDM density – OTN/DWDM satellites
• 100GE density – CMOS optics, CPAK/CFP4
• 10GE density (TGE breakout cables) and GE density (GE satellites)
Thank You.
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Connect 52

Anatomy of Core Network Elements Josef Ungerman

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Anatomy of Core Network Elements Josef Ungerman

Uploaded by

Copyright:

Available Formats

Dubrovnik, Croatia, South East Europe

20-22 May, 2013

On the Origin of Species

Peripherals CPU Route DRAM Routing and Forwarding Processor

Port Adapters, SIP/SPA

Peripherals CPU Route DRAM RP (Route Processor)

Control NPU headers TM Packet DRAM FP (Forwarding Processor)

Port Adapters, SIP/SPA

Port Adapters, SIP/SPA

Port Adapters – MPA

Ctrl Mem Ctrl Mem

ASIC (Application Specific Integrated Circuit)

Input Demux Feedback

• performance [10’s of Mpps] + programmability [months]

• cost vs. performance vs. flexibility vs. latency vs. power

• high development cost

• usage example: coreedge, aggregation routers

(corollary). Good, Fast, Cheap:

Pick any two (you can’t have all three).”

•PLU (Packet Lookup Unit) – L3 lookup data (FIB itself)

PLU Leaf TLU-0 TLU-1 TLU-2 TLU-3

Pointer chain allowing In-place Modify (PIC)

Stable O(1) lookup performance

L2 Switching (also VPLS)

Stable O(1) lookup performance

various other lookups

Resources & Memory Interconnect 2010 QFA

“What would Cinderella pick

“Good multiprocessors are built

Distribute & Gather Logic TM ASIC

Processing Processing Proc. Proc. Proc. TM ASIC

Distribute & Gather Logic

2008 NP [Trident]: 2011 NP [Typhoon]:

SWITCHING FABRIC Egress

TX ASR9010 – 7.04Tbps (1+1)

• packet loss, high jitter

Binary Ingress Replication

• dumb switching fabric

cell buffer Good efficiency

cell buffer buffer

The lower ingress speedup, the more cell

• Cisco CRS (1296 slots!)

• constructing large switching fabric out of smaller SF elements … … …

60’s: V. Beneš – special case of rearrangeably non-blocking Clos (n = m = 2)

8 planes CRS-1 – Benes

2-7 planes ASR9000 – Clos

SPP (Silicon Packet Processor)

SIP- MSC-40 – 40G FP-40 – 40G PLIM

4, 8 or 16 Linecard slots + 2 RP slots

CPU CPU CPU CPU

SIP- MSC-40 – 40G FP-40 – 40G PLIM

FP-140 – 140G MSC-140– 140G

CPU CPU CPU CPU

SIP- MSC-40 – 40G FP-40 – 40G PLIM

FP-140 – 140G MSC-140– 140G

4, 8 or 16 Linecard slots + 2 RP slots

CPU CPU CPU CPU

SIP- MSC-40 – 40G FP-40 – 40G PLIM

FP-140 – 140G MSC-140– 140G

4, 8 or 16 Linecard slots + 2 RP slots

CPU CPU CPU CPU

SIP- MSC-40 – 40G FP-40 – 40G PLIM

FP-140 – 140G MSC-140– 140G

4, 8 or 16 Linecard slots + 2 RP slots