Professional Documents
Culture Documents
BRKARC-2350 IOS Routing Internals PDF
BRKARC-2350 IOS Routing Internals PDF
BRKARC-2350
#clmel
Agenda
• Router Components
• Moving Packets
• CEF, CPU and Memory
• Outbound Load Sharing
• Routing Convergence
Improvements
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Agenda
Router Components
Data and Control Planes
– Software Based Routers
– Hardware Based Routers
– Hybrid Routers
• Moving Packets
• CEF, CPU and Memory
• Outbound Load Sharing
• Routing Convergence
Improvements
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Router Components
Data and Control Planes
• Control Plane Brains
– Control Traffic
• Routing Updates (BGP, EIGRP, OSPF, etc.)
• SSH
• SNMP
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
Agenda
Router Components
– Data and Control Planes
Software Based Routers
– Hardware Based Routers
– Hybrid Routers
• Moving Packets
• CEF, CPU and Memory
• Outbound Load Sharing
• Routing Convergence
Improvements
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
Router Components
Software Based Routers
• Software Based
– Shared control and data plane
– General Purpose CPU (slow and smart)
• Runs at CPU speed
• Speed/flexibility tradeoff
– CPU responsible for all operations
800/2800/2900/3900/7200 Series Routers are software based
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
Router Components
Software Based Routers
I/O
Memory
RX Ring CPU
Process
Tx Ring Memory
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
Agenda
Router Components
– Data and Control Planes
– Software Based Routers
Hardware Based Routers
– Hybrid Routers
• Moving Packets
• CEF, CPU and Memory
• Outbound Load Sharing
• Routing Convergence
Improvements
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
Router Components
Hardware Based Routers
• Hardware based
– Separated control and data plane
– CPU + ASIC (Application Specific Integrated Circuit)
– ASIC designed specifically to move packets (fast and dumb)
– CPU manages control plane
– CPU only moves packets the ASIC can’t
– Data Plane packets sent to the CPU are “punted”
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
Router Components
Hardware Based Routers
IO
Memory
CPU
Process
Memory
RAM
RX Ring
Tx Ring Forwarding
TCAM ASIC
Network Interface
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
Router Components
Hardware Based Routers
Control
Plane IO
Memory
Data CPU
Process
Plane
Memory
RAM
RX Ring
Tx Ring Forwarding
TCAM ASIC
Network Interface
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
Agenda
Router Components
– Data and Control Planes
– Software Based Routers
– Hardware Based Routers
Hybrid Routers
• Moving Packets
• CEF, CPU and Memory
• Outbound Load Sharing
• Routing Convergence
Improvements
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
Router Components
Hybrid Routers
• Hardware assisted
– Separated control and data plane
– CPU + NP (Network Processor)
– NP is multi-core specialised processor
– NP is optimised to move packets
– CPU manages control plane
– CPU only moves packets the NP can’t
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
Router Components
Hybrid Routers
Control
Plane IO
Memory
Data CPU
Process
Plane
Memory
RAM
RX Ring
Tx Ring Forwarding
TCAM ASIC
Network Interface
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
Router Components
Hybrid Routers
Control
Plane IO
Memory
Data CPU
Process
Plane
Memory
RAM
RX Ring
Tx Ring Dataplane
NP
Memory
Network Interface
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
Agenda
• Router Components
Moving Packets
Process Switching
– CEF Switching
• CEF, CPU and Memory
• Outbound Load Sharing
• Routing Convergence
Improvements
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
Overview
• CEF Switching and Process Switching
– Fast Switching is deprecated as of 12.4(20)T
– Not covered today
• CEF Switching is the default
• Process Switching is the fallback
– Anything CEF can’t handle
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
Process Switching
CPU
Interrupt
IO
Memory
L2 Hdr e0/0 e0/1
Rx Ring Tx Ring
Packet
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
Process Switching
Schedules IP
CPU
Input
Interrupt
IO
Memory
L2 Hdr e0/0 e0/1
Rx Ring Tx Ring
Packet
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
Process Switching
IP Scheduler
CPU
Input
Runs
IO
Memory
e0/0 Packet e0/1
Rx Ring Tx Ring
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
Process Switching
Find
IP Scheduler
Route CPU
Input
Routing L2 Hdr
Table Find
Adjacency
ARP
Table
IO
Memory
e0/0 Packet e0/1
Rx Ring Tx Ring
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
Process Switching
IP Scheduler
CPU
Input
Routing L2 Hdr
Table
ARP
Table
IO
Memory
e0/0 Packet e0/1
Rx Ring Tx Ring
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
Process Switching
IP Scheduler
CPU
Input
Routing
Table
ARP
Table
IO
Memory
e0/0 L2 Hdr e0/1
Rx Ring PacketTx Ring
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
Process Switching
• Process Switching is BAD
Router#show ip route 172.16.1.1
• Multiple lookups Routing entry for 172.16.1.1/32
Known via "bgp 65530", distance 20, metric 0
• Inefficient data structures * 10.0.0.1, from 10.0.0.1, 00:00:07 ago
Router#show ip route 10.0.0.1
• Process scheduling Routing entry for 10.0.0.1/32
Known via "static", distance 1, metric 0
* 192.168.1.1
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
Agenda
• Router Components
Moving Packets
– Process Switching
CEF Switching
• CEF, CPU and Memory
• Outbound Load Sharing
• Routing Convergence
Improvements
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
The FIB (Forwarding Information Base)
“Show IP CEF”
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
CEF Switching
CPU
Interrupt
IO
Memory
L2 Hdr e0/0 e0/1
Rx Ring Tx Ring
Packet
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
CEF Switching
IP Scheduler
CPU
Input
Route +
L2 Lookup
L2 Hdr CEF
Interrupt Table
IO
Memory
L2 Hdr e0/0 e0/1
Rx Ring Tx Ring
Packet
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
CEF Switching
CPU
Route +
L2 Lookup
L2 Hdr CEF
Interrupt Table
IO
Memory
e0/0 Packet e0/1
Rx Ring Tx Ring
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
CEF Switching
CPU
Route +
L2 Lookup
CEF
Interrupt Table
IO
Memory
e0/0 L2 Hdr e0/1
Rx Ring Tx Ring
Packet
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
CEF Switching - Summary
• Interrupt removes process scheduling
• Pre-compiled Interface + L2 information (cache)
• CEF table data structure improvement
– RIB is a hash
– CEF is a mtrie
• Single lookup for all necessary forwarding information
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
CEF Switching - Features
• Supported in CEF • Process Switching Only
– QoS – ACL Logging
– ACL – Packets destined to the router
– Zone Based Firewall – No L2 Adjacency
– NAT
– NetFlow
– IPSec
– GRE
– PBR
– Many more!
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
Agenda
• Router Components
• Moving Packets
CEF, CPU and Memory
Processes and Interrupts
– Routing Memory Utilisation
• Outbound Load Sharing
• Routing Convergence
Improvements
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
CEF and CPU Utilisation
• CPU does everything
• Total CPU vs. Interrupts Total CPU – Interrupts =
– SPF, BGP Utilisation Due to
Processes
– Routed Packets
CPU utilization for five seconds: 5%/2%; one minute: 3%; five minutes: 2%
PID Runtime (ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
...
2 68 585 116 1.00% 1.00% 0% 0 IP Input
17 88 4232 20 0.20% 1.00% 0% 0 BGP Router
18 152 14650 10 0% 0% 0% 0 BGP Scanner
...
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
CPU Utilisation Examples
1. CPU Utilisation due to moderate traffic rates
CPU utilization for five seconds: 47%/46%; one minute: 40%; five minutes: 39%
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
CPU Utilisation Examples
1. CPU Utilisation due to moderate traffic rates
CPU utilization for five seconds: 47%/46%; one minute: 40%; five minutes: 39%
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
CPU Utilisation Examples
1. CPU Utilisation due to moderate traffic rates
CPU utilization for five seconds: 47%/46%; one minute: 40%; five minutes: 39%
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
Process Priority
• Processes assigned priority
Critical
– Critical/High/Medium/Low
High
Medium
CPU Low
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
Process Priority
• Processes assigned priority
3 Critical
– Critical/High/Medium/Low
• Priority Scheduler
High
1 2 Medium
CPU Low
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
Process Priority
• Processes assigned priority
3 Critical
– Critical/High/Medium/Low
• Priority Scheduler
4 High
1 2 Medium
CPU Low
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
Process Priority
• Processes assigned priority
Critical
– Critical/High/Medium/Low
• Priority Scheduler
4 High
1 2 Medium
3
CPU Low
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
Process Priority
• Processes assigned priority
Critical
– Critical/High/Medium/Low
• Priority Scheduler
4 High
1 2 Medium
CPU Low
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
Process Priority
• Processes assigned priority
Critical
– Critical/High/Medium/Low
• Priority Scheduler
High
1 2 Medium
4
CPU Low
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
Process Priority
• Processes assigned priority
Critical
– Critical/High/Medium/Low
• Priority Scheduler
High
1 2 Medium
CPU Low
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
Process Priority
• Processes assigned priority
5 Critical
– Critical/High/Medium/Low
• Priority Scheduler
High
• Run to Completion Model
2 Medium
1
CPU Low
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
Process Priority
• Processes assigned priority
5 Critical
– Critical/High/Medium/Low
• Priority Scheduler
High
• Run to Completion Model
– Processes choose to suspend
2 Medium
1
CPU Low
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
Process Priority
• Processes assigned priority
5 Critical
– Critical/High/Medium/Low
• Priority Scheduler
High
• Run to Completion Model
– Processes choose to suspend
1 2 Medium
CPU Low
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 55
Process Priority
• Processes assigned priority
Critical
– Critical/High/Medium/Low
• Priority Scheduler
High
• Run to Completion Model
– Processes choose to suspend
– Interrupts break the rules
1 2 Medium
Interrupt! CPU 5
Low
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 56
Agenda
• Router Components
• Moving Packets
CEF, CPU and Memory
– Processes and Interrupts
Routing Memory Utilisation
• Outbound Load Sharing
• Routing Convergence
Improvements
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
Routing Process Memory
• Routing Protocol, RIB, and CEF each take their own memory
• RIB built from Routing Protocols
• CEF built from RIB
Proc Mem
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
Memory Impact of Multiple Prefixes
ISP B
ISP A ISP C
400k 400k
400k Prefixes Prefixes
Prefixes
3 BGP Peers
400k Identical Routes
15.2(2)T
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 59
Memory Impact of Multiple Prefixes
Memory Utilisation
50%
0 peers
40% 0 BGP entries 45% ISP A ISP C
41%
0 CEF entries 38%
30% ISP B
20%
3 BGP Peers
10%
5% 400k Identical Routes
0% 15.2(2)T
0 Peers 1 Peer 2 Peers 3 Peers
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
Memory Impact of Multiple Prefixes
Memory Utilisation
50%
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
Equal Cost Loadsharing
OSPF Cost 20
OSPF
B
172.16.2.0/24 Area 0
OSPF Cost 20
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 66
Routing Table – Equal Cost Routes
RouterB#show ip route 172.16.2.0
Routing entry for 172.16.2.0/24
Known via "ospf 1", distance 110, metric 20, type intra area
Last update from 172.16.1.1 on Ethernet0/0, 1d02h ago
Routing Descriptor Blocks:
* 192.168.100.1, from 192.168.200.1, 1d02h ago, via Ethernet0/1
Route metric is 20, traffic share count is 1
172.16.1.1, from 192.168.200.1, 1d02h ago, via Ethernet0/0
Route metric is 20, traffic share count is 1
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 68
Routing Table – Equal Cost Routes
RouterB#show ip route 172.16.2.0
Routing entry for 172.16.2.0/24
Known via "ospf 1", distance 110, metric 20, type intra area
Last update from 172.16.1.1 on Ethernet0/0, 1d02h ago
Routing Descriptor Blocks:
* 192.168.100.1, from 192.168.200.1, 1d02h ago, via Ethernet0/1
Route metric is 20, traffic share count is 1
172.16.1.1, from 192.168.200.1, 1d02h ago, via Ethernet0/0
Route metric is 20, traffic share count is 1
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 69
Routing Table – Equal Cost Routes
RouterB#show ip route 172.16.2.0
Routing entry for 172.16.2.0/24
Known via "ospf 1", distance 110, metric 20, type intra area
Last update from 172.16.1.1 on Ethernet0/0, 1d02h ago
Routing Descriptor Blocks:
* 192.168.100.1, from 192.168.200.1, 1d02h ago, via Ethernet0/1
Route metric is 20, traffic share count is 1
172.16.1.1, from 192.168.200.1, 1d02h ago, via Ethernet0/0
Route metric is 20, traffic share count is 1
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 70
CEF Hashing
• CEF hash is deterministic
– Same input always provides the same output
Packet 1 = src 10.1.1.1 dst 10.2.2.2
Packet 2 = src 10.1.1.1 dst 10.3.3.3
1
Without randomisation every
D router makes the same
1 B decision
2 E
A
2 1 F Downstream routers never
C
2 loadshare
G
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 77
CEF Hashing Algorithm
• Default hash is “Universal”
• Source IP + Destination IP + Universal Identifier
• Universal ID prevents polarisation
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 78
CEF Loadsharing Options
• Per-Packet
– More even load sharing
– Jitter
– Out of Order packets (bad for lots of applications)
• Per-Destination (default)
– Can be less even load sharing
– Ordered delivery
– Hashing challenges
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 79
CEF Hashing
RouterB#show ip CEF 172.16.2.1 internal
172.16.2.0/24, epoch 0, RIB[I], refcount 5, per-destination sharing
…
ifnums:
Ethernet0/0(3): 172.16.1.1
Ethernet0/1(4): 192.168.200.1
path 08172748, path list 100071A8, share 1/1, type attached nexthop, for IPv4
nexthop 172.16.1.1 Eth0/0, adj IP adj out Eth0/0, addr 172.16.1.1 081E35A0
path 08172898, path list 100071A8, share 1/1, type attached nexthop, for IPv4
nexthop 192.168.200.1 Eth0/1, adj IP adj out Eth0/1, addr 192.168.200.1 0F75D9F8
flags: Per-session, for-rx-IPv4, 2buckets
2 hash buckets
< 0 > IP adj out of Ethernet0/0, addr 172.16.1.1 081E35A0
< 1 > IP adj out of Ethernet0/1, addr 192.168.200.1 0F75D9F8
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 80
CEF Hashing
RouterB#show ip CEF 172.16.2.1 internal
172.16.2.0/24, epoch 0, RIB[I], refcount 5, per-destination sharing
…
ifnums:
Ethernet0/0(3): 172.16.1.1
Ethernet0/1(4): 192.168.200.1
path 08172748, path list 100071A8, share 1/1, type attached nexthop, for IPv4
nexthop 172.16.1.1 Eth0/0, adj IP adj out Eth0/0, addr 172.16.1.1 081E35A0
path 08172898, path list 100071A8, share 1/1, type attached nexthop, for IPv4
nexthop 192.168.200.1 Eth0/1, adj IP adj out Eth0/1, addr 192.168.200.1 0F75D9F8
flags: Per-session, for-rx-IPv4, 2buckets
2 hash buckets
< 0 > IP adj out of Ethernet0/0, addr 172.16.1.1 081E35A0
< 1 > IP adj out of Ethernet0/1, addr 192.168.200.1 0F75D9F8
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 81
CEF Hashing
RouterB#show ip CEF 172.16.2.1 internal
172.16.2.0/24, epoch 0, RIB[I], refcount 5, per-destination sharing
…
ifnums:
Ethernet0/0(3): 172.16.1.1
Ethernet0/1(4): 192.168.200.1
path 08172748, path list 100071A8, share 1/1, type attached nexthop, for IPv4
nexthop 172.16.1.1 Eth0/0, adj IP adj out Eth0/0, addr 172.16.1.1 081E35A0
path 08172898, path list 100071A8, share 1/1, type attached nexthop, for IPv4
nexthop 192.168.200.1 Eth0/1, adj IP adj out Eth0/1, addr 192.168.200.1 0F75D9F8
flags: Per-session, for-rx-IPv4, 2buckets
2 hash buckets
< 0 > IP adj out of Ethernet0/0, addr 172.16.1.1 081E35A0
< 1 > IP adj out of Ethernet0/1, addr 192.168.200.1 0F75D9F8
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 82
CEF Hashing
RouterB#show ip CEF 172.16.2.1 internal
172.16.2.0/24, epoch 0, RIB[I], refcount 5, per-destination sharing
…
ifnums:
Ethernet0/0(3): 172.16.1.1
Ethernet0/1(4): 192.168.200.1
path 08172748, path list 100071A8, share 1/1, type attached nexthop, for IPv4
nexthop 172.16.1.1 Eth0/0, adj IP adj out Eth0/0, addr 172.16.1.1 081E35A0
path 08172898, path list 100071A8, share 1/1, type attached nexthop, for IPv4
nexthop 192.168.200.1 Eth0/1, adj IP adj out Eth0/1, addr 192.168.200.1 0F75D9F8
flags: Per-session, for-rx-IPv4, 2buckets
2 hash buckets
< 0 > IP adj out of Ethernet0/0, addr 172.16.1.1 081E35A0
< 1 > IP adj out of Ethernet0/1, addr 192.168.200.1 0F75D9F8
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 83
CEF Hashing
RouterB#show ip CEF 172.16.2.1 internal
172.16.2.0/24, epoch 0, RIB[I], refcount 5, per-destination sharing
…
ifnums:
Ethernet0/0(3): 172.16.1.1
Ethernet0/1(4): 192.168.200.1
path 08172748, path list 100071A8, share 1/1, type attached nexthop, for IPv4
nexthop 172.16.1.1 Eth0/0, adj IP adj out Eth0/0, addr 172.16.1.1 081E35A0
path 08172898, path list 100071A8, share 1/1, type attached nexthop, for IPv4
nexthop 192.168.200.1 Eth0/1, adj IP adj out Eth0/1, addr 192.168.200.1 0F75D9F8
flags: Per-session, for-rx-IPv4, 2buckets
2 hash buckets
< 0 > IP adj out of Ethernet0/0, addr 172.16.1.1 081E35A0
< 1 > IP adj out of Ethernet0/1, addr 192.168.200.1 0F75D9F8
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 84
CEF Hashing
2 hash buckets
< 0 > IP adj out Ethernet0/0, addr 172.16.1.1 081E35A0
< 1 > IP adj out Ethernet0/1, addr 192.168.200.1 0F75D9F8
Eth0/0, 172.16.1.1
Source IP 0
CEF
Destination IP Hash or
1
Universal ID
Eth0/1, 192.168.200.1
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 85
CEF Hashing
RouterB#show ip CEF exact-route 192.168.2.38 172.16.2.24
192.168.2.38 -> 172.16.2.24 => IP adj out Ethernet0/1, addr 192.168.200.1
Eth0/0, 172.16.1.1
Source IP 0
CEF
Destination IP Hash or
1
Universal ID
Eth0/1, 192.168.200.1
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 86
Equal Cost Multipath - Summary
• CEF is built from the routing table
• Load sharing is part of routing decision
• Not 100% equal
• Based on Source IP + Destination IP + Universal ID
• Only one router
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 87
Agenda
• Router Components
• Moving Packets
• CEF, CPU and Memory
Outbound Load Sharing
– CEF Equal Cost Multipath
(ECMP)
Load Sharing with
Performance Routing (PfR)
• Routing Convergence
Improvements
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 88
Loadsharing Across Routers
CEF ECMP works per-router
No dynamic way to get even sharing across routers
20%
ISP1
WAN
Site
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 89
PfR Operations
Command and Control Infrastructure
Border Routers (BRs) communicate load to Master Controller (MC)
20%
ISP1
BR
WAN
Load Information Site
MC
BR
60% ISP2
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 90
PfR Operations
Master Controller analyses reports from Border Routers
20%
ISP1
? BR
WAN
Load Information Site
MC
BR
60% ISP2
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 91
PfR Operations
Master Controller analyses reports from Border Routers
MC detects policy violation
20%
ISP1
! BR
WAN
Load Information Site
MC
BR
60% ISP2
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 92
PfR Operations
Master Controller pushes routing updates
20%
ISP1
! BR
WAN
Routing Updates Site
MC
BR
60% ISP2
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 93
PfR Operations
Master Controller pushes routing updates
Border Routers adjust routing impacting load
48%
ISP1
BR
WAN
Routing Updates Site
MC
BR
40% ISP2
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 94
PfR Operations
Border Routers continue reporting
48%
ISP1
BR
WAN
Load Information Site
MC
BR
40% ISP2
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 95
PfR Operations
Border Routers continue reporting
Master Control continues analysing
48%
ISP1
? BR
WAN
Load Information Site
MC
BR
40% ISP2
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 96
PfR Summary
• PfR “lifecycle”
• Policy Enforcement
– BGP Local Preference
– Static Routes
– PBR
• PfR provides routing intelligence
• CEF and RIB are the same
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 97
Agenda
• Router Components
• Moving Packets
• CEF, CPU and Memory
• Outbound Load Sharing
Routing Convergence
Improvements
Fast Convergence Overview
– OSPF LFA
– BGP PIC
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 98
Routing Convergence – What’s to improve?
• Routing changes are bad
• Small changes can require (potentially) large recalculation
• Routing Protocols are slow
– Failure detection is fast
– Event propagation + calculation is the bottleneck
• Chain Reaction
– Protocol Change -> RIB Change -> CEF Change
• Protocol can already know what to do before failure
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 99
Agenda
• Router Components
• Moving Packets
• CEF, CPU and Memory
• Outbound Load Sharing
Routing Convergence
Improvements
– Fast Convergence Overview
OSPF LFA
– BGP PIC
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 101
OSPF Overview
• Link State Algorithm
– LSDB provides a view of the entire network
• Network changes exchanged via LSA (Link State Advertisement)
– Multiple events cause throttling (5000ms default)
• SPF algorithm determines best path
– Runs on receipt of LSA, delayed 5000ms (default)
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 102
OSPF Convergence
• Convergence =
Failure Detection + Event Propagation + SPF + FIB Update
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 103
OSPF Loop Free Alternate
10.1.1.0/24
A C
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 104
OSPF Loop Free Alternate
• OSPF presents a primary and backup to CEF
– Backup calculated from secondary SPF run
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 105
OSPF Loop Free Alternate
• Aims for <50ms reconvergence
• Triggers as soon as the failure is
detected
– NO fast hellos
– Use BFD!
5427
• Not enabled by default
– Added to 7600/ASR1000 in 15.1(3)S
– Added to NX-OS in 5.0(2)
8
LFA No LFA
milliseconds
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 106
OSPF Loop Free Alternate
• Fast failure detection is key!
• Single Box
• Not a replacement for SPF Tuning
A C
WAN
B D
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 107
Agenda
• Router Components
• Moving Packets
• CEF, CPU and Memory
• Outbound Load Sharing
Routing Convergence
Improvements
– Fast Convergence Overview
– OSPF LFA
BGP PIC
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 116
BGP Prefix Independent Convergence
• Today’s FIB is flat
10.1.1.0/24 192.168.1.1
10.1.2.0/24 192.168.1.1
10.1.3.0/24 192.168.1.1
400k prefixes
• 400k routes -> 400k updates
A B
• BGP often has same next hop
400k prefixes
• We can do better!
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 117
BGP Prefix Independent Convergence
• Instead of flat FIB, Hierarchical
10.1.1.0/24 192.168.1.1
10.1.2.0/24 192.168.1.1
10.1.3.0/24 192.168.1.1
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 118
BGP Prefix Independent Convergence
• Instead of flat FIB, Hierarchical
10.1.1.0/24
10.1.3.0/24
A CE2
D 10.1.1.0/24
10.1.2.0/24
IGP Next Hop 10.1.3.0/24
Prefixes BGP Next Hop ….
B
10.1.1.0/24
C
10.1.2.0/24
10.1.3.0/24
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 120
BGP PIC Core
Primary: A-B-C
Backup: A-D-C
CE1
B C
A CE2
D 10.1.1.0/24
10.1.2.0/24
IGP Next Hop 10.1.3.0/24
Prefixes BGP Next Hop ….
B
10.1.1.0/24
C
10.1.2.0/24
10.1.3.0/24
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 121
BGP PIC Core
Primary: A-B-C
Backup: A-D-C
CE1
B C
A CE2
D 10.1.1.0/24
10.1.2.0/24
IGP Next Hop 10.1.3.0/24
Prefixes BGP Next Hop ….
B
10.1.1.0/24
C
10.1.2.0/24
10.1.3.0/24
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 122
BGP PIC Core
Primary: A-B-C
Backup: A-D-C
CE1
B C
A CE2
D 10.1.1.0/24
10.1.2.0/24
IGP Next Hop 10.1.3.0/24
Prefixes BGP Next Hop ….
B
10.1.1.0/24
C IGP Next Hop
10.1.2.0/24
10.1.3.0/24 D
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 123
BGP PIC Core
Core
100000
10000
PIC
LoC (ms)
1000
no PIC
100
10
12 0
15 0
17 0
20 0
22 0
25 0
27 0
30 0
32 0
35 0
00
0
10 0
1
0
0
0
00
00
00
00
50
00
50
00
50
00
50
00
50
00
25
50
75
Prefix
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 128
Q&A
Complete Your Online Session Evaluation
Give us your feedback and receive a
Cisco Live 2015 T-Shirt!
Complete your Overall Event Survey and 5 Session
Evaluations.
BRKARC-2350 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public