BRKARC-1006-SD-WAN Vedge and XE SD-WAN Platform Architecture Overview PDF

#CLUS
SD-WAN vEdge and

XE SD-WAN Platform
Architecture Overview
Jason Yang – Technical Marketing Engineer
Sutheendiran Vijendiran – Software Engineer
BRKARC-1006
#CLUS
Your SD-WAN learning map at CLUS
Sunday Monday Tuesday Wednesday Thursday
TECCRS-2014
BRKRST-2791
Deep Dive
Policy BRKCRS-2113 BRKARC-1006
BRKRST-2558 Cloud onRamp Platform ARC
SD-WAN as a BRKARC-1006
Managed Service Platform ARC
BRKARC-1004 BRKRST-2559
TECRST-2191 On-prem
BRKRST-2095 Solution View
Deployment / BCP Deployment
Migration
BRKSEC-2342
Security
TECSEC-2355
Security BRKRST-2560
BRKCRS-2110 BRKCRS-2110 Analytics / ML
The foundation The foundation
#CLUS BRKARC-1006 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Agenda
• Introduction to Cisco SD-WAN Platforms
• vEdge and XE SD-WAN Platforms Architecture
• Life of a packet in SD-WAN Platforms
• vEdge and XE SD-WAN Software Architecture
• Migration from classic IOS XE to XE SD-WAN
• XE SD-WAN Platforms Serviceability
• Conclusion
What Will Not Be Covered….
• How SD-WAN works
• Platform Performance and Scale
• SD-WAN/Platform License
• Virtual/Cloud Platforms
Cisco Webex Teams
Questions?
Use Cisco Webex Teams to chat
with the speaker after the session
How
1 Find this session in the Cisco Live Mobile App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space
Webex Teams will be moderated cs.co/ciscolivebot# BRKARC-1006

by the speaker until June 16, 2019.
#CLUS © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
Introduction to
Cisco SD-WAN
Platforms
#CLUS
Cisco SD-WAN Architecture
Management Plane Analytics
• Single pane of glass • Machine Learning
vManage vAnalytics
• Monitoring and Troubleshooting Cloud Management • Carrier Performance
Analytics
• RBAC and APIs APIs • Bandwidth Forecasting
Control Plane 3rd Party

Automation
• SDN Architecture
• Intelligent Routing and Security
Distribution
• Horizontal Scale, Low Complexity vSmart Controllers
Data Plane Quality of Security Application
Services Visibility
• Physical or Virtual
MPLS 4G
• Zero Touch Provisioning
INET
• On-Premise or Cloud SD-WAN Routers
What are they?

Cloud Data Center CoLo Campus Branch How they work?
USERS
Cisco SD-WAN Routing Portfolio
Branch Hub
Cisco ENCS ISR 1000 ISR 4000 ASR 1000
XE SD-WAN
BRKARC-1006
• Service chaining virtual • Hardware and software redundancy

• WAN and voice module
functions • Integrated wired and • High-performance service with
flexibility
• Options for WAN wireless access hardware assist
• Compute with UCS E
connectivity • PoE/PoE+
• Integrated Security stack
• Open for 3rd party services
& apps • WAN Optimization
vEdge 5000
vEdge 100 vEdge 1000 & 2000
vEdge
• Modular
• 4G LTE & Wireless • Fixed/Pluggable • RPS
Module
Virtual and Cloud

CSR 1000V
vEdge Cloud Not in the scope of BRKARC-1006
• Software Router Platform
• Can be deployed in private,
• Cisco DNA virtualization
• Extend enterprise routing,
public, and hybrid cloud security & management to cloud
vEdge Platform Overview (1)
vEdge 100 vEdge 1000 vEdge 2000 vEdge 5000
I/O 5x 10/100/1000 Mbps 8x 1G SFP 4x 1G SFP, 2x PIM* 4x NIM*

RJ-45 PIM: 2x10G SFP, 8x 1G SFP NIM: 8x1G Copper, 8x1G SFP, 4x10G SFP
External USB vEdge 100: N/A 2x USB3.0 2x USB3.0 2x USB3.0

vEdge 100m & 100wm: 1x USB3.0
PoE vEdge 100: N/A N/A N/A N/A

vEdge 100m & 100wm: 1x port
4G LTE vEdge 100: N/A N/A N/A N/A

vEdge 100m & 100wm: 1x port
Console Port Mini-USB Mini-USB, RJ-45 Mini-USB, RJ-45 Mini-USB, RJ-45
Mgmt port N/A RJ-45 (10/100/1000) RJ-45 (10/100/1000) RJ-45 (10/100/1000)
Power Supply vEdge 100: External AC-DC Adapter External AC-DC Adapter Hot-swappable PSU / 1+1 Hot-swappable PSU / Yes
/ Redundancy vEdge 100m & 100wm: Internal / Yes active-active
fixed AC Adapter / N/A
FANs vEdge 100: N/A 2, Front to back 2, hot swappable, Front to 4, hot swappable, Front to back
vEdge 100m & 100wm: 1x fan, fixed back
HxWxD 1.72x7.5x5.5 in. 1.75x7.5x10 in. 1.75x17.25x18.5 in. 1.75x17.25x22.8 in.
* PIM: Pluggable Interface Module, NIM: Network Interface Module – not the same NIM as ISR4000
vEdge Platform Overview (2)
vEdge 100 vEdge 1000 vEdge 2000 vEdge 5000
CPU Cavium 7020 2-cores Cavium 6130 4-cores Cavium 6880 32-cores Intel Haswell-EP 14-
@800MHz @1.0GHz @1.2GHz cores @2.2GHz
Encryption N/A N/A N/A Intel QAT
Acceleration
Memory 2GB DDR3 4GB DDR3 8GB DDR3 32GB DDR4
Flash 4GB 8GB 8GB 120GB
Cisco ASR1000 Series Routers
2.5 Gbps to 200 Gbps – Available Today
XE SD-WAN
*CEF IMIX on IOS-XE
Fixed Chassis IOS-XE Modular Chassis
Compact, Power Router Business Critical Resiliency Instant On Service Delivery

High Services performance Secure solutions
Separate control and forwarding planes
Investment protection Optimal Application experience
Multiple form factors HW and SW redundancy
Software consumption model
#CLUS © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
ASR 1000 Fixed Chassis Overview
ASR 1001-X ASR 1002-X ASR 1001-HX ASR 1002-HX
SPA Slots 1 3 N/A N/A
EPA Slots N/A N/A N/A 1
NIM Slots 1 N/A N/A 1
Built-In GE 6 6 8 8
Built-In TenGE 2 N/A 4 + 4 (configurable 10GE/GE) 8
CPU 2.0GHz quad-core 2.13GHz quad-core 2.5GHz quad-core 2.5GHz quad-core
Memory 8GB/16GB 4GB/ 8GB/16GB 8GB/16GB 16GB/32GB
eUSB(8GB) eUSB(8GB) eUSB(32GB)

Storage eUSB(32GB)
SSD (200GB, 400GB) Optional HDD (160GB) SSD (200GB, 400GB)
Height 1.75” (1RU) 3.5” (2RU) 1.75” (1RU) 3.5” (2RU)
Throughput 2.5 to 20Gbps 5 to 36Gbps 60Gbps 100Gbps
Maximum Output Power 250W 470W 360W 500W
Airflow Front to back Front to back Front to back Front to back
SD-WAN w/ Security
LTE Advanced
Cisco ISR 4000 Series Router Unified Communications
Compute UCS-E
10G WAN & Macsec
ISR 4351 ISR 4461

Up to 2 Gbps* Up to 10 Gbps*
ISR 4331 ISR 4451

Up to 2 Gbps* Up to 4 Gbps*
ISR 4221 ISR 4321 ISR 4431

Up to 1.2 Gbps* Up to 1.5 Gbps* Up to 4 Gbps*
*CEF IMIX on IOS-XE
Unified Communications Proactive Security Application Assurance Virtualization

Voice survivability & High quality Video Branch Threat Defense App. Optimization App hosting & VNFs
SD-WAN DSL [G.FAST/35b]
SD-WAN Security Wifi [ME]
Cisco ISR 1000 Series LTE Advanced PoE/PoE+
ISR 1111X-8P
Up to 350 Mbps*
ISR 1109-4P ISR 1101-4P ISR 111x-8P

Up to 200 Mbps* Up to 250 Mbps* Up to 350 Mbps*
ISR 111x-4P
ISR 1109-2P Up to 250 Mbps*
Up to 200 Mbps*
**
*CEF IMIX on IOS-XE
**VDSL2, ADSL2+ only
M2M (ATM/KIOSK/POS)
Extended temperature Remote Workers MSP CPE Branch in a box
& Dual Modem
SD-WAN
ENCS 5000 Series LTE Advanced

Modularity
- Enterprise Network Compute System
RAID Storage
ENCS 5100 Series

ENCS 5412
12Core(ISRv+5VNF)
ENCS 5400 Series

ENCS 5408
8Core(ISRv+3VNF)
ENCS 5104 ENCS 5406

4Core (ISRv+1VNF) 6Core(ISRv+2VNF)
Routing and Compute Virtualized Services Open for 3rd party apps NFVIS Hypervisor
XE SD-WAN Platform Support Matrix
July 2018 (16.9) Dec 2018 (16.10) March 2019 (16.11)
ISR 1K ISR 1K
 C1111-8P  C1111-4P, C1116-4P,C1117-4P
 C1111-8P LTEEA/LA,  C1111-4P LTEEA/LA
 C1117-4PLTEEA/LA,  C1116-4PLTEEA
Platfforms
ISR 4K  C1117-4PM,C1117-4PMLTEEA
 ISR 4221  C1111X-8P (8GB DRAM)
 ISR 4321/4331/4351 ISR 1K Wireless
 C1111-8PLTEEAW, C1111-8PW ISR 4K
ASR 1K
ISR 4K  ISR 4461
 ASR 1001-X/1002-X
 ASR 1001-HX/1002-HX  ISR 4431/4451 (Feb 2019)
Virtual Virtual Virtual
 ISRv-ENCS 5412  ISRv-ENCS 5104, 5406, 5408  CSR-AWS Only
T1/E1 Serial
DSL-IPoE, (Feb 2019)
 NIM-1/2/4/8MFT-T1/E1  NIM-1T/2T/4T
 NIM-VAB-A, NIM-VAB-M
DSL-PPPoE, PPPoA
 NIM-VAB-A, NIM-VAB-M
Ethernet-PPPoE
4G/LTE
Module
 NIM-LTEA-EA/LA
Ethernet-L3
 NIM-1GE-CU-SFP
 NIM-2GE-CU-SFP
 SM-X-4X1G-1X10G
 SM-X-6X1G
Ethernet-L2
 NIM-ES2-4/8
 NIM-ES2-4/8-P
vEdge and XE SD-
WAN Platforms
Architecture
vEdge 100 1x10/100/1000 RJ-45
100Mbps AES-256
throughput, with five fixed
10/100/1000 Mbps ports.
Comes in three different
flavors
GPS module
Power module
LED
Reset Button
USB-B
4x10/100/1000M RJ-45 vEdge 100b Ethernet only
vEdge 100m Ethernet and integrated

2G/3G/4G modem
vEdge 100wm Ethernet and integrated

2G/3G/4G modem + Wireless
LAN
vEdge 100 System on
a Chip
Compress DPI (HFA) Sec PowerMin
(SoC)
Authentik
/Decomp Cores Vault Management
Timers
Neuron
Search Crypto Crypto
Packet
Built on Cavium
Packet
FPA Security CN7020
2
Security
OCTEON
Application • High-Speed Interconnects
DMA Acceleration
Manager
MIPS64r3
Integer Core
cnMIPS
II
MIPS64r3
Integer Core processor
64K Icache cores 64K Icache
Malloc
Packet
32K Dcache ……… 32K Dcache
SSO Input
1x 10/100/1000M
RGMII
I/O & Low Latency
RJ-45
Co-Proc Crossbar at
QSMMII Networks Core frequency
4x 10/100/1000M
RJ-45 eMMC
Packet Coherent Hyper Access
Mini-USB Misc I/O Output L2 Cache Memory Controller DRAM
• Security accelerators
GPS • Application accelerators
vEdge – 1000
Built-in I/O:
8x1G SFP Mgmt GE
USB
Console &
mini USB
1 Gbps AES-256
throughput, with 8 ports
of fixed GE SFP SFP LEDs SD
vEdge 1000 - System on a Chip (SoC)
Timers SSO DPI
Authentik
Engine
FPA De-dup
DMA
Power (De)Comp Crypto Crypto

Optimizer Packet Packet
Security Security
Acclr. MIPS64r2 CN6130 MIPS64r2

4x1GE SFP 4xSGMII
4 Integer Core
Manager Integer Core
cnMIPS II
37K Icache 37K Icache
cores
Mgmt GE PCIe0
32K Dcache 32K Dcache
……… 2K Write Buffer
Packet 2K Write Buffer
Malloc Input
4x1GE SFP 4xSGMII
Boot/ I/O Bridge

Flash
(NOR/CF/e
Console MMC)
Mini-USB 2xUART, Packet

2xUSB2.0 Output 1MB Hyper Access
USB w/PHY Layer2 Cache Memory Controller DRAM
USB
SDXC
vEdge 2000
2x PIM: 2x10GE SFP or 8x1G SFP* Built-in I/O: 4x1G SFP USB
Mgmt GE
1 3 vEdge-2000
0 2
PIM0 PIM1
SFP LEDs SD Console and

Mini-USB
Pluggable Interface Modules (PIMs) Supported transceivers OIR components
• You can install the PIM-8x1GE-SFP in either PIM Slot 0 or PIM Slot 1. • SFP-1GE-Base-T • PIMs
• However, when it is installed in PIM Slot 0, only four ports are usable. • SFP-1GE-LX • Power Supply
• When it is installed in PIM Slot 1, all eight ports are usable. • SFP-1GE-SX • Transceivers
• There is no such restriction with the PIM-2x10GE-SFP+ • SFP-10GE-SR • Fan Tray
• SFP-10GE-LR
vEdge 2000 - System on a Chip (SoC)
DMA
FPA Application RAID/

Compress
Acceleration
Decomp
TCP
Manager XOR
SSO Boot/ Flash
(NOR/CF/eM
MC)
2xRXAUI or
PIM0
4xSGMII Power
Optimizer Mgmt GE
1xXAUI or
4xSGMII Crypto Crypto
Packet Packet
PIM1 Security Security Secure Console
1xXAUI or CN6880 Vault
4xSGMII MIPS64r2 MIPS64r2 Mini-USB
Integer Core 32 Integer Core
cnMIPS II Misc I/O USB
4x1GE SFP 4xSGMII 37K Icache 37K Icache
cores USB
Interlaken Write Back Buffer Write Back Buffer
……… HFA
SDXC
(Pattern
Matching)
Packet
Input
I/O Coproc TPM
Packet Network Network
Output
Hyper Hyper
Access
4MB Shared Access
DRAM Memory L2 Cache Memory DRAM
Controller Controller
vEdge 5000
LCD Mgmt port 2x USB

4x NIM: NIM-8-1GE-SFP, NIM-8-1GE-RJ45, NIM-4-10GE-SFP
vEdge-5000
PWR
STA
HDD
NIM0 NIM1 NIM2 NIM3
Console
NIM supported transceivers OIR components
• SFP-1GE-SX • NIMs
• SFP-1GE-LX • Power Supply
• SFP-1GE-EX • Fan Tray
• SFP-1GE-Base-T • Transceivers
• SFP-10GE-SR
• SFP-10GE-LR
vEdge 5000 – Intel based architecture
Intel Haswell-EP
DRAM
Core1 Core2 Core3 Core4 Core5 Core6 Core7
Core8 Core9 Core10 Core11 Core12 Core13 Core14
PCIe DMI PCIe PCIe PCIe PCIe
Flash NIM0 NIM1 NIM2 NIM3
TPM
PCH
Wellsburg
BIOS
20 Gbps AES-256
Mgmt GE USB Console throughput, with 4 Network
USB Interface Modules
vEdge Control Plane/Data Plane Cores Distribution
CP cores DP cores Threads/core

vEdge 100 1 1 1
vEdge 1000 1 3 1
vEdge 2000 2 30 1
vEdge 5000 1 13 2
XE SD-WAN Platforms Architecture
- ASR1000 fixed chassis, ASR1002-HX in example
Resource Pkts Buffer Resource Pkts Buffer
TCAM
DRAM DRAM DRAM DRAM
(80Mbit)
(2GB) (512MB) (2GB) (512MB)
PPEs QFP1 PPEs QFP2

PPE1 PPE2 PPE3 PPE1 PPE2 PPE3
Console Management
USB
& Aux Ethernet NVRAM
PPE4 PPE62 PPE4 PPE62
BQS BQS Boot Flash
CPU CPU Memory
Dispatcher Dispatcher
2.5 GHz Quad- I2C Chassis
Pkt Buffer Pkt Buffer core Management Bus
75Gbps 75Gbps
Interconnect
75Gbps Crypto Memory
(4GB)
150Gbps
Interface Aggregation ASIC
11Gbps 80Gbp 8Gbps 120Gbps

s
8x10 I 2C
NIM 8xGE EPA
GE Serdes Interface
Hypertransport
ASR 1000 System Resources Comparison
ASR 1001-X ASR 1002-X ASR 1001-HX ASR 1002-HX
System throughput
2.5 - 20Gbps 5 - 36Gbps 60Gbps 100Gbps
(IMIX)
QFP cores 31 62 62 124
Clock Rate 1.5 GHz 1.2 GHz 1.5 GHz 1.5 GHz
QFP Resource Mem 4GB (unified) 1GB 4GB 4GB
Packet Buffer 256MB 512MB 512MB 1GB

TCAM 10 Mb 40 Mb 40Mb 80 Mb
Intel quad-core Intel quad-core Intel quad-core Intel quad-core
Control CPU
2.0GHz 2.13GHz 2.5GHz 2.5GHz
Control CPU Memory 8GB/16GB 4GB/8GB/16GB 8GB/16GB 16GB/32GB
- ISR4400
DDR3 Control Plane Data Plane DDR3
DRAM (4 cores) (6-10 core) DRAM
PPE1 PPE2 PPE3 PPE4 PPE5

IOSd SVC SVC SVC Crypto Crypto Crypto Crypto Crypto
BQS
PPE6 PPE7 PPE8 PPE9
Crypto Crypto
IOS-XE
Crypto
1
Crypto
2
3
QFP code
• Physical processing separation as on ASR1K

• Dedicated CPU sockets for Control Plane & Data Plane
• Control Plane run by IOS-XE on 4 core X86 platform
• Data Plane run by micro code on 6 or 10 cores
• Dedicated forwarding, crypto and scheduling resources
• Architecture, albeit smaller scale, identical to ASR1k
- ISR4461
DRAM DDR4 Control Plane
(4 cores)
DDR4 DRAM
Up to 32GB
4GB Fixed
PPE1 PPE2 PPE3 PPE4 PPE5 PPE6 PPE7 PPE8
IOSd SVC SVC SVC Crypto Crypto Crypto Crypto Crypto Crypto Crypto Crypto
BQS
PPE9 PPE10 PPE11 PPE12 PPE13 PPE14 PPE15
Crypto Crypto Crypto Crypto Crypto
IOS-XE
Crypto
1
Crypto
2
3
QFP code
• Same physical processing separation as on ASR1K / ISR4400

• Control Plane 4-core Intel Xeon architecture
• Control Plane DDR4 DRAM – 8GB default, up to 32GB
• Data Plane 16-core Cavium SoC architecture
• Data Plane DDR4 DRAM – 4GB Fixed
- ISR4300, ISR4200 & ISR1100
4331 / 4351
4321, 4221 & 1111
Data Plane Control Plane
DRAM (4 cores) DRAM
(4 cores) SVC/ Control Plane
IOS
Dark (2 cores)
PPE1 PPE2 IOS SVC
PPE I/O Data Plane

Crypto (2 cores)
I/O
PPE3 SVC2 SVC3
Crypto
IOS-XE
IOS-XE
• Unified architecture - Single socket CPU
• Multiple CPU cores providing the distributed control plane
• Control & Dataplane cores run by IOS-XE
• Dedicated forwarding, crypto and scheduling resources
• Same base function as ASR1K
• Service Core not yet used on Cisco 1100
- Enterprise Network Compute System
ENCS 5400 Series
VNF 1 ISRv VNF 2

(NIC aware) (NIC aware)
HW offload for
VM-VM traffic Software
switched path
X86 / NFVIS
6 SR-IOV LAN
Networks
Internal NIC CIMC Lights-out
management
(10G)
High-speed Switch
backplane
X86 CIMC
VLAN-aware NIM
MGMT MGMT
HW switch Data Path
PoE
Control Path
Cellular, T1, Dual-PHY

Dedicated management ports
DSL, LAN, GE WAN GE or LAN uplink
Life of a packet in
SD-WAN Platform
vEdge Packet Flow
– Packet Arrives
Timers SSO
Authentik
DPI  Parse the packet as it arrives
Engine
FPA De-dup
 L2 type, error checks (FCS, Frame
DMA Error, Length Mismatch, etc.
Power
Optimizer
(De)Comp Crypto
Security
Packet
Crypto
Security
Packet  IPv4 and IPv6 checks (checksum,
4xSGMII Acclr.
Manager
MIPS64r2
Integer Core
CN6130
4
cnMIPS II
MIPS64r2
Integer Core malformed, TTL, options etc.)
 L4 (TCP/UDP) checks (checksum,
PCIe0 cores
………
length, TCP flags, bad port)
Packet 2K Write Buffer 2K Write Buffer
Malloc Input
4xSGMII
I/O Bridge
 Ingress classification based on
diffserv on a per-port basis and
Boot/
Flash
(NOR/CF/e
MMC)
2xUART,
2xUSB2.0
Packet
Output 1MB Hyper Access
Memory Controller
can be dynamically remapped.
Layer2 Cache
 Create up to 7-tuple hash tag
w/PHY
vEdge Packet Flow
– Packet Ready to be processed
Timers SSO
Authentik
DPI
Engine  Receive and buffer the packet
 Write all the packet into the buffers (L2
FPA De-dup
cache and/or DRAM)

DMA
 Allocate a buffer and link the buffer

Security Security
CN6130 MIPS64r2
and align the packet data.
4xSGMII Acclr. MIPS64r2
Manager Integer Core 4 Integer Core
cnMIPS II
cores
 Performs “add_work” operation to add

PCIe0 32K Dcache
32K Dcache
Malloc
the pointer to the input work queue

Input
4xSGMII
Boot/
Flash
I/O Bridge maintained by the SSO unit
(NOR/CF/e
MMC)  Based on QoS priority, ingress order
and current locks for the flow.
2xUART, Packet
w/PHY Layer2 Cache Memory Controller
vEdge Packet Flow
– cnMIPS core gets work
 cnMIPS core “asks” for a piece of work
Timers SSO DPI
De-dup
Authentik
Engine  SSO return the pointer to the packet data
buffer.
FPA
DMA
 The core processes the packet data
based on Feature Invocation Array (FIA),
Security Security
4xSGMII Acclr.
Manager
MIPS64r2
Integer Core
CN6130
4
MIPS64r2
Integer Core reading and writing the packet data in
cnMIPS II
PCIe0
37K Icache
32K Dcache
cores
………
37K Icache
32K Dcache L2/DRAM.
Malloc
Packet
Input
2K Write Buffer 2K Write Buffer
 After processing the packet data, the
4xSGMII core sends the packet data buffer pointer
Boot/
Flash
I/O Bridge
and the data offset to the appropriate
packet output queue in the Packet Output
(NOR/CF/e
MMC)
Packet
unit.
2xUART,
 The core is freed to do other “work”
vEdge Packet Flow
– cnMIPS core sends packets
Timers SSO
Authentik
DPI
Engine  The Packet Output unit is responsible
for packet transmission.
FPA De-dup
DMA
 Packet queued for output on specific
Power (De)Comp Crypto
port, each port has multiple prioritized
Crypto
Security Security
CN6130
queues
4xSGMII Acclr. MIPS64r2 MIPS64r2
Manager Integer Core 4 Integer Core
cnMIPS II
cores
 Sends the packet data from internal
PCIe0 32K Dcache
32K Dcache
Malloc
memory to the output port over the bus.
Input
 Packet buffer memory can automatically

4xSGMII
Boot/ I/O Bridge
be returned to the free pool after

Flash
(NOR/CF/e
MMC)
transmission completion.
2xUART, Packet
XE SD-WAN Packet Flow
- ASR1001-X WAN-to-LAN direction in example
• Interconnect receives IPsec packet from the
transport VPN and writes it to packet memory
TCAM • The dispatcher assigns it to one of PPE threads.
(10Mbit) QFP
PPE1 PPE2 PPE3 • The PPE processes the packet according to SD-
WAN FIA.
Resource
DRAM
(4GB) PPE4 PPE31 • The PPE recognizes the packet needs crypto
Packet
BQS
assist and modifies the internal packet header
Buffer
DRAM
accordingly (SA lookup).
(512MB)
• The packet is written to a specific re-cycle queue.
Dispatcher
• Packet is de-queued from the BQS chip and

Pkt Buffer
Crypto
(Nitrox-II passed to the Crypto Engine
CN6645
10 Cores) Interconnect • Crypto Engine decrypts the packet and returns
the packet to the dispatcher
• Dispatcher assigns the returned packet to one of
TenGE0
the PPE threads.
GE0 GE2 GE4
TenGE1
Data GE1 GE3 GE5
• PPEs continue to work on packet according to the
FIA, and send out to service VPN.
XE SD-WAN Packet Flow
- ISR4331 LAN-to-WAN direction in example
• Ethernet LAN Switch NIM receives the data packet
Data Plane • Through Multigigabit Fabric the packet reach Data
(4 cores)
Plane, and written into the Global Packet Memory
PPE1 PPE2 (GPM).
FPGE
DRAM FPGE • Go through distributor, the packet is assigned to a PPE
FPGE
PPE3
I/O thread.
Crypto
• The PPE thread processes the packet according to SD-

WAN FIA, during which it will be ready to encryption the
packet.
Multigigabit • The packet is then copied out of GPM and sent to
Fabric packet buffer. BQS will pick up the packet from packet
buffer and send it to crypto thread for encryption.
• Once this is done, packets will be re-injected back into
NIM
the distributor and back to PPE thread. Packet will
continue with other feature processing, and send to
BQS thread and out to the WAN interface (Front Panel
GE).
vEdge and XE SD-WAN
Software Architecture
vEdge Software Architecture
• vDaemon: SDWAN Software Process SDWAN
• Confd: Configuration Process Policy & FIB FTM FPM
• Sysmgr: System Manager Process

RTM
• Chmgr: Chassis Manager Process
Protocols
• TTM: Tunnel Table Manager TTM OMP
• OMP: Overlay Management Protocol

System Infra Sysmgr Chmgr
• RTM: Route Table Manager
Config
• FPM: Forwarding Policy Manager Management vDaemon
mgr
• FTM: Forwarding Table Manager
XE SD-WAN Software Architecture
I/O FP Subsystem RP Subsystem
Subsystem
IOSd SDWAN
CMAN-
CMAN-CC FP
Confd
Configmgr
FMAN-FP SSH DHCP NBAR NTP
NGIO Client
Driver CMAN-RP OSPF BGP NAT OMP Agent OMP vDaemon
QFP
FPGE/FPTE
Ucode SDWAN
FMAN-RP VRF IDB RIB TTM Sysmgr
HQF Subsystem
DPDK FTM
FIB FPM
Polaris Kernel
Life of a Packet (FIAs): From LAN to WAN
Lookup
SDWAN App-
IP Dest Data Process Goto
Interface NBAR Route
Lookup Policy & OCE Output
ACL Policy
Walk
IPSEC
Label Tunnel Pre- Encrypt Layer 2
TX
Add Encap Route (Transport Encap
mode)
Color Coding: LAN Interface Tunnel Interface WAN Interface
Life of a Packets (FIAs): From WAN to LAN
SDWAN SDWAN
IP Dest SDWAN IPSEC Goto
WAN interface
lookup For-us Decrypt Output
Filter ACL
VPN VPN IP Dst App-

Data Goto
Label transition lookup in route
Policy Output
Lookup to IP vrf Policy
L2
TX
Encap
Color Coding: LAN Interface Tunnel Interface WAN Interface
SD-WAN QoS
Data Policy
5 tuples or DPI matching
- Policing
- Classify into queue
- Rewrite inner DSCP
Port Shaping
DPI Classification Port/VLAN Policing
Ingress Egress
Port/VLAN Policing DSCP re-write rules
Access-Control List Access-Control List Queueing

5 tuples matching 5 tuples matching
- Rewrite inner DSCP - Rewrite inner DSCP
Policing
Rate
Tokens
Token Bucket
• Single Rate Policer
- Forward traffic conforming to policer rate
- Drop traffic exceeding policer rate
Configurable burst rate
Ingress Interface
Egress Interface
-
• Ingress and Egress* Policing

- Interface/VLAN based (vEdge only)
- Access list classification
- Flow policing, match on 5-tuple
Classification Policing Queuing - Data Policy classification (ingress only)
o Flow policing, match on 5-tuple
o Application policing, match on DPI
Marking/Re-marking
• Classification
- Flow, match on 5-tuple (ACL, Data Policy)
- Application, match on DPI (Data Policy)
Ingress Interface
Egress Interface
• Ingress interface marks/remarks inner
DSCP bits
- Copied to encapsulation DSCP bits
• Egress marks/remarks outer encapsulation
DSCP bits
Classification Marking, - Inner DSCP bits not modified
Re-marking
- Transport network QoS
Shaping
Rate
Tokens
Token Bucket • Shaper
- Forward shaper rate conforming
traffic
Ingress Interface
Egress Interface
- Queue shaper rate exceeding
traffic
o Weighted Round-Robin
• Egress-only Shaping*
- Interface based
Shaping Queuing
Queuing
• Classification
- Flow, match on 5-tuple (ACL, Data Policy)
- Application, match on DPI (Data Policy)
• Per-Egress Interface Queuing

Q0 - Q0 is LLQ, strict policer on vEdge, conditional policer on
Ingress Interface
Egress Interface
Q1 XE SD-WAN
Q2 - control traffic (DTLS/TLS, BFD) goes into Q0
o Not subjected to LLQ policer drop (vEdge)
Q7 o DTLS/TLS subjected to LLQ policer drop, BFD not
subjected to LLQ policer drop (XE SD-WAN)
• Scheduling for Q1-Q7 is WRR*
Classification Queuing • Drop is RED** or tail drop

* Weighted Round-Robin - RED drop profiles are linear, i.e. X% queue depth results
** Random Early Discard
in X% drop probability (vEdge)
- WRED is done per precedence by default per queue (XE
SD-WAN)
vEdge vs. XE SD-WAN QoS (1)
vEdge XE SD-WAN
policy data-policy p1 vpn-list v1 policy data-policy p1 vpn-list v1
sequence 10 sequence 10
Data-Policy match dscp 1 match dscp 1
action accept set forwarding-class c1 action accept set forwarding-class c1
policy policy
class-map class-map
class c0 queue 0 class c0 queue 0
! THE CLASSES IN IOS CLASSMAP CORRESPONDS TO
Class QUEUES AND NOT SDWAN CLASS
!
class-map queue0
match qos-group 0
vpn vpn-id policy-map shape_interface-name
interface interface-name class class-default
qos-map map1 shape average rate-in-bps
rewrite-rule rewrite1 service-policy map1
Interface
shaping-rate rate-in-kbps sdwan
interface interface-name
rewrite-rule rewrite1
service-policy output shape_interface-name
vEdge vs. XE SD-WAN QoS (2)
vEdge XE SD-WAN
policy ! qos scheduler configuration translated to cisco policy-map
qos-scheduler qos0 policy-map map1
class queue0 class queue0
scheduling llq priority percent 20
bandwidth percent 20 class queue1
buffer-percent 20 bandwidth remaining percent 10
qos-scheduler qos1 random-detect
class queue1 class queue3
bandwidth-percent 10 bandwidth remaining percent 50
QoS buffer-percent 10 random-detect
Scheduler drops red-drop class class-default
qos-scheduler qos2 //Replace queue2 with class-default to match vEdge behavior
class queue2 bandwidth remaining percent 20
bandwidth-percent 20 random-detect
buffer-percent 20
drops red-drop
qos-scheduler qos3
class queue3
bandwidth-percent 50
buffer-percent 20
SD-WAN Data Plane Security
IPsec Connection
2048-Bit RSA Encryption and ESP
(Authentication)
AES-256 (Encryption)
ESP, HMAC-SHA1,
and Anti-Replay
(Integrity)
Data Plane Encryption
• Data plane encryption and key generation OMP Route Packet OMP Route
are done by AES-256 System IP Address Packet
vSmart
• Same key to encrypt outgoing packets Color
and to decrypt incoming packets. Encapsulation
• Each router periodically generates an AES Site ID

key for its data path (specifically, one key AES Key
per TLOC) and transmits this key to the …
vSmart
• A simple and scalable key exchange
process that does not use per-pair keys
• The liveness of SAs between router peers
is tracked by monitoring BFD packets
Data Plane Integrity
AH + ESP/UDP Encapsulation
• ESP: protects the inner header, data Added

Outer Header
By UDP Header
packet payload, and ESP trailer in all AH+ ESP Header | Sequence | IV
Authenticated
data packets. ESP/
UDP
SPI | Number |
MPLS Label | EXP | S | TTL
• AH: protects the entire data packet, Original Inner Header: IP Header
Encrypted
including the inner and outer headers, packet Inner Header: TCP Header
data packet payload, and ESP trailer. Payload Data

ESP Trailer | ESP Trailer | ESP Next
• Anti-replay: number all data packets Added
Padding | Padding | Header
| Length |
and to ensure that receiving routers By
AH+ AH Next | AH |Reserved
Header | Payload |
accept only packets with unique ESP/
UDP | Length |
numbers. Security Parameters Index (SPI)
Sequence Number
• Carry VPN information in data packets ICV Checksum
App Classification Engine
• vEdge – Qosmos • XE SDWAN – NBAR2/SD-AVC
• Methods: • Network Based Application Recognition
Engine, not just DPI, but combination of
 DNS Caching other optimization techniques:
 Explicit
DNS Snooping
 Pattern matching
Statistical Classification
 Port-based classification over SSL (Machine Learning)
 Protocol Data Signature

Socket Cache
 Session Behavior
 Session Correlation Service Discovery
 Statistical Protocol Identification

Custom App and DNS-AS
SD-AVC
• A network service which ensures Application
recognition for visibility, Analytics and application vManage
SD-AVC Containers
based policy solutions.
• Analytics processing at a network level vManage
• Synchronizing application state between network
nodes, solve asymmetric routing issue SD-AVC Dispatcher
• First packet classification using dynamic L3/L4

classification rules, required for DCA
• Serves as a gateway for external sources: O365, Application SD-AVC
DNS-AS, CUCM, MS Lync, OpenDNS, Rules
Pack Update
Sensor Data
(JSON UDP)
• Auto-learning and auto-signature algorithms

• Provides pack update capability at a network level for
NBAR2
thousands of devices Agent
XE SDWAN Consumer XE SDWAN Sensor
Migration from classic
IOS XE to XE SD-WAN
Migrate using bootstrap configuration generated
from vManage – step 1
Step-1: Import the viptela signed serial list file

under WAN Edge list and upload the vedge serial
Pre-requisites:
list, validate the device and send it to the
controllers. vManage version: 18.4.0 and above
XE-SDWAN version: 16.10.1 and above
Migration step -2 Create a device template and attach the template
to the ISR/ASR device which was imported and
validated from step1.
Device to
be attached
Migration step - 3 There will be a pop-up window, which asks for the details
to fill in before attaching the template to the device
Provide the device details

in respective fields
Update once
details are filled in
next screen would be preview

of the config and attach that
template to the device.
Migration step - 4
Push feature template task will be shown as "Done-
Scheduled"
since the device is still not online yet.
And we notice the process in the background.
Background process
Migration step – 5 & 6
Shows device is in
sync pending state
Navigate to Devices tab and confirm the device is in

vManage mode with assigned template.
Here, click on the 3 dots located on the extreme right
side to choose the "Generate Bootstap Configuration".
prompted with 2 options to choose here,

select cloud-init and click on "OK"
Migration step - 7
Bootstrap configuration will be generated and prompted with a popup

window. Make sure to save the file to local
Verify the configuration you are trying to bootstrap and click on the disk with name "ciscosdwan.cfg"
Download icon
Migration step – 8 & 9
Step 8: Copy the xe-sdwan image and bootstrap config

file "ciscosdwan.cfg" to routers bootflash:  br1_pop2#copy usb0:ciscosdwan.cfg bootflash:
Destination filename [ciscosdwan.cfg]?
which is running non-sdwan code (polaris). Copy in progress...C
4755 bytes copied in 0.052 secs (91442 bytes/sec)
 br1_pop2#copy usb0:isr4400-ucmk9.16.10.214.SSA.bin
bootflash:
Step 9: Check the bootvariables to make sure
the config-register is set to 0x2102 and take You save the XE
the backup of the existing running image under a
different directory
configuration from the device.
and delete it from
And make sure device boots up with XE- bootflash
SDWAN as this will be the only image available
under the bootflash
 br1_pop2#dir bootflash: | i bin
20 -rw- 587682473 Feb 19 2019
00:38:38+00:00 isr4400-ucmk9.16.10.214.SSA.bin
Migration - Final
Step 10: Write Erase the configuration and reload
the router. When the router boots with XE-SDWAN Note: Incase of ON-PREM
image, it will look for ciscosdwan.cfg file under deployments, make sure to install
bootflash: and if the file is detected then the PNP the enterprise root certificate to
process will be aborted and the router boots up bringup control-connections.
with the configuration present under this config file
show sdwan control
Verification: connections
Make sure the device forms control-connections and bfd session
properly and also from vManage 'WAN edge list' section confirm
the device is 'insync' status.
XE SD-WAN Platforms
Serviceability
Possible causes for control connection failure
Connectivity issues Certificate issues
Serial number(s) not present

DTLS Connection Failure
Certificate revoked/invalidated
TLOC Disabled
Certificate Verification Failed
Transient Conditions
Org. Name Mismatch
DTLS connection failure
Probable causes Debugging steps:
 NH not reachable PING Def-GW

 Def-GW not installed in RIB Ping vBond if ICMP is allowed on the
 DTLS port not open in the vBond
Controllers Traceroute to vBond DNS Address
2019-03-22T10:49:04-0700
2019-03-22T10:49:03-0700
Transient Conditions
Following are some Transient conditions where the control connections flap.
System-IP change on the vEdge
Tear-down msg. to vBond [control connection to vBond is transient]
This can be verified using the “show control connections” output as shown below
Disconnect vBond after register reply
System-IP Changed
2019-03-25T19:50:04+000
2019-03-25T19:49:03+000
2019-03-25T19:49:03+000
2019-03-25T19:49:03+000
2019-03-25T19:45:04+000
Serial Number(s) NOT present
Challenge response rejected

by peer
2019-03-28T11:28-0700
2019-03-28T11:23-0700
2019-03-28T11:23-0700
2019-03-28T11:23-0700
Peer Board ID Cert not

verified
2019-03-28T10:22-0700
2019-03-28T11:23-0700
Certificate installation failed
 Certification verification failure is when certificate

cannot be verified with the root cert installed.
Fail to verify Peer Certificate
2019-03-29T10:21-0400
2019-03-29T10:21-0400
Organization-name Mismatch
 For a given a overlay, the Org. Name has to match

across all the controllers and vEdges so that control
connections can come up.
 If not, you will see “Certificate Org. name mismatch”
as seen below in the “show control connections”
output.
Certificate Org name mismatch
2019-03-29T12:25-0500
2019-03-29T12:25-0500
vManage Dashboard
Control up
partial
Control down
Control Up: Total number of

devices with the required
number of operational control
plane connections to a vSmart
controller.
Partial: Total number of

devices with some, but not all,
operational control plane
connections to vSmart
controllers.
Control Down: Total number

of devices with no control plane
connection to a vSmart
controller.
System Status – Sample
Software Upgrade
Upgrade software version of the XE SD-WAN router
NOTE: If the software upgrade is NOT successful and the device loses its
connectivity after upgrade, it will automatically roll-back to the previous
Software version
Device Reboot
System Alarm - Types
Major Alarm - RED
 One or more hardware components on the router has failed.
 One or more hardware components on the router has exceeded
the temperature threshold.
Minor Alarm - YELLOW

 Indicates a warning on the router that, if left unattended,
might result in an interruption in router operation or
degradation in router performance.
Checking Alarms
Collecting Show Admin Tech
Generate Show-admin Tech
Packet Capture - vEdge
pcap file will be

downloaded
We can also do TCP dump on vEdge cli

-
Vedge# tcpdump vpn 1
More info: -
https://sdwan-docs.cisco.com/Product_Documentation/Command_Reference/Operational_Commands/tcpdump
The Packet Tracer and FIA Debugger
• In XE SD-WAN, we provide a tool known as Packet Trace which enables us
to visualize the Feature Invocation Array (FIA) to see each feature that
touches a packet and any important decision points to be recorded.
• This is a commonly-used and very useful tool for troubleshooting
forwarding issues, namely:
1. Packet drops through the device
2. Existence (or lack thereof) of data traffic
ASR1000#debug platform condition interface gig0/0/0 ingress

ASR1000#debug platform condition start
ASR1000#debug platform packet-trace enable
ASR1000#..!send traffic
ASR1000#show platform packet-trace statistics
ASR1000# show platform packet-trace packet 8
Packet Tracer Example

Packet: 8
Summary
CBUG ID: 2353
Input : GigabitEthernet0/0/0
Output : Tunnel0
State : DROP 20 (QosPolicing) Feature: FIA_TRACE
Timestamp Entry : 0x8a0062d0 - IPV4_NAT_OUTPUT_FIA
Start : 342606524252755 ns (03/20/2015 12:31:00.116449
LapsedUTC)
time: 14328 ns
Stop : 342606524298005 ns (03/20/2015 12:31:00.116494
Feature:UTC)
FIA_TRACE
Path Trace Entry : 0x8a0064bc - IPV4_OUTPUT_THREAT_DEFENSE
Feature: IPV4 Lapsed time: 124 ns
Source : 10.1.82.17 Feature: FIA_TRACE
Destination : 10.8.5.10 Entry : 0x8a0062d8 - IPV4_VFR_REFRAG
Protocol : 17 (UDP) Lapsed time: 124 ns
SrcPort : 10714 Feature: FIA_TRACE
DstPort : 514 Entry : 0x8a005be4 - IPV4_OUTPUT_L2_REWRITE
Feature: FIA_TRACE Lapsed time: 2826 ns
Entry : 0x8a0062b0 - IPV4_OUTPUT_VFR Feature: FIA_TRACE
Lapsed time: 106 ns Entry : 0x8a006980 - OUTPUT_DROP
Feature: FIA_TRACE Lapsed time: 71 ns
Entry : 0x8a005fe4 - MC_OUTPUT_GEN_RECYCLE Feature: FIA_TRACE
Lapsed time: 302 ns Entry : 0x8a006524 - IPV4_OUTPUT_QOS
Feature: NAT Lapsed time: 18773 ns
Direction : IN to OUT Packet Copy Out
Action : Translate Source 45000000 00000000 ff2fca7a c0a83801 c0a83802 00000800
Old Address : 10.1.82.17 10714 4598001c 00000000 3b116425 0a020201 0a08050a 11950202
New Address : 10.2.2.1 04501 0008d132
Conclusion
Key Takeaways
• Cisco has the most comprehensive SD-WAN portfolios to meet
your needs
• Native vEdge
• ASR1k with built-in SD-WAN acceleration ASIC (QFP) in Hub/DC/Colo
• The industry leading branch routers ISR4k/ISR1k/ENCS with rich services.
• Easy migration from XE to SD-WAN

• Operational simplicity
References
• BRKARC-2001 - Cisco ASR1000 Routers: Architectural Overview and Use
Cases
• BRKARC-3001 - Cisco Integrated Services Router - Architectural Overview
and Use Cases
• BRKRST-3404 - How to choose the correct Branch device
• BRKARC-2012 - Enterprise Network Functions Virtualization (ENFV)
Architecture, Configuration and Troubleshooting
• BRKARC-1005 - Cisco ENCS 5400,5100 - Architecture
• BRKARC-3147 - Advanced troubleshooting of the ASR1K and ISR (IOS-
XE) made easy
Complete your
online session • Please complete your session survey
evaluation after each session. Your feedback
is very important.
• Complete a minimum of 4 session
surveys and the Overall Conference
survey (starting on Thursday) to
receive your Cisco Live water bottle.
• All surveys can be taken in the Cisco Live
Mobile App or by logging in to the Session
Catalog on ciscolive.cisco.com/us.
Cisco Live sessions will be available for viewing
on demand after the event at ciscolive.cisco.com.
Continue your education
Demos in the
Walk-in labs
Cisco campus
Meet the engineer

Related sessions
1:1 meetings
Thank you
#CLUS
#CLUS

BRKARC-1006-SD-WAN Vedge and XE SD-WAN Platform Architecture Overview PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

BRKARC-1006-SD-WAN Vedge and XE SD-WAN Platform Architecture Overview PDF

Uploaded by

Copyright:

Available Formats

#CLUS

SD-WAN vEdge and

Webex Teams will be moderated cs.co/ciscolivebot# BRKARC-1006

Control Plane 3rd Party

What are they?

• Service chaining virtual • Hardware and software redundancy

Virtual and Cloud

I/O 5x 10/100/1000 Mbps 8x 1G SFP 4x 1G SFP, 2x PIM* 4x NIM*

External USB vEdge 100: N/A 2x USB3.0 2x USB3.0 2x USB3.0

PoE vEdge 100: N/A N/A N/A N/A

4G LTE vEdge 100: N/A N/A N/A N/A

Console Port Mini-USB Mini-USB, RJ-45 Mini-USB, RJ-45 Mini-USB, RJ-45

Mgmt port N/A RJ-45 (10/100/1000) RJ-45 (10/100/1000) RJ-45 (10/100/1000)

HxWxD 1.72x7.5x5.5 in. 1.75x7.5x10 in. 1.75x17.25x18.5 in. 1.75x17.25x22.8 in.

Flash 4GB 8GB 8GB 120GB

*CEF IMIX on IOS-XE

Fixed Chassis IOS-XE Modular Chassis

Compact, Power Router Business Critical Resiliency Instant On Service Delivery

SPA Slots 1 3 N/A N/A

EPA Slots N/A N/A N/A 1

NIM Slots 1 N/A N/A 1

Built-In TenGE 2 N/A 4 + 4 (configurable 10GE/GE) 8

CPU 2.0GHz quad-core 2.13GHz quad-core 2.5GHz quad-core 2.5GHz quad-core

Memory 8GB/16GB 4GB/ 8GB/16GB 8GB/16GB 16GB/32GB

eUSB(8GB) eUSB(8GB) eUSB(32GB)

Height 1.75” (1RU) 3.5” (2RU) 1.75” (1RU) 3.5” (2RU)

Throughput 2.5 to 20Gbps 5 to 36Gbps 60Gbps 100Gbps

Maximum Output Power 250W 470W 360W 500W

Airflow Front to back Front to back Front to back Front to back

ISR 4351 ISR 4461

ISR 4331 ISR 4451

ISR 4221 ISR 4321 ISR 4431

*CEF IMIX on IOS-XE

Unified Communications Proactive Security Application Assurance Virtualization

Cisco ISR 1000 Series LTE Advanced PoE/PoE+

ISR 1109-4P ISR 1101-4P ISR 111x-8P

ENCS 5000 Series LTE Advanced

ENCS 5100 Series

ENCS 5400 Series

ENCS 5104 ENCS 5406

4x10/100/1000M RJ-45 vEdge 100b Ethernet only

vEdge 100m Ethernet and integrated

vEdge 100wm Ethernet and integrated

Power (De)Comp Crypto Crypto

Acclr. MIPS64r2 CN6130 MIPS64r2

4x1GE SFP 4xSGMII

Boot/ I/O Bridge

Mini-USB 2xUART, Packet

SFP LEDs SD Console and

FPA Application RAID/

LCD Mgmt port 2x USB

NIM0 NIM1 NIM2 NIM3

Core8 Core9 Core10 Core11 Core12 Core13 Core14

PCIe DMI PCIe PCIe PCIe PCIe

Flash NIM0 NIM1 NIM2 NIM3

CP cores DP cores Threads/core

PPEs QFP1 PPEs QFP2

BQS BQS Boot Flash

CPU CPU Memory

11Gbps 80Gbp 8Gbps 120Gbps

Packet Buffer 256MB 512MB 512MB 1GB

PPE1 PPE2 PPE3 PPE4 PPE5

• Physical processing separation as on ASR1K