BRKDCN 3378

Building Data Centre
Networks with VXLAN

BPG-EVPN
Lukas Krattiger
Principal Engineer @CCIE21921
BRKDCN-3378
Cisco Spark
Questions?
Use Cisco Spark to communicate
with the speaker after the session
How
1. Find this session in the Cisco Live Mobile App
2. Click “Join the Discussion”
3. Install Spark or go directly to the space
4. Enter messages/questions in the space
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Session Objective
• A short Overview on Overlays
• Standards and Implementation on VXLAN BGP EVPN
• A walk-thru on Control- & Data-Plane
• Details around Tenant Routed Multicast (TRM)
• Overview and Details around EVPN Multi-Site
• VXLAN OAM – Operation, Administration and Management
BRKDCN-3378 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
Agenda
• Introduction to Overlays
• VXLAN with BGP EVPN
• Standards and Implementation
• Control & Data Plane
• Tenant Routed Multicast (TRM)
• Multi-Site
• VXLAN OAM
Introduction to Overlays
Overlay Taxonomy - Underlay
Layer-3
Interface Spine Spine Spine Spine
Peering
Underlay
Edge Device Leaf Leaf Leaf Leaf Leaf Leaf Leaf
LAN
Segment
Hypervisor Baremetal Hypervisor Hypervisor Baremetal Hypervisor Baremetal Baremetal
Virtual
Server Physical
Server
Overlay Taxonomy - Overlay
Tunnel Encapsulation
Spine
(VNI Namespace)
Spine Spine Spine
Overlay
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP
LAN
Segment
Hypervisor Baremetal Hypervisor Hypervisor Baremetal Hypervisor Baremetal Baremetal
Virtual
Server Physical VTEP: VXLAN Tunnel End-Point
Server VNI/VNID: VXLAN Network Identifier
Understanding Overlay Technologies
Overlay Services
• Layer-2 Underlay Transport
Tunnel Encapsulation
• Layer-3 Network
• Layer-2 and Layer-3
Data-Plane
Control-Plane • Overlay Layer-2/Layer-3 Unicast Traffic
• Peer-Discovery • Overlay Broadcast, Unknown Unicast,
• Route Learning and Distribution Multicast traffic (BUM traffic)
• Local Learning forwarding
• Remote Learning • Ingress Replication (Unicast)
• Multicast
Agenda
• Multi-Site
• VXLAN OAM
Standards and Implementation
What is … ?
• VXLAN • EVPN
• Standards based Encapsulation • Standards based Control-Plane
• RFC 7348 • RFC 7432
• Uses UDP-Encapsulation • Uses Multiprotocol BGP
• Transport Independent • Uses Various Data-Planes

• Layer-3 Transport (Underlay) • VXLAN (EVPN-Overlay), MPLS,
Provider Backbone (PBB)
• Flexible Namespace
• 24-bit field (VNID) provides ~16M • Many Use-Cases Covered
unique identifier • Bridging, MAC Mobility, First-Hop &
• Allows Segmentations Prefix Routing, Multi-Tenancy (VPN)
Introducing Ethernet VPN (EVPN)
EVPN MP-BGP – RFC 7432
MPLS Provider Backbone Bridges Overlay (NVO3)
(draft-ietf-l2vpn-evpn) (draft-ietf-l2vpn-pbb-evpn) (draft-ietf-bess-evpn-overlay)
VXLAN and EVPN related RFCs & Drafts (IETF)
ID Title Category
RFC 7348 Virtual Extensible Local Area Network Data Plane
RFC 7432 BGP MPLS based Ethernet VPNs Control Plane
draft-ietf-bess-evpn-overlay A Network Virtualisation Overlay Solution using EVPN Control Plane
draft-ietf-bess-evpn-inter-subnet-forwarding Integrated Routing and Bridging in EVPN Control Plane
draft-ietf-bess-l2vpn-evpn-prefix-advertisement IP Prefix Advertisement in E-VPN Control Plane
draft-tissa-nvo3-oam-fm NVO3 Fault Management / OAM Management Plane
Different Type of EVPN Use-Cases
Use-
Case
Layer-2 &
Layer-2
Layer-3
Asymmetric IRB Symmetric IRB

(VLAN-Aware) (VLAN-Based)
VRF to VRF
ID Title Category
Integrated Routing and Bridging in EVPN
• Symmetric Inter-Subnet
Forwarding
• Bridge->Route/Route->Bridge
• Symmetric VNI in both
directions
• Adjacency contains Remote
VTEP,VRF
• Optimal for Scale
• Flexible Configuration
VTEP = VXLAN Tunnel End-Point

VRF = Virtual Routing and Forwarding
VNI = VXLAN Network Identifier
ID Title Category
EVPN Layer-2 Service Interface
• Single Subnet per EVI

• VLAN-based
• Per EVI BGP Route Distinguisher / Router Target per EVI / VNI
• BGP Route-Target constrain mechanism to limit propagation (import/export)
• 1:1 mapping
• EVI to Single Broadcast Domain
(Bridge Domain)
• Ethernet Tag ID must be 0
VID = VLAN ID
EVI = EVPN Virtual Instance BRKDCN-3378 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
ID Title Category
IP-VRF-to-IP-VRF Model in EVPN
• Interface-Less Model
• Route-Type 5 only
• Next-Hop is remote VTEP
• Two extended communities
• Encapsulation Extended
Community
• Router’s MAC Address (remote
VTEP)
Route Type 2 = MAC/IP Route

Route Type 5 = IP Prefix Route
Control- & Data-Plane
Host Advertisements
Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq.
2 0000.3001.1101 / 48 3001, 65500:3001 10.200.200.101

Spine Spine Spine Spine
2 0000.3001.1102 / 48 3001, 65500:3001 10.200.200.104

• Host MAC (Route Type 2)
2 0000.3002.2101 / 48
Overlay 3002, 65500:3002 • MAC
10.200.200.107
• MPLS Label1 (L2VNI*)

Leaf
101010110101
01010101010
Leaf Leaf Leaf
101010110101
01010101010
Leaf Leaf Leaf
101010110101
01010101010
• Route Target for MAC-VRF
• MAC attributes are Mandatory
Baremetal Baremetal Baremetal
Host A Host B Host C

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
*L2VNI: VNI for all Bridging operation (”VLAN-VNI”) BRKDCN-3378 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
Ethernet Tag
Ethernet Identifier
V2# show bgp l2vpn evpn 0000.3001.1101 (Ethtag)
Segment
Identifier (ESI) MAC Address MAC
Route Type: Length Address
BGP routing table information
MAC/IP for VRF default, address family L2VPN EVPN
Route Distinguisher: 10.10.10.101:32777
BGP routing table entry for [2]:[0]:[0]:[48]:[0000.3001.1101]:[0]:[0.0.0.0]/216,
version 4
Paths: (1 available, best #1)
Flags: (0x000202) on xmit-list, is not in l2rib/evpn, is locked
Advertised path-id 1
Next-Hop
IP Address
Path type: internal, path
L2VNIis valid, is best path, no labeled nexthop
AS-Path: NONE, path(MPLS
sourced internal to AS
Label1)
10.200.200.101 (metric 3) fromL2VNI
10.10.10.201 (10.10.10.201)
Encap:8
Route Target VXLAN
Origin IGP, MED not set, localpref 100, weight 0
Received label 3001
Extcommunity: RT:65500:3001 ENCAP:8
Originator: 10.10.10.101 Cluster list: 10.10.10.201
Host Advertisements
2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101 /32 • Host

5000, 65500:5000 MAC+IP (Route Type 2)
10.200.200.101
•
MAC and IP
2 0000.3001.1102 / 48 3001, 65500:3001 192.168.10.102 /32 5000, 65500:5000
• 10.200.200.104
MPLS Label1 (L2VNI)
2 0000.3002.2101 / 48
Overlay 3002, 65500:3002 192.168.20.101 /32 5000, 65500:5000• Route Target for MAC-VRF
10.200.200.107
• MPLS Label2 (L3VNI*)
Leaf
101010110101
Leaf Leaf Leaf
101010110101
Leaf Leaf Leaf
101010110101
• Route Target for IP-VRF
•
01010101010 01010101010 01010101010
Router MAC
• IP Attributes are Optional

Baremetal Baremetal Baremetal • Populated through ARP/ND
MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101
*L3VNI: VNI for all Routing operation (”VRF-VNI”) BRKDCN-3378 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 56
Ethernet Tag
Ethernet Identifier
V2# show bgp l2vpn evpn 0000.3001.1101 (Ethtag)
Segment
Identifier (ESI) MAC Address MAC
Route Type: Length Address
MAC/IP for VRF default, address family L2VPN EVPN
BGP routing table entry for [2]:[0]:[0]:[48]:[0000.3001.1101]:[32]:[192.168.10.101]/272,
version 4
Paths: (1 available, best #1) IP Address
Length
Flags: (0x000202) on xmit-list, is not in l2rib/evpn, is locked IP Address
Next-Hop L3VNI
IP Address
Path type: internal,L2VNI
path is (MPLS
valid, is best path, no labeled nexthop
Label2)
AS-Path: NONE, path
(MPLSsourced
Label1) internal to AS
10.200.200.101 (metric 3) from 10.10.10.201 (10.10.10.201)
Encap:8
Origin IGP, MED not set, localpref 100, weight 0 VXLAN
Received label 3001 5000
Extcommunity: RT:65500:3001 RT:65500:5000 ENCAP:8 Router MAC:0200.0ade.de01
L2VNI L3VNI
Route Target Router MAC
Route Target
Subnet Route Advertisements
Type IP / Length L3VNI / RT Next-Hop Seq.
5 192.168.10.0 /24 5000, 65500:5000 10.200.200.101

• Internal and External Subnet
Prefixes (Route Type 5)
Overlay • IP Prefix
• MPLS Label (L3VNI)
Leaf Leaf Leaf Leaf Leaf Leaf Leaf
• Route Target for IP-VRF
•
101010110101
01010101010
Router MAC
• Populated through External

Routing Protocol
Subnet A
192.168.10.0/24
Ethernet Tag
Ethernet Identifier
Segment
V2# show bgp l2vpn evpn 192.168.10.0 (Ethtag)
Identifier (ESI) IP Address
Route Type: IP Address
Length family
IP Prefix for VRF default, address L2VPN EVPN
BGP routing table entry for [5]:[0]:[0]:[24]:[192.168.10.101]/224,
version 4
Paths: (1 available, best #1)
Flags: (0x000202) on xmit-list, is not in l2rib/evpn, is locked
Next-Hop
IP Address
Path type: internal, path
L3VNIis valid, is best path, no labeled nexthop
AS-Path: NONE, path(MPLS
sourced internal to AS
Label)
10.200.200.101 (metric 3) fromL3VNI
10.10.10.201 (10.10.10.201)
Encap:8
Origin IGP, MED not set, localpref 100, weight 0 Router MAC
Route Target VXLAN
Received label 5000
Extcommunity: RT:65500:5000 ENCAP:8 Router MAC:0200.0ade.de01
Introducing VXLAN
MAC 802.1q IP Payload CRC

Src and Dst
Src, Dst VTEP VTEP IP
and Hop-by- UDP Dst VXLAN
Address Port 4789 VNI
Hop MAC Original Layer-2 Frame
Outer MAC Outer IP UDP VXLAN Inner MAC Inner IP Payload CRC
Data-Plane (VXLAN) 20-byte + 8-byte +8-byte + 14-byte* = 50 Bytes

of total overhead
UDP Src Port

Hash of L2/L3/L4
headers of original
Frame
*plus 4-byte if IEEE 802.1q exists as part of Inner MAC Header BRKDCN-3378 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
VXLAN Frame Format – MAC in IP Encapsulation
Field Value Bites Total Field Value Bites Total
Dest. MAC Address Next-Hop MAC Address 48 Source Port L2/L3/L4 Hash 16
(4 Bytes Optional)
8 Bytes
Src. MAC Address Next-Hop MAC Address 48 Destination Port 4789 (UDP) 16
14 Bytes
VLAN Type 0x8100 16 UDP Length 16
VLAN ID Tag 16 Checksum 0x0000 16
Ether Type 0x0800 16
Outer MAC Outer IP UDP VXLAN Inner MAC Payload CRC
Field Value Bites Total

Field Value Bites Total
VXLAN Flags RRRRIRRR 8
IP Header Misc. Data 72
8 Bytes
Reserved 24
Protocol 0x11 (UDP) 8
20 Bytes
VNI 16M Possible Segments 24

Header Checksum Various 16
Reserved 8
Source IP Src, VTEP IP 32
Destination IP Dest. VTEP IP 32 BRKDCN-3378 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 61
VXLAN and BGP EVPN – Putting it Together
Control-Plane (BGP EVPN)
3001 5000
2 0000.3001.1101/48 192.168.10.101/32 10.200.200.101
65500:3001 65500:5000
Dst VTEP IP L2VNI Dst MAC Dst IP

10.200.200.101 3001 0000.3001.1101 192.168.10.101
Data-Plane (VXLAN)
Bridging
Packet Walk – ARP Request
2 0000.3001.1101 / 48 3001, 65500:3001 10.200.200.101
SIP DIP VXLAN Overlay

SMAC DMAC
ARP Request for
192.168.10.102
10.200.200.101 239.0.0.1 3001 0000.3001.1101 FFFF.FFFF.FFFF
VTEP VTEP VTEP VTEP VTEP VTEP VTEP

ARP Request for 192.168.10.102
SMAC: DMAC:
ARP Request for 192.168.10.102 0000.3001.1101 FFFF.FFFF.FFFF
SMAC: DMAC:
0000.3001.1101 FFFF.FFFF.FFFF

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101
Packet Walk – ARP Response
Type
Type MAC
MAC/ Length
/ Length L2VNI
L2VNI/ RT
/ RT IPIP/ Length
/ Length L3VNI
L3VNI/ RT
/ RT Next-Hop
Next-Hop Seq.
Seq.
22 0000.3001.1101
0000.3001.1101/ 48
/ 48 3001,
3001,65500:3001
65500:3001 192.168.10.101/32 5000, 65500:5000 10.200.200.101
10.200.200.101
SIP DIP VXLAN Overlay

SMAC DMAC
ARP Response for
192.168.10.102
10.200.200.104 10.200.200.101 3001 0000.3001.1102 0000.3001.1101
ARP Response for 192.168.10.102 ARP Response for 192.168.10.102
SMAC: DMAC: SMAC: DMAC:

0000.3001.1102 0000.3001.1101 0000.3001.1102 0000.3001.1101

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101
Packet Walk – Bridging
2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101/32 5000, 65500:5000 10.200.200.101
2 0000.3001.1102 / 48 3001, 65500:3001 192.168.10.102/32 5000, 65500:5000 10.200.200.104
SIP DIP VXLAN SMAC Overlay DMAC SIP DIP

Payload
10.200.200.101 10.200.200.104 3001 0000.3001.1101 0000.3001.1102 192.168.10.101 192.168.10.102
SMAC DMAC SIP DIP
0000.3001.1101 0000.3001.1102 192.168.10.101 192.168.10.102

SMAC DMAC SIP DIP
0000.3001.1101 0000.3001.1102
Baremetal 192.168.10.101 192.168.10.102 Baremetal Baremetal

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101
Integrated Routing and Bridging in EVPN
• Symmetric Inter-Subnet • Asymmetric Inter-Subnet

Forwarding Forwarding
• Bridge->Route/Route->Bridge • Bridge->Route->Bridge
• Symmetric VNI in both • Different (Asymmetric) VNI
directions depending on directions
• Adjacency contains Remote • Adjacency contains Remote
VTEP,VRF VTEP,VRF and End-Points
• Optimal for Scale • Potential Sub-Optimal for Scale
• Flexible Configuration • Consistent Configuration
VTEP = VXLAN Tunnel End-Point

VRF = Virtual Routing and Forwarding
Operational Models for Asymmetric Inter-Subnet
Forwarding
(draft-ietf-bess-evpn-inter-subnet-forwarding – Section 4)
• Asymmetric IRB
VNI 40000 (L2VNI)
VNI 30000 (L2VNI)

V1 V2
MAC IP IP MAC
192.168.22.33 192.168.33.44
Bridge -> Route -> Bridge
Forwarding
ARP and Adjacency Table
• Asymmetric IRB
VNI 40000 (L2VNI)
VNI 30000 (L2VNI)

V1 V2
MA MA
MAC IP IP MAC
C C
a.a.a b.b.b
192.168.22.33 192.168.33.44
ARP table V1 ARP table V2

a.a.a, 192.168.22.33, VLAN 30 b.b.b, 192.168.33.44, VLAN 40
b.b.b, 192.168.33.44, VNI 40000 a.a.a, 192.168.22.33, VNI 30000
Forwarding
Routing Table
• Asymmetric IRB
VNI 40000 (L2VNI)
VNI 30000 (L2VNI)

V1 V2
MA MA
MAC IP IP MAC
C C
192.168.22.33
a.a.a 192.168.33.44
b.b.b
192.168.22.33 192.168.33.44
Routing table V1 Routing table V2

192.168.22.33, local, VLAN 30 192.168.33.44, local, VLAN 40
192.168.33.44, local, VNI 40000 192.168.22.33, local, VNI 30000
Forwarding
(draft-ietf-bess-evpn-inter-subnet-forwarding – Section 4.1)
• Asymmetric IRB
VNI 40000 (L2VNI)
VNI 30000 (L2VNI)

V1 V2
MA MA
MAC IP IP MAC
C C
192.168.22.33 192.168.33.44
[2]:[0]:[0]:[48]:[0050.569f.d495]:[32]:[192.168.33.44]
BGP 10.22.22.34 (Next-Hop)
Update L2VNI: 30000
Packet Walk – Asymmetric IRB (A to C)
2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101/32 10.200.200.101
2 0000.3002.2102 / 48 3002, 65500:3002 192.168.20.101/32 10.200.200.107

Payload
10.200.200.101 10.200.200.107 3002 2020.0000.AAAA 0000.3002.2101 192.168.10.101 192.168.20.101
SMAC DMAC SIP DIP
2020.0000.AAAA 0000.3002.2101 192.168.10.101 192.168.20.101
SMAC DMAC SIP DIP
0000.3001.1101 2020.0000.AAAA

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101
Packet Walk – Asymmetric IRB (C to A)
2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101/32 10.200.200.101
2 0000.3002.2102 / 48 3002, 65500:3002 192.168.20.101/32 10.200.200.107

Payload
10.200.200.107 10.200.200.101 3001 0000.3002.2101 2020.0000.AAAA 192.168.20.101 192.168.10.101
SMAC DMAC SIP DIP
2020.0000.AAAA 0000.3001.1101 192.168.20.101 192.168.10.101
SMAC DMAC SIP DIP
0000.3002.2101 2020.0000.AAAA 192.168.20.101 192.168.10.101


MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101
Operational Models for Symmetric Inter-Subnet
Forwarding
(draft-ietf-bess-evpn-inter-subnet-forwarding – Section 5)
• Symmetric IRB
VNI 50000 (L3VNI)
V1 V2
MAC IP IP MAC
192.168.22.33 192.168.33.44
Bridge -> Route -> Route -> Bridge
Forwarding
ARP and Adjacency Table
• Symmetric IRB
VNI 50000 (L3VNI)
V1 V2
MAC IP IP MAC
a.a.a b.b.b
192.168.22.33 192.168.33.44
ARP table V1 ARP table V2

a.a.a, 192.168.22.33, VLAN 30 b.b.b, 192.168.33.44, VLAN 40
Forwarding
Routing Table
• Symmetric IRB
VNI 50000 (L3VNI)
V1 V2
MAC IP IP MAC
a.a.a b.b.b
192.168.22.33 192.168.33.44
Routing table V1 Routing table V2

192.168.22.33, V2, VNI 50000 192.168.33.44, V1, VNI 50000
Forwarding
(draft-ietf-bess-evpn-inter-subnet-forwarding – Section 5.1.1)
• Symmetric IRB
VNI 50000 (L3VNI)
V1 V2
MAC IP IP MAC
192.168.22.33 192.168.33.44
[2]:[0]:[0]:[48]:[0050.569f.d495]:[32]:[192.168.33.44]
BGP 10.22.22.34 (Next-Hop)
Update L2VNI: 30000
L3VNI: 50000
VXLAN and BGP EVPN – Putting it Together
Extended Community
Router MAC
Control-Plane (BGP EVPN) 0200.0ade.de01
3001 5000
2 0000.3001.1101/48 192.168.10.101/32 10.200.200.101
65500:3001 65500:5000
Dst VTEP IP L3VNI Router MAC Dst IP

10.200.200.101 5000 0200.0ade.de01 192.168.10.101
Data-Plane (VXLAN)
Routing
Routing and the Router MAC – Ethernet
Router MAC
SMAC DMAC SIP DIP

Payload
0200.0ade.de01 0200.0ade.de07 192.168.10.101 192.168.20.101
SMAC DMAC SIP DIP SMAC DMAC SIP DIP

Payload Payload
0000.3001.1101 2020:0000:AAAA 192.168.10.101 192.168.20.101 2020.0000AAAA 0000.3002.2101 192.168.10.101 192.168.20.101
Switch Switch
SVI10 SVI20
192.168.10.1 192.168.20.1
interface: Eth2/1 interface: Eth2/1

MAC: 0200.0ade.de01 MAC: 0200.0ade.de07
Baremetal IP: 10.200.200.1 IP: 10.200.200.7 Baremetal
Host A Host C
MAC: 0000.3001.1101 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.20.101
Routing and the Router MAC – VXLAN
Router MAC
SIP DIP VXLAN SMAC DMAC SIP DIP

Payload
10.200.200.1 10.200.200.7 5000 0200.0ade.de01 0200.0ade.de07 192.168.10.101 192.168.20.101
SMAC DMAC SIP DIP SMAC DMAC SIP DIP

Payload Payload
0000.3001.1101 2020:0000:AAAA 192.168.10.101 192.168.20.101 2020.0000AAAA 0000.3002.2101 192.168.10.101 192.168.20.101
SVI10 SVI20
192.168.10.1 VTEP
VXLAN VTEP 192.168.20.1
interface: NVE1 interface: NVE1

MAC: 0200.0ade.de01 MAC: 0200.0ade.de07
Baremetal IP: 10.200.200.1 IP: 10.200.200.7 Baremetal
Host A Host C
MAC: 0000.3001.1101 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.20.101
Packet Walk – Symmetric IRB (A to C)
2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101/32 5000, 65500:5000 10.200.200.101
2 0000.3002.2102 / 48 3002, 65500:3002 192.168.20.101/32 5000, 65500:5000 10.200.200.107

Payload
10.200.200.101 10.200.200.107 5000 0200.0ade.de01 0200.0ade.de07 192.168.10.101 192.168.20.101
SMAC DMAC SIP DIP
2020.0000.AAAA 0000.3002.2101 192.168.10.101 192.168.20.101
SMAC DMAC SIP DIP
0000.3001.1101 2020.0000.AAAA

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101
Packet Walk – Symmetric IRB (C to A)
2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101/32 5000, 65500:5000 10.200.200.101
2 0000.3002.2102 / 48 3002, 65500:3002 192.168.20.101/32 5000, 65500:5000 10.200.200.107

Payload
10.200.200.107 10.200.200.101 5000 0200.0ade.de07 0200.0ade.de01 192.168.20.101 192.168.10.101
SMAC DMAC SIP DIP
2020.0000.AAAA 0000.3001.1101 192.168.20.101 192.168.10.101
SMAC DMAC SIP DIP
0000.3002.2101 2020.0000.AAAA 192.168.20.101 192.168.10.101


MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101
Packet Walk – Routing
2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101/32 5000, 65500:5000 10.200.200.101
2 0000.3002.2102 / 48 3002, 65500:3002 192.168.20.101/32 5000, 65500:5000 10.200.200.107

Payload
10.200.200.101 10.200.200.107 5000 0200.0ade.de01 0200.0ade.de07 192.168.10.101 192.168.20.101
SMAC DMAC SIP DIP

Router MAC
2020.0000.AAAA 0000.3002.2101 192.168.10.101 192.168.20.101
SMAC DMAC SIP DIP
0000.3001.1101 2020.0000.AAAA

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101
Packet Walk – Routing (Silent Host)
2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101 5000, 65500:5000 10.200.200.101
5 192.168.20.0/24 5000, 65500:5000 10.200.200.105
5 192.168.20.0/24 5000, 65500:5000 10.200.200.107

Payload
10.200.200.101 10.200.200.105 5000 0200.0ade.de01 0200.0ade.de05 192.168.10.101 192.168.20.101
SMAC DMAC SIP DIP
0000.3001.1101 2020.0000.AAAA

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101
Packet Walk – Routing (Silent Host)
2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101 5000, 65500:5000 10.200.200.101
5 192.168.20.0/24 5000, 65500:5000 10.200.200.107
10.200.200.105
5 192.168.20.0/24 5000, 65500:5000 10.200.200.107
2 0000.3002.21o1 / 48 3002, 65500:3002 192.168.20.101 5000, 65500:5000 10.200.200.107
SIP DIP VXLAN SMAC DMAC Overlay
SIP DIP
Payload
10.200.200.101 10.200.200.105 5000 0200.0ade.de01 0200.0ade.de05 192.168.10.101 192.168.20.101

ARP Response for 192.168.20.101
SMAC: DMAC:
0000.3002.2101 2020.0000.AAAA
ARP Request for 192.168.20.101

SMAC DMAC SIP DIP SMAC: DMAC:
2020.0000.AAAA FFFF.FFFF.FFFF
0000.3001.1101 2020.0000.AAAA

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101
Agenda
• Multi-Site
• VXLAN OAM
Multicast Forwarding
Tenant Routed Multicast (TRM)
Same Subnet Forwarding no IGMP Snooping
Traditional Forwarding in VXLAN Overlays
Spine Spine
Site-External DCI
(IP Routing and Increased
MTU Support)
VXLAN EVPN
VTEP VTEP VTEP VTEP
Baremetal Baremetal Baremetal Baremetal Baremetal
SRC-10 RCVR-10 RCVR-11 RCVR-14

224.10.10.10 10.10.10.10 10.10.10.11 10.10.10.14
10.10.10.100
BRKDCN-3378 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Same Subnet Forwarding no IGMP Snooping
Spine Spine
Site-External •DCI
”Single Copy” in Core – Treated as BUM
MTU Support)• Same Subnet Only
• No Pruning on Local Interface or Remote VTEP
VXLAN EVPNInterface
VTEP VTEP VTEP VTEP

224.10.10.10 10.10.10.10 10.10.10.11 10.10.10.14
10.10.10.100
Same Subnet Forwarding with IGMP Snooping
Spine Spine
Site-External DCI
MTU Support)
VXLAN EVPN
VTEP VTEP VTEP VTEP

224.10.10.10 10.10.10.10 10.10.10.11 10.10.10.14
10.10.10.100
Spine Spine
• Pruning on Local Interface
VXLAN EVPN
• VXLAN is ”pruned off” if no interest Receiver exists
behind any Remote VTEP
VTEP VTEP VTEP VTEP

224.10.10.10 10.10.10.10 10.10.10.11 10.10.10.14
10.10.10.100
Spine Spine
Site-External DCI
MTU Support)
VXLAN EVPN
VTEP VTEP VTEP VTEP

224.10.10.10 10.10.10.10 10.10.10.11 10.10.10.14
10.10.10.100
Spine Spine
VXLAN EVPN
• VXLAN is NOT pruned if interest Receiver exists behind
one Remote VTEP
VTEP VTEP VTEP VTEP

224.10.10.10 10.10.10.10 10.10.10.11 10.10.10.14
10.10.10.100
Different Subnet Forwarding – Router on-a-Stick
Spine Spine
Site-External DCI
MTU Support)
VXLAN EVPN
VTEP VTEP VTEP VTEP
Baremetal Baremetal Baremetal 10.10.10.254 Baremetal Baremetal
20.20.20.254
224.10.10.10 10.10.10.10 20.20.20.20 10.10.10.14
10.10.10.100
Different Subnet Forwarding – Router on-a-Stick
Spine Spine
Multiple Copy in Core – Treated as BUM
MTU Support)• Different Subnet possible – RPF Challenges
VXLAN EVPN
• VXLAN is NOT pruned if interest Receiver exists behind
one Remote VTEP
VTEP VTEP VTEP VTEP
Baremetal Baremetal Baremetal 10.10.10.254 Baremetal Baremetal
20.20.20.254
224.10.10.10 10.10.10.10 20.20.20.20 10.10.10.14
10.10.10.100
Functional Components
https://tools.ietf.org/html/draft-sajassi-bess-evpn-mvpn-
Tenant Routed Multicast (TRM) seamless-interop
Spine Spine
Site-External DCI
MTU Support)
VXLAN EVPN
VTEP VTEP VTEP VTEP
DR DR DR DR
SRC-10 RCVR-10 RCVR-20 RCVR-30 RCVR-11

224.10.10.10 10.10.10.10 20.20.20.20 30.30.30.30 10.10.10.11
10.10.10.100
Functional Components
https://tools.ietf.org/html/draft-sajassi-bess-evpn-mvpn-
Tenant Routed Multicast (TRM) seamless-interop
Spine Spine
Site-External DCI
Underlay: MTU Support)
• PIM-based Underlay Transport (PIM ASM)
• Separate Multicast Groups from Layer-2 VNI
• Leveraging same redundant Underlay Rendezvous-
VXLAN EVPN
Point (i.e. PIM Anycast-RP) Single Packet in Core
VTEP VTEP VTEP VTEP
DR DR DR DR
Overlay:
• BGP-based Control-Plane using ngMVPN (Next-
Generation Multicast VPN)
• Using existing BGP Route-Reflector
• Rendezvous-Point-less
Baremetal Baremetal
• Efficient Single Copy in Multicast Underlay

SRC-10 RCVR-10 RCVR-20 RCVR-30 RCVR-11
224.10.10.10 10.10.10.10 20.20.20.20 • Always-Route approach
30.30.30.30 (per-VLAN config)
10.10.10.11
10.10.10.100 • Distributed Anycast Designated Router (DR)
• VPC – Virtual Port-Channel
• Integration with non-TRM VTEP
Forwarding Behaviour
Tenant Routed Multicast (TRM)
BRKDCN-3378
Same Subnet Forwarding – Local and Remote
Snooping
TRM Forwarding (Layer-2 only Mode)
Spine
Site-External DCI Spine
MTU Support)
VXLAN EVPN
VTEP VTEP VTEP VTEP

DR DR DR DR

224.10.10.10 10.10.10.10 20.20.20.20 10.10.10.14
10.10.10.100
Same Subnet Forwarding – Local and Remote
Snooping
TRM Forwarding (Layer-2 only mode)
Spine
Site-External DCI Spine
MTU Support)
VXLAN EVPN
VTEP VTEP VTEP VTEP

DR DR DR DR

224.10.10.10 10.10.10.10 20.20.20.20 10.10.10.14
10.10.10.100
Different and Same Subnet Forwarding
TRM Forwarding (Layer-3 Mode)
Spine Spine
Site-External DCI
MTU Support)
VXLAN EVPN
VTEP VTEP VTEP VTEP
DR DR DR DR

224.10.10.10 10.10.10.10 20.20.20.20 10.10.10.14
10.10.10.100
Local and Remote Forwarding
TRM Forwarding (Layer-3 Mode)
Spine Spine
Site-External DCI
TTL Decrement
MTU Support)
VXLAN EVPN
VTEP VTEP VTEP VTEP
DR DR DR DR
No TTL Decrement
(bridged) TTL Decrement
(routed)

224.10.10.10 10.10.10.10 20.20.20.20 10.10.10.14
10.10.10.100
Overlay Rendezvous Point
Spine Spine
Site-External DCI
• RP-less
MTU Support) • Distributed Anycast RP (NGMVPN-based)
• Shortest Path Tress (SPT only)
VXLAN EVPN • Requires per-Tenant Loopback, Multicast enabled
• External RP
VTEP VTEP VTEP • Centralised
VTEP RP (PIM-based)
DR DR DR • Shared Tree
DR and Shortest Path Tree (cut over)
• Requires External PIM-based RP

224.10.10.10 10.10.10.10 20.20.20.20 10.10.10.14
10.10.10.100
TRM Control- & Data-Plane
BRKDCN-3378
Underlay Multicast Tree – PIM ASM
Spine Spine
Layer-3 VNI: 50001
Default MDT: 239.1.1.2
Site-External DCI
Route-Target: 65502:50001
MTU Support)
S,G – (10.0.0.1, 239.1.1.2/32)
VRF IIF: NVE-Loopback S,G – (10.0.0.2, 239.1.1.2/32)
(Underlay)
Tenant1 VXLAN EVPNS,G
OIF: Uplink (Underlay)
IIF: NVE-Loopback – (10.0.0.3, 239.1.1.2/32)
(Underlay)
*,G – (*, 239.1.1.2/32) OIF: Uplink (Underlay)
IIF: NVE-Loopback S,G – (10.0.0.4, 239.1.1.2/32)
(Underlay)
IIF: Uplink (Underlay) OIF: Uplink (Underlay)
IIF: NVE-Loopback (Underlay)
OIF: NVE1 (Underlay) OIF: Uplink (Underlay)
VTEP VTEP VTEP VTEP
DR DR DR DR

224.10.10.10 10.10.10.10 20.20.20.20 10.10.10.14
10.10.10.100
Underlay Multicast Tree – PIM ASM
Spine Spine
Layer-3 VNI: 50001
PIM ASM required for Underlay
Route-Target: 65502:50001 • Separate Groups for BUM and MDT
MTU Support)
S,G – (10.0.0.1, 239.1.1.2/32)
VRF IIF: NVE-Loopback S,G
• Default MDT initiates on VTEP startup (*, G)
– (10.0.0.2, 239.1.1.2/32)
(Underlay)
Tenant1 VXLAN
OIF: Uplink (Underlay) EVPN • Per-VTEP
IIF: NVE-Loopback S,G – (10.0.0.3,
(Underlay) (S,G)239.1.1.2/32)
imitated on VTEP startup
*,G – (*, 239.1.1.2/32) OIF: Uplink (Underlay)
IIF: NVE-Loopback S,G – (10.0.0.4, 239.1.1.2/32)
(Underlay)
IIF: Uplink (Underlay) OIF: Uplink (Underlay)
IIF: NVE-Loopback (Underlay)
OIF: NVE1 (Underlay) OIF: Uplink (Underlay)
VTEP VTEP VTEP VTEP
DR DR DR DR

224.10.10.10 10.10.10.10 20.20.20.20 10.10.10.14
10.10.10.100
NGMVPN – Source Active Advertisement (MVPN
Type 5)
Spine Spine
Layer-3 VNI: 50001
Site-External DCI
MTU Support)
VRF
Tenant1 VXLAN EVPN
*,G – (*, 239.1.1.2/32) Source Active (NGMVPN Type 5)
IIF: Uplink (Underlay)
Originator: Leaf #1
OIF: NVE1 (Underlay)
S,G: 10.10.10.100,
VTEP VTEP 224.10.10.10
VTEP VTEP
DR DR DR DR

224.10.10.10 10.10.10.10 20.20.20.20 10.10.10.14
10.10.10.100
MRoute – Overlay Multicast Tree (Source Join)
Spine Spine
Layer-3 VNI: 50001
Site-External DCI
MTU Support)
VRF
Tenant1 VXLAN EVPN
*,G – (*, 239.1.1.2/32)
S,G – (10.10.10.100, 224.10.10.10) VTEP VTEP VTEP VTEP

S,G – (10.10.10.100, 224.10.10.10)
IIF: VLAN100 (Host-facing) DR DR DR DR IIF: VRF-L3VNI (Overlay)
OIF: None OIF: None

224.10.10.10 10.10.10.10 20.20.20.20 10.10.10.14
10.10.10.100
NGMVPN – Source Tree Join (MVPN Type 7)
Spine Spine
Layer-3 VNI: 50001
Site-External DCI
MTU Support)
VRF
Tenant1 VXLAN EVPN
*,G – (*, 239.1.1.2/32) Source Tree Join (NGMVPN Type 5)
Originator: Leaf #4
S,G: 10.10.10.100, 224.10.10.10
VTEP VTEP VTEP VTEP
DR DR DR DR

224.10.10.10 10.10.10.10 20.20.20.20 10.10.10.14
10.10.10.100
MRoute – Overlay Multicast Tree (Receiver Join)
Spine Spine
Layer-3 VNI: 50001
Site-External DCI
MTU Support)
VRF
Tenant1 VXLAN EVPN
*,G – (*, 239.1.1.2/32)
S,G – (10.10.10.100, 224.10.10.10) VTEP VTEP VTEP VTEP

S,G – (10.10.10.100, 224.10.10.10)
IIF: VLAN100 (Host-facing) DR DR DR DR IIF: VRF-L3VNI (Overlay)
OIF: VRF-L3VNI (Overlay), VLAN100 (Host-facing) OIF: VLAN100 (Host-facing)

224.10.10.10 10.10.10.10 20.20.20.20 10.10.10.14
10.10.10.100
Agenda
• Multi-Site
• VXLAN OAM
VXLAN BGP EVPN Multi-Site
Inter-X Connectivity
VXLAN Multi-Pod VXLAN Multi-Fabric VXLAN Multi-Site
EVPN Control- BGP EVPN EVPN Control- EVPNFabric

Control-Plane EVPNFabric
Control-Plane EVPNFabric
Control-Plane
#1 BGP EVPN EVPNFabric
Control-Plane
#2
Fabric #1 Fabric #2 #1 #2
Plane Domain 1 Plane Domain 2 Domain 1 Domain 2 Domain 1 Domain 2
Overlay Overlay Overlay Overlay Overlay Overlay

VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE
P P P P P P P P P P P P P P P P P P P P P P P P
Bar Bar Bar Bar

em em em em Bar Bar Bar Bar Bar Bar Bar Bar
eta eta eta eta em em em em em em em em
l l l l etal etal etal etal etal etal etal etal
DCI DCI
Single Data-Plane – End-to-End Data-Plane Domain 1 Data-Plane Domain 2 Data-Plane Domain 1 Data-Plane Domain 2
Data-Plane Data-Plane
• Single Fabric with End-to-End • Multiple Fabrics – Normalised • Multiple Fabrics with
Encapsulation through Ethernet Integrated DCI
• Build Hierarchy in the Underlay • Multiple Fabrics Interconnect • Integrated DCI – Scaling
– Flatten it in the Overlay using DCI (Layer 2 and Layer 3) within and between Fabrics
Functional Components https://tools.ietf.org/html/draft-sharma-multi-site-evpn
Site-External DCI
Border Gateways MTU Support)
(Key Functional Components of
VXLAN Multi-Site Architecture)
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW
Spine Spine Spine Spine Spine Spine Spine Spine
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP
Site 1 Site n
Site-Internal Fabric
(Common VXLAN and
BGP-EVPN Functions) BRKDCN-3378 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Hierarchical Overlay Domains
Overlay Multi-Site
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW
Spine Overlay Site 1

Spine Spine Spine Spine Overlay Site n
Spine Spine Spine
Site 1 Site n
Anycast Border Gateway (1)
Anycast Border Gateway
• Up to 4 Border Gateways
• Border Gateway
BGW BGW BGW BGW • Deploying at Leaf – 7.0(3)I7(1)
VTEP VTEP VTEP VTEP
• Deploying at Spine – 7.0(3)I7(2)
Site 1
• Common Virtual IP (VIP) across BGW
Border VIP
10.1.1.111 • VIP for communication between the Border
BGW BGW BGW BGW
Gateways in different Sites
VTEP
PIP-BGW1
VTEP
PIP-BGW2
VTEP
PIP-BGW3
VTEP
PIP-BGW4 • VIP for communication between Border
10.1.1.101 10.1.1.102 10.1.1.103 10.1.1.104
Gateway and Leaf within a Site
Border VIP
• Individual Primary IP (PIP) per BGW
10.1.1.111
• Used for Broadcast, Unknown Unicast and
Multicast (BUM) replication
• PIP for communication with Single-Homed
endpoints (routed only), intra- and inter-Site
Site 1
Type: 03 IP: 10.1.1.101
4 System MAC: 00:00:00:00:00:01
Ethernet Segment: 00:03:09 VNI: 30010 • Per-VNI Designated Forwarder (DF) election
• Each BGW can serve as DF for a single or a
BGW BGW BGW BGW
set of Layer-2 VNIs
VTEP VTEP VTEP VTEP
• DF election and assignment is automatic
DF
30010
DF
30011
DF
30012
DF
30099
• Using BGP EVPN Route Type 4 for DF election
• Operator Managed Assignment (Type: 03)
BGP EVPN
• Six Octet Site Identifier (System MAC:
00:00:00:00:00:01)
RR RR
Spine Spine
• Multi-Site Discriminator (Ethernet-Segment:
00:03:09)
• Originators IP Address (PIP): 10.1.1.101
• Layer-2 VNI: 30010
Site 1
Failure Detection on BGWs – Fabric Isolation (1)
• The Site-Internal interfaces on BGW nodes are

constantly tracked to determine their status (‘evpn
Site-External
Multi-Site VIP
multisite fabric-tracking’ command)
10.111.111.1
BGW BGW BGW BGW

VTEP VTEP VTEP VTEP
PIP-BGW2 PIP-BGW3 PIP-BGW4

10.200.200.22 10.200.200.23 10.200.200.24
Site-Internal
Spine Spine
Site 1
 The Site-Internal interfaces on BGW nodes are

Site-External
Multi-Site VIP
10.111.111.1
• If all the Site-Internal interfaces are detected as
BGW BGW BGW BGW down:
VTEP VTEP VTEP VTEP
1. The isolated BGW stops advertising PIP/VIP
addresses toward the Site-External network
10.200.200.22 10.200.200.23 10.200.200.24
The remaining BGWs perform new DF elections for
Site-Internal
2.
the L2VNIs owned by the isolated BGW
Spine Spine
Site 1
• The Site-Internal interfaces on BGW nodes are

Site-External
Multi-Site VIP
10.111.111.1
• If all the Site-Internal interfaces are detected as
BGW BGW BGW BGW down:
VTEP VTEP VTEP VTEP
1. The isolated BGW stops advertising PIP/VIP
addresses toward the Site-External network
10.200.200.22 10.200.200.23 10.200.200.24
The remaining BGWs perform new DF elections for
Site-Internal
2.
the L2VNIs owned by the isolated BGW
Spine Spine • As a result, the BGW becomes isolated from both
the Site-Internal and Site-External networks
Site 1
• Seamless BGW node reinsertion using a “delay-
restore” timer for the VIP address
Failure Detection on BGWs – DCI Isolation
DC Core
(Layer-3 Unicast) • The Site-External interfaces on BGW nodes are
also tracked to determine their status (‘evpn
Site-External
multisite dci-tracking’ command)

• If all the Site-External interfaces are detected as
BGW BGW BGW BGW down, the isolated BGW node:
VTEP VTEP VTEP VTEP
1. Stops advertising VIP VTEP address toward the Site-
PIP-BGW1 PIP-BGW2 PIP-BGW3 PIP-BGW4
Internal network
10.200.200.21 10.200.200.22 10.200.200.23 10.200.200.24 Withdraws BGP EVPN Type-4 advertisements
Site-Internal
2.
Multi-Site VIP (triggering a new DF election between other BGWs)
10.111.111.1
3. Starts functioning as a regular VTEP (PIP still up)
• As a result, the BGW continues to operate as a

Site 1 Site-Internal VTEP
• Seamless BGW node reinsertion using a “delay-
restore” timer for the VIP address
Multi-Site Control- & Data-Plane
BRKDCN-3378
Multi-Site Control Plane Deployment Considerations
• MP-eBGP EVPN only inter-Sites
• Next-hop behaviour (VXLAN tunnel termination and reorigination) and loop protection
(as-path attribute)
• Two main options for underlay and overlay control plane deployment
1. I-E-I (Recommended)
• Intra-Site: IGP (OSPF, IS-IS) as underlay CP, iBGP as overlay CP
• Inter-Sites: eBGP for both underlay and overlay CPs
2. E-E-E
• Intra-Site and Inter-Sites: eBGP for both underlay and overlay CPs
• Full mesh of MP-eBGP EVPN adjacencies across sites

• Recommended to deploy a couple of Route-Servers with 3 or more sites
• RS in a separate AS only perform control plane functions (“eBGP Route-Reflectors”,
IETF RFC 7947)
• RS functions: EVPN routes reflection, next-hop-unchanged, route-target rewrite
Multi-Site Overlay Control Plane – back-to-back
Site 3
Site N
Site 2
eBGP EVPN
Full mesh
Site 1
Multi-Site Overlay Control Plane – Route-Server
Site 3
eBGP EVPN
RS
Site N
Site 2
RS
eBGP EVPN
Site 1
Multi-Site Overlay Control Plane – Tenants
RS
L3VNI: 50001 L3VNI: 50001

Route-Target: 65501:50001 DC Core Route-Target: 65502:50001
(Layer-3 Unicast)
VRF VRF
Tenant1 Tenant1
DCI
…. ….
VTEP VIP1 VTEP VTEP VIP2 VTEP
10.1.1.111 10.2.2.222
Fabric BGW BGW BGW BGW
VXLAN EVPN
iBGP-EVPN VXLAN EVPN
iBGP-EVPN
RR RR
L2VNI: 30010 (VLAN 10)
Site1 L2VNI: 30020 (VLAN 20)
L3VNI: 50001 (Tenant1) L3VNI: 50001 (Tenant1) L3VNI: 50001 (Tenant1)
Host1 Host2 Host3

0000.3010.1101 0000.3020.2101 0000.3010.1102
192.168.10.101 192.168.20.101 192.168.10.102
Multi-Site Overlay Control Plane – Site1
RS
L3VNI: 50001 L3VNI: 50001

(Layer-3 Unicast)
2 0000.3010.1101/48 30010, 65501:30010 192.168.10.101/32 50001, 65501:50001 10.1.1.1

VRF VRF
Tenant1 2 0000.3020.2101/48 30020, 65501:30020 192.168.20.101/32 Tenant1 10.1.1.111
50001, 65501:50001
DCI 2 0000.3010.1102/48 30010, 65501:30010 192.168.10.102/32 50001, 65501:50001 10.1.1.111
…. ….
10.1.1.111 10.2.2.222
VXLAN EVPN VXLAN EVPN

RR RR
L2VNI: 30010 (VLAN 10)
Host1 Host2 Host3

0000.3010.1101 0000.3020.2101 0000.3010.1102
192.168.10.101 192.168.20.101 192.168.10.102
Multi-Site Overlay Control Plane – Site2
RS
L3VNI: 50001 L3VNI: 50001

(Layer-3 Unicast)
2 VRF 30010, 65502:30010

0000.3010.1101/48 192.168.10.101/32 50001, 65502:50001 10.2.2.222 VRF
2 Tenant1 30020, 65502:30020
0000.3020.2101/48 192.168.20.101/32 50001, 65502:50001 10.2.2.1 Tenant1
DCI
2 0000.3010.1102/48 30010, 65502:30010 192.168.10.102/32 50001, 65502:50001 10.2.2.3
…. ….
10.1.1.111 10.2.2.222

RR RR
L2VNI: 30010 (VLAN 10)
Host1 Host2 Host3

0000.3010.1101 0000.3020.2101 0000.3010.1102
192.168.10.101 192.168.20.101 192.168.10.102
Multi-Site Overlay Control Plane – Between Sites
RS
L3VNI: 50001 L3VNI: 50001

(Layer-3 Unicast)
VRF VRF
Tenant1 Tenant1
DCI
…. ….
Type MAC / Length
10.1.1.111 L2VNI / RT IP / Length L3VNI / RT Next-Hop
10.2.2.222 Seq.
Fabric BGW 2 0000.3010.1101/48BGW30010, 65599:30010 192.168.10.101/32 BGW
50001, 65599:50001 10.1.1.111 BGW
2 0000.3020.2101/48 30020, 65599:30020 192.168.20.101/32 50001, 65599:50001 10.2.2.222

VXLAN
2
EVPN
0000.3010.1102/48 30010, 65599:30010 192.168.10.102/32
VXLAN
50001, 65599:50001
EVPN
10.2.2.222
RR RR
L2VNI: 30010 (VLAN 10)
Host1 Host2 Host3

0000.3010.1101 0000.3020.2101 0000.3010.1102
192.168.10.101 192.168.20.101 192.168.10.102
Multi-Site Overlay Data Plane – Overview
Inter-site VXLAN
Data Plane
De-capsulation and DC Core

Re-encapsulation on De-capsulation and
BGW (L2 or L3 lookup)
(Layer-3 Unicast)
Re-encapsulation on
BGW (L2 or L3 lookup)
DCI
…. ….
10.1.1.111 10.2.2.222

Intra-site VXLAN
Data Plane
Site1 Site2
Host1 Host2 Host3

0000.3010.1101 0000.3020.2101 0000.3010.1102
192.168.10.101 192.168.20.101 192.168.10.102
Multi-Site Packet Walk (BUM)
BRKDCN-3378
Multi-Site – BUM Traffic Distribution
Overlay Multi-Site
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW

Spine Spine Spine
BUM
Site 1 Site n
Baremetal
Multi-Site – BUM Replication Modes (Multicast
Sites)
Overlay Multi-Site
Ingress Replication
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW

Spine Spine Spine
Multicast Multicast
Site 1 Site n
Multi-Site – BUM Replication Modes (All Ingress
Replication)
Overlay Multi-Site
Ingress Replication
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW

Spine Spine Spine
Ingress Replication Ingress Replication
Site 1 Site n
Multi-Site – BUM Replication Modes (Mixed Site)
Overlay Multi-Site
Ingress Replication
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW

Spine Spine Spine
Ingress Replication Multicast
Site 1 Site n
Multi-Site – BUM Traffic Enforcement
Overlay Multi-Site
Storm Control
VTEP VTEP Broadcast 0-100% VTEP VTEP
BGW BGW
Unknown Unicast 0-100% BGW BGW
Multicast 0-100%
Spine Spine Spine
BUM
Site 1 Site n
Baremetal
Layer 2 (BUM) – Site 1
Bridge

Payload
L10 DGROUP 30010 H1-MAC ALL-F H1-IP ALL-255
DF
Leaf10 replicates VTEP 30010 VTEP
traffic intra-Site BGW11 BGW21
2
VTEP VTEP
VXLAN EVPN VXLAN EVPN VXLAN EVPN
Leaf10 Site1 DCI Site2 Leaf20
DF
VTEP 30010 VTEP
1 Host 1 sends a
BGW12 BGW22
L2 BUM frame
Baremetal Baremetal
Host 1 Host 2
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102
Layer 2 (DF and Split Horizon) – Site 1
Bridge

Payload
L10 DGROUP 30010 H1-MAC ALL-F H1-IP ALL-255
DF
VTEP 30010 VTEP
BGW11 BGW21
VTEP BUM Forward VTEP

DF
VTEP 30010 VTEP
BGW12 BGW22
Baremetal Baremetal
Host 1 Host 2
0000.3010.1101 Drop due to Designated Forwarder (DF) rule 0000.3010.1102
192.168.10.101 192.168.10.102
Drop due to Split-Horizon rule

Layer 2 (BUM) – DCI
Bridge
BGW11 BGW21 30010 H1-MAC ALL-F H1-IP ALL-255 Payload
BGW11 BGW22 30010 H1-MAC ALL-F H1-IP ALL-255
DF
VTEP 30010
BGW11 replicates traffic VTEP
BGW11
inter-Sites toward remote BGW21
BGW nodes
VTEP BUM Forward 3 VTEP
DF
VTEP 30010 VTEP
BGW12 BGW22
Baremetal Baremetal
Host 1 Host 2
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102
Layer 2 (DF and Split Horizon) – DCI
Bridge
BGW11 BGW21 30010 H1-MAC ALL-F H1-IP ALL-255 Payload
BGW11 BGW22 30010 H1-MAC ALL-F H1-IP ALL-255
DF
VTEP 30010 VTEP
BGW11 BGW21
VTEP BUM Forward VTEP

DF
VTEP 30010 VTEP
BGW12 BGW22
BUM Forward
Baremetal Baremetal
Host 1 Host 2
192.168.10.101 192.168.10.102

Bridge

Payload
BGW22 DGROUP 30010 H1-MAC ALL-F H1-IP ALL-255
DF
VTEP 30010 VTEP
BGW11 BGW21 BGW22 replicates

traffic intra-Site
VTEP
4 VTEP
DF
VTEP 30010 VTEP
BGW12 BGW22
BUM Forward
Baremetal Baremetal
Host 1 Host 2
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102
Layer 2 (DF and Split Horizon) – Site 2
Bridge

Payload
BGW22 DGROUP 30010 H1-MAC ALL-F H1-IP ALL-255
DF
VTEP 30010 VTEP
BGW11 BGW21
VTEP VTEP
DF
VTEP 30010 VTEP
BGW12 BGW22
BUM Forward
Baremetal Baremetal
Host 1 Host 2
192.168.10.101 192.168.10.102

Bridge
DF
VTEP 30010 VTEP
BGW11 BGW21
VTEP VTEP
DF
VTEP 30010 VTEP
BGW12 BGW22
5
Leaf20 sends traffic to
local Host 2
Baremetal Baremetal
Host 1 Host 2
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102
Multi-Site and External Layer 3 Connectivity
• The BGW nodes can also be used to provide Layer-3 external
connectivity to each site
• Different connectivity models are supported
• VRF-Lite peering with external WAN Edge routers
• MP-BGP EVPN peering with external WAN Edge routers
(Shared Border deployment model, aka GOLF)
• Dedicated or shared pair of WAN Edge routers across sites
• External Layer-3 network may be different from the DCI network

used for inter-site communication
Border Gateway and VRF-Lite
Separate routing peering

Dedicated physical interfaces / VRF-A VRF-B VRF-C
for each VRF (IGP or
sub-interfaces* for each VRF, External eBGP)
separate from interfaces used Connectivity
for Multi-Site traffic
Site-External
Multi-Site
Overlay
BGW BGW BGW BGW BGW BGW BGW BGW

Site-Internal
Site 1 Site N
*No current SVIs support on BGWs
Border Gateway and Shared External Connectivity
Various hand-off options
depending on deployed HW Shared Border operates like a
VRF-A VRF-B VRF-C traditional VXLAN EVPN
(VRF-Lite, MPLS-VPN, LISP)
External VTEP (Layer 3 only)
Connectivity
Single MP-BGP EVPN VXLAN Data Plane
instance to exchange VTEP VTEP
between BGW and WAN
Site-External
routes for all VRFs Edge Router

Shared Border
Multi-Site
Overlay
BGW BGW BGW BGW BGW BGW BGW BGW

Site-Internal
Site 1 Site N
Agenda
• Multi-Site
• VXLAN OAM
Operations, Administration and Management (OAM)
• OAM – processes, activities,
tools and standards
• Various Mode of Operation
• Pro-Active
• Controlling a Situation
• Re-Active
• Responding to a Situation
VXLAN OAM - OAM Model of Operation
V V V V V V V
Endpoint Locator Ping / Path MTU Pathtrace Pro-Active Monitoring

• Locate End-Host and • Check liveliness of End- • Trace paths to End-Host • Proactive Monitoring with
Segment Identifier Host and Tunnel-Endpoint Threshold and State
Notifications
• Track History of End- • Option to specify Payload • Get Path, Interface and
Host Parameters Error statistics along path
• Provide Fabric Host- • Specify Payload
Count and Activity Parameters for Path
Selection
NGOAM or VXLAN OAM
• Next Generation OAM for Data

Centre Fabrics
• Running on Nexus 9000, Nexus
7000 and Nexus 5600
• VXLAN Today
• All IP Tomorrow
• Various Methods to Execute and

Retrieve Data
• Command Line Interface (CLI)
• NX-API
• DCNM (using NX-API)
Endpoint Traceroute – VXLAN OAM
• Endpoint Reachability
• Uses ICMP
• VTEP to Endpoint reachability
Overlay What is the Path • VTEP to VTEP reachability
to Host A?

• Validates Overlay Path
• Single Specified Path
• Multiple, Specified Path
• Provides Overlay to Underlay

Baremetal Baremetal Baremetal correlation
MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101
How Would a Normal Traceroute Look Alike?
Eth1/5
10.1.1.17
Spine
NVE1
10.200.200.18 What is the Path
to Host A?
VTEP VTEP
L15# traceroute 192.168.10.101 source 10.50.1.15 vrf BLUE
Baremetal
AS#65500
traceroute to 192.168.10.101 (192.168.10.101) from 10.50.1.15 (10.50.1.15), 30 hops max, 40 byte packets
Host(10.50.1.18)
1 10.50.1.18 A 0.96 ms 0.817 ms 0.746 ms
MAC: 0000.3001.1101
2 2 192.168.10.101 (192.168.10.101) 4.751 ms 0.69 ms 0.697 ms
IP: 192.168.10.101
Which Path did my Traceroute take?

Spine
Eth1/5
10.1.2.17
Endpoint Traceroute – VXLAN OAM – Close-Up
L15# traceroute nve ip 192.168.10.101 vrf BLUE source 10.50.1.15 sport 35977 verbose
Codes: '!' - success, 'Q' - request not sent, '.' - timeout,

'D' - Destination Unreachable, 'X' - unknown return code,
'm' - malformed request(parameter problem),
'c' - Corrupted Data/Test, '#' - Duplicate response
Traceroute Request to peer ip 10.200.200.18 source ip 10.200.200.15

Sender handle: 94
1 !Reply from 10.1.1.17,time = 1 ms Spine Ingress Interface IP
2 !Reply from 10.200.200.18,time = 1 ms
Destination VTEP IP
3 !Reply from 192.168.10.101,time = 4 ms
Host A IP
Spine Ingress Interface and Destination VTEP IP Address
are Underlay Information – additions vs. standard Traceroute
Pathtrace for Enhanced Network Visibility
What is the Path

• Application Specific Pathtrace
from Host C to • Uses “draft-tissa-nvo3-oam-fm”
Host A for HTTP?
•
Endpoint to Endpoint Pathtrace
• Adds Interface Load and Error
Overlay Statistic of the Path
• Uses Protocol Information
• Validates Specific or All Path

• Provides Overlay to Underlay
correlation
Host A Host B Host C • Superset of NVE Ping/Traceroute

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101
Pathtrace – VXLAN OAM – Close-Up
L15# pathtrace nve ip unknown vrf BLUE Known or Unknown VTEP IP Address
payload
ip 192.168.10.101 192.168.20.101 Dst Endpoint IP / Src Endpoint IP
port 54321 80 Source Port / Destination Port
proto 6
payload-end TCP (IANA Protocol Number 6)

'D' - Destination Unreachable, 'X' - unknown return code,
Path trace Request to peer ip 10.200.200.18 source ip 10.200.200.15

Sender handle: 142
Hop Code ReplyIP IngressI/f EgressI/f State

====================================================
1 !Reply from 10.1.1.17, Eth1/5 Eth1/8 UP / UP
2 !Reply from 10.200.200.18, Eth1/54 Unknown UP / DOWN
Why are we Specifying Payload Information?
101010110101010
10101010
• VXLAN provides variable UDP

Baremetal Baremetal
Host A Spine Host B

MAC: 0000.3001.1101
IP: 192.168.10.101
Source Port in Outer Header
MAC: 0000.3001.1102
IP: 192.168.10.102
• Hash of the inner Layer-2/Layer-

VTEP Spine
3/Layer-4 Headers of the original
VTEP
Ethernet Frame.
• Enables entropy for ECMP Load
balancing in the Network
Which Path did your Application Traffic took?
Pathtrace – VXLAN OAM – Close-Up
L15# pathtrace nve ip unknown vrf BLUE payload ip 192.168.10.101 ...

'D' - Destination Unreachable,
Spine Ingress 'X' Interface
Interface, Egress - unknown return
and code,VTEP IP Address
Destination
are Underlay Information – additions vs. standard and NVE Traceroute
Path trace Request to peer ip 10.200.200.18 source ip 10.200.200.15

Sender handle: 142
Spine Ingress Interface IP Spine Ingress Interface Spine Egress Interface
Hop Code ReplyIP IngressI/f EgressI/f State

====================================================
1 !Reply from 10.1.1.17, Eth1/5 Eth1/8 UP / UP Interface Status
2 !Reply from 10.200.200.18, Eth1/54 Unknown UP / DOWN
Destination VTEP IP Destination Leaf Ingress Interface
Database Output – VXLAN OAM – Close-Up
L15# show ngoam pathtrace database session 168 detail
Pathtrace entry for session id 168 OAM Session ID

================================
Start time: Tue Jun 13 01:18:39.710 PDT

End time: Tue Jun 13 01:18:39.735 PDT
Last Clear of Summary Statistics: Never

Pathtrace Requests: sent (2)/received (0)/timeout (0)/unsent (0)
Pathtrace Replies: sent (0)/received (2)/unsent (0)/Duplicate (0)
! Reply from 10.1.1.17 on Eth1/5, state UP. Sent on Eth1/8, state UP.
Interface stats for interface: Eth1/5
-------------------------------
Rx Len : 84
Rx Bytes : 66113123 Interface Statistics
Rx Pkt rate : 0
Rx Byte rate : 0
Rx Load : 0 BRKDCN-3378 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 193
Summary
Summary
• Overview on VXLAN Overlay
• Control- and Data-Plane interactions
• Some info around Multicast forwarding
• How Multi-Site enhanced VXLAN EVPN
• Operations is key – VXLAN OAM
If you haven’t
had enough
VXLAN BGP
EVPN
Links & Resources
• VXLAN Multi-Site Intro
• https://blogs.cisco.com/datacenter/vxlan-innovations-vxlan-evpn-multi-site-part-2-of-2
• VXLAN Multi-Site @ Cisco Live online
• https://www.ciscolive.com/global/on-demand-library/?search=BRKDCN-2035#/
• ”eBGP” for EVPN
• https://learningnetwork.cisco.com/blogs/community_cafe/2017/11/02/vxlan-ebgp-evpn-
the-incarnation-of-a-hybrid-guest-post
• Configuration Example
• https://communities.cisco.com/community/technology/datacenter/data-center-
networking/blog/2015/05/19/vxlanevpn-configuration-example
Q&A
Complete Your Online
Session Evaluation
• Give us your feedback and
receive a Cisco Live 2018 Cap
by completing the overall event
evaluation and 5 session
evaluations.
• All evaluations can be completed
via the Cisco Live Mobile App.
Don’t forget: Cisco Live sessions will be
available for viewing on demand after the
event at www.CiscoLive.com/Global.
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Thank you

BRKDCN 3378

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

BRKDCN 3378

Uploaded by

Copyright:

Available Formats

Building Data Centre

Networks with VXLAN

Hypervisor Baremetal Hypervisor Hypervisor Baremetal Hypervisor Baremetal Baremetal

Hypervisor Baremetal Hypervisor Hypervisor Baremetal Hypervisor Baremetal Baremetal

• Transport Independent • Uses Various Data-Planes

EVPN MP-BGP – RFC 7432

MPLS Provider Backbone Bridges Overlay (NVO3)

(draft-ietf-l2vpn-evpn) (draft-ietf-l2vpn-pbb-evpn) (draft-ietf-bess-evpn-overlay)

RFC 7348 Virtual Extensible Local Area Network Data Plane

RFC 7432 BGP MPLS based Ethernet VPNs Control Plane

draft-ietf-bess-evpn-overlay A Network Virtualisation Overlay Solution using EVPN Control Plane

draft-ietf-bess-evpn-inter-subnet-forwarding Integrated Routing and Bridging in EVPN Control Plane

draft-ietf-bess-l2vpn-evpn-prefix-advertisement IP Prefix Advertisement in E-VPN Control Plane

draft-tissa-nvo3-oam-fm NVO3 Fault Management / OAM Management Plane

Asymmetric IRB Symmetric IRB

RFC 7348 Virtual Extensible Local Area Network Data Plane

RFC 7432 BGP MPLS based Ethernet VPNs Control Plane

draft-ietf-bess-evpn-overlay A Network Virtualisation Overlay Solution using EVPN Control Plane

draft-ietf-bess-evpn-inter-subnet-forwarding Integrated Routing and Bridging in EVPN Control Plane

draft-ietf-bess-l2vpn-evpn-prefix-advertisement IP Prefix Advertisement in E-VPN Control Plane

draft-tissa-nvo3-oam-fm NVO3 Fault Management / OAM Management Plane

VTEP = VXLAN Tunnel End-Point

RFC 7348 Virtual Extensible Local Area Network Data Plane

RFC 7432 BGP MPLS based Ethernet VPNs Control Plane

draft-ietf-bess-evpn-overlay A Network Virtualisation Overlay Solution using EVPN Control Plane

draft-ietf-bess-evpn-inter-subnet-forwarding Integrated Routing and Bridging in EVPN Control Plane

draft-ietf-bess-l2vpn-evpn-prefix-advertisement IP Prefix Advertisement in E-VPN Control Plane

draft-tissa-nvo3-oam-fm NVO3 Fault Management / OAM Management Plane

• Single Subnet per EVI

RFC 7348 Virtual Extensible Local Area Network Data Plane

RFC 7432 BGP MPLS based Ethernet VPNs Control Plane

draft-ietf-bess-evpn-overlay A Network Virtualisation Overlay Solution using EVPN Control Plane

draft-ietf-bess-evpn-inter-subnet-forwarding Integrated Routing and Bridging in EVPN Control Plane

draft-ietf-bess-l2vpn-evpn-prefix-advertisement IP Prefix Advertisement in E-VPN Control Plane

draft-tissa-nvo3-oam-fm NVO3 Fault Management / OAM Management Plane

Route Type 2 = MAC/IP Route

2 0000.3001.1101 / 48 3001, 65500:3001 10.200.200.101

2 0000.3001.1102 / 48 3001, 65500:3001 10.200.200.104

• MPLS Label1 (L2VNI*)

• MAC attributes are Mandatory

Baremetal Baremetal Baremetal

Host A Host B Host C

2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101 /32 • Host

• IP Attributes are Optional

5 192.168.10.0 /24 5000, 65500:5000 10.200.200.101

• Populated through External

MAC 802.1q IP Payload CRC

Data-Plane (VXLAN) 20-byte + 8-byte +8-byte + 14-byte* = 50 Bytes

UDP Src Port

VLAN ID Tag 16 Checksum 0x0000 16

Ether Type 0x0800 16

Outer MAC Outer IP UDP VXLAN Inner MAC Payload CRC

Field Value Bites Total

VNI 16M Possible Segments 24

Dst VTEP IP L2VNI Dst MAC Dst IP

Spine Spine Spine Spine

SIP DIP VXLAN Overlay

VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Baremetal Baremetal Baremetal

Host A Host B Host C

Spine Spine Spine Spine

SIP DIP VXLAN Overlay

VTEP VTEP VTEP VTEP VTEP VTEP VTEP