BRKMPL 3333

#CLUS
EVPN
Network Virtualization Solution for Next
Generation DCs & DC Interconnect
Ali Sajassi - Distinguished Engineer
Lukas Krattiger – Principal Engineer
Jiri Chaloupka - TME
BRKMPL-3333
#CLUS
Session Objectives
• To review EVPN strengths and its role as unified Overlay Control
Plane
• To review main use cases in DC, SP, & Ent and how EVPN enables
such use cases
#CLUS BRKMPL-3333 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Cisco Webex Teams
Questions?
Use Cisco Webex Teams to chat
with the speaker after the session
How
1 Find this session in the Cisco Live Mobile App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space
Webex Teams will be moderated cs.co/ciscolivebot#BRKMPL-3333

by the speaker until June 16, 2019.
#CLUS © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
Agenda
• Overview
• History, Summary of Services
• Services - Part I
• Secure EVPN & Multi-homing
Services – Part II
• IRB, Multicast, and DC Interconnect
• Services - Part III
• EVPN & L3VPN Interworking, EVPN & VPLS Interworking,
and VPWS & FXC
• Conclusion & Q&A
EVPN Overview
EVPN History
2010 - 2012 2013 - 2015 2016-2019
Following drafts were Multicast Services

EVPN was introduced introduced - Optimized ingress
First - EVPN IRB replication
- EVPN DCI - IGMP & MLD proxy
Following drafts were - Virtual ES - IRB Multicast
introduced next: - VPLS seamless
- PBB-EVPN integration for all- Enhancements
- EVPN-VPWS active multi-homing - L3VPN multi-homing
- EVPN-Overlay - Inter-AS IRB
- EVPN-ETREE - DF Election algorithms
- Mobility
- FXC
Secure EVPN
EVPN – Next Gen Overlay Technology
EVPN
Overlay (Provides many services and supports many encapsulations)
Supported Encap: MPLS, VxLAN, NVGRE, GUE, GENEVE, SRv6
MPLS IP Segment
Underlay
(LDP & RSVP-TE) (v4 & v6) Routing
EVPN: Unified Overlay Solution
Data Center Service Provider Enterprise
De-facto overlay solution for Next gen all-in-one VPN EVPN is used to address scale
transforming SP COs to NG solution to provide L2VPN, and efficiency issues as
DC to support virtualization, L3VPN, IRB, and more w/ a Enterprises need to migrate
multi-tenancy, flexible MH, single solution from traditional L2
flexible workload placement & access/aggregation networks
mobility, full cross-sectional BW Provides distributed vPE w/ to multi-tenancy fabric w/ the
utilization in fabric and access. white boxes for scaling out same characteristics as NG
instead of scaling up DC.
Supporting multiple Underlay:
MPLS, VxLAN, SR, GENVE Backhauling solution to vPE
EVPN: An All-in-one Solution
Services: E-LAN E-LINE E-TREE NVO Multicast L3VPN
(DC Fabric)
EVPN Solutions EVPN EVPN-VPWS EVPN-ETREE EVPN-Overlay EVPN-Mcast EVPN-L3VPN

PBB-EVPN EVPN-FXC EVPN-IRB
RFCs/Drafts: RFC 7209 RFC 8214 RFC 8317 RFC 8365 4 WG draft 2 WG drafts
RFC 7432 1 WG draft 2 WG drafts 1 Indv draft
RFC 7623
RFC 8584
5 WG drafts
7 Indv drafts
It provides all these service while supporting multi-tenancy,

workload mobility, full cross-sectional BW utilization, flexible
multi-homing, etc.
EVPN for SP Distributed vPE
Unified VPN PE1
CE2
nPE
Solution EVPN uPE
EVPN
CE3 CE PW
CE1 Metro WAN PE2
VPLS
Metro CE4
• EVPN has emerged as such unified solution because of:

• Offering E-LAN services that is significantly enhanced because of uPE
per host-address policy (MAC and/or IP), extensive multi-homing EVPN-VPWS Distributed
capabilities, and simplifying integration w/ access networks
nPE using
Scalable and flexible P2P services with all-active multi-homing EVPN
•
(EVPN-VPWS and EVPN-FXC) which weren’t feasible before
CE
• BW-optimized P2MP (E-TREE) services with enhanced filtering
capabilities at the ingress nodes which weren’t possible before
1. EVPN glues a set of white boxes into a single vPE
• Ease of provisioning via auto-sensing, auto-discovery, and auto-
provisioning of access network
system supporting L2 and L3VPN services
2. EVPN provides backhauling to vPE system. Such
backhauling is not possible with traditional PW
EVPN for DC and DCI
Flexible WAN Flexible client site
Interconnect connectivity
Multi- DC1 PE
Client
tenancy EVPN
SDN Controller
PE PE
Spine Spine WAN DC2
PE EVPN
EVPN PE
Leaf Leaf Leaf Leaf PE Spine Spine

DCI/WAN
Full cross-sectional
Flexible workload placement BW utilization in fabric Leaf Leaf Leaf Leaf
w/ VM mobility and access links
server
VF2
VF3
Multi-site Data Center Interconnection w/ EVPN VF4
Some Buzzwords & Their Meanings
DC1 Client
• Full cross-sectional BW Utilization WAN DC2

EVPN
• Multi-tenancy
• Flexible client site connectivity
• Flexible workload placement w/ VM mobility V
V
F
V
F
2
F
3
• Flexible WAN Interconnect 4
• Multi-site DC interconnect
Full Cross-Sectional BW Utilization
EVPN-VxLAN
EVPN-MPLS
(e.g., 48 out of 48 paths
VxLAN or VPLS = 100%)
Traditional L2 network (e.g., 4 out of 48 paths
Spine Spine Spine Spine
(bridging) = 8%)
(e.g., 1 out of 48 path Spine Spine Spine Spine Leaf Leaf Leaf Leaf Leaf Leaf Leaf
= 2%)
Leaf Leaf Leaf Leaf Leaf Leaf Leaf

Baremetal Baremetal
Baremetal Baremetal
Baremetal Baremetal
Full Cross-Sectional BW Utilization – Cont. (2)
To support full cross-sectional BW utilization, following EVPN
constructs are needed:
• EVPN All-Active MH: Providing per-flow load-balancing between CE and multi-homing

PEs
• Aliasing: Ingress PE forwarding traffic to all multi-homing egress PEs even though it has
only learned the MAC route from one of them
• BGP multi-pathing: Per-flow load-balancing between ingress PE and multi-homing
egress PEs
• IGP multi-pathing: Per-flow load balancing among ECMPs of an ingress PE and an
egress PE
MAC1 CE1 PE1 PE3 CE2 MAC2
Load balancing
PE2 PE4
Multi-tenancy
(multiple overlay networks)
Spines
EVI 20
EVI 10
Leafs
• Underlay: physical topology + • Overlay: Virtual topology among

associated routing/signaling protocol endpoints (NVEs)
protocol • Deals with reachability between tenant
• High redundancy in fabric (link/node) routes and MAC addresses
• IP underlay: IGP (or BGP) • EVPN uses BGP as routing protocol for
• MPLS underlay: LDP, RSVP-TE, SR distribution of tenants IP
• In DC: Leaf/Spine (CLOS) Topology prefixes/addresses and MACs
• Uses IP or MPLS as underlay
Multi-Tenancy – Cont. (2)
(components of overlay networks) • A tenant can have one or more EVIs
• An EVI identifies an overlay network for
MPLS just like VNI for VxLAN
Spine
EVI is an EVPN instance

spanning Leafs that
EVI 20 participate in that EVPN
EVI 10
Leaf
MAC- MAC- MAC- MAC- MAC-
EVPN uses BGP route target
MAC-
VRF VRF VRF VRF VRF VRF filtering to enable leafs that
don’t belong to a specific EVI,
not import any MACs for that
EVI, providing efficient scalability
VM VM VM VM VM
EVPN can use BGP RT
constraint (RTC) and Outbound
Route Filtering (ORF) for further
filtering of MAC routes at RR
Multi-Tenancy – Cont. (3)
(components of overlay networks)
VLAN-Aware Bundle Service I/F VLAN-based Service I/F VLAN Bundle Service I/F
PE PE PE
Bridge
Bridge VLAN Bridge
Domain 1
VLAN-1 Domain 1 VLAN-1 1,2 Domain 1
(BD1)
(BD1) MPLS or IP (BD1)
tunnel MAC-VRF1
Bridge (Ethernet Bridge
Domain 2 payload) Bridge Domain 2
VLAN-2 (BD2) VLAN-2 Domain 2 (BD2)
VLAN
(BD2)
MAC-VRF 3,4 MAC-VRF
MAC-VRF1
RD1 / RT1 RD1 / RT1
• 1:1 mapping of VLAN to BD • 1:1 mapping of VLAN to BD • N:1 mapping of VLANs to BD

• N:1 mapping of BDs to MAC- • 1:1 mapping of BD to MAC- • N:1 mapping of BDs to MAC-
VRF VRF VRF
• Can save in RT configuration • With auto-derive RT, no issue • Not typical because of MAC
• Lose granularity of router with RT config aliasing issue
filtering per BD • Can do router filtering per BD
• Recommended type to use
Flexible Client Site Connectivity
Prior to EVPN EVPN
CE1 PE1 CE1 PE1
CE2 PE2 CE2 PE2

CE3
PE3
CE3 PE3
Client
CE4
PE4
CE4 PE4
• Limited to Dual-homing • N-way redundancy
• Increased cost because of inter-chassis links • No inter-chassis links + fast convergence
• Inflexible configuration • Very flexible
• Proprietary (vPC, Virtual Switch, Cluster) • Works for both MPLS & IP transport
• Standard based with multitude of applications: SP

access, SP inter-domain, intra-DC, and inter-DC
Flexible Client Site Connectivity – Cont. (2)
Multi-home Multi-home
Single Home
Device/Network Single- Device/Network All-
Device/Network
Active (Per-VLAN) LB Active (Per-Flow) LB
(SHD/SHN))
ESI-0 ESI-0 ESI-1 ESI-1

ESI-1 ESI-1
VM
VM VM VM
• Ethernet Segment • Typically used for MHN in • Typically used for MHD in DCs
Identifier (ESI) of ‘0’ SPs
• Per-flow LB for known unicast
• No DF election • Per VLAN DF election for traffic
all traffic
• Per-VLAN DF for BUM traffic
(default) but can be per flow
Flexible Client Site Connectivity – Cont. (3)
To support flexible client site connectivity, following EVPN
constructs are needed:
MAC1 MAC2
• Split-horizon: BUM traffic CE1 PE1 PE3 CE2
doesn’t get loopback to the Echo !
originating CE device.
PE2 PE4
• DF selection: Either PE3 or MAC1 CE1 PE1 PE3 CE2 MAC2

PE4 forward the broadcast
traffic to the far-end dual- Duplicate !
homed device CE2.
PE2 PE4
Flexible Workload Placement w/ VM mobility
• It can be done w/ centralized GW but results • To forward both intra & inter-subnet traffic
in suboptimum inter-subnet forwarding (and optimally, EVPN-IRB is used (it also helps
scale issues at the GW!) with horizontal scaling)
• Optimizing full cross-sectional BW but • Flexible workload placement is thus relevant

wasting it here !! in context of EVPN-IRB
Centralized DCI GW
L3
GW / DCI GW
L2
Spine Spine
Leaf Leaf
L3 Leaf Leaf Leaf Leaf Leaf Leaf Leaf
L2
Subnet 1/VLAN1 Subnet 2/VLAN2

Subnet 1/VLAN1 Subnet 2/VLAN2
Flexible Workload Placement w/ VM mobility –
Cont. (2)
• To support Flexible workload placement w/ VM mobility,

following EVPN constructs are needed:
1. EVPN-IRB (Integrated Routing & Bridging)
2. Distributed Anycast GW
3. EVPN MAC mobility procedures
Cont. (3)
EVPN-IRB (Integrated Routing & Bridging)
IP-VPN Instance represents a

Spine collection of IP-VRFs and
their associated connectivity
for a given tenant
Leaf IP-VRF IP-VRF IP-VRF IP-VRF IRB Interface: a virtual interface

MAC- MAC- MAC- MAC- MAC- MAC-
connecting a MAC-VRF to an IP-
MAC-
VRF VRF VRF VRF VRF VRF VRF VRF
VM VM VM VM VM
Inter-subnet - EVPN can use BGP RT constraint
Routed (RTC) and Outbound Route
Intra-subnet - Filtering (ORF) for further filtering
Bridged of IP routes at RR
Cont. (4)
Distributed Anycast Gateway with BGP-EVPN
Spine Identical Anycast Gateway Virtual IP
Distributed Anycast Gateway and MAC address are configured on
serves as the gateway for all IRB i/fs for the tenant
connected hosts (overlay
anycast !!)
Host GW
IP-x, MAC-x
IP-VRF IP-VRF IP-VRF IP-VRF
MAC- MAC- MAC- MAC- MAC- MAC- MAC- Leaf

VRF VRF VRF VRF
VRF VRF VRF
VM VM VM VM VM
Host-2 Host-3
All the IRB I/Fs perform active Host-1
IP-H2, MAC-H2 IP-H3, MAC-H3
forwarding IP-H1, MAC-H1
Bridge-domain 1 Bridge-domain 2 Bridge-domain 1
Cont. (5)
• Baseline EVPN MAC Mobility Procedure (sec. 15 of EVPN)
• First time advertisement doesn’t include MAC Mobility seq #
• When a PE learns a MAC locally for which it learned it from a remote PE,
then it advertises that MAC with mobility-seq# + 1 (ESI is different)
• When a PE learns a MAC locally for which it learned it from another multi-
homing PE, the it advertises that MAC with no seq#, if the received route
didn’t have a seq# (ESI is the same)
• When a PE learns a MAC locally for which it learned it from another multi-
homing PE, the it advertises that MAC with the same seq#, if the received
route had a seq# (ESI is the same)
Cont. (5)
• EVPN MAC Mobility Enhancements
• draft-malhotra-bess-evpn-irb-extended-mobility-01
• It enhanced for IRB use cases where 1:1 fixed mapping assumption
between MAC & IP of a workload no longer holds
1. VM move results in VM IP moving to a new MAC association
• (IPa, MACa -> IPa, MACb)
2. VM move results in VM MAC moving to a new IP association

• (IPa, MACa -> IPb, MACa)
Services - Part I
Secure EVPN
Use Cases
Secured DCI SD-WAN

DC1
DC2
Internet /
WAN
EVPN Public Network
Branch
Campus
Branch
What is Secure-EVPN? Signaling Channel
Secured Traffic
(draft-sajassi-bess-secure-evpn-01.txt) RR
Channel
PE2
PE2
PE1
PE1 PE3
PE3
PEn PE4
PEn PE4
• With secure-EVPN, # of signaling sessions is

• Currently a full-mesh of IPsec SAs require a full- reduced from O(N^2) to O(N)
mesh of IKv2 signaling sessions • BGP is used instead of IKv2 for exchange of info
in establishing IPsec Sas
• This reduction helps secure-EVPN to scale easily
for VPNs with thousands of sites
General Requirements
• Support of different VPN solutions over unsecure Internet – i.e.,
EVPN, IP-VPN, MVPN, VPLS, etc.
• With primary application for EVPN
• Leverage P2MP signaling of BGP to setup P2P SAs (secured
tunnels) among participating PEs
• Provide VPN services over the secured tunnels
User/Tenant Requirements
• To protect user layer 2 & 3 data and control traffic
• Implies user control & routing info advertised in BGP must be protected
• Which in turn implies BGP session must be protected
• To protect User both unicast & mcast data

• To support following policy & DH group list
• a single policy & DH group for all Sas
• multiple policies & DH groups among SAs
User/Tenant Requirements – Cont.
• To support following granularity of IPsec Security Association (SA) tunnels
a) Per PE: A single IPsec tunnel between a pair of PEs to be used for all tenants'
traffic supported by the pair of PEs
b) Per tenant: A single IPsec tunnel per tenant per pair of PEs
c) Per subnet: A single IPsec tunnel per subnet (e.g., per VLAN) of a tenant on a
pair of PEs
d) Per IP address: A single IPsec tunnel per pair of IP addresses of a tenant on a
pair of PEs
e) Per MAC address: A single IPsec tunnel per pair of MAC addresses of a tenant
on a pair of PEs
f) Per Attachment Circuit: A single IPsec tunnel per pair of Attachment Circuits
between a pair of PEs
General Approach
• Leverage existing Tunnel Encap attribute to signal security info
• Security info can be considered as “color” of the tunnel !!
• Define a new encapsulation sub-TLV for VxLAN-in-IPsec

• Use this sub-TLV to pass security info
• DH groups & public values
• Policy list
• Pseudo Rando Func list
• Use existing BGP mechanism for tunnel setup

• Distribute tunnel attribute w/ existing routes (or new routes) in P2MP fashion
• Use hierarchy where if a child route doesn’t have its own attribute, it inherits the attribute of its
parent
• Route per PE -> Route per Tenant/subnet -> route pre MAC/IP
Signaling for cntrl
traffic SA setup
Solution Overview SA for cntrl Traffic
Signaling for data

traffic SA setup
• Secure control channel between each
PE and the RR (e.g., using existing RR SA for Data Traffic
scheme such as IKv2)
• Setup BGP session over this secure tunnel PE2
• Use this secured BGP channel for P2MP

signaling to establish P2P IPsec SAs
PE3
• No need for P2P signaling to establish P2P
SA for data traffic
• Reducing # of msg exchanges from O(N^2)
to O(N)
PEn PE4
• Each PE advertises to other PEs the info
needed for establishing P2P SAs
Solution Overview – Cont.
• When a PE device first comes up and wants to setup an IPsec SA
between itself and each of the interested remote PEs, it generates
a DH pair for each of its intended IPsec SA using an algorithm
defined in the IKEv2 Diffie-Hellman Group Transform IDs [IKEv2-
IANA].
• The originating PE distributes DH public value along with a nonce
(using IPsec Tunnel TLV in Tunnel Encapsulation Attribute) to other
remote PEs via the RR.
• Each receiving PE uses this DH public number and the
corresponding nonce in creation of IPsec SA pair to the
originating PE
Encapsulations
Two types of IPSec encapsulations for our applications
MAC Header
MAC Header EtherType: IPv4/IPv6
EtherType: IPv4/IPv6 IP Header

Protocol = ESP
IP Header
1. VxLAN Encap within ESP Protocol = UDP ESP Header
UDP Header UDP Header

L4DPort = VxLAN L4DPort=VxLAN
VXLAN Header VXLAN Header
ENCRYPTED
IPsec encap in transport mode
without outer UDP header Payload Payload
ESP Trailer
CRC
(NP = UDP)_
CRC
Encapsulations – Cont.
Two types of IPSec encapsulations for our applications RFC 3948
MAC Header
MAC Header EtherType: IPv4/IPv6
EtherType: IPv4/IPv6 IP Header

2. VxLAN Encap within ESP within IP Header
Protocol = UDP
UDP Protocol = UDP UDP Header

L4DPort=4500 (ESP)
UDP Header
L4DPort = VxLAN ESP Header
IPsec encap in transport mode with VXLAN Header UDP Header

L4DPort=VxLAN
outer UDP header per [RFC3948] VXLAN Header
• Needed to NAT traversal or per flow
ENCRYPTED
Payload
LB using UDP header

Payload
CRC
ESP Trailer
(NP=UDP)
CRC
Inheritance of Security Policy
• IPsec tunnels for EVPN & other VPNs can be setup at different level of
granularity
• For example, if an IPsec tunnel is needed between a pair of ACs, then
IPsec tunnel attribute is carried along with the EVPN route representing
each AC
• In the absence of such coloring (e.g., sending IPsec tunnel attribute
explicitly along an EVPN route), the route inherits the IPsec tunnel of next
level up (of its parent)
• For example, in the absence of Ipsec tunnel attribute for EVPN route
representing AC, the AC route inherits IPsec tunnel for tenants or peer
Routes to be used with IPsec Tunnel Encap
Attribute
Functionality EVPN IP-VPN MVPN VPLS

(SA per pair of)
PE IPv4/v6 route IPv4/v6 route IPv4/v6 route IPv4/v6 route
Tenant IMET loopback I-PMSI N/A
Subnet of tenant IMET N/A N/A VPLS AD route
IP of tenant RT2 or RT5 VPN-v4/v6 route S-PMSI N/A

(*,G) or (S,G)
MAC of tenant RT2 N/A N/A N/A
NA: Not Applicable

For SA establishment, at the minimum, a PE needs to advertise to
other PEs, its ID, a notification to indicate if this is its initial
contact, key exchange including DH public number and DH group, and
INTERNET DRAFT Secure EVPN October 20, 2018
Nonce. When a single policy is used among all SAs, it is assumed that
Min set this single policy is configured by the management system in all the
PE devices and thus there is no need to signal it. The information
group to use. ID sub-TLV would not be needed in BGP because tunnel
that carries
attribute already need to originator
be signaledID.
(using RFC7296
Section notations)
5 details these are:
sub-
Minimum
TLVs asSet
part of IPsec tunnel TLV in BGP Tunnel Encapsulation
Attribute. ID, [N(INITIAL_CONTACT),] KE, Ni; where
ID payload is defined in section 3.5 of [RFC7296]

3.1.2 Single Policy N (Notify) Payload in section 3.10 of [RFC7296]
KE (Key Exchange) payload in section 3.4 of [RFC7296]
If a single policy Ni (Nonce)
needs to bepayload
signaledin among
section
per3.9 of [RFC7296]
tenant or per
INTERNET
subnet DRAFT Secure
among a set of PEs, then in EVPN October 20, 2018
addition to the information
KE payload
described in section contains
3.1.1, the Association
Security DH public number andneeds
sub-TLV also to
identifies
be which DH
signaled as well. The payload for this sub-TLV is defined in section
match
3.3 of is found inand
[RFC7296] thedetailed
initiator’s list. 5.3.
in section
Single Policy
In order
ID, toSajassi
supportetmulti-policy
[N(INITIAL_CONTACT),SA,
al. a policy
KE, Ni list
Expires Mayis20,
signaled
2019 in [Page 9]
addition to the information described in section 3.1.1. Furthermore,
SA (Security
in order to supportAssociation) payload
multi-DH-groups, a in section
DH group 3.3along
list of [RFC7296]
with its
nonce list are signaled instead of a single DH group and a single
Anonce
single
as SA payload in
described identifies a single IPsec policy. One important
section 3.1.1.
Policy List and DH group List
restriction on the SA Payload is that an standard IKE SA payload can
contain multiple transform;
ID, [N(INITIAL_CONTACT), however,
[SA], [KE], [CONTROLLER-IKE]
[Ni] restricts the
SA payload to only a single transform for each transform type as
described
[SA] in
listsection A.3.1
of IPsec of [CONTROLLER-IKE].
policies (i.e., list of SA payloads)
[KE] list of KE payloads
3.1.3 Policy-list & DH-group-list
EVPN Multi-Homing
Use Cases
Flexible Client Site Connectivity
Distributed vPE
PE
uPE Client
EVPN-
CE VPWS
PE
Flexible WAN/DC Connectivity

DC1
Flexible Backhaul of user traffic to DC DC2
DC2 WAN
EVPN
EVPN WAN
Metro EVPN
EVPN - load-balancing modes
All-Active Single-Active Port-Active
(per flow) (per VLAN) (per port)
PE1 PE2 PE1 PE2 PE1 PE2
V1 V1 V1 V2 V1, V2
CE1 CE2 CE3
Single LAG at the CE Multiple LAGs at the CE Multiple LAGs at the CE

VLAN goes to both PE VLAN active on single PE Port active on single PE
Traffic hashed per flow Traffic hashed per VLAN Traffic hashed per port
Benefits: Bandwidth, Convergence Benefits: Billing, Policing Benefits: Protocol Simplification
EVPN Multi-homing Sequence
Segment Auto-Discovery
ESI Auto-Sensing
Redundancy Group
Membership Auto-Discovery
DF Election & VLAN Carving
ESI Label & MH type Discovery
EVPN Multihoming Sequence
Spine Spine
ESI Auto-Sensing
VTEP1 VTEP2 VTEP3
ES1
LACP PDU
Host LACP info
LACP System ID (MAC) 6 Bytes 0011.0022.0033 Host

Host
LACP System Priority 2 Bytes 0000
Host2
LACP Port Key 2 Bytes 0018 Host1 (MAC2)
(MAC1)
Spine Spine
ESI Auto-Sensing
VTEP1 VTEP2 VTEP3

Auto-Generated ESI (10 Bytes)
Type Priority MAC Port
01 0000 0011.0022.0033 0018

ES1
LACP PDU
Host LACP info

Host
Host2
(MAC1)
Spine Spine
Various Modes
Available
ESI Auto-Sensing
VTEP1 VTEP2 VTEP3

Auto-Generated ESI (10 Bytes)
Type Priority MAC Port
01 0000 0011.0022.0033 0018

ES1
LACP PDU
Host LACP info

Host
Host2
(MAC1)
Spine Spine
ESI Auto-Sensing
VTEP1 VTEP2 VTEP3

Redundancy Group
VTEP1 ES Route (Type4)
RD VTEP1
ESI ES1
ES1
ES Import Target 0011.0022.0033
Based on 6 Bytes Host
MAC Address Host
Host2
Host1 (MAC2)
(MAC1)
Layer-2
Layer-3
Spine Spine
ESI Auto-Sensing
VTEP1 VTEP2 VTEP3

Redundancy Group
VTEP1 ES Route (Type4) VTEP2 ES Route (Type4)
RD VTEP1 RD VTEP2
ESI ES1
ES1 ESI ES1
ES Import Target 0011.0022.0033 ES Import Target 0011.0022.0033
Based on 6 Bytes Host
MAC Address Host
Host2
Host1 (MAC2)
(MAC1)
Layer-2
Layer-3
Spine Spine
Ordered List of
Segment Auto-Discovery Discovered VTEP
(lowest IP)
VTEP Ordered List VTEP Ordered List
Position VTEP Position VTEP

ESI Auto-Sensing
0 VTEP1 0 VTEP1
1 VTEP2 1 VTEP2
VTEP1 VTEP2 VTEP3
Redundancy Group
ES1
DF Election and VLAN Carving
Host
Host
Host2
Host1 (MAC2)
(MAC1)
Layer-2
Layer-3
Spine Spine

ESI Auto-Sensing
0 VTEP1 0 VTEP1
1 VTEP2 1 VTEP2
VTEP1 VTEP2 VTEP3
Redundancy Group
ModuloES1
Operation
DF Election and VLAN Carving VID VID mod 2 Result of Modulo
100 0 Operation to
101 1 determine DF Host
Host
102 0
Host2
VID mod N (N = # of PEs) 103 Host1 1 (MAC2)
(MAC1)
Layer-2
Layer-3
Spine Spine

ESI Auto-Sensing
0 VTEP1 0 VTEP1
1 VTEP2 1 VTEP2
VTEP1 VTEP2 VTEP3
Redundancy Group
DF for VID ModuloES1

Operation
DF for VID
DF Election and VLAN Carving 100, 102 101, 103
VID VID mod 2
100 0
101 1 Host
Host
102 0
Host2
VID mod N (N = # of PEs) 103 Host1 1 (MAC2)
(MAC1)
Layer-2
Layer-3
Spine Spine
ESI Auto-Sensing
VTEP1 VTEP2 VTEP3

Redundancy Group
VTEP1 Per ES Ethernet A-D Route
(Type1)
RD VTEP1 ES1
DF Election and VLAN Carving ESI ES1
Ethtag MAX-ET
Label 0
Host
Host
ESI Label & MH type Discovery Host1

Host2
(MAC2)
(MAC1)
Layer-2
Layer-3
Spine Spine
ESI Auto-Sensing
VTEP1 VTEP2 VTEP3

Redundancy Group
VTEP1 Per ES Ethernet A-D Route
(Type1) VTEP2 Per ES Ethernet A-D Route
(Type1)
RD VTEP1 ES1RD VTEP2
DF Election and VLAN Carving ESI ES-1
ESI ES-1
Ethtag MAX-ET
Ethtag MAX-ET
Label 0
Label 0 Host
Host
ESI lable Extended Community
ESI Label & MH type Discovery Label-1
ESI lable Extended Community Host2
Label-2 (MAC2)
Enhancements to DF Election
• Default VLAN-carving DF election has the following short-comings:
1. When a link/node failure occurs, it shuffles VLANs on healthy links
2. It does not factor in status of a VLAN on an ES (or status of MAC-VRF) in
DF election
PE1
PE1
99, 102, 105

PE2
PE2 CE1
CE1
100, 103, 106 100, 102, 104, MPLS
MPLS
106
PE3
PE3
101, 104, 107 99, 101, 103,

105, 107
Enhancements to DF Election – Cont. (2)
(draft-ietf-bess-evpn-df-election-framework)
• This draft use Highest Random Weight (HRW) function for DF

election
• It factors ESI in the weight function for better load distribution
• Wrand(v, Es, Si) = (1103515245((1103515245*Si+12345)XOR
D(v,Es))+12345)(mod 2^31); D() is CRC-32
• It introduces a new DF Election EC w/ DF-type field and DF-capability
bits
• It use DF-type=0x1 for HRW & one of the capability bits for AC-
Influenced DF election
• For AC-Influenced, once a PE makes order list of PEs for a given VID, it
prunes it based on AC-status (if Ether A-D per EVI route is not present)
Enhancements to DF Election – Cont. (3)
(draft-sajassi-bess-evpn-fast-df-recovery)
• This draft introduces handshake OR time-stamp capabilities for

faster convergence upon a PE recovery (or adding a new PE)
• These two new capability use two of the bits of the capability bits in
DF Election EC
• The procedure for both handshaking & time-stamping are detailed
in the draft
• These capabilities can work with different DF election types
Enhancements to DF Election – Cont. 4)
(draft-sajassi-bess-evpn-per-mcast-flow-df-election)
• In some scenarios, DF election for mcast traffic per VLAN is not

enough
• IPTV where a single VLAN is used for all IPTV traffic
• This draft enhances RFC 7432 DF election procedure to enable
mcast flow DF election
• It uses the same rand() function as HRW with the addition mcast
flow to that function
• affinity (S,G,V, ESI, Address(i)) = (1103515245. ((1103515245*Address(i) +
12345) XOR D(S,G,V,ESI))+12345) (mod 2^31)
• (S,G,V, ESI) = CRC_32(S,G,V, ESI)
Service Part II
Use Cases Spine Spine
Leaf Leaf Leaf Leaf
Server Server
Server
Distributed Anycast
Gateway for Data Center PE PE
PE PE PE
Server
All Active FHRP

Server
N-way FHRP for Client Connectivity
EVPN IRB for DC
Fabric Evolution
EVPN Integrated Routed and Bridge (IRB)
Spine Spine
• Pervasive First-Hop Gateway

Approach
Leaf Leaf Leaf Leaf • All Active with No Hello Protocol
• Common MAC & IP on all PE (Leaf)
VLAN GREEN - IP 10.1.1.1/24 • SVI/BVI GREEN – 10.1.1.1/24
VLAN RED - IP 10.2.2.1/24
• SVI/BVI RED – 10.2.2.1/24
• Anycast Gateway MAC – 2020.0000.AAAA
• Optimizes Inter-Subnet
Forwarding
Server Server Server
• No Hair-Pinning
• Reduces Unknown Unicast
Supports Host Mobility
•
Components
IRB PE
MAC-VRF:
RD1/RT1
IRB I/F
Bridge IP-VRF:
Domain RD10/RT10
(BD1) MPLS or IP Tunnel
Ethernet I/F
(IP Payload)
MAC-VRF: MPLS or IP Tunnel

RD2/RT2 (Ethernet Payload)
IRB I/F
Bridge
Domain
(BD2)
Components – MAC-VRF
EVPN Integrated Routed and Bridge (IRB) IRB PE
MAC-VRF Properties:
• Defined as per RFC7432 MAC-VRF:
RD1/RT1
• Properties: IRB I/F
Bridge IP-VRF:
• RD and RT(s) Domain RD10/RT10
(BD1)
• VLAN-Aware
Ethernet I/F
• N BDs per MAC-VRF
• Individual Eth-Tag per BD MAC-VRF: MPLS or IP Tunnel
(Ethernet Payload)
• VLAN-Base RD2/RT2
IRB I/F
• 1 BD per MAC-VRF Bridge
Domain
• Eth-tag always 0 (BD2)
• Each BD may be linked to an IRB Interface
(Inter-Subnet forwarding)
• MPLS or IP tunnels (Ethernet Payload) Bridge Table is Populated with local
MACs, MACs learned on ACs or via
BGP EVPN
Components – IP-VRF
IP-VRF Properties
• Defined as per RFC7432 MAC-VRF:
RD1/RT1
• Properties: IRB I/F
Bridge IP-VRF:
• RD and RT(s) Domain RD10/RT10
(BD1)
MPLS or IP Tunnel
• Regular Interfaces
Ethernet I/F
(IP Payload)
• IRB Interfaces
MAC-VRF:
• MPLS or IP tunnels (IP Payload) RD2/RT2
IRB I/F
Bridge
Domain
(BD2)
Routing Table is Populated with

ARP Table is Populated with EVPN RT- Interface Routes, SAFI 128 Routes,
2s, ARP and ND Resolution SAFI 1 Routes, EVPN RT-2 and EVPN
RT-5 Routes
Automated Route Distinguisher and Route-Target
VXLAN EVPN to MPLS EVPN Gateway
Automated Route Distinguisher Automated Route-Target (RT)
(RD)
• Simplifies Unique RT assignment on a
• Simplifies Unique RD assignment on a per EVI
per EVI
• Using Format of ASN + EVI
• Using Type-1 Format • 2-byte ASN + 4-byte Value
• 4-byte IP Address + 2-byte Value • BGP ASN + EVI/VNI ID
• RID Loopback IP + internal EVI index
• Internal EVI Index is BD + 32767

Route Distinguisher Route-Target
4-byte 2-byte 2-byte 4-byte
10.65.50.101 34768 65501 30001
Configuration – MAC-VRF
MAC-VRF Configuration :
vlan101
name GREEN (BD1) MAC-VRF:
vn-segment 30001 RD1/RT1
vlan202
Bridge
name RED (BD2) Domain
vn-segment 30002 (BD1)
Ethernet I/F
evpn
vni 30001 l2 MPLS or IP Tunnel
rd auto MAC-VRF:
(Ethernet Payload)
route-target both auto RD2/RT2
vni 30002 l2 Bridge
rd auto Domain
route-target both auto (BD2)
interface nve1
member vni 30001
macst-group 239.1.1.1
member vni 30002
ingress-replication protocol bgp
Example based on NX-OS Configuration with
VXLAN #CLUS BRKMPL-3333 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 71
Configuration – IP-VRF
IP-VRF Configuration :
vrf-context TENANT-YELLOW
vni 50001 MAC-VRF:
rd auto RD1/RT1
address-family ipv4/ipv6 unicast
Bridge IP-VRF:
route-target both auto Domain RD10/RT10
route-target both auto evpn (BD1)
MPLS or IP Tunnel
Ethernet I/F
vlan2501 (IP Payload)
name YELLOW
vn-segment 50001 MAC-VRF:
RD2/RT2
interface vlan2501 Bridge
vrf member TENANT-YELLOW Domain
ip forward (BD2)
ipv6 address use-link-local-only
interface nve1
member vni 50001 associate-vrf

Configuration – IRB Interface
IRB Interface Configuration :
MAC-VRF:
RD1/RT1
fabric forwarding anycast-gateway-mac IRB I/F
Bridge IP-VRF:
2020.0000.AAAA Domain RD10/RT10
(BD1)
MPLS or IP Tunnel
Ethernet I/F
interface vlan101
description GREEN (IP Payload)
vrf member TENANT-YELLOW MPLS or IP Tunnel
ip address 10.1.1.1/24 MAC-VRF:
(Ethernet Payload)
fabric forwarding mode anycast-gateway RD2/RT2
IRB I/F
Bridge
interface vlan202 Domain
description RED (BD2)
vrf member TENANT-YELLOW
ip address 10.2.2.1/24
fabric forwarding mode anycast-gateway

Symmetric and Asymmetric IRB
Symmetric IRB Asymmetric IRB
IP-VRF IP-VRF IP-VRF IP-VRF
MAC- MAC- MAC- MAC- MAC- MAC- MAC- MAC-

VRF VRF VRF VRF VRF VRF VRF VRF
Symmetric IRB
Symmetric IRB
• ARP/MAC state is localized to PE (Leaf)
Spine Spine
• ARP/ND entry only where Host is present
• Allows Better Horizontal Scaling in DC
• Allows more efficient Encapsulation with
MPLS and VXLAN GPE
IP-VRF IP-VRF
• Symmetric Processing
MAC- MAC- MAC- MAC- • Imposition PE, at ingress, performs lookup
VRF VRF VRF VRF
and Bridge/Route operation
• Deposition PE, at egress, performs lookup
and Route/Bridge operation
• Implemented by Cisco
Asymmetric IRB
Asymmetric IRB • ARP/MAC state is on every PE (Leaf)
• ARP/ND entry must be present everywhere
Spine Spine
• Limits Scale
• Requires Ethernet-based Encapsulation
IP-VRF IP-VRF
• Asymmetric Processing
• Imposition PE, at ingress, performs lookup
MAC- MAC- MAC- MAC- and Bridge/Route operation
VRF VRF VRF VRF
• Deposition PE, at egress, performs only
Bridge operation
Symmetric IRB Operation
Control-Plane (Symmetric)
Border Border
Spine Spine
Leaf Leaf Leaf Leaf

MAC-H4/IP-H4 via ARP/ND
Server Server Server Server
Host-1 Host-2 Host-3 Host-4

MAC-H1, IP-H1 MAC-H2, IP-H2 MAC-H3, IP-H3 MAC-H4, IP-H4
BD-1 BD-2 BD-1 BD-2
Control-Plane (Symmetric) MAC/IP Route (RT-2)
RD: VPN RD
MAC: MAC-H4
EVPN Integrated Routed and Bridge (IRB) IP: IP-H4
Label1: MAC VRF Label
Border Border Label2: IP VRF Label
Ext. Community/Attributes:
2 Route Target: MAC-VRF
Route Target: IP-VRF
RR Next Hop: IP-L4
Spine Spine
Leaf Leaf Leaf Leaf


BD-1 BD-2 BD-1 BD-2
RD: VPN RD
MAC: MAC-H4
Add IP-H4 to IP-VRF Table Ext. Community/Attributes:
IP-H4 nh IP-L4 [Label2] 3 2 Route Target: MAC-VRF
RR Next Hop: IP-L4
Spine Spine
Leaf Leaf Leaf Leaf


BD-1 BD-2 BD-1 BD-2
RD: VPN RD
MAC: MAC-H4
Add IP-H4 to IP-VRF Table Ext. Community/Attributes:
IP-H4 nh IP-L4 [Label2] 3 2 Route Target: MAC-VRF
RR Next Hop: IP-L4
Spine Spine
Add MAC-H4 to MAC Table

MAC-H4 nh IP-L4 [Label1]
Leaf Leaf Leaf Leaf
Add IP-H4 to IP-VRF Table MAC-H4/IP-H4 via ARP/ND
IP-H4 nh IP-L4 [Label2] 4
1
Add MAC-H4, IP-H4, to ARP
suppression cache

BD-1 BD-2 BD-1 BD-2
State (Symmetric)
Border Border
Spine Spine
L1 L4
MAC Table: MAC-H4 nh IP-L4 [Label1] MAC Table: MAC-H1 nh IP-L1 [Label1]
IP-VRF Table: IP-H4 nh IP-L4 [Label2] Leaf Leaf Leaf Leaf

IP-VRF Table: IP-H1 nh IP-L1 [Label2]
ARP Suppress: IP-H4 nh MAC-H4 ARP Suppress: IP-H1 nh MAC-H1

BD-1 BD-2 BD-1 BD-2
Forwarding (Symmetric)
Border Border
1. Host-1 sends an ARP request for its First-Hop

Gateway, with Anycast Gateway IP Address as the
target IP Address (SVI1’s IP address: IP-SVI1)
Spine Spine
2. ARP request is received at L1 and sends it to
Control-Plane
3. L1 sends ARP reply with MAC-BVI1 as MAC address
Leaf Leaf Leaf Leaf
1
3

BD-1 BD-2 BD-1 BD-2
Host-1 – ARP Cache
IP-SVI1 nh MAC-BVI1 #CLUS BRKMPL-3333 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 83
Border Border
6. Host-1 sends data packet with MAC-SVI1 as the 6. L1 encapsulates packet using Label2 as the VPN label
destination MAC and IP-H4 as the destination IP and Next-Hop as IP-L4. Route
address.
Spine 7 Spine
7. Packet is forwarded towards L4
5. L1 receives the packet, since the destination MAC is
equal to the SVI MAC. IP lookup is performed for IP-
H4. Lookup results with IP-L4 as next hop and Label2
as VPN label
Leaf 6 Leaf Leaf Leaf

BD-1 BD-2 BD-1 BD-2
Border Border
8. L4 receives encapsulated packet, uses Label2 to

determine the VPN table to do the inner IP lookup with
resulting egress SVI to forward the packet
Spine Spine
9. Host-H4 receives data packet with MAC-H4 as the
destination MAC and consumes it
Leaf Leaf Leaf Leaf

8

BD-1 BD-2 BD-1 BD-2
EVPN Multicast
(Bridged & Routed)
Use Cases RCVR
RCVR
RCVR PE WAN
Spine Spine RCVR
DC
Leaf Leaf Leaf Leaf
SRC
Seamless Integration
RCVR RCVR between MVPNs
SRC
Server
East-West Multicast Routing PE
in Data Center
L2VPN
Service PE
PE
RCVR
Server
SRC
Server
Improved Bridged Multicast

for L2VPN Services
Different Subnet Forwarding – Router on-a-Stick
Traditional Forwarding in Overlays
S
SRC
TOR1 TOR2
SRC-10 Leaf Leaf
224.10.10.10 10.1.1.254
10.1.1.100 10.3.3.254
Spine
VLAN 101 (Green)
Spine
R
TOR3 TOR4 RCVR
RCVR-21
Leaf Leaf
10.3.3.21
VXLAN EVPN
VLAN 101 (Green) VLAN 101 (Green)
Different Subnet Forwarding – Router on-a-Stick
Traditional Forwarding in Overlays
• Multiple Copy in Core – Treated as BUM
S
SRC
• Different Subnet possible – RPF Challenges
TOR1 TOR2
• Pruning on Local Interface
SRC-10 Leaf Leaf
224.10.10.10 • Tunnel is NOT pruned if interest Receiver exists behind
10.1.1.254
10.1.1.100 one Remote VTEP 10.3.3.254
Spine
VLAN 101 (Green)
Spine
R
TOR3 TOR4 RCVR
RCVR-21
Leaf Leaf
10.3.3.21
VXLAN EVPN
VLAN 101 (Green) VLAN 101 (Green)
EVPN and Multicast Forwarding
Spine Spine
• Efficient Forwarding of Multicast

• Distributed Designated Router (DR)
Leaf Leaf Leaf Leaf • Single Copy in Core
• Flexible Rendezvous-Point
VLAN GREEN - IP 10.1.1.1/24
VLAN BLUE - IP 10.3.3.1/24
• SP, Financial and Media Demands
• Multicast for Signalization vs. Heavy
Multicast Forwarding
• Selective Multicast Tunnels
Functional Components
Spine Spine
Site-External DCI
(IP Routing and Increased
MTU Support)
•
VXLAN EVPN
•
VTEP VTEP VTEP VTEP

•
DR DR DR DR
Baremetal Baremetal Baremetal Baremetal Baremetal
SRC-10 RCVR-10 RCVR-20 RCVR-30 RCVR-11

224.10.10.10 10.1.1.10 10.2.2.20 10.3.3.30 10.1.1.11
10.1.1.100
Functional Components
Tenant Routed Multicast Spine Spine
Site-External DCI
(IP Routing and Increased
• MTU Support)
•
• VXLAN EVPN
•
VTEP VTEP VTEP VTEP
DR DR DR DR
•
Baremetal Baremetal Baremetal • Baremetal Baremetal
•
SRC-10 RCVR-10 RCVR-20 RCVR-30 RCVR-11
224.10.10.10 10.1.1.10 10.2.2.20 • 10.3.3.30 10.1.1.11
10.1.1.100 •
•
•
DC TOR with BGP Control-Plane
Data Center Fabric Multicast
feature ngmvpn
router bgp 65101

router-id 10.65.10.24 TOR1 TOR2
neighbor 10.65.10.111
Leaf Leaf
remote-as 65101
update-source loopback0
address-family ipv4 mvpn
send-community
Spine
send-community extended
address-family l2vpn evpn
send-community
AS65101
send-community extended
Spine
vrf TENANT1
address-family ipv4 unicast
advertise l2vpn evpn TOR3 TOR4
Leaf Leaf
VXLAN EVPN
Layer-3 Extension with Multicast
interface Vlan2001
no shutdown
vrf member TENANT1
ip forward TOR1 TOR2
*ip pim sparse-mode
Leaf Leaf
vrf context TENANT1

vni 50001
rd auto
Spine
route-target both auto
route-target both auto mvpn
AS65101
route-target both auto evpn
Spine
interface nve1
source-interface loopback1 TOR3 TOR4
host-reachability protocol bgp
Leaf Leaf
member vni 50001 associate-vrf
mcast-group 239.3.3.3
*PIM under Interface is to initialize Multicast.

VXLAN EVPN
There is no PIM Peering present in the Overlay
Integrated Anycast RP (RP Everywhere)
ip multicast overlay-spt-only
vrf context TENANT1

vni 50001 TOR1 TOR2
ip pim rp-address 10.50.0.1
rd auto
Leaf RP Leaf RP
route-target both auto
route-target both auto mvpn
Spine
route-target both auto evpn
interface loopback51
AS65101
vrf member TENANT1
Spine
ip address 10.50.0.1/32
ip pim sparse-mode
TOR3 TOR4
Leaf RP Leaf RP
VXLAN EVPN
Integrated Route and Bridge (IRB) with Multicast
interface Vlan101
no shutdown
vrf member TENANT1
ip address 192.168.10.1/24 TOR1 TOR2
*ip pim sparse-mode
Leaf Leaf
fabric forwarding mode anycast-gateway
Spine
interface Vlan202
no shutdown AS65101
vrf member TENANT1 Spine
ip address 192.168.20.1/24
*ip pim sparse-mode
fabric forwarding mode anycast-gateway TOR3 TOR4
Leaf Leaf
*PIM under Interface is to initialize Multicast.

There is no PIM Peering present in the Overlay
VXLAN EVPN
EVPN DC
Interconnect
What we are going to Discuss in this Section
EVPN DC Interconnect
• Intra Data Center Deployment
• Distributed IP Anycast Gateway
DC#2 • Layer-2/Layer-3 with Host Mobility
• Multi-Tenancy
Server Server
• Inter Data Center Connection
• Hierarchical
DC#1 • Optimized Layer-2 Extension (L2VPN)
• Layer-3 Multi-Tenancy (L3VPN)
• IP or MPLS Encapsulation
Server Server
• Layer-2 and Layer-3 Multi-Tenancy

• Stretched IP Subnet
• Host Mobility
• East-West Communication between DCs
Scenario Overview
iBGP
Core
AS65501 AS65501
DC GW DC GW DC GW DC GW
MPLS EVPN
eBGP eBGP

VXLAN EVPN VXLAN EVPN
iBGP iBGP
AS65101 AS65202
Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf
Scenario Overview
iBGP
Core
RID: 10.65.50.102 RID: 10.65.50.101 RID: 10.65.50.202 RID: 10.65.50.202
AS65501 AS65501
MPLS EVPN
eBGP eBGP
RID: 10.65.10.112 RID: 10.65.10.111 VXLAN EVPN VXLAN EVPN
iBGP iBGP
AS65101 AS65202
RID: 10.65.10.21
RID: 10.65.10.22
RID: 10.65.10.23
RID: 10.65.10.24 #CLUS BRKMPL-3333 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 108
DC GW with BGP Control-Plane
Configuration for EVPN Overlay Control-Plane
router bgpiBGP
65501
bgp router-id 10.65.50.101 | Router ID
Core
RID: 10.65.50.102 RID: 10.65.50.101 RID: 10.65.50.202 RID: 10.65.50.202
neighbor 10.65.50.201 | Remote DC GW
AS65501 remote DCI remote-as 65501 AS65501
| Remote DC ASN
DC GW DC GW update-source Loopback0 DC GW | Router
DC GW ID
address-family
MPLS EVPN l2vpn evpn
import re-originate stitching-rt
advertise l2vpn evpn re-originated
eBGP eBGP
neighbor 10.65.10.111 | Spine
Spine Spine remote-as 65101 Spine | Spine Spine ASN (Local DC)

ebgp-multihop 10
update-source Loopback1 | VTEP ID
iBGP iBGP
AS65101 address-family l2vpn evpn AS65202
encapsulation-type vxlan | Fabric Side Encap
import stitching-rt reoriginate
advertise l2vpn evpn re-originated stitching-rt
multipath | EVPN Multi-Path
RID: 10.65.10.21 route-policy pass-all in
RID: 10.65.10.22 route-policy pass-all out
RID: 10.65.10.23
Layer-2 DC GW - EVPN MPLS to EVPN VXLAN
Two Different Configuration Mode
ESI-based Multi-Homing* Anycast-based Multi-Homing*
• TOR uses Multi-Homing to DC GW • TOR uses vPC
• Using all-active Ethernet Segment • Using Anycast VTEP approach
• DC GW uses Multi-Homing to TOR • DC GW uses Anycast IP

• Using all-active Ethernet Segment • Using Anycast VTEP approach
• DF election happen on all participating • DF election happens within Anycast

Nodes (EVPN-based) complex (not EVPN-based)
• Traffic between TOR and DC GW is • Traffic between TOR and DC GW is
balanced by MAC Multi-Pathing (MAC balanced by Underlay Routing
ECMP) Protocol (IP ECMP)
*Check Device Capabilities before selecting Multi-Homing mode

Scenario Overview – ESI-Based Multi-Homing
ESI-Based Multi-Homing
iBGP
Core
RID: 10.65.50.102 RID: 10.65.50.101 RID: 10.65.50.202 RID: 10.65.50.202
AS65501 AS65501
MPLS EVPN
eBGP eBGP
iBGP iBGP
AS65101 AS65202
RID: 10.65.10.21
RID: 10.65.10.22
RID: 10.65.10.23
Underlay Configuration
ESI-Based Multi-Homing
interface iBGP
loopback0
ip address 10.65.50.101/32 | Router ID
Core
interface loopback1
RID: 10.65.50.102 RID: 10.65.50.101 RID: 10.65.50.202 RID: 10.65.50.202
ip address 10.65.10.1/32 | VTEP ID
AS65501 AS65501
DC GW DC GW router isis MPLS DC GW DC GW
router-id
MPLS10.65.50.101
EVPN | Router ID
segment-routing mpls | Core Side Encap
interface interface TenGigE 0/0/0/1 | Link to Core
eBGP interface interface TenGigE 0/0/0/2 | Link to Core eBGP
interface Loopback10 | Router ID

router isis VXLAN
router-id 10.65.10.1 | VTEP ID
iBGP iBGP
AS65101 interface interface TenGigE 0/0/1/1 AS65202
| Link to Spine
interface interface TenGigE 0/0/1/2 | Link to Spine
interface Loopback1 | VTEP ID
interface Loopback2 | Anycast VTEP ID
RID: 10.65.10.21
RID: 10.65.10.22
RID: 10.65.10.23
VXLAN VTEP & ESI-based Multi-Homing
Core
ESI Type0: 65.50.10.65.50.10.10.10.10 ESI Type0: 65.50.20.65.50.20.20.20.20

ES-Import RT:
DC GWaaaa.bbbb.cccc
DC GW ES-ImportDCRT:
GW cccc.bbbb.aaaa
DC GW
VXLAN VTEP & ESI-based Multi-Homing
interface nve 1
source-interface Loopback1 | VTEP ID
member Core
vni 30001 | Layer-2 VNI
mcast-group 239.1.1.1 | BUM Replication
ESI Type0: 65.50.10.65.50.10.10.10.10 ESI Type0: 65.50.20.65.50.20.20.20.20
ES-Import RT:
DC GWaaaa.bbbb.cccc
DC GW evpn ES-ImportDCRT:
GW cccc.bbbb.aaaa
DC GW
interface nve1
ethernet-segment
identifier type 0 65.50.10.65.50.10.10.10.10 | ESI Type0
bgp route-target aaaa.bbbb.cccc | ES-Import RT
Scenario Overview – Anycast VTEP
Anycast VTEP
iBGP
Core
RID: 10.65.50.102 RID: 10.65.50.101 RID: 10.65.50.202 RID: 10.65.50.202
AnycastAS65501
VTEP: 10.65.10.12 AS65501
MPLS EVPN
eBGP eBGP
iBGP iBGP
AS65101 AS65202
RID: 10.65.10.21
RID: 10.65.10.22
RID: 10.65.10.23
Underlay Configuration
Anycast VTEP
interface iBGP
loopback0
ip address 10.65.50.101/32 | Router ID
Core
interface loopback1
RID: 10.65.50.102 RID: 10.65.50.101 RID: 10.65.50.202 RID: 10.65.50.202
ip address 10.65.10.1/32 | VTEP ID
AnycastAS65501
VTEP: 10.65.10.12 AS65501
DC GW DC GW interface loopback2 DC GW DC GW
ip address 10.65.10.12/32
MPLS EVPN | Anycast VTEP ID
router isis MPLS

eBGP router-id 10.65.50.101 | Router ID eBGP
segment-routing mpls | Core Side Encap
Spine Spine interface interface TenGigE 0/0/0/1Spine | LinkSpineto Core
RID: 10.65.10.112 RID: 10.65.10.111 VXLAN EVPN
interface VXLAN EVPN
interface TenGigE 0/0/0/2 | Link to Core
interface Loopback10 | Router ID
iBGP iBGP
AS65101 AS65202
router isis VXLAN
router-id 10.65.10.1 | VTEP ID
RID: 10.65.10.21 interface Loopback1 | VTEP ID
RID: 10.65.10.22 interface Loopback2 | Anycast VTEP ID
RID: 10.65.10.23
VXLAN VTEP & Anycast-based Multi-Homing
Core
Anycast VTEP: 10.65.10.12 Anycast VTEP: 10.65.10.22

VXLAN VTEP & Anycast-based Multi-Homing
Core
Anycast VTEP: 10.65.10.12 Anycast VTEP: 10.65.10.22

interface nve 1
source-interface Loopback1 | VTEP ID
anycast source-interface loopback2 | Anycast VTEP ID
Spine Spine
member vni 30001 Spine
| Layer-2
Spine
VNI
mcast-group 239.1.1.1 | BUM Replication
Layer-2 Service (EVI)
Core
EVI: 40001
MPLS EVPN

L2VNI: 30001 VXLAN EVPN VXLAN EVPN
L2VNI: 50001
Layer-2 Service (EVI)
Core
EVI: 40001
l2vpn
DC GW DC GW bridge group 2001 DC GW | Bridge
DC GW Group
MPLS EVPN 2001

bridge-domain | Bridge Domain
evi 40001 | EVI mapping
member vni 30001 | VNI mapping

L2VNI: 30001 VXLAN EVPN VXLAN EVPN
L2VNI: 50001
Route-Target Stitching
iBGP
Route-Target Core Route-Target

AS65501 EVI: 40001
AS65501 EVI: 40001
eBGP eBGP
iBGP iBGP
Route-Target Route-Target
AS65101 L2VNI: 30001 AS65202 L2VNI: 50001
Route-Target Stitching
iBGP
Route-Target Core Route-Target

AS65501 EVI: 40001
AS65501 EVI: 40001
evpn
evi 40001 | EVPN Virtual Instance
bgp
eBGP route-target import 65501:40001 | DCI RT eBGP
route-target export 65501:40001 | DCI RT
Spine Spine
route-target import 65101:30001 stitching
Spine
| VXLAN RT
Spine
route-target export 65101:30001 stitching | VXLAN RT

iBGP iBGP
Route-Target Route-Target
AS65101 L2VNI: 30001 AS65202 L2VNI: 50001
Services - Part III
Service Provider Network - Simplification Journey
Compass
Unified MPLS EPN 5.0 Metro Fabric
Provisioning NETCONF NETCONF
YANG YANG
Programmability
L2/L3VPN Services LDP BGP LDP BGP BGP
Inter-Domain CP BGP-LU BGP-LU

FRR or TE RSVP
IGP with SR
LDP IGP with SR
Intra-Domain CP
IGP
From Mac Bridging to Mac Routing
Common BGP Control Plane
EVPN, VPNv4/6 Overlay
Evolution:
Underlay
Segment Routing (SR: MPLS, SRv6) SR, VXLAN SR, VXLAN
Data Center Network

Service Provider Network overlap
Leaf
VM
PE1 DCI1
Spine Spine
Leaf
VM
A1 Access WAN/Core
Leaf
PE2 DCI2 VM
BGP: VPNv4/6 VPLS Overlay

Existing Solution: Fabric-Path (Trill)
LDP: VPLS, PW Fabric-Path
MPLS: LDP, RSVP-TE MPLS, L2 L2, IP Underlay
EVPN - Ethernet VPN
• Concepts are same!!! Pick your side!
Pick your side!

SP1 SP2
PE2 PE4
CE1
PE1 PE3
L1 L2 L3 L4
C1 C2
VM VM VM VM
EVPN &
VPNv4/6
Interconnect
EVPN and VPNv4/6 Interconnect
• DCI/BL provides EVPN to VPNv4/6 stitching
• DCI/BL participates in L3 Routing not in L2 Bridging
• DCI/BL is mandatory, because of summarization!!!
BGP - L3VPN VPNv4/6 BGP - EVPN
A3 P3 SP2 L2
CE1 A1 Access ABR1 Core DCI/BL1

CO CE2
A2 P2 SP1 L1
• DCI/BL provides EVPN to VPNv4/6 stitching Interconnect
• DCI/BL participates in L3 Routing not in L2 Bridging
• DCI/BL is mandatory, because of summarization!!!
prefix-CE2/24 RT5 prefix-CE2/24
prefix-CE1/24 RT5 prefix-CE1/24
RT2 MAC/IP = CE1/32

X
A3 P3 SP2 L2

CO CE2
A2 P2 SP1 L1
prefix-CE2/24 RT: VRF A RT5 prefix-CE2/24 RT: VRF A Stitching
RT2 MAC/IP = CE1/32 RT: VRF A Stitching

X
DCI/BL
VRF A
RD DCI:0
RT import/export: VRF A Stitching
RT import/export: VRF A
EVPN to VPNv4/6 Re-Advertise
RT2 MAC/IP = CE1/32 RT: VRF A Stitching

1. Import: RT: VRF A Stitching DCI/BL - BGP Configuration X
router bgp 1
address-family l2vpn evpn DCI/BL
import stitching-rt re-originate
advertise vpnv4 unicast re-originated stitching-rt
3. Filter RT2 => /32 Router ! VRF A
address-family vpnv4 unicast
RD DCI:0
route-policy rt2-filter out RT import/export: VRF A Stitching
advertise vpnv4 unicast re-originated RT import/export: VRF A
!
2. Advertise to vpnv4: VRF A
VPNv4/6 to EVPN Re-Advertise
2. Advertise to EVPN: RT: VRF A Stitching DCI/BL - BGP Configuration

router bgp 1
address-family l2vpn evpn DCI/BL
import stitching-rt re-originate
advertise vpnv4 unicast re-originated stitching-rt
! VRF A
RD DCI:0
route-policy rt2-filter out RT import/export: VRF A Stitching
advertise vpnv4 unicast re-originated RT import/export: VRF A
1. Import: VRF A
!
EVPN and VPNv4 InterConnect
Anycast IRB 192.168.2.1/24
RR101 Emulates R2
RR103 RR104
IRB
R39
RR101 R2
LACP
VPNv4: 9.9.9.101/24
H2: 192.168.2.20/24 H2 IRB
R38 R35 R28 R52
IRB
LACP R37 R34 R26 R51
H1: 192.168.1.10/24 H1
IRB
R36
Anycast IRB 192.168.1.1/24

BGP - EVPN BGP - L3VPN VPNv4/6
R36: BGP Configuration - RT-5
router bgp 1
bgp router-id 3.3.3.36
!
!
neighbor-group rr
remote-as 1
update-source Loopback0
advertise vpnv4 unicast
!
vrf a
RT-5
rd auto
additional-paths receive
maximum-paths ibgp 2
!
R36: RT-5 Prefix
R36#show bgp vpnv4 unicast
Status codes: s suppressed, d damped, h history, * valid,>best

i - internal, r RIB-failure, S stale, N Nexthop-discard
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 3.3.3.36:0 (default for vrf a)
*>192.168.1.0/24 0.0.0.0 0 32768 ?
* i 3.3.3.37 0 100 0 ?
*>i192.168.1.10/32 3.3.3.37 100 0 i
*>i192.168.2.0/24 3.3.3.38 0 100 0 ?
* i 3.3.3.39 0 100 0 ?
*>i192.168.2.20/32 3.3.3.38 100 0 i
* i 3.3.3.39 100 0 i
Advertised prefix RT-5
R36: RT-5 Route
R36#show bgp l2vpn evpn rd 3.3.3.37:0 [5][0][24][192.168.1.0]/80
Tue Oct 16 03:35:06.480 UTC
BGP routing table entry for [5][0][24][192.168.1.0]/80, Route Distinguisher: 3.3.3.37:0
Versions:
Process bRIB/RIB SendTblVer
Speaker 84912 84912
Last Modified: Oct 16 03:23:18.399 for 00:11:48
Paths: (2 available, best #1) RT-5 prefi VRF A R37 RD
Not advertised to any peer x
Path #1: Received by speaker 0
Not advertised to any peer
Local
3.3.3.37 (metric 30) from 3.3.3.103 (3.3.3.37) VRF Agg label
Received Label 64008
Origin incomplete, metric 0, localpref 100, valid, internal, best, group-best, import-candidate, not-in-vrf
Received Path ID 0, Local Path ID 1, version 84912
Extended community: Flags 0x6: RT:100:100
Originator: 3.3.3.37, Cluster list: 3.3.3.103
EVPN ESI: 0000.0000.0000.0000.0000, Gateway Address : 0.0.0.0 VRF A Route-Target
Local
3.3.3.37 (metric 30) from 3.3.3.104 (3.3.3.37) VRF Agg label
Origin incomplete, metric 0, localpref 100, valid, internal, not-in-vrf
Extended community: Flags 0x6: RT:100:100
Originator: 3.3.3.37, Cluster list: 3.3.3.104 VRF A Route-Target
EVPN ESI: 0000.0000.0000.0000.0000, Gateway Address : 0.0.0.0
RP/0/RP0/CPU0:R36#
R36: VRF A - Routing Table
R36#show route vrf a
C 192.168.1.0/24 is directly connected, 04:55:09, BVI100

L 192.168.1.1/32 is directly connected, 04:55:09, BVI100
B 192.168.1.10/32 [200/0] via 3.3.3.37 (nexthop in vrf default)
B 192.168.2.0/24 [200/0] via 3.3.3.38 (nexthop in vrf default), 00:40:26
[200/0] via 3.3.3.39 (nexthop in vrf default), 00:40:26
RP/0/RP0/CPU0:R36
DCI R26: VRF Configuration
vrf a
import route-target
100:100 stitching VRF a RT - CO
999:100
!
VRF a RT - Core
export route-target
100:100 stitching
999:100
!
!
!
R26#show route vrf a connected

Wed Oct 17 03:28:28.244 UTC
% No matching routes found

No Local L3 Interface in VRF 100
DCI R26: BGP Configuration
R26#show run router bgp 1 router bgp 1
Mon Oct 15 21:01:43.943 UTC !
router bgp 1 neighbor 1.1.1.101
bgp router-id 1.1.1.26 use neighbor-group rr-core
ibgp policy out enforce-modifications RR Next-Hop-change !
address-family vpnv4 unicast neighbor 3.3.3.103
! use neighbor-group rr
address-family l2vpn evpn !
! neighbor 3.3.3.104
neighbor-group rr use neighbor-group rr
remote-as 1 !
update-source Loopback0 vrf a
address-family l2vpn evpn EVPN AF - CO rd auto
import stitching-rt re-originate address-family ipv4 unicast
route-policy vpnv4-filter in additional-paths receive
route-policy vpnv4-community-set out maximum-paths ibgp 2
advertise vpnv4 unicast re-originated stitching-rt !
! !
! !
neighbor-group rr-core
remote-as 1
update-source Loopback0
route-policy evpn-filter in
route-reflector-client BGP VRF
route-policy rt2-filter out
advertise vpnv4 unicast re-originated
!
DCI R26: BGP EVPN/VPNv4 Route Leaking Configuration
address-family l2vpn evpn Filter routes with VPNv4 route-policy rt2-filter
import stitching-rt re-originate community if destination in (0.0.0.0/0 ge 32) then
route-policy vpnv4-filter in drop
route-policy vpnv4-community-set out else
advertise vpnv4 unicast re-originated stitching-rt Set VPNv4 community set community evpn
! endif
end-policy
address-family vpnv4 unicast !
import re-originate stitching-rt Filter routes with EVPN
route-policy evpn-filter in community route-policy evpn-filter
route-reflector-client if community matches-any evpn then
route-policy rt2-filter out Filter /32 routes and drop
advertise vpnv4 unicast re-originated set EVPN community else
! pass
endif
end-policy
route-policy vpnv4-community-set
set community vpnv4
community-set evpn end-policy
1:111
end-set
! route-policy vpnv4-filter
community-set vpnv4 if community matches-any vpnv4 then
1:222 drop
end-set else
! pass
endif
end-policy
!
DCI R26: BGP EVPN/VPNv4 Route Leaking
import stitching-rt re-originate 1. Import RT 100:100 and re-
route-policy vpnv4-filter in originate with RT 999:100
route-policy vpnv4-community-set out
advertise vpnv4 unicast re-originated stitching-rt 4. Advertise re-originated
! routes with RT 100:100
address-family vpnv4 unicast 3. Import RT 999:100 and re-
import re-originate stitching-rt originate with RT 100:100
route-policy evpn-filter in
route-reflector-client
route-policy rt2-filter out 2. Advertise re-originated
advertise vpnv4 unicast re-originated routes with RT 999:100
!
vrf a
import route-target
100:100 stitching VRF a RT - CO
999:100
!
VRF a RT - Core
export route-target
100:100 stitching
999:100
!
!
!
Anycast IRB 192.168.2.1/24
RR101 Emulates R2
RR103 RR104
IRB
R39
RR101 R2
LACP
VPNv4: 9.9.9.101/24
H2: 192.168.2.20/24 H2 IRB
R38 R35 R28 R52
IRB
H1: 192.168.1.10/24 H1
IRB
R36
Anycast IRB 192.168.1.1/24

BGP - EVPN BGP - L3VPN VPNv4/6
Anycast IRB 192.168.2.1/24
RR101 Emulates R2
RR103 RR104
IRB
RR101 R2
LACP R39
VPNv4: 9.9.9.101/24
H2: 192.168.2.20/24 H2 IRB
R38 R35 R28 R52

R2 VPNv4 - Core Route
IRB
H1: 192.168.1.10/24 H1 R26#show route vrf a

IRB
R36 B 9.9.9.0/24 [200/0] via 1.1.1.101 (nexthop in vrf default), 1d07h
Anycast IRB 192.168.1.1/24 [200/0] via 3.3.3.37 (nexthop in vrf default), 15:12:55
H1 Host-Address B BGP[200/0]
192.168.2.0/24 - EVPNvia 3.3.3.38 (nexthopBGP - L3VPN
in vrf default),VPNv4/6
1d07h
[200/0] via 3.3.3.39 (nexthop in vrf default), 1d07h
B 192.168.2.20/32 [200/0] via 3.3.3.38 (nexthop in vrf default), 1d07h
H2 Host-Address [200/0] via 3.3.3.39 (nexthop in vrf default), 1d07h
Anycast IRB 192.168.2.1/24
RR101 Emulates R2
RR103 RR104
IRB
RR101 R2
LACP R39
VPNv4: 9.9.9.101/24
H2: 192.168.2.20/24 H2 IRB
R38 R35 R28 R52
IRB
H1: 192.168.1.10/24 H1
IRB
R36
r2#show route vrf a
Anycast IRB 192.168.1.1/24
C 9.9.9.0/24 is directly connected, 2w0d, Loopback9
L 9.9.9.101/32 is directly connected, 2w0d, Loopback9
H1 - Prefix B 192.168.1.0/24 [200/0] via 1.1.1.26 (nexthop in vrf default), 15:16:22
H2 - Prefix [200/0] via 1.1.1.28 (nexthop in vrf default), 1d07h
Anycast IRB 192.168.2.1/24
RR101 Emulates R2
RR103 RR104
IRB
RR101 R2
LACP R39
VPNv4: 9.9.9.101/24
H2: 192.168.2.20/24 H2 IRB
R38 R35 R28 R52
IRB
LACP R37 R34 R26 R51 R2 - Prefix
H1: 192.168.1.10/24 H1
IRB R36#show route vrf a
R36 B 9.9.9.0/24 [200/0] via 1.1.1.26 (nexthop in vrf default), 16:53:11

Anycast IRB 192.168.1.1/24
C 192.168.1.0/24 is directly connected, 1d12h, BVI100
L 192.168.1.1/32 is directly connected, 1d12h, BVI100
R2 VPNv4 - Core Route
[200/0] via 3.3.3.37 (nexthop in vrf default), 16:45:48 H1 Host-Address
[200/0] via 3.3.3.39 (nexthop in vrf default), 1d07h H2 Host-Address
r2#show route vrf a
C 9.9.9.0/24 is directly connected, 2w0d, Loopback9

L 9.9.9.101/32 is directly connected, 2w0d, Loopback9
B 192.168.1.0/24 [200/0] via 1.1.1.26 (nexthop in vrf default), 15:16:22 H1 - Prefix
H2 - Prefix

[200/0] via 1.1.1.28 (nexthop in vrf default), 16:53:11 R2 - Prefix
C 192.168.1.0/24 is directly connected, 1d12h, BVI100
L 192.168.1.1/32 is directly connected, 1d12h, BVI100
Low-Latency Traffic Flex-Algo128: Delay
L2/L3VPN - EVPN (SLA Low-Latency -
BGP Community)
A3 P3 SP2 L2
$ $ $ $

CO CE2
A2 P2 SP1 L1
L3VPN - VPNv4 (SLA Low-Latency - BGP Community) L2/L3VPN - EVPN (SLA Low-Latency -
BGP Community)
A3 P3 SP2 L2
$ $ $ $

CO CE2
A2 P2 SP1 L1
L3VPN - VPNv4 (SLA Low-Latency - BGP Community) L2/L3VPN - EVPN (SLA Low-Latency -
BGP Community)
ODN A3 P3 SP2 L2
$ $ $ $

CO CE2
A2 P2 SP1 L1
Flex-Algo 128: ABR1 Flex-Algo 128: BL1

Label L1 or L2 (MultiPath)
Flex-Algo 128: BL1 Service
Stack Service
Service
EVPN-VPWS
Multihomed
Service
EVPN-VPWS
• Benefits of EVPN applied to point-to-point
services
• No signaling of PWs. Instead signals MP2P LSPs PE2 PE4
instead (ala L3VPN)
• All-active CE multi-homing (per-flow LB) CE1 MPLS CE2
• Single-active CE multi-homing (per-service LB) PE1 PE3
• Relies on a sub-set of EVPN routes to

advertise Ethernet Segment and AC
reachability
• PE discovery & signaling via a single protocol –
BGP
• Per-EVI Ethernet Auto-Discovery route
H-EVPN &
EVPN Headend (PWHE + EVPN)
PWHE PWHE
A1 PE1 PE3 A3
Multi/Single-Homed Multi/Single-Homed
Multi/Single-Homed
CE Single-Active All/Single-Active CE
All/Single-Active
EVPN EVPN-VPWS
EVPN-VPWS
VPNv4/6
A2 PE2 PE4 A4
PWHE PWHE
EVPN-VPWS - Testbed
Startup Sequence is almost identical with EVPN except:
RT3 and RT2 are not required
RR103 RR104
LACP R39
H2
R38 R35
R37 R34
H1 R36
Config: EVPN-VPWS
R36 R38/R39
l2vpn l2vpn
xconnect group 500 xconnect group 500
p2p 500 p2p 500
interface Bundle-Ether100 interface Bundle-Ether100
neighbor evpn evi 500 target 3839 source 36 neighbor evpn evi 500 target 36 source 3839
! !
! !
! !
Leaf/Access R36: L2vpn xconnect status & Data
Plane verification
R36#show l2vpn xconnect
Legend: ST = State, UP = Up, DN = Down, AD = Admin Down, UR = Unresolved,
SB = Standby, SR = Standby Ready, (PP) = Partially Programmed
XConnect Segment 1 Segment 2

Group Name ST Description ST Description ST
------------------------ ----------------------------- -----------------------------
500 500 UP BE100 UP EVPN 500,3839,68106 UP
----------------------------------------------------------------------------------------
R36#show mpls forwarding labels 68106

Local Outgoing Prefix Outgoing Next Hop Bytes
Label Label or ID Interface Switched
------ ----------- ------------------ ------------ --------------- ------------
68106 68107 EVPN:500 3.3.3.38 0
68107 EVPN:500 3.3.3.39 0
Leaf/Access R36: RT-1 Per EVI Ethernet Auto-Discovery
R36#show bgp l2vpn evpn rd 3.3.3.36:500 [1][0038.3900.0000.0000.1100][3839]/120
BGP routing table entry for [1][0038.3900.0000.0000.1100][3839]/120, Route Distinguisher: 3.3.3.36:500

Versions:
Process bRIB/RIB SendTblVer
RT- AC-
316 1 ESI R38/R39
Speaker 316 ID
Last Modified: Jan 27 08:24:37.527 for 00:01:42
Paths: (2 available, best #1)
Local
3.3.3.38 (metric 30) from 3.3.3.103 (3.3.3.38)
Origin IGP, localpref 100, valid, internal, best, group-best, import-candidate, imported, rib-install
Extended community: RT:1:500
Source AFI: L2VPN EVPN, Source VRF: default, Source Route Distinguisher: 3.3.3.38:500
Local
3.3.3.39 (metric 30) from 3.3.3.103 (3.3.3.39)
Origin IGP, localpref 100, valid, internal, import-candidate, imported, rib-install
Leaf/Access R36: EVPN-VPWS Instance
R36#show evpn evi vpn-id 500 detail
VPN-ID Encap Bridge Domain Type

---------- ------ ---------------------------- -------------------
500 MPLS VPWS:500 VPWS (vlan-unaware)
Stitching: Regular
Unicast Label : 0
Multicast Label: 0
Flow Label: N EVPN-VPWS
Control-Word: Enabled
Forward-class: 0
• No RT2 – MAC
Advertise MACs: No • No RT3 - BUM
Advertise BVI MACs: No
Aliasing: Enabled
UUF: Enabled
Re-origination: Enabled
Multicast source connected: No
Statistics:
Packets Sent Received
Total : 0 0
Unicast : 0 0
BUM : 0 0
Bytes Sent Received
Total : 0 0
Unicast : 0 0
BUM : 0 0
RD Config: none
RD Auto : (auto) 3.3.3.36:500
RT Auto : 1:500
Route Targets in Use Type
------------------------------ ---------------------
1:500 Import
1:500 Export
EVPN-VPWS
Flexible Cross-
Connect (FXC)
EVPN – Flexible Cross-Connect Service
Challenge:
How to bring multiple access services from different sources using a single EVPN E-LINE
tunnel?
A1
CE1 CE1
A1
EVPN
PE
CE2 MPLS PE CE2
Forwarder
MUX
A2 CEn DEMUX
CEn
Normalized VLAN
VLAN translation over unique tunnel
EVPN – Flexible Cross-Connect Service
Request:
Can local switching preferred over ELINE tunnel?
A1
CE1 CE1
A1
EVPN
PE
CE2 MPLS PE CE2
Forwarder
MUX
A2 CEn DEMUX
CEn
Local Switching is
preferred on Backup connectivity Normalized VLAN
matching VLAN over EVPN ELINE VLAN translation over unique tunnel
Flexible Cross-Connect Service: Local Switching
Purpose:
Bring access services (e.g OLT) into BNG with redundancy
VRFs
BNG
VLANs EVPN
A1 A2
rewrite
VLANs
OLT OLT OLT x1000
ELINE
(backup tunnel)
Flexible Cross-Connect (FXC) - Testbed
Startup Sequence is almost identical with EVPN except:
RT3 and RT2 are not required
RR103 RR104
LACP R39
H2
R38 R35
R37 R34
H1 R36
Config: FXC VLAN-Unaware & VLAN-Aware
R36/R38/R39 VLAN-Unaware
l2vpn
RR103 RR104
flexible-xconnect-service vlan-unaware 600
interface Bundle-Ether100.10
interface Bundle-Ether100.20 LACP R39
neighbor evpn evi 600 target 363839
!
H2
R36/R38/R39 VLAN-Aware R38 R35

R36#show run l2vpn
l2vpn
flexible-xconnect-service vlan-aware evi 600
interface Bundle-Ether100.10
interface Bundle-Ether100.20 R37 R34
!
H1 R36
Leaf/Access R36: L2vpn xconnect status & Data
Plane verification
VLAN-Unaware VLAN-Aware
R36#show l2vpn flexible-xconnect-service R36#show l2vpn flexible-xconnect-service
Legend: ST = State, UP = Up, DN = Down, AD = Admin Down, UR = Unresolved, Legend: ST = State, UP = Up, DN = Down, AD = Admin Down, UR = Unresolved,
SB = Standby, SR = Standby Ready, (PP) = Partially Programmed SB = Standby, SR = Standby Ready, (PP) = Partially Programmed
Flexible XConnect Service Segment Flexible XConnect Service Segment

Name ST Type Description ST Name ST Type Description ST
------------------------- ---------------------------------- ------------------------- ----------------------------------
600 UP AC: BE100.10 UP evi:600 UP AC: BE100.10 UP
AC: BE100.20 UP AC: BE100.20 UP
PW: EVPN 600,363839,64011 UP PW: EVPN 600 UP
--------------------------------------------- ---------------------------------------------
R36#show mpls forwarding labels 64011 R36#show evpn internal-label
Local Outgoing Prefix Outgoing Next Hop Bytes VPN-ID Encap Ethernet Segment Id EtherTag Label
Label Label or ID Interface Switched ---------- ------ --------------------------- -------- --------
------ ----------- ------------------ ------------ --------------- ------------
64011 64039 EVPN:600 3.3.3.38 0 600 MPLS 0038.3900.0000.0000.1100 10 64012
64037 EVPN:600 3.3.3.39 0 Summary pathlist:
0x02000006 3.3.3.38 64041
0x02000007 3.3.3.39 64040
600 MPLS 0038.3900.0000.0000.1100 20 64013

Summary pathlist:
0x02000006 3.3.3.38 64041
0x02000007 3.3.3.39 64040
Leaf/Access R36: RT-1 Per EVI Ethernet Auto-Discovery
VLAN-Unaware VLAN-Aware
R36#show bgp l2vpn evpn rd 3.3.3.36:600 [1][0038.3900.0000.0000.1100][363839]/120 R36#show bgp l2vpn evpn rd 3.3.3.36:600
Thu Jun 6 05:40:06.781 UTC
BGP routing table entry for [1][0038.3900.0000.0000.1100][363839]/120, Route Distinguisher: 3.3.3.36:600 Status codes: s suppressed, d damped, h history, * valid, > best
Versions: i - internal, r RIB-failure, S stale, N Nexthop-discard
Process bRIB/RIB SendTblVer RT- Origin codes: i - IGP, e - EGP, ? - incomplete
Speaker 105 105 Target/Service-id Network Next Hop Metric LocPrf Weight Path
Last Modified: Jun 6 05:32:38.947 for 00:07:281 Route Distinguisher: 3.3.3.36:600 (default for vrf fxc:evi:600)
Paths: (2 available, best #1) ESI R38/R39 *> [1][0036.3700.0000.0000.1100][10]/120
Not advertised to any peer 0.0.0.0 0 i
Path #1: Received by speaker 0 *> [1][0036.3700.0000.0000.1100][20]/120
Not advertised to any peer 0.0.0.0 0 i
Local *>i[1][0038.3900.0000.0000.1100][10]/120
3.3.3.38 (metric 30) from 3.3.3.103 (3.3.3.38) 3.3.3.38 100 0 i
Received Label 64039 * i 3.3.3.39 100 0 i
Origin IGP, localpref 100, valid, internal, best, group-best, import-candidate, imported, rib-install *>i[1][0038.3900.0000.0000.1100][20]/120
Received Path ID 0, Local Path ID 1, version 103 3.3.3.38 100 0 i
Extended community: RT:1:600 * i 3.3.3.39 100 0 i
Local
Dot1q tag
3.3.3.39 (metric 30) from 3.3.3.103 (3.3.3.39)
dot1q tag ID
Origin IGP, localpref 100, valid, internal, import-candidate, imported, rib-install
ID
What will be tag ID value in case of QinQ?
Originator: 3.3.3.39, Cluster list: 3.3.3.103 example:
interface Bundle-Ether100.10 l2transport
encapsulation dot1q 10 second-dot1q 100
!
[1][0038.3900.0000.0000.1100][41060]/120
Formula: tag ID
(first_tag_value * 4096) + second_tag_value
(10 * 4096) + 100 = 41060
Leaf/Access R36: EVPN-VPWS Instance
R36#show evpn evi vpn-id 600 detail R36#show evpn evi vpn-id 600 detail
Thu Jun 6 06:25:06.940 UTC
VPN-ID Encap Bridge Domain Type
---------- ------ ---------------------------- ------------------- VPN-ID Encap Bridge Domain Type
600 MPLS VPWS:600 VPWS (vlan-unaware) ---------- ------ ---------------------------- -------------------
Stitching: Regular 600 MPLS fxc:600 VPWS (vlan-aware)
Unicast Label : 0 Stitching: Regular
Multicast Label: 0 Unicast Label : 0
Flow Label: N EVPN-VPWS Multicast Label: 0
• No RT2 – MAC
Flow Label: N EVPN-VPWS
Forward-class: 0
• No RT3 - BUM
• No RT2 – MAC
Advertise MACs: No Forward-class: 0
Advertise BVI MACs: No Advertise MACs: No • No RT3 - BUM
Aliasing: Enabled Advertise BVI MACs: No
UUF: Enabled Aliasing: Enabled
Re-origination: Enabled UUF: Enabled
Multicast source connected: No Re-origination: Enabled
Multicast source connected: No
Statistics: Statistics:
Packets Sent Received Packets Sent Received
Total : 0 0 Total : 0 0
Unicast : 0 0 Unicast : 0 0
BUM : 0 0 BUM : 0 0
Bytes Sent Received Bytes Sent Received
Total : 0 0 Total : 0 0
Unicast : 0 0 Unicast : 0 0
BUM : 0 0 BUM : 0 0
RD Config: none RD Config: none
RD Auto : (auto) 3.3.3.36:600 RD Auto : (auto) 3.3.3.36:600
RT Auto : 1:600 RT Auto : 1:600
Route Targets in Use Type Route Targets in Use Type
------------------------------ --------------------- ------------------------------ ---------------------
1:600 Import 1:600 Import
1:600 Export 1:600 Export
EVPN
Interconnect/Migration
(L2 Services)
EVPN L2 Interconnect
Everything in one Bridge Domain
• Legacy L2: REP, G8032, STP, etc.
LACP
• VPLS VPWS
• EVPN-VXLAN/EVPN-MPLS CE A3
• EoMPLS(PW)
• Ethernet – MultiHomed, SingleHomed
Leaf
VM
A3
Spine Spine
DCI/PE DCI/PE
PE1
Leaf
VM
STP/REP/ MPLS MPLS Core
A1 G.8032…. Core
Leaf
VM DCI/PE DCI/PE
PE2
A2 EVPN - VXLAN
EVPN-MPLS
EVPN - MPLS
A1 A2
VPLS
EVPN & VPLS Interconnect
CE2 A2 R37 PE2

LACP
MPLS Core/Access MPLS Core CE3
R36 PE1
CE1 A1
VPLS EVPN
R36/R37 Configuration
evpn l2vpn
evi 100 bridge group 100
advertise-mac bridge-domain 100
! access-vfi 1
virtual vfi 1 neighbor 3.3.3.37 pw-id 37
ethernet-segment !
identifier type 0 11.11.11.11.11.11.11.11.11 neighbor 3.3.3.38 pw-id 38
!
neighbor 3.3.3.39 pw-id 39
Virtual Ethernet Segment (vES) !
!
• VPLS is Single-Active Access to EVPN evi 100
Virtual Ethernet-Segment (vES)
R36#show evpn ethernet-segment detail
Ethernet Segment Id Interface Nexthops

------------------------ ---------------------------------- --------------------
0011.1111.1111.1111.1111 VFI:1 3.3.3.36
3.3.3.37
ES to BGP Gates : Ready
ES to L2FIB Gates : Ready
Virtual Access :
Name : VFI_1
State : Up
Num PW Up : 1
ESI type : 0
Value : 11.1111.1111.1111.1111
ES Import RT : 1111.1111.1111 (from ESI)
Source MAC : 0000.0000.0000 (N/A)
Topology :
Operational : MH, Single-active
Configured : Single-active (AApS) (default)
Service Carving : Auto-selection
Peering Details : 3.3.3.36[MOD:P:00] 3.3.3.37[MOD:P:00]
Service Carving Results:
Forwarders : 2
Permanent : 0
Elected : 2
Not Elected : 0
MAC Flushing mode : Invalid
Peering timer : 3 sec [not running]
Recovery timer : 30 sec [not running]
Carving timer : 0 sec [not running]
Local SHG label : 64006
Remote SHG labels : 1
64009 : nexthop 3.3.3.37
EVPN & VPLS Seamless
Integration - Migration
VPLS & EVPN Seamless Integration - Migration
VFI1 is by default in Split Horizon Group 1
CE2 R37 R39 CE4
• SHG1 protects loops in MPLS Core
• Full Mesh of pseudowires(PW) is required
for Any-to-Any forwarding
R36 MPLS
VFI1
PW_R37 UP
BD1 PW_R38 UP
PW_R39 UP
CE1
R38 CE3
l2vpn
bridge group 100
bridge-domain 100
vfi 1
!
!
!
!
CE2 R37 R39 CE4
R36 MPLS
VFI1
PW_R37 UP
EVI100 is also by default in Split Horizon Group 1
BD1 PW_R38 UP • R36 doesn’t forward data between VFI1 and EVI100
CE1 X PW_R39 UP
EVI100 R38 CE3
l2vpn
bridge group 100
bridge-domain 100
vfi 1
!
!
!
evi 100
!
CE2 R37 R39 CE4
R36 MPLS
VFI1
PW_R37 UP
EVI1 is also by default in Split Horizon Group 1
BD1 PW_R38 DOWN • R36 doesn’t forward data between VFI1 and EVI100
CE1 X PW_R39 UP
BGP EVPN
EVI100 R38 CE3 R36&R38 run BGP EVPN
• PW_R38 goes DOWN
• Data Forwarding between R36 and R38 via EVI100
l2vpn
bridge group 100
bridge-domain 100
vfi 1
!
!
!
evi 100
!
Conclusion
Conclusion:
Now we know why EVPN is a Unified Overlay Solution
Data Center Service Provider Enterprise
De-facto overlay solution for Next gen all-in-one VPN EVPN is used to address scale
transforming SP COs to NG solution to provide L2VPN, and efficiency issues as
DC to support virtualization, L3VPN, IRB, and more w/ a Enterprises need to migrate
multi-tenancy, flexible MH, single solution from traditional L2
flexible workload placement & access/aggregation networks
mobility, full cross-sectional BW Provides distributed vPE w/ to multi-tenancy fabric w/ the
utilization in fabric and access. white boxes for scaling out same characteristics as NG
instead of scaling up DC.
Supporting multiple Underlay:
MPLS, VxLAN, SR, GENVE Backhauling solution to vPE
EVPN: An All-in-one Solution
Services: E-LAN E-LINE E-TREE NVO Multicast L3VPN
(DC Fabric)
EVPN Solutions EVPN EVPN-VPWS EVPN-ETREE EVPN-Overlay EVPN-Mcast EVPN-L3VPN

PBB-EVPN EVPN-FXC EVPN-IRB
RFCs/Drafts: RFC 7209 RFC 8214 RFC 8317 RFC 8365 4 WG draft 2 WG drafts
RFC 7432 1 WG draft 2 WG drafts 1 Indv draft
RFC 7623
RFC 8584
5 WG drafts
7 Indv drafts
It provides all these service while supporting multi-tenancy,

workload mobility, full cross-sectional BW utilization, flexible
multi-homing, etc.
Complete your
online session • Please complete your session survey
evaluation after each session. Your feedback
is very important.
• Complete a minimum of 4 session
surveys and the Overall Conference
survey (starting on Thursday) to
receive your Cisco Live water bottle.
• All surveys can be taken in the Cisco Live
Mobile App or by logging in to the Session
Catalog on ciscolive.cisco.com/us.
Cisco Live sessions will be available for viewing
on demand after the event at ciscolive.cisco.com.
Continue your education
Demos in the
Walk-in labs
Cisco campus
Meet the engineer

Related sessions
1:1 meetings
Thank you
#CLUS
#CLUS

BRKMPL 3333

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

BRKMPL 3333

Uploaded by

Copyright:

Available Formats

#CLUS

Webex Teams will be moderated cs.co/ciscolivebot#BRKMPL-3333

2010 - 2012 2013 - 2015 2016-2019

Following drafts were Multicast Services

Data Center Service Provider Enterprise

EVPN Solutions EVPN EVPN-VPWS EVPN-ETREE EVPN-Overlay EVPN-Mcast EVPN-L3VPN

It provides all these service while supporting multi-tenancy,

• EVPN has emerged as such unified solution because of:

Leaf Leaf Leaf Leaf PE Spine Spine

Multi-site Data Center Interconnection w/ EVPN VF4

• Full cross-sectional BW Utilization WAN DC2

• Flexible WAN Interconnect 4

Leaf Leaf Leaf Leaf Leaf Leaf Leaf

Leaf Leaf Leaf Leaf Leaf Leaf Leaf

• EVPN All-Active MH: Providing per-flow load-balancing between CE and multi-homing

• Underlay: physical topology + • Overlay: Virtual topology among

EVI is an EVPN instance

• 1:1 mapping of VLAN to BD • 1:1 mapping of VLAN to BD • N:1 mapping of VLANs to BD

CE2 PE2 CE2 PE2

• Limited to Dual-homing • N-way redundancy

• Increased cost because of inter-chassis links • No inter-chassis links + fast convergence

• Inflexible configuration • Very flexible

• Standard based with multitude of applications: SP

ESI-0 ESI-0 ESI-1 ESI-1

doesn’t get loopback to the Echo !

• DF selection: Either PE3 or MAC1 CE1 PE1 PE3 CE2 MAC2

• Optimizing full cross-sectional BW but • Flexible workload placement is thus relevant

Subnet 1/VLAN1 Subnet 2/VLAN2

• To support Flexible workload placement w/ VM mobility,

IP-VPN Instance represents a

Leaf IP-VRF IP-VRF IP-VRF IP-VRF IRB Interface: a virtual interface

MAC- MAC- MAC- MAC- MAC- MAC- MAC- Leaf

2. VM move results in VM MAC moving to a new IP association

Secured DCI SD-WAN

• With secure-EVPN, # of signaling sessions is

• To protect User both unicast & mcast data

• Define a new encapsulation sub-TLV for VxLAN-in-IPsec

• Use existing BGP mechanism for tunnel setup

Solution Overview SA for cntrl Traffic

Signaling for data

• Use this secured BGP channel for P2MP

MAC Header EtherType: IPv4/IPv6

EtherType: IPv4/IPv6 IP Header

1. VxLAN Encap within ESP Protocol = UDP ESP Header

UDP Header UDP Header

VXLAN Header VXLAN Header

MAC Header EtherType: IPv4/IPv6

EtherType: IPv4/IPv6 IP Header

UDP Protocol = UDP UDP Header

IPsec encap in transport mode with VXLAN Header UDP Header

outer UDP header per [RFC3948] VXLAN Header

• Needed to NAT traversal or per flow

LB using UDP header

Functionality EVPN IP-VPN MVPN VPLS

Tenant IMET loopback I-PMSI N/A

Subnet of tenant IMET N/A N/A VPLS AD route

IP of tenant RT2 or RT5 VPN-v4/v6 route S-PMSI N/A

NA: Not Applicable

ID payload is defined in section 3.5 of [RFC7296]

Flexible WAN/DC Connectivity

PE1 PE2 PE1 PE2 PE1 PE2

CE1 CE2 CE3

Single LAG at the CE Multiple LAGs at the CE Multiple LAGs at the CE