IOS-XR EVPN Deep Dive v3

• Check internal wiki first
• Download always latest/greatest version

of this deck
http://go2.cisco.com/evpn
© 2022 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Deck Update tracking
• I’m not tracking cosmetic changes or minor updates ;) be sure you always download latest version
• Feb12/2019 – Cisco Live Barcelona deck – baseline
• Jun 6/2019 – FXC, VPLS migration, VPLS&EVPN Interconnect
• Jun 7/2019 – Troubleshooting Hints
• Jun 28/2019 – VPWS&EVPN Interconnect
• Aug 1/2019 – EVPN Timers
• Nov 2019 – BGP L3 Interconnect change
• Apr 2020 – EVPN Principles – more details
• Apr 2020 – 7.1.1 EVPN Single-Active EVPN-VPWS Single-Active, evpn balancing modes, symmetric/asymmetric IRB,
centralized GW
• Jun 2020 – EVPN/EVPN-VPWS EVI best practice
• Apr 2022 – additional features section (CW NCS)
• Apr 2022 – Transport integration
• Apr 2022 – Single-Flow-Active (SFA)
• Apr 2022 – EVPN Headend

IOS-XR EVPN Deep Dive
Jiri Chaloupka – Principal Technical Marketing Engineer
April 2022
Agenda
• EVPN Basic Principles
• EVPN L2 All-Active Multihomed Service
• EVPN Distributed L3 Anycast Gateway
• EVPN & VPNv4/6 Interconnect
• EVPN Single-Active
• EVPN Routes - Summary
• EVPN-VPWS All-Active Multihomed Service
• EVPN-VPWS Flexible Cross-Connect (FXC)
• EVPN Interconnect & Seamless Integration/Migration (L2 Services)
• EVPN Troubleshooting Hints
• EVPN Timers
• Conclusion
From Mac Bridging to Mac Routing
Common BGP Control Plane
EVPN, VPNv4/6 Overlay
Evolution:
Underlay
Segment Routing (SR: MPLS, SRv6) SR, VXLAN SR, VXLAN
Data Center Network

Service Provider Network overlap
Leaf
VM
PE1 DCI1
Spine Spine
Leaf
VM
A1 Access WAN/Core
Leaf
PE2 DCI2 VM
BGP: VPNv4/6 VPLS Overlay

Existing Solution: Fabric-Path (Trill)
LDP: VPLS, PW Fabric-Path
MPLS: LDP, RSVP-TE MPLS, L2 L2, IP Underlay

Service Provider Network - Simplification Journey
Compass
Unified MPLS EPN 5.0 Metro Fabric
Provisioning NETCONF NETCONF
YANG YANG
Programmability
PCE
L2/L3VPN Services LDP BGP LDP BGP BGP
Inter-Domain CP BGP-LU BGP-LU

FRR or TE RSVP
IGP with SR
LDP IGP with SR
Intra-Domain CP
IGP
Next-Generation Solutions for L2VPN
Solving VPLS challenges for per-flow Redundancy
M1 M2
CE1 PE1 PE3 CE2
Echo !
• Existing VPLS solutions do not offer an All-
PE2 PE4
Active per-flow redundancy
• Looping of Traffic Flooded from PE M1 Duplicate !
M2
CE1 PE1 PE3 CE2
• Duplicate Frames from Floods from the Core
• MAC Flip-Flopping over Pseudowire PE2 PE4
• E.g. Port-Channel Load-Balancing does not

produce a consistent hash-value for a frame with M1 M2
the same source MAC (e.g. non MAC based CE1 PE1
MAC
Flip-Flop PE3 CE2
Hash-Schemes)
PE2 PE4
MPLS Transport & BGP Service
BGP L3VPN/ L3 EVPN BGP L2VPN EVPN
BGP Signaling BGP Signaling BGP Signaling BGP Signaling
PE2 PE4 PE2 PE4
CE1 MPLS CE2 CE1 MPLS CE2
PE1 PE3 PE1 PE3

Data Plane Data Plane
IP Packet Transport IP Packet L2 Frame Transport L2 Frame

MPLS Label MPLS Label
Service Service
BGP Label BGP Label
IP Packet L2 Frame
EVPN – Basic Principles
EVPN Advantages:
Integrated • Integrated Layer 2 and Layer 3 VPN services
Services • L3VPN-like principles and operational experience for scalability and control
• All-active Multi-homing & PE load-balancing (ECMP)
• Fast convergence (link, node, MAC moves)
Network
• Control-Place (BGP) learning. PWs are no longer used.
Efficiency
• Optimized Broadcast, Unknown-unicast, Multicast traffic delivery
• Choice of MPLS, VxLAN or SRv6 data plane encapsulation
Service • Support existing and new services types (E-LAN, E-Line, E-TREE)
Flexibility • Peer PE auto-discovery. Redundancy group auto-sensing
Fully support IPv4 and IPv6 in the data plane and control plane
Investment •
Protection • Open-Standard and Multi-vendor support
Concepts
EVPN Instance (EVI) Ethernet Segment BGP Routes BGP Route Attributes
SHD CE1
Route Types Extended Communities
ESI1 PE1 [1] Ethernet Auto-Discovery (AD) Route ESI MPLS Label
BD EV
I
[2] MAC/IP Advertisement Route ES-Import

MHD CE2
[3] Inclusive Multicast Route MAC Mobility
BD
EV
PE2
I
ESI2 [4] Ethernet Segment Route Default Gateway

PE
[5] IP Prefix Advertisement Route Encapsulation
• EVI identifies a VPN in the • Represents a ‘site’ connected • New SAFI [70] • New BGP extended
network to one or more PEs • Routes serve control communities defined
• Encompass one or more • Uniquely identified by a 10- plane purposes, • Expand information
bridge-domains, byte global Ethernet including: carried in BGP routes,
depending on service Segment Identifier (ESI) MAC address reachability including:
interface type • Could be a single device MAC address moves
MAC mass withdrawal
Port-based or an entire network Redundancy mode
Split-Horizon label adv.
VLAN-based (shown above) Single-Homed Device (SHD) MAC / IP bindings of a GW
Aliasing
VLAN-bundling Multi-Homed Device (MHD) Split-horizon label encoding
Multicast endpoint discovery
Single-Homed Network (SHN) Redundancy group discovery Data plane Encapsulation
Multi-Homed Network (MHN) Designated forwarder election
IP address reachability
© 2022 Cisco and/or its affiliates. All rights reserved. Cisco Confidential L2/L3 Integration
EVPN - Load-Balancing Modes
6.X.X 7.1.1 7.1.2
All-Active Single-Active Port-Active
(per flow) (per VLAN) (per port)
PE1 PE2 PE1 PE2 PE1 PE2
V1 V1 V1 V2 V1, V2
CE1 CE2 CE3
Single LAG at the CE Multiple LAGs at the CE Single LAGs at the CE

VLAN goes to both PE VLAN active on single PE Port active on single PE
Traffic hashed per flow Traffic hashed per VLAN Traffic hashed per port
Benefits: Bandwidth, Convergence Benefits: Billing, Policing Benefits: Protocol Simplification
EVPN - load-balancing modes
Multi-Homed Interface Default Mode Supported Mode
Bundle Interface All-Active All-Active
Single-Active
Port-Active
Physical Interface Single-Active Single-Active

(gig, tengig, etc..)
Static Anycast All-Active All-Active
Pseudowire (PW)
Access VPLS (access-VFI) Single-Active Single-Active
Route Distinguisher (RD) & Route Target (RT) – VPNv4/6
Reminder
• Route Distinguisher - makes IPv4/IPv6 prefix globally unique
• Route Target Extended Community – “tag” IPv4/IPv6 prefix for selective import/export
RD
1:1
VRF_A
RT
Export 10:10 VRF_A
RT
BGP VPNv4:
X.X.X.0/24 Import 10:10
1:1_X.X.X.0/24 RT:10:10
2:2_X.X.X.0/24 RT:20:20
RD X.X.X.0/24
2:2
VRF_B VRF_B
RT RT
Export 20:20 Import 20:20
X.X.X.0/24 X.X.X.0/24
Route Distinguisher (RD) & Route Target (RT) – VPNv4/6
Reminder
• Route Distinguisher - makes IPv4/IPv6 prefix globally unique
• Route Target Extended Community – “tag” IPv4/IPv6 prefix for selective import/export
RD
1:1
VRF_A
RT VRF_A
Export 10:10 RT
BGP VPNv4: Import 10:10
X.X.X.0/24 1:1_X.X.X.0/24 RT:10:10
2:2_Y.Y.Y.0/24 RT:10:10 X.X.X.0/24
RD Y.Y.Y.0/24
2:2
VRF_B VRF_B
RT RT
Export 10:10 Import 20:20
Y.Y.Y.0/24 Empty
EVPN - Route Distinguisher (RD) and Route Target (RT) Allocation
Per-Node/Per-EVI RD - [BGP-RouterID]:[EVI-ID] -> Similar to VRF RD in L3VPN
EVPN RT1, RT2, RT3 R36 RD

3.3.3.36:0
Per-Node/Per-EVI RT - [BGP-AS]:[EVI-ID] -> Similar to VRF RT in L3VPN
RD
3.3.3.36:100
Per-Node RD - [BGP-routerid]:0,1,2,…. -> DF Election, Mass-Withdraw
BD1 EVI100
EVPN RT1, RT4
.1
MAC-A RT
BE1
R36 example BGP RouterID 3.3.3.36, BGP-AS: 1, EVI 100: IP-A 1:100
BE1 - ESI1
Per-Node RD: 3.3.3.36:0,1,2 Vlan1
Per-Node/Per-EVI RD: 3.3.3.36:100 Vlan2

RT
BE 1
MAC-B 1:101
Per-Node/Per-EVI RT: 1:100
.2
IP-B
BD2 EVI101
Why more Per-Node RD? RD

Maximum Route-Targets (RTs) per route is 400 3.3.3.36:101
EVPN - Instance
• EVPN Multipoint (ELAN) Instance is identified by local unique EVI ID
• Best Practice is to Autogenerate Route-Target (RT) => EVI ID becomes global unique per EVPN Instance
• IOS-XR Autogenerates RT by default
R36 RD
3.3.3.36:0
RD
3.3.3.36:100
BD1 EVI100
1
MAC-A RT
.
BE1
IP-A 1:100
BE1 - ESI1
Vlan1
Vlan2
RT
BE 1
MAC-B 1:101
.2
IP-B
BD2 EVI101
RD
3.3.3.36:101
Service ID available from IOS-XR 7.1.1+
EVPN-VPWS - Instance Source/Target is used before IOS-XR 7.1.1
See more detail in “EVPN-VPWS” section
• EVPN-VPWS (ELINE) Instance is identified by EVI ID and Service ID
• Best Practice is to Autogenerate Route-Target (RT) => EVI ID becomes global unique
• IOS-XR Autogenerates RT by default
• Service ID* provides additional instance granularity under single EVI ID
R36 RD R36 RD
Different EVI ID is used per EVPN-VPWS Instance 3.3.3.36:0 Single EVI ID for multiple EVPN-VPWS Instances 3.3.3.36:0
RD RD
3.3.3.36:100 3.3.3.36:102
P2P EVI100 P2P EVI102

Service ID RT Service ID RT
.1
1
MAC-A MAC-A
.
BE1
BE1
10 1:100 10 1:102
IP-A IP-A
BE1 - ESI1 BE1 - ESI1
Vlan1 Vlan1
Vlan2 Vlan2
BE 1
BE 1
Service ID RT Service ID RT
MAC-B 20 1:101 MAC-B 20 1:102
.2
.2
IP-B P2P EVI101 IP-B P2P EVI102
RD RD
© 2022 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 3.3.3.36:101 3.3.3.36:102
EVPN-VPWS EVI ID – Best Practice
• One EVI ID for All EVPN-VPWS Instances per Site Pair
• Example: EVI ID: 102, Service ID 10
• Site(PE1/PE2) and Site (PE3/PE4) => EVI 102 EVI ID: 102, Service ID 20
• Site(PE1/PE2) and Site (PE5) => EVI103
• Site(PE3/PE4) and Site (PE5) => EVI104
CE1 PE2 PE4 CE2
CE3 PE1 PE3 CE4

MPLS
CE3
PE5 CE4
EVI ID: 103, Service ID 10
EVPN - Ethernet VPN
• Concepts are same!!! Pick your side!
Pick your side!

SP1 SP2
PE2 PE4
CE1
PE1 PE3
L1 L2 L3 L4
C1 C2
VM VM VM VM
EVPN - Ethernet-Segment for Multi-Homing
L1 and L2 (L3 and L4) have to know if they multi-home same broadcast domain
SP1 SP2
The bundle on the Leafs
connecting to a node should have
Identical ES identifier (ESI)
L1 L2 L3 L4
Unique 10-byte global identifier

per Ethernet Segment Ethernet Segment represents a node
C1 C2 connected multiple Leaves
VM VM VM VM
EVPN - Ethernet VPN
MAC address advertisement and MAC address table synchronization
Leaves run Multi-Protocol BGP to advertise & learn MAC addresses over the Network
MAC addresses are advertised to rest of Leaves
L3/4 – Learn MAC address advertised by L1
L2 – uses MAC address advertised by L1 to synchronize MAC address table
-> L2 forwards MAC via local ETH interface represented by same Ethernet Segment between L1 and L2
SP1 SP2
MAC advertisement & learning/synchronization

via BGP EVPN NLRI
L1 L2 L3 L4
Data Plane learning
from the hosts
All Active multi-homing
C1 C2 Ethernet Segment
VM VM VM VM
EVPN L2 All-Active Multihomed
Service
EVPN - Testbed
RR103 RR104
LACP R39
H2
R38 R35
LACP R37 R34
H1
R36
EVPN Configuration CE has to receive same lacp system
MAC
lacp system mac 3637.3637.3637 l2vpn

bridge group 100
interface Bundle-Ether100 bridge-domain 100
l2transport interface Bundle-Ether100
! !
! evi 100
!
evpn RT-2 MAC advertise !
evi 100 !
advertise-mac !
! Core Isolation
group 1
core interface TenGigE0/0/0/38
core interface TenGigE0/0/0/39
!
interface Bundle-Ether100
ethernet-segment
identifier type 0 36.37.00.00.00.00.00.11.00
!
core-isolation-group 1
!
!
EVPN Configuration - BGP
router bgp 1
bgp router-id 3.3.3.36
address-family l2vpn evpn
!
neighbor-group rr
remote-as 1
update-source Loopback0 BGP EVPN CP
!
neighbor 3.3.3.103
use neighbor-group rr
!
neighbor 3.3.3.104
!
!
Ethernet Segment
R36#show evpn ethernet-segment
Mon Oct 15 13:27:44.402 UTC
Ethernet Segment Id Interface Nexthops

------------------------ ---------------------------------- --------------------
0036.3700.0000.0000.1100 BE100 3.3.3.36
3.3.3.37
Ethernet Segment
R36#show evpn ethernet-segment esi 0036.3700.0000.0000.1100 detail
.....
------------------------ ---------------------------------- --------------------
0036.3700.0000.0000.1100 BE100 3.3.3.36
3.3.3.37
ES to BGP Gates : Ready
ES to L2FIB Gates : Ready
Main port :
Interface name : Bundle-Ether100
Interface MAC : 008a.9644.d8dd
IfHandle : 0x0800001c
State : Up
Redundancy : Not Defined
ESI type : 0
Value : 36.3700.0000.0000.1100
ES Import RT : 3637.0000.0000 (from ESI)
Source MAC : 0000.0000.0000 (N/A)
Topology :
Operational : MH, All-active
Configured : All-active (AApF) (default)
Service Carving : Auto-selection
Peering Details : 3.3.3.36[MOD:P:00] 3.3.3.37[MOD:P:00]
Service Carving Results:
Forwarders : 1
Permanent : 0
Elected : 1
Not Elected : 0
MAC Flushing mode : STP-TCN
Peering timer : 3 sec [not running]
Recovery timer : 30 sec [not running]
Carving timer : 0 sec [not running]
Local SHG label : 64005
Remote SHG labels : 1
64005 : nexthop 3.3.3.37
EVPN Instance View
R36#show evpn evi vpn-id 100 detail
VPN-ID Encap Bridge Domain Type

---------- ------ ---------------------------- -------------------
100 MPLS 100 EVPN
Stitching: Regular
Unicast Label : 68096
Multicast Label: 64000
Flow Label: N
Control-Word: Enabled
Forward-class: 0
Advertise MACs: Yes
Advertise BVI MACs: No
Aliasing: Enabled
UUF: Enabled
Re-origination: Enabled
Multicast source connected: No
Statistics:
Packets Sent Received
Total : 0 0
Unicast : 0 0
BUM : 0 0
Bytes Sent Received
Total : 0 0
Unicast : 0 0
BUM : 0 0
RD Config: none
RD Auto : (auto) 3.3.3.36:100
RT Auto : 1:100
Route Targets in Use Type
------------------------------ ---------------------
1:100 Import
1:100 Export
EVPN – BUM Ingress Replication
Two service labels per EVPN instance
BUM Label – to forward Broadcast, Unknown Unicast and Multicast
Unicast Label – to forward Unicast
SP1 SP2
BU
BU
BU
M
M
M
L1 L2 L3 L4
BU
M
C1 C2
VM VM VM VM
EVPN BGP - Inclusive Multicast Route 0x3
• Usage:
• Multicast tunnels used to transport Broadcast, Multicast and Unknown Unicast frames (BUM)
Unique per Advertising PE per EVI

8 bytes RD Set to VLAN or I-SID for VLAN-Aware
4 bytes Ethernet Tag ID Bundling Service interface, otherwise 0
1 bytes
IP address length
IP Address Length
IPv4 or IPv6 address
4 or 16 bytes Originating Router’s IP add.
Route Type specific encoding of E-VPN NLRI
PMSI Tunnel Attribute - RFC6514
Flags based on RFC6514

1 bytes Flags Ingress Replication/mLDP etc.
1 bytes Tunnel Type
Multicast MPLS Label
3 bytes MPLS Label
When the Tunnel Type is set to Ingress Replication, the Tunnel Identifier
variable Tunnel Identifier carries the unicast tunnel endpoint IP address of the local PE that is to be this
PE's receiving endpoint address for the tunnel.
R36: RT-3 Inclusive Multicast
R36#show bgp l2vpn evpn rd 3.3.3.36:100 [3][0][32][3.3.3.36]/80
Mon Oct 15 13:10:17.010 UTC
BGP routing table entry for [3][0][32][3.3.3.36]/80, Route Distinguisher: 3.3.3.36:100
Versions:
Process bRIB/RIB SendTblVer RT-3
Speaker 39774 39774
Last Modified: Aug 31 01:37:02.399 for 6w3d
Paths: (1 available, best #1)
Advertised to update-groups (with more than one peer):
0.2
Path #1: Received by speaker 0
0.2
Local
0.0.0.0 from 0.0.0.0 (3.3.3.36)
Origin IGP, localpref 100, valid, redistributed, best, group-best, import-candidate
Received Path ID 0, Local Path ID 1, version 39774
Extended community: RT:1:100 EVI 100 Route-Target
PMSI: flags 0x00, type 6, label 64120, ID 0x03030324
Ingress Replication Multicast (BUM) Label
EVPN – Designated Forwarder (DF)
Challenge:
How to prevent duplicate copies of flooded traffic from being delivered to a multi-homed Ethernet Segment?
If (L3 and L4) Multi-Homing access via same Ethernet Segment -> only one of them can forward traffic to access
Same for (L1 and L2)
Why extra BUM Label?

What if Unicast Traffic is sent to L3 or L4 (not flooded)? -> DF Election applies only to BUM (from Core to Access)
DF, Redirect, Fast Re-Route (FRR), etc.
Service Label informs egress Leaf if traffic is BUM or Unicast
SP1 SP2
L1 L2 L3 L4
NDF DF
C1 Duplicate C2
DF Election per EVI/ESI - Algorithm
Service Carving
Nodes Position EVIs
R36
R37
0
1
+ 100
EVI-ID modulo Number of Nodes = Position

100 modulo 2 = 0
R36 is DF for EVI-100
Who will be DF for EVI-101?

Ethernet Segment - DF Election
R36#show evpn ethernet-segment esi 0036.3700.0000.0000.1100 carving detail
……
------------------------ ---------------------------------- --------------------
0036.3700.0000.0000.1100 BE100 3.3.3.36
3.3.3.37
Main port :
Interface MAC : 008a.9644.d8dd
State : Up
ESI type : 0
Value : 36.3700.0000.0000.1100
Source MAC : 0000.0000.0000 (N/A)
Topology :
Operational : MH, All-active
Configured : All-active (AApF) (default)
Forwarders : 1
Permanent : 0
Elected : 1
EVI E : 100
Not Elected : 0
© Cisco
2022 and/or
its
64005
affiliates. :
Allnexthop 3.3.3.37
rights reserved. Cisco Confidential
EVPN BGP - Ethernet Segment Router 0x4
• Usage:
• Auto-discovery of multi-homed Ethernet Segments
• Designated Forwarder election
• Tagged with ES-Import Extended Community
• PEs apply route filtering based on ES-Import community. Thus, Ethernet Segment route is imported only
by the PEs that are multi-homed to the same Ethernet segment
• ES-Import extended community is not the same as the Route Target (RT) extended community
Unique per Advertising PE

8 bytes RD
ESI of Ethernet Segment
10 bytes Ethernet Segment Identifier
1 bytes IP Address Length
IPv4 or IPv6 address
4 or 16 bytes Originating Router’s IP add.

ES-Import Extended Community
Usage:
• Used to tag the Ethernet Segment route
• Limits the scope of Ethernet Segment routes distribution to PEs connected to the same multi-homed
Segment
0x06
0x02 MAC Address portion of the ESI

6 bytes ES-Import
R36: RT-4 Ethernet Segment Router
R36#show bgp l2vpn evpn rd 3.3.3.36:0 [4][0036.3700.0000.0000.1100][32][3.3.3.36]/128
Mon Oct 15 03:24:50.736 UTC
BGP routing table entry for [4][0036.3700.0000.0000.1100][32][3.3.3.36]/128, Route Distinguisher: 3.3.3.36:0
Versions:
Process bRIB/RIB SendTblVer
Ethernet Segment Identifier (ESI)
Speaker 82835 82835 RT-4
Last Modified: Oct 14 21:32:13.399 for 05:52:37
0.2
0.2
Local
0.0.0.0 from 0.0.0.0 (3.3.3.36)
Origin IGP, localpref 100, valid, redistributed, best, group-best, import-candidate, rib-install
Extended community: EVPN ES Import:3637.0000.0000 DF Election:00:0:00
Nodes which share same ESI import this route
EVPN – Split Horizon
Challenge:
How to prevent flooded traffic from echoing back to a multi-homed Ethernet Segment?
Transport
BUM Label Label
SP1 SP2
SH Label
L1 L2
C1 Echo !
VM VM
EVPN – Split Horizon
Challenge:
How to prevent flooded traffic from echoing back to a multi-homed Ethernet Segment?
Transport
BUM Label Label
SP1 SP2
SH Label
L1 L2
C1 Echo !
C11
VM VM VM VM
EVPN – MAC Mass-Withdraw
Challenge:
How to inform other Leafs of a failure affecting many MAC addresses quickly while the
control-plane re-converges?
SP2 MAC1  ESI1  Leaf1 + Leaf2

MAC1 can be reached SP1
via ESI1
L1 L2 L3 L4
MAC1 can NOT be
reached via ESI1
C1 C2
VM VM VM VM
ESI1 MAC1
EVPN BGP - Ethernet Auto-discovery Route 0x1
Two flavors:
Per-ESI Ethernet A-D route Per-EVI Ethernet A-D route
• Advertise Split-Horizon Label associated with
an Ethernet Segment
• Used for MAC Mass-Withdraw
• Tagged with ESI MPLS Label Extended
Community

8 bytes RD
MUST be set to MAX-ET
4 bytes Ethernet Tag ID
MUST be set to 0
3 bytes MPLS Label
MAX-ET=0xFFFFFFFF
ESI Label Extended Community
Usage:
• Used to tag the Ethernet AD Route per ESI
• Advertises the Split-Horizon Label for the Ethernet Segment
• Indicates the Redundancy Mode: Single Active vs. All-Active
0x06
0x01 Bit 0: Redundancy Mode (single active vs. all active)

1 byte Flags
Set to 0
2 bytes Reserved
3 bytes ESI MPLS Label Ethernet Segment Split-Horizon Label
R36: RT-1 Per ESI Ethernet Auto-Discovery
R36#show bgp l2vpn evpn rd 3.3.3.36:0 [1][3.3.3.36:1][0036.3700.0000.0000.1100][4294967295]/184
Sun Oct 14 20:56:59.687 UTC
BGP routing table entry for [1][3.3.3.36:1][0036.3700.0000.0000.1100][4294967295]/184, Route Distinguisher: 3.3.3.36:0
Versions:
Process bRIB/RIB SendTblVer RD - unique per advertising Ethernet Segment Identifier (ESI)
Speaker 76372 76372 RT-1 node (R36 unique)
Local Label: 0
Last Modified: Sep 18 23:02:40.399 for 3w4d
0.2
0.2
Local
0.0.0.0 from 0.0.0.0 (3.3.3.36)
Extended community: EVPN ESI Label:0x00:64005 RT:1:100 EVI(s) Route-Target
All EVI(s) which use this ESI
Redundancy mode Split-Horizon Label

All-Active: 0x00
© 2022 Cisco and/or its affiliates. All rights0x01
Single-Active: reserved. Cisco Confidential
R36, R37, R38, R39 - EVPN Startup
R36 - Example
1. RT4: DF Election & Multi-Homed Ethernet
Segment Auto-Discovery
LACP R39
Service Carving: 100 modulo 2 = 0 H2
R36 is DF for EVI-100
R38 R35
RT-4 - DF Election
LACP R37 R34

RD: 1.1.1.36:1
H1 ESI: 0036.3700.0000.0000.1100
R36 Ext-Com: 3637.0000.0000 (RT)
R36 - Example
LACP R39
2. RT1: Per ESI Ethernet Auto-Discovery
(Split-Horizon, Mass-Withdraw) H2
RT-1 - Per ESI Ethernet AD
R38 R35
RD: 1.1.1.36:1
ESI: 0036.3700.0000.0000.1100
LACP R37 R34 Flag:0x00 All-Active

Ext-Com:
Split-Horizon Label: 64005
H1 Ext-Com: 1:100 (RT)
R36
R36 - Example
LACP R39
3. RT3: Inclusive Multicast RT-3 - Inclusive Multicast
R38 R35
RD: 1.1.1.36:100
Ext-Com: Type 6 Ingress-Replication
Multicast(BUM) Label: 64120
LACP R37 R34
Ext-Com: 1:100 (RT)
H1
R36
BUM Forwarding

LACP R39
3. RT3: Inclusive Multicast
R38 R35
LACP R37 R34
H1 Transport Label R38-9
R36 BUM Label R38-9/EVI100
BUM - Traffic
IR BUM - Traffic
BUM Forwarding

LACP R39
R38 R35
X
LACP R37 R34
Transport Label R37
H1 BUM Label R37/EVI100
R36 SH Label R37/ESIx
BUM - Traffic BUM - Traffic

IR
EVPN BGP - MAC Advertisement Route 0x2
ESI of Ethernet Segment on which MAC
Address was learnt. All 1s ESI for PBB-
8 bytes RD EVPN
Set to VLAN or I-SID for VLAN-Aware
Bundling Service interface, otherwise 0
Allows for MAC Address ‘summarization’, i.e.
1 byte MAC Address Length hierarchical MAC Addresses. Typically set to
6 bytes MAC Address 48
Could be C-MAC Address (EVPN) or B-MAC
1 byte IP Address Length Address (PBB-EVPN)
To distinguish IPv4 vs. IPv6 addresses.
4 or 16 IP Address
3 bytes MPLS Label

Used for ARP flood suppression or for
Integrated Routing and Bridging (IRB).
Route Type specific encoding of EVPN NLRI
VPN Label downstream assigned
MAC Mobility Extended Community
• Used to tag the MAC Advertisement route
• EVPN: Indicates that a MAC address has moved from one PE to another
0x06
0x00
Set to 0
2 bytes Reserved
Indicates the count of MAC address mobility
4 bytes Sequence Number events
R36: RT-2 MAC Advertisement
R36#show bgp l2vpn evpn rd 3.3.3.36:100 [2][0][48][0062.ec71.fbd7][0]/104
Mon Oct 15 04:33:39.527 UTC
BGP routing table entry for [2][0][48][0062.ec71.fbd7][0]/104, Route Distinguisher: 3.3.3.36:100
Versions:
Speaker 83317 83317 RT-2 Advertised MAC
Local Label: 64004
0.2
0.2
Local
0.0.0.0 from 0.0.0.0 (3.3.3.36)
Extended community: SoO:3.3.3.37:100 RT:1:100 R36 Re-Advertised
EVPN ESI: 0036.3700.0000.0000.1100
Not advertised to any peer
Local
3.3.3.37 (metric 30) from 3.3.3.103 (3.3.3.37)
Received Label 64004
Origin IGP, localpref 100, valid, internal, import-candidate, imported, rib-install
Extended community: SoO:3.3.3.37:100 RT:1:100 R37 MAC DP Learned and
Originator: 3.3.3.37, Cluster list: 3.3.3.103 Advertised
EVPN ESI: 0036.3700.0000.0000.1100
© 2022
Source AFI:
Cisco and/or L2VPN
its affiliates. EVPN,
All rights Source
reserved. VRF: default, Source Route Distinguisher: 3.3.3.37:100
Cisco Confidential
R36: RT-2 MAC Advertisement
R36#show evpn evi mac
Mon Oct 15 20:57:14.505 UTC
VPN-ID Encap MAC address IP address Nexthop Label

---------- ------ -------------- ---------------------------------------- --------------------------------------- --------
100 MPLS 0062.ec71.1000 :: 3.3.3.38 64006
100 MPLS 0062.ec71.1000 :: 3.3.3.39 64006
100 MPLS 0062.ec71.fbd7 :: 3.3.3.37 64004
100 MPLS 0062.ec71.fbd8 :: Bundle-Ether100 64004
100 MPLS 0062.ec71.fbd9 :: 3.3.3.37 64004
100 MPLS 0062.ec71.fbe0 :: 3.3.3.38 64006
100 MPLS 0062.ec71.fbe0 :: Learned
and
Advertised
3.3.3.39 64006
100 MPLS 0062.ec71.fbe1 :: MAC
3.3.3.38 64006
100 MPLS 0062.ec71.fbe1 :: 3.3.3.39 64006
R36 - Example
LACP R39
3. RT3: Inclusive Multicast RT-2 - MAC Advertisement
4. RT2: MAC Advertisement R38 R35
RD: 1.1.1.36:100
ESI: 0036.3700.0000.0000.1100
LACP R37 R34

MAC: 0062.ec71.fbd7
H1 Label: 64004
R36 Ext-Com: 1:100 (RT)
L2 Frame SMAC:
0062.ec71.fbd7
Unicast Forwarding
L2 Frame Flow1
Transport Label R36
1. RT4: DF Election & Multi-Homed Ethernet DMAC: H1
Segment Auto-Discovery RT-2 MAC Label/EVI

LACP R39
2. RT1: Per ESI Ethernet Auto-Discovery L2 Frame Flow1
DMAC: H1
LACP R37 R34
H1
R36
L2 Frame Flow1
DMAC: H1
EVPN – Aliasing
Challenge:
How to load-balance traffic towards a multi-homed device across multiple Leafs when
MAC addresses are learnt by only a single Leaf?
MAC1 can also be

SP2 MAC1  ESI1  Leaf1 + Leaf2
reached via ESI1 SP1
L1 L2 L3 L4
MAC1 can be
reached via ESI1
C1 C2
VM VM VM VM
ESI1 MAC1
57
EVPN BGP - Ethernet Auto-discovery Route 0x1
Two flavors:
Per-ESI Ethernet A-D route Per-EVI Ethernet A-D route
• Advertise Split-Horizon Label associated with an • Advertise VPN label used for Aliasing or
Ethernet Segment Backup-Path
• Used for MAC Mass-Withdraw
• Tagged with ESI MPLS Label Extended
Community
8 bytes RD ESI of Ethernet Segment
10 bytes Ethernet Segment Identifier Set to VLAN or I-SID for VLAN-
MUST be set to MAX-ET Aware Bundling Service interface,
MUST be set to 0 otherwise 0
3 bytes MPLS Label VPN (Aliasing) Label per (ESI,
Route Type specific encoding of E-VPN NLRI Ethernet Tag)
MAX-ET=0xFFFFFFFF
R36: RT-1 Per EVI Ethernet Auto-Discovery
RP/0/RP0/CPU0:R36#show bgp l2vpn evpn rd 3.3.3.36:100 [1][0036.3700.0000.0000.1100][0]/120
Mon Oct 15 03:35:13.604 UTC
BGP routing table entry for [1][0036.3700.0000.0000.1100][0]/120, Route Distinguisher: 3.3.3.36:100
Versions:
Process bRIB/RIB SendTblVer Ethernet Segment Identifier (ESI)
Speaker 79640 7964 RT-1
Last Modified: Oct 12 17:40:06.399 for 2d09h
0.2
0.2
Local
0.0.0.0 from 0.0.0.0 (3.3.3.36)
Local
3.3.3.37 (metric 30) from 3.3.3.103 (3.3.3.37) Aliasing Label allocated by R37 for EVI 100
Extended community: RT:1:100 EVI 100 Route-Target
Originator: 3.3.3.37, Cluster list: 3.3.3.103
© Source AFI: L2VPN EVPN, Source VRF: default, Source Route Distinguisher: 3.3.3.37:100
2022 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
R36 - Example
LACP R39
3. RT3: Inclusive Multicast RT-1 - Per EVI Ethernet AD
RD: 1.1.1.36:100
5. RT1: Per EVI Ethernet Auto-Discovery
ESI: 0036.3700.0000.0000.1100
LACP R37 R34

Aliasing-Label: 64004
H1 Ext-Com: 1:100 (RT)
R36
Unicast Forwarding
L2 Frame Flow1
Transport Label R36
Segment Auto-Discovery RT-2 MAC Label/EVI100

LACP R39
DMAC: H1
5. RT1: Per EVI Ethernet Auto-Discovery
LACP R37 R34
H1
R36
L2 Frame Flow1
DMAC: H1
Unicast Forwarding
L2 Frame Flow2
DMAC: H1
L2 Frame Flow1
Transport Label R36

LACP R39
DMAC: H1
3. RT3: Inclusive Multicast Transport Label R37
RT1 Label/EVI100
5. RT1: Per EVI Ethernet Auto-Discovery L2 Frame Flow2
DMAC: H1 L2 Frame Flow2
DMAC: H1
LACP R37 R34
H1
R36
L2 Frame Flow1
DMAC: H1
Unicast Forwarding
L2 Frame Flow2
DMAC: H1
L2 Frame Flow1
Transport Label R36

LACP R39
DMAC: H1
3. RT3: Inclusive Multicast Transport Label R37
RT1 Label/EVI100
5. RT1: Per EVI Ethernet Auto-Discovery L2 Frame Flow2
DMAC: H1 L2 Frame Flow2
Per Flow Balancing via R36 and R37 - Aliasing DMAC: H1
LACP R37 R34
H1
Per Flow Balancing via R36 and R37 - Aliasing
R36
L2 Frame Flow1
DMAC: H1
EVPN – MAC Mobility
Challenge:
How to detect the correct location of MAC after the movement of host from one Ethernet
Segment to another also called “MAC move”?
MAC IP ESI Seq. Next-Hop

Sequence number and Next-Hop value
will be changed after the host move SP1 SP
MAC-1 IP-1 0 1 Leaf-3

MAC-1 IP-1 0 0 Leaf-1
L1 L2 L3 L4 Sequence number is incremented and
Next-hop is changed to Leaf-3
C1 C2
VM
Host move
EVPN Distributed L3 Anycast
Gateway
EVPN – Distributed Symmetric Anycast Gateway
Leaves run Multi-Protocol BGP to advertise & learn MAC + HOST IP addresses over the Network
MAC + IP addresses are advertised to rest of Leaves
L3/4 – Learn MAC + IP HOST address advertised by L1
-> L2/L3 update MAC address table + IP Forwarding table
L2 – uses MAC address advertised by L1 to synchronize MAC address table
-> L2 forwards MAC via local ETH interface represented by same Ethernet Segment between L1 and L2
L2 – uses MAC + IP HOST address advertised by L1 to synchronize ARP/ND information
-> L2 forwards IP via local ETH interface
Identical Anycast Gateway Virtual IP
Distributed Anycast Gateway serves as and MAC address are configured on all
the gateway for connected hosts SP1 SP2 the Leafs
BVI BVI BVI BVI

GW GW GW GW
L1 L2 L3 L4
All the BVIs perform active forwarding in
contrast to active/standby like First-hop
routing protocol
C1 C2
VM VM VM VM
EVPN – IRB in Network Fabric
Intra-subnet
Forwarding
Inter-subnet
Forwarding
SP1 SP2
BVI BVI BVI BVI

GW GW GW GW
L1 L2 L3 L4
C1 C2 C3 C4
VM VM VM VM
Distributed vs Centralized Routing
Layer2 Bridging mandatory between Leaves only Layer2 Bridging mandatory between Leaves and DCI
IRB
L4 X.X.X.H2/24 L4 X.X.X.H2/24
H2 H2
IRB
DCI2 SP2 L3 IRB

DCI2 SP2 L3
CO IRB
CO
IRB
L2 L2
DCI1 SP1 DCI1 SP1
H1 H1
X.X.X.H1/24 X.X.X.H1/24
L1 IRB L1
• Optimized forwarding of east-west traffic • All east<->west routed traffic traverses to centralized gateways
• ARP/MAC state localized to Leafs • Centralized gateways have full ARP/MAC state in the DCI
• Helps with horizontal scaling of DC • Scale challenge
Symmetric vs Asymmetric - Integrated Routing and Bridging (IRB)
Symmetric Asymmetric
IRB
IRB IRB
L4 Y.Y.Y.H2/24 L4 Y.Y.Y.H2/24
H2 H2
IRB
SP2 L3 SP2 L3 IRB

IRB
CO CO IRB
IRB IRB
L2 L2
SP1 SP1
H1 H1
IRB
X.X.X.H1/24
L1 IRB
X.X.X.H1/24 L1 IRB
• Ingress and Egress Leaf – Routing and Bridging • Ingress Leaf – Routing and Bridging
• ARP/MAC Entries optimization • Egress Leaf – Bridging Only!
• L1/L2 MAC/ARP of Hosts from X.X.X.0/24 only • ARP/MAC Entries optimization
• L3/L4 MAC/ARP of Hosts from Y.Y.Y.0/24 only • L1/L2 MAC/ARP of Hosts from X.X.X.0/24 and Y.Y.Y.0/24
• • L3/L4 MAC/ARP of Hosts from Y.Y.Y.0/24 and X.X.X.0/24
Horizontally scalable solution
• Limited Scale
EVPN Distributed L3 Anycast GW - Symmetric IRB
Anycast IRB 192.168.2.1/24
RR103 RR104
IRB
LACP R39
H2: 192.168.2.20/24 H2 IRB
R38 R35
IRB
LACP R37 R34
H1: 192.168.1.10/24 H1
IRB
R36
Anycast IRB 192.168.1.1/24
EVPN Configuration - IRB
cef adjacency route override rib
prefer adjacency /32 (ARP) route over RIB
evpn
no evi 100
IOS-XR 6.0+
no advertise-mac AIB has the lowest priority by default (LSD>RIB>AIB)
!
vrf a Not needed! We need MAC/IP RT-2

address-family ipv4 unicast
import route-target
100:100
!
export route-target
100:100
! VRF configuration
!
!
interface BVI100
host-routing MAC/IP RT2
vrf a
ipv4 address 192.168.1.1 255.255.255.0
mac-address 3637.3637.3637 Anycast Distributed IRB: Same IP and MAC
! R36,R37
EVPN Configuration - BGP VRF
router bgp 1
address-family vpnv4 unicast
!
!
neighbor-group rr
remote-as 1
update-source Loopback0
!
neighbor 3.3.3.103
!
neighbor 3.3.3.104
!
vrf a
rd auto
additional-paths receive
maximum-paths ibgp 2 BGP Multi-Path for Inter-subnet forwarding
redistribute connected
!
!
R36: RT-2 MAC/IP Advertisement
R36#show bgp l2vpn evpn rd 3.3.3.36:100 [2][0][48][0062.ec71.fbd7][32][19$
Tue Oct 16 02:47:45.576 UTC
BGP routing table entry for [2][0][48][0062.ec71.fbd7][32][192.168.1.10]/136, Route Distinguisher: 3.3.3.36:100
Versions:
Speaker 84847 84847 RT-2 Advertised MAC IP
0.2
0.2
Local
0.0.0.0 from 0.0.0.0 (3.3.3.36)
Second Label 64008
Extended community: SoO:3.3.3.37:100 RT:1:100 RT:100:100
EVPN ESI: 0036.3700.0000.0000.1100
RT EVI 100 and RT VRF A
RT-2 per-BD label
Local
3.3.3.37 (metric 30) from 3.3.3.103 (3.3.3.37) VRF Agg label
Received Label 64004, Second Label 64008
Extended community: SoO:3.3.3.37:100 RT:1:100 RT:100:100 RT EVI 100 and RT VRF A
EVPN ESI: 0036.3700.0000.0000.1100
© 2022
Source
Cisco and/orAFI: L2VPN
its affiliates. EVPN,
All rights Source
reserved. VRF: default, Source Route Distinguisher: 3.3.3.37:100
Cisco Confidential
RP/0/RP0/CPU0:R36#
R36: RT-2 MAC/IP
R36#show evpn evi mac
Tue Oct 16 02:52:22.437 UTC
VPN-ID Encap MAC address IP address Nexthop Label

---------- ------ -------------- ---------------------------------------- --------------------------------------- --------
100 MPLS 0062.ec71.fbd7 192.168.1.10 3.3.3.37 64004
65535 N/A 008a.9644.d8d8 :: Local 0
Learned and Advertised RT-2 per-BD label

MAC and IP
R36: VRF Routes
R36#show route vrf a
Tue Oct 16 02:46:34.463 UTC
Codes: C - connected, S - static, R - RIP, B - BGP, (>) - Diversion path

D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP
i - ISIS, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, su - IS-IS summary null, * - candidate default
U - per-user static route, o - ODR, L - local, G - DAGR, l - LISP
A - access/subscriber, a - Application route
M - mobile route, r - RPL, t - Traffic Engineering, (!) - FRR Backup path
Gateway of last resort is not set
C 192.168.1.0/24 is directly connected, 03:37:59, BVI100

L 192.168.1.1/32 is directly connected, 03:37:59, BVI100
B 192.168.1.10/32 [200/0] via 3.3.3.37 (nexthop in vrf default)
B 192.168.2.20/32 [200/0] via 3.3.3.38 (nexthop in vrf default), 03:28:28
[200/0] via 3.3.3.39 (nexthop in vrf default), 03:28:28
EVPN Learned Route BGP Multi Path to H2 connected to R38 and R39
R36: AIB preference
R36#show cef vrf a 192.168.1.10
Tue Oct 16 02:48:21.376 UTC
192.168.1.10/32, version 9605, internal 0x1020001 0x0 (ptr 0x97c135fc) [1], 0x0 (0x97dda968), 0x0 (0x0)
Updated Oct 15 23:14:52.111
local adjacency 192.168.1.10
Prefix Len 32, traffic index 0, Adjacency-prefix, precedence n/a, priority 3
via 192.168.1.10/32, BVI100, 3 dependencies, weight 0, class 0 [flags 0x0]
path-idx 0 NHID 0x0 [0x98750da0 0x0]
next hop 192.168.1.10/32
local adjacency
Host Available via Local

adjancency - AIB
cef adjacency route override rib
prefer adjacency /32 (ARP) route over RIB
IOS-XR 6.0+
AIB has the lowest priority by default (LSD>RIB>AIB)
R36: VRF A - CEF
R36#show cef vrf a 192.168.2.20/32
Tue Oct 16 03:15:50.092 UTC
192.168.2.20/32, version 9613, internal 0x5000001 0x0 (ptr 0x97c14154) [1], 0x0 (0x0), 0x208 (0x98a06600)
Updated Oct 15 23:18:06.305
Prefix Len 32, traffic index 0, precedence n/a, priority 3
via 3.3.3.38/32, 5 dependencies, recursive, bgp-multipath [flags 0x6080]
path-idx 0 NHID 0x0 [0x97256420 0x0]
recursion-via-/32 VRF Agg label
next hop VRF - 'default', table - 0xe0000000
next hop 3.3.3.38/32 via 16038/0/21
next hop 35.36.1.35/32 Te0/0/0/39 labels imposed {16038 64004}
via 3.3.3.39/32, 5 dependencies, recursive, bgp-multipath [flags 0x6080]
path-idx 1 NHID 0x0 [0x97257178 0x0]
recursion-via-/32
next hop 3.3.3.39/32 via 16039/0/21
Inter-Subnet Multi-Path and

ECMP
R36 - Example Anycast IRB 192.168.2.1/24
1. RT4: DF Election & Multi-Homed Ethernet IRB

LACP R39
(Split-Horizon, Mass-Withdraw) H2 IRB
3. RT3: Inclusive Multicast RT-2 - MAC Advertisement
4. RT2: MAC/IP Advertisement R38 R35
RD: 1.1.1.36:100
IRB ESI: 0036.3700.0000.0000.1100
LACP R37 R34

MAC: 0062.ec71.fbd7
H1 Label: 64004(BD) + 64008(VRF)
R36 IP: 192.168.1.10
L2 Frame SMAC: Ext-Com: 1:100 (RT) + VRF RT

0062.ec71.fbd7
IP Header SurceIP:
192.168.1.10 Anycast IRB 192.168.1.1/24
BGP Layer3 - Interconnect
BGP Layer3 Interconnect
Principles
• DCI/BL provides Layer3 Interconnect
• DCI/BL participates in L3 Routing, but not in Layer2 Bridging
• DCI/BL summarization is required/recommended Layer2 Bridging Required over Leaves
IRB
L4 X.X.X.H2/24
H2
Z.Z.Z.CE1/24 DCI2 SP2 L3 IRB
CE1
PE/DCI
3 CORE CO
IRB
L2
DCI1 SP1
H1
X.X.X.H1/24
L1 IRB
DCI/BL Summarization
Host-Routes are not required outside CO/DC
L3/4 VRF FIB:
PE/DCI3 VRF FIB: DCI1/2 VRF FIB: X.X.X.H1 -> L1, L2
X.X.X.H1 -> DC1, DCI2 X.X.X.H1 -> L1, L2 X.X.X.H2 -> IRB(local)
X.X.X.H2 -> DCI1, DCI2 X.X.X.H2 -> L3, L4 X.X.X.0/24 -> IRB(local)
X.X.X.0/24 -> DC1, DCI2 X.X.X.0/24 -> L1, L2, L3, L4 Z.Z.Z.0/24 -> DCI1, DCI2
Z.Z.Z.0/24 -> CE1 Z.Z.Z.0/24 -> PE3
IRB
L4 X.X.X.H2/24
H2
Z.Z.Z.CE1/24 DCI2 SP2 L3 IRB L1/2 VRF FIB:

X.X.X.H1 -> IRB(local)
CE1
PE/DCI
3 CORE CO X.X.X.H2 -> L3, L4
X.X.X.0/24 -> IRB(local)
IRB Z.Z.Z.0/24 -> DCI1, DCI2
L2
DCI1 SP1
H1
X.X.X.H1/24
L1 IRB
Control Plane
BGP - L3VPN VPNv4/6 BGP - L3VPN VPNv4/6 Option #1 – VPNv4/6 & VPNv4/6
BGP – EVPN L3 BGP – EVPN L3
Option #2 – EVPN & EVPN
BGP - L3VPN VPNv4/6 BGP – EVPN L3
Option #3 – VPNv4/6 & EVPN
IRB
L4 X.X.X.H2/24
H2
CE1
PE/DCI
3 CORE CO
IRB
L2
DCI1 SP1
H1
X.X.X.H1/24
L1 IRB
Option #1 – VPNv4/6 & VPNv4/6
BGP - L3VPN VPNv4/6 BGP - L3VPN VPNv4/6
VPNv4: Z.Z.Z.0/24 VPNv4: Z.Z.Z.0/24
VPNv4: X.X.X.0/24 VPNv4: X.X.X.0/24
VPNv4: X.X.X.H1, X.X.X.H2

X
IRB
L4 X.X.X.H2/24
H2
CE1
PE/DCI
3 CORE CO
IRB
L2
DCI1 SP1
H1
X.X.X.H1/24
© 2022 Cisco and/or its affiliates. All rights reserved. Cisco Confidential L1 IRB
RT5: Z.Z.Z.0/24 RT5 Prefix: Z.Z.Z.0/24
RT5: X.X.X.0/24 RT5 Prefix: X.X.X.0/24
RT2 MAC/IP: X.X.X.H1, X.X.X.H2

X
IRB
L4 X.X.X.H2/24
H2
CE1
PE/DCI
3 CORE CO
IRB
L2
DCI1 SP1
H1
X.X.X.H1/24
VPNv4: Z.Z.Z.0/24 RT5 Prefix: Z.Z.Z.0/24
VPNv4: X.X.X.0/24 RT5 Prefix: X.X.X.0/24
RT2 MAC/IP: X.X.X.H1, X.X.X.H2

X
IRB
L4 X.X.X.H2/24
H2
CE1
PE/DCI
3 CORE CO
IRB
L2
DCI1 SP1
H1
X.X.X.H1/24
VRF A
RT import/export RD DCI:0 RT import/export stitching
BGP - L3VPN VPNv4/6 BGP - L3VPN VPNv4/6 Option #1 – VPNv4/6 & VPNv4/6
IRB
L4 X.X.X.H2/24
H2
CE1
PE/DCI
3 CORE CO
IRB
L2
DCI1 SP1
H1
X.X.X.H1/24
Control Plane Options Highlight
• Option #1 – VPNv4/6 & VPNv4/6
+ VPNv4/6 Industry proofed solution for Layer3 VPN
+ DCI doesn’t need to understand BGP EVPN AF
- Leaf has to peer with Route-Reflector via both BGP EVPN and VPNv4/6 AF
EVPN AF to support L2 stretch (MAC advertisement) across DC/CO between Leaves
EVPN AF to sync ARP/ND for Multi-Homed All-Active
- DC/CO Route-Reflector has to support both BGP EVPN and VPNv4/6 AF
- Leaf has to advertise VM Host-Routes via VPNv4/6
• Option #2 – EVPN & EVPN

+ Single BGP Address Family End-To-End in Network
- Existing L3 VPNv4/6 services has to to migrated to L3 EVPN
No technical benefit to migrate existing L3 VPNv4/6 to L3 EVPN
• Option #3 – VPNv4/6 & EVPN

+ Recommended solution which benefits from both Options #1 and #2
+ New DC/CO - Leaf, Route-Reflector use single BGP AF EVPN
+ Existing L3 VPNv4/6 services stay untouched
End-To-End MPLS Data Plane DCI1/2 VRF FIB:
X.X.X.H1 -> L1, L2
PE/DCI3 VRF FIB: X.X.X.H2 -> L3, L4
X.X.X.H1 -> DC1, DCI2 X.X.X.0/24 -> L1, L2, L3, L4
X.X.X.H2 -> DCI1, DCI2 Z.Z.Z.0/24 -> PE3 IRB
L4 X.X.X.H2/24
X.X.X.0/24 -> DC1, DCI2
Z.Z.Z.0/24 -> CE1
H2
CE1
PE/DCI
3 CORE CO
IRB
L2
DCI1 SP1
H1
X.X.X.H1/24
L1 IRB
X.X.X.H1 DCI: VRF IP Lookup! X.X.X.H1
Transport Label DCI1/2 PHP VRF Leaf 1/2 PHP VRF
Service Label VRF X.X.X.H1 VRF X.X.X.H1

X.X.X.H1 X.X.X.H1
BGP Layer3 Interconnect L3/4 VRF FIB:
End-To-End MPLS Data Plane DCI1/2 VRF FIB: X.X.X.H1 -> L1, L2
X.X.X.H1 -> L1, L2 X.X.X.H2 -> IRB(local)
X.X.X.H2 -> L3, L4 X.X.X.0/24 -> IRB(local)
X.X.X.0/24 -> L1, L2, L3, L4 Z.Z.Z.0/24 -> DCI1, DCI2
Z.Z.Z.0/24 -> PE3 IRB
L4 X.X.X.H2/24
H2

CE1
PE/DCI
L2
DCI1 SP1
H1
X.X.X.H1/24
L1 IRB
Z.Z.Z.CE1 DCI: VRF IP Lookup! Z.Z.Z.CE1
VRF PHP PE3 VRF PHP DC1/2 Transport Label

Z.Z.Z.CE1 VRF Z.Z.Z.CE1 VRF Service Label
Z.Z.Z.CE1 Z.Z.Z.CE1
CO - EVPN VXLAN Data Plane DCI1/2 VRF FIB:
X.X.X.H1 -> L1, L2
PE/DCI3 VRF FIB: X.X.X.H2 -> L3, L4
X.X.X.H1 -> DC1, DCI2 X.X.X.0/24 -> L1, L2, L3, L4
X.X.X.H2 -> DCI1, DCI2 Z.Z.Z.0/24 -> PE3 IRB
L4 X.X.X.H2/24
X.X.X.0/24 -> DC1, DCI2
Z.Z.Z.0/24 -> CE1
H2
CE1
PE/DCI
3 CORE CO
IRB
L2
DCI1 SP1
H1
X.X.X.H1/24
L1 IRB
X.X.X.H1 DCI: VRF IP Lookup! X.X.X.H1
Transport Label DCI1/2 PHP VRF Leaf 1/2 - IP VXLAN VNI

Service Label VRF X.X.X.H1
Inner ETH Header X.X.X.H1
© 2022 Cisco and/or its affiliates. All rights reserved. Cisco Confidential RFC7348
X.X.X.H1
BGP Layer3 Interconnect L3/4 VRF FIB:
CO - EVPN VXLAN Data Plane DCI1/2 VRF FIB: X.X.X.H1 -> L1, L2
X.X.X.H1 -> L1, L2 X.X.X.H2 -> IRB(local)
X.X.X.H2 -> L3, L4 X.X.X.0/24 -> IRB(local)
X.X.X.0/24 -> L1, L2, L3, L4 Z.Z.Z.0/24 -> DCI1, DCI2
Z.Z.Z.0/24 -> PE3 IRB
L4 X.X.X.H2/24
H2

CE1
PE/DCI
L2
DCI1 SP1
H1
X.X.X.H1/24
L1 IRB
Z.Z.Z.CE1 DCI: VRF IP Lookup! Z.Z.Z.CE1
VRF PHP PE3 DCI 1/2 - IP VXLAN VNI

Z.Z.Z.CE1 VRF
Inner ETH Header Z.Z.Z.CE1 RFC7348
Z.Z.Z.CE1
Data Plane Highlight - MPLS
• MPLS Data Plane
+ The packet structure is always identical, regardless of BGP VPNv4/6 or L3 EVPN Control Plane
Less Complexity, Simple Troubleshooting
+ MPLS Load-Balancing (ECMP) by Inner IP Header Lookup
+ Segment Routing provides Traffic Engineering and Fast Re-Reroute (FRR) capability
BGP L3 EVPN or VPNv4/6 MPLS Packet

Transport Label
VRF or Prefix Label
Original IP Packet
Data Plane Highlight - IP
• VXLAN Data Plane – RFC7348
- EVPN Signaling only
- RFC7348 requires Inner Ethernet encapsulation => Unnecessary overhead for L3 Forwarding
Outer IP
IP + UDP Header VXLAN Header - VNI Inner ETH Header Original IP Packet
- Inner Ethernet Header encapsulation/decapsulation typically done by Integrated Routing and Bridging (IRB) Interface
IRB requires Bridge-Domain
DCI doesn’t participate in L2 Forwarding => Bridge-Domain (BD) requires unnecessary HW resources
+ VXLAN draft-ietf-nvo3-vxlan-gpe can simplify
• SRv6
+ Transport and Service is integrated in Outer IPv6 Header
+ The packet structure is always identical, regardless of BGP VPNv4/6 or L3 EVPN Control Plane
Less Complexity, Simple Troubleshooting
Outer IPv6
Original IP Packet
Header
+ Load-Balancing (ECMP) by Flow-Label in outer IPv6 header

+ Doesn’t require additional header compared to VXLAN
+ Same Principles as Segment Routing MPLS
© 2022
Optional Segment Routing Header (SRH) can extend Traffic Engineering, Service Chaining and
Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Fast Re-Reroute (FRR) capabilities
EVPN and VPNv4/6 Interconnect
BGP - L3VPN VPNv4/6 BGP - EVPN
prefix-CE2/24 RT: VRF A RT5 prefix-CE2/24 RT: VRF A Stitching
RT2 MAC/IP = CE1/32 RT: VRF A Stitching

X
DCI/BL
VRF A
RD DCI:0
RT import/export: VRF A Stitching
RT import/export: VRF A
EVPN to VPNv4/6 Re-Advertise
RT2 MAC/IP = CE1/32 RT: VRF A Stitching

1. Import: RT: VRF A Stitching DCI/BL - BGP Configuration X
router bgp 1
address-family l2vpn evpn DCI/BL
import stitching-rt re-originate
advertise vpnv4 unicast re-originated stitching-rt
3. Filter RT2 => /32 Router ! VRF A
RD DCI:0
import re-originate stitching-rt
route-policy rt2-filter out RT import/export: VRF A Stitching
advertise vpnv4 unicast re-originated RT import/export: VRF A
!
2. Advertise to vpnv4: VRF A
VPNv4/6 to EVPN Re-Advertise
2. Advertise to EVPN: RT: VRF A Stitching DCI/BL - BGP Configuration

router bgp 1
address-family l2vpn evpn DCI/BL
import stitching-rt re-originate
advertise vpnv4 unicast re-originated stitching-rt
! VRF A
RD DCI:0
route-policy rt2-filter out RT import/export: VRF A Stitching
advertise vpnv4 unicast re-originated RT import/export: VRF A
1. Import: VRF A
!
EVPN and VPNv4 InterConnect
Anycast IRB 192.168.2.1/24
RR101 Emulates R2
RR103 RR104
IRB
RR101 R2
LACP R39
VPNv4: 9.9.9.101/24
H2: 192.168.2.20/24 H2 IRB
R38 R35 R28 R52
IRB
LACP R37 R34 R26 R51
H1: 192.168.1.10/24 H1
IRB
R36
Anycast IRB 192.168.1.1/24

BGP - EVPN BGP - L3VPN VPNv4/6
R36: BGP Configuration - RT-5
router bgp 1
!
!
neighbor-group rr
remote-as 1
advertise vpnv4 unicast
!
vrf a
RT-5
rd auto
additional-paths receive
maximum-paths ibgp 2
!
R36: RT-5 Prefix
R36#show bgp vpnv4 unicast
Status codes: s suppressed, d damped, h history, * valid,>best

i - internal, r RIB-failure, S stale, N Nexthop-discard
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 3.3.3.36:0 (default for vrf a)
*>192.168.1.0/24 0.0.0.0 0 32768 ?
* i 3.3.3.37 0 100 0 ?
*>i192.168.1.10/32 3.3.3.37 100 0 i
*>i192.168.2.0/24 3.3.3.38 0 100 0 ?
* i 3.3.3.39 0 100 0 ?
*>i192.168.2.20/32 3.3.3.38 100 0 i
* i 3.3.3.39 100 0 i
Advertised prefix RT-5
R36: RT-5 Route
R36#show bgp l2vpn evpn rd 3.3.3.37:0 [5][0][24][192.168.1.0]/80
Tue Oct 16 03:35:06.480 UTC
BGP routing table entry for [5][0][24][192.168.1.0]/80, Route Distinguisher: 3.3.3.37:0
Versions:
Speaker 84912 84912
RT-5 VRF A R37 RD
prefix
Local
Origin incomplete, metric 0, localpref 100, valid, internal, best, group-best, import-candidate, not-in-vrf
Extended community: Flags 0x6: RT:100:100
EVPN ESI: 0000.0000.0000.0000.0000, Gateway Address : 0.0.0.0 VRF A Route-Target
Local
Origin incomplete, metric 0, localpref 100, valid, internal, not-in-vrf
Extended community: Flags 0x6: RT:100:100
Originator: 3.3.3.37, Cluster list: 3.3.3.104 VRF A Route-Target
EVPN ESI: 0000.0000.0000.0000.0000, Gateway Address : 0.0.0.0
RP/0/RP0/CPU0:R36#
R36: VRF A - Routing Table
C 192.168.1.0/24 is directly connected, 04:55:09, BVI100

L 192.168.1.1/32 is directly connected, 04:55:09, BVI100
RP/0/RP0/CPU0:R36
DCI R26: VRF Configuration
vrf a
import route-target
100:100 stitching VRF a RT - CO
999:100
!
VRF a RT - Core
export route-target
100:100 stitching
999:100
!
!
!
R26#show route vrf a connected

Wed Oct 17 03:28:28.244 UTC
% No matching routes found

No Local L3 Interface in VRF 100
DCI R26: BGP Configuration
R26#show run router bgp 1 router bgp 1
Mon Oct 15 21:01:43.943 UTC !
router bgp 1 neighbor 1.1.1.101
bgp router-id 1.1.1.26 use neighbor-group rr-core
ibgp policy out enforce-modifications RR Next-Hop-change !
address-family vpnv4 unicast neighbor 3.3.3.103
! use neighbor-group rr
address-family l2vpn evpn !
! neighbor 3.3.3.104
neighbor-group rr use neighbor-group rr
remote-as 1 !
update-source Loopback0 vrf a
address-family l2vpn evpn EVPN AF - CO rd auto
import stitching-rt re-originate address-family ipv4 unicast
route-policy vpnv4-filter in additional-paths receive
route-policy vpnv4-community-set out maximum-paths ibgp 2
advertise vpnv4 unicast re-originated stitching-rt !
! !
! !
neighbor-group rr-core
remote-as 1
route-policy evpn-filter in
route-reflector-client BGP VRF
route-policy rt2-filter out
advertise vpnv4 unicast re-originated
!
DCI R26: BGP EVPN/VPNv4 Route Leaking Configuration
address-family l2vpn evpn Filter routes with VPNv4 route-policy rt2-filter
import stitching-rt re-originate community if destination in (0.0.0.0/0 ge 32) then
route-policy vpnv4-filter in drop
route-policy vpnv4-community-set out else
advertise vpnv4 unicast re-originated stitching-rt Set VPNv4 community set community evpn
! endif
end-policy
address-family vpnv4 unicast !
import re-originate stitching-rt Filter routes with EVPN
route-policy evpn-filter in community route-policy evpn-filter
route-reflector-client if community matches-any evpn then
route-policy rt2-filter out Filter /32 routes and set drop
advertise vpnv4 unicast re-originated EVPN community else
! pass
endif
end-policy
route-policy vpnv4-community-set
set community vpnv4
community-set evpn end-policy
1:111
end-set
! route-policy vpnv4-filter
community-set vpnv4 if community matches-any vpnv4 then
1:222 drop
end-set else
! pass
endif
end-policy
!
DCI R26: BGP EVPN/VPNv4 Route Leaking
import stitching-rt re-originate 1. Import RT 100:100 and re-
route-policy vpnv4-filter in originate with RT 999:100
route-policy vpnv4-community-set out
advertise vpnv4 unicast re-originated stitching-rt 4. Advertise re-originated
! routes with RT 100:100
address-family vpnv4 unicast 3. Import RT 999:100 and re-
import re-originate stitching-rt originate with RT 100:100
route-policy evpn-filter in
route-reflector-client
route-policy rt2-filter out 2. Advertise re-originated
advertise vpnv4 unicast re-originated routes with RT 999:100
!
vrf a
import route-target
100:100 stitching VRF a RT - CO
999:100
! VRF a RT - Core
export route-target
100:100 stitching
999:100
!
!
!
Anycast IRB 192.168.2.1/24
RR101 Emulates R2
RR103 RR104
IRB
RR101 R2
LACP R39
VPNv4: 9.9.9.101/24
H2: 192.168.2.20/24 H2 IRB
R38 R35 R28 R52
IRB
H1: 192.168.1.10/24 H1
IRB
R36
Anycast IRB 192.168.1.1/24

BGP - EVPN BGP - L3VPN VPNv4/6
Anycast IRB 192.168.2.1/24
RR101 Emulates R2
RR103 RR104
IRB
RR101 R2
LACP R39
VPNv4: 9.9.9.101/24
H2: 192.168.2.20/24 H2 IRB
R38 R35 R28 R52

R2 VPNv4 - Core Route
IRB
H1: 192.168.1.10/24 H1 R26#show route vrf a

IRB
R36 B 9.9.9.0/24 [200/0] via 1.1.1.101 (nexthop in vrf default), 1d07h
Anycast IRB 192.168.1.1/24 [200/0] via 3.3.3.37 (nexthop in vrf default), 15:12:55
H1 Host-Address BGP[200/0]
B 192.168.2.0/24 - EVPN BGP - L3VPN VPNv4/6
via 3.3.3.38 (nexthop in vrf default), 1d07h
[200/0] via 3.3.3.39 (nexthop in vrf default), 1d07h
B 192.168.2.20/32 [200/0] via 3.3.3.38 (nexthop in vrf default), 1d07h
H2 Host-Address [200/0] via 3.3.3.39 (nexthop in vrf default), 1d07h
Anycast IRB 192.168.2.1/24
RR101 Emulates R2
RR103 RR104
IRB
RR101 R2
LACP R39
VPNv4: 9.9.9.101/24
H2: 192.168.2.20/24 H2 IRB
R38 R35 R28 R52
IRB
H1: 192.168.1.10/24 H1
IRB
R36
r2#show route vrf a
Anycast IRB 192.168.1.1/24
C 9.9.9.0/24 is directly connected, 2w0d, Loopback9
L 9.9.9.101/32 is directly connected, 2w0d, Loopback9
H1 - Prefix B 192.168.1.0/24 [200/0] via 1.1.1.26 (nexthop in vrf default), 15:16:22
H2 - Prefix [200/0] via 1.1.1.28 (nexthop in vrf default), 1d07h
Anycast IRB 192.168.2.1/24
RR101 Emulates R2
RR103 RR104
IRB
RR101 R2
LACP R39
VPNv4: 9.9.9.101/24
H2: 192.168.2.20/24 H2 IRB
R38 R35 R28 R52
IRB
LACP R37 R34 R26 R51 R2 - Prefix
H1: 192.168.1.10/24 H1
IRB R36#show route vrf a
R36 B 9.9.9.0/24 [200/0] via 1.1.1.26 (nexthop in vrf default), 16:53:11

Anycast IRB 192.168.1.1/24
C 192.168.1.0/24 is directly connected, 1d12h, BVI100
L 192.168.1.1/32 is directly connected, 1d12h, BVI100
R2 VPNv4 - Core Route
[200/0] via 3.3.3.37 (nexthop in vrf default), 16:45:48 H1 Host-Address
[200/0] via 3.3.3.39 (nexthop in vrf default), 1d07h H2 Host-Address
r2#show route vrf a
C 9.9.9.0/24 is directly connected, 2w0d, Loopback9

L 9.9.9.101/32 is directly connected, 2w0d, Loopback9
B 192.168.1.0/24 [200/0] via 1.1.1.26 (nexthop in vrf default), 15:16:22 H1 - Prefix
H2 - Prefix

[200/0] via 1.1.1.28 (nexthop in vrf default), 16:53:11 R2 - Prefix
C 192.168.1.0/24 is directly connected, 1d12h, BVI100
L 192.168.1.1/32 is directly connected, 1d12h, BVI100
EVPN Single-Active
EVPN - Testbed
RR103 RR104
Single-Active
R39
H2
R38 R35
LACP R37 R34
H1
R36
All-Active - Example
R36#show evpn internal-label
VPN-ID Encap Ethernet Segment Id EtherTag Label

---------- ------ --------------------------- -------- --------
100 MPLS 0038.3900.0000.0000.1100 0 68103
Summary pathlist:
0x02000001 3.3.3.38 68096
0x02000002 3.3.3.39 68096
R36#show mpls forwarding labels 68103 detail
Local Outgoing Prefix Outgoing Next Hop Bytes

Label Label or ID Interface Switched
------ ----------- ------------------ ------------ --------------- ------------
68103 68096 EVPN:100 3.3.3.38 0
Updated: Jan 27 07:50:05.582
Version: 42, Priority: 3
Label Stack (Top -> Bottom): { 68096 }
NHID: 0x0, Encap-ID: 0x1386f00000002, Path idx: 0, Backup path idx: 0, Weight: 0
MAC/Encaps: 0/4, MTU: 0
Packets Switched: 0
68096 EVPN:100 3.3.3.39 0

Updated: Jan 27 07:50:05.582
NHID: 0x0, Encap-ID: 0x1387100000002, Path idx: 1, Backup path idx: 0, Weight: 0
©
2022 Cisco and/or its affiliates.
Packets Switched: All rights 0
reserved. Cisco Confidential
Single-Active – Configuration and Verification
R36#show evpn internal-label
Remote R38/R39
VPN-ID Encap Ethernet Segment Id EtherTag Label
---------- ------ --------------------------- -------- -------- evpn
100 MPLS 0038.3900.0000.0000.1100 0 68103 interface Bundle-Ether100
Summary pathlist: ethernet-segment
0x02000001 3.3.3.38 68096 load-balancing-mode single-active
0x00000000 3.3.3.39 (B) 68096 !
R36#show mpls forwarding labels 68103 detail !
Sun Jan 27 07:52:03.877 UTC !
------ ----------- ------------------ ------------ --------------- ------------
68103 68096 EVPN:100 3.3.3.38 0
Updated: Jan 27 07:51:14.370
Path Flags: 0x400 [ BKUP-IDX:1 (0x0) ]
NHID: 0x0, Encap-ID: 0x1386f00000002, Path idx: 0, Backup path idx: 1, Weight: 0
Packets Switched: 0
68096 EVPN:100 3.3.3.39 0 (!)

Updated: Jan 27 07:51:14.370
Path Flags: 0x300 [ IDX:1 BKUP, NoFwd ]
NHID: 0x0, Encap-ID: 0x1387100000002, Path idx: 1, Backup path idx: 0, Weight: 0
©
2022 Cisco
Packets Switched:
and/or its affiliates. All rights 0
reserved. Cisco Confidential
(!): FRR pure backup
Single-Active ethernet-segment carving detail
R38#show evpn ethernet-segment esi 0038.3900.0000.0000.1100 carving detail

------------------------ ---------------------------------- --------------------
0038.3900.0000.0000.1100 BE100 3.3.3.38
3.3.3.39
Main port :
Interface MAC : 008a.967f.30dd
State : Up
ESI type : 0
Value : 38.3900.0000.0000.1100
Source MAC : 0000.0000.0000 (N/A)
Topology :
Operational : MH, Single-active
Configured : Single-active (AApS)
Forwarders : 1
Permanent : 0
Elected : 1
EVI E : 100
Not Elected : 0
Remote
© 2022 SHGits affiliates.
Cisco and/or labelsAll: rights
1 reserved. Cisco Confidential
68098 : nexthop 3.3.3.39
EVPN Port-Active
EVPN - Testbed
RR103 RR104
LACP R39
H2
R38 R35
R36/R37
R37 R34
evpn LACP
X
ethernet-segment H1
load-balancing-mode port-active
!
R36
!
!
Port-Active –Verification
R36#show bundle R37#show bundle
Bundle-Ether100 Bundle-Ether100
Status: Up Status: LACP OOS (out of service)
Local links <active/standby/configured>: 1 / 0 / 1 Local links <active/standby/configured>: 0 / 1 / 1
Local bandwidth <effective/available>: 10000000 (10000000) kbps Local bandwidth <effective/available>: 0 (0) kbps
MAC address (source): 008a.9644.d8de (Chassis pool) MAC address (source): 008a.9644.08de (Chassis pool)
Inter-chassis link: No Inter-chassis link: No
Minimum active links / bandwidth: 1 / 1 kbps Minimum active links / bandwidth: 1 / 1 kbps
Maximum active links: 64 Maximum active links: 64
Wait while timer: 2000 ms Wait while timer: 2000 ms
Load balancing: Load balancing:
Link order signaling: Not configured Link order signaling: Not configured
Hash type: Default Hash type: Default
Locality threshold: None Locality threshold: None
LACP: Operational LACP: Operational
Flap suppression timer: Off Flap suppression timer: Off
Cisco extensions: Disabled Cisco extensions: Disabled
Non-revertive: Disabled Non-revertive: Disabled
mLACP: Not configured mLACP: Not configured
IPv4 BFD: Not configured IPv4 BFD: Not configured
IPv6 BFD: Not configured IPv6 BFD: Not configured
Port Device State Port ID B/W, kbps

Port Device State Port ID B/W, kbps -------------------- --------------- ----------- -------------- ----------
-------------------- --------------- ----------- -------------- ---------- Te0/0/0/0 Local Standby 0x8000, 0x0001 10000000
Te0/0/0/0 Local Active 0x8000, 0x0001 10000000 Link is in standby due to bundle out of service state
Link is Active
R37#show int bundle-ether 100
Bundle-Ether100 is down, line protocol is down
R37#show int tenGigE 0/0/0/0
TenGigE0/0/0/0 is up, line protocol is up
EVPN Single-Flow-Active (SFA)
EVPN Load-Balancing Modes
Single-Flow-Active (SFA)
7.3.1
Single-Homed Single-Flow-Active
STP/REP/G.8032 “break” L2 loop MST-AG/REP-AG/G.8032 “break” L2 loop
A3 A3
PE1 PE1
STP/REP/ MPLS MST/REP/ MPLS

A1 G.8032…. Core A1 G.8032…. Core
PE2 PE2
A2 A2
EVPN-MPLS EVPN-MPLS
EVPN Load-Balancing Modes
7.3.1
X
Single-Homed Single-Flow-Active (SFA)
STP/REP/G.8032 “break” L2 loop MST-AG/REP-AG/G.8032 “break” L2 loop
A3 A3
PE1 PE1
STP/REP/ MPLS MST/REP/ MPLS

A1 G.8032…. Core A1 G.8032…. Core
PE2 PE2
A2 A2
EVPN-MPLS EVPN-MPLS
EVPN Single-Flow-Active (SFA) - Configuration
PE1/PE2
evpn
ethernet-segment
identifier type 0 36.37.36.37.36.37.36.37.01
load-balancing-mode single-flow-active
convergence
mac-mobility
A3 PE37
P2
X
A2
MST/REP/ MPLS PE38
G8032
P1
A1 PE36
RT-1 Per ESI Ethernet Auto-Discovery

Sun Oct 14 20:56:59.687 UTC
Versions:
Process bRIB/RIB SendTblVer RD - unique per advertising Ethernet Segment Identifier (ESI)
Speaker 76372 76372 RT-1 node (R36 unique)
Local Label: 0
Last Modified: Sep 18 23:02:40.399 for 3w4d
0.2
0.2
Local
0.0.0.0 from 0.0.0.0 (3.3.3.36)
Extended community: EVPN ESI Label:0x02:64005 RT:1:100 EVI(s) Route-Target
All EVI(s) which use this ESI
Redundancy mode Split-Horizon Label
All-Active: 0x00
Single-Active: 0x01
Single-Flow-Active: 0x02 NEW! draft-brissette-bess-evpn-l2gw-proto
• PE36/PE37 are both DF (L2 legacy protocol must break a loop)
• PE36 advertise A2 MAC+IP EVPN RT2 with BGP Local-Preference 100
• PE37 synchronize A2 ARP/ND (EVPN RT2 MAC+IP advertised by PE36)
• FIB Next-Hop -> PE36
37#show arp vrf a

192.168.100.100 - a0aa.cccc.cccc EVPN_SYNC ARPA BVI100
37#show cef vrf a 192.168.100.100

A3 PE37
Prefix Len 32, traffic index 0, precedence n/a, priority 3
P2
via 3.3.3.36/32, 5 dependencies, recursive [flags 0x6000]
path-idx 0 NHID 0x0 [0x89dc1908 0x0]
X MST/REP/
A2 PE38
recursion-via-/32 G8032
next hop 3.3.3.36/32 via 16036/0/21 P1
next hop 35.37.1.35/32 Te0/0/0/39 labels imposed {16036 28103} A1 PE36
• PE36/PE37 are both DF (L2 legacy protocol must break a loop)
• PE36 advertise A2 MAC+IP EVPN RT2 with BGP Local-Preference 100
• PE37 synchronize A2 ARP/ND (EVPN RT2 MAC+IP advertised by PE36)
• FIB Next-Hop -> PE36
• PE37 Re-advertise A2 MAC+IP RT2 with BGP Local-Preference 80
• PE38 prefers A2 via PE36 (BGP LP 100)
EVPN Re-originated RT2 A2-MAC+IP LocalPref 80
A3 PE37
P2
X MST/REP/
A2 PE38
G8032
P1
A1 PE36
EVPN RT2 A2-MAC+IP LocalPref 100

R38#show bgp l2vpn evpn rd 3.3.3.38:100 [2][0][48][a0aa.cccc.cccc][32][192.168.100.100]/136
3.3.3.36 (metric 30) from 3.3.3.103 (3.3.3.36)
Origin IGP, localpref 100, valid, internal, best, group-best, import-candidate, imported, rib-install
EVPN ESI: 0036.3736.3736.3736.3701
Source AFI: L2VPN EVPN, Source VRF: default, Source Route Distinguisher: 3.3.3.36:100
Local
3.3.3.37 (metric 30) from 3.3.3.103 (3.3.3.37)
EVPN ESI: 0036.3736.3736.3736.3701
EVPN Single-Flow-Active (SFA) – MAC Move
• A2 moves because of L2 access topology change
• PE37 receives packet with source MAC A2
• PE37 advertises EVPN RT2 BGP Local-Preference 100 and MAC Mobility sequence number +1
36#show bgp l2vpn evpn rd 3.3.3.36:100 [2][0][48][a0aa.cccc.cccc][32][192.168.100.100]/136

0.0.0.0 from 0.0.0.0 (3.3.3.36)
Second Label 28103
Extended community: Flags 0xe: SoO:3.3.3.37:100 EVPN MAC Mobility:0x00:3 RT:1:100 RT:100:100
EVPN ESI: 0036.3736.3736.3736.3701

A3 PE37
P2
MST/REP/ PE38
A2
G8032
X P1
A1 PE36
EVPN Single-Flow-Active (SFA) – MAC Move
• A2 moves because of L2 access topology change
• PE37 receives packet with source MAC A2
• PE37 advertises EVPN RT2 BGP Local-Preference 100 and sequence number +1
• PE36 MAC Flush BD port (based on ESI)
• Packet with destination of A2-MAC sent to PE36 by remote PE is flooded via EVI to PE37
• PE37 programs FIB based on synchronized ARP/ND (Speculative ARP)

A3 PE37
P2
MST/REP/ PE38
A2
G8032
X P1
A1 PE36
DF Election Convergence Improvements
evpn
ethernet-segment
identifier type 0 36.37.36.37.36.37.36.37.01
load-balancing-mode single-active BGP Next-Hop Tracking for RT4
convergence Node Failure Convergence Improvement
nexthop-tracking
reroute
NTP Timestamping for RT4
R37#show evpn ethernet-segment carving detail

Service Carving Synchronization:
Mode : NTP_SCT
Peer Updates :
3.3.3.36 [SCT: 2020-10-28 12:57:47:456146]
3.3.3.37 [SCT: 2020-10-28 12:57:47:451599] NTP Timestamping for RT4
R37#show ntp status

Clock is synchronized, stratum 3, reference is 10.255.11.1
R37#show bgp l2vpn evpn rd 3.3.3.36:0 [4][0036.3736.3736.3736.3701][32]

3.3.3.36 (metric 30) from 3.3.3.103 (3.3.3.36)
Origin IGP, localpref 100, valid, internal, best, group-best, import-candidate, not-in-vrf
Extended community: EVPN ES Import:3637.3637.3637 DF Election:0:0x0008:0 EVPN NTP: 3812880149.4488
Convergence Profiles – [Latest]
Access ring failure PE node failure PE-CE link failure PE node restoration PE-CE link restoration
ASR9k NCS5500 ASR9k NCS5500 ASR9k NCS5500 ASR9k NCS5500 ASR9k NCS5500
L2 Service - Bridging
EVPN All-Active NA NA
EVPN Single-Active NA NA
EVPN Port-Active NA NA
EVPN Single-Flow-Active
L3 Service - IRB
EVPN Single-Flow-Active
VPWS Service
Sometime
Meet sub-sec Seconds
Sub-second
EVPN Routes - Summary
EVPN Routes – Cheat Sheet
BGP Signaling PE1 – Advertises:
PE2 PE4
PE1 RT-4 Ethernet Segment Route
CE1 MPLS CE2 • I have ESI1 in case when someone needs this information for
EVI1-L Designated Forwarder(DF) Election
PE1 PE3
BE1-SHL
Data Plane
EVI1-BUML RT-1 Per ESI Ethernet Auto-Discovery (AD) Route
Transport
L2 Frame
MPLS Label
L2 Frame
• I have ESI1
Service
• ESI1 is All-Active
BGP Label
L2 Frame
BD1 EVI1 • AC with ESI1 is connected to EVI1 and EVI2
BD1 MAC • My Split Horizon Label for ESI1 isBE1-SHL
MAC-A
.1
BVI1
MAC-A -> BE1.1
BE1
IP-A VRF1 ARP RT-1 Per EVI Ethernet Auto-Discovery (AD) Route(s)
BE1 - ESI1 • EVI1 per-EVI (Aliasing) Label isEVI1-L
Vlan1 IP-A MAC-A -> BVI1
VRF1 IP-B MAC-B -> BVI2 • EVI2 per-EVI (Aliasing) Label isEVI2-L
Vlan2 VRF1-AGGL
BVI2 RT-3 Inclusive Multicast Route(s)

B E1
MAC-B BD2 MAC • EVI1 Label for BUM traffic is EVI1-BUML

IP-B MAC-B -> BE1.2 • EVI2 Label for BUM traffic is EVI2-BUML
.2
BD2 EVI2
RT-2 MAC/IP Advertisement Route(s)
• MAC-A in EVI1 via label
EVI1-L and IP-A in VRF1 via label VRF1-AGGL
EVI2-L
• MAC-B in EVI2 via labelEVI2-L and IP-B in VRF1 via label VRF1-AGGL
BE1-SHL
EVI2-BUML RT-5 Prefix Advertisement Route(s)
• IPv4/6 prefix of BVI1 in VRF1 via labelVRF1-AGGL
• IPv4/6 prefix of BVI2 in VRF1 via labelVRF1-AGGL
EVPN-VPWS
Multihomed Service
EVPN-VPWS
• Benefits of EVPN applied to point-to-point services
• No signaling of PWs. Instead signals MP2P LSPs instead
(ala L3VPN)
• All-active CE multi-homing (per-flow LB)
PE2 PE4
• Single-active CE multi-homing (per-service LB)
CE1 MPLS CE2
• Relies on a sub-set of EVPN routes to advertise
Ethernet Segment and AC reachability PE1 PE3
• PE discovery & signaling via a single protocol – BGP
• Per-EVI Ethernet Auto-Discovery route
EVPN vs EVPN-VPWS - Balancing Mode
Single-Active
• Both EVPN and EVPN-VPWS advertise RT1(per-ESI) PE2 PE4
• Signal All-Active or Single-Active CE1 MPLS CE2
PE1 PE3
• Remote node performs per-flow load-balancing -> All-Active mode
• How remote node knows who is Active in Single-Active mode?

• EVPN
• Remote node follows MAC (RT2) advertisement -> node advertising MAC is active
• EVPN-VPWS
• Additional signaling per-service is required to inform remote node who is Active
EVPN-VPWS Layer 2 Attributes Extended Community
RFC8214 IOS-XR 7.1.1
+-------------------------------------------+
| Type (0x06) / Sub-type (0x04) (2 octets) | 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
+-------------------------------------------+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Control Flags (2 octets) | | MBZ |C|P|B| (MBZ = MUST Be Zero)
+-------------------------------------------+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| L2 MTU (2 octets) |
+-------------------------------------------+ Control-Word(C) = 4
| Reserved (2 octets) |
Primary(P) = 2
Backup(B) = 1
+-------------------------------------------+
L2 MTU is a 2-octet value indicating the MTU in bytes
EVPN-VPWS
All-Active
EVPN-VPWS - Testbed
Startup Sequence is almost identical with EVPN except:
RT3 and RT2 are not required
RR103 RR104
LACP R39
H2
R38 R35
R37 R34
H1 R36
Config: EVPN-VPWS
R36 R38/R39
l2vpn l2vpn
xconnect group 500 xconnect group 500
p2p 500 p2p 500
interface Bundle-Ether100 interface Bundle-Ether100
neighbor evpn evi 500 target 3839 source 36 neighbor evpn evi 500 target 36 source 3839
! !
! !
! !
From IOS-XR 7.1.1 Simplified configuration option is available (see next slides)
R36: L2vpn xconnect status & Data Plane verification
R36#show l2vpn xconnect
Legend: ST = State, UP = Up, DN = Down, AD = Admin Down, UR = Unresolved,
SB = Standby, SR = Standby Ready, (PP) = Partially Programmed
XConnect Segment 1 Segment 2

Group Name ST Description ST Description ST
------------------------ ----------------------------- -----------------------------
500 500 UP BE100 UP EVPN 500,3839,68106 UP
----------------------------------------------------------------------------------------
R36#show mpls forwarding labels 68106

------ ----------- ------------------ ------------ --------------- ------------
68106 68107 EVPN:500 3.3.3.38 0
68107 EVPN:500 3.3.3.39 0
R36#show bgp l2vpn evpn rd 3.3.3.36:500 [1][0038.3900.0000.0000.1100][3839]/120
Control-Word(C) = 4
BGP routing table entry for [1][0038.3900.0000.0000.1100][3839]/120, Route Distinguisher: 3.3.3.36:500 Primary(P) = 2
Versions:
Backup(B) = 1
Process bRIB/RIB SendTblVer RT-1 AC-
ESI R38/R39
Speaker 316 316 ID
Last Modified: Jan 27 08:24:37.527 for 00:01:42
Not advertised to any peer Control-Word + Primary
Local
3.3.3.38 (metric 30) from 3.3.3.103 (3.3.3.38)
MTU 1500B
Extended community: EVPN L2 ATTRS:0x06:1500 RT:1:500
Local
3.3.3.39 (metric 30) from 3.3.3.103 (3.3.3.39) MTU 1500B
R36: EVPN-VPWS Instance View
R36#show evpn evi vpn-id 500 detail

---------- ------ ---------------------------- -------------------
500 MPLS VPWS:500 VPWS (vlan-unaware)
Stitching: Regular
Unicast Label : 0
Multicast Label: 0
Flow Label: N EVPN-VPWS
Forward-class: 0
• No RT2 – MAC
Advertise MACs: No • No RT3 - BUM
Advertise BVI MACs: No
Aliasing: Enabled
UUF: Enabled
Re-origination: Enabled
Statistics:
Packets Sent Received
Total : 0 0
Unicast : 0 0
BUM : 0 0
Bytes Sent Received
Total : 0 0
Unicast : 0 0
BUM : 0 0
RD Config: none
RD Auto : (auto) 3.3.3.36:500
RT Auto : 1:500
Route Targets in Use Type
------------------------------ ---------------------
1:500 Import
1:500 Export
Config: EVPN-VPWS
R36 R38/R39
l2vpn l2vpn
p2p 500 p2p 500
! !
! !
! !
From IOS-XR 7.1.1 Simplified configuration option is available

if ”target id” and “source id” has same value => “service id” can be used
R36 R38/R39
l2vpn l2vpn
p2p 500 p2p 500
neighbor evpn evi 500 service 333 neighbor evpn evi 500 service 333
! !
! !
! !
EVPN-VPWS
Single-Active
EVPN-VPWS - Testbed
RR103 RR104
Single-Active
R39
H2
R38 R35
LACP R37 R34
H1
R36
Config: EVPN-VPWS
R36 R38/R39
l2vpn l2vpn
p2p 500 p2p 500
! !
! !
! !

------------------------ ----------------------------- -----------------------------
500 500 UP BE100 UP EVPN 500,3839,24004 UP
----------------------------------------------------------------------------------------
R36#show mpls forwarding labels 24004

------ ----------- ------------------ ------------ --------------- ------------ Active
24004 28127 EVPN:500 3.3.3.39 0
28127 EVPN:500 3.3.3.38 0 (!)
Standby
R36#show bgp l2vpn evpn rd 3.3.3.36:500 [1][0038.3900.0000.0000.1100][3839]/120
Tue Apr 14 07:47:20.033 UTC Control-Word(C) = 4
BGP routing table entry for [1][0038.3900.0000.0000.1100][3839]/120, Route Distinguisher: 3.3.3.36:500 Primary(P) = 2
Versions:
Backup(B) = 1
Process bRIB/RIB SendTblVer RT-1 AC-
ESI R38/R39
Speaker 430 430 ID
Last Modified: Apr 14 07:47:09.651 for 00:00:10
Not advertised to any peer Control-Word + Backup
Local
3.3.3.38 (metric 30) from 3.3.3.103 (3.3.3.38)
MTU 1500B
Local
3.3.3.39 (metric 30) from 3.3.3.103 (3.3.3.39) MTU 1500B
EVPN-VPWS
Flexible Cross-Connect (FXC)
EVPN – Flexible Cross-Connect Service
Challenge:
How to bring multiple access services from different sources using a single EVPN E-LINE tunnel?
A1
CE1 CE1
A1
EVPN
CE2 PE
CE2 MPLS PE Forwarder
MUX
A2 CEn DEMUX
CEn
Normalized VLAN
VLAN translation over unique tunnel
150
EVPN – Flexible Cross-Connect Service
Request:
Can local switching preferred over ELINE tunnel?
A1
CE1
EVPN
CE2 PE
Forwarder
MUX
CEn DEMUX
Normalized VLAN
VLAN translation over unique tunnel
151
Flexible Cross-Connect Service: Local Switching
Purpose:
Bring access services (e.g OLT) into BNG with redundancy
VRFs
BNG
Local Switching is
VLANs EVPN preferred on matching
VLAN
A1 MPLS A2
rewrite
VLANs
OLT OLT OLT x1000
Backup connectivity
over EVPN ELINE
ELINE
(backup tunnel)
Flexible Cross-Connect (FXC) - Testbed
RR103 RR104
LACP R39
H2
R38 R35
R37 R34
H1 R36
Config: FXC VLAN-Unaware & VLAN-Aware
R36/R38/R39 VLAN-Unaware
l2vpn
RR103 RR104
flexible-xconnect-service vlan-unaware 600
interface Bundle-Ether100.10
interface Bundle-Ether100.20 LACP R39
neighbor evpn evi 600 target 363839
!
H2
R36/R38/R39 VLAN-Aware R38 R35

R36#show run l2vpn
l2vpn
flexible-xconnect-service vlan-aware evi 600
interface Bundle-Ether100.10
interface Bundle-Ether100.20 R37 R34
!
H1 R36
VLAN-Unaware VLAN-Aware
R36#show l2vpn flexible-xconnect-service R36#show l2vpn flexible-xconnect-service
Legend: ST = State, UP = Up, DN = Down, AD = Admin Down, UR = Unresolved, Legend: ST = State, UP = Up, DN = Down, AD = Admin Down, UR = Unresolved,
SB = Standby, SR = Standby Ready, (PP) = Partially Programmed SB = Standby, SR = Standby Ready, (PP) = Partially Programmed
Flexible XConnect Service Segment Flexible XConnect Service Segment

Name ST Type Description ST Name ST Type Description ST
------------------------- ---------------------------------- ------------------------- ----------------------------------
600 UP AC: BE100.10 UP evi:600 UP AC: BE100.10 UP
AC: BE100.20 UP AC: BE100.20 UP
PW: EVPN 600,363839,64011 UP PW: EVPN 600 UP
--------------------------------------------- ---------------------------------------------
R36#show mpls forwarding labels 64011 R36#show evpn internal-label
Local Outgoing Prefix Outgoing Next Hop Bytes VPN-ID Encap Ethernet Segment Id EtherTag Label
Label Label or ID Interface Switched ---------- ------ --------------------------- -------- --------
------ ----------- ------------------ ------------ --------------- ------------
64011 64039 EVPN:600 3.3.3.38 0 600 MPLS 0038.3900.0000.0000.1100 10 64012
64037 EVPN:600 3.3.3.39 0 Summary pathlist:
0x02000006 3.3.3.38 64041
0x02000007 3.3.3.39 64040
600 MPLS 0038.3900.0000.0000.1100 20 64013

Summary pathlist:
0x02000006 3.3.3.38 64041
0x02000007 3.3.3.39 64040
VLAN-Unaware VLAN-Aware
R36#show bgp l2vpn evpn rd 3.3.3.36:600 [1][0038.3900.0000.0000.1100][363839]/120 R36#show bgp l2vpn evpn rd 3.3.3.36:600
Thu Jun 6 05:40:06.781 UTC
BGP routing table entry for [1][0038.3900.0000.0000.1100][363839]/120, Route Distinguisher: 3.3.3.36:600 Status codes: s suppressed, d damped, h history, * valid, > best
Versions: i - internal, r RIB-failure, S stale, N Nexthop-discard
Process bRIB/RIB SendTblVer Origin codes: i - IGP, e - EGP, ? - incomplete
Speaker 105 105 RT-1 Target/Service-id Network Next Hop Metric LocPrf Weight Path
Last Modified: Jun 6 05:32:38.947 for 00:07:28 Route Distinguisher: 3.3.3.36:600 (default for vrf fxc:evi:600)
Paths: (2 available, best #1) ESI R38/R39 *> [1][0036.3700.0000.0000.1100][10]/120
Not advertised to any peer 0.0.0.0 0 i
Path #1: Received by speaker 0 *> [1][0036.3700.0000.0000.1100][20]/120
Not advertised to any peer 0.0.0.0 0 i
Local *>i[1][0038.3900.0000.0000.1100][10]/120
3.3.3.38 (metric 30) from 3.3.3.103 (3.3.3.38) 3.3.3.38 100 0 i
Received Label 64039 * i 3.3.3.39 100 0 i
Origin IGP, localpref 100, valid, internal, best, group-best, import-candidate, imported, rib-install *>i[1][0038.3900.0000.0000.1100][20]/120
Received Path ID 0, Local Path ID 1, version 103 3.3.3.38 100 0 i
Extended community: RT:1:600 * i 3.3.3.39 100 0 i
Local
3.3.3.39 (metric 30) from 3.3.3.103 (3.3.3.39)
Dot1q tag
dot1q tag ID
ID
Extended community: RT:1:600
What will be tag ID value in case of QinQ?
Originator: 3.3.3.39, Cluster list: 3.3.3.103 example:
interface Bundle-Ether100.10 l2transport
encapsulation dot1q 10 second-dot1q 100
!
[1][0038.3900.0000.0000.1100][41060]/120
Formula: tag ID
(first_tag_value * 4096) + second_tag_value
(10 * 4096) + 100 = 41060
R36: EVPN-VPWS Instance View
R36#show evpn evi vpn-id 600 detail R36#show evpn evi vpn-id 600 detail
Thu Jun 6 06:25:06.940 UTC
---------- ------ ---------------------------- ------------------- VPN-ID Encap Bridge Domain Type
600 MPLS VPWS:600 VPWS (vlan-unaware) ---------- ------ ---------------------------- -------------------
Stitching: Regular 600 MPLS fxc:600 VPWS (vlan-aware)
Unicast Label : 0 Stitching: Regular
Multicast Label: 0 Unicast Label : 0
Flow Label: N EVPN-VPWS Multicast Label: 0
• No RT2 – MAC Flow Label: N EVPN-VPWS
Forward-class: 0 Control-Word: Enabled • No RT2 – MAC
Advertise MACs: No • No RT3 - BUM Forward-class: 0
Advertise BVI MACs: No Advertise MACs: No • No RT3 - BUM
Aliasing: Enabled Advertise BVI MACs: No
UUF: Enabled Aliasing: Enabled
Re-origination: Enabled UUF: Enabled
Multicast source connected: No Re-origination: Enabled
Statistics: Statistics:
Packets Sent Received Packets Sent Received
Total : 0 0 Total : 0 0
Unicast : 0 0 Unicast : 0 0
BUM : 0 0 BUM : 0 0
Bytes Sent Received Bytes Sent Received
Total : 0 0 Total : 0 0
Unicast : 0 0 Unicast : 0 0
BUM : 0 0 BUM : 0 0
RD Config: none RD Config: none
RD Auto : (auto) 3.3.3.36:600 RD Auto : (auto) 3.3.3.36:600
RT Auto : 1:600 RT Auto : 1:600
Route Targets in Use Type Route Targets in Use Type
------------------------------ --------------------- ------------------------------ ---------------------
1:600 Import 1:600 Import
©
20221:600 its
Cisco and/or affiliates.
All rights
reserved.
Cisco
Export
Confidential 1:600 Export
EVPN Interconnect/Migration
(L2 Services)
EVPN L2 Interconnect – Let’s connect everything together
Everything in one Bridge Domain
• Legacy L2: REP, G8032, STP, etc.
LACP
• VPLS VPWS
• EVPN-VXLAN/EVPN-MPLS CE A3
• EoMPLS(PW)
• Ethernet – MultiHomed, SingleHomed
Leaf
VM
A3 DCI/PE DCI/PE
Spine Spine
PE1
Leaf
VM
STP/REP/ MPLS MPLS Core
A1 G.8032…. Core
Leaf
VM DCI/PE DCI/PE
PE2
A2 EVPN - VXLAN
EVPN-MPLS
EVPN - MPLS
A1 A2
VPLS
EVPN & VPLS Seamless
Integration - Migration
VPLS & EVPN Seamless Integration - Migration
VFI1 is by default in Split Horizon Group 1
CE2 R37 R39 CE4
• SHG1 protects loops in MPLS Core
• Full Mesh of pseudowires(PW) is required
for Any-to-Any forwarding
R36 MPLS
VFI1
PW_R37 UP
BD1 PW_R38 UP
PW_R39 UP
CE1
R38 CE3
l2vpn
bridge group 100
bridge-domain 100
vfi 1
neighbor x.x.x.37 pw-id 37
!
!
!
!
CE2 R37 R39 CE4
R36 MPLS
VFI1
PW_R37 UP
EVI100 is also by default in Split Horizon Group 1
BD1 PW_R38 UP • R36 doesn’t forward data between VFI1 and EVI100
CE1 X PW_R39 UP
EVI100 R38 CE3
l2vpn
bridge group 100
bridge-domain 100
vfi 1
!
!
!
evi 100
!
CE2 R37 R39 CE4
R36 MPLS
VFI1
PW_R37 UP
EVI1 is also by default in Split Horizon Group 1
BD1 PW_R38 DOWN • R36 doesn’t forward data between VFI1 and EVI100
CE1 X PW_R39 UP
BGP EVPN
EVI100 R38 CE3 R36&R38 run BGP EVPN
• PW_R38 goes DOWN
• Data Forwarding between R36 and R38 via EVI100
l2vpn
bridge group 100
bridge-domain 100
vfi 1
!
!
!
evi 100
!
PW -> EVPN-VPWS Seamless
Migration
EVPN-VPWS/Legacy-PW Seamless Migration
Supported Modes
CE1 PE38 MPLS PE39 CE2 Discovery: Static/BGP-AD
Signaling: LDP, BGP
LDP based PW
R38 Configuration
l2vpn
xconnect group test
p2p test
interface TenGigE0/0/0/0
neighbor ipv4 3.3.3.39 pw-id 10

------------------------ ----------------------------- -----------------------------
test test UP Te0/0/0/0 UP 3.3.3.39 10 UP
----------------------------------------------------------------------------------------
CE1 PE38 MPLS PE39 CE2
LDP based PW
R38 Configuration
l2vpn
xconnect group test Allows Tengig0/0/0/0 to be migrated
p2p test
vpws-seamless-integration
Existing LDP based PW is UP and forwarding data
New EVPN-VPWS service is ready and is signaled via BGP EVPN AF
p2p test-new
neighbor evpn evi 1000 service 10

------------------------ ----------------------------- -----------------------------
test test UP Te0/0/0/0 UP 3.3.3.39 10 UP
----------------------------------------------------------------------------------------
test test-new DN Te0/0/0/0 UP EVPN 1000,10,None DN
----------------------------------------------------------------------------------------
R38#show bgp l2vpn evpn rd 3.3.3.38:1000

Route Distinguisher: 3.3.3.38:1000 (default for vrf VPWS:1000)
*> [1][0000.0000.0000.0000.0000][10]/120
0.0.0.0 0 i
CE1 PE38 MPLS PE39 CE2
LDP based PW - DOWN

EVPN-VPWS - UP
R38 Configuration R39 Configuration
EVPN-VPWS is UP
l2vpn
xconnect group test
l2vpn LDP PW is Down and service is in “Seamless Inactive” mode
p2p test
xconnect group test p2p test can be removed
p2p test
p2p test-new
p2p test-new

------------------------ ----------------------------- -----------------------------
test test DN Te0/0/0/0 SB(SI) 3.3.3.39 10 UP
----------------------------------------------------------------------------------------
test test-new UP Te0/0/0/0 UP EVPN 1000,10,3.3.3.39 UP
----------------------------------------------------------------------------------------
R38#show bgp l2vpn evpn rd 3.3.3.38:1000

Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 3.3.3.38:1000 (default for vrf VPWS:1000)
*> [1][0000.0000.0000.0000.0000][10]/120
0.0.0.0 0 i
* ©i 2022 Cisco and/or its affiliates.3.3.3.39
All rights reserved. Cisco Confidential 100 0 i
Usecases
New Node (PE40) insertion/replacement
LDP PW #1 PE38 EVPN-VPWS Seamless Migration configuration

CE1 PE38 PE39 CE2
#2 PE40 EVPN-VPWS Configuration
PE40 -> CE2 AC is down (not-connected/down)
PE38 <-> PE39 LDP PW is UP
PE40
#3 CE2 -> PE39 link remove and connect to PE40
PE38 <-> PE39 PW DOWN
PE40 -> Signal EVPN-VPWS
# PE38 <-> PE40 EVPN-VPWS UP
CE1 PE38 LDP PW PE39 CE2
EVP
N -VPW
S PE40
Usecases
Active/Backup PW – Multi-Homed CE
• CE Ethernet Bundle to PE39/40 with maximum link = 1
LDP PW • Link to PE40 is not active
CE1 PE38 PE39 CE2
Backu #1 PE38 EVPN-VPWS Seamless Migration configuration
p PW #2 PE40 EVPN-VPWS Configuration
PE40 PE40 -> CE2 AC is down (not active)
PE38 <-> PE39 LDP PW is UP
#3 CE2 changes ethernet bundle link priorities
PE38 <-> PE39 PW DOWN
PE40 -> Signal EVPN-VPWS
CE1 PE38 LDP PW PE39 CE2 # PE38 <-> PE40 EVPN-VPWS UP
EVP
N -VPW
S PE40
EVPN & VPLS/VPWS
Interconnect
EVPN & VPLS Interconnect
CE2 A2 R37 PE2

LACP
MPLS Core/Access MPLS Core CE3
R36 PE1
CE1 A1
VPLS EVPN
R36/R37 Configuration R36 Configuration R37 Configuration

evpn l2vpn l2vpn
evi 100 bridge group 100 bridge group 100
advertise-mac bridge-domain 100 bridge-domain 100
! access-vfi 1 access-vfi 1
virtual vfi 1 neighbor x.x.x.A1 pw-id 1 neighbor x.x.x.A1 pw-id 10
ethernet-segment ! !
identifier type 0 11.11.11.11.11.11.11.11.11 neighbor x.x.x.A2 pw-id 2 neighbor x.x.x.A2 pw-id 20
! !
! !
Virtual Ethernet Segment (vES) evi 100 evi 100
• VPLS is Single-Active Access to EVPN
Virtual Ethernet-Segment (vES)
R36#show evpn ethernet-segment detail

------------------------ ---------------------------------- --------------------
0011.1111.1111.1111.1111 VFI:1 3.3.3.36
3.3.3.37
Virtual Access :
Name : VFI_1
State : Up
Num PW Up : 1
ESI type : 0
Value : 11.1111.1111.1111.1111
Source MAC : 0000.0000.0000 (N/A)
Topology :
Operational : MH, Single-active
Configured : Single-active (AApS) (default)
Forwarders : 2
Permanent : 0
Elected : 2
Not Elected : 0
MAC Flushing mode : Invalid
64009 : nexthop 3.3.3.37
EVPN & VPWS (Active/Backup) Interconnect
R37 PE2
Backup-PW LACP
CE1 A1 MPLS Core/Access MPLS Core CE3
Active-PW
R36 PE1
EVPN
R36 Configuration R37 Configuration

l2vpn l2vpn
bridge group 100 bridge group 100
bridge-domain 100 bridge-domain 100
neighbor x.x.x.A1 pw-id 1 neighbor x.x.x.A1 pw-id 10
! !
evi 100 evi 100
• VPWS Active/Backup is Single-Homed from EVPN point of view => VPWS ESI = 0
• A1 Configuration without modification
EVPN & VPWS (Static-Anycast) Interconnect
Anycast Pseudonode R3637
R37 PE2
MPLS Core/Access LACP
CE1 A1 MPLS Core CE3
Active-PW
R36 PE1
EVPN
A1 Configuration R36/R37 Configuration R36/R37 Configuration

l2vpn evpn l2vpn
xconnect group 100 evi 100 bridge group 100
p2p 100 advertise-mac bridge-domain 100
interface TenGigE0/0/0/0 ! neighbor x.x.x.A1 pw-id 1
neighbor ipv4 x.x.36.37 pw-id 1 virtual neighbor x.x.x.A1 pw-id 1 mpls static label local 3637 remote 100
mpls static label local 100 remote 3637 ethernet-segment !
identifier type 0 11.11.11.11.11.11.11.11.11 evi 100
Virtual Ethernet Segment (vES)

• VPWS is All-Active Access to EVPN
EVPN ETREE
EVPN ETREE – RT Constrains (Scenario 1a) 7.2.1
• Host connected to Leaf can talk ONLY to device connected to Root
• H1, H2, H3 can talk to H4
• H1, H2, H3 CANNOT talk to each other Leaf Additional Configuration

Root Configuration Prevents H1 and H2 to talk locally
evpn l2vpn
evi 100 bridge group evpn
Leaf4 bgp bridge-domain evpn100
route-target export 1:1000 interface TenGigE0/0/0/0
route-target import 1:1000 split-horizon group
route-target import 1:100 !
H3 ! interface Bundle-Ether100
split-horizon group
!
Leaf3
MPLS Root1 H4
H2
Leaf2
Leaf Configuration
evpn
H1 evi 100
bgp
route-target export 1:100
Leaf1 route-target import 1:1000
!
etree
© 2022 Cisco and/or its affiliates. All rights reserved. Cisco Confidential rt-leaf <- MAC Synchronization
!
EVPN ETREE Leaf Label (Scenario 1b)
Root Configuration
No specific Root Configuration
l2vpn
bridge group test • ASR9k/NCS add Leaf ACs to SHG2 automatically
Leaf4 bridge-domain test => Prevents local Leaf to Leaf AC forwarding
!
evi 300
H3
Leaf3
MPLS Root1 H4
H2
Leaf2
Leaf Configuration
l2vpn
H1 bridge group test
bridge-domain test
etree
leaf
Leaf1 !
!
evi 300
EVPN ETREE Leaf Label (Scenario 1b) - BUM
Leaf Configuration
l2vpn
Each Leaf (device with at least one Leaf AC) advertises RT1 per-ESI
bridge group test with ESI 0 with ETREE extended community to distribute ETREE Label
bridge-domain test
etree R28#show bgp l2vpn evpn rd 1.1.1.28:0 [1][1.1.1.28:1][0000.0000.0000.0000.0000][4294967295]/184
leaf Wed Mar 23 03:41:36.734 UTC
! Versions:
interface Bundle-Ether100 Process bRIB/RIB SendTblVer
! Speaker 1481327 1481327
Local Label: 0
evi 300 Last Modified: Mar 23 03:21:20.580 for 00:20:17
0.2
Leaf4 Advertised to update-groups (with more than one peer):
0.2
Local
0.0.0.0 from 0.0.0.0 (1.1.1.28)
H3 Received Path ID 0, Local Path ID 1, version 1481327
Extended community: EVPN E-TREE:0x00:24010 RT:1:3000
Leaf3 ETREE Label works same as Split-Horizon Label (SHL)

SHL prevents BUM forwarding between two ACs with the same ESI
H2 ETREE Label prevents forwarding between Leaves ACs
Leaf2 Leaf to Leaf BUM traffic has ETREE Label

If Traffic with ETREE label is received cannot be forwarded to Leaf AC
H1 Root to Leaf or Leaf to Root BUM traffic doesn’t have ETREE label
BUM between Root <-> Leaf is allowed
Leaf1
EVPN ETREE Leaf Label (Scenario 1b) - Unicast
Leaf Configuration
l2vpn
Leaf Advertises local MAC with ETREE extended community
bridge group test Same extended community was used to distribute ETREE Label
bridge-domain test
RP/0/RSP0/CPU0:R28#show bgp l2vpn evpn bridge-domain test [2][0][48][682c.7b24.c63d][0]/104
etree
Wed Mar 23 04:13:10.244 UTC
leaf BGP routing table entry for [2][0][48][682c.7b24.c63d][0]/104, Route Distinguisher: 1.1.1.28:300
! Versions:
Speaker 1481349 1481349
! Local Label: 24012
evi 300 Last Modified: Mar 23 03:21:48.580 for 00:51:22
0.2
Leaf4 Advertised to update-groups (with more than one peer):
0.2
Local
0.0.0.0 from 0.0.0.0 (1.1.1.28)
H3 Received Path ID 0, Local Path ID 1, version 1481349
Extended community: SoO:1.1.1.28:300 EVPN E-TREE:0x01:0 RT:1:300
EVPN ESI: 0026.2826.2826.2826.2802
Leaf3 ETREE Label is set to 0, but Leaf Flag is set to 1
H2 Unicast traffic is filtered by ingress node

If traffic is originated from Leaf AC and destination is local/remote Leaf AC frame is dropped
Leaf2
H1
Leaf1
EVPN ETREE Leaf Label (Scenario 2) per-AC
ASR9k only Root/Leaf Configuration
l2vpn
bridge group test
bridge-domain test
interface Bundle-Ether100 <- interface to H4
Leaf4 interface Bundle-Ether200 <- interface to H5
etree
leaf
!
H3 !
evi 300
H5
Leaf3 AC
af
Le
MPLS
H2 Root
Leaf2 Leaf R H4
oot A Leaf Configuration
C
Same as Scenario 1b
H1
l2vpn
bridge group test
bridge-domain test
Leaf1 etree
leaf
!
!
evi 300
EVPN ETREE Leaf Label (Scenario 2) per-AC
ASR9k only
Root/Leaf Configuration Leaf node works same as Leaf in Scenario 1b
NCS interoperate with ASR9k leaf/root (scenario 2)
l2vpn
bridge group test
bridge-domain test Root/Leaf node advertises RT1 per-ESI
interface Bundle-Ether100 <- interface to H4 with ESI 0 with ETREE extended community to distribute ETREE Label
interface Bundle-Ether200 <- interface to H5
etree
leaf BUM Traffic from Leaf to Root/Leaf must be tagged by ETREE label
!
! Root/Leaf node forward this traffic to root AC (H4)
evi 300 Root/Leaf node prevents forwarding to leaf AC (H5)
BUM Traffic from Root/Leaf to Leaf:

H5 tagged by ETREE Label if originated by Leaf AC (H5)
AC
af not tagged by ETREE label if originated by Root AC (H5)
Le
Root Root/Leaf advertises local MAC:
Leaf R H4 with ETREE extended community if originated by Leaf AC (H5)
oot A
C
without ETREE extended community if originated by Root AC (H4)
Unicast traffic is filtered by ingress node

If traffic is originated from Leaf AC and destination is local/remote Leaf AC frame is dropped
EVPN ETREE Summary
Scenario 1a: RT Constrains is simple and HW “friendly”
Unicast/BUM filtering by ingress node => scale benefit
doesn’t support IRB
Scenario 1b: Simple configuration, but additional ETREE label must be imposed for BUM
BUM filtered by egress node
Support IRB
Scenario 2: Same principle as Scenario 1b

ASR9k allows to combine Root/Leaf ACs in the same Bridge-Domain
Support IRB
EVPN FRR
Fast Convergence (FRR Data Plane) - Core
Core Failure (Link/Node) – PIC Core
Technology: RSVP-TE/LFA/rLFA/TI-LFA
Transport: IGP -> MPLS, SRv6
Overlay Service: Service Independent
Device: P-Router, Spine
PE2 PE4 L2
X P2 X S2
X
P1 X
S1
PE1 PE3 L1
Fast Convergence (FRR Control Plane) – DC Leaf/TOR
MAC Mobility
VM/MAC Move
Technology: EVPN Mac Mobility (EVPN RT-2)
Transport: Transport Independent
Overlay Service: EVPN MAC IP ESI Seq. Next-Hop
Device: Leaf/TOR MAC-1 IP-1 0 1 Leaf-3/4
Sequence number is incremented and

L4 Next-hop is changed to Leaf-3/4
VM1
L3 S2
Move
Sequence number and Next-Hop value
will be changed after the host move
L2 S1
VM1
MAC-1 IP-1 0 0 Leaf-1/2 L1
Fast Convergence (FRR CP/DP) – Edge/Leaf/TOR
Leaf/TOR Failure (Link) – EVPN Mass Withdraw

Technology: EVPN RT1 Mass Withdraw
Transport: Transport Independent
Overlay Service: EVPN
Device: Leaf/TOR/Access/Edge
I1 PE2 PE4 X
MAC-CE1 -> ESI1 -> PE1
ES -> PE2
P2
CE1
X P1
ES
I1 PE1 PE3
RT1 ESI1 Mass Withdraw
Fast Convergence (FRR Data Plane) – Edge L3VPN
Edge Failure (Link) – BGP PIC Edge

Technology: BGP PIC Edge
Transport: MPLS, SRv6 (Transport Independent)
Overlay Service: L3VPN
Device: Access/PE
BGP CE-PE is mandatory!!!
B GP
e PE2
P2
CE1 L3VPN
X P1
eB PE1
G P
Fast Convergence (FRR Data Plane) – Edge L2VPN NEW!
Available in IOS-XR 7.3.1
Edge Failure (Link) – EVPN FRR

Technology: EVPN FRR
Transport: Transport Independent (7.3.1 MPLS)
Overlay Service: EVPN
Device: Access/PE/Leaf/TOR
All-Active Single-Active
PE2 PE2
P2 P2
CE1 EVPN CE1 EVPN

X P1 X P1
PE1 PE1
Fast Convergence (EVPN FRR Data Plane) – Edge
• Single-Active NDF filter traffic in both directions

• Re-Directed traffic will be re-directed back to PE1 (L3 Loop) or dropped
• Solution is to bypass NDF => Only redirected packet can bypass NDF!
• Extra FRR label is used to bypass NDF
• FRR Label is used for both All-Active and Single-Active access
PE2 F PE2
ND
P2 P2
CE1 EVPN CE1 EVPN

X P1 X P1
PE1 DF PE1
EVPN FRR - Configuration
evpn evpn
ethernet-segment ethernet-segment
identifier type 0 36.37.36.37.36.37.36.37.01 identifier type 0 36.37.36.37.36.37.36.37.01
convergence load-balancing-mode single-active
reroute convergence
reroute
Local Bypass
R37#show mpls forwarding

28104 Pop PW(127.0.0.1:432181084161) BE100 point2point 0
RT-1 Per EVI/ESI Ethernet Auto-Discovery
FRR Label
L36#show bgp l2vpn evpn rd 3.3.3.36:100 [1][0036.3736.3736.3736.3701][0]/120

0.0.0.0 from 0.0.0.0 (3.3.3.36)
Extended community: EVPN ESI Label:0x00:28104 RT:1:100 RT-1 Ethernet Segment Identifier (ESI)
Local
3.3.3.37 (metric 30) from 3.3.3.103 (3.3.3.37) Aliasing Label
Extended community: EVPN ESI Label:0x00:28104 RT:1:100
RP/0/RP0/CPU0:L36#
EVI
RT
FRR Label
DF Election Convergence Improvements
evpn
ethernet-segment
identifier type 0 36.37.36.37.36.37.36.37.01
load-balancing-mode single-active BGP Next-Hop Tracking for RT4
convergence Node Failure Convergence Improvement
nexthop-tracking
reroute
NTP Timestamping for RT4
R37#show evpn ethernet-segment carving detail

Service Carving Synchronization:
Mode : NTP_SCT
Peer Updates :
3.3.3.36 [SCT: 2020-10-28 12:57:47:456146]
3.3.3.37 [SCT: 2020-10-28 12:57:47:451599] NTP Timestamping for RT4
R37#show ntp status

Clock is synchronized, stratum 3, reference is 10.255.11.1
R37#show bgp l2vpn evpn rd 3.3.3.36:0 [4][0036.3736.3736.3736.3701][32]

3.3.3.36 (metric 30) from 3.3.3.103 (3.3.3.36)
Origin IGP, localpref 100, valid, internal, best, group-best, import-candidate, not-in-vrf
Extended community: EVPN ES Import:3637.3637.3637 DF Election:0:0x0008:0 EVPN NTP: 3812880149.4488
Virtual Ethernet-Segment vESI
Virtual Ethernet Segment (vESI) AC-based
Regular Main-Port based ESI is configured per-main port => BE100
ESI1
CE1 PE25 BE100.10 and BE100.20 inherit BE100 main-port ESI
PE28 is not able to split same ESI between PE25 and PE30
BE100.10
L2 ESI1
MPLS vESI allows to configure ESI per-AC => each sub-interface has own ESI
Switch
PE28 Each Sub-interface BE100.10 and BE100.20 runs DF Election independently
BE100.20
ESI2
ESI2
CE2 PE30
PE28 - Configuration
evpn
virtual interface Bundle-Ether100.10
ethernet-segment
identifier type 0 28.28.28.28.28.28.28.28.28
!
Versions:
ESI1
Speaker 1481392 1481392
CE1 PE25 Local Label: 0
Last Modified: Mar 23 05:11:53.580 for 00:01:05
BE100.10 0.2
ESI1
L2 MPLS
Switch
PE28 Advertised to update-groups (with more than one peer):
0.2
BE100.20
Local
ESI2
0.0.0.0 from 0.0.0.0 (1.1.1.28)
Extended community: EVPN ESI Label:0x00:24014 Router MAC:00a7.4213.3f2c RT:1:300
ESI2
CE2 PE30
RP/0/RSP0/CPU0:R28#show bgp l2vpn evpn rd 1.1.1.28:0 [4][0028.2828.2828.2828.2828][32][1.1.1.28]/128
Wed Mar 23 05:14:31.116 UTC
BGP routing table entry for [4][0028.2828.2828.2828.2828][32][1.1.1.28]/128, Route Distinguisher: 1.1.1.28:0
Versions:
PE28 - Configuration Speaker 1481388 1481388
Last Modified: Mar 23 05:11:50.580 for 00:02:41
evpn 0.2
virtual interface Bundle-Ether100.10 Path #1: Received by speaker 0
ethernet-segment Advertised to update-groups (with more than one peer):
0.2
identifier type 0 28.28.28.28.28.28.28.28.28 Local
! 0.0.0.0 from 0.0.0.0 (1.1.1.28)
Extended community: EVPN ES Import:2828.2828.2828 Router MAC:00a7.4213.3f2c DF Election:0:0x0008:0
RT1 per-ESI and RT4 are allocated for each sub-interface
ESI1 In case of main port BE100 failure BGP must withdraw:
CE1 PE25 RT1 and RT4 for each sub-interface
BE100.10
ESI1
Convergence would be affected!
L2 MPLS
Switch
PE28
BE100.20
ESI2 Both RT1 per-ESI and RT4 are tagged by Router MAC
ESI2
PE28 generates also Grouping ESI (gESI)
CE2 PE30 gESI is autogenerated, but can be also configured manually
gESI auto-generation: ESI type 3 + interface MAC + ffff
PE28 - Configuration R28#show bgp l2vpn evpn rd 1.1.1.28:0 [1][1.1.1.28:2][0300.a742.133f.2cff.ffff][4294967295]/184

Wed Mar 23 05:13:38.554 UTC
BGP routing table entry for [1][1.1.1.28:2][0300.a742.133f.2cff.ffff][4294967295]/184, Route Distinguisher: 1.1.1.28:0
evpn Versions:
virtual interface Bundle-Ether100.10 Speaker 1481391 1481391
ethernet-segment Local Label: 0
identifier type 0 28.28.28.28.28.28.28.28.28 Last Modified: Mar 23 05:11:53.580 for 00:01:45
! Advertised to update-groups (with more than one peer):
0.2
0.2
Local
0.0.0.0 from 0.0.0.0 (1.1.1.28)
© 2022 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Extended community: EVPN ESI Label:0x00:0 RT:1:300
gESI adding another indirection to forwarding resolution
ESI1
CE1 PE25 From Remote node point of view:
BE100.10
MAC_CE1 -> vESI_BE100.10 -> PE28
MPLS Remote
ESI1
L2
Switch
PE28 PE -> PE25
BE100.20
ESI2
With gESI:
ESI2
CE2 PE30 MAC_CE1 -> gESI_BE100 -> vESI_BE100.10 -> PE28
-> PE25
In case of BE100 failure PE28 withdraw RT1 BE100 gESI

PE28 - Configuration PE25/PE30 can start DF election for all shared vESI
evpn
Remote PE remove PE28 from gESI_BE100 FW chain
virtual interface Bundle-Ether100.10
ethernet-segment
identifier type 0 28.28.28.28.28.28.28.28.28
!
EVPN Multicast sync
RT7/8
EVPN – Native Multicast in the Network Fabric
Source
BL BL
PIM State sync
in EVPN
SP SP
L3
---
L2 L L L L
EVI-x
IGMP Join / Leave
EVI-y
C C
IRB
mcast Receiver Receiver
evpn 199
EVPN Selective Multicast
RT6
EVPN ELAN L2 Selective Multicast – Route-Type 6
RT6 to IGMP Join (Proxy) not supported
Receiver1
Multicast must be received by PE5/6 from source without IGMP join
PE5/PE6 selectively ingress-replicate multicast to PE1 and PE2

IGMP Join
PE1
I1
ES EVPN RT6
Selective Multicast
EVPN RT7
Receiver2
IGMP Join sync
ES
PE2 PE5
I1
Receiver3
Source1
PE3 PE6
Receiver7
PE4
EVPN Centralized GW
CGW
EVPN Centralized Gateway (CGW)
CGW - Configuration
evpn
virtual access-evi
ethernet-segment
A1 identifier type 0 77.77.77.77.77.77.77.77.77
l2vpn
bridge group test
CE1 bridge-domain test
access-evi 300
routed interface BVI300
A2 CGW1
L2 EVPN L3 VPN
Core
A3 CGW2 Access - Configuration

evpn
evi 300
CE2
advertise-mac
l2vpn
A4 bridge group test
bridge-domain test
!
evi 300
EVPN Centralized Gateway (CGW)
R28#show evpn ethernet-segment

------------------------ ---------------------------------- --------------------
0077.7777.7777.7777.7777 Access-EVI:all 1.1.1.26
A1 1.1.1.28
RP/0/RSP0/CPU0:R28#show arp vrf a

CE1 -------------------------------------------------------------------------------
0/0/CPU0
-------------------------------------------------------------------------------
Address Age Hardware Addr State Type Interface
A2 CGW1 192.168.250.1 - a011.1111.1111 Interface ARPA BVI300
192.168.250.10 - 28ac.9ea7.d41b EVPN_SYNC ARPA BVI300
L2 EVPN L3 VPN
Core
CGW in Single-Active mode from Access-to-CGW (South->North)
Based on Access-EVI DF election NDF CGW BVI is added to Core SHG
A3 CGW2
prevents traffic from access-EVI go to BVI
allows traffic from BVI to Access-EVI
CE2
Single-Active South->North
All-Active North->South
A4
Distributed vs Centralized Gateway
• Distributed Anycast Gateway is our priority!
• Best Scalable solution
• Optimal L2/L3 forwarding
EVPN Headend
L3 EVPN-Headend Vlan-Unaware
HE Modes (PE):
1. Single-Active/Port-Active from Access and All-Active from Core (default)
2. All-Active
A2 PE2
PE1/PE2 Configuration
HE evpn
EVPN VPNv4/6 interface PW-Ether 1

ethernet-segment
CE1 identifier type 0 9.8.7.6.5.4.3.2.1
HE
A1 PE1
l2vpn
xconnect group xc100
p2p evpn-headend
Access Modes (A): interface PW-Ether1
1. All-Active EVPN-VPWS neighbor evpn evi 1 target 1 source 1
2. Port-Active EVPN-VPWS
3. Single-Active (main port only)
Pref-Based/AC Driven DF
election
Access-Driven DF Election
EVPN-VPWS only!!!!
PE3
PE1/PE2/PE3 Configuration
evpn
CE1 PE2 EVPN ethernet-segment
identifier type 0 36.37.36.37.36.37.36.37.01
load-balancing-mode port-active
service-carving preference-based
access-driven
weight <value>
PE1
Transport Integration
EVPN & EVPN-VPWS On-Demand Next Hops (ODN) 7.2.1
DNX Platform
• RT1 and RT3 are advertised with color (color specifies SLA)
R37 R39
CE1 MPLS CE2

R36 Configuration
segment-routing
R36 R38
traffic-eng
on-demand color 100
dynamic
metric
type igp R38 Configuration
R36 Verification route-policy C100
if evpn-route-type is 1 or evpn-route-type is 3 then
R36#show bgp l2vpn evpn rd 3.3.3.36:100 set extcommunity color c100
Route Distinguisher: 3.3.3.36:100 (default for vrf evpn100) endif
*>i[1][0038.3938.3938.3938.3901][0]/120 end-policy
3.3.3.38 C:100 100 0 i
*>i[3][0][32][3.3.3.38]/80 extcommunity-set opaque c100
3.3.3.38 C:100 100 0 i 100
end-set
R36#show segment-routing traffic-eng policy evpn

evi 100
Color: 100, End-point: 3.3.3.38 bgp
Name: srte_c_100_ep_3.3.3.38 route-policy export C100
Status: !
© 2022
Admin: up itsOperational:
Cisco and/or up forCisco
affiliates. All rights reserved. 00:03:45
Confidential !
EVPN Per-Flow Traffic Steering
R37 R39
CE1 MPLS CE2
R36 R38
R36/37 Configuration
class-map match-any test
match cos 5
end-class-map segment-routing
traffic-eng
on-demand color 100
policy-map per-flow dynamic
class test metric
set forward-class 5 type igp
!
!
interface Bundle-Ether999 !
l2transport on-demand color 1000
service-policy input per-flow per-flow
forward-class 5 color 100
EVPN Troubleshooting Hints
#1. Don’t fix what’s not broken….
Troubleshooting Hint #2
Transport Verification
• Verify End-To-End Label-Switched Path (LSP)
• show route, show cef, show mpls forwarding, etc.
BGP Control Plane
• BGP Session Status
• show bgp l2vpn evpn summary
• BGP EVPN – Route Types

• show bgp l2vpn evpn summary
EVPN Control Plane – L2
• Ethernet Segment Verification
• show evpn ethernet-segment [detail]
• Received and Advertised MAC addresses

• show evpn evi mac
• show l2route evpn mac all
• EVI Instance Summary: unicast/multicast label, RD, RT, etc..

• show evpn evi vpn-id [evi id] detail
EVPN Data Plane – L2
• Nexthop Verification - Unicast
• show l2route evpn mac all
• show evpn internal-label
• Nexthop Verification – Broadcast/Multicast

• show evpn evi inclusive-multicast detail
• Bridge-Domain Status
• show l2vpn bridge-domain bd-name [bd name]
• Bridge-Domain Local/Remote MAC addresses

• show l2vpn forwarding bridge-domain [bd group:bd name] mac-address detail location [LC location]
EVPN Control Plane – L3
• Received and Advertised Host Routes
• show l2route evpn mac-ip all
• Routes in VRF
• show route vrf [vrf name]
EVPN Data Plane – L3
• Nexthop Verification
• show route vrf [vrf name]
• show cef vrf [vrf name]
• + all L3VPN troubleshooting hints
Timers
EVPN Timers
Default
Timer Range Value Trigger Applicability Action Sequence
Postpone EVPN startup procedure and

Hold AC link(s) down to prevent CE to
Single-Homed, All-Active, Single- PE forwarding. Startup-cost-in timer
startup-cost-in 30-86400 disabled node recovered Active allows PE to set core protocols first 1
Postpone EVPN Startup procedure.

Recovery timer allows PE to set access
node recovered, Single-Homed (ESI configured), protocols (STP) before reachability
recovery 0-3600s 30s interface recovered Single-Active towards EVPN core is advertised 2
Starts after sending EVPN RT4 to
postpone rest of EVPN startup
procedure. Peering timer allows remote
PE (multihoming AC with same ESI) to
node recovered, process RT4 before DF election will
peering 0-300s 3s interface recovered All-Active, Single-Active happen 3
• Available in EVPN global configuration mode and in evpn interface sub-configuration mode
• Startup-cost-in is available in EVPN global configuration mode only
• Timers are triggered in sequence (if applicable)
• Cost-out in EVPN global configuration mode brings down AC link(s) to prepare node for reload/SW upgrade
© 2022 Cisco and/or its affiliates. All rights reserved. Cisco Confidential *node recovered – All required SW components are loaded
*interface recovered – link status is up
Summary
EVPN – Service Layering Access Aggregation Core
A AG PE P PE
Multicast CE
FXC
EVPN-HE
E-TREE A AG PE P PE
P2P
L2 Bridging
L3 Routing BL BL
IRB
SP SP
SP SP
SP SP
SP SP
L L L
L L L
L L L
L L L L
L L L L
L L L L
C
C CE CE
Conclusion
• EVPN is an very important complement to BGP based services
• BGP is Unified Services Control Plane across SP Network
• EVPN All-Active Multihomed Service with Distributed Anycast Gateway & Integration to
L3VPN simplifies SPDC/NextGen-CO/WAN Integration
NETCONF
Provisioning YANG
Programmability PCE
L2/L3VPN Services LDP BGP LDP BGP BGP
Inter-Domain CP BGP-LU BGP-LU

FRR or TE RSVP
IGP with SR
LDP IGP with SR
Intra-Domain CP
IGP
EVPN - Stay Up-To-Date
• https://e-vpn.io/
• Upcoming Conferences: https://e-vpn.io/conferences/
Additional services related
features
MPLS L3VPN Service
PE1 MPLS PE3
CE1 P1 CE2
PE2 PE4
Ethernet MPLS L3VPN Service Ethernet
L2 Frame Transport L2 Frame

MPLS Label
IP Packet Service IP Packet
MPLS Label
IP Packet
• P-Node doesn’t know about overlay service (no signaling)
• P-Node identifies overlay service (L2/L3) based on inner header
• If inner payload starts
• 0x4 => L3 IPv4 Header
IP/UDP/TCP header hash to provide per-flow ECMP
MPLS L2VPN Service – Without Control-Word
PE1 MPLS PE3
CE1 P1 CE2
PE2 PE4

MPLS Label
MPLS Label
• P-Node doesn’t know about overlay service (no signaling)
L2 Frame
• P-Node identifies overlay service (L2/L3) based on inner header
IP Packet • If inner payload starts
IP/UDP/TCP header hash to provide per-flow ECMP
• If MAC Address starts with 0x4 or 0x6 P-node expect L3VPN (no ethernet header)
Hash is calculated from “random data” with expectation of L3 IP Header
If MAC doesn’t start 0x4 or 0x6 L2 payload is correctly identified without Control-Word
Hash for L2 payload is calculated from Top Label to provide ECMP
MPLS L2VPN Service – Control-Word
PE1 MPLS PE3
CE1 P1 CE2
PE2 PE4

MPLS Label
MPLS Label
Control
Word • P-Node doesn’t know about overlay service (no signaling)
L2 Frame • P-Node identifies L2 Service based on Control-Word
IP Packet
• Hash for L2 payload is calculated from Top Label to provide ECMP
• Service Label in our example
EVPN-VPWS Layer 2 Attributes Extended Community
RFC8214 IOS-XR 7.1.1
+-------------------------------------------+
| Type (0x06) / Sub-type (0x04) (2 octets) | 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
+-------------------------------------------+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Control Flags (2 octets) | | MBZ |C|P|B| (MBZ = MUST Be Zero)
+-------------------------------------------+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| L2 MTU (2 octets) |
+-------------------------------------------+ Control-Word(C) = 4
| Reserved (2 octets) |
Primary(P) = 2
Backup(B) = 1
+-------------------------------------------+
L2 MTU is a 2-octet value indicating the MTU in bytes
EVPN Control-Word (CW)
• In IOS-XR Control-Word for L2VPN EVPN services is enabled by default
• Control-Word can be disabled per EVI
evpn
evi 100
control-word-disable
• EVPN-VPWS (ELINE) Control-Word already supported by NCS/ASR9k

• Control-Word Signaling identifies mismatch
• EVPN ELAN doesn’t provide Control Word signaling! (IETF work in progress)
EVPN ELAN prior 7.4.1

• Service between ASR9k <-> NCS => ASR9k must disable Control-Word
• Service between NCS <-> NCS => Control-Word is not supported and not used
How to Upgrade to EVPN ELAN 7.4.1+

Existing Service between ASR9k <-> NCS: ASR9k Control-Word is disabled
NCS must disable Control-Word or ASR9k must enable Control Word
Existing Service between NCS <-> NCS: Control-Word is not supported and not used
If upgrade is done per-side => NCS must disable Control-Word
DNX FEC optimization
NCS LER ECMP-FEC Optimization (L2/L3 Recursive Services)
• Forwarding chain programing of BGP multipath requires use of ECMP-FEC
• We can refer to these as “Overlay/Service/VPN” ECMP-FECs
Overlay ECMP-FECs are different from those used to program the Underlay
•
• IOS-XR implementation allocates one EVPN label per local EVI
• Equivalent to per-VRF label allocation in L3 VPN
• ECMP-FEC consumption for EVPN active/active multi-homing
• One (1) ECMP-FEC per EVI per remote active/active multi-homed UNI
• Example:
• EVPN service spanning total of 10 sites with A/A MH
• At any given PE: Overlay (service-related) ECMP-FEC consumption == 9 ECMP-FECs
• Any other MH cases (single-homed/single active), do not require an ECMP-FEC
Disclaimer: ECMP-FEC Optimization slides are stolen from Jose Liste ppt
Example: EVPN Active-Active Multi-Homing – KNOWN Unicast Segment Routing
WITHOUT ECMP-FEC LER opt. enabled
NCS 5500 – SR ECMP-FEC LER Opt.

LEM 786K ECMP-FEC 4K FEC 124K EEDB 96K*
MAC (EVI X)
ECMP-FEC
d46d.5040.0001 ID=20007 Push [EVPN_label L1]
Overlay
ID=4
num_entries=2
d47d.5050.0002 ID=20008 Push [EVPN_label L2]
“Overlay” ECMP-FEC for
BGP multipath
ip2mpls
Underlay
ECMP-FEC ID=20010 Push 16008
1.1.1.8/32 ID=7
num_entries=2 ID=20011
16008, swap 16008
Push 16008
mpls2mpls
ip2mpls
ECMP-FEC ID=20012 Push 16009
1.1.1.9/32 ID=8
16009, swap 16009
Push 16009
shared ECMP-FEC ID=20014
mpls2mpls
ID=9
Local-OIF-node_4-arp
ASIC Resources Local-OIF-node_5-arp
Dedicated ECMP-FEC per
labeled destination / BGP LEM
NHs !!! ECMP-FEC
© 2022 Cisco and/or its affiliates. All rights reserved. Cisco Confidential FEC
EEDB [*] Effective EEDB size for MPLS = 72K
Example: EVPN Active-Active Multi-Homing – KNOWN Unicast Segment Routing
WITH ECMP-FEC LER opt. enabled
NCS 5500 – SR ECMP-FEC LER Opt.

LEM 786K ECMP-FEC 4K FEC 124K EEDB 96K*
MAC (EVI X)
ECMP-FEC
d46d.5040.0001 ID=4 ID=20007 Push [EVPN_label L1, 16008]
num_entries=2
Overlay
d47d.5050.0002 ID=20008 Push [EVPN_label L2, 16009]
“Overlay” ECMP-FEC for
BGP multipath
COLLAPSED encap – push
Underlay
VPN and Transport labels
ip2mpls
shared ECMP-FEC ID=20009 Local-OIF-node_4-arp
1.1.1.8/32; push 16008 ID=7
16008; swap 16008
Local-OIF-node_5-arp
mpls2mpls
Shared “Underlay” ECMP-

ip2mpls FEC
1.1.1.9/32; push 16009

ASIC Resources
16009; swap 16009 LEM
mpls2mpls ECMP-FEC
© 2022 Cisco and/or its affiliates. All rights reserved. Cisco Confidential FEC
EEDB [*] Effective EEDB size for MPLS = 72K
TODO
• Multicast
• FXC

IOS-XR EVPN Deep Dive v3

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

IOS-XR EVPN Deep Dive v3

Uploaded by

Copyright:

Available Formats

• Check internal wiki first

• Download always latest/greatest version

• Apr 2022 – EVPN Headend

Data Center Network

BGP: VPNv4/6 VPLS Overlay

MPLS: LDP, RSVP-TE MPLS, L2 L2, IP Underlay

L2/L3VPN Services LDP BGP LDP BGP BGP

Inter-Domain CP BGP-LU BGP-LU

• MAC Flip-Flopping over Pseudowire PE2 PE4

• E.g. Port-Channel Load-Balancing does not

PE2 PE4 PE2 PE4

CE1 MPLS CE2 CE1 MPLS CE2

PE1 PE3 PE1 PE3

IP Packet Transport IP Packet L2 Frame Transport L2 Frame

Protection • Open-Standard and Multi-vendor support

[2] MAC/IP Advertisement Route ES-Import

ESI2 [4] Ethernet Segment Route Default Gateway

PE1 PE2 PE1 PE2 PE1 PE2

CE1 CE2 CE3

Single LAG at the CE Multiple LAGs at the CE Single LAGs at the CE

Physical Interface Single-Active Single-Active

Per-Node/Per-EVI RD - [BGP-RouterID]:[EVI-ID] -> Similar to VRF RD in L3VPN

EVPN RT1, RT2, RT3 R36 RD

Per-Node/Per-EVI RD: 3.3.3.36:100 Vlan2

Why more Per-Node RD? RD

P2P EVI100 P2P EVI102

CE3 PE1 PE3 CE4

EVI ID: 103, Service ID 10

Pick your side!

Unique 10-byte global identifier

MAC advertisement & learning/synchronization

LACP R37 R34

lacp system mac 3637.3637.3637 l2vpn

Ethernet Segment Id Interface Nexthops

VPN-ID Encap Bridge Domain Type

Unique per Advertising PE per EVI

Route Type specific encoding of E-VPN NLRI

Flags based on RFC6514

Ingress Replication Multicast (BUM) Label

Why extra BUM Label?

EVI-ID modulo Number of Nodes = Position

R36 is DF for EVI-100

Who will be DF for EVI-101?

Unique per Advertising PE

Route Type specific encoding of E-VPN NLRI

0x02 MAC Address portion of the ESI

Nodes which share same ESI import this route

SP2 MAC1  ESI1  Leaf1 + Leaf2

Unique per Advertising PE

Route Type specific encoding of E-VPN NLRI

0x01 Bit 0: Redundancy Mode (single active vs. all active)

3 bytes ESI MPLS Label Ethernet Segment Split-Horizon Label

Redundancy mode Split-Horizon Label

LACP R37 R34

R36 Ext-Com: 3637.0000.0000 (RT)

LACP R37 R34 Flag:0x00 All-Active

1. RT4: DF Election & Multi-Homed Ethernet

LACP R37 R34

H1 Transport Label R38-9

R36 BUM Label R38-9/EVI100

1. RT4: DF Election & Multi-Homed Ethernet

R36 SH Label R37/ESIx

BUM - Traffic BUM - Traffic