L2GOLF Deployment with N9K ToRs

DCI Series-4
danma@cisco.com
2017.10.30
This slide is targeted for L2GOLF customer production deployment with N9K
VxLAN Spines/ToRs

For detail L2GOLF theory, refer TAC TOI

EDCS-1572869 (PI)
EDCS-1548459 (PD)
L2GOLF Overview
Since 6.1.1, ASR9K can work as DCI Gateway provides L2 connectivity
between VxLAN Fabric Data Centers – L2GOLF

DCI router works as EVPN-VXLAN to EVPN-MPLS Layer 2 Gateway

L2GOLF Overview
MAC learning disabled on VxLAN port and EVI port.
Unknown unicast flooding is blocked on VxLAN port in 6.1.1, while 6.2.1 by
default doesn’t block unknown unicast flooding on VxLAN port, we can use CLI to
block.
Bridge shutdown is blocked.
EVI port can not be brought down by security features such as MAC secure and
MAC limit
NVE or VxLAN port between dual-homing PE only supports All-active load-
balancing by now, single-active is not supported.
L2GOLF Typical Design
WAN RR

MPLS SR
Core
DCI GW DCI GW
VTEP VTEP

eBGP eBGP

DC Spine DC Spine
VXLAN VXLAN
Overlay Overlay
iBGP RR RR RR RR
iBGP

VTEP VTEP
DC Leaf DC Leaf
SVI same SVI same
IP/MAC IP/MAC
L2GOLF Design Considerations-1

 Normally DCI has two separated IGP process running for MPLS side and VxLAN
side, and they have different loopback interfaces in two IGP processes.

 RFC 7432 requires Type-3 route use a common loopback address as Originating
Router's IP Address field value for all the EVIs on the PE. XR uses BGP router ID as
common ip address for all EVIs Type-3 Originating ip address field value. So we
recommend to use core-facing loopback as BGP router ID since it is routable and
has LSP in WAN MPLS side.
L2GOLF Design Considerations-2

 Same as L3GOLF, same RD is not supported for route reoriginate in L2GOLF

deployment. ToR, DCI and remote DCI need use different RD for one BD or one MAC
VRF. Suggest use RD Type-1 format or auto-RD
Loopback IPv4 address : nn

 Without Spine, ToRs can’t receive EVPN prefixes each other via DCI, the reason is
that ASR9K DCI will not reoriginate then advertise EVPN prefixes from one ToR to
another ToR, so:
(1) ToRs can establish full-mesh iBGP sessions to exchange EVPN prefixes
(2) or setup a RR for ToRs exchanging EVPN prefixes within DC
Two Multi-homing Deployment Models
Anycast VTEP Multi-homing vs. ESI-Based Multi-homing

 Depending on the capability of ToRs, ASR9K DCI supports two multi-homing

deployment models
o Anycast VTEP Multi-homing VXLAN GW model
o ESI-Based Multi-homing VXLAN GW model

 Different mechanisms inside DC fabric for multi-homing and load-balancing between

ToRs and DCIs
 Same implementation on the MPLS WAN side
N9K ToR capability

 N9K old version may not support the followings:

Anycast VTEP Multi-homing VXLAN GW model
ESI-Based Multi-homing VXLAN GW model
 Recommend upgrading to 7.0.3I6(1) or 7.0.3I7(1) latest CCO version
EVPN-VXLAN to EVPN-MPLS L2 DCI GW
Anycast VTEP Multi-homing VXLAN GW model
Anycast VTEP Multi-homing VxLAN GW

anycast
source IP
vPC L2 overlay
MPLS underlay
multi-homing
nodes share
VTEP

ToR PE
IPV4 Forwarding MPLS Forwarding IPV4 Forwarding

• Anycast VxLAN operation model:

• ToR run vPC.
• DCI PE use anycast IP for VxLAN.
• Traffic between ToR and PE is load balanced by IGP.
Anycast VTEP Multi-homing Overview

• Interwork with Cisco Nexus 9K with following assumptions

• A pair of ToRs present as a virtual VTEP by running vPC
• ToRs support manually configured Anycast remote VTEP which is different
against the next-hop IP address learned from EVPN BGP update
• A pair of ASR9K DCI use an Anycast VTEP IP to receive VXLAN
packet from ToR, but unique physical VTEP IP to send VXLAN to
ToR.
ASR 9000 EVPN-VXLAN to EVPN-MPLS L2 GW
Anycast VTEP Multi-homing overview
Anycast VTEP Multi-homing VXLAN L2 GW model
Anycast VTEP for ToR to DCI unicast

An anycast VTEP loopback and a

DCI-1 EVPN DF election
• Int lo 1

NULL ESI is used on redundant DCI Ipv4 add 5.5.5.5/32

GWs to be advertised towards the Int lo 2 VTEP: 5.5.5.5 VTEP: 6.6.6.6

fabric
Ipv4 add 2.2.2.2/32
Any-cast VTEP 2.2.2.2
interface nve1
• Appear as a single VXLAN VTEP to source-interface loopback1
fabric ToR VTEPs. anycast source-interface
loopback2
unicast
• Used by ToR to tunnel the unicast to
DCI. DCI-2

Work with existing ToR VTEP VXLAN /

Int lo 1
• Ipv4 add 6.6.6.6/32 VIP: 1.1.1.1
EVPN implementations on N9000 that VMAC: M1
don’t have support for MAC ECMP / Int lo 2
Ipv4 add 2.2.2.2/32
multi-path and mass-withdraw.
interface nve1
• Avoids the need to implement fabric source-interface loopback1
side per-ESI EAD and mass MAC anycast source-interface
loopback2
withdraw on DCI GW
Anycast VTEP Multi-homing VXLAN L2 GW model
Physical VTEP for DCI to ToR BUM
EVPN DF election
An unique physical VTEP IP will also
DCI-1
• Int lo 1

be configured on the NVE on each Ipv4 add 5.5.5.5/32 Selected DF for

BUM
DCI router to be used as the source Int lo 2 VTEP: 5.5.5.5 VTEP: 6.6.6.6
for all multicast encapsulated VXLAN
Ipv4 add 2.2.2.2/32
Any-cast VTEP 2.2.2.2

traffic. interface nve1

source-interface loopback1

Used by DCIs to tunnel the unicast to BUM

anycast source-interface
• loopback2
ToRs.
• Avoid conflict with EVPN DF selection DCI-2
based BUM blocking towards Fabric. Int lo 1
Ipv4 add 6.6.6.6/32 VIP: 1.1.1.1
• Avoids the need to implement fabric Int lo 2
VMAC: M1
side inclussive multicast route(RT-3) Ipv4 add 2.2.2.2/32

• Ingress replication on VXLAN side interface nve1

is not supported for BUM, instead,

source-interface loopback1
anycast source-interface
underlay Multicast routing(enabling loopback2

PIM) is mandatory here.

Anycast VTEP Multi-homing VXLAN L2 GW model
Physical VTEP for DCI to ToR unicast
EVPN DF election
Current ASR9000 implementation
DCI-1
• Int lo 1

has a limitation that it cannot use two Ipv4 add 5.5.5.5/32

different source IPs for unicast and Int lo 2 VTEP: 5.5.5.5 VTEP: 6.6.6.6
multicast VXLAN encapsulations for
Ipv4 add 2.2.2.2/32
Any-cast VTEP 2.2.2.2

the same VNI / BD. interface nve1

source-interface loopback1

As a result, ASR9K DCI will use unicast

anycast source-interface
• loopback2
physical VTEP IP as the source for
all VXLAN encapsulated traffic DCI-2

towards the fabric, including unicast.

Int lo 1
Ipv4 add 6.6.6.6/32 VIP: 1.1.1.1
VMAC: M1
Int lo 2
Ipv4 add 2.2.2.2/32

interface nve1
source-interface loopback1
anycast source-interface
loopback2
Anycast VTEP Multi-homing VXLAN L2 GW model
How DCI interwork with ToR with two different VTEP sources

ASR9K DCI only advertise Anycast

DCI-1 EVPN DF election
• Int lo 1
source IP as VTEP via BGP EVPN Ipv4 add 5.5.5.5/32

towards ToRs so that ToRs know to use Int lo 2 VTEP: 5.5.5.5 VTEP: 6.6.6.6
that address as Dest. VTEP for VXLAN
Ipv4 add 2.2.2.2/32
Any-cast VTEP 2.2.2.2

encapsulation. interface nve1

source-interface loopback1
• VXLAN packet received by ToRs with anycast source-interface
loopback2 unicast
ASR9K physical VTEP IP as source
VTEP will be dropped by ToRs by DCI-2
default since it’s unknown by ToRs. Int lo 1
Ipv4 add 6.6.6.6/32 VIP: 1.1.1.1
• N9K needs configure “peer-vtep” CLI VMAC: M1
to provision physical ASR9K VTEP IPs
Int lo 2
Ipv4 add 2.2.2.2/32
statically, so that traffic sourced from
these VTEPs is forwarded .
interface nve1
source-interface loopback1 ToR
anycast source-interface interface nve1
loopback2 member vni 5001
mcast-group 224.1.1.10
peer-vtep 5.5.5.5
peer-vtep 6.6.6.6
Anycast VTEP Multi-homing DCI configuration
with N9K vPC ToR
DC5: Anycast VTEP Multi-homing L2GOLF
RR

MPLS Core (SR or LDP)

BE1 BE1 BE2 BE1 BE2 BE2

BE2 BE1
DCI4_1 ASR9k DCI4_2 DCI5_1 ASR9k DCI5_2

BE13 BE14 BE23 BE24 BE14 BE23 BE24

BE13
eBGP Spine eBGP

Po13 Po23 Po14 Po24 VxLAN VxLAN Po13 Po23 Po14 Po24
Overlay Overlay
Spine2-1 Spine2-2 Spine5-1 Spine5-2

RR RR RR RR
iBGP iBGP

Leaf2-1 Leaf2-2 Leaf5-1 Leaf5-2

Po40 Po40
ESI A/A vPC

DC2: ESI-Based Multi-homing DC5: Anycast VTEP Multi-homing

DCI5 config (page-1): Anycast VTEP Multi-homing
router ospf Core router bgp 100
router-id 100.100.5.11 <<<< loopback0 bgp router-id 100.100.5.11 <<<< loopback0
segment-routing mpls address-family l2vpn evpn
area 0 !
interface Bundle-Ether1 neighbor 20.1.1.1 <<<< towards WAN RR or remote DCI
interface Bundle-Ether2 remote-as 100
interface Loopback0 update-source Loopback0
! address-family l2vpn evpn
router ospf Edge-1 import re-originate stitching-rt
router-id 100.100.5.1 <<<< loopback1 advertise l2vpn evpn re-originated
area 0 !
interface Bundle-Ether13 neighbor 100.100.5.3 <<<<< towards VxLAN Spines/ToRs
interface Bundle-Ether14 remote-as 400
interface Loopback1 ebgp-multihop 10
interface Loopback2 <<<< same ip on two DCI update-source Loopback1
! address-family l2vpn evpn
! import stitching-rt reoriginate
! multipath
route-policy pass-all in
encapsulation-type vxlan
route-policy pass-all out
advertise l2vpn evpn re-originated stitching-rt
DCI5 config (page-2): Anycast VTEP Multi-homing

redundancy interface nve1

iccp redundancy
group 11 backbone mpls
mode singleton iccp group 11
backbone !
interface Bundle-Ether1 backbone vxlan
interface Bundle-Ether2 iccp group 12
! !
group 12 source-interface Loopback1
mode singleton anycast source-interface Loopback2 <<<< same ip on two DCI
backbone !
interface Bundle-Ether13
interface Bundle-Ether14
! evpn
! interface nve1
ethernet-segment
identifier type 0 55.55.55.55.55.55.55.55.55 <<<< same on two DCI
bgp route-target aaaa.aaaa.aaaa <<<< same on two DCI
!
 DCI5 config (page-3): Anycast VTEP Multi-homing
evpn router pim
evi 6001 address-family ipv4
bgp rp-address 100.100.5.10 <<<< points to Spine PIM Anycast RP
route-target import 100:6001 !
route-target export 100:6001
route-target import 400:6001 stitching
route-target export 400:6001 stitching multicast-routing
! interface Loopback1
enable
l2vpn !
bridge group 6001 interface Loopback2
bridge-domain 6001 enable
evi 6001 !
member vni 6001 interface Bundle-Ether13
! enable
!
interface nve 1 interface Bundle-Ether14
member vni 6001 enable
mcast-group 229.1.1.1 !
host-reachability protocol bgp
!
L2GOLF DC underlay Multicast Design
DC GW

VTEP/ VTEP/
VXLAN Gateway VXLAN Gateway

PIM
Anycast RP
DC Aggregation(Spine)

VXLAN
Overlay
RR RR

VTEP VTEP VTEP

DC Access(Leaf)

Note: DCI underlay Multicast only needs enable on VxLAN side,

MPLS side doesn’t need enable multicast
 DC5 N9K Spine (page-1): PIM Anycast RP config
DC5 N9K Spine-1: DC5 N9K Spine-2:
feature ospf feature ospf
feature bgp feature bgp
feature pim feature pim

ip pim rp-address 100.100.5.10 group-list 229.1.1.0/24 ip pim rp-address 100.100.5.10 group-list 229.1.1.0/24
ip pim ssm range 232.0.0.0/8 ip pim ssm range 232.0.0.0/8
ip pim anycast-rp 100.100.5.10 100.100.5.3 ip pim anycast-rp 100.100.5.10 100.100.5.3
ip pim anycast-rp 100.100.5.10 100.100.5.4 ip pim anycast-rp 100.100.5.10 100.100.5.4

interface loopback0 interface loopback0

ip address 100.100.5.3/32 ip address 100.100.5.4/32
ip router ospf 1 area 0.0.0.0 ip router ospf 1 area 0.0.0.0
ip pim sparse-mode ip pim sparse-mode

interface loopback1 interface loopback1

ip address 100.100.5.10/32 ip address 100.100.5.10/32
ip router ospf 1 area 0.0.0.0 ip router ospf 1 area 0.0.0.0
ip pim sparse-mode ip pim sparse-mode
DC5 N9K Spine (page-2): BGP config
route-map Next-Hop-Unchanged permit 10
set ip next-hop unchange

router bgp 400

router-id 100.100.5.3
neighbor 100.100.5.1 <<<<< eBGP session with DCI routers
remote-as 100
update-source loopback0
ebgp-multihop 10
address-family l2vpn evpn
send-community
send-community extended
route-map Next-Hop-Unchanged out

neighbor 100.100.5.5 <<<<< ibgp session with ToRs

remote-as 400
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
route-reflector-client
N9K ToR vPC Diagram
 DC5 N9K ToR (page-1): vPC config
DC5 N9K ToR1: DC5 N9K ToR2:
nv overlay evpn nv overlay evpn
feature ospf feature ospf
feature bgp feature bgp
feature pim feature pim
feature interface-vlan feature interface-vlan
feature vn-segment-vlan-based feature vn-segment-vlan-based
feature lacp feature lacp
feature vpc feature vpc
feature lldp feature lldp
feature nv overlay feature nv overlay

route-map pass-all permit 10 route-map pass-all permit 10

hardware access-list tcam region vacl 0 <<<< adjust TCAM hardware access-list tcam region vacl 0 <<<< adjust TCAM
hardware access-list tcam region arp-ether 256 hardware access-list tcam region arp-ether 256

ip pim rp-address 100.100.5.10 group-list 229.1.1.0/24 ip pim rp-address 100.100.5.10 group-list 229.1.1.0/24
ip pim ssm range 232.0.0.0/8 ip pim ssm range 232.0.0.0/8

interface loopback0 interface loopback0

ip address 100.100.5.5/32 ip address 100.100.5.6/32
ip address 100.100.5.30/32 secondary <<<<< Anycast VTEP ip address 100.100.5.30/32 secondary <<<<< Anycast VTEP
ip router ospf 1 area 0.0.0.0 ip router ospf 1 area 0.0.0.0
ip pim sparse-mode ip pim sparse-mode
 DC5 N9K ToR (page-2): vPC config
DC5 N9K ToR1: DC5 N9K ToR2:

track 1 interface Ethernet2/1 line-protocol track 1 interface Ethernet2/1 line-protocol <<<<< track Core uplink
track 2 interface Ethernet2/2 line-protocol track 2 interface Ethernet2/2 line-protocol <<<<< track Core uplink
track 3 interface port-channel10 line-protocol track 3 interface port-channel10 line-protocol
<<<<< track vPC peer-link port-channel
track 40 list boolean or track 40 list boolean or
object 1 object 1
object 2 object 2
object 3 object 3
 DC5 N9K ToR (page-3): vPC config
DC5 N9K ToR1: DC5 N9K ToR2:
vpc domain 10 <<<<< vPC domain vpc domain 10 <<<<< vPC domain
peer-switch peer-switch
role priority 1 role priority 1
peer-keepalive destination 1.5.18.56 source 1.5.18.55 peer-keepalive destination 1.5.18.55 source 1.5.18.56
peer-gateway peer-gateway
track 40 track 40
fast-convergence <<<<< 7.0(3)I7(1) CLI fast-convergence <<<<< 7.0(3)I7(1) CLI
ipv6 nd synchronize ipv6 nd synchronize
ip arp synchronize ip arp synchronize

interface port-channel10 <<<<< vPC peer link interface port-channel10 <<<<< vPC peer link
switchport mode trunk switchport mode trunk
spanning-tree port type network spanning-tree port type network
vpc peer-link vpc peer-link

interface Vlan3900 <<<<< L3 Backup over vPC Peer Link interface Vlan3900 <<<<< L3 Backup over vPC Peer Link
mtu 9216 mtu 9216
ip address 100.56.1.1/30 ip address 100.56.1.2/30
ip ospf network point-to-point ip ospf network point-to-point
ip router ospf 1 area 0.0.0.0 ip router ospf 1 area 0.0.0.0
ip pim sparse-mode ip pim sparse-mode

interface port-channel40 <<<<< connected to CE switch interface port-channel40 <<<<< connected to CE switch
switchport mode trunk switchport mode trunk
switchport trunk allowed vlan 2-3,101-104,1000-2999 switchport trunk allowed vlan 2-3,101-104,1000-2999
lacp vpc-convergence <<<<< 7.0(3)I7(1) CLI lacp vpc-convergence <<<<< 7.0(3)I7(1) CLI
vpc 40 <<<<< vPC port-channel ID vpc 40 <<<<< vPC port-channel ID
 DC5 N9K ToR (page-4): vPC status
DC5-Leaf-1# show vpc DC5-Leaf-2# show vpc
Legend: Legend:
(*) - local vPC is down, forwarding via vPC peer-link (*) - local vPC is down, forwarding via vPC peer-link

vPC domain id : 10 vPC domain id : 10

Peer status : peer adjacency formed ok Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive vPC keep-alive status : peer is alive
Configuration consistency status : success Configuration consistency status : success
Per-vlan consistency status : success Per-vlan consistency status : success
Type-2 consistency status : success Type-2 consistency status : success
vPC role : primary vPC role : secondary
Number of vPCs configured : 1 Number of vPCs configured : 1
Peer Gateway : Enabled Peer Gateway : Enabled
Dual-active excluded VLANs : - Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled Graceful Consistency Check : Enabled
Auto-recovery status : Disabled Auto-recovery status : Disabled
Delay-restore status : Timer is off.(timeout = 30s) Delay-restore status : Timer is off.(timeout = 30s)
Delay-restore SVI status : Timer is off.(timeout = 10s) Delay-restore SVI status : Timer is off.(timeout = 10s)
Operational Layer3 Peer-router : Disabled Operational Layer3 Peer-router : Disabled

vPC Peer-link status vPC Peer-link status

--------------------------------------------------------------------- ---------------------------------------------------------------------
id Port Status Active vlans id Port Status Active vlans
-- ---- ------ ------------------------------------------------- -- ---- ------ -------------------------------------------------
1 Po10 up 1-3,11-14,101-104,1000-2999,3900 1 Po10 up 1-3,11-14,101-104,1000-2999,3900

vPC status vPC status

---------------------------------------------------------------------------- ----------------------------------------------------------------------------
Id Port Status Consistency Reason Active vlans Id Port Status Consistency Reason Active vlans
-- ------------ ------ ----------- ------ --------------- -- ------------ ------ ----------- ------ ---------------
40 Po40 up success success 2-3,101-104,10 40 Po40 up success success 2-3,101-104,10
00-2999 00-2999
DC5 N9K ToR (page-5): L2VNI config for Type-2 MAC route
DC5 N9K ToR1: DC5 N9K ToR2:

fabric forwarding anycast-gateway-mac 0005.0005.0005 fabric forwarding anycast-gateway-mac 0005.0005.0005

<<<<< SVI GW virtual MAC <<<<< SVI GW virtual MAC

vlan 101 vlan 101

vn-segment 6001 <<<<< L2VNI vn-segment 6001 <<<<< L2VNI

interface Vlan101 interface Vlan101

vrf member v5011 vrf member v5011
ip address 55.101.1.254/24 <<<<< same ip ip address 55.101.1.254/24 <<<<< same ip
fabric forwarding mode anycast-gateway fabric forwarding mode anycast-gateway

evpn evpn
vni 6001 l2 vni 6001 l2
rd 100.100.5.5:6001 <<<<< or RD auto rd 100.100.5.6:6001 <<<<< or RD auto
route-target import auto <<<<< AS_Number:VNI route-target import auto <<<<< AS_Number:VNI
route-target export auto route-target export auto

interface nve1 interface nve1

source-interface loopback0 source-interface loopback0
host-reachability protocol bgp host-reachability protocol bgp
member vni 6001-6004 member vni 6001-6004
suppress-arp suppress-arp
mcast-group 229.1.1.1 mcast-group 229.1.1.1
 DC5 N9K ToR (page-6): L3VNI config for Type-2 MAC+IP
DC5 N9K ToR1: DC5 N9K ToR2:

vlan 11 vlan 11
vn-segment 5011 <<<<< L3VNI vn-segment 5011 <<<<< L3VNI

interface Vlan11 interface Vlan11

no shutdown no shutdown
vrf member v5011 vrf member v5011
ip forward ip forward

vrf context v5011 vrf context v5011

vni 5011 vni 5011
rd 100.100.5.5:5011 <<<<< or RD auto rd 100.100.5.6:5011 <<<<< or RD auto
address-family ipv4 unicast address-family ipv4 unicast
route-target import 400:5011 <<<<< or RT auto route-target import 400:5011 <<<<< or RT auto
route-target import 400:5011 evpn <<<<< RT auto evpn route-target import 400:5011 evpn <<<<< or RT auto evpn
route-target export 400:5011 route-target export 400:5011
route-target export 400:5011 evpn route-target export 400:5011 evpn

interface nve1 interface nve1

source-interface loopback0 source-interface loopback0
host-reachability protocol bgp host-reachability protocol bgp
member vni 5011-5014 associate-vrf member vni 5011-5014 associate-vrf
 DC5 N9K ToR (page-7): BGP config
DC5 N9K ToR1: DC5 N9K ToR2:

interface nve1 interface nve1

source-interface loopback0 source-interface loopback0
host-reachability protocol bgp host-reachability protocol bgp
member vni 5011-5014 associate-vrf member vni 5011-5014 associate-vrf
member vni 6001-6004 member vni 6001-6004
suppress-arp suppress-arp
mcast-group 229.1.1.1 mcast-group 229.1.1.1
peer-vtep 100.100.5.1 peer-vtep 100.100.5.1
peer-vtep 100.100.5.2 peer-vtep 100.100.5.2

router bgp 400 router bgp 400

router-id 100.100.5.5 router-id 100.100.5.6
address-family l2vpn evpn address-family l2vpn evpn
maximum-paths ibgp 2 maximum-paths ibgp 2
neighbor 100.100.5.3 <<<<< iBGP with Spines neighbor 100.100.5.3 <<<<< iBGP with Spines
remote-as 400 remote-as 400
update-source loopback0 update-source loopback0
address-family l2vpn evpn address-family l2vpn evpn
send-community extended send-community extended
neighbor 100.100.5.4 <<<<< iBGP with Spines neighbor 100.100.5.4 <<<<< iBGP with Spines
remote-as 400 remote-as 400
update-source loopback0 update-source loopback0
address-family l2vpn evpn address-family l2vpn evpn
send-community extended send-community extended
vrf v5011 vrf v5011
address-family ipv4 unicast address-family ipv4 unicast
advertise l2vpn evpn advertise l2vpn evpn
redistribute direct route-map pass-all redistribute direct route-map pass-all
maximum-paths ibgp 2 maximum-paths ibgp 2
EVPN-VXLAN to EVPN-MPLS L2 DCI GW
ESI-Based Multi-homing VXLAN GW model
ESI-Based Multi-homing with EVPN-MPLS DCI
ToR provide dual-homing to PE
PE provide dual-homing to ToR
PE
L2 overlay
ToR
MPLS underlay

IPV4 Forwarding MPLS Forwarding IPV4 Forwarding

• ESI Based MH with EVPN MPLS DCI:

• ToR use all-active Ethernet Segment 1 (ES1).
• DCI PE use all-active Ethernet Segment 2 (ES2).
• DF election happen on all 4 nodes.
• Traffic from PE to local ToR is load balanced on ToR nodes.
• PE can use EVI value hash to chose remote TOR (R6.1.1). PE can also use flow info (L2/L3 header) to chose remote
ToR (R6.2.1).
ESI-Based Multi-homing L2 GW overview

• Interworks with any ToR with following assumptions

• ToR need to support EVPN RT-1, RT-2, RT-4 and EVPN based split-horizon, DF, and load
balancing based on aliasing and BGP multi-path.
• All-active ESI need to be manually configured or auto-sensed(in case of MC-LAG is used
between ToRs and VM) on two ToRs to which a VM is duel-homed.

• No anycast VTEP on ASR9K DCI need to be used, two ASR9K work as two
standalone DCI VTEPs instead.
• VXLAN facing, DCI need to support EVPN RT-1, RT-2, RT-4 and EVPN based split-
horizon, DF, and load balancing based on aliasing and BGP multi-path.
• All-active ESI need to be manually configured on two DCIs to represent the EVI.

• EVPN aliasing, DF and split-horizon described in previous slides(section 3) is

used in this model.
ESI-Based Multi-homing L2 GW overview cont.

• An Ethernet segment is attached to a pair of standalone ToRs(ToR-1 and ToR-

2)
• Both ToRs use same all-active ESI(ESI-1) to present the attached ethernet
segment and advertise an EVPN RT-4(ethernet segment route) to DCI.
• Certain host MAC is learnt by one of the ToR(ToR-1) and advertised along with
ESI to DCI using RT-2, a BGP route for the MAC pointing to ToR-1 is installed
in DCI.
• Given the same ESI is also learned from ToR-2, DCI aliasing ToR-2 to the MAC
route, hence a BGP route for the MAC pointing to ToR-1 is installed in DCI as
well.(BGP multi-path must be enabled in DCIs)
• The same thing happened in the reverse direction.
ESI-Based Multi-homing:
DCI configuration with N9K ESI A/A ToR
DC2: ESI-based Multi-homing L2GOLF
RR

MPLS Core (SR or LDP)

BE1 BE1 BE2 BE1 BE2 BE2

BE2 BE1
DCI4_1 ASR9k DCI4_2 DCI5_1 ASR9k DCI5_2

BE13 BE14 BE23 BE24 BE14 BE23 BE24

BE13
eBGP Spine eBGP

Po13 Po23 Po14 Po24 VxLAN VxLAN Po13 Po23 Po14 Po24
Overlay Overlay
Spine2-1 Spine2-2 Spine5-1 Spine5-2

RR RR RR RR
iBGP iBGP

Leaf2-1 Leaf2-2 Leaf5-1 Leaf5-2

Po40 Po40
ESI A/A vPC

DC2: ESI-Based Multi-homing DC5: Anycast VTEP Multi-homing

DCI2 config (page-1): ESI-Based Multi-homing
router isis Core router bgp 100
router-id 100.100.4.11 <<<< loopback0 bgp router-id 100.100.4.11 <<<< loopback0
segment-routing mpls address-family l2vpn evpn
interface Bundle-Ether1 !
interface Bundle-Ether2 neighbor 20.1.1.1 <<<< towards WAN RR or remote DCI
interface Loopback0 remote-as 100
! update-source Loopback0
router isis Edge-1 address-family l2vpn evpn
router-id 100.100.4.1 <<<< loopback1 import re-originate stitching-rt
interface Bundle-Ether13 advertise l2vpn evpn re-originated
interface Bundle-Ether14 !
interface Loopback1 neighbor 100.100.4.3 <<<<< towards VxLAN Spines/ToRs
! remote-as 500
! ebgp-multihop 10
update-source Loopback1
address-family l2vpn evpn
import stitching-rt reoriginate
multipath
route-policy pass-all in
encapsulation-type vxlan
route-policy pass-all out
advertise l2vpn evpn re-originated stitching-rt
DCI2 config (page-2): ESI-Based Multi-homing

redundancy interface nve1

iccp redundancy
group 11 backbone mpls
mode singleton iccp group 11
backbone !
interface Bundle-Ether1 backbone vxlan
interface Bundle-Ether2 iccp group 12
! !
group 12 source-interface Loopback1
mode singleton !
backbone
interface Bundle-Ether13
interface Bundle-Ether14 evpn
! interface nve1
! ethernet-segment
identifier type 0 44.44.44.44.44.44.44.44.44 <<<< same on two DCI
bgp route-target bbbb.bbbb.bbbb <<<< same on two DCI
!
 DCI2 config (page-3): ESI-Based Multi-homing
evpn router pim
evi 6001 address-family ipv4
bgp rp-address 100.100.4.10 <<<< points to Spine PIM Anycast RP
route-target import 100:6001 !
route-target export 100:6001
route-target import 500:6001 stitching
route-target export 500:6001 stitching multicast-routing
! interface Loopback1
enable
l2vpn !
bridge group 6001 interface Bundle-Ether13
bridge-domain 6001 enable
evi 6001 !
member vni 6001 interface Bundle-Ether14
! enable
!
interface nve 1
member vni 6001
mcast-group 229.1.1.1
host-reachability protocol bgp
!
DCI2 config (page-4): ESI-Based Multi-homing
 We don’t officially claim EVPN support on Typhoon LC, but if you use Typhoon
LC to do ESI-based Multi-homing L2GOLF lab testing, need enable the following
CLI for Typhoon LC since Typhoon LC doesn’t support VxLAN flow-based load-
balancing, it only supports per-EVI load-balancing.
interface nve2
member vni 6001-6004
mcast-group 229.1.1.1
host-reachability protocol bgp
load-balance per-evi < ------- This CLI is must for Typhoon LC after 6.2.1
!

 6.1.1 only supports VxLAN per-EVI load-balancing on both Tomahawk and

Typhoon, so the above CLI is not needed in 6.1.1. Since 6.2.1, Tomahawk by
default is doing VxLAN flow-based load-balancing.
N9K Limitation for ESI-Based Multi-homing
 ESI-based multihoming is supported on the Cisco Nexus 9300 Series switches
only and it is not supported on the Cisco Nexus 9300-EX and 9500 Series
switches.
 The Cisco Nexus 9500 Series switches can be used as Spine switches but they
cannot be used as VTEPs.
 Beginning with Cisco NX-OS Release 7.0(3)I5(2), ARP suppression is supported
with ESI-Based multihoming.
 Curently only support dual-homing with two switches
 To enable ESI-based multihoming, the Spine switches should be running the
minimum software version as Cisco NX-OS Release 7.0(3)I5(2) or later.
 Switchport trunk native VLAN is not supported on the trunk interfaces.
 Cisco recommends enabling LACP on ES PO.
 IPV6 is not currently supported.
 DC2 N9K Spine (page-1): PIM Anycast RP config
DC2 N9K Spine-1: DC2 N9K Spine-2:
feature isis feature isis
feature bgp feature bgp
feature pim feature pim

ip pim rp-address 100.100.4.10 group-list 229.1.1.0/24 ip pim rp-address 100.100.4.10 group-list 229.1.1.0/24
ip pim ssm range 232.0.0.0/8 ip pim ssm range 232.0.0.0/8
ip pim anycast-rp 100.100.4.10 100.100.4.3 ip pim anycast-rp 100.100.4.10 100.100.4.3
ip pim anycast-rp 100.100.4.10 100.100.4.4 ip pim anycast-rp 100.100.4.10 100.100.4.4

interface loopback0 interface loopback0

ip address 100.100.4.3/32 ip address 100.100.4.4/32
ip router ospf 1 area 0.0.0.0 ip router ospf 1 area 0.0.0.0
ip pim sparse-mode ip pim sparse-mode

interface loopback1 interface loopback1

ip address 100.100.4.10/32 ip address 100.100.4.10/32
ip router ospf 1 area 0.0.0.0 ip router ospf 1 area 0.0.0.0
ip pim sparse-mode ip pim sparse-mode
DC2 N9K Spine (page-2): BGP config
route-map Next-Hop-Unchanged permit 10
set ip next-hop unchange

router bgp 500

router-id 100.100.4.3
neighbor 100.100.4.1 <<<<< eBGP session with DCI routers
remote-as 100
update-source loopback0
ebgp-multihop 10
address-family l2vpn evpn
send-community
send-community extended
route-map Next-Hop-Unchanged out

neighbor 100.100.4.5 <<<<< ibgp session with ToRs

remote-as 500
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
route-reflector-client
DC2 N9K ToR (page-1): ESI-Based Multi-homing config
DC2 N9K ToR1: DC2 N9K ToR2:
nv overlay evpn nv overlay evpn
feature isis feature isis
feature bgp feature bgp
feature pim feature pim
feature interface-vlan feature interface-vlan
feature vn-segment-vlan-based feature vn-segment-vlan-based
feature lacp feature lacp
feature vpc feature vpc
feature lldp feature lldp
feature nv overlay feature nv overlay

route-map pass-all permit 10 route-map pass-all permit 10

ip pim rp-address 100.100.4.10 group-list 229.1.1.0/24 ip pim rp-address 100.100.4.10 group-list 229.1.1.0/24
ip pim ssm range 232.0.0.0/8 ip pim ssm range 232.0.0.0/8

hardware access-list tcam region vacl 0 <<<< adjust TCAM hardware access-list tcam region vacl 0 <<<< adjust TCAM
hardware access-list tcam region arp-ether 256 hardware access-list tcam region arp-ether 256
hardware access-list tcam region vpc-convergence 256 hardware access-list tcam region vpc-convergence 256

interface loopback0 interface loopback0

ip address 100.100.2.5/32 ip address 100.100.2.6/32
ip router isis 1 ip router isis 1
ip pim sparse-mode ip pim sparse-mode
DC2 N9K ToR (page-2): ESI-Based Multi-homing config
DC2 N9K ToR1: DC2 N9K ToR2:

evpn esi multihoming evpn esi multihoming

interface Ethernet2/1 <<<<< connected to spine interface Ethernet2/1 <<<<< connected to spine
evpn multihoming core-tracking evpn multihoming core-tracking
mtu 9216 mtu 9216
ip address 102.35.1.1/31 ip address 102.36.1.1/31
isis network point-to-point isis network point-to-point
ip router isis 1 ip router isis 1
ip pim sparse-mode ip pim sparse-mode

interface Ethernet2/2 <<<<< connected to spine interface Ethernet2/2 <<<<< connected to spine
evpn multihoming core-tracking evpn multihoming core-tracking
mtu 9216 mtu 9216
ip address 102.45.1.1/31 ip address 102.46.1.1/31
isis network point-to-point isis network point-to-point
ip router isis 1 ip router isis 1
ip pim sparse-mode ip pim sparse-mode

interface port-channel40 <<<<< connected to CE switch interface port-channel40 <<<<< connected to CE switch
switchport mode trunk switchport mode trunk
switchport trunk allowed vlan 2,101-104,1000-2999 switchport trunk allowed vlan 2,101-104,1000-2999
ethernet-segment 1 ethernet-segment 1
system-mac 0025.0025.0025 <<<<< same to form ESI system-mac 0025.0025.0025 <<<<< same to form ESI
 DC2 N9K ToR (page-3): ESI-Based Multi-homing status
DC2-Leaf-1# show nve ethernet-segment
ESI: 0300.2500.2500.2500.0001
Parent interface: port-channel40
ES State: Up
Port-channel state: Up
NVE Interface: nve1
NVE State: Up
Host Learning Mode: control-plane
Active Vlans: 101-104,
DF Vlans: 102,104,
Active VNIs: 6001-6004,
CC failed for VLANs:
VLAN CC timer: 0
Number of ES members: 2
My ordinal: 0
DF timer start time: 00:00:00
Config State: config-applied
DF List: 100.100.2.5 100.100.2.6
ES route added to L2RIB: True
EAD/ES routes added to L2RIB: True
EAD/EVI route timer age: not running
----------------------------------------
DC2-Leaf-1#
DC2-Leaf-2# show nve ethernet-segment
ESI: 0300.2500.2500.2500.0001
Parent interface: port-channel40
ES State: Up
Port-channel state: Up
NVE Interface: nve1
NVE State: Up
Host Learning Mode: control-plane
Active Vlans: 101-104,
DF Vlans: 101,103,
Active VNIs: 6001-6004,
CC failed for VLANs:
VLAN CC timer: 0
Number of ES members: 2
My ordinal: 1
DF timer start time: 00:00:00
Config State: config-applied
DF List: 100.100.2.5 100.100.2.6
ES route added to L2RIB: True
EAD/ES routes added to L2RIB: True
EAD/EVI route timer age: not running
----------------------------------------
DC2-Leaf-2#
DC2 N9K ToR (page-4): L2VNI config for Type-2 MAC route
DC2 N9K ToR1: DC2 N9K ToR2:

fabric forwarding anycast-gateway-mac 0002.0002.0002 fabric forwarding anycast-gateway-mac 0002.0002.0002

<<<<< SVI GW virtual MAC <<<<< SVI GW virtual MAC

vlan 101 vlan 101

vn-segment 6001 <<<<< L2VNI vn-segment 6001 <<<<< L2VNI

interface Vlan101 interface Vlan101

vrf member v5011 vrf member v5011
ip address 25.101.1.254/24 ip address 25.101.1.254/24
fabric forwarding mode anycast-gateway fabric forwarding mode anycast-gateway

evpn evpn
vni 6001 l2 vni 6001 l2
rd 100.100.2.5:6001 <<<<< or RD auto rd 100.100.2.6:6001 <<<<< or RD auto
route-target import auto <<<<< AS_Number:VNI route-target import auto <<<<< AS_Number:VNI
route-target export auto route-target export auto

interface nve1 interface nve1

source-interface loopback0 source-interface loopback0
host-reachability protocol bgp host-reachability protocol bgp
member vni 6001-6004 member vni 6001-6004
suppress-arp suppress-arp
mcast-group 229.1.1.1 mcast-group 229.1.1.1
 DC2 N9K ToR (page-5): L3VNI config for Type-2 MAC+IP
DC2 N9K ToR1: DC2 N9K ToR2:

vlan 11 vlan 11
vn-segment 5011 <<<<< L3VNI vn-segment 5011 <<<<< L3VNI

interface Vlan11 interface Vlan11

no shutdown no shutdown
vrf member v5011 vrf member v5011
ip forward ip forward

vrf context v5011 vrf context v5011

vni 5011
rd 100.100.2.5:5011 <<<<< or
address-family ipv4 unicast
route-target import 400:5011
route-target import 400:5011
route-target export 400:5011
route-target export 400:5011
vni 5011
RD auto rd 100.100.2.6:5011 <<<<< or RD auto
address-family ipv4 unicast
<<<<< or RT auto route-target import 400:5011 <<<<< or RT auto
evpn <<<<< or RT auto evpn route-target import 400:5011 evpn <<<<< RT auto evpn
route-target export 400:5011
evpn route-target export 400:5011 evpn

interface nve1 interface nve1

source-interface loopback0 source-interface loopback0
host-reachability protocol bgp host-reachability protocol bgp
member vni 5011-5014 associate-vrf member vni 5011-5014 associate-vrf
 DC2 N9K ToR (page-6): BGP config
DC2 N9K ToR1: Dc2 N9K ToR2:
interface nve1 interface nve1
source-interface loopback0 source-interface loopback0
host-reachability protocol bgp host-reachability protocol bgp
member vni 5011-5014 associate-vrf member vni 5011-5014 associate-vrf
member vni 6001-6004 member vni 6001-6004
suppress-arp suppress-arp
mcast-group 229.1.1.1 mcast-group 229.1.1.1

router bgp 500 router bgp 500

router-id 100.100.2.5 router-id 100.100.2.6
address-family l2vpn evpn address-family l2vpn evpn
maximum-paths ibgp 2 maximum-paths ibgp 2
neighbor 100.100.2.3 <<<<< iBGP with Spines neighbor 100.100.2.3 <<<< iBGP with Spines
remote-as 500 remote-as 500
update-source loopback0 update-source loopback0
address-family l2vpn evpn address-family l2vpn evpn
send-community extended send-community extended
neighbor 100.100.2.4 <<<<< iBGP with Spines neighbor 100.100.2.4 <<<<< iBGP with Spines
remote-as 500 remote-as 500
update-source loopback0 update-source loopback0
address-family l2vpn evpn address-family l2vpn evpn
send-community extended send-community extended
vrf v5011 vrf v5011
address-family ipv4 unicast address-family ipv4 unicast
advertise l2vpn evpn advertise l2vpn evpn
redistribute direct route-map pass-all redistribute direct route-map pass-all
maximum-paths ibgp 2 maximum-paths ibgp 2
ASR9K working as EVPN-VxLAN ToR
ASR9K working as EVPN-VxLAN ToR

 If you don’t have N9K Spine/ToR, you also can use ASR9K
routers as EVPN-VxLAN ToR

 When ASR9K works as ToR, we can configure regular RT to

match DCI’s stitching RT
 router bgp 400

ASR9K as EVPN-VxLAN ToR

bgp router-id 100.100.5.7 <<<< loopback0
address-family l2vpn evpn
!
neighbor 100.100.5.1 <<<<< towards DCI
evpn remote-as 100
evi 6001 ebgp-multihop 10
bgp update-source Loopback0
route-target import 400:6001 <<< match DCI’s stitching-RT address-family l2vpn evpn
route-target export 400:6001 import stitching-rt reoriginate <<< remove this CLI
! multipath
advertise-mac route-policy pass-all in
! encapsulation-type vxlan
route-policy pass-all out
interface NVE 1 advertise l2vpn evpn re-originated stitching-rt
source-interface Loopback0 <<<<< remove stitching-rt
member vni 6001
mcast-group 229.1.1.1 multicast-routing
host-reachability protocol bgp interface Loopback0
! enable
!
l2vpn interface Bundle-Ether17
bridge group 6001 enable
bridge-domain 6001
interface TenGigE0/1/0/3.6001 router pim
evi 6001 address-family ipv4
member vni 6001 rp-address 100.100.5.10
XR 6.3.2 EVPN-VxLAN Ingress-replication
EVPN-VxLAN Ingress-Replication

Since 6.3.2 ASR9K supports EVPN-VxLAN Ingress-replication on

Tomahawk LC, Typhoon LC doesn’t support

ASR9K DCI: N9K ToR:

interface nve 1 interface nve1

member vni 6001-6004 host-reachability protocol bgp
mcast-group 229.1.1.1 source-interface loopback0
host-reachability protocol bgp member vni 6001-6004
! suppress-arp
source-interface Loopback0 mcast-group 229.1.1.1
ingress-replication protocol bgp ingress-replication protocol bgp
!
Same AS Considerations
Same AS Considerations

If two DC sites have same AS, we can use any one of the following
method:
 DCI ASR9Ks do as-override towards Spines, or
 Spines and ToRs use allowas-in, or
 DCI ASR9Ks use route-policy to manipulate AS_Path, or
 Spines use route-map to manipulate AS_Path