Network Virtualization Evd

EXTREME VALIDATED DESIGN
Network Virtualization
Network Virtualization in Fabric
in IP IP Fabric
withwith
BGPBGP
EVPN
EVPN
Version 2.0
9035383
February 2018
© 2018, Extreme Networks, Inc. All Rights Reserved.
Extreme Networks and the Extreme Networks logo are trademarks or registered trademarks of Extreme Networks, Inc. in the United States and/or other countries. All
other names are the property of their respective owners. For additional information on Extreme Networks Trademarks please see
www.extremenetworks.com/company/legal/trademarks. Specifications and product availability are subject to change without notice.
© 2017, Brocade Communications Systems, Inc. All Rights Reserved.
Brocade, the B-wing symbol, and MyBrocade are registered trademarks of Brocade Communications Systems, Inc., in the United States and in other countries. Other
brands, product names, or service names mentioned of Brocade Communications Systems, Inc. are listed at http://www.brocade.com/en/legal/brocade-Legal-
intellectual-property/brocade-legal-trademarks.html. Other marks may belong to third parties.
Notice: This document is for informational purposes only and does not set forth any warranty, expressed or implied, concerning any equipment, equipment feature,
or service offered or to be offered by Brocade. Brocade reserves the right to make changes to this document at any time, without notice, and assumes no
responsibility for its use. This informational document describes features that may not be currently available. Contact a Brocade sales office for information on
feature and product availability. Export of technical data contained in this document may require an export license from the United States government.
The authors and Brocade Communications Systems, Inc. assume no liability or responsibility to any person or entity with respect to the accuracy of this document
or any loss, cost, liability, or damages arising from the information contained herein or the computer programs that accompany it.
The product described by this document may contain open source software covered by the GNU General Public License or other open source license agreements. To find
out which open source software is included in Brocade products, view the licensing terms applicable to the open source software, and obtain a copy of the programming
source code, please visit http://www.brocade.com/support/oscd.
2
Contents
Contents ............................................................................................................................................................................................................................... 3
List of Figures........................................................................................................................................................................................................................ 5
Preface .................................................................................................................................................................................................................................. 7
Extreme Validated Designs .......................................................................................................................................................................................... 7
Purpose of This Document ........................................................................................................................................................................................... 7
Target Audience ........................................................................................................................................................................................................... 7
Authors ......................................................................................................................................................................................................................... 8
Document History ........................................................................................................................................................................................................ 8
Introduction ......................................................................................................................................................................................................................... 9
Technology Overview ......................................................................................................................................................................................................... 11

Terminology ............................................................................................................................................................................................................... 12
Functional Components of IP Fabric ........................................................................................................................................................................... 13
Leaf-Spine Layer 3 Clos Topology (Two-Tier) ........................................................................................................................................................ 13
Optimized 5-Stage Layer 3 Clos Topology (Three-Tier) ......................................................................................................................................... 14
Edge Services and Border Leafs .......................................................................................................................................................................... 15
Extreme IP Fabric Underlay Routing ................................................................................................................................................................... 15
Network Virtualization with BGP VxLAN Based EVPN ........................................................................................................................................ 19
Validated Designs ............................................................................................................................................................................................................... 31

eBGP Deployment Model ........................................................................................................................................................................................... 31
Hardware and Software Matrix ................................................................................................................................................................................... 33
3-Stage Fabric .............................................................................................................................................................................................................. 34
Fabric Infrastructure Configuration.................................................................................................................................................................... 34
BGP Underlay Configuration .............................................................................................................................................................................. 37
BGP EVPN and VxLAN Overlay............................................................................................................................................................................ 39
5-Stage Fabric .............................................................................................................................................................................................................. 47
Spine Configuration ............................................................................................................................................................................................ 48
Super-Spine Configuration.................................................................................................................................................................................. 49
Edge-Leaf Configuration ..................................................................................................................................................................................... 50
Use Cases ........................................................................................................................................................................................................................... 52

Simple 3-stage BGP VxLAN Based EVPN Fabric Illustration ..................................................................................................................................................... 53
Configuration ...................................................................................................................................................................................................... 54
Verification ......................................................................................................................................................................................................... 61
L2 and L3 Extension between Racks ........................................................................................................................................................................... 65
Configuration ...................................................................................................................................................................................................... 67
Verification ......................................................................................................................................................................................................... 69
VLAN Scoping at the ToR Level ................................................................................................................................................................................... 78
Configuration ...................................................................................................................................................................................................... 79
Verification ......................................................................................................................................................................................................... 82
VLAN Scoping at the Port Level within a ToR .............................................................................................................................................................. 87
Configuration ...................................................................................................................................................................................................... 87
Verification ......................................................................................................................................................................................................... 91
Layer-2 Handoff with VPLS ......................................................................................................................................................................................... 95
Configuration ...................................................................................................................................................................................................... 95
Verification ....................................................................................................................................................................................................... 100
Layer-3 Handoff with MPLS/L3VPN .......................................................................................................................................................................... 103
Configuration .................................................................................................................................................................................................... 103
Verification ....................................................................................................................................................................................................... 110
3
Design Considerations ...................................................................................................................................................................................................... 112
BGP Route scale considerations ............................................................................................................................................................................... 112
BGP TTL Security ...................................................................................................................................................................................................... 112
Appendix1: VDX Leaf configuration ................................................................................................................................................................................. 113

Node ID Configuration ............................................................................................................................................................................................. 113
IP Fabric Infrastructure Links.................................................................................................................................................................................... 114
Loopback Interfaces ................................................................................................................................................................................................. 114
vLAG Pair/ToR .......................................................................................................................................................................................................... 114
Node ID Configuration on vLAG Pair ................................................................................................................................................................ 115
ISL Configuration .............................................................................................................................................................................................. 115
Server Port-Channel Configuration .................................................................................................................................................................. 115
Loopback interfaces ......................................................................................................................................................................................... 116
BGP Configuration .................................................................................................................................................................................................... 116
Tenant Provisioning ................................................................................................................................................................................................. 116
Anycast Gateway MAC Configuration .............................................................................................................................................................. 117
Enable Conversational Learning of MAC and ARP/ND Host Entries ................................................................................................................. 117
VRFs, Server VLANs, and Subnets Configuration .............................................................................................................................................. 117
Advertise Tenant Layer 3 Routes from the Leaf ............................................................................................................................................... 119
Enable EVPN Instance for the Tenant VLAN Segments .................................................................................................................................... 119
vLAG Pair Configuration ................................................................................................................................................................................... 120
References ........................................................................................................................................................................................................................ 123
4
List of Figures
Figure 1 Leaf-Spine L3 Clos Topology ......................................................................................................................................................................... 14
Figure 2 Optimized 5-Stage L3 Clos Topology............................................................................................................................................................. 15
Figure 3 eBGP for Underlay ........................................................................................................................................................................................ 17
Figure 4 iBGP for Underlay ......................................................................................................................................................................................... 18
Figure 5 VTEPs and L2 Extension with Flood and Learn.............................................................................................................................................. 20
Figure 6 Routing Between VxLAN networks in Flood and Learn topology.................................................................................................................. 21
Figure 7 VTEPs and L2 Extension with BGP EVPN Control-plane ................................................................................................................................ 22
Figure 8 ARP Suppression ........................................................................................................................................................................................... 25
Figure 9 VLAN Scoping at the Leaf/ToR Level ............................................................................................................................................................. 26
Figure 10 VLAN Scoping at the Port Level within a ToR ............................................................................................................................................... 26
Figure 11 Asymmetric IRB ............................................................................................................................................................................................ 27
Figure 12 Symmetric IRB .............................................................................................................................................................................................. 28
Figure 13 Multitenancy ................................................................................................................................................................................................ 29
Figure 14 MCT Pair for Dual-homing and Leaf Redundancy ........................................................................................................................................ 30
Figure 15 EBGP based 3-stage IP Fabric ....................................................................................................................................................................... 31
Figure 16 EBGP based Optimized 5-stage IP Fabric...................................................................................................................................................... 32
Figure 17 Illustration topology for a simple 3-stage fabric .......................................................................................................................................... 53
Figure 18 L2/L3 Extension between Racks ................................................................................................................................................................... 66
Figure 19 VLAN Scoping at the ToR Level..................................................................................................................................................................... 78
Figure 20 Port VLAN scoping within the ToR ............................................................................................................................................................... 87
Figure 21 Layer-2 Handoff with VPLS ........................................................................................................................................................................... 95
Figure 22 Layer-3 Handoff with MPLS/L3VPN ............................................................................................................................................................ 103
5
6
Preface
Extreme Validated Designs
Helping customers consider, select, and deploy network solutions for current and planned needs is our mission. Extreme Validated Designs offer a fast track
to success by accelerating that process.
Validated designs are repeatable reference network architectures that have been engineered and tested to address specific use cases and deployment
scenarios. They document systematic steps and best practices that help administrators, architects, and engineers plan, design, and deploy physical and
virtual network technologies. Leveraging these validated network architectures accelerates deployment speed, increases reliability and predictability,
and reduces risk.
Extreme Validated Designs incorporate network and security principles and technologies across the ecosystem of service provider, data center,
campus, and wireless networks. Each Extreme Validated Design provides a standardized network architecture for a specific use case, incorporating
technologies and feature sets across Extreme products and partner offerings.
All Extreme Validated Designs follow best-practice recommendations and allow for customer-specific network architecture variations that deliver
additional benefits. The variations are documented and supported to provide ongoing value, and all Extreme Validated Designs are continuously
maintained to ensure that every design remains supported as new products and software versions are introduced.
By accelerating time-to-value, reducing risk, and offering the freedom to incorporate creative, supported variations, these validated network
architectures provide a tremendous value-add for building and growing a flexible network infrastructure.
Purpose of This Document

This Extreme validated design provides guidance for designing and implementing IP fabric in a data center network using Extreme hardware and
software. It details the Extreme reference architecture for enabling Network virtualization using BGP VxLAN based EVPN.
It should be noted that not all features such as automation practices, zero-touch provisioning, and monitoring of the Extreme IP Fabric are included in
this document. Future versions of this document are planned to include these aspects of the Extreme IP Fabric solution.
The design practices documented here follow the best-practice recommendations, but there are variations to the design that are supported as well.
Target Audience
This document is written for Extreme systems engineers, partners, and customers who design, implement, and support data center networks. This
document is intended for experienced data center architects and engineers. This document assumes that the reader has a good understanding of data
center switching and routing features and of Multi-Protocol BGP/MPLS VPN[5] for understanding multitenancy in VXLAN EVPN networks.
7
Authors
• Krish Padmanabhan
Sr Principal Engineer, System and Solution Engineering
• Eldho Jacob
Principal Engineer, System and Solution Engineering
The authors would like to acknowledge the following at Extreme Networks for their technical guidance in developing this validated design:
• Abdul Khader
Director, System and Solution Engineering
• Vivek Baveja
Director, Product Management
The authors would also like to acknowledge the following for their meticulous review of the document.
• Wim van Laarhoven
• Lavanya Venkatesan
Document History
Date Part Number Description
February 2018 9035383 Network Virtualization with BGP EVPN in IP Fabric (SLX platforms)
8
Introduction
Extreme has expanded its product portfolio with SLX platforms and SLX-OS positioned for network virtualization architectures to meet the growing
customer demand for higher levels of scale, agility, and operational efficiency.
This document describes cloud-optimized network designs using Extreme IP Fabrics for building data-center sites. The configurations and design
practices documented here are fully validated and conform to the Extreme IP Fabric reference architectures. The intention of this Extreme Validated
Design document is to provide reference configurations and document best practices for building cloud-scale data-center networks using Extreme SLX
switches and Extreme IP Fabric architectures.
This document describes the following architectures:
• Extreme IP Fabric deployed in 3-stage and optimized 5-stage folded Clos topologies
• Network virtualization and multi-tenancy using BGP EVPN in these 3-stage and 5-stage fabrics
9
10
Technology Overview
Extreme IP Fabric provides a Layer 3 Clos deployment architecture for data center sites. In an Extreme IP Fabric, all links in the Clos topology are Layer
3 links. It includes the networking architecture; the protocols used to build the network; turnkey automation features used to provision, manage, and
monitor the networking infrastructure; and the hardware differentiation with Extreme SLX and VDX switches. The following sections describe the
validated design for data center sites with Extreme IP Fabrics. Because the infrastructure is built on IP, advantages like the following are leveraged:
loop-free communication using industry-standard routing protocols, ECMP, very high solution scale, and standards-based interoperability.
11
Terminology
Term Description
ARP Address Resolution Protocol

AS Autonomous System
ASN Autonomous System Number
BD Bridge Domain
BFD Bidirectional Forwarding Detection
BGP Border Gateway Protocol
BUM Broadcast, Unknown unicast, and Multicast
CE Classical Ethernet
DCI Data Center Interconnect
eBGP External Border Gateway Protocol
ECMP Equal Cost Multi-Path
EVPN Ethernet Virtual Private Network
GVLAN Global VLAN
iBGP Internal Border Gateway Protocol
IP Internet Protocol
IRB Integrated Routing and Bridging
MAC Media Access Control
MP-BGP Multi-Protocol Border Gateway Protocol
MPLS Multi-Protocol Label Switching
ND Neighbor Discovery
NLRI Network Layer Reachability Information
OpEx Operational Expenses
PE Provider Edge Router
PoD Point of Delivery
QoS Quality of Service
RD Route Distinguisher
RIOT Routing In/Out of Tunnels (in a single pass in the ASIC or SoC)
RT Route Target
ToR Top of Rack switch
UDP User Datagram Protocol
vLAG Virtual Link Aggregation Group
VLAN Virtual Local Area Network
VM Virtual Machine
VNI VXLAN Network Identifier
VPLS Virtual Private Lan Service
VPN Virtual Private Network
VRF VPN Routing and Forwarding instance
VTEP VXLAN Tunnel Endpoint
VXLAN Virtual Extensible Local Area Network
12
Functional Components of IP Fabric
Leaf-Spine Layer 3 Clos Topology (Two-Tier)
The leaf-spine topology has become the de facto standard for networking topologies when building medium- to large-scale data center
infrastructures. The leaf-spine topology is adapted from Clos telecommunications networks. The Extreme IP Fabric within a PoD resembles a two-tier
or 3-stage folded Clos fabric. The two-tier leaf-spine topology is shown in Figure 1.
The bottom layer of the IP fabric has the leaf devices (top-of-rack switches), and the top layer has spines. The role of the leaf is to provide connectivity
to the endpoints in the data center network. These endpoints include tenant workloads such as compute and storage devices, as well as other
networking devices like routers, switches, load balancers, firewalls, and any other physical or virtual networking endpoints. Because all endpoints
connect only to the leaf, policy enforcement, including security, traffic-path selection, QoS marking, traffic policing, and shaping, is implemented at
the leaf.
More importantly, Leafs act as the first hop gateways with Anycast gateway addresses for the server segments to facilitate mobility with the VXLAN
overlay.
A set of Leafs act as Border-leafs or Edge-leafs that provide connectivity to services such as Firewall, load balancer, storage etc., and external
connectivity to the PoD.
The role of the spine is to provide connectivity between leafs. The major role of the spine is to participate in the control-plane and data- plane
operations for traffic forwarding between leafs. The spine devices serve two purposes: BGP control plane (route distribution using BGP protocol and its
extensions), And data-plane IP forwarding based on the outer IP header in the underlay network. Since there are no network endpoints connected to
the spine, tenant VRFs or VXLAN segments are not created on spines. Their routing table size requirements are also very light to accommodate just the
underlay reachability. Note that all spine devices need not act as BGP route reflectors; only selected spines in the spine layer can act as BGP route
reflectors in the overlay design. More details are provided in BGP EVPN Control Plane.
As a design principle, the following requirements apply to the leaf-spine topology:
• Each compute/storage rack has a Leaf or ToR (Top of the Rack) switch. The rack may have a pair of redundant switches in a MCT/vLAG pair
referred to as a dual ToR or a MCT pair ToR or a vLAG pair ToR. Dual ToR provides node and link redundancy to the workloads in the rack.
• Leafs and border-leafs connects to all spines in the PoD. These links are referred to as Fabric Infrastructure Links.
• Spines are not interconnected with each other.
• Leafs are not interconnected with each other for data-plane purposes. (Leafs in a MCT or vLAG pair are interconnected for control-plane
operations such as forming a server-facing LAG.)
• The network endpoints do not connect to the spines.
This type of topology has the predictable latency and also provides the ECMP forwarding in the underlay network. The number of hops between two
leaf devices is always two within the fabric. This topology also enables easier scale out in the horizontal direction as the data center expands and is
limited by the port density and bandwidth supported by the spine devices.
This validated design recommends the same hardware in the spine layer. Mixing different hardware is not recommended.
IP Fabric Infrastructure Links

All fabric nodes—leafs, spines, and border-leafs —are interconnected with Layer 3 IPv4 interfaces. The validated design has,
• 40-GbE links are used between the fabric nodes.
• All these links are configured as Layer 3 interfaces with /31 IPv4 address.
13
• The MTU for these links is set to jumbo MTU. This is a requirement to handle the VXLAN encapsulation of Ethernet frames.
• Multiple parallel links between two nodes in the fabric must be avoided.
Server-Facing Links
The server-facing or access links are on the leaf nodes connecting the workloads. These links are either individual or a LAG in case of a dual-
ToR. In the validated design,
• 10/25-GbE links as trunk ports with associated VLANs.
• Spanning-tree is typically disabled for server connectivity unless there are downstream L2 switches.
Figure 1 Leaf-Spine L3 Clos Topology
Internet
MPLS Network
SPINE
WAN EDGE
BORDER LEAF
Rack-1 Rack-2 Rack-N Edge Services
Optimized 5-Stage Layer 3 Clos Topology (Three-Tier)

Multiple PoDs based on leaf-spine topologies can be connected for higher scale in an optimized 5-stage folded Clos (three-tier) topology. This
topology adds a new tier to the network, known as a super-spine. This architecture is recommended for interconnecting several EVPN VXLAN PoDs.
Super-spines function similar to spines: BGP control plane and IP forwarding based on the outer IP header in the underlay network. No endpoints are
connected to the super-spine. Figure 2 shows four super-spine switches connecting the spine switches across multiple data center PoDs.
The connection between the spines and the super-spines follows the Clos principles:
• Each spine connects to all super-spines in the network. In the validated design, both 40 and 100-GbE links were tested independently. It is
not recommended to mix the links of different bandwidths between two layers of the IP fabric.
• Border-leafs connect to the super-spines.
• Neither spines nor super-spines are interconnected with each other.
14
Figure 2 Optimized 5-Stage L3 Clos Topology
Edge Services and Border Leafs

For two-tier and three-tier data center topologies, the role of the border leaf in the network is to provide external connectivity to the data center
site. In addition, since all traffic enters and exits the data center through the border leaf switches, they present the ideal location in the network to
connect network services like firewalls, load balancers, and edge VPN routers. The border leaf switches connect to the WAN edge devices in the
network to provide external connectivity to the data center site. As a design principle, two border leaf switches are recommended for redundancy.
The WAN edge devices provide the interfaces to the Internet and DCI solutions. For DCI, these devices function as the Provider Edge (PE) routers,
enabling connections to other data center sites through WAN technologies like Multiprotocol Label Switching (MPLS) VPN and Virtual Private LAN
Services (VPLS). The Extreme validated design for DCI solutions is discussed in a separate validated design document.
There are several ways that the border leafs connect to the data center site. In three-tier (super-spine) architectures, the border leafs are connected to
the super-spines as depicted in Figure 2. In two-tier topologies, the border leafs are connected to the spines as depicted in Figure 1. Certain topologies
may use the spine as border leafs (known as a border spine), overloading two functions into one. This topology adds additional forwarding
requirements to spines—they need to be aware of the tenants, VNIs, and VXLAN tunnel encapsulation and de-encapsulation functions.
Extreme IP Fabric Underlay Routing

IP fabric collectively refers to the following:
• IPv4 network address assignments to the links connecting the nodes in the fabric: spines, leafs, super-spines, and border leafs.
• Control-plane protocol used for reachability between the nodes. A smaller scale topology might benefit from a link-state protocol such as
OSPF. (Note that this is not validated in this design, though it is supported). Large scale topologies, however, typically use BGP. Extreme
validated design recommends BGP as the protocol for underlay network reachability.
• Resiliency feature such as BFD.
15
There are several underlay deployment options. When using BGP as the only routing protocol in the fabric, there are two models:
• eBGP for Underlay—eBGP peering between each tier of nodes: between the leaf and the spine; between the spine and the super-spine; and
between the super-spine and the border leaf.
• iBGP for Underlay—iBGP peering between the leaf and the spine within the PoD and spines as BGP route reflectors. eBGP peering between
the PoDs through the super-spine layer for inter-PoD reachability. (Note that this is not validated in this design, though it is supported).
eBGP for Underlay

This deployment model refers to the usage of eBGP peering between the leaf and the spine in the fabric. In this model, each leaf node is assigned its
own autonomous system (AS) number. The other nodes are grouped based on their role in the fabric, and each of these groups is assigned a separate
AS number, as shown in Figure 3. Using eBGP in an IP fabric is simple and also provides the ability to apply BGP policies for traffic engineering on a
per-leaf or per-rack basis since each leaf or rack in a PoD is assigned a unique AS number. Private AS numbers are used in this validated design. One
design consideration for the AS number assignment is that a 2-byte AS number provides a maximum of 1023 private AS numbers (ASN 64512 to ASN
65534); We recommend using 4-byte private AS numbers (ASN 4,200,000,000 to 4,294,967,294) for scalability when there are multiple PoDs.
• Each leaf in a PoD is assigned its own AS number. (Note that a dual-ToR is considered as a single leaf and both nodes in this pair will have
same AS number).
• All spines inside a PoD belong to one AS.
• All super-spines are configured in one AS.
• Edge or border leafs belong to a separate AS.
• Each leaf peers with all spines using eBGP.
• Each spine peers with all super-spines using eBGP.
• There is no eBGP peering between leafs.
• There is no eBGP peering between spines.
• There is no eBGP peering between super-spines.
16
Figure 3 eBGP for Underlay
iBGP for Underlay

In this deployment model, each PoD and edge services PoD is configured with a unique AS number, as shown in Figure 4. The spines and leafs in a
PoD are configured with the same AS number. The iBGP design is different than the eBGP design because iBGP must be fully meshed with all BGP-
enabled devices in an IP fabric. In order to avoid the full mesh of BGP peering, route reflectors must be used in the fabric. Each spine acts as a route-
reflector for underlay IPv4 routes to the leaf nodes inside the PoD. iBGP peering is between the spine and the leaf in a PoD, and all spines in a PoD
act as BGP route reflectors to the leafs for the underlay.
Note that this model is given for informational purposes only and not part of this validated design. However iBGP model is supported.
17
Figure 4 iBGP for Underlay
18
Network Virtualization with BGP VxLAN Based EVPN
Network virtualization is the process of creating virtual, logical networks on physical infrastructures. With network virtualization, multiple physical
networks can be consolidated to form a logical network. Conversely, a physical network can be segregated to form multiple virtual networks. Virtual
networks are created through a combination of hardware and software elements spanning the networking, storage, and computing infrastructure.
Network virtualization solutions leverage the benefits of software in terms of agility and programmability, along with the performance acceleration
and scale of application-specific hardware.
Virtual Extensible LAN (VXLAN) is an overlay technology that provides Layer 2 connectivity for workloads residing across the data center network.
VXLAN creates a logical network overlay on top of physical networks, extending Layer 2 domains across Layer 3 boundaries. VXLAN provides
decoupling of the virtual topology provided by the VXLAN tunnels from the physical topology of the network. It leverages Layer 3 benefits in the
underlay, such as load balancing on redundant links, which leads to higher network utilization. In addition, VXLAN provides a large number of logical
network segments, allowing for large-scale multitenancy in the network. VXLAN is based on the IETF RFC 7348 standard. VXLAN has a 24-bit Virtual
Network ID (VNI) space, which allows for 16 million logical networks compared to a traditional VLAN, which supports a maximum of 4096 logical
segments. VXLAN eliminates the need for Spanning Tree Protocol (STP) in the data center network, and it provides increased scalability and improved
resiliency. VXLAN has become the de facto standard for overlays that are terminated on physical switches or virtual network elements.
The traditional Layer 2 extension mechanisms using VXLAN rely on "Flood and Learn" mechanisms. These mechanisms are very inefficient, delaying
MAC address convergence and resulting in unnecessary flooding. Also, in a data center environment with VXLAN- based Layer 2 extension
mechanisms, a Layer 2 domain and an associated subnet might exist across multiple racks and even across all racks in a data center site. With
traditional underlay routing mechanisms, routed traffic destined to a VM or a host belonging to the subnet follows an inefficient path in the network,
because the network infrastructure is aware only of the existence of the distributed Layer 3 subnet, but it is not aware of the exact location of the
hosts behind a leaf switch.
With the Extreme BGP-EVPN, network virtualization is achieved by creating a VXLAN-based overlay network. It leverages BGP EVPN to provide a
control plane for the virtual overlay network. BGP EVPN enables control-plane learning for end hosts behind remote VXLAN tunnel endpoints
(VTEPs). This learning includes reachability for Layer 2 MAC addresses and Layer 3 host routes.
Some key features and benefits of Extreme BGP-EVPN network virtualization are summarized as follows:
Active-active MCT/vLAG pairs—Multi-Chassis port channel for dual homing of network endpoints are supported at the leaf. Both switches in a
MCT/vLAG pair participate in the BGP-EVPN operations and are capable of actively forwarding traffic.
Static anycast gateway—with static anycast gateway technology, each leaf is assigned the same default gateway IP and MAC addresses for all
connected subnets. This ensures that local traffic is terminated and routed at Layer 3 at the leaf. This also eliminates any suboptimal inefficiencies
found with centralized gateways. All leafs are simultaneously active forwarders for all default traffic for which they are enabled. Also, because the
static anycast gateway does not rely on any control-plane protocol, it can scale to large deployments.
Efficient VXLAN routing—with the gateway moved to the leaf, routing of packets between VXLAN networks occur at the leaf. Routed traffic from the
network endpoints is terminated in the leaf and is then encapsulated in the VXLAN header to be sent to the remote site. Similarly, traffic from the
remote leaf node is VXLAN-encapsulated and gets decapsulated and routed to the destination. This VXLAN routing operation in to and out of the
tunnel (RIOT) on the leaf switches is enabled in the Extreme SLX and VDX platform ASICs. VXLAN routing performed in a single pass is more efficient
than competitive ASICs.
Data-plane IP and MAC learning and Control-plane —With IP host routes and MAC addresses learned from the data plane and advertised with BGP
EVPN, the leaf switches are aware of the reachability of the hosts in the network. Any traffic destined to the hosts takes the most efficient route in the
network.
Layer 2 and Layer 3 multitenancy—BGP EVPN provides the control plane for VRF routing and for Layer 2 VXLAN extension. BGP EVPN enables a
multitenant infrastructure and extends it across the data center to enable traffic isolation between the Layer 2 and Layer 3 domains, while
providing efficient routing and switching between the tenant endpoints.
Dynamic tunnel discovery—With BGP EVPN, the remote VTEPs are automatically discovered. The resulting VXLAN tunnels are also automatically
created. This significantly reduces operational expense (OpEx) and eliminates errors in configuration.
19
ARP/ND suppression—The BGP-EVPN EVI leafs discover remote IP and MAC addresses and use this information to populate their local ARP tables.
Using these entries, the leaf switches respond to any local ARP queries. This eliminates the need for flooding ARP requests in the network
infrastructure.
Conversational ARP/ND learning—Conversational ARP/ND reduces the number of cached ARP/ND entries by programming only active flows into the
forwarding plane. This helps to optimize utilization of hardware resources. In many scenarios, there are software requirements for ARP and ND
entries beyond the hardware capacity. Conversational ARP/ND limits storage-in-hardware to active ARP/ND entries; aged-out entries are deleted
automatically.
VM mobility support—if a VM moves behind a leaf switch, with data-plane learning, the leaf switch discovers the VM and learns its addressing
information. It advertises the reachability to its peers, and when the peers receive the updated information for the reachability of the VM, they
update their forwarding tables accordingly. BGP-EVPN-assisted VM mobility leads to faster convergence in the network.
Open standards and interoperability—BGP EVPN is based on the open standard protocol and is interoperable with implementations from other
vendors. This allows the BGP-EVPN-based solution to fit seamlessly in a multivendor environment.
VXLAN Layer 2 Extension Using Flood and Learn

Let's consider the simple topology shown in Figure 5, which represents VXLAN extension, to understand how VXLAN flood and learn works before going
into the details of control-based VXLAN using BGP EVPN and the various network functions that the EVPN control plane enables.
Figure 5 VTEPs and L2 Extension with Flood and Learn
Ingress Replication
IP Network Underlay
I MAC-H2
MAC-H1 10.10.1.2
VTEP-A VTEP-B
10.10.1.1
10.10.10.1 10.10.10.2
VLAN10, VNI10 VLAN20, VNI10
VTEP-C
10.10.10.3
VTEP-C L2 table
VLAN30, VNI10
MAC VTEP-IP
H1 10.10.10.1
H2 10.10.10.2
MAC-H3 H3 Local Eth port
10.10.1.3
VXLAN tunnel end point (VTEP) may be implemented in hardware (leaf or ToR switch) or in virtualized environments. Each VTEP has a unique IP
address and MAC address. Each VTEP can reach other VTEPs over the underlay IP network.
Each VTEP has its own end host/server segment connected to it. In this topology, all hosts belong to one Layer 2 broadcast domain or, in simple
terms, one VLAN and one IP subnet. The local VLAN numbers may be different in each VTEP, but they are bound to one VNI number, which is
common on all VTEPs. So for all practical purposes, the LAN segment is now identified by a VXLAN VNI, and the VLAN numbers are only locally
significant.
The logical dashed lines shown inside the IP network between the VTEPs represent the head-end or ingress replication paths. This is used to send
what is known as the BUM traffic: Broadcast, Unknown Unicast, and Multicast frames on the Layer 2 segment. The VTEP unicasts these packets to all
other VTEPs connected to a VXLAN segment. This may require additional configuration or provisioning of tunnels on each VTEP device to all other
devices.
20
Let's consider that H1 wants to communicate with H2:
• H1 sends an ARP request.
• VTEP-A learns H1 as a local MAC and also maps this host to the VNI, and because the packet is a broadcast packet, it is encapsulated into the
VXLAN packet and replicated; it is then unicast to each of the remote VTEPs participating in this VNI segment. The outer-src-ip is set to
10.10.10.1, and the outer-dst-ip is the remote VTEP IP.
• This packet is sent to every VTEP.
• VTEP-B and VTEP-C decapsulate the packet and flood it into their local VXLAN network.
• They also learn three pieces of information: the source-ip of VTEP-A, the inner-src-mac of H1, and the VNI. This creates an L2-MAC-to-VTEP-
IP binding: {mac H1, VTEP-ip 10.10.10.1, VNI 10}.
• When H2 responds to the ARP request, the packet is unicast to H1. This packet is encapsulated in a VXLAN packet by VTEP-B and sent as a
unicast IP packet based on its routing table:
o Outer-IP header—dst: 10.10.10.1, src 10.10.10.2
• VTEP-A decapsulates the packet and sends it to H1. It also creates an L2-MAC-to-VTEP-IP binding: {MAC H2, VTEP-IP 10.10.10.2, VNI 10}.
• Now the communication between H1 and H2 will be unicast. VTEP-A and VTEP-B now know sufficient information to encapsulate the packets
directly between them.
When the hosts are in different subnets, we need a Layer 3 gateway in the network to connect to all VNI segments. As seen in Figure 6, VTEP-C is
configured with all VNI numbers in the network and acts as the router or gateway between these VNI segments (see the blue and red dotted arrows
routing between VLAN10 and VLAN20). When hosts send ARP messages for the gateway in their respective VLANs, VTEP-C will respond. For first-hop
router redundancy, multiple VTEPs may be configured with all VNIs, and they may run an FHRP protocol between them.
Figure 6 Routing Between VxLAN networks in Flood and Learn topology
VTEP-A - 10.10.10.1 VTEP-B - 10.10.10.2

IP network underlay Host-B

Host-A
VLNA20
VLAN10
VTEP-C 10.10.10.3
VLAN10, VNI10, GW-IP
Host-C
VLAN10
21
BGP EVPN for VXLAN
As we have seen in the VXLAN flood and learn case, the MAC learning is data frame-driven and flooding of broadcast or unknown unicast frames
depends on ingress replication by VTEPs in the network.
With the BGP EVPN control plane, the MAC learning happens via BGP similar to IPv4/IPv6 route learning in a Layer 3 network. This reduces flooding in
the underlay network except for remarkably silent hosts. This control-plane-based MAC learning enables several additional functions with BGP as the
unified control plane for both Layer 2 and Layer 3 forwarding in the overlay network.
In Figure 7, each VTEP, being a BGP speaker, advertises the MAC and IP addresses of its local hosts to other VTEPs using the BGP EVPN control plane. A
BGP route reflector may be used for distribution of this information to the VTEPs. Both VTEP discovery and MAC/IP or MAC/IPv6 host learning happen
through the control plane.
Since IPv4/IPv6 addresses are also exchanged in the control plane, each VTEP may act as a gateway for the VNI subnets configured on it. A centralized
Layer 3 gateway is not required. This feature is also referred to as distributed gateway. Also, since each VTEP is aware of MAC/IP or MAC/IPv6 host
bindings, ARP requests need not be flooded between the VTEPS. The VTEP may respond to the ARP requests on behalf of the target host, if the host
address has already been learned. This is referred to as ARP/ND suppression in the fabric.
Figure 7 VTEPs and L2 Extension with BGP EVPN Control-plane
BGP sessions between VTEPs

Exchanging EVPN Address family
VTEP-A 10.10.10.1 VTEP-B 10.10.10.2
MAC-H1 I MAC-H2
10.10.1.1 10.10.1.2
VTEP-C 10.10.10.3
VLAN30, VNI10
MAC-H3
10.10.1.3
BGP EVPN control-plane-based learning allows more flexibility to control the information flow between the VTEPs. It enables layer-2 multi-tenancy
using MAC-VRF constructs. In a simple terms, each VLAN or a Bridge-domain can be considered as a MAC-VRF, and MAC addresses from the remote
VTEPs get downloaded into it.
BGP-EVPN also enables layer-3 multitenancy using VRFs similar to MPLS-VPN. Each VTEP may host several tenants and each tenant with a set of
VXLAN segments. Depending on the interest, other VTEPs may import the tenant-specific information. This way both Layer 2 and Layer 3 extensions
can be provisioned on a tenant basis.
BUM traffic is accommodated using ingress replication at the VTEP. Since VTEP discovery also happens through the control plane, setting up ingress
replication does not require additional provisioning or configuration about remote VTEPs.
Let’s look at the functional components of the BGP EVPN implementation of a data center IP Fabric.
VTEP
In an IP fabric, the leaf and border leaf act as VTEPs. Note that only one VTEP is allowed per device. Every VTEP has an overlay interface, which
identifies the VTEP IP address. The VTEP information is exchanged, and remote VTEPs are discovered over BGP EVPN.
22
Static Anycast Gateway
Each leaf or VTEP has a set of server-facing VLANs that are mapped to VXLAN segments by a VNI number. These VLAN segments have an associated VE
interface (a Layer 3 interface for the VLAN). Each tenant VLAN has anycast gateway IPv4/IPv6 addresses and associated anycast gateway MAC
addresses. These gateway IP/IPv6 addresses and gateway MAC address are consistent for the VLAN segments shared on all leafs in the fabric.
Overlay Gateway
Each VTEP or leaf is configured with an overlay gateway. This defines the VTEP IP address, which is used as the source IP when encapsulating packets
and is used as the next-hop IP in the EVPN NLRIs. In this validated design, we are using an IPv4 underlay; hence the overlay interface is associated with
the IPv4 address of a loopback interface on the leaf.
BGP EVPN Control Plane

The BGP EVPN control plane is used for VTEP discovery to learn MAC/IP routes from other VTEPs. The exchange of this information takes place using
EVPN NLRIs. The NLRI uses the existing AFI of 25 (L2VPN). IANA has assigned BGP EVPNs a SAFI value of 70. The NLRI also carries a tunnel
encapsulation attribute. For an IP fabric using VXLAN encapsulation, the attribute is set to VXLAN.
In the leaf-spine topology (3-stage Clos or 5-stage Clos), all leafs and border leafs should be enabled with the BGP EVPN Address- Family to exchange
EVPN routes (NLRI) and participate in VTEP discovery. Spine and super-spines do not participate in the VTEP functionality. However, selected spines in
the spine layer are enabled with the BGP EVPN Address-Family for distribution of routes, and all leafs including border leafs must be peered with these
spines who have the BGP EVPN Address-Family enabled.
In the deployment model where eBGP is used, a minimum of two spines in a 3-stage PoD should be enabled with the EVPN Address-Family. Note that
all spines participate in the eBGP underlay, but only a few designated spines participate in the EVPN.
In the deployment model where iBGP is used, two spines are selected as route reflectors for the EVPN Address-Family, and each VTEP leaf has two
iBGP neighbors that are the two spine BGP route reflectors. Each spine BGP route reflector has all VTEP leaf nodes as route-reflector clients and
reflects EVPN routes for the VTEP leaf nodes.
In the 5-stage Clos topology, a minimum of two super-spines should be enabled with the EVPN Address-Family.
EVPN Route Types

EVPN uses different route types to carry various network-layer reachability information. The following are the well-known route types defined in BGP
EVPN:
• Route Type-1—Ethernet Auto Discovery route. This route is used in multi-homing cases to achieve split-horizon, aliasing, and fast
convergence.
• Route Type-2—MAC/IP advertisement route:
– MAC-only route that carries {MAC address of the host, L2VNI of the VXLAN segment}. This route carries only the Layer 2 information of
a host. Whenever a VTEP learns a MAC from its server-facing subnets, it advertises this route into BGP.
– MAC/IP route that carries {MAC address of the host, IPv4/IPv6 address of the host, L2VNI of the VXLAN segment, L3VNI of the tenant
VRF of the host}. This route carries both the Layer 2 and Layer 3 information of the hosts. This route is advertised by the VTEP when it
learns the IPv4/IPv6 host addresses via ARP or ND from the server-facing subnets. This information enables ARP/ND suppression on
other VTEPs.
• Route Type-3—Inclusive Multicast Ethernet Tag route. This route is required for sending BUM traffic to all VTEPs interested for a given
bridge domain or VXLAN segment.
• Route Type-4—Ethernet Segment route. This route is used for multi-homing of server VLAN segments. Note that only MCT or VLAG- based
multi-homing is supported.
• Route Type-5—IPv4/IPv6 prefix advertisement route {IPv4/IPv6 route, L3VNI, Router-MAC}. This route is advertised for every Layer 3 server-
facing subnet behind a VTEP or external routes.
23
Tunnel Attribute
Extended community type 0x3, sub-type 0x0c, and tunnel encapsulation type 0x8 (VXLAN). This is included with all EVPN routes.
Layer 3 VNI or Tenant VRF
Each tenant VRF is configured with a unique Layer 3 VNI. This is required for inter-subnet routing. This VNI must be the same for a tenant VRF on all
VTEPs including the border leaf. Both Type-2 and Type-5 routes carry this Layer 3 VNI.
Router-MAC Extended Community
Extended community type EVPN (0x06) and sub-type 0x03.
The router-mac is the MAC address of the VTEP advertising a route. This is also required along with the Layer 3 VNI for inter-subnet routing, as
explained in Integrated Routing and Bridging section; and it is carried in both Type-2 MAC/IP routes and Type-5 prefix routes. In the data plane, this
MAC address is used as the inner destination MAC address when a packet is routed.
MAC-Mobility Attribute
Extended community type EVPN (0x06) and sub-type 0x00. Carries a 32-bit sequence number.
This enables MAC or station moves between the VTEPs. When a MAC moves, for example, from VTEP-1 to VTEP-2, VTEP-2 advertises a MAC (or
MAC/IP) route with a higher sequence number. This update triggers a best-path calculation on other VTEPs, thereby detecting the host move to VTEP-
2.
ARP Suppression
Control-plane distribution of MAC/IP addresses enables ARP suppression in the fabric for Layer 2 extensions between racks. A portion of the fabric is
shown in Figure 8 to illustrate the ARP suppression functionality in the fabric.
When the hosts come up, they typically ARP for the gateway IP that is hosted by leafs. Let's consider the case where H2 ARPs for the gateway address.
Note that both leafs have the same anycast gateway address for the host VXLAN segment.
• Leaf2 learns the MAC/IP (or ARP) binding for H2.
• Leaf2 will advertise the MAC/IP route into the BGP EVPN Address-Family.
• Leaf1 will learn this route and populate it in its MAC/IP binding table.
• H1 sends an ARP request to H2. Leaf1 will respond on behalf of H2.
• Extending the same information flow for H1, when Leaf2 learns H1's MAC/IP route, it will respond to ARP requests on behalf of H1.
Compared to the data-plane-based learning in Layer 2 extension technologies such as VPLS or VXLAN flood and learn, where ARP traffic is also sent
over an overlay network, VXLAN EVPN significantly reduces ARP/ND flooding in the fabric.
24
Figure 8 ARP Suppression
(2) BGP update to Leaf1 – M2/10.10.1.2;

Leaf updates its host table
Leaf1 Leaf2
VNI 10 VNI 10
Anycast Gateway Anycast Gateway
(3) H1 sends ARP for H2; (1) H2 sends ARP for Gateway IP;
Leaf1 responds on behalf of H2 Leaf2 update its host table
H1 H2
MAC M1 MAC M2
IP 10.10.1.1 IP 10.10.1.2
VLAN 10 VLAN 10
VLAN Scoping
As discussed earlier, in VXLAN networks, each VLAN is mapped to a VNI number of a VXLAN segment. This provides an interesting option to break
the 4K limit of the 802.1Q VLAN space. The VLAN tag (or c-tag) on the wire or the port VLAN membership may be locally scoped or locally significant
at the leaf level or at the port level within a leaf.
VLAN Scoping at the Leaf Level

In this case, the VLANs are scoped at the leaf or ToR level. Refer to Figure 9.
In this example,
- VLAN 10 is mapped to VNI 10 on Leaf1
- VLAN 20 is mapped to VNI 10 on Leaf2.
By mapping to the same VNI, the two VLAN segments (VLAN 10 and VLAN 20) are on the same bridge domain. With this mapping, hosts on these
VLANs have Layer 2 extension between them, and they belong to one VXLAN segment identified by the VNI 10.
25
Figure 9 VLAN Scoping at the Leaf/ToR Level
Leaf1 Leaf2
VLAN10  VNI 10 VLAN20  VNI 10
Anycast GW 10.10.1.254 Anycast GW 10.10.1.254
L2 Extension or same Bridge-domain
H1 H2
MAC M1 MAC M2
IP 10.10.1.1 IP 10.10.1.2
VLAN 10 VLAN 20
VLAN Scoping at the Port Level within a Leaf

VLAN scoping at the port level can be accomplished using the Bridge-domain feature of SLX devices. (VDX platforms support the same functionality
with virtual fabric GVLAN feature). This basically abstracts a VLAN or bridge domain and decouples the VLAN tag (or c-tag) on the wire.
Refer to Figure 10. In this example, Port1, VLAN tag 10, and Port2, VLAN tag 20, are mapped to bridge-domain BD 100 , and BD 100 is mapped to VNI
4196. With this mapping, the hosts H1 (VLAN 10), H2 (VLAN 20), and H3 (VLAN 501) are bound to one VXLAN segment identified by the VNI 4196.
Figure 10 VLAN Scoping at the Port Level within a ToR
Leaf1 Leaf2
BD 100  VNI 4196 BD 100  VNI 4196
BD 100: BD 100:
- Port1, tag 10 Port2 Port1, tag 501
- Port2, tag 20 Port1 Port1 Anycast GW 10.10.1.254
Anycast GW 10.10.1.254
L2 Extension or same Bridge-domain
H1 H2 H3
MAC M1 MAC M2 MAC M3
IP 10.10.1.1 IP 10.10.1.2 IP 10.10.1.3
VLAN tag 10 VLAN tag 20 VLAN tag 501
26
Conversational Learning
Conversational learning helps conserve the hardware forwarding table by programming only those ARP/ND or MAC entries for which there are active
conversations or traffic flows. With this feature, the control plane may hold more host entries than what the hardware table can support. When there
is sufficient space in hardware, all host entries are programmed. When there is no space, conversational learning kicks in and starts aging out the
inactive entries. Note that the host subnets are inserted into the hardware (LPM table) regardless of the activity. The host entries are inserted in the
hardware (/32 IPv4 or /128 IPv6 host route table) based on the traffic.
Integrated Routing and Bridging

With the anycast gateway function, each VTEP or leaf acts as an Integrated Routing and Bridging (IRB) device providing Layer 2 extension as well Layer
3 routing between the VXLAN segments in a tenant. Note that the tenant may span multiple leafs. There are two variations of IRB implementation in
the IP fabric: asymmetric IRB and symmetric IRB.
Asymmetric IRB
Figure 11 Asymmetric IRB
Leaf1 Leaf2
VLAN 10  VNI 10, Anycast GW 10.10.1.254 VLAN 10  VNI 10, Anycast GW 10.10.1.254
Tenant VRF SALES Tenant VRF SALES
H1 H2 H3
IP 10.10.1.1 IP 10.10.1.2 IP 10.20.1.1
VLAN 10, VNI 10 VLAN 10, VNI 10 VLAN 20, VNI 20
In Figure 11, a tenant, SALES, is provisioned in the fabric with two VNI segments, VNI 10 and VNI 20. Leaf1 has servers connected to it on VNI 10 only.
However, it is provisioned with both VLAN 10 and VLAN 20 and mapped to VNI 10 and VNI 20 respectively. Similar configuration is done on Leaf2. Both
Leafs act as first-hop gateways for these VLANs with anycast gateway address.
If H1 in VNI 10 needs to communicate with H3 in VNI 20, Leaf1 routes the packet first between the segments and then bridges the packet on VNI 20
and the packet is sent on the Overlay. Leaf2 will decapsulate the VXLAN headers and send the packet to H3.
Essentially, the ingress VTEP both routes and bridges the packet; this method is referred as asymmetric IRB. This also means that every VTEP must be
configured with all VLANs irrespective of the existence of local workloads on those VLANs.
Symmetric IRB
Figure 12 depicts symmetric IRB. Here, every tenant is assigned a Layer 3 VNI. This is analogous to a Layer 3 routing interface between two switches.
This VNI must be the same for a given tenant on all leafs where it is provisioned.
27
The MAC/IP host routes are advertised by the VTEP with the L2 VNI as well as an L3 VNI and the router-mac address of the VTEP. When a packet is
routed over the L3 VNI, the dst-mac of the inner Ethernet payload is set to the router-mac of the remote VTEP. In Figure 12, routing from H1 to H3
always occurs over this L3 VNI. That is, both leaf devices route the packet once: by the ingress leaf from the server VLAN/VNI to the L3 VNI and by the
egress leaf from the L3 VNI to the server VLAN/VNI.
A significant advantage of this method is that all VNIs of a given tenant need not be created on all leafs. They are created only when there is server
connectivity to those VNIs. In Figure 12, Leaf1 is not configured with VNI 20. Also note that on Leaf2, even though VNI 10 is present, a packet from H3
to H1 will be routed directly on to the L3 VNI of the tenant. This adds the additional requirement that the host routes on all VXLAN segments in a given
tenant need to be downloaded to the leaf's forwarding table.
Figure 12 Symmetric IRB
L3 VNI 2000
Leaf2
Leaf1 RMAC RM2
RMAC RM1 VLAN 10  VNI 10, Anycast GW 10.10.1.254
Tenant VRF Sales, L3 VNI 2000 Tenant VRF Sales, L3 VNI 2000
H1 H2 H3
IP 10.10.1.1 IP 10.10.1.2 IP 10.20.1.1
Extreme IRB Implementation

Both symmetric and asymmetric IRB methods are implemented on Extreme switches. If the target VNI segment is configured on a VTEP, asymmetric
IRB is performed. Otherwise, the packet is routed over the L3 VNI or symmetric routing occurs. Every tenant VRF is assigned with an L3 VNI.
In the Extreme BGP EVPN implementation, we get the best of both schemes:
• There is no need to create all server VNIs on all leafs for a tenant.
• If a target VNI segment is not local and is extended behind one or more remote VTEPs, download the host routes on that target segment
into hardware based on traffic activity. Traffic to these hosts will be routed over the L3 VNI.
Multitenancy
Layer 2 multitenancy is achieved by a MAC-VRF construct used for extending a VLAN between multiple VTEPs or ToRs.
In BGP EVPN, multiple tenants can co-exist at the Layer 3 level and share a common IP transport network while having their own separate routing
domain in the VXLAN overlay network. Every tenant in the EVPN network is identified by a VRF (VPN routing and forwarding instance), and these
tenant VRFs can span multiple leafs in a data center. (Similar to Layer 3 MPLS VPNs with tenant VRFs on multiple PE devices.). Each VRF can have a set
of server-facing VLANs, routing interfaces for those VLANs with anycast gateways, and a Layer 3 VNI used for symmetric routing purposes. This VNI
should be the same if the same tenant VRF is provisioned on other leafs including a border leaf.
We recommend the separation of the tenant routing domain from the underlay routing domain (or default VRF), which is used for setting up the
overlays or tunnels between the VTEPs.
28
Even if Layer 3 multitenancy is not required in a deployment (this is the case with a single tenant), we recommend moving the server subnets to a
separate VRF and keeping a clear separation of underlay and overlay routing domains. By using a separate VRF for server subnets, there is a visibility
into host routes and we can leverage the host route optimization in the data plane. A tenant VRF also allows provisioning a L3 VNI that enables
symmetric IRB.
A tenant VRF may not be needed in the case of pure L2 or VLAN extension between the VTEPs.
Figure 13 Multitenancy
L3 VNI 5003
L3 VNI 5001
L3 VNI 5002
Leaf1 Leaf2 Border-leaf
VRF101 VRF101
VRF101
L3 VNI 5001 L3 VNI 5001
L3 VNI 5001
VLAN 200, 201 VLAN 201. 202 VRF11
L3 VNI 5002
VRF11 VRF21
L3 VNI 5002 L3 VNI 5003 VRF21
VLAN 100,102 VLAN 300, 301 L3 VNI 5003
Ingress Replication
Although host reachability information is exchanged over the control plane to drastically reduce flooding in a VLAN, certain situations require the
flooding of frames, as in traditional Ethernet networks such as but not limited to:
• MAC aging
• Silent hosts
• L2 multicast or broadcast
Ingress replication is a technique used to accommodate flooding in such cases by the VTEPs in the IP fabric. Each VTEP for a given VXLAN segment (or
server VLAN) computes the list of VTEPs having the same segment using the IMR (Inclusive Multicast Route) routes. Whenever the VTEP must flood a
frame in a VXLAN segment, it replicates the frame in hardware and unicasts the frame to each of the VTEPs in the IMR list for that segment.
MCT Pair
SLX platforms support MCT between two nodes. This is the recommended solution for leaf level redundant connectivity for the workloads. Multi-
homing is supported only using a MCT pair. Multi-homing to two separate VTEPs or leaf nodes, is not supported.
When the two leafs are in a MCT pair, they act as one logical VTEP or endpoint. As shown in Figure 14, both leafs are configured with the same VTEP IP
address. From other VTEPs in the network, this pair appears as a single VTEP. This is very important because having two physical switches in this mode
on each rack does not result in an increased number of VTEPs or additional tunneling requirements on other VTEPs in the network.
VDX platforms support similar functionality with two devices in a vLAG pair.
29
Figure 14 MCT Pair for Dual-homing and Leaf Redundancy
Leaf1 Leaf2
VTEP-IP 10.121.1.1 VTEP-IP 10.121.1.1
ICL
H1 H2 H3
IP 10.10.1.1 IP 10.10.1.2 IP 10.20.1.1
30
Validated Designs
This section provides the details of the deployment model with the validated configuration templates. Extreme validated design recommends a
deployment model for the IP fabric deployment, which uses eBGP as the control protocol between the tiers of the nodes in the Fabric. Depending on
the scale, the user may choose either a 3-stage or a 5-stage fabric. The number of racks inside a PoD is determined by the port density of the Spines.
The location of the workloads and connectivity challenges also determine the size of a PoD.
eBGP Deployment Model

This design using eBGP as a routing protocol within the data center is based on the IETF draft: Use of BGP for routing in large-scale data centers.[2]
Figure 15 shows the design for a 3-stage IP fabric using eBGP as the control protocol to exchange both underlay (IPv4 unicast) and overlay
(L2VPN/EVPN) routes. Note that the border leafs are connected to the spines in this design. BGP uses IPv4 as transport for peering between the tiers.
There is no BGP peering between the nodes in the same tier.
Figure 15 EBGP based 3-stage IP Fabric
SPINE
AS 4200000000 Internet
MPLS Network
WAN EDGE
Rack-1 Rack-2 Rack-N

AS 4200000001 AS 4200000003 AS 4200000031 Edge Services
AS 4200007000
Leaf Border-Leaf
31
The design shown in Figure 16 is a 5 stage IP Fabric where a super-spine layer interconnects multiple 3-stage PoDs.
Figure 16 EBGP based Optimized 5-stage IP Fabric
Super-Spine
AS 4200003000
Internet MPLS Network
WAN EDGE
AS 4200000000 AS 4200002000
Spine Spine
Edge Services
AS 4200007000
Border-Leaf
Rack-1 Rack-2 Rack-N Rack-1 Rack-2 Rack-N

AS 4200000001 AS 4200000003 AS 4200000031 AS 4200002001 AS 4200002003 AS 4200002031
Leaf Leaf
POD1 POD2
32
Hardware and Software Matrix
TABLE 1 Platforms Used in This Validated Design
Places in the Network Platform Software Version
Leaf Nodes SLX 9140 SLX OS 17s.1.02

Spine Nodes SLX 9240 SLX OS 17s.1.02
Super-Spine Nodes SLX 9850 SLX OS 17r.1.01ad
Border Leaf SLX 9140 SLX OS 17s.1.02
WAN Edge Router SLX 9540 SLX OS 17r.1.01ad
33
3-Stage Fabric
Fabric Infrastructure Configuration
This section covers the aspects of provisioning the building blocks of the IP fabric underlay infrastructure. This involves the common configurations
on the fabric nodes, the loopback interfaces used as the router ID and VTEP address, and the interfaces or links between the fabric nodes (also
referred to as the fabric infrastructure links).
Building blocks of the Fabric:
• Nodes – Leaf, Spine and Edge-leaf.
• Fabric infrastructure links connecting the Nodes.
• Loopback interface with a unique IPv4 address as Router-ID on each node.
• Loopback interface with a unique IPv4 address as VTEP-IP on Leaf and Edge-Leaf.
• Dual homing and redundancy at the Leaf and Edge-Leaf using a pair of nodes in MCT or vLAG.
Global MTU configuration

mtu 9216
ipv6 mtu 9100
ip mtu 9100

All nodes in the IP fabric—leafs, spines, and super-spines—are interconnected with Layer 3 interfaces. In the validated design,
• 40-G links are used between the nodes.
• The MTU for these links is set to Jumbo MTU. This is a requirement to handle the VXLAN encapsulation of Ethernet frames. In the
configuration shown below, the IPv4 MTU of the link between Spine and Leaf is set to 9100.
The link configurations on two nodes – a spine and a leaf – interconnected with a fabric link, are shown below.
interface Ethernet 0/51

description Link to Spine1
ip address 10.0.1.0/31
no shutdown
!

description Link to Leaf1
no shutdown
!
Loopback Interfaces
Each device in the fabric needs one loopback interface with a unique IPv4 address for the purpose of Router-ID.
34
interface Loopback 1
ip address 10.1.1.11/32
no shutdown
!
ip router-id 10.1.1.11
Each leaf and border leaf needs a loopback interface with a unique IPv4 address to use as VTEP-IP. This interface is used under overlay gateway
configuration as shown below. This is not required on spine and super-spines.
no shutdown
!
overlay-gateway mct-leaf
type layer2-extension
ip interface Loopback 2
map vni auto
activate
!
SLX-9140 MCT pair as Leaf

SLX platforms support MCT pair for dual-homing of server connections. In this design, we have used SLX-9140 in pairs as Leaf and Edge-Leaf devices.
This section will provide step by step configuration of a MCT pair Leaf. The configuration templates are shown below side by side for the devices.
Each device in the MCT pair connects to every Spine in the network with L3 links also referred to as Fabric Infrastructure Links.
Each device must be configured with their own loopback interfaces for unique router-ids.
interface Loopback 1 interface Loopback 1

ip address 10.1.1.11/32 ip address 10.1.1.12/32
no shutdown no shutdown
! !
ip router-id 10.1.1.11 ip router-id 10.1.1.12
However both of them share same VTEP IP address. This is achieved by configuring same IP address on another loopback interface.

! !
overlay-gateway mct-leaf overlay-gateway mct-leaf
type layer2-extension type layer2-extension
ip interface Loopback 2 ip interface Loopback 2
map vni auto map vni auto
activate activate
! !
Cluster Configuration
This involves the following steps:
• Node ID for each device in the pair
• MCT control vlan and a VE interface with IPv4 address
• ICL member ports and a port-channel. We strongly recommend a minimum of two ports in this bundle. They must of the same type and
speed. In the validated design, as shown in the illustration below, the MCT pair is interconnected by two 40G Ethernet ports for ICL port-
channel.
35
• EVPN instance and BGP peering between the peers. The BGP AS number is same as the one assigned to the Leaf pair in the fabric. Both
devices in a cluster use the same AS number.
The server facing port-channel configuration is provided under “Tenant Provisioning” section under “Network Virtualization with BGP EVPN”.
VDX platforms support dual-homing with vLAG feature. Please refer to the appendix for provisioning VDX vLAG as ToR.
interface Ethernet 0/49 interface Ethernet 0/49

description MCT peer-intf member description MCT peer-intf member
channel-group 1 mode active type standard channel-group 1 mode active type standard
! !
description MCT peer-intf member description MCT peer-intf member
! !
interface Port-channel 1 interface Port-channel 1
description MCT peer-intf LAG description MCT peer-intf LAG
MCT-Peer1 MCT-Peer2
ICL Links
mLAG
node-id 1 node-id 2
cluster management principal-priority 1 !
! !
vlan 4090 vlan 4090
router-interface Ve 4090 router-interface Ve 4090
description MCT control-vlan description MCT control-vlan
! !
interface Ve 4090 interface Ve 4090
! !
cluster pod1-cluster 1 cluster pod1-cluster 1
peer-interface Port-channel 1 peer-interface Port-channel 1
peer 10.0.1.9 peer 10.0.1.8
df-load-balance df-load-balance
deploy deploy
! !
evpn default evpn default
route-target both auto ignore-as route-target both auto ignore-as
rd auto rd auto
! !
router bgp router bgp
neighbor 10.0.1.9 remote-as 4200000001 neighbor 10.0.1.8 remote-as 4200000001
neighbor 10.0.1.9 bfd neighbor 10.0.1.8 bfd
address-family ipv4 unicast address-family ipv4 unicast
no neighbor 10.0.1.9 activate no neighbor 10.0.1.8 activate
address-family l2vpn evpn address-family l2vpn evpn
neighbor 10.0.1.9 encapsulation nsh neighbor 10.0.1.8 encapsulation nsh
neighbor 10.0.1.9 activate neighbor 10.0.1.8 activate
36
BGP Underlay Configuration
When enabling network virtualization with EVPN overlay, the underlay configuration needs to accommodate the BGP peers that exchange only
IPv4 routes and the BGP peers that exchange both IPv4 and EVPN routes. This is accomplished by using BGP peer groups.
• Two spines exchange only IPv4 Address-Family routes.
• Two spines exchange both IPv4 and EVPN Address-Family routes.
Leaf Configuration
This is applicable to all leafs. They form bgp peering with all Spines inside the 3-stage PoD.
For MCT pair leaf and border-leaf, each node in the pair must be configured with BGP peering to spines in a similar manner. Nodes in a MCT pair will
advertise a common VTEP IP address into the underlay.
• Configure the directly connected IP addresses of the spines into a peer groups: spine-group.
• Enable MD5 authentication and BFD to the spine-group.
• Enable the IPv4 Address-Family, and advertise the VTEP IP address.
router bgp
local-as 4200000001 4 Byte AS Number of this leaf.
capability as4-enable Enable 4 Byte AS capability.
fast-external-fallover
bfd interval 300 min-rx 300 multiplier 3
!
neighbor spine-group peer-group
neighbor spine-group remote-as 4200000000 Peer-group config pointing to spines.
neighbor spine-group description To spine Enable MD5 authentication and BFD
neighbor spine-group password <password>
neighbor spine-group bfd
!
neighbor 10.0.1.1 peer-group spine-group
neighbor 10.0.1.7 peer-group spine-group Enable IPv4 Address-Family.
! Advertise the VTEP-IP.
address-family ipv4 unicast Enable graceful-restart.
network 10.1.1.1/32
maximum-paths 8
graceful-restart
!
37
Border/Edge leaf Configuration
The configuration is similar to Leaf, but the local-as number is different. Both Edge-leafs are in the same AS. In addition, border-leaf may also have BGP
peering to WAN/DCI Edge nodes.
router bgp
local-as 4200007000
capability as4-enable
!
neighbor spine-group peer-group
neighbor spine-group remote-as 4200000000
neighbor spine-group description To spine
neighbor spine-group password <password>
neighbor spine-group bfd
!
!
address-family ipv4 unicast
network 10.61.1.1/32
maximum-paths 8
graceful-restart
!
Spine Configuration
This is applicable to all spines in the 3-stage fabric. Note that each Leaf or a MCT pair Leaf and Border-leaf pair are in separate autonomous systems.
The remote-as must be specified separately and can’t be configured under peer-groups.
• Configure the directly connected leafs' IP addresses in one peer group: leaf-group.
• Configure the directly connected edge-leafs' IPs into a peer group: edge-group.
• Enable MD5 authentication and BFD to all peer groups.
• Enable IPv4 Address-Family.
38
router bgp 4 Byte AS number of this device
local-as 4200000000 Enable 4 byte AS number capability
!
neighbor edge-group peer-group Configure a peer-group “edge-group” for
neighbor edge-group remote-as 4200000021 the directly connected links’ IPv4
neighbor edge-group password <password> addresses of the border-leafs.
neighbor edge-group bfd Enable MD5 authentication and BFD
!
neighbor 10.32.1.1 peer-group edge-group
neighbor 10.32.2.1 peer-group edge-group
!
neighbor leaf-group peer-group
Configure a peer-group “leaf-group”
neighbor leaf-group description To leaf/TOR
neighbor leaf-group password <password> Enable MD5 authentication and BFD
neighbor leaf-group bfd
!
neighbor 10.0.1.0 remote-as 4200000001
neighbor 10.0.1.0 peer-group leaf-group
neighbor 10.0.2.0 peer-group leaf-group Add the directly connected leafs’ IPv4
neighbor 10.0.3.0 remote-as 4200000003 addresses into leaf-group.
neighbor 10.0.3.0 peer-group leaf-group Each leaf is in a different AS, so it must
neighbor 10.0.4.0 remote-as 4200000004 be specified seperately and not with
neighbor 10.0.4.0 peer-group leaf-group peer-group
neighbor 10.0.6.0 peer-group leaf-group Enable IPv4 address-family
!
maximum-paths 8
graceful-restart
!
!
BGP EVPN and VxLAN Overlay

MTU
For the tenant workloads, the MTU is controlled by the global configuration commands shown in the Fabric Infrastructure Configuration section. The
global IPv4 MTU is set to 9100 bytes. As the overlay is built on IPv4, the tenant workloads cannot send packets – L2 bridged or IPv4/IPv6 routed
packets – of size greater than 9100 bytes after VxLAN encapsulation.
Overlay Gateway Configuration

Following are the steps involved in configuring the overlay gateway or VTEP on a leaf and border leaf.
• Create an overlay gateway, and assign it a name.
• Enable Layer 2 extension.
• Associate the loopback interface whose IPv4 address is used as the VTEP IP. (Loopback 2 interface)
• Map the VLANs to the VNI number. In this validated design, we're using the auto mapping of VLAN to a VNI. For instance, VLAN 101 is
mapped to VNI 101. (This simplified mapping option should work for most implementations unless there is a specific requirement to map the
server VLAN range to a specific VNI range in the VXLAN domain.)
39
overlay-gateway leaf1 Configure the overlay gateway with a name
!
!
ip interface Loopback 2 Enable Layer 2 extension
!
map vlan vni auto
!
activate Specify the loopback interface
! used as the VTEP IP
VLANs are auto mapped to VNIs

Activate the overlay gateway VLAN 101 is mapped to VxLAN VNI 101
BGP Overlay Configuration
Leaf and Border/Edge Leaf Configuration

This is applicable to all leafs and border-leafs. Note that on MCT pair leafs, both nodes in the pair must be independently configured with this
template.
• Configure l2vpn evpn AFI/SAFI.
• Activate the connected spines under the EVPN Address-Family.
• Enable the next-hop unchanged configuration to the peers. When EVPN routes are advertised into eBGP by a node, the next hop is set to its
peering address. This follows standard BGP behavior. The next hop should always point to the IP address of the VTEP that originated these
routes.
Address family l2vpn and SAFI evpn

router bgp
address-family l2vpn evpn
graceful-restart
neighbor spine-group encapsulation vxlan
neighbor spine-group activate Activate the evpn exchange with the spines peer-group
neighbor spine-group next-hop-unchanged
!
!
Do not change the next-hop. The next-hop is the
VTEP IP of the advertising Leaf and remains the
same in the fabric
Spine Configuration
This is applicable to the two spines in the 3-stage fabric to exchange EVPN routes with leafs and edge leafs.
• Enable the EVPN Address-Family.
• Since spines do not have either L2 or L3 tenants, these routes get filtered because of the route-targets. To avoid this spines must retain all
EVPN routes so they can be propagated to all leafs and border-leafs in the fabric.
• Activate both leaf and border-leaf peer-groups into the EVPN Address-Family.
• Enable the next-hop unchanged configuration to the peers. When EVPN routes are advertised into eBGP by a node, the next hop is set to its
peering address. This follows standard BGP behavior. The next hop should always point to the IP address of the VTEP that originated these
routes.
40
router bgp Address family l2vpn evpn
graceful-restart
! This enables Spines to advertise all evpn
retain route-target all routes it receives from its peers without
! any filtering
neighbor edge-group encapsulation vxlan
neighbor edge-group next-hop-unchanged
neighbor edge-group activate Activate evpn route exchange with edge-
! leafs peer-group
neighbor leaf-group encapsulation vxlan As mentioned in leaf config, set nexthop
neighbor leaf-group next-hop-unchanged unchanged
neighbor leaf-group activate
!
!
Activate the evpn exchange with the leaf
peer-group. Again, set nexthop unchanged
Tenant Provisioning
Tenant provisioning refers to the configuration on leafs to enable server/workload VLANs and network connectivity to tenant VRF contexts, and
mapping these VLANs and VRFs to the overlay control and forwarding planes to establish Layer 2 extension between the racks and Layer 3
multitenancy. This section is common both 3-stage and 5-stage Clos fabrics.
VLANs are referred to as MAC-VRF in BGP EVPN and they provide multi-tenancy at Layer-2. VLAN or a Broadcast-Domain (BD) extension between two
racks or two PoDs is Layer-2 extension and uses the MAC-VRF construct in EVPN.
VRFs provide multi-tenancy at the L3 level. For instance, a tenant may have multiple VLAN subnets, and extended across multiple Racks or PoDs and
requires Inter-VLAN routing. The L3 subnets of these VLANs are configured within a virtual routing context referred to as L3 VRF or simply VRF. Even
for deployments that do not require multi-tenancy at Layer-3, we recommend using one VRF for the workload connectivity if gateway services are
needed at the Fabric Leafs. The workload or server facing VLAN subnets must never be in the underlay or default VRF context of the Leafs.
Enable Conversational Learning of MAC, ARP and ND Entries

This is applicable to all leafs and border-leafs in the fabric for conservation of L2 and L3 host forwarding table space.
host-table aging-mode conversational
mac-address-table learning-mode conversational
Anycast Gateway MAC Configuration

Anycast gateway MAC configuration is applied to all leafs including border-leafs in the data center. (Border-leafs may host VE interfaces for the
VLAN segments extended over L2 networks or VPLS. However we do not recommend workloads directly attached to the border-leafs.) This is used
as the gateway MAC or router MAC for all server-facing subnets. This enables seamless workload move within and across the PoDs in the data
center. We recommend setting the U/L bit to 1 in the MAC address to indicate that it is a locally administered MAC address and not to conflict with
any real MAC addresses.
The MAC addresses must be different for IPv4 and IPv6, but the OUI portion (first three bytes) must be same.
ip anycast-gateway-mac 0201.0101.0101
ipv6 anycast-gateway-mac 0201.0102.0202
Tenant VRF
The underlay routing domain is in the default VRF of a leaf device. This provides the reachability and provisioning of tunnels or overlays to other VTEPs
in the network. VRF provides the multi-tenancy at layer-3. For server subnets or workloads, we recommend using a separate VRF. This is the separate
underlay and overlay routing domains. This also allows the use of an L3 VNI for symmetric IRB, host route visibility, and optimization in the forwarding
plane.
41
The following are the steps involved in tenant VRF configuration.
1. Assign a unique RD. Every tenant must have a unique RD value per leaf/ToR where it is provisioned. In the validated design, we are using
the following format: IPv4_Address:nn where
• IPv4_Address is the router ID of the VTEP.
• “nn” is a unique number for the tenant VRF. This value is re-used on other leafs as well where the same tenant is provisioned.
For example, vrf201 has the following RD values on leafs where it is provisioned.
– On leaf1: 10.1.1.11:101
– On leaf5: 10.1.2.11:101
– On border-leaf1: 10.123.4.1:201
2. Assign a unique L3 VE routing interface. This gets mapped to a L3 VNI used for symmetric routing in the tenant VRF.
3. Assign import and export route targets for IPv4 and IPv6 tenant routes.
In the configuration templates below, the following tenant profile is enabled on a leaf:
Configure the Tenant VRF Profile:
• Name: vrf101
• L3 Routing interface for L3 VNI: 3001
• IPv4 and IPv6 forwarding: enabled
• Route-target 101:101
• Server-facing VLAN 101 and L3 VE interface for first-hop routing
RD is route-distinguisher unique per tenant

vrf vrf101
rd 10.1.1.11:101
!
evpn irb ve 3001 L3 VE routing interface required for each
! tenant for symmetric routing
route-target export 101:101 evpn
route-target import 101:101 evpn
! Route target for export and import of tenant
address-family ipv6 unicast IPv4 and IPv6 routes. This is similar to
route-target export 101:101 evpn MPLS/VPNs
!
Assign a Layer 3 Interface for the L3 VNI of the Tenant VRF:
This is the routing interface for the Integrated Routing and Bridging (IRB) operation on the leaf. The bridge-domain is equivalent to an underlying VLAN
for the L3 interface.
42
Bridge-domain for the L3 VNI
bridge-domain 3001 Specify the Router interface for this Bridge-domain
router-interface ve 3001
!
interface Ve 3001 Router interface for the L3 VNI
vrf forwarding vrf101 Associate the interface to the tenant VRF
ipv6 address use-link-local-only
no shutdown
! Enable V6 forwarding on this interface
Server VLAN, and Subnet Configuration

• Create a Server-Facing VLAN
• Assign a VE (L3) Interface for the Server-Facing VLAN
• Associate the VE to the correct tenant VRF
• Assign IPv4 and IPv6 anycast gateways.
vlan 101
description L2 Tenant Vlan; Tenant VRF vrf101
router-interface Ve 101 Routing interface for the VLAN
suppress-nd
suppress-arp
! VLAN subnet belongs to the tenant vrf101
interface Ve 101
vrf forwarding vrf101
ip anycast-address 10.0.101.254/24 IPv4 anycast gw address. Same address on all
ipv6 anycast-address fdf8:10:0:65::254/96 Leafs where this vlan is extended
no shutdown
!
IPv6 anycast gw address. Same address on all
leafs where this VLAN is extended
Next step is to enable this VLAN on the server facing port of the Leaf.
Provision Server VLAN on Server-Facing Link

The server-facing or access links are on the leaf nodes. In the validated design,
• 10-G links are used for server-facing VLANs.
• These links are configured as Layer 2 trunks with VLANs associated.
• The MTU for these links is set to the default: 1500 bytes.
• Spanning tree is disabled. 1
1 If there are L2 switches or bridges between leaf and servers, spanning tree protocol must be enabled. If there is a possibility of enabling bridges
inadvertently under the leaf nodes, we recommend enabling spanning tree and configuring the server ports as edge ports.
POD1-Leaf3(conf-if-eth-0/34)# spanning-tree autoedge
43
interface Ethernet 0/34 Enabled as a trunk port
switchport Add the required VLANs to the trunk port
switchport mode trunk
switchport trunk allowed vlan add 101
switchport trunk tag native-vlan
spanning-tree shutdown
no shutdown
L2 and L3 extension into Overlay

Once the server-facing VLANs are created and mapped to VNI segments on the leaf, those VNI segments must be enabled into the control plane. As
was done for the tenant VRF, the VNI segments also require an RD (route distinguisher) and an RT (route target). This is also defined as the MAC-VRF
and enables learning remote MAC addresses when the same VLAN segment is extended to other leafs or VTEPs in the fabric.
The RD and RT configuration is set to auto in this design for simplicity and may be followed for most of the deployments. Advanced users may define a
different scheme of RD and RT. A user-defined RD/RT is not covered in this document.
For L3 symmetric IRB, the bridge-domain associated with the L3 VNI of the tenant VRF is added to the EVPN instance.
Route target to enable import and export of EVPN routes from

remote VTEPS. Route target is automatically generated.
evpn default
route-target both auto ignore-as
ignore-as keyword allows downloading routes from VTEPs in
rd auto
remote AS. (as in EBGP underlay where each leaf is in a separate
bridge-domain add 3001
vlan add 101 AS)
!
RD is set to Auto for simplified configuration
Enable the server vlan’s VNI. This enables L2 Enable the Bridge-domain used for L3 VNI of the tenant VRF.
extension of the workload VLAN. For any For additional tenant VRFs, add the respective L3 VNI’s bridge-
additional vlans, add the respective VNIs domains.
Advertise Tenant VLAN Subnets from the Leaf

IPv4
router bgp Activate layer3 IPv4 routes exchange from the
address-family ipv4 unicast vrf vrf101 tenant
redistribute connected
maximum-paths 8
! Advertise the connected subnets inside the tenant
! vrf – vlan subnets in this case
IPv6
router bgp Activate layer3 IPv4 routes exchange from the
address-family ipv6 unicast vrf vrf101 tenant
maximum-paths 8
! Advertise the connected subnets inside the tenant
! vrf – vlan subnets in this case
MCT Pair Leaf Configuration

MCT pair is the solution for redundant or dual ToR. Most of the configuration is similar to the individual or stand-alone ToR/Leaf. It requires a few
additional configuration steps when provisioning redundant connectivity using LAG or port-channel to the servers.
44
The configuration is broadly divided in two blocks.
• MCT cluster configuration shown in the section: Fabric Infrastructure Configuration > SLX-9140 MCT pair as Leaf
• MCT tenant VLAN, VRF and L2/L3 extension over VxLAN overlay.
The configuration of two switches in Dual-ToR MCT pair is shown side-by-side for comparison.
• Loopback1 interface has unique IP address on each node, this is used as the IP Router-ID for the node.
• Loopback2 interface has the same IP address on both nodes, this is used as VETP-IP under overlay-gateway. Refer to the section Fabric
Infrastructure Configuration > SLX-9140 MCT pair as Leaf
Conversational Learning
host-table aging-mode conversational host-table aging-mode conversational

mac-address-table learning-mode conversational mac-address-table learning-mode conversational
Anycast gateway
ip anycast-gateway-mac 0201.0101.0101 ip anycast-gateway-mac 0201.0101.0101

ipv6 anycast-gateway-mac 0201.0102.0202 ipv6 anycast-gateway-mac 0201.0102.0202
Overlay gateway and EVPN Instance
overlay-gateway mct-leaf overlay-gateway mct-leaf

map vni auto map vni auto
activate activate
! !
rd auto rd auto
! !
BGP EVPN for the Fabric
Enable peering with the spines.

neighbor spine-group encapsulation vxlan neighbor spine-group encapsulation vxlan
neighbor spine-group allowas-in 1 neighbor spine-group allowas-in 1
neighbor spine-group activate neighbor spine-group activate
! !
45
MC-LAG or Dual-Homed Server Port-Channel
This section shows the creation of dual-homed server connectivity using a MLAG. Here a server is connected to the dual ToR on a two port bundle or a
LAG. This LAG is deployed or activated in the MCT cluster. The MCT number identifies that this particular bundle is dual-homed between the two
switches in the MCT cluster.

description MCT lag member to Server Racks description MCT lag member to Server Racks
! !
description MCT LAG description MCT LAG
switchport switchport
switchport mode trunk-no-default-native switchport mode trunk-no-default-native
! !
client MCT 101 client MCT 101
client-interface Port-channel 101 client-interface Port-channel 101
deploy deploy
! !
Tenant VLAN and Layer2 Extension
vlan 101 vlan 101

description L2 Tenant vlan description L2 Tenant vlan
! !
switchport trunk allowed vlan add 101 switchport trunk allowed vlan add 101
! !
vlan add 101 vlan add 101
! !
46
Tenant VRF and Layer3 Extension
vrf vrf101 vrf vrf101

rd 10.1.1.11:101 rd 10.1.1.12:101
evpn irb ve 3001 evpn irb ve 3001
route-target export 101:101 evpn route-target export 101:101 evpn
route-target import 101:101 evpn route-target import 101:101 evpn
! !
! !
bridge-domain 3001 bridge-domain 3001
router-interface ve 3001 router-interface ve 3001
description l3vni for vrf101 description l3vni for vrf101
! !
vrf forwarding vrf101 vrf forwarding vrf101
ipv6 address use-link-local-only ipv6 address use-link-local-only
! !
bridge-domain add 3001 bridge-domain add 3001
! !
address-family ipv4 unicast vrf vrf101 address-family ipv4 unicast vrf vrf101
redistribute connected redistribute connected
maximum-paths 8 maximum-paths 8
! !
First Hop Routing for Tenant VLAN

vlan 101 vlan 101
suppress-nd suppress-nd
suppress-arp suppress-arp
! !
ip anycast-address 10.0.101.254/24 ip anycast-address 10.0.101.254/24
ipv6 anycast-address fdf8:10:0:65::254/96 ipv6 anycast-address fdf8:10:0:65::254/96
! !
5-Stage Fabric
This configuration is applicable to the model shown in Figure 16, where eBGP is used as the control protocol for underlay. A 5-stage
fabric includes multiple 3-stage PoDs interconnected by a set of Super-spines. This section includes the incremental configuration
required on top of a 3-stage fabric. The configuration templates shown for each PIN in a PoD and the Super-spines that can be used
as reference and replicated across devices in multiple PoDs.
The fabric links and router-id configurations for super-spines are similar to the configurations on Spines. Super-spines also do not act
as VTEPs and they don’t need a VTEP IP address. Refer to Fabric Infrastructure Configuration under 3-stage Fabric section.
In a 5-stage fabric, the Edge-leafs are connected to the Super-spines.
Key points to consider when building 5-stage Fabric:
• Super-spine tier is in one AS, i.e. every super-spine is configured with the same BGP AS number.
47
• Each spine in a POD is connected to every Super-spine with IPv4 Fabric links.
• Each spine in a POD will have EBGP peering with every Super-spine and exchange both underlay and overlay routes.
• Each Edge leaf is connected to every super-spine with IPv4 Fabric links.
• Edge leafs exchange IPv4 underlay and overlay routes with the super-spines.
Spine Configuration
This is applicable to the two spines designated to exchange only IPv4 routes with leafs and super-spines.
• Configure the directly connected leafs IP addresses in one peer group: leaf-group.
• Configure the directly connected super-spine IPs into another peer group: superspine-group.
• Enable MD5 authentication and BFD to all peers.
• Enable the IPv4 Address-Family. Both peer-groups are activated by default for this AFI.
• Enable the L2VPN/EVPN AFI and activate both leaf-group and superspine-group peer-groups.
48
router bgp
local-as 4200000000
!
neighbor leaf-group peer-group
neighbor leaf-group description To leaf/TOR
neighbor leaf-group password <password>
neighbor leaf-group bfd
!
!
neighbor superspine-group peer-group
neighbor superspine-group remote-as 4200000020
neighbor superspine-group password 2 <password>
neighbor superspine-group bfd
!
neighbor 10.0.51.0 peer-group super-spine-group
neighbor 10.0.52.0 peer-group super-spine-group
!
maximum-paths 8
graceful-restart
!
graceful-restart
retain route-target all
!
neighbor leaf-group encapsulation vxlan
neighbor leaf-group next-hop-unchanged
neighbor leaf-group activate
!
neighbor super-spine-group encapsulation vxlan
neighbor super-spine-group next-hop-unchanged
neighbor super-spine-group activate
!
!
Super-Spine Configuration
• Create a peer group for each PoD:
o pod1-spine-group — Add the directly connected neighbor addresses of all spines in PoD1 to this group.
o pod2-spine-group — Add the directly connected neighbor addresses of all spines in PoD2 to this group.
• Create a separate peer group for the edge leafs — edge-group. Add the directly connected neighbor addresses of edge leafs to this group.
• Enable IPv4 AFI. All peer-groups are activated under this AFI by default
• Enable L2VPN/EVPN AFI. Activate spine and edge groups.
49
router bgp
local-as 4200000010
!
neighbor edge-leaf peer-group
neighbor edge-leaf remote-as 4200007000
neighbor edge-leaf password <password>
neighbor edge-leaf bfd
!
neighbor 10.0.61.0 peer-group edge-leaf
neighbor 10.0.62.0 peer-group edge-leaf
!
neighbor pod1-spine-group peer-group
neighbor pod1-spine-group remote-as 4200000000
neighbor pod1-spine-group password <password>
neighbor pod1-spine-group bfd
!
neighbor 10.0.51.1 peer-group pod1-spine-group
!
neighbor pod2-spine-group peer-group
neighbor pod2-spine-group remote-as 4200002000
neighbor pod2-spine-group password <password>
neighbor pod2-spine-group bfd
!
!
maximum-paths 8
graceful-restart
!
graceful-restart
retain route-target all
neighbor pod2-spine-group encapsulation vxlan
neighbor pod2-spine-group next-hop-unchanged
neighbor pod2-spine-group activate
neighbor pod1-spine-group encapsulation vxlan
neighbor pod1-spine-group next-hop-unchanged
neighbor pod1-spine-group activate
neighbor edge-leaf encapsulation vxlan
neighbor edge-leaf next-hop-unchanged
neighbor edge-leaf activate
!
!
Edge-Leaf Configuration
The configuration of edge or border leafs is similar to that of leafs. They peer with the super-spines. They exchange both IPv4 routes EVPN routes with
super-spines.
• Configure another peer group super-spine -group. Add the super-spine addresses to this group. These super-spines exchange both IPv4 and
EVPN routes.
• Enable the IPv4 Address-Family, and advertise the VTEP IP address.
• Enable L2VPN/EVPN AFI-SAFI. Activate the peer group superspine-group.
50
• In addition, the border-leaf may be configured with peering to WAN Edge or DCI Edge devices.
router bgp
local-as 4200000061
!
neighbor super-spine-group peer-group
neighbor super-spine-group remote-as 4200000010
neighbor super-spine-group description To super spines
neighbor super-spine-group password <password>
neighbor super-spine-group bfd
!
neighbor 10.0.61.1 peer-group evpn-super-spine-group
neighbor 10.0.61.3 peer-group evpn-super-spine-group
!
network 10.61.1.1/32
maximum-paths 8
graceful-restart
!
graceful-restart
neighbor super-spine-group encapsulation vxlan
neighbor super-spine-group next-hop-unchanged
neighbor super-spine-group activate
!
51
Use Cases
In this section we illustrate the use cases by using sections of the validated design network topology as appropriate. This will help the reader to
further understand the deployment scenarios.
52
Simple 3-stage BGP VxLAN Based EVPN Fabric Illustration
This case is included for the purpose of illustrating a complete fabric by putting together all the building blocks. As shown in the diagram below, it
includes 2 spines and 3 workload racks. To illustrate various platforms that can be used as Leaf, we have chosen a MCT pair based of SLX 9140 and a
vLAG pair based of VDX-6740 switches. We also included a stand-alone SLX-9140 leaf.
Please note that the number of Spines depend on the oversubscription ratio desired in the fabric.
Figure 17 Illustration topology for a simple 3-stage fabric
SLX-9240
Eth 0/1-5 Spines
Eth 0/1-5
Eth 0/49-50 Fo 2/0/49-50

Eth 0/51-52 Eth 0/51-52 Fo 1/0/49-50
Leaf1-1 Leaf1-2 Leaf3 Leaf4-1 Leaf4-2

SLX-9140 0/49-50 SLX-9140 SLX-9140 VDX-6740 Ports 51-52 VDX-6740
0/1 0/1 0/1 0/1 0/1
Rack1: VTEP 10.1.1.1 Rack2: VTEP 10.1.3.1 Rack3: VTEP 10.1.6.1
53
Configuration
Configuration on the SLX-9240 Spines
Note: Spines require only the underlay configuration.
UNDERLAY CONFIGURATION
SLX: Spine1 Config SLX: Spine2 Config
switch-attributes host-name SLX-Spine1 switch-attributes host-name SLX-Spine2

! !
mtu 9216 mtu 9216
ip mtu 9100 ip mtu 9100
! !
description Link to SLX-9140 Leaf description Link to SLX-9140 Leaf
description Link to VDX Leaf description Link to VDX Leaf
description Link to VDX Leaf description Link to VDX Leaf
! !
local-as 4200000000 local-as 4200000000
capability as4-enable capability as4-enable
fast-external-fallover fast-external-fallover
bfd interval 300 min-rx 300 multiplier 3 bfd interval 300 min-rx 300 multiplier 3
neighbor leaf-group peer-group neighbor leaf-group peer-group
neighbor leaf-group description To leaf/TOR neighbor leaf-group description To leaf/TOR
neighbor leaf-group password password neighbor leaf-group password password
neighbor leaf-group bfd neighbor leaf-group bfd
neighbor 10.0.1.0 peer-group leaf-group neighbor 10.0.1.2 peer-group leaf-group
54
retain route-target all retain route-target all
neighbor leaf-group encapsulation vxlan neighbor leaf-group encapsulation vxlan
neighbor leaf-group next-hop-unchanged neighbor leaf-group next-hop-unchanged
neighbor leaf-group activate neighbor leaf-group activate
Configuration on the SLX-9140 Leaf MCT pair
SLX: MCT Leaf peer1 SLX: MCT Leaf peer2 SLX: non-MCT Leaf
switch-attributes host-name SLX-Leaf1-1 switch-attributes host-name SLX-Leaf1-2 switch-attributes host-name SLX-Leaf3
! Underlay configuration ! Underlay configuration ! Underlay configuration

interface Loopback 1 interface Loopback 1 interface Loopback 1
ip address 10.1.1.11/32 ip address 10.1.1.12/32 ip address 10.1.3.2/32
no shutdown no shutdown no shutdown
interface Loopback 2 interface Loopback 2 interface Loopback 2
ip router-id 10.1.1.11 ip router-id 10.1.1.12 ip router-id 10.1.3.2
mtu 9216 mtu 9216 mtu 9216
ipv6 mtu 9100 ipv6 mtu 9100 ipv6 mtu 9100
ip mtu 9100 ip mtu 9100 ip mtu 9100
host-table aging-mode conversational host-table aging-mode conversational host-table aging-mode conversational

host-table aging-time conversational 300 host-table aging-time conversational 300 host-table aging-time conversational 300
mac-address-table learning-mode conversational mac-address-table learning-mode conversational mac-address-table learning-mode conversational
mac-address-table aging-time conversational 300 mac-address-table aging-time conversational 300 mac-address-table aging-time conversational 300
mac-address-table aging-time 1800 mac-address-table aging-time 1800 mac-address-table aging-time 1800
! Fabric Interface config ! Fabric Interface config ! Fabric Interface config

interface Ethernet 0/51 interface Ethernet 0/51 interface Ethernet 0/49
description Link to Spines description Link to Spines description Link to Spines
description Link to Spines description Link to Spines description Link to Spines
! MCT configuration ! MCT configuration ! BGP configuration for underlay

node-id 1 node-id 2 router bgp
cluster management principal-priority 1 vlan 4090 local-as 4200000003
vlan 4090 router-interface Ve 4090 capability as4-enable
router-interface Ve 4090 description MCT control-vlan fast-external-fallover
description MCT control-vlan interface Ve 4090 bfd interval 300 min-rx 300 multiplier 3
interface Ve 4090 ip address 10.0.1.9/31 neighbor evpn-spine peer-group
ip address 10.0.1.8/31 no shutdown neighbor evpn-spine remote-as 4200000000
no shutdown neighbor evpn-spine description To spine
neighbor evpn-spine password password
!! ICL configuration !! ICL configuration neighbor evpn-spine bfd
interface Ethernet 0/49 interface Ethernet 0/49 neighbor 10.0.3.1 peer-group evpn-spine
description MCT ICL LAG member description MCT ICL LAG member neighbor 10.0.3.3 peer-group evpn-spine
channel-group 1 mode active type standard channel-group 1 mode active type standard address-family ipv4 unicast
no shutdown no shutdown network 10.1.3.1/32
interface Ethernet 0/50 interface Ethernet 0/50 maximum-paths 8
description MCT ICL LAG member description MCT ICL LAG member
55
description MCT ICL LAG description MCT ICL LAG
speed 40000 speed 40000
peer 10.0.1.9 peer 10.0.1.8
deploy deploy
rd auto rd auto
! BGP configuration for underlay and MCT ! BGP configuration for underlay and MCT
local-as 4200000001 local-as 4200000001
neighbor evpn-spine peer-group neighbor evpn-spine peer-group
neighbor evpn-spine remote-as 4200000000 neighbor evpn-spine remote-as 4200000000
neighbor evpn-spine description To spine neighbor evpn-spine description To spine
neighbor evpn-spine password password neighbor evpn-spine password password
neighbor evpn-spine bfd neighbor evpn-spine bfd
neighbor 10.0.1.1 peer-group evpn-spine neighbor 10.0.2.1 peer-group evpn-spine
network 10.1.1.1/32 network 10.1.1.1/32
OVERLAY CONFIGURATION
SLX: MCT Leaf peer1 SLX: MCT Leaf peer2 SLX: non-MCT Leaf
ip anycast-gateway-mac 0201.0101.0101 ip anycast-gateway-mac 0201.0101.0101 ip anycast-gateway-mac 0201.0101.0101

ipv6 anycast-gateway-mac 0201.0102.0202 ipv6 anycast-gateway-mac 0201.0102.0202 ipv6 anycast-gateway-mac 0201.0102.0202
overlay-gateway slx-mct-leaf overlay-gateway slx-mct-leaf overlay-gateway slx-leaf3
type layer2-extension type layer2-extension type layer2-extension
ip interface Loopback 2 ip interface Loopback 2 ip interface Loopback 2
map vni auto map vni auto map vni auto
activate activate activate
vlan 101 vlan 101 vlan 101

description L2 Tenant extended vlan description L2 Tenant extended vlan description L2 Tenant extended vlan
router-interface Ve 101 router-interface Ve 101 router-interface Ve 101
suppress-nd suppress-nd suppress-nd
suppress-arp suppress-arp suppress-arp
vlan 131 vlan 131 vlan 331
description L2 Tenant non-extended vlan description L2 Tenant non-extended vlan description L2 Tenant non-extended vlan
suppress-nd suppress-nd suppress-nd
suppress-arp suppress-arp suppress-arp
bridge-domain 3001 bridge-domain 3001 bridge-domain 3001

description l3vni for vrf101 description l3vni for vrf101 description l3vni for vrf101
vrf vrf101 vrf vrf101 vrf vrf101

56
rd 10.1.1.11:101 rd 10.1.1.12:101 rd 10.1.3.2:101
evpn irb ve 3001 evpn irb ve 3001 evpn irb ve 3001
address-family ipv4 unicast address-family ipv4 unicast address-family ipv4 unicast
route-target export 101:101 evpn route-target export 101:101 evpn route-target export 101:101 evpn
route-target import 101:101 evpn route-target import 101:101 evpn route-target import 101:101 evpn
address-family ipv6 unicast address-family ipv6 unicast address-family ipv6 unicast
route-target export 101:101 evpn route-target export 101:101 evpn route-target export 101:101 evpn
route-target import 101:101 evpn route-target import 101:101 evpn route-target import 101:101 evpn
evpn default evpn default evpn default

route-target both auto ignore-as route-target both auto ignore-as route-target both auto ignore-as
rd auto rd auto rd auto
bridge-domain add 3001 bridge-domain add 3001 bridge-domain add 3001
vlan add 101,131 vlan add 101,131 vlan add 101,331
! LAG towards Host ! LAG towards Host ! Ethernet Interface towards Host
description MCT lag member to Server Racks description MCT lag member to Server Racks switchport
channel-group 101 mode active type standard channel-group 101 mode active type standard switchport mode trunk-no-default-native
no shutdown no shutdown switchport trunk allowed vlan add 101,331
interface Port-channel 101 interface Port-channel 101 no shutdown
speed 10000 speed 10000 interface Ve 3001
switchport switchport vrf forwarding vrf101
switchport mode trunk-no-default-native switchport mode trunk-no-default-native ipv6 address use-link-local-only
switchport trunk allowed vlan add 101,131 switchport trunk allowed vlan add 101,131 no shutdown
no shutdown no shutdown interface Ve 101
cluster pod1-cluster 1 cluster pod1-cluster 1 vrf forwarding vrf101
client MCT 101 client MCT 101 ip anycast-address 10.0.101.254/24
client-interface Port-channel 101 client-interface Port-channel 101 ip arp learn-any
deploy deploy ipv6 anycast-address fdf8:10:0:65::254/96
no shutdown
interface Ve 3001 interface Ve 3001 interface Ve 331
vrf forwarding vrf101 vrf forwarding vrf101 vrf forwarding vrf101
ipv6 address use-link-local-only ipv6 address use-link-local-only ip anycast-address 10.1.75.254/24
no shutdown no shutdown ip arp learn-any
interface Ve 101 interface Ve 101 ipv6 anycast-address fdf8:10:0:14b::254/96
vrf forwarding vrf101 vrf forwarding vrf101 no shutdown
ip anycast-address 10.0.101.254/24 ip anycast-address 10.0.101.254/24 !
ip arp learn-any ip arp learn-any router bgp
ipv6 anycast-address fdf8:10:0:65::254/96 ipv6 anycast-address fdf8:10:0:65::254/96 address-family ipv4 unicast vrf vrf101
no shutdown no shutdown redistribute connected
interface Ve 131 interface Ve 131 maximum-paths 8
vrf forwarding vrf101 vrf forwarding vrf101 address-family ipv6 unicast vrf vrf101
ip anycast-address 10.0.131.254/24 ip anycast-address 10.0.131.254/24 redistribute connected
ip arp learn-any ip arp learn-any maximum-paths 8
ipv6 anycast-address fdf8:10:0:83::254/96 ipv6 anycast-address fdf8:10:0:83::254/96 address-family l2vpn evpn
no shutdown no shutdown neighbor evpn-spine encapsulation vxlan
! ! neighbor evpn-spine activate
router bgp router bgp neighbor evpn-spine next-hop-unchanged
maximum-paths 8 maximum-paths 8 !!! User must clear BGP sessions from exec-prompt
address-family ipv6 unicast vrf vrf101 address-family ipv6 unicast vrf vrf101 !! “clear ip bgp neighbor all”
neighbor evpn-spine encapsulation vxlan neighbor evpn-spine encapsulation vxlan
neighbor evpn-spine activate neighbor evpn-spine activate
neighbor evpn-spine enable-peer-as-check neighbor evpn-spine enable-peer-as-check
!!! User must clear BGP sessions from exec-prompt !!! User must clear BGP sessions from exec-prompt
!! “clear ip bgp neighbor all” !! “clear ip bgp neighbor all”
57
Configuration on the VDX-6740 Leaf vLAG pair
VDX : vLAG Node1 VDX : vLAG Node2
!! Configuring vcs-id and rbridge-id is not a requirement if both vLAG peers !! Configuring vcs-id and rbridge-id is not a requirement if both vLAG peers
!! are pre-configured to be in the same vcs fabric. !! are pre-configured to be in the same vcs fabric.
!! Box would require reboot after this command. !! Box would require reboot after this command.
vcs vcsid 1 rbridge-id 1 logical-chassis enable vcs vcsid 1 rbridge-id 2 logical-chassis enable
!! After VCS fabric is up, its expected that the configuration are performed from !! After VCS fabric is up, its expected that the configuration are
the primary node. !! performed from the primary node.
!! Underlay Configuration !! Underlay Configuration

switch-attributes 1 host-name VDX-Leaf4-1 switch-attributes 2 host-name VDX-Leaf4-2
mtu 9216 mtu 9216
ipv6 mtu 9100 ipv6 mtu 9100
! !
mac-address-table learning-mode conversational mac-address-table learning-mode conversational
mac-address-table aging-time conversational 300 mac-address-table aging-time conversational 300
mac-address-table aging-time 1800 mac-address-table aging-time 1800
! !
rbridge-id 1 rbridge-id 2
host-table aging-time conversational 300 host-table aging-time conversational 300
!! Link to spines !! Link to spines

interface FortyGigabitEthernet 1/0/49 interface FortyGigabitEthernet 2/0/49
description Link to Spines description Link to Spines
no fabric isl enable no fabric isl enable
no fabric trunk enable no fabric trunk enable
no ip proxy-arp no ip proxy-arp
description Link to Spines description Link to Spines
no ip proxy-arp no ip proxy-arp
!! ISL trunks between vLAG switches !! ISL trunks between vLAG switches
description vLAG ISL description vLAG ISL
fabric isl enable fabric isl enable
fabric trunk enable fabric trunk enable
description vLAG ISL description vLAG ISL
58
!! BGP underlay config !! BGP underlay config
local-as 4200000006 local-as 4200000006
neighbor evpn-spine peer-group neighbor evpn-spine peer-group
neighbor evpn-spine remote-as 4200000000 neighbor evpn-spine remote-as 4200000000
neighbor evpn-spine password password neighbor evpn-spine password password
neighbor evpn-spine bfd neighbor evpn-spine bfd
network 10.1.6.1/32 network 10.1.6.1/32
OVERLAY CONFIGURATION
VDX : vLAG Node1 VDX : vLAG Node2
overlay-gateway vdx-leaf-pair overlay-gateway vdx-leaf-pair

attach rbridge-id add 1-2 attach rbridge-id add 1-2
map vlan vni auto map vlan vni auto
activate activate
interface Vlan 101 interface Vlan 101

description L2 Tenant extended vlan description L2 Tenant extended vlan
description L2 Tenant non-extended vlan description L2 Tenant non-extended vlan

description l3vni for vrf101 description l3vni for vrf101
rd 10.1.6.11:1 rd 10.1.6.12:1
vni 7097 vni 7097
evpn-instance pod1-vdx evpn-instance pod1-vdx
rd auto rd auto
vni add 101,431 vni add 101,431
! LAG towards Host ! LAG towards Host

switchport mode trunk switchport mode trunk
59
no switchport trunk tag native-vlan no switchport trunk tag native-vlan
switchport trunk allowed vlan add 101,431 switchport trunk allowed vlan add 101,431
interface TenGigabitEthernet 1/0/1 interface TenGigabitEthernet 2/0/1
description MCT lag member to Server Racks description MCT lag member to Server Racks
shutdown shutdown
ip arp learn-any ip arp learn-any
ip arp-aging-timeout 25 ip arp-aging-timeout 25
ipv6 anycast-address fdf8:10:0:1af::254/96 ipv6 anycast-address fdf8:10:0:1af::254/96
neighbor evpn-spine activate neighbor evpn-spine activate
neighbor evpn-spine allowas-in 1 neighbor evpn-spine allowas-in 1
neighbor evpn-spine enable-peer-as-check neighbor evpn-spine enable-peer-as-check
!!! User must clear BGP sessions from exec-prompt !!! User must clear BGP sessions from exec-prompt
!! “clear ip bgp neighbor all” !! “clear ip bgp neighbor all”
60
Verification
Verification after Underlay & Spine configuration
SLX-Leaf1-1# show ip bgp summary
BGP4 Summary
Router ID: 10.1.1.11 Local AS Number: 4200000001
Confederation Identifier: not configured
Confederation Peers:
Maximum Number of IP ECMP Paths Supported for Load Sharing: 8
Number of Neighbors Configured: 2, UP: 2
Number of Routes Installed: 3, Uses 324 bytes
Number of Routes Advertising to All Neighbors: 3 (2 entries), Uses 104 bytes
Number of Attribute Entries Installed: 3, Uses 342 bytes BGP session to spines should be in
Neighbor Address AS# State Time Rt:Accepted Filtered Sent ToSend established state
10.0.1.1 4200000000 ESTAB 0h10m44s 1 0 1 0
10.0.1.3 4200000000 ESTAB 0h10m44s 1 0 2 0
SLX-Leaf1-1#
SLX-Leaf1-1# show bgp evpn summary
BGP4 Summary
Number of Routes Installed: 0
Number of Routes Advertising to All Neighbors: 0 (0 entries)
Number of Attribute Entries Installed: 0
Neighbor Address AS# State Time Rt:Accepted Filtered Sent ToSend Cluster client BGP session should
10.0.1.9 4200000001 ESTAB 0h10m 9s 0 0 0 0 be UP
SLX-Leaf1-1#
SLX-Leaf1-1# show vlan 4090
VLAN Name State Ports Classification
(R)-RSPAN (u)-Untagged
(t)-Tagged
================ =============== ========================== =============== ====================
4090 VLAN4090 ACTIVE Po 1(t)
SLX-Leaf1-1# Control-vlan 4090 should be

SLX-Leaf1-1# show port-channel summary Active & ICL port-channel should
be UP.
Flags: D - Down P - Up in port-channel (members)
U - Up (port-channel) * - Primary link in port-channel
S - Switched I - Insight Enabled Cluster-management session
M - Not in use. Min-links not met
===== =============== ========== ===============
should be active.
Group Port-channel Protocol Member ports
===== =============== ========== ===============
1 Po 1 (SU) LACP Eth 0/49 (P)
Eth 0/50 (P)
SLX-Leaf1-1#
SLX-Leaf1-1# show cluster management
Total Number of Nodes in Cluster : 2
Node-Id Switch MAC IP Address Status

----------------------------------------------------------------------
1 >60:9C:9F:B0:F5:00* 10.0.1.8 Co-ordinator
2 60:9C:9F:B0:D8:00 10.0.1.9 Connected
SLX-Leaf1-1#
SLX-Leaf1-1# show ip route bgp
IP Routing Table for VRF "default-vrf"
Total number of IP routes: 10
'*' denotes best ucast next-hop
'[x/y]' denotes [preference/metric]
10.1.3.1/32 BGP should install ECMP path

*via 10.0.1.1, Eth 0/51, [20/0], 0m36s, eBgp, tag 0 towards remote VTEP ip address
*via 10.0.1.3,
10.1.6.1/32
Eth 0/52, [20/0], 0m36s, eBgp, tag 0
on Leaf3 & Leaf4
*via 10.0.1.1, Eth 0/51, [20/0], 12m5s, eBgp, tag 0
SLX-Leaf1-1#
SLX-Leaf1-1#
61
SLX-Leaf3# show ip route bgp
IP Routing Table for VRF "default-vrf"
'*' denotes best ucast next-hop BGP route o/p for remote vTEP ip
addresses from Leaf3 & Leaf4
10.1.1.1/32
10.1.6.1/32
*via 10.0.3.1, Eth 0/49, [20/0], 1h11m, eBgp, tag 0
*via 10.0.3.3, Eth 0/50, [20/0], 1h11m, eBgp, tag 0
SLX-Leaf3#
VDX-Leaf4-1# show ip route bgp

Type Codes - B:BGP D:Connected O:OSPF S:Static U:Unnumbered +:Leaked route; Cost - Dist/Metric
BGP Codes - i:iBGP e:eBGP
OSPF Codes - i:Inter Area 1:External Type 1 2:External Type 2 s:Sham Link
Destination Gateway Port Cost Type Uptime
10.1.1.1/32 10.0.6.1 Fo 1/0/49 20/0 Be 30m20s
10.0.6.3 Fo 1/0/50 20/0 Be 30m20s
10.1.3.1/32 10.0.6.1 Fo 1/0/49 20/0 Be 18m51s
10.0.6.3 Fo 1/0/50 20/0 Be 18m51s
VDX-Leaf4-1#
VDX-Leaf4-2# show ip route bgp

Type Codes - B:BGP D:Connected O:OSPF S:Static U:Unnumbered +:Leaked route; Cost - Dist/Metric
BGP Codes - i:iBGP e:eBGP
OSPF Codes - i:Inter Area 1:External Type 1 2:External Type 2 s:Sham Link
Destination Gateway Port Cost Type Uptime
10.1.1.1/32 10.0.7.1 Fo 2/0/49 20/0 Be 30m10s
10.0.7.3 Fo 2/0/50 20/0 Be 30m10s
10.1.3.1/32 10.0.7.1 Fo 2/0/49 20/0 Be 18m41s
10.0.7.3 Fo 2/0/50 20/0 Be 18m41s
VDX-Leaf4-2#
Verification after Overlay configuration

SLX-Leaf1-1# show overlay-gateway
Overlay Gateway "slx-mct-leaf", ID 1, node-ids 1-2
Type Layer2-Extension, Tunnel mode VXLAN
IP address 10.1.1.1 ( Loopback 2 ), Vrf default-vrf
Admin state up
Number of tunnels 2
Packet count: RX 50 TX 6250
Byte count : RX 7652 TX 803052
SLX-Leaf1-1#
SLX-Leaf1-1#
SLX-Leaf1-1# show bgp evpn summ
BGP4 Summary Verify overlay-gateway is UP on all leafs.
Ensure BGP neighbors are establieshed
If things are in order VxLAN tunnels to
Number of Routes Advertising to All Neighbors: 81 (57 entries), Uses 2964 bytes neighbors should be up
Number of Attribute Entries Installed: 112, Uses 12768 bytes Execute these on all Leafs.
Neighbor Address AS# State Time Rt:Accepted Filtered Sent ToSend
10.0.1.1 4200000000 ESTAB 4h44m43s 31 24 24 0
10.0.1.3 4200000000 ESTAB 4h44m43s 31 24 24 0
10.0.1.9 4200000001 ESTAB 6h 4m27s 30 0 33 0
SLX-Leaf1-1#
SLX-Leaf1-1# show tunnel br
Tunnel 61441, mode VXLAN, node-ids 1-2
Admin state up, Oper state up
Source IP 10.1.1.1, Vrf default-vrf
Destination IP 10.1.3.1

Admin state up, Oper state up
62
SLX-Leaf1-1# show cluster
Cluster pod1-cluster 1
================
Cluster State: Deployed Cluster should be UP on MCT leaf.
Client Isolation Mode: Loose
DF Hold Time: 3 Cluster clients shoud be UP
Configured Member Vlan Range: 101,131
Active Member Vlan Range: 101,131
Cluster Control Vl
an: 4090
Configured Member BD Range: 3001
Active Member BD Range: 3001
No. of Peers: 1
No. of Clients: 3
Peer Info:
==========
Peer IP: 10.0.1.9, State: Up
Peer Interface: Port-channel 1
ICL Tunnel Type: NSH, State: Up
Client Info:
============
Name Id ESI Interface Local/Remote State
---- -- --- --------- ------------------
MCT 101 0:0:0:0:0:0:0:1:0:65 Port-channel 101 Up / Up
tu61441 2064 0:0:a:1:3:1:a:1:1:1 Tunnel-61441 Up / Up
tu61442 2065 0:0:a:1:6:1:a:1:1:1 Tunnel-61442 Up / Up
SLX-Leaf1-1#
SLX-Leaf1-1# show vlan brief
Total Number of VLANs configured : 4
(t)-Tagged
================ =============== ========================== =============== ====================
1 default INACTIVE(no member port)
Vlan 101 is an extended L2 tenant,
101 VLAN0101 ACTIVE Po 1(t) hence it shows the VTEP end
Po 101(t)
Tu 61441(t) vni 101 points where the vlan is extended
Tu 61442(t) vni 101

Vlan 131 is present only on Leaf1
Po 101(t) & hence doesn’t have tunnel
membership.
SLX-Leaf1-1# BD 3001 is for L3VNI & the

SLX-Leaf1-1# show bridge-domain 3001
Bridge-domain 3001 vrf/L3VNI is present on two other
------------------------------- VTEP endpoints
Bridge-domain Type: MP
Description: l3vni for vrf101
Number of configured end-points: 3 , Number of Active end-points: 3
VE id: 3001, if-indx: 1207962553
VNI: 7097, Tunnels: 2(2 up)
Tunnels: tu61441.7097 tu61442.7097
VLAN: N/A, Tagged ports: 1(1 up), Un-tagged ports: 0 (0 up)
Tagged Ports: po1.7609
Un-tagged Ports:
SLX-Leaf1-1#
63
SLX-Leaf1-1# show ip arp suppression-cache
Flags: L - Locally Learnt Adjacency
R - Remote Learnt Adjacency
RS - Remote Static Adjacency
Vlan/Bd IP Mac Interface Age Flags
---------------------------------------------------------------------------------------------------
0101 (V) 10.0.101.3 50eb.1a95.6bf7 Po 101 00:23:04 L
0101 (V) 10.0.101.101 0011.9400.044d Po 101 00:00:41 L
0101 (V) 10.0.101.102 0011.9400.044e Po 101 00:00:39 L
0101 (V) 10.0.101.201 0010.9400.1b96 Tu 61441 (10.1.3.1) Never R
0101 (V) 10.0.101.202 0010.9400.1b97 Tu 61441 (10.1.3.1) Never R
0101 (V) 10.0.101.211 0010.9400.2046 Tu 61442 (10.1.6.1) Never R
0101 (V) 10.0.101.212 0010.9400.2047 Tu 61442 (10.1.6.1) Never R
0131 (V) 10.0.131.101 0011.9400.04e3 Po 101 00:00:41 L
0131 (V) 10.0.131.102 0011.9400.04e4 Po 101 00:00:39 L
SLX-Leaf1-1#
SLX-Leaf1-1# show ipv6 nd suppression-cache
Vlan/Bd IP Mac Interface Age
Flags
-------------------------------------------------------------------------------------------------------------------------
0101 (V) fdf8:10:0:65::101 0011.9400.044d Po 101 00:15:23 L
0101 (V) fdf8:10:0:65::102 0011.9400.044e Po 101 00:15:21 L
0101 (V) fdf8:10:0:65::201 0010.9400.1b96 Tu 61441 (10.1.3.1) Never R
0101 (V) fdf8:10:0:65::202 0010.9400.1b97 Tu 61441 (10.1.3.1) Never R
0101 (V) fdf8:10:0:65::211 0010.9400.2046 Tu 61442 (10.1.6.1) Never R
0101 (V) fdf8:10:0:65::212 0010.9400.2047 Tu 61442 (10.1.6.1) Never R
...
SLX-Leaf1-1#
Host IPv6 ND entries learnt locally Host ARP entries learnt locally and
and over BGP from remote VTEPs over BGP from remote VTEPs are
are shown here. shown here.
SLX-Leaf1-1# show mac-address-table bridge-domain 3001

Type Code - CL:Cluster Local MAC CCL:Cluster Client Local MAC
CR:Cluster Remote MAC CCR:Cluster Client Remote MAC
BDId Mac-address Type State Ports/LIF/PW
3001 (B) 50eb.1a94.9953 EVPN-Static Active Tu 61442 (10.1.6.1)
3001 (B) 50eb.1a95.7ec3 EVPN-Static Active Tu 61442 (10.1.6.1)
3001 (B) 609c.9fb0.d801 CR Active Po 1
3001 (B) 609c.9fb1.5601 EVPN-Static Active Tu 61441 (10.1.3.1) BGP route o/p for non-
Total MAC addresses : 4 extended vlans are shown
here.
SLX-Leaf1-1# show ip route bgp vrf vrf101
IP Routing Table for VRF "vrf101"
Total number of IP routes: 6 Network routes for non-
'*' denotes best ucast next-hop extended or remote vlans
are installed as EVPN type-
10.1.75.0/24 5 routes in routing-table

*via 10.1.3.1%default-vrf, Ve 3001, [20/0], 4h29m, eBgp, tag 0, (VNI 7097, GW MAC 609c.9fb1.5601, Tu 61441)
10.1.75.201/32
*via 10.1.3.1%default-vrf, Ve 3001, [20/0], 7m55s, eBgp, tag 0, (VNI 7097, GW MAC 609c.9fb1.5601, Tu 61441) VRF BD/L3VNI shows the
10.1.75.202/32 reachability to remote
*via 10.1.3.1%default-vrf, Ve 3001, [20/0], 7m55s, eBgp, tag 0, (VNI 7097, GW MAC 609c.9fb1.5601, Tu 61441)
10.1.175.0/24 VTEPs and their router-
*via 10.1.6.1%default-vrf, Ve 3001, [20/0], 4h31m, eBgp, tag 0, (VNI 7097, GW MAC 50eb.1a94.9953, Tu 61442) MAC addresses.
*via 10.1.6.1%default-vrf, Ve 3001, [20/0], 4h31m, eBgp, tag 0, (VNI 7097, GW MAC 50eb.1a95.7ec3, Tu 61442)
10.1.175.101/32
*via 10.1.6.1%default-vrf, Ve 3001, [20/0], 0m3s, eBgp, tag 0, (VNI 7097, GW MAC 50eb.1a95.7ec3, Tu 61442)
10.1.175.102/32
*via 10.1.6.1%default-vrf, Ve 3001, [20/0], 0m3s, eBgp, tag 0, (VNI 7097, GW MAC 50eb.1a95.7ec3, Tu 61442)
SLX-Leaf1-1# show ipv6 route bgp vrf vrf101

IPv6 Routing Table for VRF "vrf101"
Total number of IPv6 routes: 7
fdf8:10:0:14b::/96
*via 10.1.3.1%default-vrf, Ve 3001, [20/0], 4h47m, eBgp, tag 0, (VNI 7097, GW MAC 609c.9fb1.5601, Tu 61441)
fdf8:10:0:1af::/96
*via 10.1.6.1%default-vrf, Ve 3001, [20/0], 4h49m, eBgp, tag 0, (VNI 7097, GW MAC 50eb.1a94.9953, Tu 61442)
*via 10.1.6.1%default-vrf, Ve 3001, [20/0], 4h49m, eBgp, tag 0, (VNI 7097, GW MAC 50eb.1a95.7ec3, Tu 61442)
#
64
L2 and L3 Extension between Racks
Figure 18 shows a section of the topology to illustrate the following with configuration and verification. Two racks are shown in the diagram.
• Rack1 has a redundant MCT ToR, leaf1-1 and leaf1-2, referred to as leaf1 collectively.
• Rack3 has an individual ToR, Leaf3.
• A tenant VRF vrf1 is provisioned on both racks.
• VLAN to VNI mapping is set to auto.
• The tenant has two server VLANs 101 and 131 mapped to VNIs 101 and 131 respectively.
• Tenant VRF is provisioned with a L3 VNI for routing between VLANs. This is also auto mapped based on the Bridge-domain identifier. (For BD,
VNI = 4096 + BD-Number)
• Server VLAN 101 is extended (L2 extension) between these two racks. VLAN/VNI 101 is provisioned on both racks, and there are hosts on
these racks.
• Server VLAN 131 is a VLAN provisioned on Rack1 only, but it belongs to the same tenant. Routing between VNI 101 and 131 is required within
this tenant both in the same rack and across the racks (L3 extension).
• This example also illustrates the symmetric and asymmetric routing operation.
The configuration on leafs is identical on each leaf except for the VTEP IP, router ID, and RD configurations. The vLAG pair is represented with one
VTEP IP address. The use of anycast gateway addresses for the server-facing VLAN interfaces simplifies the configuration drastically.
65
Figure 18 L2/L3 Extension between Racks
Spine Layer
Bridge-Domain 3001
L3 VNI 7097
Leaf1-1 10.1.1.1 Leaf1-2 10.1.1.1 Leaf3 10.1.3.1
Po 104 Po 101
VLAN 101/VNI 101

10.0.101.201/0010.9400.1b96
VLAN 131/VNI 131 VLAN 101/VNI 101 10.0.101.202/0010.9400.1b97
10.0.131.101/0011.9400.04e3 10.0.101.101/0011.9400.044d Tenant vrf101
10.0.131.102/0011.9400.04e4 10.0.101.102/0011.9400.0443
Tenant vrf101 Tenant vrf101
66
Configuration
Check the MCT configuration on Leaf1
In SLX-OS the MCT cluster configuration are done independently while the logical vTEP configuration is done only on the principal node of the cluster.
For the MCT pair, Leaf1-1 is the primary node for cluster management. The cluster configuration are done on both Leaf1-1 & Leaf1-2 but for cluster services like
Logical vTEP the EVPN overlay-gateway configuration is done on the principal node.
Leaf3 is an individual ToR and won’t require the cluster configuration.
POD1-Leaf1-1# show cluster management

Total Number of Nodes in Cluster : 2
Node-Id Switch MAC IP Address Status

----------------------------------------------------------------------
1 >60:9C:9F:B0:F5:00* 10.0.1.8 Co-ordinator
2 60:9C:9F:B0:D8:00 10.0.1.9 Connected
POD1-Leaf1-1#
POD1-Leaf1-1# show cluster 1

Cluster pod1-cluster 1
================
Cluster State: Deployed
DF Hold Time: 3
Configured Member Vlan Range: 101,131
Active Member Vlan Range: 101,131
Cluster Control Vlan: 4090
Configured Member BD Range: 3001
Active Member BD Range: 3001
No. of Peers: 1
No. of Clients: 2
Peer Info:
==========
Peer Interface: Port-channel 1
ICL Tunnel Type: NSH, State: Up
Client Info:
============
---- -- --- --------- ------------------
Configuration on the Leaf1 MCT Pair

The configuration is shown in three parts for clarity. Common configuration such as port channel and VLANs are shown in one block. The tenant, Layer
3 interfaces, and BGP-EVPN configuration is shown in the second block. The common overlay-gateway configuration is shown in the third block.
The configuration is pretty much the same except for the router ID and RD of the tenant VRF. This makes it easier to automate the provisioning on
various nodes.
Tenant and L2 Extension Between Racks in a 3-Stage Clos Fabric
!MCT pair
vlan 101
description VLAN 101, VNI 101, Tenant vrf101
router-interface Ve 101
!
vlan 131
!
bridge-domain 3001
description VLAN 3001, L3 VNI 7097, Tenant vrf101
!
interface Port-channel 101
!
!
68
! !
! !
! !
rd 10.1.1.11:101 rd 10.1.1.12:101
! !
! !
rd auto rd auto
duplicate-mac-timer 5 max-count 3 duplicate-mac-timer 5 max-count 3
vlan add 101,131 vlan add 101,131
! !
! !
! !
! !
! !
! !
! !
Please note that overlay-gateway configuration is applied from the primary node on the two-node MCT pair.
overlay-gateway mct-leaf
map vni auto
activate
!
67
Configuration on Leaf3
vlan 101
!
bridge-domain 3001
description VLAN 3001, L3 VNI 7097, Tenant vrf101
!
!
no shutdown
!
no shutdown
!
!
vrf vrf101
rd 10.1.3.2:101
evpn irb ve 3001
!
!
evpn default
rd auto
duplicate-mac-timer 5 max-count 3
bridge-domain add 3001
vlan add 101
!
router bgp
address-family ipv4 unicast vrf vrf101
maximum-paths 8
!
maximum-paths 8
!
!
interface Ve 101
ip anycast-address 10.0.101.254/24
ipv6 anycast-address fdf8:10:0:65::254/96
no shutdown
!
interface Ve 3001
no shutdown
!
overlay-gateway leaf3
map vni auto
activate
!
68
Verification
Verify VLAN Extension between the Racks
Check the L2 extended VLAN on each node. This should show the local L2 trunk ports and also the tunnels to all remote VTEPs where the same VLAN
segment is extended.
In the following output from the Leaf1 MCT peer, there are five tunnels for VLAN 101, which indicates that the same VLAN/VNI segment is
provisioned on five other VTEPs or ToRs. Note that one of the tunnels, Tu 61446, is destined to Leaf3. Also note that there are four underlay next
hops to reach this tunnel destination in the fabric.
POD1-Leaf1-1# show vlan 101

(t)-Tagged
================ =============== ========================== =============== ====================
Po 101(t)
Tu 61441(t) vni 101
Tu 61443(t) vni 101 List of local server facing ports
Tu 61444(t) vni 101 and tunnels to each VTEP
Tu 61445(t) vni 101
Tu 61446(t) vni 101 where the VLAN is extended
POD1-Leaf1-1# show tunnel 61446
Ifindex 0x7c00f006, Admin state up, Oper state up Tu 61446 is destined to leaf3’s
vtep-ip 10.1.3.1
Overlay gateway "mct-leaf", ID 1
Configuration source BGP-EVPN
MAC learning BGP-EVPN
Active next hops on node 1:
IP: 10.0.1.5, Vrf: default-vrf
Egress L3 port: Eth 0/53, Outer SMAC: 609c.9fb0.f539
Underlay IP next-hops from
Outer DMAC: 609c.9fb0.ac05 each MCT peer to reach the
Egress L2 Port: Eth 0/53, Outer ctag: 0, stag:0, Egress mode: Local remote VTEP leaf3. This shows
IP: 10.0.1.1, Vrf: default-vrf 4 paths as there are 4 spine
Egress L3 port: Eth 0/51, Outer SMAC: 609c.9fb0.f537 links
Outer DMAC: 609c.9fb0.7205
Egress L2 Port: Eth 0/51, Outer ctag: 0, stag:0, Egress mode: Local


Egress L3 port: Eth 0/54, Outer SMAC: 609c.9fb0.f53a

Egress L3 port: Eth 0/53, Outer SMAC: 609c.9fb0.d839
Outer DMAC: 609c.9fb0.ac06



Egress L3 port: Eth 0/54, Outer SMAC: 609c.9fb0.d83a

Byte count : RX 2150260 TX 5673182
POD1-Leaf1-1#
69
In the following output shown from Leaf3, Tunnel 61441 is destined to the vLAG Leaf1 pair's VTEP IP: 10.1.1.1.
VLAN member ports and tunnels
POD1-Leaf3# show vlan 101

(t)-Tagged
================ =============== ========================== =============== ====================
101 VLAN0101 ACTIVE Eth 0/34(t) List of local member ports and
Tu 61441(t)
Tu 61443(t)
vni
vni
101
101
tunnels to each VTEP where
Tu 61444(t) vni 101 the vlan is extended
Tu 61445(t) vni 101
Tu 61446(t) vni 101
POD1-Leaf3# show tunnel 61441 Tu 61441 is destined to leaf5’s

Tunnel 61441, mode VXLAN, node-ids 1 vtep-ip 10.121.1.1
Ifindex 0x7c00f001, Admin state up, Oper state up
Overlay gateway "pod1-leaf3", ID 1
IP: 10.0.3.5, Vrf: default-vrf Underlay IP next-hops from
Egress L3 port: Eth 0/51, Outer SMAC: 609c.9fb1.5637
Outer DMAC: 609c.9fb0.ac07 Leaf3 to reach the remote vtep
Egress L2 Port: Eth 0/51, Outer ctag: 0, stag:0, Egress mode: Local on MCT pair leaf1-1/1-2. This
IP: 10.0.3.1, Vrf: default-vrf shows 4 paths as there are 4
Egress L3 port: Eth 0/49, Outer SMAC: 609c.9fb1.5635 spine links



Byte count : RX 6832086 TX 2073558
POD1-Leaf3#
VLAN Layer 3 Interface State on the MCT Pair
POD1-Leaf1-1# show ip int ve 101

Ve 101 is up protocol is up
Vlan is 101
Hardware is Virtual Ethernet, address is 609c.9fb0.f501
Current address is 609c.9fb0.f501
Interface index (ifindex) is 1207959653
Primary Internet Address is 10.0.101.254/24 broadcast is 10.0.101.255
IP MTU is 9000
...
Vrf : vrf101
Vlan is 131
IP MTU is 9000
...
Vrf : vrf101
Bridge domain is 3001
IP unassigned
IP MTU is 9000
...
Vrf : vrf101
70
Vlan is 101
IP MTU is 9000
...
Vrf : vrf101
Vlan is 131
IP MTU is 9000
...
Vrf : vrf101
IP unassigned
IP MTU is 9000
...
Vrf : vrf101
VLAN Layer 3 Interface State on the Leaf3 ToR
POD1-Leaf3# show ip int ve 101

Vlan is 101
Hardware is Virtual Ethernet, address is 609c.9fb1.5601
Current address is 609c.9fb1.5601
IP MTU is 9000
...
Vrf : vrf101
IP unassigned
IP MTU is 9000
...
Vrf : vrf101
71
Local Host Entries on Each Leaf
Depending on the port-channel hashing on server-facing links, the ARP entries may be learned on any of the nodes in the MCT pair. Make sure that all
host entries are learned collectively in the MCT pair.
POD1-Leaf1-1# show arp ve 101 vrf vrf101

Entries in VRF vrf101 : 11
Address Mac-address Interface MacResolved Age Type
--------------------------------------------------------------------------------
...
10.0.101.101 0011.9400.044d Ve 101 yes 00:06:43 Dynamic
10.0.101.102 0011.9400.044e Ve 101 yes 00:06:43 Dynamic
...
POD1-Leaf1-2# show ip arp suppression-cache vlan 101
---------------------------------------------------------------------------------------------------
...
0101 (V) 10.0.101.101 0011.9400.044d Po 101 Never R
0101 (V) 10.0.101.102 0011.9400.044e Po 101 Never R
...
--------------------------------------------------------------------------------
10.0.131.101 0011.9400.04e3 Ve 131 yes 00:00:39 Dynamic
10.0.131.102 0011.9400.04e4 Ve 131 yes 00:00:39 Dynamic
...
POD1-Leaf1-2# show ip arp suppression-cache vlan 131
---------------------------------------------------------------------------------------------------
0131 (V) 10.0.131.101 0011.9400.04e3 Po 104 Never R
0131 (V) 10.0.131.102 0011.9400.04e4 Po 104 Never R
...
POD1-Leaf3# show arp ve 101 vrf vrf101

--------------------------------------------------------------------------------
10.0.101.201 0010.9400.1b96 Ve 101 yes 00:00:12 Dynamic
10.0.101.202 0010.9400.1b97 Ve 101 yes 00:00:12 Dynamic
...
POD1-Leaf3#
72
Remote Host Entries in the Extended VLAN
The following table from Leaf3 shows the BGP and ARP entries of the remote hosts behind the Leaf1 pair. Note that the next hop is set to 10.1.1.1,
which is a common VTEP IP of the vLAG pair. This causes the redundant leaf to appear as one VTEP in the underlay network, and load balancing is
accomplished.
In the ARP suppression-cache, both the local and remote entries are indicated with different flags. “R” for remote entries signify that they were learned
over BGP EVPN and the local entries are marked as "L”.
POD1-Leaf3# show bgp evpn routes type arp 10.0.101.101 mac 0011.9400.044d ethernet-tag 0
Status A:AGGREGATE B:BEST b:NOT-INSTALLED-BEST C:CONFED_EBGP D:DAMPED
E:EBGP H:HISTORY I:IBGP L:LOCAL M:MULTIPATH m:NOT-INSTALLED-MULTIPATH
S:SUPPRESSED F:FILTERED s:STALE
1 Prefix: ARP:[0][0011.9400.044d]:[IPv4:10.0.101.101], Status: BE, Age: 3h50m24s
NEXT_HOP: 10.1.1.1, Learned from Peer: 10.0.3.1 (4200000000)
LOCAL_PREF: 100, MED: none, ORIGIN: incomplete, Weight: 0
AS_PATH: 4200000000 4200000001
Extended Community: RT 59905:1073741925 RT 42000:1 RT 101:101
ExtCom:06:03:60:9c:9f:b0:f5:01 RT 59905:268435557 RT 59905:101 ExtCom:03:0c:00:00:00:00:00:08
Extended Community: ExtCom: Tunnel Encapsulation (Type Vxlan)
Adj_RIB_out count: 3, Admin distance 20
L2 Label: 101 L3 Label: 7097 (VNI) Router Mac : 609c.9fb0.f501
ESI : 00.000000000000000000
RD: 10.1.1.11:32869
2 Prefix: ARP:[0][0011.9400.044d]:[IPv4:10.0.101.101], Status: E, Age: 3h50m24s
AS_PATH: 4200000000 4200000001
ESI : 00.000000000000000000
RD: 10.1.1.11:32869
POD1-Leaf3# show ip arp suppression-cache vlan 101
---------------------------------------------------------------------------------------------------
0101 (V) 10.0.101.103 0011.9400.044f Tu 61441 (10.1.1.1) Never R
0101 (V) 10.0.101.104 0011.9400.0450 Tu 61441 (10.1.1.1) Never R
0101 (V) 10.0.101.105 0011.9400.0451 Tu 61441 (10.1.1.1) Never R
0101 (V) 10.0.101.106 0011.9400.0452 Tu 61441 (10.1.1.1) Never R
0101 (V) 10.0.101.107 0011.9400.0453 Tu 61441 (10.1.1.1) Never R
0101 (V) 10.0.101.201 0010.9400.1b96 Eth 0/34 00:21:38 L
0101 (V) 10.0.101.202 0010.9400.1b97 Eth 0/34 00:21:38 L
Local host ARP entries

Remote host ARP entries
Verify Tenant Extension between the Racks

Tenant extension ensures routing between the VXLAN segments within the same tenant.
As shown in Figure 18, VNI segment 131 is provisioned only on the MCT ToR but is part of the tenant on both ToRs. Let's go over a list of verification
steps required to ensure that communication between the hosts in VNI 101 on Leaf3 and hosts in VNI 131 on MCT Leaf1.
73
RMAC of Each Node
There is one RMAC assigned to every VTEP. This information can be obtained by looking at any of the L3 interfaces or the L3 VNI's associated VLAN
interface. For the MCT pair, even though they have same VTEP IP, they are assigned a unique router MAC.
POD1-Leaf1-1# show ip int ve 101 POD1-Leaf1-2# show ip int ve 101

Ve 101 is up protocol is up Ve 101 is up protocol is up
Vlan is 101 Vlan is 101
Hardware is Virtual Ethernet, address is 609c.9fb0.f501 Hardware is Virtual Ethernet, address is 609c.9fb0.d801
Current address is 609c.9fb0.f501 Current address is 609c.9fb0.d801
Interface index (ifindex) is 1207959653 Interface index (ifindex) is 1207959653

Vlan is 101
74
L3 VNI State on the Nodes
Bridge-domain 3001 (VNI 7097) is assigned to the tenant VRF. Make sure that the MCT ToR and Leaf3 have tunnels established to each other and that
this bridge-domain is activated on it.
As seen in the following table for the output from Leaf1, the tunnel source is the VTEP IP of the vLAG, 10.1.1.1, and the destination IP is the VTEP IP of
Leaf3, 10.1.3.1. (Notice additional tunnels in the list; these are destined to other VTEPs where the same tenant is provisioned.)
POD1-Leaf1-1# show bridge-domain 3001

Bridge-domain 3001
-------------------------------
VE id: 3001, if-indx: 1207962553
Tunnels: tu61444.7097 tu61441.7097 tu61443.7097 tu61445.7097 tu61446.7097
VLAN: N/A, Tagged ports: 1(1 up), Un-tagged ports: 0 (0 up)
Un-tagged Ports:
POD1-Leaf1-1#







POD1-Leaf1-1#
75
L3 Bridge-domain/VNI state from Leaf3
POD1-Leaf3# show bridge-domain 3001

Bridge-domain 3001
-------------------------------
VE id: 3001, if-indx: 1207962553
Tunnels: tu61446.7097 tu61443.7097 tu61444.7097 tu61445.7097 tu61441.7097
POD1-Leaf3# show tunnel 61441

Tunnel 61441, mode VXLAN, node-ids 1



Verify the Route to the Remote Subnet of the Same Tenant
The following table shows the BGP entry on Leaf3 for the remote subnet of vlan/VNI 131.
There will be 8 entries in the BGP table: 2 originators in the MCT pair and those two entries are learned from four spines exchanging EVPN routes.
Again, the next hop is the same for all paths due to the common VTEP IP used by the MCT pair.
76
POD1-Leaf3# show bgp evpn routes type ipv4-prefix 10.0.131.0/24 tag 0
1 Prefix: IP4Prefix:[0][10.0.131.0/24], Status: BE, Age: 3d1h49m53s
AS_PATH: 4200000000 4200000001
Extended Community: RT 42000:1 RT 101:101 ExtCom:06:03:60:9c:9f:b0:f5:01 ExtCom:03:0d:00:00:00:00:00:00 RT
59905:1073748921 ExtCom:03:0c:00:00:00:00:00:08
Default Extd Gw Community: Received
Label: 7097 (VNI) Router Mac : 609c.9fb0.f501
RD: 10.1.1.11:1
2 Prefix: IP4Prefix:[0][10.0.131.0/24], Status: E, Age: 3d1h49m53s
AS_PATH: 4200000000 4200000001
59905:1073748921 ExtCom:03:0c:00:00:00:00:00:08
RD: 10.1.1.11:1
AS_PATH: 4200000000 4200000001
59905:1073748921 ExtCom:03:0c:00:00:00:00:00:08
RD: 10.1.1.11:1
4 Prefix: IP4Prefix:[0][10.0.131.0/24], Status: BE, Age: 2d21h2m32s
AS_PATH: 4200000000 4200000001
Extended Community: RT 42000:1 RT 101:101 ExtCom:06:03:60:9c:9f:b0:d8:01 ExtCom:03:0d:00:00:00:00:00:00 RT
59905:1073748921 ExtCom:03:0c:00:00:00:00:00:08
Label: 7097 (VNI) Router Mac : 609c.9fb0.d801
RD: 10.1.1.12:1
AS_PATH: 4200000000 4200000001
59905:1073748921 ExtCom:03:0c:00:00:00:00:00:08
RD: 10.1.1.12:1
AS_PATH: 4200000000 4200000001
59905:1073748921 ExtCom:03:0c:00:00:00:00:00:08
RD: 10.1.1.12:1
...
POD1-Leaf3# show ip bgp routes 10.0.131.0/24 vrf vrf101

Number of BGP Routes matching display condition : 2
Prefix Next Hop MED LocPrf Weight Status
1 10.0.131.0/24 10.1.1.1 none 100 0 BME
AS_PATH: 4200000000 4200000001
Label: 7097 Router Mac : 609c.9fb0.f501
2 10.0.131.0/24 10.1.1.1 none 100 0 ME
AS_PATH: 4200000000 4200000001
Label: 7097 Router Mac : 609c.9fb0.d801
Last update to IP routing table: 2d21h8m42s
Route is not advertised to any peers
POD1-Leaf3# show ip route 10.0.131.0/24 vrf vrf101
10.0.131.0/24
*via 10.1.1.1%default-vrf, Ve 3001, [20/0], 2d21h, eBgp, tag 0, (VNI 7097, GW MAC 609c.9fb0.d801, Tu 61441)
*via 10.1.1.1%default-vrf, Ve 3001, [20/0], 2d21h, eBgp, tag 0, (VNI 7097, GW MAC 609c.9fb0.f501, Tu 61441)
POD1-Leaf3#
77
VLAN Scoping at the ToR Level
VLAN scoping is briefly discussed in the Technology Overview section.
Refer to Figure 19 for the topology used to illustrate the VLAN scoping at the leaf or ToR level. For the purpose of illustration, we’ve chosen a MCT pair
and an individual leaf.
As seen in the diagram, each leaf has a server VLAN that requires a Layer 2 extension to the other rack. Also note that the VLAN numbers are different
on each Rack. By mapping these VLANs to the same VNI number—5257 in this case—we achieve bridging or L2 extension between them. The servers
now have L2 adjacency between them. In other words, they are in the same bridge domain or broadcast domain. In essence, the VLAN tag on the wire
between the servers and the leaf is decoupled from the bridge domain. This VLAN tag need not be identical on both sides to have Layer 2 adjacency or
extension. In other words, the VLAN number is relevant only at the ToR level.
VLAN scoping uses bridge-domain on the Leafs. This can be implemented using traditional VLANs also. However, the VLAN on each leaf must be
mapped manually to a common VNI. Automatic mapping of VLAN to VNI cannot be used. For example:
• Leaf1 – VLAN 161 is mapped to VNI 161
• Leaf2 – VLAN 361 is mapped to VNI 161
Bridge-domain simplifies the configuration for VLAN scoping case using auto VLAN to VNI mapping.
Figure 19 VLAN Scoping at the ToR Level

FIGURE 21 VLAN Scoping at the ToR Level
VNI 5257
Spines
L3 VNI 7097
Leaf1-1 Leaf1-2 Leaf3

VTEP-IP 10.1.1.1 VTEP-IP 10.1.1.1 VTEP-IP 10.1.3.1
BD 1161/VNI 5257 BD 1161/VNI 5257 BD 1161/VNI 5257
BD 1161/VE 1161 BD 1161/VE 1161 BD 1161/VE 1161

Anycast GW 10.4.137.254 Anycast GW 10.4.137.254 Anycast GW 10.4.137.254
Po 107 Eth 0/34
VLAN 161 VLAN 361

10.4.137.101/0011.9400.0579 10.4.137.201/0010.9400.2622
10.4.137.102/0011.9400.057a 10.4.137.202/0010.9400.2623
78
Configuration
The configuration steps are similar to the L2 extension illustrated in the use-case “L2 and L3 Extension between Racks”. The difference is that vlan-
scoping is achieved using bridge-domain instead of vlan configuration. Difference in configuration is shown below comparing an L2 tenant
configuration based of vlan and of bridge-domain.
vlan 101 interface Port-channel 107

description L2 Tenant vlan switchport
! switchport mode trunk-no-default-native
At the port-level vlans are marked to a
interface Port-channel 101 logical-interface port-channel 107.161
switchport vlan 161 logical interface and the logical-
switchport mode trunk-no-default-native ! interface are bound to the bridge-
switchport trunk allowed vlan add 101 bridge-domain 1161 domain
! description L2 Tenant vlan
evpn default logical-interface port-channel 107.161
vlan add 101 !
! evpn default
bridge-domain add 1161 Bridge-domain will map to a L2 VNI
! based on manual “BD to vni map” or
“map vni auto” under overlay-gateway
configuration
Bridge-domain mapping to a vni can be done using either of these methods
• With “map vni auto” command under “overlay-gateway” configuration, bridge-domain will get vni automatically assigned. VNI for BD
will be equal to “4096 + BD value”, so for example BD 100 will map to vni 4196 with “map vni auto”. The recommended practice is to
use this method of vni assignment.
• User could also manually map BD to a desired VNI value. This is done using BD-to-VNI mapping configuration under the overlay gateway
configuration. However every VLAN and BD must be manually mapped to VNIs.
The table below summarizes the provisioning of L2 extension on leafs.
Leaf 1 Leaf 3
Server traffic is tagged with VLAN 161. Server traffic is tagged with VLAN 361.
Create logical interface to mark VLAN 161. Create logical interface to mark VLAN 361.
Add logical interface as member of bridge-domain 1161 Add logical interface as member of bridge-domain 1161.
Add the BD 1161 under evpn which will map it to vni 5257. Add the BD 1161 under evpn which will map it to vni 5257.
Assign VE 1161 as router-interface for BD 1161. Assign VE 1161 as router-interface for BD 1161.
Create the VE 1161 Layer 3 interface for first-hop routing. Create the VE 1161 Layer 3 interface for first-hop routing.
Assign the anycast GW 10.4.137.254 address to VE 1161. Assign the anycast GW 10.4.137.254 address to VE 1161.
Complete configurations and verification steps on leafs in the Figure 21 topology are given in the sections that follow.

The configuration is shown in three parts for clarity:
• Common configurations, such as port channel and VLANs, are shown in one block.
• The tenant, Layer 3 interfaces, and BGP EVPN configurations are shown in the second block.
• The common overlay-gateway configuration is shown in the third block.
79
logical-interface port-channel 107.161
vlan 161
!
bridge-domain 1161
description L2 Tenant vlan
!
bridge-domain 3001
description BD 3001, L3 VNI 7097, Tenant vrf101
!
evpn default
bridge-domain add 1161, 3001
!

! !
! !
! !
rd 10.1.1.11:101 rd 10.1.1.12:101
! !
! !
rd auto rd auto
! !
! !
! !
! !
! !
! !
80
overlay-gateway mct-leaf1
map vni auto
activate
!
logical-interface ethernet 0/34.361
vlan 361
!
bridge-domain 1161
!
bridge-domain 3001
!
evpn default
!
81
no shutdown
!
no shutdown
!
!
vrf vrf101
rd 10.1.3.2:101
evpn irb ve 3001
!
!
evpn default
rd auto
!
router bgp
maximum-paths 8
!
maximum-paths 8
!
!
interface Ve 1161
ip arp learn-any
ipv6 anycast-address fdf8:10:0:489::254/96
no shutdown
!
interface Ve 3001
no shutdown
!
map vni auto
activate
!
Verification
Check the L2-extended bridge-domain on each node. This should show the local L2 logical ports & local vlan marking and also the tunnels to all remote
VTEPs where the same bridge-domain/VNI is extended.
In the output below from the Leaf1 MCT pair, there are three tunnels for bridge-domain 1161, which indicates that the same Bridge-domain/VNI
segment is provisioned on three other VTEPs or ToR. Note that one of the tunnels, Tu 61445, is destined to Leaf3. Also note that there are four
underlay next hops to reach this tunnel destination in the fabric.
82
Bridge-domain 1161
-------------------------------
Description:
VE id: 1161, if-indx: 1207960713
VLAN: 161, Tagged ports: 1(1 up), Un-tagged ports: 0 (0 up)
Un-tagged Ports: List of local server facing ports
VNI: 5257, Tunnels: 3(3 up) and tunnels to each VTEP
Tunnels: tu61445.5257 tu61446.5257 tu61441.5257
VLAN: N/A, Tagged ports: 1(1 up), Un-tagged ports: 0 (0 up) where the VLAN is extended
Un-tagged Ports:
Tu 61446 is destined to leaf3’s
vtep-ip 10.1.3.1
Configuration source BGP-EVPN Underlay IP next-hops from
MAC learning BGP-EVPN each vLAG peer to reach the
IP: 10.0.1.5, Vrf: default-vrf remote VTEP leaf3. This shows
Egress L3 port: Eth 0/53, Outer SMAC: 609c.9fb0.f539 4 paths as there are 4 spine
links








Byte count : RX 2150260 TX 5673182
POD1-Leaf1-1#
In the output below from Leaf3, Tunnel 61442 is destined to the vLAG Leaf1 pair's VTEP IP 10.121.1.1.
83
Bridge-domain 1161
-------------------------------
Description:
Number of configured end-points: 4 , Number of Active end-points: 4 List of local member ports and
VE id: 1161, if-indx: 1207960713 tunnels to each VTEP where
Tagged Ports: eth0/34.361 the vlan is extended
Un-tagged Ports:
Tunnels: tu61445.5257 tu61441.5257 tu61446.5257
POD1-Leaf3# show tunnel 61441 vtep-ip 10.1.1.1
Active next hops on node 1: Underlay IP next-hops from
Egress L3 port: Eth 0/51, Outer SMAC: 609c.9fb1.5637 Leaf3 to reach the remote vtep
Outer DMAC: 609c.9fb0.ac07 on MCT pair leaf1-1/1-2. This
shows 4 paths as there are 4
IP: 10.0.3.1, Vrf: default-vrf spine links




Byte count : RX 6832086 TX 2073558
POD1-Leaf3#
VLAN Layer 3 Interfaces State on the MCT Pair

Bridge domain is 1161 Bridge domain is 1161
Primary Internet Address is 10.4.137.254/24 broadcast is 10.4.137.255 Primary Internet Address is 10.4.137.254/24 broadcast is 10.4.137.255
IP MTU is 9000 IP MTU is 9000
Proxy Arp is not Enabled Proxy Arp is not Enabled
ICMP unreachables are always sent ICMP unreachables are always sent
ICMP mask replies are never sent ICMP mask replies are never sent
IP fast switching is enabled IP fast switching is enabled
Vrf : vrf101 Vrf : vrf101

Bridge domain is 3001 Bridge domain is 3001
IP unassigned IP unassigned
IP MTU is 9000 IP MTU is 9000
Proxy Arp is not Enabled Proxy Arp is not Enabled
ICMP unreachables are always sent ICMP unreachables are always sent
ICMP mask replies are never sent ICMP mask replies are never sent
IP fast switching is enabled IP fast switching is enabled
Vrf : vrf101 Vrf : vrf101
84
VLAN Layer 3 Interfaces State on the Leaf3 ToR
POD1-Leaf3# show ip interface ve 1161

IP MTU is 9000
Proxy Arp is not Enabled
ICMP unreachables are always sent
ICMP mask replies are never sent
IP fast switching is enabled
Vrf : vrf101
POD1-Leaf3# show ip interface ve 3001

IP unassigned
IP MTU is 9000
Proxy Arp is not Enabled
ICMP unreachables are always sent
ICMP mask replies are never sent
IP fast switching is enabled
Vrf : vrf101

--------------------------------------------------------------------------------
10.4.137.101 0011.9400.0579 Ve 1161 yes 00:05:30 Dynamic
10.4.137.102 0011.9400.057a Ve 1161 yes 00:05:21 Dynamic
...
--------------------------------------------------------------------------------
10.4.137.101 0011.9400.0579 Ve 1161 yes 00:06:03 Dynamic
10.4.137.102 0011.9400.057a Ve 1161 yes 00:05:54 Dynamic
...

--------------------------------------------------------------------------------
10.4.137.201 0010.9400.2622 Ve 1161 yes 00:11:11 Dynamic
10.4.137.202 0010.9400.2623 Ve 1161 yes 00:10:51 Dynamic
...
POD1-Leaf3#
The table below from Leaf3 shows the BGP and ARP entries of a remote host behind the Leaf1 pair for bridge-domain 1161 or vni 5257. Note that the
next hop is set to 10.1.1.1, which is a common VTEP IP of the MCT pair.
In the ARP suppression-cache, both the local and remote entries are indicated with different flags. “R” for remote entries signify that they were
learned over BGP EVPN and the local entries are marked as "L”.
85
POD1-Leaf3# show bgp evpn routes type arp 10.4.137.101 mac 0011.9400.0579 ethernet-tag 0
1 Prefix: ARP:[0][0011.9400.0579]:[IPv4:10.4.137.101], Status: BE, Age: 0h26m19s
AS_PATH: 4200000000 4200000001
ESI : 00.000000000000000000
RD: 10.1.1.11:38025
2 Prefix: ARP:[0][0011.9400.0579]:[IPv4:10.4.137.101], Status: E, Age: 0h26m19s
AS_PATH: 4200000000 4200000001
ESI : 00.000000000000000000
RD: 10.1.1.11:38025
...
POD1-Leaf3# show ip arp suppression-cache bridge-domain 1161

---------------------------------------------------------------------------------------------------
1161 (B) 10.4.137.101 0011.9400.0579 Tu 61441 (10.1.1.1) Never R
1161 (B) 10.4.137.102 0011.9400.057a Tu 61441 (10.1.1.1) Never R
1161 (B) 10.4.137.103 0011.9400.057b Tu 61441 (10.1.1.1) Never R
1161 (B) 10.4.137.104 0011.9400.057c Tu 61441 (10.1.1.1) Never R
1161 (B) 10.4.137.201 0010.9400.2622 Eth 0/34 00:16:33 L
1161 (B) 10.4.137.202 0010.9400.2623 Eth 0/34 00:16:13 L

86
VLAN Scoping at the Port Level within a ToR
Port VLAN scoping enables complete abstraction of a bridge domain where the VLAN tags on the server-side data frame on two ports can be different
and still be bridged between the ports. The VLAN tag is localized at the port level rather than at the ToR level.
Refer to the topology shown in Figure 19.
On the vLAG leaf, there are two port channels or LAG bundles: po107 and po110. Each has server traffic tagged with an 802.1q VLAN tag of 162 and
262, respectively. From the port VLAN scoping perspective, these tags are referred to as c-tags. The {port,vlan} is added as a member of a virtual-fabric
VLAN. In this case, there is a fabric VLAN ID 6000. (Note that this number is above the 802.1q VLAN range of 4096.)
In summary, BD 1162 comprises two members (port, vlan). (Unlike the ports in traditional VLAN cases.)
• (po107, vlan tag 162)
• (po110, vlan tag 262)
On Leaf3, VLAN 362 is mapped to BD 1162/VNI 5258. On the Leaf1 pair, bridge-domain 1162 is mapped to VNI 5258. Thus we're providing Layer 2
extension within and between the ToRs for server-side traffic with different dot1q VLAN tags.
Figure 20 Port VLAN scoping within the ToR
L2 VNI 5258
Spines
L3 VNI 7097
Leaf1-1 Leaf1-2 Leaf3
VTEP-IP 10.1.1.1 VTEP-IP 10.1.1.1 VTEP-IP 10.1.3.1
BD 1162/VNI 5258 BD 1162/VNI 5258 BD 1162/VNI 5258
VE 1162 VE 1162 VE 1162

Anycast GW 10.4.138.254 Anycast GW 10.4.138.254 Anycast GW 10.4.138.254
Po 110 Eth 0/34
Po 107
VLAN tag 362

VLAN tag 162 VLAN tag 262 10.4.138.201/0010.9400.262c
10.4.138.101/0011.9400.0583 10.4.138.121/0010.9401.16ef 10.4.138.202/0010.9400.262d
10.4.138.102/0011.9400.0584 10.4.138.122/0010.9401.16f0 Tenant vrf101
Configuration
A sample configuration is given below as a quick reference for port-VLAN scoping. In this example, {po107, tag 162} and
[po110, tag 262] are mapped to BD 1162/ VNI 5258. .With this configuration, it is possible to bridge traffic on these ports with the specified dot1q
tags.
87
switchport
switchport mode trunk-no-default-native
vlan 162
!
switchport
vlan 262
!
bridge-domain 1162 p2mp

The configuration is shown in three parts for clarity:
• Common configurations, such as port channel and bridge-domains, are shown in one block.
• The tenant, Layer 3 interfaces, and BGP EVPN configurations are shown in the second block under each RBridge ID.
• The common overlay-gateway configuration is shown in the third block.

vlan 162
!
vlan 262
!
bridge-domain 1162
!
bridge-domain 3001
!
evpn default
!
88
! !
! !
! !
rd 10.1.1.11:101 rd 10.1.1.12:101
! !
! !
rd auto rd auto
bridge-domain add 1162, 3001 bridge-domain add 1162,3001
! !
! !
! !
! !
ipv6 anycast-address fdf8:10:0:48a::254/96 ipv6 anycast-address fdf8:10:0:48a::254/96
! !
! !
overlay-gateway mct-leaf1
map vni auto
activate
!
89
vlan 362
!
bridge-domain 1162
!
bridge-domain 3001
!
evpn default
!
no shutdown
!
no shutdown
!
!
vrf vrf101
rd 10.1.3.2:101
evpn irb ve 3001
!
!
evpn default
rd auto
!
router bgp
maximum-paths 8
!
maximum-paths 8
!
!
interface Ve 1162
ip arp learn-any
ipv6 anycast-address fdf8:10:0:48a::254/96
no shutdown
!
interface Ve 3001
no shutdown
!
90
map vni auto
activate
!
Verification
Check the L2 extended bridge-domain/VNI on each node. This should show the local L2 member ports and also the tunnels to all remote VTEPs where
the same VNI segment is extended.
In the output below from the Leaf1 MCT pair for the bridge-domain, there are few VxLAN tunnels, which indicates that the same Bridge-domain/VNI
segment is provisioned on other VTEPs or ToRs. Note that one of the tunnels, Tu 61446, is destined to Leaf3. Also note that there are four underlay
next hops to reach this tunnel destination in the fabric.
91
Bridge-domain 1162
-------------------------------
Description: L2 Tenant vlan
VE id: 1162, if-indx: 1207960714
Un-tagged Ports: List of local server facing ports
VLAN: 262, Tagged ports: 1(1 up), Un-tagged ports: 0 (0 up) and tunnels to each VTEP
Un-tagged Ports: where the VLAN is extended
Tunnels: tu61445.5258 tu61446.5258 tu61441.5258
VLAN: N/A, Tagged ports: 1(1 up), Un-tagged ports: 0 (0 up) Tu 61446 is destined to leaf3’s
vtep-ip 10.1.3.1
Un-tagged Ports:

Overlay gateway "mct-leaf", ID 1 Underlay IP next-hops from
Source IP 10.1.1.1, Vrf default-vrf each vLAG peer to reach the
Configuration source BGP-EVPN remote VTEP leaf3. This shows
MAC learning BGP-EVPN 4 paths as there are 4 spine
links








Byte count : RX 2150260 TX 5673182
POD1-Leaf1-1#
92
In the output below from Leaf3, Tunnel 61441 is destined to the MCT Leaf1 pair's VTEP IP 10.1.1.1
Bridge-domain 1162
-------------------------------
Description:
Number of configured end-points: 4 , Number of Active end-points: 4 List of local member ports and
VE id: 1162, if-indx: 1207960714 tunnels to each VTEP where
Tagged Ports: eth0/34.362 the vlan is extended
Un-tagged Ports:
Tunnels: tu61445.5258 tu61441.5258 tu61446.5258
POD1-Leaf3# show tunnel 61441 vtep-ip 10.1.1.1
Active next hops on node 1: Underlay IP next-hops from
Egress L3 port: Eth 0/51, Outer SMAC: 609c.9fb1.5637 Leaf3 to reach the remote vtep
Outer DMAC: 609c.9fb0.ac07 on MCT pair leaf1-1/1-2. This
shows 4 paths as there are 4
IP: 10.0.3.1, Vrf: default-vrf spine links




Byte count : RX 6832086 TX 2073558
POD1-Leaf3#

--------------------------------------------------------------------------------
10.4.138.101 0011.9400.0583 Ve 1162 yes 00:21:52 Dynamic
10.4.138.102 0011.9400.0584 Ve 1162 yes 00:21:52 Dynamic
...
10.4.138.121 0010.9401.16ef Ve 1162 yes 00:09:31 Dynamic
10.4.138.122 0010.9401.16f0 Ve 1162 yes 00:09:31 Dynamic
...
POD1-Leaf1-2# show ip arp suppression-cache bridge-domain 1162
---------------------------------------------------------------------------------------------------
1162 (B) 10.4.138.101 0011.9400.0583 Po 107 Never R
1162 (B) 10.4.138.102 0011.9400.0584 Po 107 Never R
...
POD1-Leaf1-1# show mac-address-table bridge-domain 1162 | inc 0583|0584|16ef|16f0
1162 (B) 0010.9401.16ef Dynamic-CCL Active Po 110.262
1162 (B) 0010.9401.16f0 Dynamic-CCL Active Po 110.262
1162 (B) 0011.9400.0583 Dynamic-CCL Active Po 107.162
1162 (B) 0011.9400.0584 Dynamic-CCL Active Po 107.162
POD1-Leaf1-1#
93
--------------------------------------------------------------------------------
10.4.138.201 0010.9400.262c Ve 1162 yes 00:23:51 Dynamic
10.4.138.202 0010.9400.262d Ve 1162 yes 00:23:40 Dynamic
...
POD1-Leaf3#
The table below from Leaf3 shows the BGP and ARP entries of the remote hosts behind the Leaf1 pair. Note that the next hop is set to 10.1.1.1, which
is a common VTEP IP of the MCT pair.
In the ARP table, both the local and remote entries are indicated with different types: BGP-EVPN for remote entries, signifying that they were learned
over BGP EVPN; Dynamic for local entries. Note that the remote host entries are imported into the virtual interface of local BD 1162 on Leaf3.
POD1-Leaf3# show bgp evpn routes type arp 10.4.138.101 mac 0011.9400.0583 ethernet-tag 0
1 Prefix: ARP:[0][0011.9400.0583]:[IPv4:10.4.138.101], Status: BE, Age: 3d4h5m25s
AS_PATH: 4200000000 4200000001
ESI : 00.000000000000000000
RD: 10.1.1.11:38026
2 Prefix: ARP:[0][0011.9400.0583]:[IPv4:10.4.138.101], Status: E, Age: 3d4h5m25s
AS_PATH: 4200000000 4200000001
ESI : 00.000000000000000000
RD: 10.1.1.11:38026
...
POD1-Leaf3# show ip arp suppression-cache bridge-domain 1162

---------------------------------------------------------------------------------------------------
1162 (B) 10.4.138.101 0011.9400.0583 Tu 61441 (10.1.1.1) Never R
1162 (B) 10.4.138.102 0011.9400.0584 Tu 61441 (10.1.1.1) Never R
1162 (B) 10.4.138.103 0011.9400.0585 Tu 61441 (10.1.1.1) Never R
1162 (B) 10.4.138.104 0011.9400.0586 Tu 61441 (10.1.1.1) Never R
1162 (B) 10.4.138.201 0010.9400.262c Eth 0/34 00:14:24 L
1162 (B) 10.4.138.202 0010.9400.262d Eth 0/34 00:14:13 L

94
Layer-2 Handoff with VPLS
This use-case illustrates VLAN or Layer-2 extension between two EVPN fabrics over VPLS. The choice of VPLS depends on the VNI ranges used in these
two fabrics. This use case may also be extended to interconnecting an EVPN fabric to a traditional L2 network.
If both fabrics used consistent VNI range i.e. VNI X maps to same VLAN segment on both fabrics and the workloads on them are L2 adjacent, BGP EVPN
based Fabric extension would be an appropriate option.
As shown in the topology below, a pair of switches in MCT pair act as border-leaf that terminate the VxLAN encapsulated packets from the EVPN fabric
and hand off classical Ethernet packets to the MPLS edge devices. Similarly on the other direction, these border-leafs map and encapsulate the CE
packets into VNI segments towards the EVPN fabrics.
The following devices are used in this illustration and these are part of the validated design.
PIN Platform
Edge/Border-Leaf SLX 9140 MCT pair
MPLS Edge SLX 9540 MCT pair
Leaf SLX 9140 MCT pair
Note that only the relevant portion of the validated design topology is shown in the diagram and configuration section will include only the relevant
and incremental configuration needed for the use-case. For building the Fabric Underlay and Overlay, please refer to the validated design section.
Figure 21 Layer-2 Handoff with VPLS
MPLS
Super-spines
DCI-1 DCI-2 Spines

S i
DC1 DCI DC2 DCI
Leaf1-1 Leaf1-2
Leaf1-1 Leaf1-2
Po 211 Po 212 Edge-leaf Pair Edge Leaf Pair
DC1 DC2
Configuration
Configuration steps:
• Border/Edge-leaf MCT pair configuration. Refer to the validated design section for more details on MCT pair configuration.
95
• Configure the VLAN and VNI mappings for all VLAN segments that must be extended outside the fabric.
• Enable the VLANs on the CE edge ports or port-channels connected to the DCI Edge nodes.
• Configure VE interface on the border-leafs for these VLAN segments to enable ARP suppression for remote hosts in the other data centers. In
other words, this border-leaf will respond for ARP requests sent from external hosts to internal hosts.
In this use-case, VLAN 101 from DC1 is extended to DC2.
SLX-9140 MCT Pair Border-Leaf

Configurations of the two nodes in the pair are shown side by side.
mtu 9216 mtu 9216

! !
! !
! !
vlan 4090 vlan 4090
! !
! !
description MCT peer-link LAG description MCT peer-link LAG
! !
description LAG to DCI description LAG to DCI
! !
vlan 101 vlan 101
description Extended over DCI description Extended over DCI
! !
! !
96
cluster management node-id 1 cluster management node-id 2
node-id 1 !
cluster management principal-priority 1
!
cluster Edge-Leaf 6162 cluster Edge-Leaf 6162
peer 10.61.62.1 peer 10.61.62.0
deploy deploy
client DCI 1 client DCI 1
deploy deploy
! !
Overlay gateway configuration is applied from the principal node.
overlay-gateway Edge-Leaf
map vni auto
activate
!

rd auto rd auto
vlan add 101 vlan add 101
! !
SLX-9540 MCT Pair DCI Edge

The DCI Edge is built using two SLX-9540 routers in a MCT pair and their configurations are shown side by side.
A sample MPLS and VPLS configuration is included for completeness. MPLS/VPLS configuration detail is beyond the scope of this document. There are
various ways of setting up the MPLS tunnels (LDP or traffic engineering) and VPLS domains over them.
97
Cluster, Access Circuit and BGP
! !
ip route 10.56.56.1/32 10.55.56.1 ip route 10.55.55.1/32 10.55.56.0
! !
vlan 4090 vlan 4090
description MCT peering vlan description MCT peering vlan
! !
! !
mtu 9216 mtu 9216
switchport mode trunk switchport mode trunk
switchport trunk tag native-vlan switchport trunk tag native-vlan
! !
description MCT LAG to Edge-Leaf description MCT LAG to Edge-Leaf
logical-interface port-channel 2.101 logical-interface port-channel 2.101
vlan 101 vlan 101
! !
! !
cluster DCI-cluster 5556 cluster DCI-cluster 5556
member bridge-domain add 101 member bridge-domain add 101
peer-interface Ve 4090 peer-interface Ve 4090
peer 10.56.56.1 peer 10.55.55.1
client-isolation loose client-isolation loose
deploy deploy
client Edge-Leaf 1 client Edge-Leaf 1
esi 0:0:0:0:0:0:0:1:1 esi 0:0:0:0:0:0:0:1:1
deploy deploy
! !
client-pw client-pw
esi 0:0:0:0:0:0:0:2:2 esi 0:0:0:0:0:0:0:2:2
deploy deploy
! !
! !
local-as 100 local-as 100
neighbor 10.56.56.1 update-source loopback 1 neighbor 10.55.55.1 update-source loopback 1
neighbor 10.56.56.1 encapsulation mpls neighbor 10.55.55.1 encapsulation mpls
! !
98
MPLS Link and IGP
router isis router isis
net 49.0001.0010.0055.5501.00 net 49.0001.0010.0056.5601.00
fast-flood 5 fast-flood 5
is-type level-1 is-type level-1
log adjacency log adjacency
lsp-gen-interval 5 lsp-gen-interval 5
lsp-refresh-interval 64000 lsp-refresh-interval 64000
max-lsp-lifetime 65535 max-lsp-lifetime 65535
partial-spf-interval 5000 150 300 partial-spf-interval 5000 150 300
spf-interval level-1 5 150 300 spf-interval level-1 5 150 300
metric-style wide level-1 metric-style wide level-1
default-metric 10 default-metric 10
! !
ip router isis ip router isis
! !
description BL to MPLS cloud description BL to MPLS cloud
isis auth-mode md5 level-1 isis auth-mode md5 level-1
isis auth-key level-1 $9$BwrsDbB+tABWGWpINOVKoQ== isis auth-key level-1 $9$BwrsDbB+tABWGWpINOVKoQ==
isis point-to-point isis point-to-point
! !
99
MPLS LSP and VPLS Pseudowire
pw-profile vpls pw-profile vpls
mtu 9100 mtu 9100
! !
router mpls router mpls
policy policy
traffic-engineering isis level-1 traffic-engineering isis level-1
ingress-tunnel-accounting ingress-tunnel-accounting
transit-session-accounting transit-session-accounting
! !
ldp ldp
lsr-id 55.1.1.1 lsr-id 56.1.1.1
! !
dynamic-bypass dynamic-bypass
enable-all-interfaces enable-all-interfaces
! !
mpls-interface ethernet 0/48 mpls-interface ethernet 0/48
! !
mpls-interface ve 4090 mpls-interface ve 4090
! !
path to-MLX-1 path to-MLX
hop 55.0.1.1 strict hop 56.0.1.1 strict
! !
lsp to-MLX-DC2 lsp to-MLX-DC2
to 66.1.1.1 to 66.1.1.1
primary-path to-MLX-1 primary-path to-MLX
adaptive adaptive
frr frr
facility-backup facility-backup
! !
enable enable
! !
bridge-domain 101 p2mp bridge-domain 101 p2mp
vc-id 101 vc-id 101
peer 66.1.1.1 load-balance peer 66.1.1.1 load-balance
statistics statistics
logical-interface port-channel 2.101 logical-interface port-channel 2.101
pw-profile vpls pw-profile vpls
bpdu-drop-enable bpdu-drop-enable
local-switching local-switching
! !
Verification
Verification steps are shown on the DC1 nodes, and the same steps can be followed on the DC2 nodes as well.
Verification involves the following.
• VLAN state on the border-leaf.
o VxLAN Tunnel membership towards the internal Leafs where the VLAN is provisioned.
o CE Trunk port connected to the DCI Edge node.
• VLAN state on the DCI Edge.
o CE Trunk port to border-leaf
o VPLS Pseudowire to remote DCI Edge.
• Local and Remote MAC learning on both border-leaf and DCI Edge.
100
Edge/Border-Leaf MCT Pair
The output below taken from one of the nodes of the MCT pair shows the VLAN state, its member ports and tunnels to the Leafs inside the DC1 where
the VLAN is provisioned. The VLAN is extended on the CE trunk port-channel (PO 2) connected to the DCI/VPLS Edge switch.
SLX-EdgeLeaf1# show vlan 101

(t)-Tagged
================ =============== ========================== =============== ==================== List of ports connected to DCI
101 VLAN0101 ACTIVE Po 1(t) (Po1 being ISL and Po2 is CE-
Po 2(t)
Tu 61441(t) vni 101 trunk to VPLS Edge) and
Tu 61443(t) vni 101 tunnels to each Leafs where
the VLAN is extended
Tu 61444(t) vni 101
Tu 61445(t) vni 101
SLX-EdgeLeaf1#SLX-EdgeLeaf1# show tunnel 61441

Overlay gateway "Edge-Leaf", ID 1 Tu 61441 is destined to DC1
leaf1’s vtep-ip 10.1.1.1
Outer DMAC: 609c.9f5d.7a12

Outer DMAC: 609c.9f11.336f

Outer DMAC: 609c.9f11.3370

Byte count : RX 1891000 TX 1669934
SLX-EdgeLeaf1#
MAC and ARP entries of local and remote hosts (highlighted in the output below is the host in DC2)
SLX-EdgeLeaf1# show mac-address-table vlan 101 Host 0010.9400.1b96/10.0.101.201

Type Code - CL:Cluster Local MAC CCL:Cluster Client Local MAC
CR:Cluster Remote MAC CCR:Cluster Client Remote MAC is in DC2 and is learnt from DCI on
VlanId/BDId Mac-address Type State Ports/LIF/PW the trunk Po 2
101 (V) 0010.9400.1b96 Dynamic-CCL Active Po 2
101 (V) 0011.9400.044d EVPN Active Tu 61441 (10.1.1.1)
Host 0011.9400.044d /10.0.101.101
SLX-EdgeLeaf1# show ip arp suppression-cache vlan 101
Flags: L - Locally Learnt Adjacency is behind the Leaf on local fabric of
R - Remote Learnt Adjacency DC1
---------------------------------------------------------------------------------------------------
0101 (V)
0101 (V)
10.0.101.101
10.0.101.201
0011.9400.044d Tu 61441 (10.1.1.1)
0010.9400.1b96 Po 2
Never
Never
R
R
Arp suppression-cache shows there
reachability.
SLX-EdgeLeaf2# show mac-address-table vlan 101 | inc 1b96|044d
101 (V) 0010.9400.1b96 Dynamic-CCL Active Po 2
101 (V) 0011.9400.044d EVPN Active Tu 61441 (10.1.1.1)
SLX-EdgeLeaf2# show ip arp suppression-cache vlan 101 | inc 1b96|044d

0101 (V) 10.0.101.101 0011.9400.044d Tu 61441 (10.1.1.1) Never R
0101 (V) 10.0.101.201 0010.9400.1b96 Po 2 00:09:16 L
101
DCI VPLS/Edge
Cluster, Bridge-domain and Pseudowire
DCI-9540-1# show bridge-domain 101
Bridge-domain 101
-------------------------------
Bridge-domain Type: MP, VC-ID: 101 MCT Enabled: TRUE
Number of configured end-points: 3, Number of Active end-points: 2
VE if-indx: 0, Local switching: TRUE, bpdu-drop-enable: TRUE
MAC Withdrawal: Disabled
PW-profile: vpls, mac-limit: 0
Un-tagged Ports:
Total VPLS peers: 1 (1 Operational):
VC id: 101, Peer address: 66.1.1.1, State: Operational, uptime: 12 min 20 sec
Load-balance: True, Cos Enabled: False,
Tunnel cnt: 1
rsvp to-MLX-DC2 (cos_enable:False cos_value:0)
Assigned LSPs count:0 Assigned LSPs:
Local VC lbl: 983116, Remote VC lbl: 983121,
Local VC MTU: 9100, Remote VC MTU: 9100,
Local VC-Type: 5, Remote VC-Type: 5
Local PW preferential Status: Standby, Remote PW preferential Status: Active
DCI-9540-2# show bridge-domain 101 VPLS PW’s works in

Bridge-domain 101
------------------------------- active/standby mode on an
Bridge-domain Type: MP, VC-ID: 101 MCT Enabled: TRUE MCT cluster & hence
Number of configured end-points: 3, Number of Active end-points: 3
VE if-indx: 0, Local switching: TRUE, bpdu-drop-enable: TRUE
Pseudowire is active only on
MAC Withdrawal: Disabled one of the cluster nodes.
PW-profile: vpls, mac-limit: 0
Un-tagged Ports:
Total VPLS peers: 1 (1 Operational):
VC id: 101, Peer address: 66.1.1.1, State: Operational, uptime: 27 min 30 sec
Load-balance: True, Cos Enabled: False,
Tunnel cnt: 1
rsvp to-MLX-DC2 (cos_enable:False cos_value:0)
Assigned LSPs count:0 Assigned LSPs:
Local VC lbl: 983134, Remote VC lbl: 983110,
Local VC MTU: 9100, Remote VC MTU: 9100,
Local VC-Type: 5, Remote VC-Type: 5
Local PW preferential Status: Active, Remote PW preferential Status: Active
DCI-9540-2# show cluster 5556

Cluster DCI-cluster 5556
================
Cluster State: Deployed
Configured Member Vlan Range:
Active Member Vlan Range:
Configured Member BD Range: 61-70,101-130,161-190
Active Member BD Range: 61-70,101-130,161-190
No. of Peers: 1
No. of Clients: 2
Peer Info:
==========
Peer Interface: Vlan 4090
Client Info:
============
---- ---- ----------- --------- ------------------
Edge-Leaf 1 0:0:0:0:0:0:0:1:1 Port-channel 2 Up / Up
Client-PW 34816 0:0:0:0:0:0:0:2:2 PW Up / Up
Local and Remote MAC entries
DCI-9540-2# show mac-address-table bridge-domain 101 Host 0010.9400.1b96 behind DC2 fabric is
BDId Mac-address Type State Ports/LIF/PW
101 (B) 0010.9400.1b96 Dynamic-CL Active 66.1.1.1 learned ove the VPLS Pseudowire.
101 (B) 0011.9400.044d Dynamic-CCL Active Po 2.101
DCI-9540-1# show mac-address-table bridge-domain 101 | inc 2046|44d|1b96 Host 0011.9400.044d behind DC1 fabric is
101 (B) 0010.9400.1b96 CR Active 10.56.56.1 learned over the port-channel logical
101 (B) 0011.9400.044d CCR Active Po 2.101
interface between this DCI node and border-
leaf.
102
Layer-3 Handoff with MPLS/L3VPN
This use-case illustrates Layer-3 extension between two EVPN fabrics over MPLS/L3VPN network. This use case may also be extended to
interconnecting an EVPN fabric to a traditional multi-tenant L3 network.
If both fabrics used consistent L3 VNI range i.e. VNI X maps to same L3 tenant VRF on both fabrics, BGP EVPN based fabric extension would be an
appropriate option.
As shown in the topology below, a pair of switches in MCT pair act as border-leaf that terminate the VxLAN encapsulated packets from the EVPN fabric
and hands off Layer-3 routed packets to the MPLS edge devices which act as L3VPN PE routers. Similarly, on the other direction, these border-leafs
map and encapsulate the Layer-3 routed packets into appropriate tenant L3 VNI segments towards the EVPN fabrics.
The following devices are used in this illustration and these are part of the validated design.
PIN Platform
Edge/Border-Leaf SLX 9140 MCT pair
MPLS Edge SLX 9540 MCT pair
Leaf SLX 9140 MCT pair
Note that only the relevant portion of the validated design topology is shown in the diagram and configuration section will include only the relevant
and incremental configuration needed for the use-case. For building the Fabric Underlay and Overlay, please refer to the validated design section.
Figure 22 Layer-3 Handoff with MPLS/L3VPN
MPLS
Super-spines
DCI-1 DCI-2 Spines

S i
DC1 DCI Edge DC2 DCI Edge

AS #64511
Leaf1-1 Leaf1-2
Leaf1-1 Leaf1-2
Po 211 Po 212 Edge-leaf Pair Edge Leaf Pair
AS #4200007000
DC1 DC2
Configuration
Following are the configuration steps involved:
• Provision the tenant VRFs on the border-leaf pair. This is similar to tenant VRF provisioning on the leaf in the EVPN/VxLAN fabric.
103
• Provision the tenant VRFs on the DCI edge pair. This pair acts as PE edge for L3VPN.
• Provision IPv4 and IPv6 interfaces between the border-leafs and DCI edge nodes in the tenant L3 VRF context.
o VLAN and VE interfaces are used for this connectivity.
o Each node in the border-leaf MCT pair is connected to both DCI edge nodes by L2 trunk ports.
o Each trunk will carry a separate VLAN (for isolation) and has an associated VE interface inside the tenant VRF.
o This will be repeated for every tenant VRF on each of the MCT nodes.
• Enable BGP peering to DCI nodes inside the tenant VRF context. This exchanges the IPv4 and IPv6 routes to/from the DCI nodes. DCI node
will also have a VRF for the tenant VRF. (Border-leaf acts as CE and DCI node as PE. They exchange routes into the tenant VRFs on each side).
This is also known as vrf-lite peering. Each tenant VRF will have a separate BGP peering session with each of the DCI nodes.
• Set up MPLS/L3VPN on the DCI nodes (acting as L3VPN PE routers)
SLX-9140 MCT Pair Border-Leaf
mtu 9216 mtu 9216

! !
! !
! !
vlan 4090 vlan 4090
! !
! !
! !
104
cluster management node-id 1 cluster management node-id 2
node-id 1 !
cluster management principal-priority 1
!
cluster Edge-Leaf 6162 cluster Edge-Leaf 6162
peer 10.61.62.1 peer 10.61.62.0
deploy deploy
client DCI 1 client DCI 1
deploy deploy
! !
local-as 4200007000 local-as 4200007000
network 10.61.1.1/32 network 10.61.1.1/32
! !
! !
overlay-gateway Edge-Leaf
map vni auto
activate
!
105
vrf vrf1 vrf vrf1
rd 10.61.1.11:1 rd 10.61.1.12:1
! !
! !
! !
vlan 2601 vlan 2603
vlan 2602 vlan 2604

! !
ipv6 address fdf8:10:55:61::0/127 ipv6 address fdf8:10:55:62::2/127
! !
! !
description link to DCI-1 for vrf-lite handoff description link to DCI-1 for vrf-lite handoff
! !
description link to DCI-2 for vrf-lite handoff description link to DCI-2 for vrf-lite handoff
! !
local-as 4200007000 local-as 4200007000
! !
neighbor fdf8:10:55:61::1 remote-as 64511 neighbor fdf8:10:55:62::3 remote-as 64511
neighbor fdf8:10:55:61::1 activate neighbor fdf8:10:55:62::3 activate
! !
106
SLX-9540 MCT Pair DCI Edge
IGP and MPLS configuration
router isis router isis
net 49.0001.0010.0055.5501.00 net 49.0001.0010.0056.5601.00
fast-flood 5 fast-flood 5
is-type level-1 is-type level-1
log adjacency log adjacency
lsp-gen-interval 5 lsp-gen-interval 5
lsp-refresh-interval 64000 lsp-refresh-interval 64000
max-lsp-lifetime 65535 max-lsp-lifetime 65535
partial-spf-interval 5000 150 300 partial-spf-interval 5000 150 300
spf-interval level-1 5 150 300 spf-interval level-1 5 150 300
metric-style wide level-1 metric-style wide level-1
default-metric 10 default-metric 10
! !
! !
description BL to MPLS cloud description BL to MPLS cloud
isis auth-mode md5 level-1 isis auth-mode md5 level-1
isis auth-key level-1 $9$BwrsDbB+tABWGWpINOVKoQ== isis auth-key level-1 $9$BwrsDbB+tABWGWpINOVKoQ==
isis point-to-point isis point-to-point
! !
107
The MPLS tunneling configuration is beyond the scope of this document. A sample configuration is provided below for completeness.
router mpls router mpls
policy policy
traffic-engineering isis level-1 traffic-engineering isis level-1
ingress-tunnel-accounting ingress-tunnel-accounting
transit-session-accounting transit-session-accounting
! !
ldp ldp
lsr-id 55.1.1.1 lsr-id 56.1.1.1
! !
dynamic-bypass dynamic-bypass
enable-all-interfaces enable-all-interfaces
! !
mpls-interface ethernet 0/48 mpls-interface ethernet 0/48
! !
mpls-interface ve 4090 mpls-interface ve 4090
! !
path to-DCI_2 path to-DCI_1
! !
path to-MLX-1 path to-MLX
! !
lsp to-DCI_2 lsp to-DCI_1
to 56.1.1.1 to 55.1.1.1
primary-path to-DCI_2 primary-path to-DCI_1
adaptive adaptive
frr frr
! !
enable enable
! !
lsp to-MLX-DC2 lsp to-MLX-DC2
to 66.1.1.1 to 66.1.1.1
primary-path to-MLX-1 primary-path to-MLX
adaptive adaptive
frr frr
! !
enable enable
! !
108
VRF-lite peering between Border-leaf and DCI nodes

description link to EdgeLeaf1 for vrf-lite handoff description link to EdgeLeaf1 for vrf-lite handoff
! !
description link to EdgeLeaf2 for vrf-lite handoff description link to EdgeLeaf2 for vrf-lite handoff
! !
vlan 2601 vlan 2602
vlan 2603 vlan 2604
! !
! !
! !
vrf vrf1 vrf vrf1
rd 100:1 rd 100:1
route-target export 100:1 route-target export 100:1
route-target import 100:1 route-target import 100:1
! !
route-target export 100:1 route-target export 100:1
route-target import 100:1 route-target import 100:1
! !
neighbor 10.55.62.2 activate maximum-paths 8 neighbor 10.56.62.2 activate maximum-paths 8
! !
! !
109
neighbor 66.1.1.1 update-source loopback 2 neighbor 66.1.1.1 update-source loopback 2
! !
address-family vpnv4 unicast address-family vpnv4 unicast
neighbor 66.1.1.1 send-community extended neighbor 66.1.1.1 send-community extended
! !
address-family vpnv6 unicast address-family vpnv6 unicast
neighbor 66.1.1.1 send-community extended neighbor 66.1.1.1 send-community extended
! !
Verification
Edge/Border-Leaf MCT Pair
The verification outputs are shown from one of the nodes in the MCT pair.

(t)-Tagged
================ =============== ========================== =============== ====================
2601 VLAN2601 ACTIVE Eth 0/17(t) Vlan 2601 & 2602 are two vlans
towards the two DCI edge
VLAN Name State Ports Classification devices
(t)-Tagged
================ =============== ========================== =============== ====================
2602 VLAN2602 ACTIVE Eth 0/18(t)
SLX-EdgeLeaf1# show ip int br | inc 260

Ve 2601 10.55.61.0 vrf1 up up
Ve 2602 10.56.61.0 vrf1 up up
SLX-EdgeLeaf1# show ip bgp summ vrf vrf1
BGP4 Summary Two v4 & v6 BGP neighbors are
Router ID: 10.55.61.0 Local AS Number: 4200007000 formed towards the two DCI
Confederation Peers: Edge devices.
Number of Attribute Entries Installed: 16, Uses 1824 bytes
10.55.61.1 64511 ESTAB 19h31m12s 30 0 35 0
10.56.61.1 64511 ESTAB 19h31m 0s 30 0 65 0
SLX-EdgeLeaf1# show ipv6 bgp summ vrf vrf1

BGP4 Summary
Number of Attribute Entries Installed: 12, Uses 1368 bytes
fdf8:10:55:61::1 64511 ESTAB 19h31m44s 30 0 35 0
fdf8:10:56:61::1 64511 ESTAB 19h31m11s 30 0 65 0
110
SLX-EdgeLeaf1# show ip bgp vrf vrf1
Total number of BGP Routes: 189
Status codes: s suppressed, d damped, h history, * valid, > best, i internal, S stale 10.0.131.0/24 are behind
Origin codes: i - IGP, e - EGP, ? - incomplete Leafs on local fabric
Network Next Hop RD MED LocPrf Weight Path
*> 10.0.131.0/24 10.1.1.1 none 100 0 4200003000 4200000000 4200000001 ?
* 10.0.131.0/24 10.1.1.1 none 100 0 4200003000 4200000000 4200000001 ?
*> 10.1.75.0/24 10.55.61.1 none 100 0 64511 4200000200 4200000201 ?
* 10.1.75.0/24 10.56.61.1 none 100 0 64511 4200000200 4200000201 ?
*i 10.1.75.0/24 10.61.62.1 none 100 0 64511 4200000200 4200000201 ? 10.1.75.0/24 is behind DCI
SLX-EdgeLeaf1# show ip route 10.1.75.0/24 vrf vrf1
IP Routing Table for VRF "vrf1" edge or DC2 and is learned
Total number of IP routes: 87 over the vrf-lite peers
10.1.75.0/24
*via 10.55.61.1, Ve 2601, [20/0], 19h37m, eBgp, tag 0
*via 10.56.61.1, Ve 2602, [20/0], 19h37m, eBgp, tag 0 Route info shows vrf-lite &
SLX-EdgeLeaf1# show ip route 10.0.131.0/24 vrf vrf1 EVPN routes
10.0.131.0/24
*via 10.1.1.1%default-vrf, Ve 3001, [20/0], 19h58m, eBgp, tag 0, (VNI 7097, GW MAC 609c.9fb0.d801, Tu 61441)
*via 10.1.1.1%default-vrf, Ve 3001, [20/0], 19h58m, eBgp, tag 0, (VNI 7097, GW MAC 609c.9fb0.f501, Tu 61441)
SLX-EdgeLeaf1#
DCI MPLS/L3VPN Edge

DCI-9540-1# show ip route 10.0.131.0/24 vrf vrf1
Route info shows
10.0.131.0/24
*via 10.55.61.0, Ve 2601, [20/0], 1h17m, eBgp, tag 0
10.0.131.0/24 is learned from
*via 10.55.62.2, Ve 2603, [20/0], 1h17m, eBgp, tag 0 Border-Leafs of DC1
DCI-9540-1# show ip route 10.1.75.0/24 vrf vrf1
10.1.75.0/24 is learned over
10.1.75.0/24 L3VPN network with the next-
*via DIRECT, Lsp 2, [200/0], 1d23h, iBgp+(default-vrf), tag 0
hop being the mpls tunnel
DCI-9540-1# show mpls lsp
Note: LSPs marked with * are taking a Secondary Path
LSP To Admin Oper Tunnel Up/Dn Retry Active
Name Address State State Intf Times Num Path
MCT_10.56.56.1_ 10.56.56.1 UP UP tnl7 1 0 --
1207963642
to-MLX-DC2 66.1.1.1 UP UP tnl2 1 0 to-MLX-1
DCI-9540-1#
111
Design Considerations
BGP Route scale considerations
In the overlay or EVPN fabric, following route types are used for the workloads:
• Host MAC only route
• Host MAC + IP ARP route
• Host MAC + IPv6 ND route
• Host Subnet or Prefix route for IPv4
• Host Subnet of Prefix route for IPv6
• IMR Route per VLAN or VNI
When a dual-stack host is provisioned, 3 routes are inserted into BGP EVPN Address-family: a MAC only route for the host’s MAC address, an ARP or
MAC/IP route and a ND or MAC/IPv6 route. On top of this, a Prefix route for the Vlan Subnet on which the host resides.
On a typical Rack of 2000 VMs or hosts – assuming all of them support dual-stack – it is expected to insert 6000 BGP EVPN entries apart from the host
subnets (2 * Subnets – one for IPv4 and one for IPv6).
These routes are sent to Spines. The Spines will send the routes to other Leafs. In this case, each leaf will be receiving a number of copies equal to the
number of Spines.
When same VLANs/VNIs or tenant VRFs are not extended or provisioned on another ToR, the associated host routes and subnet routes will be filtered
out based on the route targets and therefore won’t be in the BGP RIB.
These are certain considerations one must take into account when designing the fabric. The network designers and architects are encouraged to
consult the verified scale parameters of the platforms used in the fabric.
For large scale DC fabrics, one may also consider using only two spines to exchange EVPN routes instead of all spines. This will reduce the number of
copies of BGP paths on the Leaf nodes. Similarly, two Super-spines may be designated to exchange EVPN routes.
BGP TTL Security

This is applicable for eBGP peering only. This configuration can be applied to a specific neighbor or a peer group.
112
Appendix1: VDX Leaf configuration
This section provides the configuration steps for VDX platforms used as Leaf in the BGP EVPN fabric.
Node ID Configuration
The VDX platforms used as leaf, spine, and super-spine nodes are enabled with VCS ID 1 by default. Since these nodes will be independent in IP fabric,
we need to ensure that they do not form a VCS fabric between them. This is achieved by configuring a unique VCS ID on each node.
In the validated design, each node—spine, leaf, super-spine, and edge leaf—is configured with a unique VCS ID. The RBridge ID may be re-used. We
recommend using rbridge-id 1 for individual leafs and rbridge-ids 1 and 2 for vLAG pair.
!Note: this is done from the exec prompt

POD1-Leaf5# vcs vcsid 409 set-rbridge-id 1
Enable Virtual-Fabric on all leafs and edge leafs:
POD1-Leaf5(config)# vcs virtual-fabric enable
The vLAG pair is assigned its own unique VCS ID, and each node in the vLAG pair has a separate RBridge ID. For example, in the validated design, Leaf1
is a 2-node vLAG pair.
vLAG peer 1:
POD1-Leaf1-1# vcs vcsid 1 set-rbridge-id 1
vLAG peer 2:
POD1-Leaf1-2# vcs vcsid 1 set-rbridge-id 2
Verify the configuration:
POD1-Leaf1-1# show vcs

Config Mode : Distributed
VCS Mode : Logical Chassis
VCS ID : 1
VCS GUID : c98c32fb-1472-4cc5-a7fa-aeb8570627f9
Total Number of Nodes : 2
Rbridge-Id WWN Management IP VCS Status Fabric Status HostName
--------------------------------------------------------------------------------------------------------------
1 10:00:00:27:F8:F0:76:E0* 10.20.34.45 Online Online POD1-Leaf1-1
2620:100:0:fa48:34::45
2 >10:00:00:27:F8:F0:5B:B0 10.20.34.46 Online Online POD1-Leaf1-2
2620:100:0:fa48:34::46
From the primary node of the vLAG pair, enable virtual-fabric. For instance, as shown above, the Rbridge 2 is the primary node in the Leaf1 vLAG pair.
POD1-Leaf1-2(config)# vcs virtual-fabric enable
113
All nodes in the IP fabric—leafs, spines, and super-spines—are interconnected with Layer 3 interfaces. In the validated design,
• 40-G links are used between the nodes.
• The MTU for these links is set to Jumbo MTU. This is a requirement to handle the VXLAN encapsulation of Ethernet frames.
Disable the fabric ISL and trunk features.
interface FortyGigabitEthernet 1/0/97

mtu 9216
no fabric isl enable
no fabric trunk enable
ip mtu 9100
ip address 10.11.1.1/31
no shutdown
Loopback Interfaces
Each device in the fabric needs one loopback interface with a unique IPv4 address for the purpose of Router-ID.
rbridge-id 1
no shutdown
ip address 10.121.1.11/32
Configure the IP Router-ID using the IP address of the loopback 2 interface.
rbridge-id 1
Each leaf needs a loopback interface with a unique IPv4 address to use as VTEP-IP.
rbridge-id 1
no shutdown
ip address 10.121.1.1/32
vLAG Pair/ToR
vLAG configuration involves a few additional configuration steps apart from those shown above.
• Node ID configuration on the pair of devices.
• Inter-switch links or ISL configuration on both devices.
• Loopback interfaces for router-id and VTEP-IP. The pair act as one logical VTEP, so will share a common VTEP IP address. However each node
in the pair will have a separate router-id.
• Fabric link configuration. This is same as the configuration shown for individual Leaf/ToR. Each node in the pair will have separate links
connecting the Spines.
• Configure the server-facing port channels, and add the required VLANs on them.
114
Node ID Configuration on vLAG Pair
Refer to the Node ID configuration section for assigning the Node ID to the vLAG pair.
• Pod1-Leaf1-1, rbridge-id 1
• Pod1-Leaf1-2, rbridge-id 2
POD1-Leaf1-1# show vcs

Config Mode : Distributed
VCS Mode : Logical Chassis
VCS ID : 1
VCS GUID : c98c32fb-1472-4cc5-a7fa-aeb8570627f9
Total Number of Nodes : 2
Rbridge-Id WWN Management IP VCS Status Fabric Status HostName
--------------------------------------------------------------------------------------------------------------
1 >10:00:00:27:F8:F0:76:E0* 10.20.34.45 Online Online POD1-Leaf1-1
2620:100:0:fa48:34::45
2 10:00:00:27:F8:F0:5B:B0 10.20.34.46 Online Online POD1-Leaf1-2
2620:100:0:fa48:34::46
ISL Configuration
As shown in the illustration below, the vLAG pair is interconnected by two 40G Ethernet ports for ISL.
Server Port-Channel Configuration

A typical vLAG configuration is shown below. This is also part of the tenant provisioning.


Rbridge 1 Rbridge 2
Te 1/0/7 ISL Links Te 2/0/7
vLAG

vlag ignore-split vlag ignore-split
spanning-tree shutdown spanning-tree shutdown
! !
interface TenGigabitEthernet 1/0/7 interface TenGigabitEthernet 2/0/7
lacp timeout long lacp timeout long
! !
115
Loopback interfaces
Each node has a unique router-id. But both of them share a common VTEP-IP and act as a logical VTEP. In the configuration below, loopback-2
interface has same IP address on both nodes.
! !
! !
! !
BGP Configuration
For VDX platforms, BGP configuration is done in the rbridge configuration mode. For vLAG pair, the BGP configuration is done for each rbridge node.
Configuration for an individual node is shown below. This must be replicated with the appropriate neighbor IP addresses of the Spines for a dual or
VLAG ToR. Note that the leaf negotiates and exchange EVPN-AFI with only two spines.
!POD1-leaf1-1
rbridge-id 1
router bgp
local-as 4200000001
!
neighbor spine-evpn-group peer-group
neighbor spine-evpn-group remote-as 4200000000
neighbor spine-evpn-group password 2 $PVNHITJVPWQ=
neighbor spine-evpn-group bfd
!
neighbor 10.12.1.0 peer-group spine-evpn-group
neighbor 10.13.1.0 peer-group spine-evpn-group
!
neighbor spine-ip-group peer-group
neighbor spine-ip-group remote-as 4200000000
neighbor spine-ip-group password 2 $PVNHITJVPWQ=
neighbor spine-ip-group bfd
!
neighbor 10.11.1.0 peer-group spine-ip-group
neighbor 10.14.1.0 peer-group spine-ip-group
!
network 10.121.1.1/32
maximum-paths 8
graceful-restart
!
graceful-restart
neighbor spine-evpn-group activate
neighbor spine-evpn-group allowas-in 1
neighbor spine-evpn-group next-hop-unchanged
!
Tenant Provisioning
Tenant provisioning refers to the configuration on leafs to enable server VLANs and networks connectivity to tenant VRF contexts and mapping these
VLANs and VRFs to the overlay control and forwarding planes to establish Layer 2 extension and multitenancy. This is applicable to both 3-stage and 5-
stage Clos fabrics.
116
Anycast Gateway MAC Configuration
Anycast gateway MAC configuration is applied to all leafs (except edge leafs) in the data center. This is used as the gateway MAC or router-mac for all
server facing subnets. This enables seamless workload move within and across the PoDs in the data center. It is recommended to set the U/L bit is set
to 1 in the mac-address to indicate that it is a locally administered MAC address and not to conflict with any real MAC addresses.
The MAC addresses must be different for IPv4 and IPv6, but the OUI portion (first three bytes) must be same.
rbridge-id 1
!
rbridge-id 1
!
Enable Conversational Learning of MAC and ARP/ND Host Entries

mac-address-table learning-mode conversational
rbridge-id 1
host-table aging-mode conversational
!
VRFs, Server VLANs, and Subnets Configuration

Following are the steps involved in tenant VRF configuration.
1. Assign a unique RD. Every tenant must have a unique RD value per leaf/ToR where it is provisioned. In the validated design, we are using the
following format: IPv4_Address:nn where
• IPv4_Address is the router-id of the VTEP and
• nn is a unique number for the tenant VRF. This value is re-used on other leafs as well where the same tenant is provisioned.
For example, vrf201 has the following RD values on leafs where it is provisioned.
o On leaf1: 10.121.1.11:201
o On leaf5: 10.121.1.51:201
o On border-leaf1: 10.123.4.1:201
2. Assign a unique L3 VNI number.
3. Assign import and export route-targets for IPv4 and IPv6 tenant routes.
117
In the configuration templates below, the following Tenant profile is enabled on a leaf:
Configure Tenant VRF Profile:
Name: vrf101
L3 VNI: 7101
IPv4 and IPv6: enabled
Route-target 101:101
Server-facing VLAN 2001
RD is route-distinguisher unique
rbridge-id 45 per tenant
vrf vrf101
rd 10.121.1.11:101
! VNI is the unique L3 VNI required
vni 7101 for each tenant for symmetric
! routing
route-target import 101:101 evpn Route target for export and
! import of tenant IPv4 and IPv6
address-family ipv6 unicast routes. This is similar to
route-target export 101:101 evpn MPLS/VPNs
!
!
!
Assign Layer 3 Interface for the L3 VNI of the Tenant VRF:
This is the routing interface for the Integrated Routing-Bridging (IRB) operation on the leaf.
L3 VNI number configured under the tenant

rbridge-id 45 vrf – vrf101, vni 7101
interface Ve 7101
ipv6 address use-link-local-only Associate with the correct tenant vrf –
no shutdown vrf101
!
!
Enable V6 forwarding on this interface
Create Server-Facing VLAN:
interface Vlan 2001

description VLAN 2001, VNI 2001, Tenant vrf101; extended to POD2
!
Enable the VLAN on Server port:
interface TenGigabitEthernet 1/0/4 Enabled as a trunk port

switchport Add the required VLANs to the trunk port
no shutdown
118
Assign VE (L3) Interface for the Server-Facing VLAN:
rbridge-id 45
interface Ve 2001
vrf forwarding vrf101 VLAN subnet belongs to the tenant vrf101
!
ipv6 anycast-address fd2d:d47f:107:1::254/64
ipv6 nd cache expire 270
! IPv6 anycast gw address. Same address on all
ip anycast-address 10.107.1.254/24 leafs where this VLAN is extended
ip arp-aging-timeout 4 IPv6 neighbor cache timeout set to < 5 mins (
! mac aging-time of 5 mins)
no shutdown
!
!
IPv4 anycast gw address. Same address on all
Leafs where this vlan is extended
IPv4 ARP cache timeout set to < 4 mins ( mac
aging-time of 5 mins)
Advertise Tenant Layer 3 Routes from the Leaf

IPv4
rbridge-id 45 Activate layer3 IPv4 routes exchange from the

router bgp tenant
maximum-paths 8 Advertise the connected subnets inside the tenant
! vrf – vlans in this case
!
!
IPv6
Activate layer3 IPv6 routes exchange from the

rbridge-id 45 tenant
router bgp
maximum-paths 8 Advertise the connected subnets inside the
! tenant vrf – vlans in this case
!
!
Enable EVPN Instance for the Tenant VLAN Segments

Once the server facing VLANs are created and mapped to VNI segments on the leaf, those VNI segments need to be enabled into the control plane. As
was done for the tenant VRF, the VNI segments also require a RD (route-distinguisher) and RT (route-target). This is also defined as MAC-VRF and
enables learning remote mac addresses when the same VLAN segment is extended to other leafs or VTEPs in the fabric.
The RD and RT configuration is set to auto in this design for simplicity and may be followed for most of the deployments. Advanced users may define a
different scheme of RD and RT. User-defined RD/RT is not covered in this document.
119
rbridge-id 45 Route target to enable import of host routes (mac/ip) from remote
evpn-instance pod1-leaf1 VTEPS, ignore-as keyword allows downloading routes from VTEPs
! in remote AS. (as in EBGP underlay where each leaf is in a separate
route-target both auto ignore-as AS)
rd auto
! RD is set to Auto for simplified configuration
vni add 2001
!
! Enable the server vlan’s VNI
For any additional vlans, add the respective VNIs
vLAG Pair Configuration

VLAG pair or redundant ToR requires a few additional configuration steps:
• Same VTEP-IP
• Separate or unique Router-IDs
Please note that the configuration for both switches in the vLAG pair can be done from the primary node. The configuration required on each node of
Dual-ToR vLAG pair is shown in two config blocks side-by-side. The global configuration is shown in one config block.
• Loopback1 interface has unique IP address on each node, this is used as the IP Router-ID for the node.
• Loopback2 interface has the same IP address on both nodes, this is used as VETP-IP under overlay-gateway.
• Attach both RBridge IDs under the overlay gateway.
• Create the tenant VLAN and VE interface. Enable the VLAN on the server facing vLAG.
• Extend the L2 segment under the EVPN instance on each node or rbridge.
• Advertise L3 subnets of tenant VRF into BGP EVPN.
! !
! !
! !
! !
rd 10.121.1.11:101 rd 10.121.1.12:101
vni 7101 vni 7101
! !
! !
! !
! !
120
Overlay Gateway
attach rbridge-id add 1-2
map vlan vni auto
activate
!
Server facing VLAN and LAG
interface Vlan 2001

!
vlag ignore-split
switchport
no shutdown
!
interface TenGigabitEthernet 1/0/7
channel-group 113 mode active type standard
lacp timeout long
no shutdown
!
interface TenGigabitEthernet 2/0/7
channel-group 113 mode active type standard
lacp timeout long
no shutdown
!
VE interface and anycast gateway for the Server facing VLAN
ipv6 anycast-address fd2d:d47f:107:1::254/64 ipv6 anycast-address fd2d:d47f:107:1::254/64
ipv6 nd cache expire 270 ipv6 nd cache expire 270
! !
! !
Extend Tenant VLAN segment into EVPN
evpn-instance pod1-leaf1 evpn-instance pod1-leaf1
rd auto rd auto
vni add 2001 vni add 2001
! !
! !
121
Advertise Tenant Subnet routes
! !
! !
! !
! !
122
References
1. BGP MPLS-Based Ethernet VPN
https://tools.ietf.org/html/rfc7432
2. Use of BGP for routing in large-scale data centers

https://datatracker.ietf.org/doc/draft-ietf-rtgwg-bgp-routing-large-dc/
3. Integrated Routing and Bridging in EVPN

https://datatracker.ietf.org/doc/draft-ietf-bess-evpn-inter-subnet-forwarding/
4. RFC 4760: Multiprotocol Extensions for BGP-4

https://datatracker.ietf.org/doc/rfc4760/
5. RFC 4364: BGP/MPLS IP Virtual Private Networks (VPNs)

https://datatracker.ietf.org/doc/rfc4364/
6. A Network Virtualization Overlay Solution using EVPN

https://datatracker.ietf.org/doc/draft-ietf-bess-evpn-overlay/
7. Extreme Networks Data center Switching Products portal and documentation

https://www.extremenetworks.com/products/switching/data-center-switching/
https://www.extremenetworks.com/support/documentation/
123

Network Virtualization Evd

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Network Virtualization Evd

Uploaded by

Copyright:

Available Formats

EXTREME VALIDATED DESIGN

© 2017, Brocade Communications Systems, Inc. All Rights Reserved.

Technology Overview ......................................................................................................................................................................................................... 11

Validated Designs ............................................................................................................................................................................................................... 31

Use Cases ........................................................................................................................................................................................................................... 52

Appendix1: VDX Leaf configuration ................................................................................................................................................................................. 113

References ........................................................................................................................................................................................................................ 123

Purpose of This Document

This document describes the following architectures:

ARP Address Resolution Protocol

As a design principle, the following requirements apply to the leaf-spine topology:

• Spines are not interconnected with each other.

• The network endpoints do not connect to the spines.

IP Fabric Infrastructure Links

• 40-GbE links are used between the fabric nodes.

• 10/25-GbE links as trunk ports with associated VLANs.

Figure 1 Leaf-Spine L3 Clos Topology

Rack-1 Rack-2 Rack-N Edge Services

Optimized 5-Stage Layer 3 Clos Topology (Three-Tier)

• Border-leafs connect to the super-spines.

• Neither spines nor super-spines are interconnected with each other.

Edge Services and Border Leafs

Extreme IP Fabric Underlay Routing

• Resiliency feature such as BFD.

eBGP for Underlay

• All spines inside a PoD belong to one AS.

• All super-spines are configured in one AS.

• Edge or border leafs belong to a separate AS.

• Each leaf peers with all spines using eBGP.

• Each spine peers with all super-spines using eBGP.

• There is no eBGP peering between leafs.

• There is no eBGP peering between spines.

• There is no eBGP peering between super-spines.

iBGP for Underlay

VXLAN Layer 2 Extension Using Flood and Learn

Figure 5 VTEPs and L2 Extension with Flood and Learn

• H1 sends an ARP request.

• This packet is sent to every VTEP.

o Outer-IP header—dst: 10.10.10.1, src 10.10.10.2

Figure 6 Routing Between VxLAN networks in Flood and Learn topology

VTEP-A - 10.10.10.1 VTEP-B - 10.10.10.2

IP network underlay Host-B

Figure 7 VTEPs and L2 Extension with BGP EVPN Control-plane

BGP sessions between VTEPs

BGP EVPN Control Plane

EVPN Route Types

• Route Type-2—MAC/IP advertisement route:

Layer 3 VNI or Tenant VRF

Router-MAC Extended Community

Extended community type EVPN (0x06) and sub-type 0x03.

• Leaf2 learns the MAC/IP (or ARP) binding for H2.

• H1 sends an ARP request to H2. Leaf1 will respond on behalf of H2.

(2) BGP update to Leaf1 – M2/10.10.1.2;

VLAN Scoping at the Leaf Level

- VLAN 10 is mapped to VNI 10 on Leaf1

- VLAN 20 is mapped to VNI 10 on Leaf2.

L2 Extension or same Bridge-domain

VLAN Scoping at the Port Level within a Leaf

Figure 10 VLAN Scoping at the Port Level within a ToR

L2 Extension or same Bridge-domain

Integrated Routing and Bridging

Figure 12 Symmetric IRB

Extreme IRB Implementation