BRKDCT 3378 PDF

Building DataCenter
Networks with VXLAN

BGP-EVPN
Lukas Krattiger– Principal Technical Marketing Engineer
BRKDCT-3378
@CCIE21921
Session Objectives
• Focus on Data Center Networks and
Fabrics with Overlays
• Closer Look on Packet Encapsulation
(VXLAN)
• Encapsulation and Forwarding
• Underlay – the Transport for the Overlay
• Closer Look on Packet Encapsulation

(BGP EVPN)
• Control-Plane – Exchanging Information
• Optimizing the Forwarding
Session Non-Objectives
• Deep-Dive into FabricPath
• There are many Sessions and
Recordings
• Comparison between different
Orchestration and Management
Tools
• Automation Workflows or Services
Catalogs
Albert Einstein
Agenda
• Introduction to Data Center Fabrics

• VXLAN with BGP EVPN
• Overview
• Underlay
• Control & Data Plane
• Multi-Tenancy
• “Stories” and Use-Cases

• Fabric Management & Automation
6
Introduction to Data
Center Fabrics
7
Data Center “Fabric” Journey (Standalone)
STP
VPC
FabricPath
VXLAN
MAN/WA
N
FabricPath VXLAN
/BGP /EVPN
MAN/WA MAN/WA
N N
Data Center Fabric Properties
Extended Namespace
Scalable Layer-2 Domains
Integrated Route and Bridge
Multi-Tenancy
Hybrid Overlays
Inter-Pod connectivity
Overlay Based Data Center Fabrics
RR RR • Desirable Attributes:
• Mobility
• Segmentation
• Scale
• Automated & Programmable
• Abstracted consumption models
• Full Cross Sectional Bandwidth
• Layer-2 + Layer-3 Connectivity
• Physical + Virtual
10
Overlay Based Data Center: Edge Devices
Network Overlays Host Overlays Hybrid Overlays
V
V
V
V
V
V
• Router/Switch end-points • Virtual end-points only • Physical and Virtual

• Protocols for Resiliency/Loops • Single admin domain • Resiliency + Scale
• Traditional VPNs • VXLAN, NVGRE, STT • X-Organizations/Federation
• VXLAN, OTV, VPLS, LISP, FP • Open Standards
• Any subnet, anywhere, rapidly
RR RR
• Reduced Failure Domains
• Extensible Scale & Resiliency
• Profile Controlled Configuration
 Full Bi-Sectional Bandwidth (N Spines)
 Any/All Leaf Distributed Default Gateways
 Any/All Subnets on Any Leaf

Spine/Leaf Topologies
• High Bi-Sectional Bandwidth
• Wide ECMP: Unicast or Multicast
• Uniform Reachability, Deterministic
Latency
• High Redundancy: Node/Link
Failure
• Line rate, low latency, for all traffic
Variety of Fabric Sizes More Spine, More Bandwidth, More Resiliency
• Fabric size: Hundreds to 10s of
Thousands of 10G ports
• Variety of Building Blocks:
• Varying Size
• Varying Capacity
• Desired oversubscription
• Modular and Fixed
• Scale Out Architecture

• Add compute, service, external
connectivity as the demand grows
VXLAN with BGP EVPN
15
Agenda

• Overview
• Underlay
• Multi-Tenancy

16
Overview
Classic Ethernet IEEE 802.1Q
Classic Ethernet
Frame Format Frame
DMAC SMAC 802.1Q Etype Payload CRC
• Traditionally VLAN is expressed

over 12 bits (802.1Q tag) Destination MAC (DMAC)
• Limits the maximum number of Source MAC (SMAC)
segments in a Data Center to 4096 TPID TCI
VLANs 4 bytes 802.1Q 0x8100

(16 bits)
PCP CFI VID
(3 bits) (1 bits) (12 bits)
Ether Type (Etype)
Data (Payload)
CRC/FCS
VLAN ID
12 bits
TPID = Tag Protocol Identifier, TCI = Tag Control Information, PCP = Priority Code Point,
CFI = Canonical Format Indicator, VID = VLAN Identifier
Overview
Introducing VXLAN
• Traditionally VLAN is expressed
over 12 bits (802.1Q tag)
• Limits the maximum number of
segments in a Data Center to 4096
VLANs
• VXLAN leverages the VNI field with Classical Ethernet Frame DMAC SMAC 802.1Q Etype Payload CRC
a total address space of 24 bits 50 bytes Original CE Frame
VXLAN Outer IP UDP VxLAN 802.1Q CRC

• Support of ~16M segments MAC
Frame (14) (20) (8) (8)
DMAC SMAC
optional
Etype Payload
(new)
Cisco DFA
Frame
•
8 bits 24 bits 24 bits 8 bits
The VXLAN Network Identifier ags Reserved VNIVNI Reserved
(VNI/VNID) is part of the VXLAN

Header
VNI
Next-Hop MAC Address
VXLAN Frame Format Dest. MAC Address 48

Src VTEP MAC Address
MAC-in-IP Encapsulation Src. MAC Address 48
VLAN Type 14 Bytes

16
0x8100 (4 Bytes Optional) IP Header
72
Misc. Data
VLAN ID
16
Outer MAC Header Tag
Protocol 0x11 (UDP) 8
Ether Type
16
Underlay
0x0800 Header
16 20 Bytes
Checksum
Source IP 32
50 (54) Bytes of Overhead
Outer IP Header Src and Dst

addresses of the
Source Dest. IP 32
16 VTEPs
Port
UDP Header VXLAN Port 16 Hash of the inner L2/L3/L4 headers of
8 Bytes the original frame.
UDP Length 16 Enables entropy for ECMP Load
VXLAN Header UDP 4789 balancing in the Network.
Checksum 0x0000 16
Overlay
VXLAN Flags
8
RRRRIRRR
Original Layer-2 Frame Allows for 16M
possible
Reserved 24
Segments
8 Bytes
VNI 24
Reserved 8
 Extended Namespace
Scalable Layer-2 Domains
Multi-Tenancy
Understanding Overlay Technologies
Overlay Services
• Layer 2 Underlay Transport
Tunnel Encapsulation
• Layer 3 Network
• Layer 2 and Layer 3
Control Plane Data Plane

• Overlay Layer 2/Layer 3 Unicast traffic
• Peer Discovery mechanism
• Overlay Broadcast, Unknown Unicast,
• Route Learning and Distribution
Multicast traffic (BUM traffic) forwarding
– Local Learning
– Ingress Replication
– Remote Learning
– Multicast
21
Why VXLAN?
VXLAN provides a Network with
Segmentation, IP Mobility, and Scale
• “Standards” based Overlay (RFC 7348)
• Leverages Layer-3 ECMP – all links forwarding
• Increased Name-Space to 16M identifier
• Integration of Physical and Virtual
• It’s SDN 
22
VXLAN Taxonomy (1)
IP Interface Edge Device
Edge Device
Local LAN
Edge Device Segment
IP Interface Edge Device
Edge Device
Edge Device Physical Servers
Local LAN Virtual Servers

Segment
23
VXLAN Taxonomy (2)
VTEP
VTEP
Local LAN
VTEP Segment
VTEP
VTEP
VTEP Physical Servers
VTEP: VXLAN Tunnel End-Point

Local LAN Virtual Servers VNI/VNID: VXLAN Network Identifier
Segment
24
Getting the Puzzle Together!
Driving
Standards based
Overlay-
Evolution with
VXLAN BGP
EVPN
What is VXLAN with BGP EVPN?
• Standards based Overlay (VXLAN) with Standards

based Control-Plane (BGP)
• Layer-2 MAC and Layer-3 IP information distribution by
Control-Plane (BGP)
• Forwarding decision based on Control-Plane
(minimizes flooding)
• Integrated Routing/Bridging (IRB) for Optimized
Forwarding in the Overlay
• Multi-Tenancy At Scale
26
EVPN – Ethernet VPN
Control-
EVPN MP-BGP - RFC 7432
Plane
Multi-Protocol Label Switching Provider Backbone Bridges Network Virtualization Overlay

Data-
(MPLS) (PBB) (NVO)
Plane
draft-ietf-l2vpn-evpn draft-ietf-l2vpn-pbb-evpn draft-ietf-bess-evpn-overlay
 EVPN over NVO Tunnels (ie VXLAN) for Data Center

Fabric encapsulations
 Provides Layer-2 and Layer-3 Overlays over simple IP
Networks
Cisco’s VXLAN related IETF RFCs & Drafts
ID Title Category
RFC 7348 Virtual eXtensible Local Area Network Data Plane
RFC 7432 BGP MPLS based Ethernet VPNs Control Plane
draft-ietf-bess-evpn-overlay A Network Virtualization Overlay Solution using EVPN Control Plane
draft-ietf-bess-evpn-inter-subnet-forwarding Integrated Routing and Bridging in EVPN Control Plane
draft-ietf-bess-l2vpn-evpn-prefix- IP Prefix Advertisement in E-VPN Control Plane

advertisement
draft-tissa-nvo3-oam-fm NVO3 Fault Management / OAM Management Plane
28
VXLAN Evolution
• Multi-Protocol BGP (MP-BGP) based Control-Plane using
Protocol Learning
EVPN NLRI (Network Layer Reachability Information)
• Workload MAC and IP • Make Forwarding decisions at VTEPs for Layer-2 (MAC)
Addresses learnt by VXLAN and Layer-3 (IP); Integrated Route/Bridge (IRB)
Edge Devices (NVEs)
• Advertises Layer-2 and • Reduce Flooding
Layer-3 Address-to-VTEP
Association (Overlay • Reduce impact of ARP on the Network
Control-Plane)
• Flood Prevention • Standards Based (IETF draft)
• Optimized ARP forwarding
VXLAN Evolution
• Forward based on MAC or IP address learnt via Control-
IP Services
Plane (MP-BGP EVPN)
• VXLAN Routing • Make routing decisions at VTEPs
• Distributed Anycast
Gateway (requires Overlay • Scale and Multipathing (ECMP)
Control-Plane)
• Multi-Tenancy
• Leverage Layer-3 Gateway capabilities along with
Protocol Information
• LISP-ish / LISP-like approach for Host/IP Mobility
• Location (VTEP), Identifier (MAC, IP of End-Host)
Getting the Puzzle Together!
Optimized Networks with VXLAN
Underlay
Overlay
Integrated (VXLAN)
Route/Bridge
BGP
(EVPN)
http://packetpushers.net/show-233-cisco-nexus-using-bgp-
as-a-vxlan-control-plane-sponsored/
http://www.slideshare.net/robboyd/techwisetv-
workshop-secrets-of-scalable-multitenancy
http://blogs.cisco.com/cin/network-like-its-1999-with-bgp-evpn http://blogs.cisco.com/datacenter/vxlanevpn-
standards-based-overlay-with-control-plane
32
Agenda

• Overview
• Underlay
• Multi-Tenancy

33
Deployment Considerations
• MTU and Overlays
• Unicast Routing Protocol and IP
Addressing
• Multicast for BUM* Traffic
Replication
*BUM: Broadcast, Unknown Unicast & Multicast

MTU and VXLAN
• VXLAN adds 50 Bytes (or 54 Bytes)
Outer MAC Header to the Original Ethernet Frame
Underlay
• Avoid Fragmentation by adjusting
the IP Networks MTU
50 (54) Bytes of Overhead
Outer IP Header
• Data Centers often require Jumbo

UDP Header
MTU; most Server NIC do support
VXLAN Header
up to 9000 Bytes
• Using a MTU of 9216* Bytes
Overlay
accommodates VXLAN Overhead
Original Layer-2 Frame plus Server max. MTU
*Cisco Nexus 5600/6000 switches only support 9192 Byte for Layer-3 Traffic
Building your IP Network – Interface Principles (1)
• Know your IP addressing and IP Rendezvous-Point Loopback
10.254.254.1
scale requirements Routing Loopback
10.10.10.203/32
• Separate VTEP from Routing p2p Links

Protocol from RP* Loopback 10.1.1.2/30
V
• Best to use individual Aggregates
for the Underlay V
• Unicast Routing p2p** Links p2p Links V
10.1.1.1/30
• Unicast Routing Loopbacks V
• VTEP (NVE) Loopback V p2p Agg: 10.1.1.0/24
• Multicast Routing Loopback (RP) RID Agg: 10.10.10.0/24
Routing Loopback V VTEP Agg: 10.200.200.0/24
10.10.10.101/32
RP Agg: 10.254.254.0/24
• IPv4 only (today) VTEP Loopback
10.200.200.101/32
*RP: Rendezvous-Point (Multicast)
**p2p: Point-to-Point
Building your IP Network – Interface Principles (2)
• Routed Ports/Interfaces
• Layer-3 Interfaces between Spine and
Leaf (no switchport)
• For each Point-2-Point (P2P)
connection, minimum /31 required
V
• Alternative, use IP Unnumbered (/32)
• Use Loopback as Source-Interface V

V
for VTEP (NVE*) V
V
V
*NVE: Network Virtualization Edge

VTEP: VXLAN Tunnel End-Point
Building your IP Network – Some Math
Example from depicted topology:

4 Spine * 6 Leaf = 24 Point-2-Point (P2P) Links
24 Links * 2 (/31) + 10 RID* + 6 VTEP + 4 Spine
= 48 IP Addresses for P2P Links

= 20 IP Addresses for Loopback Interfaces
V
68 IP Addresses required == /25 Prefix V

V
A More Realistic Scenario: V
4 Spine * 40 Leaf = 160 Point-2-Point (P2P) Link
160 Links * 4 (/30) + 44 RID* + 80 VTEP + 4 Spine V
V
= 640 IP Addresses for P2P Links
= 128 IP Addresses for Loopback Interface
768 IP Addresses required == /22 Prefix *RID: Router ID; Unicast Routing Loopback
IP Unnumbered– Simplifying the Math
Example from depicted topology:

4 Spine + 6 Leaf = 10 Individual Devices
= 6 IP Addresses for Loopback Interface (Used for VTEP)

= 10 IP Address Loopback Interface (RID* & IP Unnumbered)
V
16 IP Addresses required == /28 Prefix
V
V
A More Realistic Scenario: V
4 Spine + 40 Leaf = 44 Individual Devices
V
= 40 IP Addresses for Loopback Interface (Used for VTEP) V
= 44 IP Addresses for Loopback Interface (RID* & IP Unnumbered)
84 IP Addresses required == /25 Prefix

*RID: Router ID; Unicast Routing Loopback
Building your IP Network – Routing Protocols; OSPF
• OSPF – watch your Network type!
• Network Type Point-2-Point (P2P)
• Preferred (only LSA type-1)
• No DR/BDR election
• Suits well for routed interfaces/ports
(optimal from a LSA Database V
perspective)
• Full SPF calculation on Link Change V
• Network Type Broadcast V
• Suboptimal from a LSA Database V
perspective (LSA type-1 & 2) V
• DR/BDR election V
• Additional election and Database
Overhead
Building your IP Network – Routing Protocols; IS-IS
• IS-IS – what was this CLNS?
• Independent of IP (CLNS)
• Well suited for routed interfaces/ports
• No SPF calculation on Link change;
only if Topology changes V
• Fast Re-convergence
• Not everyone is familiar with it V
V
V
V
V
*CLNS: Connection-Less Network Service

Building your IP Network – Routing Protocols; eBGP
• eBGP – Service Provider style
• Two Different Models
• Two-AS
• Multi-AS
• BGP is a Distance Vector
V
• AS* are used to calculate the Path
(AS_Path)
V
• If Underlay is eBGP, your Overlay
V
becomes eBGP
V
V
V
*AS: Autonomous System

• eBGP – TWO-AS, yes it works!
• Total of 8 eBGP Peering (with 4
Spine)
• eBGP peering for Underlay-Routing based
on physical interface
AS#65500
• 4 Spines = 4 BGP Peering per Leaf V
• Advertise all Infrastructure Loopbacks
• eBGP peering for Overlay-Routing V
(EVPN) V
• Loopback to Loopback Peering V
• 4 Spines = 4 BGP Peering V
• Requires some BGP config knobs V
• Disable BGP AS-Path check
• Next-Hop needs to be Unchanged
• Retain all Routes on Spine (not a RR)
• eBGP – Multi-AS
• Total of 8 eBGP Peering (with 4
Spine)
• eBGP peering for Underlay-Routing based
on physical interface
AS#65500
• 4 Spines = 4 BGP Peering per Leaf V
• Advertise all Infrastructure Loopbacks
• eBGP peering for Overlay-Routing V
(EVPN) V
• Loopback to Loopback Peering V
• 4 Spines = 4 BGP Peering V
• Requires some BGP config knobs V
• Next-Hop needs to be Unchanged
• Retain all Routes on Spine (not a RR)
Multicast Enabled Underlay
May use PIM-ASM or PIM-BiDir (Different hardware has different capabilities)
ASR 1000
Nexus 1000v Nexus 3000 Nexus 5600 Nexus 7000/F3 Nexus 9000 ASR 9000
CSR 1000
Multicast Mode IGMP v2/v3 PIM ASM PIM BiDir PIM ASM / PIM BiDir PIM ASM PIM BiDir PIM ASM / PIM BiDir
• Spine and Aggregation Switches make good Rendezvous-Point (RP) Locations in

Topologies
• Reserve a range of Multicast Groups (Destination Groups/DGroups) to service the Overlay
and optimize for diverse VNIs
• In Spine/Leaf topologies with lean Spine
• Use multiple Rendezvous-Point across the multiple Spines
• Map different VNIs to different Rendezvous-Point for simple load balancing measure
• Use Redundant Rendezvous-Pint
• Design a Multicast Underlay for a Network Overlay, Host VTEPs will leverage this Network
45
Multicast Enabled Underlay – PIM ASM*
• PIM Sparse-Mode (ASM)
• Redundant Rendezvous-Point using RP
PIM Anycast-RP or MSDP
• Source-Tree or Unidirectional V
Shared-Tree (Source-Tree shown)
• Shared-Tree will always use RP for V
forwarding V
V
• 1 Source-Tree per Multicast-Group V
per VTEP (each VTEP is Source & VTEP1 (S,G) Tree
V
VTEP2 (S,G) Tree
Receiver)
RP Rendezvous-Point
*ASM: Any-Source Multicast
Multicast Enabled Underlay – BiDir-PIM*
• Bidirectional PIM (BiDir)
• Redundant Rendezvous-Point using RP
Phantom-RP
• Building Bi-Directional Shared-Tree V
• Uses shortest path between Source
and Receiver with RP as routing- V
vector V
V
• 1 Shared-Tree per Multicast-Group V
V
VTEPs (*,G) Tree
RP Rendezvous-Point
*BiDir-PIM: Bidirectional PIM
To Remember - Multicast Enabled Underlay
• Multi-Destination Traffic (Broadcast, Unknown Unicast, etc.) needs to be

replicated to ALL VTEPs serving a given VNI
• Each VTEP is Multicast Source & Receiver
• For a given VNI, all VTEPs act as a Sender and a Receiver
• Head-End Replication will depend on hardware scale/capability
• Resilient, efficient, and scalable Multicast Forwarding is highly desirable
• Choose the right Multicast Routing Protocol for your need (type/mode)
• Use redundant Multicast Rendezvous Points (Spine/Aggregation generally preferred)
• 99% percent of Overlay problems are in the Underlay (OTV experience)
48
Agenda

• Overview
• Underlay
• Multi-Tenancy

49
Multiprotocol BGP (MP-BGP) Primer
• Multiprotocol BGP (MP-BGP)
• Extension to Border Gateway
RR RR
Protocol (BGP) - RFC 4760
• VPN Address-Family:
• Allows different types of address
families (e.g. VPNv4, VPNv6, L2VPN V2
EVPN, MVPN) V1
• Information transported across single
BGP peering
RR BGP Route-Reflector
V3 iBGP Peering*
*eBGP supported without BGP Route-Reflector
• VPN segmentation for tenant routing VRF Info
(Multi-Tenancy) VRF Info Name: VRF-A
Name: VRF-A RD: 15:10.0.0.2 (auto)
• Route Distinguisher (RD) RD: Imp
3:10.0.0.1 (auto) RRRoute-Target
RR 65500:50000 (auto)
Imp Route-Target 65500:50000 Exp Route-Target 65500:50000 (auto)
• 8-byte field of VRF parameters VRF Info (auto)
Exp Route-Target 65500:50000
Name: (auto)VRF-A
• value to make VPN prefix unique: RD: 62:10.0.0.3 (auto)
• RD + VPN prefix Imp Route-Target 65500:50000 (auto)
Exp Route-Target 65500:50000 (auto)
V1 V2
V3 iBGP Peering
• Cisco’s VXLAN/EVPN does provide VRF Info
automated Route Distinguisher (RD) VRF Info Name: VRF-A
Name: VRF-A RD: 15:10.0.0.2 (auto)
• Automatic uses Type 1 format RD: Imp
3:10.0.0.1 (auto) RRRoute-Target
RR 65500:50000 (auto)
Imp Route-Target 65500:50000 Exp Route-Target 65500:50000 (auto)
• 4-byte IP Address (Router ID) VRF Info (auto)
Exp Route-Target 65500:50000
Name: (auto)VRF-A
• 4-byte Value (VRF ID) RD: 62:10.0.0.3 (auto)
Imp Route-Target 65500:50000 (auto)
vrf context VRF-A Exp Route-Target 65500:50000 (auto)
vni 50000 V2
rd auto V1
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
route-target both auto RR BGP Route-Reflector
route-target both auto evpn V3 iBGP Peering
Multiprotocol BGP (MP-BGP) Primer BGP Advertisement
VPN-EVPN: RD:[MAC_A][IP_A]
• VPN Segmentation for tenant BGP Next-Hop: V1
Route Target: 65500:50000
routing (Multi-Tenancy) Label (L3VNI): 50000
RR RR
• Selective distribute VPN routes -
Route Target (RT)
• 8-byte field of VRF parameter
MAC_A / IP_A >>
• unique value to define the LOCAL
Route-Type2
import/export rules for VPN prefix V1 V2
MAC_A / IP_A >> V1
Route-Type2
V3 iBGP Peering
Host A
MAC_A / IP_A
Multiprotocol BGP (MP-BGP) Primer BGP Advertisement
VPN-EVPN: RD:[MAC_A][IP_A]
• Cisco’s VXLAN/EVPN does provide BGP Next-Hop: V1
Route Target: 65500:50000
automated Route Target (RT) Label (L3VNI): 50000
• 8-byte Route Target (2 x 4-byte) RR RR
• ASN : VNI
MAC_A / IP_A >>

vrf context VRF-A LOCAL
vni 50000 Route-Type2
V2
rd auto V1
MAC_A / IP_A >> V1
address-family ipv4 unicast Route-Type2
route-target both auto RR BGP Route-Reflector
route-target both auto evpn V3 iBGP Peering
Host A
MAC_A / IP_A
Overlay with Optimized Routing
EVPN Control Plane -- Host and Subnet Route Distribution
BGP Update
RR RR • Host-MAC
Spine • Host-IP
Border • Internal IP Subnet
• External Prefixes
V
V
V
V
V
V
BGP Adjacencies
Route-Reflectors deployed
RR
for scaling purposes (iBGP)
55
Overlay with Optimized Routing
EVPN Control Plane -- Host and Subnet Route Distribution
BGPMultiprotocol
Scalable Multi-Tenancy with Update BGP
RR RR • Host-MAC
Spine • Host-IP
EVPN Address-Family:
Border Host MAC+IP,• internal/external IP
Internal IP Subnet
Subnets
• External Prefixes
V
BGP enhanced for Fast Convergence at Large Scale
V Extensions for Fast and Seamless Host Mobility

V
V
Distributed Gateway with Traffic Flow Symmetry
V
V
BGP Adjacencies ARP Suppression
Route-Reflectors deployed
RR
for scaling purposes (iBGP)
56
Host Advertisement
Route MAC, IP L2VNI L3VNI NH Encap Seq
(“VLAN”) (“VRF”)
• Host Attaches Type
2 MAC_A, IP_A 30001 50001 IP_V1 8:VXLAN 0

• Host “A” attaches to Edge Device
(VTEP) RR RR
• VTEP V1 advertises Host “A”

reachability information
• MAC and L2VNI [mandatory]
• IP and L3VNI [optional] V1 V2
• depending on ARP
• Additional route attributes

advertised RR BGP Route-Reflector
• MPLS Label1 (L2VNI) V3 iBGP Peering*
• MPLS Label2 (L3VNI) Host A
MAC_A / IP_A
• Extended Communities
Route Type: Ethernet Segment Ethernet Tag MAC Address
MAC Address IP Address Length IP Address
2 - MAC/IP Identifier Identifier Length
V2# show bgp l2vpn evpn 192.168.1.73
BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 10.0.0.1:32868
BGP routing table entry for
[2]:[0]:[0]:[48]:[0050.56a3.c2bb]:[32]:[192.168.1.73]/272,
version 4
Paths: (1 available, best #1)
Flags: (0x000202) on xmit-list, is not in l2rib/evpn, is locked
Advertised path-id 1
L3VNI Path type: internal, path is valid, is best path, no labeled nexthop
AS-Path: NONE, path sourced internal to AS
L2VNI 10.0.0.1 (metric 3) from 10.0.0.111 (10.0.0.111)
Origin IGP, MED not set, localpref 100, weight 0
Received label 30001 50001
Extcommunity: RT:65501:30001 RT:65501:50001 ENCAP:8 Router MAC:5087.89d4.5495
Originator: 10.0.0.1 Cluster list: 10.0.0.111
Remote VTEP Route Target: Route Target: Overlay Encapsulation: Router MAC of
IP Address L2VNI (VLAN) L3VNI (VRF) 8 - VXLAN Remote VTEP
58
Protocol Learning & Distribution
RR RR
MAC, IP L2VNI L3VNI NH MAC, IP L2VNI L3VNI NH
MAC_A, IP_A 30001 50001 local MAC_B, IP_B 30001 50002 local
1
1
1 V2
V1
MAC, IP L2VNI L3VNI NH
MAC_C, IP_C 30001 50001 local
MAC_Y, IP_Y 30002 50001 local

Host A
MAC_A / IP_A V3 Host B
MAC_B / IP_B
VTEPs advertise End-Host reachability

1 information (MAC,IP) within MP-BGP
Virtual Switch
Host C Host Y
MAC_C / IP_C MAC_Y / IP_Y
RR RR
2
2
2 V2
V1

Host A
MAC_B / IP_B
BGP Route-Reflector “reflects” Overlay related

2 reachability information to other VTEPs
Virtual Switch
Host C Host Y
3 3
RR RR
MAC_B, IP_B 30001 50001 IP_V2 MAC_A, IP_A 30001 50001 IP_V1
MAC_C, IP_C 30001 50001 IP_V3 MAC_C, IP_C 30001 50001 IP_V3
2
VMAC_Y, IP_Y 30002 50001 IP_V3
MAC_Y, IP_Y 30002 50001
V1IP_V3
3

Host A
MAC_A / IP_A V3 MAC_A, IP_A 30001 50001 IP_V1
Host B
MAC_B / IP_B
MAC_B, IP_B 30001 50001 IP_V2
VTEPs receive respective reachability information
3 and installs them related to route-policy into RIB/FIB
Virtual Switch
Host C Host Y
Subnet Route Advertisement
Route MAC, IP L3VNI NH Encap
(“VRF”)
• IP Prefix Redistribution Type
5 Subnet_A/24 50001 IP_V1 8:VXLAN

• From “Direct” (connected), “Static” or
dynamically learned Routes RR RR
• VTEP V1 advertises local Subnet

through redistribution of “Direct”
(connected) routes
• IP Prefix, IP Prefix Length, and L3VNI V1 V2

advertised
• MPLS Label (L3VNI) RR BGP Route-Reflector
• Extended Communities V3 iBGP Peering*
(“VRF”)
• If multiple VTEP announce same IP Type
5 Subnet_A/24 50000
50001 IP_V1 8:VXLAN
Prefix, Equal Cost Multipath (ECMP)
5 Subnet_A/24 50001 IP_V2 8:VXLAN
will apply 5 Subnet_A/24 RR
50001 RR IP_V3 8:VXLAN
• VTEP V1 advertises local Subnet

through redistribution of “Direct”
(connected) routes
• IP Prefix, IP Prefix Length, and L3VNI V1 V2

advertised
• MPLS Label (L3VNI) RR BGP Route-Reflector
• Extended Communities V3 iBGP Peering*
(“VRF”)
• IP Prefix Learning Type
5 Subnet_X/24 50001 IP_V1 8:VXLAN

• via BGP with VRF-Lite (Inter-AS
Option A) RR RR
• via LISP on Nexus 7000/7700
• via other routing protocol (static or
dynamic)
• VTEP V1 participated in external V1 V2
Peering (LISP, BGP, OSPF etc.)
and advertises learned IP Prefixes
into the Fabric
• IP Prefix RR BGP Route-Reflector
• IP Prefix Length V3 iBGP Peering*

• L3VNI
Route Type: Ethernet Segment Ethernet Tag
IP Prefix Length IP Prefix GW IP Address
5 – IP Prefix Identifier Identifier
V2# show bgp l2vpn evpn 192.168.2.0
BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 10.0.0.1:3
BGP routing table entry for [5]:[0]:[0]:[24]:[192.168.2.0]:[0.0.0.0]/224, version 3
Paths: (1 available, best #1)
Flags: (0x000002) on xmit-list, is not in l2rib/evpn, is locked
Advertised path-id 1
Path type: internal, path is valid, is best path, no labeled nexthop
AS-Path: NONE, path sourced internal to AS
L3VNI 10.0.0.1 (metric 3) from 10.0.0.111 (10.0.0.111)
Origin incomplete, MED 0, localpref 100, weight 0
Received label 50001
Extcommunity: RT:65501:50001 ENCAP:8 Router MAC:5087.89d4.5495
Originator: 10.0.0.1 Cluster list: 10.0.0.111
Remote VTEP Route Target: Overlay Encapsulation: Router MAC of

IP Address L3VNI (VLAN) 8 - VXLAN Remote VTEP
65
ARP Suppression
VXLAN/EVPN
RR RR
MAC, IP VNI NH MAC, IP VNI NH
MAC_B, IP_B 30001 IP_V2 MAC_A, IP_A 30001 IP_V1
MAC_C, IP_C 30001 IP_V3 MAC_C, IP_C 30001 IP_V3
MAC_Y, IP_Y 30002 IP_V3 MAC_Y, IP_Y 30002 IP_V3

ARP Request for IP_B
Src MAC: MAC_A
Dst MAC: FF:FF:FF:FF:FF:FF V1 V2
1
ARP Response for IP_B
2 Src MAC: MAC_B
MAC, IP VNI NH
Dst MAC: MAC_A
MAC_A, IP_A 30001 IP_V1
MAC_B, IP_B 30001 IP_V2

Host A
MAC_B / IP_B
1 ARP Request sent for IP_B sent from Host A

Virtual Switch
V1 knows about IP_B and can respond.

2
No need for ARP forwarding across the Network Host C Host Y
ARP Handling on Lookup “Miss” (1)
VXLAN/EVPN
Missing RR RR
MAC, IP VNI NH
MAC, IP “B”VNI NH
MAC_C, IP_C 30001 IP_V3 MAC_A, IP_A 30001 IP_V1
MAC_Y, IP_Y 30002 IP_V3 2 MAC_C, IP_C 30001 IP_V3
MAC_Y, IP_Y 30002 IP_V3

Src MAC: MAC_A
Dst MAC: FF:FF:FF:FF:FF:FF V1 V2 ARP Request for IP_B
Src MAC: MAC_A
Dst MAC: FF:FF:FF:FF:FF:FF
1 2
MAC, IP VNI NH
MAC_A, IP_A 30001 IP_V1
Host A
MAC_B / IP_B
Src MAC: MAC_A
Dst MAC: FF:FF:FF:FF:FF:FF
1 ARP Request sent for IP_B sent from Host A
Virtual Switch
Miss of IP_B. Forward ARP Request to all

2
Ports except source-port (ARP snooping) Host C Host Y
ARP Handling on Lookup “Miss” (2)
VXLAN/EVPN
RR RR
MAC, IP VNI NH
MAC, IP VNI NH MAC_A, IP_A 30001 IP_V1
MAC_C, IP_C 30000
30001 V3
IP_V3 MAC_C, IP_C 30001 IP_V3
4
MAC_Y, IP_Y 30001
30002 V3
IP_V3 MAC_Y, IP_Y 30002 IP_V3
ARP Response for IP_B
Src MAC: MAC_B
MAC_B, IP_B
V1
30001 IP_V2 V2 ARP Response from IP_B
Src MAC: MAC_B
Dst MAC: MAC_A Dst MAC: MAC_A
4
3
MAC, IP VNI NH
MAC_A, IP_A 30000

30001 V1
IP_V1
MAC_B, IP_B 30001 IP_V2

Host A
MAC_B / IP_B
3 ARP Response is sent to V2

Virtual Switch
V2 will populate this information in the

4
control-plane (learn) and forward it subsequently Host C Host Y
Packet Forwarding (Bridge)
VXLAN/EVPN
RR RR
MAC, IP VNI NH MAC, IP VNI NH
MAC_A, IP_A 30001 Local MAC_B, IP_B 30001 Local
MAC_B, IP_B 30001 IP_V2 MAC_A, IP_A 30001 IP_V1

SMAC: MAC_A SMAC: MAC_A
DMAC: MAC_B DMAC: MAC_B
SIP: IP_A
V1 V2 SIP: IP_A
DIP: IP_B DIP: IP_B
1
2 3 4
SMAC: MAC_V1 SMAC: hop-by-hop
Underlay
DMAC: hop-by-hop DMAC: MAC_V2
Underlay
SIP: IP_V1 SIP: IP_V1
DIP: IP_V2 DIP: IP_V2
Host A
MAC_A / IP_A
UDP V3 UDP Host B
MAC_B / IP_B
VXLAN VNID: 30001 VXLAN VNID: 30001
Overlay
Overlay
SMAC: MAC_A SMAC: MAC_A

DMAC: MAC_B DMAC: MAC_B
SIP: IP_A SIP: IP_A

DIP: IP_B DIP: IP_B
Packet Forwarding (Route)
VXLAN/EVPN
RR RR
MAC, IP VNI NH VRF MAC, IP VNI NH VRF
MAC_A, IP_A 30001 Local 50001 MAC_A, IP_A 30001 Local 50001
MAC_F, IP_F 30005 IP_V2 50001 MAC_F, IP_F 30005 E1/4 50001
SMAC: MAC_A SMAC: MAC_GW
DMAC: MAC_GW DMAC: MAC_F
SIP: IP_A
V1 V2 SIP: IP_A
DIP: IP_F DIP: IP_F
1
2 3 4
Underlay
Underlay
Host A
MAC_A / IP_A
UDP V3 UDP Host F
MAC_F, IP_F
Overlay
Overlay

SIP: IP_A SIP: IP_A

DIP: IP_F DIP: IP_F
Packet Forwarding (Route) – Silent Host
VXLAN/EVPN
RR RR
MAC, IP VNI NH VRF MAC, IP VNI NH VRF
MAC_A, IP_A 30000 Local 50001 MAC_A, IP_A 30000 Local 50001
Subnet F 30005 IP_V2 50001 Subnet F 30005 E1/4 50001

SIP: IP_A
V1 V2 SIP: IP_A
DIP: IP_F DIP: IP_F
1
2 3 4
Underlay
Underlay
Host A
MAC_A / IP_A
UDP V3 UDP Host F
MAC_F, IP_F
Overlay
Overlay

SIP: IP_A SIP: IP_A

DIP: IP_F DIP: IP_F
 Scalable Layer-2 Domains
Multi-Tenancy
Anycast – One-to-Nearest Association
• a network addressing and RR RR
routing methodology
• datagrams sent from a single
sender to the topologically
✖
nearest node
✔
• group of potential receivers, ✖
all identified by the same
destination address
✔
73
Distributed IP Anycast Gateway
• Distributed Inter-VXLAN Routing at
Access Layer (Leaf) RR RR
• All Leafs share same gateway IP and

MAC Address for a given Subnet
SVI 100
• Gateway is always active SVI 200
• no redundancy protocol, hello SVI 100
exchange etc. SVI 100

SVI 200
SVI 200
SVI 100
• Distributed state - Smaller ARP SVI 200
SVI 100
tables SVI 200
SVI 100
• Only local attached End-Points SVI 200
(Servers)
SVI 100, Gateway IP: 192.168.1.1, Gateway MAC: AG:AG:AG:AG:AG:AG
SVI 200, Gateway IP: 10.10.10.1, Gateway MAC: AG:AG:AG:AG:AG:AG
RR RR
Spine
bridge SVI 100, Gateway IP: 192.168.1.1
SVI 200, Gateway IP: 10.10.10.1
route V
route
SVI 100
Host3
V MAC: CC:CC:CC:CC:CC:CC
V SVI 200 IP: 192.168.1.33
VLAN 100
V VXLAN VNI 30001
V Host2
MAC: BB:BB:BB:BB:BB:BB
V IP: 10.10.10.22
SVI 100 VLAN 200
Host1 VXLAN VNI 30002
MAC: AA:AA:AA:AA:AA:AA
IP: 192.168.1.11
VLAN 100
VXLAN VNI 30001
RR RR
Any Subnet Routed Anywhere – Any VTEP can serve any Subnet
Spine
bridge SVI 100, Gateway IP: 192.168.1.1
Integrated Route & Bridge (IRB) - Route whenever you can,
SVI 200, Gateway IP: Bridge when needed
10.10.10.1
route V
route
– Optimized
No Hairpinning SVI 100 East/West and North/South Routing
Host3
V SVI 200Seamless IP: 192.168.1.33
Mobility - All Leaf share same Gateway MAC
VLAN 100
V VXLAN VNI 30001
V Host2Failure Domain – Layer-2/Layer-3 Boundary at Leaf

Reduced
V IP: 10.10.10.22
SVI 100 VLAN 200
Host1
Optimal
VXLAN Scalability
VNI 30002 – Route Distributed & closest to the Host
IP: 192.168.1.11
VLAN 100
VXLAN VNI 30001
Integrated Routing and Bridging (IRB)
VXLAN/EVPN based overlays follow
two slightly different Integrated RR RR
Routing and Bridging (IRB) semantics

• Asymmetric
• Uses an “asymmetric path” from the
Host towards the egressing port of the
VTEP vs. the way back
• Symmetric*
• Uses an “symmetric path” from the
Host towards the egressing port of the
VTEP vs. the way back
*Implemented by Cisco’s VXLAN/EVPN

Consistent Configuration
• Logical Configuration (VLAN, VRF,
VNI) consistently instantiated on ALL RR RR
Leafs
• Optimal for Consistency SVI 100
SVI 200
• Every VLAN/VNI Everywhere
SVI 300
SVI 100
• Sub-Optimal for Scale SVI 100
SVI 200
• Instantiates Resources (VLAN/VNI) SVI 100

SVI 200
SVI 300
even if no End-Point uses it SVI 200

SVI 300
SVI 100
SVI 300
SVI 100 SVI 200
SVI 200 SVI 300
SVI 300
Scoped Configuration
• Logical Configuration (VLAN, VRF,
VNI) scoped to Leafs with respective RR RR
connected End-Points
• Optimal for Scale SVI 200
• Instantiates Resources (VLAN/VNI)

where End-Points are connected SVI 300
SVI 100
• Consistency with End-Points SVI 200
SVI 100
• Configuration Consistency depends SVI 300
SVI 200
on End-Points
SVI 100 SVI 300
SVI 200
Asymmetric IRB
• Similar to todays Inter-VLAN routing
RR RR
• Requires to follow a consistent
configuration of VLAN and L2VNI
across all Switches
• Post routed traffic will leverage
destination Layer 2 Segment
SVI 200
SVI 300
(L2VNI), same as for bridged traffic
SVI 300
SVI 200
SVI 300
✖
RR
SVI 200
RR
SVI 200
SVI 300
Asymmetric IRB
Asymmetric IRB
L2VNI 30001
L2VNI 30002
Leaf
V V
SVI 300 SVI 200 SVI 200 SVI 300
Host1 Host2 Host3 Host4

MAC: AA:AA:AA:AA:AA:AA MAC: BB:BB:BB:BB:BB:BB MAC: CC:CC:CC:CC:CC:CC MAC: DD:DD:DD:DD:DD:DD
IP: 192.168.1.11 IP: 10.10.10.22 IP: 10.10.10.33 IP: 192.168.1.44
VLAN 300 VLAN 200 VLAN 200 VLAN 300
VXLAN VNI 30003 VXLAN VNI 30002 VXLAN VNI 30002 VXLAN VNI 30003
Symmetric IRB
• Similar to Transit Routing Segments
RR RR
• Scoped Configuration of
VLAN/L2VNI; only required where
End-Points (Server) reside
• New VNI (L3VNI) introduced per
virtual routing and forwarding (VRF)
SVI 200
SVI 300
context
• Routed traffic uses transit VNI
(L3VNI), while bridged traffic uses
SVI 300
SVI 200
L2VNI
SVI 300
RR
SVI 200
RR
SVI 200
SVI 300
Symmetric IRB
Symmetric IRB
L3VNI 50001 (VRF)
Leaf
V V
SVI 300 SVI 200 SVI 200 SVI 300

IP: 192.168.1.11 IP: 10.10.10.22 IP: 10.10.10.33 IP: 192.168.1.44
 Integrated Route and Bridge
Multi-Tenancy
Agenda

• Overview
• Underlay
• Multi-Tenancy

87
What is Multi-Tenancy
• A mode of operation, where multiple independent instances (tenant)

operate in a shared environment.
• Each instance (i.e. VRF/VLAN) is logically isolated, but physically

integrated.
88
Where can we apply Multi-Tenancy
Multi-Tenancy at Layer-2 Multi-Tenancy at Layer-3
• Per-Switch VLAN-to-VNI mapping • VRF-to-VNI mapping
• Per-Port VLAN Significance • MP-BGP for scaling with VPNs

Layer-2 Multi-Tenancy
RR RR
Spine
bridge
V
VLAN 100
Host3
V IP: 192.168.1.33
VLAN 100
V VXLAN VNI 30001
V
V
VLAN 100
Host1
IP: 192.168.1.11
VLAN 100
VXLAN VNI 30001
Layer-2 Multi-Tenancy – Bridge Domains
VXLAN Overlay
(VNI 30001)
Leaf
V
VLAN 100
Bridge Domain V
VLAN 100
Host1 Host3
MAC: AA:AA:AA:AA:AA:AA MAC: CC:CC:CC:CC:CC:CC
IP: 192.168.1.11 IP: 192.168.1.33
VLAN 100 VLAN 100
VXLAN VNI 30001 VXLAN VNI 30001
Layer-2 Multi-Tenancy – Bridge Domains
VXLAN Overlay
The Bridge Domain
(VNI is the Layer-2 Segment from Host to Host
30001)
In VXLAN, the Bridge Domain consists of three Components

Leaf
V
VLAN 100
Bridge Domain
1) The Ethernet Segment (VLAN), between Host
V and Switch
VLAN 100
2) The Hardware Resources (Bridge Domain) within the Switch
3) The VXLAN Segment (VNI) between Switch and Switch
Host1 Host3
MAC: AA:AA:AA:AA:AA:AA MAC: CC:CC:CC:CC:CC:CC
IP: 192.168.1.11 IP: 192.168.1.33
VLAN 100 VLAN 100
VXLAN VNI 30001 VXLAN VNI 30001
VLAN-to-VNI mapping
VXLAN Overlay
(VNI 30001)
Leaf
V V
VLAN 100 VLAN 100
Host1 Host2 Host3

MAC: AA:AA:AA:AA:AA:AA MAC: BB:BB:BB:BB:BB:BB MAC: CC:CC:CC:CC:CC:CC
IP: 192.168.1.11 IP: 192.168.1.22 IP: 192.168.1.33
VLAN 100 VLAN 100 VLAN 100
VXLAN VNI 30001 VXLAN VNI 30001 VXLAN VNI 30001
CLI Modes - VLAN based (per-Switch)
Leaf#1 • VLAN to VNI configuration on a per-
vlan 100 switch basis
vn-segment 30001
• VLAN becomes “Switch Local
Leaf#2 Identifier”
vlan 100 • VNI becomes “Network Global
vn-segment 30001 Identifier”
Per-Switch VLAN-to-VNI mapping
VXLAN Overlay
(VNI 30001)
Leaf
V V
VLAN 100 VLAN 200
Host1 Host2 Host3

IP: 192.168.1.11 IP: 192.168.1.22 IP: 192.168.1.33
CLI Modes - VLAN based (per-Switch)
Leaf#1 • VLAN to VNI configuration on a per-
vlan 100 switch basis
vn-segment 30001
• VLAN becomes “Switch Local
Leaf#2 Identifier”
vlan 200 • VNI becomes “Network Global
vn-segment 30001 Identifier”
• 4k VLAN limitation has been
removed
Per-Port VLAN-to-VNI mapping
VXLAN Overlay
(VNI 30001)
Leaf
V V
Host1 Host2 Host3

IP: 192.168.1.11 IP: 192.168.1.22 IP: 192.168.1.33
CLI Modes - VLAN based (per-Port)
Leaf#1
vlan 2500
vn-segment 30001
interface Ethernet 1/8

switchport mode trunk
switchport vlan mapping enable
switchport vlan mapping 100 2500

switchport mode trunk
switchport vlan mapping enable
switchport vlan mapping 200 2500
CLI Modes - Bridge-Domain based (per-Port)
Leaf#1
bridge-domain 100
member vni 30001
encapsulation profile vni VLAN100-30001

dot1q 100 vni 30001
encapsulation profile vni VLAN200-30001
no switchport
dot1q 200 vni 30001 service instance 1 vni
encapsulation profile VLAN100-30001 default

no switchport
service instance 1 vni
encapsulation profile VLAN200-30001 default
Layer-3 Multi-Tenancy
VRF-A (VNI 50001)
VRF-B (VNI 50002)
RR RR
Spine SVI 100, Gateway IP: 192.168.1.1 (VRF-A)
route SVI 200, Gateway IP: 10.10.10.1 (VRF-B)
SVI 300, Gateway IP: 172.16.1.1 (VRF-B)
route V
SVI 300
Host3
V IP: 172.16.1.33 (VRF-B)
V SVI 200 VLAN 300
V
V Host2
IP: 10.10.10.22 (VRF-B)
V VLAN 200
SVI 100
Host1
IP: 192.168.1.11 (VRF-A)
VLAN 100
Layer-3 Multi-Tenancy – VRF-VNI or L3VNI
VRF-A VRF-B
(VNI 50001) (VNI 50002)
Leaf
V V V
SVI 100 SVI 200 SVI 300
Host1 Host2 Host3

IP: 192.168.1.11 (VRF-A) IP: 10.10.10.22 (VRF-B) IP: 172.16.1.33 (VRF-B)
VRF-A VRF-B
(VNI 50001) (VNI 50002)
Leaf
V V V
SVI 100 SVI 200 SVI 300
Host1 Host2 Host3

VRF-A VRF-B
(VNI 50001) (VNI 50002)
Leaf
Routing Routing
DomainV
SVI 100
V
SVI 200
Domain V
SVI 300
VRF-A VRF-B
Host1 Host2 Host3
VRF-A VRF-B
The Routing Domain
(VNI 50001) is the VRF owning multiple
(VNI 50002)across multiple Switches
Subnets
Leaf
Routing Routing
In VXLAN EVPN, the Routing Domain consists of three Components
DomainV
SVI 100
1) The
SVI 200
Domain
V Routing Domains (VRF), local
V to the Switch
SVI 300
2) The Routing Domain (L3VNI) between the Switches
VRF-A VRF-B
3) Multi-Protocol BGP with EVPN Address-Family
Host1 Host2 Host3

Layer-3 Multi-Tenancy – VRF-Lite
VLAN 1001
VLAN 1002
Leaf
V V
SVI 100 SVI 200 SVI 300 SVI 400
Subnet1 Subnet2 Subnet3 Host4

IP: 192.168.1.0/24 (VRF-A) IP: 10.10.10.0/24 (VRF-B) IP: 172.16.1.0/24 (VRF-B) IP: 10.44.44.0/24 (VRF-A)
VLAN 1001
Ethernet
VLAN 1002
Leaf
V V
SVI 100 SVI 200 SVI 300 SVI 400

vrf context VRF-B vrf context VRF-B
interface eth1/10.1002
VLAN 1001 interface eth1/10.1002
encapsulation dot1q 1002
vrf member VRF-B
Ethernet
VLAN 1002
vrf member VRF-B
ip address 10.2.2.1/24 ip address 10.2.2.2/24
ip router ospf 100 area 0.0.0.0 ip router ospf 100 area 0.0.0.0
router ospf 100 router ospf 100

Leaf
vrf VRF-B vrf VRF-B
V
vrf context VRF-A vrf context VRF-A
V
SVI 100 SVI 200 SVI 300 SVI 400
interface eth1/10.1001 interface eth1/10.1001
encapsulation dot1q 1001 encapsulation dot1q 1001
vrf member VRF-A vrf member VRF-A
ip router ospf 100 area 0.0.0.0 ip router ospf 100 area 0.0.0.0
router ospf 100 router ospf 100

vrf VRF-A vrf VRF-A
Layer-3 Multi-Tenancy – MPLS L3VPN
VPN Label “Blue”
MPLS
VPN Label “Red”
Leaf
V V
SVI 100 SVI 200 SVI 300 SVI 400

Layer-3 Multi-Tenancy – MPLS L3VPN
rd 1.1.1.1:200 VPN Label “Blue” rd 1.1.1.2:200
address-family ipv4 unicast address-family ipv4 unicast
route-target import 200:200
route-target export 200:200
MPLS
VPN Label “Red”
route-target import 200:200
route-target export 200:200

rd 1.1.1.1:100 rd 1.1.1.2:100
Leaf address-family ipv4 unicast
route-target import 100:100 route-target import 100:100
route-targetVexport 100:100 route-targetVexport 100:100
router SVI
bgp100
65500 SVI 200 routerSVI
bgp30065500SVI 400
neighbor 1.1.1.2 remote-as 65500 neighbor 1.1.1.1 remote-as 65500
address-family vpnv4 unicast address-family vpnv4 unicast
send-community extended send-community extended
vrf VRF-A vrf VRF-A
vrf VRF-B vrf VRF-B
109
Layer-3 Multi-Tenancy – VXLAN EVPN
L3VNI 50001
VXLAN
L3VNI 50002
Leaf
V V
SVI 100 SVI 200 SVI 300 SVI 400

IP: 192.168.1.11 (VRF-A) IP: 10.10.10.22 (VRF-B) IP: 172.16.1.33 (VRF-B) IP: 10.44.44.44 (VRF-A)
VLAN 100 VLAN 200 VLAN 300 VLAN 400 110
Layer-3 Multi-Tenancy – VXLAN EVPN
vni 50002 L3VNI 50001 vni 50002
rd auto rd auto
VXLAN
L3VNI 50002
route-target both auto evpn route-target both auto evpn

vni 50001
Leaf vni 50001
rd auto rd auto
V address-family ipv4 unicast
V
route-target both auto route-target both auto
route-target
SVI 100 both auto evpn
SVI 200 route-target
SVI 300 both auto evpn
SVI 400
router bgp 65500 router bgp 65500

address-family l2vpn evpn address-family l2vpn evpn
send-community extended send-community extended
vrf VRF-A vrf VRF-A
Host1 advertise l2vpn Host2
evpn Host3 Host4
advertise l2vpn evpn
vrf VRF-B
MAC: AA:AA:AA:AA:AA:AA MAC: BB:BB:BB:BB:BB:BB vrf VRF-B
MAC: CC:CC:CC:CC:CC:CC MAC: DD:DD:DD:DD:DD:DD
address-family
IP: 192.168.1.11 (VRF-A) ipv4 unicast (VRF-B)
IP: 10.10.10.22 IP: 172.16.1.33address-family
(VRF-B) IP:ipv4 unicast(VRF-A)
10.44.44.44
VLAN 100 advertise l2vpn evpn
VLAN 200 VLAN 300 advertise l2vpn evpn
VLAN 400 111
Integrated Route & Bridge + Multi-Tenancy
VRF-A (VNI 50001)
RR RR
Spine
bridge SVI 100, Gateway IP: 192.168.1.1 (VRF-A)
L2VNI 30001 SVI 200, Gateway IP: 10.10.10.1 (VRF-A)
route V
L3VNI 50001 SVI 100
Host3
V SVI 200 IP: 192.168.1.33
VLAN 100
V VXLAN VNI 30001
V Host2
V IP: 10.10.10.22
SVI 100 VLAN 200
IP: 192.168.1.11
VLAN 100
112
VXLAN VNI 30001
Integrated Route & Bridge + Multi-Tenancy
VRF-A (VNI 50001)
RR RR
Spine
SVI 100, Gateway IP: 192.168.1.1 (VRF-A)
SVI 200, Gateway IP: 10.10.10.1 (VRF-A)
V
SVI 100
Host3
V SVI 200 IP: 192.168.1.33 (VRF-A)
VLAN 100
V VXLAN VNI 30001
V Host2
V IP: 10.10.10.22 (VRF-A)
SVI 100 VLAN 200
IP: 192.168.1.11 (VRF-A)
VLAN 100
VXLAN VNI 30001 113
 Integrated Route and Bridge
 Multi-Tenancy
Agenda

• Overview
• Underlay
• Multi-Tenancy

115
“Stories” and Use-Cases
116
• Yesterday: VXLAN, yet another
VXLAN Overlay
• Data-Plane only (Multicast based
applicability •
Flood & Learn)
Today: VXLAN for the creation of
evolves as the scalable DC Fabrics – Intra-DC
• Control-Plane, active VTEP discovery,
Multicast and Unicast (Head-End
Control Plane Replication)
evolves!
Story #1: Scalable Data Center Fabric
• VXLAN based Data Center Fabric

• BGP EVPN Control-Protocol (Overlay)
• OSPF for Underlay Routing (Unicast)
• PIM ASM with Anycast-RP for BUM Replication (Underlay)
• Distributed IP Anycast Gateway
118
Story #1: Scalable Data Center Fabric (1)
Spine
Border Leaf
Leaf
Leaf
Leaf
Leaf
p2p Agg: 10.1.1.0/24 Leaf

RID Agg: 10.10.10.0/24
VTEP Agg: 10.200.200.0/24
RP Agg: 10.254.254.0/24
119
interface loopback0 interface loopback0
ip router ospf UNDERLAY area 0.0.0.0 ip router ospf UNDERLAY area 0.0.0.0
interface loopback0
router ospf UNDERLAY
ip address 10.10.10.102/32 router ospf UNDERLAY
router-id
ip router 10.10.10.101
ospf UNDERLAY area 0.0.0.0 router-id 10.10.10.201
interface loopback0
interface
router Ethernet1/1
ospf
ip addressUNDERLAY
10.10.10.103/32 interface Ethernet1/1
mturouter-id
9192
ip router10.10.10.102
ospf UNDERLAY area 0.0.0.0 mtu 9192
ipinterface
ospf
router network point-to-point
Ethernet1/1
ospf UNDERLAY ip ospf network point-to-point
ip mtu
router
9192ospf 10.10.10.103
router-id UNDERLAY area 0.0.0.0 ip router ospf UNDERLAY area 0.0.0.0
ip ip
pimaddress
sparse-mode
10.1.1.5/30 ip pim sparse-mode
… ipinterface
ospf network point-to-point
Ethernet1/1
ip mtu
router
9192ospf UNDERLAY area 0.0.0.0 interface Ethernet1/2
ip ip
pimaddress
sparse-mode
10.1.1.9/30 mtu 9192
… ip ospf network point-to-point ip address 10.1.1.6/30
ip router ospf UNDERLAY area 0.0.0.0 ip ospf network point-to-point
ip pim sparse-mode ip router ospf UNDERLAY area 0.0.0.0
… ip pim sparse-mode
p2p Agg: 10.1.1.0/24 interface Ethernet1/3

RID Agg: 10.10.10.0/24 mtu 9192
VTEP Agg: 10.200.200.0/24 ip address 10.1.1.10/30
RP Agg: 10.254.254.0/24 ip ospf network point-to-point
ip router ospf UNDERLAY area 0.0.0.0
ip pim sparse-mode
120
…
ip address 10.10.10.202/32 RP RP ip address 10.10.10.203/32

ip pim sparse-mode ip pim sparse-mode
ip pim anycast-rp 10.254.254.1 10.254.254.202 ip pim anycast-rp 10.254.254.1 10.254.254.202

ip pim anycast-rp 10.254.254.1 10.254.254.203 ip pim anycast-rp 10.254.254.1 10.254.254.203
ip pim rp-address 10.254.254.1 ip pim rp-address 10.254.254.1
interface loopback0
ip address 10.10.10.103/32
interface loopback0
ip address 10.10.10.102/32
p2p Agg: 10.1.1.0/24 ip pim rp-address 10.254.254.1
RID Agg: 10.10.10.0/24 interface loopback0 RP Rendezvous-Point
VTEP Agg: 10.200.200.0/24 ip address 10.10.10.101/32
ip pim rp-address 10.254.254.1
RP Agg: 10.254.254.0/24 ip router ospf UNDERLAY area 0.0.0.0
ip pim rp-address 10.254.254.1

121
VTEP
VTEP
VTEP
VTEP
interface loopback1
VTEP ip address 10.200.200.103/32
interface loopback1
ip address 10.200.200.102/32
p2p Agg: 10.1.1.0/24 VTEPinterface loopback1
ip routerinterface
ospf UNDERLAY
nve1 area 0.0.0.0
ip address 10.200.200.101/32
RID Agg: 10.10.10.0/24 source-interface loopback1
ip routerinterface
ospf UNDERLAY
nve1 area 0.0.0.0
VTEP Agg: 10.200.200.0/24 host-reachability protocol bgp
RP Agg: 10.254.254.0/24 source-interface loopback1
interface nve1
host-reachability protocol bgp
source-interface loopback1
122
router-id 10.10.10.202 router-id 10.10.10.203
RR RR
neighbor 10.10.10.0/24 remote-as 65500 neighbor 10.10.10.0/24 remote-as 65500
update-source loopback0 update-source loopback0
send-community both send-community both
route-reflector-client VTEP
route-reflector-client
VTEP
VTEP
router bgp 65500
bgp router-id
router VTEP 65500 10.10.10.103
router-id neighbor 10.10.10.202 remote-as 65500
10.10.10.102
router bgp 65500
update-source
neighbor 10.10.10.202 loopback0
remote-as 65500
router-id
VTEP 10.10.10.101address-family
update-source loopback0 l2vpn evpn
neighbor 10.10.10.202 remote-as 65500
send-community both
address-family
neighbor
send-community 10.10.10.203 remote-as 65500
p2p Agg: 10.1.1.0/24 VTEP address-family l2vpn evpn both
neighbor update-source
10.10.10.203 loopback0
remote-as 65500
RID Agg: 10.10.10.0/24 send-community both
address-family l2vpn evpn RR BGP Route-Reflector
update-source loopback0
VTEP Agg: 10.200.200.0/24 send-community both
address-family
RP Agg: 10.254.254.0/24
address-family l2vpn evpn both
send-community
send-community both
123
vlan 100
vn-segment 30001
name Blue
vlan 200
vn-segment 30002
name Green
VTEP
evpn
vni 30001
rd auto
VTEP vni 30002
rd auto
VTEP route-target both auto
interface nve1
VTEP source-interface loopback1
VTEP member vni 30001
mcast-group 239.239.239.1
member vni 30002
p2p Agg: 10.1.1.0/24 VTEP mcast-group 239.239.239.2
RID Agg: 10.10.10.0/24
VTEP Agg: 10.200.200.0/24
RP Agg: 10.254.254.0/24
124
interface Vlan100 vrf context VRF-A
mtu 9192 vni 50001
vrf member VRF-A rd auto
ip address 192.168.1.1/24 tag 21921 address-family ipv4 unicast
fabric forwarding mode anycast-gateway route-target both auto
interface Vlan200 VTEP
mtu 9192 route-target both auto
vrf member VRF-A route-target both auto evpn
ip address 10.10.10.1/24 tag 21921
fabric forwarding mode anycast-gateway interface nve1
VTEP member vni 50001 associate-vrf
router bgp 65500

VTEP vrf VRF-A
VTEP advertise l2vpn evpn
redistribute direct route-map TAG
p2p Agg: 10.1.1.0/24 VTEP

RID Agg: 10.10.10.0/24 route-map TAG permit 10
VTEP Agg: 10.200.200.0/24 match tag 21921
RP Agg: 10.254.254.0/24
125
interface Ethernet 2/1.10
vrf member VRF-A
ip address 172.16.0.1/30

vrf member VRF-B VTEP
ip address 172.16.0.1/30
router bgp 65500

vrf VRF-A VTEP
address-family ipv4 unicast interface Ethernet 1/15.21
advertise l2vpn evpn vrf member VRF-A
VTEP
aggregate-address 10.10.10.0/24 summary-only ip address 172.16.0.2/30
aggregate-address 192.168.1.0/24 summary-only encapsulation dot1q 5
neighbor 172.16.0.1 remote-as 65599 VTEP WAN
update-source Ethernet2/1.10 interface Ethernet 1/15.22
address-family ipv4 unicast vrf member VRF-B
VTEP
… ip address 172.16.0.2/30
p2p Agg: 10.1.1.0/24 VTEP
router bgp 65599
RID Agg: 10.10.10.0/24
vrf VRF-A
VTEP Agg: 10.200.200.0/24
RP Agg: 10.254.254.0/24
update-source Ethernet1/15.21
address-family ipv4 unicast 126
…
Story #2: Scalable Data Center Fabric
• VXLAN based Data Center Fabric

• BGP EVPN Control-Protocol (Overlay)
• eBGP for Underlay Routing (Unicast)
• eBGP Multi-AS Design
• Ingress Replication for BUM (Underlay)
• Distributed IP Anycast Gateway
127
Spine AS65500
AS65555
AS65504
AS65503
AS65503
AS65502
p2p Agg: 10.1.1.0/24 AS65501

RID Agg: 10.10.10.0/24
VTEP Agg: 10.200.200.0/24
RP Agg: 10.254.254.0/24
128
ip address 10.10.10.101/32 tag 12345 ip address 10.10.10.201/32 tag 12345
interface
interface Ethernet1/1
loopback0 interface Ethernet1/1
mtuip9192
address 10.10.10.102/32 tag 12345 mtu 9192
interface
interface Ethernet1/1
loopback0
router
mtuipbgp
9192 65501
address 10.10.10.103/32 tag 12345 interface Ethernet1/2
router-id
ip address10.10.10.101
10.1.1.5/30 mtu 9192
interface Ethernet1/1 ip address 10.1.1.6/30
redistribute
router bgp
mtu 9192 direct
65502 route-map UL-TAG
template peer10.10.10.102
router-id
ip address SPINE-UNDERLAY
10.1.1.9/30 interface Ethernet1/3
remote-as 65500 ipv4 unicast
address-family mtu 9192
address-family
redistribute
router bgp 65503ipv4
directunicast
route-map UL-TAG ip address 10.1.1.10/3
neighbor
template10.10.10.2
peer10.10.10.103
router-id SPINE-UNDERLAY
inherit peer 65500
remote-as SPINE-UNDERLAY
address-family ipv4 unicast router bgp 65500
neighbor 10.10.10.L1-S2
address-family
redistribute ipv4 unicast
direct route-map UL-TAG router-id 10.10.10.201
inherit
neighborpeer SPINE-UNDERLAY
10.10.10.6
template peer SPINE-UNDERLAY address-family ipv4 unicast
neighbor
inherit10.10.10.L1-S3
peer 65500
remote-as SPINE-UNDERLAY redistribute direct route-map UL-TAG
inherit
10.10.10.L1-S2
address-family ipv4 unicast neighbor 10.10.10.1 remote-as 65501
neighbor
10.10.10.10 address-family ipv4 unicast
p2pinherit
Agg: 10.1.1.0/24
neighborpeer
inherit SPINE-UNDERLAY
10.10.10.L1-S3
peer SPINE-UNDERLAY neighbor 10.10.10.5 remote-as 65502
RID
… Agg: 10.10.10.0/24
inherit
10.10.10.L1-S2 address-family ipv4 unicast
VTEP Agg:
neighbor10.200.200.0/24
peer SPINE-UNDERLAY neighbor 10.10.10.9 remote-as 65503
RP Agg: 10.254.254.0/24
inherit
10.10.10.L1-S3 address-family ipv4 unicast
… inherit peer SPINE-UNDERLAY …
neighbor 10.10.10.L1-S4 route-map TAG-UL permit 10
129
inherit peer SPINE-UNDERLAY match tag 12345
…
VTEP
VTEP
VTEP
VTEP
interface loopback1
VTEP ip address 10.200.200.103/32 tag 12345
interface loopback1
ip address 10.200.200.102/32
interface nve1 tag 12345
p2p Agg: 10.1.1.0/24 VTEPinterface loopback1 source-interface loopback1
ip address 10.200.200.101/32
interface nve1 tag 12345
RID Agg: 10.10.10.0/24 host-reachability protocol bgp
VTEP Agg: 10.200.200.0/24 source-interface loopback1
interface nve1
RP Agg: 10.254.254.0/24 source-interface loopback1
130
router-id 10.10.10.202 router-id 10.10.10.203
nexthop route-map NHUNCH nexthop route-map NHUNCH
retain route-target all retain route-target all
VTEP
disable-connected-check disable-connected-check
route-map NHUNCH out route-map NHUNCH out
neighbor 10.10.10.102 remote-as 65502 VTEP 10.10.10.102 remote-as 65502
neighbor
update-source loopback0 update-source loopback0
address-family l2vpn evpn VTEP address-family l2vpn evpn
disable-connected-check disable-connected-check
route-map NHUNCH out VTEP route-map NHUNCH out
update-source loopback0 VTEP update-source loopback0
disable-connected-check
p2p Agg: 10.1.1.0/24 VTEP disable-connected-check
route-map
RID Agg: 10.10.10.0/24 NHUNCH out route-map NHUNCH out
VTEP Agg: 10.200.200.0/24
RP Agg: 10.254.254.0/24
route-map NHUNCH permit 10
set ip next-hop unchanged
131
router bgp 65503 VTEP

router-id 10.10.10.103
router bgp 65502 address-family l2vpn evpn
router-id 10.10.10.102 send-community both
VTEP
neighbor 10.10.10.202 remote-as 65500 disable-connected-check
update-source loopback0 neighbor 10.10.10.203 remote-as 65500
router bgp 65501 address-family l2vpn evpn VTEP update-source loopback0
router-id 10.10.10.101 send-community both address-family l2vpn evpn
neighbor 10.10.10.202 remote-as 65500 disable-connected-check send-community both
update-source loopback0 VTEP
address-family l2vpn evpn update-source loopback0
send-community both address-family
VTEP l2vpn evpn
disable-connected-check send-community both
update-source
p2p Agg: 10.1.1.0/24 loopback0 VTEP
address-family
RID Agg: 10.10.10.0/24 l2vpn evpn
send-community
VTEP Agg: both
10.200.200.0/24
RP Agg:disable-connected-check
10.254.254.0/24
132
vlan 100
vn-segment 30001
name Blue
vlan 200
vn-segment 30002
name Green
VTEP
evpn
vni 30001
rd auto
route-target both 65500:30001
VTEP vni 30002
rd auto
VTEP route-target both 65500:30002
interface nve1
VTEP member vni 30001
ingress-replication protocol bgp
member vni 30002
p2p Agg: 10.1.1.0/24 VTEP ingress-replication protocol bgp
RID Agg: 10.10.10.0/24
VTEP Agg: 10.200.200.0/24
RP Agg: 10.254.254.0/24
133
interface Vlan100 vrf context VRF-A
mtu 9192 vni 50001
vrf member VRF-A rd auto
ip address 192.168.1.1/24 tag 21921 address-family ipv4 unicast
fabric forwarding mode anycast-gateway route-target both 65500:50001
route-target both 65500:50001 evpn
interface Vlan200 VTEP
mtu 9192 route-target both 65500:50001
vrf member VRF-A route-target both 65500:50001 evpn
ip address 10.10.10.1/24 tag 21921
fabric forwarding mode anycast-gateway interface nve1
VTEP member vni 50001 associate-vrf
router bgp 655xx

VTEP vrf VRF-A
VTEP advertise l2vpn evpn
redistribute direct route-map TAG
p2p Agg: 10.1.1.0/24 VTEP

RID Agg: 10.10.10.0/24 route-map TAG permit 10
VTEP Agg: 10.200.200.0/24 match tag 21921
RP Agg: 10.254.254.0/24
134
vrf member VRF-A
ip address 172.16.0.1/30

vrf member VRF-B VTEP
ip address 172.16.0.1/30
router bgp 65555

vrf VRF-A VTEP
address-family ipv4 unicast interface Ethernet 1/15.21
advertise l2vpn evpn vrf member VRF-A
VTEP
aggregate-address 10.10.10.0/24 summary-only ip address 172.16.0.2/30
aggregate-address 192.168.1.0/24 summary-only encapsulation dot1q 5
neighbor 172.16.0.1 remote-as 65599 VTEP WAN
update-source Ethernet2/1.10 interface Ethernet 1/15.22
address-family ipv4 unicast vrf member VRF-B
VTEP
… ip address 172.16.0.2/30
p2p Agg: 10.1.1.0/24 VTEP
router bgp 65599
RID Agg: 10.10.10.0/24
vrf VRF-A
VTEP Agg: 10.200.200.0/24
RP Agg: 10.254.254.0/24
update-source Ethernet1/15.21
address-family ipv4 unicast 135
…
• Yesterday: VXLAN, yet another
VXLAN Overlay
• Data-Plane only (Multicast based
applicability •
Flood & Learn)
Today: VXLAN for the creation of
evolves as the scalable DC Fabrics – Intra-DC
• Control-Plane, active VTEP discovery,
Multicast and Unicast (Head-End
Control Plane Replication)
• Future: VXLAN for DCI – Inter-DC
evolves! • DCI Enhancements (ARP
caching/suppress, Multi-Homing,
Failure Domain isolation, Loop
Protection etc.)
What is the Elephant in the Room?
137
Note sure if it is a Elephant
VXLAN for Interconnecting Networks
Story #3: Inter-Fabric Connectivity
• Option 1: End-to-End Fabric Stretch

• Option 2: Fabric-DCI-Fabric (2-box)
• Option 3: Fabric-DCI-Fabric L3-DCI (1-box)
• Option 4: Fabric-DCI-Fabric L2-DCI (1-box)
139
Inter-Fabric Connectivity (Option 1)
• Multiple BGP-EVPN Control-Plane
EVPN Control-Plane Domain 1 Domains
• End-to-End reachability for VTEP
V • End-to-End reachability for BUM
V
Replication
V • Multicast / Ingress Replication
V
V
V
• End-to-End Data-Plane
encapsulation
V
V
VXLAN Encapsulation
EVPN Control-Plane Domain 2
Inter-Fabric Connectivity (Option 2)
• Normalization via Ethernet (MPLS,
VRF-lite & IEEE 802.1Q Trunk) at
V
V the Border
V • Separate Data-Plane (DP)
V
DCI
V
encapsulation per Domain
V
• Multicast / Ingress Replication
V
V
VXLAN Encapsulation
EVPN Control-Plane Domain 2 DCI Encapsulation
Inter-Fabric Connectivity (Option 3 / Option 4)
• Integrated Hand-Off with Data-Plane
separation
V
V • Option 3 – L3 DCI
• L3-LISP, MPLS, EVPN
V
V
• Option 4 – L2 DCI
V • OTV, L2-LISP, EVPN
V
• Separate Data-Plane (DP)
V encapsulation per Domain
V • Multicast / Ingress Replication
VXLAN Encapsulation
EVPN Control-Plane Domain 2 DCI Encapsulation
Inter-Fabric Connectivity
Option 1 Option 2 Option 3/4
Underlay Control Plane Unified Underlay Domain Separated Underlay Domains Separated Underlay Domains
Overlay Control Plane Separated Overlay Control-Plane Domains
Overlay Data Plane Single Data-Plane Separated Data-Planes Separated Data-Planes
Unified Underlay Domain (All

BUM Replication in DCI Dependency on DCI Choice (Unicast/Multicast)
Multicast or All Ingress Replication)
ARP Flood Suppression

yes yes yes
(DCI)
Unknown Unicast Flood

no yes yes
Suppression (DCI)
Broadcast
no yes yes
Suppression/Limit (DCI)
Layer-2 Loop Prevention Loop mitigation (Edge Protection) VPC at Border Loop mitigation (At DCI)
Fabric Management &
Automation
144
How to achieve Data Center Automation
• Simplify
• Do not start with the most difficult task (low hanging Fruits)
• Standardize
• Find common Denominators and create Templates
• Automate repetitive Tasks
• Use Templates for Simple Tasks and use Automation (e.g. create VLAN, SVI, VRF)
• Abstract
• Take a step back and look at the WHOLE
• Cisco ACI
145
Anatomy of Data Center Automation
VMM Chef Openstack
Puppet NX-API Ansible
API
Underlay Overlay Hybrid Overlay Inter-Domain

Simplified
Management Provisioning & Management
Management and Multi-Fabric
- integration of
- Network Element- - Overlay Services - Seamless LISP
Physical and Virtual
Management (Layer 2/Layer 3) and MPLS
Workload
-Topology Overview
Mobility, Service Agility
- Service Chaining
VTEPs
integration
-Configuration - Optimizing Inter-

Domain integration
Deployment Multi-tenancy
-Cross DC Mobility
IP Fabric
Network Infrastructure
Fabric Management & Operations
Element Day-0: Day- 1: Day-2:
management:
Configuration Configuration and Visibility,
Hardware (POAP) Configuration Configuration
Management, Management increments,
Health Status, and Underlay compare changes.
Inventory Management Automated
Configuration
Compute
Integration
Troubleshooting
Simplifying Management & Fabric Visibility
• Device Auto-Configuration
• Cabling Plan Consistency Check
• Automated Network Provisioning
• Common point of fabric access
• Tenant, Virtual Fabric & Host
Visibility
Device Auto-Configuration: Day 0
• Underlay Configuration:
• Physical interface IP configuration
• Loopback interface IP configuration
• Multicast Configuration for the Underlay (BUM)
• Routing protocol for the underlay configuration
• vPC domain
• BGP EVPN + RR configuration
• VTEP configuration
149
Device Auto-Configuration: Day 0.5
• Tenant Configuration including:

• VPC configuration for downstream connectivity
• Interface configuration
• Host Ports and Port-Channels
150
Device Auto-Configuration: Day 1
• Tenant Configuration including:

• VLAN configuration
• VRF configuration
• VNI configuration
• SVI (BDI) configuration
• BGP VRF (L3 Tenant) + EVPN (L2 Tenant)
• Distributed IP Anycast Gateway configuration
151
Device Auto-Configuration (POAP)
Day 0, Day 0.5 and Day 1
1. Easy way to unbox, rack the device, and not enter any base CLI
configuration. Just rack, power, and plug into the management
network.
2. Provides a standard and consistent configuration across of the data
center network devices.
3. Provides a standard and consistent images to deploy to all of the
data center devices.
152
Q&A
153
Recommended Reading
Using TRILL, FabricPath, and VXLAN:
Designing Massively Scalable Data
Centers (MSDC) with Overlays
• Sanjay K. Hooda
• Shyam Kapadia
• Padmanabhan Krishnan
ISBN-10: 1-58714-393-3
ISBN-13: 978-1-58714-393-9
Recommended Viewing
Cisco Programmable Fabric Using
VXLAN with BGP EVPN LiveLessons
• David Jansen
• Lukas Krattiger
ISBN-10: 0-13-427229-3
ISBN-13: 978-0-13-427229-0
Thank you
156

BRKDCT 3378 PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

BRKDCT 3378 PDF

Uploaded by

Copyright:

Available Formats

Building DataCenter

Networks with VXLAN

• Closer Look on Packet Encapsulation

• Introduction to Data Center Fabrics

• “Stories” and Use-Cases

• Router/Switch end-points • Virtual end-points only • Physical and Virtual

 Full Bi-Sectional Bandwidth (N Spines)

 Any/All Leaf Distributed Default Gateways

 Any/All Subnets on Any Leaf

• Scale Out Architecture

• Introduction to Data Center Fabrics

• “Stories” and Use-Cases

• Traditionally VLAN is expressed

VLANs 4 bytes 802.1Q 0x8100

Ether Type (Etype)

a total address space of 24 bits 50 bytes Original CE Frame

VXLAN Outer IP UDP VxLAN 802.1Q CRC

(VNI/VNID) is part of the VXLAN

VXLAN Frame Format Dest. MAC Address 48

MAC-in-IP Encapsulation Src. MAC Address 48

VLAN Type 14 Bytes

Outer IP Header Src and Dst

Control Plane Data Plane

IP Interface Edge Device

IP Interface Edge Device

Edge Device Physical Servers

Local LAN Virtual Servers

VTEP Physical Servers

VTEP: VXLAN Tunnel End-Point

• Standards based Overlay (VXLAN) with Standards

Multi-Protocol Label Switching Provider Backbone Bridges Network Virtualization Overlay

 EVPN over NVO Tunnels (ie VXLAN) for Data Center

RFC 7348 Virtual eXtensible Local Area Network Data Plane

RFC 7432 BGP MPLS based Ethernet VPNs Control Plane

draft-ietf-bess-evpn-overlay A Network Virtualization Overlay Solution using EVPN Control Plane

draft-ietf-bess-evpn-inter-subnet-forwarding Integrated Routing and Bridging in EVPN Control Plane

draft-ietf-bess-l2vpn-evpn-prefix- IP Prefix Advertisement in E-VPN Control Plane

• Introduction to Data Center Fabrics

• “Stories” and Use-Cases

*BUM: Broadcast, Unknown Unicast & Multicast

• Data Centers often require Jumbo

• Separate VTEP from Routing p2p Links

• Use Loopback as Source-Interface V

*NVE: Network Virtualization Edge

Example from depicted topology:

= 48 IP Addresses for P2P Links

68 IP Addresses required == /25 Prefix V

Example from depicted topology:

= 6 IP Addresses for Loopback Interface (Used for VTEP)

84 IP Addresses required == /25 Prefix

*CLNS: Connection-Less Network Service

*AS: Autonomous System

• Spine and Aggregation Switches make good Rendezvous-Point (RP) Locations in

• Multi-Destination Traffic (Broadcast, Unknown Unicast, etc.) needs to be

• Introduction to Data Center Fabrics

• “Stories” and Use-Cases

MAC_A / IP_A >>

V Extensions for Fast and Seamless Host Mobility

2 MAC_A, IP_A 30001 50001 IP_V1 8:VXLAN 0

• VTEP V1 advertises Host “A”

• Additional route attributes

V2# show bgp l2vpn evpn 192.168.1.73

MAC, IP L2VNI L3VNI NH