BRKDCN-2125

Overlay Management and
Visibility with VXLAN
Shyam Kapadia, Principal Engineer

Tweet: @shyamkapadia
BRKDCN-2125
Cisco Webex Teams
Questions?
Use Cisco Webex Teams to chat
with the speaker after the session
How
1 Find this session in the Cisco Events Mobile App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space
BRKDCN-2125 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Who am I?
• Born & brought-up in India
• PhD in Computer Science from USC (Fight on!)
• 12 years at Cisco
• Developer, Author, Inventor…
• Hobbies: Movies, Sports (Cricket, NFL, NBA)
Agenda
• Overlays and Network abstraction

• Underlay-Overlay Correlation
• Motivation for Overlay OAM
• Operations, Administration and Management (OAM)

• VXLAN OAM – NVE Ping, Traceroute, Pathtrace
• Endpoint Visibility
• EVPN Multi-Site
• Examples
Overlays &
Network
Abstraction
Agenda


• EVPN Multi-Site
• Examples
Taxonomy - Underlay
Layer-3
Interface Spine Spine Spine Spine
Peering
Underlay
Edge Device Leaf Leaf Leaf Leaf Leaf Leaf Leaf
LAN
Segment
Hypervisor Baremetal Hypervisor Hypervisor Baremetal Hypervisor Baremetal Baremetal
Virtual
Server Physical
Server
Taxonomy - Overlay
Tunnel Encapsulation
Spine
(VNI Spine
Namespace)
Spine Spine
Overlay
VTEP Leaf Leaf Leaf Leaf Leaf Leaf Leaf
LAN
Segment
Hypervisor Baremetal Hypervisor Hypervisor Baremetal Hypervisor Baremetal Baremetal
Virtual
Server Physical VTEP: VXLAN Tunnel Endpoint
Server VNI/VNID: VXLAN Network Identifier
Understanding Overlay Technologies
Overlay Services
• Layer-2 Underlay Transport
Tunnel Encapsulation
• Layer-3 Network
• Layer-2 and Layer-3
Data-Plane
Control-Plane • Overlay Layer-2/Layer-3 Unicast
• Peer-Discovery Traffic
• Route Learning and Distribution • Overlay Broadcast, Unknown Unicast,
• Local Learning Multicast (BUM) traffic forwarding
• Remote Learning • Ingress Replication (Unicast)
• Multicast
Plumbing
VTEP VTEP
• Underlay – The Plumbing
Underlay Overlay
• Physical Connectivity
• What is an Overlay?
Spine • Overlay – Tunnels

• Logical Transport Service
Leaf • Virtual Private
Leaf Network (VPN)
• What is an Underlay?
Spine
Endpoint Connectivity vs. Endpoint Presence
VTEP • Connectivity
VTEP– The Plumbing
Underlay Overlay
• Endpoint physically connected

Baremetal
• ”I need a Interface” Baremetal
Spine
• Presence – Tunnels
• Endpoint logically present
• ”I need other Endpoints to
Leaf Leaf
Communicate”
Spine
Overlay Forwarding – Bridging
• Endpoint to Endpoint
Spine Spine Spine Spine Communication
• Layer-2 Service (VPN) for
VNI 30001 (L2VNI) Overlay Bridging
• Leaf to Leaf Communication

Leaf Leaf Leaf Leaf Leaf Leaf Leaf • The VTEP on the Leaf originates
the Encapsulation
• The Spine sees only

encapsulated packets
Baremetal Baremetal • No Knowledge about Endpoints
Host A Host B • Everything is IP/UDP*
VLAN 10 VLAN 10
*VXLAN, LISP, OTV use IP/UDP Encapsulation BRKDCN-2125 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
Overlay Forwarding – Pinging Host B
tc@boxA:$̃ ping HostB
PING HostB (192.168.10.102): 56 data bytes
64 bytes from 192.168.10.102 : icmp_seq=0 ttl=64 time=0.653 ms
Spine Spine Spine Spine
--- 192.168.10.102 ping statistics ---

2 packets transmitted, 2 packets received, 0% packet loss
VNI 30001 (L2VNI) Overlay tc@boxA:$̃
Leaf Leaf Leaf Leaf Leaf Leaf Leaf
Baremetal Baremetal
Host A Host B
VLAN 10 VLAN 10
Overlay Forwarding – A Bridged Ping to Host B

Outer SIP Outer DIP Protocol Outer Outer VXLAN Inner SMAC Inner DMAC Inner SIP Inner DIP ICMP
SPORT DPORT
Payload
10.200.200.101 10.200.200.104 UDP 45631 4789 30001
Overlay
0000.3001.1101 0000.3001.1102 192.168.10.101 192.168.10.102 Echo
Request
SMAC DMAC dot1Q SIP DIP ICMP

0000.3001.1101 0000.3001.1102 10 192.168.10.101 192.168.10.102 Echo
Request
Baremetal 0000.3001.1101
Baremetal 0000.3001.1102 10 192.168.10.101 192.168.10.102 Echo
Request
Host A Host B
MAC: 0000.3001.1101 MAC: 0000.3001.1102
IP: 192.168.10.101 IP: 192.168.10.102
Overlay Forwarding to B – Seen by the Underlay

Outer SIP Outer DIP Protocol Outer Outer
SPORT DPORT
10.200.200.101 10.200.200.104 UDP 45631 4789
Baremetal Baremetal
Host A Host B
MAC: 0000.3001.1101 MAC: 0000.3001.1102
IP: 192.168.10.101 IP: 192.168.10.102
Overlay Forwarding – Which Underlay Path?
Spine
• Network Virtualization Overlay
Spine
(NVO) – Network Abstraction

Spine Spine
from Physical Connectivity

• Entropy – Overlays provide Path
Leaf Leaf Leaf Leaf Leaf Leaf Leaf Diversity across Equal Cost
Multipath (ECMP)
• Correlation – Overlays don’t
present or represent the
Baremetal Baremetal
Underlay
Host A Host B
VLAN 10 VLAN 10
Entropy – the VXLAN Example
Entropy
happens here
Outer MAC Outer IP UDP VXLAN Inner MAC Inner IP Payload CRC
Data-Plane (VXLAN)
Path Diversity – Overlay Example (VXLAN)
Outer-Header
Baremetal
SIP 10.200.200.101 Baremetal
DIP 10.200.200.104
Host A Host B
MAC: 0000.3001.1101 Protocol UDP MAC: 0000.3001.1102
IP: 192.168.10.101 IP: 192.168.10.102
SPORT 52562
Original Packet Original Packet
SMAC 0000.3001.1101
DPORT
VXLAN
4789
30001
Overlay SMAC 0000.3001.1101
DMAC 0000.3001.1102 Leaf Leaf DMAC 0000.3001.1102

Inner-Header
dot1Q 10 dot1Q 10
SMAC 0000.3001.1101
SIP 192.168.10.101 SIP 192.168.10.101
DMAC 0000.3001.1102
DIP 192.168.10.102 DIP 192.168.10.102
dot1Q 10
Protocol TCP Protocol TCP
SIP 192.168.10.101
SPORT 56425 SPORT 56425
DIP 192.168.10.102
DPORT 25 DPORT 25
Protocol TCP
SPORT 56425
DPORT 25
Path Diversity – Overlay Example (VXLAN)
• VXLAN Entropy is based on 5-

Outer-Header
Baremetal
SIP 10.200.200.101
Tuple plus MAC addresses Baremetal
Host A DIP 10.200.200.104 • Layer-2 Source MAC (SMAC)Host B

MAC: 0000.3001.1101 Protocol UDP • Layer-2 Destination MACMAC: (DMAC)
0000.3001.1102
IP: 192.168.10.101 IP: 192.168.10.102
SPORT 52562 • Layer-3 Source IP (SIP)
Original Packet
DPORT 4789
Overlay • Layer-3 Destination IPOriginal
(DIP) Packet
SMAC 0000.3001.1101
VXLAN 30001 • IP Protocol SMAC 0000.3001.1101
DMAC 0000.3001.1102 Leaf Leaf DMAC 0000.3001.1102

Inner-Header
• Protocol Source Port (SPORT)
dot1Q 10 dot1Q 10
SIP 192.168.10.101
SMAC 0000.3001.1101
• Protocol Destination Port
SIP
(DPORT)
192.168.10.101
DMAC 0000.3001.1102
• Some Platforms include IEEE

DIP 192.168.10.102 DIP 192.168.10.102
dot1Q 10
Protocol TCP Protocol TCP
SIP 192.168.10.101
SPORT 56425
DIP 192.168.10.102 802.1q information (VLAN ID) SPORT 56425
DPORT 25 DPORT 25
Protocol TCP
SPORT 56425 • Various Hashing Algorithms

DPORT 25
Path Diversity – Underlay Example (Layer-3
ECMP)
Outer-Header
Baremetal
SIP 10.200.200.101 Baremetal
Spine
Host A DIP 10.200.200.104 • Layer-3 ECMP Hashing is Host B
MAC: 0000.3001.1101 MAC: 0000.3001.1102
typically based on 5-Tuple
Protocol UDP
IP: 192.168.10.101 IP: 192.168.10.102
SPORT 52562
Original Packet • Layer-3 Source IP (SIP)
Original Packet
SMAC 0000.3001.1101
DPORT
VXLAN
4789
30001
Underlay • Layer-3 Destination IPSMAC
(DIP) 0000.3001.1101
DMAC 0000.3001.1102 Leaf Spine • IP Protocol
Leaf DMAC 0000.3001.1102
dot1Q 10 • Protocol Source Port (SPORT)

dot1Q 10
SIP 192.168.10.101
• Protocol Destination Port
SIP
(DPORT)
192.168.10.101
DIP 192.168.10.102 DIP 192.168.10.102
Protocol TCP
• Platform specific Protocol TCP
SPORT 56425 SPORT 56425
DPORT 25 DPORT 25
10101011010101
010101010
Baremetal Baremetal
Host A Spine
Spine Host B
MAC: 0000.3001.1101 MAC: 0000.3001.1102
IP: 192.168.10.101 IP: 192.168.10.102
Leaf Leaf
Spine
Spine
10101011010101
010101010
Baremetal
• VXLAN uses variable UDP Host B

Baremetal
Host A Spine
Spine
MAC: 0000.3001.1101
IP: 192.168.10.101
Source Port in Outer Header
MAC: 0000.3001.1102
IP: 192.168.10.102
• Hash of the inner Layer-

Leaf
Spine
2/Layer-3/Layer-4
Leaf Headers of
the original Ethernet Frame
Spine
• Enables entropy for ECMP Load

balancing in the Network
Agenda


• EVPN Multi-Site
• Examples
Ping/Traceroute in the Overlay – Bridged
--- 192.168.10.102 ping statistics ---

Baremetal Baremetal
Host A Host B
192.168.10.101 192.168.10.102
VLAN 10 VLAN 10
Ping/Traceroute in the Overlay – Bridged
--- 192.168.10.102 ping statistics ---

tc@boxA:$̃ traceroute HostB

Leaf
Leaf Leaf Leaf Leaf Leaf
traceroute to Leaf
HostB (192.168.10.102), 30 hops max, 38 byte packets
1 192.168.10.102 (192.168.10.102) 0.302 ms 0.251 ms 0.259 ms
tc@boxA:$̃
Baremetal Baremetal
Host A Host B
192.168.10.101 192.168.10.102
VLAN 10 VLAN 10
FYI
Centralized IP Gateway – VXLAN Flood & Learn
• Centralized First-Hop Routing

Spine Spine Spine Spine on a Set of Devices
• First-Hop Redundancy Protocol
192.168.20.1
192.168.10.1 0005.73A0.0014 (FHRP) Approach – i.e. HSRP,
0005.73A0.000A
VRRP
Leaf Leaf Leaf Leaf Leaf Leaf Leaf • Gateway is always active?!
• Depends on VPC
• Centralized, Inter-VLAN/VNI
VPC+HSRP
Routing
• Centralized MAC & ARP State
• Large Configuration State at GW
Ping/Traceroute in the Overlay – Routed with
HSRP
tc@boxA:$̃ ping HostC
PING HostC (192.168.20.101): 56 data bytes
Spine Spine Spine VNI 3002 (L2VNI)
Spine
--- 192.168.20.101 ping statistics ---

VNI 3001 (L2VNI)
tc@boxA:$̃
VPC+HSRP
Baremetal Baremetal
Host A Host C
192.168.10.101 192.168.20.101
VLAN 10 VLAN 20
Ping/Traceroute in the Overlay – Routed with
HSRP
Spine Spine Spine VNI 3002 (L2VNI)
Spine
--- 192.168.20.101 ping statistics ---

VNI 3001 (L2VNI)
tc@boxA:$̃
tc@boxA:$̃ traceroute HostC

traceroute to HostC (192.168.20.101), 30 hops max, 38 byte packets
1 192.168.10.1 (192.168.10.1) 0.593 ms 0.303 ms 0.312 ms
2 192.168.20.1 (192.168.20.1) 0.661 ms 0.400 ms 0.378 ms
3 192.168.20.101 (192.168.20.101) 0.509 ms 0.387 ms 0.217 ms
VPC+HSRP
tc@boxA:$̃
Baremetal Baremetal
Host A Host C
192.168.10.101 192.168.20.101
VLAN 10 VLAN 20
Distributed IP Anycast Gateway – VXLAN EVPN
• Distributed First-Hop Routing on

Edge Device
192.168.20.1 • All Edge Device share same

192.168.10.1 2020.0000.AAAA
2020.0000.AAAA Gateway IP and MAC address
• Pervasive Gateway approach
Leaf Leaf
• Gateway is always active

• No redundancy protocol for

hello or state exchange
• Distributed and smaller state

• Only local Endpoints ARP entries
Distributed IP Anycast Gateway – VXLAN EVPN
• Distributed First-Hop Routing on

Edge Device
192.168.10.1
192.168.20.1
2020.0000.AAA
• All Edge Device share same
2020.0000.AAA
A
A Gateway IP and MAC address
• Pervasive Gateway approach
Leaf Leaf
• Gateway is always active

• No redundancy protocol for

hello or state exchange
• Distributed and smaller state

• Only local Endpoints ARP entries
FYI
Anycast – One-to-Nearest Association
Spine Spine Spine Spine • Network Addressing and

Routing Methodology
Overlay
• Datagrams sent from a single
Leaf Leaf
Sender to the Topologically
Leaf Leaf
Nearest Node
Leaf Leaf Leaf
• Group of potential Receivers, all

identified by the same
Baremetal Baremetal
Baremetal
Destination Address
Host A Host B Host C
*L3VNI: VNI for all Routing operation (”VRF-VNI”) 33

BRKDCN-2125 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
Ping/Traceroute in the Overlay – Routed with EVPN

Spine Spine
Spine Spine
Overlay
--- 192.168.20.101 ping statistics ---
VNI 50000 (L3VNI) 2 packets transmitted, 2 packets received, 0% packet loss
tc@boxA:$̃
Baremetal Baremetal
Host A Host C
192.168.10.101 192.168.20.101
VLAN 10 VLAN 20
*L3VNI: VNI for all Routing operation (”VRF-VNI”) BRKDCN-2125 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
Ping/Traceroute in the Overlay – Routed with EVPN

Spine Spine
Spine Spine
Overlay
--- 192.168.20.101 ping statistics ---
VNI 50000 (L3VNI) 2 packets transmitted, 2 packets received, 0% packet loss
tc@boxA:$̃

tc@boxA:$̃ traceroute HostC
traceroute to HostC (192.168.20.101), 30 hops max, 38 byte packets
1 192.168.10.1 (192.168.10.1) 0.593 ms 0.303 ms 0.312 ms
2 192.168.20.1 (192.168.20.1) 0.661 ms 0.400 ms 0.378 ms
3 192.168.20.101 (192.168.20.101) 0.509 ms 0.387 ms 0.217 ms
tc@boxA:$̃
Baremetal Baremetal
Host A Host C
192.168.10.101 192.168.20.101
VLAN 10 VLAN 20
Distributed Anycast Gateway and Ping
• Distributed Anycast Gateway

Spine Spine Spine Spine only exists towards LAN
Segment
192.168.10.1
2020.0000.AAAA
Overlay • The Distributed Anycast
Gateway MAC and IP is not
advertised over BGP EVPN
RR# show bgp l2vpn evpn 2020.0000.AAAA

BGP routing table information for VRF default, address family L2VPN EVPN
RR#
Baremetal Baremetal
RR# show bgp l2vpn evpn 192.168.10.1

Host A Host C
192.168.10.101
BGP routing table information for VRF default, address family L2VPN EVPN
192.168.20.101
VLAN 10 RR# VLAN 20
Ping To Distributed Anycast Gateway – Local
• Simple Ping on LAN Segment

• Distributed Anycast Gateway is

Overlay represented by an SVI (Switch
192.168.10.1
2020.0000.AAAA
Virtual Interface)
0000.3001.1101 2020.0000.AAAA 10 192.168.10.101 192.168.10.1 Echo

Request
Baremetal

Host A
2020.0000.AAAA 0000.3001.1101 10 192.168.10.1 192.168.10.101 Echo
192.168.10.101 Reply
VLAN 10
Ping To Distributed Anycast Gateway – VPC
• Simple Ping on LAN

Spine Spine Spine Spine Segment
Overlay • Distributed Anycast

192.168.10.1
2020.0000.AAAA
192.168.10.1
2020.0000.AAAA
Gateway is represented by
VPC Leaf Leaf Leaf Leaf
an SVI (Switch Virtual
Leaf Leaf Leaf
Interface)
• Port-Channel hashing
decides to which Switch to
0000.3001.1101 2020.0000.AAAA 10 192.168.10.101 192.168.10.1 Echo
Baremetal
Request
go
Host A
2020.0000.AAAA 0000.3001.1101 10 192.168.10.1 192.168.10.101 Echo
192.168.10.101 Reply
VLAN 10
Ping From Distributed Anycast Gateway – VPC
• Ping from Distributed

Spine Spine Spine Spine Anycast Gateway (SVI) to
Local Host
Overlay
192.168.10.1
2020.0000.AAAA
192.168.10.1
2020.0000.AAAA
• From Switch to Endpoint;
VPC Leaf Leaf Leaf Leaf Leaf
local virtual Port-Channel
Leaf Leaf
(VPC) interface is used

• From Endpoint to Leaf;
Port-Channel hashing
0000.3001.1101 2020.0000.AAAA 10 192.168.10.101 192.168.10.1 Echo
Baremetal
Reply
decides what Interface to
use
Host A
2020.0000.AAAA 0000.3001.1101 10 192.168.10.1 192.168.10.101 Echo
192.168.10.101
VLAN 10
Request • No response if Source and
Destination Switch is different
Ping To Distributed Anycast Gateway – Local
• Simple Ping on LAN

Segment across IP Subnet
• Distributed Anycast
192.168.20.1
192.168.10.1
2020.0000.AAAA
Overlay
2020.0000.AAAA
Gateway is represented by
Leaf
an SVI (Switch Virtual
Interface)
Leaf Leaf Leaf Leaf Leaf Leaf
• Local Inter-VLAN Routing

to reach Local SVI
0000.3001.1101 2020.0000.AAAA 10 192.168.10.101 192.168.20.1 Echo

Request
Baremetal

Host A
2020.0000.AAAA 0000.3001.1101 10 192.168.20.1 192.168.10.101 Echo
192.168.10.101 Reply
VLAN 10
Ping To Distributed Anycast Gateway – Remote
192.168.20.1/32 (?)
• Ping into the Blue

Spine
• Distributed Anycast
Spine Spine Spine
Gateway can be on any

192.168.10.1
2020.0000.AAAA
Overlay 192.168.20.1
2020.0000.AAAA
Switch
Leaf Leaf Leaf Leaf Leaf Leaf Leaf • Distributed Anycast
Gateway is only advertised
as IP Subnet (direct route)
0000.3001.1101 2020.0000.AAAA 10 192.168.10.101 192.168.20.1 Echo

Request • VXLAN Entropy and ECMP
Hashing defines Destination
Baremetal Baremetal
Host A Host C
192.168.10.101 192.168.20.101
VLAN 10 SMAC DMAC dot1Q SIP VLANDIP20 ICMP
2020.0000.AAAA 0000.3001.1101 10 192.168.20.1 192.168.10.101 Echo

Reply
Ping From Distributed Anycast Gateway – Remote
• Ping from Distributed Anycast

Gateway (SVI) to Remote Host
• From Switch to Endpoint;
192.168.10.1
2020.0000.AAAA
Overlay
192.168.10.1
2020.0000.AAAA
192.168.20.1
2020.0000.AAAA
shortest path
• From Endpoint to Leaf;
response could go to any
Switch with Distributed

Anycast Gateway IP Address
• No response if Source and
2020.0000.AAAA 0000.3002.2101 10 192.168.10.1 192.168.20.101 Echo
Request Destination Switch is different
Baremetal Baremetal

Host A Host C
192.168.10.101 0000.3002.2101 2020.0000.AAAA 10 192.168.20.101
192.168.20.101 192.168.10.1 Echo
VLAN 10 VLAN 20 Reply
Problem – Predictable Overlay Connectivity Test
From To Comment Result Predictability

• Endpoint to SVI Connectivity Endpoint SVI Local Switch OK 100%
Tests are not consistent with Same Subnet

SVI Endpoint Local Switch OK 100%
”traditional” Networking Same Subnet
• Nature of Anycast IP Addressing Endpoint SVI VPC Port-Channel

Same Subnet
OK 100%
• Overlay Entropy SVI Endpoint VPC Port-Channel Not OK <50%

• ECMP Hashing Same Subnet
• Port-Channel Hashing Endpoint SVI Local Switch

Different Subnet
OK 100%
SVI Endpoint Local Switch OK 100%

Different Subnet
Endpoint SVI Remote Switch OK 100%

Different Subnet
SVI Endpoint Remote Switch Not OK <50%

Different Subnet
Ping From/To Loopback – Local with/without VPC
• Simple Ping to a Routed IP

Address
10.50.1.12 Overlay 10.50.1.15 10.50.1.18 • Uses Distributed Anycast

10.50.1.11
Gateway to Reach Local or
Remote Loopbacks
Leaf
VPC Leaf Leaf Leaf Leaf Leaf Leaf
• Allows most Flexible

Connectivity Tests
Baremetal Baremetal
interface loopback10
Host A Host C
192.168.10.101 192.168.20.101 vrf member BLUE
VLAN 10 VLAN 20 ip address 10.50.1.L#/32 tag 12345
Solution – Predictable Overlay Connectivity Test
From To Comment Result Predictability

• Avoid False Positives Endpoint Loopback Local Switch OK 100%
• Create a per-VRF Loopback Same Subnet
Loopback Endpoint Local Switch OK 100%
• Execute Connectivity Tests against Same Subnet
the Loopback Endpoint Loopback VPC Port-Channel OK 100%
Same Subnet
• Loopback Connectivity Tests - Loopback Endpoint VPC Port-Channel

Same Subnet
OK 100%
100% Predictable in any Case Endpoint Loopback Local Switch OK 100%

• Endpoint to Loopback Different Subnet
• Loopback to Endpoint Loopback Endpoint Local Switch

Different Subnet
OK 100%
• Loopback to Loopback Endpoint Loopback Remote Switch OK 100%

Different Subnet
• A Loopback can Save your Day! Loopback Endpoint Remote Switch OK 100%
Different Subnet
Operations,
Administration,
and Management
(OAM)
Agenda


• EVPN Multi-Site
• Examples
Operations, Administration, and Management
(OAM)
• OAM – processes, activities,
tools and standards
• Various Modes of Operation
• Pro-Active
• Controlling a Situation
• Re-Active
• Responding to a Situation
VXLAN OAM – Re-Active
V V V V V V V
Ping / Path MTU Pathtrace

• Check liveliness of End- • Trace paths to End-Host
Host and Tunnel-Endpoint
• Option to specify • Get Path, Interface and
Payload Parameters Error statistics along
path
• Specify Payload
Parameters for Path
Selection
VXLAN OAM – Pro-Active
V V V V V V V
Endpoint Locator Pro-Active Monitoring

• Locate End-Host and • Proactive Monitoring
Segment Identifier with Threshold and State
Notifications
• Track History of End-
Host
• Provide Fabric Host-
Count and Activity
VXLAN OAM - OAM Model of Operation
V V V V V V V
Endpoint Locator Ping / Path MTU Pathtrace Pro-Active Monitoring

• Locate End-Host and • Check liveliness of End- • Trace paths to End-Host • Proactive Monitoring
Segment Identifier Host and Tunnel-Endpoint with Threshold and State
Notifications
• Track History of End- • Option to specify • Get Path, Interface and
Host Payload Parameters Error statistics along
path
• Provide Fabric Host-
Count and Activity • Specify Payload
Parameters for Path
Selection
Agenda


• EVPN Multi-Site
• Examples
NGOAM or VXLAN OAM
• Next Generation OAM for Data

Center Fabrics
• Running on Nexus 9000, Nexus
7000, Nexus 5600, Nexus 3000
• VXLAN Today
• All IP Tomorrow
• Various Methods to Execute and

Retrieve Data
• Command Line Interface (CLI)
• NX-API
• DCNM (using NX-API)
VXLAN OAM – Pre-Requisites
• Enable “feature ngoam”

Spine Spine Spine Spine • Required on VTEPs and
intermediate Devices (i.e.
Overlay Spines)
• Activate OAM filters

Leaf Leaf Leaf Leaf Leaf Leaf Leaf • Configure “ngoam install acl”
• Required on all VTEPs
• Have a Loopback with Unique IP

• Source for Loopback Messages
Baremetal Baremetal
Baremetal • VRF-aware (per VRF Loopback)
Host A Host B Host C • IP must be reachable in Overlay
MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101
VXLAN OAM – Pre-Requisites (Command Line)
feature ngoam feature ngoam
Overlay
feature ngoam Leaf Leaf Leaf Leaf Leaf Leaf Leaf feature ngoam
ngoam install acl ngoam install acl
interface loopback 10 interface loopback 10

vrf member BLUE vrf member BLUE
ip address 10.50.1.15/32 ip address 10.50.1.18/32
Baremetal Baremetal
Baremetal

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101
VXLAN OAM – Reachability Verification
L15# ping 10.50.1.18 source 10.50.1.15 vrf BlUE PING

10.50.1.18 (10.50.1.18) from 10.50.1.15: 56 data bytes
64 bytes from 10.50.1.18: icmp_seq=0 ttl=254 time=1.041 ms
Overlay
--- 10.50.1.18 ping statistics ---
L15# show ip route 10.50.1.18 vrf BLUE 5 packets transmitted, 5 packets received, 0.00% packet loss
IP Route Table for VRF "BLUE” round-trip min/avg/max = 0.704/0.92/1.303 ms
'*' denotes best ucast next-hop
'**' denotes best mcast
Leaf next-hop
Leaf Leaf Leaf Leaf Leaf Leaf
'[x/y]' denotes [preference/metric]

'%<string>' in via output denotes VRF <string>
10.50.1.18/32, ubest/mbest: 1/0

*via 10.200.200.18%default, [200/0], 00:11:11, bgp-65501,
internal, tag 65501 (evpn) segid: 50001 tunnelid: 0xac8c812 encap: VXLAN
BGP-EVPN: VNI=50001
Baremetal
(EVPN) Baremetal
Baremetal
client-specific data: 4d
recursive
MAC:next hop: 10.200.200.18/32%default
0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
extendedIP:route information: BGPIP:origin
192.168.10.101 AS 65501 BGP peerIP:AS
192.168.10.102 65501
192.168.20.101
Endpoint Reachability – VXLAN OAM
Spine Spine Spine Spine • Endpoint Reachability

Is Host A • Uses ICMP
Overlay alive? • VTEP to Endpoint reachability
• VTEP to VTEP reachability
Leaf Leaf Leaf Leaf Leaf Leaf Leaf • Validates ECMP Path
• Single Random Path
• Multiple, Random/Specified Path
• Provides VXLAN Outer UDP

Baremetal Baremetal
Baremetal Source Port (SPORT) as output
MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101
Spine
Is Host A alive?
VTEP VTEP
Baremetal AS#65500
L15# ping nve ip 192.168.10.101 vrf BLUE source 10.50.1.15 verbose
Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
Host A 'D' - Destination Unreachable, 'X' - unknown return code,
MAC: 0000.3001.1101 'm' - malformed request(parameter problem),
IP: 192.168.10.101
'c' - Corrupted Data/Test, '#' - Duplicate response
Spine
Sender handle: 32
! sport 35977 size 56,Reply from 192.168.10.101,time = 1 ms
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/2 ms

Total time elapsed 89 ms
VTEP Reachability – VXLAN OAM
Spine
Is VTEP18
Loopback10 alive?
IP: 10.50.1.18
L15# ping nve ip 10.50.1.18 vrf BLUE source 10.50.1.15 sport 41803 verbose
VTEP VTEP
Codes: '!' - success, 'Q' - request not sent, '.' - AS#65500
timeout,
'D' - Destination Unreachable, 'X' - unknown return code,
'm' - malformed request(parameter problem),
Sender handle: 62
! sport 41803 size 56,Reply from 10.50.1.18,time = 1 msSpine
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

Total time elapsed 87 ms
CLI Options – ICMP-based NVE Ping
ping nve ip Destination Host/Loopback vrf VRF-Name source Source Loopback verbose
ping nve ip Destination Host/Loopback vrf VRF-Name source Source Loopback sport Outer Source Port verbose
ping nve ip Destination Host/Loopback vrf VRF-Name source Source Loopback egress Uplink Interface verbose
• Issues Ping to Host or Loopback • Use a specific VXLAN Outer

IP address Source Port
• Otherwise Random Generated
• Specifies the VRF where Source VXLAN Source Ports are used
and Destination Endpoint exists
• Use specific egress Interface
• Choose the local Loopback IP as • i.e. Uplink towards Spine
a Source IP address for the NVE • Otherwise ECMP hashing is used
Ping with Random or defined VXLAN
Source Port
CLI Options – NVE Ping with MAC
ping nve mac Destination Host MAC Local-VLAN profile Profile # verbose
ping nve mac Destination Host MAC Local-VLAN profile Profile # sport Outer Source Port verbose
ping nve mac Destination Host MAC Local-VLAN profile Profile # egress Uplink Interface verbose
• Issues Ping to Destination MAC • OAM response returned from

destination leaf
• Input requires VLAN mapped
with L2VNI where Destination • Use specific egress Interface
MAC resides • Otherwise ECMP hashing is used
with Random or defined VXLAN
• Uses nv03 tissa draft Source Port
ngoam profile 4
oam-channel 2
Endpoint Traceroute – VXLAN OAM
Spine Spine Spine Spine • Endpoint Traceroute

• Uses ICMP
What is the
Overlay Path to Host A? • VTEP to Endpoint
• VTEP to VTEP
Leaf Leaf Leaf Leaf Leaf Leaf Leaf • Validates Overlay Path
• Single Specified Path
• Multiple, Specified Path
• Provides Overlay to Underlay

Baremetal Baremetal
Baremetal correlation
MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101
How would a normal Traceroute look like?
Eth1/5
10.1.1.17
Spine
NVE1
10.200.200.18 What is the
L15# traceroute 192.168.10.101 source 10.50.1.15 vrf BLUE Path to Host A?
traceroute to 192.168.10.101 (192.168.10.101) from 10.50.1.15 (10.50.1.15), 30 hops max, 40 byte packets
VTEP VTEP
1 10.50.1.18 (10.50.1.18) 0.96 ms 0.817 ms 0.746 ms
2 Baremetal
2 192.168.10.101 (192.168.10.101) 4.751 msAS#65500
0.69 ms 0.697 ms
Host A
MAC: 0000.3001.1101
IP: 192.168.10.101
Spine
Eth1/5
10.1.2.17
Which Path did my Traceroute take? © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
Endpoint Traceroute – VXLAN OAM
Eth1/5
10.1.1.17
Spine
NVE1
10.200.200.18 What is the
Path to Host A?
L15# traceroute nve ip 192.168.10.101

VTEP vrf BLUE source 10.50.1.15 sport 35977 verboseVTEP
Baremetal AS#65500
Host A
'm' - malformed request(parameter
MAC: 0000.3001.1101 problem),
IP: 192.168.10.101
Spine
Traceroute Request to peer ip 10.200.200.18 source ip 10.200.200.15
Sender handle: 94
1 !Reply from 10.1.1.17,time = 1 ms
Eth1/5
3 !Reply from 192.168.10.101,time = 4 ms 10.1.2.17
Endpoint Traceroute – VXLAN OAM – Close-Up
L15# traceroute nve ip 192.168.10.101 vrf BLUE source 10.50.1.15 sport 35977 verbose

Traceroute Request to peer ip 10.200.200.18 source ip 10.200.200.15

Sender handle: 94
1 !Reply from 10.1.1.17,time = 1 ms Spine Ingress Interface IP
Destination VTEP IP
Host A IP
Spine Ingress Interface and Destination VTEP IP Address
are Underlay Information – additions vs. standard Traceroute
CLI Options – ICMP-based NVE Traceroute
traceroute nve ip Destination Host/Loopback vrf VRF-Name source Source Loopback verbose
traceroute nve ip Destination Host/Loopback vrf VRF-Name source Source Loopback sport Outer Source Port verbose
traceroute nve ip Destination Host/Loopback vrf VRF-Name source Source Loopback egress Uplink Interface verbose
• Issues Traceroute to Host or • Use a specific VXLAN Outer

Loopback IP address Source Port
• Otherwise Random Generated
• Specifies the VRF where Source VXLAN Source Ports are used
and Destination Endpoint exists
• Use specific egress Interface
• Chose the local Loopback IP as • i.e. Uplink towards Spine
a Source IP address for the NVE • Otherwise ECMP hashing is used
Ping with Random or defined VXLAN
Source Port
Ping/Traceroute with DCNM
Source Switch
Destination Switch
VRF
Ping/Traceroute with DCNM
Demo – VXLAN OAM NVE Ping and
Traceroute
Pathtrace for Enhanced Network Visibility
What is the Path • Application Specific Pathtrace

from Host C to
Uses “draft-tissa-nvo3-oam-fm”
Host A for HTTP? •

• Endpoint to Endpoint Pathtrace
Overlay • Adds Interface Load and Error
Statistics of the Path
• Uses Protocol Information
• Validates Specific or All Paths

• Provides Overlay to Underlay
correlation
Baremetal Baremetal
Baremetal
• Superset of NVE Ping/Traceroute
MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.10.102 IP: 192.168.20.101
Spine
Is Host A alive?
VTEP
AS#6550 VTEP
Baremetal
0 Baremetal
Host A Host C
MAC: 0000.3001.1101 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.20.101
Spine
Pathtrace with Known VTEP
Eth1/5
10.1.1.17
What is the Path from
Host C to Host A for
HTTP traffic?
Spine
NVE1
10.200.200.18 NVE1
10.200.200.15
L15# pathtrace nve ip 10.200.200.18 vrf BLUE payload ip 192.168.10.101 192.168.20.101 port 54321 80 proto 6 payload
VTEP VTEP
'!' - success, 'Q' - request not sent, '.' AS#65500
Codes:Baremetal - timeout,
Baremetal
'm' -Host
malformed
A request(parameter problem),
Host C
MAC: 0000.3001.1101 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.20.101
Path trace Request to peer ip 10.200.200.18 source ip 10.200.200.15
Sender handle: 142 Spine
Hop Code ReplyIP IngressI/f EgressI/f State

====================================================
1 !Reply from 10.1.1.17, Eth1/5 Eth1/8 UP / UP Eth1/5
10.1.2.17
2 !Reply from 10.200.200.18, Eth1/54 Unknown UP / DOWN
Pathtrace with Unknown VTEP
Eth1/5
10.1.1.17
What is the Path
from Host C to
Host A for HTTP?
Spine
NVE1
10.200.200.18 NVE1
10.200.200.15
L15# pathtrace nve ip unknown vrf BLUE payload ip 192.168.10.101 192.168.20.101 port 54321 80 proto 6 payload-end
Codes: '!' - success, 'Q' - request not sent, '.' - AS#6550

VTEP VTEP
timeout,
Baremetal
'D' - Destination 0
Unreachable, 'X' - unknown return code, Baremetal

Host A
'c' - Corrupted Data/Test, '#' - Duplicate response Host C
MAC: 0000.3001.1101 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.20.101
Sender handle: 142 Spine

====================================================
1 !Reply from 10.1.1.17, Eth1/5 Eth1/8 UP / UP Eth1/5
2 !Reply from 10.200.200.18, Eth1/54 Vlan10 UP / UP 10.1.2.17
Pathtrace – VXLAN OAM – Close-Up
L15# pathtrace nve ip unknown vrf BLUE Known or Unknown VTEP IP Address
payload
ip 192.168.10.101 192.168.20.101 Destination Endpoint IP / Source Endpoint IP
port 54321 80 Source Port / Destination Port
proto 6
payload-end TCP (IANA Protocol Number 6)


Sender handle: 142

====================================================
1 !Reply from 10.1.1.17, Eth1/5 Eth1/8 UP / UP
2 !Reply from 10.200.200.18, Eth1/54 Vlan10 UP / UP
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
Why are we specifying Payload information?
10101011010101
010101010
Baremetal
• VXLAN provides variable UDP

Baremetal
Host A Spine
Spine Host B
MAC: 0000.3001.1101
IP: 192.168.10.101
Source Port in Outer Header
MAC: 0000.3001.1102
IP: 192.168.10.102
• Hash of the inner Layer-

Leaf
Spine
2/Layer-3/Layer-4
Leaf Headers of
the original Ethernet Frame.
Spine
• Enables entropy for ECMP Load

balancing in the Network
Which Path did your Application Traffic take?
Pathtrace – VXLAN OAM – Close-Up
L15# pathtrace nve ip unknown vrf BLUE payload ip 192.168.10.101 ...

Spine IngressUnreachable,
'D' - Destination Interface, Egress
'X'Interface and return
- unknown Destination VTEP IP Address
code,
are Underlay Information – additions vs. standard and NVE Traceroute

Sender handle: 142
Spine Ingress Interface IP Spine Ingress Interface Spine Egress Interface

====================================================
1 !Reply from 10.1.1.17, Eth1/5 Eth1/8 UP / UP Interface Status
Destination VTEP IP Destination Leaf Ingress Interface
Pathtrace – VXLAN OAM – Extensions (Routing)
L15# pathtrace nve ip unknown vrf BLUE Known or Unknown VTEP IP Address
payload
port 54321 80 Source Port / Destination Port

proto 6
verbose Verbose (for additional information)

req-stats
Request Interface Statistics
Pathtrace – VXLAN OAM – Extensions (Bridging)
L15# pathtrace nve ip unknown Known or Unknown VTEP IP Address

payload
mac 0000.3001.1101 0000.3001.1102 Destination Endpoint MAC / Source Endpoint MAC
port 54321 80
proto 6 Source Port / Destination Port
vni 30010
verbose Layer-2 VNI (in case of Bridging)
req-stats Verbose (for additional information)
Request Interface Statistics
Pathtrace with Unknown VTEP – Request
Statistics
Eth1/5
10.1.1.17
What is the Path
from Host C to
Host A for HTTP?
Spine
NVE1
10.200.200.18 NVE1
10.200.200.15
L15# pathtrace nve ip unknown vrf BLUE payload ip 192.168.10.101 192.168.20.101 payload-end verbose req-stats
VTEP
AS#6550
VTEP
Baremetal
Sender handle: 168 0 Baremetal
Hop Host A ReplyIP

Code IngressI/f EgressI/f State Host C
MAC: 0000.3001.1101
==================================================== MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.20.101
1 !Reply from 10.1.1.17, Eth1/5 Eth1/8 UP / UP
Input Stats: PktRate:0 ByteRate:0 Load:0 Bytes:66113123Spine
unicast:140952 mcast:252611 bcast:2 discards:0 errors:0 unkno
Output Stats: PktRate:0 ByteRate:0 load:0 bytes:51359028 unicast:100504 mcast:252545 bcast:6 discards:0 errors:0 band

Eth1/5
Input Stats: PktRate:0 ByteRate:0 Load:0 Bytes:52302926 unicast:99998 mcast:263225 bcast:4 discards:0 errors:0 unknow
10.1.2.17
Request Stats – VXLAN OAM – Close-Up
L15# pathtrace nve ip unknown vrf BLUE

payload ip 192.168.10.101 192.168.20.101 payload-end
verbose req-stats

Sender handle: 168 OAM Session ID

==================================================== Interface
1 !Reply from 10.1.1.17, Eth1/5 Eth1/8 UP / UP Statistics
Input Stats: PktRate:0 ByteRate:0 Load:0 Bytes:66113123 unicast:140952 mcast:252611 bcast:
Output Stats: PktRate:0 ByteRate:0 load:0 bytes:51359028 unicast:100504 mcast:252545 bcast

Input Stats: PktRate:0 ByteRate:0 Load:0 Bytes:52302926 unicast:99998 mcast:263225 bcast:4
Database Output – VXLAN OAM – Close-Up
L15# show ngoam pathtrace database session 168 detail
Pathtrace entry for session id 168 OAM Session ID

================================
Start time: Tue Jun 13 01:18:39.710 PDT

End time: Tue Jun 13 01:18:39.735 PDT
Last Clear of Summary Statistics: Never

Pathtrace Requests: sent (2)/received (0)/timeout (0)/unsent (0)
Pathtrace Replies: sent (0)/received (2)/unsent (0)/Duplicate (0)
! Reply from 10.1.1.17 on Eth1/5, state UP. Sent on Eth1/8, state UP.
Interface stats for interface: Eth1/5
-------------------------------
Rx Len : 84
Rx Bytes : 66113123
Interface Statistics
Rx Pkt rate : 0
Rx Byte rate : 0
Rx Load : 0 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
Endpoint Pathtrace with DCNM
Source IP
Destination IP
VRF
Payload Information (Optional)
Endpoint Pathtrace with DCNM
Demo – VXLAN OAM NVE PathTrace
Agenda


• EVPN Multi-Site
• Examples
Where is my Endpoint?
found Overlay
• Find Your Endpoints

• Correlate All Network Related
Information for a given Endpoint
• Keep Endpoints Location History
Endpoint Locator (EPL) – Architecture
• Endpoint Locator (EPL)

• Application in DCNM
• Peers with the Overlay Control-
Plane (i.e. BGP EVPN)
• BGP Receiver only (Passive)
• Searchable and Scalable

Database for Real-Time and
Historic Data
BGP
• Stores every Endpoint Control-
Plane Event
EPL
• Correlates with Inventory Data
Endpoint Locator
Demo –Endpoint Locator with DCNM
Agenda


• EVPN Multi-Site
• Examples
EVPN Multi-Site
Site-External DCI
Border Gateways (IP Routing and Increased
(Key Functional MTU Support)
Components of
VXLAN Multi-Site
Architecture) BGW BGW BGW BGW
Spine Spine Spine Spine Spine Spine Spine Spine
Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf
Site 1 Site n
Site-Internal Fabric
(Common VXLAN and BGP-EVPN Functions)
Hierarchical Overlay Domains
Overlay Multi-Site
BGW BGW BGW BGW
Spine
Overlay Site 1
Overlay Site n
Spine Spine Spine
Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf
Site 1 Site n
Baremetal Baremetal
Y2018
VXLAN XConnect

• MPLS Pseudowire like
Tunneling with VXLAN
• Tunnel all control & data
VTEP VTEP VTEP VTEP VTEP VTEP VTEP
packets between VTEPs
• Attachment point is part of a
vlan 10 vlan 10
unique provider VNI
vn-segment 10000 vn-segment 10000
• P2P
xconnect xconnect
interface Ethernet1/1 interface Ethernet1/1
switchport mode dot1q-tunnel switchport mode dot1q-tunnel
switchport access vlan 10 switchport access vlan 10
Y2018
VXLAN XConnect
vlan 20
vn-segment 20000
xconnect
interface Ethernet1/2
switchport mode dot1q-tunnel
switchport access vlan 20
• MPLS Pseudowire like
Tunneling with VXLAN
• Tunnel all control & data
packets between VTEPs
• Attachment point is part of a
vlan 10 vlan 10
unique provider VNI
• P2P
xconnect xconnect
interface Ethernet1/1 vlan 20
switchport mode dot1q-tunnel vn-segment 20000
switchport access vlan 10 xconnect
switchport access vlan 20 BRKDCN-2125 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 94
OAM for VXLAN XConnect
vlan 20
vn-segment 20000
xconnect
• Pro-active OAM
• Monitor reachability to the

remote VTEPs/VNI
• Configurable heart-beat
interval
• Failure Propagation
vlan 10 vlan 10
xconnect
• Convey local vni/vlan/interface
xconnect
interface Ethernet1/1 vlan 20
vn-segment 20000
failures to the remote VTEPs
switchport access vlan 10 xconnect
• Error-disabled mode
switchport access vlan 20 BRKDCN-2125 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 95
Network Insights
Network Insights Applications
Apps
DCNM APIC
Platform
App Hosting Framework App Hosting Framework
App Store App Store
Data collection and ingestion Data correlation and analysis Data visualization and action
Visibility Insights Proactive Troubleshooting

Learn from your network and See problems before Find root cause faster with
recognize anomalies your end users do granular details
Data Center Visibility Use Cases
Network Health Path and Latency Network Performance

CPU and memory Measurement
• • Interface utilization
utilization • End-to-end visibility
• Buffer monitoring
• Forwarding table utilization • Path tracing over time
• Microburst detection
• Protocol state and events • Flow latency monitoring
• Drop event correlation
• Environmental data
Network Insights Resources
• Analysis and correlation of software and hardware telemetry data

with focus on Day 2 network operations use-cases
• Focus on identifying anomalies and providing quick drill-down to
specific issues
NIR
Network Insights Resources - Customer Benefits
Fabric-Wide Capacity
Resource Utilization
Planning, Trend Monitoring
Environmental
Avoid Environmental (CPU,
Monitoring Power, Memory, Fan, Storage
Related Failures
Network Identify/Predict Failing

Operations Statistics Devices
Insights
Resources
Troubleshoot Application Latency
Identify Traffic/Protocol
Flow Analytics behavior
Event Analytics Identify Subtle

Path-Related issues
Key Takeaways
• Modern Overlays require Modern Tools

• ICMP isn’t sufficient
• Who can Ping What, When?
• VXLAN OAM/NGOAM for Single & Multi-Site deployments

• Pro-active OAM
• Enables use cases like Xconnect tunnel monitoring
• Use all available tools for quicker resolution

• VXLAN OAM, Endpoint Locator, Network Insights with DCNM
Complete your
online session
survey • Please complete your session survey
after each session. Your feedback
is very important.
• Complete a minimum of 4 session
surveys and the Overall Conference
survey (starting on Thursday) to
receive your Cisco Live t-shirt.
• All surveys can be taken in the Cisco Events
Mobile App or by logging in to the Content
Catalog on ciscolive.com/emea.
Cisco Live sessions will be available for viewing on

demand after the event at ciscolive.com.
Continue your education
Demos in the
Walk-in labs
Cisco campus
Meet the engineer

Related sessions
1:1 meetings
Thank you

BRKDCN-2125

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

BRKDCN-2125

Uploaded by

Copyright:

Available Formats

Overlay Management and

Visibility with VXLAN

Shyam Kapadia, Principal Engineer

• Born & brought-up in India

• PhD in Computer Science from USC (Fight on!)

• Developer, Author, Inventor…

• Hobbies: Movies, Sports (Cricket, NFL, NBA)

• Overlays and Network abstraction

• Operations, Administration and Management (OAM)

• Overlays and Network abstraction

• Operations, Administration and Management (OAM)

Hypervisor Baremetal Hypervisor Hypervisor Baremetal Hypervisor Baremetal Baremetal

Hypervisor Baremetal Hypervisor Hypervisor Baremetal Hypervisor Baremetal Baremetal

Spine • Overlay – Tunnels

• Endpoint physically connected

• Leaf to Leaf Communication

• The Spine sees only

--- 192.168.10.102 ping statistics ---

Leaf Leaf Leaf Leaf Leaf Leaf Leaf

Spine Spine Spine Spine

SMAC DMAC dot1Q SIP DIP ICMP

SMAC DMAC dot1Q SIP DIP ICMP

Spine Spine Spine Spine

10.200.200.101 10.200.200.104 UDP 45631 4789

Leaf Leaf Leaf Leaf Leaf Leaf Leaf

(NVO) – Network Abstraction

from Physical Connectivity

DMAC 0000.3001.1102 Leaf Leaf DMAC 0000.3001.1102

• VXLAN Entropy is based on 5-

Host A DIP 10.200.200.104 • Layer-2 Source MAC (SMAC)Host B

DMAC 0000.3001.1102 Leaf Leaf DMAC 0000.3001.1102

• Some Platforms include IEEE

SPORT 56425 • Various Hashing Algorithms

dot1Q 10 • Protocol Source Port (SPORT)

DIP 192.168.10.102 DIP 192.168.10.102

SPORT 56425 SPORT 56425

• VXLAN uses variable UDP Host B

• Hash of the inner Layer-

• Enables entropy for ECMP Load

• Overlays and Network abstraction

• Operations, Administration and Management (OAM)

--- 192.168.10.102 ping statistics ---

Leaf Leaf Leaf Leaf Leaf Leaf Leaf

--- 192.168.10.102 ping statistics ---

tc@boxA:$̃ traceroute HostB

• Centralized First-Hop Routing

--- 192.168.20.101 ping statistics ---

Leaf Leaf Leaf Leaf Leaf Leaf Leaf

--- 192.168.20.101 ping statistics ---

tc@boxA:$̃ traceroute HostC

• Distributed First-Hop Routing on

192.168.20.1 • All Edge Device share same

• Gateway is always active

• No redundancy protocol for

• Distributed and smaller state

• Distributed First-Hop Routing on

• Gateway is always active

• No redundancy protocol for

• Distributed and smaller state

Spine Spine Spine Spine • Network Addressing and

• Group of potential Receivers, all

*L3VNI: VNI for all Routing operation (”VRF-VNI”) 33

tc@boxA:$̃ ping HostC

Leaf Leaf Leaf Leaf Leaf Leaf Leaf