Brkaci 2008

#CiscoLive
End-2-End Policy from the

Campus to the DC and Back, A
Packet Journey with SDA to ACI
Ramses Smeyers, Principal Consulting Engineer, CX

BRKACI-2008
#CiscoLive
Why do we Build Networks?
Doable..
but kind of
a science
project!
End-Users Data &

FW rules
Applications
#CiscoLive BRKACI-2008 © 2021 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Agenda
• Introduction
• Management and Policy Plane
• A day in the life of a packet
Introduction
Phase 2 (LA)
Limited Availability
General information
• Limited Availability is the phase in a product lifecycle, that comes before

“General Availability” phase
• It is necessary to approve the design and its limitations with Cisco BU
• The LA solutions are supported by Cisco TAC
• The software is already published on CCO and DC App Center
• Note: Partners/Customers must get documentation through Cisco
Good News
SD-Access and ACI Fabric Similarities
SD-Access Fabric ACI Fabric
• Underlay • Underlay
• Overlay • Overlay
• Logical constructs • Logical constructs
• Virtual Network • VRF
• SGT • EPG
• User Endpoint • App Endpoint
• Group Based Access Control • Contract
Phase 2 (LA)
SDA-ACI Phase 2 Architecture
MGMT. Kafka enables a scalable and
Kafka Bus
Controller APIC
1 PLANE open Pub/Sub-based messaging
Peering bus for controllers’ federation
App Groups
POLICY Enables identity federation and
User Groups
APIC
2 PLANE automated network hand-off with

domain admins in control
Campus SDA Fabric CONTROL Automated VRF-aware Network

BGP-EVPNVXLAN
3 PLANE Handoff with BGP EVPN
San Francisco New York City
ACI Data Center ACI Data Center
Users
LISP Underlay Network COOP Data plane based learning of IP
DATA
1
VXLAN SGT (16 bits) VXLAN SGT (16 bits) iVXLAN EPG (16 bits)
4 PLANE
to group bindings for scale &
Header VNID (24 bits) Header VNID (24 bits) Header VNID (24 bits)
simplicity
EPG-SGT Translate SGT-EPG Translate
Phase 2 (LA)
Software and Hardware Supported Versions
SDA-ACI Components Version
SDA Border ASR1001-X, ASR1001-HX, ASR1002-X, 17.3.2a

ASR1002-HX (8G min RAM)
CAT9500H
SDA Edge All SDA supported switches. FE-switches 17.3.2a
DNA-C 2.1.2.4
ISE 3.0 patch 1
APIC APIC-CLUSTER-M3 5.1(1h)

APIC-CLUSTER-L3
ACI Border Leaf C9300-GX/FX/FX2/FXP/EX 15.1(1h)
ACI Spine C9300-GX 15.1(1h)

C9300C
X9700-EX/FX/GX
MDC 1.0.0.188
MDM 1.0.0.188
Phase 2 (LA)
Scalability Values
Configurable Option Scale

Virtual Networks (VNs) extended into the ACI fabric 32
Remote EPGs (Security Group Tag (SGT)) 500
Remote SDA endpoints 256000*
Local EPGs 500
Local endpoints 16000
Contract rules 64000
* To achieve the maximum scale for learned endpoints, you must configure the switch's Forwarding Scale Profile
policy, choosing the High IPv4 EP profile that is supported only on FX, FXP and GX ACI leaf models
Management
and Policy Plane
Phase 2 Management Domain - Kafka
• The phase 2 management plane
uses a Kafka messaging bus Kafka Messaging Bus (VN, SGT/EPG Group, Contract*, …) Exchange
• Kafka is a pub/sub messaging bus

which runs on APIC Kafka Kafka Kafka
Each APIC in the APIC cluster will be a
Kafka broker
• Kafka clients subscribe (read) and pxGrid
publish (write) messages to Kafka

DNAC ISE MDC MDM
In this integration ISE and MDC are
Kafka clients
• DNAC does not communicate MDC (Multi-Domain Connector) and MDM (Multi-Domain
directly to Kafka. DNAC Manager) are two new apps that run on the APIC
communicates with ISE via pxGrid
and ISE communicates with Kafka
MDC and MDM Apps
• MDM
• Responsible for domain registration and management
• Manages Kafka topics
• Certificate Management
• MDM app must be installed for domain registration and management. No configuration is
done from the MDM app
• MDC
• APIC configuration, config-sync/recovery
• Manages all multi-domain workflows
• Class-id management
• All multi-domain configuration and management is done from the MDC app
Kafka Connections
• All APICs in the APIC cluster run Kakfa,

each APIC is a Kafka broker
• APICs communicate over the fabric infra
network on TCP port 9092
Infra network
• Kafka clients communicate to the brokers
over TCP port 9093 on the OOB TCP Port 9092 TCP Port 9092
management network APIC APIC APIC
• TCP port 9093 is not permitted by default

over the OOB management network OOB network
An OOB management contract is required to

TCP Port 9093 TCP Port 9093
allow TCP port 9093
• Domain registration will fail if TCP port 9093
is not permitted by an OOB management
contract (ACI domain and SDA domain) ISE MDM MDC
A day in the life
of a packet
Our setup for today
DNA-Center ISE
SGT/IP and EPG/IP
C
BGP/EVPN
VRF: Internal03 SD-Access

B
Fabric Site B L3
BL
OSPF OSPF
1/14
Doctor client PC VM
192.168.1.3/32 10.42.42.101
Consumer Provider
From SDA à
ACI
Doctor logs into PC C
SD-Access
B
Fabric Site B
edge-2#show run int GigabitEthernet 1/0/14

Building configuration...
SD-Access
B
Fabric Site B
Current configuration : 331 bytes
!
interface GigabitEthernet1/0/14
switchport access vlan 1021
switchport mode access
device-tracking attach-policy IPDT_POLICY
load-interval 30
dot1x timeout tx-period 7
dot1x max-reauth-req 3
no macro auto processing
source template DefaultWiredDot1xClosedAuth
spanning-tree portfast
spanning-tree bpduguard enable
end
edge-2#
SD-Access
B
Fabric Site B
SD-Access
B
Fabric Site B
edge-2#show authentication sessions interface GigabitEthernet 1/0/14

Interface MAC Address Method Domain Status Fg Session ID
------------------------------------------------------------------------------------
Gi1/0/14 a036.9f8f.7a72 dot1x DATA Auth 450210AC0000005BF320DD33
Doctor logs into PC
edge-2#show authentication sessions interface GigabitEthernet 1/0/14 details
Interface: GigabitEthernet1/0/14
IIF-ID: 0x1297C0CE C
MAC Address: a036.9f8f.7a72
IPv6 Address: Unknown
IPv4 Address: 192.168.1.3
User-Name: derek SD-Access
Device-type: Microsoft-Workstation B
Fabric Site B
Device-name: MSFT 5.0
Status: Authorized
Domain: DATA
Oper host mode: multi-auth
Oper control dir: both
Session timeout: N/A
Acct update timeout: 172800s (local), Remaining: 172770s
Common Session ID: 450210AC0000005BF320DD33
Acct Session ID: 0x00000007
Handle: 0x30000051
Current Policy: PMAP_DefaultWiredDot1xClosedAuth_1X_MAB
Local Policies:
Server Policies:
SGT Value: 16
Method status list:

Method State
dot1x Authc Success
edge-2#
SD-Access
B
edge-2#show cts role-based sgt-map vrf Internal03 all Fabric Site B
%IPv6 protocol is not enabled in VRF Internal03
Active IPv4-SGT Bindings Information
IP Address SGT Source

============================================
192.168.1.1 2 INTERNAL
192.168.1.3 16 LOCAL
IP-SGT Active Bindings Summary

============================================
Total number of LOCAL bindings = 1
Total number of INTERNAL bindings = 1
Total number of active bindings = 2
edge-2#
Doctor logs into PC
C
edge-2#show cts environment-data 12-00:Development_Servers
CTS Environment Data 13-00:Test_Servers
==================== 14-00:PCI_Servers
Current state = COMPLETE 15-00:BYOD SD-Access
B
Last status = Successful 16-00:Doctors Fabric Site B
Local Device SGT: 17-00:Nurses
SGT tag = 2-00:TrustSec_Devices 255-00:Quarantined_Systems
Server List Info: 10001-
Installed list: CTSServerList1-0001, 1 server(s): 00:Ecommerce_Web13ACISDA56ad54a9EPG
*Server: 172.16.201.217, port 1812, A-ID Environment Data Lifetime = 86400 secs
FD8E99B10C8188CA4F373AAB06C76091 Last update time = 15:05:08 UTC Mon Mar 1 2021
Status = ALIVE Env-data expires in 0:01:34:05 (dd:hr:mm:sec)
auto-test = TRUE, keywrap-enable = FALSE, idle- Env-data refreshes in 0:01:34:05 (dd:hr:mm:sec)
time = 60 mins, deadtime = 20 secs Cache data applied = NONE
Security Group Name Table: State Machine is running
0-00:Unknown
2-00:TrustSec_Devices
3-00:Network_Services
4-00:Employees
5-00:Contractors
6-00:Guests
7-00:Production_Users
8-00:Developers
9-00:Auditors
10-00:Point_of_Sale_Systems
11-00:Production_Servers
Doctor logs into PC
Doctor sends traffic – SDA side C
SD-Access
B
Fabric Site B
SD-Access
B
Fabric Site B
SD-Access
B
Fabric Site B
edge-2#show cts role-based permissions

IPv4 Role-based permissions default: SD-Access
B
Permit IP-00 Fabric Site B
RBACL Monitor All for Dynamic Policies : FALSE
RBACL Monitor All for Configured Policies : FALSE
edge-2#
SDA does egress policy

towards access ports
For ACI Integration Policy
is on ACI side
Doctor sends traffic – SDA side
C
edge-2#show ip route vrf Internal03
Routing Table: Internal03 SD-Access

Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP B
Fabric Site
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area B
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

E1 - OSPF external type 1, E2 - OSPF external type 2, m - OMP
n - NAT, Ni - NAT inside, No - NAT outside, Nd - NAT DIA
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
H - NHRP, G - NHRP registered, g - NHRP registration summary
o - ODR, P - periodic downloaded static route, l - LISP
a - application route
+ - replicated route, % - next hop override, p - overrides from PfR
& - replicated local route overrides by connected Doctor client PC
192.168.1.3/32
Gateway of last resort is not set
192.168.1.0/24 is variably subnetted, 3 subnets, 2 masks

C 192.168.1.0/24 is directly connected, Vlan1021
L 192.168.1.1/32 is directly connected, Vlan1021
l 192.168.1.3/32 [10/1] via 192.168.1.3, 00:11:47, Vlan1021
edge-2#
Doctor sends traffic – SDA side - LISP
C
edge-2#show run int vlan 1021
Building configuration...
SD-Access
Current configuration : 322 bytes B
Fabric Site B
!
interface Vlan1021
description Configured from Cisco DNA-Center
mac-address 0000.0c9f.fb98
vrf forwarding Internal03
ip address 192.168.1.1 255.255.255.0
ip helper-address 172.16.201.201
no ip redirects
ip route-cache same-interface
no lisp mobility liveness test
Doctor client PC
lisp mobility 192_168_1_0-Internal03-IPV4
192.168.1.3/32
end
edge-2#show ip vrf
Name Default RD Interfaces
Internal03 <not set> Vl1021
LI0.4099
Mgmt-vrf <not set> Gi0/0
edge-2#
C
edge-1# sh run | b lisp

router lisp
… SD-Access
B
instance-id 4099 Fabric Site B
remote-rloc-probe on-route-change
dynamic-eid 192_168_1_0-Internal03-IPV4
database-mapping 192.168.1.0/24 locator-set rloc_88061184-0323-4d56-82c5-49da9d05fde2
exit-dynamic-eid
!
service ipv4
eid-table vrf Internal03
map-cache 0.0.0.0/0 map-request
exit-service-ipv4
! Doctor client PC
exit-instance-id 192.168.1.3/32
!
edge-2#show lisp instance-id 4099 ipv4 database 192.168.1.3/32
LISP ETR IPv4 Mapping Database for EID-table vrf Internal03 (IID 4099), LSBs: 0x1 C
Entries total 2, no-route 0, inactive 0
192.168.1.3/32, dynamic-eid 192_168_1_0-Internal03-IPV4, inherited from default locator-set rloc_88061184-0323-4d56-82c5-

49da9d05fde2 SD-Access
Uptime: 00:17:21, Last-change: 00:17:21 B
Domain-ID: unset
Fabric Site B
Locator Pri/Wgt Source State
172.16.2.70 10/10 cfg-intf site-self, reachable
Map-server Uptime ACK Domain-ID
172.16.1.254 00:17:08 Yes 0
edge-2#
edge-2#sh ip int brief | i Loopback0

Loopback0 172.16.2.70 YES NVRAM up up
border01#sh ip lisp | i Map

Map Server (MS): enabled
Map Resolver (MR): enabled Doctor client PC
ITR Map-Resolver(s): 172.16.1.254
ETR Map-Server(s): 172.16.1.254
192.168.1.3/32
ITR Solicit Map Request (SMR): accept and process
Map-cache:
Map-cache limit: 32768
Map-cache activity check period: 60 secs
border01#sh lisp site | i Site|Register|192.168.1.3

LISP Site Registration Information
Site Name Last Up Who Last Inst EID Prefix
Register Registered ID
00:20:13 yes# 172.16.2.70:21016 4099 192.168.1.3/32
C
edge-2#lig instance-id 4099 10.42.42.101

Mapping information for EID 10.42.42.101 from 172.16.1.254 with RTT 1 msecs
0.0.0.0/1, uptime: 01:43:10, expires: 00:14:59, via map-reply, forward-native SD-Access
B
Encapsulating to proxy ETR Fabric Site B
edge-2#
edge-2#lig instance-id 4099 10.42.42.101

Mapping information for EID 10.42.42.101 from 172.16.1.254 with RTT 1 msecs
0.0.0.0/1, uptime: 01:43:10, expires: 00:14:59, via map-reply, forward-native
Encapsulating to proxy ETR
edge-2#sh ip lisp instance-id 4099 map-cache 10.42.42.101
LISP IPv4 Mapping Cache for EID-table vrf Internal03 (IID 4099), 6 entries
Doctor client PC
0.0.0.0/1, uptime: 01:43:29, expires: 00:14:39, via map-reply, forward-native 192.168.1.3/32
Sources: map-reply
State: forward-native, last modified: 01:43:29, map-source: 172.16.1.254
Idle, Packets out: 62(35712 bytes) (~ 00:10:13 ago)
Encapsulating to proxy ETR
edge-2#
Doctor sends traffic – SDA side – L2VPN
C
border01#show ip route vrf Internal03
Gateway of last resort is not set SD-Access

B
Fabric Site B
10.0.0.0/24 is subnetted, 1 subnets
B 10.42.42.0 [20/0] via 10.92.92.231, 1d00h, Vlan3002
B 172.16.201.0/24 [20/0] via 172.16.4.2, 1d01h
192.168.1.0/24 is variably subnetted, 3 subnets, 2 masks Doctor client PC
B 192.168.1.0/24 [200/0], 1d08h, Null0 192.168.1.3/32
C 192.168.1.1/32 is directly connected, Loopback1021
l 192.168.1.3/32 [250/1], 00:24:55, Null0
border01#
No egress policy on
border
Doctor sends traffic – SDA side - L2VPN
C
border01#show ip route vrf Internal03 10.42.42.101
Routing Table: Internal03 SD-Access

Routing entry for 10.42.42.0/24 B
Fabric Site B
Known via "bgp 65003", distance 20, metric 0
Tag 101, type external
Last update from 10.92.92.231 on Vlan3002, 11:41:20 ago
Routing Descriptor Blocks:
* 10.92.92.231 (default), from 10.92.92.228, 11:41:20 ago, via Vlan3002
opaque_ptr 0x7F31DCDA5720
Route metric is 0, traffic share count is 1
AS Hops 1
Route tag 101 Doctor client PC
MPLS label: none VM
192.168.1.3/32
MPLS Flags: NSF 10.42.42.101
border01#
Pointing to default
route table
ACI / SDA connection - Underlay
Cp-TEP
10.92.92.228/32
Eth 1/41
OSPF
OSPF 10.82.82.6/30
BL
Gi0/1/0.901 Leaf 103
G 1/0/24 10.82.82.5/30
VPC Domain
10.82.82.1/30
B G 0/2/1
F
L3
Anycast-TEP
10.92.92.231/32
10.82.82.2/30 Gi0/2/0.901
Loopback1 / Tunnel0
10.82.83.4/32
10.82.82.9/30
Eth 1/41
BL
10.82.82.10/30
Leaf 104
Cp-TEP
10.92.92.227/32
SDA L3 ACI
C
border01#show ip route

B
Fabric Site B
O 10.82.82.4/30 [110/2] via 10.82.82.2, 12:21:50, Vlan901
O 10.82.82.8/30 [110/2] via 10.82.82.2, 12:21:49, Vlan901
O 10.82.84.1/32 [110/3] via 10.82.82.2, 12:11:57, Vlan901
O 10.82.84.2/32 [110/3] via 10.82.82.2, 12:11:57, Vlan901
O E2 10.92.92.227/32 [110/20] via 10.82.82.2, 12:11:57, Vlan901 Doctor client PC
O E2 10.92.92.228/32 [110/20] via 10.82.82.2, 12:11:57, Vlan901 192.168.1.3/32
O E2 10.92.92.231/32 [110/20] via 10.82.82.2, 12:11:57, Vlan901
….
border01#
C
ACI à MDC à System Configuration SD-Access

B
Fabric Site B
Doctor client PC
192.168.1.3/32
interface nve1
no ip address C
source-interface Loopback1
host-reachability protocol bgp
vxlan udp port 48879
group-based policy
SD-Access
B
member vni 4099 vrf Internal03 Fabric Site B
end
interface Loopback1
description Loopback for ACI-SDA
ip address 10.82.83.4 255.255.255.255
end
border01#show int tunnel0

Tunnel0 is up, line protocol is up
Hardware is Tunnel
Interface is unnumbered. Using address of Loopback1 (10.82.83.4) Doctor client PC
MTU 17864 bytes, BW 100 Kbit/sec, DLY 50000 usec, 192.168.1.3/32
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation TUNNEL, loopback not set
Keepalive not set
Tunnel linestate evaluation up
Tunnel source 10.82.83.4
Tunnel protocol/transport MUDP/IP
…
border01#
C
border01#show bgp l2vpn evpn summary

BGP router identifier 172.16.1.254, local AS number 65003
BGP table version is 12, main routing table version 12 SD-Access
B
6 network entries using 2304 bytes of memory Fabric Site B
8 path entries using 1792 bytes of memory
5/4 BGP path/bestpath attribute entries using 1440 bytes of memory
2 BGP AS-PATH entries using 48 bytes of memory
3 BGP extended community entries using 124 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 5708 total bytes of memory
BGP activity 16/3 prefixes, 25/7 paths, scan interval 60 secs
6 networks peaked at 13:25:20 Mar 1 2021 UTC (1d01h ago)
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd Doctor client PC
10.92.92.227 4 101 720 794 7 0 0 11:55:22 1 192.168.1.3/32
10.92.92.228 4 101 720 800 7 0 0 11:55:17 1
border01#
border01#show bgp l2vpn evpn
BGP table version is 7, local router ID is 172.16.1.254 C
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
t secondary path, L long-lived-stale,
SD-Access
B
Origin codes: i - IGP, e - EGP, ? - incomplete Fabric Site B
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path

Route Distinguisher: 1:4099 (default for vrf Internal03)
*> [5][1:4099][0][24][172.16.201.0]/17
172.16.4.2 65535 65000 ?
*> [5][1:4099][0][24][192.168.1.0]/17
0.0.0.0 32768 i
* [5][1:4099][0][30][172.16.4.0]/17
0.0.0.0 0 32768 i
*> 172.16.4.2 0 65535 65000 ? Doctor client PC
*> [5][1:4099][0][30][172.16.4.4]/17 192.168.1.3/32
172.16.4.2 0 65535 65000 ?
* 0.0.0.0 0 32768 i
Route Distinguisher: 10.92.92.227:2
*> [5][10.92.92.227:2][0][24][10.42.42.0]/17
When the SDA border
10.92.92.231
0 65535 101 ? sends the packet to ACI
*> [5][10.92.92.228:2][0][24][10.42.42.0]/17
10.92.92.231 0 65535 101 ?
it rewrites the VNID to
border01# the ACI VNID
C
border01#show nve vni

Interface VNI Multicast-group VNI state Mode VLAN cfg vrf SD-Access
B
nve1 4099 N/A Up L3CP 3002 CLI Internal03 Fabric Site B
border01# show run int nve1

interface nve1
no ip address
source-interface Loopback1
host-reachability protocol bgp
vxlan udp port 48879
group-based policy
member vni 4099 vrf Internal03
end
Doctor client PC
border01#show nve peers
Interface VNI Type Peer-IP RMAC/Num_RTs eVNI state flags UP time
192.168.1.3/32
nve1 4099 L3CP 10.92.92.231 0200.0a5c.0000 2392064 UP A/M/4 11:59:44
When the SDA border

sends the packet to ACI
the ACI VNID
border01#show bgp l2vpn evpn detail
… C
BGP routing table entry for [5][10.92.92.227:2][0][24][10.42.42.0]/17, version 7
Paths: (1 available, best #1, table EVPN-BGP-Table)
Advertised to update-groups: SD-Access
1 B
Refresh Epoch 1 Fabric Site B
101
10.92.92.231 (metric 20) (via default) from 10.92.92.227 (10.0.136.71)
Origin incomplete, metric 0, localpref 100, weight 65535, valid, external, best
EVPN ESI: 00000000000000000000, Gateway Address: 0.0.0.0, VNI Label 2392064, MPLS VPN Label 0
Extended Community: SoO:101:33554415 RT:1:4099 ENCAP:8
Router MAC:0200.0A5C.0000
rx pathid: 0, tx pathid: 0x0
Updated on Mar 2 2021 22:29:21 UTC
…
bdsol-aci12-leaf3# moquery -c l3Ctx | egrep "^dn|^scope|^name"

dn : sys/ctx-[vxlan-16777200] Doctor client PC
name : black-hole
scope : 16777200 192.168.1.3/32
dn : sys/ctx-[vxlan-2949121]
name : management
scope : 2949121
dn
name
: sys/ctx-[vxlan-2621440]
: __mdp_54_DEFAULT_TENANT:Internal03
When the SDA border
scope
dn
: 2621440
: sys/ctx-[vxlan-2392064]
sends the packet to ACI
name
scope
: __mdp_54_DEFAULT_TENANT:Internal03_outer
: 2392064 it rewrites the VNID to
the ACI VNID
Doctor sends traffic – ACI side – Flow – Step 1
2
r
S
u te
3_o BL
rn al0
n te
NA
N T:I 4
_T E ed
T c
UL for
Border to
D EF A nen
U 3
spine in 4_ l03
outer p _5 064 na
2 er
md 39 Int
shadow VRF __ I: 2 T:
N
V
_T EN
AN
5
LT
1 DE
F AU
VRF: ACISDA:ACISDA VM
54_ 40 VNI: 2686976 10.42.42.101
_ 4
dp 1
_m 262
_ I:
VN
BL L
Leaf 103 Leaf 101
Leaf 104 Leaf 102
Doctor sends traffic – ACI side - Flow
bdsol-aci12-leaf3# show vrf
VRF-Name VRF-ID State Reason
__mdp_54_DEFAULT_TENANT:Internal 5 Up --
03
__mdp_54_DEFAULT_TENANT:Internal 6 Up --
03_outer
black-hole 3 Up --
management 2 Up --
overlay-1 4 Up --
BL
bdsol-aci12-leaf3# show ip route vrf __mdp_54_DEFAULT_TENANT:Internal03_outer
IP Route Table for VRF "__mdp_54_DEFAULT_TENANT:Internal03_outer"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
10.42.42.0/24, ubest/mbest: 1/0

*via 10.92.92.230%overlay-1, [1/0], 13:49:28, static
172.16.4.0/30, ubest/mbest: 1/0
*via 10.82.83.4%overlay-1, [20/0], 13:59:38, bgp-101, external, tag 65003, rwVnid: vxlan-4099
172.16.4.4/30, ubest/mbest: 1/0 VM
*via 10.82.83.4%overlay-1, [20/0], 13:59:38, bgp-101, external, tag 65003, rwVnid: vxlan-4099 10.42.42.101
172.16.201.0/24, ubest/mbest: 1/0
192.168.1.0/24, ubest/mbest: 1/0
bdsol-aci12-leaf3#
Spine remaps traffic
bdsol-aci12-spine1# show dcimgr repo sclass-maps
2
----------------------------------------------------------
Remote | Local
site Vrf PcTag | Vrf PcTag Rel-state
r
S ----------------------------------------------------------
1000 2392064 16 | 2621440 49153 [formed]
u te bdsol-aci12-spine1#
3_o BL
rn al0
n te
NA
N T:I 4
_T E ed
T c
UL for
D EF A nen
U 3
4_ l03
p _5 064 na
2 er
md 39 Int
__ I: 2 T:
N
V
_T EN
AN
5
LT
1 DE
F AU
54_ 40 VNI: 2686976 10.42.42.101
_ 4
dp 1
_m 262
_ I:
VN
BL L
Leaf 103 Leaf 101
Leaf 104 Leaf 102
BL
----------------------------------------------------------
Remote | Local
----------------------------------------------------------
1000 2392064 16 | 2621440 49153 [formed]
VM
10.42.42.101
2
r
S
u te
3_o BL
rn al0
n te
NA
N T:I 4
_T E ed
T c
UL for
D EF A nen
U 3
4_ l03
p _5 064 na
2 er
md 39 t Traffic send
__ I: 2 T :In back to
N
V
T_
TE
N AN border in
inside
5
L
1 DE
F AU shadow VRF
54_ 40 VNI: 2686976 10.42.42.101
_ 4
dp 1
_m 262
_ I:
VN
BL L
Leaf 103 Leaf 101
Leaf 104 Leaf 102
----------------------------------------------------------
Remote | Local
----------------------------------------------------------
1000 2392064 16 | 2621440 49153 [formed]
BL
module-1# show platform internal hal l3 routes | inc 2621440
|2621440| 0.0.0.0/ 0| UC| 5c| 19|
TCAM| 6| 0| 6|A| 75a9| 402e| NA| NA| NA| NA|
0| 0| 0| 0| 0| 1|
Hex/Dec
module-1# show platform internal hal l3 routes route-id 92 | inc "Next base Id"
Next base Id : 0x75a9
module-1# show platform internal hal l3 nexthops | grep 75a9
75a9 Y T F 0 1801000e 1801000e 0 1 1 0 0 402e 10 2 402e 0 20301e 0 0
0 0 0 1 0 0 0 00:0d:0d:0d:0d:0d 0 0 0 0 0 0 0 0 0 10.92.92.231
module-1# VM
10.42.42.101
Traffic send
from border
to access leaf
in dest. 2
VNI/pcTag
r
S
u te
3_o BL
rn al0
n te
NA
N T:I 4
_T E ed
T c
UL for
D EF A nen
U 3
4_ l03
p _5 064 na
2 er
md 39 Int
__ I: 2 T:
N
V
_T EN
AN
5
LT
1 DE
F AU
54_ 40 VNI: 2686976 10.42.42.101
_ 4
dp 1
_m 262
_ I:
VN
BL L
Leaf 103 Leaf 101
Leaf 104 Leaf 102
bdsol-aci12-leaf3# show endpoint
Legend:
s - arp H - vtep V - vpc-attached p - peer-aged
R - peer-attached-rl B - bounce S - static M - span
D - bounce-to-proxy O - peer-attached a - local-aged m - svc-mgr
L - local E - shared-service
+-----------------------------------+---------------+-----------------+--------------+-------------+
VLAN/ Encap MAC Address MAC Info/ Interface
Domain VLAN IP Address IP Info BL
+-----------------------------------+---------------+-----------------+--------------+-------------+
__mdp_54_DEFAULT_TENANT:Internal03 192.168.1.3 p tunnel15
…
bdsol-aci12-leaf3# show ip route vrf __mdp_54_DEFAULT_TENANT:Internal03

IP Route Table for VRF "__mdp_54_DEFAULT_TENANT:Internal03"
10.42.42.0/24, ubest/mbest: 1/0, attached, direct, pervasive VM

*via 10.0.32.67%overlay-1, [1/0], 15:33:11, static, tag 4294967292, rwVnid: vxlan-2686976
172.16.4.0/30, ubest/mbest: 1/0 10.42.42.101
…
192.168.1.0/24, ubest/mbest: 1/0
bdsol-aci12-apic1# moquery -c fvCtx -d "uni/tn-ACISDA/ctx-ACISDA" | egrep "^dn|scope"

dn : uni/tn-ACISDA/ctx-ACISDA
scope : 2686976
bdsol-aci12-spine1# show ip int brief | grep 32.67
lo9 10.0.32.67/32 protocol-up/link-up/admin-up
bdsol-aci12-spine2# show ip int brief | grep 10.0.32.67
lo9 10.0.32.67/32 protocol-up/link-up/admin-up
bdsol-aci12-spine2# show coop internal info ip-db key 2686976 10.42.42.101
IP address : 10.42.42.101 BL
Vrf : 2686976
Flags : 0
EP bd vnid : 16121790
EP mac : 00:50:56:B6:3F:9D
Publisher Id : 10.0.8.64
Record timestamp : 03 03 2021 13:59:32 564569819
Publish timestamp : 03 03 2021 13:59:32 565091249
Seq No: 0
Remote publish timestamp: 01 01 1970 00:00:00 0
URIB Tunnel Info
Num tunnels : 1
Tunnel address : 10.0.32.64 VM
Tunnel ref count : 1
10.42.42.101
bdsol-aci12-apic1# moquery -c fvAEPg -d "uni/tn-ACISDA/ap-E-commerce/epg-Web" | egrep "^dn|pcTag"
dn : uni/tn-ACISDA/ap-E-commerce/epg-Web
pcTag : 5474
bdsol-aci12-leaf3# show zoning-rule | grep 5474

| 4106 | 5474 | 0 | implicit | uni-dir | enabled | 2621440 | | deny,log | shsrc_any_any_deny(12) |
| 4107 | 49153 | 5474 | 5 | bi-dir | enabled | 2621440 | ACISDA:Campus-2-Web | permit | fully_qual(7) |
| 4108 | 5474 | 49153 | 5 | uni-dir-ignore | enabled | 2621440 | ACISDA:Campus-2-Web | permit | fully_qual(7) |
bdsol-aci12-leaf3#
Policy applied in
internal/campus
shadow VRF on
BL
BL
VM
10.42.42.101
BL
VM
10.42.42.101
BL
VM
10.42.42.101
2
r
S
u te
3_o BL
rn al0
n te
NA
N T:I 4
_T E ed
T c
UL for
D EF A nen
U 3
4_ l03
p _5 064 na
Traffic leaked
2 er
md 39 t into destination
__ I: 2 :In VRF
N NT
V
_TE
NA 5
LT
1 DE
F AU
54_ 40 VNI: 2686976 10.42.42.101
_ 4
dp 1
_m 262
_ I:
VN
BL L
Reminder: ACI Route leaking
Leaf 103 Leaf 101
Configure the subnet on the
Leaf 104 Leaf 102 provider EPG
bdsol-aci12-leaf1# show endpoint
Legend:
s - arp H - vtep V - vpc-attached p - peer-aged
R - peer-attached-rl B - bounce S - static M - span
D - bounce-to-proxy O - peer-attached a - local-aged m - svc-mgr
L - local E - shared-service
+-----------------------------------+---------------+-----------------+--------------+-------------+
VLAN/ Encap MAC Address MAC Info/ Interface
Domain VLAN IP Address IP Info BL
+-----------------------------------+---------------+-----------------+--------------+-------------+
13 vlan-1035 0050.56b6.3f9d LV po2
ACISDA:ACISDA vlan-1035 10.42.42.101 LV po2
13 vlan-1035 0050.5687.6567 LpV po2
ACISDA:ACISDA vlan-1035 10.42.42.102 LV po2
overlay-1 10.0.136.64 L lo0
overlay-1 10.0.32.64 L lo1
11/overlay-1 vxlan-16777209 40f0.7843.26b0 L eth1/2
11/overlay-1 vxlan-16777209 3c57.311a.4384 L eth1/1
11/overlay-1 vxlan-16777209 2cf8.9b29.2f1e L eth1/3
VM
10.42.42.101
Doctor sends traffic – ACI side – Complete flow
Spine remaps traffic
Traffic send
from border bdsol-aci12-spine1# show dcimgr repo sclass-maps
2
to access leaf ----------------------------------------------------------
in dest. Remote | Local
VNI/pcTag site Vrf PcTag | Vrf PcTag Rel-state
r
S ----------------------------------------------------------
1000 2392064 16 | 2621440 49153 [formed]
u te bdsol-aci12-spine1#
3_o BL
rn al0
n te
NA
N T:I 4
_T E ed
T c
UL for
Border to
D EF A nen
U 3
spine in 4_ l03
outer p _5 064 na
Traffic leaked
2 er
md 39 t Traffic send into destination
shadow VRF __ I: 2 :In back to VRF
N NT
V
_TE
NA border in
inside
5
LT
1 DE
F AU shadow VRF
54_ 40 VNI: 2686976 10.42.42.101
_ 4
dp 1
_m 262
_ I:
VN
BL L
Leaf 103 Leaf 101
Leaf 104 Leaf 102
From ACI à
SDA
Next slides will
cover the
difference
compared to
SDA à ACI
General flow
• Step 1: Access leaf sends traffic to border leaf
• Route leaking, towards internal VRF
• Step 2: Border leaf sends traffic in campus shadow VRF to spine
• Step 3: Spine changes VRF/pcTag
• Step 4: Spine sends traffic ACI Border leaf in outer VRF
• Step 5: SDA Border receives traffic
• Step 6: SDA Border forwards tragic to Edge
• Step 7: Edge delivers packet
Return traffic - ACI side – Access leaf
bdsol-aci12-leaf1# show ip route vrf ACISDA:ACISDA
IP Route Table for VRF "ACISDA:ACISDA"
10.42.42.0/24, ubest/mbest: 1/0, attached, direct, pervasive

BL
*via 10.0.32.67%overlay-1, [1/0], 16:46:55, static, rwVnid: vxlan-2686976
10.42.42.1/32, ubest/mbest: 1/0, attached, pervasive
*via 10.42.42.1, vlan12, [0/0], 21:13:55, local, local
172.16.4.0/30, ubest/mbest: 2/0
*via 10.0.136.71%overlay-1, [200/0], 16:46:54, bgp-101, internal, tag 65003, rwVnid: vxlan-2621440
172.16.4.4/30, ubest/mbest: 2/0
172.16.201.0/24, ubest/mbest: 2/0
*via 10.0.136.71%overlay-1, [200/0], 16:46:54, bgp-101, internal, tag 65003, rwVnid: vxlan-2621440 VM
192.168.1.0/24, ubest/mbest: 2/0 10.42.42.101
bdsol-aci12-leaf1#
How do we receive
__mdp_54_DEFAULT_TENANT:Internal03_outer
__mdp_54_DEFAULT_TENANT:Internal03
2392064
2621440 these routes ?
ACISDA:ACISDA 2686976
Return traffic - ACI side – Access leaf
bdsol-aci12-leaf1# show bgp vpnv4 unicast vrf ACISDA:ACISDA
BGP routing table information for VRF overlay-1, address family VPNv4 Unicast
BGP table version is 27, local router ID is 10.0.136.64
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-injected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup

BL
Route Distinguisher: 101:2686976 (VRF ACISDA:ACISDA)
*|i172.16.4.0/30 10.0.136.71 100 0 65003 65000 ?
*>i 10.0.136.69 100 0 65003 65000 ?
*|i172.16.4.4/30 10.0.136.71 100 0 65003 65000 ?
*>i 10.0.136.69 100 0 65003 65000 ?
*|i172.16.201.0/24 10.0.136.71 100 0 65003 65000 ?
*>i 10.0.136.69 100 0 65003 65000 ?
*|i192.168.1.0/24 10.0.136.71 0 100 0 65003 i
*>i 10.0.136.69 0 100 0 65003 i
VM
bdsol-aci12-leaf1# acidiag fnvread | egrep "136.71|136.69"
103 1 bdsol-aci12-leaf3 FDO24310LXQ 10.0.136.69/32 leaf active 0 10.42.42.101
104 1 bdsol-aci12-leaf4 FDO24311CJN 10.0.136.71/32 leaf active 0
Return traffic - ACI side – Border inside VRF

BL
172.16.4.0/30, ubest/mbest: 1/0
172.16.4.4/30, ubest/mbest: 1/0
172.16.201.0/24, ubest/mbest: 1/0
192.168.1.0/24, ubest/mbest: 1/0
VM
10.42.42.101
Return traffic - ACI side – Border inside VRF

BL
172.16.4.0/30, ubest/mbest: 1/0
172.16.4.4/30, ubest/mbest: 1/0
172.16.201.0/24, ubest/mbest: 1/0
192.168.1.0/24, ubest/mbest: 1/0
VM
10.42.42.101
Return traffic - ACI side – Spine

----------------------------------------------------------
Remote | Local
site Vrf PcTag | Vrf PcTag Rel-state BL
----------------------------------------------------------
1000 2392064 16 | 2621440 49153 [formed]
module-1# show platform internal hal l3 routes | inc 2392064

|2392064| 0.0.0.0/ 0| UC| 5d| 1a|
TCAM| 406| 0| 406|A| 75a9| 402e| NA| NA| NA| NA|
0| 0| 0| 0| 0| 1|
module-1# show platform internal hal l3 nexthops | grep 75a9

75a9 Y T F 0 1801000e 1801000e 0 1 1 0 0 402e 10 2 402e 0 20301e 0 0
0 0 0 1 0 0 0 00:0d:0d:0d:0d:0d 0 0 0 0 0 0 0 0 0 10.92.92.231
module-1# VM
10.42.42.101
Return traffic - ACI side – Border outer VRF
bdsol-aci12-leaf3# show ip route vrf __mdp_54_DEFAULT_TENANT:Internal03_outer
IP Route Table for VRF "__mdp_54_DEFAULT_TENANT:Internal03_outer"
10.42.42.0/24, ubest/mbest: 1/0

BL
*via 10.92.92.230%overlay-1, [1/0], 16:57:36, static
172.16.4.0/30, ubest/mbest: 1/0
172.16.4.4/30, ubest/mbest: 1/0
172.16.201.0/24, ubest/mbest: 1/0
192.168.1.0/24, ubest/mbest: 1/0
bdsol-aci12-leaf3#
VM
10.42.42.101
ACI / SDA connection - Underlay
Cp-TEP
10.92.92.228/32
Eth 1/41
OSPF
OSPF 10.82.82.6/30
BL
Gi0/1/0.901 Leaf 103
G 1/0/24 10.82.82.5/30
VPC Domain
10.82.82.1/30
B G 0/2/1
F
L3
Anycast-TEP
10.92.92.231/32
10.82.82.2/30 Gi0/2/0.901
Loopback1 / Tunnel0
10.82.83.4/32
10.82.82.9/30
Eth 1/41
BL
10.82.82.10/30
Leaf 104
Cp-TEP
10.92.92.227/32
SDA L3 ACI
Return traffic - ACI side – Border outer VRF
bdsol-aci12-leaf3# show bgp ipv4 unicast 192.168.1.3 vrf __mdp_54_DEFAULT_TENANT:Internal03_outer
BGP routing table information for VRF __mdp_54_DEFAULT_TENANT:Internal03_outer, address family IPv4 Unicast
BGP routing table entry for 192.168.1.0/24, version 4 dest ptr 0xa03109a4
Paths: (1 available, best #1)
Flags: (0x0c001a 00000000) on xmit-list, is in urib, is best urib route, is in HW, exported
vpn: version 16, (0x100002) on xmit-list
Multipath: eBGP iBGP
BL
Advertised path-id 1, VPN AF advertised path-id 1
Path type: external 0xc0000028 0x0 ref 0 adv path ref 2, path is valid, is best path, remote nh not
installed
Imported from 1:4099:[5]:[0]:[0]:[24]:[192.168.1.0]:[0.0.0.0]/120
AS-Path: 65003 , path sourced external to AS
10.82.83.4 (metric 42) from 10.82.83.4 (172.16.1.254)
Origin IGP, MED 0, localpref 100, weight 0 tag 0, propagate 0
Aggregated by 172.16.1.254, aggregator AS 65003, atomic-aggregate set
Received label 4099
Extcommunity:
ENCAP:8 VM
Router MAC:7035.093c.804b
VNID:4099 10.42.42.101
When the ACI border

VRF advertise information:
Path-id 1 not advertised to any peer
VPN AF advertise information:

Path-id 1 advertised to peers:
sends the packet to SDA,
10.0.136.65 10.0.136.66
bdsol-aci12-leaf3#
the SDA VNID
Return traffic - SDA side – border
C
border01#show ip route vrf Internal03

B
Fabric Site B
10.0.0.0/24 is subnetted, 1 subnets
B 10.42.42.0 [20/0] via 10.92.92.231, 17:07:35, Vlan3002
B 172.16.201.0/24 [20/0] via 172.16.4.2, 2d02h
B 192.168.1.0/24 [200/0], 2d10h, Null0
l 192.168.1.3/32 [250/1], 23:13:19, Null0
border01# Doctor client PC
192.168.1.3/32
Normal lookup in
LISP
Return traffic - SDA side – edge
C
edge-2#show cts role-based permissions

IPv4 Role-based permissions default:
Permit IP-00 SD-Access
B
RBACL Monitor All for Dynamic Policies : FALSE Fabric Site B
RBACL Monitor All for Configured Policies : FALSE
edge-2#show cts role-based sgt-map all


============================================
172.16.2.69 2 INTERNAL
172.16.2.70 2 INTERNAL
IP-SGT Active Bindings Summary

============================================
Total number of INTERNAL bindings = 2 Doctor client PC
Total number of active bindings = 2 192.168.1.3/32

================================================================
edge-2#
Return traffic - SDA side – edge C
• You can opt to apply Policy in ISE SD-Access

B
• Configure SXP on border Fabric Site B
• Use interface create for ACI Handoff in Internal03 VRF

(default created by integration)
• Policy applied egress on edge
• Useful commands
• Edge
Doctor client PC
• show cts role-based permissions 192.168.1.3/32
• show cts role-based counters

• Border
• show cts sxp sgt-map vrf Internal03
Continue your education
Demos in the Cisco campus
Meet the engineer 1:1 meetings
Walk-in labs
Related sessions
Thank you
#CiscoLive
#CiscoLive

Brkaci 2008

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Brkaci 2008

Uploaded by

Copyright:

Available Formats

#CiscoLive

End-2-End Policy from the

Ramses Smeyers, Principal Consulting Engineer, CX

End-Users Data &

• Limited Availability is the phase in a product lifecycle, that comes before

• Logical constructs • Logical constructs

• Virtual Network • VRF

• User Endpoint • App Endpoint

• Group Based Access Control • Contract

2 PLANE automated network hand-off with

Campus SDA Fabric CONTROL Automated VRF-aware Network

EPG-SGT Translate SGT-EPG Translate

SDA Border ASR1001-X, ASR1001-HX, ASR1002-X, 17.3.2a

SDA Edge All SDA supported switches. FE-switches 17.3.2a

ISE 3.0 patch 1

APIC APIC-CLUSTER-M3 5.1(1h)

ACI Border Leaf C9300-GX/FX/FX2/FXP/EX 15.1(1h)

ACI Spine C9300-GX 15.1(1h)

Configurable Option Scale

Remote EPGs (Security Group Tag (SGT)) 500

Remote SDA endpoints 256000*

Local EPGs 500

Local endpoints 16000

Contract rules 64000

• Kafka is a pub/sub messaging bus

publish (write) messages to Kafka

• All APICs in the APIC cluster run Kakfa,

• TCP port 9093 is not permitted by default

An OOB management contract is required to

SGT/IP and EPG/IP

VRF: Internal03 SD-Access

edge-2#show run int GigabitEthernet 1/0/14

edge-2#show authentication sessions interface GigabitEthernet 1/0/14

Method status list:

IP Address SGT Source

IP-SGT Active Bindings Summary

edge-2#show cts role-based permissions

SDA does egress policy

Routing Table: Internal03 SD-Access

N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

192.168.1.0/24 is variably subnetted, 3 subnets, 2 masks

edge-1# sh run | b lisp

192.168.1.3/32, dynamic-eid 192_168_1_0-Internal03-IPV4, inherited from default locator-set rloc_88061184-0323-4d56-82c5-

edge-2#sh ip int brief | i Loopback0

border01#sh ip lisp | i Map

border01#sh lisp site | i Site|Register|192.168.1.3

edge-2#lig instance-id 4099 10.42.42.101

edge-2#lig instance-id 4099 10.42.42.101

Gateway of last resort is not set SD-Access

Routing Table: Internal03 SD-Access

Gateway of last resort is not set SD-Access

ACI à MDC à System Configuration SD-Access

border01#show int tunnel0

border01#show bgp l2vpn evpn summary

Network Next Hop Metric LocPrf Weight Path

border01#show nve vni

border01# show run int nve1

When the SDA border

bdsol-aci12-leaf3# moquery -c l3Ctx | egrep "^dn|^scope|^name"

10.42.42.0/24, ubest/mbest: 1/0

bdsol-aci12-spine1# show dcimgr repo sclass-maps

bdsol-aci12-spine1# show dcimgr repo sclass-maps

bdsol-aci12-leaf3# show ip route vrf __mdp_54_DEFAULT_TENANT:Internal03