Brkaci 2001

Integration and Interoperation of
Existing Nexus Networks into an

ACI Architecture
Michael Herbert Principal Engineer INSBU
BRKACI-2001
Cisco Spark
Questions?  
Use Cisco Spark to communicate  
with the speaker after the session
How
1. Find this session in the Cisco Live Mobile App
2. Click “Join the Discussion”
3. Install Spark or go directly to the space
4. Enter messages/questions in the space
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Agenda
▪ Introduction to the Intent/Application Based Data Centre
• What is ACI?
• What do we mean by Intent
• What do we mean by Applications Based Infrastructure
▪ ACI Foundations
▪ Integrating Existing Nexus Layer 2 and Layer 3
▪ Integrating DCI and WAN
▪ Integrating with Other Security Domains (TrustSec, SDA, …)
▪ What about Public Cloud?
Next Gen Forwarding & Networking 
STP
VPC FabricPath
VXLAN
Data Plane Based Endpoint Discovery
Control Plane Based Endpoint Location Tracking

VXLAN VXLAN
/EVPN /ACI APIC
MAN/WAN MAN/WAN
BRKACI-2001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
Identity Based Networking  VTEP IP Group VNID Tenant Packet
Policy
Shared Application
Servers
Devices and users are Services
authenticated and
authorised into end-point Ingress and
ACI Fabric
groups (aka EPG’s or
Egress ➔
Enforcement
SGT’s) APIC
End Point Group Tags Enterprise

Backbone
(EPG’s, SGT’s) are
encoded in a VXLAN
header
Campus Switch Campus Switch
Policies between scalable
groups are established
following the provider/ Employee Tag
consumer model Supplier Tag

Non-Compliant Employee Voice Voice Employee Non-Compliant Tag
VLAN A VLAN B
Identity Based Networking  VTEP IP Group VNID Tenant Packet
Policy
Shared Application
Servers
Devices and users are Services
authenticated and
authorised into end-point Ingress and
ACI Fabric
Egress ➔
Enforcement
SGT’s) APIC
End Point Group Tags Enterprise

Backbone
encoded in a VXLAN
header
Policies between scalable
following the provider/ Employee Tag

Non-Compliant Employee Voice Voice Employee Supplier Non-Compliant Non-Compliant Tag
VLAN A VLAN B
Directory Enabled Networking 
Data Base Defined Networking APIC
Common Operational Properties - AD, LDAP, …

System Management, Change Management, System Integrity, Correlation
Application Centric Infrastructure (ACI) 
Identity and Intent Based Infrastructure
Embedded L4 Security
Next Gen Stateful L4-7
Embedded Sensors Visibility and Control
Policy Discovery, Identity and Policy
Monitoring and Control Federation
Multi-Tier Sensor
Data Gathering
MACSEC and INS-SEC Firewall at Each

Encryption Leaf switch
Servers (Physical, Virtual, Containers, Micro
Services)
Branch Web1 App1 DB Identity & Policy

QoS QoS QoS
Filter Service Filter

Driven Security
Architecture
Application Centric Infrastructure (ACI) 
Identity and Intent Based Infrastructure
Embedded L4 Security
Next Gen Stateful L4-7
Embedded Sensors Visibility and Control
Policy Discovery, Identity and Policy
Monitoring and Control Federation
Multi-Tier Sensor
Data Gathering
MACSEC and INS-SEC Firewall at Each

Encryption Leaf switch
Servers (Physical, Virtual, Containers, Micro
Services)
Branch Web1 App1 DB Identity & Policy

QoS QoS QoS
Filter Service Filter

Driven Security
Architecture
Agenda
• What is ACI?
▪ ACI Foundations
Gartner’s View 
Intent Based Infrastructure
http://blogs.gartner.com/andrew-lerner/2017/02/07/intent-based-networking/
Treating Infrastructure Like a System 
Intent Based Control
Define Business outcomes
(Actual Requirements - Policies)
Assurance Functions Logical Description ‘Write to’

(System Correctness and SLA Business Policies
& Outcomes
of Requirements Operations
Monitoring)
Centralised Data
Monitoring, Data Insights & Controllers Model (Database)
Discovery, Learning Analytics & Orchestrators
‘Read from’
Operations Streaming Telemetry
and Systems Coordinated Updates to
Programmable
Feedback Infrastructure Infrastructure Components
Data Centre Vision 
Inter-dependent
Data Centre Vision  feedback loops
ACI, UCS (Intent
1. Deployment and Based Automation)
Infrastructure
Provisioning Automation
Security
Cisco
Guarantees Network
CloudCentre Application
(Common Deployment Compliance Assurance
Consistency
Consumption across Engine
Hybrid IT)
(Formal
Methodologies)
ADM 
Tetration Analytics Security 2. Operations and
Forensics
(Machine Learning Based Operations Management
and Security)
Agenda
• What is ACI?
▪ ACI Foundations
What Do We Mean by “Application”?
What Do We Mean by “Application Centric”?
VISIBILITY
Cisco CloudCentre
Application Profile
Application
Service Storage
Network
Profiles Profiles SECURITY
Profiles
Cisco ACI
PERFORMANCE
Cisco Workload
Optimisation Manager
Cisco Tetration
What Do We Mean by “Application Centric”? Analytics™
Correlation of Enforcement and Telemetry
• Cisco Tetration and ACI are designed to provide

complementary visibility, security and
operations
• Tetration platform provides network
performance monitoring functionalities in Cisco
ACI™ mode
• Following Cisco Nexus® 9000 series hardware
is required:
• Cisco Nexus 9300-FX based leaf switches
• Cisco Nexus 9500 series spine switches with N9K-
X9736C-FX line cards
• These functionalities require Cisco ACI release
3.1 or later
Cisco Tetration

operations
ACI™ mode
is required:
3.1 or later
Cisco Tetration

operations
ACI™ mode
is required:
3.1 or later
Cisco Tetration
Correlation with the view from the Server
Flow Inventory
Flow details
Process
Inventory
Process
details
What Do We Mean by “Application Centric”?
Correlating the Application ‘Transaction’ View with the Infrastructure View
APIC APIC
Exchanges Information on
Application Tiers, Nodes, Services, Get the context related to
Service Endpoints, End host, affected EP’s
Business Transactions, Health Create Troubleshooting
Status, Faults session
Provide an ‘Application Allow Application Admin

Transaction’ View of the to launch ACI
Network Troubleshooting Tools
Objective: View of Application as Related to Infrastructure 
Multi-Domain View
Consistent Governance
Cisco Tetration
Analytics™
Application
Owner,
Administrator, …
Public Cloud
vPod
Infrastructure
Service VM Service VM Service VM
Hypervisor Hypervisor Hypervisor Administration
Agenda
▪ ACI Foundations
▪ Forwarding and Network Availability
▪ ACI Constructs

ACI Fabric – An IP network with an Integrated Overlay
APIC
VTEP GPO VXLAN MACIP Payload IP Transport
VTEP VTEP VTEP VTEP VTEP VTEP
VTEP
vSwitch VTEP
vSwitch
• Cisco’s ACI solution leverages an integrated VXLAN based overlay

• IP Network for Transport
• VXLAN based tunnel end points (VTEP)
• VTEP discovery via infrastructure routing
• Directory (Mapping) service for EID (host MAC and IP address) to VTEP lookup
Host Level Forwarding Granularity
IP Forwarding: MAC Forwarding:
Forwarded using DIPi Forwarded using DMAC

address, HW learning of IP address, HW learning of
address MAC address
10.1.3.11 10.1.3.35 10.6.3.2 10.6.3.17
• Forward based on destination IP Address for intra and inter subnet (Default Mode)
• Bridge semantics are preserved for intra subnet traffic (no TTL decrement, no MAC
header rewrite, etc.)
• Non-IP packets will be forwarded using MAC address. Fabric will learn MAC’s for non-IP
packets, IP address learning for all other packets
• Route if MAC is router-mac, otherwise bridge (standard L2/L3 behaviour)
Removing the Classic L2/L3 Boundaries
Layer 2 and Layer 3 integrated forwarding
10.1.1.10 10.1.3.11 10.6.3.2 10.1.3.35 10.1.3.11 10.1.3.35

10.1.1.10 10.6.3.2
Distributed Default Gateway Directed ARP Forwarding and/or Proxy-

ARP based default GW
• ACI Fabric supports full layer 2 and layer 3 forwarding semantics, no changes required to applications or end point IP
stacks
• ACI Fabric provides optimal forwarding for layer 2 and layer 3
• Fabric provides a pervasive SVI which allows for a distributed default gateway
ACI Multi-Pod  BiDir PIM Multicast Requirement in Backbone
(HER planned for future release)
Inter-Pod Network
Pod ‘N’
Pod ‘A’
Layer 2/3 Host Mobility
Single APIC Cluster
▪ Multiple ACI Pods connected by an IP Inter-Pod L3 ▪ Forwarding control plane (IS-IS, COOP) fault isolation
network, each Pod consists of leaf and spine nodes (BGP between pods)
▪ Managed by a single APIC Cluster (single database ▪ Data Plane VXLAN encapsulation between Pods (any
domain) that can be geographically distributed (10 msec layer 2 VLAN and layer 3 subnet can be extended
RTT, will increase to 50 msec) across pods)
ACI Multi-Site 
Network and Identity Extended between Fabrics
Identity information carried across Fabrics Network information carried across Fabrics
(Availability Zones) (Availability Zones)
VTEP IP Class-ID VNID Tenant Packet
No Multicast Requirement in
Backbone (HER for any BUM
IP Network traffic)
MP-BGP - EVPN
Multi-Site Orchestrator
ACI Remote Physical Leaf
Available from ACI 3.1 Release All local to remote traffic is forwarded
via ACI VXLAN Data-Plane
VXLAN
Data-Plane
IP WAN/IPN
(No Multicast Required)
Any traffic that requires use of the Spine vSwitch

Hypervisor
Proxy will be forwarded to the primary site
* Spine services will be required in scenario’s like copy

All local traffic is switched directly between
service, BD Proxy Services, PBR endpoints, both virtual and bare metal
ACI Remote Virtual Leaf (Virtual Pod)
Scheduled for 2HCY18 DME/PE COOP BGP
Oracle RR
BGP EVPN Carries

Host and Network
Routing End to End
AVE AVE AVE

Hypervisor Hypervisor Hypervisor
Web App DB
IP Network
AVE vSwitch vSwitch DME/PE COOP BGP

Oracle RR
Hypervisor Hypervisor Hypervisor GPO VXLAN Provides
Network and Identity
Based Segmentation
Web App DB End to End
AVE AVE AVE
Web App DB
ACI Anywhere
Scheduled for 2HCY18
Reachability - SDWAN
(Viptela), Direct
Connect, ExpressRoute
Web App DB
IP Network
AVE vSwitch vSwitch COOP BGP

DME/PE Oracle
Hypervisor Hypervisor Hypervisor GPO VXLAN Provides RR
Network and Identity
Based Segmentation
Web App DB End to End
AVE AVE AVE
Web App DB
ACI Fault and Policy Domains 
DC Wide Operational Domain – Single Domain for security groups (EPG’s), connectivity, …, with scoping for changes
Common WAN Services
DC Core
MP-BGP - EVPN
Pod ‘A’ Pod ‘B’ Pod ‘C’ Pod ‘D’

Switch
Maintenance
Zones
APIC Cluster APIC Cluster
Network Configuration Network Configuration Network Configuration Network Configuration

and Fault Domain and Fault Domain and Fault Domain and Fault Domain
Availability Zone Availability Zone
Agenda
▪ ACI Foundations
▪ Forwarding and Network Availability
▪ ACI Constructs

Some new (or not so new) terms: Tenants, VRF (Context), Bridge Domains, Application Network
What is an EPG? 
A Logical Group of Endpoints Attached to the Network
BD1 BD2 BD2

BD3
EPG1 EPG2 EPG3 EPG4 EPG5 EPG6 EPG7 EPG8 EPG9
• All of the endpoints (things attached to the network) in the same EPG are treated to the same rules (policy)
• A security group using the same access lists (similar to an SGT in TrustSec)
• A services group using the same QoS rules, same L4-7 services, …
• It could be as simple as all the servers on the same VLAN or subnet
What are Contracts and Filters? 
The Network Rules Tenant
VRF
• Contracts are semantics to Bridge Domains Bridge Domains
specify EPG to EPG
communication in ACI EPGs EPGs EPGs EPGs
• Communication policy includes C C C
filters (ACLs), QoS, Route

C
Leaking, L4-7 Service Graphs
• Contract filters are similar to
Access Control Lists (e.g. match
this TCP port)
• Contracts can be defined VRF
between EPGs or between L3out Bridge Domains

C
External EPGs and regular EPGs EPGs C EPGs
What is an Application Profile? 
A Logical Group of EPG’s and Associated Contracts
Health scores, statistics, logs

and audit data automatically
correlated and rolled up at
Application Profile level
EPG, uEPG, domain

associations, contract relations
and L4-7 Configuration
Developer: My application template NOC: What I monitor and troubleshoot
What is a Tenant? 
A Virtual Private Cloud
Outside View: A Tenant is a group that owns a virtual Tenant
private cloud instance
Inside View: The portion of the database that your

login has access to
Context Context
BD BD BD
Subnet Subnet Subnet Subnet Subnet

A B C B C
Okay but Where the *&$# Did My Network Go?  
Bridge Domains and Contexts
• Context == Private Network
• Unique Layer 3 (L3) forwarding domain
• It’s a VRF
• Bridge Domain (BD)

• Unique Layer 2 (L2) forwarding domain
• By default it defines the Broadcast Domain
• Is associated with a VRF
• Subnet
• Is a subnet
• It is associated with a Bridge Domain 10.10.20.0/24
10.10.10.0/24
• You can have multiple subnets associated with one
192.168.4.0/24
BD (think secondary IP’s)
ACI Networking Foundations  
BDs and EPGs
Layer 2
Layer 2
Layer 2
Extend L2 domain beyond ACI fabric - 2 options

1. Manually assign a port to a VLAN which in turn mapped to an EPG. This extend EPG beyond ACI fabric
(EPG == VLAN)
2. Create a L2 connection to outside network. Extend bridge domain beyond ACI fabric. Allow contract
between EPG inside ACI and EPG outside of ACI
ACI Networking Foundations  
BDs and EPGs
Layer 2
Layer 2
Layer 2
Lets Look at the

Links
Extend L2 domain beyond ACI fabric - 2 options

1. Manually assign a port to a VLAN which in turn mapped to an EPG. This extend EPG beyond ACI fabric
(EPG == VLAN)
2. Create a L2 connection to outside network. Extend bridge domain beyond ACI fabric. Allow contract
between EPG inside ACI and EPG outside of ACI
VLAN, EPG and BD 
VLAN == EPG
Layer 2
VLAN 30
VLAN 20
BD
EPG
Existing
App 100.1.1.5
100.1.1.3 100.1.1.99 100.1.1.7 100.1.1.3
• VLANs are localised to the leaf ports

• The same subnet, bridge domain, EPG can be configured as a ‘different’ VLAN on each leaf
switch
VLAN, EPG and BD 
VLAN == EPG
VLAN 10
VLAN 30
VLAN 10 VLAN 10 VLAN 10 Layer 2
VLAN 20
BD
EPG Existing
App
100.1.1.3 100.1.1.99 100.1.1.7 100.1.1.5 100.1.1.3
• Single Policy Group (one extended EPG)

• Leverage vPC for interconnect (diagram shows a single port-channel which is an option)
• BPDU should be enabled on the interconnect ports on the ‘vPC’ domain
VLAN, EPG and BD 
Sub-divide the ’Broadcast Domain’
VLAN 10
Layer 2
VLAN 30
VLAN 10 VLAN 10 VLAN 10
VLAN 20
EPG EPG BD
Outside C Inside Existing
100.1.1.3 100.1.1.99 100.1.1.7 100.1.1.5 App
100.1.1.3
• External EPG (policy between the L2 outside EPG and internal EPG)
• Leverage vPC for interconnect (diagram shows a single port-channel which is an option)
• BPDU should be enabled on the interconnect ports on the ‘vPC’ domain
• L2 outside forces the same external VLAN << fewer operational errors
Where did HSRP go? 
Anycast Default Gateway
WAN
Anycast Default Gateway
L2 Trunk
10.10.10.3 10.20.20.50
10.10.10.5
10.20.20.7
10.10.10.20
100.1.1.3 100.1.1.99 10.20.20.20
100.1.1.7
Agenda
▪ ACI Foundations
Attach the existing legacy network to the ACI fabric via L2
double-sided vPC
Spines
Leafs
North-to-South
East-to-West
Existing Network Infrastructure
Move the Default Gateway to ACI or...
ACI
EPG1 The VRF and Bridge Domain on ACI becomes
the default gateway for servers
New Servers Attach to ACI
EPG Legacy
Legacy Network
Server A Server B
MAC A MAC B
Move the Firewall to ACI and Keep Default Gateway on the
Firewall
ACI
EPG1 EPG Firewall MPLS
New Servers Attach to ACI
EPG Legacy
North-to-South Legacy Network
Server A Server B
East-to-West
MAC A MAC B
Eventually Move the Server NICs to ACI Leafs 
Spines
Firewalls
Leafs
Servers
MPLS
Virtual Machines
Legacy Network
Inside a VRF, Policy Enforcement is a Binary Decision
• Policy Enforce: no communication without contracts
• Policy Unenforced: all communication allowed
VRF – MyVRF VRF – MyVRF

L3Out L3Out
EPG-A EPG-B EPG-C EPG-A EPG-B EPG-C

External External
EPG EPG
Tools to simplify Contract (ACL) Management During
Migration - vzAny
Tenant ONE Tenant Shared Services
EPG1 VRF1 VRF Services

EPG Shared
Services
vzAny
EPG2
EPG3
Leverage an allow ‘any’ rule - vzAny
Start off with Contract Preferred Groups 
All EPG’s within a preferred group are trusted
VRF – MyVRF
Preferred Group
No need for contracts L3Out
VLAN10 VLAN20 VLAN30
or to understand External
application behaviour EPG
because EPGs are
configured in Preferred
Group
Migrate Network Centric Model – EPG = VLAN
VLAN10 VLAN20
VLAN30
Simplify Transition from Network Centric (VLAN == EPG) to
Application Centric
VRF – MyVRF
Preferred Group
L3Out
VLAN10 VLAN20 VLAN30
External
EPG
When you identify

endpoints that require Tomcat MySQL
isolation or contracts you
configure more EPG/
uEPG APP-01 VLAN30
Considerations when Migrating an Existing L2 Network to an
ACI fabric
▪ Make sure how BD forwards broadcasts and BPDU traffic

▪ Understand built-in Loop prevention mechanisms in ACI
▪ Make sure you understand how Spanning-Tree interacts with ACI
▪ Limit the interactions with TCN BPDUs
▪ Harden the Fabric with MCP and with EP move detection
▪ Be careful with ARP (or proxy ARP)
ACI Fabric Loop Detection
▪ ACI prevents loops from being introduced in the
fabric as follows: APIC
▪ With LLDP, by detecting if a leaf port is connected

to another leaf of the same fabric
▪ With the Miscabling Protocol (MCP)
▪ By detecting MAC moves
▪ With BPDU-guard on physical ports
▪ With BPDU-guard on virtual ports (in the VMM LLDP BPDU
domain) MCP
▪ By forwarding Spanning-Tree BPDUs (but ACI LLDP Loop

doesn't run STP per se) within an EPG Detection MCP Loop
Detection
(supported with STP Loop
11.1 release) Detection
Enabling MCP Globally 
Creating an MCP Interface Policy 
Global MCP Policy Properties
Initial Delay (sec): The delay time before MCP starts taking action based on the value of
Loop Protection Action value configured by users. From the system bootup until the initial
delay timer timeout, the MCP will only create syslog entry if a loop is detected. 180 seconds
by default.
Loop Detect Multiplication Factor: The multiplication factor which MCP uses to determine
when a loop is formed. It denotes the number of continuous packets a port has to receive
before claiming a loop is formed. 3 by default.
Loop Protection Action: This determines how MCP will take action when a loop is
detected. MCP would error-disable the port or send syslog only based on this value.
Transmission Frequency (sec): How often we will send MCP PDUs. 2 seconds by default.
By default Flooding is BD-wide, not just Restricted to an
EPG
BD1
EPG1 EPG2 leaf1 BD1 leaf3 BD1 BD1 leaf4
EPG1 EPG2 EPG1 EPG2

leaf2
1/1 1/2 1/3 1/4 1/5 1/6 1/7 1/8
VLAN 5, 6, 7, 8 VLAN 9,10,11,12
VLAN 5, VLAN 6 VLAN 9, VLAN 10
VLAN 5, VLAN 7
Broadcast VLAN 9, VLAN 11
BPDU Forwarding Within the EPG is Based on the VLAN
leaf4
leaf1 BD1 leaf3 BD1
BD1
EPG1 leaf2 EPG2 EPG1 EPG2
1/1 1/2 1/3 1/4 1/5 1/6 1/7 1/8
VLAN 5, VLAN 6 VLAN 9, VLAN 10
BPDUs
VLAN 5, VLAN 7
BPDUs
VLAN 9, VLAN 11
TCNs are Useful in this Topology should the Active Path
Change from switch1 to switch2
leaf4
leaf1 BD1 leaf3 BD1 BD1
EPG1 leaf2 EPG2 EPG1 EPG2
switch1 switch2
STP Root Switch
But is there a need for TCNs in ACI with this topology? No 
Limit STP impact on the fabric with vPC and Portfast/Trunkfast 
There is only one L2 path from ACI
Spines to the L2 network outside
Leafs
Servers
Virtual Machines
With Topologies that Require TCNs, Limit the Impact of
TCNs
▪ When ACI receives a TCN on a VLAN it flushes the endpoints of the BD that are
associated with that VLAN
▪ If you have an EPG that has local endpoints connected
▪ And
▪ Is also connected to the outside via L2
▪ Use a different VLAN for the locally attached endpoints
▪ And
▪ For the L2 extended network
For legacy networks connected to ACI consider BD set to unknown
unicast flooding First frame is flooded
Spines
MAC A
MAC A
Server 1
MAC 1
Virtual Machines
Checklist for Migration / Coexistence of Legacy Infrastructure
with ACI
▪ With clustered servers check how do they use ARP:
▪ Check the type of servers that you need to • E.g. if you have MNLB servers you need to use dedicated
connect to ACI: L2 BDs
• If one node is master at any given time is the only one
▪ Which teaming do they use? answering ARP request while all other servers forward
• If they use Transmit Load Balancing change the traffic with the same source IP you may need a dedicated
teaming configuration L2 BD
• If you are using virtualised servers make sure they ▪ Make sure BPDUs are passed through the fabric only if
use mac pinning equivalents or port-channelign necessary
▪ Are servers doing routing? If so you need to ▪ Limit BPDU TCN impact on the fabric to the EPGs/
connected them to a L3out VLANs that really require the TCN
▪ Are servers aggregating multiple clients like a ▪ If there is only one L2 path outside of the ACI fabric,
firewall doing source NAT? enable spanning-tree portfast / trunkfast on the outside
L2 switches
• If yes make sure that the number of source IP per
MAC is within the limits ▪ Set BDs that have outside L2 connectivity to unknown
• Make sure individual IP addresses are aged out unicast flooding unless you can remove BPDU TCNs
independently and you force all migrated servers to be discovered by
ACI
Where to Go for More Best Practices Information
✓ ACI Design Guide
https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-
paper-c11-737909.pdf
✓ About endpoint learning and BD settings

https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-
infrastructure/white-paper-c11-739989.html
✓ About Service Graphs
https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-
infrastructure/white-paper-c11-734298.html
✓ About the use of PBR in a Service Graph
https://www.cisco.com/c/en/us/solutions/data-center-virtualization/application-centric-infrastructure/
white-paper-c11-739971.html
Agenda
▪ ACI Foundations
Integrating an Existing DC with OTV 
Fabric ‘A’
BD extend via OTV/
VPLS with multiple
L2Out
OTV
EVPN
10.10.10.8 10.20.20.32 10.10.10.9

10.10.10.5 10.10.10.34 Subnet 10.10.10.0/24
DG = 10.10.10.1 DG = 10.10.10.1
• ACI fabrics can connect to any existing layer 2 DCI system, OTV, VPLS, VXLAN with EVPN, …
• Same basic L2 connection as connecting to any other Nexus or Catalyst switch
http://www.cisco.com/c/en/us/solutions/data-center-virtualization/application-centric-infrastructure/
white-paper-c11-737077.html
Integrating new ACI Fabrics with Existing OTV 
Fabric ‘A’ Fabric ‘B’

BD extend via OTV/
VPLS with multiple
L2Out
OTV
EVPN
• ACI fabrics can connect to any existing layer 2 DCI system, OTV, VPLS, VXLAN
with EVPN, …
• Same basic L2 connection as connecting to any other Nexus or Catalyst switch
• Improvements with 11.2 ACI release provide an automatic common default
gateway across both connected fabrics
• Prior to 11.2 manual configuration is required for default GW and ARP config
Support for the same Pervasive GW IP & MAC on
independent fabrics
Fabric ‘A’ Fabric ‘B’
BD extend via OTV/
VPLS with multiple
L2Out
OTV
EVPN
10.10.10.5 10.10.10.34 Subnet 10.10.10.0/24 10.10.10.57 10.10.10.39

DG = 10.10.10.1 DG = 10.10.10.1 DG = 10.10.10.1 DG = 10.10.10.1
• Support for the same default gateway address on independent fabrics with BD extended
(same pervasive default gateway)
• Same subnet exists in independent fabrics with layer 2 transport (OTV, VPLS, VXLAN
EVPN)
• Common gateway to allow attachment of device with hard coded DG at either site
• Replicates the current vPC HSRP localisation design
Removing the Classic L2/L3 Boundaries
DCI is different now
Site 1 DP-ETEP B Site 2

DP-ETEP A
IP Backbone
S1 S2 S3 S4 S5 S6 S7 S8
= VXLAN Encap/Decap
EP1 EP2
Communication between endpoints in separate sites (Layer 2 and/or Layer 3) is enabled

simply by creating and pushing a contract between the endpoints’ EPGs
ACI & SDWAN Integration 
Target 2HCY18
▪ Purpose is to provide end to end path visibility/control to perform application based routing from
the DC to the branch WAN edge
▪ APIC and vManage (DNAC in the future) will exchange group, path and policy requirements
▪ Application traffic will receive appropriate WAN service level
ACI Control/forwarding Segment SDWAN Control/forwarding Segment
1 Internet
Path
Path 2
App A
MPLS
4G LTE
Path
3
Where to Go for More Information
✓ ACI Stretched Fabric White Paper

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/kb/b_kb-aci-stretched-
fabric.html#concept_524263C54D8749F2AD248FAEBA7DAD78
✓ ACI Multi-Pod White Paper
http://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-
centric-infrastructure/white-paper-c11-737855.html?cachemode=refresh
✓ ACI Dual Fabric Design Guide
http://www.cisco.com/c/en/us/solutions/data-center-virtualization/application-centric-
infrastructure/white-paper-c11-737077.html?cachemode=refresh
✓ ACI and GOLF High Level Integration Paper
http://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/
application-centric-infrastructure/white-paper-c11-736899.html
Agenda
▪ ACI Foundations
Identity and Access Control  VTEP IP Group
Policy
VNID Tenant Packet
(TrustSec)
Shared Application
Servers
▪ Devices and users are Services
authenticated and
Ingress and
authorised into end-point ACI Fabric
Egress ➔
Enforcement
SGT’s) APIC
▪ End Point Group Tags Enterprise

Backbone
encoded in a VXLAN
header
▪ Policies between scalable
Employee Tag
following the provider/
Non-Compliant Employee Voice Voice Employee Non-Compliant Tag
VLAN A VLAN B
Identity and Access Control  VTEP IP Group
Policy
VNID Tenant Packet
(TrustSec)
Shared Application
Servers
▪ Devices and users are Services
authenticated and
Ingress and
authorised into end-point ACI Fabric
Egress ➔
Enforcement
SGT’s) APIC
▪ End Point Group Tags Enterprise

Backbone
encoded in a VXLAN
header
▪ Policies between scalable
Employee Tag
following the provider/
Non-Compliant Employee Voice Voice Employee Supplier Non-Compliant Non-Compliant Tag
VLAN A VLAN B
ACI Policy and Security Fundamentals   Contract
EPGs, Identity and Access Control Filters
Web Tier App Tier
▪ Each EPG is identified by a specific Group Policy End Points End Points
Filter
‘class-id’
Filter
▪ All traffic sourced from an endpoint is identified by the
class-id corresponding to it’s EPG membership Filter
▪ When one endpoint communicates with another

endpoint the Fabric checks that the class-id of the
source is permitted to communicate to the class-id of Provider identified by
Consumer identified by
the destination using the specific ports as defined by the D_Class
the S_Class (Source
the filters in the contract EPG Class)
(Destination EPG
Class)
▪ Communication policy is ‘directional’ (policy checks
both that both Web to App and App to Web are
allowed)
▪ Note: An EPG has a unique class-id, ‘source class’
and ‘destination class’ only refer to relative policy
Independent Policy for
enforcement (which direction is being enforced) reverse direction
ACI Policy and Security Fundamentals   Contract
EPGs, Identity and Access Control Filters
Web Tier App Tier
▪ Each EPG is identified by a specific Group Policy End Points End Points
Filter
‘class-id’
Filter
▪ All traffic sourced from an endpoint is identified by the
class-id corresponding to it’s EPG membership Filter
▪ When one endpoint communicates with another

endpoint the Fabric checks that the class-id of the
source is permitted to communicate to the class-id of Provider identified by
Consumer identified by
the destination using the specific ports as defined by the D_Class
the S_Class (Source
the filters in the contract EPG Class)
(Destination EPG
Class)
▪ Communication policy is ‘directional’ (policy checks
both that both Web to App and App to Web are
allowed)
▪ Note: An EPG has a unique class-id, ‘source class’
and ‘destination class’ only refer to relative policy
Independent Policy for
enforcement (which direction is being enforced) reverse direction
ACI Multi-Site 
Network and Identity Extended between Fabrics
Network information carried across Identity information carried across
Fabrics (Availability Zones) Fabrics (Availability Zones)
VTEP IP VNID Class-ID Tenant Packet No Multicast Requirement in

Backbone, Head-End
Replication (HER) for any
IP Network Layer 2 BUM traffic)
MP-BGP - EVPN
Federating Identity between Domains
TrustSec-ACI Integration
▪ Sharing Groups between TrustSec and ACI domains began with ISE 2.1
▪ Allow TrustSec (SDA) security groups to be used in ACI policies
▪ Allow ACI EndPoint Groups to be used in policies in TrustSec domain
SDA Policy Domain ACI Policy Domain
Campus / Branch / Non-ACI DC

ISE 2.1
APIC  Data Centre
TrustSec Policy Domain DC
APIC Policy Domain
Voice Employee Supplier BYOD

ACI Fabric Web App DB
Voice Data
VLAN VLAN
SDA and ACI 
Border Leaf Leveraging IP Based EPG
ISE & APIC
SDA ISE Policy Domain Exchange Groups ACI Policy Domain
and Member
Security Groups information End Point Groups
Cisco APIC-DC
Cisco ISE ISE notifies APIC

about IP endpoint
APIC-EM IP, SGT mappings to EPG mappings IP-ClassId, VNI bindings
SDA & iWAN
L3Out
External EPG
User Switch Router* Classified and ACI Nexus9000 ACI Fabric Server
implements policy Border Leaf
Classification
LISP,SGT & VXLAN
Extended Access Control with Cisco Firepower
TrustSec & ACI Policy Groups in Access Rule
Extended Access Control with Cisco Firepower
TrustSec & ACI Policy Groups in Access Rule
Agenda
▪ ACI Foundations
ACI Anywhere 
ACI Network and Policy Extension Multisite Orchestrator
IP Network
vSpine +
vLeaf
AVE AVE AVE AVE
AVE
Hypervisor Hypervisor Hypervisor Hypervisor
Remote Physical Leaf AVE with non-N9K
(N9K)
Site 1 Site 2
ACI Anywhere
Multi-Cloud Policy and Management Extension
Multisite Orchestrator
IP Network
Site 1 Site 2
Why use Cloud Constructs? 
Policy Mapping - AWS
User Account Tenant
Virtual Private Network VRF
VPC subnet BD Subnet
Security Group EPG
Contracts
Security Group Rule EPG Contracts
Outbound rule Consumed contracts

Source/Destination: Subnet or IP or Any or ‘Internet’
Protocol
Port
Inbound rule Provided contracts

EC2 Instance
Network Adapter
Why use Cloud Constructs?  
Policy Mapping - Azure
Resource Group Tenant
Virtual Network VRF
Subnet BD Subnet
Application Security Group EPG

(ASG)
Network Security Group

(NSG) EPG Contracts
Outbound rule Consumed contracts

Source/Destination: ASG or Subnet or IP or Any or ‘Internet’
Protocol
Port
Inbound rule Provided contracts

Virtual Machine
Network Adapter
Connect the User Domain to the Application
End to End Identity and Intent Based Infrastructure
APIC & ACI Multi-Site
Tetration Analytics
CloudCentre Platform
vPod
Service Service Service
VM VM VM
DB App Web
Application User Group
Group Based Policy (Intent) 83

BRKACI-2001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Q&A
84
Complete Your Online  
Session Evaluation
• Give us your feedback and receive
a Cisco Live 2018 Cap by
completing the overall event
evaluation and 5 session
evaluations.
• All evaluations can be completed
via the Cisco Live Mobile App.
Don’t forget: Cisco Live sessions will be

available for viewing on demand after the
event at www.CiscoLive.com/Global.
Thank you
86

Brkaci 2001

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Brkaci 2001

Uploaded by

Copyright:

Available Formats

Integration and Interoperation of

Existing Nexus Networks into an

Data Plane Based Endpoint Discovery

Control Plane Based Endpoint Location Tracking

End Point Group Tags Enterprise

consumer model Supplier Tag

End Point Group Tags Enterprise

consumer model Supplier Tag

Common Operational Properties - AD, LDAP, …

MACSEC and INS-SEC Firewall at Each

Branch Web1 App1 DB Identity & Policy

Filter Service Filter

MACSEC and INS-SEC Firewall at Each

Branch Web1 App1 DB Identity & Policy

Filter Service Filter

Assurance Functions Logical Description ‘Write to’

Correlation of Enforcement and Telemetry

• Cisco Tetration and ACI are designed to provide

Correlation of Enforcement and Telemetry

• Cisco Tetration and ACI are designed to provide

Correlation of Enforcement and Telemetry

• Cisco Tetration and ACI are designed to provide

Provide an ‘Application Allow Application Admin

▪ Integrating Existing Nexus Layer 2 and Layer 3

VTEP GPO VXLAN MACIP Payload IP Transport

VTEP VTEP VTEP VTEP VTEP VTEP

• Cisco’s ACI solution leverages an integrated VXLAN based overlay

IP Forwarding: MAC Forwarding:

Forwarded using DIPi Forwarded using DMAC

10.1.3.11 10.1.3.35 10.6.3.2 10.6.3.17

10.1.1.10 10.1.3.11 10.6.3.2 10.1.3.35 10.1.3.11 10.1.3.35

Distributed Default Gateway Directed ARP Forwarding and/or Proxy-

Layer 2/3 Host Mobility

Single APIC Cluster

Any traffic that requires use of the Spine vSwitch

* Spine services will be required in scenario’s like copy

BGP EVPN Carries

AVE AVE AVE

AVE vSwitch vSwitch DME/PE COOP BGP

AVE vSwitch vSwitch COOP BGP

Common WAN Services

Pod ‘A’ Pod ‘B’ Pod ‘C’ Pod ‘D’

APIC Cluster APIC Cluster

Network Configuration Network Configuration Network Configuration Network Configuration

Availability Zone Availability Zone

▪ Integrating Existing Nexus Layer 2 and Layer 3

BD1 BD2 BD2

EPG1 EPG2 EPG3 EPG4 EPG5 EPG6 EPG7 EPG8 EPG9

• Communication policy includes C C C

filters (ACLs), QoS, Route

between EPGs or between L3out Bridge Domains

Health scores, statistics, logs

EPG, uEPG, domain

Developer: My application template NOC: What I monitor and troubleshoot

Inside View: The portion of the database that your

Subnet Subnet Subnet Subnet Subnet

• Bridge Domain (BD)

Extend L2 domain beyond ACI fabric - 2 options

Lets Look at the

Extend L2 domain beyond ACI fabric - 2 options

• VLANs are localised to the leaf ports

• Single Policy Group (one extended EPG)

Anycast Default Gateway