ACI Multi-Site Deployment

ACI Multi-Site
Deployment
John Weston – Technical Marketing Engineer

BRKACI-3502
Cisco Spark
Questions?
Use Cisco Spark to communicate
with the speaker after the session
How
1. Find this session in the Cisco Live Mobile App
2. Click “Join the Discussion”
3. Install Spark or go directly to the space
4. Enter messages/questions in the space
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Session Objectives
At the end of the session, the participants should be able to:
• Articulate the different deployment options to interconnect
Cisco ACI networks (Multi-Pod vs. Multi-Site)
• Understand the functionalities and specific design
considerations associated to the ACI Multi-Site architecture
Initial assumption:
• The audience already has a good knowledge of ACI main
concepts (Tenant, BD, EPG, L2Out, L3Out, etc.)
BRKACI-3502 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Agenda
• ACI Network and Policy Domain • Control and Data Plane
Evolution • Connecting to the External Layer 3
• ACI Multi-Pod Quick Review Domain
• Network Services Integration
• ACI Multi-Site Deep Dive
• Virtual Machine Manager (VMM)
• Overview and Use Cases Integration
• Introducing ACI Multi-Site Orchestrator
• Migration Scenarios
(MSO)
• Inter-Site Connectivity Deployment • Conclusions and Q&A
Considerations
ACI Network and Policy Domain
Evolution
Introducing: Application Centric Infrastructure (ACI)
Web App DB
Outside QoS QoS QoS
(Tenant Service
Filter Filter
VRF)
APIC
Application Policy
ACI Fabric Infrastructure Controller
Integrated GBP VXLAN Overlay
Cisco ACI
Fabric and Policy Domain Evolution
ACI Single Pod Fabric ACI Stretched Fabric ACI Multi-Pod Fabric
IPN
Pod ‘A’ Pod ‘n’
DC1 APIC Cluster DC2 MP-BGP - EVPN
…
APIC Cluster
ACI 1.0 - ACI 1.1 - Geographically ISE 2.1 & ACI 1.2 ACI 2.0 - Multiple Networks ACI 3.0 – Multiple Availability ACI 3.1/3.2 - Remote Leaf
Leaf/Spine Single Stretch a single Pod Federation of Identity (Pods) in a single Availability Zones (Fabrics) in a Single and vPod extends an
Pod Fabric and Interconnect Zone (Fabric) Region ’and’ Multi-Region Availability Zone (Fabric) to
TrustSec and ACI using Policy Management remote locations
IP based EPG/SGT
IP
Fabric ‘A’ Fabric ‘n’
MP-BGP - EVPN
…
ISE
ACI Multi-Site
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
Regions and Availability Zones
OpenStack and AWS Definitions
OpenStack
• Regions - Each Region has its own full OpenStack

deployment, including its own API endpoints, networks
and compute resources
• Availability Zones - Inside a Region, compute nodes can
be logically grouped into Availability Zones, when launching
new VM instance, we can specify AZ or even a specific
node in a AZ to run the VM instance
Amazon Web Services

• Regions – Separate large geographical areas, each
composed of multiple, isolated locations known as
Availability Zones
• Availability Zones - Distinct locations within a region
that are engineered to be isolated from failures in other
Availability Zones and provide inexpensive, low latency
network connectivity to other Availability Zones in the
same region
Terminology
• Pod – A Leaf/Spine network sharing a common control plane (ISIS, BGP,
COOP, …)
 Pod == Network Fault Domain
• Fabric – Scope of an APIC Cluster, it can be one or more Pods
 Fabric == Availability Zone (AZ) or Tenant Change Domain
• Multi-Pod – Single APIC Cluster with multiple leaf spine networks
 Multi-Pod == Multiple Networks within a Single Availability Zone (Fabric)
• Multi-Fabric – Multiple APIC Clusters + associated Pods (you can have
Multi-Pod with Multi-Fabric)*
 Multi-Fabric == Multi-Site == a DC infrastructure region with multiple AZs
* Available from ACI release 3.2

Typical Requirement
Creation of Two Independent Fabrics/AZs
Fabric ‘A’ (AZ 1)
Fabric ‘B’ (AZ 2)
Application
workloads
deployed across
availability zones BRKACI-3502 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
Typical Requirement
Creation of Two Independent Fabrics/AZs
Multi-Pod Fabric ‘A’ (AZ 1)

‘Classic’ Active/Active
Pod ‘1.A’ Pod ‘2.A’
ACI Multi-Site
Multi-Pod Fabric ‘B’ (AZ 2)

Application
Pod ‘1.B’workloads Pod ‘2.B’
deployed across
ACI Multi-Pod Quick Review
ACI Multi-Pod
Overview
VXLAN
Inter-Pod Network
Pod ‘A’ Pod ‘n’
MP-BGP - EVPN
APIC Cluster
IS-IS, COOP, MP-BGP IS-IS, COOP, MP-BGP
Availability Zone
• Multiple ACI Pods connected by an IP Inter-Pod • Forwarding control plane (IS-IS, COOP)
L3 network, each Pod consists of leaf and spine fault isolation
nodes • Data Plane VXLAN encapsulation between
• Managed by a single APIC Cluster Pods
• Single Management and Policy Domain • End-to-end
BRKACI-2003
BRKACI-3502© 2018policy
Cisco and/or enforcement
its affiliates. All rights reserved. Cisco Public 14
Single Availability Zone with Maintenance &
Configuration Zones
Scoping ‘Network Device’ Changes
Maintenance Zones – Groups of
switches managed as an “upgrade”
Inter-Pod Network
group
ACI Multi-Pod
Fabric
APIC Cluster
Configuration Zone ‘A’ Configuration Zone ‘B’

• Configuration Zones can span any required set of switches, simplest approach may be to map a
configuration zone to an availability zone, applies to infrastructure configuration and policy only
Single Availability Zone with Tenant Isolation
Isolation for ‘Virtual Network Zone and Application’ Changes
Inter-Pod Network
ACI Multi-Pod
Fabric
APIC Cluster
Tenant ‘Prod’ Configuration/Change Domain Tenant ‘Dev’ Configuration/Change Domain
• The ACI ‘Tenant’ construct provide a domain of application and associated virtual
network policy change
• Domain of operational change for an application (e.g. production vs. test)
ACI Multi-Pod
Supported Topologies
Intra-DC Two DC sites directly connected
1G/10G/40G/100G
10G*/40G/100G 10G*/40G/100G
POD 1 10G*/40G/100G 10G*/40G/100G
POD n POD 1 Dark fibre/DWDM POD 2
(up to 50** msec RTT)
…
APIC Cluster APIC Cluster
3 (or more) DC Sites directly connected Multiple sites interconnected by a

1G/10G/40G/100G generic L3 network
10G*/40G/100G
POD 1 10G*/40G/100G POD 2
Dark fibre/DWDM 10G*/40G/100G 10G*/40G/100G
(up to 50 msec RTT)
L3
10G*/40G/100G
10G*/40G/100G (up to 50msec RTT)
10G*/40G/100G
POD 3 * 10G only with QSA adapters on EX/FX and 9364C spines
BRKACI-3502 **© 50
2018msec support
Cisco and/or added
its affiliates. in SW
All rights release
reserved. Cisco2.3(1)
Public 17
ACI Multi-Site Deep Dive
Overview and Use
Cases
ACI Multi-Site VXLAN
ACI 3.0 Release
Overview IP Network
MP-BGP - EVPN
Multi-Site Orchestrator
Site 1 Site 2
REST
GUI
API Availability Zone ‘B’
Availability Zone ‘A’
• Separate ACI Fabrics with independent APIC clusters • MP-BGP EVPN control plane between sites
• ACI Multi-Site Orchestrator pushes cross-fabric • Data Plane VXLAN encapsulation across
configuration to multiple APIC clusters providing sites
scoping of all configuration changes • End-to-end policy definition and enforcement
ACI Multi-Site
Main Use Cases
Scale-Up Model to Build a Large Data Centre Interconnect (DCI)

Intra-DC Network
ACI Multi-Site
Network and Identity Extended between Fabrics
Network information carried across Identity information carried across
Fabrics (Availability Zones) Fabrics (Availability Zones)
VTEP IP VNID Class-ID Tenant Packet

No Multicast Requirement
in Backbone, Head-End
IP Network Replication (HER) for any
Layer 2 BUM traffic)
MP-BGP - EVPN
ACI Multi-Site
Inter-Site Policies and ‘Shadow’ EPGs
IP
Network
• Inter-Site policies defined on DP-ETEP A DP-ETEP B
the ACI Multi-Site Orchestrator S1 S2 S3 S4 S5 S6 S7 S8
are pushed to the respective

APIC domains
 End-to-end policy consistency
 Creation of ‘shadow’ EPGs to EP1 EP2
Site 1 Site 2
locally represent the policies
EP1 EP1
•
C EP2 EPG
Policies are enforced at the EPG
C EP2 EPG
‘Shadow’
‘Shadow
EPG
Objects’
EPGs
ingress leaf node, once it has
learned on the data plane info
for remote endpoint EP1
C
EP2
EPG EPG
ACI Multi-Site VNID  16678781
Namespace Normalisation IP Network Class-ID: 49153 Translation of Class-ID, VNID
(scoping of name spaces)
VNID  16678781 Spine Translation Table
Class-ID: 49153
Rem. Site Local Site
VNID 16678781 16547722

MP-BGP - EVPN Class-ID 49153 32770
VNID  16547722
Class-ID: 32770
VNID  16678781
Class-ID: 49153 EP1 EP2 Site 2
Site 1 EPG
C EPG
Leaf to Leaf VTEP, Class-ID is local to the Fabric
Leaf to Leaf VTEP, Class-ID is local to the Fabric
VNID Class-ID Tenant Packet
VNID Class-ID Tenant Packet VNID Class-ID Tenant Packet
• Maintain separate name spaces with ID translation performed on the spine nodes
• Requires specific HW on the spine to support for this functionality
• Multi-Site Orchestrator instructs local APIC to program translation tables on spines
ACI Multi-Site
Hardware Requirements
• Support all ACI leaf switches (1st Generation, -EX and -FX)
• Only –EX spine (or newer) to Can have only a subset
of spines connecting to
connect to the inter-site network IP Network
the IP network
• New 9364C non modular spine

1st Gen 1st Gen -EX -EX
(64x40G/100G ports) supported for
Multi-Site from ACI 3.1 release (shipping)
• 1st generation spines (including 9336PQ)
not supported
• Can still leverage those for intra-site leaf
to leaf communication
ACI Multi-Site Networking Options
Per Bridge Domain Behaviour
Layer 3 only across sites IP Mobility without BUM flooding Layer 2 adjacency across Sites
1 2 3
ISN ISN ISN
Site Site Site Site 2
Site Site Site
1 2 1 2 1 2
 Bridge Domains and subnets not  Same IP subnet defined in separate  Interconnecting separate sites for
extended across Sites Sites fault containment and scalability
reasons
 Layer 3 Intra-VRF or Inter-VRF  Support for IP Mobility (‘cold’ and
communication (shared services ‘live’* VM migration) and intra-  Layer 2 domains stretched across
across VRFs/Tenants) subnet communication across sites Sites, support for ‘live’* VM
migration and application clustering
 No Layer 2 BUM flooding across
sites  Layer 2 BUM flooding across
sites
MSO GUI MSO GUI MSO GUI

(BD) (BD) (BD)
*’Live’ migration officially supported

BRKACI-3502 © 2018 Cisco from ACI
and/or its release
affiliates. 3.2
All rights reserved. Cisco Public 26
ACI Multi-Site
Scalability Values Supported in 3.1 Release
Scale Parameter Stretched Objects Site Local Objects
Sites 8 N/A
Leaf scale 800 across all sites 200 per site
Tenants 200 2500 Stretched Object

Scale
VRFs 400 3000
IP Subnets 2000 10000
BD 2000 10000
EPGs 2000 10000
Endpoints 50000 100000
Contracts 2000 2000 APIC Domain 1 APIC Domain 2

Scale Scale
L3Outs External EPGs 500 (prefixes) 2400
IGMP Snooping 8,000 8,000
ACI Multi-Site
CloudSec Encryption for VXLAN Traffic Encrypted Fabric to Fabric Traffic
[ GCM-AES-128 (32-bit PN), GCM--AES-256
(32-bit PN), GCM-AES-128-XPN (64-bit PN),
VTEP Information Clear Text GCM-AES-256-XPN (64-bit PN)])
VTEP IP MACSEC VXLAN Tenant Packet
Inter-Site Network
MP-BGP - EVPN
Support planned for a future ACI release for FX line cards and 9364C platform
ACI Multi-Site
Spines in Separate Sites Connected Back-to-Back
Dark fibre/DWDM
10G*/40G/100G
Dark fibre/DWDM
10G*/40G/100G
ISN
Site 1 Site 2 Site 1 Site 2
Site 3 Site 3
• Multiple DC sites directly connected • ‘Hybrid’ topology with some sites directly
• Supported from ACI release 3.2 (Q2CY18) connected and other reachable via the ISN
• 10G connection supported with QSA adapter • Supported in a future ACI release (2HCY18)
on spine nodes
Introducing ACI
Multi-Site
Orchestrator
ACI Multi-Site
Multi-Site Orchestrator (MSO)
• Micro-services architecture
 Multiple MSO nodes are created and run
concurrently (active/active)
REST
API
GUI  vSphere VM only form factor initially (physical
appliance planned for ACI 3.2 release)
ACI Multi-Site
• OOB Mgmt connectivity to the APIC clusters
deployed in separate sites
VM VM VM
 Support for 500 msec to 1 sec RTT
• Main functions offered by MSO:
Hypervisor  Monitoring the health-state of the different ACI Sites
 Provisioning of day-0 configuration to establish
….. inter-site EVPN control plane
Site 1 Site 2 Site n  Defining and provisioning policies across sites
 Day-2 operation functionalities
ACI Multi-Site
MSO Deployment Considerations
Intra-DC Deployment Interconnecting DCs over WAN
New York
Site3
IP Network
WAN
Milan Rome
Hypervisor Hypervisor Hypervisor Site1 Site2
VM VM VM
ACI Multi-Site Orchestrator

Hypervisor Hypervisor
ACI Multi-Site
VM VM Orchestrator VM
• Hypervisors can be connected directly to the DC OOB network • Up to 150 msec RTT latency supported between MSO nodes
• Each MSO node has a unique routable IP • Higher latency (500 msec to 1 sec RTT) between MSO nodes
• Async calls from MSO to APIC and managed APIC clusters
• If possible deploy MSO nodes in separate sites for availability
purposes (network partition
BRKACI-3502
scenarios)
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
ACI Multi-Site
MSO Dashboard
• Health/Faults for all

managed sites
• Easily way to identify
stretched policies across
sites
• Quickly search for any
deployed inter-site policy
• Provide direct access to
the APIC GUIs in
different sites
ACI Multi-Site
MP-BGP/EVPN Infra Configuration
• Configure Day-0 infra

policies
• Select spines establishing
MP-BGP EVPN peering
with remote sites
• Site/Pod Data Plan TEPs
(DP-ETEPs)
• Spine Control Plane TEPs
(CP-ETEPs)
ACI Multi-Site
MSO Schema and Templates
• Template = APIC policy definition
(App & Network) Schema
POLICY
• Template is the scope/granularity TemplateDEFINITION
Template
of what can be pushed to sites EP1
EPG
C EP2
EPG
• Template is associated to all

managed sites or to a subset SITE
of sites LOCAL
• Schema = container of Templates

sharing a common use-case
Site 1 Site 2
• Scope of change: policies in EFFECTIVE
EFFECTIVE
different templates can be POLICY POLICY
pushed to separate sites at

different times
ACI Multi-Site ACI 3.2 Release
Day-2 Operations: Full-Stack Consistency Checker
• Multi-Site Infra: Unicast, Multicast, BGP

TEPs and Tunnel state
• Multi-Site Tenant and EPG granularity:
 Inspect and validate full-stack programming:
MSC, APICs and Spine translations
 Validate the consistency of local and remote
MP-BGP EVPN
inter-site EPGs, BD, VRF, External EPG,
policies, etc.
Spines VXLAN Spines
 Root cause configuration programming
issues without calling TAC
• GUI and APIs will both be supported
APIC vs. ACI Multi-Site Orchestrator Functions
• Central point of management and • Complementary to APIC

configuration for the Fabric • Provisioning and managing of “Inter-Site
• Responsible for all Fabric local functions Tenant and Networking Policies”
• Fabric discovery and bring up • Scope of changes
• Fabric access policies • Granularly propagate policies to multiple APIC
• Service graphs clusters
• Domains creation (VMM, Physical, etc.)
• … • Can import and merge configuration from
• Integration with third party services different APIC cluster domains
• Maintains runtime data (VTEP address, VNID, • End-to-end visibility and troubleshooting
Class_ID, GIPo, etc.) • No run time data, configuration repository
• No participation in the fabric control and data • No participation in the fabric control and data
planes planes
Inter-Site
Connectivity
Deployment
Considerations
ACI Multi-Site
Inter-Site Network (ISN) Requirements
Inter-Site Network
MP-BGP - EVPN
• Not managed by APIC, must be separately configured (day-0 configuration)

• IP topology can be arbitrary, not mandatory to connect to all spine nodes, can extend long distance
(across the globe)
• Main requirements:
 OSPF on the first hop routers to peer with the spine nodes and exchange site specific E-TEP reachability
 Increased MTU support (at least +100B) to allow site-to-site VXLAN traffic
Control and
Data Plane
ACI Multi-Site
BGP Inter-Site Peers
• Spines connected to the Inter-Site Network
IP perform two main functions:
Network
1. Establishing MP-BGP EVPN peerings with
BGP Inter-Site
remote sites
Peers
 One dedicated Control Plane ETEP address (CP-ETEP)
is assigned to each spine running MP-BGP EVPN must
1st Gen 1st Gen be a globally routable address
DP-ETEP, HER-ETEP
 Full mesh MP-BGP EVPN peerings with BGP Inter-Site
CP-ETEP CP-ETEP Peers in remote sites
1 2
 Received EVPN information is synced (via COOP) with
the other local spines that are not BGP Inter-Site Peers
2. Forward inter-sites data-plane traffic
 Anycast Data Plane ETEP (DP-ETEP): assigned to all
the spines connected to the ISN and used to receive
L2/L3 unicast traffic
 Anycast HER* ETEP (HER-ETEP): assigned to all the
spines connected to the ISN and used to receive L2
BUM traffic
*HER: Head-End Replication BRKACI-3502 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
ACI Multi-Site IP Network Routing Table
Exchanging TEP Information across Sites DP-ETEP A, HER-ETEP A

CP-ETEP S3-S4
DP-ETEP B, HER-ETEP B
• OSPF between spines and CP-ETEP S5-S8
Inter-Site network (only
supported option in 3.0)
OSPF Inter-Site
• Exchange of External Spine OSPF
Network
TEP addresses (DP-ETEPs and
CP-ETEPs) across sites DP-ETEP A
S5
DP-ETEP B
S6 S7 B S8
S1 S2 HER-ETEP
S3 S4A HER-ETEP
TEP Pool information not advertised CP-ETEPs S3-S4 IS-IS to OSPF CP-ETEPs S5-S8
mutual redistribution
to the Inter-Site network TEP Pool 1 TEP Pool 2
Recommended to use not
overlapping TEP Pools if possible
• Multicast support not required in Site 1 Site 2

the Inter-Site Network
Leaf Routing Table Leaf Routing Table
Head-End Replication (HER) for L2
BUM traffic (only for stretched BDs) IP Prefix Next-Hop IP Prefix Next-Hop
DP E-TEP B Pod1-S3, Pod1-S4 DP E-TEP A Pod2-S1, Pod2-S2,
Pod2-S3, Pod2-S4
ACI Multi-Site
Inter-Site MP-BGP EVPN Control Plane
• MP-BGP EVPN used to
S3-S4 Table S5-S8 Table
communicate Endpoint (EP)
EP1 Leaf 1 EP2 Leaf 4
information across Sites MP-BGP EVPN
EP2 DP-ETEP B EP1 DP-ETEP A
MP-iBGP or MP-EBGP peering
supported across sites
Remote host route entries (EVPN IP
Type-2) are associated to the remote Network
site Anycast DP-ETEP address DP-ETEP B
DP-ETEP A
S1 S2 S3 S4 S5 S6 S7 S8
• Automatic filtering of endpoint
information across Sites
COOP COOP
Host routes are exchanged only if
there is a cross-site contract
requiring communication between EP2
EP1
endpoints Site 1 Site 2
Define and push inter-site policy

EP1 EP2
EPG
C EPG
ACI Multi-Site Policy information (EP1’s
Inter-Sites Unicast Data Plane Class-ID) carried across Pods
S2 has remote info for EP2
and encapsulates traffic to
remote DP ETEP B
EP1 AddressLeaf 4
S6 translates the VNID
EP2 Leaf 4
EP2 DP ETEP B S4 rewrites the S-VTEP and Class-ID to local DP ETEP A
IP values and sends traffic to
EP1
to be DP ETEP A
the local leaf
4 4
Site 1 3 5 Site 2
DP-ETEP A DP-ETEP B
S1 S2Proxy AS3 S4 S5 S6Proxy BS7 S8

EP2 e1/1
EP1 e1/3 EP1 DP ETEP A
5 * Proxy B
* Proxy A
Leaf learns remote Site
EP2 unknown, traffic is 2 location info for EP1
EP1 EP2
encapsulated to the local Proxy
A Spine VTEP (adding S_Class 1 6
information) EP1 sends traffic destined If policy allows it, EP2
to remote EP2 receives the packet
EP1 EP2
EPG
C EPG = VXLAN Encap/Decap
ACI Multi-Site Policy information (EP1’s
Inter-Site Data Plane (2) S_Class) carried across Pods
S3 translates the VNID

and S_Class to local EP1 Leaf 4
values and sends traffic to EP2 DP ETEP A
IP S6 rewrites the S-VTEP
the local leaf to be DP ETEP B
10 9
Site 1 DP-ETEP B Site 2
DP-ETEP A
S1 S2 S3 S4 S5 S6 S7 S8
EP1 e1/3
EP2 DP ETEP B EP1 DP ETEP A
** Proxy A
8 * Proxy B
11 Leaf encapsulates traffic to
Leaf learns remote Site remote DP ETEP address
location info for EP2 EP1 EP2
12 7
EP1 receives the packet EP2 sends traffic back to
remote EP1
EP1 EP2
EPG
C EPG = VXLAN Encap/Decap
ACI Multi-Site
Inter-Site Data Plane (3) = VXLAN Encap/Decap
From this point EP1 to EP2 communication is encapsulated Leaf to Remote Spine DP ETEPs in both directions
IP
Site 1 DP-ETEP B Site 2

DP-ETEP A
S1 S2 S3 S4 S5 S6 S7 S8
**
EP1 EP2
EP1 EP2
EPG
C EPG
ACI Multi-Site
Layer 2 BUM Traffic Data Plane
S3 is elected as Multi-Site forwarder for GIPo 1 S7 translates the VNID and the
BUM traffic  it creates an unicast VXLAN GIPo values to locally significant
packet with DP-ETEP A as S_VTEP and ones and associates the frame to
Multicast HER-ETEP B* as D_VTEP IP an FTAG tree
3 4
DP-ETEP A HER-ETEP B
S1 S2 S3 S4 S5 S6 S7 S8
BUM frame is flooded along the
tree associated to GIPo. VTEP
2 5
* learns VM1 remote location
*
EP1 DP-ETEP A
BUM frame is associated to
GIPo1 and flooded intra-site via Proxy B
*
the corresponding FTAG tree EP1 EP2
1 6
GIPo1 = Multicast Group EP1 generates a BUM EP2 receives the BUM
associated to EP1’s BD frame
frame
*This is a different ETEP address than the one used for inter-site L3 unicast communication
Connecting to the
External Layer 3
Domain
Connecting to the External Layer 3 Domain
‘Traditional’ L3Out on the BL Nodes
Client
PE
PE
WAN
PE
L3Out PE
• Connecting to WAN Edge devices at

Border Leaf nodes
Definition of a L3Out logical construct
• VRF-lite hand-off for extending L3
Border Leafs multi-tenancy outside the ACI fabric
Each tenant defines one (or more) L3Out
with a set of Logical Nodes, Logical
Interfaces, peering protocol
Connecting to the External Layer 3 For More Information on
ACI and GOLF Integration:
Domain Different WAN
LABACI-2101
‘GOLF’ Design Hand-Off options:
VRF-Lite, MPLS-
= VXLAN Encap/Decap VPN, LISP* Client
PE
PE
WAN
PE
PE
GOLF Routers (ASR 9000, ASR

DCI 1000, Nexus 7000)
OTV/VPLS
 Direct or indirect connection from spines to WAN Edge
(GOLF) routers
 Better scalability, one protocol session for all VRFs, no longer
constraint by border leaf HW table
 VXLAN handoff with MP-BGP EVPN
 Simplified tenant L3Out configuration
 Support for host routes advertisement out of the ACI Fabric
 VRF configuration automation on GOLF router through
OpFlex exchange
*LISP hand-off only on Nexus 7000 (8.2 release) BRKACI-3502 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
Multi-Site and L3Out
Endpoints Always Use Local L3Outs for Outbound Traffic
Supported Design Not Supported Design
Inter-Site Network Inter-Site Network

X
L3Out L3Out L3Out

Site 1 Site 2 Site 1
WAN WAN
EPG Web C1 ExtEPG-1 EPG Web C1 ExtEPG-1 EPG Web C1 ExtEPG-1 EPG Web C1 ExtEPG-1
Note: the same consideration applies to both Border Leaf L3Outs and GOLF L3Outs
Multi-Site and L3Out
Endpoints always Use Local L3Outs for Outbound Traffic
Inter-Site Network
Site 1 Site 2
Active Traffic dropped X Active

because of lack of
EPG Web C1 ExtEPG-1 state in the FW EPG Web C1 ExtEPG-2
Need a better solution than use of L3Outs to integrate perimeter FW!

Multi-Site and Network Services
Integration
Multi-Site and Network Services
Integration Models
ISN
• Active and Standby pair deployed across Pods
• Currently supported only if the FW is in L2 mode or in L3
mode but acting as default gateway for the endpoints
Active Standby
ISN
• Active/Active FW cluster deployed across Sites
• Not currently supported (scoped for a future ACI

Active/Active Cluster
release)
ISN • Most common deployment model for ACI Multi-Site

• Option 1: supported from 3.0 if the FW is connected in L3
mode to the fabric  mandates the deployment of traffic
ingress optimisation
• Option 2: supported from 3.2 release with the use of
Active/Standby Active/Standby Service Graph with© 2018
BRKACI-3502 Policy Based
Cisco and/or Redirection
its affiliates. All rights reserved. (PBR)
Cisco Public 55
Independent Active/Standby FW Pairs across Sites
Option 1: FWs Connected via L3Out
Inter-Site Network
Site 1 Site 2
L3Out L3Out
Site 1 Site 2
10.10.10.10 10.10.10.11
Active/Standby Active/Standby
EPG Web C1 ExtEPG-1 EPG Web C1 ExtEPG-2
• Ingress optimisation usually requires host route advertisement on the L3Out

 Currently supported only on GOLF L3Outs
 Native support on ACI Border Leaf nodes may come in© 2018
BRKACI-3502 2HCY18
Cisco and/or its affiliates. All rights reserved. Cisco Public 56
Independent Active/Standby FW Pairs ACI 3.2 Release
across Sites
Option 2: Use of Policy Based Redirection (Inbound Traffic)
Inter Site
Network
Site1 Site2
Compute leaf
always applies EPG EPG
the PBR policy Ext C Web Compute leaf
always applies
Provider Consumer the PBR policy
L3Out-Site1 L3Out-Site2
L3 Mode L3 Mode EPG

ResilientEPG
local FW Service Active/Standby Active/Standby
Web Cluster,
(Active/Standby Web
Active/Active Cluster,
Multiple Independent PBR
nodes)
• Inbound traffic can enter any site when destined to a stretched subnet (if ingress optimisation is not
possible or deployed)
• PBR policy is always applied on the compute leaf node where the destination endpoint is connected
• PBR always redirect traffic to the local FW nodes
• Mandates the deployment of ACI EX/FX leaf nodes BRKACI-3502 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
across Sites
Option 2: Use of Policy Based Redirection (Outbound Traffic)
Inter Site
Network
Site1 Site2
Compute leaf
always applies EPG EPG
the PBR policy Ext C Web Compute leaf
always applies
Provider Consumer the PBR policy
L3Out-Site1 L3Out-Site2
L3 Mode L3 Mode EPG

EPG
Web Web
• Outbound Traffic must always use a local L3Out connection

• PBR policy always applied on the compute leaf node where the destination endpoint is connected
• PBR always redirect traffic to the local FW nodes
• Mandates the deployment of ACI EX/FX leaf nodes
across Sites
Option 2: Use of Policy Based Redirection (East-West Traffic)
Inter Site
Network
Site1 Site2
Consumer leaf EPG EPG

always applies App C Web
the PBR policy
Provider Consumer
L3 Mode L3 Mode EPG

EPG
Web App
 Both EPGs can be stretched across sites

 Consumer leaf always redirect traffic to a local FW service node
 Provider leaf must not apply PBR policy to ensure proper traffic stitching to the FW node
that has built connection state
across Sites
Option 2: Use of Policy Based Redirection (East-West Traffic)
Inter Site
Network
Site1 Site2
Consumer leaf EPG EPG

always applies App C Web Provider leaf
the PBR policy does not apply
Provider Consumer
the PBR policy
L3 Mode L3 Mode EPG

EPG
Web App
 Both EPGs can be stretched across sites

 Consumer leaf always redirect traffic to a local FW service node
 Provider leaf must not apply PBR policy to ensure proper traffic stitching to the FW node
that has built connection state
Multi-Site and Virtual
Machine Manager
(VMM) Integration
ACI Multi-Site and VMM Integration
Option 1 – Separate VMM per Site
ISN
Site 1 VMM Domain VMM Domain Site 2

DC1 DC2
VMM 1 VMM 2
HV vSwitch1
HV HV Managed by HV vSwitch2
HV HV
VMM 1
Managed
by VMM 2
• Typical deployment model for an ACI Multi-Site

• Creation of separate VMM domains in each site, which are then exposed to the
Multi-Site Orchestrator
ACI Multi-Site and VMM Integration
Option 2 – Single VMM Managing Host Clusters in Separate Sites
ISN

DC1 DC2
VMM 1
HV vSwitch1
HV HV HV vSwitch2
HV HV
Managed
by VMM 1
• Even the deployment of a single VMM leads to the creation of separate VMM
domains across sites
ACI Multi-Site and vCenter Integration
Live Migration across Sites
ISN

DC1 DC2
vCenter vCenter
Server 1 Server 2
HV HVVDS1 HV
EPG1 HV HVVDS2 HV
EPG1
Live vMotion
• Live virtual machines migration across sites is supported only with vCenter
deployments (both for single or multiple vCenter options)
 Requires vSphere 6.0 and above
Migration
Scenarios
ACI Multi-Site
Migration Scenarios
Site 1 Site 1
1 2
Site 2 Site 2
2 2 1 3
Site 1 Site 2 Site 1 Site 2

Green Field Green Field Brown Field Green Field
1. Model new tenant and inter-site policies 1. Import existing tenant policies from site 1 to a
on the ACI Multi-Site Orchestrator and new template on ACI Multi-Site Orchestrator
associate the template to the sites
2. Associate the template also to site 2
2. Push policies to the ACI sites
3. Push template policies to site 2
ACI Multi-Site
Migration Paths Import Policies
from Site 1 and
Push it to Site 2
Fabric 1 ISN
‘Brownfield’ ACI Fabric to Site 1 Site 2
Multi-Site
ISN
Pod ‘A’ IPN Pod ‘B’ Multi-Pod to Pod ‘A’ IPN
Pod ‘B’
‘Hierarchical Multi-Site’ Site 2
APIC Cluster
APIC Cluster
Multi-Pod From ACI 3.2 release Site 1
Fabric 1 Fabric 2 ISN

Multi-Fabric Design to
Inter-Site Site 1 Site 2
App Multi-Site
L2/L3
DCI
Multi-Fabric Scoped for the future BRKACI-3502 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 67
Conclusion
ACI Multi-Pod & Multi-Site
A Reason for Both
Multi-Pod Fabric ‘A’ (AZ 1)

Pod ‘1.A’ Pod ‘2.A’
ACI Multi-Site
Multi-Pod Fabric ‘B’ (AZ 2)

Application
Pod ‘1.B’workloads Pod ‘2.B’
deployed across
Where to Go for More Information
 ACI Multi-Pod White Paper

http://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-
infrastructure/white-paper-c11-737855.html?cachemode=refresh
 ACI Multi-Pod Configuration Paper
https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-
infrastructure/white-paper-c11-739714.html
 ACI Multi-Site Cisco Live Las Vegas 2017
https://www.ciscolive.com/online/connect/sessionDetail.ww?SESSION_ID=95450&backBtn=true
 ACI Multi-Site White Paper
https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-
infrastructure/white-paper-c11-739609.html
Q&A
Complete Your Online
Session Evaluation
• Give us your feedback and
receive a Cisco Live 2018 Cap
by completing the overall event
evaluation and 5 session
evaluations.
• All evaluations can be completed
via the Cisco Live Mobile App.
Don’t forget: Cisco Live sessions will be
available for viewing on demand after the
event at www.CiscoLive.com/Global.
Thank you

ACI Multi-Site Deployment

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

ACI Multi-Site Deployment

Uploaded by

Copyright:

Available Formats

ACI Multi-Site

John Weston – Technical Marketing Engineer

DC1 APIC Cluster DC2 MP-BGP - EVPN

• Regions - Each Region has its own full OpenStack

Amazon Web Services

* Available from ACI release 3.2

Fabric ‘A’ (AZ 1)

Fabric ‘B’ (AZ 2)

Multi-Pod Fabric ‘A’ (AZ 1)

Pod ‘1.A’ Pod ‘2.A’

Multi-Pod Fabric ‘B’ (AZ 2)

Configuration Zone ‘A’ Configuration Zone ‘B’

Tenant ‘Prod’ Configuration/Change Domain Tenant ‘Dev’ Configuration/Change Domain

APIC Cluster APIC Cluster

3 (or more) DC Sites directly connected Multiple sites interconnected by a

Scale-Up Model to Build a Large Data Centre Interconnect (DCI)

VTEP IP VNID Class-ID Tenant Packet

the ACI Multi-Site Orchestrator S1 S2 S3 S4 S5 S6 S7 S8

are pushed to the respective

VNID 16678781 16547722

• New 9364C non modular spine

MSO GUI MSO GUI MSO GUI

*’Live’ migration officially supported

Leaf scale 800 across all sites 200 per site

Tenants 200 2500 Stretched Object

IP Subnets 2000 10000

EPGs 2000 10000

Endpoints 50000 100000

Contracts 2000 2000 APIC Domain 1 APIC Domain 2

IGMP Snooping 8,000 8,000

VTEP IP MACSEC VXLAN Tenant Packet

ACI Multi-Site Orchestrator

• Health/Faults for all

• Configure Day-0 infra

• Template is associated to all

• Schema = container of Templates

pushed to separate sites at

Day-2 Operations: Full-Stack Consistency Checker

• Multi-Site Infra: Unicast, Multicast, BGP

• Central point of management and • Complementary to APIC

• Not managed by APIC, must be separately configured (day-0 configuration)

Exchanging TEP Information across Sites DP-ETEP A, HER-ETEP A

• Multicast support not required in Site 1 Site 2

Define and push inter-site policy

S1 S2Proxy AS3 S4 S5 S6Proxy BS7 S8

VTEP IP VNID Class-ID Tenant Packet

S3 translates the VNID

Site 1 DP-ETEP B Site 2

• Connecting to WAN Edge devices at

GOLF Routers (ASR 9000, ASR

Supported Design Not Supported Design

Inter-Site Network Inter-Site Network

L3Out L3Out L3Out

Active Traffic dropped X Active

Need a better solution than use of L3Outs to integrate perimeter FW!

• Active/Active FW cluster deployed across Sites

• Not currently supported (scoped for a future ACI

ISN • Most common deployment model for ACI Multi-Site

EPG Web C1 ExtEPG-1 EPG Web C1 ExtEPG-2

• Ingress optimisation usually requires host route advertisement on the L3Out

L3 Mode L3 Mode EPG

L3 Mode L3 Mode EPG

• Outbound Traffic must always use a local L3Out connection

Consumer leaf EPG EPG