DGTL-BRKACI-2690-How To Extend Your ACI Fabric To Public Cloud

#CiscoLive
How to extend your ACI Fabric

to Public Cloud
(AWS & Azure)
Azeem Suleman, Principal Engineer
Lilian Quan, Principal Engineer
@suleman_azeem
DGTL-BRKACI-2690
#CiscoLive
Agenda
• Introduction
• Architecture
• Demo
• Use Cases
• Case Study
• References & Q&A
#CiscoLive DGTL-BRKACI-2690 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Introduction
ACI Extensions to Multi-Cloud
Multi-Site Orchestrator
VM VM VM
VM VM VM
VM VM VM
Cloud Site(s) On-Premises Cloud Site(s)
Cloud ACI Solution Components
MSO – Multisite
Orchestrator
Build Multi-Cloud Connections ACI Cloud

Orchestration Plane Deploy cloud controllers Controller
Author cross-cloud policies API
Management Endpoint
ORCHESTRATION Group (EPG)
Deploy cross-cloud policies Endpoint

(Instance)
Cloud Controllers Render cloud-specific constructs
Author policies in single cloud mode
CONTROL
Cloud APIs Cloud programming layer
Secure IPSEC
Data Channel
On Premises
Network and Cloud Native

VM VM VM
Security Policy Constructs
VM VM VM
Resources
ACI Multi-Site
Multi-Site Orchestrator (MSO)
3 VM Cluster
Any Routed IP Network
Site 1 Site 2 Site N
VM VM VM VM VM VM VM VM VM VM VM VM VM VM
VM VM VM VM VM VM VM
No Multicast <= 1s RTT Required (MSO à APIC) Single central management (MSO)
Phased Changes (Zones) Up to 12 Sites, distributed gateway Automated L2 DCI VXLAN extension
ACI Multi-Site
Software and Hardware Requirements
• Support all ACI leaf switches (1st Gen, EX, FX, FX2) Any Routed Network
• Modular Spine with EX/FX line card to

connect to the inter-site network
Can have only a subset
1st Gen 1st Gen -EX -EX of spines connecting to
• 9364c or 9332x fixed spine supported for the IP network
Multi-Site from ACI 3.1 release (shipping)
• 1st generation spines (including 9336PQ)
not supported
• Can still leverage those for intra-site leaf
to leaf communication
Fundamentals - AWS
• Regions – Think of it as multiple data center with more than Region
one physical location. Pod or site could be used for ACI
• Availability Zones (AZ) – Set of buildings, Internet uplinks Subnet
and power. Think of it as a data center but may contains Availability Zone 1 Availability Zone 2
more than one physical location. Path or node attachment
could be used in ACI
• Virtual Private Cloud (VPC) – Set of subnets with one ore ACI
more CIDR blocks running in a single region across multiple Pod
data centers (AZ). Similar to VRF VRF
• Subnet – Range of IP addresses. Each subnet must reside BD

within one AZ and can’t span zones. Minimum subnet size is Subnet Path Node Attachment
/28. BD Subnet
Fundamentals (Cont.) - AWS Route
table Router
Route
table
Network ACL Network ACL
• Security Group – Act as a firewall for associated EC2 instance

(VM), controlling both inbound and outbound traffic at network Security Group Security Security
Group Group
interface (EP) level. Equivalent to EPG with white-list
• Security Group Rule – Rules applied to inbound traffic (ingress)
or outbound traffic (egress). Combination of contracts and filters Subnet 1 Subnet 2
in ACI
L3out
• Network ACL – Used to deny / permit select traffic at a subnet VRF
level. Network ACLs are stateless. In ACI, it is similar to taboo
and grey-list contracts Routes
PSVI
Routes
• Route Table – Can be associated with multiple subnets. Acts Taboo Taboo
like a source-based policy-based routing (PBR) rule.
EPG EPG EPG
BD Subnet 1 BD Subnet 2
Connectivity Terms
For your info
& reference
AWS Only – External Connectivity
• Internet Gateway (IGW) – Horizontally scaled, redundant and highly available VPC component
that allows communication between instances in your VPC and the Internet
• NAT Gateway – Acts like an ECMP route to a set of NAT devices

• Virtual Private Gateway (VGW) – is the VPN concentrator. It terminates VPN and AWS Direct
Connect. Also provides BGP control plane for route-exchange
• Virtual Private Network (VPN) – comes in two flavors: VPNs provided through VGW and
instances running VPN software
• Direct Connect (DX) – Private dedicated link to an AWS region (not encrypted). Used for
speed and throughout.
• In ACI, IGW / VGW / DX is equivalent to L3out
For your info
& reference
Unified Cloud Native Networks - Mapping

Cisco ACI
Tenant
VRF
Azure Cloud AWS Cloud
Resource Group Bridge Domain Subnet
Account
Virtual Network EPG
VPC
Subnet Contracts, Filters
Subnet
App Security Group Consumed Contracts
Security Group
Network Security Group Provided Contracts
Security Group Rule
Inbound Rule Inbound Rule
Outbound Rule Outbound Rule
For your info
& reference
Cloud Hierarchy
ACI AWS Azure GCP

Security Domain OU Organization Organization
Tenant Account Subscription Project
Site / Pod Region Region Region
VRF / Context VPC VNet VPC
Path / Node Availability
AZ Subnet
Attachment sets/zones
BD Subnet Subnet Subnet Zone
Challenges in building a Multi Cloud environment
Building an automated and Maintain consistent policy, Requires a single pane of

secure interconnect between security and analytics for glass to manage policies
environments (on-premises, workloads deployed across across on-premise and cloud
Cloud and / or Multi-Cloud) environment (on-premises, locations
with ease of provisioning and Cloud and / or Multi-Cloud)
monitoring at scale locations
ACI Extension to Cloud
Cisco Multi-Site Orchestrator
Public Clouds
On-Premises DC
IP SG
SG Rule
SG
SG Rule
SG
Web APP DB
Network
EPG EPG EPG
Web
Contract
APP Contract
DB AWS Region
Internet
IP
VM VM VM Network ASG ASG ASG
NSG NSG
Web APP DB
Azure Region
Consistent Policy Enforcement on- Automated Inter-connect Simplified Operations with

Premises & Public Cloud provisioning end-to-end visibility
Architecture
Cloud APIC Architecture
• Cloud APIC image downloaded from Cloud
Provider Marketplace
• Interconnect connectivity automation (BGP-

Web Server (NGINX) EVPN, VXLAN)
Policy Distributor (PD) • Automates and manages cloud routers lifecycle
• Translates ACI Policy to cloud native Constructs

Policy Manager (PM)
• Deploys cloud resources and infrastructure
components
Cloud Policy Element
• Intuitive GUI and REST API North Bound Interface
Connector
• cAPIC manages 1 or more regions
API (AWS, Azure...) NetConf
(CSR1000v) • Supports Gov Cloud too
Cloud APIC Resources
• m5.2xlarge EC2 Instance type • Standard D8s v3 VM type

• 3.1 GHz Intel Xeon Platinum 8xxx • 2.3-2.4 GHz Intel Xeon E5-2673
series processors v4/v3 processors
• Balance of compute, memory and • Balance of compute, memory and
networking resources networking resources
Memory Network
vCPU Storage Network
(GiB) Performance
100–300
8 32 2xvNIC High
Gig
Cloud EPG and Cloud ExtEPG
• Cloud EPG:
A collection of network interfaces on the cloud provider, which will share the same
security policy. Can have endpoints in one or more subnets as well as can span across
regions. Tied to a VRF
• Cloud Ext EPG:

A set of subnets that represent the outside world compared to the cloud provider.
Outside world can either be another site or Internet.
Example: IPv4 internet as outside, cloudExtEPg will be identified with the subnet
0.0.0.0/0
Cloud Endpoint (EP) Classification
• Set of rules run against the cloud instances, once there is a match the endpoint (NIC)
is assigned to the Cloud EPG
• 4 classifier for endpoint assignment to Cloud EPG
a. Predefined
1. IP Address / Subnet
2. Region
3. Zone
b. Custom
1. Tags / label
• An endpoint can be classified into multiple Cloud EPGs

• Match operators are supported
Cloud EP Classification
Operators
• Below are few examples. You can use any combination
Key Operator Value
IP Address / Subnet =, != 10.10.10.1, 10.10.10.0/24
Region In, Not in us-west-1, us-east-1
Zone In, Not in us-west-1a, us-west-1b
Has key
Custom Application web, db
Doesn’t have key
Cloud EP Classification
Micro Segmentation
• Example 1: Cloud EPG ”Dev”

Condition Key(s) + Operator(s) + Value(s)
Match Custom:department == Engineering, Region In (us-west-1, us-east-1),

expression custom:Role NotIn (Management, ITStaff)
• Example 2: Cloud EPG ”Finance”
Condition Key(s) + Operator(s) + Value(s)
Match Custom:department == Finance, Region In (us-west-1, us-east-1),

expression IP == 172.16.0.0/24
Hub and Spoke Topology
Beside simple hub and spoke topology, to better understand
lets follow how packet is sent from spoke 1 to spoke 2:
• Spoke1 VM sends a packet to Spoke2
• In Azure, User Defined Route Table (UDR) on the Spoke1 VM
subnet, with the next-hop as the Network Virtual Appliance IP to
reach Spoke2 VM Subnet, so Azure will route the packet to the NVA
(Spoke and Hub VNets are peered)
• Because the Hub VNet is peered with both spokes, it’s able to reach
Spoke2 directly, so the NVA forward the packet the VM2 in Spoke2.
From Cloud ACI perspective, this simple hub and spoke design still
poses some challenge for example: scale of NVA, VRF information
retention (to extend ACI policy model to Azure). To overcome this,
we introduce Network Load Balancer in front of CSR for scale out of
CSRs and we use ACLs to identify the VRF from the source IP.
Cloud Infra - AWS
AWS Region
On-Premises
Infra VPC
CSR1kv CSR1kv
TGW TGW
Infra Tenant
User VPC-1 User VPC-2
SG(EPG)-1 SG(EPG)-2 SG(EPG)-3 SG(EPG)-4
User Tenant
Cloud Infra - Azure
Multisite Orchestrator
ACI DC
On-Premises Public Cloud
Infra VNET
CSR1kv CSR1kv
VM VM VM
VNET Peering
User VNET 1 User VNET 2
• Infra VPC/VNET will be used as hub
Infra VPC VPC to connect different Cloud ACI
sites, including on-premises ACI
Fabric and public cloud sites.
• Note:
ACI DC
On-Premises Public Cloud • From ACI 5.0 onwards, CSRs in Infra
VPC in AWS are used to connect
Infra VPC VPCs in regions within a site if the
AWS regions involved support Transit
AZ-1 AZ-2
gateway. For traffic between VPCs in
regions supporting transit gateway,
VM VM VM
Infra VPC CSR is not in the path of the
packet.
• You would need an AWS account
which will act similar to Fabric admin
• Need to have proper IP subnet
planned ahead of the deployment
• User VPC will be created by Cloud
User VPC APIC where all application policy will
be enforced
Multisite Orchestrator • Need an AWS account which will act
ACI DC as Tenant admin before creating
On-Premises Public Cloud user VPC
• IP subnets need to be unique within
Infra VPC a User VPC and across other VPCs
AZ-1 AZ-2 which are part of the same VRF and
Customer App between VPCs across VRFs which
VM VM VM
Region User VPC have communication with each other
• User VPC communicates with
another User VPC through TGW and
TGW inter-region peering. (Prior to
ACI 5.0, inter-VPC communication
goes through the Infra VPC
Infra VPC in Multi-Region
Multiple Hub & Spoke
ACI DC
Infra VPC Infra VPC
AZ-1 AZ-2 User VPC AZ-1 AZ-2 User VPC
VM VM VM
Region - 1 Region - 2
Infra VPC in Multi-Region
Single Hub / Multiple Spoke
ACI DC
Infra VPC
AZ-1 AZ-2 User VPC User VPC
VM VM VM Region - 1 Region - 2
User VPC User VPC
User VPC in Multiple Region
Multiple Hub & Multiple Spoke
ACI DC
Infra VPC Infra VPC
AZ-1 AZ-2 AZ-1 AZ-2

Customer App Customer App
Region - 1 User VPC Region - 2 User VPC

VM VM VM
Demo
Packet Walk
Instances within a VPC
AWS Cloud
AWS Region
Infra VPC
CSR1000V
• All traffic between instances
CSR1000V
within the same VPC, can
directly communicate to each
other based on the respective
security group policies
programmed by Cloud APIC
User VPC-1 User VPC-2
Epg-1 Epg-1 Epg-2 Epg-3 Security Group Availability Zone
CSR1000V End Point Group

Instance-1 Instance-2 Instance-3 Instance-4
SG-1 SG-1 SG-2 SG-3 Cloud APIC Transit Gateway
(TGW)
Packet Walk
-EC2 instances in Same EPG, Same Region
AWS Cloud
AWS Region
Infra VPC • Instance-1 &
Instance-2 are
CSR1000V CSR1000V part of same EPG
User VPC-1
Route table
Security group has a rule to allow
The
all traffic within route
Inbound
the same table
rule shows
in
security that the
the Destination
group (intra-EPG destination
Security group has
traffic) is local
allow all within
the same Security Group (Intra-
Instance-1 sends a packet EPGto traffic)
Instance-1 Traffic reaches Instance-2 Instance-2
Instance-2 in same EPG
Epg-1
(same Security Group) Epg-1
SG-1 SG-1
EPG1 is translated to Security

Group-1
Instances across VPC
AWS Cloud
• For instances in two different VPC’s
AWS Region
communicating to each other, the
Infra VPC
traffic has to exit the VPC either via
CSR1000V CSR1000V
VGW or via TGW of the user VPC
and reach CSR in infra VPC.
• Once the traffic reaches the
CSR1000v in infra VPC, packets are
routed to the destination based on
User VPC-1 User VPC-2 the configured policies
Availability
Epg-1 Epg-1 Epg-2 Epg-3 Security Group
Zone
CSR1000V End Point Group

Instance-1 Instance-2 Instance-3 Instance-4
SG-1 SG-1 SG-2 SG-3 Cloud APIC Transit Gateway
(TGW)
Instance-1 & Instance-2 are
Packet Walk part of two EPGs in two VRFs
Instances in two User VPCs EPGs are translated to

Security Group’s and attached
to Network interfaces
AWS Cloud Instance-1 sends a packet to
AWS Region Instance-2
Infra VPC Based on contract between EPG-

1 & EPG-2, the rules are
programmed on security groups
CSR1000V CSR1000V
The route table shows that the
destination is reachable via TGW
User VPC-1 User VPC-2 TGW Sends the packet to the

User VPC-2 via attachment
If User VPCs are in different

Route table region, packet goes through
TGW in local region and then to
Route table TGW in remote region (inter-
Instance-1 Instance-2 region TGW peering) then to
destination VPC
Epg-1
SG-1 Epg-2
SG-2 Traffic is permitted based on the
inbound rules of the security
group in the destination instance
Traffic reaches Instance-2

VM-1 & VM-2 are part of two
Packet Walk
EPGs in two VRFs
EPGs are translated to ASG /

Instances in two User VNETs NSG and attached to Network
interfaces
VM-1 sends a packet to VM-2

Region - 1
Based on contract between
Infra VNET EPG-1 & EPG-2, the rules are
programmed on NSG
CSR1kv CSR1kv The Route table is

programmed with UDR for
VM-2’s CIDR pointing to NLB
in the infra VNET
NLB sprays traffic to one of
VNET Peering the CSR
CSR checks for route entry in
VRF corresponding to User
VNET1 and if present,
forwards the traffic out of
one of its NICs, where route
Route Table Route Table is installed for User VNET2
VM-1 VM-2 due to VNET peering
Epg-1 User VNET 2 Epg-2

SG-2 Traffic is permitted based on
User VNET 1 SG-1
the inbound rules of the NSG
in the destination VM
Traffic reaches Instance-2

Instances in a VPCs and on-Premises
On-Premise Public Cloud
Site A Site B
Region 1
Infra VPC • For traffic from Instances in
VXLAN TUNNEL
(DATA PLANE) a VPC to on-premise,
traffic reaches CSR in Infra
BGP EVPN
Control Plane CSR1000v CSR1000v
VPC and over the VXLAN
tunnel to the ACI Spines
AZ-1 AZ-2
on-premise
• Spine forwards the traffic
User VPC - 1 User VPC -2
to the corresponding leaf
VM
EPG-1 EPG-1 EPG-2 EPG-3 on which the EP is located
EPG-1 Security Group (SG) Availability Zone (AZ)
CSR-1000V AWS Internet Gateway

(IGW)
SG-1 SG-1 SG-2 SG-3 AWS Virtual Private Gateway
Cloud APIC
Instance 01 Instance 02 Instance 03 Instance 04 (VGW)
Instance-1 & Instance-2 are
Packet Walk part of two EPGs in two VRFs
Instances in AWS to On-premise EPGs are translated to Security

Group’s and attached to
Multi-Site Orchestrator Public Cloud Network interfaces
On-Premise AWS Cloud
Instance-1 sends a packet to
AWS Region Site B Instance-2
Infra VPC Based on contract between EPG-

IPSec VPN Tunnel (Underlay) 1 & EPG-2, the rules are
programmed on security groups
CSR1000V CSR1000V
The route table shows that the

destination is reachable via TGW
User VPC-1 TGW Sends the packet to the

CSR via attachment
CSR sends packet via tunnel to

Route table on-Premises spine and spine
VM forwards to leaf
EPG-1 Instance-1
Traffic is permitted based on the
contract to EPG1’s (EP`) at the
Epg-1
SG-1 destination leaf
Traffic reaches the destination

end point (EPG-1)
Use Cases
Use Cases Summary
Network Segmentation Services Integration Operations and
Connectivity Visibility
• Automation of • Application • Cloud Native • Consistent Policy
Interconnect
• Micro-segmentation • Application Load • Topology View
• VPN Balancing
• Custom • Configuration Drift
• Transit Gateway Network Load
• Network •
• Custom Naming
VNET Peering Balancer
•
• Region, Zone, IP
Resources
Internet Gateway Address • Container Services
•
• Automation of North
Other options • PaaS bound APIs & Tools
• • Tenant
(Manually • 3rd Party L4-L7 Cloud Formation
supported) • Policy enforcement •
with Multi-Cloud • Palo Alto, Check • Azure Resource

• Direct Connect (DX) Point Manager
• Security Group
Express Route (ER) management Terraform
•
• F5, Citrix, •
ASAv/FTDv • Ansible
Network Connectivity
Virtual Private Network (VPN)
Site A Site B
AWS Cloud
IPSec VPN Tunnel (Underlay) Region
Shared Services VPC

BGP-EVPN Session (Control Plane) Use VPC 1
VXLAN Tunnel (Data Plane)
Customer
Premise User VPC 2
Internet
Router
AWS Transit Gateway
VM VM VM
• IPSec VPN connection between customer Premise Router before ACI fabric and CSR1kv over Internet.
• VXLAN data-plane connects ACI fabric and Cloud site
• BGP-EVPN routing reachability between ACI fabric and Cloud Site
Direct Connect (DX)
Cisco Multi-Site Orchestrator
On-Premises DC – Cisco ACI AWS Cloud

Region
Infra VPC
BGP EVPN (Overlay) Use VPC 1
IPSec VPN Tunnel (Underlay)
User VPC 2
Direct Connect (DX)
AWS Transit Gateway

DX Gateway
VM VM VM
• IPSec VPN connection between customer Premise Router before ACI fabric and CSR1kv over DX.
• VXLAN data-plane connects ACI fabric and Cloud site
• BGP-EVPN routing reachability between ACI fabric and Cloud Site
Cloud ACI: Network Connectivity In Public Cloud
Infra VNET (Hub)
Infra VPC
ASG NSG
CSR1kv CSR1kv CSR1kv CSR1kv
Shared Services &

NLB 3rd Party FW
Transit Gateway (TGW)

VNET Peering VNET Peering
User VPC 1 User VPC 2 User VNET 1 User VNET 2

UDR
0.0.0.0/0 -> FW IP
Transit Gateway (TGW) VNET Peering

Available in 5.0(1) Available in 5.0(2)
Automate Network Connectivity In The Cloud Across (1) AWS VPC (2) Azure VNET
#CiscoLive © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
AWS Direct Connect (DX)
AWS Cloud
East Corporate DC – Cisco ACI
US East1 US East2
1. Select Your Region AZ1 AZ1
2. Create a Connection CSR 1000v CSR 1000v
AZ2 AZ2
VGW VGW
3. Receive LOA-CFA ASR CSR 1000v CSR 1000v
4. Cross Connect IP DX DX GW
Network
US West2
5. Create Virtual Interface ASR
US West1
West Corporate DC – Cisco ACI AZ1 VGW AZ1 VGW

a. 802.1Q VLAN & eBGP Session
CSR1000v CSR1000v
b. Private VIF
AZ2 AZ2
I. DX Gateway
CSR1000v CSR1000v
II. Virtual Private Gateway
Shared Service VPC
6. Configure Customer Router
Segmentation
Application Stretch

• Stretch tenant/vrf across on-
APIC Cloud APIC premises and cloud sites
Tenant • During peak times easily deploy

VRF application tiers and resources in
BD1/Subnet CIDR 2 the cloud site
1Web-EPG1 Web-EPG2
• Consistent segmentation policy and

enforcement within and across on-
https https
premises and cloud sites
BD3/Subnet3 CIDR 4
App-EPG1 App-EPG2 • Application stack failover between
sites (active/disaster recovery)
Stretched EPG with Consistent Segmentation
APIC Cloud APIC

• Web Tier and App Tier are
Tenant stretched and securely segmented
VRF across on-premise and public cloud
BD/Subnet1 CIDR 2 sites
EPG - Web
• Consistent segmentation policy and

https, redis
enforcement for endpoints of
Web/App Tier are independent of
BD3/Subnet3 CIDR 4
location
EPG - App
Shared Services for Hybrid-Cloud

• Provides a capability to
APIC Cloud APIC deploy shared service
across hybrid cloud
Tenant 1 Route Tenant 2 Tenant 3
Leaking
VRF2 VRF3 • Shared Service
VRF1
CIDR 2 CIDR 4 deployed in 1 Site can
dns Web-EPG Web-EPG be consumed by
endpoints across other
BD/Subnet1
https https, redis
sites
DNS-EPG
CIDR 3 CIDR 5
• Contract will leak
App-EPG App-EPG subnet between VRFs
for reachability
Cloud and On-Premise L3Outs
Site B
• Cloud local L3out via

Region 1 Infra VPC IGW
L3out
CSR CSR
• On-Prem local L3out
Site A
AZ-1 AZ-2
• On-Prem site
IPSec Tunnel VGW VGW IPSec Tunnel endpoints cannot use
User VPC - 1 User VPC -2 Cloud L3out
EPG-1 EPG-1 EPG-2 EPG-3
IGW • Shared On-Prem L3out
L3out L3out for Cloud VPCs
IGW
SG-1 SG-1 SG-2 SG-3
Instance 01 Instance 02 Instance 03 Instance 04
Cloud First
• Cloud APIC only without on-premises

ACI or MSO
• Abstract AWS / Azure networking
constructs from user that is familiar
with ACI, delivering ACI-consistent
policy and operational model
• Deploy EPG and contracts on top of
AWS public cloud
Cloud First (Cont.)
• Cloud APIC only without on-
premises ACI or MSO
• Abstract AWS / Azure networking
• Deploy EPG and contracts on top
of AWS / Azure public cloud
Cloud First (Cont.)
• Cloud APIC only without on-
premises ACI
• Abstract AWS networking
• Deploy EPG and contracts on top of
AWS public cloud
Services Integration
L4-L7 Services Use Cases
Internet
Gateway
• Internet • Spoke to Spoke Availability Availability
App VPC
Availability Availability Zone 1 Zone 2
• Via Hub Service device Zone 1 Zone 2 • Local region
(FW)
App
App • Inter region
• Via local service
• Zoning, Shared TGW attachment to App VPC & WebVPC
• Patch updates Services or compliance
ALB 1 VIP requirement ALB
• Egress route table is subnet (per AZ) subnet
modified to include • Egress route table is
UDR / specific CIDR / FW UnTrust subnet modified in user VPC / FW Trust
Web VPC
subnet
subnet for internet 2 IP Address
VNET to get redirected
traffic via Hub or FW trust subnet to FW device fronted FW untrust
Shared VPC (NAT supported) by ALB / NLB subnet
ALB /NLB NLB / ILB

subnet subnet
Web Web
Services Deployment Matrix
Deployment Consumer EPG

Services Reachability
Type Deployment
Cloud Native (e.g. Intra Site

PaaS via Private IP (Private Link) (Consumer within the Cloud APIC
Storage) managed site)
PaaS Managed Intra Site, Inter Site

(PaaS service resides SQL, APIM, Databricks, via Private IP (Subnet IP (Consumer from another Cloud APIC
in customer VNET) AAD, SQL MI, CosmosDB space) site),
Internet via Firewall
Third-party services
SaaS via Private Links Intra Site
Operations and
Visibility
Cloud APIC Infra steps
• Deploy Cloud APIC using Cloud Formation (AWS) / ARM (Azure) Template
• Cloud APIC setup wizard (automated)
• AWS / Azure regions managed by Cloud APIC
• CSR1000v bring up and connectivity
• Tunnel creation
• Inter-site connectivity to on-premise / Cloud
• Inter-region connectivity
Cloud APIC – Bring Up
• Cloud APIC AMI / Image is available from AWS
and Azure marketplace
• Cloud Formation / ARM Template does:
• Launch Cloud APIC EC2 / VM Instance
• Create management and Infra Interface with IP
address from Infra VPC pool
• Assign elastic IP to the management interface to
enable communication with the Internet
• Create Internet Gateway on the Infra VPC / VNET
and setup the route table to point to Internet
Gateway
• Program security group rules on management
interface to allow https / ssh access from
configured external networks
Deployment Steps
• Multisite Orchestrator
• Site registration
• Configure Infra – BGP EVPN Session is up
• Create Tenant
• Create Schema
• Add Sites to Schema
• Site local properties
• Deploy the Schema
Operations
• We have covered multiple aspects of operation lifecycle:
• Dashboard
• Topology View
• Tag / Filter-based search
• Custom Resource Naming
• Configuration Drifts
• Visore object browser
• Firmware Management
• Statistics & Event Analytics
• Active Sessions and Tech Support
• Backup & Restore
• Remote Locations
Topology View
• Show logical topology of

different Cloud constructs
• Display options for these:
• Availability Zone
• VPC / VNET
• Routers
• Security Groups
• Tenant
• EPG
Tag / Filter-based search
• Tag / Filter attribute search can be used at
multiple levels e.g. Tenant / EPG / EP /
Event Analytics.
• We allow both single or multiple search
strings conditions in Cloud APIC
Custom Resource Naming
• Default naming policy need to be defined before any
Example of resource naming at the setup:
resource is deployed by Cloud APIC
• Virtual Network: vnet_${tenant}_${ctx}
• Resource Group: rg_${tenant}_${ctx}_${region}
• Subnet: snet_${subnet}
• Application Security Group: asg_${epg}_${app}
• Network Security Group: nsg_${epg}_${app}
• NSG Rule: ${priority}
The ${keyword} part are special variable used to

automatically add the respective name of a related
Mo. For example if your tenant is “T1” and your VRF
is “VRF1”, your VNET name would be
“vnet_T1_VRF1” using the default naming policy
Configuration Drifts
• Admin deletes one outbound
rule using Cloud provider
console (by mistake)
• Drift pinpoints the contract and
provides details which
provider and consumers are
affected
• When the change happened
and what severity it belongs to
Visore – Web Base MO Query and Browser Tool
https://<IP address>/visore.html
APIC Management Information Model Reference
https://<IP address>/doc/html
Policy based upgrade
• Similar steps as APIC
• Under Firmware
Management select
image location
• Schedule a time to
upgrade
• Once done, it will show
upgrade got completed
Tech Support
• We will collect
the following:
• CSR1kv
• Logs
• Cloud APIC
• Configuration
• Logs
• Core Files
Statistics
• We will show multiple

statistics:
• Inter-site
• Inter-region
• Inter-VPC
• Cloud EPG / Endpoints
• AWS TGW Stats
• Cloud Routers
Case Study
Cloud First
Azure Only
Region - WestUS
Resource Group / Subscription # 2
Resource Group / Subscription # 1 VNET
Peering
UDR
VM NSG
CSR1000v CSR1000v 0.0.0.0/0 -> FW IP

NLB
Cloud ACI Subnet
VM
Subnet - 1 ASG
Virtual Network – 1 (Spoke)

NSG VM VM NSG
Shared Services Subnet Common App Subnet

Resource Group / Subscription # 3
VNET
Peering VM NSG
NSG
ER Gateway
Gateway Subnet Firewall Subnet NLB
VM
Infra Virtual Network (Hub) Subnet - 2 ASG
Virtual Network – 2 (Spoke)
Internet
Hybrid Cloud
Infra Ready
Cisco Multisite Orchestrator
Global VNET Peering

Region - WestUS Region - EastUS
NSG NSG VM
VM
CSR1000v CSR1000v CSR1000v CSR1000v
er T
NLB NLB
in g
Pe NE
VN erin
VM
Pe VM
V
ET g
ASG ASG
VN erin
er T
g
Pe NE
Pe
ER GW NSG NSG ER GW
in
ET g
V
Palo Alto FW Gateway Gateway Palo Alto FW

VM NSG NSG VM
Infra Virtual Network (Hub) Infra Virtual Network (Hub)
VM VM
ASG ASG
Express Express
Internet
Route Route
CNF CNF
Branch On-Prem
Hybrid Cloud
Cloud Native and PaaS Services
Cisco Multisite Orchestrator
Global VNET Peering
Region - WestUS RG # 2
Virtual Network – 2 (Spoke) Subscription # 1 Infra Virtual Network (Hub) Infra Virtual Network (Hub) Subscription # 1 Virtual Network – 2 (Spoke)
RG # 2 Region - EastUS
Service Tag Service Tag
Outbound CSR1kv CSR1kv
Outbound
NSG CSR1kv CSR1kv NLB VM NSG
NSG Rule VM Cloud ACI Subnet
NLB Cloud ACI Subnet NSG Rule
VM APIM VM
Public PaaS ASG APIM ASG
Public PaaS
Services Services
VNET VNET
Peering Peering
Subscription# 1 Virtual Network – 1 (Spoke) Virtual Network – 1 (Spoke) Subscription# 1
NSG VM VM NSG
Subne Subne
Private PaaS Private
t-3 VM VM
Private
t-3 Private PaaS
Endpoint ASG Endpoint
Services ASG
Services
CNF CNF
Express Express
Route Internet Route
Branch On-Premise
References
References
• Cisco ACI
• Cisco Cloud ACI
• Cisco Cloud APIC on AWS Marketplace
• Cisco Cloud APIC on Azure Marketplace
• Cisco Cloud ACI on AWS Whitepaper
• Cisco Cloud ACI on Microsoft Azure Whitepaper
Thank you
#CiscoLive
#CiscoLive

DGTL-BRKACI-2690-How To Extend Your ACI Fabric To Public Cloud

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

DGTL-BRKACI-2690-How To Extend Your ACI Fabric To Public Cloud

Uploaded by

Copyright:

Available Formats

#CiscoLive

How to extend your ACI Fabric

Cloud Site(s) On-Premises Cloud Site(s)

Build Multi-Cloud Connections ACI Cloud

Deploy cross-cloud policies Endpoint

Cloud APIs Cloud programming layer

Network and Cloud Native

Any Routed IP Network

Site 1 Site 2 Site N

• Modular Spine with EX/FX line card to

• Subnet – Range of IP addresses. Each subnet must reside BD

Network ACL Network ACL

• Security Group – Act as a firewall for associated EC2 instance

AWS Only – External Connectivity

• NAT Gateway – Acts like an ECMP route to a set of NAT devices

Unified Cloud Native Networks - Mapping

ACI AWS Azure GCP

Building an automated and Maintain consistent policy, Requires a single pane of

Consistent Policy Enforcement on- Automated Inter-connect Simplified Operations with

• Interconnect connectivity automation (BGP-

Policy Distributor (PD) • Automates and manages cloud routers lifecycle

• Translates ACI Policy to cloud native Constructs

• m5.2xlarge EC2 Instance type • Standard D8s v3 VM type

• Cloud Ext EPG:

• An endpoint can be classified into multiple Cloud EPGs

• Below are few examples. You can use any combination

Key Operator Value

IP Address / Subnet =, != 10.10.10.1, 10.10.10.0/24

Region In, Not in us-west-1, us-east-1

Zone In, Not in us-west-1a, us-west-1b

• Example 1: Cloud EPG ”Dev”

Match Custom:department == Engineering, Region In (us-west-1, us-east-1),

• Example 2: Cloud EPG ”Finance”

Condition Key(s) + Operator(s) + Value(s)

Match Custom:department == Finance, Region In (us-west-1, us-east-1),

User VPC-1 User VPC-2

SG(EPG)-1 SG(EPG)-2 SG(EPG)-3 SG(EPG)-4

User VNET 1 User VNET 2

On-Premises Public Cloud

Infra VPC Infra VPC

AZ-1 AZ-2 User VPC AZ-1 AZ-2 User VPC

AZ-1 AZ-2 User VPC User VPC

User VPC User VPC

Infra VPC Infra VPC

AZ-1 AZ-2 AZ-1 AZ-2

Region - 1 User VPC Region - 2 User VPC

Epg-1 Epg-1 Epg-2 Epg-3 Security Group Availability Zone

CSR1000V End Point Group

EPG1 is translated to Security

CSR1000V End Point Group

Packet Walk part of two EPGs in two VRFs

Instances in two User VPCs EPGs are translated to

Infra VPC Based on contract between EPG-

User VPC-1 User VPC-2 TGW Sends the packet to the

If User VPCs are in different

Traffic reaches Instance-2

EPGs are translated to ASG /

VM-1 sends a packet to VM-2

CSR1kv CSR1kv The Route table is

Epg-1 User VNET 2 Epg-2

Traffic reaches Instance-2

CSR-1000V AWS Internet Gateway

Packet Walk part of two EPGs in two VRFs