ACI-3456 Openstack

BRKACI-3456
Mastering ACI and

OpenStack
Domenico Dastoli
Technical Marketing Engineer – DCNBU
Agenda
• ACI and OpenStack, why?
• ACI Unified Plugin for Openstack:
• Installations
• Components
• Network Benefits
• Operate OpenStack:
• Create Tenant Networks
• External connectivity
• OS to other workloads
• VNFs
BRKACI-3456 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
ACI and OpenStack
OpenStack
• What’s OpenStack?
• “OpenStack is a Cloud Operating System, that takes resources such as compute, storage, network,
virtualization technologies and controls those resources at a data center level.”
• Who's building OpenStack?
• Originally founded by Rackspace Hosting and NASA in July 2010
• Adopted by Ubuntu in 2011
• Now driven by OpenStack Foundation (established in September, 2012)
• Who’s backing OpenStack?

• More than 200 companies including Cisco - 87000+ people from 180 countries
• Who’s using OpenStack
• Enterprise, service providers, SMBs, researchers…
• In private public and Telco cloud
OpenStack Component overview: Neutron
OpenStack Neutron Challenges
Non distributed No underlay

L3 services visibility
Complexity of
Performance troubleshooting
Why Cisco ACI and OpenStack?
Distributed, Scalable Hardware-Accelerated
Virtual Networking Performance
• Full Neutron Node datapath replace

• Fully distributed Layer 2, anycast
gateway, DHCP, and metadata
• Distributed NAT and floating
IP address
Integrated Overlay and Operations and

Underlay Telemetry
• Automatic VXLAN tunnels at top of rack

• No wasted CPU cycles for tunneling
• Optional use of SRIOV, OVS DPDK (VPP
support roadmap)
• VNFs automated BGP peering SVI with
64-way line-rate Load Balancing

Underlay Telemetry

Underlay Telemetry
• Fully managed underlay network

through Cisco ® APIC
• Capability to connect physical servers
and multiple hypervisors to overlay
networks

Underlay Telemetry
• Troubleshooting across physical and

virtual environments
• Health scores and capacity planning per
tenant network
ACI Unified Plugin for
OpenStack
ACI Unified Plugin for OpenStack
• Full Policy Based Network Automation Extended to the Hypervisor
Key Benefits
Unified networking: Containers, VMs, and
bare-metal
VM distributed networking
EXT
Secure multi-tenancy
Visibility: Live statistics in APIC per VM

OpFlex OVS OpFlex OVS and health metrics
Node Node
VIM
ACI Unified plugin is a Neutron ML2 Plugin
• ML2 plugin: a framework allowing
OpenStack Networking to utilize the
variety of layer 2 networking Neutron Server
technologies.
• When running the ACI integration, The
following Type and Mechanism Drivers
will be used: ML2 Plug-in
API Extensions
• Type Drivers
• opflex Type Manager Mechanism Manager
• Mechanism Drivers
Linux Bridge
TypeDriver
TypeDriver
TypeDriver
TypeDriver
Population
apic_aim
Microsoft
apic_aim
SR-IOV
Hyper-V
vSwitch
Layer 2
OpFlex
•
VXLAN
Nexus
Cisco
Cisco
VLAN
Open
GRE
ACI Unified Plugin options
• Opflex mode allows creation of neutron networks based on
• VLAN
• VXLAN
• APIC AIM Mechanism driver enables the user to operate OpenStack

projects in:
• Neutron standard ML2 mode
• Group Based Policy (GBP) mode
Installation of
OpenStack
Installation of OpenStack and ACI Plugin
• On Cisco.com:
• https://www.cisco.com/c/en/us/support/cloud-systems-management/application-
policy-infrastructure-controller-apic/tsd-products-support-series-
home.html#OpenStack_Installation_Guides
• Manual installation:
• Prone to errors and discouraged. We limit the support for production environments
and documentation available on github.
• RHEL OSP Director, Juju Charms and Cisco VIM: full support for automated
installation and upgrade
VIM
Scalability – check out Release Notes for update!
Limit Type Maximum
• Limits keep changing and
can be found in Release
Host / Leaf 40
Notes:
https://www.cisco.com/c/e
n/us/support/cloud- VPC Links / Leaf 40
systems-
management/application-
policy-infrastructure- Endpoints / Leaf 2000
controller-apic/tsd-
products-support-series-
Endpoints / Host 400
home.html#Cisco_APIC_Op
enStack_and_Container_Pl
ugins_Release_Notes Virtual Endpoints / Leaf 40000
MultiPod and OpenStack
• It is possible to seamlessly extend the OpenStack cluster across different
data centers, both increasing redundancy and allowing disaster recovery
scenarios.
Multiple OpenStack Cluster on same ACI fabric
EXT
• You can have multiple

OpenStack Clusters on
the same ACI fabric, i.e.
Production and Testing
APP DB APP DB
etc.
APP DB APP DB
OS Cluster-01 OS Cluster-02
What are the
components and how
do they work?
ACI Neutron Plugin Main Components
ML2
Plugin Neutron-server
AIM
Controller node
VMM
and
APIC Model
Main Components are:
• ACI ML2 Plugin
• AIM
• Opflex Agent (Optional) OpFlex
• OpFlex Proxy Proxy
• VMM Manager
VLAN or
VXLAN
L2/L3 Bond0
vlan/vXlan OpFlex-agent
br-fabric
br-int
ta p483a c b1f ta p483a c b1f
vm1 vm2 vm3 vm4
ACI Integration Module
• The AIM daemon is running on the
Controller nodes and is responsible to
configure ACI through REST API call
AIM Daemon based on the OpenStack policy model
defined.
• It continuously monitors ACI resources
created from OpenStack and reverts
back any changes if not aligned with
AIM DB, part of the OpenStack DB.
Containerised AIM for Red Hat OSP
• As from In OSP12 and OSP13 AIM runs as a container running in host
mode:
[root@overcloud-controller-0 heat-admin]# docker ps | grep aim
1595784748d5 10.10.250.69:8787/rhosp13/openstack-ciscoaci-aim:latest
"kolla_start" 8 days ago Up About an hour (healthy) ciscoaci_aim
[root@overcloud-controller-0 heat-admin]#
[root@overcloud-controller-0 heat-admin]# docker inspect 1595784748d5|egrep "NetworkMode|PidMode"
"NetworkMode": "host",
"PidMode": "host",
[root@overcloud-controller-0 heat-admin]# docker inspect --format '{{ .State.Pid }}' 1595784748d5
8788
[root@overcloud-controller-0 heat-admin]# ps -ef |grep 8788
root 8788 8582 0 13:43 ? 00:00:03 /usr/bin/python /bin/supervisord -c
/etc/aim/aim_supervisord.conf
root 22006 8788 0 13:45 ? 00:00:01 /usr/bin/python2 /usr/bin/aim-event-service-polling --
config-file=/etc/aim/aim.conf --log-file=/var/log/aim/aim-event-service-polling.log
root 22007 8788 3 13:45 ? 00:03:55 /usr/bin/python2 /usr/bin/aim-aid --config-
file=/etc/aim/aim.conf --log-file=/var/log/aim/aim-aid.log
root 22059 8788 0 13:45 ? 00:00:02 /usr/bin/python2 /usr/bin/aim-event-service-rpc --config-
file=/etc/aim/aim.conf --log-file=/var/log/aim/aim-event-service-rpc.log
root 736371 708659 0 15:42 pts/0 00:00:00 grep --color=auto 8788
ACI Neutron Plugin Operation Workflow
Application Network Profile
EPG EPG EPG DB
3 C1 WEB C2 APP C3
Create Application Policy
ACI
5
Fabric
2 Push Policy
Automatically Push What if you want to modify

Network Profiles to
APIC and keep it
ACI values of objects
sync created by OpenStack?
AIMCTL CLI
Create Network, Subnet,
Security Groups, Policy
NETWORK ROUTING SECURITY
AIM
1
OPEN VIRTUAL SWITCH OPEN VIRTUAL SWITCH OPEN VIRTUAL SWITCH
NEUTRON NOVA
4 Web App Web App DB App DB
OpenStack Admin
Instantiate VMs
HYPERVISOR HYPERVISOR HYPERVISOR
AIM Controller CLI
• AIM Controller CLI
allows to:
• Query for existing ACI [heat-admin@overcloud-controller-0 ~]$ aimctl manager bridge-domain-find
objects +--------------------------------------+------------------------------------------+
| tenant_name | name |
• Query for existing ACI |--------------------------------------+------------------------------------------|
objects details | prj_b75263c9e55c48cab43b2edaf59e5d9a | net_63bac97c-eb1f-4b9b-a8da-9a0c478789bb |
• Update ACI objects | common | openshift-39_bd_kubernetes-service |
values | prj_b75263c9e55c48cab43b2edaf59e5d9a | net_8f2d85b6-89c5-4e93-8339-267e52a01ac9 |
• Create ACI objects | common | BD-T1-SITE2 |
| common | BD-T1-SITE1 |
| common | default |
+--------------------------------------+------------------------------------------+
[heat-admin@overcloud-controller-0 ~]$ aimctl manager bridge-domain-show
prj_b75263c9e55c48cab43b2edaf59e5d9a net_8f2d85b6-89c5-4e93-8339-267e52a01ac9
AIM Controller CLI

+---------------------------+--------------------------------------------------+
| Property | Value |
|---------------------------+--------------------------------------------------|
AIM Controller CLI

| tenant_name | prj_b75263c9e55c48cab43b2edaf59e5d9a |
• | name | net_8f2d85b6-89c5-4e93-8339-267e52a01ac9 |
allows to: | enable_arp_flood | True |
[heat-admin@overcloud-controller-0 ~]$ aimctl manager bridge-domain-find
• Query for existing ACI | display_name | serverNET |
+--------------------------------------+------------------------------------------+
objects | limit_ip_learn_to_subnets | True
| tenant_name | name
|
|
• Query for existing ACI | enable_routing | True |

|--------------------------------------+------------------------------------------|
objects details | ip_learning | True |

| prj_b75263c9e55c48cab43b2edaf59e5d9a | net_63bac97c-eb1f-4b9b-a8da-9a0c478789bb |
| l2_unknown_unicast_mode | proxy |
Update ACI objects
| common | openshift-39_bd_kubernetes-service |
• | vrf_name | DefaultVRF |
values
| prj_b75263c9e55c48cab43b2edaf59e5d9a | net_8f2d85b6-89c5-4e93-8339-267e52a01ac9 |
| ep_move_detect_mode | garp |
| common | BD-T1-SITE2 |
• Create ACI objects | monitored
| common
| False
| BD-T1-SITE1
|
|
| l3out_names | [] |
| common | default |
| epoch | 2 |
+--------------------------------------+------------------------------------------+
| dn | uni/tn-prj_b75263c9e55c48cab43b2edaf59e5d9a/BD-ne|
| | t_8f2d85b6-89c5-4e93-8339-267e52a01ac9 |
+---------------------------+--------------------------------------------------+
[heat-admin@overcloud-controller-0 ~]$ aimctl manager bridge-domain-update
AIM Controller CLI

prj_b75263c9e55c48cab43b2edaf59e5d9a net_8f2d85b6-89c5-4e93-8339-267e52a01ac9 --
limit_ip_learn_to_subnets False
+---------------------------+--------------------------------------------------+
| Property | Value |
AIM Controller CLI

|---------------------------+--------------------------------------------------|
• | tenant_name | prj_b75263c9e55c48cab43b2edaf59e5d9a |
allows to: | name | net_8f2d85b6-89c5-4e93-8339-267e52a01ac9 |
• Query for existing ACI | enable_arp_flood | True |
objects | display_name | serverNET |
| limit_ip_learn_to_subnets | False |
• Query for existing ACI | enable_routing | True |
objects details | ip_learning | True |
• Update ACI objects | l2_unknown_unicast_mode | proxy |
values | vrf_name | DefaultVRF |
| ep_move_detect_mode | garp |
• Create ACI objects | monitored | False |
| l3out_names | [] |
| epoch | 3 |
| dn | uni/tn-prj_b75263c9e55c48cab43b2edaf59e5d9a/BD-ne|
| | t_8f2d85b6-89c5-4e93-8339-267e52a01ac9 |
+---------------------------+--------------------------------------------------+
AIM Controller CLI
• AIM Controller CLI
allows to:
• Query for existing ACI
objects
• Query for existing ACI
objects details
• Update ACI objects
values
• Create ACI objects
Neutron Opflex Agent
• The Neutron Opflex Agent runs on both
the compute and the controller. It is
responsible to communicate with the
Neutron Opflex and neutron server.
OpFlex agents OpFlex Agent (old name Agent OVS)
(optional) • The OpFlex Agent runs on the compute

and controller nodes. It is responsible to
communicate with the OVS and the leaf
node to register to ACI fabric.
Containerised Agents for Red Hat OSP
• As from In OSP12 and OSP13 Agents run as a container running in host
mode. Both agents run on the same container:
[root@overcloud-compute-0 heat-admin]# docker ps | grep opflex
53e4e43035d1 10.10.250.69:8787/rhosp13/openstack-ciscoaci-opflex:latest "kolla_start"
8 days ago Up 2 hours (healthy) ciscoaci_opflex_agent
[root@overcloud-compute-0 heat-admin]# docker inspect 53e4e43035d1|egrep "NetworkMode|PidMode"
"NetworkMode": "host",
"PidMode": "host",
[root@overcloud-compute-0 heat-admin]# docker inspect --format '{{ .State.Pid }}' 53e4e43035d1
5847
[root@overcloud-compute-0 heat-admin]# ps -ef |grep 5847
root 5847 5816 0 13:39 ? 00:00:03 /usr/bin/python /bin/supervisord -c /etc/opflex-agent-
ovs/opflex_supervisord.conf
root 6301 5847 0 13:39 ? 00:00:04 /usr/bin/opflex_agent --log /var/log/opflex/opflex-
agent.log -c /etc/opflex-agent-ovs/opflex-agent-ovs.conf -c /etc/opflex-agent-ovs/plugins.conf.d -c
/etc/opflex-agent-ovs/conf.d
root 6302 5847 0 13:39 ? 00:00:00 /usr/bin/mcast_daemon --log /var/log/opflex/mcast.log
42435 6303 5847 0 13:39 ? 00:00:29 /usr/bin/python2 /usr/bin/neutron-opflex-agent --config-
file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --log-file
/var/log/neutron/neutron-opflex-agent.log
root 59984 18603 0 15:48 pts/2 00:00:00 grep --color=auto 5847
[root@overcloud-compute-0 heat-admin]#
OpFlex Architecture
• Neutron-opFlex-
agent: Receives
updates from Neutron
about new endpoints
and updates EP and
Service files
OpFlex Proxy
• OpFlex-agent: Runs
OpFlex protocol with
OpFlex-agent
Neutron-
Neutron-Server
ACI leaf proxy and
OpFlex-agent
programs open
vswitch via OpenFlow
os-compute-01 os-controller-01
OpFlex Architecture
Neutron-opFlex-
•
agent: Receives
updates from Neutron
()[root@overcloud-compute-1 /]# more
/var/lib/opflex-agent-ovs/endpoints/041b5a38-
c38b-4a9b-8de3-
about new endpoints
da69872abb41_fa\:16\:3e\:0f\:99\:a7.ep
{
"dhcp4": {
and updates EP and
"ip-address-mapping": [
{
Service files
OpFlex Proxy
"mapped-ip": "192.168.1.29",
"policy-space-name":
"prj_e8b2df07409b4b998dece8c00dc374bb",
•
"floating-ip": "10.104.31.11",
"endpoint-group-name":
OpFlex-agent: Runs
"OpenStack|EXT-l3out1", OpFlex protocol with
}
OpFlex-agent
Neutron- ],
Neutron-Server
ACI leaf proxy and
"endpoint-group-name":
OpFlex-agent
"OpenStack|net_1a7a4bb6-eaec-4aba-8f22 programs open
vswitch via OpenFlow
}
os-compute-01 os-controller-01
ACI + OpenStack – With OpFlex
Architecture ML2
APIC Driver
converts neutron
AIM
Controller node to Cisco ACI
Policy
OpFlex for OVS

Cisco ACI fabric
provides line • Open Source OpFlex agent
rate distributed Opflex Proxy
routing and Receives extends ACI into the host
switching Policies from
capabilities APIC • OpFlex Proxy exposes new open
API in ACI fabric
OpFlex
Proxy
VLAN or
VXLAN
L2/L3
OpFlex Agent
Bond0 Receives
vlan/vXlan OpFlex-agent Policies from
OVS Rules ACI leaf
programmed by br-fabric
the OpFlex br-int
Agent are used ta p483a c b1f ta p483a c b1f
for policy
enforcement
vm1 vm2 vm3 vm4
What if you do not install OpFlex agents
on the compute node?
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
ACI + OpenStack – Without OpFlex Support
Architecture ML2
APIC Driver
converts neutron
AIM
Policy
PhysDom Integration
Cisco ACI fabric
provides line • In some scenarios Opflex agent is
rate distributed
routing and not installed, in which case we talk
switching about PhysDom integration
capabilities
• A typical use case for this is when
we have SR-IOV based compute
hosts not having OVS
VLAN
• In this model the EPG is configured
L2/L3 PhysDom and
with VLAN static binding to the
static binding
Bond0
vlan on the ACI leaf compute nodes.
switches
os-compute-01
IPtables are br-ex
programmed by br-int
neutron server
ta p483a c b1f ta p483a c b1f
vm1 vm2 vm3 vm4
Compute nodes: OpFlex vs non OpFlex
Distributed Distributed OVS rules

Distributed DHCP Metadata
Mode / Optimization Routing on Routing on or
NAT Optimization Optimization
host Leaf IPTables
OpFlex Yes Yes Yes Yes Yes OVS Rules
Non OpFlex/PhysDom No Yes No No No IPTables
Acceleration mode
OVS-DPDK
possible on the same OVS with OpFlex SR-IoV
no OpFlex
host
• You can mix and OVS with OpFlex Yes Yes No
match modes on the
same hosts SR-IoV Yes Yes Yes
OVS-DPDK no OpFlex No Yes Yes
ACI Unified Plugin Components - ACI Side
• ACI VMM Manager.
• Runs on the APIC cluster and communicates with the ACI Integration Module
• Receives configuration, learns Eps
• VMM object model for OpenStack represents OS nodes, Neutron resources, Nova
resources.
• ACI Leaf Opflex Proxy.
• Runs on the ACI leaf switches to scale out policy propagation to the connected
nodes.
• Transfers the OpFlex policies required by agent of the node.
ACI Tenant VMs view
ACI VMM View
All OS nodes
All
networks in
OS
All VMs running

on a OS node
More about OpFlex
mode architecture
optimizations
OpFlex and Policy Enforcement (OVS)
ML2
APIC Driver
converts neutron
AIM
Policy
br-fab_vxlan0
NAT, routing
Cisco ACI fabric and
provides line
br-fabric encapsulation
rate distributed Opflex Proxy qpi-2
qpi-1
routing and Receives
switching Policies from
capabilities APIC
qpf-1 qpf-2
OpFlex
Proxy Security
br-int enforcement
VLAN or
VXLAN tap1 tap2
L2/L3
OpFlex Agent
Bond0 Receives
vm1 vm2
vXlan OpFlex-agent Policies from
OVS Rules ACI leaf nginx-1
programmed by br-fabric
the OpFlex br-int
Agent are used ta p483a c b1f ta p483a c b1f
for policy
enforcement
vm1 vm2 vm3 vm4
Distributed Functions
Distributed Routing and Policy Enforcement
DESCRIPTION
• Traditionally in OpenStack the

routing is done on the servers
hosting neutron services only.
EPG net01 EPG net02 • With ACI integration the opflex-
agent is taking care of the routing
overcloud-compute-0 NO NEED FOR NEUTRON of the VMs. Since each compute
L3 AGENT
Compute
node node has a opflex-agent, the
routing is done in a distributed
Neutron-L3-Agent
OpFlex-agent
ta p483a c b1f
manner.
ta p483a c b1f
tap446aaf1g tap673e74ca tap483acb1f tap413e41cd
192.168.21.22
• Also, the opflex-agent performs
192.168.1.3 10.12.0.15
local policy enforcement through
192.168.1.20
vm1 vm2 vm3 vm4
OVS rules locally on the same
OvS will act as hypervisor where the instance
distributed virtual lives.
router
()[root@overcloud-compute-1 /]# ovs-ofctl -O OpenFlow13 dump-flows br-int | grep 192.168.
cookie=0x8, duration=2109.367s, table=1, n_packets=0, n_bytes=0, send_flow_rem
DESCRIPTION
priority=8064,ip,reg0=0x2,nw_src=192.168.22.3 actions=ct(commit,zone=NXM_NX_REG6[0..15]),goto_table:3
cookie=0x8, duration=2109.367s, table=1, n_packets=0, n_bytes=0, send_flow_rem • Traditionally in OpenStack the
priority=8064,ip,reg0=0x2,nw_src=192.168.22.23 hosting neutron services only.
L3 AGENT
Compute
Neutron-L3-Agent
OpFlex-agent
ta p483a c b1f
manner.
ta p483a c b1f
192.168.21.22
192.168.1.3 10.12.0.15
192.168.1.20
vm1 vm2 vm3 vm4
hypervisor where the instance
lives.
DESCRIPTION

L3 AGENT
Compute
Horizon
Neutron-L3-Agent View
OpFlex-agent
ta p483a c b1f
manner.
ta p483a c b1f
192.168.21.22
192.168.1.3 10.12.0.15
192.168.1.20
vm1 vm2 vm3 vm4
lives.
DESCRIPTION

L3 AGENT
Compute
Neutron-L3-Agent
OpFlex-agent
ta p483a c b1f
manner.
ta p483a c b1f
192.168.21.22
192.168.1.3 10.12.0.15
192.168.1.20
vm1 vm2 vm3 vm4
ACI View
lives.
NAT Function performed in the OVS locally
EXT
contract
DESCRIPTION
NAT network Border Leafs
Neutron network L3outExtEPG • Floating IP configured by
0.0.0.0/0 OpenStack Neutron using
NAT network
standard mechanism
opflex-
Neutron network • OVS performs NAT function
using OpenFlow rules from
OvS functions as distributed virtual 10.12.0.13 10.12.0.15 OpFlex agent for Floating IP
router for VMs. If destination vm1 vm2
network is external to OpenStack

router directly connected subnets os-bm-node-01
and NAT policy is defined, it will
source NAT the VM IP.
SNAT and FIP
Each compute node will be

assigned with one SNAT IP
address. VMs with no FIP will be
NATted with the specific SNAT
IP of the node hosted them
Floating IP will appear here with

the name of the VM where the
FIP has been assigned to
DHCP Optimization
DESCRIPTION
• Traditionally VMs are getting IP

from Neutron DHCP Server
OpFlex-agent will
retrieve VM IP • Agent-OVS learns info of the
information from VM from Endpoint Files
EP file and provide
• Agent-OVS responds to the
to the VM through DHCP Allocation and Options
VMs with DHCP responses
DHCP DORA
• DHCP allocation and options
Neutron-DHCP-Agent
Compute
node
passed back to Neutron server.
OpFlex-agent t
DNSmasq
a
p
4
8
3
a
c
b
1
f
10.11.0.21 10.11.0.33
vm1 vm2
os-network-01
Metadata Optimization
DESCRIPTION
• Traditionally in OS VMs get the

meatadata information from the
Metadata-agent service running on Neutron
will retrieve VM Server
Metadata
information from • Neutron metadata agent is
reading the Service File
nova-api and VM Metadata
provide to the VM • Metadata agent locally performs
proxy
Compute
node
Nova-API • Metadata agent updates the
Metadata
Metadata-agent t
a neutron server with VM
Service
p
4
8
Metadata
3
a
c
b
1
f
Nova-api 10.11.0.21 10.11.0.33

vm1 vm2
os-network-01
ML2 (Neutron) vs GBP
mode
ML2 (Neutron) – APIC Mapping
Openstack/Neutron
• With the ML2 APIC Object
Object
Standard Neutron
model, the following Project Tenant and Application Profile Name
mapping happens.
Network EPG and Bridge Domain
• All the operations
are done on Subnet Subnet
OpenStack through
Horizon, CLI or Heat Contract, consumed and provided by any
Router EPGs corresponding to the Neutron
Networks connected to the router.
Security Group Host Protection Profile (HPP) policies
GBP – APIC Mapping
Openstack/GBP Object APIC Object
• With the GBP Model
the following mapping Project Tenant and Application Profile Name
happens.
• GBP offers much more L3 Policy VRF
granularity and
flexibility compare to
L2 Policy Bridge Domain and associated Subnet
standard neutron.
• GBP comes with CLI,
Policy Group EPG
Heat and Horizon
plugins
Policy Ruleset Contract
GBP Policy RuleSets
L3 Policy
• GBP Policy RuleSets Policy Group provide Policy RuleSet Set consume Policy Group
represents the ACI
Policy concept. Policy Rule
Policy Target Policy Target
• ACI contracts, filters Policy Rule
and actions are
mapped with GBP Policy Target
Policy Rule
Policy Target
Rulesets
Policy Target Policy Target
L2 Policy Classifier Action L2 Policy
Provider EPG Consumer EPG
contract
ML2 vs GBP model – what is best?
• GBP:
• Application Centric
• Security groups are created as ACI contracts AND OVS rules. So they are visible
on ACI and will be enforced both in HW (ACI leaf) and SW (OVS).
• Introduces new REST APIs: if any existing templates, you will need to adapt
• ML2:
• Network Centric
• Standard way of creating neutron networks
• REST API will not change: any heat or CLI template will keep working
• Security Groups visible in ACI as HPP: they are implemented as OVS rules
Demo
BRKACI-3456
Topology and Steps
• Create Net01
• Create Net11
• Attach 2 VMs per network
• Create Router and a gateway for
external connectivity
• Connect the networks to the router
External Network
External Connectivity
• Connectivity for a tenant can be either shared or dedicated.
• A shared external network is visible by all OpenStack projects.
• A dedicated connectivity for the OpenStack project.
• It would be possible to have a mixed environment both with shared and
dedicated external connectivity.
Tenant Pasta&Co Tenant Pizza&Co Tenant Pasta&Co Tenant Pizza&Co
net1 net2 net3 net4 net1 net2 net3 net4
Dediacated Dediacated
Shared L3 out
L3 out L3 out
WWW WWW WWW
How to create the L3out on ACI
• Although the OpenStack plugin could create automatically an L3out on ACI,
the best practice is to create it manually
• Defining manually an L3out supports all the L3out features:
• VPC
• Dynamic routing protocols
• Route engineering
• Etc.
• The L3out can be created with XML templates or in any ways you are
familiar with.
• Once the L3out is available, ACI AIM plugin on OpenStack can import it and
start controlling the L3out.
External Network with
Distributed NAT (DNAT)
Creation of the L3out Dedicated
• A dedicated L3out must be created in the OpenStack created tenant.
• In the L3out creation, it must be specified:
• Interfaces and their IP information
• Dynamic routing if any
• External EPG
• You should NOT add any contract as they will be added later automatically
by the plugin.
• If you require SNAT or FIP, the L3 out must be defined in a different VRF
from the one created by OpenStack!
Create OpenStack External Network with DNAT
[stack@dom-undercloud ~]$ neutron net-create external-net-CL --router:external --apic:distinguished_names type=dict
ExternalNetwork=uni/tn-prj_97390b780c7545d393d9314d34e69cfa/out-externalNet/instP-extEpg
+----------------------------+----------------------------------------------------------------------------------------+
| Field | Value |
+----------------------------+----------------------------------------------------------------------------------------+
| admin_state_up | True |
| apic:distinguished_names | {"ExternalNetwork": "uni/tn-prj_97390b780c7545d393d9314d34e69cfa/out-externalNet |
| | /instP-extEpg", "BridgeDomain": "uni/tn-prj_97390b780c7545d393d9314d34e69cfa/BD-EXT- |
| | externalNet", "VRF": "uni/tn-prj_97390b780c7545d393d9314d34e69cfa/ctx-DefaultVRF", |
| | "EndpointGroup": "uni/tn-prj_97390b780c7545d393d9314d34e69cfa/ap-OpenStack/epg-EXT- |
| | externalNet"} |
| apic:nat_type | distributed |
| apic:synchronization_state | synced |
| id | f085fe67-42e1-4b3c-8951-e5d9932222ca |
| is_default | False |
| name | external-net-CL |
| port_security_enabled | True Creating neutron external |
| provider:network_type | opflex |
| provider:physical_network | physnet1 network bound to the |
| provider:segmentation_id
| revision_number
|
| 4
L3out imported with the |
|
| router:external
| shared
| True
| False
aimctl manager. |
|
| status | ACTIVE |
| subnets | |
+----------------------------+----------------------------------------------------------------------------------------+
External SNAT or Floating IP Pool Definition
[stack@dom-undercloud ~]$ neutron subnet-create external-net-CL 10.104.21.0/24 --name ext-subnet --disable-dhcp
--gateway 10.104.21.1 --apic:snat_host_pool True
Created a new subnet:
+----------------------------+--------------------------------------------------+
| Field | Value |
+----------------------------+--------------------------------------------------+
| allocation_pools | {"start": "10.104.21.2", "end": "10.104.21.254"} |
| apic:distinguished_names | {} |
| apic:snat_host_pool | True |
| apic:synchronization_state | N/A |
| cidr | 10.104.21.0/24 |
| dns_nameservers
| enable_dhcp
|
| False
|
|
Creating neutron
| gateway_ip | 10.104.21.1 | external network
| host_routes | |
| id | 5344832d-dd03-40d7-a4d2-3f04c86fbb9d | SNAT pool and
| ip_version | 4 |
| ipv6_address_mode | | attaching the router
| ipv6_ra_mode
| name
|
| ext-subnet
|
|
to the external net.
| network_id | f085fe67-42e1-4b3c-8951-e5d9932222ca |
| revision_number | 2 |
| service_types | |
| subnetpool_id | |
| tenant_id | 97390b780c7545d393d9314d34e69cfa |
+----------------------------+--------------------------------------------------+
[stack@dom-undercloud ~]$ openstack router set --external-gateway external-net-CL CLrouter
Using Floating IP
[stack@dom-undercloud ~]$ neutron subnet-create external-net-CL 10.104.31.0/24 --name ext-subnet-FIP --allocation-pool

start=10.104.31.10,end=10.104.31.100 --disable-dhcp --gateway 10.104.31.1
+----------------------------+---------------------------------------------------+
| Field | Value |
+----------------------------+---------------------------------------------------+
| allocation_pools | {"start": "10.104.31.10", "end": "10.104.31.100"} |
| apic:distinguished_names
| apic:snat_host_pool
| {}
| False
|
|
Creating floating IP is
| cidr
| enable_dhcp
| 10.104.31.0/24
| False
|
|
as simple as adding
| gateway_ip | 10.104.31.1 | another subnet to the
| host_routes | |
| id | d9bb7111-b668-4823-932d-68fa211aa69b | external network.
| ip_version | 4 |
| name | ext-subnet-FIP |
| network_id | b90bfad9-4ed3-477f-996a-4222ae0768dd |
| project_id | 11fa0c41388f4d3fbf3f2f6d6184f687 |
| service_types | |
| tenant_id | 11fa0c41388f4d3fbf3f2f6d6184f687 |
+----------------------------+---------------------------------------------------+
[stack@dom-undercloud ~]$
NAT Function performed in the OVS locally
EXT
contract
DESCRIPTION
NAT network Border Leafs
Neutron network L3outExtEPG • Floating IP configured by
0.0.0.0/0 OpenStack Neutron using
NAT network
standard mechanism
opflex-
Neutron network • OVS performs NAT function
using OpenFlow rules from
OvS functions as distributed virtual 10.12.0.13 10.12.0.15 OpFlex agent for Floating IP
router for VMs. If destination vm1 vm2
network is external to OpenStack

router directly connected subnets os-bm-node-01
and NAT policy is defined, it will
source NAT the VM IP.
Life of a packet – Floating IP
Life of a packet – FIP EXT
20.0.0.1
BD NAT external
Border Leafs
FIP 10.104.31.0/24
ExtEPG
br-fab_vxlan0
br-fabric
qpi-1 VM1 wants to
contact client
qpf-1 20.0.0.1.
br-int VM1 has FIP
tap1 10.104.31.11.
vm1 192.168.1.29
fa:16:3e:0f:99:a7
FIP 10.104.31.11 dom-node-01
20.0.0.1
()[root@overcloud-compute-1 /]# more /var/lib/opflex-agent-

ovs/endpoints/041b5a38-c38b-4a9b-8de3-
da69872abb41_fa\:16\:3e\:0f\:99\:a7.ep
BD NAT external {
"dhcp4": { Border Leafs
FIP 10.104.31.0/24 "ip-address-mapping": [
{
"mapped-ip": "192.168.1.29",
br-fab_vxlan0 "policy-space-name":
"prj_e8b2df07409b4b998dece8c00dc374bb",
"floating-ip": "10.104.31.11",
br-fabric "endpoint-group-name": "OpenStack|EXT-l3out1",
qpi-1 }
],
"endpoint-group-name": "OpenStack|net_1a7a4bb6-eaec-4aba-
8f22-e09c9ee696d2",
qpf-1
}
br-int
tap1
vm1 192.168.1.29
fa:16:3e:0f:99:a7
FIP 10.104.31.11 dom-node-01
20.0.0.1
BD NAT external
Border Leafs
FIP 10.104.31.0/24
ExtEPG
br-fab_vxlan0
br-fabric
qpi-1
qpf-1
1
DST_MAC SRC_MAC DST_IP SRC_IP DST_PORT SRC_PORT
br-int 00:22:bd:f8:19:ff fa:16:3e:0f:99:a7 20.0.0.1 192.168.1.29 80 40183
tap1
vm1 192.168.1.29
fa:16:3e:0f:99:a7
FIP 10.104.31.11 dom-node-01
20.0.0.1
DST_MAC SRC_MAC DST_IP SRC_IP DST_PORT SRC_PORT
00:22:bd:f8:19:ff fa:16:3e:ab:ff:0d 20.0.0.1 10.104.21.13 80 40183
BD NAT external 2 Border Leafs

FIP 10.104.31.0/24
ExtEPG
br-fab_vxlan0
()[root@overcloud-compute-1 /]# ovs-ofctl -O Openflow13 dump-flows br-
br-fabric fabric |egrep “10.104.31.11|set”
cookie=0x0, duration=772.165s, table=10, n_packets=1, n_bytes=98,
qpi-1 priority=10,ip,reg6=0x1,reg7=0x7a8004,metadata=0x2/0xff,nw_src=192.168.1.29
actions=set_field:fa:16:3e:0f:99:a7->eth_src,set_field:00:22:bd:f8:19:ff-
>eth_dst,set_field:10.104.31.11->ip_src,dec_ttl,load:0x7a8004-
>NXM_NX_REG0[],load:0x2->NXM_NX_REG4[],load:0x3->NXM_NX_REG5[],load:0x2-
qpf-1 >NXM_NX_REG6[],load:0->NXM_NX_REG7[],load:0x400-
>OXM_OF_METADATA[],resubmit(,3)
br-int
tap1
OVS will encapsulate the
vm1 192.168.1.29 packet with the NAT
fa:16:3e:0f:99:a7
FIP 10.104.31.11 dom-node-01
external EPG encap
The External network in ACI
3. The NATted IP
in ACI is
represented by
the ‘external’
EPG 2. OVS applies
NAT rules
4. Traffic is sent
to external router 1. VM traffic
through ACI reaches OVS
user
tenant
The External Network EPG will be created in the tenant itself.

A contract to allow connectivity between the EPG and the L3out
will be created automatically.
What if the L3out should
be shared by multiple
OpenStack projects?
Create L3 out on ACI – Shared
• The shared external network must
be defined in the Common tenant in
ACI
• You must define L3out and extEPG
The External network in ACI
common
tenant
3. The NATted IP
in ACI is
represented by
the ‘external’
EPG 2. OVS applies
NAT rules
4. Traffic is sent
to external router 1. VM traffic
through ACI reaches OVS
user
tenant
The External Network EPG will be created in the tenant itself.

A contract to allow connectivity between the EPG and the L3out
will be created automatically.
Dedicated Tenant
External Network no NAT
Creation of the L3out Dedicated
• A dedicated L3out must be created in the OpenStack created tenant.
• In the L3out creation, it must be specified:
• Interfaces and their IP information
• Dynamic routing if any
• External EPG
• You should NOT add any contract as they will be added later automatically
by the plugin.
• When disabling Distributed NAT, L3 out must be defined in the VRF created
by OpenStack, i.e. DefaultVRF!
Create OpenStack External Network
[stack@dom-undercloud ~]$ neutron net-create external-net-CL --router:external --apic:distinguished_names type=dict
ExternalNetwork=uni/tn-prj_97390b780c7545d393d9314d34e69cfa/out-externalNet/instP-extEpg --apic:nat_type ""
+----------------------------+----------------------------------------------------------------------------------------+
| Field | Value |
+----------------------------+----------------------------------------------------------------------------------------+
| admin_state_up | True |
| apic:distinguished_names | {"ExternalNetwork": "uni/tn-prj_97390b780c7545d393d9314d34e69cfa/out-externalNet |
| | /instP-extEpg", "BridgeDomain": "uni/tn-prj_97390b780c7545d393d9314d34e69cfa/BD-EXT- |
| | externalNet", "VRF": "uni/tn-prj_97390b780c7545d393d9314d34e69cfa/ctx-DefaultVRF", |
| | "EndpointGroup": "uni/tn-prj_97390b780c7545d393d9314d34e69cfa/ap-OpenStack/epg-EXT- |
| | externalNet"} |
| apic:nat_type | |
| apic:synchronization_state | synced |
| id | f085fe67-42e1-4b3c-8951-e5d9932222ca |
| is_default | False |
| name | external-net-CL |
| port_security_enabled
| provider:network_type
| True
| opflex
Creating neutron external |
|
| provider:physical_network | physnet1 network bound to the |
| provider:segmentation_id | |
| revision_number | 4 L3out. |
| router:external | True |
| shared | False |
| status | ACTIVE |
| subnets | |
+----------------------------+----------------------------------------------------------------------------------------+
[stack@dom-undercloud ~]$ openstack router set --external-gateway external-net-CL CLrouter
Connect BM or VMs to
OpenStack
OpenStack and other workload types
• AIM is only responsible to sync configuration pushed from OS.
• Any other configuration which does not change the direct status of OS
created objects could be done by an ACI Admin.
• The intention would be to allow BM or other VM types to be instantiated in
the same subnet created by OpenStack.
• Those VMs/BMs should be allowed to talk to OS VMs.
How to do this?
• Recommendation is:
• Create separate EPG from OS EPG
• Attach the BM/VM EPG to same BD as
OS
• Make sure DHCP in OS is not
overlapping IPs
• Add manually a contract between OS
EPG and BM/VM EPG
APIC administrator adds
Consumer EPG
Provider Contract
Filters
DB EGP and contract
EPG Name between net01 EPG and
net01 Database
db-access TCP/3309 DB EPG.
(OpenStack) (Bare Metal)
Fabric Admin
net01-bd 10.12.1.1/24
EPG database EPG net01
10.12.1.100
OpenStack administrator
creates net01 EPG and BD
OpenStack
10.12.0.21 10.12.0.22 10.12.1.15 10.12.1.51
Admin
VM1 VM2 VM3 VM3
os-node-01 os-node-02
neutron net-create net01

openstack subnet create --network net01 --gateway 10.12.1.1 --subnet-range 10.12.1.0/24 subnet01
openstack router create router01
neutron router-interface-add router01 subnet01
ACI Plugin and VNFs
Challenges of VNFs
• Configuration of dynamic routing
protocol between fabric switch and
VNF
• Support for dynamic VNF deployed in
distributed fashion
• Traffic distribution among VNF
• Wider ECMP than normally seen with
physical appliance VNFs VNFs
• Ensure evenly traffic distribution among

VNF Rack-1 Rack-2
Protocol peering
• Optimal performance with VNF
What is Neutron SVI feature
• ACI plugin for OpenStack enables distributed route peering between the
switches and OpenStack VNFs:
• Based on the creation or destruction of VNFs, Neutron SVI feature dynamically and
automatically create and destroy SVI on the underlay and enables line rate routing
capabilities and up to 64-way ECMP to the VNFs.
ACI L3out
• Up to 6 pairs of switches under same L3out
• VNFs across distributed sites (MultiPod)
• Supports bonding with VPC (L2 segment extended via ACI fabric)
• Support BFD for fast VM failure detection
VNF-1 VNF-2 VNF-3 VNF-4

•
Neutron SVI
neutron net-create LBSVI --provider:network_type vlan --provider:physical_network
physnet1 --apic:svi True --apic:bgp_enable True --apic:bgp_asn 2010
openstack subnet create --ip-version 4 --subnet-range 172.168.0.0/24 --gateway

172.168.0.1 --network LBSVI LBSUBNET --no-dhcp
openstack port create apic-svi-port:node-101 --network LBSVI --device-owner apic:svi

--fixed-ip subnet=LBSUBNET,ip-address=172.168.0.11
openstack port create apic-svi-port:node-102 --network LBSVI --device-owner apic:svi
--fixed-ip subnet=LBSUBNET,ip-address=172.168.0.12
Creates the L3out on ACI
openstack port create LB1PORT --network LBSVI --fixed-ip subnet=LBSUBNET,ip-
address=172.168.0.21 with BGP enabled
LB1=$(openstack port list | awk '/LB1/ {print $2}')
nova boot --flavor m1.tiny --image LB1 --nic port-id=$LB1 vLB1
Defines network for the

leaf and the IP address of
each physical leaf
Allows to attach VMs to the
L3out network. ACI will
dynamically BGP peer with
those
Distribute traffic evenly to VNFs
Each VNF within the cluster receives equal amount of data traffic, regardless
where it is attached
Each border leaf evenly
distribute traffic flows to each Ingress leaf evenly distribute
VNF instance. traffic flows to each border
Traffic destining to some leaf where VNFs are
VNFs will be forwarded via attached
fabric
.1 .2 .3 .4
VNID1
.11 .12 .13
VNFs 100.1.1.0/24 VNFs
Rack-1 Rack-2
Rack-2BRKACI-3456 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 171
Want to know more?
For Your
Documentation (Cont.) Reference
• ACI Unified Plugin for OpenStack Architecture Guide:

• https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/1-
x/openstack/b_ACI_with_OpenStack_OpFlex_Architectural_Overview/b_ACI_with
_OpenStack_OpFlex_Architectural_Overview_chapter_010.html
• APIC OpenStack Plugin Datasheet:
• http://www.cisco.com/c/en/us/solutions/collateral/data-center-
virtualization/openstack-at-cisco/datasheet-c78-732353.html
• ACI Product Information:
• https://www.cisco.com/c/en/us/support/cloud-systems-management/application-
policy-infrastructure-controller-apic/tsd-products-support-series-home.html
Cisco Webex Teams
Questions?
Use Cisco Webex Teams (formerly Cisco Spark)
to chat with the speaker after the session
How
1 Find this session in the Cisco Events Mobile App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space
cs.co/ciscolivebot#BRKACI-3456
Complete your online
session survey
• Please complete your Online Session
Survey after each session
• Complete 4 Session Surveys & the Overall
Conference Survey (available from
Thursday) to receive your Cisco Live T-
shirt
• All surveys can be completed via the Cisco
Events Mobile App or the Communication
Stations
Don’t forget: Cisco Live sessions will be available for viewing

on demand after the event at ciscolive.cisco.com
Continue Your Education
Demos in Meet the Related

Walk-in
the Cisco engineer sessions
self-paced
Showcase labs 1:1
meetings
Thank you

ACI-3456 Openstack

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

ACI-3456 Openstack

Uploaded by

Copyright:

Available Formats

BRKACI-3456

Mastering ACI and

• Who’s backing OpenStack?

Non distributed No underlay

• Full Neutron Node datapath replace

Integrated Overlay and Operations and

• Automatic VXLAN tunnels at top of rack

Integrated Overlay and Operations and

Integrated Overlay and Operations and

• Fully managed underlay network

Integrated Overlay and Operations and

• Troubleshooting across physical and

Visibility: Live statistics in APIC per VM

• APIC AIM Mechanism driver enables the user to operate OpenStack

• You can have multiple

vm1 vm2 vm3 vm4

Create Application Policy

Automatically Push What if you want to modify

• Update ACI objects | common | openshift-39_bd_kubernetes-service |

values | prj_b75263c9e55c48cab43b2edaf59e5d9a | net_8f2d85b6-89c5-4e93-8339-267e52a01ac9 |

• Create ACI objects | common | BD-T1-SITE2 |

AIM Controller CLI

AIM Controller CLI

• Query for existing ACI | enable_routing | True |

objects details | ip_learning | True |

AIM Controller CLI

AIM Controller CLI

allows to: | name | net_8f2d85b6-89c5-4e93-8339-267e52a01ac9 |

• Query for existing ACI | enable_arp_flood | True |

objects | display_name | serverNET |

• Query for existing ACI | enable_routing | True |

objects details | ip_learning | True |

• Update ACI objects | l2_unknown_unicast_mode | proxy |

values | vrf_name | DefaultVRF |

• Create ACI objects | monitored | False |

OpFlex agents OpFlex Agent (old name Agent OVS)

(optional) • The OpFlex Agent runs on the compute

OpFlex for OVS

vm1 vm2 vm3 vm4

vm1 vm2 vm3 vm4

Distributed Distributed OVS rules

Non OpFlex/PhysDom No Yes No No No IPTables

OVS-DPDK no OpFlex No Yes Yes

All VMs running

vm1 vm2 vm3 vm4

• Traditionally in OpenStack the

tap446aaf1g tap673e74ca tap483acb1f tap413e41cd

tap446aaf1g tap673e74ca tap483acb1f tap413e41cd

• Traditionally in OpenStack the

tap446aaf1g tap673e74ca tap483acb1f tap413e41cd

• Traditionally in OpenStack the

tap446aaf1g tap673e74ca tap483acb1f tap413e41cd

network is external to OpenStack

Each compute node will be

Floating IP will appear here with

• Traditionally VMs are getting IP

• Traditionally in OS VMs get the

Nova-api 10.11.0.21 10.11.0.33

Security Group Host Protection Profile (HPP) policies

L2 Policy Classifier Action L2 Policy

Provider EPG Consumer EPG

net1 net2 net3 net4 net1 net2 net3 net4

WWW WWW WWW