You are on page 1of 154

#CLUS

Your First Seven
Days Of ACI
Joseph Ristaino – Technical Leader, DCBU
ACI Escalation
Carlo Schmidt – Technical Leader – ACI
Solution Support
BRKACI-1001

#CLUS
Agenda
• Day 1: Why ACI?
• Day 2: Infrastructure and Policies
• Day 3: Forwarding Overview
• Day 4: Network Centric Migrations
• Day 5: Multi Location Deployments
• Day 6: Troubleshooting Tools
• Day 7: Additional Resources

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Cisco Webex Teams
Questions?
Use Cisco Webex Teams (formerly Cisco Spark)
to chat with the speaker after the session

How
1 Find this session in the Cisco Events App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space

Webex Teams will be moderated cs.co/ciscolivebot#BRKACI-1001
by the speaker until June 18, 2018.

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Acronyms/Definitions Reference Slide Icon 

Acronyms Definitions Acronyms Definitions
ACI Application Centric Infrastructure SVI Switch Virtual Interface

ACL Access Control List VIC Virtual Interface Card

API Application Programming Interface VNID Virtual Network Identifier
APIC Application Policy Infrastructure VPC Virtual Port-Channel
Controller

BD Bridge Domain VRF Virtual Routing and Forwarding
COOP Council of Oracle Protocol VTEP VXLAN Tunnel Endpoint

ECMP Equal Cost Multi Pathing VXLAN Virtual Extensible LAN

EP Endpoint

EPG Endpoint Group

KVM Keyboard, Video, and Mouse

MP-BGP Multi Protocol BGP

pcTag Policy Control Tag

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
Day 1: Why ACI?
Why ACI?
Challenges of Today
Management
• CLI to every Device
Core • Manual Configuration –
Takes Time
• Harder as we scale!

Dist Functionality
• Spanning Tree to Prevent
Loops
• Static Configuration
• Allow all Traffic by Default
Access • Coordination between
Network and Server Team

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
Why ACI?
ACI Overview Application Centric Infrastructure
Software Defined Networking built on Nexus 9000
Control Plane is Decoupled From the Data Plane
Spines
Spine1# show module
Mod Ports Module-Type Model Status
--- ----- ----------------------------------- --------------- ------
2 32 32p 40/100G Ethernet Module N9K-X9732C-EX ok
22 0 Fabric Module N9K-C9504-FM-E ok
23 0 Fabric Module N9K-C9504-FM-E ok
24 0 Fabric Module N9K-C9504-FM-E ok
26 0 Fabric Module N9K-C9504-FM-E ok
27 0 Supervisor Module N9K-SUP-A Active
Leafs 28 0 Supervisor Module N9K-SUP-A Standby

Leaf4# show module

Mod Ports Module-Type Model Status
--- ----- ---------------------------------- ------------------ ------
----
APIC APIC APIC 1 54 48x10/25G+6x40/100G Switch N9K-C93180YC-EX ok

APIC Cluster
#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
What is ACI?
ACI Overview
Management
• Fabric is managed by APIC
Spines • All configuration exposed via API
• Switches join fabric in a few clicks!

Functionality
Leafs
• No spanning Tree – ECMP Routing
• Dynamic Configuration
• Whitelist Model (customizable)

APIC APIC APIC

APIC Cluster
#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
R L3 Routing
Why ACI? V VLAN
Functionality GW Gateway (SVI)
T VXLAN Tunnel Endpoint
(TEP)

T T
ISIS/BGP Overlay

T T T T T T

APIC
GW
GW V
APIC R
APIC External
L2 & L3
Bare Metal Network
APIC Cluster Hypervisors
#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
Why ACI?
Functionality

A layer 3 network running ISIS is configured automatically by your APIC cluster
to provide a routed underlay network between leafs and spines – user does not
have to understand and build underlay

A overlay network is built using a enhanced version of VXLAN to allow layer 2
switching across the fabric as well as per VRF routing across the fabric – user
does not have to understand how to build overlay

VXLAN VNIDs are used to separate layer 2 switching as well as layer 3 routing

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
Why ACI?
Management Overview

• GUI gives full
visibility into the
entire system
• Controller
status shows
state of the
APIC Cluster.
• “Fully Fit”
means all
APIC’s are in
sync and
communicating

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
Why ACI?
Management Overview

• Faults are
thrown for
various reasons
to warn user of
issues in the
environment.
• Faults are
classified based
on severity of
the error

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
Why ACI?
Management Overview

Looks like we
had an issue!

• Health scores are driven
based on faults and
events
• Can be viewed system
wide or per object

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
Why ACI?
Management Overview

• Fabric
Inventory and
Topology are
centrally
managed.

Clicking on
Objects will drill
down further

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
Why ACI?
Management Overview

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
Day 2: Infrastructure
and Policies
Infrastructure and Policies
APIC Components APIC
ACI MGMT

A S A S

UCS C220

1) Cisco VIC 1225 (Copper or Fiber)
2) Two 10Gb port for connections to ACI Switches
A – Active 3) 1Gb Copper Ethernet port for CIMC
S - Standby 4) Console Port
5) Two 1Gb Copper Ethernet Ports for OOB MGMT
#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
Infrastructure and Policies
Best Practice
OOB
Network

ACI Spine Switches
1 OOB MGMT per SUP
1 Console per SUP
40/100 Gb connections to Leafs

ACI Leaf Switches
1 OOB MGMT
1 Console
40/100 Gb connections to Spines

Console
Server

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
Checklist
 CIMC
 Management
 NTP
 TACACS
 Backups
Infrastructure and Services
CIMC

• Use for APIC
Hardware
Diagnostics and
Remote Access
• Use to install the
APIC Software

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
Infrastructure and Services
CIMC

• CIMC KVM Provides
Remote Access
• Equivalent of
Console

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
Checklist
 CIMC
 Management
 NTP
 TACACS
 Backups
Infrastructure and Services
Required Addressing

1. Infra Subnet
2. Infra VLAN
3. BD Multicast Range
1 4. OOB Network IP’s
(CIMC included)
2
3

4 NOTE: Infrastructure subnet and
BD MCAST is used internally for
APICs and Switches!

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
Management - Required Addressing
Planning
Requirements Notes

Fabric Name Has to be consistent on all APICs Fabric1
Fabric ID Set to 1 (Default) 1
TEP Pool Recommended a /19 network. APIC will assign IPs from this 10.0.0.0/16
pool to Leafs, Spines and other Fabric specific services. Avoid
IP space which APIC might have to communicate with. E.g.:
vCenter or other integrated services
GIPO Pool Multicast network for flooding inside ACI. Not exposed to 225.0.0.0/15
external network unless using Multipod

Infra VLAN VLAN will be reserved for internal ACI communication. Cannot 3967
be deployed toward user servers

APIC OOB IP 1 IP per APIC, has to be out of band. Inband can be configured
later.
Switch 1 IP per switch, can have inband, out of band or both.
Management IP

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
Infrastructure and Services
APIC UI
APIC Management

apic 1

API

APIC Cluster
CLI (ssh)
apic 2

apic 3
#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
Infrastructure and Services spine 1 spine 2

Switch Management

ACI Fabric

leaf 1 leaf 2 leaf 3 leaf 4 leaf 5
APIC

Leaf and Spine Access
- Console
- SSH – Direct or via APIC
- REST API
#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
Checklist
 CIMC
 Management
 NTP
 TACACS
 Backups
Infrastructure and Services
NTP & PTP

• APIC’s send time in control
plane messaging
• Certificates
• Tech Supports 
• Atomic Counters!

• If Fabric is Gen 2 or newer
(EX/FX), Spine can act as a PTP APIC APIC APIC
master as well
• Allows user to measure latency
between EndPoints and leafs

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
Checklist
 CIMC
 Management
 NTP
 TACACS
 Backups
Infrastructure and Services
AAA
Allows users to
authenticate with certain
privilege levels

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
Infrastructure and Services
AAA jristain@apic1:~> moquery -c aaaModLR | grep -C 12 "2017-02-13T15"
<snip>
# aaa.ModLR
“Oh no! We lost connectivity to id
affected
: 8589940567
: uni/tn-Joey-Tenant/BD-Joey-BD3
servers on February 12th at 3pm cause : transition

EST!?”
changeSet : arpFlood (Old: no, New: yes), unkMacUcastAct (Old: proxy, New: flood)
childAction :
clientTag :
code : E4206171
created : 2017-02-13T15:06:07.249+00:00
descr : BD Joey-BD3 modified
dn : subj-[uni/tn-Joey-Tenant/BD-Joey-BD3]/mod-8589940567
ind : modification
modTs : never
rn : mod-8589940567
sessionId : Ld0sxAcCRfmb2Qb+W+XbUg==
severity : info
status :
trig : config
txId : 4611686018449066821
user : remoteuser-jristain

Logs changes per user!!

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
Checklist
 CIMC
 Management
 NTP
 TACACS
 Backups
Infrastructure and Services
Backups – Configuration Export
• JSON/XML export of the current fabric
configuration
• Can set on a scheduler
• Exports to a Remote Location
(FTP/SCP/SFTP) - DISASTER RECOVERY

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
Infrastructure and Services
Backups - Snapshots

Creates a Config Backup that is stored on the APIC by default
Run on a Per Fabric or Tenant Basis

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
Infrastructure and Services
Backups - Snapshots

• Rollback feature allows config
rollback between 2 Object

snapshots
• Can also compare
differences between a Changed To

previous SS Changed From

Changed From

Changed To

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
Infrastructure and Services
CIMC, NTP, AAA, and Backup Planning
Requirements Notes

CIMC IP per Unique IP address used for IP KVM built into APIC.
APIC Must use dedicated port
NTP Server NTP Server which all nodes inside fabric will use
User TACAS/ RBAC or RADIUS Server for accounting.
Management Custom local user account can be used too

Scheduled Multicast network for flooding inside ACI. Not
backup exposed to external network unless using
Multipod
Backup Server Server outside of ACI Fabric running FTP, SFTP or
SCP Server

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
Checklist
 CIMC
 Management
 NTP
 TACACS
 Backups
Fabric and Tenant
Policies
Fabric and Tenant Policies
Access Policies S10 S20
vPC Policy

vPC Policy

L1 L2 L3 L4

Server vPC Port-Channel
Server Nexus 7000
vPC Port-Channel
#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
Fabric and Tenant Policies
Access Policies
Access policies refer to the configuration that is applied for physical and virtual
(hypervisors/VMs) devices attached to the fabric.

Broken into a few major areas:

Global Policy Switch Policy Interface Policy
• Pools • Policies • Policies
• Domains • Policy Groups • Policy Groups
• Attachable Access Entity Profiles • Profiles • Profiles

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
Fabric and Tenant Policies
vPC Domain Policy

• No Peer-Link
• No Peer-
Keepalive
• Uses Fabric
Links for
Communication

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
Fabric and Tenant Policies
Access Policies S10 S20 103-104 vPC

101-102 vPC

L1 L2 L3 L4

Server vPC Port-Channel
Server Nexus 7000

vPC Port-Channel
#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
Fabric and Tenant Policies
Port-Channels

Legacy NXOS Config
Nexus7710# show run int po 10

interface port-channel10
switchport mode trunk
vpc 10

Nexus7710# show run interface Ethernet1/10

interface Ethernet1/10
speed 10000
lldp transmit
lldp receive
channel-group 10 mode active

Unspecified fields use
default values
#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
Fabric and Tenant Policies
Access Policies S10 S20
103-104 vPC

101-102 vPC

L1 L2 L3 L4

Server BareMetal01-vPC
Server Nexus 7000
N7710-vPC
#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
AEP

The AEP is used to associate a domain to one or more interface policy
groups. In most deployments it is recommended to use a single AEP if
VMM integration is not being used. If the ACI Fabric will be integrated
with n VMM domains, use 1 + n to determine how many AEPs are
needed
The Domain is used to specify what type of path (vlan) can be
deployed on a interface. If a AEP does not contain a “External Routed
Domain”, the interface can not be used to deploy a L3Out.

In Most deployments a single VLAN pool can be used with 1 Physical
Domain and 1 External Routed Domain.

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
Relationship View
Access Policies Workflow Example
Switch Profile Leaf-101 vPC-101-102

Interface Profile Leaf-101 vPC-101-102

Interface Selector P1-5_WinAD P6-7-N7K-vPC

Interface Block 1/1-5 1/6-7 1/6-7

Interface Policy Group Win2016Serv
N7K-vPC

Interface Policies CDP_On LLDP_Off BPDU_Guard LACP

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
Management - Required Addressing
Planning
Requirements Notes Example

AEP 1 AEP for all Policy groups. Map all domains to this Policy group Prod_AEP

Domain 1 Physical Domain, 1 External Routed Domain phys
L3Out
VLAN Pool 1 VLAN pool for all statically deployed vlans. 1 VLAN pool for Static_VLANs
Dynamically deployed VLANs. These pools should not overlap. VMM_Domain

Switch Profile 1 Profile per switch for Orphan Ports, 1 Profile per vPC Domain vPC-101-102, Leaf101,
(Containing both switches) Leaf102
Interface Profile Create a 1 to 1 mapping to switch Profile vPC-101-102, Leaf101,
Leaf102

Interface Selector Name after Server, Include Port ID. P11-N7710-vPC

Policy Group 1 Policy Group per Port-Channel/ vPC. Policy Groups can be N7710-vPC
reused for access ports. Assign AEP to Policy Group
#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
Global Policy
Pools (Vlan / VXLAN) Pool1 Pool2
A resource pool of encapsulations that can
be allocated within the fabric.

Domains (Physical / VMM / External Bridged / External
Routed)
DomPhy1 DomL2Ext1
Administrative domain which selects a vlan/vxlan pool for
allocation of encaps within the domain

Attachable Access Entity Profiles (AEP)
AEP
Selects one or more domains and is referenced/applied by TenantA
interface policy groups.

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
Global Policy - Attachable Entity Profiles
Configuration:
• Create a VLAN/VXLAN pool with a range
of encapsulations Pool1 Pool2 Pool3 Pool4
• Create a domain (physical, l2/l3 external, DomPhy1 DomVm1 DomL2 DomL3
or VMM) and associate pool
• Associate domain to AEP AEP AEP AEP
• Associate interface policy group to AEP Statics VMs External
switch/interface selectors will apply the
config through the interface policy group
assign to specific ports
1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4
What have we accomplished?
• Specified what domains and
corresponding pools are allowed per
interface in the fabric!

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
Access Policies SWITCH POLICY

Policies define protocol / feature configurations

Policy Groups select which policies should be applied

Profiles associate policy groups to switches or interfaces,
through the use of selectors

Switch Policy Types: Interface Policy Types:
VPC Domain Link-level Storm Control
Spanning-tree (MST) CDP Data plane policing
BFD LLDP MCP
Fibre-channel SAN / Node Port-channel / LAG L2 (Vlan local / global) INTERFACE POLICY
Port-channel member Firewall
Spanning-tree

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
Interface Policy Groups
Used to specify which interface policies to be applied to a particular interface type.
It also associates an AEP (which defines which domains are allowed on the interface).

Types:
VPC Domain 1
Access port (EP1)
Access Bundle Groups
• Virtual Port-channel (EP2)
• Port-channel (EP3)

EP1 EP2 EP3

Note: Separate policy groups should be created for each port-channel (standard or VPC) that you
need to configure. All interfaces on leaf that are associated with a particular access bundle group
reside in same channel.

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
Port-Channel Policies
Classical vPC Domain configuration ACI Port-Channel Policies
Required configuration of domain, peer-link, and Specify mode, minimum / maximum links, and
peer-keepalive link on both devices in domain related protocol options (relating to LACP)

interface Ethernet1/5-6
lacp port-priority 32768
lacp rate normal
channel-group 10 mode on

interface Ethernet1/10-11
lacp port-priority 32768
lacp rate fast
channel-group 20 mode active

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
Access Policy Example
General Configuration (reused for many interfaces): AEP Pool1
1) Configure a physical domain and vlan pool CiscoLive
DomPhy1
2) Create an AEP and associate physical domain
3) Create switch/interfaces profiles for leaf (LEAF101) Switch Profile
• very easy to apply configurations if you create a
switch/interface profile for each leaf and one for each LEAF101
VPC domain pair
Leaf_101
4) Configure Interface policies (LACP / LLDP)
Interface Profile
LACP Active LEAF101
Policies

LLDP Rx / Tx enabled

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
Creating Physical Domain / AEP / Vlan Pool

In dropdown:
Click Create Attachable Entity Profile

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 55
Creating Physical Domain / AEP / Vlan Pool

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 56
Creating Physical Domain / AEP / Vlan Pool

Click + to add vlan
range

In dropdown:
Click Create VLAN Pool

Specify start and
end vlans in
range

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
Create Interface Profile for each leaf / VPC
domain

Enter name and submit

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
Create Switch Profile for each leaf / VPC domain

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 59
Create Switch Profile for each leaf / VPC domain

Enter name

Click + to add selector

Select the Interface Profile
created for this leaf earlier

Enter a name and choose
appropriate leaf or leafs
(for vpc pair)

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
Create common protocol configurations
Example demonstrates a common lacp port-channel policy

Use a descriptive name

Select the protocol

Configure
options/knobs

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 61
Access Policy Example
Interface specific (each time you add a new interface): AEP Pool1
1) Create policy group for device (VPC / PC / Access) CiscoLive
DomPhy1
2) Within the policy group, select the desired policies / AEP
3) Associate interfaces to policy group via desired leaf Switch Profile
profile
• use specific leaf profile if access or PC LEAF101
• use VPC leaf profile if policy group is VPC Leaf_101

Interface Profile
LACP Active PC_Server_1 LEAF101
Policies Policy Groups blk_1/1-2

LLDP Rx / Tx enabled Access_Servers blk_1/47-48

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 62
Create policy groups
Note:
A separate policy
Descriptive name
group should be
created for each
PC/VPC that you will
deploy
Associate your desired
interface policies
(otherwise default)

Associate your AEP to select
which domains this interface
can deploy

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
Create interface selectors / associate policy
group

Click + to add selector

Choose interface profile
to add selectors

Use a descriptive name

Specify interface/range

Associate the policy group to
deploy on interfaces

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
Example policy scheme
Switch Profile Leaf101 Leaf101_102

Interface Profile Leaf101 Leaf101_102

Interface Selector linux windows n7k_pc10 asa_cl1_pc1 n7k1_pc10 n7k2_pc10

Interface Block 1/20-25 1/30-35 1/10-11 1/45-48 1/10 1/20

1/1-4

Interface Policy Group linux-access windows-access asa_vpc_ccl asa_vpc_data n7k_vpc10

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 65
vPC Protection Group Policy

vPC Domain 1
vPC Domain 1 vPC Domain 2
Classical vPC Domain configuration ACI vPC Domain configuration
Required configuration of domain, peer-link, and Specify the Domain ID and the two Leaf switch IDs
peer-keepalive link on both devices in domain that form the domain pair
vpc domain 1
peer-keepalive destination 172.168.1.2 / VPC Protection Group
source 172.168.1.1 vrf vpc-keepalive
peer-gateway
ip arp synchronize
Name: vPC-Domain100
ID: 100
interface port-channel 20 Switch1: 101
vpc peer-link
Switch2: 102

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 66
VPC Protection Group (example configuration)
GUI sequence:
Tabs:
Fabric -> Access Policies

Navigation Tree:
Switch Policies -> Policies -
> VPC Domain -> Default

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 67
Fabric and Tenant Policies
Tenant Policies S10 S20

Extend VLAN to legacy Net
L1 L2 L3 L4 Allow Layer 2
Connectivity to 7K

Server
Server Nexus 7000

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 68
Fabric and Tenant Policies
Tenant Policies – Key concepts
Tenants are a Logical Grouping containing Policies. Resources in the
Common Tenant can be used in User Tenants

VRFs are used to separate routing tables inside the ACI Fabric. 1 or more VRFs
can be used.

Bridge Domains define your Broadcast/ Flood domain
Unique VXLAN VNID is used per Bridge Domain
Configure ARP Optimization and L2 Unknown Unicast Proxy
Subnet (SVI) can be defined under the BD and is mapped to a single VRF

Bridge Domain VRF Tenant

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 69
Fabric and Tenant Policies
Tenant Policies – Key concepts one EPG to another
EPGs defines a collation of policy assigned to a group of devices
Contracts, QoS, SPAN requirements
L4-L7 policies (PBR, Load balancing, Firewalls)
EPG is most commonly determined by ingress VLAN & Port

Contracts are a collection of filters which allow traffic to pass between EPGs
Contacts are similar to access-lists. Consumer is Source, Provider is Destination
Filters contain a list of protocols and ports

EPG

Bridge VRF
ICMP Contract
Domain Tenant
EPG

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 70
Fabric and Tenant Policies
Tenant View

EPGs

Bridge Domains

VRFs

Contracts

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 71
Fabric and Tenant Policies
Deploying a VRF

Change the VRF from a
White-List model to an
“Allow All” Model

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 72
Fabric and Tenant Policies
Deploying a Bridge Domain

Associate Bridge Domain to VRF

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 73
N7710# configure terminal

Fabric and Tenant Policies Enter configuration commands, one per line.
N7710(config)# interface port-channel 1
End with CNTL/Z.

Deploying an EndPoint Group N7710(config-if)# switchport trunk allowed vlan add 100

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 74
Fabric and Tenant Policies
Tenant Policies
S10 S20

Extend VLAN to legacy
Net L1 L2 L3 L4 Allow Layer 2
Connectivity to 7K

L2 Path

Server
Server Nexus 7000

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 75
Fabric and Tenant Policies
Planning
Requirements Notes Example

Tenant 1 Tenant can be used company. Tenants can also separate functions of a Prod/Dev
business. NOTE: Shorter names are easier when using CLI
VRF 1 or more VRFs per Tenant PROD-MAIN
DEV-TEST,DEV-PROD
Bridge Domain Recommended to have 1 BD per Legacy VLAN. For Network Centric VLAN_100,VLAN_101
Migrations, 1 BD should be used for each EPG. BD_vMotion

Application Profile Logical Container for EPGs. 1 AP is sufficient in most installations. NOTE: Prod-AP
This is strictly a management entity. No policies are defined on this object.
EndPoint Group Ports/VLANs (static path bindings) are added to EPGs to define what VLAN_100
Endpoints get defined in what EPGs. QOS/Contracts, etc are added to VLAN_101
EPGs. For Network Centric Migrations, 1 EPG should be used for each vMotion
Legacy VLAN.
Contracts Contracts can be re-used across multiple EPGs. If we compare this to an Web
ACL, the Consumer is the Source, and the Provider is the Destination.
Filters Add Required Ports and Protocols to allow communication. Only what is SRC: Any, DST:80
specified in the filter  contract will be allowed between EPGs providing SRC: Any, DST:443
and consuming that contract.

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 76
Fabric and Tenant Policies
Tenant Policies S10 S20
VMM enabled EPGs

Static Path

L1 L2 L3 L4

Hypervisor Cluster
Nexus 7000

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 77
Cisco ACI Hypervisor Integration
Application Network Profile
5 EPG EPGAP
EPG DB
WEB P
APIC L/B
F/W
Create Application Policy

APIC Admin

ACI
9
Fabric
Push Policy (Lazy)

1
6
Cisco APIC and Learn location of ESX
4
VMware vCenter Initial Automatically Map Host through LLDP
Handshake EPG To Port Groups

VIRTUAL DISTRIBUTED SWITCH
2 Create VDS
WEB PORT GROUP APP PORT GROUP DB PORT GROUP
Create Port
vCenter 7
Groups
Server
Web App DB Web Web DB
8 Attach Hypervisor
VI/Server Admin Instantiate VMs, to VDS
Assign to Port Groups 3
HYPERVISOR HYPERVISOR

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 78
Fabric and Tenant Policies
Layer 3 Connectivity
S10 S20

Layer 3 Access To
Core L1 L2 L3 L4 Provide External
Access to Server

ACI Layer 2
Server Layer 3
Core

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 79
Basic Connectivity node-103
RID: #
node-104
RID: #
IP: A IP: B

Layer3 Out: L3Out-1
VRF: VRF-V1
Layer-3 Domain: DomL3 vlan-x
Logical Node Profile: node-103-104

node: node-103 node: node-104 L3Out-1
Router-ID: # Router-ID: #
VRF-V1
Logical Interface Profile: ipv4-lif
Create the L3Out
path: topology/pod- • Associate VRF and L3 Domain
1/…vpcX • Create Logical Node Profile and associate fabric
type: ext-svi, encap: vlan-x nodes to the L3Out.
IP-A, IP-B, MTU, MAC • Create Logical Interface Profile
• Specify Path attributes containing physical interface,
encapsulation, and IPs

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 80
Fabric and Tenant Policies
Creating a Layer 3 Out

• External Routed Networks allow
us to peer with external routers
• Dynamic Protocols
• EIGRP
• OSPF
• BGP
• Static Routing

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 81
Fabric and Tenant Policies
Route Reflectors

• Fabric nodes communicate
using MP-BGP. L1 L2 L3 L4
• BGP advertises routes from
Border Leaf to Compute
Leafs.
• Runs in overlay-1 VRF ACI

Server
0.0.0.0/0
10.0.0.0/24

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 82
Fabric and Tenant Policies
Route Reflectors S10 S20

L1 L2 L3 L4

192.168.160.64 192.168.160.65

leaf3# show ip route vrf A:A
IP Route Table for VRF “A:A”
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

0.0.0.0/0, ubest/mbest: 1/0
*via 192.168.160.64%overlay-1, [200/1], 03w21d, bgp-90002, internal, tag 90002
*via 192.168.160.65%overlay-1, [200/1], 03w21d, bgp-90002, internal, tag 90002
10.0.0.0/24, ubest/mbest: 1/0
*via 192.168.160.64%overlay-1, [200/1], 03w21d, bgp-90002, internal, tag 90002
*via 192.168.160.65%overlay-1, [200/1], 03w21d, bgp-90002, internal, tag 90002

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 83
Fabric and Tenant Policies
Planning

Requirements Notes Example

BGP Route Use an AS Number not already in your environment. The AS number is 65000
Reflector only exposed to the external network when peering BGP with devices.
Private AS number can be used. NOTE: CHANGING THE AS NUMBER
IS DISRUPTIVE!
External Routed This is your Layer 3 Object. It contains the entire Layer 3 path L3out-To-Core
Network configuration.
Node Profile Defines which nodes are part of the Layer 3 out Domain. Here is Leaf101, Leaf102
where you define your Router ID’s and Static Routes. Leaf101-102

Logical Interface Defines which interfaces are used for peering. Support Types are Port10
Profile Routed Interfaces, Routes Sub-Interfaces, and SVIs. This is also vPC-To-Core
where you define the IP/MTU/VLAN is SVI or Sub-Interface.
Networks (External This is where you define the external subnets you want to apply policy Ext_EPG  0.0.0.0/0 subnet
EPG) to. You do this by listing the subnets and applying contracts. NOTE:
multiple all 0’s subnets should not be configured in the same VRF.

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 84
Fabric and Tenant Policies
Layer 3 Connectivity
S10 S20

Layer 3 Access To
Core L1 L2 L3 L4 Provide External
Access to Server

ACI Layer 2
Server Layer 3
Core

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 85
Agenda
• Day 1: Why ACI?
• Day 2: Infrastructure and Policies
• Day 3: Forwarding Overview
• Day 4: Network Centric Migrations
• Day 5: Multi Location Deployments
• Day 6: Troubleshooting Tools
• Day 7: Additional Resources

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 86
Day 3: Forwarding
Overview
What is an Endpoint?
Traditional Endpoint

L2 – MAC Table Eth1/1 Eth1/2
- MAC Address
- VLAN 10 20
- Interface

L3 – ARP Table
- IP / MAC
000a.000a.000a 000b.000b.000b
- Interface
192.168.1.100/24 192.168.2.100/24
- VRF

N5K# show mac address-table | grep 000a N5K# show mac address-table | grep 000b
• 10 000a.000a.000a dynamic 0 Eth1/1 • 20 000b.000b.000b dynamic 0 Eth1/2
N5K# show ip arp vrf default | grep 000a N5K# show ip arp vrf default | grep 000b
192.168.1.1 00:00:01 000a.000a.000a Vlan10 192.168.2.1 00:00:01 000b.000b.000b Vlan20

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 88
What is an Endpoint?
Eth1/1 Eth1/2
ACI Endpoint 10 20

- MAC or MAC/IP  IP is /32 or
/128 Route
- VLAN  EPG (pcTag) APIC
- Interface 000a.000a.000a 000b.000b.000b
- VRF 192.168.1.100/24 192.168.2.100/24
- Flags  Local, vPC, static, etc. EPG1 EPG2

apic1# show endpoints ip 192.168.1.100
Dynamic Endpoints:
Tenant : CL
Application : CL
AEPg : EPG1

End Point MAC IP Address Node Interface Encap
----------------- ---------------------------------------- ---------- ------------------------------ ---------------
00:0A:00:0A:00:0A 192.168.1.100 101 102 eth1/1 vlan-10

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 89
What is an Endpoint?
Eth1/1 Eth1/2
ACI Endpoint 10 20

- MAC or MAC/IP  IP is /32 or
/128 Route
- VLAN  EPG (pcTag)
- Interface 000a.000a.000a 000b.000b.000b
- VRF 192.168.1.100/24 192.168.2.100/24
- Flags  Local, vPC, static, etc. EPG1 EPG2

Leaf1# show endpoint mac 000a.000a.000a detail
Legend:
s - arp O - peer-attached a - local-aged S - static
V - vpc-attached p - peer-aged M - span L - local
B - bounce H - vtep
+-----------------------------------+---------------+-----------------+--------------+-------------+----------------+
VLAN/ Encap MAC Address MAC Info/ Interface Endpoint Group
Domain VLAN IP Address IP Info Info
+-----------------------------------+---------------+-----------------+--------------+-------------+----------------+
16 vlan-10 000a.000a.000a L eth1/1 CL:CL:EPG1
CL:17 vlan-10 192.168.1.100 L eth1/1

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 90
Endpoint Learning - ARP
ACI Leafs learn via ARP!
Eth1/1 Eth1/2

ARP Request Who has 192.168.1.101??
DMAC FFFF.FFFF.FFFF
000a.000a.000a 000b.000b.000b
SMAC 000a.000a.000a
192.168.1.100/24 192.168.1.101/24
Eth: 0x0806 Eth: 0x0806 EPG1 EPG1

Hdr/Opcode Hdr/Opcode
Frame Unicast Routing? EP Contents
Sender MAC 000a.000a.000a
ARP No MAC (Sender MAC)
Sender IP 192.168.1.100
ARP Yes MAC (Sender MAC),
Target MAC 0000.0000.0000 IP (Sender-IP)
Target IP 192.168.1.101

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 91
Endpoint Learning– Routed Frames
Routed Frame triggers an EP Learn
Eth1/1 Eth1/2

000a.000a.000a 000b.000b.000b
DMAC BD MAC 192.168.1.100/24 192.168.2.100/24
EPG1 EPG2
SMAC 000a.000a.000a

802.1Q 10
Frame Unicast Routing EP Contents
SIP 192.168.1.100 IPV4/6 Yes MAC (L2 SRC MAC),
IP (SRC IP)
DIP 192.168.2.100

Protocol 1

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 92
Anycast Gateway
S10 S20

• Gateway IP is programmed on all leafs
that need it
• Deterministic Traffic Flow to Gateway
• Consistent Latency across all Devices L1 L2 L3 L4
Towards Gateway
BD1 BD1 BD1 BD2 BD2

EP1 –EPG1 EP2 - EPG1 EP3 – EPG2
BD1 BD1 BD2

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 93
Proxy Routing
3  EP synced to
• Leafs report EP’s to spine other Spines
once Learnt  EP published
to Spine
S10 S20
• Spines maintain a database 2
of all Endpoints Learnt in the
Fabric, and on what Leaf(s)
they exist.
• Used for “Hardware Proxy”  EP L1 L2 L3 L4
BD Mode. learnt
on Leaf

1

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 94
ARP Flooding
EP1 ARP’s for EP2
• Behavior is the same as
Traditional Switches S10 S20  ARP is flooded
3 to all leafs that
• ARP is flooded using BD
Multicast Group to all Leafs have the BD
that have the BD 2  Flooded
ARP is
in
BD, copy to
Spine
 ARP L1 L2 L3 L4
1 Received
on L1 BD1 BD1 BD1
 L2 sends
4 ARP out
ports in BD
 L3 sends
ARP to EP2
000a.000a.000a 000b.000b.000b
192.168.1.100/24 192.168.1.101/24
EP1 - EPG1 EP2 - EPG1
BD1 BD1
#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 95
ARP Optimization – Unicast Routing
EP1 ARP’s for EP2
• ACI can Unicast ARP to avoid
unnecessary Flood traffic.  S10 S20
Requires Unicast Routing on 3  Spine knows Target
BD IP is on L3, Unicast
2  L1 doesn’t to L3
know Target
IP  Send to  L3 learns EP1 from
Spine! L1

L1 L2 L3 L4

BD1 BD1
 ARP
1 Received
on L1 4  L3 sends
ARP to EP2

000a.000a.000a 000b.000b.000b
192.168.1.100/24 192.168.1.101/24
EP1 - EPG1 EP2 - EPG1
#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 96
Known Unicast – Layer 2
EP1 pings EP2
S10 S20 3  Packet is sent
Outer Outer  L1 looks at the from L1 directly
DMAC and to L3 through
SIP L1 spines
2 knows it exists
DIP L3 on L3 in EPG1

VXLAN BD1
L1 L2 L3 L4
Inner Inner
BD1 BD1

DMAC BBBB 1  ICMP 4  L3 sends
Received ICMP to EP2
SMAC AAAA on L1

SIP 192.168.1.100

DIP 192.168.1.101 000a.000a.000a 000b.000b.000b
Protocol ICMP
192.168.1.100/24 192.168.1.101/24
EP1 - EPG1 EP2 - EPG1
#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 97
Known Unicast – Layer 3
EP1 pings EP2
S10 S20  Packet is sent
4
2  L1 looks at the from L1
DST IP and directly to L3
knows it exists through spines
Subnet under BD acts as GW on L3 in EPG2
If traffic is destined to the GW X
MAC, we do an IP Lookup in the
VRF L1 L2 L3 L4

BD1 BD2
 ICMP
1  L3 sends
EPG1
Received 5 ICMP to EP2
EPG2 on BD GW 3
ICMP ICMP
000a.000a.000a 000b.000b.000b
192.168.1.100/24 192.168.2.100/24
EP1 - EPG1 EP2 – EPG2
#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 98
Day 4:
Network Centric
Migrations
Physical Layer
S10 S20

L1 L3 L4

L2

vPC to allow L2 VLANs

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 100
Checklist
 Physical Layer 
 Layer 2
 Layer 3
Network Centric Design
L2 Migration Recommendations BD_VLAN100

Each Legacy VLAN requires a unique Bridge Domain +
Settings: Unicast Routing Disabled
EPG
Unknown L2 Flooding VLAN_100

ARP Flooding
=
Each Legacy VLAN has a unique EPG
Legacy VLAN
100
What have we Accomplished?
Each Legacy VLAN maps to a unique Bridge Domain

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 102
Conceptual View
Legacy ACI

VRF CiscoLive

BD_VLAN100 BD_VLAN101 BD_VLAN102

EPG EPG EPG
VLAN_100 VLAN_101 VLAN_102

VLAN100 VLAN101 VLAN102
#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 103
Conceptual View
S10 S20

SVI/VLAN:100
192.168.100.1
SVI/VLAN:101 L1 L3 L4
192.168.101.1
SVI/VLAN:102
192.168.102.1 L2

L2 Extension

BD_100 BD_101 BD_102
EPG 100 EPG 101 EPG 102
VLAN100 VLAN101 VLAN102
#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 104
Spanning-tree in ACI
• ACI Fabric does not run Spanning-tree
• BPDUs are flooded in ‘EPG VNID’ (use same VLAN pool for all ports deploying legacy VLANs)
• ACI Fabric does snoop BPDUs and will flush Endpoints (Mac & IP) when TCNs are received
• Learning is disabled when excessive BPDUs are received

• External Spanning-tree devices should be configured with “spanning-
tree link-type shared”
• Use “show mcp internal info vlan encap_vlan” to see TCNs
Leaf101# show mcp internal info vlan 100
-------------------------------------------------
PI VLAN: 13 Up
Encap VLAN: 100
PVRSTP TC Count: 11
RSTP TC Count: 0
Last TC flush at Mon May 1 19:32:22 2017
on Tunnel13

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 105
Verification
APIC GUI shows connected Endpoints (MAC and or IP) per EPG and Path
E.g.: 5C:83:8F:69:BB:C9 (N7K) connected via Nodes-101-102/N7710-vPC

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 106
Checklist
 Physical Layer 
 Layer 2
 Layer 3
Network Centric Design
L3 Migration Requirements
Configure “Layer 3 Out” to create a routed connection to legacy network
Routed Interface
Routed subinterface Subnet
EPG
Switched Virtual Interface (SVI) Bride Domain VLAN_100

Bridge Domain with “Unicast Routing” enabled
Subnet defined on BD
L3Out
L3Out associated with BD
EPG has contract to L3Out Network
Dynamic Routing Routing Protocol

OSPF/ EIGRP/ BGP/ Static
#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 108
Conceptual View
S10 S20

L3 Extension
SVI/VLAN:100
192.168.100.1
SVI/VLAN:101 L1 L3 L4
192.168.101.1
SVI/VLAN:102
192.168.102.1 L2

L2 Extension

BD_100 BD_101 BD_102
EPG 100 EPG 101 EPG 102
VLAN100 VLAN101 VLAN102
#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 109
L3 Migration Considerations
1) Disable External GW!
2) Bridge Domain Settings
Unicast routing Enabled – Minor Service Impact
L2 Unknown Unicast H/W Proxy – Service Impact
ARP Flooding Optimized - In conjunction with L2
Unknown Unicast
Limit IP learning to Subnet
Off Subnet Learns are cleared
Learning is disabled for 2 minutes
3) Global Settings
Enforce Subnet Check - adds prefix check to all BD’s

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 110
Verification
APIC GUI now shows IP information since UC Routing is enabled on BD
E.g.: 192.168.102.11 connected via Nodes-101-102/BareMetal02-vPC

Recommended Content! – ACI Endpoint Learning White Paper
https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-739989.html

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 111
Verification

GUI

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 112
Verification

Leaf101# show ip ospf neighbors vrf CiscoLive:VRF1
OSPF Process ID default VRF CiscoLive:VRF1
Total number of neighbors: 1
Neighbor ID Pri State Up Time Address Interface
192.168.255.255 1 FULL/BDR 02:27:05 192.168.255.2 Eth1/13

SSH Leaf101# show ip route vrf CiscoLive:VRF1 10.0.0.0/8
IP Route Table for VRF "CiscoLive:VRF1"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

10.0.0.0/8, ubest/mbest: 1/0
*via 192.168.255.2, eth1/13, [110/5], 01:45:34, ospf-default

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 113
Common Pitfalls
Old Gateway still Active!
S10 S20

L3 Extension
SVI/VLAN:100 SVI/VLAN:100
192.168.100.1 192.168.100.1
SVI/VLAN:101 L1 L3 L4 SVI/VLAN:101
192.168.101.1 192.168.101.1
SVI/VLAN:102 SVI/VLAN:102
192.168.102.1 L2 192.168.102.1

L2 Extension

BD_100 BD_101 BD_102
EPG 100 EPG 101 EPG 102
VLAN100 VLAN101 VLAN102

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 114
Common Pitfalls
Windows Dynamic Load Balancing S10 S20

Problem:
Traffic is Sourced with the same IP but
from both NIC’s using different MACs
ACI Fabric sees frequent IP Move L1 L2 L3 L4
between MAC’s when Routing is
Enabled!

Solution:
Use “Hyper-V Port” to force single
MAC to IP Communication

NIC1: MAC A NIC1: MAC A
NIC2: MAC B IP: 192.168.100.11
IP: 192.168.100.10

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 115
Checklist
 Physical Layer 
 Layer 2
 Layer 3
Day 5: Multi-Location
Deployment Options
Stretched Fabric IS-IS

S10 S20 S11 S21

L1 L2 L3 L4 L5 L6 L7 L8 L9 L10

APIC APIC APIC APIC

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 118
Stretched Fabric
Advantages
• All one Fabric
• No Additional Routed Infrastructure
• Simple Provisioning – If cabling is in
place S10 S20 S11 S21

Limitations
• Single APIC Failure Domain L1 L2 L3 L4 L5 L6 L7 L8 L9 L10

• L1 Connectivity between Transit
Leafs and spines (dark fiber)
• Same Control Plane Instance
APIC APIC
APIC

Across Sites

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 119
IPN MTU Requirements: 9150 Bytes

Multipod IS-IS
IPv4 Multicast
IPN IPN OSPF
Network

S10 S20 S11 S21

L1 L2 L3 L4 L7 L8 L9 L10

APIC APIC APIC APIC

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 120
Multipod
Advantages
• All one Fabric
• Policy Stretched across sites IPN IPN
• Separate Control Plane Instances
per site S10 S20 S11 S21

• Increases Leaf Scale to 400

Limitations L1 L2 L3 L4 L7 L8 L9 L10

• Single APIC Failure Domain
• Need dedicated Routing Devices as
Inter-Pod Network (IPN) Routers. APIC APIC
APIC

• Requires PIM BI-Dir to route BUM
traffic between sites.
• 50ms max latency between pods

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 121
IPN MTU Requirements: 9150 Bytes

Remote Leaf
IS-IS
IPV4 “Inter-site” OSPF
ISN Network ISN
Primary Site Remote Office/ DC

S10 S20 RL1 RL2

L1 L2 L3 L4

APIC APIC APIC

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 122
Remote Leaf
Advantages
• All one Fabric
• Easy Addition of small site to ISN ISN

existing APIC
• Spines not required in Remote Site. S10 S20 RL1 RL2

• Connects to existing routing
infrastructure
• No Multicast required L1 L2 L3 L4

Limitations
• All traffic goes to “main” site before APIC APIC APIC

other sites.
• 140ms Latency Restriction
• Port Count

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 123
IPN MTU Requirements: 9150 Bytes

Multi-Site IS-IS
IPV4 “Inter-site” OSPF
IPN Network IPN

S10 S20 S10 S20

L1 L2 L3 L4 L1 L2 L3 L4
ACI
Multi-Site
ACI
Controller
Multi-Site
ACI
Controller
Multi-Site
Controller

APIC APIC APIC APIC APIC APIC

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 124
Multi-Site

Advantages IPN IPN

• Two Independent Fabrics (APIC Clusters)
S10 S20 S10 S20
• Policy is synchronized using Multi-Site
Controller
• Connects to existing routing infrastructure
• No Multicast required L1 L2 L3 L4 L1 L2 L3 L4

ACI
Multi-Site
ACI
Controller

Limitations
Multi-Site
ACI
Controller
Multi-Site
Controller

• 500ms – 1s latency for OOB APIC APIC APIC APIC APIC APIC

MSC  APIC connectivity
• Not all Site Specific Config can be done
from MSC

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 125
Day 6:
Troubleshooting Tools
Faults Available in 2.2(2e)!

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 127
EP Tracker

“We had a
problem at
14:21!!!”

Attach/Detach events
are logged for each EP

IP Was Moving???
#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 128
Atomic Counters
S10

Used to measure packet loss in Overlay
Logs packet count between EP’s on different Leafs Tx Rx

Specific Filter can be set L1 L2
Requires NTP!
Leaf Direction Filter Packet Count
L1 Tx ICMP 500
L2 Rx ICMP 500
192.168.101.10 192.168.102.11

Ping –c 500 192.168.102.11 –c
500

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 129
Atomic Counters

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 130
Atomic Counters

NO Packet Loss In Overlay

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 131
S10

SPAN
EP Learnt
• ACI allows for SPAN of EPG L1 L2
• ERSPAN Destination must be an IP EP ERSPAN
Learnt in ACI
10.10.10.10
• EP Can run Wireshark or Tshark Leaf101# show monitor session all
session 1
---------------
description : Span session 1
type : erspan
version : 2
oper version : 1
EPG 100 state : up (active)
erspan-id : 1
SPAN Source SPAN granularity :

Destination
vrf-name : CiscoLive:VRF1
acl-name :
ip-ttl : 64
EPG ERSPAN ip-dscp
specified
: ip-dscp not

destination-ip : 10.10.10.10/32

Port ERPSAN/Local origin-ip
mode
: 1.1.1.1
: access
Port source VLANs
rx
:
: 100
tx : 100
both : 100
filter VLANs : filter not specified

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 132
Troubleshooting Wizard - Faults

Shows Faults
in the Path

Builds Topology of Flow

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 133
Troubleshooting Wizard – Drop Stats

Shows Drops on Every Hop.
Green Arrows portray no Drops NOTE: Some Drops are expected.
Look for Drops like “Buffer” and “Error”!

Recommended Content! – Understanding Drop Faults in ACI
http://www.cisco.com/c/en/us/support/docs/cloud-systems-management/application-policy-
infrastructure-controller-apic/210539-Explanations-of-Packet-Drop-Faults-in-AC.html

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 134
Troubleshooting Wizard - Contracts

Shows Contracts for
Flows

Implicit Deny Allow SSH

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 135
Troubleshooting Wizard – Atomic Counters

No Drops!

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 136
Troubleshooting Wizard – SPAN
Ability to SPAN to APIC or other devices
attached to the Fabric

User can select which ports to SPAN

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 137
Capacity Dashboard

Contract TCAM is
Full!

Capacity Dashboard panel displays your usage by range and percentage.
Use this to plan your fabric Scale.

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 138
App Center
Enhanced Endpoint Tracker

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 139
Enhanced Endpoint Tracker

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 140
#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 141
#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 142
App Center
Elam Assistant

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 143
ELAM Assistant

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 144
ELAM Assistant

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 145
Day 7: Additional
Resources
Support Forums

TAC Engineers are Subscribed
Easy Portal to Post Non Impacting
Questions or Concerns
Has Documentation written by
CSE’s and Technical Leaders

https://supportforums.cisco.com/t5/application-centric/bd-p/12206936-discussions-aci

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 147
Facebook Group

Many Customers and Cisco
Employees
Great Real World Deployment
Advice
Great way to meet others working
with ACI

Great Community 

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 148
Solutions Support
One TAC team to support all
aspects of ACI
Engineers are familiar with 3rd
party products like VMWare
Case does not get handed off
when it is a Switching vs. Routing
issue.
ACI Team takes ownership

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 149
JumpStart
Program designed by TAC

Two 3-hour WebEx session with
TAC

Talk to your Cisco Account team
to get scheduled for your
JumpStart!

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 150
Complete your online session evaluation

Give us your feedback to be entered
into a Daily Survey Drawing.
Complete your session surveys through
the Cisco Live mobile app or on
www.CiscoLive.com/us.
Don’t forget: Cisco Live sessions will be available for viewing
on demand after the event at www.CiscoLive.com/Online.

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 151
Continue
your Demos in
the Cisco
Walk-in
self-paced
Meet the
engineer
Related
sessions
education campus labs 1:1
meetings

#CLUS BRKACI-1001 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 152
Thank you

#CLUS
#CLUS