ACI Operations and

Troubleshooting
Azeem Suleman, Principal Engineer, Insieme Business Unit

@azeem_suleman
Agenda
• Cisco ACI Overview
• Demo
• Cisco ACI Operations
• What’s new in Cisco ACI 1.2(x)
• NX-OS Style CLI and Basic GUI
• Configuration Rollback and Traffic Map
• Unmanaged Node for L4-7 Services
• Intra-EPG Isolation

• Troubleshooting
• Q&A
Cisco ACI Overview
Application Centric Infrastructure (ACI)
Rapid Deployment of Applications onto
Networks with Scale, Security and Full Visibility

APPLICATION CENTRIC
NEXUS 9500 AND 9300 POLICY CONTROLLER

ACI
Architecture

Spine Nodes

Leaf Nodes

AVS

EPG “Internet” Service Producers EPG “Users”

EPG “Files”

Service Consumers
Application Policy Model and Instantiation
Application
Client
Application policy model: Defines
the application requirements Storage Storage
(application network profile)
Web Tier App Tier DB Tier

Policy instantiation: Each device

dynamically instantiates the required
changes based on the policies
VM VM VM VM VM VM VM

10.2.4.7 10.9.3.37 10.32.3.7

All forwarding in the fabric is managed through the application network profile
• IP addresses are fully portable anywhere within the fabric
• Security and forwarding are fully decoupled from any physical or virtual network attributes
• Devices autonomously update the state of the network based on configured policy requirements
Access Methodology
• CLI (Command-line interface)
• Means of interacting with a computer program where user issues commands to the
program in the form of successive lines of text (command lines)

• GUI (Graphical user interface)

• Interface that allows users to interact with devices through graphical icons and visuals

• Programmable interface
• Software components / objects exposed to be called directly by other programs

• Open Source Tool

• ACI Toolkit – Configuration Roll Back, Endpoint Tracker and other applications
ACI Toolkit
• Simple toolkit built on top of APIC API NX-OS Custom
• Set of simple python classes like Linux Python
• Python Library CLI Commands Scripts
• Used to generate REST API calls
• Runs locally
ACI Toolkit
• Small number of classes
• ~30 currently
• “Intuitive” names

• Not full functionality, most common

• Focused primarily on configuration APIC
• Preserves the ACI basic concepts
• Tenants, EPGs, Contracts, etc.
Demo
Cisco ACI Operations
Life of an Operations Engineer
• Pre-deployment (Fulfillment)
• Fast provisioning end-to-end
• Check to verify / validate what is configured exists on the ACI Fabric
• Make sure ACI Fabric and integrated devices (L4-7) will do exactly what is expected
• Apps come-up with no surprises
• Validate policy before apps are powered on

• Post-deployment (Assurance)
• Proactively know if something is wrong
• Before incident happens or issue is reported
• Easily troubleshoot the fabric as one system
Cisco ACI Fabric
Cisco ACI Deployment Lifecycle
Monitoring Managing Troubleshooting
• Audit Logs
• Faults
• iPing
• Image Management
• Events
• iTraceroute
• Health Score • Config Export / Import
• Endpoint Tracker
• Atomic Counter • Fabric Inventory
• ERSPAN (local laptop /
• Contract deny logs • Show Usage APIC Wireshark)

• Statistics • Traffic Map

• Configuration Rollback
• Capacity Dashboard • Managed Object
Browser (Visore)
Show Usage
• Prior to commit changes, allow user(s) to view impacts of configuration changes
• Example shows the message when user tried to delete a VLAN pool
Capacity Dashboard
Cisco ACI Optimiser

https://www.youtube.com/watchv=m7_C5htXAr4&feature=youtu.be
What’s New in Cisco ACI
1.2(x)
 Cisco NX-OS Style of CLI on Cisco APIC
Leaf 102

Leaf 101 Leaf 102

foo Eth 1/1-48 Eth 1/1-48 Tenant T1

demo-apic1(fabric-exec)# show mac address-table address

--------------------------------------------------------------------------------
Node leaf101 Output: demo-apic1# show running-config tenant t1
------------------------------------------------------------------------------- tenant t1
(config)# leaf 102 vrf context v1
VLAN MAC Address Type age Secure Ports
(config-leaf)# interface port-channel foo ---------+-----------------+--------+---------+------+---------+---------+ bridge-domain bd1
(config-leaf-if)# no shut 102 4403.a77a.547c dynamic - F F po8 vrf member v1
-------------------------------------------------------------------------------- exit

Searching the MAC Showing the configuration for

Configuring port channels
address table in leaf switches a tenant and leaf
Cisco NX-OS Style of CLI Modes

NXOS Config Mode

config terminal apic1(config)#
Exec Mode
(default)
Configuration commands
apic1#

All Exec and Bash Mode

Show commands bash
apic1:~>
Current commands
Overview

• The APIC NX-OS style of

CLI reuses the exact same
Cisco® NX-OS style REST API as used by the
APIC NX-OS APIC
GUI.
of CLI runs on CLI GUI
the Cisco APIC, • The show version and
not on the leaf and show running commands
are back (you can view the APIC REST API
spine switches.
entire running
configuration).
Show Version - Output
azesulem$ ssh admin@172.31.218.86
Application Policy Infrastructure Controller
admin@172.31.218.86's password:
apic1#

apic1# show version

Role Id Name Version
---------- ---------- ------------------------ --------------------
controller 1 apic1 1.2(0.245)
controller 2 apic2 1.2(0.245)
controller 3 apic3 1.2(0.245)
leaf 101 leaf1 n9000-11.2(0.83)
leaf 102 leaf2 n9000-11.2(0.83)
leaf 103 leaf3 n9000-11.2(0.83)
spine 104 spine1 n9000-11.2(0.83)
spine 105 spine2 n9000-11.2(0.83)
Tenant, Bridge Domain - Configuration
• Use CLI with Cisco® NX-OS look and feel to create tenants, VRF instances, and bridge domains.
• Use CLI to enable distributed anycast gateway for the bridge domain.

apic1# config terminal

apic1(config)#
apic1(config)# tenant test-tenant-cli
apic1(config-tenant)# vrf context vrf-cli
apic1(config-tenant)# bridge-domain BD-1
apic1(config-tenant-bd)# vrf member vrf-cli
apic1(config-tenant-bd)# unicast routing
apic1(config-tenant-bd)# arp flooding

apic1(config-tenant)# interface bridge-domain BD-1

apic1(config-tenant-interface)# ip address 7.7.7.1/24
Basic GUI
• The Basic GUI mode shows only the most commonly used features and emphasises ease of use.
• Some features are simply not exposed: L4-L7 integration, advanced routing (L3Out), etc.
Purpose of the Basic GUI
With the Cisco ACI 1.2 release, Release 1.2(x), Cisco ACI™ introduces an
alternative user interface to the existing GUI.
The goals of this GUI are as follows:
Reduce the time needed for deployment:
• Shorten the time needed to test Cisco ACI
• Provide ease of use in implementing Cisco ACI

Reduce the need for new learning:

• Provide network engineers with configurations based on current and traditional networking concepts (ACLs, VLANs,
subnets, etc.) as much as possible

Switching back and forth between the Advanced and Basic GUIs is
not recommended.
Address the markets for specific customers:
• Provide a tool for commercial customers
• Simplify the most common operations
Drag and Drop Configuration
For both Advanced and Basic GUIs
Drag-and-drop configuration is available
for the following features:
• EPGs
• Attributed-based EPGs
• Association of EPG with VMM and
physical domain
• Contracts
• External EPG for L2Out
• External EPG for L3Out
Simplified Interface Configuration
• One place to configure everything related to interface
• Creation of port channels and virtual port channels (vPCs)
• Interface-level configuration: speed, link debounce, LLDP, and Cisco® Discovery Protocol
• Layer 2 protocol
• VLAN and VMM domain association
 Configuration Rollback
Diff Tool
• Deleted managed objects
You can use are re-created.
configuration rollback • A special REST API is
• Created managed objects
to undo the changes available that shows the
are deleted.
made between two differences between
• Modified managed two snapshots:
snapshots. Objects
objects are reverted to their apichost/mqapi2/snapshots.diff.xml?s1d
are processed n=SNAPSHOT_ONE_DN&s2dn=SNAPS
prior state.
as follows: HOT_TWO_DN
Remote archives are
not supported.
Configuration Rollback
Configuration Rollback
• Snapshot for whole fabric / per tenant basis. You can create snapshots manually or periodic
• Example shows the difference between 2 snapshots at fabric level

Undo just
Diff the Diff
Endpoint Tracker
Traffic Map
Help visualise and quickly spot high traffic density and underutilized
nodes in the Cisco ACI™ fabric.

A grid is presented with a list of node IDs or vPC pairs on each axis.
Traffic flow between a given pair of nodes or between a vPC pair is
presented using colour-coded cells on the heat map.

Traffic density is presented in a range of colours, from lightest (yellow),

to shades of orange, to red (highest). Traffic statistics are collected
using atomic counters.

• You can order by name or by traffic.

• Traffic can be seen by:
- Sent packets
- Received packets
- Dropped packets
- Excess packets
Service Insertion Overview
Any form factor of L4-L7
Virtual Appliance Chassis

Manual (legacy)
• Manual traffic steering Manual
• Manual L4-L7 configuration
Any L4-L7 Vendor

Managed Mode
ACI Fabric
• Automated traffic steering
• Automated L4-L7 configuration
Fully Automated
Growing list of Eco system L4-
L7 Partners with Device
Package

Unmanaged Mode Semi Automated

• Automated traffic steering
Any L4-L7 Vendor
• Manual L4-L7 configuration
Service Insertion for Any Layer 4-7 Device
(No device package)
Description
• Unmanaged L4-L7 devices to be used as service node in a service graph between EPGs.
• This approach allows the network team to handle the network automation part for the service devices with Cisco® APIC. However, configuration
and management can continue to follow their current model.
• This approach also helps those L4-L7 devices for which a device package is not available.

1: Configure Cisco ACI® fabric for

L4-L7 service appliance – network
part only.

2: Administrator configures L4-L7

service appliance in the usual way
(CLI or GUI).
L4-L7 Admin
Service Graph with “Unmanaged” Device

UI hides all other

settings related to the
package, configuration
parameters, and
connectivity when the
managed mode is
not selected.
Micro-Segmentation
• Micro-Segmentation
• Regular EPG = Static Micro-Segment Micro-Segmentation ACI Status
EPG of a bridge-domain Release
VMWare + AVS 1.1(x) Shipping
• Dynamic EPG = Micro-Segment EPG Microsoft Hyper-V 1.2(1x) Shipping
using Attributes + contract (optional)
Multi-Hypervisor 1.2(1x) Shipping
• Attributes = VM attributes or VMWare DVS 1.2(2x) Shipping
Networking attributes such as IP, Intra-EPG Isolation 1.2(2x) Shipping
MAC
Intra-EPG Isolation + Micro- 1.2(2x) Shipping
• Key use cases: Segmentation
1. Quarantine (i.e. no EPG contract)
2. Micro-Segments (with contract policy)
Intra-EPG Isolation
• 1.2(2x) release added Intra-EPG
Isolation support
1. VMware DVS (i.e. AVS not
required)
Intra-EPG
2. Bare Metal
Segmentation
• When Intra-EPG isolation is
enabled “ALL” endpoints in EPG
are isolated
• (All Intra-EPG Isolation endpoints
must be in the same EPG)
• Can isolate Physical and Virtual
endpoint in same EPG
Troubleshooting
Troubleshooting Tools
• Traditionally CLI is our friend in most cases. But
it’s used for box-by-box

• Now in a distributed system where we have a

large number of devices connected together,
individual CLI will be tough to use.
 APIC Management Information Model Reference

From APIC GUI

https://<apic>/doc/html
 Visore
Object Browser – Web based MO Query Tool

fabricNode
https://<IP>/visore.html adSt on
childAction

delayedHeartbeat no
dn topology/pod-1/node-101

fabricSt active

id 101

lcOwn local

modTs 2015-04-08T14:38:44.546+02:00

model N9K-C9396PX

monPolDn uni/fabric/monfab-default

name bdsol-9396px-02
<?xml version="1.0" encoding="UTF-8"?><imdata totalCount="1"><fabricNode
adSt="on" childAction="" delayedHeartbeat="no" dn="topology/pod-1/node-101" role leaf

fabricSt="active" id="101" lcOwn="local" modTs="2015-04-08T14:38:44.546+02:00" serial SAL18CLUS15

model="N9K-C9396PX" monPolDn="uni/fabric/monfab-default" name="bdsol-9396px- status
02" role="leaf" serial="SAL18CLUS15" status="" uid="0" vendor="Cisco Systems, Inc" uid 0
version=""/></imdata>
vendor Cisco Systems, Inc

version

icurl 'http://apic/api/node/class/fabricNode.xml?query-target-filter=and(eq(fabricNode.id,"101"))' 20
API Inspector
Enables to see RESTAPI calls from APIC WebUI

82

admin@apic1> moquery -d "/topology/HDfabricOverallHealth5min-0"

Total Objects shown: 1
# fabric.OverallHealthHist5min
index : 0
childAction :
cnt : 31
dn : /topology/HDfabricOverallHealth5min-0
healthAvg : 82
healthMax : 82
healthMin : 82
healthSpct : 0
healthThr :
healthTr : 0
lastCollOffset : 310
modTs : never
repIntvEnd : 2015-04-10T19:24:03.530+01:00
repIntvStart : 2015-04-10T19:18:53.442+01:00
rn : HDfabricOverallHealth5min-0
status :
Prefer JSON or XML instead of text in moquery?
-> no problem 18

just specify “–o json” or “-o xml” with moquery

Troubleshooting Overview
• We have developed “visibility & troubleshooting” sub-section under operations
which will help in guided troubleshooting.
• In this section will walkthrough key areas flow and discuss only 2 sections end-
to-end.
• Fabric Discovery Process
• Fabric upgrade / downgrade
• Management (Out-of-Band and In-band)
• Virtualisation Integration
• Fabric Connectivity
• External Networking
• Layer 2
• Layer 3
• Layer4-7 Services Insertion
 Fabric-wide monitoring
Statistics Faults Diagnostics

Thresholds

Faults,
Health Scores

Troubleshooting, Drill Downs

Stats
Atomic
Counters
Fabric
Traceroute SPAN
On-Demand
Diagnostics
ELAM …
 Visibility and Troubleshooting - Wizard

1 2

0 define session name 3

1 select end point 1
2 select end point 2
We define session name and select End Points we’d like to troubleshoot visually
3 start

92
Example connectivity diagram generated for the selected
two end points.

We can further select info for particular datapath

93
APIC Logs Switch Logs
• /var/log/dme/log • /var/log/dme/log
• /var/log/dme/oldlog • /var/log/dme/oldlog

admin@apic1:~> cd /var/log/dme/log admin@apic1:~> cd /var/log/dme/log

admin@apic1:log> ls –altr * admin@apic1:log> ls –altr *
admin@apic1:log> ls –al svc_ifc_policymgr.* admin@apic1:log> ls -al svc_ifc_policyelem.*
…
Fabric Discovery Process
• In this section will walkthrough key areas flow only:

• LLDP Neighbour Discovery

• DHCP - TEP IP address assigned to the node
• Node software upgraded
• Policy Element session between APIC and Node
Fabric Discovery Process
LLDP Neighbour Discovery
• Verify LLDP neighbours – show lldp neighbours.

leaf101# show lldp neighbors

Capability codes:
(R) Router, (B) Bridge, (T) Telephone, (C) DOCSIS Cable Device
(W) WLAN Access Point, (P) Repeater, (S) Station, (O) Other
Device ID Local Intf Hold-time Capability Port ID
apic1 Eth1/1 120 90:e2:ba:4b:fa:d4
Fabric Discovery Process
LLDP Neighbour Discovery (Cont.)
• Verify state of the port and any errors / mismatch.

• If APIC is not present and no neighbours shown. Please check lldp process.
• show processes | grep lldp
leaf101# show processes | grep lldp
5619 S 41a497e7 1 - lldp

• Confirm cable connection of leaf or to the APIC

• LLDP in/out of the APIC interface can be confirmed with the command – show lldptool in/out <intf>
• Use the ip link show command to verify the ports of the bond interface
• show lldptool command will display the details of the TLV fields in the lldp updates
Fabric Discovery Process
DHCP – TEP IP Address assigned to the node
• Verify the allocated address and its pingable from APIC – acidiag fnvread.

admin@apic1:~> acidiag fnvread

ID Name Serial Number IP Address Role State LastUpdMsgId
-------------------------------------------------------------------------------------------------
101 leaf101 SAL17267Z9U 10.0.192.64/32 leaf active 0
102 leaf102 SAL1733B948 10.0.192.92/32 leaf inactive 0x200000018261b
201 spine201 FGE173400AK 10.0.192.94/32 spine active 0
202 spine202 FGE17420181 10.0.192.93/32 spine active 0

Total 4 nodes
Fabric Discovery Process
DHCP – TEP IP Address assigned to the node (Cont.)
• If node doesn’t ping then verify LLDP. Also check these:
• show dhcp internal info client – verify the client information present
• show ip route vrf overlay-1 – verify the infra routing is working

• On APIC you can verify the dhcp process:

• ps – ef | grep dhcpd
Fabric Discovery Process
DHCP – TEP IP Address assigned to the node (Cont.)
leaf1# cat /mit/sys/summary
• If node doesn’t ping and lldp is correct. # System
address : 10.0.152.95
Check on the state on node: childAction :
configIssues :
currentTime : 2016-03-08T01:53:20.852+00:00
• /mit/sys/summary dn : sys
fabricId : 1
fabricMAC : 00:22:BD:F8:19:FF
• in-service: node has an IP address and is id : 101
inbMgmtAddr : 0.0.0.0
running the configured firmware inbMgmtAddr6 : 0.0.0.0
lcOwn : local
• out-of-service: node doesn’t have ip address modTs
mode
: 2016-03-05T23:53:48.313+00:00
: unspecified
monPolDn : uni/fabric/monfab-default
• invalid-ver: switch has detected that it is not name : leaf1
oobMgmtAddr : 0.0.0.0
running the software that the node oobMgmtAddr6 : 0.0.0.0
configuration file is configured podId :1
rn : sys
role : leaf
serial : SAL1832Y6R4
state : in-service
 Virtualisation Integration
• Hypervisors: VMware, Microsoft and KVM
• VMware supported versions: 5.1, 5.5 and 6.x
• Need to have proper vCenter credentials.
• APIC and vCenter can communicate. If inband management then need a
contract.
• Make sure management policy is associated.
• Common issues in VMM integration:
1. Inventory doesn’t get populated or controller state shows ‘unknown / offline’
2. Virtual Distributed Switch not present in vCenter.
3. Port-group not present in vCenter.
4. Virtual Machine is not able to ping the leaf (gateway).
 Cisco ACI – Hypervisor Integration (vDS)
5 Application Network Profile
EPG EPG EPG
Create Application F/W WEB L/B APP DB
APIC Policy
APIC Admin

9 ACI
Fabric
Push Policy
1 (Lazy)
Cisco APIC connects to 6
VMware vCenter: 4 APIC learns the
• Initiates vDS creation Map EPG To Port location of ESX
• Collects VM, HV inventory Groups and Host through
• Creates a persistent initiate port- LLDP
connection to listen for group creation
2 Create VIRTUAL DISTRIBUTED SWITCH
vCenter events VDS WEB PORT APP PORT
DB PORT GROUP
GROUP GROUP
vCenter 7 Create Port
Server Groups
We
App DB Web Web DB
vCenter admin VI/Server admin b
VI/Server Admin 8 3
Instantiates VMs, Attaches
Assign to Port Groups Hypervisor to HYPERVISOR HYPERVISOR
VDS
 High-Level Workflow
Fabric Tenant
VMM Domain

Switch Profile EPG

Switch Selector

Interface Profile
Attachment
Entity Profile

Interface Policy Group

Dynamic VLAN pool
Scenario – Policy present on the leaf
• Virtual Machine is not able to ping the leaf (gateway)
• VM is present under EPG (means EPG policy is downloaded to the leaf)
• Verify lifecycle.
• Verify VM and vNIC is in operational state.
• Verify that there are no faults under EPG e.g. ‘invalid-path’, ‘invalid-vlan’
• Follow the data path debugging.
References
• Operating Cisco Application Centric Infrastructurehttps://learningnetwork.cisco.com/docs/DOC-
27047

• Troubleshooting Cisco Application Centric Infrastructurehttps://aci-troubleshooting-

book.readthedocs.org/en/latest/

• Cisco Application Centric Infrastructure

Fundamentalshttp://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/1-x/aci-fundamentals/b_ACI-
Fundamentals.html

• Cisco Application Centric Infrastructure Design

Guidehttp://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-
infrastructure/white-paper-c11-731960.html

• Cisco APIC Troubleshooting

Guidehttp://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/1-
x/troubleshooting/b_APIC_Troubleshooting.html
Q&A
Complete Your Online Session Evaluation
Give us your feedback and receive a
Cisco 2016 T-Shirt by completing the
Overall Event Survey and 5 Session
Evaluations.
– Directly from your mobile device on the Cisco Live
Mobile App
– By visiting the Cisco Live Mobile Site
http://showcase.genie-connect.com/ciscolivemelbourne2016/
– Visit any Cisco Live Internet Station located
throughout the venue
Learn online with Cisco Live!
T-Shirts can be collected Friday 11 March Visit us online after the conference
for full access to session videos and
at Registration presentations.
www.CiscoLiveAPAC.com
Thank you