Brkarc 2016

#CLUS
Troubleshooting
Cisco DNA SD-
Access from API and
Maglev
Parthiv Shah, Technical Leader, Escalation
Akshay Manchanda, Technical Leader, TAC
BRKARC-2016
#CLUS
Agenda
• Cisco DNA Architecture Overview
• Maglev Based Troubleshooting
• Installation/Services Debugging
• Log Collection
• ISE and DNA-Centre Integration
• Device Discovery/Provisioning
• API Based Troubleshooting

• How to Access
• Problem and Solution
#CLUS BRKARC-2016 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Agenda
• Cisco DNA Architecture Overview
• Maglev Based Troubleshooting
• Installation/Services Debugging
• Log Collection
• ISE and DNA-Centre Integration
• Device Discovery/Provisioning
• API Based Troubleshooting

• How to Access
• Problem and Solution
Cisco Webex Teams
Questions?
Use Cisco Webex Teams to chat
with the speaker after the session
How
1 Find this session in the Cisco Live Mobile App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space
Webex Teams will be moderated cs.co/ciscolivebot#BRKARC-2016

by the speaker until June 16, 2019.
#CLUS © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
Objectives and Assumptions
Objectives
After completing this module you will:
• Understand the Basic DNA Architecture Overview
• Understand Cisco DNAC Maglev Based Troubleshooting
• Understand Cisco DNAC API Based Troubleshooting
Assumptions
Audience must be familiar with
• Working knowledge of APIC-EM and PKI.
• Working knowledge of Routing/Switching and Cisco Fabric architecture.
• This session will not cover Cisco Fabric or ISE troubleshooting.
Cisco DNA
Architecture
Overview
The Cisco DNA Center Appliance
Fully Integrated Automation & Assurance
• Centralized Deployment - Cloud Tethered
• Built-In Telemetry Collectors (FNF, SNMP, Syslog, etc)
• Built-In Contextual Connectors (ISE/pxGrid, IPAM, etc)
• Multi-Node High Availability (3 Node, Automation)
• RBAC, Backup & Restore, Scheduler, APIs
Cisco DNA Center Platform
1RU Server (Small form factor)
DN2-HW-APL •
•
UCS 220 M5S: 64-bit x86
vCPU: 44 core (2.2GHz) / 56C / 112C
DNAC 1.2 Scale: Per Node • RAM: 256GB DDR4
• 5,000 Nodes (1K Devices + 4K APs) • Control Disks: 2 x 480GB SSD RAID1
• 25,000 Clients (Concurrent Hosts)
• System Disks: 6 x 1.9TB SSD M-RAID
DNAC 1.3 Scale: Per Node • Network: 2 x 10GE SFP+
• Please refer DNAC 1.3 Data Sheet • Power: 2 x 770W AC PSU
Single Appliance for Cisco DNAC (Automation + Assurance)

Cisco DNA Solution Cisco DNA Center
Cisco Enterprise Portfolio Simple Workflows
DESIGN PROVISION POLICY ASSURANCE
Cisco DNA Center

Identity Services Engine Network Control Platform Network Data Platform
Routers Switches Wireless Controllers Wireless APs
Cisco DNA Center
Cisco SD-Access – Key Components
ISE Appliance Cisco DNA Center

Appliance
API Cisco DNA Center API
Design | Policy | Provision | Assurance
API
Cisco& ISE
Identity Policy Automation
NCP Assurance
NDP
API API
Identity Services Engine Network Control Platform Network Data Platform
NETCONF
SNMP
SSH
AAA
RADIUS
EAPoL
Fabric HTTPS
NetFlow
Syslogs
Cisco Switches | Cisco Routers | Cisco Wireless
Cisco DNA Center and ISE integration
Identity and Policy Automation
Cisco Identity Services Engine
Authentication Groups and

Authorisation Policies
Policies
pxGrid
Campus Fabric REST APIs
Fabric Policy
Management Authoring
Workflows
Cisco DNA Center
Cisco DNA Center and ISE integration
ISE node roles in SD-Access
Admin/Operate
Cisco DNA Center
Devices REST pxGrid
Things
Config Sync Context
ISE-PSN ISE-PAN ISE-PXG

Users
Authorisation Policy Exchange Topics
TrustSecMetaData
If Employee then VN/SGT-10
SGT Name: Employee = SGT-10
Network
Users SGT Name: Contractor = SGT-20
Devices If Contractor then VN/SGT-20 ...
SessionDirectory*
If Things then VN/SGT-30
Bob with Win10 on CorpSSID
ISE-MNT
Cisco DNA Center Solution Basic Pre-requisite
• Hardware
• Supported Cisco DNA Center Appliance (DN2-HW-APL / DN2-HW-APL-L / DN2-HW-APL-XL)
• Supported switch/router/WLC/AP models
• Software
• Check various platform for recommended IOS-XE software version
• Check License for planned platforms
• Recommended ISE and Cisco DNA Center software
• Underlay/Overlay
• IP address plan for Cisco DNA Center and ISE
• Check for underlay network / routing configured correctly and devices are reachable
• Reachability to Internet – Direct or Proxy connection
• Access to an NTP server
• Make sure Cisco DNA Center appliance is close to real time using CIMC
Cisco DNA Center
Troubleshooting
Cisco DNA Center
SD-Access 4 Step Workflow
Design Provision Policy Assurance

Prov i s i on Assure
Assure Assure Assure
• Global Settings • Fabric Domains • Virtual Networks • Health Dashboard

• Site Profiles • CP, Border, Edge • ISE, AAA, Radius • 360o Views
• DDI, SWIM, PNP • FEW, OTT WLAN • Endpoint Groups • FD, Node, Client
• User Access • External Connect • Group Policies • Path Traces
Planning & Preparation

Installation & Integration
Cisco DNA Center – Maglev Logical Architecture
App Stack 1 App Stack 2 App Stack N
APIs, SDK & Packaging APIs, SDK & Packaging

Standards Standards
Maglev Services
IaaS
(Baremetal, ESXi, AWS, OpenStack etc)
Cisco SD-Access (Fusion) Package Services
Trap events, host discovery we leverage ipam-service IP Address manager
apic-em-event-service
snmp traps so they are handled here.
Critical during Provisioning
apic-em-inventory- Provides communication service between network-orchestration-service
orchestation.
manager-service inventory and discovery service
orchestration-engine-service Orchestration Service
Certificate authority and enables controller
apic-em-jboss-ejbca
authority on the DNAC.
pnp-service PNP Tasks
apic-em-network- Configure devices. Critical service to check
programmer-service during provisioning. policy-analysis-service Policy related
apic-em-pki-broker-
PKI Certificate authority policy-manager-service Policy related
service
command-runner- Responsible for Command Runner related Core database management
postgres
service task system
distributed-cache- rbac-broker-service RBAC

Infrastructure
service
sensor-manager Sensor Related
dna-common-service DNAC-ISE integration task
site-profile-service Site Profiling
dna-maps-service Maps Related services Core service during Provisioning
spf-device-manager-service
phase
dna-wireless-service Wireless Core service during Provisioning
spf-service-manager-service
phase
identity-manager-
DNAC-ISE integration task swim-service SWIM
pxgrid-service
Assurance Services Base Services
cassandra Database cassandra Core Database
collector-agent Collector Agents
catalogserver Local Catalog Server for update
collector-manager Collector Manager
elasticsearch Search elasticsearch Elastic Search Container
ise ISE data collector glusterfs-server Core Filesystem
kafka Communication service
identitymgmt Identity Managenent container
mibs-container SNMP MIBs
netflow-go Netflow data collector influxdb Database
pipelineadmin kibana-logging Kibana Logging collector
pipelineruntime-jobmgr
kong Infrastructure service
pipelineruntime-taskmgr
pipelineruntime-taskmgr maglevserver Infrastructure
pipelineruntime- mongodb Database
taskmgr-data
pipelineruntime- rabbitmq Communication service
taskmgr-timeseries Various Pipelines and Task nanager
workflow-server
snmp SNMP Colelctor
syslog Syslog Collector workflow-ui
trap Trap Collector workflow-worker Various Update workflow task
Most Commonly Used Maglev CLI
$ maglev $ magctl
Usage: maglev [OPTIONS] COMMAND [ARGS]... Usage: magctl [OPTIONS] COMMAND [ARGS]...
Tool to manage a Maglev deployment Tool to manage a Maglev deployment
Options:
Options:
--version Show the version and exit.
--version Show the version and exit. -d, --debug Enable debug logging
-d, --debug Enable debug logging --help Show this message and exit.
-c, --context TEXT Override default CLI context
--help Show this message and exit.
Commands: Commands:
backup Cluster backup operations api API related operations
appstack AppStack related operations
catalog Catalog Server-related management operations completion Install shell completion
completion Install shell completion disk Disk related operations
context Command line context-related operations glusterfs GlusterFS related operations
cronjob Cluster cronjob operations iam Identitymgmt related operations
job Cluster job operations job Job related operations
login Log into the specified CLUSTER logs Log related operations
logout Log out of the cluster maglev Maglev related commands
node Node related operations
maintenance Cluster maintenance mode operations service Service related operations
managed_service Managed-Service related runtime operations tenant Tenant related operations
node Node management operations token Token related operations
package Package-related runtime operations user User related operations
restore Cluster restore operations workflow Workflow related operations
service Service-related runtime operations
system System-related management operations
system_update_addon System update related runtime operations
system_update_package System update related runtime operations
Collecting Integrating
Logs ISE
Bring-up
Issues
Provisioning Discovery
Issues Issues
Cisco DNA Center Services are not coming up
Have Patience
120 to 180 minutes bring-up time
• Check network connectivity

• Check NTP/DNS server reachability
• Check any specific service not coming up
• During install or update use GUI
Avoid console login or don’t run

any system related commands
Install Failure
If you are unable to run maglev/magctl commands after install:
• Check RAID configuration and install error messages
• USB 3.0 is recommended for installation.
• Avoid KVM and/or USB 2.0 or NFS mount method for installation
• Use Windows 10 or Linux/Mac based system to build burn ISO image.
• Check for Error or Exception in following log files:
• /var/log/syslog
• /var/log/maglev_config_wizard.log
Package Status – GUI / CLI
How to Check Package Status from GUI How to Check Package Status from CLI
System Settings  App Management: Packages & Updates maglev package status
System Settings  Software Updates  Installed Apps
Check for any status
not “DEPLOYED”
Check for
“Failed”
Verify H/W profile complies with requirements
Verify sufficient disk and memory available
Verify number of CPUs to be minimum 88

and minimum memory is 256 GB.
Check Health Status of Cisco DNAC Cluster
Should show
Result as
SUCCESS
(Continued)
Troubleshooting – Kubernetes & Docker
Docker health check

The "Active" line should
show as "running".
Package Update
Package Update Troubleshooting
2-Step Update Process – System Update and Application Package update
Fail to Download Packages:
• Check connectivity to Internet
• During update download internet connectivity is mandatory
Fail to install packages:
• During install internet connectivity is mandatory
• Check if there is any failure displayed in GUI
• Check the status from CLI if there is any error
Package Update Ordering
https://www.cisco.com/c/en/us/td/docs/cloud-systems-management/network-automation-and-
management/dna-center/1-1/rn_release_1_1_2_2/b_dnac_release_notes_1_1_2_2.html#task_nj3_nww_qcb
Proxy Setting check If Proxy server
configured then
check for Proxy
server
Check Parent
Catalog server and
Repository
System Update Check
maglev system_updater update_info
Failure Output
Displays the
current and new
version
Failure State and

Sub-state
Progress Percentage
To Check the live log during update

$ magctl service logs -rf system-updater | lql
$ magctl service logs -rf workflow-worker | lql
Package Mapping – GUI v/s CLI
CLI Package Name GUI Display Name GUI Display Name CLI Package Name
application-policy Automation - Application Policy Automation - Application Policy application-policy
assurance Assurance - Base Assurance - Base assurance
automation-core NCP - Services Assurance - Path Trace path-trace
base-provisioning-core Automation - Base Assurance - Sensor sensor-automation
command-runner Command Runner Automation - Base base-provisioning-core
core-network-visibility Network Controller Platform Automation - Device Onboarding device-onboarding
device-onboarding Automation - Device Onboarding Automation - Image Management image-management
image-management Automation - Image Management Automation - SD Access sd-access
iwan IWAN Automation - Sensor sensor-automation
migration-support Automation - WAAS waas
ncp-system NCP - Base Command Runner command-runner
ndp Network Data Platform IWAN iwan
ndp-base-analytics Network Data Platform - Base Analytics NCP - Base ncp-system
ndp-platform Network Data Platform - Core NCP - Services automation-core
Ndp-ui Network Data Platform - Manager Network Controller Platform core-network-visibility
Network-visibility Network Controller Platform Network Controller Platform Network-visibility
path-trace Assurance - Path Trace Network Data Platform ndp
sd-access Automation - SD Access Network Data Platform - Base Analytics ndp-base-analytics
system System Or Infrastructure Network Data Platform - Core ndp-platform
waas Automation - WAAS Network Data Platform - Manager Ndp-ui
sensor-automation Automation - Sensor System Or Infrastructure system
sensor-automation Assurance - Sensor migration-support
Package Deploy Failure and Recovery
$ maglev package status
maglev-1 [main - https://kong-frontend.maglev-system.svc.cluster.local:443]
NAME DEPLOYED AVAILABLE STATUS
-----------------------------------------------------------------------------------
network-visibility 2.1.1.60067 - UPGRADE_ERROR - maglev_workflow.workflow.exceptions.TaskCallableExecutionError:
(1516326117.1073043, 1516327147.0490577, 'TimeoutError', 'Timeout of 1020 seconds has expired while watching for k8s changes for apic-em-jboss-
ejbca ')
$ maglev catalog package display network-visibility | grep fq Find the package name
fqn: network-visibility:2.1.1.60067
$ maglev catalog package delete network-visibility:2.1.1.60067 Delete the package

Ok
$ maglev package undeploy network-visibility. Undeploy failed package – Don’t use it as it can be
Undeploying packages 'network-visibility:2.1.1.60067'
Package will start getting undeployed momentarily
destructive and can lose the database
$ maglev catalog package pull network-visibility:2.1.1.60067

Package pull initiated
Pull the package again
Use "maglev catalog package status network-visibility:2.1.1.60067" to monitor the progress of the operation
Once above steps completed, go to GUI and download the package again and install it.
Or you can use “maglev package deploy <>”
High Availability
High Availability(HA) Overview
• Minimize Downtime for Cisco DNAC Clsuter
• HA cluster consists of multiple nodes that communicate and share/replicate information to
ensure high system availability, reliability, and scalability
• Cisco DNAC HA is limited to 3 nodes (active  active).
• Can Handle maximum one node failure
• Components scaled as part of HA :
• Managed Service Addons: Rabbitmq, Kong, Cassandra DB, Mongo DB, Postgres DB, Glusterfs, Elastic search,
Minio
• Maglev Core Service Addons: Maglevserver, Identity Management, agent, fluent-es, keepalived, platform-ui
• K8S Components: kube-apiserver , etcd , calico, kube-controller-manager , kube-dns , kube-proxy , kube-
scheduler
Creation of 3 node cluster
Switch 1 Switch 2 Switch 3

Switch 1
Cisco Cisco Cisco

Cisco
DNAC1 DNAC2 DNAC3
DNAC1
Cluster nodes MUST be on the same version
To Configure node-2 point to first node-1 as first step of software install
Repeat the same for node-3 after node-2 completes installation
Redistribute services through System 360 enables the cluster to act as a single unit
Install Initial Cisco DNA Center Node
Kong
Fusion Services NDP Services
CatalogServer
MaglevServer DockerRegistry
WorkflowServer WorkflowWorker
GlusterFS MongoDB Kubernetes
RabbitMQ Cassandra Docker
Install Additional Cisco DNA Center Nodes
Kong Kong Kong
Fusion Services NDP Services
CatalogServer
MaglevServer DockerRegistry
WorkflowServer WorkflowWorker
GlusterFS MongoDB Kubernetes GlusterFS MongoDB Kubernetes GlusterFS MongoDB Kubernetes
RabbitMQ Cassandra Docker RabbitMQ Cassandra Docker RabbitMQ Cassandra Docker
Distribute Services
Kong Kong Kong
Fusion Services NDP Services Fusion Services NDP Services Fusion Services NDP Services
CatalogServer MaglevServer WorkflowServer DockerRegistry WorkflowWorker
GlusterFS MongoDB Kubernetes GlusterFS MongoDB Kubernetes GlusterFS MongoDB Kubernetes
RabbitMQ Cassandra Docker RabbitMQ Cassandra Docker RabbitMQ Cassandra Docker
Bringing up Cisco DNA Center 3 node cluster
• Always ensure the seed Cisco DNA Center node is up and running before
adding other cluster nodes
• After forming the cluster, make sure that all the nodes are in READY state
when you run ‘kubectl get nodes’ command from CLI.
• Enabling HA should only be done after confirming that the 3-node cluster
is successfully formed and operational with full stack deployed.
• DO NOT try to add two nodes in parallel i.e. add nodes sequentially.
Cisco DNA Center settings after second node install
Enable Service Distribution Not

showing up after the second node
is installed as HA requires 3
nodes.
Cisco DNA Center settings after third node install
Enable Service Distribution shows

up after the third node is installed
as HA requires 3 nodes.
Enabling HA using CLI

$ maglev service nodescale refresh
Scheduled update of service scale (task_id=afeca07f-5a87-410a-be48-3eef76b08db6)
Enable Service Distribution
Service Distribution happened
Check services on each node
Automation Behavior on node failure
Switch 1 Switch 2 Switch 3 Switch 1 Switch 2 Switch 3
Cisco Cisco Cisco Cisco Cisco Cisco

DNAC1 DNAC2 DNAC3 DNAC1 DNAC2 DNAC3
Node fails, automation services are automatically distributed
Current re-distribution takes 25 minutes (unplanned)
Node failure restore (RMA) will require re-distribution of services needs (25 minutes – can be planned outage)
Link failure - no significant delay in redistribution of services when link comes back up
Failure of two nodes will bring the cluster down
UI Notification on HA failure
Persistent notification of failure:

1. Node
2. Services
3. Interfaces
Node Failure UI Notifications
Node down notification
 2nd and 3rd node will form a quorum

 UI won’t be available till services are
distributed
Node Failure UI Notifications
Service Failure UI Notifications
Nodes are up but one or more

services are down
Service Failure UI Notifications
Some services are pending and

not ready
Cluster Link Failure Notifications
Node down Some services

showing status
as NodeLost
Cluster Link Came Up
Node down Banner changed from Node Lost to Services

temporarily Disrupted. When all the services
are up, this banner should go away also.
Cluster Link Came Up
Node Up Fully restored so banner gone
Network Link failure
No Impact but No Notifications
Remove a node from cluster (RMA use case)
• If a node in a one of the node in cluster is in failed state and is not recovering after
several hours, users should remove it from the cluster
by running CLI : $ maglev node remove <node_ip>
Gracefully removing a node

• If for any reason, customer want to remove one of the active nodes in cluster, use
the following steps:
• Move services on the given host another node by issuing:
$ maglev node drain <node_ip>
• Once all services are up and running, power down the node and remove it from the
cluster: $ maglev node remove <node_ip>
HA Commands Cheat Sheet
HA commands:
• maglev service nodescale status
• maglev service nodescale refresh
• maglev service nodescale progress
maglev service nodescale history
•
Check All 3 nodes available
• maglev node remove <node_ip>
• maglev node allow <node_ip>
• maglev cluster node display
Logs ISE
Bring-up
Issues
Issues Issues
UI Debugging from Browser
Use Browser Debugging mode to find out API or GUI related Errors
For Chrome/Firefox Browsers

• Enable Debugging mode by going to
Menu  More Tools  Developer mode
• Select Console from top menu
• For clarity clear existing log.
• Run the task from Cisco DNA Center GUI
• Capture the console screenshot to

identify API/Error details.
UI Debugging from Browser
Firebug is another Tool for debugging mode.
• Install Firebug add-on in Firefox Browser
• Enable Firebug add-on
• Launch Firebug and Go to Console
• Run the task and it will capture detailed API information and related operation
Post/Get Operation and API name Task Success / Fail Code
Live Log - Service
Log Files:
• To follow/tail the current log of any service:

magctl service logs –r -f <service-name>
EX: magctl service logs -r -f spf-service-manager-service
Note: remove -f to display the current logs to the terminal
• To get the complete logs of any service:

• Get the container_id using:
docker ps | grep <service-name> | grep -v pause | cut -d' ' -f1
• Get logs using: docker logs <container_id>
Check Service Log in GUI
Click on Kibana Icon
Click on Service Counts

Monitoring / Log Explorer / Workflow
System Settings  System360:  Tools
https://<dnacenter_ip>/dna/systemSettings
Cisco DNA Center’s Monitoring Dashboard
Monitoring Cisco DNA Center Memory, CPU &
Bandwidth
Check Service Log using Log Explorer
Log Messages
Changing Cisco DNA Center Logging Levels
How to Change the Logging Level
• Navigate to the Settings Page:  System Settings  Settings  Debugging Levels
• Select the service of interest
• Select the new Logging Level
• Set the duration Cisco DNA Center should
keep this logging level change
• Intervals: 15 / 30 / 60 minutes or forever
Required information to report an issue
• RCA file [Sun Feb 11 14:26:00 UTC] maglev@10.90.14.247 (maglev-master-1)
$ rca
• SSH to server using maglev user
===============================================================
ssh –p 2222 maglev@<dnacenter_ip_address> Verifying ssh/sudo access
===============================================================
• rca
[sudo] password for maglev: <passwd>
Done
mkdir: created directory '/data/rca'
• Generated file can be copied using scp/sftp from changed ownership of '/data/rca' from root:root to maglev:maglev
external server ===============================================================

Verifying administration access
scp –P 2222 ===============================================================
[administration] password for 'admin': <passwd>
maglev@<dnacenter_ip_address>:<rca_filename> User 'admin' logged into 'kong-frontend.maglev-
system.svc.cluster.local' successfully
Important : For 3-node Cluster RCA needs to capture
from all 3 node individually
===============================================================
RCA package created on Sun Feb 18 14:26:14 UTC 2018
===============================================================
• Error Screenshot from UI 2018-02-18 14:26:14 | INFO | Generating log for 'date'...
tar: Removing leading `/' from member names
• API Debug log using /etc/cron.d/
/etc/cron.d/.placeholder
browser debugging mode /etc/cron.d/clean-elasticsearch-indexes
/etc/cron.d/clean-journal-files
Logs ISE
Bring-up Issues
Issues Issues
Cisco DNA Center – ISE Integration
Administration  pxGrid Services
• Pxgrid service should be enabled on ISE.
• SSH needs to be enabled on ISE.
• Superadmin credentials will be used for trust establishment for SSH/ERS
communication. By default ISE Super admin has ERS credentials
• ISE CLI and UI user accounts must use the same username and password
• ISE admin certificate must contain ISE IP or FQDN in either subject name or SAN.
• DNAC system certificate must contain DNAC IP or FQDN in either subject name
or SAN.
• Pxgrid node should be reachable on eth0 IP of ISE from DNAC.
• Bypass Proxy for DNAC on ISE server
Cisco DNA Center – ISE Integration Workflow
After Trust establishment

Check the subscriber
status in ISE pxGrid
Offline, Pending approval, Online
Trust Status on Cisco DNA •Center
Identity source status: (Under System360)
• AAA server Status (Settings – Auth/Policy Server) • INIT
• INPROGRESS • Available/Unavailable (PxGRID state)
• ACTIVE • TRUSTED/UNTRUSTED
• FAILED
• RBAC_FAILURE
Troubleshooting
ISE - Cisco DNA Center Integration
Checking pxGrid service status

• Login to ISE server using SSH
• Run “show application status ise” to check for the services running.
Increasing log level to debug

• Go to Administration  Logging  Debug Log Config
• Select the ISE server and Edit
• Find pxGrid, ERS, Infrastructure Service from the list.
Click Log Level button and select Debug Level
Troubleshooting
On Cisco DNA Center check On ISE check logs

• network-design-service • ERS
• identity-manager-pxGrid-service • pxGrid
• Cisco DNA Center common-service • Infrastructure Service logs
Example Error:
2017-08-01 05:24:36,794 | ERROR | pool-1-thread-1 | identity-manager-pxGrid-service |
c.c.e.i.u.pxGridConfigurationUtils | An error occurred while retrieving pxGrid endpoint
certificate. Request: PUT https://bldg24-ise1.cisco.com:9060/ers/config/endpointcert/
certRequest HTTP/1.1, Response: HttpResponseProxy{HTTP/1.1 500 Internal Server Error
[Cache-Control: no-cache, no-store, must-revalidate, Expires: Thu, 01 Jan 1970 00:00:00 GMT,
Set-Cookie: JSESSIONIDSSO=9698CC02E88780EC4415A6DE80C37355; Path=/; Secure; HttpOnly, Set-
Cookie: APPSESSIONID=03A609099AD604812984C6DF27CF7A19; Path=/ers; Secure; HttpOnly, Pragma:
no-cache, Date: Tue, 01 Aug 2017 05:24:36 GMT, Content-Type: application/json;charset=utf-8,
Content-Length: 421, Connection: close, Server: ] ResponseEntityProxy{[Content-Type:
application/json;charset=utf-8,Content-Length: 421,Chunked: false]}} |
Troubleshooting
How To Capture ISE Log bundle:
• Go to Operation  Download Logs
• Select ISE server
• Select any additional log to be captured
• Select Encryption and create bundle
• Download bundle
Logs ISE
Bring-up Issues
Issues Issues
Step 1
Verify all devices are green after Discovery
Step 2
Check if all devices in Managed state
New Configuration after Discovery
FE250#show archive config differences flash:underlay system:running-config
!Contextual Config Diffs:
+device-tracking tracking
+device-tracking policy IPDT_MAX_10
+limit address-count 10
+no protocol udp
+tracking enable
+crypto pki trustpoint TP-self-signed-1978819505

+enrollment selfsigned
+subject-name cn=IOS-Self-Signed-Certificate-1978819505 New RSA Keys are created
+revocation-check none
+rsakeypair TP-self-signed-1978819505
+crypto pki trustpoint 128.107.88.241

+enrollment mode ra Secure connection to Cisco DNA Center
+enrollment terminal using the interface 1 IP address as the
+usage ssl-client certificate name
See Notes for Complete Configurations
Troubleshooting – Discovery/Inventory
• Check for IP address reachability from DNAC
to the device
• Check username/password configuration in
Settings
• Check whether telnet/ssh option is properly
selected
• Check using manual telnet/ssh to the
device from DNAC or any other client
• Check SNMP community configuration
matches on switch and DNA-C
• Discovery View will provide additional
information.
Services Involved on DNA:

apic-em-inventory-manager-service
Logs ISE
Bring-up Issues
Issues Issues
Verifying Config Push
• While Cisco DNA Center is evolving to use NETCONF and YANG APIs, at this
time it pushes most configuration by SSH.
• Exact configuration commands can be seen via show history all
FE2050#show history all
CMD: 'enable' 13:29:55 UTC Tue Jan 16 2018
CMD: 'terminal length 0' 13:29:55 UTC Tue Jan 16 2018
CMD: 'terminal width 0' 13:29:55 UTC Tue Jan 16 2018
CMD: 'show running-config' 13:29:55 UTC Tue Jan 16 2018
CMD: 'config t' 13:29:56 UTC Tue Jan 16 2018
CMD: 'no ip domain-lookup' 13:29:56 UTC Tue Jan 16 2018
CMD: 'no ip access-list extended DNA Center_ACL_WEBAUTH_REDIRECT' 13:29:57 UTC Tue Jan 16 2018
*Jan 16 13:29:57.023: %DMI-5-SYNC_NEEDED: Switch 1 R0/0: syncfd: Configuration change requiring
running configuration sync detected - 'no ip access-list extended DNA
Center_ACL_WEBAUTH_REDIRECT'. The running configuration will be synchronized to the NETCONF
running data store.
CMD: 'ip tacacs source-interface Loopback0' 13:29:57 UTC Tue Jan 16 2018
CMD: 'ip radius source-interface Loopback0' 13:29:57 UTC Tue Jan 16 2018
CMD: 'cts role-based enforcement vlan-list 1022' 13:29:57 UTC Tue Jan 16 2018
AAA Configuration
FE2050#show running-config | sec aaa
aaa new-model AAA server (ISE) is now
aaa group server radius dnac-group used to authenticate
server name dnac-radius_172.26.204.121 device logins
ip radius source-interface Loopback0
aaa authentication login default group dnac-group local
aaa authentication enable default enable
aaa authentication dot1x default group dnac-group
aaa authorization exec default group dnac-group local
aaa authorization network default group dnac-group
aaa authorization network dnac-cts-list group dnacs-group
aaa accounting dot1x default start-stop group dnac-group
aaa server radius dynamic-author

client 172.26.204.121 server-key cisco123
FE2050#show aaa servers

RADIUS: id 1, priority 1, host 172.26.204.121, auth-port 1812, acct-port 1813 AAA server up and
State: current UP, duration 546s, previous duration 0s running from IOSd
Dead: total time 0s, count 0
Platform State from SMD: current UNKNOWN, duration 546s, previous duration 0s
SMD Platform Dead: total time 0s, count 0
Global Cisco TrustSec (CTS) Configurations
Global AAA Configuration for all IOS Switches

aaa new-model
!
aaa authentication dot1x default group ise-group
TrustSec authorization should use cts-list AAA servers aaa authorization network default group ise-group
aaa authorization network cts-list group ise-group
cts authorization list cts-list aaa accounting dot1x default start-stop group ise-group
!
For SGT policy enforcement, if switch has to access control aaa server radius dynamic-author
client <Switch_IP> server-key cisco
cts role-based enforcement !
cts role-based enforcement vlan-list <VLANs> radius server ise
address ipv4 <ISE_IP> auth-port 1812 acct-port 1813
pac key <PAC_Password>
!
aaa group server radius ise-group
server name ise
!
ISE and ‘Network Device’ Transact Securely Using PAC keys
Switch authenticates with Cisco ISE for Secure EAP FAST Channel
Environmental Data Switch# cts credential id <device_id> password <cts_password>
TrustSec Egress Policy
RADIUS EAP FAST Channel RADIUS PAC* keys pushed by ISE. Switch uses this to talk to ISE securely
ISE IOS bldg24-edge-3650-1#show cts pacs
AID: 5079AA777CC3205E5D951003981CBF95
PAC-Info:
PAC-type = Cisco Trustsec
AID: 5079AA777CC3205E5D951003981CBF95
I-ID: FDO1947Q1F1
A-ID-Info: Identity Services Engine
Credential Lifetime: 15:30:58 PST Mon May 28 2018
PAC-Opaque:
000200B800010211000400105079AA777CC3205E5D951003981CBF950006009C0003
0100C25BAEC6DC8B90034431914E48C335DC000000135A95A90900093A8087E1E4
7B8EA12456005D6E38C41F69C19F86B884B370177982EB65469F1E5F6B2B6D96B7
1C99DA19B240FE080757F8F8BBD543AE830A5959EA4A999C310CE1FEC427213AA
552406796C8DDDA695DBCF08FB3473249DCC025598D27CD280E4D01E7877F14C6
F211CC3BAB5E3B836A6B42A9C5EE4E0E6F997549D10561
Refresh timer is set for 11w3d
Environmental Data
Switch# show cts environment-data
CTS Environment Data
====================
Current state = COMPLETE
Last status = Successful
Local Device SGT:
SGT tag = 2-00:TrustSec_Infra_SGT
Server List Info:
Installed list: CTSServerList1-0001, 1 server(s):
IOS *Server: 10.1.1.222, port 1812, A-ID 3E465B9E3F4E012E6AD3159B403B5004
ISE Status = DEAD
Security Group Name Table:
auto-test = TRUE, keywrap-enable = FALSE, idle-time = 60 mins, deadtime
= 20 secs 0-00:Unknown
Multicast Group SGT Table: 2-00:TrustSec_Infra_SGT
Security Group Name Table:
0-00:Unknown
10-00:Employee_FullAccess
2-00:TrustSec_Infra_SGT 20-00:Employee_BYOD
10-00:Employee_FullAccess 30-00:Contractors
20-00:Employee_BYOD
30-00:Contractors 100-00:PCI_Devices
100-00:PCI_Devices 110-00:Web_Servers
110-00:Web_Servers
120-00:Mail_Servers
120-00:Mail_Servers
255-00:Unregist_Dev_SGT 255-00:Unregist_Dev_SGT
Environment Data Lifetime = 86400 secs
Last update time = 21:57:24 UTC Thu Feb 4 2016
Env-data expires in 0:23:58:00 (dd:hr:mm:sec)
Env-data refreshes in 0:23:58:00 (dd:hr:mm:sec)
Cache data applied = NONE
State Machine is running
If CTS is not Configured, Verify the Device is a NAD
Configuration Configuration not pushed to the
Issues network device
Check
state?
Device should be Reachable and Managed
Debug Inventory Issue
% 10.9.3.0 overlaps with Vlan12
Fix the configuration on the device
(config)#no vrf definition Campus
Navigate to Device inventory
Select the device and click “Resync”
Loopback 0 If you are using Automated Underlay
skip this setup
interface Loopback0
ip address <> This is only required for Manual
Underlay configuration
ip router isis
Don’t forget to select the device and click “Resync”
SD-Access Fabric
Provisioning
Fabric Edge Configuration
LISP configuration
VRF/VLAN configuration
SVI configuration
Interface configuration
SDA Provisioning – Workflow
Services Involved Start Provisioning from UI
NB API Pre-Process-Cfs-Step Determine all the namespaces this config applies to
SPF Service Validate-Cfs-Step Validate whether this config is consistent and conflict free
Process-Cfs-Step
Persist the data and take snapshot for all namespaces
in a single transaction
Target-Resolver-Cfs-Step
Orchestration Determine the list of devices this config should go to
SPF Device
Engine Translate-Cfs-Step Per device convert the config to the config that needs to go to the device
Messaging
Deploy-Rfs-Task Convert the config to Bulk Provisioning Message to send it to NP
Network
Programmer Rfs-Status-Updater-
Task Update the Device config Status based on response from NP
Rfs-Merge-Step
Complete Update the task with an aggregate merged message
SDA Provisioning – Task Status Check
Click on View Target Device List
Click on Show task
Status Check the status
Click on See Details

VLAN and VRF Configuration
FE2050#show run | beg vrf FE2050#show run | sec vlan
vrf definition BruEsc ip dhcp snooping vlan 1021-1024
rd 1:4099 vlan 1021
! name 192_168_1_0-BruEsc
address-family ipv4 vlan 1022
route-target export 1:4099 name 192_168_100_0-BruEsc
route-target import 1:4099 vlan 1023
exit-address-family name 192_168_200_0-DEFAULT_VN
vrf definition DEFAULT_VN cts role-based enforcement vlan-list 1021-1023
rd 1:4099
!
address-family ipv4
route-target export 1:4099
route-target import 1:4099
exit-address-family
One VLAN per IP Address Pool

One VRF per VN DHCP Snooping and CTS are enabled
Closed Authentication Configuration
IBNS 2.0 Template Interface Configuration
template DefaultWiredDot1xClosedAuth FE2051#show run int gi 1/0/1
dot1x pae authenticator switchport mode access
switchport access vlan 2047 device-tracking attach-policy IPDT_MAX_10
switchport mode access authentication timer reauthenticate server
switchport voice vlan 4000 dot1x timeout tx-period 7
dot1x max-reauth-req 3
mab
source template DefaultWiredDot1xClosedAuth
access-session closed
spanning-tree portfast
access-session port-control auto
authentication periodic
authentication timer reauthenticate server
service-policy type control subscriber PMAP_ D
Troubleshooting – Device / Fabric Provision Issues
Services involved:
• orchestration-engine-service • spf-device-manager-service
• spf-service-manager-service • apic-em-network-programmer-service
Cisco SD-Access
Fabric
Troubleshooting
DHCP
DHCP Packet Flow in Campus Fabric
B DHCP
1 The DHCP client generates a
DHCP request and broadcasts it
on the network
FE1 BDR
1 2 FE uses DHCP Snooping to add
it’s RLOC as the remote ID in
Option 82 and sets giaddress the
2 Anycast SVI
Using DHCP Relay the request is
forwarded to the Border.
4 3 DHCP Server replies with offer
3
5 to Anycast SVI.
4 Border uses the remote ID in
option 82 to forward the packet.
5 FE installs the DHCP binding
and forwards the reply to client
DHCP Binding on Fabric Edge
FE#show ip dhcp snooping binding

MacAddress IpAddress Lease(sec) Type VLAN Interface
------------------ --------------- ---------- ------------- ---- --------------------
00:13:a9:1f:b2:b0 10.1.2.99 691197 dhcp-snooping 1021 TenGigabitEthernet1/0/23
FE#debug ip dhcp snooping ? Debug ip dhcp snooping

H.H.H DHCP packet MAC address Enables showing detail with regards to DHCP snooping
agent DHCP Snooping agent and the insertion of option 82 remote circuit
event DHCP Snooping event Debug ip dhcp server packet
packet DHCP Snooping packet Enables debug with regards to the relay function , insertion
redundancy DHCP Snooping redundancy giaddress and relay functionality to the Server
Debug dhcp detail
Adds additional detail with regards to LISP in DHCP debugs
Received DHCP Discover
015016: *Feb 26 00:07:35.296: DHCP_SNOOPING: received new DHCP packet from input interface
(GigabitEthernet4/0/3)
015017: *Feb 26 00:07:35.296: DHCP_SNOOPING: process new DHCP packet, message type: DHCPDISCOVER,
input interface: Gi4/0/3, MAC da: ffff.ffff.ffff, MAC sa: 00ea.bd9b.2db8, IP da: 255.255.255.255, IP
sa: 0.0.0.0, DHCP ciaddr: 0.0.0.0, DHCP yiaddr: 0.0.0.0, DHCP siaddr: 0.0.0.0, DHCP giaddr: 0.0.0.0,
DHCP chaddr: 00ea.bd9b.2db8, efp_id: 374734848, vlan_id: 1022
Adding Relay Information Option

015018: *Feb 26 00:07:35.296: DHCP_SNOOPING: add relay information option.
015019: *Feb 26 00:07:35.296: DHCP_SNOOPING: Encoding opt82 CID in vlan-mod-port format
015020: *Feb 26 00:07:35.296: :VLAN case : VLAN ID 1022
015021: *Feb 26 00:07:35.296: VRF id is valid
015022: *Feb 26 00:07:35.296: LISP ID is valid, encoding RID in srloc format
015023: *Feb 26 00:07:35.296: DHCP_SNOOPING: binary dump of relay info option, length: 22 data:
0x52 0x14 0x1 0x6 0x0 0x4 0x3 0xFE 0x4 0x3 0x2 0xA 0x3 0x8 0x0 0x10 0x3 0x1 0xC0 0xA8 0x3 0x62
015024: *Feb 26 00:07:35.296: DHCP_SNOOPING: bridge packet get invalid mat entry: FFFF.FFFF.FFFF,
packet is flooded to ingress VLAN: (1022)
015025: *Feb 26 00:07:35.296: DHCP_SNOOPING: bridge packet send packet to cpu port: Vlan1022.
Option 82 0x3 0xFE = 3FE = VLAN ID 1022 LISP Instance-id 4099 RLOC IP 192.168.3.98
0x4 = Module 4 , 0x3 = Port 3
Continue with Option 82
015026: *Feb 26 00:07:35.297: DHCPD: Reload workspace interface Vlan1022 tableid 2.
015027: *Feb 26 00:07:35.297: DHCPD: tableid for 1.1.2.1 on Vlan1022 is 2
015028: *Feb 26 00:07:35.297: DHCPD: client's VPN is Campus.
015029: *Feb 26 00:07:35.297: DHCPD: No option 125
015030: *Feb 26 00:07:35.297: DHCPD: Option 125 not present in the msg.
015032: *Feb 26 00:07:35.297: DHCPD: Sending notification of DISCOVER:
015033: *Feb 26 00:07:35.297: DHCPD: htype 1 chaddr 00ea.bd9b.2db8
015034: *Feb 26 00:07:35.297: DHCPD: circuit id 000403fe0403 Circuit ID
015035: *Feb 26 00:07:35.297: DHCPD: table id 2 = vrf Campus 0x3 0xFE = 3FE = VLAN ID 1022
015036: *Feb 26 00:07:35.297: DHCPD: interface = Vlan1022 0x4 = Module 4 , 0x3 = Port 3
015037: *Feb 26 00:07:35.297: DHCPD: class id 4d53465420352e30
Sending Discover to DHCP server Anycast Gateway IP address
015040: *Feb 26 00:07:35.297: DHCPD: Looking up binding using address 1.1.2.1

015041: *Feb 26 00:07:35.297: DHCPD: setting giaddr to 1.1.2.1.
015042: *Feb 26 00:07:35.297: DHCPD: BOOTREQUEST from 0100.eabd.9b2d.b8 forwarded to 192.168.12.240.
015043: *Feb 26 00:07:35.297: DHCPD: BOOTREQUEST from 0100.eabd.9b2d.b8 forwarded to 192.168.12.241.
Forwarding ACK
015089: *Feb 26 00:07:35.302: DHCPD: Reload workspace interface LISP0.4099 tableid 2.
015090: *Feb 26 00:07:35.302: DHCPD: tableid for 1.1.7.4 on LISP0.4099 is 2
015091: *Feb 26 00:07:35.302: DHCPD: client's VPN is .
015092: *Feb 26 00:07:35.302: DHCPD: No option 125
015093: *Feb 26 00:07:35.302: DHCPD: forwarding BOOTREPLY to client 00ea.bd9b.2db8.
015094: *Feb 26 00:07:35.302: DHCPD: Forwarding reply on numbered intf
015096: *Feb 26 00:07:35.302: DHCPD: Clearing unwanted ARP entries for multiple helpers
015097: *Feb 26 00:07:35.303: DHCPD: src nbma addr as zero
015098: *Feb 26 00:07:35.303: DHCPD: creating ARP entry (1.1.2.13, 00ea.bd9b.2db8, vrf Campus).
015099: *Feb 26 00:07:35.303: DHCPD: egress Interfce Vlan1022
015100: *Feb 26 00:07:35.303: DHCPD: unicasting BOOTREPLY to client 00ea.bd9b.2db8 (1.1.2.13).
015101: *Feb 26 00:07:35.303: DHCP_SNOOPING: received new DHCP packet from input interface (Vlan1022)
015102: *Feb 26 00:07:35.303: No rate limit check because pak is routed by this box
015103: *Feb 26 00:07:35.304: DHCP_SNOOPING: process new DHCP packet, message type: DHCPACK, input
interface: Vl1022, MAC da: 00ea.bd9b.2db8, MAC sa: 0000.0c9f.f45d, IP da: 1.1.2.13, IP sa: 1.1.2.1,
DHCP ciaddr: 0.0.0.0, DHCP yiaddr: 1.1.2.13, DHCP siaddr: 0.0.0.0, DHCP giaddr: 1.1.2.1, DHCP chaddr:
00ea.bd9b.2db8, efp_id: 374734848, vlan_id: 1022
Client Adding to Device Tracking
015104: *Feb 26 00:07:35.304: DHCP_SNOOPING: binary dump of option 82, length: 22 data:
0x52 0x14 0x1 0x6 0x0 0x4 0x3 0xFE 0x4 0x3 0x2 0xA 0x3 0x8 0x0 0x10 0x3 0x1 0xC0 0xA8 0x3 0x62
015105: *Feb 26 00:07:35.304: DHCP_SNOOPING: binary dump of extracted circuit id, length: 8 data:
0x1 0x6 0x0 0x4 0x3 0xFE 0x4 0x3
015106: *Feb 26 00:07:35.304: DHCP_SNOOPING: binary dump of extracted remote id, length: 12 data:
0x2 0xA 0x3 0x8 0x0 0x10 0x3 0x1 0xC0 0xA8 0x3 0x62
015107: *Feb 26 00:07:35.304: actual_fmt_cid OPT82_FMT_CID_VLAN_MOD_PORT_INTF global_opt82_fmt_rid
OPT82_FMT_RID_DEFAULT_GLOBAL global_opt82_fmt_cid OPT82_FMT_CID_DEFAULT_GLOBAL cid: sub_option_length 6
015108: *Feb 26 00:07:35.304: DHCP_SNOOPING: opt82 data indicates local packet
015109: *Feb 26 00:07:35.304: DHCP_SNOOPING: opt82 data indicates local packet
015117: *Feb 26 00:07:35.405: DHCP_SNOOPING: add binding on port GigabitEthernet4/0/3 ckt_id 0
GigabitEthernet4/0/3
015118: *Feb 26 00:07:35.405: DHCP_SNOOPING: added entry to table (index 1125)
015119: *Feb 26 00:07:35.405: DHCP_SNOOPING: dump binding entry: Mac=00:EA:BD:9B:2D:B8 Ip=1.1.2.13 Lease=21600
Type=dhcp-snooping Vlan=1022 If=GigabitEthernet4/0/3
015120: *Feb 26 00:07:35.406: No entry found for mac(00ea.bd9b.2db8) vlan(1022) GigabitEthernet4/0/3
015121: *Feb 26 00:07:35.406: host tracking not found for update add dynamic
Client Added to Device Tracking
(1.1.2.13, 0.0.0.0, 00ea.bd9b.2db8) vlan(1022)
015122: *Feb 26 00:07:35.406: DHCP_SNOOPING: remove relay information option.
015123: *Feb 26 00:07:35.406: platform lookup dest vlan for input_if: Vlan1022, is NOT tunnel, if_output:
Vlan1022, if_output->vlan_id: 1022, pak->vlan_id: 1022
015124: *Feb 26 00:07:35.406: DHCP_SNOOPING: direct forward dhcp replyto output port: GigabitEthernet4/0/3.
Available API’s and
DNA Platform
Troubleshooting
What is an API (Application Programmable Interface)?
What is Representational State Transfer
(RestFul) API’s
GET
POST
Application A Application B
PUT
DELETE
Data format of the payload is JSON (JavaScript Object Notation)
{ "title": "A Wrinkle in Time", "author": "Madeline L'Engle" }
Different Ways for Consuming API’s
DNA Center Platform as a Service
DNA Center API Tester
Native Restful Clients like RESTLET and POSTMAN
Native Scripting in any programming language like Python, Java, C
Method 1
DNA Center
Platform as a
Service
Enable the REST API bundle from DNA-Center
Enable the
REST API
bundle to start
REST calls
Access the API’s from the Developer Toolkit
List of available API’s
Discovery API Call from DNAC Platform
Get
Discovery by Make a REST
Index Range Call from DNA
GUI Itself
Discovery API Call from DNAC Platform (Cont)
{
"response": [
{
"name": "C9800-CL",
"discoveryType": "Range",
You can also "ipAddressList": "10.122.145.235-10.122.145.235",
"deviceIds": "36f02621-5b65-4c15-8374-8f9e5b1e72ee",
"userNameList": "admin",
do Discovery "passwordList": "NO!$DATA!$",
"ipFilterList": "",
"enablePasswordList": "NO!$DATA!$",
by ID "snmpRoCommunity": "",
"protocolOrder": "ssh",
"discoveryCondition": "Complete",
"discoveryStatus": "Inactive",
"timeOut": 5,
"numDevices": 1,
"retryCount": 3,
"isAutoCdp": false,
"globalCredentialIdList": [
"c39a97e7-54c1-4a4a-a9d8-15d0ec142f30",
"47658146-ed68-4208-b208-bb01060236b2"
],
"preferredMgmtIPMethod": "None",
"netconfPort": "830",
"id": "133"
}
],
"version": "1.0"
}
Method 2
Using DNA Center

API Tester
Get Device Provisioning Config
Part 1- find the Device ID
DNA API Tester URL:

https://<Cisco DNA Center IP Address>/dna/apitester
Copy the Device

ID to use in next
API Call
Get Device Provisioning Config
Part 2- Find the Provisioning Config Status Based on Device-ID and
flag for IsLatest: true
Method 3:
Using Native REST

Tools like
RESTLET
Authenticate by Generating a Token
https://developer.cisco.com/docs/dna-center/#!generating-and-using-an-authorization-token/generating-and-using-an-authorization-
token
{
"Token": "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiI1YjhlZmE4OGZjNGE5YjAwODlkZmM3ZDIiLCJhdXRoU291cmNlIjoiaW50ZXJuYWwiLCJ0ZW5hbnROYW1lIjoiVE5UMCIsInJvbG
VzIjpbIjViOGVmYTg2ZmM0YTliMDA4OWRmYzdkMSJdLCJ0ZW5hbnRJZCI6IjViOGVmYTg1ZmM0YTliMDA4OWRmYzdjZiIsImV4cCI6MTU1NzE3NDYwMywidXNlcm5 hbWUiOiJhZG1pbiJ9.JlkLC2ig-
DCdkqFEQ1wQjow4eaoYqi_ApfbEl8aMIhY"
}
Getting All VLAN ID’s in the Fabric and Underlay
https://<DNAC IP/FQDN>/dna/intent/api/v1/topology/vlan/vlan-names
Find the GET Request for pulling the Templates
Run the GET template API from the DNA Center
This is the URL that you’ll

define in postman to send
the GET request
Running a GET to Cisco DNA Center
Use inherit auth from

parent to send the actual
GET request.
Every template has a

templateId that you can
later use to query one
specific template
Verify the template from Cisco DNA Center
Getting the details of a template using API GET
This is the tempalteId we

before
This is giving us all the

details of the template
Creating a project using API
This is the URL you need to

send your POST to create
the project
This is the model that Cisco

DNA Center expects in the
POST. If you check the model
you’ll see if fields are
mandatory#CLUS
or optional. BRKARC-2016 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 129
Now you can send your post to

the URL you found using the
model schema described in
Cisco DNA Center
If successful, you’ll
see a tasked, URL and
a version number.
Project created in Cisco DNA Center
The project
is created You can check the template-
programmer logs to see how it was
created
Defining and running the API POST for a template
Verify what’s the URL you need to

send your POST to create the
template
Defining and running the API POST for a template
Send the POST using the variables

described in the method. Replace
${projectid} the real projectid
In this case I’m creating a template

named “postman-created-template”
that will be part of the project
“postman-template-name”
Verifying template under the defined project
How DNA Center
Uses API’s
Internally
Kong is the backend API Server for DNA Center
Inventory
SPF Service
Topology Service
Kong
Service
Check the API’s currently used by individual services
$ magctl appstack status | grep kong
$ magctl api routes | grep pool
Using Chrome
Developer Tools to
Troubleshoot
Issues
Launching Developer Tools on a Browser
Network API calls on Developer Tools
Checking a Specific API Call from Developer Tools
Saving the API requests for a particular session
Understanding
Certificates and
Common Issues hit
due to Improper
Certificates
Operations on DNA Center that makes use of
Certificates
Identity Services Engine(ISE)/ IP address Manager Integration (IPAM)
Software Image Management(SWIM) / Plug and Play (PnP)/Lan

Automation
Wireless Lan Controller for Assurance
Understanding the Key Fields of a Certificate
Who issued the certificate
To whom the certificate was issued
Alternate identities which the certificate is valid for
Understanding the Chain of Trust in Certificates
Verifying Certificate from Cisco DNA Center GUI
Step-3
1 Step-2
2
4
3
Step-1
Make sure all interface IP’s

and VIP are included in the
SAN field of the DNAC
certificate
Checking Certificate from Cisco DNA Center CLI
$ echo | openssl s_client -showcerts -servername <Cisco DNA Center IP Address> -connect <Cisco DNA Center IP Address>:443 2>/dev/null | openssl x509 -inform pem -
noout -text
1
2
3
How to Revert to Self Signed Certificate on DNA
Center
cd /home/maglev 1. Generate the certificate:
vi register.conf
sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -
[req]
keyout key.pem -out cert.pem -config request.conf -extensions
distinguished_name = req_distinguished_name
x509_extensions = v3_req 'v3_req’
prompt = no
[req_distinguished_name]
C = IN
ST = MH
L = Mumbai
2. Verification of the IP’s in the certificate:
O = CUSTOMER
OU = MyDivision
CN = DOMAIN openssl x509 -inform pem -text -noout -in cert.pem
[v3_req]
basicConstraints = critical, CA:TRUE
keyUsage = keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth Download the cert.pem and key.pem file from DNA Center and upload on the
subjectAltName = @alt_names
DNA Center
[alt_names]
IP.1 = X.X.X.X
IP.2 = 172.20
<esc>
:wq
Upload the Certificate on the DNA Center
Post Certificate Change Checks
ISE DNA Center Integration
WLC Assurance – Manually put the new DNA certificate on WLC
Pushing the New DNA Center Certificate on WLC
(Cisco Controller) >show network assurance summary • Login into the DNA Center SSH on Port 2222
Server url............................. https://xx.xx.xx.xx

Wsa Service............................ Enabled • Copy the token from the below command:
wsa Onchange Mode...................... Enabled
wsa Sync Interval...................... Fixed $ cat .maglevconf
wsa Subscription Topics................ all
NAC Data Publish Status:
Last Error.......................... Wed Apr 25 07:54:01 2018 Peer • Generate .pem file . The file needs to be transferred to WLC .
certificate cannot be authenticated with given CA certificates, SSL
certificate problem: unable to get local issuer certificate $ curl http://<DNAC IP address>/ca/pem > dna_cert.pem
Last Success........................ None
JWT Token Config.................... Not Available
JWT Last Success.................... None • Configure WLC
JWT Last Failure.................... None
(Cisco Controller) >config network assurance url <DNAC IP address>
Sensor Backhaul settings: (Cisco Controller) >config network assurance id-token <The Token that generated in
Ssid................................ Not Configured DNAC>
Authentication...................... Open
Sensor provisioning:
Status.............................. Disabled • Transfer the DNAC generated pem.file to WLC either through ftp/tftp/sftp. this can
Interface Name...................... None be done via WLC GUI or CLI
WLAN ID............................. None
SSID................................ None
(Cisco Controller) > From the WLC GUI:
Commands > Download File >

File-Type: NA-Serv-CA Certificate
Validate DNA-
Center Networking
Check the assigned IP addresses to DNA-Center
and the Virtual IP addresses
$ ip a | grep enp
$ etcdctl get /maglev/config/cluster/cluster_network
Check the Intra-Cluster link details
etcdctl get /maglev/config/node-<DNAC IP address>/network| python -mjson.tool
Validate High
Availability and
Cluster Health for
Cisco DNA Center
$ etcdctl get /maglev/node_scale/status
Completed
$ etcdctl cluster-health
member 93186661b8b32a0 is healthy: got healthy result from http://10.1.1.4:2379
member 1141887decc0d774 is healthy: got healthy result from http://10.1.1.2:2379
member a76429d777a6ffeb is healthy: got healthy result from http://10.1.1.1:2379
cluster is healthy
$ kubectl get nodes

NAME STATUS AGE VERSION
10.1.1.1 Ready 228d v1.7.3
10.1.1.2 Ready 228d v1.7.3
10.1.1.4 Ready 231d v1.7.3
Provisioning
Operations Stuck
for a Long Time
Firstly, Be Patient
Make Sure the SPF and Programmer Services are
Running
magctl appstack status | grep -e spf -e network-programmer -e rabbit
Check for queued messages in the RabbitMQ
$ magctl service exec rabbitmq-0 "rabbitmqctl list_queues"
Critical Check Points
and Known Issues
for DNAC-ISE
Integration
Three Step Integration of ISE with DNA Center
1. For SD-Access capabilities, ISE needs to have ISE Base and ISE Plus License installed
2. Make sure all required Ports are opened as per this guide:
1. https://www.cisco.com/c/en/us/td/docs/cloud-systems-management/network-automation-and-management/dna-center/1-
2/install/b_dnac_install_1_2/b_dnac_install_1_2_chapter_0101.html?bookSearch=true#reference_wtq_lkk_tdb
3. Make sure the ISE is on a supported version.
1. https://www.cisco.com/c/en/us/solutions/enterprise-networks/software-defined-access/compatibility-matrix.html
4. The ISE CLI and GUI user accounts must use the same user name and password.
5. DNA Center Certificate should have all physical IP’s and Virtual IP’s in the SAN Field of the Certificate.
6. The ISE admin certificate must contain the ISE IP address or fully-qualified domain name (FQDN) in either the certificate subject name or the SAN.
7. DNA Center and Cisco ISE IP/FQDN must be present in the proxy exceptions list IF there is a web-proxy between Cisco ISE and DNA Center.
8. DNA Center and Cisco ISE nodes cannot be behind a NAT device.
9. Cisco DNA Center and Cisco ISE cannot integrate if the ISE Admin and ISE pxGrid certificates are issued by different enterprise certificate authorities.
Three Step Integration of ISE with DNA Center
DNAC SSH’s into Port 9060 (TCP) XMPP Port 5222

ISE on Port 22 ERS calls (TCP)
(TCP)
Make Sure NTP is in sync on both DNA and ISE
The certificate download process will fail if the times are not in sync on DNA Center and ISE.
Please make sure NTP is in sync on both ISE and DNA Center before integrating ISE and DNA Center.
On DNA Center, use the below command:
ISE CLI/GUI Password Change OR Pxgrid
Certificate Expired/Replaced
1. Change the password
here if the password
from ISE side has
changed.
2. Even if there is a
certificate change, we
can just update the
password on this same
pacge and it will
exchange the
certificate change.
Pulling Client and
Inventory Health
Reports from
Cisco DNA Center
Navigate to Data and Reporting Page
Devices with Client Health for

Inventory Based Code version customized
Reports Report interval
Sample Report for Inventory
Sample Report for Client Health
API Documentation
https://developer.cisco.com/dnacenter/
https://developer.cisco.com/site/dna-center-rest-
api/
Complete your
online session • Please complete your session survey
evaluation after each session. Your feedback
is very important.
• Complete a minimum of 4 session
surveys and the Overall Conference
survey (starting on Thursday) to
receive your Cisco Live water bottle.
• All surveys can be taken in the Cisco Live
Mobile App or by logging in to the Session
Catalog on ciscolive.cisco.com/us.
Cisco Live sessions will be available for viewing
on demand after the event at ciscolive.cisco.com.
Continue your education
Demos in the
Walk-in labs
Cisco campus
Meet the engineer

Related sessions
1:1 meetings
NDA Roadmap Sessions at Cisco Live
Customer Connection Member Exclusive
Join Cisco’s online user group to …
Connect online with 29,000 peer and Cisco NETWORKING ROADMAPS SESSION ID DAY / TIME
experts in private community forums
Roadmap: SD-WAN and Routing CCP-1200 Mon 8:30 – 10:00
Roadmap: Machine Learning and

CCP-1201 Tues 3:30 – 5:00
Learn from experts and stay informed Artificial Intelligence
about product roadmaps Roadmap: Wireless and Mobility CCP-1202 Thurs 10:30 – 12:00
 Roadmap sessions at Cisco Live
 Monthly NDA briefings
Give feedback to Cisco product teams Join at the Customer Connection Booth
(in the Cisco Showcase)
 Product enhancement ideas
 Early adopter trials Member Perks at Cisco Live
 User experience insights • Attend NDA Roadmap Sessions
• Customer Connection Jacket
Join online: www.cisco.com/go/ccp • Member Lounge
Thank you
#CLUS
#CLUS
Additional Slides
for
Reference
Locator/ID Separation Protocol (LISP) Internet
Groper – “lig”
FE1#lig 18.18.18.18 instance-id 4099
Mapping information for EID 18.18.18.18 from 172.16.1.2 with RTT 7 msecs
18.18.18.18/32, uptime: 00:00:00, expires: 23:59:59, via map-reply, complete
Locator Uptime State Pri/Wgt
10.2.120.4 00:00:00 up 10/10
FE1#lig self instance-id 4099

10.2.1.40/32, uptime: 00:00:00, expires: 23:59:59, via map-reply, self, complete
Locator Uptime State Pri/Wgt
10.2.120.2 00:00:00 up, self 10/10
FE1#lig 17.17.17.17 instance-id 4099

16.0.0.0/4, uptime: 00:00:00, expires: 00:14:59, via map-reply, forward-native
Encapsulating to proxy ETR
SD-Access Data
Plane
Troubleshooting
Thank you
#CLUS
Package Update – GUI v/s CLI
How to get GUI name from CLI
maglev catalog package display base-provision-core | grep display
$ maglev catalog package display $ maglev catalog package display automation-core | grep display
maglev-1 [main - https://kong-frontend.maglev-system.svc.cluster.local:443] displayName: NCP - Services
NAME VERSION STATE INFO [Fri Jan 19 00:25:39 UTC] maglev@172.27.255.230 (maglev-master-1) ~
-------------------------------------------------------------------- $ maglev catalog package display base-provision-core | grep display
application-policy 2.1.1.170016 READY displayName: Automation - Base
assurance 1.0.5.583 READY
automation-core 2.1.1.60067 READY
base-provision-core 2.1.1.60067 READY $ maglev catalog package status network-visibility
command-runner 2.1.1.60067 READY maglev-1 [main - https://kong-frontend.maglev-system.svc.cluster.local:443]
device-onboarding 2.1.1.60067 READY
image-management 2.1.1.60067 READY KIND RESOURCE STATE MESSAGE
ncp-system 2.1.1.60067 READY -------------------------------------------------------------------------------------------------------
ndp-base-analytics 1.0.7.823 PARTIAL Package needs to bePackage
pulled/downloaded network-visibility:2.1.3.60048 READY
ndp-platform 1.0.7.724 PARTIAL Package needs to bePlugin
pulled/downloaded fusion/cli-template/devicecontrollability-cli-template-plugin:7.7.3.60048 READY
ndp-ui 1.0.7.919 PARTIAL Package needs to bePlugin
pulled/downloaded fusion/cli-template/perfmon-cli-template-plugin:7.7.3.60048 READY
network-visibility 2.1.1.60067 READY Plugin fusion/cli-template/wlc-dynamic-qos-cli-template-plugin:7.7.3.60048 READY
path-trace 2.1.1.60067 READY .
sd-access 2.1.1.60067 READY .
sensor-assurance 1.0.5.301 PARTIAL Package needs to be. pulled/downloaded
sensor-automation 2.1.1.60067 READY ServiceBundle fusion/apic-em-event-service:7.1.3.60048 READY
system 1.0.4.661 PARTIAL Package needs to beServiceBundle
pulled/downloaded fusion/apic-em-inventory-manager-service:7.1.3.60048 READY
ServiceBundle fusion/apic-em-jboss-ejbca:7.1.3.60048 READY
.
.
State to be ready .
ServiceBundleGroup fusion/apicem-core:2.1.3.60048 READY
ServiceBundleGroup fusion/dna-maps:2.1.3.60048 READY
ServiceBundleGroup maglev-system/apicem-core-ui:2.1.3.60048 READY
ServiceBundleGroup maglev-system/dna-maps-ui:2.1.3.60048 READY
Package Update
$ maglev catalog package display network-visibility (Continued)
_capabilityStatus: - level: 1
modified: 1513910885.2623622 minLevel: 0
_dependsOn: name: ncp:device-templating
capabilities: Package Dependencies _pullStatus: {}
- level: 1 abstract: A fundamental building block for all DNA Center Applications.
name: ncp:platform-base abstract: A fundamental building block for all DNA Center Applications.
- level: 1 description: 'A fundamental building block for DNA Automation, Network Controller
name: maglev:platform
_id: 5a39e478378cef79fe8ec4c4 Platform (NCP) offers capabilities such as such as Discovery, Inventory,
_provides: Topology, Site and Grouping services, Site Profiles, etc. DNA Center
capabilities: Package Capabilities Applications will leverage these capabilities to interact with devices on
the network, to provision, apply policies, or query the network.
- level: 1
minLevel: 0 '
name: ncp:service-provisioning-support displayName: Network Controller Platform
- level: 1 fqn: network-visibility:2.1.1.60067 Display name as
minLevel: 0
name: ncp:device-on-demand-read
info: ''
kind: Package shown in GUI
- level: 1 manifestVersion: v1
minLevel: 0 name: network-visibility
name: ncp:device-inventory requiresPull: false
- level: 1 serviceGroups:
minLevel: 0 - fusion/apicem-core:2.1.1.60067
name: ncp:platform-common - fusion/dna-maps:2.1.1.60067
- level: 1 - maglev-system/apicem-core-ui:2.1.1.60067
minLevel: 0 - maglev-system/dna-maps-ui:2.1.1.60067
name: ncp:floor-maps state: READY
- level: 1 status:
minLevel: 0 state: READY
name: ncp:device-model-config tenantId: SYS0
version: 2.1.1.60067
Package Deploy Failure and Recovery
How to Check Workflows from GUI

System Settings  System360: Tools (right-side)
 Workflows
https://<dnacenter_ip>/app/system/workflow/
Failure scenario
Cisco DNA Center Services not coming up
How to Check Service Status from GUI
System Settings  System360: Services
https://<dnacenter_ip>/dna/systemSettings
How to Check Service Status from CLI
• SSH to Cisco DNA Center server
• Check for Service Instance status using “magctl appstack status <service>”
• Various States – Running, Terminating, Unresponsive, Error, crashdump, stopped
$ magctl appstack status fusion
NAME READY STATUS RESTARTS AGE IP NODE
apic-em-event-service-1698386882-cxvkb 1/1 Running 0 1d 10.10.243.107 192.168.240.11
apic-em-inventory-manager-service-3938287905-2ghz4 1/1 Running 0 1d 10.10.243.70 192.168.240.11
apic-em-jboss-ejbca-2091556107-t632h 1/1 Running 0 1d 10.10.243.105 192.168.240.11
apic-em-network-programmer-service-1178764915-blkpg 1/1 Running 0 1d 10.10.243.90 192.168.240.11
apic-em-pki-broker-service-4242378431-08dzw 1/1 Running 0 1d 10.10.243.111 192.168.240.11
app-policy-provisioning-service-1453250883-n3pkw 1/1 Running 0 1d 10.10.243.74 192.168.240.11
...
Check for services restarts count / error

• magctl appstack status | awk '$5 !~ /^0/'
$ magctl appstack status | awk '$5 !~ /^0/'
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE
kube-system kube-controller-manager-192.168.240.11 1/1 Running 23 243d 192.168.240.11 192.168.240.11
kube-system kube-scheduler-192.168.240.11 1/1 Running 42 243d 192.168.240.11 192.168.240.11
maglev-system catalogserver-3012330575-wlrnl 1/1 Running 2 1d 10.10.243.48 192.168.240.11
maglev-system encryptionmanager-1445236960-2tc93 1/1 Running 1 1d 10.10.243.14 192.168.240.11
maglev-system kibana-logging-2447148192-z8qrx 1/1 Running 3 65d 10.10.243.28 192.168.240.11
maglev-system kong-2609876156-kls0g 2/2 Running 2 1d 10.10.243.37 192.168.240.11
maglev-system system-updater-1441312065-d3fkt 1/1 Running 1 1d 10.10.243.17 192.168.240.11
maglev-system telegraf-3824842432-h92w8 2/2 Running 2 1d 10.10.243.62 192.168.240.11
Check Cisco DNA Center server
resources • Check CPU usage “top”
• Disk Throughput Check “iostat”
Assurance Behavior on node failure
Switch 1 Switch 2 Switch 3 Switch 1 Switch 2 Switch 3
Cisco Cisco Cisco Cisco Cisco Cisco

DNAC1 DNAC2 DNAC3 DNAC1 DNAC2 DNAC3
Assurance runs on Cisco DNAC1 – single node
Assurance is impacted if Cisco DNAC1 fails – no impact if any other node fails
To restore assurance, the failed node must be removed from cluster and assurance restarted in another active
node
Failure of Assurance enabled node will result in loss of assurance data
Restore the last Assurance backup through UI

Assurance Services Failure UI Notifications
Node down notification
 2nd and 3rd node will form a quorum

 UI won’t be available till services are
distributed
Assurance Services Failure UI Notifications
Graceful node power cycle in cluster
Perform below steps to gracefully restart a node in cluster

• Restart/reboot the node and once node shows as Ready on ”kubectl get nodes”
• Execute below command to add the node back to cluster
$ maglev node allow <node_ip>
• Perform $ maglev service nodescale refresh
Clustering Capabilities
Failover:
• Stateful set services get replicated onto all the master nodes to enable stateful failover of
requests in the event that processes servicing these requests fail.
• This is achieved by the anti-node affinity configured for the services that are getting scaled..
Load Balancing:
• With a load-balancing mechanism in place, the requests are distributed across the nodes. If
any of the instances fail, requests to the failed instance can be sent to the surviving instances.
(eg. Kong HA provides load balancing capability as part of HA )
Rebalancing of services:
• Services get rebalanced onto the number of nodes available ( 3 in our case) instead of running
only on one node.

Brkarc 2016

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Brkarc 2016

Uploaded by

Copyright:

Available Formats

#CLUS

• API Based Troubleshooting

• API Based Troubleshooting

Webex Teams will be moderated cs.co/ciscolivebot#BRKARC-2016

Single Appliance for Cisco DNAC (Automation + Assurance)

DESIGN PROVISION POLICY ASSURANCE

Cisco DNA Center

Routers Switches Wireless Controllers Wireless APs

ISE Appliance Cisco DNA Center

Design | Policy | Provision | Assurance

Cisco Switches | Cisco Routers | Cisco Wireless

Authentication Groups and

Cisco DNA Center

Cisco DNA Center

Devices REST pxGrid

ISE-PSN ISE-PAN ISE-PXG

Design Provision Policy Assurance

• Global Settings • Fabric Domains • Virtual Networks • Health Dashboard

Planning & Preparation

APIs, SDK & Packaging APIs, SDK & Packaging

distributed-cache- rbac-broker-service RBAC

trap Trap Collector workflow-worker Various Update workflow task

• Check network connectivity

Avoid console login or don’t run

Verify sufficient disk and memory available

Verify number of CPUs to be minimum 88

Docker health check

Failure State and

To Check the live log during update

$ maglev catalog package delete network-visibility:2.1.1.60067 Delete the package

$ maglev catalog package pull network-visibility:2.1.1.60067

Switch 1 Switch 2 Switch 3

Cisco Cisco Cisco

Cluster nodes MUST be on the same version

To Configure node-2 point to first node-1 as first step of software install

Repeat the same for node-3 after node-2 completes installation

Fusion Services NDP Services

GlusterFS MongoDB Kubernetes

RabbitMQ Cassandra Docker

Kong Kong Kong

Fusion Services NDP Services

GlusterFS MongoDB Kubernetes GlusterFS MongoDB Kubernetes GlusterFS MongoDB Kubernetes

RabbitMQ Cassandra Docker RabbitMQ Cassandra Docker RabbitMQ Cassandra Docker

Kong Kong Kong

CatalogServer MaglevServer WorkflowServer DockerRegistry WorkflowWorker

GlusterFS MongoDB Kubernetes GlusterFS MongoDB Kubernetes GlusterFS MongoDB Kubernetes

RabbitMQ Cassandra Docker RabbitMQ Cassandra Docker RabbitMQ Cassandra Docker

Enable Service Distribution Not

Enable Service Distribution shows

Enabling HA using CLI

Service Distribution happened

Cisco Cisco Cisco Cisco Cisco Cisco

Node fails, automation services are automatically distributed

Current re-distribution takes 25 minutes (unplanned)

Failure of two nodes will bring the cluster down

Persistent notification of failure:

Node down notification

 2nd and 3rd node will form a quorum

Nodes are up but one or more

Some services are pending and

Node down Some services

Node down Banner changed from Node Lost to Services

Node Up Fully restored so banner gone