Brkewn 3012

Performance
Monitoring on AireOS
controllers and APs
Javier Contreras Albesa

BRKEWN-3012
Cisco Spark
Questions?
Use Cisco Spark to chat with the
speaker after the session
How
1. Find this session in the Cisco Live Mobile App
2. Click “Join the Discussion”
3. Install Spark or go directly to the space
4. Enter messages/questions in the space
Cisco Spark spaces will be cs.co/ciscolivebot#BRKEWN-3012

available until July 3, 2017.
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Agenda
• Chinese Medicine - Prevention
• KPI - Health
• Translating KPI to AireOS
• IOS KPI
• AP-COS KPI
• Q&A
Chinese
Medicine
Chinese Medicine – Prevention
Performance Monitoring as Preventive Medicine
 We want to stay Healthy

 Focus on prevention and good
habits
 Not on “fixing” after the
crash/illness
 What to monitor to ensure the
Wireless network remains Healthy
BRKEWN-3012 © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
Key Performance Indicators
Looking for Symptoms
 Key Performance Indicators

 Can be used to monitor success of an organization
 Define a set of values against which to measure
 Defining KPIs for network infrastructure subsystems
 KPI as health indicators: is the network device

healthy?
Doctor am I sick?
Looking for trouble before it happens
 Not good: going to doctor after 5 days with 42C fever

 Preventive Detection: monitoring health overall
 Good detection:
 Memory utilization increasing every day
 Task tending to high CPU
 Radio channel tending to high utilization
 Not optimal detection:

 Uptime
 Channel meltdown
What are Good Health Indicators?
Where to look
 Resources are Finite

 CPU time
 Memory
 Timers
 ARP entries
 Client entries
 Queues
 Longer Uptimes/Capwap
 You can always exhaust resources
 Utilization remains in expected range under intended
load
What are Good Health Indicators?
Where to look
 Negative Events
 Radio Resets (not all!)
 Crashes
 Drops
 Hangs
 Not servicing clients
How can I obtain this?
Getting the data
 PI will not provide it all

 It is a more “network wide status” tool
 This is a per device check

 For the moment, scripting tools can help
 Health data now in 8.3.120.0

 WLCCA can process it
Translating
KPIs to AireOS
WLC Architecture
What we need to Look At
Multicore CPU Multicore CPU

(Control) (Data)
Control Plane
PCIe Switch
I/O Subsystem
Service
Port
0 1 6 7
AP Client
Network
WLC Architecture

(Control) (Data)
Data Plane
PCIe Switch
I/O Subsystem
Service
Port
0 1 6 7
AP Client
Network
WLC Architecture

(Control) (Data)
PCIe Switch
I/O Subsystem
Service
Port
0 1 6 7
AP Client
Network
AP
WLC Architecture

(Control) (Data)
PCIe Switch
I/O Subsystem
Service
Port
0 1 6 7
AP Client
Network
RF
AireOS KPIs SPAM
It is a complex world DTLS
(CAPWAPP)
APF
(802.11)
RRM
PEM
Dot1x MSCB OS Service

(Keys/Auth)
IP Stack Port
Driver
AAA
Subsystems
Data Plane
AireOS Basic Health
Does it have Fever?
 Where to Put the Thermometer?

 Control Plane
 Data Plane
AireOS KPIs
Let’s measure – Control Plane
 Queue Utilization  Client Count

 Per Process CPU  PMK Count
 CPU per core
 DHCP stats
 Timers
 ARP Switch/Kernel
 Memory Utilization
 Mbufs
AireOS KPIs
Let’s measure - Data Plane
 State monitoring
 DP-CP Drops
 Fragmentation
 Packet Pools
One CPU to bind them…
AireOS KPIs
CPU Monitoring
(Cisco Controller) >show cpu
Current CPU(s) load: 4%
Individual CPU load: 0%/0%, 0%/0%, 0%/1%, 100%/0%, 0%/1%, 0%/1%,

0%/1%, 0%/1%, 0%/1%, 0%/1%, 0%/1%, 0%/1%, 0%/0%, 0%/0%, 0%/1%,
0%/0%, 0%/0%, 0%/0%, 0%/0%, 0%/1%, 0%/0%, 0%/0%, 0%/0%, 0%/0%
AireOS KPIs
Task-CPU Utilization
(WLC-5508-7) >show process cpu
Name Priority CPU Use (usr/sys)% hwm CPU Reaper
System Reset Task (240/ 7) 0 ( 0/ 0)% 0 9
reaperWatcher ( 3/ 96) 0 ( 0/ 0)% 0 7 I
osapiReaper ( 10/ 94) 0 ( 0/ 0)% 0 1 I
TempStatus (240/ 7) 0 ( 0/ 0)% 0 9 I
HAPeerToPeerCommTa ( 90/ 64) 0 ( 0/ 0)% 0 6
pktDebugSocketTask (255/ 1) 0 ( 0/ 0)% 0 6
webauthRedirect ( 95/ 62) 100 ( 95/ 4)% 100 9 T 300
emWeb (240/ 7) 0 ( 0/ 0)% 100 1 T 300
CSCuo86819 WLC crashes in 7.6.120.0 - memory corruption caused by Webauth
Queues In London
AireOS KPIs
Queue Utilization
 Used for messaging between tasks

 Around 128-150 queues in 8.4/8.3
 Some are critical for the well being of the WLC
(5520-2) >show queue-info
Total message queue count = 128
* Column Breached shows the number of times the queue breached the 95% mark
Q-Id Queue Name Allocated MsgSize InUse MaxUsed Breached* Details
--------------------------------------------------------------------------------------------
1 PRINTF-Q 256 80 0 0 0 N
2 GRE Queue 100 8 0 1 0 N
3 dtlqueue 4096 2 0 3 0 N
4 dtlarpqueue 4096 24 0 8 0 Y
AireOS KPIs
Queue Utilization
 Inuse: how many messages in the queue

 MaxUsed: top message count ever
 Breached: how many times we reached 95%
utilization
AireOS KPIs
Queue Utilization
(WiSM-slot1-1) >show queue-info
Q-Id Queue Name Allocated MsgSize InUse MaxUsed Breached* Details
--------------------------------------------------------------------------------------------
1 PRINTF-Q 256 80 0 0 0 N
2 GRE Queue 100 8 0 5 0 N
3 dtlqueue 4096 2 0 3 0 N
4 dtlarpqueue 4096 16 0 75 0 Y
..
70 nfv9queue 4096 316 0 2 0 Y
71 RogueApInfoChangedDB 4000 4 4000 4000 68 N
72 RogueApMeasChangedDB 4000 4 4000 4000 1 N
73 RogueClientInfoChangedDB 5000 4 5000 5000 1 N
74 RogueClientMeasChangedDB 5000 4 5000 5000 1 N
AireOS KPIs
Queue Utilization
86 RRM-DCLNT-5_0-Q 375 0 4 0 N
87 RRM-MGR-5_0-Q 375 0 20 0 N
88 RRM-PERIODIC-TASK-Q 93 0 1 0 N
89 SNTP-Q 3 2 3 20010 N
90 CDP-Q 512 0 13 0 N
91 BCAST-Q 2048 0 2 0 Y
92 BCAST-DATA-Q 512 0 2 0 Y
93 IDS Track Event Msg Queue 1024 0 8 0 N
AireOS KPIs
Queue Decoder Ring
 Dtlarpqueue: Data transformation layer ARP queue

 SNMP-Q: SNMP requests
 Debugger/logger-q: Debug and Syslog Messages
 SPAM: Messages between AP and WLC
 NMSP*: location
 Dot1x: all authentication (PSK/dot1x)
 APF*: 802.11 client state machine
 BCAST*: multicast/broadcast
A queue like no
other
AireOS KPIs
Radius Queues
 Radius messages between WLC and AAA server

 8.0MR3 and lower: 255 standing requests (ID)
 8.0MR4 and higher: 16 queues per server (ID/Source
Port)
(5520-2) >show radius queue
Max Radius Queues Per Server..................... 16
Max Radius Buffers Available..................... 8128
Currently number of Buffers consumed............ 0
Radius Authentication Messages Stats
Total Auth Req sent(allocated).................. 140
Total Auth Resp rcvd(freed)..................... 140
Total Auth Req Pkts Dropped(no buffer).......... 0
AireOS KPIs
Radius Queues
(Cisco Controller) >show radius queue
Radius Queue
Auth Alloc : 122401616
Acct Alloc : 31288769
Auth Free : 122401615
Acct Free : 31288769
Auth Alloc Err : 0
Acct Alloc Err : 0
AireOS KPIs
DHCP
(CH-8510W-DC-1) show>dhcp stats
DHCP Proxy Behaviour: enabled
DHCP Server IP Address:.......................... 10.5.0.10 DHCP NAK Count:................................ 3573
DHCP RX DISCOVER Count:........................ 3117892 Tx Fails:...................................... 26765
DHCP TX DISCOVER Count:........................ 3175288 Last Rx Time:.................................. 1 seconds ago
DHCP ACK Count:................................ 4644105 Last Tx Time:.................................. 3 seconds ago
DHCP REQUEST Count:............................ 7641156
DHCP INFORM Count:............................. 200006
DHCP DECLINE Count:............................ 0
DHCP RELEASE Count:............................ 987
DHCP REPLY Count:.............................. 0
DHCP OFFER Count:.............................. 1283401
I remember you..
AireOS KPIs
Memory Monitoring
 Memory Utilization
 Increases with load, remains steady on same load
level
 Most tasks use OSAPI Buffer based: time, owner,
trailers
 MBUFs
AireOS KPIs
Memory Monitoring
Memory Memory
1000 650
800 600
600
550
400
200 500
0 450
Memory Memory
AireOS KPIs
Memory the easy way
 show memory history (8.0.132.0 and higher)

(5520-2) >show memory history
-------------------------- System Memory History -------------------------
System Name ..................................... 5520-2
Software Version ................................ 8.4.1.189
Current Time .................................... Sun Jan 29 17:59:44 2017
System UP Time .................................. 6 days 4 hrs 12 mins 48 secs
------------------------- System Memory at BOOT ------------------------
Memory Snapshot Time ............................ Mon Jan 23 14:17:18 2017
Total System Memory ............................. (32386132 KB) 31627 MB
Total System Free Memory......................... (29429840 KB) 28740 MB (90 %)
AireOS KPIs
Memory the easy way
 Hundreds of data points

------------------------- System Memory Snapshot 1 Count: 289 -------
Memory Snapshot Time ............................ Sun Jan 29 14:54:31 2017
Total System Memory ............................. (32386132 KB) 31627 MB
Total System Free Memory......................... (29408732 KB) 28719 MB (90 %)
Total Memory in Buffers.......................... (9960 KB)
Total Memory in Cache............................ (703068 KB) 686 MB
Total Active Memory.............................. (2200724 KB) 2149 MB
Total InActive Memory............................ (597932 KB) 583 MB
AireOS KPIs
Memory the easy way
 MBUF Utilization
------------------------- MBUF Information ------------------------------
Maximum number of Mbufs.......................... 24576
Number of Mbufs Free............................. 24560
Number of Mbufs In Use........................... 16
AireOS KPIs
Memory the easy way
 Buffers in use and Top users
------------------------- Pmalloc Pools Information --------------------
Index Pool-Size Chunks-In-Pool Chunks-In-Use Memory(Size/Used/Free)KB
0 16 200000 40881 21875 /19388 /2486
1 64 160000 6794 25000 /15424 /9575
2 128 93600 31317 20475 /12689 /7785
3 256 20000 6406 6875 /3476 /3398
..
-------------- TOP-20 Pools Consuming Tasks (Time: 465689 Secs) ------------
Index Name PID 16-byte 64-byte 128-byte 256-byte 384-byte 512-byte 1024-byte 2048-byte 4096-byte Raw-Pool Total-Pools
001 fp_main_task 2182 3011 4289 155 280 55 9 124058 60 24586 1767 158270
002 emWeb 2664 37373 657 31015 5987 30 4 21 2 7 9 75105
A little bit of
everything…
AireOS KPIs
Happy Uptime
(5520-2) >show ap uptime
Number of APs.................................... 2
Global AP User Name.............................. admin
Global AP Dot1x User Name........................ Not Configured
AP Name Ethernet MAC AP Up Time Association Up Time
------------------ ----------------- ----------------------- -----------------------
ap1800-3850-3-11 80:e8:6f:d0:12:08 5 days, 19 h 57 m 32 s 5 days, 19 h 55 m 54 s
barbados-beta-floor1 58:ac:78:dc:d3:a0 5 days, 19 h 58 m 24 s 1 days, 04 h 53 m 52 s
AireOS KPIs
Coredump Available
(5520-2) >show coredump summary
Core Dump upload is disabled
Core Dump file on flash:
SW Version....................................... 8.2.102.89
Time Stamp....................................... Tue Apr 26 15:01:08 2016
File Size........................................ 50431374
File Name Suffix................................. _8.2.102.89_5520-

2_1461682868.gz
AireOS KPIs
Time(rs) are gold
(5500-2) >show system timers summary
Total Timers..................................... 150000
Allocated.................................... 2538
Free......................................... 147462
Peak Usage................................... 2842
Alloc Operations................................. 275207
Free Operations.................................. 272669
Alloc Failures................................... 0
AireOS KPIs
5520/8540 Chassis Monitoring
(5520-2) >show imm chassis fan
FAN1_TACH1 | 0Fh | ok | 29.1 | 8200 RPM
FAN1_TACH2 | 10h | ok | 29.1 | 9600 RPM
FAN2_TACH1 | 11h | ok | 29.2 | 8200 RPM
FAN2_TACH2 | 12h | ok | 29.2 | 9600 RPM
FAN3_TACH1 | 13h | ok | 29.3 | 8200 RPM
FAN3_TACH2 | 14h | ok | 29.3 | 10000 RPM
FAN4_TACH1 | 15h | ok | 29.4 | 8200 RPM
FAN4_TACH2 | 16h | ok | 29.4 | 10000 RPM
FAN5_TACH1 | 17h | ok | 29.5 | 8200 RPM
FAN5_TACH2 | 18h | ok | 29.5 | 10000 RPM
AireOS KPIs
My Dear Clients
(Cisco Controller) >show client state summary
Client State Summary
====================
State Number of Clients
----- -----------------
START 8
8021X_REQD 25
DHCP_REQD 54
WEBAUTH_REQD 260
RUN 146
----- -----------------
Total 493
AireOS KPIs
My Dear Clients
(Controller_2) >show pmk-cache all
Number of PMK Cache Entries: 156
Type Station Lifetime VLAN Override IP Override Audit-Session-ID
------ -------------- -------- ------------------ --------------- ----------------
RSN 04:54:53:4b:63:5c 86301 0.0.0.0
RSN 04:54:53:4c:22:d9 57521 0.0.0.0
RSN 08:70:45:92:7d:f0 86363 0.0.0.0
AireOS KPIs
ARPs
(5520-2) >show arp stats
ARP statistics:
Max=client+1000
Current Entries 236
Dup Additions 382947
Bad Additions 0
Bad Deletions 0
(5520-2) >show arp kernel
IP address HW type Flags HW address Mask Device
4.0.7.161 0x1 0x2 00:00:05:00:00:04 * dtl0.4
4.0.7.122 0x1 0x2 00:00:06:00:00:09 * dtl0.4
4.0.7.117 0x1 0x2 00:00:06:00:00:1d * dtl0.4
192.168.5.104 0x1 0x2 58:ac:78:dc:d3:a0 * dtl0.5
AireOS KPIs
LAG
 This won’t happen unless configured

 If you change it, reboot
(5500-4-82) >show lag summary
LAG Disable is in transition. Pls Reboot the switch
The Data
Plane: happy
frames
Data Plane
Happy to forward your frames
 Takes care of all frame forwarding for clients

 Also: AVC, Encryption, Tunneling
 Closely monitored by control plane
 Its own memory, CPU, software
 Its own stats and performance indicators
Data Plane
Happy to forward your frames
 Main command:
(5500-2) >debug fastpath dump fpapool
FP0.01:
FP0.01:(#)Buffer Pool = avail / size (in-use) min-avail (max in-use)
..
FP0.01:Buffer overload back-pressure limits:
FP0.01:Entered _GREENZONE 1 times. _GREENLINE is 27000 buffers available.
FP0.01:Entered YELLOWZONE 0 times. YELLOWLINE is 27000 buffers available.
FP0.01:Entered ORANGEZONE 0 times. ORANGELINE is 26000 buffers available.
FP0.01:Entered ___REDZONE 0 times. ___REDLINE is 10000 buffers available.
FP0.01:Entered _BLACKZONE 0 times. _BLACKLINE is 5000 buffers available.
Data Plane
Happiness gone
 Green Zone: Normal Scenario

 Yellow Zone: Traffic Peaks
 Orange Zone: Multicast is dropped, CAPWAP is top
priority
 Red/Black: Heavy packet drop -> recovery through
crash
FP0.02:Entered _GREENZONE 59575 times. _GREENLINE is 27000 buffers available.
FP0.02:Entered YELLOWZONE 59575 times. YELLOWLINE is 27000 buffers available.
FP0.02:Entered ORANGEZONE 1 times. ORANGELINE is 26000 buffers available.
FP0.02:Entered ___REDZONE 0 times. ___REDLINE is 10000 buffers available.
FP0.02:Entered _BLACKZONE 0 times. _BLACKLINE is 5000 buffers available.
Data Plane – Control Plane
Let’s Talk
(5520-2) >debug fastpath dump dpcp-stats
FP0.20:Queue TxCount RateLmtDrop PktLmtDrop
FP0.20:Q0: Internal DP --- 85587 0 0
FP0.20:Q1: Mobility Ctrl --- 19 0 0
FP0.20:Q2: Capwap --- 237449 0 0
FP0.20:Q3: Client Traffic --- 1610 0 0
FP0.20:Q4: ARP, Client services --- 33729 0 0
FP0.20:Q5: IPv6 Client services --- 16634 0 0
FP0.20:Q6: Infra services --- 42932 0 0
FP0.20:Q7: Probe --- 276962 0 0
FP0.20:Q8: IAPP, Rogue --- 85139 1 0
FP0.20:Q9: HA --- 8714 0 0
FP0.20:Q10: WebAuth --- 0 0 0
IOS APs
IOS APs
Simpler
 IOS as monolithic entity

 Main things to monitor
 CPU
 Memory
 Interfaces
 Radios
 CAPWAP
IOS APs
Radio Interfaces
 Show controller d0 | d1
 Lot of data, not all human readable
 Not intended for “general use”, but treasure of
information
IOS APs
Show Controller - Monster
interface Dot11Radio0Radio AL 2.4GHz, Base Address b0aa.7733.bf10, BBlock version 0.00, Software version
4.18.2Serial number: FOC19066XZD Unused dynamic SQRAM memory: 0x0000C858 (50 KB)Unused dynamic SDRAM memory:
0x00082674 (521 KB)Spectrum FW version: 1.15.4Number of supported simultaneous BSSID on Dot11Radio0: 16Carrier Set:
Spain (ES) (-E)Uniform Spreading Required: NoConfigured Frequency: 2412 MHz Channel 1Serving Frequency: 2412 MHz
Channel 1 20MHzAllowed Frequencies: 2412( 1) 2417( 2) 2422( 3) 2427( 4) 2432( 5) 2437( 6) 2442( 7)
2447( 8) 2452( 9) 2457( 10) 2462( 11) 2467( 12) 2472( 13) Listen Frequencies: 2412( 1) 2417( 2)
2422( 3) 2427( 4) 2432( 5) 2437( 6) 2442( 7) 2447( 8) 2452( 9) 2457( 10) 2462( 11) 2467( 12)
2472( 13) 2484( 14) Beacon Flags: 0, Interface Flags 2020105, Interface Events 20, Mode 9; Beacons are enabled;
Probes are enabledConfigured TxPower: 16 dBm (Level Index 1)Allowed Power Levels:
16 13 10 7 4 1 dBmAllowed Client Power Levels: 16 13 10 7 4 1 dBmAntenna:
Rx[right-a left-b middle-c ]
Tx[right-a left-b middle-c cck all ofdm all]
Internal Gain [Allowed 8, In
Use 8] (dBi x 2), In Use 8] (dBi x 2), In Use 8] (dBi x 2), In Use 8] (dBi x 2), Reported 0, Configured 0, In
Use 8] (dBi x 2)Active Rates: 9.0 basic-12.0 18.0 24.0 36.0 48.0 54.0 m0-2 m1-2 m2-2 m3-2 m4-2 m5-2 m6-2 m7-2 m8-
2 m9-2 m10-2 m11-2 m12-2 m13-2 m14-2 m15-2Current Rates: 9.0 basic-12.0 18.0 24.0 36.0 48.0 54.0 m0-2 m1-2 m2-2
m3-2 m4-2 m5-2 m6-2 m7-2 m8-2 m9-2 m10-2 m11-2 m12-2 m13-2 m14-2 m15-2Allowed Rates: 1.0 2.0 5.5 6.0 9.0 11.0 12.0
18.0 24.0 36.0 48.0 54.0All Rates: 1.0 2.0 5.5 11.0 6.0 9.0 12.0 18.0 24.0 36.0 48.0 54.0 m0-2 m1-2 m2-2 m3-2 m4-2
m5-2 m6-2 m7-2 m8-2 m9-2 m10-2 m11-2 m12-2 m13-2 m14-2 m15-2Default Rates: basic-1.0 basic-2.0 basic-5.5 basic-
11.0 6.0 9.0 12.0 18.0 24.0 36.0 48.0 54.0 m0-2 m1-2 m2-2 m3-2 m4-2 m5-2 m6-2 m7-2 m8-2 m9-2 m10-2 m11-2 m12-2 m13-
2 m14-2 m15-2Best Range Rates: basic-1.0 2.0 5.5 11.0 6.0 9.0 12.0 18.0 24.0 36.0 48.0 54.0 m0-2 m1-2 m2-2 m3-2
m4-2 m5-2 m6-2 m7-2 m8-2 m9-2 m10-2 m11-2 m12-2 m13-2 m14-2 m15-2Best Throughput Rates: basic-1.0 basic-2.0 basic-
5.5 basic-11.0 basic-6.0 basic-9.0 basic-12.0 basic-18.0 basic-24.0 basic-36.0 basic-48.0 basic-54.0 m0-2 m1-2 m2-2
m3-2 m4-2 m5-2 m6-2 m7-2 m8-2 m9-2 m10-2 m11-2 m12-2 m13-2 m14-2 m15-2Current Voice Rates: basic-5.5 basic-11.0
6.0 12.0 24.0 [disabled until voice packet-discard enabled]Default Voice Rates: basic-5.5 basic-11.0 6.0 12.0
24.0Managment Rates: 9.0 basic-12.0Multicast Rates: basic-12.0 Multicast Active rate: 12.0RTS Rates: basic-
12.0RTS Active rate 12.0RTS Max rate 12.0 Local Packet Rate Probing is disabled : Packet Size 20Data Rate
Sensitivity (rate, SNR dB, Contention dBm)( 1.0, 10, -93) ( 2.0, 10, -92) ( 5.5, 11, -90) (11.0, 14, -90) (
6.0, 15, -89) ( 9.0, 16, -88) (12.0, 18, -88) (18.0, 19, -86) (24.0, 20, -85) (36.0, 24, -81) (48.0,
27, -78) (54.0, 31, -77) ERP settings: short slot time.Neighbors in non-erp mode:
IOS APs
Taming the monster
 Interface status
Beacon Flags: 0, Interface Flags 20105, Interface Events
0, Mode 9; Beacons are enabled; Probes are enabled
 Channel set to zero

Configured Frequency: 2412 MHz Channel 1
 Warn on low PoE scenario

Radio on Low Power Mode due to PoE, restricted to 3
antennas
• QBSS set to higher than 80% (0xCC)

QBSS Load: 0x24
IOS APs
Taming the monster
 TX blocks - Pending
Transmit queues: Limit 13919 Current 4 In-Progress 0
---- Active ------ In-Progress -----------
Cnt Quo Bas Max Cl Cnt Quo Bas Uplink
0 64 0 0 0 0 5 0 Voice
0 512 0 0 0 0 60 0 Video
0 1024 0 0 0 0 200 0 Best
4 1024 0 0 0 0 200 0 MC0
0 0 0 0 0 0 7 0
• High Radio reset counts

Radio resets - total:14 retries:0 failed:0
IOS APs
Taming the monster
 RF jammer detected
Jammer detected: 1 / 0
• Beacon Stop
Beacon stopped count: 0
Counts > 120 0
Counts > 90 0
Counts > 60 0
Counts > 30 2
Counts > 15 0
Counts > 10 0
Counts > 5 1
IOS APs
Taming the monster
• Silent radio hang
• No traffic received/sent when clients are around
Host Rx K Bytes: 2243173 / 0 Host Tx K Bytes: 3382831 / 0
Unicasts Rx: 368819860 / 0 Unicasts Tx: 457956326 / 0
Unicasts to host: 368819860 / 0 Unicasts by host: 415499647 / 0
Broadcasts Rx: 348114707 / 0 Broadcasts Tx: 354596429 / 439
Beacons Rx: 342650083 / 0 Beacons Tx: 353388822 / 439
 PCI Resets
Radio PCI resets:0
AP-COS
1800/2800/3800/
1560
Wave2
AP-COS
Different architecture (simplified view)
Light
ssh
httpd capwapd WCP watchdog
CLI
Capwap hostapd LED
brain Platform
Device Management Config/system mng Wireless Control Management
Linux Cisco
Networking /clickfile Linux
stack Services system
Kernel
Click DataPath
Ethernet Radio Flash
Driver Driver Driver
AP-COS
Different organism, somewhat different health metrics
 Some structures remain similar

 Some have changed, sh tech as easiest way
 Main things to monitor
 CPU
 Memory
 Interfaces
 Radios
 CAPWAP
AP-COS
Memory - CPU
 Memory
***** show memory summary *****
total used free shared buffers
Mem: 1022672 379456 643216 0 0
 CPU
***** show processes cpu *****
Mem: 379456K used, 643216K free, 107792K shrd, 0K buff, 107916K cached
CPU: 0% usr 4% sys 0% nic 95% idle 0% io 0% irq 0% sirq
Load average: 0.08 0.10 0.13 2/111 22143
PID PPID USER STAT VSZ %VSZ CPU %CPU COMMAND
4710 2 root SWN 0 0% 1 4% [kclick]
AP-COS
Temperature
sh thermal-control-summary
This CLI produces large dump, do you want to proceed?(y/n)y
AP thermal condition
--------------------
fair
Temperature readings
--------------------
A38x CPU temp = 57 C
I2C-0-4f temp = 35 C
AP-COS
Radio Crashes
Radio Core files
***** show flash cores *****
Directory of /storage/cores/
total 1524
-rw-r--r-- 1 root root 1430212 Feb 22 17:06 AP1801-3802e-up-dd_core-
radio0FW-8.2.145.44.2017-02-22-17-06-27.tgz
-rw-r--r-- 1 root root 123448 Feb 20 10:58 AP1801-
3802e.all.10.0.cleanair.tgz
---------------------------------------------------------------------------
Filesystem Size Used Available Use% Mounted on
flash 57.5M 1.7M 52.8M 3% /storage
AP-COS
5 last set of logs
 AP syslog data
***** show flash syslogs *****
Directory of /storage/syslogs/
total 296
-rw-r--r-- 1 root root 7969 Feb 8 17:36 176
-rw-r--r-- 1 root root 20479 Feb 7 19:15 176.0
-rw-r--r-- 1 root root 4838 Feb 8 17:36 176.last_write
-rw-r--r-- 1 root root 20480 Feb 7 17:55 176.start
-rw-r--r-- 1 root root 25 Feb 8 17:37 176.watchdog_status
-rw-r--r-- 1 root root 9976 Feb 9 04:36 177
-rw-r--r-- 1 root root 20407 Feb 8 17:42 177.0
AP-COS
Client Information
 Different components
barbados-beta-floor1#sh dot11 clients 0
AP Mode - FlexConnect
Client MAC Slot ID WLAN ID AID WLAN Name RSSI Maxrate WGB
00:1E:E5:DF:A3:C4 1 3 1 psk-w2 -40 M15 No
barbados-beta-floor1#sh cont dot11Radio 1 client
mac radio vap aid state encr Maxrate is_wgb_wired wgb_mac_addr
00:1E:E5:DF:A3:C4 1 2 5 FWD AES_CCM128 M15 false 00:00:00:00:00:00
AP-COS
Client Information – 8.3MR2
AP58AC.78DC.AAA0#sh client summary
Radio Driver client Summary: WCP client Summary:
============================== =====================
wifi0 mac radio vap aid state encr Maxrate is_wgb_wired

wgb_mac_addr
[*02/08/2017 13:36:58.7460]
74:23:44:3B:06:07 0 0 2 FWD OPEN M7 false
[*02/08/2017 13:36:58.7460] Total STA List Count 2 00:00:00:00:00:00
[*02/08/2017 13:36:58.7461] | NO| MAC|STATE| 3C:97:0E:D5:E8:21 0 0 1 FWD OPEN M7 false

00:00:00:00:00:00
[*02/08/2017 13:36:58.7461] -----------------------------
NSS client Summary:
[*02/08/2017 13:36:58.7461] | 1|74:23:44:3b: 6: 7| 8|
=====================
[*02/08/2017 13:36:58.7461] | 2|3c:97: e:d5:e8:21| 8|
Current Count: 4
wifi1
| MAC | OPAQUE |PRI POL|VLAN|BR|TN|QCF|BSS|RADID|MYMAC|
[*02/08/2017 13:36:58.7479]
|58:AC:78:DC:AA:A0|00000000| 3| 0| 1| 1| 0| 2| 3| 1|
[*02/08/2017 13:36:58.7479] Total STA List Count 0
|58:AC:78:DC:AA:A1|00000000| 3| 0| 1| 1| 0| 2| 3| 1|
[*02/08/2017 13:36:58.7479] | NO| MAC|STATE|
|3C:97:0E:D5:E8:21|00000000| 1| 0| 1| 1| 0| 0| 0| 0|
[*02/08/2017 13:36:58.7479] -----------------------------
|74:23:44:3B:06:07|00000000| 1| 0| 1| 1| 0| 0| 0| 0|
AP-COS
Client Information Details – 8.3MR2
b2802# show controllers dot11Radio 1 client 3C:A9:F4:64:F7:6C HT/VHT Rate Statistics:
... (Rate/SS/Width : Rx, Tx, Tx-Retries)
... 0/1/20 : 10, 0, 0
Legacy Rate Statistics: 1/1/20 : 1, 0, 0
(Mbps : Rx, Tx, Tx-Retries) 2/2/20 : 0, 9, 54
6 Mbps : 16, 0, 0 3/2/20 : 0, 2, 1
AP-COS
Radio Status
 Available in show tech
Associated client list:
====================
Total Client Count:0
Radio Total_clients Legacy_clients HT/VHT20_clients HT/VHT40_clients VHT80_clients
0 0 0 0 0 0
1 0 0 0 0 0
QBSS Load: cca_load: 0x3, rx_load: 0x0, tx_load: 0x0
AP-COS
Radio Status
 Statistics
Tx Watchdog stats:
Tx SW Watchdog 0 / 0
Tx HW Watchdog 0 / 0
Beacons missed: 0-30s 31-60s 61-90s 90s+
0 0 0 0
intf TxData TxUC TxMBC TxBytes TxMBBytes TxFail TxDcrd RxData RxUC RxMBC RxBytes RxErr UCTxRt
MCTxRt stats_ago
apr0v2 0 0 0 0 0 0 0 0 0 0 0 0 0
0 1.700000
AP-COS
Who is around
 RRM Neighbors
3802I_sw2_shield_gig15_C53#show rrm neighbor-list
========================================================
RRM Neighbors Slot 0
===================================
MAC Addr RSSI Srv.Chan
RRM Neighbors Slot 1
===================================
MAC Addr RSSI Srv.Chan
58:AC:78:DF:8C:1F -29 120
00:D7:8F:52:4F:6F -26 52
AP-COS
Radio Reset History
=================================================================================
AP58AC.78DC.AE00#show history interface dot11Radio all reset
=================================================================================
Timestamp Slot Client count Reset reason
Fri Jan 27 02:51:10 2017 0 0 84(radio failure (beacons stuck))
Fri Jan 27 02:51:27 2017 0 0 2(Radio firmware crashed)
AP-COS
Network Sub System
3802I_sw2_shield_gig15_C53#show controllers nss stats
NSS Statistics
Wired0 Port Statistics:
ID : 0 TYPE : 0
RX PKTS : 3428064 TX PKTS : 656862
RX OCTETS : 418299830 TX OCTETS : 151041084
RX ERR : 0 TX ERR : 0
Wired1 Port Statistics:
ID : 1 TYPE : 0
RX PKTS : 0 TX PKTS : 0
RX OCTETS : 0 TX OCTETS : 0
RX ERR : 0 TX ERR : 0
AP-COS
WLAN stats
AP00FE.C82D.EF30#sh interfaces dot11Radio 0 wlan 6 Packets Errored = 0
statistics
Rx errors = 0
VAP Level Stats: apr0v5 (under radio wifi0)
Rx Dropped = 0
Tx Data Packets = 120
Tx failures = 0
Tx Data Bytes = 54501
Tx Dropped = 0
Tx Data Payload Bytes = 49941
Host Discard = 0
Rx Data Packets = 166
Rx MIC Errors = 0
Rx Data Bytes = 27204
Last Tx rate for unicast Packets = 11000
Rx Data Payload Bytes = 24880
Last Tx rate for unicast Packets(mcs) = 22
Tx Unicast Data Packets = 120
Last Tx rate for multicast Packets = 48000
Rx Unicast Data Packets = 166
Last Tx rate for multicast Packets(mcs) = 96
Tx Multi/Broadcast Data Packets = 0
Total beacons sent to fw in SWBA intr = 3066
Rx Multi/Broadcast Data Packets = 0
Packets Dropped = 0
Key things to remember
Key “Take Aways”
 Health prevention is good for you and for your network
 There are ways to detect problems before they happen
 Key metrics can be good indicators of how things are doing
Complete Your Online
Session Evaluation
• Give us your feedback to be
entered into a Daily Survey
Drawing. A daily winner will
receive a $750 gift card.
• Complete your session surveys
through the Cisco Live mobile
app or on www.CiscoLive.com/us.
Don’t forget: Cisco Live sessions will be

available for viewing on demand after the
event at www.CiscoLive.com/Online.
Continue Your Education
• Demos in the Cisco campus
• Walk-in Self-Paced Labs
• Lunch & Learn
• Meet the Engineer 1:1 meetings
• Related sessions
Thank you

Brkewn 3012

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Brkewn 3012

Uploaded by

Copyright:

Available Formats

Performance

Javier Contreras Albesa

Cisco Spark spaces will be cs.co/ciscolivebot#BRKEWN-3012

 We want to stay Healthy

 Key Performance Indicators

 Defining KPIs for network infrastructure subsystems

 KPI as health indicators: is the network device

 Not good: going to doctor after 5 days with 42C fever

 Not optimal detection:

 Resources are Finite

 PI will not provide it all

 This is a per device check

 Health data now in 8.3.120.0

Multicore CPU Multicore CPU

Multicore CPU Multicore CPU

Multicore CPU Multicore CPU

Multicore CPU Multicore CPU

Dot1x MSCB OS Service

 Where to Put the Thermometer?

 Queue Utilization  Client Count

(Cisco Controller) >show cpu

Current CPU(s) load: 4%

Individual CPU load: 0%/0%, 0%/0%, 0%/1%, 100%/0%, 0%/1%, 0%/1%,

(WLC-5508-7) >show process cpu

Name Priority CPU Use (usr/sys)% hwm CPU Reaper

System Reset Task (240/ 7) 0 ( 0/ 0)% 0 9

reaperWatcher ( 3/ 96) 0 ( 0/ 0)% 0 7 I

osapiReaper ( 10/ 94) 0 ( 0/ 0)% 0 1 I

TempStatus (240/ 7) 0 ( 0/ 0)% 0 9 I

HAPeerToPeerCommTa ( 90/ 64) 0 ( 0/ 0)% 0 6

pktDebugSocketTask (255/ 1) 0 ( 0/ 0)% 0 6

webauthRedirect ( 95/ 62) 100 ( 95/ 4)% 100 9 T 300

emWeb (240/ 7) 0 ( 0/ 0)% 100 1 T 300

CSCuo86819 WLC crashes in 7.6.120.0 - memory corruption caused by Webauth

 Used for messaging between tasks

Total message queue count = 128

Q-Id Queue Name Allocated MsgSize InUse MaxUsed Breached* Details

2 GRE Queue 100 8 0 1 0 N

 Inuse: how many messages in the queue

Q-Id Queue Name Allocated MsgSize InUse MaxUsed Breached* Details

2 GRE Queue 100 8 0 5 0 N

70 nfv9queue 4096 316 0 2 0 Y

71 RogueApInfoChangedDB 4000 4 4000 4000 68 N

72 RogueApMeasChangedDB 4000 4 4000 4000 1 N

73 RogueClientInfoChangedDB 5000 4 5000 5000 1 N

74 RogueClientMeasChangedDB 5000 4 5000 5000 1 N

93 IDS Track Event Msg Queue 1024 0 8 0 N

 Dtlarpqueue: Data transformation layer ARP queue

 Radius messages between WLC and AAA server

Max Radius Queues Per Server..................... 16

Max Radius Buffers Available..................... 8128

Currently number of Buffers consumed............ 0

Radius Authentication Messages Stats

Total Auth Req sent(allocated).................. 140

Total Auth Resp rcvd(freed)..................... 140

Total Auth Req Pkts Dropped(no buffer).......... 0

(Cisco Controller) >show radius queue

Auth Alloc : 122401616

Acct Alloc : 31288769

Auth Free : 122401615

Acct Free : 31288769

Auth Alloc Err : 0

Acct Alloc Err : 0