AIX 5.3 Virtualization Updates & Tools: Advanced Technical Support, Americas

Advanced Technical Support, Americas
AIX 5.3
Virtualization Updates & Tools
Noel Ricca nricca@us.ibm.com

ATS System p Performance
Including materials from Luc Smolders, Dan Braden, Steve Nasypany
IBM Confidential 8/30/2007 © 2007 IBM Corporation

Recent Changes/Things to Know
Purpose of presentation is to provide information on new

tunings in AIX and changes to Performance Tools
Agenda
– Virtual Processor Issues and Tools
– Virtual I/O Server Tuning and Tools
– LPAR and CEC Recordings/Reports
– TL-06 Impacts on Tools
2 © 2007 IBM Corporation

Virtual Processors and Tools

Virtual Processors - Folding

Dynamically adjusting active Virtual Processors
– System consolidates loads onto a minimal number of VPs
• Scheduler computes utilization of VPs every second
– If VPs needed to host physical utilization is less than the current active VP count, a
VP is put to sleep
– If VPs needed are greater than the current active VPs, more are enabled
– On by default in AIX 5.3 ML3
• vpm_xvcpus tunable
Increases processor utilization and affinity
– Inactive VPs don’t get dispatched and waste physical CPU cycles
– Fewer VPs can be more accurately dispatched to physical resources by Hypervisor
When to adjust
– Burst/Batch workloads with short response-time requirements may need sub-
second dispatch latency
• Disable or manually tune the number of VPs
– # schedo –o vpm_xvcpus=[-1 | N]
– Where N specifies the number of VPs to enable in addition to the number
of VPs needed to consume physical CPU utilization

Virtual Processors - Tools

Folding
– Tools still show data by logical processors, whether folding is active or not!
– mpstat –s will show physical and associated logical utilization
• Disable/enable folding to see how loads are consolidated
Too many VPs
– High context switch rates
• mpstat –a: ilcs field monitor the number of involuntary logical context switches
• No rules-of-thumb, must baseline normal operation and watch for changes as loads increase
– Lock contention analysis with tools like trace and splat
Too few VPs
– Available processor cycles in pool
• Available pool processors: lparstat / topas –C available pool processor “app” value
• Shared physical processor utilization totals: topas –C/-R
– High CPU utilization, High entitled capacity, work not getting done
• Lparstat / vmstat high %user + %sys, %entc
Monitoring Context Switches with mpstat
– Hypervisor
• vlcs voluntarily gives cycles to shared pool or to another VP in the same partition
• ilcs involuntary, VP forced to yield (entitlement consumed, etc)
– Operating system
• cs – ics voluntary, thread yields (completes work, issues a read(), etc)
• ics involuntary, thread forced to yield (reaches time-slice dispatch length)

CPU Monitoring - mpstat command

¾Shows detailed logical processor information
¾up to 29 new metrics (when using -a option)
¾default mode shows
¾utilization metrics (%user, %sys, %idle, %wait)
¾major and minor page faults (with and without disk I/O)
¾number of syscalls and interrupts
¾dispatcher metrics
¾number of migrations
¾voluntary and involuntary context switches
¾logical processor affinity (percentage of redispatches inside MCM)
¾run queue size
¾fraction of processor consumed
¾percentage of entitlement consumed (shared mode)
¾number of logical context switches (shared mode)
¾hardware preemptions
¾Focus of AIX development, sar command is updated only as required to remain functional
¾-d shows detailed software and hardware dispatchers metrics, MCM affinity (dedicated only)
¾-i shows detailed interrupt metrics
¾-s shows SMT utilization

mpstat Example CPU user & sys values are relative to physical consumed
# mpstat 1 So a lcpu may look “busy” but actual usage is low

System configuration: lcpu=2 ent=0.5
cpu min maj mpc int cs ics rq mig lpa sysc us sy wa id pc %ec lcs U – Unused capacity
0 0 0 0 176 128 59 1 0 100 54 33 38 0 29 0.00 0.5 131
1 0 0 0 10 0 0 0 0 - 0 0 3 0 97 0.00 0.3 131 ALL – System Wide Usage
U - - - - - - - - - - - - 0 99 0.50 99.3 -
ALL 0 0 0 186 128 59 1 0 100 54 0 0 0 100 0.00 0.7 131
cpu Logical CPU number

min Minor page fault – no disk I/O required to satisfy
maj Major page fault – disk I/O required to satisfy fault
mpc Total number of mpc interrupts – Interprocessor calls
int Total number of interrupts

cs Total number of context switches
ics Total number of involuntary context switches
rq Total number of processes on the run queue
mig Total number of thread migration to another logical processor
lpa Total number of re-dispatches within affinity domain 3
sysc Total number of system calls
us / sys / wa / id The percentage of physical processor utilization consumed. Note: interpret relative to pc.
Fraction of physical processor consumed (shared partition or when SMT enabled). The pc of the cpuid
pc
U row represents the number of unused physical processors relative to entitlement
%ec The percentage of entitled capacity consumed.
lcs Total number of logical CPU context switches
mpstat –s
# mpstat -s
Proc0 Proc2 Proc4 Proc6

80% 78% 75% 82% [shared mode only]
cpu0 cpu1 cpu2 cpu3 cpu4 cpu5 cpu6 cpu7
40% 40% 68% 10% 35% 40% 41% 41%
(delta PURR / delta time-base) * 100

Represents the percentage of dispatch cycles given to a
logical processor
Interpreted as the percentage of physical processor
consumed by a logical processor
Only AIX tool that will dynamically show Virtual Processor
Folding (adjust vpm_xvcpus tunable)
Schedo –p –o vpm_xvcpus = 0 | -1
Physical Processor / Virtual Process Busy – with SMT enabled each physical process has
proc
two logical processors.
Logical CPU number and the overall busy percentage, which is the sum or user + system
cpu
mode utilization. Gives the relative SMT split between processors.

lparstat Review
root@sq09.dfw.ibm.com /tmp # lparstat -h 1 4
System configuration: type=Shared mode=Capped smt=On lcpu=4 mem=4096 psize=2 ent=0.40
%user %sys %wait %idle physc %entc lbusy app vcsw phint %hypv hcalls
Additional information when
----- ---- ----- ----- ----- ----- ------ --- ---- ----- ----- ------ “-h” flag is specified
84.9 2.0 0.2 12.9 0.40 99.9 27.5 1.59 521 2 13.5 2093
86.5 0.3 0.0 13.1 0.40 99.9 25.0 1.59 518 1 13.1 490
%user / %sys / Shows the percentage of the entitled processing capacity used. So, you would say that the system is consuming
%wait / %idle 86.9% (84.9 + 2) of four-10th of a physical processor. For dedicated partitions, the entitled capacity = # of physical
processors
physc Shows the number of physical processors consumed. For a capped partition this number will not exceed the entitled
capacity. For an uncapped partition this number could match the number of processors in the shared pool; however,
this my be limited based on the number of on-line Virtual Processors.
%entc Shows the percentage of entitled capacity consumed. For a capped partition the percentage will not exceed 100%;
however, for uncapped partitions the percentage can exceed 100%.
lbusy Shows the percentage of logical processor utilization that occur while executing in user and system mode. Note: In
this example we’re using approx 25% of the logical processors. This is the “traditional” measure of CPU utilization Shared
using time-based sampling. As this value approaches 100% it may indicate that the partition could make use of Mode
additional VPs. Only
app Shows the number of available processors in the shared pool. The shared pool ‘psize’ is 2 processors. Must set
‘Allow shared processor pool utilization authority’. View the “properties” for a partition and click the Hardware tab,
then Processors and Memory.
vcsw Shows the number of virtual context switches.
phint Shows the number of phantom interrupts. A phantom interrupt is an interrupt that belongs to another shared partition.
%hypv / hcalls Shows the percentage of time spent in the hypervisor and the number of hypervisor calls.

Virtual I/O Server

Virtual SCSI - CPU

Know the applications I/O behavior
– Review information on application block sizes
– Determine I/O Per Second (IOPS) and bytes transferred
• iostat -D
• filemon
– Otherwise, you need to size to maximum I/O rates
• Will waste entitlement when using dedicated partitions
• Likely just guessing with shared partitions
Disk guidelines for IOPS
• Use 60 IOPS for planning for 7200 RPM disks
• Use 75 IOPS for planning for 10,000 RPM disks
• Use 90 IOPS for planning for 15,000 RPM disks
• Most disk subsystems get better IOPS on their disks
– Use 150 IOPS for DS4000, DS6000 and DS8000 disks

Virtual SCSI - CPU

CPU sizing method for VSCSI server
– Determine approximate cycles per second for an I/O type
• Available in VIOS Planning Guide
– http://www14.software.ibm.com/webapp/set2/sas/f/vios/documentation/
home.html
– Allocation =
• (# of IOPS X Cycles per second) / CPU Frequency

Virtual SCSI - CPU

Number of 4KB 8KB 32KB 64KB 128KB
CPU cycles
per operation*
Physical Disk 45000 47000 58000 81000 120000
LVM 49000 51000 59000 74000 105000
* Based on 1.65 GHz POWER5
Example: Two clients, using physical disk storage

Client1: peak of 10,000 IOPS of size 32KB
Client2: peak of 5,000 IOPS of size 64KB
Allocation = (# of IOPS X Cycles per second) / CPU Frequency
(10,000 X 58,000 + 5,000 X 81,000)/1,650,000,000 = 0.60 processors
* Can adjust cpu cycles required for other processors by computing a ratio between processor speeds

Virtual SCSI – Tuning/Tools

Measure I/O
– Just as you do with a non-VIOS system
– Use iostat –aD for detailed disk and adapter information
• Mitigate any high IOPS
– Transfers per second (tps)
– Now broken down by read/write transfers, most customers don’t know
• Look for unbalanced IO on PVs
– May indicate data layout issues
• Look for high wait/service times, high queue full (sqfull)
• Adapter totals (-a) can be associated with HW limits
– Use filemon trace tool
• To determine average block sizes
• Identify hot LVs and PVs
– Look for high read/write times
> Writes > 2 msec (cached), > 10 msec (non-cached)
> Reads > 20 msec
– Look for large deltas between PV and LV layers
> File system buffer tunings (cross check vmstat –v counters)
Server-specific tools
– viostat – wrapper around iostat
– topas – global values, disk and network metrics
– netstat – network statistics
– Most modes can be run as padmin, but root shell may have more options
I/O Tuning – iostat -aD Read/write IOPS

# iostat -a –D rps/wps
System configuration: lcpu=2 drives=3 paths=1 vdisks=1
Adapter: PV
scsi0 xfer: bps tps bread bwrtn Virtual adapter
0.0 0.0 0.0 0.0
Paths/Disk: Paths
hdisk0_path0 xfer: %tm_act bps tps bread bwrtn
0.0 0.0 0.0 0.0 0.0
read: rps avgserv minserv maxserv timeouts fails
0.0 0.0 0.0 0.0 0 0
write: wps avgserv minserv maxserv timeouts fails
0.0 0.0 0.0 0.0 0 0
queue: avgtime mintime maxtime avgwqsz avgsqsz sqfull
0.0 0.0 0.0 0.0 0.0 0
Vadapter
vsci0 xfer: tps bread bwrtn partition-id
0.0 0.0 0.0 ####
read: avgserv minserv maxserv Use –l option for wide
0.0 0.0 0.0 column, one device
write: avgserv minserv maxserv per line format
0.0 0.0 0.0
queue: avgtime mintime maxtime avgsqsz qfull
0.0 0.0 0.0 0.0 0
Disk:
hdisk10 xfer: %tm_act bps tps bread bwrtn
0.0 0.0 0.0 0.0 0.0
0.0 0.0 0.0 0.0 0 0
0.0 0.0 0.0 0.0 0 0
0.0 0.0 0.0 0.0 0.0 0
© 2007 IBM Corporation

I/O Tuning – iostat -D Service times you could only get from filemon before
hdisk1 xfer: %tm_act bps tps bread bwrtn

87.7 62.5M 272.3 62.5M 823.7
271.8 9.0 0.2 168.6 0 0
0.5 4.0 1.9 10.4 0 0
1.1 0.0 14.1 0.2 1.2 2374
Virtual adapter’s extended throughput report (-D)
Metrics related to transfers (xfer:) All –D outputs are rates, except
tps Indicates the number of transfers per second issued to the adapter. sqfull, which is an interval delta
recv The total number of responses received from the hosting server to this adapter.
sent The total number of requests sent from this adapter to the hosting server.
partition id The partition ID of the hosting server, which serves the requests sent by this adapter. Can’t exceed queue_depth for the disk
Adapter Read/Write Service Metrics (read:)
avgserv Indicates the average time. Default is in milliseconds.
If this is often > 0, then increase queue_depth
minserv Indicates the minimum time. Default is in milliseconds.
maxserv Indicates the maximum time. Default is in milliseconds.
Adapter Wait Queue Metrics (wait:)
avgtime Indicates the average time spent in wait queue. Default is in milliseconds. Average IO Sizes
mintime Indicates the minimum time spent in wait queue. Default is in milliseconds. read = bread/rps
maxtime Indicates the maximum time spent in wait queue. Default is in milliseconds.
avgwqsz Indicates the average wait queue size. write = bwrtn/wps
qvgsqsz Indicates the average service queue size – Waiting to be sent to the disk.
sqfull Indicates the number of times the service queue becomes full.

I/O Tuning – filemon LV

#filemon –T25000000 –Pv –Opv,lv –o myfile.out; sleep 60; trcstop
------------------------------------------------------------------------
Detailed Logical Volume Stats (512 byte blocks) Response Time (msecs)
------------------------------------------------------------------------
Reads < 20 msecs
VOLUME: /dev/ofs002lv description: N/A Write w/cache < 2 msecs
reads: 10172 (0 errs)
read sizes (blks): avg 69.8 min 8 max 256 sdev 101.8
read times (msec): avg 2.513 min 0.213 max 108.228 sdev 5.321
read sequences: 6296
read seq. lengths: avg 112.7 min 8 max 10240 sdev 731.9
writes: 16480 (0 errs)
write sizes (blks): avg 17.9 min 8 max 256 sdev 15.2
write times (msec): avg 1.014 min 0.228 max 35.433 sdev 1.539
write sequences: 16414
write seq. lengths: avg 17.9 min 8 max 704 sdev 18.5
seeks: 22502 (84.4%)
seek dist (blks): init 512740232,
avg 10093106.9 min 8 max 480236768 sdev 52384904.5
time to next req(msec): avg 2.263 min 0.014 max 171.678 sdev 5.326
throughput: 8306.9 KB/sec
utilization: 0.51
Key Indicator of a performance bottleneck Sequential vs. Random Workload

Response times are too long # of Writes = # of Write Sequences, workload is random
Significant delta between the Logical and # of Writes = 1, workload is sequential
Physical layer response times.

I/O Tuning – filemon PV

Average IO sizes
Blks are 512 bytes in AIX
69 x 512 = ~32KB average size
------------------------------------------------------------------------
Detailed Physical Volume Stats (512 byte blocks)
------------------------------------------------------------------------
VOLUME: /dev/hdisk1 description: N/A

reads: 10172 (0 errs)
read sizes (blks): avg 69.8 min 8 max 256 sdev 101.8
read times (msec): avg 2.188 min 0.001 max 108.213 sdev 4.835
read sequences: 6296
read seq. lengths: avg 112.7 min 8 max 10240 sdev 731.9
writes: 16487 (0 errs)
write sizes (blks): avg 17.9 min 8 max 256 sdev 15.2
write times (msec): avg 0.400 min 0.001 max 19.784 sdev 0.783
write sequences: 16421
write seq. lengths: avg 18.0 min 8 max 704 sdev 18.5
seeks: 22509 (84.4%)
seek dist (blks): init 791801736,
avg 10156034.0 min 8 max 6810 sdev 5275.1
seek dist (%tot blks):init 56.76268,
avg 0.72807 min 0.00000 max 48.82100 sdev 3.78211
time to next req(msec): avg 2.262 min 0.013 max 171.681 sdev 5.325
throughput: 8309.9 KB/sec
utilization: 1.00

Virtual Ethernet
Packets transferred in memory between partitions on the same
server
– Higher throughput than physical ethernet
– Physical devices do not support MTU 65394
Throughput linearly scales with processor entitlements
– MTU 9000 is 3X MTU 1500
– MTU 65394 is 7X MTU 9000
– Try to use the highest MTU
No unique TCP/IP tunables methodology
TCP Checksum Offloading
– Because virtual network does not suffer from physical network
link errors, checksums do not need to be generated (this is the
default in later AIX 5.3 levels)
• # chdev –l <device> -a chksum_offload=yes

Shared Ethernet
Heavy network load, use same sizings as dedicated systems
– MTU 1500, 1 CPU
– MTU 9000, 0.5 CPU
Shared processors
– Shared processors can result in higher latency, decreasing throughput
– For bursty network loads, use uncapped and allow for more entitlement than
would be allocated for a dedicated partition hosting the same application
Tools
• lsattr –El en#
• topas
• entstat, netstat
• seastat
– Tool from Nigel Griffith simplifies output, provides intervals
– http://www-941.ibm.com/collaboration/wiki/display/WikiPtype/nmon
Whenever there is a VIO Client/Server issue, check if there is a CPU constraint
first
– Add entitlement (shared) or increase CPUs (dedicated)
– Use larger MTU sizes if possible

Tools for
LPAR & CEC
Historical Performance

LPAR & CEC Historical Tools

AIX
– New AIX function with LPAR and CEC performance history
– Local Recording/Reports
– CEC Recording/Reports
– SMIT configuration updates (TL06)
Other
– IBM – Free
• Nigel Griffiths Nmon and seastat
• Stephen Atkins Nmon Analyzer and consolidation tools
• Lparmon – IBM Briefing Center, Alphaworks
• Gmon – Alphaworks?
– Tivoli
• Integrated Tivoli Monitor (ITM SE)
– Open Solutions
• Ganglia (see earlier Wiki link for more info)

Topas Local Recording

What?
– Records a pre-defined set of AIX performance metrics, 24 X 7
– Does not actually use topas itself to record the data
• But records the metric-set commonly monitored by topas
• Hence referred to as ‘topas’ or ‘local’ recording
Why?
– Customers need historical data and reports
• Record data needed to create ‘RMF-like’ reports (ala System z)
When?
– Introduced in AIX 5.3 ML 4
How do I enable it?
– Automatic with install of AIX 5.3 TL5
– Manually configurable with
• /usr/lpp/perfagent/config_aixwle.sh [add|delete]

Local Recording
Metrics
– CPU/LPAR
• User, system, idle, wait, logical/physical busy, available pool processor,
pool size
• Entitlement, entitlement consumed
• CPU counts: physical, logical, virtual
– Memory
• Real: size, %client, %comp, %noncomp, %pinned, # frames
• Virtual: file page in/out, page in/out, page faults, frames stolen, IO requests
initiated by VMM
– System Calls/IO
• Total, read, write, fork, exec, read/write bytes
– Global Processes
• Number of processes, run queue, swap queue, process context switches
• PLM supplied utilization and load average
• No per-process/thread metrics

Local Recording
Metrics (cont.)
– Disk
• Busy, read/write blocks, average wait time, average service time
– Network
• LAN: KB in/out, transmit drops, receive drops, frames in/out
• Network Interface: input/output packets, input/output KB
• UDP: send/receive datagrams, full input socket count
• TCP: send/receive packets, bytes, totals
– Filesystem
• Size, free, %used
– Paging Space
• Size, %free, total size, total free
– NFS
• Client/Server calls, NFS v2 &v3 only
– Workload Manager
• CPU, memory, disk consumed percentage per superclass

Local Recording
Where is the data?
– Agent records to /etc/perf/daily/ directory
– Recordings name format: xmwlm.YYMMDD
How long is the recording kept?
– Initially, each 24-hour ‘daily’ recording was retained for 48 hours
– Now supports retension of one week of daily recordings
• Last 7 days
• APAR IY91882
– Customers should generate reports daily or archive recordings
as their needs require
How much disk space does this take?
– Dependent upon the number of devices
• System with a low number of disks: 2 MB/day
• 10 MB/day for every 100 disks

Topasout Local Reports

What?
– Common report generation tool for new AIX recordings
– Based on Performance Toolbox post-processor utility ptxtab
When?
– AIX 5.3 ML4 and TL5
Time filtering options
– -i MM Generate reports with time intervals of 5, 10, 15, 30 or 60 minutes
– -b HHMM Begin time of report in hours and minutes (range of 0000-2400)
– -e HHMM End time of report in hours and minutes (range of 0000-2400)
Report Types
– Formatted
• AIX 5.3 TL5
• ‘RMF-style’ text reports
• -R [ detailed | summary | disk | lan ] recording_file_name
• Output is written to standard out
– Redirect to customize file naming

Local Reports
Report Types (cont.)
– Nmon Analyzer
• AIX 5.3 TL5
• Format supported by Stephen Atkins Excel utility
– Same tool that supports Nmon
– Breakdown and graphs by system resources
• topasout -a recording_file_name
• Output is written directly to a file
– Format: xmwlm.YYMMDD.csv
– Ready for Import into Analyzer, currently version 3.2.5
• Http://www-941.haw.ibm.com/collaboration/wiki/display/Wikiptype/nmonanalyser
• APAR IY87993

Local Reports
Report Types (cont.)
– Legacy PTX
• AIX 5.3 ML4
• Pre-existing formats common with Performance Toolbox
reports
• -c recording_file_name Comma-separated output
• -s recording_file_name Spreadsheet import
• Output is written directly to a file
– Format: xmwlm.YYMMDD_01

Topasout Local Report – Detailed

•Detailed report: topasout –R detailed
Report: System Detailed --- hostname: ptoolsl1 version 1.0
Start:12/21/05 10.00.00 Stop:12/21/05 11.00.00 Int: 5 Min Range: 60 Min
Time: 10.00.00 --------------------------------------------------------------
CPU UTIL MEMORY PAGING EVENTS/QUEUES NFS
Kern 12.0 PhyB 0.7 Sz,GB 16.0 Sz,GB 4.0 Cswth 3213 SrvV2 32
User 8.0 Ent 0.5 InU 4.3 InU 2.3 Syscl 43831 CltV2 12
Wait 0.0 EntC 15.2 %Comp 3.1 Flt 221 RunQ 1 SrvV3 44
Idle 78.0 LP 4 %NonC 9.0 Pg-I 87 WtQ 0 CltV3 18
SMT ON Mode Shr %Clnt 2.0 Pg-O 44 VCSW 1214
Network KBPS I-Pack O-Pack KB-I KB-O

en0 0.6 7.5 0.5 0.3 0.3 Layout similar to topas
en1 22.3 820.1 124.3 410.0 61.2
lo0 0.0 0.0 0.0 0.0 0.0
Disk Busy% KBPS TPS KB-R KB-W

hdisk0 0.0 0.0 0.0 0.0 0.0
hdisk1 0.0 0.0 0.0 0.0 0.0
Time: 10.05.00 --------------------------------------------------------------
CPU UTIL MEMORY PAGING EVENTS/QUEUES NFS
Idle 78.0 LP 4 %NonC 9.0 Pg-I 87 WtQ 0 CltV3 18
SMT ON Mode Shr %Clnt 2.0 Pg-O 44 VCSW 1214

en0 0.6 7.5 0.5 0.3 0.3
en1 22.3 820.1 124.3 410.0 61.2
lo0 0.0 0.0 0.0 0.0 0.0

hdisk0 0.0 0.0 0.0 0.0 0.0
hdisk1 0.0 0.0 0.0 0.0 0.0

Topasout Local Report - Summary

zDedicated partitions: topasout –R summary
Report: System Summary - hostname: ptoolsl1 version 1.0

Mem: 16.2 GB Dedicated SMT:OFF Logical CPUs: 2
Time InU Us Sy Wa Id PhysB RunQ WtQ CSwitch Syscall PgFault
-------------------------------------------------------------------------------
14.00.00 21.1 11 8 0 81 0.2 1 0 3432 5050 17
14.05.00 21.1 16 5 0 79 0.3 1 0 532 3104 14
14.10.00 21.2 13 7 0 20 0.2 1 0 652 4326 13
zShared partitions: topasout –R summary

Report: System Summary - hostname: ptoolsl1 version 1.0
Psize:1.0 Mem: 16.2 GB Shared SMT:OFF Logical CPUs: 2
Time InU Us Sy Wa Id PhysB Ent %EntC RunQ WtQ CSwitch Syscall PgFault
-------------------------------------------------------------------------------
10.00.00 21.1 11 8 0 81 0.2 0.5 23.2 1 0 3432 5050 17
10.05.00 21.1 16 5 0 79 0.3 0.5 25.0 1 0 532 3104 14
10.10.00 21.2 13 7 0 20 0.2 0.5 23.4 1 0 652 4326 13

Topasout Local Report – Adapter I/O

zDisk report: topasout –R disk
Report: Total Disk I/O Summary - hostname: ptoolsl1 version:1.0

Start:04/25/06 00.00.00 Stop:04/26/06 00.00.00 Int:05 Min Range:1440 Min
Mem: 8.0 GB Dedicated SMT:ON Logical CPUs:16
Time InU PhysB %Bsy MBPS TPS MB-R MB-W
-------------------------------------------------------------------------------
00.00.05 6.5 12.50 45.5 120.5 300.1 100.1 20.4
00.00.10 6.7 13.40 55.0 240.0 320.2 240.0 0.0
00.00.15 7.0 14.70 60.4 160.2 350.3 40.1 120.1
00.00.20 7.4 15.50 72.3 200.7 410.5 20.3 180.4
zLAN report: topasout –R lan

Report: Total LAN I/O Summary - hostname: ptoolsl1 version:1.0
Start:03/12/06 17.15.00 Stop:03/12/06 20.30.00 Int:05 Min Range: 195 Min
Psize:1.0 Mem: 16.2 GB Shared SMT:OFF Logical CPUs: 2
Time InU PhysB MBPS I-Pack O-Pack MB-I MB-O Rcvdrp Xmtdrp
-------------------------------------------------------------------------------
17.15.00 3.2 6.30 20.0 310.5 120.2 16.2 3.8 120 160
17.20.00 3.3 6.45 22.3 220.3 225.7 11.1 11.2 118 165
17.25.00 3.2 6.15 18.5 275.6 158.0 11.6 6.9 121 162
17.30.00 3.4 6.55 19.4 270.2 156.9 11.3 6.1 124 154

Topas CEC Monitoring/Recording

What?
– Collect and aggregate performance data from AIX LPARs within a CEC
Why?
– Customers need overall CEC utilization
When?
– Monitoring: AIX 5.3 ML3
– Recording: AIX 5.3 TL5
– SMIT Panels: AIX 5.3 TL6
How to enable pre-TL6?
– Manually configurable with
• Monitoring
– User –C option
• Recording
– /usr/lpp/perfagent/config_topas.sh [ add | delete ]
– Topas –R option
– Adds /etc/inittab entry

Topas CEC Monitoring Review (AIX 5.3 ML 3)

z topas
–C
zUpper section displays aggregated CEC information
zLower section displays shared/dedicated data – closely mimics lparstat
Topas CEC Monitor Interval: 10 Thu Jul 28 17:04:57 2006
Partition Info Memory (GB) Processor
Monitored : 6 Monitored :24.6 Monitored :1.2 Shr Physical Busy: 0.27
UnMonitored: - UnMonitored: - UnMonitored: - Ded Physical Busy: 2.70
Shared : 3 Available :24.6 Available : -
Dedicated : 3 UnAllocated: 0 UnAllocated: - Hypervisor
Capped : 1 Consumed : 2.7 Shared :1.5 Virt. Context Switch: 632
Uncapped : 1 Dedicated : 5 Phantom Interrupts : 7
Pool Size : 3
Avail Pool :2.6
Host OS M Mem InU Lp Us Sy Wa Id PhysB Ent %EntC Vcsw PhI
-------------------------------------shared-------------------------------------
ptoolsl3 A53 c 4.1 0.4 2 14 1 0 84 0.08 0.50 15.0 208 0
ptoolsl2 A53 C 4.1 0.4 4 20 13 5 62 0.17 0.50 36.5 219 5
ptoolsl5 A53 U 4.1 0.4 4 0 0 0 99 0.02 0.50 0.1 205 2
------------------------------------dedicated-----------------------------------
ptoolsl1 A53 S 4.1 0.5 4 20 10 0 70 0.60
ptoolsl4 A53 4.1 0.5 2 100 0 0 0 2.00
ptoolsl6 A52 4.1 0.5 1 5 5 12 88 0.10
•M – System Mode
•c means capped, C - capped with SMT
•u means shared, U - uncapped with SMT
•S means SMT
Topas CEC Reports

What?
– Reports that mimic topas CEC monitoring format as closely as possible
When?
– Command line: AIX 5.3 TL5
– SMIT Panels: AIX 5.3 TL6
Time filtering options

– -i MM Generate reports with time intervals of 5, 10, 15, 30 or 60 minutes
– -b HHMM Begin time of report in hours and minutes (range of 0000-2400)
– -e HHMM End time of report in hours and minutes (range of 0000-2400)
Report Types
– Formatted
• -R [ detailed | summary ] recording_file_name
• Written to standard out, redirect to customize file naming
– Legacy PTX
• -c recording_file_name Comma-delimited format
• -s recording_file_name Spreadsheet format
• Output format: topas_cec.YYMMDD_01
• APAR IY93513 includes recent fixes
– No Nmon Analyzer format at this time

Topasout CEC Report - Detailed

zDetailed CEC: topasout –R detailed [recording name]
Report: Topas CEC Detailed --- hostname: ptoolsl1 version:1.0
Start:05/02/06 07.00.00 Stop:05/02/06 17.00.00 Int:05 Min Range:600 Min
Time: 07.00.00 -----------------------------------------------------------------
Partition Info Memory (GB) Processors
Monitored : 8 Monitored : 0.0 Monitored : 7 Shr Physical Busy: 2.2
UnMonitored: 0 UnMonitored: 0.0 UnMonitored: 0 Ded Physical Busy: 0.4
Shared : 3 Available :32.0 Available : 7
Dedicated : 2 UnAllocated: 0 UnAllocated: 1 Hypervisor Layout similar to topas -C
Capped : 1 Consumed : 8.7 Shared : 4 Virt. Context Switch:332
Pool Size : 2
Avail Pool : 1
--------------------------------shared------------------------------------------
ptools1 A53 u 1.1 0.4 4 15 3 0 82 1.30 0.50 22.0 200 5
ptools5 A53 U 12 10 1 12 3 0 85 0.20 0.25 0.3 121 3
ptools3 A53 C 5.0 2.6 1 10 1 0 89 0.15 0.25 0.3 52 2
-------------------------------dedicated----------------------------------------
ptools4 A53 S 0.6 0.3 2 12 3 0 85 0.60
ptools6 A52 1.1 0.1 1 11 7 0 82 0.50
ptools8 A52 1.1 0.1 1 11 7 0 82 0.50
Time: 07.05.00 -----------------------------------------------------------------
Partition Info Memory (GB) Processors
UnMonitored: - UnMonitored: 0.0 UnMonitored: 0 Ded Physical Busy: 0.4
Shared : 3 Available :32.0 Available : 7
Dedicated : 2 UnAllocated: - UnAllocated: 1 Hypervisor
Capped : 2 Consumed : 8.7 Shared : 4 Virt. Context Switch:332
Pool Size : 2
Avail Pool : 1
--------------------------------shared------------------------------------------
ptools1 A53 u 1.1 0.4 4 15 3 0 82 1.30 0.50 22.0 200 5
ptools5 A53 U 12 10 1 12 3 0 85 0.20 0.25 0.3 121 3
ptools3 A53 C 5.0 2.6 1 10 1 0 89 0.15 0.25 0.3 52 2
-------------------------------dedicated----------------------------------------
ptools4 A53 S 0.6 0.3 2 12 3 0 85 0.60
…
Topasout CEC Report - Summary

z Summary CEC: topasout –R summary [recording name]
Report: Topas CEC Summary --- hostname: ptoolsl1 version:1.0
Start:02/09/06 00.00.00 Stop:02/09/06 23.55.00 Int: 5 Min Range:1440 Min
Partition Mon: 7 UnM: 1 Shr: 4 Ded: 3 Cap: 3 UnC: 1
-CEC------ -Processors-------------------- -Memory (GB)------------
Time ShrB DedB Mon UnM Avl UnA Shr Ded PSz APP Mon UnM Avl UnA InU
00.05.00 3.2 1.1 5 2 7 1 4 3 2 1 16.0 0.0 32.0 0.0 8.1
00.10.00 2.9 0.9 5 2 7 1 4 3 2 1 16.0 0.0 32.0 0.0 8.3
00.15.00 2.1 1.3 5 2 7 1 4 3 2 1 16.0 0.0 32.0 0.0 8.5
...
configuration change at 02.15.00

Partition Mon: 8 UnM: 0 Shr: 4 Ded: 4 Cap: 3 UnC: 1
-CEC------ -Processors-------------------- -Memory (GB)------------
Time ShrB DedB Mon UnM Avl UnA Shr Ded PSz APP Mon UnM Avl UnA InU
02.15.00 3.1 2.5 7 0 7 1 4 5 2 1 18.0 0.0 32.0 0.0 9.1
02.20.00 1.9 1.5 7 0 7 1 4 5 2 1 18.0 0.0 32.0 0.0 6.8
02.25.00 2.0 3.3 7 0 7 1 4 5 2 1 18.0 0.0 32.0 0.0 7.8
...

Topas CEC Recording - Problems

Topas –C | -R do not resolve remote hosts
– The CEC function reuses a registered inetd protocol, known as xmquery
• Part of the Performance Toolbox/Aide product for System p AIX
• If the Performance Aide is installed, xmservd is the operating agent started by xmquery
• If the Performance Aide is not installed, xmtopas is the AIX agent started by xmquery
– Topas utilizes the Remote Statistics (RSi) API to connect the local and remote agents
• When initialized, this API issues an xmquery protocol on a specific port. Systems inetd
service see this query, and start the agent
• Agent then replies to host which issued the query
– The xmquery protocol call defaults to poll systems only within the local subnet
• If LPARs within the same CEC operate on multiple subnets:
– Create a $HOME/Rsi.hosts file
– List of fully qualified hostname or IP address for remote systems, one entry per line!
– API will initialize and use this file to poll systems in other subnets
Topas is slow to start, recognize new hosts
– Reconfiguration code has been optimized in TL5. While startup time may require up to one
minute, new systems should be recognized within 1-2 minutes.
Check for service updates to AIX filesets for latest fixes
– bos.perf.tools contains topas
– perfagent.tools contains metrics instrumentation and API

AIX TL06

40 Advanced Technical Support, Americas
AIX Recordings - 5.3 TL6 Update

IBM Global Services
•SMIT panels Configure Topas Options
• setup access to partitions Add Host to topas external subnet search file
(Rsi.hosts)
not on local subnet List hosts in topas external subnet search file
• turn on/off CEC and local recordings

(Rsi.hosts)
Configure Recordings
• display recording status List Available Recordings
• generate reports
Show current recordings status
Generate Report
►to file
►to printer Configure Recordings
►to stdout
• eliminates Enable CEC Recording

Disable CEC Recording
►need to know file location and names Enable Local Recording
Disable Local Recording
►topasout syntax
Type Date Start Stop

local 07/01/23 00:02:42 23:58:53
Recording Status
local 07/01/24 00:03:18 23:59:45
========= ======
local 07/01/25 00:04:45 23:56:11
CEC Not Enabled
local 07/01/26 00:01:11 23:57:37
Local Not Enabled
local 07/01/27 00:04:13 23:59:34
WLE Not Enabled
local 07/01/28 00:04:04 23:57:52
Barcelona 2006
local 07/01/29 00:02:52 01:12:54
What's new in AIX 5.3 © IBM Corporation 2007

Dedicated idle cycles donation

IBM Global Services
•Similar in concept to idle partitions ceding idle cycles to shared pool

•Differences
• there is a guaranteee not to get phantom interrupts (interrupts for other partitions)
• the partition keeps running on the same physical processors
• must be enabled on HMC
•New phyp instrumentation collects
• donated cycles
►voluntarily donated by an idle dedicated partition to shared pool
• stolen cycles
►cyclesstolen by phyp from a dedicated partition to run maintenance tasks (hypervisor overhead)
►can happen whether donation is enabled or not (just wasn’t instrumented before)
•Tools metrics impact

• processors belonging to donating dedicated partitions are counted in pool size
• PURR stops on context switches
►similarto what happens to shared partitions
►tools will compensate so that dedicated percentages are still relative to total capacity
•Tools updated
• lparstat, mpstat andBarcelona
sar 2006
• topas and topasout reports
Dedicated idle cycles donation - lparstat

IBM Global Services
•New mode # lparstat 1 3
$ lparstat -i
System configuration: type=Dedicated mode=Donating
Node Name : smt=On lcpu=2 mem=800
va01
Partition Name :
va %user %sys %wait %idle physc vcsw
Partition Number : 2 ----- ---- ----- ----- ----- -------
Type : 0.1 0.4 0.0 99.5 0.68 670234
Dedicated-SMT 0.0 0.2 0.0 99.8 0.68 670234
Mode : 0.0 0.2 0.0 99.8 0.68 670234
Donating
Entitled Capacity :
1.00
Partition Group-ID :
32770
Shared Pool ID : - donation causes
Online Virtual CPUs : 1 hardware context
Maximum Virtual CPUs : 1 switches
Minimum Virtual CPUs : 1
Online Memory :
800 MB Stay relative to
Maximum Memory : partition capacity.
1024 MB
Minimum Memory :
128 MB In this case one shows actual physical processor
Variable Capacity Weight : - processor consumption:
Minimum Capacity :
1.00
Maximum Capacity :
number of physical processors
1.00 minus donated and stolen cycles
Capacity Increment :
1.00
Barcelona 2006
Maximum Physical CPUs in system : 4
Active Physical CPUs in system : 4
Active CPUs in Pool : -
Unallocated Capacity : -
Physical CPU Percentage :
What's new in AIX 5.3
100.00% © IBM Corporation 2007
U ll t d W i ht
IBM Global Services
Dedicated idle cycles donation - lparstat details
•New -d flags shows more details %idon, %bdon: percentages of
idle and busy times donated
•Example with donation enabled %istol, %bstol: percentages of

idle and busy times stolen
# lparstat –d
System configuration: type=Dedicated mode=Donating smt=On lcpu=2 mem=800
%user %sys %wait %idle %idon %bdon %istol %bstol

----- ---- ----- ----- ------ ----- ----- ------
0.1 0.2 2.1 97.7 12.79 6.8 4.8 2.75
•Example without donation and in combination with -h

# lparstat -dh
System configuration: type=Dedicated mode=Capped smt=On lcpu=2 mem=800
%user %sys %wait %idle %hypv hcalls %istol %bstol

----- ---- ----- ----- ----- ------ ------ ------
0.1 0.2 2.1 97.7 0.0 391 4.8 2.75
Barcelona 2006

Dedicated idle cycles donation - sar and mpstat

IBM Global Services
•sar
• automatically displays phyc when donation is enabled
•mpstat
• automaticaly displays pc and lcs if donation is enabled
• new -h option to show more details on hypervisor related statistics
►donation enabled
System configuration: lcpu=2 mode=Donating
cpu pc ilcs vlcs idon bdon istol bstol
0 0.3 50327 687231635 10.2 4.5 0.59 0.32
1 0.5 61702 684989764 10.2 4.5 0.59 0.32
ALL 0.8 112029 1372221399 20.4 9.0 1.18 0.64
idon, bdon: percentages of idle

►donation disabled
and busy times donated:
System configuration: lcpu=2 mode=Capped
cpu pc ilcs vlcs istol bstol istol, bstol: percentages of idle
0 0.3 503727 687231635 0.59 0.32 and busy times stolen
1 0.41 61702 684989764 0.59 0.32
ALL 0.71 565429 1372221399 1.18 0.64
►shared partition
System configuration: lcpu=2 ent=0.5 mode=Uncapped
cpu Barcelona
pc ilcs 2006
vlcs
0 0.6 503727 687231635
1 0.6 61702 684989764
ALL 0.8 565429 1372221399

Dedicated idle cycles donation - topas -L

IBM Global Services
Interval: 2 Logical Partition: Fri Sep 22 09:01:46 2006

Donating SMT ON Online Memory: 3200.0
Partition CPU Utilization Online Virtual CPUs: 1 Online Logical CPUs: 2
%user %sys %wait %idle %hypv hcalls %istl %bstl %idon %bdon vcsw
1 1 0 98 1 200 0 2.1 3.5 10.0 1.0 includes same
updates as
===============================================================================
LCPU minpf majpf intr csw icsw runq lpa scalls usr sys _wt idl pclparstat and
Cpu0 0 0 190 176 84 0 100 5089 1 2 0 97 0.52mpstat
Cpu1 0 0 14 0 0 0 0 0 0 0 0 100 0.48
•Topasout report
Report: System Detailed --- hostname: ptoolsl1 version: 1.2
new version
number to mark
Time: 10.00.00 --------------------------------------------------------------
CPU UTIL MEMORY PAGING EVENTS/QUEUES NFS the new format
Idle 78.0 bdon 0.1 %NonC 9.0 Pg-I 87 WtQ 0 CltV3 18
SMT ON idon 1.0 %Clnt 2.0 Pg-O 44 VCSW 1214
LP 4 bstl 0.5
Mode Don istl 0.0

Barcelona 2006
en0 0.6 7.5 0.5 0.3 0.3
-
hdisk0 0.0 0.0 0.0 0.0 0.0
Dedicated idle cycles donation - topas -C

IBM Global Services
•Example of topasout report for CEC recording

Report: Topas CEC Detailed --- hostname: ptoolsl1 version: 1.2 donated
Start:02/09/06 06.30.00 Stop:02/09/06 07.30.00 Int:60 Min Range: 600 Min cycles
Partition Info Memory (GB) Processors Avail Pool: 1.3
UnMonitored: - UnMonitored: 0.0 UnMonitored: 0 Ded Physical Busy: 0.4 stolen cycles
Shared : 6 Available :32.0 Available : 7 Donated Physical CPUs:0.7
Uncapped : 1 UnAllocated: - UnAllocated: 1 Stolen Pysical CPUs: 0.1
Capped : 7 Consumed : 8.7 Shared : 4 Hypervisor
Dedicated : 2 Dedicated : 3 Virt. Context Switch:332
Donating : 2 Donated : 1 Phantom Interrupts : 2
Pool Size : 2
donated
Host OS M Mem InU Lp Us Sy Wa Id PhysB Vcsw Ent %EntC PhI
--------------------------------shared------------------------------------------
processors
ptools1 A53 u 1.1 0.4 1 15 3 0 82 1.30 200 0.50 22.0 5
ptools5 A53 U 12 10 2 12 3 0 85 0.20 121 0.25 0.3 3
ptools3 A53 C 5.0 2.6 2 10 1 0 89 0.15 52 0.25 0.3 2 donating
ptools7 A53 c 2.0 0.4 1 0 1 0 99 0.05 2 0.10 0.3 2 partitions
Host OS M Mem InU Lp Us Sy Wa Id PhysB Vcsw %istl %bstl %bdon %idon
------------------------------dedicated-----------------------------------------
ptools4 A53 D 0.6 0.3 2 12 3 0 85 0.60 110 1 2 0 5
ptools6 A52 d 1.1 0.1 1 11 7 0 82 0.50 50 10 5 10 0
ptools8 A52 1.1 0.1 1 11 7 0 82 0.50 5 0 1 - -
ptools2 A52 1.12006
Barcelona 0.1 1 11 7 0 82 0.50 4 0 2 - -
Time: 07.30.00 -----------------------------------------------------------------

IBM Global Services
Thanks
Barcelona 2006

References and Sources
Cler, C., (2006). Configuring POWER5 Shared Pool Partitions Presentation

IBM System p Advanced POWER Virtualization Best Practices, Redbook Draft, Oct. 2006.
AIX 5L Virtualization Performance Management Course. 2006. IBM Unix Software Support Education.
Advanced POWER Virtualization on IBM eServer p5 Servers: Architecture and Performance Considerations.
2006.
AIX 5.3 Product Documentation. Retrieved Sept 2005, from
http://publib.boulder.ibm.com/infocenter/pseries/index.jsp?topic=/com.ibm.pseries.doc/hardware.htm
Braden, B., (2006-2007). Disk Sizing, Data Layout and Tuning Presentation.
Barker, R., (2006). VIO Server Guidelines Presentation.
Smolders, L., (2006-2007). Performance Tools Presentations.
Mathis, H.M., Mericas, A.E, McCalpin J.D., Eickemeyer R. J., and Kunkel S.R.. Characterization of
simultaneous multithreading (SMT) efficiency in POWER5. IBM Journal of Research and Development
Volume 49, Number 4/5 2005. Retrieved November 2005 from
http://www.research.ibm.com/journal/rd/494/mathis.html

AIX 5.3 Virtualization Updates & Tools: Advanced Technical Support, Americas

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

AIX 5.3 Virtualization Updates & Tools: Advanced Technical Support, Americas

Uploaded by

Copyright:

Available Formats

Advanced Technical Support, Americas

Noel Ricca nricca@us.ibm.com

IBM Confidential 8/30/2007 © 2007 IBM Corporation

Recent Changes/Things to Know

 Purpose of presentation is to provide information on new

2 © 2007 IBM Corporation

Virtual Processors and Tools

3 © 2007 IBM Corporation

Virtual Processors - Folding

4 © 2007 IBM Corporation

Virtual Processors - Tools

5 © 2007 IBM Corporation

CPU Monitoring - mpstat command

6 © 2007 IBM Corporation

# mpstat 1 So a lcpu may look “busy” but actual usage is low

cpu Logical CPU number

int Total number of interrupts

Proc0 Proc2 Proc4 Proc6

(delta PURR / delta time-base) * 100

8 © 2007 IBM Corporation

9 © 2007 IBM Corporation

Virtual I/O Server

10 © 2007 IBM Corporation

Virtual SCSI - CPU

11 © 2007 IBM Corporation

Virtual SCSI - CPU

12 © 2007 IBM Corporation

Virtual SCSI - CPU

Example: Two clients, using physical disk storage

13 © 2007 IBM Corporation

Virtual SCSI – Tuning/Tools

I/O Tuning – iostat -aD Read/write IOPS

© 2007 IBM Corporation

hdisk1 xfer: %tm_act bps tps bread bwrtn

16 © 2007 IBM Corporation

I/O Tuning – filemon LV

Key Indicator of a performance bottleneck Sequential vs. Random Workload

17 © 2007 IBM Corporation

I/O Tuning – filemon PV

VOLUME: /dev/hdisk1 description: N/A

18 © 2007 IBM Corporation

19 © 2007 IBM Corporation

20 © 2007 IBM Corporation

21 © 2007 IBM Corporation

LPAR & CEC Historical Tools

22 © 2007 IBM Corporation

Topas Local Recording

23 © 2007 IBM Corporation

24 © 2007 IBM Corporation

25 © 2007 IBM Corporation

26 © 2007 IBM Corporation

Topasout Local Reports

27 © 2007 IBM Corporation

28 © 2007 IBM Corporation

29 © 2007 IBM Corporation

Topasout Local Report – Detailed

Network KBPS I-Pack O-Pack KB-I KB-O

Disk Busy% KBPS TPS KB-R KB-W

Network KBPS I-Pack O-Pack KB-I KB-O

Disk Busy% KBPS TPS KB-R KB-W

© 2007 IBM Corporation

Topasout Local Report - Summary

Report: System Summary - hostname: ptoolsl1 version 1.0

Purpose of presentation is to provide information on new

Time filtering options

Cler, C., (2006). Configuring POWER5 Shared Pool Partitions Presentation