The IBM PowerVP Investigation Guide Table of Contents

The IBM PowerVP Investigation Guide..........................................................................................1 Purpose of this Investigation Guide............................................................................................1 PowerVP Synopsis......................................................................................................................2 What gap does PowerVP fill .....................................................................................................! "efinition of general ter#s used with PowerVP$.......................................................................% When should I #onitor with PowerVP .....................................................................................& "o I still need #y 'S()ased perfor#an*e tools ........................................................................& +ow should I set the *olor *oded ,P- utili.ation thresholds ...................................................& If I see red/ do I have a perfor#an*e pro)le# ...........................................................................0 +ow should I set the *olor *oded lin1 utili.ation thresholds ....................................................0 +ow should I #onitor #y syste# with PowerVP .....................................................................0 +ow *an I try opti#i.ation ideas ..............................................................................................2 +ow *an PowerVP help with "P' ...........................................................................................2 +ow do I #ap virtual partitions to physi*al *onfigurations ......................................................2 +ow *an I tune for )etter affinity ..............................................................................................3 ,an I i#pa*t the future dire*tion of PowerVP ..........................................................................3

Purpose of this Investigation Guide

Once you have PowerVP (Power Virtualization Performance) installed and monitoring your Power Systems, this investigation guide will help you better understand the information available and optimize the performance of your systems.

Please use the PowerVP nstallation and !ser "uide to assist you with the prere#uisites for using this investigation guide. $hat document will provide you with% Power Systems hardware re#uirements for models that support running PowerVP &e#uirements for operation systems for ' (, )* i, +inu,, or V OS &e#uirements for firmware levels nstallation of PowerVP -onfiguration PowerVP for monitoring and customizing it for your environment Starting.stopping PowerVP monitoring /avigating within and between PowerVP displays 0igh1level definitions of PowerVP functions &eal1time monitoring and replay of a saved log
$his Investigation Guide will provide more detailed definitions of the monitored system resources and the performance metrics used. t will attempt to help you define criteria for your utilization thresholds with color illustrations. t will provide some best practices to help you interpret the results and optimize your system performance. 2ou 1

are encouraged to read this investigation guide li3e a paper, and not 4ust refer to parts of it as a reference.

PowerVP Synopsis
PowerVP (Power Virtualization Performance) is a new )* licensed program product than can help you understand and monitor the performance of your )* Power Systems. -lients typically understand the performance of a given logical partition with the help of a comprehensive portfolio of OS1based performance tools from ' (, )* i, and +inu,. 0owever, as the Power based systems have evolved, understanding the performance of the entire Power System as it hosts multiple logical partitions has become more comple, with the age of virtualization and cloud computing. PowerVP was created to fill this gap as it monitors and illustrates the performance of an entire system (or frame ). PowerVP will allow you to monitor overall performance and allow you to drill down into more detailed hardware and software views to help you identify and resolve performance issues and to optimize the performance of your Power system. PowerVP includes agent components as well as a monitor. $he system level agent can run in any partition (' (, )* i, +inu,, or V OS). $his agent will collect a variety of information from the operating system and PowerV* interfaces to characterize the configuration and performance of the entire system. $he "! will display this data for the system level and hardware node level views. Optional partition1level agents can run in each partition (' (, )* i, +inu,, V OS). $he "! will display this data for the partition level drill1down views. $he "! can run on any networ3 connected wor3station and will provide you with an easy1to1use e,perience. PowerVP illustrates Power Systems hardware topology in con4unction with resource utilization metrics to help you better understand the system. $hese resources include nodes, processor modules, chips, cores, Powerbus lin3s, memory controller lin3s, "( .O bus, dis3 drives, 5thernets, etc. $hese resource utilizations are portrayed using a colorized heat techni#ue. $hese colors and thresholds can be customized to suite a specific client6s performance re#uirements in a meaningful way. 7or e,ample, green may indicate normal, yellow can signal caution, and red may indicate that some resource is e,tremely busy and may re#uire action. PowerVP also allows mapping between real and virtual processor resources. 7or e,ample, clic3 on a partition and see which physical cores are associated with that partition. PowerVP is a real1time monitor that can collect and update the performance information as fre#uently as every second. $he "! allows you to view this data in real1time. $he data can also be recorded in a repository for later playbac3. 8uring playbac3, PowerVP provides 8V&1li3e functions to play, fast forward, rewind, 4ump, pause, or stop. PowerVP utilizes a drill1down approach for performance analysis. $he system1level view will illustrate overall system1level performance with processor modules and inter1node 2

lin3s. -lic3ing on a specific hardware node will drill into that node showing the utilization of each core and intra1node lin3s. ' listing of each partition is present on both views showing the entitlement, utilization, and physical hardware mapping. -lic3ing on a specific partition will provide its detailed performance statistics including application efficiency. -ollectively, these performance metrics can help you optimize performance with resource balancing, improved affinity, and application efficiency. *any of the functions in PowerVP are suitable and have value for all users (clients, field support). Some of the more detailed metrics (such as -P and bus utilizations) within PowerVP are aimed at more technical users. -onsider adding PowerVP to your performance toolbo, to help simplify the management of your Power system6s performance.

What gap does PowerVP fill?

0istorically, clients have a detailed understanding of performance within a given partition (+P'&). 5ach operating system has a host of command line tools that help describe performance for the -P!, virtualization, .O, etc. 5ach operating system has one or more monitoring tools that help you understand utilizations, resource consumption, and usage characteristics of your +P'&. $his type of performance analysis allows you understand and optimize the performance of your applications running in that partition. $he adoption of virtualization has increased greatly in the last few years. *ost systems typically have several partitions9 some systems have hundreds of partitions. *any clients are moving towards cloud computing or (Po8) processing on demand which dynamically involves many systems with many partitions. !sing traditional OS1based performance tools and monitors are still important for managing.optimizing application performance within an +P'&. 0owever, there is an increased need to understand, manage, and optimize the performance of your overall system (or frame, -5-) or group of systems (or cloud, Po8). n doing this, clients don:t want to monitor each +P'& individually and then have to piece this 4igsaw puzzle together. PowerVP fills this gap in providing an easy1to1understand overall view of performance. 7rom a single partition, you can gather information for the entire system using new.enhanced interfaces to the PowerV* hypervisor. ;ith these new interfaces, PowerVP is also able to illustrate resources and their utilization that normally don:t appear in traditional performance tools within an +P'&. 7or e,ample, 3nowing the utilization of individual processor cores and their mapping to +P'&s can help you understand, manage, and balance your system resources. 'nother good e,ample, 3nowing the utilization of internal buses (Powerbus, "( .O bus, and the memory controller bus) can help you understand affinity and optimize performance. 'lso, PowerV* lets you consume all this information real1time.

Definition of general terms used with PowerVP

$he following terms are used within PowerVP and are described here as they should be interpreted when using PowerVP. $hey arranged in a logical reading order. System ' physical system is the entire Power System, including all resources for -P!, memory, storage, etc. $his physical system may contain one or more partitions, or virtual systems. Some refer to this as a frame or a -5-. 7or the purpose of interpreting PowerVP, please do not interchange the terms system (physical) and partition (virtual system). Partition% ' logical partition (+P'&) is division of a system:s resources, such that it can run independently with its own operating system. ' physical system can have one or more +P'&s (virtual systems). $hese +P'&s can be dedicated or shared (capped or uncapped). ' hypervisor, li3e PowerVP, manages these partitions. Please refer to )* nformation -enter for PowerV* for more descriptions of these related terms (virtual system, entitlement, hardware thread, V OS, dedicated donate, processor folding, partition types, etc.). !ardware node% 5,cept for the smallest Power Systems, there is a componentization of the physical system into boo3s, drawers, or nodes. 7or e,ample, Power <<=.<>= has up to four drawers, Power <?@ has up to eight boo3s. So"#et ' soc3et is a physical connection on a Power System that connects a one processor module. $hese modules can be either an S-* (single chip module ) or a 8-* (dual chip module). Pro"essor Module ' processor module is an orderable physical entity that connects to a soc3et. $hese processor modules can be in the format of an S-* or a 8-*. ;ith PO;5&<, these modules contain processor cores, caches, and other components. 7or PO;5&<, a 8-* implies two processor chips. $hip ' processor chip is physical integrated circuit that contains processor cores and.or caches. PO;5&< chips contain up to eight cores with on1chip +A, +B, and +C caches. $his document doesn:t describe all Power System configurations9 but there are significant differences between PO;5&D.@.E.< chips as well as model footprints within those architecture families. $ore ' processor core is a single physical processing unit. ;ith PO;5&<, up to eight of these cores e,ist on a single chip. 5ach core can have up to four hardware threads dispatched to it simultaneously using S*$D. $hese hardware threads can be called logical cores. *any tal3 about a system with a total number of physical cores, for e,ample, a ED1core system. +P'&s can have an entitlement in terms of a number of cores. $P% $he term -P! is used to collectively refer to the -P! resources (core, soc3et, chip, system) for a given entity (partition, system) when discussing metrics li3e -P! utilization, -P! time, -P! cycles, etc. $he term -P! is not used

e,plicitly as a specific resource name as it is often confusing. Some refer a -P! to a soc3et, some to a processor module, and some to a processor core. %tili&ation !tilization is a base performance term that is the percentage of time that a resource is busy. t is normally in the form a percentage, typically from =F to A==F. Of course, some shared +P'&s may have a utilization of greater than A==F if it consumes more -P! resources from the shared pool than its entitlement states. $P% utili&ation% $his term is much more comple, than you might e,pect. t can refer simply to the percentage of time that the -P! resources are busy. 0owever, with the advent of S*$ levels (more than one hardware thread dispatched to a core), multi1core systems, and comple, processor pipes, -P! utilization becomes more complicated. 5ach operating system may provide and interpret -P! utilization differently. ' ( and )* i provide utilizations that consider S*$ levels and hardware thread dispatch conditions. 7rom this, -P! utilization is rendered where a linear relationship is e,pected between system throughput and -P! utilization. Of course this comes with many assumptions (sufficient other resources for that wor3load to scale, only true for the actual wor3load used to tune the utilization while other wor3loads may scale significantly differently, and on and on). +inu, operation systems currently provide -P! utilizations that are based more on occupancy (hardware thread occupying a given core). $he more that you understand on this topic, the more you realize that other metrics are also needed to best understand your system.application (such as scaling characteristics, instructions consumed, run cycles consumed, contention issues, etc.). Power'us (W) *) +) ,) -) B) $. $hese are a set of lin3s or buses within Power Systems. ;ithin PowerVP, those lin3s that are labeled ;, (, 2, or G are lin3s within a hardware node9 those lin3s that are labeled ' or ) are lin3s between hardware nodes. $hese Powerbus lin3s carry data between a given chip and other resources outside that chip (cache, memory, .O). PowerVP portrays these lin3s and their utilizations. 0aving a higher Powerbus utilization implies that there is a higher rate of data transfer. Memory $ontroller (M$. $hese are a set of lin3s that connect the memory to the soc3et. $hese *- buses carry data between the memory controller and the chip. PowerVP portrays these lin3s and their utilizations. 0aving a higher rate of data transfer implies that there is a higher rate of data transfer. I/0 (G*. 'us $hese are a set of lin3s or buses within a Power System that connect the .O subsystems to the chip. $hese lin3s carry data for storage .O and networ3 .O. PowerVP portrays these lin3s, their utilizations, as well as their inbound.outbound data rate. 0aving a higher "( bus utilization implies that there is a higher rate of data transfer. $y"les per instru"tion ($PI. -P is a standard measurement of application efficiency. t is the number of cycles consumed divided by the number of 4

(machine) instructions completed. /ormally, a lower -P is better than a higher -P . ' -P can be measured for a given core, a processor module, a hardware node, or an +P'& with PowerVP. 7rom an +P'& perspective, you can brea3 down -P! utilization, into -P components (e.g., load.store unit, floating point, global completion table). $PI sta"# analysis -P! utilization can be bro3en down into -P components. +oad.Store !nit (+S! -P ) reflects the cycles consumed for accessing data (+A, cache, +B cache, +C cache, memory). 7loating Point (7(! -P ) reflects cycles consumed on e,ecuting floating point. "lobal -ompletion $able ("-$ -P ) reflects cycles consumed waiting on the global completion table for pipelining out1of1order instruction e,ecution. PowerVP analysis typically focuses on +S! -P . 1S% $PI sta"# analysis /ormally the largest component of -P! utilization is the +S! -P for O+$P applications. n other words, accessing data consumes a ma4ority of the -P! resources. ' characterization of the time accessing data from +A cache, +B cache, +C cache, and memory9 this also notes whether the accesses are for cache.memory for a given chip or for another chip on the same processor module or hardware node or distant hardware node.

When should I monitor with PowerVP?

2ou should proactively use your performance management tooling to understand your system(s) performance. t is best to have baseline information that reflects current performance levels. f you try to optimize performance later, you:ll have a HbeforeI baseline for your HafterI improvement attempts. f you have a performance issue in the future, it is also good to have a Hbefore.normalI baseline. deally you could run PowerVP all the time. &emember that you can only monitor (whether real1time or with play1bac3) information that was recorded with PowerVP as it cannot use historic data collected from other OS1based monitors.

Do I still need my 0S2'ased performan"e tools?

PowerVP was created to compliment your current suite of performance tools in your toolbo,. PowerVP focuses on new views that typically are not available with your OS1 based performance tools. So, please use these tools together to monitor and optimize your system.application performance.

!ow should I set the "olor "oded $P% utili&ation thresholds?

$he utilization of a particular resource (core, dis3 drive, bus) simply indicates its level of being busy doing wor3. 0igh or low doesn:t necessarily imply good or bad. f there is important wor3 offered to your Power System, you would hope that it would be e,ecuted now9 and processing that wor3 will drive higher resource utilization. f there is spare resource utilization, you may wish that low1priority batch 4obs will be able to ta3e advantage of it9 and processing that wor3 will drive higher resource utilization. &

f there is spare resource utilization, you may wish to be energy conscious and have some cores put to sleep9 the result of this action will drive the utilization of the remaining resources to a higher utilization. $he point here is that high utilization isn:t necessarily a bad thing. t is important to do sizing and capacity planning such that your system resources can handle the anticipated load as well as reasonable pea3s in wor3load. n this planning, it is also important to plan some headroom (i.e., spare utilization). Part of the purpose of headroom, is to be able to do most of your wor3 with -P! utilization level low enough not to have too much #ueuing time. $he other purpose of headroom is to be able to handle wor3load pea3s9 perhaps during some of those pea3s, you can deal with having additional response time due to the #ueuing multiplier effect. )est practices for headroom levels consider many factors (number of cores, resource type, partition type, partition size, etc). Please use the )* Systems ;or3load 5stimator to best size your new system, migration, or consolidation at www.ibm.com.systems.support.tools.estimator. $herefore, you should customize these color1coded thresholds to meet the re#uirements of your business. 2ou can set the number of thresholds, the utilization levels, and the colors. 2ou might start with the default levels.colors and modify them to your customized environment. $his customization will also help set your e,pectations and actions as to what to do when those levels are e,ceeded. 7or e,ample if you see that your -P! utilization is generally red for an hour during your business day, do you% #uic3ly ma3e a change to increase the entitlement of a high priority +P'&J or consider a hardware upgrade to migrate to a newer.larger Power System in the near futureJ or consider activating some additional capacity on demandJ or do you 4ust 3now that you hit a red zone normally in the pea3 of your average dayJ

If I see red) do I have a performan"e pro'lem?

Probably not, please reread the previous section. &ed simply indicates a high utilization for a given resource.

!ow should I set the "olor "oded lin# utili&ation thresholds?

$he availability of the instrumentation for the Powerbus, *- bus, and "( bus is relatively recent. $he default utilization thresholds.colors have been set as a starting point. ;e will continue to monitor these default thresholds and improve them using various wor3loads within )* and from feedbac3 from clients using PowerVP. 0igh or imbalanced Powerbus utilization can be an indication that improvements can be made to affinity (see the affinity section).

!ow should I monitor my system with PowerVP?

t depends on the nature of your business and the health of your servers. 2ou may wish to always record data and monitor it in real time, and then use the play bac3 functions to do drill1down analysis as needed. $o drill deeper, simply use the PowerVP 0

navigation (clic3ing and hovering) to loo3 more closely at hardware nodes, bus utilizations, and partition details. &emember that you can only monitor (whether real1 time or with play1bac3) information that was recorded with PowerVP as it cannot use historic data collected from other OS1based monitors. t is also possible to record PowerVP data without running the monitor "! . Some clients have tal3ed about having the PowerVP system1level display pro4ected on their wall or on the Hbig screenI. $hey may customize PowerVP to alert them with particular colors for certain utilization levels.

!ow "an I try optimi&ation ideas?

2ou should get a good HbeforeI monitoring interval prior to any changing of application or configuration. n doing so, note that day.time for the replay, or ta3e screen shots, or note the -P! utilizations and -P levels. 7or the +P'& drill1down panels, you can mar3 the bar charts with blue mar3s indicating the current levels. $hen do the change to your application or configuration that you assume will provide an optimization. 'fter this change settles down loo3 again at the PowerVP data to confirm the performance level after the change. $ypically one would want to 3eep the wor3load level e#uivalent to ma3e appropriate comparisons. /ow you can loo3 for improvement indicators% -P! utilization reduction, -P reduction, movement of the +S! -P brea3down from right to left (remote memory to local memory, remote caches to local caches, +C to +B, etc.), reduction of the utilizations of buses (Powerbus, *- bus, "( bus).

!ow "an PowerVP help with DP0?

$he 8ynamic Platform Optimizer (8PO) can help optimize your virtualization configuration. Kust li3e ma3ing any other change on your own, you can use PowerVP to help validate the performance benefit of 8PO.

!ow do I map virtual partitions to physi"al "onfigurations?

$he new hypervisor interfaces created for PowerVP provide topology information. $he illustrations on the PowerVP monitor show the specific e,istence and topology of cores, chips, processor modules, hardware nodes, and lin3s. 5ach ma4or view also has an +P'& section to show the virtual perspective by listing the partitions. PowerVP is able to help you map the virtual partitions to the physical configuration. )y clic3ing on a dedicated partition, PowerVP highlights that partition in a uni#ue color and also lights up the -P! resources (cores) in the same color. $hen you can loo3 at this mapping. deally, from the perspective of a given partition, the cores allocated would be grouped close together in the configuration to ma,imize the locality of data access. 7or shared processor partitions, PowerVP will map a given partition to a shared processor pool. $as3s in shared partitions can be dispatched to any core in the shared processor pool. f your server configuration has recently changed by adding partitions or changing the entitlement, you may want to consider running 8PO to optimize the configuration for performance.

!ow "an I tune for 'etter affinity?

;ith )* Power Systems, having good affinity is important for good performance. Servers that use a nodal design to increase their capacity (e.g., /!*'1li3e architectures ) are especially sensitive to affinity considerations. Processor affinity suggests that your wor3 be dispatched to hardware threads to the cores.chips.nodes with the highest li3elihood of being close in pro,imity where your data is. *emory affinity suggests that memory allocated to your wor3 be close in pro,imity to the cores processing your wor3. deally, your wor3 would dispatched to the same core to optimize the chance of having a hot cache (vs. a dirty cache, or having to do remote cache accesses) or at least to the same soc3et or node to optimize the chance of having local memory accesses (vs. having to do remote.distant memory accesses). *uch of the -P! resource consumed for your application can be attributed to accessing data (that is, consuming cycles waiting for accesses from cache or memory). ;hen accessing data, the goal is to consume as few cycles as possible. 7or PO;5&<, it is preferred to access data from this list in this order of preference% +A cache, local +B cache, local +C cache, cache on another core on the same chip, cache on another chip on the same hardware node, cache on another hardware node, memory on your soc3et, memory on another soc3et on the same hardware node, memory on another hardware node. $here might be a A=== times difference in run cycles consumed from one e,treme to another. $o improve your affinity you could try a number of things. Leep in mind that this may be an advanced technical topic. 'pplication coding ad4ustments may ma,imize cache line optimization. !sing pre1fetch or not may provide trade1offs between -P , throughput, and response time. !sing virtualization, such as dedicated partitions, may force better affinity. !sing other OS1provided functions (&S5$, subsystems, ;P'&, affinity system values, etc.) may force better affinity. *any of these topics are discussed in papers at% www.ibm.com.systems.power.software.ai,.whitepapers.perfMfa#.html or www.ibm.com.systems.power.software.i.management.performance.resources.html
.

$an I impa"t the future dire"tion of PowerVP?

)* has responded to the performance management re#uirements for Power System clients and has delivered PowerVP to help meet those overall re#uirements. &emember, PowerVP is intended to be used along with the suite of all the e,isting performance tools. ;ith this initial version of PowerVP, )* welcomes feedbac3 for this new monitoring tool to help guide future new enhancements. Please contact your )* mar3eting representative to provide any feedbac3.