M03 PerfinaVirtEnv

Performance in a Virtualized Environment
Module 3
© 2014 VMware Inc. All rights reserved

You Are Here
Course Introduction Storage Optimization
VMware Management Resources

CPU Optimization
Performance in a Virtualized
Environment
Memory Performance
Network Scalability
Virtual Machine and Cluster
Network Optimization Optimization
Storage Scalability Host and Management Scalability
VMware vSphere: Optimize and Scale 3-2

Importance
Virtual machines and physical computers have many important

differences regarding performance. These differences include the
monitoring tools available and the overall performance
troubleshooting methodology.
The virtual machine monitor (VMM) is a key component in the
VMware vSphere® environment. The mode in which the VMM runs
can significantly affect application performance.

Module Lessons
Lesson 1: Performance Overview

Lesson 2: Virtual Machine Monitor
Lesson 3: Monitoring Tools

Lesson 1:
Performance Overview

Learner Objectives
By the end of this lesson, you should be able to meet the following
objectives:
 Review components of the VMware® ESXi™ architecture
 Describe fundamentals of performance
 Define a performance problem
 Discuss the vSphere performance troubleshooting methodology

Virtualization Architectures
Hosted architecture Bare-metal architecture

 VMware® Workstation™ (hypervisor)
 VMware Fusion®  ESXi

ESXi Architecture
guest file CPU is controlled by the

guest
TCP/IP system scheduler and virtualized
by the VMM.
monitor virtual machine monitor (VMM)
virtual NIC virtual SCSI

Memory is allocated by
memory
scheduler allocator
virtual switch file system the VMkernel and
virtualized by the VMM.
NIC drivers I/O drivers
VMkernel
Network and I/O devices

are proxied though
physical host native device drivers.

Virtualization Performance: First Dimension
First dimension:
 Single virtual machine on a single physical virtual machine
host: vCPU
• The hypervisor sits between the virtual
machine and the physical hardware.
Factor that affects performance: hypervisor
 VMM overhead
PCPU
host

Virtualization Performance: Second Dimension
Second dimension:
 Multiple virtual machines on a single physical host:
• The hypervisor sits between the virtual machines and physical hardware.
Factor that affects performance:
 Scheduling overhead and insufficient resources such as network and
storage bandwidth
virtual machine virtual machine virtual machine
hypervisor
PCPU
host

Virtualization Performance: Third Dimension
Third dimension:
 VMware vSphere® Distributed Resource Scheduler™:
• Reduces the impact of second dimension performance issues
Factor that affects performance:
 Aggressive VMware vSphere® vMotion® migration
virtual virtual virtual virtual virtual virtual

machine machine machine machine machine machine
hypervisor hypervisor
vSphere
host vMotion host

Performance Factors in a vSphere Environment
Hardware:
 CPU
 Memory
 Storage
 Network
Software:
 VMM
 Virtual machine settings
 Applications

Best Practices for Performance
Understand and account for

the target application’s profile.
Optimally configure virtual

machines.
Improve virtualization
performance by using newer
platforms when available.
Monitor consolidated
workloads closely to discover
new bottlenecks quickly.

Performance Problems
 A performance problem should be defined in the context of an ongoing

performance management and capacity planning process.
 Depending on the methodology in place, a performance problem might
be defined as:
• When an application fails to meet its predetermined service-level agreement
• When application response falls outside a predetermined range of baseline
performance
• When users report slow response time or poor throughput

Performance Troubleshooting Methodology
A performance troubleshooting methodology must provide guidance

on how to find the root cause of performance problems and how to
fix the cause when determined.
To accomplish this, you have to address the following questions:
 What are the success criteria for this problem?
 Where do you start looking for problems?
 How do you know what to look for to identify a problem?
 How do you find the root cause of a problem that you have identified?
 What do you change to fix the root cause?
 Where do you look next if no problem is found?

Basic Troubleshooting Checklist for ESXi Hosts
1. Check for VMware® Tools™ status.
2. Check for resource pool

CPU saturation. 11. Check for using only one vCPU
in an SMP virtual machine.
16. Check for low guest CPU
3. Check for host CPU saturation.
utilization.
12. Check for high CPU ready time
4. Check for guest CPU saturation. on virtual machines running in
17. Check for past virtual machine
under-utilized hosts.
memory swapping.
5. Check for active virtual machine
memory swapping. 13. Check for slow storage device. 18. Check for high memory
demand in a resource pool.
6. Check for virtual machine swap wait.
14. Check for unexpected increase
in I/O latency on a shared 19. Check for high memory
7. Check for active virtual machine
storage device. demand in a host.
memory compression.
8. Check for an overloaded 20. Check for high guest memory

15. Check for unexpected increase demand.
storage device.
in data transfer rate on
network controllers.
9. Check for dropped receive
packets.
10. Check for dropped transmit

packets.
Definite problems Likely problems Possible problems

Review of Learner Objectives
You should be able to meet the following objectives:

 Review components of the ESXi architecture
 Describe fundamentals of performance
 Define a performance problem
 Discuss the vSphere performance troubleshooting methodology

Lesson 2:
Virtual Machine Monitor

Learner Objectives
objectives:
 Describe the VMM
 Discuss the monitor modes

Virtual Machine Monitor
The VMM provides virtual x86

Guest OS Guest OS
hardware to a virtual
machine’s guest operating
system.
virtual virtual
machine machine VMM modes:
VMkernel
monitor monitor
 Software virtualization
 Hardware virtualization
virtual NIC virtual SCSI
memory
scheduler virtual switch file system
allocator
NIC drivers I/O drivers
physical hardware

Monitor Mode
The virtual CPU is comprised of the following:

 The virtual instruction set
 The virtual memory management unit (MMU)
The combination of techniques used to virtualize the instruction set
and memory is called the monitor execution mode, or monitor mode.
The VMM can implement the monitor mode with either software or
hardware techniques, or a combination of both.

Challenges of x86 Hardware Virtualization
x86 operating systems run directly

on the hardware and assume that direct
they own the hardware. execution of
user and OS
 The operating system must Ring 3
requests
execute its privileged instructions
in ring 0. Ring 2
Virtualizing the x86 architecture
requires placing a virtualization Ring 1
layer under the operating system.
Ring 0

CPU Software Virtualization: Binary Translation
Binary translation is the original

approach to virtualizing the direct
execution of
(32-bit) x86 instruction set.
user requests
With binary translation: Ring 3
 The VMM runs in ring 0 for

isolation and performance. Ring 2
 The guest operating system Ring 1 binary

runs in ring 1. translation
 Applications continue to run in Ring 0
of OS
requests
ring 3.

CPU Hardware Virtualization (First Generation)
In addition to software virtualization, the following hardware

virtualization is available:
 Intel VT-x
 AMD-V
Both designs have similar goals:
 Design the hardware to make running virtual machines easier for a
VMM.
 Allow a VMM to do away with binary translation while still being able to
fully control the execution of a virtual machine.

Intel VT-x and AMD-V
Both Intel VT-x and AMD-V direct

have a CPU execution mode execution of
user requests
feature that does the Ring 3
following:
 Allows the VMM to run in a Ring 2
root mode below ring 0
 Automatically traps privileged Ring 1
and sensitive calls to the
hypervisor Ring 0 OS
 Stores the guest operating requests
system state in virtual Root mode trap to
machine control structures privilege VMM
level without
(Intel VT-x) or virtual
binary
machine control blocks
translation.
(AMD-V)

CPU Virtualization Overhead
Time spent by the VMM does the

following:
 Increases host CPU utilization
sensitive operations
 Increases latency (for example, direct
virtual machine exit latency) additional
execution
CPU time
Throughput is unaffected when
CPU headroom is available.

Memory Management Concepts
Processes see virtual process 2

process 1
memory, not physical virtual
memory. memory
 The guest operating

system uses page tables to physical memory
map virtual memory
addresses to physical
memory addresses.
The MMU and translation TLB
look-aside buffer (TLB) virtual physical virtual to physical mapping

address address
quickly translate virtual
addresses to physical
addresses.
 If an address is not found
in the TLB, then the page
table is consulted.

MMU Virtualization
VM 1 VM 2
process 1 process 2 process 1 process 2
virtual memory
guest physical
memory
host physical memory
To run multiple virtual machines on a single system, another level of

memory virtualization must be used: host physical memory.
The VMM maps guest physical addresses (GA) to host physical
addresses (HA).
To support the guest operating system, the MMU must be virtualized
by using the following:
 Software technique: shadow page tables
 Hardware technique: Intel EPT and AMD RVI

Software MMU: Shadow Page Tables
VM 1 VM 2
virtual process 1 process 2 process 1 process 2
memory (VA)
guest physical
memory (GA)
host physical memory (HA)
Shadow page tables are created for each primary page table. They
consist of two mappings: VA to GA and GA to HA. The page tables
accelerate memory access.

Hardware MMU Virtualization
AMD RVI and TLB VA to GA mapping

VA GA
Intel EPT permit two
levels of address guest PT
mapping in the pointer
hardware:
guest
 These two levels
eliminate the need VMM
for the VMM to nested PT
synchronize shadow pointer
page tables with
guest page tables. GA to HA mapping

Memory Virtualization Overhead
Software MMU virtualization incurs CPU overhead when the following

occur:
 New processes are created:
• New address spaces need to be created.
 Context switches are made:
• Address spaces are switched.
 A large number of processes are running:
• Shadow page tables need to be maintained.
 Pages are allocated and deallocated
Hardware MMU virtualization incurs CPU overhead only when a TLB
miss occurs.

Choosing the Monitor Mode
The VMM can choose from three monitor modes:

 BT: Binary translation and shadow page tables
 HV: AMD-V or Intel VT-x and shadow page tables
 HWMMU: AMD-V with RVI, or Intel VT-x with EPT
The VMM determines a set of possible monitor modes to use, then
picks one. The decision is based on the following:
 The physical CPU’s features and guest operating system type
 Configuration file settings


 Describe the VMM
 Discuss the monitor modes

Lesson 3:
Monitoring Tools

Learner Objectives
objectives:
 Discuss the available monitoring tools
 Use VMware® vCenter Server™ performance charts
 Use the resxtop command
 Discuss guest operating system-based performance tools
 Describe how to choose the correct tool for a given situation

Commonly Used Performance Monitoring Tools
VMware® performance charts:

 Performance charts are viewed with VMware vSphere® Web Client or
VMware vSphere® Client™:
• vSphere Web Client can be used when connected to vCenter Server.
• vSphere Client can be used when connected to vCenter Server or directly to
an ESXi host.
 Two views are available:
• Overview: A set of predefined charts that display the performance for the
last 24 hours.
• Advanced: You can create and save customized charts, both real-time and
historical.
The resxtop utility
Guest operating system-based tools:
 Perfmon, Iometer, and so on
 Some performance tools and benchmarking tools might not work
accurately in a virtual machine.

Overview Performance Charts
VMware performance charts:

 Viewed with vSphere Web Client:
Partial Overview
Panel for a Host’s
Performance Charts

Advanced Performance Charts
VMware performance charts are viewed with vSphere Web Client:
Partial Advanced
Panel for a Host’s
Performance Charts

Customize Performance Charts
Previously Timespan
Chart Type Objects
Saved
Chart Options Rollups
Counters
Statistics Type

Timespan: Real-Time and Historical
vCenter Server stores statistics at different granularities.
Timespan Data Frequency Number of Samples

Real-Time (past hour) 20 seconds 180
Past Day 5 minutes 288
Past Week 30 minutes 336
Past Month 2 hours 360
Past Year 1 day 365

Chart Types
Line Graph:
 Each instance is shown separately.
Stacked Graph:
 Graphs are stacked on top of one another.
 Use stacked graphs to compare data
across virtual machines. Example shows
breakdown of CPU use:
• Per virtual machine
• Per vCPU by each virtual machine
Stacked Graph per virtual machine:
 Available to ESXi hosts only:
• Used to compare metrics between virtual
machines

Objects and Counters
Objects are instances or aggregations of devices.

 Examples:
• vCPU0, vCPU1, vmhba1:1:2, aggregate over all NICs
Counters identify which statistics to collect.
 Examples:
• CPU: used time, ready time, usage (%)
• NIC: network packets received
• Memory: memory swapped

Statistics Type
The statistics type is the unit of measurement used during the

statistics interval.
Statistics Type Description Example
Rate Value over the current interval CPU usage (MHz)

Delta Change from previous interval CPU ready time
Absolute Absolute value (independent of interval) Memory active

Rollup
Rollup is the conversion function between statistics intervals.

 5 minutes of past-hour statistics are converted to 1 past-day value.
• 15 twenty-second statistics are rolled up into a single value.
 30 minutes of past-day statistics are converted to 1 past-week value.
• 6 five-minute statistics are rolled up into a single value.
Rollup Type Conversion Function Sample Statistic
Average Average of data points CPU usage (average)

Summation Sum of data points CPU ready time (milliseconds)
Latest Last data point Uptime (days)
Other rollup types: Minimum, Maximum

Exporting Charts
Click the Export icon above the chart to export performance data as
an image or a set of values.
The chart can be exported in these formats:
 PNG
 JPEG
 CSV

resxtop Utility
Use the resxtop utility to examine real-time and historical resource

usage for ESXi hosts.
resxtop can be run in these modes:
 Interactive mode
 Batch mode
 Replay mode

Using resxtop Interactively
To start resxtop interactively:

1. Log in to a system installed with VMware vSphere® Command-Line
Interface (vCLI).
2. Run resxtop with one or more connection parameters. Examples:
• # resxtop --server vc01.vmeduc.com
--username administrator --vihost esxi01.vmeduc.com
• # resxtop --server esxi01.vmeduc.com --username mike

Navigating resxtop
When using resxtop in interactive mode, type a character to change

the screen or behavior. Commands are case-sensitive.
c CPU view (default) n Network view

m Memory view f/F Add or remove statistic
columns
d Disk (adapter) view
V Virtual machine view
u Disk (device) view (uppercase V)
v Virtual disk view (lowercase v) h Help
q Quit

Sample Output from resxtop
CPU (lowercase c):

host
statistics
per world
statistics
Type V (uppercase V):
per virtual
machine
statistics

Using resxtop in Batch and Replay Modes
To run resxtop in batch mode and print all performance counters:

 resxtop –a –b > analysis.csv
 The –a option shows all statistics.
Always start your virtual machines before running resxtop in batch
mode.
 resxtop produces virtual machine data based only on the virtual
machines that are running at the time that the command is launched.
To run resxtop in replay mode:
 Use vm-support to create a file with sampled performance data and
use resxtop to replay the file. For example:
• vm-support –p –d 300 –i 30
• Unzip and untar the resulting .tar file so that resxtop can use it
in replay mode.
• resxtop –r filename

Analyzing Batch Mode Output with Perfmon

Guest Operating System-Based Performance Tools
In general, guest operating system-based performance tools are not

aware of the virtual nature of the hardware that they are running on
and do not provide accurate metrics.
 Guest tools are safe to use that do the following:
• Measure non-time-based metrics (such as free/swapped memory times)
• Strictly generate load (such as Iometer).
 Often, application-specific counters are available only through guest-
based tools:
• For example, for Microsoft server applications such as SQL Server or
Exchange
 VMware Tools in vSphere includes a Perfmon DLL that provides
additional counters that give the guest visibility into host CPU and
memory usage.

Perfmon DLL in VMware Tools
The Perfmon DLL provides processor and

memory counters to access host statistics
from inside a virtual machine.

Choosing the Correct Tool (1)
vSphere Web Client and vSphere Client:

 Are the primary tools for observing performance and configuration data
for one or more ESXi hosts.
 Provide access to the most important configuration and performance
information.
 Do not require high levels of privilege to access the data.
resxtop:
 Gives access to detailed performance data of a single ESXi host.
 Provides fast access to a large number of performance metrics.
 Requires administrator-level access.

Choosing the Correct Tool (2)
Guest operating system-based monitoring tools can lead to

inaccurate metrics in some situations (for example, overcommitted
CPU resources).
Accuracy of in-guest tools is dependent on the guest operating
system and kernel version being used.

Lab 2: VMware Monitoring Tools
Monitor memory, CPU, and storage performance

1. Prepare for the Lab
2. Start Database Activity in a Virtual Machine
3. Examine Host and Storage Overview Charts
4. Configure Advanced Performance Charts
5. Start resxtop
6. Explore resxtop Output
7. Run resxtop in Batch Mode
8. Use Windows Performance Monitor to Display Captured Statistics
9. Clean Up for the Next Lab


 Discuss the available monitoring tools
 Use vCenter Server performance charts
 Use the resxtop command
 Discuss guest operating system-based performance tools
 Describe how to choose the correct tool for a given situation

Key Points
 The vSphere troubleshooting methodology is to follow a fixed-order

flow through the most common observable performance problems.
 For most workloads, using the default monitor mode is recommended.
 vSphere tools for monitoring performance are vCenter Server
performance charts and resxtop.
 Guest-based performance monitoring is an inaccurate means of
evaluating performance in virtual deployments.
Questions?

M03 PerfinaVirtEnv

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

M03 PerfinaVirtEnv

Uploaded by

Copyright:

Available Formats

Performance in a Virtualized Environment

© 2014 VMware Inc. All rights reserved

Course Introduction Storage Optimization

VMware Management Resources

Storage Scalability Host and Management Scalability

VMware vSphere: Optimize and Scale 3-2

© 2014 VMware Inc. All rights reserved

Virtual machines and physical computers have many important

VMware vSphere: Optimize and Scale 3-3

© 2014 VMware Inc. All rights reserved

Lesson 1: Performance Overview

VMware vSphere: Optimize and Scale 3-4

© 2014 VMware Inc. All rights reserved

VMware vSphere: Optimize and Scale 3-5

© 2014 VMware Inc. All rights reserved

VMware vSphere: Optimize and Scale 3-6

© 2014 VMware Inc. All rights reserved

Hosted architecture Bare-metal architecture

VMware vSphere: Optimize and Scale 3-7

© 2014 VMware Inc. All rights reserved

guest file CPU is controlled by the

monitor virtual machine monitor (VMM)

virtual NIC virtual SCSI

Network and I/O devices

VMware vSphere: Optimize and Scale 3-8

© 2014 VMware Inc. All rights reserved

VMware vSphere: Optimize and Scale 3-9

© 2014 VMware Inc. All rights reserved

virtual machine virtual machine virtual machine

VMware vSphere: Optimize and Scale 3-10

© 2014 VMware Inc. All rights reserved

virtual virtual virtual virtual virtual virtual

VMware vSphere: Optimize and Scale 3-11

© 2014 VMware Inc. All rights reserved

VMware vSphere: Optimize and Scale 3-12

© 2014 VMware Inc. All rights reserved

Understand and account for

Optimally configure virtual

VMware vSphere: Optimize and Scale 3-13

© 2014 VMware Inc. All rights reserved

 A performance problem should be defined in the context of an ongoing

VMware vSphere: Optimize and Scale 3-14

© 2014 VMware Inc. All rights reserved

A performance troubleshooting methodology must provide guidance

VMware vSphere: Optimize and Scale 3-15

© 2014 VMware Inc. All rights reserved

1. Check for VMware® Tools™ status.

2. Check for resource pool

8. Check for an overloaded 20. Check for high guest memory

10. Check for dropped transmit

Definite problems Likely problems Possible problems

VMware vSphere: Optimize and Scale 3-16

© 2014 VMware Inc. All rights reserved

You should be able to meet the following objectives:

VMware vSphere: Optimize and Scale 3-17

© 2014 VMware Inc. All rights reserved

VMware vSphere: Optimize and Scale 3-18

© 2014 VMware Inc. All rights reserved

VMware vSphere: Optimize and Scale 3-19

© 2014 VMware Inc. All rights reserved

The VMM provides virtual x86

NIC drivers I/O drivers

VMware vSphere: Optimize and Scale 3-20