M09 MemoryOptimization

Memory Optimization
Module 9
© 2014 VMware Inc. All rights reserved

You Are Here
Course Introduction Storage Optimization
VMware Management Resources

CPU Optimization
Performance in a Virtualized
Environment
Memory Optimization
Network Scalability
Virtual Machine and Cluster
Network Optimization Optimization
Storage Scalability Host and Management Scalability
VMware vSphere: Optimize and Scale 9-2

Importance
While VMware vSphere® employs various mechanisms to efficiently

allocate memory, you might still encounter a situation where virtual
machines are allocated insufficient physical memory.
You should know how to monitor memory usage of both the host and
virtual machines. You should also know how to troubleshoot
common memory performance problems, such as those involving an
excessive demand for memory.

Module Lessons
Lesson 1: Memory Virtualization Concepts

Lesson 2: Monitoring Memory Activity
Lesson 3: Troubleshooting Memory Performance Problems

Lesson 1:
Memory Virtualization Concepts

Learner Objectives
By the end of this lesson, you should be able to meet the following
objectives:
 Describe how memory is used in a virtualized environment
 Explain each memory-reclamation technique
 Explain how memory overcommitment impacts performance

Virtual Memory Overview
virtual machine
guest virtual memory guest

applications
guest physical memory guest

operating system
host physical memory hypervisor

Application and Guest Operating System Memory Management
virtual machine
application
allocated list free list
guest
OS

Virtual Machine Memory Management
Virtual machine starts with no host application

physical memory allocated to it.
Host physical memory is allocated on
demand.
guest OS
 Guest operating system will not
explicitly allocate memory.
 Memory is allocated on the virtual
machine’s first access to memory (read
or write). hypervisor

Memory Reclamation
Guest physical memory is not freed in

the typical sense:
 Memory is moved to the free list. application
The hypervisor is not aware of when guest OS
the guest frees the memory: free list
 It has no access to the guest’s free list. guest OS

 The virtual machine can accrue lots of
host physical memory.
The hypervisor cannot directly reclaim
the host physical memory that is in the hypervisor
guest OS free list.

Guest Operating System Memory Terminology
memory size
total amount of memory
presented to a guest OS
allocated memory free memory

memory assigned to memory not assigned
applications and the OS
active memory idle memory

allocated memory recently allocated memory not
accessed or used by recently accessed or
applications and the OS used

Virtual Machine Memory-Reclamation Techniques
The hypervisor relies on the following memory-reclamation

techniques to free host physical memory:
 Transparent page sharing (enabled by default)
 Ballooning
 Memory compression
 Host-level (hypervisor) swapping

Transparent Page Sharing
Content-based page sharing reclaims memory with minimal

overhead.

Memory Ballooning in the Guest Operating System
The memory balloon driver (vmmemctl) collaborates with the virtual

machine to reclaim pages that are considered least valuable by the
guest operating system.

Reclaiming Memory with Ballooning
Ballooning preferentially selects free or idle virtual machine memory.

 This is because the guest operating system allocates from free
memory.
But if asked to reclaim too much, ballooning will eventually start
reclaiming active memory.
free memory idle memory active memory
guest
OS
hypervisor

Memory Compression
The VMware® ESXi™ host stores pages in a compression cache

rather than swap them out.
compression cache
A C
A B VM A B C VM D
B D
swap swap
A file B file
without compression cache with compression cache

Host Cache
Swap to host cache allows users to configure a special swap cache

on solid-state drive (SSD) storage. page table
0
1
1
0
SSD
1
0
0
1
main memory

Reclaiming Memory with Host Swapping
Host-level swapping randomly selects guest physical memory to

reclaim, which might be a virtual machine’s active memory.
free memory idle memory active memory
guest
OS
hypervisor

Reason for Reclaiming Memory
The hypervisor reclaims memory to support ESXi memory

overcommitment.
A host’s memory is overcommitted when the total amount of guest
physical memory is greater than the amount of host physical
memory.
VM1 (2GB) VM2 (2GB) VM3 (2GB)

VM
memory
hypervisor
(4GB)

Sliding Scale Mem.MinFreePct
Mem.MinFreePct is the amount of memory the VMkernel should

keep free.
 The VMkernel uses a sliding scale to determine the Mem.MinFreePct
threshold based on the amount of memory installed in the host.
Memory Installed Free State

Threshold Range
0 through 4GB (first 4GB) 6%
4 through 12GB (next 8GB) 4%
12 through 28GB (next 16GB) 2%
Remaining memory 1%

Criteria for Reclaiming Host Memory
Host physical memory is reclaimed based on four host free-memory

states, reflected by four thresholds.
If Host Free Memory Drops Use the Following

Toward the Following Reclamation Technique
Threshold
High None
Soft Ballooning
Hard Compression and swapping
Low All options are used as

necessary.

Memory Reclamation Review
The hypervisor uses memory-reclamation techniques to reclaim host

physical memory.
 Transparent page sharing (TPS) is enabled by default on hosts using small
pages.
• TPS is a low-overhead task.
 Ballooning (can cause guest operating system paging) is more efficient
than host-level swapping:
• Both ballooning and swapping can result in memory pages being written to disk.
• Because the guest operating system is aware of its memory usage, the guest
operating system can page more efficiently than the host.
 Memory compression:
• Compresses pages in memory rather than writing them to disk.
 Host-level swapping:
• This technique quickly reclaims memory.
• It has higher performance overhead than the other techniques.
Memory reclamation enables memory overcommitment.

Memory Overhead
Memory overhead has the following components:

 VMkernel: 100MB+
• Depends on the number of devices attached to the host
 Virtual machine memory overhead:
• Varies, based on size of guest memory and number of vCPUs
Examples:
1GB RAM 16GB RAM

2 vCPUs 4 vCPUs
=> ~30MB => ~151MB
overhead overhead

System Swap
System swap is a memory reclamation process that can take

advantage of unused memory resources across an entire system.
System swap is a tradeoff between the impact of reclaiming the
memory from another process and the ability to assign the memory
to a virtual machine that can use it.
 Space required is 1GB.
Available options for background storage are:
 Datastore: Allows the use of the datastore specified (except VMware®
Virtual SAN™ datastore).
 Host Cache: Allows the use of a part of the host cache.
 Preferred swap file location: Allows the use of the preferred swap file
location that ESXi has determined.

Configuring System Swap
Select host > Manage > Settings > System Swap.

Review of Learner Objectives
You should be able to meet the following objectives:

 Describe how memory is used in a virtualized environment
 Explain each memory-reclamation technique
 Explain how memory overcommitment impacts performance

Lesson 2:
Monitoring Memory Activity

Learner Objectives
objectives:
 Monitor host memory usage and virtual machine memory usage
 Monitor host swapping activity
 Monitor host ballooning activity

Monitoring Virtual Machine Memory Usage
Consumed host memory:

 Host physical memory allocated to a
guest
Active Guest Memory:
 Guest physical memory actively used
Monitoring the memory usage is useful for quickly
analyzing a virtual machine’s status.

Memory Usage Metrics Inside the Guest Operating System
Why is guest/host memory usage different than what I see inside the
guest operating system?
 Guest physical memory:
• Guest has better visibility while estimating active memory.
• ESXi active memory estimate technique can take time to converge.
 Host physical memory:
• Host memory usage does not correspond to any memory metric within the
guest.
• Host memory usage size is based on a virtual machine’s relative priority on
the physical host and memory usage by the guest.

Consumed Host Memory and Active Guest Memory
Consumed host memory > active guest memory:

 If memory is not overcommitted, this combination is good.
 The consumed host memory represents the highest amount of memory
used by the guest.
Consumed host memory <= active guest memory:
 Active guest memory does not completely reside in host physical
memory.
 This combination points to potential performance degradation.
When might active guest memory not reside

in host physical memory?

Monitoring Memory Usage Using resxtop
Possible states:
high, soft, hard,
and low
PCI hole
VMKMEM
Host Physical Memory (PMEM)
VMKMEM: Memory managed by the VMkernel

Monitoring Host Swapping in the vSphere Web Client
Host-level swapping severely affects the performance of the virtual

machine being swapped, as well as other virtual machines.

Host Swapping Activity in resxtop: Memory Screen
Memory screen (type m) for virtual machine worlds
Total memory
swapped for all
virtual machines on
host
swap reads swap writes

per second per second
swap space swap space

currently used target

Host Swapping Activity in resxtop: CPU Screen
CPU screen (type c) for virtual machine worlds
percentage of time
virtual machine has
waited for swap
activity

Host Ballooning Activity in vSphere Web Client
Ballooning has a lower performance penalty than swapping.

 High ballooning activity can affect performance (high guest operating
system paging).

Host Ballooning Activity in resxtop
memory balloon
statistics for the host
Is the balloon
driver installed?
physical memory physical

being held for memory
other VMs target to
reclaim

Introduction to Lab 13: Monitoring Memory Performance
Case 1: Case 2:
Baseline Data after
data workloads started
# ./starttest2 vApp: RAM-HOG (Limit 1,000MB)

Linux01
Linux01
Workload Workload
01 … 06
VM VM
128MB 128MB 1GB
1GB RAM RAM RAM RAM
your ESXi host

Lab 13: Monitoring Memory Performance
Use performance charts and the resxtop command to monitor

memory performance under load
1. Prepare for the Lab
2. Generate Database Activity in the Test Virtual Machine
3. Check for Overcommittment of Virtual Machine Memory
4. Configure an Advanced Performance Chart
5. Configure resxtop to Report Virtual Machine Memory Statistics.
6. Observe Baseline Memory Statistics
7. Import the RAM-HOG vApp
8. Start a Memory Test on Each RAM-HOG Workload
9. Record Memory Statistics
10. Clean Up for the Next Lab

Review of Lab 13: Monitoring Memory Performance
Review the lab questions:

 Did the values for MCTLSZ and MCTLTGT converge?
 Did the values for SWCUR and SWTGT converge?
 Which virtual machines do not have the balloon driver installed?
 Which virtual machines experienced swapping activity?
• Did the %SWPWT value exceed 5 percent for any of these virtual
machines?
 Based on the opm value that you recorded, did the performance of the
starttest2 script degrade?


 Monitor host memory usage and virtual machine memory usage
 Monitor host swapping activity
 Monitor host ballooning activity

Lesson 3:
Troubleshooting Memory Performance
Problems

Learner Objectives
objectives:
 Describe various memory performance problems
 Discuss causes of memory performance problems
 Propose solutions to correct memory performance problems

Review: Basic Troubleshooting Checklist for ESXi Hosts
1. Check for VMware® Tools™ status.
2. Check for resource pool

CPU saturation. 11. Check for using only one vCPU
in an SMP virtual machine.
16. Check for low guest CPU
3. Check for host CPU saturation.
utilization.
12. Check for high CPU ready time
4. Check for guest CPU saturation. on virtual machines running in
17. Check for past virtual machine
under-utilized hosts.
memory swapping.
5. Check for active virtual machine
memory swapping. 13. Check for slow storage device. 18. Check for high memory
6. Check for virtual machine swap demand in a resource pool.
wait. 14. Check for unexpected increase
in I/O latency on a 19. Check for high memory
7. Check for active virtual machine
shared storage device. demand in a host.
memory compression.
8. Check for an overloaded 20. Check for high guest memory

15. Check for unexpected increase demand.
storage device.
in data transfer rate
on network controllers.
9. Check for dropped receive
packets.
10. Check for dropped transmit

packets.
Definite problems Likely problems Possible problems

Active Host-Level Swapping (1)
No
Yes
No Yes
Yes

Active Host-Level Swapping (2)
No
Yes

Causes of Active Host-Level Swapping
The basic cause of host-level swapping:

 Memory overcommitment from using memory-intensive virtual
machines whose combined configured memory is greater than the
amount of host physical memory available
Causes of active host-level swapping:
 Excessive memory overcommitment
 Memory overcommitment with memory reservations
 Balloon drivers in virtual machines not running or disabled

Resolving Host-Level Swapping
Host-level swapping can be resolved in the following ways:

 Reduce the level of memory overcommitment.
 Enable the balloon driver in all virtual machines.
 Add memory to the host.
 Reduce memory reservations.
 Use resource controls to dedicate memory to critical virtual machines.

Reducing Memory Overcommitment
Ways to reduce the level of memory overcommitment:

 Add physical memory to the ESXi host.
 Reduce the number of virtual machines running on the ESXi host.
 Increase available memory resources by adding the host to a VMware
vSphere® Distributed Resource Scheduler™ cluster.

Enabling Balloon Driver in Virtual Machines
To enable the balloon driver in a virtual machine, install VMware

Tools.
If a virtual machine has critical memory needs, use resource controls
to satisfy those needs.
A best practice is to always

keep the balloon driver enabled.

Memory Hot Add
Add memory resources to a powered-on virtual machine that meets

the following conditions:
 The virtual machine has a guest operating system that supports
memory hot add functionality.
 The virtual machine is using at least hardware version 7 (VMware®
ESX®/ESXi 4.0 or later compatibility).
 VMware Tools is installed.

Memory Hot Add Procedure

Reduce a Virtual Machine’s Memory Reservation
Reevaluate a virtual machine’s memory reservation if:

 That virtual machine’s reservation cause the host to swap virtual
machines without reservations.
Reduce a virtual machine’s memory reservation if:
 The virtual machine is not using its full reservation.
If the reservation cannot be reduced, memory overcommitment must
be reduced.

Dedicating Memory to Critical Virtual Machines
Use memory reservations to prevent critical virtual machines from

swapping:
 Use as a last resort.
 This might move the swapping problem to other virtual machines.

Guest Operating System Paging
Monitor the host’s ballooning activity.

 If ballooning > 0 for the host, look at ballooning activity for the virtual
machines with performance problems.
• If ballooning > 0 for those virtual machines, check for high paging activity
within the guest operating systems.
Cause Solution
If overall memory demand is Use resource controls to direct

high, the ballooning mechanism resources to critical VMs.
might reclaim active memory Reduce memory
from the application or guest OS. overcommitment on the host.

Example: Ballooning Versus Swapping
Swap target is
more for the virtual
machines without
the balloon driver.
Virtual
MCTL: N – Balloon machine with
memory-
driver not active, balloon driver
hog virtual
tools probably not swaps less.
machines
installed

High Guest Memory Demand
Monitor the virtual machine’s memory usage:

 If average > 80% or peaks > 90%, high guest memory demand might
be causing problems within the guest operating system.
Cause Solution
Application and guest operating Configure additional memory for

system are using a high the virtual machine.
percentage of memory Tune the application to reduce
allocated to them. its memory demand.

When Swapping Occurs Before Ballooning
Many virtual machines are powered on at the same time.

 Virtual machines might access a large portion of their allocated
memory.
 At the same time, balloon drivers have not yet started.
 This causes the host to swap virtual machines.
Host-level swapping slows the boot process but is not a problem
after the guest is booted up.
A virtual machine’s memory that is currently swapped out to disk will
not cause a performance problem if the memory is never accessed.

Memory Performance Best Practices
 Allocate enough memory to hold the working set of applications you will
run in the virtual machine, thus minimizing swapping.
 Never disable the balloon driver. Always keep it enabled.
 Keep transparent page sharing enabled
 Avoid overcommitting memory to the point that it results in heavy
memory reclamation.

Lab 14: Diagnosing Memory Performance Issues
Use a troubleshooting checklist to detect possible performance

problems related to memory swapping and paging
1. Generate Database Activity in the Test Virtual Machine
2. Configure Advanced Performance Charts
3. Configure resxtop to Report Virtual Machines Memory Statistics.
4. Check for Active Host-Level Memory Swapping
5. Determine Which Virtual Machines Are Affected by Host-Level Memory
Swapping
6. Check for Past Host-Level Swapping
7. Determine Which Virtual Machines Are Affected by Past Host-Level
Swapping
9. Check for Guest Operating System Paging at the Host Level
10. Check for Guest Operating System Paging in Each Virtual Machine
11. Check for High Guest Memory Demand
12. Clean Up for the Next Lab


 Describe various memory performance problems
 Discuss causes of memory performance problems
 Propose solutions to correct memory performance problems

Key Points
 The hypervisor uses memory-reclamation techniques to reclaim host

physical memory. Transparent page sharing, ballooning, memory
compression, and host-level swapping are used to reclaim memory.
 A host’s swap rates and ballooning activity are key memory
performance metrics.
 The basic cause of memory swapping is memory overcommitment of
memory-intensive virtual machines.
Questions?

M09 MemoryOptimization

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

M09 MemoryOptimization

Uploaded by

Copyright:

Available Formats

Memory Optimization

© 2014 VMware Inc. All rights reserved

Course Introduction Storage Optimization

VMware Management Resources

Storage Scalability Host and Management Scalability

VMware vSphere: Optimize and Scale 9-2

© 2014 VMware Inc. All rights reserved

While VMware vSphere® employs various mechanisms to efficiently

VMware vSphere: Optimize and Scale 9-3

© 2014 VMware Inc. All rights reserved

Lesson 1: Memory Virtualization Concepts

VMware vSphere: Optimize and Scale 9-4

© 2014 VMware Inc. All rights reserved

VMware vSphere: Optimize and Scale 9-5

© 2014 VMware Inc. All rights reserved

VMware vSphere: Optimize and Scale 9-6

© 2014 VMware Inc. All rights reserved

guest virtual memory guest

guest physical memory guest

host physical memory hypervisor

VMware vSphere: Optimize and Scale 9-7

© 2014 VMware Inc. All rights reserved

allocated list free list

VMware vSphere: Optimize and Scale 9-8

© 2014 VMware Inc. All rights reserved

Virtual machine starts with no host application

VMware vSphere: Optimize and Scale 9-9

© 2014 VMware Inc. All rights reserved

Guest physical memory is not freed in

 It has no access to the guest’s free list. guest OS

VMware vSphere: Optimize and Scale 9-10

© 2014 VMware Inc. All rights reserved

allocated memory free memory

active memory idle memory

VMware vSphere: Optimize and Scale 9-11

© 2014 VMware Inc. All rights reserved

The hypervisor relies on the following memory-reclamation

VMware vSphere: Optimize and Scale 9-12

© 2014 VMware Inc. All rights reserved

Content-based page sharing reclaims memory with minimal

VMware vSphere: Optimize and Scale 9-13

© 2014 VMware Inc. All rights reserved

The memory balloon driver (vmmemctl) collaborates with the virtual

VMware vSphere: Optimize and Scale 9-14

© 2014 VMware Inc. All rights reserved

Ballooning preferentially selects free or idle virtual machine memory.

VMware vSphere: Optimize and Scale 9-15

© 2014 VMware Inc. All rights reserved

The VMware® ESXi™ host stores pages in a compression cache

without compression cache with compression cache

VMware vSphere: Optimize and Scale 9-16

© 2014 VMware Inc. All rights reserved

Swap to host cache allows users to configure a special swap cache

VMware vSphere: Optimize and Scale 9-17

© 2014 VMware Inc. All rights reserved

Host-level swapping randomly selects guest physical memory to

free memory idle memory active memory

VMware vSphere: Optimize and Scale 9-18

© 2014 VMware Inc. All rights reserved

The hypervisor reclaims memory to support ESXi memory

VM1 (2GB) VM2 (2GB) VM3 (2GB)

VMware vSphere: Optimize and Scale 9-19

© 2014 VMware Inc. All rights reserved