ESX MEMORY

MANAGEMENT

December 18, 2019

DXC Proprietary and Confidential

Talk Overview

• Introduction
• ESX Memory Management Overview
• Memory Reclamation Mechanism
• Allocation Policy

DXC Proprietary and Confidential December 18, 2019 2

Introduction

• VMware ESX is a hypervisor designed to efficiently manage hardware resources

including CPU, memory, storage, and network among multiple concurrent virtual
machines. This paper describes the basic memory management concepts in ESX and
the configuration options available
• ESX uses high-level resource management policies to compute a target memory
allocation for each virtual machine (VM) based on the current system load and
parameter settings for the virtual machine (shares, reservation, and limit).
• In the cases where host memory is overcommitted, the target allocations are still
achieved by invoking several lower-level mechanisms to reclaim memory from virtual
machines.

DXC Proprietary and Confidential December 18, 2019 3

Talk Overview

• Introduction
• ESX Memory Management Overview
• Memory Reclamation Mechanism
• Allocation Policy

DXC Proprietary and Confidential December 18, 2019 4

ESX Memory Management Overview

0.1 Terminology:
The following terminology is used throughout this paper.
• Host physical memory1 refers to the memory that is visible to the hypervisor as available
on the system.
• Guest physical memory refers to the memory that is visible to the guest operating system
running in the virtual machine.
• Guest virtual memory refers to a continuous virtual address space presented by the guest
operating system to applications. It is the memory that is visible to the applications running
inside the virtual machine.

DXC Proprietary and Confidential December 18, 2019 5

0.2 Memory Virtualization Basics:
Virtual memory creates a uniform virtual address space for applications and allows the operating
system and hardware to handle the address translation between the virtual address space and the
physical address space.

Fig.Memory Levels

DXC Proprietary and Confidential December 18, 2019 6

0.3 Memory Management Basics in ESX

This topic discuss about how the application, guest operating system, hypervisor, and virtual
machine manage memory at their respective layers :

• An application starts and uses the interfaces provided by the operating system to explicitly
allocate or deallocate the virtual memory during the execution.
• The operating system establishes the definitions of “allocated” or “free” physical memory.
• The hypervisor knows when to allocate host physical memory for a virtual machine
because the first memory access from the virtual machine to a host physical memory will
cause a page fault that can be easily captured by the hypervisor. However, it is difficult for
the hypervisor to know when to free host physical memory upon virtual machine memory
deallocation because the guest operating system free list is generally not publicly
accessible.

DXC Proprietary and Confidential December 18, 2019 7

Talk Overview

• Introduction
• ESX Memory Management Overview
• Memory Reclamation Mechanism
• Allocation Policy

DXC Proprietary and Confidential December 18, 2019 8

Memory Reclamation Mechanism
• 0.1 Overview
• The hypervisor cannot reclaim host physical memory upon virtual machine memory
deallocation, it must reserve enough host physical memory to back all virtual machine’s
guest physical memory (plus their overhead memory) in order to prevent any virtual
machine from running out of host physical memory.
• The host memory is overcommitted when the total amount of guest physical memory of
the running virtual machines is larger than the amount of actual host memory
• ESX supports memory over commitment through reclamation techniques due to two
important benefits it provides: Higher memory utilization and higher consolidation ratio.

• These techniques are transparent page sharing, ballooning, memory compression and
host swapping.

DXC Proprietary and Confidential December 18, 2019 9

0.2 Transparent Page Sharing (TPS)

When multiple virtual machines are running, some of them may have identical sets of
memory content. This presents opportunities for sharing memory across virtual machines
(as well as sharing within a single virtual machine).
ESX uses hashing to identify potentially identical pages.

DXC Proprietary and Confidential December 18, 2019 10

0.3 Ballooning
• Ballooning is designed to force the VM's guest operating system to decide which memory
pages are less important and swap them to disk -- freeing more physical memory space
for the VM.
• The guest operating system is not aware that it is running inside a virtual machine and is
not aware of the states of other virtual machines on the same host. When the hypervisor
runs multiple virtual machines and the total amount of the free host memory becomes
low, none of the virtual machines will free guest physical memory because the guest
operating system cannot detect the host’s memory shortage.
• Ballooning makes the guest operating system aware of the low memory status of the
host and reclaims the memory for the host.

DXC Proprietary and Confidential December 18, 2019 11

0.4 Memory Compression
• ESXi provides a memory compression cache to improve virtual machine performance
when you use memory over commitment.
• Memory compression is enabled by default. When a host's memory becomes
overcommitted, ESXi compresses virtual pages and stores them in memory.
• Because accessing compressed memory is faster than accessing memory that is
swapped to disk, memory compression in ESXi allows you to overcommit memory without
significantly hindering performance. When a virtual page needs to be swapped, ESXi first
attempts to compress the page. Pages that can be compressed to 2 KB or smaller are
stored in the virtual machine's compression cache, increasing the capacity of the host.

N.B: Memory compression only occurs when there’s a host memory shortage and ballooning
has not achieved the desired effect. ESXi will not proactively compress memory pages when
host memory is under committed.

DXC Proprietary and Confidential December 18, 2019 12

0.5 Hypervisor Swapping

• In the cases where ballooning, transparent page sharing and memory compression are
not sufficient to reclaim memory, ESX employs hypervisor swapping to reclaim memory.
• At virtual machine startup, the hypervisor creates a separate swap file for the virtual
machine. Then, if necessary, the hypervisor can directly swap out guest physical
memory to the swap file, which frees host physical memory for other virtual machines.

DXC Proprietary and Confidential December 18, 2019 13

Limitation:
• Page selection problems: Under certain circumstances, hypervisor swapping may
severely penalize guest performance. This occurs when the hypervisor has no
knowledge about which guest physical pages should be swapped out, and the swapping
may cause unintended interactions with the native memory management policies in the
guest operating system.
• Double paging problems: Another known issue is the double paging problem. Assuming
the hypervisor swaps out a guest physical page, it is possible that the guest operating
system pages out the same physical page, if the guest is also under memory pressure.
This causes the page to be swapped in from the hypervisor swap device and
immediately to be paged out to the virtual machine’s virtual swap device. Page selection
and double-paging problems exist because the information needed to avoid them is not
available to the hypervisor.
• High swap-in latency: Swapping in pages is expensive for a VM. If the hypervisor swaps
out a guest page and the guest subsequently accesses that page, the VM will get
blocked until the page is swapped in from disk. High swap-in latency, which can be tens
of milliseconds, can severely degrade guest performance.

DXC Proprietary and Confidential December 18, 2019 14

Talk Overview

• Introduction
• ESX Memory Management Overview
• Memory Reclamation Mechanism
• Allocation Policy

DXC Proprietary and Confidential December 18, 2019 15

Allocation Policy
• 0.1 Overview:

• A key concept that is very important to understand is

how Shares, Limits and Reservations work.
• Why do we have these controls? Well to put it simply if we are over provisioning our
ESXi host on memory and CPU, we need a tool to make sure that the right machines
get the correct amount of resources.
• Example: To ensure that the critical VM always gets 3 GB Memory (Reservation) and
the Test VM never gets more than 1GB Memory (Limit).

DXC Proprietary and Confidential December 18, 2019 16

0.2 Reservations:
A reservation is a guarantee on either memory or cpu for a virtual machine.
• On memory it is a guarantee for access to physical memory for the virtual machine.
• While every virtual machine has a swap file. The swap file size is defined as (swap file =
configured memory – memory reservation).
• When the virtual machine is running, the vmkernel allocates memory to it when the VM
requests it. The vmkernel will always try to map the memory to physical memory, but if
the ESXi host is running low on memory the memory has to come from the swap file.
This has a huge performance penalty.
Example: Lets a virtual machine configured with 2 GB memory and you configure with 1 GB
reservation.
• When the virtual machine powers on a 1 GB swap file (.vswp) is created on a datastore
• The 1 GB reservation guarantees that the VM will always a least get 1 GB of physical
memory. If the ESXi host is running low the remaining 1 GB can come from the swap file
on disk!

DXC Proprietary and Confidential December 18, 2019 17

0.3 Limits:
On memory the limit defines what is the maximum amount of physical memory the virtual
machine can use.
This is a very dangerous setting! If you set the limit lower than the configured memory for a
VM, it will cause swapping and balloon activity for the virtual machine.
Example: You have a virtual machine configured with 4 GB memory and you configure a 1
GB limit
The virtual machine guest operating system sees 4GB of memory (inside windows task
manager for instance) but the ESXi is not allowed to give it more than 1 GB of physical
memory.
The virtual machine will probably request more that 1 GB for its application and when this
happens ballooning and hypervisor swapping will start.

DXC Proprietary and Confidential December 18, 2019 18

0.4 Shares:
Shares specify the relative importance of a virtual machine (or resource pool). If a virtual
machine has twice as many shares of a resource as another virtual machine, it is entitled to
consume twice as much of that resource when these two virtual machines are competing for
resources.
Shares are typically specified as High, Normal, or Low and these values specify share values
with a 4:2:1 ratio, respectively.

Default CPU and Memory share values for the VM(Fig)

Example : VM A has 1000 shares and VM B has 1000 shares and they are
both competing for the RAM Memory. In this case each virtual machine will get 1/2 or 50%
access and they will have the same performance.

DXC Proprietary and Confidential December 18, 2019 19

 Q&A

Footnotes/references, if needed, appear here in Arial 7pt.

Text grows up as lines are added. Copy and paste this box onto other slides.

DXC Proprietary and Confidential December 18, 2019 20

Thank you.

DXC Proprietary and Confidential