6/3/2014 Document 361468.

1
https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=iegb54m6x_4&id=361468.1 1/5
HugePages on Oracle Linux 64-bit (Doc ID 361468.1)
Modified: 21-Aug-2013 Type: REFERENCE
In this Document
Purpose
Scope
Details
Introduction
Why Do You Need HugePages?
How to Configure
Check and Validate the Configuration
Troubleshooting
Known Problems and Limitations
Further Reading
References
APPLIES TO:
Oracle Database - Enterprise Edition - Version 9.2.0.1 and later
Linux OS - Version Enterprise Linux 4.0 to Oracle Linux 6.0 with Unbreakable Enterprise Kernel [2.6.32] [Release RHEL4
to OL6]
Linux x86-64
Oracle Linux
Red Hat Enterprise Linux (RHEL)
SUSE Linux Enterprise Server (SLES)
PURPOSE
This document aims to provide.
Basic configuration of HugePages on 64-bit Linux
Fundemental reasons to use HugePages on Linux
References to known problems
References to technical background on HugePages
SCOPE
Information in this document is useful for Linux system administrators and Oracle database administrators working with
system administrators.
This document covers information about Linux HugePages for 64-bit architectures. For more generic and uses on 32-bit
and for references please see Document 361323.1
The configuration steps provided here is primarily for Oracle Linux. Still the same concepts and configurations should
apply to other Linux distributions.
6/3/2014 Document 361468.1
https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=iegb54m6x_4&id=361468.1 2/5
DETAILS
Introduction
HugePages is a feature of the Linux kernel which allows larger pages to manage memory as the alternative to the
small 4KB pagesize. For a detailed introduction, see Document 361323.1
Why Do You Need HugePages?
HugePages is crucial for faster Oracle database performance on Linux if you have a large RAM and SGA. If your
combined database SGAs is large (like more than 8GB, can even be important for smaller), you will need HugePages
configured. Note that the size of the SGA matters. Advantages of HugePages are:

Larger Page Size and Less # of Pages: Default page size is 4K whereas the HugeTLB size is 2048K. That
means the system would need to handle 512 times less pages.
Reduced Page Table Walking: Since a HugePage covers greater contiguous virtual address range than a
regular sized page, a probability of getting a TLB hit per TLB entry with HugePages are higher than with regular
pages. This reduces the number of times page tables are walked to obtain physical address from a virtual
address.
Less Overhead for Memory Operations: On virtual memory systems (any modern OS) each memory
operation is actually two abstract memory operations. With HugePages, since there are less number of pages to
work on, the possible bottleneck on page table access is clearly avoided.
Less Memory Usage: From the Oracle Database perspective, with HugePages, the Linux kernel will use less
memory to create pagetables to maintain virtual to physical mappings for SGA address range, in comparison to
regular size pages. This makes more memory to be available for process-private computations or PGA usage.
No Swapping: We must avoid swapping to happen on Linux OS at all Document 1295478.1. HugePages are not
swappable (whereas regular pages are). Therefore there is no page replacement mechanism overhead.
HugePages are universally regarded as pinned.
No 'kswapd' Operations: kswapd will get very busy if there is a very large area to be paged (i.e. 13 million
page table entries for 50GB memory) and will use an incredible amount of CPU resource. When HugePages are
used, kswapd is not involved in managing them. See also Document 361670.1
There is a general misconception where the HugePages is a feature specific to 32-bit Linux. HugePages is a
generic feature available to all word-sizes and architectures. Just that there are some specifics with 32-bit
platforms. Please see Document 361323.1 for further references.

How to Configure
The configuration steps below will guide you to do a persistent system configuration where you would need to do a
complete reboot of the system. Please plan your operations accordingly:
Step 1: Have the memlock user limit set in /etc/security/limits.conf file. Set the value (in KB) slightly smaller than
installed RAM. e.g. If you have 64GB RAM installed, you may set:
* soft memlock 60397977
* hard memlock 60397977
There is no harm in setting this value large than your SGA requirements.
The parameters will be set by default on:
6/3/2014 Document 361468.1
https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=iegb54m6x_4&id=361468.1 3/5
Oracle Linux with oracle-validated package (See Document 437743.1) installed.
Oracle Exadata DB compute nodes
Step 2: Re-logon to the Oracle product owner account (e.g. 'oracle') and check the memlock limit
$ ulimit -l
60397977
Step 3: If you have Oracle Database 11g or later, the default database created uses the Automatic Memory
Management (AMM) feature which is incompatible with HugePages. Disable AMM before proceeding. To disable, set
the initialization parameters MEMORY_TARGET and MEMORY_MAX_TARGET to 0 (zero). Please see Document
749851.1 for further information in case you encounter the error below:
ORA-00845: MEMORY_TARGET not supported on this system
Step 4: Make sure that all your database instances are up (including ASM instances) as they would run on production.
Use the script hugepages_settings.sh in Document 401749.1 to calculate the recommended value for the
vm.nr_hugepages kernel parameter. e.g.:
$ ./hugepages_settings.sh
...
Recommended setting: vm.nr_hugepages = 1496
$
Note: You can also calculate a proper value for the parameter yourself but that is not advised if you do not have
extensive experience with HugePages and concepts.
Step 5: Edit the file /etc/sysctl.conf and set the vm.nr_hugepages parameter there:
...
vm.nr_hugepages = 1496
...
This will make the parameter to be set properly with each reboot.
Step 6: Stop all the database instances and reboot the server
(Although settings could have been done dynamically they would not be effective to the extent we require before a
reboot. The best practice is to do a persistent system configuration and perform a reboot to complete the configuration
as presented through the steps above)
What If the Database / SGA Configurations Change?
The performed configuration is basically based on the RAM installed and combined size of SGA of database
instances you are running. Based on that when:
Amount of RAM installed for the Linux OS changed
New database instance(s) introduced
SGA size / configuration changed for one or more database instances
you should revise your HugePages configuration to make it suitable to the new memory framework. If not you may
6/3/2014 Document 361468.1
https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=iegb54m6x_4&id=361468.1 4/5
Symptom Possible Cause Troubleshooting Action
System is running
out of memory or
swapping
Not enough HugePages to cover the SGA(s) and therefore
the area reserved for HugePages are wasted where SGAs
are allocated through regular pages.
Review your HugePages
configuration to make sure that
all SGA(s) are covered.
Databases fail to
start
memlock limits are not set properly Make sure the settings in
limits.conf apply to database
owner account.
One of the database
fail to start while
another is up
The SGA of the specific database could not find available
HugePages and remaining RAM is not enough.
Make sure that the RAM and
HugePages are enough to cover
all your database SGAs
Cluster Ready
Services (CRS) fail to
start
HugePages configured too large (maybe larger than
installed RAM)
Make sure the total SGA is less
than the installed RAM and re-
calculate HugePages.
HugePages_Total =
HugePages_Free
HugePages are not used at all. No database instances are
up or using AMM.
Disable AMM and make sure that
the database instances are up.
See Doc ID 1373255.1
Database started
successfully and the
performance is slow
The SGA of the specific database could not find available
HugePages and therefore the SGA is handled by regular
pages, which leads to slow performance
Make sure that the HugePages
are many enough to cover all
your database SGAs
experience one or more problems below on the system:
Poor database performance
System running out of memory or excessive swapping
Database instances cannot be started
Crucial system services failing
Check and Validate the Configuration
After the system is rebooted, make sure that your database instances (including the ASM instances) are started.
Automatic startup via OS configuration or CRS, or manual startup (whichever method you use) should have been
performed. Check the HugePages state from /proc/meminfo. e.g.:

# grep HugePages /proc/meminfo
HugePages_Total: 1496
HugePages_Free: 485
HugePages_Rsvd: 446
HugePages_Surp: 0
The values in the output will vary. To make sure that the configuration is valid, the HugePages_Free value should be
smaller than HugePages_Total and there should be some HugePages_Rsvd. HugePages_Rsvd counts free pages that
are reserved for use (requested for an SGA, but not touched/mapped yet).
The sum of Hugepages_Free and HugePages_Rsvd may be smaller than your total combined SGA as instances allocate
pages dynamically and proactively as needed.
Troubleshooting
Some of the common problems and how to troubleshoot them are listed in the following table:

6/3/2014 Document 361468.1
https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=iegb54m6x_4&id=361468.1 5/5










Known Problems and Limitations
Below are some of the known and related problems and limitations related to the HugePages feature:
Document 749851.1 HugePages and Oracle Database 11g Automatic Memory Management (AMM) on Linux
Document 829850.1 Hugepages Are Not Used by Database Buffer Cache
Document 803238.1 Oracle Not Utilizing Hugepages
Document 728063.1 Setup HugePages in an Guest Does Not Work with Oracle VM 2.1 or 2.1.1
Document 550443.1 HugePages Not Released On Oracle RDBMS Instance Shutdown with RHEL / EL 5 Update 1
(5.1)
Document 860350.1 /proc/meminfo Does Not Provide HugePages Information on Oracle Enterprise Linux (OEL5)
Document 1557478.1 ALERT: Disable Transparent HugePages on SLES11, RHEL6, OEL6 and UEK2 Kernels
Further Reading
To be able to do advanced / manual configurations with HugePages you need to understand the implementation and
theory behind the concept. You may read the following for further information:
Document 361323.1 HugePages on Linux: What It Is... and What It Is Not...
Document 401749.1 Shell Script to Calculate Values Recommended Linux HugePages / HugeTLB Configuration
Document 397568.1 Linux IA64 example of allocating 48GB SGA using hugepages
REFERENCES
NOTE:401749.1 - Shell Script to Calculate Values Recommended Linux HugePages / HugeTLB Configuration
NOTE:749851.1 - HugePages and Oracle Database 11g Automatic Memory Management (AMM) on Linux
NOTE:361670.1 - Slow Performance with High CPU Usage on 64-bit Linux with Large SGA
NOTE:361323.1 - HugePages on Linux: What It Is... and What It Is Not...
NOTE:1373255.1 - 11.2.0.1/11.2.0.2 to 11.2.0.3 Grid Infrastructure and Database Upgrade on Exadata Database
Machine
NOTE:1392497.1 - USE_LARGE_PAGES To Enable HugePages
NOTE:1557478.1 - ALERT: Disable Transparent HugePages on SLES11, RHEL6, OEL6 and UEK2 Kernels