Oracle OpenWorld 2019

SAN FRANCISCO

Copyright © 2019 Oracle and/or its affiliates.

Safe Harbor

The following is intended to outline our general product direction. It is intended for information purposes
only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code,
or functionality, and should not be relied upon in making purchasing decisions. The development, release,
timing, and pricing of any features or functionality described for Oracle’s products may change and
remains at the sole discretion of Oracle Corporation.

Statements in this presentation relating to Oracle’s future plans, expectations, beliefs, intentions and
prospects are “forward-looking statements” and are subject to material risks and uncertainties. A detailed
discussion of these factors and other risks that affect our business is contained in Oracle’s Securities and
Exchange Commission (SEC) filings, including our most recent reports on Form 10-K and Form 10-Q
under the heading “Risk Factors.” These filings are available on the SEC’s website or on Oracle’s website
at http://www.oracle.com/investor. All information in this presentation is current as of September 2019
and Oracle undertakes no duty to update any statement in light of new information or future events.

Copyright © 2019 Oracle and/or its affiliates.

 Maximize Performance with
Oracle Linux
Aruna Ramakrishna

Principal Member of Staff

Oracle Linux and VM Development

Copyright © 2019 Oracle and/or its affiliates.

Agenda

• Maximize performance with Oracle Linux

• Oracle Linux troubleshooting
Community contributions

Oracle developers Oracle Linux team Collaborating with

contributed 5.2% ranked 3rd for the Linux
of patches to the upstream community to
core kernel, 1.8% contributions in 2018, develop mitigations
by lwn.net (behind for CPU-related
overall
Red Hat, SUSE) security issues
Hugepages

• 4KB pages = large number of page table entries

• 256GB RAM = ~67 million pages
• Page table size = a few GB!
• Performance penalties due to TLB misses, more RAM accesses
• Hugepages
• 2MB, 1GB pages
• Smaller page tables, faster lookup; TLB hits
• Not swappable
Regular vs. transparent hugepages

• Regular hugepages
• Hugepage-aware applications
• Database SGAs
• Transparent hugepages
• Does not require application to be hugepage-aware
• Process heap and stack space backed by hugepages
Configuring hugepages

• Set parameter: ‘use_large_pages=only’

Configuring hugepages

• Hugepages cannot be swapped out under memory pressure

• HugePages_Free: too many unused hugepages?
• Hugepage pool distributed across NUMA nodes
• cat /sys/devices/system/node/node*/meminfo |
fgrep Huge
Is the system running out of memory?

• free
• Low “free” memory value is not alarming
• Most of the available memory is usually used for page cache
• vmstat
• Look for swap activity
• cat /proc/meminfo
• top
free, vmstat
free, vmstat
free, vmstat
vmstat
/proc/meminfo

• MemAvailable includes buffer and page cache

memory, which are easily reclaimable
/proc/meminfo

• MemAvailable includes buffer and page cache

memory, which are easily reclaimable
/proc/meminfo

• MemAvailable includes buffer and page cache

memory, which are easily reclaimable
/proc/meminfo

• MemAvailable includes buffer and page cache

memory, which are easily reclaimable
/proc/meminfo

• MemAvailable includes buffer and page cache

memory, which are easily reclaimable
Out of memory killer
Out of memory killer
Page allocation failure
vm.min_free_kbytes

• General rule of thumb: 512 MB or 0.5% of total memory

• Minimum free memory reserves, to avoid system hang/freeze/lock up
under memory pressure
• More reserves = lesser chance of an allocation failing
• Memory depletion leads to higher allocation latencies due to
compaction
• Might need finer tuning depending on your workload
• See MOS note 2501269.1
Other tunables

• vm.swappiness
• vm.dirty_background_ratio
• vm.dirty_ratio
Collecting diagnostic logs: sosreport

• Collects output of: system commands, logs, procfs files, etc.

into an archive
• Data is collected after the fact - one-shot
• Run: ‘sosreport’ after a problem has occurred
oswatcher

• Continuous, periodic data collection of various system

commands (mpstat, vmstat, iostat, etc.)
• Useful for diagnosing performance issues
• Monitors system behavior leading up to a problem
• See MOS note 301137.1
kdump

• Vmcores are the best troubleshooting tool!

• Dump of system memory to disk
• Requires non-trivial downtime and disk space
• Bigger the system, larger the size of the vmcore = more
downtime
• Not enabled by default
/etc/kdump.conf

• Local hard disk preferable

• core_collector: filter out unnecessary pages
Things to watch out for, while configuring kdump

• Ensure that
• There’s sufficient space to dump vmcore on local disk
• kdump is running: ‘service kdump status'
• crashkernel has enough memory reserved
• See MOS note 1507025.1
crashkernel
Panics are good to
unmask a problem

• To generate vmcore when the machine runs

into an OOM:
• sysctl vm.panic_on_oom=1
Serial console logs

• Useful for debugging after a system hang or crash

• In grub:
• console=tty0 console=ttyS0,115200n8
• Pay attention to the baud rate - don’t use 9600!
• Leads to hung tasks, softlockup
Always update your systems

• Ksplice
• Rebootless upgrades
• Security fixes can be patched with zero downtime
• Supports the Linux kernel, Xen hypervisor, glibc & openssl
• Stay up-to-date with all available ksplice patches with
‘uptrack-upgrade’
• See MOS note 1363347.1
Improving diagnosability for Oracle Linux

• First Failure Data Capture (FFDC) using ftrace ring buffers

• Logging verbose debug messages in in-memory ring
buffers
• Buffers can be saved to disk after a failure
• Buffers can be extracted from kdump kernel without
capturing the entire vmcore
Oracle Cloud Infrastructure

• Oracle-provided images refreshed on a monthly basis

• New images released with security mitigations as soon as
embargo is lifted
• Supports Bring Your Own Image too
• Yum servers, ksplice channels mirrored in OCI regions
• Oracle Linux support provided at no additional cost when
subscribed to OCI
Takeaways

Enable Verify Check serial

hugepages vm.min_free_kbytes console baud rate
MOS notes
MOS note Topic Summary

2501269.1 vm.min_free_kbytes Set it to 512MB or 0.5% of RAM,

whichever is higher

301137.1 oswatcher For periodic data collection

1507025.1 kdump Enable kdump to capture

vmcore for FFDC

1363347.1 ksplice Zero downtime upgrades!

Don’t miss

Oracle Linux: A Cloud-Ready, Optimized

Platform for Oracle Cloud Infrastructure
Wed, Sept 18, 04:45 PM - 05:30 PM
Questions?

Thank you!
Session Survey

Help us make the content

even better. Please complete
the session survey in the
Mobile App.

Copyright © 2019 Oracle and/or its affiliates.

Safe Harbor

The preceding is intended to outline our general product direction. It is intended for information purposes
only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code,
or functionality, and should not be relied upon in making purchasing decisions. The development, release,
timing, and pricing of any features or functionality described for Oracle’s products may change and
remains at the sole discretion of Oracle Corporation.

Statements in this presentation relating to Oracle’s future plans, expectations, beliefs, intentions and
prospects are “forward-looking statements” and are subject to material risks and uncertainties. A detailed
discussion of these factors and other risks that affect our business is contained in Oracle’s Securities and
Exchange Commission (SEC) filings, including our most recent reports on Form 10-K and Form 10-Q
under the heading “Risk Factors.” These filings are available on the SEC’s website or on Oracle’s website
at http://www.oracle.com/investor. All information in this presentation is current as of September
2019 and Oracle undertakes no duty to update any statement in light of new information or future events.

Copyright © 2019 Oracle and/or its affiliates.