Unit 17:

AIX Tuning Tips for Oracle

© Copyright IBM Corporation 2013

Agenda
• Overview

• Memory

• CPU

• I/O

• Network

• Miscellaneous

© Copyright IBM Corporation 2013

What exactly is performance?
• Performance is the major factor on which the productivity of a
system depends
• Performance is dependent on a combination of:
– Throughput
– Response time

© Copyright IBM Corporation 2013

Components of system performance
• Central processing unit (CPU) resources
– Processor speed and number of processors
– Performance of software that controls CPU scheduling
• Memory resources
– Random access memory (RAM) speed, amount of memory, and
caches
– Virtual Memory Manager (VMM) performance
• I/O resources
– Disk latencies, number of disks and I/O adapters
– Device driver and kernel performance
• Network resources
– Network adapter performance and physical network itself
– Software performance of network applications
© Copyright IBM Corporation 2013
Impact of virtualization
• Virtualization affects how you manage AIX performance:
– The memory and processor capacity is determined by the Power
Hypervisor
– Memory and processors may be shared
– Physical adapters to storage or to the network may be in the virtual
I/O server

© Copyright IBM Corporation 2013

Performance analysis tools

© Copyright IBM Corporation 2013

Performance tuning tools

© Copyright IBM Corporation 2013

Agenda

• Overview

• Memory

• CPU

• I/O

• Network

• Miscellaneous

© Copyright IBM Corporation 2013

Review:Memory structures
Instance
User
Process
Shared Pool Database Large
Buffer Java Pool
Cache Pool
Library
Cache
Redo … … Streams Pool
Server Data
Directory log
Process …
Cache buffer … Sort Extent
Pool
PGA PM
ON
SM
ON
DB
WR
LG
WR
CK
PT
Oth
ers

Flashback
Buffer

Redo
Data Control
Parameter Log Archived
Files Files Files Log Files
File

Passwo
rd File Database

© Copyright IBM Corporation 2013

Virtual Memory
Real Memory

Virtual memory
Page Frame
Segment 0 Segment (256M)
Segment 1
Segment 2
Segment 3
Segment 4
...... Disk
Segment n-1
Segment n

Page
Size:4K/64K/16M/16G

© Copyright IBM Corporation 2013

VMM terminology
• Segment types:
– Persistent: File caching for JFS Segments
– Client: File caching for all other
Program text(Persistent)
file systems,such as JFS2 or
NFS
– Working: Private memory Process Data file(Persistent)
allocations

Data file (Client)

• Segment classification:
Threads
– Computational:
• Working segments Program text (Client)
• Program text (binary object
executables)
– Non-computational (file Process private-stack and data(Working)
memory):
• Persistent segments
Shared library data (Working)
• Client segments

© Copyright IBM Corporation 2013

Free list and page replacement

Non-pinned Paging
working storage space

Free File Persistent: JFS

list File cache Client: JFS2,
system
NFS, and others

Pinned-memory
(cannot be stolen)

© Copyright IBM Corporation 2013

When to steal pages based on free pages
Begin stealing when free Stop stealing when free
pages in a mempool is pages in a mempool is
less than minfree equal to maxfree
(default: 960) (default: 1088)

maxfree

minfree Number of
Number of free pages
free pages
Free list Free list
© Copyright IBM Corporation 2013
What types of pages are stolen?

lru_file_repage = 1 lru_file_repage= 0

Tries to only steal numperm > maxperm

file pages
maxperm
(from maxperm%
Tries to only steal
default=90%)
file pages
If file repage rate > (non-computational,
computational repage numperm> minperm either persistent
rate AND
Then steal
or client)
computational pages numperm< maxperm
Else steal file pages

minperm
(from minperm%
Steals the least default=3%) Steals the least
recently used recently used
pages numperm < minperm pages

© Copyright IBM Corporation 2013

Active Memory Sharing
Dedicated memory Shared memory (AMS)
LPARs with over-commitment

AIX AIX AIX

AIX AIX AIX
realmem=2 GB realmem=2 GB realmem=2 GB

Requests Receives Page Requests a

physical physical loaned or
page loan
page stolen

Power Hypervisor
(shared memory pool)

8G AMS VIOS
2G 1G 2G
Paging spaces
Physical memory

© Copyright IBM Corporation 2013

Active Memory Expansion (AME)
LPAR’s
Expanded
Memory
LPAR’s Actual
Logical Memory

Uncompressed
HMC administrator sets
Memory
expansion factor in the
True
ucomprsd
partition profile
exp_factor * true_mem =
XG
1.5X G target_exp_mem
Compress/
Compressed target_mem
True uncompre Memory
ss (expanded)
comprsd

Expansion factor = Memory

1.5 Deficit

© Copyright IBM Corporation 2013

Overall recommendations(1/2)
• If memory is overcommitted and impacting performance:
– Add logical memory (for example, use DLPAR to increase allocation)
– Reduce demand (especially wasteful demand)
– Consider sharing memory using AMS
– Consider implementing expanded memory with AME
• The primary tuning recommendation is: If it is not bro ken, do not fix it!
• The AIX6 or AIX7 vmo tuning defaults are well tuned for most systems
– Use of outdated tuning recommendations can cause problems
– If back-leveled at AIX 5L V5.3, use the AIX6 default vmo parameter values
as a starting point
• If free list is driven to zero or sustained below minfree, increasing
minfreeand maxfreemay be beneficial.
– maxfree = minfree + (maxpgahead or j2_maxPageReadAhead)
• Increasing minperm% may be help when working segments dominate, if:
– Computational allocations are due to wasteful application memory
management, perhaps even a memory leak
– And I/O performance is being impacted

© Copyright IBM Corporation 2013

Overall recommendations(2/2)
• No requirements for allocating SGA
• SGA may be pinned on AIX, but IS NOT RECOMMENDED
– we have better ways of keeping sga resident in memory
• Do not over commit real memory!!!!
– Server should be configured with enough physical memory to satisfy
memory requirements
• Paging space
– With AIX demand paging, paging space does not have to be large½
memory + 4 GB
• Monitor paging activity:
– vmstat -s
– sar -r
– nmon
• Resolve paging issues:
– Reduce filesystem cache size (MAXPERM, MAXCLIENT)
– Reduce Oracle SGA or PGA (9i or later) size
– Add physical memory

© Copyright IBM Corporation 2013

Determining SGA size
• Statspack:

SGA Memory Summary for DB: test01 Instance: test01 Snaps: 1046 -1047

SGA regions Size in Bytes

------------------------------ ----------------
Database Buffers 16,928,210,944
Fixed Size 768,448
Redo Buffers 2,371,584
Variable Size 1,241,513,984
----------------
sum 18,172,864,960

• SQLPLUS:
SQL> show sga
SQL> show parameters

© Copyright IBM Corporation 2013

Agenda

• Overview

• Memory

• CPU

• I/O

• Network

• Miscellaneous

© Copyright IBM Corporation 2013

Processes and threads

© Copyright IBM Corporation 2013

CPU considerations
Oracle parameters based on the # of CPUs
• DB_WRITER_PROCESSES
• Degree of parallelism
– user level
– table level
– query level
– PARALLEL_MAX_SERVERS or AUTOMATIC_PARALLEL_TUNING
(CPU_COUNT * PARALLEL_THREADS_PER_CPU)
• CPU_COUNT
• FAST_START_PARALLEL_ROLLBACK – should be using UNDO instead
• CBO – execution plan may be affected; check explain plan

© Copyright IBM Corporation 2013

Agenda

• Overview

• Memory

• CPU

• I/O

• Network

• Miscellaneous

© Copyright IBM Corporation 2013

Oracle server architecture
Instance
User
Process Shared Pool Database Large
Buffer Java Pool
Cache Pool
Library
Cache
… … Streams Pool
Server Data Redo log
Directory buffer
Process …
Cache …
Sort Extent Pool
PGA PM SM DB LG CK
Ot
her
ON ON WR WR PT
s
Flashback
Buffer

Redo
Data Control Log Archived
Parameter
File Files Files Files Log Files

Passwor
d File Database

24
© Copyright IBM Corporation 2013
Oracle RAC options for data storage

ACFS GPFS ASM

Database Files √ √ √

Redo Log Files √ √ √

Control Files √ √ √

Archive log Files √ √

Oracle Binaries √ √

OCR √ √ √

Voting Disk √ √ √

© Copyright IBM Corporation 2013

Data layout for optimal I/O performance (1 of 3)
• Stripe and mirror everything (SAME) approach:
• Goal is to balance I/O activity across all disks, loops, adapters, and so
on
• Avoid/Eliminate I/O hotspots
• Manual file-by-file data placement is time consuming, resource
intensive and iterative

Example:
1. Use RAID-5 or RAID-10 to create striped
LUNs (hdisks)
• Four 3+P (or 2+2) RAID arrays
• One (or more) LUNs per RAID array
• Each LUN is spread across 4 drives

© Copyright IBM Corporation 2013

Data layout for optimal I/O performance (2 of 3)

• Stripe and mirror everything (SAME) approach:

• Goal is to balance I/O activity across all disks, loops, adapters, and
so on
• Avoid/Eliminate I/O hotspots
• Manual file-by-file data placement is time consuming, resource
intensive and iterative
Example:
1. Use RAID-5 or RAID-10 to create striped
LUNs (hdisks)
• Four 3+P (or 2+2) RAID arrays
• One (or more) LUNs per RAID array
• Each LUN is spread across 4 drives
2. Stripe or spread individual objects across
multiple LUNs (hdisks) for maximum
distribution
• Each object is spread across 4 LUNs (16
drives)

© Copyright IBM Corporation 2013

Data layout for optimal I/O performance (3 of 3)

• GPFS
– Stripes by default when multiple one LUNs configured per
filesystem.
– Strip size is configurable
• ASM
– Stripes by default when multiple one LUNs configured per
ASM disk group.
– Strip size is 128k (Fine-grained) or 1M (Coarse-grained)
• single-instance filesystems
– Use AIX PP or LV striping

© Copyright IBM Corporation 2013

Data layout: LV and PP striping
• LV Striping
– Oracle recommends stripe width of a multiple of db_block_size *
db_file_multiblock_read_count Usually around 1 MB
– Valid LV Strip sizes:
• AIX 5.2: 4k, 8k, 16k, 32k, 64k, 128k, 256k, 512k, 1 MB
• AIX 5.3: AIX 5.2 Strip sizes + 2M, 4M, 16 MB, 32M, 64M, 128M

– Use AIX Logical Volume 0 offset (9i Release 2 or later) for RAW LVs
Use Scalable Volume Groups (VGs), or use “mklv –T O” with Big
VGs Requires AIX APAR IY36656 and Oracle patch (bug 2620053)
• PP Striping
– Use minimum Physical Partition (PP) size (mklv -t, -s parms) Spread
AIX Logical Volume (LV) PPs across multiple hdisks in VG (mklv –e x)

© Copyright IBM Corporation 2013

Single instance environments: Filesystems
• Filesystems
– JFS:no longer being enhanced
• Better for lots of small file creates and deletes
– JFS2: generally the preferred single-instance filesystem
• Better for large files/filesystems
• Mount options:
– Buffer Caching (default): stage data in fs buffer cache
– Direct I/O (DIO): no caching on reads
– Concurrent I/O (CIO):DIO + no write lock (JFS2 only)
– Release Behind Read (RBR): During sequential reads, memory pages
released after pages copied to internal buffers
– Release Behind Write (RBW): During sequential writes, memory
pages released after pages written to disk

© Copyright IBM Corporation 2013

Single instance environments: Cached versus non-cached (direct) I/O

• File system caching tends to benefit heavily sequential workloads with

low write content. To enab le caching for JFS/JFS2:
– Use default filesystem mount options
– Set Oracle filesystemio_options=ASYNCH (default)
• DIO tends to benefit heavily random access workloads and CIO tends to
benefit heavy update workloads. To disable JFS, JFS2 caching:
– In 9i, set filesystemio_options=ASYNCH and use dio or cio mount
option
– In 10g and 11g, set filesystemio_options=SETALL
• When using DIO/CIO, fs buffer cache isn’t used. Consider the
following db changes:
– Increase db_cache_size
– Increase db_file_mu ltiblock_read_count
• Read Metalink Note #s 272520. 1, 257338.1, 36 0287.1, 232935.1

© Copyright IBM Corporation 2013

Single instance environments: I/O tuning (ioo)
• READ-AHEAD (only applicable to JFS/JFS2 with caching enabled)
– MINPGAHEAD (JFS) or j2_minPageReadAhead (JFS2)
• Default: 2
• Starting value: MAX(2,DB_BLOCK_SIZE / 4096)
– MAXPGAHEAD(JFS) or j2_maxPageReadAhead (JFS2)
• Default: 8 (JFS), 128 (JFS2)
• Set equal to (or multiple of) size of largest Oracle I/O request
– DB_BLOCK_SIZE * DB_FILE_MULTI_BLOCK_READ_COUNT
• Number of buffer structures per filesystem:
– j2_dynamicBuffe rPreallocation
• Default: 16
• Monitor with “vmstat –v”, increa se if value of “external pager
filesystem I/Os blocked with no fsbuf “ is increasing

© Copyright IBM Corporation 2013

Single instance environments: Oracle database files

• Database Files (DBF)

– I/O size is db_block_size or db_block_size * db_file_multiblock_read_count
– Use CIO or no mount options for extremely sequential I/O
– If block size is >=4096, use a filesystem block size of 4096, else use 2048
• Redo Log/Control Files
– I/O size is always a multiple of 512 bytes
– Use CIO or DIO and set filesystem block size to 512
• Archive Log Files
– Do not use CIO or DIO
– ‘rbrw’ mount option can be advantageous
• Flashback Log Files
– Writes are sequential, sized as a multiple of db_block_size
– By default, dbca will use a single location – the flash recovery area - for flashback logs, archive
logs, and backup logs
– Flashback Log files should use CIO, DIO, or rbrw
• Oracle Binaries
– Do not use CIO or DIO

© Copyright IBM Corporation 2013

Asynchronous I/O for filesystem environments (1 of 2)

• AIX parameters
– minservers: minimum # of AIO server processes(systemwide)
• AIX 5.3 default = 1 (systemwi de), 6.1 default = 3 (per CPU)

– maxservers: maximum # of AIO server processes

• AIX 5.3 default = 10 (per CPU), 6.1 default = 30 (per CPU)
– maxreqs: maximum # of concurrent AIO requests
• AIX 5.3 default = 4096, 6.1 default = 65536
– “Enable” at system restart
– Typical 5.3 settings: minservers=100, maxservers=200, maxreqs=65536
– CIO uses fastpath AIO in AIX 6.1
– For CIO fastpath AIO in 5.3 Tl5+, set fsfastpath=1
> Not persistent across reboot
• Oracle parameters
– disk_asynch_io = TRUE
– filesystemio_options = {ASYNCH | SETALL}
– db_writer_processes (let default)

© Copyright IBM Corporation 2013

Asynchronous I/O for filesystem environments (2 of 2)

• Monitor Oracle usage:

– Watch alert log and *.trc files in BDUMP directory for
warning message:
• Warning “lio_listio returned EAGAIN”
• Monitor from AIX:
– “pstat –a | grep aios”
– Use “-A” option for NMON
– iostat –Aq (new in AIX 5.3)

© Copyright IBM Corporation 2013

GPFS tunables
• See metalink note 302806.1
• Async I/O:
– Oracle parameter filesystemio_options is ignored
– Set Oracle parameter disk_asynch_io=TRUE
– Prefetchthreads= exactly what the name says
• Usually set prefetchthreads=64 (the default)
– Worker1threads= GPFS asynch I/O
• Set worker1threads=550-prefetchthreads
– Set aio maxservers=(wo rker1threads /#cpus) + 10
• Other settings:
– GPFS block size is configurable; most will use 512KB-1MB
– Pagepool– GPFS fs buffer cache, not used for RAC but may be for
binaries. Default=64M
• mmchconfig pagepool=100M
– Ipqmaxlen=512
# no –r –o ipqmaxlen=512

© Copyright IBM Corporation 2013

ASM configurations
• AIX parameters
– Async I/O needs to be enabled, but default values may be used
• ASM instance parameters
– ASM_POWER_LIMIT=1
• Makes ASM rebalancing a low-priority operation.
– May be changed dynamically.
– It is common to set this value to 0, then increase to a higher
value during maintenance windows
– PROCESSES=25+ 15n, where n=# of instances using ASM
• DB instance parameters
– disk_asynch_io=TRUE
– filesystemio_options=ASYNCH
– Increase processes by 16
– Increase Large_Pool by 600k
– Increase Shared_Pool by [(1M per 100GB of usable space) + 2M]

© Copyright IBM Corporation 2013

Agenda

• Overview

• Memory

• CPU

• I/O

• Network

• Miscellaneous

© Copyright IBM Corporation 2013

Network parameters: All environments
• Set sb_max >= 1 MB (1048576) (generally ok by default)
• Set tcp_sendspace = 262144
• Set tcp_recvspace = 262144
• Set rfc1323=1

© Copyright IBM Corporation 2013

Additional network parameters for RAC
• Set udp_sendspace = db_block_size * db_file_multiblock_read_count
+4k
– (Not less than 65536)
• Set udp_recvspace = 10 * udp_sendspace
– Must be < sb_max
– Increase if buffer overflows occur
• Ipqmaxlen=512 for GPFS environments
• Use Jumbo Frames if suppor ted at the switch layer
• Examples:
no -a |grep udp_sendspace
no –o -p udp_sendspace=65536
netstat -s |grep "soc ket buffer overflows"

© Copyright IBM Corporation 2013

Agenda

• Overview

• Memory

• CPU

• I/O

• Network

• Miscellaneous

© Copyright IBM Corporation 2013

Miscellaneous parameters
• /etc/security/limits
– Set to “-1” for everything ex cept core for Oracle user
• Sys0 attribute maxuproc >= 4096
• Environment variables: AIXTHREAD_SCOPE=S
• Use 64-bit AIX kernel
• Time synchronization – use the “-x” flag with xntpd

© Copyright IBM Corporation 2013