You are on page 1of 70

IBM Systems & Technology Group Technical Conference

14 18 April, 2008, Sevilla, Spain

Oracle/AIX Performance and Tuning Session ID : pS014


Version 2.0 21.04.2008

Fabienne Lepetit Oracle/IBM Joint Solutions Center Sbastien Chabrolles IBM Power Benchmark PSSC Montpellier
Fabienne.Lepetit@oracle.com / s.chabrolles@fr.ibm.com

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

This presentation
is a Oracle/IBM joint work : IBM PSSC Power benchmark center & IBM/Oracle Joint Solutions Center.

is based on our experience and feedback of benchmarks

is a checklist which can be used in most production environment

can be widely distributed to BPs and customers

has been built with many written informations in order to be easily read and presented dont hesitate to reuse it!

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

Learning objectives

Configure AIX to support Oracle with the best performance

Setup Oracle to take advantage of AIX features

Understand the joint effort to tune both Oracle and AIX and how they interact

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

Agenda
Oracle Server Architecture Memory
AIX VMM tuning Oracle Memory Structure Allocation

IO
AIX LVM Striping AIX mount option Oracle ASM Asynchronous IO

CPU Monitoring & Tuning


Methodology & tools Focus on Two Oracle Tools: AWR & Grid Control

Conclusion

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

Agenda
Oracle Server Architecture Memory
AIX VMM tuning Oracle Memory Structure Allocation

IO
AIX LVM Striping AIX mount option Oracle ASM Asynchronous IO

CPU Monitoring & Tuning


Methodology & tools Focus on Two Oracle Tools: AWR & Grid Control

Conclusion

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

Oracle Server Architecture


how to manage Oracle memory structures

Background processes

DBWR

LGWR

SMON

DIA0

Oracle instance
Memory structures

System Global Area


Buffer Cache Shared SQL Log Buffer

Program Global Area


Private SQL Area

how to implement the data on disk

Oracle database

Database files DB Files Redologs Control files

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

Oracle Server Architecture (contd)


The components of an Oracle server are : database = set of datafiles (data + redologs + control files) instance = memory structures allocated at startup + background processes (automatically forked at startup) associated to a database Database files datafiles contains data (tables, indexes) redologs contains redo entries control files record the physical structure of the database Oracle memory structures = System Global Area (SGA) + Process Global Area (PGA) SGA shared memory region that contains data and control information for one Oracle instance allocated at instance startup and deallocated at instance shutdown each instance has its own SGA PGA memory buffer that contains data and control information for a server process PGA is created by Oracle when a server process is started contains private SQL area
7 4/22/2008 2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

Oracle Server Architecture (contd) Focus on the SGA


The three main components of the SGA are : buffer cache: - store the most recently used blocks of data - improves performance by reducing disk I/O activity shared pool contains : - shared SQL area (parse tree and execution plan for SQL statements) - dictionnary cache - library cache redolog buffer stores redo entries (log of changes made to the database) written to the online redo log (used during database recovery)
System Global Area
Buffer Cache SharedSQL Log Buffer

Program Global Area


Private SQL Area

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

Agenda
Oracle Server Architecture Memory
AIX VMM tuning Oracle Memory Structure Allocation

IO
AIX LVM Striping AIX mount option Oracle ASM Asynchronous IO

CPU Monitoring & Tuning


Methodology & tools Focus on Two Oracle Tools: AWR & Grid Control

Conclusion

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

Virtual Memory Management


1 - AIX is started, applications load some computational pages into the memory. As a UNIX system, AIX will try to take advantage of the free memory by using it as a cache file to reduce the IO on the physical drives. 2 - The activity is increasing, the DB needs more memory but there is no free pages available. LRUD (AIX page stealer) is starting to free some pages into the memory. 3 - With the default setting, LRUD will page out some computational pages instead of removing some pages from the File System Cache.

Objective :
Tune the VMM to protect computational pages (Programs, SGA, PGA) from being paged out and force the LRUD to steal pages from FS-CACHE only.

10

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

Memory : VMM Tuning for Filesystems (lru_file_repage)


This VMM tuning tip is applicable for AIX 5.2 ML4+ or AIX 5.3 ML1+

lru_file_repage :
This new parametter prevents the computational pages to be paged out. By setting lru_file_repage=0 (1 is the default value) youre telling the VMM that your preference is to steal pages from FS Cache only.

New vmo tuning approach : Recommendations based on VMM developer experience.

Use vmo p o to set the following parametters : lru_file_repage = 0 minfree = 120 x # lcpus

Increase if page stealing activity, vmstat po column

maxfree = minfree +(maxpaghead x # lcpus)

minperm% = from 3 to 10 maxperm% = maxclient% = from 70 to 90 strict_maxclient = 1


11 4/22/2008 2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

Memory : VMM Tuning for Filesystems (legacy)


Recommended VMM tuning for AIX version prior to 5.2 ML4 or AIX 5.3 ML1.

The following VMM tuning as the same objective : protect the computational pages to be page out.

vmo legacy tuning approach :

Use vmo p o to set the following parametters : minfree = 120 x # lcpus

Increase if page stealing activity, vmstat po column

maxfree = minfree +(maxpaghead x # lcpus)

minperm% = from 3 to 10 maxperm% = maxclient% = from 5 to 15 but < minperm% strict_maxclient = 0

12

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

Memory : AIX Memory Management for Oracle


System-wide contention scope (user thread/kernel thread one-to-one mapping) reduces memory usage of Oracle process & makes AIX post/wait mechanism most effective In Oracle9i, set AIX environment variable AIXTHREAD_SCOPE=S Since Oracle10g, AIXTHREAD_SCOPE=S is Internally set Pinning shared memory prevents paging out SGA (only if App + DB on same AIX and >80% computationnal pages) On AIX part vmo p o v_pinshm = 1 Leave maxpin% at the default of 80% unless the SGA exceeds 77% of real memory : vmo p o maxpin%=[(total mem-SGA size)*100/total mem] + 3 On Oracle part (9i/10g) LOCK_SGA = TRUE Large Page Support improves performance of prefetching On AIX part vmo r o lgpg_size = 16777216 o lgpg_regions=(SGA size / 16 MB) chuser capabilities=CAP_BYPASS_RAC_VMM,CAP_PROPAGATE oracle (allow Oracle user ID to use Large Pages) ldedit b lpdata oracle (to allow oracle binary to use large page data) export LDR_CNTRL=LARGE_PAGE_TEXT=Y@LARGE_PAGE_DATA=M (prior Oracle user ID to starting Oracle instance and listener to allow both large page text and large page data)

13

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

Memory : Oracle Memory Structures Allocation


8i : Manual management

All Oracle memory structures were manually managed and were mostly static (db_block_buffers)

9i : Dynamic memory resizing


db_cache_size (dynamic parameter)
sga_max_size

(static parameter) maximum size of the SGA for the lifetime of the instance.

pga_aggregate_target

(dynamic parameter) specifies the target aggregate PGA memory available to all server processes attached to the instance

parameter : db_cache_advice (dynamic parameter) enables or disables statistics gathering used for predicting behavior with different cache sizes.
additional

10g : Automatic Shared Memory Management (ASMM)


sga_target

(dynamic) if set the db_cache_size is automatically sized (shared_pool_size, large_pool_size, java_pool_size as well). Can be increased up to sga_max_size. To use Automatic Shared Memory Management, sga_target must be <>0.

14

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

Memory : Oracle Memory Structures Allocation

NEW! 11g

11g : Automatic Memory Management (AMM)


memory_target

(dynamic parameter) specifies the total memory size to be used by the instance SGA and PGA. Exchanges between SGA and PGA are done according to the needs. If sga_target and pga_aggregate_target are not set the policy is to give 60% of memory_target to the SGA and 40% to the PGA.

memory_max_target

(static parameter) specifies the maximum memory size for the instance.

To use Automatic Memory Management, memory_target must be <>0. See Metalink notes 443746.1 and 452512.1 explaining AMM and these new parameters.

15

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

Memory : Use jointly AIX dynamic LPAR and Oracle dynamic allocation of memory + AMM
Initial configuration
Memory_max_size = 18 GB

Final configuration Scenario :


Oracle tuning advisor indicates that SGA+PGA need to be increased to 11GB: memory_target can be increased dynamically to 11GB but real memory is only 12GB, so it needs to be increased as well. Memory_max_size = 18 GB Real memory = 15 GB

Real memory = 12 GB

AIX + free
Memory_target = 11 GB

AIX + free
Memory_target = 8 GB

Step 1 - Increase physical memory allocated to the LPAR to 15 GB

ssh hscroot@hmc "chhwres -r mem -m <system> -o a -p <LPAR name> -q 3072"

SGA + PGA

Step 2 - Increase SGA+PGA allocated to the instance of the database to 11GB

SGA + PGA

alter system set memory_target=11GB;

Memory allocated to the system has been increased dynamically, using AIX DLPAR Memory allocated to Oracle (SGA and PGA) has been increased on the fly
16 4/22/2008 2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

Agenda
Oracle Server Architecture Memory
AIX VMM tuning Oracle Memory Structure Allocation

IO
AIX LVM Striping AIX mount option Oracle ASM Asynchronous IO

CPU Monitoring & Tuning


Methodology & tools Focus on Two Oracle Tools: AWR & Grid Control

Conclusion

17

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

IO : Database Layout

Having a good Storage configuration is a key point : Because disk is the slowest part of an infrastructure Reconfiguration can be difficult and time consuming

Stripe and mirror everything (S.A.M.E) approach: Goal is to balance I/O activity across all disks, loops, adapters, etc... Avoid/Eliminate I/O hotspots

rec O om rac me le nd a ti o

Manual file-by-file data placement is time consuming, resource intensive and iterative Additional advices to implement SAME :
apply the SAME strategy to data, indexes if possible separate redologs (+archivelogs)

18

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

IO : RAID Policy with ESS, DS6/8K


Storage
RAID-5 vs. RAID-10 Performance Comparison I/O Profile Sequential Read Sequential Write Random Read Random Write RAID-5 Excellent Excellent Excellent Fair RAID-10 Excellent Good Excellent Excellent

HW Striping

LUN 1 LUN 2 LUN 3 LUN 4

With Enterprise class storage (with huge cache), RAID-5 performances are comparable to
RAID-10 one (for most customer workloads) Consider RAID-10 for workloads with a high percentage of random write activity (> 25%) and high I/O access densities (peak > 50%)

Use RAID-5 or RAID-10 to create striped LUNs If possible try to minimize the number of LUNs per RAID array to avoid contention on physical disk.

19

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

IO : 2nd Striping (LVM)


AIX
Volume Group

Storage

HW Striping

LV (raw or with jfs2)

hdisk1 hdisk2 hdisk3 hdisk4

LUN 1 LUN 2 LUN 3 LUN 4

1. 2. 3. 4.

Luns are striped across physical disks (stripe-size : ~ 64k, 128k, 256k) LUNs are seen as hdisk device on AIX server. Create AIX Volume Group(s) (VG) with LUNs from multiple arrays Logical Volume striped across hdisks (stripe-size : 8M, 16M, 32M, 64M) => each read/write access to the LV are well balanced accross LUNs and use the maximum number of physical disks for best performance.

20

LVM Striping

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

IO : LVM Spreading / Striping


2 different ways to stripe on the LVM layer :

1. Stripe using Logical Volume (LV)

Create Logical Volume with the striping option : mklv S <stripe-size> ...

Valid LV Stripe sizes: AIX 5.2: 4k, 8k, 16k, 32k, 64k, 128k, 256k, 512k, 1M 1. AIX 5.3+: AIX 5.2 Stripe sizes + 2M, 4M, 16M, 32M, 64M, 128M

2. PP striping (AKA spreading)

Create a Volume Group with a 8M,16M or 32M PPsize. (PPsize will be the Stripe size) AIX 5.2 : Choose a Big VG and change the t factor : # mkvg B t <factor> s <PPsize>... AIX 5.3 +: Choose a Scalable Volume Group : # mkvg S s <PPsize> ... Create LV with Maximum range of physical volume option to spread PP on different hdisk in a Round Robin fashion : # mklv e x ...

jfs2 filesystem creation advice : If you create a jfs2 filesystem on a striped (or PP spreaded) LV, use the INLINE logging option. It will avoid hot spot by creating the log inside the filesystem (which is striped) instead of using a uniq PP stored on 1 hdisk. # crfs a logname=INLINE
21 4/22/2008 2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

IO : Disk Subsystem Advices


Check if the definition of your disk subsystem is present in the ODM. If the description shown in the output of lsdev Cc disk the word Other, then it means that AIX doesnt have a correct definition of your disk device in the ODM and use a generic device definition.
# lsdev Cc disk

Generic device definition bad performance In general, a generic device definition provides far from optimal performance since it doesnt properly customize the hdisk device : exemple : hdisk are created with a queue_depth=1 1. Contact your vendor or go to their web site to download the correct ODM definition for your storage subsystem. It will setup properly the hdisk accordingly to your hardware for optimal performance. If AIX is connected to the storage subsystem with several Fiber Channel Cards for performance, dont forget to install a multipath device driver or path control module.
- sdd or sddpcm for IBM DS6000/DS8000 - powerpath for EMC disk subsystem - hdlm for Hitachi etc....
22 4/22/2008 2008 IBM Corporation

2.

IBM Systems & Technology Group Technical Conference 2008

Agenda
Oracle Server Architecture Memory
AIX VMM tuning Oracle Memory Structure Allocation

IO
AIX LVM Striping AIX mount option Oracle ASM Asynchronous IO

CPU Monitoring & Tuning


Methodology & tools Focus on Two Oracle Tools: AWR & Grid Control

Conclusion

23

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

IO : Filesystems Mount Options (DIO, CIO)


If Oracle data are stored in a Filesystem, some mount option can improve performance :

Direct IO (DIO) introduced in AIX 4.3. Data is transfered directly from the disk to the
application buffer, bypassing the file buffer cache hence avoiding double caching (filesystem cache + Oracle SGA). Emulates a raw-device implementation. To mount a filesystem in DIO $ mount o dio /data

Concurrent IO (CIO) introduced with jfs2 in AIX 5.2 ML1


Implicit use of DIO. No Inode locking : Multiple threads can perform reads and writes on the same file at the same time. Performance achieved using CIO is comparable to raw-devices. To mount a filesystem in CIO: $ mount o cio /data
24

Bench throughput over run duration higher tps indicates better performance.

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

IO : Benefits of CIO for Oracle


Benefits :
1. 2. 3. Avoid double caching : Some data are already cache in the Application layer (SGA) Give a faster access to the backend disk and reduce the CPU utilization Disable the inode-lock to allow several threads to read and write the same file (CIO only)

Restrictions :
1. Because data transfer is bypassing AIX buffer cache, jfs2 prefetching and write-behind cant be used. These functionnalities can be handled by Oracle. (Oracle parameter) db_file_multiblock_read_count = 8, 16, 32, ... , 128 according to workload 2. When using DIO/CIO, IO requests made by Oracle must by aligned with the jfs2 blocksize to avoid a demoted IO (Return to normal IO after a Direct IO Failure) => When you create a JFS2, use the mkfs o agblksize=XXX Option to adapt the FS blocksize with the application needs. Rule : IO request = n x agblksize Exemples: if DB blocksize > 4k ; then jfs2 agblksize=4096 Redolog are always written in 512B block; So jfs2 agblksize must be 512

25

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

IO : Direct IO demoted
Extract from Oracle AWR (test made in Montpellier with Oracle 10g) Waits on redolog (with demoted IO, FS blk=4k) Waits% Time -outs Total Wait Time (s) 2,229,324 0.00 62,628 Avg wait (ms) 28 Waits /txn 1.53

log file sync

Waits on redolog (without demoted IO, FS blk=512) Waits% 494,905 Time -outs Total Wait Time (s) 0.00 1,073 Avg wait (ms) 2 Waits /txn 1.00

log file sync

How to detect demoted IO : Trace command to check demoted io : # trace aj 59B,59C ; sleep 2 ; trcstop ; trcrpt o directio.trcrpt # grep demoted directio.trcrpt

26

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

CIO/DIO implementation Advices


with Standard mount options mount -o rw Oracle bin and shared lib. Cached by AIX mount -o rw Cached by Oracle Cached by AIX mount -o rw Cached by Oracle Cached by AIX mount -o rw Cached by AIX mount -o rw Oracle Control files Cached by AIX with optimized mount options mount -o rw Cached by AIX mount -o cio *(1) Cached by Oracle mount -o cio (jfs2 + agblksize=512) Cached by Oracle mount -o rbrw Use JFS2 write-behind but are not kept in AIX Cache. mount -o rw Cached by AIX

Oracle Datafiles

Oracle Redolog

Oracle Archivelog

*(1) : to avoid demoted IO : jfs2 agblksize = Oracle DB block size / n

27

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

Agenda
Oracle Server Architecture Memory
AIX VMM tuning Oracle Memory Structure Allocation

IO
AIX LVM Striping AIX mount option Oracle ASM Asynchronous IO

CPU Monitoring & Tuning


Methodology & tools Focus on Two Oracle Tools: AWR & Grid Control

Conclusion

28

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

IO : Oracle Automatic Storage Management (ASM)


ASM is a database file system which provide cluster file system and volume manager capabilities. Alternative to conventional file-system and LVM functions. For Oracle datafiles. Integrated to Oracle DB with no additional cost, for single or RAC databases With ASM, the management of Oracle datafiles is the same for the DBA on all platforms (Unix, Linux, Windows) Datafiles are striped across all ASM disks - I/O is spread evenly to prevent hot spots and maximize performance Online add/drop of disk devices with automatic online redistribution of data

Oracle

AIX

Storage

hdisk1

LUN 1 LUN 2 LUN 3 LUN 4

ASM DISKGROUP

hdisk2 hdisk3 hdisk4

ASM striping
29 4/22/2008 2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

IO : Oracle Automatic Storage Management (ASM)

Database on ASM

ASM Instance

ASM DISKGROUP create tablespace DATA datafile +DG/data01.dbf size 15G;

ASM is implemented as a special kind of Oracle instance, with its own SGA and background processes (by default, less than 100M of SGA are allocated for the ASM instance) and generate very little activity except during rebalance operations. A single ASM instance can service one or more instance databases. Can maintain redundant copies of data An ASM-managed database will have approximately the same performance as a database implemented in raw-devices.
30 4/22/2008 2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

Agenda
Oracle Server Architecture Memory
AIX VMM tuning Oracle Memory Structure Allocation

IO
AIX LVM Striping AIX mount option Oracle ASM Asynchronous IO

CPU Monitoring & Tuning


Methodology & tools Focus on Two Oracle Tools: AWR & Grid Control

Conclusion

31

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

IO : Asynchronous IO (AIO)
Allows multiple requests to be sent without to have to wait until the disk subsystem has completed the physical IO. Utilization of asynchronous IO is strongly advised whatever the type of file-system and mount option implemented (JFS, JFS2, CIO, DIO).

Application
2

aio Q

aioservers

Disk

Posix vs Legacy Since AIX5L V5.3, two types of AIO are now available : Legacy and Posix. For the moment, the Oracle code is using the Legacy AIO servers.
32 4/22/2008 2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

IO : Asynchronous IO (AIO) tuning


important tuning parameters :
check AIO configuration with : AIX 5.X : lsattr El aio0 AIX 6.1 : ioo L | grep aio

maxreqs : size of the Asynchronous queue.


minserver : number of kernel proc. Aioservers to start (AIX 5L system wide). maxserver : maximum number of aioserver that can be running per logical CPU Rule of thumb : maxservers should be = (10 * <# of disk accessed concurrently>) / # cpu maxreqs (= a multiple of 4096) should be > 4 * #disks * queue_depth but only tests allow to set correctly minservers and maxservers Monitoring : In Oracles alert.log file, if maxservers set to low : Warning: lio_listio returned EAGAIN Performance degradation may be seen #aio servers used can be monitored via ps k | grep aio | wc l , iostat A or nmon (option A)
33 4/22/2008 2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

IO : Asynchronous IO (AIO) fastpath


With fast_path, IO are queued directly from the application into the LVM layer without any aioservers kproc operation. Better performance compare to non-fast_path No need to tune the min and max aioservers No ioservers proc. => ps k | grep aio | wc l is not relevent, use iostat A instead

Application
2
Raw Devices / ASM :

AIX Kernel

Disk

check AIO configuration with : lsattr El aio0 enable asynchronous IO fast_path. : AIX 5L : chdev -a fastpath=enable -l aio0 (default since AIX 5.3) AIX 6.1 : ioo p o aio_fastpath=1 (default setting)

FS with CIO/DIO and AIX 5.3 TL5+ :


Activate fsfast_path (comparable to fast_path but for FS + CIO/DIO) AIX 5L : adding the following line in /etc/inittab: aioo:2:once:aioo o fsfast_path=1 AIX 6.1 : ioo p o aio_fsfastpath=1 (default setting)

34

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

IO : AIO,DIO/CIO & Oracle Parameters


How to set filesystemio_options parameter Possible values ASYNCH : enables asynchronous I/O on file system files (default) DIRECTIO : enables direct I/O on file system files (disables AIO) SETALL : enables both asynchronous and direct I/O on file system files NONE : disables both asynchronous and direct I/O on file system files Since the version 10g, Oracle will open data files located on the JFS2 file system with the O_CIO option if the filesystemio_options initialization parameter is set to either directIO or setall.

Advice : set this parameter to setall Note : set the disk_asynch_io parameter to true as well

35

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

Agenda
Oracle Server Architecture Memory
AIX VMM tuning Oracle Memory Structure Allocation

IO
AIX LVM Striping AIX mount option Oracle ASM Asynchronous IO

CPU Monitoring & Tuning


Methodology & tools Focus on Two Oracle Tools: AWR & Grid Control

Conclusion

36

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

CPU : Simultaneous Multi-Threading (SMT)


Feature available on POWER5 and POWER6 system running AIX
5.3 or 6.1 allows 2 instruction path to share access to the POWER execution units on every clock cycle instruction path = virtual processor = logical POWER processor (i.e. a server with 4 physical processors will have 8 logical processors with SMT enabled) SMT allows to use resources that would otherwise usually be unexploited and can achieve significant throughput gains up to 40% greater performance No source code modification needed to use SMT How to control SMT: smtcl [m on | off] -m on enables the SMT mode, -m off disables it -w boot makes SMT change effective on reboot -w now makes SMT change effective immediately but not persistent across reboots
37 4/22/2008 2008 IBM Corporation

thread1

thread2

Register thread1

Register thread2

P6 Core

IBM Systems & Technology Group Technical Conference 2008

CPU: Micro-Partitioning
Micro-Partitioning give the possibility to share processors between several LPARs. It improve CPU utilisation by reducing the number of idle cycle. (available on POWER5 and POWER6 system running AIX 5.3+)

DLPAR Configuration Total Throughput (bytes):2,037,677,193 Action_Transaction: 59.07 secs

SPLPAR Configuration Total Throughput (bytes):2,715,373,065 Action_Transaction: 53.256 secs

An example given based on a PeopleSoft/Oracle10g RAC benchmark (2,400 Virtual Users) ran at PSSC MOP (DLPAR performance vs. SPLPAR one)
38 4/22/2008 2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

CPU: Compatibility Matrix DLPAR SMT

AIX 5.2 AIX 5.3 AIX 6.1

Oracle 9i Oracle 10g Oracle 11g


39 4/22/2008 2008 IBM Corporation

Micro- Partition

IBM Systems & Technology Group Technical Conference 2008

Agenda
Oracle Server Architecture Memory
AIX VMM tuning Oracle Memory Structure Allocation

IO
AIX LVM Striping AIX mount option Oracle ASM Asynchronous IO

CPU Monitoring & Tuning


Methodology & tools Focus on Two Oracle Tools: AWR & Grid Control

Conclusion

40

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

Monitoring & Tuning : Methodology


DBA System administrator

OS monitoring

Oracle monitoring log of changes

design/parameters changes impact ?

tuning is an iterative process a minimun of changes should be performed at each iteration tuning is a joint effort between the system administrator and the DBA dont hesitate to keep a log to track changes and the impact of these changes
41 4/22/2008 2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

Monitoring & Tuning (contd) : System Monitoring Tools

vmstat useful for obtaining an overall picture of CPU, paging, and memory usage lvmstat useful to get logical volume I/O statistics iostat allows to get statistics on disk activity (and also on terminals, CPU, adapters) lparstat reports logical partition related information and statistics nmon complete tool which gives information on all the components of the system filemon uses the trace facility to report on the I/O activity of physical volumes, logical volumes, individual files, and the Virtual Memory Manager xmperf allows to define monitoring environments to supervise the performance of the local and remote systems netstat allows to monitor network activity ....

42

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

Monitoring & Tuning (contd) : Oracle Monitoring Tools


AWR is a standard tool which allow the collection, automation, storage and viewing of performance data. Replaces utlbstat/utlestat and Statspack. Grid Control (10g) provide a graphical, webbased, centralized management of databases. Tkprof allows to take and exploit SQL traces, gives explain plans etc Toad, SpotLight graphical tools (Quest)

43

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

Monitoring & Tuning (contd) : Methodology Summary


Performance issue? Monitor the system, diagnose the problem. Tools to use, questions to ask

CPU
topas, nmon vmstat (us, sys, id, r) sar P ALL CPU bound? runqueue? workload ? parallelization of workload?

MEMORY topas, nmon vmstat (pi, po) pagingout activity? memory allocated to Oracle? other applications running? other Oracle instances?

IO storage monitoring tool vmstat (wa) iostat, nmon, filemon accesses well spread? hot spots? jfs mount options? AIO? Network

Metalink note 282036.1 alert.log bdump,cdump,udump init.ora/spfile AWR Grid Control strategy of analyze? memory allocated to instances (SGA, PGA) tkprof (parsing, explain plans, bind variables)

( and check ulimits + AIXTHREAD_SCOPE=S...)

Topas, nmon netstat,netpmon

44

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

Agenda
Oracle Server Architecture Memory
AIX VMM tuning Oracle Memory Structure Allocation

IO
AIX LVM Striping AIX mount option Oracle ASM Asynchronous IO

CPU Monitoring & Tuning


Methodology & tools Focus on Two Oracle Tools: AWR & Grid Control

Conclusion

45

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

Automatic Workload Repository (AWR)


is the tool to use to collect information about the behavior of a database!
Collects, automates, stores performance statistics for problem determination. AWR is setuped during the creation of the database is automatic generates reports in html format reports more information than Statspack prefer AWR to statspack reports!

Snapshots (set of statistics gathered at a single time) are taken automatically each 60 minutes or can be run manually: execute dbms_workload_repository.create_snapshot();

AWR reports are generated by running $ORACLE_HOME/rdbms/admin/awrrpt.sql and providing 2 snapshot Ids. Can be also generated using the Grid Control.

To enable most of the new statistical gathering and advisors, ensure that the parameter STATISTICS_LEVEL is set to TYPICAL (recommended).
46 4/22/2008 2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

Automatic Workload Repository (contd)


Advised to run manually additional snapshots during performance issue phases / optimization campaign No shutdown between two snapshots! Choose the most representative snapshots IDs to generate the report.

snapshot period analyzed in AWR report

Compare reports to mesure the impact of tuning and to check the behavior of the database Use Metalink and/or the Oracle documentation to get the definition of events, statistics etc

47

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

Performance tuning advisors


Automatic Database Diagnostic Monitor (ADDM) Based on AWR, invoked automatically every time that a new AWR snapshot is generated Powerful self-diagnostic engine which : analyze the system indentify major problems recommend corrective actions!!! ADDM reports are generated by running $ORACLE_HOME/rdbms/admin/addmrpt.sql and providing 2 snapshot Ids Can also be generated using the Grid Control. In 11g, for RAC clusters, ADDM analyze the system at the cluster level.
NEW! 11g

SQL Tuning Advisor (STA) Detects missing or stale statistics, missing indexes Suggests new execution plan which can be applied to the system and used without any change in the application Advised to use STA through Grid Control but also possible to invoke it using DBMS_SQLTUNE package.
48 4/22/2008 2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

AWR Report : Header & Hit Ratios

49

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

AWR Report : Top 5 timed events

This Top 5 Timed Events table is very important it lists the events to focus on and will lead the performance analysis. According the event listed in the table, refer to the corresponding section in the report.

50

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

AWR Report : SQL ordered by

Sections SQL statement ordered by present SQL statements ordered by differents criterias. They are useful to detect a problem of parsing, a response time of a particular statement etc SQL is clickable and the whole statement is presented later in the report.

51

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

AWR Report : Cache sizes and advisors

Buffer Pool Advisory help to tune the buffer pool, giving estimation of the impact of changes of the size of the buffer pool. Advisors exist for: - PGA memory - Shared Pool - Sga_target -

52

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

Oracle Grid Control

Targets : databases

53

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

Oracle Grid Control

Focus on one database

54

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

Oracle Grid Control

Performance graphs

55

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

Oracle Grid Control

Advisor Central

56

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

Oracle Grid Control

ADDM (Automatic Database Diagnostic Monitor)

Focus needed on these statements!


57 4/22/2008 2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

Oracle Grid Control

ADDM (Automatic Database Diagnostic Monitor)

Run the advisor to get a tuning hint!

Statement with most impact on performances


58 4/22/2008 2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

Oracle Grid Control

SQL Tuning Advisor

You can implement the advice (or not!)


59 4/22/2008 2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

Oracle Grid Control

Focus on one query

60

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

Oracle Grid Control

Focus on one query

61

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

Oracle Grid Control

Grid is the most powerful tool to identify issues, get advices, tune

your database

It facilitates DB administration tasks and allows to perform the maintenance of the database (backups etc)

Targets monitored by the Grid are the hosts, databases, Oracle Applications Servers, Oracle eBusiness Suite etc

Allows to have a central point for the management of the targets

Helps the DBA, but does not replace him/her! ;-)

62

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

Agenda
Oracle Server Architecture Memory
AIX VMM tuning Oracle Memory Structure Allocation

IO
AIX LVM Striping AIX mount option Oracle ASM Asynchronous IO

CPU Monitoring & Tuning


Methodology & tools Focus on Two Oracle Tools: AWR & Grid Control

Conclusion

63

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

Conclusion
Implement and get the best of Oracle on AIX is the result of a joint effort between the OS administrator and the Oracle DBA. The tuning process is made of a first guess parameter setting and then, iterations to change progressively the configuration to get the best results Prefer JFS2 + Concurrent IO or ASM. Always use Asynchronous IO servers and, if available, use FastPath. Implement SMT if available on your system. Use AIX, Oracle and storage monitoring tools
(AIX : Nmon, Topas Oracle : AWR & Grid Control Storage : TPC)

and feel free to contact us for additional questions!!!

oraclibm@fr.ibm.com

64

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

Addendum - Useful note for Oracle running on AIX platforms


Before installing an Oracle product on AIX, it is strongly advised to have a look at the Metalink note 282036.1 : Minimum Software Versions and Patches Required to Support Oracle Products on IBM pSeries It contains all mandatory AIX PTFs & APARs and Oracle patches to install on a system is updated on a regular basis and

65

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

Addendum - Certification matrix : Oracle database Enterprise Edition on IBM AIX based Systems
10th of april 2008

When upgrading to Oracle 10GR2 on AIX 6.1, download and run an updated version of rootpre script before proceeding with the Oracle Database install (refer to bug 6613550 for more details).

66

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

Oracle/AIX Performance and Tuning Session ID : pS014 Fabienne.Lepetit@oracle.com s.chabrolles@fr.ibm.com

67

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

Reference Materials DRAFT -

Thanks to the authors of the following documents : Improving Database Performance with AIX Concurrent I/O S. Kashyap, B. Olszewski, R Hendrickson Performance Management Guide AIX5L version 5.3 documentation Oracle 9i & 10g on IBM AIX5L : Tips & Considerations IBM/Oracle ICC Oracle Architecture and Performance Tuning on AIX R. Ballough Configuring IBM TotalStorage for Oracle OLTP Applications D. Martin Tuning IBM AIX5L for an Oracle Database D. Massanari, M. Riviere

68

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

EMEA Oracle/IBM Joint Solutions Center


C/O Compagnie IBM France Rue de la Vieille Poste 34006 Montpellier Cedex - FRANCE

oraclibm@fr.ibm.com
Tel : +33 4 67 34 67 49

69

4/22/2008

2008 IBM Corporation

IBM Systems & Technology Group Technical Conference 2008

Trademarks
The following are trademarks of the International Business Machines Corporation in the United States, other countries, or both.
Not all common law marks used by IBM are listed on this page. Failure of a mark to appear does not mean that IBM does not use the mark nor does it mean that the product is not actively marketed or is not significant within its relevant market. Those trademarks followed by are registered trademarks of IBM in the United States; all others are trademarks or common law marks of IBM in the United States.

For a complete list of IBM Trademarks, see www.ibm.com/legal/copytrade.shtml:


*, AS/400, e business(logo), DBE, ESCO, eServer, FICON, IBM, IBM (logo), iSeries, MVS, OS/390, pSeries, RS/6000, S/30, VM/ESA, VSE/ESA, WebSphere, xSeries, z/OS, zSeries, z/VM, System i, System i5, System p, System p5, System x, System z, System z9, BladeCenter

The following are trademarks or registered trademarks of other companies.


Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries. Cell Broadband Engine is a trademark of Sony Computer Entertainment, Inc. in the United States, other countries, or both and is used under license therefrom. Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. UNIX is a registered trademark of The Open Group in the United States and other countries. Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. ITIL is a registered trademark, and a registered community trademark of the Office of Government Commerce, and is registered in the U.S. Patent and Trademark Office. IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency, which is now part of the Office of Government Commerce.
* All other products may be trademarks or registered trademarks of their respective companies. Notes: Performance is in Internal Throughput Rate (ITR) ratio based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput improvements equivalent to the performance ratios stated here. IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply. All customer examples cited or described in this presentation are presented as illustrations of the manner in which some customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics will vary depending on individual customer configurations and conditions. This publication was produced in the United States. IBM may not offer the products, services or features discussed in this document in other countries, and the information may be subject to change without notice. Consult your local IBM business contact for information on the product or services available in your area. All statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. Information about non-IBM products is obtained from the manufacturers of those products or their published announcements. IBM has not tested those products and cannot confirm the performance, compatibility, or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. Prices subject to change without notice. Contact your IBM representative or Business Partner for the most current pricing in your geography.

70

4/22/2008

2008 IBM Corporation

You might also like