IBM Systems & Technology Group Technical Conference

14 – 18 April, 2008, Sevilla, Spain

Oracle/AIX Performance and Tuning Session ID : pS014
Version 2.0 21.04.2008

Fabienne Lepetit – Oracle/IBM Joint Solutions Center Sébastien Chabrolles – IBM – Power Benchmark PSSC Montpellier
Fabienne.Lepetit@oracle.com / s.chabrolles@fr.ibm.com

4/22/2008

© 2008 IBM Corporation

IBM Systems & Technology Group – Technical Conference 2008

This presentation…
… is a Oracle/IBM joint work : IBM PSSC Power benchmark center & IBM/Oracle Joint Solutions Center.

… is based on our experience and feedback of benchmarks

… is a checklist which can be used in most production environment

… can be widely distributed to BPs and customers

… has been built with many written informations in order to be easily read and presented – don’t hesitate to reuse it!

2

4/22/2008

© 2008 IBM Corporation

IBM Systems & Technology Group – Technical Conference 2008

Learning objectives

Configure AIX to support Oracle with the best performance

Setup Oracle to take advantage of AIX features

Understand the joint effort to tune both Oracle and AIX and how they interact

3

4/22/2008

© 2008 IBM Corporation

IBM Systems & Technology Group – Technical Conference 2008 Agenda Oracle Server Architecture Memory AIX VMM tuning Oracle Memory Structure Allocation IO AIX LVM Striping AIX mount option Oracle ASM Asynchronous IO CPU Monitoring & Tuning Methodology & tools Focus on Two Oracle Tools: AWR & Grid Control Conclusion 4 4/22/2008 © 2008 IBM Corporation .

IBM Systems & Technology Group – Technical Conference 2008 Agenda Oracle Server Architecture Memory AIX VMM tuning Oracle Memory Structure Allocation IO AIX LVM Striping AIX mount option Oracle ASM Asynchronous IO CPU Monitoring & Tuning Methodology & tools Focus on Two Oracle Tools: AWR & Grid Control Conclusion 5 4/22/2008 © 2008 IBM Corporation .

IBM Systems & Technology Group – Technical Conference 2008 Oracle Server Architecture how to manage Oracle memory structures Background processes DBWR LGWR SMON … DIA0 Oracle instance Memory structures System Global Area Buffer Cache Shared SQL Log Buffer Program Global Area Private SQL Area how to implement the data on disk Oracle database Database files DB Files Redologs Control files 6 4/22/2008 © 2008 IBM Corporation .

indexes…) redologs – contains redo entries control files – record the physical structure of the database Oracle memory structures = System Global Area (SGA) + Process Global Area (PGA) SGA shared memory region that contains data and control information for one Oracle instance allocated at instance startup and deallocated at instance shutdown each instance has its own SGA PGA memory buffer that contains data and control information for a server process PGA is created by Oracle when a server process is started contains private SQL area 7 4/22/2008 © 2008 IBM Corporation .IBM Systems & Technology Group – Technical Conference 2008 Oracle Server Architecture (cont’d) The components of an Oracle server are : database = set of datafiles (data + redologs + control files) instance = memory structures allocated at startup + background processes (automatically forked at startup) associated to a database Database files datafiles – contains data (tables.

dictionnary cache .library cache • redolog buffer stores redo entries (log of changes made to the database) written to the online redo log (used during database recovery) System Global Area Buffer Cache SharedSQL Log Buffer Program Global Area Private SQL Area 8 4/22/2008 © 2008 IBM Corporation .store the most recently used blocks of data .improves performance by reducing disk I/O activity • shared pool contains : .shared SQL area (parse tree and execution plan for SQL statements) .IBM Systems & Technology Group – Technical Conference 2008 Oracle Server Architecture (cont’d) – Focus on the SGA The three main components of the SGA are : • buffer cache: .

IBM Systems & Technology Group – Technical Conference 2008 Agenda Oracle Server Architecture Memory AIX VMM tuning Oracle Memory Structure Allocation IO AIX LVM Striping AIX mount option Oracle ASM Asynchronous IO CPU Monitoring & Tuning Methodology & tools Focus on Two Oracle Tools: AWR & Grid Control Conclusion 9 4/22/2008 © 2008 IBM Corporation .

applications load some computational pages into the memory. the DB needs more memory but there is no free pages available. Objective : Tune the VMM to protect computational pages (Programs.IBM Systems & Technology Group – Technical Conference 2008 Virtual Memory Management 1 . 2 .With the default setting. PGA) from being paged out and force the LRUD to steal pages from FS-CACHE only. LRUD (AIX page stealer) is starting to free some pages into the memory.AIX is started. 10 4/22/2008 © 2008 IBM Corporation .The activity is increasing. LRUD will page out some computational pages instead of removing some pages from the File System Cache. 3 . SGA. AIX will try to take advantage of the free memory by using it as a cache file to reduce the IO on the physical drives. As a UNIX system.

► “New” vmo tuning approach : Recommendations based on VMM developer experience.3 ML1+ ► lru_file_repage : ● ● This new parametter prevents the computational pages to be paged out.2 ML4+ or AIX 5. vmstat “po” column maxfree = minfree +(maxpaghead x # lcpus) minperm% = from 3 to 10 maxperm% = maxclient% = from 70 to 90 strict_maxclient = 1 11 4/22/2008 © 2008 IBM Corporation .IBM Systems & Technology Group – Technical Conference 2008 Memory : VMM Tuning for Filesystems (lru_file_repage) This VMM tuning tip is applicable for AIX 5. By setting lru_file_repage=0 (1 is the default value) you’re telling the VMM that your preference is to steal pages from FS Cache only. Use “vmo –p –o” to set the following parametters : lru_file_repage = 0 minfree = 120 x # lcpus Increase if page stealing activity.

3 ML1.2 ML4 or AIX 5. vmstat “po” column maxfree = minfree +(maxpaghead x # lcpus) minperm% = from 3 to 10 maxperm% = maxclient% = from 5 to 15 but < minperm% strict_maxclient = 0 12 4/22/2008 © 2008 IBM Corporation . The following VMM tuning as the same objective : protect the computational pages to be page out.IBM Systems & Technology Group – Technical Conference 2008 Memory : VMM Tuning for Filesystems (“legacy”) Recommended VMM tuning for AIX version prior to 5. ► vmo “legacy” tuning approach : Use “vmo –p –o” to set the following parametters : minfree = 120 x # lcpus Increase if page stealing activity.

CAP_PROPAGATE oracle (allow Oracle user ID to use Large Pages) ● ldedit –b lpdata oracle (to allow oracle binary to use large page data) ● export LDR_CNTRL=LARGE_PAGE_TEXT=Y@LARGE_PAGE_DATA=M (prior Oracle user ID to starting Oracle instance and listener to allow both large page text and large page data) 13 4/22/2008 © 2008 IBM Corporation . AIXTHREAD_SCOPE=S is Internally set Pinning shared memory – prevents paging out SGA (only if App + DB on same AIX and >80% computationnal pages) ► On AIX part ● vmo –p –o v_pinshm = 1 ● Leave maxpin% at the default of 80% unless the SGA exceeds 77% of real memory : vmo –p –o maxpin%=[(total mem-SGA size)*100/total mem] + 3 ► On Oracle part (9i/10g) ● LOCK_SGA = TRUE Large Page Support – improves performance of prefetching ► On AIX part ● vmo –r –o lgpg_size = 16777216 –o lgpg_regions=(SGA size / 16 MB) ● chuser capabilities=CAP_BYPASS_RAC_VMM. set AIX environment variable AIXTHREAD_SCOPE=S ► Since Oracle10g.IBM Systems & Technology Group – Technical Conference 2008 Memory : AIX Memory Management for Oracle System-wide contention scope (user thread/kernel thread one-to-one mapping) – reduces memory usage of Oracle process & makes AIX post/wait mechanism most effective ► In Oracle9i.

►additional 10g : Automatic Shared Memory Management (ASMM) ►sga_target (dynamic) – if set the db_cache_size is automatically sized (shared_pool_size. large_pool_size. To use Automatic Shared Memory Management.IBM Systems & Technology Group – Technical Conference 2008 Memory : Oracle Memory Structures Allocation 8i : Manual management ► All Oracle memory structures were manually managed and were mostly static (db_block_buffers…) 9i : Dynamic memory resizing db_cache_size (dynamic parameter) ►sga_max_size (static parameter) – maximum size of the SGA for the lifetime of the instance. 14 4/22/2008 © 2008 IBM Corporation . java_pool_size as well). ►pga_aggregate_target (dynamic parameter) – specifies the target aggregate PGA memory available to all server processes attached to the instance parameter : db_cache_advice (dynamic parameter) – enables or disables statistics gathering used for predicting behavior with different cache sizes. Can be increased up to sga_max_size. sga_target must be <>0.

To use Automatic Memory Management. ►memory_max_target (static parameter) – specifies the maximum memory size for the instance.1 explaining AMM and these new parameters. memory_target must be <>0. Exchanges between SGA and PGA are done according to the needs. 15 4/22/2008 © 2008 IBM Corporation .IBM Systems & Technology Group – Technical Conference 2008 Memory : Oracle Memory Structures Allocation NEW! 11g 11g : Automatic Memory Management (AMM) ►memory_target (dynamic parameter) – specifies the total memory size to be used by the instance – SGA and PGA. See Metalink notes 443746. If sga_target and pga_aggregate_target are not set the policy is to give 60% of memory_target to the SGA and 40% to the PGA.1 and 452512.

so it needs to be increased as well.IBM Systems & Technology Group – Technical Conference 2008 Memory : Use jointly AIX dynamic LPAR and Oracle dynamic allocation of memory + AMM Initial configuration Memory_max_size = 18 GB Final configuration Scenario : Oracle tuning advisor indicates that SGA+PGA need to be increased to 11GB: memory_target can be increased dynamically to 11GB but real memory is only 12GB.Increase physical memory allocated to the LPAR to 15 GB ssh hscroot@hmc "chhwres -r mem -m <system> -o a -p <LPAR name> -q 3072" SGA + PGA ► Step 2 .Increase SGA+PGA allocated to the instance of the database to 11GB SGA + PGA alter system set memory_target=11GB. using AIX DLPAR Memory allocated to Oracle (SGA and PGA) has been increased on the fly 16 4/22/2008 © 2008 IBM Corporation . Memory allocated to the system has been increased dynamically. Memory_max_size = 18 GB Real memory = 15 GB Real memory = 12 GB AIX + free Memory_target = 11 GB AIX + free Memory_target = 8 GB ► Step 1 .

IBM Systems & Technology Group – Technical Conference 2008 Agenda Oracle Server Architecture Memory AIX VMM tuning Oracle Memory Structure Allocation IO AIX LVM Striping AIX mount option Oracle ASM Asynchronous IO CPU Monitoring & Tuning Methodology & tools Focus on Two Oracle Tools: AWR & Grid Control Conclusion 17 4/22/2008 © 2008 IBM Corporation .

IBM Systems & Technology Group – Technical Conference 2008

IO : Database Layout

Having a good Storage configuration is a key point : Because disk is the slowest part of an infrastructure Reconfiguration can be difficult and time consuming

Stripe and mirror everything (S.A.M.E) approach: Goal is to balance I/O activity across all disks, loops, adapters, etc... Avoid/Eliminate I/O hotspots

rec O om rac me le nd a ti o

n

Manual file-by-file data placement is time consuming, resource intensive and iterative Additional advices to implement SAME :
• apply the SAME strategy to data, indexes • if possible separate redologs (+archivelogs)

18

4/22/2008

© 2008 IBM Corporation

IBM Systems & Technology Group – Technical Conference 2008

IO : RAID Policy with ESS, DS6/8K
Storage
RAID-5 vs. RAID-10 Performance Comparison I/O Profile Sequential Read Sequential Write Random Read Random Write RAID-5 Excellent Excellent Excellent Fair RAID-10 Excellent Good Excellent Excellent

HW Striping

LUN 1 LUN 2 LUN 3 LUN 4

With Enterprise class storage (with huge cache), RAID-5 performances are comparable to
RAID-10 one (for most customer workloads) Consider RAID-10 for workloads with a high percentage of random write activity (> 25%) and high I/O access densities (peak > 50%)

Use RAID-5 or RAID-10 to create striped LUNs If possible try to minimize the number of LUNs per RAID array to avoid contention on physical disk.

19

4/22/2008

© 2008 IBM Corporation

IBM Systems & Technology Group – Technical Conference 2008

IO : 2nd Striping (LVM)
AIX
Volume Group

Storage

HW Striping

LV (raw or with jfs2)

hdisk1 hdisk2 hdisk3 hdisk4

LUN 1 LUN 2 LUN 3 LUN 4

1. 2. 3. 4.

Luns are striped across physical disks (stripe-size : ~ 64k, 128k, 256k) LUNs are seen as hdisk device on AIX server. Create AIX Volume Group(s) (VG) with LUNs from multiple arrays Logical Volume striped across hdisks (stripe-size : 8M, 16M, 32M, 64M) => each read/write access to the LV are well balanced accross LUNs and use the maximum number of physical disks for best performance.

20

LVM Striping

4/22/2008

© 2008 IBM Corporation

Stripe using Logical Volume (LV) ► Create Logical Volume with the striping option : mklv –S <stripe-size> . 16M. 512k. 32k.. jfs2 filesystem creation advice : If you create a jfs2 filesystem on a striped (or PP spreaded) LV.16M or 32M PPsize..3 +: Choose a Scalable Volume Group : # mkvg –S –s <PPsize> ..3+: AIX 5. 128M 2.. 256k. 1M 1. use the INLINE logging option. (PPsize will be the “Stripe size”) ● AIX 5... Create LV with “Maximum range of physical volume” option to spread PP on different hdisk in a Round Robin fashion : # mklv –e x .2 : Choose a Big VG and change the “t factor” : # mkvg –B –t <factor> –s <PPsize>. # crfs –a logname=INLINE … 21 4/22/2008 © 2008 IBM Corporation . ● AIX 5..2 Stripe sizes + 2M. 4M. 8k.IBM Systems & Technology Group – Technical Conference 2008 IO : LVM Spreading / Striping 2 different ways to stripe on the LVM layer : 1. 128k. 16k. PP striping (AKA spreading) ► ► Create a Volume Group with a 8M. AIX 5. It will avoid « hot spot » by creating the log inside the filesystem (which is striped) instead of using a uniq PP stored on 1 hdisk. 64k. 64M.2: 4k. Valid LV Stripe sizes: ● AIX 5.. 32M.

a generic device definition provides far from optimal performance since it doesn’t properly customize the hdisk device : exemple : hdisk are created with a queue_depth=1 1. # lsdev –Cc disk Generic device definition bad performance In general..sdd or sddpcm for IBM DS6000/DS8000 . Contact your vendor or go to their web site to download the correct ODM definition for your storage subsystem. If AIX is connected to the storage subsystem with several Fiber Channel Cards for performance.powerpath for EMC disk subsystem .. It will setup properly the “hdisk” accordingly to your hardware for optimal performance. If the description shown in the output of “lsdev –Cc disk” the word “Other”. don’t forget to install a multipath device driver or path control module.. then it means that AIX doesn’t have a correct definition of your disk device in the ODM and use a generic device definition.IBM Systems & Technology Group – Technical Conference 2008 IO : Disk Subsystem Advices Check if the definition of your disk subsystem is present in the ODM.hdlm for Hitachi etc. . . 22 4/22/2008 © 2008 IBM Corporation 2.

IBM Systems & Technology Group – Technical Conference 2008 Agenda Oracle Server Architecture Memory AIX VMM tuning Oracle Memory Structure Allocation IO AIX LVM Striping AIX mount option Oracle ASM Asynchronous IO CPU Monitoring & Tuning Methodology & tools Focus on Two Oracle Tools: AWR & Grid Control Conclusion 23 4/22/2008 © 2008 IBM Corporation .

4/22/2008 © 2008 IBM Corporation .3. • Data is transfered directly from the disk to the application buffer. • Performance achieved using CIO is comparable to raw-devices. • Emulates a raw-device implementation. To mount a filesystem in DIO $ mount –o dio /data Concurrent IO (CIO) – introduced with jfs2 in AIX 5. some mount option can improve performance : Direct IO (DIO) – introduced in AIX 4. bypassing the file buffer cache hence avoiding double caching (filesystem cache + Oracle SGA).IBM Systems & Technology Group – Technical Conference 2008 IO : Filesystems Mount Options (DIO.2 ML1 • Implicit use of DIO. • No Inode locking : Multiple threads can perform reads and writes on the same file at the same time. To mount a filesystem in CIO: $ mount –o cio /data 24 Bench throughput over run duration – higher tps indicates better performance. CIO) If Oracle data are stored in a Filesystem.

16. . then jfs2 agblksize=4096 Redolog are always written in 512B block.. use the “mkfs –o agblksize=XXX” Option to adapt the FS blocksize with the application needs. . 3. Rule : IO request = n x agblksize Exemples: if DB blocksize > 4k . 32. 128 according to workload 2. Because data transfer is bypassing AIX buffer cache. jfs2 prefetching and write-behind can’t be used.IBM Systems & Technology Group – Technical Conference 2008 IO : Benefits of CIO for Oracle Benefits : 1. ⇒ (Oracle parameter) db_file_multiblock_read_count = 8. IO requests made by Oracle must by aligned with the jfs2 blocksize to avoid a demoted IO (Return to normal IO after a Direct IO Failure) => When you create a JFS2. Avoid double caching : Some data are already cache in the Application layer (SGA) Give a faster access to the backend disk and reduce the CPU utilization Disable the inode-lock to allow several threads to read and write the same file (CIO only) Restrictions : 1. 2. These functionnalities can be handled by Oracle.. When using DIO/CIO. So jfs2 agblksize must be 512 25 4/22/2008 © 2008 IBM Corporation .

00 log file sync How to detect demoted IO : Trace command to check demoted io : # trace –aj 59B. trcrpt –o directio.905 Time -outs Total Wait Time (s) 0. FS blk=4k) Waits% Time -outs Total Wait Time (s) 2.59C . FS blk=512) Waits% 494.00 1. sleep 2 .628 Avg wait (ms) 28 Waits /txn 1.53 log file sync Waits on redolog (without demoted IO.trcrpt # grep demoted directio.00 62.trcrpt 26 4/22/2008 © 2008 IBM Corporation .IBM Systems & Technology Group – Technical Conference 2008 IO : Direct IO demoted Extract from Oracle AWR (test made in Montpellier with Oracle 10g) Waits on redolog (with demoted IO.324 0.073 Avg wait (ms) 2 Waits /txn 1.229. trcstop .

IBM Systems & Technology Group – Technical Conference 2008 CIO/DIO implementation Advices with Standard mount options mount -o rw Oracle bin and shared lib. Cached by AIX mount -o rw Cached by Oracle Cached by AIX mount -o rw Cached by Oracle Cached by AIX mount -o rw Cached by AIX mount -o rw Oracle Control files Cached by AIX with optimized mount options mount -o rw Cached by AIX mount -o cio *(1) Cached by Oracle mount -o cio (jfs2 + agblksize=512) Cached by Oracle mount -o rbrw Use JFS2 write-behind … but are not kept in AIX Cache. mount -o rw Cached by AIX Oracle Datafiles Oracle Redolog Oracle Archivelog *(1) : to avoid demoted IO : jfs2 agblksize = Oracle DB block size / n 27 4/22/2008 © 2008 IBM Corporation .

IBM Systems & Technology Group – Technical Conference 2008 Agenda Oracle Server Architecture Memory AIX VMM tuning Oracle Memory Structure Allocation IO AIX LVM Striping AIX mount option Oracle ASM Asynchronous IO CPU Monitoring & Tuning Methodology & tools Focus on Two Oracle Tools: AWR & Grid Control Conclusion 28 4/22/2008 © 2008 IBM Corporation .

Alternative to conventional file-system and LVM functions.IBM Systems & Technology Group – Technical Conference 2008 IO : Oracle Automatic Storage Management (ASM) • ASM is a database file system which provide cluster file system and volume manager capabilities. • Integrated to Oracle DB with no additional cost. Linux. the management of Oracle datafiles is the same for the DBA on all platforms (Unix.I/O is spread evenly to prevent hot spots and maximize performance • Online add/drop of disk devices with automatic online redistribution of data Oracle AIX Storage hdisk1 LUN 1 LUN 2 LUN 3 LUN 4 ASM DISKGROUP hdisk2 hdisk3 hdisk4 ASM striping 29 4/22/2008 © 2008 IBM Corporation . For Oracle datafiles. for single or RAC databases • With ASM. Windows) • Datafiles are striped across all ASM disks .

• ASM is implemented as a special kind of Oracle instance. 30 4/22/2008 © 2008 IBM Corporation . with its own SGA and background processes (by default.dbf’ size 15G.IBM Systems & Technology Group – Technical Conference 2008 IO : Oracle Automatic Storage Management (ASM) Database on ASM ASM Instance ASM DISKGROUP create tablespace DATA datafile ‘+DG/data01. • Can maintain redundant copies of data • An ASM-managed database will have approximately the same performance as a database implemented in raw-devices. • A single ASM instance can service one or more instance databases. less than 100M of SGA are allocated for the ASM instance) and generate very little activity except during rebalance operations.

IBM Systems & Technology Group – Technical Conference 2008 Agenda Oracle Server Architecture Memory AIX VMM tuning Oracle Memory Structure Allocation IO AIX LVM Striping AIX mount option Oracle ASM Asynchronous IO CPU Monitoring & Tuning Methodology & tools Focus on Two Oracle Tools: AWR & Grid Control Conclusion 31 4/22/2008 © 2008 IBM Corporation .

JFS2. 1 Application 2 aio Q 3 aioservers 4 Disk Posix vs Legacy Since AIX5L V5. • Utilization of asynchronous IO is strongly advised whatever the type of file-system and mount option implemented (JFS.IBM Systems & Technology Group – Technical Conference 2008 IO : Asynchronous IO (AIO) • Allows multiple requests to be sent without to have to wait until the disk subsystem has completed the physical IO. For the moment. the Oracle code is using the Legacy AIO servers. 32 4/22/2008 © 2008 IBM Corporation . CIO.3. DIO). two types of AIO are now available : Legacy and Posix.

X : lsattr –El aio0 AIX 6. if maxservers set to low : “Warning: lio_listio returned EAGAIN” “Performance degradation may be seen” #aio servers used can be monitored via “ps –k | grep aio | wc –l” . • maxserver : maximum number of aioserver that can be running per logical CPU Rule of thumb : maxservers should be = (10 * <# of disk accessed concurrently>) / # cpu maxreqs (= a multiple of 4096) should be > 4 * #disks * queue_depth but only tests allow to set correctly minservers and maxservers Monitoring : In Oracle’s alert. Aioservers to start (AIX 5L system wide). • minserver : number of kernel proc.1 : ioo –L | grep aio • maxreqs : size of the Asynchronous queue.log file. “iostat –A” or nmon (option A) 33 4/22/2008 © 2008 IBM Corporation .IBM Systems & Technology Group – Technical Conference 2008 IO : Asynchronous IO (AIO) tuning • important tuning parameters : check AIO configuration with : AIX 5.

: AIX 5L : chdev -a fastpath=enable -l aio0 (default since AIX 5.3) AIX 6. Better performance compare to non-fast_path No need to tune the min and max aioservers No ioservers proc.1 : ioo –p –o aio_fsfastpath=1 (default setting) 34 4/22/2008 © 2008 IBM Corporation .1 : ioo –p –o aio_fastpath=1 (default setting) • FS with CIO/DIO and AIX 5. IO are queued directly from the application into the LVM layer without any “aioservers kproc” operation. use “iostat –A” instead 1 3 Application 2 • Raw Devices / ASM : AIX Kernel Disk check AIO configuration with : lsattr –El aio0 enable asynchronous IO fast_path.IBM Systems & Technology Group – Technical Conference 2008 IO : Asynchronous IO (AIO) fastpath With fast_path.3 TL5+ : Activate fsfast_path (comparable to fast_path but for FS + CIO/DIO) AIX 5L : adding the following line in /etc/inittab: aioo:2:once:aioo –o fsfast_path=1 AIX 6. => “ps –k | grep aio | wc –l” is not relevent.

DIO/CIO & Oracle Parameters How to set filesystemio_options parameter Possible values ASYNCH : enables asynchronous I/O on file system files (default) DIRECTIO : enables direct I/O on file system files (disables AIO) SETALL : enables both asynchronous and direct I/O on file system files NONE : disables both asynchronous and direct I/O on file system files Since the version 10g.IBM Systems & Technology Group – Technical Conference 2008 IO : AIO. Advice : set this parameter to ‘setall’… Note : set the disk_asynch_io parameter to ‘true’ as well 35 4/22/2008 © 2008 IBM Corporation . Oracle will open data files located on the JFS2 file system with the O_CIO option if the filesystemio_options initialization parameter is set to either directIO or setall.

IBM Systems & Technology Group – Technical Conference 2008 Agenda Oracle Server Architecture Memory AIX VMM tuning Oracle Memory Structure Allocation IO AIX LVM Striping AIX mount option Oracle ASM Asynchronous IO CPU Monitoring & Tuning Methodology & tools Focus on Two Oracle Tools: AWR & Grid Control Conclusion 36 4/22/2008 © 2008 IBM Corporation .

a server with 4 physical processors will have 8 logical processors with SMT enabled) • SMT allows to use resources that would otherwise usually be unexploited and can achieve significant throughput gains – up to 40% greater performance • No source code modification needed to use SMT How to control SMT: smtcl [–m on | off] “-m on” enables the SMT mode.e.3 or 6.IBM Systems & Technology Group – Technical Conference 2008 CPU : Simultaneous Multi-Threading (SMT) • Feature available on POWER5 and POWER6 system running AIX 5. “-m off” disables it “-w boot” makes SMT change effective on reboot “-w now” makes SMT change effective immediately but not persistent across reboots 37 4/22/2008 © 2008 IBM Corporation thread1 thread2 Register thread1 Register thread2 P6 Core .1 • allows 2 instruction path to share access to the POWER execution units on every clock cycle • instruction path = “virtual processor” = logical POWER processor (i.

It improve CPU utilisation by reducing the number of idle cycle.037. SPLPAR one) 38 4/22/2008 © 2008 IBM Corporation .IBM Systems & Technology Group – Technical Conference 2008 CPU: Micro-Partitioning Micro-Partitioning give the possibility to share processors between several LPARs.373.256 secs An example given based on a PeopleSoft/Oracle10g RAC benchmark (2.715.065 Action_Transaction: 53.07 secs SPLPAR Configuration Total Throughput (bytes):2.400 Virtual Users) ran at PSSC MOP (DLPAR performance vs.677.3+) DLPAR Configuration Total Throughput (bytes):2.193 Action_Transaction: 59. (available on POWER5 and POWER6 system running AIX 5.

1 Oracle 9i Oracle 10g Oracle 11g 39 4/22/2008 © 2008 IBM Corporation Micro.IBM Systems & Technology Group – Technical Conference 2008 CPU: Compatibility Matrix DLPAR SMT AIX 5.3 AIX 6.Partition .2 AIX 5.

IBM Systems & Technology Group – Technical Conference 2008 Agenda Oracle Server Architecture Memory AIX VMM tuning Oracle Memory Structure Allocation IO AIX LVM Striping AIX mount option Oracle ASM Asynchronous IO CPU Monitoring & Tuning Methodology & tools Focus on Two Oracle Tools: AWR & Grid Control Conclusion 40 4/22/2008 © 2008 IBM Corporation .

IBM Systems & Technology Group – Technical Conference 2008 Monitoring & Tuning : Methodology DBA System administrator OS monitoring Oracle monitoring log of changes design/parameters changes impact ? tuning is an iterative process a minimun of changes should be performed at each iteration tuning is a joint effort between the system administrator and the DBA don’t hesitate to keep a log to track changes and the impact of these changes 41 4/22/2008 © 2008 IBM Corporation .

and memory usage lvmstat – useful to get logical volume I/O statistics iostat – allows to get statistics on disk activity (and also on terminals.. logical volumes.IBM Systems & Technology Group – Technical Conference 2008 Monitoring & Tuning (cont’d) : System Monitoring Tools vmstat – useful for obtaining an overall picture of CPU. CPU. paging. individual files. and the Virtual Memory Manager xmperf – allows to define monitoring environments to supervise the performance of the local and remote systems netstat – allows to monitor network activity . adapters) lparstat – reports logical partition related information and statistics nmon – complete tool which gives information on all the components of the system filemon – uses the trace facility to report on the I/O activity of physical volumes.. 42 4/22/2008 © 2008 IBM Corporation ..

Grid Control (10g) provide a graphical. webbased. SpotLight … graphical tools (Quest) 43 4/22/2008 © 2008 IBM Corporation . storage and viewing of performance data. automation. Replaces utlbstat/utlestat and Statspack. centralized management of databases. Tkprof allows to take and exploit SQL traces. gives explain plans etc… Toad.IBM Systems & Technology Group – Technical Conference 2008 Monitoring & Tuning (cont’d) : Oracle Monitoring Tools AWR is a standard tool which allow the collection.

filemon accesses well spread? hot spots? jfs mount options? AIO? Network Metalink note 282036. id.. r) sar –P ALL CPU bound? runqueue? workload ? parallelization of workload? MEMORY topas. bind variables…) ( and check ulimits + AIXTHREAD_SCOPE=S.ora/spfile AWR Grid Control strategy of analyze? memory allocated to instances (SGA. po) pagingout activity? memory allocated to Oracle? other applications running? other Oracle instances? IO storage monitoring tool vmstat (wa) iostat. nmon vmstat (pi. diagnose the problem. nmon. nmon netstat. sys..1 alert.udump… init.) Topas. nmon vmstat (us. questions to ask… CPU topas. Tools to use.log bdump.IBM Systems & Technology Group – Technical Conference 2008 Monitoring & Tuning (cont’d) : Methodology Summary Performance issue? Monitor the system.cdump. explain plans. PGA) tkprof (parsing.netpmon 44 4/22/2008 © 2008 IBM Corporation .

IBM Systems & Technology Group – Technical Conference 2008 Agenda Oracle Server Architecture Memory AIX VMM tuning Oracle Memory Structure Allocation IO AIX LVM Striping AIX mount option Oracle ASM Asynchronous IO CPU Monitoring & Tuning Methodology & tools Focus on Two Oracle Tools: AWR & Grid Control Conclusion 45 4/22/2008 © 2008 IBM Corporation .

46 4/22/2008 © 2008 IBM Corporation . ensure that the parameter STATISTICS_LEVEL is set to TYPICAL (recommended).IBM Systems & Technology Group – Technical Conference 2008 Automatic Workload Repository (AWR) … is the tool to use to collect information about the behavior of a database! • Collects. Can be also generated using the Grid Control. • AWR is setuped during the creation of the database is automatic generates reports in html format reports more information than Statspack prefer AWR to statspack reports! • Snapshots (set of statistics gathered at a single time) are taken automatically each 60 minutes or can be run manually: execute dbms_workload_repository. • AWR reports are generated by running $ORACLE_HOME/rdbms/admin/awrrpt. automates.create_snapshot().sql and providing 2 snapshot Ids. stores performance statistics for problem determination. • To enable most of the new statistical gathering and advisors.

statistics etc… 47 4/22/2008 © 2008 IBM Corporation .IBM Systems & Technology Group – Technical Conference 2008 Automatic Workload Repository (cont’d) • Advised to run manually additional snapshots during performance issue phases / optimization campaign • No shutdown between two snapshots! • Choose the most representative snapshots IDs to generate the report. snapshot period analyzed in AWR report • Compare reports to mesure the impact of tuning and to check the behavior of the database •Use Metalink and/or the Oracle documentation to get the definition of events.

ADDM analyze the system at the cluster level.IBM Systems & Technology Group – Technical Conference 2008 Performance tuning advisors Automatic Database Diagnostic Monitor (ADDM) • Based on AWR. invoked automatically every time that a new AWR snapshot is generated • Powerful self-diagnostic engine which : analyze the system indentify major problems recommend corrective actions!!! • ADDM reports are generated by running $ORACLE_HOME/rdbms/admin/addmrpt. for RAC clusters. • In 11g. 48 4/22/2008 © 2008 IBM Corporation .sql and providing 2 snapshot Ids – Can also be generated using the Grid Control. NEW! 11g SQL Tuning Advisor (STA) • Detects missing or stale statistics. missing indexes… • Suggests new execution plan which can be applied to the system and used without any change in the application • Advised to use STA through Grid Control – but also possible to invoke it using DBMS_SQLTUNE package.

IBM Systems & Technology Group – Technical Conference 2008 AWR Report : Header & Hit Ratios … 49 4/22/2008 © 2008 IBM Corporation .

IBM Systems & Technology Group – Technical Conference 2008 AWR Report : Top 5 timed events This “Top 5 Timed Events” table is very important – it lists the events to focus on and will lead the performance analysis. refer to the corresponding section in the report. 50 4/22/2008 © 2008 IBM Corporation . According the event listed in the table.

a response time of a particular statement etc… SQL is clickable and the whole statement is presented later in the report.IBM Systems & Technology Group – Technical Conference 2008 AWR Report : SQL ordered by… Sections « SQL statement ordered by… » present SQL statements ordered by differents criterias. 51 4/22/2008 © 2008 IBM Corporation . They are useful to detect a problem of parsing.

PGA memory .Shared Pool . Advisors exist for: .Sga_target -… 52 4/22/2008 © 2008 IBM Corporation . giving estimation of the impact of changes of the size of the buffer pool.IBM Systems & Technology Group – Technical Conference 2008 AWR Report : Cache sizes and advisors Buffer Pool Advisory help to tune the buffer pool.

IBM Systems & Technology Group – Technical Conference 2008 Oracle Grid Control Targets : databases 53 4/22/2008 © 2008 IBM Corporation .

IBM Systems & Technology Group – Technical Conference 2008 Oracle Grid Control Focus on one database 54 4/22/2008 © 2008 IBM Corporation .

IBM Systems & Technology Group – Technical Conference 2008 Oracle Grid Control Performance graphs 55 4/22/2008 © 2008 IBM Corporation .

IBM Systems & Technology Group – Technical Conference 2008 Oracle Grid Control Advisor Central 56 4/22/2008 © 2008 IBM Corporation .

IBM Systems & Technology Group – Technical Conference 2008 Oracle Grid Control ADDM (Automatic Database Diagnostic Monitor) Focus needed on these statements! 57 4/22/2008 © 2008 IBM Corporation .

IBM Systems & Technology Group – Technical Conference 2008 Oracle Grid Control ADDM (Automatic Database Diagnostic Monitor) Run the advisor to get a tuning hint! Statement with most impact on performances 58 4/22/2008 © 2008 IBM Corporation .

IBM Systems & Technology Group – Technical Conference 2008 Oracle Grid Control SQL Tuning Advisor You can implement the advice (or not!) 59 4/22/2008 © 2008 IBM Corporation .

IBM Systems & Technology Group – Technical Conference 2008 Oracle Grid Control Focus on one query 60 4/22/2008 © 2008 IBM Corporation .

IBM Systems & Technology Group – Technical Conference 2008 Oracle Grid Control Focus on one query 61 4/22/2008 © 2008 IBM Corporation .

get advices. but does not replace him/her! .IBM Systems & Technology Group – Technical Conference 2008 Oracle Grid Control • Grid is the most powerful tool to identify issues. databases. Oracle eBusiness Suite etc… • Allows to have a central point for the management of the targets • Helps the DBA. tune your database • It facilitates DB administration tasks and allows to perform the maintenance of the database (backups etc…) • Targets monitored by the Grid are the hosts.-) 62 4/22/2008 © 2008 IBM Corporation . Oracle Applications Servers.

IBM Systems & Technology Group – Technical Conference 2008 Agenda Oracle Server Architecture Memory AIX VMM tuning Oracle Memory Structure Allocation IO AIX LVM Striping AIX mount option Oracle ASM Asynchronous IO CPU Monitoring & Tuning Methodology & tools Focus on Two Oracle Tools: AWR & Grid Control Conclusion 63 4/22/2008 © 2008 IBM Corporation .

• The tuning process is made of a first guess parameter setting and then. use FastPath. if available. Topas – Oracle : AWR & Grid Control – Storage : TPC…) • … and feel free to contact us for additional questions!!! oraclibm@fr. • Use AIX. Oracle and storage monitoring tools (AIX : Nmon.ibm. • Always use Asynchronous IO servers and.com 64 4/22/2008 © 2008 IBM Corporation . • Implement SMT if available on your system.IBM Systems & Technology Group – Technical Conference 2008 Conclusion • Implement and get the best of Oracle on AIX is the result of a joint effort between the OS administrator and the Oracle DBA. iterations to change progressively the configuration to get the best results… • Prefer JFS2 + Concurrent IO or ASM.

1 : Minimum Software Versions and Patches Required to Support Oracle Products on IBM pSeries It contains all mandatory AIX PTFs & APARs and Oracle patches to install on a system is updated on a regular basis and 65 4/22/2008 © 2008 IBM Corporation .Useful note for Oracle running on AIX platforms •Before installing an Oracle product on AIX. it is strongly advised to have a look at the Metalink note 282036.IBM Systems & Technology Group – Technical Conference 2008 Addendum .

66 4/22/2008 © 2008 IBM Corporation .IBM Systems & Technology Group – Technical Conference 2008 Addendum .1.Certification matrix : Oracle database Enterprise Edition on IBM AIX based Systems 10th of april 2008 When upgrading to Oracle 10GR2 on AIX 6. download and run an updated version of rootpre script before proceeding with the Oracle Database install (refer to bug 6613550 for more details).

Lepetit@oracle.IBM Systems & Technology Group – Technical Conference 2008 Oracle/AIX Performance and Tuning Session ID : pS014 Fabienne.com 67 4/22/2008 © 2008 IBM Corporation .com s.chabrolles@fr.ibm.

Riviere 68 4/22/2008 © 2008 IBM Corporation . Ballough • Configuring IBM TotalStorage for Oracle OLTP Applications – D.3 documentation • Oracle 9i & 10g on IBM AIX5L : Tips & Considerations – IBM/Oracle ICC • Oracle Architecture and Performance Tuning on AIX – R. Martin • Tuning IBM AIX5L for an Oracle Database – D. R Hendrickson • Performance Management Guide – AIX5L version 5. B. M. Kashyap. Massanari.IBM Systems & Technology Group – Technical Conference 2008 Reference Materials – DRAFT - Thanks to the authors of the following documents : • Improving Database Performance with AIX Concurrent I/O – S. Olszewski.

ibm.IBM Systems & Technology Group – Technical Conference 2008 EMEA Oracle/IBM Joint Solutions Center C/O Compagnie IBM France Rue de la Vieille Poste 34006 Montpellier Cedex .com Tel : +33 4 67 34 67 49 69 4/22/2008 © 2008 IBM Corporation .FRANCE oraclibm@fr.

Itanium. Intel.com/legal/copytrade. DBE. xSeries®. other countries. System p.shtml: *. Not all common law marks used by IBM are listed on this page. other countries. Windows NT. compatibility. IBM®. e business(logo)®. see www. System i5. and the Windows logo are trademarks of Microsoft Corporation in the United States. All statements regarding IBM's future direction and intent are subject to change or withdrawal without notice. This publication was produced in the United States. IBM may not offer the products. ITIL is a registered trademark. and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States. Consult your local IBM business contact for information on the product or services available in your area. all others are trademarks or common law marks of IBM in the United States. PostScript. Intel Xeon. RS/6000®. AS/400®. or both and is used under license therefrom. Information about non-IBM products is obtained from the manufacturers of those products or their published announcements. Those trademarks followed by ® are registered trademarks of IBM in the United States. Intel Inside logo. pSeries®. or both. and/or other countries. Inc. Windows. The actual throughput that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream. Regardless. VSE/ESA. Notes: Performance is in Internal Throughput Rate (ITR) ratio based on measurements and projections using standard IBM benchmarks in a controlled environment. * All other products may be trademarks or registered trademarks of their respective companies. z/VM®. Intel SpeedStep. the I/O configuration. in the United States. Java and all Java-based trademarks are trademarks of Sun Microsystems. Intel Inside. Actual environmental costs and performance characteristics will vary depending on individual customer configurations and conditions. Failure of a mark to appear does not mean that IBM does not use the mark nor does it mean that the product is not actively marketed or is not significant within its relevant market. our warranty terms apply. Therefore. IBM (logo)®. or both. and is registered in the U. IBM has not tested those products and cannot confirm the performance. services or features discussed in this document in other countries. iSeries®. ESCO. UNIX is a registered trademark of The Open Group in the United States and other countries. IBM hardware products are manufactured from new parts. S/30. System x. other countries. and represent goals and objectives only. no assurance can be given that an individual user will achieve throughput improvements equivalent to the performance ratios stated here. 70 4/22/2008 © 2008 IBM Corporation . other countries. z/OS®. System z. and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. and the information may be subject to change without notice. System z9®. the Adobe logo. System i. Cell Broadband Engine is a trademark of Sony Computer Entertainment. the storage configuration. Celeron. Prices subject to change without notice.IBM Systems & Technology Group – Technical Conference 2008 Trademarks The following are trademarks of the International Business Machines Corporation in the United States. or both. and the workload processed. IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency. Patent and Trademark Office. Intel Centrino. which is now part of the Office of Government Commerce. All customer examples cited or described in this presentation are presented as illustrations of the manner in which some customers have used IBM products and the results they may have achieved. FICON. System p5. or both. WebSphere®. and a registered community trademark of the Office of Government Commerce. eServer. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. other countries. in the United States. For a complete list of IBM Trademarks. Inc. OS/390®. or any other claims related to non-IBM products. MVS. or new and serviceable used parts.S. VM/ESA®. Microsoft. Intel Centrino logo. BladeCenter® The following are trademarks or registered trademarks of other companies. Contact your IBM representative or Business Partner for the most current pricing in your geography. Linux is a registered trademark of Linus Torvalds in the United States. zSeries®. Intel logo. Adobe.ibm.