Professional Documents
Culture Documents
OracleAIX Performance and Tuning
OracleAIX Performance and Tuning
Fabienne Lepetit Oracle/IBM Joint Solutions Center Sbastien Chabrolles IBM Power Benchmark PSSC Montpellier
Fabienne.Lepetit@oracle.com / s.chabrolles@fr.ibm.com
4/22/2008
This presentation
is a Oracle/IBM joint work : IBM PSSC Power benchmark center & IBM/Oracle Joint Solutions Center.
has been built with many written informations in order to be easily read and presented dont hesitate to reuse it!
4/22/2008
Learning objectives
Understand the joint effort to tune both Oracle and AIX and how they interact
4/22/2008
Agenda
Oracle Server Architecture Memory
AIX VMM tuning Oracle Memory Structure Allocation
IO
AIX LVM Striping AIX mount option Oracle ASM Asynchronous IO
Conclusion
4/22/2008
Agenda
Oracle Server Architecture Memory
AIX VMM tuning Oracle Memory Structure Allocation
IO
AIX LVM Striping AIX mount option Oracle ASM Asynchronous IO
Conclusion
4/22/2008
Background processes
DBWR
LGWR
SMON
DIA0
Oracle instance
Memory structures
Oracle database
4/22/2008
4/22/2008
Agenda
Oracle Server Architecture Memory
AIX VMM tuning Oracle Memory Structure Allocation
IO
AIX LVM Striping AIX mount option Oracle ASM Asynchronous IO
Conclusion
4/22/2008
Objective :
Tune the VMM to protect computational pages (Programs, SGA, PGA) from being paged out and force the LRUD to steal pages from FS-CACHE only.
10
4/22/2008
lru_file_repage :
This new parametter prevents the computational pages to be paged out. By setting lru_file_repage=0 (1 is the default value) youre telling the VMM that your preference is to steal pages from FS Cache only.
Use vmo p o to set the following parametters : lru_file_repage = 0 minfree = 120 x # lcpus
The following VMM tuning as the same objective : protect the computational pages to be page out.
12
4/22/2008
13
4/22/2008
All Oracle memory structures were manually managed and were mostly static (db_block_buffers)
(static parameter) maximum size of the SGA for the lifetime of the instance.
pga_aggregate_target
(dynamic parameter) specifies the target aggregate PGA memory available to all server processes attached to the instance
parameter : db_cache_advice (dynamic parameter) enables or disables statistics gathering used for predicting behavior with different cache sizes.
additional
(dynamic) if set the db_cache_size is automatically sized (shared_pool_size, large_pool_size, java_pool_size as well). Can be increased up to sga_max_size. To use Automatic Shared Memory Management, sga_target must be <>0.
14
4/22/2008
NEW! 11g
(dynamic parameter) specifies the total memory size to be used by the instance SGA and PGA. Exchanges between SGA and PGA are done according to the needs. If sga_target and pga_aggregate_target are not set the policy is to give 60% of memory_target to the SGA and 40% to the PGA.
memory_max_target
(static parameter) specifies the maximum memory size for the instance.
To use Automatic Memory Management, memory_target must be <>0. See Metalink notes 443746.1 and 452512.1 explaining AMM and these new parameters.
15
4/22/2008
Memory : Use jointly AIX dynamic LPAR and Oracle dynamic allocation of memory + AMM
Initial configuration
Memory_max_size = 18 GB
Real memory = 12 GB
AIX + free
Memory_target = 11 GB
AIX + free
Memory_target = 8 GB
SGA + PGA
SGA + PGA
Memory allocated to the system has been increased dynamically, using AIX DLPAR Memory allocated to Oracle (SGA and PGA) has been increased on the fly
16 4/22/2008 2008 IBM Corporation
Agenda
Oracle Server Architecture Memory
AIX VMM tuning Oracle Memory Structure Allocation
IO
AIX LVM Striping AIX mount option Oracle ASM Asynchronous IO
Conclusion
17
4/22/2008
IO : Database Layout
Having a good Storage configuration is a key point : Because disk is the slowest part of an infrastructure Reconfiguration can be difficult and time consuming
Stripe and mirror everything (S.A.M.E) approach: Goal is to balance I/O activity across all disks, loops, adapters, etc... Avoid/Eliminate I/O hotspots
rec O om rac me le nd a ti o
Manual file-by-file data placement is time consuming, resource intensive and iterative Additional advices to implement SAME :
apply the SAME strategy to data, indexes if possible separate redologs (+archivelogs)
18
4/22/2008
HW Striping
With Enterprise class storage (with huge cache), RAID-5 performances are comparable to
RAID-10 one (for most customer workloads) Consider RAID-10 for workloads with a high percentage of random write activity (> 25%) and high I/O access densities (peak > 50%)
Use RAID-5 or RAID-10 to create striped LUNs If possible try to minimize the number of LUNs per RAID array to avoid contention on physical disk.
19
4/22/2008
Storage
HW Striping
1. 2. 3. 4.
Luns are striped across physical disks (stripe-size : ~ 64k, 128k, 256k) LUNs are seen as hdisk device on AIX server. Create AIX Volume Group(s) (VG) with LUNs from multiple arrays Logical Volume striped across hdisks (stripe-size : 8M, 16M, 32M, 64M) => each read/write access to the LV are well balanced accross LUNs and use the maximum number of physical disks for best performance.
20
LVM Striping
4/22/2008
Create Logical Volume with the striping option : mklv S <stripe-size> ...
Valid LV Stripe sizes: AIX 5.2: 4k, 8k, 16k, 32k, 64k, 128k, 256k, 512k, 1M 1. AIX 5.3+: AIX 5.2 Stripe sizes + 2M, 4M, 16M, 32M, 64M, 128M
Create a Volume Group with a 8M,16M or 32M PPsize. (PPsize will be the Stripe size) AIX 5.2 : Choose a Big VG and change the t factor : # mkvg B t <factor> s <PPsize>... AIX 5.3 +: Choose a Scalable Volume Group : # mkvg S s <PPsize> ... Create LV with Maximum range of physical volume option to spread PP on different hdisk in a Round Robin fashion : # mklv e x ...
jfs2 filesystem creation advice : If you create a jfs2 filesystem on a striped (or PP spreaded) LV, use the INLINE logging option. It will avoid hot spot by creating the log inside the filesystem (which is striped) instead of using a uniq PP stored on 1 hdisk. # crfs a logname=INLINE
21 4/22/2008 2008 IBM Corporation
Generic device definition bad performance In general, a generic device definition provides far from optimal performance since it doesnt properly customize the hdisk device : exemple : hdisk are created with a queue_depth=1 1. Contact your vendor or go to their web site to download the correct ODM definition for your storage subsystem. It will setup properly the hdisk accordingly to your hardware for optimal performance. If AIX is connected to the storage subsystem with several Fiber Channel Cards for performance, dont forget to install a multipath device driver or path control module.
- sdd or sddpcm for IBM DS6000/DS8000 - powerpath for EMC disk subsystem - hdlm for Hitachi etc....
22 4/22/2008 2008 IBM Corporation
2.
Agenda
Oracle Server Architecture Memory
AIX VMM tuning Oracle Memory Structure Allocation
IO
AIX LVM Striping AIX mount option Oracle ASM Asynchronous IO
Conclusion
23
4/22/2008
Direct IO (DIO) introduced in AIX 4.3. Data is transfered directly from the disk to the
application buffer, bypassing the file buffer cache hence avoiding double caching (filesystem cache + Oracle SGA). Emulates a raw-device implementation. To mount a filesystem in DIO $ mount o dio /data
Bench throughput over run duration higher tps indicates better performance.
4/22/2008
Restrictions :
1. Because data transfer is bypassing AIX buffer cache, jfs2 prefetching and write-behind cant be used. These functionnalities can be handled by Oracle. (Oracle parameter) db_file_multiblock_read_count = 8, 16, 32, ... , 128 according to workload 2. When using DIO/CIO, IO requests made by Oracle must by aligned with the jfs2 blocksize to avoid a demoted IO (Return to normal IO after a Direct IO Failure) => When you create a JFS2, use the mkfs o agblksize=XXX Option to adapt the FS blocksize with the application needs. Rule : IO request = n x agblksize Exemples: if DB blocksize > 4k ; then jfs2 agblksize=4096 Redolog are always written in 512B block; So jfs2 agblksize must be 512
25
4/22/2008
IO : Direct IO demoted
Extract from Oracle AWR (test made in Montpellier with Oracle 10g) Waits on redolog (with demoted IO, FS blk=4k) Waits% Time -outs Total Wait Time (s) 2,229,324 0.00 62,628 Avg wait (ms) 28 Waits /txn 1.53
Waits on redolog (without demoted IO, FS blk=512) Waits% 494,905 Time -outs Total Wait Time (s) 0.00 1,073 Avg wait (ms) 2 Waits /txn 1.00
How to detect demoted IO : Trace command to check demoted io : # trace aj 59B,59C ; sleep 2 ; trcstop ; trcrpt o directio.trcrpt # grep demoted directio.trcrpt
26
4/22/2008
Oracle Datafiles
Oracle Redolog
Oracle Archivelog
27
4/22/2008
Agenda
Oracle Server Architecture Memory
AIX VMM tuning Oracle Memory Structure Allocation
IO
AIX LVM Striping AIX mount option Oracle ASM Asynchronous IO
Conclusion
28
4/22/2008
Oracle
AIX
Storage
hdisk1
ASM DISKGROUP
ASM striping
29 4/22/2008 2008 IBM Corporation
Database on ASM
ASM Instance
ASM is implemented as a special kind of Oracle instance, with its own SGA and background processes (by default, less than 100M of SGA are allocated for the ASM instance) and generate very little activity except during rebalance operations. A single ASM instance can service one or more instance databases. Can maintain redundant copies of data An ASM-managed database will have approximately the same performance as a database implemented in raw-devices.
30 4/22/2008 2008 IBM Corporation
Agenda
Oracle Server Architecture Memory
AIX VMM tuning Oracle Memory Structure Allocation
IO
AIX LVM Striping AIX mount option Oracle ASM Asynchronous IO
Conclusion
31
4/22/2008
IO : Asynchronous IO (AIO)
Allows multiple requests to be sent without to have to wait until the disk subsystem has completed the physical IO. Utilization of asynchronous IO is strongly advised whatever the type of file-system and mount option implemented (JFS, JFS2, CIO, DIO).
Application
2
aio Q
aioservers
Disk
Posix vs Legacy Since AIX5L V5.3, two types of AIO are now available : Legacy and Posix. For the moment, the Oracle code is using the Legacy AIO servers.
32 4/22/2008 2008 IBM Corporation
Application
2
Raw Devices / ASM :
AIX Kernel
Disk
check AIO configuration with : lsattr El aio0 enable asynchronous IO fast_path. : AIX 5L : chdev -a fastpath=enable -l aio0 (default since AIX 5.3) AIX 6.1 : ioo p o aio_fastpath=1 (default setting)
34
4/22/2008
Advice : set this parameter to setall Note : set the disk_asynch_io parameter to true as well
35
4/22/2008
Agenda
Oracle Server Architecture Memory
AIX VMM tuning Oracle Memory Structure Allocation
IO
AIX LVM Striping AIX mount option Oracle ASM Asynchronous IO
Conclusion
36
4/22/2008
thread1
thread2
Register thread1
Register thread2
P6 Core
CPU: Micro-Partitioning
Micro-Partitioning give the possibility to share processors between several LPARs. It improve CPU utilisation by reducing the number of idle cycle. (available on POWER5 and POWER6 system running AIX 5.3+)
An example given based on a PeopleSoft/Oracle10g RAC benchmark (2,400 Virtual Users) ran at PSSC MOP (DLPAR performance vs. SPLPAR one)
38 4/22/2008 2008 IBM Corporation
Micro- Partition
Agenda
Oracle Server Architecture Memory
AIX VMM tuning Oracle Memory Structure Allocation
IO
AIX LVM Striping AIX mount option Oracle ASM Asynchronous IO
Conclusion
40
4/22/2008
OS monitoring
tuning is an iterative process a minimun of changes should be performed at each iteration tuning is a joint effort between the system administrator and the DBA dont hesitate to keep a log to track changes and the impact of these changes
41 4/22/2008 2008 IBM Corporation
vmstat useful for obtaining an overall picture of CPU, paging, and memory usage lvmstat useful to get logical volume I/O statistics iostat allows to get statistics on disk activity (and also on terminals, CPU, adapters) lparstat reports logical partition related information and statistics nmon complete tool which gives information on all the components of the system filemon uses the trace facility to report on the I/O activity of physical volumes, logical volumes, individual files, and the Virtual Memory Manager xmperf allows to define monitoring environments to supervise the performance of the local and remote systems netstat allows to monitor network activity ....
42
4/22/2008
43
4/22/2008
CPU
topas, nmon vmstat (us, sys, id, r) sar P ALL CPU bound? runqueue? workload ? parallelization of workload?
MEMORY topas, nmon vmstat (pi, po) pagingout activity? memory allocated to Oracle? other applications running? other Oracle instances?
IO storage monitoring tool vmstat (wa) iostat, nmon, filemon accesses well spread? hot spots? jfs mount options? AIO? Network
Metalink note 282036.1 alert.log bdump,cdump,udump init.ora/spfile AWR Grid Control strategy of analyze? memory allocated to instances (SGA, PGA) tkprof (parsing, explain plans, bind variables)
44
4/22/2008
Agenda
Oracle Server Architecture Memory
AIX VMM tuning Oracle Memory Structure Allocation
IO
AIX LVM Striping AIX mount option Oracle ASM Asynchronous IO
Conclusion
45
4/22/2008
Snapshots (set of statistics gathered at a single time) are taken automatically each 60 minutes or can be run manually: execute dbms_workload_repository.create_snapshot();
AWR reports are generated by running $ORACLE_HOME/rdbms/admin/awrrpt.sql and providing 2 snapshot Ids. Can be also generated using the Grid Control.
To enable most of the new statistical gathering and advisors, ensure that the parameter STATISTICS_LEVEL is set to TYPICAL (recommended).
46 4/22/2008 2008 IBM Corporation
Compare reports to mesure the impact of tuning and to check the behavior of the database Use Metalink and/or the Oracle documentation to get the definition of events, statistics etc
47
4/22/2008
SQL Tuning Advisor (STA) Detects missing or stale statistics, missing indexes Suggests new execution plan which can be applied to the system and used without any change in the application Advised to use STA through Grid Control but also possible to invoke it using DBMS_SQLTUNE package.
48 4/22/2008 2008 IBM Corporation
49
4/22/2008
This Top 5 Timed Events table is very important it lists the events to focus on and will lead the performance analysis. According the event listed in the table, refer to the corresponding section in the report.
50
4/22/2008
Sections SQL statement ordered by present SQL statements ordered by differents criterias. They are useful to detect a problem of parsing, a response time of a particular statement etc SQL is clickable and the whole statement is presented later in the report.
51
4/22/2008
Buffer Pool Advisory help to tune the buffer pool, giving estimation of the impact of changes of the size of the buffer pool. Advisors exist for: - PGA memory - Shared Pool - Sga_target -
52
4/22/2008
Targets : databases
53
4/22/2008
54
4/22/2008
Performance graphs
55
4/22/2008
Advisor Central
56
4/22/2008
60
4/22/2008
61
4/22/2008
Grid is the most powerful tool to identify issues, get advices, tune
your database
It facilitates DB administration tasks and allows to perform the maintenance of the database (backups etc)
Targets monitored by the Grid are the hosts, databases, Oracle Applications Servers, Oracle eBusiness Suite etc
62
4/22/2008
Agenda
Oracle Server Architecture Memory
AIX VMM tuning Oracle Memory Structure Allocation
IO
AIX LVM Striping AIX mount option Oracle ASM Asynchronous IO
Conclusion
63
4/22/2008
Conclusion
Implement and get the best of Oracle on AIX is the result of a joint effort between the OS administrator and the Oracle DBA. The tuning process is made of a first guess parameter setting and then, iterations to change progressively the configuration to get the best results Prefer JFS2 + Concurrent IO or ASM. Always use Asynchronous IO servers and, if available, use FastPath. Implement SMT if available on your system. Use AIX, Oracle and storage monitoring tools
(AIX : Nmon, Topas Oracle : AWR & Grid Control Storage : TPC)
oraclibm@fr.ibm.com
64
4/22/2008
65
4/22/2008
Addendum - Certification matrix : Oracle database Enterprise Edition on IBM AIX based Systems
10th of april 2008
When upgrading to Oracle 10GR2 on AIX 6.1, download and run an updated version of rootpre script before proceeding with the Oracle Database install (refer to bug 6613550 for more details).
66
4/22/2008
67
4/22/2008
Thanks to the authors of the following documents : Improving Database Performance with AIX Concurrent I/O S. Kashyap, B. Olszewski, R Hendrickson Performance Management Guide AIX5L version 5.3 documentation Oracle 9i & 10g on IBM AIX5L : Tips & Considerations IBM/Oracle ICC Oracle Architecture and Performance Tuning on AIX R. Ballough Configuring IBM TotalStorage for Oracle OLTP Applications D. Martin Tuning IBM AIX5L for an Oracle Database D. Massanari, M. Riviere
68
4/22/2008
oraclibm@fr.ibm.com
Tel : +33 4 67 34 67 49
69
4/22/2008
Trademarks
The following are trademarks of the International Business Machines Corporation in the United States, other countries, or both.
Not all common law marks used by IBM are listed on this page. Failure of a mark to appear does not mean that IBM does not use the mark nor does it mean that the product is not actively marketed or is not significant within its relevant market. Those trademarks followed by are registered trademarks of IBM in the United States; all others are trademarks or common law marks of IBM in the United States.
70
4/22/2008