IBM Oracle Solutions Advanced Technical Support - Americas

Oracle on AIX – Configuration & Tuning

R. Ballough, IBM Advanced Technical Support

Oracle on AIX Workshop

© 2006 IBM Corporation

IBM Oracle Solutions - Advanced Technical Support - Americas

Legal information
The information in this presentation is provided by IBM on an "AS IS" basis without any warranty, guarantee or assurance of any kind. IBM also does not provide any warranty, guarantee or assurance that the information in this paper is free from errors or omissions. Information is believed to be accurate as of the date of publication. You should check with the appropriate vendor to obtain current product information. Any proposed use of claims in this presentation outside of the United States must be reviewed by local IBM country counsel prior to such use. IBM,^ , and pSeries are trademarks or registered trademarks of the International Business Machines Corporation. Oracle and Oracle9i are trademarks or registered trademarks of Oracle Corporation. All other products or company names are used for identification purposes only, and may be trademarks of their respective owners.

2

Oracle for AIX Workshop

© 2006 IBM Corporation

IBM Oracle Solutions - Advanced Technical Support - Americas

Agenda
Basic AIX Configuration/Tuning for Oracle
– Memory – CPU – I/O – Network – Miscellaneous

RAC Configuration RAC Tuning

3

Oracle for AIX Workshop

© 2006 IBM Corporation

IBM Oracle Solutions - Advanced Technical Support - Americas

AIX Configuration for Oracle “starting points”
The suggestions presented here are considered to be basic configuration “starting points” for general Oracle workloads Customer workloads will vary Ongoing performance monitoring and tuning is recommended to ensure that the configuration is optimal for the particular workload characteristics

4

Oracle for AIX Workshop

© 2006 IBM Corporation

IBM Oracle Solutions - Advanced Technical Support - Americas

Oracle Server Architecture – Memory Structures
PMON SMON
Archive Logs

Program Global Area (PGA)

System Global Area (SGA)

ARC0 Redo Log Buffer LGWR

Private SQL Area

Shared Pool

Database Buffer Cache

DBWR User

CHKP
Control Files

Online Redo Logs

D000

Database Files

5

Oracle for AIX Workshop

© 2006 IBM Corporation

Virtual Memory Manager (VMM) The AIX “vmo” command provides for the display and/or update of several parameters which influence the way AIX manages physical memory – The “-a” option displays current parameter settings vmo –a – The “-o” option is used to change parameter values vmo –o minfree=1440 – The “-p” option is used to make changes persist across a reboot vmo –p –o minfree=1440 A number of the default “vmo” settings are not optimized for database workloads and should be modified for Oracle environments 6 © 2005 IBM Corporation © 2003 IBM Corporation .Americas Memory Tools .IBM Advanced Technical Support .

Americas General Memory Tuning Memory Use testsys 8/2/2005 %comp 100 90 80 70 60 50 40 30 20 10 0 8:03 8:04 8:05 8:06 8:07 8:08 8:09 8:10 8:11 8:12 8:13 8:14 8:15 8:16 8:17 8:18 8:19 8:20 8:21 8:22 8:23 8:24 8:25 8:26 8:27 8:28 8:29 8:30 8:31 8:32 8:33 8:34 8:35 %file 7 Oracle for AIX Workshop © 2006 IBM Corporation .Advanced Technical Support .IBM Oracle Solutions .

IBM Oracle Solutions .3 To start.max % real memory for fs buffer cache 40-60%: JFS or JFS2 filesystems without DIO or CIO <= 20%: Raw logical volumes JFS or JFS2 with DIO or CIO GPFS Never more than 20 GB prior to AIX 5. MAXCLIENT% .Americas VMM Tuning Pre AIX 5.Advanced Technical Support . set to vmtune "numperm" value Reduce until vmstat freed (fr) to scanned (sr) ratio is 4:1 8 Oracle for AIX Workshop © 2006 IBM Corporation .2 ML4 MINPERM% – minimum % real memory for fs buffer cache 15-20%: JFS or JFS2 filesystems without DIO or CIO 5%: RAW logical volumes JFS or JFS2 with DIO or CIO GPFS MAXPERM%.

Americas VMM Tuning – AIX 5. Default value of “0” means no preemption. STRICT_MAXPERM=0 (default) STRICT_MAXCLIENT=1 (default) 9 Oracle for AIX Workshop © 2006 IBM Corporation .IBM Oracle Solutions . MAXCLIENT%=80% or higher make this a threshold which is > (1-computational memory) LRU_FILE_REPAGE=0 LRU_POLL_INTERVAL=10ms LRU_FILE_REPAGE=0 is a “hint” to lrud to ignore repage rates when determining what to page out – effectively favoring paging out file pages (filesystem buffer cache) rather than computational pages LRU_POLL_INTERVAL indicates the time period after which LRUD pauses and interrupts can be serviced.Advanced Technical Support .2ML4+ MINPERM% =5% MAXPERM%.

IBM Oracle Solutions .Americas Understanding Memory Pools Memory pools are configured at boot time based on physical hardware configuration Number can be seen with ‘vmstat –v |grep pools’ Size can only be seen using KDB LRUD operates per memory pool SGA is allocated equally from memory pools If free list has been depleted in a memory pool.3 ML3 with APAR IY69237 modifies VMM to consider all free frames in system regardless of memory pool before reclaiming memory 10 Oracle for AIX Workshop © 2006 IBM Corporation . LRUD will scan/reclaim memory in that pool Consider implementing one of the following: Setting memory_affinity=0 ignores physical hardware configuration and allocates evenly sized memory pools AIX 5.Advanced Technical Support .

8) x 6 x 2) vmo –o minfree=1440 –o maxfree=1536 -p 11 Oracle for AIX Workshop © 2006 IBM Corporation .Americas VMM Page Stealing Thresholds The following define thresholds for the VMM page stealing process (lrud): minfree Set minfree = 120 x # logical CPUs /#mem pools Consider increasing if vmstat “fre” column frequently approaches zero or if “vmstat –s” shows significant “free frame waits” maxfree Set maxfree = minfree + (MAX(maxpgahead. maxpgahead=8 and j2_maxPageReadAhead=8: minfree = 1440 = 120 x 6 x 2 maxfree = 1536 = 1440 + (max(8. j2_maxPageReadAhead) * # logical CPUs)/ # mem pools Example: For a 6-way LPAR with SMT enabled.IBM Oracle Solutions .Advanced Technical Support .

MAXCLIENT) Reduce Oracle SGA or PGA (9i or later) size Add physical memory 12 Oracle for AIX Workshop © 2006 IBM Corporation .Advanced Technical Support .IBM Oracle Solutions . paging space does not have to be large ½ memory + 4GB Monitor paging activity: vmstat -s sar -r nmon Resolve paging issues: Reduce Filesystem cache size (MAXPERM. but IS NOT RECOMMENDED – we have better ways of keeping sga resident in memory Do not over commit real memory!!!! Server should be configured with enough physical memory to satisfy memory requirements Paging space With AIX demand paging.Americas General AIX memory tuning No requirements for allocating SGA SGA may be pinned on AIX.

etc. ( DB_CACHE_SIZE.. SGA_TARGET (10g) – PGA_AGGREGATE_TARGET can be dynamically resized – SGA_TARGET and PGA_AGGREGATE_TARGET are not hard limits Something to keep in mind for use of DLPAR with memory…. 13 © 2005 IBM Corporation © 2003 IBM Corporation . • • • SQL> alter system set db_cache_size=2048m scope=both.. SHARED_POOL_SIZE.Americas Changing SGA size Memory Oracle 9i or 10g – SGA can be dynamically resized. but has an upper bound by the parameter SGA_MAX_SIZE.IBM Advanced Technical Support .

241.928.172.448 2.944 768.IBM Advanced Technical Support .960 SQLPLUS: SQL> show sga SQL> show parameters 14 © 2005 IBM Corporation © 2003 IBM Corporation .210.371.984 ---------------- sum 18.584 1.864.Americas Determining SGA size Statspack: SGA Memory Summary for DB: test01 Instance: test01 Snaps: 1046 -1047 SGA regions -----------------------------Database Buffers Fixed Size Redo Buffers Variable Size Size in Bytes ---------------16.513.

IBM Oracle Solutions .Advanced Technical Support .Americas Agenda Basic AIX Configuration/Tuning for Oracle – Memory – CPU – I/O – Network – Miscellaneous 15 Oracle for AIX Workshop © 2006 IBM Corporation .

query level .PARALLEL_MAX_SERVERS or AUTOMATIC_PARALLEL_TUNING (CPU_COUNT * PARALLEL_THREADS_PER_CPU) – CPU_COUNT – FAST_START_PARALLEL_ROLLBACK – should be using UNDO instead – CBO – execution plan may be affected.IBM Advanced Technical Support . check explain plan 16 © 2005 IBM Corporation © 2003 IBM Corporation .Americas Oracle parameters based on # CPUs – DB_WRITER_PROCESSES – Degree of Parallelism – – – .table level .

Virtual cpus)<= physical processors in shared pool .Virtual CPUs should be the nearest integer >= capping limit .Americas CPU Considerations Use SMT with AIX 5.IBM Advanced Technical Support .CAPPED .UNCAPPED Virtual CPUS should be set to the max peak demand requirement DLPAR considerations Oracle 9i – Oracle CPU count does not recognize change in # cpus – AIX scheduler can still use the added CPUs Oracle 10g – Oracle CPU count recognizes change in # cpus 17 © 2005 IBM Corporation © 2003 IBM Corporation .3/Power5 environments Micropartitioning considerations .

Advanced Technical Support .IBM Oracle Solutions .Americas Agenda Basic AIX Configuration/Tuning for Oracle – Memory – CPU – I/O – Network – Miscellaneous 18 Oracle for AIX Workshop © 2006 IBM Corporation .

Files PMON SMON Archive Logs Program Global Area (PGA) System Global Area (SGA) ARC0 Redo Log Buffer LGWR Private SQL Area Shared Pool Database Buffer Cache DBWR User CHKP Control Files Online Redo Logs D000 Database Files 19 Oracle for AIX Workshop © 2006 IBM Corporation .Americas Oracle Server Architecture .IBM Oracle Solutions .Advanced Technical Support .

Americas Flashback database design 20 Oracle for AIX Workshop © 2006 IBM Corporation .IBM Oracle Solutions .Advanced Technical Support .

Americas Options for storing Oracle data files Filesystems – Single-instance: – JFS. Veritas CFS Raw Automatic Storage Management (ASM) – new in 10g 21 © 2005 IBM Corporation © 2003 IBM Corporation . Veritas VxFS – Clustered: – GPFS.IBM Advanced Technical Support . JFS2.

striping on the front end as well for maximum distribution Physical Partition Spreading (mklv –e x) –orLarge Grained LVM striping (>= 1MB stripe size) http://www-1..Americas Data Layout for Optimal I/O Performance Stripe and mirror everything (SAME) approach: Goal is to balance I/O activity across all disks.com/support/techdocs/atsmastr. loops.ibm. etc. Avoid/Eliminate I/O hotspots Manual file-by-file data placement is time consuming. resource intensive and iterative Use RAID-5 or RAID-10 to create striped LUNs (hdisks) Create AIX Volume Group(s) (VG) w/ LUNs from multiple arrays.nsf/WebIndex/WP100319 22 © 2005 IBM Corporation © 2003 IBM Corporation .IBM Advanced Technical Support .. adapters.

16 MB. 8k. or use “mklv –T O” with Big VGs • Requires AIX APAR IY36656 and Oracle patch (bug 2620053) PP Striping – Use minimum Physical Partition (PP) size (mklv -t.2 Strip sizes + 2M. 64k. 64M.IBM Advanced Technical Support . 32M.Americas Data Layout cont’d… Stripe using Logical Volume (LV) or Physical Partition (PP) striping LV Striping – Oracle recommends stripe width of a multiple of Db_block_size * db_file_multiblock_read_count Usually around 1 MB – Valid LV Strip sizes: AIX 5. 128M – Use AIX Logical Volume 0 offset (9i Release 2 or later) • Use Scalable Volume Groups (VGs). 16k. 1 MB AIX 5. -s parms) Spread AIX Logical Volume (LV) PPs across multiple hdisks in VG • (mklv –e x) 23 © 2005 IBM Corporation © 2003 IBM Corporation . 32k. 4M. 256k.3: AIX 5. 512k. 128k.2: 4k.

DIO and CIO must be specified at the filesystem level In 10g.Advanced Technical Support .Americas Single Instance Environments . Oracle issues o_cio and o_dio calls as appropriate 24 Oracle for AIX Workshop © 2006 IBM Corporation . memory pages released after pages written to disk In 9i.Filesystems Filesystems JFS – no longer being enhanced Better for lots of small file creates & deletes JFS2 – generally the preferred single-instance filesystem Better for large files/filesystems Mount options: Buffer Caching (default)– stage data in fs buffer cache Direct I/O (DIO)– no caching on reads Concurrent I/O (CIO) – DIO + no write lock (JFS2 only) Release Behind Read (RBR)– During sequential reads.IBM Oracle Solutions . memory pages released after pages copied to internal buffers Release Behind Write (RBW) – During sequential writes.

set filesystemio_options=SETALL and use dio or cio mount option In 10g.1 25 Oracle for AIX Workshop © 2006 IBM Corporation .Advanced Technical Support .IBM Oracle Solutions . set filesystemio_options=SETALL When using DIO/CIO. To enable caching for JFS/JFS2: Use default filesystem mount options Set Oracle filesystemio_options=ASYNCH DIO tends to benefit heavily random access workloads and CIO tends to benefit heavy update workloads. To disable JFS.Americas Single-instance environments Cached vs. JFS2 caching: In 9i. Consider the following db changes: Increase db_cache_size Increase db_file_multiblock_read_count Read Metalink Note #272520. non-Cached (Direct) I/O File System caching tends to benefit heavily sequential workloads with low write content. fs buffer cache isn’t used.

use a filesystem block size of 4096.Oracle Database File Access Data Base Files (DBF) I/O size is db_block_size or db_block_size * db_file_multiblock_read_count Use CIO or no mount options for extremely sequential I/O If block size is >=4096.Americas Single-instance environments . else use 2048 Redo Log/Control Files I/O size is always a multiple of 512 bytes Use CIO or DIO and set filesystem block size to 512 Archive Log Files Do not use CIO or DIO ‘rbrw’ mount option can be advantageous Oracle Binaries Do not use CIO or DIO 26 © 2005 IBM Corporation © 2003 IBM Corporation .IBM Advanced Technical Support .

I/O Tuning (ioo) READ-AHEAD (Only applicable to JFS/JFS2 with caching enabled) MINPGAHEAD (JFS) or j2_minPageReadAhead (JFS2) Default: 2 Starting value: MAX(2.Advanced Technical Support .IBM Oracle Solutions .Americas Single-instance environments . Starting Value: 568 Monitor with “vmstat –v”.DB_BLOCK_SIZE / 4096) MAXPGAHEAD (JFS) or j2_maxPageReadAhead (JFS2) Default: 8 (JFS). increase if value of “external pager filesystem I/Os blocked with no fsbuf “is increasing 27 Oracle for AIX Workshop © 2006 IBM Corporation . 128 (JFS2) Set equal to (or multiple of) size of largest Oracle I/O request DB_BLOCK_SIZE * DB_FILE_MULTI_BLOCK_READ_COUNT Number of buffer structures per filesystem: NUMFSBUFS (JFS2): Default: 196. increase if value of “filesystem I/Os blocked with no fsbuf” is increasing j2_dynamicBufferPreallocation Default: 16 Monitor with “vmstat –v”.

maxreqs=16384 Oracle parameters disk_asynch_io = TRUE filesystemio_options = {ASYNCH | SETALL} db_writer_processes = default Monitor usage: Watch alert.log for errors: Warning “lio_listo returned EAGAIN” Monitor from AIX “pstat –a | grep aios” Use “-A” option for NMON Iostat –Aq (new in AIX 5.Americas Asynchronous I/O for filesystem environments AIX parameters minservers = 10 * # cpus maxservers = (10 * # disks) / # cpus maxreqs = a multiple of 4096 > 4 * #disks * queue_depth “enable” at system restart Typical settings: minservers=100.Advanced Technical Support .3) 28 Oracle for AIX Workshop © 2006 IBM Corporation .IBM Oracle Solutions . maxservers=200.

but default values may be used ASM instance parameters – ASM_POWER_LIMIT=1 Makes ASM rebalancing a low-priority operation.Advanced Technical Support .Americas ASM configurations AIX parameters – Async I/O needs to be enabled.IBM Oracle Solutions . It is common to set this value to 0. then increase to a higher value during maintenance windows PROCESSES=25+ 15n. where n=# of instances using ASM – DB instance parameters – – – – – disk_asynch_io=TRUE filesystemio_options=ASYNCH Increase Processes by 16 Increase Large_Pool by 600k Increase Shared_Pool by [(1M per 100GB of usable space) + 2M] 29 Oracle for AIX Workshop © 2006 IBM Corporation . May be changed dynamically.

Advanced Technical Support .Americas Agenda Basic AIX Configuration/Tuning for Oracle – Memory – CPU – I/O – Network – Miscellaneous RAC Configuration RAC Tuning 30 Oracle for AIX Workshop © 2006 IBM Corporation .IBM Oracle Solutions .

Americas Network Parameters – all environments Set sb_max >= 1 MB (1048576) (generally ok by default) Set tcp_sendspace = 262144 Set tcp_recvspace = 262144 Set rfc1323=1 Also confirm these are set properly at network interface level 31 Oracle for AIX Workshop © 2006 IBM Corporation .Advanced Technical Support .IBM Oracle Solutions .

Advanced Technical Support .Americas Miscellaneous parameters /etc/security/limits Set to “-1” for everything except core for Oracle user Sys0 attribute maxuproc >= 4096 Environment variables: AIXTHREAD_SCOPE=S NUM_SPAREVP=1 (AIX 5.1 only) Use a 64-bit kernel 32 Oracle for AIX Workshop © 2006 IBM Corporation .IBM Oracle Solutions .

IBM Oracle Solutions .Advanced Technical Support .Americas Agenda Basic AIX Configuration/Tuning for Oracle – Memory – CPU – I/O – Network – Miscellaneous RAC Configuration/Tuning 33 Oracle for AIX Workshop © 2006 IBM Corporation .

Americas RAC Architecture Summary Centralized Management Console Users Low Latency Interconnect VIA or Proprietary Full Cache Fusion Cache-to-cache data shipping Shared cache eliminates slow I/O Enhanced IPC High Speed Switch or Interconnect Clustered Database Servers Cache Fusion Shared Cache Hub or Switch Fabric Allows flexible and transparent deployment Storage Area Network Shared Disk Subsystem Drive and Exploit Industry Advances in Clustering Oracle for AIX Workshop 34 © 2006 IBM Corporation .Advanced Technical Support .IBM Oracle Solutions .

Advanced Technical Support .Americas Logical Shared Disk Architecture N e tw o r k In te r c o n n e c t N ode 1 N ode 2 N ode 3 Shared D is k Each RAC node requires access to all shared disk: Physical Shared disk typically Fiber Channel or SAN attached Best Practices: 1) Use 2 or more HBAs using multipathing software for load balancing and path failover 2) plan for fabric redundancy 35 Oracle for AIX Workshop © 2006 IBM Corporation .IBM Oracle Solutions .

ibm.com) Other Notes: ORACLE_HOME directories must be local unless using GPFS or SFRAC. OS level backups. Required for Standard Edition RAC VERITAS SFRAC (http://support.veritas.Americas Storage Software Options for Oracle RAC: GPFS (http://publib.01+ 9i: GPFS 2. 5.Advanced Technical Support .3.2. 2.3.2. scripts to manage disk space. ORACLE_HOME 36 Oracle for AIX Workshop Information subject to change. 5. etc. © 2006 IBM Corporation .4. 10g REQUIRES separate LUNs for Voting Disk and OCR partitions unless a clustered filesystem or HACMP is used Use separate directories for CRS.1+ Link above includes references to supported storage platforms HACMP Raw Logical Volumes 10g: 5.1.3 Automatic Storage Management (10g only) Check with Storage Vendor for support/recommended configuration Check with Internal operations groups for management implications eg.1.html) See metalink note 302806.0. 5.x.3+ 10gR1: GPFS 2.2. and additional requirements may apply. 2. 2.3. 4.5.1.boulder. ASM.3 not certified with 10g 9i: 4.com/clresctr/library/gpfs_aix_faq.1.2 – NOTE: HACMP 5.IBM Oracle Solutions .0. 5. 2.1 for details 10gR2: GPFS 2.

Americas Oracle Options for Data Storage RAW Data Base Files Redo Log Files Control Files Archive Log Files Oracle Binaries OCR Voting Disk GPFS ASM 37 Oracle for AIX Workshop © 2006 IBM Corporation .IBM Oracle Solutions .Advanced Technical Support .

com Check note # 282036.Advanced Technical Support . but cannot function as clusterware Latest Requirements can be found at http://metalink.5.3 recommended Veritas SFRAC 4.0 Other: PSSP 3.5 is required for use of SP switch and can replace HACMP as clusterware Oracle 10g RAC: Oracle CRS provides basic clusterware capability HACMP required for raw logical volume environments only .5.IBM Oracle Solutions .Americas Clustering Software Requirements for RAC on pSeries: Oracle 9i RAC: HACMP .1 for detail 38 Oracle for AIX Workshop © 2006 IBM Corporation .oracle.2 currently most recent certified version Veritas SFRAC may be used if desired Other: PSSP 3.5 is required for use of SP switch.

Advanced Technical Support .IBM Oracle Solutions .Americas Oracle 10g R2: Clusterware Provides same base functions as 10g R1 CRS Extended to provide protection for 3rd Party Application components – Employs Application Virtual IPs for transparent network connectivity – May require Clustered File System support for Application Configuration Files – Applications do not necessarily have to reside on DB node 39 Oracle for AIX Workshop © 2006 IBM Corporation .

IBM Oracle Solutions .1 bug 10g requires one virtual IP address for client traffic for each node Validate that the application uses OCI calls and can take advantage of TAF JDBC Thick does not support use of OCI TAF Supports failure/reconnect of failed connections Optional select statement failover OR Hardware load balancer/sprayer Software load balancer DNS CNAMES 40 Oracle for AIX Workshop © 2006 IBM Corporation .Americas Public Interface Requirements for RAC on pSeries: Public interface At least one network interface for client traffic May use virtual ethernet if sufficient bandwidth – watch for 10.Advanced Technical Support .0.2.

2.0.Advanced Technical Support .Americas RAC Interconnect Configuration P u b lic N e tw o rk P riva te N e tw o rk P riva te N e tw o rk (P rim a ry) (B a c k u p ) RA C In t er c n on ec t Node 1 Node 2 Node 3 Gigabit Ethernet satisfactory for most applications IBM HPS offers lower latency/higher bandwidth IP over InfiniBand 10.3ad Link Aggregation AIX options replace Oracle FTP-IP code May also be used with 9i CLUSTER_INTERCONNECTS parameter 41 Oracle for AIX Workshop © 2006 IBM Corporation .IBM Oracle Solutions .1+ UDP network tuning required for optimum performance Switched Networks REQUIRED – crossover cables not supported Virtual Ethernet not yet supported Oracle9i: Oracle “Fault Tolerant IP” (FTP-IP) feature provides Interconnect Network Fault Tolerance Oracle10g: Etherchannel or 802.

can be 10/100 ethernet VERITAS SFRAC: cluster interconnect: recommended 3 Ethernet interfaces. recommend 2 gigabit Ethernet GPFS solutions: cluster interconnect: same as above additional: GPFS traffic can share with the cluster interconnect! ASM: cluster interconnect: same as above 42 Oracle for AIX Workshop © 2006 IBM Corporation .Advanced Technical Support . recommend 2 gigabit Ethernet HACMP/GPFS solutions: cluster interconnect: same as above additional: private subnet required for GPFS traffic.IBM Oracle Solutions .Americas Private Interconnect Requirements for RAC on pSeries: Private interconnect Oracle 9i RAC: HACMP/RAW solutions: cluster interconnect: minimum of 1. minimum 2 Oracle 10g RAC: HACMP/RAW solutions cluster interconnect: minimum of 1.

Advanced Technical Support . Oracle searches HACMP Network Topology to identify up to 3 eligible networks for Interconnect use – Normally.IBM Oracle Solutions . plus 1 add’l public one Networks are operated in High Availability “failover” (primary/backup/backup) mode via Oracle provided “Fault Tolerant IP” code Oracle “Fault Tolerant IP” code registers DB connections with HACMP Event Manager (haemd) to assist with network failover 43 Oracle for AIX Workshop © 2006 IBM Corporation . the 2 private networks.Americas RAC Interconnect: 9i R2 (Default Configuration) Private Networks (normally 2) defined to HACMP At instance startup.

0.Americas RAC Interconnect: 9i R2 (Default Configuration) Node 1 Oracle FT-IP AIX/HACMP Node 2 Oracle FT-IP AIX/HACMP en1 en0 en3 en2 10.4 44 © 2006 IBM Corporation © 2003 IBM Corporation .2 en1 en0 en3 en2 • Fault Tolerant • Primary/Backup • No Bandwidth Aggregation Private Switch1 Switch2 Public 10.0.0.3 10.0.0.1 10.IBM Advanced Technical Support .0.0.0.

3ad Link Aggregation recommended Network interface names must be the same on all nodes – eg. 45 Oracle for AIX Workshop © 2006 IBM Corporation . it must be a cluster interconnect on all nodes.IBM Oracle Solutions .Advanced Technical Support .Americas 10g RAC Interconnect Configuration P u b lic N e tw o rk P riva te N e tw o rk P riva te N e tw o rk (P rim a ry) (B a c k u p ) RA C In t er c n on ec t Node 1 Node 2 Node 3 Best practice: Use 2 Gigabit Ethernet interfaces for interconnect Oracle9i “Fault Tolerant IP” feature not available in 10g Etherchannel and 802. if en0 is a cluster interconnect.

Advanced Technical Support .g.3ad Link Aggregation Requires protocol capable switches Supports bandwidth aggregation and load balancing 46 Oracle for AIX Workshop © 2006 IBM Corporation .IBM Oracle Solutions .g. en0) MUST be the same on all nodes – Private interface names (e. en6) SHOULD be the same on all nodes – Private networks should be non-routable Fault Tolerance provided by either: – Logical Network with Primary/Backup adapters (9iR2 Alternative 2) – EtherChannel or 802.Americas RAC Interconnect Configuration: 10g HACMP Network Topology no longer used Oracle “Fault Tolerant IP” code has been removed Public and Private (RAC Interconnect) network names are specified at cluster configuration time – Public interface names (e.

IBM Advanced Technical Support .Americas RAC Interconnect: 10g (Bandwidth Aggregation/Backup) Node 1 Oracle Node 2 Oracle 10.0.5 en6 en1 en0 AIX/TCP-IP Fault Tolerant Load Balanced or Active/Standby en6 en1 en0 AIX/TCP-IP 10.0.0.6 en3 en2 en4 en3 en2 en4 • Fault Tolerant • Load Balanced • Bandwidth Aggregation Private Switch1 Switch2 Public 47 © 2006 IBM Corporation © 2003 IBM Corporation .0.

IBM Advanced Technical Support .Americas Additional Network Parameters for RAC: Set udp_sendspace = db_block_size * db_file_multiblock_read_count +4k • (not less than 65536) Set udp_recvspace = 10 * udp_sendspace – Must be < sb_max – Increase if buffer overflows occur Ipqmaxlen=512 for GPFS environments Use Jumbo Frames if supported at the switch layer Time synchronization – use the “-x” flag with xntpd Examples: no -a |grep udp_sendspace no –o -p udp_sendspace=65536 netstat -s |grep "socket buffer overflows" 48 © 2006 IBM Corporation © 2003 IBM Corporation .

Americas GPFS tunables See metalink note 302806. most will use 512KB-1MB Pagepool – GPFS fs buffer cache.1 Async I/O: Oracle parameter filesystemio_options is ignored Set Oracle parameter disk_asynch_io=TRUE Worker1threads = GPFS asynch I/O Prefetchthreads= exactly what the name says Worker1threads +prefetchThreads<=550 Usually set prefetchthreads=default (64) and worker1threads=550prefetchthreads Set aio maxrequests=(worker1threads/#cpus) + 10 Other settings: GPFS block size is configurable.IBM Oracle Solutions . should be set to approximate capacity of I/O subsystem Ipqmaxlen=512 No –r –o ipqmaxlen=512 49 Oracle for AIX Workshop © 2006 IBM Corporation .Advanced Technical Support . Default=150MBpS. Default=64M mmchconfig pagepool=100M MaxMBpS = maximum I/O that GPFS can submit per second. not used for RAC but may be for binaries.

it can be desirable to increase the pagerpool parameter OCR. use separate filesystems for binaries than Oracle data files Use local filesystems for CRS HOME When I/O to GPFS filesystems outside of Oracle instance I/O is expected.Americas GPFS & Oracle – best practices Have < 10 filesystems Because of different usage patterns. vote placement is recommended on raw luns.IBM Advanced Technical Support . rather than as GPFS files 50 © 2006 IBM Corporation © 2003 IBM Corporation .

Americas Scalability: OLTP Environments Scale-out tends to be good for OLTP environments when: – There is low to moderate update activity – The workload is relatively uniform and predictable – The application is well designed and there are minimal lock/latch or serialization related contention issues – A functional partitioning strategy is used to direct users to a limited number of nodes –However. Oracle E-Business Suite. non-RAC result on same hardware No demonstrable $/tpmC advantage vs. IBM p595 non-RAC results – Limited number of SAP. PeopleSoft. other results Most customer RAC environments are 2 or 3 nodes A deployment involving a small number (2 or 3) of large nodes carries significantly less business risk than one involving many (4+) small nodes 51 © 2006 IBM Corporation © 2003 IBM Corporation . functional partitioning may reduce load balancing effectiveness Industry benchmark proof points are limited (as of 11/1/05): – 1 out of 178 TPC-C benchmarks used RAC (16-node HP Integrity rx5670) Relatively poor scale-out vs.IBM Advanced Technical Support .

IBM Advanced Technical Support . partitioned tables which have even key/data distribution – Data has a low “locality of reference” (low buffer cache hit%) – Are used predominantly for large query with Parallel Query Option (PQO) and PARALLEL_AUTOMATIC_TUNING=TRUE – Query response time is not critical Oracle focus on RAC based TPC-H benchmarks (as of 11/1/05): – 8 of 17 clustered results (all DB) were with Oracle RAC – 8 of 23 Oracle DB results were with Oracle RAC Small number of known Oracle accounts > 1TB and/or involving more than 6 nodes If you want to scale-out beyond 4-nodes. plan to do comprehensive stress testing before production deployment 52 © 2006 IBM Corporation © 2003 IBM Corporation .Americas Scalability: DSS Environments Scale-out tends to be good for Decision Support environments when: – There is minimal data update or DDL operations during peak shift – Data is predominantly in large.

ibm.com/documentation/index.oracle.com/products/database/clustering/RACWhitepapers.Advanced Technical Support .oracle.com – 282036.IBM Oracle Solutions .1 IBM Redbooks: (www.Americas Oracle Documentation Oracle Reference Manuals: http://otn.ibm.pdf Oracle Metalink: http://metalink.com/technology/products/database/asm/pdf/asm_10gr2_bptw p_sept05.oracle.com/redbooks) Techdocs – Technical Sales Library: http://www.com/support/metalink/index.html Oracle Database 10g Release 2 Automatic Storage Management Overview and Technical Best Practices http://www.com/support/techdocs 53 Oracle for AIX Workshop © 2006 IBM Corporation .html Oracle Whitepapers: http://otn.oracle.html Oracle (Metalink) Certification Info: http://otn.oracle.

ibm.ibm.com – 302806.1: IBM General Parallel File System (GPFS) and Oracle RAC on AIX 5L and IBM eServer pSeries 54 Oracle for AIX Workshop © 2006 IBM Corporation .html GPFS for AIX Library: www.com/servers/eserver/clusters/library/ – GPFS for AIX Frequently Asked Questions Oracle Metalink: http://metalink.com/servers/eserver/pseries/software/sp/gpfs.Americas GPFS Documentation GPFS for AIX home page: www.ibm.oracle.html – Concepts.com/servers/eserver/pseries/library/gpfs.IBM Oracle Solutions . Planning and Installation Guide" (GA22-7453) – Administration and Programming Reference (SA22-7452) – Problem Determination Guide (GA22-7434) Clusters Library: http://www.Advanced Technical Support .

Advanced Technical Support .IBM Oracle Solutions .Americas QUESTIONS ANSWERS 55 Oracle for AIX Workshop © 2006 IBM Corporation .

Sign up to vote on this title
UsefulNot useful