IBM Global Services

Oracle on AIX : Best Practices

Dr. Andreas Gruber mySAP Technology Services

November 2003

2003 IBM Corporation

IBM Global Services

Agenda
1 Introduction 2 Design of Disk Layout for SAP/Oracle Databases 3 Performance Tuning for the Journaled Filesystem 4 AIX 5L and the Enhanced Journaled Filesystem (JFS2) 5 Multiple SAP Databases on a Single System

Oracle on AIX : Best Practices

2003 IBM Corporation

IBM Global Services

Agenda
1 Introduction 2 Design of Disk Layout for SAP/Oracle Databases 3 Performance Tuning for the Journaled Filesystem 4 AIX 5L and the Enhanced Journaled Filesystem (JFS2) 5 Multiple SAP Databases on a Single System

Oracle on AIX : Best Practices

2003 IBM Corporation

IBM Global Services

Introduction
"A best practice is a technique or methodology that, through experience and research, has proven to reliably lead to a desired result." http://searchVB.com

Knowledge transfer from SAP infrastructure projects based on

IBM pSeries, Enterprise Storage Server, EMC Symmetrix, Storage Area Networks Tivoli Storage Manager and Data Protection for SAP

Performance tuning as a holistic approach that involves all components of a system Explanation and advertising of new features implemented by hardware and software components

Oracle on AIX : Best Practices

2003 IBM Corporation

IBM Global Services Introduction > Oracle Performance Tuning

Oracle Performance Tuning

List of Priorities
Business Rules Data and Application Design Logical Structure of the Database Database Operations and Access Paths Memory Allocation I/O and Physical Structure Resource Contention Underlying Platform(s)
AIX / Hardware AIX / Hardware AIX / Hardware

Oracle on AIX : Best Practices

2003 IBM Corporation

IBM Global Services

Agenda
1 Introduction 2 Design of Disk Layout for SAP/Oracle Databases 3 Performance Tuning for the Journaled Filesystem 4 AIX 5L and the Enhanced Journaled Filesystem (JFS2) 5 Multiple SAP Databases on a Single System

Oracle on AIX : Best Practices

2003 IBM Corporation

IBM Global Services Disk Layout > Old Style

Old Style Disk Layout for SAP/Oracle Database

Use of single and separated disks
SCSI-Attachment SSA-Loops

Placement of files according to the following criteria

Type of Files (Redolog, Archives, Datafiles) Data / Index Tablespaces Master Data / Transaction Data

Elimination of arising hot spots via reorganization of tables into separate tablespaces Problem
Wasting of disk resources (e.g. Redologs) Complex and difficult administration of scattered data files

Oracle on AIX : Best Practices

2003 IBM Corporation

IBM Global Services Disk Layout > New Paradigm

New Paradigm for SAP/Oracle Database

Intelligent Disksubsystems
Internal Read/Write Caches (0,5 - 64 GByte) Internal RAID-5 or RAID-10 (or similar) implementations SCSI or Fibre Channel-Attachment

Oracle's S.A.M.E Strategy (Stripe And Mirror Everything)

EMC Symmetrix / DMX Striping Stripe almost everything Don't subset by individual disks Hardware striping for Log Files Use 1 MB stripe size Mirror for data protection Use "Outside Edge" of disks IBM Enterprise Storage Server Stripe across ESS RAID Arrays Physical Partition striping (16/32 MB)

Mirroring

Use RAID -5 or RAID -10 Use "Middle" for ESS arrays to minimize seek

Placement Policy

Oracle on AIX : Best Practices

2003 IBM Corporation

IBM Global Services Disk Layout > Example

Disk Layout Example for SAP/Oracle Database

Oracle on AIX : Best Practices

2003 IBM Corporation

IBM Global Services Disk Layout > Golden Rules for ESS (Shark)

Golden Rules for ESS (Shark)

Number of Disksubsystems
Should be based on throughput needs Should not be based on capacity needs (Terabytes)

Decision Support Systems (e.g. BW)

100-200 GByte Data per 2GBit Fibre Channel Port Moderate Cache Requirements

OLTP Systems
200-300 GB Data per 2GBit Fibre Channel Port Large Cache Requirements

Number of RAID Arrays according to throughput requirements

50 MB/s sustained sequential Read/Write Performance

10

Oracle on AIX : Best Practices

2003 IBM Corporation

IBM Global Services

Agenda
1 Introduction 2 Design of Disk Layout for SAP/Oracle Databases 3 Performance Tuning for the Journaled Filesystem 4 AIX 5L and the Enhanced Journaled Filesystem (JFS2) 5 Multiple SAP Databases on a Single System

11

Oracle on AIX : Best Practices

2003 IBM Corporation

IBM Global Services Performance Tuning for JFS > JFS vs. Raw Devices

Journaled Filesystem vs. Raw Devices

"Raw Devices" correspond to Logical Volumes in AIX
Disk devices (/dev/rhdisk10) are not allowed to be accessed directly

Filesystems are easier to manage for most administrators Journaled Filesystems ensure the integrity of the filesystem structure and data
Locking of filesystem structures prevents simultaneous access

JFS access to data always uses the AIX Virtual Memory Manager and the real memory as a buffer for files
Improves performance when data is accessed multiple times Oracle data blocks are buffered twice and consume memory

12

Oracle on AIX : Best Practices

2003 IBM Corporation

IBM Global Services Performance Tuning for JFS > Areas of Tuning

Areas of Tuning

13 Oracle on AIX : Best Practices

Oracle Parameters init.ora Asynchronous I/O Read-Ahead File Caching Database Layout Multipath and SAN Cache and internal access Number and speed of drives
2003 IBM Corporation

IBM Global Services Performance Tuning for JFS > File Caching

File Caching
Memory is categorised into two different types
Computational memory: Consists of the pages that belong to working-storage or executable files. File memory: Consists of the remaining pages. These are usually pages from permanent data files in persistent storage.

AIX is using up all memory until only "minfree" pages are left
The percentage of memory that is used for file cache (numperm) can be seen with the command /usr/samples/kernel/vmtune (AIX 4.3 + 5.1) /usr/bin/vmstat -v (AIX 5.2) minperm If numperm falls below this level, the page-replacement algorithm steals both file and computational pages, regardless of repage rates. maxperm If numperm rises above this level, the page-replacement algorithm steals only file pages.
2003 IBM Corporation

14

Oracle on AIX : Best Practices

IBM Global Services Performance Tuning for JFS > VMM Tuning

Virtual Memory Manager (VMM) Tuning

Tuning of the file cache will only improve performance if the system was memory constrained before Tuning is meant to avoid duplicate buffering of Oracle data blocks which leads to a waste of physical memory If the SAP R/3 database server has more than 2 GB memory and is only used for SAP services
minperm should be set equivalent to ~ 150 MB maxperm should be set equivalent to ~ 300 MB In case of 3 GB memory this corresponds to /usr/samples/kernel/vmtune -p 5 -P 10

The number of pages on the freelist

minfree = 120 * Number of CPUs

15

Oracle on AIX : Best Practices

2003 IBM Corporation

IBM Global Services Performance Tuning for JFS > Tuning Read-Ahead

Tuning Read-Ahead
VMM tries to anticipate the future need for pages of a sequential file by observing the pattern a program uses to access the file.
minpgahead The number of pages read ahead when the VMM first detects the sequential access pattern. If the program continues to access the file sequentially, the next read ahead will be doubled. maxpgahead The maximum number of pages the VMM will read ahead in a sequential file.

Recommended values for SAP database servers

minpgahead maxpgahead maxfree 2 8 * #CPUs rounded to the next power of 2 minfree + maxpgahead

In case of a 6 CPU System this corresponds to ( < AIX 5.2) /usr/samples/kernel/vmtune -r 2 -R 64 -f 720 -F 784

16

Oracle on AIX : Best Practices

2003 IBM Corporation

IBM Global Services Performance Tuning for JFS > Read-Ahead for backup

Read-Ahead for backup with TSM and TDP

Backup of Oracle databases with TSM and TDP heavily rely on read-ahead
Performance Improvements of 100% have been observed

Journaled Filesystem
maxpgahead = 256

Enhanced Journal Filesystem (JFS2)

j2_maxPageReadAhead = 256

High values of read-ahead might have negative effects on cache efficiency of disk subsystems for SAP production
Switch to these parameters only during backup

17

Oracle on AIX : Best Practices

2003 IBM Corporation

IBM Global Services Performance Tuning for JFS > Asynchronous I/O

Asynchronous I/O
Asynchronous I/O is an AIX operating system feature consisting of
Request Queue for I/O write requests Kernel processes (aioservers) that take requests from the queue

Programs must use the corresponding application programming interface in order to use asynchronous I/O
no wait for the completion of a write request no blocking of application execution

Parameters for tuning asynchronous I/O

minservers The minimum number of aioservers that are started for asynchronous disk I/O. The default value is 1. maxservers The maximum number of aioservers that are started for asynchronous disk I/O. The default value is 10. Since each aioserver uses memory, this number should not be much larger than the expected amount of simultaneous asynchronous disk I/O requests. maxreqs Maximum number of asynchronous disk I/O requests that can be stored in the queue. The default value is 4096.
2003 IBM Corporation

18

Oracle on AIX : Best Practices

IBM Global Services Performance Tuning for JFS > AIO Tuning Recommendation

AIO Tuning Recommendations

Asynchronous I/O is treated like a device and can be managed via smit aio
autoconfig should be switched on, to make it "available" after reboot

Recommended starting values are

minserver maxserver maxreqs 2 or Number of CPUs -1, whatever is larger Two times the number of datafiles 12288

The aioserver processes are started as oracle child processes

for large SAP systems this number may become very large and the maximum number of processes per user has to be adopted chdev -l sys0 -a maxuproc=2000

The number of active aioservers can best be monitored with the performance tool "nmon"
http://www-106.ibm.com/developerworks/eserver/articles/analyze_aix/

19

Oracle on AIX : Best Practices

2003 IBM Corporation

IBM Global Services

Agenda
1 Introduction 2 Design of Disk Layout for SAP/Oracle Databases 3 Performance Tuning for the Journaled Filesystem 4 AIX 5L and the Enhanced Journaled Filesystem (JFS2) 5 Multiple SAP Databases on a Single System

20

Oracle on AIX : Best Practices

2003 IBM Corporation

IBM Global Services AIX 5L and JFS2 > New features

New features with AIX 5L

Support for POWER4 processors and LPAR technology Workload Management for CPU, Memory and I/O Alternatively 32-Bit or 64-Bit AIX Kernel
64-Bit Kernel recommended for JFS2 Max. Filesize 16TB (vs. 64 GB) (vs. 1 TB)

Enhanced Journal Filesystem (JFS2)

Max. Filesystemsize 16 TB Dynamic number of i-nodes B-Tree directory structure Higher performance for creation and deletion of files

AIX 5.2 implements new commands for performance tuning

21

Administration via smit Reboot safe

2003 IBM Corporation

Oracle on AIX : Best Practices

IBM Global Services AIX 5L and JFS2 > Direct I/O

Direct I/O for JFS2 with AIX 5L

Reads under cached I/O - cache miss Reads with direct I/O

22

Oracle on AIX : Best Practices

2003 IBM Corporation

IBM Global Services AIX 5L and JFS2 > Implications of direct I/O

Implications and restrictions of direct I/O

In order to activate direct I/O, the filesystem must be mounted with the option "mount -o dio" Read-Ahead is disabled for these filesystem The alignment of the read requests must be on 4K boundaries
In AIX 5.2 ML01 this alignment must be according to "agblksize", a parameter which is specified during file system creation If the alignment criteria is not met, normal cached access takes place but the cache is discarded immediately Oracle Online-Redologs should be located in separate filesystems with an allocation group blocksize (agblksize) of 512 Byte

Applications that see performance benefits when using raw logical volumes for storage are likely to benefit from the use of Direct I/O

23

e.g. Oracle
2003 IBM Corporation

Oracle on AIX : Best Practices

IBM Global Services AIX 5L and JFS2 > Locking and serialization

Locking and serialization

Filesystem Inodes contain meta information about files or directories
if a file is changed, also the inode is changed if the inode is changed the file might be unchanged

Inode locks prevent concurrent modifications of files that lead to inconsistencies of data
JFS2 uses a read-shared write-exclusive lock Read and write serialization granularity is only at file level

Database applications are more sophisticated and have a finer granularity concerning locking and serializaton of access to data
Table locking Row-level locking

24

Oracle on AIX : Best Practices

2003 IBM Corporation

IBM Global Services AIX 5L and JFS2 > Concurrent I/O

Concurrent I/O
AIX 5.2 ML01 provides JFS2 with the concurrent I/O feature
Multiple threads can simultaneously perform read and write operations on a shared file The option can either be used to open files or a whole filesystem can be mounted with mount -o cio

If only a subset of files in a filesystem should be opened in concurrent mode, the subdirectory can be mounted as namefs
mount v namefs o cio /somefs/subsomefs /somefs

Direct I/O is implicitly used with concurrent I/O Extension or truncation of files results in a change of the inode lock from shared to exclusive during the time of modification

25

Oracle on AIX : Best Practices

2003 IBM Corporation

IBM Global Services AIX 5L and JFS2 > Benchmark results

Benchmark results

26

Oracle on AIX : Best Practices

2003 IBM Corporation

IBM Global Services AIX 5L and JFS2 > Tuning Parameters for JFS2

Tuning Parameters for JFS2

New tuning commands with AIX 5.2 replace vmtune and schedtune
vmo, ioo, schedo

Values are preserved after reboot, if pre AIX 5.2 compatibility mode is switched off
chdev -l sys0 -a pre520tune=disable

JFS2 file cache does not use the minperm/maxperm values

maxclient Hard limit of file pages for JFS2 and NFS and must be set smaller than or equal to maxperm

Read-Ahead parameters must be set separately for JFS2 via ioo

j2_minPageReadAhead j2_maxPageReadAhead

27

Oracle on AIX : Best Practices

2003 IBM Corporation

IBM Global Services

Agenda
1 Introduction 2 Design of Disk Layout for SAP/Oracle Databases 3 Performance Tuning for the Journaled Filesystem 4 AIX 5L and the Enhanced Journaled Filesystem (JFS2) 5 Multiple SAP Databases on a Single System

28

Oracle on AIX : Best Practices

2003 IBM Corporation

IBM Global Services Multiple SAP Databases > Motivation

Motivation for consolidation of databases

Servers have become more and more powerful
Provide more performance than can be used by one database Less operating system images to maintain Number of SAP systems and databases is exploding, budgets are shrinking

Workload Management enables balancing of resources

Better usage of computing resources, including I/O and network Peak usage can be satisfied

Dynamic Logical Partitioning enables on-demand resources

CPUs and memory can be moved around freely Storage technology enables flexible disk assignment Hot-swapping of adapters

29

Oracle on AIX : Best Practices

2003 IBM Corporation

IBM Global Services Multiple SAP Databases > Guidelines and hints

Guidelines and hints for Oracle

Naming and numbering conventions
Enforce User-ID and Group-ID standards Use naming conventions for Networks, Volume Groups and Volumes

Oracle Installation Information

Work with symbolic links for each SID for the installation directories /oracle/jre, /oracle/oui, /oracle/inventory (SAP OSS Note 350251)

AIX kernel and Oracle kernel extensions

Oracle 8 is only supported as 32-Bit version on AIX 5L Oracle 8 kernel extensions only work with the 32-Bit AIX 5L kernel Oracle 8 Patch 2896876 (Metalink note 231901.1) provides extension for 64-Bit AIX 5L Kernel

Oracle listeners
Separate listeners should be configured for each instance with unique port numbers
2003 IBM Corporation

30

Oracle on AIX : Best Practices

IBM Global Services Multiple SAP Databases > Guidelines and hints SAP

Guidelines and hints for SAP

Transport Directory
Each system environment should have a separate transport directory, e.g. /usr/sap/trans<SID> The following variables should be set in the SAP system DIR_TRANS=/usr/sap/trans<SID> DIR_EPS_ROOT=/usr/sap/trans<SID>/EPS

Performance Collector saposcol

Start the saposcol from an SID independent path such as /etc/sap/saposcol Create an entry in the inittab file
mkitab i strload saposcol:2:once:su sidadm c /etc/sap/saposcol l

31

Oracle on AIX : Best Practices

2003 IBM Corporation

IBM Global Services Multiple SAP Databases > Workload Management

Workload Management
Workload Management included in AIX at no additional costs
Easy setup for SAP and Oracle (30 Minutes) Passive mode for testing, performance monitoring and accounting

Management of CPU Resources

Management of memory easier with SAP and Oracle parameters

Soft limits and hard limits possible

No wasting of resources

Classification via User-IDs or Tags

orasid and sidadm are perfectly suited for easy classification Separation of classification of Dialog/Background workprocesses through direct assignment

32

Oracle on AIX : Best Practices

2003 IBM Corporation

IBM Global Services Multiple SAP Databases > Workload Management Example

Workload Management Example

Superclasses for 4 SAP systems I1A,IPC, KPC and IPB Subclasses SAP, Oracle DB, Live Cache for APO system I1A

33

Oracle on AIX : Best Practices

2003 IBM Corporation

IBM Global Services

Summary
Modern disk subsystems facilitate database layouts through large caches and internal striping and mirroring Correct configuration of file cache in AIX prevents waste of memory Sufficient asynchronous I/O server processes are key to good performance JFS2 offers almost raw device performance with concurrent I/O Workload Manager in passive mode ideal for monitoring

34

Oracle on AIX : Best Practices

2003 IBM Corporation

IBM Global Services

Questions ... ?

November 2003

2003 IBM Corporation

IBM Global Services

Agenda
1 Introduction 2 Design of Disk Layout for SAP/Oracle Databases 3 Performance Tuning for the Journaled Filesystem 4 AIX 5L and the Enhanced Journaled Filesystem (JFS2) 5 Multiple SAP Databases on a Single System 6 References

36

Oracle on AIX : Best Practices

2003 IBM Corporation

IBM Global Services

References
IBM Redbooks and Redpapers at www.redbook.ibm.com
"A holistic approach to a reliable infrastructure for SAP R/3", SG24-5050 "Consolidating multiple SAP systems on one pSeries", REDP-3626-00

SAP Market Place at service.sap.com

"Database Layout for R/3 installation under Oracle"

Oracle Whitepapers at technet.oracle.com

"Optimal Storage Configuration Made Easy"

Enterprise Storage Server white papers for Oracle and SAP

http://www.storage.ibm.com/disk/ess/whitepaper.htm

AIX Whitepapers, e.g. on Concurrent I/O

http://www-1.ibm.com/servers/eserver/pseries/library/wp_aix_lit.html

37

Oracle on AIX : Best Practices

2003 IBM Corporation