IBM System Storage and SAP High Availability Installations on AIX

including PowerHA, Power Server and VIOS aspects

Version Date: December 20, 2010 Version 2 Katharina Probst Walter Orb

IBM Boeblingen Lab, Germany SAP on Power Systems Development @ SAP Copyright IBM Corp. 2009

Copyright IBM Corp. 2010

Table of contents
Table of contents.................................................................................................................................. 2 Abstract................................................................................................................................................. 3 Introduction........................................................................................................................................... 4
Dual VIOS...................................................................................................................................................4.......... AIX Operating System................................................................................................................................4.......... PowerHA.....................................................................................................................................................5.......... SAP.............................................................................................................................................................6..........

Storage Setup........................................................................................................................................ 7
Eliminate Hardware SPoF...........................................................................................................................8.......... Eliminate SPoF in the Path.........................................................................................................................8.......... Data Protection using Storage Features.....................................................................................................9.......... Data Redundancy.......................................................................................................................................9.......... Storage Volumes........................................................................................................................................9.......... Storage Volume Groups.............................................................................................................................11........

Resources............................................................................................................................................ 12 About the authors............................................................................................................................... 12 Trademarks and special notices........................................................................................................ 13

Copyright IBM Corp. 2010

Abstract
This paper is intended to give an understanding of technical matters when setting up high availability SAP environments using IBM System Storage DS and Power Systems on AIX. While there are good papers about how to set up storage and making the databases well performing in general, we focus here particularly on high availability (HA) installations and disaster recovery (DR) aspects in the SAP context. For those skilled in the art it is obvious that IBM Power Systems technology including AIX, VIOS concepts and the used Software namely PowerHA and SAP Software require a corresponding storage layout to work. This document is written on base of the following levels/technologies:

IBM PowerHA for AIX Version 5.5 (HACMP) IBM System p5 Server with POWER5+ technology IBM AIX Version 6.1 VIOS 2.1 SAP NetWeaver 7.00 IBM System Storage DS8300 Model 2107-9A2

Copyright IBM Corp. 2010

Introduction
Applications, servers and storage are often administered by different divisions or persons, what often leads to an isolated view. Hence the intention is to show the end-to-end concept from a storage point of view for HA (high availability) enabled SAP landscapes. The Application: Today, SAP supports a special HA installation option. This option separates the database, the SAP Central Services (enqueue and message server), the Enqueue Replication Server and the Instances into independent components. This design allows each group to move independently between nodes, not requiring to move entire SAP systems. To enable a move of an application the management of these resources is required. One possibility is to use PowerHA (HACMP). The Server: The virtualization technologies provided when using AIX on Power Systems became key for SAP landscapes by providing more flexibility and better utilization. One item that will be later discussed in more detail is the dual VIOS concept, allowing the virtualization of disk and network and sharingof few physical resources by many partitions while maintaining redundancy. The following chapters will discuss each of the topics mentioned above in an end-to-end view. The focus is to describe what needs to be accomplished on the storage side to increase the robustness of the concept. This is achieved by synchronization of the concept with the available technologies and software and elimination of single point of failures (SPoF). Dual VIOS A Virtual IO Server is an appliance that provides virtual storage and shared Ethernet attachment. Today there is more than one way to set it up. In general, the main benefit is to attach fewer physical adapters for Ethernet and storage to the VIOS servers and virtualize them for many LPARS sharing the same physical server. This reduces the amount of required physical adapter substantially. Whereas earlier, at least 2 Ethernet and 2 fibre channel adapters per partition were required to eliminate single point of failures (SPoF). With just 4 adapters of each type and a dual VIOS concept we can fulfill this requirement for an entire server. Of course, for performance considerations there might be more physical adapters required. Due to the high value of virtualization and its benefits for cost reduction, a dual VIOS concept is highly recommended to be part of every SAP landscape. The requirement for the storage layout in combination with PowerHA is to create a 1:1 association between storage volume and AIX volume group on VIOS level to mange the disks by two VIOS Servers. Rule 1: Do not split or merge storage volumes on the VIOS level. To learn more about the different options to use and to setup a VIOS please see the Advanced POWER Virtualization on IBM System p: Virtual I/O Server Deployment Examples document at http:// www.redbooks.ibm.com/abstracts/redp4224.html. AIX Operating System The AIX operating system is required to run the applications. If the operating system fails a fall-over needs to be initiated. Although it can be entirely automated by PowerHA, storage redundancy can help to reduce the likelihood of a fall-over. The recommendation is to mirror the rootvg to at least two storage volumes on different storage boxes using a LVM mirror. This provides the benefit of un-disruptive operation of the operating system in case of a

Copyright IBM Corp. 2010

disk, cabling or storage subsystem failure within one site. The LVM mirror today supports up to three mirror copies. Rule 2: Create a LVM mirror volume for the rootvg on a separate storage subsystem. A good reference for the different mirroring options can be found in AIX Logical Volume Manager, from A to Z: Introduction and Concepts in Chapter 2 at http://www.redbooks.ibm.com/redbooks/pdfs/sg245432.pdf The rootvg has a slightly different procedure in comparison to mirroring non bootable disks. Step-by-step instructions can be found here: http://publib.boulder.ibm.com/infocenter/aix/v6r1/index.jsp?topic=/ com.ibm.aix.baseadmn/doc/baseadmndita/mirrorootvg.htm PowerHA The PowerHA product is a piece of software that manages applications (not the OS) and the corresponding resources. This cluster software monitors all nodes to detect failures in the application, the operating system, the network and storage connectivity to react on failures. It starts, stops, and moves applications and their resources on the defined cluster nodes based on policies and scripts. To make it work the executables and the data needs to be accessible on the node where the application is currently running. This has two aspects in regards to the layout of the storage: a) the resources such as disks need to move between nodes to make the application available (HA). b) the data needs to be kept redundant to protect from a disaster (DR). In order to have the data and executables accessible PowerHA can use two different approaches. If concurrent write is handled by the application a clustered filesystem (GPFS) can be used to have all data on all nodes at any time. The recommended approach today for SAP is to use enhanced concurrent volume groups in non-concurrent mode which are created on shared storage volumes. Shared storage volumes have a path to all nodes the application can potentially run on. Now PowerHA can varyon and varyoff the volume groups on the node where the volume is required. Rule 3: PowerHA with enhanced concurrent volume groups require a minimum of oneshared storage volume for each moving application. A well designed HA concept also requires a DR concept. Meaning the duplication of data to recover from a disaster. On the one hand this protects the business relevant data and on the other the files required for an application files to restart it on the fall-over node. The scop of this concept reaches from Tape Backup, RAID levels, storage mirroring to LVM mirroring. Depending on the PowerHA release LVM mirroring, GLVM mirroring, storage mirroring (Metro Mirror and Global Mirror) are integrated into PowerHA, and by that are cluster-aware. For a one-site-HA-solution each volume used for a SAP component should be mirrored to a second storage subsystem using LVM. The advantage of LVM mirroring is the seamless switch to a synchronous copy. A two site solution needs to transfer the data to a remote location where LVM usually is not suitable any more. Here storage based mirroring technologies can be used. Details are discussed in the section Data Protection using Storage Features. The other protection mechanisms are not directly linked to a SAP storage layout, but a necessity to protect the data. Rule 4: Each storage volume used for a SAP component needs at least one additional volume on a different storage subsystem to mirror the data and the SAP system files. An overview of the different DR and HA concepts including the mirroring options can be found in the Resource section.

Copyright IBM Corp. 2010

SAP The SAP installation consists of the SAP executable and a database. First we look at the SAP executable, followed by the database aspects. SAP provides the standalone enqueue and the enqueue replication server by a new HA installation option. The benefit is to split the SAP application into different components which:

can be moved between cluster nodes using PowerHA. Thus they have to be installed in their own volume groups see the PowerHA chapter before. are required on each node or are bound to a single node.

The benefit is the flexibility of not being forced to move the entire SAP system when performing administrative fallovers. Also, if one component dies a selective action can take place. Finally the replicated enqueue server is a backup of the enqueue table on a second node in order to protectthe state in a fall-over situation. The following tables summarizes SAP components per stack. As discussed before, each component needs at least one storage volume and by that the minimum amount of storage volumes (without counting the ones for mirroring) is described in these tables. 1. ABAP / JAVA Stack Instance Central Services ERS Description ABAP central Services: ASCS JAVA central Services: SCS Hot standby enqueue table Node behavior Moves between nodes each resides on a shared storage volume. Can be implemented to move between nodes or is installed on each node into a local filesystem. a shared volume for option one, one dedicated volume per node for option two. One copy on all nodes one dedicated volume per node (often mounted under /usr/sap). This can be either normal NFS mount and by that transparent to this setup. When using a NFS crossmount a shared volume is required. Instances are bound to a single node one dedicated volume per instance is required (min. two instances are required).

SYS

The /usr/sap/<SID>/SYS directory includes /sapmnt and /usr/sap/ trans /usr/sap/<SID>/<INSTANCE>

SAP global

Instance

2. Double Stack Instance ASCS SCS Description ABAP central services JAVA central services Node behavior Moves between nodes resides on a shared storage volume. Moves between nodes resides on a shared storage volume.

Copyright IBM Corp. 2010

Instance 2 x ERS

Description

Node behavior

Can be implemented to move between nodes or is Hot standby enqueue table, one installed on each node into a local filesystem. two shared volumes for option one, two dedicated for the JAVA and one for the volumes per node for option two. ABAP enqueue. The /usr/sap/<SID>/SYS directory includes /sapmnt and /usr/sap/ trans /usr/sap/<SID>/<INSTANCE> One copy on all nodes one dedicated volume per node (often mounted under /usr/sap). See above.

SYS

SAP global

Instance

See above.

Rule 5: Each moving SAP component requires a minimum of one shared storage volume. The local components are recommended to have their own storage volumes too. It is recommended to mirror all of them to a second storage subsystem which doubles the amount of required storage volumes. The database availability can be achieved in different ways, depending on the used database technology. Basically there are the following possibilities: Shared disk installation as a cold standby solution Clustered database (DB2 PureScale, Oracle RAC) Second DB instance as hot stand-by solution (DB2 HA/DR, Oracle Data Guard) covering the DR aspect for the database. Please check the support status of your combination of PowerHA, SAP, DB and OS. In general data, log , and other files are put onto different AIX volume groups, having their own volumes on the storage subsystem. The common approach for a SAP DB layout is to create storage volumes for each /<db>/ <SID>/sapdata* file system, at least one for the log-files and a last one for the rest. Other tools as for example the Tivoli Storage FlashCopy Manager might have additional requirements for the volume group layout, so it is recommendedto use a least in three different volume groups fordata, log and exe. Please check the references in the Resources Section for recommendations on how to implement a storage layout for a well performing database. Rule 6: For a shared disk installation one shared storage volume might technically be sufficient for a shared disk database installation. A better, but still basic approach is to use a minimum of six shared storage volumes. Assign four storage volumes to the data volume group and create four filesystems for the data, followed by the log-files and the rest. The size of the SAP volumes depends on many factors. Please refer to the SAP installation guides you can obtain from your SAP representative or www.sap.com. Be aware that the installation guides describe the absolute minimum requirements to run the installation. Usually you will need more.

Storage Setup
The task of the storage is to protect the data and the SAP executables. The Recovery Time Objective (RTO) is a variable describing the amount of lost data in case of a disaster. The RTO goes directly along with the used mirroring technology, the distance to mirror and the quality of the network. Also the connectivity between the server and the storage and volume layout influences the robustness.

Copyright IBM Corp. 2010

We discuss here a one site HA solution with full HA capabilities, meaning a HA concept also taking DR into account. More details and the transfer to a more complex setup can be found in IBM Storage and SAP evaluation of HA and DR concepts which is listed in the Resource section. Eliminate Hardware SPoF The basic equipment for a one site full HA solution consists of a minimum of:

two Power Servers two SAN Switches and two Storage Units

as shown in the picture on the right. The servers are required to be protected against a server outage to let the application switch over to the second node. Also a SAN or network switch can become a SPoF if it is not redundant. Finally the storage subsystems are physically duplicated to protect data by mirroring and enable the application to get undisrupted data access. Eliminate SPoF in the Path The next step is to ensure that beginning from the application running in a LPAR down to the storage volumes all paths are kept redundant. From a Power server view the path for the fiber channel can look as illustrated in the following picture:

Power Server

LPAR Dual VIOS concept Min. two physical adapter for each VIOS Each VIOS is attached to min. two switches.

Switch

Each switch is to be zoned that each VIOS sees both storage systems through each switch it is attached to.

Storage

Use at least two I/O enclosures to get to a volume.

For Ethernet the same rule of redundancy has to be applied accordingly to the technology.

LPAR Dual VIOS concept Min. two physical Ethernet adapter for each VIOS. Each VIOS is attached to min. two different networks.

Copyright IBM Corp. 2010

Data Protection using Storage Features In order to protect data IBM storage provides a set of technologies suitable for SAP landscapes. The availability of the feature depends on the model and the purchased licenses. Full Disk Encryption (FDE): This storage security feature is hardware based, well performing encryption of the physical disks. It is based on special FDE-disks and key servers holding the keys for encryption and decryption. This protects the content on the disks for example during relocation or on the way to recycling as long the key servers are kept safe. RAID: RAID protects from disk failures within a box. The DS8000 models provides RAID level 5 (Single disk failure) up to level 10 (full duplication and striping). The chosen level has an direct effect on the amount of required disk space. It is best practice not to mix the RAID level for a SAP landscape. Storage Mirroring: Point to Point Remote Copy (PPRC) comes in two flavors. Metro Mirror is a synchronous mirroring technology often used to connect data centers within city boundaries where LVM reaches its limits. Global Mirror is asynchronous and by that suitable for long, unlimited distances. These technologies are used if a secondary site is to be connected into a HA/DR setup. More details can be found in IBM Storage and SAP evaluation of HA and DR concepts listed in the Resource section. Data Redu ndancy To protect the environment from a loss of any kind of data, several mechanisms need to go hand in hand. First of all the data needs to be protected from disk failures. This is the first mandatory step when initializing the storage arrays: specifying a RAID level. With RAID 5 effectively 6 out of 8 disks can be used for data. The other two disks in the rank are used for spare to replace the broken disk and parity to recalculate the lost data. RAID 6 and RAID 10 increase the robustness against disk failures within one rank. On the other hand it is not recommended for SSD drives and reduces the net capacity by requiring more disks for spare and parity. The RAID level to choose for SAP applications is RAID 5 for most cases. The next step is to protect against the connectivity lost of one storage system. This can be caused by broken cables, loss of power, or any kind of site disaster. Hence the recommendation is to mirror the data to at least one additional, independent storage subsystem within one site and to a secondary remote site. Within one site LVM mirroring is widely used to mirror all SAP, DB and rootvg storage volumes. The copy within one site minimizes the RPO in case of a single hardware or connectivity failure. To provide additional robustness a second site protects against an entire data center outage. For that purpose storage based mirroring (PPRC) of the SAP and DB related volumes is recommended. The RTO value increases depending on the chosen technology (synchronous or asynchronous), the distance and the bandwidth. Storage Volumes First of all the mirroring requires at least to duplicate the storage layout to a second storage subsystem. This means duplicate the physical resources, power supply, , the volumes and their corresponding volume groups. If the quorum check on OS level is used a Quorum Disk on an third storage box is recommended to avoid a failing fall-over of the SAP application in case one volume is lost. Furthermore, the zoning of the switches need to allow the servers to share volumes as described earlier in more detail.

Copyright IBM Corp. 2010

Here a summary of the minimum recommended storage volume requirement:

AIX

Min. one volume per node for the rootvg One volume for paging space

SAP, PowerHA and dual VIOS

Min. one (shared) volume for the database logfiles Min. one (shared) volume for the /<db> file system Min. one (shared) volume per /<db>/<SID>/sapdata* file system, combined into one volume group on AIX. Min. one shared volume for each moving component: Central Services, ERS (optional), NFS Min. one local volume for each instance Min. one local volume per node for the /usr/sap file system

Data Redundancy

Mirror the volumes listed before to at least one second storage system duplicate amount of volumes RAID 5 or RAID 6 Backup solution (for example Tivoli Storage Manager for ERP)

The overall minimum amount of storage volumes is 29 for one SAP full HA installation. A SAP landscape has at least three installations (test, quality assurance and production). Now we are counting already around 90 volumes. In order not to loose track of the volumes a concept of assigning the LUNS can help. A LUN is a 4 digit hex number to identify a storage volume within one storage box. The original meaning of the LUN needs to reflect the original purpose. LUNs are having the following Format: a b cc (each letter represents one hex number). The original meaning is: a is the group identifier. Best practice is to use a value of 0 for System z, all other values for other Systems. ab defines the LSS. Volumes where this value is even are handled by the first managing Power Server in the storage system, uneven by the second. If one power Server fails all are handled by the second. Hence the value ensures that the workload is spread between the two Power servers. cc specifies the device number.

1 0

Copyright IBM Corp. 2010

One possible concept amongst lots of different possibilities to add some additional information to it can be as followed: a is assigned to a SAP SID. PRD=volumes starting with 1, QAS= volumes starting with 2, ab use it to spread workload evenly. cc identify the purpose. On all storage systems, the volumes that are mirrored to each other, have the same number. This method does not remove the necessity to document it, but helps to find mistakes in the end. Also for the VIOS setup it helps to pre-order the disks for the PowerHA setup on AIX on all nodes in the same way to avoid mistakes in the setup. NOTE: Planning for an SAN Mirror integration the LSS has additional meaning. Consistency groups are defined on base of LSS, meaning for example the Database data for each database needs to have an unique LSS number to avoid bottlenecks. Storage Volume Gr oups The storage volume groups are grouping storage volumes. The concept becomes clear when looking at a native storage to server attachment (without using storage virtualization technoligies like SAN Volume Controller ) This . also means that each server sided fiber channel port can be attached to one storage volume group, not more. The other way round: a storage volume group can be attached to multiple server ports. This means a 1:n relation between fiber channel port on the server and storage volume groups. Volumes for SAP applications with PowerHA are classified from an application view to be available on node1, on node2, or shared between node1 and node2. Recalling the information of the previous chapters describing the elimination of SPoF and the 1:n relation between fibre-channel port and storage volume groups, we can assign one storage volume group to each server. Otherwise the path redundancy is not fulfilled anymore. When using more adapters, more storage volume groups can be created if desired. For the described hardware setup the storage volume group A must contain all node1 volumes and the shared volumes. The node2 storage volumes and the shared volumes are assigned to volume group B as illustrated in the picture above.

1 1

Copyright IBM Corp. 2010

Resources
These Web sites provide useful references to supplement the information contained in this document:

Logical configuration concepts for IBM System Storage DS8000 series in large data centers ftp://service.boulder.ibm.com/storage/isv/LogicalConfigConceptsDS8000.pdf DS8000 Performance guide www.ibm.com/support/techdocs WP101602 IBM Redbooks http://www.redbooks.ibm.com/ IBM Storage and SAP evaluation of HA and DR concepts http://www.ibm.com/support/techdocs WP101616

About the authors

Katharina Probst Walter Orb probst@de.ibm.com walter.orb@de.ibm.com

1 2

Copyright IBM Corp. 2010

Trademarks and special notices

Copyright IBM Corporation 2008. All rights Reserved. References in this document to IBM products or services do not imply that IBM intends to make them available in every country. IBM and the IBM logo are trademarks of International Business Machines Corporation in the United States, other countries, or both: http://www.ibm.com/legal/ copytrade.shtml Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. Intel, Intel Inside (logos), MMX, and Pentium are trademarks of Intel Corporation in the United States, other countries, or both. UNIX is a registered trademark of The Open Group in the United States and other countries. Linux is a trademark of Linus Torvalds in the United States, other countries, or both. SET and the SET Logo are trademarks owned by SET Secure Electronic Transaction LLC. ABAP, SAP EarlyWatch, SAP GoingLive, SAP NetWeaver, SAP, and SAP logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries. Other company, product, or service names may be trademarks or service marks of others. Information is provided "AS IS" without warranty of any kind. All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer. Information concerning non-IBM products was obtained from a supplier of these products, published announcement material, or other publicly available sources and does not constitute an endorsement of such products by IBM. Sources for non-IBM list prices and performance numbers are taken from publicly available information, including vendor announcements and vendor worldwide homepages. IBM has not tested these products and cannot confirm the accuracy of performance, capability, or any other claims related to non-IBM products. Questions on the capability of non-IBM products should be addressed to the supplier of those products. All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. Contact your local IBM office or IBM authorized reseller for the full text of the specific Statement of Direction. Some information addresses anticipated future capabilities. Such information is not intended as a definitive statement of a commitment to specific levels of performance, function or delivery schedules with respect to any future products. Such commitments are only made in IBM product announcements. The information is presented here to communicate IBM's current investment and development activities as a good faith effort to help with our customers' future planning. Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput or performance improvements equivalent to the ratios stated here. Photographs shown are of engineering prototypes. Changes may be incorporated in production models. Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk.

1 3

Copyright IBM Corp. 2010