Sun Microsystems GmbH

Sun Microsystems GmbH

High Availability with SAP enqueue replication and Sun Cluster

Author: Created: Last Changed: Version:

Jrg Meiners, Sun Microsystems GmbH 16.01.2009 17.03.2009 02

Page 1 of 21

Sun Microsystems GmbH

Table of Contents
1 - Conceptual overview...........................................................................................................................................................3 2 - About this paper..................................................................................................................................................................4 3 - Description of the target setup.............................................................................................................................................5 3.1 - Cluster configuration...................................................................................................................................................5 3.2 - Oracle database..........................................................................................................................................................5 3.3 - SAP System Central Services (SCS)................................................................................................................................6 3.4 - SAP Replicated Enqueue Server....................................................................................................................................6 3.5 - SAP Central Instance....................................................................................................................................................6 4 - Performing the installation..................................................................................................................................................8 4.1 - Solaris and Sun Cluster installation..............................................................................................................................8 4.2 - Basic Cluster configuration...........................................................................................................................................8 4.3 - Cluster data services configuration (part I)....................................................................................................................8 4.4 - SAP/Oracle Installation..............................................................................................................................................10 4.5 - Prepare SAP/Oracle switchover...................................................................................................................................11 4.6 - Test SAP/Oracle switchover........................................................................................................................................12 4.7 - Enable enqueue replication........................................................................................................................................12 4.8 - Test the enqueue replication......................................................................................................................................13 4.9 - Cluster data services configuration (part II).................................................................................................................14 4.10 - Enable and test the cluster.......................................................................................................................................16 5 - Appendix...........................................................................................................................................................................17 5.1 - Default profile...........................................................................................................................................................17 5.2 - ERS11 start profile......................................................................................................................................................17 5.3 - ERS11 instance profile................................................................................................................................................17 5.4 - SCS01 start profile......................................................................................................................................................18 5.5 - SCS01 instance profile................................................................................................................................................19 5.6 - JC00 start profile........................................................................................................................................................19 5.7 - JC00 instance profile...................................................................................................................................................20 5.8 - Script startsap_cluster.sh...........................................................................................................................................20 5.9 - Script stopsap_cluster.sh...........................................................................................................................................21

Page 2 of 21

Sun Microsystems GmbH

1 - Conceptual overview
The Enqueue Server is a critical single point of failure in SAP architectures. And furthermore - as the Enqueue Server holds its data in memory - a failure would mean to loose the central SAP enqueue table. As a consequence, all ongoing SAP transactions having an active lock would abort. And special only to the SAP JAVA-stack, all SAP application instances will automatically restart as soon as the Enqueue Server is online again (and having a new empty enqueue table with a new ID). To overcome this drawback, SAP strongly recommends to use the standalone Enqueue Server (as part of the SAP System Central Services) together with the Replication Enqueue Server especially in productive JAVA-stack systems. The Replication Enqueue Server runs on another host and contains a replica of the enqueue table (the replication table). Enqueue Server and Replicated Enqueue Server are running in a synchronous replication mode. Thus if the Enqueue Server fails and becomes online again, the enqueue table is not lost. Instead, the Enqueue Server can recover the enqueue table from the replication table. The SAP enqueue replication concept is based on an underlying HA software and can't work without that. Sun Cluster supports this architecture out-of-the-box by providing best suitable product features: Provides a robust and reliable cluster framework. Provides data agents for the basic infrastructure (Storage, Network). Provides data agents for the SAP and database applications. In this context especially for the SAP SCS (System Central Services: Message Server and Enqueue Server) and the Replicated Enqueue Server. Handles the particular needs of Enqueue Server and Replicated Enqueue Server in a failover environment If Enqueue Server fails, Sun Cluster will restart the Enqueue Server on the cluster node (= physical server) that is currently hosting the Replicated Enqueue Server. This is mandatory, because the restarted Enqueue Server will recover the enqueue table from the replication table using a server-local access to the shared memory segment that holds the replicated table. After table recovery has finished, the Replicated Enqueue Server will be stopped. Sun Cluster will restart the Replicated Enqueue Server on another cluster node (different than the node hosting the Enqueue Server) if another cluster node is available. Thus keeping up the enqueue replication concept and maximal availability. To learn more about the SAP enqueue replication and Sun Cluster, please refer to: Sun Cluster 3.2 documentation (http://docs.sun.com/app/docs/prod/sun.cluster32) Sun Cluster 3.2 Data Service for SAP Web Application Server Guide (http://docs.sun.com/app/docs/doc/820-2568) SAP documentation (http://help.sap.com SAP NetWeaver 7.0 Library Look for the chapter "Standalone Enqueue Server")

Page 3 of 21

Sun Microsystems GmbH

2 - About this paper

This paper describes the implementation of a SAP High Availability Architecture based on the SAP enqueue replication concept and a Sun infrastructure. An installation-HowTo is being provided, using: Sun Solaris 10 (Update 5/08) Sun Cluster 3.2.0 (Update 2/08) Oracle Database 10g Enterprise Edition Release 10.2.0.2.0 SAP NetWeaver Portal 7.0 (SP16, Kernel 7.00 patch# 159) The HowTo is focused on all necessary implementation and configuration tasks that are specific for the SAP Enqueue replication scenario. A standard installation of Solaris, Sun Cluster and SAP Portal is assumed and a description of this process is out of scope. Also it is not intended to describe all aspects of an High End SAP/Oracle/Cluster implementation. Many things in this example installation are implemented as simple as possible (like filesystem layout, parameter tuning, ...) to hold this paper small and to focus on SAP enqueue replication setup.

Page 4 of 21

Sun Microsystems GmbH

3 - Description of the target setup

3.1 - Cluster configuration
Cluster Name: sap1 Node Names: sscsapx1,sscsapx2 Transport Adapters: bge1,e1000g1 (on both nodes) Quorum Device Name: d11 (c4t600015D00022B30000000000000072CDd0) Public Network sscsapx1 IPMP-group: sc_ipmp0 bge0 (sscsapx1 192.168.14.35) e1000g0 (sscsapx1b 192.168.14.1) Public Network sscsapx4 IPMP-group: sc_ipmp0 bge0 (sscsapx2 192.168.14.36) e1000g0 (sscsapx2b 192.168.14.6)

SAP CI Logical Hostname SAP NW WAS Oracle DB Group Logical Hostname Oracle Listener Oracle Database Failover Filesystem ZFS (/oracle/PE2) SAP Replicated Enqueue Group Logical Hostname SAP Replicated Enqueue Server SAP SCS Group Logical Hostname NFS Share (/sapmnt/PE2, /usr/sap/trans) SAP Message Server SAP Enqueue Server Global Filesystem UFS+SVM (/usr/sap/PE2, /sapmnt/PE2, /usr/sap/trans) Local (root FS): /oracle/client /oracle/stage/102_64 Local (root FS): /oracle/client /oracle/stage/102_64

Figure 1: Overview Cluster Resourcegroups

3.2 - Oracle database

DB name PE2, installed on logical hostname pe2oralh.
Cluster-Resourcegroup: pe2ora; preferred node: sscsapx1 Cluster-resource pe2ora_stor Type SUNW.HAStoragePlus Properties zpool, zfs Devices: d10 (c4t600015D00022B30000000000000072D1d0) Failover filesystem Mountpoints: /oracle/PE2 (150 GB)

Page 5 of 21

Sun Microsystems GmbH

pe2ora_lh pe2ora_lsnr pe2ora_db SUNW.LogicalHostname SUNW.oracle_listener SUNW.oracle_server ip / hostname = 192.168.14.24 / pe2oralh

Notes regarding SAP/Oracle filesystem configuration: /oracle/client: Local directory in root-fs. To be copied on both cluster nodes. /oracle/stage/102_64: Local directory in root-fs of the install server. Not needed for running the database. /sapmnt/PE2 and /usr/sap/PE2: Will be accessible to the database by the cluster global filesystem. No need to copy anything.

3.3 - SAP System Central Services (SCS)

SAP SID PE2, Instance No. 01 (instance name SCS01), installed on logical hostname pe2scslh.
Cluster-Resourcegroup: pe2scs; preferred node: sscsapx1 Cluster-resource pe2scs_stor Type SUNW.HAStoragePlus Properties SVM diskset pe2scs_stor Devices: d11 (c4t600015D00022B30000000000000072CDd0) Global filesystems Mountpoints: /usr/sap/PE2 (20 GB), /sapmnt/PE2 (10 GB), /usr/sap/trans (10 GB) ip / hostname = 192.168.14.23 / pe2scslh (Just FYI: SAP Message Server) (Just FYI: SAP Enqueue Server) Exports /sapmnt/PE2 and /usr/sap/trans

pe2scs_lh pe2scs_ms pe2scs_enq pe2scs_nfs

SUNW.LogicalHostname SUNW.sapscs SUNW.sapenq SUNW.nfs

Note: Any necessary application shared data access (for example to /sapmnt/PE2) is done by using the global filesystem(s) within the cluster. This is also to avoid any problems with NFS cross-mounts between cluster nodes (that are generally not supported in a Sun Cluster configuration). The NFS server configured in this resourcegroup is to provide all cluster-external NFS-clients like additional SAP application instances with access to the shared data.

3.4 - SAP Replicated Enqueue Server

SAP SID PE2, Instance No. 11 (instance name ERS11), installed on logical hostname pe2erslh.
Cluster-Resourcegroup: pe2ers; preferred node: sscsapx2 Cluster-resource pe2ers_lh pe2ers_enq Type SUNW.LogicalHostname SUNW.saprepl Properties ip / hostname = 192.168.14.25 / pe2erslh (Just FYI: SAP Replicated Enqueue Server)

Notes regarding SAP/Oracle filesystem configuration: No dedicated filesystems needed. Global filesystems (/usr/sap/PE2, /sapmnt/PE2) will be used for instance specific data.

3.5 - SAP Central Instance

SAP SID PE2, Instance No. 00 (instance name JC00), installed on logical hostname pe2cilh.
Cluster-Resourcegroup: pe2ci; preferred node: sscsapx1 Cluster-resource Type Properties

Page 6 of 21

Sun Microsystems GmbH

pe2ci_lh pe2ci_jc00 SUNW.LogicalHostname SUNW.sapwebas ip / hostname = 192.168.14.22 / pe2cilh (Just FYI: SAP Web Application Server)

Notes regarding SAP/Oracle filesystem configuration: No dedicated filesystem needed. Global filesystems (/usr/sap/PE2, /sapmnt/PE2) will be used for instance specific data.

Page 7 of 21

Sun Microsystems GmbH

4 - Performing the installation

4.1 - Solaris and Sun Cluster installation
Install Sun Solaris 10 (Update 5/08) and Sun Cluster 3.2.0 (Update 01) on the servers sscsapx1 and sscsapx2. A description is out of scope of this paper. Note: The documentation Sun Cluster Quick Start Guide for Solaris OS (http://docs.sun.com/app/docs/doc/8202556) provides a good example (cookbook) for setting up the cluster hardware, installing the Solaris and Sun Cluster Software. Note: Using Solaris Volume Manager (SVM), take care not to use the same metadevice name for the filesystem / globaldevices on both nodes. The cluster will not work properly. Instead take different metadevice names (for example d200 on sscsapx1 and d400 on sscsapx2).

4.2 - Basic Cluster configuration

This step describes some basic cluster configurations. To be executed, if not done before. Add Quorum Device
#cldevicelistv #clquorumaddd11 #clquorumlist #clquorumstatus (FYI:toviewtheavailablediddevices) (addsd11asquorumdevice) (FYI:toviewtheconfiguredquorumdevices) (FYI:toviewthequorumstatus)

Network configuration

#eepromlocalmacaddress?=true

(tobeexecutedonbothclusternodes)

Data Services registration

#clresourcetyperegisterSUNW.HAStoragePlus #clresourcetyperegisterSUNW.LogicalHostname #clresourcetyperegisterSUNW.oracle_listener #clresourcetyperegisterSUNW.oracle_server #clresourcetyperegisterSUNW.sapscs #clresourcetyperegisterSUNW.sapenq #clresourcetyperegisterSUNW.saprepl #clresourcetyperegisterSUNW.nfs #clresourcetyperegisterSUNW.sapwebas

Hostnames Add all logical hostnames needed to the local /etc/hosts on sscsapx1 and sscsapx2
192.168.14.24pe2oralhpe2oralh.abc.de#logicalhostforOracleDB 192.168.14.23pe2scslhpe2scslh.abc.de#logicalhostforSAPSCS01 192.168.14.25pe2erslhpe2erslh.abc.de#logicalhostforSAPERS11 192.168.14.22pe2cilhpe2cilh.abc.de#logicalhostforSAPJC00

4.3 - Cluster data services configuration (part I)

This step will configure the storage, filesystems and IP addresses. Hereby providing the necessary infrastructure platform to install SAP/Oracle afterwards. Oracle database a) Create failover filesystem (zpool and zfs filesystem)

#zpoolcreatepe2ora_stor/dev/dsk/c4t600015D00022B30000000000000072D1d0 #zfscreatepe2ora_stor/oraclePE2 #zfssetmountpoint=/oracle/PE2pe2ora_stor/oraclePE2 #zpoolexportpe2ora_stor #zpoolimportpe2ora_stor (onthecurrentlyhostingnode) (onthetargetnode)

Test:

b) Create resourcegroup with storage and logical hostname Page 8 of 21

Sun Microsystems GmbH

#clresourcegroupcreatensscsapx1,sscsapx2pe2ora #clresourcecreatetHAStoragePlusgpe2orapzpools=pe2ora_storpAffinityOn=truepe2ora_stor #clreslogicalhostnamecreategpe2orahpe2oralhNsc_ipmp0@sscsapx1,sc_ipmp0@sscsapx2pe2ora_lh

c) Enable and test Resourcegroup

#clresourcegrouponlinepe2ora #clresourcegroupswitchnsscsapx2pe2ora #clresourcegroupswitchnsscsapx1pe2ora

SAP System Central Services (SCS) a) Create SVM Diskset and cluster devicegroup Check if /kernel/drv/md.conf identical on both cluster nodes
#cldevicelistv #metasetspe2scs_storahsscsapx1sscsapx2 #cldevicegroupshow #cldevicegroupstatus #cldevicegroupsetp<name>=<value>pe2scs_stor #cldevicegroupswitchnsscsapx2pe2scs_stor #cldevicegroupswitchnsscsapx1pe2scs_stor

(FYI:Justtoreview;lookford11) (FYI:createsdevicegrouppe2scs_stor) (FYI:Toviewexistingdevicegroups) (FYI:Toviewstatusofdevicegroups) (Optional.Nothingmandatoryhere!!)

Test cluster devicegroup:

b) Create global filesystems (UFS)

#metasetspe2scs_stora/dev/did/rdsk/d11 #metainitspe2scs_stord100pe/dev/did/rdsk/d1110G #metainitspe2scs_stord101p/dev/did/rdsk/d11s020G #metainitspe2scs_stord102p/dev/did/rdsk/d11s0all #newfs/dev/md/pe2scs_stor/rdsk/d100 #newfs/dev/md/pe2scs_stor/rdsk/d101 #newfs/dev/md/pe2scs_stor/rdsk/d102 #mkdirp/sapmnt/PE2 (tobeexecutedonbothclusternodes) #mkdirp/usr/sap/PE2 (tobeexecutedonbothclusternodes) #mkdirp/usr/sap/trans (tobeexecutedonbothclusternodes)

c) Add entries to /etc/vfstab (both cluster nodes):

/dev/md/pe2scs_stor/dsk/d100/dev/md/pe2scs_stor/rdsk/d100/sapmnt/PE2ufs2yesglobal,logging /dev/md/pe2scs_stor/dsk/d101/dev/md/pe2scs_stor/rdsk/d101/usr/sap/PE2ufs2yesglobal,logging /dev/md/pe2scs_stor/dsk/d102/dev/md/pe2scs_stor/rdsk/d102/usr/sap/transufs2yesglobal,logging #sccheck #mount/sapmnt/PE2 #mount/usr/sap/PE2 #mount/usr/sap/trans

d) Create resourcegroup with storage, logical hostname and NFS server

#clresourcegroupcreatensscsapx1,sscsapx2pe2scs #clresourcecreatetHAStoragePlusgpe2scspGlobalDevicePaths=pe2scs_storpAffinityOn=true pe2scs_stor #clreslogicalhostnamecreategpe2scshpe2scslhNsc_ipmp0@sscsapx1,sc_ipmp0@sscsapx2pe2scs_lh #clresourcegroupsetpPathprefix=/usr/sap/PE2pe2scs #mkdirp/usr/sap/PE2/SUNW.nfs shareFnfsorw/sapmnt/PE2 shareFnfsorw/usr/sap/trans #clresourcecreategpe2scstSUNW.nfspResource_dependencies=pe2scs_storpe2scs_nfs

Add these two entries (shares) to /usr/sap/PE2/SUNW.nfs/dfstab.pe2scs_nfs

e) Enable and test Resourcegroup

#clresourcegrouponlinepe2scs #clresourcegroupswitchnsscsapx2pe2scs #clresourcegroupswitchnsscsapx1pe2scs

SAP Central Instance a) Create Resourcegroup with logical hostname

#clresourcegroupcreatensscsapx1,sscsapx2pe2ci #clreslogicalhostnamecreategpe2cihpe2cilhNsc_ipmp0@sscsapx1,sc_ipmp0@sscsapx2pe2ci_lh

b) Enable and test Resourcegroup

#clresourcegrouponlinepe2ora #clresourcegroupswitchnsscsapx2pe2ci #clresourcegroupswitchnsscsapx1pe2ci

SAP Replicated Enqueue Server a) Create Resourcegroup with logical hostname

#clresourcegroupcreatensscsapx2,sscsapx1pe2ers

Page 9 of 21

Sun Microsystems GmbH

#clreslogicalhostnamecreategpe2ershpe2erslhNsc_ipmp0@sscsapx2,sc_ipmp0@sscsapx1pe2ers_lh

b) Enable and test Resourcegroup

#clresourcegroupswitchnsscsapx1pe2ci #clresourcegroupswitchnsscsapx2pe2ci

4.4 - SAP/Oracle Installation

SAP/Oracle standard installation Perform a standard installation of SAP NetWeaver Portal 7.0 and Oracle Database 10g on the prepared storage and logical hosts. Any special configuration needs regarding failover, Sun Cluster or enqueue replication will be described/done not here, but in the following steps. Oracle DB on node sscsapx1: Name PE2, configured on logical hostname pe2oralh SAP SCS on node sscsapx1: SAP SID PE2, Instance No. 01 (instance name SCS01), configured on logical hostname pe2scslh SAP CI on node sscsapx1: SAP SID PE2, Instance No. 00 (instance name JC00), installed on logical hostname pe2cilh SAP Replicated Enqueue Server installation Perform a standard installation of the SAP Replicated Enqueue Server on the prepared storage and logical hosts. Any special configuration needs regarding failover, Sun Cluster or Enqueue replication will be described/done not here, but in the following steps. SAP Replicated Enqueue Server on node sscsapx2: SAP SID PE2, Instance No. 11 (instance name ERS11), installed on logical hostname pe2erslh As of today (January 2009, SAP NetWeaver 7.0 EHP 1, SPS02) there is no SAP tool like sapinst to install the SAP Replicated Enqueue Server. Instead, it has to be done manually. You find the guide in the SAP documentation (http://help.sap.com SAP NetWeaver 7.0 Library Look for the chapter "Setting Up the Replication Server"). Here is a summary. It is simplified and adopted to the special needs of the cluster configuration: Directory structure Create the instance-specific subdirectories in the global filesystem /usr/sap/PE2:
/usr/sap/PE2/ERS11/exe /usr/sap/PE2/ERS11/log /usr/sap/PE2/ERS11/data /usr/sap/PE2/ERS11/work

Instance profile Create the instance profile /sapmnt/PE2/profile/PE2_ERS11_pe2erslh. You find the file for reuse in the appendix (Chapter ERS11 instance profile). Please pay attention to comments given in the file. Some changes have to be done (compared to the SAP documentation) because of Sun Cluster specifics. Start profile You may want to create a start profile. This is optional, because Sun Cluster will start the server-process directly (by executing the command: /usr/sap/PE2/SYS/exe/run/enrepserver pf=/sapmnt/PE2/profile/PE2_ERS11_pe2erslh). In this way, no start profile is being used. If you want to start the server-process manually, you can do it in the same way. Just if you want to use startsap (startsap r3 ERS11 pe2erslh), you have to provide the start profile /sapmnt/PE2/profile/START_ERS11_pe2erslh. You find the file for reuse in the appendix (Chapter ERS11 start profile).

Disable sapcpe Copying the SAP Kernel from /sapmnt/PE2/exe to the instance-specific directories /usr/sap/PE2/<instancename>/exe does not make sense, because it's all on the global filesystem(s). Instead start directly from /usr/sap/PE2/SYS/exe/run (that is a softlink to /sapmnt/PE2/exe). Set these parameters in all start profiles and instance profiles:
DIR_CT_RUN=/usr/sap/PE2/SYS/exe/run DIR_EXECUTABLE=/usr/sap/PE2/SYS/exe/run

Check and adjust environment of OS users pe2adm and orape2 to point to the correct directory
PATH=...:/usr/sap/PE2/SYS/exe/run:... DIR_LIBRARY=.../usr/sap/PE2/SYS/exe/run:... LD_LIBRARY_PATH=.../usr/sap/PE2/SYS/exe/run:...

In the instance start profiles, disable all of the sapcpe calls like this one:
Execute_00=immediate$(DIR_CT_RUN)/sapcpe$(FT_EXE)pf=$(_PF)$(_CPARG0)

Page 10 of 21

Sun Microsystems GmbH

Delete the existing (already copied) files in the instance-specific directories /usr/sap/PE2/<instancename>/ exe.

4.5 - Prepare SAP/Oracle switchover

Prepare both nodes (sscsapx1 and sscsapx2) to be capable to host the SAP/Oracle applications. If not done before (during the installation), please check if these configurations are available and identical on both cluster nodes. Home directories Relocate the home directories of the OS users pe2adm and orape2 onto global filesystems. Homedir of pe2adm: /usr/sap/PE2/pe2adm Homedir of orape2: /oracle/PE2/orape2 Create OS users and groups Create the accounts on both hosts, if missing.

#groupaddg100sapsys #groupaddg101dba #groupaddg102oper #groupaddg103sapinst #useraddg100G101,102,103u110d/usr/sap/PE2/pe2adms/bin/cshpe2adm #useraddg101G102,103u111d/oracle/PE2/orape2s/bin/cshorape2

Solaris project Check /etc/project for this entry:

PE2:200:SAPSystemPE2:orape2,pe2adm,root::process.maxsemnsems=(priv,2048,deny);project.maxsem ids=(priv,1024,deny);project.maxshmids=(priv,256,deny);project.maxshm memory=(priv,18446744073709551615,deny)

Definition of default project Check file /etc/user_attr for these entries:

pe2adm::::project=PE2 orape2::::project=PE2

Services Check file /etc/services for these entries:

sapdp003200/tcp#SAPSystemDispatcherPort sapdp013201/tcp#SAPSystemDispatcherPort sapdp113211/tcp#SAPSystemDispatcherPort sapgw003300/tcp#SAPSystemGatewayPort sapgw013301/tcp#SAPSystemGatewayPort sapgw113311/tcp#SAPSystemGatewayPort sapdp00s4700/tcp#SAPSystemDispatcherSecurityPort sapdp01s4701/tcp#SAPSystemDispatcherSecurityPort sapdp11s4711/tcp#SAPSystemDispatcherSecurityPort sapgw00s4800/tcp#SAPSystemGatewaySecurityPort sapgw01s4801/tcp#SAPSystemGatewaySecurityPort sapgw11s4811/tcp#SAPSystemGatewaySecurityPort

Hosts Check file /etc/hosts to contain all logical hosts and physical nodes. Mountpoints for failover filesystems Make sure, that the mountpoint /oracle/PE2 is available on all nodes. Java J2SE Install the most recent version of J2SE 1.4.2.x SDK on a global filesystem. Download self-extracting version von http://java.sun.com Install into directory /usr/sap/PE2/j2se Modify environment of user pe2adm: JAVA_HOME=/usr/sap/PE2/j2se Modify JC00 instance profile: jstartup/vm/home = /usr/sap/PE2/j2se Note: For Solaris x64 you need to download a version from http://java.sun.com/j2se/1.4.2/SAPsite/download.html. JCE policy files Download from http://java.sun.com/javase/downloads and extract to: Page 11 of 21

Sun Microsystems GmbH

/usr/sap/PE2/j2se/jre/lib/security/local_policy.jar /usr/sap/PE2/j2se/jre/lib/security/US_export_policy.jar

saposcol directory Create a local working directory /usr/sap/tmp for saposcol on both nodes (owner: root, group: sapsys, permissions: drwxrwxr-x). Oracle client Copy directory /oracle/client (from the Oracle-installation node) to the other nodes. Oracle inventory Relocate the Oracle inventory to shared storage. On the Oracle-installation node; Modify the entry in /var/opt/oracle/oraInst.loc: inventory_loc=/oracle/PE2/oraInventory Copy directory /var/opt/oracle to the other nodes Move directory /oracle/oraInventory to /oracle/PE2/oraInventory

4.6 - Test SAP/Oracle switchover

Check if SAP/Oracle is able to run on both cluster nodes. It makes sense to test application switchover at this point in time, without a fully completed Sun Cluster integration. Because in the case of any errors, you could be sure to have no impact of a potentially inaccurate cluster configuration. And you could focus your error analysis on the native SAP stuff. Start/Stop SAP and Oracle manually, relocate filesystems and IP adresses using Sun Cluster commands. To stop SAP, use following commands and OS user pe1adm:
#stopsapr3JC00pe2cilh #stopsapr3SCS01pe2scslh #stopsapr3ERS11pe2erslh (orkilltheenrepserverprocess,ifyouhavenotprovidedastartprofileforERS11) #sqlplus Enterusername:/assysdba SQL>shutdown #lsnrcltstop

To stop Oracle, use following commands and OS user orape2:

To start Oracle, use following commands and OS user orape2:

#lsnrcltstart #sqlplus Enterusername:/assysdba SQL>startup

To start SAP, use following commands and OS user pe1adm:

#startsapr3SCS01pe2scslh #startsapr3JC00pe2cilh #startsapr3ERS11pe2erslh (or#/usr/sap/PE2/SYS/exe/run/enrepserverpf=/sapmnt/PE2/profile/PE2_ERS11_pe2erslh,ifyouhavenot providedastartprofileforERS11)

To switch resourcegroups (containing storage and IP) between the cluster nodes:
#clrgswitchn<targetnode><resourcegroup>

with <target-node> = sscsapx1 or sscsapx2 with <resourcegroup> = pe2scs, pe2ers, pe2ora or pe2ci

4.7 - Enable enqueue replication

You need to set some SAP parameters to enable the enqueue replication. For complete list of enqueueparameters please refer to the SAP documentation (http://help.sap.com SAP NetWeaver 7.0 Library Look for the chapter "Profile Parameter Settings for the Standalone Enqueue Server"). This overview shows the enqueue (and enqueue replication) related parameterisation of this example installation. All profiles (including all parameters) are listed in the appendix of this paper for reuse. Parameters for SCS01 Configured in the instance profile: Page 12 of 21

Sun Microsystems GmbH

ipc/shm_psize_34=0 enque/server/replication=true enque/serverinst=01 enque/table_size=4096 rdisp/enqname=$(rdisp/myname) enque/snapshot_pck_ids=100

Parameters for JC00 (and all other application instances) Configured in the default profile:
enque/deque_wait_answer=TRUE

Parameters for ERS11 Nothing more to do, if you have reused the instance profile from the appendix of this paper.

4.8 - Test the enqueue replication

It makes sense to test the enqueue replication at this point in time, without a fully completed Sun Cluster integration. Because in the case of any errors, you could be sure to have no impact of a potentially inaccurate cluster configuration. And you could focus your error analysis on the native SAP stuff. For more infos, please refer to the SAP documentation (http://help.sap.com SAP NetWeaver 7.0 Library Look for the chapter "Replication Server: Check Installation"). Execute the following SAP-related commands (ensmon, enqt, ) with OS user pe2adm. Attention: Take care to execute the commands on the correct nodes as shown below. Some of the commands are working on both sides (Enqueue and Replicated Enqueue), others not. It may be astonishingly to create the enqueue table test-entries on the Replicated Enqueue Server side (and not on the Enqueue Server side). But this is NOT a typo. 1) Switch applications to their preferred node Start the applications on their preferred node. Oracle DB, SCS01, JC00 on sscsapx1. ERS11 on sscsapx2. See chapter Test SAP/Oracle switchover on how to start, stop and switch the applications (in this installation phase, without having a complete cluster integration). 2) Check if replication is active On sscsapx2 (the current node hosting ERS11, containing the Replicated Enqueue Server):
#ensmonpf=/sapmnt/PE2/profile/PE2_ERS11_pe2erslh2

The output should look like this:

Trytoconnecttohostpe2scslhservicesapdp01 getreplinforequestexecutedsuccessfully Replicationisenabledinserver,repl.serverisconnected Replicationisactive

3) Check the enqueue table ID On sscsapx1 (the current node hosting SCS01, containing the Enqueue Server):
#enqtpf=/sapmnt/PE2/profile/PE2_SCS01_pe2scslh97 #enqtpf=/sapmnt/PE2/profile/PE2_ERS11_pe2erslh97

On sscsapx2 (the current node hosting ERS11, containing the Replicated Enqueue Server): The output should be identical and look like this:

REQ EnqId:EnqTabCreaTime/RandomNumber=14.01.200918:44:311231955071/2868 ReqOrdatSrv:TimeInSecs/ReqNumberThisSec=18.01.200911:26:321232274392/1

4) Create enqueue table entries On sscsapx2 (the current node hosting ERS11, containing the Replicated Enqueue Server):
#enqtpf=/sapmnt/PE2/profile/PE2_ERS11_pe2erslh1120

This command will create 20 dummy entries in the table. 5) View enqueue table entries Page 13 of 21

Sun Microsystems GmbH On sscsapx1 (the current node hosting SCS01, containing the Enqueue Server):
#enqtpf=/sapmnt/PE2/profile/PE2_SCS01_pe2scslh20 #enqtpf=/sapmnt/PE2/profile/PE2_ERS11_pe2erslh20

On sscsapx2 (the current node hosting ERS11, containing the Replicated Enqueue Server): The output should be identical and contain 20 entries like this (and maybe some others):
000ENQTENQ_PERFXenqueueperformancetest11111111000

6) Perform switchover Stop the instance SC01 on node sscsapx1 and start it on node sscsapx2. See chapter Test SAP/Oracle switchover on how to start, stop and switch the applications (in this installation phase, without having a complete cluster integration). 7) Check the success The Enqueue Server should have stopped the Replicated Enqueue Server on node sscsapx2. Check if the OS process is gone. It is the process command name er.sapPE2_ERS11, if you have started ERS11 with startssap (having a start profile). Otherwise, if you have started the Replicated Enqueue Server directly, the process command name is enrepserver. Check if the enqueue table ID is still the same from before the switchover. Repeat step 3, solely regarding SCS01 (because ERS11 should be stopped). Remember that the node has changed from sscsapx1 to sscsapx2 ! Check if the 20 enqueue table entries still exist. Repeat step 4, solely regarding SCS01 (because ERS11 should be stopped). Remember that the node has changed from sscsapx1 to sscsapx2 ! Note. In the case, that the replication does not work correctly, you can use these files foe debugging: Look for the developer traces dev_en* in the directories /usr/sap/PE2/SCS01/work and /usr/sap/ERS11/SCS01/work. Attention: If you have started the Replicated Enqueue Server directly (not using startsap and a start profile), then you will find the trace files of the Replicated Enqueue Server in the home directory of user pe2adm. Not yet, but in the final configuration, Sun Cluster will also start the Enqueue Server directly (not using startsap and a start profile). Thus, you will find also these trace files in the home directory of user pe2adm. If needed, increase the trace level. Set parameter rdisp/TRACE = 3 in the instance profiles of SCS01 and ERS11.

4.9 - Cluster data services configuration (part II)

This step will finalize the integration of SAP/Oracle into the Sun Cluster. Provide start/stop script for SAP instances Provide these two scripts (owner: pe2adm, group: sapsys, permissions: -rwxrr--):
/usr/sap/PE2/pe2adm/startsap_cluster.sh /usr/sap/PE2/pe2adm/stopsap_cluster.sh

You find them in the appendix. Sun Cluster will call them to start and stop the SAP instances SCS01 and JC00. Oracle database a) Create Oracle monitor-user and table Start the oracle database. Use OS user orape2 to connect to the db:

#sqlplus"/assysdba" sql>createuserhaoracleidentifiedbyhaoracle; sql>alteruserhaoracledefaulttablespacesystemquota1monsystem; sql>grantselectonv_$sysstattohaoracle; sql>grantselectonv_$archive_desttohaoracle; sql>grantcreatesessiontohaoracle; sql>grantcreatetabletohaoracle; sql>exit;

b) Resource for Oracle database Stop database, if running. Then create the cluster resource:

#clresourcecreategpe2ora\ tSUNW.oracle_server\ pResource_project_name=PE2\

Page 14 of 21

Sun Microsystems GmbH

pConnect_string=haoracle/haoracle\ pORACLE_SID=PE2\ pORACLE_HOME=/oracle/PE2/102_64\ pAlert_log_file=/oracle/PE2/saptrace/background/alert_PE2.log\ pRestart_type=RESOURCE_RESTART\ presource_dependencies=pe2ora_stor\ pe2ora_db

c) Resource for Oracle listener Stop listener, if running. Then create the cluster resource:

#clresourcecreategpe2ora\ tSUNW.oracle_listener\ pResource_project_name=PE2\ pLISTENER_NAME=LISTENER\ pORACLE_HOME=/oracle/PE2/102_64\ presource_dependencies=pe2ora_stor\ pe2ora_lsnr

SAP System Central Services (SCS) a) Remove the Enqueue Server from instance start Disable the startup in the start profile "/sapmnt/PE2/profile/START_SCS01_pe2scslh". Remove these lines in the section Start SAP locking service:
_EN=en.sap$(SAPSYSTEMNAME)_$(INSTANCE_NAME) Execute_03=localrmf$(_EN) Execute_04=locallnsf$(DIR_EXECUTABLE)/enserver$(FT_EXE)$(_EN) Start_Program_01=local$(_EN)pf=$(DIR_PROFILE)/PE2_SCS01_pe2scslh

Note. You can start the Enqueue Server directly (and Cluster will do it in this way): b) Resource for Enqueue Server Stop instance SCS01, if running. Then create the cluster resource:

#/usr/sap/PE2/SCS01/exe/enserverpf=/sapmnt/PE2/profile/PE2_SCS01_pe2scslh

#clresourcecreatedgpe2scstSUNW.sapenq\ pResource_project_name=PE2\ penqueue_profile=/sapmnt/PE2/profile/PE2_SCS01_pe2scslh\ penqueue_server=/usr/sap/PE2/SYS/exe/run/enserver\ psap_user=pe2adm\ penqueue_instance_number=01\ presource_dependencies=pe2scs_stor,pe2ora_db{ANY_NODE},pe2ora_lsnr{ANY_NODE}\ pe2scs_enq

c) Resource for SCS01 (remaining Message Server) Stop instance SCS01, if running. Then create the cluster resource:

#clresourcecreatedgpe2scstSUNW.sapscs\ pResource_project_name=PE2\ psap_sid=PE2\ psap_instance_number=01\ psap_instance_name=SCS01\ pmsg_server_port=3901\ pscs_startup_script=/usr/sap/PE2/pe2adm/startsap_cluster.sh\ pscs_shutdown_script=/usr/sap/PE2/pe2adm/stopsap_cluster.sh\ presource_dependencies=pe2scs_stor,pe2ora_db{ANY_NODE},pe2ora_lsnr{ANY_NODE}\ pe2scs_ms

SAP Central Instance a) Resource for JC00 Stop instance JC00, if running. Then create the cluster resource:

#clresourcecreatedgpe2citSUNW.sapwebas\ pResource_project_name=PE2\ psap_sid=PE2\ psap_instance_number=00\ psap_instance_name=JC00\ psap_instance_type=J2EE\ pwebas_startup_script=/usr/sap/PE2/pe2adm/startsap_cluster.sh\ pwebas_shutdown_script=/usr/sap/PE2/pe2adm/stopsap_cluster.sh\ presource_dependencies=pe2scs_stor{ANY_NODE},pe2ora_db{ANY_NODE},\ pe2ora_lsnr{ANY_NODE},pe2scs_enq_1{ANY_NODE},pe2scs_ms{ANY_NODE}\ pe2ci_jc00

Page 15 of 21

Sun Microsystems GmbH

SAP Replicated Enqueue Server a) Resource for Replicated Enqueue Server (instance ERS11) Stop Replicated Enqueue Server (Instance ERS11), if running. Then create the cluster resource:

#clresourcecreatedgpe2erstSUNW.saprepl\ preplica_profile=/sapmnt/PE2/profile/PE2_ERS11_pe2erslh\ preplica_server=/usr/sap/PE2/SYS/exe/run/enrepserver\ psap_user=pe2adm\ presource_dependencies=pe2scs_stor{ANY_NODE},pe2scs_enq{ANY_NODE}\ pe2ers_enq

Group affinities Configure these resourcegroup affinities to reflect the special failover requirements of the SAP enqueue replication concept.
#clresourcegroupsetpRG_affinities=+pe2erspe2scs

The weak positive affinity setting ensures that, in case of failover, the group pe2scs (SCS01, including Enqueue Server) fails over to the node where the group pe2ers (ERS11, Replicated Enqueue Server) is running.
#clresourcegroupsetpRG_affinities=pe2scspe2ers

The strong negative affinity setting: The group pe2ers (ERS11, Replicated Enqueue Server) is never brought online nor allowed to remain online on a node on which the group pe2scs (SCS01, including Enqueue Server) is online.

4.10 - Enable and test the cluster

This chapter does not intend to describe a complete cluster acceptance test. Just the minimum functional test is shown here. Enable the resources Enable the cluster resources, configured in the step before:
#clresourceenablepe2ora_lsnr #clresourceenablepe2ora_db #clresourceenablepe2scs_enq #clresourceenablepe2scs_ms #clresourceenablepe2ci_jc00 #clresourceenablepe2ers_enq

Switchover Test if every resourcegroup can be switched over to every node.

#clrgswitchn<targetnode><resourcegroup>

with <target-node> = sscsapx1 or sscsapx2 with <resourcegroup> = pe2scs, pe2ers, pe2ora or pe2ci Check if the evacuation of the Replicated Enqueue Server is working, as specified in the section Group affinities above. SAP enqueue replication Test the enqueue replication again, as already described in the chapter Test the enqueue replication. Verify that JC00 (as enqueue client) is not doing a restart, after group pe2scs (SCS01, including Enqueue Server) has done an switchover or failover. Check the enqueue clients Check, if the JAVA enqueue clients (all SAP application instances) will continue to work without a restart, after a SCS01 failover or switchover.

Page 16 of 21

Sun Microsystems GmbH

5 - Appendix
5.1 - Default profile
File /sapmnt/PE2/profile/DEFAULT.PFL
SAPSYSTEMNAME=PE2 SAPGLOBALHOST=pe2scslh # #SAPCentralServiceInstanceforJ2EE # j2ee/scs/host=pe2scslh j2ee/scs/system=01 j2ee/ms/port=3901 SAPDBHOST=pe2oralh j2ee/dbtype=ora j2ee/dbname=PE2 j2ee/dbhost=pe2oralh DIR_PUT=/usr/sap/$(SAPSYSTEMNAME)/put #Enableenqueuereplicationforallclients(=appinstances) enque/deque_wait_answer=TRUE

5.2 - ERS11 start profile

File /sapmnt/PE2/profile/START_ERS11_pe2erslh
SAPSYSTEMNAME=PE2 #SeecommentsintheERS11instanceprofile. SAPSYSTEM=01 INSTANCE_NAME=ERS11 #SeecommentsintheERS11instanceprofile. DIR_CT_RUN=/usr/sap/PE2/SYS/exe/run DIR_EXECUTABLE=/usr/sap/PE2/SYS/exe/run SETENV_00=PATH=$(DIR_INSTANCE)/exe:%(PATH) SETENV_01=LD_LIBRARY_PATH=$(DIR_EXECUTABLE) _PF=$(DIR_PROFILE)/PE2_ERS11_pe2erslh # #Disabled!SeecommentsintheERS11instanceprofile. # #_CPARG0=list:$(DIR_EXECUTABLE)/ers.lst #Execute_00=immediate$(DIR_EXECUTABLE)/sapcpe$(FT_EXE)$(_CPARG0)pf=$(_PF) #OS_UNICODE=uc # #StartReplicatedEnqueueServer # _ER=er.sap$(SAPSYSTEMNAME)_$(INSTANCE_NAME) Execute_01=immediatermf$(_ER) Execute_02=locallnsf$(DIR_EXECUTABLE)/enrepserver$(_ER) #NotstartingitintheSAPdescribedway. #SeecommentsintheERS11instanceprofile. #Restart_Program_00=local$(_ER)pf=$(_PF)NR=01 Restart_Program_00=local$(_ER)pf=$(_PF)

5.3 - ERS11 instance profile

File /sapmnt/PE2/profile/PE2_ERS11_pe2erslh
SAPSYSTEMNAME=PE2

Page 17 of 21

Sun Microsystems GmbH

#!!ATTENTION!! #Eventhisinstancewasinstalledon#11andinstancenameisERS11, #youhavetosettheinternalnumberto01.ThatistheinstancenumberofSCS01. #Thisassuresthatbothinstances(SAPEnqueueServerANDSAPReplicatedEnqueueServer) #areusingthesameinternalmemoryrepresentationfortheLOCKtable.Thusbeingableto #synchronizeinthecaseofafailover(otherwisethesynchronisationwillfail!) #TheSAPdocumentationisdescribingadifferentway,whichcomesobviouslytothesame #result.Itstartstheprocesswithinthestartprofilewiththisparameterisation: #Restart_Program_00=local$(_ER)pf=$(_PF)NR=01(notetheendingNR=01). #ButastheclusterstartstheReplicatedEnqueueprocessdirectly(withoutusingthe #startprofile),wehavetosetthisparameterintheinstanceprofile. SAPSYSTEM=01 INSTANCE_NAME=ERS11 #Startdirectlyfrom/sapmnt/PE2/exe(thedirsspecifiedherearesoftlinksto/sapmnt/PE2/exe). #Nosapcpetoinstancespecificdirsmakesense,becauseit'sallontheglobalfilesystem(s). DIR_CT_RUN=/usr/sap/PE2/SYS/exe/run DIR_EXECUTABLE=/usr/sap/PE2/SYS/exe/run SAPLOCALHOST=pe2erslh OS_UNICODE=uc #InfosaboutSAPEnqueueServer(runnninginSCS01) rdisp/enqname=$(rdisp/myname) enque/process_location=REMOTESA enque/serverinst=01 enque/serverhost=pe2scslh #Generalinstanceparameter ipc/shm_psize_10=130000000 ipc/shm_psize_40=115000000

5.4 - SCS01 start profile

File /sapmnt/PE2/profile/START_SCS01_pe2scslh
SAPSYSTEMNAME=PE2 SAPSYSTEM=01 INSTANCE_NAME=SCS01 #DIR_CT_RUN=$(DIR_EXE_ROOT)/run #DIR_EXECUTABLE=$(DIR_INSTANCE)/exe DIR_CT_RUN=/usr/sap/PE2/SYS/exe/run DIR_EXECUTABLE=/usr/sap/PE2/SYS/exe/run SAPLOCALHOST=pe2scslh DIR_PROFILE=$(DIR_INSTALL)/profile _PF=$(DIR_PROFILE)/PE2_SCS01_pe2scslh SETENV_00=LD_LIBRARY_PATH=$(DIR_LIBRARY):%(LD_LIBRARY_PATH) SETENV_01=SHLIB_PATH=$(DIR_LIBRARY):%(SHLIB_PATH) SETENV_02=LIBPATH=$(DIR_LIBRARY):%(LIBPATH) # #CopySAPExecutables!!!Disabled!!! # #_CPARG0=list:$(DIR_CT_RUN)/scs.lst #Execute_00=immediate$(DIR_CT_RUN)/sapcpe$(FT_EXE)pf=$(_PF)$(_CPARG0) #OS_UNICODE=uc # #StartSAPmessagingservice # _MS=ms.sap$(SAPSYSTEMNAME)_$(INSTANCE_NAME) Execute_01=localrmf$(_MS) Execute_02=locallnsf$(DIR_EXECUTABLE)/msg_server$(FT_EXE)$(_MS) Start_Program_00=local$(_MS)pf=$(DIR_PROFILE)/PE2_SCS01_pe2scslh # #StartSAPlockingservice #!!!Disabled!!!WillbestarteddirectlybySunCluster # #_EN=en.sap$(SAPSYSTEMNAME)_$(INSTANCE_NAME) #Execute_03=localrmf$(_EN) #Execute_04=locallnsf$(DIR_EXECUTABLE)/enserver$(FT_EXE)$(_EN) #Start_Program_01=local$(_EN)pf=$(DIR_PROFILE)/PE2_SCS01_pe2scslh

Page 18 of 21

Sun Microsystems GmbH

5.5 - SCS01 instance profile

File /sapmnt/PE2/profile/PE2_SCS01_pe2scslh
SAPSYSTEMNAME=PE2 SAPSYSTEM=01 INSTANCE_NAME=SCS01 #DIR_CT_RUN=$(DIR_EXE_ROOT)/run #DIR_EXECUTABLE=$(DIR_INSTANCE)/exe DIR_CT_RUN=/usr/sap/PE2/SYS/exe/run DIR_EXECUTABLE=/usr/sap/PE2/SYS/exe/run SAPLOCALHOST=pe2scslh OS_UNICODE=uc # #SAPMessagingServiceforJava # rdisp/msserv=0 rdisp/msserv_internal=3901 ms/standalone=1 ms/server_port_0=PROT=HTTP,PORT=81$$ # #SAPLockingService # enque/serverinst=01 enque/table_size=4096 rdisp/enqname=$(rdisp/myname) enque/snapshot_pck_ids=100 enque/server/replication=true ipc/shm_psize_34=0 ipc/shm_psize_10=130000000 ipc/shm_psize_40=115000000

5.6 - JC00 start profile

File /sapmnt/PE2/profile/START_JC00_pe2cilh
SAPSYSTEMNAME=PE2 SAPSYSTEM=00 INSTANCE_NAME=JC00 #DIR_CT_RUN=$(DIR_EXE_ROOT)/run #DIR_EXECUTABLE=$(DIR_INSTANCE)/exe DIR_CT_RUN=/usr/sap/PE2/SYS/exe/run DIR_EXECUTABLE=/usr/sap/PE2/SYS/exe/run SAPLOCALHOST=pe2cilh DIR_PROFILE=$(DIR_INSTALL)/profile _PF=$(DIR_PROFILE)/PE2_JC00_pe2cilh SETENV_00=LD_LIBRARY_PATH=$(DIR_LIBRARY):%(LD_LIBRARY_PATH) SETENV_01=SHLIB_PATH=$(DIR_LIBRARY):%(SHLIB_PATH) SETENV_02=LIBPATH=$(DIR_LIBRARY):%(LIBPATH) # #CopySAPExecutables!!!Disabled!!! # #_CPARG0=list:$(DIR_CT_RUN)/j2eeinst.lst #Execute_00=immediate$(DIR_CT_RUN)/sapcpe$(FT_EXE)pf=$(_PF)$(_CPARG0) # #CopySAPExecutables # #_CPARG1=list:$(DIR_CT_RUN)/igsexe.lst #Execute_01=immediate$(DIR_CT_RUN)/sapcpe$(FT_EXE)pf=$(_PF)$(_CPARG1) # #StartJavaapplicationserver # _JC=jc.sap$(SAPSYSTEMNAME)_$(INSTANCE_NAME) Execute_02=localrmf$(_JC) Execute_03=locallnsf$(DIR_EXECUTABLE)/jcontrol$(FT_EXE)$(_JC) Start_Program_00=local$(_JC)pf=$(_PF) # #Startinternetgraphicsserver # _IG=ig.sap$(SAPSYSTEMNAME)_$(INSTANCE_NAME) Execute_04=localrmf$(_IG)

Page 19 of 21

Sun Microsystems GmbH

Execute_05=locallnsf$(DIR_EXECUTABLE)/igswd_mt$(_IG) Start_Program_01=local$(_IG)mode=profilepf=$(_PF)

5.7 - JC00 instance profile

File /sapmnt/PE2/profile/PE2_JC00_pe2cilh
SAPSYSTEMNAME=PE2 SAPSYSTEM=00 INSTANCE_NAME=JC00 #DIR_CT_RUN=$(DIR_EXE_ROOT)/run #DIR_EXECUTABLE=$(DIR_INSTANCE)/exe DIR_CT_RUN=/usr/sap/PE2/SYS/exe/run DIR_EXECUTABLE=/usr/sap/PE2/SYS/exe/run SAPLOCALHOST=pe2cilh jstartup/trimming_properties=off jstartup/protocol=on jstartup/vm/home=/usr/sap/PE2/j2se jstartup/max_caches=500 jstartup/release=700 jstartup/instance_properties=$(jstartup/j2ee_properties):$(jstartup/sdm_properties) j2ee/dbdriver=/oracle/client/10x_64/instantclient/ojdbc14.jar igs/listener/rfc/disable=1 PHYS_MEMSIZE=512 exe/saposcol=$(DIR_CT_RUN)/saposcol # #Jcontrol:MigratedProfileParameter #createatThuJun1215:06:302008 # j2ee/instance_id=ID0016265 #

5.8 - Script startsap_cluster.sh

File /usr/sap/PE2/pe2adm/startsap_cluster.sh
#!/bin/sh ######################pleasecustomize....####################### HOME=`getentpasswdpe2adm|cutd:f6s` .$HOME/.profile INSTANCE=$2 LOGICAL_HOST=`unamen` if["$INSTANCE"="SCS01"];then LOGICAL_HOST=pe2scslh; fi if["$INSTANCE"="JC00"];then LOGICAL_HOST=pe2cilh; fi SAPSID=PE2 STARTSAP="/usr/sap/${SAPSID}/SYS/exe/run/startsap" #######################endofcustomizing....####################### SCRIPTNAME=`basename$0` LOG_FILE=/tmp/${SCRIPTNAME}_${INSTANCE}.log exec>>$LOG_FILE echo"================================================================================" echo"`date`:Script$SCRIPTNAMEstartedwithoptions$*..." GREP=/usr/bin/grep NEWTASK=/usr/bin/newtask PS=/usr/bin/ps SLEEP=/usr/bin/sleep INSTID=`echo$INSTANCE|sed's/^[AZaz]*//g'` #start$INSTANCE....

Page 20 of 21

Sun Microsystems GmbH

echo"execute$STARTSAP$INSTANCE$LOGICAL_HOST" $NEWTASK$STARTSAP$INSTANCE$LOGICAL_HOST

5.9 - Script stopsap_cluster.sh

File /usr/sap/PE2/pe2adm/stopsap_cluster.sh
#!/bin/sh ######################pleasecustomize....####################### HOME=`getentpasswdpe2adm|cutd:f6s` .$HOME/.profile INSTANCE=$2; LOGICAL_HOST=`unamen` if["$INSTANCE"="SCS01"];then LOGICAL_HOST=pe2scslh; fi if["$INSTANCE"="JC00"];then LOGICAL_HOST=pe2cilh; fi SAPSID=PE2 STOPSAP="/usr/sap/${SAPSID}/SYS/exe/run/stopsap" SAPCCMSR="/usr/sap/${SAPSID}/SYS/exe/run/sapccmsr" #######################endofcustomizing....####################### SCRIPTNAME=`basename$0` LOG_FILE=/tmp/${SCRIPTNAME}_${INSTANCE}.log exec>>$LOG_FILE echo"================================================================================" echo"`date`:Script$SCRIPTNAMEmitOptionen$*gestartet..." ######################stopccmsr###################### echo"execute${SAPCCMSR}stopj2eepf=/usr/sap/${SAPSID}/SYS/profile/${SAPSID}_${INSTANCE}_$ {LOGICAL_HOST}" ${SAPCCMSR}stopj2eepf=/usr/sap/${SAPSID}/SYS/profile/${SAPSID}_${INSTANCE}_${LOGICAL_HOST} ######################stop$INSTANCE....###################### echo"execute$STOPSAPstartupsrv$INSTANCE2$LOGICAL_HOST" $STOPSAPstartupsrv$INSTANCE$LOGICAL_HOST echo"execute$STOPSAP$INSTANCE$LOGICAL_HOST" $STOPSAP$INSTANCE$LOGICAL_HOST #IGSneedssometimeforstoppinallprecesses,sowait... if["`pgrepf/usr/sap/${SAPSID}/${INSTANCE}/exe/igs`"!=""] then /usr/bin/sleep30 fi ######################stopjlaunchprocesses###################### INSTANCEtemp=`echo${INSTANCE}|awk'/^D[09][09]$/{print$1}'` if["$INSTANCEtemp"];then echo"killingjlaunchprocesses" /usr/bin/pkillf/usr/sap/${SAPSID}/${INSTANCE}/exe/jlaunch fi ######################cleanupIPC###################### instance_nr=`echo$INSTANCE|sed's/[AZ]*//'` echo"/usr/sap/${SAPSID}/SYS/exe/run/cleanipc$instance_nrremove" /usr/sap/${SAPSID}/SYS/exe/run/cleanipc"$instance_nr"remove ######################stopsaposcol###################### #Note:Notreallycorrect.saposcolisnotinstancespecific #BestwouldbetostartitoutsideofSAPorClustercontrolassystemservice(SMF) echo"execute/usr/sap/${SAPSID}/SYS/exe/run/saposcolk" /usr/sap/${SAPSID}/SYS/exe/run/saposcolk echo"`date`:Script$SCRIPTNAMEfinished

Page 21 of 21