You are on page 1of 91

Table of Contents

Rac11gR1OnHPUX............................................................................................................................................1 1. *Introduction.......................................................................................................................................1 1.1. *What you need to know....................................................................................................1 1.1.1. Software required for install (Itanium or PA-RISC)..........................................1 1.1.2. Processor Model.................................................................................................1 1.2. *Installation steps ..........................................................................................................................................1 1.3. *Schematic....................................................................................................................................................2 1.3.1. Hardware/software configuration BEFORE Oracle software install............................................2 1.3.2. Hardware/software configuration AFTER Oracle software install..............................................2 1.4. *Installation Method.....................................................................................................................................3 2. *Prepare the cluster nodes for Oracle RAC.....................................................................................................3 2.1. *User Accounts and Software Directories.......................................................................................4 2.1.1. *Creating the OSDBA (DBA) Group..............................................................................4 2.1.2. *Creating an OSOPER Group (Optional) ........................................................................4 2.1.3. *Creating the Oracle Software Owner User....................................................................4 2.1.4. *Creating an Oracle Software Owner User.....................................................................5 2.1.5. *Modifying an Oracle Software Owner User..................................................................5 2.1.6. *Creating Software Directories.......................................................................................5 2.2. *SSH Setup...................................................................................................................................................6 2.2.1. *Checking Existing SSH Configuration on the System...............................................................6 2.2.2. Configuring SSH ...........................................................................................................................6 2.3. *Hardware Requirements and Preparation...................................................................................................7 2.3.1. *Hardware Requirements ..............................................................................................................7 2.4. *Kernel Parameters.......................................................................................................................................8 2.5. *Required HPUX Patches.............................................................................................................................8 2.5.1. *HPUX 11.23 Patches..................................................................................................................9 2.5.2. *HPUX 11.31 Patches..................................................................................................................9 2.6. *Network Requirements and Preparation.....................................................................................................9 2.6.1. *Network Ping Tests...................................................................................................................10 2.6.2. *Network Adapter configuration................................................................................................10 2.7. *Stage the Oracle Software.........................................................................................................................10 2.8. *Check Preparations...................................................................................................................................11 3. *Prepare the Shared Storage for Oracle RAC This section describes how to prepare the shared storage for Oracle RAC..................................................................................................................................................14 3.1. Configuring Raw Disk for Storage To configure raw disks for database file storage, follow the following steps:...............................................................................................................................14 4. Oracle Clusterware Installation and Configuration.......................................................................................15 4.1. Oracle Clusterware Install ..............................................................................................................16 5. Oracle Clusterware patching..........................................................................................................................24 5.1. Oracle Clusterware patching complete..........................................................................................31 6. Oracle ASM Home Software Install..............................................................................................................31 6.1. CVU check.....................................................................................................................................31 6.2. Oracle ASM Home Software Install..............................................................................................31 6.2.1. Oracle ASM software home has been installed.............................................................38 7. Oracle ASM Software Home Patching..........................................................................................................38 7.1. Start the OUI for Oracle ASM Software Home Patching..............................................................38 8. Oracle RAC Database Listener Creation.......................................................................................................45 8.1. Create Node specific network listeners ..........................................................................................45 8.2. Completed Node specific network listeners creation .....................................................................49 9. Oracle ASM Instance and diskgroup Creation..............................................................................................49 9.1. Create ASM Instance and add the +DATA and +FLASH diskgroups..........................................49 9.2. Completed creation of ASM Instance and add of the +DATA and +FLASH diskgroups............58 10. Oracle RAC Database Home Software Install.............................................................................................58 10.1. CVU check - Pre Database Install...............................................................................................58 10.2. Oracle RAC Database Home Software Install.............................................................................58 i

Table of Contents
Rac11gR1OnHPUX 10.2.1. Oracle RAC Database software home has been installed............................................66 11. Oracle RAC Software Home Patching........................................................................................................66 11.1. Using the installer to patch the Oracle RAC Software Home.....................................................66 12. Oracle RAC Database Creation...................................................................................................................73 12.1. use dbca to create the RAC database...........................................................................................73

ii

Rac11gR1OnHPUX

1. *Introduction
1.1. *What you need to know
For those of you starting an Oracle RAC install for the first time, all this will be new. For those of you with Oracle 10g RAC on HPUX experience, there are just a few changes. Disk requirements for Oracle Clusterware devices The storage requirement for the OCR & vote devices used by Oracle Clusterware has been increased to 280 MB (256 MB, plus an additional buffer to allow for fdisk variation in space allocation). Better support for IP address The previous release had some restrictions that could be easily worked around, with regard to IP addresses from the non-routable range. These restrictions have been removed in the Oracle Clusterware 11g version and the workarounds are no longer required.

1.1.1. Software required for install (Itanium or PA-RISC)


hpia64_11gR1_database_1of2.zip and hpia64_11gR1_database_2of2.zip hpux.parisc_11gR1_database_1of2.zip and hpux.parisc_11gR1_database_2of2.zip

1.1.2. Processor Model


This paper covers both HPUX(PA-RISC) and HPUX(IA64) installs. Please note that The OS Version of Oracle you install must match the OS Chip Version. The following HPUX versions are supported: HP-UX 11.31 Itanium HP-UX 11.31 PA-RISC HP-UX 11.23 Itanium HP-UX 11.23 PA-RISC

1.2. *Installation steps


To summarise the install steps are: Preparation Pre-reqs. to make sure the cluster is setup OK. Stage all the software on one node, typically Node1 Establish Oracle Clusterware Install the Oracle Clusterware (using the push mechanism to install on the other nodes in the cluster) Patch the Clusterware to the latest patchset Establish ASM Install an Oracle Software Home for ASM Rac11gR1OnHPUX 1

Patch the ASM Home to the latest patchset Create the Listeners Create the ASM Instances and initial Disk Group Establish RAC Database Install an Oracle Software Home for RAC Database Patch the RAC Database Home to the latest patchset Create the RAC Database Instances

1.3. *Schematic
The following is a schematic of the software & hardware layout of a 2 node RAC cluster. As explained in this document the actual number of LUNs required will vary depending on your mirroring requirements.

1.3.1. Hardware/software configuration BEFORE Oracle software install

1.3.2. Hardware/software configuration AFTER Oracle software install

Notice that there are a number of LUNs present to both machines from the shared storage. 4 of 500MB LUNs

1.2. *Installation steps

2 are used for the OCR devices (/dev/rdsk/c4t0d1,/dev/rdsk/c6t0d1) 2 are used for the vote devices (/dev/c4t0d2,/dev/c6t0d2) 12 of 15890MB LUNs 10 are used for the +DATA ASM diskgroup (/dev/rdsk/c4t0d[3-7],/dev/rdsk/c6t0d[3-7]) 2 are used for the +FLASH diskgroup. (/dev/rdsk/c4t0d10,/dev/rdsk/c6t0d10) Also notice there is a dedicated switch on the cluster interconnect.

1.4. *Installation Method


This document details one method of installing a 2-node Oracle 11gR1 RAC cluster on HPUX: Character Device: The Oracle Clusterware, ASM & Oracle Database Homes binaries are established on the local disk of each of the RAC nodes. The files required by Oracle Clusterware must be on Character Devices. The database data-files are on ASM. The local Homes for Oracle Clusterware, ASM and RAC take up a total of 6.5GB on each node's local disk filesystem. It should be noted that there are other installation configurations. This document does not cover these methods: OCFS: The Oracle Cluster, ASM & Oracle Database Homes are established on the local disk of each of the RAC nodes. The files required by Oracle Clusterware are on OCFS. The database data-files are also on OCFS. NFS: The Oracle Cluster, ASM & Oracle Database Homes are established on the local disk of each of the RAC nodes. The files required by Oracle Clusterware are on Block devices. The database data-files are also on NFS File Server. ServiceGuard: Various combinations of HP-ServiceGuard and Oracle Clusterware/ASM are supported for the Oracle 11gR1 Database. Installation details of these configurations are not covered in this document. There are other possible combinations although Oracle recommends one of the above methods. You cannot place the Clusterware devices on NFS File Server. NFS File Server is not covered here

2. *Prepare the cluster nodes for Oracle RAC


This installation routine presumes that you have a 2-node HPUX cluster. There are a number of items that require checking before the install commences. Getting this bit right will enhance your install experience. 1. User Accounts and Software Directories 2. SSH Setup 3. Hardware Requirements and Preparation 4. Kernel Parameters 5. Required HPUX Patches 6. Network Requirements and Preparation 7. Stage the Oracle Software 8. Check Preparations It is essential that these items are checked and are configured correctly before the install commences.

1.3.2. Hardware/software configuration AFTER Oracle software install

2.1. *User Accounts and Software Directories


2.1.1. *Creating the OSDBA (DBA) Group
You must create an OSDBA group in the following circumstances: An OSDBA group does not exist, for example, if this is the first installation of Oracle Database software on the system An OSDBA group exists, but you want to give a different group of operating system users database administrative privileges in a new Oracle installation To determine whether the OSDBA group exists, enter the following command:
# grep OSDBA_group_name /etc/group

If the OSDBA group does not exist or if you require a new OSDBA group, then create it as follows. In the following command, use the group name dba unless a group with that name already exists. The default OSDBA group name is dba.
# /usr/sbin/groupadd -g 201 dba

Check too whether the oinstall group exists. If it does not, then add this group too.
# /usr/sbin/groupadd -g 200 oinstall

2.1.2. *Creating an OSOPER Group (Optional)


Create an OSOPER group only if you want to identify a group of operating system users with a limited set of database administrative privileges (SYSOPER operator privileges). For most installations, it is sufficient to create only the OSDBA group. If you want to use an OSOPER group, then you must create it in the following circumstances: If an OSOPER group does not exist, for example, if this is the first installation of Oracle Database software on the system If an OSOPER group exists, but you want to give a different group of operating system users database operator privileges in a new Oracle installation If you require a new OSOPER group, then create it as follows. In the following command, use the group name oper unless a group with that name already exists.
# /usr/sbin/groupadd -g 202 oper

2.1.3. *Creating the Oracle Software Owner User


You must create an Oracle software owner user in the following circumstances: If an Oracle software owner user does not exist, for example, if this is the first installation of Oracle software on the system If an Oracle software owner user exists, but you want to use a different operating system user, with different group membership, to give database administrative privileges to those groups in a new Oracle Database installation To determine whether an Oracle software owner user named oracle exists, enter the following command:
# id oracle

2.1. *User Accounts and Software Directories

If the oracle user exists, then the output from this command is similar to the following:
uid=440(oracle) gid=200(oinstall) groups=201(dba),202(oper)

If the user exists, then determine whether you want to use the existing user or create another oracle user. If you want to use the existing user, then ensure that the user's primary group is the Oracle Inventory group and that it is a member of the appropriate OSDBA and OSOPER groups.

2.1.4. *Creating an Oracle Software Owner User


In the following procedure, use the user name oracle unless a user with that name already exists. If the Oracle software owner user does not exist or if you require a new Oracle software owner user, then create it as follows: 1.To create the oracle user, enter a command similar to the following:
# /usr/sbin/useradd -g oinstall -G dba[,oper] oracle

In this command: The -g option specifies the primary group, which must be the Oracle Inventory group, for example oinstall The -G option specifies the secondary groups, which must include the OSDBA group and if required, the OSOPER group 2.Set the password of the oracle user:
# passwd oracle

2.1.5. *Modifying an Oracle Software Owner User


If the oracle user exists, but its primary group is not oinstall or it is not a member of the appropriate OSDBA or OSOPER groups, then enter a command similar to the following to modify it. Specify the primary group using the -g option and any required secondary group using the -G option:
# /usr/sbin/usermod -g oinstall -G dba[,oper] oracle

2.1.6. *Creating Software Directories


On each of the nodes, issue the following commands as root, to create the required directories with the correct ownership and permissions:
# mkdir /var/opt/oracle # chown oracle:oinstall /var/opt/oracle # chmod 755 /var/opt/oracle # mkdir -p /opt/oracle/product/CRS # mkdir -p /opt/oracle/product/RDBMS # chown -R oracle:dba /opt/oracle # chmod -R 755 /opt/oracle

Make sure that the user environment variables are set correctly:
# export ORACLE_BASE=/opt/oracle/product # export ORACLE_HOME=$ORACLE_BASE/RDBMS # export ORA_CRS_HOME=$ORACLE_BASE/CRS

2.1.3. *Creating the Oracle Software Owner User

2.2. *SSH Setup


To configure SSH, you must first create RSA or DSA keys on each cluster node, and then copy all the keys generated on all cluster node members into an authorized keys file that is identical on each node. Note that the SSH files must be readable only by root and by the software installation user (oracle, crs, asm), as SSH ignores a private key file if it is accessible by others. When this is done, then start the SSH agent to load keys into memory. In the examples that follow, the RSA key is used. You must configure SSH separately for each Oracle software installation owner that you intend to use for installation.

2.2.1. *Checking Existing SSH Configuration on the System


To determine if SSH is running, enter the following command:
# pgrep sshd

If SSH is running, then the response to this command is one or more process ID numbers. In the home directory of the software owner that you want to use for the installation (crs, oracle), use the command ls -al to ensure that the .ssh directory is owned and writable only by the user. You need either an RSA or a DSA key for the SSH protocol. RSA is used with the SSH 1.5 protocol, while DSA is the default for the SSH 2.0 protocol. With OpenSSH, you can use either RSA or DSA. The instructions that follow are for SSH1. If you have an SSH2 installation, and you cannot use SSH1, then refer to your SSH distribution documentation to configure SSH1 compatibility or configure SSH2 with DSA.

2.2.2. Configuring SSH


The following steps are to be run by the software owner ("oracle" user in our case) On the First Node, then on the Second Node:
# mkdir ~/.ssh # chmod 755 ~/.ssh # /usr/bin/ssh-keygen -t rsa

Note that in this case, we will leave the passphrase empty. The identification information has been saved in /home/oracle/.ssh/id_rsa. The public key has been saved in /home/oracle/.ssh/id_rsa.pub. The contents of the id_rsa.pub file of both nodes need to be put into a file called /home/oracle/.ssh/authorized_keys on both nodes. Basically, the files are identical, so create the file on node 1 and copy it to the same location on node 2.
# # # # cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys ssh oracle@node2 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys chmod 644 ~/.ssh/authorized_keys scp ~/.ssh/authorized_keys node2:~/.ssh/authorized_keys

Now, check connectivity in each direction, from each of the cluster servers, first from node1 and then repeat from node2.
# ssh node1 date # ssh node2 date

At the end of this process, the public hostname for each member node should be registered in the known_hosts file for all other cluster member nodes.

2.2. *SSH Setup

2.3. *Hardware Requirements and Preparation


2.3.1. *Hardware Requirements
Cluster servers must all be configured and prepared sufficient to the following specifications: at least 1GB of physical RAM. Use either of the following commands to verify the amount of physical memory installed:
# /usr/contib/bin/machinfo | grep -i Memory # /usr/sbin/dmesg | grep "Physical:"

swap space equivalent to a multiple of the physical memory, according to the following situations. If RAM is less than 2GB, swap space should be double the size of RAM. If RAM is between 2GB and 8GB, then the recommended swap space should be equal to the RAM. Otherwise, if RAM is over 8GB, then allocate a swap space .75 times the size of the RAM. Use the following command to confirm the size of swap:
# /usr/sbin/swapinfo -a

ensure that there is at least 400MB free space in the /tmp directory. To verify the free space available in /tmp, issue the following command:
# bdf /tmp

the Oracle Clusterware home must have at least 650MB of available disk space. the Oracle database home should have at least 4GB of available space confirm that the OS-version is correct, and 64-bit, issue the follow commands:
# uname -a # /bin/getconf KERNEL_BITS

confirm that Async I/O is enabled (default for HPUX 11.31 and 11.23):
# ll /dev/async

the output should look something like:


# crw-rw-rw- 1 bin bin 101 0x000000 May 16 07:23 /dev/async

in order to successfully relink the Oracle executables (if required) after installation, ensure that the following symbolic links have been created:
# # # # # # # # # # cd ln ln ln ln ln ln ln ln ln /usr/lib -s /usr/lib/libX11.3 libX11.sl -s /usr/lib/libXIE.2 libXIE.sl -s /usr/lib/libXext.3 libXext.sl -s /usr/lib/libXhp11.3 libXhp11.sl -s /usr/lib/libXi.3 libXi.sl -s /usr/lib/libXm.4 libXm.sl -s /usr/lib/libXp.2 libXp.sl -s /usr/lib/libXt.3 libXt.sl -s /usr/lib/libXtst.2 libXtst.sl

make sure that each of the cluster nodes is set to the same date and time. Oracle strongly recommends using the Network Time Protocol feature of most operating systems for this purpose, with all nodes using the same reference Network Time Protocol server.

2.3. *Hardware Requirements and Preparation

2.4. *Kernel Parameters


Verify that the kernel parameters shown listed below are set correctly (higher values are allowed). Also, check the most recently published documentation for the Oracle Cluseterware and Database software for any changes. nproc => 4096 ksi_alloc_max => (nproc*8) executable_stack => 0 max_thread_proc => 1024 maxdsiz => 1073741824 (1 GB) maxdsiz_64bit => 2147483648 (2 GB) maxssiz => 134217728 (128 MB) maxssiz_64bit => 1073741824 (1 GB) maxuprc => ((nproc*9)/10) msgmap (11.23 only) => (msgtgl+2) msgmni => nproc msgseg => (nproc*4) (at least 32767) msgtgl => nproc ncsize => ninode+1024 nfile => (15*nproc+2048) (for db's with a high number of data files, assume (number of Oracle processes)*(number of Oracle data files)+2048) nflocks => nproc ninode => (8*nproc+2048) nkthread => (((nproc*7)/4)+16) semmni => nproc semmns => (semmni*2) semmnu => (nproc-4) semvmx => 32767 shmmax => size of physical memory or 1073741824, whichever is greater, must be equal to or greater than the size of the SGA shmmni => 512 shmseg => 120 vps_ceiling => 64 (up to 16384 for a large SGA) Modify the kernel parameter settings by using either the kcweb application
# /usr/sbin/kcweb -F

or by using the kmtune command line utility (kctune on Itanium):


# kmtune parameter>=value

List the parameters changed:


# kmtune -D

2.5. *Required HPUX Patches


To determine which operating system patches are installed, enter the following command:
# /usr/sbin/swlist -l patch

To determine if a specific operating system patch has been installed, enter the following command:
# /usr/sbin/swlist -l patch <patch_number>

2.4. *Kernel Parameters

To determine which operating system bundles are installed, enter the following command:
# /usr/sbin/swlist -l bundle

2.5.1. *HPUX 11.23 Patches


HPUX 11.23 with Sept 2004 or newer base, and Mar 2007 Patch bundle for HP-UX 11iV2-B.11.23.0703 PHKL_33025 file system tunables cumulative patch PHKL_34941 Improves Oracle Clusterware restart and diagnosis PHCO_32426 reboot(1M) cumulative patch PHCO_36744 LVM patch [replaces PHCO_35524] PHCO_37069 libsec cumulative patch PHCO_37228 libc cumulative patch) [replaces PHCO_36673] PHCO_38120 kernel configuration commands patch PHKL_34213 vPars CPU migr, cumulative shutdown patch PHKL_34989 getrusage(2) performance PHKL_36319 mlockall(2), shmget(2) cumulative patch) [replaces PHKL_35478] PHKL_36853 pstat patch PHKL_37803 mpctl(2) options, manpage, socket count) [replaces PHKL_35767] PHKL_37121 sleep kwakeup performance cumulative patch [replaces PHKL_35029] PHKL_34840 slow system calls due to cache line sharing PHSS_37947 linker + fdp cumulative patch) [replaces PHSS_35979] PHNE_37395 cumulative ARPA Transport patch

2.5.2. *HPUX 11.31 Patches


PHKL_37296 vfs module patch PHKL_37452 vm cumulative patch [replaces PHKL_35900, PHKL_35936] PHKL_37453 esdisk cumulative patch [replaces PHKL_36249] PHKL_37454 esctl cumulative patch [replaces PHKL_36248] PHCO_37476 libc cumulative patch PHCO_37807 Cumulative Patch for bcheckrc PHSS_37948 linker + fdp cumulative patch PHSS_37954 Integrity Unwind Library PHNE_35894 Networking commands cumulative patch

2.6. *Network Requirements and Preparation


You will need a minimum of 2 network adapters per machine, 1 adapter is the public interface. The other adapter, the private interface, is used for Oracle RAC node membership and inter-instance traffic. The private interface adapters should be connected to each other via a network switch which is used exclusively for the interconnect traffic. In 2 node environments do not be tempted to use a 'Crossover Cable'. Best practice is to use more than 2 adapters, Before installation of the Oracle software you take 2 of the adapters on the private interface and team them together to create what looks like a single, highly available adapter for the interconnect traffic. You can then take the remaining 3rd adapter and use it for the public interface; some customers even team their public adapters as well. In that case you would need to start with 4 adapters, 2 public & 2 private. Ensure you team the adapters before starting installation. Please be aware that even with 2-node cluster you must use a dedicated network switch on the interconnect, Crossover cables are not supported You will need a total of 3 IP addresses per node:

2.5. *Required HPUX Patches

The public IP address, which should be recorded in hosts file on each node and, if available, DNS. This IP Address should be bound to the public adapter before starting the install. It should be a static, not DHCP, address. The private IP address, which should be from a different subnet than the public IP address. This address does not require registering in DNS but you should place an entry in the hosts file on each node. This IP Address should be bound to the private adapter before starting the install. It should be a static, not DHCP, address. Oracle recommends that you use a subnet reserved for private networks, such as 10.0.0.0 or 192.168.0.0 A VIP address, which should be from the same subnet as the public IP address and should be recorded in DNS and the hosts file on each node. This IP Address should NOT be bound to the public adapter before starting the install. Oracle Clusterware is responsible for binding this address. It should be a static, not DHCP, address. Refer to Metalink Note 296874.1 "Configuring the HPUX Operating System for the Oracle 10g VIP". If you do not have a DNS server then make sure both the public and the VIP addresses are entered into all the relevant hosts files that are normally found in the /etc/hosts file.

2.6.1. *Network Ping Tests


There are a series of 'ping' tests that should be completed, and then the network adapter binding order should be checked. You should ensure that the public IP addresses resolve correctly and that the private addresses are of the form 'nodename-priv' and resolve on both nodes via the hosts file. Public Ping test Pinging node1 from node1 should return node1's public IP address Pinging node2 from node1 should return node2's public IP address Pinging node1 from node2 should return node1's public IP address Pinging node2 from node2 should return node2's public IP address Private Ping test Pinging node1 private from node1 should return node1's private IP address Pinging node2 private from node1 should return node2's private IP address Pinging node1 private from node2 should return node1's private IP address Pinging node2 private from node2 should return node2's private IP address VIP Ping test Pinging the VIP address at this point should fail. VIPs will be activated at the end of the Oracle Clusterware install. If any of the above tests fail you should fix name/address resolution by updating the DNS or local hosts files on each node before continuing with the installation.

2.6.2. *Network Adapter configuration


If your network adapters allow configuration you should make sure that they are configured for Full Duplex and at the 'fastest' speed consistent among nodes. They should not be left to 'auto-negotiate'.

2.7. *Stage the Oracle Software


It is recommended that you stage the required software onto a local drive on Node 1 of your cluster. The 11gR1 now integrates the Oracle Clusterware, Database and Client install into one DVD with one runInstaller program. 2.6. *Network Requirements and Preparation 10

2.8. *Check Preparations


Now you can run the CVU to check the state of the cluster prior to the install of the Oracle Clusterware Software. CVU can be run from the installation media. From the clsuterware directory (just below the stage directory), issue the following command to run the CVU utitility:
# runcluvfy.sh stage -pre crsinst -n node1,node2 -r 11gR1 -verbose

This will produce a listing of pre-installation tests and success or failure states for each test, much like the following: Show CVU output Hide CVU output

runcluvfy.sh stage -pre crsinst -n atlrp441,atlrp442 -r 11gR1 -verbose Performing pre-checks for cluster services setup Checking node reachability... Check: Node reachability from node "atlrp441" Destination Node Reachable? ------------------------------------ -----------------------atlrp441 yes atlrp442 yes Result: Node reachability check passed from node "atlrp441".

Checking user equivalence... Check: User equivalence for user "oracle" Node Name Comment ------------------------------------ -----------------------atlrp442 passed atlrp441 passed Result: User equivalence check passed for user "oracle". Checking administrative privileges... Check: Existence of user "oracle" Node Name User Exists Comment ------------ ------------------------ -----------------------atlrp442 yes passed atlrp441 yes passed Result: User existence check passed for "oracle". Check: Existence of group "oinstall" Node Name Status Group ID ------------ ------------------------ -----------------------atlrp442 exists 402 atlrp441 exists 402 Result: Group existence check passed for "oinstall". Check: Membership of user "oracle" in group "oinstall" [as Primary] Node Name User Exists Group Exists User in Group Primary ---------------- ------------ ------------ ------------ -----------atlrp442 yes yes yes yes atlrp441 yes yes yes yes Result: Membership check for user "oracle" in group "oinstall" [as Primary] Administrative privileges check passed. Checking node connectivity... Interface information for node "atlrp442" Interface Name IP Address Subnet ---------------- ----------------------eth0 10.143.181.34 10.143.181.0 eth2 10.143.207.17 10.143.207.0

Comment -----------passed passed passed.

Subnet Gateway -----------10.143.181.34 10.143.207.17

Default Gateway -----------UNKNOWN UNKNOWN

Hardware Address -----------00:0E:0C:08:06:4B 00:04:23:A6:CD:BE

Interface information for node "atlrp441" Interface Name IP Address Subnet ---------------- ------------ -----------eth0 10.143.181.33 10.143.181.0 eth2 10.143.207.16 10.143.207.0

Subnet Gateway -----------10.143.181.33 10.143.207.16

Default Gateway -----------UNKNOWN UNKNOWN

Hardware Address -----------00:0E:0C:08:08:F4 00:04:23:A6:CD:6A

Check: Node connectivity of subnet "10.143.181.0" Source Destination

Connected?

2.8. *Check Preparations

11

------------------------------ ------------------------------ ---------------atlrp442:eth0 atlrp441:eth0 yes Result: Node connectivity check passed for subnet "10.143.181.0" with node(s) atlrp442,atlrp441. Check: Node connectivity of subnet "10.143.207.0" Source Destination Connected? ------------------------------ ------------------------------ ---------------atlrp442:eth2 atlrp441:eth2 yes Result: Node connectivity check passed for subnet "10.143.207.0" with node(s) atlrp442,atlrp441. Interfaces found on subnet "10.143.181.0" that are likely candidates for a private interconnect: atlrp442 eth0:10.143.181.34 atlrp441 eth0:10.143.181.33 Interfaces found on subnet "10.143.207.0" that are likely candidates for a private interconnect: atlrp442 eth2:10.143.207.17 atlrp441 eth2:10.143.207.16 WARNING: Could not find a suitable set of interfaces for VIPs. Result: Node connectivity check passed. Checking system requirements for 'crs'... Check: Total memory Node Name Available ------------ -----------------------atlrp442 7.98GB (8371808KB) atlrp441 7.98GB (8371808KB) Result: Total memory check passed. Check: Free disk space in "/tmp" dir Node Name Available ------------ -----------------------atlrp442 5.02GB (5262320KB) atlrp441 5.02GB (5262320KB) Result: Free disk space check passed. Check: Swap space Node Name Available ------------ -----------------------atlrp442 15.98GB (16760416KB) atlrp441 15.98GB (16760416KB) Result: Swap space check passed. Check: System architecture Node Name Available ------------ -----------------------atlrp442 PA-RISC atlrp441 PA-RISC Result: System architecture check passed. Check: Kernel version Node Name Available ------------ -----------------------atlrp442 HP-UX B.11.31 atlrp441 HP-UX B.11.31 Result: Kernel version check passed.

Required -----------------------1GB (1048576KB) 1GB (1048576KB)

Comment ---------passed passed

Required -----------------------400MB (409600KB) 400MB (409600KB)

Comment ---------passed passed

Required -----------------------1.5GB (1572864KB) 1.5GB (1572864KB)

Comment ---------passed passed

Required -----------------------PA-RISC PA-RISC

Comment ---------passed passed

Required -----------------------HP-UX B.11.31 HP-UX B.11.31

Comment ---------passed passed

Check: Package existence for "OS-Core:B.11.31" Node Name Status ------------------------------ -----------------------------atlrp441 OS-Core:B.11.31 atlrp442 OS-Core:B.11.31 Result: Package existence check passed for "OS-Core:B.11.31".

Comment ---------------passed passed

Check: Operating system patch for "PHKL_36248" => Superceded by PHKL_37454 Node Name Applied Required Comment ------------ ------------------------ ------------------------ ---------atlrp441 missing PHKL_36248 failed atlrp442 missing PHKL_36248 failed Result: Operating system patch check failed for "PHKL_36248". Check: Operating system patch for "PHKL_36249" => Superceded by Node Name Applied Required ------------ ------------------------ -----------------------atlrp441 missing PHKL_36249 atlrp442 missing PHKL_36249 Result: Operating system patch check failed for "PHKL_36249". Check: Operating system patch for "PHKL_35900" => Superceded by Node Name Applied Required ------------ ------------------------ -----------------------atlrp441 missing PHKL_35900 atlrp442 missing PHKL_35900 Result: Operating system patch check failed for "PHKL_35900". PHKL_37453 Comment ---------failed failed

PHSS_37452 Comment ---------failed failed

Check: Operating system patch for "PHKL_35936" => Superceded by PHSS_37452 Node Name Applied Required Comment

2.8. *Check Preparations

12

------------ ------------------------ -----------------------atlrp441 missing PHKL_35936 atlrp442 missing PHKL_35936 Result: Operating system patch check failed for "PHKL_35936". Check: Operating system patch for "PHSS_37948" Node Name Applied Required ------------ ------------------------ -----------------------atlrp441 PHSS_37948 PHSS_37948 atlrp442 PHSS_37948 PHSS_37948 Result: Operating system patch check passed for "PHSS_37948". Check: Package existence for "ACXX:C.06.10" Node Name Status ------------------------------ -----------------------------atlrp441 ACXX:C.06.10 atlrp442 ACXX:C.06.10 Result: Package existence check passed for "ACXX:C.06.10". Check: Package existence for "C-ANSI-C:C.06.10" Node Name Status ------------------------------ -----------------------------atlrp441 C-ANSI-C:C.06.10 atlrp442 C-ANSI-C:C.06.10 Result: Package existence check passed for "C-ANSI-C:C.06.10". Check: Operating system patch for "PHKL_37296" Node Name Applied Required ------------ ------------------------ -----------------------atlrp441 PHKL_37296 PHKL_37296 atlrp442 PHKL_37296 PHKL_37296 Result: Operating system patch check passed for "PHKL_37296". Check: Operating system patch for "PHCO_37476" Node Name Applied Required ------------ ------------------------ -----------------------atlrp441 PHKL_34941 PHCO_37476 atlrp442 PHKL_34941 PHCO_37476 Result: Operating system patch check passed for "PHCO_37476". Check: Operating system patch for "PHCO_37807" Node Name Applied Required ------------ ------------------------ -----------------------atlrp441 missing PHCO_37807 atlrp442 missing PHCO_37807 Result: Operating system patch check passed for "PHCO_37807". Check: Operating system patch for "PHSS_37948" Node Name Applied Required ------------ ------------------------ -----------------------atlrp441 PHSS_37948 PHSS_37948 atlrp442 PHSS_37948 PHSS_37948 Result: Operating system patch check passed for "PHCO_37807". Check: Package existence for "Perl5-64:D.5.8.8" Node Name Status ------------------------------ -----------------------------atlrp441 Perl5-64:D.5.8.8.B atlrp442 Perl5-64:D.5.8.8.B Result: Package existence check passed for "Perl5-64:D.5.8.8". Check: Group existence for "dba" Node Name Status Comment ------------ ------------------------ -----------------------atlrp442 exists passed atlrp441 exists passed Result: Group existence check passed for "dba". Check: Group existence for "oinstall" Node Name Status Comment ------------ ------------------------ -----------------------atlrp442 exists passed atlrp441 exists passed Result: Group existence check passed for "oinstall". Check: User existence for "nobody" Node Name Status Comment ------------ ------------------------ -----------------------atlrp442 exists passed atlrp441 exists passed Result: User existence check passed for "nobody". System requirement failed for 'crs'

---------failed failed

Comment ---------passed passed

Comment ---------------passed passed

Comment ---------------passed passed

Comment ---------passed passed

Comment ---------passed passed

Comment ---------passed passed

Comment ---------passed passed

Comment ---------------passed passed

Pre-check for cluster services setup was unsuccessful on all the nodes.

2.8. *Check Preparations

13

Show all

Hide all

Note that in this case the CVU failed, but only because some of the patches it was checking for had been superceded with more recent versions.

3. *Prepare the Shared Storage for Oracle RAC This section describes how to prepare the shared storage for Oracle RAC
For all installations, you must choose the storage option that you want to use for Oracle Clusterware files, Automatic Storage Manager (ASM) and Oracle Real Application Clusters databases (Oracle RAC). Oracle Clusterware files include: Voting disks, used to monitor cluster node status, and Oracle Cluster Registry (OCR) which contains configuration information about the cluster. There are two ways of storing Oracle Clusterware files: Raw Devices: Oracle Clusterware files can be placed on RAW devices based on shared disk partitions. A supported shared file system - Supported file systems include the following: A supported cluster file system Network File System (NFS): A file-level protocol that enables access and sharing of files In the example given in this document, Raw Devices are used.

3.1. Configuring Raw Disk for Storage To configure raw disks for database file storage, follow the following steps:
Make sure that the disks are available, enter the following command on every node:
# ioscan -fnNkCdisk

If the ioscan command does not display device name information for a device that you want to use, enter the following command to install the special device files for any new devices:
# insf -e

The disk names for the same disk can be different on each node. A disk can be identified as the same one via the WWID. The WWID of a disk can be checked via the following command:
# scsimgr lun_map -D /dev/rdsk/disk25 | grep WWID

The System Management Homepage also shows the WWID for each disk:

3. *Prepare the Shared Storage for Oracle RAC This section describes how to prepare the shared storage for14 Oracle RA

For each disk that you want to use, enter the following command on any node to verify that it is not already part of an LVM volume group:
# pvdisplay /dev/rdsk/ora_data1

If this command displays volume group information, the disk is already part of a volume group. The disks that you choose must not be part of an LVM volume group. We recommend to create a special Oracle device directory, and to use mknod to create device paths in this special Oracle folder. This has the advantage, that you get same names for OCR and Voting files across all nodes in the cluster. In addition, it ensures that the permissions of these Oracle device files remain untouched of 'insf e'.
# mkdir /dev/oracle # ll /dev/rdsk/disk25 crw-r----- 1 bin sys 23 0x000019 Jan 16 12:16 /dev/rdisk/disk25 # mknod /dev/oracle/asmdisk1 c 23 0x000019

Later during set-up, for the ASM instance, set the ASM_DISKSTRING parameter to /dev/oracle/* Now when 'insf -e' is run, it only touches 'standard' DSF, rather than this special ones. Modify the owner, group, and permissions on the character raw device files (ASM and Voting Disks) on all nodes:
# chown oracle:dba /dev/oracle/* # chmod 660 /dev/oracle/*

Modify the owner, group, and permissions on the character raw device files (OCR files) on all nodes:
# chown root:dba /dev/oracle/OCR # chmod 640 /dev/oracle/OCR

4. Oracle Clusterware Installation and Configuration


The next step is to install the Oracle Clusterware layer. Oracle Clusterware is an essential component of the Oracle RAC database infrastructure. Oracle Clusterware does not require any other clustering software.

3.1. Configuring Raw Disk for Storage To configure raw disks fordatabase file storage, follow the following15 steps:

4.1. Oracle Clusterware Install


Start the installer by running runInstaller from the staged installation media.
$ ./runInstaller &

Notes Notice that the Oracle 11g Installer now combines the Oracle Database, Client and Clusterware components Action Select the Oracle Clusterware radio button Click Next

Notes The OUI will name the Oracle Clusterware Home'OraCrs11g_home'. If you change this you should make sure that the name you use is unique Actions Specify a location for the Oracle Clusterware Home Click Next 4.1. Oracle Clusterware Install 16

Notes The installer will validate the state of the cluster before continuing. If there are issues you should rectify them before continuing Actions Click Next

Notes Each Cluster requires a name, this should be unique within your organisation, The default is a substring of the node name followed by _cluster This is where you specify details of all the nodes in the cluster. The installer will default names for the node it is running on. You must add other nodes manually Oracle defaults the names to 'nodename', 'nodename-priv', 'nodename-vip' Actions Confirm the Cluster Name selected is acceptable Confirm the Details for the current node are OK. The defaults are: Public Node Name : must resolve via hosts and or DNS to the public IP address and must be live 4.1. Oracle Clusterware Install 17

Private Node Name : must resolve via hosts to the interconnect IP address and must be live Virtual Host Name : must resolve via hosts and or DNS to a new IP address and must not be live If these are not correct select the node entry and click Edit... to modify OR Click Add... to add more nodes to the cluster

Notes If you have more nodes repeast the Add... cycle Actions Click Next

Notes Here you can see we have successfully configured the network adapter usage. Ideally you will have only 1 adapter set as public and 1 adapter set as private. Other adapters, if available, set to do not use. If you have multiple public or multiple private adapters it is better to team them at the OS adapter driver level before commencing the install.Type : Private Actions Click Next 4.1. Oracle Clusterware Install 18

Notes Here we specify the shared storage devices that will be used by Oracle Clusterware. Ideally you will have 2 devices for the OCR, Oracle will mirror to these devices to protect you from a single OCR device failure. Also you will have an additional 3 vote devices to protect your cluster from the failure of a single vote device. Actions Select the Normal radio button Enter the device to be used for the First OCR Enter the device to be used for the Second OCR Click Next

Notes Next we specify the devices to be used for the Oracle Clusterware vote disks Actions Select the Normal radio button Enter the device to be used for the First vote disk Enter the device to be used for the Second vote disk Enter the device to be used for the Third vote disk 4.1. Oracle Clusterware Install 19

Click Next

Notes the installer lists a summary of the planned actions Actions Click Install

Notes the OUI first copies software to the local node and then copies the software to the remote nodes Actions none required

4.1. Oracle Clusterware Install

20

Notes The installer requires commands to be run as root on each of the nodes Actions On the first node open a root shell window & run the root.sh script Show root.sh output Hide root.sh output

# /opt/oracle/product/CRS/root.sh WARNING: directory '/opt/oracle/product' is not owned by root WARNING: directory '/opt/oracle' is not owned by root WARNING: directory '/opt' is not owned by root Checking to see if Oracle CRS stack is already configured Setting the permissions on OCR backup directory Setting up Network socket directories Oracle Cluster Registry configuration upgraded successfully The directory '/opt/oracle/product' is not owned by root. Changing owner to root The directory '/opt/oracle' is not owned by root. Changing owner to root The directory '/opt' is not owned by root. Changing owner to root Successfully accumulated necessary OCR keys. Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897. node <nodenumber>: <nodename> <private interconnect name> <hostname> node 1: atlrp441 atlrp441-priv atlrp441 node 2: atlrp442 atlrp442-priv atlrp442 Creating OCR keys for user 'root', privgrp 'sys'.. Operation successful. Now formatting voting device: /dev/oracle/vote1 Now formatting voting device: /dev/oracle/vote2 Now formatting voting device: /votedisk/vote3 Format of 3 voting devices complete. Startup will be queued to init within 30 seconds. Adding daemons to inittab Expecting the CRS daemons to be up within 600 seconds. Cluster Synchronization Services is active on these nodes. atlrp441 Cluster Synchronization Services is inactive on these nodes. atlrp442 Local node checking complete. Run root.sh on remaining nodes to start CRS daemons.

Show all

Hide all

Once this has completed successfully on the first node, open a root shell window on the next node & run the root.sh script

4.1. Oracle Clusterware Install

21

Show root.sh output

Hide root.sh output

# /opt/oracle/product/CRS/root.sh WARNING: directory '/opt/oracle/product' is not owned by root WARNING: directory '/opt' is not owned by root Checking to see if Oracle CRS stack is already configured Setting the permissions on OCR backup directory Setting up Network socket directories Oracle Cluster Registry configuration upgraded successfully The directory '/opt/oracle/product' is not owned by root. Changing owner to root The directory '/opt' is not owned by root. Changing owner to root clscfg: EXISTING configuration version 4 detected. clscfg: version 4 is 11 Release 1. Successfully accumulated necessary OCR keys. Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897. node <nodenumber>: <nodename> <private interconnect name> <hostname> node 1: atlrp441 atlrp441-priv atlrp441 node 2: atlrp442 atlrp442-priv atlrp442 clscfg: Arguments check out successfully. NO KEYS WERE WRITTEN. Supply -force parameter to override. -force is destructive and will destroy any previous cluster configuration. Oracle Cluster Registry for cluster has already been initialized Startup will be queued to init within 30 seconds. Adding daemons to inittab Expecting the CRS daemons to be up within 600 seconds. Cluster Synchronization Services is active on these nodes. atlrp441 atlrp442 Cluster Synchronization Services is active on all the nodes. Oracle CRS stack installed and running under init(1M) Running vipca(silent) for configuring nodeapps Creating Creating Creating Starting Starting Starting Done. VIP GSD ONS VIP GSD ONS application application application application application application resource resource resource resource resource resource on on on on on on (2) (2) (2) (2) (2) (2) nodes... nodes... nodes... nodes... nodes... nodes...

Show all

Hide all

Notes a series of configuration assistants are run at the end of the install Actions 4.1. Oracle Clusterware Install 22

none required

Notes The installer has completed installation and configuration of Oracle Clusterware Actions Click Exit

Verify your CRS installation by executing the olsnodes command from the /opt/oracle/product/CRS/bin directory:
# olsnodes -n # atlrp441 1 # atlrp442 2

Now you should see the following processes running: oprocd -- Process monitor for the cluster. evmd -- Event manager daemon that starts the racgevt process to manage callouts. ocssd -- Manages cluster node membership and runs as oracle user; failure of this process results in cluster restart. crsd -- Performs high availability recovery and management operations such as maintaining the OCR. Also manages application resources and runs as root user and restarts automatically upon failure. You can check whether the Oracle processes evmd, occsd, and crsd are running by issuing the following command.
# ps -ef | grep d.bin

At this point, you have completed phase one, the installation of Cluster Ready Services Please note that Oracle added the following three lines to the automatic startup file /etc/inittab h1:3:respawn:/sbin/init.d/init.evmd run >/dev/null 2>&1 </dev/null h2:3:respawn:/sbin/init.d/init.cssd fatal >/dev/null 2>&1 </dev/null h3:3:respawn:/sbin/init.d/init.crsd run >/dev/null 2>&1 </dev/null

4.1. Oracle Clusterware Install

23

Oracle Support recommends NEVER modifying these entries in the inittab or modifying the init scripts unless you use this method to stop a reboot loop or are given explicit instructions from Oracle support. To ensure that the Oracle Clusterware install on all the nodes is valid, the following should be checked on all the nodes:
# /opt/oracle/product/CRS/bin/crsctl check crs

5. Oracle Clusterware patching


At this point we have installed Oracle Clusterware 11.1.0.6. In this section we will patch the Oracle Clusterware to the latest release of Oracle 11gR1 - 11.1.0.7. The patchset can be downloaded from metalink. Note: The Oracle Clusterware software must be at the same or newer level as any other ORACLE_HOME directory. Therefore, you should always upgrade Oracle Clusterware before you upgrade any other ORACLE_HOME directory. This patch does not allow you to upgrade other ORACLE_HOME directories, before you upgrade Oracle Clusterware home directory. We will apply this patchset in a rolling upgrade fashion. Before we start we can query the clusterware versions
# /opt/oracle/product/CRS/bin/crsctl query crs Oracle Clusterware version on node [atlrp441] is [11.1.0.6.0] # /opt/oracle/product/CRS/bin/crsctl query crs Oracle Clusterware active version on the cluster is [11.1.0.6.0]

softwareversion activeversion

Enter the following commands to start Oracle Universal Installer, where patchset_directory is the directory where you unpacked the patch set software:
$ cd patchset_directory/Disk1 $ ./runInstaller

5. Oracle Clusterware patching

24

Notes The installer appears Actions Click Next

Notes You should ensure that the Clusterware home is selected in the first drop down list box The installer should default the directory to the correct location Actions Ensure the correct home is selected Click Next

5. Oracle Clusterware patching

25

Notes the installer detects that this is a clustered home and automatically selects all the nodes in the cluster Actions Click Next

5. Oracle Clusterware patching

26

Notes Some parameters are validated by the installer Actions Click Next

Notes This is a summary of the actions the installer will complete Actions Click Install

5. Oracle Clusterware patching

27

Notes the installer stages the patch on all the nodes in the cluster Actions No action required

Notes 5. Oracle Clusterware patching 28

At the end the installer lists the mandatory steps that must be completed to apply this patch Actions Log in as the root user and enter the following command to shut down the Oracle Clusterware: # CRS_home/bin/crsctl stop crs Run the root111.sh script. It will automatically start the Oracle Clusterware on the patched node:
# CRS_home/install/root111.sh

Show root111.sh output (all nodes except last)

Hide root111.sh output (all nodes except last)

# /opt/oracle/product/CRS/install/root111.sh Creating pre-patch directory for saving pre-patch clusterware files Completed patching clusterware files to /opt/crs Relinking some shared libraries. Relinking of patched files is complete. Preparing to recopy patched init and RC scripts. Recopying init and RC scripts. Startup will be queued to init within 30 seconds. Starting up the CRS daemons. Waiting for the patched CRS daemons to start. This may take a while on some systems. . 11107 patch successfully applied. clscfg: EXISTING configuration version 4 detected. clscfg: version 4 is 11 Release 1. Successfully accumulated necessary OCR keys. Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897. node <nodenumber>: <nodename> <private interconnect name> <hostname> node 1: atlrp441 atlrp441-priv atlrp441 node 2: atlrp442 atlrp442-priv atlrp442 Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. clscfg -upgrade completed successfully Creating '/opt/oracle/product/CRS/install/paramfile.crs' with data used for CRS configuration Setting CRS configuration values in /opt/oracle/product/CRS/install/paramfile.crs Setting cluster unique identifier Restarting Oracle clusterware Stopping Oracle clusterware Stopping resources. This could take several minutes. Successfully stopped Oracle Clusterware resources Stopping Cluster Synchronization Services. Shutting down the Cluster Synchronization Services daemon. Shutdown request successfully issued. Waiting for Cluster Synchronization Services daemon to stop Cluster Synchronization Services daemon has stopped Starting Oracle clusterware Attempting to start Oracle Clusterware stack Waiting for Cluster Synchronization Services daemon to start Waiting for Cluster Synchronization Services daemon to start Waiting for Cluster Synchronization Services daemon to start Waiting for Cluster Synchronization Services daemon to start Cluster Synchronization Services daemon has started Event Manager daemon has started Cluster Ready Services daemon has started

Show root111.sh output on last node

Hide root111.sh output on last node

# /opt/oracle/product/CRS/install/root111.sh Creating pre-patch directory for saving pre-patch clusterware files Completed patching clusterware files to /opt/crs Relinking some shared libraries. Relinking of patched files is complete. Preparing to recopy patched init and RC scripts. Recopying init and RC scripts. Startup will be queued to init within 30 seconds.

5. Oracle Clusterware patching

29

Starting up the CRS daemons. Waiting for the patched CRS daemons to start. This may take a while on some systems. . 11107 patch successfully applied. clscfg: EXISTING configuration version 4 detected. clscfg: version 4 is 11 Release 1. Successfully accumulated necessary OCR keys. Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897. node <nodenumber>: <nodename> <private interconnect name> <hostname> node 1: atlrp441 atlrp441-priv atlrp441 node 2: atlrp442 atlrp442-priv atlrp442 Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. clscfg -upgrade completed successfully Creating '/opt/oracle/product/CRS/install/paramfile.crs' with data used for CRS configuration Setting CRS configuration values in /opt/oracle/product/CRS/install/paramfile.crs

Continue with these two steps on all nodes, one by one, this will achieve a rolling upgrade of the Oracle Clusterware. When done, verify the Oracle Clusterware is running on all nodes before exiting the installer. $ CRS_home/bin/crsctl check crs Cluster Synchronization Services appears healthy Cluster Ready Services appears healthy Event Manager appears healthy
# /opt/oracle/product/CRS/bin/crsctl query crs Oracle Clusterware version on node [atlrp441] is [11.1.0.7.0] # /opt/oracle/product/CRS/bin/crsctl query crs Oracle Clusterware active version on the cluster is [11.1.0.7.0]

softwareversion activeversion

Notes Once the Oracle Clusterware is running on all nodes, exit the installer Actions

5. Oracle Clusterware patching

30

Click Yes

5.1. Oracle Clusterware patching complete

6. Oracle ASM Home Software Install


6.1. CVU check 6.2. Oracle ASM Home Software Install
Start the installer by running "runInstaller" from the staged installation media.
export ORACLE_HOME=/opt/oracle/product/asm ./runInstaller &

In this section we establish the Oracle software for the ASM home.

Notes Here we will create a ASM Software home on all the nodes in the cluster Actions Select the Oracle Database 11g radio button Click Next

5.1. Oracle Clusterware patching complete

31

Notes Here we can choose the type of database software to be installed Actions Select the Enterprise Edition radio button Click Next

6.2. Oracle ASM Home Software Install

32

Notes Here we specify the location of various components. An 11g install makes more use of the ORACLE_BASE. Most logs will be stored in sub directories under the oracle base. The oracle base will be common to all installs. Modify as required. Also this is where you specify the location of the ASM software home. I usually modify the Home name and Home Path to include the word ASM. This makes it easier to identify later on. If you change the path you should ensure that you do not use the exact same path as the Oracle Clusterware home Actions Confirm entries are OK Click Next

Notes The installer will detect the presence of Oracle Clusterware and uses this to populate the dialog box. To build a cluster which includes all nodes you must ensure that there are check-boxes next to the node names Actions Click Select All Click Next

6.2. Oracle ASM Home Software Install

33

Notes The installer will then complete some Product-Specific Prerequisite checks. These should all pass OK as you have already run the CVU check Actions Click Next

6.2. Oracle ASM Home Software Install

34

Notes We are going to install a Software only home and then subsequently configure the software Actions Select the Install Software Only radio button Click Next

Notes We must assign the Unix group name of the priveleged users Action Choose oinstall (default) from the drop down lists Click Next

6.2. Oracle ASM Home Software Install

35

Notes A summary of the install will be displayed In the first step, the installer will copy the software to all of the identified cluster nodes Actions Click Install

6.2. Oracle ASM Home Software Install

36

Notes At various steps, the installer will pause, with instructions to run scrips (such as root.sh) Action open a shell window on each node and follow instructions provided by installer

# /opt/oracle/product/asm/root.sh Running Oracle 11g root.sh script... The following environment variables are set as: ORACLE_OWNER= oracle11 ORACLE_HOME= /opt/oracle/product/asm Enter the full pathname of the local bin directory: [/usr/local/bin]: /usr/local/bin is read only. Continue without copy (y/n) or retry (r)? [y]: y Warning: /usr/local/bin is read only. No files will be copied.

Creating /etc/oratab file... Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root.sh script. Now product-specific root actions will be performed. Finished product-specific root actions.

Notes Run the root scripts - they should only take a few seconds to run on each node, one at a time Action run the scripts indicated in the previous screen on both nodes then return to the installer and Click OK

6.2. Oracle ASM Home Software Install

37

Notes After the software install completes you will see this End of Installation dialog Actions Click Exit

6.2.1. Oracle ASM software home has been installed

7. Oracle ASM Software Home Patching


Once ASM software has been installed, the next step in the process is to apply the 11.1.0.7 patchset to the ASM ORACLE_HOME.

7.1. Start the OUI for Oracle ASM Software Home Patching
Enter the following commands to start Oracle Universal Installer, where patchset_directory is the directory where you unpacked the patch set software:
$ cd patchset_directory/Disk1 $ ./runInstaller

Notes the Universal Installer screen appears Action On the Welcome screen, click Next.

6.2.1. Oracle ASM software home has been installed

38

Notes Specify the name and the location of the asm home Action Check that the name and location are correct

Notes: 7.1. Start the OUI for Oracle ASM Software Home Patching 39

here you can specify your metalik credentials for this install. If you leave both fields blank you can opt out of notifications (see next screen). Action Enter your email address Enter your Metalink password Click Next

Notes:* If you leave both fields blank, you will opt out of notifications and get the above screen. Action Click Yes

7.1. Start the OUI for Oracle ASM Software Home Patching

40

Notes the installer detects that this is a clustered home and automatically selects all the nodes in the cluster Actions Click Next

7.1. Start the OUI for Oracle ASM Software Home Patching

41

Notes Some parameters are validated by the installer Action Click Next

Notes This is a summary of the actions the installer will complete Actions Click Install

7.1. Start the OUI for Oracle ASM Software Home Patching

42

Notes The installer copies the patch to all the nodes in the cluster Actions No action required

Notes 7.1. Start the OUI for Oracle ASM Software Home Patching 43

The installer pauses, root.sh need to be run as root on both nodes of the cluster Action open a shell window on each node and run root.sh (one after the other) then return to the installer and Click OK
#/opt/oracle/product/asm/root.sh Running Oracle 11g root.sh script... The following environment variables are set as: ORACLE_OWNER= oracle11 ORACLE_HOME= /opt/oracle/product/asm Enter the full pathname of the local bin directory: [/usr/local/bin]: /usr/local/bin is read only. Continue without copy (y/n) or retry (r)? [y]: y Warning: /usr/local/bin is read only. No files will be copied.

Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root.sh script. Now product-specific root actions will be performed. Finished product-specific root actions.

Notes The output from the root.sh script should be similar to this it should only take a few seconds to run on each node Action run the scripts on all nodes then return to the installer and Click OK

Notes 7.1. Start the OUI for Oracle ASM Software Home Patching 44

After the software install completes you will see this End of Installation dialog, exit the installer Actions Click Yes

8. Oracle RAC Database Listener Creation


8.1. Create Node specific network listeners
The Oracle network listeners traditionally run from the ASM home. Here we are going to create the listeners using netca from the ASM home.

Notes Ensure the ORACLE_HOME environement variable isse tto the recently installed ASM home location Action change to the ASM home bin directory run ./netca

Notes Netca detects that the Oracle Clusterware layer is running and offers Cluster or Single Node configuration Actions Select the Cluster configuration radio button Click Next

8. Oracle RAC Database Listener Creation

45

Notes Netca uses Oracle Clusterware to determine all the nodes in the cluster Actions Click Select all nodes Click Next

Notes You get various options we need to configure listeners Actions Select the Listener configuration radio button Click Next

Notes

8.1. Create Node specific network listeners

46

We need to add a listener Actions Select the Add radio button Click Next

Notes Here you get the opportunity to name the listener Do not change this. The listeners will eventually be called LISTENER_nodename1 & LISTENER_nodename2. This is important for RAC Actions Click Next

Notes Oracle Net supports various network protocols, although TCP is the most common. 8.1. Create Node specific network listeners 47

Actions Ensure the Selected Protocols list includes TCP Click Next

Notes It is possible to choose a non-default Port I do not recommend this Actions Ensure the Use the Standard port number of 1521 radio button is set Click Next Notes After configuring the node listeners you get the opportunity to configure more network components Actions Select the No radio button Click Next

8.1. Create Node specific network listeners

48

Notes You get the opportunity to configre other networking components Action Click the Finish button to exit the tool Notes You can see the listener resources inside Oracle Clusterware Action run the ./crs_stat -t comand from the Oracle Clusterware home bin directory

8.2. Completed Node specific network listeners creation


You have now completed the RAC cluster nodes Network listener configuration. A new managed resource will have been added to Oracle Clusterware OCR for each listener.

9. Oracle ASM Instance and diskgroup Creation


9.1. Create ASM Instance and add the +DATA and +FLASH diskgroups
Notes We use the dbca from the ASM install to create the ASM instances Actions ensure the ORACLE_HOME environment variable is set to the ASM home directory run ./dbca from the ASM home bin directory

8.2. Completed Node specific network listeners creation

49

Notes dbca detects the Oracle Clusterware layer is running and offers to create either cluster or single instance database Actions Select the Oracle Real Application Clusters database radio button Click Next

9.1. Create ASM Instance and add the +DATA and +FLASH diskgroups

50

Notes We use dbca to create RAC databases and ASM instances Actions Select the Configure Automatic Storage Management radio button Click Next Notes You need to make sure you create ASM instances on all the cluster nodes Actions Click Select All Click Next

9.1. Create ASM Instance and add the +DATA and +FLASH diskgroups

51

Notes Here we specify the password for the ASM Oracle SYS user Actions Enter the SYS password Enter the same password for the Confirm SYS password Click Next

9.1. Create ASM Instance and add the +DATA and +FLASH diskgroups

52

Notes dbca will create and start ASM instances Actions Click OK

9.1. Create ASM Instance and add the +DATA and +FLASH diskgroups

53

Notes This should take a few seconds Actions no action

Notes ASM requires disks to be group together into diskgroups. This section will be used to create 2 disk groups +DATA and +FLASH Actions Click Create New

9.1. Create ASM Instance and add the +DATA and +FLASH diskgroups

54

Notes At the moment no disks are visible Actions Click Change Discovery Path

9.1. Create ASM Instance and add the +DATA and +FLASH diskgroups

55

Notes Now we will assign disks to specific disk groups and create the DATA diskgroup Use a discovery string to enable dbca to find the available disks Actions In the Disk Group Name enter DATA Select the External Redundancy radio button Select the Show All radio button Select the 6 disks to be used for the DATA diskgroup Click OK Notes A progress message will be displayed - this should not take very long Actions none

Notes Here we can see the DATA diskgroup has been created and is mounted on 2/2 instances. We now need to create the FLASH diskgroup Actions Click Create New Notes Repeat the above steps, this time allocating the disks for the FLASH disk group 9.1. Create ASM Instance and add the +DATA and +FLASH diskgroups 56

Notes Here we can see the DATA and FLASH diskgroups have been created and are mounted on 2/2 instances. This completed the ASM configuration. Actions Click Finish

9.1. Create ASM Instance and add the +DATA and +FLASH diskgroups

57

Notes A confirmation dialog box Actions Click No run ./crs_stat -t from the Oracle Clusterware bin directory

9.2. Completed creation of ASM Instance and add of the +DATA and +FLASH diskgroups
Congratulations you have installed ASM, Network listeners and created 2 ASM diskgroups.

10. Oracle RAC Database Home Software Install


10.1. CVU check - Pre Database Install
Before installing the RAC Database software you should use the CVU utility to confirm the status of the cluster.

10.2. Oracle RAC Database Home Software Install


Start the installer by running "runInstaller" from the staged installation media.
export ORACLE_HOME=/opt/oracle/product/RAC11 ./runInstaller &

In this section we establish the Oracle software for the RAC Database home. 9.2. Completed creation of ASM Instance and add of the +DATA and +FLASH diskgroups 58

Notes We use the same installer for Database homes as Clusterware homes Actions Select the Oracle Database 11g radio button Click Next

10.2. Oracle RAC Database Home Software Install

59

Notes Here we can choose the type of database software to be installed Actions Select the Enterprise Edition radio button Click Next

Notes Here we specify the location of various components. If you change the path you should ensure that you do not use the exact same path as the Oracle Clusterware or ASM homes Actions Confirm entries are OK Click Next

10.2. Oracle RAC Database Home Software Install

60

Notes The installer will detect the presence of Oracle Clusterware and uses this to populate a dialog box. To build a cluster which includes all nodes you must ensure that there are check-boxes next to the node names Actions Click Select All Click Next

10.2. Oracle RAC Database Home Software Install

61

Notes The installer will then complete some Product-Specific Prerequisite checks. These should all pass OK - as you have already run the CVU check Actions Click Next

10.2. Oracle RAC Database Home Software Install

62

Notes We are going to install a Software only home and then subsequently configure the software Actions Select the Install Software Only radio button Click Next

Notes Assign the Unix groups for the privileged db users (default is oinstall) Action Click Next

10.2. Oracle RAC Database Home Software Install

63

Notes Here we can see a summary of the install. Actions Click Install

Notes 10.2. Oracle RAC Database Home Software Install 64

The installer will copy the software to all nodes in the cluster The installer will pause and display instructions for running scripts, such as root.sh Be sure to follow directions carefully, running scripts one at a time, on each node Action open a shell and follow instructions closely then return to the installer and Click OK

Notes After the software install completes you will see this End of Installation dialog Actions Click Exit

10.2. Oracle RAC Database Home Software Install

65

Notes Are you sure ? Actions Click Yes

10.2.1. Oracle RAC Database software home has been installed

11. Oracle RAC Software Home Patching


Once RAC software has been installed, the next step in the process is to apply the 11.1.0.7 patchset to the RAC ORACLE_HOME.

11.1. Using the installer to patch the Oracle RAC Software Home
Enter the following commands to start Oracle Universal Installer, where patchset_directory is the directory where you unpacked the patch set software:
$ cd patchset_directory/Disk1 $ ./runInstaller

10.2.1. Oracle RAC Database software home has been installed

66

Notes the Universal Installer screen appears Actions On the Welcome screen, click Next.

Notes 11.1. Using the installer to patch the Oracle RAC Software Home 67

Specify the name and the location of the Database home Actions Check that the name and location are correct, Click Next

Notes here you can specify your metalik credentials for this install. If you leave both fields blank you can opt out of notifications (see next screen). Actions Enter your email address Enter your Metalink password Click Next

11.1. Using the installer to patch the Oracle RAC Software Home

68

Notes If you leave both fields blank, you will opt out of notifications and get the above screen. Actions Click Yes

Notes 11.1. Using the installer to patch the Oracle RAC Software Home 69

the installer detects that this is a clustered home and automatically selects all the nodes in the cluster Actions Click Next

Notes Some parameters are validated by the installer Actions Click Next

11.1. Using the installer to patch the Oracle RAC Software Home

70

Notes This is a summary of the actions the installer will complete Actions Click Install

Notes 11.1. Using the installer to patch the Oracle RAC Software Home 71

The installer copies the patch to all the nodes in the cluster Actions No action required

Notes The installer pauses, root.sh need to be run as root on both nodes of the cluster Actions open a shell window on each node and run root.sh (one after the other), output is shown below then return to the installer and Click OK

# /opt/oracle/product/RAC11/root.sh Running Oracle 11g root.sh script... The following environment variables are set as: ORACLE_OWNER= oracle11 ORACLE_HOME= /opt/oracle/product/RAC11 Enter the full pathname of the local bin directory: [/usr/local/bin]: /usr/local/bin is read only. Continue without copy (y/n) or retry (r)? [y]: y Warning: /usr/local/bin is read only. No files will be copied. Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root.sh script. Now product-specific root actions will be performed. Finished product-specific root actions.

11.1. Using the installer to patch the Oracle RAC Software Home

72

Notes Exit the installer Actions Click Yes

12. Oracle RAC Database Creation


12.1. use dbca to create the RAC database
Notes You must launch dbca from the RAC database home Action Ensure the ORACLE_HOME environment variable is set to the new RAC Home change to the RAC Home bin directory run ./dbca

12. Oracle RAC Database Creation

73

Notes dbca detects the Oracle Clusterware layer is running and offers to create either cluster or single instance database Actions Select the Oracle Real Application Clusters database radio button Click Next

12.1. use dbca to create the RAC database

74

* Notes We use dbca to create RAC databases Actions Select the Create a Database radio button Click Next Notes You need to make sure you create RAC database instances on all the cluster nodes Actions Click Select All Click Next

12.1. use dbca to create the RAC database

75

Notes Here you get to select the type of database Actions Select the General Purpose... radio button Click Next

12.1. use dbca to create the RAC database

76

Notes Now you specify the preface for the SIDs for the RAC database instances recommendation is to keep it simple Actions Enter APX in the Global Database name, The SID should autofill Click Next

12.1. use dbca to create the RAC database

77

Notes Here you get to configure Oracle Enterprise Manager. If you have no Grid Control server then the best method is to use dbcontrol as detailed here. Actions Check the Configure Enterprise manager checkbox Select the Configure Database Control for local management radio button Click Next

12.1. use dbca to create the RAC database

78

Notes Here we specify the password for the Oracle Database users. In this example we are setting the same password for all users. You should set a password scheme that meets your requirements. Actions Select the Use the Same Administrative Password radio button Enter the password Enter the same password for the Confirm password Click Next

12.1. use dbca to create the RAC database

79

Notes Here you specify where you would like your database datafiles stored. We are going to use the ASM diskgroups we created earlier. Actions Select the Automatic Storage Management (ASM) radio button Click Next

12.1. use dbca to create the RAC database

80

Notes dbca displays the diskgroups we previously created. We are going to use the DATA diskgroup Actions Check the box on the DATA diskgroup line Click Next

12.1. use dbca to create the RAC database

81

Notes Here we can adjust where various files are placed Actions Select the Use Oracle-managed Files radio button Ensure +DATA is in the Database Area field Click Next

12.1. use dbca to create the RAC database

82

Notes Here we specify recovery configuration information. We are going to use a flash recovery area. Actions Check the Specify Flash Recovery Area checkbox In the Flash recovery area enter +FLASH Set the flash recovery size remember the size of the +FASH diskgroup Check the Enable Archiving checkbox Click Next

12.1. use dbca to create the RAC database

83

Notes Here you can choose to add the Oracle Sample schemas Actions Check the Sample schemas checkbox Click Next

12.1. use dbca to create the RAC database

84

Notes Here you can modify the default initialisation parameters. Actions Click Next

12.1. use dbca to create the RAC database

85

Notes 11g introduces enhanced security settings Actions Select the Keep the enhanced radio button Click Next

Notes 11g will set up some automatic maintenance tasks Actions Check the Enable automatic maintenance tasks checkbox Click Next

12.1. use dbca to create the RAC database

86

Notes Here you can review the placement of various database files Actions Click Next

12.1. use dbca to create the RAC database

87

Notes You can decide to generate database creation scripts Actions Check the Create Database checkbox Click Next Notes A summary screen will be displayed Actions Click Next

Notes The database is created Actions none

12.1. use dbca to create the RAC database

88

Notes After the database is created the summary screen is displayed note the URL for the Database Control Actions Click Exit Notes The Database instances will be started on all nodes Actions none Notes You can see that cluster has started the database instances on each n node Action check the status of the db instances run ./crs_stat -t

12.1. use dbca to create the RAC database

89