Table of Contents

Rac11gR1OnLinux.............................................................................................................................................1 1. *Introduction.......................................................................................................................................1 1.1. *What you need to know....................................................................................................1 1.1.1. Software required for install...............................................................................1 1.1.1.1. 32-Bit Required RPM's .......................................................................2 1.1.1.2. 64-Bit Required RPM's .......................................................................2 1.1.2. Processor Model........................................................................................................................................3 . 1.2. *Installation steps ..........................................................................................................................................3 1.3. *Schematic....................................................................................................................................................4 1.3.1. Hardware/software configuration BEFORE Oracle software install............................................4 1.3.2. Hardware/software configuration AFTER Oracle software install..............................................4 1.4. *Installation Method.....................................................................................................................................5 2. *Prepare the cluster nodes for Oracle RAC.....................................................................................................6 2.1. *User Accounts................................................................................................................................6 2.1.1. *Creating the OSDBA (DBA) Group..............................................................................6 2.1.2. *Creating an OSOPER Group (Optional) ........................................................................6 2.1.3. *Creating an OSASM Group...........................................................................................7 2.1.4. *Creating an OINSTALL Group.....................................................................................7 2.1.5. *Determining Whether an Oracle Software Owner User Exists.....................................7 2.1.6. *Creating an Oracle Software Owner User.....................................................................8 2.1.7. *Modifying an Oracle Software Owner User..................................................................8 2.1.8. *Configuring SSH on Cluster Member Nodes................................................................8 2.1.9. *Checking Existing SSH Configuration on the System..................................................8 2.1.10. *To configure SSH, Create RSA Keys On Each Node using the following steps:.......9 2.1.11. *Add All Keys to a Common authorized_keys File......................................................9 2.1.12. *Enabling SSH User Equivalency on Cluster Member Nodes....................................10 2.1.13. *Setting Display and X11 Forwarding Configuration.................................................11 2.2. *Networking...............................................................................................................................................12 2.2.1. *Network Ping Tests...................................................................................................................13 2.2.2. *Network Adapter configuration................................................................................................14 2.3. *Time Sync.................................................................................................................................................14 2.4. *Configuring Kernel Parameters................................................................................................................14 2.5. *Setting Shell Limits for the oracle User....................................................................................................15 2.6. *Configuration of the Hangcheck-timer Module........................................................................................16 2.6.1. *Hangcheck-timer Module verification procedure.....................................................................16 2.7. *Platform Specific Setup............................................................................................................................17 2.8. *Stage the Oracle Software.........................................................................................................................18 2.9. *Cluster Verification Utility stage check....................................................................................................18 3. *Prepare the shared storage for Oracle RAC.................................................................................................20 3.1. Example of Configuring Block Device Storage for Oracle Clusterware .......................................21 3.1.1. Example of Creating a Udev Permissions File for Oracle Clusterware........................21 3.1.2. Platform Specific Settings.............................................................................................22 4. Oracle Clusterware Installation and Configuration.......................................................................................22 4.1. CVU Pre Oracle Clusterware install check ....................................................................................22 4.2. Oracle Clusterware Install.............................................................................................................26 . 4.2.0.1. Verify cluster resources are online.............................................................................41 4.2.0.2. Use CVU to verify the Oracle Clusterware install........................................41 5. Oracle Clusterware patching..........................................................................................................................43 5.1. Oracle Clusterware patching complete..........................................................................................50 6. Oracle ASM Home Software Install..............................................................................................................50 6.1. CVU check before creating ASM instance....................................................................................50 6.2. Oracle ASM Home Software Install..............................................................................................56 6.2.1. Oracle ASM software home has been installed.............................................................64 7. Oracle ASM Software Home Patching..........................................................................................................64 7.1. Start the OUI for Oracle ASM Software Home Patching..............................................................64 i

Table of Contents
Rac11gR1OnLinux 8. Oracle RAC Database Listener Creation.......................................................................................................71 8.1. Create Node specific network listeners ..........................................................................................71 8.2. Verify the Listener resources are online........................................................................................76 8.3. Completed Node specific network listeners creation .....................................................................76 9. Oracle ASM Instance and diskgroup Creation..............................................................................................77 9.1. Create ASM Instance and add the +DATA and +FLASH diskgroups using the Database Assistant................................................................................................................................................77 9.2. Verify ASM instances are online...................................................................................................91 9.3. Completed creation of ASM Instance and add of the +DATA and +FLASH diskgroups............92 10. Oracle RAC Database Home Software Install.............................................................................................92 10.1. CVU check -pre Database Instance Install..................................................................................92 10.2. Oracle RAC Database Home Software Install.............................................................................97 10.2.1. Oracle RAC Database software home has been installed..........................................106 11. Oracle RAC Software Home Patching......................................................................................................106 11.1. Using the installer to patch the Oracle RAC Software Home...................................................106 12. Oracle RAC Database Creation.................................................................................................................113 12.1. CVU Database Verification Check............................................................................................113 12.2. use dbca to create the RAC database.........................................................................................115 12.3. Verify all database resource are online......................................................................................133 12.4. RAC database creation is complete...........................................................................................134

ii

Rac11gR1OnLinux

1. *Introduction
1.1. *What you need to know
For those of you starting an Oracle RAC install for the first time, all this will be new. For those of you with Oracle 10g RAC on Linux experience, there are just a few changes. Disk requirements for Oracle Clusterware devices The storage requirement for the OCR & vote devices used by Oracle Clusterware has been increased to 280 MB (256 MB, plus an additional buffer to allow for fdisk variation in space allocation). Better support for IP address The previous release had some restrictions that could be easily worked around, with regard to IP addresses from the non-routable range. These restrictions have been removed in the Oracle Clusterware 11g version and the workarounds are no longer required.

1.1.1. Software required for install
• Linux x86 Base Oracle 11gR1 DVD • Linux x86-64 (AMD64/EM64T) Base Oracle 11gR1 DVD Oracle Enterprise Linux: as part of the Oracle Unbreakable Linux Program, Oracle offers for download or on CD: Oracle Enterprise Linux 4 fully compatible with Red Hat Enterprise Linux 4 AS/ES Oracle Enterprise Linux 5 fully compatible with Red Hat Linux 5 Server and Advanced Platform. If you install the Oracle Validated Configuration RPM, then it sets and verifies system parameters based on recommendations from the Oracle Validated Configurations program, and installs any additional packages needed for installing Oracle Clusterware and Oracle Database. It also updates sysctl.conf settings, system startup parameters, user limits, and driver parameters to values that testing shows will provide better performance. • Note: Additional information on the Oracle Validated Configurations can be found at the following URL: http://www.oracle.com/technology/tech/linux/validated-configurations/index.html This release is certified for the following distributions: • Oracle Enterprise Linux 4 • RedHat Enterprise Linux 4 AS/ES • Oracle Enterprise Linux 5 • Red Hat Linux 5 Server and Advanced Platform. • SLES-10 • Asianux Server 3 • Asianux 2.0 SP2 Do not mix Operating Systems between nodes in a cluster. For example, do not configure a cluster with Enterprise Linux 4 on one node and Enterprise Linux 5 on another node. Rac11gR1OnLinux 1

16.1 make-3.0 make-3.3.3 compat-libstdc++-33-3. Enterprise Linux 5 Red Hat Enterprise Linux 5 binutils-2.91.2-18 compat-libstdc++-33-3.11 SUSE 10 Packages binutils-2.80-5 sysstat-5.105-2 libaio-0. Enterprise Linux 4 Red Hat Enterprise Linux 4 binutils-2. Do not customize RPMs during installation.125 elfutils-libelf-devel-0.2.8.1.92.0.4.11 unixODBC-devel-2.3.2.1.4.5-2 gcc-c++-3.1.0 libstdc++-devel-4.1.2.0 unixODBC-2.2 compat-libstdc++-33-3.3.2 unixODBC-2.3 elfutils-libelf-0.2 libaio-devel-0. This installation includes most of the required packages and helps you limit manual checks of package dependencies.1.106 libgcc-4.11 unixODBC-devel-2.15.5 compat-libstdc++-5.92.5-2 make-3.2 glibc-devel-2.50.5-12 glibc-devel-2. Enterprise Linux 4 Red Hat Enterprise Linux 4 binutils-2.Install your Linux operating system with the default software packages (RPMs).5 compat-libstdc++-5.1-52 gcc-c++-4.5 gcc-4.5 unixODBC-2.2.97-5 elfutils-libelf-devel-0.4.1 sysstat-7.1.8.50.4-2 libaio-devel-0. 32-Bit Required RPM's The following packages (or later versions) must be installed: Asianux 2.19 libaio-devel-0.4.2.2.11 unixODBC-devel-2. 1.1.0. Enterprise Linux 5.3.0.4-31.3.0.5-2 libstdc++-devel-3.2.1 SUSE 10 Packages binutils-2.1.19 glibc-devel-2.3 compat-libstdc++-33-3.0.3 (32 bit) elfutils-libelf-0.3.4-2.19 glibc-common-2.0 libstdc++-4.7-22.2.16.17.1.125 gcc-4.1-52 libstdc++-4.125 gcc-4.0.5-12 glibc-common-2. Red Hat Enterprise Linux 5 binutils-2.4-2.125 elfutils-libelf-devel-0.2.2.1.1.0.7 libelf-0.5-12 glibc-headers-2. Oracle continues to work with Linux vendors to certify Oracle products running on • Red Hat Enterprise Linux • SUSE Linux Enterprise Server • Asianux For additional information on Linux Operating System support refer to Metalink Note 266043.1.19 glibc-headers-2.97 elfutils-libelf-devel-0.5 Asianux 3.2.2.2.0 1.1.3.1-52 glibc-2.17.0.1.5-2 glibc-2.3-61 elfutils-libelf-0.0.3.97-5 gcc-3.105-2 libgcc-3.15.6-2.11 Asianux 3.81-1.1.1 libstdc++-devel-4.80 sysstat-6.0 glibc-2.91.1.2 libelf-0.5 gcc-4.5 libstdc++-3. 64-Bit Required RPM's The following packages (or later versions) must be installed: Asianux 2.4-31.104 libaio-0.4-2.el5 compat-libstdc++-33-3.97 gcc-3.1.9 1.106 libaio-0.3.6 compat-libstdc++-33-3.0.4.3 (32 bit) elfutils-libelf-0.11 ksh-93r-12.1.0.4-2.1.3.4.3-47. Software required for install 2 .104 libgcc-4.3.

2 To determine whether the required packages are installed.1 (32 bit) libstdc++-4.1.1 libstdc++-4.3.When available you must install the 64bit Itanium IA64 versions of the Oracle software.1. • Linux x86-64 (AMD64/EM64T). 1. then install it.3.5 glibc-devel-2.3.4 (32-bit) libaio-0.80 sysstat-6.2.5-12 (32 bit) glibc-common-2.2.19 glibc-2.3.5 glibc-2.5 glibc-devel-2.0 gcc-c++-4. 64-Bit Required RPM's 3 .1. • Lunix x86 32bit .0 libstdc++-devel-4. Processor Model This paper covers both 32bit and 64bit install.106 (32 bit) libaio-devel-0.106 libgcc-4.19 (32 bit) glibc-common-2.1.4-2.1 (32 bit) libstdc++-devel 4.you must install the 32bit versions of the Oracle software. Please note that The OS Version of Oracle you install must match the OS Chip Version.3.105 libgcc-3.4.5 (32-bit) libstdc++-3.0. • Lunix 64bit Itanium IA64 .80 sysstat-5.0.1.0 make-3.3.NOT AVAILABLE .3.4-31.0 glibc-2.4.5 make-3. enter commands similar to the following: # rpm -q package_name If a package is not installed.104 (32 bit) libgcc-4.1.1.3.3.81 sysstat-7. typically Node1 • Establish Oracle Clusterware ♦ Install the Oracle Clusterware (using the push mechanism to install on the other nodes in the cluster) ♦ Patch the Clusterware to the latest patchset • Establish ASM ♦ Install an Oracle Software Home for ASM ♦ Patch the ASM Home to the latest patchset ♦ Create the Listeners ♦ Create the ASM Instances and initial Disk Group • Establish RAC Database 1.1.5 libgcc-3.4.3.5 gcc-c++-4.4-2. 1.1.gcc-c++-3.4 (32 bit) libaio-0. to make sure the cluster is setup OK.105 libaio-0.3.5 libstdc++-3.4.2 glibc-32bit-2.4 glibc-devel-32bit-2.4.3.0 libstdc++-4.106 libaio-0. *Installation steps To summarise the install steps are: • Preparation ♦ Pre-reqs.5-12 (32 bit) libaio-0.105 (32 bit) libaio-devel-0.5-12 glibc-2.2.2 (32 bit) glibc-devel-2.0.4 glibc-devel-2.104 libaio-32bit-0.you must install the 64bit X64 versions of the Oracle software.5 (32 bit) libstdc++-devel 3.1.1 make-3. ♦ Stage all the software on one node.1.4 glibc-devel-2.4-31.1 libgcc-4.1.3.4.1.1 glibc-2.

Hardware/software configuration BEFORE Oracle software install 1. As explained in this document the actual number of LUN’s required will vary depending on your mirroring requirements.♦ Install an Oracle Software Home for RAC Database ♦ Patch the RAC Database Home to the latest patchset ♦ Create the RAC Database Instances 1.2.3.1. *Installation steps 4 . *Schematic The following is a schematic of the software & hardware layout of a 2node RAC cluster.3. 1.2.3. Hardware/software configuration AFTER Oracle software install 1.

4. The local Homes for Oracle Clusterware. Hardware/software configuration AFTER Oracle software install 5 . 1. ASM & Oracle Database Homes are established on the local disk of each of the RAC nodes./dev/sdc5. The files required by Oracle Clusterware are on Block devices. ASM and RAC take up a total of 6.Notice that there are a number of LUNs present to both machines from the shared storage. The database data-files are on ASM. 1.2.5GB on each nodes local disk filesystem. ASM & Oracle Database Homes binaries are established on the local disk of each of the RAC nodes. It should be noted that there are other installation configurations. The database data-files are also on OCFS. /dev/sdd5) • 16 of 13358MB LUNs ♦ 8 are used for the +DATA ASM diskgroup (/dev/sdc[8-15]) ♦ 8 are used for the +FLASH diskgroup. *Installation Method This document details one method of installing a 2-node Oracle 11gR1 RAC cluster on Linux: • Block Device: The Oracle Clusterware. This document does not cover these methods: • OCFS: The Oracle Cluster. The files required by Oracle Clusterware are on OCFS.3. (/dev/sdd[8-15]) Also notice there is a dedicated switch on the cluster interconnect. • 5 of 290MB LUNs ♦ 2 are used for the OCR devices (/dev/sdb1./dev/sdc1) ♦ 3 are used for the vote devices (/dev/sdb5.

*User Accounts 2. If you want to use an OSOPER group. NFS File Server is not covered here 2.4. 1.1. There are other possible combinations although Oracle recommends one of the above methods.1. Time Sync 4. enter the following command: # grep OSDBA_group_name /etc/group If the OSDBA group does not exist or if you require a new OSDBA group.2.1. In the following command. use the and image for each section to show/hide the section or you can Expand all or Collapse all by clicking these buttons. 2. For most installations. User Accounts 2.1. The files required by Oracle Clusterware are on Block devices. This is implemented using the Twisty Plugin which requires Java Script to be enabled on your browser. There are a number of items that require checking before the install commences. This installation routine presumes that you have a 2-node Linux cluster. Getting this bit right will enhance your install experience. but you want to give a different group of operating system users database administrative privileges in a new Oracle installation To determine whether the OSDBA group exists. Stage the Oracle Software 5. # /usr/sbin/groupadd dba Note: The default OSDBA group name is dba. ASM & Oracle Database Homes are established on the local disk of each of the RAC nodes. it is sufficient to create only the OSDBA group. • You cannot place the Clusterware devices on NFS File Server. then create it as follows. 2. Networking 3. *Installation Method 6 . for example. use the group name dba unless a group with that name already exists. then you must create it in the following circumstances: 1. Run CVU It is essential that these items are checked and are configured correctly before the install commences. *Creating an OSOPER Group (Optional) Create an OSOPER group only if you want to identify a group of operating system users with a limited set of database administrative privileges (SYSOPER operator privileges). *Creating the OSDBA (DBA) Group You must create an OSDBA group in the following circumstances: • An OSDBA group does not exist.• NFS: The Oracle Cluster. *Prepare the cluster nodes for Oracle RAC The guides include hidden sections. if this is the first installation of Oracle Database software on the system • An OSDBA group exists. The database data-files are also on NFS File Server.

enter the following command: # id oracle If the oracle user exists.5. *Creating an OINSTALL Group You must create an OINSTALL group in the following circumstances: • An OINSTALL group does not exist. enter the following command: # grep OSASM_group_name /etc/group If the OSASM group does not exist or if you require a new OSASM group. for example. If you want to use an OSASM group. In the following command. then create it as follows.1. # /usr/sbin/groupadd oinstall Note: The default OINSTALL group name is oinstall. use the group name oper unless a group with that name already exists. # /usr/sbin/groupadd oper 2. use the group name asadmin unless a group with that name already exists. *Determining Whether an Oracle Software Owner User Exists To determine whether an Oracle software owner user named oracle exists.1. then you must create it in the following circumstances: • If an OSASM group does not exist. then the output from this command is similar to the following: 2. if this is the first installation of Oracle Database software on the system • If an OSASM group exists. if this is the first installation of Oracle Database software on the system • If an OSOPER group exists. In the following command. enter the following command: # grep OINSTALL_group_name /etc/group If the OINSTALL group does not exist or if you require a new OINSTALL group.1. but you want to give a different group of operating system users database administrative privileges in a new Oracle installation To determine whether the OINSTALL group exists. then create it as follows.2.4. use the group name dba unless a group with that name already exists.3. then create it as follows. for example. In the following command. *Creating an OSOPER Group (Optional) 7 . # /usr/sbin/groupadd asmadmin 2. *Creating an OSASM Group Create an OSASM group only if you want SYSASM as a system privilege that enables the separation of the SYSDBA database administration privilege from the ASM storage administration privilege. if this is the first installation of Oracle Database software on the system • An OINSTALL group exists. for example.1. 2.• If an OSOPER group does not exist. but you want to give a different group of operating system users database operator privileges in a new Oracle installation If you require a new OSOPER group. but you want to give a different group of operating system users database operator privileges in a new Oracle installation To determine whether the OSASM group exists.

8. but its primary group is not oinstall or it is not a member of the appropriate OSDBA or OSOPER groups. you must first create RSA or DSA keys on each cluster node. crs.Set the password of the oracle user: # passwd oracle 2. which must include the OSDBA group and if required. If you want to use the existing user. then start the SSH agent to load keys into memory.dba or dba. which must be the Oracle Inventory group. and then copy all the keys generated on all cluster node members into an authorized keys file that is identical on each node. You need either an RSA or a DSA key for the SSH protocol.To create the oracle user. *Modifying an Oracle Software Owner User If the oracle user exists. the OSOPER group.1. then enter a command similar to the following to modify it. In the home directory of the software owner that you want to use for the installation (crs.5 protocol. then ensure that the user’s primary group is the Oracle Inventory group and that it is a member of the appropriate OSDBA and OSOPER groups.oper] oracle 2. then create it as follows: 1.9. When this is done.1. enter the following command: $ pgrep sshd If SSH is running. *Checking Existing SSH Configuration on the System To determine if SSH is running. use the command ls -al to ensure that the . *Determining Whether an Oracle Software Owner User Exists 8 . as SSH ignores a private key file if it is accessible by others. enter a command similar to the following: # /usr/sbin/useradd -g oinstall -G dba[.uid=440(oracle) gid=200(oinstall) groups=201(dba).202(oper) If the user exists.oper] oracle In this command: • The -g option specifies the primary group. You must configure SSH separately for each Oracle software installation owner that you intend to use for installation. the RSA key is used. while 2. then the response to this command is one or more process ID numbers.1. asm). In the examples that follow. use the user name oracle unless a user with that name already exists. If the Oracle software owner user does not exist or if you require a new Oracle software owner user. *Configuring SSH on Cluster Member Nodes To configure SSH. then determine whether you want to use the existing user or create another oracle user. Note that the SSH files must be readable only by root and by the software installation user (oracle. for example oinstall • The -G option specifies the secondary groups.oper 2. 2. Specify the primary group using the -g option and any required secondary group using the -G option: # /usr/sbin/usermod -g oinstall -G dba[.1.ssh directory is owned and writable only by the user. oracle). *Creating an Oracle Software Owner User In the following procedure.6.1.5.7. RSA is used with the SSH 1. 2.

With OpenSSH. *Add All Keys to a Common authorized_keys File Complete the following steps: 1.. Never distribute the private key to anyone not authorized to perform Oracle software installations.DSA is the default for the SSH 2.502(crs) 3.pub files from each server. 2. using the RSA key. Log in as the software owner (in this example.1. Type "yes" and press ENTER. As the oracle user: $ ssh stnsp001 "cat ~/.pub" >> ~/. To ensure that you are logged in as the Oracle user. For example: [crs@stnsp001 ~]$ id uid=502(crs) gid=501(oinstall) groups=501(oinstall).ssh directory in the crs user's home directory. You will then see the message: 2.11.pub file and the private key to the ~/.137.ssh/authorized_keys $ ssh stnsp002 "cat ~/.10. and set permissions on it to ensure that only the crs user has read and write permissions: [crs@stnsp001 ~]$ mkdir ~/. the crs user). and that the user ID matches the expected user ID you have assigned to the Oracle user.ssh [crs@stnsp001 ~]$ chmod 700 ~/.8. This command writes the RSA public key to the ~/. Repeat steps 1 through 4 on each node that you intend to make a member of the cluster.ssh/authorized_keys * Each time you connect from any node to a new hostname for the first time. Ensure that Oracle user group and user and the terminal window process group and user IDs are identical. and you cannot use SSH1.ssh 4. replacing "stnsp001" with a different node name each time. Combine the contents of the id_rsa. Enter the following command: [crs@stnsp001 ~]$ /usr/bin/ssh-keygen -t rsa At the prompts: • Accept the default location for the key file (press Enter). • Enter and confirm a pass phrase unique for this installation user.502(crs) [crs@stnsp001 ~]$ id crs uid=502(crs) gid=501(oinstall) groups=501(oinstall).215)' can't be established. (You can do this all on one node) run the command below for each node that is a part of the cluster.ssh/id_rsa. enter the commands id and id oracle. you will see a message similar to: "The authenticity of host 'stnsp001 (10. If you have an SSH2 installation.. *To configure SSH.0 protocol. Create RSA Keys On Each Node using the following steps: 1. The instructions that follow are for SSH1. 2.ssh/id_rsa. If necessary.9. you can use either RSA or DSA.pub" >> ~/. then refer to your SSH distribution documentation to configure SSH1 compatibility or configure SSH2 with DSA.ssh/id_rsa. Are you sure you want to continue connecting (yes/no)?" RSA key fingerprint is .1. 2.ssh/id_rsa file. 5.1. *Checking Existing SSH Configuration on the System 9 . create the .

" "oracle@stnsp001's password:" Type password and press ENTER. complete the following procedure.215' (RSA) to the list of known hosts. Enter Yes.ssh/ You are prompted to accept an RSA key.8. where hostname1. 2. where the crs user path is /home/crs: [[crs@stnsp001 crs]$ scp ~/. where xxx represents parts of a valid IP address: [crs@stnsp001 . log in as the crs user.8.ssh/id_rsa': Mon May 19 11:08:13 PST 2008 [crs@stnsp001 crs]$ ssh stnsp001.oracle. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'stnsp001. and you see that the node you are copying to is added to the known_hosts file.xxx.com.us.us.pub keys that you have created. In the .ssh/id_rsa': Mon May 19 11:08:13 PST 2008 2. provide the password for the oracle user.xxx.215)' can't be established. . are the public hostnames (alias and fully qualified domain name) of nodes in the cluster to run SSH from the local node to each node. *Enabling SSH User Equivalency on Cluster Member Nodes After you have copied the authorized_keys file that contains all keys to each node in the cluster.ssh/scp authorized_keys stnsp002:~/.ssh directory on a remote node.1. which should be the same on all nodes in the cluster.ssh/authorized_keys stnsp002:~/.ssh/ crs@stnsp002's password: authorized_keys 100% 828 7. and the file authorized_keys.ssh]$ ~/.1. and so on. Repeat steps 2 from each node to each other member node in the cluster.12.215)' can't be established. in the order listed. The following example is with SCP.xxx. 2. the Oracle Clusterware software owner is named crs: 1. RSA key fingerprint is 7z:60:60:zz:48:48:z1:a0:f7:4e.8. . Enter passphrase for key '/home/crs/.com date The authenticity of host 'stnsp001."Warning: Permanently added 'stnsp001. hostname2. For example: [crs@stnsp001 crs]$ ssh stnsp001 date The authenticity of host 'stnsp001 (xxx. including from the local node to itself.com (xxx.8.oracle.oracle.10.8. with the Oracle Clusterware owner crs.xxx.xxx. 2. and from each node to each other node: [crs@stnsp001 ~]$ ssh stnsp001 date [crs@stnsp001 ~]$ ssh stnsp002 date . you should see the id_rsa.11. RSA key fingerprint is 7z:60:60:zz:48:48:z1:a0:f7:4e. Enter passphrase for key '/home/crs/.ssh directory.215' (RSA) to the list of known hosts.215' (RSA) to the list of known hosts. Your output should be similar to the following. use SCP (Secure Copy) or SFTP (Secure FTP) to copy the authorized_keys file to the oracle user . *Add All Keys to a Common authorized_keys File 10 . on a node called stnsp002.us. The authorized_keys file is copied to the remote node. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'stnsp001.137. When prompted.5MB/s 00:00 3. On the system where you want to run OUI. In this example. Use the following command syntax. On the local node.xxx.

you need to inform the ssh-agent (already running for desktop) of the passphrase so that ssh clients would not be prompted for a passphrase.ssh]$ ssh-add Enter passphrase for /home/crs/. proceed to "Setting Display and X11 Forwarding Configuration" below.ssh/id_dsa: *TYPE YOUR PASSPHRASE* Identity added: /home/oracle/.ssh/id_dsa (/home/oracle/. At the end of this process. 3. 2. the public hostname for each member node should be registered in the known_hosts file for all other cluster member nodes. but your ssh configuration has X11 forwarding enabled. and the local node has only one visual (which is typical).ssh/id_rsa (/home/crs/.[crs@stnsp001 crs]$ ssh stnsp002 date Enter passphrase for key '/home/crs/. and desire password-less login (needed for 10g RAC install). using fake authentication data for X11 forwarding. Korn. *Enabling SSH User Equivalency on Cluster Member Nodes 11 .1. 4. . *Setting Display and X11 Forwarding Configuration • If you are on a remote terminal. and further client requests are satisfied by the ssh-agent.ssh/id_dsa) At the prompt. then verify that the ~/.ssh/id_rsa) These commands start the ssh-agent on the node.ssh]$ exec /usr/bin/ssh-agent $SHELL [crs@stnsp001 .ssh/id_rsa Identity added: /home/crs/. and Bash shells [crs@stnsp001 ~]$ export DISPLAY=hostname:0 2.1. If you are using a remote client to connect to the local node." then this means that your authorized keys file is configured correctly.ssh/authorized_keys file on that node contains the correct public keys. Repeat step 2 on each cluster node member. enter the pass phrase for each key that you generated. For example: [crs@stnsp001 ~]$ ssh stnsp002 date Mon May 19 23:34:42 UTC 2008 [crs@stnsp001 ~]$ ssh stnsp001 date Mon May 19 23:34:48 UTC 2008 [crs@stnsp001 ~]$ ssh stnsp002 If any node prompts for a password or pass phrase.12. then you can now use the ssh or scp commands without being prompted for a password or a pass phrase. then use the following syntax to set the DISPLAY environment variable: Bourne. and you see a message similar to "Warning: No xauth data. Once you notify ssh-agent the passphrase is cached for duration of the GUI desktop login.13. $ ssh-add Enter passphrase for /home/oracle/. To correct this. For example: [crs@stnsp001 .ssh/id_rsa': Mon May 19 11:08:35 PST 2008 . If you have configured SSH correctly. and that you have created an Oracle software owner with identical group membership and IDs. (Optional) If you entered passphrase above. and load the RSA keys into memory so that you are not prompted to use pass phrases when issuing SSH commands.

It should be a static. if you are using the Bash shell. This IP Address should NOT be bound to the public adapter before starting the install. B. Oracle Clusterware is responsible for binding this address. In 2 node environments do not be tempted to use a 'Crossover Cable'. In that case you would need to start with 4 adapters. 1 adapter is the public interface. address • The private IP address. DNS. which should be from a different subnet than the public IP address. or make a note of your SSH pass phrase. It should be a static. some customers even team their public adapters as well.1. which should be from the same subnet as the public IP address and should be recorded in DNS and the hosts file on each node. The other adapter. Using any text editor. and remember to repeat step 4 before you start OUI from a different terminal session. For example: Host * ForwardX11 no • If you are not plannig to run the installer on the console you may need to run the following command to allow the X session to be accessed remotely xhost + 2. It should 2. Ensure you team the adapters before starting installation. *Networking You will need a minimum of 2 network adapters per machine. 2 public & 2 private. if available. then enter the following command: $ export DISPLAY=stnsp001:0 • To ensure that X11 forwarding will not cause the installation to fail. highly available adapter for the interconnect traffic. and if your hostname is stnsp001. not DHCP. You can then take the remaining 3rd adapter and use it for the public interface. *Setting Display and X11 Forwarding Configuration 12 . Crossover cables are not supported You will need a total of 3 IP addresses per node: • The public IP address. Before installation of the Oracle software you take 2 of the adapters on the private interface and team them together to create what looks like a single. create a user-level SSH client configuration file for the Oracle software owner user. not DHCP. Please be aware that even with 2-node cluster you must use a dedicated network switch on the interconnect. The private interface adapters should be connected to each other via a network switch which is used exclusively for the interconnect traffic. as follows: A.ssh/config file. the private interface. This address does not require registering in DNS but you should place an entry in the hosts file on each node.2.13. is used for Oracle RAC node membership and inter-instance traffic. Make sure that the ForwardX11 attribute is set to no.Note: You must run OUI from this session. address • A VIP address. This IP Address should be bound to the private adapter before starting the install. which should be recorded in hosts file on each node and. C shell: [crs@stnsp001 ~]$ setenv DISPLAY hostname:0 For example. This IP Address should be bound to the public adapter before starting the install. edit or create the software installation owner’s ~/. Best practice is to use more than 2 adapters.

137.8 MiB) TX bytes:43154372 (41.8.255.com stnsp001-vip # Also in DNS if available 10.137.6 MiB) TX bytes:398 (398.15.1 MiB) Base address:0x2800 Memory:fe880000-fe8a0000 eth1 eth2 A sample /etc/hosts file # Do not remove the following line.137.8 GiB) TX bytes:138428916 (132.20.oracle. *Networking 13 . and then the network adapter binding order should be checked. not DHCP.20.com stnsp001 # Also in DNS if available 10.252.oracle. address If you do not have a DNS server then make sure both the public and the VIP addresses are entered into all the relevant hosts files that are normally found in the /etc/hosts file.2.171 Bcast:10.com stnsp001-rac It is important to confirm that networking is configured correctly before commencing install.1 localhost.0.248.137.215 stnsp001.2.215 Bcast:10. • Public Ping test Pinging stnsp001 from stnsp001 should return stnsp001's public IP address Pinging stnsp002 from stnsp001 should return stnsp002's public IP address Pinging stnsp001 from stnsp002 should return stnsp001's public IP address Pinging stnsp002 from stnsp002 should return stnsp002's public IP address 2.oracle.us.137.8.0 inet6 addr: fe80::20e:cff:fe08:8f2/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:111936568 errors:0 dropped:0 overruns:0 frame:0 TX packets:617958 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:3027847550 (2.0 b) Base address:0x2840 Memory:fe7e0000-fe800000 Link encap:Ethernet HWaddr 00:04:23:A6:CD:61 inet addr:10. You should ensure that the public IP addresses resolve correctly and that the private addresses are of the form 'nodename-priv' and resolve on both nodes via the hosts file. or various programs # that require network functionality will fail. *Network Ping Tests There are a series of 'ping' tests that should be completed.localdomain localhost 10.us.0.8.137.171 stnsp001-rac.0 inet6 addr: fe80::204:23ff:fea6:cd61/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:41400340 errors:0 dropped:0 overruns:0 frame:0 TX packets:37403509 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:844975237 (805.23.0 MiB) Base address:0x2880 Memory:fe7c0000-fe7e0000 Link encap:Ethernet HWaddr 00:0E:0C:08:08:F3 inet6 addr: fe80::20e:cff:fe08:8f3/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:1041506 errors:0 dropped:0 overruns:0 frame:0 TX packets:5 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:86613582 (82.1. Show Networking setup information Hide Networking setup information You should use ifconfig on each node to confirm the correct IP addresses are bound to the correct adapters /sbin/ifconfig -a eth0 Link encap:Ethernet HWaddr 00:0E:0C:08:08:F2 inet addr:10.255 Mask:255. Show all Hide all 2.us. 127.255.233 stnsp001-vip.137.255 Mask:255.be a static.

At the very least you should ensure that the system clocks on all nodes are as close as possible.2. verify that the kernel parameters shown in the following table are set to values greater than or equal to the recommended value shown. 2. VIPs will be activated at the end of the Oracle Clusterware install. Typically this can be achieved using Network Time Server protocol. and to change them if necessary. semmns. Parameter Value File semmsl 250 /proc/sys/kernel/sem semmns 32000 semopm 100 semmni 128 shmmax Half the size of physical memory (in bytes) /proc/sys/kernel/shmmax shmmni 4096 /proc/sys/kernel/shmmni shmall 2097152 /proc/sys/kernel/shmall file-max 65536 /proc/sys/fs/file-max ip_local_port_range Minimum: 9000 Maximum:65500 /proc/sys/net/ipv4/ip_local_port_range rmem_default 262144 /proc/sys/net/core/rmem_default rmem_max 4194304 /proc/sys/net/core/rmem_max wmem_default 262144 /proc/sys/net/core/wmem_default wmem_max 1048576 /proc/sys/net/core/wmem_max aio-max-nr 1048576 /proc/sys/fs/aio-max-nr To view the current value specified for these kernel parameters. and shmmni file-max 2. If any of the above tests fail you should fix name/address resolution by updating the DNS or local hosts files on each node before continuing with the installation. 2.4. follow these steps: 1. *Time Sync There is a general requirement for Oracle RAC that the time on all the nodes be the same. *Configuring Kernel Parameters On all cluster nodes. Enter the commands shown in the following table to view the current values of the kernel parameters: Parameters semmsl. shmmax.3. and semmni for shmall. *Network Ping Tests Command # /sbin/sysctl -a | grep sem # /sbin/sysctl -a | grep shm # /sbin/sysctl -a | grep file-max 14 . They should not be left to 'auto-negotiate'. *Network Adapter configuration If your network adapters allow configuration you should make sure that they are configured for Full Duplex and at the 'fastest' speed consistent among nodes.2. 2.semopm.1. The procedure following the table describes how to verify and set the values.2.• Private Ping test Pinging stnsp001 private from stnsp001 should return stnsp001's private IP address Pinging stnsp002 private from stnsp001 should return stnsp002's private IP address Pinging stnsp001 private from stnsp002 should return stnsp001's private IP address Pinging stnsp002 private from stnsp002 should return stnsp002's private IP address • VIP Ping test Pinging the VIP address at this point should fail.

d/login file.ip_local_port_range # /sbin/sysctl -a | grep ip_local_port_range rmem_default. Add or edit the following line in the /etc/pam. If the value of any kernel parameter is less than the recommended value. you must increase the following shell limits for the oracle user |Shell Limit |Item in limits.conf file.conf file when it restarts: # /sbin/chkconfig boot. add the following lines to the /etc/profile file (or the file /etc/profile. Note: Include lines only for the kernel parameter values that you want to change.rmem_default = 262144 net. Bash. *Setting Shell Limits for the oracle User To improve the performance of the software on Linux systems.sysctl on 4.shmmni = 4096 kernel. On Red Hat systems. Depending on the oracle user's default shell. rmem_max.shmall = 2097152 kernel.file-max = 65536 net. enter the following command: /sbin/sysctl -p 3. wmem_default.shmmax = 2147483648 kernel. they persist when you restart the system.conf |Hard Limit| Maximum number of open file descriptors nofile 65536 Maximum number of processes available to a single user nproc 16384 To increase the shell limits: 1. if any of the current system parameter values are greater than the recommended values.rmem_max = 4194304 net.sem). and add or edit lines similar to the following: kernel.conf file: oracle oracle oracle oracle soft hard soft hard nproc 2047 nproc 16384 nofile 1024 nofile 65536 2.conf file available in the active kernel memory. *Configuring Kernel Parameters 15 . you must specify all four values.5. set the kernel parameter disable_cap_mlock as follows: disable_cap_mlock = 1 5.ipv4. then keep using the larger values. For the semaphore parameters (kernel.core. create or edit the /etc/sysctl. On SUSE systems only.local on SUSE systems): 2. enter the following command to cause the system to read the /etc/sysctl. or Korn shell. or run the command sysctl -p to make the changes in the /etc/sysctl.core. either restart the computer. then complete the following process: Using any text editor. make the following changes to the default shell startup file: • For the Bourne. Repeat steps 1 and 2 on all other nodes in the cluster. However.conf file. and wmem_max # /sbin/sysctl -a | grep net.sem = 250 32000 100 128 fs.core.4. if it does not already exist: session required /lib/security/pam_limits.core aio-max-nr # /sbin/sysctl -a | grep aio-max-nr 2.wmem_default = 262144 net.core.so 3. After updating the values of kernel parameters in the /etc/sysctl.conf file. to have these changes take effect immediately so that you do not have to restart the system. 2.0 only.wmem_max = 1048576 By specifying the values in the /etc/sysctl. On SUSE Linux Enterprise Server 9.ip_local_port_range = 9000 65500 net. Add the following lines to the /etc/security/limits.

6 enter a command similar to the following to start the module located in the directories of the current kernel version: 2. in seconds. The default value is 60 seconds. 2. The hangcheck_reboot parameter: If the value of hangcheck_reboot is equal to or greater than 1.login file (or the file /etc/csh. The hangcheck_margin parameter: it defines how long the timer waits. then the hangcheck-timer module restarts the system. Log in as root. The hangcheck_tick parameter: it defines how often.local on SUSE systems): if ( $USER == "oracle" ) then limit maxproc 16384 limit descriptors 65536 endif 1. Oracle recommends to set it to 10 (hangcheck_margin=10) 3. It should always be set to 1. then ulimit -p 16384 ulimit -n 65536 else ulimit -u 16384 -n 65536 fi fi • For the C shell (csh or tcsh). If the kernel fails to respond within the sum of the hangcheck_tick and hangcheck_margin parameter values. 2. then the hangcheck-timer module restarts the system. *Setting Shell Limits for the oracle User 16 . If a kernel/device driver hang occurs. then if [ $SHELL = "/bin/ksh" ].6. There are 3 parameters used to control the behavior of the module: 1. Enter the following command on each node to determine which kernel modules are loaded: # /sbin/lsmod 3. *Configuration of the Hangcheck-timer Module Before installing Oracle Real Application Clusters on Linux systems. then the module restarts the node in seconds. If the hangcheck-timer module is not listed for any node: on Kernel 2. add the following lines to the /etc/csh. Oracle recommends to set it to 1 (hangcheck_tick=1). The default value is 180 seconds.1.login. in seconds. If the hangcheck_ reboot parameter is set to zero. verify that the hangcheck-timer module (hangcheck-timer) is loaded and configured correctly. *Hangcheck-timer Module verification procedure 1. the hangcheck-timer checks the node for hangs. and enter the following command to check the kernel version: # uname -a 2.5.6. then the hangcheck-timer module will not restart the node. for a response from the kernel. Repeat this procedure on all other nodes in the cluster. 2.4 enter a command similar to the following to start the module on that node: # /sbin/insmod hangcheck-timer hangcheck_tick=1 hangcheck_margin=10 hangcheck_reboot=1 on Kernel 2.if [ $USER = "oracle" ]. hangcheck-timer monitors the Linux kernel for extended operating system hangs that could affect the reliability of a RAC node and cause a database corruption.

local file. or if a reboot is not possible.ko hangcheck_tick=1 hangcheck_margin=10 hangcheck_reboot=1 In the preceding command example.6.6.d/rc.9-42.local file. the variable kernel_version is the kernel version running on your system that you have got from uname -a. To confirm that the hangcheck module is loaded. Show How to Configure SELinux on OEL5 and RHEL5: Hide Configure SELinux on OEL5 and RHEL5: As per Metalink Note:454196.6.d/boot.9-42. issue as root: # setenforce 0 This will put SELinux into permissive mode immediately.9-42.# insmod /lib/modules/kernel_version/kernel/drivers/char/hangcheck-timer. For a different workaround refer to the note for details.ELsmp ro root=LABEL=/ rhgb quiet selinux=0 initrd /initrd-2. Show How to Ensure latest glibc is installed: Hide Ensure latest glibc is installed: As per Metalink Note:731599.ELsmp) root (hd0. *Platform Specific Setup Show How to Disable RUN_PARALLEL on SLES (SuSE Linux): Hide Disable RUN_PARALLEL on SLES: As per Metalink Note:464061. For example: title Red Hat Enterprise Linux AS (2. ensure your glibc rpm is updated as follows: 2. to avoid false node reboots. • on SUSE: On SUSE systems.ELsmp.6. verify that the local system startup file contains the command shown in the previous step.7.img • Do one of the above and then reboot the server for change to take effect.0) kernel /vmlinuz-2. • Appending kernel boot options: ♦ Edit the kernel boot line and append "selinux=0" (to completely disable). 4. or "enforcing=0" (to set to permissive mode) to the kernel boot options. enter the following command: # lsmod | grep hang The output should be similar to the following: hangcheck_timer 3289 0 5.1. 2. The default setting of "yes" prevents the Oracle Clusterware to start in most cases. or add it if necessary: • on Red Hat: On Red Hat Enterprise Linux systems. add the command to the /etc/init. *Hangcheck-timer Module verification procedure 17 . To ensure that the module is loaded every time the system restarts.1. add the command to the /etc/rc. on Oracle Enterprise Linux 5 & Redhat Enterprise Linux 5 it's required to either disable or set SELinux to permissive mode using one of the methods below: • Editing /etc/selinux/config: ♦ Change the SELINUX value to "SELINUX=disabled" or "SELINUX=permissive".1.1. on SuSE Linux disable RUN_PARALLEL by setting it to "no" in /etc/sysconfig/boot.

You will have separate downloads for : • Oracle Clusterware and • ASM/Database 2.sh stage -post hwos -n stnsp001. As per Metalink Note: 391790.7. The supplied builtin tmpwatch cronjob: /usr/sbin/tmpwatch 720 /var/tmp is safe since it does not delete any socket or special files.9.8.1. Database and Client install into one DVD with one runInstaller program.3. it is recommended to run CVU to verify the nodes are configured correctly: runcluvfy. Unbreakable Linux customers or Redhat Network subscribers can issue: # up2date glibc Refer to above note for further details.stnsp002 -verbose Performing post-checks for hardware and operating system setup 2. Ensure that you use only 32 bit versions of the Oracle Software on a 32bit OS and 64 bit versions of the Oracle Software on a 64bit OS A suggested stage is as follows • Oracle 11g Release 1 EE (11./runcluvfy. Important.com/technology/products/database/clustering/cvu/cvu_download_homepage.4-2.stnsp002 -verbose Show CVU -post hwos output Hide CVU output .6) (1 DVD ) /mnt The 11gR1 now integrates the Oracle Clusterware.• EL4 customers: install glibc-2. 2. contact Novell for fix.html After the hardware and OS has been configured.oracle directory. *Platform Specific Setup 18 . only regular files. CVU can be run from the installation media.5-24 (or above) or upgrade to EL5u2 • SLES10 customers: bug 416838 filed for this problem. *Stage the Oracle Software It is recommended that you stage the required software onto a local drive on Node 1 of your cluster. Show How to disable deletion of critical Clusterware files: Hide deleting critical Clusterware files: The Oracle Clusterware places important socket files in /var/tmp/. *Cluster Verification Utility stage check Now you can run the CVU (Cluster Verification Utility) to check the state of the Operating System configuration.1. ensure that such deletion does not occur.oracle. which can be removed by Linux crontab cleanup jobs.40 (or above) or upgrade to EL4u7 • EL5 customers: install glibc-2. but it is recommended to download the latest version from: http://www. However custom scripts or 3rd party cleanup jobs might cleanup such files and cause Clusterware failures.sh stage -post hwos -n stnsp001.0. If you download the software from OTN you will not get the integrated installer.

20.8.216 stnsp001 eth0:10.216 10.9.8.8.---------------stnsp002:eth1 stnsp001:eth1 yes Result: Node connectivity check passed for subnet "10.0.24.0" Source Destination Connected? -----------------------------.137.24.Checking node reachability.0 eth2 10.-----------.8.1 00:0E:0C:08:08:F2 0.1 00:0E:0C:08:08:F3 0.0 10.1 00:04:23:A6:CD:BD Interface information for node "stnsp001" Interface Name IP Address Subnet ---------------.8.20.137..137.0 eth2 10.24.0 10.0 10.0" with node(s) stnsp002.215 Interfaces found on subnet "10..0 10. Interfaces found on subnet "10.24.stnsp001.137.1 00:0E:0C:08:06:4F 0. Interface information for node "stnsp002" Interface Name IP Address Subnet ---------------.0 10.0.137.8.207 stnsp001 eth1:10.-----------eth0 10.0.137.0 Subnet Gateway Default Gateway Hardware Address -----------.137. *Cluster Verification Utility stage check 19 . Check: Node connectivity of subnet "10.20.stnsp001.137.137.24.137.0.206 Interfaces found on subnet "10.-----------.137.8.215 10.-----------.8.1 00:04:23:A6:CD:61 Check: Node connectivity of subnet "10.1 00:0E:0C:08:06:4E 0.137.0.24. Checking user equivalence.171 10...137.137.0 10.0 eth1 10.137.20. Result: Node connectivity check passed.137.0" with node(s) stnsp002..-----------.0" that are likely candidates for a private interconnect: stnsp002 eth2:10.-----------------------------. 2.-----------eth0 10.20..20.137.-----------0.137...24.-----------------------stnsp001 yes stnsp002 yes Result: Node reachability check passed from node "stnsp001".0" that are likely candidates for a private interconnect: stnsp002 eth1:10.-----------------------stnsp002 passed stnsp001 passed Result: User equivalence check passed for user "oracle11".24.137.0.137.---------------stnsp002:eth2 stnsp001:eth2 yes Result: Node connectivity check passed for subnet "10.171 WARNING: Could not find a suitable set of interfaces for VIPs.137.8.137.8.-----------------------------.0.137. Checking shared storage accessibility.0.20.-----------------------------.137.137.137.0" that are likely candidates for a private interconnect: stnsp002 eth0:10.20.0.8.137.172 stnsp001 eth2:10.137.206 10.24.137.0" Source Destination Connected? -----------------------------.20. Checking node connectivity.0" with node(s) stnsp002.---------------stnsp002:eth0 stnsp001:eth0 yes Result: Node connectivity check passed for subnet "10.137.137. Check: Node reachability from node "stnsp001" Destination Node Reachable? -----------------------------------.8.137.207 10. Check: Node connectivity of subnet "10. Check: User equivalence for user "oracle11" Node Name Comment -----------------------------------.0 Subnet Gateway Default Gateway Hardware Address -----------.0.172 10.0.stnsp001.0" Source Destination Connected? -----------------------------.0.8.-----------0.8.0 eth1 10.137.8.

The output of fdisk –s /dev/sd[b-e] command may look as follows. you must choose the storage option that you want to use for Oracle Clusterware files. • Oracle Cluster Registry (OCR) which contains configuration information about the cluster. Post-check for hardware and operating system setup was successful. You are presented with a bunch of disks from the storage array. voting disk and database files. now configure storage for cluster registry. There are two ways of storing Oracle Clusterware files: • Block or Raw Devices: Oracle Clusterware files can be placed on either Block or RAW devices based on shared disk partitions. /dev/sdb: /dev/sdc: /dev/sdd: /dev/sde: 116924416 116924416 116924416 116924416 In this example we have four 116Gig LUN’s: /dev/sdb /dev/sdc /dev/sdd /dev/sde 116G 116G 116G 116G 3.WARNING: Package cvuqdisk not installed.stnsp001". For example: [root@node1]$ fdisk -s /dev/sdb1 281106 As root. stnsp001 NFS -----------------------------------/nas Sharing Nodes (2 in count) -----------------------stnsp002 stnsp001 Shared storage check was successful on nodes "stnsp002. Oracle recommends using Block devices for easier usage. Automatic Storage Manager (ASM) and Oracle Real Application Clusters databases (Oracle RAC). used to monitor cluster node status. Therefore. Oracle configuration software checks to ensure that devices contain a minimum of 256MB of available disk space. *Prepare the shared storage for Oracle RAC 20 . Oracle Clusterware files include: • Voting disks. Show all Hide all 3. This is due to current fdisk restrictions. the actual device created may be smaller than the size requested. Oracle recommends using at least 280MB for the device size. You can check partition sizes by using the command syntax fdisk -s partition. based on the cylinder geometry of the disk. *Prepare the shared storage for Oracle RAC This section describes how to prepare the shared storage for Oracle RAC For all installations. • A supported shared file system: Supported file systems include the following: ♦ A supported cluster file system ♦ Network File System (NFS): A file-level protocol that enables access and sharing of files When you create partitions with fdisk by specifying a device size. such as +256M. You do not have to use the same storage option for each file type.

1./sbin/fdisk /dev/sdb) 3. default 1) Using default value 1 Last cylinder or +size or +sizeM or +sizeK (1-1024. such as vi.3. default 1024):+280m Command (m for help):w The partition table has been altered! Calling ioctl () to re-read partition table. Log in as root. Login as the root user on the remote nodes and execute the following: [root@stnsp002] # /sbin/partprobe /dev/sdb1 Note: Oracle recommends that you create partitions for Oracle Clusterware files on physically separate disks. Example of Configuring Block Device Storage for Oracle Clusterware The procedure to create partitions for Oracle Clusterware files on block devices is asfollows: 1. Synching disks.1. using the syntax device_partitions:root:oinstall:0640 3. Start a text editor. Change to the /etc/udev/permissions.d directory: # cd /etc/udev/permissions. The user account with which you perform the installation (oracle or crs) must have write permissions to create the files in the path that you specify. Example of Creating a Udev Permissions File for Oracle Clusterware The procedure to create a permissions file to grant oinstall group members write privileges to block devices is as follows: 1. and enter the partition information where you want to place the OCR and voting disk files. log in as root 2. Example of Configuring Block Device Storage for Oracle Clusterware 21 . The following is an example of how to use fdisk to create one partition on a shared storage block disk device for an OCR file: [root@stnsp001] # /sbin/fdisk /dev/sdb The number of cylinders for this disk is set to 1024.1. Use the command syntax /sbin/partprobe diskpath on each node in thecluster to update the kernel partition table for the shared storage device on eachnode.1. Create a new partition. default 1): Using default value 1 Last cylinder or +size or +sizeM or +sizeK (1-1024. 2. 3. and make the partition 280 MB in size for both OCR andvoting disk partitions. Command (m for help): n Command action e extended P primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-1024. 4. Enter the fdisk command to format a specific storage disk (for example.d 1.

stnsp002 -r 11gR1 -verbose Performing pre-checks for cluster services setup Checking node reachability. save the file as 49-oracle. and SUSE Enterprise Server 10 systems.1. sdc5 and sda5.1. 1. and Red Hat Enterprise Linux 4 systems.. Oracle Clusterware Installation and Configuration This page includes hidden sections.. and to grant the Oracle Clusterware owner (in this example crs) permissions to place voting disks on sdb5. save the file as 51-oracle. assign the permissions in the udev file to the devices: # /sbin/udevstart Use the following procedure above to create additional partitions to use for the OCR. add the following information to the file: # OCR disks sdb1:root:oinstall:0640 sdc1:root:oinstall:0640 # Voting disks sdb5:crs:oinstall:0640 sdc5:crs:oinstall:0640 sdd5:crs:oinstall:0640 1. Check: Node reachability from node "stnsp001" Destination Node Reachable? 3. • On Asianux 3. Refer to the OS documentation for additional information on using the fdisk command.1. 3.Note that Oracle recommends that you place the OCR and the voting disk files on separate physical disks. to grant oinstall members access to SCSI disks to place OCR files on sdb1 and sdc1. Voting. verify the nodes are configured correctly for the CRS install runcluvfy. Enterprise Linux 4. on SuSE Linux SLES9 disable HOTPLUG_USE_SUBFS by setting it to "no" in /etc/sysconfig/hotplug.permissions. 4.stnsp002 -r 11gR1 -verbose Show CVU -pre crsinst output Hide CVU output runcluvfy. For example. Save the file: • On Asianux 2. Enterprise Linux 5. CVU Pre Oracle Clusterware install check Prior to installing CRS.1. Oracle Clusterware is an essential component of the Oracle RAC database infrastructure. The default setting of "yes" causes problems for multipath devices.1. Using the following command. use the and image for each section to show/hide the section Show all Hide all or you can 'Show' or 'Hide' all by selecting here : Next we will install the Oracle Clusterware layer.sh stage -pre crsinst -n stnsp001.sh stage -pre crsinst -n stnsp001. Red Hat Enterprise Linux 5.2. Oracle Clusterware does not require any other clustering software.permissions. and ASM disks. 4. Example of Creating a Udev Permissions File for Oracle Clusterware 22 . Platform Specific Settings Show How to Disable HOTPLUG_USE_SUBFS on SLES9 (SuSE Linux): Hide Disable HOTPLUG_USE_SUBFS on SLES9: As per Metalink Note:357472.

0 Subnet Gateway Default Gateway Hardware Address -----------.. Check: Membership of user "oracle11" in group "oinstall" [as Primary] Node Name User Exists Group Exists User in Group Primary Comment ---------------. Checking node connectivity.137.---------------stnsp002:eth0 stnsp001:eth0 yes Result: Node connectivity check passed for subnet "10. Check: Existence of group "oinstall" Node Name Status Group ID -----------.0" that are likely candidates for a private interconnect: stnsp002 eth2:10.8.137...-----------.8.-----------------------stnsp002 yes passed stnsp001 yes passed Result: User existence check passed for "oracle11".0" that are likely candidates for a private interconnect: stnsp002 eth0:10.8.137.-----------------------stnsp002 exists 1422 stnsp001 exists 1422 Result: Group existence check passed for "oinstall".137.20.137.137.0 10.137. Check: Node connectivity of subnet "10.-----------------------.137.1 00:04:23:A6:CD:BD Interface information for node "stnsp001" Interface Name IP Address Subnet ---------------.215 10.171 10.137.0.0.0 eth2 10. Check: User equivalence for user "oracle11" Node Name Comment -----------------------------------.-----------eth0 10.-----------.0 10..0" with node(s) stnsp002.-----------------------stnsp002 passed stnsp001 passed Result: User equivalence check passed for user "oracle11". Checking user equivalence.-----------stnsp002 yes yes yes yes passed stnsp001 yes yes yes yes passed Result: Membership check for user "oracle11" in group "oinstall" [as Primary] passed.-----------0.1 00:0E:0C:08:06:4E 0.8.8.137.0" with node(s) stnsp002.-----------------------stnsp001 yes stnsp002 yes Result: Node reachability check passed from node "stnsp001".8. CVU Pre Oracle Clusterware install check 23 .-----------.216 10.0 eth2 10.-----------------------------------.8.0..8.1 00:04:23:A6:CD:61 Check: Node connectivity of subnet "10.-----------.0" Source Destination Connected? -----------------------------.137.8.137.137.8.20.8.216 stnsp001 eth0:10.20.172 stnsp001 eth2:10..0.20.1.215 Interfaces found on subnet "10.-----------------------. Administrative privileges check passed.20.137.-----------eth0 10.0.-----------.8. Check: Existence of user "oracle11" Node Name User Exists Comment -----------.20.0 Subnet Gateway Default Gateway Hardware Address -----------.1 00:0E:0C:08:08:F2 0.0. Interface information for node "stnsp002" Interface Name IP Address Subnet ---------------.20.-----------0.137.137.stnsp001.20.---------------stnsp002:eth2 stnsp001:eth2 yes Result: Node connectivity check passed for subnet "10.0.-----------.0 10.0 10.137.137. Interfaces found on subnet "10.137.0.0" Source Destination Connected? -----------------------------.137.137. Checking administrative privileges.137.-----------------------------.172 10.-----------------------------.171 4.-----------.8.stnsp001.20.-----------.

2" Node Name Status Comment -----------------------------.5" Node Name Status -----------------------------. Checking system requirements for 'crs'.-----------------------stnsp002 235.2-18 passed stnsp001 binutils-2..9-34.92.5GB (1572864KB) 1.15.1.95GB (6234256KB) Result: Total memory check passed.6.4.92. Check: Kernel version Node Name Available -----------.3.-----------------------stnsp002 5.3.15. Check: Package existence for "gcc-3.4.1.80-5 Result: Package existence check passed for "make-3.-----------------------------stnsp002 make-3.3. Check: Swap space Node Name Available -----------.0.80" Node Name Status -----------------------------.14GB (210907900KB) Result: Free disk space check passed.5GB (1572864KB) Comment ---------passed passed Required -----------------------i686 i686 Comment ---------passed passed Required -----------------------2.-----------------------------stnsp002 gcc-3.15.3.95GB (6234256KB) stnsp001 5.91GB (6192456KB) Result: Swap space check passed.0.-----------------------stnsp002 2.6.-----------------------stnsp002 i686 stnsp001 i686 Result: System architecture check passed.5-2 stnsp001 gcc-3.9-34. Check: Package existence for "libaio-devel-0.---------------stnsp002 binutils-2.80". Result: Node connectivity check passed.9 Comment ---------passed passed Check: Package existence for "make-3.0.5-2 Result: Package existence check passed for "gcc-3. Check: Package existence for "libaio-0.-----------------------------stnsp002 libaio-devel-0.2".0. CVU Pre Oracle Clusterware install check 24 .-----------------------------.83GB (247283204KB) stnsp001 201.5".-----------------------------stnsp002 libaio-0.0.4.105".-----------------------stnsp002 5.3.0.ELsmp Result: Kernel version check passed. Comment ---------------passed passed Check: Package existence for "binutils-2.ELsmp stnsp001 2.6. Check: Total memory Node Name Available -----------.. Check: Free disk space in "/tmp" dir Node Name Available -----------.105-2 Result: Package existence check passed for "libaio-0.6.1.0.15.92.9 2.91GB (6192456KB) stnsp001 5.2-18 passed Result: Package existence check passed for "binutils-2.WARNING: Could not find a suitable set of interfaces for VIPs.105" Node Name Status -----------------------------.0.105-2 Comment ---------------passed passed Comment ---------------passed passed Comment ---------------passed 4. Check: System architecture Node Name Available -----------.105" Node Name Status -----------------------------.11.80-5 stnsp001 make-3.11.105-2 stnsp001 libaio-0.92.4.3. Required -----------------------1GB (1048576KB) 1GB (1048576KB) Comment ---------passed passed Required -----------------------400MB (409600KB) 400MB (409600KB) Comment ---------passed passed Required -----------------------1.

2.2.2.19" Node Name Status -----------------------------. Comment ---------------passed passed Comment ---------------passed passed Check: Package existence for "libstdc++-devel-3.-----------------------------stnsp002 libstdc++-3.5" Node Name Status -----------------------------. Check: Package existence for "glibc-2.4-2.11" Node Name Status Comment -----------------------------.3.-----------------------stnsp002 exists passed stnsp001 exists passed Result: Group existence check passed for "oinstall".RHEL4.5". Check: User existence for "nobody" Node Name Status -----------.1 passed Result: Package existence check passed for "unixODBC-devel-2. CVU Pre Oracle Clusterware install check 25 . Check: Package existence for "unixODBC-2. Check: Package existence for "libgcc-3.-----------------------stnsp002 exists passed stnsp001 exists passed Result: Group existence check passed for "dba".4-2.-----------------------.1 Result: Package existence check passed for "unixODBC-2.4.rhel4 stnsp001 sysstat-5.stnsp001 libaio-devel-0.5-2 Result: Package existence check passed for "libgcc-3. Comment ---------------passed passed Check: Package existence for "elfutils-libelf-devel-0.97" Node Name Status Comment -----------------------------.1.-----------------------------stnsp002 unixODBC-2.5-2 Result: Package existence check passed for "libstdc++-3.RHEL4.3.4.0.5" Node Name Status -----------------------------.0.4.4-2.19 stnsp001 glibc-2.1 stnsp001 unixODBC-2.3.-----------------------.97-5 passed stnsp001 elfutils-libelf-devel-0.-----------------------------.4-2.0. Check: Group existence for "dba" Node Name Status Comment -----------.4.rhel4 Result: Package existence check passed for "sysstat-5.4.-----------------------------stnsp002 sysstat-5.11-1.3.4.-----------------------------.2.4.5" Node Name Status -----------------------------.-----------------------------.------------------------ Comment ---------------passed passed Comment ------------------------ 4. Check: Group existence for "oinstall" Node Name Status Comment -----------.11" Node Name Status -----------------------------.11-1.---------------stnsp002 libstdc++-devel-3.5".97-5 passed Result: Package existence check passed for "elfutils-libelf-devel-0.2.5-7. Check: Package existence for "libstdc++-3.11".-----------------------------stnsp002 glibc-2. Comment ---------------passed passed Check: Package existence for "unixODBC-devel-2.11-1.3.---------------stnsp002 unixODBC-devel-2.-----------------------------stnsp002 libgcc-3.5-2 stnsp001 libstdc++-3.5-2 passed Result: Package existence check passed for "libstdc++-devel-3.3.RHEL4.4.5-2 stnsp001 libgcc-3.19".19 Result: Package existence check passed for "glibc-2.2.2.5" Node Name Status Comment -----------------------------.5".105". Check: Package existence for "sysstat-5.5-7.4.11-1.5".97".---------------stnsp002 elfutils-libelf-devel-0.5-2 passed stnsp001 libstdc++-devel-3.1 passed stnsp001 unixODBC-devel-2.11".0.RHEL4.2.4.4.105-2 passed Result: Package existence check passed for "libaio-devel-0.4.

Show all Hide all 4. Client and Clusterware components • Action ♦ Select the Oracle Clusterware radio button ♦ Click Next 4.2. Oracle Clusterware Install Start the installer by running "runInstaller" from the staged installation media./runInstaller • Notes ♦ Notice that the Oracle 11g Installer now combines the Oracle Database.2. .stnsp002 exists passed stnsp001 exists passed Result: User existence check passed for "nobody". Oracle Clusterware Install 26 . System requirement passed for 'crs' Pre-check for cluster services setup was successful.

2. Oracle Clusterware Install 27 .• Notes ♦ The OUI will name the Oracle Clusterware Home 'OraCrs11g_home'. ♦ If you change this you should make sure that the name you use is unique • Actions ♦ Specify a location for the Oracle Clusterware Home ♦ Click Next 4.

The defaults are: ◊ Public Node Name : must resolve via hosts and or DNS to the public IP address and must be live ◊ Private Node Name : must resolve via hosts to the interconnect IP address and must be live ◊ Virtual Host Name : must resolve via hosts and or DNS to a new IP address and must not be live ◊ If these are not correct select the node entry and click Edit. If there are issues you should rectify them before continuing • Actions ♦ Click Next • Notes ♦ Each Cluster requires a name.. to modify OR ♦ Click Add. this should be unique within your organisation. 'nodename-priv'. The installer will default names for the node it is running on.. 'nodename-vip' • Actions ♦ Confirm the Cluster Name selected is acceptable ♦ Confirm the Details for the current node are OK.. You must add other nodes manually ♦ Oracle defaults the names to 'nodename'.• Notes ♦ The installer will validate the state of the cluster before continuing. to add more nodes to the cluster 4. Oracle Clusterware Install 28 .. The default is a substring of the node name followed by _cluster ♦ This is where you specify details of all the nodes in the cluster.2.

Oracle Clusterware Install 29 .2.• Notes ♦ Here you specify the details of the node you wish to add to the cluster nodes list • Actions ♦ Enter the new node details ◊ Public Node Name : must resolve via hosts and or DNS to the public IP address and must be live ◊ Private Node Name : must resolve via hosts to the interconnect IP address and must be live ◊ Virtual Host Name : must resolve via hosts and or DNS to a new IP address and must not be live ♦ Click OK to return to the node list for the cluster 4.

.2.. cycle • Actions ♦ Click Next • Notes 4. Oracle Clusterware Install 30 .• Notes ♦ If you have more nodes repeast the Add.

.♦ The installer lists all the Network adapters. • Actions ♦ Select the Adapter eth0 ♦ Click Edit. which is incorrect. If it guesses incorrectly you must change the usage. • Notes ♦ Here you specify the ‘Interface type’ Oracle will use this specific network for • Actions ♦ Select the Interface Type : Public radio button ♦ Click OK 4. The installer will try and guess the use of an adapter based on the IP address bound. You should have one Adapter correctly identified as type 'Public' and at least one adapter correctly identified as type 'Private'.2. Oracle Clusterware Install 31 . Here it has guessed that all adapters are Private..

2.Type : Private • Actions ♦ Click Next 4. Oracle Clusterware Install 32 . set to ‘do not use’. Other adapters.• Notes ♦ Here you can see we have successfully configured the network adapter usage. Ideally you will have only 1 adapter set as public and 1 adapter set as private. If you have multiple public or multiple private adapters it is better to team them at the OS adapter driver level before commencing the install. if available.

Also you will have an additional 3 vote devices to protect your cluster from the failure of a single vote device. • Actions ♦ Select the Normal radio button ♦ Enter the device to be used for the First OCR ♦ Enter the device to be used for the Second OCR ♦ Click Next 4.• Notes ♦ Here we specify the shared storage devices that will be used by Oracle Clusterware. Ideally you will have 2 devices for the OCR.2. Oracle Clusterware Install 33 . Oracle will mirror to these devices to protect you from a single OCR device failure.

• Notes ♦ Next we specify the devices to be used for the Oracle Clusterware vote disks • Actions ♦ Select the Normal radio button ♦ Enter the device to be used for the First vote disk ♦ Enter the device to be used for the Second vote disk ♦ Enter the device to be used for the Third vote disk ♦ Click Next 4.2. Oracle Clusterware Install 34 .

2. Oracle Clusterware Install 35 .• Notes ♦ the installer lists a summary of the planned actions • Actions ♦ Click Install • Notes 4.

2.♦ the installer installs the software onto the local node • Actions ♦ none required • Notes ♦ the installer installs the software onto the remote node • Actions ♦ none required 4. Oracle Clusterware Install 36 .

Now formatting voting device: /dev/sdb5 Now formatting voting device: /dev/sdc5 Now formatting voting devices /dev/sdd5 Format of 3 voting devices complete. • Notes ♦ The output from the root. stnsp002 Local node checking complete. Operation successful.0/crs/root. Run root.sh script should be similar to this 4. Adding daemons to inittab Expecting the CRS daemons to be up within 600 seconds. Changing owner to root Successfully accumulated necessary OCR kegs..0' is not owned by root.0' is not owned by root Checking to see if Oracle CRS stack is already configured Setting the permissions on OCR backup directory Setting up Network socket directories Oracle Cluster Registry configuration upgraded successfully The directory '/scratch/11. Cluster Synchronization Services is active on these nodes. Startup will be queued to init within 30 seconds.• Notes ♦ The installer requires commands to be run as root on each of the nodes ♦ Be sure to follow directions carefully. privgrp 'root'. Oracle Clusterware Install 37 . Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.sh WARNING: directory '/scratch/11. running scripts one at a time.sh on remaining nodes to start CRS daemons.1. on each node • Action ♦ On the first node open a root shell window and run the root. stnsp001 Cluster Synchronization Services is inactive on these nodes.00# /scratch/11.sh script bash-3.1.2.1. node <nodenunber>: <nodenane> <private interconnect name> <hostnane> node 1: stnsp001 stnsp001-rac stnsp001 node 2: stnsp002 stnsp002-rac stnsp002 Creating OCR kegs for user 'root'.

clscfg: version 4 is 11 Release 1.. Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.. nodes... nodes.1. nodes.0/crs/root.0' is not owned be root. Waiting for the Oracle CRSD and EVMD to start Waiting for the Oracle CRSD and EVMD to start Oracle CRS stack installed and running under init(lM) Running vipca(silent) for configuring nodeapps Creating Creating Creating Creating Creating Creating Done. nodes...0' is not owned by root Checking to see if Oracle CRS stack is already configured Setting the permissions on OCR backup directory Setting up Network socket directories Oracle Cluster Registry configuration upgraded successfully The directory '/scratch/11.1. node <nodenunber>: <nodenane> <private interconnect name> <hostnane> node 1: stnsp001 stnsp001-rac stnsp001 node 2: stnsp002 stnsp002-rac stnsp002 clscfg: Arguments check out successfully.1. Successfully accumulated necessary OCR keys.. Adding deamons to inittab Expecting the CRS daemons to be up within 600 seconds.. Cluster Synchronization Services is active on these nodes. -force is destructive and will destroy any previous cluster configuration. stnsp001 stnsp002 Cluster Synchronization Services is active on all the nodes.sh WARNING: directory '/scratch/11... • Notes ♦ The output from the root.♦ here the Oracle Clusterware is configured ♦ This may take some time to run • Action ♦ open a root shell on the first node ♦ run the identified command as root on the first node ♦ You must wait for the command to complete before continuing bash-3. NO KEYS WERE WRITTEN.00# /scratch/11. Oracle Cluster Registry for cluster has already been initialized Startup will be queued to init within 30 seconds. nodes. Changing owner to root clscfg: EXISTING configuration version 4 detected..sh script should be similar to this ♦ here the Oracle Clusterware is configured ♦ This may take some time to run • Action ♦ open a root shell on the second node ♦ run the identified command as root on the second node ♦ You must wait for the command to complete before continuing 4. VIP GSD ONS VIP GSD ONS application application application application application application resource resource resource resource resource resource on on on on on on (2) (2) (2) (2) (2) (2) nodes.. Supply -force parameter to override. Oracle Clusterware Install 38 .2.

Oracle Clusterware Install 39 .2.• Notes ♦ You can now contnue with the install • Action ♦ return to the installer ♦ Click OK 4.

Oracle Clusterware Install 40 .2.• Notes ♦ a series of configuration assistants are run at the end of the install • Actions ♦ none required • Notes ♦ The installer has completed installation and configuration of Oracle Clusterware • Actions ♦ Click Exit 4.

002.2.2..1.0./crs_stat -t Name Type Target State Host -----------------------------------------------------------ora.1.. Verify cluster resources are online [oracle11@stnsp001 bin]$ cd /scratch/11.002.vip application ONLINE ONLINE stnsp002 • Notes ♦ You can see the resources configured inside Oracle Clusterware • Action ♦ change directory to Oracle Clusterware home bin directory ♦ run the .0...ons application ONLINE ONLINE stnsp001 ora./cluvfy stage -post crsinst -n stnsp001.001..gsd application ONLINE ONLINE stnsp002 ora.001.stnsp002 -verbose 4.0/crs/bin [oracle11@stnsp001 bin]$ ........2...2.0.. Verify cluster resources are online 41 .002.vip application ONLINE ONLINE stnsp001 ora./crs_stat -t comand 4.• Actions ♦ Click Yes 4. Use CVU to verify the Oracle Clusterware install Verify the Oracle Clusterware installation using the Oracle Cluster Verification Utility: ..1..ons application ONLINE ONLINE stnsp002 ora.001..gsd application ONLINE ONLINE stnsp001 ora.

2. Checking data integrity of OCR./cluvfy stage -post crsinst -n stnsp001. OCR integrity check passed.. Check: Liveness for "CRS daemon" Node Name Running -----------------------------------.. local-only configurations.stnsp002 Performing post-checks for cluster services setup Checking node reachability..Show CVU -post crsinst output Hide CVU output -verbose .. Checking cluster integrity... Checking user equivalence.. Checking daemon liveness. All nodes free of non-clustered.. Checking the version of OCR. Node Name -----------------------------------stnsp001 stnsp002 Cluster integrity check passed Checking OCR integrity.. Use CVU to verify the Oracle Clusterware install 42 . Check: User equivalence for user "oracle11" Node Name Comment -----------------------------------. 4.. Checking the absence of a non-clustered configuration. Checking daemon liveness... Node Name Status -----------------------------------... OCR of correct Version "2" exists. Check: Node reachability from node "stnsp001" Destination Node Reachable? -----------------------------------..-----------------------stnsp001 yes stnsp002 yes Result: Node reachability check passed from node "stnsp001".0.. Cluster manager integrity check passed. Checking Cluster manager integrity..2. Data integrity check for OCR passed..-----------------------stnsp002 passed stnsp001 passed Result: User equivalence check passed for user "oracle11"... Checking CSS daemon..-----------------------stnsp002 running stnsp001 running Result: Daemon status check passed for "CSS daemon"... Uniqueness check for OCR device passed..-----------------------stnsp002 yes stnsp001 yes Result: Liveness check passed for "CRS daemon". Checking CRS integrity.

-----------------------. CRS OK? -----------------------yes yes Comment ---------passed passed Comment ---------passed passed Comment ---------passed passed Post-check for cluster services setup was successful.-----------------------stnsp002 no exists stnsp001 no exists Result: Check passed.-----------------------stnsp002 yes stnsp001 yes Result: Liveness check passed for "EVM daemon". The patchset can be downloaded from metalink. Checking node application existence.1.. Oracle Clusterware patching 43 .7. CRS integrity check passed. Checking existence of VIP node application Node Name Required Status -----------. Checking existence of GSD node application Node Name Required Status -----------... In this section we will patch the Oracle Clusterware to the latest release of Oracle 11gR1 .0. Checking existence of ONS node application Node Name Required Status -----------.. Show all Hide all 5. Liveness of all Node Name -----------stnsp002 stnsp001 the daemons CRS daemon -----------------------yes yes CSS daemon -----------------------yes yes EVM daemon ---------yes yes Checking CRS health..1. Oracle Clusterware patching At this point we have installed Oracle Clusterware 11.6.-----------------------stnsp002 no exists stnsp001 no exists Result: Check passed.Check: Liveness for "CSS daemon" Node Name Running -----------------------------------.-----------------------. Checking daemon liveness. you should always upgrade Oracle Clusterware before you upgrade any other 5.. Therefore.-----------------------stnsp002 yes exists stnsp001 yes exists Result: Check passed.11.-----------------------.0.-----------------------stnsp002 yes stnsp001 yes Result: Liveness check passed for "CSS daemon". Check: Health of CRS Node Name -----------------------------------stnsp002 stnsp001 Result: CRS health check passed. Check: Liveness for "EVM daemon" Node Name Running -----------------------------------. Note: The Oracle Clusterware software must be at the same or newer level as any other ORACLE_HOME directory.

1.0] Enter the following commands to start Oracle Universal Installer./runInstaller • Notes ♦ The installer appears • Actions ♦ Click Next 5.1.0.0. Oracle Clusterware patching 44 .1.6. where patchset_directory is the directory where you unpacked the patch set software: $ cd patchset_directory/Disk1 $ . before you upgrade Oracle Clusterware home directory. This patch does not allow you to upgrade other ORACLE_HOME directories.0/crs/bin/crsctl query crs softwareversion Oracle Clusterware version on node [stnsp001] is [11.0/crs/bin/crsctl query crs activeversion Oracle Clusterware active version on the cluster is [11.ORACLE_HOME directory. We will apply this patchset in a non rolling upgrade fashion. Before we start we can query the clusterware versions #/scratch/11.1.6.0] #/scratch/11.

Oracle Clusterware patching 45 .• Notes ♦ You should ensure that the Clusterware home is selected in the first drop down list box ♦ The installer should default the directory to the correct location • Actions ♦ Ensure the correct home is selected ♦ Click Next 5.

Oracle Clusterware patching 46 .• Notes ♦ the installer detects that this is a clustered home and automatically selects all the nodes in the cluster • Actions ♦ Click Next • Notes ♦ Some parameters are validated by the installer • Actions ♦ Click Next 5.

Oracle Clusterware patching 47 .• Notes ♦ This is a summary of the actions the installer will complete • Actions ♦ Click Install • Notes 5.

5.sh script. Recopying init and RC scripts.♦ the installer stages the patch on all the nodes in the cluster • Actions ♦ No action required • Notes ♦ At the end the installer lists the mandatory steps that must be completed to apply this patch • Actions ♦ Log in as the root user and enter the following command to shut down the Oracle Clusterware: # CRS_home/bin/crsctl stop crs • ♦ Run the root111. Oracle Clusterware patching 48 .sh output (all nodes except last) bash-3. It will automatically start the Oracle Clusterware on the patched node: # CRS_home/install/root111. clscfg: version 4 is 11 Release 1. Relinking of patched files is complete.sh Creating pre-patch directory for saving pre-patch clusterware files Completed patching clusterware files to /opt/crs Relinking some shared libraries. .00# /scratch/11. clscfg: EXISTING configuration version 4 detected. This may take a while on some systems. Starting up the CRS daemons. Preparing to recopy patched init and RC scripts.0/crs/install/root111. 11107 patch successfully applied.sh output (all nodes except last) Hide root111.1. Startup will be queued to init within 30 seconds.sh Show root111. Waiting for the patched CRS daemons to start.

Startup will be queued to init within 30 seconds. Shutdown request successfully issued. $ CRS_home/bin/crsctl check crs Cluster Synchronization Services appears healthy Cluster Ready Services appears healthy Event Manager appears healthy # /scratch/11.0/crs/install/root111.. Waiting for Cluster Synchronization Services daemon to stop Cluster Synchronization Services daemon has stopped Starting Oracle clusterware Attempting to start Oracle Clusterware stack Waiting for Cluster Synchronization Services daemon to start Waiting for Cluster Synchronization Services daemon to start Waiting for Cluster Synchronization Services daemon to start Waiting for Cluster Synchronization Services daemon to start Cluster Synchronization Services daemon has started Event Manager daemon has started Cluster Ready Services daemon has started Show root111. This could take several minutes.0/crs/bin/crsctl query crs softwareversion Oracle Clusterware version on node [stnsp001] is [11. Relinking of patched files is complete. Oracle Clusterware patching 49 . When done.1.1. Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897. clscfg: version 4 is 11 Release 1. Successfully stopped Oracle Clusterware resources Stopping Cluster Synchronization Services. this will achieve a rolling upgrade of the Oracle Clusterware. clscfg: EXISTING configuration version 4 detected. . clscfg -upgrade completed successfully Creating '/scratch/11. Operation successful.7.0.sh output on last node bash-3. Operation successful. Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.0] 5. clscfg -upgrade completed successfully Creating '/scratch/11.crs' with data used for CRS configuration Setting CRS configuration values in /scratch/11.1.1.1.7.0/crs/install/paramfile.1.0/crs/install/paramfile.0/crs/install/paramfile. This may take a while on some systems.Successfully accumulated necessary OCR keys. 11107 patch successfully applied. Shutting down the Cluster Synchronization Services daemon.crs Setting cluster unique identifier Restarting Oracle clusterware Stopping Oracle clusterware Stopping resources.1.sh output on last node Hide root111. verify the Oracle Clusterware is running on all nodes before exiting the installer. privgrp 'root'.1.00# /scratch/11.crs Continue with these two steps on all nodes.0/crs/install/paramfile. one by one.0. Waiting for the patched CRS daemons to start.sh Creating pre-patch directory for saving pre-patch clusterware files Completed patching clusterware files to /opt/crs Relinking some shared libraries.0] # /scratch/11.crs' with data used for CRS configuration Setting CRS configuration values in /scratch/11. Successfully accumulated necessary OCR keys. Starting up the CRS daemons.. node <nodenumber>: <nodename> <private interconnect name> <hostname> node 2: stnsp002 stnsp002-priv stnsp002 Creating OCR keys for user 'root'. privgrp 'root'. Preparing to recopy patched init and RC scripts. Recopying init and RC scripts. node <nodenumber>: <nodename> <private interconnect name> <hostname> node 1: stnsp001 stnsp001-priv stnsp001 Creating OCR keys for user 'root'.1.0/crs/bin/crsctl query crs activeversion Oracle Clusterware active version on the cluster is [11.

-----------------------stnsp001 yes stnsp002 yes Result: Node reachability check passed from node "stnsp001".. CVU check before creating ASM instance Verify the cluster is configured correctly for an instance creation . Oracle Clusterware patching complete 50 ..1..1. Check: Node reachability from node "stnsp001" Destination Node Reachable? -----------------------------------.• Notes ♦ Once the Oracle Clusterware is running on all nodes.stnsp002 -r 11gR1 -verbose Performing pre-checks for database installation Checking node reachability. Oracle ASM Home Software Install 6. Oracle Clusterware patching complete 6. Checking user equivalence.sh stage -pre dbinst -n stnsp001./runcluvfy./runcluvfy. exit the installer • Actions ♦ Click Yes 5.stnsp002 -r 11gR1 -verbose Show CVU -pre dbinst output Hide CVU output .sh stage -pre dbinst -n stnsp001..1. 5.

.-----------.8.8.0.0 eth0 10.8.24.8.206 10.-----------------------------stnsp002:eth0 stnsp002:eth0 stnsp002:eth0 stnsp001:eth0 stnsp002:eth0 stnsp001:eth0 stnsp002:eth0 stnsp001:eth0 stnsp002:eth0 stnsp001:eth0 stnsp001:eth0 stnsp001:eth0 Connected? ---------------yes yes yes yes yes yes 6.1.137.0 10.0.137.0. Checking node connectivity.215 10.0 Subnet Gateway Default Gateway Hardware Address -----------.-----------eth0 10.137.0 Subnet Gateway Default Gateway Hardware Address -----------..0 eth1 10.137.20.20.0.-----------------------stnsp002 exists 8500 stnsp001 exists 8500 Result: Group existence check passed for "dba". Check: Membership of user "oracle11" in group "dba" Node Name User Exists Group Exists User in Group Primary ---------------.-----------------------.1 00:0E:0C:08:06:4E 0.8.8.137.0 10.137.137.8.8.0 10.0 eth2 10.-----------.137.137.216 10.137.-----------stnsp002 yes yes yes yes passed stnsp001 yes yes yes yes passed Result: Membership check for user "oracle11" in group "oinstall" [as Primary] passed.137.0 eth2 10.-----------.0.137.1 00:04:23:A6:CD:61 Check: Node connectivity of subnet "10.-----------0.137.20.-----------------------.207 10.0" Source Destination -----------------------------.8.137.-----------------------.-----------------------stnsp002 yes passed stnsp001 yes passed Result: User existence check passed for "oracle11"..1 00:04:23:A6:CD:BD Interface information for node "stnsp001" Interface Name IP Address Subnet ---------------.Check: User equivalence for user "oracle11" Node Name Comment -----------------------------------.0 10.24.-----------. Checking administrative privileges. Check: Existence of group "oinstall" Node Name Status Group ID -----------.-----------stnsp002 yes yes yes no stnsp001 yes yes yes no Result: Membership check for user "oracle11" in group "dba" passed.8.0.0 10.8.1 00:0E:0C:08:08:F3 0.137.0 eth1 10. Administrative privileges check passed.137.0.-----------.0 10.1 00:0E:0C:08:08:F2 0. Comment -----------passed passed Interface information for node "stnsp002" Interface Name IP Address Subnet ---------------.0.1 00:0E:0C:08:08:F2 0.-----------.0 eth0 10.20.-----------------------stnsp002 passed stnsp001 passed Result: User equivalence check passed for user "oracle11".0 10.0.-----------.137.172 10.1 00:0E:0C:08:06:4F 0.137.24.-----------------------stnsp002 exists 1422 stnsp001 exists 1422 Result: Group existence check passed for "oinstall".233 10.137.137.234 10. Check: Existence of group "dba" Node Name Status Group ID -----------.0.8.-----------.0.8.0.137.0.171 10.8.137.8.0 10.-----------eth0 10.0.8. CVU check before creating ASM instance 51 . Check: Existence of user "oracle11" Node Name User Exists Comment -----------.137.24.-----------.0. Check: Membership of user "oracle11" in group "oinstall" [as Primary] Node Name User Exists Group Exists User in Group Primary Comment ---------------.137.137.8.-----------0.-----------.1 00:0E:0C:08:06:4E 0.-----------..0.

80-5 Result: Package existence check passed for "make-3.137. Check: Kernel version Node Name Available -----------.1.137.137. Comment ---------------passed passed 6.8.234 stnsp001 eth0:10.216 eth0:10.137.stnsp001. Required -----------------------1GB (1048576KB) 1GB (1048576KB) Comment ---------passed passed Required -----------------------400MB (409600KB) 400MB (409600KB) Comment ---------passed passed Required -----------------------1.93GB (252637044KB) stnsp001 240.91GB (6192456KB) stnsp001 5.-----------------------------stnsp002 make-3.137.---------------stnsp002:eth2 stnsp001:eth2 yes Result: Node connectivity check passed for subnet "10.137.0" with node(s) stnsp002.95GB (6234256KB) Result: Total memory check passed.20.137.6.-----------------------stnsp002 5.137.0" that are likely candidates for a private interconnect: stnsp002 eth1:10.80" Node Name Status -----------------------------.-----------------------stnsp002 240.137.0" with node(s) stnsp002.95GB (6234256KB) stnsp001 5.-----------------------stnsp002 i686 stnsp001 i686 Result: System architecture check passed.11.5GB (1572864KB) Comment ---------passed passed Required -----------------------i686 i686 Comment ---------passed passed Required -----------------------2.11.24.137.0.0. Check: Node connectivity of subnet "10.206 Interfaces found on subnet "10.171 WARNING: Could not find a suitable set of interfaces for VIPs.1.137.6.stnsp001.24.8.24. Checking system requirements for 'database'.9 Comment ---------passed passed Check: Package existence for "make-3.8.20.137.215 eth0:10. CVU check before creating ASM instance 52 .24.8.207 stnsp001 eth1:10.137.0" with node(s) stnsp002.0" that are likely candidates for a private interconnect: stnsp002 eth0:10.9-34. Check: Free disk space in "/tmp" dir Node Name Available -----------.0" Source Destination Connected? -----------------------------.20.-----------------------------.8.137.80".9-34.ELsmp Result: Kernel version check passed.Result: Node connectivity check passed for subnet "10.stnsp001. Check: Total memory Node Name Available -----------..ELsmp stnsp001 2.20. Interfaces found on subnet "10.-----------------------stnsp002 5.0. Check: System architecture Node Name Available -----------. Check: Swap space Node Name Available -----------.233 Interfaces found on subnet "10.6.93GB (252637044KB) Result: Free disk space check passed.8.---------------stnsp002:eth1 stnsp001:eth1 yes Result: Node connectivity check passed for subnet "10.6.1..9 2.0" Source Destination Connected? -----------------------------.24.137.-----------------------------.137.0.80-5 stnsp001 make-3. Result: Node connectivity check passed.5GB (1572864KB) 1.20.172 stnsp001 eth2:10.0" that are likely candidates for a private interconnect: stnsp002 eth2:10.91GB (6192456KB) Result: Swap space check passed. Check: Node connectivity of subnet "10.-----------------------stnsp002 2.

rhel4 stnsp001 sysstat-5.-----------------------------stnsp002 libstdc++-3.-----------------------------stnsp002 gcc-3.5" Node Name Status -----------------------------.105-2 Result: Package existence check passed for "libaio-0.Check: Package existence for "binutils-2.15. CVU check before creating ASM instance 53 .-----------------------------.2. Comment ---------------passed passed Comment ---------------passed passed Check: Package existence for "libstdc++-devel-3.4.97-5 passed Result: Package existence check passed for "elfutils-libelf-devel-0.5-2 passed stnsp001 libstdc++-devel-3.5".5".11".2.-----------------------------stnsp002 sysstat-5.5" Node Name Status -----------------------------.5".2.3.3.1.5-2 stnsp001 libgcc-3.2.RHEL4.RHEL4.-----------------------------stnsp002 libaio-0.105".11".-----------------------------.4.0.4.105-2 stnsp001 libaio-0.11-1.2.5" Node Name Status Comment -----------------------------.4.105-2 passed stnsp001 libaio-devel-0.0.-----------------------------. Check: Package existence for "unixODBC-2. Check: Package existence for "sysstat-5. Check: Package existence for "libaio-0.2-18 passed Result: Package existence check passed for "binutils-2.92.92. 6. Check: Package existence for "libgcc-3.2-18 passed stnsp001 binutils-2.1 Result: Package existence check passed for "unixODBC-2.4.105" Node Name Status -----------------------------.0. Comment ---------------passed passed Comment ---------------passed passed Check: Package existence for "libaio-devel-0.1 stnsp001 unixODBC-2.11" Node Name Status Comment -----------------------------.---------------stnsp002 binutils-2.5" Node Name Status -----------------------------.0.3.15.---------------stnsp002 libstdc++-devel-3.4.105" Node Name Status Comment -----------------------------.11" Node Name Status -----------------------------.2.15.4.5-2 passed Result: Package existence check passed for "libstdc++-devel-3.105".97".4.RHEL4.0.4.97" Node Name Status Comment -----------------------------.4.---------------stnsp002 unixODBC-devel-2.3.---------------stnsp002 libaio-devel-0.1 passed Result: Package existence check passed for "unixODBC-devel-2.---------------stnsp002 elfutils-libelf-devel-0.5" Node Name Status -----------------------------.0.-----------------------------.4.11-1.0.-----------------------------.2.5-7.4.4.1 passed stnsp001 unixODBC-devel-2.rhel4 Result: Package existence check passed for "sysstat-5.-----------------------------stnsp002 libgcc-3.4. Check: Package existence for "libstdc++-3.5-2 Result: Package existence check passed for "libstdc++-3.5-7.105-2 passed Result: Package existence check passed for "libaio-devel-0.5-2 Result: Package existence check passed for "gcc-3.15.11-1.4.-----------------------------stnsp002 unixODBC-2.4.5-2 stnsp001 libstdc++-3.2" Node Name Status Comment -----------------------------.5-2 stnsp001 gcc-3. Check: Package existence for "gcc-3.RHEL4. Comment ---------------passed passed Check: Package existence for "elfutils-libelf-devel-0.92. Comment ---------------passed passed Check: Package existence for "unixODBC-devel-2.92.5".2.3.0.5".3.3.5-2 Result: Package existence check passed for "libgcc-3.3.11-1.2".97-5 passed stnsp001 elfutils-libelf-devel-0.

Comment ---------passed passed Required -----------------------128 128 "semmni".3. Comment ---------passed passed Required -----------------------65536 65536 "file-max".4-2.-----------------------stnsp002 2147483648 stnsp001 2147483648 Result: Kernel parameter check passed for Check: Kernel parameter for "shmmni" Node Name Configured -----------. Comment ---------passed passed Required -----------------------32000 32000 "semmns". Comment ---------passed passed Required -----------------------100 100 "semopm". Comment ---------passed passed Required -----------------------4096 4096 "shmmni".3.-----------------------stnsp002 4194304 stnsp001 4194304 Comment ---------passed passed Required -----------------------4194304 4194304 Comment ---------passed passed 6.-----------------------stnsp002 6553600 stnsp001 6553600 Result: Kernel parameter check passed for Comment ---------------passed passed Required -----------------------250 250 "semmsl".4-2.-----------------------stnsp002 4096 stnsp001 4096 Result: Kernel parameter check passed for Check: Kernel parameter for "file-max" Node Name Configured -----------.3.1. Comment ---------passed passed Required -----------------------536870912 536870912 "shmmax". Check: Kernel parameter for "semmsl" Node Name Configured -----------.19 stnsp001 glibc-2. CVU check before creating ASM instance 54 .-----------------------stnsp002 32000 stnsp001 32000 Result: Kernel parameter check passed for Check: Kernel parameter for "semopm" Node Name Configured -----------.-----------------------.-----------------------stnsp002 100 stnsp001 100 Result: Kernel parameter check passed for Check: Kernel parameter for "semmni" Node Name Configured -----------. Comment ---------passed passed Required -----------------------2097152 2097152 "shmall".-----------------------stnsp002 4194304 4194304 stnsp001 4194304 4194304 Result: Kernel parameter check passed for "rmem_default". Check: Kernel parameter for "rmem_max" Node Name Configured -----------.-----------------------stnsp002 250 stnsp001 250 Result: Kernel parameter check passed for Check: Kernel parameter for "semmns" Node Name Configured -----------.-----------------------stnsp002 128 stnsp001 128 Result: Kernel parameter check passed for Check: Kernel parameter for "shmall" Node Name Configured -----------.4-2.-----------------------------stnsp002 glibc-2.3.19 Result: Package existence check passed for "glibc-2.4-2.19" Node Name Status -----------------------------.19". Comment ---------passed passed Check: Kernel parameter for "rmem_default" Node Name Configured Required -----------.Check: Package existence for "glibc-2.-----------------------stnsp002 2097152 stnsp001 2097152 Result: Kernel parameter check passed for Check: Kernel parameter for "shmmax" Node Name Configured -----------.

Checking daemon liveness. Check: Kernel parameter for "wmem_default" Node Name Configured Required -----------.. CVU check before creating ASM instance 55 .-----------------------stnsp002 exists passed stnsp001 exists passed Result: User existence check passed for "nobody".-----------------------.. Check: Health of CRS Node Name -----------------------------------stnsp002 stnsp001 Result: CRS health check passed. Checking daemon liveness. System requirement passed for 'database' Checking CRS integrity.-----------------------.-----------------------stnsp002 1048576 262144 stnsp001 1048576 262144 Result: Kernel parameter check passed for "wmem_max".-----------------------. Check: User existence for "nobody" Node Name Status Comment -----------.1.....-----------------------stnsp002 yes stnsp001 yes Result: Liveness check passed for "CSS daemon". Check: Liveness for "EVM daemon" Node Name Running -----------------------------------.-----------------------stnsp002 yes stnsp001 yes Result: Liveness check passed for "EVM daemon". Check: Liveness for "CRS daemon" Node Name Running -----------------------------------.Result: Kernel parameter check passed for "rmem_max". Check: Liveness for "CSS daemon" Node Name Running -----------------------------------. Checking daemon liveness. CRS OK? -----------------------yes yes 6. Check: Group existence for "dba" Node Name Status Comment -----------.-----------------------stnsp002 exists passed stnsp001 exists passed Result: Group existence check passed for "dba".... Check: Kernel parameter for "wmem_max" Node Name Configured Required -----------.-----------------------.-----------------------stnsp002 yes stnsp001 yes Result: Liveness check passed for "CRS daemon". Liveness of all Node Name -----------stnsp002 stnsp001 the daemons CRS daemon -----------------------yes yes Comment ---------passed passed Comment ---------passed passed CSS daemon -----------------------yes yes EVM daemon ---------yes yes Checking CRS health..-----------------------stnsp002 524288 262144 stnsp001 524288 262144 Result: Kernel parameter check passed for "wmem_default".

CRS integrity check passed. Pre-check for database installation was successful.

Show all

Hide all

6.2. Oracle ASM Home Software Install
Start the installer by running "runInstaller" from the staged installation media.
./runInstaller

In this section we establish the Oracle software for the ASM home.

• Notes ♦ Here we will create a ASM Software home on all the nodes in the cluster • Actions ♦ Select the Oracle Database 11g radio button ♦ Click Next

6.2. Oracle ASM Home Software Install

56

• Notes ♦ Here we can choose the type of database software to be installed • Actions ♦ Select the Enterprise Edition radio button ♦ Click Next

6.2. Oracle ASM Home Software Install

57

• Notes ♦ Here we specify the location of various components. An 11g install makes more use of the ORACLE_BASE. Most logs will be stored in sub directories under the oracle base. The oracle base will be common to all installs. Modify as required. Also this is where you specify the location of the ASM software home. I usually modify the Home name and Home Path to include the word ASM. This makes it easier to identify later on. If you change the path you should ensure that you do not use the exact same path as the Oracle Clusterware home • Actions ♦ Confirm entries are OK ♦ Click Next

• Notes ♦ The installer has detected the presence of Oracle Clusterware and uses this to populate this dialog box. To build a cluster which includes all nodes you must ensure that there are check-boxes next to the node names • Actions ♦ Click Select All ♦ Click Next

6.2. Oracle ASM Home Software Install

58

• Notes ♦ The installer will then complete some Product-Specific Prerequisite checks.2. These should all pass OK – as you have already run the CVU check • Actions ♦ Click Next 6. Oracle ASM Home Software Install 59 .

• Notes ♦ We are going to install a Software only home and then subsequently configure the software • Actions ♦ Select the Install Software Only radio button ♦ Click Next • Notes ♦ Assign the Unix groups for the privileged ASM users (default is oinstall) • Action ♦ Click Next 6. Oracle ASM Home Software Install 60 .2.

Oracle ASM Home Software Install 61 .• Notes ♦ Here we can see a summary of the install.2. • Actions ♦ Click Install • Notes 6.

on each node • Action ♦ open a shell window on each node ♦ run the root.1.1. The following environment variables are set as: ORACLE_OWNER= oracle11 ORACLE_HOME= /scratch/product/11. Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root. Now product-specific root actions will be performed. • Notes 6.♦ Here the installer copies the software to all nodes in the cluster • Actions ♦ none required • Notes ♦ The installer pauses.sh Running Oracle 11g root. Finished product-specific root actions.2. Oracle ASM Home Software Install 62 .0/asm/root.sh script. running scripts one at a time....00# /scratch/product/11..sh script bash-3.0/asm Enter the full pathname of the local bin directory: [/usr/local/bin]: /usr/local/bin is read only. Continue without copy (y/n) or retry (r)? [y]: y Warning: /usr/local/bin is read only. Creating /etc/oratab file.sh script. some scripts need to be run as root on all nodes of the cluster ♦ Be sure to follow directions carefully. No files will be copied.

Oracle ASM Home Software Install 63 .♦ The output from the root.2.sh script should be similar to this ♦ it should only take a few seconds to run on each node • Action ♦ run the scripts on all nodes ♦ then return to the installer and Click OK • Notes ♦ After the software install completes you will see this End of Installation dialog • Actions ♦ Click Exit 6.

Oracle ASM Software Home Patching Once ASM software has been installed.2.0.2. Oracle ASM software home has been installed 7.1./runInstaller 6. the next step in the process is to apply the 11. 7.1. where patchset_directory is the directory where you unpacked the patch set software: $ cd patchset_directory/Disk1 $ . Oracle ASM software home has been installed 64 .• Notes ♦ Are you sure ? • Actions ♦ Click Yes 6. Start the OUI for Oracle ASM Software Home Patching Enter the following commands to start Oracle Universal Installer.1.7 patchset to the ASM ORACLE_HOME.1.

click Next.1.• Notes ♦ the Universal Installer screen appears • Action ♦ On the Welcome screen. Start the OUI for Oracle ASM Software Home Patching 65 . • Notes 7.

Start the OUI for Oracle ASM Software Home Patching 66 . • Action ♦ Enter your email address ♦ Enter your Metalink password ♦ Click Next 7.1.♦ Specify the name and the location of the asm home • Action ♦ Check that the name and location are correct • Notes: ♦ here you can specify your metalik credentials for this install. ♦ If you leave both fields blank you can opt out of notifications (see next screen).

• Action ♦ Click Yes • Notes 7. Start the OUI for Oracle ASM Software Home Patching 67 . you will opt out of notifications and get the above screen.1.• Notes:* ♦ If you leave both fields blank.

1. Start the OUI for Oracle ASM Software Home Patching 68 .♦ the installer detects that this is a clustered home and automatically selects all the nodes in the cluster • Actions ♦ Click Next • Notes ♦ Some parameters are validated by the installer • Action ♦ Click Next 7.

1.• Notes ♦ This is a summary of the actions the installer will complete • Actions ♦ Click Install • Notes 7. Start the OUI for Oracle ASM Software Home Patching 69 .

Finished product-specific root actions.1.1.sh script.0/asm Enter the full pathname of the local bin directory: [/usr/local/bin]: /usr/local/bin is read only. • Notes ♦ The output from the root...sh need to be run as root on both nodes of the cluster • Action ♦ open a shell window on each node and run root. Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root. The following environment variables are set as: ORACLE_OWNER= oracle11 ORACLE_HOME= /scratch/product/11.1.sh (one after the other) ♦ then return to the installer and Click OK bash-3.sh Running Oracle 11g root.00# /scratch/product/app/oracle11/product/11.sh script.sh script should be similar to this 7.♦ The installer copies the patch to all the nodes in the cluster • Actions ♦ No action required • Notes ♦ The installer pauses. root. Start the OUI for Oracle ASM Software Home Patching 70 . Now product-specific root actions will be performed. No files will be copied. Continue without copy (y/n) or retry (r)? [y]: y Warning: /usr/local/bin is read only.0/asm/root.

/netca • Notes ♦ run the Network Configuration Assistant • Action ♦ Set environment variable ORACLE_HOME to ASM home location 8. Here we are going to create the listeners using netca from the ASM home. Oracle RAC Database Listener Creation 71 .♦ it should only take a few seconds to run on each node • Action ♦ run the scripts on all nodes ♦ then return to the installer and Click OK • Notes ♦ After the software install completes you will see this End of Installation dialog. Oracle RAC Database Listener Creation 8. Create Node specific network listeners The Oracle network listeners traditionally run from the ASM home.0/asm/bin [oracle11@stnsp001 bin]$ .1.1. exit the installer • Actions ♦ Click Yes 8. [oracle11@stnsp001 bin]$ cd /scratch/product/11.

/netca • Notes ♦ Netca detects that the Oracle Clusterware layer is running and offers Cluster or Single Node configuration • Actions ♦ Select the Cluster configuration radio button ♦ Click Next • Notes ♦ Netca uses Oracle Clusterware to determine all the nodes in the cluster • Actions ♦ Click Select all nodes ♦ Click Next 8. Create Node specific network listeners 72 .1.♦ change directory to the ASM home bin directory ♦ run .

• Notes ♦ You get various options – we need to configure listeners • Actions ♦ Select the Listener configuration radio button ♦ Click Next • Notes ♦ We need to add a listener • Actions ♦ Select the Add radio button ♦ Click Next 8.1. Create Node specific network listeners 73 .

• Actions ♦ Ensure the Selected Protocols list includes TCP ♦ Click Next 8. This is important for RAC • Actions ♦ Click Next • Notes ♦ Oracle Net supports various network protocols.• Notes ♦ Here you get the opportunity to name the listener – Do not change this. although TCP is the most common.1. The listeners will eventually be called LISTENER_nodename1 & LISTENER_nodename2. Create Node specific network listeners 74 .

• Notes ♦ It is possible to choose a non-default Port – I do not recommend this • Actions ♦ Ensure the Use the Standard port number of 1521 radio button is set ♦ Click Next

• Notes ♦ After configuring the node listeners you get the opportunity to configure more network components • Actions ♦ Select the No radio button ♦ Click Next

8.1. Create Node specific network listeners

75

• Notes ♦ You get the opportunity to configre other networking components • Action ♦ Click the Finish button to exit the tool

8.2. Verify the Listener resources are online
[oracle11@stnsp001 bin]$ cd /scratch/11.1.0/crs/bin [oracle11@stnsp001 bin]$ ./crs_stat -t Name Type Target State Host -----------------------------------------------------------ora....01.lsnr application ONLINE ONLINE stnsp001 ora....001.gsd application ONLINE ONLINE stnsp001 ora....001.ons application ONLINE ONLINE stnsp001 ora....001.vip application ONLINE ONLINE stnsp001 ora....02.lsnr application ONLINE ONLINE stnsp002 ora....002.gsd application ONLINE ONLINE stnsp002 ora....002.ons application ONLINE ONLINE stnsp002 ora....002.vip application ONLINE ONLINE stnsp002 • Notes ♦ You can see the listener resources inside Oracle Clusterware • Action ♦ change directory to the Oracle Clusterware home bin directory ♦ run the ./crs_stat -t comand

8.3. Completed Node specific network listeners creation
You have now completed the RAC cluster nodes Network listener configuration. A new managed resource will have been added to Oracle Clusterware OCR for each listener.

8.2. Verify the Listener resources are online

76

9. Oracle ASM Instance and diskgroup Creation
9.1. Create ASM Instance and add the +DATA and +FLASH diskgroups using the Database Assistant
cd /scratch/product/11.1.0/asm/bin ./dbca

• Notes ♦ We use the dbca from the ASM install to create the ASM instances • Actions ♦ ensure the ORACLE_HOME environment variable is set to the ASM home directory ♦ run ./dbca from the ASM home bin directory

• Notes ♦ dbca detects the Oracle Clusterware layer is running and offers to create either cluster or single instance database • Actions ♦ Select the Oracle Real Application Clusters database radio button ♦ Click Next

9. Oracle ASM Instance and diskgroup Creation

77

1. Create ASM Instance and add the +DATA and +FLASH diskgroupsusing the Database Assistant 78 .• Notes ♦ We use dbca to create RAC databases and ASM instances • Actions ♦ Select the Configure Automatic Storage Management radio button ♦ Click Next 9.

• Notes ♦ You need to make sure you create ASM instances on all the cluster nodes • Actions ♦ Click Select All ♦ Click Next 9.1. Create ASM Instance and add the +DATA and +FLASH diskgroupsusing the Database Assistant 79 .

Create ASM Instance and add the +DATA and +FLASH diskgroupsusing the Database Assistant 80 .• Notes ♦ Here we specify the password for the ASM Oracle SYS user • Actions ♦ Enter the SYS password ♦ Enter the same password for the Confirm SYS password ♦ Click Next 9.1.

Create ASM Instance and add the +DATA and +FLASH diskgroupsusing the Database Assistant 81 .• Notes ♦ dbca will create and start ASM instances • Actions ♦ Click OK 9.1.

Create ASM Instance and add the +DATA and +FLASH diskgroupsusing the Database Assistant 82 . This section will be used to create 2 disk groups +DATA and +FLASH • Actions ♦ Click Create New 9.• Notes ♦ This should take a few seconds • Actions ♦ no action • Notes ♦ ASM requires disks to be group together into diskgroups.1.

• Notes ♦ At the moment no disks are visible • Actions ♦ Click Change Discovery Path 9. Create ASM Instance and add the +DATA and +FLASH diskgroupsusing the Database Assistant 83 .1.

Create ASM Instance and add the +DATA and +FLASH diskgroupsusing the Database Assistant 84 .1.• Notes ♦ here we specifya filter to allow us to see the disks on the shared array • Action ♦ Enter a filter to allow the installer to see the disks ♦ Click OK • Notes ♦ Here you can now see thr disksw are visible • Action ♦ none requried 9.

1.• Notes ♦ Now we will assign disks to specific disk groups and create the DATA diskgroup • Actions ♦ In the Disk Group Name enter DATA ♦ Select the External Redundancy radio button ♦ Select the Show All radio button ♦ Select the 6 disks to be used for the DATA diskgroup ♦ Click OK 9. Create ASM Instance and add the +DATA and +FLASH diskgroupsusing the Database Assistant 85 .

1. Create ASM Instance and add the +DATA and +FLASH diskgroupsusing the Database Assistant 86 .• Notes ♦ A progress message – this should not take very long • Actions ♦ none 9.

1. We now need to create the FLASH diskgroup • Actions ♦ Click Create New • Notes ♦ We need to allow the installer to see the disks we haev reserved for the FLASH disk group • Action ♦ Click Change Discovery Path 9. Create ASM Instance and add the +DATA and +FLASH diskgroupsusing the Database Assistant 87 .• Notes ♦ Here we can see the DATA diskgroup has been created and is mounted on 2/2 instances.

• Notes ♦ We need to modify the disk discovery string • Action ♦ Modify the string ♦ Click OK 9. Create ASM Instance and add the +DATA and +FLASH diskgroupsusing the Database Assistant 88 .1.

Create ASM Instance and add the +DATA and +FLASH diskgroupsusing the Database Assistant 89 .* Notes • ♦ Now we will assign disks to specific disk groups and create the FLASH diskgroup For a change we will create this diskgroup with Normal redundancy • Actions ♦ In the Disk Group Name enter FLASH ♦ Select the Normal radio button ♦ Select the Show Candidate disks radio button ♦ Select the remaining disks allocated for the FLASH diskgroup ♦ Click OK 9.1.

• Notes ♦ A progress message – this should not take very long • Actions ♦ none 9.1. Create ASM Instance and add the +DATA and +FLASH diskgroupsusing the Database Assistant 90 .

SM2.gsd application ONLINE ONLINE stnsp002 ora.002.• Notes ♦ Here we can see the DATA and FLASH diskgroups have been created and are mounted on 2/2 instances.gsd application ONLINE ONLINE stnsp001 ora...ons application ONLINE ONLINE stnsp002 9...001...../crs_stat -t Name Type Target State Host -----------------------------------------------------------ora.ons application ONLINE ONLINE stnsp001 ora... Verify ASM instances are online [oracle11@stnsp001 bin]$ cd /scratch/11.2....2.0/crs/bin [oracle11@stnsp001 bin]$ .002.01.02.. • Actions ♦ Click Finish • Notes ♦ A confirmation dialog box • Actions ♦ Click No 9. Verify ASM instances are online 91 .....lsnr application ONLINE ONLINE stnsp002 ora.. This completed the ASM configuration.asm application ONLINE ONLINE stnsp001 ora.vip application ONLINE ONLINE stnsp001 ora......lsnr application ONLINE ONLINE stnsp001 ora.asm application ONLINE ONLINE stnsp002 ora.SM1.001.1....001.

CVU check -pre Database Instance Install Before installing the RAC Database software you should use the CVU utility to confirm the status of the cluster. 10.. Check: Existence of user "oracle11" Node Name User Exists Comment -----------. you do not have the execute the command again..stnsp002 -r 11gR1 -verbose • Note: If you executed the CVU -pre dbinst command during the ASM instance installation.. Check: Node reachability from node "stnsp001" Destination Node Reachable? -----------------------------------...sh stage -pre dbinst -n stnsp001.-----------------------stnsp002 passed stnsp001 passed Result: User equivalence check passed for user "oracle11". Oracle RAC Database Home Software Install 10.3.. Check: User equivalence for user "oracle11" Node Name Comment -----------------------------------. Network listeners and created 2 ASM diskgroups. Completed creation of ASM Instance and add of the +DATA and +FLASH diskgroups 92 .-----------------------./runcluvfy. Checking administrative privileges. Check: Existence of group "oinstall" Node Name Status -----------..-----------------------stnsp001 yes stnsp002 yes Result: Node reachability check passed from node "stnsp001".------------------------ Group ID ------------------------ 9.. . Completed creation of ASM Instance and add of the +DATA and +FLASH diskgroups Congratulations you have installed ASM.1.ora.stnsp002 -r 11gR1 -verbose Performing pre-checks for database installation Checking node reachability./crs_stat -t 9.vip application • Notes ONLINE ONLINE stnsp002 ♦ Here we can see the asm instances now as part of Oracle Clusterware • Action ♦ change directory to the Oracle Clusterware bin directory ♦ run . Checking user equivalence.-----------------------stnsp002 yes passed stnsp001 yes passed Result: User existence check passed for "oracle11".3./runcluvfy.sh stage -pre dbinst -n stnsp001.. Show CVU -pre dbinst output Hide CVU output .002.

-----------eth0 10.24.---------------stnsp002:eth2 stnsp001:eth2 yes Result: Node connectivity check passed for subnet "10.137.0" Source Destination Connected? -----------------------------.24.0.137.0.8.137.137.stnsp002 exists 1422 stnsp001 exists 1422 Result: Group existence check passed for "oinstall".8.206 10.137.-----------------------------.1 00:0E:0C:08:06:4E 0.137.0" with node(s) stnsp002.216 10.233 10. Comment -----------passed passed Interface information for node "stnsp002" Interface Name IP Address Subnet ---------------.0" Source Destination Connected? -----------------------------.0 Subnet Gateway Default Gateway Hardware Address -----------.8.8.0.8.-----------.0" that are likely candidates for a private interconnect: 10.137.8.137.-----------stnsp002 yes yes yes yes passed stnsp001 yes yes yes yes passed Result: Membership check for user "oracle11" in group "oinstall" [as Primary] passed.0.0 10.0" with node(s) stnsp002.0 10.0.-----------0.20.stnsp001.24.1 00:0E:0C:08:06:4E 0.-----------.8.-----------------------------.-----------.20.-----------.-----------.20.137.0 eth1 10.137.24.20.8.0 eth2 10.137. Check: Node connectivity of subnet "10.8.8.137.137.1 00:0E:0C:08:08:F2 0. Administrative privileges check passed.0 eth2 10.8.8.0.-----------.20.0" Source Destination Connected? -----------------------------.24.0 Subnet Gateway Default Gateway Hardware Address -----------.-----------. Check: Existence of group "dba" Node Name Status Group ID -----------.137.---------------stnsp002:eth1 stnsp001:eth1 yes Result: Node connectivity check passed for subnet "10.171 10.137.0 10.137.0" with node(s) stnsp002.8.8.0.0.8.0 10.0 eth0 10.0.0.8.0.24.0.137.216 eth0:10.8.-----------------------stnsp002 exists 8500 stnsp001 exists 8500 Result: Group existence check passed for "dba".215 eth0:10.---------------stnsp002:eth0 stnsp002:eth0 yes stnsp002:eth0 stnsp001:eth0 yes stnsp002:eth0 stnsp001:eth0 yes stnsp002:eth0 stnsp001:eth0 yes stnsp002:eth0 stnsp001:eth0 yes stnsp001:eth0 stnsp001:eth0 yes Result: Node connectivity check passed for subnet "10.137.0.stnsp001..137.1 00:0E:0C:08:08:F3 0.0.8.-----------.137.233 Interfaces found on subnet "10.1 00:0E:0C:08:08:F2 0.137.-----------eth0 10.-----------stnsp002 yes yes yes no stnsp001 yes yes yes no Result: Membership check for user "oracle11" in group "dba" passed.172 10.24.137.137.20.137.137.137.137.-----------.137. Check: Node connectivity of subnet "10.137.234 stnsp001 eth0:10.137.137.0 10.137. Check: Membership of user "oracle11" in group "dba" Node Name User Exists Group Exists User in Group Primary ---------------.0 eth0 10.-----------0.1 00:04:23:A6:CD:61 Check: Node connectivity of subnet "10.1 00:04:23:A6:CD:BD Interface information for node "stnsp001" Interface Name IP Address Subnet ---------------.0 10.215 10.137.8.8.0 10.0 eth1 10.8.-----------------------------.-----------.137.207 10.8. CVU check -pre Database Instance Install 93 .. Check: Membership of user "oracle11" in group "oinstall" [as Primary] Node Name User Exists Group Exists User in Group Primary Comment ---------------.234 10.stnsp001.-----------.137.1 00:0E:0C:08:06:4F 0. Interfaces found on subnet "10.1. Checking node connectivity.0.137.0 10.0.0" that are likely candidates for a private interconnect: stnsp002 eth0:10.8.-----------------------.

1.4.80" Node Name Status -----------------------------.2" Node Name Status Comment -----------------------------.171 WARNING: Could not find a suitable set of interfaces for VIPs.15.24.11.-----------------------------stnsp002 make-3.0.105-2 stnsp001 libaio-0.0.91GB (6192456KB) stnsp001 5. Check: Kernel version Node Name Available -----------.105" Node Name Status -----------------------------.137.5GB (1572864KB) Comment ---------passed passed Required -----------------------i686 i686 Comment ---------passed passed Required -----------------------2.-----------------------------stnsp002 libaio-0.stnsp002 eth1:10.20.-----------------------stnsp002 209.5GB (1572864KB) 1.95GB (6234256KB) stnsp001 5.92.1.1.6.06GB (219216588KB) stnsp001 240. Check: Free disk space in "/tmp" dir Node Name Available -----------.137.4.. Required -----------------------1GB (1048576KB) 1GB (1048576KB) Comment ---------passed passed Required -----------------------400MB (409600KB) 400MB (409600KB) Comment ---------passed passed Required -----------------------1..2-18 passed stnsp001 binutils-2.6.-----------------------stnsp002 5.137. Check: Swap space Node Name Available -----------.0.206 Interfaces found on subnet "10.0. Check: Package existence for "gcc-3.-----------------------------.9 2.92.24.5" Node Name Status -----------------------------.15.9-34.2".0.5".3.95GB (6234256KB) Result: Total memory check passed. Checking system requirements for 'database'.4.4.6.11.6.9 Comment ---------passed passed Check: Package existence for "make-3.5-2 Result: Package existence check passed for "gcc-3.80-5 stnsp001 make-3.15. CVU check -pre Database Instance Install 94 .0.ELsmp Result: Kernel version check passed.105-2 Comment ---------------passed passed Comment ---------------passed passed 10.-----------------------stnsp002 5.-----------------------stnsp002 i686 stnsp001 i686 Result: System architecture check passed. Check: Total memory Node Name Available -----------.0.137.---------------stnsp002 binutils-2.-----------------------------stnsp002 gcc-3.80".92.137.92.ELsmp stnsp001 2.9-34.207 stnsp001 eth1:10.-----------------------stnsp002 2.3. Result: Node connectivity check passed.3.91GB (6192456KB) Result: Swap space check passed.20. Check: System architecture Node Name Available -----------.15.2-18 passed Result: Package existence check passed for "binutils-2. Check: Package existence for "libaio-0.20.5-2 stnsp001 gcc-3.0" that are likely candidates for a private interconnect: stnsp002 eth2:10.93GB (252637044KB) Result: Free disk space check passed.172 stnsp001 eth2:10.80-5 Result: Package existence check passed for "make-3.0. Comment ---------------passed passed Check: Package existence for "binutils-2.

RHEL4.105".5".RHEL4.3.---------------stnsp002 unixODBC-devel-2.5". Comment ---------------passed passed Check: Package existence for "unixODBC-devel-2.19" Node Name Status -----------------------------.5" Node Name Status -----------------------------.2.5-2 stnsp001 libgcc-3.1 passed Result: Package existence check passed for "unixODBC-devel-2.97" Node Name Status Comment -----------------------------.-----------------------------. Comment ---------------passed passed Comment ---------------passed passed Check: Package existence for "libstdc++-devel-3.---------------stnsp002 elfutils-libelf-devel-0. Check: Kernel parameter for "semmsl" Node Name Configured -----------.-----------------------------stnsp002 libstdc++-3.3.5-7.0.11".2.2. Check: Package existence for "libstdc++-3.0.5-2 Result: Package existence check passed for "libstdc++-3.5" Node Name Status -----------------------------. Check: Package existence for "libaio-devel-0.1 passed stnsp001 unixODBC-devel-2.5-2 stnsp001 libstdc++-3.3. Comment ---------passed passed Required -----------------------32000 Comment ---------passed 10.4.5-2 passed Result: Package existence check passed for "libstdc++-devel-3.105".3.4.11-1. Check: Package existence for "sysstat-5.4-2.5".3.1 Result: Package existence check passed for "unixODBC-2.11" Node Name Status -----------------------------.3.4-2.5-7.11-1.4.4-2.5".11-1.-----------------------------.19 Result: Package existence check passed for "glibc-2.-----------------------------stnsp002 libgcc-3.97-5 passed stnsp001 elfutils-libelf-devel-0.2. Check: Package existence for "glibc-2.Result: Package existence check passed for "libaio-0.5-2 Result: Package existence check passed for "libgcc-3. Check: Package existence for "unixODBC-2.97-5 passed Result: Package existence check passed for "elfutils-libelf-devel-0.2.11-1.RHEL4.105" Node Name Status Comment -----------------------------. CVU check -pre Database Instance Install 95 .4.11" Node Name Status Comment -----------------------------.11".105-2 passed Result: Package existence check passed for "libaio-devel-0.rhel4 Result: Package existence check passed for "sysstat-5.-----------------------------.2.5-2 passed stnsp001 libstdc++-devel-3.3.19 stnsp001 glibc-2.19".5" Node Name Status Comment -----------------------------.-----------------------------.2.4.0.105-2 passed stnsp001 libaio-devel-0.4-2. Comment ---------------passed passed Check: Package existence for "elfutils-libelf-devel-0.4.97".-----------------------stnsp002 32000 Comment ---------------passed passed Required -----------------------250 250 "semmsl".1.-----------------------------stnsp002 glibc-2.-----------------------------stnsp002 unixODBC-2.4.2.5" Node Name Status -----------------------------. Check: Package existence for "libgcc-3.4.3.4.1 stnsp001 unixODBC-2.0.-----------------------stnsp002 250 stnsp001 250 Result: Kernel parameter check passed for Check: Kernel parameter for "semmns" Node Name Configured -----------.---------------stnsp002 libstdc++-devel-3.4.4.3.-----------------------------stnsp002 sysstat-5.4.rhel4 stnsp001 sysstat-5.RHEL4.---------------stnsp002 libaio-devel-0.

Check: Kernel parameter for "semopm" Node Name Configured -----------.-----------------------stnsp002 100 stnsp001 100 Result: Kernel parameter check passed for Check: Kernel parameter for "semmni" Node Name Configured -----------.-----------------------. Check: Kernel parameter for "wmem_max" Node Name Configured -----------. CVU check -pre Database Instance Install 96 . Comment ---------passed passed Check: Kernel parameter for "wmem_default" Node Name Configured Required -----------.-----------------------stnsp002 4194304 4194304 stnsp001 4194304 4194304 Result: Kernel parameter check passed for "rmem_default".-----------------------stnsp002 1048576 stnsp001 524288 Result: Kernel parameter check passed for Check: Group existence for "dba" Node Name Status -----------.-----------------------stnsp002 4194304 stnsp001 4194304 Result: Kernel parameter check passed for Comment ---------passed passed Required -----------------------4194304 4194304 "rmem_max". Check: Kernel parameter for "rmem_max" Node Name Configured -----------. Comment ---------passed passed Comment ------------------------ 10.-----------------------stnsp002 2097152 stnsp001 2097152 Result: Kernel parameter check passed for Check: Kernel parameter for "shmmax" Node Name Configured -----------.-----------------------stnsp002 6553600 stnsp001 6553600 Result: Kernel parameter check passed for passed Required -----------------------100 100 "semopm".-----------------------.-----------------------stnsp002 4096 stnsp001 4096 Result: Kernel parameter check passed for Check: Kernel parameter for "file-max" Node Name Configured -----------.------------------------ Comment ---------passed passed Required -----------------------262144 262144 "wmem_max".-----------------------stnsp002 2147483648 stnsp001 1073741824 Result: Kernel parameter check passed for Check: Kernel parameter for "shmmni" Node Name Configured -----------. Comment ---------passed passed Required -----------------------4096 4096 "shmmni". Comment ---------passed passed Check: Kernel parameter for "rmem_default" Node Name Configured Required -----------. Comment ---------passed passed Required -----------------------128 128 "semmni".stnsp001 32000 32000 Result: Kernel parameter check passed for "semmns".-----------------------stnsp002 524288 262144 stnsp001 524288 262144 Result: Kernel parameter check passed for "wmem_default". Comment ---------passed passed Required -----------------------65536 65536 "file-max".-----------------------stnsp002 128 stnsp001 128 Result: Kernel parameter check passed for Check: Kernel parameter for "shmall" Node Name Configured -----------.1. Comment ---------passed passed Required -----------------------2097152 2097152 "shmall". Comment ---------passed passed Required -----------------------536870912 536870912 "shmmax".

Checking daemon liveness. CRS OK? -----------------------yes yes Show all Hide all 10. System requirement passed for 'database' Checking CRS integrity.-----------------------stnsp002 exists passed stnsp001 exists passed Result: User existence check passed for "nobody"... Check: Liveness for "CRS daemon" Node Name Running -----------------------------------. Check: Liveness for "EVM daemon" Node Name Running -----------------------------------. Checking daemon liveness.-----------------------./runInstaller 10. Checking daemon liveness..-----------------------stnsp002 yes stnsp001 yes Result: Liveness check passed for "CRS daemon". Liveness of all Node Name -----------stnsp002 stnsp001 the daemons CRS daemon -----------------------yes yes CSS daemon -----------------------yes yes EVM daemon ---------yes yes Checking CRS health. CRS integrity check passed.. Check: Health of CRS Node Name -----------------------------------stnsp002 stnsp001 Result: CRS health check passed.. ...stnsp002 exists passed stnsp001 exists passed Result: Group existence check passed for "dba"..-----------------------stnsp002 yes stnsp001 yes Result: Liveness check passed for "EVM daemon".-----------------------stnsp002 yes stnsp001 yes Result: Liveness check passed for "CSS daemon".. Oracle RAC Database Home Software Install 97 . Check: Liveness for "CSS daemon" Node Name Running -----------------------------------. Pre-check for database installation was successful.2. Check: User existence for "nobody" Node Name Status Comment -----------.2.. Oracle RAC Database Home Software Install Start the installer by running "runInstaller" from the staged installation media.

2. Oracle RAC Database Home Software Install 98 . • Notes ♦ We use the same installer for Database homes as Clusterware homes • Actions ♦ Select the Oracle Database 11g radio button ♦ Click Next 10.In this section we establish the Oracle software for the RAC Database home.

Oracle RAC Database Home Software Install 99 .• Notes ♦ Here we can choose the type of database software to be installed • Actions ♦ Select the Enterprise Edition radio button ♦ Click Next 10.2.

• Notes ♦ Here we specify the location of various components. To build a cluster which includes all nodes you must ensure that there are check-boxes next to the node names • Actions ♦ Click Select All ♦ Click Next 10. If you change the path you should ensure that you do not use the exact same path as the Oracle Clusterware or ASM homes • Actions ♦ Confirm entries are OK ♦ Click Next • Notes ♦ The installer has detected the presence of Oracle Clusterware and uses this to populate this dialog box.2. Oracle RAC Database Home Software Install 100 .

• Notes ♦ The installer will then complete some Product-Specific Prerequisite checks.2.as you have already run the CVU check • Actions ♦ Click Next 10. These should all pass OK . Oracle RAC Database Home Software Install 101 .

• Notes ♦ We are going to install a Software only home and then subsequently configure the software • Actions ♦ Select the Install Software Only radio button ♦ Click Next • Notes ♦ Assign the Unix groups for the privileged db users (default is oinstall) • Action ♦ Click Next 10. Oracle RAC Database Home Software Install 102 .2.

2. Oracle RAC Database Home Software Install 103 . • Actions ♦ Click Install • Notes 10.• Notes ♦ Here we can see a summary of the install.

Now product-specific root actions will be performed. running scripts one at a time. Continue without copy (y/n) or retry (r)? [y]: y Warning: /usr/local/bin is read only. Finished product-specific root actions. such as root.sh • Notes ♦ The installer pauses. Oracle RAC Database Home Software Install 104 .0/asm Enter the full pathname of the local bin directory: [/usr/local/bin]: /usr/local/bin is read only. on each node • Action ♦ open a shell window on each node and execute the root script bash-3.0/db/root.00# /scratch/product/11. • Notes ♦ The output from the root. some scripts need to be run as root on all ndes of the cluster ♦ Be sure to follow directions carefully.1..1. Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root. The following environment variables are set as: ORACLE_OWNER= oracle11 ORACLE_HOME= /scratch/product/11. The installer will pause and display instructions for running scripts..2.sh Running Oracle 11g root.sh script should be similar to this ♦ it should only take a few seconds to run on each node 10. No files will be copied.sh script.♦ The installer copies the software to all nodes in the cluster • Action ♦ Wait.sh script.

Oracle RAC Database Home Software Install 105 .• Action ♦ run the scripts on all nodes ♦ then return to the installer and Click OK • Notes ♦ After the software install completes you will see this End of Installation dialog • Actions ♦ Click Exit 10.2.

Using the installer to patch the Oracle RAC Software Home Enter the following commands to start Oracle Universal Installer.0. where patchset_directory is the directory where you unpacked the patch set software: $ cd patchset_directory/Disk1 $ . Oracle RAC Database software home has been installed 11.1. Oracle RAC Database software home has been installed 106 .1. the next step in the process is to apply the 11. Oracle RAC Software Home Patching Once RAC software has been installed.1.2. 11.2./runInstaller 10.7 patchset to the RAC ORACLE_HOME.1.• Notes ♦ Are you sure ? • Actions ♦ Click Yes 10.

Using the installer to patch the Oracle RAC Software Home 107 . click Next.1. • Notes 11.• Notes ♦ the Universal Installer screen appears • Actions ♦ On the Welcome screen.

♦ Specify the name and the location of the Database home • Actions ♦ Check that the name and location are correct. ♦ If you leave both fields blank you can opt out of notifications (see next screen). Click Next • Notes ♦ here you can specify your metalik credentials for this install. Using the installer to patch the Oracle RAC Software Home 108 .1. • Actions ♦ Enter your email address ♦ Enter your Metalink password ♦ Click Next 11.

Using the installer to patch the Oracle RAC Software Home 109 . you will opt out of notifications and get the above screen.• Notes ♦ If you leave both fields blank.1. • Actions ♦ Click Yes • Notes 11.

♦ the installer detects that this is a clustered home and automatically selects all the nodes in the cluster • Actions ♦ Click Next • Notes ♦ Some parameters are validated by the installer • Actions ♦ Click Next 11.1. Using the installer to patch the Oracle RAC Software Home 110 .

Using the installer to patch the Oracle RAC Software Home 111 .• Notes ♦ This is a summary of the actions the installer will complete • Actions ♦ Click Install • Notes 11.1.

sh script.0/db Enter the full pathname of the local bin directory: [/usr/local/bin]: /usr/local/bin is read only. 11.sh script. Using the installer to patch the Oracle RAC Software Home 112 . No files will be copied.1. Now product-specific root actions will be performed. The following environment variables are set as: ORACLE_OWNER= oracle11 ORACLE_HOME= /scratch/product/app/oracle11/product/11.0/db/root..sh (one after the other). Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root. Finished product-specific root actions.00# /scratch/product/app/oracle11/product/11.sh need to be run as root on both nodes of the cluster • Actions ♦ open a shell window on each node and run root.1.. output is shown below ♦ then return to the installer and Click OK bash-3. root.sh Running Oracle 11g root. Continue without copy (y/n) or retry (r)? [y]: y Warning: /usr/local/bin is read only.♦ The installer copies the patch to all the nodes in the cluster • Actions ♦ No action required • Notes ♦ The installer pauses.1.

1.-----------------------stnsp001 yes stnsp002 yes Result: Node reachability check passed from node "stnsp001"./runcluvfy. Oracle RAC Database Creation 113 . 12.stnsp002 -d /scratch/product/app/oracle11/product/11.-----------------------stnsp002 passed stnsp001 passed Result: User equivalence check passed for user "oracle11". Check: Node reachability from node "stnsp001" Destination Node Reachable? -----------------------------------. CVU Database Verification Check Verify the cluster before creating a cluster database .stnsp002 -d <DB:ORACLE_HOME> -verbose Show CVU -pre dbinst output Hide CVU output .1.0/db -verbose Performing pre-checks for database configuration Checking node reachability..• Notes ♦ Exit the installer • Actions ♦ Click Yes 12.sh stage -pre dbcfg -n stnsp001. Checking user equivalence.. Oracle RAC Database Creation 12..sh stage -pre dbcfg -n stnsp001. Check: User equivalence for user "oracle11" Node Name Comment -----------------------------------./runcluvfy..

0 10..0" Source Destination Connected? -----------------------------.137.0 Subnet Gateway Default Gateway Hardware Address -----------.---------------stnsp002:eth2 stnsp001:eth2 yes Result: Node connectivity check passed for subnet "10.0 eth2 10.-----------.137.8.0.8.0 10.0.20.0.1 00:0E:0C:08:06:4E 0.0.0.137. Check: Existence of user "oracle11" Node Name User Exists Comment -----------.137.-----------------------.137.206 10.24.24.8.0.0 eth0 10.137.-----------------------------.-----------stnsp002 yes yes yes no stnsp001 yes yes yes no Result: Membership check for user "oracle11" in group "dba" passed.8.137.8.137.-----------stnsp002 yes yes yes yes passed stnsp001 yes yes yes yes passed Result: Membership check for user "oracle11" in group "oinstall" [as Primary] passed.8.137.8.137.0 eth2 10.0 10. Check: Membership of user "oracle11" in group "dba" Node Name User Exists Group Exists User in Group Primary ---------------.137.0" that are likely candidates for a private interconnect: stnsp002 eth0:10.8.171 10..0.137.8.0.137.0 Comment -----------passed passed Subnet Gateway Default Gateway Hardware Address -----------.Checking administrative privileges.137.233 Interfaces found on subnet "10.137.8.137.0 10.0" Source Destination Connected? -----------------------------.-----------------------------.0 10.1 00:0E:0C:08:08:F2 0.137.234 stnsp001 eth0:10. Check: Existence of group "dba" Node Name Status Group ID -----------. Interfaces found on subnet "10.137. Administrative privileges check passed.24.233 10.137.0.---------------stnsp002:eth1 stnsp001:eth1 yes Result: Node connectivity check passed for subnet "10. Check: Existence of group "oinstall" Node Name Status Group ID -----------.137.0.-----------.0.stnsp001.-----------eth0 10.137.-----------0.137.8..-----------------------stnsp002 exists 1422 stnsp001 exists 1422 Result: Group existence check passed for "oinstall".-----------.0" with node(s) stnsp002.137.216 10.137. Checking node connectivity. Check: Node connectivity of subnet "10. CVU Database Verification Check 114 .0.0.137.-----------.-----------0.20.0" with node(s) stnsp002.stnsp001.8.20.137.215 eth0:10.207 10.8.24.0 10. Interface information for node "stnsp002" Interface Name IP Address Subnet ---------------.0 eth1 10.0" that are likely candidates for a private interconnect: stnsp002 eth1:10. Check: Membership of user "oracle11" in group "oinstall" [as Primary] Node Name User Exists Group Exists User in Group Primary Comment ---------------.20.1 00:0E:0C:08:06:4E 0.1 00:0E:0C:08:06:4F 0.137.24.24.-----------.0.0 10.8.137.0 10.0" with node(s) stnsp002.-----------.20.0" Source Destination Connected? -----------------------------.137.216 eth0:10.-----------.0 eth0 10.stnsp001.0 eth1 10.8.---------------stnsp002:eth0 stnsp002:eth0 yes stnsp002:eth0 stnsp001:eth0 yes stnsp002:eth0 stnsp001:eth0 yes stnsp002:eth0 stnsp001:eth0 yes stnsp002:eth0 stnsp001:eth0 yes stnsp001:eth0 stnsp001:eth0 yes Result: Node connectivity check passed for subnet "10.137.-----------------------.24.-----------------------.-----------------------------.137.-----------.137.1.215 10.172 10.207 12.137.8.1 00:04:23:A6:CD:BD Interface information for node "stnsp001" Interface Name IP Address Subnet ---------------..234 10.20.-----------.0.8.8.-----------eth0 10.137.1 00:0E:0C:08:08:F3 0.8.137.8.0.1 00:0E:0C:08:08:F2 0. Check: Node connectivity of subnet "10.-----------.8.-----------------------stnsp002 exists 8500 stnsp001 exists 8500 Result: Group existence check passed for "dba".8.8.137.-----------.24.1 00:04:23:A6:CD:61 Check: Node connectivity of subnet "10.-----------------------stnsp002 yes passed stnsp001 yes passed Result: User existence check passed for "oracle11".137.

-----------------------stnsp002 yes stnsp001 yes Result: Liveness check passed for "CRS daemon". Check: Health of CRS Node Name -----------------------------------stnsp002 stnsp001 Result: CRS health check passed.20..24.20. CRS OK? -----------------------yes yes Show all Hide all 12./dbca and • Notes ♦ You must launch dbca from the RAC database home • Action ♦ Ensure the ORACLE_HOME environment variable is set to the new RAC Home ♦ change to the RAC Home bin directory ♦ run . Checking CRS integrity. Check: Liveness for "CSS daemon" Node Name Running -----------------------------------.137....2.. Pre-check for database configuration was successful. use dbca to create the RAC database 115 . Liveness of all Node Name -----------stnsp002 stnsp001 the daemons CRS daemon -----------------------yes yes CSS daemon -----------------------yes yes EVM daemon ---------yes yes Checking CRS health.-----------------------stnsp002 yes stnsp001 yes Result: Liveness check passed for "CSS daemon". Check: Liveness for "CRS daemon" Node Name Running -----------------------------------. Checking daemon liveness.2./dbca 12... Check: Liveness for "EVM daemon" Node Name Running -----------------------------------. use dbca to create the RAC database [oracle11@stnsp001 bin]$ cd /scratch/product/11.. Checking daemon liveness.1. CRS integrity check passed.137.137.206 Interfaces found on subnet "10.0/db/bin [oracle11@stnsp001 bin]$ . Result: Node connectivity check passed.-----------------------stnsp002 yes stnsp001 yes Result: Liveness check passed for "EVM daemon".. Checking daemon liveness.0" that are likely candidates for a private interconnect: stnsp002 eth2:10..172 stnsp001 eth2:10.20.171 WARNING: Could not find a suitable set of interfaces for VIPs.137.stnsp001 eth1:10.

• Notes ♦ dbca detects the Oracle Clusterware layer is running and offers to create either cluster or single instance database • Actions ♦ Select the Oracle Real Application Clusters database radio button ♦ Click Next 12. use dbca to create the RAC database 116 .2.

• Notes ♦ We use dbca to create RAC databases • Actions ♦ Select the Create a Database radio button ♦ Click Next 12. use dbca to create the RAC database 117 .2.

2.• Notes ♦ You need to make sure you create RAC database instances on all the cluster nodes • Actions ♦ Click Select All ♦ Click Next 12. use dbca to create the RAC database 118 .

.. radio button ♦ Click Next 12. use dbca to create the RAC database 119 .2.• Notes ♦ Here you get to select the type of database • Actions ♦ Select the General Purpose.

use dbca to create the RAC database 120 . The SID should autofill ♦ Click Next 12.2.• Notes ♦ Now you specify the preface for the SIDs for the RAC database instances – recommendation is to keep it simple • Actions ♦ Enter APX in the Global Database name.

• Notes ♦ Here you get to configure Oracle Enterprise Manager. If you have no Grid Control server then the best method is to use dbcontrol as detailed here. • Actions ♦ Check the Configure Enterprise manager checkbox ♦ Select the Configure Database Control for local management radio button ♦ Click Next

12.2. use dbca to create the RAC database

121

• Notes ♦ Here we specify the password for the Oracle Database users. In this example we are setting the same password for all users. You should set a password scheme that meets your requirements. • Actions ♦ Select the Use the Same Administrative Password… radio button ♦ Enter the password ♦ Enter the same password for the Confirm password ♦ Click Next

12.2. use dbca to create the RAC database

122

• Notes ♦ Here you specify where you would like your database datafiles stored. We are going to use the ASM diskgroups we created earlier. • Actions ♦ Select the Automatic Storage Management (ASM) radio button ♦ Click Next

12.2. use dbca to create the RAC database

123

use dbca to create the RAC database 124 .• Notes ♦ dbca displays the diskgroups we previously created. We are going to use the DATA diskgroup • Actions ♦ Check the box on the DATA diskgroup line ♦ Click Next 12.2.

2. use dbca to create the RAC database 125 .• Notes ♦ Here we can adjust where various files are placed • Actions ♦ Select the Use Oracle-managed Files radio button ♦ Ensure +DATA is in the Database Area field ♦ Click Next 12.

• Actions ♦ Check the Specify Flash Recovery Area checkbox ♦ In the Flash recovery area enter +FLASH ♦ Set the flash recovery size – remember the size of the +FASH diskgroup ♦ Check the Enable Archiving checkbox ♦ Click Next 12.2. use dbca to create the RAC database 126 . We are going to use a flash recovery area.• Notes ♦ Here we specify recovery configuration information.

use dbca to create the RAC database 127 .2.• Notes ♦ Here you can choose to add the Oracle Sample schemas • Actions ♦ Check the Sample schemas checkbox ♦ Click Next 12.

use dbca to create the RAC database 128 .2. • Actions ♦ Click Next 12.• Notes ♦ Here you can modify the default initialisation parameters.

use dbca to create the RAC database 129 .• Notes ♦ 11g introduces enhanced security settings • Actions ♦ Select the Keep the enhanced… radio button ♦ Click Next • Notes ♦ 11g will set up some automatic maintenance tasks • Actions ♦ Check the Enable automatic maintenance tasks checkbox ♦ Click Next 12.2.

use dbca to create the RAC database 130 .• Notes ♦ Here you can review the placement of various database files • Actions ♦ Click Next 12.2.

• Notes ♦ You can decide to generate database creation scripts • Actions ♦ Check the Create Database checkbox ♦ Click Finish • Notes ♦ A summary screen • Actions ♦ Click OK 12.2. use dbca to create the RAC database 131 .

2.• Notes ♦ The database is being created • Actions ♦ none 12. use dbca to create the RAC database 132 .

3......SM2...ons application ONLINE ONLINE stnsp001 ora. Verify all database resource are online 133 ...001.X1....3..01./crs_stat -t Name Type Target State Host -----------------------------------------------------------ora.inst application ONLINE ONLINE stnsp002 ora.001....vip application ONLINE ONLINE stnsp001 ora.lsnr application ONLINE ONLINE stnsp001 ora.1..gsd application ONLINE ONLINE stnsp001 ora..asm application ONLINE ONLINE stnsp001 ora.. Verify all database resource are online [oracle11@stnsp001 bin]$ cd /scratch/11...X2.• Notes ♦ After the database is created the summary screen is displayed – note the URL for the Database Control • Actions ♦ Click Exit • Notes ♦ The Database instances will be started on all nodes • Actions ♦ none 12...APX.SM1..0/crs/bin [oracle11@stnsp001 bin]$ .asm application ONLINE ONLINE stnsp002 12.inst application ONLINE ONLINE stnsp001 ora.001.db application ONLINE ONLINE stnsp001 ora.

..vip • Notes application application application application ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE stnsp002 stnsp002 stnsp002 stnsp002 ♦ You can see that Oracle clusterware has started the database instances on each node • Action ♦ change directory to the Oracle Clusterware home bin directory ♦ type ..002.ora.lsnr ora...ons ora.gsd ora..002. RAC database creation is complete 134 . RAC database creation is complete 12.4....002..../crs_stat -t 12.02.4.

Sign up to vote on this title
UsefulNot useful