Professional Documents
Culture Documents
RACGuides Rac10gR2OnLinux
RACGuides Rac10gR2OnLinux
Rac10gR2OnLinux.............................................................................................................................................1 1. *Introduction.......................................................................................................................................1 1.1. *What you need to know....................................................................................................1 1.1.1. Software required for install...............................................................................2 1.1.2. Processor Model.................................................................................................2 1.1.3. Required RPM packages .....................................................................................2 1.1.3.1. 32-Bit Required RPM's .......................................................................3 1.1.3.2. 64-Bit Required RPM's .......................................................................4 1.1.3.3. *Oracle Enterprise Linux:..................................................................6 1.2. *Installation steps ..........................................................................................................................................6 1.3. *Schematic....................................................................................................................................................7 1.3.1. Hardware/software configuration BEFORE Oracle software install............................................7 1.3.2. Hardware/software configuration AFTER Oracle software install..............................................7 1.4. *Installation Method.....................................................................................................................................8 2. *Prepare the cluster nodes for Oracle RAC.....................................................................................................9 2.1. *User Groups and Accounts............................................................................................................9 2.1.1. *Creating the OSDBA (dba) Group .................................................................................9 2.1.2. *Creating an OSOPER Group (Optional) ......................................................................10 2.1.3. *Creating the Oracle Inventory Group (oinstall)...........................................................10 2.1.4. *Creating the Oracle Software Owner User..................................................................10 2.1.4.1. *Determining Whether an Oracle Software Owner User Exists...................11 2.1.4.2. *Creating an Oracle Software Owner User...................................................11 2.1.4.3. *Modifying an Oracle Software Owner User................................................11 2.1.5. *Verifying That the User nobody Exists.................................................................................................11 2.1.6. *Creating Identical Users and Groups on Other Cluster Nodes..............................................................12 2.2. *SSH Setup.................................................................................................................................................12 2.2.1. *Create RSA Keys On Each Node using the following steps:...................................................12 2.2.2. *Add All Keys to a Common authorized_keys File...................................................................13 2.2.3. *Enabling SSH User Equivalency on Cluster Member Nodes...................................................14 2.3. *Configuring the oracle's User Environment ..............................................................................................15 2.4. *Network Requirements.............................................................................................................................16 2.4.1. *Network Hardware Requirements .............................................................................................16 2.4.2. *IP Address Requirements ..........................................................................................................17 2.4.3. *Network Ping Tests...................................................................................................................18 2.4.4. *Network Adapter configuration................................................................................................18 2.5. *Time Sync.................................................................................................................................................18 2.6. *Configuring Kernel Parameters................................................................................................................19 2.7. *Setting Shell Limits for the oracle User....................................................................................................20 2.8. *Configuration of the Hangcheck-timer Module........................................................................................21 2.8.1. *Hangcheck-timer Module verification procedure.....................................................................21 2.9. *Platform Specific Setup............................................................................................................................22 2.10. *Required Software Directories................................................................................................................23 2.10.1. *Oracle Base Directory.............................................................................................................23 2.10.2. *Oracle Inventory Directory.....................................................................................................23 2.10.3. *Oracle Clusterware Home Directory......................................................................................23 2.10.4. *Oracle Home Directory...........................................................................................................24 2.10.5. *Identifying Existing Oracle Directories..................................................................................24 2.11. *CVU Stage Check...................................................................................................................................25 3. *Prepare the shared storage for Oracle RAC.................................................................................................26 3.1. *Create Partitions...........................................................................................................................27 3.1.1. Example of Configuring Block Device Storage for Oracle Clusterware .......................28 3.1.2. Example of Creating a Udev Permissions File for Oracle Clusterware........................28 3.1.3. Platform Specific Settings.............................................................................................29 3.2. *Installing the cvuqdisk Package for Linux ................................................................................................29 3.3. *CVU..........................................................................................................................................................30 i
Table of Contents
Rac10gR2OnLinux 4. Oracle Clusterware Installation and Configuration.......................................................................................30 4.1. *CVU Pre Oracle Clusterware install check ..................................................................................30 4.2. *Installing Oracle Clusterware with OUI......................................................................................30 4.2.1. Oracle Clusterware has been installed...........................................................................38 4.3. *CVU Post Oracle Clusterware install check.............................................................................................38 4.4. *Changing diagwait parameter to delay node reboot.................................................................................39 5. Oracle Clusterware patching..........................................................................................................................39 6. Install Oracle ASM Software only Home......................................................................................................43 7. Oracle ASM Software Home Patching..........................................................................................................48 8. Oracle RAC Database Listener Creation.......................................................................................................52 8.1. Create Node specific network listeners ..........................................................................................52 9. Oracle ASM Instance and diskgroup Creation..............................................................................................57 9.1. Create ASM Instance and add the +DATA and +FLASH diskgroups..........................................57 10. Oracle RAC Database Home Software Install.............................................................................................66 10.1. CVU check - Pre Database Install...............................................................................................66 10.2. Oracle RAC Database Home Software Install.............................................................................67 11. Oracle RAC Software Home Patching........................................................................................................72 12. Oracle RAC Database Creation...................................................................................................................76 12.1. use dbca to create the RAC database...........................................................................................76
ii
Rac10gR2OnLinux
1. *Introduction
1.1. *What you need to know
Oracle Clusterware With Oracle Database 10g Release 2 (10.2), Cluster Ready Services, or CRS, is now called Oracle Clusterware. Cluster Verification Utility (CVU) This utility (also known as cvu or cluvfy) checks the state of your cluster at various stages of the install, from initial hardware and operating system configuration to post-installation cluster database configuration. It should catch operating system configuration issues that otherwise would have caused an unnecessary installation failure. You can download the latest version of CVU from otn. http://www.oracle.com/technology/products/database/clustering/cvu/cvu_download_homepage.html Clusterware Home Starting with Oracle Database 10g Release 2 (10.2), Oracle Clusterware should be installed in a separate Oracle Clusterware home directory. You should not install Oracle Clusterware in a release-specific Oracle home mount point, typically /u01/app/oracle/product/10.2.0. Automatic Storage Management (ASM) In this document, ASM is the standard method of storage for the database files (not the Oracle Home). ASM can provide Automatic striping, mirroring, and Volume Management. This document does not cover using OCFS and RAW devices for storage, though they are still available. ASM Instance With Oracle Database 10g Release 2 (10.2), a single ASM instance for each node is now able to serve disk groups to all the database instances in the cluster, whether or not the database is a RAC database or a Single Instance database. Automatic Storage Management should be installed in a separate ASM home directory. Virtual IP (VIP) Oracle Database uses the VIP address to provide a secondary IP for the main network adapter for the node. Clients connect to the VIP to gain access to the database. The purpose of the VIP is to improve detection of node failure by clients, and facilitate failover. The VIP is not a cluster IP. Installing 10gR2 RAC on Oracle Enterprise Linux 5(OEL5) or RHEL5 or SLES10 If you are installing Oracle RAC on one of these systems, there are issues that you must be aware of. You should check notes WebIV:414163.1 (10gR2 RAC Install issues on Oracle EL5 or RHEL5 or SLES10), WebIV:419646.1 (Requirements For Installing Oracle10gR2? On RHEL/OEL 5 (x86)) and WebIV:421308.1 (Requirements For Installing Oracle10gR2? On RHEL/OEL 5 (x86_64)) before beginning the installation. Disk requirements for Oracle Clusterware files The storage requirement for the OCR & vote devices used by Oracle Clusterware has been increased in 11g to 280 MB (256 MB, plus an additional buffer to allow for fdisk variation in space allocation). It is a best practice to use the same storage requirements for the clusterware files in 10g. Certification Rac10gR2OnLinux 1
Ensure that you have a certified combination of the operating system and an Oracle Database software release by referring to the OracleMetaLink? certification information.
Operating systems x86 (64-bit) Red Hat Enterprise Linux AS/ES 3 (Update 4 or later) Red Hat Enterprise Linux AS/ES 4 (Update 1 or later) Oracle Enterprise Linux 4 ( OEL4 2.6.9-11.EL) Red Hat Enterprise Linux AS/ES 5 Oracle Enterprise Linux 5 ( OEL5 ) SUSE Linux Enterprise Server 9 (Service Pack 2 or later) SUSE Linux Enterprise Server 10 Asianux 2.0 For additional information on Linux Operating System support refer to Metalink Note 266043.1.
glibc-devel-2.3.3-98.47.i586.rpm gcc-3.3.3-43.34.i586.rpm. libstdc++-devel-3.3.3-43.34.i586.rpm gcc-c++-3.3.3-43.34.i586.rpm db1-1.85-85.1.i586.rpm orbit-0.5.17-330.1.i586.rpm gnome-libs-1.4.1.7-671.1.i586.rpm. plotutils-2.4.1-575.1.i586.rpm gnuplot-3.7.3-256.1.i586.rpm sysstat-5.0.1-35.4.i586.rpm. SuSE SLES10: glibc-devel-2.4-31.2.i586.rpm gcc-4.1.0-28.4.i586.rpm libstdc++-devel-4.1.0-28.4.i586.rpm gcc-c++-4.1.0-28.4.i586.rpm libaio-devel-0.3.104-14.2.i586.rpm
compat-libstdc++-33-3.2.3-47.3.i386.rpm control-center-2.8.0-12.x86_64.rpm gcc-3.4.3-47.x86_64.rpm gcc-c++-3.4.3-47.x86_64.rpm glibc-2.3.4-2.9.x86_64.rpm glibc-2.3.4-2.9.i386.rpm glibc-common-2.3.4-2.9.x86_64.rpm glibc-devel-2.3.4-2.9.x86_64.rpm glibc-devel-2.3.4-2.9.i386.rpm glibc-headers-2.3.4-2.9.x86_64.rpm glibc-kernheaders-2.4-9.1.87.x86_64.rpm gnome-libs-1.4.1.2.90-44.1.x86_64 libaio-0.3.103-3.i386.rpm libaio-0.3.103-3.x86_64.rpm libgcc-3.4.3-9.EL4 libstdc++-3.4.3-22.1.x86_64 libstdc++-devel-3.4.3-22.1.x86_64 make-3.80-5.x86_64.rpm pdksh-5.2.14-30.x86_64.rpm sysstat-5.0.5-1.x86_64.rpm util-linux-2.12a-16.EL4.23.x86_64 (for raw devices) xorg-x11-deprecated-libs-6.8.1-23 xscreensaver-4.18-5.rhel4.2.x86_64.rpm [Red Hat,Oracle] Enterprise Linux 5.0 compat-gcc-34-3.4.6-4 compat-gcc-34-c++-3.4.6-4 compat-libstdc++-33-3.2.3-61.i386.rpm compat-libstdc++-33-3.2.3-61.x86_64.rpm gcc-c++-4.1.1-52.el5.x86_64.rpm and all its dependent packages: ibstdc++-devel-4.1.1-52.el5.x86_64.rpm glibc-headers-2.5-12.x86_64.rpm glibc-devel-2.5-12.i386.rpm glibc-devel-2.5-12.x86_64.rpm ibgomp-4.1.1-52.el5.x86_64.rpm gcc-4.1.1-52.el5.x86_64.rpm libXp-1.0.0-8.i386.rpm sysstat-7.0.0-3.el5.x86_64.rpm util-linux-2.13-0.44.el5.x86_64 (for raw devices) SUSE SLES9: binutils-2.15.90.0.1.1-32.5 db1-1.85-85.1 gcc-3.3.3-43.24 gcc-c++-3.3.3-43.24 glibc-2.3.3-98.28 glibc-32bit-9-200506071326 glibc-devel-2.3.3-98-47 glibc-devel-32bit-9-200506062332 libaio-0.3.102.1.2 libaio-devel-0.3.102.1.2 1.1.3.2. 64-Bit Required RPM's 5
gnome-libs-1.4.1.7-671.1 gnuplot-3.7.3-256.1 (sysstat required RPM) libstdc++-3.3.3-43.24 libstdc++-devel-3.3.3-43.24 make-3.80-184.1 orbit-0.5.17-330.1 (gnome-libs required RPM) plotutils-2.4.1-575.1 (sysstat required RPM) pdksh-5.2.14-780.1 sysstat-5.0.1-35.1 XFree86-libs-4.3.99.902.43.48 SUSE SLES10: glibc-devel-2.4-31.2.x86_64.rpm gcc-4.1.0-28.4.x86_64.rpm libstdc++-devel-4.1.0-28.4.x86_64.rpm gcc-c++-4.1.0-28.4.x86_64.rpm glibc-devel-32bit-2.4-31.2.x86_64.rpm libaio-devel-0.3.104-14.2.x86_64.rpm
Create the Listeners Create the ASM Instances and initial Disk Group Establish RAC Database Install an Oracle Software Home for RAC Database Patch the RAC Database Home to the latest patchset Create the RAC Database Instances Note: Since RHEL5/OEL5 and SLES10 are not recognized by runInstaller, you will need to execute it with the -ignoreSysPrereqs option.
1.3. *Schematic
The following is a schematic of the software & hardware layout of a 2node RAC cluster. As explained in this document the actual number of LUNs required will vary depending on your mirroring requirements.
Notice that there are a number of LUNs present to both machines from the shared storage. 5 of 270MB LUNs 2 are used for the OCR devices 3 are used for the vote devices 9 of 1GB LUNs 6 are used for the +DATA ASM diskgroup 3 are used for the +FLASH diskgroup. Also notice there is a dedicated switch on the cluster interconnect.
NFS: The Oracle Cluster, ASM & Oracle Database Homes are established on the local disk of each of the RAC nodes. The files required by Oracle Clusterware are on Block devices. The database data-files are also on NFS File Server. There are other possible combinations although Oracle recommends one of the above methods. You cannot place the Clusterware devices on directnfs. Directnfs is not covered here
This installation routine presumes that you have a 2-node Linux cluster. There are a number of items that require checking before the install commences. Getting this bit right will enhance your install experience. 1. User Groups and Accounts 2. SSH Setup 3. Configuring the oracle's User Environment 4. Network Requirements 5. Time sync 6. Kernel Parameters 7. Setting Shell Limits for oracle user 8. Configuration of the Hangcheck-Timer Module 9. Platform Specific Setup 10. Required Software Directories 11. CVU Stage Check
following command, use the group name dba unless a group with that name already exists. # /usr/sbin/groupadd dba
10
11
# /usr/sbin/useradd nobody
2.2.1. *Create RSA Keys On Each Node using the following steps:
1. Log in as the software owner (in this example, the oracle user). 2. To ensure that you are logged in as the Oracle user, and that the user ID matches the expected user ID you have assigned to the Oracle user, enter the commands id and id oracle. Ensure that Oracle user group and user and the terminal window process group and user IDs are identical. For example: # id uid=502(oracle) gid=501(oinstall) groups=501(oinstall),502(oracle) # id oracle uid=502(oracle) gid=501(oinstall) groups=501(oinstall),502(oracle) 3. If necessary, create the .ssh directory in the crs user's home directory, and set permissions on it to ensure that only the crs user has read and write permissions: # mkdir ~/.ssh # chmod 700 ~/.ssh 4. Enter the following command: # /usr/bin/ssh-keygen -t rsa
12
At the prompts: Accept the default location for the key file (press Enter). Enter and confirm a pass phrase unique for this installation user. This command writes the RSA public key to the ~/.ssh/id_rsa.pub file and the private key to the ~/.ssh/id_rsa file. Never distribute the private keys to anyone not authorized to perform Oracle software installations. 5. Repeat steps 1 through 4 on each node that you intend to make a member of the cluster, using the RSA keys.
2.2.1. *Create RSA Keys On Each Node using the following steps:
13
At the end of this process, the public hostname for each member node should be registered in the known_hosts file for all other cluster member nodes. If you are using a remote client to connect to the local node, and you see a message similar to "Warning: No xauth data; using fake authentication data for X11 forwarding," then this means that your authorized keys file is configured correctly, but your ssh configuration has X11 forwarding enabled. To correct this, proceed as follows: a. Using any text editor, edit or create the ~oracle/.ssh/config file. b. Make sure that the ForwardX11? attribute is set to no. For example: Host * ForwardX11 no 3. Repeat step 2 on each cluster node member. 4. (Optional) If you entered passphrase above, and desire password-less login (needed for 10g RAC install), you need to inform the ssh-agent (already running for desktop) of the passphrase so that ssh clients would not be prompted for a passphrase. Once you notify ssh-agent the passphrase is cached for duration of the GUI 2.2.3. *Enabling SSH User Equivalency on Cluster Member Nodes 14
desktop login, and further client requests are satisfied by the ssh-agent. At the prompt, enter the pass phrase for each key that you generated. For example: # exec /usr/bin/ssh-agent $SHELL # ssh-add Enter passphrase for /home/crs/.ssh/id_rsa Identity added: /home/crs/.ssh/id_rsa (/home/crs/.ssh/id_rsa) These commands start the ssh-agent on the node, and load the RSA keys into memory so that you are not prompted to use pass phrases when issuing SSH commands. If you have configured SSH correctly, then you can now use the ssh or scp commands without being prompted for a password or a pass phrase. For example: # ssh stnsp002 date Mon May 19 23:34:42 UTC 2008 # ssh stnsp001 date Mon May 19 23:34:48 UTC 2008 If any node prompts for a password or pass phrase, then verify that the ~/.ssh/authorized_keys file on that node contains the correct public keys, and that you have created an Oracle software owner with identical group membership and IDs.
15
In this example, local_host is the host name or IP address of the system that you want to use to display Oracle Universal Installer (your workstation or PC). /tmp directory: If you determined that the /tmp directory has less than 400 MB of free disk space, then identify a file system with at least 400 MB of free space and set the TEMP and TMPDIR environment variables to specify a temporary directory on this file system Note: You cannot use a shared file system as the location of the temporary file directory (typically /tmp) for Oracle RAC installation. Install the cvuqdisk Package for Linux: If you are using Red Hat or SUSE Linux, then you must download and install the operating system package cvuqdisk. Without cvuqdisk, CVU is unable to discover shared disks, and you receive the error message "Package cvuqdisk not installed" when you run CVU. Use the cvuqdisk rpm for your hardware (i386, or for Itanium, ia64). To install the cvuqdisk RPM, complete the following procedure: Locate the cvuqdisk RPM package, which is in the directory clusterware/rpm on the installation media. If you have already installed Oracle Clusterware, then it is located in the directory CRS_home/rpm. Copy the cvuqdisk package to each node on the cluster. You should ensure that each node is running the same version of Linux. Log in as root. Using the following command, check to see if you have an existing version of the cvuqdisk package: # rpm -qi cvuqdisk If you have an existing version, then enter the following command to de-install the existing version: #rpm -e cvuqdisk Set the environment variable CVUQDISK_GRP to point to the group that will own cvuqdisk, typically oinstall. Use the following command to install the cvuqdisk package: #rpm -iv cvuqdisk-1.0.1-1.rpm Note: If you prefer, you can choose to disable CVU shared disk checks by adding the following line to the file CRS_ home/cv/admin/cvuconfig: CV_RAW_CHECK_ENABLED=FALSE
For example: if eth0 is the public interface on node1, it must be the public interface on node2. You should configure the private interfaces on the same network adapters as well. If eth1 is the private interface for node1, then eth1 should be the private interface for node2. 3. For the public network, each network adapter must support TCP/IP. 4. For the private network, the interconnect must support the user datagram protocol (UDP) using high-speed network adapters and switches that support TCP/IP (Gigabit Ethernet or better recommended). 5. For the private network, the endpoints of all designated interconnect interfaces must be completely reachable on the network. There should be no node that is not connected to every private network. You can test whether an interconnect interface is reachable using a ping command. Note: UDP is the default interconnect protocol for Oracle RAC, and TCP is the interconnect protocol for Oracle Clusterware. Token-Ring is not supported for the interconnect.
eth1
eth2
17
RX packets:41400340 errors:0 dropped:0 overruns:0 frame:0 TX packets:37403509 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:844975237 (805.8 MiB) TX bytes:43154372 (41.1 MiB) Base address:0x2800 Memory:fe880000-fe8a0000
It is important to confirm that networking is configured correctly before commencing install. Show all Hide all
18
system, enter the following command: /sbin/sysctl -p 3. Repeat steps 1 and 2 on all other nodes in the cluster. On SUSE systems only, enter the following command to cause the system to read the /etc/sysctl.conf file when it restarts: # /sbin/chkconfig boot.sysctl on 4. On SUSE Linux Enterprise Server 9.0 only, set the kernel parameter disable_cap_mlock as follows: disable_cap_mlock = 1 5. After updating the values of kernel parameters in the /etc/sysctl.conf file, either restart the computer, or run the command sysctl -p to make the changes in the /etc/sysctl.conf file available in the active kernel memory.
2. Add or edit the following line in the /etc/pam.d/login file, if it does not already exist: session required /lib/security/pam_limits.so
3. Depending on the oracle user's default shell, make the following changes to the default shell startup file: For the Bourne, Bash, or Korn shell, add the following lines to the /etc/profile file (or the file /etc/profile.local on SUSE systems): if [ $USER = "oracle" ]; then if [ $SHELL = "/bin/ksh" ]; then ulimit -p 16384 ulimit -n 65536 else ulimit -u 16384 -n 65536 fi fi For the C shell (csh or tcsh), add the following lines to the /etc/csh.login file (or the file /etc/csh.login.local on SUSE systems): if ( $USER == "oracle" ) then limit maxproc 16384 limit descriptors 65536 endif 1. Repeat this procedure on all other nodes in the cluster. 2.7. *Setting Shell Limits for the oracle User 20
22
is safe since it does not delete any socket or special files, only regular files. However custom scripts or 3rd party cleanup jobs might cleanup such files and cause Clusterware failures.
24
./runcluvfy.sh stage -post hwos -n stnsp001,stnsp002 -verbose Performing post-checks for hardware and operating system setup Checking node reachability... Check: Node reachability from node "stnsp001" Destination Node Reachable? ------------------------------------ -----------------------stnsp001 yes stnsp002 yes Result: Node reachability check passed from node "stnsp001".
Checking user equivalence... Check: User equivalence for user "oracle11" Node Name Comment ------------------------------------ -----------------------stnsp002 passed stnsp001 passed Result: User equivalence check passed for user "oracle11". Checking node connectivity...
Interface information for node "stnsp002" Interface Name IP Address Subnet ---------------- ------------ -----------eth0 10.137.8.216 10.137.8.0 eth1 10.137.24.207 10.137.24.0 eth2 10.137.20.172 10.137.20.0
Subnet Gateway Default Gateway Hardware Address ------------ ------------ -----------0.0.0.0 10.137.8.1 00:0E:0C:08:06:4E 0.0.0.0 10.137.8.1 00:0E:0C:08:06:4F 0.0.0.0 10.137.8.1 00:04:23:A6:CD:BD
Interface information for node "stnsp001" Interface Name IP Address Subnet ---------------- ------------ -----------eth0 10.137.8.215 10.137.8.0 eth1 10.137.24.206 10.137.24.0 eth2 10.137.20.171 10.137.20.0
Subnet Gateway Default Gateway Hardware Address ------------ ------------ -----------0.0.0.0 10.137.8.1 00:0E:0C:08:08:F2 0.0.0.0 10.137.8.1 00:0E:0C:08:08:F3 0.0.0.0 10.137.8.1 00:04:23:A6:CD:61
Check: Node connectivity of subnet "10.137.8.0" Source Destination Connected? ------------------------------ ------------------------------ ---------------stnsp002:eth0 stnsp001:eth0 yes Result: Node connectivity check passed for subnet "10.137.8.0" with node(s) stnsp002,stnsp001. Check: Node connectivity of subnet "10.137.24.0" Source Destination Connected? ------------------------------ ------------------------------ ---------------stnsp002:eth1 stnsp001:eth1 yes Result: Node connectivity check passed for subnet "10.137.24.0" with node(s) stnsp002,stnsp001.
25
Check: Node connectivity of subnet "10.137.20.0" Source Destination Connected? ------------------------------ ------------------------------ ---------------stnsp002:eth2 stnsp001:eth2 yes Result: Node connectivity check passed for subnet "10.137.20.0" with node(s) stnsp002,stnsp001. Interfaces found on subnet "10.137.8.0" that are likely candidates for a private interconnect: stnsp002 eth0:10.137.8.216 stnsp001 eth0:10.137.8.215 Interfaces found on subnet "10.137.24.0" that are likely candidates for a private interconnect: stnsp002 eth1:10.137.24.207 stnsp001 eth1:10.137.24.206 Interfaces found on subnet "10.137.20.0" that are likely candidates for a private interconnect: stnsp002 eth2:10.137.20.172 stnsp001 eth2:10.137.20.171 WARNING: Could not find a suitable set of interfaces for VIPs. Result: Node connectivity check passed.
Checking shared storage accessibility... WARNING: Package cvuqdisk not installed. stnsp001 NFS -----------------------------------/nas Sharing Nodes (2 in count) -----------------------stnsp002 stnsp001
Shared storage check was successful on nodes "stnsp002,stnsp001". Post-check for hardware and operating system setup was successful.
Show all
Hide all
26
There are two ways of storing Oracle Clusterware files: Block or Raw Devices: Oracle Clusterware files can be placed on either Block or Raw devices based on shared disk partitions. Oracle recommends using Block devices for easier usage. Refer to note WebIV:401132.1 if you want to use block devices. A supported shared file system: Supported file systems include the following: A supported cluster file system: OCFS (Linux Kernel 2.4),OCFS2 (Linux Kernel 2.6) or GPFS (IBM POWER). Network File System (NFS): A file-level protocol that enables access and sharing of files Note 1. If you do not have a storage option that provides external file redundancy, then you must configure at least three voting disk areas to provide voting disk redundancy. 2. If you are using NFS, then you must set the values for the NFS buffer size parameters rsize and wsize to at least 16384. Oracle recommends that you use the value 32768. 3. You cannot use ASM to store Oracle Clusterware files, because these files must be accessible before any ASM instance starts. There three ways of storing Oracle Database and recovery files: Automatic Storage Management (ASM). This is the recommended option. A supported shared file system Raw partitions (database files only)
In this example we have four 116Gig LUNs: /dev/sdb /dev/sdc /dev/sdd /dev/sde 116G 116G 116G 116G
27
Login as the root user on the remote nodes and execute the following: [root@stnsp002] # /sbin/partprobe /dev/sdb1 Note: Oracle recommends that you create partitions for Oracle Clusterware files on physically separate disks. The user account with which you perform the installation (oracle or crs) must have write permissions to create the files in the path that you specify.
grant the Oracle Clusterware owner (in this example crs) permissions to place voting disks on sdb5, sdc5 and sda5, add the following information to the file: # OCR disks sdb1:root:oinstall:0640 sdc1:root:oinstall:0640 # Voting disks sdb5:crs:oinstall:0640 sdc5:crs:oinstall:0640 sdd5:crs:oinstall:0640 4. Save the file: On Asianux 2, Enterprise Linux 4, and Red Hat Enterprise Linux 4 systems, savet he file as 49-oracle.permissions. On Asianux 3, Enterprise Linux 5, Red Hat Enterprise Linux 5, and SUSE Enterprise Server 10 systems, save the file as 51-oracle.permissions. 1. Using the following command, assign the permissions in the udev file to the devices: # /sbin/udevstart Use the following procedure above to create additional partitions to use for the OCR, Voting, and ASM disks. Refer to the OS documentation for additional information on using the fdisk command.
29
5. Set the environment variable CVUQDISK_GRP to point to the group that will own cvuqdisk, typically oinstall. 6. Use the following command to install the cvuqdisk package: rpm -iv cvuqdisk-1.0.1-1.rpm
3.3. *CVU
To check for all shared storage available across all nodes on the cluster, use the following command: runcluvfy.sh comp ssa -n node_list If you want to check the shared accessibility of a specific shared storage type to specific nodes in your cluster, then use the following command syntax: runcluvfy.sh comp ssa -n node_list -s storageID_list For example: runcluvfy.sh comp ssa -n node1,node2 -s /dev/sdb,/dev/sdc
Next we will install the Oracle Clusterware layer. Oracle Clusterware is an essential component of the Oracle RAC database infrastructure. Oracle Clusterware does not require any other clustering software. You must not install any other Cluster Software. To install Oracle Clkusterware you will need: To determine your cluster name To determine public node names, private node names, and virtual node names for each node in the cluster To Identify shared storage for Oracle Clusterware files (3 voting disks and 2 OCRs) and prepare disk partitions if necessary
to invoke the installer check again note WebIV:414163.1 before installing): $ runInstaller -ignoreSysPrereqs // This will bypass the OS check //
Notes The OUI will ask for the inventory directory. The default is ORACLE_BASE/oraInventory. It will also ask for the Operating System group that will have write permission to the inventory. Actions Specify a location for the Inventory directory and specify the Operating System group Click Next
Notes The OUI will name the Oracle Clusterware Home'OraCrs10g_home'. If you change this you should make sure that the name you use is unique Actions
31
Notes The installer will validate the state of the cluster before continuing. If there are issues you should rectify them before continuing Actions Click Next
Notes Each Cluster requires a name, this should be unique within your organisation, The default is crs. This is where you specify details of all the nodes in the cluster. The installer will default names for the node it is running on. You must add other nodes manually Oracle defaults the names to 'nodename', 'nodename-priv', 'nodename-vip' Actions 4.2. *Installing Oracle Clusterware with OUI 32
Confirm the Cluster Name selected is acceptable Confirm the Details for the current node are OK. The defaults are: Public Node Name : must resolve via hosts and or DNS to the public IP address and must be live Private Node Name : must resolve via hosts to the interconnect IP address and must be live Virtual Host Name : must resolve via hosts and or DNS to a new IP address and must not be live If these are not correct select the node entry and click Edit to modify. Click Add to enter a new node
Notes Here you specify the details of the node you wish to add to the cluster nodes list Actions Enter the new node details Public Node Name : must resolve via hosts and or DNS to the public IP address and must be live Private Node Name : must resolve via hosts to the interconnect IP address and must be live Virtual Host Name : must resolve via hosts and or DNS to a new IP address and must not be live Click OK to return to the node list for the cluster
33
* Notes The installer lists all the Network adapters. You should have one Adapter correctly identified as type 'Public' and at least one adapter correctly identified as type 'Private'. The installer will try and guess the use of an adapter based on the IP address bound. If it guesses incorrectly you must change the usage. Here it has guessed that all adapters are Private, which is incorrect. Actions Select the Adapter eth0 Click Edit... Select the Interface Type : Public radio button Click OK You should see that you have successfully configured the network adapter usage. Ideally you will have only 1 adapter set as public and 1 adapter set as private. If you have multiple public or multiple private adapters it is better to team them at the OS adapter driver level before commencing the install. Click Next
Here we specify the shared storage devices that will be used by Oracle Clusterware. Ideally you will have 2 devices for the OCR, Oracle will mirror to these devices to protect you from a single OCR device failure. Also you will have an additional 3 vote devices to protect your cluster from the failure of a single vote device. Actions Select the Normal radio button Enter the device to be used for the First OCR Enter the device to be used for the Second OCR Click Next
Notes Next we specify the devices to be used for the Oracle Clusterware vote disks Actions Select the Normal radio button Enter the device to be used for the First vote disk Enter the device to be used for the Second vote disk Enter the device to be used for the Third vote disk Click Next
35
Notes the installer lists a summary of the planned actions Actions Click Install
Notes the installer installs the software onto the local node Actions none required
36
Notes The installer requires commands to be run as root on each f the nodes Action On the first node open a root shell window
Notes here the Oracle Clusterware is configured Action run the identified command as root on the first node This may take some time to run You must wait for the command to complete before continuing Notes open a root shell on the second node 4.2. *Installing Oracle Clusterware with OUI 37
Action run the identified command as root on the first node This may take some time to run Notes You can now contnue with the install Action return to the installer Click OK Notes a series of configuration assistants are run at the end of the install Actions none required Notes The installer has completed installation and configuration of Oracle Clusterware Actions Click Exit To see the resources configured by Oracle Clusterware, run the ./crs_stat -t comand from the Oracle Clusterware home bin directory [root@stnsp013 bin]# ./crs_stat -t Name Type Target State Host -----------------------------------------------------------ora....013.gsd application ONLINE ONLINE stnsp013 ora....013.ons application ONLINE ONLINE stnsp013 ora....013.vip application ONLINE ONLINE stnsp013 ora....014.gsd application ONLINE ONLINE stnsp014 ora....014.ons application ONLINE ONLINE stnsp014 ora....014.vip application ONLINE ONLINE stnsp014
39
Notes Specify the name and the location of the clusterware home Action Check that the name and location are correct
Notes Both nodes in the RAC Cluster are displayed Action Click Next
40
Notes The installer will validate the state of the cluster before continuing. Action If there are issues you should rectify them before continuing. Click Next
Notes the installer lists a summary of the planned actions Actions Click Install
41
Notes The installer requires to run root102.sh as root on each the node Action (to be performed on each node, one after the other) Log in as the root user and enter the following command to shut down the Oracle Clusterware: # CRS_home/bin/crsctl stop crs Run the root102.sh script. It will automatically start the Oracle Clusterware on the patched node: 5. Oracle Clusterware patching 42
# CRS_home/install/root102.sh
Notes Here we can choose the type of database software to be installed Action Select the Enterprise Edition radio button Click Next
43
Notes Here we specify the name and location of ASM home. Modify as required (usually the Home name and Home Path should include the word ASM) Action Specify a location for the Oracle ASM Home Click Next
Notes The installer has detected the presence of Oracle Clusterware and uses this to populate this dialog box. To build a cluster which includes all nodes you must ensure that there are check-boxes next to the node names Action Click Select All Click Next
44
Notes The installer will validate the state of the cluster before continuing. If there are issues you should rectify them before continuing Actions Click Next
Notes We are going to install a Software only home and then subsequently configure the software Actions Select the Install Software Only radio button Click Next
Notes Here we can see a summary of the install. 6. Install Oracle ASM Software only Home 45
Notes Here the installer copies the software to all nodes in the cluster Actions none required
Notes The installer pauses, some scripts need to be run as root on both nodes of the cluster Action open a shell window on each node
46
Notes here we run the root scripts - they should only take a few seconds to run on each node Action run the scripts indicated in the previous screen on both nodes (one after the other) then return to the installer and Click OK
Notes After the software install completes you will see this End of Installation dialog Actions Click Exit to exit the installer ans click Yes to confirm Oracle ASM software home has been installed
47
Notes Specify the name and the location of the asm home Action Check that the name and location are correct
48
Notes Both nodes in the RAC Cluster are displayed Action Click Next.
Notes The installer will run Product-specific Prerequisite Checks . Action If there are issues you should rectify them before continuing. Click Next
49
Notes On this screen, you will register with Oracle Configuration Manager (OCM) Action If you want to register with OCM, enter details regarding the CSI Number, OracleMetaLink? Account user name and Country code (optional - The registration and configuration can also be done manually after patchset installation) Click Next
Notes Here we can see a summary of the install. Actions Click Install
50
Notes The installer pauses, root.sh need to be run as root on both nodes of the cluster Action open a shell window on each node and run root.sh (one after the other) then return to the installer and Click OK
Action The installation is done. Click Exit to exit the installer Click Yes in the box Oracle ASM software home has been patched to 10.2.0.4
51
Notes Netca detects that the Oracle Clusterware layer is running and offers Cluster or Single Node configuration Actions Select the Cluster configuration radio button Click Next
52
Notes Netca uses Oracle Clusterware to determine all the nodes in the cluster Actions Click Select all nodes Click Next
Notes You need to configure listeners Actions Select the Listener configuration radio button Click Next
Notes You need to add a listener Actions Select the Add radio button Click Next
53
Notes The default name is LISTENER. Do not change this. The listeners will eventually be called LISTENER_nodename1 & LISTENER_nodename2. This is important for RAC Actions Click Next
Notes Oracle Net supports various network protocols, although TCP is the most common. Actions Ensure the Selected Protocols list includes TCP Click Next
54
Notes It is possible to choose a non-default Port This is not recommended. Actions Ensure the Use the Standard port number of 1521 radio button is set Click Next
Notes After configuring the node listeners you get the opportunity to configure more network components Actions Select the No radio button Click Next
55
------------------- Notes You get the opportunity to configure other networking components Action Click the Finish button to exit the tool You can see the listener resources inside Oracle Clusterware by running crs_stat. $ ./crs_stat -t Name Type Target State Host -----------------------------------------------------------ora....13.lsnr application ONLINE ONLINE stnsp013 ora....013.gsd application ONLINE ONLINE stnsp013 ora....013.ons application ONLINE ONLINE stnsp013 ora....013.vip application ONLINE ONLINE stnsp013 ora....14.lsnr application ONLINE ONLINE stnsp014 ora....014.gsd application ONLINE ONLINE stnsp014 8.1. Create Node specific network listeners 56
ONLINE ONLINE
ONLINE ONLINE
stnsp014 stnsp014
A new managed resource has been added to Oracle Clusterware OCR for each listener. You have now completed the RAC cluster nodes Network listener configuration. Completed Node specific network listeners creation
Notes dbca detects the Oracle Clusterware layer is running and offers to create either cluster or single instance database Actions Select the Oracle Real Application Clusters database radio button Click Next
57
Notes We use dbca to create RAC databases and ASM instances Actions Select the Configure Automatic Storage Management radio button Click Next
Notes You need to make sure you create ASM instances on all the cluster nodes Actions Click Select All Click Next
9.1. Create ASM Instance and add the +DATA and +FLASH diskgroups
58
Notes Here we specify the password for the ASM Oracle SYS user Actions Enter the SYS password Enter the same password for the Confirm SYS password Click on the parameter file button Click Next
Notes dbca will create and start ASM instances Actions Click OK
9.1. Create ASM Instance and add the +DATA and +FLASH diskgroups
59
Notes ASM requires disks to be group together into diskgroups. This section will be used to create 2 disk groups +DATA and +FLASH Actions Click Create New
9.1. Create ASM Instance and add the +DATA and +FLASH diskgroups
60
Notes At the moment no disks are visible Actions Click Change Discovery Path
Notes Here we specifya filter to allow us to see the disks on the shared array Action Enter a filter to allow the installer to see the disks Click OK
9.1. Create ASM Instance and add the +DATA and +FLASH diskgroups
61
Notes Here you can now see thr disksw are visible Action none requried
Notes Now we will assign disks to specific disk groups and create the DATA diskgroup Actions In the Disk Group Name enter DATA Select the External Redundancy radio button Select the Show All radio button Select the 6 disks to be used for the DATA diskgroup Click OK
9.1. Create ASM Instance and add the +DATA and +FLASH diskgroups
62
Notes A progress message this should not take very long Actions none
Notes Here we can see the DATA diskgroup has been created and is mounted on 2/2 instances. We now need to create the FLASH diskgroup Actions Click Create New
9.1. Create ASM Instance and add the +DATA and +FLASH diskgroups
63
Notes We need to allow the installer to see the disks reserved for the FLASH disk group Action Click Change Discovery Path
Notes We need to modify the disk discovery string and to add the new path Action Modify the string Click OK
9.1. Create ASM Instance and add the +DATA and +FLASH diskgroups
64
* Notes Now we will assign disks to specific disk groups and create the FLASH diskgroup. This diskgroup is created with Normal redundancy Actions In the Disk Group Name enter FLASH Select the Normal radio button Select the Show Candidate disks radio button Select the remaining disks allocated for the FLASH diskgroup Click OK A progress message will be displayed.
Notes Here we can see the DATA and FLASH diskgroups have been created and are mounted on 2/2 instances. This completed the ASM configuration. Actions Click Finish
9.1. Create ASM Instance and add the +DATA and +FLASH diskgroups
65
Notes A confirmation dialog box Actions Click No The asm instances now are part of Oracle Clusterware [oracle11@stnsp013 bin]$ ./crs_stat -t Name Type Target State Host -----------------------------------------------------------ora....SM1.asm application ONLINE ONLINE stnsp013 ora....13.lsnr application ONLINE ONLINE stnsp013 ora....013.gsd application ONLINE ONLINE stnsp013 ora....013.ons application ONLINE ONLINE stnsp013 ora....013.vip application ONLINE ONLINE stnsp013 ora....SM2.asm application ONLINE ONLINE stnsp014 ora....14.lsnr application ONLINE ONLINE stnsp014 ora....014.gsd application ONLINE ONLINE stnsp014 ora....014.ons application ONLINE ONLINE stnsp014 ora....014.vip application ONLINE ONLINE stnsp014 Creation of ASM Instances and addition of the +DATA and +FLASH diskgroups are completed Congratulations you have installed ASM, Network listeners and created 2 ASM diskgroups
66
Notes Here we can choose the type of database software to be installed Action Select the Enterprise Edition radio button Click Next
Notes Here we specify the name and location of ASM home. Modify as required (usually the Home name and Home Path should include the word ASM) Action Specify a location for the Oracle ASM Home Click Next
67
Notes The installer has detected the presence of Oracle Clusterware and uses this to populate this dialog box. To build a cluster which includes all nodes you must ensure that there are check-boxes next to the node names Action Click Select All Click Next
Notes The installer will validate the state of the cluster before continuing. If there are issues you should rectify them before continuing Actions Click Next
68
Notes The installer has detected another instance (ASM) and asks if you want to upgrade Actions Click No
Notes We are going to install a Software only home and then subsequently configure the software Actions Select the Install Software Only radio button Click Next
69
Notes Here we can see a summary of the install. Actions Click Install
Notes Here the installer installs and copies the software to all nodes in the cluster Actions none required
70
Notes The installer pauses, some scripts need to be run as root on both nodes of the cluster Action open a shell window on each node
Notes here we run the root scripts - they should only take a few seconds to run on each node Action run the scripts indicated in the previous screen on both nodes (one after the other) then return to the installer and Click OK
71
Notes After the software install completes you will see this End of Installation dialog Actions Click Exit to exit the installer ans click Yes to confirm Oracle RAC software home has been installed
72
Notes Specify the name and the location of the RAC home Action Check that the name and location are correct
Notes Both nodes in the RAC Cluster are displayed Action Click Next.
Notes The installer will run Product-specific Prerequisite Checks . Action If there are issues you should rectify them before continuing. Click Next
73
Notes On this screen, you will register with Oracle Configuration Manager (OCM) Action If you want to register with OCM, enter details regarding the CSI Number, OracleMetaLink? Account user name and Country code (optional - The registration and configuration can also be done manually after patchset installation) Click Next
Notes Here we can see a summary of the install. Actions Click Install
74
Notes The installer pauses, root.sh need to be run as root on all nodes of the cluster Action open a shell window on each node and run root.sh (one after the other) then return to the installer and Click OK
75
Action The installation is done. Click Exit to exit the installer Click Yes in the box Oracle RAC software home has been patched to 10.2.0.4
Notes dbca detects the Oracle Clusterware layer is running and offers to create either cluster or single instance database Actions
76
Select the Oracle Real Application Clusters database radio button Click Next
Notes We use dbca to create RAC databases Actions Select the Create a Database radio button Click Next
Notes You need to make sure you create RAC database instances on all the cluster nodes Actions Click Select All Click Next
77
Notes Here you get to select the type of database Actions Select the General Purpose... radio button Click Next
Notes Now you specify the preface for the SIDs for the RAC database instances recommendation is to keep it simple Actions Enter your database name in the Global Database name, The SID should autofill Click Next
78
Notes Here you get to configure Oracle Enterprise Manager. If you have no Grid Control server then the best method is to use dbcontrol as detailed here. Actions Check the Configure Enterprise manager checkbox Select the Configure Database Control for local management radio button Click Next
Notes Here we specify the password for the Oracle Database users. In this example we are setting the same password for all users. You should set a password scheme that meets your requirements. Actions Select the Use the Same Administrative Password radio button Enter the password Enter the same password for the Confirm password Click Next
79
Notes Here you specify where you would like your database datafiles stored. We are going to use the ASM diskgroups we created earlier. Actions Select the Automatic Storage Management (ASM) radio button Click Next
Notes dbca displays the diskgroups we previously created. We are going to use the DATA diskgroup Actions Check the box on the DATA diskgroup line Click Next
80
Notes We will use Oracle Managed files . All the database files will be created on the DATA diskgroup. Actions Click Next
Notes Here we specify recovery configuration information. We are going to use a flash recovery area. Actions Check the Specify Flash Recovery Area checkbox In the Flash recovery area enter +FLASH Set the flash recovery size remember the size of the +FASH diskgroup Check the Enable Archiving checkbox Click Next
81
Notes Here you can choose to add the Oracle Sample schemas Actions Check the Sample schemas checkbox Click Next
Notes Here you can define services Actions We will not define additional services during the database creation Click Next
82
Notes Here you can modify the default initialisation parameters. Actions Click Next
Notes Here you can review the placement of various database files Actions Click Next
83
Notes You can decide to generate database creation scripts Actions Check the Create Database checkbox Click Next
84
Notes After the database is created the summary screen is displayed. Note the URL for the Database Control Actions Click on Password Management if you want to manage and/or unlock passwords Click Exit
85
Notes The Database instances will be started on all nodes Actions none You can see that the cluster has registered started the database instances on each node # cd <CRS_home>/bin # ./crs_stat -t Name Type Target State Host -----------------------------------------------------------ora....SM1.asm application ONLINE ONLINE stnsp013 ora....13.lsnr application ONLINE ONLINE stnsp013 ora....013.gsd application ONLINE ONLINE stnsp013 ora....013.ons application ONLINE ONLINE stnsp013 ora....013.vip application ONLINE ONLINE stnsp013 ora....SM2.asm application ONLINE ONLINE stnsp014 ora....14.lsnr application ONLINE ONLINE stnsp014 ora....014.gsd application ONLINE ONLINE stnsp014 ora....014.ons application ONLINE ONLINE stnsp014 ora....014.vip application ONLINE ONLINE stnsp014 ora.wlm.db application ONLINE ONLINE stnsp013 ora....m1.inst application ONLINE ONLINE stnsp013 ora....m2.inst application ONLINE ONLINE stnsp014 RAC database creation is complete
86