This action might not be possible to undo. Are you sure you want to continue?
By Frits Hoogland
Version 1.1 Removed usage of raw devices (deprecated in RHEL5), now usage by directly accessing the blockdevices (possible since 10.1.2.0.2; ML357492.1)
For research, testing and development purposes only. This is a non oracle supported configuration! Get Centos 5.1 disks from: http://isoredirect.centos.org/centos/5/isos/i386/. Choose mirror link from “Actual Country –“, download CentOS-5.1-i386-bin-1of6.iso, 2of6, 3of6 and 4of6. I assume the virtual machines have access to the internet. The used VMware software is VMware server 1.0.3 Please mind no VMware player nor VMware workstation can be used (shared scsi devices are not possible using this vmware products). 1. Create new virtual machine in vmware - Typical - Linux, version “Other Linux 2.6.x kernel” - Name: “centos51-rac-1”, location “/vmware/vms/centos51-rac-1” - Use bridged networking - Disk size: 20G, uncheck “allocate all disk space now”, uncheck “split disk into 2GB files” 2. Edit virtual machine settings - Set memory: 1G (1024MB) - Hard Disk (SCSI 0:0) is alright, this is for the linux installation - CD-ROM 1 (IDE 1:0): select “Use ISO image”, and point it to the first centos 5.1 iso, CentOS-5.1-i386-bin-1of6.iso. - Remove “floppy 1” - Add Ethernet Adapter, Host-only. - Add Hard Disk, Create a new virtual disk, SCSI, Disk Size 10GB, uncheck “allocate all disk space now”, uncheck “split disk into 2GB files”, disk file location “/vmware/vms/centos51-rac-shared/centos51-rac-shared/centos51-racshared.vmdk” - Click on the newly created hard disk, and click “Advanced…”, select “SCSI 1:0” from “Virtual Device Node”. - Edit the virtual machine settings file (“/vmware/vms/centos51-rac-1/centos51-rac1.vmx” in this case) The disk which we are going to use shared is on SCSI device 1, the host disks/non shared are on SCSI device 0. Now we are going to enable SCSI reservation for the shared disk. Add the following two lines: scsi1.sharedBus = virtual disk.locking = “false” 3. Linux installation for centos51-rac-1 - Startup the virtual machine centos51-rac-1.
Because there is no operating system on the internal disks, the installer from the ISO will boot. Press enter to commence installation CD Found: choose “Skip”, enter Now the graphical installer starts. Choose next at the splash-screen What language would you like to use during the installation process? Choose “English (English)” (default), and next Select the appropriate keyboard for the system. Choose your keyboard type, next Warning: The partition table on the device sda was unreadable. This is alright, the (virtual) disk is still untouched, so there is nothing on it. Choose “Yes” Warning: The partition table on the device sdb was unreadable. This is alright too, this (virtual) disks is untouched too. Choose “Yes” Installation requires partitioning of your hard drive. Choose “Create custom layout”, next Partition screen o New Filesystem type: swap Allowable Drives: Uncheck “sdb” Size (MB): 2000 OK o New Mount Point: / Allowable Drives: Uncheck “sdb” Additional Size Options: Fill to maximum allowable size OK Next
Bootloader screen Everything is set correctly here, choose next Network Devices Enable “Active on Boot” for eth1 Push the “Edit” button In the Enable IPv4 support area: Select “Manual configuration” IP Address: 10.10.10.1 Prefix: 24 Hostname: Manually: centos51-rac-1 (eth0 is the public NIC, which is connected to my network. My DHCP server releases static ip-addresses, which means it is safe to use that) Please click into the map to choose a region: Choose your locality, next Root password Choose twice, then click next Software packages Uncheck “Desktop – Gnome”, check “Server – GUI”, next
Click next to begin installation of CentOS. A “Required Install Media” screen will appear. It lists the 4 CD’s mentioned earlier. Click Continue. The installation starts. - At some point, the installer asks for CentOS disc 2 (“Please insert CentOS disc 2 to continue”). Click on the CD ROM icon in the lower right corner. This will open the Virtual Machine Settings window. Choose the second disc (CentOS-5.1-i386-bin2of6.iso), then click OK. Click also on OK in the Change CDROM window of the virtual machine. Repeat those steps for disc 3 and 4. - Congratulations, the installation is complete. Click “Reboot” 4. First time boot - After the installation, the fresh installed linux system boots, and asks some questions: - Welcome: “Forward” - Firewall: Disabled, Forward, Yes (for real production environments, firewalling is mandatory, but for this test setup it has no added value) - SELinux: Disabled, Forward, Yes - Enable kdump: Leave unchecked, Forward - Date and time: Click on the Network Time Protocol tab. Check “enable network time protocol” Delete the 3 already existing ntp servers. Add “pool.ntp.org” as a time server. - Create User: Username: admin Full name: admin Password: ****** Confirm Password: ****** “Forward” - Sound Card: No sound cards were detected. Forward - Additional CDs: Finish - The system must now reboot for some of your selections to take effect. OK 5. System ready - Logon using the admin account. - Right click on the background, and choose “Open Terminal” - Update the system with the latest packages (in the terminal): [admin@centos51-rac-1 ~]$ su – Password: [root@centos51-rac-1 ~]# yum update ... Total download size: 112 M Is this ok [y/N]: y ... Importing GPG key 0xE8562897 “CentOS-5 Key (CentOS 5 Official Signing Key) email@example.com” from http://mirror.centos.org/centos/RPM-GPG-KentOS-5 Is this ok [y/N]: y
... Complete! [root@centos51-rac-1 ~]# - Install the required packages from the Oracle Database Quick installation Guide for Linux x86 [root@centos51-rac-1 ~]# yum install compat-libstdc++-33 elfutils-libelf-devel glibc-devel gcc gcc-c++ libaiodevel libstdc++-devel sysstat unixODBC-devel ... Total download size: 23 M Is this ok [y/N]: y ... Complete! [root@centos51-rac-1 ~]# 6. Install second linux machine. - Follow steps: 1 : machine name now should be “centos51-rac-2” 2 : do not add the extra Hard Disk. (this is done manually in the vmx file later) 3 : Network devices: eth1: ip address should be 10.10.10.2 hostname “centos51-rac-2” 4 : no comments 5 : no comments - Add the following lines to the settings file (/vmware/vms/centos51-rac-2/centos51rac-2.vmx) scsi1.present = “TRUE” scsi1:0.present = “TRUE” scsi1:0.filename = “/vmware/vms/centos51-racshared/centos51-rac-shared.vmdk” scsi1.virtualDev = “lsilogic” scsi1.sharedBus = virtual disk.locking = “false” scsi1:0.redo = “” scsi1:0.writeThrough = “TRUE” - Reboot centos51-rac-2 7. Configure network. (on both machines) Modify the following line in the /etc/hosts file (remove machinename from localhost): Old: 127.0.0.1 centos51-rac-1 localhost.localdomain localhost New: 127.0.0.1 localhost.localdomain localhost Add the following lines to the /etc/hosts file (replace public and vip addresses with your own ip addresses!): # public names 192.168.0.64 centos51-rac-1 192.168.0.63 centos51-rac-2 # private interconnect
2 # vip 192.10. Synching disks. default 38): [enter] Last cylinder or +size or +sizeK (1-1305.168..102 centos51-rac-1-priv centos51-rac-2-priv centos51-rac-1-vip centos51-rac-2-vip 8.10. Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-1305). default 1305): +300M Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 2 First cylinder (38-1305.10. default 75): [enter] Last cylinder or +size or +sizeK (1-1305.10. [root@centos51-rac-1 ~]# cd /etc/udev/rules. Prepare shared disks (fdisk only needs to be done on one of the machines) [root@centos51-rac-1 ~]# fdisk /dev/sdb . default 1305): +300M Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 3 First cylinder (75-1305.10. [root@centos51-rac-1 ~]# Now the partitions are created.. default 1): [enter] Last cylinder or +size or +sizeK (1-1305.0. Permissions must be set on every machine. The next step is to correct the device permissions.d .0. default 1305): [enter] Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table.1 10.101 192.168.
4-1.d]# groupadd –g 505 asmdba [root@centos51-rac-1 rules. KERNEL==”sdb1”.0.0. OWNER=”root”.rules KERNEL==”raw1”. RUN+=”/bin/raw /dev/raw/raw2 %N” Create a new file called 99-raw.rpm oracleasm2.4.rpm Preparing . set password for the user “oracle” [root@centos51-rac-1 rules. and download: .0.1.rpm oracleasmlib-2.6..d]# groupadd –g 504 asm [root@centos51-rac-1 rules.d]# groupadd –g 502 dba [root@centos51-rac-1 rules.18-53.el5. GROUP=”oinstall”.el5.d]# vi 60-raw.el5.Usage of raw devices is deprecated in RHEL5 These steps still can be followed. MODE=”0640” 9.Userspace library (oracleasmlib-2.rpm) .i686.Kernel driver (oracleasm-2.el5. and enter the following information: [root@centos51-rac-1 rules. MODE=”0640” KERNEL==”sdb2”. ######################## [100%] 1:oracleasm-support ######################## [100%] 2:oracleasm-2. OWNER=”oracle”. add the following lines: [root@centos51-rac-1 rules. In 60-raw.Support files (oracleasm-support-2. OWNER=”root”.d]# vi 99-raw.i386.3-1.rules.i386.oracle.rules.oper. but it is advised to use the rules setting below this area.d]# vi 50-udev.1.0. Install ASMLIB ASMLIB must be installed on every machine! Goto http://www. GROUP=”oinstall”. OWNER=”oracle”.i386.4.0. KERNEL==”sdb2”.d]# groupadd –g 501 oinstall [root@centos51-rac-1 rules.4-1.0.rules KERNEL==”sdb1”.com/technology/software/tech/linux/asmlib/rhel5. [root@centos51-rac-1 rules.rpm) [root@centos51-rac-1 ~]# rpm –Uvh oracleasm-support-2.el5-2.18-53.rpm) . RUN+=”/bin/raw /dev/raw/raw1 %N” ACTION==”add”.el5-2.d]# groupadd –g 503 oper [root@centos51-rac-1 rules.4 ######################## [100%] 3:oracleasmlib ######################## [100%] Initialise ASMLIB on both machines: .el5. Create oracle users and groups Users and groups must be added on every machine.d]# passwd oracle 10.rules ACTION==”add”. GROUP=”oinstall”.18-53. MODE=”0640” KERNEL==”raw2”.1.rules at the bottom of the file: [root@centos51-rac-1 rules.4-1.41.i386.6.3-1.html. MODE=”0640” Add to the file 50-udev..asm –d /home/oracle –s /bin/bash –c “Oracle software owner” oracle Next.6.d]# useradd –m –u 501 –g oinstall –G dba.i686. GROUP=”oinstall”.el5.
issue a scan command: [root@centos51-rac-1 ~]# /etc/init.ssh/id_rsa. or specifying a real fully qualified domain name (FQDN).ssh/id_rsa): [enter] Create directory „/home/oracle. Ctrl-C will about.ssh‟. The current values will be shown in brackets („‟). Enter file in which to save the key (/home/oracle/.d/oracleasm listdisks VOL1 [root@centos51-rac-1 ~]# 11.d/oracleasm scandisks Scanning system for ASM disks: [ OK ] [root@centos51-rac-1 ~]# /etc/init. Enter passphrase (empty for no passphrase): [enter] Enter same passphrase again: [enter] Your identification has been saved in /home/oracle/.[root@centos51-rac-1 ~]# /etc/init. Default user to own the driver interface : oracle Default group to own the driver interface : oinstall Start Oracle ASM library driver on boot (y/n) [n]: y Fix permissions of the Oracle ASM disks on boot (y/n) [y]: y Writing Oracle ASM library driver configuration: [ OK ] Creating /dev/oracleasm mount point: [ OK ] Loading modules “oracleasm”: [ OK ] Mounting ASMlib driver filesystem: [ OK ] Scanning system for ASM disks: [ OK ] Create ASM disks. Because we focus on the Oracle RAC database. The following questions will determine whether the driver is loaded on boot and what permissions it will have. Set hosts equivalence for the oracle user using ssh public keys [root@centos51-rac-1 ~]# su – oracle [oracle@centos51-rac-1 ~]$ ssh-keygen –t rsa Generating public/private rsa key pair. Resolving this can either be disabling sendmail. . it will cause sendmail to wait for 1 minute during startup. we just disable sendmail. Disable sendmail Because we have specified a “short” name for the machines. [root@centos51-rac-1 ~]# chkconfig -–del sendmail 12. Do this on both machines.d/oracleasm configure Configuring the Oracle ASM Library driver This will configure the on-boot properties of the Oracle ASM library driver.d/oracleasm createdisk VOL1 /dev/sdb3 Marking disk “/dev/sdb3” as an ASM disk: [ OK ] [root@centos51-rac-1 ~]# On all the other machines. This step can be done on only one machine: [root@centos51-rac-1 ~]# /etc/init. Hitting <ENTER> without typing an answer will keep that current value.
Let’s add this key to the authorized keys on the other machine: [oracle@centos51-rac-1 ~]$ ssh-copy-id –i ~/.core.pub.pub oracle@centos51-rac-1 Do the same (generate keys.63)‟ can‟t be stablished.core. set oracle user equivalence for other and current machine) for machine centos51-rac-2. with “ssh „oracle@centos51rac-2‟”. 13.192.conf Add the following lines: kernel.Your public key has bene saved in /home/oracle/.rmem_default = 4194304 net. Are you shure you want to continue connecting (yes/no)? yes Warning: Permanently added „centos51-rac-2.168.0.wmem_default = 262144 net.core. [root@centos51-rac-1 ~]# vi /etc/sysctl. and check in: . RSA key fingerprint is f2:31:69:7b:72:24:bf:1a:16:05:a2:01:73:7a:99:7f.63‟ (RSA) to the list of known hosts.ssh/authorized_keys To make sure we haven‟t added extra keys that you weren‟t expecting.168.file-max = 65536 net.rmem_max = 4194304 net.pub oracle@centos51-rac-2 28 The authenticity of host „centos51-rac-2 (192. The key fingerprint is: 70:2c:af:49:72:60:a1:0a:db:da:9f:47:83:34:78 Now a public/private keypair has been generated.sem = 250 32000 100 128 fs. oracle@centos51-rac-2‟s password: [enter password] Now try logging into the machine.ip_local_port_range = 1024 65000 net.ssh/id_rsa.wmem_max = 262144 14.ipv4.ssh/id_rsa.0.core. [oracle@centos51-rac-1 ~]$ Redo this step for the current machine (!!) [oracle@centos51-rac-1 ~]$ ssh-copy-id –i ~/. .ssh/id_rsa. Set kernel parameters This must be done on both machines. Set shell limits for the oracle user This must be done on both machines.
6 ORACLE_SID=rac1 PATH=$PATH:$ORACLE_HOME/bin/:$ORA_CRS_HOME/bin Please mind that this must be done on the second node too.0. This must be done on both machines.0.1. but ORACLE_SID needs to be set to ‘rac2’! 16.1. Centos comes with a “centos-release” file. [oracle@centos51-rac-1 ~]$ cd /oracle/sw [oracle@centos51-rac-1 sw]$ unzip linux_11gR1_clusterware. Get rpm-build to be able to build rpm’s: [root@centos51-rac-1 ~]# yum install rpm-build .[root@centos51-rac-1 ~]# vi /etc/security/limits. Pre-generate oracle directories This must be done on both machines..so 15.oinstall /oracle 17.bash_profile Add the following lines: export export export export export ORACLE_BASE=/oracle ORACLE_HOME=/oracle/db/11.6 ORA_CRS_HOME=/oracle/crs/11. Set environment variables in oracle profile: [oracle@centos51-rac-1 ~]$ vi . [root@centos51-rac-1 ~]# mkdir /oracle [root@centos51-rac-1 ~]# chown oracle. Get redhat-release package.zip . we need to install that.. Because cluvfy (cluster verification utility) checks the operating system version using the redhat-release package.conf Add the following lines: oracle oracle oracle oracle soft hard soft hard nproc 2047 nproc 16384 nofile 1024 nofile 65536 [root@centos51-rac-1 ~]# vi /etc/pam.d/login Add the following line: session required /lib/security/pam_limits. Get and extract clusterware software I’ve downloaded the clusterware software in the /oracle/sw directory. [oracle@centos51-rac-1 sw]$ 18.
..1.0.centos51-rac-2 –verbose .redhat.i386.centos51-rac-2 –verbose .0..2.i386. the allocated memory should be the only thing which failed: .2.2.------centos51-rac-1 1011.rpm Preparing.src..rpm .. Install the cvuqdisk package from the clusterware software rpm directory (on both machines) [root@centos51-rac-1 ~]# rpm –Uvh /oracle/sw/clusterware/rpm/cvuqdisk-1./runcluvfy.. Check the output of cluvfy. System requirement failed for „crs‟ Pre-check for cluster services setup was unsuccessful on all the nodes.0.rpm Preparing.sh stage –pre crsinst – n centos51-rac-1. Run cluvfy to check clusterware requirements [oracle@centos51-rac-1 sw]$ cd /oracle/sw/clusterware [oracle@centos51-rac-1 sw]$ ..sh stage –post hwos –n centos51-rac-1..06MB (1035324KB) 1GB (1048576KB) failed centos51-rac-1 1011.1-1./runcluvfy. 20.rpm .. Install newly generated rpm: [root@centos51-rac-1 ~]# rpm –Uvh /usr/src/redhat/RPMS/i386/redhat-release-5Server5.com/pub/redhat/linux/enterprise/5Server/en/os/ SRPMS/redhat-release-5Server-5. ############################ [100%] 1:cvuqdisk ############################ [100%] Next...-------------------. ########################### [100%] 1:redhat-release ########################### [100%] 19. Run cluvfy to verify hardware and operating system setup. Check: Total memory Node Name Available Required Comment -------------...1. Wrote: /usr/src/redhat/RPMS/i386/redhat-release-5Server5. ..1. run cluvfy: [oracle@centos51-rac-1 sw]$ .--------------... .1.rpm Build package: [root@centos51-rac-1 ~]# rpmbuild -–rebuild redhat-release5Server-5.0.06MB (1035324KB) 1GB (1048576KB) failed Result: Total memory check failed..2. Get source rpm of redhat-release [root@centos51-rac-1 ~]# wget ftp://ftp..src.0..
. . due to smartctl not functioning correctly on other than pure SCSI devices.centos51rac-2”.. Click Next. These error messages are alright. Install clusterware [oracle@centos51-rac-1 sw]$ /oracle/sw/clusterware/runInstaller .centos51-rac-2 WARNING: Unable to determine the sharedness of /dev/sdb on the nodes: centos51-rac-1.centos51-rac-2 Shared storage check failed on nodes “centos51-rac-1. 21. Post-check for hardware and operating system setup was unsuccessful on all the nodes.Checking shared storage accessibility… WARNING: Unable to determine the sharedness of /dev/sda on the nodes: centos51-rac-1.
Then click “Next” .Click OK Specify “/oracle/oraInventory” as the full path for the inventory directory.
Then click “Next” .1.Specify “/oracle/crs/11.0.6” as path.
Add a cluster node (using the “Add” button). and specify Public Node Name: centos51-rac-2 Private Node Name: centos51-rac-2-priv Virtual Host Name: centos51-rac-2-vip Then click “Next” .Leave “centos5-cluster” as clustername.
.Click on eth0. then on the “Edit” button. Then click “Next”. Set the interface to “Public”.
Choose “External Redundancy”. Specify “/dev/sdb1” at “Specify OCR Location” Then click “Next” .
.Specify “External Redundancy” Specify “/dev/sdb2” as “Voting Disk Location” Then click “Next”.
Click “Install” .
sh.sh. Some assistants will be run (without manual intervention needed). press “OK”. then to centos51-rac-2 (as root) and execute orainstRoot. Appears (and prompt is returned to the terminal). After that. After that. Then run root. logon to centos51-rac-1 (as root). Wait until the prompt comes back and the message: (this can take some time!) Local node checking complete.sh on remaining nodes to start CRS daemons. Run root.Logon to centos51-rac-1 (as root) and execute orainstRoot.sh. and execute root.sh on centos51-rac-2 (as root). .
[oracle@centos51-rac-1 ~]$ cd /oracle/sw [oracle@centos51-rac-1 sw]$ unzip linux_11gR1_database. [oracle@centos51-rac-1 sw]$ 23.centos51-rac-2 –r 11gR1 -verbose This should only fail for the memory check. [root@centos51-rac-1 ~]# chown oracle /oracle 24.. Set ownership of the /oracle directory back to oracle Set the ownership back on both machines. Run cluvfy to validate RAC database requirements.. 25. Install oracle database software [oracle@centos51-rac-1 ~]$ cd /oracle/sw/database [oracle@centos51-rac-1 ~]$ .Press “Exit” 22./runInstaller .. Get and extract database software I’ve downloaded the database software in the /oracle/sw directory. .zip .. [oracle@centos51-rac-1 ~]$ cluvfy stage –pre dbinst –n centos51-rac-1.
Click “Next” .
Select “Enterprise Edition” (default). click “Next” .
All path’s are filled out correct. Click “Next” .
. then click “Next”.Select Node “centos51-rac-2” too (or click “Select All”).
and behind “Checking PATH environment variable” Then click “Next” .Click on the square behind “Checking Network Configuration requirements”.
.Select “Install Software Only”. then click “Next”.
Check the groups: Database Administrator (OSDBA) Group: dba Database Operator (OSOPER) Group: oper ASM administrator (OSASM) Group: asm Then click “Next” .
Click “Install” .
Then click “OK” Click “Exit” 26.6/root.0.Execute “/oracle/db/11.sh” on both centos51-rac-1 and centos51-rac-2 as root. Create listener [oracle@centos51-rac-1 ~]$ netca Oracle Net Services Configuration: .1. Press [enter] on the question about the full path name of the local bin directory.
Click “Next” Click “Next” .
then “Next” Click “Next” .Select “Listener configuration” (default).
Click “Next” Click “Next” .
Click “Next” Click “Next” .
. Oracle Net Services configuration successful.centos51-rac-2 –d $ORACLE_HOME –verbose This should report: Pre-check for database configuration was successful. The exit code is 0 27.. Listener configuration complete. Run cluvfy to check cluster database configuration [oracle@centos51-rac-1 ~]$ cluvfy stage –pre dbcfg –n centos51rac-1. Create RAC database [oracle@centos51-rac-1 ~]$ dbca . 28. centos51-rac-2..Click “Finish” Configuring Listener:LISTENER centos51-rac-1..
click “Next” . click “Next” Select “Create a Database” (default).Select “Oracle Real Application Clusters database” (default).
then click “Next” Select “Custom Database”. then click “Next” .Click “Select All”.
Then click “Next” Leave the defaults (as shown in the screenshot). then click “Next” .Enter “rac” at “Global Database Name”. Sid will be filled out automatically.
and type (and retype) the password you choose.Select “Use the same administrative password for all accounts”. Then click “Next” .
then click “Next”.Select “Automatic Storage Management (ASM)”. Then click “Next” Click “OK” . type and retype the ASM sys password. Choose.
Click “Create New” .
Set “Redundancy” to “External”. Then click “OK” . And Click on the square before “ORCL:VOL1”.Enter “VOL” as “Disk Group Name”.
Click “Next” Click “Next” .
Click “Next” Click “Next” .
Click “Next” Click “Next” .
Click “Next” Click “Next” .
Click “Next” .
Click “OK” .
Click “Exit” After pressing “Exit”.portal_db?selected=11&frame=) Build your own oracle RAC cluster on Oracle Enterprise Linux and iSCSI by Jeff Hunter (http://www.html) . References: Oracle 11g documentation (http://www.com/technology/pub/articles/hunter_rac11gr1_iscsi. After the instances have been started. a window appears which tells us the assistant is starting the cluster database instances.oracle.oracle.com/pls/db111/portal. it disappears.