You are on page 1of 199

Building a Virtual Oracle RAC – Project Introduction

Filed under: Networking in Virtual Environment, Oracle Database, Virtualization — Tags:
bridged, configuration, host-only, installation, iSCSI server, iSCSI target, Linux, network, oel,
Openfiler, oracle, rac, vbox, virtual box — oraclelabs @ 17:18

19 Votes

Introduction: This learning experiment is about building a totally virtual Oracle RAC
environment at home using only tools and software available to wide public, preferably open
source (when possible).

My project is generally based on an excellent article by Jeffrey Hunter (and other authors) –
Build Your Own Oracle RAC Cluster on Oracle Enterprise Linux and iSCSI

Jeffrey’s article gives instructions on how to build a physical setup. I do recommend reading this
article or at least giving it some 10 minutes scan before you attempt to build a RAC. In that
article Jeffrey estimates a cost to build a production RAC in physical environment between $20K
and $30K. He shows that for the purposes of learning the technology a “mostly physical” RAC
with some parts virtualized can be built for under $2700. I am going to take the virtualization to
the extreme and build a RAC inside a single PC. Yes, I do not care now about performance, and
yes, I am willing to buy some additional memory for the PC. You can not really expect to run
two Linux nodes with databases and a storage (SAN) simulation on a PC with 1GB of memory.
My estimated cost is, therefore, somewhere under $100, if all we need is a memory upgrade.
Sounds good? It is still a lot of work and it will probably take you many hours to do it right. The
benefit is a hands on knowledge, which is priceless.

Now to the choice of hardware and software.

 PC or server with some 200GB space and 3.5GB memory (XP Professional can handle
up to 4GB).
 The host OS: Windows XP SP3 professional (although SP2 will probably do as well)
 Virtualization software: Sun’s Virtual Box 3.1.2 (or higher). Works with no issues on
both Intel and AMD processors.
 Database nodes: Oracle Enterprise Linux 5 update 3 as a guest OS, running Oracle
 SAN/NAS emulation software: Openfiler 2.3

My reasons for choosing particular software:

Host OS does not really matter, virtualization software can run on many different platforms. If
you desire so, you can run this whole project using Linux as a host. Virtual Box is selected
because it is an open source software produced by Sun Microsystems (which belongs to Oracle

Why Sun Virtual Box and not VMware? VMware is a popular product but I do not see a
consistency in their support of free versions of the software. In my experiments with VMware I
have encountered some obstacles while running Linux guests (particularly in networking and
CPU clock synchronization). Virtual Box is an open source product and quite a mature one, well
documented and supported. So far I was able to run OEL4 and OEL5 with no issues (even with
no guest additions installed). I like the product’s built-in networking, it seems to be more flexible
and stable. These are all my personal preferences, of course.

Why Oracle’s Linux? This particular flavor of Linux is a modified Red Hat Enterprise Linux 5.
Oracle has made a few adjustments that allow Oracle 10g R2 database (32 bit) to install with no
issues on it (usually, no additional RPM’s required). Since I am aiming to install Oracle in the
guest(s) thus the choice of this Linux flavor. Besides, this Linux comes with OCFS2 and most of
the ASMLib software packages installed. If you have a licensing question about Oracle
Enterprise Linux (OEL) I can answer it. You only need a license if you want Oracle’s support.
For learning purposes you do not need a license. If you will need to download some rpm libraries
and updates you can get those for free, the YUM configuration can be pointed to a free
repository (not discussed here).

Now the database itself. Which edition of are we going to use? There is no clarity in that
article on this matter, for now I assume the Standard Edition will do. There are some additional
pieces of software from Oracle (or used by Oracle) to be installed, those are: OCFS2, ASMLib
2.0 and TAF.

Now that we know what we need to get started let’s split the work into more manageable mini
projects, follow them just in this order, and hopefully all pieces will fit:

 Task 1. Installing and configuring Sun Virtual Box.

 Task 2. Installing Openfiler in Sun Virtual Box.

 Task 3. Configuring Openfiler to simulate iSCSI targets.

 Task 4. Creating a Linux guest, configuring hardware and Installing Linux.

 Task 5. Patching Linux guest.

 Task 6. Configuring iSCSI targets in Linux and partitioning the volumes.

 Task 7. Preparing Linux guest for Oracle installation – part1

 Task 8. Cloning a Linux guest.

 Task 9. Preparing Linux guest for Oracle installation – part2

 Task 10. Oracle installation in RAC nodes – part 1.

 Task 11. Oracle installation in RAC nodes – part 2.

 Task 12. Creating Oracle database in RAC nodes.

 Task 13. Having some fun with RAC.

Virtual Oracle RAC. Task 1: Install and configure Sun Virtual Box

3 Votes

Link back: This guide is a part of the Virtual Oracle RAC project, the index to the whole project
is here.

Here is an overview of network setup that we will do in Virtual Box environment. This picture
below shows three sets of network adapters in guests.

eth0 is for public network

eth1 is for private network

of course. Create it. The notion of "network" in this particular part of Virtual Box may be likened to creating a router with a static IP. Do proceed.Download and install Sun Virtual Box (follow Sun’s instructions. access Virtual Box preferences Make sure Host-Only network is installed (it must be there by default) and it is the only network present. Run Virtual Box. This "router" will be accessible from host’s network but not from outside of it (not from other hosts on your network). if it is not there. that during the installation of networks there may be pup-ups asking you if you want to proceed with unsigned driver installation. the installation is quite straightforward). . Notice.

. close all settings dialogs and wait till another adapter is created. it is placed on a totally separate network from your host’s LAN adapter): Now create another (second) Host-Only adapter: Click on "Ok".Configure Host-Only adapter as below (yes.

Task 2: Openfiler – installation in Sun Virtual Box 4 Votes Link back: This guide is a part of the Virtual Oracle RAC project. Some projects I looked at use Linux guests (in virtual machine) to simulate iSCSI using .e. to your router). Disclaimer: in my network setup I have used a domain name that can be taken or belong to someone. These iSCSI targets can be created in a variety of ways. If you are planning to experiment with Oracle RAC you will need a SAN (NAS). this one does not have to be created. Reminder: we are in a virtualized environment so we do not need to run to a store and buy real hard drives. the index to the whole project is here. We will use software to simulate hard drives and disk volumes. It all stems down to having iSCSI targets (server) somewhere on your network. Host-Only networks is here but not connected to your gateway (i. Virtual Oracle RAC.Save your changes and enter "ipconfig" command from host command prompt. it appears automatically when "Bridged" adapters are created in guests. I used it just for illustration only. At this point the host networks will look like this: As you can see. Presently it appears that all possible and impossible domain names are taken but I have no claims on the domain name. it is by pure coincidence and I apologize for any inconvenience it may have caused. If such is the case. From the network diagram you will notice that there is a "Bridged" network "router".

org Get Openfiler installation media from .3 (x86) in Sun Virtual Box. I named the guest Openfiler2 just in case to avoid any possible trouble later when we come to the networking part. Openfiler is essentially free. Let’s create a new guest that will become our Openfiler server. for instance. My choice is Openfiler. If you want to know more about Openfiler (other than what you can find on their web site) you can refer to www. Seems to work fine. I am going to install Openfiler 2. . Such is.rpath. which is based on a small Linux distro (well. remember?).iSCSI server features built into some Linux flavours. SUSE Linux Enterprise Server 10. however if you want to get the Admin Guide book it will cost you 40 GBP. of course) and gives you nice web based interface. "Red Hat" is the default and I did not bother to change it. it comes in a small image file of some 300MB size (best things come in a small package. I’d rather go with open source software for this purpose.

.For memory. this will take a chunk off your real memory so do not be too generous. Remember. 256MB RAM should suffice for now.

e. i. We will create a new hard disk. The virtual hard drive in this case is the "internal hard drive" in the Openfiler2 machine. the hard drive that will hold the OS. . This one is not the hard drive that will be used for iSCSI simulation (be patient).Now this.

This screen (can not even call it a dialog) is kind of redundant but we like to be welcomed anyway. right? .

We accept the default of dynamically expanding storage. .

Set the size to 800MB initially Click on Finish and … your new guest with its hard drive are created. .

org/wiki/NX_bit if you really really want to. Look up this part at http://en. . If it is not enabled.Now go back to the guest machine settings.wikipedia. there is some work to do here. This is very important! Enable PAE/NX support as shown below (if it not enabled by default in your version). Navigate to the System and then to Processor tab. your installation will fail at the very end (as it did for me).

Close the settings screen for the guest and bring up Virtual Media Manager (that’s a very cool thing. The result will look like this: . but wait a second. We need to plug some disks into the Openfiler to get our iSCSI targets. you have seen a part of it already when we created hard dist for Openfiler2): Use "New" dialogue to create a virtual hard drive in a file named "openfiler_rac" and sized at 40GB (just like if you were creating a file holding a guest VM). right? Where are those disks coming from? Here is what we need to do.Almost ready to run the installer.

. Now see what happens next. Add the hard disk to the vacant slot in the IDE controller (the capacity of one controller seems to be three IDE devices). Bring up the Openfiler2 guest settings and go to "Hard Disks" section.The "openfiler_rac" storage will be used in our RAC project to hold iSCSI drives.

we define two host-only adapters and one bridged adapter. another article suggested using an additional Linux guest machine for such a purpose (to simulate hard drives). why to go to such extent of complexity when Virtual Box provides much simpler means? So. The bridged adapter will later be used during NTP setup. I must point out that I have seen an article on the internet suggesting the use of physical external USB hard drive as a source disk for Openfiler (in a Virtual Box environment). Adapter 1: . I am wondering.When you run installation of your guest these disks will be detected as normal IDE drives. next in our setup is the network.

Adapter 2: .

they are not important to you): .And bridged Adapter 3 (do not copy down my MAC addresses.

Bring up the Virtual Media Manager again and go to CD/DVD images tab. . Then your adapters will simply vanish. This problem may not present itself in future releases of Virtual Box and Openfiler (hopefully). that is. Now all we need to do is to insert the installation media and push the reset button. This has to do with a driver supplied by VMware to Openfiler to support virtual network adapters (can be a problem in this particular version of software). To make the story short.Why this particular adapter type (Intel PRO)? Why not to go with the default? It worked for us well in other Linux installation. But how? Easy enough. only this Intel PRO type seems to survive after restarting the guest. until first reboot of the guest. The default type will work for you as well.

Register the installation media (the image file) and notice that the grayed line at the bottom reads "Not Attached". . which means that this file. is not used by any component at this time. although registered with the Media Manager.

if it is not there it can be added easily.Now bring back the Openfiler2 settings and go to Storage tab. There should be a DVD drive in our IDE controller. Click on "CD/DVD Device": .

Select the appropriate media (so here is the connection between registered media files and DVD drive in our virtual guest!) .

Now we see the the media file (image) appeared in the CD drive: . to/graphical-installation. Just start your guest.200 odbn1-vip. assign IP addresses shown below in Openfiler section at the bottom: # Notice that loopback address do not have a real host name in it 127.10.1 localhost.localdomain localhost # eth0 # Database nodes – Public Network odbn1 odbn1-vip .0.11 odbn1.12 odbn2.harzion. it will detect installation media and boot into installer.openfiler. When network cards detected during installation. Follow the generic installation guide for Openfiler here http://www.harzion.2.10. we are ready to start odbn2 # Database nodes – Public Virtual IP (VIP) addresses 10.harzion.10.0.

2. The /etc/hosts file shown above we will place in our Openfiler once installation is openfiler-priv The gateway.12 odbn2-priv.11 odbn2-priv # Openfiler eth0 – public 10. will be at openfiler # Openfiler eth1 – private 10.1 and hostname is "openfiler" odbn1-priv 10.1. of course. All future interactions with the running Openfiler are via a web interface.2.harzion.harzion. Notice that naming of the guest in Virtual Box has no effect on hostnames so do not confuse these two kind of names.10.20 openfiler-priv.10.20 openfiler.10.10.201 odbn2-vip # eth1 # Database nodes – Private Interconnect 10.harzion. When the guest boots it displays a reminder on where the web GUI sits: . same file will be present on all other hosts involved in the project.

Who could have guessed… I was going to try "1234". Virtual Oracle (notice that it deals with physical hardware rather than virtual but it makes no difference to us).oracle. Task 3: Openfiler – iSCSI targets setup .Default web user is: "openfiler" and the password is "password". like in "Spaceballs" movie. one of my favourites. I used a very good practical guide on how to create virtual volumes in the Openfiler that can be found here: http://www.

10.10.2.  Verify that “iSCSI target server” service is running: .2. the index to the whole project is here.1.2. as shown below (we have placed it in our Openfiler already): # Notice that loopback address do not have a real host name in it 127.html (the article gives instructions on how to build physical of course. we can start configuring our Oracle RAC. Now that we have Openfiler and OEL guests in Virtual Box.localdomain localhost# Database nodes – Private Interconnect – (eth0) 10. If you have followed instructions in my other articles on installing OEL your network will differ a bit and will need some while we are trying here to do the same in virtual environment).1. we will cover that.0.2. As a general guideline I am going to use this (excellent) article – http://www.10.1 Votes Link back: This guide is a part of the Virtual Oracle RAC project. The starting point is assigning names and network addresses to all our components.2.11 odbn1-priv 10.11 odbn1-pub 10. all hosts in our network will have exactly same /etc/hosts file.10.12 odbn2-priv # Database nodes – Public Network – (eth1) 10.1 localhost.200 odbn1-vip 10.0. Roll up your sleeves.20 openfiler Now we do following steps:  Enable iSCSI services from [Services] / [Manage Services] page.10.12 odbn2-pub # Database nodes – Public Virtual IP (VIP) addresses – (eth1:1) 10.201 odbn2-vip # Openfiler – private and public interfaces 10.20 openfiler-priv 10.1.

10.20): . and eth1 on 10. Two of them have static IP and are active already.  Network Access Configuration Navigate to [System] / [Network Setup]. one needs a setup (correction – eth0 and eth1 show reversed. notice our three network interfaces. eth0 should be on 10.

Click on eth2 interface and select DHCP protocol: Confirm default MTU on next screen: The result will look like this (actual IP not shown for eth2): .

Now click on [ Clock Setup ] as shown below: .Now enable two hosts (two RAC nodes) to communicate to Openfiler as below (notice the slightly unusual netmask).

there was not much success either.At this moment you probably ask “why don’t we use the host PC’s clock or install Guest Additions to keep guest clock synchronized with the host PC?”. I have not been successful so far with Guest Additions so far. It seems that UNIX or Linux world does not understand Windows SNTP in Windows XP.10. the clock has drifted wildly and some other problems were introduced (such as an occasional keyboard mapping mess up).pool.1 to get time from domain controller (somehow Windows XP will relay this to DC). it is a legitimate question and here is the or similar. Put a name or an IP of the NTP server you are going to use as a time source. As for synchronizing with host PC’s Windows Time service (w32time). So here is a sort of compromise that may work for most of the cases.1. If you do not have access to internet but instead have a Windows Domain Server (so you are in a corporate environment) you may be lucky and our Linux guest will be able to get time from the domain controller. Use internet NTP server if you have access to internet.ntp. Well. If you are setting up this project at home chances are you have access to internet and so you can use of the “pool” servers. . such as 0. at least what I can come up with. Use 10.

1.10. View /var/log/messages file.10.conf file on openfiler2 and add two more lines to it: restrict 10. In order to allow RAC nodes to query this NTP server we have to edit /etc/ntp.1. it may have these lines added after you click on “Setup synchronization”: By this time openfiler2 will have its NTP receiving time but it will not distribute it to others.Open an SSH window and connect to openfiler2 as root.11 nomodify notrap restrict 10.12 nomodify notrap After that you will need to restart the NTP service: # service ntpd restart .

scroll to “Create a partition in /dev/hdb” and click on “Create” thus accepting defaults: . that RAC nodes will be able to sync their clocks from openfiler2 even if openfiler2 itself won’t be able to get precise time from other sources (as long as it permits to query itself). Use this source of information on NTP servers and clients to fix your problem. Most probably you will see a picture like this: Navigate to [Block Devices] section (see below): Click on the link for “hdb”.If you have trouble with this NTP setup you should examine /var/log/messages and /etc/ntp. so this resolves a potential problem with date/time and Oracle installer later on. Notice.conf files.  Physical Storage Navigate to [Volumes].

The result of this action presented below: Now navigate to [Volumes] / [Volume Groups] and create a new volume group as below: .

092 iSCSI . numbers are slightly changed though).092 iSCSI racdb-asm2 racdb – ASM Volume 2 8.092 iSCSI racdb-asm3 racdb – ASM Volume 3 8.092 iSCSI racdb-asm4 racdb – ASM Volume 4 8. iSCSI / Logical Volumes Volume Name Volume Description Required Space (MB) Filesystem Type racdb-crs racdb – Oracle Clusterware 2.048 iSCSI racdb-asm1 racdb – ASM Volume 1 8.Not tired yet? Navigate to [Volumes] / [Add Volume] and scroll down to “Create a volume” in “rac1″ and use the table below (borrowed from original article mentioned in the beginning.

Here is the picture of first volume being created (after this one you will have to click on [Add
Volume] again):

In the end you will get a setup like this:

I am getting tired a bit but hold on a little longer, we are getting there. It’s actually getting quite
exciting as we are going to create iSCSI targets now.
Navigate to [Volumes] / [iSCSI Targets] and proceed to add five iSCSI targets with following
names (system will suggest default names that you will change, it is only a matter of aesthetics).
The names will be as below:

We are not going to change any settings for our targets (leave all defaults).
Select first target and click “Change” (this is going to affect sub-tabs on this page). Click on
“LUN Mapping” sub-tab. You will see this (look at it for a while and it will make sense):

If you did not guess yet, you will have to click on “Map” button thus linking LUN
“/dev/rac1/racdb-crs” with iSCSI target “”.
After that proceed to “Network ACL” sub-tab and grant our RAC nodes the access to this target

I am sending special thanks to Jeffrey Hunter. get the DVD image. . Virtual Oracle RAC. whose article was guiding me through this setup so far. The media comes in five ISO CD images or one DVD image. the index to the whole project is here.1. Now go to first sub-tab and select next target. At this point we are done with Openfiler. This part of the project provides instructions on installing Oracle Enterprise Linux 5 Update 3 (OEL5 U3) in Virtual Box 3. Repeat this process until you linked them all. Task 4: Installing Oracle Enterprise Linux 5 in Sun Virtual Box Rate This Link back: This guide is a part of the Virtual Oracle RAC project. I am installing a 32-bit variant here. Why this Linux is treated as Red Hat was explained in the project index.2 (r56127) Download "Oracle Enterprise Linux 5 Update 3" from Oracle’s eDelivery site. then link it with appropriate LUN and grant below: The last sub-tab – “CHAP” is not relevant.

Create the first node (new guest in Virtual Box): Give this machine 512MB of memory for now. when we start installing Oracle we may need to increase the allocation (or maybe add more base memory to the PC): . Later.

Choose to create a new hard disk (which will result in a file that holds an image of this machine): .

You get to the next window where where you just click Next: I recommend going with dynamically expanding storage option: .

Choose an appropriate location for the file and give it a max size of 40GB (it will be initially created quite small): .

Verify the settings for the storage and click Finish (there will be another screen with confirmation): .

Once empty and inactive machine is created we need to go over its settings and do some changes. then comes Hard Disk and floppy is not selected: . Update boot order so CD/DVD is first.

You will need to create three network adapters. first one for private interface: .

Second adapter is for public interface: .

And the third adapter is for connecting to Internet (since we do not want to bridge either private or public interfaces to host’s Internet): .

Now we are ready to proceed with installing the Linux onto our guest machine. When these adapters are discovered by hardware installation wizard you need to know which is which. Guests can only see media (regular files. directories) that are added to Media Manager: . it is slower but saves you a lot of time later on).Notice MAC addresses of the cards (they will be different in your case but you may assign them manually and make them just as in this guide). image files. Add installation media image in Virtual Media Manager (do yourself a favour. drives. download a single DVD image.

like in the picture below): .Mount (it will become "attached" in Media Manager) the DVD image in guest (your guest has to have an IDE device that emulates CD/DVD drive.

Start installation by powering up the guest (I had to pause the guest to take a snapshot): .

just one more layer of abstraction. Volume Groups is a new thing here but do not be afraid. OEL5 differs from OEL4 in its management of partitioning. it is a relatively simple setup. like having a software RAID built on top of your (single) hard drive: .If boot order is correct you will be presented with a few dialogues where you mostly accept defaults until you arrive at storage decisions.

Now configuring network (watch out for MAC addresses.11 instead of what is shown in the picture below): . So. the order of adapters here will not necessarily be the same as you see them in guest settings).2. first adapter will get a static address on private network (please use 10.10.

11 instead of what is shown in the picture below): .10.Second adapter will also get a static IP.1. it is on public network (please use 10.

make up yours: . Give this machine a hostname.Third adapter will get DHCP configuration. make all three cards active on boot. Domain name is mandatory for some services (they will fail on startup if there is no domain) but it does not have to be a real one. Finally.

When you finally get to the software configuration section choose manual mode and make sure the following is selected:  Desktop Environments  o GNOME Desktop Environment  Applications .Next screens finalize the initial setup and let you choose software you want to be installed.

When done with all that the installation will proceed and the machine will reboot into your new OS. Hopefully. This part of the project provides instructions on preparing Oracle software installation in Oracle Enterprise Linux 5 (which itself is in Virtual Box). when you are asked about SELinux – disable that one too. Virtual Oracle RAC. You may also create non-root user now or delay that till later. the index to the whole project is here. . Watch for any error messages during reboot.  o Editors o Graphical Internet o Text-based Internet  Development  o Development Libraries o Development Tools o Legacy Software Development  Servers  o Server Configuration Tools  Base System  o Administration Tools o Base o Java o Legacy Software Support o System Tools o X Window System Few more things: when you get to the Firewall dialog – disable the Firewall. Task 5: Patching Oracle Enterprise Linux 5 for Oracle Installation 2 Votes Link back: This guide is a part of the Virtual Oracle RAC project.

core.d/init.rmem_default=1048576 net. it will be placed in all of our Linux machines. Notice.core.core.In previous guides we have installed OEL5 U3 as a guest machine in Virtual Box.d/init. add following lines to the config file: net. that node name(s) are not present in the loopback address.rmem_default=1048576 sysctl -w net.d/iptables stop Now is the time to install Linux packages needed by Oracle.wmem_max=262144 Same changes have to be made permanently in /etc/sysctl. Network verification and configuration: Please refer to the /etc/hosts file that was introduced in Openfiler installation and setup.core. thus saving a lot of time and effort. This is our first node of the RAC.conf file.core.rmem_max=1048576 net.rmem_max=1048576 sysctl -w net.core.core.core. Once we finished with all software installations on that node we will simply clone it.d/iptables status Use the command below if the firewall is active: /etc/rc. Edit /etc/hosts on “odbn1″ by copy and paste it or use ftp. . they should not be there for the RAC to work properly.wmem_default=262144 sysctl -w net.wmem_max=262144 Verify that firewall is indeed stopped by running this command: /etc/rc. Some adjustments must be done in the kernel for the network (those commands do not make the changes permanent): sysctl -w net. That’s right.wmem_default=262144 net.

Execute these commands: . It does not matter which interface you use now for the session. select “more CD/DVD Images”): Select the proper image (if it is not displayed you will have to launch Media Manager first and register the image file): Use SSH terminal (PuTTY for instance) to log into “odbn1″ with root credentials.Attach the installation DVD to the “odbn1″ machine (right click on CD image in the running guest window. public or private.

* rpm -Uvh libstdc++-4.* rpm -Uvh unixODBC-2.2.* rpm -Uvh glibc-2. Now we will install iSCSI package (supplied but not installed by default): rpm -Uvh iscsi-initiator-utils-6.el5.* rpm -Uvh openmotif-2.* rpm -Uvh libaio-0.* rpm -Uvh unixODBC-devel-2.* rpm -Uvh glibc-headers-2.* rpm -Uvh glibc-common-2.i386.* rpm -Uvh libgcc-4.* rpm -Uvh glibc-devel-2.* We can see that some of the packages were there and some were installed.18.* rpm -Uvh libstdc++-devel-4. Good.* rpm -Uvh gcc-c++-4. rpm -Uvh binutils-2.* rpm -Uvh elfutils-libelf-devel-0. if some of them are already here the particular command will be skipped.* rpm -Uvh elfutils-libelf-0.* rpm -Uvh compat-libstdc++-296* rpm -Uvh compat-libstdc++-33* rpm -Uvh libaio-devel-0.mkdir -p /media/cdrom mount -r /dev/cdrom /media/cdrom cd /media/cdrom/Server Now let’s run installation for all packages that needed by Oracle.* rpm -Uvh sysstat-7.rpm .* rpm -Uvh gcc-4.* rpm -Uvh make-3.0.868-0.* rpm -Uvh libXp-1.

Virtual Oracle RAC. Start the “iscsid” serivce: # service iscsid start Run the following commands to make sure the services will run next time after reboot: # chkconfig iscsid on # chkconfig iscsi on Now. Configuring iSCSI services. Run this command (shows available iSCSI targets): # iscsiadm -m discovery -t sendtargets -p openfiler-priv . we check if our iSCSI service can communicate with Openfiler. the index to the whole project is here. 5 Votes Link back: This guide is a part of the Virtual Oracle RAC project. This part of the project provides instructions on configuring iSCSI initiators in Linux. Task 6: Configuring iSCSI targets in Linux and partitioning the volumes.

so you can run discovery again. Go back and set it all to “Allow” for all targets. There was something called “Network ACL” setup. when the iSCSI initiator discovers the targets it configures the services to start up automatically on reboot and log into the targets. Remember now? It is set to “Deny” by default for each target. that may be likened to a firewall in the openfiler. The change will take effect immediately.Is your output any different from what is shown above? Got less targets or none at all? And you say that your openfiler is up and running and you followed all instructions to the letter. let’s go back and recall how we configured our iSCSI targets. Well. This is what you should see during restart if iSCSI setup was done properly: If login to the targets did not happen on reboot you will need to execute the commands below: . We can test this now by rebooting our linux machine. As it turns out.

The mapping of our iSCSI targets to local SCSI device names is random and may change after reboot.# iscsiadm -m node -T iqn. which is a Dynamic Device Management -p 10. do not confuse this “node” with RAC nodes) The “–op” option means update configuration property specified by “-n” option.openfiler:racdb. it is “node. # cd /etc/udev/rules.1.2006-01.d/ -p .20 –op update -n node.startup” in this case.rules” with the following content: # /etc/udev/ -p 10. I am showing only first of them.1. The command below (no operation specified) can be used to query configuration of a target: # iscsiadm -m node -T iqn. BUS==”scsi”.rules KERNEL==”sd*”. Linux “talks” to iSCSI targets using local device names.20 Making device names persistent.1.10.d/ Create a file called “ Do these two commands for each of the targets appropriately. This is done using “udev”.2006-01. The mapping of targets to the local devices is illustrated here: Since we want to have a permanent and consistent mapping across all RAC It is a problem that needs fixing.20 -l # iscsiadm -m node -T iqn.2006-01. PROGRAM=”/etc/udev/scripts/iscsidev.openfiler:racdb.startup -v automatic The “-l” option means – log into the target (“a node”.10.openfiler:racdb. we are going to create persistent local SCSI device names.

sh Let’s restart the iSCSI initiator service: # service iscsi stop # service iscsi start Here is the outcome of all that (the image below may be too wide but I did not want the lines to wrap): .SYMLINK+=”iscsi/%c/part%n” Navigate to another directory: # cd /etc/udev/scripts We create here a new shell script called “iscsidev.%b”.sh” with the following content: #!/bin/sh # FILE: /etc/udev/scripts/ BUS=${1} HOST=${BUS%%:*} [ -e /sys/class/iscsi_host ] || exit 1 file=”/sys/class/iscsi_host/host${HOST}/device/session*/iscsi_session*/targetname” target_name=$(cat ${file}) # This is not an open-scsi drive if [ -z "${target_name}" ]. then exit 1 fi echo “${target_name##*.}” Make the new script executable: # chmod 755 /etc/udev/scripts/iscsidev.

Well. it did not exist before) and compare them to mapping in /dev/disk/by-path: . how do we know if what we’ve done actually worked? We look at the names in /dev/iscsi (this directory has just been created after we ran these commands.

asm2 /dev/iscsi/asm2/part . We can reboot the odbn1 machine and see that iscsi devices are still there and properly mapped.2006-01.openfiler:racdb.2006-01.openfiler:racdb.asm1 /dev/iscsi/asm1/part iqn.And … Take for instance “/dev/iscsi/asm1/part”. it corresponds to the “asm1″ target (through /dev/sda). Mapping of iSCSI Target Name to Local Device Name iSCSI Target Name Local Device Name iqn. Now we have persistent local names for our

Notice: some of the material for this article was taken from Oracle author’s Before we start creating partitions it makes sense to showdown our virtual machines and take a snapshot of them. Creating partitions on iSCSI /dev/iscsi/crs/part Next.openfiler:racdb.asm4 /dev/iscsi/asm4/part iqn. I recommend reading that article if you need more detailed information. This provides an option of reverting to a known state of machine if something goes wrong. The following table lists the five iSCSI volumes and what file systems they will support: Oracle Shared Drive Configuration File System iSCSI Target (short) Mount Size ASM Diskgroup Name File Types Type Name Point Oracle Cluster Registry (OCR) File – OCFS2 crs 2GB /u02 (~250 MB) Voting Disk – (~20MB) ASM asm1 8GB ORCL:VOL1 +RACDB_DATA1 Oracle Database Files ASM asm2 8GB ORCL:VOL2 +RACDB_DATA1 Oracle Database Files ASM asm3 8GB ORCL:VOL3 +FLASH_RECOVERY_AREA Oracle Flash Recovery Area ASM asm4 8GB ORCL:VOL4 +FLASH_RECOVERY_AREA Oracle Flash Recovery Area Total 36GB .com. Assuming that snapshot is taken and our machines are back online.iqn.asm3 /dev/iscsi/asm3/part we will have to create partitions in our SCSI volumes.

number 1. Red arrows mark your input. Remember to always create a primary partition. # fdisk /dev/iscsi/asm1/part Repeat the command sequence for volumes asm2 through asm4 and then for crs.In the picture below shown the fdisk dialog that creates a primary partition of a maximum available size. of max size): . where you either type values or accept defaults. which is shown below.

So.Verify new partitions Keep in mind that the mapping of iSCSI target names and local SCSI device names will be different on each of our RAC nodes (it may even change on each particular node after a reboot). This does not present a problem as we are using local device names presented to us by “udev”. if you have not restarted your node after partitioning. run following command as root: # partprobe And now we will query the partitions with fdisk command: # fdisk -l Here is the results: .


We will try to do as much work on a single node of the RAC before we clone it. Following commands are taking care of just that (run them as root). the clone will have all these settings and configurations already done. Virtual Oracle RAC. After that some changes will have to be done on both nodes (or more. since the storage is shared between all nodes.This is all for the volumes and the partitions at this stage of our project. Some of them we are not going to use and so they can be disabled. Optional: Freeing up some system resources This Linux distribution comes with many services activated by default. When we clone our Linux guest (a node) at a later time. if you are going for more than two nodes). the index to the whole project is here. the "#" in the beginning of the command line denotes that you are logged in as user "root". the "#" is not a part of the command. If you are not familiar with UNIX and Linux. Task 7: Preparing Linux guest for Oracle installation – part 1 5 Votes Link back: This guide is a part of the Virtual Oracle RAC project. If not specified otherwise. Notice that partitioning is only done once. the commands in this chapter are to be executed in SSH session as a user "root". Before we run installation process for Oracle software there is some preparation needed. # chkconfig sendmail off # chkconfig bluetooth off # chkconfig ip6tables off # chkconfig yum-updatesd off # chkconfig cups off Use this command to verify the effect of the deactivation: # chkconfig –list . The change will take place after next reboot.

SSH tools to communicate with our Linux systems we do not need to run our guests at level 5. This part is easy for us. If you ever need to get to the X Windows Desktop again all you have to do is to issue "init 5" command as root.e. if you think of this. while by default the system will proceed to level 5 (where it starts GUI desktop – X Windows).Another important change concerns default run level of the system. any other shared storage solution) all nodes will authenticate themselves to the OCSF with their user IDs. When using OCFS (and. changes will take effect after reboot. . no shutdown or reboot is necessary. As we are going to use "telnet". When you are done you can issue "init 3" command to bring the system back to level 3.lst Make changes in the menu. # cd /boot/grub # vi menu. Create Groups The following O/S groups will be created: Oracle Oracle Group UNIX Group Description Privilege Name name Oracle Inventory and Software . This is achieved by using Oracle Cluster File System (OCFS2 in our case) . In order to provide consistent access to files these IDs must match on all nodes. i. Special considerations for user "oracle" and UNIX groups RAC technology requires that all nodes of the cluster have a shared access to the Oracle Clusterware software. oinstall Owner Database Administrator SYSDBA OSDBA dba Database Operator SYSOPER OSOPER oper .lst as shown in this picture: Again. because we create our IDs only once and all our other nodes will be produced by cloning (that does not alter IDs). Here is how to change the default run level. all users and groups that we create must have exactly same numerical IDs on different nodes. Under normal circumstances runlevel 3 is sufficient. In other words.

# groupadd -g 501 oinstall # groupadd -g 502 dba #groupadd -g 503 oper Create user "oracle" #useradd -m -u 501 -g oinstall -G dba. then enter the following command to create it: # useradd nobody Create the Oracle Base Directory Let’s take a look at our internal disk on odbn1 machine. enter the following command: # id nobody 2. If the user nobody does not exist (English is an interesting language if you think of that). To determine if the user exists.oper -d /home/oracle -s /bin/bash -c "Oracle Software Owner" oracle Verify the created user: # id oracle Set the password for the oracle account: # passwd oracle Verify That the User "nobody" exists (If you have installed the suggested Linux version you may skip this check) 1. there are two ways to produce the report (not all UNIX flavors support -H option): .

The next step is to create a new directory that will be used to store the Oracle Database software.7GB is occupied by Linux itself and its swap. # mkdir -p /u01/app/oracle The "-p" option tell the mkdir command to create the whole chain of directories at once. we specify the correct owner and group (why? because we work as "root" user and newly created objects will have root’s properties): # chown oracle:oinstall /u01/app/oracle And now permissions: # chmod 775 /u01/app/oracle Create the Oracle Clusterware Home Directory This location is later referred to by $ORA_CRS_HOME environment variable. After the directory is created.So we have 33GB of space to work with. 2. This location is later referred to by $ORACLE_BASE environment variable. # mkdir -p /u01/app/crs # chown oracle:oinstall /u01/app/crs # chmod 775 /u01/app/crs Create Mount Point for OCFS2 / Clusterware # mkdir /u02 # chown oracle:oinstall /u02 # chmod 775 /u02 Create Login Script for oracle User Account Login as user "oracle" and create the following .bash_profile: .

Note: After we clone our first node into other nodes we will have to change a number of things
in each of them. One of such changes is to make sure that each RAC node has a unique Oracle
SID (defined in the user’s profile).

# su – oracle

Use some editor to create (or rather edit, since in OEL5 it is already there) the following profile:

.bash_profile for "oracle" User Account
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc

export JAVA_HOME=/usr/local/java

# User specific environment and startup programs
export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/10.2.0/db_1
export ORA_CRS_HOME=/u01/app/crs
export ORACLE_PATH=$ORACLE_BASE/common/oracle/sql:.:$ORACLE_HOME/rdbms/admin
export CV_JDKHOME=/usr/local/java

# Each RAC node must have a unique ORACLE_SID. (i.e. racdb1, racdb2,...)
export ORACLE_SID=racdb1

export PATH=.:${JAVA_HOME}/bin:${PATH}:$HOME/bin:$ORACLE_HOME/bin
export PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin
export PATH=${PATH}:$ORACLE_BASE/common/oracle/bin
export ORACLE_TERM=xterm
export TNS_ADMIN=$ORACLE_HOME/network/admin
export ORA_NLS10=$ORACLE_HOME/nls/data
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib
export THREADS_FLAG=native
export TEMP=/tmp
export TMPDIR=/tmp

Swap Space Considerations

Exit your "oracle" user session and now you are "root" again.

We will need at least 1GB of total memory (real + swap) to install Oracle Database 10g Release

Here is how to check how much we’ve got (another way is to run the "top" command):

# cat /proc/meminfo | egrep ‘MemTotal|SwapTotal’

Configure kernel parameters and shell limits for Oracle

Changes to /etc/sysctl.conf file.

You can make changes to your kernel configuration as follows (copy and paste text from "cat" to
"EOF", do not include "#"):

# cat >> /etc/sysctl.conf <<EOF

kernel.shmmni = 4096

kernel.sem = 250 32000 100 128

fs.file-max = 65536

net.ipv4.ip_local_port_range = 1024 65000


Just in case I am listing the contents of /etc/sysctl.conf file after the changes:

kernel.shmmax = 4294967295

kernel.shmall = 268435456

kernel.shmmni = 4096

kernel.sem = 250 32000 100 128

fs.file-max = 65536

net.ipv4.ip_local_port_range = 1024 65000





Activate your changes with following command (or reboot the machine):

# sysctl -p

You can verify the new settings with this command:

#/sbin/sysctl -a

Setting Shell Limits for the oracle User

Run following command to change limits for user "oracle" (while you are still logged in as

# cat >> /etc/security/limits.conf <<EOF

oracle soft nproc 2047

oracle hard nproc 16384

oracle soft nofile 1024

oracle hard nofile 65536


# cat >> /etc/pam.d/login <<EOF

session required /lib/security/


Shell settings

C shell and its variants use /etc/csh.login profile (this one is loaded before local profile). Add
following lines to it:

# cat >> /etc/csh.login <<EOF

if ( \$USER == "oracle" ) then

limit maxproc 16384

limit descriptors 65536



conf To test the module and its new setting run following commands: # modprobe hangcheck-timer # tail -10 /var/log/messages . Korne and Bash) load /etc/profile.All other shells (such as Bourne. then ulimit -p 16384 ulimit -n 65536 else ulimit -u 16384 -n 65536 fi umask 022 fi EOF Configure the hangcheck-timer Kernel Module The module "hangcheck-timer. then if [ \$SHELL = "/bin/ksh" ].ko" Set hangcheck timer parameter to values recommended by Oracle: # echo "options hangcheck-timer hangcheck_tick=30 hangcheck_margin=180" >> /etc/modprobe.ko" is already installed in OEL5 and this can be verified by this command: # find /lib/modules -name "hangcheck-timer. so it has to be amended like this: # cat >> /etc/profile <<EOF if [ \$USER = "oracle" ].

I chose to use Openfiler machine as a source of time.127.0 address (which is optional). except for the default line with 127.20 mask 255. The easy way to fix this problem is to set up NTP client on our machine(s). Restart ntp daemon: # service ntpd stop # service ntpd start If succeeded.20 restrict 10.conf.1.255 nomodify notrap noquery No other NTP servers should be specified in this file. Use the "tail" command in one SSH window to verify that NTP setup is progressing properly (leave this window and come back to it later).1.conf ntp.10. the response will be "OK" and look like this: The output of the "tail" command in first window may look like this by now: . # tail -f /var/log/messages In another SSH window make following changes: # cd /etc Make a backup of NTP settings # cp ntp.255.255.10. rather than pointing to external NTP server.Set up NTP client for synchronized time on RAC nodes Oracle installer will have a trouble if RAC nodes have different date and time showing by their clocks.bak Edit the NTP settings file to include these lines (pointing to openfiler-priv network interface): server 10.1.

We can also use "ntpq" to verify the settings (or "ntpdc" utility with "peers" command): # ntpq -p If you look carefully at the IP addresses you will notice that "step time server" shows IP of the "router". all is working properly. It is clarified when you ran "ntpq" command. rather than that of the Openfiler. As long as "st" column (Stratum) shows value other than 16 and "jitter" is reasonably small. The result of this setup is that odbn1 has its clock synchronized with that of the Openfiler.core.core. Although we used Openfiler’s IP in the configuration file the command shows proper server name in the output.wmem_default=262144 net.wmem_max=262144 .rmem_max=1048576 net.rmem_default=1048576 net.core. Add following lines to the /etc/sysctl. RAC related changes to the system configuration files Have this changes done as root.conf file net.core.

18-128.i686.shmall = 268435456 kernel.i686.ip_local_port_range=1024 65000 Have kernel load the new parameters: # sysctl -p Install the OCFS2 The OCFS2 software packages is supplied on CD #3 of the OEL5 U3 (but not installed by default).rpm . 32000 100 128 fs.0 This library is also supplied on CD #3.el5-1.7-1.el5.rpm # rpm -Uvh ocfs2console-1.file-max=65536 net.9-1. Install and verify it by issuing the following commands: # rpm -Uvh oracleasm-support-2.rpm Verify the installation of packages: # rpm -qa | grep ocfs2 | sort Install ASMLib 2.2.2.ipv4.i386.2-1.6.i386.el5.shmmax = 4294967295 kernel.5-1.el5.shmmni = 4096 kernel.6.el5-2.If you are using Linux other than OEL5 U3 you may also need to add these lines: kernel.el5.7-1.rpm # rpm -Uvh oracleasm-2.rpm # rpm -Uvh ocfs2-2. Have disk #3 mounted (mount the image through the Virtual Box as you did before) and issue following commands: # mount -r /dev/cdrom /media/cdrom # cd /media/cdrom/Server # rpm -Uvh ocfs2-tools-1.el5.

i386. Virtual Oracle RAC.0.el5. Open the command line window.0. the index to the whole project is here.4-1.4-1. You may keep Virtual Box GUI open.i386.C:\Program Files\Sun\VirtualBox\ cd /d G:\VBmachine mkdir odbn2 . Task 8: Cloning a Linux guest. Get the oracleasmlib- 2.rpm from Oracle. Have this rpm copied onto the guest by means of sftp into a new directory /root/rpms Install the rpm by running these commands: # cd /root/rpms # rpm -Uvh oracleasmlib-2. use “oracle” instead of “sun” if you have latest version of product): set PATH=%PATH%.# rpm -qa | grep oracleasm | sort # cd / # eject Install ASMLib Userspace Library This one is not shipped along with CDs and will have to be downloaded. Type in the following commands (with a correction to your locations. Shut down your virtual machines. 3 Votes Link back: This guide is a part of the Virtual Oracle RAC project.el5.rpm Looks like we are done with most of the preparations that we can do on a single node before cloning it.

only absolute path works (at least that’s the case for release 3. if you are tempted to use the setvdiuuid tool I would advise against that as it may save you cloning time (5 min) but prove to give some trouble later. fill out the details like this: The memory is 512MB for now: .vdi G:\VBmachine\odbn2\odbn2. A few important moments here.2 of Virtual Box. The VBoxManage utility will report errors if source machine is running.VBoxManage clonehd G:\VBmachine\odbn1\odbn1. it will also report an error if you use relative path to the vdi file.vdi This is how it looks in the process. Also.1. Bring up the Virtual Box GUI and click on “New” machine. Next step is to add (or rather register) our newly cloned machine to the Virtual Box.

but it is not registered yet so it is not on the list.Now. . the disk is “existing”.

Click on the folder button and Media Manager interface appears. Click on “Add”. add it and then “select” it: .vdi). navigate to our new file (odbn2.

Now we are back to the new machine dialog with cloned file available to be used. Click “Next”. .

Do not start this machine or you will ruin it.And “Finish”. .

any plain editor) open odbn1. Hardware setup is separate and contained in an xml file in the same folder as the vdi file. In such case ou will need to delete the SATA controller. We need to edit the xml file to give our new machine proper setup.xml file but do not start the machine yet.What will happen if we run this new machine right now? The Virtual Box knows nothing about this machine’s hardware at the moment. Copy and paste (overwrite) the whole network section from first to the second. Close the Virtual Box GUI (if it was open) and start it up again. In a text editor (Textpad will do. Do not start odbn1 as well until instructed to do so. All these settings are not cloned or copied over when we clone vdi image. attach an IDE controller and then attach the image of the disk (the vdi file) to it. It may be quite different with VMWare but here the image is not self contained. This configuration section is contained between tags <Network> and </Network>. Navigate to Storage section and remove Floppy controller as we have no need for it.xml and odbn2. then click on “Settings” for “odbn2″ machine. Notice: in newer versions of Virtual Box the default type of disk controller is SATA rather than IDE. Only now our new settings will be read and accepted.xml files. If we run the machine now with all the default settings from that file the Linux will run new hardware setup process and overwrite our careful network setup. Save the modified odbn2. .

otherwise they would not start. . The iSCSI seems to be working as well. Now we can start the odbn2 machine (it is just a precise clone of odbn1 for now). Notice that all network interfaces are started properly.Then go to System section and uncheck Floppy from the boot order: Start the openfiler machine now. When we start odbn2 we would like to see if it can communicate to openfiler. that means MAC addresses in the hardware (network) section matched those in the Linux itself.

Do the same for the next adapter: # vi ifcfg-eth1 Now change host name: # cd /etc/sysconfig # vi network .Now let’s convince odbn1 clone that it is odbn2 (plastic surgery of sorts). Bring up SSH terminal as if you were connecting to odbn1. # cd /etc/sysconfig/network-scripts/ # vi ifcfg-eth0 Edit the file replacing 11 with 12 in the IP address. log in as root.

. Use “ifconfig” command to verify that IP addresses indeed have changed: We also run “hostname” command to see if host name is correct: Let’s see if openfiler is visible to the new server and can properly map the iSCSI targets. On the startup the cloned machine will notice its new name: Bring up an SSH window and connect to the IP address of odbn2.Change odbn1 to odbn2 in the HOSTNAME: Save the file and reboot the machine: # reboot Watch the machine messages in the GUI.

Include also an xml file from C:\Documents and Settings\%USER%\. Virtual Oracle RAC.e.While we are at it.xml That file is essential for the backup and restore of Virtual Box setup. Task 9: Preparing Linux guest for Oracle installation – part 2 . # su – oracle # vi .VirtualBox\VirtualBox.bash_profile I would now take a backup of the whole farm (i. let’s also change the ORACLE_SID in the oracle’s profile to “racdb2″. three virtual machines) by shutting them down and copying all vdi and xml files someplace else.

the arrows indicating your input: . the index to the whole project is here.ssh/id_rsa. Files are copied by Oracle Installer (OI) to other nodes using ssh and scp commands and OI expects "user equivalence" to be setup on all nodes. Log on as the oracle UNIX user account (on both nodes): # su – oracle Create the .2 Votes Link back: This guide is a part of the Virtual Oracle RAC project. This means UNIX id used for installation ("oracle" user in our case) is trusted on all nodes and ssh/scp commands won’t ask for password. you will need it later). We are going to set up this equivalency. it may or may not be same on all nodes. When prompted: accept the default location for the key private key in ~/.ssh $ chmod 700 ~/. write it down.ssh directory in the oracle user’s home directory (on both nodes): $ rm -R ~/. Do it on both nodes.ssh Generate an RSA key pair (public and private key) for the SSH protocol.ssh/id_rsa The results so far will look like this (first node). Configure SSH on RAC Nodes Oracle Clusterware and the Oracle Database software are installed from one node in a RAC cluster. enter and confirm a pass phrase (make up something.ssh $ mkdir ~/. $ /usr/bin/ssh-keygen -t rsa The command above creates: public key in ~/.

pub >> ~/. Keep the RSA passphrase and oracle’s password handy.ssh/id_rsa. this is done in order to produce additional SSH configuration file "known_hosts" and properly add contents of "id_rsa. When all keys are collected this file will be copied to other" to "authorized_keys". The following commands are run from one node (odnb1): $ ssh odbn1 cat ~/.ssh/authorized_keys Notice: yes. red arrows denote your input: . you are connecting with SSH from odbn1 to odbn1. This "authorized_keys" file is a collection of RSA public keys from all nodes in cluster.and second node: Create "authorized_keys" file on first node only.

ssh/authorized_keys What has been done just now? We have connected to odbn2 and appended the contents of its "id_rsa.ssh/id_rsa. This looks like that so far: Examine your "authorized_keys" file on odbn1.ssh/authorized_keys Change permissions on the "authorized_keys" file on both nodes: $ chmod 600 ~/.pub" file to the "authorized_keys" file on >> ~/.ssh/authorized_keys Test connection from odbn1 (first to odbn1 itself. then to odbn2): $ ssh odbn1 uname -a $ ssh odbn2 uname -a . you will see it includes keys from both nodes.ssh/authorized_keys odbn2:. Now copy this file to the other node (you are still being asked for oracle’s password): $ scp ~/.Still from the same host odbn1 issue this command: $ ssh odbn2 cat ~/.

that is): Test connection on private and public interfaces as well (this is needed later by Oracle Installer): $ ssh odbn1-priv $ ssh odbn1-pub .Notice: we are not prompted for the oracle’s password. Now we are going to get rid of that prompt either. Enter following commands on odbn1: $ exec /usr/bin/ssh-agent $SHELL $ /usr/bin/ssh-add The ssh-agent is a process that will supply the passphrase to new SSH processes in this session. only for the passphrase of the key file. When we restart the nodes later (or leave this session) the two commands (ssh-agent and ssh- add) are to be re-run. Try running same ssh commands as before and you will see no prompts whatsoever (when running from from odbn1.

Taking care of memory As mentioned before. Start all the machines again. . at this point we have configured the SSH to satisfy the Oracle Installer (we hope).Well. Virtual Oracle RAC. Take this chance to run a backup of the whole project while guests are shut down. the index to the whole project is here. now comes the time to increase the virtual machines memory to 1024MB. Task 10: Oracle installation in RAC nodes – part 1 1 Votes Link back: This guide is a part of the Virtual Oracle RAC project. then change Base Memory settings on both nodes.

Xming or other. I will be using Cygwin/X here. run in CMD window (or via an icon): . This can be any server of your choosing. Make sure your X Server includes xhost utility.Getting an X Server running on the PC There are few small things to take care of before we launch Oracle Installer. such as Cygwin/X. we need an X Server on our PC. sometimes it is not installed by default. This is how to locate the xhost utility during instalation: Have your X Server installed and started on your PC. Namely. If using Cygwin.

0 and bring up an xterm window. In this xterm window give an access control command: $ xhost +10. if you are using PuTTY make sure the X11 forwarding is not selected for the RAC node sessions: .12 This xhost command has enabled the obdn1 host to use your X Server to display x clients. The access control matters of X Server are not covered clearly in documentation and articles would suggest using X11 forwarding in SSH.exe -l -c /usr/bin/startxwin.10.11 $ xhost +10.1.10. I think I have just saved you a few days of frustration.exe This will usually start X server on display 0.C:\bin\cygwin2\bin\run. If xterm window did not appear have it started manually. Therefore you should check that the SSH sessions you run from your PC to the RAC nodes are not using the X11 forwarding. The X11 forwarding in SSH will indeed take care of all the neccessary settings automaticatlly.1.exe /usr/bin/bash. however this setup won’t be usable by Oracle Installer. So.

from that same SSH session you have enabled SSH no- password connections (go back to refresh your memory). Ff you have restarted servers or logged in again in SSH you will need to repeat steps with ssh-agent and ssh-add as described before (on odbn1 only).10.1:0 Test that your session can run some X windows application and show it on your PC’s X Server: $ xclock & This should spawn an xclock application and show it on your PC via X Server. Configure OCFS2 . Having endured so far we are now going to do more exciting things.10. Notice: The IP address we supplied in DISPLAY variable is the virtual "router" on our host PC.1. since X Server expect connection from 10. If this did not work you will have to troubleshoot your setup and proceed with Oracle installation only when resolved.11.As user "oracle".1. we should "talk" to it on the same network. $ export DISPLAY=10. from the odbn1 node.

e. that although we will be using the private network IP addresses we will still use hostnames as they appear on public network. There would not be such a confusion with just one network (either private or public). OCFS2 will be configured to use private network i.10.10. As user "root".1.11 odbn1-priv 10. go back if you missed them. hostnames as in etc/hosts: 10. If you already cloned the node then do this step on both nodes.1:0 Now.1.12 odbn2-priv Notice. This will start the OCFS2 Cluster Stack and bring up next dialog ("Node Configuration") .Notice: OCFS2 package (RPM) installation instructions are in Task 7 . I know. on the odbn1 node. bring up OCFS2 configuration tool to edit /etc/ocfs2/cluster.conf file: # ocfs2console & Select [ Cluster ] – [ Configure Nodes ]. start a new SSH session and set the DISPLAY varaible: # export DISPLAY=10.1.10. it is confusing.

Add two RAC nodes as below: .There will be a pop-up message: Dismiss it and proceed to "Node Configuration".

conf file will look like this after configuration is done (do not just copy this file over to other node): . The /etc/ocfs2/cluster.When you click "Apply" the screen changes to node status and node numbers: Exit the console and repeat the same process on the other node of the cluster supplying same exact data in the same order.

O2CB Cluster Service O2CB is the OCFS2′s cluster stack of services that are listed below:  NM: Node Manager that keep track of all the nodes in the cluster.d/o2cb offline ocfs2 # /etc/init. its owners and status  CONFIGFS: User space driven configuration file system mounted at /config  DLMFS: User space interface to the kernel space DLM All of the listed services are included in the single service o2cb (/etc/init.conf  HB: Heart beat service that issues up/down notifications when nodes join or leave the cluster  TCP: Handles communication between the nodes  DLM: Distributed lock manager that keeps track of all locks.d/o2cb).d/o2cb unload # /etc/init. Now issue following commands to have O2CB start on boot up and have other needed attributes (on both nodes): # /etc/init.d/o2cb configure Below shown non-default responses to the configuration questions (marked by arrows): .

Repeat the same steps on the other node. Issue this command to review the service status so far: # /etc/init.d/o2cb status .

ocfs2 -b 4K -C 32K -N 4 -L oracrsfiles /dev/iscsi/crs/part1 The following output is produced: Now mount the newly formatted partition (that was labeled "oracrsfiles") under directory /u02: # mount -t ocfs2 -o datavolume. Verify that the file system is mounted correctly: # mount .Make sure that o2cb serivce will be started at proper run level: # chkconfig –list o2cb Levels 3 through 5 should be "on": Format and mount the OCFS2 Filesystem This task is to be done from one node only # mkfs.nointr -L "oracrsfiles" /u02 Repeat the mount command on the other node.

nointr 0 0 This is how it will look like: Adjust ownership and permissions on the new filesystem (from one node only): # chown oracle:oinstall /u02 # chmod 775 /u02 Create directories to contain Oracle Clusterware shared files (also from one node only): .Repeat verification on the other node. Your output should be similar to the one below (newly mounted file system is highlighted): Make these changes permanent by adding the following line to the /etc/fstab file on both RAC nodes: LABEL=oracrsfiles /u02 ocfs2 _netdev.datavolume.

# mkdir -p /u02/oradata/racdb # chown -R oracle:oinstall /u02/oradata # chmod -R 775 /u02/oradata Reboot both nodes.d/oracleasm configure The following picture shows your input marked by red arrows: . Use "mount" command to see if the new file system is still there and is mounted in the proper place. The configuration task is to be done on both nodes: # /etc/init. You can also use o2cb status check to make sure the serive is functioning properly (from both nodes): # /etc/init. go back if you missed this step. If you have already cloned the node then do this step on both nodes. Check if our setup can survive the reboot.0 Notice: ASMLib package (RPM) was added in Task 7 .d/o2cb status Configure ASMLib 2.

d/oracleasm createdisk VOL1 /dev/iscsi/asm1/part1 # /etc/init.Create ASM disks We are creating the ASM disks from one node only. Run following commands: # /etc/init.d/oracleasm createdisk VOL2 /dev/iscsi/asm2/part1 # /etc/init.d/oracleasm createdisk VOL3 /dev/iscsi/asm3/part1 # /etc/init.d/oracleasm listdisks .d/oracleasm createdisk VOL4 /dev/iscsi/asm4/part1 The commands above have created ASM disks in iSCSI targets that are mapped to local devices: The newly created disks can be listed by using this command: # /etc/init.

com/technetwork/database/10201linuxsoft- 097986.On the other node we only need to do a scandisk to recognize the new volumes and then list them: # /etc/init. just like in the picture below: Oracle Database 10g Release 2 (10. into some directory  unzip the archives and re-pack them into an ISO image.1.0.0) 10201_companion_linux32.1.0) 10201_database_linux32.d/oracleasm listdisks Prepare Oracle software media Download the following from http://www.d/oracleasm scandisks # /etc/init. You can either:  copy the archives (with sftp) onto odbn1 node as user "oracle". So. it is a one time install).0. have the archives unzipped into separate directories on your PC. then mount the image in VB guest I will be using the second option as not to inflate the file system of the virtual machine (after all.2.html: Software Title File Name Oracle Clusterware Release 2 ( Oracle Database 10g Companion CD Release 2 (10. then use a utility such as Infra Recorder to pack the software into a DVD ISO Now there is two ways to proceed.0) 10201_clusterware_linux32.

iso": .Name this image "rac32media.

1-1. Then mount it in the nodes like this: # mount -r /dev/cdrom /media/cdrom # cd /media/cdrom Install cvuqdisk package This package is needed later by Cluster Verification Utility (CVU). The SSH agent will have to be initiated (again) for this purpose. Issue the following commands (do it on both nodes): # cd /media/cdrom/10201_clusterware_linux32/clusterware/rpm # export CVUQDISK_GRP=oinstall # rpm -iv cvuqdisk-1.rpm Verify installation of the package: # ls -l /usr/sbin/cvuqdisk Install and run Cluster Verification Utility (CVU) This utility will be run from odbn1 node as user "oracle".Add newly created image to Virtual Media Manager and then mount it in both VB machines. then execute following commands: . Open another SSH session (we may want to use our "root" session later) and login as "oracle".0.

$ mkdir -p jdk14 $ unzip jrepack.odbn2 -verbose | tee stage -pre crsinst -n odbn1./runcluvfy.1 (has to do with non routable ranges of IP addresses).zip -d jdk14 $ export CV_HOME=/home/oracle/bin/cluvfy $ export CV_JDKHOME=/home/oracle/bin/cluvfy/jdk14 Run pre-requisites check report $ . as it is caused by a known bug documented in Metalink Note 338924. There is a messages in connectivity section reading "Could not find a suitable set of interfaces for VIPs". There are several errors like "Check: Package existence" that can be ignored. we can now check that all pre- installation tasks for Oracle Clusterware are completed by executing the following command as the "oracle" UNIX user account from linux1: $ mkdir -p /home/oracle/bin/cluvfy $ cd /home/oracle/bin/cluvfy $ cp /media/cdrom/10201_clusterware_linux32/clusterware/cluvfy/* .txt Review the CVU report (cvu_report_pre. .$ exec /usr/bin/ssh-agent $SHELL $ /usr/bin/ssh-add Install the CVU utility: Verifying Oracle Clusterware Requirements with CVU Once all prerequisites for running the CVU utility have been met. This one can be ignored too.txt).

Connectivity errors must be resolved.If you get any errors other than those in my report there might be a problem with your setup.txt ./ stage -post hwos -n odbn1.odbn2 -verbose | tee cvu_report_2. For reference and troublesooting see Metalink ID 338924.1 and Forum Run the Hardware and OS Setup check report $ .

it should say "Post-check for hardware and operating system setup was successful. proper report will look like this: When reports are ok and all serious problems have been resolved we can move on. The two shared Oracle Clusterware file groups are:  Oracle Cluster Registry (OCR) File 1 : /u02/oradata/racdb/OCRFile File 2 : /u02/oradata/racdb/OCRFile_mirror Size : (2 * 100MB) = 200M . Install Oracle Clusterware The two shared files (file groups) used by Oracle Clusterware will be stored on the Oracle Cluster File System (OFCS2) we created earlier." The utility checks connectivity again and then checks storage.Review the report.

click Next: . click "Next" as in the picture below: Have Name and Path filled as shown.  CRS Voting Disk File 1 : /u02/oradata/racdb/CSSFile File 2 : /u02/oradata/racdb/CSSFile_mirror1 File 3 : /u02/oradata/racdb/CSSFile_mirror2 Size : (3 * 20MB) = 60MB Issue these commands as user "oracle": $ cd ~ $ /media/cdrom/10201_clusterware_linux32/clusterware/runInstaller -ignoreSysPrereqs & The Oracle Installer will appear.

click Next: .In the next screen the installer runs various checks (the checks must be all right if our manual check reports were fine) and shows the final result.

click on Add and supply information for the other node as shown so it looks like this: .The Cluster Configuration screen appears with one node filled out already.

.Click "Next". The Network Interface configuration screen appears (it shows subnets and network adapter names). We need a public interface in this configuration. both interfaces are shown as "Private". Select "eth0" and click "Edit". As you can see.

Designate "eth0" as public interface. Click "Ok" .

Our interfaces and their purpose look like this now: .

Click "Next" and OCR location dialogue appears.

Supply following OCR locations in the next screen (leave Normal Redundancy selected):

Specify OCR Location: /u02/oradata/racdb/OCRFile
Specify OCR Mirror Location: /u02/oradata/racdb/OCRFile_mirror

Click Next:

Supply following locations of voting disks in the next screen (again, leave Normal Redundancy

Voting Disk Location: /u02/oradata/racdb/CSSFile
Additional Voting Disk 1 Location: /u02/oradata/racdb/CSSFile_mirror1
Additional Voting Disk 2 Location: /u02/oradata/racdb/CSSFile_mirror2

Click Next:

Summary screen appears. Click Install:

the software is being copied over to the other node in the cluster . as the install progresses.Notice how.

If this happens on odbn2 it is safe to reboot it and wait till installer continues (say 5 minutes after reboot).Notice: Sometimes one of the virtual machines may crash on memory related errors (running out of usable memory on the host PC). You will need to:  cancel the installer  restart machines if they are down  delete subdirectories under of the /u01/app/crs and /u01/app/oracle  shut down all VM’s  free up host PC’s memory or reboot it  restart VM’s  restart the Installer  good luck Upon completion of the remote copy operations (ssh and tar processes) the next screen shows up: . If this happens on odbn1 it is more complicated.

And pop up screen shows: .

sh script on odbn1 and wait until it script on odbn1: .sh script on both nodes. Important: run orainstRoot.Now pay attention closely. Output of the root. Important: run /u01/app/crs/root. one at a time. Important: Do not close this Oracle Installer dialog until instructed to do so. then proceed with the same script on the odbn2.

The /var/log/messages shows this on the first node at the completion if the script: .

Now proceed with the root. The output is here: .sh on odbn2 node (read a note further on).

here is the manual fix to this trouble.0) is Edit "/u01/app/crs/bin/vipca" file on both nodes adding the following statement after line 124: unset LD_ASSUME_KERNEL The snapshot below shows edited text with added "unset" command: .The error at the end (libpthread.

Next. edit "/u01/app/crs/bin/srvctl" file on both nodes adding same statement after line 167 so it will look like this: .

To take care of this problem in future.10. Run vipca from odbn2 (still as root): # export DISPLAY=10. we will re-run it manually.1:0 # /u01/app/crs/bin/vipca The vipca fails again with this cryptic message: " [Error 0(Native: listNetInterfaces:[3])]" . when more software is installed into new Oracle Homes or patches applied to existsing software.1. the following files have to be re-edited: <ORA_CRS_HOME>/bin/vipca <ORA_CRS_HOME>/bin/srvctl <RDBMS_HOME>/bin/srvctl <ASM_HOME>/bin/srvctl Since "vipca" utility failed.

Notice: If you have received another cryptic message [Exception in thread "main"] this simply means you forgot to set the DISPLAY variable. let’s run vipca again (still on odbn2) # /u01/app/crs/bin/vipca The following screen pops up (click on Next): . Run following commands as root from odbn2 (notice that the IP addresses here are of the networks. rather than specific adapters): # /u01/app/crs/bin/oifcfg setif -global eth0/ # /u01/app/crs/bin/oifcfg setif -global eth1/10.10.10. This trouble has to be manually fixed.0:cluster_interconnect Let’s see now the resulting configuration of VIP network: # /u01/app/crs/bin/oifcfg getif So.The message has to do (again) with the range of IP addresses we selected for the interfaces.

Agree wih the next screen (eth0 is indeed the public interface): .

Click "Next". Remove the domain name from the second alias. .As soon as we supply first IP alias name as "odbn1-vip" the rest of fields is magically filled out.

Click "Finish". .Summary screen appears.

The progress screen is shown: .

this screen shows. Click "Ok": The summary appears: .When this is finished and you uncross your fingers.

The installer continues .Exit the vipca. Now you can back to Clusterware installer dialog and dismiss the "Execute Configuration scripts" dialogue.

And … fails on Cluster Verification Utility: .

We ignore this message and exit the installer.d/init.* Here are the results (one node shown. the other must show identical results): . Now let’s run a few verification commands on both nodes: # /u01/app/crs/bin/olsnodes -n # /u01/app/crs/bin/crs_stat -t -v # /u01/app/crs/bin/crsctl check crs # ls -l /etc/init.

the index to the whole project is here. Verifications before installation Now we are ready to install Oracle database software but before that we need to run the verification utility again (database pre-installation requirements).Now we are ready to install Oracle database software. Virtual Oracle RAC. Task 11: Oracle installation in RAC nodes – part 2 Rate This Link back: This guide is a part of the Virtual Oracle RAC project. Run these commands from odbn1 as user oracle: $ cd /home/oracle/bin/cluvfy $ export CV_HOME=/home/oracle/bin/cluvfy .

txt Review the report.$ export CV_JDKHOME=/home/oracle/bin/cluvfy/jdk14 $ .sh stage -pre dbinst -n odbn1.odbn2 -r 10gR2 -verbose | tee cvu_report_3. There are some errors that are expected and can be ignored./runcluvfy. in particular: "Could not find a suitable set of interfaces for VIPs" and two errors with "Package existence check failed for …" The connectivity section looks like this (see below). . it says "failed" but we can safely continue with installation.


click "Next": In the next screen choose "Custom" and click on "Next": .Oracle Database software installation As "root" execute this command from odbn1: # mount -r /dev/cdrom /media/cdrom As "oracle" execute these commands from odbn1: $ cd ~ $ /media/cdrom/10201_database_linux32/database/runInstaller & The installer window appears.

Agree with defaults in the next screen and click "Next": .

Click "Next": .In the next screen make sure "Cluster Installation" and all nodes are selected.

Acknowledge the summary screen and click "Next": .

Click "Next": . Hopefully you will get a clean bill of health.The verifications screen appears.

In the next screen correct the name of group for OSOPER as shown. Click "Next": .

Click "Next": .In this screen select "Install Database Software Only".

Click "Install": .Summary screen appears.

.Installation proceeds.

In such case you can un-pause (resume) the machine and/or reboot it. It may happen that during the installation one of the machines will stop responding complaining about low memory (it will be automatically paused and Oracle installer will appear stuck). the clusterware) will reboot itself and then the installer will proceed. I have observed this behaviour in the phase of "Remote Operations In Progress". You can log into them and run "top" to make sure the machines are alive. If all goes well. this configuration pop-up appears: .Make sure the Virtual Box machines (GUI) are somewhere visible. In some cases the machine (well.

Run the "" script on odbn1 and then on odbn2 (one at a time only). The output from odbn1 is show here: .

This bug fix was mentioned before.Exit the OUI. A little cleanup needs to be done. Edit following file (on both nodes): $ vi $ORACLE_HOME/bin/srvctl Add the following text at line 168: unset LD_ASSUME_KERNEL Save the file and exit. Adding TNS Listener This is to be done as user "oracle" from odbn1 only: $ netca & The NETCA screen appears. This concludes the software installation phase. proceed with "Cluster configuration": Make sure both nodes selected and click "Next": .

Select first option: .

Agree to "Add": Accept default LISTENER name: .

Accept default protocol: .

proceed (this may take some time. screen appears frozen): .And default port: Leave "No" for "configure another listener".

Then confirmation screen appears. click "Next": .

Virtual Oracle RAC. now click on "Finish": We can verify the results on both nodes as shown here: and… We are done with listeners.Hey. right? Right. we’ve been here. Task 12: Creating Oracle database in RAC nodes .

Start Database Configuration Assistant from odbn1 node: $ dbca & Leave fisrt option selected: Leave "Create a Databse" selected: . the index to the whole project is here.Rate This Link back: This guide is a part of the Virtual Oracle RAC project.

Click "Select All" nodes: .

Select "Custom Database": .

it has to match what you have in your "hosts" file. it is not a real internet domain.We have discussed the importance of the domain name in the networking of the RAC in the very beginning of this article.Fill out the Global Database Name. . Still. Although the domain name has to be there. .

I am configuring a lightweight RAC configuration so I am not installing OEM or other advanced
options (it is up to you if want OEM, Java etc):

Specify password for all admin accounts (and write it down)

Select ASM as a storage option:

Specify SYS password for ASM instance and fill out location of SPFILE as shown (SPFILE has to be on a shared disk): .

Acknowledge the confirmation: Promising signs… .

click "Ok": .When next screen appears click "Create New" disk group: Fill out details as shown.

Click "Create New" again: .

Fill out details as shown: .

In the next screen select RACDB_DATA1 as a storage option for the database (unselect first group): .

Agree to the "Oracle Managed Files" option: .

Fill out details for Flash Recovery as shown (total size of the disk group minus some 10%): .

As mentioned before. but you can add more options: . I am installing barebone database.

I am even removing standard components as I am not going to use them in this configuration: .

In the next screen click "Add": Fill out "Service Name" as racdb_taf (this is for Transparent Application Failover): Leave both nodes as "Preferred" and select "Basic" as TAF Policy: .

I am accepting default settings (you can change them as you wish): .

Same with storage options. I am accepting all defaults: .

Click "Finish": .Leave "Create Database" option selected and select "Generate Database Creation Scripts" as well (for later review).

Cross fingers and click "Ok" .Summary screen pops up.

Database installation progresses and in the end this screen appers. click "Exit": .

To make sure all registered RAC applications are running (as user "oracle" from any node): .A few moments later the RAC starts (this phase may take a while): Then the OUI will close by itself. Edit the following file (on both nodes): $ vi $ORACLE_HOME/bin/srvctl Add the following text at line 168: unset LD_ASSUME_KERNEL Save the file and exit. Let’s run a few checks. Now we should have a fully functional Oracle RAC. A little cleanup needs to be done (again).

info.idevelopment.$ $ORA_CRS_HOME/bin/crs_stat -t To verify the racdb_taf service. start a sqlplus (on any node): $ sqlplus ‘/as sysdba’ Give following command: SQL> show parameter service See that there are two services displayed. Just for reference Stopping the Oracle RAC 10g Environment on a single node $ export ORACLE_SID=racdb1 $ srvctl stop instance -d racdb -i racdb1 . racdb_taf’ scope=both. If there is only one or none they can be added manually: SQL> alter system set service_names = ‘racdb.

run a long query and then kill one of the nodes.201)(PORT = 1521)) (LOAD_BALANCE = yes) (CONNECT_DATA = . What I really want to do now is to connect to the RAC database. Task 13. the index to the whole project is here. Having some fun with RAC 1 Votes Link back: This guide is a part of the Virtual Oracle RAC project.$ srvctl stop asm -n odbn1 $ srvctl stop nodeapps -n odbn1 Starting the Oracle RAC 10g environment on a single node $ export ORACLE_SID=racdb1 $ srvctl start nodeapps -n odbn1 $ srvctl start asm -n odbn1 $ srvctl start instance -d racdb -i racdb1 Start or stop all Instances of the RAC at once $ srvctl start database -d racdb $ srvctl stop database -d racdb Virtual Oracle RAC.2. We could do a few more things to make the good work perfect. Configure client side TNS (tnsnames. for instance we can enable archive log mode in our RAC and set up backup with RMAN but this is already outside of the scope of this little (I am kidding. Wondering if RAC will stand to its reputation.200)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = 10.10.2. it feels quite big to me) project.10.ora) Add following lines: RACDB_TAF = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = 10.

org) (FAILOVER_MODE = (TYPE = SELECT) (METHOD = BASIC) (RETRIES = 180) (DELAY = 5) ) ) ) In some cases you will need to add a following line to sqlnet. help t2.ora (to overcome possible ORA- 12170 error): sqlnet. .inbound_connect_timeout = 30 Connecting with sqlplus Bring up a sqlplus in your host PC and connect to the cluster: SQL> connect system/password@RACDB_TAF Run this "long" query that will keep the RAC occupied for a while: SQL:> select * from help t1.(SERVER = DEDICATED) (SERVICE_NAME = racdb_taf.harzion.

we will reboot odbn1 node (I did the "shutdown" but you can do "reboot"): .While the query’s output is running (it may run like this for hours).

Next picture shows that OCFS was stopped properly: .

The other node is running happily if quite busy all the meanwhile: .

OCFS loaded.And the sqlplus is still producing the output… The odbn1 node is coming back. then ASM follows .

then they will start appearing.If you log in to the console and run the "top" command filtering user "oracle" (press "u" to specify user). . you will notice that for a minute or two there will be no processes for "oracle".

and finally you will have plenty of them. just as on the other node: .

just for fun. Go back. Now. I will reboot the second node. then most probably you have skipped the step where we edited "svrctl" script. . odbn2.If noting happens within 5 minutes though. edit and reboot again. I do not have much hope that my query will survive this too. It would be too good.

Sure enough. the query dies as well. although the first node is running again: . just as OCFS is going down.

I consider this project concluded.Well. thanks for your patience and comments. .