Building an Inexpensive Oracle RAC 10g R2 on Linux - (RHEL 5.

3 / iSCSI)
by Jeff Hunter, Sr. Database Administrator

Contents 1. Overview 2. Oracle RAC 10g Overview 3. Shared-Storage Overview 4. iSCSI Technology 5. Hardware and Costs 6. Install the Linux Operating System 7. Install Required Linux Packages for Oracle RAC 8. Network Configuration 9. Install Openfiler 10. Configure iSCSI Volumes using Openfiler 11. Configure iSCSI Volumes on Oracle RAC Nodes 12. Create "oracle" User and Directories 13. Configure the Linux Servers for Oracle 14. Configure the "hangcheck-timer" Kernel Module 15. Configure RAC Nodes for Remote Access using SSH 16. All Startup Commands for Both Oracle RAC Nodes 17. Install and Configure Oracle Cluster File System (OCFS2) 18. Install and Configure Automatic Storage Management (ASMLib 2.0) 19. Download Oracle RAC 10g Software 20. Pre-Installation Tasks for Oracle10g Release 2 21. Install Oracle Clusterware 10g Software 22. Install Oracle Database 10g Software 23. Install Oracle Database 10g Companion CD Software 24. Create TNS Listener Process 25. Create the Oracle Cluster Database 26. Post-Installation Tasks - (Optional) 27. Verify TNS Networking Files 28. Create / Alter Tablespaces 29. Verify the RAC Cluster & Database Configuration 30. Starting / Stopping the Cluster 31. Transparent Application Failover - (TAF) 32. Troubleshooting 33. Conclusion 34. Building an Oracle RAC Cluster Remotely 35. Acknowledgements 36. About the Author

Overview One of the most efficient ways to become familiar with Oracle Real Application Cluster (RAC) 10g technology is to have access to an actual Oracle RAC 10g cluster. In learning this new technology, you will soon start to realize the benefits Oracle RAC 10g has to offer like fault tolerance, new levels of security, load balancing, and the ease of upgrading capacity. The problem though is the price of the hardware required for a typical production RAC configuration. A small two node cluster, for example, could run anywhere from US$10,000 to well over US$20,000. This would not even include the heart of a production RAC environment, the shared storage. In most cases, this would be a Storage Area Network (SAN), which generally start at US$10,000. For those who want to become familiar with Oracle RAC 10g without a major cash outlay, this guide provides a low-cost alternative to configuring an Oracle RAC 10g Release 2 system using commercial off-the-shelf components and downloadable software at an estimated cost of US$2,500 to US$3,000. The system will consist of a dual node cluster (two i386 nodes with a single processor), both running Linux (CentOS 5.3 or Red Hat Enterprise Linux 5 Update 3), Oracle10g Release 2 for Linux x86, OCFS2, and ASMLib 2.0. All shared disk storage for Oracle RAC will be based on iSCSI using Openfiler Release 2.3 (Final) x86_64 running on a third node (known in this article as the Network Storage Server). Powered by rPath Linux, Openfiler is a free browser-based network storage management utility that delivers filebased Network Attached Storage (NAS) and block-based Storage Area Networking (SAN) in a single framework. The entire software stack interfaces with open source applications such as Apache, Samba, LVM2, ext3, Linux NFS and iSCSI Enterprise Target. Openfiler combines these ubiquitous technologies into a small, easy to manage solution fronted by a powerful web-based management interface. Openfiler supports CIFS, NFS, HTTP/DAV, FTP, however, we will only be making use of its iSCSI capabilities to implement an inexpensive SAN for the shared storage components required by Oracle RAC 10g. The operating system and Openfiler application will be installed on one internal SATA disk. A second internal 73GB 15K SCSI hard disk will be configured as a single "Volume Group" that will be used for all shared disk storage requirements. The Openfiler server will be configured to use this volume group for iSCSI based storage and will be used in our Oracle RAC 10g configuration to store the shared files required by Oracle Clusterware as well as all Oracle ASM volumes. This article is provided for educational purposes only, so the setup is kept simple to demonstrate ideas and concepts. For example, the disk mirroring configured in this article will be setup on one physical disk only, while in practice that should be performed on multiple physical drives.

In addition, each Linux node will only be configured with two network interfaces — one for the public networ (eth0) and one that will be used for both the Oracle RAC private interconnect "and" the network storage server for shared iSCSI access (eth1). For a production RAC implementation, the private interconnect should be at least Gigabit (or more) with redundant paths and "only" be used by Oracle to transfer Cluster Manager and Cache Fusion related data. A third dedicated network interface (i.e. eth2) should be configured on another

redundant Gigabit network for access to the network storage server (Openfiler). Finally, while this article provides detailed instructions for successfully installing a complete Oracle RAC 10g system, it is by no means a substitute for the official Oracle documentation. In addition to this article, users should also consult the following Oracle documents to gain a full understanding of alternative configuration options, installation, and administration with Oracle RAC 10g. Oracle's official documentation site is docs.oracle.com. Oracle Clusterware and Oracle Real Application Clusters Installation Guide - 10g Release 2 (10.2) for Linux Oracle Clusterware and Oracle Real Application Clusters Administration and Deployment Guide - 10g Release 2 (10.2) 2 Day + Real Application Clusters Guide - 10g Release 2 (10.2) Although in past articles I used raw partitions for storing files on shared storage, here we will make use of the Oracle Cluster File System V2 (OCFS2) and Oracle Automatic Storage Management (ASM). The two Oracle RAC nodes will be configured as follows:
Oracle Database Files RAC Node Name linux1 linux2 racdb1 racdb2 Instance Name Database Name racdb racdb $ORACLE_BAS E /u01/app/oracle /u01/app/oracle File System Volume Manager for DB Files ASM ASM

Oracle Clusterware Shared Files File Type Oracle Cluster Registry (OCR) File Name iSCSI Volume Name Mount Point File System

/u02/oradata/racdb/OCRFile /u02/oradata/racdb/OCRFile_mirr crs or /u02/oradata/racdb/CSSFile /u02/oradata/racdb/CSSFile_mirro r1 crs /u02/oradata/racdb/CSSFile_mirro r2

/u02

OCFS2

Voting Disk

/u02

OCFS2

With Oracle Database 10g Release 2 (10.2), Cluster Ready Services, or CRS, is now called Oracle Clusterware. The Oracle Clusterware software will be installed to /u01/app/crs on both of the nodes that make up the RAC cluster. Starting with Oracle Database 10g Release 2 (10.2), Oracle Clusterware should be installed in a separate Oracle Clusterware home directory which is non-release specific (/u01/app/oracle/product/10.2.0/... for example) and must never be a subdirectory of the ORACLE_BASE directory (/u01/app/oracle for example). This is a change to the Optimal Flexible Architecture (OFA) rules. Note that the Oracle Clusterware and Oracle Real

0 / FireWire) Building an Inexpensive Oracle RAC 9i on Linux . The Oracle Clusterware software will be installed to /u01/app/crs on both of the nodes that make up the RAC cluster. and the Oracle Clusterware home must never be a subdirectory of the ORACLE_BASE directory.e. online redo logs. "Preinstallation".2 / FireWire) Building an Inexpensive Oracle RAC 10g Release 1 on Linux .(B14203-08 HAS CONFLICTING CRS_HOME LOCATIONS ) and is fixed in Oracle 11g. It is also possible to use RAW devices for these files. I tend to stick with Dell hardware given their superb quality and compatibility with Linux. These two files will be installed on shared storage using Oracle's Cluster File System. and shared storage medium: Building an Inexpensive Oracle RAC 10g Release 2 on Linux . Operating System. however.. makes the article that much more interesting! This article is only designed to work as documented with absolutely no substitutions! The only exception here is the choice of vendor hardware (i.(WBEL 3. The Oracle database files could have just as well been stored on the Oracle Cluster File System (OFCS2). The following is a list of past articles which describe configuring a similar Oracle RAC Cluster using various versions of Oracle. Stallard Technologies has a proven track record of delivering the best value on pre-owned hardware combined with a commitment to superior customer service.(Fedora Core 1 / FireWire) . Ensure that the hardware you purchase from the vendor is supported on Red Hat Linux 5. the "Oracle Cluster Registry (OCR)" file and the "Voting Disk" file be shared with both nodes in the cluster.(CentOS 4. networking equipment. I highly recommend purchasing pre-owned or refurbished Dell hardware from a reputable company like Stallard Technologies. the Clusterware software requires that two of its files. Using ASM. For example. it is not possible to use ASM for these two shared Clusterware files. Release 2 (OCFS2). visit their website or contact William Buchanan.Application Clusters installation documentation from Oracle incorrectly state that the Oracle Clusterware home directory can be a subdirectory of the ORACLE_BASE directory. The Oracle Database 10g Release 2 software will be installed into a separate Oracle Home. in the section "Oracle Clusterware home directory". All of the Oracle physical database files (data. in Chapter 2. This issue is tracked with Oracle documentation bug "5843155" . For a test system of this nature. however. archived redo logs) will be installed to shared volumes being managed by Automatic Storage Management (ASM). To learn more about Stallard Technologies.2. Inc. This is incorrect. and internal / external hard drives). it incorrectly lists the path /u01/app/oracle/product/crs as a possible Oracle Clusterware home (or CRS home) path. namely /u01/app/oracle/product/10. The default ORACLE_BASE path is /u01/app/oracle. control files.0/db_1 on both of the nodes that make up the RAC cluster. however. machines. I base my recommendation on my own outstanding personal experience with their organization.

The first successful cluster product was developed by DataPoint in 1977 named ARCnet. With the release of Oracle 6 for the Digital VAX cluster product. but made for a complex environment to setup and manage given the multiple layers involved. but didn't really take off in the commercial market. the other surviving server (or servers) can take over the workload from the failed server and the application continue to function normally as if nothing has happened. Oracle Corporation was the first commercial database to support clustering at the database level. however. Like OPS. By Oracle 7. this became known as the Integrated Distributed Lock Manager (IDLM) and relied on an additional layer known as the Operating System Dependant (OSD) layer. before Oracle realized the need for a more efficient and scalable distributed lock manager (DLM) as the one included with the VAX/VMS cluster product was not well suited for database applications. With Oracle 9i. . By Oracle 10g. With the release of Oracle Database 10g Release 2 (10. The key benefit of clustering is to provide a highly available framework where the failure of one node (for example a database server) does not bring down an entire application. the failure of one node will not cause the loss of access to the database. and at the same time since all instances access the same database. Cluster Ready Services was renamed to Oracle Clusterware. it might be helpful to first clarify what a cluster is. This framework required vendor-supplied clusterware which worked well.Oracle RAC 10g Overview Before introducing the details for building a RAC cluster. and performance benefits by allowing the system to scale out. Oracle Real Application Clusters (RAC). load balancing. RAC provides fault tolerance. It wasn't until the 1980's when Digital Equipment Corporation (DEC) released its VAX cluster product for the VAX/VMS operating system.2). This guide uses Oracle Clusterware 10g.2 which gave birth to Oracle Parallel Server (OPS) . is the successor to Oracle Parallel Server. In later releases of Oracle. It wasn't long. A cluster is a group of two or more interconnected computers or servers that appear as if they are one server to end users and applications and generally share the same set of physical disks. When using Oracle 10g or higher.the first database to run the parallel server. In the case of failure with one of the servers. The ARCnet product enjoyed much success by academia types in research labs. Oracle decided to design and write their own DLM for the VAX/VMS cluster product which provided the fine-grain block level locking required by the database. OPS was extended to included support for not only the VAX/VMS cluster product but also with most flavors of UNIX. Oracle's own DLM was included in Oracle 6. but to also create their own clusterware product in future releases. Using the same IDLM. Oracle Clusterware is the only clusterware that you need for most platforms on which Oracle RAC operates (except for Tru cluster. The concept of clustering computers actually started several decades ago. introduced with Oracle9i. Oracle introduced a generic lock manager which was integrated into the Oracle kernel. Oracle RAC allows multiple instances to access the same database (storage) simultaneously. By Oracle 8. This new model paved the way for Oracle to not only have their own DLM. You can still use clusterware from other vendors if the clusterware is certified for Oracle RAC. CRS was only available for Windows and Linux. Oracle 9i could still rely on external clusterware but was the first release to include their own clusterware product named Cluster Ready Services (CRS). Oracle's clusterware product was available for all operating systems. where you need vendor clusterware).

you need servers that . Some vendors use an approach known as a Federated Cluster. Each instance in the cluster must be able to access all of the data. then the requesting instance can read that data (after acquiring the required locks). in which data is spread across several machines rather than shared by all. As mentioned earlier. The other instances in the cluster must be able to access them (read-only) in order to recover that instance in the event of a system failure. or switched topologies (FC-SW).000 for a two-node cluster. Pre-configured Oracle RAC solutions are available from vendors such as Dell. but for administrators and developers who are used to GPL-based Linux prices. The redo log files for an instance are only writeable by that instance and will only be read from another instance during system failure.000 to US$5.g.000. With OPS a request for data from one instance to another required the data to be written to disk first. however. Protocols supported by Fibre Channel include SCSI and IP. control files and parameter file for all other instances in the cluster. For more background about Oracle RAC. however. Oracle's approach to clustering leverages the collective processing power of all the nodes in the cluster and at the same time provides failover security.000. With Oracle RAC. Just the fibre channel switch alone can start at around US$1. fibre channel is a high-speed serial-transfer interface that is used to connect systems and storage devices in either point-topoint (FC-P2P). The data disks must be globally available in order to allow all instances to access the database. The UNDO. This article. which can reach prices of about US$300 for a 36GB drive. With cache fusion. This does not even include the fibre channel storage array and high-end drives. however. A less expensive alternative to fibre channel is SCSI.12 gigabits per second in each direction. Fibre channel configurations can support as many as 127 nodes and have a throughput of up to 2. is read all the time during normal database operation (e. fibre channel is one of the most popular solutions for shared storage. redo log files. Specifically. Shared-Storage Overview Today. Not all database clustering solutions use shared storage. at around US$2. which does not include the cost of the servers that make up the cluster. multiple instances use the same set of disks for storing data. data is passed along a high-speed interconnect using a sophisticated locking algorithm. Fibre channel. IBM and HP for production environments. iSCSI. SCSI technology provides acceptable performance for shared storage. even SCSI can come in over budget. arbitrated loop (FC-AL).25 Gbps is expected. and 4. Each instance has its own redo log files and UNDO tablespace that are locally read-writeable. on the other hand. The biggest difference between Oracle RAC and OPS is the addition of Cache Fusion. focuses on putting together your own Oracle RAC 10g environment for development and testing by using Linux servers and a low cost shared disk solution. Another popular solution is the Sun NFS (Network File System) found on a NAS. A typical fibre channel setup which includes fibre channel cards for the servers is roughly US$10. is very expensive. visit the Oracle RAC Product Center on OTN.At the heart of Oracle RAC is a shared disk subsystem. for CR fabrication). It can be used for shared storage but only if you are using a network appliance or something similar.

Oracle is able to optimize the I/O path between the Oracle software and the NFS server resulting in significant performance gains. better scalability. many product manufacturers have interpreted the Fibre Channel specifications differently from each other which has resulted in scores of interconnect problems. iSCSI is a data transport protocol defined in the SCSI-3 specifications framework and is similar to Fibre Channel in that it is responsible for carrying block-level data over a storage network. iSCSI SANs remain the leading competitor to FC SANs. Database servers depend on this type of . The first is price. the Internet Small Computer System Interface. but given the low-end hardware being used. is an Internet Protocol (IP)-based storage networking standard for establishing and managing connections between IP-based storage devices. While the costs involved in building a FC SAN have come down in recent years. Today. a new feature known as Direct NFS Client integrates the NFS client functionality directly in the Oracle software. Several of the advantages to FC SAN include greater performance. TCP as the transport protocol. and most important to us — support for server clustering! Still today. see the Oracle White Paper entitled "Oracle Database 11g Direct NFS Client". Based on an earlier set of ANSI protocols called Fiber Distributed Data Interface (FDDI). Through this integration. the only technology that existed for building a network based storage solution was a Fibre Channel Storage Area Network (FC SAN). This solution offers a low-cost alternative to fibre channel for testing and educational purposes. The shared storage that will be used for this article is based on iSCSI technology using a network storage server installed with Openfiler. increased disk utilization. improved availability. it is not recommended to be used in a production environment. One of the key drawbacks that has limited the benefits of using NFS and NAS for database storage has been performance degradation and complex configuration requirements. To learn more about Direct NFS Client. iSCSI Technology For many years. FC SANs suffer from three major disadvantages. and clients. With the introduction of Oracle 11g. however. Standard NFS client software (client systems that use the operating system provided NFS driver) is not optimized for Oracle database file I/O access patterns. and read/write block sizes of 32K. Block-level communication means that data is transferred between the host and the client in chunks called blocks. See the Certify page on Oracle Metalink for supported Network Attached Storage (NAS) devices that can be used with Oracle RAC. Since its adoption.guarantee direct I/O over NFS. 2003 by the Internet Engineering Task Force (IETF). The third disadvantage is the fact that a Fibre Channel network is not Ethernet! It requires a separate network technology along with a second set of skill sets that need to exist with the datacenter staff. better known as iSCSI. Direct NFS Client can simplify. Ratified on February 11. the cost of entry still remains prohibitive for small companies with limited IT budgets. Fibre Channel has recently been given a run for its money by iSCSI-based storage systems. Fibre Channel was developed to move SCSI commands over a storage network. When purchasing Fibre Channel components from a common manufacturer. the performance optimization of the NFS client configuration for database workloads. The second is incompatible hardware components. With the popularity of Gigabit Ethernet and the demand for lower cost. and in many cases automate. hosts. this is usually not a problem.

and robust reliability. For the purpose of this article. most of the processing of the data (both TCP and iSCSI) is handled in software and is much slower than Fibre Channel which is handled completely in hardware. and mission critical reliability will undoubtedly continue to choose Fibre Channel. The iSCSI software initiator is generally used with a standard network interface card (NIC) — a Gigabit Ethernet card in most cases. The TCP/IP protocol. The beauty of iSCSI is its ability to utilize an already familiar IP network as its transport mechanism. iSCSI HBAs are available from a number of vendors. however. iSCSI Initiator Basically. including Adaptec. and QLogic. For many the solution is to do away with iSCSI software initiators and invest in specialized cards that can offload TCP/IP and iSCSI processing from a server's CPU. however. Fibre Channel has clearly demonstrated its capabilities over the years with its capacity for extremely high speeds. Like a FC SAN. and gigabytes (GB) per second with some of the more common ones highlighted in grey. large complex connectivity. I provide the maximum transfer rates in kilobits (kb). an iSCSI SAN should be a separate physical network devoted entirely to storage. So with all of this talk about iSCSI. The overhead incurred in mapping every SCSI command onto an equivalent iSCSI transaction is excessive. An iSCSI initiator can be implemented using either software or hardware. Software iSCSI initiators are available for most major operating system platforms. For each interface. kilobytes (KB). The iSCSI initiator software will need to exist on each of the Oracle RAC nodes (linux1 and linux2). gigabits (Gb). which is basically just a specialized Ethernet card with a SCSI ASIC on-board to offload all the work (TCP and SCSI commands) from the system CPU. megabytes (MB). Customers who have strict requirements for high performance storage. many of its early critics were quick to point out some of its inherent shortcomings with regards to performance. its components can be much the same as in a typical IP network (LAN). Intel. iSCSI Target An iSCSI target is the "server" component of an iSCSI network. an iSCSI initiator is a client device that connects and initiates requests to some service offered by a server (in this case an iSCSI target). A hardware initiator is an iSCSI HBA (or a TCP Offload Engine (TOE) card).communication (as opposed to the file level communication used by most NAS systems) in order to work properly. Alacritech. Before closing out this section. For the purpose of this article. it is only important to understand the difference between an iSCSI initiator and an iSCSI target. is very complex and CPU intensive. the node openfiler1 will be the iSCSI target. iSCSI comes with its own set of acronyms and terminology. megabits (Mb). flexibility. This is typically the storage device that contains the information you want and answers requests from the initiator(s). . These specialized cards are sometimes referred to as an iSCSI Host Bus Adaptor (HBA) or a TCP Offload Engine (TOE) card. does this mean the death of Fibre Channel anytime soon? Probably not. With iSCSI. we will be using the free Linux Open-iSCSI software driver found in the iscsi-initiator-utils RPM. While iSCSI has a promising future. I thought it would be appropriate to present the following chart that shows speed comparisons of the various types of disk interfaces and network technologies. Also consider that 10-Gigabit Ethernet is a reality today! As with any new technology. For this article.

25 1.(IEEE1394b) Gigabit Ethernet PCI .064 1.(133 MHz / 32-bit) Speed Kb KB 920 115 Mb 0.(SATA II) Ultra320 SCSI FC-AL Fibre Channel PCI-Express x1 .2 1.11b wireless Wi-Fi (2.5 3 5 6.(bidirectional) PCI .4 GHz band) SCSI-2 (Fast SCSI / Fast Narrow SCSI) 100Base-T Ethernet (Fast Ethernet) ATA/100 (parallel) IDE Fast Wide SCSI (Wide SCSI) Ultra SCSI (SCSI-3 / Fast-20 / Ultra Narrow) Ultra IDE Wide Ultra SCSI (Fast Wide 20) Ultra2 SCSI FireWire 400 .375 0.5 20 20 33 40 40 50 60 80 80 100 125 133 150 160 160 266 266 266 300 320 400 500 532 533 1 1.(SATA I) Wide Ultra3 SCSI Ultra160 SCSI PCI .28 2.375 1.4 2.115 0.128 2.2 4 4.(66 MHz / 32-bit) AGP 1x .11g wireless WLAN (2.5 12.(66 MHz / 32-bit) Serial ATA II .4 GHz band) USB 1.264 Gb GB 115 14.28 1.6 16.92 10 11 12 24 40 54 80 100 100 160 160 264 320 320 400 480 640 640 800 1000 1064 1200 1280 1280 2128 2128 2128 2400 2560 3200 4000 4256 4264 MB 0.014 133.256 4.(33 MHz / 64-bit) PCI .Disk Interface / Network / BUS Serial Parallel (standard) 10Base-T Ethernet IEEE 802.7 .1 Parallel (ECP/EPP) SCSI-1 IEEE 802.56 3.0 Wide Ultra2 SCSI Ultra3 SCSI FireWire 800 .128 2.115 1.128 2.75 10 12.(33 MHz / 32-bit) Serial ATA I .(IEEE1394a) USB 2.(66 MHz / 64-bit) AGP 2x .

80GHz . The Dell Dimension includes an integrated 10/100 Ethernet adapter that will be used to connect to the public network. Gigabit Ethernet Intel 10/100/1000Mbps PCI Desktop Adapter .1 Hardware and Costs The hardware used to build our example Oracle RAC 10g environment consists of three Linux servers (two Oracle RAC nodes and one Network Storage Server) and components that can be purchased at many local computer stores or over the Internet (i.Serial ATA III .(133 MHz / 64-bit) AGP 4x .(Connected to KVM Switch) 1 .Ethernet LAN Card Used for RAC interconnect to linux2 and Openfiler networked storage. Each Linux server for Oracle RAC should contain two NIC adapters. Inc.528 10 16 32 64 1 1 1. The second NIC adapter will be used for the private network (RAC interconnect and Openfiler networked storage).e.(Broadcom BCM4401) .No Keyboard.(linux1) Dell Dimension 2400 Series .40GB 7200 RPM Internal Hard Drive .3.(bidirectional) 4800 6400 600 800 1064 1066 1250 2000 2133 4000 8000 4.(SATA III) PCI-X .25 2 4 8 17. I used a Gigabit Ethernet switch (and 1Gb Ethernet cards) for the private network.8 6. Monitor. For the purpose of this article.5" Floppy .064 2.Integrated 10/100 Ethernet .2GB DDR SDRAM (at 333MHz) . Oracle RAC Node 1 .(PWLA8391GT) US$35 US$600 .Integrated Intel 3D AGP Graphics . Select the appropriate NIC adapter that is compatible with the maximum data transmission speed of the network switch to be used for the private network.512 8.).(bidirectional) AGP 8x .3ae) PCI-Express x4 .(IEEE 802.4 8.(100 MHz / 64-bit) PCI-X .CDROM (48X Max Variable) .(bidirectional) PCI-Express x16 .(266 MHz / 32-bit) 10G Ethernet .Intel(R) Pentium(R) 4 Processor at 2. Stallard Technologies. or Mouse .(533 MHz / 32-bit) PCI-Express x8 .

No Keyboard. I could have made an extra partition on the 500GB internal SATA disk for the iSCSI target.5" Floppy . I used a Gigabit Ethernet switch (and 1Gb Ethernet cards) for the private network.No Keyboard.0GHz Xeon / 1MB Cache / 800FSB (SL7PE) .Oracle RAC Node 2 .73GB 15K SCSI Internal Hard Disk .(Connected to KVM Switch) Note: The operating system and Openfiler application will be installed on the 500GB internal SATA disk.(PWLA8391GT) US$35 US$600 Network Storage Server .(openfiler1) Dell PowerEdge 1800 .Single embedded Intel 10/100/1000 Gigabit NIC .(linux2) Dell Dimension 2400 Series . For the purpose of this article.CDROM (48X Max Variable) .Integrated Intel 3D AGP Graphics . Monitor.Integrated 10/100 Ethernet .6GB of ECC Memory . but decided to make use of the faster SCSI disk for this US$800 . The Openfiler server will be configured to use this second hard disk for iSCSI based storage and will be used in our Oracle RAC 10g configuration to store the shared files required by Oracle Clusterware as well as all Oracle ASM volumes.16x DVD Drive . The Dell Dimension includes an integrated 10/100 Ethernet adapter that will be used to connect to the public network. or Mouse .(Connected to KVM Switch) 1 .2GB DDR SDRAM (at 333MHz) .500GB SATA Internal Hard Disk . Monitor.3. A second internal 73GB 15K SCSI hard disk will be configured for the database storage.Dual 3. For example. Gigabit Ethernet Intel 10/100/1000Mbps PCI Desktop Adapter . Each Linux server for Oracle RAC should contain two NIC adapters. Select the appropriate NIC adapter that is compatible with the maximum data transmission speed of the network switch to be used for the private network. The second NIC adapter will be used for the private network (RAC interconnect and Openfiler networked storage).40GB 7200 RPM Internal Hard Drive .Ethernet LAN Card Used for RAC interconnect to linux1 and Openfiler networked storage.Integrated Graphics .(Broadcom BCM4401) .80GHz .Intel(R) Pentium(R) 4 Processor at 2. Please be aware that any type of hard disk (internal or external) should work for database storage as long as it can be recognized by the network storage server (Openfiler) and has adequate space. or Mouse .

(Connect linux1 to interconnect Ethernet switch) Category 6 patch cable . keyboard. 1 . Gigabit Ethernet D-Link 8-port 10/100/1000 Desktop Switch . video monitor and mouse. Select the appropriate NIC adapter that is compatible with the maximum data transmission speed of the network switch to be used for the private network. A KVM switch is a hardware device that allows a user to control multiple computers from a single keyboard. Mouse Switch —better known as a KVM Switch.(Connect openfiler1 to public network) Category 6 patch cable . A more practical solution would be to configure a dedicated computer which would include a single monitor. For the purpose of this article.(DGS-2208) 6 . Gigabit Ethernet Intel 10/100/1000Mbps PCI Desktop Adapter .example. The Dell PowerEdge 1800 machine includes an integrated 10/100/1000 Ethernet adapter that will be used to connect to the public network. and mouse in order to access its console. keyboard.Ethernet LAN Card Used for networked storage on the private network. as the number of servers to manage increases. and mouse that would have direct access to the console of each server.Ethernet Switch Used for the interconnect between linux1-priv and linux2-priv. I used a Gigabit Ethernet switch (and 1Gb Ethernet cards) for the private network.(Connect linux2 to interconnect Ethernet switch) Category 6 patch cable . The second NIC adapter will be used for the private network (Openfiler networked storage).(PWLA8391GT) US$35 Miscellaneous Components 1 .(Connect linux1 to public network) Category 6 patch cable . US$340 . When managing a very small number of servers. This switch will also be used for network storage traffic for Openfiler.(Connect openfiler1 to interconnect Ethernet switch) US$10 US$10 US$10 US$10 US$10 US$10 US$50 Optional Components KVM Switch This article requires access to the console of all nodes (servers) in order to install the operating system and perform several of the configuration tasks.Network Cables Category 6 patch cable . For the purpose of this article. This solution is made possible using a Keyboard. Video. I used a Gigabit Ethernet switch (and 1Gb Ethernet cards) for the private network. The Network Storage Server (Openfiler server) should contain two NIC adapters. However. it might make sense to connect each server with its own monitor. this solution becomes unfeasible.(Connect linux2 to public network) Category 6 patch cable .

Total US$2. let's take a conceptual look at what the environment would look like after connecting all of the hardware components (click on the graphic below to view larger image): Figure 1: Architecture As we start to go into the details of the installation. please see the article "KVM Switches For the Home and the Enterprise".Avocent provides a high quality and economical 4-port switch which includes four 6' cables: SwitchView® 1000 . I will indicate at the beginning of each .(4SV1000BND1-001) For a detailed explanation and guide on the use and KVM switches. Now that we have talked about the hardware that will be used in this example.555 We are about to start the installation process. it should be noted that most of the tasks within this document will need to be performed on both Oracle RAC nodes (linux1 and linux2).

Install the Linux Operating System Perform the following installation on both Oracle RAC nodes in the cluster! This section provides a summary of the screens used to install the Linux operating system.iso CentOS-5.3-i386-bin-3of6.3-x86_64-bin-1of7.3-x86_64-bin-2of7.3-x86_64-bin-3of7.iso (619 MB) (554 MB) (637 MB) (629 MB) (635 MB) .3-x86_64-bin-5of7.iso CentOS-5.iso (3. 32-bit (x86) Installations       CentOS-5.3-i386-bin-5of6.iso CentOS-5.3-i386-bin-2of6.3-i386-bin-4of6.iso CentOS-5.3-i386-bin-6of6.iso CentOS-5. Downloading CentOS CentOS.iso CentOS-5.7 GB) 64-bit (x86_64) Installations      CentOS-5.iso CentOS-5.3-i386-bin-DVD.iso CentOS-5.section whether or not the task(s) should be performed on both Oracle RAC nodes or on the network storage server (openfiler1).3-x86_64-bin-4of7.iso CentOS-5. This guide is designed to work with CentOS Release 5 Update 3 for x86 and follows Oracle's suggestion of performing a "default RPMs" installation type to ensure all expected Linux O/S packages are present for a successful Oracle RDBMS installation.3-i386-bin-1of6.iso (624 MB) (636 MB) (634 MB) (632 MB) (638 MB) (625 MB) If the Linux RAC nodes have a DVD installed.org Download and burn the following ISO images to CD/DVD for CentOS Release 5 Update 3 for either x86 or x86_64 depending on your hardware architecture. you may find it more convenient to make use of the single DVD image:  CentOS-5.

redhat. After several seconds. and mouse.3-x86_64-bin-7of7. the installer should then detect the video card. After downloading and burning the CentOS images (ISO files) to CD/DVD. tab over to [Skip] and hit [Enter].iso (637 MB) CentOS-5. here are just two (of many) software packages that can be used: UltraISO Magic ISO Maker Installing CentOS This section provides a summary of the screens used to install CentOS. Welcome to CentOS At the welcome screen. For more detailed installation instructions.3-x86_64-bin-6of7. If you are not familiar with this process and do not have the required software to burn images to CD. At the boot: prompt. power it on. I would suggest. the media burning software would have warned us.3-x86_64-bin-DVD. You may already be familiar with and have the proper software to burn images to CD.iso (4. monitor. click [Next] to continue. insert CentOS Disk #1 into the first server (linux1 in this example). Media Test When asked to test the CD media. If there were any errors. you may find it more convenient to make use of the single DVD image:  CentOS-5. Before installing the Linux operating system on both nodes. there are many options for burning these images (ISO files) to a CD. you should have the two NIC interfaces (cards) installed.2 GB) If you are downloading the above ISO files to a MS Windows machine. hit [Enter] to start the installation process. that the instructions I have provided below be used for this Oracle RAC 10g configuration. The installer then goes into GUI mode.  CentOS-5. it is possible to use the manuals from Red Hat Linux http://www. After completing the Linux installation on the first node.com/docs/manuals/. . and answer the installation screen prompts as noted below.iso (638 MB) If the Linux RAC nodes have a DVD installed. however. perform the same Linux installation on the second node while substituting the node name linux1 for linux2 and the different IP addresses were appropriate. Boot Screen The first screen is the CentOS boot screen.

048MB of RAM installed.5 times the size of RAM Between 2. To increase the size of the swap partition.192MB . to add another 512MB to swap. and the rest going to the root (/) partition. you can easily change that from this screen. double the amount of RAM (systems with <= 2. Make the appropriate selection for your configuration and click [Next] to continue.Language / Keyboard Selection The next two screens prompt you for the Language and Keyboard settings. the installer will create the same disk configuration as just noted but will create them using the Logical Volume Manager (LVM). you can accept the defaults.032MB . For example.e. Click [Yes] to acknowledge this warning. 36.048MB RAM) for swap. First.049MB and 8.048MB RAM) or an amount equal to RAM (systems with > 2. I will accept all automatically preferred sizes. it will partition the first hard drive (/dev/hda for my configuration) into two partitions — one for the /boot partition (/dev/hda1) and the remainder of the disk dedicate to a LVM named VolGroup00 (/dev/hda2).one for the root filesystem (/) and another for swap.024MB and 2. click [OK] on the "Edit LVM Volume Group: VolGroup00" dialog. [Edit] and decrease the size of the root file system (/) by the amount you want to add to the swap partition. The following is Oracle's requirement for swap space: Available RAM Swap Space Required Between 1. Click [Next] to continue.512MB = 35.75 times the size of RAM For the purpose of this install. . Now add the space you decreased from the root file system (512MB) to the swap partition. Once you are satisfied with the disk layout. In almost all cases. The LVM Volume Group (VolGroup00) is then partitioned into two LVM partitions . When completed. If for any reason.192MB Equal to the size of RAM More than 8. [Edit] the volume group VolGroup00. This will bring up the "Edit LVM Volume Group: VolGroup00" dialog. Partitioning The installer will then allow you to view (and modify if needed) the disk partitions it automatically selected. the automatic layout does not configure an adequate amount of swap space. You will then be prompted with a dialog window asking if you really want to remove all Linux partitions.520MB). the installer will choose 100MB for /boot. The main concern during the partitioning phase is to ensure enough swap space is allocated as required by Oracle (which is a multiple of the available RAM). Starting with RHEL 4. (Including 4.096MB for swap since I have 2. For example. Disk Partitioning Setup Select [Remove all partitions on selected drives and create default layout] and check the option to [Review and modify partitioning layout].048MB 1. For most automatic layouts. you would decrease the size of the root file system by 512MB (i. click [Next] to continue.

The settings you make here will. accept all default values and click [Next] to continue.Check ON the option to [Enable IPv4 support] . Verify that the option "Enable IPv4 support" is selected. depend on your network configuration. Put eth1 (the interconnect) on a different subnet than eth0 (the public network): eth0: .0 . The key point to make is that the machine should never be configured with DHCP since it will be used to host an Oracle instance.255.168.Check ON the option to [Enable IPv4 support] .Check OFF the option to use [Dynamic IP configuration (DHCP)] .Check OFF the option to use [Dynamic IP configuration (DHCP)] .100 Prefix (Netmask): 255. The installer may choose to not activate eth1 by default.255. there will be several changes that need to be made to the network configuration. You will also need to configure the server with a real host name.2. Time Zone Selection Select the appropriate time zone for your environment and click [Next] to continue. Finish this dialog off by supplying your gateway and DNS servers. Click off the option to "Enable IPv6 support".Boot Loader Configuration The installer will use the GRUB boot loader by default. This screen should have successfully detected each of the network devices.Check OFF the option to [Enable IPv6 support] Continue by manually setting your hostname.255.1. You may choose to use different IP addresses for both eth0 and eth1 that I have documented in this guide and that is OK. Second.0 . To use the GRUB boot loader. make sure that each of the network devices are checked to [Active on boot]. First.168.(select Manual configuration) IPv4 Address: 192. [Edit] both eth0 and eth1 as follows. of course. I used "linux1" for the first node and "linux2" for the second.255.(select Manual configuration) IPv4 Address: 192.100 Prefix (Netmask): 255. Set Root Password . Since we will be using this machine to host an Oracle instance.Check OFF the option to [Enable IPv6 support] eth1: . Click off the option to use "Dynamic IP configuration (DHCP)" by selecting the "Manual configuration" radio button and configure a static IP address and Netmask for your environment. You will need to configure the machine with static IP addresses. Network Configuration I made sure to install both NIC interfaces (cards) in each of the Linux machines before starting the operating system installation.

Select a root password and click [Next] to continue. Package Installation Defaults By default, CentOS Linux installs most of the software required for a typical server. There are several other packages (RPMs), however, that are required to successfully install the Oracle software. The installer includes a "Customize software" selection that allows the addition of RPM groupings such as "Development Libraries" or "Legacy Library Support". The addition of such RPM groupings is not an issue. De-selecting any "default RPM" groupings or individual RPMs, however, can result in failed Oracle Clusterware and Oracle Database installation attempts. For the purpose of this article, select the radio button [Customize now] and click [Next] to continue. This is where you pick the packages to install. Most of the packages required for the Oracle software are grouped into "Package Groups" (i.e. Application -> Editors). Since these nodes will be hosting the Oracle Clusterware and Oracle RAC software, verify that at least the following package groups are selected for install. For many of the Linux package groups, not all of the packages associated with that group get selected for installation. (Note the "Optional packages" button after selecting a package group.) So although the package group gets selected for install, some of the packages required by Oracle do not get installed. In fact, there are some packages that are required by Oracle that do not belong to any of the available package groups (i.e. libaio-devel). Not to worry. A complete list of required packages for Oracle Clusterware 10g and Oracle RAC 10g for CentOS 5 will be provided at the end of this section. These packages will need to be manually installed from the CentOS CDs after the operating system install. For now, install the following package groups:
 

 

Desktop Environments o GNOME Desktop Environment Applications o Editors o Graphical Internet o Text-based Internet Development o Development Libraries o Development Tools o Legacy Software Development Servers o Server Configuration Tools Base System o Administration Tools o Base o Java o Legacy Software Support o System Tools o X Window System

In addition to the above packages, select any additional packages you wish to install for this node keeping in mind to NOT de-select any of the "default" RPM packages. After selecting the packages to install click [Next] to continue. About to Install This screen is basically a confirmation screen. Click [Next] to start the installation. If you are installing CentOS using CDs, you will be asked to switch CDs during the installation process depending on which packages you selected. Congratulations And that's it. You have successfully installed CentOS on the first node (linux1). The installer will eject the CD/DVD from the CD-ROM drive. Take out the CD/DVD and click [Reboot] to reboot the system. Post Installation Wizard Welcome Screen When the system boots into CentOS Linux for the first time, it will prompt you with another Welcome screen for the "Post Installation Wizard". The post installation wizard allows you to make final O/S configuration settings. On the "Welcome" screen, click [Forward] to continue. Firewall On this screen, make sure to select the [Disabled] option and click [Forward] to continue. You will be prompted with a warning dialog about not setting the firewall. When this occurs, click [Yes] to continue. SELinux On the SELinux screen, choose the [Disabled] option and click [Forward] to continue. You will be prompted with a warning dialog warning that changing the SELinux setting will require rebooting the system so the entire file system can be relabeled. When this occurs, click [Yes] to acknowledge a reboot of the system will occur after firstboot (Post Installation Wizard) is completed. Kdump Accept the default setting on the Kdump screen (disabled) and click [Forward] to continue. Date and Time Settings Adjust the date and time settings if necessary and click [Forward] to continue. Create User

Create any additional (non-oracle) operating system user accounts if desired and click [Forward] to continue. For the purpose of this article, I will not be creating any additional operating system accounts. I will be creating the "oracle" user account during the Oracle database installation later in this guide. If you chose not to define any additional operating system user accounts, click [Continue] to acknowledge the warning dialog. Sound Card This screen will only appear if the wizard detects a sound card. On the sound card screen click [Forward] to continue. Additional CDs On the "Additional CDs" screen click [Finish] to continue. Reboot System Given we changed the SELinux option (to disabled), we are prompted to reboot the system. Click [OK] to reboot the system for normal use. Login Screen After rebooting the machine, you are presented with the login screen. Log in using the "root" user account and the password you provided during the installation.

Perform the same installation on the second node After completing the Linux installation on the first node, repeat the above steps for the second node (linux2). When configuring the machine name and networking, ensure to configure the proper values. For my installation, this is what I configured for linux2: First, make sure that each of the network devices are checked to [Active on boot]. The installer may choose to not activate eth1. Second, [Edit] both eth0 and eth1 as follows. Verify that the option "Enable IPv4 support" is selected. Click off the option for "Use dynamic IP configuration (DHCP)" and configure a static IP address and Netmask for your environment. Click off the option to "Enable IPv6 support". You may choose to use different IP addresses for both eth0 and eth1 that I have documented in this guide and that is OK. Put eth1 (the interconnect) on a different subnet than eth0 (the public network): eth0: - Check ON the option to [Enable IPv4 support] - Check OFF the option to [Use dynamic IP configuration (DHCP)] - (select Manual configuration) IPv4 Address: 192.168.1.101

several will be missing either because they were considered optional within the package group or simply didn't exist in any package group! The packages listed in this section (or later versions) are required for Oracle Clusterware 10g Release 2 and Oracle RAC 10g Release 2 running on the CentOS 5 or Red Hat Enterprise Linux 5 platform.Check OFF the option to [Enable IPv6 support] Continue by setting your hostname manually.255.0.Check OFF the option to [Enable IPv6 support] eth1: .1-52 gcc-c++-4.125-3. the next step is to verify and install all packages (RPMs) required by both Oracle Clusterware and Oracle RAC. verify the software requirements documented in this section before starting the Oracle installs.1-52 .255. Install Required Linux Packages for Oracle RAC Install the following required Linux packages on both Oracle RAC nodes in the cluster! After installing CentOS Linux. Although many of the required packages for Oracle were installed in the section Install the Linux Operating System.0 .1.6-2.(select Manual configuration) IPv4 Address: 192.168.101 Prefix (Netmask): 255.el5 elfutils-libelf-devel-0.el5 compat-libstdc++-296-2.125 gcc-4.0 .Check OFF the option to [Use dynamic IP configuration (DHCP)] . To ensure that these checks complete successfully.96-138 compat-libstdc++-33-3.255.17. The Oracle Universal Installer (OUI) performs checks on your machine during installation to verify that it meets the appropriate operating system package requirements.Check ON the option to [Enable IPv4 support] . Finish this dialog off by supplying your gateway and DNS servers. I used "linux2" for the second node.2.50.Prefix (Netmask): 255.255.1. 32-bit (x86) Installations        binutils-2.2.3-61 elfutils-libelf-0.

1 openmotif-2.3.2.1-52 libstdc++-4. the RPM command will simply ignore the install and print a warning message to the console that the package is already installed.2. This article does not cover the installation of Oracle demos.5-12 glibc-devel-2.106 libgcc-4.11 unixODBC-devel-2.* rpm -Uvh elfutils-libelf-0.* rpm -Uvh gcc-4.3 sysstat-7.1.* rpm -Uvh glibc-headers-2.1-52.* rpm -Uvh glibc-common-2.[CD #1] mkdir -p /media/cdrom mount -r /dev/cdrom /media/cdrom cd /media/cdrom/CentOS rpm -Uvh binutils-2. an easier method is to run the rpm -Uvh PackageName command from the five CDs as follows.* cd / eject # From CentOS 5.1.* rpm -Uvh unixODBC-2.0.3 (x86).* rpm -Uvh libstdc++-devel-4.106 libaio-devel-0.1.0 unixODBC-2.* rpm -Uvh make-3.11 Note that the openmotif RPM packages are required to install Oracle demos.* . While it is possible to query each individual package to determine which ones are missing and need to be installed.2.5-12 libaio-0.* rpm -Uvh glibc-2. # From CentOS 5.1 libstdc++-devel-4.0.e15 libXp-1.5-12 glibc-headers-2.[CD #2] mount -r /dev/cdrom /media/cdrom cd /media/cdrom/CentOS rpm -Uvh elfutils-libelf-devel-0.(x86) CDs.* rpm -Uvh gcc-c++-4.81-1. and CD #3 on the CentOS 5 .5-12 glibc-common-2.* rpm -Uvh libaio-0.               glibc-2.3.* rpm -Uvh libgcc-4.* rpm -Uvh glibc-devel-2. CD #2. Each of the packages listed above can be found on CD #1.0-8 make-3.* rpm -Uvh libstdc++-4. For packages that already exist and are up to date.3 (x86) .

5-12 libaio-0.3 (x86) .17.1 (32 bit) libstdc++-4.6-2.2.106 (32 bit) libaio-devel-0.5-12 glibc-devel-2.1.e15 .* rpm -Uvh unixODBC-devel-2.5-12 glibc-2.1.125 gcc-4.el5 compat-libstdc++-296-2.1.[CD #3] mount -r /dev/cdrom /media/cdrom cd /media/cdrom/CentOS rpm -Uvh compat-libstdc++-296* rpm -Uvh compat-libstdc++-33* rpm -Uvh libaio-devel-0.* rpm -Uvh openmotif-2.125-3.3-61 (32 bit) elfutils-libelf-0.96-138 compat-libstdc++-33-3.3-61 compat-libstdc++-33-3.3.1 libgcc-4.1 libstdc++-4.el5 elfutils-libelf-0.3.1 glibc-2.1.5-12 (32 bit) glibc-headers-2.5-12 (32 bit) glibc-common-2.* cd / eject 64-bit (x86_64) Installations                        binutils-2.* rpm -Uvh libXp-1.* cd / eject # From CentOS 5.1 gcc-c++-4.106 libgcc-4.50.125-3.cd / eject # From CentOS 5.1 (32 bit) libstdc++-devel-4.5 glibc-devel-2.[CD #4] mount -r /dev/cdrom /media/cdrom cd /media/cdrom/CentOS rpm -Uvh sysstat-7.1-52.3 (x86) .106 libaio-0.el5 (32 bit) elfutils-libelf-devel-0.1.1.1.3.0.2.

[CD #2] mount -r /dev/cdrom /media/cdrom cd /media/cdrom/CentOS rpm -Uvh elfutils-libelf-devel-0.* rpm -Uvh libgcc-4.* rpm -Uvh gcc-c++-4.2.2. For packages that already exist and are up to date.0.* rpm -Uvh unixODBC-2.* cd / eject # From CentOS 5.[CD #1] mkdir -p /media/cdrom mount -r /dev/cdrom /media/cdrom cd /media/cdrom/CentOS rpm -Uvh binutils-2. CD #3.1 openmotif-2.* rpm -Uvh glibc-common-2.3 (x86_64) .0-8 libXp-1.11 (32 bit) unixODBC-devel-2.        libXp-1. an easier method is to run the rpm -Uvh PackageName command from the five CDs as follows.81-1.* rpm -Uvh gcc-4. # From CentOS 5.* rpm -Uvh libaio-0. the RPM command will simply ignore the install and print a warning message to the console that the package is already installed.3 (x86_64).[CD #3] mount -r /dev/cdrom /media/cdrom cd /media/cdrom/CentOS rpm -Uvh compat-libstdc++-296* rpm -Uvh compat-libstdc++-33* .3 sysstat-7.0. Each of the packages listed above can be found on CD #1.0 unixODBC-2.* rpm -Uvh glibc-devel-2.2.2.3 (x86_64) .0.* rpm -Uvh glibc-headers-2.* rpm -Uvh glibc-2.* rpm -Uvh libstdc++-4. This article does not cover the installation of Oracle demos.11 unixODBC-2. and CD #4 on the CentOS 5 (x86_64) CDs.0-8 (32 bit) make-3.* rpm -Uvh elfutils-libelf-0.* cd / eject # From CentOS 5.11 Note that the openmotif RPM packages are required to install Oracle demos.* rpm -Uvh libstdc++-devel-4.* rpm -Uvh make-3. CD #2. While it is possible to query each individual package to determine which ones are missing and need to be installed.

we need to configure the network on both Oracle RAC nodes for access to the public network as well as their private interconnect.* unixODBC-devel-2.* # From CentOS 5. it is important to not skip this section as it contains critical steps that are required for a successful RAC environment. the interconnect should be at least gigabit (or more) and only be used by Oracle as well as having the network storage server (Openfiler) on a separate gigabit network. Both of the Oracle RAC nodes should have one static IP address for the public network and one static IP address for the private cluster interconnect. Configuring Public and Private Network In our two node example. For a production RAC implementation.[CD #4] mount -r /dev/cdrom /media/cdrom cd /media/cdrom/CentOS rpm -Uvh sysstat-7.* openmotif-2.3 (x86_64) . You must have one network interface for the public network and another network interface for the private interconnect. you need static IP addresses! The private interconnect should only be used by Oracle to transfer Cluster Manager and Cache Fusion related data along with data for the network storage server (Openfiler). Note that Oracle does not support using the public network interface for the interconnect.* cd / eject Network Configuration Perform the following network configuration on both Oracle RAC nodes in the cluster! Although we configured several of the network settings during the installation of CentOS.* libXp-1. We now need to configure the /etc/hosts file as well as adjusting several of the network settings for the interconnect. Introduction to Network Settings During the Linux O/S install we already configured the IP address and host name for both of the Oracle RAC nodes. Do not use DHCP naming for the public IP address or the interconnects.rpm -Uvh rpm -Uvh rpm -Uvh rpm -Uvh cd / eject libaio-devel-0. .

localdomain localhost # Public Network .168.(linux1) Device IP Address Subnet Gateway Purpose eth0 192.168.2.255.168.100 255.(eth0) 192.195 openfiler1-priv Oracle RAC Node 2 .1.255. ::1 localhost6.0 192.168.168.1.195 openfiler1 192.1.0.The easiest way to configure network settings in CentOS is with the program Network Configuration.255.100 linux1 .1.1 localhost. This application can be started from the command-line as the "root" user account as follows: # /usr/bin/system-config-network & Do not use DHCP naming for the public IP address or the interconnects . Our example configuration will use the following settings: Oracle RAC Node 1 .255.168.255.168.(eth1) 192.1.we need static IP addresses! Using the Network Configuration application.0 192.0.2.168.168.localdomain6 localhost6).101 255.1.255.168.100 linux1 192.(eth0) 192.2.0 Connects linux1 (interconnect) to linux2 (linux2priv) /etc/hosts 127.1 Connects linux1 to the public network eth1 192.0.101 linux2 # Private Interconnect .(eth0:1) 192.(linux2) Device IP Address Subnet Gateway Purpose eth0 192.168.2.0 priv) /etc/hosts 127.1.1 localhost.101 255.255.1.(eth1) 192.2.168.100 linux1-priv 192.localdomain localhost # Public Network .168.1.1 Connects linux2 to the public network Connects linux2 (interconnect) to linux1 (linux1eth1 192.168. Notice that the /etc/hosts settings are the same for both nodes and that I removed any entry that has to do with IPv6 (for example. you need to configure both NIC devices as well as the /etc/hosts file.100 255.200 linux1-vip 192.201 linux2-vip # Private Storage Network for Openfiler .168. Both of these tasks can be completed using the Network Configuration GUI.1.0.101 linux2-priv # Public Virtual IP (VIP) addresses .255.

192.1. The public virtual IP addresses will be configured automatically by Oracle Clusterware when you run the Oracle Universal Installer. All of this will be explained much later in this article! In the screen shots below. this is the Host Name/IP Address that will be configured in the client(s) tnsnames.168. All virtual IP addresses will be activated when the srvctl start nodeapps -n <node_name> command is run.168.168.1. This is the Host Name/IP Address that will be configured in the client(s) tnsnames.(eth1) 192.(eth1) 192.168.168.1.195 openfiler1 192.200 linux1-vip 192.195 openfiler1-priv Note that the virtual IP addresses only need to be defined in the /etc/hosts file (or your DNS) for both Oracle RAC nodes.2.ora file (more details later).168. only Oracle RAC Node 1 (linux1) is shown. Be sure to make all the proper network settings to both Oracle RAC nodes.101 linux2 # Private Interconnect .2.168.ora file for each Oracle Net Service Name. .1.101 linux2-priv # Public Virtual IP (VIP) addresses .201 linux2-vip # Private Storage Network for Openfiler . which starts Oracle's Virtual Internet Protocol Configuration Assistant (VIPCA).100 linux1-priv 192.2. Although I am getting ahead of myself.(eth0:1) 192.

Figure 2: Network Configuration Screen .Node 1 (linux1) .

Figure 3: Ethernet Device Screen .eth0 (linux1) .

eth1 (linux1) .Figure 4: Ethernet Device Screen .

0 inet6 addr: fe80::20e:cff:fe64:d1e5/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:11 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.168.0 b) Base address:0xddc0 Memory:fe9c0000-fe9e0000 .100 Bcast:192.1.100 Bcast:192.255.168.Figure 5: Network Configuration Screen .3 MiB) TX bytes:145612 (142. The following example is from linux1: # /sbin/ifconfig -a eth0 Link encap:Ethernet HWaddr 00:14:6C:76:5C:71 inet addr:192.255./etc/hosts (linux1) Once the network is configured.255.1 KiB) Interrupt:169 Base address:0xef00 eth1 Link encap:Ethernet HWaddr 00:0E:0C:64:D1:E5 inet addr:192.255 Mask:255.2. you can use the ifconfig command to verify everything is working.255.168.1.2.255 Mask:255.0 inet6 addr: fe80::214:6cff:fe76:5c71/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:3059 errors:0 dropped:0 overruns:0 frame:0 TX packets:1539 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:3460697 (3.0 b) TX bytes:782 (782.168.

With TAF successfully configured. Going one step further is making use of Transparent Application Failover (TAF). the client receives a TCP reset. Without using VIPs.0.0 b) TX bytes:0 (0. 2.0 b) sit0 About Virtual IP Why do we have a Virtual IP (VIP) in 10g? Why does it just return a dead connection when its primary node fails? It's all about availability of the application. In the case of connect.8 MiB) Link encap:IPv6-in-IPv4 NOARP MTU:1480 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0. the next address in tnsnames is used.8 MiB) TX bytes:1991946 (1. When a node fails. In the case of SQL.0. two things happen.0. this usually causes them to see errors on their connections to the old address. rather than waiting on a very long TCP/IP time-out (~10 minutes).(TAF)". Source . you don't really have a good HA solution without using VIPs. it is possible to completely avoid ORA-3113 errors all together! TAF will be discussed in more detail in the section "Transparent Application Failover . If the machine name is listed in the in the loopback address entry as below: . clients connected to a node that died will often wait a 10 minute TCP timeout period before getting an error. which will send error RST packets back to the clients.1) Confirm the RAC Node Name is Not Listed in Loopback Address Ensure that the node names (linux1 or linux2) are not included for the loopback address in the /etc/hosts file.Metalink: "RAC Frequently Asked Questions" (Note:220970. For directly connected clients. This results in the clients getting errors immediately.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:1764 errors:0 dropped:0 overruns:0 frame:0 TX packets:1764 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:1991946 (1. The new node re-arps the world indicating a new MAC address for the address.lo Link encap:Local Loopback inet addr:127. This means that when the client issues SQL to the node that is now down. or traverses the address list while connecting. 1. this is ORA-3113.1 Mask:255. Subsequent packets sent to the VIP go to the new node. When this occurs.0. As a result. the VIP associated with it is supposed to be automatically failed over to some other node.

0. All rights reserved.localdomain localhost it will need to be removed as shown below: If the RAC node name is listed for the loopback address. The ONS log file will contain lines similar to the following: Oracle Database 10g CRS Release 10.localdomain localhost If an entry does not exist for localhost in the /etc/hosts file.0.0. .localdomain and localhost are included for the loopback address in the /etc/hosts file for each of the Oracle RAC nodes: 127. you will receive the following error during the RAC installation: ORA-00603: ORACLE server session terminated by fatal error or ORA-29702: error occurred in Cluster Group Service operation Confirm localhost is defined in the /etc/hosts file for the loopback address Ensure that the entry for localhost. Oracle Clusterware will be unable to start the application resources — notably the ONS process.ons'. Oracle strongly suggests to adjust the default and maximum receive buffer size (SO_RCVBUF socket option) to 1024KB and the default and maximum send buffer size ( SO_SNDBUF socket option) to 256KB.2.0.0.1 linux1 localhost..1.linux1.ons.729: [ RACG][3086871296][13316][3086871296][ora.0. For example: CRS-0215 could not start resource 'ora.0.log" for more details. 2007-04-14 13:10:02.1 localhost. 2005 Oracle.linux1.ons]: Failed to get IP for localhost (1) Failed to get IP for localhost (1) Failed to get IP for localhost (1) onsctl: ons failed to start .0. The error would indicate "Failed to get IP for localhost" and will be written to the log file for ONS. Adjusting Network Settings With Oracle 9.127.localdomain localhost localhost.1 127. such as Cache Fusion and Cluster Manager buffer transfers between instances within the RAC cluster.. Check log file "/u01/app/crs/log/linux1/racg/ora.0 Production Copyright 1996. Oracle makes use of UDP as the default protocol on Linux for inter-process communication (IPC).1 and later.2.linux1.

rmem_max=1048576 net.wmem_max=262144 net.core.rmem_default=1048576 net.rmem_default=1048576 # +---------------------------------------------------------+ # | Maximum setting in bytes of the socket "receive" buffer | . You should now make the above changes permanent (for each reboot) by adding the following lines to the /etc/sysctl. The receive buffer cannot overflow because the peer is not allowed to send data beyond the buffer size window. This could cause the sender to overwhelm the | | receiver.core.core. | +---------------------------------------------------------+ # +---------------------------------------------------------+ # | Default setting in bytes of the socket "receive" buffer | # | which may be set by using the SO_RCVBUF socket option.0.core. potentially causing the sender to overwhelm the receiver. and the default | | and maximum send buffer size (SO_SNDBUF socket option) | | to 256KB.The receive buffers are used by TCP and UDP to hold received data until it is read by the application.wmem_default=262144 net.1 and onwards.rmem_default = 1048576 # sysctl -w net. Oracle strongly suggests to | | adjust the default and maximum receive buffer size | | (SO_RCVBUF socket option) to 1024KB. This means that datagrams will be discarded if they don't fit in the socket receive buffer. The default and maximum window size can be changed in the /proc file system without reboot: # sysctl -w net.wmem_max = 262144 The above commands made the changes to the already running OS.rmem_max = 1048576 # sysctl -w net. The receive buffer cannot overflow because | | the peer is not allowed to send data beyond the buffer | | size window.core.core. This means that datagrams will be | | discarded if they don't fit in the socket receive | | buffer. Oracle now makes use | | of UDP as the default protocol on Linux for | | inter-process communication (IPC).core.core.2.conf file for both nodes in your RAC cluster: # # # # # # # # # # # # # # # # # # # # +---------------------------------------------------------+ | ADJUSTING NETWORK SETTINGS | +---------------------------------------------------------+ | With Oracle 9. such as Cache Fusion | | and Cluster Manager buffer transfers between instances | | within the RAC cluster. | # +---------------------------------------------------------+ net. The receive buffers are used by TCP and UDP | | to hold received data until it is read by the | | application.wmem_default = 262144 # sysctl -w net.core.

Flushing firewall rules: [ OK ] 7.or to simply have the firewall option turned off. the solution is to remove the udp ICMP (iptables) rejection rule .core. | # +---------------------------------------------------------+ net. Setting chains to policy ACCEPT: filter [ Unloading iptables modules: [ OK ] OK ] . The following commands should be executed as the root user account: 1. If the firewall option is stopped (like it is in my example below) you do not have to proceed with the following steps.wmem_max=262144 Check and turn off UDP ICMP rejections: During the Linux installation process. If UDP ICMP is blocked or rejected by the firewall.# | which may be set by using the SO_RCVBUF socket option.wmem_default=262144 # +---------------------------------------------------------+ # | Maximum setting in bytes of the socket "send" buffer | # | which may be set by using the SO_SNDBUF socket option.d/iptables stop 5.d/init. # /etc/rc.log file: 08/29/2005 22:17:19 oac_init:2: Could not connect to server. Check to ensure that the firewall option is turned off.core.d/init.d/iptables status Firewall is stopped. 3. | # +---------------------------------------------------------+ net. I indicated to not configure the firewall option. | # +---------------------------------------------------------+ net.rmem_max=1048576 # +---------------------------------------------------------+ # | Default setting in bytes of the socket "send" buffer | # | which may be set by using the SO_SNDBUF socket option. When the Oracle Clusterware process fails. 2. # /etc/rc. This has burned me several times so I like to do a double-check that the firewall option is not configured and to ensure udp ICMP filtering is turned off. By default the option to configure a firewall is selected by the installer. If the firewall option is operating you will need to first manually disable UDP ICMP rejections: 4.core. the Oracle Clusterware software will crash after several minutes of running. you will have something similar to the following in the <machine_name>_evmocr. 6. The Oracle Clusterware software will then start to operate normally and not crash. clsc retcode = 9 08/29/2005 22:17:19 a_init:12!: Client init unsuccessful : [32] ibctx:1:ERROR: INVALID FORMAT proprinit:problem reading the bootblock or superbloc 22 When experiencing this type of error.

the next step is to install the Openfiler software to the network storage server (openfiler1). I could have made an extra partition on the 500GB internal SATA disk for the iSCSI target. LVM2. version 2. NFS. Openfiler combines these ubiquitous technologies into a small. but decided to make use of the faster SCSI disk for this example. Please be aware that any type of hard disk (internal or external) should work for database storage as long as it can be recognized by the network storage server (Openfiler) and has adequate space. Then. easy to manage solution fronted by a powerful web-based management interface. The Openfiler server will be configured to use this volume group for iSCSI based storage and will be used in our Oracle RAC 10g configuration to store the shared files required by Oracle Clusterware as well as all Oracle ASM volumes.3 (Final Release) for either x86 or x86_64 depending on your hardware architecture. Samba. Openfiler supports CIFS. For example. HTTP/DAV. Later in this article. you will then need to burn the ISO image to CD. however. FTP.8.openfiler.com/ Download Openfiler Use the links below to download Openfiler NAS/SAN Appliance. please visit their website at http://www. the network storage server will be configured as an iSCSI storage device for all Oracle RAC 10g shared storage requirements. we will only be making use of its iSCSI capabilities to implement an inexpensive SAN for the shared storage components required by Oracle RAC 10g. Powered by rPath Linux. Linux NFS and iSCSI Enterprise Target. The entire software stack interfaces with open source applications such as Apache. Openfiler is a free browser-based network storage management utility that delivers filebased Network Attached Storage (NAS) and block-based Storage Area Networking (SAN) in a single framework. . This example uses x86_64. A second internal 73GB 15K SCSI hard disk will be configured as a single "Volume Group" that will be used for all shared disk storage requirements. to turn UDP ICMP rejections off for next server reboot (which should always be turned off): # chkconfig iptables off Install Openfiler Perform the following installation on the network storage server (openfiler1)! With the network configured on both Oracle RAC nodes. ext3. To learn more about Openfiler. The operating system and Openfiler application will be installed on one internal SATA disk. After downloading Openfiler.

The only manual change required was for configuring the local network settings.com/docs/. . After the reboot.openfiler-2. I opted to install Openfiler with all default options. I would suggest.3-x86_64-disc1. any external hard drives (if connected) will be discovered by the Openfiler server. here are just two (of many) software packages that can be used: UltraISO Magic ISO Maker Install Openfiler This section provides a summary of the screens used to install the Openfiler software. there are many options for burning the ISO image (ISO file) to a CD. that the instructions I have provided below be used for this Oracle RAC 10g configuration. please visit http://www. services and drivers are started and recognized.3-x86-disc1.iso Processor Type Size SHA1SUM x86 322 MB 8343a9ae07a0590d88484b3dac117c843626bf9c openfiler-2. Before installing the Openfiler software to the network storage server. For more detailed installation instructions.openfiler. If you are not familiar with this process and do not have the required software to burn images to CD. you should have both NIC interfaces (cards) installed and any external hard drives connected and turned on (if you will be using external hard drives). Once the install has completed. You may already be familiar with and have the proper software to burn images to CD. however. For the purpose of this article. the server will reboot to make sure all required components.iso Processor Type Size SHA1SUM x86_64 336 MB dae55d478ade41389cff3718579745b1a95f6d41 If you are downloading the above ISO file to a MS Windows machine.

hit [Enter] to start the installation process. Keyboard Configuration The next screen prompts you for the Keyboard settings. I also keep the checkbox [Review (and modify if needed) the partitions created] selected. Welcome to Openfiler NSA At the welcome screen. After several seconds. monitor. click [Next] to continue. I selected ONLY the 500GB SATA internal hard drive [sda] for the operating system and Openfiler application installation. insert the CD into the network storage server (openfiler1 in this example). the next screen will ask if you want to "remove" or "keep" old partitions. Automatic Partitioning If there were a previous installation of Linux on this machine. tab over to [Skip] and hit [Enter]. Click [Yes] to acknowledge this warning. the media burning software would have warned us. Select [Automatically partition] and click [Next] continue. Boot Screen The first screen is the Openfiler boot screen. Select the option to [Remove all partitions on this system]. I de-selected the 73GB SCSI internal hard drive since this disk will be used exclusively in the next section to create a single "Volume Group" that will be used for all iSCSI based shared disk storage requirements for Oracle RAC 10g. You will then be prompted with a dialog window asking if you really want to remove all partitions. Make the appropriate selection for your configuration. the installer should then detect the video card. and answer the installation screen prompts as noted below. If there were any errors. Partitioning . The installer then goes into GUI mode. power it on. Click [Next] to continue. Media Test When asked to test the CD media. Although the official Openfiler documentation suggests to use Manual Partitioning. I opted to use "Automatic Partitioning" given the simplicity of my example configuration. At the boot: prompt. For my example configuration.After downloading and burning the Openfiler ISO image (ISO file) to CD. and mouse. Disk Partitioning Setup The next screen asks whether to perform disk partitioning using "Automatic Partitioning" or "Manual Partitioning with Disk Druid".

0 Continue by setting your hostname manually. You may choose to use different IP addresses for both eth0 and eth1 and that is OK.The installer will then allow you to view (and modify if needed) the disk partitions it automatically chose for hard disks selected in the previous screen.255.Check OFF the option to [Configure using DHCP] .Leave the [Activate on boot] checked ON . make sure that each of the network devices are checked to [Active on boot]. In this example. In almost all cases. In the next section. The installer may choose to not activate eth1 by default.255.2. [Edit] both eth0 and eth1 as follows. the installer will choose 100MB for /boot.255.Netmask: 255.255. Make the appropriate selection for your location. I will create the required partition for this particular hard disk. .1. First. the installer found the 73GB SCSI internal hard drive as /dev/sdb.168. and the rest going to the root (/) partition for that disk (or disks). an adequate amount of swap.Leave the [Activate on boot] checked ON . This screen should have successfully detected each of the network devices. configure eth1 (the storage network) to be on the same subnet you configured for eth1 on linux1 and linux2: eth0: . I will "Delete" any and all partitions on this drive (there was only one.IP Address: 192. For my example configuration. For now.Netmask: 255.Check OFF the option to [Configure using DHCP] .0 eth1: . /dev/sdb1).IP Address: 192. Time Zone Selection The next screen allows you to configure your time zone information. Second. The installer will also show any other internal hard disks it discovered. however.195 . I am satisfied with the installers recommended partitioning for /dev/sda.195 .168. I used a hostname of "openfiler1". Set Root Password Select a root password and click [Next] to continue. Network Configuration I made sure to install all NIC interfaces (cards) in the network storage server before starting the Openfiler installation. You must. Finish this dialog off by supplying your gateway and DNS servers.

Congratulations And that's it. For example: https://openfiler1. This allows convenient name resolution when testing the network for the cluster. create a new volume group.About to Install This screen is basically a confirmation screen. The default administration login credentials for Openfiler are:   Username: openfiler Password: password The first page the administrator sees is the [Status] / [System Information] screen.info:446/ From the Openfiler Storage Control Center home page. log in as an administrator. To use Openfiler as an iSCSI storage server. I typically copy the contents of the /etc/hosts file from one of the Oracle RAC nodes to the new Openfiler server. you should now be presented with a text login screen and the URL(s) to use for administering the Openfiler server. Configure iSCSI Volumes using Openfiler Perform the following configuration tasks on the network storage server (openfiler1)! Openfiler administration is performed using the Openfiler Storage Control Center — a browser based tool over an https connection on port 446. Services . and finally. The installer will eject the CD from the CD-ROM drive. identify and partition the physical storage. configure network access. we have to perform six major tasks. Take out the CD and click [Reboot] to reboot the system. You have successfully installed Openfiler on the network storage server. If everything was successful after the reboot.idevelopment. create new iSCSI targets for each of the logical volumes. Click [Next] to start the installation. set up iSCSI services. create all logical volumes. Modify /etc/hosts File on Openfiler Server Although not mandatory.

The ietd program implements the user level part of iSCSI Enterprise Target software for building an iSCSI storage system on Linux..To control services.. . click on the 'Enable' link under the 'iSCSI target server' service name. we should be able to SSH into the Openfiler server and see the iscsi-target service running: [root@openfiler1 ~]# service iscsi-target status ietd (pid 3839) is running. we use the Openfiler Storage Control Center and navigate to [Services] / [Manage Services]: Figure 6: Enable iSCSI Openfiler Service To enable the iSCSI service. the 'iSCSI target server' status should change to 'Enabled'. With the iSCSI target enabled. After that.

255. we will want to add both Oracle RAC nodes individually rather than allowing the entire 192. Note that iSCSI volumes will be created later on in this section. Lastly. configuring network access is accomplished using the Openfiler Storage Control Center by navigating to [System] / [Network Setup]. When entering each of the Oracle RAC nodes. use a subnet mask of 255. when entering the actual node in the 'Network/Host' field. As a convention when entering nodes.2. Also note that this step does not actually grant the appropriate permissions to the iSCSI volumes required by both Oracle RAC nodes. That will be accomplished later in this section by updating the ACL for each new logical volume. note that the 'Name' field is just a logical name used for reference only. It is important to remember that you will be entering the IP address of the private network (eth1) for each of the RAC nodes in the cluster. when entering actual hosts in our Class C network. For the purpose of this article. The following image shows the results of adding both Oracle RAC nodes: Figure 7: Configure Openfiler Network Access for Oracle RAC Nodes .Network Access Configuration The next step is to configure network access in Openfiler to identify both Oracle RAC nodes ( linux1 and linux2) that will need to access the iSCSI volumes through the storage (private) network. The "Network Access Configuration" section (at the bottom of the page) allows an administrator to setup networks and/or hosts that will be allowed to access resources exported by the Openfiler appliance.0 network to have access to Openfiler resources. As in the previous section.255. I simply use the node name defined for that IP address.168. Next.255. always use its IP address even though its host name may already be defined in your /etc/hosts file or DNS.

storage arrays. or ANY other storage can be connected to the Openfiler server and served to the clients. we are presented with the options to 'Edit' or 'Create' a partition. In our case. To see this and to start the process of creating our iSCSI volumes. navigate to [Volumes] / [Block Devices] from the Openfiler Storage Control Center: Figure 8: Openfiler Physical Storage . By clicking on the /dev/sdb link. external FireWire drives. external USB drives. Storage devices like internal IDE/SATA/SCSI/SAS disks. Once these devices are discovered at the OS level.Physical Storage In this section. Openfiler Storage Control Center can be used to set up and manage all of that storage. Since we will be creating a single primary partition that spans the entire disk. we have a 73GB internal SCSI hard drive for our shared storage needs. most of the options can be left to their default . we will be creating the five iSCSI volumes to be used as shared storage by both of the Oracle RAC nodes in the cluster. This involves multiple steps that will be performed on the internal 73GB 15K SCSI hard disk connected to the Openfiler server.Block Device Management Partitioning the Physical Disk The first step we will perform is to create a single primary partition on the /dev/sdb internal hard disk. On the Openfiler server this drive is seen as /dev/sdb (MAXTOR ATLAS15K2_73SCA).

setting where the only modification would be to change the 'Partition Type' from 'Extended partition' to 'Physical volume'. Here are the values I specified to create the primary partition on /dev/sdb: Mode: Primary Partition Type: Physical volume Starting Cylinder: 1 Ending Cylinder: 8924 The size now shows 68.36 GB. . This results in a new partition (/dev/sdb1) on our internal hard disk: Figure 9: Partition the Physical Volume Volume Group Management The next step is to create a Volume Group. We will be creating a single volume group named rac1 that contains the newly created primary partition. To accept that we click on the "Create" button.

From the Openfiler Storage Control Center, navigate to [Volumes] / [Volume Groups]. There we would see any existing volume groups, or none as in our case. Using the Volume Group Management screen, enter the name of the new volume group (rac1), click on the checkbox in front of /dev/sdb1 to select that partition, and finally click on the 'Add volume group' button. After that we are presented with the list that now shows our newly created volume group named "rac1":

Figure 10: New Volume Group Created

Logical Volumes We can now create the five logical volumes in the newly created volume group (rac1). From the Openfiler Storage Control Center, navigate to [Volumes] / [Add Volume]. There we will see the newly created volume group (rac1) along with its block storage statistics. Also available at the bottom of this screen is the option to create a new volume in the selected volume group - (Create a volume in "rac1"). Use this screen to create the following five logical (iSCSI) volumes. After creating each logical volume, the application will point

you to the "Manage Volumes" screen. You will then need to click back to the "Add Volume" tab to create the next logical volume until all five iSCSI volumes are created: iSCSI / Logical Volumes Volume Name Volume Description racdb-crs racdb-asm1 racdb-asm2 racdb-asm3 racdb-asm4 racdb - Oracle Clusterware racdb - ASM Volume 1 racdb - ASM Volume 2 racdb - ASM Volume 3 racdb - ASM Volume 4 Required Space (MB) Filesystem Type 2,048 16,984 16,984 16,984 16,984 iSCSI iSCSI iSCSI iSCSI iSCSI

In effect we have created five iSCSI disks that can now be presented to iSCSI clients ( linux1 and linux2) on the network. The "Manage Volumes" screen should look as follows:

Figure 11: New Logical (iSCSI) Volumes

iSCSI Targets

com. the following table lists the new iSCSI target names (the Target IQN) and which iSCSI logical volume it will be mapped to: iSCSI Target / Logical Volume Mappings Target IQN iqn.openfiler:racdb.one for each of the iSCSI logical volumes.com. grant both of the Oracle RAC nodes access to the new iSCSI target.openfiler:racdb. an iSCSI target will need to be created for each of these five volumes. A default value is automatically generated for the name of the new iSCSI target (better known as the "Target IQN").openfiler:racdb.ASM Volume 3 racdb . there will be a one-to-one mapping between an iSCSI logical volume and an iSCSI target.com. Verify the grey sub-tab "Target Configuration" is selected. Create New Target IQN From the Openfiler Storage Control Center.2006-01.ASM Volume 4 iqn.2006-01. navigate to [Volumes] / [iSCSI Targets].2006-01. Each iSCSI logical volume will be mapped to a specific iSCSI target and the appropriate network access permissions to that target will be granted to both Oracle RAC nodes. An example Target IQN is "iqn. The example below illustrates the three steps required to create a new iSCSI target by creating the Oracle Clusterware / racdbcrs target (iqn.2006-01. For the purpose of this article.2006-01.asm2 racdb-asm2 iqn.2006-01.com. Before an iSCSI client can have access to them. the universal name for the new iSCSI target).openfiler:tsn. create a unique Target IQN (basically.com. map one of the iSCSI logical volumes created in the previous section to the newly created iSCSI target.ASM Volume 2 racdb .crs). This three step process will need to be repeated for each of the five new iSCSI targets listed in the table above.ASM Volume 1 racdb . This page allows you to create a new iSCSI target.2006-01.openfiler:racdb.asm1 racdb-asm1 iqn. Please note that this process will need to be performed for each of the five iSCSI logical volumes created in the previous section.asm4 racdb-asm4 We are now ready to create the five new iSCSI targets .ae4683b67fd3": .openfiler:racdb. There are three steps involved in creating and configuring an iSCSI target.crs iSCSI Volume Name Volume Description racdb-crs racdb .com.At this point we have five iSCSI logical volumes.com.asm3 racdb-asm3 iqn.Oracle Clusterware racdb .openfiler:racdb. however. and finally. For the purpose of this article.

For the purpose of this article. For the first iSCSI target (Oracle Clusterware / racdb-crs). I will modify the default Target IQN by replacing the string "tsn. This will create a new iSCSI target and then bring up a page that allows you to modify a number of settings for the new iSCSI target. none of settings for the new iSCSI target need to be changed.crs" as shown in Figure 13 below: Figure 13: Create New iSCSI Target : Replace Default Target IQN Once you are satisfied with the new Target IQN. click the "Add" button. LUN Mapping .Figure 12: Create New iSCSI Target : Default Target IQN I prefer to replace the last segment of the default Target IQN with something more meaningful.ae4683b67fd3" with "racdb.

Next. Locate the appropriate iSCSI logical volume (/dev/rac1/racdb-crs in this case) and click the "Map" button. You do not need to change any settings on this page. Your screen should look similar to Figure 14 after clicking the "Map" button for volume /dev/rac1/racdb-crs: Figure 14: Create New iSCSI Target : Map LUN Network ACL . use the pull-down menu to select the correct iSCSI target and hit the "Change" button. click on the grey sub-tab named "LUN Mapping" (next to "Target Configuration" sub-tab). If not. the next step is to map the appropriate iSCSI logical volumes to it. Under the "Target Configuration" sub-tab. verify the correct iSCSI target is selected in the section "Select iSCSI Target".After creating the new iSCSI target.

These are the two nodes that will need to access the new iSCSI targets through the storage (private) network.Before an iSCSI client can have access to the newly created iSCSI target. it needs to be granted the appropriate permissions. Click on the grey sub-tab named "Network ACL" (next to "LUN Mapping" sub-tab). . we configured network access in Openfiler for two hosts (the Oracle RAC nodes). change the "Access" for both hosts from 'Deny' to 'Allow' and click the 'Update' button: Figure 15: Create New iSCSI Target : Update Network ACL Go back to the Create New Target IQN section and perform these three tasks for the remaining four iSCSI logical volumes while substituting the values found in the "iSCSI Target / Logical Volume Mappings" table . We now need to grant both of the Oracle RAC nodes access to the new iSCSI target. Awhile back. Configure iSCSI Volumes on Oracle RAC Nodes Configure the iSCSI initiator on both Oracle RAC nodes in the cluster! Creating partitions. For the current iSCSI target. should only be executed on one of nodes in the RAC cluster. however.

linux1 and linux2. Unix. Apple Mac. load CD #1 into each of the Oracle RAC nodes and perform the following: # # # # # mount -r /dev/cdrom /media/cdrom cd /media/cdrom/CentOS rpm -Uvh iscsi-initiator-utils-* cd / eject Configure the iSCSI (initiator) service After verifying that the iscsi-initiator-utils package is installed on both Oracle RAC nodes. We will also configure the iscsi service to automatically start which logs into iSCSI targets needed at system startup. To determine if this package is installed (which in most cases. /dev/iscsi/asm1) for each of the iSCSI target names discovered using udev. the clients are two Linux servers. This is a change from previous versions of CentOS (4. The iSCSI software initiator will be configured to automatically log in to the network storage server (openfiler1) and discover the iSCSI volumes created in the previous section.3 includes the Open-iSCSI iSCSI software initiator which can be found in the iscsi-initiator-utils RPM. In this section we will be configuring the iSCSI software initiator on both of the Oracle RAC nodes. running CentOS 5.3.e.) for which iSCSI support (a driver) is available. Before we can do any of this. The software is included in the iscsi-initiator-utils package which can be found on CD #1. etc. we must first install the iSCSI initiator software! Installing the iSCSI (initiator) service With CentOS 5. Having a consistent local SCSI device name and which iSCSI target it maps to is required in order to know which volume (device) is to be used for OCFS2 and which volumes belong to ASM. however.x) which included the Linux iscsi-sfnet software driver developed as part of the Linux-iSCSI Project. CentOS 5.3. start the iscsid service and enable it to automatically start when the system boots. MS Windows. In our case. All iSCSI management tasks like discovery and logins will use the command-line interface iscsiadm which is included with Open-iSCSI. Starting iSCSI daemon: # chkconfig iscsid on # chkconfig iscsi on [ [ OK OK ] ] . the Open-iSCSI iSCSI software initiator does not get installed by default. perform the following on both Oracle RAC nodes: # rpm -qa --queryformat "%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})\n"| grep iscsi-initiatorutils If the iscsi-initiator-utils package is not installed.An iSCSI client can be any system (Linux. it will not be). # service iscsid start Turning off network shutdown. We will then go through the steps of creating persistent local SCSI device names (i.

1 iqn.2006-01.168.openfiler:racdb.2.2. Having a consistent local SCSI device name and which iSCSI target .2006-01.2006-01.168.com.2006-01.startup -v automatic # iscsiadm -m node -T iqn.crs -p 192.com.2006-01.openfiler:racdb.openfiler:racdb.2006-01.2006-01. we will go through the steps to create persistent local SCSI device names for each of the iSCSI target names.168.com.openfiler:racdb.195 --op update -n node.168.168.I believe this is required given the discovery (above) shows the targets using the IP address.168.195 --op update -n node.openfiler:racdb.startup -v automatic # iscsiadm -m node -T iqn.com.195:3260.openfiler:racdb.2.openfiler:racdb.168.openfiler:racdb.com.com.Now that the iSCSI service is started.195 -l iqn.com.195 -l iqn.168.com. Note that I had to specify the IP address and not the host name of the network storage server (openfiler1-priv) .168. This should be performed on both Oracle RAC nodes to verify the configuration is functioning properly: # iscsiadm -m discovery -t sendtargets -p openfiler1-priv 192. This needs to be run on both Oracle RAC nodes.2.168.crs -p 192.openfiler:racdb.195:3260.195:3260.openfiler:racdb.com.168.com.195 --op update -n node.2.openfiler:racdb.195 -l iqn.2.asm4 -p 192.195:3260.2.1 iqn.openfiler:racdb.168. perform the following on both Oracle RAC nodes: # iscsiadm -m node -T iqn.2006-01. use the iscsiadm command-line interface to discover all available targets on the network storage server.2.asm3 -p 192.2006-01.1 iqn.com.2.asm2 -p 192.1 iqn.2.asm3 192.asm1 -p 192.195 --op update -n node.openfiler:racdb.2006-01. This will be done using udev.2006-01.startup -v automatic Create Persistent Local SCSI Device Names In this section.crs Manually Log In to iSCSI Targets At this point the iSCSI initiator service has been started and each of the Oracle RAC nodes were able to discover the available targets from the network storage server. As with the manual log in process described above.asm3 -p 192. # # # # # iscsiadm iscsiadm iscsiadm iscsiadm iscsiadm -m -m -m -m -m node node node node node -T -T -T -T -T iqn.2006-01.startup -v automatic # iscsiadm -m node -T iqn.195:3260.asm2 -p 192.com.2.com.195 -l iqn.asm4 192.195 -l Configure Automatic Log In The next step is to ensure the client will automatically log in to each of the targets listed above when the machine is booted (or the iSCSI initiator service is started/restarted).2.asm4 -p 192.openfiler:racdb.2.168.asm1 192.asm2 192.195 --op update -n node.1 iqn.asm1 -p 192.2.com.2006-01.2006-01.startup -v automatic # iscsiadm -m node -T iqn.openfiler:racdb.2.168.168. The next step is to manually log in to each of the available targets which can be done using the iscsiadm command-line interface.2006-01.com.

.asm4 iqn.openfiler:racdb. print $9 " " $10 " " $11}') ip-192.com.2006-01.d/55-openiscsi.2006-01...2.2006-01./sdd ip-192.asm2 iqn. after a reboot it may be determined that the iSCSI target iqn.asm4-lun-0 -> .com.com.com.com.e.2006-01./sda ip-192.168. the target iqn.rules and contain only a single line of name=value pairs used to receive events we are interested in. It is therefore impractical to rely on using the local SCSI device name given there is no way to predict the iSCSI target mappings after a reboot..2006-01./. the client logging in to an iSCSI target).195:3260-iscsi-iqn.com./sde Using the output from the above listing. Rules that match may provide additional device information or specify a device node name and multiple symlink names and instruct udev to run additional programs (a SHELL script for example) as part of the device event handling process. ls -l *openfiler* | awk '{FS=" ".com./..com. The file will be named /etc/udev/rules. When udev receives a device event (for example./.2006-01. we can establish the following current mappings: Current iSCSI Target Name to local SCSI Device Name Mappings iSCSI Target Name iqn.195:3260-iscsi-iqn. udev provides a dynamic device directory using symbolic links that point to the actual device using a configurable set of rules.openfiler:racdb.168.asm1 iqn.rules on both Oracle RAC nodes: .168. The first step is to create a new rules file.crs SCSI Device Name /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde This mapping.195:3260-iscsi-iqn..openfiler:racdb. What we need is a consistent device name we can reference (i.com.asm2-lun-0 -> .asm1 may get mapped to /dev/sda.openfiler:racdb. it matches its configured rules against the available device attributes provided in sysfs to identify the device.2006-01.195:3260-iscsi-iqn.openfiler:racdb.com.2./sdc ip-192. may change every time the Oracle RAC node is rebooted.crs-lun-0 -> .asm1-lun-0 -> ./.. This is where the Dynamic Device Management tool named udev comes in.2006-01.openfiler:racdb.asm1 gets mapped to the local SCSI device /dev/sdd..openfiler:racdb.195:3260-iscsi-iqn./sdb ip-192.asm3 iqn. For example. /dev/iscsi/asm1) that will always point to the appropriate iSCSI target through reboots.2.2006-01. It will also define a call-out SHELL script (/etc/udev/scripts/iscsidev.openfiler:racdb.2006-01. When either of the Oracle RAC nodes boot and the iSCSI initiator service is started.2006-01.2.com.openfiler:racdb./. however.openfiler:racdb.2006-01.. it will automatically log in to each of the targets configured in a random fashion and map them to the next available local SCSI device name. I can actually determine the current mappings for all targets by looking at the /dev/disk/by-path directory: # (cd /dev/disk/by-path.openfiler:racdb.168.com.2.asm3-lun-0 -> .openfiler:racdb.it maps to is required in order to know which volume (device) is to be used for OCFS2 and which volumes belong to ASM. For example.. Create the following rules file /etc/udev/rules.d/55-openiscsi.168.sh) to handle the event.

2006-01.sh Now that udev is configured.asm2. then exit 1 fi # Check if QNAP drive check_qnap_target_name=${target_name%%:*} if [ $check_qnap_target_name = "iqn. change it to executable: # chmod 755 /etc/udev/scripts/iscsidev. target: iqn.rules KERNEL=="sd*". then target_name=`echo "${target_name%.sh on both Oracle RAC nodes: /etc/udev/scripts/iscsidev. Let's first create a separate directory on both Oracle RAC nodes where udev scripts can be stored: # mkdir -p /etc/udev/scripts Next. PROGRAM="/etc/udev/scripts/iscsidev.168.d/55-openiscsi.com. create the UNIX shell script /etc/udev/scripts/iscsidev.sh %b".2.sh BUS=${1} HOST=${BUS%%:*} [ -e /sys/class/iscsi_host ] || exit 1 file="/sys/class/iscsi_host/host${HOST}/device/session*/iscsi_session*/targetname" target_name=$(cat ${file}) # This is not an open-scsi drive if [ -z "${target_name}" ].168. BUS=="scsi". restart the iSCSI service on both Oracle RAC nodes: # service iscsi stop Logging out of session [sid: 1. portal: 192.com.*}"` fi echo "${target_name##*.2.rules # /etc/udev/rules. target: iqn.com.195.qnap" ].openfiler:racdb.3260] .195.3260] Logging out of session [sid: 2.asm1.openfiler:racdb.d/55-openiscsi. portal: 192.2004-04.sh #!/bin/sh # FILE: /etc/udev/scripts/iscsidev./etc/udev/rules.}" After creating the UNIX SHELL script.2006-01.SYMLINK+="iscsi/%c/part%n" We now need to create the UNIX SHELL script that will be called when this event is received.

2006-01.168.2006-01.com.195.openfiler:racdb.asm3.asm2. portal: 192.2006-01.195.openfiler:racdb.195.com.. portal: 192.openfiler:racdb.2.asm1.195.com. target: iqn.crs.com. portal: 192.168. portal: 192. target: iqn. target: iqn.2.openfiler:racdb.2006-01.168.openfiler:racdb.3260] Logging out of session [sid: 4.crs.168.openfiler:racdb.2. portal: 192.crs. portal: 192.openfiler:racdb.2.195.com.asm3.asm3.195.openfiler:racdb. portal: 192.2.3260]: successful Logout of [sid: 2.3260]: successful Login to [iface: default. target: iqn. target: iqn.2. target: iqn.168.2.195. target: iqn.168.168.com.2006-01.2. target: iqn.168.asm3.2006-01.3260]: successful [ OK ] Let's see if our hard work paid off: # ls -l /dev/iscsi/* /dev/iscsi/asm1: total 0 lrwxrwxrwx 1 root root 9 Aug 16 00:49 part -> .3260] Logging in to [iface: default.asm4.2006-01.2006-01.168..195.3260] Logging in to [iface: default.195.2006-01.195.com.openfiler:racdb. portal: 192.openfiler:racdb.com. target: iqn. target: iqn.2.3260]: successful Login to [iface: default.com.2.openfiler:racdb.openfiler:racdb.168. portal: 192.2..200601.asm2.com.195.3260] Login to [iface: default. portal: 192./. target: iqn. target: iqn.2. target: iqn.openfiler:racdb.com.195.168.168.2006-01.com.openfiler:racdb.3260]: successful Logout of [sid: 4.asm1. portal: 192. portal: 192. Starting iSCSI daemon: [ OK ] [ OK ] Setting up iSCSI targets: Logging in to [iface: default.2. portal: 192.3260] Logging out of session [sid: 5. target: iqn. portal: 192.195. target: iqn.2.2. portal: 192. portal: 192.168./sdc /dev/iscsi/asm3: .2.asm4.openfiler:racdb.168.openfiler:racdb..2006-01./.asm4.Logging out of session [sid: 3.3260] Logout of [sid: 1.com.com.crs.asm4.asm2. target: iqn.168.3260]: successful Stopping iSCSI daemon: [ OK ] # service iscsi start iscsid dead but pid file exists Turning off network shutdown. portal: 192.3260] Logging in to [iface: default.195. target: iqn.3260] Logging in to [iface: default.com.3260]: successful Login to [iface: default.2.openfiler:racdb.2.168.2006-01.com.openfiler:racdb.com.2006-01.2006-01. target: iqn.195.2006-01.3260]: successful Logout of [sid: 3.2006-01.195.2006-01.com.3260]: successful Logout of [sid: 5.195. portal: 192.195./sda /dev/iscsi/asm2: total 0 lrwxrwxrwx 1 root root 9 Aug 16 00:49 part -> .168.168.asm1.3260]: successful Login to [iface: default.

com../sde /dev/iscsi/crs: total 0 lrwxrwxrwx 1 root root 9 Aug 16 00:49 part -> .2006-01.openfiler:racdb. For example. The following table lists the five iSCSI volumes and what file systems they will support: Oracle Shared Drive Configuration File System Type OCFS2 crs iSCSI Target (short) Name Mount Point ASM Diskgroup Name File Types Oracle Cluster Registry (OCR) File .asm2 iqn.2006-01. online redo log files...openfiler:racdb.openfiler:racdb.(~250 MB) Size 2GB /u02 ./sdd The listing above shows that udev did the job it was suppose to do! We now have a consistent set of local device names that can be used to reference the iSCSI targets. We will then be using Automatic Storage Management (ASM) to create four ASM volumes.com. we can safely assume that the device name /dev/iscsi/asm1/part will always reference the iSCSI target iqn.asm3 iqn.. As mentioned earlier in this article.com. I will be using Oracle's Cluster File System./.2006-01.openfiler:racdb.asm4 iqn.openfiler:racdb.total 0 lrwxrwxrwx 1 root root 9 Aug 16 00:49 part -> ./.openfiler:racdb. Release 2 (OCFS2) to store the two files to be shared for Oracle's Clusterware software.asm1.2006-01. two for all physical database files (data/index files../sdb /dev/iscsi/asm4: total 0 lrwxrwxrwx 1 root root 9 Aug 16 00:49 part -> .crs Local Device Name /dev/iscsi/asm1/part /dev/iscsi/asm2/part /dev/iscsi/asm3/part /dev/iscsi/asm4/part /dev/iscsi/crs/part Create Partitions on iSCSI Volumes We now need to create a single primary partition on each of the iSCSI volumes that spans the entire size of the volume..com. and control files) and two for the Flash Recovery Area (RMAN backups and archived redo log files).2006-01.com.2006-01.asm1 iqn./. We now have a consistent iSCSI target name to local device name mapping which is described in the following table: iSCSI Target Name to Local Device Name Mappings iSCSI Target Name iqn.com.

17817403392 bytes 64 heads.Voting Disk . You can safely ignore any warnings that may indicate the device does not contain a valid DOS partition (or Sun. SGI or OSF disklabel). default 1): 1 Last cylinder or +size or +sizeM or +sizeK (1-16992.e.8GB 17. In this example.8GB 17. default 16992): 16992 Command (m for help): p Disk /dev/iscsi/asm1/part: 17.2G B ORCL:VOL 1 ORCL:VOL 2 ORCL:VOL 3 ORCL:VOL 4 +RACDB_DATA1 +RACDB_DATA1 +FLASH_RECOVERY_ARE A +FLASH_RECOVERY_ARE A Oracle Database Files Oracle Database Files Oracle Flash Recovery Area Oracle Flash Recovery Area As shown in the table above. 32 sectors/track. For each of the five iSCSI volumes. 16992 cylinders Units = cylinders of 2048 * 512 = 1048576 bytes Device Boot Start End Blocks Id System . linux1) # --------------------------------------# fdisk /dev/iscsi/asm1/part Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-16992.8GB 73.(~20MB) ASM ASM ASM ASM Total asm1 asm2 asm3 asm4 17.8 GB.8GB 17. you can use the default values when creating the primary partition as the default action is to use the entire disk. we will need to create a single Linux primary partition on each of the five iSCSI volumes. The fdisk command is used in Linux for creating (and removing) partitions. I will be running the fdisk command from linux1 to create a single primary partition on each iSCSI target using the local device names created by udev in the previous section:      /dev/iscsi/asm1/part /dev/iscsi/asm2/part /dev/iscsi/asm3/part /dev/iscsi/asm4/part /dev/iscsi/crs/part Creating the single partition on each of the iSCSI volumes must only be run from one of the nodes in the Oracle RAC cluster! (i.

Syncing disks.8 GB. 17817403392 bytes 64 heads. default 1): 1 Last cylinder or +size or +sizeM or +sizeK (1-16992. 32 sectors/track. Syncing disks. # --------------------------------------# fdisk /dev/iscsi/asm2/part Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-16992.8 GB. 16992 cylinders Units = cylinders of 2048 * 512 = 1048576 bytes Device Boot /dev/iscsi/asm2/part1 Start 1 End 16992 Blocks 17399792 Id 83 System Linux Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table./dev/iscsi/asm1/part1 1 16992 17399792 83 Linux Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. default 1): 1 Last cylinder or +size or +sizeM or +sizeK (1-16992. 16992 cylinders Units = cylinders of 2048 * 512 = 1048576 bytes Device Boot /dev/iscsi/asm3/part1 Command (m for help): w Start 1 End 16992 Blocks 17399792 Id 83 System Linux . 17817403392 bytes 64 heads. 32 sectors/track. default 16992): 16992 Command (m for help): p Disk /dev/iscsi/asm2/part: 17. # --------------------------------------# fdisk /dev/iscsi/asm3/part Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-16992. default 16992): 16992 Command (m for help): p Disk /dev/iscsi/asm3/part: 17.

# --------------------------------------# fdisk /dev/iscsi/asm4/part Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-16960. default 1): 1 Last cylinder or +size or +sizeM or +sizeK (1-1009. 1009 cylinders Units = cylinders of 4154 * 512 = 2126848 bytes Device Boot /dev/iscsi/crs/part1 Start 1 End 1009 Blocks 2095662 Id 83 System Linux Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. default 1): 1 Last cylinder or +size or +sizeM or +sizeK (1-16960. Syncing disks. . 2147483648 bytes 67 heads.The partition table has been altered! Calling ioctl() to re-read partition table. # --------------------------------------# fdisk /dev/iscsi/crs/part Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-1009. default 16960): 16960 Command (m for help): p Disk /dev/iscsi/asm4/part: 17. default 1009): 1009 Command (m for help): p Disk /dev/iscsi/crs/part: 2147 MB. 32 sectors/track.7 GB. Syncing disks. 16960 cylinders Units = cylinders of 2048 * 512 = 1048576 bytes Device Boot /dev/iscsi/asm4/part1 Start 1 End 16960 Blocks 17367024 Id 83 System Linux Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. 17783848960 bytes 64 heads. 62 sectors/track.

62 sectors/track.7 GB. 16992 cylinders Units = cylinders of 2048 * 512 = 1048576 bytes Device Boot /dev/sdd1 Start 1 End 16992 Blocks 17399792 Id 83 System Linux . 63 sectors/track. you should now inform the kernel of the partition changes using the following command as the "root" user account from all remaining nodes in the Oracle RAC cluster (linux2). From linux2. 32 sectors/track. 17783848960 bytes 64 heads. 17817403392 bytes 64 heads. 4863 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot /dev/hda1 * /dev/hda2 Start 1 14 End 13 4863 Blocks 104391 38957625 Id 83 8e System Linux Linux LVM Disk /dev/sda: 2147 MB.Syncing disks.8 GB. Verify New Partitions After creating all required partitions from linux1. 1009 cylinders Units = cylinders of 4154 * 512 = 2126848 bytes Device Boot /dev/sda1 Start 1 End 1009 Blocks 2095662 Id 83 System Linux Disk /dev/sdb: 17. 16960 cylinders Units = cylinders of 2048 * 512 = 1048576 bytes Device Boot /dev/sdb1 Start 1 End 16960 Blocks 17367024 Id 83 System Linux Disk /dev/sdc: 17. 2147483648 bytes 67 heads.0 GB. 17817403392 bytes 64 heads. 32 sectors/track. 40000000000 bytes 255 heads.8 GB. run the following commands: # partprobe # fdisk -l Disk /dev/hda: 40. Note that the mapping of iSCSI target names discovered from Openfiler and the local SCSI device name will be different on both Oracle RAC nodes. This is not a concern and will not cause any problems since we will not be using the local SCSI device names but rather the local device names created by udev in the previous section. 32 sectors/track. 16992 cylinders Units = cylinders of 2048 * 512 = 1048576 bytes Device Boot /dev/sdc1 Start 1 End 16992 Blocks 17399792 Id 83 System Linux Disk /dev/sdd: 17.

2.195:3260-iscsi-iqn.asm1-lun-0 -> ..openfiler:racdb./sdb ip-192./sdc1 ip-192.com.168. and all required directories./sdd1 ip-192.2.openfiler:racdb../sdd ip-192.com./sde1 ip-192./. 16992 cylinders Units = cylinders of 2048 * 512 = 1048576 bytes Device Boot /dev/sde1 Start 1 End 16992 Blocks 17399792 Id 83 System Linux As a final step you should run the following command on both Oracle RAC nodes to verify that udev created the new symbolic links for each new partition: # (cd /dev/disk/by-path.168.asm4-lun-0 -> .195:3260-iscsi-iqn. 17817403392 bytes 64 heads.168.2006-01.com. recommended O/S groups.2006-01.168.195:3260-iscsi-iqn.asm2-lun-0-part1 -> .openfiler:racdb..2.2006-01.asm3-lun-0-part1 -> .195:3260-iscsi-iqn..openfiler:racdb. print $9 " " $10 " " $11}') ip-192..com. 32 sectors/track..2.195:3260-iscsi-iqn././sde ip-192.openfiler:racdb.com.com.168./.asm2-lun-0 -> .195:3260-iscsi-iqn./../.openfiler:racdb.168.168.asm3-lun-0 -> . The following O/S groups will be created: Description Oracle Inventory and Software Owner Database Administrator Database Operator SYSDBA SYSOPER OSDBA OSOPER Oracle Privilege Oracle Group Name UNIX Group name oinstall dba oper ./..2006-01.com.2.2006-01.2006-01.com...2006-01.195:3260-iscsi-iqn.crs-lun-0-part1 -> ..8 GB..2006-01./.crs-lun-0 -> ./.././sdc ip-192../sda ip-192. We will be using these new device names when configuring the volumes for OCFS2 and ASMlib later in this guide:      /dev/iscsi/asm1/part1 /dev/iscsi/asm2/part1 /dev/iscsi/asm3/part1 /dev/iscsi/asm4/part1 /dev/iscsi/crs/part1 Create "oracle" User and Directories Perform the following tasks on both Oracle RAC nodes in the cluster! In this section we will create the oracle UNIX user account.com../.195:3260-iscsi-iqn.Disk /dev/sde: 17...2.195:3260-iscsi-iqn..openfiler:racdb.2.openfiler:racdb. ls -l *openfiler* | awk '{FS=" ".168.168.2.2.openfiler:racdb.2006-01.2006-01.195:3260-iscsi-iqn.2..168.asm4-lun-0-part1 -> .openfiler:racdb.com./sda1 The listing above shows that udev did indeed create new device names for each of the new partitions./sdb1 ip-192..asm1-lun-0-part1 -> .

502(dba). and oper group and oracle user account: # groupadd -g 501 oinstall # groupadd -g 502 dba # groupadd -g 503 oper # useradd -m -u 501 -g oinstall -G dba. Note that members of the UNIX group oinstall are considered the "owners" of the Oracle software. for example starting up and shutting down databases. we are creating the oracle user account to have all responsibilities! This guide adheres to the Optimal Flexible Architecture (OFA) for naming conventions used in creating the directory structure. To use this group. the UID of the UNIX user " oracle" and GID of the UNIX group "oinstall" must be the same on both of the Oracle RAC nodes in the cluster. Members of the dba group can administer Oracle databases. dba. complete the following procedure to verify that the user nobody exists on the system: 1. Verify That the User nobody Exists Before installing the Oracle software. If either the UID or GID are different. For this article. In this article. Release 2 (OCFS2) to store the files required to be shared for the Oracle Clusterware software. I will use 501 for the "oracle" UID and 501 for the "oinstall" GID. The default name for this group is oper. the files on the OCFS2 file system will show up as "unowned" or may even be owned by a different user.503(oper) Set the password for the oracle account: # passwd oracle Changing password for user oracle. New UNIX password: xxxxxxxxxxx Retype new UNIX password: xxxxxxxxxxx passwd: all authentication tokens updated successfully. To determine if the user exists. choose the "Custom" installation type to install the Oracle database software. # id nobody uid=99(nobody) gid=99(nobody) groups=99(nobody) .We will be using the Oracle Cluster File System. Create Group and User for Oracle Lets start this section by creating the UNIX oinstall. Members of the optional group oper have a limited set of database administrative privileges such as managing and running backups. enter the following command: 2.oper -d /home/oracle -s /bin/bash -c "Oracle Software Owner" oracle # id oracle uid=501(oracle) gid=501(oinstall) groups=501(oinstall). When using OCFS2.

If the user nobody does not exist. 3. and permissions for it. these directories would be created on a separate file system. This is being done for the sake of simplicity and is not recommended as a general practice. Please note that this is being done for the sake of simplicity and is not recommended as a general practice. then you do not have to create that user. Normally. /u01/app/oracle owned by oracle:oinstall with 775 permissions. you will have the following:    /u01 owned by root. Perform the following on both Oracle RAC nodes: # mkdir -p /u01/app/oracle # chown -R oracle:oinstall /u01/app/oracle # chmod -R 775 /u01/app/oracle At the end of this procedure. the following assumes that the directories are being created in the root file system. After the directory is created. Repeat this procedure on all the other Oracle RAC nodes in the cluster. Create the Oracle Clusterware Home Directory Next. The following assumes that the directories are being created in the root file system. create a new directory that will be used to store the Oracle Clusterware software. group. Normally. these directories would be created on a separate file system. This ownership and permissions enables the OUI to create the oraInventory directory. . Create the Oracle Base Directory The next step is to create a new directory that will be used to store the Oracle Database software. When configuring the oracle user's environment (later in this section) we will be assigning the location of this directory to the $ORACLE_BASE environment variable. in the path /u01/app/oracle/oraInventory. As noted in the previous section. /u01/app owned by root. When configuring the oracle user's environment (later in this section) we will be assigning the location of this directory to the $ORA_CRS_HOME environment variable.If this command displays information about the nobody user. you must then specify the correct owner. then enter the following command to create it: # /usr/sbin/useradd nobody 4.

bash_profile: When you are setting the Oracle environment variables for each Oracle RAC node.oracle . Create Mount Point for OCFS2 / Clusterware Let's now create the mount point for the Oracle Cluster File System.After the directory is created. Perform the following on both Oracle RAC nodes: # mkdir -p /u02 # chown -R oracle:oinstall /u02 # chmod -R 775 /u02 Create Login Script for oracle User Account To ensure that the environment is setup correctly for the "oracle" UNIX userid on both Oracle RAC nodes. Perform the following on both Oracle RAC nodes: # mkdir -p /u01/app/crs # chown -R oracle:oinstall /u01/app/crs # chmod -R 775 /u01/app/crs At the end of this procedure. Release 2 (OCFS2) that will be used to store the two Oracle Clusterware shared files. These permissions are required for Oracle Clusterware installation and are changed during the installation process. I used: For this example. use the following . I used:   linux1 : ORACLE_SID=racdb1 linux2 : ORACLE_SID=racdb2 Log in to each node as the oracle user account: # su . you will have the following:    /u01 owned by root. you must then specify the correct owner. /u01/app owned by root.bash_profile for Oracle User . and permissions for it. group. /u01/app/crs owned by oracle:oinstall with 775 permissions. ensure to assign each RAC node a unique Oracle SID! For this example.

. ~/.0/db_1 ORA_CRS_HOME=/u01/app/crs ORACLE_PATH=$ORACLE_BASE/dba_scripts/common/sql:. racdb2.# . (i.bashrc ]. racdb1. This section ..) export ORACLE_SID=racdb1 export export export export export export export export export export export export export export export export export PATH=..:${JAVA_HOME}/bin:${PATH}:$HOME/bin:$ORACLE_HOME/bin PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin PATH=${PATH}:$ORACLE_BASE/dba_scripts/common/bin ORACLE_TERM=xterm TNS_ADMIN=$ORACLE_HOME/network/admin ORA_NLS10=$ORACLE_HOME/nls/data NLS_DATE_FORMAT="DD-MON-YYYY HH24:MI:SS" LD_LIBRARY_PATH=$ORACLE_HOME/lib LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/oracm/lib LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib CLASSPATH=$ORACLE_HOME/JRE CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib THREADS_FLAG=native TEMP=/tmp TMPDIR=/tmp Configure the Linux Servers for Oracle Perform the following configuration procedures on both Oracle RAC nodes in the cluster! The kernel parameters and shell limits discussed in this section will need to be defined on both Oracle RAC nodes in the cluster every time the machine is booted.2.e.:$ORACLE_HOME/rdbms/admin CV_JDKHOME=/usr/local/java # Each RAC node must have a unique ORACLE_SID.bashrc fi alias ls="ls -FA" alias s="screen -DRRS iPad -t iPad" export JAVA_HOME=/usr/local/java # User export export export export export specific environment and startup programs ORACLE_BASE=/u01/app/oracle ORACLE_HOME=$ORACLE_BASE/product/10. then .bash_profile # Get the aliases and functions if [ -f ~/.

There are several different ways to configure (set) these parameters. setting the IP local port range. This way you do not have to use a raw device or even more drastic. Instructions for placing them in a startup script ( /etc/sysctl.conf) are included in the section "All Startup Commands for Both Oracle RAC Nodes". Overview This section focuses on configuring both Oracle RAC Linux servers . activating all kernel parameters for the system. type: # cat /proc/meminfo | grep SwapTotal SwapTotal: 4128760 kB If you have less than 2048MB of memory (between your RAM and SWAP). (Note: An inadequate amount of swap during the installation will cause the Oracle Universal Installer to either "hang" or "die") To check the amount of memory you have. I will be making all changes permanent (through reboots) by placing all commands in the /etc/sysctl. Memory and Swap Space Considerations Installing Oracle Database 10g Release 2 on RHEL/OEL 5 requires a minimum of 1024MB of memory.conf file. rebuild your system.getting each one prepared for the Oracle RAC 10g installation. and finally how to verify the correct date and time for both nodes in the cluster. make a file that will act as additional swap space. setting shell limits for the oracle user. type: # cat /proc/meminfo | grep MemTotal MemTotal: 2074084 kB To check the amount of swap you have allocated. For the purpose of this article. let's say about 500MB: # dd if=/dev/zero of=tempswap bs=1k count=500000 Now we should change the file permissions: # chmod 600 tempswap . setting shared memory and semaphores. you can add temporary swap space by creating a temporary swap file. setting the maximum number of file handles. As root. This includes verifying enough memory and swap space.provides information about setting those kernel parameters required for Oracle.

Also note that when setting the four semaphore values that all four values need to be entered on one line.shmmax Use the default values if they are the same or larger than the required values.core. you can simply copy / paste the following to both Oracle RAC nodes while logged in as root: cat >> /etc/sysctl. This article assumes a fresh new install of CentOS 5 and as such. This being the case.wmem_max=262144 RHEL/OEL 5 already comes configured with default values defined for the following kernel parameters: kernel.core. The values given are minimums.file-max = 65536 net.wmem_default=262144 net.sem = 250 32000 100 128 fs. On both Oracle RAC nodes. verify that the kernel parameters described in this section are set to values greater than or equal to the recommended values.rmem_default=1048576 net.shmmni = 4096 kernel.Finally we format the "partition" as swap and add it to the swap space: # mke2fs tempswap # mkswap tempswap # swapon tempswap Configuring Kernel Parameters and Shell Limits The kernel parameters and shell limits presented in this section are recommended values only as documented by Oracle. so if your system uses a larger value. For production database systems.conf <<EOF kernel.core.ip_local_port_range = 1024 65000 net.shmall = 268435456 kernel.core.ipv4. do not change it: kernel.sem = 250 32000 100 128 .shmmni = 4096 kernel.rmem_max=1048576 net.shmall kernel. Configuring Kernel Parameters Oracle Database 10g Release 2 on RHEL/OEL 5 requires the kernel parameter settings shown below. many of the required kernel parameters are already set (see above).shmmax = 4294967295 kernel. Oracle recommends that you tune these values to optimize the performance of the system.

run the following as root on both Oracle RAC nodes in the cluster: # sysctl -p net.core.shmall = 268435456 kernel.ipv4.default.wmem_max = 262144 .core.rp_filter = 1 net.core.core.core.file-max = 65536 # /sbin/sysctl -a | grep ip_local_port_range net.sem = 250 32000 100 128 fs.wmem_max = 262144 kernel.ipv4.[rw]mem' net.sem = 250 32000 100 128 # /sbin/sysctl -a | grep file-max fs.ipv4.core.ipv4.shmmax = 4294967295 kernel. so there's no need to reboot the system after making kernel parameter changes.rmem_default = 1048576 net.rmem_max = 1048576 net.conf startup file.ip_forward = 0 net.msgmnb = 65536 kernel.shmall = 268435456 net.conf.hugetlb_shm_group = 0 kernel.rmem_max = 1048576 net.rmem_default = 1048576 net.shmmni = 4096 kernel.ipv4.core. To activate the new kernel parameter values for the currently running system.default.file-max = 65536 net.sysrq = 0 kernel.conf.ipv4.file-max = 65536 net.shmmax = 4294967295 # /sbin/sysctl -a | grep sem kernel.ipv4.shmmni = 4096 kernel.ip_local_port_range = 1024 65000 Verify the new kernel parameter values by running the following on both Oracle RAC nodes in the cluster: # /sbin/sysctl -a | grep shm vm.msgmax = 65536 kernel.wmem_default = 262144 net.wmem_default = 262144 net.ip_local_port_range = 1024 65000 EOF The above command persisted the required kernel parameters through reboots by inserting them in the /etc/sysctl.accept_source_route = 0 kernel.tcp_syncookies = 1 kernel.ip_local_port_range = 1024 65000 # /sbin/sysctl -a | grep 'core\.core_uses_pid = 1 net. Linux allows modification of these kernel parameters to the current system while it is up and running.core.fs.

then if [ \$SHELL = "/bin/ksh" ]. Bash.            For the Bourne.Setting Shell Limits for the oracle User To improve the performance of the software on Linux systems. add the following lines to the /etc/csh. run the following as root: cat >> oracle oracle oracle oracle EOF /etc/security/limits.login <<EOF if ( \$USER == "oracle" ) then limit maxproc 16384 limit descriptors 65536 endif EOF . add the following lines to the /etc/profile file by running the following command: cat >> /etc/profile <<EOF if [ \$USER = "oracle" ].d/login <<EOF session required /lib/security/pam_limits. or Korn shell. Oracle recommends you increase the following shell limits for the oracle user: Shell Limit Maximum number of open file descriptors Item in limits.so EOF Update the default shell startup file for the "oracle" UNIX account. then ulimit -p 16384 ulimit -n 65536 else ulimit -u 16384 -n 65536 fi umask 022 fi EOF       For the C shell (csh or tcsh).conf Hard Limit nofile 65536 16384 Maximum number of processes available to a single user nproc To make these changes.login file by running the following command: cat >> /etc/csh.conf <<EOF soft nproc 2047 hard nproc 16384 soft nofile 1024 hard nofile 65536 cat >> /etc/pam.

. Oracle strongly recommends using the Network Time Protocol feature of most operating systems for this purpose. the Oracle Universal Installer (OUI) first installs the software to the local node running the installer (i. with all nodes using the same reference Network Time Protocol server.(more errors on this node) Please note that although this would seem like a severe error from the OUI. may not always be an option. ensure that the date and time of the node you are performing the software installations from (linux1) is less than all other nodes in the cluster (linux2). If the date and time on the node performing the install is greater than that of the node it is copying to. The "tar" command DOES actually extract the files. In this case. Before starting any of the above noted installations. the OUI will throw an error from the " tar" command indicating it is attempting to extract files stamped with a time in the future: Error while copying directory /u01/app/crs with exclude file list 'null' to nodes 'linux2'. linux1).e. Accessing a Network Time Protocol server. ensure that each member node of the cluster is set as closely as possible to the same date and time. [PRKC-1002 : All the submitted commands did not execute successfully] --------------------------------------------linux2: /bin/tar: . and the Companion CD. the OUI will execute the UNIX "tar" command on each of the remote nodes to extract the files that were archived and copied over. however. The software is then copied remotely to all of the remaining nodes in the cluster (i.e. During the remote copy process. I generally use a 20 second difference as shown in the following example: Setting the date and time from linux1: # date -s "8/16/2009 01:12:00" Setting the date and time from linux2: # date -s "8/16/2009 01:12:20" The two-node RAC configuration described in this article does not make use of a Network Time Protocol server. the Database.Setting the Correct Date and Time on All Cluster Nodes During the installation of Oracle Clusterware. Configure the "hangcheck-timer" Kernel Module .. it can safely be disregarded as a warning. they will be missing the time field until the time on the remote server is greater than the timestamp of the file./bin/olsnodes: time stamp 2009-08-16 09:21:34 is 735 s in the future . however. linux2)./bin/lsnodes: time stamp 2009-08-16 09:21:34 is 735 s in the future /bin/tar: . when manually setting the date and time for the nodes in the cluster. when you perform a listing of the files (using ls -l) on the remote node.

18-128.1 used a userspace watchdog daemon called watchdogd to monitor the health of the cluster and to restart a RAC node in case of a failure. Starting with Oracle 9. There is a configurable threshold to hang-check that.1 and 9.0. It will set a timer and check the timer after a certain amount of time. The hangcheck-timer. Use the following to ensure that you have the module included: # find /lib/modules -name "hangcheck-timer. Configuring and Loading the hangcheck-timer Module There are two key parameters to the hangcheck-timer module: . Although the hangcheck-timer module is not required for Oracle Clusterware (Cluster Manager) operation. however. the timer resets the node. if exceeded will reboot the machine.0.ko) in the /lib/modules/2.12 and higher.18-128.ko Module The hangcheck-timer was normally shipped only by Oracle. The hangcheck-timer module uses the Time Stamp Counter (TSC) CPU register which is a counter that is incremented at each clock signal.2.6.ko" /lib/modules/2.6.ko In the above output. this module is now included with Red Hat Linux AS starting with kernel versions 2.Perform the following configuration procedures on both Oracle RAC nodes in the cluster! Oracle 9. If the system hangs or pauses. Installing the hangcheck-timer. we care about the hangcheck timer object ( hangcheck-timer.el5/kernel/drivers/char directory.9-e. The hangcheck-timer should already be included.2.el5/kernel/drivers/char/hangcheck-timer.0. Much more information about the hangcheck-timer project can be found here.4. the watchdog daemon has been deprecated by a Linux kernel module named hangchecktimer which addresses availability and reliability problems much better. The TCS offers much more accurate time measurements since this register is updated by the hardware automatically. it is highly recommended by Oracle. The hang-check timer is loaded into the Linux kernel and checks if the system hangs.2 (and still available in Oracle10g Release 2).ko Module The hangcheck-timer module uses a kernel-based timer that periodically checks the system task scheduler to catch delays in order to determine the health of the system.

So to keep myself sane and able to sleep at night. It is only out of pure habit that I continue to include a modprobe of the hangcheck-timer kernel module in the /etc/rc.e. hangcheck-margin: This parameter defines the maximum hang delay that should be tolerated before hangcheck-timer resets the RAC node.local file. The two hangcheck-timer module parameters indicate how long a RAC node must hang before it will reset the system. These values need to be available after each reboot of the Linux server. Oracle recommends setting it to 30 seconds. it will use the values defined by the entry I made in the /etc/modprobe. It defines the margin of error in seconds. A node reset will occur when the following is true: system hang time > (hangcheck_tick + hangcheck_margin) Configuring Hangcheck Kernel Module Parameters Each time the hangcheck-timer kernel module is loaded (manually or by Oracle) it needs to know what value to use for each of the two parameters we just discussed: (hangcheck-tick and hangcheck-margin). Someday I will get over it.local).local You don't have to manually load the hangcheck-timer kernel module using modprobe or insmod after each reboot. To do this. I always configure the loading of the hangcheck-timer kernel module on each startup as follows: # echo "/sbin/modprobe hangcheck-timer" >> /etc/rc. but realize that it does not hurt to include a modprobe of the hangcheck-timer kernel module during startup. The default value is 60 seconds.conf file as follows: # echo "options hangcheck-timer hangcheck_tick=30 hangcheck_margin=180" >> /etc/modprobe. It is for this reason that it is not required to perform a modprobe or insmod of the hangcheck-timer kernel module in any of the startup files (i. make an entry with the correct values to the /etc/modprobe. The hangcheck-timer module will be loaded by Oracle (automatically) when needed.conf Each time the hangcheck-timer kernel module gets loaded. .  hangcheck-tick: This parameter defines the period of time between checks of system health. The default value is 180 seconds. Oracle recommends setting it to 180 seconds. Manually Loading the Hangcheck Kernel Module for Testing Oracle is responsible for loading the hangcheck-timer kernel module when required.conf file. /etc/rc.

Now. the software owner for Oracle Clusterware and the Oracle Database software will be " oracle". use the modprobe command. The SSH configuration described in this article uses SSH1. we will configure SSH using SSH1. If SSH is not available. You need either an RSA or a DSA key for the SSH protocol.conf file. As configured earlier in this article. User equivalence enables the oracle UNIX user account to access all other nodes in the cluster (running commands and copying files) without the need for a password. then OUI attempts to use rsh and rcp instead.g. RSA is used with the SSH 1.9. For the purpose of this article.0 protocol.conf file. These services. Oracle added support in 10g Release 1 for using the SSH tool suite for setting up user equivalence. to test the hangcheck-timer kernel module to verify it is picking up the correct parameters we defined in the /etc/modprobe. margin is 180 seconds). are disabled by default on most Linux systems. To manually load the hangcheck-timer kernel module and verify it is using the correct values defined in the /etc/modprobe.0 (tick is 30 seconds. The use of RSH will not be discussed in this article. Before Oracle Database 10g. Although you could load the hangcheck-timer kernel module by passing it the appropriate parameters (e.conf file. Configure RAC Nodes for Remote Access using SSH Perform the following configuration procedures on both Oracle RAC nodes in the cluster! Before you can install Oracle RAC 10g. you can use either RSA or DSA. however. . we want to verify that it is picking up the options we set in the /etc/modprobe. user equivalence had to be configured using remote shell (RSH). Aug 16 16:46:59 linux1 kernel: Hangcheck: Using get_cycles(). you must configure secure shell (SSH) for the UNIX user account you plan to use to install Oracle Clusterware and the Oracle Database software. run the following command: # modprobe hangcheck-timer # grep Hangcheck /var/log/messages | tail -2 Aug 16 16:46:59 linux1 kernel: Hangcheck: starting hangcheck timer 0. The goal here is to setup user equivalence for the oracle UNIX user account. insmod hangcheck-timer hangcheck_tick=30 hangcheck_margin=180).5 protocol. while DSA is the default for the SSH 2. With OpenSSH. The installation and configuration tasks described in this section will need to be performed on both Oracle RAC nodes.

This type of configuration is beyond the scope of this article and will not be discussed. Run this command on both Oracle RAC nodes in the cluster to verify the SSH daemons are installed and running! To find out more about SSH. then refer to your SSH distribution documentation to configure SSH1 compatibility or to configure SSH2 with DSA. and you cannot use SSH1. . and user equivalence is described in the following sections. This means that you must be able to run the secure shell commands (ssh or scp) on the Linux server you will be running the OUI from against all other Linux servers in the cluster without being prompted for a password. must to be enabled for RAC and patchset installations as well as creating the clustered database. enter the following command: # pgrep sshd 2360 If SSH is running. however. why do we have to setup user equivalence? Installing Oracle Clusterware and the Oracle Database software is only performed from one node in a RAC cluster. The content of the RSA public keys will then need to be copied into an authorized key file which is then distributed to both Oracle RAC nodes in the cluster. The methods required for configuring SSH1. Configuring the Secure Shell To determine if SSH is installed and running. it will use the ssh and scp commands to run remote commands on and copy files (the Oracle software) to all other nodes within the RAC cluster. When running the Oracle Universal Installer (OUI) on that particular node. an RSA key. refer to the man page: # man ssh Creating the RSA Keys on Both Oracle RAC Nodes The first step in configuring SSH is to create an RSA public/private key pair on both Oracle RAC nodes in the cluster. Log on as the "oracle" UNIX user account. Please note that these steps will need to be completed on both Oracle RAC nodes in the cluster: 1. This configuration. Please note that the use of secure shell is not required for normal RAC operation. The command to do this will create a public and private key for RSA (for a total of two keys per node). then the response to this command is a list of process ID number(s).If you have an SSH2 installation. So. Use the following steps to create the RSA key pair. The oracle UNIX user account on the node running the OUI (runInstaller) must be trusted by all other nodes in your RAC cluster.

Complete the following steps on one of the nodes in the cluster to create and then distribute the authorized key file.pub public key from both Oracle RAC nodes in the cluster to the authorized key file just created (~/. You will be prompted for the oracle UNIX user account password for both Oracle RAC nodes accessed. Enter and confirm a pass phrase.1 oracle oinstall 395 Aug 16 17:05 id_rsa.ssh/id_rsa. In this step. This should be different from the "oracle" UNIX user account password however it is not a requirement. it is then distributed to all other nodes in the cluster. Again. Once the authorized key file contains all of the public keys. create it now: 2.ssh/id_rsa. $ cd ~/. $ ls -l *. I am using linux1: 1. determine if an authorized key file already exists on the node ( ~/.pub file and the private key to the ~/. This command will write the public key to the ~/. If necessary.ssh/authorized_keys 3. use SCP (Secure Copy) or SFTP (Secure FTP) to copy the content of the ~/.oracle 2.pub public key created in the previous section. First.pub The listing above should show the id_rsa.pub -rw-r--r-. $ mkdir -p ~/. $ touch ~/. 5.ssh $ chmod 700 ~/. Note that you should never distribute the private key to anyone! 5. .ssh/authorized_keys). For the purpose of this article. Repeat the above steps for each Oracle RAC node in the cluster. Now that both Oracle RAC nodes contain a public and private key for RSA. create the .ssh/id_rsa file. In most cases this will not exist since this article assumes you are working with a new install.ssh 4.# su .ssh directory in the "oracle" user's home directory and set the correct permissions on it: 3.ssh 4. If the file doesn't exist. An authorized key file is nothing more than a single file that contains a copy of everyone's (every node's) RSA public key. Enter the following command to generate an RSA key pair (public and private key) for the SSH protocol: $ /usr/bin/ssh-keygen -t rsa At the prompts: o o Accept the default location for the key files. this will be done from linux1.ssh/authorized_keys). you will need to create an authorized key file on one of the nodes.

the only remaining node is linux2. You should not see this message again when you connect from this system to the same node.ssh/id_rsa. $ ssh linux1 hostname Enter passphrase for key '/home/oracle/.8KB/s 00:00 9.168.101)' can't be established.ssh/authorized_keys The authenticity of host 'linux1 (192.1.The following example is being run from linux1 and assumes a two-node cluster.ssh/authorized_keys 8.168.168.1. with nodes linux1 and linux2: $ ssh linux1 cat ~/.1.100)' can't be established. At this point. 6. 13. In our two-node cluster example. we have the RSA public key from every node in the cluster in the authorized key file (~/.ssh/authorized_keys linux2:.168. RSA key fingerprint is 9d:dd:6d:7f:8c:0b:46:bb:93:77:05:a4:b8:d8:4b:31.ssh/authorized_keys) on linux1. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'linux1. Change the permission of the authorized key file for both Oracle RAC nodes in the cluster by logging into the node and running the following: $ chmod 600 ~/. Use the scp command to copy the authorized key file to all remaining nodes in the RAC cluster: 7. For example. you will see a message similar to the following: The authenticity of host 'linux1 (192. We now need to copy it to the remaining nodes in the cluster.168. 12.ssh/id_rsa': xxxxx linux1 .ssh/authorized_keys The authenticity of host 'linux2 (192. test the following from linux1: 11. RSA key fingerprint is d8:2e:03:d5:18:56:60:f3:5b:fa:ba:29:c9:f4:be:d5.pub >> ~/. Are you sure you want to continue connecting (yes/no)? yes Enter yes at the prompt to continue.pub >> ~/. At this point. if you use ssh to log in to or run a command on another node.101' (RSA) to the list of known hosts. oracle@linux1's password: xxxxx $ ssh linux2 cat ~/.1.192. oracle@linux2's password: xxxxx authorized_keys 100% 790 0. RSA key fingerprint is d8:2e:03:d5:18:56:60:f3:5b:fa:ba:29:c9:f4:be:d5.ssh/authorized_keys 10.1.100)' can't be established. you are prompted for the pass phrase that you specified when you created the RSA key.ssh/id_rsa. $ scp ~/.192. 14.100' (RSA) to the list of known hosts. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'linux2. oracle@linux2's password: xxxxx The first time you use SSH to connect to a node from a particular system.

Log on to the node where you want to run the OUI from ( linux1) as the "oracle" UNIX user account. Enter the following commands: 3. $ ssh linux1 "date. you will be able to use the ssh and scp commands without being prompted for a password or pass phrase from this terminal session: 7. To enable user equivalence for the current terminal shell session. all Oracle installations will be performed from linux1. Enter passphrase for /home/oracle/. Even though SSH is configured on both Oracle RAC nodes in the cluster.ssh/id_rsa: xxxxx Identity added: /home/oracle/. Sun Aug 16 17:37:47 EDT 2009 9. Enter passphrase for key '/home/oracle/. are modified so that they act only when the shell is an interactive shell. you need to enable user equivalence for the terminal session you plan to run the OUI from. If SSH is configured correctly. 11. or ask any questions.15. 6.oracle 2. you must enable user equivalence for the terminal shell session as this is not done by default. If you log out and log back in to the node you will be performing the Oracle installation from. $ ssh linux2 "date. For the purpose of this article. $ /usr/bin/ssh-add 5. Sun Aug 16 17:38:20 EDT 2009 linux2 . $ ssh linux2 hostname 16. apart from the host name. then the Oracle installation can fail.hostname" 12. You should ensure that any part of a login script(s) that generate any output.ssh/id_rsa) At the prompt.ssh/id_rsa (/home/oracle/. Make any changes required to ensure that only the host name is displayed when you enter these commands. it will need to run the secure shell tool commands (ssh and scp) without being prompted for a pass phrase. Enabling SSH User Equivalency for the Current Shell Session When running the OUI.ssh/id_rsa': xxxxx linux2 If you see any other messages or text. Before running the OUI. linux1 10. perform the following steps: 1.hostname" 8. enter the pass phrase for each key that you generated. # su . User equivalence will need to be enabled on any new terminal shell session before attempting to run the OUI. using the secure shell tool commands will still prompt for a pass phrase. $ exec /usr/bin/ssh-agent $SHELL 4.

linux2 Note that having X11 Forwarding enabled will cause the Oracle installation to fail. and Bash shells: $ DISPLAY=<Any X-Windows Host>:0 $ export DISPLAY C shell: $ setenv DISPLAY <Any X-Windows Host>:0 After setting the DISPLAY variable to a valid X Windows display. apart from the date and hostname. Korn. set the environment variable DISPLAY to a valid X Windows display: Bourne. Make any changes required to ensure that only the date is displayed when you enter these commands." then this means that your authorized keys file is configured correctly. To correct this problem. if you see any other messages or text. and you see a message similar to: "Warning: No xauth data. using fake authentication data for X11 forwarding. From the terminal session enabled for user equivalence (the node you will be performing the Oracle installations from). For example: $ export DISPLAY=melody:0 $ ssh linux2 hostname Warning: No xauth data. however. Also. The Oracle Universal Installer is a GUI interface and requires the use of an X Server. create a user-level SSH client configuration file for the "oracle" . are modified so that they act only when the shell is an interactive shell. or ask any questions. 13. You should ensure that any part of a login script(s) that generate any output. If any of the nodes prompt for a password or pass phrase then verify that the ~/. you should perform another test of the current terminal session to ensure that X11 forwarding is not enabled: $ ssh linux1 hostname linux1 $ ssh linux2 hostname linux2 If you are using a remote client to connect to the node performing the installation. your SSH configuration has X11 forwarding enabled.ssh/authorized_keys file on that node contains the correct public keys. using fake authentication data for X11 forwarding.The commands above should display the date set on each Oracle RAC node along with its hostname. then the Oracle installation can fail.

then stty intr ^C fi     C shell: test -t 0 if ($status == 0) then stty intr ^C endif If there are hidden files that contain stty commands that are loaded by the remote shell. then OUI indicates an error and stops the installation.e.UNIX user account that disables X11 Forwarding: o o o Using a text editor. any hidden files on the system (i. insert the following into the ~/.cshrc. . All Startup Commands for Both Oracle RAC Nodes Verify that the following startup commands are included on both of the Oracle RAC nodes in the cluster! . You must run the Oracle Universal Installer from this terminal session or remember to repeat the steps to enable user equivalence (steps 2. 3. and 4 from this section) before you start the Oracle Universal Installer from a different terminal session. . edit or create the file ~/.ssh/config Make sure that the ForwardX11 attribute is set to no.profile) will cause the installation process to fail if they contain stty commands. Remove any stty Commands When installing the Oracle software. To avoid this problem. or Korn shell: if [ -t 0 ].ssh/config file: Host * ForwardX11 no 14. you must modify these files to suppress all output on STDERR as in the following examples:    Bourne. . For example. Bash.bashrc.

semaphores.conf All parameters and values to be used by kernel modules. and entries that have been discussed so far that will need to be included in the startup scripts for each Linux node in the RAC cluster. parameters.default. See sysctl(8) and # sysctl. commands. In this section. # Controls IP packet forwarding net. /etc/modprobe. I provide all of the commands.ip_forward = 0 # Controls source route verification net.Up to this point.ipv4. and entries from previous sections that need to occur on both Oracle RAC nodes when they are booted.ipv4.conf(5) for more details.conf # Kernel sysctl configuration file for Red Hat Linux # # For binary values. /etc/sysctl.conf. /etc/modprobe. I indicate in blue the entries that should be included in each of the startup files in order to provide a successful RAC node. /sbin/modprobe -r --ignore-remove snd-intel8x0 options hangcheck-timer hangcheck_tick=30 hangcheck_margin=180 /etc/sysctl.ipv4. }.sysrq = 0 . 1 is enabled.rp_filter = 1 # Do not accept source routing net.conf We wanted to adjust the default and maximum send buffer size as well as the default and maximum receive buffer size for the interconnect. we have talked in great detail about the parameters and resources that need to be configured on both nodes in the Oracle RAC 10g configuration. This section will review those parameters. This file also contains those parameters responsible for configuring shared memory.conf alias eth0 r8169 alias eth1 e1000 alias scsi_hostadapter ata_piix alias snd-card-0 snd-intel8x0 options snd-card-0 index=0 options snd-intel8x0 index=0 remove snd-intel8x0 { /usr/sbin/alsactl store 0 >/dev/null 2>&1 || : . file handles.accept_source_route = 0 # Controls the System Request debugging functionality of the kernel kernel.default. and local IP range for use by the Oracle instance. 0 is disabled. For each of the startup files below.conf.

2.wmem_default=262144 # +---------------------------------------------------------+ # | Maximum setting in bytes of the socket "send" buffer | # | which may be set by using the SO_SNDBUF socket option. such as Cache Fusion | | and Cluster Manager buffer transfers between instances | | within the RAC cluster.msgmax = 65536 # # # # # # # # # # # # # # # # # # # # +---------------------------------------------------------+ | ADJUSTING NETWORK SETTINGS | +---------------------------------------------------------+ | With Oracle 9. The receive buffer cannot overflow because | | the peer is not allowed to send data beyond the buffer | | size window.rmem_default=1048576 # +---------------------------------------------------------+ # | Maximum setting in bytes of the socket "receive" buffer | # | which may be set by using the SO_RCVBUF socket option.ipv4.0.core_uses_pid = 1 # Controls the use of TCP syncookies net. | # +---------------------------------------------------------+ net. | . This could cause the sender to overwhelm the | | receiver.1 and onwards. | # +---------------------------------------------------------+ net.msgmnb = 65536 # Controls the default maxmimum size of a mesage queue kernel. | +---------------------------------------------------------+ # +---------------------------------------------------------+ # | Default setting in bytes of the socket "receive" buffer | # | which may be set by using the SO_RCVBUF socket option. The receive buffers are used by TCP and UDP | | to hold received data until it is read by the | | application.# Controls whether core dumps will append the PID to the core filename # Useful for debugging multi-threaded applications kernel. Oracle now makes use | | of UDP as the default protocol on Linux for | | inter-process communication (IPC).core. in bytes kernel.core. Oracle strongly suggests to | | adjust the default and maximum receive buffer size | | (SO_RCVBUF socket option) to 1024MB.core. This means that datagrams will be | | discarded if they don't fit in the socket receive | | buffer.tcp_syncookies = 1 # Controls the maximum size of a message. and the default | | and maximum send buffer size (SO_SNDBUF socket option) | | to 256KB.rmem_max=1048576 # +---------------------------------------------------------+ # | Default setting in bytes of the socket "send" buffer | # | which may be set by using the SO_SNDBUF socket option. | # +---------------------------------------------------------+ net.

core_uses_pid = 1 net.shmmax = 4294967295 # Controls the maximum number of shared memory segments.wmem_max=262144 # # # # # # # # +---------------------------------------------------------+ | ADJUSTING ADDITIONAL KERNEL PARAMETERS FOR ORACLE | +---------------------------------------------------------+ | Configure the kernel parameters for all Oracle Linux | | servers by setting shared memory and semaphores.accept_source_route = 0 kernel. ensure that each of these parameters are truly in effect by running the following command on both Oracle RAC nodes in the cluster: # sysctl -p net.conf.file-max=65536 # +---------------------------------------------------------+ # | LOCAL IP RANGE | # ----------------------------------------------------------+ net.# +---------------------------------------------------------+ net.tcp_syncookies = 1 kernel.ipv4.core.ip_forward = 0 net.sem=250 32000 100 128 # +---------------------------------------------------------+ # | FILE HANDLES | # ----------------------------------------------------------+ fs.conf file. in bytes kernel. in pages kernel. and setting | | the IP local port range. Then.sysrq = 0 kernel.default.rp_filter = 1 net.ipv4.ipv4.shmmni = 4096 # +---------------------------------------------------------+ # | SEMAPHORES | # | ---------| # | | # | SEMMSL_value SEMMNS_value SEMOPM_value SEMMNI_value | # | | # +---------------------------------------------------------+ kernel. | | setting the maximum amount of file handles.msgmnb = 65536 .default. | +---------------------------------------------------------+ # +---------------------------------------------------------+ # | SHARED MEMORY | # +---------------------------------------------------------+ # Controls the maximum shared segment size.conf.ip_local_port_range=1024 65000 Verify that each of the required kernel parameters (above) are configured in the /etc/sysctl.ipv4.ipv4.shmall = 268435456 # Controls the maximum number of shared memory segments system wide kernel.

1.shmmax = 4294967295 kernel.(eth1) 192.rmem_max = 1048576 net.168.103 nascar 192.122 switch1 192.1.101 linux2-priv # Public Virtual IP (VIP) addresses .168.1.190 george 192.1.101 linux2 # Private Interconnect .195 openfiler1-priv # Miscellaneous Nodes 192.1 router 192.localdomain localhost # Public Network .168.1.105 packmule 192.102 alex 192.file-max = 65536 net.rules .wmem_max = 262144 kernel.1. or various programs # that require network functionality will fail.1.shmmni = 4096 kernel.sem = 250 32000 100 128 fs.100 linux1 192.168.wmem_default = 262144 net.core.ip_local_port_range = 1024 65000 /etc/hosts All machine/IP entries for nodes in the RAC cluster.1.1. /etc/hosts # Do not remove the following line.rmem_default = 1048576 net.100 linux1-priv 192.168.ipv4.(eth0) 192.core.1.1.106 melody 192.core.1 localhost.d/55-openiscsi.168.245 accesspoint /etc/udev/rules.1.(eth1) 192.msgmax = 65536 net.168.168.168.168.201 linux2-vip # Private Storage Network for Openfiler .1.120 cartman 192.1.168.1.168.195 openfiler1 192.200 linux1-vip 192.168.kernel.168.168.(eth0:1) 192.0.0.168.168.shmall = 268435456 kernel. 127.2.core.2.168.2.121 domo 192.

sh Call-out SHELL script that handles the events passed to it from the udev rules file (above) and used to mount iSCSI volumes.rules # /etc/udev/rules.local #!/bin/sh # # This script will be executed *after* all the other init scripts.d/55-openiscsi. BUS=="scsi". /etc/udev/scripts/iscsidev. /etc/rc. /etc/udev/rules.SYMLINK+="iscsi/%c/part%n" /etc/udev/scripts/iscsidev.sh #!/bin/sh # FILE: /etc/udev/scripts/iscsidev. touch /var/lock/subsys/local .local Loading the hangcheck-timer kernel module.sh BUS=${1} HOST=${BUS%%:*} [ -e /sys/class/iscsi_host ] || exit 1 file="/sys/class/iscsi_host/host${HOST}/device/session*/iscsi_session*/targetname" target_name=$(cat ${file}) # This is not an open-scsi drive if [ -z "${target_name}" ]. then exit 1 fi echo "${target_name##*.rules KERNEL=="sd*". PROGRAM="/etc/udev/scripts/iscsidev.sh %b". This file contains all name=value pairs used to receive events and the call-out SHELL script to handle the event.d/55-openiscsi. # You can put your own initialization stuff in here if you don't # want to do the full Sys V style init stuff.}" /etc/rc.Rules file to be used by udev to mount iSCSI volumes.

is a Cluster File System which allows all nodes in a cluster to concurrently access a device via the standard file system interface. however. but doesn't hurt) | +---------------------------------------------------------+ /sbin/modprobe hangcheck-timer Install and Configure Oracle Cluster File System (OCFS2) Most of the installation and configuration procedures in this section should be performed on both Oracle RAC nodes in the cluster! Creating the OCFS2 filesystem. one can store not only database related files on a shared disk. OCFS (Release 1) was released in December 2002 to enable Oracle Real Application Cluster (RAC) users to run the clustered database without having to deal with RAW devices. Overview It is now time to install the Oracle Cluster File System. OCFS2. See the following page for more information on OCFS2 (including Installation Notes) for Linux: OCFS2 Project Documentation Download OCFS2 . It has been designed to be a general purpose cluster file system. control files. I will be using the latest release of OCFS2 to store the two files that are required to be shared by the Oracle Clusterware software. Release 2 (OCFS2). In this article. archive logs.# # # # +---------------------------------------------------------+ | HANGCHECK TIMER | | (I do not believe this is required. This allows for easy management of applications that need to run across a cluster. redo logs. developed by Oracle Corporation. etc. The file system was designed to store database related files. OCFS2 is the next generation of the Oracle Cluster File System. I will also be using this space to store the shared ASM SPFILE for all Oracle RAC instances. such as data files. With it. Along with these two files. should only be executed on one of nodes in the RAC cluster. but also store Oracle binaries and configuration files (shared Oracle Home) making management of RAC even easier.

el5PAE-1. pygtk2 (EL4) or python-gtk (SLES9) 1.6.2-1.el5.2-1.rpm . platform. etc).rpm . kernel version and the kernel flavor (smp. The ocfs2console application requires e2fsprogs. namely.x86_64.99.rpm .18-128. python 2.3.rpm .(OCFS2 tools) ocfs2console-1.6.rpm .el5.el5xen-1.6. The latest kernel module is available for download from http://oss.i686.el5. With CentOS 5. glib2 2. use the OCFS2 release that matches your kernel version.6.4.(Package for PAE kernel) ocfs2-2.3 or later and ocfs2-tools.6. Always download the latest OCFS2 kernel module that matches the distribution.2-1. hugemem.2-1.el5-1. The available OCFS2 kernel modules for Linux kernel 2.com/projects/ocfs2/files/ and the tools from http://oss.el5 #1 SMP Wed Jan 21 10:44:23 EST 2009 i686 i686 i386 GNU/Linux Install OCFS2 .4.(OCFS2 console) 64-bit (x86_64) Installations ocfs2-2.el5.el5. download both the OCFS2 tools and the OCFS2 console applications: ocfs2-tools-1.4.2-1 at the time of this writing).x86_64. I am using kernel release 2.x86_64.6.2-1.6. If you were curious as to which OCFS2 driver release you need.oracle.18-128.18-128. To determine your kernel release: $ uname -a Linux linux1 2.oracle.rpm .com/projects/ocfs2-tools/files/.el5.4.el5.18-128.(Package for xen kernel) Next. 32-bit (x86) Installations ocfs2-2.rpm .18-128.6.4.4.i686.x86_64.4. download both the OCFS2 tools and the OCFS2 console applications: ocfs2-tools-1.el5.(Package for default kernel) ocfs2-2. vte 0.(OCFS2 tools) ocfs2console-1.4. The OCFS2 distribution comprises of two sets of RPMs. the kernel module and the tools.10 or later.4. Download the appropriate RPMs starting with the latest OCFS2 kernel module (the driver).18-128.2.(OCFS2 console) The OCFS2 Console is optional but highly recommended.el5.rpm .2-1.(Package for default kernel) ocfs2-2.11.First.i386. psmp.i386.16 or later.el5.el5xen-1.(Package for xen kernel) Next.2-1.i686.rpm .18-128.18-128.3 or later.2-1.el5 are listed below. The appropriate OCFS2 kernel module was found in the latest release of OCFS2 at the time of this writing (OCFS2 Release 1. let's download the latest OCFS2 distribution.el5-1.4.2-1.

i386.i686.el5. run the "Security Level Configuration" GUI utility: # /usr/bin/system-config-securitylevel & This will bring up the following screen: . During the installation of CentOS. we Disabled SELinux on the SELinux screen. The installation process is simply a matter of running the following command on both Oracle RAC nodes in the cluster as the root user account: # rpm -Uvh ocfs2-2.rpm \ ocfs2-tools-1. you can use the tool system-config-securitylevel to disable SELinux.rpm \ ocfs2console-1.i386.3) you will need to disable SELinux (using tool system-config-securitylevel) to get the O2CB service to execute.2-1.el5.6. you did not disable SELinux during the installation phase. If.4. ########################################### 1:ocfs2-tools ########################################### 2:ocfs2-2.single processor / x86 machines..el5.4.el5 ########################################### 3:ocfs2console ########################################### [100%] [ 33%] [ 67%] [100%] Disable SELinux (RHEL4 U2 and higher) Users of RHEL4 U2 and higher (CentOS 5.2-1.el5-1. however. If you did not follow the instructions to disable the SELinux option during the installation of CentOS (or if you simply want to verify it is truly disable).18-128.2-1.rpm Preparing.4..6. If you are using RHEL4 U2 or higher (which includes us since we are using CentOS 5.3 is based on RHEL5 U3) are advised that OCFS2 currently does not work with SELinux enabled.I will be installing the OCFS2 files onto two .18-128.

click the SELinux tab and select the "Disabled" option. Simply acknowledge this warning by clicking "Yes". you will be presented with a warning dialog. After clicking [OK]. Your screen should now look like the following after disabling the SELinux option: .Figure 16: Security Level Configuration Opening Screen / Firewall Disabled Now.

conf file is not present.Figure 17: SELinux Disabled If you needed to disable SELinux in this section on any of the nodes. but will also create and start the cluster stack O2CB. we will not only create and configure the /etc/ocfs2/cluster. In other words. To avoid a network delay being interpreted as a node disappearing on the net which could lead to a node-self-fencing.conf file using ocfs2console. a private interconnect is recommended. The IP address. however. (as will be the case in our example). The next step is to generate and configure the /etc/ocfs2/cluster. It is safe to use the same private interconnect for both Oracle RAC and OCFS2. need not be the one associated with that hostname. OCFS2 will not attempt to match the node name (hostname) with the specified IP address. it does require the nodes to be alive on the network and sends regular keepalive packets to ensure that they are.0) for all of its network traffic as recommended by Oracle. In this section. A popular question then is what node name should be used when adding nodes to an OCFS2 cluster and should it be related to the IP address? When adding nodes to an OCFS2 cluster.2. While OCFS2 does not take much bandwidth.168. those nodes will need to be rebooted to implement the change. the node name being entered must match the hostname of the machine or the OCFS2 console will throw an error. the ocfs2console tool will create this file along with a new cluster stack service (O2CB) with a . SELinux must be disabled before you can continue with configuring OCFS2! Configure OCFS2 OCFS2 will be configured to use the private network (192.conf file on both Oracle RAC nodes in the cluster. The easiest way to accomplish this is to run the GUI tool ocfs2console. When the /etc/ocfs2/cluster. any valid IP address on that node can be used.

2. This will start the OCFS2 Cluster Stack (Figure 19).168. I added both nodes using linux1 / 192.100 for the first node and linux2 / 192.. perform the following steps: 1. Leave the IP Port set to its default value of 7777.].default cluster name of ocfs2. o In the "Add Node" dialog.2. o This will bring up the "Add Node" dialog. On the "Node Configuration" dialog. ..101 for the second node. Acknowledge this Information dialog box by clicking [Close]. Select [Cluster] -> [Configure Nodes. This will need to be done on both Oracle RAC nodes in the cluster as the root user account: # ocfs2console & This will bring up the GUI as shown below: Figure 18: ocfs2console Screen Using the ocfs2console GUI tool.168. click the [Add] button. enter the Host name and IP address for the first node in the cluster.2. You will then be presented with the "Node Configuration" dialog. In my example.

Note: The node name you enter "must" match the hostname of the machine and the IP addresses will use the private interconnect. o Click [Close] on the "Node Configuration" dialog. 3. o Click [Apply] on the "Node Configuration" dialog . Figure 19: Starting the OCFS2 Cluster Stack The following dialog shows the OCFS2 settings I used for the node linux1 and linux2: Figure 20: Configuring Nodes for OCFS2 See the Troubleshooting section if you get the error: o2cb_ctl: Unable to access cluster service while creating node After exiting the ocfs2console. exit the application using [File] -> [Quit].conf similar to the following. you will have a /etc/ocfs2/cluster.All nodes should now be "Active" as shown in Figure 20. After verifying all values are correct. This needs to be performed on both Oracle RAC nodes in the cluster. This process needs to be completed on both Oracle RAC nodes in the cluster and the OCFS2 configuration file should be exactly the same for both of the nodes: .

d/o2cb status Driver for "configfs": Loaded Filesystem "configfs": Mounted Driver for "ocfs2_dlmfs": Loaded Filesystem "ocfs2_dlmfs": Mounted Checking O2CB cluster ocfs2: Online Heartbeat dead threshold = 31 Network idle timeout: 30000 Network keepalive delay: 2000 Network reconnect delay: 2000 . O2CB.d/o2cb)./etc/ocfs2/cluster.conf node: ip_port = 7777 ip_address = 192.168.2.100 number = 0 name = linux1 cluster = ocfs2 node: ip_port = 7777 ip_address = 192.101 number = 1 name = linux2 cluster = ocfs2 cluster: node_count = 2 name = ocfs2 O2CB Cluster Service Before we can do anything with OCFS2 like formatting or mounting the file system.conf HB: Heart beat service that issues up/down notifications when nodes join or leave the cluster TCP: Handles communication between the nodes DLM: Distributed lock manager that keeps track of all locks. we need to first have OCFS2's cluster stack. running (which it will be as a result of the configuration process performed above). The following commands are for documentation purposes only and do not need to be run when installing and configuring OCFS2 for this article!           /etc/init. its owners and status CONFIGFS: User space driven configuration file system mounted at /config DLMFS: User space interface to the kernel space DLM All of the above cluster services have been packaged in the o2cb system service (/etc/init.168.2. Here is a short listing of some of the more useful commands and options for the o2cb system service. The stack includes the following services:       NM: Node Manager that keep track of all the nodes in the cluster.

d/o2cb unload Unmounting ocfs2_dlmfs filesystem: OK Unloading module "ocfs2_dlmfs": OK Unmounting configfs filesystem: OK Unloading module "configfs": OK The above command will unload all OCFS2 modules.     /etc/init.2.d/o2cb load Loading module "configfs": OK Mounting configfs filesystem at /config: OK Loading module "ocfs2_nodemanager": OK Loading module "ocfs2_dlm": OK Loading module "ocfs2_dlmfs": OK Mounting ocfs2_dlmfs filesystem at /dlm: OK Loads all OCFS2 modules.  /etc/init. If however you are using a release of OCFS2 prior to 1. ocfs2. Set the on-boot properties as follows: . This bug was fixed in release 1.Checking O2CB heartbeat: Not active  /etc/init. Configure O2CB to Start on Boot and Adjust O2CB Heartbeat Threshold You now need to configure the on-boot properties of the OC2B driver so that the cluster stack services will start on each boot. please see the Troubleshooting section for a workaround to this bug.       /etc/init. a bug existed where the driver would not get loaded on each boot even after configuring the on-boot properties to do so.1 of OCFS2 and does not need to be addressed in this article.1.d/o2cb offline ocfs2 Stopping O2CB cluster ocfs2: OK The above command will offline the cluster we created. Perform the following on both Oracle RAC nodes in the cluster: With releases of OCFS2 prior to 1.d/o2cb online ocfs2 Starting O2CB cluster ocfs2: OK The above command will online the cluster we created. ocfs2.2. You will also be adjusting the OCFS2 Heartbeat Threshold from its default setting of 31 to 61.2.1.

Ctrl-C will abort.d/o2cb offline ocfs2 # /etc/init. use the menu [Tasks] . I will be executing all commands in this section from linux1 only. creating the OCFS2 file system should only be executed on one of nodes in the RAC cluster. The format operation needs the cluster to be online. we created the directory /u02 under the section Create Mount Point for OCFS2 / Clusterware which will be used as the mount point for the OCFS2 cluster file system.ocfs2. however.[Format]. If the O2CB cluster is offline.d/o2cb unload # /etc/init. From the ocfs2console utility. The following questions will determine whether the driver is loaded on boot. Note that it is possible to create and mount the OCFS2 file system using either the GUI tool ocfs2console or the command-line tool mkfs. we can use the Oracle executable mkfs. We can now start to make use of the iSCSI volume we partitioned for OCFS2 in the section "Create Partitions on iSCSI Volumes". The instructions below. Earlier in this document. as it needs to ensure that the volume is not mounted on some other node in the cluster. start it.ocfs2. The current values will be shown in brackets ('[]'). For the purpose of this example. Hitting without typing an answer will keep that current value. Load O2CB driver on boot (y/n) [n]: y Cluster stack backing O2CB [o2cb]: o2cb Cluster to start on boot (Enter "none" to clear) [ocfs2]: ocfs2 Specify heartbeat dead threshold (>=7) [31]: 61 Specify network idle timeout in ms (>=5000) [30000]: 30000 Specify network keepalive delay in ms (>=1000) [2000]: 2000 Specify network reconnect delay in ms (>=2000) [2000]: 2000 Writing O2CB configuration: OK Loading filesystem "configfs": OK Mounting configfs filesystem at /sys/kernel/config: OK Loading filesystem "ocfs2_dlmfs": OK Mounting ocfs2_dlmfs filesystem at /dlm: OK Starting O2CB cluster ocfs2: OK Format the OCFS2 File System Unlike the other tasks in this section. demonstrate how to create the OCFS2 file system using the command-line tool mkfs. This section contains the commands to create and mount the file system to be used for the Cluster Manager.d/o2cb configure Configuring the O2CB driver.# /etc/init. To create the file system. I run the following command only from linux1 as the root user account using the local SCSI device name mapped to .ocfs2. This will configure the on-boot properties of the O2CB driver.

Also note that I specified a label named "oracrsfiles" which will be referred to when mounting or un-mounting the volume: # mkfs.2 Cluster stack: classic o2cb Filesystem label=oracrsfiles Block size=4096 (bits=12) Cluster size=32768 (bits=15) Volume size=2145943552 (65489 clusters) (523912 blocks) 3 cluster groups (tail covers 977 clusters. run the following checks to ensure the file system is mounted correctly. we can mount it. then I'll show how to include it in the /etc/fstab to have it mount on each boot. however.nointr -L "oracrsfiles" /u02 If the mount was successful.gid=5. Mounting the cluster file system will need to be performed on both Oracle RAC nodes in the cluster as the root user account using the OCFS2 label oracrsfiles! First. Let's first do it using the command-line. This should be performed on both nodes in the RAC cluster: # mount /dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw. Remember that this needs to be performed as the root user account on both nodes in the cluster: # mount -t ocfs2 -o datavolume.ocfs2 successful Mount the OCFS2 File System Now that the file system is created. you will simply get your prompt back.mode=620) /dev/hda1 on /boot type ext3 (rw) .4.the iSCSI volume for crs — /dev/iscsi/crs/part1. here is how to manually mount the OCFS2 file system from the command-line. Use the mount command to ensure that the new file system is really mounted. rest cover 32256 clusters) Journal size=67108864 Initial number of node slots: 4 Creating bitmaps: done Initializing superblock: done Writing system files: done Writing superblock: done Writing backup superblock: 1 block(s) Formatting Journals: done Formatting slot map: done Writing lost+found: done mkfs. We should.ocfs2 1.ocfs2 -b 4K -C 32K -N 4 -L oracrsfiles /dev/iscsi/crs/part1 mkfs.

should not be mounted with this mount option.1. you mounted the newly created file system using the OCFS2 label "oracrsfiles". Cluster Registry (OCR). Oracle plans to add support for a global heartbeat. Any other type of volume. Oracle database users must mount any volume that will contain the Voting Disk file.tmpfs on /dev/shm type tmpfs (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) domo:Public on /domo type nfs (rw.121) configfs on /sys/kernel/config type configfs (rw) ocfs2_dlmfs on /dlm type ocfs2_dlmfs (rw) /dev/sdd1 on /u02 type ocfs2 (rw. You installed the OCFS2 software packages which will be used to store the shared files needed by Cluster Manager. Configure OCFS2 to Mount Automatically at Startup Let's take a look at what you've have done so far. The nointr option ensures that the I/O's are not interrupted by signals. Redo logs. which will make most mounts instant. However. you loaded the OCFS2 module into the kernel and then formatted the clustered file system. you should still check those options by running the following on both Oracle RAC nodes in the cluster as the root user account: . Start by adding the following line to the /etc/fstab file on both Oracle RAC nodes in the cluster: LABEL=oracrsfiles /u02 ocfs2 _netdev.nointr. If you have been following along with the examples in this article.addr=192. including an Oracle home (which I will not be using for this article). The _netdev mount option is a must for OCFS2 volumes.datavolume.nointr 0 0 Notice the "_netdev" option for mounting this file system. This section walks through the steps responsible for mounting the new OCFS2 file system each time the machine(s) are booted using its label. Now. This mount option indicates that the volume is to be mounted after the network is started and dismounted before the network is shutdown. Archive logs and Control files with the datavolume mount option so as to ensure that the Oracle processes open the files with the o_direct flag. Why does it take so much time to mount the volume? It takes around 5 seconds for a volume to mount.ko kernel module is being loaded and that the file system will be mounted during the boot process. Finally. After going through the install.datavolume. In a later release. let's make sure that the ocfs2.168. Data files._netdev. the actions to load the kernel module and mount the OCFS2 file system should already be enabled.heartbeat=local) Please take note of the datavolume option I am using to mount the new file system. It does so as to let the heartbeat thread stabilize.

Let's fix that: # chown oracle:oinstall /u02 # chmod 775 /u02 Let's now go back and re-check that the permissions are correct for both Oracle RAC nodes in the cluster: # ls -ld /u02 drwxrwxr-x 3 oracle oinstall 3896 Aug 16 21:17 /u02 Create Directory for Oracle Clusterware Files The last mandatory task is to create the appropriate directory on the new OCFS2 file system that will be used for the Oracle Clusterware shared files. 2:on 5:on 6:off Check Permissions on New OCFS2 File System Use the ls command to check ownership. the oracle user account (and the oinstall group) will not be able to write to this directory. The following tasks only need to be executed on one of nodes in the RAC cluster. # mkdir -p /u02/oradata/racdb # chown -R oracle:oinstall /u02/oradata # chmod -R 775 /u02/oradata # ls -l /u02/oradata total 0 drwxrwxr-x 2 oracle oinstall 3896 Aug 16 21:33 racdb . I will be executing all commands in this section from linux1 only. The following tasks only need to be executed on one of nodes in the RAC cluster. We will also modify the permissions of this new directory to allow the "oracle" owner and group "oinstall" read/write access.# chkconfig --list o2cb o2cb 0:off 1:off 3:on 4:on The flags that I have marked in bold should be set to "on". Let's first check the permissions: # ls -ld /u02 drwxr-xr-x 3 root root 3896 Aug 16 21:17 /u02 As you can see from the listing above. I will be executing all commands in this section from linux1 only. The permissions should be set to 0775 with owner " oracle" and group "oinstall".

two disk groups). In this article. You will be required to create RAW devices for all disk partitions used by ASM.Install and Configure Automatic Storage Management (ASMLib 2. ASM will be used as the shared file system and volume manager for Oracle Database files (data. however. ASM with Standard Linux I/O: This method does not make use of ASMLib. archived redo logs) as well as the Flash Recovery Area. I will be using the "ASM with ASMLib I/O" method. is ASMLib required for ASM? Not at all. In this article. Those performance metrics and testing details are out of scope of this article and therefore will not be discussed. ASM is built into the Oracle kernel and can be used for both single and clustered instances of Oracle. we will install and configure ASMLib 2. I plan on performing several tests in the future to identify the performance gains in using ASMLib. Automatic Storage Management simplifies database administration by eliminating the need for the DBA to directly manage potentially thousands of Oracle database files requiring only the management of groups of disks allocated to the Oracle Database. even with rapidly changing data usage patterns.0 which is a support library for the Automatic Storage Management (ASM) feature of the Oracle Database. All of the files and directories to be used for Oracle will be contained in a disk group — (or for the purpose of this article. Keep in mind that ASMLib is only a support library for the ASM software. ASM automatically performs load balancing in parallel across all available disk drives to prevent hot spots and maximize performance. In fact. will only need to be performed on a single node within the cluster. ASMLib allows an Oracle Database using ASM more efficient and capable access to the disk groups it is using. Introduction In this section. So. The ASM software gets installed as part of the Oracle Database installation which occurs later in this guide. Oracle states in Metalink Note 275315. online redo logs.0) Most of the installation and configuration procedures in this section should be performed on both of the Oracle RAC nodes in the cluster! Creating the ASM disks.1 that "ASMLib was provided to enable ASM I/O to Linux disks without the limitations of the standard UNIX I/O API". control files. . there are two different methods to configure ASM on Linux:   ASM with ASMLib I/O: This method creates all Oracle database files on raw block devices managed by ASM using ASMLib calls. Oracle database files are created on raw character devices managed by ASM using standard Linux I/O system calls. RAW devices are not required with this method as ASMLib works with block devices.

el5.18-128.rpm .i386. If you would like to learn more about Oracle ASMLib 2.(Userspace library) 64-bit (x86_64) Installations oracleasm-2.5-1. you will need to create one.x86_64.rpm .0.18-128.rpm .rpm .(Package for xen kernel) Next.6.(Package for xen kernel) Next. download the following ASMLib tools: oracleasm-support-2.i686.6.18128. download the following ASMLib tools: oracleasm-support-2.0.0.0 drivers while finishing off the section with a demonstration of how to create the ASM disks.rpm .el5 #1 SMP Wed Jan 21 10:44:23 EST 2009 i686 i686 i386 GNU/Linux If you do not currently have an account with Oracle OTN. visit http://www.0.5-1.6.0.i686.el5. we need to download the appropriate version of the ASMLib driver for the Linux kernel which in my case is kernel 2.rpm .0.el5-2.0 libraries and the driver from OTN.el5 running on two single processor / x86 machines: # uname -a Linux linux1 2.5-1.(Package for PAE kernel) oracleasm-2.x86_64.18-128.We start this section by first downloading the ASMLib drivers (ASMLib Release 2.i386. the latest release of the ASMLib driver is 2.1.com/technology/tech/linux/asmlib/ Download the ASMLib 2. We will then install and configure the ASMLib 2.rpm .4-1.el5.el5. This is a FREE account! Oracle ASMLib Downloads for Red Hat Enterprise Linux Server 5 32-bit (x86) Installations oracleasm-2.el5.el5xen-2.rpm .0.el5.(Package for default kernel) oracleasm-2. Like the Oracle Cluster File System.el5.0.6.3-1.i686.el5.(Userspace library) .3-1.0) specific to our Linux kernel. At the time of this writing.el5xen-2.el5.6.el5PAE-2.18-128.(Driver support files) oracleasmlib-2.0.4-1.el5-2.oracle.(Driver support files) oracleasmlib-2.6.x86_64.5-1.(Package for default kernel) oracleasm-2.1.5-1.18-128.6.x86_64.18-128.5-1.rpm .0 Packages We start this section by downloading the latest ASMLib 2.

0 Packages This installation needs to be performed on both nodes in the RAC cluster as the root user account: # rpm -Uvh oracleasm-2. and the flash recovery area. control files.rpm \ oracleasm-support-2.Install ASMLib 2. we now need to configure and load the ASM kernel module.0 Packages for Linux.0.el5.18-128.rpm \ oracleasmlib-2. This will configure the on-boot properties of the Oracle ASM library driver. On the other Oracle RAC node.el########################################### 3:oracleasmlib ########################################### [100%] [ 33%] [ 67%] [100%] Configure and Load the ASMLib 2. I will be running these commands on linux1.4-1. we configured (partitioned) four iSCSI volumes to be used by ASM.0.. . This task needs to be run on both nodes in the RAC cluster as the root user account: # /etc/init. you will need to perform a scandisk to recognize the new volumes.3-1.i386.6. ########################################### 1:oracleasm-support ########################################### 2:oracleasm-2. ASM will be used for storing Oracle database files like online redo logs. you should then run the oracleasm listdisks command on both Oracle RAC nodes to verify that all ASM disks were created and available. Use the local device names that were created by udev when configuring the four ASM volumns. The following questions will determine whether the driver is loaded on boot and what permissions it will have.d/oracleasm configure Configuring the Oracle ASM library driver.el5.1. Ctrl-C will abort.rpm Preparing.5-1. When that is complete. Hitting without typing an answer will keep that current value. Default user to own the driver interface []: oracle Default group to own the driver interface []: oinstall Start Oracle ASM library driver on boot (y/n) [n]: y Scan for Oracle ASM disks on boot (y/n) [y]: y Writing Oracle ASM library driver configuration: done Initializing the Oracle ASMLib driver: [ OK ] Scanning the system for Oracle ASMLib disks: [ OK ] Create ASM Disks for Oracle Creating the ASM disks only needs to be performed from one node in the RAC cluster as the root user account.0 Packages Now that we downloaded and installed the ASMLib 2. In the section "Create Partitions on iSCSI Volumes".6..i686. The current values will be shown in brackets ('[]').i386.el5-2.el5. archived redo log files. database files.18-128.

try listing all ASM disks that were used by the previous install using: # /etc/init.d/oracleasm listdisks VOL1 VOL2 VOL3 VOL4 As you can see.d/oracleasm deletedisk Removing ASM disk "VOL4" [ OK ] VOL1 VOL2 VOL3 VOL4 To create the ASM disks using the iSCSI target names to local device name mappings.d/oracleasm createdisk VOL3 /dev/iscsi/asm3/part1 Marking disk "VOL3" as an ASM disk: [ OK ] # /etc/init. If you do receive a failure. the results show that I have four ASM volumes already defined.d/oracleasm createdisk VOL2 /dev/iscsi/asm2/part1 Marking disk "VOL2" as an ASM disk: [ OK ] # /etc/init.d/oracleasm scandisks Scanning the system for Oracle ASMLib disks: [ OK ] We can now test that the ASM disks were successfully created by using the following command on both nodes in the RAC cluster as the root user account: # /etc/init. go ahead and remove them using the following commands. you must perform a scandisk to recognize the new volumes: # /etc/init.d/oracleasm deletedisk Removing ASM disk "VOL3" [ OK ] # /etc/init. If you have the four volumes already defined from a previous run.If you are repeating this article using the same hardware (actually.d/oracleasm deletedisk Removing ASM disk "VOL2" [ OK ] # /etc/init.d/oracleasm createdisk VOL4 /dev/iscsi/asm4/part1 Marking disk "VOL4" as an ASM disk: [ OK ] On all other nodes in the RAC cluster.d/oracleasm listdisks VOL1 VOL2 VOL3 . you may get a failure when attempting to create the ASM disks. the same shared logical drives).d/oracleasm deletedisk Removing ASM disk "VOL1" [ OK ] # /etc/init.d/oracleasm createdisk VOL1 /dev/iscsi/asm1/part1 Marking disk "VOL1" as an ASM disk: [ OK ] # /etc/init. use the "oracleasm createdisk" commands (below) to create the new volumes. After removing the previously created volumes. # /etc/init. type the following: # /etc/init.

VOL4 Download Oracle RAC 10g Software The following download procedures only need to be performed on one node in the cluster! Overview The next step is to download and extract the required Oracle software packages from the Oracle Technology Network (OTN): If you do not currently have an account with Oracle OTN. linux1. You will perform all Oracle software installs from this machine.html 64-bit (x86_64) Installations http://www. A full description of the license agreement is available on OTN. In this example.oracle. Downloading and Extracting the Software . 32-bit (x86) Installations http://www. This is a FREE account! Oracle offers a development and testing license free of charge. is provided and the license does not permit production use. you will be downloading the required Oracle software to linux1 and saving them to /home/oracle/orainstall.oracle. Log in to the node that you will be performing all of the Oracle installations from ( linux1) as the "oracle" user account.com/technology/software/products/database/oracle10g/htdocs/10201linuxsoft.html You will be downloading and extracting the required software from Oracle to only one of the Linux nodes in the cluster—namely. No support. however.com/technology/software/products/database/oracle10g/htdocs/10201linx8664soft. The Oracle installer will copy the required software packages to all other nodes in the RAC configuration using the remote access method we setup in the section Configure RAC Nodes for Remote Access using SSH. you will need to create one.

0.zip Then extract the Oracle Database Software: $ cd /home/oracle/orainstall $ unzip 10201_database_linux32.2. we will use /home/oracle/orainstall.2. As the oracle user account.0) software for either Linux x86 or Linux x86-64. It is responsible for performing various system checks to assist you with confirming the Oracle RAC nodes are properly configured for Oracle Clusterware and Oracle Real Application Clusters installation. it is highly recommended to run the Cluster Verification Utility (CVU) to verify the hardware and software configuration.oracle mkdir -p /home/oracle/orainstall cd /home/oracle/orainstall unzip 10201_clusterware_linux32. Oracle Database 10g Release 2 (10. In this example.zip Pre-Installation Tasks for Oracle10g Release 2 Perform the following checks on both Oracle RAC nodes in the cluster! Before installing the Oracle Clusterware and Oracle RAC software. download the Oracle Clusterware Release 2 (10.1.2. Note that the CVU is also run automatically at the end of the Oracle Clusterware installation as part of the Configuration Assistants process. Prerequisites for Using Cluster Verification Utility Install cvuqdisk RPM (RHEL Users Only) .1.zip Finally. All downloads are available from the same page. CVU is a command-line utility provided on the Oracle Clusterware installation media.0. The CVU only needs to be run from the node you will be performing the Oracle installations from (linux1 in this article). extract the three packages you downloaded to a temporary directory. extract the (optional) Oracle Companion CD Software: $ cd /home/oracle/orainstall $ unzip 10201_companion_linux32.0).First.0.1.0). Extract the Oracle Clusterware package as follows: # $ $ $ su . and Oracle Database 10g Companion CD Release 2 (10.

This means you will need to install the cvuqdisk RPM to both linux1 and linux2. Locate and copy the cvuqdisk RPM from linux1 to linux2 as the "oracle" user account: $ ssh linux2 "mkdir -p /home/oracle/orainstall/clusterware/rpm" $ scp /home/oracle/orainstall/clusterware/rpm/cvuqdisk-1.ssh/id_rsa) Checking Pre-Installation Tasks for CRS with CVU . For the purpose of this article. we need to set an environment variable named CVUQDISK_GRP to point to the group that will own the cvuqdisk utility.The first pre-requisite for running the CVU pertains to users running Oracle Enterprise Linux. To enable user equivalence for the current terminal shell session. Red Hat Linux. CentOS.ssh/id_rsa (/home/oracle/. then you must download and install the package cvuqdisk to both of the Oracle RAC nodes in the cluster.0.0. Note that before installing the cvuqdisk RPM. Without cvuqdisk. perform the following steps remembering to enter the pass phrase for each key that you generated when prompted: # su .1-1. CVU will be unable to discover shared disks and you will receive the error message "Package cvuqdisk not installed" when you run CVU.rpm Preparing packages for installation.1 root oinstall 4168 Jun 2 2005 /usr/sbin/cvuqdisk Verify Remote Access / User Equivalence The CVU should be run from linux1 — the node we will be performing all of the Oracle installations from. the Oracle Clusterware media was extracted to the /home/oracle/orainstall/clusterware directory on linux1.oracle $ exec /usr/bin/ssh-agent $SHELL $ /usr/bin/ssh-add Enter passphrase for /home/oracle/.0. export CVUQDISK_GRP # rpm -iv cvuqdisk-1. log in as the oracle user account and verify remote access / user equivalence is configured to all nodes in the cluster. If you are using any of the above listed operating systems. and SuSE. The default group is oinstall which is the group we are using for the oracle UNIX user account in this article..rpm linux2:/home/oracle/orainstall/clusterware/rpm Perform the following steps as the "root" user account on both Oracle RAC nodes to install the cvuqdisk RPM: # cd /home/oracle/orainstall/clusterware/rpm # CVUQDISK_GRP=oinstall. When using the secure shell method.1-1. Before running CVU. The cvuqdisk RPM can be found on the Oracle Clusterware installation media in the rpm directory..1-1 # ls -l /usr/sbin/cvuqdisk -rwsr-x--. cvuqdisk-1. user equivalence will need to be enabled for the terminal shell session before attempting to run the CVU.ssh/id_rsa: xxxxx Identity added: /home/oracle/.

96.128 compat-gcc-c++-7. This can be considered close enough and safe to continue with the installation.128 compat-libstdc++-7.sh stage -pre crsinst -n linux1. I highly recommend both Oracle RAC nodes have 2GB of RAM memory or higher for performance reasons. you may receive an error during the "Total memory" check: Check: Total memory Node Name Available -----------. Required -----------------------1GB (1048576KB) 1GB (1048576KB) Comment ---------failed failed As you can see from the output above. the requirement is for 1GB of memory (1048576 KB).168.65MB (1033880KB) Result: Total memory check failed.100 ERROR: Could not find a suitable set of interfaces for VIPs.168. As I mentioned earlier in this article.3-2.1: Suitable interfaces for the private interconnect on subnet "192.3-2. Although your system may have 1GB of memory installed in each of the Oracle RAC nodes. The first error is with regards to finding a suitable set of interfaces for VIPs which can be safely ignored.128 .3-2.zip -d jdk14 CV_HOME=/home/oracle/orainstall/clusterware/cluvfy. this error can be safely ignored.0.168. export CV_HOME CV_JDKHOME=/home/oracle/orainstall/clusterware/cluvfy/jdk14. If your system only has 1GB of RAM memory. we can now check that all pre-installation tasks for Oracle Clusterware are completed by executing the following command as the "oracle" UNIX user account (with user equivalence enabled) from linux1: $ $ $ $ $ $ cd /home/oracle/orainstall/clusterware/cluvfy mkdir -p jdk14 unzip jrepack.96. The last set of errors that can be ignored deal with specific RPM package versions that are not required with CentOS 5. Result: Node connectivity check failed.14-5 compat-gcc-7.2. This is a bug documented in Metalink Note 338924.96. export CV_JDKHOME .96.Once all prerequisites for running the CVU utility have been met.linux2 -verbose Review the CVU report.0": linux2 eth1:192.-----------------------linux2 1009. As documented in the note.3-2.2.2.65MB (1033880KB) linux1 1009. For example:      compat-db-4. the Linux kernel is calculating it to be 1033880 KB which comes out to be 14696 KB short.128 compat-libstdc++-devel-7.101 linux1 eth1:192. Note that there are several errors you may ignore in this report./runcluvfy.

Another reason for this error was documented by Bane Radulovic at Oracle Corporation.4.3-2.4.6-4 compat-libstdc++-33-3.linux2 -verbose Review the CVU report..linux1.sourceforge.linux2. For example. Again.3-61 Checking the Hardware and Operating System Setup with CVU The next CVU check to run will verify the hardware and operating system setup." portion of the report: Checking shared storage accessibility. compat-libstdc++-devel-7. in CentOS 5 Update 3.sh stage -post hwos -n linux1. the check for finding a suitable set of interfaces for VIPs will fail and can be safely ignored Also note you may receive warnings in the "Checking shared storage accessibility.. this too can be safely ignored. The first came from Metalink indicating that cluvfy currently does not work with devices other than SCSI devices.33 [i686-redhat-linux-gnu] Copyright (C) 2002-4 Bruce Allen Home page is http://smartmontools.linux1. the check itself may fail. His research shows that CVU calls smartclt on Linux. a check against /dev/sde shows: # /usr/sbin/smartctl -i /dev/sde smartctl version 5.6-4 compat-gcc-34-c++-3.linux1".linux1.. For example.2.128 While these specific packages are listed as missing in the CVU report. If this occurs. no workaround exists other than to use manual methods for detecting shared devices. run the following as the "oracle" UNIX user account from linux1: $ cd /home/oracle/orainstall/clusterware/cluvfy $ .linux2.. these would be:    compat-gcc-34-3. While we know the disks are visible and shared from both of our Oracle RAC nodes in the cluster.linux2. please ensure that the correct versions of the compat-* packages are installed on both of the Oracle RAC nodes in the cluster.linux2./runcluvfy. As with the previous check (pre-installation tasks for CRS). Several reasons for this have been documented.net/ Device: Openfile Virtual disk Version: 0 Serial number: Device type: disk Local Time is: Mon Sep 3 02:02:53 2007 EDT Device supports SMART and is Disabled . At the time of this writing.96.linux1. This would include devices like EMC PowerPath and volume groups like those from Openfiler. WARNING: Unable to determine the sharedness of /dev/sde on nodes: linux2. and the problem is that smartclt does not return the serial number from our iSCSI devices.linux1 Shared storage check failed on nodes "linux2.

you will be asked for the nodes involved and to configure in the RAC cluster. what exactly is the Oracle Clusterware responsible for? It contains all of the cluster and database configuration metadata along with several system management features for RAC. the Oracle Database software only needs to be run from one node. This is the only node we need to perform the install from. It allows the DBA to register and invite an Oracle instance (or instances) to the cluster.Temperature Warning Disabled or Not Supported At the time of this writing. Oracle Clusterware Shared Files The two shared files (actually file groups) used by Oracle Clusterware will be stored on the Oracle Cluster File System.the Oracle Clusterware." If the heartbeat fails for any of the nodes. Install Oracle Clusterware 10g Software Perform the following installation procedures from only one of the Oracle RAC nodes in the cluster (linux1)! The Oracle Clusterware software will be installed to both of the Oracle RAC nodes in the cluster by the Oracle Universal Installer. During the installation of Oracle Clusterware. The OUI will copy the software packages to all nodes configured in the RAC cluster. it checks with the Oracle Clusterware configuration files (on the shared disk) to distinguish between a real node failure and a network failure. often called the "heartbeat. the OUI will copy the required software to all nodes using the remote access we configured in the section "Configure RAC Nodes for Remote Access using SSH". Oracle Clusterware will send messages (via a special ping operation) to all nodes configured in the cluster. The two shared Oracle Clusterware file groups are:  Oracle Cluster Registry (OCR) . it is unknown if the Openfiler developers have plans to fix this. the Oracle Universal Installer (OUI) used to install the Oracle Database software (next section) will automatically recognize these nodes. we downloaded and extracted the install files for Oracle Clusterware to linux1 in the directory /home/oracle/orainstall/clusterware. After installing Oracle Clusterware. During the installation phase. In a previous section. So. Overview We are ready to install the Cluster part of the environment . During normal operation. Release 2 (OFCS2) we created earlier. Like the Oracle Clusterware install you will be performing in this section.

If you are using a remote client to connect to the node performing the installation (SSH or Telnet to linux1 from a workstation configured with an X Server). clients can connect from any host Log in as the oracle User Account and Set DISPLAY (if necessary) # su . the ASM instance would need to be run first. Finally. The problem is that these files need to be in place and accessible BEFORE any Oracle instances can be started. log in as the oracle user account.oracle $ $ $ $ # IF YOU ARE USING A REMOTE CLIENT TO CONNECT TO THE # NODE PERFORMING THE INSTALL DISPLAY=<your local workstation>:0. verify remote access / user equivalence to all nodes in the cluster: Verify Server and Enable X Server Access # hostname linux1 # xhost + access control disabled.0 export DISPLAY Verify Remote Access / User Equivalence Verify you are able to run the Secure Shell commands (ssh or scp) on the Linux server you will be running the Oracle Universal Installer from against all other Linux servers in the cluster without being prompted for a password. or another vendor's clustered file system. user equivalence will need to be enabled for the terminal shell session before attempting to run the OUI. For ASM to be available. you should first verify you are logged onto the server you will be running the installer from (i. Verifying Terminal Shell Environment Before starting the Oracle Universal Installer. shared RAW devices. Next. When using the secure shell method. Also note that the two shared files could be stored on the OCFS2. . File 1 : /u02/oradata/racdb/OCRFile File 2 : /u02/oradata/racdb/OCRFile_mirror Size : (2 * 100MB) = 200M CRS Voting Disk o File 1 : /u02/oradata/racdb/CSSFile o File 2 : /u02/oradata/racdb/CSSFile_mirror1 o File 3 : /u02/oradata/racdb/CSSFile_mirror2 o Size : (3 * 20MB) = 60MB o o o It is not possible to use Automatic Storage Management (ASM) for the two shared Oracle Clusterware files: Oracle Cluster Registry (OCR) or the CRS Voting Disk files. you will need to set the DISPLAY variable to point to your local workstation. To enable user equivalence for the current terminal shell session.e. linux1) then run the xhost command as root from the console to allow X Server connections.

ssh/id_rsa: xxxxx Identity added: /home/oracle/.1.0 Virtual Node Name linux1-vip linux2-vip Interface Type Public Private Specify Home Details Product-Specific Prerequisite Checks Starting with Oracle Database 10g Release 2 (10.perform the following steps remembering to enter the pass phrase for the RSA key you generated when prompted: $ exec /usr/bin/ssh-agent $SHELL $ /usr/bin/ssh-add Enter passphrase for /home/oracle/. you will need to manually verify the check that failed by clicking on the checkbox. For my installation. all checks passed with no problems. If any of the checks fail.2) with RAC.ssh/id_rsa) $ ssh linux1 "date.2. Cluster Name: crs Public Node Name Specify Cluster Configuration linux1 linux2 Interface Name Specify Network Interface Usage eth0 eth1 Specify Oracle Cluster Registry (OCR) Private Node Name linux1-priv linux2-priv Subnet 192.hostname" Tue Aug 18 11:44:49 EDT 2009 linux2 Installing Oracle Clusterware Perform the following tasks to install the Oracle Clusterware: $ cd ~oracle $ /home/oracle/orainstall/clusterware/runInstaller -ignoreSysPrereqs Screen Name Welcome Screen Specify Inventory directory and credentials Response Click Next Accept the default values: Inventory directory: /u01/app/oracle/oraInventory Operating System group name: oinstall Set the Name and Path for the ORACLE_HOME (actually the $ORA_CRS_HOME that I will be using in this article) as follows: Name: OraCrs10g_home Path: /u01/app/crs The installer will run through a series of checks to determine if the node meets the minimum requirements for installing and configuring the Oracle Clusterware software.ssh/id_rsa (/home/oracle/.168. Oracle Clusterware . Click Next to continue.hostname" Tue Aug 18 11:44:16 EDT 2009 linux1 $ ssh linux2 "date.0 192.168.

Run the following on both nodes in the RAC cluster to correct this problem: # chmod 644 /etc/oraInst. For the purpose of this example." If the permissions to /etc/oraInst. the following error will occur: "CRS is not installed on any of the nodes.it should be 0022. you must configure at least three voting disks.sh After the installation has completed. Navigate to the /u01/app/oracle/oraInventory directory and run orainstRoot.1 root root 63 Aug 19 10:55 /etc/oraInst. (starting with the node you are performing the install from). Note that to take advantage of the benefits of multiple voting disks. you could configure only one voting disk.sh on both nodes in the RAC cluster. By enabling multiple voting disk configuration.loc" are 644 (-rw-r--r--) and owned by root.sh.sh Within the same new console window on both Oracle RAC nodes in the cluster. CSS has been modified to allow you to configure CSS with multiple voting disks. enhancing cluster reliability.loc -rw-r--r-. the redundant voting disks allow you to configure a RAC database with multiple voting disks on independent shared physical disks.sh on both nodes before running root. This option facilitates the use of the iSCSI network protocol.1).(the location of the oraInventory directory cannot be determined). during the Oracle Clusterware post-installation process (while running the Oracle Clusterware Verification Utility). In 10g Release 1 (10. For the purpose of this example.loc # ls -l /etc/oraInst. and other Network Attached Storage (NAS) storage solutions. you will be prompted to run the orainstRoot.Location provides for the creation of a mirrored Oracle Cluster Registry (OCR) file. For example. Note: After executing the orainstRoot.sh and root. I did choose to mirror the voting disk by keeping the default option of "Normal Redundancy": Voting Disk Location: /u02/oradata/racdb/CSSFile Additional Voting Disk 1 Location: /u02/oradata/racdb/CSSFile_mirror1 Additional Voting Disk 2 Location: /u02/oradata/racdb/CSSFile_mirror2 Specify Voting Disk Location Summary Click Install to start the installation! orainstRoot. I did choose to mirror the OCR file by keeping the default option of "Normal Redundancy": Specify OCR Location: /u02/oradata/racdb/OCRFile Specify OCR Mirror Location: /u02/oradata/racdb/OCRFile_mirror Starting with Oracle Database 10g Release 2 (10. it is possible you didn't run orainstRoot. (starting with the node you are performing the install from). the umask setting may be off .loc Execute Configuration Scripts root.2) with RAC.sh script. verify the permissions of the file "/etc/oraInst. as the root user account. stay logged in as the root .loc are not set correctly. Also. Open a new console window on both Oracle RAC nodes in the cluster.sh on both nodes. Problems can occur during the installation of Oracle if the oracle user account does not have read permissions to this file .

unset LD_ASSUME_KERNEL by adding one line. If the Oracle Clusterware home directory is a sub-directory of the ORACLE_BASE directory (which should never be!).0: cannot open shared object file: No such file or directory Note that srvctl will produce similar output until the workaround described below is performed. Navigate to the /u01/app/crs directory and locate the root.so. Issue #1 At the end of root. .sh on the last node. To workaround Issue #1 above.sh on the Last Node will Fail! After receiving any of the errors described in this section. These warnings can be safely ignored.sh file on both nodes in the RAC cluster one at a time.sh script may take awhile to run. edit vipca (in the CRS bin directory on both nodes) to undo the setting of LD_ASSUME_KERNEL. when installed.19 export LD_ASSUME_KERNEL fi unset LD_ASSUME_KERNEL <<== Line to be added Similarly for srvctl (in both the CRS and. This error occurs because these releases of the Linux kernel fix an old bug in the Linux threading that Oracle worked around using LD_ASSUME_KERNEL settings in both vipca and srvctl.sh file for each node in the cluster . RDBMS and ASM bin directories on both nodes). add an unset command to ensure LD_ASSUME_KERNEL is not set as follows: if [ "$arch" = "i686" -o "$arch" = "ia64" ] then LD_ASSUME_KERNEL=2. Do not hit the OK button on the "Execute Configuration Scripts" dialog until all of the issues described in this section have been resolved. Running root. After the IF statement around line 120. The root. Run the root. please leave the OUI up.sh script on both nodes.user account. you will receive several warnings regarding permission while running the root. vipca will fail to run with the following error: Oracle CRS stack installed and running under init(1M) Running vipca(silent) for configuring nodeapps /u01/app/crs/jdk/jre//bin/java: error while loading shared libraries: libpthread. this workaround is no longer valid on OEL5 or RHEL5 or SLES10 hence the failures.4.(starting with the node you are performing the install from).

19 export LD_ASSUME_KERNEL unset LD_ASSUME_KERNEL <<== Line to be added Note: Remember to re-edit these files on both nodes: <ORA_CRS_HOME>/bin/vipca <ORA_CRS_HOME>/bin/srvctl <RDBMS_HOME>/bin/srvctl <ASM_HOME>/bin/srvctl # (If exists) after applying the 10. vipca will now fail to run with the following error if the VIP IP's are in a non-routable range [10.2.2. Please keep in mind that vipca is a GUI and will need to set your DISPLAY variable accordingly to your X server.x.2. If you try to run the above command. The goal to this workaround is to get the output of "$ORA_CRS_HOME/bin/oifcfg getif" to include both public and cluster_interconnect interfaces. This issue was raised with development and is fixed in the 10.0 eth1 192.4. you will notice it provides nothing which means we have some work to do! [root@linux2 bin]# /u01/app/crs/bin/oifcfg getif The first step is to identify the current interfaces and IP addresses: [root@linux2 ~]# /u01/app/crs/bin/oifcfg iflist eth0 192.3 patch sets.x.0 is the cluster_interconnect interface.2 or 10. Issue #2 After resolving the issue above. After working around Issue #1 above.2. . or 192.0. as these patchset will still include those settings unnecessary for OEL5 or RHEL5 or SLES10.168.168. 172.x.2. Also note that we are explicitly unsetting LD_ASSUME_KERNEL and not merely commenting out its setting to handle a case where the user has it set in their environment (login shell).around line 168 should look like this: LD_ASSUME_KERNEL=2.0.x.0 Remember during the Oracle Clusterware install that 192.x.1.1.(16-31).168.x]: [root@linux2 ~]# $ORA_CRS_HOME/bin/vipca Error 0(Native: listNetInterfaces:[3]) [Error 0(Native: listNetInterfaces:[3])] There are several ways to workaround Issue #2.168.x.4 patchsets.168.0 is my public interface while 192.0. manually re-run vipca (GUI) as root from the last node in which the error occurred.

0:public [root@linux2 ~]# /u01/app/crs/bin/oifcfg setif -global eth1/192.0 Summary: Click Finish Configuration Assistant Progress Dialog: Click OK after configuration is complete.2. Verify Oracle Clusterware Installation After the installation of Oracle Clusterware.0 global public eth1 192.0 global cluster_interconnect Manually Run vipca from Last Node After resolving all of the issues above. manually re-run vipca (GUI) as root from the last node in which the errors occurred.255.1. Configuration Results: Click Exit Go back to the OUI and acknowledge the "Execute Configuration scripts" dialog window.2.1.255. . Please keep in mind that vipca is a GUI and will need to set your DISPLAY variable accordingly to your X server: [root@linux2 ~]# /u01/app/crs/bin/vipca When the "VIP Configuration Assistant" appears.201 Subnet Mask: 255.255.0 Node Name: linux2 IP Alias Name: linux2-vip IP Address: 192.0:cluster_interconnect Let's know run the "/u01/app/crs/bin/oifcfg getif" command again to verify its output: [root@linux2 ~]# /u01/app/crs/bin/oifcfg getif eth0 192. we can run through several tests to verify the install was successful.168.1.168. End of installation At the end of the installation. this is how I answered the screen prompts: Welcome: Click Next Network interfaces: Select only the public interface . we can manually set the public / private interfaces accordingly using the "setif" option of the $ORA_CRS_HOME/bin/oifcfg command: [root@linux2 ~]# /u01/app/crs/bin/oifcfg setif -global eth0/192.168.1.255. exit from the OUI.168.Using this information.eth0 Virtual IPs for cluster nodes: Node Name: linux1 IP Alias Name: linux1-vip IP Address: 192. Run the following commands on both nodes in the RAC cluster as the oracle user account.168.200 Subnet Mask: 255.168.

create the database using the Database Configuration Assistant (DBCA) after the install.Check Cluster Nodes $ $ORA_CRS_HOME/bin/olsnodes -n linux1 1 linux2 2 Confirm Oracle Clusterware Function $ $ORA_CRS_HOME/bin/crs_stat -t -v Name Type R/RA F/FT Target State Host ---------------------------------------------------------------------ora.1.linux1.0) with RAC.2.d/init.linux2.ons application 0/3 0/0 ONLINE ONLINE linux2 ora. Overview After successfully installing the Oracle Clusterware software.gsd application 0/5 0/0 ONLINE ONLINE linux1 ora. instead.d/init.gsd application 0/5 0/0 ONLINE ONLINE linux2 ora.0.linux1. You will.vip application 0/0 0/0 ONLINE ONLINE linux2 Check CRS Status $ $ORA_CRS_HOME/bin/crsctl check crs CSS appears healthy CRS appears healthy EVM appears healthy Check Oracle Clusterware Auto-Start Scripts $ ls -l /etc/init.cssd /etc/init.evmd Install Oracle Database 10g Software Perform the following installation procedures from only one of the Oracle RAC nodes in the cluster (linux1)! The Oracle Database software will be installed to both of Oracle RAC nodes in the cluster by the Oracle Universal Installer. For the purpose of this example.crsd /etc/init.crs /etc/init.d/init.d/init.linux2.linux2.linux1.vip application 0/0 0/0 ONLINE ONLINE linux1 ora.* -r-xr-xr-x 1 root root 1951 -r-xr-xr-x 1 root root 4699 -r-xr-xr-x 1 root root 35379 -r-xr-xr-x 1 root root 3175 Aug Aug Aug Aug 19 19 19 19 11:03 11:03 11:03 11:03 /etc/init.ons application 0/3 0/0 ONLINE ONLINE linux1 ora. the next step is to install Oracle Database 10g Release 2 (10.d/init. you will forgo the "Create Database" option when installing the software. .

the Oracle 10g database software only needs to be run from one node. Note that you can utilize the same terminal shell session used in the previous section which in this case.ssh/id_rsa) $ ssh linux1 "date.Like the Oracle Clusterware install (previous section). Instructions for configuring CVU can be found in the section "Prerequisites for Using Cluster Verification Utility discussed earlier in this article.0 export DISPLAY Verify Remote Access / User Equivalence Verify you are able to run the Secure Shell commands (ssh or scp) on the Linux server you will be running the Oracle Universal Installer from against all other Linux servers in the cluster without being prompted for a password. user equivalence will need to be enabled for the terminal shell session before attempting to run the OUI. .hostname" Wed Aug 19 22:07:44 EDT 2009 linux2 Run the Oracle Cluster Verification Utility Before installing the Oracle Database Software. Verifying Terminal Shell Environment As discussed in the previous section. The OUI will copy the software packages to all nodes configured in the RAC cluster.ssh/id_rsa (/home/oracle/. (Install Oracle Clusterware 10g Software). To enable user equivalence for the current terminal shell session. you do not have to perform any of the actions described below with regards to setting up remote access and the DISPLAY variable: Log in as the oracle User Account and Set DISPLAY (if necessary) # su . When using the secure shell method. the terminal shell environment needs to be configured for remote access and user equivalence to all nodes in the cluster before running the Oracle Universal Installer.oracle $ $ $ $ # IF YOU ARE USING A REMOTE CLIENT TO CONNECT TO THE # NODE PERFORMING THE INSTALL DISPLAY=<your local workstation>:0.ssh/id_rsa: xxxxx Identity added: /home/oracle/. perform the following steps remembering to enter the pass phrase for the RSA key you generated when prompted: $ exec /usr/bin/ssh-agent $SHELL $ /usr/bin/ssh-add Enter passphrase for /home/oracle/.hostname" Wed Aug 19 22:07:13 EDT 2009 linux1 $ ssh linux2 "date. we should run the following database pre-installation check using the Cluster Verification Utility (CVU).

perform the following checks: Specify Hardware Cluster Installation Mode   Ensure the Oracle Clusterware is running on the node in question. export CV_HOME CV_JDKHOME=/home/oracle/orainstall/clusterware/cluvfy/jdk14. Install Oracle Database 10g Release 2 Software Install the Oracle Database 10g Release 2 software as follows: $ cd ~oracle $ /home/oracle/orainstall/database/runInstaller -ignoreSysPrereqs Screen Name Welcome Screen Response Click Next Select the type of installation you would like to perform: "Enterprise Edition". For the purpose of this article. Ensure you are able to reach the node in question from the node you are performing the installation from.linux2 -r 10gR2 -verbose Review the CVU report. Expected result: 3036MB . Click Select All to select all servers: linux1 and linux2. Available Product Components (Custom Database Installs Only) Select the components that you plan on using for your database environment. you will need to manually verify the check that failed by clicking on the checkbox. or "Custom".sh stage -pre dbinst -n linux1./runcluvfy. These two errors can be safely ignored... I selected the "Custom" option. select "Custom". The installer will run through a series of checks to determine if the node meets the minimum requirements for installing and configuring the Oracle database software.2. Product-Specific Prerequisite Checks It is possible to receive an error about the available swap space not meeting its minimum requirements: Checking available swap space requirements. If any of the checks fail. Note that this report will contain the same errors we received when checking preinstallation tasks for CRS — failure to find a suitable set of interfaces for VIPs and the failure to find specific RPM packages that are not required with CentOS 5. If the installation stops here and the status of any of the RAC nodes is "Node not reachable".$ $ $ $ cd /home/oracle/orainstall/clusterware/cluvfy CV_HOME=/home/oracle/orainstall/clusterware/cluvfy. "Standard Edition". Select Installation Type If you need advanced components like Oracle Label Security or if you simply want to customize the environment.0/db_1 Select the Cluster Installation option then select all nodes available. Specify Home Details Set the Name and Path for the ORACLE_HOME as follows: Name: OraDb10g_home1 Location: /u01/app/oracle/product/10. export CV_JDKHOME .

2.0: cannot open shared object file: No such file or directory This error occurs because these releases of the Linux kernel fix an old bug in the Linux threading that Oracle worked around using LD_ASSUME_KERNEL settings in both vipca and srvctl.sh.Actual Result: 1983MB In most cases. edit $ORACLE_HOME/bin/srvctl (in the Database bin directory on all nodes) to undo the setting of LD_ASSUME_KERNEL by adding one line. For me. Privileged Operating System Groups (Custom Database Installs Only) Unset LD_ASSUME_KERNEL in SRVCTL During the Oracle Clusterware installation in the previous section. you will be prompted to run the root. Summary Click Install to start the installation! After the installation has completed. you will have the minimum required swap space (as shown above) and this can be safely ignored. this workaround is no longer valid on OEL5 or RHEL5 or SLES10 hence the failures. To workaround this issue.2.19 . End of installation At the end of the installation. Create Database Remember that we will create the clustered database as a separate step using dbca. Those same modifications need to be performed with the new srvctl in the Database bin directory: $ $ORACLE_HOME/bin/srvctl /u01/app/oracle/product/10.. Navigate to the /u01/app/oracle/product/10.0/db_1 directory and run root. this was "linux1". Execute Configuration Scripts First. Simply click the check-box for "Checking available swap space requirements..sh script will need to be run on both nodes in the RAC cluster one at a time starting with the node you are running the database installation from. It is important to keep in mind that the root. exit from the OUI. go back to the OUI and acknowledge the "Execute Configuration scripts" dialog window.sh script.0/db_1/jdk/jre/bin/java: error while loading shared libraries: libpthread." and click Next to continue. After running the root.4. we needed to modify both vipca and srvctl in the CRS bin directory.so. open a new console window on the node you are installing the Oracle Database 10g software from as the root user account. Select the UNIX groups that will be used for each of the Oracle group names as follows: Database Administrator (OSDBA) Group: dba Database Operator (OSOPER) Group: oper Select the option to Install database Software only. around line 168: LD_ASSUME_KERNEL=2.sh script on both nodes in the cluster.

2.4 patchsets. after applying the 10.export LD_ASSUME_KERNEL unset LD_ASSUME_KERNEL <<== Line to be added Note: Remember to re-edit these files on all nodes: <ORA_CRS_HOME>/bin/vipca <ORA_CRS_HOME>/bin/srvctl <RDBMS_HOME>/bin/srvctl <ASM_HOME>/bin/srvctl # (If exists) .3 patch sets.2.2.. Install Oracle Database 10g Companion CD Software Perform the following installation procedures from only one of the Oracle RAC nodes in the cluster (linux1)! The Oracle Database 10g Companion CD software will be installed to both of Oracle RAC nodes in the cluster by the Oracle Universal Installer. Please keep in mind that this is an optional step.0. Also note that we are explicitly unsetting LD_ASSUME_KERNEL and not merely commenting out its setting to handle a case where the user has it set in their environment (login shell).2 or 10.1. Overview After successfully installing the Oracle Database software.2. The type of installation to perform will be the Oracle Database 10g Products installation type.. my testing database will often make use of the Java Virtual Machine (Java VM) and Oracle interMedia and therefore will require the installation of the Oracle Database 10g Companion CD. This issue was raised with development and is fixed in the 10. the next step is to install the Oracle Database 10g Companion CD Release 2 software (10.0. the ORA-29558:JAccelerator (NCOMP) not installed error occurs when a database that uses Java VM is upgraded to the patch release.0).0. . as these patchset will still include those settings unnecessary for OEL5 or RHEL5 or SLES10. This installation type includes the Natively Compiled Java Libraries (NCOMP) files to improve Java performance. For the purpose of this article. If you do not install the NCOMP files.0.

When using the secure shell method. To enable user equivalence for the current terminal shell session. (Install Oracle Database 10g Software). perform the following steps remembering to enter the pass phrase for the RSA key you generated when prompted: $ exec /usr/bin/ssh-agent $SHELL $ /usr/bin/ssh-add Enter passphrase for /home/oracle/. The OUI will copy the software packages to all nodes configured in the RAC cluster.0 export DISPLAY Verify Remote Access / User Equivalence Verify you are able to run the Secure Shell commands (ssh or scp) on the Linux server you will be running the Oracle Universal Installer from against all other Linux servers in the cluster without being prompted for a password. you do not have to perform any of the actions described below with regards to setting up remote access and the DISPLAY variable: Log in as the oracle User Account and Set DISPLAY (if necessary) # su .ssh/id_rsa (/home/oracle/. Verifying Terminal Shell Environment As discussed in the previous section. the terminal shell environment needs to be configured for remote access and user equivalence to all nodes in the cluster before running the Oracle Universal Installer. Note that you can utilize the same terminal shell session used in the previous section which in this case.hostname" Wed Aug 19 22:07:13 EDT 2009 linux1 $ ssh linux2 "date.oracle $ $ $ $ # IF YOU ARE USING A REMOTE CLIENT TO CONNECT TO THE # NODE PERFORMING THE INSTALL DISPLAY=<your local workstation>:0.Like the Oracle Clusterware and Database install (previous sections).ssh/id_rsa: xxxxx Identity added: /home/oracle/.hostname" Wed Aug 19 22:07:44 EDT 2009 linux2 Install Oracle Database 10g Companion CD Software Install the Oracle Database 10g Companion CD Software as follows: $ cd ~oracle $ /home/oracle/orainstall/companion/runInstaller -ignoreSysPrereqs . user equivalence will need to be enabled for the terminal shell session before attempting to run the OUI.ssh/id_rsa) $ ssh linux1 "date. the Oracle Database 10g Companion CD software only needs to be run from one node.

you will need to manually verify the check that failed by clicking on the checkbox.0/db_1 The Cluster Installation option will be selected along with all of the available nodes in the cluster by default. click Install to start the installation! At the end of the installation. Ensure you are able to reach the node in question from the node you are performing the installation from.Screen Name Welcome Screen Select a Product to Install Response Click Next Select the Oracle Database 10g Products 10.2.0 option.2. If any of the checks fail. all checks passed with no problems. Stay with these default options and click Next to continue.0. Click Next to continue. Summary End of installation On the Summary screen. Specify Home Details Specify Hardware Cluster Installation Mode Product-Specific Prerequisite Checks The installer will run through a series of checks to determine if the node meets the minimum requirements for installing and configuring the Oracle Database 10g Companion CD Software. Overview The DBCA requires the Oracle TNS Listener process to be configured and running on all nodes in the RAC cluster before it can create the clustered database. perform the following checks:   Ensure the Oracle Clusterware is running on the node in question. Create TNS Listener Process Perform the following configuration procedures from only one of the Oracle RAC nodes in the cluster (linux1)! The Network Configuration Assistant (NETCA) will setup the TNS listener in a clustered configuration on both of Oracle RAC nodes in the cluster.1. For my installation. Set the destination for the ORACLE_HOME Name and Path to that of the previous Oracle Database 10g software install as follows: Name: OraDb10g_home1 Path: /u01/app/oracle/product/10. If the installation stops here and the status of any of the RAC nodes is "Node not reachable". . exit from the OUI.

All changes will be made and replicated to all nodes in the cluster.oracle $ $ $ $ # IF YOU ARE USING A REMOTE CLIENT TO CONNECT TO THE # NODE PERFORMING THE INSTALL DISPLAY=<your local workstation>:0. On one of the nodes (I will be using linux1) bring up the NETCA and run through the process of creating a new TNS listener process and also configure the node for local access. you do not have to perform any of the actions described below with regards to setting up remote access and the DISPLAY variable: Log in as the oracle User Account and Set DISPLAY (if necessary) # su . To enable user equivalence for the current terminal shell session. run the following: $ netca & . Verifying Terminal Shell Environment As discussed in the previous section.ssh/id_rsa) $ ssh linux1 "date.ssh/id_rsa: xxxxx Identity added: /home/oracle/.The process of creating the TNS listener only needs to be performed on one node in the cluster. (Install Oracle Database 10g Companion CD Software). When using the secure shell method. user equivalence will need to be enabled for the terminal shell session before attempting to run the OUI. the terminal shell environment needs to be configured for remote access and user equivalence to all nodes in the cluster before running the Network Configuration Assistant (NETCA). perform the following steps remembering to enter the pass phrase for the RSA key you generated when prompted: $ exec /usr/bin/ssh-agent $SHELL $ /usr/bin/ssh-add Enter passphrase for /home/oracle/.hostname" Wed Aug 19 22:07:13 EDT 2009 linux1 $ ssh linux2 "date.ssh/id_rsa (/home/oracle/. Note that you can utilize the same terminal shell session used in the previous section which in this case.0 export DISPLAY Verify Remote Access / User Equivalence Verify you are able to run the Secure Shell commands (ssh or scp) on the Linux server you will be running the Oracle Universal Installer from against all other Linux servers in the cluster without being prompted for a password.hostname" Wed Aug 19 22:07:44 EDT 2009 linux2 Run the Network Configuration Assistant To start the NETCA.

Select Naming Methods configuration. The following screens are now like any other normal listener configuration. Listener Configuration Next 6 Screens Type of Configuration The following screens are: Selected Naming Methods: Local Naming Naming Methods Configuration Naming Methods configuration complete! [ Next ] You will be returned to this Welcome (Type of Configuration) Screen.linux1.LISTENER_LINUX1.lsnr TYPE=application TARGET=ONLINE STATE=ONLINE on linux1 ===================== $ hostname linux2 $ ps -ef | grep lsnr | grep -v 'grep' | grep -v 'ocfs' | awk '{print $9}' LISTENER_LINUX2 $ $ORA_CRS_HOME/bin/crs_stat ora.lsnr NAME=ora. Select Listener configuration. Verify TNS Listener Configuration The Oracle TNS listener process should now be running on both nodes in the RAC cluster: $ hostname linux1 $ ps -ef | grep lsnr | grep -v 'grep' | grep -v 'ocfs' | awk '{print $9}' LISTENER_LINUX1 $ $ORA_CRS_HOME/bin/crs_stat ora. Type of Configuration Click Finish to exit the NETCA.lsnr TYPE=application . Screen Name Select the Type of Oracle Net Services Configuration Select the nodes to configure Type of Configuration Response Select Cluster configuration Select all of the nodes: linux1 and linux2.LISTENER_LINUX2.LISTENER_LINUX2.lsnr NAME=ora.linux2. You can simply accept the default parameters for the next six screens: What do you want to do: Add Listener name: LISTENER Selected protocols: TCP Port number: 1521 Configure another listener: No Listener configuration complete! [ Next ] You will be returned to this Welcome (Type of Configuration) Screen.linux1.The following table walks you through the process of creating a new Oracle listener for our RAC environment.linux2.LISTENER_LINUX1.

TARGET=ONLINE STATE=ONLINE on linux2

Create the Oracle Cluster Database

The database creation process should only be performed from one of the Oracle RAC nodes in the cluster (linux1)!

Overview We will be using the Oracle Database Configuration Assistant (DBCA) to create the clustered database. Before executing the Database Configuration Assistant, make sure that $ORACLE_HOME and $PATH are set appropriately for the $ORACLE_BASE/product/10.2.0/db_1 environment. You should also verify that all services we have installed up to this point (Oracle TNS listener, Oracle Clusterware processes, etc.) are running before attempting to start the clustered database creation process.

Verifying Terminal Shell Environment As discussed in the previous section, (Install Oracle Database 10g Companion CD Software), the terminal shell environment needs to be configured for remote access and user equivalence to all nodes in the cluster before running the Network Configuration Assistant (NETCA). Note that you can utilize the same terminal shell session used in the previous section which in this case, you do not have to perform any of the actions described below with regards to setting up remote access and the DISPLAY variable: Log in as the oracle User Account and Set DISPLAY (if necessary)
# su - oracle $ $ $ $ # IF YOU ARE USING A REMOTE CLIENT TO CONNECT TO THE # NODE PERFORMING THE INSTALL DISPLAY=<your local workstation>:0.0 export DISPLAY

Verify Remote Access / User Equivalence Verify you are able to run the Secure Shell commands (ssh or scp) on the Linux server you will be running the Oracle Universal Installer from against all other Linux servers in the cluster without being prompted for a password. When using the secure shell method, user equivalence will need to be enabled for the terminal shell session before attempting to run the OUI. To enable user equivalence for the current terminal shell session,

perform the following steps remembering to enter the pass phrase for the RSA key you generated when prompted:
$ exec /usr/bin/ssh-agent $SHELL $ /usr/bin/ssh-add Enter passphrase for /home/oracle/.ssh/id_rsa: xxxxx Identity added: /home/oracle/.ssh/id_rsa (/home/oracle/.ssh/id_rsa) $ ssh linux1 "date;hostname" Wed Aug 19 22:07:13 EDT 2009 linux1 $ ssh linux2 "date;hostname" Wed Aug 19 22:07:44 EDT 2009 linux2

Run the Oracle Cluster Verification Utility Before creating the Oracle clustered database, we should run the following database configuration check using the Cluster Verification Utility (CVU). Instructions for configuring CVU can be found in the section "Prerequisites for Using Cluster Verification Utility discussed earlier in this article.
$ $ $ $ cd /home/oracle/orainstall/clusterware/cluvfy CV_HOME=/home/oracle/orainstall/clusterware/cluvfy; export CV_HOME CV_JDKHOME=/home/oracle/orainstall/clusterware/cluvfy/jdk14; export CV_JDKHOME ./runcluvfy.sh stage -pre dbcfg -n linux1,linux2 -d ${ORACLE_HOME} -verbose

Review the CVU report. Note that this report will contain the same error we received when checking preinstallation tasks for CRS — failure to find a suitable set of interfaces for VIPs. This error can be safely ignored.

Create the Clustered Database To start the database creation process, run the following:
$ dbca & Screen Name Welcome Screen Operations Node Selection Database Templates Database Identification Response Select Oracle Real Application Clusters database. Select Create a Database. Click the Select All button to select all servers: linux1 and linux2. Select Custom Database Select: Global Database Name: racdb.idevelopment.info SID Prefix: racdb

I used idevelopment.info for the database domain. You may use any domain. Keep in mind that this domain does not have to be a valid DNS domain. Management Option Database Credentials Storage Options Leave the default options here which is to Configure the Database with Enterprise Manager / Use Database Control for Database Management I selected to Use the Same Password for All Accounts. Enter the password (twice) and make sure the password does not start with a digit number. For this article, we will select to use Automatic Storage Management (ASM). Supply the SYS password to use for the new ASM instance. Also, starting with Oracle10g Release 2, the ASM instance server parameter file (SPFILE) needs to be on a shared disk. You will need to modify the default entry for "Create server parameter file (SPFILE)" to reside on the OCFS2 partition as follows: /u02/oradata/racdb/dbs/spfile+ASM.ora. All other options can stay at their defaults. You will then be prompted with a dialog box asking if you want to create and start the ASM instance. Select the OK button to acknowledge this dialog. The OUI will now create and start the ASM instance on all nodes in the RAC cluster. To start, click the Create New button. This will bring up the "Create Disk Group" window with the four volumes we configured earlier using ASMLib. If the volumes we created earlier in this article do not show up in the "Select Member Disks" window: (ORCL:VOL1, ORCL:VOL2, ORCL:VOL3, and ORCL:VOL4) then click on the "Change Disk Discovery Path" button and input "ORCL:VOL*". For the first "Disk Group Name" I used the string RACDB_DATA1. Select the first two ASM volumes (ORCL:VOL1 and ORCL:VOL2) in the "Select Member Disks" window. Keep the "Redundancy" setting to Normal. After verifying all values in this window are correct, click the OK button. This will present the "ASM Disk Group Creation" dialog. When the ASM Disk Group Creation process is finished, you will be returned to the "ASM Disk Groups" windows. Click the Create New button again. For the second "Disk Group Name", I used the string FLASH_RECOVERY_AREA. Select the last two ASM volumes (ORCL:VOL3 and ORCL:VOL4) in the "Select Member Disks" window. Keep the "Redundancy" setting to Normal. After verifying all values in this window are correct, click the OK button. This will present the "ASM Disk Group Creation" dialog. When the ASM Disk Group Creation process is finished, you will be returned to the "ASM Disk Groups" window with two disk groups created and selected. Select only one of the disk groups by using the checkbox next to the newly created Disk Group Name RACDB_DATA1 (ensure that the disk group for FLASH_RECOVERY_AREA is not selected) and click Next to continue. Database File Locations I selected to use the default which is Use Oracle-Managed Files: Database Area: +RACDB_DATA1

Create ASM Instance

ASM Disk Groups

When the Oracle Database Configuration Assistant has completed. For this test configuration.ons application ONLINE ONLINE linux1 ora.asm application ONLINE ONLINE linux2 ora.b2.X2.lsnr application ONLINE ONLINE linux2 ora. I also always select to "Generate Database Creation Scripts"..ons application ONLINE ONLINE linux2 ora. all windows and dialog boxes will disappear.vip application ONLINE ONLINE linux1 ora. My disk group has a size of about 17GB. This option is available since we installed the Oracle Companion CD software.lsnr application ONLINE ONLINE linux1 ora. I left them all at their default settings. exit from the DBCA.linux1. Click OK on the "Summary" screen. you will have a fully functional Oracle RAC cluster running! $ $ORA_CRS_HOME/bin/crs_stat -t Name Type Target State Host -----------------------------------------------------------ora.gsd application ONLINE ONLINE linux2 ora..b1... another dialog will come up indicating that it is starting all Oracle instances and HA service "racdb_srvc"....inst application ONLINE ONLINE linux1 ora.db application ONLINE ONLINE linux1 ora. When finished. click the [Browse] button and select the disk group name +FLASH_RECOVERY_AREA.linux2.. the database creation will start.linux1. Creation Options Click Finish to start the database creation process.. I left them all at their default settings..SM2.vip application ONLINE ONLINE linux2 ora..asm application ONLINE ONLINE linux1 ora. although it is perfectly OK to select the Sample Schemas.linux2.X1.. This may take several minutes to complete. Leave both instances set to Preferred and for the "TAF Policy" select Basic.Check the option for Specify Flash Recovery Area. After acknowledging the database creation report and script generation dialog.. When defining the Flash Recovery Area size.SM1..racdb. and enter the Service Name: racdb_srvc.. I used a "Flash Recovery Area Size" of 15 GB (15360 MB). For the Flash Recovery Area. Recovery Configuration Database Services Initialization Parameters Change any parameters for your environment.inst application ONLINE ONLINE linux2 . Note: When exiting the DBCA you will not receive any feedback from the dialog End of Database Creation window for around 30-60 seconds. At the end of the database creation. click Add..gsd application ONLINE ONLINE linux1 ora. After awhile. Database Storage Change any parameters for your environment.. use the entire volume minus 10% for overhead — (17-10%=15 GB).linux1.linux2. Database Content I left all of the Database Components (and destination tablespaces) set to their default value.. Keep the default option Create Database selected.

info.info SQL> alter system set service_names = 2 'racdb.srv application ora..sql script to recompile all invalid PL/SQL packages now instead of when the packages are accessed for the first time.db1.sql .info.----------. Re-compile Invalid Objects Run the utlrp. $ sqlplus / as sysdba SQL> @?/rdbms/admin/utlrp.info. but was never updated as a service for each Oracle instance.ora.info' scope=both.idevelopment.. This step is optional but recommended.idevelopment. then you will need to manually add the service to both instances: SQL> show parameter service NAME TYPE VALUE -------------------. Post-Installation Tasks ..-------------------------service_names string racdb...(Optional) This chapter describes several optional tasks that can be applied to your new Oracle 10g environment in order to enhance availability as well as database management.ora. we added a service named " racdb_srvc" that will be used to connect to the database with TAF enabled.idevelopment. racdb_srvc If the only service defined was for racdb. Use the following to verify the racdb_srvc service was successfully added: SQL> show parameter service NAME TYPE VALUE -------------------.idevelopment.srv application ONLINE ONLINE ONLINE ONLINE ONLINE ONLINE linux1 linux1 linux2 Create the racdb_srvc Service During the creation of the Oracle clustered database.-------------------------------service_names string racdb. racdb_srvc.cs application ora..db2.idevelopment... During several of my installs.srvc.. the service was added to the tnsnames.----------.

Oracle will make a copy of the online redo log before it gets reused. I will use the node linux1 which runs the racdb1 instance: 1. To facilitate media recovery. use the following tasks to put a RAC enabled database into archive log mode. When the current online redolog fills. $ sqlplus / as sysdba SQL> alter system set cluster_database=false scope=spfile sid='racdb1'. however most DBA's opt to bypass this option during initial database creation. It is also worth mentioning that each instance has exclusive write access to its own online redolog files. The same holds true for a single instance configuration. Using the local instance. In a correctly configured RAC environment. linux1) and disable the cluster instance parameter by setting cluster_database to FALSE from the current instance: 2. Each Oracle instance will use its group of online redologs in a circular manner. It is therefore a requirement that online redo logs be located on a shared storage device (just like the database files). The size of an online redolog file is completely independent of another intances' redolog size. Once an online redolog fills. Oracle will switch to the next one. it may be different depending on the workload and backup / recovery considerations for each node. it is a simple task to put the database into archive log mode. In an Oracle RAC environment. From one of the nodes in the Oracle RAC configuration. Enable archiving: SQL> alter database archivelog. Log in to one of the nodes (i. If the database is in "Archive Log Mode". each instance can read another instance's current online redolog file to perform instance recovery if that instance was terminated abnormally. In cases like this where the database is in no archive log mode. however. This is a process known as archiving. Oracle moves to the next one. Oracle tracks and logs all changes to database blocks in online redolog files. A thread must contain at least two online redologs (or online redolog groups). each instance will have its own set of online redolog files known as a thread. For the purpose of this article. MOUNT the database: 5. Shutdown all instances accessing the clustered database: $ srvctl stop database -d racdb 4. The single instance must contain at least two online redologs (or online redolog groups). Note however that this will require a short database outage.e. . 3. Although in most configurations the size is the same. The Database Configuration Assistant (DBCA) allows users to configure a new database to be in archive log mode. Oracle allows the DBA to put the database into "Archive Log Mode" which makes a copy of the online redolog after it fills (and before it gets reused).Enabling Archive Logs in a RAC Environment Whether a single instance or clustered database. As already mentioned. Oracle writes to its online redolog files in a circular manner. $ sqlplus / as sysdba SQL> startup mount 6.

7. the dba_scripts_archive_Oracle. performance. For example. unzip the archive file to the $ORACLE_BASE directory. As the oracle user account. Log in to the local instance and verify Archive Log Mode is enabled: 12.zip archive will be copied to /u01/app/oracle. TAF) back up using srvctl: $ srvctl start service -d racdb 11. Oldest online log sequence 18. Although these views provide a simple and easy mechanism to query critical information regarding the database. security. Archive destination 17.zip archive to the $ORACLE_BASE directory of each node in the cluster. For the purpose of this example.zip /u01/app/oracle $ cd /u01/app/oracle . Re-enable support for clustering by modifying the instance parameter cluster_database to TRUE from the current instance: SQL> alter system set cluster_database=true scope=spfile sid='racdb1'. Shutdown the local instance: SQL> shutdown immediate 9. In this section you will download and install a collection of Oracle DBA scripts that can be used to manage many aspects of your database including space management. Database log mode 15. $ sqlplus / as sysdba 13. SQL> archive log list 14.zip. it helps to have a collection of accurate and readily available SQL scripts to query these views. Bring all instance back up using srvctl: $ srvctl start database -d racdb 10. Next log sequence to archive Current log sequence 84 Archive Mode Enabled USE_DB_RECOVERY_FILE_DEST 83 84 After enabling Archive Log Mode. Automatic archival 16. backups.e. 8. (Optional) Bring any services (i. each instance in the RAC configuration can automatically archive redologs! Download and Install Custom Oracle Database Scripts DBA's rely on Oracle's data dictionary views and dynamic performance views in order to support and better manage their databases. The Oracle DBA scripts archive can be downloaded using the following link http://www. and session management.idevelopment. download the dba_scripts_archive_Oracle. Next.info/data/Oracle/DBA_scripts/dba_scripts_archive_Oracle. perform the following on both nodes in the Oracle RAC cluster as the oracle user account: $ mv dba_scripts_archive_Oracle.

283.----------UNDO PERMANENT PERMANENT PERMANENT PERMANENT UNDO TEMPORARY LOCAL LOCAL LOCAL LOCAL LOCAL LOCAL LOCAL MANUAL AUTO AUTO MANUAL AUTO MANUAL MANUAL 1.741.sql script: SQL> @help. TS Type Ext.704 60 ONLINE INDX 65.zip The final step is to verify (or set) the appropriate environment variable for the current UNIX shell to ensure the Oracle SQL scripts can be run from SQL*Plus while in any directory.--------ONLINE UNDOTBS1 9.$ unzip dba_scripts_archive_Oracle. Mgt.----------- 8.024 1.457.282.bash_profile login script that was created in the section Create Login Script for oracle User Account. For example.030.073.240 2 ONLINE TEMP 92.000 2.--------------------.544.472 15 ONLINE SYSTEM 505.860.329.688 9 ------.225.144 To obtain a list of all available Oracle DBA scripts while logged into SQL*Plus.---------.536 0 ONLINE UNDOTBS2 22.824 1.073.741.201.288.257. you should now be able to run any of the SQL scripts in your $ORACLE_BASE/dba_scripts/common/sql while logged into SQL*Plus.--------avg 23 sum 1.043.824 -----------------.483.sql ======================================== .457.024 524. Seg.--------. Mgt.800 1.274. For UNIX verify the following environment variable is set and included in your login shell script: ORACLE_PATH=$ORACLE_BASE/dba_scripts/common/sql:. run the help. to query tablespace information while logged into the Oracle database as a DBA user: SQL> @dba_tablespaces Status Tablespace Name bytes) Pct.:$ORACLE_HOME/rdbms/admin export ORACLE_PATH Note that the ORACLE_PATH environment variable should already be set in the .147.152 7 rows selected. Tablespace Size Used (in -----------.544 72 ONLINE USERS 321. Now that the Oracle DBA scripts have been unzipped and the UNIX environment variable ( $ORACLE_PATH) has been set to the appropriate directory.968 1 ONLINE SYSAUX 378.648 838.-----------------. Used --------.283.732.

To resolve this problem.sql asm_diskgroups.sql asm_disks.sql wm_refresh_workspace. Since the environment described in this article makes use of the Oracle Clustered File System (OCFS2). I present the steps required to configure a shared Oracle password file between all instances in the Oracle clustered database.sql wm_freeze_workspace. a common solution is to place a single database password file on a shared / clustered file system and then create symbolic links from each of the instances to this single version of the database password file.sql perf_top_sql_by_disk_reads.Automatic Shared Memory Management ======================================== asmm_components.sql ======================================== Workspace Manager ======================================== wm_create_workspace. But what about the other instances in the cluster? The database password file on other instances do not get updated and will not contain the user who was just granted the SYSDBA role.sql ======================================== Automatic Storage Management ======================================== asm_alias.sql asm_disks_perf. Therefore a program (like RMAN) that tries to log in as this new user with SYSDBA privileges will fail if it tries to use an instance with a password file that does not contain his or her name.sql wm_get_workspace. the instance records this in the database password file for the instance you are logged into.sql wm_merge_workspace.sql asm_clients.sql asm_files2.sql wm_enable_versioning.sql asm_templates.sql wm_goto_workspace. When a database user is granted the SYSDBA role.> perf_top_sql_by_buffer_gets.sql asm_files.sql wm_unfreeze_workspace. we will use it to store the single version of the database password file.sql wm_disable_versioning.sql wm_workspaces.sql asm_drop_files.sql Create Shared Oracle Password Files In this section.sql wm_remove_workspace. The password file for the database in UNIX is located at $ORACLE_HOME/dbs/orapw<ORACLE_SID> for each instance and contains a list of all database users that have SYSDBA privileges.sql < --. .SNIP --.

(linux1): $ mkdir -p /u02/oradata/racdb/dbs 2. this will not matter since any missing entries can be easily added by granting them the SYSDBA role . 1. Create the database password directory on the clustered file system mounted on /u02/oradata/racdb. when a user is granted the SYSDBA role. From the second node in the cluster (linux2): 8.(plus the fact that this is a fresh install and unlikely you created any SYSDBA users at this point!). $ ln -s /u02/oradata/racdb/dbs/orapw+ASM $ORACLE_HOME/dbs/orapw+ASM2 $ ln -s /u02/oradata/racdb/dbs/orapwracdb $ORACLE_HOME/dbs/orapwracdb2 Now. In most cases. $ mv $ORACLE_HOME/dbs/orapwracdb1 /u02/oradata/racdb/dbs/orapwracdb 5.In this section. $ rm $ORACLE_HOME/dbs/orapwracdb2 10. $ mv $ORACLE_HOME/dbs/orapw+ASM1 /u02/oradata/racdb/dbs/orapw+ASM 4. Perform the following from only one node in the cluster as the oracle user account . Note that the database server does not need to be shutdown while performing the following actions. From one node in the cluster (linux1).ora . From linux1 as the oracle user account: 3. $ rm $ORACLE_HOME/dbs/orapw+ASM2 9. move the database password files to the database password directory on the clustered file system. 6. we will also be including the Oracle password file for the ASM instance. $ ln -s /u02/oradata/racdb/dbs/orapw+ASM $ORACLE_HOME/dbs/orapw+ASM1 $ ln -s /u02/oradata/racdb/dbs/orapwracdb $ORACLE_HOME/dbs/orapwracdb1 7. all instances will have access to the same password file: SQL> GRANT sysdba TO scott. 11. Verify TNS Networking Files Ensure that the TNS networking files are configured on both Oracle RAC nodes in the cluster! listener. Chose a node that contains a database password file that has the most recent SYSDBA additions.

For clarity. The listener.2.2.168.0/db_1/network/admin/tnsnames.2. LISTENER_LINUX1 = (DESCRIPTION_LIST = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = linux1-vip)(PORT = 1521)(IP = FIRST)) (ADDRESS = (PROTOCOL = TCP)(HOST = 192.ora Here is a copy of my tnsnames. RACDB = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = linux1-vip)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = linux2-vip)(PORT = 1521)) (LOAD_BALANCE = yes) (CONNECT_DATA = (SERVER = DEDICATED) .0/db_1) (PROGRAM = extproc) ) ) tnsnames. tnsnames. This file should already be configured on both of the Oracle RAC nodes in the RAC cluster.100)(PORT = 1521)(IP = FIRST)) ) ) SID_LIST_LISTENER_LINUX1 = (SID_LIST = (SID_DESC = (SID_NAME = PLSExtProc) (ORACLE_HOME = /u01/app/oracle/product/10.ora # Generated by Oracle configuration tools.ora file that was configured by Oracle and can be used for testing the Transparent Application Failover (TAF).ora Network Configuration File: # /u01/app/oracle/product/10. I included a copy of the listener.ora file should be properly configured and no modifications should be needed.linux1 Network Configuration File: # /u01/app/oracle/product/10.1.ora # listener.ora.ora.linux1 # Generated by Oracle configuration tools.We already covered how to create a TNS listener configuration file (listener.ora) for a clustered environment in the section Create TNS Listener Process.ora # tnsnames.0/db_1/network/admin/listener.ora file from my node linux1: listener. You can include any of these entries on other client machines that need access to the clustered database.

info) ) ) RACDB1 = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = linux1-vip)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = racdb.idevelopment.idevelopment.idevelopment.info) (FAILOVER_MODE = (TYPE = SELECT) (METHOD = BASIC) (RETRIES = 180) (DELAY = 5) ) ) ) LISTENERS_RACDB = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = linux1-vip)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = linux2-vip)(PORT = 1521)) ) EXTPROC_CONNECTION_DATA = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC0)) ) (CONNECT_DATA = (SID = PLSExtProc) (PRESENTATION = RO) ) .info) (INSTANCE_NAME = racdb1) ) ) RACDB2 = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = linux2-vip)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = racdb.(SERVICE_NAME = racdb.idevelopment.info) (INSTANCE_NAME = racdb2) ) ) RACDB_SRVC = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = linux1-vip)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = linux2-vip)(PORT = 1521)) (LOAD_BALANCE = yes) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = racdb_srvc.

) Connecting to Clustered Database From an External Client This is an optional step.ora files. substitute the data file names that were created in your environment where appropriate. Use another machine (i.e.168.100 192.168. but I like to perform it in order to verify my TNS files are configured correctly. If you are using a large drive for the shared storage.1.201 linux1 linux2 linux1-vip linux2-vip Try to connect to the clustered database using all available service names defined in the tnsnames. the machine you are connecting from should be able to resolve the following host names in the local hosts file or through DNS: 192.695345467 . file_name from dba_data_files union select tablespace_name.1.ora and tnsnames.1.1. TABLESPACE_NAME FILE_NAME --------------.101 192. Note: Please keep in mind that the database file names (OMF files) being listed in these examples may differ from what the Oracle Database Configuration Assistant (DBCA) creates for your environment. file_name from dba_temp_files. Important Note: Verify that the machine you are connecting from can resolve all host names exactly how they appear in the listener. When working through this section. For the purpose of this document. you may want to make a sizable testing database. we left all tablespaces set to their default size.200 192.ora) from either of the nodes in the cluster that were created for the clustered database. Below are several optional SQL commands for modifying and creating all tablespaces for the test database.263.168.-------------------------------------------------EXAMPLE +RACDB_DATA1/racdb/datafile/example. The following query can be used to determine the file names for your environment: SQL> 2 3 4 5 select tablespace_name.ora file: C:\> C:\> C:\> C:\> sqlplus sqlplus sqlplus sqlplus system/manager@racdb2 system/manager@racdb1 system/manager@racdb_srvc system/manager@racdb Create / Alter Tablespaces When creating the clustered database.168. a Windows machine connected to the network) that has Oracle installed and add the TNS entries (in the tnsnames.

283.265.648 838.400 1.695345475 +RACDB_DATA1/racdb/datafile/users.024 524.741.695345415 +RACDB_DATA1/racdb/tempfile/temp. SQL> alter database datafile '+RACDB_DATA1/racdb/datafile/users.544 10 ONLINE SYSAUX 270.261.262.024 .--------------------.840 60 ONLINE EXAMPLE 83.261.695345415' resize 800m.695345485' resize 1024m.264.260.080 0 TS Type Ext.262. SQL> alter database datafile '+RACDB_DATA1/racdb/datafile/sysaux.971.286.966. Mgt.072 0 ONLINE SYSTEM 503.000 52 ONLINE USERS 131.695345459 +RACDB_DATA1/racdb/datafile/undotbs1.695345451' resize 500m. Mgt.457. Tablespace Size Used (in -----------. SQL> alter tablespace undotbs1 add datafile '+RACDB_DATA1' size 1024m 2 autoextend on next 50m maxsize 2048m.860.483.283.-----------------.073.457.147. SQL> alter database datafile '+RACDB_DATA1/racdb/datafile/system.----------UNDO PERMANENT PERMANENT PERMANENT PERMANENT PERMANENT UNDO LOCAL LOCAL LOCAL LOCAL LOCAL LOCAL LOCAL MANUAL AUTO AUTO MANUAL AUTO AUTO MANUAL 1. resource.259.536 0 ONLINE UNDOTBS2 1.---------. SQL> alter tablespace undotbs2 add datafile '+RACDB_DATA1' size 1024m 2 autoextend on next 50m maxsize 2048m.695345459' resize 1024m. SQL> 2 3 4 create tablespace indx datafile '+RACDB_DATA1' size 1024m autoextend on next 50m maxsize unlimited extent management local autoallocate segment space management auto.972. Used --------.800 157.820.259.824 1.288. SQL> alter database tempfile '+RACDB_DATA1/racdb/tempfile/temp.265.695345441 +RACDB_DATA1/racdb/datafile/undotbs2.695345485 sysdba" SQL> create user scott identified by tiger default tablespace users.--------.336.695345451 +RACDB_DATA1/racdb/datafile/system.000 2. SQL> grant dba. SQL> alter tablespace users add datafile '+RACDB_DATA1' size 1024m autoextend off.--------ONLINE UNDOTBS1 132. connect to scott.SYSAUX SYSTEM TEMP UNDOTBS1 UNDOTBS2 USERS $ sqlplus "/ as +RACDB_DATA1/racdb/datafile/sysaux. Seg. Here is a snapshot of the tablespaces I have defined for my test database environment: Status Tablespace Name bytes) Pct.544 53 ONLINE INDX 65.

165.018. Administering node applications.----------- 8. I will only be performing checks from linux1.824 -----------------. and Oracle Enterprise Manager agents (for maintenance purposes).--------avg 22 sum 1. Administering ASM instances.073. TEMPORARY 2 LOCAL MANUAL 1.ONLINE TEMP 25.429. There are five node-level tasks defined for SRVCTL:      Adding and deleting node level applications. Status of all instances and services $ srvctl status database -d racdb Instance racdb1 is running on node linux1 Instance racdb2 is running on node linux2 . however.316.440 8 rows selected. Starting and stopping a group of programs that includes virtual IP addresses.544 Verify the RAC Cluster & Database Configuration The following RAC verification checks should be performed on both Oracle RAC nodes in the cluster! For this article.741. Oracle Notification Services.824 ------. Setting and unsetting the environment for node-level applications.382. Overview This section provides several srvctl commands and SQL queries that can be used to validate your Oracle RAC configuration. listeners.

2. ONS.(VIP. Listener) $ srvctl config nodeapps -n linux1 -a -g -s -l . racdb2 Status of node applications on a particular node $ srvctl status nodeapps -n linux1 VIP is running on node: linux1 GSD is running on node: linux1 Listener is running on node: linux1 ONS daemon is running on node: linux1 Status of an ASM instance $ srvctl status asm -n linux1 ASM instance +ASM1 is running on node linux1.2. GSD. List all configured databases $ srvctl config database racdb Display configuration for our RAC database $ srvctl config database -d racdb linux1 racdb1 /u01/app/oracle/product/10.Status of a single instance $ srvctl status instance -d racdb -i racdb2 Instance racdb2 is running on node linux2 Status of a named service globally across the database $ srvctl status service -d racdb -s racdb_srvc Service racdb_srvc is running on instance(s) racdb1.0/db_1 linux2 racdb2 /u01/app/oracle/product/10.0/db_1 Display all services for the specified cluster database $ srvctl config service -d racdb racdb_srvc PREF: racdb1 racdb2 AVAIL: Display the configuration for node applications .

695352447 +RACDB_DATA1/racdb/datafile/sysaux.256.--.0/db_1 All running instances in the cluster SELECT inst_id . INST_ID INST_NO INST_NAME PAR STATUS DB_STATUS STATE HOST -------.263.695345409 +FLASH_RECOVERY_AREA/racdb/onlinelog/group_3.695345467 +RACDB_DATA1/racdb/datafile/indx. database_status db_status .259.2.255.257.259.695345451 +RACDB_DATA1/racdb/datafile/system.: /linux1-vip/192. host_name host FROM gv$instance ORDER BY inst_id. Display the configuration for the ASM instance(s) $ srvctl config asm -n linux1 +ASM1 /u01/app/oracle/product/10.695345391 +RACDB_DATA1/racdb/datafile/example. instance_number inst_no . Listener exists.270.-----------.695350869 +FLASH_RECOVERY_AREA/racdb/onlinelog/group_4.261. ONS daemon exists. NAME ------------------------------------------+FLASH_RECOVERY_AREA/racdb/controlfile/current. status .695345415 . active_state state .------1 1 racdb1 YES OPEN ACTIVE NORMAL linux1 2 2 racdb2 YES OPEN ACTIVE NORMAL linux2 All data files which are in the disk group select union select union select union select name from v$datafile member from v$logfile name from v$controlfile name from v$tempfile.---------.168.695350875 +RACDB_DATA1/racdb/controlfile/current.260.1.255. parallel .200/255.-------.695345395 +FLASH_RECOVERY_AREA/racdb/onlinelog/group_1.0/eth0 GSD exists.VIP exists.695345403 +FLASH_RECOVERY_AREA/racdb/onlinelog/group_2. instance_name inst_name .--------.256.------.258.

695350865 +RACDB_DATA1/racdb/onlinelog/group_4. Ensure that you are logged in as the "oracle" UNIX user.267. a popular question might be.695350871 +RACDB_DATA1/racdb/tempfile/temp.260. Or you may find that Enterprise Manager is not running and need to start it.257. All ASM disk that belong to the 'RACDB_DATA1' disk group SELECT path FROM v$asm_disk WHERE group_number IN (select group_number from v$asm_diskgroup where name = 'RACDB_DATA1'). Enterprise Manager Database Console.695345399 +RACDB_DATA1/racdb/onlinelog/group_2. We have all of the required software installed and configured plus we have a fully functional clustered database. PATH ---------------------------------ORCL:VOL1 ORCL:VOL2 Starting / Stopping the Cluster At this point.+RACDB_DATA1/racdb/datafile/undotbs1.265. however. This section provides the commands responsible for starting and stopping the cluster environment.695345441 +RACDB_DATA1/racdb/datafile/undotbs1. This would include Oracle Clusterware.264. If you have followed the instructions in this article. etc.695352519 +RACDB_DATA1/racdb/datafile/undotbs2.695352363 +RACDB_DATA1/racdb/onlinelog/group_1. all services should start automatically on each reboot of the Linux nodes. With all of the work we have done up to this point.269. I will be running all of the commands in this section from linux1: # su .258.262. all Oracle instances.695352571 +RACDB_DATA1/racdb/datafile/users. There are times.695345459 21 rows selected.695345475 +RACDB_DATA1/racdb/datafile/undotbs2.695345405 +RACDB_DATA1/racdb/onlinelog/group_3. "How do we start and stop services?".266.oracle $ hostname .695345485 +RACDB_DATA1/racdb/datafile/users. everything has been installed and configured for Oracle RAC 10g.272.271. when you might want to shutdown a node and manually start it back up.

Oracle RAC 10g provides a superior solution with its advanced failover mechanisms. Once the node applications are successfully started. TNS Listener. To answer many of these high availability requirements. and ONS). and ONS). Think about what it would take to ensure a downtime of no more than .linux1 Stopping the Oracle RAC 10g Environment The first step is to stop the Oracle instance.5 hours or even no downtime during the year.(TAF) Overview It is not uncommon for businesses of today to demand 99. $ $ $ $ $ export ORACLE_SID=racdb1 srvctl start nodeapps -n linux1 srvctl start asm -n linux1 srvctl start instance -d racdb -i racdb1 emctl start dbconsole Start / Stop All Instances with SRVCTL Start / Stop all of the instances and its enabled services. then bring up the ASM instance.999% availability for their enterprise applications. GSD. Finally. Oracle RAC 10g includes the required components that all work within a clustered configuration . When considering the availability of the Oracle database. Once the instance (and related services) is down. businesses are investing in mechanisms that provide for automatic failover when one participating system fails. I just included this for fun as a way to bring down all instances! $ srvctl start database -d racdb $ srvctl stop database -d racdb Transparent Application Failover . shutdown the node applications (Virtual IP. bring up the Oracle instance (and related services) and the Enterprise Manager Database console. Finally. TNS Listener. GSD.99% or even 99. $ $ $ $ $ export ORACLE_SID=racdb1 emctl stop dbconsole srvctl stop instance -d racdb -i racdb1 srvctl stop asm -n linux1 srvctl stop nodeapps -n linux1 Starting the Oracle RAC 10g Environment The first step is to start the node applications (Virtual IP. then bring down the ASM instance.

All database connections (and processes) that loose connections are reconnected to another node within the cluster. A major component of Oracle RAC 10g that is responsible for failover processing is the Transparent Application Failover (TAF) option. The failover is completely transparent to the user. It provides all of the necessary configuration parameters for load balancing and failover.responsible for providing continuous availability ..ora file. My intention here is to present a brief overview and example of how it works.ora.ora file on a non-RAC client machine (if you have a Windows machine lying around). Certain configuration steps.when one of the participating systems fail within the cluster.. (Actually. we need to verify that a valid entry exists in the tnsnames. This means that your application (client) code does not need to change in order to take advantage of TAF. You can copy the contents of this entry to the %ORACLE_HOME%\network\admin\tnsnames. Please note that a complete discussion on failover in Oracle RAC 10g would be an article in of its own.) During the creation of the clustered database in this article. you only need a client install of the Oracle software. Setup tnsnames. One important note before continuing is that TAF happens automatically within the OCI libraries.info) (FAILOVER_MODE = (TYPE = SELECT) (METHOD = BASIC) (RETRIES = 180) (DELAY = 5) . using the Java thin client will not be able to participate in TAF since it never reads the tnsnames. Ensure that you have Oracle RDBMS software installed. however.ora file on the client machine (my Windows laptop is being used in this example) in order to connect to the new Oracle clustered database: tnsnames.idevelopment. will need to be done on the Oracle TNS file tnsnames. I created a new service that will be used for testing TAF named racdb_srvc.ora File Entry for Clustered Database . RACDB_SRVC = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = linux1-vip)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = linux2-vip)(PORT = 1521)) (LOAD_BALANCE = yes) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = racdb_srvc. Keep in mind that at the time of this article. the users are automatically migrated to the other available systems.ora File Before demonstrating TAF. This final section provides a short demonstration on how automatic failover works in Oracle RAC 10g.

host_name . and if a failover has occurred. NULL .) ) ) . We will be using this query throughout this example. NULL AS failover_method . COLUMN COLUMN COLUMN COLUMN instance_name host_name failover_method failed_over FORMAT FORMAT FORMAT FORMAT a13 a9 a15 a11 SELECT instance_name . failover_method . NULL AS failover_method . failed_over FROM v$session WHERE username = 'SYSTEM'. failover method. failover_type . log in to the clustered database using the racdb_srvc service as the SYSTEM user: C:\> sqlplus system/manager@racdb_srvc COLUMN COLUMN COLUMN COLUMN instance_name host_name failover_method failed_over FORMAT FORMAT FORMAT FORMAT a13 a9 a15 a11 SELECT instance_name . SQL Query to Check the Session's Failover Information The following SQL query can be used to check a session's failover type. NULL AS failed_over FROM v$instance UNION SELECT NULL . Transparent Application Failover Demonstration From a Windows machine (or other non-RAC client machine). NULL AS failover_type . NULL AS failover_type .. NULL AS failed_over . host_name ..

failed_over FROM v$session WHERE username = 'SYSTEM'. host_name . NULL AS failed_over FROM v$instance UNION SELECT NULL .------------. . we can use the srvctl command-line utility as follows: # su .oracle $ srvctl status database -d racdb Instance racdb1 is running on node linux1 Instance racdb2 is running on node linux2 $ srvctl stop instance -d racdb -i racdb1 -o abort $ srvctl status database -d racdb Instance racdb1 is not running on node linux1 Instance racdb2 is running on node linux2 Now let's go back to our SQL session and rerun the SQL statement in the buffer: COLUMN COLUMN COLUMN COLUMN instance_name host_name failover_method failed_over FORMAT FORMAT FORMAT FORMAT a13 a9 a15 a11 SELECT instance_name . failover_type .----------racdb1 linux1 SELECT BASIC NO DO NOT logout of the above SQL*Plus session! Now that we have run the query (above).--------. we should now shutdown the instance racdb1 on linux1 using the abort option. NULL AS failover_type . failover_type .--------------. failover_method . NULL . INSTANCE_NAME HOST_NAME FAILOVER_TYPE FAILOVER_METHOD FAILED_OVER ------------. NULL .FROM v$instance UNION SELECT NULL . NULL AS failover_method . failed_over FROM v$session WHERE username = 'SYSTEM'. failover_method . To perform this operation.

--------. we can see that the above session has now been failed over to instance racdb2 on linux2. If the machine name is listed in the in the loopback address entry as below: 127. For example: CRS-0215 could not start resource 'ora.INSTANCE_NAME HOST_NAME FAILOVER_TYPE FAILOVER_METHOD FAILED_OVER ------------.2.localdomain localhost it will need to be removed as shown below: If the RAC node name is listed for the loopback address.1 localhost.log" for more details.--------------.localdomain and localhost are included for the loopback address in the /etc/hosts file for each of the Oracle RAC nodes: 127.0.ons.0. Troubleshooting Confirm the RAC Node Name is Not Listed in Loopback Address Ensure that the node names (linux1 or linux2) are not included for the loopback address in the /etc/hosts file.----------racdb2 linux2 SELECT BASIC YES SQL> exit From the above demonstration.localdomain localhost If an entry does not exist for localhost in the /etc/hosts file. The ONS log file will contain lines similar to the following: Oracle Database 10g CRS Release 10.------------.0.linux1.ons'.1 linux1 localhost. .localdomain localhost localhost.1 127.0. Check log file "/u01/app/crs/log/linux1/racg/ora. The error would indicate "Failed to get IP for localhost" and will be written to the log file for ONS.0 Production Copyright 1996.0.0.0.linux1.1. you will receive the following error during the RAC installation: ORA-00603: ORACLE server session terminated by fatal error or ORA-29702: error occurred in Cluster Group Service operation Confirm localhost is defined in the /etc/hosts file for the loopback address Ensure that the entry for localhost. Oracle Clusterware will be unable to start the application resources — notably the ONS process.

The "tar" command DOES actually extract the files. During the remote copy process. the Oracle Universal Installer (OUI) first installs the software to the local node running the installer (i.(more errors on this node) Please note that although this would seem like a severe error from the OUI. [PRKC-1002 : All the submitted commands did not execute successfully] --------------------------------------------linux2: /bin/tar: . they will be missing the time field until the time on the remote server is greater than the timestamp of the file. when you perform a listing of the files (using ls -l) on the remote node.. The software is then copied remotely to all of the remaining nodes in the cluster (i. and the Companion CD.. the Database./bin/lsnodes: time stamp 2009-08-16 09:21:34 is 735 s in the future /bin/tar: . the OUI will execute the UNIX "tar" command on each of the remote nodes to extract the files that were archived and copied over./bin/olsnodes: time stamp 2009-08-16 09:21:34 is 735 s in the future . it can safely be disregarded as a warning... ensure that the date and time of the node you are performing the software installations from (linux1) is less than all other nodes in the cluster (linux2).linux1. Setting the Correct Date and Time on Both Oracle RAC Nodes During the installation of Oracle Clusterware. may not always be an option. Before starting any of the above noted installations.729: [ RACG][3086871296][13316][3086871296][ora. with all nodes using the same reference Network Time Protocol server. If the date and time on the node performing the install is greater than that of the node it is copying to. I generally use a 20 second difference as shown in the following example: Setting the date and time from linux1: # date -s "8/16/2009 01:12:00" Setting the date and time from linux2: . however. In this case. the OUI will throw an error from the " tar" command indicating it is attempting to extract files stamped with a time in the future: Error while copying directory /u01/app/crs with exclude file list 'null' to nodes 'linux2'. ensure that each member node of the cluster is set as closely as possible to the same date and time. linux2). however.e. linux1).2005 Oracle. 2007-04-14 13:10:02. Oracle strongly recommends using the Network Time Protocol feature of most operating systems for this purpose. when manually setting the date and time for the nodes in the cluster.e. All rights reserved.ons]: Failed to get IP for localhost (1) Failed to get IP for localhost (1) Failed to get IP for localhost (1) onsctl: ons failed to start . Accessing a Network Time Protocol server.

14 iotype_init(91) register fileio iotype_init(91) register blockio iotype_init(91) register nullio open_path(120) Can't open /dev/rac1/crs -2 fileio_attach(268) -2 open_path(120) Can't open /dev/rac1/asm1 -2 fileio_attach(268) -2 open_path(120) Can't open /dev/rac1/asm2 -2 fileio_attach(268) -2 open_path(120) Can't open /dev/rac1/asm3 -2 fileio_attach(268) -2 open_path(120) Can't open /dev/rac1/asm4 -2 fileio_attach(268) -2 Please note that I am not suggesting that this only occurs with USB drives connected to the Openfiler server. Openfiler . When the Openfiler server is rebooted.00 GB] inherit '/dev/rac1/asm1' [115. you should first check the status of all logical volumes using the lvscan command from the Openfiler server: # lvscan inactive inactive inactive inactive inactive '/dev/rac1/crs' [2. It may occur with other types of drives.94 GB] inherit Notice that the status for each of the logical volumes is set to inactive .Logical Volumes Not Active on Boot One issue that I have run into several times occurs when using a USB drive connected to the Openfiler server.version 0.94 GB] inherit '/dev/rac1/asm4' [115. Method 1 .4.(also available through dmesg): iSCSI Enterprise Target Software .94 GB] inherit '/dev/rac1/asm3' [115. it is not able to load the logical volumes and writes the following message to /var/log/messages .(the status for each logical volume on a working system would be set to ACTIVE).# date -s "8/16/2009 01:12:20" The two-node RAC configuration described in this article does not make use of a Network Time Protocol server. I currently know of two methods to get Openfiler to automatically load the logical volumes on reboot.94 GB] inherit '/dev/rac1/asm2' [115. however I have only seen it with USB drives! If you do receive this error. the system is able to recognize the USB drive however. both of which are described below.

00 GB] inherit '/dev/rac1/asm1' [115.static -v 0 if [ -x /sbin/kpartx ]. reboot the Openfiler server to ensure each of the logical volumes will be set to ACTIVE after the boot process. then /sbin/dmsetup ls --target multipath --exec "/sbin/kpartx - . Then.static ].One of the first steps is to shutdown both of the Oracle RAC nodes in the cluster .sysinit script (described below). Found volume group "rac1" using metadata type lvm2 # vgchange -ay 5 logical volume(s) in volume group "rac1" now active After setting each of the logical volumes to active.. After making the changes to the /etc/rc.94 GB] inherit '/dev/rac1/asm4' [115. take 2 if [ -c /dev/mapper/control ].. then if [ -x /sbin/multipath.(linux1 and linux2). Method 2 This method was kindly provided by Martin Jones.94 GB] inherit '/dev/rac1/asm3' [115. This may take a while.94 GB] inherit As a final test.sysinit script on the Openfiler server with the changes (highlighted in blue) proposed by Martin: Make Modifications to /etc/rc.. check that the iSCSI target service is running: # service iscsi-target status ietd (pid 2668) is running. His workaround includes amending the /etc/rc. use the lvscan command again to verify the status: # lvscan ACTIVE ACTIVE ACTIVE ACTIVE ACTIVE '/dev/rac1/crs' [2. verify the external drives are powered on and then reboot the Openfiler server. from the Openfiler server. The following is a small portion of the /etc/rc. Finally. restart each of the Oracle RAC nodes in the cluster .sysinit script to basically wait for the USB disk ( /dev/sda in my example) to be detected. then modprobe dm-multipath >/dev/null 2>&1 /sbin/multipath..(linux1 and linux2). After you have verified that each of the logical volumes will be active on boot. manually set each of the logical volumes to ACTIVE for each consecutive reboot: # # # # # lvchange lvchange lvchange lvchange lvchange -a -a -a -a -a y y y y y /dev/rac1/crs /dev/rac1/asm1 /dev/rac1/asm2 /dev/rac1/asm3 /dev/rac1/asm4 Another method to set the status to active for all logical volumes is to use the Volume Group change command as follows: # vgscan Reading all physical volumes.sysinit # LVM2 initialization.94 GB] inherit '/dev/rac1/asm2' [115.

a" fi fi if [ -x /sbin/dmraid ]; then modprobe dm-mirror > /dev/null 2>&1 /sbin/dmraid -i -a y fi #----#----#----MJONES - Customisation Start # Check if /dev/sda is ready while [ ! -e /dev/sda ] do echo "Device /dev/sda for first USB Drive is not yet ready." echo "Waiting..." sleep 5 done echo "INFO - Device /dev/sda for first USB Drive is ready." #----#----#----MJONES - Customisation END

if [ -x /sbin/lvm.static ]; then if /sbin/lvm.static vgscan > /dev/null 2>&1 ; then action $"Setting up Logical Volume Management:" /sbin/lvm.static vgscan --mknodes --ignorelockingfailure && /sbin/lvm.static vgchange -a y --ignorelockingfailure fi fi fi # Clean up SELinux labels if [ -n "$SELINUX" ]; then for file in /etc/mtab /etc/ld.so.cache ; do [ -r $file ] && restorecon $file >/dev/null 2>&1 done fi

Finally, restart each of the Oracle RAC nodes in the cluster - (linux1 and linux2).

OCFS2 - Configure O2CB to Start on Boot With the releases of OCFS2 prior to 1.2.1, there is a bug that exists where the driver does not get loaded on each boot even after configuring the on-boot properties to do so. Note that this section does not apply here since the version of OCFS2 used in this article is greater than release 1.2.1.

After attempting to configure the on-boot properties to start on each boot according to the official OCFS2 documentation, you will still get the following error on each boot:
... Mounting other filesystems: mount.ocfs2: Unable to access cluster service Cannot initialize cluster mount.ocfs2: Unable to access cluster service Cannot initialize cluster [FAILED] ...

Red Hat changed the way the service is registered between chkconfig-1.3.11.2-1 and chkconfig-1.3.13.2-1. The O2CB script used to work with the former. Before attempting to configure the on-boot properties:
        

REMOVE the following lines in /etc/init.d/o2cb
### BEGIN INIT INFO # Provides: o2cb # Required-Start: # Should-Start: # Required-Stop: # Default-Start: 2 3 5 # Default-Stop: # Description: Load O2CB cluster services at system boot. ### END INIT INFO

      

Re-register the o2cb service.
# chkconfig --del o2cb # chkconfig --add o2cb # chkconfig --list o2cb o2cb 0:off 1:off

2:on

3:on

4:on

5:on

6:off

# ll /etc/rc3.d/*o2cb* lrwxrwxrwx 1 root root 14 Sep 29 11:56 /etc/rc3.d/S24o2cb -> ../init.d/o2cb

The service should be S24o2cb in the default runlevel. After resolving the bug I listed above, we can now continue to set the on-boot properties as follows:
# /etc/init.d/o2cb offline ocfs2 # /etc/init.d/o2cb unload # /etc/init.d/o2cb configure Configuring the O2CB driver. This will configure the on-boot properties of the O2CB driver. The following questions will determine whether the driver is loaded on boot. The current values will be shown in brackets ('[]'). Hitting <ENTER> without typing an answer will keep that current value. Ctrl-C will abort. Load O2CB driver on boot (y/n) [n]: y Cluster to start on boot (Enter "none" to clear) [ocfs2]: ocfs2

Writing O2CB configuration: OK Loading module "configfs": OK Mounting configfs filesystem at /config: OK Loading module "ocfs2_nodemanager": OK Loading module "ocfs2_dlm": OK Loading module "ocfs2_dlmfs": OK Mounting ocfs2_dlmfs filesystem at /dlm: OK Starting cluster ocfs2: OK

OCFS2 - o2cb_ctl: Unable to access cluster service while creating node While configuring the nodes for OCFS2 using ocfs2console, it is possible to run into the error:
o2cb_ctl: Unable to access cluster service while creating node

This error does not show up when you startup ocfs2console for the first time. This message comes up when there is a problem with the cluster configuration or if you do not save the cluster configuration initially while setting it up using ocfs2console. This is a bug! The work-around is to exit from the ocfs2console, unload the o2cb module and remove the ocfs2 cluster configuration file /etc/ocfs2/cluster.conf. I also like to remove the /config directory. After removing the ocfs2 cluster configuration file, restart the ocfs2console program. For example:
# /etc/init.d/o2cb offline ocfs2 # /etc/init.d/o2cb unload Unmounting ocfs2_dlmfs filesystem: OK Unloading module "ocfs2_dlmfs": OK Unmounting configfs filesystem: OK Unloading module "configfs": OK # rm -f /etc/ocfs2/cluster.conf # rm -rf /config # ocfs2console &

This time, it will add the nodes!

OCFS2 - Adjusting the O2CB Heartbeat Threshold With previous versions of this article, (using FireWire as opposed to iSCSI for the shared storage), I was able to install and configure OCFS2, format the new volume, and finally install Oracle Clusterware (with its two required shared files; the voting disk and OCR file), located on the new OCFS2 volume. While I was able to install Oracle Clusterware and see the shared drive using FireWire, however, I was receiving many lock-ups and hanging after about 15 minutes when the Clusterware software was running on both nodes. It always varied on which node would hang (either linux1 or linux2 in my example). It also didn't matter whether there was a high I/O load or none at all for it to crash (hang).

. we would need to adjust O2CB_HEARTBEAT_THRESHOLD to 61 as shown below: (61 . Kernel panic . the following describes how to manually adjust the O2CB heartbeat threshold. First.1) * 2 = 120 seconds Let's see now how to manually increase the O2CB heartbeat threshold from 7 to 61. We first need to modify the file /etc/sysconfig/o2cb and set O2CB_HEARTBEAT_THRESHOLD to 61: /etc/sysconfig/o2cb # . During the installation and configuration of OCFS2. This is a configurable parameter that is used to compute the time it takes for a node to "fence" itself.not syncing: ocfs2 is very sorry to be fencing this system by panicing The solution I used was to increase the O2CB heartbeat threshold from its default value of 7. Some setups may require an even higher setting. we would have a fence time of: (7 . we adjusted this value in the section "Configure O2CB to Start on Boot and Adjust O2CB Heartbeat Threshold".. to 61.After looking through the trace files for OCFS2. This task will need to be performed on all Oracle RAC nodes in the cluster. This can be done by querying the /proc file system as follows: # cat /proc/fs/ocfs2_nodemanager/hb_dead_threshold 7 We see that the value is 7. it is used in the formula below to determine the fence time (in seconds): [fence time in seconds] = (O2CB_HEARTBEAT_THRESHOLD . with an O2CB heartbeat threshold of 7.2. On the console would be a message similar to the following: . If you encounter a kernel panic from OCFS2 and need to increase the heartbeat threshold. Index 0: took 0 ms to do submit_bio for read Index 1: took 3 ms to do waiting for read completion Index 2: took 0 ms to do bio alloc write Index 3: took 0 ms to do bio add page write Index 4: took 0 ms to do submit_bio for write Index 5: took 0 ms to do checking slots Index 6: took 4 ms to do waiting for write completion Index 7: took 1993 ms to do msleep Index 8: took 0 ms to do allocating bios for read Index 9: took 0 ms to do bio alloc read Index 10: took 0 ms to do bio add page read Index 11: took 0 ms to do submit_bio for read Index 12: took 10006 ms to do waiting for read completion (13.3):o2hb_stop_all_regions:1888 ERROR: stopping heartbeat on all active regions.1) * 2 So. it was apparent that access to the voting disk was too slow (exceeding the O2CB heartbeat threshold) and causing the Oracle Clusterware software (and the node) to crash.2-1).1) * 2 = 12 seconds If we want a larger threshold (say 120 seconds). let's see how to determine what the O2CB heartbeat threshold is currently set to. but what does this value represent? Well. If you are using an earlier version of OCFS2 tools (prior to ocfs2-tools release 1. use the same procedures described in the section "Configure O2CB to Start on Boot and Adjust O2CB Heartbeat Threshold".

Again.d/o2cb configure. this should be performed on all Oracle RAC nodes in the cluster. O2CB_IDLE_TIMEOUT_MS= # O2CB_KEEPALIVE_DELAY_MS: Max time in ms before a keepalive packet is sent O2CB_KEEPALIVE_DELAY_MS= # O2CB_RECONNECT_DELAY_MS: Min time in ms between connection attempts O2CB_RECONNECT_DELAY_MS= After modifying the file /etc/sysconfig/o2cb. # Please use that method to modify this file # # O2CB_ENABELED: 'true' means to load the driver on boot. This will configure the on-boot properties of the O2CB driver. O2CB_HEARTBEAT_THRESHOLD=61 # O2CB_IDLE_TIMEOUT_MS: Time in ms before a network connection is considered dead.# This is a configuration file for automatic startup of the O2CB # driver. It is generated by running /etc/init. O2CB_BOOTCLUSTER=ocfs2 # O2CB_HEARTBEAT_THRESHOLD: Iterations before a node is considered dead. Ctrl-C will abort. we need to alter the o2cb configuration.d/o2cb configure Configuring the O2CB driver. Load O2CB driver on boot (y/n) [n]: y Cluster to start on boot (Enter "none" to clear) [ocfs2]: ocfs2 Specify heartbeat dead threshold (>=7) [7]: 61 Specify network idle timeout in ms (>=5000) [10000]: 10000 Specify network keepalive delay in ms (>=1000) [5000]: 5000 Specify network reconnect delay in ms (>=2000) [2000]: 2000 Writing O2CB configuration: OK Loading module "configfs": OK Mounting configfs filesystem at /config: OK Loading module "ocfs2_nodemanager": OK Loading module "ocfs2_dlm": OK Loading module "ocfs2_dlmfs": OK Mounting ocfs2_dlmfs filesystem at /dlm: OK . Hitting <ENTER> without typing an answer will keep that current value. the name of a cluster to start. # umount /u02 # /etc/init.d/o2cb unload # /etc/init. The current values will be shown in brackets ('[]'). The following questions will determine whether the driver is loaded on boot. O2CB_ENABLED=true # O2CB_BOOTCLUSTER: If not empty.d/o2cb offline ocfs2 # /etc/init.

In some cases. running root. please leave the OUI up.4. when installed.4. the O2CB heartbeat threshold value had to be increased to as high as 601 in order to prevent OCFS2 from panicking the kernel. add an unset command to ensure LD_ASSUME_KERNEL is not set as follows: if [ "$arch" = "i686" -o "$arch" = "ia64" ] then LD_ASSUME_KERNEL=2. This error occurs because these releases of the Linux kernel fix an old bug in the Linux threading that Oracle worked around using LD_ASSUME_KERNEL settings in both vipca and srvctl. RDBMS and ASM bin directories on all nodes). edit vipca (in the CRS bin directory on all nodes) to undo the setting of LD_ASSUME_KERNEL.sh Fails on the Last Node After the Oracle Clusterware install process.0: cannot open shared object file: No such file or directory After receiving this error. After the IF statement around line 120. Do not hit the OK button on the "Execute Configuration Scripts" dialog until all of the issues described in this section have been resolved.so.19 export LD_ASSUME_KERNEL fi unset LD_ASSUME_KERNEL <<== Line to be added Similarly for srvctl (in both the CRS and. this workaround is no longer valid on OEL5 or RHEL5 or SLES10 hence the failures. Note that srvctl will produce similar output until the workaround described below is performed. around line 168 should look like this: LD_ASSUME_KERNEL=2.sh on the last node will fail while attempting to configure vipca at the end of the script: Oracle CRS stack installed and running under init(1M) Running vipca(silent) for configuring nodeapps /u01/app/crs/jdk/jre//bin/java: error while loading shared libraries: libpthread. Oracle Clusterware Installation: Running root. unset LD_ASSUME_KERNEL by adding one line.Starting O2CB cluster ocfs2: OK We can now check again to make sure the settings took place in for the o2cb cluster stack: # cat /proc/fs/ocfs2_nodemanager/hb_dead_threshold 61 It is important to note that the value of 61 I used for the O2CB heartbeat threshold will not work for all configurations. To workaround this issue.19 export LD_ASSUME_KERNEL unset LD_ASSUME_KERNEL <<== Line to be added .

0 global public eth1 192.0 global cluster_interconnect After resolving all of the issues above.168..0 Remember during the Oracle Clusterware install that 192.0.168.1. This issue was raised with development and is fixed in the 10.168.x]: [root@linux2 ~]# $ORA_CRS_HOME/bin/vipca Error 0(Native: listNetInterfaces:[3]) [Error 0(Native: listNetInterfaces:[3])] There are several ways to workaround this issue.2.2. as these patchset will still include those settings unnecessary for OEL5 or RHEL5 or SLES10.x.2. Also note that we are explicitly unsetting LD_ASSUME_KERNEL and not merely commenting out its setting to handle a case where the user has it set in their environment (login shell).168. Please keep in mind that vipca is a GUI and will need to set your DISPLAY variable accordingly to your X server: .2.168. or 192.x.2.0 eth0 192. manually re-run vipca (GUI) as root from the last node in which the errors occurred..0:cluster_interconnect Let's know run the "$ORA_CRS_HOME/bin/oifcfg getif" command again to verify its output: [root@linux2 ~]# $ORA_CRS_HOME/bin/oifcfg getif eth0 192.1. vipca will now fail to run with the following error if the VIP IP's are in a non-routable range [10.0:public # $ORA_CRS_HOME/bin/oifcfg setif -global eth1/192.0 is the cluster_interconnect interface.3 patch sets.(16-31).x.2.0.2. you will notice it provides nothing which means we have some work to do! The first step is to identify the current interfaces and IP addresses: [root@linux2 ~]# $ORA_CRS_HOME/bin/oifcfg iflist eth1 192. After working around the LD_ASSUME_KERNEL issue above. The goal to this workaround is to get the output of "$ORA_CRS_HOME/bin/oifcfg getif" to include both public and cluster_interconnect interfaces.0 is my public interface while 192.1. we can manually set the public / private interfaces accordingly using the setif option of the $ORA_CRS_HOME/bin/oifcfg command: # $ORA_CRS_HOME/bin/oifcfg setif -global eth0/192.168.168. after applying the 10.168.0. 172.2 or 10.4 patchsets.1.x.168.x. If you try to run the above command.x.Note: Remember to re-edit these files on all nodes: <ORA_CRS_HOME>/bin/vipca <ORA_CRS_HOME>/bin/srvctl <RDBMS_HOME>/bin/srvctl <ASM_HOME>/bin/srvctl # (If exists) . Using this information.

but shared with me his experience and knowledge of the product and how to best utilize it for Oracle RAC. Building an Oracle RAC Cluster Remotely An iDevelopment. The RAC solution presented in this article can be put together for around US$2. If the lab is available. This book comes highly recommended . there are several other individuals that deserve credit in making this article a success.700 and will provide the DBA with a fully functional Oracle10g Release 2 RAC cluster. Please email him at "Bryan" AT "AHCCINC" DOT "COM" with the words "RAC LAB" as the subject. First. much of the content regarding the history of Oracle RAC can be found in his very popular book Oracle Database 10g Real Application Clusters Handbook. A special thanks to K Gopalakrishnan for his assistance in delivering the Oracle RAC 10g Overview section of this article. however. For those DBAs. His research and hard work made the task of configuring Openfiler seamless. it should never be considered for a production environment.000 to US$20. I would like to thank Bane Radulovic from the Server BDE Team at Oracle. you can start building your RAC now! Acknowledgements An article of this magnitude and complexity is generally not the work of one person alone. While the hardware used for this article should be stable enough for educational purposes. Although I was able to author and successfully demonstrate the validity of the components that make up this configuration. Bane was also involved with hardware recommendations and testing. that want to become more familiar with the features and benefits of Oracle10g RAC will find the costs of configuring even a small RAC cluster costing in the range of US$15. The cost is $14 for 7 days full access.000.# $ORA_CRS_HOME/bin/vipca Conclusion Oracle10g RAC allows the DBA to configure a database solution with superior fault tolerance and load balancing. Bane not only introduced me to Openfiler.info reader is now offering 3 computers that will allow you to REMOTELY build an Oracle RAC Cluster from scratch using a VNC enabled KVM. In this section. This article has hopefully given you an economical solution to setting up and configuring an inexpensive Oracle10g Release 2 RAC Cluster using CentOS 5 and iSCSI technology.

and Windows server environment. Jeff's other interests include mathematical encryption theory. Jeff currently works as a Senior Database Administrator for The DBA Zone. Linux. database security. Intel. About the Author Jeffrey Hunter is an Oracle Certified Professional. Avocent Corporation. Database Administrator and Software Engineer for over 17 years and maintains his own website site at: http://www. please visit his website at www. California. Java and PL/SQL programming. SIIG. He has been a Sr. and an Oracle ACE. developing high availability solutions. His work includes advanced performance tuning.for both DBAs and Developers wanting to successfully implement Oracle RAC and fully understand how many of the advanced services like Cache Fusion and Global Resource Directory operate. located in Pittsburgh.iDevelopment. Stallard Technologies. Seagate. Inc. Jeff graduated from Stanislaus State University in Turlock. provided information on Oracle RAC that could not be found in any other Oracle documentation. capacity planning.puschitz. This article. LDAP. Author. Also. . D-Link.info. and physical / logical database design in a UNIX. If you are interested in examining technical articles on Linux internals and indepth Oracle configurations written by Werner Puschitz. Java Development Certified Professional. Lastly.com. Pennsylvania. and LaCie. I would like to express my appreciation to the following vendors for generously supplying the hardware for this article. Inc. this article may have never come to fruition. writing web-based database administration tools. with a Bachelor's degree in Computer Science. Without his hard work and research into issues like configuring OCFS2 and ASMLib. along with several others of his. thanks to Tzvika Lemel for his comments and suggestions on using Oracle's Cluster Verification Utility (CVU). and of course Linux. I would next like to thank Oracle ACE Werner Puschitz for his outstanding work on "Installing Oracle Database 10g with Real Application Cluster (RAC) on Red Hat Enterprise Linux Advanced Server 3".. programming language processors (compilers and interpreters) in Java and C.