Oracle Real Application Cluster Field DBA Admin Handbook

Flavia D’Souza

www.Lulu.com

Oracle Real Application Cluster Field DBA Admin Handbook By Flavia D’Souza

Copyright @ 2008 www.Lulu.com . All rights reserved

No part of this Publication may be reproduced, or transmitted in any form or by any means, electronic, photocopying, recording, or otherwise, without the prior permission of the Publisher.

Printing History August 2008: First Edition Minor Formatting

September 2008:

ISBN: 978-1-4092-2478-5

ii

This book is dedicated to the people who Inspire me
to do what I enjoy to do –My Family. To my parents who taught me the value of hardwork, persistence and humbleness. To my husband who is supportive and my daughter for her appreciative gesture and to the people who inspire me through their inspirational writing to contribute something back to the society.

iii

iv .

.139 Chapter 10 Performance Monitoring on RAC database ..............31 Chapter 4 Administering Database Instances in RAC.................49 Chapter 5 Storage Administration ......262 v ................................. RAC related useful Metalink Notes .................... Metalink information Search .....................................................13 Chapter 3 Oracle Clusterware Administration ......................................................203 Chapter 13 Design and Development Techniques ..................1 Chapter 2 RAC Architecture and Concepts.........................242 4......................161 Chapter 11 Administrative Options .....................................................219 Glossary .......................183 Chapter 12 Workload management...........243 5.......................241 3...............................239 1............................................................Contents at a Glance Chapter 1 Oracle Real Application Cluster Architecture ...........................93 Chapter 7 Adding and deleting nodes and instances .. RAC Frequently Asked Questions ...............129 Chapter 9 Managing Backup & Recovery............. RAC related useful Weblink References .........................240 2....75 Chapter 6 Administering Services .....223 Reference ....... Handy commands.........................................................109 Chapter 8 Recovery Manager Configuration and Archiving ....................

vi .

...............................................................................2 Main Components of Oracle Real Application Cluster 10g ............17 Shared storage Technologies and RAC .................13 RAC Software Principles................1 What is a Cluster ?.......19 Cluster Stack with RAC ...................................................3 Real Application Cluster Main Processes .....6 RAC database Storage Principles ................................................25 Additional Memory Requirement for RAC.............26 RAC and Services........5 Oracle Clusterware High Availability and the Application Programming Interface .................................................16 Storage options for RAC .....................................11 Chapter 2 RAC Architecture and Concepts......................8 Definitions of different storage in Real Application Clusters ...........................................9 What is a raw device? ..................................................................................10 Automatic Storage Management ....................17 The Oracle Clusterware Voting Disk and Oracle Cluster Registry .......................9 What is a Cluster File system (CFS) ?..............................................21 RAC and Instance/Crash Recovery .................................................................................28 Database Control and RAC .28 vii .................................14 RAC Software Storage Principle ..........................................................................................................7 RAC and Shared Storage Technologies ............................................................................................................................4 Storage Principles for RAC Software and CRS ................................................................................Contents Chapter 1 Oracle Real Application Cluster Architecture .......26 RAC and Global Dynamic Performance Views ..........27 Virtual IP Addresses and RAC ...........14 The Oracle Clusterware Software Components .......................

..................................51 Starting up and Shutdown with Enterprise Manager............................................................57 Cluster specific parameters used in Real Application Clusters Databases ................52 Starting up and Shutting down with srvctl ...............................................................................59 ADDING A NODE TO 10g RAC CLUSTER ......50 Administering Real Application Clusters with Enterprise Manager.60 REMOVING A NODE FROM 10g RAC CLUSTER ..32 Recovering Voting Disks....................45 Chapter 4 Administering Database Instances in RAC...35 Managing Backups and Recovery of the OCR Using OCR Backup Files .........................................................................................Chapter 3 Oracle Clusterware Administration .......................................................50 Administering Real Application Clusters with srvctl ....39 OCRCHECK Utility to verify the OCR Integrity .................................................................................43 Upgrading and Downgrading the OCR Configuration in Real Application Clusters ....................67 viii ...31 Administering Oracle Clusterware Components ......................................................................33 Repairing an Oracle Cluster Registry Configuration on a Local Node....................................................................51 Starting and Stopping Instances and Real Application Clusters Databases .................................33 Oracle Cluster Registry administration in Real Application Clusters ..52 Initialization Parameter Used in Real Application Clusters ..............36 Diagnosing OCR Problems with the OCRDUMP and OCRCHECK Utilities.......................32 Backing up Voting Disks..............................40 Administering the Oracle Cluster Registry with OCR Exports..............................32 Changing the Voting Disk Configuration after Installing Real Application Clusters..............49 Database Component Overview .....................

.....................85 Administering ASM Instances and ASM Disk Groups with Enterprise Manager in RAC ...............................................................................94 Grid workload dispatching ....86 Administering the Oracle Cluster Registry in Real Application Clusters ...................................91 Chapter 6 Administering Services ..................................97 Application services ..................83 Automatic Storage Management ..........................78 Overview of Storage in Oracle Real Application Clusters ..................................87 Administering Oracle Cluster Registry Backup Files in Real Application Clusters.................94 Using Services in Real Application Clusters Environments...........76 ASM General Architecture ................................................98 ix ........93 Services and High Availability in Real Application Clusters ...................................................................................................................................95 Adding and Modifying Services..........................................................................................98 Internal services ...................................82 Administering ASM Instances and Disk Groups with Enterprise Manager in RAC ..........88 Administering the OCR with OCR Exports ...............................................................75 What is Automatic Storage Management ............83 Administering ASM Instances with SRVCTL in RAC..................................................................97 High Availability of Services in RAC ................................81 Standalone ASM Disk Group Management ...................85 Administering ASM Instances with SRVCTL in RAC.................79 Automatic Storage Management in Real Application Clusters .............................94 Traditional workload dispatching ..........................90 The ocrconfig Tool Command Syntax and Options ....................................................Chapter 5 Storage Administration .....85 Modifying Disk Group Config for ASM in RAC......87 Restoring the Oracle Cluster Registry from Automatically Generated OCR Backups ..........

...........................................................................................................132 RMAN Archiving Configuration Scenarios .....................................................................................................121 ASM Instance Clean-Up Procedures for Node Deletion ........112 Adding Nodes that Already have Clusterware and Oracle Software to a Cluster........140 Media Failure protection on RAC database ....................................................103 Administering Services in Real Application Clusters with SRVCTL...120 Deleting Instances from Real Application Clusters Databases ...........Using the DBCA to Add and Modify Services ...............................................101 Administering Services in Real Application Cluster with Enterprise Manager .........................................133 Chapter 9 Managing Backup & Recovery...........................................................................................129 Overview of Configuring RMAN for Real Application Clusters ......106 Chapter 7 Adding and deleting nodes and instances .............................................................................................124 Deleting Nodes from Oracle Clusters on Windows-Based Platforms ............................127 Chapter 8 Recovery Manager Configuration and Archiving ................140 x .....130 Configuring the RMAN Control File Autobackup Feature ........................................................125 ASM Instance Cleanup Procedures after Node Deletion on Windows-Based Platforms .................109 Overview of Node Addition Processes....................................................110 Adding Nodes at the Oracle Clusterware Layer ........139 Objectives .............................104 Everything switches to Services .......................................100 Changing VIP Addresses...........................131 Archived Redo Logs management using RMAN in Real Application Clusters .

..........................................................150 RMAN Backup Scenarios for Real Application Clusters .........154 RMAN Restore Scenarios for Real Application Clusters .162 Performance Statistics on Real Application Clusters ........................................169 Monitoring Performance with Oracle xi .........166 GCS Statistics and Cache Fusion Transfer Impact Analysis ............................................164 RAC Statistics and Events in AWR and Statspack Reports................................166 GCS and GES Statistics Analysis in Performance Monitoring .......................................158 Chapter 10 Performance Monitoring on RAC database ......150 Managing Backup and Recovery on RAC database....Managed Backup Methods ..................................................................................................149 Recovering the Voting Disk ........................................142 RAC and Oracle Recovery Manager (RMAN) .................................143 Automatic Control File Backups .............................................161 Monitoring Real Application Clusters Databases .......................................141 RAC and the Flash Recovery Area...................168 Analyzing Response Times Based on Wait Events.........Archived Log File Configurations..............148 Managing the OCR ........143 User.....................................165 Wait Events in Real Application Clusters .........................156 Parallel Recovery with RMAN..................................164 Automatic Workload Repository in Real Application Clusters Environments..........................154 Cluster File System Backup Scheme..................................................142 Snapshot Control File Management .............145 Parallel Recovery in RAC .........143 Channel Connections to Cluster Instances while using RMAN.........................................148 Recovering the OCR..................................147 RAC and Fast-Start Parallel Rollback ...........................................................................................................................................

......179 Chapter 11 Administrative Options .........................189 Defining Network Interfaces with OIFCFG ........................183 Discover Nodes and Instances Using Enterprise Manager .......172 Real-Time Performance Monitoring .................................................205 Connection Load Balancing..........174 Database Throughput Charts ................................................................................................196 SRVCTL Cluster Database Configuration Tasks ........................................196 Overview of SRVCTL for Administering Real Application Clusters................................................192 Additional Tracing for Real Application Clusters High Availability...................195 Using Instance-Specific Alert Files in Real Application Clusters........................204 Service Deployment Options ...................................................198 Chapter 12 Workload management.............................................................................................................196 Administering RAC using Server Control (srvctl) ............................197 SRVCTL Command Reference .................................................................................................................................................................................178 Cluster Database Instance Performance Page.................191 Using Log Files in Real Application Clusters .......Enterprise Manager.............................................................................................184 Enterprise Manager Pages for Real Application Clusters.........203 Workload Management and Application High Availability.........185 Enterprise Manager Jobs in Real Application Clusters187 Performing Scheduled Maintenance Using Defined Blackouts in Enterprise Manager ....................193 Enabling Additional Debugging Information for Cluster Ready Services ....189 Troubleshooting ..........................208 xii ......189 OIFCFG Command-Line Tool..........188 Administering System and Network Interfaces with OIFCFG (Oracle Interface Configuration) tool ........

........................................................219 Workload Management and Service topologies in Real Application Clusters ....................................221 Service Relocation management by Oracle Clusterware ............................................................................................220 Automatic Workload Repository (AWR) .............................214 Enabling ODP.................................................................................216 Administering Services with the PL/SQL DBMS_SERVICE Package .............................211 Load Balancing Advisory .........212 Enabling Java Database Connectivity Clients to Receive FAN Events ..223 xiii .........210 Managing Unplanned Outages..................215 Administering Services with the Database Configuration Assistant......NET Clients to Receive FAN High Availability Events.................................213 Using FAN with thick JDBC and thin JDBC Clients ..................216 Administering Services with SRVCTL.........................................218 Chapter 13 Design and Development Techniques ...210 Application High Availability with Services & FAN ........................................217 Service Thresholds and Alerts .......220 Recommended Real Application Clusters Service Configurations.....221 Setting Service Levels and Thresholds ...............213 Enabling Oracle Call Interface Clients to Receive FAN High Availability Events......................................214 Administering Services with Enterprise Manager ..................216 Measuring Performance by Service using the Automatic Workload Repository ....................................................222 Glossary .....Fast Application Notification..........................................................................................................................211 Using Fast Application Notification (FAN) Callouts ..........................................214 Administering Services ...

...241 a...... Handy commands.... Misc checks ..................239 1......................................................241 3................242 4................................ Knowledge Search....... Desupport notices & Product Availability..........264 iii.............................243 5......................... RAC Frequently Asked Questions .........240 2.........................Reference .........................................266 xiv .......... Patch and Updates .......... Important logs for investigation by Oracle ........241 b...... Metalink information Search ................. RAC related useful Metalink Notes ........ Starting Up and Shutting Down with srvctl .................262 i.......... Oracle's Certification Matrices............................262 ii...........241 c....................... RAC related useful Weblink References ..............

Chapter 1 Oracle Real Application Cluster Architecture .

The RAC infrastructure is also a key component for implementing the Oracle enterprise grid computing architecture. Combining smaller servers into a cluster can be achieved to create scalable environment that support mission critical business applications RAC uses Oracle Clusterware for the infrastructure to bind multiple servers so that they operate as a single system. The Oracle Clusterware is also a required component for using RAC A cluster comprises of multiple interconnected 1 . Oracle Clusterware is a portable cluster management solution that is integrated with the Oracle database. Oracle Database 10g Real Application Clusters (RAC) enables the clustering of the Oracle Database. The basic principle behind the Real Application Cluster is greater throughput and scalability due to the combined power of multiple instances running on multiple servers RAC provides high availability and scalability for all application types.Oracle Real Application Cluster Field DBA Admin Handbook What is a Cluster ? servers or computers that appear as if they are one single server to end users and applications. Multiple instances access a single physical database prevents the server from being a single point of failure. A RAC database comprises of multiple instances residing on different computers to access a common database residing on shared storage.

each Oracle instance usually runs on a separate server. event management. connection management. Oracle Clusterware is software. storage management. application management. Oracle Clusterware 2.Chapter 1: Oracle Real Application Cluster Architecture In Oracle Database 10g. enables the servers to be bound together to operate as if they were one server. load balancing and availability. The Oracle Clusterware comprises of two clusterware components: a voting disk to record node membership information and the Oracle Cluster Registry (OCR) to record 3 . Oracle Software In RAC10g Clusterware is called CRS layer which resides below Oracle software Layer. Second layer is the Oracle software itself RAC is the Oracle Database option that provides a single system image for multiple servers to access one Oracle database. Oracle Provides you with an integrated software solution that addresses cluster management. In RAC. These capabilities are addressed while hiding the complexity through simple-to-use management tools and automation Oracle Real Application Clusters 10g provides an integrated cluster ware layer that delivers a complete environment for applications RAC uses Oracle Clusterware for the infrastructure to bind multiple servers so that they operate as a single system Main Components of Oracle Real Application Cluster 10g It comprises of two main components 1.

Some of the tools used are Server Control (SRVCTL). The Oracle Clusterware enables you to create a clustered pool of storage to be used by any combination of single-instance and RAC databases.Oracle Real Application Cluster Field DBA Admin Handbook cluster configuration information. They manage what is called the global resources      LMON: Global Enqueue Service Monitor LMD0: Global Enqueue Service Daemon LMSx: Global Cache Service Processes. 4 . In Oracle Clusterware each node is connected to a private network by way of a private interconnect. DBCA and Enterprise Manager. Oracle Clusterware is the only clusterware that you need for most platforms on which RAC operates. Where x can range from 0 to j LCK0: Lock Process DIAG : Diagnosibility Process There are several tools that are used to manage the various resources available on the cluster at a global level. You can also use clusterware from other vendors if the clusterware is certified for RAC. Oracle Clusterware is a portable cluster management solution that is integrated with the Oracle database. The Oracle Clusterware comprises several background processes that facilitate cluster operations such as Cluster Synchronization Service (CSS) and Event Management (EVM) Real Application Cluster Main Processes The main processes involved in Oracle RAC are primarily used to maintain database coherency among each instance.

The RAC infrastructure is also a key component for implementing the Oracle enterprise grid computing architecture. You install CRS. you install the Oracle Database software with RAC components and create a cluster database.Chapter 1: Oracle Real Application Cluster Architecture The combined processing power of the multiple servers can provide greater throughput and scalability than is available from a single server. In the second phase. 5 . The oracle home that you use for the CRS software must be different from the one that is used for the RAC Software.Cluster Software and Oracle software is usually installed on a regular file system that is local to each node. CRS and RAC software is installed on cluster shared storage Note:. RAC enables you to combine smaller commodity servers into a cluster to create scalable environments that support mission critical business applications Storage Principles for RAC Software and CRS The Oracle Software 10g Real Application Clusters installation is a two-phase installation in the first Phase. RAC is the Oracle Database option that provides a single system image for multiple servers to access one Oracle database. This permits online patch upgrades without shutting down the database and also eliminates the software as a single point of failure. In RAC. each Oracle instance runs on a separate server. RAC is a unique technology that provides high availability and scalability for all application types. Having multiple instances access a single database prevents the server from being a single point of failure.

OCR and voting files must be on redundant. Oracle Clusterware components can respond to status changes to restart applications and processes according to defined high availability rules. Its size is around 100MB. The recommended best practice location for those files is raw devices.  The voting file is essentially used by the Cluster Synchronization Services daemon for node monitoring information across the cluster . Virtual Interconnect protocol addresses etc. In 6 .Oracle Real Application Cluster Field DBA Admin Handbook In addition. It maintains information about the high-availability components in your cluster such as the cluster node list. The Oracle Cluster Registry (OCR) file is also a key component of the CRS.CRS layer being the first layer compared to RAC database layer. and CRS application resource profiles such as services. Its size is set around 20 MB. Oracle Clusterware High Availability and the Application Programming Interface Oracle Clusterware provides a high availability application programming interface (API) that you can use to enable Oracle Clusterware to manage applications or processes that run a cluster To maintain high availability. reliable storage such as RAID.  Note :. The file maintained automatically by administrative tools such as SRVCTL. two CRS files must be stored on shared storage. Cluster database instance to node mapping. the voting and OCR files should not be stored in ASM because they must be accessible before starting any oracle instance. If you are using ASM instance to handle the storage.

and all the redo log groups of individual instances must be stored on shared devices for instance or crash recovery purposes. And they must be stored on a file system. If you are not using a CFS. In addition each instance must have it’s own tablespace on shared device using the recommended automatic undo management feature. then it must be stored in a cluster 7 . The purpose of redo groups on shared device is to get access from other instances in case of crash recovery. all data files in RAC must reside on shared devices (either raw or cluster filesystem) . If you are using a cluster file system (CFS). Each undo tablespace must be shared by all instances for recovery purposes. Each instance in the RAC database must have at least two redo log groups. stop. This is the primary difference between RAC storage and storage for singleinstance oracle database.Chapter 1: Oracle Real Application Cluster Architecture addition. If you are using the recommended flash recovery area feature. you can use the Oracle Clusterware high availability framework by registering your applications with Oracle Clusterware and configuring the clusterware to start. or relocate your application processes RAC database Storage Principles In order to be shared by all the instances that access the same database. Archive logs on RAC database should not be on raw device because archive log names are automatically generated and different for each instance. for example by using a network file system (NFS) across nodes. Each instance’s online redo log groups are called an instance’s thread of online redo. you are always forced to make the archives available to the other cluster members at the time of recovery. It enables you to access these archive files from any node at any time.

Traditionally. Note :. 8 . you must store files in either raw volumes without any file system.Oracle Real Application Cluster Field DBA Admin Handbook directory or shared file system so that all instances can access it. Traditional file system do not support simultaneous mounting by more than one system. Next in the evolutionary scale came Network Attached Storage (NAS) that took the storage devices away from the server an connected them directly to the network The choice of file system is critical for RAC deployment. Therefore. Note:. on client server systems. or in a file system that supports concurrent access by multiple systems. data was stored on devices either inside or directly attached to the server. or a cluster File system RAC and Shared Storage Technologies  Supported shared storage for Oracle grids:  Network attached Storage  Storage Area Network  Supported file systems for Oracle grids: Raw Volumes  Cluster file system  ASM Storage Area Network(SAN) represents the evolution of data storage technology to this point.iSCSI is important to SAN technology because it enables a SAN to be deployed in a local area network (LAN).A shared directory can be an ASM disk group.

Raw Partition: A raw partition is a portion of a physical disk that is accessed at the lowest possible level. A raw partition is created when an extended partition is created and logical partitions are assigned to it without any formatting. Real Application Clusters require that each instance be able to access a set of unformatted devices on a shared disk subsystem. and online redo logs. Once formatting is complete. it is called cooked partition SCSI. data files. server parameter files. Archive logs should not be stored on raw devices. iSCSI 9 . Oracle instances in Real Application Clusters use raw devices to update control files. ASM AND CLUSTER FILESYSTEMS WITH REAL APPLICATION CLUSTERS Shared Disk Subsystems for Real Application Clusters On most platforms. Raw devices are used for Real Application Clusters since they enable the sharing of disks. SAN and NAS.Chapter 1: Oracle Real Application Cluster Architecture wide area network (WAN). Raw devices must be precreated using OS specific commands before the database can be created. or Metropolitan area Network (MAN) Definitions of different storage in Real Application Clusters RAW DEVICES. All instances in the cluster share these files. These shared disks are also referred to as raw devices. What is a raw device? Raw Device: A raw device is a disk drive that does not yet have a file system set up.

use of iSCSI with RAC requires either a cluster file system or use of raw volumes. NAS: Network Attached Storage is a special purpose server with its own embedded software that offers cross platform file sharing across the network. Because it operates in block mode. 10 . SAN: Storage Area Network is a shared dedicated high-speed network connecting storage elements and the backend of the servers.although a private network is recommended. iSCSI: Another form of network attached storage that communicates in block mode over Ethernet (Gigabit Ethernet) to special storage subsystems. What is a Cluster File system (CFS) ? A cluster file system (CFS) is a file system that may be accessed (read and write) by all members in a cluster at the same time. iSCSI uses standard hardware and software to communicate . This implies that all members of a cluster have the same view.Oracle Real Application Cluster Field DBA Admin Handbook Although not directly related to CFS and raw devices questions arise around the storage technologies being used. SCSI: Disk drives are connected individually to the host machine by small computer system interfaces (SCSI) through one of a number of disk controllers. Like NFS attached storage.

11 . ASM distributes I/O load across all available resource to optimize performance while removing the need for manual I/O tuning (spreading out the database files avoids hotspots). you can store the files that Real Application Clusters requires directly on the cluster file system.Chapter 1: Oracle Real Application Cluster Architecture If your platform supports an Oracle certified cluster file system. Automatic Storage Management Automatic Storage Management (ASM) is a new feature of Oracle Database 10g which provides a vertical integration of the file system and volume manager specifically built for the Oracle database files.

Oracle Real Application Cluster Field DBA Admin Handbook 12 .

Chapter 2 RAC Architecture and Concepts .

You see few additional background processes associated with RAC instance.Global Enqueue Service Monitor LMD 0:. They manage what is called the global resources:     LMON:. load balancing and availability. These capabilities are addressed while hiding the complexity through simple-to-use management tools and automation. several platform-specific processes or services will also be running on each node in the cluster to support Oracle Clusterware Oracle Clusterware Processes on UNIX-Based Systems  crsd—Performs high availability recovery and management operations such as maintaining the OCR and managing application resources. connection management.Global Enqueue Service Daemon LMSx :. or by a user in the admin group on Mac OS X13 . application management.Diagnosibility Process The Oracle Clusterware Software Components When Oracle Clusterware operates. This process runs as the root user.Global Cache Service Processes LCK 0 :. event management. RAC Software Principles Addition to the background processes found in single instance database. These processes are primarily used to maintain database coherency among each instance.Oracle Real Application Cluster Field DBA Admin Handbook Oracle Real Application Cluster 10g provides you with an integrated software solution that addresses cluster management.Lock Process DIAG :. storage management.

.

oprocd—Cluster Process monitor.    Various global resources on the cluster such as RAC databases. This process restarts automatically upon failure. Instances. This process only appears on platforms that do not use vendor clusterware with Oracle Clusterware Oracle Clusterware Services on Windows-Based Systems Clusterware processes on Windows are called Clusterware services and they are  OracleCRService— Maintaining application resources and OCR is the function of this service. OracleCSService—Manages cluster node membership and runs as oracle user. CRS node Application and 15 . ocssd—This process runs as oracle user and manages cluster node membership. OraFenceService—Process monitor for the cluster. This process runs as the root user. This process restarts automatically upon failure. failure of this process results in cluster restart. or by a user in the admin group on Mac OS X-based systems.Oracle Real Application Cluster Field DBA Admin Handbook     based systems. This process also starts the racgevt process to manage FAN server callouts. OracleEVMService—This is the Event manager daemon. Services. Apart from this it performs high availability recovery. This process also starts the racgevt process to manage FAN server callouts. Note that this process only appears on platforms that do not use vendor clusterware with Oracle Clusterware. evmd—Event manager daemon. Any failure of this process results in cluster restart.

Virtual Interconnect Protocol addresses. This file is maintained automatically by administrative tools such as SRVCTL. However Voting disk and OCR files must be stored on your shared storage. It’s size is set to around 20MB The Oracle cluster registry (OCR) file is also a key component of the CRS. In the second phase. It maintains information about the high-availability components in your cluster such as the cluster node list.  The voting file is essentially used by the Cluster Synchronization service daemon for node monitoring information across the cluster. The Oracle home that you use for the CRS software must be different from the one that is used for the RAC Software. Local installation permits online node patch upgrades and eliminates the software as a single point of failure.Chapter 2: RAC Architecture and Concepts also optional Automatic Storage Management (ASM) can be managed by tools Server Control (SRVCTL). software is usually installed on a regular file system that is local to each node. Though shared storage is an accepted option to install CRS and RAC software. RAC Software Storage Principle The oracle database 10g Real Application Clusters installation is a two-phase installation. and CRS application resource profiles such as services. you install the oracle database software with RAC components and create a cluster database. Its size is around 100MB  Oracle Database 10g provides Automatic Storage Management (ASM) and oracle recommends to use this to 16 . cluster database instance to node mapping. In the first phase. you install CRS. DBCA and Enterprise Manager.

2. The voting disk must reside on shared disk. then you should use external mirroring to provide redundancy. which Oracle recommends and it’s oracle product Oracle Cluster File System (OCFS).Oracle Real Application Cluster Field DBA Admin Handbook manage your disk.If you define a single voting disk. Oracle Database software manages disk access and the Oracle software is certified for use on a variety of storage architectures Storage options for RAC     ASM. RAC uses the voting disk to determine which instances are members of a cluster. Oracle recommends that you have multiple voting disks.Manages cluster membership by way of a health check and arbitrates cluster ownership among the instances in case of network failures. For high availability. However you may also use a cluster-aware volume manager. Each node in the cluster 17 . OCR relies on a distributed shared-cache architecture for optimizing queries against the cluster repository. which is available for Linux and Windows platforms. OCR File :. or a third-party cluster file system that is certified for RAC A network file system Raw devices The Oracle Clusterware Voting Disk and Oracle Cluster Registry The Oracle Clusterware has following two critical files: 1. Voting Disk: . The Oracle Clusterware enables multiple voting disks Note:.Cluster configuration information is maintained in Oracle Cluster Registry file.

loc Oracle Cluster Registry (OCR) :. services and node applications. SRVCTL.Maintains cluster configuration information as well as configuration information about any cluster database within the cluster.The name of the configuration file is ocr. they communicate through their local OCR process to the OCR process that is performing input/output (I/O) for writing to the repository on disk. or the Database Configuration Assistant (DBCA). The OCR client applications are Oracle Universal Installer (OUI). When OCR client application needs to update the OCR. The OCR also manages information about processes that Oracle Clusterware controls. 18 . Database Upgrade Assistant(DBUA). Database Configuration Assistant (DBCA).You can replace a failed OCR online.loc and the configuration file variable is ocrconfig. The OCR stores configuration information in a series of key-value pairs within a directory tree structure. NetCA and Virtual Internet Protocol Configuration assistant (VIPCA). The Oracle Clusterware can multiplex the OCR and Oracle recommends that you use this feature to ensure cluster high availability Note:. specifically databases. The OCR must reside on shared disk that is accessible by all of the nodes in your cluster. OCR also maintains dependency and status information for application resources defined within CRS.Chapter 2: RAC Architecture and Concepts maintains an in-memory copy of OCR. Note:. instances. and you can update the OCR through supported APIs such as Enterprise Manager. along with an OCR process that accesses its OCR cache. Enterprise Manger (EM). the Server Control Utility (SRVCTL).

Traditionally. data was stored on devices either inside or directly attached to the server. Next in the evolution scale came Network Attached Storage (NAS) that took the storage devices away from the server and connected them directly to the network. Oracle Cluster File systems Oracle cluster file system (OCFS) is a shared file system designed specially for oracle Real Application Cluster Automatic Storage Management (ASM)  Automatic and high-performance cluster file system 19 . In RAC deployment choosing the appropriate file system is critical. or on a file system that supports concurrent access by multiple systems. Supported shared storage for Oracle grids   Network attached Storage Storage area Network 2. on client server systems.Oracle Real Application Cluster Field DBA Admin Handbook Shared storage Technologies and RAC 1. Supported file systems for Oracle grids    Raw Volumes Cluster file system ASM Storage Area Network (SAN) represents the evolution of data storage technology. you must store files in either raw volumes without any file system. Because traditional file systems do not support simultaneous mounting by more than one system.

ASM is the oracle’s strategic and stated direction as to where oracle database files should be stored. Note:. 4. 3. Simple management Use of OMF with RAC Single Oracle Software installation Autoextend 20 . ASM facilitates management of dynamic database environment by allowing DBA’s to increase the database size without having to shut down the database to adjust the storage allocations. 2. However OCFS will continue to be developed and supported for those who are using it. It provides a vertical integration of the file system and the volume manager that is specially built for Oracle Database files The ASM distributes I/O load across all available resourses to optimize performance while removing the need for manual I/O tuning.Chapter 2: RAC Architecture and Concepts     Manages Oracle Database files Data spread across disks to balance load Integrated mirroring across disks Solves many storage management Challenges The automatic storage management(ASM) is a new feature in oracle Database 10g. The ASM can maintain redundant copies of data to provide fault tolerance. Comparison between RAW or CFS  Using CFS 1.

Performance 2.Oracle Real Application Cluster Field DBA Admin Handbook  Using raw 1. User Datagram Protocol (UDP) on Gigabit Ethernet is used on all Unix platform as one of the primary protocols and 21 . Use when CFS not available 3. Cannot be used for archive log files You can use a cluster file system or place files on raw devices. Cluster file systems provide the following advantages:     Greatly simplify the installation and administration of RAC Use of Oracle Managed Files with RAC Single Oracle Software installation Autoextend enabled on oracle data files Uniform accessibility to archive logs in case of physical node failure RAW devices Implications    Raw devices are always used when CFS is not available or not supported by oracle Raw devices offer best performance without any intermediate layer between oracle and the disk Autoextend fails on raw devices if the space is exhausted Cluster Stack with RAC Interconnect software protocol supported by oracle on each node in a cluster is a part and parcel of Cluster stack and this supports inter-instance communications and Transmission control/Internet Protocol (TCP/IP) to support CRS polling.

RAC uses a Global 22 . Any Interconnect used in cluster stack must be certified by oracle for your platform Storage solution you choose on RAC also must be certified by oracle similar to the interconnect for your platform. Global Resources Coordination Cluster operations require synchronization among all instances to control shared access to resources. Locking prevents two processes from changing the same resource (or row) at the same time. If a CFS is unavailable on the target platform. Coordination between processes on different nodes is important and this is maintained by Internode synchronization. Internode synchronization guarantees that each instance sees the most recent version of a block in its buffer cache. This coordination prevents from changing the same resources at the same time. then the database area can be created either on ASM or on raw devices (with the required volume manager) and the flash recovery area must be created on the ASM Necessity of Global Resources In single-Instance environment locking coordinates access to a common resource such as a row in a table. then both the database area and flash recovery can be created on either CFS or ASM.Chapter 2: RAC Architecture and Concepts interconnect for RAC inter-instance IPC communication. In RAC environment. If a cluster file system (CFS) is available on the target platform. There are other supported vendor-specific interconnect protocols available and some of them are Remote Shared Memory for SCI and SunFire Link interconnects and Hyper Messaging Protocol for Hyper fabric interconnects.

by the first instance. Cache coherency is the technique of keeping multiple copies of a block consistent between different oracle instances. The Global Cache Service (GCS) and Global Enqueue Service (GES) manage the information in the GRD Each instance maintains a part of the GRD in its System Global Area (SGA). This dirty image of the block is also called a past image of the block. 3. or dirtied. and the content of the block is represented by its SCN. Any given time only one copy of the block exists cluster wide. the first instance is the holder. The second instance in the cluster attempting to modify the block submits a request to the GCS.Oracle Real Application Cluster Field DBA Admin Handbook Resource Directory (GRD) to record information about how resources are used within a cluster database. 1. The first instance retains the dirty buffer for recovery purposes. GCS implements cache coherency by using what is called the Cache Fusion algorithm Global Cache Coordination in RAC10g Here is a scenario where data block has been changed. The GCS and GES nominate one instance to manage all information about a particular resource. Also each instance knows which instance masters which resource. 4. the second instance informs the GCS that it holds the block. This instance is called the resource master. The GCS transmits the request to the holder of the Block. Maintaining Cache coherency is an important part of RAC activity. On receipt of the block. In this case. A past image block cannot be modified further. 2. The first instance receives the request message and sends the block to the second instance. 23 .

the holder of the current version of the block. A write request for a data block can originate in any instance that has the current or past image of the block. 4. the GCS orders all past image holders to discard their past images. After receipt of the notification. GCS Coordination in RAC10g to Write data block to Disk: Following example focuses on the scenario how oracle perform a checkpoint at any time or replace buffers in the cache due to free buffer requests. The first instance sends a write request to the GCS. At any given time multiple versions of the same data block with different changes can exist in the caches of instances in the cluster. The second instance records the completion of the write operation with the GCS 5. The second instance receives the write request and writes the block to disk.Chapter 2: RAC Architecture and Concepts Note: The data block is not written to disk before the resource is granted to the second instance. In this scenario. 3. GCS also must ensure that all previous versions are purged from the other caches. A write protocol managed by the GCS ensures that only the most current version of the data is written to disk. assume that the first instance holding a past image of buffer requests that Oracle writes the buffer to the disk: 1. 2. The GCS forwards the request to the second instance. These past images are no longer needed for recovery. 24 .

25 . RAC and Instance/Crash Recovery When an instance fails and the failure is detected by another instance. nevertheless. only one I/O is performed to write the most current version of the block to disk. SMON determines the set of blocks that need recovery. the second instance performs the recovery on failed instance OR any of the surviving instance can do the Instance crash recovery on crashed Instance.  Additional Memory Requirement for RAC compared to Single Instance databases Heuristics for scalability cases:   15% more shared pool 10% more buffer cache To find out current values and resource limit. and at the same time GCS resources are remastered. transactions can continue to modify data blocks as long as these transactions have already acquired the necessary resources.   During the first phase of recovery. all GCS resource requests and write requests are temporarily suspended and GCS resource requests and resources lose their masters. GES remasters the enqueues followed by GCS remasters it resources. During this time.Oracle Real Application Cluster Field DBA Admin Handbook Note: In this case. This set is called the recovery set. One of the surviving instances can grab the Instance Recovery enqueue after enqueues are reconfigured.

Because blocks may be cached across instances. current_utilization. Additional Memory Requirement for RAC RAC specific memory is mostly allocated in the shared pool at SGA creation time. there is a corresponding GV$ view except for a few exceptions. Actual resource usage can be monitored by querying the CURRENT_UTILIZATION and MAX_UTILIZATION columns for the GCS and GES entries in the V$RESOURCE_LIMIT view of each instance. about 10% more buffer cache and 15% more shared pool needs to be increased to run RAC Instances. Nevertheless these values are mostly upper bounds. When migrating your oracle database from single instance to RAC. In contrast. considering the workload requirements per instance the same as with the single-instance environment. standard dynamic performance views store information about the local instance only. you must also account for bigger buffer caches. RAC and Global Dynamic Performance Views Global dynamic performance views store information about all shared instances accessing one RAC database. max_utilization from v$resource_limit where resource_name like 'g%s_%'. The INST_ID column 26 . each GV$ view possesses an additional column named INST_ID. However values are heuristics and based on RAC sizing experience. In addition to the V$ information. For each of the V$ views available in RAC instance.Chapter 2: RAC Architecture and Concepts SQL>select resource_name.

then you do not get a result either. the value of the PARALLEL_MAX_SERVERS initialization parameter must be set to at least 1 on each instance. In the two cases above. if all the parallel servers are busy on a particular node. The parallel execution coordinator is running on the instance that the client connects to. then you do not get a result from that node. and one slave is allocated in each instance to query the underlying V$ view for that instance. Also. Services enable you to group database workloads and route the work to the optimal instances that are assigned to process the service. Applications that you assign to services transparently acquire the defined workload management characteristics.In order to query the GV$ views.If you have 5 instance RAC database. Many Oracle database features are integrated with services. such as Resource Manager. You can query GV$ views from any started instance. including high availability and load balancing rules. For example :. you would see 5 values under INST_ID column in G$ view. Note:. This is because GV$ views use a special form of parallel execution. Services are entities that you can define in RAC databases. RAC and Services Services are a logical abstraction for managing workloads Services—Oracle Database 10g introduced a powerful automatic workload management facility called services to enable the enterprise grid vision. If PARALLEL_MAX_SERVERS is set to 0 on a particular node. Services can be used to define the resources that Oracle assigns to process workloads and to monitor workload resources.Oracle Real Application Cluster Field DBA Admin Handbook displays the instance number from which the associated V$ view information is obtained. you do not get a warning or an error message. which enables you to restrict the resources that a 27 .

Services enable the automatic recovery of work. nevertheless you can still manage the instances in RAC environment without Enterprise Manager(EM). Enterprise Manager (EM) grid Control or the Database Control is the recommended management tool for the cluster as well as the database. and priorities. Services are built into the oracle database providing single system image for workloads. Virtual IP Addresses and RAC Virtual IP addresses (VIP) are all about availability of applications when an entire node fails. prioritisations for workloads. and the Oracle Scheduler. Some database features are also integrated with Oracle Streams. These attributes are handled by each instance in the cluster by using metrics. by using other tools or simply querying the data dictionary views. The number of instances offering the service is transparent to the application. Advanced Queuing. service level thresholds. performance measures for real transactions. to achieve queue location transparency. alerts. Each service represents a workload with common attributes. scheduler job classes. and alerts and actions when performance goals are violated. 28 . A Service can span one or more instances of an oracle database in a cluster.Chapter 2: RAC Architecture and Concepts service can use within an instance. and resource manager. Database Control and RAC With Real Application Clusters 10g. to map services to specific job classes. and a single instance can support multiple services.

RAC is supported in both modes. Alternatively. Grid Control can be used to manage multiple databases.various instances in database and Application server etc in your enterprise across different ORACLE_HOME directories. Servers. and other target types such as listeners. cluster ware . Database Control is configured within the same ORACLE_HOME of your database target and can be used to manage only one database at a time. 29 .Oracle Real Application Cluster Field DBA Admin Handbook EM has two different management frameworks: Grid control and Database Control.

Chapter 2: RAC Architecture and Concepts 30 .

Chapter 3 Oracle Clusterware administration .

Note:.You can dynamically add voting disks after you complete the Oracle clusterware and RAC installation processes. use the following procedures to regularly backup your voting disks and to recover them as needed. Perform this operation on every voting disk as needed where voting_disk_name is the name of the active voting disk and backup_file_name is the name of the file to which you want to back up the voting disk contents: dd if=voting_disk_name of=backup_file_name Recovering Voting Disks Run the following command to recover a voting disk where backup_file_name is the name of the voting disk backup file and voting_disk_name is the name of the active voting disk: 32 .Oracle Real Application Cluster Field DBA Admin Handbook Administering Oracle Clusterware Components The Oracle Clusterware (CRS) includes two important components: The voting disk and the Oracle Cluster Registry (OCR). The voting disk is a file that manages information about node membership and the OCR is a file that manages the RAC database configuration information. Voting Disks administration in Real Application Clusters Oracle recommends that you select the option to configure multiple voting disks during Oracle Clusterware installation for better availability. After installation. Backing up Voting Disks Following command can be run to back up a voting disk.

Chapter 3: Oracle Clusterware administration

dd if=backup_file_name of=voting_disk_name Note: If you have multiple voting disks, then you can remove the voting disks and add them back into your environment using crsctl utility Changing the Voting Disk Configuration after Installing Real Application Clusters You can dynamically add and remove voting disks after installing Real Application Clusters. Do this using the following commands where path is the fully qualified path for the additional voting disk. Run the following command as the root user to add a voting disk: $crsctl add css votedisk path Run the following command as the root user to remove a voting disk: $crsctl delete css votedisk path Note :- If your cluster is down, then you can use -force option to modify the voting disk configuration with either of these commands without interacting with active Oracle Clusterware daemons. However, using the -force option while any cluster node is active may corrupt your configuration. Oracle Cluster Registry Application Clusters administration in Real

The Oracle installation process for RAC gives you the option of automatically mirroring the OCR. This creates a second OCR to duplicate the original OCR. You can put the mirrored OCR on an Oracle cluster file system disk, on a shared raw device, or on a shared raw logical volume. 33

Oracle Real Application Cluster Field DBA Admin Handbook

Adding an Oracle Cluster Registry OCR can be added to an OCR location after completing a RAC installation or upgrade. If you already mirrored the OCR, then you do not need to add an OCR location; Note :- RAC environments do not support more than two OCRs, a primary OCR and a second OCR. Replacing an Oracle Cluster Registry Mirrored OCR can be replaced provided that OCR that you are not going to replace is online, and Oracle Clusterware is running on the node on which e you are going to perform the replace operation. Note:- The OCR that you are replacing can be either online or offline. In addition, if your OCR resides on a cluster file system file or if the OCR is on an network file system, then create the target OCR file before continuing with this procedure. Following command is run to replace the OCR using either destination_file or disk to indicate the target OCR: $ocrconfig -replace ocr destination_file or disk To replace an OCR mirror location using destination_file or disk to indicate the target OCR: $ocrconfig -replace ocrmirror destination_file or disk Repairing an Oracle Cluster Registry Configuration on a Local Node either

34

Chapter 3: Oracle Clusterware administration

You may need to repair an OCR configuration on a particular node if your OCR configuration changes while that node is stopped. For example, you may need to repair the OCR on a node that was not up while you were adding, replacing, or removing an OCR. To repair an OCR configuration, run the following command on the node on which you have stopped the Oracle Clusterware daemon: $ocrconfig –repair ocrmirror device_name Note:- This operation only changes the OCR configuration on the node from which you run this command. For example, if the OCR mirror device name is /dev/raw1, then use the command syntax ocrconfig -repair ocrmirror /dev/raw1 on this node to repair its OCR configuration. Removing an Oracle Cluster Registry You may want to remove an OCR location reduce OCRrelated overhead or to stop mirroring your OCR or you may have moved your OCR to redundant storage such as RAID. To remove an OCR location, at least one other OCR must be online. Caution: Do not perform this OCR removal procedure unless there is at least one other active OCR online. Following command can be run on any node in the cluster to remove the OCR ocrconfig -replace ocr Run the following command on any node in the cluster to remove the mirrored OCR

35

If you remove a primary OCR. 36 . You can use any backup software to copy the automatically generated backup files at least once daily to a different device from where the primary OCR resides. The default location for generating backups on UNIX-based systems is CRS_home/cdata/cluster_name where cluster_name is the name of your cluster. The CRSD process that creates the backups also creates and retains an OCR backup for each full day and at the end of each week. The Windows-based default location for generating backups uses the same path structure. The first method uses automatically generated OCR file copies and the second method uses manually created OCR export files. At any one time. Managing Backups and Recovery of the OCR Using OCR Backup Files Two methods are most popular for copying OCR content and using it for recovery.Oracle Real Application Cluster Field DBA Admin Handbook ocrconfig -replace ocrmirror These commands update the OCR configuration on all of the nodes on which Oracle Clusterware is running. then the mirrored OCR becomes the primary OCR. The Oracle Clusterware automatically creates OCR backups every four hours. Oracle always retains the last three backup copies of the OCR. Note:. Note: When removing an OCR location.You cannot customize the backup frequencies or the number of files that Oracle retains. the remaining OCR must be online.

Chapter 3: Oracle Clusterware administration Note:. 2. as a definitive verification that the OCR failed on both the primary OCR and the OCR mirror. Note:. Perform the restore by applying an OCR backup file that you identified in step 1 using the following command where file_name is the name of the OCR that you want to restore.Make sure that the OCR devices that you specify in the OCR configuration exist and that these OCR devices are valid before running this command. 3. ocrconfig -restore file_name 37 . It would normally return failure message if there is a issue or corruption in OCR. Stop Oracle Clusterware on all of the nodes in your RAC database by executing the init.You must be root user to run ocrconfig commands. Review the contents of the backup using ocrdump -backupfile file_name where file_name is the name of the backup file. Attempt to correct the problem using one of the following platform-specific OCR diagnostic procedures. Restoring the Oracle Cluster Registry from Automatically Generated OCR Backups If an application fails. run an ocrcheck. If diagnosis doesn’t help you may have to restore OCR from backup Restoring the Oracle Cluster Registry on UNIX-Based Systems 1. Identify the OCR backups using the ocrconfig showbackup command. Then before attempting to restore the OCR. try to restart the application and if you think the error is related to OCR.crs stop command on all of the nodes.

Restart Oracle Clusterware on all of the nodes in your cluster by restarting each node or by running the init. 4. 3. OracleCRService. 38 . Run the following command to verify the OCR integrity where the -n all argument retrieves a listing of all of the cluster nodes that are configured as part of your cluster cluvfy comp ocr -n all [-verbose] Restoring the Oracle Cluster Registry on Windows-Based Systems 1. Review the contents of the backup using ocrdump backupfile file_name where file_name is the name of the backup file. Make sure that the OCR devices that you specify in the OCR configuration exist and that these OCR devices are valid. disable the following OCR clients and stop them using the Service Control Panel: OracleClusterVolumeService.Oracle Real Application Cluster Field DBA Admin Handbook 4. Execute the restore by applying an OCR backup file that you identified in Step 1 with the ocrconfig -restore file name command. Start all of the services that were stopped in step2. OracleCSService. 5. On all of the remaining nodes. Use the following procedure to restore the OCR on Windows-based systems: Identify the OCR backups using the ocrconfig -showbackup command.crs start command. Restart all of the nodes and resume operations in cluster mode. and the OracleEVMService. 2.

ocrdump [file_name|-stdout] [-backupfile backup_file_name] [-keyname keyname] [-xml] [noheader] Table 3-1 Options Description Name of a file to which you want OCRDUMP to write output. Run the following command to verify the OCR integrity where the -n all argument retrieves a listing of all of the cluster nodes that are configured as part of your cluster cluvfy comp ocr -n all [-verbose] Diagnosing OCR Problems with the OCRDUMP and OCRCHECK Utilities OCRDUMP and OCRCHECK utilities are used to diagnose OCR problems as described under the following topics: Using the OCRDUMP Utility Use the OCRDUMP utility to write the OCR contents to a file so that you can examine the OCR content. Does not print the time at which you ran file_name -stdout -keyname -xml -noheader 39 . The predefined output location that you can redirect with.Chapter 3: Oracle Clusterware administration 5. a filename. The name of an OCR key whose subtree is to be dumped. for example. Writes the output in XML format.

OCRID. The following is a sample of the OCRCHECK output: Status of Oracle Cluster Registry is as follows : Version : 2 Total space (kbytes) : 262144 Used space (kbytes) : 16256 Available space (kbytes) : 245888 ID : 1918913332 Device/File Name : /dev/raw/raw1 Device/File integrity check succeeded Device/File Name : /oradata/mirror. You can query the backups using the ocrconfig -showbackup command. The OCRCHECK utility displays the version of the OCR's block format. of which you want to view.ocr Device/File integrity check succeeded Cluster registry integrity check succeeded 40 . backup_file_name The name of the backup file the content OCRCHECK Utility to verify the OCR Integrity OCRCHECK utility is used to verify the OCR integrity. total space available and used space. and the OCR locations that you have configured. -backupfile Option to identify a backup file. It also returns an individual status for each file as well as result for the overall OCR integrity check. OCRCHECK performs a block-by-block checksum operation for all of the blocks in all of the OCRs that you have configured.Oracle Real Application Cluster Field DBA Admin Handbook Options Description the command and when the OCR configuration occurred.

when the issue is confined only to local node. However. you can use other nodes to start your cluster database.ini Oracle Cluster Registry Data Loss Protection Mechanism The OCR has a mechanism that prevents data loss due to accidental overwrites. If you configure a mirrored OCR and if the OCR cannot access the two mirrored OCR locations and also cannot verify that the available OCR contains the most recent configuration. However. the Oracle Clusterware alert log. thus enabling you to use the updated OCR file to start your cluster. or both. Oracle displays an alert message in Enterprise Manager. Oracle displays an alert message in either Enterprise Manager. the Oracle Clusterware alert log files.In such cases. In such cases. Sometimes this problem is local to only one node. this can result in the loss of data that was not available at the time that the previous known good state was created. or both. then you can override the protection mechanism. In situation like this data loss is protected by prohibiting the node from re-start Note:.Overriding the OCR using this procedure can result in the loss of OCR updates that were made between the time of 41 . To change amount of logging. edit the file CRS_Home/srvm/admin/ocrlog. then the OCR prevents further modification to the available OCR. Note:.Chapter 3: Oracle Clusterware administration OCRCHECK creates a log file in the directory CRS_Home/log/hostname/client. if you are unable to start any cluster nodes in your environment and if you cannot repair the OCR. This procedure enables you to start the cluster using the available OCR.

loc on Unix-based systems and the Registry on Windows-based systems) with other nodes on which Oracle Clusterware is running. Do this by comparing the node's OCR configuration (ocr. If the configurations match. 6. In other words. 3. 1.Oracle Real Application Cluster Field DBA Admin Handbook the last known good OCR update made to the currentlyaccessible OCR and the time at which you performed this procedure. then run ocrconfig -repair. Administering the Oracle Cluster Registry with OCR Exports 42 . then run the command ocrconfig -overwrite to bring up the node. Do this by taking output from the ocrdump command and determine whether it has your latest updates. running the ocrconfig -overwrite command can result in data loss if the OCR that you are using to perform the overwrite does not contain the latest configuration updates for your cluster environment. Ensure that the most recent OCR contains the latest OCR updates. 2. 4. 5. If you cannot resolve the CLSD message. Attempt to resolve the cause of the a CLSD-1009 or CLSD-1011 message. If the configurations do not match. How to overwrite the OCR if a node cannot start up and if the alert log contains CLSD-1009 or CLSD-1011 message. then ensure that the node can access all of the configured OCRs by running an ls command on Unix-based systems or a dir command on Windows-based systems. Oracle issues a warning when one of the configured OCR locations is not available or if the configuration is incorrect.

then restore your configuration using one of the following platform-specific procedures: Note:. Some of these changes are often not restored when you restore the OCR.Chapter 3: Oracle Clusterware administration In addition to using the automatically created OCR backup files. if you have unresolvable configuration problems. This may result in an OCR that has contents that do not match the state of the rest of your system. modifying Oracle Clusterware resources. or creating a database. or if you are unable to restart your clusterware after such changes. configuration changes also cause file and database object creation. For example. Importing Oracle Cluster Registry Content on UNIXBased Systems Use the following procedure to import the OCR on UNIXbased systems: 1. This exports the OCR content to a file format.Most configuration changes that you make not only change the OCR contents. Do not perform an OCR restore as a correction to revert to previous configurations if some of these configuration changes should fail. To restore the OCR use the -import option if your configuration changes cause errors. such as adding or deleting nodes from your environment. Identify the OCR export file that you want to import by identifying the OCR export file that you previously created using the ocrconfig -export file_name command. Do this by using the ocrconfig -export command. 43 . you should also export the OCR contents before and after making significant configuration changes.

OracleEVMService. Restart all of the affected services on all nodes. 3. Perform the import by applying an OCR export file that you identified in Step 1 using the following command where file_name is the name of the OCR file from which you want to import OCR information: ocrconfig -import file_name 4. OracleCSService. 5.Oracle Real Application Cluster Field DBA Admin Handbook 2. 3. 2. 4. OracleCMService. and the OracleCRService. Identify the OCR export file that you want to import by running the ocrconfig -showbackup command. Import an OCR export file using the ocrconfig import command from one node. Run the following Cluster Verification Utility (CVU) command to verify the OCR integrity where the -n all argument retrieves a listing of all of the cluster nodes that are configured as part of your cluster cluvfy comp ocr -n all [-verbose] Importing Oracle Cluster Registry Content on WindowsBased Systems Use the following procedure to import the OCR on Windowsbased systems: 1. Stop the following OCR clients on each node in your RAC environment using the Service Control Panel: OracleClusterVolumeService.crs stop command on all of the nodes. Stop Oracle Clusterware on all of the nodes in your RAC database by executing the init. 44 . Restart Oracle Clusterware on all of the nodes in your cluster by restarting each node.

The CLUSTER_INTERCONNECTS initialization parameter is useful only in a UNIX-based environments where UDP IPC is enabled. It enables you to specify multiple IP addresses. To downgrade. separated by colons. then you can use the cluvfy command to verify the integrity of the OCR. Run the following Cluster Verification Utility (CVU) command to verify the OCR integrity where node_list is a list of all of the nodes in your cluster database: cluvfy comp ocr -n all [-verbose] Upgrading and Downgrading the OCR Configuration in Real Application Clusters When you install Oracle Clusterware. If you are upgrading the OCR to Oracle Database 10g release 10. you cannot use the Cluster Verification Utility (CVU) commands to verify the OCR for pre-10.2 release formats. The CLUSTER_INTERCONNECTS initialization parameter requires the IP address of the interconnect instead of the device name. you can use the CLUSTER_INTERCONNECTS initialization parameter to specify an alternative interconnect for the private network.2. 45 . RAC network traffic is distributed between the specified IP addresses. Administering Multiple Cluster Interconnects on UINIXBased Platforms On Unix based platforms. Oracle automatically runs the ocrconfig -upgrade command. If you are downgrading. follow the downgrade instructions for each component and also downgrade the OCR using the ocrconfig -downgrade command.Chapter 3: Oracle Clusterware administration 5.

212 Use the ifconfig or netstat command to display the IP address of a device. GES.S IMPLEX> inet 129.34.RUNNING.NOCH ECKSUM> inet 127.NOCHEC KSUM. run the following command as the root user: # /usr/sbin/ifconfig -a fta0: flags=c63<UP. For example.139.BROADCAST.LOOPBACK.137.1 netmask ff000000 ipmtu 4096 ics0: flags=1100063<UP.137.SIMPLEX.NOTRAILERS.0. and IPQ IPC traffic.MULTICAST.212 for all GCS.Oracle Real Application Cluster Field DBA Admin Handbook Overall cluster stability and performance may improve when you force Oracle over a different interconnect by setting the CLUSTER_INTERCONNECTS initialization parameter.1 netmask ffffff00 broadcast 10.0.137.NOTRAILERS.NOARP. This command provides a map between device names and IP addresses.34.BROADCAST.34.0.0. set the CLUSTER_INTERCONNECTS parameter as follows: CLUSTER_INTERCONNECTS=129.MULTICAST.255 ipmtu 1500 lo0: flags=100c89<UP.0. to determine the IP address of a device.RUNNING.CLUIF> inet 10.34.255 ipmtu 7000 sl0: flags=10<POINTOPOINT> tun0: flags=80<NOARP> 46 .0. to use the network interface whose IP address is 129.212 netmask fffffc00 broadcast 129. For example.

and ics0: devices in that order.212 and the interface ics0: has an IP address of 10. then list them in the same order for all instances of the same database.0. If the interconnect IP address specified is incorrect or does not exist on the system. For example.Chapter 3: Oracle Clusterware administration In the preceding example. then inter-node traffic may pass through bridges and routers or there may not be a path between the two nodes at all. fta0:.1.34. then Oracle Database uses the default cluster interconnect device. if the parameter for instance 1 on node 1 lists the IP addresses of the alt0:. If you specify multiple IP addresses for this parameter.137.    47 . then the parameter for instance 2 on node 2 must list the IP addresses of the equivalent network adapters in the same order.0. If you do not follow this rule. the interface fta0: has an IP address of 129. Specify the CLUSTER_INTERCONNECTS initialization parameter in the parameter file. setting a different value for each database instance. Bear in mind the following important points when using the CLUSTER_INTERCONNECTS initialization parameter:  The IP addresses specified for the different instances of the same database on different nodes must belong to network adapters that connect to the same network.

Oracle Real Application Cluster Field DBA Admin Handbook 48 .

Sign up to vote on this title
UsefulNot useful