Configuring HP Serviceguard Toolkit for Oracle Data Guard

Technical white paper

Table of contents
Introduction ......................................................................................................................................... 2 Terms and definitions ........................................................................................................................... 2 Oracle Data Guard overview ................................................................................................................ 2 Physical and logical standby databases .............................................................................................. 2 Active Data Guard ........................................................................................................................... 3 Data Guard Broker........................................................................................................................... 3 Role transitions ................................................................................................................................ 3 Serviceguard support for Oracle Data Guard.......................................................................................... 3 Dependencies .................................................................................................................................. 4 Supported configurations .................................................................................................................. 4 Continentalclusters environment ......................................................................................................... 6 Metrocluster and EDC configurations .................................................................................................. 8 Metrocluster in Continentalclusters environment.................................................................................. 10 Supporting multiple instances of single-instance ODG configuration ..................................................... 11 HA for Data Guard Broker .............................................................................................................. 11 Installation and configuration of the toolkit ............................................................................................ 12 Setting up the application ............................................................................................................... 12 Setting up the toolkit ....................................................................................................................... 12 ODG package configuration example .............................................................................................. 14 Adding the package to the cluster .................................................................................................... 19 ODG maintenance ......................................................................................................................... 19 Troubleshooting ................................................................................................................................. 20 Known problems and workarounds ...................................................................................................... 21 For more information .......................................................................................................................... 22 Appendix A ...................................................................................................................................... 22 Configuring ODG example ............................................................................................................. 22 Preparing the primary database....................................................................................................... 22 Creating a physical standby database.............................................................................................. 23

Introduction
This document describes how the HP Serviceguard Toolkit for Oracle Data Guard assists in easy integration of Oracle Data Guard (ODG) with HP Serviceguard. ODG is the host-based datareplication software for Oracle Database. It provides management, monitoring, and automation features to create and maintain one or more standby databases to protect data from failures, disasters, human error, and data corruptions. To provide High Availability (HA) so that data replication continues in the face of failures, ODG can be deployed in a Serviceguard cluster.

Terms and definitions
Term ASM ECMT EDC LVM MNP ODG RAC Definition Automatic Storage Management Enterprise Cluster Master Toolkit Extended Distance Cluster Logical Volume Manager Multi-node Package Oracle Data Guard Real Application Clusters

Oracle Data Guard overview
Data Guard provides a comprehensive set of services that create, maintain, manage, and monitor one or more standby databases to enable production Oracle databases to survive disasters and data corruption. Data Guard maintains these standby databases as transactionally consistent copies of the production database. If the production database is unavailable in case of a planned or unplanned outage, then Data Guard can switch any standby database to take over in the production role, thus reducing the downtime associated with the outage. Data Guard can be used with traditional backup, restoration, and cluster techniques to provide a high level of data protection and data availability. A Data Guard configuration consists of one production database, known as the primary database, and up to nine standby databases.

Physical and logical standby databases
A standby database can be either a physical standby database or a logical standby database. A physical standby database provides a physically identical copy of the primary database, with on-disk database structures that are identical to the primary database on a block-for-block basis. The database schema, including indexes, is the same. A physical standby database is kept synchronized with the primary database, through Redo Apply, which recovers the redo data received from the primary database and applies the redo to the physical standby database. A logical standby database contains the same logical information as the production database, although the physical organization and structure of the data can be different. The logical standby database is kept synchronized with the primary database through SQL Apply, which transforms the data in the redo data received from the primary database into SQL statements and then executes the SQL statements on the standby database. A logical standby database can be used for other business purposes in addition to disaster recovery requirements. The users can access a logical standby database for queries and reporting purposes at any time.

2

each database continues to participate in the Data Guard configuration with its new role. it can be reinstated as a standby for the new primary database once the reason for the failure is corrected. reporting. Data Guard integration with Serviceguard Extended Distance Cluster (EDC). while continuously applying changes received from the production database. Oracle Active Data Guard enhances the quality of service by offloading resource-intensive activities from a production database to one or more synchronized standby databases. Web-based access. or Continentalclusters provides disaster protection and recovery for Oracle databases as well as for the applications that use the databases. This protection mode provides the highest level of data protection that is possible without compromising the availability of a primary database. and so on. the role of a database can be changed using either a switchover or a failover operation. The broker’s interfaces improve usability and centralize management and monitoring of the Data Guard configuration. Flashback Database removes the need to re-create the primary database after a failover. If Flashback Database 3 is enabled on the primary database. the redo data needed to recover a transaction must be written to both the online redo log and to at least one synchronized standby database before the transaction commits. enabling you to return a database to its state at a time in the recent past. Oracle Active Data Guard enables read-only access to a physical standby database for queries. and can provide additional benefits of high availability and disaster protection against planned or unplanned outages at the production site. Data Guard Broker The Data Guard broker is a distributed management framework that automates the creation. volume groups. After a switchover. some data loss may occur. Switchover This operation allows the primary database to switch roles with one of its standby databases. Active Data Guard Active Data Guard is a new feature for Oracle 11g. 3 . To provide this level of protection. maintenance. Using Data Guard. Serviceguard has built-in monitoring capabilities to monitor system resources like network. Metrocluster. 1 2 3 This protection mode ensures that zero data loss occurs if a primary database fails. Serviceguard support for Oracle Data Guard Providing HA for ODG is essential for mission-critical business that requires ODG to be deployed in an environment that has a wide range of applications. Integrating ODG with Serviceguard using the toolkit has the following additional advantages: 1. and monitoring of Data Guard configurations. 2. Oracle Active Data Guard also enables the use of fast incremental backups when offloading backups to a standby database. Provides HA for Data Guard processes and the Data Guard broker (if used) for both primary and the standby databases. and to initiate failover following detection. sorting. Data Guard provides disaster protection and recovery only for Oracle databases. Flashback Database is similar to conventional point-in-time recovery in its effects. Role transitions An Oracle database can be operated in one of two roles: primary or standby. If the primary database was not operating in either maximum protection mode 1 or maximum availability 2 mode before the failure. Failover This operation changes a standby database to the primary role in response to a primary database failure. and filesystems. There is no data loss during a switchover.The standby databases in a Data Guard configuration can be a combination of physical and logical standby databases.

The ODG Toolkit has a hard dependency on the ECMT Oracle Toolkit. stop. In the case of a single-instance database.com/go/hpux-serviceguard-docs Supported configurations Single-instance Oracle database Figure 1: ODG Toolkit in a single-instance Oracle database environment SG Cluster 1 SG Cluster 2 Data Guard replication Package for primary DB Node 1 Node 2 Package for standby DB Node 1 Node 2 Primary DB Standby DB The ODG Toolkit can be implemented as a combinational package with the ECMT Oracle Toolkit. and the ASM instances. NOTE: For information about supportability and compatibility with various versions of Serviceguard and HP-UX. 3. Dependencies The ODG Toolkit requires the ECMT Oracle Toolkit so that together they provide HA to single-instance ODG. This is advantageous from the complete application stack. Serviceguard’s robust failover mechanism can be extended to provide reliable automatic Data Guard role management (failover and switchover) operations. 4 . Instant Capacity. This enables workload management. A combinational package is one in which two applications are packaged together by combining their respective Serviceguard modules into one package. The ODG Toolkit integration similarly allows the users to take advantage of these capabilities for failure scenarios that affect ODG replication. The Serviceguard Toolkit for Oracle Data Guard consists of a set of shell scripts that are used to start. stop. Serviceguard is integrated with Insight Dynamics—VSE for Integrity Servers. as opposed to accounting for the Oracle Database alone. please refer to the supportability matrix available at http://www. the listener. and other VSE technologies to be controlled in concert with Serviceguard. Hence. and monitor ODG primary and standby database instances. this toolkit leverages scripts from the ECMT Oracle Toolkit to start.Serviceguard has extensive dependencies that allow these application interdependencies to be built in for a complete environment/deployment. and monitor the database. customers who do not already have the ECMT product must purchase it along with the ODG Toolkit.hp. 4.

the package will failover if either the Oracle database or any of the Data Guard processes fail. then the command to create a combinational package is: cmmakepkg –m ecmt/oracle/oracle –m tkit/dataguard/dataguard <pkg_file_name> where: – ecmt/oracle/oracle is the Oracle toolkit module shipped with ECMT Oracle Toolkit. If the customer does not have an ECMT Oracle package running and wants to create the Data Guard package afresh. the Oracle database will be brought up first using the ECMT Oracle Toolkit. then: – If the ECMT Oracle package is a legacy-style package. Configuring an ODG Toolkit involves two scenarios: 1. it must be migrated to modular style. its value will be tkit/dataguard/dataguard – output_file_name is the template file that gets generated with the values of the ECMT Oracle database module populated in it. The recommended way is to discard the legacy package and create the Data Guard package afresh in modular style. The user should set the “ACTIVE_STANDBY” parameter to “yes” when he wants to use this feature. In case of Data Guard package. Note that since the primary database and all the standby databases are configured in separate clusters. NOTE: The package parameter. – If the ECMT Oracle package is a modular-style package. – tkit/dataguard/dataguard is the Data Guard toolkit module shipped with the ODG Toolkit. Active Data Guard is supported in Oracle database version 11gR1 or later. The user must edit this file and enter values for the Data Guard specific package attributes and then apply the package using the cmapplyconf command. You can achieve this using the “cmmigratepkg” tool. the standby database will be started up to the “open” state. then the application is monitored. then the Data Guard processes will be started by the ODG Toolkit. and then apply the file using cmapplyconf command to create the package. Subsequently.As a combinational package. a new combinational package must be created. 2. the Data Guard module can be inserted into it by using the following command: cmmakepkg –i <pkg_ascii_file> -m <module_file_name> <output_file_name> where: – pkg_ascii_file is the package file of the existing ECMT Oracle package. separate combinational packages must be created for each primary database and standby database. Since the Oracle database and the Data Guard are packaged together. migrating the legacy package to modular style is not the recommended way. – pkg_file_name is the template file that gets generated. This is because the ODG Toolkit does not support legacy-style packaging. However. The user needs to edit this file and enter values for the ECMT Oracle specific package attributes and also for the Data Guard specific package attributes. 5 . If a customer already has an ECMT Oracle Toolkit package running and wants to convert the Oracle database into a Data Guard setup. “START_MODE” must be set to “mount” when an ECMT Oracle Toolkit is used in combination with an ODG Toolkit. This can be generated using the command “cmgetconf –p <pkg_name> <output_filename>“ – module_file_name is the name of the module to be included in the running package. NOTE: In the case of Active Data Guard.

Recovery package: This package is also created using the ODG Toolkit on the Recovery Cluster. 3. this package will be in halted state. This package will bring the Oracle database on the Recovery Cluster in Standby mode and will start monitoring the standby database processes. It will be configured to bring up the Oracle database on the Recovery Cluster in Primary mode. Initially. then this package will halt the Oracle database and will failover the package to another node within the Recovery Cluster. Primary package: This package is created using the ODG Toolkit on the Primary Cluster. If the standby database fails. Even though both these clusters can have more than two nodes. All the above three packages are configured as failover packages. 6 . Data Guard is configured using the ODG Toolkit and the Oracle database is placed on a disk that is shared between the nodes in the Primary Cluster. we have used only two nodes in the clusters in our example. 2. then the Primary package will be failed over to another node within the Primary Cluster. This package will bring up the Oracle database on the Primary Cluster as a primary database and will start monitoring the primary database processes. The standby database is created in the Recovery Cluster and is also placed on a shared disk. If the primary database fails on Node 1. In the Primary Cluster.Continentalclusters environment Figure 2: ODG Toolkit in a Continentalclusters environment Continentalcluster SG Cluster 1 ( Primary Cluster) SG Cluster 2 (Recovery Cluster) Primary package Recovery group Data Receiver package Recovery package Node 1 Node 2 Node 1 Node 2 Primary DB Standby DB Figure 2 depicts a typical Continentalclusters environment. Data Receiver package: This package is created using the ODG Toolkit on the Recovery Cluster. A Recovery group is created in the Continentalclusters environment with the following three packages in it: 1. which consists of a Primary Cluster (SG Cluster 1) and a Recovery Cluster (SG Cluster 2).

When the primary database fails. Thus. then Serviceguard configured on the Recovery Cluster allows the database to failover to another node within the Recovery Cluster. This is because Continentalclusters does not want the packages to be brought up when the cluster is brought up. the Primary package and the Data Receiver package will be up and running and the Recovery package will be in halted state. a typical Data Guard environment is set up with data replication done from the primary database on the Primary Cluster to the standby database on the Recovery Cluster. which will bring up the database as a primary database. the AUTO_RUN attribute of the packages must be disabled at the time of bringing up the packages in the Recovery group. Restoring the cluster in a Continentalclusters to its original state is a manual process. Halt the Recovery package. 7 . the administrator must run the “cmrecovercl” command on the Recovery Cluster to bring up the Recovery package.Initially. thus providing HA to the primary database. Note that in this case. Start the Data Receiver package—this will bring up the database on the Recovery Cluster as a standby database. However. This is because there can be only one Data Receiver package and only one Recovery package within a Recovery group. When the Primary Cluster goes down. 2. The following steps must be performed to restore the clusters back to their original state: 1. package switching can be manually enabled using the “cmmodpkg” command after the packages in the Recovery group are started. if the standby database fails. Serviceguard configured on the Primary Cluster fails over the database to another node within the Primary Cluster. In such a case. and then bring up the Recovery package. Similarly. the local failover of the package within the cluster at the time of package failure will not happen. In a Continentalclusters environment. 3. the ODG Toolkit in Continentalclusters will not prevent users from configuring other standby databases that are placed outside the Continentalclusters setup. Multiple standby databases cannot be supported in a Continentalclusters setup. Start the Primary package—this will bring up the database on Primary Cluster as a primary database. But if this attribute is not enabled. which will halt the standby database. This command will first halt the Data Receiver package. The Primary package will bring up the Oracle database on the Primary Cluster in Primary mode and the Data Receiver package will bring up the Oracle database on the Recover Cluster in Standby mode. role management is handled by ODG Toolkit.

The integration of ODG with Metrocluster or EDC gives organizations this choice.Metrocluster and EDC configurations Metrocluster is an HP high availability product for Serviceguard customers who require integrated disaster recovery solutions. either locally or at other data centers. While Metrocluster provides storage-based replication of data across metropolitan distances. HP Metrocluster uses Serviceguard clustering technology to form a single cluster of systems that are located apart from each other at different data centers at metropolitan distances. Having both storage replication and ODG replication can help to safeguard customer data. organizations often like to build in additional data replicas. Single-instance Data Guard configuration in Metrocluster Figure 3: Single-instance Data Guard configuration in Metrocluster with standby database residing outside Metrocluster Metrocluster Data Center 1 (Primary site) Primary package Primary DB Data Center 2 (Recovery site) Node 1 Node 2 Node 3 Node 4 Primary DB Data replication in Metrocluster Primary DB Third location (Serviceguard cluster) Standby package Standby DB Node 1 Node 2 Standby DB 8 .

it will continue to send the archived logs to the standby database located at the third location. there will be only one server at the third location and the standby database instance has to be brought up manually. Once the primary database instance comes up at Data Center 2. The third location is a separate Serviceguard cluster with two nodes and a shared disk. then the primary database instance will be failed over to Data Center 2 and will continue to function as a primary database. which is not configured within the Metrocluster environment. The standby database instance is configured at a third location (say site 3). Since Metrocluster has achieved data replication from Data Center 1 to Data Center 2. Data Guard in EDC environment Figure 4: ODG in EDC environment in which both the primary and the standby are single-instance databases Site A Package for primary DB EDC Package for standby DB Site B Node 1 Node 2 Node 3 Node 4 Primary DB Standby DB 9 . The primary database instance will be brought up on any one node in Data Center 1 while other nodes in the data center will serve as backup nodes for that instance. It is not necessary to configure the standby database in Data Center 2 because the data replication is taken care of by the Metrocluster solution. If high availability is not desired for the standby database then the third location need not be configured in a Serviceguard cluster. if Data Center 1 is down.In Figure 3. there will not be any problems when starting the primary database at Data Center 2 using the replicated data from the shared disk. and the database and the archived logs will be located on the shared disk. In such a situation. The Data Guard primary database instance is configured in Data Center 1 such that the archived logs and the primary database reside on the disk that is shared locally between the nodes in that data center. In case of a disaster. Metrocluster is configured with two data centers: one as the primary site and the other as the recovery site. Array-based replication technology enables the data written to the storage array in Data Center 1 to be replicated to the storage array located in Data Center 2. The standby database instance will be brought up on any one node in the third location.

The Primary Cluster has two data centers (sites). All the four nodes are present inside the same Serviceguard cluster. the role of Standby on Site B has to be manually changed to primary. The primary database of Data Guard setup is configured as a Primary package in the Primary Cluster (Metrocluster instance). When Site A goes down. since the role transitions will not be supported in the first release. Failover from Site A to Site B and vice versa is not allowed. each of which have storage array for storing the data. Both the primary and standby databases are configured as single instance Serviceguard failover packages inside the EDC. The Primary Cluster is configured as a Metrocluster spread over two different sites that are geographically dispersed within the confines of a metropolitan area. The Recovery Cluster is configured on the third site as a Serviceguard cluster. 10 .Figure 4 shows a Data Guard setup in an EDC environment. Metrocluster enables data written to the primary site to be replicated to the recovery site and hence the primary database instance can failover to any node within the Primary Cluster. Metrocluster in Continentalclusters environment Figure 5: Single-instance Data Guard setup in a Continentalclusters environment where the Primary Cluster is configured as a Metroclusters Continentalcluster Data Center 1 (Primary site) Primary package Primary DB Primary Cluster (Metrocluster ) Data Center 2 ( Recovery site) Node 1 Node 2 Node 3 Node 4 Primary DB Data replication in Metrocluster Primary DB Recovery Cluster (Third site) Data Receiver package Standby DB Recovery package Node 1 Node 2 Standby DB Figure 5 describes a Continentalclusters setup with two clusters spread over a total of three different sites.

HA for Data Guard Broker HA for Data Guard Broker is not supported on a single-instance database.The standby database is configured as a Data Receiver package in the Recovery Cluster. This means that they should have different volume groups for storing the database and redo logs. It will receive the archived redo logs sent by the primary database instance irrespective of the node on which the primary database is running. 11 . Two single-instance Data Guard primary databases are configured in Cluster 1 and one standby database is configured for each of the primary databases in Cluster 2. However. They should use different Oracle database listeners and IP addresses or ports. Multiple standby configurations are not supported within a Continentalclusters setup. Both the primary database in Cluster 1 and the standby database in Cluster 2 and their corresponding redo logs are located on shared disks such that each can be accessed from any nodes in their respective clusters. When the Primary Cluster fails. all the instances must be configured in such a way that they function independently. Thus there will be no instance of standby database running once the role of the standby database is changed to primary. This will enable failover of the primary and standby packages within the cluster. Cross subnet failover is allowed across the sites. Once this command is run. the user must run the “cmrecovercl” command to halt the Data Receiver package and start the Recovery package. the ODG Toolkit in Continentalclusters will not prevent customers from configuring standby databases that are placed outside the Continentalclusters environment. the Recovery package starts and brings up the Oracle database as a primary database. Note that both these clusters are independent of each other and there cannot be package failover between the two clusters. thus providing HA for both of them. Supporting multiple instances of single-instance ODG configuration Figure 6: Multiple Data Guard instances in one Serviceguard cluster SG Cluster 1 SG Cluster 2 Primary DB Package 2 Primary DB Package 1 Primary DB Package 3 Standby DB Standby DB Package 4 Node 2 Standby DB Node 1 Node 1 Node 2 Primary DB Standby DB To support multiple Data Guard instances in one Serviceguard cluster. Figure 6 shows a supported configuration where two two-node Serviceguard clusters are configured to provide HA to the Data Guard configurations.

This is a configuration file that is read by the toolkit script.” or “open. NOTE: In the event of package failover to the adoptive node. The following variables are contained in hadg. If the file exists and if the maintenance feature is enabled.” NOTE: For the ODG Toolkit.” MAINTENANCE_FLAG: The maintenance flag is used to bring this toolkit into maintenance mode. make sure that the database instance on the adoptive node will be able to communicate in the Data Guard configuration. After the maintenance work. This will be used for starting and stopping the database.conf (user configuration file) This script contains a list of predefined variables that the user must customize for use with a particular database instance.debug file under this directory. This uniquely identifies an Oracle database instance. For example: ORACLE_ADMIN=oracle SID_NAME: The Oracle session name. In this case. Two more scripts (tkit_module. START_MODE: This parameter determines the startup mode for Oracle database.” “mount. which are used for modular packaging. This requires appropriate configuration of the listener and network services. To put the toolkit into maintenance mode.1). create the dataguard. ensure that the ECMT B. After installing the ODG Toolkit. These scripts are: .conf: TKIT_DIR: This directory is synonymous with the package directory and holds the toolkit configuration file.Installation and configuration of the toolkit Setting up the application ODG is included with the Enterprise Edition and Standard Edition of the Oracle database software. two scripts (hadg. please refer to Appendix A.sh and tkit_gen.sh.hadg. This parameter directs “cmapplyconf” to generate the hadg. Oracle must be installed on all the nodes of the cluster. by the user “oracle” and shared storage has to be configured.sh) and one file (dataguard. 4 Setting up the toolkit This toolkit has to be used in combination with the ECMT Oracle module for a single-instance Oracle database.conf file under this directory. If set to “yes. hadg.” this will enable the maintenance feature in the toolkit.debug” in the package directory.06. make sure that the instance is 4 Listener and network services can be configured by using Oracle’s netmgr utility. ORACLE_ADMIN: User name of the Oracle database administrator. For information on configuring ODG.00 has been installed. monitoring is paused. The default value is “open.” It can take the options “nomount. The package will not be failed over to the adoptive node even though the instance has been brought down for maintenance. always specify the value of this parameter as “mount. 12 . and database instance may be brought down for maintenance.conf) and a README file will be installed in the /opt/cmcluster/toolkit/dataguard directory. will be installed in the /etc/cmcluster/scripts/tkit/dataguard directory and /etc/cmcluster/modules/tkit/dataguard respectively. ORACLE_HOME: The base directory of Oracle where it is installed. The Serviceguard Toolkit for ODG will look out for a file “dataguard.sh and hadg.

The default value is 30 seconds. The TIME_OUT variable is used to protect against a worst-case scenario where a hung database or ASM instance prevents the halt script from completing.sh) This script contains a list of internally used variables and functions that support the starting and stopping of an ODG instance. This script will be called by tkit_module.conf) in the package directory (TKIT_DIR).On package startup. 13 . it stops the Data Guard instance and monitor process. This will enable the toolkit to continue monitoring the database server application.” DG_BROKER: Specifies whether the ODG broker is to be used or not. The TIME_OUT variable has no effect on package failover times. the value of this parameter has to be “yes” in the package configuration file of the recovery package.debug” from the package directory. NOTE: If the maintenance flag is set to “no.” then the above feature will not be available and the toolkit cannot be brought into maintenance mode.” The default value is “no. This parameter can be set to “yes” or “no. TIME_OUT: The time for which the toolkit waits for a completion of a normal shutdown before initiating forceful halt of the application. The Active Data Guard Option available with Oracle Database 11g Enterprise Edition enables you to open a physical standby database for read-only access.Toolkit Configuration File Generator Script (tkit_gen. therefore preventing the standby node from starting the instance. ODG Broker management is not supported in this release hence this parameter must be set to “no. .Attribute Definition File (Data Guard) The ADF is used to generate a package ASCII template file.On package halt. NOTE: The following three scripts are used only during the modular method of packaging.” The default value is “no.brought up properly.sh to perform the following: .sh) This script is called by the Master Control Script and acts as an interface between the Master Control Script and the Toolkit interface script (hadg.sh).Module Script (tkit_module. in seconds. ACTIVE_STANDBY: This parameter determines whether the database instance is an active standby or not.” START_STANDBY_AS_PRIMARY: Specifies whether the standby database has to be started as the primary database by failing over the primary database. It is also responsible for calling the Toolkit Configuration File Generator Script (described below). MONITOR_INTERVAL: The time interval. Main Script (hadg. . it starts the Data Guard instance (primary/standby) and launches monitor processes. ALERT_MAIL_ID: This parameter is used to specify the email address for sending alerts. . This parameter can be used in a Continentalcluster environment.sh) This script is called by the Module Script when the package configuration is applied using “cmapplyconf” to generate the user configuration file (hadg. this script will wait between checks to make sure that the Oracle instance is running.” This can take values “yes” or “no. Delete the file “dataguard. . The default value is 30 seconds.

it takes it from the installation directory. Note that the package configuration file shown below contains attributes of both the ECMT Oracle Toolkit and the ODG Toolkit.sh script in the configuration directory. Create a directory in /etc/cmcluster.conf b. The ODG Toolkit can be configured in one of the following methods: a) Install directory method: Here the scripts remain in the installation directory. Follow the instructions in the chapter “Building an HA Cluster Configuration” in the manual “Managing HP Serviceguard“ to create the logical volume infrastructure on shared disks. Only the parameters that are to be modified in dgpkg.ODG package configuration example This section explains an ODG package configuration example. The example configuration uses the installation directory mode operation. the ODG Toolkit module has been used with the ECMT Oracle modules as follows: #cmmakepkg -m ecmt/oracle/oracle -m tkit/dataguard/dataguard dgpkg. This example on ODG Package Setup and Configuration is for an ODG configuration using LVM. In the example below. The disk must be available to all clustered nodes that will be configured to run the ODG Toolkit.” This directory will eventually become TKIT_DIR and “cd” to this directory. Specifying configuration parameters in the package Once the package configuration file has been created. Note that the ODG Toolkit has to be used in combination with the ECMT Toolkit (in case of a single-instance database). for example. Serviceguard first tries to use the hadg. Run the following commands to create the package configuration file templates. Creating a package configuration Create two packages: one for the primary database on the Primary Cluster and the other for the standby database on the Standby Cluster. NOTE: The following attributes are specific to the Oracle Toolkit in ECMT # # “package_name” is the name that is used to identify the package. # Package names must be unique within a cluster.conf specifically for this configuration are shown here. The users can modify the scripts in the configuration directory to add any specific requirements. # 14 . A combinational package is the one in which two applications are packaged together by combining their respective Serviceguard modules into one package. The example illustrates the creation of a package for ODG in a single-instance environment. b) Configuration directory method: Here the user has to copy the scripts from the installation directory (including contents of subdirectories) to the configuration directory and define this location in the parameter “TKIT_DIR” in the package configuration file. the user must specify various parameter values. a. If the script is not found in the configuration directory. Create file systems on all logical volumes on the volume groups. “dataguard.

# run_script_timeout halt_script_timeout 300 610 -----------------------------------------------------------------------# “script_log_file“ is the full path name for the package control script log # file. # “halt_script_timeout” is the number of seconds allowed for the package to halt. # package_type failover -----------------------------------------------------------------------# # “run_script_timeout” is the number of seconds allowed for the package to start. script_log_file /etc/cmcluster/dataguard/pkg. # The package_type attribute specifies the behavior for this package.log --------------------------------------------------------------------------# # Define package configuration directory # ecmt/oracle/oracle/TKIT_DIR -----------------------------------------------------------------------# # # ecmt/oracle/oracle/INSTANCE_TYPE -----------------------------------------------------------------------# # Define Oracle home database Define the instance type /etc/cmcluster/dataguard 15 . # package_description “Serviceguard Package” -----------------------------------------------------------------------# # “package_type” is the type of package.package_name dgpkg -----------------------------------------------------------------------# # “package_description” specifies the application that the package runs.

# ecmt/oracle/oracle/ORACLE_HOME -----------------------------------------------------------------------# # Define user name of Oracle database administrator # ecmt/oracle/oracle/ORACLE_ADMIN -----------------------------------------------------------------------# # Define oracle session name # ecmt/oracle/oracle/SID_NAME -----------------------------------------------------------------------# # Define Oracle database startup mode # ecmt/oracle/oracle/START_MODE -----------------------------------------------------------------------# # Define whether the database instance is using ASM or not # ecmt/oracle/oracle/ASM -----------------------------------------------------------------------# # Define ASM disk groups used by the database instance # #ecmt/oracle/oracle/ASM_DISKGROUP ----------------------------------------------------------------------# # Define the volume groups used in the ASM disk groups for the # database instance # #ecmt/oracle/oracle/ASM_VOLUME_GROUP -----------------------------------------------------------------------# # The ASM home no mount ORCL oracle /var/orahome 16 .

sh oracle_monitor” LISTENER_ORCL yes oracle 17 . service_name service_cmd oracle_service_test “$SGCONF/scripts/ecmt/oracle/tkit_module.# #ecmt/oracle/oracle/ASM_HOME -----------------------------------------------------------------------# # Define user name of Oracle ASM administrator # ecmt/oracle/oracle/ASM_USER ---------------------------------------------------------------------# # The ASM session name # #ecmt/oracle/oracle/ASM_SID -----------------------------------------------------------------------------# # Define whether the configured listener has to be started with the server # ecmt/oracle/oracle/LISTENER -----------------------------------------------------------------------# # Define the Oracle listener name(s) # ecmt/oracle/oracle/LISTENER_NAME -----------------------------------------------------------------------# # Define the listener password(s) # #ecmt/oracle/oracle/LISTENER_PASS -----------------------------------------------------------------------# #ecmt/oracle/oracle/LISTENER_RESTART # #ecmt/oracle/oracle/LISTENER_RESTART -----------------------------------------------------------------------NOTE: The following are the service commands for the combinational package.

sh oracle_hang_monitor 30 failover” none no 300 dataguard_service_test “$SGCONF/scripts/tkit/dataguard/tkit_module.service_restart none no 300 oracle_listener_service_test “$SGCONF/scripts/ecmt/oracle/tkit_module. -----------------------------------------------------------------------# # # tkit/dataguard/dataguard/ACTIVE_STANDBY -----------------------------------------------------------------------# # # tkit/dataguard/dataguard/DG_BROKER -----------------------------------------------------------------------# # # tkit/dataguard/dataguard/START_STANDBY_AS_PRIMARY -----------------------------------------------------------------------# no Define START_STANDBY_AS_PRIMARY no Define DG_BROKER no Define ACTIVE_STANDBY 18 .sh dataguard_monitor“ none no 300 service_fail_fast_enabled service_halt_timeout service_name service_cmd service_restart service_fail_fast_enabled service_halt_timeout service_name service_cmd service_restart service_fail_fast_enabled service_halt_timeout service_name service_cmd service_restart service_fail_fast_enabled service_halt_timeout NOTE: The following attributes are specific to the ODG Toolkit.sh oracle_monitor_listener” none no 300 oracle_hang_service_test “$SGCONF/scripts/ecmt/oracle/tkit_module.

conf $ cmmodpkg -e -n <node1> -n <node2> dgpkg $ cmmodpkg -e dgpkg For more information on adding a package to the cluster. see the “Managing HP Serviceguard” document available at: http://www.” “fs_mount_opt.hp. add the package to the Serviceguard cluster and start it up.com/go/hpux-serviceguard-docs ODG maintenance There might be situations where the ODG database instances have to be taken down for maintenance purposes like changing configuration without having the instance migrate to standby node. $ cmapplyconf -P dgpkg. 19 .” “fs_umount_opt. The following procedure should be followed: The toolkit maintenance feature is enabled only when the configuration variable MAINTENANCE_FLAG is set to “yes” in the toolkit configuration file. # vg vgora -----------------------------------------------------------------------# # “fs_name.# # Define email address for sending alerts #tkit/dataguard/dataguard/ALERT_MAIL_ID -----------------------------------------------------------------------# # “vg” is used to specify which volume groups are used by this package. Note that the toolkit maintenance feature is different from the Serviceguard’s package maintenance feature.” “fs_directory.” “fs_fsck_opt.” # and “fs_type” specify the filesystems that are used by this package. # fs_name fs_directory fs_type fs_mount_opt #fs_umount_opt #fs_fsck_opt -----------------------------------------------------------------------/dev/vgora/lvol1 /oradb “vxfs” “-o rw” Adding the package to the cluster After the setup is complete.

20 . 1. Verify ODG setup: To verify a Data Guard configuration. $ cd /etc/cmcluster/pkg/dgpkg/ $ $PWD/hadg. If required.hp. see the “Managing HP Serviceguard” guide available at http://www. package directory is /etc/cmcluster/pkg/dgpkg. 5.sh stop 4.debug Toolkit monitor scripts (both database instance and listener monitoring scripts). Also check if the RFS process is running on the standby database by querying the table “V$MANAGED_STANDBY. stop the database instance(s) as shown below: $ cd /etc/cmcluster/pkg/dgpkg/ $ $PWD/hadg. Allow monitoring scripts to continue normally as shown below: $ rm -f /etc/cmcluster/pkg/dgpkg/dataguard. $ cmmodpkg -e dgpkg Note: If a package failure occurs during maintenance operations. would now stop monitoring these daemon processes. Perform maintenance actions (Example: Database maintenance).com/go/hpux-serviceguard-docs Troubleshooting This section provides the guidelines to verify if the ODG package has been configured properly or not. the package does not automatically failover to an adoptive node. Disable the failover of the package through cmmodpkg command. and the ORACLE_HOME is configured as /orahome.Note: The example assumes that the package name is dgpkg. 7. 3.debug The message “Starting Oracle Data Guard monitoring again after maintenance” appears in the Serviceguard Package Control script log. Enable the package failover. Check especially for error messages. $ cmmodpkg -d dgpkg 2. Restart the Oracle database instance again if you manually stopped it prior to maintenance. Create an empty file /etc/cmcluster/pkg/dgpkg/dataguard. obtain information about the archiving status of the primary database by querying the V$ARCHIVE_DEST view. which continuously monitor ODG process daemons’ processes. The messages.sh start 6. You must manually start the package on the adoptive node.debug as shown below: $ touch /etc/cmcluster/pkg/dgpkg/dataguard.” If the standby site is not receiving the logs. Pause the monitor script. check if the standby database is able to receive the redo logs. “Serviceguard Toolkit for ODG pausing Data Guard monitoring and entering maintenance mode” appears in the Serviceguard Package log file. For more information on manually starting the package on an adoptive node. The following steps help the user to troubleshoot the possible causes for the package failure: 1.

The listener is not started. .--------. . check the following list of possible issues. Verify the Toolkit setup: Check whether the package configuration attributes specified in the package configuration file are valid or not.The standby instance is not started.ora file has not been configured correctly at the standby site. .The listener. .For example. . If the replication does not happen from the primary database to the standby database then check if the online redo log files are configured or not on the primary as well as the standby.ora file at the primary site.The service name listed in the LOG_ARCHIVE_DEST_n parameter of the primary initialization parameter file is incorrect. you used a backup from the wrong database. check the Oracle Alert log for errors. 2. .You used an invalid backup as the basis for the standby database (for example. 21 . . Also. or did not create the standby control file using the correct method).------------------------------------------------------------------1 VALID 2 ERROR /vobs/oracle/work/arc_dest/arc standby1 ORA-16012: Archivelog standby database Error identifier mismatch 3 INACTIVE 4 INACTIVE 5 INACTIVE 5 rows selected If the output of the query does not help.The LOG_ARCHIVE_DEST_STATE_n parameter specifying the state of the standby archiving destination has the value DEFER. Known problems and workarounds 1. enter: SQL> SELECT dest_id “ID” 2> status “DB_status” 3> destination “Archive_dest” 4> error “Error” 5> FROM v$archive_dest. ID DB_status Archive_dest -. .You have added a standby archiving destination to the primary initialization parameter file. but have not yet enabled the change.The service name for the standby instance is not configured correctly in the tnsnames.

Note that these are generic steps to configure the Data Guard. Enable force logging: Place the primary database in FORCE LOGGING mode after the database creation using the 22 . Detailed information about configuring Data Guard can be obtained from the Oracle Web site (http://www.hp. then the ODG Toolkit package will fail. see: • HP Serviceguard Solutions: http://www. Details about creating a logical standby database can be found at the Oracle Web site (http://download.oracle. The standby database can be created in two ways: • Physical standby database • Logical standby database Configuring the physical standby database in a single-instance Oracle database is described in this section. Legacy-type support is required to support this feature.com/docs/cd/E11882_01/server. If customer already has a database and wants to make use of Data Guard for replication to another standby site.com). which can be run directly by the user. The following steps should be followed to create the primary database: 1. the data replication can happen only from the site where the primary is running. the role management is handled by the ODG Toolkit and in other cases. This is a restriction put by Data Guard. ODG Toolkit does not support legacy-style configuration. Multiple standby database instances cannot be supported in the Continentalclusters setup since For more information To learn more. The Data Guard broker should not be configured for automatic role transitions. Preparing the primary database Before creating the standby database. Data Guard replicates only updates to the database. there can be only one Recovery package and only one Data Receiver package within the Recovery group.2. it is the customer’s responsibility to set up the standby in the same state as the primary.oracle.htm). This is similar to configuring ECMT Toolkit. In case of 4.hp. The following steps describe how to configure the Data Guard. Other standby instances can be configured outside the CC environment. In case of a Metrocluster environment where only the primary database is configured within the Metrocluster and the standby database is configured on a third site that is not part of the Metrocluster. 5.112/e10700/create_ls. the primary database must be configured first. Customers have to use third party or Oracle-supplied tools to back up and copy the existing database to the standby site. 3. and not the ODG Toolkit. Even if the proximity of the other site is better.com/go/hpux-serviceguard-docs Appendix A Configuring ODG example ODG is included with the Enterprise Edition and Personal Edition of the Oracle database software. There are use-cases in which only the primary database is configured in a SG cluster and the standby is not part of any cluster. Continentalclusters scenarios.com/go/serviceguardsolutions • Technical Documentation: www. data replication cannot happen from that site. if the role of any database is changed.

Every database in a Data Guard configuration must use a password file. Set up the environment to support the standby database: a. 4. the online redo logs are also accessible to the new instance on the adoptive node. 23 .rdo”) SIZE 500K.rdo.following SQL statement: SQL> ALTER DATABASE FORCE LOGGING. Create a password file: Create a password file if one does not already exist. 4. Creating a physical standby database 1. and the password for the SYS user must be identical on every system for redo data transmission to succeed. SQL> STARTUP MOUNT. Prepare an initialization parameter file for the standby database. Create a control file for the standby database: SQL> STARTUP MOUNT. Create a backup copy of the primary database data files: You can use any backup copy of the primary database to create the physical standby database as long as you have the necessary archived redo log files to completely recover the database. and set the password for the SYS user to the same password used by the SYS user on the primary database. 3. The password for the SYS user on every database in a Data Guard configuration must be identical for redo transmission to succeed. define initialization parameters that control log transport services while the database is in the primary role. If the backup procedure required you to shut down the primary database. 3. Oracle recommends that you use the Recovery Manager utility (RMAN). SQL> ALTER DATABASE OPEN. as shown in the following example: SQL> ALTER DATABASE CREATE STANDBY CONTROLFILE AS “/tmp/standby. 5. SQL> ALTER DATABASE ARCHIVELOG. 2. Here’s a sample SQL statement that adds a new group of redo logs to the database: SQL> ALTER DATABASE ADD LOGFILE (“<oracle_redo_file_1>. Create a password file.ctl”. issue the following SQL*Plus statement to start the primary database: 2. issue the following statements to put the primary database in ARCHIVELOG mode and enable automatic archiving: SQL> SHUTDOWN IMMEDIATE. It is recommended that the online redo log files are created on the shared disk so that when a local failover of the primary database instance occurs. and open the primary database to user access. Create a password file.” “<oracle_redo_file_2>. SQL> ALTER DATABASE OPEN. Setting the primary database initialization parameters: On the primary database. Then create the control file for the standby database. Copy files from the primary system to the standby system: Use an operating system copy utility to copy the following files from the primary system to the standby system: Backup data files Standby control file Initialization parameter file 5. Enable archiving: If archiving is not enabled.

EXPIRE_TIME parameter to two minutes in the SQLNET. host address. port. Temporary files enable disk sorting when the database is open in read-only mode and prepare the database for future role transitions. To restart the listeners (to pick up the new definitions). The Oracle Net service name must resolve to a connect descriptor that uses the same protocol.ORA parameter file on the standby system. c. use Oracle Net Manager to configure a listener for the respective databases. Start the physical standby database. On both the primary and standby sites. Configure listeners for the primary and standby databases. To add temporary files to the physical standby database. The connect descriptor must also specify that a dedicated server be used. On an idle standby database.ora of a node contains the entries of all those nodes that it wants to communicate with. 6. b. Identify the table spaces that should contain temporary files. Create a new temporary file for the physical standby database. Create a server parameter file for the standby database. Enable broken-connection detection on the standby system. a. TABLESPACE_NAME -------------------------------TEMP1 TEMP2 Add new temporary files to the standby database. Creating a new temporary file on the physical standby database at this point rather than later is beneficial. enter the following LSNRCTL utility commands on both the primary and standby systems: % lsnrctl stop % lsnrctl start Ensure that the tnsnames. 24 . On both the primary and standby systems. For example: SQLNET. use Oracle Net Manager to create a network service name for the primary and standby databases that will be used by log transport services. perform the following tasks: 1.EXPIRE_TIME=2 d. On the standby database. issue the following SQL statements to start and mount the database in read-only mode: SQL> STARTUP OPEN READ ONLY. Create Oracle Net service names. and SID that you specified when you configured the listeners for the primary and standby databases. Start the physical standby database: Perform the following steps to start the physical standby database and Redo Apply. use the SQL CREATE statement to create a server parameter file for the standby database from the text initialization parameter file SQL> CREATE SPFILE FROM PFILE=“initstandby. Enable broken-connection detection by setting theSQLNET.ora”.b. Do this by entering the following command on the standby database: SQL> SELECT TABLESPACE_NAME FROM DBA_TABLESPACES 2> WHERE CONTENTS = “TEMPORARY”. e.

FIRST_TIME. you should first identify the existing archived redo log files on the standby database. The following steps show how to perform these tasks. On the standby database. APPLIED FROM V$ARCHIVED_LOG ORDER BY SEQUENCE#. query the V$ARCHIVED_LOG view to verify the redo data was received and archived on the standby database: SQL> SELECT SEQUENCE#. SQL> SELECT SEQUENCE#. On the standby database. a. Force a log switch to archive the current online redo log file. To force a log switch so that redo data is transmitted immediately. and then check the standby database again. Verify that the new redo data was archived on the standby database. A log switch occurs by default when an online redo log file becomes full. use the following ALTER SYSTEM statement on the primary database. Identify the existing archived redo log files. b. d. issue the ALTER SYSTEM ARCHIVE LOG CURRENT statement to force a log switch and archive the current online redo log file group: SQL> ALTER SYSTEM ARCHIVE LOG CURRENT.For each table space identified in the previous query. query the V$ARCHIVED_LOG view to verify the archived redo log files were applied. On the standby database. The following example adds a new temporary file called TEMP1 with size and reuse characteristics that match the primary database temporary files: SQL> ALTER TABLESPACE TEMP1 ADD TEMPFILE 2> “/arch1/standby/temp01. The transmission of redo data to the remote standby location does not occur until after a log switch. NEXT_TIME 2 FROM V$ARCHIVED_LOG ORDER BY SEQUENCE#. For example: SQL> SELECT SEQUENCE#. force a log switch and archive a few online redo log files on the primary database. Also. Start Redo Apply.dbf” 3> SIZE 40M REUSE. Test archival operations to the physical standby database. For example: SQL> ALTER SYSTEM SWITCH LOGFILE. d. c. This statement automatically mounts the database. c. 7. Verify new archived redo log files were applied. On the primary database. the statement includes the DISCONNECT FROM SESSION option so that Redo Apply runs in a background session. 25 . Verify that the physical standby database is performing properly: To see that redo data is being received on the standby database. issue the following command to start Redo Apply: SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION. FIRST_TIME. On the standby database. NEXT_TIME 2> FROM V$ARCHIVED_LOG ORDER BY SEQUENCE#. query the V$ARCHIVED_LOG view to identify existing files in the archived redo log. add a new temporary file to the standby database.

L. Upgrade the data protection mode: The Data Guard configuration is initially set up in the maximum performance mode (the default).com/bizsupport/feedback/ww/webfeedback. enabling you to return a database to its state at a time in the recent past. Data Guard can recover and apply more redo data from standby redo log files than from the archived redo log files alone. configuring standby redo logs is recommended on all standby databases.P. HP shall not be liable for technical or editorial errors or omissions contained herein. The standby redo logs should exist on both primary and standby databases and have the same size and names.com/go/sgoracledg Share with colleagues © Copyright 2010 Hewlett-Packard Development Company. Flashback Database removes the need to re-create the primary database after a failover. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Nothing herein should be construed as constituting an additional warranty. Enable Flashback Database: HP welcomes your input. through our technical documentation feedback website: http://www.hp. However. Configure standby redo logs: Standby redo logs are required for standby databases running in the maximum protection mode and maximum availability mode.Further preparations At this point. Flashback Database is similar to conventional point-in-time recovery in its effects. Flashback Database is faster than pointin-time recovery because it does not require restoring data files from backup or the extensive application of redo data. 3. or both. The information contained herein is subject to change without notice. Created September 2010 . You can enable Flashback Database on the primary database. 2. the standby database. the physical standby database is running and can provide the maximum performance level of data protection. Please give us comments about this white paper. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. This can be change to either maximum protection mode or maximum availability mode. visit: http://www. 4AA2-7717ENW. or suggestions for related documentation.hp. because during a failover.html To learn more about the HP Serviceguard Toolkit for Oracle Data Guard. The following list describes additional preparations you can take on the physical standby database: 1.