You are on page 1of 22

7 Using Oracle Clusterware to Manage Active Standby PairsOracle Clusterware moni tors and controls applications to provide high

availability. This chapter descri bes how to use Oracle Clusterware to manage availability for a TimesTen active s tandby pair. For more information about Oracle Clusterware, see Oracle Clusterware Administra tion and Deployment Guide in the Oracle Database documentation. This chapter includes the following topics: Overview The cluster.oracle.ini file Creating and initializing a cluster Recovering from failures Planned maintenance Overview Figure 7-1 shows an active standby pair with one read-only subscriber in the sam e local network. The active database, the standby database and the read-only sub scriber are on different nodes. There are two nodes that are not part of the act ive standby pair that are also running TimesTen. An application updates the acti ve database. An application reads from the standby and the subscriber. All of th e nodes are connected to shared storage. Figure 7-1 Active standby pair with one subscriber Description of "Figure 7-1 Active standby pair with one subscriber" You can use Oracle Clusterware to start, monitor and automatically fail over Tim esTen databases and applications in response to node failures and other events. See "Planned maintenance" and "Recovering from failures" for details. Oracle Clusterware can be implemented at two levels of availability for TimesTen . The basic level of availability manages two master nodes and up to 127 read-on ly subscriber nodes in the cluster. The active standby pair is defined with loca l host names or IP addresses. If both master nodes fail, user intervention is ne cessary to migrate the active standby scheme to new hosts. When both master node s fail, Oracle Clusterware notifies the user. The advanced level of availability uses virtual IP addresses for the active, sta ndby and read-only subscriber databases. Extra nodes can be included in the clus ter that are not part of the initial active standby pair. If a failure occurs, t he use of virtual IP addresses allows one of the extra nodes to take on the role of a failed node automatically. If your applications connect to TimesTen in a client/server configuration, autom atic client failover enables the client to reconnect automatically to the master database with the active role after a failure. See "Working with the TimesTen C lient and Server" in Oracle TimesTen In-Memory Database Operations Guide and "TT C_FailoverPortRange" in Oracle TimesTen In-Memory Database Reference. The ttCWAdmin utility is used to administer TimesTen active standby pairs in a c luster that is managed by Oracle Clusterware. The configuration for each active standby pair is manually created in an initialization file called cluster.oracle

.ini by default. The information in this file is used to create Oracle Clusterwa re resources. Resources are used to manage each TimesTen daemon, database, Times Ten processes, user applications and virtual IP addresses. For more information about the ttCWAdmin utility, see "ttCWAdmin" in Oracle TimesTen In-Memory Databa se Reference. For more information about the cluster.oracle.ini file, see "The c luster.oracle.ini file". Active standby configurations Use Oracle Clusterware to manage only these configurations: Active standby pair with or without read-only subscribers Active standby pair (with or without read-only subscribers) with AWT cache group s Active standby pair (with or without read-only subscribers) with read-only cache groups Active standby pair (with or without read-only subscribers) with AWT cache group s and read-only cache groups Required privileges See "ttCWAdmin" in Oracle TimesTen In-Memory Database Reference for information about the privileges required to execute ttCWAdmin commands. Hardware and software requirements Oracle Clusterware release 11.1.0.7 is supported with TimesTen active standby pa ir replication. See Oracle Clusterware Administration and Deployment Guide for n etwork and storage requirements and information about Oracle Clusterware configu ration files. Oracle Clusterware and TimesTen should be installed in the same location on all nodes. All machines should use Network Time Protocol (NTP) or a similar system so that clocks on the machines remain within 250 milliseconds of each other. Restricted commands and SQL statements When you use Oracle Clusterware to manage an active standby pair, you cannot use these commands and SQL statements: CREATE ACTIVE STANDBY PAIR, ALTER ACTIVE STANDBY PAIR and DROP ACTIVE STANDBY PA IR SQL statements The -repStart and -repStop options of the ttAdmin utility The -cacheStart and -cacheStop options of the ttAdmin utility after the active s tandby pair has been created The -duplicate option of the ttRepAdmin utility The ttRepStart and ttRepStop built-in procedures In addition, do not call ttDaemonAdmin -stop before calling ttCWAdmin -shutdown. The TimesTen integration with Oracle Clusterware accomplishes these operations w ith the ttCWAdmin utility and the attributes in the cluster.oracle.ini file. For more information about the built-ins and utilities, see Oracle TimesTen In-M emory Database Reference. For more information about the SQL statements, see Ora

You can include spaces for readability if desired. [basicDSN] in the cluster. The information in this file is used to create Oracle Clusterware resources that manage TimesTen databa ses. The ttCWAdmin utility is used to administer TimesTen active standby pairs in a c luster that is managed by Oracle Clusterware. The entry name and the DSN must be the same as the database file name.oracle.ini file described in "Configuring basic availability" must exist in the sys.host2 This is an example of a cluster.cle TimesTen In-Memory Database SQL Reference. The user creates the cluster.oracle.ini file are descri bed in Chapter 8.oracle. See "Defi ning the DSNs for the databases" for a description of the database file name.ini file for an active standby pair with one subscriber on host3: .ini files for these configurations: Configuring basic availability Configuring advanced availability Including cache groups in the active standby pair Including the active standby pair in a cache grid Implementing application failover Recovering from permanent failure of both master nodes Using the RepDDL attribute Configuring basic availability This example shows an active standby pair with no subscribers.odbc. The list of hosts is delimited by commas.ini file as a text file and places it in the daemon home directory on the host for the active database. The cluster.oracle. The entry name in the cluster.oracle.ini file must be the same as an existing DS N: In the sys.ini file.ini file on Linux or UNIX In a system DSN on Windows For example.ini file Create an initialization file called cluster. [basicDSN] MasterHosts=host1. This section includes sample cluster.ini.oracle.oracle. The hosts for the active database and the standby database are host1 and host2.oracle. "TimesTen Configuration Attributes for Oracle Clusterware". By default this dire ctory is: The install_dir/info directory on Linux and UNIX The c:\TimesTen\install_dir\srv\info directory on Windows All of the attributes that can be used in the cluster.odbc. user applications and virtual IP addresses. TimesTen processes.

assign port numbers for the active and standby databases by setting the GridPort attribute.host4 MasterVIP=192.255.host2 SubscriberHosts=host3 Configuring advanced availability In this example.255.168.1.host3.2 SubscriberVIP=192.168.1.host3 SubscriberHosts=host4.0 CacheConnect=y Including the active standby pair in a cache grid If the active standby pair is a member of a cache grid. Including cache groups in the active standby pair If the active standby pair replicates one or more AWT or read-only cache groups. 192.168. The active standby pair is a member of a cache grid . set the CacheConnect attribute to y.168.[basicSubscriberDSN] MasterHosts=host1.host5 MasterVIP=192.1. This example specifies an active standby pair with one subscriber in an advanced availability configuration. MasterVIP specifies the virtual IP addresses defi ned for the master databases. The active standby pair replicates one or more cach e groups. VIPInterface is the name of the public network adaptor.168.1. VIPNetMask defines the netmask of the virtual IP addresses.2 SubscriberVIP=192. 192.255.2 VIPInterface=eth0 VIPNetMask=255.168.255.3 VIPInterface=eth0 VIPNetMask=255.1. There is one extra node that can be us ed for failing over the master databases and one extra node that can be used for the subscriber database.1.host3 SubscriberHosts=host4.1.0 Ensure that the extra nodes: Have TimesTen installed Have the direct-linked application installed if this is part of the configuratio n.168. the hosts for the active database and the standby database are host1 and host2.255. 192.0 This example has one subscriber on host4. [advancedCacheDSN] MasterHosts=host1. host5 MasterVIP=192.1. VIPNetMask defines the netmask of the virtual IP addresses.host2. [advancedSubscriberDSN] MasterHosts=host1.host2.1.3 VIPInterface=eth0 VIPNetMask=255. This example specifies an active standby pair with no subscribers in an advanced availability configuration.168.1. [advancedDSN] MasterHosts=host1. There are no subscriber nodes. MasterVIP and SubscriberVIP specify the virtual IP add resses defined for the master and subscriber databases. VIPInterface is the name of the public network ada ptor. .255. host3 and host4 are extra nodes that can be used for failover.host2. See "Implementing application failover".1.

0 CacheConnect=y GridPort=16101.2 VIPInterface=eth0 VIPNetMask=255. it takes action to recover the failed application.sh AppStopCmd=/mycluster/reader/app_stop.0 AppName=reader AppType=Standby AppStartCmd=/mycluster/reader/app_start. The possib le values are Active.255.1.host4 MasterVIP=192.Determines the database to which the application is linked. The reader application is an application that queries t he data in the standby database.255.oracle.oracle. These attributes have default values. The TimesTen application monitor process uses the user-supplied script or progra m specified by AppCheckCmd to monitor the application. The required attributes in the cluster.255. [appDSN] MasterHosts=host1.ini file for failing over an appli cation are: AppName .168.host2.host3 MasterVIP=192.1.2 VIPInterface=eth0 VIPNetMask=255.sh AppStartCmd. 16102 Implementing application failover TimesTen integration with Oracle Clusterware can facilitate the failover of an a pplication that is linked to any of the databases in the active standby pair. When Oracle Clusterware detects a nonzero value. cons ider a valid cluster.168. The script for starting. The quotes are needed because there are spaces in the path.host2.Command line for executing an application that checks the status o f the application specified by AppName AppType . -stop and -check).1. 192.[advancedGridDSN] MasterHosts=host1. 192.168.1. Subscriber (all) and Subscriber[index].host3. For example.Command line for stopping the application AppCheckCmd .168.sh AppCheckCmd=/mycluster/reader/app_check. . AppStopCmd and Ap pCheckCmd.255. The script that checks th e status of the application must be written to return 0 for success and a nonzer o number for failure. In this example the application is directly linked to the active database. you can also set AppFailureThreshold. a nd checking the application takes arguments for the DSN and the action to take ( -start. Standby.Command line for starting the application AppStopCmd .1. Bo th direct-linked and client/server applications that are on the same machine as Oracle Clusterware and TimesTen can be managed.ini file on Windows. Note the double quotes for the specified paths in AppStartCmd. and AppScrip tTimeout. This example shows advanced availability configured for an active standby pair w ith with no subscribers.Name of the application to be managed by Oracle Clusterware AppStartCmd .1. stopping. AppFailoverDelay. AppStopCmd and AppCheckCmd can include arguments. Optionally.

exe" -dsn myDSN -stop AppCheckCmd="C:\Program Files\UserApps\UpdateApp.168.1. You can include blank lines f or readability. RepBackupDir specifies a directory on shared storage. AutoRecover is set to y. AutoRecover is set to y.[appWinDSN] MasterHosts=host1.host2. Automatic recovery of temporary dual failure requires: RETURN TWOSAFE is not specified for the active standby pair.sh AppCheckCmd=/mycluster/update/app2_check.sh AppStopCmd=/mycluster/update/app2_stop.host3. The active standby pair does not replicate cache groups.0 AppName=UpdateApp AppType=Active AppStartCmd="C:\Program Files\UserApps\UpdateApp.255.host2.2 VIPInterface=eth0 VIPNetMask=255. AppStopCmd and AppChe ckCmd immediately following the AppName attribute.1. Oracle Clusterware can automati cally recover the master databases. Use AppName to name th e application and provide values for AppType. For example: [app2DSN] MasterHosts=host1. If both master nodes fail permanently.0 AppName=reader AppType=Standby AppStartCmd=/mycluster/reader/app_start. AppStartCmd.host3. Oracle Clusterware can automatically reco ver the master databases to two new nodes if: Advanced availability is configured (virtual IP addresses and at least four host s).255. A cache grid is not configured.1. 192.exe" -dsn myDSN -check You can configure failover for more than one application. RETURN TWOSAFE is not specified.168.2 VIPInterface=Local Area Connection VIPNetMask=255.1.255.host4 MasterVIP=192.sh AppStopCmd=/mycluster/reader/app_stop.168. RepBackupDir specifies a directory on shared storage.exe" -dsn myDSN -start AppStopCmd= "C:\Program Files\UserApps\UpdateApp.1.168.255.sh Recovering from permanent failure of both master nodes If both master nodes fail and then come back up.host4 MasterVIP=192.sh AppCheckCmd=/mycluster/reader/app_check. RepBackupPeriod is set to a value greater than 0. .1.sh AppName=update AppType=Active AppStartCmd=/mycluster/update/app2_start. 192.

0 AutoRecover=y RepBackupDir=/shared_drive/dsbackup RepBackupPeriod=3600 If you have cache groups in the active standby pair or prefer to recover manuall y from failure of both master hosts. see the description of the ttBackup utility in Oracle TimesTen In-Memory Database Refer ence. When you specify a value for RepDDL. ensure that AutoRecover is set to n (the de fault).1. Include those replication settings in the RepDDL attribute.1.168.host3. The default value for AutoRecover is n.1. You can specify the optional attribute RepfullbackupCycle to manage when TimesTen performs subsequent full backup. Manual recovery requires: RepBackupDir specifies a directory on shared storage RepBackupPeriod must be set to a value greater than 0 This example shows attribute settings for manual recovery. TimesTen perform s backups for any master database that becomes active.ora cle. For more information about the interaction between backups and log records.255. If RepBackupDir and RepBackupPeriod are configured for backups.255. It does not delete backup s that were performed for a database that used to be active and has become the s tandby unless the database becomes active again. By default.168. TimesTen perf orms a full backup after every five incremental backups.1.2 VIPInterface=eth0 VIPNetMask=255.255. do not specify ReturnServi ceAttribute. [manrecoveryDSN] MasterHosts=host1. use the <DSN> macro for the database file n . Ensure that the shared storage has enough space for two complete database backups. MasterStoreAttribute or SubscriberStoreAttribute in the cluster.RepBackupPeriod must be set to a value greater than 0. 192.1.1. [autorecoveryDSN] MasterHosts=host1. ttCWAdmin -restore automatic ally chooses the correct backup files. This example shows attribute settings for automatic recovery.0 RepBackupDir=/shared_drive/dsbackup RepBackupPeriod=3600 Using the RepDDL attribute The RepDDL attribute represents the SQL statement that creates the active standb y pair. so it is not included in the file.255.host2.2 VIPInterface=eth0 VIPNetMask=255.168.ini file. You can use it to exclude tables. TimesTen first performs a full backup of the active database and then performs i ncremental backups. Ensure that the values of RepBackupPeriod and RepfullbackupCycle are small enough to prevent a large amount of log records in the transaction log file.ini file.host2. 192.oracle. The RepDDL attribute is optional.host4 MasterVIP=192.168. If you include RepDDL in the cluster. cach e groups and sequences from the active standby pair.host3 MasterVIP=192. Incremental backups increase the amount of log records in the transaction log fi les.

87.0. it attempts to connect to the rem ote daemon. where n is a number from 1 to the total number of SubscriberHosts attri bute values or 1 to the total number of SubscriberVIP attribute values if virtua l IP addresses are used.1.1.11.1.0.11.host2. Similarly. This is an example of specifying the ROUTE clause in RepDDL: [routeDSN] MasterHosts=host1.0.0. \ EXCLUDE SEQUENCE ttuser. which provides the private host names and public host n ames of the local and remote hosts as well as the remote daemon port number. cache groups and sequences from the active standby pair: [excludeDSN] MasterHosts=host1.168.255. in order of priority: From the ROUTE clause in the RepDDL setting.3" PRIORITY 1\ MASTERIP "10. TimesTen substitutes the correct values from the MasterHosts or MasterVIP attributes.87.255.empcount The replication agent transmitter obtains route information as follows.host4 SubscriberHosts=host5.1.11.168.204" PRIORITY 3\ SUBSCRIBERIP "140.204" PRIORITY 3\ ROUTE MASTER <DSN> ON <MASTERHOST[2]> SUBSCRIBER <DSN> ON <MASTERHOST[1]>\ MASTERIP "192.87.salaries. use the <SUBSCRIBERHOST[n]> macro to specify subscriber hos t names. \ EXCLUDE CACHE GROUP terry. using information that Oracle Clusterware maintains about the replic ation scheme.0.1" PRIORITY 2\ MASTERIP "140. The private host name is preferred over the public host name.1.host3.168.203" PRIORITY 3\ .87.1.host4 RepDDL=CREATE ACTIVE STANDBY PAIR \ <DSN> ON <MASTERHOST[1]>. if a ROUTE clause is specified.3" PRIORITY 1\ SUBSCRIBERIP "192.203" PRIORITY 3\ SUBSCRIBERIP "140.168. From Oracle Clusterware.host2.168. Do not specify a ROUTE clause if you are configuring advanced availability.ame prefix. If they fail.2" PRIORITY 1\ SUBSCRIBERIP "192. Use the RepDDL attribute to exclude tables.2 SubscriberVIP=192.0 RepDDL=CREATE ACTIVE STANDBY PAIR \ <DSN> ON <MASTERHOST[1]>.2" PRIORITY 1\ MASTERIP "10.11. From the active and standby hosts.salupdate.1. Use the <MASTERHOST[1]> and <MASTERHOST[2]> macros to specify the ma ster host names.host3.host6 MasterVIP=192. The replication agent transmitter cannot connect to the IPC socket. <DSN> ON <MASTERHOST[2]> SUBSCRIBER <DSN> ON <SUBSCRIBERHOST[1]>\ EXCLUDE TABLE pat.3 VIPInterface=eth0 VIPNetMask=255. <DSN> ON <MASTERHOST[2]>\ ROUTE MASTER <DSN> ON <MASTERHOST[1]> SUBSCRIBER <DSN> ON <MASTERHOST[2]>\ MASTERIP "192.168.0.1" PRIORITY 2\ SUBSCRIBERIP "10.2" PRIORITY 2\ MASTERIP "140.0.168. 192. depending on whether your configuration uses virtual IP a ddresses.1.2" PRIORITY 2\ SUBSCRIBERIP "10. then the replication agent choo ses the connection method based on host name.0.

. If you want to configure a remote disaster recovery subscriber. Oracle Clusterware starts automatically after successful installation. The installer uses these values to create the ttcrsagent. See Oracle Clusterware installation documentation for your platform. On Linux. Install TimesTen on each host Install TimesTen in the same location on each host in the cluster. The hosts included in the cluster. On Windows. with host names separa ted by commas. including ext ra hosts. perform these tasks: Install Oracle Clusterware Install TimesTen on each host Register the TimesTen cluster information Start the TimesTen cluster agent Create and populate a TimesTen database on one host Create a cluster.Creating and initializing a cluster To create and initialize a cluster. See "TimesTen Installation" in Oracle TimesTen In-Memory Data base Installation Guide for details. including spare hosts. This list must be the same on each node. The location must be the same on each node. see "Configuring a disaster recovery subscriber".options file. You can also use ttmodinstall -crs to creat e the file after installation. The Oracle Clusterware location. Install Oracle Clusterware Install Oracle Clusterware. see "Inclu ding more than one active standby pair in a cluster". execute ttmodinstall -crs on each node after installation to create the ttcrsagent. The port numb er can be different on each node. By default the installation occurs on all hosts conc urrently.ini file Create the virtual IP addresses (optional) Create an active standby pair replication scheme Start the active standby pair Load cache groups If you plan to have more than one active standby pair in the cluster. the installer prompts you for values for: The TCP/IP port number associated with the TimesTen cluster agent.oracle.options file on Linux a nd UNIX platforms. TimesTen uses the default TimesTen port.sh to perform identical installations on additional hosts if desir ed. If you do not provide a port number. On Linux and UNIX. you can use the -record and -batch opti ons for setup.

odbc. Oracle Clusterware begins monitorin g the TimesTen daemon and will restart it if it fails. On UNIX. Note: You must stop the TimesTen cluster agent on the local host before you execute a ttDaemonAdmin -stop command. enter this command: ttCWAdmin -ocrConfig As long as Oracle Clusterware and TimesTen are installed on the hosts. For example: ttCWAdmin -init When the TimesTen cluster agent has started. Create the virtual IP addresses (optional) For advanced availability.ini file on the host where you intend the active databas e to reside.odbc.For more information about ttmodinstall.oracle. Start the TimesTen cluster agent Start the TimesTen cluster agent by executing the ttCWAdmin -init command on one of the hosts. Create and populate a TimesTen database on one host Create a database on the host where you intend the active database to reside. see "ttmodinstall" in Oracle TimesTen I n-Memory Database Reference.ini files that are ide ntical to the sys. Create identical sys.oracle. For ex ample: ttCWAdmin -createVIPs -dsn myDSN Create an active standby pair replication scheme Create an active standby pair replication scheme by executing the ttCWAdmin -cre ate command on any host. This internal user will be used to create that the user will not need a password for an internal user information in the sys. AWT cache groups and read-only cache group s.ini file on the host where you intend the active database to reside.odbc. this step never needs to be repeated. Register the TimesTen cluster information TimesTen cluster information is stored in the Oracle Cluster Registry. See "The cluster. Do not load the cache groups.oracle. This command prompts for an encryption pass phrase gain.odbc.i the active standby pair. or as the instance administrator on Windows. create sys. . you must execute this command as the root user.ini files on other hosts On all hosts that will be in the cluster. Otherwise the cluster agent will restart the daemon . execute the ttCWAdmin -createVIPs command on any host in the cluster. For example: ttCWAdmin -create -dsn myDSN The command prompts you to choose the host on which the active database will res ide. The command also prompts for the user ID and with the ADMIN privilege if it does not find this ni file.ini file" for details about its contents an d location.ini file Create a cluster. Create a cluster. Create schema objects such as tables. As the ro ot user on UNIX or Linux platforms. Th e DSN must be the same as the database file name.

perform these tasks: Use the RepDDL or RemoteSubscriberHosts Clusterware attribute to provide informa tion about the remote disaster recovery subscriber. To use Oracle Clusterware to manage an active standby pair that has a remote dis aster recovery subscriber.0 Perform these tasks for additional replication schemes: Create and populate the databases.168.If CacheConnect is enabled. include additional configurations in the cluster.255.host3 SubscriberHosts=host4. Including more than one active standby pair in a cluster If you want to use Oracle Clusterware to manage more than one active standby pai r in a cluster.host3 SubscriberHosts=host4.168.1.6 VIPInterface=eth0 VIPNetMask=255. See "Using a disaster recovery subscriber in an active sta ndby pair". host5 MasterVIP=192. Configuring a disaster recovery subscriber You can create an active standby pair on the primary site with a remote disaster recovery subscriber. Create the virtual IP addresses. For example. For example: ttCWAdmin -start -dsn myDSN Load cache groups If the active standby pair includes cache groups.oracle.ini file contains configuration information f or two active standby pair replication schemes: [advancedSubscriberDSN] MasterHosts=host1.host2.1. The user must perform a switchover if the p rimary site fails.1. 192.5 SubscriberVIP=192. this cluster.ini file . Start the active standby pair Start the active standby pair replication scheme by executing the ttCWAdmin -sta rt command on any host.1. Use the ttCWAdmin -createVIPs command. Use the ttCWAdmin -create com mand.168. use the LOAD CACHE GROUP state ment to load the cache groups from the Oracle tables. host5 MasterVIP=192.168.oracle. Oracle Clusterware manages the active standby pair but does not mana ge the disaster recovery subscriber. Start the active standby pair.1. The Oracle password is used to set the autorefresh states for cac he groups.2 SubscriberVIP=192. the command prompts for the user password for the Or acle database.host2.3 VIPInterface=eth0 VIPNetMask=255. For example: [advancedDRsubDSN] .168.1.255.255. Use the ttCWAdmin -start command.255.0 [advSub2DSN] MasterHosts=host1.4.168.1. 192. Create the active standby pair replication scheme.

Recovering from failures Oracle Clusterware can recover automatically from many kinds of failures.1. If application failover is configured.255.MasterHosts=host1. then the application begins updating the new active database. then that database becomes the standby database.168. Load the cache groups that are replicated by the active standby pair. This does not create the disaster recovery subscriber. If it is successful. This s ection describes several failure scenarios and how Oracle Clusterware manages th e failures.3 VIPInterface=eth0 VIPNetMask=255.168. Set up the disaster recovery subscriber using the procedure in "Rolling out a di saster recovery subscriber".1. Oracle Clus terware automatically changes the state of the standby database to 'ACTIVE'.2 SubscriberVIP=192. Use ttCWAdmin -start to start the active standby pair replication scheme. Figure 7-2 shows that the state of the standby database has changed to 'ACTIVE' and that the application is updating the new active database. Figure 7-3 shows a cluster where the former active node becomes the standby node .1. Figure 7-3 Standby database starts on former active host Description of "Figure 7-3 Standby database starts on former active host" . This section includes these topics: When an active database or its host fails When a standby database or its host fails When read-only subscribers or their hosts fail When failures occur on both master nodes When more than two master hosts fail When an active database or its host fails If there is a failure on the node where the active database resides.255.host3 SubscriberHosts=host4. 192. host5 RemoteSubscriberHosts=host6 MasterVIP=192. Figure 7-2 Standby database becomes active Description of "Figure 7-2 Standby database becomes active" Oracle Clusterware tries to restart the database or host where the failure occur red.1.0 CacheConnect=y Use ttCWAdmin -create to create the active standby pair replication scheme on th e primary site.host2.168.

Oracle Clusterware first tries to res tart the database or host. Oracle Clusterware starts a standby database on one of the extra nodes. If it cannot restart the standby database on the same host and advanced availability is configured. . Oracle Clusterware starts the subscriber database on an extra node. Oracle Clusterware starts the sta ndby database on an extra node. When failures occur on both master nodes This section includes these topics: Automatic recovery Manual recovery for advanced availability Manual recovery for basic availability Manual recovery to the same master nodes when databases are corrupt Manual recovery when RETURN TWOSAFE is configured Automatic recovery Oracle Clusterware can achieve automatic recovery from temporary failure on both master nodes after the nodes come back up if: RETURN TWOSAFE is not specified for the active standby pair. Figure 7-5 Standby database on new host Description of "Figure 7-5 Standby database on new host" When read-only subscribers or their hosts fail If there is a failure on a subscriber node. Figure 7-4 shows a cluster in which the standby database is started on one of th e extra nodes. Oracle Clusterware first tries to re start the database or host. Figure 7-5 shows the standby database on a new host after a failure. RepBackupPeriod is set to a value greater than 0. Figure 7-4 Standby database starts on extra host Description of "Figure 7-4 Standby database starts on extra host" When a standby database or its host fails If there is a failure on the standby node. If it cannot restart the database on the same host a nd advanced availability is configured. AutoRecover is set to y. RepBackupDir specifies a directory on shared storage.If the failure of the former active node is permanent and advanced availability is configured.

ttCWAdmin -restore -dsn manrecoveryDSN If there are cache groups in the database. ttCWAdmin -start -dsn manrecoveryDSN Manual recovery for basic availability This section assumes that the failed master nodes will be recovered to new hosts on which TimesTen and Oracle Clusterware have been installed. AutoRecover is set to y. drop and re-create the cache groups. RepBackupDir specifies a directory on shared storage.ini file.ini file for examples. Ensure that there is not already a database on the host with the same DSN as the database you want to restore. If the new hosts are not already specified by MasterHosts and SubscriberHosts in the cluster. then modify the file to include the new hosts. To perform manual recovery in a basic availability configuration. .oracle. These steps use t he manrecoveryDSN database and cluster.ini file. To perform manual recovery in an advanced availability configuration. Re-create the active standby pair replication scheme. These steps use manrecoveryDSN. RETURN TWOSAFE is not specified for the active standby pair. These steps use t he basicDSN database and cluster.oracle. ttCWAdmin -init -hosts localhost Restore the backup database.oracle. ttCWAdmin -create -dsn manrecoveryDSN Start the active standby pair replication scheme. RepBackupPeriod is set to a value greater than 0.ini file for examples. perform th ese tasks: Ensure that the TimesTen cluster agent is running on the local host.oracle.Oracle Clusterware can achieve automatic recovery from permanent failure on both master nodes if: Advanced availability is configured (virtual IP addresses and at least four host s). The active standby pair does not replicate cache groups. See "Recovering from permanent failure of both master nodes" for examples of clu ster. A cache grid is not configured. Manual recovery for advanced availability This section assumes that the failed master nodes will be recovered to new hosts on which TimesTen and Oracle Clusterware have been installed.oracle. This step is not necessary for manrecoveryDSN be cause extra hosts are already specified in the cluster.ini files. perform these steps: Acquire new hosts for the databases in the active standby pair.

ttCWAdmin -start -dsn basicDSN Manual recovery to the same master nodes when databases are corrupt Failures can occur on both master nodes so that the databases are corrupt. destroy the databa ses by using the ttDestroy utility.Ensure that the TimesTen cluster agent is running on the local host.ini example includes backup configuration in case the databa se logs are not available: [basicTwosafeDSN] MasterHosts=host1. ttCWADmin -restore -dsn basicDSN If there are cache groups in the database. The SubscriberHosts entry changes from host2 to host20. ttCWAdmin -create -dsn basicDSN Start the active standby pair replication scheme.host2 ReturnServiceAttribute=RETURN TWOSAFE .oracle. You must re-create the active standby pair from the database logs or restore the databases from a backup first. ttCWAdmin -start -dsn basicDSN Manual recovery when RETURN TWOSAFE is configured You can configure an active standby pair to have a return service of RETURN TWOS AFE by using the ReturnServiceAttribute Clusterware attribute in the cluster. ttCWADmin -restore -dsn basicDSN If there are cache groups in the database. ttDestroy basicDSN Restore the backup database.ora cle. Re-create the active standby pair replication scheme. [basicDSN] MasterHosts=host10.oracle.host20 Re-create the active standby pair replication scheme. However. RETURN TWOSAFE is desig ned so that it is not possible to restart replication or reconnect applications automatically. The MasterHosts entry changes from h ost1 to host10. This example uses the basicDSN database. the database logs may be availa ble on one or both nodes after both nodes fail. This cluster. drop and re-create the cache groups.ini file. drop and re-create the cache groups. When RETURN TWOSAFE is configured. This example uses the basicDSN dat abase.ini fil e. perform the following steps: Ensure that the replication agent and the cache agent are stopped and that appli cations are disconnected from both databases. If yo u want to recover to the same master nodes. ttCWAdmin -init -hosts localhost Restore the backup database. ttCWAdmin -create -dsn basicDSN Start the active standby pair replication scheme. Ensure that there is not already a database on the host with the same DSN as the database you want to restore. ttCWAdmin -stop -dsn basicDSN On the node where you want the new active database to reside. Update the MasterHosts and SubscriberHosts entries in the cluster.

restore the database from backups.oracle. ttCWAdmin -restore -dsn basicTwosafeDSN Start the active standby pair replication scheme. The command prompts you to choose the host on which the active database will reside. See "Manual recovery for advanced availability" and "Manual recovery for basic a vailability" for guidelines on subsequent actions to take. ttCWAdmin -create -dsn basicTwosafeDSN If neither database is usable. Identify or obtain at least two healthy hosts for the active and standby databas es.ini fil e. Use these guidelines: Address the root cause of the failure if it is something like a power outage or network failure.RepBackupDir=/shared_drive/dsbackup RepBackupPeriod=3600 Perform these recovery tasks: Ensure that the replication agent and the cache agent are stopped and that appli cations are disconnected from both databases. ttCWAdmin -stop -dsn basicTwosafeDSN Drop the active standby pair. Planned maintenance This section includes the following topics: Changing the schema Performing a rolling upgrade of Oracle Clusterware software Upgrading TimesTen Adding a read-only subscriber to an active standby pair Removing a read-only subscriber from an active standby pair Adding an active standby pair to a cluster Removing an active standby pair from a cluster Adding a host to the cluster Removing a host from the cluster Reversing the roles of the master databases . ttCWAdmin -drop -dsn basicTwosafeDSN Decide whether the former active or standby database is more up to date and re-c reate the active standby pair using the chosen database. Update the MasterHosts and SubscriberHosts entries in the cluster. ttCWAdmin -start -dsn basicTwosafeDSN When more than two master hosts fail Approach a failure of more than two master hosts as a more extreme case of dual host failure.

ttCWAdmin -stop -dsn advancedCacheDSN Drop the active standby pair. Re-create the active standby pair replication scheme. ttCWAdmin -start -dsn advancedCacheDSN Performing a rolling upgrade of Oracle Clusterware software See Oracle Clusterware Administration and Deployment Guide. ttCWAdmin -drop -dsn advancedDSN Stop the TimesTen cluster agent on the host. Including tables. These c ommands use the advancedCacheDSN as an example. perform the following tasks: Stop the replication agents on the databases in the active standby pair. Only the PORT and TIMEOUT attributes can be set fo r subscribers. If you are upgrading between major releases. ttCWAdmin -drop -dsn advancedCacheDSN Modify the schema as desired. perform these tasks: Stop the replication agents on the databases in the active standby pair. Upgrading TimesTen To upgrade TimesTen. This removes the host from the clus ter and stops the TimesTen daemon. See "Database Upgrades" in Oracle TimesTen In-Memory Da tabase Installation Guide. ttCWAdmin -stop -dsn advancedDSN Drop the active standby pair. Every node in the cluster must have Times Ten from the same major release. ttCWAdmin -create -dsn advancedCacheDSN Start the active standby pair replication scheme. . ttCWAdmin -shutdown -hosts localhost Upgrade TimesTen on the desired hosts. sequences or cache groups from the active standby pair replica tion scheme To change the schema of the active standby pair. us e the ttMigrate utility. sequences or cache groups in the active standby pair replicati on scheme Excluding tables.Moving a database to a different host Performing host or network maintenance Performing maintenance on the entire cluster Changing user names or passwords Changing the schema Changing the schema of the active standby includes these actions: Adding or dropping a subscriber database Altering database attributes.

oracle. Create the active standby pair replication scheme. add a virtual IP address to the SubscriberVIP attribute. This includes the host in the cluster and star ts the TimesTen daemon. This example uses the advancedSubs criberDSN. ttCWAdmin -create -dsn advancedSubscriberDSN Start the active standby pair replication scheme. Create the active standby pair replication scheme. ttCWAdmin -init Create the active standby pair replication scheme. ttCWAdmin -start -dsn advancedSubscriberDSN Removing a read-only subscriber from an active standby pair To remove a read-only subscriber from an active standby pair. Add the subscriber to the SubscriberHosts attribute.ini file. . ttCWAdmin -create -dsn advancedSubscriberDSN Start the active standby pair replication scheme.oracle. See "Configuring advanced availability" for an example using these attributes. which has a subscriber and is configured for advanced availability. If the cluster is configured for advanced availability. Remove a virtual IP the SubscriberVIP attribute or remove the attribute altogeth er if there are no subscribers left in the active standby pair. ttCWAdmin -start -dsn advancedDSN Adding a read-only subscriber to an active standby pair To add a read-only subscriber to an active standby pair. which already has a subscriber and is configured for advanced availab ility. ttCWAdmin -create -dsn advancedDSN Start the active standby pair replication scheme.ini file. ttCWAdmin -drop -dsn advancedSubscriberDSN Modify the cluster. This example uses the advancedSubs criberDSN. ttCWAdmin -stop -dsn advancedSubscriberDSN Drop the active standby pair. ttCWAdmin -drop -dsn advancedSubscriberDSN Modify the cluster. perform these step s: Stop the replication agents on all databases.Start the TimesTen cluster agent. ttCWAdmin -stop -dsn advancedSubscriberDSN Drop the active standby pair. Remove the subscriber from the SubscriberHosts attribute or remove the attribute altogether if there are no subscribers left in the active standby pair. perform these steps: Stop the replication agents on all databases.

oracle. ttCWAdmin -createVIPs -dsn advSub2DSN Create the new active standby pair replication scheme. which was added in "Adding an active standby pair to a cl uster".1.0 [advSub2DSN] MasterHosts=host6.255. [advancedSubscriberDSN] MasterHosts=host1.ini file (optional). host5 MasterVIP=192. ttCWAdmin -dropVIPs -dsn advSub2DSN Modify the cluster. Modify the cluster. On Linux and UNIX. .255.1.4. This ex ample uses advSub2DSN.ttCWAdmin -start -dsn advancedSubscriberDSN Adding an active standby pair to a cluster To add an active standby pair (with or without subscribers) to a cluster that is already managing an active standby pair.255.168. ttCWAdmin -stop -dsn advSub2DSN Drop the active standby replication scheme.168. perform these tasks: Create and populate a database on the host where you intend the active database to reside initially.1. Remove the entries for advSub2DSN .oracle. p erform these tasks: Stop the replication agents on all databases in the active standby pair.255. ttCWAdmin -start -dsn advSub2DSN Removing an active standby pair from a cluster To remove an active standby pair (with or without subscribers) from a cluster. oracle.host3 SubscriberHosts=host4.5 SubscriberVIP=192. 192.1.6 VIPInterface=eth0 VIPNetMask=255.host7.168. See "Create and populate a TimesTen database on one host". host10 MasterVIP=192.host2. ttCWAdmin -create -dsn advSub2DSN Start the new active standby pair replication scheme.3 VIPInterface=eth0 VIPNetMask=255. ttCWAdmin -drop -dsn advSub2DSN Drop the virtual IP addresses for the active standby pair.2 SubscriberVIP=192.ini file.168.host8 SubscriberHosts=host9.1. the user must be root to do this. This example adds advSub2DSN to the cluster.168. 192.ini file that already contains the configuration for advancedSubscriberDS N.168.0 Create new virtual IP addresses. log onto each host that was included in th e configuration for this active standby pair and use the ttDestroy utility.1.1. The new active standby pair is on different hosts from the original active st andby pair. If you want to destroy the databases.

ttDestroy advSub2DSN For more information about ttDestroy. oracle. enter a command similar to the following: ttCWAdmin -delSubscriberHosts "subhost1" -dsn advancedSubscriberDSN Reversing the roles of the master databases After a failover. The examples in this section use the advancedSubscriberDSN. To remove two spare master host from the cluster. SubscriberHosts must list at least one more host than the number of subscriber databases if one of the subscriber hosts is to be remo ved. The -relocate option is useful for relocating a database if you need to take the host offline. Adding a host to the cluster Adding a host requires that the cluster be configured for advanced availability. Thus the newly migrated active database b ecomes the standby and the standby becomes the active. Ensure that there are no open transactions before you use the com mand.host9" -dsn advancedSubscriberDSN To remove a spare subscriber hosts from the cluster. MasterHosts must list more than two hosts if one of the mast er hosts is to be removed.ini file. the active and standby databases are on different hosts than t hey were before the failover.host9" -dsn advancedSubscriberDSN To add a spare subscriber host to a cluster. you can use the -relocat e option of the ttCWAdmin utility to move a database from the local host to the next available spare host specified in the MasterHosts attribute in the cluster. To add two spare master hosts to a cluster. The examples in this section use the advancedSubscriberDSN. For example: ttCWAdmin -switch -dsn basicDSN Moving a database to a different host When a cluster is configured for advanced availability. enter a command similar to the following: ttCWAdmin -delMasterHosts "host8. enter a command similar to the foll owing: ttCWAdmin -addSubscriberHosts -hosts "subhost1" -dsn advancedSubscriberDSN Removing a host from the cluster Removing a host from the cluster requires that the cluster be configured for adv anced availability. the -rel ocate option first reverses the roles. see "ttDestroy" in Oracle TimesTen In-Memo ry Database Reference. If there are open transactions. For example: ttCWAdmin -relocate -dsn advancedDSN Performing host or network maintenance If you need to upgrade the operating system or hardware for a host or perform ne . If the database on the local host has the active role. the command will fai l. Ensure that there are no open transactions before using the -switch option of the ttCWAdmin utility to reverse the roles of the ac tive and standby databases. enter a command similar to the follo wing: ttCWAdmin -addMasterHosts -hosts "host8.

Oracle Clus terware also stores the cache administration user name and password. These c ommands use the advancedCacheDSN as an example. Perform the host maintenance. Then enable automatic startup and start Oracle Clusterw are: # crsctl enable crs # crsctl start crs See Oracle Clusterware Administration and Deployment Guide for more information about these commands. ttCWAdmin -stop -dsn advancedCacheDSN Drop the active standby pair. To change t he user name or password for the internal user or the cache administration user. If there are cache groups in the active standby pair. E xecute these Oracle Clusterware commands as root or OS administrator: # crsctl stop crs # crsctl disable crs Shut down TimesTen. Performing maintenance on the entire cluster When all of the hosts in the cluster need to be brought down. Oracle Clusterware prompts for the user name and password of the internal user. shut down Oracle Clusterware and disable automatic startup. To change the user name or password of the internal user that created the active standby pair replication or to change the cache administration user name or pas sword. stop Oracle Cluste rware on each host individually. Then enable automatic startup and start Oracle Clu sterware: # crsctl enable crs # crsctl start crs See Oracle Clusterware Administration and Deployment Guide for more information about these commands. you must re-create the cluster. See "Shutting down a TimesTen application" in Oracle TimesTe n In-Memory Database Operations Guide. ttCWAdmin -drop -dsn advancedCacheDSN Change the appropriate user name or password: Change the internal user name or password by using the CREATE USER or ALTER USER .twork maintenance. perform these tasks: Stop the replication agents on the databases in the active standby pair. Perform the maintenance. Changing user names or passwords When you create the active standby pair replication scheme with the ttCWAdmin -c reate command. Execute these Oracle Clusterware commands as ro ot or OS administrator: # crsctl stop crs # crsctl disable crs Shut down TimesTen. See "Shutting down a TimesTen application" in Oracle TimesTe n In-Memory Database Operations Guide.

statements. ttCWAdmin -start -dsn advancedCacheDSN . ttCWAdmin -create -dsn advancedCacheDSN Start the active standby pair replication scheme. Re-create the active standby pair replication scheme. See "Setting the cache administration user name and pass word" in Oracle In-Memory Database Cache User's Guide. Change the cache administration user name or password by using the ttCacheUidPwd Set built-in procedure. See "Creating or identifying users to the database" in Oracle Times Ten In-Memory Database Operations Guide.