white paper

Eliminating Database Downtime When

Upgrading or Migrating from Oracle
8i or 9i to Oracle 10g or 11g
Publication Date: February 2009
Author: Alok Pareek, Vice President of Technology, GoldenGate Software, Inc.

Abstract:
Eliminating database downtime poses a significant challenge for IT organizations that need to upgrade or migrate
mission-critical database environments running Oracle database software versions 8i or 9i to Oracle 10g or 11g. That
is particularly true for critical applications that must provide continuous or near-continuous operations to users who
increasingly expect uninterrupted availability of online service. Any outage of an application or website, even if that
outage is scheduled or “planned,” has an impact on the revenue and reputation of the business.
The purpose of this paper is to present a solution that tackles the specific challenge of upgrading or migrating an
Oracle 8i or 9i database to Oracle software version 10g or 11g – without taking any database downtime. Key technology
components used in this solution are GoldenGate Software’s platform for transactional data management and Oracle’s
Cross Platform Transportable Tablespace feature.
The details within this paper will showcase key technical strengths of the solution, including how to achieve a
rolling upgrade/migration for this scenario; using a clone database to offload instantiation and conversions; keeping
transactions in synch across the databases; managing “partial” or phased migrations or upgrades; conducting data
verification post-upgrade/migration; and implementing an easy and reliable failover strategy.

© 2009 GoldenGate Software, Inc.

white paper

Table of Contents
Section Page Number
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
2. Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
3. Concepts and Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
GoldenGate’s Technology for Transactional Data Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Database Upgrade . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Database Migration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Standby Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Transportable Tablespace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Cross Platform Transportable Tablespace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Clone Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
RMAN (Recovery Manager) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
4. A Zero-Downtime Approach Using GoldenGate and Oracle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Upgrade vs. Migration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
5. Upgrade and Migration Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Business Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Technical Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
6. Available Solutions: 8i or 9i ➔ 10g or 11g Upgrade or Migration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Database Upgrade Assistant . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Export/Import . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
SQL Plus As Copy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Transportable Tablespaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
SQL*Loader . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Oracle Data Guard, Oracle Streams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
The GoldenGate Zero-Downtime Upgrades Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
7. Overview of GoldenGate’s Technology Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Capture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
On-Disk Queues aka “Trail files” . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Delivery (Apply) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
GoldenGate Veridata® . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
8. Overview of The Oracle Cross Platform Transportable Tablespace Feature . . . . . . . . . . . . . . . . . . . . . . . . 13
9. Detailed Steps Using a Platform Migration Case Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Migration without Failback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Migration with Failback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Failback Steps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Partial Migration Using Bi-Directional Synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
10. Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
About the Author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

© 2009 GoldenGate Software, Inc.

2
 1. INTRODUCTION
Today, mission-critical applications must provide continuous operations to users who increasingly expect
uninterrupted availability of online services. Any outage of an application or website, even if that outage
is scheduled, has an impact on the revenue and reputation of the business. In fact, many e-commerce
businesses report that a half-day outage would incur millions of dollars in financial losses. For databases that
host the data store for these mission-critical applications, availability requirements have become stringent.
From that high availability standpoint, there are essential events that still require application downtime:
modifying hardware or database software, upgrading applications, applying software patches, migrating to
different computing architectures, etc. Since such events are not conceived via system or data failures, they
are aptly classified as planned outages. Empirical data from non-trivial applications deployed in very large
database (VLDB) environments demonstrates that minimizing downtime to handle planned outages is a
complex, time consuming, error prone, and costly process.
For all of those reasons, many organizations choose to postpone those essential IT events as long as possible.
If such an event is a move to Oracle 10g or 11g, there is a better solution available today.
The purpose of this paper is to present a solution that takes the specific problem of either upgrading or
migrating an Oracle 8i or 9i database to Oracle software version 10g or 11g without taking any database
downtime (excluding application switchover time).
Eliminating database downtime for upgrades or migrations in mission-critical Oracle database environments
running Oracle database software versions 8i or 9i poses a significant challenge for IT organizations. Oracle
Corp. does not provide a database rolling upgrade solution when upgrading the database from Oracle version
8i or 9i to Oracle version 10g or 11g.1 As such, the solution described and illustrated in this paper is the
only proven solution that limits database downtime to mere seconds during the migration or upgrade from
8i or 9i to Oracle 10.x or higher. In the case of applications running on 8i, there’s a strong sense of urgency
and concern to upgrade to Oracle version 10, as the 8i product has reached end-of-life support from Oracle.
The key technology components used in this solution to achieve zero-database-downtime migrations are
GoldenGate Software, Inc.’s software platform for transactional data management and Oracle’s Cross Platform
Transportable Tablespace feature. Oracle’s Cross Platform Transportable Tablespace feature facilitates high-
speed data movement of Oracle data from one machine/platform to another; it is a recommended though not
mandatory method during instantiation of a target database that eventually participates in a rolling upgrade.

© 2009 GoldenGate Software, Inc.

1
white paper

2. GOALS
This paper is created primarily for advanced database users.
The main goal of this paper is to increase awareness of a solution for minimizing database downtime
during a database upgrade or migration under the circumstances outlined in the introduction. As such,
resource issues such as disk space, software-licensing costs, etc., are not discussed – not because such
issues are unimportant, but because the need to minimize downtime or completely avoid it outweighs other
considerations in demanding high availability environments.
Furthermore, this document is not intended to repeat the upgrade steps, typical warnings, and preparatory
details associated with an Oracle database upgrade. There are many articles and notes that exist on the
Oracle MetaLink customer support portal, in various books, and on the web, which are helpful towards
understanding the operational procedures and complexities associated with upgrades and migrations.
As one of the lead developers of the Cross Platform Transportable Tablespace feature in 10g at Oracle,
I struggled with minimizing downtime when migrating very large 8i or 9i databases from one platform to
another. My aim therefore is to present a novel solution that allows for a database upgrade/migration from
8i or 9i to 10g/11g without taking database downtime or having to make tablespaces read-only, or take any
locks. The solution presented leverages the benefits of real-time synchronization, clone databases, rolling
upgrades, transportable and cross-platform transportable tablespaces, and failback. Every effort is made to
avoid overhead at the primary database to ensure application availability (activities such as instantiation, file
conversion for cross platform transport, etc., are offloaded to a clone).
A secondary goal is also to explain how to minimize the “wall clock time” required to perform the entire
upgrade. Experienced users will realize that wall clock time can be on the order of days or weeks, yet the
downtime to the application can still be near zero. The key here, as will be explained in the following pages,
is to offload work processing to additional database copies.

3. CONCEPTS AND TERMINOLOGY

Throughout this document, we refer to the production database as the SOURCE and the secondary copy as
the TARGET database.
The following concepts and terminology deserve review to fully comprehend the solution proposed in this
white paper:

3.1 GoldenGate’s Technology for Transactional Data Management

GoldenGate’s software platform provides guaranteed change data capture, routing, transformation, and
delivery, across heterogeneous business systems in real time. It captures transactions non-intrusively from
a source database by reading transaction logs, transforms when needed, propagates, and applies those
transactions with guaranteed integrity to another database (target) in real time. GoldenGate’s processes run
continuously (and can even be bi-directional) and support high-volume, rapidly changing environments,
moving thousands of transactions per second with very low impact. The target database is a transactional
replica at a logical level; it can be functionally leveraged for multiple applications including a rolling database
upgrade.

© 2009 GoldenGate Software, Inc.

2
 Aside from the differentiating ability of GoldenGate to handle heterogeneous database environments, readers
familiar with Oracle data movement technologies may ruminate on the perceived resemblance between
Oracle Streams and GoldenGate. This collation, however, is purely conceptual. The underlying architectures
are significantly different. For example, Streams in Oracle 9i has problems scaling in high-throughput
environments. In addition, it is limited in its support for data types (e.g. LONG) and for certain kernel
functionality (e.g. log_parallelism). These issues are not present in GoldenGate’s platform.
GoldenGate’s software is designed for managing high transaction volumes in real time without impacting
throughput or commit time latencies on the source database. The transaction capture component does not
rely on the shared memory of the database instance to stage transactions or associated metadata, thereby
avoiding any resource contention with the ongoing application workload.
Not relevant to this specific topic, but noteworthy for this audience nevertheless, is that GoldenGate can also
be used when the target and source database software come from different vendors, and GoldenGate can
apply changed data in real-time to JMS message queues or topics as well.

3.2 Database Upgrade

A database upgrade advances the Oracle software release number from one version to another. There are two
primary ways to perform an upgrade:

3.2.1 In place
An in-place upgrade renders the database inaccessible for business applications while the database software
is being upgraded. This procedure entails running an upgrade script (e.g. for u0902000.sql), recompiling
invalid PL/SQL etc., and experiencing downtime that is usually not acceptable in most mission-critical
environments.
The intent of this paper is to not to address in-place upgrades.

3.2.2 Rolling Upgrade

The term rolling upgrade refers to upgrading different databases or different instances of the same database
(in a Real Application Clusters environment) one at a time, without stopping the database. Oracle doesn’t
allow a rolling database upgrade in Oracle 9i.
A rolling upgrade (database) consists of the following high-level steps:
1. The application points to the production database running software version VOLD
2. A secondary logical database copy is constructed running software version VOLD
3. The secondary database copy is upgraded to the next database version VNEW
4. The secondary and primary databases are synchronized
5. The primary database is shut down
6. The application is pointed to the secondary database
7. The primary is kept in the same version VOLD for failback reasons

© 2009 GoldenGate Software, Inc.

3
white paper

3.3 Database Migration

A migration allows for the underlying operating system or hardware platform to be changed. In Oracle, on-
disk file formats are not homogeneous across platforms. Under 10g compatibility, the on-disk structures for
platforms that appear in v$TRANSPORTABLE_PLATFORM are identical, but the endian format could differ.
Moving an Oracle database across different operating systems is a common requirement in many computing
environments.

3.4 Standby Database

A standby database is a copy of the main production database that is maintained for high availability or
disaster tolerance purposes. The standby database can be physical or logical.
In the physical standby copy, the database is kept on recovery mode (i.e. the redo logs of the production
database are applied to a mounted copy of the production database – the hardware/OS limitations here are
obvious since redo across Oracle platforms is not compatible).
A logical standby is a copy of the production database that contains the same objects, but doesn’t physically
match the structure of the production database copy. For example, a standalone table EMP in the physical
database could be represented as obj# 3143 whereas its logical equivalent at the standby site could be
represented as obj# 5931. It is maintained by instantiating a logical equivalent of the production database
and replaying the SQL that modifies the production database at the standby site.

3.5 Transportable Tablespace

Transportable Tablespace was a feature introduced in Oracle 8i that allows for non-system tablespaces to
be moved from one database to another by physically grafting the tablespaces’ datafiles into the target
database’s control file and importing object metadata into the target database’s dictionary.
It has three main phases:
1. Exporting the metadata (dictionary data for the objects)
2. Transferring the datafiles from one database to another
3. Importing the metadata and datafiles

3.6 Cross Platform Transportable Tablespace

Cross Platform Transportable Tablespace is an extension of the Transportable Tablespace feature and allows
tablespaces to be transported across database platforms. This feature can only be used after the database
compatibility has been advanced to 10.0.0.0 or higher.
This whitepaper makes use of this feature along with GoldenGate’s software to migrate an 8i or 9i database
to a 10g or 11g database.

3.7 Clone Database

A clone database is a database constructed using a restored backup of an existing database, recovered to a
point in time, and opened.

3.8 RMAN (Recovery Manager)

RMAN (Recovery Manager) is an Oracle tool that manages the process of making backups and managing
the process of restoring and recovering from them. It is also used for conversion of endian systems during a
cross platform CONVERT.

© 2009 GoldenGate Software, Inc.

4
 4. A ZERO-DOWNTIME APPROACH USING GOLDENGATE AND ORACLE
Since an in-place upgrade requires application downtime during the upgrade process, it is not feasible in critical HA
environments. Therefore we consider only rolling upgrades henceforth.
Using GoldenGate in conjunction with Oracle technologies, a rolling upgrade or a rolling migration can be performed
without taking any application downtime—other than the very minimal time required for application switchover,
typically less than one minute and in most cases, only seconds.

4.1 Upgrade vs. Migration

In general, a migration is more complex than an upgrade given that a migration includes both a software
upgrade and a migration to a different platform, and often there are on-disk incompatibilities across the
datafiles/logfiles/controlfiles of the source and target database platforms.
As such, this paper will use a real-life case study of migrating an existing Oracle database on the Sun
(Solaris™ OE (32-bit)) platform running on 9i to Linux (Linux IA (64-bit)) on 10g.
Readers who are interested only in an upgrade should skip the cross platform transport steps covered in
Section 8.

5. UPGRADE AND MIGRATION CHALLENGES

Such upgrade and migration projects pose challenges for mitigating business risk and navigating technology issues.
Some of the more common and critical challenges include:

5.1 Business Challenges

5.1.1 Impact on Revenue
Downtime equates to lost revenue. Many businesses (e.g. retail, travel, banking, etc.) no longer have the
extended downtime windows for planning upgrades/migrations. Many database environments, for scalability
or cost saving reasons, would like to take advantage of recent trends in technology such as deploying low
cost storage (new hardware), or other advances in clusterware and software. The following table (Figure 1)
illustrates the average downtime costs associated with various businesses.

Industry Segment/Application Average Cost per Hour of Downtime

Financial Markets $6.45 million
Credit Card Sales $2.65 million
Pay-Per-View Television $150,000
Home Shopping $113,000
Airline Reservations $89,000
Teleticket Sales $69,000
Shipping $28,000
Source: International Data Corporation and Sun Microsystems (http://www.sun.com/datacenter/continuity/availability/)

Figure 1: Average Cost of Downtime for Business Applications

© 2009 GoldenGate Software, Inc.

5
white paper

5.1.2 Customer Expectations/Loyalty

In the Internet era, businesses must continue to support online processing beyond conventional business
hours. For the user, the cost of switching over to the competition is non existent or minimal. For example, let’s
look at what happened when eBay took a 22-hour outage several years ago on a Thursday. In a combined
study by Nielsen Media Research and NetRatings Inc., which measures website usage, it was found that
during eBay’s downtime, the average time spent per person at Yahoo’s auction site the next day increased
from 7 minutes to 18 minutes and the number of people using Yahoo’s site skyrocketed from 62,000 on
Thursday to 135,000 on Saturday. Financial losses to eBay were estimated at $3 million to $5 million.2

5.1.3 Interdependencies, Business Reputation

E-commerce has resulted in digital business partnerships where non-availability of critical data at one site
may also result in loss of business services at multiple sites (via data access from either web services,
direct querying, or pre-configured batch loading). Extended downtime is becoming difficult to plan for
since significant coordination is required with business partners. This results in postponement of upgrades/
migrations, which has associated indirect costs (e.g. running the existing software with complex workarounds
for bugs, not being able to take advantage of new software functionality in the later releases, etc.).

5.2 Technical Challenges

The following presents some of the serious issues that need to be dealt with when considering a zero-
database-downtime upgrade/migration. Once these issues are well understood, one gains clarity into why a
unified solution using GoldenGate and Oracle’s Cross Platform Transportable Tablespace provides the best
way to perform the upgrade or migration for HA environments.

5.2.1 Instantiation
The first problem that needs to be addressed when performing a rolling upgrade is choosing the method
by which the target database is instantiated. In other words, how do you create a first version of the target
database?
The complexity of the problem is magnified when the target database is on a different platform, since physical
backups cannot simply be restored at the target and converted into a clone.
To handle multi-terabyte sized databases, procedures such as export/import are tedious and error prone.
Failures during the instantiation or global data consistency are not adequately addressed with methods that
are time consuming, unless some form of locking is used. Extracting data from the production database
over a period of time results in a performance impact on the source database that is unacceptable. There’s
also interference with the normal mechanics of the database cache manager affecting buffer allocation and
flushing algorithms, resource management algorithms, etc., since the typical OLTP database is not optimized
to handle long running queries, table scans, etc.
A good instantiation solution therefore must properly address:
½½ Global data consistency
½½ Efficiency (speed)
½½ Performance (i.e. low impact on the production database)
½½ Manageability

© 2009 GoldenGate Software, Inc.

6
 5.2.2 Incremental Data Movement
The next post-instantiation challenges are to determine:
i) The point that demarcates the last committed transaction picked up by the instantiation process
with the subsequent transactions submitted against the production database; and
ii) The method by which the subsequent transactions get propagated to the target database.
Depending on the instantiation method, one or both problems may be pertinent. The following example
depicts this challenge (See Figure 2).

Production database: Begin Instantiation: End Instantiation:

generating redo at database clock at database clock at
SCN 23.12345 SCN 23.12355 SCN 23.98745

Time

T1 T3 T100

Figure 2: High-Level Instantiation of a New Target Database for Upgrades/Migrations

Example:
1. Tables MASTER and SLAVE have a parent-child relationship via a foreign key constraint.
Bi-Directional TDM Configuration
2. The instantiation method (e.g. export, SQL copy, etc.) picks up a read consistent version of table
Source Target
MASTER as of SCNSource
Capture 23.12360.
Trail Target Trail Delivery

3. The instantiation method picks up a read consistent version of table SLAVE as of SCN 23.12777.
Route
LAN/WAN/Web/IP
4. Transaction TM inserts new row R1 into table MASTER at SCN 24.12890.
There are several problems
Delivery
with this method. First, how are transactions
Target Trail
handled
Source Trail
that are active as of SCN
Capture
23.12360 but commit after SCN 23.98745? Second, the MASTER and SLAVE tables are from different points
in time, and as a result are inconsistent with each other from a referential integrity point of view. And third,
the data submitted subsequent to SCN 23.12360 for the MASTER table (in this case row R1) still needs to
be moved.
Thus, a good solution that addresses incremental data movement must:
Compare & Verify
i) Ensure global consistency
ii) Identify a clean instantiation termination point
iii) Allow transactions from that point onward to be propagated to the target database
Capture
iv) Handle any failures during incremental data propagation (e.g. loss of network, media failure, etc.)
2 3 6 7
exp_norows.dmp
exp_xtts.dmp
Dprod Dprod backup Dpitr
9i / Sun Solaris 9i ➔ 10g
4 Sun Solaris
© 2009 GoldenGate Software, Inc.
7
8
white paper

5.2.3 Performance Impact

Another major concern during the upgrade is to understand what kind of impact is experienced by the
application running on the production database. The upgrade steps should not degrade performance on the
production database such that the application service levels (SLAs) are compromised.

5.2.4 Staging Areas

Adequate disk resources need to be configured and managed when addressing rolling upgrades or migrations.
Especially when a no-downtime migration is required – proper consideration must be given to handling the
disk requirements for the clone database.

5.2.5 Change Management

Changes other than those made by DML need to be addressed during the upgrade/migration process.
Examples of these are datafile additions, PL/SQL package creations, etc. In general if the overall solution is
fast (i.e. O[hours] ) such changes can be restricted since they are infrequent operations.

5.2.6 Special Handling of Legacy Data or Certain Datatypes

Any datatypes that are not supported by the solution must be identified ahead of time and dealt with separately.
These are usually objects that Export/Import doesn’t work with, such as certain application tables in the SYS
schema and typically those that are fairly small in size. These simply can be recreated at the target.
Additional handling is required to support certain datatypes when upgrading/migrating from Oracle 8i
environments.

5.2.7 Verification
Testing that the source and target database are synchronized needs to be conducted post upgrade/migration
— any out-of-synch conditions at the new source and/or failback database could pose risks to the business
and should be identified and acted upon. Conducting this verification should be possible even as ongoing
transactions are being processed at the source database.
A good solution must therefore address fast and efficient comparison of data across the two databases,
i.e. the database should not acquire any locks during the verification period. In-flight transactions must be
accounted for during the comparison.

5.2.8 Failback
Once the upgrade/migration takes place, a failback strategy must be in place to avoid any downtime and
data loss in case the new environment is not stable. Therefore, all transactions that have been processed
at the target (the new production database) need to be propagated to the original production database in a
consistent and manageable manner for a successful failback.

© 2009 GoldenGate Software, Inc.

8
 6. AVAILABLE SOLUTIONS:
8i or 9i ➔ 10g or 11g UPGRADE OR MIGRATION
Unload/ Export/ Transportable Backup/Roll Standby Databases
Scenario: GoldenGate
Load Import Tablespaces Forward Data Guard Streams

8i/9i ➔ 10g/11g Yes Yes Yes No No No1 Yes

8i/9i ➔ 10g/11g
Yes Yes No No No No1 Yes
cross platform
9i ➔ 10g/11g
Yes Yes Yes No No No1 Yes
RAC/ASM
8i ➔ 9i Yes Yes Yes No No No Yes
8i ➔ 9i
Yes Yes No No Yes2 No3 Yes
cross platform
Non-Oracle ➔
Yes No No No No No3 Yes
10g/11g

Downtime Weeks/Days Hours/Minutes Minutes/Seconds

Extended Downtime Real-Time

1. From Oracle 9.2 to Oracle 10g+ it may work for very limited data types and low-volume databases.
2. Requires transient logical standby and does not allow the use of standby systems for failover during the upgrade process.
3. May work for limited data types and low-volume databases.

Figure 3: An Overview of Downtime Categories for Different Solutions, Given Various Oracle Upgrade
and Migration Scenarios
There are a number of possible solutions that may be evaluated to help with a project to move onto 10g or 11g. The
figure above (Figure 3) depicts whether these solutions support different possible migration or upgrade scenarios,
along with associated time categories—from extended downtime to real-time switchover.
Those and several other common methods and solutions are summarized below with their respective advantages
and disadvantages, illustrating where significant shortfalls exist for environments seeking minimal downtime and high
availability:

6.1 Database Upgrade Assistant

This method only works for an in-place upgrade. This doesn’t solve the rolling upgrade scenario, nor does it
address platform migration.

6.2 Export/Import
A full export can be done from an 8i or 9i database, followed by an import into an existing vanilla 10g database.
For Oracle 10g to 11g upgrades you can use the Data Pump Export utility and import the data into an existing
vanilla 11g database using Data Pump Import. Oracle’s Export utility extracts data from the database and
writes it to a proprietary file (called an export dump file). Import processes the data and recreates all the
objects (tables, indexes, packages, etc.). Thus the time required for an export/import upgrade is high. With
Oracle 10g Data Pump you can parallelize the process more easily but the time required will be still significant.

© 2009 GoldenGate Software, Inc.

9
white paper

An export in FULL mode does have advantages in that it moves all database objects; it also supports both
upgrades and migrations. However, in a high availability environment, there are three significant disadvantages
to using Export/Import:
1. V
 olume-Time Relationship: The time required to extract the data is dependent on the size of the data.
Extracting and reinserting high volumes of data can take days or even weeks for large databases.
2. Ongoing Transactions: Ongoing changes are not captured by Export, nor is there a way to demarcate
the boundaries across SCNS that allow the incremental data to be re-exported. This effectively
implies application downtime during the export/import process.
3. Recoverability: Failures during export and/or import are not easy to recover from.

6.3 SQLPlus As Copy

Using this method, all schemas/tables can be captured from one database and recreated on another database
over a database link. An example of schema copy is as follows:
COPY FROM SCOTT/TIGER@LOCALDB TO SCOTT/TIGER@REMOTEDB
This method doesn’t scale nor can it be viewed as a serious way to move high volumes of data. It has a high
database impact, is unmanageable, and doesn’t support incremental data movement.

6.4 Transportable Tablespaces

Transportable tablespaces have been used effectively to minimize planned outages, since the time to transport
avoids extracting data and instead relies on integrating datafiles directly into the target database. It’s not a
complete solution, because it doesn’t address incremental data movement and also imposes restrictions on
the production application by rendering the user data to read-only workloads. Also, the method cannot be
used for migrations.
The high level flow of the upgrade process using transportable tablespaces is as follows:
1. Create a target database in 10g or 11g using install utilities. Also install any components and
applications as necessary. Pre-create all users, roles, Java packages, views, and additional objects
not picked up by transport.
2. Extract any unsupported objects using export or recreate object commands.
3. Make all non SYSTEM tablespaces read only.
4. Unplug the non SYSTEM tablespaces using the export utility – this extracts only the dictionary
metadata associated with the non SYSTEM tablespaces.
5. Transport the datafiles in the non SYSTEM tablespaces to the target database.
6. Plug the datafiles into the target database using the import utility in transportable mode.
7. Run manual verification tests to ensure that the data is synchronized with the primary.
This method can achieve the upgrade at disk speeds of file transfer; but the drawback is that the application
is inaccessible – so the downtime could be extended – and as previously noted, does not support migrations.

© 2009 GoldenGate Software, Inc.

10
 6.5 SQL*Loader
This method involves scanning all the tables and writing the data into flat files, and using the SQL loader
utility to reload the data into the target database. It is tedious, slow, and doesn’t address all database object
movement. It suffers from the same disadvantages as Export/Import and SQL Copy

6.6 Oracle Data Guard, Oracle Streams

The above two solutions do not support a rolling upgrade from 8i or 9i to 10g or 11g. For 10g to 11g
upgrades you can use a physical standby configuration but you have to use transient logical standby during
the upgrade process. In addition, during the upgrade process the standby system will not be available for
failover. Please note that there are some limitations with using logical standby particularly with data types.
(See References, 3 and 4).

6.7 The GoldenGate Zero-Downtime Upgrades Solution

Using its real-time, heterogeneous data movement technology GoldenGate offers Zero-Downtime Migration
and Upgrade solutions that address a minimal database downtime from 8i or 9i to 10g or 11g.
For example, the upgrade to Oracle 10g using GoldenGate consists of the following high-level steps:
1. Perform Initial Setup of a standby database running 8i or 9i software using an existing database
backup.
2. Upgrade the standby database to 10g.
3. Synchronize the standby database with the production database.
4. Test in active/live mode.
5. Switchover the application to the standby database.
6. Upgrade the primary database to 10g after comprehensive application testing at standby.
For a migration scenario to Oracle 10g, the high-level steps of GoldenGate’s Zero-Downtime Upgrades
solution are as follows:
1. Perform Initial Setup of a clone database running 8i or 9i software using an existing database backup
on the same platform.
2. Upgrade the clone database to 10g, advance compatibility to 10.0.0.0.
3. Create a vanilla 10g database on the new platform with 10.0.0.0 compatibility.
4. Move the data from the clone database to the new target database using Oracle’s Cross Platform
Transportable Tablespace feature.
5. Synchronize the new target database with the original 8i or 9i production database.
6. Test in active/live mode.
7. Switchover the application to the standby database.
8. Upgrade the primary database to 10g after comprehensive application testing at standby.
From here, Sections 7-8 describe the technologies that will be used to conduct the zero-downtime upgrade or
migration. Section 9 discusses the steps in greater detail. Since a migration is a more complicated procedure
than an upgrade, the detailed steps use a case study of migrating a 9i database on one platform to a 10g
database on another platform.
Please note that for upgrading or migrating to Oracle 11g, database versions prior to Oracle 9.2.0.4 require
the step to upgrade to Oracle 10g first before upgrading to 11g. For upgrading from 10g to 11g you can refer
to Oracle’s documented approach.3

© 2009 GoldenGate Software, Inc.

11
 Production database: Begin Instantiation: End Instantiation:
generating redo at database clock at database clock at
SCN 23.12345 SCN 23.12355 SCN 23.98745
white paper

Time

T1 T3 T100
7. OVERVIEW OF GOLDENGATE’S TECHNOLOGY COMPONENTS
There are a number of key technology components that comprise GoldenGate’s software platform and are
relevant to the subject at hand

Bi-Directional TDM Configuration

Source Target
Capture Source Trail Target Trail Delivery

Route
LAN/WAN/Web/IP

Delivery Target Trail Source Trail Capture

Compare & Verify

Figure 4: H
 igh-Level Architecture Diagram of Goldengate’s Software Products, Running
Bi-Directionally and Verifying Data Between Two Databases
Capture
7.1 Capture 2 3 6 7

GoldenGate Capture is a process that runs on the source database system.

®
It exp_norows.dmp
is multithreaded
in an Oracle
exp_xtts.dmp
RAC environment.
Dprod The process mines transactions from
Dprod backup the Oracle redo log
Dpitr and propagates transactions over
the9inetwork to an on-disk queue. Only committed transactions are 9i
/ Sun Solaris written
➔ 10gto the queues.
4 Sun Solaris
7.1.1 Initialization
GoldenGate Capture can be used either to initialize the target database
8 directly, or to start propagating
transactions against an existing target database from a given point.
Apply
11 12
7.1.2 Change Capture IGNORE = Y
10
Only table DML operations are captured from the source database, i.e. updates to indexes, system tables,
13
etc. are not captured.
Compare & Verify
5 Dtarget
10g Linux
7.2 On-Disk Queues or “Trail Files”
GoldenGate® Trail Files can be conceptualized as a persistent ordered set of committed transactions generated
by the GoldenGate Capture process. GoldenGate Trail Files describe DML operations (inserts, updates, and
deletes) along with transactional context as captured from the source database.
Capture
2 3 6 7
exp_norows.dmp
exp_xtts.dmp
Dprod Apply Dprod backup Dpitr
9i / Sun Solaris 9i ➔ 10g
16 4 Sun Solaris

8
Apply
© 2009 GoldenGate Software, Inc.
12
11 12
Failover IGNORE = Y
 7.3 Delivery (Apply)
The GoldenGate® Delivery process runs on the target system. It reads the captured data from the Trail Files
and applies the captured transactions at the target using dynamic SQL. To maintain synchronization between
the source and target databases, GoldenGate applies data changes to the target tables using native database
calls, statement caches, and local database access. To ensure data and referential integrity, GoldenGate
applies data changes in the transaction commit order that occurred at the source database.

7.4 GoldenGate Veridata®

GoldenGate Veridata is a stand-alone, high-speed comparison solution that identifies and reports data
discrepancies between two databases without interrupting ongoing business processes. It allows data
discrepancies to be isolated for testing and troubleshooting. GoldenGate Veridata is ideal for conducting data
validation after the rolling upgrade, once the source and target are fully operational and running different
versions of Oracle. It can also help to determine if a failback is needed, in case of any risky data anomalies.

8. OVERVIEW OF THE ORACLE CROSS PLATFORM TRANSPORTABLE

TABLESPACE FEATURE
The Cross Platform Transportable Tablespace procedure, to transport all non SYSTEM tablespaces, is
described below since it will be used as part of the rolling upgrade/migration process:
1. Ensure compatibility of source and target database is at 10.0.0.0 or higher.
2. Make all non SYSTEM tablespaces read only.
3. Unplug the non SYSTEM tablespaces using the export utility – this extracts only the dictionary
metadata associated with the non SYSTEM tablespaces.
4. Convert the datafiles associated with the non SYSTEM tablespaces using the RMAN CONVERT
functionality. If the cross platform transport is across the same endian system, then conversion may
not be required.
5. Transport the converted datafiles in the non SYSTEM tablespaces to the target database.
6. Plug the datafiles into the target database using the import utility in transportable mode.

© 2009 GoldenGate Software, Inc.

13
 Bi-Directional TDM Configuration
white paperSource Target
Capture Source Trail Target Trail Delivery

Route
LAN/WAN/Web/IP

Delivery Target Trail Source Trail Capture

9. DETAILED STEPS USING A PLATFORM MIGRATION CASE EXAMPLE
We use a case study of moving an Oracle 9i database on Sun Solaris to Oracle 10g on Linux and discuss the
detailed implementation steps. These steps can be simplified and adopted to accomplish a rolling upgrade.
Two scenarios are described next: A migration without the addition of a failback solution, and the same
migration with the addition of a failback. Compare & Verify

9.1 Migration without Failback

Capture
2 3 6 7
exp_norows.dmp
exp_xtts.dmp
Dprod Dprod backup Dpitr
9i / Sun Solaris 9i ➔ 10g
4 Sun Solaris

8
Apply
11 12
IGNORE = Y
10

13
Compare & Verify
5 Dtarget
10g Linux

Figure 5: Cross-Platform Migration (without Failback)

1. Address change management by restricting creation of newer packages, tablespaces, etc. during
the migration process. (Not depicted)
Capture
2. Start GoldenGate
2 Capture process (captures consistent
3 scn = Qscn) at 6the 7production database
Dprod. exp_norows.dmp
exp_xtts.dmp
3. Do a point-in-time recovery of an existing backup of Dprod until Qscn in a separate staging area. Call
Dprod Apply Dprod backup Dpitr
thisSolaris
9i / Sun database Dpitr. 9i ➔ 10g
16 4 Sun Solaris
4. Upgrade Dpitr to 10g on Solaris. Advance compatibility to 10.0 or higher.
5. Set up a vanilla 10g database on Linux. Call this database Dtarget.
8
Apply
6. Take a full export without rows at Dpitr. Call the generated export exp_norows.dmp. This will pick up
any objects that are not picked up as part 11of 12
the transportable tablespace export.
Failover IGNORE = Y
7. Unplug the user tablespaces from Dpitr 14using the Oracle Cross Platform Transportable
10 Tablespace
Capture
feature
13 using source side endian conversion. (Note the conversion would not be required if
the endian
Comparesystems
& Verify were the same.) This is the step that avoids any performance degradation and
5 Dtarget
does not require any quiescing at Dprod. This step will 10g create a small export dump file. Call this
Linux
exp_xtts.dmp.

© 2009 GoldenGate Software, Inc.

14
 Source Target
Capture Source Trail Target Trail Delivery

Route
LAN/WAN/Web/IP

Delivery Target Trail Source Trail Capture

8. Plug the set of tablespaces from Dpitr into Dtarget using the Cross Platform Transportable Tablespace
feature. Use the exp_xtts.dmp file created from Step 7. Note that the plugged in tablespaces are in
read-only mode.
Compare & Verify
9. Make the set of user tablespaces in Dtarget Read Write. (Not depicted)
10. Do a NOROWS import with IGNORE=Y option at Dtarget using the exp_norows.dmp dump file.
11. Start GoldenGate Apply process at Dtarget and synchronize up to the changes generated since
Qscn. Capture

12. If any data

2 types are not supported by Transportable
3 Tablespace or GoldenGate,
6 7 then do a special
export/import of these objects from Dprod to Dtarget. exp_norows.dmp
exp_xtts.dmp
13. Dprod
Use GoldenGate Veridata to verify
Dprodthatbackup
the data at Dprod and Dtarget is synchronized.
Dpitr
9i / Sun Solaris 9i ➔ 10g
14. Switchover the application from Dprod to Dtarget. (Not
4 depicted)
Sun Solaris
The above procedure offloads any quiescing, conversion work to a clone database, and takes advantage of
GoldenGate’s incremental real-time data capture and delivery to eliminate
8 the downtime to zero (excluding
the application switchover time). Apply
11 12
As an alternative to using the Cross Platform Transportable Tablespace feature in step 6, you can use a full
IGNORE = Y
10 not apply. And in step
export with data and import into a vanilla database, in which case steps 7, 8, and 9 do
10 you can13import the data as well as all database objects.
Compare & Verify
For upgrading from Oracle 10g to 11g you can use Data Pump Dtarget
5 Export in parallel to improve performance.
10g Linux

9.2 Migration with Failback

Capture
2 3 6 7
exp_norows.dmp
exp_xtts.dmp
Dprod Apply Dprod backup Dpitr
9i / Sun Solaris 9i ➔ 10g
16 4 Sun Solaris

8
Apply
11 12
Failover IGNORE = Y
14 10
Capture
13
Compare & Verify
5 Dtarget
10g Linux

Figure 6: Cross-Platform Migration (with Failback)

© 2009 GoldenGate Software, Inc.

15
white paper

1. Address change management by restricting creation of newer packages, tablespaces, etc. during
the migration process. (Not depicted)
2. Start GoldenGate Capture process (captures consistent scn = Qscn) at the production database
Dprod.
3. Do a point-in-time recovery of an existing backup of Dprod until Qscn in a separate staging area. Call
this database Dpitr.
4. Upgrade Dpitr to 10g on Solaris. Advance compatibility to 10.0 or higher.
5. Set up a vanilla 10g database on Linux. Call this database Dtarget.
6. Take a full export without rows at Dpitr. Call the generated export exp_norows.dmp. This will pick up
any objects that are not picked up as part of the transportable tablespace export.
7. Unplug the user tablespaces from Dpitr using the Oracle Cross Platform Transportable Tablespace
feature using source side endian conversion. (Note the conversion would not be required if
the endian systems were the same.) This is the step that avoids any performance degradation and
does not require any quiescing at Dprod. This step will create a small export dump file. Call this
exp_xtts.dmp.
8. Plug the set of tablespaces from Dpitr into Dtarget using the Oracle Cross Platform Transportable
Tablespace feature. Use the exp_xtts.dmp file created from Step 7. Note that the plugged in
tablespaces are in read-only mode.
9. Make the set of user tablespaces in Dtarget Read Write.
10. Do a NOROWS import with IGNORE=Y option at Dtarget using the exp_norows.dmp dump file.
11. Start the GoldenGate Delivery process at Dtarget and synchronize up to the changes generated since
Qscn.
12. If any data types are not supported by Transportable Tablespace or GoldenGate then do a special
export/import of these objects from Dprod to Dtarget.
13. After GoldenGate eliminates the lag between Dprod and Dtarget use GoldenGate Veridata to verify
that the data at Dprod and Dtarget is synchronized.
14. Start GoldenGate Capture process at Dtarget.
15. Switchover the application from Dprod to Dtarget. (Not depicted)
16. Start GoldenGate Apply on Dprod.

9.3 Failback Steps

During the real-time bi-directional data movement GoldenGate allows two databases to support transaction
processing. This makes failback a trivial process, if the new database is not stable:
1. Stop the application at Dtarget (the new primary) running 10g or 11g
2. Once GoldenGate has applied all transactions from Dtarget to Dprod, switch application to Dprod
3. Declare Dprod as the new primary

9.4 Partial Migration Using Bi-Directional Synchronization

In certain environments, it may be possible to migrate a limited number of applications over to the target
database on 10g or 11g while running the rest of the applications against the 8i or 9i source database. This
can be done based on logical application partitioning or geographical partitioning.

© 2009 GoldenGate Software, Inc.

16
 For example, if the application supports 100 bank branches, only one branch might be switched over to
the new database until the new 10g (or 11g) environment is deemed stable after appropriate testing and
validation. GoldenGate can be run bi-directionally to synchronize the data across the source and the target
systems, thus supporting multi-master database environments.

10. SUMMARY
To summarize, the key technical strengths of the solution are as follows:
½½ A rolling upgrade/migration – using two databases
½½ No instantiation using primary database – by offloading to a clone database
½½ Conversions – offloaded to a clone staging database
½½ Synchronize transactions across databases
½½ Verify data replication and transaction integrity – using GoldenGate Veridata
½½ Easy failover strategy – using a bi-directional GoldenGate configuration
Using GoldenGate’s software platform and Oracle’s Cross Platform Transportable Tablespace feature, a rolling
upgrade or even a migration can be done with zero database downtime and only very minimal application
switchover downtime. This combined solution enables companies to start utilizing new database features
provided in Oracle 10g and 11g releases without impacting business operations.

ABOUT THE AUTHOR

Alok Pareek is the Vice President of Technology for GoldenGate Software, previously working as its Software Architect
for High Availability solutions. Prior to joining GoldenGate, he had over 10 years of server development experience at
Oracle in the Recovery/High Availability area. His significant (patent-filed) contributions at Oracle include development
of the Cross Platform Transportable Tablespace feature (10g), Multi threaded redo generation (9i), Multiple block size
cache support (9i), and Whole database transport (10.2). He was responsible for the redo generation component of
the database from 8i to 10.2. He led the technical team responsible for high-speed data movement across platforms
as part of Oracle’s cost-cutting measures. He has presented at major industry events, numerous regional Oracle user
groups, and various user conferences on the topics of recovery and high availability. Alok holds a graduate degree in
Computer Science from Stanford University, where his research area was Database Systems.

REFERENCES
1. 1
 0.10 Upgrade with Logical Standby Database
(http://download.oracle.com/docs/cd/B19306_01/server.102/b14238/toc.htm)
Using a logical standby database enables you to accomplish upgrades for database software and patch sets with almost
no downtime. Note: This capability is not available for upgrading from Oracle9i to Oracle Database 10g.
2. D
 embeck, Chet. “Yahoo Cashes in on Ebay’s Outage,” E-Commerce Times, June 18, 1999
(http://www.ecommercetimes.com/perl/story/545.html)
3. Oracle 11g upgrade documentation (http://download.oracle.com/docs/cd/B28359_01/server.111/b28300/toc.htm)
4. O
 racle MetaLink note about 10g to 11g upgrade with the transient logical standby
(https://metalink.oracle.com/metalink/plsql/ml2_documents.showDocument?p_database_id=NOT&p_id=300479.1)
• L owell, David E.; Saito, Yasushi; Samberg, Eileen J. Devirtualizable Virtual Machines Enabling General, Single-Node,
Online Maintenance, Hewlett Packard Laboratories
• GoldenGate 8.0 Operations Guide for Windows and UNIX

© 2009 GoldenGate Software, Inc.

17
white paper

ABOUT GOLDENGATE SOFTWARE

GoldenGate Software Inc. is a leading provider of high availability and real-time data integration solutions for improving
the availability, accessibility, and performance of critical data across heterogeneous enterprise IT environments. More
than 500 customers worldwide, including Visa, Bank of America, US Bank, UBS, Sabre Holdings, DIRECTV, Comcast,
MGM Mirage, Chase Paymentech, AMD, Mayo Foundation, Retail Decisions, and Overstock.com, rely on GoldenGate
solutions. The company broadens its global market reach through strategic relationships with leading technology
vendors including ACI Worldwide, Amdocs, Business Objects, Cerner, Fujitsu, GE Healthcare, HP, IBM, Ingres,
Microsoft, Oracle, and Teradata.

CONTACT INFORMATION

GoldenGate Software, Inc.

301 Howard Street
San Francisco, CA 94105 USA
Tel: +1 415-777-0200
Email: info@goldengate.com

For a list of worldwide offices and for more company information, please visit:

www.goldengate.com

© 2009 GoldenGate Software, Inc.

18