My Personal Journey with Oracle's GoldenGate

Kevin Lidh
13 August 2010
Table of Contents
Things of note before we start that I learned along the way......................................................................3
Before we begin.........................................................................................................................................4
Install GoldenGate software.......................................................................................................................5
Create GoldenGate directories...................................................................................................................5
Create GoldenGate administrator account on source and target................................................................6
Configure and start the manager................................................................................................................7
The trials of my “initial load”....................................................................................................................8
Create extract group on the source side (Solaris).......................................................................................9
Create checkpoint table and replicat group on target side (RHEL5).........................................................9
Verify everything is working....................................................................................................................10
And the final test......................................................................................................................................10
In conclusion............................................................................................................................................12
Things of note before we start that I learned along the way
• The directories you create with the “create subdirs” command will be created in your present
working directory so be sure you are where you want them to go.
• Even though some documents will tell you to add the install directory (where you untar the
software) to your path, everything you run looks for files or directories from your present
working directory so you pretty much have to be sitting in the install directory to run anything
GoldenGate related.
• Ensure your ORACLE_SID is set before you do anything in GoldenGate.
• Your extract and replicat group names can only be up to 8 characters.
Before we begin
My goal was to migrate a schema from one platform and Oracle version to a completely different
platform and version of Oracle. I already had a Solaris x86-64 virtual machine and a CentOS 5 virtual
machine (think RHEL5) from earlier tests so I decided those would be my platforms. GoldenGate has
a 10g version for my Solaris environment so I installed Oracle 10gR2 (10.2.0.1.0) and upgraded it to
10.2.0.4.0. I created a database called ggtest and created a simple schema called appuser with a
mycustomer, oldcustomer, and custorder table as well as a foreign key constraint and a sequence.
[/u02/GoldenGate]
$ uname -a
SunOS soleil 5.10 Generic_142901-03 i86pc i386 i86pc
[/u02/GoldenGate]
$ sqlplus appuser/appuser

SQL*Plus: Release 10.2.0.4.0 - Production on Fri Aug 13 12:25:17 2010

Copyright (c) 1982, 2007, Oracle. All Rights Reserved.

Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

appuser@GGTEST> desc mycustomer

Name Null? Type
----------------------------------------------------------------- --------
--------------------------------------------
ID NOT NULL NUMBER
NAME VARCHAR2(255)
ADDRESS VARCHAR2(255)
PHONE VARCHAR2(100)
EMAIL VARCHAR2(255)
CUSTDATE DATE

appuser@GGTEST> desc oldcustomer

Name Null? Type
----------------------------------------------------------------- --------
--------------------------------------------
NAME VARCHAR2(255)
ADDRESS VARCHAR2(255)
PHONE VARCHAR2(100)
EMAIL VARCHAR2(255)
CUSTDATE DATE

appuser@GGTEST> desc custorder

Name Null? Type
----------------------------------------------------------------- --------
--------------------------------------------
CUSTID NOT NULL NUMBER
NUMWIDGETS NOT NULL NUMBER
ORDERDATE NOT NULL DATE

appuser@GGTEST>

From here, before I could really test GoldenGate, I needed to create a transaction generator. So I wrote
a quick-and-dirty transaction generator in Java using jdbc. This only creates one connection and
performs a random number of customer adds, customer removals, and customer orders. I also
populated the oldcustomer and mycustomer tables with 169 customers, all with unique names. So now
I'm ready to install and test GoldenGate.

Install GoldenGate software

The GoldenGate software is platform and Oracle version specific. I tried to run the 11g version of
software against the 10g version of the database and it complained.
I wound up installing (unzip and untar) on both servers in the /u02/GoldenGate directory so it would be
easy to find.

Create GoldenGate directories

I created the directories on the target side first.
[/u02/GoldenGate]
$ ggsci

Oracle GoldenGate Command Interpreter for Oracle

Version 10.4.0.19 Build 002
Solaris, x64, 64bit (optimized), Oracle 10 on Sep 23 2009 15:11:52

Copyright (C) 1995, 2009, Oracle and/or its affiliates. All rights reserved.

GGSCI (soleil) 1> create subdirs

Creating subdirectories under current directory /u02/GoldenGate

Parameter files /u02/GoldenGate/dirprm: created

Report files
Checkpoint files
Process status files
SQL script files
Database definitions files
Extract data files
Temporary files
Veridata files
Veridata Lock files
Veridata Out-Of-Sync files
Veridata Out-Of-Sync XML files
Veridata Parameter files
Veridata Report files
Veridata Status files
Veridata Trace files
Stdout files
/u02/GoldenGate/dirrpt: created
/u02/GoldenGate/dirchk: created
/u02/GoldenGate/dirpcs: created
/u02/GoldenGate/dirsql: created
/u02/GoldenGate/dirdef: created
/u02/GoldenGate/dirdat: created
/u02/GoldenGate/dirtmp: created
/u02/GoldenGate/dirver: created
/u02/GoldenGate/dirver/lock: created
/u02/GoldenGate/dirver/oos: created
/u02/GoldenGate/dirver/oosxml: created
/u02/GoldenGate/dirver/params: created
/u02/GoldenGate/dirver/report: created
/u02/GoldenGate/dirver/status: created
/u02/GoldenGate/dirver/trace: created
/u02/GoldenGate/dirout: created

GGSCI (soleil) 2>

Then I created the directories on the target side. Note the version of GoldenGate is different as the
versions of my databases are different.
[/u02/GoldenGate]
$ ggsci

Oracle GoldenGate Command Interpreter for Oracle

Version 10.4.0.19 Build 002
Linux, x64, 64bit (optimized), Oracle 11 on Sep 17 2009 23:51:28
Copyright (C) 1995, 2009, Oracle and/or its affiliates. All rights reserved.

GGSCI (centora.lidh.com) 1> create subdirs

Creating subdirectories under current directory /u02/GoldenGate

Parameter files /u02/GoldenGate/dirprm: created

Report files
Checkpoint files
Process status files
SQL script files
Database definitions files
Extract data files
Temporary files
Veridata files
Veridata Lock files
Veridata Out-Of-Sync files
Veridata Out-Of-Sync XML files
Veridata Parameter files
Veridata Report files
Veridata Status files
Veridata Trace files
Stdout files
/u02/GoldenGate/dirrpt: created
/u02/GoldenGate/dirchk: created
/u02/GoldenGate/dirpcs: created
/u02/GoldenGate/dirsql: created
/u02/GoldenGate/dirdef: created
/u02/GoldenGate/dirdat: created
/u02/GoldenGate/dirtmp: created
/u02/GoldenGate/dirver: created
/u02/GoldenGate/dirver/lock: created
/u02/GoldenGate/dirver/oos: created
/u02/GoldenGate/dirver/oosxml: created
/u02/GoldenGate/dirver/params: created
/u02/GoldenGate/dirver/report: created
/u02/GoldenGate/dirver/status: created
/u02/GoldenGate/dirver/trace: created
/u02/GoldenGate/dirout: created

GGSCI (centora.lidh.com) 2>

Create GoldenGate administrator account on source and target

I won't go into a lot of detail as to missing steps but I did miss some steps along the way. Also, since
the commands and results are essentially the same on both source and target, I'm only going to show
the first but remember to run them on both.
sys@GGREP> create tablespace ggs_data datafile '/u02/oradata/ggrep/ggs_data01.dbf' size
200M;

Tablespace created.

sys@GGREP> create user ggs_owner identified by ggs_owner default tablespace ggs_data

temporary tablespace temp;

User created.

sys@GGREP> grant connect,resource to ggs_owner;

Grant succeeded.

sys@GGREP> grant select any dictionary, select any table to ggs_owner;

Grant succeeded.

sys@GGREP> grant create table to ggs_owner;

Grant succeeded.

sys@GGREP> grant flashback any table to ggs_owner;

Grant succeeded.

sys@GGREP> grant execute on dbms_flashback to ggs_owner;

Grant succeeded.

sys@GGREP> grant execute on utl_file to ggs_owner;

Grant succeeded.

sys@GGREP>

The missing steps were to turn the recyclebin off and bouncing the databases and then the following
scripts and command:
SQL> @$GGATE/marker_setup.sql
SQL> @$GGATE/ddl_setup.sql
SQL> @$GGATE/role_setup.sql
SQL> grant GGS_GGSUSER_ROLE to ggate;
SQL> @$GGATE/ddl_enable.sql

At this point, in both databases (although I'm sure its only necessary on the source), I turned on
supplemental logging.
sys@GGTEST> alter database add supplemental log data (all) columns;

Database altered.

sys@GGREP> alter database add supplemental log data (all) columns;

Database altered.

Configure and start the manager

As noted in my initial notes, you have to do this from the root directory of the GoldenGate install. I
first set up the source side.
[/u02/GoldenGate]
$ ggsci

Oracle GoldenGate Command Interpreter for Oracle

Version 10.4.0.19 Build 002
Solaris, x64, 64bit (optimized), Oracle 10 on Sep 23 2009 15:11:52

Copyright (C) 1995, 2009, Oracle and/or its affiliates. All rights reserved.

GGSCI (soleil) 1> edit params mgr

"/u02/GoldenGate/dirprm/mgr.prm" 3 lines, 106 characters

port 7809
userid ggs_owner, password ggs_owner
purgeoldextracts /u02/GoldenGate/dirdat/ex, usecheckpoints

GGSCI (soleil) 2> start manager

Manager started.
GGSCI (soleil) 3> info manager

Manager is running (IP port soleil.7809).

GGSCI (soleil) 4>

I then set up the target side.

[/u02/GoldenGate]
$ ggsci

Oracle GoldenGate Command Interpreter for Oracle

Version 10.4.0.19 Build 002
Linux, x64, 64bit (optimized), Oracle 11 on Sep 17 2009 23:51:28

Copyright (C) 1995, 2009, Oracle and/or its affiliates. All rights reserved.

GGSCI (centora.lidh.com) 1> edit params mgr

port 7809
userid ggs_owner, password ggs_owner
purgeoldextracts /u02/GoldenGate/dirdat/ex, usecheckpoints

GGSCI (centora.lidh.com) 2> start manager

Manager started.

GGSCI (centora.lidh.com) 3> info manager

Manager is running (IP port centora.lidh.com.7809).

GGSCI (centora.lidh.com) 4>

So at this point, the managers are running and the communication port is 7809 on both nodes.

The trials of my “initial load”

There is a GoldenGate method of doing an initial load of the data. Basically it's an initial state of the
source. You start with an empty schema (no rows) and then configure a one time load. I created the
schema but I couldn't get the one time load to work. After several failed attempts, I gave up. I wound
up just setting up a database link and moving the data that way. I've read reports of other people having
issues with the GoldenGate method but I'm pretty sure I did something wrong.
All things considered, there are many methods for doing an initial load of data whether you going to
replicate with Oracle Streams, GoldenGate, or SharePlex. My method wasn't necessarily wrong but in
a large enterprise environment, it probably isn't the most efficient. I'll work on data load scenarios
later.
So at this point, I now have my data in sync but want to start my application.
Create extract group on the source side (Solaris)
There are certain lines that require a semicolon and some that don't. Don't always trust what you read
on the internet, definitely check the documentation.
GGSCI (soleil) 1> info all

Program Status Group Lag Time Since Chkpt

MANAGER RUNNING

GGSCI (soleil) 2> add extract ext1, tranlog, begin now

EXTRACT added.

GGSCI (soleil) 3> add exttrail /u02/GoldenGate/dirdat/lt, extract ext1

EXTTRAIL added.

GGSCI (soleil) 4> edit params ext1

-- extract group --
extract ext1
-- connection to database --
userid ggs_owner, password ggs_owner
-- hostname and port for trail
rmthost centora, mgrport 7809
-- path and name for trail --
rmttrail /u02/GoldenGate/dirdat/lt
-- DDL support
ddl include mapped objname appuser.*;
-- DML
table appuser.*;

Create checkpoint table and replicat group on target side

(RHEL5)
I had to create a checkpoint table which is how GoldenGate tracks where it is in the replication process.
Streams uses the SCN.
GGSCI (centora.lidh.com) 3> edit params ./GLOBAL

ggschema ggs_owner
checkpointtable ggs_owner.checkpoint

GGSCI (centora.lidh.com) 5> dblogin userid ggs_owner

Password:
Successfully logged into database.

GGSCI (centora.lidh.com) 6> add checkpointtable ggs_owner.checkpoint

Successfully created checkpoint table GGS_OWNER.CHECKPOINT.

GGSCI (centora.lidh.com) 7> add replicat rep1, exttrail

/u02/GoldenGate/dirdat/lt,checkpointtable ggs_owner.checkpoint
REPLICAT added.

GGSCI (centora.lidh.com) 8> edit params rep1

-- replicat group --
replicat rep1
-- source and target definitions --
assumetargetdefs
-- target database login --
userid ggs_owner, password ggs_owner
-- file for discarded transaction --
discardfile /u02/GoldenGate/discard/rep1_discard.txt, append,megabytes 10 --NOTE: Have
to create the directory first. Now it's running!
-- DDL support --
DDL
-- specify table mapping --
map appuser.*, target appuser.*;

Verify everything is working

The pro to using GoldenGate is that it is this simple to set up. When we set up a similar migration
environment using Oracle Streams to go from Solaris SPARC to AIX, it was much more complex. The
pro to Streams is that if you have an Enterprise license, there's no extra cost. GoldenGate has a pretty
steep license cost. Anyway, let's ensure it's running.
GGSCI (soleil) 3> info all

Program Status Group Lag Time Since Chkpt

MANAGER RUNNING
EXTRACT RUNNING EXT1 00:00:00 00:00:00

GGSCI (centora.lidh.com) 3> info all

Program Status Group Lag Time Since Chkpt

MANAGER RUNNING
REPLICAT RUNNING REP1 00:00:00 00:00:02

And the final test

The real test to see if this environment is working is to modify data. So I kicked off my transaction
generator and let it run for five or so minutes. This is the last of the log file:
appQuery: generating report
-- there are 157 current customers --
-- there are 181 potential customers --

custAdd: Adding Preston to customer list

custAdd: Adding Lynch to customer list
custAdd: Adding Chapman to customer list
Control-C caught. Shutting down ggTransDriver...
custRemove: Removing Murray from customer list
custRemove: Customer Murray has 0 existing orders
custOrder: Burks ordered 69 widgets
custOrder: Perry ordered 10 widgets
ended transaction loop.

So before the transaction loop ended, the customer Perry ordered 10 widgets. First, lets ensure that this
customer order is actually in the source database.
appuser@GGTEST> select * from custorder where custid = (select id from mycustomer where
name = 'Perry');

CUSTID NUMWIDGETS ORDERDATE

---------- ---------- --------------------
20140 10 13-AUG-2010 10:28:36

appuser@GGTEST>

So the record is there. Now let's see if the record has made it to the target database.
appuser@GGREP> select * from custorder where custid = (select id from mycustomer where name
= 'Perry');

CUSTID NUMWIDGETS ORDERDATE

---------- ---------- --------------------
20140 10 13-AUG-2010 10:28:36

appuser@GGREP>

And it is. So my first test implementation of GoldenGate has been a success. I can run my transaction
generator for hours and the data will move to a different Oracle version on a different platform. If you
didn't notice, when I configured GoldenGate, I also told it to migrate DDL. So let's run a quick little
test to see if DDL makes it over.
appuser@GGTEST> create table newtable (col1 varchar2(20));

Table created.

appuser@GGTEST> insert into newtable values ('Hello new table');

1 row created.

appuser@GGTEST> commit;

Commit complete.

appuser@GGTEST> select * from newtable;

COL1
--------------------
Hello new table

appuser@GGTEST>

Then I blindly query the target side:

appuser@GGREP> select * from newtable;

COL1
--------------------
Hello new table

appuser@GGREP>

So I was able to create a brand new table on the source side and add a record and it was all replicated to
the target side. Another success.
In conclusion
So for now I've proved that I can configure GoldenGate and do simple table migration. On my “to do”
list for future GoldenGate testing:
1. Expand this test to include different data types including longs and LOBs. Longs are every
migration's enemy.
2. Get a successful GoldenGate initial load done
3. Do a migration of a database that is in use the entire time. This would be more realistic of
environments I could come across.
4. Do a test in other operating systems like Solaris SPARC, HP-UX, AIX and even VMS.