PTS OGG Workshop v4

PlatformTechnology Solutions
Oracle GoldenGate Workshop

Version 4.0
1
Oracle Platform Technology Solutions
Copyright 2010 Oracle and/or its affiliates. All rights reserved..
Authors:
Sean Stacey
Contributors / Reviewers:
Loren Penton
Robert Mackowiak
Rick Pandya
Viksit Puri
Efrain Sanchez
Oracle Corporation
World Headquarters
500 Oracle Parkway
Redwood Shores, CA 94065
U.S.A.
Worldwide Inquiries:
Phone: +1.650.506.7000
Fax: +1.650.506.7200
www.oracle.com
Oracle Corporation provides the software
that powers the Internet.
Oracle is a registered trademark of Oracle Corporation. Various
product and service names referenced herein may be trademarks
of Oracle Corporation. All other product and service names
mentioned may be trademarks of their respective owners.
Copyright 2010 Oracle Corporation
All rights reserved.
2
Oracle PTS GoldenGate Workshop

Version 4.0
Contents
Workshop Overview................................................................................................................................ 4
Assumptions........................................................................................................................................... 4
Workshop Prerequisites.......................................................................................................................... 4
Special Credits........................................................................................................................................ 4
Products We Will Be Using In This Workshop........................................................................................ 5
Important Information.............................................................................................................................. 5
Accounts and Passwords........................................................................................................................ 5
Lab 1: Getting Started........................................................................................................................... 6

Lab 2: Installing and Configuring Oracle GoldenGate Software........................................................... 22
Lab 3: Setting Up the Initial Load Process............................................................................................ 45
Lab 4: Setting Up One-way Replication................................................................................................ 68
Lab 5: Setting Up Bi-Directional Replication......................................................................................... 82
Lab 6: Leveraging Oracle GoldenGate Transformations...................................................................... 92
Lab 7: Leveraging Oracle GoldenGate for Filtering..............................................................................102
Appendix A: Useful Information............................................................................................................112

Appendix B: Schema Definitions for the Workshop..............................................................................113
Appendix C: Troubleshooting................................................................................................................116
Appendix D: Useful GGSCI Commands............................................................................................... 117
3
Overview
Oracle GoldenGate is a very powerful heterogeneous database replication utility which is not only capable
of supporting realtime Replication across different Oracle database versions and platforms but also across
database products.
This workshop will be focusing on demonstrating the Heterogeneous capabilities of replication between
MySQL and an Oracle Database.
This is a hands-on-workshop, and as such you will be guided through the process of installation,
configuration and execution of Oracle GoldenGate running with both Oracle and MySQL with the intent of
getting you quickly started on using these products.
Assumptions
This workshop is intended to teach someone who has no prior knowledge or experience in using Oracle
GoldenGate.
It is assumed that the code samples supplied within this workshop are for demonstration purposes only
and as such are not guaranteed nor supported for production deployment
It is not the intent of this workshop to make someone proficient in either the Oracle RDBMS nor the
MySQL database.
Workshop Pre Requisites
Basic skills in Linux operation as well as editing text files in Linux.
Understanding of the SQL language is necessary.
Special Credits
Loren Penton - Principal Member of Technical Staff, Oracle GoldenGate development. For Guidance
and Assistance in content creation. Loren built an Oracle GoldenGate workshop kit
upon which this workshop has been based.
4
Products We Will Be Using In This Workshop
Oracle GoldenGate on Oracle version 11.1.1.0.0
Oracle GoldenGate on MySQL version 11.0
Oracle VirtualBox Version 3.2.8
Oracle Enterprise Linux 5 Update 2, 32bit
Oracle Database 11g Release 2, Enterprise Edition, Version 11.2.0.1
Oracle MySQL 5.1 with the InnoDB engine
Important Information About This Workshop
GoldenGate on MySQL v11 software is pre-G.A. at the time of writing this workshop.
Both of the Oracle and MySQL Databases have already been installed and have been preset to startup
at boot time.
The GoldenGate Reference Guide is preloaded in the Virtual Environment in PDF format.
A shortcut to the Oracle Online Documentation Library for Database 11g Release 2 is available as a
bookmark in the browser.
A file containing some useful tips on using MySQL is available on the desktop
Our workshop will be using a basic database schema representative of a banking OLTP system.
Accounts and Passwords used in this workshop
Account
Password
Type
Type
root
oracle
OS
Linux Super User account
oracle
oracle
OS
Linux Oracle Database owner
mysql
oracle
OS
Linux MySQL Database owner
mygg
oracle
OS
GoldenGate MySQL user
orgg
oracle
OS
GoldenGate Oracle user
orgg
orgg
DB
Oracle Database user account for GoldenGate
mygg
mygg
DB
MySQL Database user account for GoldenGate
5
Lab 1:
Getting Started
Everything has been said before, but since nobody listens we have
to keep going back and beginning all over again."
- Andre Gide
Overview
There a series of setup steps we will need to complete in order to get our labs working. During
this first Lab we will be setting up both of the databases that we will be using for the remainder
of this workshop. We will start with the Oracle database first and then the MySQL database.
Upon completion of this lab we will have both databases created. The Oracle database objects
will be empty while the MySQL database objects will not only be populated with seed data but
we will also have a stored procedure ready for generating our workload as necessary.
6
1.1
Log into your virtual machine as the user oracle with the password oracle. Disregard the
following popup window...
Note: This warning message is due to directory permissions that have been explicitly set
for ease of use in this workshop. This is not an artifact of using GoldenGate nor from any
other software being used in this workshop.
1.2
Once you have logged into the machine, if there is not already a Terminal window open, go
ahead and open a new Terminal window by double-clicking the Terminal Icon in your desktop:
1.3
Change to the orgg user with the password oracle as follows:

[oracle@localhost ~] $ su orgg
Password: oracle
7
1.4
View the Environment variables set for this account session:

[oracle@localhost ~] $ env
Note: The Oracle GoldenGate user (orgg in this case) must be set up to include the
ORACLE_SID and ORACLE_HOME environment variables. Also the
LD_LIBRARY_PATH should be set to include the $ORACLE_HOME/lib directory.
8
1.5
Make sure you can connect into the Oracle database and also see whether it is running by
logging into the database:
[orgg@localhost ~] $ sqlplus '/ as sysdba'
sql> select instance_name, status, host_name, version
from v$instance ;
1.6
In order for GoldenGate to capture data changes from Oracle's redo logs for updates to Primary
Keys and chained rows, we will need to enable Supplemental Logging in the Oracle database.
This is done at the database level while connected as a database user with the ALTER
SYSTEM privileges. This can be done using the following commands (while still logged in as
the SYS user in this case):
SQL>
alter database add supplemental log data ;
SQL>
alter system switch logfile ;
SQL>
SQL>
select supplemental_log_data_min from v$database
9
1.7
While still logged in as the SYS user we will now go ahead and create the Oracle database user
orgg with a password orgg. This is the Oracle GoldenGate user that we will be using for the
remainder of this workshop.
Note: For ease of use during this workshop we will be granting DBA prvileges to the orgg
user. This is NOT a best practice nor do we endorse this in a production
environment.
SQL>
2
create user orgg identified by orgg

default tablespace users quota unlimited on users ;
SQL>
grant connect, resource, dba to orgg ;
SQL>
exit
10
1.8
Now we will login to the Oracle database as the newly created orgg user with the password of
orgg. Note that at this stage orgg does not have any database objects defined.
[oracle@localhost ~]$ sqlplus orgg/orgg
SQL> select table_name from user_tables ;
SQL> @scripts/orgg_ddl.sql
SQL> select table_name from user_tables ;
11
1.9
If everything looks okay let's go ahead and setup the MySQL database as well as the mygg
user account that we will be using for this workshop:
Open a new terminal window by double-clicking the Terminal icon on the desktop. Then change to the
mysql user:
[oracle@localhost ~]$ su mysql
Password: oracle
[mysql@localhost ~]$ cd /usr/local/mysql
[mysql@localhost mysql]$
bin/mysqladmin version
This should return information about the MySQL database as follows:
12
1.10
Now that we have established MySQL is running, let's go ahead and exit from the mysql operating
system account and change to the mygg user with the password oracle.
As this is the first time we are using the mygg Linux user account, let's check the shell environment
settings for this user:
[mysql@localhost mysql] $ exit
[oracle@localhost ~]$ su mygg
Password: oracle
[mygg@localhost ~]$ env
Note: The LD_LIBRARY_PATH for the mygg OS user is different from the setting for the
orgg user. In this case we are pointing the LD_LIBRARY_PATH to the Oracle
GoldenGate on MySQL directory that we will be installing .
13
1.11
Once we have changed users, we will once again access MySQL. However this time (only) we
are going to use the special root / admin account so that we can create our Database and
associated MySQL user account that we will be using in our workshop.
[mygg@localhost ~]$ mysql -h 127.0.0.1 -u root -p
Enter password: oracle
mysql>
show databases ;
14
1.12
We are now ready to create the MySQL database called MYBANK and the MySQL mygg user account
(with password mygg). This will be the main account we will be using through out the remainder of our
labs. All SQL scripts and GoldenGate operations that touch MySQL will be using this Database and
account combination.
While still logged in as the root user enter the following:
mysql> create database MYBANK ;
mysql> grant all on MYBANK.* to mygg@localhost identified by "mygg" ;
mysql> show databases ;
mysql> exit
15
1.13
We are now ready to create the MYBANK database objects and populate the seed data that will be
used for simulating our transaction workload. In addition to the tables and data we will also be creating
a stored procedure called simtxn which can be called at will to simulate our workload. This step is
contained in several mysql scripts it is recommended that you take a look at these scripts before
running them so you have a basic understanding of the objects we will be creating, as well as how they
are created
[mygg@localhost ~]$ mysql MYBANK -u mygg -p
Enter password: mygg
mysql>
source scripts/mybank_ddl.sql
Note: The "ERROR 1051 (42502): Unknown Table" messages are nothing to be
alarmed about. These are due to the tables not existing prior to running the script and are
therefore expected in this situation.
16
1.14
Now confirm that the tables have been correctly created.

mysql>
show table status ;
mysql>
delimiter //
Note: It is important to ensure that the "Engine" column is listed as being "InnoDB" ,
GoldenGate Extract replication cannot be guaranteed with the MySQL default MyISAM
engine.
17
1.15
The next step is to seed the MYBANK database with data that we will be using for the labs in
our workshop. This is done with a script called mybank_seed.sql which is located in the:
/home/mygg/scripts/mybank_seed.sql
mysql>
source scripts/mybank_seed.sql
Note: You can expect the output from this script to scroll off the screen, there will also be
many warning messages which you can ignore.
18
1.16
Once the script to seed the database has completed running, we need to create the transaction
simulator. In this case we are using is a MySQL stored procedure- simtxn. The source code
for the simtxn stored procedure is located in the file: /home/mygg/scripts/mybank_simtxn.sql
After creating the stored procedure we will execute it once:
mysql>
source scripts/mybank_simtxn.sql
mysql>
call simtxn();
->
//
Note: Calling the simtxn stored procedure can take several minutes depending on the performance
of your laptop. Do not be alarmed...
1.17
Let's check and see how things look now there are two ways to see how the tables were populated:
mysql>
select count(*) from ACCOUNT //
mysql>
select count(*) from ACCOUNT_TRANS //
19
1.18
We will also check the other tables in the newly created and populated MYBANK database.
mysql>
select count(*) from BRANCH //
mysql>
select count(*) from BRANCH_ATM //
mysql>
select count(*) from TELLER //
mysql>
select count(*) from TELLER_TRANS //
Note: The following ROW COUNTS should match:

- ACCOUNT = 1600
- BRANCH = 40
- TELLER = 800
20
1.19
The last step in this lab is to enable checkpointing for GoldenGate in MySQL.
mysql> source scripts/chkpt_mysql.sql
mysql> show tables //
Lab 1 Summary
You have now completed the following preparation tasks for our workshop:
The Oracle database schema and objects have been created
The Oracle database has been configured for Supplemental Logging
The MySQL database and objects have been created
The transaction simulation process for generating the workload has been installed and setup
The MySQL database has been set up for GoldenGate checkpointing via the database
21
Lab 2:
Installing and Configuring Oracle GoldenGate Software
Overview
In this lab we will install and configure the Oracle GoldenGate software. We will be creating two
physically separated GoldenGate environments, as we will be replicating between two
completely different databases. Take note, the binary bits are completely different for each
database platform being replicated, and because of this, we will be installing the Oracle
GoldenGate software into two physically seperated directories for ease of use. The software for
Oracle GoldenGate on Oracle will be downloaded from OTN, while the software for Oracle
GoldenGate on MySQL has been already pre-staged for us.
Upon completing this lab exercise our systems will be running Oracle GoldenGate on Oracle
version 10.4 and Oracle GoldenGate on MySQL version 11.
Note: One of the most significant features of the Oracle GoldenGate software is it's portable
packaging model and ease of installation, that should become apparant by the end of this lab.
The process of installing Oracle GoldenGate consists of simply unpacking a file into a directory.
22
2.1
If not already logged into your VirtualBox system as the Oracle user do so now. From a
terminal window - change to the mygg user and then navigate to the /u01/mysqlgg directory.
This is a predefined directory that we will be using to run Oracle GoldenGate on MySQL from.
[oracle@localhost ~]$
su mygg
Password: oracle
[mygg@localhost ~]$
cd /u01/mysqlgg
[mygg@localhost mysqlgg]$
pwd
ls
cat my.ini
Note: The my.ini file is a mysql configuration file. It is typically used by the MySQL
engine to set systemwide defaults when starting the MySQL database. In this case we
are creating a local copy of this file to be used by Oracle GoldenGate on MySQL.
It is necessary for the Oracle GoldenGate on MySQL engine to know where the (redo)
log files are located for it be able to capture database changes. Ordinarily we might
consider placing a soft-link to the version of the file owned by the mysql operating
system user.
23
2.2
The zipfile: ggs_redhatAS40_x86_MySQL_32bit_vV10M2_045.zip contains a tarball of the

Oracle GoldenGate software, which will need to be un-tarred.
[mygg@localhost mysqlgg]$ unzip
ggs_redhatAS40_x86_MySQL_32bit_vV10M2_045.zip
[mygg@localhost mysqlgg]$ tar xvf
ggs_redhatAS40_x86_MySQL_32bit_vV10M2_045.tar
2.3
Following the unzip operation, you should see 51 items in this directory.
ls |wc -l
24
2.4
We are almost done with the installation of Oracle GoldenGate on MySQL. To verify that Oracle
GoldenGate is correctly installed and working run the following commands.
ldd ggsci
ldd mgr
Note: the Linux ldd (aka List Dynamic Dependencies) utiliity will return an error if any
required libraries are missing. These errors must be corrected before continuing as it
indicates that the Oracle GoldenGate software is not correctly installed/configured.
The $LD_LIBRARY_PATH environment variable is what defines the location of these "lib"
files.
25
2.5
Now we just need to create the subdirectories for the environment. This directory structure is
used by the Oracle GoldenGate and is necessary for Oracle GoldenGate to readily locate the
files and objects used for runtime configuration and replication. Creating the directory structure
is easily done using the GoldenGate Software Command Interface (ggsci) utility:
GGSCI
> create subdirs
GGSCI
> exit
./ggsci
26
2.6
Oracle GoldenGate on MySQL is now installed.

We should also be able to see these directories when we are back at the shell prompt. In this
case we should be in /u01/mysqlgg/ directory and logged in as the mygg user:
ls -ltr
27
2.7
The newly created dir* directories will be empty other than the /dirver directory.
We can easily verify this by issuing the following Linux command while logged in as the mygg
user:
ls dir*
28
2.8
As our final step for installing the Oracle GoldenGate on MySQL we will now create the
GLOBALS parameter file.
The GLOBALS parameter file is used to set instance-wide parameters for the entire Oracle
GoldenGate Instance. An alternative to using the GLOBALS parameter file is to set parameters
at runtime for a specific GoldenGate process.
The GLOBALS parameter file is located in the directory in which Oracle GoldenGate software
for MySQL is installed in this case it is: /u01/mysqlgg/ We will be adding a single line-entry.
Note: You can still use the vi editor if you would prefer.
vi GLOBALS
Add the following line to the file:

checkpointtable MYBANK.GGS_CHECKPOINT
We have now completed the installation of Oracle GoldenGate on MySQL.
29
2.9
Now that we have completed the installation of Oracle GoldenGate on MySQL, we need to
install the Oracle GoldenGate on Oracle software.
For this step we will be downloading the software from the Oracle Technology Network. The
quickest way to do this is to startup the firefox browser from the Linux desktop by doubleclicking the firefox icon.
The default homepage for the firefox browsers in this workshop have been confiugured to open
at the Oracle GoldenGate landing pad on Oracle Technology Network (aka OTN). If you
experience any difficulties in accessing the page the url is :
http://www.oracle.com/technology/products/goldengate/index.html
Click on the download link highlighted in blue
30
2.10
On the next page it is easy to slip up and inadvertently choose the Linux 64Bit download under
the popular download section, however we will be using the Linux 32bit product.
This is available via the E-Delivery.com link at the bottom of the page (Highlighted in Blue):
2.11
This will bring you to the following screen where you will need to click the "Continue" link
(unless you wish to choose a language different from the default setting of English):
31
2.12
Ensure you answer all the fields correctly in the following screen:
2.13
Scroll down to the bottom of the same screen to read and ensure you are in compliance with
the Export Restrictions. Only after checking both of the boxes can you click on the "Continue"
button.
32
2.14
This will bring you to the following screen:

- In the Product Pack drop-down choose "Oracle Fusion Middleware"and
- In the Platform drop-down choose "Linux x86"
and
- click go.
This will list a number of Oracle GoldenGate products.
2.15
Scroll to bottom of the list for the product pack we will be using Oracle GoldenGate on Oracle
v11.1.1.0.0 Media Pack for Linux x86 B60214-01.
Check the radio button in this row and click the "Continue" button at the bottom of the page.
33
2.16
This will bring you to the following screen where you will check the radio-button alongside
"Oracle GoldenGate V11.1.1.0.0 for Oracle 11g on Linux x86 V22228-01 58M"
2.17
Click the Save button on the pop-up that appears. (this will save the file to your Linux Desktop)
34
2.18
While saving the file you should then see the following download dialogue:
Wait for this to complete.
2.19
This will place the following zip file on your Linux desktop
35
2.20
You now have a local copy of the Oracle GoldenGate on Oracle software package. However
the zipfile has been downloaded into the local user account of oracle and you will need this to
be placed in the directory that will be used for the rest of our workshop for managing the
replication for the Oracle Database.
Note: In our workshop environment, we will be performing these initial file-copy steps
using the Linux root (super-user) account to overcome permission restrictions. Hence
this step would not be necessary if the download was performed by the orgg Linux user
in the first place.
After opening a terminal:

su -
Password: oracle
[root@localhost ~]#
cd /u01/oraclegg/
[root@localhost oraclegg]#
mv ~oracle/Desktop/V22228-01.zip .
ls -l
chown orgg.orgg V22228-01.zip
ls -l
36
2.21
Now that the file is in the correct location for our labs and owned by the orgg Linux user, we
can now exit from the root user account. We will commence the Oracle GoldenGate on Oracle
install starting with unzipping and then untarring the newly downloaded zip file while logged in
as the orgg Linux user.
[root@localhost ~]#
exit
su orgg
Password: oracle
[orgg@localhost ~]$
cd /u01/oraclegg/
[orgg@localhost oraclegg]$
ls -ltr
unzip V22228-01.zip
ls -ltr
tar xvf
ggs_Linux_x86_ora11g_32bit_v11_1_1_0_0_078.tar
or another way:
tar xvf *.tar
37
2.22
If everything has run through correctly you should now have 82 objects in your "/u01/oraclegg"
directory. We can check this using the following command:
ls |wc -l
38
2.23
Verify that the software for Oracle GoldenGate on Oracle is correctly installed by issuing the
Linux ldd (list dynamic depencies) command on Oracle GoldenGate programs:
ldd ggsci
ldd mgr
Note: the use of the $ORACLE_HOME/lib directory (you may recall this is the taken from
the $LD_LIBRARY_PATH environment setting for the orgg Linux user account. The ldd
command can be run on other Oracle GoldenGate programs including replicat and mgr if
desired.
If there are any errors displayed here, they will need to be corrected before proceeding.
39
2.24
We will now be creating the subdirectories for the Oracle GoldenGate on Oracle software. Just
as in the installation of the Oracle GoldenGate on MySQL this is done through the GoldenGate
application itself.
While logged in as the orgg user, ensure you are in the /u01/oraclegg/ directory:
GGSCI
>
create subdirs
GGSCI
>
exit
./ggsci
40
2.25
The newly created sub-directories can now be seen from the Linux shell. Also, all of the dir*
directories will be empty at this stage other than the /dirver subdirectory.
We can easily verify this by running the Linux "ls dir*" command while logged in as the orgg
Linux user.
ls -ltr
ls dir*
41
2.26
We will now create a file called the GLOBALS parameter file.

The GLOBALS parameter file is used to set instance-wide parameters for the entire Oracle
GoldenGate Instance. An alternative to using the GLOBALS parameter file is to set parameters
at runtime for a specific GoldenGate process.
The GLOBALS parameter file is located in the directory in which Oracle GoldenGate softare is
installed in this case it is: /u01/oraclegg/ We will be adding a single line-entry.
Also we are going to use an alternative editor to vi this time. The gedit editor is a graphical
editor available in the Linux Gnome desktop.
Note: You can still use the vi editor if you would prefer.
gedit GLOBALS
Add the following line to the file:

checkpointtable orgg.GGS_CHECKPOINT
42
2.27
Save the file and exit the gedit session. The newly created GLOBALS file should be the very
last file added to the /u01/oraclegg/ directory:
ls -ltr
43
2.28
We will now add the Oracle GoldenGate checkpoint table to the Oracle database. In our last
few steps we added the GLOBALS parameter file to our Oracle GoldenGate on Oracle
instance. The single parameter we have defined is used to set the name of the Oracle
GoldenGate Checkpoint Table.
Checkpoints are used during the online change synchronization to store the current read and
write position of a process. Checkpoints ensure that data changes marked for synchronization
are extracted, and they prevent redundant extractions. They provide fault tolerance by
preventing the loss of data should the system, the network, or a GoldenGate process need to
be restarted.
This is a more durable approach as well as a deployment best practice.
In this step we will also be logging into the Oracle database through Oracle GoldenGate for the
first time, as this is a necessary step to configure the Oracle GoldenGate Checkpoint Table.
ggsci
GGSCI
>
dblogin userid orgg, password orgg
GGSCI
>
add checkpointtable
You can now exit the ggsci session

GGSCI
>
exit
You have now completed the following installation tasks for our workshop:
The Oracle GoldenGate on MySQL software has been downloaded and installed
The Oracle GoldenGate on Oracle software has been downloaded and installed
The Oracle Database has been configured for GoldenGate Checkpointing
44
Lab 3:
Setting up the Initial Load process
Overview
When setting up Replication between two systems, it is a typical requirement to have both
systems in-synch with one another. In many cases this may mean transferring large volumes of
existing data to get both sites at the point in which they can be kept in synch moving forward.
This is referred to as "Instantiating" your target database.
There are many ways to do this and the factors that will influence your decision on which
approach to take, are varied at best. How large is the database and data being replicated? Is
it the entire database system? What is the distance between the sites? What is the speed of
the network / wire to transfer the data? Are there time-constraints in getting the sites in synch
with one another? Are both sites running the same database product? What about the
hardware and operating systems?, etc etc.
In other words there are many options to choose from when instantiating the target database,
and in many cases the best approach may be defined by one of your core requirements leaving
you with no other choice on the path you need to take. Without getting into the pros and cons
of which approach is best suited for your individual needs we will be looking at how Oracle
GoldenGate can be used for instantiating your target database. In a production deployment this
step may be bypassed if the databases are instantiated using another (external) method.
Thus, in this lab we will be learning how to setup our initial extract process from our Source
database (MySQL) and then appying the captured data to our Target database (Oracle).
45
3.1
If you are not already logged into your VirtualBox system as the Oracle Linux user do so
now.
From a terminal window- change to the mygg user and then navigate to the /u01/mysqlgg
directory. This is the predefined directory that you have just installed the Oracle GoldenGate on
MySQL software.
su mygg
Password: oracle
[mygg@localhost ~]$
cd /u01/mysqlgg
pwd
ls -ltr
46
3.2
Run the GoldenGate Software Command Interface (ggsci) utility and then edit the parameter file
for the GoldenGate Process Manager (mgr).
[mygg@localhost mysql]$ ggsci
GGSCI
> info all
GGSCI
> edit param mgr
This will open the following vi editing session go ahead and enter the following lines (verbatim).
Save your changes and exit back to the ggsci command prompt
Note: The parameters we are setting are:

PORT
The port on which the Manager listens for requests. In this case
we are using the default port assignment.
DYNAMIC PORT LIST
Sets the port-range for dynamic allocation by the Manager.
PURGEOLDEXTRACTS
This purges trail data that is no longer needed
USECHECKPOINTS
Only purge when all processes are finished with a file as

indicated by the checkpoint. This setting ensures no data will be
deleted until all processes are finished with it and is the default
setting.
CHECKMINUTES
Determines how often the Manager will cycle through

maintenance activities
47
3.3
Next, you will create a parameter file called eload.prm. While still in the GoldenGate Software
Command Interface (ggsci) utility edit the parameter file for eload. Once again, the edit
command will start a vi session.
The file you will be creating/editing is called: /u01/mysqlgg/dirprm/eload.prm.
As an alternative you can always cd into the /dirprm directory and manually create and edit the
file without using the ggsci utility.
GGSCI
> edit param ELOAD
Make sure you Save your changes and Exit back to the ggsci command prompt.
48
3.4
After you have finished editing the eload.prm file exit from ggsci and take a look at the newly
created parameter files via the operating system.
The two newly created parameter files are located in the sub-directory: /u01/mysqlgg/dirprm/
GGSCI
> exit
ls -l dirprm
49
3.5
There is nothing preventing you from modifying the GoldenGate parameter (or prm) files
created by ggsci with an external editor such as gedit or vi. This capability is a handy feature
that can be further leveraged to reuse existing GoldenGate parameter files to creating new
GoldenGate parameter files, or for correcting typographic errors and spelling mistakes.
This can be easily tested by using the gedit utility:
gedit dirprm/eload.prm
50
3.6
You will now be setting up the Extract Task Group to capture the initial load data from the
MySQL database MYBANK. If there is a parameter file predefined for this Extract it will be
used. In this case, the file we have just created in the previous step eload.prm will be used.
[mygg@localhost mysqlgg]$ ggsci
GGSCI
>
ADD EXTRACT ELOAD, SOURCEISTABLE
GGSCI
>
info all
GGSCI
>
status *,task
Note: After successfully creating the ELOAD EXTRACT Task Group we were not able to
see ELOAD listed when we typed "info all". This is because it is a special process that
once run will read every row from each table specifiied in the eload.prm file for the
MYBANK database. These rows will then be transmitted and read by the Replicat Task
running on the TARGET database system and inserted row by row. The Task will
automatically shutdown once all rows have been read, transmitted and inserted. We are
however able to see the ELOAD Extract Task listed when we query the status of all tasks.
51
3.7
Keep this (mygg) terminal window open and open another terminal window. In this new
terminal window change to the orgg user.
su orgg
Password: oracle
[orgg@localhost ~]$
cd /u01/oraclegg
[orgg@localhost ~]$
ggsci
GGSCI
>
edit param mgr
Save your changes and exit back to the ggsci command prompt.
Note: The port number is the most important parameter being assigned here. This is
simply because we are running two different installations of Oracle GoldenGate in our
single host environment. So it is very important that we ensure that there are no conflicts
with any of our manager ports.
52
3.8
We are now ready to set up our Replicat task on our Target Oracle database (orgg). We will
first create a parameter file using the ggsci edit command as follows:
GGSCI
>
edit param rload
This will open the vi editor and a newly created parameter file called /u01/oraclegg/dirprm/rload.prm.
Add the following lines verbatim:
Make sure you Save your changes and exit back to the ggsci command prompt.
Note: In this file we are specifying that it is a Replicat process and providing the
following parameter settings:
USERID and PASSWORD
- The database login credentials.
ASSUMETARGETDEF
- When the Source and Target tables specified

with a MAP statement have identical column
structure, this will direct GoldenGate not to
look up the source structures from a source
definition file.
We will be covering this in Lab 6
MAP SOURCE.* TARGET.*
- This directs GoldenGate how to "map" the

table data across the Source and the Target
databases
53
3.9
Specify the Replicat Task Group that is will be running on the Target database (orgg in this
case). In this case we are using the SPECIALRUN clause which informs GoldenGate that we
will be running an initial load process.
GGSCI
>
ADD REPLICAT RLOAD, SPECIALRUN
GGSCI
>
info all
GGSCI
>
status *,task
Note: Once again the rload task will not be listed when using the ggsci command- "info
all", but instead we will need to use the "status *, task" command instead.
54
3.10
We have now setup the "framework" for directing the inital load process. However we do not
have a mechanism to actually do the work of extracting the data from the Source database,
transporting the data to the Target database and then performing the change data apply in the
Target database. We are now going to set this up.
We will start with the Source (mygg). This is the MySQL database MYBANK..
You should still be in a ggsci console if not, start ggsci from the /u01/mysqlgg directory in the mygg
Terminal window
GGSCI
> edit param emastr
This will open a newly created parameter file called /u01/mysqlgg/dirprm/emastr.prm. Add the
following lines verbatim:
A description of the Parameters we have set in this parameter file is covered on the next page.
55
Parameters Set in the EMASTR Extact parameter file: emastr.prm
EXTRACT
Specify the name of the EXTRACT Process
EXTTRAIL
Specifies the Trail name to be used as the Data Source
SOURCEDB
The Source database - this parameter is not required for an

Oracle database
USERID , PASSWORD
Database User Id and Password
STATOPTIONS
Specifies Statistics to be displayued when running the STATS

command - Specifying the RESETREPORTSTATS parameter will
reset the statistics for the Extract process when a new process
report is created - Not specifying this would include statistics from
different Reporting.
REPORT AT
Informs the system to generate a report every day at the specified

time (HH:MM).
REPORTROLLOVER
This will force the Report files to age on a regular schedule

instead of when an Extract process starts.
REPORTCOUNT
Reports the number of records processed - and in this case every

60 seconds.
56
3.11
Setup the EMASTR Extract process on the Source (mysql - mygg) Database:
GGSCI > ADD EXTRACT EMASTR, TRANLOG, BEGIN NOW, DESCRIPTION "Capture change
data from Redo"
GGSCI
> ADD EXTTRAIL ./dirdat/em, EXTRACT EMASTR, MEGABYTES 50
Then set up the Data Pump Process called PMASTR in this case. The first step here is to create the
parameter file for the data pump process: pmastr.
GGSCI
> edit param pmastr
57
3.12
Add the following entries to the pmastr parameter file:
Parameters set in the PMASTR Datapump parameter file: pmastr.prm
EXTRACT
Specify the name of the EXTRACT (DataPump) Process
RMTHOST
This is the Hostname or IP Address of the Remote Server and

GoldenGate instance - we are using 127.0.0.1 for the localhost
MGRPORT
The port on which the remote Manager Process is listening

in this case this is the orgg Oracle GoldenGate on Oracle port
RMTTRAIL
Database User ID and Password
PASSTHRU
This Parameter is for Extract DataPump only - This informs the

process to not use data definitions from the database and that the
data can be propagated to an intermediary system that may not
have a database installed on it.
More details on this parameter is available in page 252 of the
GoldenGate Reference Guide.
TABLE
Specify the Tables to be processed - As illustrated here wildcards

can be used.
58
3.13
Create the Data Pump (PMASTR) process as follows: (in mysql / mygg)
GGSCI > ADD EXTRACT PMASTR, EXTTRAILSOURCE ./dirdat/em, DESCRIPTION "Reads
from dirdat/em on SOURCE and writes to dirdat/pm on TARGET"
GGSCI
> ADD RMTTRAIL ./dirdat/pm, EXTRACT PMASTR, MEGABYTES 50
GGSCI
> info all
We can now see the newly created EMASTR Extract and PMASTR DataPump
processes. But before we can start the process we need to ensure the Replicat Process
is configured on the Target database...
59
3.14
You are now ready to add the Replicat process to the Target Oracle Database via ggsci
console. Make sure you are logged in as orgg. This will open a newly created empty
parameter file called /u01/oraclegg/dirprm/rmastr.prm.
Of special consideration here are the two parameters HANDLECOLLISIONS and END RUNTIME.
When HANDLECOLLISIONS is active, Replicat performs the following operations:
1.
Duplicate inserts: If replicat attempts to write a row that already exists in the Target, the
Target row is overwritten
2.
Missing updates or deletes: If replicat attempts to update or delete a row that is missing,
the data is discarded.
When END RUNTIME is active it tells Replicat to shutdown if it encounters a record in the Extract Trail
with a commit timestamp equal-to or after the process start time.
GGSCI
ggsci
> edit param rmastr
Add the following lines verbatim:
Save your changes and exit back to the ggsci prompt.
60
3.15
When you have returned to the GGSCI prompt enter the following:
(orgg)
GGSCI > ADD REPLICAT RMASTR, EXTTRAIL ./dirdat/pm, DESCRIPTION

"Change Data Apply Reads from /dirdat/pm"
61
3.16
The last step on the Target system (orgg in this case) before firing everything up is to ensure
that the Process Manager is up and running. Otherwise when we start the DataPump process
on the Source System (mygg) it could potentially error out (or ABEND) as it will not be able to
communicate with the target host/Database (orgg) .
Start the MGR process on the TARGET database (orgg):
[orgg@localhost oraclegg]$ ggsci
GGSCI
> start mgr
GGSCI
> info all
62
STEP 3.17 IS TO BE BYPASSED DURING THIS WORKSHOP

3.17
You are are now ready to start the Extract processes on the Source System (mygg)
GGSCI
> start mgr
GGSCI
> start emastr
GGSCI
> start pmastr
Then Check the Status of the emastr and pmastr processes:

GGSCI
> status emastr
GGSCI
> status pmastr
63
3.18
If everything looks okay in that it is running withount any reported error messages, let's go
ahead start the mygg processes: MGR and ELOAD (or Initial Load) process on the mygg
system.
GGSCI
> start mgr
GGSCI
> start eload
GGSCI
> status *,task
GGSCI
> status er *
64
3.19
Once you have established that these processes are running without any issues Leave this
window open while you open a new window (or navigate to a window opened as the orgg user)
on the Target (orgg) system.
[orgg@localhost ~]$
cd /u01/oraclegg/
ggsci
GGSCI
>
DBLOGIN USERID orgg, PASSWORD orgg
GGSCI
>
start rmastr
GGSCI
>
status *,task
NOTE: If you experience any problems, errors or abends with any processes you can check
these by typing:
GGSCI
>
view report rmastr
65
3.20
Also the Report mechanism is useful tool to assess the processes after they have completed. If
you are interested view these two Reports:
mygg: GGSCI
>
view report eload
orgg: GGSCI
>
view report rload
Note: No screenshots for the view report commands have

been provided due to the length of the report outputs.
You can scroll through the reports using the <space-bar>
to page forward one page at a time.
66
3.21
We have created a utility script called chkdb. This script can be run from a shell prompt to
report the rowcounts for each table in the Oracle Target Database (orgg).
Here's a sample of what your output should look like if everything is working as aniticipated. The
number of rows in your BRANCH_ATM and TELLER_TRANS tables may vary. This is due to the
random values being used to generate the transactions for these two tables.
67
Lab 4:
Setting Up One-way Replication
Overview
In this lab we are going to setup Oracle GoldenGate to capture changes from our SOURCE
Database in MySQL and then to apply those changes to our TARGET Database in Oracle. This
lab is the shortest of all the labs. The reason for this is that all of the work in setting up this lab
has already been done in the previous lab. The Replicat process we have set up on our Target
database just needs to be slightly modified to permit constant Change Data Apply processes to
take place.
68
4.1
Our first step is to simply comment out 2 lines in our rmastr.prm file on our Target Database
(orgg). In this example we will use gedit to modify the parameter file directly.
Comment out HANDLECOLLISIONS and END RUNTIME parameters as these were necessary
for the initial load process. This is discussed in greater detail in the previous lab, when we
created the rmastr parameter file.
Note: Use "-- " to comment out a line in the parameter file.
gedit /u01/oraclegg/dirprm/rmastr.prm
Make sure to SAVE your changes before exiting back to the shell prompt.
69
4.2
We are now ready to restart the rmastr process

GGSCI
> info rmastr
GGSCI
> start rmastr
GGSCI
> lag rmastr
70
4.3
On the Target database (orgg) we will run the chkdb script to see the row counts before we
insert additional data changes from our Source (mygg) database.
[orgg@localhost oraclegg]$ chkdb
71
4.4
In another Terminal window logged in as the mygg user, start the simtxn transaction simulator
stored procedure in MySQL:
mysql>
mysql MYBANK -u mygg -pmygg
call simtxn ;
Note: the simtxn stored procedure, will take longer to run this time round as it uses preexisting data to seed each consecutive run. Hence, you can expect this process to run
for a minimum of 5 minutes under normal circumstances
.
72
4.5
Meanwhile, return to the Terminal window for the orgg user and run the chkdb shell script
again. You should expect to see the row count gradually increasing:
[orgg@localhost oraclegg]$ watch chkdb
You can leave this running for a little while - it will automatically refresh the screen every
2 seconds and you should see the row counts for ACCOUNT_TRANS, BRANCH_ATM
and TELLER_TRANS gradually increasing. When you are satisfied with what you are
seeing or the inserts have stopped you can terminate the watch session by issuing a
<CTRL> C.
73
4.6
Another useful utility operation is to determine the time lag between data being processed by
the Extract process (emastr) and the timestamp for that record in the Data Source (mygg)
based on the data timestamp.
Perform the following operation on the Source database (mygg)
GGSCI
>
lag emastr
Here we are running the lag command on the Source system (mygg in this case)
74
4.7
As with the Lag Extract, there is also a Lag Replicat, this provides us with the lag (or time
difference) between the timestamp of the data being processed by the Replicat process in this
case rmastr and the timestamp of the data being written to the trail.
GGSCI
> lag rmastr
The following screenshot demonstrates the processing that is taking place on-the-fly on the Target
Database Replicat process (orgg). One moment the lag request showed nothing happening, a short
time later, when it was re-executed it had already completed.
75
4.8
Check the detailed status of the emastr Extract process on the Source Database (mygg). From
within the ggsci utility there are two steps we can take to see how the Source Extract process is
running. The first is to query the the process itself from within ggsci using the detailed
information option and the second is to view the emastr report:
STEP AGGSCI
>
info emastr detail
76
4.8
STEP BGGSCI
>
view report emastr
This report is very detailed, hit the <space bar> to scroll through the file. At the very bottom of the report
it lists the number of rows processed.
77
4.9
The very last exercise in this lab is the final proofpoint test... If everything is working as
expected, then we should expect to see the same number of rows in both the Source and the
Target databases. Keeping in mind that the transaction simulation does inserts, deletes and
updates.
The easiest way to demonstrate this is by running a simple SQL query against each database. In this
step we've chosen the ACCOUNT_TRANS and TELLER_TRANS tables.
mysql MYBANK -u mygg -pmygg
mysql>
select count(*) from ACCOUNT_TRANS ;
mysql>
select count(*) from TELLER_TRANS ;
[orgg@localhost oraclegg]$ sqlplus orgg/orgg

SQL>
select count(*) from ACCOUNT_TRANS ;
SQL>
select count(*) from TELLER_TRANS ;
You have now configured your environment for one-way replication from MySQL to Oracle.
78
Clean-Up The Environment

The Extract and Replicat processes we created in the prior 2 labs will not be required for any of the
remaining lab exercises. We should clear out the processes that were created during this lab so they
will not interfere with any of the wildcard commands we will be using from here on out.
The data we have gerenated in both the Source and the Target databases will however remain the
same.
Step 1. clean up the Source Database Extract and DataPump:
GGSCI
> stop emastr
GGSCI
> stop pmastr
GGSCI
> delete emastr
GGSCI
> delete pmastr
ggsci
79
GGSCI
> delete eload
GGSCI
> status *, task
GGSCI
> info all
Step 2. Clean up the Replicat process on the Target database:

GGSCI
> info all
GGSCI
> dblogin userid orgg, password orgg
GGSCI
> stop rmastr
GGSCI
> delete rmastr
GGSCI
> delete rload
80
81
Lab 5:
Setting up Bi-Directional Replication
Overview
In this lab we will be building upon what we have learned thus far and configuring Bi-Directional
replication between both of our databases. In other words, data changes happening in either database
whether it be the MySQL database or the Oracle database, will be captured and applied to the other
database. In any given point in time, the data residing in both databases will be in-sync with oneanother.
As illustrated above, we will be setting up an Extract, a DataPump and a Replicat process for each of
the Databases.
82
5.1
When capturing data changes in real-time from an Oracle database, it is important to ensure
that the Oracle database is set to log the primary key values whenever it logs a row change.
This is so GoldenGate can read the Primary Key values from the redo. The default behaviour in
Oracle only logs the changed column values and the rowid. This can done via the ggsci "ADD
TRANDATA <OWNER>.<TABLE_NAME>" command. This is a non-complex but repetitive task
so it has been pre-scripted in a file called TRANDATA.OBY. Execute the file as follows:
5.1.1
View the TRANDATA.OBY on the Oracle Database (orgg)
[orgg@localhost oraclegg]$ cat /u01/stage/TRANDATA.OBY
83
5.1.2
GGSCI
Now run the OBEY File on the Oracle Database (orgg)

>
obey /u01/stage/TRANDATA.OBY
(Ignore the error messages about Logging being already disabled)
84
5.2
In order for Bi-Directional Replication to take place we need two copies of each of the
processes we used in the previous 2 labs. In other words, each database will be running an
EXTRACT process, a DATAPUMP process and a REPLICAT process. The main goal of this lab
is to help you understand the interaction and dependencies of each of these components. So
rather then put you through the motions of typing all the contents of the each of the parameter
files, these have already been provided for you. In addition to the parameter files there are two
Obey files that have pre-scripted the setup of this environment for you. The naming convention
for all of the processes and associated files are based on the simple rule of <DB><Process> as
follows:
myxt = MySQL Extract
orxt = Oracle Extract
mydp = MySQL DataPump
ordp = Oracle DataPump
myrt = MySQL Replicat
orrt = Oracle Replicat
Our first task is to copy or move the parameter files to the correct location for the approach database.
As orgg [orgg@localhost oraclegg]$ cp /u01/stage/or??.prm /u01/oraclegg/dirprm

As mygg [mygg@localhost mysqlgg]$ cp /u01/stage/my??.prm /u01/mysqlgg/dirprm
Once the files are in their correct locations it is worthwhile taking a look at each file using either
cat or view or gedit. Of particular interest are the myrt.prm and the ordp.prm files.as these
files are configured specifically with MySQL in mind to include the concept of a "database". As
such they require the additional handle of SOURCEDB when calling a database operation.
This is not a requirement in the Oracle database model so we have not yet seen this format in
our lab.
85
5.3
Our next step is to create the processes on each database.

It does not make a difference which system we perform these steps on first. We will start with the
MySQL database (mygg).
ggsci
GGSCI > ADD EXTRACT MYXT, TRANLOG, BEGIN NOW, DESCRIPTION "Capture change
data for MySQL"
GGSCI
> ADD EXTTRAIL ./dirdat/mx, EXTRACT MYXT, MEGABYTES 10
GGSCI > ADD EXTRACT MYDP, EXTTRAILSOURCE ./dirdat/mx, DESCRIPTION DataPump

reads from /dirdat/mx on MySQL - writes to /dirdat/md on Oracle
GGSCI
> ADD RMTTRAIL ./dirdat/md, EXTRACT MYDP, MEGABYTES 10
GGSCI
> dblogin SOURCEDB MYBANK, userid mygg, password mygg
GGSCI > ADD REPLICAT MYRT, EXTTRAIL ./dirdat/od, DESCRIPTION Change Data
apply from Oracle reads from ./dirdat/od
86
5.4
And then as the orgg user

ggsci
GGSCI > ADD EXTRACT ORXT, TRANLOG, BEGIN NOW, DESCRIPTION Capture change
data for Oracle
GGSCI
> ADD EXTTRAIL ./dirdat/ox, EXTRACT ORXT, MEGABYTES 10
GGSCI > ADD EXTRACT ORDP, EXTTRAILSOURCE ./dirdat/ox, DESCRIPTION Datapump

reads from /dirdat/ox on Oracle writes to /dirdat/od on MySQL
GGSCI
> ADD RMTTRAIL ./dirdat/od, EXTRACT ORDP, MEGABYTES 10
GGSCI
GGSCI > ADD REPLICAT ORRT, EXTTRAIL ./dirdat/md, DESCRIPTION Change Data
Apply to ORACLE Reads from /dirdat/md
87
5.5
We are now ready to get things started... The easiest way is to use the ggsci utility as follows:
As the mygg user
GGSCI
> info all
GGSCI
>
start *
It should look like this:
88
5.6
and then as the orgg perform this
5.7
Run this command in both the MySQL and the Oracle Database servers and wait a little while
before checking each GoldenGate instance for errors
mygg: GGSCI
> info er *
89
5.8
Now as the orgg user:

GGSCI
5.9
> info er *
Now go ahead and try some SQL DML operations in each of the databases to see how things
work. For example:
Oracle (orgg) :
SQL> select count(*) from branch ;
SQL> insert into branch values (50, 55555) ;
SQL> commit ;
SQL> select count(*) from branch ;
MySQL (mygg) :
[mygg@localhost mysqlgg]$ mysql MYBANK -u mygg -pmygg
MySQL> select count(*) from BRANCH ;
MySQL> insert into BRANCH values (44,40404) ;
MySQL> commit ;
MySQL> select count(*) from BRANCH ;
90
Here's an example of a delete operation being sychronized across the two databases...
If you have some spare time, feel free to try out some of your own experiments!
91
Lab 6:
Leveraging Oracle GoldenGate Transformations
Overview
The focus on this lab will be to demonstrate how to use data transformations for making
changes on the fly to data as it is being applied to the TARGET database. We will be using the
MySQL database as the SOURCE database and Oracle as the TARGET database.
We are already familiar with the idea of using Oracle GoldenGate to convert data from one
database to another as this has been the focus of this workshop. Oracle GoldenGate can also
perform data-conversions and transformations to tables and data as well. Table transformations
include a wide range of conversions from column splitting to column-concatenation, to creation
of columns which do not exist in the Source database, just to name a few. For data
conversions, data can be enhanced or filtered so that only a subset of the Source gets applied
to a Target database.
There are many use-cases where it makes sense to perform this operation using Oracle
GoldenGate but there are also times where it may be better to have this operation performed at
the Target Database via other Oracle ELT utilities such as the Oracle Data Integrator (ODI)
product. This lab is provided as an overview on how to leverage Oracle GoldenGate for
performing data Transformations - your usage of this feature in a real-world deployment should
be the result of careful review and consideration of your database deployment environment.
The model we are using for this lab inroduces two new tables called
ACCOUNT_BAL_HISTORY and ACCOUNT_TRANS_HISTORY. As their names suggest,
these tables will be used to store historical data based on transactions that modify the existing
MySQL database. When an insert on the Source database takes place, additional information
is generated and appended to the existing row information. This is then inserted into our two
new History tables.
92
6.1
The first step in running this lab will be creating the new tables in the Oracle Database. The
DDL for this operation is provided for you in a SQL script located in the /u01/stage directory. To
run the ddl perform the following step as the orgg user:
SQL> @/u01/stage/ora_transform_ddl.sql
6.2
For this lab the parameter files have already been created for you to ease the burden. However
it is strongly recommended that you review the Replicat parameter file: ortrr.prm This file
contains the logic and instructions for remapping the data and adding new information to the
data from the Source database.
[orgg@localhost oraclegg]$ cp /u01/stage/ortrr.prm /u01/oraclegg/dirprm
[orgg@localhost oraclegg]$ view /u01/oraclegg/dirprm/ortrr.prm
The end-section of the parameter file has the main steps required to perform the Transformation for us:
93
6.3
As the mygg user copy the pre-staged Extract and DataPump paramter files to the MySQL
GoldenGate dirprm directory for reuse.
[mygg@localhost mysqlgg]$ cp /u01/stage/mytr*.prm /u01/mysqlgg/dirprm/
GGSCI > ADD EXTRACT MYTRX, TRANLOG, BEGIN NOW, DESCRIPTION "Capture MySQL
Changes for Transformation"
GGSCI
> ADD EXTTRAIL ./dirdat/mt, EXTRACT MYTRX, MEGABYTES 10
GGSCI > ADD EXTRACT MYTRP, EXTTRAILSOURCE ./dirdat/mt, DESCRIPTION

"DataPump for MySQL Transformation Reads from /dirdat/mt writes to
/dirdat/mr on Oracle"
GGSCI
> ADD RMTTRAIL ./dirdat/mr, EXTRACT MYTRP, MEGABYTES 10
94
6.4
Create the Replicat process on the Target Oracle Database...

GGSCI
GGSCI > ADD REPLICAT ORTRR, EXTTRAIL ./dirdat/mr, DESCRIPTION "Change Data
Apply Transformation Reads from /dirdat/mr"
95
6.5
There is still one more very important step to be performed in this situation and this is to run the
defgen utility. Defgen produces a file containing all the definitions and mappings of the Source
database tables and data. The file it produces is then transferred to the Target database in text
format. The Replicat process uses these definitions to better understand the data it is mapping
and in this lab Replicat will require this file to perform our data transformations.
The first step to using defgen is to create a parameter file for defgen:
[mygg@localhost mysqlgg]$ vi /u01/mysqlgg/dirprm/defgen.prm
Save your changes and exit.
96
6.6
Once we have created this file we can run the defgen utility. The file then needs to be manually
moved to the system or location where it can be located by the Replicat process. In the steps
outlined below we change to the superuser root account due to permission restrictions when
moving the file from one Oracle GoldenGate user to another.
[mygg@localhost mysqlgg]$ defgen paramfile dirprm/defgen.prm
[mygg@localhost mysqlgg]$ su
Password: oracle
# cp /u01/mysqlgg/dirdef/mybank.def /u01/oraclegg/dirdef/
# chown orgg.orgg /u01/oraclegg/dirdef/mybank.def
# exit
97
6.7
It is a good idea to take a look at the file and ensure you have the right access permissions, etc
on the Target database orgg.
[orgg@localhost dirdef]$
ls -l /u01/oraclegg/dirdef/mybank.def
98
6.8
Let's start the Extract and DataPump on the Source MySQL database (mygg) :
GGSCI
> start mytr*
GGSCI
> status mytr*
99
6.9
Start the Replicat on the Target Oracle Database (orgg)

GGSCI
> start ortrr
GGSCI
> status ortrr
100
6.10
Last but not least generate a workload see how things work
Mysql Source Database (mygg)

[mygg@localhost mysqlgg] mysql MYBANK -u mygg -p
Enter password: mygg
mysql> call simtxn ;
Oracle Target Database (orgg) [orgg@localhost oraclegg] sqlplus orgg/orgg

select count(*) from ACCOUNT_BAL_HISTORY ;
SQL> /
SQL> /
SQL> /
101
Lab 7: Leveraging Oracle GoldenGate for Filtering

Overview
We have already seen how Oracle GoldenGate can be used to perform data Transformations. In this
lab we are going to focus on the ability for Oracle GoldenGate to perform Filter operations. In this case
we will be using Filtering to selectively Capture data changes that are of interest to us.
For this lab we are once again using the MySQL database as the Source database and the Oracle
database as the Target database. We will be implementing a process to flag transactions in the MySQL
database where the account balance drops below zero (or in other words - the account balance going
into Overdraft). In other words, we only want to capture account transactions that result in the
Account_Balance dropping below 0.
To solve this problem, we will use a FILTER operation to capture the transaction and apply this to a
newly created OVERDRAFT table in the Oracle Database.
102
7.1
Before we get started with this lab we will go ahead and cleanup the processes we created from
our previous labs. We will clean up the processes on the MySQL system first:
ggsci
> stop *
ggsci
> status *
ggsci
> dblogin SOURCEDB MYBANK, userid mygg, password mygg
ggsci
> delete *
ggsci
> status *
Note: Logging into the MySQL database is necessary in order to clear out the DB
Checkpoint processes for any Replicat Processes - in this case MYRT.
103
7.2
Now we'll clean up the GoldenGate processes for Oracle:

ggsci
> stop *
ggsci
> status *
ggsci
ggsci
> delete *
ggsci
> status *
Note: Logging into the Oracle database is necessary in order to clear out the DB
Checkpoint table for any Replicat Processes - in this case ORRT and ORTRR.
104
7.3
Now that we've cleaned things up, the first step is to create the OVERDRAFT_HISTORY table
in the Oracle Database. The DDL for this operation is provided for you in a SQL script located
in the /u01/stage directory. To run the the DDL script, perform the following step as the orgg
user:
[orgg@localhost oraclegg] $ sqlplus orgg/orgg
SQL> @/u01/stage/ora_filter_ddl.sql
105
7.4
For this lab, the parameter files have already been created for you to remove some of the
burden. It is strongly recommended however, to review the the Replicat Parameter file:
orfir.prm. As this file contains the logic and instructions for performing the Filtering operation for
data coming from the Source Database.
[orgg@localhost oraclegg] $ cp /u01/stage/orfir.prm /u01/oraclegg/dirprm
[orgg@localhost oraclegg] $ view /u01/oraclegg/dirprm/orfir.prm
Note: This parameter file is very similar to the parameter file we used in the Transformation lab.
At the bottom of the file you will notice the FILTER command - we are essentially only
replicating data where the Account_Balance falls below zero (or Overdraft in our example).
106
7.5
Now as the mygg user, copy the pre-staged Extract and DataPump parameter files to MySQL
GoldenGate dirprm directory for reuse.
Then we can go ahead and creates the Extract and DataPump for the new task.
[mygg@localhost mysqlgg]$ cp /u01/stage/myfi*.prm
/u01/mysqlgg/dirprm/

GGSCI > ADD EXTRACT MYFIX, TRANLOG, BEGIN NOW, DESCRIPTION "Capture MySQL
Changes for FILTER"
GGSCI
> ADD EXTTRAIL ./dirdat/ft, EXTRACT MYFIX, MEGABYTES 10
GGSCI > ADD EXTRACT MYFIP, EXTTRAILSOURCE ./dirdat/ft, DESCRIPTION

"DataPump for MySQL FILTER - Reads from /dirdat/ft - writes to /dirdat/fr on
Oracle"
GGSCI
> ADD RMTTRAIL ./dirdat/fr, EXTRACT MYFIP, MEGABYTES 10
GGSCI
> info all
107
7.6
Create the Replicat Process on the Target Oracle Database

GGSCI
GGSCI > ADD REPLICAT ORFIR, EXTTRAIL ./dirdat/fr, DESCRIPTION "Change Data
Apply FILTER - reads from /dirdat/fr"
108
7.7
We are now ready to start the Extract and the DataPump on the Source MySQL database
(mygg):
GGSCI
> status
GGSCI
> start myfix
GGSCI
> start myfip
GGSCI
> info all
109
7.8
Start the Replicat Process on the Target Oracle Database (orgg)

GGSCI
> status *
GGSCI
> start orfir
GGSCI
> info all
110
7.9
Last but not least we will go ahead and generate a workload to see how our FILTER operation
works.
MySQL Source Database (mygg)
[mygg@localhost mysqlgg]$ mysql MYBANK -u mygg -pmygg
mysql> call simtxn ;
Oracle Target Database (orgg)
SQL> select * from OVERDRAFT_HISTORY ;
wait a few minutes
SQL> /
You have now completed Lab 7: Leveraging Oracle GoldenGate for FILTERING.
111
Appendix A:
Additional Information
1. There are special considerations when using Oracle GoldenGate on Oracle with
Automatic Storage Management. The steps required for configuring are documented in
Chapter 4 of the Oracle GoldenGate Oracle Installation and Setup Guide version 10.4
2. When using Oracle GoldenGate on Oracle, it is recommend and a best practice to
enable archivelog mode as this will ensure that Change Data Capture information is not
lost during log switches.
3. Oracle GoldenGate has been validated and is supported with the Oracle Exadata
Database Machine.
112
Appendix B: Schema Definitions for the Workshop

MySQL Database
Table ACCOUNT
ACCOUNT_NUMBER
numeric(10,0)
ACCOUNT_BALANCE
decimal(38,2)
PRIMARY-KEY
Table ACCOUNT_TRANS
ACCOUNT_NUMBER
numeric(10,0)
PRIMARY-KEY
TRANS_NUMBER
numeric(18,0)
PRIMARY-KEY
ACCOUNT_TRANS_TS
timestamp(6)
PRIMARY-KEY
ACCOUNT_TRANS_TYPE
varchar(3)
ACCOUNT_TRANS_LOC
varchar(50)
ACCOUNT_TRANS_AMT
decimal(18,2)
Table BRANCH
BRANCH_NUMBER
numeric(10,0)
BRANCH_ZIP
numeric(5)
PRIMARY-KEY
Table TELLER
TELLER_NUMBER
numeric(10,0)
TELLER_BRANCH
numeric(10,0)
PRIMARY-KEY
Table TELLER_TRANS
TELLER_NUMBER
numeric(10,0)
PRIMARY-KEY
TRANS_NUMBER
numeric(18,0)
PRIMARY-KEY
ACCOUNT_NUMBER
numeric(10,0)
PRIMARY-KEY
ACCOUNT_TRANS_TS
timestamp(6)
PRIMARY-KEY
ACCOUNT_TRANS_TYPE
char(2)
TELLER_TRANS_AMT
decimal(18,2)
Table BRANCH_ATM
BRANCH_NUMBER
decimal (10,0)
PRIMARY-KEY
ATM_NUMBER
smallint
PRIMARY-KEY
TRANS_NUMBER
numeric(18,0)
PRIMARY-KEY
ACCOUNT_NUMBER
numeric(10,0)
ATM_TRANS_TS
timestamp(6)
ATM_TRANS_TYPE
char(2)
ATM_TRANS_AMT
decimal(18,2)
PRIMARY-KEY
113
Oracle Database
Table ACCOUNT
account_number
number(10,0)
account_balance
decimal(38,2)
primary key (account_number)
using index
Table ACCOUNT_TRANS
account_number
number(10,0)
trans_number
number(18,0)
account_trans_ts
timestamp(6)
account_trans_type
varchar2(3)
account_trans_loc
varchar2(50)
account_trans_amt
decimal(18,2)
primary key (account_number, trans_number, account_trans_ts)
using index
Table BRANCH
branch_number
number(10,0)
branch_zip
number(5)
primary key (branch_number)
using index
Table TELLER
teller_number
number(10,0)
teller_branch
number(10,0)
primary key (teller_number)
using index
Table TELLER_TRANS
teller_number
trans_number
account_number
teller_trans_ts
teller_trans_type
teller_trans_amt
primary key (teller_number,
using index
number(10,0)
number(18,0)
number(10,0)
timestamp(6)
char(2)
decimal(18,2)
trans_number, teller_trans_ts)
114
Table BRANCH_ATM
branch_number
atm_number
trans_number
account_number
atm_trans_ts
atm_trans_type
atm_trans_amt
primary key (branch_number,
using index
decimal (10,0)
smallint
number(18,0)
number(10,0)
timestamp(6)
char(2)
decimal(18,2)
atm_number, trans_number, atm_trans_ts)
Table ACCOUNT_BAL_HISTORY
account_number
number(10,0)
trans_ts
timestamp(6)
account_balance
decimal(38,2)
Table ACCOUNT_TRANS_HISTORY
account_number
number(10,0)
trans_number
number(18,0)
trans_ts
timestamp(6)
trans_type
varchar2(3)
trans_loc
varchar2(50)
trans_amt
decimal(18,2)
Table OVERDRAFT_HISTORY
account_number
number(10,0)
trans_ts
timestamp(6)
account_balance
decimal(38,2)
115
Appendix C: Troubleshooting
Most often problems arise due to typographic errors - a parameter file may have a syntax error which
can cause the the related process (Extract/DataPump/Replicat) to abend. The following set of
commands can be used to quickly diagnose what may be causing the problem:
1.
To diagnose, useful commands are:

info all
view report <process name>
edit or view the associated parameter file
edit or view ggserr.log
2.
To start diagnosis of an issue the best starting point is the "view report <group name>"
command or by looking at the ggserr.log. Often this will quickly point out a simple problem
such as a typographic error and possibly the reason why a process may have abended.
3.
To resume a process after it has abended, consideration must be given to it's previous state
and whether it can be easily recovered from the error that brought it down. e.g. if REPLICAT
has failed because of missing checkpoint information, the checkpoint table may need to be
recreated again.
4.
For this workshop, to start afresh following an undiagnosable error the following series of steps
have been found of use:
a)
b)
c)
d)
e)
stop all gg processes (e.g. stop *)

bring all check points to current state (e.g. alter extract emastr begin now)
truncate all tables (mybank_seed.sql will perform this on the MySQL database)
restart all the processes (e.g. start emastr)
restart the seed script (preferrable) or other load process
116
Appendix D: Useful GGSCI Commands
Note - Please refer to the Oracle GoldenGate Reference Guide for more detailed
information on the Oracle GoldenGate command library and syntax. The following list of
commands has been provided as a quick reference guide for commands we will be using
through out this workshop.
ADD EXTRACT | REPLICAT

Use the appropriate ADD command to create a specific group whether it be an Extract or a Replicat
group.
ADD CHECKPOINTTABLE
The ADD CHECKPOINTTABLE command is used to create a Checkpoint Table in the target database.
The Checkpoint table is used by Oracle GoldenGate to track the current position of the Replicat in the
trail. This command is used in conjunction with the DBLOGIN command to establish a database
connection prior to using this command.
ADD TRANDATA
Use the ADD TRANDATA command to enable Oracle GoldenGate to acquire the transaction information
it needs from the transaction logs. For Oracle9i and later the ADD TRANDATA command enables tablelevel supplemental logging based on column uniqueness. This command must be used in conjunction
with the DBLOGIN command to establish a database connection prior to using this command.
CREATE SUBDIRS
The CREATE SUBDIRS command is used during the installation of Oracle GoldenGate to create the
default sub-directories required for the Oracle GoldenGate software. Use the CREATE SUBDIRS
command before any other configuration tasks.
DBLOGIN [SOURCEDB <database name>, ] USERID <database user>,
password>
PASSWORD <database
Use the DBLOGIN command to connect to the specified database using the logon credentials provided.
In the case of MySQL, SQLServer and Sybase the additional SOURCEDB parameter is required to
determine which database is logged into. If the password is not specified it will be prompted for when
making the database connection.
DELETE EXTRACT
Use the DELETE EXTRACT command to remove the associated Extract group. The command will also
delete the checkpoint file but will leave the parameter file intact. The associated TRAIL files must be
manually deleted via the operating system.
Before using this command the Extract must be stopped by using the STOP EXTRACT <group name>
command.
117
DELETE
REPLICAT
Use the DELETE REPLICAT command to remove the associated Replicat group. The command will
also delete the checkpoint file but will leave the parameter file intact. This command frees up the
associated TRAIL files for purging by Manager, because the checkpoints used for the deleted group are
removed.
Before using this command the Replicat must be stopped by using the STOP REPLICAT <group name>
command.
EDIT PARAM <parameter file>
Use this command to edit the named parameter file. If the file does not already exist this operation will
create the file. In our lab environments we are running Linux so the default editor is vi.
INFO ALL
Use INFO ALL to display the status and lag (where relevant) for all Manager, Extract, and Replicat
programs on a system. The basic command, without options, displays only online (continuous)
programs. To display tasks, use either INFO ALL TASKS or INFO ALL ALLPROCESSES
INFO EXTRACT
Use INFO EXTRACT to view the following information.
Status of Extract (STARTING, RUNNING, STOPPED or ABENDED).
Approximate Extract lag.
Checkpoint information.
Process run history.
The trail(s) to which Extract is writing.
The basic command, without either the TASKS or ALLPROCESSES argument, displays
information only for online (continuous) Extract processes. Tasks are excluded.
INFO REPLICAT
Use INFO REPLICAT to retrieve the processing history of a Replicat group. The output of this
command includes:
The status of Replicat (STARTING, RUNNING, STOPPED or ABENDED).
Approximate replication lag.
The trail from which Replicat is reading.
Replicat run history, including checkpoints in the trail.
Information about the Replicat environment.
The basic command, without the TASKS or ALLPROCESSES argument, displays information only
for online (continuous) Replicat groups. Tasks are excluded.
118
LAG EXTRACT <group name>

Use LAG EXTRACT to determine a true lag time between Extract and the data source. LAG EXTRACT
calculates the lag time more precisely than INFO EXTRACT because it communicates with Extract
directly, rather than reading a checkpoint position in the trail. For Extract, lag is the difference, in
seconds, between the time that a record was processed by Extract (based on the system clock) and the
timestamp of that record in the data source.
LAG REPLICAT
<group name>
Use LAG REPLICAT to determine a true lag time between Replicat and the trail. LAG REPLICAT
estimates the lag time more precisely than INFO REPLICAT because it communicates with Replicat
directly rather than reading a checkpoint position. For Replicat, lag is the difference, in seconds,
between the time that the last record was processed by Replicat (based on the system clock) and the
timestamp of the record in the trail.
OBEY
Use OBEY to process a file that contains a list of GoldenGate commands. OBEY is useful for executing
commands that are frequently used in sequence.
status <group name> | *
STATUS MANAGER | EXTRACT | REPLICAT
<groupname | * >
, TASKS
Use the STATUS command to determine whether or not the Specified process (Manager, Extract or
Replicate) is running. In the case of the Manager process, the port number is displayed. In the case of
the * (wildcard) the status of all the processes will be listed.
START <group name> | MANAGER | *
Starts the associated process including the Manager Process. When used with a wildcard "*" will start
all processes excluding the Manager process.
STOP <group name> | MANAGER [!] | *
Stops the associated process including the Manager Process. When used with a wildcard "*" will stop
all processes excluding the Manager process.
When using STOP MANAGER ! (with the excamation point) the command bypasses the prompt
confirming the intent to shutdown the Manager process.
VIEW REPORT <group name>
Use VIEW REPORT to view the process report that is generated by Extract or Replicat. The report lists
process parameters, run statistics, error messages, and other diagnostic information.
The command displays only the current report. Reports are aged whenever a process starts. Old reports
are appended with a sequence number, for example finance0.rpt, finance1.rpt, and so forth. To view old
reports, use the [<n>] option.
119
This Document was created by Oracle Platform Technology Solutions.
120

PTS OGG Workshop v4

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

PTS OGG Workshop v4

Uploaded by

Copyright:

Available Formats

PlatformTechnology Solutions

Oracle GoldenGate Workshop

Copyright 2010 Oracle and/or its affiliates. All rights reserved..

Copyright 2010 Oracle and/or its affiliates. All rights reserved..

Oracle PTS GoldenGate Workshop

Lab 1: Getting Started........................................................................................................................... 6

Appendix A: Useful Information............................................................................................................112

Copyright 2010 Oracle and/or its affiliates. All rights reserved..

Workshop Pre Requisites

Basic skills in Linux operation as well as editing text files in Linux.

Understanding of the SQL language is necessary.

Copyright 2010 Oracle and/or its affiliates. All rights reserved..

Products We Will Be Using In This Workshop

Oracle GoldenGate on Oracle version 11.1.1.0.0

Oracle GoldenGate on MySQL version 11.0

Oracle VirtualBox Version 3.2.8

Oracle Enterprise Linux 5 Update 2, 32bit

Oracle Database 11g Release 2, Enterprise Edition, Version 11.2.0.1

Oracle MySQL 5.1 with the InnoDB engine

Important Information About This Workshop

Accounts and Passwords used in this workshop

Linux Super User account

Linux Oracle Database owner

Linux MySQL Database owner

GoldenGate MySQL user

GoldenGate Oracle user

Oracle Database user account for GoldenGate

MySQL Database user account for GoldenGate

Copyright 2010 Oracle and/or its affiliates. All rights reserved..

Copyright 2010 Oracle and/or its affiliates. All rights reserved..

Change to the orgg user with the password oracle as follows:

Copyright 2010 Oracle and/or its affiliates. All rights reserved..

View the Environment variables set for this account session:

Copyright 2010 Oracle and/or its affiliates. All rights reserved..

alter database add supplemental log data ;

alter system switch logfile ;

select supplemental_log_data_min from v$database

Copyright 2010 Oracle and/or its affiliates. All rights reserved..

create user orgg identified by orgg

grant connect, resource, dba to orgg ;

Copyright 2010 Oracle and/or its affiliates. All rights reserved..

SQL> select table_name from user_tables ;

Copyright 2010 Oracle and/or its affiliates. All rights reserved..

This should return information about the MySQL database as follows:

Copyright 2010 Oracle and/or its affiliates. All rights reserved..

Copyright 2010 Oracle and/or its affiliates. All rights reserved..

Copyright 2010 Oracle and/or its affiliates. All rights reserved..

Copyright 2010 Oracle and/or its affiliates. All rights reserved..

Copyright 2010 Oracle and/or its affiliates. All rights reserved..

Now confirm that the tables have been correctly created.

show table status ;

Copyright 2010 Oracle and/or its affiliates. All rights reserved..

Copyright 2010 Oracle and/or its affiliates. All rights reserved..

select count(*) from ACCOUNT //

select count(*) from ACCOUNT_TRANS //

Copyright 2010 Oracle and/or its affiliates. All rights reserved..

select count(*) from BRANCH //

select count(*) from BRANCH_ATM //

select count(*) from TELLER //

select count(*) from TELLER_TRANS //

Note: The following ROW COUNTS should match:

Copyright 2010 Oracle and/or its affiliates. All rights reserved..

The Oracle database schema and objects have been created

The Oracle database has been configured for Supplemental Logging