You are on page 1of 78

Using ODI for Teradata Dual-Loads

Purpose

This tutorial walks you through the steps that are needed to create Oracle Data Integrator (ODI) EL-T interfaces and packages that will simultaneously load two independent Teradata RDBMS systems configured as dual-active. The source for the data loads is an Oracle database.

Time to Complete: Approximately 120 minutes

Topics

This tutorial covers the following topics:

  • 1. Overview

  • 2. Teradata Dual-Load Architecture using ODI

  • 3. Installation & Setup Prerequisites

  • 4. Setup the ODI Environment

  • 5. Setup the Source and Destination Databases

  • 6. Create Oracle to Flat File Interface

  • 7. Create Interfaces for Scenario 1

  • 8. Create Interfaces for Scenario 2

  • 9. Execute Tests for Scenario 1

    • 10. Execute Tests for Scenario 2

    • 11. Summary

Note: The screenshots will not reflect the specific environment that you are using. They are provided to give you an idea of where to locate specific functionality in Oracle Data Integrator.

  • 1. Overview

A Teradata dual-active system is recommended for high availability of Teradata systems and guarantees business continuity by having the data always available in a secondary system when the primary becomes unavailable.

One method of loading a dual-active Teradata system is by using a dual-load approach. Dual-loads perform massive data loads and transforms from the same source into two separate Teradata systems, one of which is designated as Primary and the other as Secondary. Dual-loads can be performed using multiple scenarios, but the two most popular scenarios are:

Scenario 1 - Parallel Loads and Transforms on Primary and Secondary

Scenario 2 - Load and Transform on Primary and Replicate to Secondary

The first scenario is recommended when it is expected that the frequency of updates on the source system will be less and the data volumes involved in synchronization is high. This scenario will provide the best loading performance while guaranteeing the least overhead.

The second scenario eliminates the need to run transformations on the secondary system since the data will be loaded and transformed on the primary and then only the changed (and transformed) data will be pushed to the secondary system.

This tutorial will cover both scenarios. For more information on the architecture of Teradata dual-loads using ODI, refer to the whitepaper Enabling Teradata Dual Active using Oracle Data Integrator.

  • 2. Teradata Dual-Load Architecture using ODI

The dual-load process deals with loading two systems in parallel with the same

data. .

The loading solution must

take into account the fact that either system may be down at any point in time and miss one or several

Using ODI for Teradata Dual-Loads

loading/synchronization cycles. The loads should be smart enough to catch-up once the system becomes available.

  • 2.1. Scenario 1 - Parallel Loads and Transforms on Primary and Secondary:

In this scenario, a first job will extract data from the source Oracle system. Then, two separate processes will load each of the target Teradata systems. Transformations will be performed in parallel on both systems, and execution can be monitored independently on each of the target system.

Using ODI for Teradata Dual-Loads loading/synchronization cycles. The loads should be smart enough to catch-up once

Figure 1: Scenario 1 Parallel Loads and Transforms on Primary and Secondary

  • 2.1.1. Extract

The extraction is performed only once since reading twice from the source system would be counterproductive. But as the extraction is performed, one of the target systems (or both) could be down. To guarantee the delivery of information, extracted data must first be staged in flat files. Each file extract is numbered with a Batch ID to enable recovery when needed. Each set of extracted files is stored in a directory which is named after the Batch ID. Each table dump will be stored as a file in this directory.

The next run creates a new directory where the new files will be created. As soon as both systems have loaded a given sets of files, the files can be discarded and the directory can be deleted or archived.

The Extract can be smart enough to implement change data capture from the source system. There are multiple ways of enabling changed data capture using ODI, but for the purpose of this tutorial, we will use a simple timestamp method using which we will stamp a Date/Time field on the rows that are inserted or modified on the source table. The extract process will then store the highest processed timestamp in a variable, and use its value to determine the rows that will be extracted in the next run.

  • 2.1.2. Load

The Load operation will be exactly the same on both systems: we have to perform a traditional ELT operation, using which the files are bulk loaded into a staging area on the target system (using

Using ODI for Teradata Dual-Loads

TPT/FastLoad) and then transformed to guarantee the best possible performance. As the same load process runs on both the Primary and Secondary systems, the transformation process will be developed only once and executed against multiple systems. ODI provides several mechanisms that can be leveraged to run the same job on different physical environments, via the notion of contexts. Contexts define different physical environments for the same logical processing units. Variables can also be used here as an alternative to contexts to access the different systems dynamically.

When loads are not processed because one of the system is non-operational, the scenario must ensure recovery of failed jobs. The solution here is simply to run the process and increment the Batch ID until all available files have been processed.

  • 2.1.3. Transform

A simple transformation during the load process will be demonstrated in this tutorial. By using contexts or variables, the transformations can be developed once and executed on both systems.

  • 2.1.4. Cleanup

The staging files after being loaded to both Teradata systems would need to be cleaned up to save space. This process can be very easily built using ODI and is not discussed in this tutorial.

  • 2.2. Scenario 2 - Load and Transform on Primary and Replicate to Secondary:

This scenario will perform loads on the Primary Teradata system from the Oracle database using the steps and packages created in scenario 1. Once the data is loaded into the Primary Teradata system, the scenario 2 will replicate the data from the Primary system to the Secondary Teradata system, instead of the secondary system getting loaded from the Oracle database like scenario 1.

The extract and transformations processes created in scenario 1 will be used in this process to load into the Primary Teradata system. One the data is in the primary system, there will be a Replicate package that will replicate the data loaded from the primary system to the secondary system. The method of capturing changes in the Replicate package is to utilize a simple timestamp based mechanism that identifies the inserted/updated rows in the Primary system. The replicate package will pull these rows based on this timestamp, and the last run of the Replicate package.

The benefit of scenario 2 is that the transformations will be performed only once, on the Primary Teradata system, as opposed to scenario 1, which will run transformations on the Primary and the Secondary Teradata systems.

Using ODI for Teradata Dual-Loads

Using ODI for Teradata Dual-Loads Figure 2: Scenario 2 - Load and Transform on Primary andhere to access this OBE.  You should have downloaded the zip file that contains the ODI seed projects and models used in this tutorial. Click here to download the required files. Note 1: Before proceeding further, start the services and components to start the Oracle Database, both the Teradata systems, and Oracle Data Integrator 10g (10.1.3.5) Agent on the Teradata primary system. Note 2: For the purposes of this tutorial, the ODI software and the source Oracle Database are installed on the same machine. Also, we will be using one ODI Agent to service both Teradata systems. 4. Setup the ODI Environment 4.1. Configure ODI Topology The ODI Topology must be configured to access the Teradata systems, Oracle sources, Flat file sources and the ODI Agents. The ODI Topology defines the Physical Architecture , the Logical Architecture as well as the Contexts that associate a logical architecture with the physical. © 2010 Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 4 " id="pdf-obj-3-6" src="pdf-obj-3-6.jpg">

Figure 2: Scenario 2 - Load and Transform on Primary and Replicate to Secondary

  • 3. Installation and Setup Prerequisites

Before you start the tasks, make sure that the following requirements have been met:

The system should include the following installed products:

 

o

Oracle Database (10g, 11g or 10gXE)

o

Oracle Data Integrator 10g (10.1.3.5)

o

Primary Teradata system

o

Secondary Teradata system

o

Oracle Data Integrator 10g (10.1.3.5) Agent installations on primary Teradata

You should have successfully completed the Oracle by Example (OBE) titled Creating and Connecting to ODI Master and Work Repositoriesbefore attempting this tutorial. Click here to access this OBE.

You should have downloaded the zip file that contains the ODI seed projects and models used in this tutorial. Click here to download the required files.

Note 1: Before proceeding further, start the services and components to start the Oracle Database, both the Teradata systems, and Oracle Data Integrator 10g (10.1.3.5) Agent on the Teradata primary system.

Note 2: For the purposes of this tutorial, the ODI software and the source Oracle Database are installed on the same machine. Also, we will be using one ODI Agent to service both Teradata systems.

  • 4. Setup the ODI Environment

    • 4.1. Configure ODI Topology

The ODI Topology must be configured to access the Teradata systems, Oracle sources, Flat file sources and the ODI Agents. The ODI Topology defines the Physical Architecture, the Logical Architecture as well as the Contexts that associate a logical architecture with the physical.

Using ODI for Teradata Dual-Loads

The following sections provide the steps necessary to modify the Topology.

  • 4.1.1. Login to the Repository using Topology Manager

Click on Start -> All Programs -> Oracle -> Oracle Data Integrator -> Topology Manager” from the Start Menu.

Login to the repository using the “SUPERVISOR” account.

Using ODI for Teradata Dual-Loads The following sections provide the steps necessary to modify the Topology.
  • 4.1.2. Create Contexts

For the purpose of this lab, we will create two contexts identifying the primary and secondary Teradata systems. All loads that run on the primary Teradata system will run under the “Primary” context and the ones for secondary Teradata system will run under the “Secondary” context. The extracts from the Oracle source will run under the “Global” context.

  • 4.1.2.1. Create the Primary Context: This context will be associated with the primary

Teradata system - TD1.

Click on the “Contexts” tab from the Topology Manger window. Click on the Insert

Context

Using ODI for Teradata Dual-Loads The following sections provide the steps necessary to modify the Topology.

icon to get the dialog-box to enter the new context.

Note: If the Insert Context icon is not visible, then from the Topology Manager, click on “Windows -> Show View -> Contexts”

Click on the “Definition” tab and enter the name for the new context as “Primary”. Leave all other items as default.

Using ODI for Teradata Dual-Loads The following sections provide the steps necessary to modify the Topology.

Click “OK” to create the primary context.

  • 4.1.2.2. Create the Secondary Context: This context will be associated with the secondary

Teradata system - TD2.

Using ODI for Teradata Dual-Loads

Click on the Insert Contexticon

Using ODI for Teradata Dual-Loads Click on the “ Insert Context ” icon again to get

again to get the dialog-box to enter the new context.

Then click on the “Definition” tab. Enter the name as “Secondary”. Leave all other items as default. Click “OK” to create the Secondary Context.

The Context pane should look similar to the one below:

Using ODI for Teradata Dual-Loads Click on the “ Insert Context ” icon again to get
  • 4.1.3. Configure ODI Agents in the Topology Note: Make sure the ODI Agent is started on the primary Teradata system before proceeding with this step.

    • 4.1.3.1. Click on “Physical Architecture” tab, right-click on “Agents” and select “Insert

Agent”.

Using ODI for Teradata Dual-Loads Click on the “ Insert Context ” icon again to get
  • 4.1.3.2. Enter the below information for the Agent.

Name

TD1_Agent

Host

<Teradata_Host>

Where <Teradata_Host> is the hostname or IP Address of the primary Teradata system. Leave all other information as default.

Using ODI for Teradata Dual-Loads Click on the “ Insert Context ” icon again to get

Using ODI for Teradata Dual-Loads

Click “Apply” and then click “Test” and you should see the below message:

Using ODI for Teradata Dual-Loads Click “Apply” and then click “Test” and you should see the

Click “OK” to close the message, and “OK” to close the Agent configuration window.

You should end up with the below Agent in the Physical Architecture tab.

Using ODI for Teradata Dual-Loads Click “Apply” and then click “Test” and you should see the

Note: In this tutorial, we will be configuring only one ODI Agent on the primary Teradata system only. The same agent will handle execution of jobs on the secondary system.

Note: The source Oracle database runs on the same system as the ODI software, there is no need to configure an Agent for the source.

  • 4.1.3.3. Click on the “Logical Architecture” tab and right-click “Agents” and select “Insert

Logical Agent”.

Using ODI for Teradata Dual-Loads Click “Apply” and then click “Test” and you should see the
  • 4.1.3.4. Enter “TD_Agent” as the name.

Associate “TD1_Agent” to the Primary and the Secondary contexts. Click “OK”.

Using ODI for Teradata Dual-Loads Click “Apply” and then click “Test” and you should see the
  • 4.1.3.5. You should end-up with one Logical Agent as shown below:

Using ODI for Teradata Dual-Loads

Using ODI for Teradata Dual-Loads 4.1.4. Setup and configure the Teradata systems in the ODI Topology.
  • 4.1.4. Setup and configure the Teradata systems in the ODI Topology.

    • 4.1.4.1. Configure the primary Teradata system.

Select the “Physical Architecture” tab and expand the “Technologies” folder, right-click on “Teradata” and select “Insert Data Server.

Using ODI for Teradata Dual-Loads 4.1.4. Setup and configure the Teradata systems in the ODI Topology.
  • 4.1.4.2. In the Data Server New window, click on the “Definition” tab and enter the below

information.

Using ODI for Teradata Dual-Loads 4.1.4. Setup and configure the Teradata systems in the ODI Topology.

Using ODI for Teradata Dual-Loads

Name

TD1

Server (Data Server)

<Teradata_Server>

User

DBC

Password

Dbc

Where <Teradata_Server> is the hostname or IP Address of the Primary Teradata system.

  • 4.1.4.3. Click the JDBCtab. Click the ellipses (…) to enter the JDBC Driver information.

Select the “NCR Teradata JDBC Driver” from the drop-down.

Using ODI for Teradata Dual-Loads Name TD1 Server (Data Server) <Teradata_Server> User DBC Password Dbc Where

Click “OK” on the Drivers dialog box.

Important Note: Make sure that you copy the NCR Teradata JDBC Drivers (mainly the terajdbc4.jar, tdgssconfig.jar and tdgssjava.jar files) to the ODI_HOME/oracledi/drivers directory. ODI needs the drivers placed in this location in order to connect to Teradata via JDBC.

  • 4.1.4.4. Enter the JDBC URL.

The URL is of the form “jdbc:teradata://<host>:<port>/<server>”, where <host> is the hostname of the Teradata system, <port> the port it’s configured to listen on, and <server> is the server name. The below example assumes that the staging tables are created in the “financial” schema.

Note: If you do not have the “financial” Schema created in the Teradata environments, then you should create it prior to proceeding with the rest of the steps. Alternatively, you could replace the financial schema referenced in this tutorial with the schema of your choice.

Using ODI for Teradata Dual-Loads

Using ODI for Teradata Dual-Loads Click on “Test” to test the connection. Select the “TD1_Agent” from

Click on “Test” to test the connection.

Using ODI for Teradata Dual-Loads Click on “Test” to test the connection. Select the “TD1_Agent” from

Select the “TD1_Agent” from the drop-down. Click “Test” again. You should get a successful connection message like the one below.

Using ODI for Teradata Dual-Loads Click on “Test” to test the connection. Select the “TD1_Agent” from

Click “OK”, then click “OK” again to close the Data Server New window. This will add the new Teradata server “TD1” to the Physical Architecture.

  • 4.1.4.5. Set the physical schema and the work schema for the TD1 connection.

In the Properties box that opens, select “financial” from the “Schema (Schema)” drop-down and “financial” from the “Schema (Work Schema)” drop-down.

Using ODI for Teradata Dual-Loads

Using ODI for Teradata Dual-Loads Click “OK” . Click “OK” again when prompted for “ No

Click “OK”. Click “OK” again when prompted for No Context specified for this Schema. We will create the context in the later steps.

4.1.4.6.

Configure the Secondary Teradata System.

 

Repeat Steps 4.1.4.1 through 4.1.4.5 in order to configure the secondary Teradata system. Modify the hostname or the IP address of the secondary system as appropriate. Also note that we will be using TD1_Agent to connect to this system.

Note: If you wish to configure the secondary Teradata system on a different schema on the primary Teradata system, then replace the “financial” schema identified in steps 4.1.4.1 through 4.1.4.5 with the schema of your choice.

Name

TD2

 

Server (Data Server)

<Teradata_Server>

User

DBC

Password

Dbc

Where <Teradata_Server> is the hostname or IP Address of the Primary Teradata system.

4.1.4.7.

Create the Teradata Logical Schema.

Click on the “Logical Architecturetab and expand “Technologies -> Teradata”. Right-click on “Teradata” and select “Insert Logical Schema”.

Using ODI for Teradata Dual-Loads

Using ODI for Teradata Dual-Loads 4.1.4.8. Enter “TD” as the name. Select “TD1.financial” physical schema for
  • 4.1.4.8. Enter “TD” as the name. Select “TD1.financial” physical schema for the Primary

context and “TD2.financial” physical schema for the Secondary context. Leave other

selections as default.

Using ODI for Teradata Dual-Loads 4.1.4.8. Enter “TD” as the name. Select “TD1.financial” physical schema for

Click “OK”. The logical schema TD should be created as shown below.

Using ODI for Teradata Dual-Loads 4.1.4.8. Enter “TD” as the name. Select “TD1.financial” physical schema for
  • 4.1.5. Configure the Oracle Data source Physical and Logical Schemas.

    • 4.1.5.1. Create the Oracle schema for storing the source tables. This tutorial will use the

sales history (SH) schema that comes pre-installed with the Oracle example schemas.

SQL> create user SH identified by <password> default tablespace USERS temporary tablespace TEMP;

SQL> grant connect, resource to SH;

  • 4.1.5.2. Select the “Physical Architecture” tab and expand the “Technologies” folder,

right-click on “Oracle” and select “Insert Data Server”.

In the Data Server New window, click on the “Definition” tab and enter the below information.

Name

<Oracle_SID>

Using ODI for Teradata Dual-Loads

Server (Data Server)

<host>

User

SH

Password

<password>

Where <Oracle_SID> is the SID of the source Oracle system, <host> is the hostname or IP Address of the Oracle system, <password> is the password for SH user.

  • 4.1.5.3. Click the JDBC tab. Click the ellipses (…) to enter the JDBC Driver information.

Select the Oracle JDBC Driver” from the drop-down.

Using ODI for Teradata Dual-Loads Server (Data Server) <host> User SH Password <password> Where <Oracle_SID> is

Click “OK” on the Drivers dialog box.

  • 4.1.5.4. Enter the JDBC URL.

Click the ellipses (…) besides the JDBC URL to enter the JDBC URL information.

Select the “jdbc:oracle:thin@<host>:<port>:<sid>” from the drop-down. Where <host> is the server running the Oracle database, <port> is the listener port, and <sid> is the Database SID.

Using ODI for Teradata Dual-Loads Server (Data Server) <host> User SH Password <password> Where <Oracle_SID> is

Enter the JDBC URL that is appropriate for your environment. The URL used for this tutorial is “jdbc:oracle:thin:@xpobi-1-vm:1521:ORCL”. This means that the Oracle database server is running on xpobi-1-vm, the listener is configured on the default 1521 port, and ORCL is the database SID.

Using ODI for Teradata Dual-Loads

Using ODI for Teradata Dual-Loads Click on “Test” to test the connection. Select the “ Local

Click on “Test” to test the connection.

Using ODI for Teradata Dual-Loads Click on “Test” to test the connection. Select the “ Local

Select the Local (No Agent)from the drop-down. Click “Test” again. You should get a Successful connection window like the one below.

Using ODI for Teradata Dual-Loads Click on “Test” to test the connection. Select the “ Local

Click “OK”, then click “OK” again to close the Data Server New window. This will add the new Oracle server ORCLto the Physical Architecture.

  • 4.1.5.5. Set the Physical Schema and the Work Schema for the ORCL connection.

In the Properties box that opens, select SHfrom the “Schema (Schema)” drop-down and SHfrom the “Schema (Work Schema)” drop-down.

Using ODI for Teradata Dual-Loads

Using ODI for Teradata Dual-Loads Click “OK” and click “OK” again when prompted for “ No

Click “OK” and click “OK” again when prompted for No Context specified for this Schema. We will create the context in the later steps.

  • 4.1.5.6. Create the Oracle Logical Schema.

Click on the “Logical Architecture” tab, expand “Technologies -> Oracle”. Right-click on “Oracle” and select “Insert Logical Schema”.

Enter “ORCL_SHas the name. Select “ORCL.SH” as the Physical Schema for “Global”, “Primary” and the “Secondarycontexts.

Using ODI for Teradata Dual-Loads Click “OK” and click “OK” again when prompted for “ No

Click “OK”. The logical schema ORCL should be created as shown below.

Using ODI for Teradata Dual-Loads

Using ODI for Teradata Dual-Loads 4.1.6. Configure the Physical and Logical Schemas for the Flat Filehere to access this OBE. 4.1.6.1. Configure the Flat File Physical Source. Select the “Physical Architecture” tab and expand the “Technologies -> File ” folder, and double-click on the “FILE_GENERIC” source. 4.1.6.2. In the Data Server New window, click on the “Definition” tab and enter the below information. Name FILE_GENERIC Server (Data Server) <host> Where <host> is the hostname or IP Address of the Host that will store the files. 4.1.6.3. Click on the JDBC tab and make sure that the “JDBC Driver” and “JDBC URL” is selected as shown below. © 2010 Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 16 " id="pdf-obj-15-6" src="pdf-obj-15-6.jpg">
  • 4.1.6. Configure the Physical and Logical Schemas for the Flat File Source.

In this step, we will be modifying the existing FILE_GENERIC source that comes pre-installed in the ODI Topology.

Note: If you do not see a FILE_GENERIC source in your tree then add one. For instructions on adding File sources in ODI, refer to Oracle by example Creating ODI Project and Interface: Exporting Relational Table to Flat file”. Click here to access this OBE.

  • 4.1.6.1. Configure the Flat File Physical Source.

Select the “Physical Architecture” tab and expand the “Technologies -> Filefolder, and double-click on the “FILE_GENERIC” source.

Using ODI for Teradata Dual-Loads 4.1.6. Configure the Physical and Logical Schemas for the Flat Filehere to access this OBE. 4.1.6.1. Configure the Flat File Physical Source. Select the “Physical Architecture” tab and expand the “Technologies -> File ” folder, and double-click on the “FILE_GENERIC” source. 4.1.6.2. In the Data Server New window, click on the “Definition” tab and enter the below information. Name FILE_GENERIC Server (Data Server) <host> Where <host> is the hostname or IP Address of the Host that will store the files. 4.1.6.3. Click on the JDBC tab and make sure that the “JDBC Driver” and “JDBC URL” is selected as shown below. © 2010 Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 16 " id="pdf-obj-15-38" src="pdf-obj-15-38.jpg">
  • 4.1.6.2. In the Data Server New window, click on the “Definition” tab and enter the below

information.

Name

FILE_GENERIC

Server (Data Server)

<host>

Where <host> is the hostname or IP Address of the Host that will store the files.

Using ODI for Teradata Dual-Loads 4.1.6. Configure the Physical and Logical Schemas for the Flat Filehere to access this OBE. 4.1.6.1. Configure the Flat File Physical Source. Select the “Physical Architecture” tab and expand the “Technologies -> File ” folder, and double-click on the “FILE_GENERIC” source. 4.1.6.2. In the Data Server New window, click on the “Definition” tab and enter the below information. Name FILE_GENERIC Server (Data Server) <host> Where <host> is the hostname or IP Address of the Host that will store the files. 4.1.6.3. Click on the JDBC tab and make sure that the “JDBC Driver” and “JDBC URL” is selected as shown below. © 2010 Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 16 " id="pdf-obj-15-66" src="pdf-obj-15-66.jpg">
  • 4.1.6.3. Click on the JDBC tab and make sure that the “JDBC Driver” and “JDBC URL” is

selected as shown below.

Using ODI for Teradata Dual-Loads

Using ODI for Teradata Dual-Loads Click on “Test” and select “Local (No Agent)” for the test.

Click on “Test” and select “Local (No Agent)” for the test. You should get a “Successful Connection” message displayed. Click “OK” to clear the message.

Click “OK” to save the File Physical Source.

  • 4.1.6.4. Set the physical schema and the work schema for the FILE_GENERIC connection.

Double-click on the “FILE_GENERIC .. FILE_GENERIC source.

/demo/file”

Physical Schema under the

Using ODI for Teradata Dual-Loads Click on “Test” and select “Local (No Agent)” for the test.

In the Properties box that opens, enter “/tmp” for the “Schema (Schema)” and the “Schema (Work Schema)” fields. Leave all other fields as defaults.

Using ODI for Teradata Dual-Loads Click on “Test” and select “Local (No Agent)” for the test.

Using ODI for Teradata Dual-Loads

Click “OK” to save your changes.

  • 4.1.6.5. Create the Flat File Logical Source.

Click on the “Logical Architecture” tab, expand “Technologies -> File”. Double-click on FILE_DEMO_SRC.

Note: If you do not see a “FILE_DEMO_SRC” in your Topology then create one. Refer to the OBE “Exporting a flat file to a relational table” for more details on creating file sources.

Using ODI for Teradata Dual-Loads Click “OK” to save your changes. 4.1.6.5. Create the Flat File“Exporting a flat file to a relational table” for more details on creating file sources. 4.1.6.6. Select “ FILE_GENERIC ./tmp ” as the Physical Schemas for “Global” , “Primary” and the “Secondary” contexts. Click “OK” to save your changes. 4.2. Setup the ODI Designer environment. 4.2.1. Login to the Repository using ODI Designer . Click on “Start -> All Programs -> Oracle -> Oracle Data Integrator - > Designer” from the Start Menu. Login to the repository using the “SUPERVISOR” account. 4.2.2. Import the seed Project supplied in the zip file. The seed project “PROJ_TD_Dual_Load.xml” has the following pre-created items:  The required Teradata Knowledge Modules for dual-load  Variables required by the dual-load examples © 2010 Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 18 " id="pdf-obj-17-32" src="pdf-obj-17-32.jpg">
  • 4.1.6.6. Select FILE_GENERIC ./tmpas the Physical Schemas for “Global”, “Primary”

and the “Secondary” contexts.

Using ODI for Teradata Dual-Loads Click “OK” to save your changes. 4.1.6.5. Create the Flat File“Exporting a flat file to a relational table” for more details on creating file sources. 4.1.6.6. Select “ FILE_GENERIC ./tmp ” as the Physical Schemas for “Global” , “Primary” and the “Secondary” contexts. Click “OK” to save your changes. 4.2. Setup the ODI Designer environment. 4.2.1. Login to the Repository using ODI Designer . Click on “Start -> All Programs -> Oracle -> Oracle Data Integrator - > Designer” from the Start Menu. Login to the repository using the “SUPERVISOR” account. 4.2.2. Import the seed Project supplied in the zip file. The seed project “PROJ_TD_Dual_Load.xml” has the following pre-created items:  The required Teradata Knowledge Modules for dual-load  Variables required by the dual-load examples © 2010 Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 18 " id="pdf-obj-17-50" src="pdf-obj-17-50.jpg">

Click “OK” to save your changes.

  • 4.2. Setup the ODI Designer environment.

    • 4.2.1. Login to the Repository using ODI Designer. Click on “Start -> All Programs -> Oracle ->

Oracle Data Integrator -> Designer” from the Start Menu.

Login to the repository using the “SUPERVISOR” account.

Using ODI for Teradata Dual-Loads Click “OK” to save your changes. 4.1.6.5. Create the Flat File“Exporting a flat file to a relational table” for more details on creating file sources. 4.1.6.6. Select “ FILE_GENERIC ./tmp ” as the Physical Schemas for “Global” , “Primary” and the “Secondary” contexts. Click “OK” to save your changes. 4.2. Setup the ODI Designer environment. 4.2.1. Login to the Repository using ODI Designer . Click on “Start -> All Programs -> Oracle -> Oracle Data Integrator - > Designer” from the Start Menu. Login to the repository using the “SUPERVISOR” account. 4.2.2. Import the seed Project supplied in the zip file. The seed project “PROJ_TD_Dual_Load.xml” has the following pre-created items:  The required Teradata Knowledge Modules for dual-load  Variables required by the dual-load examples © 2010 Oracle Corporation Using ODI for Teradata Dual-Loads Pg. 18 " id="pdf-obj-17-73" src="pdf-obj-17-73.jpg">
  • 4.2.2. Import the seed Project supplied in the zip file.

The seed project “PROJ_TD_Dual_Load.xml” has the following pre-created items:

The required Teradata Knowledge Modules for dual-load

Variables required by the dual-load examples

Using ODI for Teradata Dual-Loads

Procedures to create the Oracle Table SRC_SALES_PERSON which is the source table

Procedures to create the Teradata table TD_SALES_PERSON & TD_REPL_SALES_PERSON

which are the destination tables in Teradata Systems Packages to initialize the Variables that are needed for the ELT

Packages to incrementally load into the Source table

Procedure to get the max Batch ID that the source extract is currently using

Right-click on the blank area in the Projects pane of ODI Designer and select “Import Project”.

Alternatively you can click on the Import Project icon

Using ODI for Teradata Dual-Loads  Procedures to create the Oracle Table SRC_SALES_PERSON which is the

as well.

Using ODI for Teradata Dual-Loads  Procedures to create the Oracle Table SRC_SALES_PERSON which is the
  • 4.2.3. Click on ellipses (…) to select the “Projects” directory from the supplied zip file. Select the

“TD_Dual_Load” as the project file to import.

Using ODI for Teradata Dual-Loads  Procedures to create the Oracle Table SRC_SALES_PERSON which is the

Click “OK” to import. After the import the TD_Dual_Load Project tree should expand like shown below.

Using ODI for Teradata Dual-Loads

Using ODI for Teradata Dual-Loads 4.2.4. Import the Teradata, Oracle and File models. The supplied Teradata,
  • 4.2.4. Import the Teradata, Oracle and File models.

The supplied Teradata, Flat File and Oracle models contains the below objects:

SRC_SALES_PERSON: This is the source table for the dual-load and resides in the Oracle

Database. The table contains a DATE_UPDATED column that will be used as the basis of the Changed Data Capture process. DL_VARIABLES: This table stores the value of the latest “Batch IDthat is associated with

the Oracle extract. The purpose of this is to pass the variable values between different ODI contexts. TD_SALES_PERSON: This is the destination table that exists in the Primary and Secondary Teradata Systems. This table contains a derived column FULL_NAME that is not present in the Source and a simple ETL map will derive this value.

Using ODI for Teradata Dual-Loads

TD_REPL_SALES_PERSON: This is the destination table that exists in Secondary Teradata

System. This table will be replicated from the Primary Teradata TD_SALES_PERSON table when performing dual-loads using scenario 2. File_Demo_Src: This is the intermediary Flat File that will be created to store the Oracle source table SRC_SALES_PERSON.

Click on the Models tab of ODI Designer. Right-click on the blank area in the Models pane and select

“Import Model Folder. Alternatively you can click on the Import Model Folder icon

Using ODI for Teradata Dual-Loads  TD_REPL_SALES_PERSON : This is the destination table that exists in

as well.

Using ODI for Teradata Dual-Loads  TD_REPL_SALES_PERSON : This is the destination table that exists in
  • 4.2.5. Click on ellipses (…) to select the Modelsdirectory from the supplied zip file. Select the Flat

Files, “Oracle” and “Teradata” as the models to import holding the CTRL key.

Using ODI for Teradata Dual-Loads  TD_REPL_SALES_PERSON : This is the destination table that exists in

Click “OK” to start the import. After the import is done, the model pane should expand as shown below.

Using ODI for Teradata Dual-Loads

Using ODI for Teradata Dual-Loads 5. Setup the Source and Destination Databases In this step, we
  • 5. Setup the Source and Destination Databases

In this step, we will create the source and the destination Database tables using the supplied ODI Procedures. We will also run the ODI Procedures that will populate the source table with initial data. We also need to run ODI Packages to initialize the variables used by ODI Interfaces and Packages.

  • 5.1. Create Oracle tables and populate with seed data.

The procedure “Create_Oracle_Tables” has been supplied with the seed project to create and populate the Oracle schema. Follow the below steps to execute the supplied procedure.

  • 5.1.1. In the ODI Designer “Projects” pane, expand “TD_Dual_Load -> 1. Setup -> Procedures”.

Right-click on “Create Oracle Tables” and select “Execute”.

Using ODI for Teradata Dual-Loads

Using ODI for Teradata Dual-Loads Select the “Global” context, and “Local (No Agent)” agent. Click “OK”

Select the “Global” context, and “Local (No Agent)” agent. Click “OK” to execute the procedure. The Oracle physical schema “ORCL_SH” has been associated with the Global context.

Using ODI for Teradata Dual-Loads Select the “Global” context, and “Local (No Agent)” agent. Click “OK”

You should see a Session started message. Click “OK” to acknowledge the message.

  • 5.1.2. Verify the execution using ODI Operator. Click on the “Operator” icon

Using ODI for Teradata Dual-Loads Select the “Global” context, and “Local (No Agent)” agent. Click “OK”

from ODI Designer

menu.

In the Session List pane, expand “Date -> Today” and you should see a successful execution of the “Create Oracle Tablesprocedure.

Using ODI for Teradata Dual-Loads

Using ODI for Teradata Dual-Loads 5.2. Create Teradata tables. The procedure “Create_ Teradata _Tables” has been
  • 5.2. Create Teradata tables.

The procedure “Create_Teradata_Tables” has been supplied with the seed project to create the Teradata schema. Follow the below steps to execute the supplied procedure.

  • 5.2.1. In the ODI Designer “Projects” pane, expand “TD_Dual_Load -> 1. Setup -> Procedures”.

Right-click on “Create Teradata Tables” and select “Execute”.

Select the Primarycontext, and TD_Agentagent. Click “OK” to execute the procedure. The primary Teradata physical schema TD1has been associated with the Primary context.

Using ODI for Teradata Dual-Loads 5.2. Create Teradata tables. The procedure “Create_ Teradata _Tables” has been

You should see a Session started message. Click “OK” to acknowledge the message.

  • 5.2.2. Execute the “Create Teradata Tables” procedure again. And this time select the

“Secondary” context and “TD_Agent” agent to create the Teradata schema on the secondary Teradata system TD2.

Using ODI for Teradata Dual-Loads

Using ODI for Teradata Dual-Loads Click “OK” to execute the procedure. The secondary Teradata physical schema

Click “OK” to execute the procedure. The secondary Teradata physical schema TD2has been associated with the Secondary context.

Click “OK” to acknowledge the Session Started message.

  • 5.2.3. Verify the execution using ODI Operator. Click on the “Operator” icon

Using ODI for Teradata Dual-Loads Click “OK” to execute the procedure. The secondary Teradata physical schema

from ODI Designer

menu.

In the Session List pane, expand “Date -> Today” and you should see two successful executions of the “Create Teradata Tablesprocedures.

Using ODI for Teradata Dual-Loads Click “OK” to execute the procedure. The secondary Teradata physical schema
  • 5.3. Initialize Variables used by ODI.

Execute the Package “Initialize Variables” to initialize variables used by ODI mappings. The ODI Interfaces and Packages will depend upon seed values of these variables and skipping this step will lead to errors during execution of Interfaces and Packages later.

  • 5.3.1. In the ODI Designer “Projects” pane, expand “TD_Dual_Load -> 1. Setup -> Package.

Right-click on Initialize Variablesand select “Execute”.

Using ODI for Teradata Dual-Loads

Using ODI for Teradata Dual-Loads Select the “Global” context and “Local (No Agent)” agent. Click “OK”

Select the “Global” context and “Local (No Agent)” agent. Click “OK”. This will execute the package “Initialize Variables” against the Oracle SH Schema. Click “OK” to acknowledge the Session Started message.

You need to repeat this step and execute the same package against the primary and secondary Teradata sources. This you would do by repeating the previous step for each “Primary” and “Secondary” contexts and execute the package against the “TD_Agent” agent.

Using ODI for Teradata Dual-Loads Select the “Global” context and “Local (No Agent)” agent. Click “OK”
  • 5.3.2. Verify the execution using ODI Operator.

Expand “Date -> Today” and you should three successful executions of the Initialize Variablespackage.

Using ODI for Teradata Dual-Loads

Using ODI for Teradata Dual-Loads 6. Create Oracle to Flat File Interface In this section, we
  • 6. Create Oracle to Flat File Interface

In this section, we will create ODI Interfaces and Packages that will extract the data out of the Oracle source and into a flat file. This step is common for both the Teradata dual-load scenarios.

The flat file is staged in a directory that is created when the extract process runs, and is identified uniquely by the Batch ID of the extract process. This mechanism allows us to run the extracts independently of the Teradata import process hence making the extract process completely asynchronous with the loads.

The extracts out of the Oracle SRC_SALES_PERSON table looks at the new rows inserted or modified since the time the last extract was run. The changed data capture mechanism is accomplished by using the DATE_UPDATED column on the SRC_SALES_PERSON table. The column is updated at every inserts or updates at the source database.

At the end of the Extract process, there will be three files created in the directory “SRC_SALES_PERSON.txt” which is the text dump of SRC_SALES_PERSON table, the “Primary.txt” and “Secondary.txt” files which are used as a flags to indicate whether or not the dump file was loaded into the Primary or the Secondary Teradata Systems respectively. These files are deleted after the dump is successfully loaded into its respective destination.

Using ODI for Teradata Dual-Loads

  • 6.1. Create a New Folder for the Extract process.

    • 6.1.1. In the ODI Designer “Projects” pane, right-click on “TD_Dual_Load” and select “Insert

Folder”.

Using ODI for Teradata Dual-Loads 6.1. Create a New Folder for the Extract process. 6.1.1. In
  • 6.1.2. Enter “2. Extract” as the name and click OK.

Using ODI for Teradata Dual-Loads 6.1. Create a New Folder for the Extract process. 6.1.1. In
  • 6.2. Create an Oracle Table to Flat file Interface.

    • 6.2.1. Expand the newly created “2. Extract” folder and right-click on “Interfaces” and select “Insert

Interface”. We will be creating the Interface to extract an Oracle table to a Flat file.

Using ODI for Teradata Dual-Loads

Using ODI for Teradata Dual-Loads 6.2.2. Enter “Oracle to File” in the Name field and set
  • 6.2.2. Enter “Oracle to File” in the Name field and set the Optimization Context as “Global”.

Check the “Staging Area Different From Target” and select “ORCL_SH” to indicate that the staging area for any temporary objects needed by the ELT is going to be the ORCL_SH schema.

Using ODI for Teradata Dual-Loads 6.2.2. Enter “Oracle to File” in the Name field and set

Click on the “Diagram” tab. And click “OK” if prompted for the below message.

Using ODI for Teradata Dual-Loads 6.2.2. Enter “Oracle to File” in the Name field and set
  • 6.2.3. Create the ELT map between SRC_SALES_PERSON table in ORCL_SH schema to

SRC_SALES_PERSON Flat file.

In the left pane of ODI Designer, click on “Models” tab. Expand the “Oracle -> ORCL_SH” model folder.

Using ODI for Teradata Dual-Loads

Using ODI for Teradata Dual-Loads Drag and drop the SRC_SALES_PERSON table to the Sources section of

Drag and drop the SRC_SALES_PERSON table to the Sources section of the Diagram as shown below.

Using ODI for Teradata Dual-Loads Drag and drop the SRC_SALES_PERSON table to the Sources section of

Next, expand the model folder “Flat Files -> File_Demo_Src”.

Using ODI for Teradata Dual-Loads Drag and drop the SRC_SALES_PERSON table to the Sources section of

Drag and drop the SRC_SALES_PERSON file from File_Demo_Src to the Target Datastore section of the diagram. Click “Yes” when prompted for Automatic Mapping as shown below.

Using ODI for Teradata Dual-Loads

Using ODI for Teradata Dual-Loads You should see Source and the Target columns mapped as shown

You should see Source and the Target columns mapped as shown in the below diagram.

Using ODI for Teradata Dual-Loads You should see Source and the Target columns mapped as shown
  • 6.2.4. Create a filter on DATE_UPDATED column. This is needed to implement the change data

capture logic for extracting only the newly inserted rows or modified rows from SRC_SALES_PERSON

table.

In order to create a filter, drag and drop the DATE_UPDATED column out of the SRC_SALES_PERSON table in the Sources section and into the blank area on the Sources pane.

Click on the

Using ODI for Teradata Dual-Loads You should see Source and the Target columns mapped as shown

filer icon. Make sure the filter is selected to be active by checking the “Active Filter”

box. Click on the Implementation tab and Enter:

SRC_SALES_PERSON.DATE_UPDATED BETWEEN TO_TIMESTAMP('#LastRefresh','MM/DD/YYYY HH24:MI:SS.FF') AND TO_TIMESTAMP('#CurrentTimestamp','MM/DD?YYYY HH24:MI:SS.FF')

Using ODI for Teradata Dual-Loads

Using ODI for Teradata Dual-Loads Click on the “Database” icon to check for syntactical errors in

Click on the “Database”

Using ODI for Teradata Dual-Loads Click on the “Database” icon to check for syntactical errors in

icon to check for syntactical errors in the filter expression. You should see

the below message that says that the SQL is valid for the DBMS.

Using ODI for Teradata Dual-Loads Click on the “Database” icon to check for syntactical errors in

Click “OK”. Then click on “Errors” to check for errors in the mapping. You should see the below text box stating that there were no errors detected in the diagram.

Using ODI for Teradata Dual-Loads Click on the “Database” icon to check for syntactical errors in

Click “OK” to close the text box.

  • 6.2.5. Click on the “Flow” tab. This is where we will be assigning the appropriate knowledge modules

for the ELT.

Click on “Target (FILE_GENERIC)” table. In the Properties section below, make sure the following values are set:

IKM

IKM SQL to File Append

INSERT

Yes

TRUNCATE

Yes

GENERATE_HEADER

No

Using ODI for Teradata Dual-Loads

Using ODI for Teradata Dual-Loads Click “OK” to save the Oracle to File Interface. 6.3. Create

Click “OK” to save the Oracle to File Interface.

  • 6.3. Create a package that will call the “Oracle to FileInterface.

    • 6.3.1. Create a new package. In the Projects pane of ODI Designer, expand TD_Dual_Load -> 2.

Extract. Right-click on “Packages” and select “Insert Package”.

Using ODI for Teradata Dual-Loads Click “OK” to save the Oracle to File Interface. 6.3. Create

Click on the “Definition” tab of the “Package: Newwindow and enter “Extract” as the name.

Using ODI for Teradata Dual-Loads Click “OK” to save the Oracle to File Interface. 6.3. Create
  • 6.3.2. Create the package flow. Click on the “Diagram” tab of the “Package: New” window. On the

Projects pane, expand “TD_Dual_Load -> Variables” folder. Drag and drop “BatchID” variable to the

Diagram pane as shown below.

Using ODI for Teradata Dual-Loads

Using ODI for Teradata Dual-Loads 6.3.3. Similarly, drag and drop “PathID” , “FilePath” , “LastRefresh” ,
  • 6.3.3. Similarly, drag and drop “PathID”, “FilePath”, “LastRefresh”, “CurrentTimestamp”,

“BatchID” and “LastRefresh” from the Variables folder to the Diagram pane. Note that some of the

variables are used multiple times in the diagram, and that is not a typo.

Next, drag and drop “Oracle to File” Interface from “TD_Dual_Load -> 2.Extract -> Interfaces” folder, and “Update MaxID” from “TD_Dual_Load -> 1.Setup -> Procedures” folder.

Next, add “Files -> OdiMkdir”, File -> OdiOutFile(twice) from the Toolbox pane. The way to add items from the Toolbox is to first click on the tool you want to add, and then click again on the Diagram

pane to add the tool to the diagram. When you are done adding tools, then click on the adding.

Using ODI for Teradata Dual-Loads 6.3.3. Similarly, drag and drop “PathID” , “FilePath” , “LastRefresh” ,

icon to stop

Once you are done adding all the elements, arrange the icons in the diagram such that they look similar to the picture below.

Using ODI for Teradata Dual-Loads

Using ODI for Teradata Dual-Loads 6.3.4. Add the “Next steps on Success” ( ) arrows between
  • 6.3.4. Add the “Next steps on Success” (

Using ODI for Teradata Dual-Loads 6.3.4. Add the “Next steps on Success” ( ) arrows between

) arrows between all the steps. It is important to ensure

the flow is exactly as it looks in the below diagram.

Using ODI for Teradata Dual-Loads 6.3.4. Add the “Next steps on Success” ( ) arrows between

Click on the selection pointer

Using ODI for Teradata Dual-Loads 6.3.4. Add the “Next steps on Success” ( ) arrows between

before continuing with the next steps.

  • 6.3.5. Set properties for each step. We will start with BatchID (top left in the diagram) which is the first

step that will be executed and just go along with the next step in the sequence.

  • 6.3.6. Set properties for BatchID. Highlight the BatchID step in the diagram and click on the

“General” tab of the Properties section. Make sure you set the properties exactly as shown below.

Using ODI for Teradata Dual-Loads

Using ODI for Teradata Dual-Loads 6.3.7. Set properties for PathID . Highlight the PathID step in
  • 6.3.7. Set properties for PathID. Highlight the PathID step in the diagram and set the properties as

shown below.

Using ODI for Teradata Dual-Loads 6.3.7. Set properties for PathID . Highlight the PathID step in
  • 6.3.8. Set properties for FilePath. Highlight the FilePath step in the diagram and set the properties as

shown below.

Using ODI for Teradata Dual-Loads 6.3.7. Set properties for PathID . Highlight the PathID step in
  • 6.3.9. Set properties for OdiMkDir1. Highlight the OdiMkDir1 step in the diagram and set the

properties as shown below.

Using ODI for Teradata Dual-Loads

Using ODI for Teradata Dual-Loads 6.3.10. Set properties for LastRefresh . Highlight the LastRefresh step in
  • 6.3.10. Set properties for LastRefresh. Highlight the LastRefresh step in the diagram and set the

properties as shown below.

Using ODI for Teradata Dual-Loads 6.3.10. Set properties for LastRefresh . Highlight the LastRefresh step in
  • 6.3.11. Set properties for CurrentTimestamp. Highlight the CurrentTimestamp step in the diagram

and set the properties as shown below.

Using ODI for Teradata Dual-Loads 6.3.10. Set properties for LastRefresh . Highlight the LastRefresh step in
  • 6.3.12. Set properties for Oracle to File. Highlight the Oracle to File step in the diagram and set the

properties as shown below.

Using ODI for Teradata Dual-Loads 6.3.10. Set properties for LastRefresh . Highlight the LastRefresh step in
  • 6.3.13. Set properties for OdiOutFile2. Highlight the OdiOutFile2 step in the diagram and enter the

Target File name. Make sure you hit “Enter” to make your values persist in text boxes.

Using ODI for Teradata Dual-Loads

Using ODI for Teradata Dual-Loads Click on the ellipses (…) next to the “Text” field. This

Click on the ellipses (…) next to the “Text” field. This will open the Expression Editor tool. In the top box, enter the text “Primary” as shown below.

Using ODI for Teradata Dual-Loads Click on the ellipses (…) next to the “Text” field. This

Click “OK” to save your expression.

  • 6.3.14. Set properties for OdiOutFile3. Highlight the OdiOutFile3 step in the diagram and enter the

properties as shown below. Follow the instructions mentioned in the previous step.

Using ODI for Teradata Dual-Loads Click on the ellipses (…) next to the “Text” field. This
  • 6.3.15. Set properties for Update MaxID. Highlight the Update MaxID step in the diagram and set the

properties as shown below.

Using ODI for Teradata Dual-Loads

Using ODI for Teradata Dual-Loads 6.3.16. Set properties for BatchID . Highlight the second BatchID step
  • 6.3.16. Set properties for BatchID. Highlight the second BatchID step in the diagram and set the

properties as shown below. This time we will refresh the BatchID variable.

Using ODI for Teradata Dual-Loads 6.3.16. Set properties for BatchID . Highlight the second BatchID step
  • 6.3.17. Set properties for LastRefresh. Highlight the second LastRefresh step in the diagram and set

the properties as shown below. This time we will set the LastRefresh variable to the CurrentTimestamp.

Using ODI for Teradata Dual-Loads 6.3.16. Set properties for BatchID . Highlight the second BatchID step
  • 6.3.18. Once done setting the properties of all the steps, click “OK” to create the Extract package.

  • 7. Create E-LT Interfaces for Scenario 1

In this section, we will create ODI Interfaces and Packages that will load the data from the flat file that was extracted in the prior steps, into the Teradata systems.

The Load process utilizes the Batch ID of the extract process and figures out how many batches of the extract it has to load into the Teradata system in order to keep it in sync with the source system. This process is completely asynchronous with the extracts and can be run at different frequencies altogether. Also if the load process fails for any reason, it can be re-executed and the load will pick up all the missed extracts since the last successful run.

Using ODI for Teradata Dual-Loads

The Primary.txt and Secondary.txt files that will be created by the extract process for each Batch ID, will be deleted by the load process upon a successful load of that batch. The existence (or the non-existence) of these files can utilized in the cleanup process to indicates which batches have been consumed by the Primary and the Secondary Teradata systems and are potential candidates for cleanup. If an extract file has been loaded in both the Primary and the Secondary Teradata system, then it is no longer needed and can be deleted or archived.

We will be developing one Interface and one Package to perform loads into both Teradata systems. The way we will accomplish this is by associating different Contexts to each system, and executing the Load Package by associating it to a particular Context at runtime.

  • 7.1. Create a New Folder for the Load process.

    • 7.1.1. In the ODI Designer “Projects” pane, right-click on “TD_Dual_Load” and select “Insert

Folder”.

Using ODI for Teradata Dual-Loads The Primary.txt and Secondary.txt files that will be created by the
  • 7.1.2. Enter “3. Loadas the name and click OK.

Using ODI for Teradata Dual-Loads The Primary.txt and Secondary.txt files that will be created by the
  • 7.2. Create a Flat file to Teradata Interface.

    • 7.2.1. Expand the newly created 3. Loadfolder and right-click on “Interfaces” and select “Insert

Interface”. We will be creating the Interface to load a Flat file to Teradata.

Using ODI for Teradata Dual-Loads

Using ODI for Teradata Dual-Loads 7.2.2. Enter “ File to Teradata ” in the Name field
  • 7.2.2. Enter File to Teradatain the Name field and set the Optimization Context as Primary.

Check the “Staging Area Different From Target” and select TDfrom the drop down. Then Un-

check “Staging Area Different From Target”. Your “Interface: NEW” box should look like the one below.

Using ODI for Teradata Dual-Loads 7.2.2. Enter “ File to Teradata ” in the Name field

Click on the “Diagram” tab. And click “OK” if prompted for the below message.

Using ODI for Teradata Dual-Loads

Using ODI for Teradata Dual-Loads 7.2.3. Create the ELT map to load SRC_SALES_PERSON Flat file into
  • 7.2.3. Create the ELT map to load SRC_SALES_PERSON Flat file into TD_SALES_PERSON Table

in Teradata.

In the left pane of ODI Designer, click on “Models” tab. Expand the “Flat Files -> File_Demo_Src” model folder.

Using ODI for Teradata Dual-Loads 7.2.3. Create the ELT map to load SRC_SALES_PERSON Flat file into

Drag and drop the SRC_SALES_PERSON Flat file to the Sources section of the Diagram as shown below.

Using ODI for Teradata Dual-Loads 7.2.3. Create the ELT map to load SRC_SALES_PERSON Flat file into

Next, expand the model folder Teradata -> TD_FINANCIAL.

Using ODI for Teradata Dual-Loads 7.2.3. Create the ELT map to load SRC_SALES_PERSON Flat file into

Drag and drop the TD_SALES_PERSON file to the Target Database section of the diagram. Click “Yes” when prompted for Automatic Mapping as shown below.

Using ODI for Teradata Dual-Loads

Using ODI for Teradata Dual-Loads You should see Source and the Target columns mapped as shown

You should see Source and the Target columns mapped as shown in the below diagram. The columns that should not be mapped yet are the DATE_UPDATED and FULL_NAME. We will be creating the Transformations for those columns in the next steps.

Using ODI for Teradata Dual-Loads You should see Source and the Target columns mapped as shown
  • 7.2.4. Create the Transformations for FULL_NAME and DATE_UPDATED.

Click on FULL_NAME column in the Target Datastore pane. You should see a “Mapping:

FULL_NAME” sub-window towards the bottom.

Click on the Implementation tab and enter the below expression in the text box:

TRIM(SRC_SALES_PERSON.FIRST_NAME)||' '||TRIM(SRC_SALES_PERSON.LAST_NAME)

Check the “Active Mapping” box.

Select “Staging Area” for the “Execute On” option. This will ensure the expression will be evaluated at the staging location. You window should be similar to the one below.

Using ODI for Teradata Dual-Loads

Using ODI for Teradata Dual-Loads Click on “Errors” and make sure you get “No Errors detected

Click on “Errors” and make sure you get “No Errors detected in your diagram” message returned.

Using ODI for Teradata Dual-Loads Click on “Errors” and make sure you get “No Errors detected

Click “OK”.

  • 7.2.5. Click on DATE_UPDATED column in the Target Datastore pane. You should see a “Mapping:

DATE_UPDATEDsub-window towards the bottom.

Click on the Implementation tab and enter the expression CURRENT_TIMESTAMP in the text box and check the “Active Mapping” box.

Select “Staging Area” for the “Execute On” option. This will ensure the expression will be evaluated at the staging location. You window should be similar to the one below.

Using ODI for Teradata Dual-Loads

Using ODI for Teradata Dual-Loads Click “OK” . Then click on “Errors” to check for errors

Click “OK”. Then click on “Errors” to check for errors in the mapping and make sure there are none.

Using ODI for Teradata Dual-Loads Click “OK” . Then click on “Errors” to check for errors

Click “OK” to close the text box.

  • 7.2.6. Click on the “Flow” tab of the “Interface: NEW” window. This is where we will be assigning

the appropriate knowledge modules for the ELT.

Click on SS_0 (1 FILE_GENERIC)object on the left of the flow. In the properties section below, make sure the following values are set. Leave all other values as default. The TTU_PATH property of the KM needs to be set to blanks.

Note: Ensure you hit “Enter” after making changes to each field otherwise your changes will not persist.

LKM

LKM File to Teradata (TTU) - DL

DL_DIRECTORY

#TD_Dual_Load.FilePath//<%=odiRef.getSession("CONTEXT_NAME")%>_DL_Log

Using ODI for Teradata Dual-Loads

TTU_PATH
TTU_PATH
  • 7.2.7. Click on Target+Staging Area (TD1)object on the right of the flow. In the properties section

below, make sure the following values are set. Leave all other values as default.

IKM

IKM Teradata Control Append

FLOW_CONTROL

No

COLLECT_STATS

No

Using ODI for Teradata Dual-Loads

Using ODI for Teradata Dual-Loads 7.2.8. Click on the “Controls” tab and check that the Control
  • 7.2.8. Click on the “Controls” tab and check that the Control KM is <Undefined> for this load.

Using ODI for Teradata Dual-Loads 7.2.8. Click on the “Controls” tab and check that the Control

Click “OK” to save the File to Teradata Interface.

  • 7.3. Create a package that will call the File to Teradata” Interface.

Using ODI for Teradata Dual-Loads

  • 7.3.1. Create a new package. In the Projects pane of ODI Designer, expand “TD_Dual_Load -> 3.

Load. Right-click on “Packages” and select “Insert Package”.

Using ODI for Teradata Dual-Loads 7.3.1. Create a new package. In the Projects pane of ODI

Click on the “Definition” tab of the “Package: Newwindow and enter Loadas the name.

Using ODI for Teradata Dual-Loads 7.3.1. Create a new package. In the Projects pane of ODI
  • 7.3.2. Create the package flow. Click on the “Diagram” tab of the “Package: New” window. On the

Projects pane, expand “TD_Dual_Load -> Variables” folder. Drag and drop CurrentID” variable to

the Diagram pane as shown below.

Using ODI for Teradata Dual-Loads 7.3.1. Create a new package. In the Projects pane of ODI

Using ODI for Teradata Dual-Loads

  • 7.3.3. Similarly, drag and drop “MaxID”, “PathID”, “FilePath”, CurrentIDand “CurrentID

(again), from the Variables folder to the Diagram pane. Note that some of the variables are used

multiple times in the diagram, and that is not a typo.

Next, drag and drop “File to TeradataInterface from “TD_Dual_Load -> 3.Load -> Interfaces” folder.

Next, add “Files -> OdiFileWait, “File -> OdiFileDelete” from the Toolbox pane. The way to add items from the Toolbox is to first click on the tool you want to add, and then click again on the Diagram

pane to add the tool to the diagram. When you are done adding tools, then click on the adding.

Using ODI for Teradata Dual-Loads 7.3.3. Similarly, drag and drop “MaxID” , “PathID” , “FilePath” ,

icon to stop

Once you are done adding all the elements, arrange the icons in the diagram such that they look similar to the picture below.

Using ODI for Teradata Dual-Loads 7.3.3. Similarly, drag and drop “MaxID” , “PathID” , “FilePath” ,
  • 7.3.4. Add the “Next steps on Success” (

Using ODI for Teradata Dual-Loads 7.3.3. Similarly, drag and drop “MaxID” , “PathID” , “FilePath” ,

) arrows between all the steps. It is important to ensure

the flow is exactly as it looks in the below diagram.

Using ODI for Teradata Dual-Loads 7.3.3. Similarly, drag and drop “MaxID” , “PathID” , “FilePath” ,
  • 7.3.5. Set properties for each step. We will start with CurrentID (bottom left in the diagram) which is

the first step that will be executed and just go along with the next step in the sequence.

Using ODI for Teradata Dual-Loads

  • 7.3.6. Set properties for CurrentID. Highlight the CurrentID step in the diagram and click on the

“General” tab of the Properties section. Make sure you set the properties exactly as shown below.

Using ODI for Teradata Dual-Loads 7.3.6. Set properties for CurrentID . Highlight the CurrentID step in
  • 7.3.7. Set properties for MaxID. Highlight the MaxID step in the diagram and set the properties as

shown below.

Using ODI for Teradata Dual-Loads 7.3.6. Set properties for CurrentID . Highlight the CurrentID step in
  • 7.3.8. Set properties for PathID. Highlight the PathID step in the diagram and set the properties as

shown below.

Using ODI for Teradata Dual-Loads 7.3.6. Set properties for CurrentID . Highlight the CurrentID step in
  • 7.3.9. Set properties for FilePath. Highlight the FilePath step in the diagram and set the properties as

shown below.

Using ODI for Teradata Dual-Loads

Using ODI for Teradata Dual-Loads 7.3.10. Set properties for OdiFileWait 1. Highlight the OdiFileWait 1 step
7.3.10. Set properties for OdiFileWait 1. Highlight the OdiFileWait 1 step in the diagram and set
7.3.10.
Set properties for OdiFileWait 1. Highlight the OdiFileWait 1 step in the diagram and set the
properties as shown below.
Directory
\tmp\#TD_Dual_Load.FilePath
Filename Mask
<%=odiRef.getSession("CONTEXT_NAME")%>.txt
7.3.11.
Set properties for File to Teradata. Highlight the File to Teradata step in the diagram and set
the properties as shown below.
7.3.12.
Set properties for OdiFileDelete 2. Highlight the OdiFileDelete 2 step in the diagram and set

the properties as shown below.

Directory

\tmp\#TD_Dual_Load.FilePath

Filename

<%=odiRef.getSession("CONTEXT_NAME")%>.txt

Error if no file found

Yes

Using ODI for Teradata Dual-Loads

Using ODI for Teradata Dual-Loads 7.3.13. Set properties for CurrentID. Highlight the CurrentID step in the
7.3.13. Set properties for CurrentID. Highlight the CurrentID step in the diagram that follows OdiFileDelete 2
7.3.13.
Set properties for CurrentID. Highlight the CurrentID step in the diagram that follows
OdiFileDelete 2 and set the properties as shown below.
7.3.14.
Set properties for CurrentID. Highlight the CurrentID step in the diagram that follows the
previous step’s CurrentID and set the properties as shown below.
Type
Operator
Value
Evaluate Variable
<=
#TD_Dual_Load.MaxID
7.3.15.
Once done setting the properties of all the steps, click “OK” to create the Load package.

Using ODI for Teradata Dual-Loads

  • 8. Create E-LT Interfaces for Scenario 2

In this section, we will create ODI Interfaces and Packages that will replicate data from TD1 to TD2 Teradata systems using the approach outlined in scenario 2.

The Replicate process will pull out changed data from the TD_SALES_PERSON table from the TD1 Teradata system and push the transactions to the TD_REPL_SALES_PERSON table on TD2 Teradata system. For the change capture process, the E-LT interfaces will look at the DATE_UPDATED column of the TD_SALES_PERSON table and only pull rows that were modified or inserted since the last run of the Replicate process.

This process is completely asynchronous with the extracts and the loads processes, and can be run independently and at different frequencies. Also, if the Replicate process due to the unavailability of Teradata TD2 system, the process can be rerun and all the rows that were missed since the last successful run will be replicated in this run.

  • 8.1. Create a New Folder for the Replicate process.

    • 8.1.1. In the ODI Designer “Projects” pane, right-click on “TD_Dual_Load” and select “Insert

Folder”.

Enter 4. Replicateas the name and click OK.

Using ODI for Teradata Dual-Loads 8. Create E-LT Interfaces for Scenario 2 In this section, we
  • 8.2. Create an Interface to replicate from TD1 to TD2.

    • 8.2.1. Expand the newly created 4. Replicatefolder and right-click on “Interfaces” and select

“Insert Interface”. We will be creating the Interface to replicate the TD_SALES_PERSON on TD1

Teradata system to TD_REPL_SALES_PERSON on TD2 Teradata system.

Enter Replicate TD1 to TD2in the Name field and set the Optimization Context as Secondary. Check the “Staging Area Different From Target” and select “TD” from the drop down. Your “Interface: NEW” box should look like the one below.

Using ODI for Teradata Dual-Loads 8. Create E-LT Interfaces for Scenario 2 In this section, we

Using ODI for Teradata Dual-Loads

Click on the “Diagram” tab. And click “OK” if prompted for the below message.

Using ODI for Teradata Dual-Loads Click on the “Diagram” tab. And click “OK” if prompted for
  • 8.2.2. Create the E-LT map to replicate TD_SALES_PERSON from TD1 Teradata system to

TD_REPL_SALES_PERSON in TD2 Teradata system.

In the left pane of ODI Designer, click on “Models” tab. Expand the Teradata -> TD_SALES_PERSONmodel folder.

Using ODI for Teradata Dual-Loads Click on the “Diagram” tab. And click “OK” if prompted for

Drag and drop the TD_SALES_PERSON Table to the Sources section of the Diagram as shown below.

Using ODI for Teradata Dual-Loads Click on the “Diagram” tab. And click “OK” if prompted for

Next, drag and drop the TD_REPL_SALES_PERSON Table to the Target Datastore section of the Diagram as shown below.

Using ODI for Teradata Dual-Loads

Click “Yes” when prompted for Automatic Mapping. You should see Source and the Target columns mapped appropriately based on the column names.

Using ODI for Teradata Dual-Loads Click “Yes” when prompted for Automatic Mapping . You should see
  • 8.2.3. Click on each column on TD_REPL_SALES_PERSON and make sure all the columns are set

to execute on the Staging Area. You can start by TD_REPL_SALES_PERSON.SALES_PERSON_ID

as shown below.

Using ODI for Teradata Dual-Loads Click “Yes” when prompted for Automatic Mapping . You should see

Repeat this step for the remaining columns FIRST_NAME, LAST_NAME, FULL_NAME, DATE_HIRED and DATE_UPDATED.

  • 8.2.4. Set the appropriate Contexts for the TD_SALES_PERSON source,

TD_REPL_SALES_PERSON destination.

Click on TD_SALES_PERSON in the Source section. In the properties box, set the Context as Primary. This is needed to force the source to go against the Teradata source that is setup against the Primary context, which in this case is the TD1 Teradata system.

Using ODI for Teradata Dual-Loads

Using ODI for Teradata Dual-Loads Click on TD_REPL_SALES_PERSON in the Target Datastore section. In the properties

Click on TD_REPL_SALES_PERSON in the Target Datastore section. In the properties box, set the Context as Execution Context. When we execute this interface, the Secondary context will be used and the Execution Context will be associated with the TD2 Teradata at execution time.

Also, make sure that the Update Key is set to PK_TD_REPL_SALES_PERSON. This is needed for the Teradata loading KMs when performing updates.

Using ODI for Teradata Dual-Loads Click on TD_REPL_SALES_PERSON in the Target Datastore section. In the properties
  • 8.2.5. Create a filter on DATE_UPDATED column on TD_SALES_PERSON. This is needed to

implement the change data capture logic for extracting only the newly inserted rows or modified rows from TD_SALES_PERSON table from TD1 source system.

In order to create a filter, drag and drop the DATE_UPDATED column out of the SRC_SALES_PERSON table in the Sources section and into the blank area on the Sources pane.

Click on the

Click on the filer icon. Make sure the filter is selected to be active by checking

filer icon. Make sure the filter is selected to be active by checking the “Active Filter”

box. Set the Execute On to Source. This means that the filter will be applied on the source Teradata

system (TD1).

 

Click on the Implementation tab and Enter:

TD_SALES_PERSON.DATE_UPDATED between cast('#LastREPLRefresh' as timestamp(2)) AND cast('#CurrentREPLTimestamp' as timestamp(2))

Using ODI for Teradata Dual-Loads

Using ODI for Teradata Dual-Loads Click on the “Database” icon to check for syntactical errors in

Click on the “Database”

Using ODI for Teradata Dual-Loads Click on the “Database” icon to check for syntactical errors in

icon to check for syntactical errors in the filter expression. You should see

the below message that says that the SQL is valid for the DBMS.

Using ODI for Teradata Dual-Loads Click on the “Database” icon to check for syntactical errors in

Click “OK”. Then click on “Errors” to check for errors in the mapping. You should see the below text box stating that there were no errors detected in the diagram.

Using ODI for Teradata Dual-Loads Click on the “Database” icon to check for syntactical errors in

Click “OK” to close the text box.

  • 8.2.6. Assign the appropriate Knowledge Modules to the Flow. Click on the “Flow” tab. This is where

we will be assigning the appropriate knowledge modules for the E-LT.

Click on TD_SALES_PERSON table. In the Properties section below, make sure the following values are set:

Using ODI for Teradata Dual-Loads

LKM LKM SQL to SQL DELETE_TEMPORARY_OBJECTS Yes
LKM
LKM SQL to SQL
DELETE_TEMPORARY_OBJECTS Yes

Click on TD_REPL_SALES_PERSON table. In the Properties section below, modify the below fields:

IKM

IKM Teradata Incremental Update

SYNC_JRN_DELETE

No

FLOW_CONTROL

No

COLLECT_STATS

No

Using ODI for Teradata Dual-Loads

Using ODI for Teradata Dual-Loads Click “OK” to save the Replicate TD1 to TD2 Interface. 8.3.

Click “OK” to save the Replicate TD1 to TD2 Interface.

  • 8.3. Create a package that will call the Replicate TD1 to TD2” Interface.

    • 8.3.1. Create a new package. In the Projects pane of ODI Designer, expand “TD_Dual_Load -> 4.

Replicate”. Right-click on “Packages” and select “Insert Package”.

Click on the “Definition” tab of the “Package: Newwindow and enter Replicateas the name.

Using ODI for Teradata Dual-Loads Click “OK” to save the Replicate TD1 to TD2 Interface. 8.3.

Using ODI for Teradata Dual-Loads

  • 8.3.2. Create the package flow. Click on the “Diagram” tab of the “Package: New” window. On the

Projects pane, expand “TD_Dual_Load -> Variables” folder. Drag and drop LastREPLRefresh

variable to the Diagram pane as shown below.

Using ODI for Teradata Dual-Loads 8.3.2. Create the package flow. Click on the “Diagram” tab of
  • 8.3.3. Similarly, drag and drop CurrentREPLTimestampand LastREPLRefresh(again) from

the Variables folder to the Diagram pane.

Next, drag and drop Replicate TD1 to TD2Interface from “TD_Dual_Load -> 4. Replicate -> Interfaces” folder.

Once you are done adding all the elements, arrange the icons in the diagram such that they look similar to the picture below.

Using ODI for Teradata Dual-Loads

Using ODI for Teradata Dual-Loads 8.3.4. Add the “Next steps on Success” ( ) arrows between
  • 8.3.4. Add the “Next steps on Success” (

Using ODI for Teradata Dual-Loads 8.3.4. Add the “Next steps on Success” ( ) arrows between

) arrows between all the steps. It is important to ensure

the flow is exactly as it looks in the below diagram.

Using ODI for Teradata Dual-Loads

Using ODI for Teradata Dual-Loads 8.3.5. Set properties for each step. We will start with LastREPLRefresh
  • 8.3.5. Set properties for each step. We will start with LastREPLRefresh (top left in the diagram)

which is the first step that will be executed and just go along with the next step in the sequence.

  • 8.3.6. Highlight the LastREPLRefresh (top left in the diagram) step in the diagram and click on the

“General” tab of the Properties section. Make sure you set the properties exactly as shown below.

Using ODI for Teradata Dual-Loads 8.3.5. Set properties for each step. We will start with LastREPLRefresh
  • 8.3.7. Highlight the CurrentREPLTimestamp step in the diagram and set the properties as shown

below.

Using ODI for Teradata Dual-Loads

Using ODI for Teradata Dual-Loads 8.3.8. Highlight the Replicate TD1 to TD2 step in the diagram
  • 8.3.8. Highlight the Replicate TD1 to TD2 step in the diagram and set the properties as shown below.

Using ODI for Teradata Dual-Loads 8.3.8. Highlight the Replicate TD1 to TD2 step in the diagram
  • 8.3.9. Highlight the LastREPLRefresh (bottom left in the diagram) step in the diagram and set the

properties as shown below.

Using ODI for Teradata Dual-Loads 8.3.8. Highlight the Replicate TD1 to TD2 step in the diagram

8.3.10. Once done setting the properties of all the steps, click “OK” to create the Replicate package.

  • 9. Execute Tests for Scenario 1

In this step, you will execute the packages and interfaces created in the previous steps and demonstrate the Teradata dual-load scenario as depicted in scenario 1.

The test cases will involve executing the Extract process multiple times. In between the extracts, you will perform an incremental load on the Oracle source. The idea here is to demonstrate that the updates or inserts in the source system will get propagated to the destination using the change data capture mechanisms built into the E- LT maps.

The Load package will be executed after the extracts are completed. You will also be performing sanity checks to make sure the test is successful.

In a production implementation, the execution of these packages would be automated through the use of ODI Scenarios and a scheduler.

Using ODI for Teradata Dual-Loads

  • 9.1. Execute the “Extract” Package.

    • 9.1.1. This step is going to run the Extract package for the first time in this tutorial, so this run will

extract all the rows from the Oracle database.

In the Projects pane of ODI Designer, expand “TD_Dual_Load -> 2. Extract -> Packages”.

Using ODI for Teradata Dual-Loads 9.1. Execute the “Extract” Package . 9.1.1. This step is going

Right-click on the “Extract” Package and select “Execute”.

This Package will run against the SH schema in Oracle, select the “Global” context, and “Local (No Agent)” agent.

Using ODI for Teradata Dual-Loads 9.1. Execute the “Extract” Package . 9.1.1. This step is going

Click “OK”. You should see a Session started message.

Using ODI for Teradata Dual-Loads 9.1. Execute the “Extract” Package . 9.1.1. This step is going

Click “OK”.

  • 9.1.2. Repeat the above step and execute Extract again. This time the Extract process should not

extract any rows since the source data was not modified. We will verify this in the later steps.

Using ODI for Teradata Dual-Loads

  • 9.1.3. Login to the Repository using Operator. Click on “Start -> All Programs -> Oracle -> Oracle

Data Integrator -> Operatorfrom the Start Menu. Or, if you are already logged into Designer or

Topology Manager, then click on the Operator icon

.
.

In the Session List pane, expand “Date -> Today” and you should see two successful executions of the ExtractPackage.

Using ODI for Teradata Dual-Loads 9.1.3. Login to the Repository using Operator . Click on “Start
  • 9.1.4. Using the operating system File Explorer tool, browse to the \tmp directory on the Teradata

Primary machine. The \tmp directory should have two sub-directories created underneath as shown

below. Each sub-directory represents a successful execution of the extract process.

Using ODI for Teradata Dual-Loads

Using ODI for Teradata Dual-Loads 9.1.5. Browse to the \tmp\FileSet.000000001 directory and you should see three
  • 9.1.5. Browse to the \tmp\FileSet.000000001 directory and you should see three files created with

the names Primary.txt, Secondary.txt and SRC_SALES_PERSON.txt. The SRC_SALES_PERSON.txt contains the flat file extract of the SRC_SALES_PERSON table from the Oracle SH schema. The presence of Primary.txt and Secondary.txt files indicate that the extract was not yet loaded into any of the Teradata systems. When the extract gets successfully loaded by the Load package to the Teradata systems, these files will get deleted.

Using ODI for Teradata Dual-Loads 9.1.5. Browse to the \tmp\FileSet.000000001 directory and you should see three
  • 9.1.6. Similar to the above step, browse to the \tmp\FileSet.000000002 directory and you should see

the same three files as in the previous step. Notice that the SRC_SALES_PERSON.txt file has zero bytes for this run. This is because when the extract was run the second time, it did not pull any changes from the source table because the source table was not loaded or updated between two consecutive extracts.

Using ODI for Teradata Dual-Loads 9.1.5. Browse to the \tmp\FileSet.000000001 directory and you should see three

Using ODI for Teradata Dual-Loads

Note: Look at the content of the files. The SRC_SALES_PERSON.txt for the first extract should have a total of 5 rows, since the source table on the Oracle database had 5 rows.

  • 9.2. Execute the “Load” Package on the Primary and Secondary Teradata systems.

    • 9.2.1. This step will run the Load package. The Load package will load these files to the Teradata

systems, and also perform simple transformations.

In the Projects pane of ODI Designer, expand “TD_Dual_Load -> 3. Load -> Packages”.

Using ODI for Teradata Dual-Loads Note: Look at the content of the files. The SRC_SALES_PERSON.txt for

Right-click on the LoadPackage and select “Execute”.

To run the load into the Teradata TD1 Primary system, select the Primarycontext, and TD_Agentagent in the Execution dialog box.

Using ODI for Teradata Dual-Loads Note: Look at the content of the files. The SRC_SALES_PERSON.txt for

Click “OK”. Click “OK” again when you get a confirmation that the session was started.

  • 9.2.2. Repeat the above step again but execute against the Teradata TD2 Secondary system. In the

Execution dialog box, select the Secondarycontext, and TD_Agentagent as shown below.

Using ODI for Teradata Dual-Loads Note: Look at the content of the files. The SRC_SALES_PERSON.txt for

Using ODI for Teradata Dual-Loads

  • 9.2.3. Verify that the two Loads were successfully completed using ODI Operator.

Using ODI for Teradata Dual-Loads 9.2.3. Verify that the two Loads were successfully completed using ODI
  • 9.2.4. Browse to the \tmp\FileSet.000000001 directory on the Teradata Primary system. The

directory should have two sub-directories (Primary_DL_Log and Secondary_DL_log) created as shown below. Each sub-directory stores the log files and scripts of the load process for each Teradata system. Also, note that the “Primary.txt” and “Secondary.txt” files are non-existent since the SRC_SALES_PERSON.txt dump was loaded into both Teradata systems.

Using ODI for Teradata Dual-Loads

Using ODI for Teradata Dual-Loads 9.2.5. Repeat the above step and similarly verify the sub-directories and
  • 9.2.5. Repeat the above step and similarly verify the sub-directories and files present in the

\tmp\FileSet.000000002 directory.

Note: Browse the Primary_DL_Log and Secondary_DL_log sub-directories and look at the log files generated by the load process.

  • 9.3. Perform an incremental load on the Oracle source and run an Incremental Extract.

    • 9.3.1. Run the supplied incremental load into the SRC_SALES_PERSON table in Oracle SH schema.

The load will insert 5 additional rows into the table.

In the Projects pane of ODI Designer, expand “TD_Dual_Load -> 1. Setup -> Procedures.

Using ODI for Teradata Dual-Loads 9.2.5. Repeat the above step and similarly verify the sub-directories and

Right-click on the “Incremental Load Oracle 1Procedure and select “Execute”. Select the “Global” context and “Local (No Agent)” agent. Click “OK”. This will execute the package against the Oracle database SH Schema. Click “OK” to acknowledge the Session Started message.

  • 9.3.2. Verify that the Incremental load was successfully completed through ODI Operator.

Using ODI for Teradata Dual-Loads 9.2.5. Repeat the above step and similarly verify the sub-directories and
  • 9.3.3. Get the row count of SH.SRC_SALES_PERSON table from Oracle. You can use SQL*Plus or

any SQL Query tool like SQL Developer to login to Oracle. Run the below statement to verify the row counts. The table should have a total of 10 rows.

SQL> select count(*) from SH.SRC_SALES_PERSON;

Using ODI for Teradata Dual-Loads

Using ODI for Teradata Dual-Loads 9.3.4. Execute the Extract Package to get the incremental changes from
  • 9.3.4. Execute the Extract Package to get the incremental changes from SRC_SALES_PERSON

table in the SH schema into the intermediary flat file that will be loaded into the Teradata systems in the

next steps.

In the Projects pane of ODI Designer, expand “TD_Dual_Load -> 2. Extract -> Packages”. Right- click on the “Extract” Package and select “Execute”.

This Package will run against the SH schema in Oracle, select the “Global” context, and “Local (No Agent)” agent. Click “OK”. Click “OK” to acknowledge the Session Started message.

  • 9.3.5. Verify that the Extract Package was successfully executed by using ODI Operator.

Using ODI for Teradata Dual-Loads 9.3.4. Execute the Extract Package to get the incremental changes from
  • 9.3.6. Verify the rows extracted in the flat file. Using a File Explorer tool, browse to the

\tmp\Fileset.000000003 directory on the Teradata Primary machine.

Using ODI for Teradata Dual-Loads 9.3.4. Execute the Extract Package to get the incremental changes from

Double-click the SRC_SALES_PERSON.txt file. You should see 5 rows that were loaded into the source Oracle table in the previous step.

Using ODI for Teradata Dual-Loads 9.3.4. Execute the Extract Package to get the incremental changes from
  • 9.3.7. Run the Load package to load the extracted file to the Primary and Secondary Teradata

systems.

Using ODI for Teradata Dual-Loads

In the Projects pane of ODI Designer, expand “TD_Dual_Load -> 3. Load -> Packages”. Right-click on the “Load” Package and select “Execute”.

To run the load into the Teradata TD1 Primary system, select the Primarycontext, and TD_Agentagent in the Execution dialog box.

Using ODI for Teradata Dual-Loads In the Projects pane of ODI Designer , expand “TD_Dual_Load ->

Click “OK”. You should get a confirmation that the session was started. Click “OK” to confirm.

  • 9.3.8. Repeat the above step again but execute against the Teradata TD2 Secondary system. So, in

the Execution dialog box, select the Secondarycontext, and TD_Agentagent.

Using ODI for Teradata Dual-Loads In the Projects pane of ODI Designer , expand “TD_Dual_Load ->
  • 9.3.9. Verify that the Load Package was successfully executed by using ODI Operator.

Using ODI for Teradata Dual-Loads In the Projects pane of ODI Designer , expand “TD_Dual_Load ->
  • 9.3.10. Verify the incremental loads in Teradata systems by SQL. The TD_SALES_PERSON table in

both Teradata systems should have a total of 10 rows as shown below, 5 of which were loaded by the last execution of the Load process.

Using ODI for Teradata Dual-Loads

Using ODI for Teradata Dual-Loads 10. Execute Tests for Scenario 2 It is important that you

10. Execute Tests for Scenario 2

It is important that you execute the tests for scenario 1 prior to running the scenario 2 tests. This scenario builds on top of the tests for the previous scenario. The scenario 1 should have by now, loaded 10 rows altogether on the Oracle database, and also loaded these 10 rows along with transformations, to the two Teradata systems Primary and Secondary.

The test scenarios in scenario 2 will involve executing the Replicate process multiple times to replicate data from the TD_SALES_PERSON table in TD1 to TD_REPL_SALES_PERSON table in TD2. In between running two Replicate packages, we will perform an incremental load on the Oracle source that will load 5 additional rows in the SRC_SALES_PERSON table. The idea here is to demonstrate that the updates or inserts in the source system will get propagated to the destination using the change data capture mechanisms.

In a production implementation, the execution of these packages would be automated through ODI Scenarios and scheduler.

10.1. Execute the Replicate” Package.

  • 10.1.1. Verify the number of rows in the TD_REPL_SALES_PERSON table on the Teradata TD2

system using SQL. There should be a total of 0 rows in that table since we have not replicated any data yet.

Using ODI for Teradata Dual-Loads

Using ODI for Teradata Dual-Loads 10.1.2. This step is going to run the Replicate Package for
  • 10.1.2. This step is going to run the Replicate Package for the first time in this tutorial, so this run will

perform a full load of the TD_REPL_SALES_PERSON table on TD2 from TD_SALES_PERSON on

TD1.

In the Projects pane of ODI Designer, expand “TD_Dual_Load -> 4. Replicate -> Packages”.

Using ODI for Teradata Dual-Loads 10.1.2. This step is going to run the Replicate Package for

Right-click on the ReplicatePackage and select “Execute”.

This Package will run against the TD2 Teradata system as we had discussed when creating the Replicate package, select the Secondarycontext, and TD_Agentagent.

Using ODI for Teradata Dual-Loads 10.1.2. This step is going to run the Replicate Package for

Click “OK”. You should get a confirmation that the session was started. Click “OK” to confirm.

  • 10.1.3. Verify that the Replicate Package was successfully executed by using ODI Operator.

Using ODI for Teradata Dual-Loads

Using ODI for Teradata Dual-Loads 10.1.4. Verify the number of rows loaded in the TD_REPL_SALES_PERSON table
  • 10.1.4. Verify the number of rows loaded in the TD_REPL_SALES_PERSON table on the Teradata

TD2 system using the below SQL. There should be a total of 10 rows in that table, the 10 being loaded by the Replicate package that ran in the previous step.

Using ODI for Teradata Dual-Loads 10.1.4. Verify the number of rows loaded in the TD_REPL_SALES_PERSON table

10.2. Perform an incremental load on the Oracle Source and run the incremental load process on TD1 using scenario 1 and on TD2 using scenario 2.

  • 10.2.1. In the Projects pane of ODI Designer, expand “TD_Dual_Load -> 1. Setup -> Procedures.

Using ODI for Teradata Dual-Loads 10.1.4. Verify the number of rows loaded in the TD_REPL_SALES_PERSON table

Right-click on the “Incremental Load Oracle 2procedure and select “Execute”. Select the “Global” context and “Local (No Agent)” agent. Click “OK”.

Using ODI for Teradata Dual-Loads

This step will execute the package against the Oracle database SH Schema and insert 5 additional rows into the SRC_SALES_PERSON table. Click “OK” to acknowledge the Session Started message.

  • 10.2.2. Verify that the Incremental load was successfully completed through ODI Operator.

Using ODI for Teradata Dual-Loads This step will execute the package against the Oracle database SH
  • 10.2.3. Get the row count of SH.SRC_SALES_PERSON table from Oracle. The table should now have

a total of 15 rows.

Using ODI for Teradata Dual-Loads This step will execute the package against the Oracle database SH
  • 10.2.4. Execute the Extract Package to get the incremental changes from SRC_SALES_PERSON

table in the SH schema into the intermediary flat file that will be loaded into the Teradata systems.

In the Projects pane of ODI Designer, expand “TD_Dual_Load -> 2. Extract -> Packages”. Right- click on the “Extract” Package and select “Execute”.

This Package will run against the SH schema in Oracle, select the “Global” context, and “Local (No Agent)” agent. Click “OK”. Click “OK” to acknowledge the Session Started message.

  • 10.2.5. Verify that the Extract Package was successfully executed by using ODI Operator.

Using ODI for Teradata Dual-Loads This step will execute the package against the Oracle database SH
  • 10.2.6. Run the Load package to capture incremental changes on the source into the Primary

Teradata system.

In the Projects pane of ODI Designer, expand “TD_Dual_Load -> 3. Load -> Packages”. Right-click on the “Load” Package and select “Execute”.

To run the load into the Teradata TD1 Primary system, select the Primarycontext, and TD_Agentagent in the Execution dialog box.

Using ODI for Teradata Dual-Loads This step will execute the package against the Oracle database SH

Using ODI for Teradata Dual-Loads

Click “OK”. You should get a confirmation that the session was started. Click “OK” to confirm.

  • 10.2.7. Verify that the Load Package was successfully executed by using ODI Operator.

  • 10.2.8. Run the Replicate Package to push changes from TD_SALES_PERSON on TD1, to

TD_REPL_SALES_PERSON table on TD2.

In the Projects pane of ODI Designer, expand “TD_Dual_Load -> 4. Replicate -> Packages”. Right- click on the ReplicatePackage and select “Execute”.

Select the Secondarycontext, and TD_Agentagent.

Using ODI for Teradata Dual-Loads Click “OK” . You should get a confirmation that the session

Click “OK”. Click “OK” again when you see a Session started message.

  • 10.2.9. Verify that the Replicate Package was successfully executed by using ODI Operator.

    • 10.2.10. Verify the number of rows loaded to the TD_SALES_PERSON on TD1. There should be a

total of 15 rows, the last 5 being loaded by the Replicate package from the previous step.

Using ODI for Teradata Dual-Loads Click “OK” . You should get a confirmation that the session

Using ODI for Teradata Dual-Loads

  • 10.2.11. Verify the number of rows loaded to the TD_REPL_SALES_PERSON. There should be a

total of 15 rows as well.

Using ODI for Teradata Dual-Loads 10.2.11. Verify the number of rows loaded to the TD_REPL_SALES_PERSON .

11. Summary

In this tutorial, we have successfully demonstrated ODI’s dual-load capability into Teradata, using the two scenarios that the customers most commonly use for Teradata dual-active configurations.

Using ODI for Teradata Dual-Loads

Using ODI for Teradata Dual-Loads Using ODI for Teradata Dual-Loads April 2010 Author: Maqsood Alam Oracle

Using ODI for Teradata Dual-Loads April 2010

Author: Maqsood Alam

Oracle Corporation World Headquarters 500 Oracle Parkway Redwood Shores, CA 94065 U.S.A.

Worldwide Inquiries:

Phone: +1.650.506.7000 Fax: +1.650.506.7200 www.oracle.com

Copyright © 2010, Oracle. All rights reserved.

This document is provided for information purposes only and the contents hereof are subject to change without notice. This document is not warranted to be error-free, nor subject to any other warranties or conditions, whether expressed orally or implied in law, including implied warranties and conditions of merchantability or fitness for a particular purpose. We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document. This document may not be reproduced or transmitted in any form or by any means, electronic or mechanical, for any purpose, without our prior written permission.

Oracle, JD Edwards, PeopleSoft, and Siebel are registered trademarks of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners.

Using ODI for Teradata Dual-Loads Using ODI for Teradata Dual-Loads April 2010 Author: Maqsood Alam Oracle