Professional Documents
Culture Documents
Rapport PDF
Rapport PDF
com/
This is one of our documents from ODI learning series. This document
is created for new bees in ODI. The object of the document is to assist
new bees with step by step approach to perform various tasks.
History:
Version Description Change Author Publish Date
0.1 Initial Draft Gourav Atalkar 30-June-2011
Table of Contents
1) Introduction 3
a. Objective
b. Business-Rules Driven Approach
c. Traditional ETL versus E-LT Approach
d. Understanding Oracle Data Integrator Interfaces
2) Using ODI in your Data Warehouse Project 9
a. Introduction
b. Organizing the Teams
3) Creating an ODI Project and Interface: Exporting a Flat File to a Flat File 11
a. Purpose
b. Overview
c. Creating a New Physical Schema for the Flat File Model
d. Creating a New ODI Model for the Flat File Source
e. Creating a New ODI Source Datastore for use with ODI Interface
f. Creating a New ODI Target Datastore for use with ODI Interface
g. Creating a New ODI Interface for Flat File to Flat File Transformations
4) Creating an ODI Project and Interface: Exporting a Relational Table to a Flat File 21
a. Purpose
b. Creating Interface
5) Creating ODI Interface: Exporting a Flat File to a Relational Table 25
1) Introduction
Objective
• Through a business problem case study, understand and evaluate some different
development approaches including:
• Manual coding
• Traditional ETL
Introduction to Business rules: - Business rules specify mappings, filters, joins and
constraints. They often apply to metadata to transform data and are usually described in
natural language by business users. In a typical data integration project (such as a Data
Warehouse project), these rules are defined during the specification phase in documents
written by business analysts in conjunction with project managers.
Business Rules usually define “What” to do rather than “How” to do it. They can very often be
implemented using SQL expressions, provided that the metadata they refer to is known and
qualified in a metadata repository.
Joins: A join operation links records in several data sets, such as tables or files. Joins are used
to link multiple sources. A join is implemented as a SQL expression linking the columns
(fields) of two or more data sets.
Joins can be defined regardless of the physical location of the source data sets involved. For
example, a JMS queue can be joined to a relational table.
Depending on the technology performing the join, it can be expressed as an inner join, right
outer join, left outer join and full outer join.
Filters: A filter is an expression applied to source data sets columns. Only the records
matching this filter are processed by the data flow.
Constraints: A constraint is an object that defines the rules enforced on data sets’ data. A
constraint ensures the validity of the data in a given data set and the integrity of the data of a
model. Constraints on the target are used to check the validity of the data before integration
in the target.
Traditional ETL versus E-LT Approach: Traditional ETL tools operate by first Extracting
the data from various sources, Transforming the data on a proprietary, middle-tier ETL
engine, and then Loading the transformed data onto the target data warehouse or integration
server. Hence the term “ETL” represents both the names and the order of the operations
performed, as shown in Figure below.
Understanding Oracle Data Integrator Interfaces : An interface is an ODI object stored in the
ODI Repository that enables the loading of one target datastore with data transformed from
Figure shows a screenshot of an ODI interface that loads data into the FACT_SALES target
table. Source Data is defined as a heterogeneous query on the CORRECTIONS file, the Store
and Region tables.
A Business Problem case study: Figure describes an example of a business problem to extract,
transform and load data from a Microsoft SQL Server database and a file into a target Oracle
table.
Data is coming from two Microsoft SQL Server tables (Store joined to Region) and. The target
SALES Oracle table must match some constraints such as the uniqueness of the ID column and
valid reference to the SALES_REP table.
Data must be transformed and aggregated according to some mappings as shown in Figure
Translating these business rules from natural language to SQL expressions is usually
straightforward. In our example, the rules that appear in the figure could be translated as
follows:
Constraint ID must not be null ID is set to “not null” in the data model
We will be looking at an organization that wishes to track sales from various heterogeneous data
sources issued from the production systems. The following data sources will be used:
The Repository: The Repository contains all of the metadata required for the training examples. It
is hosted, for these evaluation purposes, in a supplied database.
Orders Application: An application for tracking customer orders, hosted in a supplied database
(the "src_sales" sample database).
Sales Administration: The administration or tracking of sales, hosted in another supplied database
(the "trg_sales" sample database). We will populate this data warehouse with our transformations
The schema diagrams below define the supplied data models that we will be using throughout this
guide:
Orders Application
1) Store_table
Region_table
2) Dim Reason
Product_Table
Period_Table
Fact_table
• Identifying source data that concerns key indicators; specifying business rules to
transform source information into key indicators
• Modeling the data structure of the target warehouse to store the key indicators
• Measuring the overall accuracy of the data by setting up data quality rules
• Making key indicators and metadata available to business users through ad-hoc
query tools or predefined reports
They know the source applications and specify business rules to transform source data into
meaningful target indicators. They are in charge of maintaining translation data from operational
semantics to the unified data warehouse semantic.
ODI MODULES USED: Limited access to Models Full access to Projects Operator Navigator ODI
Console
ODI MODULES USED: Topology Navigator (limited access), Designer Navigator: Full access to
Models Restore access to Projects, ODI Console
ODI MODULES USED: Topology Navigator (full access), Designer Navigator (full access), Operator
Navigator (full access), ODI Console
ODI MODULES USED: Agents, Topology Navigator (limited access), ODI Console
ODI MODULES USED: Security Navigator (full access), Designer Navigator (read access), Topology
Navigator (read access), ODI Console
8) PROFILE: Operator
DESCRIPTION: Operators are in charge of importing released and tested scenarios into the
production environment. They schedule the execution of these scenarios. They monitor execution
logs and restart failed sessions when needed.
ODI MODULES USED: Operator Navigator ODI Console Oracle Enterprise Manager Plug-in For ODI
The ODI Master Repository contains built-in default profiles that can be assigned to users. The
following table suggests how to use these built-in profiles:
This tutorial walks you through the steps that are needed to create a project and an interface
in Oracle Data Integrator (ODI) to export a flat file to another flat file.
OVERVIEW
A common task that is performed using ODI is to export data from a flat file and load this data
into another flat file. This tutorial walks you through the steps that are needed to create a
project and an interface that will import a file to a staging area, perform some minor
transformations, and then write the data to a file. You also execute the interface and verify the
execution using ODI Operator.
Step: 1) Click Topology tab. In Topology navigator, click the Physical Architecture tab, select
Technologies > File. Right-click FILE_GENERIC, and then select New Physical Schema.
Step: 3) Open Logical Architecture, navigate to Technologies > File, right-click File and select New Logical
Schema. Enter the Logical Schema name: SRC_Text_file, select the recently created physical schema in
Global context as shown below. Click Save and close the editing windows.
Step: 1) Click the Models tab in the left panel of ODI Designer. Click the Insert Model Folder
icon, and then select New Model Folder. On the screen that appears, set the Name to Files.
Click Save icon
Step: 2) Save.
Right-click the Text model folder and select New Model. On the screen that appears, enter the
values provided in the following image. Select Reverse Engineer tab.
Step: 3) On the screen that appears, verify settings and click Save icon to save your model.
CREATING A NEW ODI SOURCE DATASTORE FOR USE WITH ODI INTERFACE
Step:1) Expand the Flat Files folder, right-click the newly created file data model, and then
select New Datastore. On the screen that appears, set Name to SRC_STORE. Click the button
next to the Resource Name field. Select the STORES.txt file and click Open.
Step: 2) Click the Files tab. Set the File Format to Fixed, and then click the Columns tab.
Step: 3) On the Columns tab, click the Reverse Engineer button. The following screen
appears. Click Yes.
Step:2) In ODI Designer, right-click the newly created source datastore STORES and select Duplicate.
On the screen that follows, click Yes. The new Copy of STORES.txt appears in the tree view.
Step:3) Right-click the newly created duplicate Copy of STORES.txt and select Edit. On the
screen that appears, set the Name to TRG_STORES. Click the button next to the Resource
Name field.
Step: 4) Select the TRG_STORES.txt file and click Open. Click Save icon. The data stores for
the source and target files have now been successfully created in the same model and can be
used in an ODI interface. Close editing tabs.
CREATING A NEW ODI INTERFACE FOR FLAT FILE TO FLAT FILE TRANSFORMATIONS
Step: 1) In ODI Designer, click the Projects tab. Expand your project, and then expand First
Folder. Right-click Interfaces and select Insert Interface.
Step: 2) On the screen that follows, enter the interface name as INT_text_file to text_file.
Select In-Memory Engine: SUNOPSIS_MEMORY_ENGINE from drop-down list. Click the
Mapping tab.
Step: 3) Click the Models tab to drag the source and target to Designer. Expand Text File
model > Flat File Model and drag the Stores datastore from the Text File model folder into
the Sources container. Drag the TRG_Stores.txt datastore from Text File model into the
Target Datastore container. When Designer asks "Do you want to perform an Automatic
Mapping?" click Yes.
Step: 4) Go to the Flow Tab Click the Srs_Set0(FILE_GENERIC) source datastore. This makes the
properties for the source appear below. In the LKM Selector section, select LKM File to SQL from the
LKM drop-down list if not already selected.
Step: 5) Click Target (FILE_GENERIC). This makes the properties for the target appear
below. For IKM, select IKM SQL to File Append from the IKM drop-down list if not already
selected. Set the IKM options TRUNCATE to True and GENERATE_HEADER to False as
shown below. Click Save icon to save your interface.
Step: 6) To test your interface, click Execute icon. The following screen appears. Retain the
defaults and click OK. On the screen that follows, click OK.
Step: 7) To verify that your interface was executed successfully; you need to open ODI
Operator. Click the ODI Operator tab. In ODI Operator, click Hierarchical Sessions. Expand All
Execution > INT_text_file to text_file > Steps > INT_text_file to text_file, and view the
execution results for each step.
Step: 2) In the Designer Navigator to the left, click the Models tab. In the tree view, expand
the File Model. Drag the TRG_Region datastore from the tree view to the Target Datastore
zone .The datastore appears in this zone.
Step: 3) Expand the Oracle Application Model Go outdoors sales and drag the SRC_REGION
datastore from the model tree to the Sources zone of your diagram (the panel to the left of the
target panel). An Automap dialog box appears. Click Yes. Now the system automatically maps
fields in the source and target datastores.
Step: 4) Click the Flow tab. Select the Distinct Rows check box.
Step: 5) Click the Save button to save your interface. If the Locking Object window appears,
select “Don’t show this window next time,” and click Yes.
Step: 6) Run this interface, and check the content of the TRG_Region table.
Step: 7) Click OK in the Execution window, and then click OK when the Session Started
message appears.
Step: 9) Go to the model tab select file model expand it and select TRG_Region data store
Right click select View Data.
Step: 2) Click Topology navigator. In Topology navigator, click the Physical Architecture tab,
select Technologies > File. Right-click and then select New Data Server.
In Definition tab specify the Name for the Data Server and go to JDBC tab. specify the JDBC
Driver and JDBC Url by selecting browse button.
Step: 4) Now in Topology navigator, click the Physical Architecture tab, select Technologies >
File. Right-click, Demonstration and then select New Physical Schema.
Step: 5) For the Directory (Schema) and Directory (Work Schema) fields, enter the path to
the directory where your input flat files are located
Step: 6) Verify that your new Physical schema was added under the Demonstration data
server in the Technologies tree view.
Step: 7) Open Logical Architecture, navigate to Technologies > File, right-click File, and
select New Logical Schema.
Step: 8) Enter the Logical Schema name: Demonstration Integration, and select the Physical
schema Demonstration in all three contexts as shown here. Click Save and close the editing
Window.
Step: 9) Verify that your new Logical schema was added under the File node in the
Technologies tree view.
Step: 11) On the screen that appears, set the Name of the project to Flat file to Relation
Table in the Name field. The Code field is filled in automatically. Click Save, and then close the
tab. The newly created Flat file to Relation Table project now appears in the Projects pane.
Create a new model folder for the flat-file source data store.
Step: 13) In ODI Designer, click the Models tab, and then click the New Model. Then select the
New Model option to insert a new model.
Step: 14) Create a new ODI source data store. On the screen that appears, set the Name to
Sales Region. Click the button next to the Resource Name field. Navigate to the
C:\oracle\Middleware\Oracle_ODI1\oracledi\Demo\Demonstration folder, and then select
the SRC Sales Region.txt file. Click Open.
Step: 15) Click the Files tab. Set the File Format to Fixed, and then click the Columns tab.
Step: 16) On the Columns tab, click the Reverse Engineer button. The following Confirmation
screen appears. Click Yes. The Column Setup Wizard appears.
Step: 17) Place the cursor, and then click the ruler at the beginning of each column as shown
next. The wizard marks each column as shown in the following screens.
Step: 20) You must create a schema to host the ODI target datastore. To create a new RDBMS
schema for the ODI datastore, perform the following steps:
To create a user, enter the following command. Verify that user odi_demo is created
successfully.
Grant connects privileges to the newly created user by executing the following SQL command:
Step: 21) Now you need to create a new ODI datastore for the target model. This datastore
will be used within the ODI Interface. To create a new ODI target datastore, perform the
following steps:
Enter the username as odi_demo with the password odi_demo. For SID enter ORCL.
Step: 22) After you have created the new database target datastore, you need to create a new
ODI target data server and the physical schema. To create the ODI target data server and
physical schema, perform the following steps:
In ODI, open ODI Topology Navigator and then select the Physical Architecture tab. Expand
Technologies, right-click Oracle, and select New Data Server. In the Data Server: New window,
enter the values provided in the following image. Click the JDBC tab.
Step: 23) Click for JDBC Driver. Select Oracle JDBC Driver. Click OK. Click for JDBC Url, select
jdbc:oracle:thin:@<host>:<port>:<sid>, and then click OK. Edit the Url to read:
jdbc:oracle:thin:@localhost:1521:ORCL for Oracle Database.
Step: 24) Click Test Connection. In the window that opens, click Yes to save your data. In the
Information window, click OK. Click Test to verify a successful connection. Click OK.
Step: 25) Expand Oracle technology node, right-click Oracle Data Server, and then select New
Physical Schema.
In the Schema (Schema) and Schema (Work Schema) field, enter your ODI_Demo schema.
Click the Save button. Close the ODI_Demo.ODI_Demo physical schema window. In the
Information window, click OK.
Step: 26) Open the Logical Architecture tab, expand Technologies > Oracle. Right-click Oracle
technology and then select New Logical Schema.
Name logical schema ODI_Demo. In all three contexts, connect this logical schema to the
ODI_Demo physical schema as shown here. Click the Save button and then close the tabs.
Step: 27) Create a new ODI target model that will be used within your ODI Interface. To
create a new ODI target model, perform the following steps:
Open ODI Designer. Click the Models tab and select New Model. On the screen that appears,
enter the values provided in the following image. Click the Reverse Engineer tab.
Step: 28) On the Reverse Engineer tab, set the Context to Development. Click the Save button
and then close the RDBMS tab.
On the screen that follows, enter the interface name as INT Flat file to RDBMS. If selected,
deselect the Staging Area Different From Target check box. Click the Mapping tab.
Step: 31) Click the Models tab to drag the source and target to the diagram. Drag the Sales
Region.txt datastore from the Flat_File model into the Sources container. Drag the
TRG_Sales_Region datastore from the RDBMS model into the Target Datastore container.
When Designer asks “Do you want to perform an Automatic Mapping?” click Yes.
Step: 32) Click the Flow tab. Click the SrcSet_0 (Demonstration) datastore. The properties for
the source appear in the following screen. For LKM, select LKM File to SQL from the LKM
drop-down list if not selected. Now you need to ensure that the temporary objects created
during the transformation are deleted. Set the LKM option DELETE_TEMPORARY_OBJECTS to
<Default>:true as shown here.
Step: 33) Click the Target datastore. Select IKM SQL Incremental Update. Set the IKM option
Flow_Control to False. Set the IKM option DELETE_ALL to True. Click Save
Step: 34) To test your interface, click the Execute button. The following screen appears.
Retain the defaults and click OK. On the next screen, click OK.
Open ODI Operator and verify that your interface was executed successfully
Double-click Step 12 and click the Execution tab. View the number of rows inserted into the
target table
Step: 2) Enter the name of the project: Procedure-demo. Click the Save button to create the
project and close the tab. The project appears in the tree view. Expand the Procedure-demo
project node.
Create Procedure
Step: 3) Open ODI Designer. In the Projects tab, select Procedure-Demo --> First Folder.
Right-click Procedures and select New Procedure.
Step: 5) In the Command: New window, enter the parameters of the procedure and the
command on the target provided in the following details shown in the image. You need this
command to drop the table in case a table with this name exists. Select the Ignore Errors
check box.
Step: 6) Select the Details tab, and then click the icon to add another step in the procedure. In
the Command: New screen, if not selected, click the Command on Target tab. Enter the
parameters of the procedure and the command provided below.
Step: 7) Click the Details tab. Click the icon to enter the step to populate the
TRG_SALES_REGION table.
Step: 8) In the Command: New screen, click the Command on Target tab. Enter the
parameters and the command shown below in the image. Click the Details tab to view your
steps.
begin
end;
Step: 10) Open ODI Operator. In ODI Operator, click the All Executions node, click
the Refresh icon, find your session, and verify that your procedure executed
successfully.
Step: 11) Open the Oracle SQL developer and expand the ODI_Demo connection. Select Tables
node and click the Refresh button. Double–click the newly created table TRG_SALES_REGION,
click the Data tab, and verify that your table TRG_SALES_REGION was successfully created and
populated with records as shown in the following screen.