/  21
Best-Practice ETL for KALIDO® 8 using
Ascential DataStage™
October 2004
Gary Powell
Senior KALIDO Application Consultant
© Copyright 2004 Kalido
www.kalido.com
2
Table of contents
1
Introduction....................................................................................................... 3
The need for data integration tools in an enterprise data warehouse............................ 3
2
How ETL changes in a KALIDO environment...................................................... 5
3
Understanding the ETL target............................................................................ 9
4
Best Practice Techniques..................................................................................11
4.1

Extracting data from the source.................................................................... 11 Working with the business model......................................................................... 11 Full or delta extraction........................................................................................ 11 Handling time-variant reference data using transaction date..................................... 12 Handling changing codes..................................................................................... 13

4.2

Transforming data...................................................................................... 13 Data summarization and allocation....................................................................... 13 Currency conversions......................................................................................... 14 Writing reusable transforms................................................................................. 15

4.3
Loading data.............................................................................................. 15
4.4

Scheduling and job sequencing..................................................................... 16 Minimize build and maintenance requirements........................................................ 16 Trap and handle errors caused by process failure.................................................... 17 Optimizing performance...................................................................................... 18

5
Conclusions.......................................................................................................20
© Copyright 2004 Kalido
www.kalido.com
3
1 Introduction

Global 2000 companies now face greater regulatory and shareholder pressure than ever to increase corporate accountability, transparency and performance. As a result, they are creating enterprise data warehouses which enable multiple, corporate-wide views of business performance across disparate systems and organizations.

Unfortunately, it takes months (12-18 is not uncommon) to custom build or modify a data warehouse, and many are never even completed. As a result, business people fail to receive the timely management information they need to make high-quality decisions.

To address this challenge, companies are taking a more iterative approach to enterprise data warehousing using KALIDO®, which automates the creation and maintenance of enterprise data warehouses and master data throughout their lifecycle. The KALIDO application suite automatically adapts data warehouses and their associated master data to new business needs based on changes made to real-world business models. Kalido customers create and modify data warehouses 75% faster and at half the cost of traditional approaches.

The need for data integration tools in an enterprise data warehouse

A data warehouse typically sources data from a wide range of IT systems and organizations. To help deliver this data to the data warehouse, companies deploy tools such as Ascential Enterprise Integration Suite™, which includes:

Ascential ProfileStage™ - data profiling to evaluate source data content and structure
Ascential QualityStage™ - data cleansing to find & reconcile low-quality or redundant data
Ascential DataStage™ - data extraction, transformation and loading (ETL)
Ascential MetaStage™ - metadata management for definitions and history of business data
Figure 1 - Kalido and Ascential Products
ANY SOURCE
CRM
ERP

SCM
RDBMS
Legacy

EAI /

Messaging Web svc XML/EDI

Legacy DW
Parallel Execution Engine
DISCOVER
DISCOVER
PREPARE
PREPARE
TRANSFORM
TRANSFORM
ProfileStage
ProfileStage
QualityStage
QualityStage
DataStage
DataStage
Meta Data Management
Real-Time Integration Services
Enterprise Connectivity
and Event Management
Service-Oriented Architecture
Data Warehouse
Schema & Content
Adaptive Services Core
Auto-Builds
Business Model &
Master Data
Region
Country
District
Target
Industry
Sector
• Size
Account
• Contract
Geo Address
Target Cust
Manager
Account Grp
Partners
Del Point
Builds
Universes
Cubes
Marts
Reports
Master Data
Auto-Builds
Transactions
Ascential Enterprise Integration Suite
KALIDO
Business
Intelligence
ANY SOURCE
CRM
ERP

SCM
RDBMS
Legacy

EAI /

Messaging Web svc XML/EDI

Legacy DW
Parallel Execution Engine
DISCOVER
DISCOVER
PREPARE
PREPARE
TRANSFORM
TRANSFORM
ProfileStage
ProfileStage
QualityStage
QualityStage
DataStage
DataStage
Meta Data Management
Real-Time Integration Services
Enterprise Connectivity
and Event Management
Service-Oriented Architecture
Data Warehouse
Schema & Content
Adaptive Services Core
Auto-Builds
Business Model &
Master Data
Region
Country
District
Target
Industry
Sector
• Size
Account
• Contract
Geo Address
Target Cust
Manager
Account Grp
Partners
Del Point
Business Model &
Master Data
Region
Country
District
Target
Industry
Sector
• Size
Account
• Contract
Geo Address
Target Cust
Manager
Account Grp
Partners
Del Point
Region
Country
District
Region
Country
District
Region
Country
District
Target
Industry
Sector
• Size
Target
Industry
Sector
• Size
Account
• Contract
Geo Address
Target Cust
Manager
Account Grp
Partners
Del Point
Account
• Contract
Geo Address
Target Cust
Manager
Account Grp
Partners
Del Point
Builds
Universes
Cubes
Marts
Reports
Universes
Universes
Cubes
Marts
Marts
Reports
Reports
Master Data
Auto-Builds
Transactions
Ascential Enterprise Integration Suite
KALIDO
Business
Intelligence

Share & Embed

More from this user

Add a Comment

Characters: ...