Introduction to DataStage

IBM Infosphere DataStage v11.5

© Copyright IBM Corporation 2015

Course materials may not be reproduced in whole or in part without the written permission of IBM.
Unit objectives
• List and describe the uses of DataStage
• List and describe the DataStage clients
• Describe the DataStage workflow
• Describe the two types of parallelism exhibited by DataStage
parallel jobs

Introduction to DataStage © Copyright IBM Corporation 2015

What is IBM InfoSphere DataStage?
• Design jobs for Extraction, Transformation, and Loading (ETL)
• Ideal tool for data integration projects – such as, data warehouses,
data marts, and system migrations
• Import, export, create, and manage metadata for use within jobs
• Build, run, and monitor jobs, all within DataStage
• Administer your DataStage development and execution environments
• Create batch (controlling) jobs
 Called job sequence

Introduction to DataStage © Copyright IBM Corporation 2015

What is Information Server?
• Suite of applications, including DataStage, that share a common:
 Repository
 Set of application services and functionality
− Provided by the Metadata Server component
• By default an application named “server1”, hosted by an IBM WebSphere
Application Server (WAS) instance
− Provided services include:
• Security
• Repository
• Logging and reporting
• Metadata management
• Managed using the Information Server Web Console client

Introduction to DataStage © Copyright IBM Corporation 2015

Information Server backbone

Information Information Information FastTrack DataStage / MetaBrokers

Data Click
Services Governance Analyzer QualityStage
Director Catalog

Metadata Metadata
Access Services Analysis Services

Metadata Server

Information Server Web Console

Introduction to DataStage © Copyright IBM Corporation 2015

Information Server Web Console

Administration Reporting

InfoSphere
Users

Introduction to DataStage © Copyright IBM Corporation 2015

DataStage architecture
• DataStage clients

Administrator Designer Director

• DataStage engines
 Parallel engine
− Runs parallel jobs
 Server engine
− Runs server jobs
− Runs job sequences

Introduction to DataStage © Copyright IBM Corporation 2015

DataStage Administrator

Project environment
variables

Introduction to DataStage © Copyright IBM Corporation 2015

DataStage Designer

Menus / toolbar

DataStage parallel
job with DB2
Connector stage

Job log

Introduction to DataStage © Copyright IBM Corporation 2015

DataStage Director

Log messages

Introduction to DataStage © Copyright IBM Corporation 2015

Developing in DataStage
• Define global and project properties in Administrator
• Import metadata into the Repository
 Specifies formats of sources and targets accessed by your jobs
• Build job in Designer
• Compile job in Designer
• Run the job and monitor job log messages
 The job log can be viewed either in Director or in Designer
− In Designer, only the job log for the currently opened job is available
 Jobs can be run from either Director, Designer, or from the command line
 Performance statistics show up in the log and also on the Designer canvas
as the job runs

Introduction to DataStage © Copyright IBM Corporation 2015

DataStage project repository

User-added folder

Standard jobs folder

Standard table
definitions folder

Introduction to DataStage © Copyright IBM Corporation 2015

Types of DataStage jobs
• Parallel jobs
 Executed by the DataStage parallel engine
 Built-in capability for pipeline and partition parallelism
 Compiled into OSH
− Executable script viewable in Designer and the log
• Server jobs
 Executed by the DataStage Server engine
 Use a different set of stages than parallel jobs
 No built-in capability for partition parallelism
 Runtime monitoring in the job log
• Job sequences (batch jobs, controlling jobs)
 A server job that runs and controls jobs and other activities
 Can run both parallel jobs and other job sequences
 Provides a common interface to the set of jobs it controls
Introduction to DataStage © Copyright IBM Corporation 2015
Design elements of parallel jobs
• Stages
 Passive stages (E and L of ETL)
− Read data
− Write data
− Examples: Sequential File, DB2, Oracle, Peek stages
 Processor (active) stages (T of ETL)
− Transform data (Transformer stage)
− Filter data (Transformer stage)
− Aggregate data (Aggregator stage)
− Generate data (Row Generator stage)
− Merge data (Join, Lookup stages)
• Links
 "Pipes” through which the data moves from stage-to-stage

Introduction to DataStage © Copyright IBM Corporation 2015

Pipeline parallelism

• Transform, Enrich, Load stages execute in parallel

• Like a conveyor belt moving rows from stage to stage
 Run downstream stages while upstream stages are running
• Advantages:
 Reduces disk usage for staging areas
 Keeps processors busy
• Has limits on scalability
Introduction to DataStage © Copyright IBM Corporation 2015
Partition parallelism
• Divide the incoming stream of data into subsets to be separately
processed by an operation
 Subsets are called partitions
• Each partition of data is processed by copies the same stage
 For example, if the stage is Filter, each partition will be filtered in exactly
the same way
• Facilitates near-linear scalability
 8 times faster on 8 processors
 24 times faster on 24 processors
 This assumes the data is evenly distributed

Introduction to DataStage © Copyright IBM Corporation 2015

Three-node partitioning
Node 1

subset1 Stage

Node 2
subset2
Data Stage

Node 3
subset3
Stage

• Here the data is split into three partitions (nodes)

• The stage is executed on each partition of data separately and in
parallel
• If the data is evenly distributed, the data will be processed three
times faster

Introduction to DataStage © Copyright IBM Corporation 2015

Job design versus execution

A developer designs the flow in DataStage Designer

… at runtime, this job runs in parallel for any number

of partitions (nodes)

Introduction to DataStage © Copyright IBM Corporation 2015

Configuration file
• Determines the degree of parallelism (number of partitions) of jobs
that use it
• Every job runs under a configure file
• Each DataStage project has a default configuration file
 Specified by the $APT_CONFIG_FILE job parameter
 Individual jobs can run under different configuration files than the project
default
− The same job can also run using different configuration files on different job runs

Introduction to DataStage © Copyright IBM Corporation 2015

Example: Configuration file

Node (partition)

Node (partition)

Resources attached
to the node

Introduction to DataStage © Copyright IBM Corporation 2015

Checkpoint
1. True or false: DataStage Director is used to build and compile your
ETL jobs
2. True or false: Use Designer to monitor your job during execution
3. True or false: Administrator is used to set global and project
properties

Introduction to DataStage © Copyright IBM Corporation 2015

Checkpoint solutions
1. False.
DataStage Designer is used to build and compile jobs.
Use DataStage Director to run and monitor jobs, but you can do this
from DataStage Designer too.
2. True.
The job log is available both in Director and Designer. In Designer,
you can only view log messages for a job open in Designer.
3. True.

Introduction to DataStage © Copyright IBM Corporation 2015

Unit summary
• List and describe the uses of DataStage
• List and describe the DataStage clients
• Describe the DataStage workflow
• Describe the two types of parallelism exhibited by DataStage parallel
jobs

Introduction to DataStage © Copyright IBM Corporation 2015