Overview of Datastage Parallelism

Server & Parallel Jobs
• ‘Parallel Jobs’ separate from the ‘Server Jobs’
• separate palette, different stages

• A single project can contain parallel as well as server jobs, can be called using the same

sequencer • Use Parallel jobs where there is a large volume of data to be processed • Overhead for management of parallel processes • Hardware support for parallelization • MPP – Single box, multiple share-nothing processors: Parallelization will give performance improvements • Clustered – Multiple boxes: Parallelization will give performance improvements • SMP processors – Shared memory, single OS • CPU-limited jobs – parallelization to increase number of processors used will help improve performance • Memory-limited jobs In SMP systems (with shared memory) may need hardware upgrade • Disk I/O limited jobs - Some SMP systems allow scalability of disk I/O, so that throughput improves as the number of processors increases. • Server jobs also support limited degree of parallelization

May 9, 2011


Types or Parallelism
• Internal Performance Enhancements
• Parallel execution of independent flows
Operator 1

Operator 2


Operator 3



Operator 4


Operator 5


Operator 6


Operator 7

Wait Point for dependent process

Pipelining: Each row is processed passed on to the next process in a pileline
Process 1 Row 4 Process 2 Row 3 Process 3 Row 2 Process 4 Row 1

Partitioning: rows split across different processes, each performing the same logic
Logical Unit
Partitioned Data

P1 P1 P1


Note that these options also depend on • support from the h/w & OS • server settings & configurations
May 9, 2011 3

DataStage Enterprise Edition
• Pipelining & Partitioning Combined
Pipeline processed for read, process & write


Oracle Oracle Oracle

N-way partitioned data read, process & write

Optimize the parallelism
• •

Better utilization of available CPU & hardware resources Overhead of copying, and preparing data, managing the processes

May 9, 2011


• By default,
• DataStage combines operators where possible • E.g. Two transform stages may combined into a single operator • Saves data copying & preparation

• This can be over-ridden at a project or stage level, if we wish to create separate processes
for each operator

May 9, 2011


Data Partitioning
Default • The configuration file decides how many process instances of each operator is created,
• • • •

e.g. if 4 nodes are defined, there is a 4-way partition of data By default, Auto-Partitioning is set DS chooses the optimum partitioning & repartitioning mechanism “Round-robin” is applied at the first level followed by “Same” If there is a need for key-based partition upstream or down-stream, then alternative modes are chosen • e.g. in the case of a join, the data in the input link is sorted & partitioned by the join key

Degree of parallelism • This is decided by the Configuration file. • The configuration file used can be varied at the job-level to suit different job requirements • Individual stages may also be executed on a selected nodes by specifying the node map constraints • Where the overhead of partitioning is not worth the performance improvement, the entire job or a specific stage may be executed sequentially. Avoid Repartitioning & Redundant Sorting • The Designer palette indicates links at which these occur (when not auto partitioning) • Designers may try to optimize this through changing the order in which the stages occur or otherwise modifying the jobs • Sequential files are non-partitioned so are less optimal intermediate storage formats. DataSets may be used to preserve partitioning across jobs.
May 9, 2011 6

Configuration File
The Configuration File
• System size & configuration details maintained external to the job design • Can be modified to suit development & production environment, handle hardware

upgrades, etc. without redesigning/recompiling jobs • The configuration file describes available processing power in terms of processing nodes • determines how many instances of a process will be produced when you compile a parallel job. • Minimum #Nodes < ½ times #CPUs Minimum Recommended • Usual starting point for #Nodes = # CPUs • # Nodes < # CPUs if some CPUs left free for OS, DB and other applications • # Nodes > # CPUs for I/O intensive streams with poor CPU-usage • Associates the scratchdisk with each node

May 9, 2011


Configuration File
{ node "node1" { fastname "ibmsceai" pools "" resource disk "/home/dsadm/Ascential/DataStage/Datasets" {pools""} resource scratchdisk "/home/dsadm/Ascential/DataStage/Scratch" {pools ""} } node "node2" { fastname "ibmsceai" pools "" resource disk "/home/dsadm/Ascential/DataStage/Datasets" {pools""} resource scratchdisk "/home/dsadm/Ascential/DataStage/Scratch" {pools ""} } }

May 9, 2011


Partitioning & Collecting
Partitioning Techniques : to be specified at each stage (or left as default) • General load-balancing techniques • ‘Round Robin’ – most efficient • ‘Random’ • ‘Same’ – use partition strategy of input feed, no repartitioning • ‘Entire’ – each node receives every row, e.g. to create a lookup file • Hash By Key – e.g to remove duplicates, aggregate over fewer groups • Key Modulo • Range – ensure related records come together • DB2 – same algorithm used by DB2 • Auto – leave it to DataStage – Usually Initially ‘Round-robin’, then ‘Same’ Collecting • Round Robin • Ordered – Read all records from first partition, then from second and so on • Sorted Merge – Read records based on one or more columns (collecting key)

May 9, 2011


Discussion on Partitioning Data Within A Job

• Assume that a <#Node> configuration file has been set • In the simplest case,
• when no specific options have been set data is processed as <#Node> partitions • Unless downstream stages require otherwise

• partition mechanism is Round Robin to begin with • subsequent stages have “Same”, i.e no repartitioning happens
• This normally gives maximum performance benefit & resource usage

• Set/Unset Designer menu options Diagram->Link Markings

May 9, 2011


Sequential Files - Read • Normally read in sequence, i.e 1 Process to read the data & pass it to the next stage • Output data is partitioned in round-robin into <#Node> partitions Processing Stages • Receives partitioned data & propagates it using the “Same” method Sequential Files – Write • Partitioned Data collected & written sequentially

Read Sequentially & Round-Robin Partition No Repartitioning here “Collected” & written sequentially Can sort on collection

Auto Partition sets it to “Same”
May 9, 2011 12

Sequential Files

• Executes in parallel when reading multiple

• For fixed-width files, can parallelize using
multiple readers per node OR Multiple Nodes

Multiple files => 1 Partition per file

On SMP systems: set “Number of Readers Per Node”

On cluster systems, set “Read From Multiple Nodes”

May 9, 2011


• Back to EE_TRG_Demo_1

Note that we did not set any specific options for parallelism or partitioning

Read Sequentially & Apply Partition (round-robin by default)

Icon for Auto Partition

May 9, 2011


• DataStage inserts Operators for sort, partition, buffering, etc.

Aggregator Stage needs Keypartitioned data Implicit Insert: Hash Partition

Join Stage expects all input links to be Key-Partitioned & Sorted. Partitioning mode must be the same Implicit Insert: Hash Partition (if required**) & Sort on Join Key(s)

** Note that in this case data output from aggregator is not partitioned again since it is already in the required partitioning format. It is only sorted
May 9, 2011 15

• USUALLY – Auto modes works sufficiently well • When would we worry about the partitioning mechanism?
• Some cases of debugging • Performance Tuning

• Look at Link Icons to identify where partitioning, explicit repartitioning & collection has occurred • Advanced Users: look at DataStage log for
• • • • •

What partition has been implicitly applied What and how the stages have been interpreted within the OSH How data is distributed across the partitions Where & why repartitioning occurs Note that the level of reporting will depend on the environment variable settings

• Tune parallelism
• • • •

through the configuration file running specific stages sequentially or on selected node pool(s) Changing the partition mode Enabling or disabling Operator Combinability, etc.

May 9, 2011


• Most Stages have an “Advanced” tab with parallelization options
Sequential/Parallel Combine operators (where possible) or Do not combine

Combine operators (where possible) or Do not combine

Constraints or limitations on which nodes are to be used

May 9, 2011


• Each Input Link into a stage has a Partitioning tab
Select Partitioning Type

Options on sorting incoming link data.

Select Partitioning Key (if applicable)

• If stage is executed sequentially & preceding stage is parallel, then the “Collection” options are available

May 9, 2011


• How can inappropriate partitioning cause wrong results to be produced? • If the partitioning mode, on say the Aggregation Stage, is for some reason explicitly set to a nonkey mode (such as round-robin), the stage will return WRONG RESULTS

Grp Key Grp Key A A A A B B B B Amt Val 10 20 30 40 50 60 70 80 Grp Key A A B B A A B B

Amt Val 10 30 50 70

Grp Key A B

Amt Val 40 120

Grp Key A

Amt Val 40 120 60 140

Amt Val 20 40 60 80

Grp Key A B

Amt Val 60 140


Round-Robin Partition

Aggregate within the partition

Wrong Output!

May 9, 2011


Case Study 2

Master your semester with Scribd & The New York Times

Special offer for students: Only $4.99/month.

Master your semester with Scribd & The New York Times

Cancel anytime.