Overview of Datastage Parallelism

Server & Parallel Jobs
• ‘Parallel Jobs’ separate from the ‘Server Jobs’
• separate palette, different stages

• A single project can contain parallel as well as server jobs, can be called using the same

sequencer • Use Parallel jobs where there is a large volume of data to be processed • Overhead for management of parallel processes • Hardware support for parallelization • MPP – Single box, multiple share-nothing processors: Parallelization will give performance improvements • Clustered – Multiple boxes: Parallelization will give performance improvements • SMP processors – Shared memory, single OS • CPU-limited jobs – parallelization to increase number of processors used will help improve performance • Memory-limited jobs In SMP systems (with shared memory) may need hardware upgrade • Disk I/O limited jobs - Some SMP systems allow scalability of disk I/O, so that throughput improves as the number of processors increases. • Server jobs also support limited degree of parallelization

May 9, 2011

2

Types or Parallelism
• Internal Performance Enhancements
• Parallel execution of independent flows
Operator 1
1a

Operator 2

1a

Operator 3

1a

2

Operator 4

1b

Operator 5

1b

Operator 6

1b

Operator 7

Wait Point for dependent process

Pipelining: Each row is processed passed on to the next process in a pileline
Process 1 Row 4 Process 2 Row 3 Process 3 Row 2 Process 4 Row 1

Partitioning: rows split across different processes, each performing the same logic
Logical Unit
Partitioned Data

P1 P1 P1

Data

Note that these options also depend on • support from the h/w & OS • server settings & configurations
May 9, 2011 3

DataStage Enterprise Edition
• Pipelining & Partitioning Combined
Pipeline processed for read, process & write

Oracle

Oracle Oracle Oracle

N-way partitioned data read, process & write

Optimize the parallelism
• •

Better utilization of available CPU & hardware resources Overhead of copying, and preparing data, managing the processes

May 9, 2011

4

Pipelining
• By default,
• DataStage combines operators where possible • E.g. Two transform stages may combined into a single operator • Saves data copying & preparation

• This can be over-ridden at a project or stage level, if we wish to create separate processes
for each operator

May 9, 2011

5

Data Partitioning
Default • The configuration file decides how many process instances of each operator is created,
• • • •

e.g. if 4 nodes are defined, there is a 4-way partition of data By default, Auto-Partitioning is set DS chooses the optimum partitioning & repartitioning mechanism “Round-robin” is applied at the first level followed by “Same” If there is a need for key-based partition upstream or down-stream, then alternative modes are chosen • e.g. in the case of a join, the data in the input link is sorted & partitioned by the join key

Degree of parallelism • This is decided by the Configuration file. • The configuration file used can be varied at the job-level to suit different job requirements • Individual stages may also be executed on a selected nodes by specifying the node map constraints • Where the overhead of partitioning is not worth the performance improvement, the entire job or a specific stage may be executed sequentially. Avoid Repartitioning & Redundant Sorting • The Designer palette indicates links at which these occur (when not auto partitioning) • Designers may try to optimize this through changing the order in which the stages occur or otherwise modifying the jobs • Sequential files are non-partitioned so are less optimal intermediate storage formats. DataSets may be used to preserve partitioning across jobs.
May 9, 2011 6

Configuration File
The Configuration File
• System size & configuration details maintained external to the job design • Can be modified to suit development & production environment, handle hardware

upgrades, etc. without redesigning/recompiling jobs • The configuration file describes available processing power in terms of processing nodes • determines how many instances of a process will be produced when you compile a parallel job. • Minimum #Nodes < ½ times #CPUs Minimum Recommended • Usual starting point for #Nodes = # CPUs • # Nodes < # CPUs if some CPUs left free for OS, DB and other applications • # Nodes > # CPUs for I/O intensive streams with poor CPU-usage • Associates the scratchdisk with each node

May 9, 2011

7

Configuration File
Sample
{ node "node1" { fastname "ibmsceai" pools "" resource disk "/home/dsadm/Ascential/DataStage/Datasets" {pools""} resource scratchdisk "/home/dsadm/Ascential/DataStage/Scratch" {pools ""} } node "node2" { fastname "ibmsceai" pools "" resource disk "/home/dsadm/Ascential/DataStage/Datasets" {pools""} resource scratchdisk "/home/dsadm/Ascential/DataStage/Scratch" {pools ""} } }

May 9, 2011

8

Partitioning & Collecting
Partitioning Techniques : to be specified at each stage (or left as default) • General load-balancing techniques • ‘Round Robin’ – most efficient • ‘Random’ • ‘Same’ – use partition strategy of input feed, no repartitioning • ‘Entire’ – each node receives every row, e.g. to create a lookup file • Hash By Key – e.g to remove duplicates, aggregate over fewer groups • Key Modulo • Range – ensure related records come together • DB2 – same algorithm used by DB2 • Auto – leave it to DataStage – Usually Initially ‘Round-robin’, then ‘Same’ Collecting • Round Robin • Ordered – Read all records from first partition, then from second and so on • Sorted Merge – Read records based on one or more columns (collecting key)

May 9, 2011

9

Discussion on Partitioning Data Within A Job

Partitioning
• Assume that a <#Node> configuration file has been set • In the simplest case,
• when no specific options have been set data is processed as <#Node> partitions • Unless downstream stages require otherwise

• partition mechanism is Round Robin to begin with • subsequent stages have “Same”, i.e no repartitioning happens
• This normally gives maximum performance benefit & resource usage

• Set/Unset Designer menu options Diagram->Link Markings

May 9, 2011

11

Partitioning
Sequential Files - Read • Normally read in sequence, i.e 1 Process to read the data & pass it to the next stage • Output data is partitioned in round-robin into <#Node> partitions Processing Stages • Receives partitioned data & propagates it using the “Same” method Sequential Files – Write • Partitioned Data collected & written sequentially

Read Sequentially & Round-Robin Partition No Repartitioning here “Collected” & written sequentially Can sort on collection

Auto Partition sets it to “Same”
May 9, 2011 12

Partitioning
Sequential Files

• Executes in parallel when reading multiple
files

• For fixed-width files, can parallelize using
multiple readers per node OR Multiple Nodes

Multiple files => 1 Partition per file

On SMP systems: set “Number of Readers Per Node”

On cluster systems, set “Read From Multiple Nodes”

May 9, 2011

13

Partitioning
• Back to EE_TRG_Demo_1

Note that we did not set any specific options for parallelism or partitioning

Read Sequentially & Apply Partition (round-robin by default)

Icon for Auto Partition

May 9, 2011

14

Partitioning
• DataStage inserts Operators for sort, partition, buffering, etc.

Aggregator Stage needs Keypartitioned data Implicit Insert: Hash Partition

Join Stage expects all input links to be Key-Partitioned & Sorted. Partitioning mode must be the same Implicit Insert: Hash Partition (if required**) & Sort on Join Key(s)

** Note that in this case data output from aggregator is not partitioned again since it is already in the required partitioning format. It is only sorted
May 9, 2011 15

Partitioning
• USUALLY – Auto modes works sufficiently well • When would we worry about the partitioning mechanism?
• Some cases of debugging • Performance Tuning

• Look at Link Icons to identify where partitioning, explicit repartitioning & collection has occurred • Advanced Users: look at DataStage log for
• • • • •

What partition has been implicitly applied What and how the stages have been interpreted within the OSH How data is distributed across the partitions Where & why repartitioning occurs Note that the level of reporting will depend on the environment variable settings

• Tune parallelism
• • • •

through the configuration file running specific stages sequentially or on selected node pool(s) Changing the partition mode Enabling or disabling Operator Combinability, etc.

May 9, 2011

16

Partitioning
• Most Stages have an “Advanced” tab with parallelization options
Sequential/Parallel Combine operators (where possible) or Do not combine

Combine operators (where possible) or Do not combine

Constraints or limitations on which nodes are to be used

May 9, 2011

17

Partitioning
• Each Input Link into a stage has a Partitioning tab
Select Partitioning Type

Options on sorting incoming link data.

Select Partitioning Key (if applicable)

• If stage is executed sequentially & preceding stage is parallel, then the “Collection” options are available

May 9, 2011

18

Partitioning
• How can inappropriate partitioning cause wrong results to be produced? • If the partitioning mode, on say the Aggregation Stage, is for some reason explicitly set to a nonkey mode (such as round-robin), the stage will return WRONG RESULTS

Grp Key Grp Key A A A A B B B B Amt Val 10 20 30 40 50 60 70 80 Grp Key A A B B A A B B

Amt Val 10 30 50 70

Grp Key A B

Amt Val 40 120

Grp Key A

Amt Val 40 120 60 140

Amt Val 20 40 60 80

Grp Key A B

Amt Val 60 140

B A B

Round-Robin Partition

Aggregate within the partition

Wrong Output!

May 9, 2011

19

Case Study 2

Master your semester with Scribd & The New York Times

Special offer for students: Only $4.99/month.

Master your semester with Scribd & The New York Times

Cancel anytime.