 You can view details about a workflow or task in either a Gantt

1. GETTING STARTED chart view or a Task view. You can start, stop, and abort
workflows from the Workflow Monitor.
 Metadata Extensions tab
 The Workflow Monitor displays workflows that have run at least
 Execute SQL to create table
once.
 The source qualifier represents the rows that the Integration
 A transformation is a part of a mapping that generates or
Service reads from the source when it runs a session
modifies data.
 Every mapping includes a Source Qualifier transformation,
 A session is a set of instructions that tells the Integration Service
representing all data read from a source and temporarily stored
how to move data from sources to targets.
by the Integration Service.
 A session is a task, similar to other tasks available in the
Workflow Manager.
 The Advanced Transformation toolbar contains transformations
 You create a session for each mapping that you want the
such as Java, SQL, and XML Parser transformations
Integration Service to run.
 The Integration Service uses the instructions configured in the
Creating Target -
session and mapping to move data from sources to targets.
 You can also manually create a target definition, import the
definition for an existing target from a database, or create a
 A workflow is a set of instructions that tells the Integration
relational target from a transformation in the Designer
Service how to execute tasks, such as sessions, email
notifications, and shell commands.
 Click the Indexes tab to add an index to the target table.
 You create a workflow for sessions you want the Integration
 If the target database is Oracle, skip to the final step. You cannot
Service to run.
add an index to a column that already has the PRIMARY KEY
 You can include multiple sessions in a workflow to run sessions in
constraint added to it.
parallel or sequentially.
 The Integration Service uses the instructions configured in the
 Click Layout > Link Columns.
workflow to run sessions and other tasks.
 When you drag ports from one transformation to another, the
Designer copies the port description and links the original port
 **Select a code page for the database connection (DB
to its copy.
Connection Objects)
 If you click Layout > Copy Columns, every port you drag is copied,
 The source code page must be a subset of the target code page.
but not linked.
 The target code page must be a superset of the source code
 Link Columns vs. Copy columns??
page.

 By default, the Lookup transformation queries and stores the

 You can create reusable or non-reusable sessions in the Workflow
contents of the lookup table before the rest of the
Manager
transformation runs, so it performs the join through a local copy
of the table that it has cached
 When you create a workflow, you can include reusable tasks that
 If the data types, including precision and scale, of these two
you create in the Task Developer.
columns do not match, the Designer displays a message and
 You can also include non-reusable tasks that you create in the
marks the mapping invalid.
Workflow Designer.

 **Overview window
 By default, the workflow is scheduled to run on demand. The
 When you run the workflow, the Integration Service runs all
Integration Service only runs the workflow when you manually
sessions in the workflow, either simultaneously or in sequence,
start the workflow. You can configure workflows to run on a
depending on how you arrange the sessions in the workflow.
schedule

 Click the Gantt Chart tab at the bottom of the Time window to
 All workflows begin with the Start task, but you need to instruct
verify the Workflow Monitor is in Gantt Chart view.
the Integration Service which task to run next.
 Note: You can also click the Task View tab at the bottom of the
 To do this, you link tasks in the Workflow Manager
Time window to view the Workflow Monitor in Task view. You
can switch back and forth between views at any time.
 When the PowerCenter Integration Service runs workflows, you
can monitor workflow progress in the Workflow Monitor.

1
 From the list of source definitions, add the following source
definitions to the mapping:
- PROMOTIONS
- ITEMS_IN_PROMOTIONS
- ITEMS
- MANUFACTURERS
- ORDER_ITEMS
 Delete all Source Qualifier transformations that the Designer
creates when you add these source definitions
 Add a Source Qualifier transformation named SQ_AllData to the
mapping, and connect all the source definitions to it.

Sequence Generator -
 The starting number (normally 1).
 The current value stored in the repository.
 The number that the Sequence Generator transformation adds to
its current value for every request for a new ID.
 The maximum value in the sequence.
 A flag indicating whether the Sequence Generator transformation
counter resets to the minimum value once it has reached its
maximum value.

 The Sequence Generator transformation has two output ports,

NEXTVAL and CURRVAL, which correspond to the two pseudo-
columns in a sequence. When you query a value from the
NEXTVAL port, the transformation generates a new value.
 cannot add any new ports to this transformation or reconfigure
NEXTVAL and CURRVAL

Defining a Link Condition

 After you create links between tasks, you can specify conditions
for each link to determine the order of execution in the
workflow. If you do not specify conditions for each link, the
Integration Service executes the next task in the workflow by
default

 If the link condition evaluates to True, the Integration Service

runs the next task in the workflow. You can also use predefined
or user-defined workflow variables in the link condition.
 You can use the -- or // comment indicators with the Expression
Editor to add comments. Use comments to describe the
expression
 You can view results of link evaluation during workflow runs in
the workflow log

XML – Creating sources/targets etc

2. INFA ADMIN GUIDE

 A domain is a collection of nodes and services that you can group
in folders based on administration ownership

2
 One node in the domain acts as a gateway to receive service
requests from clients and route them to the appropriate service
and node.
 Services and processes run on nodes in a domain.
Service Manager - A service that manages all domain operations
Application Services - Services that represent server-based
functionality, such as the Model Repository Service and the Data
Integration Service.
 The Service Manager and application services control security.
 The Service Manager manages users and groups that can log in to
application clients and authenticates the users who log in to the
application clients.
 The Service Manager and application services authorize user
requests from application clients.
 Informatica Administrator (the Administrator tool), consolidates
the administrative tasks for domain objects such as services,
nodes, licenses, and grids. You manage the domain and the
security of the domain through the Administrator tool
 When you start Informatica services, you start the Service
Manager.
 The Service Manager runs on each node.
 If the Service Manager is not running, the node is not available.
Application service support -
 It starts and stops services and service processes based on
requests from clients.
 It also directs service requests to application services.
 The Service Manager uses TCP/IP to communicate with the
application services
Domain support -
 The functions that the Service Manager performs on a node
depend on the type of node.
 For example, the Service Manager running on the master
gateway node performs all domain functions on that node.
 The Service Manager running on any other node performs some
domain functions on that node

Nodes -
 Each node in the domain runs a Service Manager that manages
domain operations on that node
 The operations that the Service Manager performs depend on
the type of node.
 A node can be a gateway node or a worker node
 You can subscribe to alerts to receive notification about node
events such as node failure or a master gateway election.
 You can also generate and upload node diagnostics to the
Configuration Support Manager and review information such as
available EBFs and Informatica recommendations

Gateway Nodes -
 One node acts as the gateway at any given time.
 That node is called the master gateway.
 A gateway node can run application services, and it can serve as a
master gateway node.
 The master gateway node is the entry point to the domain
Application services represent server-based functionality.
Application services include the following services:
 You can configure more than one node to serve as a gateway.
- Analyst Service
 If the master gateway node becomes unavailable, the Service
- Content Management Service
Manager on other gateway nodes elects another master
- Data Integration Service
gateway node.
- Metadata Manager Service
 If you configure one node to serve as the gateway and the node
- Model Repository Service
becomes unavailable, the domain cannot accept service
- PowerCenter Integration Service
requests
- PowerCenter Repository Service
- PowerExchange Listener Service
Worker Nodes -
- PowerExchange Logger Service
 A worker node is any node not configured to serve as a gateway.
- Reporting Service
Service Manager -
- SAP BW Service
 It runs as a service on Windows and as a daemon on UNIX.
- Web Services Hub

3

High Availability
High availability consists of the following components:
Resilience - The ability of application services to tolerate
transient network failures until either the resilience timeout
expires or the external system failure is fixed.
Failover - The migration of an application service or task to
another node when the node running the service process
becomes unavailable.
Recovery - The automatic completion of tasks after a service is
interrupted. Automatic recovery is available for PowerCenter
Integration Service and PowerCenter Repository Service tasks.
You can also manually recover PowerCenter Integration Service
workflows and sessions. Manual recovery is not part of high
availability
Informatica Administrator
Use the Administrator tool to complete the following types of
tasks:
Domain administrative tasks -
 Manage logs, domain objects, user permissions, and domain
reports.
 Generate and upload node diagnostics.
 Monitor jobs and applications that run on the Data Integration
Service.
 Domain objects include application services, nodes, grids, folders,
database connections, operating system profiles, and licenses.
Security administrative tasks -
 Manage users, groups, roles, and privileges
The Administrator tool has the following tabs:
Domain -
 View and edit the properties of the domain and objects within
the domain.
 The contents that appear and the tasks you can complete on the
Domain tab vary based on the view that you select.
 You can select the following views:
 Services and Nodes - View and manage application services,
nodes, grids, licenses
 Connections - View and manage connections
Logs - View log events for the domain and services within the
domain.
Monitoring - View the status of profile jobs, scorecard jobs,
preview jobs, mapping jobs, and SQL data services for each Data
Integration Service.
Reports - Run a Web Services Report or License Management
Report.
Security - Manage users, groups, roles, and privileges.
4
Folder Management
 Folders can contain nodes, services, grids, licenses, and other
folders.
User Accounts
Default Administrator -
 The default administrator is a user account in the native security
domain.
 You cannot create a default administrator.
 You cannot disable or modify the user name or privileges of the
default administrator.
 You can change the default administrator password
Domain Administrator -
 A domain administrator can create and manage objects in the
domain, including user accounts, nodes, grids, licenses, and
application services
 However, by default, the domain administrator cannot log in to
application clients.
 The default administrator must explicitly give a domain
administrator full permissions and privileges to the application
services
Application Client Administrator -
 An application client administrator can create and manage
objects in an application client.
 You must create administrator accounts for the application clients
 By default, the application client administrator does not have
permissions or privileges on the domain.
User -
 A user with an account in the Informatica domain can perform
tasks in the application clients.
 Typically, the default administrator or a domain administrator
creates and manages user accounts and assigns roles,
permissions, and privileges in the Informatica domain.
 However, any user with the required domain privileges and
permissions can create a user account and assign roles,
permissions, and privileges
Understanding Authentication and Security Domains -
 When a user logs in to an application client, the Service Manager
authenticates the user account in the Informatica domain and
verifies that the user can use the application client.
 The Service Manager uses native and LDAP authentication to
authenticate users logging in to the Informatica domain.
 You can use more than one type of authentication in an
Informatica domain.
 By default, the Informatica domain uses native authentication.
 You can configure the Informatica domain to use LDAP
authentication in addition to native authentication.
Privileges
Informatica includes the following privileges:
Domain privileges - Determine actions on the Informatica
domain that users can perform using the Administrator tool and
the infacmd and pmrep command line programs.
Analyst Service privilege - Determines actions that users can
perform using Informatica Analyst.
Data Integration Service privilege - Determines actions on
applications that users can perform using the
Administrator tool and the infacmd command line program -
This privilege also determines whether users can drill down and
export profile results.
Metadata Manager Service privileges - Determine actions that
users can perform using Metadata Manager.
Model Repository Service privilege - Determines actions on
projects that users can perform using Informatica Analyst and
Informatica Developer.
PowerCenter Repository Service privileges - Determine
PowerCenter repository actions that users can perform using the
Repository Manager, Designer, Workflow Manager, Workflow
Monitor, and the pmrep and pmcmd command line programs.
PowerExchange application service privileges - Determine
actions that users can perform on the PowerExchange Listener
Service and PowerExchange Logger Service using the infacmd
pwx commands.
Reporting Service privileges - Determine reporting actions that
users can perform using Data Analyzer.
 You assign privileges to users and groups and to application
services.
 You assign privileges to users and groups on the Security tab of
the Administrator tool
High Availability
 If you have the high availability option, you can achieve full high
availability of internal Informatica components.
 You can achieve high availability with external components based
on the availability of those components.
 If you do not have the high availability option, you can achieve
some high availability of internal components
Example
 While you are fetching a mapping into the PowerCenter Designer
workspace, the PowerCenter Repository Service becomes
unavailable, and the request fails. The PowerCenter Repository
Service fails over to another node because it cannot restart on
the same node.
5
 The PowerCenter Designer is resilient to temporary failures and
tries to establish a connection to the PowerCenter Repository
Service.
 The PowerCenter Repository Service starts within the resilience
timeout period, and the PowerCenter Designer reestablishes the
connection.
 After the PowerCenter Designer reestablishes the connection, the
PowerCenter Repository Service recovers from the failed
operation and fetches the mapping into the PowerCenter
Designer workspace.
Resilience -
 All clients of PowerCenter components are resilient to service
failures.
 PowerCenter services may also be resilient to temporary failures
of external systems, such as database systems, FTP servers, and
message queue sources.
 For this type of resilience to work, the external systems must be
highly available
Internal Resilience
 You can configure internal resilience at the following levels:
- Domain
- Application Services
- Gateway
 The Model Repository, Data Integration Service, and Analyst
Service do not have internal resilience.
 If the master gateway node becomes unavailable and fails over to
another gateway node, you must restart these services.
 After the restart, the services do not restore the state of
operation and do not recover from the point of interruption.
 You must restart jobs that were previously running during the
interruption
High Availability in the Base Product
 Informatica provides some high availability functionality that
does not require the high availability option.
 The base product provides the following high availability
functionality:
 Internal PowerCenter resilience - The Service Manager,
application services, PowerCenter Client, and command line
programs are resilient to temporary unavailability of other
PowerCenter internal components.
 PowerCenter Repository database resilience - The PowerCenter
Repository Service is resilient to temporary unavailability of the
repository database.
 Restart services - The Service Manager can restart application
services after a failure.
 Manual recovery of PowerCenter workflows and sessions - You
can manually recover PowerCenter workflows and sessions.
 Multiple gateway nodes - You can configure multiple nodes as
gateway.
Note: You must have the high availability option for failover and
automatic recovery

You can configure the following resilience properties for the

domain, application services, and command line programs:
Resilience timeout - The amount of time a client tries to connect
or reconnect to a service. A limit on resilience timeouts can
override the timeout.
Limit on resilience timeout - The amount of time a service waits
for a client to connect or reconnect to the service. This limit can
override the client resilience timeouts configured for a
connecting client. This is available for the domain and application
services

Configuring Service Resilience for the Domain

 The domain resilience timeout determines how long services try
to connect as clients to other services.
 The default value is 30 seconds.

 The limit on resilience timeout is the maximum amount of time

that a service allows another service to connect as a client.
 This limit overrides the resilience timeout for the connecting
service if the resilience timeout is a greater value.
 The default value is 180 seconds.

 The PowerCenter Client resilience timeout is 180 seconds and is

not configurable.
 This resilience timeout is bound by the service limit on resilience
timeout.
When you use a command line program to connect to the
domain or an application service, the resilience timeout is
determined by one of the following values:
Command line option - You can determine the resilience timeout
for command line programs by using a command line option,
-timeout or -t, each time you run a command
Environment variable - If you do not use the timeout option in
the command line syntax, the command line program uses the
value of the environment variable
INFA_CLIENT_RESILIENCE_TIMEOUT that is configured on the
client machine.
Default value - If you do not use the command line option or the
environment variable, the command line program uses the
default resilience timeout of 180 seconds.
Limit on timeout - If the limit on resilience timeout for the
service is smaller than the command line resilience timeout, the
command line program uses the limit as the resilience timeout
PowerCenter Integration Service
 PowerCenter Integration Service files include run-time files, state
of operation files, and session log files.
 The PowerCenter Integration Service creates files to store the
state of operations for the service.
 The state of operations includes information such as the active
service requests, scheduled tasks, and completed and running
processes.
 If the service fails, the PowerCenter Integration Service can
restore the state and recover operations from the point of
interruption.
To move data from sources to targets, the PowerCenter
Integration Service uses the following components:
PowerCenter Integration Service process - The PowerCenter
Integration Service starts one or more PowerCenter Integration
Service processes to run and monitor workflows. When you run a
workflow, the PowerCenter Integration Service process starts and
locks the workflow, runs the workflow tasks, and starts the
process to run sessions.
Load Balancer - The PowerCenter Integration Service uses the
Load Balancer to dispatch tasks. The Load Balancer dispatches

6
 tasks to achieve optimal performance. It may dispatch tasks to a
single node or across the nodes in a grid.
Data Transformation Manager (DTM) process - The PowerCenter
Integration Service starts a DTM process to run each Session and
Command task within a workflow. The DTM process performs
session validations, creates threads to initialize the session, read,
write, and transform data, and handles pre- and post- session
operations.
 PowerCenter Integration Service process accepts requests from
the PowerCenter Client and from pmcmd and performs the
following tasks:
- Manage workflow scheduling.
- Lock and read the workflow.
- Read the parameter file.
- Create the workflow log.
- Run workflow tasks and evaluates the conditional links
connecting tasks.
- Start the DTM process or processes to run the session.
- Write historical run information to the repository.
- Send post-session email in the event of a DTM failure.
Thread Types
 The types of threads the master thread creates depend on the
pre- and post-session properties, as well as the types of
transformations in the mapping.
The master thread can create the following types of threads:
Mapping threads -
The master thread creates one mapping thread for each session.
The mapping thread fetches session and mapping information,
compiles the mapping, and cleans up after session execution
Pre- and post-session threads -
The master thread creates one pre-session and one post-session
thread to perform pre- and post-session operations.
Reader threads
Transformation threads
The number of transformation threads depends on the
partitioning information for each pipeline
Writer threads
Reject Files
 By default, the PowerCenter Integration Service process creates a
reject file for each target in the session.
The writer may reject a row in the following circumstances:
- It is flagged for reject by an Update Strategy or Custom
transformation.
- It violates a database constraint such as primary key constraint.
- A field in the row was truncated or overflowed, and the target
database is configured to reject truncated or overflowed data.
Note: If you enable row error logging, the PowerCenter
Integration Service process does not create a reject file.
7
Recovery Tables Files
 The PowerCenter Integration Service process creates recovery
tables on the target database system when it runs a session
enabled for recovery.
Control File
 When you run a session that uses an external loader, the
PowerCenter Integration Service process creates a control file
and a target flat file.
 The control file contains information about the target flat file
such as data format and loading instructions for the external
loader.
 The control file has an extension of .ctl.
 The PowerCenter Integration Service process creates the control
file and the target flat file in the PowerCenter Integration
Service variable directory, $PMTargetFileDir, by default.
Indicator File
 If you use a flat file as a target, you can configure the
PowerCenter Integration Service to create an indicator file for
target row type information.
 For each target row, the indicator file contains a number to
indicate whether the row was marked for insert, update, delete,
or reject.
 The PowerCenter Integration Service process names this file
target_name.ind and stores it in the PowerCenter Integration
Service variable directory, $PMTargetFileDir, by default
Cache Files
 When the PowerCenter Integration Service process creates
memory cache, it also creates cache files.
 The PowerCenter Integration Service process creates cache files
for the following mapping objects:
- Aggregator transformation
- Joiner transformation
- Rank transformation
- Lookup transformation
- Sorter transformation
- XML target
 By default, the DTM creates the index and data files for
Aggregator, Rank, Joiner, Lookup transformations and XML
targets in the directory configured for the $PMCacheDir service
process variable.
 The PowerCenter Integration Service process names the index file
PM*.idx, and the data file PM*.dat.
 The PowerCenter Integration Service process creates the cache
file for a Sorter transformation in the $PMTempDir service
process variable directory.
Incremental Aggregation Files
 If the session performs incremental aggregation, the
PowerCenter Integration Service process saves index and data
cache information to disk when the session finished.
 The next time the session runs, the PowerCenter Integration
Service process uses this historical information to perform the
incremental aggregation.
 By default, the DTM creates the index and data files in the
directory configured for the $PMCacheDir service process
variable
Persistent Lookup Cache
 By default, the DTM creates the index and data files in the
directory configured for the $PMCacheDir service process
variable
Metric-based - The Load Balancer evaluates nodes in a round-
robin fashion. It checks all resource provision thresholds on each
available node and excludes a node if dispatching a task causes
the thresholds to be exceeded. The Load Balancer continues to
evaluate nodes until it finds a node that can accept the task. This
mode prevents overloading nodes when tasks have uneven
computing requirements.
Adaptive - The Load Balancer ranks nodes according to current
CPU availability. It checks all resource provision thresholds on
each available node and excludes a node if dispatching a task
causes the thresholds to be exceeded. This mode prevents
overloading nodes and ensures the best performance on a grid
that is not heavily loaded.

Load Balancer
You configure the following settings for the domain to determine
how the Load Balancer dispatches tasks:
Dispatch mode - The dispatch mode determines how the Load
Balancer dispatches tasks. You can configure the Load Balancer to
dispatch tasks in a simple round-robin fashion, in a round-robin
fashion using node load metrics, or to the node with the most
available computing resources.
Service level - Service levels establish dispatch priority among
tasks that are waiting to be dispatched. You can create different
service levels that a workflow developer can assign to workflows

You configure the following Load Balancer settings for each node:
Resources - When the PowerCenter Integration Service runs on a
grid, the Load Balancer can compare the resources required by a
task with the resources available on each node. The Load
Balancer dispatches tasks to nodes that have the required
resources. You assign required resources in the task properties.
You configure available resources using the Administrator tool or
infacmd.
CPU profile - In adaptive dispatch mode, the Load Balancer uses
the CPU profile to rank the computing throughput of each CPU
and bus architecture in a grid. It uses this value to ensure that
more powerful nodes get precedence for dispatch.
Resource provision thresholds - The Load Balancer checks one or
more resource provision thresholds to determine if it can
dispatch a task. The Load Balancer checks different thresholds
depending on the dispatch mode
The Load Balancer uses the following dispatch modes:
Round-robin - The Load Balancer dispatches tasks to available
nodes in a round-robin fashion. It checks the Maximum Processes
threshold on each available node and excludes a node if
dispatching a task causes the threshold to be exceeded. This
mode is the least compute-intensive and is useful when the load
on the grid is even and the tasks to dispatch have similar
computing requirements.
USING PMCMD
 pmcmd is a program you use to communicate with the
Integration Service.
 With pmcmd, you can perform some of the tasks that you can
also perform in the Workflow Manager, such as starting and
stopping workflows and sessions.
Use pmcmd in the following modes:
Command line mode - You invoke and exit pmcmd each time you
issue a command. You can write scripts to schedule workflows
with the command line syntax. Each command you write in
command line mode must include connection information to the
Integration Service.
Interactive mode - You establish and maintain an active
connection to the Integration Service. This lets you issue a series
of commands.

8
 You can use environment variables for user names and passwords
with pmcmd.
 You can also use environment variables to customize the way
pmcmd displays the date and time on the machine running the
Integration Service process.
 Before you use pmcmd, configure these variables on the machine
running the Integration Service process.
 The environment variables apply to pmcmd commands that run
on the node.
 Note: If the domain is a mixed-version domain, run pmcmd from
the installation directory of the Integration Service version
Running Commands in Command Line Mode
 When you run pmcmd in command line mode, you enter
connection information such as domain name, Integration
Service name, user name and password in each command.
 For example, to start the workflow “wf_SalesAvg” in folder
“SalesEast,” use the following syntax:
pmcmd startworkflow -sv MyIntService -d MyDomain -u seller3 -p
jackson -f SalesEast wf_SalesAvg
Running Commands in Interactive Mode
 Use pmcmd in interactive mode to start and stop workflows and
sessions without writing a script.
 When you use the interactive mode, you enter connection
information such as domain name, Integration Service name,
user name, and password. You can run subsequent commands
without entering the connection information for each
command.
 For example, the following commands invoke the interactive
mode, establish a connection to Integration Service
“MyIntService,” and start workflows “wf_SalesAvg” and
“wf_SalesTotal” in folder “SalesEast”:
pmcmd
pmcmd> connect -sv MyIntService -d MyDomain -u seller3 -p
jackson
pmcmd> setfolder SalesEast
pmcmd> startworkflow wf_SalesAvg
pmcmd> startworkflow wf_SalesTotal
USING PMREP
 pmrep is a command line program that you use to update
repository information and perform repository functions.
 pmrep is installed in the PowerCenter Client and PowerCenter
Services bin directories.
 Use pmrep to perform repository administration tasks such as
listing repository objects, creating and editing groups, restoring
and deleting repositories, and updating session-related
parameters and security information in the PowerCenter
repository.
9
 When you use pmrep, you can enter commands in the following
modes:
Command line mode - You can issue pmrep commands directly
from the system command line. Use command line mode to
script pmrep commands.
Interactive mode - You can issue pmrep commands from an
interactive prompt. pmrep does not exit after it completes a
command.
 You can use environment variables to set user names and
passwords for pmrep. Before you use pmrep, configure these
variables.
 The environment variables apply to pmrep commands that run
on the node.
 All pmrep commands require a connection to the repository
except for the following commands:
- Help
- ListAllPrivileges
 Use the pmrep Connect command to connect to the repository
before using other pmrep commands.
 Note: If the domain is a mixed-version domain, run pmrep from
the installation directory of the Repository Service version.
3. DESIGNER GUIDE
The Designer provides the following tools:
Source Analyzer - Import or create source definitions for flat file,
XML, COBOL, Application, and relational sources.
Target Designer - Import or create target definitions.
Transformation Developer - Create reusable transformations.
Mapplet Designer - Create mapplets.
Mapping Designer - Create mappings.
The Designer consists of the following windows:
Navigator - Connect to multiple repositories and folders. You
can also copy and delete objects and create shortcuts using the
Navigator. Workspace - View or edit sources, targets, mapplets,
transformations, and mappings. You work with a single tool at a
time in the workspace, which has two formats: default and
workbook. You can view multiple versions of an object in the
workspace.
Status bar - Displays the status of the operation you perform.
Output - Provides details when you perform certain tasks, such
as saving work or validating a mapping. Rightclick the Output
window to access window options, such as printing output text,
saving text to file, and changing the font size.
Overview - View workbooks that contain large mappings or a lot
of objects. The Overview window outlines the visible area in the
workspace and highlights selected objects in color. To open the
Overview window, click View > Overview Window.
Instance Data - View transformation data while you run the
Debugger to debug a mapping.
Target Data - View target data while you run the Debugger to
debug a mapping.
You can view a list of open windows, and you can switch from
one window to another in the Designer. To view the list of open
windows, click Window > Windows.
**Configuring Designer options
Creating a Toolbar
 You can create a new toolbar and choose buttons for the new
toolbar.
 You can create toolbars in the Designer, Workflow Manager, and
the Workflow Monitor.
Find Next
 Use the Find Next tool to search for a column or port name in:
- Transformations
- Mapplets
- Source definitions
- Target definitions
 With the Find Next tool, you can search one object at a time.
 You cannot search multiple objects at the same time.
 Use Find Next in each Designer tool.
 Select a single transformation or click in the Output window
before performing the search.
 The Designer saves the last 10 strings searched in the Find Next
box on the Standard toolbar
 You can search for a string in the Save, Generate, or Validate tabs
in the Output window.
 The Find in Workspace tool searches for a field name or
transformation name in all transformations in the workspace.
 The Find in Workspace tool lets you to search all of the
transformations in the workspace for port or transformation
names.
 You can search for column or port names or table names
matching the search string.
 You can specify whether to search across all names in the
workspace, or across the business name of a table, column, or
port.
 You can also choose to search for whole word matches for the
search string or matches which match the case of the search
string
You can complete the following tasks in each Designer tool:
10
- Add a repository.
- Print the workspace.
- View date and time an object was last saved.
- Open and close a folder.
- Create shortcuts (You cannot create shortcuts to objects in
non-shared folders)
- Check out and in repository objects.
- Search for repository objects.
- Enter descriptions for repository objects.
- View older versions of objects in the workspace.
- Revert to a previously saved object version.
- Copy objects.
- Export and import repository objects.
- Work with multiple objects, ports, or columns.
- Rename ports.
- Use shortcut keys.
 You can also view object dependencies in the Designer.
Rules and Guidelines for Viewing and Comparing Versioned
Repository Objects
 You cannot simultaneously view multiple versions of composite
objects, such as mappings and mapplets.
 Older versions of composite objects might not include the child
objects that were used when the composite object was checked
in.
 If you open a composite object that includes a child object version
that is purged from the repository, the preceding version of the
child object appears in the workspace as part of the composite
object.
 For example, you want to view version 5 of a mapping that
originally included version 3 of a source definition, but version 3
of the source definition is purged from the repository. When you
view version 5 of the mapping, version 2 of the source definition
appears as part of the mapping.
 Shortcut objects are not updated when you modify the objects
they reference. When you open a shortcut object, you view the
same version of the object that the shortcut originally
referenced, even if subsequent versions exist.
Viewing an Older Version of a Repository Object
To open an older version of an object in the workspace:
 1. In the workspace or Navigator, select the object and click
Versioning > View History.
 2. Select the version you want to view in the workspace and click
Tools > Open in Workspace.
 Note: An older version of an object is read-only, and the version
number appears as a prefix before the object name. You can
simultaneously view multiple versions of a non-composite
object in the workspace
Reverting to a Previous Object Version
 When you edit an object in the Designer, you can revert to a
previously saved version, undoing changes you entered since
the last save.
 You can revert to the previously saved versions of multiple
objects at the same time.
To revert to a previously saved version of an object:
1. Open the object in the workspace.
2. Select the object and click Edit > Revert to Saved.
3. Click Yes. If you selected more than one object, click Yes to All.
 The Designer removes all changes entered since the last time you
saved the object.
Copying Designer Objects
 You can copy Designer objects within the same folder, to a
different folder, or to a different repository.
 You can copy any of the Designer objects such as sources, targets,
mappings, mapplets, transformations, and dimensions.
 You must open the target folder before you can copy objects to it.
 The Copy Wizard checks for conflicts in the target folder and
provides choices to resolve the conflicts.
 The Copy Wizard displays possible resolutions.
 For a duplicate object, you can rename, reuse, replace, or skip
copying the object.
 To configure display settings and functions of the Copy Wizard,
click Tools > Options in the Designer.
 You can import objects from an XML file through the Import
Wizard in the Designer.
 The Import Wizard provides the same options to resolve conflicts
as the Copy Wizard
Working with Multiple Ports or Columns
 In all Designer tools, you can move or delete multiple ports or
columns at the same time.
 Note: You cannot select multiple ports or columns when editing
COBOL sources in the Source Analyzer
 Note: When you select multiple ports or columns, the Designer
disables add, copy, and paste
Working with Metadata Extensions
 You can extend the metadata stored in the repository by
associating information with individual repository objects.
 For example, you may want to store contact information with the
sources you create.
 You associate information with repository objects using metadata
extensions.
 Repository objects can contain both vendor-defined and user-
defined metadata extensions.
 You can view and change the values of vendor-defined metadata
extensions, but you cannot create, delete, or redefine them.
11
 You can create, edit, delete, and view user-defined metadata
extensions and change their values.
 You can create metadata extensions for the following objects in
the Designer:
- Source definitions
- Target definitions
- Transformations
- Mappings
- Mapplets
 You can create either reusable or non-reusable metadata
extensions.
 You associate reusable metadata extensions with all repository
objects of a certain type, such as all source definitions or all
Expression transformations.
 You associate non-reusable metadata extensions with a single
repository object, such as one target definition or one mapping.
 If you create a reusable metadata extension for a transformation,
the metadata extension applies to all transformations of that
type (for example, all Aggregator transformations or all Router
transformations), and not to all transformations.
 Note: If you make a metadata extension reusable, you cannot
change it back to non-reusable. The Designer makes the
extension reusable as soon as you confirm the action.
Editing Reusable Metadata Extensions
 If the metadata extension you want to edit is reusable and
editable, you can change the value of the metadata extension,
but not any of its properties. However, if the vendor or user who
created the metadata extension did not make it editable, you
cannot edit the metadata extension or its value.
 To restore the default value for a metadata extension, click Revert
in the UnOverride column.
Editing Non-Reusable Metadata Extensions
 If the metadata extension you want to edit is non-reusable, you
can change the value of the metadata extension and its
properties.
 You can also promote the metadata extension to a reusable
metadata extension.
 To restore the default value for a metadata extension, click Revert
in the UnOverride column.
Using Business Names
 You can add business names to sources, targets, and columns.
 Business names are descriptive names that you give to a source,
target, or column.
 They appear in the Navigator in the Business Components source
node and in the source and target nodes.
 Business names can also appear as column names of the source
and target definition in the workspace.
 You can also create source qualifiers to display business names as
column names in the Mapping and Mapplet Designers.
Using Business Documentation
 Business documentation provides details about a repository
object or transformation expression.
 You can create and edit links to business documentation that you
have developed for repository objects through the Designer.
 The documentation must reside on a local machine, network
server, or company intranet or internet web site in a Windows
environment.
 You can develop business documentation in HTML, PDF, or any
text format, for the following repository objects:
- Source and target tables and table instances
- All transformations and transformation instances
- Mapplets
- Mappings
- Business component directories
 To access business documentation, you need to complete the
following tasks:
- Specify the documentation path in the Designer.
- Create a link in the repository object.
- Click the link to view the documentation
Viewing Mapplet and Mapping Reports
 You can view PowerCenter Repository Reports for mappings and
mapplets in the Designer.
 View reports to get more information about the sources, targets,
ports, and transformations in mappings and mapplets.
 When you view a report, the Designer launches the Data Analyzer
application in a browser window and displays the report.
 You can view the following reports:
- Mapplet Composite Report
- Mapping Composite Report
 Before you run reports from the Designer, create a Reporting
Service in the PowerCenter domain that contains the
PowerCenter repository.
 When you create a Reporting Service for a PowerCenter
repository, Data Analyzer imports the PowerCenter Repository
Reports.
Viewing a Mapplet Composite Report
 The Mapplet Composite Report includes information about a
mapplet:
- All objects. Information about all objects in the mapplet.
- Lookup transformations. Lookup transformations in the
mapplet.
- Dependencies. Mappings that use the mapplet.
- Ports. Port details for the input and output ports.
- Sources. Source instances in the mapplet.
- Transformations. Transformations used in the mapplet.
To view a Mapplet Composite Report:
1. In the Designer, open a mapplet.
12
2. Right-click in the workspace and choose View Mapplet Report
Viewing a Mapping Composite Report
 View a mapping report to get more information about the objects
in a PowerCenter mapping.
 The Mapping Composite Report includes information about the
following components in the mapplet:
- Source and target fields. Fields used in mapping sources.
- Port connections. Port-level connections between objects.
- Transformation ports. Transformation ports for each
transformation in the mapping.
- Unconnected ports. Unconnected ports in mapping objects.
- Object-level connections. Connections between all objects in
the mapping.
To view a Mapping Composite Report:
1. In the Designer, open a mapplet.
2. Right-click in the workspace and choose View Mapping
Report.
You can import or create the following types of source
definitions in the Source Analyzer:
- Relational tables, views, and synonyms
- Fixed-width and delimited flat files that do not contain binary
data.
- COBOL files
- XML files
- Web Services Description Language (WSDL)
- Data models using certain data modeling tools through - -
Metadata Exchange for Data Models (an add-on product)
 You can import sources that use multibyte character sets.
 Source code pages must be a superset of the target code pages.
 Source definitions can be single- or multi-group.
 A single-group source has a single group in the source definition.
 Relational sources use a single-group source definition.
 A multi-group source has multiple groups in the source definition.
 Non-relational sources such as XML sources use multi-group
source definitions.
Editing Relational Source Definitions
 You might want to manually edit a source definition to record
properties that you cannot import from the source.
 You can edit a relational source definition to create key columns
and key relationships.
 These relationships can be logical relationships. They do not have
to exist in the database.
Working with COBOL Sources
 To provide support for mainframe source data, you can import a
COBOL file as a source definition in the Designer.
 COBOL files are fixed-width files that may contain text and binary
data.
 PowerCenter supports the following code pages for COBOL files: identification division.
- 7-bit ASCII program-id. mead.
- EBCDIC-US environment division.
- 8-bit ASCII select file-one assign to "fname".
- 8-bit EBCDIC data division.
- ASCII-based MBCS file section.
- EBCDIC-based MBCS fd FILE-ONE.
 You can import shift-sensitive COBOL files that do not contain copy “sample.cpy”.
shift keys. working-storage section.
 Define the shift states for each column in the COBOL source procedure division.
definition. stop run.

 COBOL sources often de-normalize data and compact the
equivalent of separate table records into a single record.
 You use the Normalizer transformation to normalize these
records in the mapping.
 COBOL files often represent the functional equivalent of multiple
source tables within the same set of records.
 When you review the structure of the COBOL file, you can adjust
the description to identify which groups of fields constitute a
single pseudo-table
Working with COBOL Copybooks
 The Designer cannot recognize a COBOL copybook (.cpy file) as a
COBOL file (.cbl file) because it lacks the proper format.
 To import a COBOL copybook in the Designer, you can insert it
into a COBOL file template by using the COBOL statement
“copy.”
 After you insert the copybook file into the COBOL file template,
you can save the file as a .cbl file and import it in the Designer.
 If the .cbl file and the .cpy file are not in the same local directory,
the Designer prompts for the location of the .cpy file.
Components in a COBOL Source File
 When you import a COBOL source, the Designer scans the file for
the following components:
- FD Section
- Fields
- OCCURS
- REDEFINES
FD Section
 The Designer assumes that each FD entry defines the equivalent
of a source table in a relational source and creates a different
COBOL source definition for each such entry.
 For example, if the COBOL file has two FD entries, CUSTOMERS
and ORDERS, the Designer creates one COBOL source definition
containing the fields attributed to CUSTOMERS, and another
with the fields that belong to ORDERS
Fields
 The Designer identifies each field definition, reads the datatype,
and assigns it to the appropriate source definition.

 When the COBOL copybook file contains tabs, the Designer
expands tabs into spaces.
 By default, the Designer expands a tab character into eight
spaces.
 You can change this default setting in powrmart.ini.
 You can find powrmart.ini in the root directory of the
PowerCenter Client installation.
 To change the default setting, add the following text to
powrmart.ini:
[AnalyzerOptions]
TabSize=n
where n is the number of spaces the Designer reads for every
tab character.
 To apply changes, restart the Designer.
 For example, the COBOL copybook file is called sample.cpy. The
COBOL file below shows how to use the copy statement to
insert the sample copybook into a COBOL file template:
13
OCCURS
 COBOL files often contain multiple instances of the same type of
data within the same record.
 For example, a COBOL file may include data about four different
financial quarters, each stored in the same record.
 When the Designer analyzes the file, it creates a different column
for each OCCURS statement in the COBOL file.
 These OCCURS statements define repeated information in the
same record. Use the Normalizer transformation to normalize
this information.
 For each OCCURS statement, the Designer creates the following
items:
- One target table when you drag the COBOL source definition
into the Target Designer.
- A primary-foreign key relationship
- A generated column ID (GCID)
REDEFINES
 COBOL uses REDEFINES statements to build the description of
one record based on the definition of another record.
 When you import the COBOL source, the Designer creates a
single source that includes REDEFINES.
 The REDEFINES statement lets you specify multiple PICTURE
clauses for the sample physical data location.
 Therefore, you need to use Filter transformations to separate the
data into the tables created by REDEFINES.
 For each REDEFINES:
- The Designer creates one target table when you drag the
COBOL source definition into the Target Designer.
- The Designer creates one primary-foreign key relationship.
- The Designer creates a generated key (GK).
- You need a separate Filter transformation in the mapping.
Rules and Guidelines for Delimited File Settings
Delimited files are character-oriented and line sequential. Use the
following rules and guidelines when you configure delimited files:
- The column and row delimiter character, quote character, and
escape character must all be different for a source definition. These
properties must also be contained in the source or target file code
page.
- The escape character and delimiters must be valid in the code page
of the source or target file. Use the following rules and guidelines
when you configure delimited file sources:
- In a quoted string, use the escape character to escape the quote
character. If the escape character does not immediately precede a
quote character, the Integration Service reads the escape character
as an ordinary character.
- Use an escape character to escape the column delimiter. However,
in a quoted string, you do not need to use an escape character to
escape the delimiter since the quotes serve this purpose. If the
escape character does not immediately precede a delimiter
character, the Integration Service reads the escape character as an
ordinary character.
- When two consecutive quote characters appear within a quoted
string, the Integration Service reads them as one quote character.
For example, the Integration Service reads the following quoted
string as I’m going tomorrow: 2353,‘I’’m going tomorrow’MD
- The Integration Service reads a string as a quoted string only if the
quote character you select is the first character of the field.
- If the field length exceeds the column size defined in the Source
Qualifier transformation, the Integration Service truncates the field.
- If the row of data exceeds the larger of the line sequential buffer
length or the total row size defined in the Source Qualifier
transformation, the Integration Service drops the row and writes it
to the session log file. To determine the row size defined in the
Source Qualifier transformation, add the column precision and the
delimiters, and then multiply the total by the maximum bytes per
character.
Working with File Lists
14
 A file list is a file that contains the names and directories of each
source file you want the Integration Service to use.
 When you configure a session to read a file list, the Integration
Service reads rows of data from the different source files in the
file list.
 To configure the mapping to write the source file name to each
target row, add the CurrentlyProcessedFileName port to the
flat file source definition.
 The Integration Service uses this port to return the source file
name.
Creating Target Definitions
 You can create the following types of target definitions in the
Target Designer:
Relational - Create a relational target for a particular database
platform. Create a relational target definition when you want to
use an external loader to the target database.
Flat file - Create fixed-width and delimited flat file target
definitions.
XML file - Create an XML target definition to output data to an
XML file.
 You can create target definitions in the following ways:
- Import the definition for an existing target. Import the target
definition from a relational target or a flat file. The Target
Designer uses a Flat File Wizard to import flat files.
- Create a target definition based on a source definition. Drag a
source definition into the Target Designer to make a target
definition.
- Create a target definition based on a transformation or
mapplet. Drag a transformation into the Target Designer to
make a target definition.
- Manually create a target definition. Create a target definition in
the Target Designer.
- Design several related target definitions. Create several related
target definitions at the same time. You can create the overall
relationship, called a schema, and the target definitions, through
wizards in the Designer.
 The Cubes and Dimensions Wizards follow common principles of
data warehouse design to simplify the process of designing
related targets.
Creating a Target Definition from a Transformation
 To create a relational target definition that closely matches a
transformation in the repository; you can create the target from
the transformation.
 Drag a transformation from the Navigator to the Target Designer,
or create a target from a transformation in the Mapping
Designer workspace.
 Create target definitions from the following types of
transformations:
- Single-group transformations. Create a single target definition
from a transformation with one output group.
 - Multiple-group transformations. Create multiple target
definitions from a transformation with multiple output groups.
- Normalizer transformations. Create a target definition from a
source qualifier or pipeline Normalizer transformation.
- Mapplets. Create one or more target definitions from a
mapplet instance in a mapping.
 When you create a target definition from a transformation, the
target database type is the same as the repository database by
default.
 After you create the target definition in the repository, you can
edit it. For example, you might want to change the target type.
Creating a Target from a Transformation with One Output Group
 When you create a target from a transformation with one output
group, the Designer creates one target.
 All the output ports become input ports in the target. The name
of the target is the same as the transformation name.
Creating a Target from a Transformation with Multiple Output
Groups
 When you create targets from a transformation with more than
one output group, the Designer creates one target for each
output group in the transformation.
 When the transformation is a plug-in or Custom transformation,
the Designer retains the primary key-foreign key relationships
between the groups in the target definitions.
Creating a Target from a Normalizer Transformation
 You can create a target from a source qualifier or pipeline
Normalizer transformation.
 When you create a target from a Normalizer transformation,
the Designer creates one target and includes all the columns
from the Normalizer.
 It does not create separate targets to represent the record
hierarchies or multiple-occurring fields in the Normalizer
transformation
Creating a Target from a Mapplet
 You can create a target from a mapplet that is under a mapping
Transformation Instances node.
 When you drag the mapplet instance to the Target Designer, the
Designer creates a target for each output group in the mapplet.
 Note: You cannot create a target when you drag a transformation
instance from a mapplet to the Target Designer
Data Transformation Source and Target
 Use a Data Transformation source or target to process data in any
file format such as Excel spreadsheets or PDF documents.
 You can also transform data in formats such as HL7, EDI-X12,
EDIFACT, SWIFT, NACHA, FIXBAI2, and DTCC.
15
 A Data Transformation source or target calls a Data
Transformation service from a PowerCenter session.
 Data transformation is the application that transforms the file
formats.
 The Data Transformation service is a Data Transformation service
that is deployed to the Data Transformation repository and is
ready to run
Data Transformation Service Types
 When you create a project in Data Transformation Studio, you
choose a Data Transformation service type to define the project.
Data Transformation has the following types of services that
transform data:
Parser - Converts source documents to XML. The input can have
any format. The output of a parser is always XML.
Serializer - Converts an XML file to another document. The input
is XML. The output can be any format.
Mapper - Converts an XML source document to another XML
structure or schema. The input is XML. The output is XML .
Transformer - Modifies the data in any format. Adds, removes,
converts, or changes text. Use transformers with a parser,
mapper, or serializer. You can also run a transformer as
standalone component.
Streamer - Splits large input documents, such as multiple
gigabyte data streams, into segments. The Streamer splits
documents that have multiple messages or multiple records in
them.
MAPPINGS -
Object Dependency
 Some objects in a mapping are also stored as independent
objects in the repository:
- Sources
- Targets
- Reusable transformations
- Mapplets
 The mapping is dependent on these objects.
 When this metadata changes, the Designer and other
PowerCenter Client applications track the effects of these
changes on mappings.
 In these cases, you may find that mappings become invalid even
though you do not edit the mapping.
 When a mapping becomes invalid, the Integration Service cannot
run it properly, and the Workflow Manager invalidates the
session.
 The only objects in a mapping that are not stored as
independent repository objects are the non-reusable
transformations that you build within the mapping.
 These non-reusable transformations are stored within the
mapping only.
Exporting and Importing a Mapping
 You export a mapping to an XML file and import a mapping from
an XML file through the Designer.
 You might want to use the export and import feature to copy a
mapping to the same repository, a connected repository, or a
repository to which you cannot connect
Invalidating Sessions
 When you edit and save a mapping, some changes cause the
session to be invalid even though the mapping remains valid.
 The Integration Service does not run invalid sessions. If you edit a
mapping, the Designer invalidates sessions when you perform
the following actions:
- Add or remove sources or targets.
- Remove mapplets or transformations.
- Replace a source, target, mapplet, or transformation when
importing or copying objects
- Add or remove Source Qualifiers or COBOL Normalizers, or
change the list of associated sources for these transformations.
- Add or remove a Joiner or Update Strategy transformation.
- Add or remove transformations from a mapplet in the
mapping.
- Change the database type for a source or target.
Deleting a Mapping
 You may delete mappings that you no longer use. When you
delete a mapping, you do not delete any sources, targets,
mapplets, or reusable transformations defined outside the
mapping.
 Note: If you enable version control, deleted mapping remains
checked out until you check it in. To check in a deleted
mapping, click Versioning > Find Checkouts. Select the deleted
mapping and click Tools > Check In.
Viewing Link Paths to a Port
 When displaying both link paths, the Designer traces the flow of
data from one column in the source, in and out of each
transformation, and into a single port in the target.
 For unconnected transformations, the Designer does not display
a link path.
 For connected Lookup transformations, the Designer shows each
output port dependent upon the input ports involved in the
lookup condition.
 For Custom transformations, the Designer shows that an output
port depends on all input ports by default. However, if you
define port relationships in a Custom transformation, the
Designer shows the dependent ports you define.
 Note: You can configure the color the Designer uses to display
connectors in a link path. When configuring the format options,
choose the Link Selection option.
Viewing Source Column Dependencies
16
 To view column dependencies, right-click a target column in a
mapping and choose Show Field Dependencies.
 The Designer displays the Field Dependencies dialog box which
lists all source columns connected to the target column.
 When you define a port expression that performs a calculation
using multiple source columns, and then connect that port to a
target column, the Field Dependencies dialog box lists all source
columns you use in the expression
Options for Linking Ports
 When you link transformations, you can link with one of the
following options:
- One to one. Link one transformation or output group to one
transformation, input group, or target only.
- One to many.
- Link one port to multiple transformations, input groups, or
targets.
- Link multiple ports in one transformation or output group to
multiple transformations, input groups, or targets.
- Many to one. Link many transformations to one
transformation, input group, or target.
Rules and Guidelines for Connecting Mapping Objects
- If the Designer detects an error when you try to link ports
between two mapping objects, it displays a symbol indicating
that you cannot link the ports.
- Follow logic of data flow in the mapping. You can link the
following types of ports:
- The receiving port must be an input or input/output port.
- The originating port must be an output or input/output port.
- You cannot link input ports to input ports or output ports to
output ports.
- You must link at least one port of an input group to an
upstream transformation.
- You must link at least one port of an output group to a
downstream transformation.
- You can link ports from one active transformation or one
output group of an active transformation to an input group of
another transformation.
- You cannot connect an active transformation and a passive
transformation to the same downstream transformation or
transformation input group.
- You cannot connect more than one active transformation to
the same downstream transformation or transformation input
group.
- You can connect any number of passive transformations to
the same downstream transformation, transformation input
group, or target
- You can link ports from two output groups in the same
transformation to one Joiner transformation configured for
sorted data if the data from both output groups is sorted.
 - You can only link ports with compatible datatypes. The
Designer verifies that it can map between the two datatypes
before linking them. The Integration Service cannot transform
data between ports with incompatible datatypes. While the
datatypes do not have to be identical, they do have to be
compatible, such as Char and Varchar.
- You must connect a source definition to a source qualifier only.
You then link the source qualifier to targets or other
transformations.
- You can link columns to a target definition in a mapping, but
you cannot copy columns into a target definition in a mapping.
Use the Target Designer to add columns to a target definition.
- The Designer marks some mappings invalid if the mapping
violates data flow validation.
Propagating Ports and attributes
The Designer does not propagate changes to the following mapping
objects:
- Unconnected transformations
- Reusable transformations
- Mapplets
- Source and target instances
- SDK Source Qualifier
Rules and Guidelines for Propagating Ports and Attributes
- The Designer does not propagate to implicit dependencies within
the same transformation.
- When you propagate a port description, the Designer overwrites
the description for the port in the other transformations in the
mapping
- When you propagate backward along the link path, verify that the
change does not cause the Integration Service to fail the session. For
example, if you propagate changes to a source qualifier, the
Integration Service might generate invalid SQL when it runs the
session. If you change the port name “CUST_ID” to “CUSTOMER_ID,”
the Integration Service might generate SQL to select the wrong
column name if the source table uses “CUST_ID.”
- When you propagate port attributes, verify that the change does
not cause the Designer to invalidate the mapping. For example,
when you change the datatype of a port from integer to string and
propagate the datatype to other transformations, the Designer
invalidates the mapping if a calculation uses one of the changed
ports. Validate the mapping after you propagate ports. If the
Designer invalidates the mapping, click Edit > Revert to Saved to
revert to the last saved version of the mapping.
- When you propagate multiple ports, and an expression or condition
depends on more than one propagated port, the Designer does not
propagate attributes to implicit dependencies if the attributes do not
match. For example, you have the following expression in an
Expression transformation:
Item_desc_out = Substr(ITEM_NAME, 0, 6) || Substr(ITEM_DESC, 0,
6)
17
The precision of Item_desc_out is 12, ITEM_NAME is 10, and
ITEM_DESC is 10. You change the precision of ITEM_DESC to 15. You
select parse expressions to infer dependencies and propagate the
port attributes of ITEM_NAME and ITEM_DESC. The Designer does
not update the precision of the Item_desc_out port in the
Expression transformation since the ITEM_NAME and ITEM_DESC
ports have different precisions.
Creating Target Files by Transaction
 You can generate a separate output file each time the Integration
Service starts a new transaction.
 You can dynamically name each target flat file.
 To generate a separate output file for each transaction, add a
FileName port to the flat file target definition.
 When you connect the FileName port in the mapping, the
Integration Service creates a separate target file at each commit.
 The Integration Service names the output file based on the
FileName port value from the first row in each transaction.
 By default, the Integration Service writes output files to
$PMTargetFileDir
Rules and Guidelines for Creating Target Files by Transaction
- You can use a FileName column with flat file targets.
- You can add one FileName column to the flat file target
definition.
- You can use a FileName column with data from real-time
sources.
- A session fails if you use a FileName column with merge files,
file lists, or FTP targets.
- If you pass the same file name to targets in multiple
partitions, you might get unexpected results.
- When a transformation drops incoming transaction boundaries
and does not generate commits, the Integration Service writes
all rows into the same output file. The output file name is the
initial value of the FileName port.
Rejecting Truncated and Overflow Data
 When a conversion causes an overflow, the Integration Service,
by default, skips the row.
 The Integration Service does not write the data to the reject file.
 For strings, the Integration Service truncates the string and
passes it to the next transformation.
 The Designer provides an option to let you include all truncated
and overflow data between the last transformation and target
in the session reject file. If you select Reject Truncated/Overflow
Rows, the Integration Service sends all truncated rows and any
overflow rows to the session reject file or to the row error logs,
depending on how you configure the session.
Rules and Guidelines for Configuring the Target Update Override
- If you use target update override, you must manually put all
database reserved words in quotes.
 - You cannot override the default UPDATE statement if the target
column name contains any of the following characters:
' , ( ) < > = + - * / \ t \ n \ 0 <space>
- You can use parameters and variables in the target update
query. Use any parameter or variable type that you can define in
the parameter file.
 You can enter a parameter or variable within the UPDATE
statement, or you can use a parameter or variable as the update
query.
 For example, you can enter a session parameter,
$ParamMyOverride, as the update query, and set
$ParamMyOverride to the UPDATE statement in a parameter
file.
- When you save a mapping, the Designer verifies that you
have referenced valid port names. It does not validate the SQL.
- If you update an individual row in the target table more than
once, the database only has data from the last update.
 If the mapping does not define an order for the result data,
different runs of the mapping on identical input data may result
in different data in the target table.
- A WHERE clause that does not contain any column references
updates all rows in the target table, or no rows in the target
table, depending on the WHERE clause and the data from the
mapping.
 For example, the following query sets the EMP_NAME to “MIKE
SMITH” for all rows in the target table if any row of the
transformation has EMP_ID > 100:
 UPDATE T_SALES set EMP_NAME = 'MIKE SMITH' WHERE
:TU.EMP_ID > 100
- If the WHERE clause contains no port references, the mapping
updates the same set of rows for each row of the mapping.
 For example, the following query updates all employees with
EMP_ID > 100 to have the EMP_NAME from the last row in the
mapping:
 UPDATE T_SALES set EMP_NAME = :TU.EMP_NAME WHERE
EMP_ID > 100
- If the mapping includes an Update Strategy or Custom
transformation, the Target Update statement only affects
records marked for update.
- If you use the Target Update option, configure the session to
mark all source records as update.
Rules and Guidelines for Adding Pre- and Post-Session SQL
Commands
- Use any command that is valid for the database type. However,
the Integration Service does not allow nested comments, even
though the database might.
- You can use parameters and variables in the target pre- and
post-session SQL commands.
For example, you can enter a parameter or variable within the
command. Or, you can use a session parameter,
18
$ParamMyCommand, as the SQL command, and set
$ParamMyCommand to the SQL statement in a parameter file.
- Use a semicolon (;) to separate multiple statements. The
Integration Service issues a commit after each statement.
- The Integration Service ignores semicolons within /* ...*/.
- If you need to use a semicolon outside of comments, you can
escape it with a backslash (\).
- The Designer does not validate the SQL.
 Note: You can also enter pre- and post-session SQL commands on
the Properties tab of the Source Qualifier transformation.
Validating the mapping
 When you develop a mapping, you must configure it so the
Integration Service can read and process the entire mapping.
 The Designer marks a mapping invalid when it detects errors that
will prevent the Integration Service from running sessions
associated with the mapping.
 The Designer marks a mapping valid for the following reasons:
Connection validation - Required ports are connected and that
all connections are valid.
Expression validation - All expressions are valid.
Objects validation - The independent object definition matches
the instance in the mapping.
Data flow validation - The data must be able to flow from the
sources to the targets without hanging at blocking
transformations.
Connection Validation
 The Designer performs connection validation each time you
connect ports in a mapping and each time you validate or save a
mapping.
 When you connect ports, the Designer verifies that you make
valid connections.
 When you save or validate a mapping, the Designer verifies that
the connections are valid and that all required ports are
connected.
 When you save or validate a mapping, the Designer makes the
following connection validations:
- At least one source and one target must be connected.
- Source qualifiers must be mapped to a target.
- Mapplets must be connected. At least one mapplet input port
and output port is connected to the mapping. If the mapplet
includes a source qualifier that uses an SQL override, the
Designer prompts you to connect all mapplet output ports to the
mapping.
- Datatypes between ports must be compatible. If you change a
port datatype to one that is incompatible with the port it is
connected to, the Designer generates an error and invalidates the
mapping. For example, you have two Date/Time ports connected,
and you change one port to a Decimal. The Designer invalidates
the mapping. You can however, change the datatype if it remains
compatible with the connected ports, such as Char and Varchar

Data Flow Validation
 When you validate or save a mapping, the Designer verifies that
the data can flow from all sources in a target load order group to
the targets without the Integration Service blocking all sources.
 Mappings that include blocking transformations might hang at
runtime with any of the following mapping configurations:
- You connect one source pipeline to multiple input groups of
the blocking transformation
- You connect the sources and transformations in a target load
order group in such a way that multiple blocking
transformations could possibly block all source pipelines.
 Depending on the source data used in a session, a blocking
transformation might block data from one source while it waits
for a row from a different source.
 When you save or validate a mapping with one of these
configurations, the Designer marks the mapping invalid.
 When the Designer marks a mapping invalid because the
mapping violates data flow validation, you must configure the
mapping differently, or use a non-blocking transformation where
possible.
 The following figure shows mappings that are invalid because
one source provides data for multiple input groups of a blocking
transformation:
To make the mappings valid, use a non-blocking transformation for
MGT1 or create two instances of the same source and connect them
to the blocking transformation.
The following figure shows two similar mappings, one which is valid,
and one which is invalid:
19
 Mapping A contains two multigroup transformations that block
data, MGT1 and MGT2. If you could run this session, MGT1
might block data from S1 while waiting for a row from S2.
 And MGT2 might block data from S2 while waiting for a row from
S1.
 The blocking transformations would block both source pipelines
and the session would hang. Therefore, the Designer marks the
mapping invalid.
 Mapping B contains one multigroup transformation that blocks
data, MGT1.
 Blocking transformations can never block all input groups, so
MGT1 might block either S1 or S2, but never both.
 MGT2 is not a blocking transformation, so it will never block data.
 Therefore, this session will not hang at runtime due to blocking.
 The Designer marks the mapping valid
Steps to Validate a Mapping
 You can validate a mapping while you are working on it through
the Designer. Also, when you click Repository > Save, the
Designer validates all mappings since the last time you saved.
 When you validate or save a mapping the results of the validation
appear in the Output window.
 The Repository Manager also displays whether a mapping is valid.
 To validate a mapping, check out and open the mapping, and
click Mappings > Validate.
 If the Output window is not open, click View > Output Window.
Review any errors to determine how to fix the mapping.
Validating Multiple Mappings
 You can validate multiple mappings without fetching them into
the workspace.
 To validate multiple mappings you must select and validate the
mappings from either a query results view or a view object
dependencies list.
 Note: If you use the Repository Manager, you can select and
validate multiple mappings from the Navigator.
 You can save and optionally check in mappings that change from
invalid to valid status as a result of the validation.
 To validate multiple mappings:
1. Select mappings from either a query or a view dependencies list.
2. Right-click one of the selected mappings and choose Validate. The
Validate Objects dialog box displays.
3. Choose whether to save objects and check in objects that you
validate
MAPPLETS
Mapplets Overview
 When you use a mapplet in a mapping, you use an instance of
the mapplet. Like a reusable transformation, any change made
to the mapplet is inherited by all instances of the mapplet.

 Mapplets help simplify mappings in the following ways:
Include source definitions - Use multiple source definitions and
source qualifiers to provide source data for a mapping.
Accept data from sources in a mapping - If you want the mapplet to
receive data from the mapping, use an Input transformation to
receive source data.
Include multiple transformations - A mapplet can contain as many
transformations as you need.
Pass data to multiple transformations - You can create a mapplet to
feed data to multiple transformations. Each Output transformation
in a mapplet represents one output group in a mapplet.
Contain unused ports - You do not have to connect all mapplet input
and output ports in a mapping.
Understanding Mapplet Input and Output
 To use a mapplet in a mapping, you must configure it for input
and output.
 In addition to transformation logic that you configure, a mapplet
has the following components:
Mapplet input: You can pass data into a mapplet using source
definitions or Input transformations or both. When you use an Input
transformation, you connect it to the source pipeline in the mapping.
Mapplet output: Each mapplet must contain one or more Output
transformations to pass data from the mapplet into the mapping.
Mapplet ports: Mapplet ports display only in the Mapping Designer.
Mapplet ports consist of input ports from Input transformations and
output ports from Output transformations. If a mapplet uses source
definitions rather than Input transformations for input, it does not
contain any input ports in the mapping.
Mapplet Input
 Mapplet input can originate from a source definition and/or from
an Input transformation in the mapplet.
 You can create multiple pipelines in a mapplet.
 Use multiple source definitions and source qualifiers or Input
transformations.
 You can also use a combination of source definitions and Input
transformations.
Using Source Definitions for Mapplet Input
 Use one or more source definitions in a mapplet to provide
source data.
 When you use the mapplet in a mapping, it is the first object in
the mapping pipeline and contains no input ports.
Using Input Transformations for Mapplet Input
 Use an Input transformation in a mapplet when you want the
mapplet to receive input from a source in a mapping.
 When you use the mapplet in a mapping, the Input
transformation provides input ports so you can pass data
through the mapplet.
20
 Each port in the Input transformation connected to another
transformation in the mapplet becomes a mapplet input port.
 Input transformations can receive data from a single active
source.
 Unconnected ports do not display in the Mapping Designer.
 You can connect an Input transformation to multiple
transformations in a mapplet.
 However, you cannot connect a single port in the Input
transformation to multiple transformations in the mapplet.
Mapplet Output
 Use an Output transformation in a mapplet to pass data through
the mapplet into a mapping.
 A mapplet must contain at least one Output transformation with
at least one connected port in the mapplet.
 Each connected port in an Output transformation displays as a
mapplet output port in a mapping.
 Each Output transformation in a mapplet displays as an output
group in a mapping.
 An output group can pass data to multiple pipelines in a
mapping.
Viewing Mapplet Input and Output
 Mapplets and mapplet ports display differently in the Mapplet
Designer and the Mapping Designer
 The following figure shows a mapplet with both an Input
transformation and an Output transformation:
 When you use the mapplet in a mapping, the mapplet object
displays only the ports from the Input and Output
transformations. These are referred to as the mapplet input and
mapplet output ports.
 The following figure shows the same mapplet in the Mapping
Designer:
 You can expand the mapplet in the Mapping Designer by
selecting it and clicking Mappings > Expand.
 This expands the mapplet within the mapping for view.
Transformation icons within an expanded mapplet display as
shaded.
 You can open or iconize all the transformations in the mapplet
and mapping.
 You cannot edit any of the properties, navigate to other folders,
or save the repository while the mapplet is expanded.
 The following figure shows an expanded mapplet in the Mapping
Designer:
 In an expanded mapping, you do not see the Input and Output
transformations.
Creating a Mapplet
 A mapplet can be active or passive depending on the
transformations in the mapplet.
 Active mapplets contain one or more active transformations.
 Passive mapplets contain only passive transformations.
 When you use a mapplet in a mapping, all transformation rules
apply to the mapplet depending on the mapplet type.
 For example, as with an active transformation, you cannot
concatenate data from an active mapplet with a different
pipeline.
 Use the following rules and guidelines when you add
transformations to a mapplet:
- If you use a Sequence Generator transformation, you must use
a reusable Sequence Generator transformation.
- If you use a Stored Procedure transformation, you must
configure the Stored Procedure Type to be Normal.
- You cannot include PowerMart 3.5-style LOOKUP functions in a
mapplet.
- You cannot include the following objects in a mapplet:
- Normalizer transformations
- COBOL sources
- XML Source Qualifier transformations
- XML sources
- Target definitions
- Other mapplets
 Although reusable transformations and shortcuts in a mapplet
can be used, to protect the validity of the mapplet, use a copy of
a transformation instead.
21
 Reusable transformations and shortcuts inherit changes to their
original transformations. This might invalidate the mapplet and
the mappings that use the mapplet
Validating Mapplets
 The Designer validates a mapplet when you save it. You can also
validate a mapplet using the Mapplets > Validate menu
command. When you validate a mapplet, the Designer writes all
relevant messages about the mapplet in the Output window.
 The Designer validates the mapplet pipeline in the same way it
validates a mapping.
 The Designer also performs the following checks specific to
mapplets:
- The mapplet can contain Input transformations and source
definitions with at least one port connected to a transformation
in the mapplet.
- The mapplet contains at least one Output transformation with
at least one port connected to a transformation in the mapplet.
Editing Mapplets
 You can edit a mapplet in the Mapplet Designer. The Designer
validates the changes when you save the mapplet.
 When you save changes to a mapplet, all instances of the
mapplet and all shortcuts to the mapplet inherit the changes.
 These changes might invalidate mappings that use the mapplet.
 To see what mappings or shortcuts may be affected by changes
you make to a mapplet, select the mapplet in the Navigator,
right-click, and select Dependencies. Or, click Mapplets >
Dependencies from the menu.
 You can make the following changes to a mapplet without
affecting the validity of existing mappings and sessions:
- Add input or output ports.
- Change port names or comments.
- Change Input or Output transformation names or comments.
- Change transformation names, comments, or properties.
- Change port default values for transformations in the mapplet.
- Add or remove transformations in the mapplet, providing you
do not change the mapplet type from active to passive or from
passive to active.
 Use the following rules and guidelines when you edit a mapplet
that is used by mappings:
 Do not delete a port from the mapplet: The Designer deletes
mapplet ports in the mapping when you delete links to an Input
or Output transformation or when you delete ports connected
to an Input or Output transformation.
 Do not change the datatype, precision, or scale of a mapplet
port: The data type, precision, and scale of a mapplet port is
defined by the transformation port to which it is connected in
the mapplet. Therefore, if you edit a mapplet to change the
 datatype, precision, or scale of a port connected to a port in an
Input or Output transformation, you change the mapplet port.
 Do not change the mapplet type: If you remove all active
transformations from an active mapplet, the mapplet becomes
passive. If you add an active transformation to a passive
mapplet, the mapplet becomes active.
Mapplets and Mappings
 The following mappings tasks can also be performed on
mapplets:
Set tracing level: You can set the tracing level on individual
transformations within a mapplet in the same manner as in a
mapping.
Copy mapplet: You can copy a mapplet from one folder to
another as you would any other repository object. After you copy
the mapplet, it appears in the Mapplets node of the new folder. If
you make changes to a mapplet, but you do not want to
overwrite the original mapplet, you can make a copy of the
mapplet by clicking Mapplets > Copy As.
Export and import mapplets: You can export a mapplet to an
XML file or import a mapplet from an XML file through the
Designer. You might want to use the export and import feature
to copy a mapplet to another repository.
Delete mapplets: When you delete a mapplet, you delete all
instances of the mapplet. This invalidates each mapping
containing an instance of the mapplet or a shortcut to the
mapplet.
Compare mapplets: You can compare two mapplets to find
differences between them. For example, if you have mapplets
with the same name in different folders, you can compare them
to see if they differ.
Compare instances within a mapplet: You can compare instances
in a mapplet to see if they contain similar attributes. For example,
you can compare a source instance with another source instance,
or a transformation with another transformation. You compare
instances within a mapplet in the same way you compare
instances within a mapping.
Create shortcuts to mapplets: You can create a shortcut to a
mapplet if the mapplet is in a shared folder. When you use a
shortcut to a mapplet in a mapping, the shortcut inherits any
changes you might make to the mapplet. However, these changes
might not appear until the Integration Service runs the workflow
using the shortcut. Therefore, only use a shortcut to a mapplet
when you do not expect to edit the mapplet.
Add a description: You can add a description to the mapplet in
the Mapplet Designer in the same manner as in a mapping. You
can also add a description to the mapplet instance in a mapping.
When you add a description, you can also create links to
documentation files. The links must be a valid URL or file path to
reference the business documentation.
22
View links to a port: You can view links to a port in a mapplet in
the same way you would view links to a port in a mapping. You
can view the forward path, the backward path, or both paths.
Propagate port attributes: You can propagate port attributes in a
mapplet in the same way you would propagate port attributes in
a mapping. You can propagate attributes forward, backward, or in
both directions.
Using Mapplets in Mappings
 In a mapping, a mapplet has input and output ports that you can
connect to other transformations in the mapping.
 You do not have to connect all mapplet ports in a mapping.
 However, if the mapplet contains an SQL override, you must
connect all mapplet output ports in the mapping.
 Like a reusable transformation, when you drag a mapplet into a
mapping, the Designer creates an instance of the mapplet.
 You can enter comments for the instance of the mapplet in the
mapping.
 You cannot otherwise edit the mapplet in the Mapping
Designer.
 If you edit the mapplet in the Mapplet Designer, each instance of
the mapplet inherits the changes.
 The PowerCenter Repository Reports has a Mapplets list report
that you use to view all mappings using a particular mapplet.
Creating and Configuring Mapplet Ports
 After creating transformation logic for a mapplet, you can create
mapplet ports. Use an Input transformation to define mapplet
input ports if the mapplet contains no source definitions. Use an
Output transformation to create a group of output ports. Only
connected ports in an Input or Output transformation become
mapplet input or output ports in a mapping. Unconnected ports
do not display when you use the mapplet in a mapping.
 You can create a mapplet port in the following ways:
Manually create ports in the Input/output transformation: You
can create port names in Input and Output transformations. You
can also enter a description for each port name. The port has no
defined data type, precision, or scale until you connect it to a
transformation in the mapplet.
Drag a port from another transformation: You can create an
input or output port by dragging a port from another
transformation into the Input or Output transformation. The new
port inherits the port name, description, data type, and scale of
the original port. You can edit the new port name and description
in the transformation. If you change a port connection, the
Designer updates the Input or Output transformation port to
match the attributes of the new connection.
 You can view the data type, precision, and scale of available
mapplet ports when you use the mapplet in a mapping.

Connecting to Mapplet Output Groups
 Each Output transformation displays as an output group when
you use a mapplet in a mapping.
 Connect the mapplet output ports to the mapping pipeline.
Use Autolink to connect the ports.
 Use the following rules and guidelines when you connect
mapplet output ports in the mapping:
- When a mapplet contains a source qualifier that has an
override for the default SQL query, you must connect all of the
source qualifier output ports to the next transformation within
the mapplet.
- If the mapplet contains more than one source qualifier, use a
Joiner transformation to join the output into one pipeline.
- If the mapplet contains only one source qualifier, you must
connect the mapplet output ports to separate pipelines. You
cannot use a Joiner transformation to join the output.
 If you need to join the pipelines, you can create two mappings to
perform this task:
- Use the mapplet in the first mapping and write data in each
pipeline to separate targets.
- Use the targets as sources in the second mapping to join data,
and then perform any additional transformation necessary.
Setting the Target Load Plan
 When you use a mapplet in a mapping, the Mapping Designer
lets you set the target load plan for sources within the mapplet.
Pipeline Partitioning
 If you have the partitioning option, you can increase the number
of partitions in a pipeline to improve session performance.
Increasing the number of partitions allows the Integration
Service to create multiple connections to sources and process
partitions of source data concurrently.
 When you create a session, the Workflow Manager validates
each pipeline in the mapping for partitioning. You can specify
multiple partitions in a pipeline if the Integration Service can
maintain data consistency when it processes the partitioned
data.
 Some partitioning restrictions apply to mapplets.
Rules and Guidelines for Mapplets
 The following list summarizes the rules and guidelines that
appear throughout this chapter:
- You can connect an Input transformation to multiple
transformations in a mapplet. However, you cannot connect a
single port in the Input transformation to multiple
transformations in the mapplet.
- An Input transformation must receive data from a single active
source.
23
- A mapplet must contain at least one Input transformation or
source definition with at least one port connected to a
transformation in the mapplet.
- A mapplet must contain at least one Output transformation
with at least one port connected to another transformation in the
mapping.
- When a mapplet contains a source qualifier that has an override
for the default SQL query, you must connect all of the source
qualifier output ports to the next transformation within the
mapplet.
- If the mapplet contains more than one source qualifier, use a
Joiner transformation to join the output into one pipeline. If the
mapplet contains only one source qualifier, you must connect the
mapplet output ports to separate pipelines. You cannot use a
Joiner transformation to join the output.
- When you edit a mapplet, you might invalidate mappings if you
change the mapplet type from passive to active.
- If you delete ports in the mapplet when the mapplet is used in a
mapping, you can invalidate the mapping.
- Do not change the datatype, precision, or scale of a mapplet
port when the mapplet is used by a mapping.
- If you use a Sequence Generator transformation, you must use a
reusable Sequence Generator transformation.
- If you use a Stored Procedure transformation, you must
configure the Stored Procedure Type to be Normal.
- You cannot include PowerMart 3.5-style LOOKUP functions in a
mapplet.
- You cannot include the following objects in a mapplet:
- Normalizer transformations
- Cobol sources
- XML Source Qualifier transformations
- XML sources
- Target definitions
- Pre- and post- session stored procedures
- Other mapplets
Mapping Parameters and Variables
Mapping Parameters
 A mapping parameter represents a constant value that you can
define before running a session.
 A mapping parameter retains the same value throughout the
entire session.
 When you use a mapping parameter, you declare and use the
parameter in a mapping or mapplet.
 Then define the value of the parameter in a parameter file.
 The Integration Service evaluates all references to the parameter
to that value.
 When you want to use the same value for a mapping parameter
each time you run the session, use the same parameter file for
each session run.
 When you want to change the value of a mapping parameter  The following table lists the default values the Integration Service
between sessions you can perform one of the following tasks: uses for different types of data:
- Update the parameter file between sessions.
- Create a different parameter file and configure the session to
use the new file.
- Remove the parameter file from the session properties. The
Integration Service uses the parameter value in the pre-session
variable assignment. If there is no pre-session variable
Using String Parameters and Variables
assignment, the Integration Service uses the configured initial
 For example, you might use a parameter named $$State in the
value of the parameter in the mapping.
filter for a Source Qualifier transformation to extract rows for a
Mapping Variables
particular state:
 Unlike a mapping parameter, a mapping variable represents a
STATE = ‘$$State’
value that can change through the session.
 During the session, the Integration Service replaces the
 The Integration Service saves the value of a mapping variable to
parameter with a string. If $$State is defined as MD in the
the repository at the end of each successful session run and
parameter file, the Integration Service replaces the parameter
uses that value the next time you run the session.
as follows:
 When you use a mapping variable, you declare the variable in the
STATE = ‘MD’
mapping or mapplet, and then use a variable function in the
mapping to change the value of the variable.
 You can perform a similar filter in the Filter transformation using
 At the beginning of a session, the Integration Service evaluates
the PowerCenter transformation language as follows:
references to a variable to determine the start value.
STATE = $$State
 At the end of a successful session, the Integration Service saves
 If you enclose the parameter in single quotes in the Filter
the final value of the variable to the repository.
transformation, the Integration Service reads it as the string
 The next time you run the session, the Integration Service
literal “$$State” instead of replacing the parameter with “MD.”
evaluates references to the variable to the saved value.
 To override the saved value, define the start value of the variable
Variable Datatype and Aggregation Type
in a parameter file or assign a value in the pre-session variable
 The Integration Service uses the aggregate type of a mapping
assignment in the session properties.
variable to determine the final current value of the mapping
variable.
Using Mapping Parameters and Variables
 When you have a pipeline with multiple partitions, the
 When the Designer validates a mapping variable in a reusable
Integration Service combines the variable value from each
transformation, it treats the variable as an Integer datatype.
partition and saves the final current variable value into the
 You cannot use mapping parameters and variables
repository.
interchangeably between a mapplet and a mapping.
 Mapping parameters and variables declared for a mapping
 You can create a variable with the following aggregation types:
cannot be used within a mapplet.
- Count
 Similarly, you cannot use a mapping parameter or variable
- Max
declared for a mapplet in a mapping.
- Min
Initial and Default Values
 You can configure a mapping variable for a Count Aggregation
 When you declare a mapping parameter or variable in a mapping
type when it is an Integer or Small Integer.
or a mapplet, you can enter an initial value.
 You can configure mapping variables of any datatype for Max or
 The Integration Service uses the configured initial value for a
Min aggregation types.
mapping parameter when the parameter is not defined in the
 To keep the variable value consistent throughout the session run,
parameter file.
the Designer limits the variable functions you use with a variable
 Similarly, the Integration Service uses the configured initial value
based on aggregation type.
for a mapping variable when the variable value is not defined in
 For example, use the SetMaxVariable function for a variable with
the parameter file, and there is no saved variable value in the
a Max aggregation type, but not with a variable with a Min
repository.
aggregation type.
 When the Integration Service needs an initial value, and you did
 The following table describes the available variable functions and
not declare an initial value for the parameter or variable, the
the aggregation types and datatypes you use with each
Integration Service uses a default value based on the datatype
function:
of the parameter or variable.
24

Variable Functions
 Variable functions determine how the Integration Service
calculates the current value of a mapping variable in a pipeline.
 Use variable functions in an expression to set the value of a
mapping variable for the next session run.
 The transformation language provides the following variable
functions to use in a mapping:
SetMaxVariable: Sets the variable to the maximum value of a group
of values. It ignores rows marked for update, delete, or reject. To use
the SetMaxVariable with a mapping variable, the aggregation type of
the mapping variable must be set to Max.
SetMinVariable: Sets the variable to the minimum value of a group
of values. It ignores rows marked for update, delete, or reject. To use
the SetMinVariable with a mapping variable, the aggregation type of
the mapping variable must be set to Min.
SetCountVariable: Increments the variable value by one. In other
words, it adds one to the variable value when a row is marked for
insertion, and subtracts one when the row is marked for deletion. It
ignores rows marked for update or reject. To use the
SetCountVariable with a mapping variable, the aggregation type of
the mapping variable must be set to Count.
SetVariable: Sets the variable to the configured value. At the end of
a session, it compares the final current value of the variable to the
start value of the variable. Based on the aggregate type of the
variable, it saves a final value to the repository. To use the
SetVariable function with a mapping variable, the aggregation type
of the mapping variable must be set to Max or Min. The SetVariable
function ignores rows marked for delete or reject.
 Use variable functions only once for each mapping variable in a
pipeline.
 The Integration Service processes variable functions as it
encounters them in the mapping.
 The order in which the Integration Service encounters variable
functions in the mapping may not be the same for every
session run.
 This may cause inconsistent results when you use the same
variable function multiple times in a mapping.
 The Integration Service does not save the final current value of
a mapping variable to the repository when any of the
following conditions are true:
- The session fails to complete.
- The session is configured for a test load.
- The session is a debug session.
25
- The session runs in debug mode and is configured to discard
session output.
 You cannot use variable functions in the Rank or Aggregator
transformation. Use a different transformation for variable
functions.
Working with User-Defined Functions
 After you create the function, you can create the following
expression in an Expression transformation to remove leading
and trailing spaces from last names:
:UDF.REMOVESPACES(LAST_NAME)
Configuring a User-Defined Function Name
 A valid function name meets the following requirements:
- It begins with a letter.
- It can contain letters, numbers, and underscores. It cannot
contain any other character.
- It cannot contain spaces.
- It must be 80 characters or fewer.
Configuring the Function Type
 You can place user-defined functions in other user-defined
functions. You can also configure a user-defined function to be
callable from expressions. Callable means that you can place
user-defined functions in an expression.
 Select one of the following options when you configure a user-
defined function:
Public: Callable from any user-defined function, transformation
expression, link condition expression, or task expression.
Private: Callable from another user-defined function. Create a
private function when you want the function to be part of a more
complex function. The simple function may not be usable
independently of the complex function.
 After you create a public user-defined function, you cannot
change the function type to private.
 Although you can place a user-defined function in another user-
defined function, a function cannot refer to itself.
 For example, the user-defined function RemoveSpaces includes a
user-defined function TrimLeadingandTrailingSpaces.
 TrimLeadingandTrailingSpaces cannot include RemoveSpaces.
Otherwise, RemoveSpaces is invalid.
Configuring Public Functions that Contain Private Functions
 When you include ports as arguments in a private user-defined
function, you must also include the ports as arguments in any
public function that contains the private function. Use the same
datatype and precision for the arguments in the private and
public function.
 For example, you define a function to modify order IDs to include
‘INFA’ and the customer ID. You first create the following private
function called ConcatCust that concatenates ‘INFA’ with the
port CUST_ID: CONCAT (‘INFA’, CUST_ID)
 After you create the private function, you create a public function
called ConcatOrder that contains ConcatCust:
CONCAT (:UDF.CONCATCUST( CUST_ID), ORDER_ID)
 When you add ConcatCust to ConcatOrder, you add the argument
CUST_ID with the same datatype and precision to the public
function.
 Note: If you enter a user-defined function when you manually
define the public function syntax, you must prefix the user-
defined function with :UDF
 The following table describes the user-defined function
management tasks and lists where you can perform each task:
run the Debugger, the Integration Service runs a debug instance
of the reusable session and creates and runs a debug workflow
for the session.
Create a debug session instance: You can configure source,
target, and session configuration properties through the
Debugger Wizard. When you run the Debugger, the Integration
Service runs a debug instance of the debug workflow and creates
and runs a debug workflow for the session
The following figure shows the windows in the Mapping Designer
that appears when you run the Debugger:

Validating User-Defined Functions

You can validate a user-defined function from the following areas:
- Expression Editor when you create or edit a UDF
- Tools menu
- Query Results window
- View History window
 When you validate a user-defined function, the PowerCenter
Client does not validate other user-defined functions and
expressions that use the function.
 If a user-defined function is invalid, any user-defined function
 Note: You cannot create breakpoints for mapplet Input and
and expression that uses the function is also invalid.
Output transformations.
 Similarly, mappings and workflows that use the user-defined
function are invalid.
Creating Error Breakpoints
Using Debugger
 When you create an error breakpoint, the Debugger pauses when
the Integration Service encounters error conditions such as a
Debugger Session Types
transformation error or calls to the ERROR function.
 You can select three different debugger session types when you
 You also set the number of errors to skip for each breakpoint
configure the Debugger. The Debugger runs a workflow for each
before the Debugger pauses:
session type. You can choose from the following Debugger
- If you want the Debugger to pause at every error, set the
session types when you configure the Debugger:
number of errors to zero.
Use an existing non-reusable session: The Debugger uses
- If you want the Debugger to pause after a specified number of
existing source, target, and session configuration properties.
errors, set the number of errors greater than zero. For example, if
When you run the Debugger, the Integration Service runs the
you set the number of errors to five, the Debugger skips five
non-reusable session and the existing workflow. The Debugger
errors and pauses at every sixth error.
does not suspend on error.
Using ISNULL and ISDEFAULT
Use an existing reusable session: The Debugger uses existing
 You can create ISNULL and ISDEFAULT conditions in
source, target, and session configuration properties. When you
transformation and global data breakpoints.
26
 When you use the ISNULL or ISDEFAULT operator, you cannot use
the type or value in the condition.
 When you create an ISNULL condition, the Debugger pauses
when the Integration Service encounters a null input value,
and the port contains the system default value.
 When you create an ISDEFAULT condition, the Debugger pauses
in the following circumstances:
- The Integration Service encounters an output transformation
error, and the port contains a user-defined default value of a
constant value or constant expression.
- The Integration Service encounters a null input value, and the
port contains a user-defined default value of a constant value or
constant expression.
Running an Existing Session in Debug Mode
 If you choose to run an existing session in debug mode, the
Debugger Wizard displays a list of all sessions in the current
folder that use the mapping. Select the session you want to use.
 You cannot run the Debugger against a session configured with
multiple partitions or a session configured to run on a grid. You
must either change the properties of the session or choose to
create a debug session for the mapping.
 When you do not select a mapplet to debug, the Designer does
not expand it in the workspace.
 You cannot complete the following tasks for transformations in
the mapplet:
- Monitor or modify transformation data.
- Evaluate expressions.
- Edit breakpoints.
- Step to a transformation instance.
 The Debugger can be in one of the following states:
Initializing: The Designer connects to the Integration Service.
Running: The Integration Service processes the data.
Paused: the Integration Service encounters a break and pauses
the Debugger.
 Note: To enable multiple users to debug the same mapping at the
same time, each user must configure different port numbers in
the Tools > Options > Debug tab.
 The Debugger does not use the high availability functionality.
 The following table describes the different tasks you can perform
in each of the Debugger states:

Set Target Options

 On the last page of the Debugger Wizard, you can select the
following target options:
 Discard target data: You can choose to load or discard target data
when you run the Debugger. If you discard target data, the
Integration Service does not connect to the target.
 Display target data: You can select the target instances you want
to display in the Target window while you run a debug session.
 When you click Finish, if the mapping includes mapplets, the
Debugger displays the mapplet instance dialog box.
 Select the mapplets from this dialog box that you want to debug.
To clear a selected mapplet, press the Ctrl key and select the
mapplet.
 When you select a mapplet to debug, the Designer expands it to
display the individual transformations when the Debugger runs.

27

Working with Persisted Values
 When you run the Debugger against mappings with sequence
generators and mapping variables, the Integration Service might
save or discard persisted values:
Discard persisted values: The Integration Service does not save
final values of generated sequence numbers or mapping variables
to the repository when you run a debug session or you run a
session in debug mode and discard target data.
Save persisted values: The Integration Service saves final values
of generated sequence numbers and mapping variables to the
repository when you run a session in debug mode and do not
discard target data. You can view the final value for Sequence
Generator and Normalizer transformations in the transformation
properties.
Designer Behavior
When the Debugger starts, you cannot perform the following
tasks:
- Close the folder or open another folder.
- Use the Navigator.
- Perform repository functions, such as Save.
- Edit or close the mapping.
- Switch to another tool in the Designer, such as Target
Designer.
- Close the Designer.
Note: Dynamic partitioning is disabled during debugging.
Monitoring the Debugger
 When you run the Debugger, you can monitor the following
information:
Session status: Monitor the status of the session.
Data movement: Monitor data as it moves through
transformations.
Breakpoints: Monitor data that meets breakpoint conditions.
Target data: Monitor target data on a row-by-row basis.
 The Mapping Designer displays windows and debug indicators
that help you monitor the session:
Debug indicators: Debug indicators on transformations help you
follow breakpoints and data flow.
Instance window: When the Debugger pauses, you can view
transformation data and row information in the Instance window.
Target window: View target data for each target in the mapping.
Output window: The Integration Service writes messages to the
following tabs in the Output window:
- Debugger tab: The debug log displays in the Debugger tab.
- Session Log tab: The session log displays in the Session Log tab.
28
- Notifications tab: Displays messages from the Repository
Service.
 You can step to connected transformations in the mapping,
even if they do not have an associated breakpoint.
 You cannot step to the following instances:
- Sources
- Targets
- Unconnected transformations
- Mapplets not selected for debugging
Modifying Data
 When the Debugger pauses, the current instance displays in the
Instance window, and the current instance indicator displays on
the transformation in the mapping. You can make the following
modifications to the current instance when the Debugger
pauses on a data breakpoint:
Modify output data: You can modify output data of the current
transformation. When you continue the session, the Integration
Service validates the data. It performs the same validation it
performs when it passes data from port to port in a regular
session.
Change null data to not-null: Clear the null column, and enter a
value in the value column to change null data to not-null.
Change not-null to null: Select the null column to change not-null
data to null. The Designer prompts you to confirm that you want
to make this change.
Modify row types: Modify Update Strategy, Filter, or Router
transformation row types.
 For Router transformations, you can change the row type to
override the group condition evaluation for user defined groups.
 For example, if the group condition evaluates to false, the rows
are not passed through the output ports to the next
transformation or target.
 The Instance window displays <no data available>, and the row
type is filtered. If you want to pass the filtered row to the next
transformation or target, you can change the row type to
Insert.
 Likewise, for a group that meets the group condition, you can
change the row type from insert to filtered.
 After you change data, you can refresh the cache before you
continue the session.
 When you issue the Refresh command, the Designer processes
the request for the current transformation, and you can see if
the data you enter is valid.
 You can change the data again before you continue the session
Restrictions
 You cannot change data for the following output ports:
Normalizer transformation: Generated Keys and Generated
Column ID ports.
 Rank transformation: RANKINDEX port.
Router transformation: All output ports.
Sequence Generator transformation: CURRVAL and NEXTVAL
ports.
Lookup transformation: NewLookupRow port for a Lookup
transformation configured to use a dynamic cache.
Custom transformation: Ports in output groups other than the
current output group.
Java transformation: Ports in output groups other than the
current output group.
 Additionally, you cannot change data associated with the following:
- Mapplets that are not selected for debugging
- Input or input/output ports
- Output ports when the Debugger pauses on an error breakpoint
4. WORKFLOW BASICS GUIDE
Workflow Tasks
 You can create tasks in Task Developer, Workflow designer or
Worklet designer
 Tasks created in Task Dev are reusable but those created in
Workflow/worklet designer are not.
 You can create the following types of tasks in the Workflow
Manager:
Assignment: Assigns a value to a workflow variable.
Command: Specifies a shell command to run during the
workflow.
Control: Stops or aborts the workflow.
Decision: Specifies a condition to evaluate.
Email: Sends email during the workflow.
Event-Raise: Notifies the Event-Wait task that an event has
occurred.
Event-Wait: Waits for an event to occur before executing the next
task.
Session: Runs a mapping you create in the Designer.
Timer: Waits for a timed event to trigger.
Workflow Manager Windows
 The Workflow Manager displays the following windows to help
you create and organize workflows:
- Navigator
- Workspace
- Output
- Overview
29
Enhanced Security
 The Workflow Manager has an enhanced security option to
specify a default set of permissions for connection objects.
 When you enable enhanced security, the Workflow Manager
assigns default permissions on connection objects for users,
groups, and others.
 When you disable enable enhanced security, the Workflow
Manager Assigns read, write, and execute permissions to all
users that would otherwise receive permissions of the default
group.
 If you delete the owner from the repository, the Workflow
Manager assigns ownership of the object to the administrator.
Viewing and Comparing Versioned Repository Objects
 You can view and compare versions of objects in the Workflow
Manager. If an object has multiple versions, you can find the
versions of the object in the View History window. In addition to
comparing versions of an object in a window, you can view the
various versions of an object in the workspace to graphically
compare them.
 Use the following rules and guidelines when you view older
versions of objects in the workspace:
- You cannot simultaneously view multiple versions of composite
objects, such as workflows and worklets.
- Older versions of a composite object might not include the child
objects that were used when the composite object was checked
in. If you open a composite object that includes a child object
version that is purged from the repository, the preceding version
of the child object appears in the workspace as part of the
composite object. For example, you might want to view version 5
of a workflow that originally included version 3 of a session, but
version 3 of the session is purged from the repository. When you
view version 5 of the workflow, version 2 of the session appears
as part of the workflow.
 - You cannot view older versions of sessions if they reference
deleted or invalid mappings, or if they do not have a session
configuration.
Searching for Versioned Objects
 Use an object query to search for versioned objects in the
repository that meet specified conditions. When you run a
query, the repository returns results based on those conditions.
You may want to create an object query to perform the
following tasks:
 Track repository objects during development: You can add Label,
User, Last saved, or Comments parameters to queries to track
objects during development.
 Associate a query with a deployment group: When you create a
dynamic deployment group, you can associate an object query
with it.
Comparing Repository Objects
 Use the Workflow Manager to compare two repository objects of
the same type to identify differences between the objects. For
example, if you have two similar Email tasks in a folder, you can
compare them to see which one contains the attributes you
need. When you compare two objects, the Workflow Manager
displays their attributes in detail.
 You can compare objects across folders and repositories. You
must open both folders to compare the objects. You can
compare a reusable object with a non-reusable object. You can
also compare two versions of the same object.
 You can compare the following types of objects:
- Tasks
- Sessions
- Worklets
- Workflows
 You can also compare instances of the same type. For example, if
the workflows you compare contain worklet instances with the
same name, you can compare the instances to see if they differ.
 Use the Workflow Manager to compare the following instances
and attributes:
- Instances of sessions and tasks in a workflow or worklet
comparison. For example, when you compare workflows, you can
compare task instances that have the same name.
- Instances of mappings and transformations in a session
comparison. For example, when you compare sessions, you can
compare mapping instances.
- The attributes of instances of the same type within a mapping
comparison. For example, when you compare flat file sources,
you can compare attributes, such as file type (delimited or fixed),
delimiters, escape characters, and optional quotes.
30
 You can compare schedulers and session configuration objects
in the Repository Manager.
 You cannot compare objects of different types. For example, you
cannot compare an Email task with a Session task.
 When you compare objects, the Workflow Manager displays the
results in the Diff Tool window. The Diff Tool output contains
different nodes for different types of objects.
 When you import Workflow Manager Objects, you can compare
object conflicts.
 A workflow must contain a Start task. The Start task represents
the beginning of a workflow.
 When you create a workflow, the Workflow Designer creates a
Start task and adds it to the workflow.
 You cannot delete the Start task
 You may decide to delete a workflow that you no longer use.
When you delete a workflow, you delete all nonreusable tasks
and reusable task instances associated with the workflow.
Reusable tasks used in the workflow remain in the folder when
you delete the workflow.
 If you delete a workflow that is running, the Integration Service
aborts the workflow.
 If you delete a workflow that is scheduled to run, the
Integration Service removes the workflow from the schedule.
 If you want to write performance data to the repository you must
perform the following tasks:
- Configure the session to collect performance data.
- Configure the session to write performance data to repository.
- Configure Integration Service to persist run-time statistics to the
repository at the verbose level.
Guidelines for Entering Pre- and Post-Session SQL Commands
 Use the following guidelines when creating the SQL statements:
- Use any command that is valid for the database type. However,
the Integration Service does not allow nested comments, even
though the database might.
- Use a semicolon (;) to separate multiple statements. The
Integration Service issues a commit after each statement.
- The Integration Service ignores semicolons within /* ...*/.
- If you need to use a semicolon outside of comments, you can
escape it with a backslash (\).
- The Workflow Manager does not validate the SQL.
Error Handling -
 You can configure error handling on the Config Object tab.
 You can choose to stop or continue the session if the Integration
Service encounters an error issuing the pre- or post- session SQL
command
 The Workflow Manager provides the following types of shell
commands for each Session task:
Pre-session command - The Integration Service performs pre-
session shell commands at the beginning of a session. You can
configure a session to stop or continue if a pre-session shell
command fails.
Post-session success command - The Integration Service
performs post-session success commands only if the session
completed successfully.
Post-session failure command - The Integration Service performs
post-session failure commands only if the session failed to
complete.

Pre-Session Shell Command Errors

 If you select stop, the Integration Service stops the session, but
continues with the rest of the workflow.
 If you select Continue, the Integration Service ignores the errors
and continues the session.
 By default the Integration Service stops the session upon shell
command errors.

Session Configuration Object

Configuration Object and Config Object Tab Settings

 You can configure the following settings in a session configuration
object or on the Config Object tab in session properties:
Advanced - Advanced settings allow you to configure constraint-
based loading, lookup caches, and buffer sizes.
Log options - Log options allow you to configure how you want to
save the session log. By default, the Log Manager saves only the
current session log.
Error handling - Error Handling settings allow you to determine if
the session fails or continues when it encounters pre-session
command errors, stored procedure errors, or a specified number
of session errors.
Partitioning options - Partitioning options allow the Integration
Service to determine the number of partitions to create at run
time.
Session on grid - When Session on Grid is enabled, the
Integration Service distributes session threads to the nodes in a
grid to increase performance and scalability.

31
 Tasks

32
  Use the Decision task instead of multiple link conditions in a
workflow.
 Instead of specifying multiple link conditions, use the predefined
condition variable in a Decision task to simplify link conditions.

Example
 For example, you have a Command task that depends on the
status of the three sessions in the workflow. You want the
Integration Service to run the Command task when any of the
three sessions fails. To accomplish this, use a Decision task with
the following decision condition:
$Q1_session.status = FAILED OR $Q2_session.status = FAILED OR
$Q3_session.status = FAILED

 You can then use the predefined condition variable in the input
link condition of the Command task. Configure the input link
with the following link condition:
$Decision.condition = True

 You can configure the same logic in the workflow without the
Decision task.
 Without the Decision task, you need to use three link conditions
Decision Task
and treat the input links to the Command task as OR links.
 You can enter a condition that determines the execution of the
workflow, similar to a link condition with the Decision task.
Event Task
 The Decision task has a predefined variable called
 You can define events in the workflow to specify the sequence of
$Decision_task_name.condition that represents the result of the
task execution.
decision condition.
 The event is triggered based on the completion of the sequence
 The Integration Service evaluates the condition in the Decision
of tasks.
task and sets the predefined condition variable to True (1) or
False (0).
 Use the following tasks to help you use events in the workflow:
 You can specify one decision condition per Decision task.
Event-Raise task - Event-Raise task represents a user-defined
 After the Integration Service evaluates the Decision task, use the
event. When the Integration Service runs the Event-Raise task,
predefined condition variable in other expressions in the
the Event-Raise task triggers the event. Use the Event-Raise task
workflow to help you develop the workflow.
with the Event-Wait task to define events.
 Depending on the workflow, you might use link conditions
Event-Wait task - The Event-Wait task waits for an event to occur.
instead of a Decision task.
Once the event triggers, the Integration Service continues
 However, the Decision task simplifies the workflow.
executing the rest of the workflow.
If you do not specify a condition in the Decision task, the
Integration Service evaluates the Decision task to True.
 To coordinate the execution of the workflow, you may specify the
Using the Decision Task
following types of events for the Event-Wait and
33
 Event-Raise tasks:
Predefined event - A predefined event is a file-watch event. For
predefined events, use an Event-Wait task to instruct the
Integration Service to wait for the specified indicator file to
appear before continuing with the rest of the workflow. When
the Integration Service locates the indicator file, it starts the next
task in the workflow.
User-defined event - A user-defined event is a sequence of tasks
in the workflow. Use an Event-Raise task to specify the location of
the user-defined event in the workflow. A user-defined event is
sequence of tasks in the branch from the Start task leading to the
Event-Raise task.
 When all the tasks in the branch from the Start task to the Event-
Raise task complete, the Event-Raise task triggers the event. The
Event-Wait task waits for the Event-Raise task to trigger the
event before continuing with the rest of the tasks in its branch.
Timer Task
 You can specify the period of time to wait before the Integration
Service runs the next task in the workflow with the Timer task.
 You can choose to start the next task in the workflow at a
specified time and date.
 You can also choose to wait a period of time after the start time
of another task, workflow, or worklet before starting the next
task.
The Timer task has the following types of settings:
Absolute time - You specify the time that the Integration Service
starts running the next task in the workflow. You may specify the
date and time, or you can choose a user-defined workflow
variable to specify the time.
Relative time - You instruct the Integration Service to wait for a
specified period of time after the Timer task, the parent
workflow, or the top-level workflow starts.
 For example, a workflow contains two sessions. You want the
Integration Service wait 10 minutes after the first session
completes before it runs the second session. Use a Timer task
after the first session. In the Relative Time setting of the Timer
task, specify ten minutes from the start time of the Timer task.
Use a Timer task anywhere in the workflow after the Start task.
34
Sources
Allocating Buffer Memory
 When the Integration Service initializes a session, it allocates
blocks of memory to hold source and target data.
 The Integration Service allocates at least two blocks for each
source and target partition.
 Sessions that use a large number of sources or targets might
require additional memory blocks.
 If the Integration Service cannot allocate enough memory
blocks to hold the data, it fails the session.
Partitioning Sources
 You can create multiple partitions for relational, Application, and
file sources.
 For relational or Application sources, the Integration Service
creates a separate connection to the source database for each
partition you set in the session properties.
 For file sources, you can configure the session to read the source
with one thread or multiple threads.
Overriding the Source Table Name
 If you override the source table name on the Properties tab of
the source instance, and you override the source table name
using an SQL query, the Integration Service uses the source table
name defined in the SQL query.
Targets
Working with Relational Targets
 When you configure a session to load data to a relational target,
you define most properties in the Transformations view on the
Mapping tab.
Performing a Test Load
 With a test load, the Integration Service reads and transforms
data without writing to targets.
 The Integration Service reads the number you configure for the
test load.
 The Integration Service generates all session files and performs all
pre- and post-session functions, as if running the full session.
 To configure a session to perform a test load, enable test load
and enter the number of rows to test.
 The Integration Service writes data to relational targets, but
rolls back the data when the session completes.
 For all other target types, such as flat file and SAP BW, the
Integration Service does not write data to the targets.
Use the following guidelines when performing a test load:
- You cannot perform a test load on sessions using XML sources.
- You can perform a test load for relational targets when you
configure a session for normal mode.
- If you configure the session for bulk mode, the session fails.
- Enable a test load on the session Properties tab.
 You can configure the following properties for relational targets:
Target database connection - Define database connection
information.
Target properties - You can define target properties such as
target load type, target update options, and reject options.
Truncate target tables - The Integration Service can truncate
target tables before loading data.
Deadlock retries - You can configure the session to retry
deadlocks when writing to targets or a recovery table.
Drop and recreate indexes - Use pre- and post-session SQL to
drop and recreate an index on a relational target table to
optimize query speed.
Constraint-based loading - The Integration Service can load data
to targets based on primary key-foreign key constraints and active
sources in the session mapping.
Bulk loading - You can specify bulk mode when loading to DB2,
Microsoft SQL Server, Oracle, and Sybase databases.
 You can define the following properties in the session and
override the properties you define in the mapping:
Table name prefix - You can specify the target owner name or
prefix in the session properties to override the table name prefix
in the mapping.
Pre-session SQL - You can create SQL commands and execute
them in the target database before loading data to the target. For
example, you might want to drop the index for the target table
before loading data into it.
Post-session SQL - You can create SQL commands and execute
them in the target database after loading data to the target. For
example, you might want to recreate the index for the target
table after loading data into it.
Target table name - You can override the target table name for
each relational target.
Target Table Truncation
 If you enable truncate target tables with the following sessions,
the Integration Service does not truncate target tables:
Incremental aggregation - When you enable both truncate target
tables and incremental aggregation in the session properties, the
Workflow Manager issues a warning that you cannot enable
truncate target tables and incremental aggregation in the same
session.
Test load - When you enable both truncate target tables and test
load, the Integration Service disables the truncate table function,
runs a test load session, and writes a message to the session log
indicating that the truncate target tables’ option is turned off for
the test load session.
Real-time - The Integration Service does not truncate target
tables when you restart a JMS or WebSphere MQ real-time
session that has recovery data.
35
Dropping and Recreating Indexes
 After you insert significant amounts of data into a target, you
normally need to drop and recreate indexes on that table to
optimize query speed.
 You can drop and recreate indexes by:
Using pre- and post-session SQL - The preferred method for
dropping and re-creating indexes is to define an SQL statement in
the Pre SQL property that drops indexes before loading data to
the target. Use the Post SQL property to recreate the indexes
after loading data to the target. Define the Pre SQL and Post SQL
properties for relational targets in the Transformations view on
the Mapping tab in the session properties.
Using the Designer - The same dialog box you use to generate
and execute DDL code for table creation can drop and recreate
indexes. However, this process is not automatic
Constraint-Based Loading
 In the Workflow Manager, you can specify constraint-based
loading for a session.
 When you select this option, the Integration Service orders the
target load on a row-by-row basis.
 For every row generated by an active source, the Integration
Service loads the corresponding transformed row first to the
primary key table, then to any foreign key tables.
 Constraint-based loading depends on the following requirements:
Active source - Related target tables must have the same active
source.
Key relationships - Target tables must have key relationships.
Target connection groups - Targets must be in one target
connection group.
Treat rows as insert - Use this option when you insert into the
target. You cannot use updates with constraint based loading.
  The Integration Service includes T_5 and T_6 in a different target
connection group because they are in a different target load
order group from the first four targets.

Bulk Loading
 You can enable bulk loading when you load to DB2, Sybase,
Oracle, or Microsoft SQL Server.
 If you enable bulk loading for other database types, the
Integration Service reverts to a normal load.

 When bulk loading, the Integration Service invokes the database

bulk utility and bypasses the database log, which speeds
performance.
 Without writing to the database log, however, the target
database cannot perform rollback.

 Note: When loading to DB2, Microsoft SQL Server, and Oracle

targets, you must specify a normal load for data driven sessions.
 In the first pipeline, target T_1 has a primary key, T_2 and T_3
When you specify bulk mode and data driven, the Integration
contain foreign keys referencing the T1 primary key. T_3 has a
Service reverts to normal load.
primary key that T_4 references as a foreign key
 Since these tables receive records from a single active source,
Committing Data
SQ_A, the Integration Service loads rows to the target in the
 When bulk loading to Sybase and DB2 targets, the Integration
following order:
Service ignores the commit interval you define in the session
1. T_1
properties and commits data when the writer block is full.
2. T_2 and T_3 (in no particular order)
 When bulk loading to Microsoft SQL Server and Oracle targets,
3. T_4
the Integration Service commits data at each commit interval.
 The Integration Service loads T_1 first because it has no foreign
Also, Microsoft SQL Server and Oracle start a new bulk load
key dependencies and contains a primary key referenced by T_2
transaction after each commit.
and T_3.
 The Integration Service then loads T_2 and T_3, but since T_2
Reserved Words
and T_3 have no dependencies, they are not loaded in any
 If any table name or column name contains a database reserved
particular order.
word, such as MONTH or YEAR, the session fails with database
 The Integration Service loads T_4 last, because it has a foreign
errors when the Integration Service executes SQL against the
key that references a primary key in T_3.
database.
 You can create and maintain a reserved words file, reswords.txt,
 After loading the first set of targets, the Integration Service
in the server/bin directory.
begins reading source B.
 When the Integration Service initializes a session, it searches for
 If there are no key relationships between T_5 and T_6, the
reswords.txt.
Integration Service reverts to a normal load for both targets.
 If the file exists, the Integration Service places quotes around
 If T_6 has a foreign key that references a primary key in T_5, since
matching reserved words when it executes SQL against the
T_5 and T_6 receive data from a single active source, the
database.
Aggregator AGGTRANS, the Integration Service loads rows to the
 Use the following rules and guidelines when working with
tables in the following order:
reserved words.
- T_5
- The Integration Service searches the reserved words file when
- T_6
it generates SQL to connect to source, target, and lookup
databases.
 T_1, T_2, T_3, and T_4 are in one target connection group if you
- If you override the SQL for a source, target, or lookup, you
use the same database connection for each target, and you use
must enclose any reserved word in quotes.
the default partition properties.
 T_5 and T_6 are in another target connection group together if
Working with Active Sources
you use the same database connection for each target and you
use the default partition properties.
36
 An active source is an active transformation the Integration
Service uses to generate rows. An active source can be any of
the following transformations:
- Aggregator
- Application Source Qualifier
- Custom, configured as an active transformation
- Joiner
- MQ Source Qualifier
- Normalizer (VSAM or pipeline)
- Rank
- Sorter
- Source Qualifier
- XML Source Qualifier
- Mapplet, if it contains any of the above transformations

 Note: The Filter, Router, Transaction Control, and Update Strategy

transformations are active transformations in that they can
change the number of rows that pass through. However, they
are not active sources in the mapping because they do not
generate rows. Only transformations that can generate rows are
active sources.
Writing to Fixed-Width Files with Flat File Target Definitions
 When you want to output to a fixed-width flat file based on a flat
file target definition, you must configure precision and field
width for the target field to accommodate the total length of the
target field.
 If the data for a target field is too long for the total length of the
field, the Integration Service performs one of the following
actions:
- Truncates the row for string columns
- Writes the row to the reject file for numeric and date time
columns

 Note: When the Integration Service writes a row to the reject file,
it writes a message in the session log.

Writing Empty Fields for Unconnected Ports in Fixed-Width File

Integration Service Handling for File Targets
Writing to Fixed-Width Flat Files with Relational Target
Definitions
Definitions
 The Integration Service does not write data in unconnected ports
to fixed-width files.
 If you want the Integration Service to write empty fields for the
unconnected ports, create output ports in an upstream
transformation that do not contain data.
 Then connect these ports containing null values to the fixed
width flat file target definition
Workflow Validation
 The Workflow Manager validates the following properties:
Expressions - Expressions in the workflow must be valid.
Tasks - Non-reusable task and Reusable task instances in the
workflow must follow validation rules.
Scheduler - If the workflow uses a reusable scheduler, the
Workflow Manager verifies that the scheduler exists.

37
 The Workflow Manager marks the workflow invalid if the
scheduler you specify for the workflow does not exist in the
folder.
 The Workflow Manager also verifies that you linked each task
properly.
Note: The Workflow Manager validates Session tasks
separately. If a session is invalid, the workflow may still be valid
Validating Multiple Workflows
 You can validate multiple workflows or worklets without fetching
them into the workspace.
 To validate multiple workflows, you must select and validate the
workflows from a query results view or a view dependencies list.
 When you validate multiple workflows, the validation does not
include sessions, nested worklets, or reusable worklet objects in
the workflows.
 You can save and optionally check in workflows that change from
invalid to valid status.
Task Validation
 The Workflow Manager validates each task in the workflow as
you create it.
 When you save or validate the workflow, the Workflow Manager
validates all tasks in the workflow except Session tasks
 When you delete a reusable task, the Workflow Manager
removes the instance of the deleted task from workflows.
 The Workflow Manager also marks the workflow invalid when
you delete a reusable task used in a workflow.
 The Workflow Manager verifies that there are no duplicate task
names in a folder, and that there are no duplicate task instances
in the workflow
 The Workflow Manager uses the following rules to validate tasks:
Assignment - The Workflow Manager validates the expression
you enter for the Assignment task. For example, the Workflow
Manager verifies that you assigned a matching datatype value to
the workflow variable in the assignment expression.
Command - The Workflow Manager does not validate the shell
command you enter for the Command task.
Event-Wait - If you choose to wait for a predefined event, the
Workflow Manager verifies that you specified a file to watch. If
you choose to use the Event-Wait task to wait for a user-defined
event, the Workflow Manager verifies that you specified an
event.
Event-Raise - The Workflow Manager verifies that you specified a
user-defined event for the Event-Raise task.
Timer - The Workflow Manager verifies that the variable you
specified for the Absolute Time setting has the Date/Time
datatype.
Start - The Workflow Manager verifies that you linked the Start
task to at least one task in the workflow.
38
 When a task instance is invalid, the workflow using the task
instance becomes invalid.
 When a reusable task is invalid, it does not affect the validity of
the task instance used in the workflow.
 However, if a Session task instance is invalid, the workflow may
still be valid.
 The Workflow Manager validates sessions differently.
Session Validation
 If you delete objects associated with a Session task such as
session configuration object, Email, or Command task, the
Workflow Manager marks a reusable session invalid.
 However, the Workflow Manager does not mark a non-reusable
session invalid if you delete an object associated with the
session.
 If you delete a shortcut to a source or target from the mapping,
the Workflow Manager does not mark the session invalid.
 The Workflow Manager does not validate SQL overrides or filter
conditions entered in the session properties when you validate a
session.
 You must validate SQL override and filter conditions in the SQL
Editor.
 If a reusable or non-reusable session instance is invalid, the
Workflow Manager marks it invalid in the Navigator and in the
Workflow Designer workspace.
 Workflows using the session instance remain valid
Expression Validation
 The Workflow Manager validates all expressions in the workflow.
You can enter expressions in the Assignment task, Decision task,
and link conditions.
 The Workflow Manager writes any error message to the Output
window
Workflow Schedules
 You can schedule a workflow to run continuously, repeat at a
given time or interval, or you can manually start a workflow
 If you configure multiple instances of a workflow, and you
schedule the workflow run time, the Integration Service runs
all instances at the scheduled time. You cannot schedule
workflow instances to run at different times.
 If you choose a different Integration Service for the workflow or
restart the Integration Service, it reschedules all workflows.
 This includes workflows that are scheduled to run continuously
but whose start time has passed and workflows that are
scheduled to run continuously but were unscheduled.
 You must manually reschedule workflows whose start time has
passed if they are not scheduled to run continuously.
 If you delete a folder, the Integration Service removes workflows
from the schedule when it receives notification from the
Repository Service.
 If you copy a folder into a repository, the Integration Service
reschedules all workflows in the folder when it receives the
notification.

 The Integration Service does not run the workflow in the

following situations:
- The prior workflow run fails. When a workflow fails, the
Integration Service removes the workflow from the schedule, and
you must manually reschedule it. You can reschedule the
workflow in the Workflow Manager or using pmcmd.
- The Integration Service process fails during a prior workflow
run. When the Integration Services process fails in a highly
available domain and a running workflow is not configured for
recovery, the Integration Service removes the workflow from the
schedule. You can reschedule the workflow in the Workflow
Manager or using pmcmd.
- You remove the workflow from the schedule. You can remove
the workflow from the schedule in the Workflow Manager or
using pmcmd.
- The Integration Service is running in safe mode. In safe mode,
the Integration Service does not run scheduled workflows,
including workflows scheduled to run continuously or run on
service initialization. When you enable the Integration Service in
normal mode, the Integration Service runs the scheduled
workflows.

Session and Workflow Logs

 The following steps describe how the Log Manager processes
session and workflow logs:
1. The Integration Service writes binary log files on the node. It
sends information about the sessions and workflows to the Log
Manager.
2. The Log Manager stores information about workflow and
session logs in the domain configuration database. The domain
configuration database stores information such as the path to the
log file location, the node that contains the log, and the
Integration Service that created the log.

39
 3. When you view a session or workflow in the Log Events
window, the Log Manager retrieves the information from the
domain configuration database to determine the location of the
session or workflow logs.
 A partition is a pipeline stage that executes in a single reader,
4. The Log Manager dispatches a Log Agent to retrieve the log
events on each node to display in the Log Events window.
 The number of partitions in any pipeline stage equals the number
 You can also configure a workflow to produce text log files
 When you configure the workflow or session to produce text log
 By default, the Integration Service creates one partition in every
files, the Integration Service creates the binary log and the text
log file
 Complete the following tasks to configure partitions for a session:
Message Severity
Log Events Window
 The Log Events window displays the following information for
each session and workflow:
Severity - Lists the type of message, such as informational or
error.
Time stamp - Date and time the log event reached the Log Agent.
Node - Node on which the Integration Service process is running.
Thread - Thread ID for the workflow or session.
Process ID - Windows or UNIX process identification numbers.
Displays in the Output window only.
Message Code - Message code and number.
Message - Message associated with the log event.
Writing to Log Files
 When you create a workflow or session log, you can configure log
options in the workflow or session properties.
 You can configure the following information for a workflow or
session log:
Write Backward Compatible Log File - Select this option to create
a text file for workflow or session logs. If you do not select the
option, the Integration Service creates the binary log only.
Log File Directory - The directory where you want the log file
created. By default, the Integration Service writes the workflow
log file in the directory specified in the service process variable,
$PMWorkflowLogDir. It writes the session log file in the directory
specified in the service process variable, $PMSessionLogDir
40
5. ADVANCED WORKFLOW GUIDE
Understanding Pipeline Partitioning
transformation, or writer thread.
of threads in the stage.
pipeline stage
- Set partition attributes including partition points, the number
of partitions, and the partition types.
- You can enable the Integration Service to set partitioning at run
time. When you enable dynamic partitioning, the Integration
Service scales the number of session partitions based on factors
such as the source database partitions or the number of nodes
in a grid.
- After you configure a session for partitioning, you can configure
memory requirements and cache directories for each
transformation.
- The Integration Service evaluates mapping variables for each
partition in a target load order group. You can use variable
functions in the mapping to set the variable values.
- When you create multiple partitions in a pipeline, the
Workflow Manager verifies that the Integration Service can
maintain data consistency in the session using the partitions.
When you edit object properties in the session, you can impact
partitioning and cause a session to fail.
- You add or edit partition points in the session properties. When
you change partition points you can define the partition type and
add or delete partitions
Partitioning Attributes
Partition points - Partition points mark thread boundaries and
divide the pipeline into stages. The Integration Service
redistributes rows of data at partition points.
Number of partitions - A partition is a pipeline stage that
executes in a single thread. If you purchase the Partitioning
option, you can set the number of partitions at any partition
point. When you add partitions, you increase the number of
processing threads, which can improve session performance.
Partition types - The Integration Service creates a default
partition type at each partition point. If you have the Partitioning
option, you can change the partition type. The partition type
controls how the Integration Service distributes data among
partitions at partition points
Partition Points
 A stage is a section of a pipeline between any two partition
points.
 When you set a partition point at a transformation, the new
pipeline stage includes that transformation.
 Figure shows the default partition points and pipeline stages for a
mapping with one pipeline:
 When you add a partition point, you increase the number of
pipeline stages by one.
 Similarly, when you delete a partition point, you reduce the
number of stages by one.
 Partition points mark the points in the pipeline where the
Integration Service can redistribute data across partitions.
Number of Partitions
 A partition is a pipeline stage that executes in a single reader,
transformation, or writer thread.
 The number of partitions in any pipeline stage equals the number
of threads in that stage
 You can define up to 64 partitions at any partition point in a
pipeline
 When you increase or decrease the number of partitions at any
partition point, the Workflow Manager increases or decreases
the number of partitions at all partition points in the pipeline.
 The number of partitions remains consistent throughout the
pipeline
 If you define three partitions at any partition point, the
Workflow Manager creates three partitions at all other
partition points in the pipeline
 The number of partitions you create equals the number of
connections to the source or target.
 If the pipeline contains a relational source or target, the number
of partitions at the source qualifier or target instance equals the
number of connections to the database.
 If the pipeline contains file sources, you can configure the session
to read the source with one thread or with multiple threads.
 For example, when you define three partitions across the
mapping, the master thread creates three threads at each
pipeline stage, for a total of 12 threads
Partitioning Multiple Input Group Transformations
41
 When you connect more than one pipeline to a multiple input
group transformation, the Integration Service maintains the
transformation threads or creates a new transformation thread
depending on whether or not the multiple input group
transformation is a partition point:
Partition point does not exist at multiple input group
transformation - When a partition point does not exist at a
multiple input group transformation, the Integration Service
processes one thread at a time for the multiple input group
transformation and all downstream transformations in the
stage.
Partition point exists at multiple input group transformation -
When a partition point exists at a multiple input group
transformation, the Integration Service creates a new pipeline
stage and processes the stage with one thread for each
partition. The Integration Service creates one transformation
thread for each partition regardless of the number of output
groups the transformation contains
Partition Types
You can define the following partition types in the Workflow
Manager:
Database partitioning - The Integration Service queries the IBM
DB2 or Oracle database system for table partition information. It
reads partitioned data from the corresponding nodes in the
database. You can use database partitioning with Oracle or IBM
DB2 source instances on a multi-node tablespace. You can use
database partitioning with DB2 targets.
Hash auto-keys - The Integration Service uses a hash function to
group rows of data among partitions. The Integration Service
groups the data based on a partition key. The Integration
Service uses all grouped or sorted ports as a compound
partition key. You may need to use hash auto-keys partitioning
at Rank, Sorter, and unsorted Aggregator transformations.
Hash user keys - The Integration Service uses a hash function to
group rows of data among partitions. You define the number of
ports to generate the partition key.
Key range - With key range partitioning, the Integration Service
distributes rows of data based on a port or set of ports that you
define as the partition key. For each port, you define a range of
values. The Integration Service uses the key and ranges to send
rows to the appropriate partition. Use key range partitioning
when the sources or targets in the pipeline are partitioned by
key range.
Pass-through - In pass-through partitioning, the Integration
Service processes data without redistributing rows among
partitions. All rows in a single partition stay in the partition after
crossing a pass-through partition point. Choose pass-through
partitioning when you want to create an additional pipeline stage
to improve performance, but do not want to change the
distribution of data across partitions.
 Round-robin - The Integration Service distributes data evenly
among all partitions. Use round-robin partitioning where you
want each partition to process approximately the same number
of rows.
Dynamic Partitioning
 When you use dynamic partitioning, you can configure the
partition information so the Integration Service determines the
number of partitions to create at run time
 The Integration Service scales the number of session partitions
at run time based on factors such as source database
partitions or the number of nodes in a grid.
 If any transformation in a stage does not support partitioning,
or if the partition configuration does not support dynamic
partitioning, the Integration Service does not scale partitions
in the pipeline. The data passes through one partition.
Note: Do not configure dynamic partitioning for a session that
contains manual partitions. If you set dynamic partitioning to a
value other than disabled and you manually partition the
session, the session is invalid
Configuring Dynamic Partitioning
Configure dynamic partitioning using one of the following
methods:
Disabled - Do not use dynamic partitioning. Define the number of
partitions on the Mapping tab.
Based on number of partitions- Sets the partitions to a number
that you define in the Number of Partitions attribute. Use the
$DynamicPartitionCount session parameter, or enter a number
greater than 1.
Based on number of nodes in grid - Sets the partitions to the
number of nodes in the grid running the session. If you configure
this option for sessions that do not run on a grid, the session runs
in one partition and logs a message in the session log.
Based on source partitioning - Determines the number of
partitions using database partition information. The number of
partitions is the maximum of the number of partitions at the
source. For Oracle sources that use composite partitioning, the
number of partitions is the maximum of the number of
subpartitions at the source.
Based on number of CPUs - Sets the number of partitions equal
to the number of CPUs on the node that prepares the session. If
the session is configured to run on a grid, dynamic partitioning
sets the number of partitions equal to the number of CPUs on
the node that prepares the session multiplied by the number of
nodes in the grid.
Rules and Guidelines for Dynamic Partitioning
 Use the following rules and guidelines with dynamic partitioning:
42
- Dynamic partitioning uses the same connection for each
partition.
- You cannot use dynamic partitioning with XML sources and
targets.
- You cannot use dynamic partitioning with the Debugger.
- Sessions that use SFTP fail if you enable dynamic partitioning.
- When you set dynamic partitioning to a value other than
disabled, and you manually partition the session on the
Mapping tab, you invalidate the session.
- The session fails if you use a parameter other than
$DynamicPartitionCount to set the number of partitions.
The following dynamic partitioning configurations cause a
session to run with one partition:
1. You override the default cache directory for an Aggregator,
Joiner, Lookup, or Rank transformation. The Integration Service
partitions a transformation cache directory when the default is
$PMCacheDir.
2. You override the Sorter transformation default work directory.
The Integration Service partitions the Sorter transformation work
directory when the default is $PMTempDir.
3. You use an open-ended range of numbers or date keys with a
key range partition type.
4. You use datatypes other than numbers or dates as keys in key
range partitioning.
5. You use key range relational target partitioning.
6. You create a user-defined SQL statement or a user-defined
source filter.
7. You set dynamic partitioning to the number of nodes in the
grid, and the session does not run on a grid.
8. You use pass-through relational source partitioning.
9. You use dynamic partitioning with an Application Source
Qualifier.
10. You use SDK or PowerConnect sources and targets with
dynamic partitioning
Cache Partitioning
 When you create a session with multiple partitions, the
Integration Service may use cache partitioning for the
Aggregator, Joiner, Lookup, Rank, and Sorter transformations.
 When the Integration Service partitions a cache, it creates a
separate cache for each partition and allocates the configured
cache size to each partition. The Integration Service stores
different data in each cache, where each cache contains only the
rows needed by that partition.
 As a result, the Integration Service requires a portion of total
cache memory for each partition.
Mapping Variables in Partitioned Pipelines
Integration Service evaluates the value of a mapping variable in
each partition separately.
 The Integration Service uses the following process to evaluate
variable values:
1. It updates the current value of the variable separately in each
partition according to the variable function used in the mapping.
2. After loading all the targets in a target load order group, the
Integration Service combines the current values from each
partition into a single final value based on the aggregation type of
the variable.
3. If there is more than one target load order group in the
session, the final current value of a mapping variable in a target
load order group becomes the current value in the next target
load order group.
4. When the Integration Service finishes loading the last target
load order group, the final current value of the variable is saved
into the repository
The following changes to mappings can cause session failure:
 Any changes you make (such as adding/altering transformations
etc) that affects the existing partitions or partition points.
Partition Points
 Partition points mark the boundaries between threads in a
pipeline and divide the pipeline into stages.
 By default, IS creates one Reader and one Writer Partition
points.
 The Integration Service redistributes rows of data at partition
points.
 You can add partition points to increase the number of
transformation threads and increase session performance.
 When you configure a session to read a source database, the
Integration Service creates a separate connection and SQL
query to the source database for each partition
 When you configure a session to load data to a relational target,
the Integration Service creates a separate connection to the
target database for each partition at the target instance.
 You configure the reject file names and directories for the target.
The Integration Service creates one reject file for each target
partition
 You can configure a session to read a source file with one thread
or with multiple threads.
 You must choose the same connection type for all partitions
that read the file.
43
 When you configure a session to write to a file target, you can
write the target output to a separate file for each partition or to
a merge file that contains the target output for all partitions.
 You can configure connection settings and file properties for each
target partition.
 When you create a partition point at transformations, the
Workflow Manager sets the default partition type.
 You can change the partition type depending on the
transformation type
Adding and Deleting Partition Points
Rules and Guidelines for Adding and Deleting Partition Points
- You cannot create a partition point at a source instance.
- You cannot create a partition point at a Sequence Generator
transformation or an unconnected transformation.
- You can add a partition point at any other transformation
provided that no partition point receives input from more than
one pipeline stage.
- You cannot delete a partition point at a Source Qualifier
transformation, a Normalizer transformation for COBOL
sources, or a target instance.
- You cannot delete a partition point at a multiple input group
Custom transformation that is configured to use one thread per
partition.
- You cannot delete a partition point at a multiple input group
transformation that is upstream from a multiple input group
Custom transformation that is configured to use one thread per
partition
- The following partition types have restrictions with dynamic
partitioning:
 Pass-through - When you use dynamic partitioning, if you change
the number of partitions at a partition point, the number of
partitions in each pipeline stage changes.
Key Range - To use key range with dynamic partitioning you must
define a closed range of numbers or date keys. If you use an
open-ended range, the session runs with one partition.
You can add and delete partition points at other
transformations in the pipeline according to the following rules:
- You cannot create partition points at source instances.
- You cannot create partition points at Sequence Generator
transformations or unconnected transformations.
- You can add partition points at any other transformation
provided that no partition point receives input from more than
one pipeline stage.
Note: When you create a custom SQL query to read database
tables and you set database partitioning, the Integration Service
reverts to pass-through partitioning and prints a message in the
session log.
Partitioning File Sources
 The Integration Service creates one connection to the file source
when you configure the session to read with one thread, and it
creates multiple concurrent connections to the file source when
you configure the session to read with multiple threads.
 Use the following types of partitioned file sources:
Flat file - You can configure a session to read flat file, XML, or
COBOL source files.
Command - You can configure a session to use an operating
system command to generate source data rows or generate a file
list.
 When connecting to file sources, you must choose the same
connection type for all partitions.
 You may choose different connection objects as long as each
object is of the same type.
 To specify single- or multi-threaded reading for flat file sources,
configure the source file name property for partitions 2-n.
 To configure for single-threaded reading, pass empty data
through partitions 2-n.
 To configure for multi-threaded reading, leave the source file
name blank for partitions 2-n.
Rules and Guidelines for Partitioning File Sources
Use the following rules and guidelines when you configure a file
source session with multiple partitions:
- Use pass-through partitioning at the source qualifier.
- Use single- or multi-threaded reading with flat file or COBOL
sources.
- Use single-threaded reading with XML sources
- You cannot use multi-threaded reading if the source files are
non-disk files, such as FTP files or WebSphere MQ sources.
44
- If you use a shift-sensitive code page, use multi-threaded
reading if the following conditions are true:
- The file is fixed-width.
- The file is not line sequential.
- You did not enable user-defined shift state in the source
definition.
- To read data from the three flat files concurrently, you must
specify three partitions at the source qualifier. Accept the default
partition type, pass-through.
- If you configure a session for multi-threaded reading, and the
Integration Service cannot create multiple threads to a file
source, it writes a message to the session log and reads the
source with one thread.
- When the Integration Service uses multiple threads to read a
source file, it may not read the rows in the file sequentially. If sort
order is important, configure the session to read the file with a
single thread. For example, sort order may be important if the
mapping contains a sorted Joiner transformation and the file
source is the sort origin.
- You can also use a combination of direct and indirect files to
balance the load.
- Session performance for multi-threaded reading is optimal with
large source files. The load may be unbalanced if the amount of
input data is small.
- You cannot use a command for a file source if the command
generates source data and the session is configured to run on a
grid or is configured with the resume from the last checkpoint
recovery strategy.
Using One Thread to Read a File Source
 When the Integration Service uses one thread to read a file
source, it creates one connection to the source.
 The Integration Service reads the rows in the file or file list
sequentially.
 You can configure single-threaded reading for direct or indirect
file sources in a session:
Reading direct files - You can configure the Integration Service to
read from one or more direct files. If you configure the session
with more than one direct file, the Integration Service creates a
concurrent connection to each file. It does not create multiple
connections to a file.
Reading indirect files - When the Integration Service reads an
indirect file, it reads the file list and then reads the files in the list
sequentially. If the session has more than one file list, the
Integration Service reads the file lists concurrently, and it reads
the files in the list sequentially.
Using Multiple Threads to Read a File Source
 When the Integration Service uses multiple threads to read a
source file, it creates multiple concurrent connections to the
source.
 The Integration Service may or may not read the rows in a file
sequentially.
 You can configure multi-threaded reading for direct or indirect
file sources in a session:
Reading direct files - When the Integration Service reads a direct
file, it creates multiple reader threads to read the file
concurrently. You can configure the Integration Service to read
from one or more direct files. For example, if a session reads
from two files and you create five partitions, the Integration
Service may distribute one file between two partitions and one
file between three partitions.
Reading indirect files - When the Integration Service reads an
indirect file, it creates multiple threads to read the file list
concurrently. It also creates multiple threads to read the files in
the list concurrently. The Integration Service may use more than
one thread to read a single file.
Partitioning Joiner Transformations
 When you create a partition point at the Joiner transformation,
the Workflow Manager sets the partition type to hash auto-
keys when the transformation scope is All Input.
 The Workflow Manager sets the partition type to passthrough
when the transformation scope is Transaction.
 You must create the same number of partitions for the master
and detail source.
 If you configure the Joiner transformation for sorted input, you
can change the partition type to pass-through.
 You can specify only one partition if the pipeline contains the
master source for a Joiner transformation and you do not add a
partition point at the Joiner transformation.
 The Integration Service uses cache partitioning when you create a
partition point at the Joiner transformation.
 When you use partitioning with a Joiner transformation, you can
create multiple partitions for the master and detail source of a
Joiner transformation.
 If you do not create a partition point at the Joiner transformation,
you can create n partitions for the detail source, and one
partition for the master source (1:n).
Note: You cannot add a partition point at the Joiner
transformation when you configure the Joiner transformation
to use the row transformation scope
 When you join data, you can partition data for the master and
detail pipelines in the following ways:
1:n - Use one partition for the master source and multiple
partitions for the detail source. The Integration Service maintains
the sort order because it does not redistribute master data
among partitions.
n:n - Use an equal number of partitions for the master and detail
sources. When you use n:n partitions, the Integration Service
processes multiple partitions concurrently. You may need to
configure the partitions to maintain the sort order depending on
the type of partition you use at the Joiner transformation.
45
Note: When you use 1:n partitions, do not add a partition point
at the Joiner transformation. If you add a partition point at the
Joiner transformation, the Workflow Manager adds an equal
number of partitions to both master and detail pipelines.
Pushdown Optimization
 When you run a session configured for pushdown optimization,
the Integration Service translates the transformation logic into
SQL queries and sends the SQL queries to the database.
 The source or target database executes the SQL queries to
process the transformations.
 The amount of transformation logic you can push to the database
depends on the database, transformation logic, mapping and
session configuration.
 The Integration Service processes all transformation logic that it
cannot push to a database
Pushdown Optimization Types
You can configure the following types of pushdown optimization:
Source-side pushdown optimization
- The Integration Service pushes as much transformation logic as
possible to the source database.
- Integration Service analyzes the mapping from the source to the
target or until it reaches a downstream transformation it cannot
push to the source database.
- Integration Service generates and executes a SELECT statement
based on the transformation logic for each transformation it can
push to the database. - Then, it reads the results of this SQL
query and processes the remaining transformations
Target-side pushdown optimization
- The Integration Service pushes as much transformation logic as
possible to the target database.
- Integration Service analyzes the mapping from the target to the
source or until it reaches an upstream transformation it cannot
push to the target database.
- It generates an INSERT, DELETE, or UPDATE statement based on
the transformation logic for each transformation it can push to
the target database.
Full pushdown optimization
- Integration Service attempts to push all transformation logic to
the target database.
- To use full pushdown optimization, the source and target
databases must be in the same relational database
management system
- If the Integration Service cannot push all transformation logic to
the database, it performs both source-side and target-side
pushdown optimization.
- When you run a session with large quantities of data and full
pushdown optimization, the database server must run a long
transaction.
 - Consider the following database performance issues when you
generate a long transaction:
- A long transaction uses more database resources.
- A long transaction locks the database for longer periods of
time. This reduces database concurrency and increases the
likelihood of deadlock.
- A long transaction increases the likelihood of an unexpected
event.
- To minimize database performance issues for long transactions,
consider using source-side or target-side pushdown optimization
Active and Idle Databases
 During pushdown optimization, the Integration Service pushes
the transformation logic to one database, which is called the
active database.
 A database that does not process transformation logic is called an
idle database.
 For example, a mapping contains two sources that are joined by a
Joiner transformation.
 If the session is configured for source-side pushdown
optimization, the Integration Service pushes the Joiner
transformation logic to the source in the detail pipeline, which is
the active database.
 The source in the master pipeline is the idle database because it
does not process transformation logic.
 The Integration Service uses the following criteria to determine
which database is active or idle:
- When using full pushdown optimization, the target database is
active and the source database is idle.
- In sessions that contain a Lookup transformation, the source or
target database is active, and the lookup database is idle.
- In sessions that contain a Joiner transformation, the source in
the detail pipeline is active, and the source in the master pipeline
is idle.
- In sessions that contain a Union transformation, the source in
the first input group is active. The sources in other input groups
are idle
Pushdown Compatibility
 To push a transformation with multiple connections to a
database, the connections must be pushdown compatible.
 Connections are pushdown compatible if they connect to
databases on the same database management system and the
Integration Service can identify the database tables that the
connections access.
 The following transformations can have multiple connections:
Joiner - The Joiner transformation can join data from multiple
source connections.
46
Union - The Union transformation can merge data from multiple
source connections.
Lookup - The connection for the Lookup transformation can differ
from the source connection.
Target - The target connection can differ from the source
connection.
 Each connection object is pushdown compatible with itself.
 If you configure a session to use the same connection object for
the source and target connections, the Integration Service can
push the transformation logic to the source or target database
Error Handling, Logging, and Recovery
 The Integration Service and database process error handling,
logging, and recovery differently.
Error Handling
 When the Integration Service pushes transformation logic to the
database, it cannot track errors that occur in the database.
 As a result, it handles errors differently than when it processes
the transformations in the session.
 When the Integration Service runs a session configured for full
pushdown optimization and an error occurs, the database
handles the errors.
 When the database handles errors, the Integration Service does
not write reject rows to the reject file.
Logging
 When the Integration Service pushes transformation logic to the
database, it cannot trace all the events that occur inside the
database server.
 The statistics the Integration Service can trace depend on the
type of pushdown optimization.
 When you push transformation logic to the database, the
Integration Service generates a session log with the following
differences:
- The session log does not contain details for transformations
processed by the database.
- The session log does not contain the thread busy percentage
when the session is configured for full pushdown optimization.
- The session log contains the number of loaded rows when the
session is configured for source-side, targetside, and full
pushdown optimization.
- The session log does not contain the number of rows read from
the source when the Integration Service uses full pushdown
optimization and pushes all transformation logic to the database.
- The session log contains the number of rows read from each
source when the Integration Service uses sourceside pushdown
optimization.
Recovery
 If you configure a session for full pushdown optimization and the
session fails, the Integration Service cannot perform incremental
recovery because the database processes the transformations.
 Instead, the database rolls back the transactions.
 If the database server fails, it rolls back transactions when it
restarts.
 If the Integration Service fails, the database server rolls back the
transaction.
 If the failure occurs while the Integration Service is creating
temporary sequence objects or views in the database, which is
before any rows have been processed, the Integration Service
runs the generated SQL on the database again.
 If the failure occurs before the database processes all rows, the
Integration Service performs the following tasks:
1. If applicable, the Integration Service drops and recreates
temporary view or sequence objects in the database to ensure
duplicate values are not produced.
2. The Integration Service runs the generated SQL on the
database again.
 If the failure occurs while the Integration Service is dropping the
temporary view or sequence objects from the database, which
is after all rows are processed, the Integration Service tries to
drop the temporary objects again
Using the $$PushdownConfig Mapping Parameter
 Depending on the database workload, you might want to use
source-side, target-side, or full pushdown optimization at
different times.
 For example, use source-side or target-side pushdown
optimization during the peak hours of the day, but use full
pushdown optimization from midnight until 2 a.m. when
database activity is low.
 To use different pushdown optimization configurations at
different times, use the $$PushdownConfig mapping parameter.
 The settings in the $$PushdownConfig parameter override the
pushdown optimization settings in the session properties.
Partitioning
 You can push a session with multiple partitions to a database if
the partition types are pass-through partitioning or key range
partitioning
Rules and Guidelines for Pushdown Optimization and
Transformations
 Use the following rules and guidelines when you configure the
Integration Service to push transformation logic to a database.
 The Integration Service processes the transformation logic if any
of the following conditions are true:
- The transformation logic updates a mapping variable and saves
it to the repository database.
47
- The transformation contains a variable port.
- The transformation meets all of the following criteria:
- Is not a Sorter transformation, Union transformation, or
target.
- Is pushed to Microsoft SQL Server, Sybase, or Teradata.
- Is downstream from a Sorter transformation, which is
downstream from a Union transformation or contains a distinct
sort.
- The session is configured to override the default values of input
or output ports.
- The database does not have an equivalent operator, variable, or
function that is used in an expression in the transformation.
- The mapping contains too many branches. When you branch a
pipeline, the SQL statement required to represent the mapping
logic becomes more complex. The Integration Service cannot
generate SQL for a mapping that contains more than 64 two-way
branches, 43 three-way branches, or 32 four-way branches. If the
mapping branches exceed these limitations, the Integration
Service processes the downstream transformations.
 The Integration Service processes all transformations in the
mapping if any of the following conditions are true:
- The session is a data profiling or debug session.
- The session is configured to log row errors.
Row Error Logging
 You can log row errors into relational tables or flat files.
 When you enable error logging, the Integration Service creates
the error tables or an error log file the first time it runs the
session. Error logs are cumulative.
 If the error logs exist, the Integration Service appends error data
to the existing error logs
 You can log source row data from flat file or relational sources
but you cannot log row errors from XML file sources.
 You can view the XML source errors in the session log
 By default, the Integration Service logs transformation errors in
the session log and reject rows in the reject file.
 When you enable error logging, the Integration Service does
not generate a reject file or write dropped rows to the session
log.
 Without a reject file, the Integration Service does not log
Transaction Control transformation rollback or commit errors.
 If you want to write rows to the session log in addition to the row
error log, you can enable verbose data tracing.
Note: When you log row errors, session performance may
decrease because the Integration Service processes one row at
a time instead of a block of rows at once
Understanding the Error Log Tables
 When you choose relational database error logging, the
Integration Service creates the following error tables the first
time you run a session:
PMERR_DATA - Stores data and metadata about a
transformation row error and its corresponding source row.
PMERR_MSG - Stores metadata about an error and the error
message.
PMERR_SESS - Stores metadata about the session.
PMERR_TRANS - Stores metadata about the source and
transformation ports, such as name and datatype, when a
transformation error occurs.
 If the error tables exist for a session, the Integration Service
appends row errors to these tables.
 Relational database error logging lets you collect row errors
from multiple sessions in one set of error tables.
 You can specify a prefix for the error tables. The error table
names can have up to eleven characters.
 The Integration Service creates the error tables without
specifying primary and foreign keys. However, you can specify
key columns
Workflow Recovery
 You can recover a workflow if the Integration Service can access
the workflow state of operation.
 The workflow state of operation includes the status of tasks in
the workflow and workflow variable values.
 The Integration Service stores the state in memory or on disk,
based on how you configure the workflow:
 Enable recovery - When you enable a workflow for recovery, the
Integration Service saves the workflow state of operation in a
shared location. You can recover the workflow if it terminates,
stops, or aborts. The workflow does not have to be running.
 Suspend - When you configure a workflow to suspend on error,
the Integration Service stores the workflow state of operation
in memory. You can recover the suspended workflow if a task
fails. You can fix the task error and recover the workflow
 The Integration Service recovers tasks in the workflow based on
the recovery strategy of the task.
 By default, the recovery strategy for Session and Command
tasks is to fail the task and continue running the workflow.
 You can configure the recovery strategy for Session and
Command tasks.
 The strategy for all other tasks is to restart the task
Workflow State of Operation
 When you enable a workflow for recovery, the Integration Service
stores the workflow state of operation in the shared location,
$PMStorageDir.
 The Integration Service can restore the state of operation to
recover a stopped, aborted, or terminated workflow.
48
 The workflow state of operation includes the following
information:
- Active service requests
- Completed and running task status
- Workflow variable values
 When you run concurrent workflows, the Integration Service
appends the instance name or the workflow run ID to the
workflow recovery storage file in $PMStorageDir.
 When you enable a workflow for recovery the Integration Service
does not store the session state of operation by default.
Session State of Operation
 When you configure the session recovery strategy to resume
from the last checkpoint, the Integration Service stores the
session state of operation in the shared location,
$PMStorageDir.
 The Integration Service also saves relational target recovery
information in target database tables.
 When the Integration Service performs recovery, it restores the
state of operation to recover the session from the point of
interruption. It uses the target recovery data to determine how
to recover the target tables
 You can configure the session to save the session state of
operation even if you do not save the workflow state of
operation.
 You can recover the session, or you can recover the workflow
from the session.
 The session state of operation includes the following information:
Source - If the output from a source is not deterministic and
repeatable, the Integration Service saves the result from the SQL
query to a shared storage file in $PMStorageDir.
Transformation - The Integration Service creates checkpoints in
$PMStorageDir to determine where to start processing the
pipeline when it runs a recovery session. When you run a session
with an incremental Aggregator transformation, the Integration
Service creates a backup of the Aggregator cache files in
$PMCacheDir at the beginning of a session run. The Integration
Service promotes the backup cache to the initial cache at the
beginning of a session recovery run.
Relational target recovery data - The Integration Service writes
recovery information to recovery tables in the target database
to determine the last row committed to the target when the
session was interrupted
Target Recovery Tables
 When the Integration Service runs a session that has a resume
recovery strategy, it writes to recovery tables on the target
database system.
 When the Integration Service recovers the session, it uses
information in the recovery tables to determine where to begin
loading data to target tables.
The Integration Service creates the following recovery tables in
the target database:
PM_RECOVERY - Contains target load information for the session
run. The Integration Service removes the information from this
table after each successful session and initializes the information
at the beginning of subsequent sessions.
PM_TGT_RUN_ID - Contains information the Integration Service
uses to identify each target on the database. The information
remains in the table between session runs. If you manually create
this table, you must create a row and enter a value other than
zero for LAST_TGT_RUN_ID to ensure that the session recovers
successfully.
PM_REC_STATE - Contains information the Integration Service
uses to determine if it needs to write messages to the target
table during recovery for a real-time session.
Task Recovery Strategies
Each task in a workflow has a recovery strategy. When the
Integration Service recovers a workflow, it recovers tasks based
on the recovery strategy:
Restart task - When the Integration Service recovers a workflow,
it restarts each recoverable task that is configured with a restart
strategy. You can configure Session and Command tasks with a
restart recovery strategy. All other tasks have a restart recovery
strategy by default.
Fail task and continue workflow - When the Integration Service
recovers a workflow, it does not recover the task. The task status
becomes failed, and the Integration Service continues running
the workflow. Configure a fail recovery strategy if you want to
complete the workflow, but you do not want to recover the task.
You can configure Session and Command tasks with the fail task
and continue workflow recovery strategy.
Resume from the last checkpoint - The Integration Service
recovers a stopped, aborted, or terminated session from the last
checkpoint. You can configure a Session task with a resume
strategy.
49
Automatically Recovering Terminated Tasks
 When you have the high availability option, you can configure
automatic recovery of terminated tasks.
 When you enable automatic task recovery, the Integration
Service recovers terminated Session and Command tasks
without user intervention if the workflow is still running.
 You configure the number of times the Integration Service
attempts to recover the task.
 Enable automatic task recovery in the workflow properties
Resuming Sessions
 When the Integration Service resumes a session, the recovery
session must produce the same data as the original session.
 The session is not valid if you configure recovery to resume from
the last checkpoint, but the session cannot produce repeatable
data
 When you recover a session from the last checkpoint, the
Integration Service restores the session state of operation to
determine the type of recovery it can perform:
Incremental - The Integration Service starts processing data at
the point of interruption. It does not read or transform rows that
it processed before the interruption. By default, the Integration
Service attempts to perform incremental recovery.
Full - The Integration Service reads all source rows again and
performs all transformation logic if it cannot perform incremental
recovery. The Integration Service begins writing to the target at
the last commit point. If any session component requires full
recovery, the Integration Service performs full recovery on the
session
  The mapping contains a Union and Custom transformation that
never produce repeatable data.
 The Lookup transformation produces repeatable data when it
receives repeatable data.
 Therefore, the target does not receive repeatable data and you
cannot configure the session to resume recovery.
 You can modify the mapping to enable resume recovery.
 Add a Sorter transformation configured for distinct output rows
immediately after the transformations that never output
repeatable data. Add the Sorter transformation after the
Custom transformation.

Working with Repeatable Data

 When you configure recovery to resume from the last checkpoint,
the recovery session must be able to produce the same data in  The Lookup transformation produces repeatable data because it
the same order as the original session. receives repeatable data from the Sorter transformation.
 When you validate a session, the Workflow Manager verifies that  The following table describes when transformations produce
the transformations are configured to produce repeatable and repeatable data:
deterministic data.
 The session is not valid if you configure recovery to resume from
the last checkpoint, but the transformations are not configured
for repeatable and deterministic data.
 Session data is repeatable when all targets receive repeatable
data from the following mapping objects:
Source - The output data from the source is repeatable between
the original run and the recovery run.
Transformation - The output data from each transformation to
the target is repeatable.

Configuring a Mapping for Recovery

 You can configure a mapping to enable transformations in the
session to produce the same data between the session and
recovery run.
 When a mapping contains a transformation that never produces
repeatable data, you can add a transformation that always
produces repeatable data immediately after it.

 The mapping contains two Source Qualifier transformations that

produce repeatable data.

50

 You can configure the Output is Repeatable and Output is
Deterministic properties for the following transformations, or
you can add a transformation that produces repeatable data
immediately after these transformations:
- Application Source Qualifier
- Custom
- External Procedure
- Source Qualifier, relational
- Stored Procedure
Steps to Recover Workflows and Tasks
 You can use one of the following methods to recover a workflow
or task:
Recover a workflow - Continue processing the workflow from the
point of interruption.
Recover a session - Recover a session but not the rest of the
workflow.
Recover a workflow from a session - Recover a session and
continue processing a workflow
 If you want to restart a workflow or task without recovery, you
can restart the workflow or task in cold start mode.
Rules and Guidelines for Session Recovery
 Configuring Recovery to Resume from the Last Checkpoint
Use the following rules and guidelines when configuring recovery
to resume from last checkpoint:
- You must use pass-through partitioning for each transformation.
- You cannot configure recovery to resume from the last
checkpoint for a session that runs on a grid.
- When you configure a session for full pushdown optimization,
the Integration Service runs the session on the database. As a
51
result, it cannot perform incremental recovery if the session fails.
When you perform recovery for sessions that contain SQL
overrides, the Integration Service must drop and recreate views.
- When you modify a workflow or session between the
interrupted run and the recovery run, you might get unexpected
results. The Integration Service does not prevent recovery for a
modified workflow. The recovery workflow or session log displays
a message when the workflow or the task is modified since last
run.
- The pre-session command and pre-SQL commands run only
once when you resume a session from the last checkpoint. If a
pre- or post- command or SQL command fails, the Integration
Service runs the command again during recovery. Design the
commands so you can rerun them.
- You cannot configure a session to resume if it writes to a
relational target in bulk mode
Unrecoverable Workflows or Tasks
 In some cases, the Integration Service cannot recover a workflow
or task. You cannot recover a workflow or task under the
following circumstances:
You change the number of partitions - If you change the number
of partitions after a session fails, the recovery session fails.
The interrupted task has a fail recovery strategy - If you
configure a Command or Session recovery to fail and continue
the workflow recovery, the task is not recoverable.
Recovery storage file is missing - The Integration Service fails the
recovery session or workflow if the recovery storage file is
missing from $PMStorageDir or if the definition of $PMStorageDir
changes between the original and recovery run.
Recovery table is empty or missing from the target database -
The Integration Service fails a recovery session under the
following circumstances:
- You deleted the table after the Integration Service created it.
- The session enabled for recovery failed immediately after the
Integration Service removed the recovery information from the
table.
 You might get inconsistent data if you perform recovery under
the following circumstances:
The sources or targets change after the initial session - If you
drop or create indexes or edit data in the source or target tables
before recovering a session, the Integration Service may return
missing or repeat rows.
The source or target code pages change after the initial session
failure - If you change the source or target code page, the
Integration Service might return incorrect data. You can perform
recovery if the code pages are two-way compatible with the
original code pages.
Stopping and Aborting
 When you stop a workflow, the Integration Service tries to stop
all the tasks that are currently running in the workflow
 If it cannot stop the workflow, you need to abort the workflow.
 The Integration Service can stop the following tasks completely:
- Session
- Command
- Timer
- Event-Wait
- Worklet
 When you stop a Command task that contains multiple
commands, the Integration Service finishes executing the
current command and does not run the rest of the commands.
 The Integration Service cannot stop tasks such as the Email task .
 For example, if the Integration Service has already started
sending an email when you issue the stop command, the
Integration Service finishes sending the email before it stops
running the workflow.
 The Integration Service aborts the workflow if the Repository
Service process shuts down
Concurrent Workflows
Use the following rules and guidelines for concurrent workflows:
- You cannot reference workflow run instances in parameter files.
To use separate parameters for each instance, you must configure
different parameter files.
- If you use the same cache file name for more than one
concurrent workflow instance, each workflow instance will be
valid. However, sessions will fail if conflicts occur writing to the
cache.
- You can use pmcmd to restart concurrent workflows by run ID
or instance name.
- If you configure multiple instances of a workflow and you
schedule the workflow, the Integration Service runs all instances
at the scheduled time. You cannot run instances on separate
schedules.
- Configure a worklet to run concurrently on the worklet General
tab.
- You must enable a worklet to run concurrently if the parent
workflow is enabled to run concurrently. Otherwise the workflow
is invalid.
- You can enable a worklet to run concurrently and place it in two
non-concurrent workflows. The Integration Service can run the
two worklets concurrently.
- Two workflows enabled to run concurrently can run the same
worklet. One workflow can run two instances of the same
worklet if the worklet has no persisted variables.
- A session in a worklet can run concurrently with a session in
another worklet of the same instance name when the session
does not contain persisted variables.
 The following transformations have restrictions with concurrent
workflows:
52
Aggregator transformation - You cannot use an incremental
aggregation in a concurrent workflow. The session fails.
Lookup transformation - Use the following rules and guidelines
for Lookup transformations in concurrent workflows:
- You can use static or dynamic lookup cache with concurrent
workflows.
- When the cache is non-persistent, the Integration Service adds
the workflow run ID as a prefix to the cache file name.
- When the cache is an unnamed persistent cache, the Integration
Service adds the run instance name as a prefix to the cache file
name.
- If the cache is a dynamic, unnamed, persistent cache and the
current workflow is configured to allow concurrent runs with the
same instance name, the session fails.
- If the lookup cache name is parameterized, the Integration
Service names the cache file with the parameter value. Pass a
different file name for each run instance.
Sequence Generator transformation - To avoid generating the
same set of sequence numbers for concurrent workflows,
configure the number of cached values in the Sequence
Generator transformation.
Grid Processing
Rules and Guidelines for Configuring a Workflow or Session to
Run on a Grid
- To run sessions over the grid, verify that the operating system
and bit mode is the same for each node of the grid. A session
might not run on the grid if the nodes run on different operating
systems or bit modes.
- If you override a service process variable, ensure that the
Integration Service can access input files, caches, logs, storage
and temporary directories, and source and target file directories.
- To ensure that a Session, Command, or predefined Event-Wait
task runs on a particular node, configure the Integration Service
to check resources and specify a resource requirement for a the
task.
- To ensure that session threads for a mapping object run on a
particular node, configure the Integration Service to check
resources and specify a resource requirement for the object.
- When you run a session that creates cache files, configure the
root and cache directory to use a shared location to ensure
consistency between cache files.
- Ensure the Integration Service builds the cache in a shared
location when you add a partition point at a Joiner
transformation and the transformation is configured for 1:n
partitioning. The cache for the Detail pipeline must be shared.
- Ensure the Integration Service builds the cache in a shared
location when you add a partition point at a Lookup
transformation, and the partition type is not hash auto-keys.
- When you run a session that uses dynamic partitioning, and you
want to distribute session threads across all nodes in the grid,
 configure dynamic partitioning for the session to use the “Based
on number of nodes in the grid” method.
- You cannot run a debug session on a grid.
- You cannot configure a resume recovery strategy for a session
that you run on a grid.
- Configure the session to run on a grid when you work with
sessions that take a long time to run.
- Configure the workflow to run on a grid when you have multiple
concurrent sessions.
- You can run a persistent profile session on a grid, but you
cannot run a temporary profile session on a grid.
- When you use a Sequence Generator transformation, increase
the number of cached values to reduce the communication
required between the master and worker DTM processes and the
repository.
- To ensure that the Log Viewer can accurately order log events
when you run a workflow or session on a grid, use time
synchronization software to ensure that the nodes of a grid use a
synchronized date/time.
- If the workflow uses an Email task in a Windows environment,
configure the same Microsoft Outlook profile on each node to
ensure the Email task can run.
Workflow Variables
 Use the following types of workflow variables:
Predefined workflow variables - The Workflow Manager
provides predefined workflow variables for tasks within a
workflow.
User-defined workflow variables - You create user-defined
workflow variables when you create a workflow.
 Use workflow variables when you configure the following types
of tasks:
Assignment tasks - Use an Assignment task to assign a value to a
user-defined workflow variable. For example, you can increment
a user-defined counter variable by setting the variable to its
current value plus 1.
Decision tasks - Decision tasks determine how the Integration
Service runs a workflow. For example, use the Status variable to
run a second session only if the first session completes
successfully.
Links - Links connect each workflow task. Use workflow variables
in links to create branches in the workflow. For example, after a
Decision task, you can create one link to follow when the decision
condition evaluates to true, and another link to follow when the
decision condition evaluates to false.
Timer tasks - Timer tasks specify when the Integration Service
begins to run the next task in the workflow. Use a user-defined
date/time variable to specify the time the Integration Service
starts to run the next task
Print 161-163
53
 Workflow Variable Start and Current Values Conceptually, the
Integration Service holds two different values for a workflow
variable during a workflow run:
- Start value of a workflow variable
- Current value of a workflow variable
 The Integration Service looks for the start value of a variable in
the following order:
1. Value in parameter file
2. Value saved in the repository (if the variable is persistent)
3. User-specified default value
4. Datatype default value
Parameters and Variables in Sessions
 Use user-defined session parameters in session or workflow
properties and define the values in a parameter file
 In the parameter file, folder and session names are case
sensitive
 User-defined session parameters do not have default values, so
you must define them in a parameter file.
 If the Integration Service cannot find a value for a user-defined
session parameter, it fails the session, takes an empty string as
the default value, or fails to expand the parameter at run time
 You can run a session with different parameter files when you use
pmcmd to start a session.
 The parameter file you set with pmcmd overrides the parameter
file in the session or workflow properties
 You cannot define built-in session parameter values in the
parameter file. The Integration Service expands these
parameters when the session runs.
Rules and Guidelines for Creating File Parameters and Database
Connection Parameters
- When you define the parameter file as a resource for a node,
verify the Integration Service runs the session on a node that can
access the parameter file. Define the resource for the node,
configure the Integration Service to check resources, and edit the
session to require the resource.
- When you create a file parameter, use alphanumeric and
underscore characters. For example, to name a source file
parameter, use $InputFileName, such as $InputFile_Data.
- All session file parameters of a particular type must have
distinct names. For example, if you create two source file
parameters, you might name them $SourceFileAccts and
$SourceFilePrices.
- When you define the parameter in the file, you can reference
any directory local to the Integration Service.
 - Use a parameter to define the location of a file. Clear the entry
in the session properties that define the file location. Enter the
full path of the file in the parameter file.
- You can change the parameter value in the parameter file
between session runs, or you can create multiple parameter files.
If you use multiple parameter files, use the pmcmd Startworkflow
command with the -paramfile or -localparamfile options to
specify which parameter file to use.
Use the following rules and guidelines when you create database
connection parameters:
- You can change connections for relational sources, targets,
lookups, and stored procedures.
- When you define the parameter, you can reference any
database connection in the repository.
- Use the same $DBConnection parameter for more than one
connection in a session.
Mapping Parameters and Variables in Sessions
 If you use mapping variables in a session, you can clear any of the
variable values saved in the repository by editing the session.
 When you clear the variable values, the Integration Service uses
the values in the parameter file the next time you run a session.
 If the session does not use a parameter file, the Integration
Service uses the values assigned in the pre-session variable
assignment.
 If there are no assigned values, the Integration Service uses the
initial values defined in the mapping.
Assigning Parameter and Variable Values in a Session
 You can update the values of certain parameters and variables
before or after a non-reusable session runs
 Note: You cannot assign parameters and variables in reusable
sessions
 You can update the following types of parameters and variables
before or after a session runs:
Pre-session variable assignment - You can update mapping
parameters, mapping variables, and session parameters before a
session runs. You can assign these parameters and variables the
values of workflow or worklet variables in the parent workflow or
worklet. Therefore, if a session is in a worklet within a workflow,
you can assign values from the worklet variables, but not the
workflow variables.
 You cannot update mapplet variables in the pre-session variable
assignment.
Post-session on success variable assignment - You can update
workflow or worklet variables in the parent workflow or worklet
after the session completes successfully. You can assign these
variables the values of mapping parameters and variables.
Post-session on failure variable assignment - You can update
workflow or worklet variables in the parent workflow or worklet
54
when the session fails. You can assign these variables the values
of mapping parameters and variables.
Passing Parameter and Variable Values between Sessions
To pass the mapping variable value from s_NewCustomers to
s_MergeCustomers, complete the following steps:
1. Configure the mapping associated with session
s_NewCustomers to use a mapping variable, for example, $
$Count1.
2. Configure the mapping associated with session
s_MergeCustomers to use a mapping variable, for example, $
$Count2.
3. Configure the workflow to use a user-defined workflow
variable, for example, $$PassCountValue.
4. Configure session s_NewCustomers to assign the value of
mapping variable $$Count1 to workflow variable $
$PassCountValue after the session completes successfully.
5. Configure session s_MergeCustomers to assign the value of
workflow variable $$PassCountValue to mapping variable $
$Count2 before the session starts.
Parameter Files
 A parameter file is a list of parameters and variables and their
associated values.
 These values define properties for a service, service process,
workflow, worklet, or session.
 The Integration Service applies these values when you run a
workflow or session that uses the parameter file
 The Integration Service reads the parameter file at the start of
the workflow or session to determine the start values for the
parameters and variables defined in the file
 Consider the following information when you use parameter files:
Types of parameters and variables - You can define different
types of parameters and variables in a parameter file. These
include service variables, service process variables, workflow and
worklet variables, session parameters, and mapping parameters
and variables.
Properties you can set in parameter files - Use parameters and
variables to define many properties in the Designer and
Workflow Manager. For example, you can enter a session
parameter as the update override for a relational target instance,
and set this parameter to the UPDATE statement in the
parameter file. The Integration Service expands the parameter
when the session runs.
Parameter file structure - Assign a value for a parameter or
variable in the parameter file by entering the parameter or
variable name and value on a single line in the form name=value.
Groups of parameters and variables must be preceded by a
heading that identifies the service, service process, workflow,
worklet, or session to which the parameters or variables apply.
 Parameter file location - Specify the parameter file to use for a
workflow or session. You can enter the parameter file name and
directory in the workflow or session properties or in the pmcmd
command line.
Parameter and Variable Types
 A parameter file can contain different types of parameters and
variables. When you run a session or workflow that uses a
parameter file, the Integration Service reads the parameter file
and expands the parameters and variables defined in the file.
 You can define the following types of parameter and variable in a
parameter file:
Service variables - Define general properties for the Integration
Service such as email addresses, log file counts, and error
thresholds. $PMSuccessEmailUser, $PMSessionLogCount, and
$PMSessionErrorThreshold are examples of service variables. The
service variable values you define in the parameter file override
the values that are set in the Administrator tool.
Service process variables - Define the directories for Integration
Service files for each Integration Service process. $PMRootDir,
$PMSessionLogDir, and $PMBadFileDir are examples of service
process variables. The service process variable values you define
in the parameter file override the values that are set in the
Administrator tool. If the Integration Service uses operating
system profiles, the operating system user specified in the
operating system profile must have access to the directories you
define for the service process variables.
Workflow variables - Evaluate task conditions and record
information in a workflow. For example, you can use a workflow
variable in a Decision task to determine whether the previous
task ran properly. In a workflow, $TaskName.PrevTaskStatus is a
predefined workflow variable and $$VariableName is a user-
defined workflow variable.
Worklet variables - Evaluate task conditions and record
information in a worklet. You can use predefined worklet
variables in a parent workflow, but you cannot use workflow
variables from the parent workflow in a worklet. In a worklet,
$TaskName.PrevTaskStatus is a predefined worklet variable and $
$VariableName is a user-defined worklet variable.
Session parameters - Define values that can change from session
to session, such as database connections or file names.
$PMSessionLogFile and $ParamName are user-defined session
parameters.
Mapping parameters - Define values that remain constant
throughout a session, such as state sales tax rates. When
declared in a mapping or mapplet, $$ParameterName is a user-
defined mapping parameter.
Mapping variables - Define values that can change during a
session. The Integration Service saves the value of a mapping
variable to the repository at the end of each successful session
run and uses that value the next time you run the session. When
55
declared in a mapping or mapplet, $$VariableName is a mapping
variable.
You cannot define the following types of variables in a parameter
file:
$Source and $Target connection variables - Define the database
location for a relational source, relational target, lookup table, or
stored procedure.
Email variables - Define session information in an email message
such as the number of rows loaded, the session completion time,
and read and write statistics.
Local variables - Temporarily store data in variable ports in
Aggregator, Expression, and Rank transformations.
Built-in variables - Variables that return run-time or system
information, such as Integration Service name or system date.
Transaction control variables - Define conditions to commit or
rollback transactions during the processing of database rows.
ABAP program variables - Represent SAP structures, fields in SAP
structures, or values in the ABAP program.
Parameter File Structure
Warning: The Integration Service uses the period character (.) to
qualify folder, workflow, and session names when you run a
workflow with a parameter file. If the folder name contains a
period (.), the Integration Service cannot qualify the names
properly and fails the workflow.
 You can define parameters and variables in any section in the
parameter file.
 If you define a service or service process variable in a workflow,
worklet, or session section, the variable applies to the service
process that runs the task.
 Similarly, if you define a workflow variable in a session section,
the value of the workflow variable applies only when the session
runs
 The following table describes the parameter file headings that
define each section in the parameter file and the scope of the
parameters and variables that you define in each section:

 If you specify the same heading more than once in a parameter
file, the Integration Service uses the information in the section
below the first heading and ignores the information in the
sections below subsequent identical headings.
 If you define the same parameter or variable in multiple
sections in the parameter file, the parameter or variable with
the smallest scope takes precedence over parameters or
variables with larger scope. For example, a parameter file
contains the following sections:
[HET_TGTS.WF:wf_TGTS_ASC_ORDR]
$DBConnection_ora=Ora2
[HET_TGTS.WF:wf_TGTS_ASC_ORDR.ST:s_TGTS_ASC_ORDR]
$DBConnection_ora=Ora3
In session s_TGTS_ASC_ORDR, the value for session parameter
$DBConnection_ora is “Ora3.” In all other sessions in the
workflow, it is “Ora2.”
Using Variables to Specify Session Parameter Files
 When you define a workflow parameter file and a session
parameter file for a session within the workflow, the Integration
Service uses the workflow parameter file, and ignores the
session parameter file.
 To use a variable to define the session parameter file name, you
must define the session parameter file name and set
$PMMergeSessParamFile=TRUE in the workflow parameter file.
 The $PMMergeSessParamFile property causes the Integration
Service to read both the session and workflow parameter files.
For example, you configured a workflow to run two concurrent
instances that contains three sessions:
56
 Create workflow variables to store the session parameter file
names. For example, you create user-defined workflow variables
$$s_1ParamFileName, $$s_2ParamFileName, and $
$s_3ParamFileName. In the session properties for each session,
set the parameter file name to a workflow variable:
 If you use a variable as the session parameter file name, and you
define the same parameter or variable in both the session and
workflow parameter files, the Integration Service sets parameter
and variable values according to the following rules:
- When a parameter or variable is defined in the same section
of the workflow and session parameter files, the Integration
Service uses the value in the workflow parameter file.
- When a parameter or variable is defined in both the session
section of the session parameter file and the workflow section
of the workflow parameter file, the Integration Service uses the
value in the session parameter file.
Using a Parameter File with pmcmd
 The -localparamfile option defines a parameter file on a local
machine that you can reference when you do not have access to
parameter files on the Integration Service machine.
 The following command starts workflowA using the parameter
file, myfile.txt:
pmcmd startworkflow -uv USERNAME -pv PASSWORD -s
SALES:6258 -f east -w wSalesAvg -paramfile '\
$PMRootDir/myfile.txt' workflowA
The following command starts taskA using the parameter file,
myfile.txt:
pmcmd starttask -uv USERNAME -pv PASSWORD -s SALES:6258 -f
east -w wSalesAvg -paramfile '\$PMRootDir/myfile.txt' taskA
Guidelines for Creating Parameter Files
List all session parameters - Session parameters do not have
default values. If the Integration Service cannot find a value for
a session parameter, it may fail the session, take an empty
string as the default value, or fail to expand the parameter at
run time. Session parameter names are not case sensitive.
List all necessary mapping parameters and variables - Mapping
parameter and variable values become start values for
parameters and variables in a mapping. Mapping parameter and
variable names are not case sensitive.
Enter folder names for non-unique session names - When a
session name exists more than once in a repository, enter the
folder name to indicate the location of the session.
Precede parameters and variables in mapplets with the mapplet
name - Use the following format:
mapplet_name.parameter_name=value
mapplet2_name.variable_name=value
Use multiple parameter files - You assign parameter files to
workflows, worklets, and sessions individually. You can specify
the same parameter file for all of these tasks or create multiple
parameter files.
When defining parameter values, do not use unnecessary line
breaks or spaces - The Integration Service interprets additional
spaces as part of a parameter name or value. Use correct date
formats for datetime values. Use the following date formats for
datetime values:
- MM/DD/RR
- MM/DD/YYYY
- MM/DD/RR HH24:MI
- MM/DD/YYYY HH24:MI
- MM/DD/RR HH24:MI:SS
- MM/DD/YYYY HH24:MI:SS
- MM/DD/RR HH24:MI:SS.MS
- MM/DD/YYYY HH24:MI:SS.MS
- MM/DD/RR HH24:MI:SS.US
- MM/DD/YYYY HH24:MI:SS.US
- MM/DD/RR HH24:MI:SS.NS
- MM/DD/YYYY HH24:MI:SS.NS
You can use the following separators: dash (-), slash (/), backslash
(\), colon (:), period (.), and space. The Integration Service ignores
extra spaces. You cannot use one- or three-digit values for year or
the “HH12” format for hour.
Do not enclose parameter or variable values in quotes - The
Integration Service interprets everything after the first equals
sign as part of the value.
Use a parameter or variable value of the proper length for the
error log table name prefix - If you use a parameter or variable
57
for the error log table name prefix, do not specify a prefix that
exceeds 19 characters when naming Oracle, Sybase, or Teradata
error log tables. The error table names can have up to 11
characters, and Oracle, Sybase, and Teradata databases have a
maximum length of 30 characters for table names. The parameter
or variable name can exceed 19 characters
Troubleshooting Parameters and Parameter Files
I have a section in a parameter file for a session, but the
Integration Service does not seem to read it.
 Make sure to enter folder and session names as they appear in
the Workflow Manager. Also, use the appropriate prefix for all
user-defined session parameters.
I am trying to use a source file parameter to specify a source file
and location, but the Integration Service cannot find the source
file.
 Make sure to clear the source file directory in the session
properties. The Integration Service concatenates the source file
directory with the source file name to locate the source file.
Also, make sure to enter a directory local to the Integration
Service and to use the appropriate delimiter for the operating
system.
I am trying to run a workflow with a parameter file and one of
the sessions keeps failing.
 The session might contain a parameter that is not listed in the
parameter file. The Integration Service uses the parameter file
to start all sessions in the workflow. Check the session
properties, and then verify that all session parameters are
defined correctly in the parameter file.
I ran a workflow or session that uses a parameter file, and it
failed. What parameter and variable values does the Integration
Service use during the recovery run?
 For service variables, service process variables, session
parameters, and mapping parameters, the Integration Service
uses the values specified in the parameter file, if they exist. If
values are not specified in the parameter file, then the
Integration Service uses the value stored in the recovery
storage file. For workflow, worklet, and mapping variables, the
Integration Service always uses the value stored in the
recovery storage file.
Tips for Parameters and Parameter Files
Use a single parameter file to group parameter information for
related sessions.
 When sessions are likely to use the same database connection or
directory, you might want to include them in the same
parameter file. When connections or directories change, you
can update information for all sessions by editing one parameter
file.

Use pmcmd and multiple parameter files for sessions with
regular cycles
 Sometimes you reuse session parameters in a cycle. For example,
you might run a session against a sales database everyday, but
run the same session against sales and marketing databases
once a week. You can create separate parameter files for each
session run. Instead of changing the parameter file in the
session properties each time you run the weekly session, use
pmcmd to specify the parameter file to use when you start the
session.
Use reject file and session log parameters in conjunction with
target file or target database connection parameters.
 When you use a target file or target database connection
parameter with a session, you can keep track of reject files by
using a reject file parameter. You can also use the session log
parameter to write the session log to the target machine.
Use a resource to verify the session runs on a node that has
access to the parameter file.
 In the Administrator tool, you can define a file resource for each
node that has access to the parameter file and configure the
Integration Service to check resources. Then, edit the session
that uses the parameter file and assign the resource. When you
run the workflow, the Integration Service runs the session with
the required resource on a node that has the resource available.
You can override initial values of workflow variables for a
session by defining them in a session section.
 If a workflow contains an Assignment task that changes the value
of a workflow variable, the next session in the workflow uses
the latest value of the variable as the initial value for the
session. To override the initial value for the session, define a
new value for the variable in the session section of the
parameter file.
You can define parameters and variables using other
parameters and variables.
 For example, in the parameter file, you can define session
parameter $PMSessionLogFile using a service process variable
as follows:
$PMSessionLogFile=$PMSessionLogDir/TestRun.txt
Incremental Aggregation
 If the source changes incrementally and you can capture changes,
you can configure the session to process those changes.
58
 This allows the Integration Service to update the target
incrementally, rather than forcing it to process the entire source
and recalculate the same data each time you run the session.
 For example, you might have a session using a source that
receives new data every day.
 You can capture those incremental changes because you have
added a filter condition to the mapping that removes pre-
existing data from the flow of data. You then enable incremental
aggregation
 When the session runs with incremental aggregation enabled for
the first time on March 1, you use the entire source.
 This allows the Integration Service to read and store the
necessary aggregate data. On March 2, when you run the
session again, you filter out all the records except those time-
stamped March 2.
 The Integration Service then processes the new data and updates
the target accordingly.
 Consider using incremental aggregation in the following
circumstances:
You can capture new source data - Use incremental aggregation
when you can capture new source data each time you run the
session. Use a Stored Procedure or Filter transformation to
process new data.
Incremental changes do not significantly change the target - Use
incremental aggregation when the changes do not significantly
change the target. If processing the incrementally changed
source alters more than half the existing target, the session may
not benefit from using incremental aggregation. In this case, drop
the table and recreate the target with complete source data
Integration Service Processing for Incremental Aggregation
 The first time you run an incremental aggregation session, the
Integration Service processes the entire source.
 At the end of the session, the Integration Service stores aggregate
data from that session run in two files, the index file and the
data file.
 The Integration Service creates the files in the cache directory
specified in the Aggregator transformation properties.
 Each subsequent time you run the session with incremental
aggregation, you use the incremental source changes in the
session.
 For each input record, the Integration Service checks historical
information in the index file for a corresponding group.
 If it finds a corresponding group, the Integration Service performs
the aggregate operation incrementally, using the aggregate data
for that group, and saves the incremental change.
 If it does not find a corresponding group, the Integration Service
creates a new group and saves the record data.
 When writing to the target, the Integration Service applies the  Avoid moving or modifying the index and data files that store
changes to the existing target. historical aggregate information.
 It saves modified aggregate data in the index and data files to be  If you move the files into a different directory, the Integration
used as historical data the next time you run the session. Service rebuilds the files the next time you run the session

 If the source changes significantly and you want the Integration High Precision – 28 digits
Service to continue saving aggregate data for future incremental
changes, configure the Integration Service to overwrite existing
aggregate data with new aggregate data.

 Each subsequent time you run a session with incremental

aggregation, the Integration Service creates a backup of the
incremental aggregation files.
 The cache directory for the Aggregator transformation must
contain enough disk space for two sets of the files.

 When you partition a session that uses incremental aggregation,

the Integration Service creates one set of cache files for each
partition.
 The Integration Service creates new aggregate data, instead of
using historical data, when you perform one of the following
tasks:
- Save a new version of the mapping.
- Configure the session to reinitialize the aggregate cache.
- Move the aggregate files without correcting the configured path
or directory for the files in the session properties.
- Change the configured path or directory for the aggregate files
without moving the files to the new location.
- Delete cache files.
- Decrease the number of partitions.

 When the Integration Service rebuilds incremental aggregation

files, the data in the previous files is lost.
Note: To protect the incremental aggregation files from file
corruption or disk failure, periodically back up the files

Reinitializing the Aggregate Files

 If the source tables change significantly, you might want the
Integration Service to create new aggregate data, instead of
using historical data.
 For example, you can reinitialize the aggregate cache if the source
for a session changes incrementally every day and completely
changes once a month.
 When you receive the new source data for the month, you might
configure the session to reinitialize the aggregate cache,
truncate the existing target, and use the new source table
during the session.
 After you run a session that reinitializes the aggregate cache, edit
the session properties to disable the Reinitialize Aggregate
Cache option.

59
 change the number of rows that pass through the
transformation.
Change the transaction boundary - For example, the Transaction
Control transformation is active because it defines a commit or
roll back transaction based on an expression evaluated for each
row.
Change the row type - For example, the Update Strategy
transformation is active because it flags rows for insert, delete,
update, or reject

 The Designer does not allow you to connect multiple active

transformations or an active and a passive transformation to the
same downstream transformation or transformation input
group because the Integration Service may not be able to
concatenate the rows passed by active transformations.

 The Sequence Generator transformation is an exception to the

rule.
 The Designer does allow you to connect a Sequence Generator
transformation and an active transformation to the same
downstream transformation or transformation input group.
 A Sequence Generator transformation does not receive data. It
generates unique numeric values

6. TRANSFORMATIONS

Active Transformations
 An active transformation can perform any of the following
actions:
Change the number of rows that pass through the
transformation - For example, the Filter transformation is active
because it removes rows that do not meet the filter condition. All
multi-group transformations are active because they might

60
 Ports
Port name - Use the following conventions while naming ports:
- Begin with a single- or double-byte letter or single- or double-
byte underscore (_).
- Port names can contain any of the following single- or double-
byte characters: a letter, number, underscore (_), $, #, or @.

 All multi-group transformations are active transformations. You

cannot connect multiple active transformations or an active and
a passive transformation to the same downstream
transformation or transformation input group.
 Some multiple input group transformations require the
Integration Service to block data at an input group while the
Integration Service waits for a row from a different input group.
 A blocking transformation is a multiple input group
transformation that blocks incoming data.

 The following transformations are blocking transformations:

- Custom transformation with the Inputs May Block property
enabled
- Joiner transformation configured for unsorted input

 The Designer performs data flow validation when you save or

Creating a Transformation
validate a mapping.
You can create transformations using the following Designer
 Some mappings that contain active or blocking transformations
tools:
might not be valid.
Mapping Designer - Create transformations that connect sources
to targets. Transformations in a mapping cannot be used in other
 Using Expression Editor - The maximum number of characters
mappings unless you configure them to be reusable.
that you can include in an expression is 32,767
Transformation Developer - Create individual transformations,
called reusable transformations that use in multiple mappings.
Mapplet Designer - Create and configure a set of  Adding Expressions to a Port - In the Data Masking
transformations, called mapplets, that you use in multiple transformation, you can add an expression to an input port.
mapping For all other transformations, add the expression to an output
port.

61

Guidelines for Configuring Variable Ports
 Consider the following factors when you configure variable ports
in a transformation:
Port order - The Integration Service evaluates ports by
dependency. The order of the ports in a transformation must
match the order of evaluation: input ports, variable ports, output
ports.
Data type - The datatype you choose reflects the return value of
the expression you enter.
Variable initialization - The Integration Service sets initial values
in variable ports, where you can create counters.
Since variables can reference other variables, the display order
for variable ports is the same as the order in which the
Integration Service evaluates each variable
 The display order for output ports does not matter since output
ports cannot reference other output ports. Be sure output ports
display at the bottom of the list of ports.
Variable Initialization
 The Integration Service does not set the initial value for variables
to NULL. Instead, the Integration Service uses the following
guidelines to set initial values for variables:
- Zero for numeric ports
- Empty strings for string ports
- 01/01/1753 for Date/Time ports with PMServer 4.0 date
handling compatibility disabled
- 01/01/0001 for Date/Time ports with PMServer 4.0 date
handling compatibility enabled
 Therefore, use variables as counters, which need an initial value.
For example, you can create a numeric variable with the
following expression:
VAR1 + 1
 This expression counts the number of rows in the VAR1 port. If
the initial value of the variable were set to NULL, the expression
would always evaluate to NULL. This is why the initial value is set
to zero.
Using Default Values for Ports
 All transformations use default values that determine how the
Integration Service handles input null values and output
transformation errors.
 Input, output, and input/output ports are created with a system
default value that you can sometimes override with a user-
defined default value.
 Default values have different functions in different types of ports:
Input port - The system default value for null input ports is NULL.
It displays as a blank in the transformation. If an input value is
NULL, the Integration Service leaves it as NULL.
62
Output port - The system default value for output transformation
errors is ERROR. The default value appears in the transformation
as ERROR(‘transformation error’). If a transformation error
occurs, the Integration Service skips the row. The Integration
Service notes all input rows skipped by the ERROR function in the
session log file.
The following errors are considered transformation errors:
- Data conversion errors, such as passing a number to a date
function.
- Expression evaluation errors, such as dividing by zero.
- Calls to an ERROR function.
Input/output port - The system default value for null input is the
same as input ports, NULL. The system default value appears as a
blank in the transformation. The default value for output
transformation errors is the same as output ports. The default
value for output transformation errors does not display in the
transformation.
Note: The Java Transformation converts PowerCenter datatypes
to Java datatypes, based on the Java Transformation port type.
Default values for null input differ based on the Java datatype.
The following table shows the system default values for ports in
connected transformations:
Entering User-Defined Default Values
 You can override the system default values with user-defined
default values for supported input, input/output, and output
ports within a connected transformation:
Input ports - You can enter user-defined default values for input
ports if you do not want the Integration Service to treat null
values as NULL.
Output ports - You can enter user-defined default values for
output ports if you do not want the Integration Service to skip
the row or if you want the Integration Service to write a specific
message with the skipped row to the session log.
Input/output ports - You can enter user-defined default values to
handle null input values for input/output ports in the same way
you can enter user-defined default values for null input values for
input ports. You cannot enter user-defined default values for
output transformation errors in an input/output port.

Note: The Integration Service ignores user-defined default
values for unconnected transformations. For example, if you call
a Lookup or Stored Procedure transformation through an
expression, the Integration Service ignores any user-defined
default value and uses the system default value only.
Use the ABORT function to abort a session when the Integration
Service encounters null input values.
Entering User-Defined Default Output Values
General Rules for Default Values
Use the following rules and guidelines when you create default
values:
 The default value must be a NULL, a constant value, a constant
expression, an ERROR function, or an ABORT function.
 For input/output ports, the Integration Service uses default
values to handle null input values. The output default value of
input/output ports is always ERROR(‘Transformation Error’).
 Variable ports do not use default values.
 You can assign default values to group by ports in the Aggregator
and Rank transformations.
 Not all port types in all transformations allow user-defined
default values. If a port does not allow user-defined default
values, the default value field is disabled.
 Not all transformations allow user-defined default values.
63
 If a transformation is not connected to the mapping data flow,
the Integration Service ignores user-defined default values.
 If any input port is unconnected, its value is assumed to be NULL
and the Integration Service uses the default value for that
input port.
 If an input port default value contains the ABORT function and
the input value is NULL, the Integration Service immediately
stops the session. Use the ABORT function as a default value to
restrict null input values. The first null value in an input port
stops the session.
 If an output port default value contains the ABORT function and
any transformation error occurs for that port, the session
immediately stops. Use the ABORT function as a default value to
enforce strict rules for transformation errors. The first
transformation error for this port stops the session.
 The ABORT function, constant values, and constant expressions
override ERROR functions configured in output port expressions.
Reusable Transformations
 The Designer stores each reusable transformation as metadata
separate from any mapping that uses the transformation.
 If you review the contents of a folder in the Navigator, you see
the list of all reusable transformations in that folder
 You can create most transformations as a non-reusable or
reusable.
 However, you can only create the External Procedure
transformation as a reusable transformation.
 When you add instances of a reusable transformation to
mappings, you must be careful that changes you make to the
transformation do not invalidate the mapping or generate
unexpected data
Instances and Inherited Changes
 Note that instances do not inherit changes to property settings,
only modifications to ports, expressions, and the name of the
transformation.
 A. AGGREGATOR
 The Integration Service performs aggregate calculations as it
reads and stores data group and row data in an aggregate cache.
 It’s unlike the Expression transformation, in that you use the
Aggregator transformation to perform calculations on groups.
 The Expression transformation permits you to perform
calculations on a row-by-row basis.
 After you create a session that includes an Aggregator
transformation, you can enable the session option, Incremental
Aggregation. When the Integration Service performs
incremental aggregation, it passes source data through the
mapping and uses historical cache data to perform aggregation
calculations incrementally.
Configuring Aggregator Transformation Properties
Modify the Aggregator Transformation properties on the
Properties tab.
Configure the following options:
Configuring Aggregate Caches
 When you run a session that uses an Aggregator transformation,
the Integration Service creates the index and the data caches in
memory to process the transformation.
 If the Integration Service requires more space, it stores overflow
values in cache files.
 You can configure the index and the data caches in the
Aggregator transformation or in the session properties. Or,
 You can configure the Integration Service to determine the cache
size at run time.
64
 Note: The Integration Service uses memory to process an
Aggregator transformation with sorted ports. The Integration
Service does not use cache memory. You do not need to
configure cache memory for Aggregator transformations that
use sorted ports.
 The result of an aggregate expression varies based on the group
by ports in the transformation.
 For example, when the Integration Service calculates the
following aggregate expression with no group by ports defined,
it finds the total quantity of items sold:
SUM( QUANTITY )
 However, if you use the same expression, and you group by the
ITEM port, the Integration Service returns the total quantity of
items sold, by item
Aggregate Functions
 Use the following aggregate functions within an Aggregator
transformation. You can nest one aggregate function within
another aggregate function.
 The transformation language includes the following aggregate
functions:
- AVG – COUNT – FIRST – LAST – MAX – MEDIAN - MIN
- PERCENTILE – STDDEV - SUM
- VARIANCE
Nested Aggregate Functions
 You can include multiple single-level or multiple nested functions
in different output ports in an Aggregator transformation.
 However, you cannot include both single-level and nested
functions in an Aggregator transformation.
 Therefore, if an Aggregator transformation contains a single-level
function in any output port, you cannot use a nested function in
any other port in that transformation.
 When you include single-level and nested functions in the same
Aggregator transformation, the Designer marks the mapping or
mapplet invalid.
 If you need to create both single-level and nested functions
create separate Aggregator transformations.
Group By Ports
 When you group values, the Integration Service produces one
row for each group.
 If you do not group values, the Integration Service returns one
row for all input rows.
 The Integration Service typically returns the last row of each
group (or the last row received) with the result of the
aggregation.
 However, if you specify a particular row to be returned (for
example, by using the FIRST function), the Integration Service
then returns the specified row
 Sorted Input Conditions
 Do not use sorted input if either of the following conditions are
true:
- The aggregate expression uses nested aggregate functions.
- The session uses incremental aggregation.
Tips for Aggregator Transformations
Limit connected input/output or output ports.
 Limit the number of connected input/output or output ports to
reduce the amount of data the Aggregator transformation stores
in the data cache.
Filter the data before aggregating it.
 If you use a Filter transformation in the mapping, place the
transformation before the Aggregator transformation to reduce
unnecessary aggregation.
B. CUSTOM TRANSFORMATION
 Custom transformations operate in conjunction with procedures
you create outside of the Designer interface to extend
PowerCenter functionality.
 You can create a Custom transformation and bind it to a
procedure that you develop using the Custom transformation
functions.
 Use the Custom transformation to create transformation
applications, such as sorting and aggregation, which require
all input rows to be processed before outputting any output
rows.
 To support this process, the input and output functions occur
separately in Custom transformations compared to External
Procedure transformations.
 The Integration Service passes the input data to the procedure
using an input function.
 The output function is a separate function that you must enter
in the procedure code to pass output data to the Integration
Service.
 In contrast, in the External Procedure transformation, an
external procedure function does both input and output, and
its parameters consist of all the ports of the transformation.
 You can also use the Custom transformation to create a
transformation that requires multiple input groups, multiple
output groups, or both
Rules and Guidelines for Custom Transformations
- Custom transformations are connected transformations. You
cannot reference a Custom transformation in an expression.
- You can include multiple procedures in one module. For
example, you can include an XML writer procedure and an XML
parser procedure in the same module.
65
- You can bind one shared library or DLL to multiple Custom
transformation instances if you write the procedure code to
handle multiple Custom transformation instances.
- When you write the procedure code, you must make sure it
does not violate basic mapping rules.
- The Custom transformation sends and receives high precision
decimals as high precision decimals.
- Use multi-threaded code in Custom transformation procedures
Creating Groups and Ports
 You can create multiple input groups and multiple output groups
in a Custom transformation.
 You must create at least one input group and one output group
 When you create a passive Custom transformation, you can
only create one input group and one output group.
Working with Port Attributes
 Ports have attributes, such as datatype and precision. When you
create a Custom transformation, you can create user-defined
port attributes.
 User-defined port attributes apply to all ports in a Custom
transformation.
 For example, you create a external procedure to parse XML data.
 You can create a port attribute called “XML path” where you can
define the position of an element in the XML hierarchy.
Custom Transformation Properties
 Properties for the Custom transformation apply to both the
procedure and the transformation.
 Configure the Custom transformation properties on the
Properties tab of the Custom transformation.
 The following table describes the Custom transformation
properties:
 Within the session - Configure the session to treat the source
rows as data driven.

 If you do not configure the Custom transformation to define the

update strategy, or you do not configure the session as data
driven, the Integration Service does not use the external
procedure code to flag the output rows.
 Instead, when the Custom transformation is active, the
Integration Service flags the output rows as insert.
 When the Custom transformation is passive, the Integration
Service retains the row type.
 For example, when a row flagged for update enters a passive
Custom transformation, the Integration Service maintains the
row type and outputs the row as update.

Working with Transaction Control

 You can define transaction control for Custom transformations
using the following transformation properties:
Transformation Scope - Determines how the Integration Service
applies the transformation logic to incoming data.
Generate Transaction - Indicates that the procedure generates
transaction rows and outputs them to the output groups.

 The following table describes how the Integration Service handles

transaction boundaries at Custom transformations:

Setting the Update Strategy

 Use an active Custom transformation to set the update strategy
for a mapping at the following levels:
Within the procedure - You can write the external procedure
code to set the update strategy for output rows. The external
Blocking Input Data
procedure can flag rows for insert, update, delete, or reject.
 By default, the Integration Service concurrently reads sources in a
Within the mapping - Use the Custom transformation in a
target load order group.
mapping to flag rows for insert, update, delete, or reject. Select
 However, you can write the external procedure code to block
the Update Strategy Transformation property for the Custom
input data on some input groups.
transformation.
66
 Blocking is the suspension of the data flow into an input group of
a multiple input group transformation.
 To use a Custom transformation to block input data, you must
write the procedure code to block and unblock data.
 You must also enable blocking on the Properties tab for the
Custom transformation
Note: When the procedure blocks data and you configure the
Custom transformation as a non-blocking transformation, the
Integration Service fails the session
Validating Mappings with Custom Transformations
 When you include a Custom transformation in a mapping, both
the Designer and Integration Service validate the mapping.
 The Designer validates the mapping you save or validate and the
Integration Service validates the mapping when you run the
session.
Validating at Design Time
 When you save or validate a mapping, the Designer performs
data flow validation.
 When the Designer does this, it verifies that the data can flow
from all sources in a target load order group to the targets
without blocking transformations blocking all sources.
Validating at Runtime
 When you run a session, the Integration Service validates the
mapping against the procedure code at runtime.
 When the Integration Service does this, it tracks whether or not
it allows the Custom transformations to block data
Locale
 The locale identifies the language and region of the characters in
the data.
 Choose a locale from the list.
 The Data Masking transformation masks character data with
characters from the locale that you choose.
 The source data must contain characters that are compatible with
the locale that you select.
Seed
 The seed value is a start number that enables the Data Masking
transformation to return deterministic data with Key Masking.
 The Data Masking transformation creates a default seed value
that is a random number between 1 and 1,000.
 You can enter a different seed value or apply a mapping
parameter value.
 Apply the same seed value to a column to return the same
masked data values in different source data.
Associated O/P
 The Associated O/P is the associated output port for an input
port.
 The Data Masking transformation creates an output port for
each input port.
 The naming convention is out_<port name>. The associated
output port is a readonly port.

C. DATA MASKING

 The Data Masking transformation modifies source data based on

masking rules that you configure for each column.
 You can maintain data relationships in the masked data and
maintain referential integrity between database tables.

 You can apply the following types of masking with the Data
Masking transformation:
Key masking - Produces deterministic results for the same source
data, masking rules, and seed value.
Random masking - Produces random, non-repeatable results for
the same source data and masking rules.
Expression masking - Applies an expression to a port to change
the data or create data.
Substitution - Replaces a column of data with similar but
unrelated data from a dictionary.
Special mask formats - Applies special mask formats to change
SSN, credit card number, phone number, URL, email address, or
IP addresses

67
 You can configure the following masking rules for key masking
string values:
Seed - Apply a seed value to generate deterministic masked data
for a column. Select one of the following options:
Value - Accept the default seed value or enter a number
between 1 and 1,000.
Mapping Parameter - Use a mapping parameter to define the
seed value. The Designer displays a list of the mapping
parameters that you create for the mapping. Choose the mapping
parameter from the list to use as the seed value.
Mask Format - Define the type of character to substitute for each
character in the input data. You can limit each character to an
alphabetic, numeric, or alphanumeric character type.
Source String Characters - Define the characters in the source
string that you want to mask. For example, mask the number sign
(#) character whenever it occurs in the input data. The Data
Masking transformation masks all the input characters when
Source String Characters is blank. The Data Masking
transformation does not always return unique data if the number
of source string characters is less than the number of result string
characters.
Result String Characters - Substitute the characters in the target
string with the characters you define in Result String Characters.
For example, enter the following characters to configure each
mask to contain all uppercase alphabetic characters:
ABCDEFGHIJKLMNOPQRSTUVWXYZ
Masking Numeric Values
 Configure key masking for numeric source data to generate
deterministic output.
 When you configure a column for numeric key masking, the
Designer assigns a random seed value to the column.
 When the Data Masking transformation masks the source data, it
applies a masking algorithm that requires the seed.
 You can change the seed value for a column to produce
repeatable results if the same source value occurs in a different
column.
 For example, you want to maintain a primary-foreign key
relationship between two tables.
 In each Data Masking transformation, enter the same seed value
for the primary-key column as the seed value for the foreign-key
column.
 The Data Masking transformation produces deterministic results
for the same numeric values.
 The referential integrity is maintained between the tables.
Masking Datetime Values
 The Data Masking transformation can mask dates between 1753
and 2400 with key masking.
 If the source year is in a leap year, the Data Masking
transformation returns a year that is also a leap year.
68
 If the source month contains 31 days, the Data Masking
transformation returns a month that has 31 days.
 If the source month is February, the Data Masking transformation
returns February.
 The Data Masking transformation always generates valid dates
Masking with Mapping Parameters
 The Integration Service applies a default seed value in the
following circumstances:
- The mapping parameter option is selected for a column but the
session has no parameter file.
- You delete the mapping parameter.
- A mapping parameter seed value is not between 1 and 1,000.
 The Integration Service applies masked values from the default
value file.
 You can edit the default value file to change the default values.
 The default value file is an XML file in the following location:
<PowerCenter Installation
Directory>\infa_shared\SrcFiles\defaultValue.xml
 The name-value pair for the seed is default_seed = "500".
 If the seed value in the default value file is not between 1 and
1,000, the Integration Service assigns a value of 725 to the seed
and writes a message in the session log.
Substitution Masking
 Substitution masking replaces a column of data with similar but
unrelated data.
 When you configure substitution masking, define the relational
or flat file dictionary that contains the substitute values.
 The Data Masking transformation performs a lookup on the
dictionary that you configure.
 The Data Masking transformation replaces source data with data
from the dictionary. Substitution is an effective way to replace
production data with realistic test data.
 You can substitute data with repeatable or non-repeatable
values.
 When you choose repeatable values, the Data Masking
transformation produces deterministic results for the same
source data and seed value.
 You must configure a seed value to substitute data with
deterministic results.
 The Integration Service maintains a storage table of source and
masked values for repeatable masking
Dictionaries
 A dictionary is a flat file or relational table that contains the
substitute data and a serial number for each row in the file.
 The Integration Service generates a number to retrieve a
dictionary row by the serial number.
 The Integration Service generates a hash key for repeatable
substitution masking or a random number for non-repeatable
masking.
 You can configure an additional lookup condition if you configure
repeatable substitution masking.
 You can configure a dictionary to mask more than one port in the
Data Masking transformation.
 The following example shows a flat file dictionary that contains
first name and gender:
SNO,GENDER,FIRSTNAME
1,M,Adam
2,M,Adeel
3,M,Adil
4,F,Alice
5,F,Alison
 Use the following rules and guidelines when you create a
dictionary:
- Each record in the dictionary must have a serial number. The
serial number does not have to be the key in a relational table.
- The serial numbers are sequential integers starting at one. The
serial numbers cannot have a missing number in the sequence.
- The serial number column can be anywhere in a dictionary row.
It can have any label.
- The first row of a flat file dictionary must have column labels to
identify the fields in each record. The fields are separated by
commas. If the first row does not contain column labels, the
Integration Service takes the values of the fields in the first row as
column names.
- A flat file dictionary must be in the $PMLookupFileDir lookup
file directory. By default, this directory is in the following location:
<PowerCenter_Installation_Directory>\server\infa_shared\LkpFil
es
- If you create a flat file dictionary on Windows and copy it to a
UNIX machine, verify that the file format is correct for UNIX. For
example, Windows and UNIX use different characters for the end
of line marker.
- If you configure substitution masking for more than one port, all
relational dictionaries must be in the same database schema.
- You cannot change the dictionary type or the substitution
dictionary name in session properties.
Storage Tables
 The Data Masking transformation maintains storage tables for
repeatable substitution between sessions.
 A storage table row contains the source column and a masked
value pair.
 Each time the Integration Service masks a value with a repeatable
substitute value, it searches the storage table by dictionary
name, locale, column name, input value, and seed.
 If it finds a row, it returns the masked value from the storage
table to the Data Masking transformation.
69
 If the Integration Service does not find a row, it retrieves a row
from the dictionary with a hash key.
Rules and Guidelines for Substitution Masking
- If a storage table does not exist for a repeatable substitution
mask, the session fails.
- If the dictionary contains no rows, the Integration Service
returns default masked values.
- When the Integration Service finds an input value with the
locale, dictionary, and seed in the storage table, it retrieves the
masked value, even if the row is no longer in the dictionary.
- If you delete a connection object or modify the dictionary,
truncate the storage table. Otherwise, you might get unexpected
results.
- If the number of values in the dictionary is less than the number
of unique values in the source data, the Integration Service
cannot mask the data with unique repeatable values. The
Integration Service returns default masked values.
Random Masking
 Random masking generates random nondeterministic masked
data.
 The Data Masking transformation returns different values when
the same source value occurs in different rows.
 You can define masking rules that affect the format of data that
the Data Masking transformation returns.
 Mask numeric, string, and date values with random masking
Masking Numeric Values
 When you mask numeric data, you can configure a range of
output values for a column.
 The Data Masking transformation returns a value between the
minimum and maximum values of the range depending on port
precision.
 To define the range, configure the minimum and maximum
ranges or a blurring range based on a variance from the original
source value.
 You can configure the following masking parameters for numeric
data:
Range - Define a range of output values. The Data Masking
transformation returns numeric data between the minimum and
maximum values.
Blurring Range - Define a range of output values that are within a
fixed variance or a percentage variance of the source data. The
Data Masking transformation returns numeric data that is close
to the value of the source data. You can configure a range and a
blurring range.
Masking String Values
 Configure random masking to generate random output for string
columns.
 To configure limitations for each character in the output string,
configure a mask format.
 Configure filter characters to define which source characters to
mask and the characters to mask them with.
 You can apply the following masking rules for a string port:
Range - Configure the minimum and maximum string length. The
Data Masking transformation returns a string of random
characters between the minimum and maximum string length.
Mask Format - Define the type of character to substitute for each
character in the input data. You can limit each character to an
alphabetic, numeric, or alphanumeric character type.
Source String Characters - Define the characters in the source
string that you want to mask. For example, mask the number sign
(#) character whenever it occurs in the input data. The Data
Masking transformation masks all the input characters when
Source String Characters is blank.
Result String Replacement Characters - Substitute the characters
in the target string with the characters you define in Result String
Characters. For example, enter the following characters to
configure each mask to contain uppercase alphabetic characters
A - Z: ABCDEFGHIJKLMNOPQRSTUVWXYZ
Masking Date Values
 To mask date values with random masking, either configure a
range of output dates or choose a variance.
 When you configure a variance, choose a part of the date to blur.
 Choose the year, month, day, hour, minute, or second.
 The Data Masking transformation returns a date that is within the
range you configure.
 You can configure the following masking rules when you mask a
datetime value:
Range - Sets the minimum and maximum values to return for the
selected datetime value
Blurring - Masks a date based on a variance that you apply to a
unit of the date. The Data Masking transformation returns a date
that is within the variance. You can blur the year, month, day, or
hour. Choose a low and high variance to apply.
Applying Masking Rules
 Apply masking rules based on the source datatype.
 When you click a column property on the Masking Properties tab,
the Designer displays masking rules based on the datatype of
the port.
 The following table describes the masking rules that you can
configure based on the masking type and the source datatype:
70
Source String Characters
 Source string characters are source characters that you choose to
mask or not mask.
 The position of the characters in the source string does not
matter.
 The source characters are case sensitive.
 You can configure any number of characters.
 When Characters is blank, the Data Masking transformation
replaces all the source characters in the column.
 Select one of the following options for source string characters:
Mask Only - The Data Masking transformation masks characters
in the source that you configure as source string characters. For
example, if you enter the characters A, B, and c, the Data Masking
transformation replaces A, B, or c with a different character when
the character occurs in source data. A source character that is not
an A, B, or c does not change. The mask is case sensitive.
Mask All Except - Masks all characters except the source string
characters that occur in the source string. For example, if you
enter the filter source character “-” and select Mask All Except,
the Data Masking transformation does not replace the “-”
character when it occurs in the source data. The rest of the
source characters change.
Example
 A source file has a column named Dependents. The Dependents
column contains more than one name separated by commas.
You need to mask the Dependents column and keep the comma
in the test data to delimit the names. For the Dependents
column, select Source String Characters. Choose Don’t Mask and
enter “,” as the source character to skip
Result String Replacement Characters
 Result string replacement characters are characters you choose
as substitute characters in the masked data.
 When you configure result string replacement characters, the
Data Masking transformation replaces characters in the source
string with the result string replacement characters.
 To avoid generating the same output for different input values,
configure a wide range of substitute characters, or mask only a
few source characters.
 The position of each character in the string does not matter.
 Select one of the following options for result string replacement
characters:
Use Only - Mask the source with only the characters you define
as result string replacement characters. For example, if you enter
the characters A, B, and c, the Data Masking transformation
replaces every character in the source column with an A, B, or c.
The word “horse” might be replaced with “BAcBA.”
Use All Except - Mask the source with any characters except the
characters you define as result string replacement characters. For
example, if you enter A, B, and c result string replacement
characters, the masked data never has the characters A, B, or c.
Example
To replace all commas in the Dependents column with
semicolons, complete the following tasks:
1. Configure the comma as a source string character and select
Mask Only.
The Data Masking transformation masks only the comma when it
occurs in the Dependents column.
2. Configure the semicolon as a result string replacement
character and select Use Only.
The Data Masking transformation replaces each comma in the
Dependents column with a semicolon
Range
 Define a range for numeric, date, or string data. When you define
a range for numeric or date values the Data Masking
transformation masks the source data with a value between the
minimum and maximum values. When you configure a range for
a string, you configure a range of string lengths.
Blurring
 Blurring creates an output value within a fixed or percent
variance from the source data value. Configure blurring to
return a random value that is close to the original value. You can
blur numeric and date values
Special Mask Formats
71
 The following types of masks retain the format of the original
data:
- Social Security numbers
- Credit card numbers
- Phone numbers
- URL addresses
- Email addresses
- IP addresses
 The Data Masking transformation returns a masked value that
has a realistic format, but is not a valid value.
 For example, when you mask an SSN, the Data Masking
transformation returns an SSN that is the correct format but is
not valid.
 You can configure repeatable masking for Social Security numbers
 When the source data format or datatype is invalid for a mask,
the Integration Service applies a default mask to the data.
 The Integration Service applies masked values from the default
value file
Default Value File
 The default value file is an XML file in the following location:
<PC Directory>\infa_shared\SrcFiles\defaultValue.xml
 The defaultValue.xml file contains the following name-value
pairs:
<?xml version="1.0" standalone="yes" ?>
<defaultValue
default_char = "X"
default_digit = "9"
default_date = "11/11/1111 00:00:00"
default_email = "abc@xyz.com"
default_ip = "99.99.9.999"
default_url = "http://www.xyz.com"
default_phone = "999 999 999 9999"
default_ssn = "999-99-9999"
default_cc = "9999 9999 9999 9999"
default_seed = "500"
/>
Rules and Guidelines for Data Masking Transformations
- The Data Masking transformation does not mask null values. If
the source data contains null values, the Data Masking
transformation returns null values. To replace null values, add an
upstream transformation that allows user-defined default values
for input ports.
- When the source data format or datatype is invalid for a mask,
the Integration Service applies a default mask to the data. The
Integration Service applies masked values from a default values
file.
- The Data masking transformation returns an invalid Social
Security number with the same format and area code as the
 source. If the Social Security Administration has issued more than
half of the numbers for an area, the Data Masking transformation
might not be able to return unique invalid Social Security
numbers with key masking.
D. EXTERNAL PROCEDURE
 External Procedure transformations operate in conjunction with
procedures you create outside of the Designer interface to
extend PowerCenter functionality.
 If you are an experienced programmer, you may want to develop
complex functions within a dynamic link library (DLL) or UNIX
shared library, instead of creating the necessary Expression
transformations in a mapping.
 To get this kind of extensibility, use the Transformation Exchange
(TX) dynamic invocation interface built into PowerCenter.
 Using TX, you can create an Informatica External Procedure
transformation and bind it to an external procedure that you
have developed.
 You can bind External Procedure transformations to two kinds of
external procedures:
- COM external procedures (available on Windows only)
- Informatica external procedures (available on Windows, AIX, HP-
UX, Linux, and Solaris)
External Procedures and External Procedure Transformations
 There are two components to TX: external procedures and
External Procedure transformations.
 An External procedure exists separately from the Integration
Service.
 It consists of C, C++, or Visual Basic code written by a user to
define a transformation.
 This code is compiled and linked into a DLL or shared library,
which is loaded by the Integration Service at runtime.
 An external procedure is “bound” to an External Procedure
transformation.
 An External Procedure transformation is created in the Designer.
It is an object that resides in the Informatica repository and
serves several purposes:
1. It contains the metadata describing the following external
procedure. It is through this metadata that the Integration
Service knows the “signature” (number and types of parameters,
type of return value, if any) of the external procedure.
2. It allows an external procedure to be referenced in a mapping.
By adding an instance of an External Procedure transformation to
a mapping, you call the external procedure bound to that
transformation.
Note: You can create a connected or unconnected External
Procedure.
72
3. When you develop Informatica external procedures, the
External Procedure transformation provides the information
required to generate Informatica external procedure stubs.
External Procedure transformations return one or no output
rows for each input row.
E. FILTER
 A filter condition returns TRUE or FALSE for each row that the
Integration Service evaluates, depending on whether a row
meets the specified condition.
 For each row that returns TRUE, the Integration Services pass
through the transformation.
 For each row that returns FALSE, the Integration Service drops
and writes a message to the session log.
 You cannot concatenate ports from more than one
transformation into the Filter transformation.
 The input ports for the filter must come from a single
transformation
 If the filter condition evaluates to NULL, the row is treated as
FALSE.
Note: The filter condition is case sensitive.
F. HTTP
 The HTTP transformation enables you to connect to an HTTP
server to use its services and applications.
 When you run a session with an HTTP transformation, the
Integration Service connects to the HTTP server and issues a
request to retrieve data from or update data on the HTTP server,
depending on how you configure the transformation:
Read data from an HTTP server - When the Integration Service
reads data from an HTTP server; it retrieves the data from the
HTTP server and passes the data to the target or a downstream
transformation in the mapping. For example, you can connect to
an HTTP server to read current inventory data, perform
calculations on the data during the PowerCenter session, and
pass the data to the target.
Update data on the HTTP server - When the Integration Service
writes to an HTTP server, it posts data to the HTTP server and
passes HTTP server responses to the target or a downstream
transformation in the mapping. For example, you can post data
providing scheduling information from upstream transformations
to the HTTP server during a session
G. IDENTITY RESOLUTION
 The Identity Resolution transformation is an active
transformation that you can use to search and match data in
Informatica Identity Resolution (IIR).
 The PowerCenter Integration Service uses the search definition
that you specify in the Identity Resolution transformation to
search and match data residing in the IIR tables.
 The input and output views in the system determine the input
and output ports of the transformation.
Groups and Ports
 An Identity Resolution transformation contains an input group
and an output group.
 The input group has ports that represent fields in the input view
of the search definition.
 The output group has ports that represent fields in the output
view of the search definition in addition to ports that describe
the result of the search.
H. JAVA
 Extend PowerCenter functionality with the Java transformation.
 The Java transformation provides a simple native programming
interface to define transformation functionality with the Java
programming language.
 You can use the Java transformation to quickly define simple or
moderately complex transformation functionality without
advanced knowledge of the Java programming language or an
external Java development environment.
 The PowerCenter Client uses the Java Development Kit (JDK) to
compile the Java code and generate byte code for the
transformation.
 The PowerCenter Client stores the byte code in the PowerCenter
repository.
 The Integration Service uses the Java Runtime Environment (JRE)
to execute generated byte code at run time.
 When the Integration Service runs a session with a Java
transformation, the Integration Service uses the JRE to execute
the byte code and process input rows and generate output
rows.
 Create Java transformations by writing Java code snippets that
define transformation logic.
 Define transformation behavior for a Java transformation based
on the following events:
- The transformation receives an input row.
- The transformation has processed all input rows.
- The transformation receives a transaction notification such as
commit or rollback.
Active and Passive Java Transformations
 When you create a Java transformation, you define its type as
active or passive.
 After you set the transformation type, you cannot change it.
 A Java transformation runs the Java code that you define on the
On Input Row tab one time for each row of input data.
73
 A Java transformation handles output rows based on the
transformation type as follows:
- A passive Java transformation generates one output row for
each input row in the transformation after processing each
input row.
- An active Java transformation generates multiple output rows
for each input row in the transformation.
 Use the generateRow method to generate each output row. For
example, if the transformation contains two input ports that
represent a start date and an end date, you can use the
generateRow method to generate an output row for each date
between the start date and the end date.
Datatype Conversion
 When a Java transformation reads input rows, it converts input
port datatypes to Java datatypes.
 When a Java transformation writes output rows, it converts Java
datatypes to output port datatypes.
 For example, the following processing occurs for an input port
with the integer datatype in a Java transformation:
1. The Java transformation converts the integer datatype of the
input port to the Java primitive int datatype.
2. In the transformation, the transformation treats the value of
the input port as the Java primitive int datatype.
3. When the transformation generates the output row, it converts
the Java primitive int datatype to the integer datatype.
 A Java transformation can have input ports, output ports, and
input/output ports. You create and edit groups and ports on the
Ports tab.
 A Java transformation always has one input group and one
output group.
 The transformation is not valid if it has multiple input or output
groups
Compiling a Java Transformation
 The PowerCenter Client uses the Java compiler to compile the
Java code and generate the byte code for the transformation.
 The Java compiler compiles the Java code and displays the results
of the compilation in the Output window on the code entry
tabs.
 The Java compiler installs with the PowerCenter Client in the
java/bin directory.
 To compile the full code for the Java transformation, click
Compile on the Java Code tab.
 When you create a Java transformation, it contains a Java class
that defines the base functionality for a Java transformation.
 The full code for the Java class contains the template class code
for the transformation, plus the Java code you define on the
code entry tabs.
 When you compile a Java transformation, the PowerCenter Client
adds the code from the code entry tabs to the template class for
the transformation to generate the full class code for the
transformation.
 The PowerCenter Client then calls the Java compiler to compile
the full class code.
 The Java compiler compiles the transformation and generates the
byte code for the transformation
Note: The Java transformation is also compiled when you click OK
in the transformation
Java Expressions
 You can invoke PowerCenter expressions in a Java transformation
with the Java programming language.
 Use expressions to extend the functionality of a Java
transformation.
 For example, you can invoke an expression in a Java
transformation to look up the values of input or output ports or
look up the values of Java transformation variables.
 To invoke expressions in a Java transformation, you generate the
Java code or use Java transformation API methods to invoke the
expression.
 You invoke the expression and use the result of the expression on
the appropriate code entry tab.
 You can generate the Java code that invokes an expression or use
API methods to write the Java code that invokes the expression
I. JOINER
 The master pipeline ends at the Joiner transformation, while
the detail pipeline continues to the target.
 The Joiner transformation accepts input from most
transformations. However, consider the following limitations on
the pipelines you connect to the Joiner transformation:
- You cannot use a Joiner transformation when either input
pipeline contains an Update Strategy transformation.
- You cannot use a Joiner transformation if you connect a
Sequence Generator transformation directly before the Joiner
transformation
 Joiner Data Cache Size - Default cache size is 2,000,000 bytes
 Joiner Index Cache Size - Default cache size is 1,000,000 bytes
 To improve performance for an unsorted Joiner transformation,
use the source with fewer rows as the master source.
 To improve performance for a sorted Joiner transformation, use
the source with fewer duplicate key values as the master
74
 By default, when you add ports to a Joiner transformation, the
ports from the first source pipeline display as detail sources.
 Adding the ports from the second source pipeline sets them as
master sources.
 If you use multiple ports in the join condition, the Integration
Service compares the ports in the order you specify.
 If you join Char and Varchar datatypes, the Integration Service
counts any spaces that pad Char values as part of the string:
Char(40) = "abcd"
Varchar(40) = "abcd"
 The Char value is “abcd” padded with 36 blank spaces, and the
Integration Service does not join the two fields because the Char
field contains trailing spaces.
 Note: The Joiner transformation does not match null values. For
example, if both EMP_ID1 and EMP_ID2 contain a row with a
null value, the Integration Service does not consider them a
match and does not join the two rows. To join rows with null
values, replace null input with default values, and then join on
the default values.
Using Sorted Input
 When you configure the Joiner transformation to use sorted data,
the Integration Service improves performance by minimizing
disk input and output.
 When you configure the sort order in a session, you can select a
sort order associated with the Integration Service code page.
 When you run the Integration Service in Unicode mode, it uses
the selected session sort order to sort character data.
 When you run the Integration Service in ASCII mode, it sorts all
character data using a binary sort order.
 To ensure that data is sorted as the Integration Service requires,
the database sort order must be the same as the user-defined
session sort order.
 When you join sorted data from partitioned pipelines, you must
configure the partitions to maintain the order of sorted data
 If you pass unsorted or incorrectly sorted data to a Joiner
transformation configured to use sorted data, the session fails
and the Integration Service logs the error in the session log file
Adding Transformations to the Mapping
 When you add transformations between the sort origin and the
Joiner transformation, use the following guidelines to maintain
sorted data:
- Do not place any of the following transformations between the
sort origin and the Joiner transformation: CARNUXX
- Custom
 - Unsorted Aggregator
- Normalizer
- Rank
- Union
- XML Parser
- XML Generator
- Mapplet, if it contains one of the above
- You can place a sorted Aggregator transformation between
the sort origin and the Joiner transformation if you use the
following guidelines:
- Configure the Aggregator transformation for sorted input.
- Use the same ports for the group by columns in the
Aggregator transformation as the ports at the sort origin.
- The group by ports must be in the same order as the ports at
the sort origin.
- When you join the result set of a Joiner transformation with
another pipeline, verify that the data output from the first Joiner
transformation is sorted.
Tip: You can place the Joiner transformation directly after the
sort origin to maintain sorted data
Example of a Join Condition
 For example, you configure Sorter transformations in the master
and detail pipelines with the following sorted ports:
1. ITEM_NO
2. ITEM_NAME
3. PRICE
 When you configure the join condition, use the following
guidelines to maintain sort order:
- You must use ITEM_NO in the first join condition.
- If you add a second join condition, you must use ITEM_NAME.
- If you want to use PRICE in a join condition, you must also use
ITEM_NAME in the second join condition.
 If you skip ITEM_NAME and join on ITEM_NO and PRICE, you
lose the sort order and the Integration Service fails the session.
Joining Two Branches of the Same Pipeline
 When you join data from the same source, you can create two
branches of the pipeline.
 When you branch a pipeline, you must add a transformation
between the source qualifier and the Joiner transformation in
at least one branch of the pipeline.
 You must join sorted data and configure the Joiner
transformation for sorted input.
 The following figure shows a mapping that joins two branches
of the same pipeline:
75
 Joining two branches might impact performance if the Joiner
transformation receives data from one branch much later than
the other branch.
 The Joiner transformation caches all the data from the first
branch, and writes the cache to disk if the cache fills.
 The Joiner transformation must then read the data from disk
when it receives the data from the second branch. This can slow
processing.
Joining Two Instances of the Same Source
 You can also join same source data by creating a second instance
of the source.
 After you create the second source instance, you can join the
pipelines from the two source instances.
 If you want to join unsorted data, you must create two
instances of the same source and join the pipelines
 The following figure shows two instances of the same source
joined with a Joiner transformation:
Note: When you join data using this method, the Integration
Service reads the source data for each source instance, so
performance can be slower than joining two branches of a
pipeline
Guidelines for Joining Data from a Single Source
 Use the following guidelines when deciding whether to join
branches of a pipeline or join two instances of a source:
- Join two branches of a pipeline when you have a large source or
if you can read the source data only once. For example, you can
only read source data from a message queue once.
- Join two branches of a pipeline when you use sorted data. If the
source data is unsorted and you use a Sorter transformation to
sort the data, branch the pipeline after you sort the data.
- Join two instances of a source when you need to add a blocking
transformation to the pipeline between the source and the Joiner
transformation.
- Join two instances of a source if one pipeline may process
slower than the other pipeline.
- Join two instances of a source if you need to join unsorted data
Blocking the Source Pipelines
 When you run a session with a Joiner transformation, the
Integration Service blocks and un-blocks the source data, based
on the mapping configuration and whether you configure the
Joiner transformation for sorted input.

Unsorted Joiner Transformation
 When the Integration Service processes an unsorted Joiner
transformation, it reads all master rows before it reads the
detail rows.
 To ensure it reads all master rows before the detail rows, the
Integration Service blocks the detail source while it caches rows
from the master source.
 Once the Integration Service reads and caches all master rows, it
unblocks the detail source and reads the detail rows.
 Some mappings with unsorted Joiner transformations violate
data flow validation.
Sorted Joiner Transformation
 When the Integration Service processes a sorted Joiner
transformation, it blocks data based on the mapping
configuration.
 Blocking logic is possible if master and detail input to the Joiner
transformation originate from different sources.
 The Integration Service uses blocking logic to process the Joiner
transformation if it can do so without blocking all sources in a
target load order group simultaneously.
 Otherwise, it does not use blocking logic.
 Instead, it stores more rows in the cache.
 When the Integration Service can use blocking logic to process
the Joiner transformation, it stores fewer rows in the cache,
increasing performance.
Caching Master Rows
 When the Integration Service processes a Joiner transformation,
it reads rows from both sources concurrently and builds the
index and data cache based on the master rows.
 The Integration Service then performs the join based on the
detail source data and the cache data.
 The number of rows the Integration Service stores in the cache
depends on the partitioning scheme, the source data, and
whether you configure the Joiner transformation for sorted
input.
 To improve performance for an unsorted Joiner transformation,
use the source with fewer rows as the master source.
 To improve performance for a sorted Joiner transformation, use
the source with fewer duplicate key values as the master.
Working with Transactions
 When the Integration Service processes a Joiner transformation,
it can apply transformation logic to all data in a transaction, all
incoming data, or one row of data at a time.
 The Integration Service can drop or preserve transaction
boundaries depending on the mapping configuration and the
transformation scope.
76
 You configure how the Integration Service applies transformation
logic and handles transaction boundaries using the
transformation scope property.
 You configure transformation scope values based on the mapping
configuration and whether you want to preserve or drop
transaction boundaries.
 You can preserve transaction boundaries when you join the
following sources:
You join two branches of the same source pipeline - Use the
Transaction transformation scope to preserve transaction
boundaries.
You join two sources, and you want to preserve transaction
boundaries for the detail source - Use the Row transformation
scope to preserve transaction boundaries in the detail pipeline.
 You can drop transaction boundaries when you join the following
sources:
You join two sources or two branches and you want to drop
transaction boundaries - Use the All Input transformation scope
to apply the transformation logic to all incoming data and drop
transaction boundaries for both pipelines.
 The following table summarizes how to preserve transaction
boundaries using transformation scopes with the Joiner
transformation:
Preserving Transaction Boundaries for a Single Pipeline
 When you join data from the same source, use the Transaction
transformation scope to preserve incoming transaction
boundaries for a single pipeline.
 Use the Transaction transformation scope when the Joiner
transformation joins data from the same source, either two
branches of the same pipeline or two output groups of one
transaction generator.
 Use this transformation scope with sorted data and any join type.
 When you use the Transaction transformation scope, verify that
master and detail pipelines originate from the same transaction
control point and that you use sorted input.
 For example, in “Preserving Transaction Boundaries for a Single
Pipeline” on page 223 the Sorter transformation is the
transaction control point.
 You cannot place another transaction control point between the
Sorter transformation and the Joiner transformation.
 In the mapping, the master and detail pipeline branches originate
from the same transaction control point, and the Integration
Service joins the pipeline branches with the Joiner
transformation, preserving transaction boundaries
 The following figure shows a mapping that joins two branches of
a pipeline and preserves transaction boundaries:
Preserving Transaction Boundaries in the Detail Pipeline
 When you want to preserve the transaction boundaries in the
detail pipeline, choose the Row transformation scope.
 The Row transformation scope allows the Integration Service to
process data one row at a time.
 The Integration Service caches the master data and matches the
detail data with the cached master data.
 When the source data originates from a real-time source, such as
IBM MQ Series, the Integration Service matches the cached
master data with each message as it is read from the detail
source.
 Use the Row transformation scope with Normal and Master
Outer join types that use unsorted data.
Dropping Transaction Boundaries for Two Pipelines
 When you want to join data from two sources or two branches
and you do not need to preserve transaction boundaries, use
the All Input transformation scope.
 When you use All Input, the Integration Service drops incoming
transaction boundaries for both pipelines and outputs all rows
from the transformation as an open transaction.
 At the Joiner transformation, the data from the master pipeline
can be cached or joined concurrently, depending on how you
configure the sort order.
 Use this transformation scope with sorted and unsorted data and
any join type.
J. LOOKUP
 Use a Lookup transformation in a mapping to look up data in a
flat file, relational table, view, or synonym
 You can import a lookup definition from any flat file or relational
database to which both the PowerCenter Client and Integration
Service can connect.
 You can also create a lookup definition from a source qualifier
77
Configure the Lookup transformation to perform the following
types of lookups:
Relational or flat file lookup -
 Perform a lookup on a flat file or a relational table.
 When you create a Lookup transformation using a relational table
as the lookup source, you can connect to the lookup source
using ODBC and import the table definition as the structure for
the Lookup transformation.
 Use the following options with relational lookups:
- Override the default SQL statement to add a WHERE clause or
to query multiple tables.
- Sort null data high or low, based on database support.
- Perform case-sensitive comparisons based on the database
support.
 When you create a Lookup transformation using a flat file as a
lookup source, the Designer invokes the Flat File Wizard.
 Use the following options with flat file lookups:
- Use indirect files as lookup sources by configuring a file list as
the lookup file name
- Use sorted input for the lookup.
- Sort null data high or low.
- Use case-sensitive string comparison with flat file lookups.
Pipeline Lookups
 Create a pipeline Lookup transformation to perform a lookup on
an application source that is not a relational table or flat file.
 A pipeline Lookup transformation has a source qualifier as the
lookup source. The source qualifier can represent any type of
source definition, including JMS and MSMQ.
 The source definition cannot have more than one group.
 When you configure a pipeline Lookup transformation, the
lookup source and source qualifier are in a different pipeline
from the Lookup transformation.
 The source and source qualifier are in a partial pipeline that
contains no target.
 The Integration Service reads the source data in this pipeline and
passes the data to the Lookup transformation to create the
cache.
 You can create multiple partitions in the partial pipeline to
improve performance.
 To improve performance when processing relational or flat file
lookup sources, create a pipeline Lookup transformation instead
of a relational or flat file Lookup transformation.
 You can create partitions to process the lookup source and pass it
to the Lookup transformation.
 Create a connected or unconnected pipeline Lookup
transformation.
Note: Do not enable HA recovery for sessions that have real-time
sources for pipeline lookups. You might get unexpected results
 Configuring a Pipeline Lookup Transformation in a Mapping
 A mapping that contains a pipeline Lookup transformation
includes a partial pipeline that contains the lookup source and
source qualifier.
 The partial pipeline does not include a target. The Integration
Service retrieves the lookup source data in this pipeline and
passes the data to the lookup cache.
 The partial pipeline is in a separate target load order group in
session properties.
 You can create multiple partitions in the pipeline to improve
performance.
 You cannot configure the target load order with the partial
pipeline.
 The following mapping shows a mapping that contains a pipeline
Lookup transformation and the partial pipeline that processes
the lookup source:
The mapping contains the following objects:
- The lookup source definition and source qualifier are in a
separate pipeline. The Integration Service creates a lookup cache
after it processes the lookup source data in the pipeline.
- A flat file source contains new department names by employee
number.
- The pipeline Lookup transformation receives Employee_Number
and New_Dept from the source file. The pipeline Lookup
performs a lookup on Employee_ID in the lookup cache. It
retrieves the employee first and last name from the lookup
cache.
- A flat file target receives the Employee_ID, First_Name,
Last_Name, and New_Dept from the Lookup transformation.
Connected or unconnected lookup -
 A connected Lookup transformation receives source data,
performs a lookup, and returns data to the pipeline.
 An unconnected Lookup transformation is not connected to a
source or target.
 A transformation in the pipeline calls the Lookup transformation
with a :LKP expression.
 The unconnected Lookup transformation returns one column to
the calling transformation.
78
Connected Lookup Transformation
 The following steps describe how the Integration Service
processes a connected Lookup transformation:
1. A connected Lookup transformation receives input values
directly from another transformation in the pipeline.
2. For each input row, the Integration Service queries the lookup
source or cache based on the lookup ports and the condition in the
transformation.
3. If the transformation is uncached or uses a static cache, the
Integration Service returns values from the lookup query.
 If the transformation uses a dynamic cache, the Integration
Service inserts the row into the cache when it does not find the
row in the cache. When the Integration Service finds the row in
the cache, it updates the row in the cache or leaves it
unchanged. It flags the row as insert, update, or no change.
4. The Integration Service passes return values from the query
to the next transformation.
 If the transformation uses a dynamic cache, you can pass rows to
a Filter or Router transformation to filter new rows to the target.
Unconnected Lookup Transformation
 An unconnected Lookup transformation receives input values
from the result of a :LKP expression in another transformation.
 You can call the Lookup transformation more than once in a
mapping.
 A common use for unconnected Lookup transformations is to
update slowly changing dimension tables.
 For more information about slowly changing dimension tables,
visit the Informatica Knowledge Base at
http://mysupport.informatica.com.
 The following steps describe the way the Integration Service
processes an unconnected Lookup transformation:
1. An unconnected Lookup transformation receives input values
from the result of a :LKP expression in another transformation, such
as an Update Strategy transformation.
2. The Integration Service queries the lookup source or cache based
on the lookup ports and condition in the transformation.
3. The Integration Service returns one value into the return port of
the Lookup transformation.
4. The Lookup transformation passes the return value into the :LKP
expression.

 The lookup table can be a single table, or you can join multiple
tables in the same database using a lookup SQL override.
 The Integration Service queries the lookup table or an in-memory
cache of the table for all incoming rows into the Lookup
transformation

Return port - Use only in unconnected Lookup transformations.

Designates the column of data you want to return based on the
lookup condition. You can designate one lookup port as the return
port

Use the following guidelines to configure lookup ports:

- If you delete lookup ports from a flat file lookup, the session fails.
- You can delete lookup ports from a relational lookup if the
mapping does not use the lookup port. This reduces the amount of
memory the Integration Service needs to run the session.

Lookup Properties
The following table describes the Lookup transformation properties:

79

Cached or un-cached lookup -
 Cache the lookup source to improve performance.
 If you cache the lookup source, you can use a dynamic or static
cache.
 By default, the lookup cache remains static and does not change
during the session.
 With a dynamic cache, the Integration Service inserts or updates
rows in the cache.
 When you cache the target table as the lookup source, you can
look up values in the cache to determine if the values exist in
the target.
 The Lookup transformation marks rows to insert or update the
target.
Lookup Query
 The Integration Service queries the lookup based on the ports
and properties you configure in the Lookup transformation.
 The Integration Service runs a default SQL statement when the
first row enters the Lookup transformation.
 If you use a relational lookup or a pipeline lookup against a
relational table, you can customize the default query with the
Lookup SQL Override property
 If you configure both the Lookup SQL Override and the Lookup
Source Filter properties, the Integration Service ignores the
Lookup Source Filter property
Default Lookup Query
 The default lookup query contains the following statements:
SELECT - The SELECT statement includes all the lookup ports in
the mapping. You can view the SELECT statement by generating
SQL using the Lookup SQL Override property. Do not add or
delete any columns from the default SQL statement.
ORDER BY - The ORDER BY clause orders the columns in the
same order they appear in the Lookup transformation. The
Integration Service generates the ORDER BY clause. You cannot
view this when you generate the default SQL using the Lookup
SQL Override property.
Overriding the Lookup Query
 Override the lookup query in the following circumstances:
Override the ORDER BY clause - Create the ORDER BY clause with
fewer columns to increase performance. When you override the
ORDER BY clause, you must suppress the generated ORDER BY
clause with a comment notation.
Note: If you use pushdown optimization, you cannot override the
ORDER BY clause or suppress the generated ORDER BY clause
with a comment notation.
A lookup table name or column names contains a reserved
word - If the table name or any column name in the lookup query
80
contains a reserved word, you must ensure that all reserved
words are enclosed in quotes.
Use parameters and variables - Use parameters and variables
when you enter a lookup SQL override. Use any parameter or
variable type that you can define in the parameter file. You can
enter a parameter or variable within the SQL statement, or you
can use a parameter or variable as the SQL query. For example,
you can use a session parameter, $ParamMyLkpOverride, as the
lookup SQL query, and set $ParamMyLkpOverride to the
SQL statement in a parameter file. The Designer cannot expand
parameters and variables in the query override and does not
validate it when you use a parameter or variable. The Integration
Service expands the parameters and variables when you run the
session.
A lookup column name contains a slash (/) character - When
generating the default lookup query, the Designer and Integration
Service replace any slash character (/) in the lookup column name
with an underscore character. To query lookup column names
containing the slash character, override the default lookup query,
replace the underscore characters with the slash character, and
enclose the column name in double quotes.
Add a WHERE clause - Use a lookup SQL override to add a
WHERE clause to the default SQL statement. You might want to
use the WHERE clause to reduce the number of rows included in
the cache. When you add a WHERE clause to a Lookup
transformation using a dynamic cache, use a Filter transformation
before the Lookup transformation to pass rows into the dynamic
cache that match the WHERE clause.
Note: The session fails if you include large object ports in a
WHERE clause.
Other - Use a lookup SQL override if you want to query lookup
data from multiple lookups or if you want to modify the data
queried from the lookup table before the Integration Service
caches the lookup rows. For example, use TO_CHAR to convert
dates to strings.
Guidelines for Overriding the Lookup Query
 Use the following guidelines when you override the lookup SQL
query:
- You can override the lookup SQL query for relational lookups.
- Generate the default query, and then configure the override.
This ensures that all the lookup/output ports are included in the
query. If you add or subtract ports from the SELECT statement,
the session fails.
- Add a source lookup filter to filter the rows that are added to
the lookup cache. This ensures the Integration Service inserts
rows in the dynamic cache and target table that match the
WHERE clause.
- To share the cache, use the same lookup SQL override for each
Lookup transformation.
- If you override the ORDER BY clause, the session fails if the
ORDER BY clause does not contain the condition ports in the
 same order they appear in the Lookup condition or if you do not
suppress the generated ORDER BY clause with the comment
notation.
- If you use pushdown optimization, you cannot override the
ORDER BY clause or suppress the generated ORDER BY clause
with comment notation.
- If the table name or any column name in the lookup query
contains a reserved word, you must enclose all reserved words in
quotes.
- You must choose the Use Any Value Lookup Policy on Mulitple
Match condition to override the lookup query for an uncached
lookup.
Handling Multiple Matches
- Use the first matching value, or use the last matching value
- Use any matching value
- Use all values
- Return an error. When the Lookup transformation uses a static
cache or no cache, the Integration Service marks the row as an
error. The Lookup transformation writes the row to the session
log by default, and increases the error count by one. When the
Lookup transformation has a dynamic cache, the Integration
Service fails the session when it encounters multiple matches.
The session fails while the Integration Service is caching the
lookup table or looking up the duplicate key values. Also, if you
configure the Lookup transformation to output old values on
updates, the Lookup transformation returns an error when it
encounters multiple matches. The transformation creates an
index based on the key ports instead of all Lookup transformation
ports.
Lookup Caches
 You can configure a Lookup transformation to cache the lookup
file or table.
 The Integration Service builds a cache in memory when it
processes the first row of data in a cached Lookup
transformation.
 It allocates memory for the cache based on the amount you
configure in the transformation or session properties.
 The Integration Service stores condition values in the index cache
and output values in the data cache.
 The Integration Service queries the cache for each row that
enters the transformation.
 The Integration Service also creates cache files by default in the
$PMCacheDir.
 If the data does not fit in the memory cache, the Integration
Service stores the overflow values in the cache files.
 When the session completes, the Integration Service releases
cache memory and deletes the cache files unless you configure
the Lookup transformation to use a persistent cache.
81
 When configuring a lookup cache, you can configure the
following options:
- Persistent cache
- Re-cache from lookup source
- Static cache
- Dynamic cache
- Shared cache
- Pre-build lookup cache
Note: You can use a dynamic cache for relational or flat file
lookups
Rules and Guidelines for Returning Multiple Rows
- The Integration Service caches all rows from the lookup source
for cached lookups.
- You can configure an SQL override for a cached or uncached
lookup that returns multiple rows.
- You cannot enable dynamic cache for a Lookup transformation
that returns multiple rows.
- You cannot return multiple rows from an unconnected Lookup
transformation.
- You can configure multiple Lookup transformations to share a
named cache if the Lookup transformations have matching
caching lookup on multiple match policies.
- A Lookup transformation that returns multiple rows cannot
share a cache with a Lookup transformation that returns one
matching row for each input row.
Configuring Unconnected Lookup Transformations
 An unconnected Lookup transformation is a Lookup
transformation that is not connected to a source or target.
 Call the lookup from another transformation with a :LKP
expression.
 You can perform the following tasks when you call a lookup from
an expression:
- Test the results of a lookup in an expression.
- Filter rows based on the lookup results.
- Mark rows for update based on the result of a lookup and
update slowly changing dimension tables.
- Call the same lookup multiple times in one mapping.
Database Deadlock Resilience
 The Lookup transformation is resilient to a database deadlock for
un-cached lookups.
 When a database deadlock error occurs, the session does not fail.
 The Integration Service attempts to re-execute the last statement
for a specified retry period.
 You can configure the number of deadlock retries and the
deadlock sleep interval for an Integration Service.
 These values also affect database deadlocks for the relational
writer. You can override these values at the session level as
custom properties.
 Configure following Integration Service Properties:
NumOfDeadlockRetries - The number of times the PowerCenter
Integration Service retries a target write on a database deadlock.
Minimum is 0. Default is 10. If you want the session to fail on
deadlock set NumOfDeadlockRetries to zero.
DeadlockSleep - Number of seconds before the PowerCenter
Integration Service retries a target write on database deadlock.
If a deadlock occurs, the Integration Service attempts to run the
statement. The Integration Service waits for a delay period
between each retry attempt. If all attempts fail due to deadlock,
the session fails. The Integration Service logs a message in the
session log whenever it retries a statement.
Tips for Lookup Transformations
Add an index to the columns used in a lookup condition:
 If you have privileges to modify the database containing a lookup
table, you can improve performance for both cached and un-
cached lookups.
 This is important for very large lookup tables.
 Since the Integration Service needs to query, sort, and compare
values in these columns, the index needs to include every
column used in a lookup condition.
Place conditions with an equality operator (=) first:
 If you include more than one lookup condition, place the
conditions in the following order to optimize lookup
performance:
- Equal to (=)
- Less than (<), greater than (>), less than or equal to (<=), greater
than or equal to (>=)
- Not equal to (!=)
Cache small lookup tables:
 Improve session performance by caching small lookup tables.
The result of the lookup query and processing is the same,
whether or not you cache the lookup table.
Join tables in the database:
 If the lookup table is on the same database as the source table in
the mapping and caching is not feasible, join the tables in the
source database rather than using a Lookup transformation.
Use a persistent lookup cache for static lookups:
 If the lookup source does not change between sessions,
configure the Lookup transformation to use a persistent lookup
cache.
 The Integration Service then saves and reuses cache files from
session to session, eliminating the time required to read the
lookup source.
82
Configure a pipeline Lookup transformation to improve
performance when processing a relational or flat file lookup
source:
 You can create partitions to process a relational or flat file lookup
source when you define the lookup source as a source qualifier.
Configure a non-reusable pipeline Lookup transformation and
create partitions in the partial pipeline that processes the
lookup source.
Lookup Caches
 The Integration Service builds a cache in memory when it
processes the first row of data in a cached Lookup
transformation.
 It allocates memory for the cache based on the amount you
configure in the transformation or session properties.
 The Integration Service stores condition values in the index cache
and output values in the data cache.
 The Integration Service queries the cache for each row that
enters the transformation
 If the data does not fit in the memory cache, the Integration
Service stores the overflow values in the cache files.
 When the session completes, the Integration Service releases
cache memory and deletes the cache files unless you configure
the Lookup transformation to use a persistent cache.
 If you use a flat file or pipeline lookup, the Integration Service
always caches the lookup source.
 If you configure a flat file lookup for sorted input, the Integration
Service cannot cache the lookup if the condition columns are
not grouped.
 If the columns are grouped, but not sorted, the Integration
Service processes the lookup as if you did not configure sorted
input
When you configure a lookup cache, you can configure the
following cache settings:
Building caches -
 You can configure the session to build caches sequentially or
concurrently.
 When you build sequential caches, the Integration Service creates
caches as the source rows enter the Lookup transformation.
 When you configure the session to build concurrent caches, the
Integration Service does not wait for the first row to enter the
Lookup transformation before it creates caches.
 Instead, it builds multiple caches concurrently.
Persistent cache -
 You can save the lookup cache files and reuse them the next time
the Integration Service processes a Lookup transformation
configured to use the cache.
 If the lookup table does not change between sessions, you can
configure the Lookup transformation to use a persistent lookup
cache

Re-cache from source -

 If the persistent cache is not synchronized with the lookup table,
you can configure the Lookup transformation to rebuild the
lookup cache.

Static cache -
 You can configure a static, or read-only, cache for any lookup
source.
 By default, the Integration Service creates a static cache.
 It caches the lookup file or table and looks up values in the cache
for each row that comes into the transformation.
 When the lookup condition is true, the Integration Service
returns a value from the lookup cache.
 The Integration Service does not update the cache while it
 The Integration Service can build lookup caches for connected
processes the Lookup transformation.
Lookup transformations in the following ways:
- Sequential caches
Dynamic cache -
- Concurrent caches
 To cache a table, flat file, or source definition and update the
 The Integration Service builds caches for unconnected Lookup
cache, configure a Lookup transformation with dynamic cache.
transformations sequentially regardless of how you configure
 The Integration Service dynamically inserts or updates data in the
cache building.
lookup cache and passes the data to the target.
 The dynamic cache is synchronized with the target.
Sequential Caches
 By default, the Integration Service builds a cache in memory
Shared cache -
when it processes the first row of data in a cached Lookup
 You can share the lookup cache between multiple
transformation.
transformations.
 The Integration Service creates each lookup cache in the pipeline
 You can share an unnamed cache between transformations in the
sequentially.
same mapping.
 The Integration Service waits for any upstream active
 You can share a named cache between transformations in the
transformation to complete processing before it starts
same or different mappings.
processing the rows in the Lookup transformation.
 Lookup transformations can share unnamed static caches within
 The Integration Service does not build caches for a downstream
the same target load order group if the cache sharing rules
Lookup transformation until an upstream Lookup transformation
match. Lookup transformations cannot share dynamic cache
completes building a cache.
within the same target load order group.
 For example, the following mapping contains an unsorted
Aggregator transformation followed by two Lookup
 When you do not configure the Lookup transformation for
transformations:
caching, the Integration Service queries the lookup table for
each input row.
 The result of the Lookup query and processing is the same,
whether or not you cache the lookup table.
 However, using a lookup cache can increase session performance.

 Configuring sequential caching may allow you to avoid building

lookup caches unnecessarily.
 For example, a Router transformation might route data to one
pipeline if a condition resolves to true, and it might route data
to another pipeline if the condition resolves to false.

83
 In this case, a Lookup transformation might not receive data at
all.
Concurrent Caches
 You can configure the Integration Service to create lookup caches
concurrently.
 You may be able to improve session performance using
concurrent caches.
 Performance may especially improve when the pipeline contains
an active transformation upstream of the Lookup
transformation.
 You may want to configure the session to create concurrent
caches if you are certain that you will need to build caches for
each of the Lookup transformations in the session.
Dynamic Lookup Cache
 The following list describes some situations when you use a
dynamic lookup cache:
Updating a master customer table with new and updated
customer information -
 Use a Lookup transformation to perform a lookup on the
customer table to determine if a customer exists in the target.
 The cache represents the customer table.
 The Lookup transformation inserts and updates rows in the cache
as it passes rows to the target.
Inserting rows into a master customer table from multiple real-
time sessions -
 Use a Lookup transformation in each session to perform a lookup
on the same customer table.
 Each Lookup transformation inserts rows into the customer table
and it inserts them in the dynamic lookup cache.
 For more information about synchronizing dynamic cache
between multiple sessions, see “Synchronizing Cache with the
Lookup Source” on page 278.
Loading data into a slowly changing dimension table and a fact
table -
 Create two pipelines and configure a Lookup transformation that
performs a lookup on the dimension table.
 Use a dynamic lookup cache to load data to the dimension table.
 Use a static lookup cache to load data to the fact table, and
specify the name of the dynamic cache from the first pipeline.
Reading a flat file that is an export from a relational table -
 Read data from a Teradata table when the ODBC connection is
slow.
 You can export the Teradata table contents to a flat file and use
the file as a lookup source.
84
 Configure the Teradata table as a relational target in the mapping
and pass the lookup cache changes back to the Teradata table.
Note - When you create multiple partitions in a pipeline that use
a dynamic lookup cache, the Integration Service creates one
memory cache and one disk cache for each transformation.
However, if you add a partition point at the Lookup
transformation, the Integration Service creates one memory
cache for each partition.
Dynamic Lookup Properties -
A Lookup transformation with a dynamic cache has the following
properties:
NewLookupRow - The Designer adds this port to a Lookup
transformation configured to use a dynamic cache. Indicates with
a numeric value whether the Integration Service inserts or
updates the row in the cache, or makes no change to the cache.
To keep the lookup cache and the target table synchronized, pass
rows to the target when the NewLookupRow value is equal to 1
or 2.
Associated Expression - Associate lookup ports or the associated
ports with an expression, an input/output port, or a sequence ID.
The Integration Service uses the data in the associated expression
to insert or update rows in the lookup cache. If you associate a
sequence ID, the Integration Service generates a primary key for
inserted rows in the lookup cache.
Ignore Null Inputs for Updates - The Designer activates this port
property for lookup/output ports when you configure the Lookup
transformation to use a dynamic cache. Select this property when
you do not want the Integration Service to update the column in
the cache when the data in this column contains a null value.
Ignore in Comparison - The Designer activates this port property
for lookup/output ports not used in the lookup condition when
you configure the Lookup transformation to use a dynamic cache.
The Integration Service compares the values in all lookup ports
with the values in their associated input ports by default. Select
this property if you want the Integration Service to ignore the
port when it compares values before updating a row.
Update Dynamic Cache Condition - Allow the Integration Service
to update the dynamic cache conditionally. You can create a
Boolean expression that determines whether to update the cache
for an input row. Or, you can enable the Integration Service to
update the cache with an expression result for an input row. The
expression can contain values from the input row or the lookup
cache
Rules and Guidelines for Dynamic Lookup Caches
Use the following guidelines when you use a dynamic lookup
cache:
- You cannot share the cache between a dynamic Lookup
transformation and static Lookup transformation in the same
target load order group.
- You can create a dynamic lookup cache from a relational table,
flat file, or source qualifier transformation.
- The Lookup transformation must be a connected
transformation.
- Use a persistent or a non-persistent cache.
- If the dynamic cache is not persistent, the Integration Service
always rebuilds the cache from the database, even if you do not
enable Re-cache from Lookup Source. When you synchronize
dynamic cache files with a lookup source table, the Lookup
transformation inserts rows into the lookup source table and the
dynamic lookup cache. If the source row is an update row, the
Lookup transformation updates the dynamic lookup cache only.
- You can only create an equality lookup condition. You cannot
look up a range of data in dynamic cache.
- Associate each lookup port that is not in the lookup condition
with an input port, sequence ID, or expression.
- Use a Router transformation to pass rows to the cached target
when the NewLookupRow value equals one or two. Use the
Router transformation to drop rows when the NewLookupRow
value equals zero, or you can output those rows to a different
target.
- Verify that you output the same values to the target that the
Integration Service writes to the lookup cache. When you choose
to output new values on update, only connect lookup/output
ports to the target table instead of input/output ports. When you
choose to output old values on update, add an Expression
transformation after the Lookup transformation and before the
Router transformation. Add output ports in the Expression
transformation for each port in the target table and create
expressions to ensure you do not output null input values to the
target.
- When you use a lookup SQL override, map the correct columns
to the appropriate targets for lookup.
- When you add a WHERE clause to the lookup SQL override, use
a Filter transformation before the Lookup transformation. This
ensures the Integration Service inserts rows in the dynamic cache
and target table that match the WHERE clause.
- When you configure a reusable Lookup transformation to use a
dynamic cache, you cannot edit the condition or disable the
Dynamic Lookup Cache property in a mapping.
- Use Update Strategy transformations after the Lookup
transformation to flag the rows for insert or update for the target.
- Use an Update Strategy transformation before the Lookup
transformation to define some or all rows as update if you want
to use the Update Else Insert property in the Lookup
transformation.
- Set the row type to Data Driven in the session properties.
85
- Select Insert and Update as Update for the target table options
in the session properties
K. NORMALIZER
 The Normalizer transformation generates a key for each source
row.
 This generated key remains same for the output group created
for each source row.
 The Integration Service increments the generated key sequence
number each time it processes a source row
You can create a VSAM Normalizer transformation or a pipeline
Normalizer transformation:
VSAM Normalizer transformation -
 A non-reusable transformation that is a Source Qualifier
transformation for a COBOL source.
 The Mapping Designer creates VSAM Normalizer columns from a
COBOL source in a mapping.
 The column attributes are read-only.
 The VSAM Normalizer receives a multiple-occurring source
column through one input port.
Pipeline Normalizer transformation -
 A transformation that processes multiple-occurring data from
relational tables or flat files.
 You create the columns manually and edit them in the
Transformation Developer or Mapping Designer.
 The pipeline Normalizer transformation represents multiple-
occurring columns with one input port for each source column
occurrence.
 When a Normalizer transformation receives more than one type
of data from a COBOL source, you need to connect the
Normalizer output ports to different targets based on the type
of data in each row.
Troubleshooting Normalizer Transformations
I cannot edit the ports in my Normalizer transformation when
using a relational source.
When you create ports manually, add them on the Normalizer
tab in the transformation, not the Ports tab.
Importing a COBOL file failed with numberrors. What should I
do?
Verify that the COBOL program follows the COBOL standard,
including spaces, tabs, and end of line characters.
The COBOL file headings should be similar to the following text:
 identification division.
program-id. mead.
environment division.
select file-one assign to "fname".
data division.
file section.
fd FILE-ONE.
The Designer does not read hidden characters in the COBOL
program. Use a text-only editor to make changes to the COBOL
file. Do not use Word or Wordpad. Remove extra spaces.
A session that reads binary data completed, but the information
in the target table is incorrect.
Edit the session in the Workflow Manager and verify that the
source file format is set correctly. The file format might be
EBCDIC or ASCII. The number of bytes to skip between records
must be set to 0.
I have a COBOL field description that uses a non-IBM COMP
type. How should I import the source?
In the source definition, clear the IBM COMP option.
In my mapping, I use one Expression transformation and one
Lookup transformation to modify two output ports from the
Normalizer transformation. The mapping concatenates them
into a single transformation. All the ports are under the same
level. When I check the data loaded in the target, it is incorrect.
Why is that?
You can only concatenate ports from level one. Remove the
concatenation.
L. RANK
 You can select only the top or bottom rank of data with Rank
transformation.
 Use a Rank transformation to return the largest or smallest
numeric value in a port or group.
 You can also use a Rank transformation to return the strings at
the top or the bottom of a session sort order.
 During the session, the Integration Service caches input data until
it can perform the rank calculations.
86
 The Rank transformation differs from the transformation
functions MAX and MIN, in that it lets you select a group of
top or bottom values, not just one value.
 You can also write expressions to transform data or perform
calculations. You can also create local variables and write non-
aggregate expressions.
 When the Integration Service runs in the ASCII data movement
mode, it sorts session data using a binary sort order.
 When the Integration Service runs in Unicode data movement
mode, the Integration Service uses the sort order configured for
the session
Rank Caches
 During a session, the Integration Service compares an input row
with rows in the data cache.
 If the input row outranks a cached row, the Integration Service
replaces the cached row with the input row.
 If you configure the Rank transformation to rank across multiple
groups, the Integration Service ranks incrementally for each
group it finds.
 The Integration Service stores group information in an index
cache and row data in a data cache.
 If you create multiple partitions in a pipeline, the Integration
Service creates separate caches for each partition.
 When you create a Rank transformation, you can configure the
following properties:
- Enter a cache directory.
- Select the top or bottom rank.
- Select the input/output port that contains values used to
determine the rank.
- You can select only one port to define a rank.
- Select the number of rows falling within a rank.
- Define groups for ranks, such as the 10 least expensive
products for each manufacturer
M. ROUTER
 A Filter transformation tests data for one condition and drops the
rows of data that do not meet the condition.
 However, a Router transformation tests data for one or more
conditions and gives you the option to route rows of data that
do not meet any of the conditions to a default output group
 When you use a Router transformation in a mapping, the
Integration Service processes the incoming data only once.
 When you use multiple Filter transformations in a mapping, the
Integration Service processes the incoming data for each
transformation.
 You cannot modify or delete output ports or their properties
 The Integration Service determines the order of evaluation for
each condition based on the order of the connected output
groups.
 The Integration Service processes user-defined groups that are
connected to a transformation or a target in a mapping.
 The Integration Service only processes user-defined groups that
are not connected in a mapping if the default group is
connected to a transformation or a target.
 If a row meets more than one group filter condition, the
Integration Service passes this row multiple times
 The Designer deletes the default group when you delete the last
user-defined group from the list.
N. SEQUENCE GENERATOR
 The Sequence Generator transformation generates numeric
values.
 Use the Sequence Generator to create unique primary key values,
replace missing primary keys, or cycle through a sequential
range of numbers.
 The Sequence Generator transformation is a connected
transformation.
 It contains two output ports that you can connect to one or more
transformations.
 The Integration Service generates a block of sequence numbers
each time a block of rows enters a connected transformation
 If you connect CURRVAL, the Integration Service processes one
row in each block.
 When NEXTVAL is connected to the input port of another
transformation, the Integration Service generates a sequence of
numbers.
 When CURRVAL is connected to the input port of another
transformation, the Integration Service generates the NEXTVAL
value plus the Increment By value
 You can make a Sequence Generator reusable, and use it in
multiple mappings
 You can use a range of values from 1 to
9,223,372,036,854,775,807 with the smallest interval of 1.
 The Sequence Generator transformation has two output ports:
NEXTVAL and CURRVAL. You cannot edit or delete these ports.
 Likewise, you cannot add ports to the transformation.
NEXTVAL
 Connect NEXTVAL to multiple transformations to generate unique
values for each row in each transformation.
 Use the NEXTVAL port to generate sequence numbers by
connecting it to a downstream transformation or target.
 You connect the NEXTVAL port to generate the sequence based
on the Current Value and Increment By properties.
87
 If the Sequence Generator is not configured to cycle through the
sequence, the NEXTVAL port generates sequence numbers up to
the configured End Value.
 For example, you might connect NEXTVAL to two targets in a
mapping to generate unique primary key values.
 The Integration Service creates a column of unique primary key
values for each target table.
 The column of unique primary key values is sent to one target
table as a block of sequence numbers.
 The other target receives a block of sequence numbers from the
Sequence Generator transformation after the first target
receives the block of sequence numbers.
 For example, you configure the Sequence Generator
transformation as follows:
Current Value = 1, Increment By = 1.
 The Integration Service generates the following primary key
values for the T_ORDERS_PRIMARY and T_ORDERS_FOREIGN
target tables:
T_ORDERS_PRIMARY TABLE: T_ORDERS_FOREIGN TABLE:
PRIMARY KEY PRIMARY KEY
1 6
2 7
3 8
4 9
5 10
 If you want the same values to go to more than one target that
receives data from a single transformation, you can connect a
Sequence Generator transformation to that preceding
transformation.
 The Integration Service processes the values into a block of
sequence numbers.
 This allows the Integration Service to pass unique values to the
transformation, and then route rows from the transformation to
targets.
 The following figure shows a mapping with a Sequence Generator
that passes unique values to the Expression transformation.
 The Expression transformation populates both targets with
identical primary key values.
 CURRVAL
 CURRVAL is NEXTVAL plus the Increment By value. You typically
only connect the CURRVAL port when the NEXTVAL port is
already connected to a downstream transformation.
 When a row enters a transformation connected to the CURRVAL
port, the Integration Service passes the last created NEXTVAL
value plus one.
 The following figure shows connecting CURRVAL and NEXTVAL
ports to a target:

 For example, you configure the Sequence Generator

transformation as follows:
Current Value = 1, Increment By = 1.

 The Integration Service generates the following values for

NEXTVAL and CURRVAL:
NEXTVAL CURRVAL
1 2
2 3
3 4
4 5
5 6

 If you connect the CURRVAL port without connecting the

NEXTVAL port, the Integration Service passes a constant value
for each row.
 When you connect the CURRVAL port in a Sequence Generator
transformation, the Integration Service processes one row in
each block.
 You can optimize performance by connecting only the NEXTVAL
port in a mapping.

Sequence Generator Transformation Properties

 The following table describes the Sequence
transformation properties you can configure:
Generator
End Value
 End Value is the maximum value you want the Integration Service
to generate.
 If the Integration Service reaches the end value and the
Sequence Generator is not configured to cycle through the
sequence, the session fails with the following error message:
TT_11009 Sequence Generator Transformation: Overflow error.

Number of Cached Values

 When you have a reusable Sequence Generator transformation in
several sessions and the sessions run at the same time, use
Number of Cached Values to ensure each session receives
unique values in the sequence.
 By default, Number of Cached Values is set to 1000 for reusable
Sequence Generators.
 For non-reusable Sequence Generator, Number of Cached
Values is set to 0 by default

88
 Reset
 If you select Reset for a non-reusable Sequence Generator
transformation, the Integration Service generates values based
on the original current value each time it starts the session.
 Otherwise, the Integration Service updates the current value to
reflect the last-generated value plus one, and then uses the
updated value the next time it uses the Sequence Generator
transformation.
 For example, you might configure a Sequence Generator
transformation to create values from 1 to 1,000 with an
increment of 1, and a current value of 1 and choose Reset.
 During the first session run, the Integration Service generates
numbers 1 through 234.
 Each subsequent time the session runs, the Integration Service
again generates numbers beginning with the current value of 1.
 If you do not select Reset, the Integration Service updates the
current value to 235 at the end of the first session run.
 The next time it uses the Sequence Generator transformation,
the first value generated is 235.
Note: Reset is disabled for reusable Sequence Generator
transformations.
O. SORTER
 You can sort data from relational or flat file sources
 When you specify multiple ports for the sort key, the Integration
Service sorts each port sequentially.
 The order the ports appear in the Ports tab determines the
succession of sort operations.
 The Sorter transformation treats the data passing through each
successive sort key port as a secondary sort of the previous port.
Sorter Cache
 You can configure a numeric value for the Sorter cache, or you
can configure the Integration Service to determine the cache
size at run time
 If you configure the Integration Service to determine the cache
size, you can also configure a maximum amount of memory for
the Integration Service to allocate to the cache.
 If the total configured session cache size is 2 GB (2,147,483,648
bytes) or greater, you must run the session on a 64-bit
Integration Service
 Before starting the sort operation, the Integration Service
allocates the amount of memory configured for the Sorter cache
size.
 If the Integration Service runs a partitioned session, it allocates
the specified amount of Sorter cache memory for each partition
 If it cannot allocate enough memory, the Integration Service fails
the session.
 For best performance, configure Sorter cache size with a value
less than or equal to the amount of available physical RAM on
the Integration Service machine.
89
 Allocate at least 16 MB (16,777,216 bytes) of physical memory to
sort data using the Sorter transformation.
 Sorter cache size is set to 16,777,216 bytes by default.
 The Integration Service requires disk space of at least twice the
amount of incoming data when storing data in the work
directory
 Use the following formula to determine the size of incoming data:
number_of_input_rows [( S column_size) + 16]
P. SOURCE QUALIFIER
 The Source Qualifier transformation represents the rows that the
Integration Service reads when it runs a session.
 Use the Source Qualifier transformation to complete the
following tasks:
Join data originating from the same source database -
You can join two or more tables with primary key foreign key
relationships by linking the sources to one Source Qualifier
transformation.
Filter rows when the Integration Service reads source data -
If you include a filter condition, the Integration Service adds a
WHERE clause to the default query.
Specify an outer join rather than the default inner join -
If you include a user-defined join, the Integration Service replaces
the join information specified by the metadata in the SQL query.
Specify sorted ports -
If you specify a number for sorted ports, the Integration Service
adds an ORDER BY clause to the default SQL query.
Select only distinct values from the source -
If you choose Select Distinct, the Integration Service adds a
SELECT DISTINCT statement to the default SQL query.
Create a custom query to issue a special SELECT statement for
the Integration Service to read source data -
For example, you might use a custom query to perform aggregate
calculations
 If the datatypes in the source definition and Source Qualifier
transformation do not match, the Designer marks the mapping
invalid when you save it.
 You specify a target load order based on the Source Qualifier
transformations in a mapping
 If one Source Qualifier transformation provides data for multiple
targets, you can enable constraint-based loading in a session to
have the Integration Service load data based on target table
primary and foreign key relationships
 You can use parameters and variables in the SQL query, user-
defined join, source filter, and pre- and post-session SQL
commands of a Source Qualifier transformation
 The Integration Service first generates an SQL query and
expands each parameter or variable.
 It replaces each mapping parameter, mapping variable, and
workflow variable with its start value.
 Then it runs the query on the source database
Source Qualifier Transformation Properties
Default Query
 For relational sources, the Integration Service generates a query
for each Source Qualifier transformation when it runs a session.
 The default query is a SELECT statement for each source column
used in the mapping.
 In other words, the Integration Service reads only the columns
that are connected to another transformation
 If any table name or column name contains a database reserved
word, you can create and maintain a file, reswords.txt,
containing reserved words.
 When the Integration Service initializes a session, it searches for
reswords.txt in the Integration Service installation directory.
 If the file exists, the Integration Service places quotes around
matching reserved words when it executes SQL against the
database.
 If you override the SQL, you must enclose any reserved word in
quotes.
 When a mapping uses related relational sources, you can join
both sources in one Source Qualifier transformation.
 During the session, the source database performs the join before
passing data to the Integration Service
90
 Use the Joiner transformation for heterogeneous sources and to
join flat files.
Viewing the Default Query
 Do not connect to the source database. You only connect to the
source database when you enter an SQL query that overrides
the default query.
 You must connect the columns in the Source Qualifier
transformation to another transformation or target before you
can generate the default query
Default Join -
 When you join related tables in one Source Qualifier
transformation, the Integration Service joins the tables based on
the related keys in each table
 This default join is an inner equijoin, using the following syntax in
the WHERE clause:
Source1.column_name = Source2.column_name
 The columns in the default join must have:
- A primary key-foreign key relationship
- Matching datatypes
Custom Join -
You might need to override the default join under the following
circumstances:
- Columns do not have a primary key-foreign key relationship.
- The datatypes of columns used for the join do not match.
- You want to specify a different type of join, such as an outer
join.
Adding an SQL Query
 The Source Qualifier transformation provides the SQL Query
option to override the default query.
 You can enter an SQL statement supported by the source
database.
 Before entering the query, connect all the input and output ports
you want to use in the mapping.
Entering a User-Defined Join
 Entering a user-defined join is similar to entering a custom SQL
query.
 However, you only enter the contents of the WHERE clause, not
the entire query.
 When you perform an outer join, the Integration Service may
insert the join syntax in the WHERE clause or the FROM clause
of the query, depending on the database syntax.
 When you add a user-defined join, the Source Qualifier
transformation includes the setting in the default SQL query.
 However, if you modify the default query after adding a user-
defined join, the Integration Service uses only the query
 defined in the SQL Query property of the Source Qualifier
transformation.
 When including a string mapping parameter or variable, use a
string identifier appropriate to the source system.
 For most databases, you need to enclose the name of a string
parameter or variable in single quotes.
Outer Join Support
 The Integration Service supports two kinds of outer joins:
Left - Integration Service returns all rows for the table to the left
of the join syntax and the rows from both tables that meet the
join condition
Right - Integration Service returns all rows for the table to the
right of the join syntax and the rows from both tables that meet
the join condition
Informatica Join Syntax
 When you enter join syntax, use the Informatica or database-
specific join syntax.
 When you use the Informatica join syntax, the Integration Service
translates the syntax and passes it to the source database during
the session.
Note: Always use database-specific syntax for join conditions.
 When you use Informatica join syntax, enclose the entire join
statement in braces ({Informatica syntax}).
 When you use database syntax, enter syntax supported by the
source database without braces.
Normal Join Syntax
{ source1 INNER JOIN source2 on join_condition }
Left Outer Join Syntax
{ source1 LEFT OUTER JOIN source2 on join_condition }
Right Outer Join Syntax
{ source1 RIGHT OUTER JOIN source2 on join_condition }
Entering a Source Filter
 You can enter a source filter to reduce the number of rows the
Integration Service queries.
 If you include the string ‘WHERE’ or large objects in the source
filter, the Integration Service fails the session.
 The Source Qualifier transformation includes source filters in
the default SQL query.
 If, however, if you modify the default query after adding a
source filter, the Integration Service uses only the query
defined in the SQL query portion of the Source Qualifier
transformation.
 You can use a parameter or variable as the source filter or include
parameters and variables within the source filter
91
Sorted Ports -
 When you use sorted ports, the Integration Service adds the
ports to the ORDER BY clause in the default query.
 The sorted ports are applied on the connected ports rather than
the ports that start at the top of the SQ
 Use sorted ports for relational sources only.
 When using sorted ports, the sort order of the source database
must match the sort order configured for the session.
 To ensure data is sorted as the Integration Service requires, the
database sort order must be the same as the user-defined
session sort order
 The Source Qualifier transformation includes the number of
sorted ports in the default SQL query.
 However, if you modify the default query after choosing the
Number of Sorted Ports, the Integration Service uses only the
query defined in the SQL Query property.
Pre and post-session SQL commands -
 You can add pre- and post-session SQL commands on the
Properties tab in the Source Qualifier transformation
 The Integration Service runs pre-session SQL commands against
the source database before it reads the source.
 It runs post-session SQL commands against the source database
after it writes to the target
Guidelines for pre- and post-session SQL commands in SQ:
- Use any command that is valid for the database type. However,
the Integration Service does not allow nested comments, even
though the database might.
- You can use parameters and variables in source pre- and post-
session SQL commands or you can use a parameter or variable as
the command. Use any parameter or variable type that you can
define in the parameter file.
- Use a semicolon (;) to separate multiple statements. The
Integration Service issues a commit after each statement.
- The Integration Service ignores semicolons within /*...*/.
- If you need to use a semicolon outside of comments, you can
escape it with a backslash (\). When you escape the semicolon,
the Integration Service ignores the backslash, and it does not use
the semicolon as a statement separator.
- The Designer does not validate the SQL.
Note: You can also enter pre- and post-session SQL commands on
the Properties tab of the target instance in a mapping
Troubleshooting Source Qualifier Transformations
I cannot connect a source definition to a target definition.
 You cannot directly connect sources to targets. Instead, you need
to connect them through a Source Qualifier transformation for
relational and flat file sources, or through a Normalizer
transformation for COBOL sources.
 I cannot connect multiple sources to one target.
 The Designer does not allow you to connect multiple Source
Qualifier transformations to a single target. There are two
workarounds:
Reuse targets - Since target definitions are reusable, you can add
the same target to the mapping multiple times. Then connect
each Source Qualifier transformation to each target.
Join the sources in a Source Qualifier transformation. Then
remove the WHERE clause from the SQL query.
The source has QNAN (not a number) values in some columns,
but the target shows 1.#QNAN.
 Operating systems have different string representations of NaN.
The Integration Service converts QNAN values to 1.#QNAN on
Win64EMT platforms. 1.#QNAN is a valid representation of
QNAN.
I entered a custom query, but it is not working when I run the
workflow containing the session.
 Be sure to test this setting for the Source Qualifier transformation
before you run the workflow. Reopen the dialog box in which
you entered the custom query. You can connect to a database
and click the Validate button to test the SQL. The Designer
displays any errors.
 The most common reason a session fails is because the
database login in both the session and Source Qualifier
transformation is not the table owner. You need to specify the
table owner in the session and when you generate the SQL
Query in the Source Qualifier transformation.
 You can test the SQL Query by cutting and pasting it into the
database client tool (such as Oracle Net) to see if it returns an
error.
I used a mapping variable in a source filter and now the session
fails.
 Try testing the query by generating and validating the SQL in the
Source Qualifier transformation. If the variable or parameter is a
string, you probably need to enclose it in single quotes. If it is a
datetime variable or parameter, you might need to change its
format for the source system.
Q. SQL TRANSFORMATION
 The SQL transformation processes SQL queries midstream in a
pipeline.
 You can insert, delete, update, and retrieve rows from a
database.
 You can pass the database connection information to the SQL
transformation as input data at run time.
 The transformation processes external SQL scripts or SQL queries
that you create in an SQL editor.
 The SQL transformation processes the query and returns rows
and database errors
92
When you create an SQL transformation, you configure the
following options:
Mode - The SQL transformation runs in one of the following
modes:
Script mode: The SQL transformation runs ANSI SQL scripts that
are externally located. You pass a script name to the
transformation with each input row. The SQL transformation
outputs one row for each input row.
Query mode: The SQL transformation executes a query that you
define in a query editor. You can pass strings or parameters to the
query to define dynamic queries or change the selection
parameters. You can output multiple rows when the query has a
SELECT statement.
Passive or active transformation - The SQL transformation is an
active transformation by default. You can configure it as a
passive transformation when you create the transformation.
Database type - The type of database the SQL transformation
connects to.
Connection type - Pass database connection information to the
SQL transformation or use a connection object.
Script Mode –
 An SQL transformation running in script mode runs SQL scripts
from text files.
 You pass each script file name from the source to the SQL
transformation ScriptName port.
 The script file name contains the complete path to the script file.
 When you configure the transformation to run in script mode,
you create a passive transformation
 The transformation returns one row for each input row
 The Integration Service ignores the output of any SELECT
statement you include in the SQL script.
 The SQL transformation in script mode does not output more
than one row of data for each input row.
 You cannot use nested scripts where the SQL script calls another
SQL script.
 A script cannot accept run-time arguments
Query Mode -
 It executes an SQL query that you define in the transformation.
 When you configure the SQL transformation to run in query
mode, you create an active transformation
 The transformation can return multiple rows for each input row.
 When you create a query, the SQL Editor validates the port
names in the query.
 It also verifies that the ports you use for string substitution are
string datatypes.
 The SQL Editor does not validate the syntax of the SQL query
 You can create the following types of SQL queries in the SQL
transformation:
Static SQL query - The query statement does not change, but you
can use query parameters to change the data. The Integration
Service prepares the query once and runs the query for all input
rows.
Dynamic SQL query - You can change the query statements and
the data. The Integration Service prepares a query for each input
row.
Static Query -
 When you create a static query, the Integration Service prepares
the SQL procedure once and executes it for each row.
 When you create a dynamic query, the Integration Service
prepares the SQL for each input row.
 You can optimize performance by creating static queries
 Bind a parameter to an input port - SQL Editor encloses the
name in question marks (?)
 When the SQL query contains a SELECT statement, the output
ports must be in the same order as the columns in the SELECT
statement.
Dynamic Query -
 To change a query statement, configure a string variable in the
query for the portion of the query you want to change.
 To configure the string variable, identify an input port by name in
the query and enclose the name with the tilde (~).
 The query changes based on the value of the data in the port.
 The transformation input port that contains the query parameter
must be a string datatype.
 You can pass the full query or pass part of the query in an input
port:
Full query - You can substitute the entire SQL query with query
statements from source data.
Partial query - You can substitute a portion of the query
statement, such as the table name.
 You can add pass-through ports to the SQL transformation
 When the source row contains a SELECT query statement, the
SQL transformation returns the data in the pass-through port in
each row it returns from the database.
93
 If the query result contains multiple rows, the SQL transformation
repeats the pass-through data in each row
Passive Mode Configuration
 When you create a SQL transformation, you can configure the
SQL transformation to run in passive mode instead of active
mode.
 You cannot change the mode after you create the transformation
Guidelines to configure the SQL transformation to run in passive
mode:
- If a SELECT query returns more than one row, the Integration
Service returns the first row and an error to the SQLError port.
The error states that the SQL transformation generated multiple
rows.
- If the SQL query has multiple SQL statements, then the
Integration Service executes all the statements. The Integration
Service returns data for the first SQL statement only. The SQL
transformation returns one row. The SQLError port contains the
errors from all the SQL statements. When multiple errors occur,
they are separated by semi-colons in the SQLError port.
- If the SQL query has multiple SQL statements and a statistics
port is enabled, the Integration Service returns the data and
statistics for the first SQL statement. The SQLError port contains
the errors for all the SQL statements.
Guidelines to configure the SQL transformation to run in query
mode:
- The number and the order of the output ports must match the
number and order of the fields in the query SELECT clause.
- The native datatype of an output port in the transformation
must match the datatype of the corresponding column in the
database. The Integration Service generates a row error when the
datatypes do not match.
- When the SQL query contains an INSERT, UPDATE, or DELETE
clause, the transformation returns data to the SQLError port,
the pass-through ports, and the NumRowsAffected port when it
is enabled. If you add output ports the ports receive NULL data
values.
- When the SQL query contains a SELECT statement and the
transformation has a pass-through port, the transformation
returns data to the pass-through port whether or not the query
returns database data. The SQL transformation returns a row
with NULL data in the output ports.
- You cannot add the "_output" suffix to output port names that
you create.
- You cannot use the pass-through port to return data from a
SELECT query.
- When the number of output ports is more than the number of
columns in the SELECT clause, the extra ports receive a NULL
value.
 - When the number of output ports is less than the number of
columns in the SELECT clause, the Integration Service generates
a row error.
- You can use string substitution instead of parameter binding in a
query. However, the input ports must be string datatypes.
Ways to connect the SQL transformation to a database:
Static connection - Configure the connection object in the
session. You must first create the connection object in Workflow
Manager.
Logical connection - Pass a connection name to the SQL
transformation as input data at run time. You must first create
the connection object in Workflow Manager.
Full database connection - Pass the connect string, user name,
password, and other connection information to the SQL
transformation input ports at run time
Note: If a session has multiple partitions, the SQL transformation
creates a separate database connection for each partition.
 The following transaction control SQL statements are not valid
with the SQL transformation:
SAVEPOINT - Identifies a rollback point in the transaction.
SET TRANSACTION - Changes transaction options.
 When you have high availability, the SQL transformation provides
database connection resiliency for static and dynamic
connections. When the Integration Service fails to connect to
the database, it retries the connection.
 You can configure the connection retry period for a connection
 When the Integration Service cannot connect to the database in
the time period that you configure, it generates a row error for a
dynamic connection or fails the session for a static connection.
Database Deadlock Resiliency
 The SQL transformation is resilient to database deadlock errors
when you enable the Session Retry on Deadlock session
property.
 The SQL transformation is resilient to database deadlock errors
in Query mode but it is not resilient to deadlock errors in Script
mode.
 If a deadlock occurs in Query mode, the Integration Service tries
to reconnect to the database for the number of deadlock retries
that you configure.
 When a deadlock occurs, the Integration Service retries the SQL
statements in the current row if the current row has no DML
statements.
 If the row contains a DML statement such as INSERT, UPDATE, or
DELETE, the Integration Service does not process the current
row again
94
For a dynamic connection, if the retry attempt fails, the
Integration Service returns an error in the SQLError port. The
Integration Service processes the next statement based on the
Continue on SQL Error within Row property. If the property is
disabled, the Integration Service skips the current row. If the
current row contains a DML statement such as INSERT, UPDATE,
or DELETE, the Integration Service increments the error counts.
For a static connection, if the retry attempts fail, the Integration
Service returns an error in the SQLError port. If the current row
contains a DML statement, then the Integration Service fails the
session. The Integration Service processes the next statement
based on Continue on SQL Error within a Row property. If the
property is disabled the Integration Service skips the current row.
<Print - 361 to 364>
R. STORED PROCEDURE
There are three types of data that pass between the Integration
Service and the stored procedure:
- Input/output parameters
- Return values
- Status codes
 If a stored procedure returns a result set rather than a single
return value, the Stored Procedure transformation takes only
the first value returned from the procedure
Status Codes
 Status codes provide error handling for the Integration Service
during a workflow.
 The stored procedure issues a status code that notifies whether
or not the stored procedure completed successfully.
 You cannot see this value.
 The Integration Service uses it to determine whether to continue
running the session or stop
Connected - The flow of data through a mapping in connected
mode also passes through the Stored Procedure transformation.
All data entering the transformation through the input ports
affects the stored procedure. You should use a connected Stored
Procedure transformation when you need data from an input
port sent as an input parameter to the stored procedure, or the
results of a stored procedure sent as an output parameter to
another transformation.
Unconnected - The unconnected Stored Procedure
transformation is not connected directly to the flow of the
mapping. It either runs before or after the session, or is called by
an expression in another transformation in the mapping.

Specifying when the Stored Procedure Runs
The following list describes the options for running a Stored
Procedure transformation:
Normal - The stored procedure runs where the transformation
exists in the mapping on a row-by-row basis. This is useful for
calling the stored procedure for each row of data that passes
through the mapping, such as running a calculation against an
input port. Connected stored procedures run only in normal
mode.
Pre-load of the Source - Before the session retrieves data from
the source, the stored procedure runs. This is useful for verifying
the existence of tables or performing joins of data in a temporary
table.
Post-load of the Source - After the session retrieves data from
the source, the stored procedure runs. This is useful for removing
temporary tables.
Pre-load of the Target - Before the session sends data to the
target, the stored procedure runs. This is useful for verifying
target tables or disk space on the target system.
Post-load of the Target - After the session sends data to the
target, the stored procedure runs. This is useful for re-creating
indexes on the database.
 You can run more than one Stored Procedure transformation in
different modes in the same mapping.
 For example, a pre-load source stored procedure can check table
integrity, a normal stored procedure can populate the table, and
a post-load stored procedure can rebuild indexes in the
database.
 However, you cannot run the same instance of a Stored
Procedure transformation in both connected and unconnected
mode in a mapping.
 You must create different instances of the transformation
 If the mapping calls more than one source or target pre- or post-
load stored procedure in a mapping, the Integration Service
95
executes the stored procedures in the execution order that you
specify in the mapping
 The Integration Service opens the database connection when it
encounters the first stored procedure.
 The database connection remains open until the Integration
Service finishes processing all stored procedures for that
connection.
 The Integration Service closes the database connections and
opens a new one when it encounters a stored procedure using
a different database connection.
 To run multiple stored procedures that use the same database
connection, set these stored procedures to run consecutively.
 If you do not set them to run consecutively, you might have
unexpected results in the target.
 For example, you have two stored procedures: Stored Procedure
A and Stored Procedure B. Stored Procedure A begins a
transaction, and Stored Procedure B commits the transaction.
 If you run Stored Procedure C before Stored Procedure B, using
another database connection, Stored Procedure B cannot
commit the transaction because the Integration Service closes
the database connection when it runs Stored Procedure C.
 Use the following guidelines to run multiple stored procedures
within a database connection:
- The stored procedures use the same database connect string
defined in the stored procedure properties.
- You set the stored procedures to run in consecutive order.
- The stored procedures have the same stored procedure type:
- Source pre-load
- Source post-load
- Target pre-load
- Target post-load
Creating a Stored Procedure Transformation
 After you configure and test a stored procedure in the database,
you must create the Stored Procedure transformation in the
Mapping Designer. There are two ways to configure the Stored
Procedure transformation:
- Use the Import Stored Procedure dialog box to configure the
ports used by the stored procedure.
- Configure the transformation manually, creating the
appropriate ports for any input or output parameters.
 Stored Procedure transformations are created as Normal type by
default, which means that they run during the mapping, not
before or after the session.
 New Stored Procedure transformations are not created as
reusable transformations.
 To create a reusable transformation, click Make Reusable in the
Transformation properties after creating the transformation.
 Note: Configure the properties of reusable transformations in the
Transformation Developer, not the Mapping Designer, to make
changes globally for the transformation.
Importing Stored Procedures
 When you import a stored procedure, the Designer creates ports
based on the stored procedure input and output parameters.
 You should import the stored procedure whenever possible.
 There are three ways to import a stored procedure in the
Mapping Designer:
- Select the stored procedure icon and add a Stored Procedure
transformation.
- Click Transformation > Import Stored Procedure.
- Click Transformation > Create, and then select Stored
Procedure.
 When you import a stored procedure containing a period (.) in
the stored procedure name, the Designer substitutes an
underscore (_) for the period in the Stored Procedure
transformation name.
Manually Creating Stored Procedure Transformations
 To create a Stored Procedure transformation manually, you need
to know the input parameters, output parameters, and return
values of the stored procedure, if there are any.
 You must also know the datatypes of those parameters, and the
name of the stored procedure.
 All these are configured through Import Stored Procedure.
 To create a Stored Procedure transformation, In the Mapping
Designer, click Transformation > Create, and then select Stored
Procedure
<Print – 384-385>
Changing the Stored Procedure
 If the number of parameters or the return value in a stored
procedure changes, you can either re-import it or edit the
Stored Procedure transformation manually.
 The Designer does not verify the Stored Procedure
transformation each time you open the mapping.
 After you import or create the transformation, the Designer does
not validate the stored procedure.
 The session fails if the stored procedure does not match the
transformation.
Configuring an Unconnected Transformation
 An unconnected Stored Procedure transformation is not directly
connected to the flow of data through the mapping.
Instead, the stored procedure runs either:
From an expression - Called from an expression written in the
Expression Editor within another transformation in the mapping.
Pre- or post-session - Runs before or after a session
96
 When using an unconnected Stored Procedure transformation
in an expression, you need a method of returning the value of
output parameters to a port. Use one of the following methods
to capture the output values:
- Assign the output value to a local variable.
- Assign the output value to the system variable PROC_RESULT.
 By using PROC_RESULT, you assign the value of the return
parameter directly to an output port, which can apply directly
to a target.
 You can also combine the two options by assigning one output
parameter as PROC_RESULT, and the other parameter as a
variable.
 Use PROC_RESULT only within an expression.
 If you do not use PROC_RESULT or a variable, the port containing
the expression captures a NULL.
 You cannot use PROC_RESULT in a connected Lookup
transformation or within the Call Text for a Stored Procedure
transformation
Expression Rules
- A single output parameter is returned using the variable
PROC_RESULT.
- When you use a stored procedure in an expression, use the :SP
reference qualifier. To avoid typing errors, select the Stored
Procedure node in the Expression Editor, and double-click the
name of the stored procedure.
- However, the same instance of a Stored Procedure
transformation cannot run in both connected and unconnected
mode in a mapping. You must create different instances of the
transformation.
- The input/output parameters in the expression must match the
input/output ports in the Stored Procedure transformation. If the
stored procedure has an input parameter, there must also be an
input port in the Stored Procedure transformation.
- When you write an expression that includes a stored procedure,
list the parameters in the same order that they appear in the
stored procedure and the Stored Procedure transformation.
- The parameters in the expression must include all of the
parameters in the Stored Procedure transformation. You cannot
leave out an input parameter. If necessary, pass a dummy
variable to the stored procedure.
- The arguments in the expression must be the same datatype
and precision as those in the Stored Procedure transformation.
- Use PROC_RESULT to apply the output parameter of a stored
procedure expression directly to a target. You cannot use a
variable for the output parameter to pass the results directly to a
target. Use a local variable to pass the results to an output port
within the same transformation.
 - Nested stored procedures allow passing the return value of one S. TRANSACTION CONTROL
stored procedure as the input parameter of another stored
 A transaction is the set of rows bound by commit or roll back
procedure. For example, if you have the following two stored
rows. You can define a transaction based on a varying number of
procedures:
input rows. You might want to define transactions based on a
- get_employee_id (employee_name)
group of rows ordered on a common key, such as employee ID
- get_employee_salary (employee_id)
or order entry date.
And the return value for get_employee_id is an employee ID
number, the syntax for a nested stored procedure is:
 In PowerCenter, you define transaction control at the following
:sp.get_employee_salary (:sp.get_employee_id
levels:
(employee_name))
Within a mapping - Within a mapping, you use the Transaction
You can have multiple levels of nested stored procedures.
Control transformation to define a transaction. You define
- Do not use single quotes around string parameters. If the input
transactions using an expression in a Transaction Control
parameter does not contain spaces, do not use any quotes. If the
transformation. Based on the return value of the expression, you
input parameter contains spaces, use double quotes.
can choose to commit, roll back, or continue without any
transaction changes.
Tips for Stored Procedure Transformations
Within a session - When you configure a session, you configure it
 Do not run unnecessary instances of stored procedures.
for user-defined commit. You can choose to commit or roll back a
 Each time a stored procedure runs during a mapping, the session
transaction if the Integration Service fails to transform or write
must wait for the stored procedure to complete in the database.
any row to the target.
You have two possible options to avoid this:
Reduce the row count - Use an active transformation prior to the
 If the mapping has a flat file target you can generate an output
Stored Procedure transformation to reduce the number of rows
file each time the Integration Service starts a new transaction.
that must be passed the stored procedure. Or, create an
You can dynamically name each target flat file.
expression that tests the values before passing them to the
stored procedure to make sure that the value does not really
Transaction Control Transformation Properties
need to be passed.
 Use the Transaction Control transformation to define conditions
Create an expression - Most of the logic used in stored
to commit and roll back transactions from transactional targets.
procedures can be easily replicated using expressions in the
 Transactional targets include relational, XML, and dynamic
Designer.
MQSeries targets

 The transaction control expression uses the IIF function to test

each row against the condition.
Use the following syntax for the expression:
Troubleshooting Stored Procedures
IIF (condition, value1, value2)
The session did not have errors before, but now it fails on the
stored procedure.
 The Integration Service evaluates the condition on a row-by-row
 The most common reason for problems with a Stored Procedure
basis.
transformation results from changes made to the stored
 The return value determines whether the Integration Service
procedure in the database. If the input/output parameters or
commits, rolls back, or makes no transaction changes to the
return value changes in a stored procedure, the Stored
row.
Procedure transformation becomes invalid. You must either
 When the Integration Service issues a commit or roll back based
import the stored procedure again, or manually configure the
on the return value of the expression, it begins a new
stored procedure to add, remove, or modify the appropriate
transaction.
ports.
 Use the following built-in variables in the Expression Editor when
you create a transaction control expression:
The session has been invalidated since I last edited the
TC_CONTINUE_TRANSACTION - The Integration Service does not
mapping. Why?
perform any transaction change for this row. This is the default
 Any changes you make to the Stored Procedure transformation
value of the expression.
may invalidate the session. The most common reason is that you
TC_COMMIT_BEFORE - The Integration Service commits the
have changed the type of stored procedure, such as from a
transaction, begins a new transaction, and writes the current row
Normal to a Post-load Source type.
to the target. The current row is in the new transaction.

97
 TC_COMMIT_AFTER - The Integration Service writes the current  Downstream transformations with the Transaction level
row to the target, commits the transaction, and begins a new transformation scope can use the transaction boundaries
transaction. The current row is in the committed transaction. defined by an upstream Transaction Control transformation.
TC_ROLLBACK_BEFORE - The Integration Service rolls back the  The following figure shows a valid mapping with a Transaction
current transaction, begin a new transaction, and write the Control transformation that is effective for a Sorter
current row to the target. The current row is in the new transformation, but ineffective for the target:
transaction.
TC_ROLLBACK_AFTER - The Integration Service writes the current
row to the target, rolls back the transaction, and begins a new
transaction. The current row is in the rolled back transaction.
 In this example, TCT1 transformation is ineffective for the target,
but effective for the Sorter transformation.
 If the transaction control expression evaluates to a value other
 The Sorter transformation’s Transformation Scope property is
than commit, roll back, or continue, the Integration Service
Transaction. It uses the transaction boundaries defined by TCT1.
fails the session.
 The Aggregator Transformation Scope property is All Input.
 It drops transaction boundaries defined by TCT1.
Using Transaction Control Transformations in Mappings
 The TCT2 transformation is an effective Transaction Control
 Transaction Control transformations are transaction generators.
transformation for the target.
 They define and redefine transaction boundaries in a mapping.
 They drop any incoming transaction boundary from an upstream
 Mapping Guidelines and Validation
active source or transaction generator, and they generate new
- If the mapping includes an XML target, and you choose to
transaction boundaries downstream.
append or create a new document on commit, the input groups
 You can also use Custom transformations configured to generate
must receive data from the same transaction control point.
transactions to define transaction boundaries.
- Transaction Control transformations connected to any target
other than relational, XML, or dynamic MQSeries targets are
 Transaction Control transformations can be effective or
ineffective for those targets.
ineffective for the downstream transformations and targets in
- You must connect each target instance to a Transaction Control
the mapping.
transformation you can connect multiple targets to a single
 The Transaction Control transformation becomes ineffective for
Transaction Control transformation.
downstream transformations or targets if you put a
- You can connect only one effective Transaction Control
transformation that drops incoming transaction boundaries after
transformation to a target.
it.
- You cannot place a Transaction Control transformation in a
 This includes any of the following active sources or
pipeline branch that starts with a Sequence Generator
transformations:
transformation.
- Aggregator with the All Input level transformation scope
- If you use a dynamic Lookup transformation and a Transaction
- Joiner with the All Input level transformation scope
Control transformation in the same mapping, a rolled-back
- Rank with the All Input level transformation scope
transaction might result in unsynchronized target data.
- Sorter with the All Input level transformation scope
- A Transaction Control transformation may be effective for one
- Custom with the All Input level transformation scope
target and ineffective for another target. If each target is
- Custom transformation configured to generate transactions
connected to an effective Transaction Control transformation, the
- Transaction Control transformation
mapping is valid.
- A multiple input group transformation, such as a Custom
- Either all targets or none of the targets in the mapping should
transformation, connected to multiple upstream transaction
be connected to an effective Transaction Control transformation.
control points
T. UNION
 Mappings with Transaction Control transformations that are
 The Integration Service processes all input groups in parallel.
ineffective for targets may be valid or invalid.
 It concurrently reads sources connected to the Union
 When you save or validate the mapping, the Designer displays a
transformation and pushes blocks of data into the input groups
message indicating which Transaction Control transformations
of the transformation.
are ineffective for targets.
 You can connect heterogeneous sources to a Union
 Although a Transaction Control transformation may be ineffective
transformation.
for a target, it can be effective for downstream transformations.

98
 The transformation merges sources with matching ports and
outputs the data from one output group with the same ports as
the input groups.
 The Union transformation is developed using the Custom
transformation
 Similar to the UNION ALL statement, the Union transformation
does not remove duplicate rows
Rules and Guidelines for Union -
- You can create multiple input groups, but only one output
group.
- All input groups and the output group must have matching
ports. The precision, datatype, and scale must be identical across
all groups.
- The Union transformation does not remove duplicate rows. To
remove duplicate rows, you must add another transformation
such as a Router or Filter transformation.
- You cannot use a Sequence Generator or Update Strategy
transformation upstream from a Union transformation.
- The Union transformation does not generate transactions
 When a Union transformation in a mapping receives data from
a single transaction generator, the Integration Service
propagates transaction boundaries.
 When the transformation receives data from multiple
transaction generators, the Integration Service drops all
incoming transaction boundaries and outputs rows in an open
transaction.
U. UPDATE STRATEGY
 In PowerCenter, you set the update strategy at two different
levels:
Within a session - When you configure a session, you can instruct
the Integration Service to either treat all rows in the same way
(for example, treat all rows as inserts), or use instructions coded
into the session mapping to flag rows for different database
operations.
Within a mapping - Within a mapping, you use the Update
Strategy transformation to flag rows for insert, delete, update, or
reject.
Note: You can also use the Custom transformation to flag rows
for insert, delete, update, or reject
Flagging Rows Within a Mapping
 For the greatest degree of control over the update strategy, you
add Update Strategy transformations to a mapping.
 The most important feature of this transformation is its update
strategy expression, used to flag individual rows for insert,
delete, update, or reject.
 The following table lists the constants for each database
operation and their numeric equivalent:
99
Forwarding Rejected Rows
 You can configure the Update Strategy transformation to either
pass rejected rows to the next transformation or drop them.
 By default, the Integration Service forwards rejected rows to the
next transformation.
 The Integration Service flags the rows for reject and writes them
to the session reject file.
 If you do not select Forward Rejected Rows, the Integration
Service drops rejected rows and writes them to the session log
file.
 If you enable row error handling, the Integration Service writes
the rejected rows and the dropped rows to the row error logs.
 It does not generate a reject file. If you want to write the dropped
rows to the session log in addition to the row error logs, you can
enable verbose data tracing.
 Update strategy expression uses the IIF or DECODE function from
the transformation language to test each row to see if it meets a
particular condition
Aggregator and Update Strategy Transformations
 When you connect Aggregator and Update Strategy
transformations as part of the same pipeline, you have the
following options:
Position the Aggregator before the Update Strategy
transformation -
 In this case, you perform the aggregate calculation, and then use
the Update Strategy transformation to flag rows that contain the
results of this calculation for insert, delete, or update.
Position the Aggregator after the Update Strategy
transformation -
 Here, you flag rows for insert, delete, update, or reject before you
perform the aggregate calculation.
 How you flag a particular row determines how the Aggregator
transformation treats any values in that row used in the
calculation.
 For example, if you flag a row for delete and then later use the
row to calculate the sum, the Integration Service subtracts the
value appearing in this row.
 If the row had been flagged for insert, the Integration Service
would add its value to the sum.
Lookup and Update Strategy Transformations
 When you create a mapping with a Lookup transformation that
uses a dynamic lookup cache, you must use Update Strategy
transformations to flag the rows for the target tables.
 When you configure a session using Update Strategy
transformations and a dynamic lookup cache, you must define
certain session properties.
 You must define the Treat Source Rows As option as Data Driven.
 Specify this option on the Properties tab in the session
properties.
 You must also define the following update strategy target table
options:
- Select Insert
- Select Update as Update
- Do not select Delete
 These update strategy target table options ensure that the
Integration Service updates rows marked for update and inserts
rows marked for insert.
 If you do not choose Data Driven, the Integration Service flags all
rows for the database operation you specify in the Treat Source
Rows As option and does not use the Update Strategy
transformations in the mapping to flag the rows.
 The Integration Service does not insert and update the correct
rows.
 If you do not choose Update as Update, the Integration Service
does not correctly update the rows flagged for update in the
target table.
As a result, the lookup cache and target table might become
unsynchronized.
Only perform inserts into a target table.
 When you configure the session, select Insert for the Treat Source
Rows As session property. Also, make sure that you select the
Insert option for all target instances in the session.
Delete all rows in a target table.
 When you configure the session, select Delete for the Treat
Source Rows As session property. Also, make sure that you
select the Delete option for all target instances in the session.
Only perform updates on the contents of a target table.
 When you configure the session, select Update for the Treat
Source Rows As session property. When you configure the
update options for each target table instance, make sure you
select the Update option for each target instance.
Perform different database operations with different rows
destined for the same target table.
 Add an Update Strategy transformation to the mapping. When
you write the transformation update strategy expression, use
either the DECODE or IIF function to flag rows for different
operations (insert, delete, update, or reject). When you
configure a session that uses this mapping, select Data Driven
for the Treat Source Rows As session property. Make sure that
you select the Insert, Delete, or one of the Update options for
each target table instance.
Reject data.
100
 Add an Update Strategy transformation to the mapping. When
you write the transformation update strategy expression, use
DECODE or IIF to specify the criteria for rejecting the row. When
you configure a session that uses this mapping, select Data
Driven for the Treat Source Rows As session property.
V. XML SOURCE QUALIFIER
 An XML Source Qualifier transformation always has one input or
output port for every column in the XML source.
 When you create an XML Source Qualifier transformation for a
source definition, the Designer links each port in the XML source
definition to a port in the XML Source Qualifier transformation.
 You cannot remove or edit any of the links.
 If you remove an XML source definition from a mapping, the
Designer also removes the corresponding XML Source Qualifier
transformation.
 You can link one XML source definition to one XML Source
Qualifier transformation
 You can link ports of one XML Source Qualifier group to ports of
different transformations to form separate data flows.
 However, you cannot link ports from more than one group in an
XML Source Qualifier transformation to ports in the same target
transformation
W. XML PARSER
 Use an XML Parser transformation to extract XML inside a
pipeline.
 The XML Parser transformation lets you extract XML data from
messaging systems, such as TIBCO or MQ Series, and from other
sources, such as files or databases.
 The XML Parser transformation functionality is similar to the XML
source functionality, except it parses the XML in the pipeline.
 For example, you might want to extract XML data from a TIBCO
source and pass the data to relational targets.
 The XML Parser transformation reads XML data from a single
input port and writes data to one or more output ports.
X. XML GENERATOR
 Use an XML Generator transformation to create XML inside a
pipeline.
 The XML Generator transformation lets you read data from
messaging systems, such as TIBCO and MQ Series, or from other
sources, such as files or databases.
 The XML Generator transformation functionality is similar to the
XML target functionality, except it generates the XML in the
pipeline.
 For example, you might want to extract data from relational
sources and pass XML data to targets.
 The XML Generator transformation accepts data from multiple
ports and writes XML through a single output port.
7. TRANSFORMATION LANGUAGE REFERENCE
Rules and Guidelines for Expression Syntax
Use the following rules and guidelines when you write
expressions:
- You cannot include both single-level and nested aggregate
functions in an Aggregator transformation.
101
- If you need to create both single-level and nested functions,
create separate Aggregator transformations.
- You cannot use strings in numeric expressions. For example, the
expression 1 + '1' is not valid because you can only perform
addition on numeric datatypes. You cannot add an integer and a
string.
- You cannot use strings as numeric parameters. For example, the
expression SUBSTR (TEXT_VAL, '1', 10) is not valid because the
SUBSTR function requires an integer value, not a string, as the
start position.
- You cannot mix datatypes when using comparison operators.
For example, the expression 123.4 = '123.4' is not valid because it
compares a decimal value with a string.
- You can pass a value from a port, literal string or number,
variable, Lookup transformation, Stored Procedure
transformation, External Procedure transformation, or the results
of another expression.
- Use the ports tab in the Expression Editor to enter a port name
into an expression. If you rename a port in a connected
transformation, the Designer propagates the name change to
expressions in the transformation.
- Separate each argument in a function with a comma.
- Except for literals, the transformation language is not case
sensitive.
- Except for literals, the Designer and PowerCenter Integration
Service ignore spaces.
- The colon (:), comma (,), and period (.) have special meaning
and should be used only to specify syntax.
- The PowerCenter Integration Service treats a dash (-) as a minus
operator.
- If you pass a literal value to a function, enclose literal strings
within single quotation marks. Do not use quotation marks for
literal numbers. The PowerCenter Integration Service treats any
string value enclosed in single quotation marks as a character
string.
- When you pass a mapping parameter or variable or a workflow
variable to a function within an expression, do not use quotation
marks to designate mapping parameters or variables or workflow
variables.
- Do not use quotation marks to designate ports.
- You can nest multiple functions within an expression except
aggregate functions, which allow only one nested aggregate
function. The PowerCenter Integration Service evaluates the
expression starting with the innermost function.
Reserved Words
Some keywords in the transformation language, such as
constants, operators, and built-in variables, are reserved for
specific functions. These include:
- :EXT
- :INFA
- :LKP
 - :MCR  Use the Treat Null In Comparison Operators As property to
- :SD configure how the PowerCenter Integration Service handles null
- :SEQ values in comparison expressions.
- :SP
- :TD This PowerCenter Integration Service configuration property
- AND affects the behavior of the following comparison operators in
- DD_DELETE expressions:
- DD_INSERT =, !=, ^=, <>, >, >=, <, <=
- DD_REJECT
- DD_UPDATE For example, consider the following expressions:
- FALSE NULL > 1
- NOT NULL = NULL
- NULL
- OR
- PROC_RESULT
- SESSSTARTTIME
- SPOUTPUT
- SYSDATE
- TRUE
- WORKFLOWSTARTTIME

The following words are reserved for workflow expressions:

- ABORTED
- DISABLED
- FAILED
Transaction Control Variables
- NOTSTARTED
 The following example uses transaction control variables to
- STARTED
determine where to process a row:
- STOPPED IIF (NEWTRAN=1, TC_COMMIT_BEFORE, TC_CONTINUE_TRANSACTION)
- SUCCEEDED  If NEWTRAN=1, the TC_COMMIT_BEFORE variable causes a
commit to occur before the current row processes.
Note: You cannot use a reserved word to name a port or local  Otherwise, the TC_CONTINUE_TRANSACTION variable forces the
variable. You can only use reserved words within transformation row to process in the current transaction.
and workflow expressions. Reserved words have predefined
meanings in expressions  Use the following variables in the Expression Editor when you
create a transaction control expression:
Working with Null Values in Boolean Expressions TC_CONTINUE_TRANSACTION - The PowerCenter Integration
 Expressions that combine a null value with a Boolean expression Service does not perform any transaction change for the current
produce results that are ANSI compliant. row. This is the default transaction control variable value.
 For example, the PowerCenter Integration Service produces the TC_COMMIT_BEFORE - The PowerCenter Integration Service
following results: commits the transaction, begins a new transaction, and writes
- NULL AND TRUE = NULL the current row to the target. The current row is in the new
- NULL AND FALSE = FALSE transaction.
TC_COMMIT_AFTER - The PowerCenter Integration Service
Working with Null Values in Comparison Expressions writes the current row to the target, commits the transaction,
 When you use a null value in an expression containing a and begins a new transaction. The current row is in the
comparison operator, the PowerCenter Integration Service committed transaction.
produces a null value. TC_ROLLBACK_BEFORE - The PowerCenter Integration Service
 However, you can also configure the PowerCenter Integration rolls back the current transaction, begin a new transaction, and
Service to treat null values as high or low in comparison write the current row to the target. The current row is in the new
operations. transaction.

DATES
102
 Date functions accept datetime values only. To pass a string to a
date function, first use TO_DATE to convert it to a datetime value.
 For example, the following expression converts a string port to
datetime values and then adds one month to each date:
ADD_TO_DATE( TO_DATE( STRING_PORT, 'MM/DD/RR'), 'MM', 1 )
 You can use dates between 1 A.D. and 9999 A.D in the Gregorian
calendar system.
Julian Day, Modified Julian Day, and the Gregorian calendar
 You can use dates in the Gregorian calendar system only.
 Dates in the Julian calendar are called Julian dates and are not
supported in Informatica.
 This term should not be confused with Julian Day or with
Modified Julian Day.
 You can manipulate Modified Julian Day (MJD) formats using the J
format string.
 The MJD for a given date is the number of days to that date since
Jan 1 4713 B.C. 00:00:00 (midnight).
 By definition, MJD includes a time component expressed as a
decimal, which represents some fraction of 24 hours.
 The J format string does not convert this time component.
 For example, the following TO_DATE expression converts strings
in the SHIP_DATE_MJD_STRING port to date values in the
default date format:
TO_DATE (SHIP_DATE_MJD_STR, 'J')
SHIP_DATE_MJD_STR RETURN_VALUE
2451544 Dec 31 1999 00:00:00.000000000
2415021 Jan 1 1900 00:00:00.000000000
 Because the J format string does not include the time portion of a
date, the return values have the time set to
00:00:00.000000000.
 You can also use the J format string in TO_CHAR expressions. For
example, use the J format string in a TO_CHAR expression to
convert date values to MJD values expressed as strings. For
example: TO_CHAR(SHIP_DATE, 'J')
and 49, the PowerCenter Integration Service returns the current
century plus the two-digit year from the source string. If the
source string year is between 50 and 99, the Integration Service
returns the previous century plus the two-digit year from the
source string.
 Current Year Between 50 and 99 - If the current year is between
50 and 99 (such as 1998) and the source string year is between
0 and 49, the PowerCenter Integration Service returns the next
century plus the two-digit year from the source string. If the
source string year is between 50 and 99, the PowerCenter
Integration Service returns the current century plus the
specified two-digit year
 The following table summarizes how the RR format string
converts to dates:
Example
The following expression produces the same return values for any
current year between 1950 and 2049:
TO_DATE( ORDER_DATE, 'MM/DD/RR' )
ORDER_DATE RETURN_VALUE
'04/12/98' 04/12/1998 00:00:00.000000000
'11/09/01' 11/09/2001 00:00:00.000000000
DIFFERENCE BETWEEN THE YY AND RR FORMAT STRINGS
 PowerCenter also provides a YY format string.
 Both the RR and YY format strings specify two-digit years.
 The YY and RR format strings produce identical results when
used with all date functions except TO_DATE.
 In TO_DATE expressions, RR and YY produce different results.

SHIP_DATE RETURN_VALUE The following table shows the different results each format string
Dec 31 1999 23:59:59 2451544 returns:
Jan 1 1900 01:02:03 2415021

RR FORMAT STRING
 The transformation language provides the RR format string to
convert strings with two-digit years to dates.
 Using TO_DATE and the RR format string, you can convert a string
in the format MM/DD/RR to a date.
 The RR format string converts data differently depending on the
 For dates in the year 2000 and beyond, the YY format string
current year.
produces less meaningful results than the RR format string. Use
 Current Year Between 0 and 49 - If the current year is between 0
the RR format string for dates in the twenty-first century
and 49 (such as 2003) and the source string year is between 0
103
 Default Date Format
 By default, the date format is MM/DD/YYYY HH24:MI:SS.US.
Note: The format string is not case sensitive. It must always be
enclosed within single quotation marks.

 The following table describes date functions that use date format
strings to evaluate input dates:

TO_CHAR Format Strings

 TO_CHAR is generally used when the target is a flat file or a
database that does not support a Date/Time datatype.
 You can convert the entire date or a part of the date to a string.

 The following table summarizes the format strings for dates in

the function TO_CHAR:

TO_DATE and IS_DATE Format Strings

 The TO_DATE function converts a string with the format you
specify to a datetime value.
 TO_DATE is generally used to convert strings from flat files to
datetime values.
Note: TO_DATE and IS_DATE use the same set of format strings.
104
 The source string format and the format string must match,
including any date separator.
 If any part does not match, the PowerCenter Integration Service
does not convert the string, and it skips the row.
 If you omit the format string, the source string must be in the
date format specified in the session
 The following table summarizes the format strings for the
functions TO_DATE and IS_DATE:
<Same as the TO_CHAR formats>
RULES AND GUIDELINES FOR DATE FORMAT STRINGS
 The format of the TO_DATE string must match the format string
including any date separators. If it does not, the PowerCenter
Integration Service might return inaccurate values or skip the
row.
 The format string must be enclosed within single quotation marks
ABORT
 When the PowerCenter Integration Service encounters an ABORT
function, it stops transforming data at that row.
 It processes any rows read before the session aborts and loads
them based on the source- or target-based commit interval and
the buffer block size defined for the session.
 The PowerCenter Integration Service writes to the target up to
the aborted row and then rolls back all uncommitted data to the
last commit point.
 You can perform recovery on the session after rollback
 If you use ABORT in an expression for an unconnected port, the
PowerCenter Integration Service does not run the ABORT
function.
Syntax
ABORT( string )
Return Value
NULL.
ABS
Returns the absolute value of a numeric value.
Syntax
ABS( numeric_value )
Return Value
Positive numeric value
ADD_TO_DATE
 Adds a specified amount to one part of a datetime value, and
returns a date in the same format as the date you pass to the
function.
 ADD_TO_DATE accepts positive and negative integer values. Use
ADD_TO_DATE to change the following parts of a date:
105
Year - Enter a positive or negative integer in the amount
argument. Use any of the year format strings: Y, YY, YYY, or YYYY.
The following expression adds 10 years to all dates in the
SHIP_DATE port: ADD_TO_DATE ( SHIP_DATE, 'YY', 10 )
Month - Enter a positive or negative integer in the amount
argument. Use any of the month format strings: MM, MON,
MONTH. The following expression subtracts 10 months from each
date in the SHIP_DATE port: ADD_TO_DATE( SHIP_DATE,
'MONTH', -10 )
Day - Enter a positive or negative integer in the amount
argument. Use any of the day format strings: D, DD, DDD, DY, and
DAY. The following expression adds 10 days to each date in the
SHIP_DATE port: ADD_TO_DATE( SHIP_DATE, 'DD', 10 )
Hour - Enter a positive or negative integer in the amount
argument. Use any of the hour format strings: HH, HH12, HH24.
The following expression adds 14 hours to each date in the
SHIP_DATE port: ADD_TO_DATE( SHIP_DATE, 'HH', 14 )
Minute - Enter a positive or negative integer in the amount
argument. Use the MI format string to set the minute. The
following expression adds 25 minutes to each date in the
SHIP_DATE port: ADD_TO_DATE( SHIP_DATE, 'MI', 25 )
Seconds - Enter a positive or negative integer in the amount
argument. Use the SS format string to set the second. The
following expression adds 59 seconds to each date in the
SHIP_DATE port: ADD_TO_DATE( SHIP_DATE, 'SS', 59 )
Milliseconds - Enter a positive or negative integer in the amount
argument. Use the MS format string to set the milliseconds. The
following expression adds 125 milliseconds to each date in the
SHIP_DATE port: ADD_TO_DATE( SHIP_DATE, 'MS', 125 )
Microseconds - Enter a positive or negative integer in the amount
argument. Use the US format string to set the microseconds. The
following expression adds 2,000 microseconds to each date in the
SHIP_DATE port: ADD_TO_DATE( SHIP_DATE, 'US', 2000 )
Nanoseconds - Enter a positive or negative integer in the amount
argument. Use the NS format string to set the nanoseconds. The
following expression adds 3,000,000 nanoseconds to each date in
the SHIP_DATE port: ADD_TO_DATE( SHIP_DATE, 'NS', 3000000 )
 If you pass a value that creates a day that does not exist in a
particular month, the PowerCenter Integration Service returns
the last day of the month.
 For example, if you add one month to Jan 31 1998, the
PowerCenter Integration Service returns Feb 28 1998.
ASCII
 The ASCII function returns the numeric ASCII value of the first
character of the string passed to the function
 You can pass a string of any size to ASCII, but it evaluates only
the first character in the string
 This function is identical in behavior to the CHRCODE function.
 If you use ASCII in existing expressions, they will still work
correctly. However, when you create new expressions, use the
CHRCODE function instead of the ASCII function.
AVG
 Returns the average of all values in a group of rows.
 Optionally, you can apply a filter to limit the rows you read to
calculate the average.
 You can nest only one other aggregate function within AVG, and
the nested function must return a Numeric datatype.
Syntax
AVG( numeric_value [, filter_condition ] )
 If a value is NULL, AVG ignores the row. However, if all values
passed from the port are NULL, AVG returns NULL
 AVG groups values based on group by ports you define in the
transformation, returning one result for each group.
 If there is not a group by port, AVG treats all rows as one group,
returning one value.
Example:
 The following expression returns the average wholesale cost of
flashlights: AVG( WHOLESALE_COST, ITEM_NAME='Flashlight' )
 You can perform arithmetic on the values passed to AVG before
the function calculates the average. For example: AVG( QTY *
PRICE - DISCOUNT )
CEIL
 Returns the smallest integer greater than or equal to the numeric
value passed to this function.
 For example, if you pass 3.14 to CEIL, the function returns 4. If
you pass 3.98 to CEIL, the function returns 4.
Likewise, if you pass -3.17 to CEIL, the function returns -3.
Syntax
CEIL( numeric_value )
CHOOSE
 Chooses a string from a list of strings based on a given position.
 You specify the position and the value.
 If the value matches the position, the PowerCenter Integration
Service returns the value.
Syntax
CHOOSE( index, string1 [, string2, ..., stringN] )
 The following expression returns the string ‘flashlight’ based on
an index value of 2: CHOOSE( 2, 'knife', 'flashlight', 'diving hood' )
106
 The following expression returns NULL based on an index value of
4: CHOOSE( 4, 'knife', 'flashlight', 'diving hood' )
CHR
 ASCII Mode - CHR returns the ASCII character corresponding to
the numeric value you pass to this function.
 Unicode Mode - returns the Unicode character corresponding to
the numeric value you pass to this function
 ASCII values fall in the range 0 to 255.
 You can pass any integer to CHR, but only ASCII codes 32 to 126
are printable characters.
Syntax
CHR( numeric_value )
 Use the CHR function to concatenate a single quote onto a string.
 The single quote is the only character that you cannot use inside
a string literal.
 Consider the following example: 'Joan' || CHR(39) || 's car'
The return value is: Joan's car
CHRCODE
 ASCII Mode - CHRCODE returns the numeric ASCII value of the
first character of the string passed to the function.
 UNICODE Mode - returns the numeric Unicode value of the first
character of the string passed to the function
COMPRESS
 Compresses data using the zlib 1.2.1 compression algorithm.
 Use the COMPRESS function before you send large amounts of
data over a wide area network.
Syntax
COMPRESS( value )
Return Value
 Compressed binary value of the input value.
 NULL if the input is a null value.
CONCAT
Syntax
CONCAT( first_string, second_string )
Return Value
String.
 If one of the strings is NULL, CONCAT ignores it and returns the
other string.
 If both strings are NULL, CONCAT returns NULL
 CONCAT does not add spaces to separate strings.
 If you want to add a space between two strings, you can write an
expression with two nested CONCAT functions.
CONVERT_BASE
 Converts a number from one base value to another base value.
Syntax
CONVERT_BASE( value, source_base, dest_base )
 The following example converts 2222 from the decimal base
value 10 to the binary base value 2: CONVERT_BASE( "2222", 10, 2 )
 The PowerCenter Integration Service returns 100010101110
 You can add a condition to filter rows out of the row set before
calculating the running total.
 Use CUME and similar functions (such as MOVINGAVG and
MOVINGSUM) to simplify reporting by calculating running
values.
Syntax
CUME( numeric_value [, filter_condition] )
DATE_COMPARE
 Returns an integer indicating which of two dates is earlier.
DATE_COMPARE returns an integer value rather than a date
value.

COUNT Return Value

 Returns the number of rows that have non-null values in a group.  -1 if the first date is earlier.
 Optionally, you can include the asterisk (*) argument to count all input  0 if the two dates are equal.
values in a transformation.  1 if the second date is earlier.
 You can nest only one other aggregate function within COUNT.  NULL if one of the date values is NULL
 You can apply a condition to filter rows before counting them.
DATE_DIFF
Syntax  Returns the length of time between two dates. You can request
COUNT( value [, filter_condition] ) the format to be years, months, days, hours, minutes, seconds,
or milliseconds, microseconds, or nanoseconds.
COUNT( * [, filter_condition] )  The PowerCenter Integration Service subtracts the second date
from the first date and returns the difference.
 COUNT groups values based on group by ports you define in the
transformation, returning one result for each group. Syntax
 If there is no group by port COUNT treats all rows as one group, DATE_DIFF( date1, date2, format )
returning one value
Return Value
CRC32  Double value. If date1 is later than date2, the return value is a
 Returns a 32-bit Cyclic Redundancy Check (CRC32) value. positive number. If date1 is earlier than date2, the return value
 Use CRC32 to find data transmission errors. is a negative number.
 You can also use CRC32 if you want to verify that data stored in a  0 if the dates are the same.
file has not been modified.  NULL if one (or both) of the date values is NULL.
 If you use CRC32 to perform a redundancy check on data in ASCII
mode and Unicode mode, the PowerCenter Integration Service
may generate different results on the same input value.
DECODE
Note: CRC32 can return the same output for different input
strings. If you want to generate keys in a mapping, use a Examples
Sequence Generator transformation. If you use CRC32 to  You might use DECODE in an expression that searches for a
generate keys in a mapping, you may receive unexpected results. particular ITEM_ID and returns the ITEM_NAME:
DECODE( ITEM_ID, 10, 'Flashlight',
Syntax 14, 'Regulator',
CRC32( value ) 20, 'Knife',
40, 'Tank',
CUME 'NONE' )
 Returns a running total. A running total means CUME returns a
total each time it adds a value. ITEM_ID RETURN VALUE
10 Flashlight
107
 14 Regulator EXP
17 NONE  Returns e raised to the specified power (exponent), where
20 Knife e=2.71828183.
25 NONE  For example, EXP(2) returns 7.38905609893065.
NULL NONE  You might use this function to analyze scientific and technical
40 Tank data rather than business data.
 EXP is the reciprocal of the LN function, which returns the natural
 DECODE returns the default value of NONE for items 17 and 25 logarithm of a numeric value.
because the search values did not match the ITEM_ID.
 Also, DECODE returns NONE for the NULL ITEM_ID. Syntax
EXP( exponent )
 The following expression tests multiple columns and conditions,
evaluated in a top to bottom order for TRUE or FALSE: Return Value
 Double value.
DECODE( TRUE,  NULL if a value passed as an argument to the function is NULL
Var1 = 22, 'Variable 1 was 22!',
Var2 = 49, 'Variable 2 was 49!', FIRST
Var1 < 23, 'Variable 1 was less than 23.',  Returns the first value found within a port or group.
Var2 > 30, 'Variable 2 was more than 30.',  Optionally, you can apply a filter to limit the rows the
'Variables were out of desired ranges.') PowerCenter Integration Service reads.
 You can nest only one other aggregate function within FIRST.
Var1 Var2 RETURN VALUE
21 47 Variable 1 was less than 23. Syntax
22 49 Variable 1 was 22! FIRST( value [, filter_condition ] )
23 49 Variable 2 was 49!
24 27 Variables were out of desired ranges. Return Value
25 50 Variable 2 was more than 30. First value in a group

ERROR
 Causes the PowerCenter Integration Service to skip a row and
issue an error message, which you define.
 The error message displays in the session log.
 The PowerCenter Integration Service does not write these
skipped rows to the session reject file.
 If a value is NULL, FIRST ignores the row. However, if all values
passed from the port are NULL, FIRST returns NULL
 FIRST groups values based on group by ports you define in the
transformation, returning one result for each group.
 If there is no group by port, FIRST treats all rows as one group,
returning one value.

 For example, you use the ERROR function in an expression, and

you assign the default value, ‘1234’, to the output port.
 Each time the PowerCenter Integration Service encounters the
ERROR function in the expression, it overrides the error with the
value ‘1234’ and passes ‘1234’ to the next transformation.
 It does not skip the row, and it does not log an error in the
session log
FLOOR
 Returns the largest integer less than or equal to the numeric
value you pass to this function.
 For example, if you pass 3.14 to FLOOR, the function returns 3.
 If you pass 3.98 to FLOOR, the function returns 3.
 Likewise, if you pass -3.17 to FLOOR, the function returns -4.

IIF( SALARY < 0, ERROR ('Error. Negative salary found. Row skipped.', Syntax
EMP_SALARY ) FLOOR( numeric_value )

SALARY RETURN VALUE Return Value

10000 10000  Integer if you pass a numeric value with declared precision
-15000 'Error. Negative salary found. Row skipped.' between 0 and 28.
NULL NULL  Double if you pass a numeric value with declared precision
150000 150000 greater than 28.
 NULL if a value passed to the function is NULL.
108
  By default, the match is case sensitive
FV
 Returns the future value of an investment, where you make Return Value
periodic, constant payments and the investment earns a  Value1 if it is the greatest of the input values, value2 if it is the
constant interest rate. greatest of the input values, and so on.
 NULL if any of the arguments is null
Syntax
FV( rate, terms, payment [, present value, type] ) IIF
 Returns one of two values you specify, based on the results of a
Example condition.
 You deposit $2,000 into an account that earns 9% annual interest
compounded monthly (monthly interest of 9%/ 12, or 0.75%). Syntax
 You plan to deposit $250 at the beginning of every month for the IIF( condition, value1 [,value2] )
next 12 months.
 The following expression returns $5,337.96 as the account  Unlike conditional functions in some systems, the FALSE (value2)
balance at the end of 12 months: condition in the IIF function is not required.
FV(0.0075, 12, -250, -2000, TRUE)  If you omit value2, the function returns the following when the
condition is FALSE:
GET_DATE_PART - 0 if value1 is a Numeric datatype.
 Returns the specified part of a date as an integer value. - Empty string if value1 is a String datatype.
 Therefore, if you create an expression that returns the month - NULL if value1 is a Date/Time datatype.
portion of the date, and pass a date such as Apr 1 1997
00:00:00, GET_DATE_PART returns 4.  For example, the following expression does not include a FALSE
condition and value1 is a string datatype so the PowerCenter
Syntax Integration Service returns an empty string for each row that
GET_DATE_PART( date, format ) evaluates to FALSE: IIF( SALES > 100, EMP_NAME )

Return Value SALES EMP_NAME RETURN VALUE

 Integer representing the specified part of the date. 150 John Smith John Smith
 NULL if a value passed to the function is NULL. 50 Pierre Bleu '' (empty string)
120 Sally Green Sally Green
NULL Greg Jones '' (empty string)

 When you use IIF, the datatype of the return value is the same
 The following expressions return the day for each date in the as the datatype of the result with the greatest precision.
DATE_SHIPPED port:
GET_DATE_PART( DATE_SHIPPED, 'D' )  For example, you have the following expression:
GET_DATE_PART( DATE_SHIPPED, 'DD' ) IIF( SALES < 100, 1, .3333 )
GET_DATE_PART( DATE_SHIPPED, 'DDD' )  The TRUE result (1) is an integer and the FALSE result (.3333) is a
GET_DATE_PART( DATE_SHIPPED, 'DY' ) decimal.
GET_DATE_PART( DATE_SHIPPED, 'DAY' )  The Decimal datatype has greater precision than Integer, so the
datatype of the return value is always a Decimal
DATE_SHIPPED RETURN VALUE
Mar 13 1997 12:00:00 AM 13  You can often use a Filter transformation instead of IIF to
June 3 1997 11:30:44PM 3 maximize session performance.
Aug 22 1997 12:00:00PM 22
NULL NULL IN
 Matches input data to a list of values. By default, the match is
case sensitive.
GREATEST
 Returns the greatest value from a list of input values. Syntax
 Use this function to return the greatest string, date, or number. IN( valueToSearch, value1, [value2, ..., valueN,] CaseFlag )
109
 FIRST_NAME RETURN VALUE
Example ramona Ramona
 The following expression determines if the input value is a safety 18-albert 18-Albert
knife, chisel point knife, or medium titanium knife. NULL NULL
 The input values do not have to match the case of the values in ?!SAM ?!Sam
the comma-separated list: THOMAS Thomas
IN( ITEM_NAME, ‘Chisel Point Knife’, ‘Medium Titanium Knife’, ‘Safety Knife’, 0 PierRe Pierre

ITEM_NAME RETURN VALUE INSTR

Stabilizing Vest 0 (FALSE)  Returns the position of a character set in a string, counting from
Safety knife 1 (TRUE) left to right.
Medium Titanium knife 1 (TRUE)
NULL Syntax
INSTR( string, search_value [,start [,occurrence [,comparison_type ]]] )
INDEXOF
 Finds the index of a value among a list of values. By default, the Return Value
match is case sensitive.  Integer if the search is successful. Integer represents the position
of the first character in the search_value, counting from left to
Syntax right.
INDEXOF( valueToSearch, string1, [string2, ..., stringN,] CaseFlag )  0 if the search is unsuccessful.
 NULL if a value passed to the function is NULL
Example
 The following expression determines if values from the  The following expression returns the position of the second
ITEM_NAME port match the first, second, or third string: occurrence of the letter ‘a’, starting at the beginning of each
INDEXOF( ITEM_NAME, ‘diving hood’, ‘flashlight’, ‘safety knife’) company name.
 Because the search_value argument is case sensitive, it skips the
ITEM_NAME RETURN VALUE ‘A’ in ‘Blue Fin Aqua Center’, and returns 0:
Safety Knife 0 INSTR( COMPANY, 'a', 1, 2 )
diving hood 1
Compass 0
safety knife 3
flashlight 2 COMPANY RETURN VALUE
Blue Fin Aqua Center 0
 Safety Knife returns a value of 0 because it does not match the Maco Shark Shop 8
case of the input value. Scuba Gear 9
Frank's Dive Shop 0
INITCAP VIP Diving Club 0
 Capitalizes the first letter in each word of a string and converts all
other letters to lowercase.  The following expression returns the position of the first
 Words are delimited by white space (a blank space, formfeed, character in the string ‘Blue Fin Aqua Center’ (starting from the
newline, carriage return, tab, or vertical tab) and characters that last character in the company name):
are not alphanumeric. INSTR( COMPANY, 'Blue Fin Aqua Center', -1, 1 )
 For example, if you pass the string ‘…THOMAS’, the function
returns Thomas. COMPANY RETURN VALUE
Blue Fin Aqua Center 1
Syntax Maco Shark Shop 0
INITCAP( string ) Scuba Gear 0
Frank's Dive Shop 0
Example VIP Diving Club 0
 The following expression capitalizes all names in the FIRST_NAME
port: INITCAP( FIRST_NAME )  You can nest the INSTR function within other functions to
accomplish more complex tasks.

110
 The following expression evaluates a string, starting from the end '+3.45E-32'
of the string. '+3.45d+32' (Windows only)
 The expression finds the last (rightmost) space in the string and '+3.45D-32' (Windows only)
then returns all characters to the left of it: '.6804'
SUBSTR( CUST_NAME,1,INSTR( CUST_NAME,' ' ,-1,1 ))  The output port for an IS_NUMBER expression must be a String
or Numeric datatype.
CUST_NAME RETURN VALUE
PATRICIA JONES PATRICIA ITEM_PRICE RETURN VALUE
MARY ELLEN SHAH MARY ELLEN '123.00' 1 (True)
'-3.45e+3' 1 (True)
 The following expression removes the character '#' from a string: '-3.45D-3' 1 (True - Windows only)
SUBSTR( CUST_ID, 1, INSTR(CUST_ID, '#')-1 ) || SUBSTR( CUST_ID, '-3.45d-3' 0 (False - UNIX only)
INSTR(CUST_ID, '#')+1 ) '3.45E-' 0 (False) Incomplete number
'' 0 (False) Consists entirely of blanks
CUST_ID RETURN VALUE '' 0 (False) Empty string
ID#33 ID33 '+123abc' 0 (False)
#A3577 A3577 ' 123' 1 (True) Leading white blanks
SS #712403399 SS 712403399 '123 ' 1 (True) Trailing white blanks
'ABC' 0 (False)
ISNULL '-ABC' 0 (False)
 Returns whether a value is NULL. ISNULL evaluates an empty NULL NULL
string as FALSE.
Note: To test for empty strings, use LENGTH. IS_SPACES
 Returns whether a string value consists entirely of spaces.
Syntax  A space is a blank space, a formfeed, a newline, a carriage return,
ISNULL( value ) a tab, or a vertical tab.
 IS_SPACES evaluates an empty string as FALSE because there
Example are no spaces. To test for an empty string, use LENGTH.
 The following example checks for null values in the items table:
ISNULL( ITEM_NAME )
Example
ITEM_NAME RETURN VALUE
 The following expression checks the ITEM_NAME port for rows
Flashlight 0 (FALSE)
that consist entirely of spaces:
NULL 1 (TRUE) IS_SPACES( ITEM_NAME )
Regulator system 0 (FALSE)
'' 0 (FALSE) Empty string is not NULL ITEM_NAME RETURN VALUE
Flashlight 0 (False)
IS_NUMBER 1 (True)
 Returns whether a string is a valid number. A valid number Regulator system 0 (False)
consists of the following parts: NULL NULL
- Optional space before the number '' 0 (FALSE) (Empty string does not
- Optional sign (+/-) contain spaces.)
- One or more digits with an optional decimal point
- Optional scientific notation, such as the letter ‘e’ or ‘E’ (and the LAST
letter ‘d’ or ‘D’ on Windows) followed by an optional sign (+/-),  Returns the last row in the selected port.
followed by one or more digits  Optionally, you can apply a filter to limit the rows the
- Optional white space following the number PowerCenter Integration Service reads.
 You can nest only one other aggregate function within LAST
 The following numbers are all valid:
' 100 ' Syntax
' +100' LAST( value [, filter_condition ] )
'-100'
'-3.45e+32' Return Value
111
 Last row in a port.
 NULL if all values passed to the function are NULL, or if no rows LENGTH
are selected (for example, the filter condition evaluates to FALSE  Returns the number of characters in a string, including trailing
or NULL for all rows). blanks.

LAST_DAY Return Value

 Returns the date of the last day of the month for each date in a  Integer representing the length of the string.
port.  NULL if a value passed to the function is NULL

Syntax LN
LAST_DAY( date )  Returns the natural logarithm of a numeric value. For example,
LN(3) returns 1.098612.
Return Value  You usually use this function to analyze scientific data rather than
 Date. The last day of the month for that date value you pass to business data.
this function  This function is the reciprocal of the function EXP.

 If a value is NULL, LAST_DAY ignores the row. However, if all Syntax

values passed from the port are NULL, LAST_DAY returns NULL LN( numeric_value )
 LAST_DAY groups values based on group by ports you define in
the transformation, returning one result for each group. If there Return Value
is no group by port, LAST_DAY treats all rows as one group,  Double value.
returning one value  NULL if a value passed to the function is NULL

Examples LOG
The following expression returns the last day of the month for  Returns the logarithm of a numeric value.
each date in the ORDER_DATE port:  Most often, you use this function to analyze scientific data rather
LAST_DAY( ORDER_DATE ) than business data
Syntax
ORDER_DATE RETURN VALUE LOG( base, exponent )
Apr 1 1998 12:00:00AM Apr 30 1998 12:00:00AM
Jan 6 1998 12:00:00AM Jan 31 1998 12:00:00AM Return Value
Feb 2 1996 12:00:00AM Feb 29 1996 12:00:00AM (Leap year)  Double value.
NULL NULL  NULL if a value passed to the function is NULL
Jul 31 1998 12:00:00AM Jul 31 1998 12:00:00AM
Example
LEAST  The following expression returns the logarithm for all values in
 Returns the smallest value from a list of input values. the NUMBERS port:
 By default, the match is case sensitive. LOG( BASE, EXPONENT )

Syntax BASE EXPONENT RETURN VALUE

LEAST( value1, [value2, ..., valueN,] CaseFlag ) 15 1 0
.09 10 -0.956244644696599
Example NULL 18 NULL
The following expression returns the smallest quantity of items 35.78 NULL NULL
ordered: -9 18 Error. (PowerCenter Integration
LEAST( QUANTITY1, QUANTITY2, QUANTITY3 ) Service does not write the row.)
0 5 Error. (PowerCenter Integration
QUANTITIY1 QUANTITY2 QUANTITY3 RETURN VALUE Service does not write the row.)
150 756 27 27 10 -2 Error. (PowerCenter Integration
NULL Service does not write the row.)
5000 97 17 17
120 1724 965 120

112
 The PowerCenter Integration Service displays an error and does  Lowercase character string. If the data contains multibyte
not write the row if you pass a negative number, 0, or 1 as a characters, the return value depends on the code page and data
base value, or if you pass a negative value for the exponent. movement mode of the Integration Service.
 NULL if a value in the selected port is NULL.
LOOKUP
Note: This function is not supported in mapplets. LPAD
 Use the Lookup transformation rather than the LOOKUP function  Adds a set of blanks or characters to the beginning of a string to
to look up values in PowerCenter mappings. set the string to a specified length.
 If you use the LOOKUP function in a mapping, you need to enable
the lookup caching option for 3.5 compatibility in the session Syntax
properties LPAD( first_string, length [,second_string] )
 This option exists expressly for PowerMart 3.5 users who want to
continue using the LOOKUP function, rather than creating Return Value
Lookup transformation  String of the specified length.
 NULL if a value passed to the function is NULL or if length is a
Syntax negative number.
LOOKUP( result, search1, value1 [, search2, value2]... )
Examples
Return Value  The following expression standardizes numbers to six digits by
 Result if all searches find matching values. If the PowerCenter padding them with leading zeros.
Integration Service finds matching values, it returns the result LPAD( PART_NUM, 6, '0')
from the same row as the search1 argument.
 NULL if the search does not find any matching values. PART_NUM RETURN VALUE
 Error if the search finds more than one matching value. 702 000702
1 000001
0553 000553
484834 484834
Example
 The following expression searches the lookup source :TD.SALES  LPAD counts the length from left to right.
for a specific item ID and price, and returns the item name if  If the first string is longer than the length, LPAD truncates the
both searches find a match: string from right to left.
LOOKUP( :TD.SALES.ITEM_NAME, :TD.SALES.ITEM_ID, 10,  For example, LPAD(‘alphabetical’, 5, ‘x’) returns the string ‘alpha’.
:TD.SALES.PRICE, 15.99 )
 If the second string is longer than the total characters needed to
ITEM_NAME ITEM_ID PRICE return the specified length, LPAD uses a portion of the second
Regulator 5 100.00 string:
Flashlight 10 15.99 LPAD( ITEM_NAME, 16, '*..*' )
Halogen Flashlight 15 15.99
NULL 20 15.99 ITEM_NAME RETURN VALUE
RETURN VALUE: Flashlight Flashlight *..**.Flashlight
Compass *..**..**Compass
 When you compare char and varchar values, the LOOKUP Regulator System Regulator System
function returns a result only if the two rows match. Safety Knife *..*Safety Knife
 This means that both the value and the length for each row must
match (use trim) LTRIM
 Removes blanks or characters from the beginning of a string.
LOWER  You can use LTRIM with IIF or DECODE in an Expression or Update
 Converts uppercase string characters to lowercase Strategy transformation to avoid spaces in a target table.
 If you do not specify a trim_set parameter in the expression:
Return Value - In UNICODE mode, LTRIM removes both single- and double-byte
spaces from the beginning of a string.
- In ASCII mode, LTRIM removes only single-byte spaces.
113

 If you use LTRIM to remove characters from a string, LTRIM
compares the trim_set to each character in the string argument,
character-by-character, starting with the left side of the string.
 If the character in the string matches any character in the
trim_set, LTRIM removes it.
 LTRIM continues comparing and removing characters until it fails
to find a matching character in the trim_set.
 Then it returns the string, which does not include matching
characters.
Syntax
LTRIM( string [, trim_set] )
Return Value
 String. The string values with the specified characters in the
trim_set argument removed.
 NULL if a value passed to the function is NULL. If the trim_set is
NULL, the function returns NULL.
Example
 The following expression removes the characters ‘S’ and ‘.’ from
the strings in the LAST_NAME port:
LTRIM( LAST_NAME, 'S.')
 Removes blanks or characters from the end of a string.
 If you do not specify a trim_set parameter in the expression:
- In UNICODE mode, RTRIM removes both single- and double-
byte spaces from the end of a string.
- In ASCII mode, RTRIM removes only single-byte spaces.
 If you use RTRIM to remove characters from a string, RTRIM
compares the trim_set to each character in the string argument,
character-by-character, starting with the right side of the string.
 If the character in the string matches any character in the
trim_set, RTRIM removes it.
 RTRIM continues comparing and removing characters until it fails
to find a matching character in the trim_set.
 It returns the string without the matching characters.
Syntax
RTRIM( string [, trim_set] )
Example
 The following expression removes the characters ‘re’ from the
strings in the LAST_NAME port:
RTRIM( LAST_NAME, 're')

LAST_NAME RETURN VALUE

LAST_NAME RETURN VALUE Nelson Nelson
Nelson Nelson Page Pag
Osborne Osborne Osborne Osborn
NULL NULL NULL NULL
S. MacDonald MacDonald Sawyer Sawy
Sawyer awyer H. Bender H. Bend
H. Bender H. Bender Steadman Steadman
Steadman teadman
 RTRIM removes ‘e’ from Page even though ‘r’ is the first character
 LTRIM removes ‘S.’ from S. MacDonald and the ‘S’ from both in the trim_set.
Sawyer and Steadman, but not the period from H. Bender.  This is because RTRIM searches, character-by-character, for the
 This is because LTRIM searches, character-by-character, for the set of characters you specify in the trim_set argument.
set of characters you specify in the trim_set argument.  If the last character in the string matches the first character in the
 If the first character in the string matches the first character in trim_set, RTRIM removes it.
the trim_set, LTRIM removes it.  If, however, the last character in the string does not match,
 Then LTRIM looks at the second character in the string. If it RTRIM compares the second character in the trim_set.
matches the second character in the trim_set, LTRIM removes it,  If the second from last character in the string matches the second
and so on. character in the trim_set, RTRIM removes it, and so on.
 When the first character in the string does not match the
corresponding character in the trim_set, LTRIM returns the  When the character in the string fails to match the trim_set,
string and evaluates the next row. RTRIM returns the string and evaluates the next row.
 In the example of H. Bender, H does not match either character in  In the last example, the last character in Nelson does not match
the trim_set argument, so LTRIM returns the string in the any character in the trim_set argument, so RTRIM returns the
LAST_NAME port and moves to the next row. string 'Nelson' and evaluates the next row.

RTRIM MAKE_DATE_TIME
114
 Returns the date and time based on the input values.
Syntax
Syntax MAX( numeric_value [, filter_condition] )
MAKE_DATE_TIME( year, month, day, hour, minute, second,
nanosecond ) Return Value
 Numeric value
Return Value  NULL if all values passed to the function are NULL or if no rows
Date as MM/DD/YYYY HH24:MI:SS. Returns a null value if you do are selected (for example, the filter condition evaluates to FALSE
not pass the function a year, month, or day. or NULL for all rows).
 Note: If the return value is Decimal with precision greater than
Example 15, you can enable high precision to ensure decimal precision up
The following expression creates a date and time from the input to 28 digits
ports:
MAKE_DATE_TIME( SALE_YEAR, SALE_MONTH, SALE_DAY, SALE_HOUR,  If a value is NULL, MAX ignores it. However, if all values passed
SALE_MIN, SALE_SEC )
from the port are NULL, MAX returns NULL.

SALE_YR SALE_MTH SALE_DAY SALE_HR SALE_MIN SALE_SEC

 MAX groups values based on group by ports you define in the
RETURN VALUE
transformation, returning one result for each group.
2002 10 27 8 36 22
 If there is no group by port, MAX treats all rows as one group,
10/27/2002 08:36:22
returning one value.
2000 6 15 15 17
06/15/2000 15:17:00
2003 1 3 22 45
01/03/2003 00:22:45
MAX/MIN (String)
04 3 30 12 5 10
 Returns the highest string value found within a port or group.
03/30/0004 12:05:10
 You can apply a filter to limit the rows in the search.
99 12 12 5 16
 You can nest only one other aggregate function within MAX.
12/12/0099 05:00:16
Note: The MAX function uses the same sort order that the Sorter
MAX/MIN (Dates)
transformation uses. However, the MAX function is case sensitive,
 Returns the latest date found within a port or group.
and the Sorter transformation may not be case sensitive.
 You can apply a filter to limit the rows in the search.
 You can also use MAX to return the latest date or the largest
 You can nest only one other aggregate function within MAX.
numeric value in a port or group.
 You can also use MAX to return the largest numeric value or the
highest string value in a port or group
Syntax
MAX( string [, filter_condition] )
Syntax
MAX( date [, filter_condition] )
Return Value
 String.
Return Value
 NULL if all values passed to the function are NULL, or if no rows
 Date.
are selected (for example, the filter condition evaluates to FALSE
 NULL if all values passed to the function are NULL, or if no rows
or NULL for all rows).
are selected (for example, the filter condition evaluates to FALSE
or NULL for all rows).
 MAX groups values based on group by ports you define in the
transformation, returning one result for each group.
MAX/MIN (Numbers)
 If there is no group by port, MAX treats all rows as one group,
 Returns the maximum numeric value found within a port or
returning one value
group.
 You can apply a filter to limit the rows in the search.
MD5
 You can nest only one other aggregate function within MAX.
 Calculates the checksum of the input value. The function uses
 You can also use MAX to return the latest date or the highest
Message-Digest algorithm 5 (MD5).
string value in a port or group.

115
 MD5 is a oneway cryptographic hash function with a 128-bit hash  It encodes both uppercase and lowercase letters in uppercase.
value.
 You can conclude that input values are different when the  METAPHONE encodes characters according to the following list of
checksums of the input values are different. Use MD5 to verify rules:
data integrity. - Skips vowels (A, E, I, O, and U) unless one of them is the first
character of the input string. METAPHONE(‘CAR’) returns ‘KR’ and
Syntax METAPHONE(‘AAR’) returns ‘AR’.
MD5( value ) - Uses special encoding guidelines

Return Value MOD

Unique 32-character string of hexadecimal digits 0-9 and a-f.  Returns the remainder of a division calculation. For example,
NULL if the input is a null value. MOD(8,5) returns 3.

MEDIAN Syntax
 Returns the median of all values in a selected port. MOD( numeric_value, divisor )
 If there is an even number of values in the port, the median is the
average of the middle two values when all values are placed Return Value
ordinally on a number line. If there is an odd number of values  Numeric value of the datatype you pass to the function. The
in the port, the median is the middle number. remainder of the numeric value divided by the divisor.
 You can nest only one other aggregate function within MEDIAN,  NULL if a value passed to the function is NULL.
and the nested function must return a Numeric datatype.
Examples
 The PowerCenter Integration Service reads all rows of data to  The following expression returns the modulus of the values in the
perform the median calculation. PRICE port divided by the values in the QTY port:
 The process of reading rows of data to perform the calculation MOD( PRICE, QTY )
may affect performance.
 Optionally, you can apply a filter to limit the rows you read to PRICE QTY RETURN VALUE
calculate the median. 10.00 2 0
12.00 5 2
Syntax 9.00 2 1
MEDIAN( numeric_value [, filter_condition ] ) 15.00 3 0
NULL 3 NULL
Return Value 20.00 NULL NULL
 Numeric value. 25.00 0 Error. Integration Service does not write row.
 NULL if all values passed to the function are NULL, or if no rows
are selected. For example, the filter condition evaluates to FALSE  The last row (25, 0) produced an error because you cannot divide
or NULL for all rows. by 0.
 To avoid dividing by 0, you can create an expression similar to the
Note: If the return value is Decimal with precision greater than following, which returns the modulus of Price divided by
15, you can enable high precision to ensure decimal precision up Quantity only if the quantity is not 0.
to 28 digits.  If the quantity is 0, the function returns NULL:
MOD( PRICE, IIF( QTY = 0, NULL, QTY ))
 MEDIAN groups values based on group by ports you define in the
transformation, returning one result for each group. PRICE QTY RETURN VALUE
 If there is no group by port, MEDIAN treats all rows as one group, 10.00 2 0
returning one value 12.00 5 2
9.00 2 1
METAPHONE 15.00 3 0
 Encodes string values. You can specify the length of the string NULL 3 NULL
that you want to encode. 20.00 NULL NULL
 METAPHONE encodes characters of the English language 25.00 0 NULL
alphabet (A-Z).
116
 The last row (25, 0) produced a NULL rather than an error
because the IIF function replaces NULL with the 0 in the QTY ROW_NO SALES RETURN VALUE
port. 1 600 NULL
2 504 NULL
MOVINGAVG 3 36 NULL
 Returns the average (row-by-row) of a specified set of rows. 4 100 NULL
 Optionally, you can apply a condition to filter rows before 5 550 1790
calculating the moving average. 6 39 1229
7 490 1215
Syntax
MOVINGAVG( numeric_value, rowset [, filter_condition] )  The function returns the sum for a set of five rows: 1790 based
on rows 1 through 5, 1229 based on rows 2 through 6, and 1215
Return Value based on rows 3 through 7
Numeric value.

 MOVINGAVG ignores null values when calculating the moving

average. However, if all values are NULL, the function returns
NULL thereafter, returns the average for the last five rows read:
MOVINGAVG( SALES, 5 )
NPER
ROW_NO SALES RETURN VALUE  Returns the number of periods for an investment based on a
1 600 NULL constant interest rate and periodic, constant payments.
2 504 NULL
3 36 NULL Syntax
4 100 NULL NPER( rate, present value, payment [, future value, type] )
5 550 358
6 39 245.8 Return Value
7 490 243 Numeric.

 The function returns the average for a set of five rows: 358 based
on rows 1 through 5, 245.8 based on rows 2 through 6, and 243
based on rows 3 through 7.
MOVINGSUM
 Returns the sum (row-by-row) of a specified set of rows.
 Optionally, you can apply a condition to filter rows before
calculating the moving sum.
Syntax
MOVINGSUM( numeric_value, rowset [, filter_condition] )
Return Value
Numeric value.
 MOVINGSUM ignores null values when calculating the moving
sum. However, if all values are NULL, the function returns NULL.
Example
 The following expression returns the sum of orders for a
Stabilizing Vest, based on the first five rows in the Sales port,
and thereafter, returns the average for the last five rows read:
MOVINGSUM( SALES, 5 )
Example
 The present value of an investment is $2,000. Each payment is
$500 and the future value of the investment is $20,000.
 The following expression returns 9 as the number of periods for
which you need to make the payments:
NPER( 0.01, -2000, -500, 20000, TRUE )
PERCENTILE
 Calculates the value that falls at a given percentile in a group of
numbers.
 You can nest only one other aggregate function within
PERCENTILE, and the nested function must return a Numeric
datatype.
 The PowerCenter Integration Service reads all rows of data to
perform the percentile calculation.
 The process of reading rows to perform the calculation may affect
performance.
 Optionally, you can apply a filter to limit the rows you read to
calculate the percentile
Syntax
PERCENTILE( numeric_value, percentile [, filter_condition ] )

117
 Return Value  The following expression returns -2111.64 as the monthly
Numeric value. payment amount of a loan:
PMT( 0.01, 10, 20000 )
 If a value is NULL, PERCENTILE ignores the row. However, if all
values in a group are NULL, PERCENTILE returns NULL. POWER
 Returns a value raised to the exponent you pass to the function.
 PERCENTILE groups values based on group by ports you define in
the transformation, returning one result for each group. Syntax
 If there is no group by port, PERCENTILE treats all rows as one POWER( base, exponent )
group, returning one value
Return Value
Example Double value.
 The PowerCenter Integration Service calculates a percentile using
the following logic: Example
 Use the following guidelines for this equation:  The following expression returns the values in the Numbers port
- x is the number of elements in the group of values for which raised to the values in the Exponent port:
you are calculating a percentile. POWER( NUMBERS, EXPONENT )
- If i < 1, PERCENTILE returns the value of the first element in the NUMBERS EXP RETURN VALUE
list. 10.0 2.0 100
- If i is an integer value, PERCENTILE returns the value of the ith 3.5 6.0 1838.265625
element in the list 3.5 5.5 982.594307804838
- Otherwise PERCENTILE returns the value of n: NULL 2.0 NULL
10.0 NULL NULL
 The following expression returns the salary that falls at the 75th -3.0 -6.0 0.00137174211248285
percentile of salaries greater than $50,000: 3.0 -6.0 0.00137174211248285
PERCENTILE( SALARY, 75, SALARY > 50000 ) -3.0 6.0 729.0
-3.0 5.5 729.0
SALARY
125000.0  The value -3.0 raised to 6 returns the same results as -3.0 raised
27900.0 to 5.5.
100000.0  If the base is negative, the exponent must be an integer.
NULL  Otherwise, the PowerCenter Integration Service rounds the
55000.0 exponent to the nearest integer value.
9000.0
85000.0 PV
86000.0  Returns the present value of an investment.
48000.0
99000.0 Syntax
RETURN VALUE: 106250.0 PV( rate, terms, payment [, future value, type] )

PMT Return Value

 Returns the payment for a loan based on constant payments and Numeric.
a constant interest rate.
Example
Syntax  The following expression returns 12,524.43 as the amount you
PMT( rate, terms, present value[, future value, type] ) must deposit in the account today to have a future value of
$20,000 in one year if you also deposit $500 at the beginning of
Return Value each period:
Numeric. PV( 0.0075, 12, -500, 20000, TRUE )

Example RAND

118
 Returns a random number between 0 and 1. This is useful for  You might use REG_EXTRACT in an expression to extract middle
probability scenarios. names from a regular expression that matches first name,
middle name, and last name.
Syntax  For example, the following expression returns the middle name
RAND( seed ) of a regular expression:
REG_EXTRACT( Employee_Name, '(\w+)\s+(\w+)\s+(\w+)',2)
Return Value
Numeric. Employee_Name Return Value
Stephen Graham Smith Graham
 For the same seed, the PowerCenter Integration Service Juan Carlos Fernando Carlos
generates the same sequence of numbers
REG_MATCH
Example  Returns whether a value matches a regular expression pattern.
 The following expression may return a value of  This lets you validate data patterns, such as IDs, telephone
0.417022004702574: numbers, postal codes, and state names.
RAND (1) Note: Use the REG_REPLACE function to replace a character
pattern in a string with a new character pattern.
RATE
 Returns the interest rate earned per period by a security. Syntax
REG_MATCH( subject, pattern )
Syntax
RATE( terms, payment, present value[, future value, type] ) Return Value
 TRUE if the data matches the pattern.
Return Value  FALSE if the data does not match the pattern.
Numeric.  NULL if the input is a null value or if the pattern is NULL.

Example
 The following expression returns 0.0077 as the monthly interest
rate of a loan:
RATE( 48, -500, 20000 )
 To calculate the annual interest rate of the loan, multiply 0.0077
by 12. The annual interest rate is 0.0924 or 9.24%.
Example
 You might use REG_MATCH in an expression to validate
telephone numbers.
 For example, the following expression matches a 10-digit
telephone number against the pattern and returns a Boolean
value based on the match:
REG_MATCH (Phone_Number, '(\d\d\d-\d\d\d-\d\d\d\d)' )

REG_EXTRACT Phone_Number Return Value

 Extracts subpatterns of a regular expression within an input 408-555-1212 TRUE
value. NULL
 For example, from a regular expression pattern for a full name, 510-555-1212 TRUE
you can extract the first name or last name. 92 555 51212 FALSE
 Note: Use the REG_REPLACE function to replace a character 650-555-1212 TRUE
pattern in a string with another character pattern 415-555-1212 TRUE
831 555 12123 FALSE
Syntax
REG_EXTRACT( subject, 'pattern', subPatternNum )  You can also use REG_MATCH for the following tasks:
- To verify that a value matches a pattern. This use is similar to
Return Value the SQL LIKE function.
 Returns the value of the nth subpattern that is part of the input - To verify that values are characters. This use is similar to the SQL
value. The nth subpattern is based on the value you specify for IS_CHAR function.
subPatternNum.
 NULL if the input is a null value or if the pattern is null  To verify that a value matches a pattern, use a period (.) and an
asterisk (*) with the REG_MATCH function in an expression.
Example

119
 A period matches any one character. An asterisk matches 0 or
more instances of values that follow it. Return Value
 String.
 For example, use the following expression to find account  Empty string if REPLACECHR removes all characters in InputString.
numbers that begin with 1835:  NULL if InputString is NULL.
REG_MATCH(ACCOUNT_NUMBER, ‘1835.*’)  InputString if OldCharSet is NULL or empty

 To verify that values are characters, use a REG_MATCH function

with the regular expression [a-zA-Z]+. a-z matches all lowercase
characters.
 A-Z matches all uppercase characters.
 The plus sign (+) indicates that there should be at least one
character.

 For example, use the following expression to verify that a list of

last names contain only characters:
REG_MATCH(LAST_NAME, ‘[a-zA-Z]+’)

REG_REPLACE
 Replaces characters in a string with another character pattern.
 By default, REG_REPLACE searches the input string for the
character pattern you specify and replaces all occurrences with
the replacement pattern.
 You can also indicate the number of occurrences of the pattern
you want to replace in the string.

Syntax REPLACESTR
REG_REPLACE( subject, pattern, replace, numReplacements )  Replaces characters in a string with a single character, multiple
characters, or no character.
Return Value  REPLACESTR searches the input string for all strings you specify
String and replaces them with the new string you specify.
Syntax
Example REPLACESTR ( CaseFlag, InputString, OldString1, [OldString2, ...
 The following expression removes additional spaces from the OldStringN,] NewString )
Employee name data for each row of the Employee_name port: Return Value
REG_REPLACE( Employee_Name, ‘\s+’, ‘ ’)  String.
 Empty string if REPLACESTR removes all characters in InputString.
Employee_Name RETURN VALUE  NULL if InputString is NULL.
Adam Smith Adam Smith  InputString if all OldString arguments are NULL or empty.
Greg Sanders Greg Sanders
Sarah Fe Sarah Fe
Sam Cooper Sam Cooper

REPLACECHR
 Replaces characters in a string with a single character or no
character.
 REPLACECHR searches the input string for the characters you
specify and replaces all occurrences of all characters with the
new character you specify.

Syntax
REPLACECHR( CaseFlag, InputString, OldCharSet, NewChar )

120

REVERSE
Reverses the input string.
Syntax
REVERSE( string )
<Print 123-127>
SETCOUNTVARIABLE
 Counts the rows evaluated by the function and increments the
current value of a mapping variable based on the count.
 Increases the current value by one for each row marked for
insertion.
 Decreases the current value by one for each row marked for
deletion.
 Keeps the current value the same for each row marked for
update or reject.
 Returns the new current value.
 At the end of a successful session, the PowerCenter Integration
Service saves the last current value to the repository.
 When used with a session that contains multiple partitions, the
PowerCenter Integration Service generates different current
values for each partition.
 At the end of the session, it determines the total count for all
partitions and saves the total to the repository.
 Unless overridden, it uses the saved value as the initial value of
the variable for the next time you use this session.
 The PowerCenter Integration Service does not save the final value
of a mapping variable to the repository when any of the
following are true:
- The session fails to complete.
121
- The session is configured for a test load.
- The session is a debug session.
- The session runs in debug mode and is configured to discard
session output.
Syntax
SETCOUNTVARIABLE( $$Variable )
SET_DATE_PART
 Sets one part of a Date/Time value to a value you specify. With
SET_DATE_PART, you can change the following parts of a date:
Year - Change the year by entering a positive integer in the value
argument. Use any of the year format strings:
Y, YY, YYY, or YYYY to set the year. For example, the following
expression changes the year to 2001 for all dates in the
SHIP_DATE port:
SET_DATE_PART( SHIP_DATE, 'YY', 2001 )
Month - Change the month by entering a positive integer
between 1 and 12 (January=1 and December=12) in the value
argument. Use any of the month format strings: MM, MON,
MONTH to set the month. For example, the following expression
changes the month to October for all dates in the SHIP_DATE
port:
SET_DATE_PART( SHIP_DATE, 'MONTH', 10 )
Day - Change the day by entering a positive integer between 1
and 31 (except for the months that have less than 31 days:
February, April, June, September, and November) in the value
argument. Use any of the month format strings (D, DD, DDD, DY,
and DAY) to set the day. For example, the following expression
changes the day to 10 for all dates in the SHIP_DATE port:
SET_DATE_PART( SHIP_DATE, 'DD', 10 )
Syntax
SET_DATE_PART( date, format, value )
Return Value
 Date in the same format as the source date with the specified
part changed.
 NULL if a value passed to the function is NULL.
Examples
 The following expressions change the hour to 4PM for each date
in the DATE_PROMISED port:
SET_DATE_PART( DATE_PROMISED, 'HH', 16 )
SET_DATE_PART( DATE_PROMISED, 'HH12', 16 )
SET_DATE_PART( DATE_PROMISED, 'HH24', 16 )
DATE_PROMISED RETURN VALUE
Jan 1 1997 12:15:56AM Jan 1 1997 4:15:56PM
Feb 13 1997 2:30:01AM Feb 13 1997 4:30:01PM
Mar 31 1997 5:10:15PM Mar 31 1997 4:10:15PM
 Dec 12 1997 8:07:33AM Dec 12 1997 4:07:33PM  Then, it saves the largest value to the repository. For example,
NULL NULL the last evaluated value for $$MaxItems in each partition is as
follows:
SETMAXVARIABLE/SETMINVARIABLE Partition Final Current Value for $$MaxItems
 Sets the current value of a mapping variable to the higher of two Partition 1 35
values: the current value of the variable or the value you specify. Partition 2 23
 Returns the new current value. Partition 3 22
 The function executes only if a row is marked as insert.
 SETMAXVARIABLE ignores all other row types and the current SETVARIABLE
value remains unchanged.  Sets the current value of a mapping variable to a value you
specify.
 At the end of a successful session, the PowerCenter Integration  Returns the specified value.
Service saves the final current value to the repository.  The SETVARIABLE function executes only if a row is marked as
 When used with a session that contains multiple partitions, the insert or update. SETVARIABLE ignores all other row types and
PowerCenter Integration Service generates different current the current value remains unchanged.
values for each partition.
 At the end of the session, it saves the highest current value  At the end of a successful session, the PowerCenter Integration
across all partitions to the repository. Service compares the final current value of the variable to the
 Unless overridden, it uses the saved value as the initial value of start value of the variable.
the variable for the next session run.  Based on the aggregate type of the variable, it saves a final
 When used with a string mapping variable, SETMAXVARIABLE current value to the repository.
returns the higher string based on the sort order selected for  Unless overridden, it uses the saved value as the initial value of
the session. the variable for the next session run.

Examples Return Value

 The following expression compares the number of items Current value of the variable.
purchased in each transaction with a mapping variable $ When value is NULL, the PowerCenter Integration Service returns
$MaxItems. the current value of $$Variable.
 It sets $$MaxItems to the higher of two values and returns the
historically highest number of items purchased in a single Examples
transaction to the MAX_ITEMS port.  The following expression sets a mapping variable $$Time to the
 The initial value of $$MaxItems from the previous session run is system date at the time the PowerCenter Integration Service
22. evaluates the row and returns the system date to the SET_$
SETMAXVARIABLE ($$MAXITEMS, ITEMS) $TIME port:
SETVARIABLE ($$Time, SYSDATE)
TRANSACTION ITEMS MAX_ITEMS
0100002 12 22 TRANSACTION TOTAL SET_$$TIME
0100003 5 22 0100002 534.23 10/10/2000 01:34:33
0100004 18 22 0100003 699.01 10/10/2000 01:34:34
0100005 35 35 0100004 97.50 10/10/2000 01:34:35
0100006 5 35 0100005 116.43 10/10/2000 01:34:36
0100007 14 35 0100006 323.95 10/10/2000 01:34:37
 At the end of the session, the PowerCenter Integration Service
 At the end of the session, the PowerCenter Integration Service saves 10/10/2000 01:34:37 to the repository as the last
saves ‘35’ to the repository as the maximum current value for $ evaluated current value for $$Time. The next time the session
$MaxItems. runs, the PowerCenter Integration Service evaluates all
 The next time the session runs, the PowerCenter Integration references to $$Time to 10/10/2000 01:34:37
Service evaluates the initial value to $$MaxItems to ‘35’.
SIGN
 If the same session contains three partitions, the PowerCenter Returns whether a numeric value is positive, negative, or 0.
Integration Service evaluates $$MaxItems for each partition.
Syntax
122
 SIGN( numeric_value ) - Skips numbers in string. For example, both SOUNDEX(‘Joh12n’)
and SOUNDEX(‘1John’) return ‘J500’.
Return Value - Returns NULL if string is NULL or if all the characters in string are
 -1 for negative values. not letters of the English alphabet.
 0 for 0.
 1 for positive values. Syntax
 NULL if NULL. SOUNDEX( string )

SOUNDEX Return Value

 Encodes a string value into a four-character string.  String.
 SOUNDEX works for characters in the English alphabet (A-Z).  NULL if one of the following conditions is true:
 It uses the first character of the input string as the first character - If value passed to the function is NULL.
in the return value and encodes the remaining three unique - No character in string is a letter of the English alphabet.
consonants as numbers. - string is empty.
 SOUNDEX encodes characters according to the following list of Example
rules:  The following expression encodes the values in the
- Uses the first character in string as the first character in the EMPLOYEE_NAME port:
return value and encodes it in uppercase. For example, both SOUNDEX( EMPLOYEE_NAME )
SOUNDEX(‘John’) and SOUNDEX(‘john’) return ‘J500’.
- Encodes the first three unique consonants following the first EMPLOYEE_NAME RETURN VALUE
character in string and ignores the rest. For example, both John J500
SOUNDEX(‘JohnRB’) and SOUNDEX(‘JohnRBCD’) return ‘J561’. William W450
- Assigns a single code to consonants that sound alike. jane J500
joh12n J500
 The following table lists SOUNDEX encoding guidelines for 1abc A120
consonants: NULL NULL

STDDEV
 Returns the standard deviation of the numeric values you pass to
this function.
 STDDEV is used to analyze statistical data.
 You can nest only one other aggregate function within STDDEV,
and the nested function must return a Numeric datatype.

Syntax
STDDEV( numeric_value [,filter_condition] )

Return Value
 Numeric value.
 NULL if all values passed to the function are NULL or if no rows
are selected (for example, the filter condition evaluates to FALSE
- Skips the characters A, E, I, O, U, H, and W unless one of them is
or NULL for all rows).
the first character in string. For example, SOUNDEX(‘A123’)
returns ‘A000’ and SOUNDEX(‘MAeiouhwC’) returns ‘M000’.
 STDDEV groups values based on group by ports you define in the
- If string produces fewer than four characters, SOUNDEX pads
transformation, returning one result for each group.
the resulting string with zeroes. For example, SOUNDEX(‘J’)
 If there is no group by port, STDDEV treats all rows as one group,
returns ‘J000’.
returning one value.
- If string contains a set of consecutive consonants that use the
same code listed in “SOUNDEX” on page 140, SOUNDEX encodes
SUBSTR
the first occurrence and skips the remaining occurrences in the
 Returns a portion of a string.
set. For example, SOUNDEX(‘AbbpdMN’) returns ‘A135’.
 SUBSTR counts all characters, including blanks, starting at the
beginning of the string.
123
 Syntax  The return value is ‘bcd’. Compare this result to the following
SUBSTR( string, start [,length] ) example:
SUBSTR('abcd', -2, 8)
Return Value  The return value is ‘cd’.
 String.
 Empty string if you pass a negative or 0 length value. SUM
 NULL if a value passed to the function is NULL.  Returns the sum of all values in the selected port.
 Optionally, you can apply a filter to limit the rows you read to
Examples calculate the total.
 The following expressions return the area code for each row in  You can nest only one other aggregate function within SUM, and
the Phone port: the nested function must return a Numeric datatype
SUBSTR( PHONE, 0, 3 ) Syntax
SUM( numeric_value [, filter_condition ] )
PHONE RETURN VALUE
809-555-0269 809 Return Value
357-687-6708 357  Numeric value.
NULL NULL  NULL if all values passed to the function are NULL or if no rows
are selected (for example, the filter condition evaluates to FALSE
 You can also pass a negative start value to return the phone or NULL for all rows).
number for each row in the Phone port.
 The expression still reads the source string from left to right when  You can perform arithmetic on the values passed to SUM before
returning the result of the length argument: the function calculates the total. For example:
SUBSTR( PHONE, -8, 3 ) SUM( QTY * PRICE - DISCOUNT )

PHONE RETURN VALUE SYSTIMESTAMP

808-555-0269 555  Returns the current date and time of the node hosting the
809-555-3915 555 PowerCenter Integration Service with precision to the
357-687-6708 687 nanosecond.
NULL NULL  The precision to which you display the date and time depends on
the platform.
 You can nest INSTR in the start or length argument to search for a  The return value of the function varies depending on how you
specific string and return its position. configure the argument:
 The following expression evaluates a string, starting from the end - When you configure the argument of SYSTIMESTAMP as a
of the string. variable, the PowerCenter Integration Service evaluates the
 The expression finds the last (rightmost) space in the string and function for each row in the transformation.
then returns all characters preceding it: - When you configure the argument of SYSTIMESTAMP as a
SUBSTR( CUST_NAME,1,INSTR( CUST_NAME,' ' ,-1,1 ) - 1 ) constant, the PowerCenter Integration Service evaluates the
function once and retains the value for each row in the
CUST_NAME RETURN VALUE transformation.
PATRICIA JONES PATRICIA
MARY ELLEN SHAH MARY ELLEN Syntax
SYSTIMESTAMP( [format] )
 The following expression removes the character '#' from a string:
SUBSTR( CUST_ID, 1, INSTR(CUST_ID, '#')-1 ) || SUBSTR( CUST_ID, TO_BIGINT
INSTR(CUST_ID, '#')+1 )  Converts a string or numeric value to a bigint value.
 TO_BIGINT syntax contains an optional argument that you can
 When the length argument is longer than the string, SUBSTR choose to round the number to the nearest integer or truncate
returns all the characters from the start position to the end of the decimal portion.
the string.  TO_BIGINT ignores leading blanks.
 Consider the following example:
SUBSTR('abcd', 2, 8)
Syntax

124
 TO_BIGINT( value [, flag] )
DATE_PROMISED RETURN VALUE
Return Value Apr 1 1998 12:00:10AM '04/01/1998 00:00:10.000000'
 Bigint. Feb 22 1998 01:31:10PM '02/22/1998 13:31:10.000000'
 NULL if the value passed to the function is NULL. Oct 24 1998 02:12:30PM '10/24/1998 14:12:30.000000'
 0 if the value passed to the function contains alphanumeric NULL NULL
characters
TO_CHAR (Numbers)
Examples  Converts numeric values to text strings. TO_CHAR also converts
The following expressions use values from the port IN_TAX: dates to strings.
TO_BIGINT( IN_TAX, TRUE ) TO_CHAR converts numeric values to text strings as follows:
- Converts double values to strings of up to 16 digits and provides
IN_TAX RETURN VALUE accuracy up to 15 digits. If you pass a number with more than 15
'7245176201123435.6789' 7245176201123435 digits, TO_CHAR rounds the number to the sixteenth digit.
'7245176201123435.2' 7245176201123435 - Returns decimal notation for numbers in the ranges (-1e16,-1e-
'7245176201123435.2.48' 7245176201123435 16] and [1e-16, 1e16). TO_CHAR returns scientific notation for
NULL NULL numbers outside these ranges.
'A12.3Grove' 0 Note: The PowerCenter Integration Service converts the values
' 176201123435.87' 176201123435 1e-16 and -1e16 to scientific notation, but returns the values 1e-
'-7245176201123435.2’ -7245176201123435 16 and -1e-16 in decimal notation.
'-7245176201123435.23' -7245176201123435
-9223372036854775806.9 -9223372036854775806 Syntax
9223372036854775806.9 9223372036854775806 TO_CHAR( numeric_value )

TO_CHAR (Dates) Return Value

 Converts dates to character strings.  String.
 TO_CHAR also converts numeric values to strings.  NULL if a value passed to the function is NULL.
 You can convert the date into any format using the TO_CHAR
format strings. TO_DATE
 Converts a character string to a Date/Time datatype. You use the
Syntax TO_DATE format strings to specify the format of the source
TO_CHAR( date [,format] ) strings.
 The output port must be Date/Time for TO_DATE expressions.
Return Value  If you are converting two-digit years with TO_DATE, use either
 String. the RR or YY format string. Do not use the YYYY format string.
 NULL if a value passed to the function is NULL.
Syntax
Examples TO_DATE( string [, format] )
 The following expression converts the dates in the
DATE_PROMISED port to text in the format MON DD YYYY: Return Value
TO_CHAR( DATE_PROMISED, 'MON DD YYYY' )  Date.
 TO_DATE always returns a date and time. If you pass a string that
DATE_PROMISED RETURN VALUE does not have a time value, the date returned always includes
Apr 1 1998 12:00:10AM 'Apr 01 1998' the time 00:00:00.000000000.
Feb 22 1998 01:31:10PM 'Feb 22 1998'  You can map the results of this function to any target column
Oct 24 1998 02:12:30PM 'Oct 24 1998' with a datetime datatype.
NULL NULL  If the target column precision is less than nanoseconds, the
PowerCenter Integration Service truncates the datetime value to
 If you omit the format argument, TO_CHAR returns a string in the match the precision of the target column when it writes
date format specified in the session, by default, MM/DD/YYYY datetime values to the target.
HH24:MI:SS.US:  NULL if you pass a null value to this function.
TO_CHAR( DATE_PROMISED )

125
 TO_DECIMAL 'A12.3Grove' 0
 Converts a string or numeric value to a decimal value.
TO_DECIMAL ignores leading blanks. TO_INTEGER
 Converts a string or numeric value to an integer. TO_INTEGER
Syntax syntax contains an optional argument that you can choose to
TO_DECIMAL( value [, scale] ) round the number to the nearest integer or truncate the
decimal portion. TO_INTEGER ignores leading blanks.

Syntax
Return Value TO_INTEGER( value [, flag] )
 If the string contains a non-numeric character, converts the
numeric portion of the string up to the first nonnumeric Return Value
character.  Integer.
 If the first numeric character is non-numeric, returns 0.  NULL if the value passed to the function is NULL.
 Decimal of precision and scale between 0 and 28, inclusive.  0 if the value passed to the function contains alphanumeric
 NULL if a value passed to the function is NULL. characters

Example Examples
 This expression uses values from the port IN_TAX. The datatype is  The following expressions use values from the port IN_TAX. The
decimal with precision of 10 and scale of 3: PowerCenter Integration Service displays an error when the
TO_DECIMAL( IN_TAX, 3 ) conversion causes a numeric overflow:
TO_INTEGER( IN_TAX, TRUE )
IN_TAX RETURN VALUE
'15.6789' 15.679 IN_TAX RETURN VALUE
'60.2' 60.200 '15.6789' 15
'118.348' 118.348 '60.2' 60
NULL NULL '118.348' 118
'A12.3Grove' 0 ‘5,000,000,000’ Error. Integration Service doesn't write row.
‘711A1’ 711 NULL NULL
'A12.3Grove' 0
TO_FLOAT ' 123.87' 123
 Converts a string or numeric value to a double-precision floating '-15.6789' -15
point number (the Double datatype).
 TO_FLOAT ignores leading blanks. TRUNC (Dates)
 Truncates dates to a specific year, month, day, hour, minute,
Syntax second, millisecond, or microsecond.
TO_FLOAT( value )  You can also use TRUNC to truncate numbers.
 You can truncate the following date parts:
Return Value Year - If you truncate the year portion of the date, the function
 Double value. returns Jan 1 of the input year with the time set to
 0 if the value in the port is blank or a non-numeric character. 00:00:00.000000000. For example, the following expression
 NULL if a value passed to this function is NULL. returns 1/1/1997 00:00:00.000000000: TRUNC(12/1/1997
3:10:15, 'YY')
Example Month - If you truncate the month portion of a date, the function
 This expression uses values from the port IN_TAX: returns the first day of the month with the time set to
TO_FLOAT( IN_TAX ) 00:00:00.000000000. For example, the following expression
returns 4/1/1997 00:00:00.000000000: TRUNC(4/15/1997
IN_TAX RETURN VALUE 12:15:00, 'MM')
'15.6789' 15.6789 Day - If you truncate the day portion of a date, the function
'60.2' 60.2 returns the date with the time set to 00:00:00.000000000. For
'118.348' 118.348 example, the following expression returns 6/13/1997
NULL NULL 00:00:00.000000000: TRUNC(6/13/1997 2:30:45, 'DD')
126
 15.9928 15.992
Syntax NULL NULL
TRUNC( date [,format] )

Return Value
 Date.
 NULL if a value passed to the function is NULL. TRUNC( PRICE, -1 )

Examples PRICE RETURN VALUE

 The following expressions truncate the year portion of dates in 12.99 10.0
the DATE_SHIPPED port: -187.86 -180.0
TRUNC( DATE_SHIPPED, 'Y' ) 56.95 50.0
TRUNC( DATE_SHIPPED, 'YY' ) 1235.99 1230.0
TRUNC( DATE_SHIPPED, 'YYY' )
TRUNC( DATE_SHIPPED, 'YYYY' ) UPPER
Converts lowercase string characters to uppercase.
DATE_SHIPPED RETURN VALUE
Jan 15 1998 2:10:30AM Jan 1 1998 12:00:00.000000000 Syntax
Apr 19 1998 1:31:20PM Jan 1 1998 12:00:00.000000000 UPPER( string )
Jun 20 1998 3:50:04AM Jan 1 1998 12:00:00.000000000
Dec 20 1998 3:29:55PM Jan 1 1998 12:00:00.000000000 VARIANCE
NULL NULL  Returns the variance of a value you pass to it. VARIANCE is used
to analyze statistical data.
TRUNC (Numbers)  You can nest only one other aggregate function within VARIANCE,
 Truncates numbers to a specific digit. You can also use TRUNC to and the nested function must return a Numeric datatype.
truncate dates.
Syntax
Syntax VARIANCE( numeric_value [, filter_condition ] )
TRUNC( numeric_value [, precision] )

 If precision is a positive integer, TRUNC returns numeric_value

with the number of decimal places specified by precision.
 If precision is a negative integer, TRUNC changes the specified
digits to the left of the decimal point to zeros.
 If you omit the precision argument, TRUNC truncates the decimal
portion of numeric_value and returns an integer.
 If you pass a decimal precision value, the PowerCenter
Integration Service rounds numeric_value to the nearest integer
before evaluating the expression.

Return Value
 Numeric value.
 NULL if one of the arguments is NULL

Examples
 The following expressions truncate the values in the Price port:
TRUNC( PRICE, 3 )

PRICE RETURN VALUE

12.9995 12.999
-18.8652 -18.865
56.9563 56.956
127
  By default, the Integration Service uses one reader thread, one
transformation thread, and one writer thread to process a
session.
 The thread with the highest busy percentage identifies the
bottleneck in the session.
8. PERFORMANCE TUNING  The session log provides the following thread statistics:
Run time - Amount of time the thread runs.
Idle time - Amount of time the thread is idle. It includes the time
Complete the following tasks to improve session performance:
the thread waits for other thread processing within the
1. Optimize the target - Enables the Integration Service to write
application. Idle time includes the time the thread is blocked by
to the targets efficiently.
the Integration Service, but not the time the thread is blocked
2. Optimize the source - Enables the Integration Service to read
by the operating system.
source data efficiently.
Busy time - Percentage of the run time the thread is by according
3. Optimize the mapping - Enables the Integration Service to
to the following formula:
transform and move data efficiently.
(run time - idle time) / run time X 100
4. Optimize the transformation - Enables the Integration Service
You can ignore high busy percentages when the total run time is
to process transformations in a mapping efficiently.
short, such as under 60 seconds. This does not necessarily
5. Optimize the session - Enables the Integration Service to run
indicate a bottleneck.
the session more quickly.
Thread work time - The percentage of time the Integration
6. Optimize the grid deployments - Enables the Integration
Service takes to process each transformation in a thread. The
Service to run on a grid with optimal performance.
session log shows the following information for the
7. Optimize the PowerCenter components - Enables the
transformation thread work time:
Integration Service and Repository Service to function optimally.
Thread work time breakdown:
8. Optimize the system - Enables PowerCenter service processes
<transformation name>: <number> percent
to run more quickly
<transformation name>: <number> percent
<transformation name>: <number> percent
Bottlenecks -
Look for performance bottlenecks in the following order:
 If a transformation takes a small amount of time, the session log
1. Target
does not include it.
2. Source
 If a thread does not have accurate statistics, because the session
3. Mapping
ran for a short period of time, the session log reports that the
4. Session
statistics are not accurate.
5. System

Eliminating Bottlenecks Based on Thread Statistics

 Use the following methods to identify performance bottlenecks:
Complete the following tasks to eliminate bottlenecks based on
Run test sessions - You can configure a test session to read from
thread statistics:
a flat file source or to write to a flat file target to identify source
- If the reader or writer thread is 100% busy, consider using string
and target bottlenecks.
datatype in the source or target ports. Non-string ports require
Analyze performance details - Analyze performance details, such
more processing.
as performance counters, to determine where session
- If a transformation thread is 100% busy, consider adding a
performance decreases.
partition point in the segment. When you add partition points to
Analyze thread statistics - Analyze thread statistics to determine
the mapping, the Integration Service increases the number of
the optimal number of partition points.
transformation threads it uses for the session. However, if the
Monitor system performance - You can use system monitoring
machine is already running at or near full capacity, do not add
tools to view the percentage of CPU use, I/O waits, and paging to
more threads.
identify system bottlenecks. You can also use the Workflow
- If one transformation requires more processing time than the
Monitor to view system resource usage
others, consider adding a pass-through partition point to the
transformation.
Using Thread Statistics
 You can use thread statistics in the session log to identify source,
Example
target, or transformation bottlenecks.
When you run a session, the session log lists run information and
thread statistics similar to the following text:
128
 ***** RUN INFO FOR TGT LOAD ORDER GROUP [1],
CONCURRENT SET [1] *****
Thread [READER_1_1_1] created for [the read stage] of partition
point [SQ_two_gig_file_32B_rows] has completed.
Total Run Time = [505.871140] secs
Total Idle Time = [457.038313] secs
Busy Percentage = [9.653215]
Thread [TRANSF_1_1_1] created for [the transformation stage] of
partition point
[SQ_two_gig_file_32B_rows] has completed.
Total Run Time = [506.230461] secs
Total Idle Time = [1.390318] secs
Busy Percentage = [99.725359]
Thread work time breakdown:
LKP_ADDRESS: 25.000000 percent
SRT_ADDRESS: 21.551724 percent
RTR_ZIP_CODE: 53.448276 percent
Thread [WRITER_1_*_1] created for [the write stage] of partition
point [scratch_out_32B] has completed.
Total Run Time = [507.027212] secs
Total Idle Time = [384.632435] secs
Busy Percentage = [24.139686]
 In this session log, the total run time for the transformation
thread is 506 seconds and the busy percentage is 99.7%.
 This means the transformation thread was never idle for the 506
seconds.
 The reader and writer busy percentages were significantly
smaller, about 9.6% and 24%.
 In this session, the transformation thread is the bottleneck in the
mapping.
 To determine which transformation in the transformation thread
is the bottleneck, view the busy percentage of each
transformation in the thread work time breakdown.
 In this session log, the transformation RTR_ZIP_CODE had a busy
percentage of 53%.
Identifying Target Bottlenecks
To identify a target bottleneck, complete the following tasks:
- Configure a copy of the session to write to a flat file target. If
the session performance increases significantly, you have a target
bottleneck. If a session already writes to a flat file target, you
probably do not have a target bottleneck.
- Read the thread statistics in the session log. When the
Integration Service spends more time on the writer thread than
the transformation or reader threads, you have a target
bottleneck.
Eliminating Target Bottlenecks
Complete the following tasks to eliminate target bottlenecks:
129
- Have the database administrator optimize database
performance by optimizing the query.
- Increase the database network packet size.
- Configure index and key constraints.
Identifying Source Bottlenecks
 You can read the thread statistics in the session log to determine
if the source is the bottleneck.
 When the Integration Service spends more time on the reader
thread than the transformation or writer threads, you have a
source bottleneck.
 If the session reads from a relational source, use the following
methods to identify source bottlenecks:
- Filter transformation
- Read test mapping
- Database query
 If the session reads from a flat file source, you probably do not
have a source bottleneck.
Using a Filter Transformation
 You can use a Filter transformation in the mapping to measure
the time it takes to read source data.
 Add a Filter transformation after each source qualifier.
 Set the filter condition to false so that no data is processed
passed the Filter transformation.
 If the time it takes to run the new session remains about the
same, you have a source bottleneck.
Using a Read Test Mapping
 You can create a read test mapping to identify source
bottlenecks. A read test mapping isolates the read query by
removing the transformation in the mapping.
 To create a read test mapping, complete the following steps:
1. Make a copy of the original mapping.
2. In the copied mapping, keep only the sources, source
qualifiers, and any custom joins or queries.
3. Remove all transformations.
4. Connect the source qualifiers to a file target.
 Run a session against the read test mapping. If the session
performance is similar to the original session, you have a source
bottleneck.
Using a Database Query
 To identify source bottlenecks, execute the read query directly
against the source database.
 Copy the read query directly from the session log. Execute the
query against the source database with a query tool such as isql.
 Measure the query execution time and the time it takes for the
query to return the first row.
Eliminating Source Bottlenecks
 Complete the following tasks to eliminate source bottlenecks:
- Set the number of bytes the Integration Service reads per line
if the Integration Service reads from a flat file source.
- Have the database administrator optimize database
performance by optimizing the query.
- Increase the database network packet size.
- Configure index and key constraints.
- If there is a long delay between the two time measurements in
a database query, you can use an optimizer hint
Identifying Mapping Bottlenecks
To identify mapping bottlenecks, complete the following tasks:
- Read the thread statistics and work time statistics in the session
log. When the Integration Service spends more time on the
transformation thread than the writer or reader threads, you
have a transformation bottleneck. When the Integration Service
spends more time on one transformation, it is the bottleneck in
the transformation thread.
- Analyze performance counters. High errorrows and
rowsinlookupcache counters indicate a mapping bottleneck.
- Add a Filter transformation before each target definition. Set
the filter condition to false so that no data is loaded into the
target tables. If the time it takes to run the new session is the
same as the original session, you have a mapping bottleneck.
Eliminating Mapping Bottlenecks
- To eliminate mapping bottlenecks, optimize transformation
settings in mappings.
 If you do not have a source, target, or mapping bottleneck, you
may have a session bottleneck.
 Small cache size, low buffer memory, and small commit intervals
can cause session bottlenecks.
Identifying Session Bottlenecks
- To identify a session bottleneck, analyze the performance
details. Performance details display information about each
transformation, such as the number of input rows, output rows,
and error rows.
Eliminating Session Bottlenecks
- To eliminate session bottlenecks, optimize the session
Using the Workflow Monitor to Identify System Bottlenecks
CPU% - The percentage of CPU usage includes other external
tasks running on the system.
Memory usage - To troubleshoot, use system tools to check the
memory usage before and after running the session and then
compare the results to the memory usage while running the
session.
Swap usage - Swap usage is a result of paging due to possible
memory leaks or a high number of concurrent tasks.
130
Identifying System Bottlenecks on Windows
Use the Windows Performance Monitor to create a chart that
provides the following information:
Percent processor time - If you have more than one CPU, monitor
each CPU for percent processor time.
Pages/second - If pages/second is greater than five, you may
have excessive memory pressure (thrashing).
Physical disks percent times - The percent of time that the
physical disk is busy performing read or write requests.
Physical disks queue length - The number of users waiting for
access to the same disk device.
Server total bytes per second - The server has sent to and
received from the network.
Identifying System Bottlenecks on UNIX
top - View overall system performance. This tool displays CPU
usage, memory usage, and swap usage for the system and for
individual processes running on the system.
iostat - Monitor the loading operation for every disk attached to
the database server. Iostat displays the percentage of time that
the disk is physically active. If you use disk arrays, use utilities
provided with the disk arrays instead of iostat.
vmstat - Monitor disk swapping actions. Swapping should not
occur during the session.
sar - View detailed system activity reports of CPU, memory, and
disk usage. You can use this tool to monitor
CPU loading - It provides percent usage on user, system, idle
time, and waiting time. You can also use this tool to monitor disk
swapping actions.
Eliminating System Bottlenecks
- If the CPU usage is more than 80%, check the number of
concurrent running tasks. Consider changing the load or using a
grid to distribute tasks to different nodes. If you cannot reduce
the load, consider adding more processors.
- If swapping occurs, increase the physical memory or reduce the
number of memory-intensive applications on the disk.
- If you have excessive memory pressure (thrashing), consider
adding more physical memory.
- If the percent of time is high, tune the cache for PowerCenter to
use in-memory cache instead of writing to disk. If you tune the
cache, requests are still in queue, and the disk busy percentage is
at least 50%, add another disk device or upgrade to a faster disk
device. You can also use a separate disk for each partition in the
session.
- If physical disk queue length is greater than two, consider
adding another disk device or upgrading the disk device. You also
can use separate disks for the reader, writer, and transformation
threads.
- Consider improving network bandwidth.
 - When you tune UNIX systems, tune the server for a major
database system.
- If the percent time spent waiting on I/O (%wio) is high, consider
using other under-utilized disks. For example, if the source data,
target data, lookup, rank, and aggregate cache files are all on the
same disk, consider putting them on different disks.
Optimizing Flat File Targets
If you use a shared storage directory for flat file targets, you can
optimize session performance by ensuring that the shared
storage directory is on a machine that is dedicated to storing and
managing files, instead of performing other tasks.
If the Integration Service runs on a single node and the session
writes to a flat file target, you can optimize session performance
by writing to a flat file target that is local to the Integration
Service process node.
Dropping Indexes and Key Constraints
 When you define key constraints or indexes in target tables, you
slow the loading of data to those tables.
 To improve performance, drop indexes and key constraints before
you run the session.
 You can rebuild those indexes and key constraints after the
session completes.
 If you decide to drop and rebuild indexes and key constraints on a
regular basis, you can use the following methods to perform
these operations each time you run the session:
- Use pre-load and post-load stored procedures.
- Use pre-session and post-session SQL commands.
Note: To optimize performance, use constraint-based loading
only if necessary
Increasing Database Checkpoint Intervals
 The Integration Service performance slows each time it waits for
the database to perform a checkpoint.
 To decrease the number of checkpoints and increase
performance, increase the checkpoint interval in the database.
 Note: Although you gain performance when you reduce the
number of checkpoints, you also increase the recovery time if
the database shuts down unexpectedly.
Using Bulk Loads
 You can use bulk loading to improve the performance of a session
that inserts a large amount of data into a DB2, Sybase ASE,
Oracle, or Microsoft SQL Server database.
 Configure bulk loading in the session properties.
 When bulk loading, the Integration Service bypasses the
database log, which speeds performance.
 Without writing to the database log, however, the target
database cannot perform rollback.
 As a result, you may not be able to perform recovery. When you
use bulk loading, weigh the importance of improved session
131
performance against the ability to recover an incomplete
session.
 When bulk loading to Microsoft SQL Server or Oracle targets,
define a large commit interval to increase performance.
 Microsoft SQL Server and Oracle start a new bulk load transaction
after each commit.
 Increasing the commit interval reduces the number of bulk load
transactions, which increases performance.
Using External Loaders
To increase session performance, configure PowerCenter to use
an external loader for the following types of target databases:
- IBM DB2 EE or EEE
- Oracle
- Sybase IQ
- Teradata
Minimizing Deadlocks
 If the Integration Service encounters a deadlock when it tries to
write to a target, the deadlock only affects targets in the same
target connection group. The Integration Service still writes to
targets in other target connection groups.
 Encountering deadlocks can slow session performance.
 To improve session performance, you can increase the number of
target connection groups the Integration Service uses to write to
the targets in a session. To use a different target connection
group for each target in a session, use a different database
connection name for each target instance.
 You can specify the same connection information for each
connection name.
Increasing Database Network Packet Size
 If you write to Oracle, Sybase ASE, or Microsoft SQL Server
targets, you can improve the performance by increasing the
network packet size.
 Increase the network packet size to allow larger packets of data
to cross the network at one time.
 Increase the network packet size based on the database you
write to:
Oracle - You can increase the database server network packet size
in listener.ora and tnsnames.ora. Consult your database
documentation for additional information about increasing the
packet size, if necessary.
Sybase ASE and Microsoft SQL Server - Consult your database
documentation for information about how to increase the packet
size.
 For Sybase ASE or Microsoft SQL Server, you must also change
the packet size in the relational connection object in the
Workflow Manager to reflect the database server packet size.
Optimizing the Query
 If a session joins multiple source tables in one Source Qualifier,
you might be able to improve performance by optimizing the
 query with optimizing hints. Also, single table select statements
with an ORDER BY or GROUP BY clause may benefit from
optimization such as adding indexes.
 Usually, the database optimizer determines the most efficient
way to process the source data.
 However, you might know properties about the source tables
that the database optimizer does not.
 The database administrator can create optimizer hints to tell the
database how to execute the query for a particular set of source
tables.
 The query that the Integration Service uses to read data appears
in the session log/SQ Transformation
 Have the database administrator analyze the query, and then
create optimizer hints and indexes for the source tables.
 Use optimizing hints if there is a long delay between when the
query begins executing and when PowerCenter receives the first
row of data.
 Configure optimizer hints to begin returning rows as quickly as
possible, rather than returning all rows at once.
 This allows the Integration Service to process rows parallel with
the query execution.
 Once you optimize the query, use the SQL override option to take
full advantage of these modifications.
 You can also configure the source database to run parallel queries
to improve performance.
Using Conditional Filters
 A simple source filter on the source database can sometimes
negatively impact performance because of the lack of indexes.
 You can use the PowerCenter conditional filter in the Source
Qualifier to improve performance.
 Whether you should use the PowerCenter conditional filter to
improve performance depends on the session.
 For example, if multiple sessions read from the same source
simultaneously, the PowerCenter conditional filter may improve
performance.
 However, some sessions may perform faster if you filter the
source data on the source database.
 You can test the session with both the database filter and the
PowerCenter filter to determine which method improves
performance.
Increasing Database Network Packet Size
 If you read from Oracle, Sybase ASE, or Microsoft SQL Server
sources, you can improve the performance by increasing the
network packet size.
 Increase the network packet size to allow larger packets of data
to cross the network at one time.
 Increase the network packet size based on the database you read
from:
 Oracle - You can increase the database server network packet size
in listener.ora and tnsnames.ora. Consult your database
132
documentation for additional information about increasing the
packet size, if necessary.
 Sybase ASE and Microsoft SQL Server - Consult your database
documentation for information about how to increase the
packet size.
 For Sybase ASE or Microsoft SQL Server, you must also change
the packet size in the relational connection object in the
Workflow Manager to reflect the database server packet size.
Connecting to Oracle Database Sources
 If you are running the Integration Service on a single node and
the Oracle instance is local to the Integration Service process
node, you can optimize performance by using IPC protocol to
connect to the Oracle database. You can set up an Oracle
database connection in listener.ora and tnsnames.ora.
Using tempdb to Join Sybase or Microsoft SQL Server Tables
 When you join large tables on a Sybase or Microsoft SQL Server
database, it is possible to improve performance by creating the
tempdb as an in-memory database to allocate sufficient
memory.
Optimizing Mappings Overview
 Focus on mapping-level optimization after you optimize the
targets and sources.
 Generally, you reduce the number of transformations in the
mapping and delete unnecessary links between transformations
to optimize the mapping.
 Configure the mapping with the least number of transformations
and expressions to do the most amount of work possible.
 Delete unnecessary links between transformations to minimize
the amount of data moved.
Optimizing Flat File Sources
- Optimize the line sequential buffer length.
- Optimize delimited flat file sources.
- Optimize XML and flat file sources.
Optimizing the Line Sequential Buffer Length
 If the session reads from a flat file source, you can improve
session performance by setting the number of bytes the
Integration Service reads per line.
 By default, the Integration Service reads 1024 bytes per line.
 If each line in the source file is less than the default setting, you
can decrease the line sequential buffer length in the session
properties.
Optimizing Delimited Flat File Sources
 If a source is a delimited flat file, you must specify the delimiter
character to separate columns of data in the source file.
 You must also specify the escape character.
 The Integration Service reads the delimiter character as a regular
character if you include the escape character before the
delimiter character.
 You can improve session performance if the source flat file does
not contain quotes or escape characters.
Optimizing XML and Flat File Sources
 XML files are usually larger than flat files because of the tag
information.
 The size of an XML file depends on the level of tagging in the XML
file.
 More tags result in a larger file size.
 As a result, the Integration Service may take longer to read and
cache XML sources.
Configuring Single-Pass Reading
 Single-pass reading allows you to populate multiple targets with
one source qualifier.
 Consider using single-pass reading if you have multiple sessions
that use the same sources.
 You can combine the transformation logic for each mapping in
one mapping and use one source qualifier for each source.
 The Integration Service reads each source once and then sends
the data into separate pipelines.
 A particular row can be used by all the pipelines, by any
combination of pipelines, or by no pipelines.
 For example, you have the Purchasing source table, and you use
that source daily to perform an aggregation and a ranking.
 If you place the Aggregator and Rank transformations in separate
mappings and sessions, you force the Integration Service to read
the same source table twice.
 However, if you include the aggregation and ranking logic in one
mapping with one source qualifier, the Integration Service reads
the Purchasing source table once, and then sends the
appropriate data to the separate pipelines.
 When changing mappings to take advantage of single-pass
reading, you can optimize this feature by factoring out common
functions from mappings.
 For example, if you need to subtract a percentage from the Price
ports for both the Aggregator and Rank transformations, you
can minimize work by subtracting the percentage before
splitting the pipeline.
Optimizing Pass-Through Mappings
To pass directly from source to target without any other
transformations, connect the Source Qualifier transformation
directly to the target.
133
 If you use the Getting Started Wizard to create a pass-through
mapping, the wizard creates an Expression transformation
between the SQ transformation and the target
Optimizing Filters
 Use one of the following transformations to filter data:
 Source Qualifier transformation - The Source Qualifier
transformation filters rows from relational sources.
 Filter transformation - The Filter transformation filters data within
a mapping. The Filter transformation filters rows from any type
of source.
 If you filter rows from the mapping, you can improve efficiency by
filtering early in the data flow.
 Use a filter in the Source Qualifier transformation to remove the
rows at the source.
 The Source Qualifier transformation limits the row set extracted
from a relational source.
 If you cannot use a filter in the Source Qualifier transformation,
use a Filter transformation and move it as close to the SQ
transformation as possible to remove unnecessary data early in
the data flow.
 The Filter transformation limits the row set sent to a target.
 Avoid using complex expressions in filter conditions.
 To optimize Filter transformations, use simple integer or
true/false expressions in the filter condition.
Note: You can also use a Filter or Router transformation to drop
rejected rows from an Update Strategy transformation if you do
not need to keep rejected rows.
Optimizing Datatype Conversions
 You can increase performance by eliminating unnecessary
datatype conversions.
 For example, if a mapping moves data from an Integer column to
a Decimal column, then back to an Integer column, the
unnecessary datatype conversion slows performance.
 Where possible, eliminate unnecessary datatype conversions
from mappings.
Use the following datatype conversions to improve system
performance:
- Use integer values in place of other datatypes when performing
comparisons using Lookup and Filter transformations. For
example, many databases store U.S. ZIP code information as a
Char or Varchar datatype. If you convert the zip code data to an
Integer datatype, the lookup database stores the zip code
94303-1234 as 943031234. This helps increase the speed of the
lookup comparisons based on zip code.
- Convert the source dates to strings through port-to-port
conversions to increase session performance. You can either
 leave the ports in targets as strings or change the ports to
Date/Time ports.
Optimizing Expressions
Complete the following tasks to isolate the slow expressions:
1. Remove the expressions one-by-one from the mapping.
2. Run the mapping to determine the time it takes to run the
mapping without the transformation.
 If there is a significant difference in session run time, look for
ways to optimize the slow expression.
Factoring Out Common Logic
 If the mapping performs the same task in multiple places, reduce
the number of times the mapping performs the task by moving
the task earlier in the mapping.
 For example, you have a mapping with five target tables.
 Each target requires a Social Security number lookup.
 Instead of performing the lookup five times, place the Lookup
transformation in the mapping before the data flow splits.
 Next, pass the lookup results to all five targets.
Minimizing Aggregate Function Calls
 When writing expressions, factor out as many aggregate function
calls as possible.
 Each time you use an aggregate function call, the Integration
Service must search and group the data.
 For example, in the following expression, the Integration Service
reads COLUMN_A, finds the sum, then reads COLUMN_B, finds
the sum, and finally finds the sum of the two sums:
SUM(COLUMN_A) + SUM(COLUMN_B)
 If you factor out the aggregate function call, as below, the
Integration Service adds COLUMN_A to COLUMN_B, then finds
the sum of both.
SUM(COLUMN_A + COLUMN_B)
Replacing Common Expressions with Local Variables
 If you use the same expression multiple times in one
transformation, you can make that expression a local variable.
 You can use a local variable only within the transformation.
Choosing Numeric versus String Operations
 The Integration Service processes numeric operations faster than
string operations.
 For example, if you look up large amounts of data on two
columns, EMPLOYEE_NAME and EMPLOYEE_ID, configuring the
lookup around EMPLOYEE_ID improves performance.
Optimizing Char-Char and Char-Varchar Comparisons
 When the Integration Service performs comparisons between
CHAR and VARCHAR columns, it slows each time it finds trailing
blank spaces in the row.
134
 You can use the TreatCHARasCHARonRead option when you
configure the Integration Service in the Informatica
Administrator so that the Integration Service does not trim
trailing spaces from the end of Char source fields
Choosing DECODE Versus LOOKUP
 When you use a LOOKUP function, the Integration Service must
look up a table in a database.
 When you use a DECODE function, you incorporate the lookup
values into the expression so the Integration Service does not
have to look up a separate table. Therefore, when you want to
look up a small set of unchanging values, use DECODE to
improve performance.
Using Operators Instead of Functions
 The Integration Service reads expressions written with operators
faster than expressions with functions.
 Where possible, use operators to write expressions.
 For example, you have the following expression that contains
nested CONCAT functions:
CONCAT (CONCAT (CUSTOMERS.FIRST_NAME, ‘’)
CUSTOMERS.LAST_NAME)
 You can rewrite that expression with the || operator as follows:
CUSTOMERS.FIRST_NAME || ‘ ’ || CUSTOMERS.LAST_NAME
Optimizing IIF Functions
 IIF functions can return a value and an action, which allows for
more compact expressions.
 For example, you have a source with three Y/N flags: FLG_A,
FLG_B, FLG_C. You want to return values based on the values of
each flag.
 You use the following expression:
IIF( FLG_A = 'Y' and FLG_B = 'Y' AND FLG_C = 'Y',
VAL_A + VAL_B + VAL_C,
IIF( FLG_A = 'Y' and FLG_B = 'Y' AND FLG_C = 'N',
VAL_A + VAL_B ,
IIF( FLG_A = 'Y' and FLG_B = 'N' AND FLG_C = 'Y',
VAL_A + VAL_C,
IIF( FLG_A = 'Y' and FLG_B = 'N' AND FLG_C = 'N',
VAL_A ,
IIF( FLG_A = 'N' and FLG_B = 'Y' AND FLG_C = 'Y',
VAL_B + VAL_C,
IIF( FLG_A = 'N' and FLG_B = 'Y' AND FLG_C = 'N',
VAL_B ,
IIF( FLG_A = 'N' and FLG_B = 'N' AND FLG_C = 'Y',
VAL_C,
IIF( FLG_A = 'N' and FLG_B = 'N' AND FLG_C = 'N',
0.0,))))))))
 This expression requires 8 IIFs, 16 ANDs, and at least 24
comparisons.
 If you take advantage of the IIF function, you can rewrite that
expression as:
 IIF(FLG_A='Y', VAL_A, 0.0)+ IIF(FLG_B='Y', VAL_B, 0.0)+
IIF(FLG_C='Y', VAL_C, 0.0)
 This results in three IIFs, two comparisons, two additions, and a
faster session.
Evaluating Expressions
If you are not sure which expressions slow performance, evaluate
the expression performance to isolate the problem.
1. Time the session with the original expressions.
2. Copy the mapping and replace half of the complex expressions
with a constant.
3. Run and time the edited session.
4. Make another copy of the mapping and replace the other half
of the complex expressions with a constant.
5. Run and time the edited session
Optimizing External Procedures
 For example, you need to create an external procedure with two
input groups.
 The external procedure reads a row from the first input group
and then reads a row from the second input group.
 If you use blocking, you can write the external procedure code to
block the flow of data from one input group while it processes
the data from the other input group.
 When you write the external procedure code to block data, you
increase performance because the procedure does not need to
copy the source data to a buffer. However, you could write the
external procedure to allocate a buffer and copy the data from
one input group to the buffer until it is ready to process the
data.
 Copying source data to a buffer decreases performance.
Optimizing Aggregator Transformations
 Aggregator transformations often slow performance because
they must group data before processing it.
 Aggregator transformations need additional memory to hold
intermediate group results.
 Use the following guidelines to optimize the performance of an
Aggregator transformation:
- Group by simple columns.
1. When possible, use numbers instead of string and dates in the
columns used for the GROUP BY.
2. Avoid complex expressions in the Aggregator expressions.
- Use sorted input.
1. The Sorted Input option decreases the use of aggregate caches
2. You can increase performance when you use the Sorted Input
option in sessions with multiple partitions
- Use incremental aggregation.
1. If you can capture changes from the source that affect less
than half the target, you can use incremental aggregation to
optimize the performance of Aggregator transformations.
2. When you use incremental aggregation, you apply captured
changes in the source to aggregate calculations in a session.
135
3. The Integration Service updates the target incrementally,
rather than processing the entire source and recalculating the
same calculations every time you run the session.
4. You can increase the index and data cache sizes to hold all data
in memory without paging to disk
- Filter data before you aggregate it.
- Limit port connections
Optimizing Custom Transformations
 The Integration Service can pass a single row to a Custom
transformation procedure or a block of rows in an array.
 You can write the procedure code to specify whether the
procedure receives one row or a block of rows.
 You can increase performance when the procedure receives a
block of rows:
- You can decrease the number of function calls the Integration
Service and procedure make. The Integration Service calls the
input row notification function fewer times, and the procedure
calls the output notification function fewer times.
- You can increase the locality of memory access space for the
data.
- You can write the procedure code to perform an algorithm on a
block of data instead of each row of data.
Optimizing Joiner Transformations
Use the following tips to improve session performance with the
Joiner transformation:
Designate the master source as the source with fewer duplicate
key values - When the Integration Service processes a sorted
Joiner transformation, it caches rows for one hundred unique
keys at a time. If the master source contains many rows with the
same key value, the Integration Service must cache more rows,
and performance can be slowed.
Designate the master source as the source with fewer rows -
During a session, the Joiner transformation compares each row of
the detail source against the master source. The fewer rows in
the master, the fewer iterations of the join comparison occur,
which speeds the join process.
Perform joins in a database when possible - Performing a join in a
database is faster than performing a join in the session. The type
of database join you use can affect performance. Normal joins
are faster than outer joins and result in fewer rows. In some
cases, you cannot perform the join in the database, such as
joining tables from two different databases or flat file systems.
To perform a join in a database, use the following options:
- Create a pre-session stored procedure to join the tables in a
database.
- Use the Source Qualifier transformation to perform the join.
Join sorted data when possible - To improve session
performance, configure the Joiner transformation to use sorted
input. When you configure the Joiner transformation to use
sorted data, the Integration Service improves performance by
minimizing disk input and output. You see the greatest
performance improvement when you work with large data sets.
For an unsorted Joiner transformation, designate the source with
fewer rows as the master source.
Optimizing Lookup Transformations
- Use the optimal database driver.
- Cache lookup tables.
 Use the appropriate cache type.
 Enable concurrent caches.
 Optimize Lookup condition matching.
- When the Lookup transformation matches lookup cache
data with the lookup condition, it sorts and orders the
data to determine the first matching value and the last
matching value.
- You can configure the transformation to return any value
that matches the lookup condition.
- When you configure the Lookup transformation to
return any matching value, the transformation returns
the first value that matches the lookup condition.
- It does not index all ports as it does when you configure
the transformation to return the first matching value or
the last matching value.
- When you use any matching value, performance can
improve because the transformation does not index on
all ports, which can slow performance.
 Reduce the number of cached rows.
- You can reduce the number of rows included in the
cache to increase performance. Use the Lookup SQL
Override option to add a WHERE clause to the default
SQL statement
 Override the ORDER BY statement.
- By default, the Integration Service generates an ORDER
BY statement for a cached lookup.
- The ORDER BY statement contains all lookup ports.
- To increase performance, suppress the default ORDER
BY statement and enter an override ORDER BY with
fewer columns.
- The Integration Service always generates an ORDER BY
statement, even if you enter one in the override.
- Place two dashes ‘--’ after the ORDER BY override to
suppress the generated ORDER BY statement.
- For example, a Lookup transformation uses the following
lookup condition:
ITEM_ID = IN_ITEM_ID
PRICE <= IN_PRICE
- The Lookup transformation includes three lookup ports
used in the mapping, ITEM_ID, ITEM_NAME, and PRICE.
136
- When you enter the ORDER BY statement, enter the
columns in the same order as the ports in the lookup
condition.
- You must also enclose all database reserved words in
quotes.
- Enter the following lookup query in the lookup SQL
override:
SELECT ITEMS_DIM.ITEM_NAME, ITEMS_DIM.PRICE,
ITEMS_DIM.ITEM_ID FROM ITEMS_DIM ORDER BY
ITEMS_DIM.ITEM_ID, ITEMS_DIM.PRICE --
 Use a machine with more memory.
- Optimize the lookup condition.
 If you include more than one lookup condition, place the
conditions in the following order to optimize lookup
performance:
Equal to (=)
Less than (<), greater than (>), less than or equal to (<=),
greater than or equal to (>=)
Not equal to (! =)
- Filter lookup rows.
- Index the lookup table.
 The Integration Service needs to query, sort, and
compare values in the lookup condition columns.
 The index needs to include every column used in a
lookup condition
- Optimize multiple lookups.
 If a mapping contains multiple lookups, even with
caching enabled and enough heap memory, the lookups
can slow performance.
 Tune the Lookup transformations that query the largest
amounts of data to improve overall performance.
 To determine which Lookup transformations process the
most data, examine the Lookup_rowsinlookupcache
counters for each Lookup transformation.
 The Lookup transformations that have a large number in
this counter might benefit from tuning their lookup
expressions.
 If those expressions can be optimized, session
performance improves
- Create a pipeline Lookup transformation and configure
partitions in the pipeline that builds the lookup source.
Optimizing Sequence Generator Transformations
 To optimize Sequence Generator transformations, create a
reusable Sequence Generator and using it in multiple mappings
simultaneously.
 Also, configure the Number of Cached Values property.
 The Number of Cached Values property determines the number
of values the Integration Service caches at one time.
 Make sure that the Number of Cached Value is not too small.
 Consider configuring the Number of Cached Values to a value
greater than 1,000.
 If you do not have to cache values, set the Number of Cache
Values to 0.
 Sequence Generator transformations that do not use cache are
faster than those that require cache.
Optimizing Sorter Transformations
- Allocate enough memory to sort the data.
- Specify a different work directory for each partition in the Sorter
transformation.
Optimizing SQL Transformations
 Each time the Integration Service processes a new query in a
session, it calls a function called SQLPrepare to create an SQL
procedure and pass it to the database.
 When the query changes for each input row, it has a performance
impact.
 When an SQL query contains commit and rollback query
statements, the Integration Service must recreate the SQL
procedure after each commit or rollback.
 To optimize performance, do not use transaction statements in
an SQL transformation query.
 When you configure the transformation to use a static
connection, you choose a connection from the Workflow
Manager connections.
 The SQL transformation connects to the database once during
the session.
 When you pass dynamic connection information, the SQL
transformation connects to the database each time the
transformation processes an input row.
Eliminating Transformation Errors
 In large numbers, transformation errors slow the performance of
the Integration Service.
 With each transformation error, the Integration Service pauses to
determine the cause of the error and to remove the row causing
the error from the data flow.
 Next, the Integration Service typically writes the row into the
session log file.
 Transformation errors occur when the Integration Service
encounters conversion errors, conflicting mapping logic, and any
condition set up as an error, such as null input.
 Check the session log to see where the transformation errors
occur.
 If the errors center on particular transformations, evaluate those
transformation constraints.
Optimizing Sessions
137
Grid -
 A grid is an alias assigned to a group of nodes that allows you to
automate the distribution of workflows and sessions across
nodes.
 When you use a grid, the Integration Service distributes workflow
tasks and session threads across multiple nodes.
 A Load Balancer distributes tasks to nodes without overloading
any node. Running workflows and sessions on the nodes of a
grid provides the following performance gains:
- Balances the Integration Service workload.
- Processes concurrent sessions faster.
- Processes partitions faster.
 The Integration Service requires CPU resources for parsing input
data and formatting the output data.
 A grid can improve performance when you have a performance
bottleneck in the extract and load steps of a session.
 A grid can improve performance when memory or temporary
storage is a performance bottleneck.
 When a PowerCenter mapping contains a transformation that has
cache memory, deploying adequate memory and separate disk
storage for each cache instance improves performance.
 Running a session on a grid can improve throughput because the
grid provides more resources to run the session.
 Performance improves when you run a few sessions on the grid
at a time.
 Running a session on a grid is more efficient than running a
workflow over a grid if the number of concurrent session
partitions is less than the number of nodes.
 When you run multiple sessions on a grid, session subtasks share
node resources with subtasks of other concurrent sessions.
 Running a session on a grid requires coordination between
processes running on different nodes.
 For some mappings, running a session on a grid requires
additional overhead to move data from one node to another
node.
 In addition to loading the memory and CPU resources on each
node, running multiple sessions on a grid adds to network
traffic.
 When you run a workflow on a grid, the Integration Service loads
memory and CPU resources on nodes without requiring
coordination between the nodes
Pushdown Optimization -
 To increase session performance, push transformation logic to
the source or target database.
Concurrent Sessions and Workflows -
 If possible, run sessions and workflows concurrently to improve
performance.
 For example, if you load data into an analytic schema, where you
have dimension and fact tables, load the dimensions
concurrently
Buffer Memory
 When the Integration Service initializes a session, it allocates
blocks of memory to hold source and target data.
 The Integration Service allocates at least two blocks for each
source and target partition.
 Sessions that use a large number of sources and targets might
require additional memory blocks.
 If the Integration Service cannot allocate enough memory blocks
to hold the data, it fails the session.
 You can configure the amount of buffer memory, or you can
configure the Integration Service to calculate buffer settings at
run time.
 To increase the number of available memory blocks, adjust the
following session properties:
DTM Buffer Size - Increase the DTM buffer size on the Properties
tab in the session properties.
Default Buffer Block Size - Decrease the buffer block size on the
Config Object tab in the session properties.
 Before you configure these settings, determine the number of
memory blocks the Integration Service requires to initialize the
session.
 Then, based on default settings, calculate the buffer size and the
buffer block size to create the required number of session
blocks.
 If you have XML sources or targets in a mapping, use the number
of groups in the XML source or target in the calculation for the
total number of sources and targets.
 For example, you create a session that contains a single partition
using a mapping that contains 50 sources and 50 targets.
Then, you make the following calculations:
1. You determine that the session requires a minimum of 200
memory blocks:
[(total number of sources + total number of targets)* 2] =
(session buffer blocks)
100 * 2 = 200
2. Based on default settings, you determine that you can change
the DTM Buffer Size to 15,000,000, or you can change the Default
Buffer Block Size to 54,000:
(Session Buffer Blocks) = (.9) * (DTM Buffer Size) / (Default Buffer
Block Size) * (number of partitions)
200 = .9 * 14222222 / 64000 * 1
Or
200 = .9 * 12000000 / 54000 * 1
138
Note: For a session that contains n partitions, set the DTM Buffer
Size to at least n times the value for the session with one
partition.
 The Log Manager writes a warning message in the session log if
the number of memory blocks is so small that it causes
performance degradation.
 The Log Manager writes this warning message even if the
number of memory blocks is enough for the session to run
successfully.
 The warning message also gives a suggestion for the proper
value.
 If you modify the DTM Buffer Size, increase the property by
multiples of the buffer block size.
Increasing DTM Buffer Size
 The DTM Buffer Size setting specifies the amount of memory the
Integration Service uses as DTM buffer memory.
 The Integration Service uses DTM buffer memory to create the
internal data structures and buffer blocks used to bring data into
and out of the Integration Service.
 When you increase the DTM buffer memory, the Integration
Service creates more buffer blocks, which improves
performance during momentary slowdowns.
 Increasing DTM buffer memory allocation generally causes
performance to improve initially and then level off.
 When you increase the DTM buffer memory allocation, consider
the total memory available on the Integration Service process
system.
 If you do not see a significant increase in performance, DTM
buffer memory allocation is not a factor in session performance.
 Note: Reducing the DTM buffer allocation can cause the session
to fail early in the process because the Integration Service is
unable to allocate memory to the required processes.
 To increase the DTM buffer size, open the session properties and
click the Properties tab.
 Edit the DTM Buffer Size property in the Performance settings.
 Increase the property by multiples of the buffer block size, and
then run and time the session after each increase
Optimizing the Buffer Block Size
 Depending on the session source data, you might need to
increase or decrease the buffer block size.
 If the machine has limited physical memory and the mapping in
the session contains a large number of sources, targets, or
partitions, you might need to decrease the buffer block size.
 If you are manipulating unusually large rows of data, increase the
buffer block size to improve performance.
 If you do not know the approximate size of the rows, determine
the configured row size by completing the following steps.
 To evaluate needed buffer block size:
 1. In the Mapping Designer, open the mapping for the session.
2. Open the target instance
3. Click the Ports tab.
4. Add the precision for all columns in the target.
5. If you have more than one target in the mapping, repeat steps
2 to 4 for each additional target to calculate the precision for
each target.
6. Repeat steps 2 to 5 for each source definition in the mapping.
7. Choose the largest precision of all the source and target
precisions for the total precision in the buffer block size
calculation.
The total precision represents the total bytes needed to move the
largest row of data.
 For example, if the total precision equals 33,000, then the
Integration Service requires 33,000 bytes in the buffers to move
that row.
 If the buffer block size is 64,000 bytes, the Integration Service can
move only one row at a time.
 Ideally, a buffer accommodates at least 100 rows at a time.
 So if the total precision is greater than 32,000, increase the size
of the buffers to improve performance.
 To increase the buffer block size, open the session properties and
click the Config Object tab.
 Edit the Default Buffer Block Size property in the Advanced
settings.
 Increase the DTM buffer block setting in relation to the size of the
rows.
 As with DTM buffer memory allocation, increasing buffer block
size should improve performance.
 If you do not see an increase, buffer block size is not a factor in
session performance.
Caches
 The Integration Service uses the index and data caches for XML
targets and Aggregator, Rank, Lookup, and Joiner
transformations.
 The Integration Service stores transformed data in the data cache
before returning it to the pipeline.
 It stores group information in the index cache.
 Also, the Integration Service uses a cache to store data for Sorter
transformations.
 To configure the amount of cache memory, use the cache
calculator or specify the cache size.
 You can also configure the Integration Service to calculate cache
memory settings at run time.
 If the allocated cache is not large enough to store the data, the
Integration Service stores the data in a temporary disk file, a
cache file, as it processes the session data.
 Performance slows each time the Integration Service pages to a
temporary file.
139
 Examine the performance counters to determine how often the
Integration Service pages to a file.
Perform the following tasks to optimize caches:
- Limit the number of connected input/output and output only
ports.
- Select the optimal cache directory location.
 If you run the Integration Service on a grid and only some
Integration Service nodes have fast access to the shared cache
file directory, configure each session with a large cache to run
on the nodes with fast access to the directory.
 If all Integration Service processes in a grid have slow access to
the cache files, set up a separate, local cache file directory for
each Integration Service process.
 An Integration Service process may have faster access to the
cache files if it runs on the same machine that contains the
cache directory.
 Note: You may encounter performance degradation when you
cache large quantities of data on a mapped or mounted drive.
Increase the cache sizes
 You configure the cache size to specify the amount of memory
allocated to process a transformation.
 The amount of memory you configure depends on how much
memory cache and disk cache you want to use.
 If you configure the cache size and it is not enough to process the
transformation in memory, the Integration Service processes
some of the transformation in memory and pages information
to cache files to process the rest of the transformation.
 Each time the Integration Service pages to a cache file,
performance slows.
 You can examine the performance details of a session to
determine when the Integration Service pages to a cache file.
 The Transformation_readfromdisk or Transformation_writetodisk
counters for any Aggregator, Rank, or Joiner transformation
indicate the number of times the Integration Service pages to
disk to process the transformation.
Use the 64-bit version of PowerCenter to run large cache sessions
 If you process large volumes of data or perform memory-
intensive transformations, you can use the 64-bit PowerCenter
version to increase session performance.
 The 64-bit version provides a larger memory space that can
significantly reduce or eliminate disk input/output.
 This can improve session performance in the following areas:
Caching - With a 64-bit platform, the Integration Service is not
limited to the 2 GB cache limit of a 32-bit platform.
Data throughput - With a larger available memory space, the
reader, writer, and DTM threads can process larger blocks of
data
 Target-Based Commit
 Each time the Integration Service commits, performance slows.
 Therefore, the smaller the commit interval, the more often the
Integration Service writes to the target database, and the slower
the overall performance
 If you increase the commit interval, the number of times the
Integration Service commits decreases and performance
improves.
 When you increase the commit interval, consider the log file
limits in the target database.
 If the commit interval is too high, the Integration Service may fill
the database log file and cause the session to fail.
 Therefore, weigh the benefit of increasing the commit interval
against the additional time you would spend recovering a failed
session.
 Click the General Options settings in the session properties to
review and adjust the commit interval.
Log Files
 A workflow runs faster when you do not configure it to write
session and workflow log files.
 Workflows and sessions always create binary logs.
 When you configure a session or workflow to write a log file, the
Integration Service writes logging events twice.
 You can access the binary logs session and workflow logs in the
Administrator tool
Error Tracing
 If a session contains a large number of transformation errors, and
you do not need to correct them, set the session tracing level to
Terse.
 At this tracing level, the Integration Service does not write error
messages or row-level information for reject data.
 If you need to debug the mapping and you set the tracing level to
Verbose, you may experience significant performance
degradation when you run the session. Do not use Verbose
tracing when you tune performance.
 The session tracing level overrides any transformation-specific
tracing levels within the mapping.
 This is not recommended as a long-term response to high levels
of transformation errors.
Post-Session Emails
 When you attach the session log to a post-session email, enable
flat file logging.
 If you enable flat file logging, the Integration Service gets the
session log file from disk.
 If you do not enable flat file logging, the Integration Service
gets the log events from the Log Manager and generates the
session log file to attach to the email.
140
 When the Integration Service retrieves the session log from the
log service, workflow performance slows, especially when the
session log file is large and the log service runs on a different
node than the master DTM.
 For optimal performance, configure the session to write to log file
when you configure post-session email to attach a session log.
Optimizing Grid Deployments Overview
 When you run PowerCenter on a grid, you can configure the grid,
sessions, and workflows to use resources efficiently and
maximize scalability.
 To improve PowerCenter performance on a grid, complete the
following tasks:
- Add nodes to the grid.
- Increase storage capacity and bandwidth.
- Use shared file systems.
- Use a high-throughput network when you complete the
following tasks:
1. Access sources and targets over the network.
2. Transfer data between nodes of a grid when using the
Session on Grid option.
Storing Files
 When you configure PowerCenter to run on a grid, you specify
the storage location for different types of session files, such as
source files, log files, and cache files.
 To improve performance, store files in optimal locations.
 For example, store persistent cache files on a high-bandwidth
shared file system.
 Different types of files have different storage requirements.
 You can store files in the following types of locations:
Shared file systems - Store files on a shared file system to enable
all Integration Service processes to access the same files. You can
store files on low-bandwidth and high-bandwidth shared file
systems.
Local - Store files on the local machine running the Integration
Service process when the files do not have to be accessed by
other Integration Service processes.
High Bandwidth Shared File System Files
 Because they can be accessed often during a session, place the
following files on a high-bandwidth shared file system:
- Source files, including flat files for lookups.
- Target files, including merge files for partitioned sessions.
- Persistent cache files for lookup or incremental aggregation.
- Non-persistent cache files for only grid-enabled sessions on a
grid.
 This allows the Integration Service to build the cache only once.
If these cache files are stored on a local file system, the
Integration Service builds a cache for each partition group.
Low Bandwidth Shared File System Files
 Because they are accessed less frequently during a session, store
the following files on a low-bandwidth shared file system:
- Parameter files or other configuration related files.
- Indirect source or target files.
- Log files.
Local Storage Files
 To avoid unnecessary file sharing when you use shared file
systems, store the following files locally:
- Non-persistent cache files for sessions that are not enabled for a
grid, including Sorter transformation temporary files.
- Individual target files for different partitions when performing a
sequential merge for partitioned sessions.
- Other temporary files that are deleted at the end of a session
run. In general, to establish this, configure $PmTempFileDir for a
local file system.
 Avoid storing these files on a shared file system, even when the
bandwidth is high
OPTIMIZING THE POWERCENTER COMPONENTS
 You can optimize performance of the following PowerCenter
components:
- PowerCenter repository
- Integration Service
 If you run PowerCenter on multiple machines, run the Repository
Service and Integration Service on different machines.
 To load large amounts of data, run the Integration Service on the
higher processing machine.
 Also, run the Repository Service on the machine hosting the
PowerCenter repository
Optimizing PowerCenter Repository Performance
- Ensure the PowerCenter repository is on the same machine as
the Repository Service process.
- Order conditions in object queries.
- Use a single-node tablespace for the PowerCenter repository if
you install it on a DB2 database.
- Optimize the database schema for the PowerCenter repository if
you install it on a DB2 or Microsoft SQL Server database.
Optimizing Integration Service Performance
- Use native drivers instead of ODBC drivers for the Integration
Service.
- Run the Integration Service in ASCII data movement mode if
character data is 7-bit ASCII or EBCDIC.
- Cache PowerCenter metadata for the Repository Service.
- Run Integration Service with high availability.
 Note: When you configure the Integration Service with high
availability, the Integration Service recovers workflows and
sessions that may fail because of temporary network or machine
failures.
141
 To recover from a workflow or session, the Integration Service
writes the states of each workflow and session to temporary
files in a shared directory. This may decrease performance
OPTIMIZING THE SYSTEM OVERVIEW
 Often performance slows because the session relies on inefficient
connections or an overloaded Integration Service process
system.
 System delays can also be caused by routers, switches, network
protocols, and usage by many users.
 Slow disk access on source and target databases, source and
target file systems, and nodes in the domain can slow session
performance.
 Have the system administrator evaluate the hard disks on the
machines.
 After you determine from the system monitoring tools that you
have a system bottleneck, make the following global changes to
improve the performance of all sessions:
Improve network speed - Slow network connections can slow
session performance. Have the system administrator determine if
the network runs at an optimal speed. Decrease the number of
network hops between the Integration Service process and
databases.
Use multiple CPUs - You can use multiple CPUs to run multiple
sessions in parallel and run multiple pipeline partitions in parallel.
Reduce paging - When an operating system runs out of physical
memory, it starts paging to disk to free physical memory.
Configure the physical memory for the Integration Service
process machine to minimize paging to disk.
Use processor binding - In a multi-processor UNIX environment,
the Integration Service may use a large amount of system
resources. Use processor binding to control processor usage by
the Integration Service process. Also, if the source and target
database are on the same machine, use processor binding to limit
the resources used by the database.
USING PIPELINE PARTITIONS
 If you have the partitioning option, perform the following tasks to
manually set up partitions:
Increase the number of partitions -
Use the following tips when you add partitions to a session:
 Add one partition at a time - To best monitor performance,
add one partition at a time, and note the session settings
before you add each partition.
 Set DTM Buffer Memory - When you increase the number of
partitions, increase the DTM buffer size. If the session
contains n partitions, increase the DTM buffer size to at least
n times the value for the session with one partition.
 Set cached values for Sequence Generator - If a session has n
partitions, you should not need to use the “Number of
Cached Values” property for the Sequence Generator
transformation. If you set this value to a value greater than 0,
 make sure it is at least n times the original value for the
session with one partition.
 Partition the source data evenly - Configure each partition to
extract the same number of rows.
 Monitor the system while running the session - If CPU cycles
are available, you can add a partition to improve
performance. For example, you may have CPU cycles
available if the system has 20 percent idle time.
 Monitor the system after adding a partition - If the CPU
utilization does not go up, the wait for I/O time goes up, or
the total data transformation rate goes down, then there is
probably a hardware or software bottleneck. If the wait for
I/O time goes up by a significant amount, then check the
system for hardware bottlenecks. Otherwise, check the
database configuration.
Select the best performing partition types at particular points in a
pipeline
- You can use multiple pipeline partitions and database partitions.
- To improve performance, ensure the number of pipeline
partitions equals the number of database partitions
- To increase performance, specify partition types at the following
partition points in the pipeline:
Source Qualifier transformation - To read data from multiple flat
files concurrently, specify one partition for each flat file in the
Source Qualifier transformation. Accept the default partition
type, pass-through.
Filter transformation - Since the source files vary in size, each
partition processes a different amount of data. Set a partition
point at the Filter transformation, and choose round-robin
partitioning to balance the load going into the Filter
transformation.
Sorter transformation - To eliminate overlapping groups in the
Sorter and Aggregator transformations, use hash auto-keys
partitioning at the Sorter transformation. This causes the
Integration Service to group all items with the same description
into the same partition before the Sorter and Aggregator
transformations process the rows. You can delete the default
partition point at the Aggregator transformation.
Target - Since the target tables are partitioned by key range,
specify key range partitioning at the target to optimize writing
data to the target
Use multiple CPUs.
Optimizing the Source Database for Partitioning
 You can add partitions to increase the speed of the query.
 Usually, each partition on the reader side represents a subset of
the data to be processed.
 Complete the following tasks to optimize the source database for
partitioning,
Tune the database - If the database is not tuned properly,
creating partitions may not make sessions quicker.
142
Enable parallel queries - Some databases may have options that
must be set to enable parallel queries. Check the database
documentation for these options. If these options are off, the
Integration Service runs multiple partition SELECT statements
serially.
Separate data into different tables’ spaces - Each database
provides an option to separate the data into different
tablespaces. If the database allows it, use the PowerCenter SQL
override feature to provide a query that extracts data from a
single partition.
Group the sorted data - You can partition and group source data
to increase performance for a sorted Joiner transformation.
Maximize single-sorted queries
Optimizing the Target Database for Partitioning
 If a session contains multiple partitions, the throughput for each
partition should be the same as the throughput for a single
partition session.
 If you do not see this correlation, then the database is probably
inserting rows into the database serially.
 To ensure that the database inserts rows in parallel, check the
following configuration options in the target database:
- Set options in the database to enable parallel inserts. For
example, set the db_writer_processes and DB2 has max_agents
options in an Oracle database to enable parallel inserts. Some
databases may enable these options by default.
- Consider partitioning the target table. If possible, try to have
each partition write to a single database partition using a Router
transformation to do this. Also, have the database partitions on
separate disks to prevent I/O contention among the pipeline
partitions.
- Set options in the database to enhance database scalability. For
example, disable archive logging and timed statistics in an Oracle
database to enhance scalability
PERFORMANCE COUNTERS OVERVIEW
 All transformations have counters. The Integration Service tracks
the number of input rows, output rows, and error rows for each
transformation.
 Some transformations have performance counters. You can use
the following performance counters to increase session
performance:
- Errorrows
- Readfromcache and Writetocache
- Readfromdisk and Writetodisk
- Rowsinlookupcache
Errorrows Counter
Transformation errors impact session performance. If a
transformation has large numbers of error rows in any of the
Transformation_errorrows counters, you can eliminate the errors
to improve performance

Readfromcache and Writetocache Counters
 If a session contains Aggregator, Rank, or Joiner transformations,
examine the Transformation_readfromcache and
Transformation_writetocache counters along with the
Transformation_readfromdisk and Transformation_writetodisk
counters to analyze how the Integration Service reads from or
writes to disk.
 To analyze the disk access, first calculate the hit or miss ratio.
 The hit ratio indicates the number of read or write operations the
Integration Service performs on the cache.
 The miss ratio indicates the number of read or write operations
the Integration Service performs on the disk.
 Use the following formula to calculate the cache miss ratio:
[(# of reads from disk) + (# of writes to disk)]/[(# of reads from
memory cache) + (# of writes to memory cache)]
Use the following formula to calculate the cache hit ratio:
[1 - Cache Miss ratio]
To minimize reads and writes to disk, increase the cache size. The
optimal cache hit ratio is 1.
Readfromdisk and Writetodisk Counters
 If a session contains Aggregator, Rank, or Joiner transformations,
examine each Transformation_readfromdisk and
Transformation_writetodisk counter.
- Allocate 50 MB to the index cache and 250 MB to the data
cache. The Integration Service accesses 50 percent of the data
from the index cache and 50 percent of the data from the data
cache. The Integration Service does not access both index and
data caches a 100 times each. Therefore, the percentage of data
that gets accessed is 50 percent.
 If the session performs incremental aggregation, the Integration
Service reads historical aggregate data from the local disk during
the session and writes to disk when saving historical data.
 As a result, the Aggregator_readtodisk and
Aggregator_writetodisk counters display numbers besides zero.
 However, since the Integration Service writes the historical data
to a file at the end of the session, you can still evaluate the
counters during the session.
 If the counters show numbers other than zero during the session
run, you can tune the cache sizes to increase performance.
 However, there is a cost associated with allocating or deallocating
memory, so refrain from increasing the cache sizes to
accommodate more data volume if you know what volume of
data the Integration Service will process.
Rowsinlookupcache Counter
 Multiple lookups can decrease session performance. To improve
session performance, tune the lookup expressions for the larger
lookup tables.

 If these counters display any number other than zero, you can
increase the cache sizes to improve session performance.
 The Integration Service uses the index cache to store group
information and the data cache to store transformed data,
which is typically larger.
 Therefore, although both the index cache and data cache sizes
affect performance, you may need to increase the data cache
size more than the index cache size.
 However, if the volume of data processed is greater than the
memory available you can increase the index cache size to
improve performance.

 For example, the Integration Service uses 100 MB to store the

index cache and 500 MB to store the data cache.
 With 200 randomly distributed accesses on each of the index and
data caches, you can configure the cache in the following ways:
- To optimize performance, allocate 100 MB to the index cache
and 200 MB to the data cache. The Integration Service accesses
100 percent of the data from the index cache and 40 percent of
the data from the data cache. The Integration Service always
accesses the index cache, and does not access the data cache 120
times. Therefore, the percentage of data that gets accessed is 70
percent.

143
144