IDQ components used

Data Profiling : To discover and understand the data pattern, anomalies and detect
data duplicates upfront

Score Cards: To measure the quality of data continuously and achieve required level
of acceptance by business

Enterprise Profiling and Data Domain : To detect Chinese (Non-Latin) content

IDQ Rules : To cleanse and standardize the data

Data Objects : To read the source data easily and write to target applications

CDO/LDO/Mapping Specifications : To transform the data from source to target

applying the necessary business rules/logic

Data Preview : To view and validate the input and the transformed output for every
transformation/ business rule applied during source to target mapping

Data Lineage : To facilitate the attribute traceability and data flow from source
to target

Dynamic Mappings : To load the data from multiple source tables/files and write it
into target tables/files having similar data structure via one single mapping

Mid Stream Profiling : To view and extract the data at every transformation with
easy drill down on the results

Business Benefits Obtained

Increased Efficiency: Enabled the Business Analysts and users to quickly access
the source and transformed data and thereby manage their day to day data quality
tasks effectively

Increased Collaboration between IT, Business and other stake holders

Powerful Profiling methods helped both Business Analysts, IT Analysts and

Developers to understand the data , including large volumes

Data Persistence Profile results stored in Informatica Profile Warehouse enabled

the users to access the data anytime easily

Online Mapping Specifications eliminated the need to maintain the traditional excel
based mapping sheets

Easy Data Export and Import in and out of the tool

Data Preview & Mid Stream Profiling provided the opportunity to drill down the
transformed data at every step, thereby eased the effort of business rule
validation manually.

Reusability: The Dynamic Mappings helped to avoid creation of multiple mappings

during source staging. Also the option of reusable Business Rules, Data Domain
groups facilitated the business analysts as the same business logic need not to be
developed multiple times and could be re-used across different SHRs.
-----------------------------------------------------------------------------------
----------------------------------------------------------

What is Pushdown Optimization?

Pushdown optimization is a way of load-balancing among servers in order to achieve

optimal performance.
Load balancing is a mechanism which distributes the workloads across the nodes in
the gird

Pushdown Optimization

One can push the transformation logic to the source/ target database using pushdown
optimization. when one use pushdown optimization, the integration service
transalates the transformation logic into SQL queries to the database. The source
or target database executes the SQL queries to process the transformations. The
amount of transformation logic one can push to the database depends on the
database, transformation logic, mapping and session configuration. The Integration
Service analyzes the transformation logic it can push to the database and executes
the SQL statement generated against the source or target tables and it processes
any transformation logic that it can push to the database.
The pushdown optimization Viewer helps to preview the SQL statements and mapping
logic that the Integration service can push to the source or target database. Also
it displays the messages related to push down optimization.

The Integration Service pushes as much transformation logic as possible to both

source and target databases. If you configure a session for full pushdown
optimization, and the Integration Service cannot push all the transformation logic
to the database, it performs source-side or target-side pushdown optimization
instead. Also the source and target must be on the same database. The Integration
Service analyzes the mapping starting with the source and analyzes each
transformation in the pipeline until it analyzes the target.

When it can push all transformation logic to the database, it generates an INSERT
SELECT statement to run on the database. The statement incorporates transformation
logic from all the transformations in the mapping. If the Integration Service can
push only part of the transformation logic to the database, it does not fail the
session, it pushes as much transformation logic to the source and target database
as possible and then processes the remaining transformation logic.

Pushdown Optimization Types

You can configure the following types of pushdown optimization:

•Source-side pushdown optimization. The Integration Service pushes as much
transformation logic as possible to the source database.
•Target-side pushdown optimization. The Integration Service pushes as much
transformation logic as possible to the target database.
•Full pushdown optimization. The Integration Service attempts to push all
transformation logic to the target database. If the Integration Service cannot push
all transformation logic to the database, it performs both source-side and target-
side pushdown optimization.
Running Source-Side Pushdown Optimization Sessions

When you run a session configured for source-side pushdown optimization, the
Integration Service analyzes the mapping from the source to the target or until it
reaches a downstream transformation it cannot push to the source database.

The Integration Service generates and executes a SELECT statement based on the
transformation logic for each transformation it can push to the database. Then, it
reads the results of this SQL query and processes the remaining transformations.

Running Target-Side Pushdown Optimization Sessions

When you run a session configured for target-side pushdown optimization, the
Integration Service analyzes the mapping from the target to the source or until it
reaches an upstream transformation it cannot push to the target database. It
generates an INSERT, DELETE, or UPDATE statement based on the transformation logic
for each transformation it can push to the target database. The Integration Service
processes the transformation logic up to the point that it can push the
transformation logic to the database. Then, it executes the generated SQL on the
Target database.

Running Full Pushdown Optimization Sessions

To use full pushdown optimization, the source and target databases must be in the
same relational database management system. When you run a session configured for
full pushdown optimization, the Integration Service analyzes the mapping from the
source to the target or until it reaches a downstream transformation it cannot push
to the target database. It generates and executes SQL statements against the source
or target based on the transformation logic it can push to the database.

Rules and Guidelines for Full Pushdown Optimization

Use the following rules and guidelines when you use pushdown optimization in
PowerCenter:•
When the Integration Service pushes transformation logic to the database, it cannot
track errors that occur in the database. When the Integration Service runs a
session configured for full pushdown optimization and an error occurs, the database
handles the errors.•
When the database handles errors, the Integration Service does not write reject
rows to the reject file.•
Mappings that generate long transactions require more database resources such as
locks and log space.•
The transaction is rolled back when an error is encountered because the database
cannot handle partial commits.•
Pushing transformation logic to the database might generate different results than
when the Integration Service runs the mapping. The database may process data
differently depending on case, data formats such as Numeric conversion to Char and
Date conversation to Char, null values in sort order, and precision.

Rules and Guidelines for Functions in Pushdown Optimization

Use the following rules and guidelines when pushing functions to a database:
•If you use ADD_TO_DATE in transformation logic to change days, hours, minutes, or
seconds, you cannot push the function to a Teradata database.
•When you push LAST_DAY () to Oracle, Oracle returns the date up to the second. If
the input date contains sub seconds, Oracle trims the date to the second.
•When you push LTRIM, RTRIM, or SOUNDEX to a database, the database treats the
argument (' ') as NULL, but the Integration Service treats the argument (' ') as
spaces.
•An IBM DB2 database and the Integration Service produce different results for
STDDEV and VARIANCE. IBM DB2 uses a different algorithm than other databases to
calculate STDDEV and VARIANCE.
•When you push SYSDATE or SYSTIMESTAMP to the database, the database server returns
the timestamp in the time zone of the database server, not the Integration Service.
•If you push SYSTIMESTAMP to an IBM DB2 or a Sybase database, and you specify the
format for SYSTIMESTAMP, the database ignores the format and returns the complete
time stamp.
•You can push SYSTIMESTAMP (‘SS’) to a Netezza database, but not SYSTIMESTAMP
(‘MS’) or SYSTIMESTAMP (‘US’).
•When you push TO_CHAR (DATE) or TO_DATE () to Netezza, dates with sub second
precision must be in the YYYY-MM-DD HH24: MI: SS.US format. If the format is
different, the Integration Service does not push the function to Netezza.

Limitations
All transformation cannot be pushed down
All functions cannot be pushed down
Usage of the feature is much dependent on the database compatibility of the
trasnformations and functions we use in mapping..
The query which IS forms automatically and incase of parallel flow IS splits the
mapping to pushdown groups and different query generated for each group,
can be viewed in Pushdown Optimization Viewer.

Performance Improvement through push down optimization

https://www.clearpeaks.com/blogs/wp-content/uploads/2016/07/Informatica-
Performance-Optimization-Techniques-Article.pdf