2/9/2011

Pushdown Optimization In Informatica

Pushdown Optimization In Informatica
Saurav Mitra

Pushdown Optimization which is a new concept in Informatica PowerCentre, allows developers to balance data transformation load among servers. This article describes pushdown techniques.

Pushdown Optimization
Pushdown optimization is a way of load-balancing among servers in order to achieve optimal performance. Veteran ETL developers often come across issues when they need to determine the appropriate place to perform ETL logic. Suppose an ETL logic needs to filter out data based on some condition. One can either do it in database by using WHERE condition in the SQL query or inside Informatica by using Informatica Filter transformation. Sometimes, we can even "push" some transformation logic to the target database instead of doing it in the source side (Especially in the case of EL-T rather than ETL). Such optimization is crucial for overall ETL performance.

How does Push-Down Optimization work?
One can push transformation logic to the source or target database using pushdown optimization. The Integration Service translates the transformation logic into SQL queries and sends the SQL queries to the source or the target database which executes the SQL queries to process the transformations. The amount of transformation logic one can push to the database depends on the database, transformation logic, and mapping and session configuration. The Integration Service analyzes the transformation logic it can push to the database and executes the SQL statement generated against the source or target tables, and it processes any transformation logic that it cannot push to the database.

Using Pushdown Optimization
Use the Pushdown Optimization Viewer to preview the SQL statements and mapping logic that the Integration Service can push to the source or target database. You can also use the Pushdown Optimization Viewer to view the messages related to pushdown optimization. Let us take an example: Image: Pushdown Optimization Example 1 Filter Condition used in this mapping is: DEPTNO>40 Suppose a mapping contains a Filter transformation that filters out all employees except those with a DEPTNO greater than 40. The Integration Service can push the transformation logic to the database. It generates the following SQL statement to process the transformation logic: INSERT INTO EMP_TGT(EMPNO, ENAME, SAL, COMM, DEPTNO)

dwbiconcepts.com/…/21-pushdown-op…

1/4

The Integration Service logs a message in the workflow log and the Pushdown Optimization Viewer when it cannot push an expression to the database. it pushes as much transformation logic to the source and target database as possible and then processes the remaining transformation logic. The Integration Service analyzes the mapping from the target to the source or until it reaches an upstream transformation it cannot push to the target database. EMPNO. Use the message to determine the reason why it could not push the expression to the database. the Integration Service might create temporary objects in the database. or UPDATE statement based on the transformation logic for each transformation it can push to the database and executes the DML. When it can push all transformation logic to the database. Also the source and target must be on the same database. The Integration Service does not extract data from the database at this time. DELETE. The statement incorporates transformation logic from all the transformations in the mapping. EMP_SRC. When we use pushdown optimization.2/9/2011 SELECT EMP_SRC.ENAME. it performs source-side or target-side pushdown optimization instead. EMP_SRC. The Integration Service analyzes the mapping from the source to the target or until it reaches a downstream transformation it cannot push to the source database and executes the corresponding SELECT statement.DEPTNO FROM EMP_SRC WHERE (EMP_SRC. The Integration Service creates temporary views in the database while pushing a Source Qualifier transformation dwbiconcepts. the Integration Service itself processes the transformation logic. If there is no equivalent operator. Using full pushdow n optimization: The Integration Service pushes as much transformation logic as possible to both source and target databases. The Integration Service creates a temporary sequence object in the database to push Sequence Generator transformation logic to the database. the Integration Service converts the expression in the transformation or in the workflow link by determining equivalent operators.SAL. or function. it does not fail the session.DEPTNO >40) Pushdown Optimization In Informatica The Integration Service generates an INSERT SELECT statement and it filters the data using a WHERE clause. It generates an INSERT. If the session is configured for full pushdown optimization. EMP_SRC. How does Integration Service handle Push Down Optimization? To push transformation logic to a database. the Integration Service pushes the Source Qualifier transformation and the Aggregator transformation to the source. and functions in the database. and pushes the Expression transformation and target to the target database.COMM. We can configure pushdown optimization in the following ways: Using source-side pushdow n optimization: The Integration Service pushes as much transformation logic as possible to the source database. The Integration Service analyzes the mapping starting with the source and analyzes each transformation in the pipeline until it analyzes the target. For example. Using target-side pushdow n optimization: The Integration Service pushes as much transformation logic as possible to the target database.com/…/21-pushdown-op… 2/4 . If the Integration Service can push only part of the transformation logic to the database. and the Integration Service cannot push all the transformation logic to the database. a mapping contains the following transformations: SourceDefn -> SourceQualifier -> Aggregator -> Rank -> Expression -> TargetDefn SUM(SAL). SUM(COMM) Group by DEPTNO RANK PORT on SAL TOTAL = SAL+COMM Image: Pushdown Optimization Example 2 The Rank transformation cannot be pushed to the database. processes the Rank transformation. variables. it generates an INSERT SELECT statement to run on the database. If you configure a session for full pushdown optimization. EMP_SRC. variable.

To change the configuration. Handling Error when Pushdown Optimization is enabled When the Integration Service pushes transformation logic to the database. When we select a pushdown option or pushdown group. 2. When we use native drivers. Microsoft SQL Server. 1. 2. it cannot track errors that occur in the database. or full pushdown optimization at different times and for that we can use the $$PushdownConfig mapping parameter. If the database server fails. 4. the database handles the errors. The possible values may be. Teradata. If the Integration Service fails. we do not change the pushdown configuration. If we configure a session for full pushdown optimization and the session fails. Configuring Parameters for Pushdown Optimization Depending on the database workload. Database that supports Informatica Pushdown Optimization We can configure sessions for pushdown optimization having any of the databases like Oracle. the Integration Service generates SQL statements using native database SQL. we must update the pushdown option in the session properties. Full [Seq View] Pushdown Optimization Viewer Use the Pushdown Optimization Viewer to examine the transformations that can be pushed to the database. After the database transaction completes. the Integration Service drops sequence and view objects created for pushdown optimization. it rolls back transactions when it restarts. To push Sequence Generator transformation logic to a database. the database rolls back the transactions. 2. 3. the database server rolls back the transaction. the Integration Service does not write reject rows to the reject file. IBM DB2. in session property for Pushdown Optimization attribute select $$PushdownConfig and define the parameter in the parameter file. the Integration Service cannot perform incremental recovery because the database processes the transformations. target-side. Instead. Sybase ASE or Databases that use ODBC drivers. Target [Seq View]. filtered lookup. When we use ODBC drivers. When the database handles errors. 1. To enable the Integration Service to create the view objects in the database we must configure the session for pushdown optimization with View. none i. The settings in the $$PushdownConfig parameter override the pushdown optimization settings in the session properties.2/9/2011 Pushdown Optimization In Informatica or a Lookup transformation with a SQL override to the database.e the integration service itself processes all the transformations. When the Integration Service runs a session configured for full pushdown optimization and an error occurs. Links Best Informatica Interview Questions & Answers Performance Comparison of Informatica Join and Database Join dwbiconcepts. Select a pushdown option or pushdown group in the Pushdown Optimization Viewer to view the corresponding SQL statement that is generated for the specified selections. we must configure the session for pushdown optimization with Sequence. the Integration Service generates SQL statements using ANSI SQL. we might want to use source-side. The Integration Service can generate more functions when it generates SQL statements using native language instead of ANSI SQL.com/…/21-pushdown-op… 3/4 . Create $$PushdownConfig parameter in the Mapping Designer . Source [Seq View]. an unconnected relational lookup.

Step by Step Approach Implementing Informatica Partitions Implementing Informatica Persistent Cache Aggregation with out Informatica Aggregator Last Updated on Tuesday.com/…/21-pushdown-op… 4/4 .How to Identify rejection reason Implementing Informatica Incremental Aggregation Using Informatica Normalizer Transformation Informatica Dynamic Lookup Cache Informatica Tuning . 11 January 2011 08:06 dwbiconcepts.2/9/2011 Pushdown Optimization In Informatica Comparing Performance of SORT operation (Order By) in Informatica and Oracle Informatica Reject File .

Sign up to vote on this title
UsefulNot useful