dbFEEDS Project Performance Tuning Guidelines– Informatica PowerCenter

Project: Stage: Phase: Work stream: Document: Subject:

db FEEDS Release 1 Design Interfaces Performance Tuning Guide Informatica – Performance tuning

Prepared by: Date: Reference: Version:

Rajesh Narayanan 31/December/2004 Draft

Page 1 of 23

dbFEEDS Project Performance Tuning Guidelines– Informatica PowerCenter

Version History:
Version Date By Changes

0.1 0.2 0.3

28/December 29/December 07/Jan

RN RN RN

First draft Added UNIX portion (section 5) Merged section2.2/2.3 from development standards.

Contributors:
Name Role Location Remarks

Approval:
Name Role Location Remarks

Sanjay De Costa Joseph Houben

Reference Documents:
Name Author Version Date

Unix system (OS) tuning

Informatica

Page 2 of 23

dbFEEDS Project Performance Tuning Guidelines– Informatica PowerCenter

CONTENTS

Page 3 of 23

dbFEEDS Project Performance Tuning Guidelines– Informatica PowerCenter

1 Document Description
This document describes the practices that can be followed by the ETL development team, in order to get the best of Informatica PowerCenter (ETL). This document mainly concentrates on optimising the performance of core ETL. In order to make ETL to achieve the optimal performance; it is imperative to strike a good balance in hardware, OS, RDBMS & Informatica PowerCenter 7.1.1. This document can be used as reference by the development team & administration team.

2 Document Organisation
This document is divided into following parts
o

Primary guidelines

-

Necessary

for

ETL

to

perform

optimally,

fundamental approach for ETL design with Informatica PC 7.1.1
o

Advanced guidelines - Guidelines can be applied on case-to-case basis, Can be followed based on the problem scenario / environment Optimising Unix system – Performance tuning system) the OS (Unix/Linux

o

3 Informatica PC Primary guidelines
3.1 Database utilisation
Utilise database for significant data handling operations, staging tables can be a real benefit for parallelism in operations. It reduces the amount of processing time by a significant amount.

3.2

Localisation
Try to localise the relational objects as far as possible. Try not to use synonyms for remote database. Usage of remote links for data processing & loading certainly slow the things down.

3.3

Removal of Database driven Sequence Generators
Usage of database oriented sequence generators proves to be a costly decision. As this requires wrapper function / store procedure call, which might degrade the performance by 3 times. Also it is not so easy to determine the bottleneck caused by database sequence generators. If it is must to use database sequence generators, then have a shared sequence generator & build a staging table from

Page 4 of 23

dbFEEDS Project Performance Tuning Guidelines– Informatica PowerCenter the flat file, add a SEQ_ID column & call a POST TARGET LOAD procedure to populate this column.
This requires a wrapper function / stored procedure call. Utilizing these stored procedures has caused performance to drop by a factor of 3 times. This slowness is not easily debugged - it can only be spotted in the Write Throughput column. Copy the map, replace the stored proc call with an internal sequence generator for a test run - this is how fast you COULD run your map. If we must use a database generated sequence number, then follow the instructions for the staging table usage. If we're dealing with GIG's or Terabytes of information - this should save you lot's of hours tuning. IF YOU MUST - have a shared sequence generator, then build a staging table from the flat file, add a SEQUENCE ID column, and call a POST TARGET LOAD stored procedure to populate that column. Place the post target load procedure in to the flat file to staging table load map. A single call to inside the database, followed by a batch operation to assign sequences is the fastest method for utilizing shared sequence generators.

3.4

Switch off the “Collect performance statistics”
This has an impact though it is minimal; removing this operation reduces reliance on the flat file operations. However it may be useful to have this option switch ON during tuning exercise.

3.5

Switch off the verbose logging
The session log has a tremendous impact on the overall performance of a session. Over-ride the session log to NORMAL logging mode. In informatica logging mechanism is not parallel; it is embedded into the operations. Also, this prevents informatica metadata table from growing. Also, it is good idea to perform some amount of automated house keeping which truncates the log from Informatica metadata at regular intervals.

3.6

Utilise staging
If the source is flat file utilise a staging table. This way you can use SQL Loader, Bulk-Load utility. Keep the basic logic in source load map; eliminate all lookups from the code. At this juncture if the reader is slow, then check for following
o

If there is an item in configuration file which sets a value to throttle reader, it will limit the read throughput. Move flat file to local disk; don’t read from network or from RAID.

o

Page 5 of 23

dbFEEDS Project Performance Tuning Guidelines– Informatica PowerCenter 3.7 Eliminate non-cached lookups Usage of non-cached lookups will hamper the performance significantly. Especially if the lookup table is “growing” or “updated” target table. This show the indexes are changing during the operation and optimizer looses the track of index & its statistics. If possible use staging table - this allows using joiner also which can increase the performance to large extent.

3.8

Tune the database
Estimate for small, medium and large source data set sizes, in terms of number of rows / average bytes per row. Also estimate the throughput for each and turnaround time for load. DBA should be provided with this information, along with tables that are expected to be high read / write .DBA should assign the right table to the right disk space that could make difference.

3.9

Availability of SWAP & TEMP space on PMSERVER
Having less disk space for SWAP & TEMP could potentially slow down the performance of entire server. To monitor this one need to watch the disk space while sessions are running. Without monitoring, it would be difficult to assess the reason, especially if mapping contains Aggregates, or lookups that uses disk cache or Joiner with heterogeneous sources.

3.10 Session Settings
Major chunk of tuning can be done in session. By switching on the “Collect performance statistics” one will come to know the parameters to be set in session level, or at least what has to be changed in database. Basically one should try to achieve OPTIMAL READ, OPTIMAL THROUGHPUT and OPTIMAL WRITE. Overtuning one of these pieces can ultimately slow down the sessions. Index Cache and Data cache are dynamically allocated first. As soon as the session is initialised, the memory for data and index caches are setup. Their sizes depend upon session settings The Reader DTM also based on dynamic allocation algorithm, it uses the memory available in chunks. Size of the chunk would be determined by the session setting “Default Buffer block size”

Page 6 of 23

dbFEEDS Project Performance Tuning Guidelines– Informatica PowerCenter Read the session throughput, then tune for the reader, see what the settings are, and send the write output to a flat file for less contention. Check the Throttle reader setting; increase the default buffer size by a factor of 64K each shot. If the reader still appears to increase during the session, then stabilize, and then try increasing Shared Session Memory from 12 MB to 24 MB. Check the writer throughput performance statistics to make sure there is NO writer bottleneck. If you have slow writer, change the map to single target table at a time to see which target is causing the slowness and tune it.
NOTE: if the reader session to flat file just doesn't ever "get fast", then we have got some basic map tuning to do. Try to merge expression objects, set the lookups to unconnected (for re-use if possible), check the Index and Data cache settings if we have aggregation, or lookups being performed. Etc... If we have a slow writer, change the map to a single target table at a time - see which target is causing the "slowness" and tune it. Make copies of the original map, and break down the copies. Once the "slower" of the N targets is discovered, talk to DBA about partitioning the table, updating statistics, removing indexes during load, etc... There are many database things you can do here. Remember the TIMING is affected by READER/TRANSFORMER/WRITER threads. With complex mappings, don't forget that each ELEMENT (field) must be weighed - in this light a firm understanding of how to read performance statistics generated by Informatica becomes important. In other words - if the reader is slow, then the rest of the threads suffer, if the writer is slow, same effect. A pipe is only as big as its smallest diameter.... A chain is only as strong as its weakest link. Sorry for the metaphors, but it should make sense.

3.11 Remove all other applications on PMServer
Except the database staging, PMServer plays well with RDBMS & its engine, but doesn’t play well with application servers, in particularly JAVA Virtual Machines, Web Servers, Security Servers, applications and Report Servers. All of these items should be broken out to other machines; this is critical to improve performance on PMServer machine.

3.12 Removal external registered modules
As far as possible, try to avoid the API’s which calls external objects, as this has been proven slow. External modules might exhibit speed problems, instead try using pre-processing / post processing with SED, AWK or GREP.

Page 7 of 23

dbFEEDS Project Performance Tuning Guidelines– Informatica PowerCenter

4 Informatica PC Advanced guidelines
4.1 Filter Expressions :
Create the filter (TRUE / FALSE) inside the port expression upstream. Complex filter expressions slow down the mapping. However it acts faster in Expression transformation with an output port for the result. Place the expression in EXPRESSION Transformation upstream from filter. Compute a single numerical flag: 1 for TRUE 0 for FALSE as output port. Push this data into the filter. This will have positive impact on performance. Use the Filter transformation early in the mapping. To maximize session performance, keep the Filter transformation as close as possible to the sources in the mapping. Rather than passing rows that you plan to discard through the mapping, you can filter out unwanted data early in the flow of data from sources to targets. Use the Source Qualifier to filter The Source Qualifier transformation provides an alternate way to filter rows. Rather than filtering rows from within a mapping, the Source Qualifier transformation filters rows when read from a source. The main difference is that the source qualifier limits the row set extracted from a source, while the Filter transformation limits the row set sent to a target. Since a source qualifier reduces the number of rows used throughout the mapping, it provides better performance.

4.2

Remove Default’s:
Having a default value including “ERROR” slows down the session. It causes unnecessary evaluation of values for every data element in the mapping. Best method of allotting default value is to have variable in expression, which returns the expected value on the condition. This will be faster than assigning default value.

4.3

Output port instead of Variable port
Variables are good for static and state driven, but slow down the performance time as they are allocated each time a row passes through expression object. Try to use Output port instead of variable port.

Page 8 of 23

dbFEEDS Project Performance Tuning Guidelines– Informatica PowerCenter 4.4 Datatype conversion:
Avoid performing implicit conversion of datatypes by connecting an Integer to string or vice versa. Instead use the function that converts the data explicitly, this avoids PMServer to decide on datatype conversion at run time.

4.5

String Functions:
String functions are costly on performance. E.g. ltrim, rtrim etc., as there involves allocate & re-allocate of memory within READER thread. Also it would be imperative to perform the string operations on the data, in which case following can be considered. Use varchar/varchar2 datatypes in database sources, if source is file then make it delimited one. Try to use LTRIM/RTRIM functions on the data coming in from a database SQL; this would be much faster than performing in ETL.

4.6

IIF Conditions caveat
As far as possible, make a logic that goes away from IIF, as IIF conditions are costly in any language. IIF creates multiple path logic inside the application & uses the decision to navigate. This might have an implication on performance as well. Other option is to use Oracle DECODE in source qualifier.

4.7

Expressions
Expressions like IS_SPACES, ISNUMBER etc. affects the performance, as this is the data validation expression that has to scan the entire string to determine the result. Try to avoid using these expressions unless there is absolute requirement for its usage.

4.8

Update Expressions for session :
In session if the option Update Else Insert is ON, then definitely performance will slow down. As, Informatica has to performs 2 operations for each rows update w.r.t PK, then if it returns 0 rows then perform Insert. As an alternative, Update Strategy can be used where rows would be marked using DD_UPDATE or DD_INSERT inside the mapping. In this case session settings can be INSERT & UPDATE AS UPDATE or UPDATE AS INSERT.

4.9

Multiple targets / sources are too slow :
Mappings with Multiple targets can eat up the performance some time. If the architecture permits then make one map per target. If the sources are from different

Page 9 of 23

dbFEEDS Project Performance Tuning Guidelines– Informatica PowerCenter ftp locations & they are flat file, then ideal choice would be FTPing the file to source to the ETL server & then process it.

4.10 Aggregator
If the mapping contains more than one aggregators, then the session will run slow, unless the cache dir is fast & disk drive access speed is high. Placing aggregator towards the end might be another option; however this will also bring down the performance. As all the I/O activity would be a bottleneck in informatica. Maplets are good source for replicating data logic, but if a maplet contains aggregator still the performance of the mapping (that contains maplet) will affect. Reduce the number of aggregators in the entire mapping to 1(if can), if possible, split the mapping to several mappings for breaking down the logics. Sorted input to aggregator will increase the performance to large extent, however if the sorted input is enabled & the data passing to aggregator is not sorted, Session will fail. Set the cache size to calculated amount using below mentioned formulae. Index size = (sum of column size in group-by ports + 17) X number of groups Data size = (sum of column size of output ports + 7) X number of groups

4.11 Joiner
Perform joins in a database. Performing a join in a database is faster than performing a join in the session. Use one of the following options:
o o

Create a pre-session stored procedure to join the tables in a database. Use the Source Qualifier transformation to perform the join.

Designate as the master source the source with the smaller number of records. For optimal performance and disk storage, designate the master source as the source with the lower number of rows. With a smaller master source, the data cache is smaller, and the search time is shorter. Set the cache size to calculated amount using below mentioned formulae. Index size = (sum of master column size in join condition +16) X number of rows in master table Data size = (sum of master column size NOT in join condition but on output ports + 8) x number of rows in master table

Page 10 of 23

dbFEEDS Project Performance Tuning Guidelines– Informatica PowerCenter 4.12 Lookups
When caching is enabled, the PowerCenter Server caches the lookup table and queries the lookup cache during the session. When this option is not enabled, the PowerCenter Server queries the lookup table on a row-by-row basis. Eliminate too many lookups. More the lookups means, the DTM reader/writer/Transform threads are not left with enough memory to be able to run efficiently (as it can). With too many lookups one need to trade in Memory contention for Disk contention. The memory contention might be worse than disk contention, as the OS ends up swapping in and out of TEMP/SWAP disk space, with small block sizes to try to locate the lookup row, and as the row goes from lookup to lookup, swapping becomes worse. Both lookups and aggregators require memory space & each of them requires Index & Data cache, ideally they share from the same heap segments. Hence, care should be taken while designing mapping that consumes the memory. In the case where a lookup uses more than one lookup condition, set the conditions with an equal sign first in order to optimize lookup performance. In the case of a cached lookup, an ORDER BY condition is issued in the SQL statement used to create the cache. Columns used in the ORDER BY condition should be indexed. The session log will contain the ORDER BY statement. Tips on Caches: Cache small lookup tables Improve session performance by caching small lookup tables. The result of the lookup query and processing is the same, whether or not you cache the lookup table. Use a persistent lookup cache for static lookup tables: If the lookup table does not change between sessions, configure the Lookup transformation to use a persistent lookup cache. The Informatica Server then saves and reuses cache files from session to session, eliminating the time required to read the lookup table. If the lookup table does not change between sessions, configure the Lookup transformation to use a persistent lookup cache. The Informatica Server then saves and reuses cache files from session to session, eliminating the time required to read the lookup table. Override the ORDER BY statement for cached lookups: By default, the Informatica Server generates an ORDER BY statement for a cached lookup that contains all lookup ports. To increase performance, you can suppress

Page 11 of 23

dbFEEDS Project Performance Tuning Guidelines– Informatica PowerCenter the default ORDER BY statement and enter an override ORDER BY with fewer columns
Place conditions with an equality operator (=) first. If a Lookup transformation specifies several conditions, you can improve lookup performance by placing all the conditions that use the equality operator first in the list of conditions that appear under the Condition tab Consider following for calculating caches for lookup Attributes Minimum Index Cache Maximum Index Cache Method 200 * [( S column size) + 16] over all condition ports # rows in lookup table [(S column size) + 16] * 2 over all condition ports # rows in lookup table [(S column size) + 8] over all outputports (not condition port) 2 * minimum data cache

Minimum Data Cache

Maximum Data Cache

4.12.1 Lookups & Aggregators Fight.
The lookups and the aggregators fight for memory space as discussed above. Each requires Index Cache, and Data Cache and they "share" the same HEAP segments inside the core. Particularly in the 4.6 / 1.6 products and prior - these memory areas become critical, and when dealing with many rows - the session is almost certain to cause the server to "thrash" memory in and out of the OS Swap space. If possible, separate the maps - perform the lookups in the first section of the maps, position the data in an intermediate target table - then a second map reads the target table and performs the aggregation (also provides the option for a group by to be done within the database)... Another speed improvement...

4.13 Maplets for complex Logic
It’s good idea to break the complex logic into maplets. This allows managing the mapping in much better & efficient way of breaking down the business logics. Always remember shorter the distance between source and Target, better the performance. With complex mappings READER,WRITER & TRANSFORM threads affects by timing, i.e. if the reader is slow, then rest of the threads suffer,similarily f he writer is slow same is the effect.

Page 12 of 23

dbFEEDS Project Performance Tuning Guidelines– Informatica PowerCenter 4.14 Database IPC settings & Priorities
If PMServer & Oracle instance are running on the same server, use IPC connection instead of TCP/IP connection. Try to change the protocol in the TNSNames.ORA and Listener.ORA files, and restart the listener on the server .However this can be used only locally, at the same time the speed increases between 2x and 5x .Another option one can think is prioritizing the database login that informatica uses to execute its task. These tasks when login to the database would override others. This would be particularly helpful in increasing the performance especially when bulk loader or SQL Loader is used.

4.15 Loading
Make sure indexes and constraints are removed before loading into relational targets & this can be created as soon as the load is completed. This would help in boost up the performance in bulk data loads. Lesser the commit interval more the time for session to complete, set the appropriate commit interval, anything above 50K is good. Partioning the data while loading is another wise option. Following are the partitions Informatica provides
o o o o

Key Range Hash Key Round Robin Pass Through

When partitioning the individual transformation it is advisable to go for the following
o o o

Aggregator Cache Lookup Cache Sorter Cache

Use hash auto key Use hash auto key partition type, equality condition Use Hash auto key or Pass-through or Key range

4.16 Memory Settings
Session Shared Memory Size controls the total amount of memory used to buffer rows internally by the reader and writer. Set session shared memory between 12MB and 25MB, remember increasing the shared memory beyond this doesn’t guarantee increase in performance rather it does decrease the performance. Buffer Block Size controls the size of the blocks that move in the pipeline. Set shared buffer block size around 128K, This would be used by informatica for handling block of rows.

Page 13 of 23

dbFEEDS Project Performance Tuning Guidelines– Informatica PowerCenter If the server has RAM over 12 GIG’s, then shared memory can be increased between 1 and 2 GIG’s. Also the shared buffer block size should set relative to shared memory settings.
The Informatica Server moves data from sources to targets based on workflow and mapping metadata stored in a repository. A workflow is a set of instructions that describes how and when to run tasks related to extracting, transforming, and loading data. The Informatica Server runs workflow tasks according to the conditional links connecting the tasks.

4.17 Reduce Number of OBJETS in a map
Frequently, the idea of these tools is to make the "data translation map" as easy as possible. All too often, that means creating "an" (1) expression for each throughput/translation (taking it to an extreme of course). Each object adds computational overhead to the session and timings may suffer. Sometimes if performance is an issue / goal, integrate several expressions in to one expression object, thus reducing the "object" overhead. In doing so – it could speed up the map.

4.18 Slow Sources - Flat Files
If we've got slow sources, and these sources are flat files, we can look at some of the following possibilities. If the sources reside on a different machine, and we've opened a named pipe to get them across the network - then we've opened (potentially) a can of worms. We’ve introduced the network speed as a variable on the speed of the flat file source. Try to compress the source file, FTP PUT it on the local machine (local to PMServer), decompress it, and then utilize it as a source. If you're reaching across the network to a relational table - and the session is pulling many rows (over 10,000) then the source system itself may be slow. We may be better off using a source system extract program to dump it to file first, and then follow the above instructions. However, there is something your SA's and Network Ops folks could do (if necessary) - this is covered in detail in the advanced section. They could backbone the two servers together with a dedicated network line (no hubs, routers, or other items in between the two machines). At the very least, they could put the two machines on the same sub-net. Now, if the file is local to PMServer but is still slow, examine the location of the file (which device is it on). If it's not on an INTERNAL DISK then it will be slower than if it were on an internal disk (C drive for you folks on NT). This doesn't mean a UNIX file LINK exists locally, and the file is remote - it means the actual file is local.

4.19 Break the mappings out

Page 14 of 23

dbFEEDS Project Performance Tuning Guidelines– Informatica PowerCenter One per target. If necessary, 1 per source per target. Why does this work? Well eliminating multiple targets in a single mapping can greatly increase speed... Basically it's like this: one session per map/target. Each session establishes its own database connection. Because of the unique database connection, the DBMS server can now handle the insert/update/delete requests in parallel against multiple targets. It also helps to allow each session to be specified for its intended purpose (no longer mixing a data driven session with INSERTS only to a single target). Each session can then be placed in to a batch marked "CONCURRENT" if preferences allow. Once this is done, parallelism of mappings and sessions become obvious. A study of parallel processing has shown again and again, that the operations can be completed sometimes in half the time of their original counterparts merely by streaming them at the same time. With multiple targets in the same mapping, a single database connection to handle multiplies diverse database statements - sometimes hitting this target, other times hitting that target. Think - in this situation it's extremely difficult for Informatica (or any other tool for that matter) to build BULK operations... even though "bulk" is specified in the session. Remember that "BULK" means this is your preference, and that the tool will revert to NORMAL load if it can't provide a BULK operation on a series of consecutive rows. Obviously, data driven then forces the tool down several other layers of internal code before the data actually can reach the database.

4.19.1 Keep the mappings as simple as possible
Bury complex logic (if you must) in to a maplet. If you can avoid complex logic all together - then that would be the key. The old rule of thumb applies here (common sense) the straighter the path between two points, the shorter the distance... Translated as: the shorter the distance between the source qualifier and the target - the faster the data loads

4.20 READER/TRANSFORMER/WRITER threads affect the TIMING
With complex mappings, don't forget that each ELEMENT (field) must be weighed - in this light a firm understanding of how to read performance statistics generated by Informatica becomes important. In other words - if the reader is slow, then the rest of the threads suffer, if the writer is slow, same effect. A pipe is only as big as its smallest diameter.... A chain is only as strong as its weakest link.

4.21 Sorting – performance issues
We can improve Aggregator transformation performance by using the Sorted Input option. When the Sorted Input option is selected, the Informatica Server assumes all data is sorted by group. As the Informatica Server reads rows for a group, it performs

Page 15 of 23

dbFEEDS Project Performance Tuning Guidelines– Informatica PowerCenter aggregate calculations as it reads. When necessary, it stores group information in memory. To use the Sorted Input option, we must pass sorted data to the Aggregator transformation. We can gain added performance with sorted ports when we partition the session. When Sorted Input is not selected, the Informatica Server performs aggregate calculations as it reads. However, since data is not sorted, the Informatica Server stores data for each group until it reads the entire source to ensure all aggregate calculations are accurate. For example, one Aggregator has the STORE_ID and ITEM Group By ports, with the Sorted Input option selected. When you pass the following data through the Aggregator, the Informatica Server performs an aggregation for the three records in the 101/battery group as soon as it finds the new group, 201/battery:
STORE_ID ITEM QTY PRICE 101 ‘battery’ 3 2.99 101 ‘battery’ 1 3.19 101 ‘battery’ 2 2.59 201 ‘battery’ 4 1.59 201 ‘battery’ 1 1.99 If you use the Sorted Input option and do not presort data correctly, the session fails.

4.21.1 Sorted Input Conditions
Do not use the Sorted Input option if any of the following conditions are true: • • • The aggregate expression uses nested aggregate functions. The session uses incremental aggregation. Input data is data-driven. You choose to treat source data as data driven in the session properties, or the Update Strategy transformation appears before the Aggregator transformation in the mapping. • If we use the Sorted Input option under these circumstances, the Informatica Server reverts to default aggregate behaviour, reading all values before performing aggregate calculations.

4.21.2 Pre-Sorting Data
To use the Sorted input option, you pass sorted data through the Aggregator. Data must be sorted as follows: • • By the Aggregator group by ports, in the order they appear in the Aggregator transformation. Using the same sort order configured for the session. If data is not in strict ascending or descending order based on the session sort order, the Informatica Server fails the session. For example, if you configure a session to use a

Page 16 of 23

dbFEEDS Project Performance Tuning Guidelines– Informatica PowerCenter French sort order, data passing into the Aggregator transformation must be sorted using the French sort order. If the session uses file sources, you can use an external utility to sort file data before starting the session. If the session uses relational sources, we can use the Number of Sorted Ports option in the Source Qualifier transformation to sort group by columns in the source database. Group By columns must be in the exact same order in both the Aggregator and Source Qualifier transformations.

4.22 Workflow Manager
The Informatica Server moves data from sources to targets based on workflow and mapping metadata stored in a repository. A workflow is a set of instructions that describes how and when to run tasks related to extracting, transforming, and loading data. The Informatica Server runs workflow tasks according to the conditional links connecting the tasks. • • • • Monitor, add, edit, delete Informatica Server info in the repository Stop the informatica Server Configure database, external loader, and FTP connections Manage sessions and batches - create, edit, delete, copy/move within a folder, start/stop, abort sessions, view session logs, details, and session performance details.

Source

Server

Target

Source Data data

Transformed

Page 17 of 23

dbFEEDS Project Performance Tuning Guidelines– Informatica PowerCenter Instructions from Metadata

` Repository

4.22.1 Monitoring and Running a Session:
When the Informatica Server runs a Session task, the Workflow Monitor creates session details that provide load statistics for each target in the mapping. We can view session details when the session runs or after the session completes. Create a worklet to reuse a set of workflow logic in several workflows. Use the Worklet Designer to create and edit worklets.

4.22.2 Informatica suggests that each session takes roughly 1 to 1 1/2 CPU's.
In keeping with this - Informatica play's well with RDBMS engines on the same machine, but does NOT get along (performance wise) with ANY other engine (reporting engine, java engine, OLAP engine, java virtual machine, etc...)

4.22.3 Place some good server load monitoring tools on the PM Server in development
Watch it closely to understand how the resources are being utilized, and where the hot spots are. Try to follow the recommendations - it may mean upgrading the hardware to achieve throughput. Look in to EMC' s disk storage array - while expensive, it appears to be extremely fast, it may improve the performance in some cases by up to 50%

4.23 Change Database Priorities for the PMServer Database User
Prioritizing the database login that any of the connections use (setup in Server Manager) can assist in changing the priority given to the Informatica executing tasks. These tasks when logged in to the database then can over-ride others. Sizing memory for these tasks (in shared global areas, and server settings) must be done if priorities are to be changed. If BCP or SQL*Loader or some other bulk-load facility is utilized, these priorities must also be set. This can greatly improve performance. Again, it's only suggested as a last resort method, and doesn't substitute for tuning the database, or the

Page 18 of 23

dbFEEDS Project Performance Tuning Guidelines– Informatica PowerCenter mapping processes. It should only be utilized when all other methods have been exhausted (tuned). Keep in mind that this should only be relegated to the production machines and only in certain instances where the Load cycle that Informatica is utilizing is NOT impeding other users.

4.24 Change the Unix User Priority
In order to gain speed, the Informatica UNIX User must be given a higher priority. The UNIX SA should understand what it takes to rank the UNIX logins, and grant priorities to particular tasks. Or - simply have the pmserver executed under a super user (SU) command; this will take care of reprioritizing Informatica's core process. This should only be used as a last resort - once all other tuning avenues have been exhausted, or if we have a dedicated UNIX machine on which Informatica is running.

4.25 Try not to load across the network
If at all possible, try to co-locate PMServer executable with a local database. Not having the database local means: 1) the repository is across the network (slow), 2) the sources / targets are across the network, also potentially slow. If we have to load across the network, at least try to localize the repository on a database instance on the same machine as the server. The other thing is: try to co-locate the two machines (pmserver and Target database server) on the same sub-net, even the same hub if possible. This eliminates unnecessary routing of packets all over the network. Having a localized database also allows us to setup a target table locally - which you can then "dump" following a load, ftp to the target server, and bulk-load in to the target table. This works extremely well for situations where append or complete refresh is taking place.

4.26 Balance between Informatica and the power of SQL and the database
Try to utilize the DBMS for what it was built for: reading/writing/sorting/grouping/filtering data en-masse. Use Informatica for the more complex logic, outside joins, data integration, multiple source feeds, etc... The balancing act is difficult without DBA knowledge. In order to achieve a balance, we must be able to recognize what operations are best in the database, and which ones are best in Informatica. This does not degrade from the use of the ETL tool, rather it enhances it - it's a MUST if you are performance tuning for high-volume throughput.

5 Performance tuning the UNIX OS

Page 19 of 23

dbFEEDS Project Performance Tuning Guidelines– Informatica PowerCenter 5.1 Process check ps-axu: Run to check for the following items:
o

Are there any processes waiting for disk access or for paging, if so check the I/O and memory subsystems. Processes that are using most of the CPU and Processes are using most of the memory. This may help you distribute the workload better.

o

5.2

Identifiying & resolving memory problem

Run vmstat S 5 confirms memory problems and check for the following:
o o

Pages-outs occurring consistently? If so, you are short of memory. Are there a high number of address translation faults? (System V only) This suggests a memory shortage. Are swap-outs occurring consistently? If so, you are extremely short of memory. Occasional swap-outs are normal; BSD systems swap-out inactive jobs. Long bursts of swap-outs mean that active jobs are probably falling victim and indicate extreme memory shortage. If you don’t have vmstat -S, look at the w and de fields of vmstat. These should ALWAYS be zero.

o

If memory seems to be the bottleneck of the system, try following remedial steps:
o

Reduce the size of the buffer cache, if your system has one, by decreasing BUFPAGES. The buffer cache is not used in system V.4 and SunOS 4.X systems. Making the buffer cache smaller will hurt disk I/O performance.

o

If you have statically allocated STREAMS buffers, reduce the number of large (2048- and 4096-byte) buffers. This may reduce network performance, but netstat-m should give you an idea of how many buffers you really need.

o

Reduce the size of your kernels tables. This may limit the systems capacity (number of files, number of processes, etc.). Try running jobs requiring a lot of memory at night. This may not help the memory problems, but you may not care about them as much.

o

Page 20 of 23

dbFEEDS Project Performance Tuning Guidelines– Informatica PowerCenter o Try running jobs requiring a lot of memory in a batch queue. If only one memory-intensive job is running at a time, your system may perform satisfactorily.
o o

Try to limit the time spent running sendmail, which is a memory hog. If you don’t see any significant improvement, add more memory.

5.3

Identifying and Resolving Disk I/O Issues

Use iostat to check i/o load and utilization, as well as CPU load. Iostat can be used to monitor the I/O load on the disks on the UNIX server. Using iostat permits monitoring the load on specific disks. Take notice of how fairly disk activity is distributed among the system disks. If it is not, are the most active disks also the fastest disks Following might help to rectify the problem due to I/O
o

Reorganize your file systems and disks to distribute I/O activity as evenly as possible. Using symbolic links helps to keep the directory structure the same throughout while still moving the data files that are causing I/O contention. Use your fastest disk drive and controller for your root file system; this will almost certainly have the heaviest activity. Alternatively, if single-file throughput is important, put performance-critical files into one file system and use the fastest drive for that file system.

o

o

o

Put performance-critical files on a file system with a large block size: 16KB or 32KB (BSD). Increase the size of the buffer cache by increasing BUFPAGES (BSD). This may hurt your systems memory performance. Rebuild your file systems periodically to eliminate fragmentation (backup, build a new file system, and restore). If you are using NFS and using remote files, look at your network situation. You don’t have local disk I/O problems. Check memory statistics again by running vmstat 5 (sar-rwpg). If your system is paging or swapping consistently, you have memory problems, fix memory problem first. Swapping makes performance worse.

o

o

o

o

Page 21 of 23

dbFEEDS Project Performance Tuning Guidelines– Informatica PowerCenter If system has disk capacity problem and is constantly running out of disk space, try the following actions:

o

Write a find script that detects old core dumps, editor backup and autosave files, and other trash and deletes it automatically. Run the script through croon.

o

Use a smaller block size on file systems that are mostly small files (e.g., source code files, object modules, and small data files).

5.4

Identifying and Resolving CPU Overload Issues

Use sar u to check for CPU loading. This provides the %usr (user), %sys (system), %wio (waiting on I/O), and %idle (% of idle time). A target goal should be %usr + %sys= 80 and %wio = 10 leaving %idle at 10. If %wio is higher, the disk and I/O contention should be investigated to eliminate I/O bottleneck on the UNIX server. If the system shows a heavy load of %sys, and %usr has a high %idle, this is indicative of memory and contention of swapping/paging problems. In this case, it is necessary to make memory changes to reduce the load on the system server. When you run iostat 5 above, also observe for CPU idle time. Is the idle time always 0, without letup? It is good for the CPU to be busy, but if it is always busy 100 percent of the time, work must be piling up somewhere. These points to CPU overload.
o

Eliminate unnecessary daemon processes. rwhod and routed are particularly likely to be performance problems, but any savings will help. Get users to run jobs at night with any queuing system that’s available always for help. You may not care if the CPU (or the memory or I/O system) is overloaded at night, provided the work is done in the morning.

o

o

Use nice to lower the priority of CPU-bound jobs will improve interactive performance. Also, using nice to raise the priority of CPU-bound jobs will expedite them but will hurt interactive performance. In general though, using nice is really only a temporary solution. If your workload grows, it will soon become insufficient. Consider upgrading your system, replacing it, or buying another system to share the load.

Page 22 of 23

dbFEEDS Project Performance Tuning Guidelines– Informatica PowerCenter 5.5 Identifying and Resolving Network Issues
One can suspect problems with network capacity or with data integrity if users experience slow performance when they are using rlogin or when they are accessing files via NFS. Look at netsat-I. If the number of collisions is large, suspect an overloaded network. If the number of input or output errors is large, suspect hardware problems. A large number of input errors indicate problems somewhere on the network. A large number of output errors suggest problems with your system and its interface to the network. If collisions and network hardware are not a problem, figure out which system appears to be slow. Use spray to send a large burst of packets to the slow system. If the number of dropped packets is large, the remote system most likely cannot respond to incoming data fast enough. Look to see if there are CPU, memory or disk I/O problems on the remote system. If not, the system may just not be able to tolerate heavy network workloads. Try to reorganize the network so that this system isn’t a file server. A large number of dropped packets may also indicate data corruption. Run netstats on the remote system, then spray the remote system from the local system and run netstat-s again. If the increase of UDP socket full drops (as indicated by netstat) is equal to or greater than the number of drop packets that spray reports, the remote system is slow network server If the increase of socket full drops is less than the number of dropped packets, look for network errors. Run nfsstat and look at the client RPC data. If the retransfield is more than 5 percent of calls, the network or an NFS server is overloaded. If timeout is high, at least one NFS server is overloaded, the network may be faulty, or one or more servers may have crashed. If badmixis roughly equal to timeout, at least one NFS server is overloaded. If timeout and retrans are high, but badxidis low, some part of the network between the NFS client and server is overloaded and dropping packets. Try to prevent users from running me /O- intensive programs across the network. The greputility is a good example of an I/O intensive program. Instead, have users log into the remote system to do their work. Reorganize the computers and disks on your network so that as many users as possible can do as much work as possible on a local system. Use systems with good network performance as file servers. If you are short of STREAMS data buffers and are running Sun OS 4.0 or System V.3 (or earlier), reconfigure the kernel with more buffers.

Page 23 of 23

Sign up to vote on this title
UsefulNot useful