PowerCenter Developer: Tips & Tricks for Mapping Designer

Lingaraju Ramasamy (Raju), Technical Architecture Manager Informatica Professional Services

2

Agenda
• Introduction • Architecture Best Practices

• Mapping Tips & Tricks
• Transformation Techniques

• Use of Metadata • Repository maintenance • Q&A

3

Introduction

4

Presenter Contact
• Lingaraju Ramasamy (Raju) • lramasamy@informatica.com

• 408-368-2475 (Mobile)
• Technical Architecture Manager, Informatica Professional Services

5

Architecture Best Practices 6 .

reusable functions 7 .Architecture Best Practices • Consistency • Applying consistent standards reduces long term complications • Naming Conventions (Velocity) • Descriptions • Environments • Documentation (Hyperlink to SharePoint) • Modularity • Develop according to a modular design • Common Error Handling • Reprocessing • Mapping Assistants • Reusability • Focus on reuse to make quick and universal modifications • Mapplets. Worklets. Transformations.

maintain and debug 8 . Incremental Loads • Simplicity • Multiple simple processes are often better than few complex processes • Multiple mappings • Simple Queries • Staging Tables • Advantages • Easy to develop. debug.Architecture Best Practices • Scalability • Keep volumes in mind in order to create efficient mappings • Caching • Queries • Partitioning • Initial vs.

Sample Complex Mapping SC_T_STR_AT TR_OUTL_ORG _WK (Oracle) SQ_SC_T_STR _ATTR_OUTL_ ORG_WK SEQ_ORG EXP_OUTL_OR G_BOOKEND SC_T_STR_AT TR_OUTL_HOL _FN (Oracle) SQ_SC_T_STR _ATTR_OUTL_ HOL_FN SEQ_HOL EXP_HOL_BOO KEND SEQ_OTHERS SEQ_OUTLET SC_T_STR_AT TR_OUTL_FN1 (Oracle) SQ_SC_T_STR _ATTR_OUTL_ FN1 EXP_OUTL_BO OKEND JAVA_GENERA TE_MSGID JAVA_GENERA TE_SESSID SEQ_PK_FK_O UTLET EXP_CREATE_ HDR_ELEMEN TS SEQ_HR_HDR SC_T_STR_AT TR_OUTL_HRS _WK1 (Oracle) SQ_SC_T_STR _ATTR_OUTL_ HRS_WK1 EXP_HRS_HDR SC_T_STR_AT TR_OUTL_HRS _WK (Oracle) SQ_SC_T_STR _ATTR_OUTL_ HRS_WK SEQ_HRS EXP_HRS_BOO KEND JNR_HRS_HDR JNR_OUTL_HR S JNR_OUTL_HR _HOL Multiple Sources –> WebServices JNR_OUTL_LA NG_HOL_HRS JNR_TEAM_ME MBER JNR_ORGANIZ ATION EXP_PASS_TH ROUGH SRT EXP_CHK_NE W_RECS TC_TRANSACT ION_RECS RTR_HDR_DE T_DATA EXP_DETAIL_H OURS SEQ_LANG SC_T_STR_AT TR_OUTL_LAN G_FN (Oracle) SQ_SC_T_STR _ATTR_OUTL_ LANG_FN EXP_LANG_BO OKEND SEQ_TEAM_ME MBER SC_T_STR_AT TR_OUTL_TM_ FN (Oracle) SQ_TEAM_ME MBER EXP_TEAM_ME MBER EXP_SEQ_SER VICE_REC_3 EXP_SEQ_DET AIL_HIERARCH Y_RECS EXP_SEQ_HEA DER_HIERARC HY_RECS11 EXP_HEADER_ HOURS EXP_SEQ_REC _BASIC EXP_SEQ_REC _TM_ZONE EXP_SEQ_DET AIL_HOLIDAY_ RECS EXP_SEQ_REC _GEO_TYPE EXP_SEQ_HEA DER_TEAM_M EMBER_RECS EXP_ACCOUNT ING_REC_6 EXP_SEQ_DAY _OF_WEEK EXP_SEQ_HEA DER_HOLIDAY _RECS EXP_SEQ_IDE NTIFICATION_ REC_4 EXP_SEQ_STA TUS_REC_5 EXP_SEQ_DET AIL_TEAM_ME MBER_RECS UNI_OUTLET EXP_HDR_BOO KEND WSC_STR_OU Tl_SV_OUTL HOL_LEVEL3 (F lat File) 9 .

Simplified Complex Mapping SEQ_LANG SC_T_STR_AT TR_OUTL_CHG _WK1 (Oracle) SQ_SC_T_STR _ATTR_OUTL_ CHG_WK EXP_SRC_BOO KEND SC_T_STR_AT TR_OUTL_LAN G_FN (Oracle) SQ_SC_T_STR _ATTR_OUTL_ LANG_FN EXP_LANG_BO OKEND SEQ_OUTLET SEQ_OUTLET SEQ_OTHERS SC_LKP_GET_ MSGID SC_T_STR_AT TR_OUTL_CHG _WK (Oracle) SQ_SC_T_STR _ATTR_OUTL_ CHG_WK EXP_OUTL_BO OKEND SC_EXP_CREA TE_HDR_ELEM ENTS SEQ_OTHERS SC_LKP_GET_ MSGID SC_T_STR_AT TR_OUTL_HRS _WK (Oracle) SC_EXP_CREA TE_HDR_ELEM ENTS JNR_OUTL_LA NG_HOL_HRS EXP_PASS_TH ROUGH SRT_INCM_RE CS EXP_CHK_NE W_RECS RTR_HDR_DE T_DATA EXP_HEADER_ LANG LANG_HEADER _SC_T_STR_A TTR_OUTL_W K (Oracle) SQ_SC_T_STR _ATTR_OUTL_ HRS_WK SEQ_HRS EXP_HRS_BOO KEND JNR_OUTL_HR S EXP_PASS_TH ROUGH SRT_INCM_RE CS EXP_CHK_NE W_RECS RTR_HDR_DE T_DATA EXP_SEQ_DAY _OF_WEEK DAY_OF_WEEK _SC_T_STR_A TTR_OUTL_W K2 (Oracle) Staging 1 SEQ_LANG SC_T_STR_AT TR_OUTL_LAN G_FN (Oracle) SQ_SC_T_STR _ATTR_OUTL_ LANG_FN EXP_LANG_BO OKEND EXP_DETAIL_L ANG LANG_DETAIL_ STR_ATTR_OU TL_WK (Oracle) EXP_HEADER_ HOURS Staging 2 SC_T_STR_AT TR_SITE_FN (O racle) SQ_SC_T_STR _ATTR_SITE_F N EXP_SRC_BOO KEND JAVA_GEN_MS G_ID HOURS_HDR_ SC_T_STR_AT TR_OUTL_WK ( Oracle) EXP_DETAIL_H OURS DETAIL_HOUR S_SC_T_STR_ ATTR_OUTL_ WK1 (Oracle) SEQ_OUTLET SC_T_STR_AT TR_OUTL_CHG _WK (Oracle) SQ_SC_T_STR _ATTR_OUTL_ CHG_WK EXP_OUTL_BO OKEND JAVA_GEN_SE SSID SEQ_OTHERS SC_LKP_GET_ MSGID SEQ_ID SC_EXP_CREA TE_HDR_ELEM ENTS JNR_OUTL_LA NG_HOL_HRS EXP_PASS_TH ROUGH SRT_INCM_RE CS EXP_CHK_NE W_RECS RTR_HDR_DE T_DATA EXP_HEADER_ LANG LANG_HEADER _SC_T_STR_A TTR_OUTL_W K (Oracle) EXP_GET_SEQ _NUM WSC_STR_ATT R_SAVE_SITE SC_T_STR_AT TR_SITE_FN1 ( Oracle) EXP_DETAIL_L ANG Staging 3 LANG_DETAIL_ STR_ATTR_OU TL_WK (Oracle) Staging –> WebServices 10 .

Mapping Tips & Tricks 11 .

Mapping Tips • Sources and Targets • Use shortcuts from shared folders • Extract only what is necessary • Limit reads on source • Distinguish between similar sources and targets • Example • DIM_CUSTOMER1 = DIM_CUSTOMER_insert • DIM_CUSTOMER2 = DIM_CUSTOMER_update 12 .

SESSSTARTTIME) 13 .Mapping Tricks Parameters & Variables • • • Reduce overhead of creating multiple mappings Replace hard coded values Use to incrementally extract data Example UpdateDateTime >= TO_DATE (‘$$PREV_RUN_TS’) : SetVariable (‘$$CURR_RUN_TS.

Mapping Tricks Parameters & Variables • Assign Parameter/Variable values in a Session • Pass values from one session to a subsequent session in same workflow/worklet • On Components Tab in Session Properties • Use user-defined workflow/worklet variables • Non-reusable Sessions only 14 .

Mapping Tricks Built-in Mapping Variables • Mapping Name • Workflow Name • Session Name • • • • • • • Integration Service Name Repository Service Name Repository User Name Folder Name Session Run Mode Source Table Names Target Table Names 15 .

Mapping Tricks Group Expression (Anchor transformation) • Add expression transformation after a source qualifier and before a target • If source or target definition changes. reconnecting ports is much easier 16 .

Mass Update • pmrep massupdate • Session properties • Session config attributes • Transformation instance attributes • Session instance run time options 17 .

Flat file • XML Files • For further analysis. client and reports 18 . use Informatica Analyst • Analyze the content.Mapping Assistants Preview Data • View Data • Accommodate anomalies early • Verification of extraction/loading strategies • Type of Data • Source/Targets • Relational. quality and structure of source data • Involves separate Profiling warehouse.

Mapping Assistants Mapping Wizard • Pass-Through • Slowly Changing Dimension • Type 1 Dimension (No History) • Type 2 Dimension (All History) • Version Data • Flag Current • Effective Date Range • Type 3 Dimension (Previous Versions) Slowly Changing Dimension Template 19 .

Mapping Assistants Mapping Analyst for Excel (MAE) • Standardize specifications • Enhance collaboration between analyst and developer • Improve documentation & audit ability of business logic Data Analyst Defines Business Terms Specifies Transformation Rules Standardize Excel format DI Developer Augments. Tunes Generated Mappings from Specifications Generate Specification Generate Mapping 20 .

Mapping Assistants Mapping Architect for Visio (MAV) • Define consistent methodology & structure for data integration projects • Build custom wizard based on pattern without coding • Generate multiple mappings at one time • Document data flow DI Architect Creates & Publishes mapping template Template File DI Developer Augments. Tunes Generated Mappings Informatica Toolbar Informatica Stencil Publish Template Drawing Window Generate Mappings Parameter File 21 .

Mapping Assistants Mapping Architect for Visio (MAV) • Case Study #1 •7 templates were used across 2 projects to generate 600 mappings •97% of mappings were automatically generated and required no additional changes •3% needed to be manually modified or custom developed Case Study #2 •1 template was used to create 150 mappings for a data migration project along with PowerCenter sessions and workflows •Total effort was less than one day •Equivalent effort to create 150 mappings manually would have been 2 weeks (10x effort) 22 .

Transformation Techniques 23 .

hints) • Can accommodate complex queries CONS • Processing impacts database resources • Lose transformation logic in metadata searched • Unable to utilize Partitioning or Pushdown Optimization options • Minimize complicated queries • Add the SQL Override Query to the Description 24 ..Transformation Tips Source Qualifier • Apply Default Query when possible • Utilize SQ Attributes (i. Source Filter) • Understand advantages and limitations of the SQL override PROS • Utilize database optimizers (i.e. indexes.e. User Defined Join..

or //) in expressions • Optimize Expressions • Numeric operations are faster than string operations • Operators are faster than functions • Un-Nest complicated logic (use IIF or DECODE) 25 .Transformation Tips Expressions • Understand Port process order • INs or IN/OUTs • VARIABLEs • OUTPUTs • Reduce code complexity • Use local variables • Redundant calculations • Check previous record • Provide comments (-.

Transformation Tips User-Defined Functions • Build complex expressions and reuse them within repository • Two Types: • Public: Callable from any transformation expression • Private: Only callable from another userdefined function • Include any valid function except aggregate functions • Can export to XML Files 26 .

rows will go to each path where the criteria is TRUE 27 .Transformation Tips Filters/Routers • Consider Source Qualifier with a filter to limit rows within relational sources • Filter as close to the source as possible • Replace multiple filters with a router • Pertaining to routers.

Transformation Tips Aggregators • Use sorted input to decrease use of aggregate caches • Limit connected input/output or output port • Filter data before aggregating • Use as early as possible Joiners • Perform joins in Source Qualifier when possible • Limit use to heterogeneous and flat file sources • Perform normal joins when possible • Join sorted input when possible • Designate the master source as the source with fewer rows 28 .

avoid when possible Can apply Parameters and Variables Can query against multiple tables in same database Suppress ORDER BY statement by appending two dashes (--) • Add indexes to database columns • Replace large lookup tables with joins in the Source Qualifier when possible • Relational Lookups should only return ports that meet the condition • Remove all ports not used downstream or in the SQL Override 29 .Transformation Tips Lookups • Using SQL Override in Lookup • • • • Similar to Source Qualifier.

. files • Stores output values in data. .dat.idx. files • Apply the Cache Calculator in Session 30 • .Transformation Tips Lookup Caches • Lookup Cache Types • Persistent Caches • Save lookup cache files for reuse • Dynamic Caches • Retains the latest changes to data as rows pass through the mapping • Updating a master table • Real-time sessions • Slowly changing dimension Cache Sizes • Eliminate Paging • Stores condition values in index.

Transformation Tips Lookups • Cache Updates • Update the dynamic lookup cache with results of an expression Use Case: Update QTY on hand for new timestamp Add WHERE incoming row timestamp > cached timestamp • SQL Overrides for Uncached Lookups • You must choose the Use Any Value on Lookup Policy on Multiple Match condition • Multiple Rows Return Use Case: Aggregate customer orders and store the total value • Database Deadlock Resilience • NumOfDeadlockRetries • DeadlockSleep 31 .

Transformation Tricks Pipeline Lookup • Perform a lookup on an application source that is not a relational table or file • Partial pipeline contains Source & Source Qualifier but no target • Integration Service reads source data and passes to Lookup Transformation to create cache • Create partitions to improve performance 32 .

every flush of the write buffers is logged 33 .Transformation Tips Transaction Control Transformation • Transaction in PowerCenter is a set of rows bound by commit or rollback • Control commit and rollback transactions based on a row or set of rows that pass through the transformation Use Case: Each invoice number is committed to the target database as a single transaction • Change Tracing Level to ‘Terse’ • At higher tracing levels.

Transformation Tips Associated Source Qualifier • Use ASQ when MQ data is flat file or COBOL • ASQ is specific to the format of the MQ data 34 .

Transformation Tips Sequence • • • • • Non-Reusable the counter is 0 Performance will be affected if cached is low Increase of caching will improve the performance This doesn’t involve any database operation The caching allows to reserve number of rows in the memory 35 .

File Source and Target Commands 36 .

• Service process variables ($PMSourceFileDir) can be used in the command. 37 .File Source and Target Commands Commands for File Sources • Use a command to generate flat file source data input rows or file list or a session • Unix – any valid UNIX command or shell script • Windows – any valid DOS or batch file.

38 .File Source Command Generating a File List • Input Type: Command (default: file) • Command Type: Command Generating File List • Command writes list of file names to stdout • PowerCenter interprets this as a file list.

Z 39 . reading compressed files • uncompress –c $PMSourceFileDir/myCompressedFile.File Source Command Generating Source Data • Input Type: Command (default: file) • Command Type: Command Generating Data • Command generates rows to stdout • Flat file reader reads directly from stdout • Removes need for staging data • Example use.

Z • Sorting output data 40 .> $PMTargetFileDir/myCompressedFile.File Target Command Processing Target Data • Output Type: Command (default: file) • Flat file writer writes to the command • Writing compressed files • compress -c .

Filename Port Source Filename • Input Filename can be processed and passed to target 41 .

Filename Port Target Filename • Write records to a dynamically named flat file 42 .

Change data detection 43 .

Change Detection for Updates MD5 or CRC32 • Challenge: a record with a lot of columns needs to be checked for changes • Solution: calculate an MD5 checksum on the columns and use a lookup to compare the value with any existing record 44 .

store MD5 value in target 45 .Sample Change data detection • Calculate MD5 for all columns except key • Create lookup for primary key and MD5 value • Perform insert/update.

Use of Metadata 46 .

Querying the PowerCenter repository • Query in designer • Limit querying on OPB tables • Use the MX views instead • Utilize Reporting Service • Use Meta Query tool • Use Batch Web Services 47 .

Reporting Service Dashboard 48 .

Repository Maintenance 49 .

QA and Prod • Archieve if required for future analysis • Purge unwanted versions • Run the purge in regular interval daily.Repository Maintenance Purge repository versions • Define version strategy for Dev. weekly or monthly pmrep connect -r $REPOSITORY_NAME -d $DOMAIN_NAME -n $ADMIN_USER -X INFA_ENCRYPTED_PASSWD pmrep purgeversion -n $VERSIONS_TO_KEEP -o $FILE_NAME 50 .

Repository Maintenance Purge repository logs • Define log strategy for Dev. QA and Prod • Archieve if required for future analysis • Purge unwanted logs • Run the purge in regular interval daily. weekly or monthly pmrep connect -r $REPOSITORY_NAME -d $DOMAIN_NAME -n $ADMIN_USER -X INFA_ENCRYPTED_PASSWD pmrep truncatelog -t $DAYS_TO_KEEP Compute statistics on metadata tables pmrep updatestatistics 51 .

informatica.com/ • http://marketplace.com • Product manuals • Informatica Professional Services 52 .com • http://velocity..informatica.informatica..Additional Informatica Resources Refer the following. • http://mysupport.

Questions? 53 .

Sign up to vote on this title
UsefulNot useful