Enhancements in Powercenter 8.6 to 8.

5
Informatica has released its latest version 8.6 covering all the hot fixes it released for the prior version 8.5 and including few new features. Since version 8, a Unified Admin Console has een designed for managing Integration and !epositor" services. #hese were discussed in earlier $logs. %hat does &owerCenter 8.6 ring new for the developers' (et us discuss &owerCenter 8.6 Client enhancements which will e useful to the developers.

1.

Creating Targets from Transformations

%e can create targets ased on transformations in the wor)space or navigator. #o create a target, *. !ight+clic) the transformation in the wor)space and select the Create and Add #arget option. ,. Alternativel", we can drag and drop the transformation in the #arget -esigner. #he target that is created has the same port definitions as the transformation from which it was created. %e can edit the target definitions later. In addition, the target t"pe is the same as that of the repositor" used.

2.

Invalid/Invalidated renamed

In &owerCenter ., the two states of o /ects were )nown as Invalid and Invalidated. #he exact meaning of these states is as follows0 Invalid 1 an o /ect will not run, Invalidated 1 an o /ect ma" e invalid or ma" not run. #he difference etween the two terms was not ver" clear. #herefore, to avoid an" confusion, in &owerCenter 8.6, the two states have een renamed as Invalid and Impacted. %hile the Invalid state still implies that an o /ect will not run, Impacted means that an o /ect is affected " a change, and therefore, ma" not run. Apart from the naming convention the icons are also changed in &owerCenter 8.

Propagating Port !escriptions

In the -esigner, in addition to the other properties of port propagation, we can edit a port description and propagate the description to other transformations in the mapping.

".

Environment #$% Enhancements

In &owerCenter 8, environment S2( can e used to execute an S2( statement at the eginning of each transaction. #he Integration Service executes transaction environment S2( at the eginning of each transaction. 3nvironment S2( can still e used to execute an S2( statement at each connection to the data ase. Use S2( commands that depend upon a transaction eing opened during the entire read or write process. 4or example, the following S2( command modifies how the session handles characters0 &%TE' #E##I() #ET )%#*%E)+T,*#E-&)TIC#.C,&'

5.

/lat /ile Enhancements

&owerCenter 8 includes enhancements for handling flat files. Some of these improve performance. 4lat files can now use Integer or -ou le data t"pes. In addition, target partitions can e merged.

2. PowerCenter 'epositor0
#he &owerCenter !epositor" is one of est metadata storage among all 3#( products. #he repositor" is sufficientl" normali5ed to store metadata at a ver" detail level6 which in turn means the Updates to the repositor" are ver" 7uic) and the overall #eam+ ased -evelopment is smooth. #he repositor" data structure is also useful for the users to do anal"sis and reporting.

Accessi ilit" to the repositor" through 89 views and S-: )it extends the repositories capa ilit" from a simple storage of technical data to a data ase for anal"sis of the 3#( metadata. &owerCenter !epositor" is a collection of ;55 ta les which can e created on an" ma/or relational data ase. #he )inds of information that are stored in the repositor" are, *. !epositor" configuration details ,. 8appings ;. %or)flows <. User Securit" 5. &rocess -ata of session runs 4or a 7uic) understanding, %hen a user creates a folder, corresponding entries are made into ta le =&$>SU$?3C#6 attri utes li)e folder name, owner id, t"pe of the folder li)e shared or not are all stored. %hen we create@import sources and define field names, datat"pes etc in source anal"5er entries are made into op >src and =&$>S!C>4(-. %hen target and related fields are createdAimported from an" data ase entries are made into ta les li)e =&$>#A!B and =&$>#A!B>4(-. #a le =&$>8A&&ICB stores mapping attri utes li)e 8apping Came, 4older Id, Dalid status and mapping comments. #a le =&$>%I-B3# stores attri utes li)e widget t"pe, widget name, comments etc. %idgets are nothing ut the #ransformations which Informatica internall" calls them as %idgets. #a le =&$>S3SSI=C stores configurations related to a session tas) and ta le =&$>CC9>A##! stores information related to connection o /ects. #a le =&$>%4(=%>!UC stores process details li)e wor)flow name, wor)flow started time, wor)flow completed time, server node it ran etc. !3&>A((>S=U!C3S, !3&>A((>#A!B3#S and !3&>A((>8A&&ICBS are few of the man" views created over these ta les.

&owerCenter applications access the &owerCenter repositor" through the !epositor" Service. #he !epositor" Service protects metadata in the repositor" " managing repositor" connections and using o /ect+loc)ing to ensure o /ect consistenc". %e can create a repositor" as glo al or local. %e can go forEglo alF to store common o /ects that multiple developers can use through shortcuts and go for local repositor" to perform of development mappings and wor)flows. 4rom a local repositor", we can create shortcuts to o /ects in shared folders in the glo al repositor". &owerCenter supports versioning. A versioned repositor" can store multiple versions of an o /ect.

. &dministration Console #he Administration Console is a we application that we use to administer the &owerCenter domain and &owerCenter securit". #here are two pages in the console, -omain &age G Securit" &age. 1e can do the following In !omain Page2 o Create G manage application services li)e Integration Service and !epositor" Service o Create and manage nodes, licenses and folders o !estart and shutdown nodes o Diew log events o =ther domain management tas)s li)e appl"ing licenses and managing grids and resources 1e can do the following in #ec3rit0 Page2 o Create, edit and delete native users and groups o Configure a connection to an (-A& director" service. Import users and groups from the (-A& director" service o Create, edit and delete !oles H!oles are collections of privilegesI o Assign roles and privileges to users and groups

o Create, edit, and delete operating s"stem profiles. An operating s"stem profile is a level of securit" that the Integration Services uses to run wor)flows ". PowerCenter Client -esigner, %or)flow 8anager, %or)flow 8onitor, !epositor" 8anager G -ata Stencil are five client tools that are used to design mappings, 8applets, create sessions to load data and manage repositor". 8apping is an 3#( code pictoriall" depicting logical data flow from source to target involving transformations of the data. -esigner is the tool to create mappings -esigner has five window panes, Source Anal"5er, %arehouse -esigner, #ransformation -eveloper, 8apping -esigner and 8applet -esigner. #o3rce &nal04er2 Allows us to import Source ta le metadata from !elational data ases, flat files, 98( and C=$=( files. %e can onl" import the source definition in the source Anal"5er and not the source data itself is to e understood. Source Anal"5er also allows us to define our own Source data definition. 1areho3se !esigner2 Allows us to import target ta le definitions which could e !elational data ases, flat files, 98( and C=$=( files. %e can also create target definitions manuall" and can group them into folders. #here is an option to create the ta les ph"sicall" in the data ase that we do not have in source anal"5er. %arehouse designer doesnFt allow creating two ta les with same name even if the columns names under them var" or the" are from different data asesAschemas. Transformation !eveloper2 #ransformations li)e 4ilters, (oo)ups, 3xpressions etc that have scope to e re+used are developed in this pane. Alternativel" #ransformations developed in 8apping -esigner can also e reused " chec)ing the optionEre+useF and " that it would e displa"ed under #ransformation -eveloper folders. -apping !esigner2 #his is the place where we actuall" depict our 3#( process6 we ring in source definitions, target definitions, transformations li)e filter, loo)up, aggregate and

develop a logical 3#( program. In this place it is onl" a logical program ecause the actual data load can e done onl" " creating a session and wor)flow. -applet !esigner2 %e create a set of transformations to e used and re+used across mappings. ". PowerCenter Client 5contd6 1or7flow -anager 2 In the %or)flow 8anager, we define a set of instructions called a wor)flow to execute mappings we uild in the -esigner. Benerall", a wor)flow contains a session and an" other tas) we ma" want to perform when we run a session. #as)s can include a session, email notification, or scheduling information.

A set of tas)s grouped together ecomes wor)let. After we create a wor)flow, we run the wor)flow in the %or)flow 8anager and monitor it in the %or)flow 8onitor. %or)flow 8anager has following three window panes,Task Developer, Create tas)s we want to accomplish in the wor)flow. Worklet Designer, Create a wor)let in the %or)let -esigner. A wor)let is an o /ect that groups a set of tas)s. A wor)let is similar to a wor)flow, ut without scheduling information. Jou can nest wor)lets inside a wor)flow. Workflow Designer, Create a wor)flow " connecting tas)s with lin)s in the %or)flow -esigner. %e can also create tas)s in the %or)flow -esigner as "ou develop the wor)flow. #he =-$C connection details are defined in %or)flow 8anager KConnections K 8enu . 1or7flow -onitor 2 %e can monitor wor)flows and tas)s in the %or)flow 8onitor. %e can view details a out a wor)flow or tas) in Bantt Chart view or #as) view. %e can run, stop, a ort, and resume wor)flows from the %or)flow 8onitor. %e can view sessions and wor)flow log events in the %or)flow 8onitor (og Diewer. #he %or)flow 8onitor displa"s wor)flows that have run at least once. #he %or)flow 8onitor continuousl" receives information from the Integration Service and !epositor" Service. It also fetches information from the repositor" to displa" historic information. #he %or)flow 8onitor consists of the following windows0

Cavigator window 1 -ispla"s monitored repositories, servers, and repositories o /ects. =utput window 1 -ispla"s messages from the Integration Service and !epositor" Service. #ime window 1 -ispla"s progress of wor)flow runs. Bantt chart view 1 -ispla"s details a out wor)flow runs in chronological format. #as) view 1 -ispla"s details a out wor)flow runs in a report format. 'epositor0 -anager %e can navigate through multiple folders and repositories and perform asic repositor" tas)s with the !epositor" 8anager. %e use the !epositor" 8anager to complete the following tas)s0 ,. Add and connect to a repositor", we can add repositories to the Cavigator window and client registr" and then connect to the repositories. ;. %or) with &owerCenter domain and repositor" connections, we can edit or remove domain connection information. %e can connect to one repositor" or multiple repositories. %e can export repositor" connection information from the client registr" to a file. %e can import the file on a different machine and add the repositor" connection information to the client registr". <. Change "our password. %e can change the password for our user account. 5. Search for repositor" o /ects or )e"words. %e can search for repositor" o /ects containing specified text. If we add )e"words to target definitions, use a )e"word to search for a target definition. 6. Diew o /ects dependencies. $efore we remove or change an o /ect, we can view dependencies to see the impact on other o /ects.

.. Compare repositor" o /ects. In the !epositor" 8anager, wean compare two repositor" o /ects of the same t"pe to identif" differences etween the o /ects. 8. #runcate session and wor)flow log entries. we can truncate the list of session and wor)flow logs that the Integration Service writes to the repositor". we can truncate all logs, or truncate all logs older than a specified date. Informatica PowerCenter 88 9e0 Concepts : 5 $" ;adri )ara0anan on ?anuar" *6th, ,LLM under Informatica 1a0. 5. 'epositor0 #ervice As we alread" discussed a out metadata repositor", now we discuss a separate,multi+ threaded process that retrieves, inserts and updates metadata in the repositor" data ase ta les, it is !epositor" Service. !epositor" service manages connections to the &owerCenter repositor" from &owerCenter client applications li)e -esigner, %or)flow 8anager, 8onitor, !epositor" manager, console and integration service. !epositor" service is responsi le for ensuring the consistenc" of metadata in the repositor". Creation < Properties2 Use the &owerCenter Administration Console Cavigator window to create a !epositor" Service. #he properties needed to create are, Service Came 1 name of the service like rep_SalesPerformanceDev (ocation 1 Domain and folder where the service is created (icense 1 license service name Code, &rimar" Code G $ac)up Codes 1 Node on which the service process runs Code&age 1 The Repository Service uses the character set encoded in the repository code page when writing data to the repository -ata ase t"pe G details 1 Type of database username pwd connect string and tablespacename

#he a ove properties are sufficient to create a repositor" service, however we can ta)e a loo) at following features which are important for etter performance and maintenance. +eneral Properties N =perating8ode0 Dalues are Cormal and 3xclusive. Use 3xclusive mode to perform administrative tas)s li)e ena ling version control or promoting local to glo al repositor" N 3na leDersionControl0 Creates a versioned repositor" )ode &ssignments2 KOigh availa ilit" optionP is licensed feature which allows us to choose &rimar" G $ac)up nodes for continuous running of the repositor" service. Under normal licenses would see onl" onl" Code to select from !ata=ase Properties N -ata aseArra"=perationSi5e0 Cum er of rows to fetch each time an arra" data ase operation is issued, such as insert or fetch. -efault is *LL N -ata ase&oolSi5e08aximum num er of connections to the repositor" data ase that the !epositor" Service can esta lish. If the !epositor" Service tries to esta lish more connections than specified for -ata ase&oolSi5e, it times out the connection attempt after the num er of seconds specified for -ata aseConnection#imeout &dvanced Properties N Comments!e7uired4or Chec)in0 !e7uires users to add comments when chec)ing in repositor" o /ects. N 3rror Severit" (evel0 (evel of error messages written to the !epositor" Service log. Specif" one of the following message levels0 4atal, 3rror, %arning, Info, #race G -e ug N 3na le!epAgentCaching03na les repositor" agent caching. !epositor" agent caching provides optimal performance of the repositor" when "ou run wor)flows. %hen "ou ena le repositor" agent caching, the !epositor" Service process caches metadata re7uested " the Integration Service. -efault is Jes. N !ACacheCapacit"0Cum er of o /ects that the cache can contain when repositor" agent caching is ena led. Jou can increase the num er of o /ects if there is availa le memor"

on the machine running the !epositor" Service process. #he value must e etween *LL and *L,LLL,LLL,LLL. -efault is *L,LLL N Allow%rites%ith!ACaching0 Allows "ou to modif" metadata in the repositor" when repositor" agent caching is ena led. %hen "ou allow writes, the !epositor" Service process flushes the cache each time "ou save metadata through the &owerCenter Client tools. Jou might want to disa le writes to improve performance in a production environment where the Integration Service ma)es all changes to repositor" metadata. -efault is Jes. Environment >aria=les #he data ase client code page on a node is usuall" controlled " an environment varia le. 4or example, =racle uses C(S>(ACB, and I$8 -$, uses -$,C=-3&AB3. All Integration Services and !epositor" Services that run on this node use the same environment varia le. Jou can configure a !epositor" Service process to use a different value for the data ase client code page environment varia le than the value set for the node. Jou might want to configure the code page environment varia le for a !epositor" Service process when the !epositor" Service process re7uires a different data ase client code page than the Integration Service process running on the same node. 4or example, the Integration Service reads from and writes to data ases using the U#4+8 code page. #he Integration Service re7uires that the code page environment varia le e set to U#4+8. Oowever, "ou have a Shift+?IS repositor" that re7uires that the code page environment varia le e set to Shift+?IS. Set the environment varia le on the node to U#4+8. #hen add the environment varia le to the !epositor" Service process properties and set the value to Shift+?IS. 6. Integration #ervice 5I#6 #he )e" functions of IS are

Interpretation of the wor)flow and mapping metadata from the repositor". 3xecution of the instructions in the metadata 8anages the data from source s"stem to target s"stem within the memor" and dis)

#he main three components of Integration Service which ena le data movement are,

Integration Service &rocess (oad $alancer -ata #ransformation 8anager

6.1 Integration #ervice Process 5I#P6 #he Integration Service starts one or more Integration Service processes to run and monitor wor)flows. %hen we run a wor)flow, the IS& starts and loc)s the wor)flow, runs the wor)flow tas)s, and starts the process to run sessions. #he functions of the Integration Service &rocess are,

(oc)s and reads the wor)flow 8anages wor)flow scheduling, ie, maintains session dependenc" !eads the wor)flow parameter file Creates the wor)flow log !uns wor)flow tas)s and evaluates the conditional lin)s Starts the -#8 process to run the session %rites historical run information to the repositor" Sends post+session emails %oad ;alancer

6.2

#he (oad $alancer dispatches tas)s to achieve optimal performance. It dispatches tas)s to a single node or across the nodes in a grid after performing a se7uence of steps. $efore understanding these steps we have to )now a out !esources, !esource &rovision #hresholds, -ispatch mode and Service levels

'eso3rces 1 we can configure the Integration Service to chec) the resources availa le on each node and match them with the resources re7uired to run the tas). 4or example, if a session uses an SA& source, the (oad $alancer dispatches the session onl" to nodes where the SA& client is installed

Three 'eso3rce Provision Thresholds, #he maximum num er of runna le threads waiting for C&U resources on the node called 8aximum C&U !un 2ueue (ength. #he maximum percentage of virtual memor" allocated on the node relative to the total ph"sical memor" si5e called 8aximum 8emor" Q. #he maximum num er of running Session and Command tas)s allowed for each Integration Service process running on the node called 8aximum &rocesses

Three !ispatch mode?s : !ound+!o in0 #he (oad $alancer dispatches tas)s to availa le nodes in a round+ro in fashion after chec)ing the K8aximum &rocessP threshold. 8etric+ ased0 Chec)s all the three resource provision thresholds and dispatches tas)s in round ro in fashion. Adaptive0 Chec)s all the three resource provision thresholds and also ran)s nodes according to current C&U availa ilit"

#ervice %evels esta lishes priorit" among tas)s that are waiting to e dispatched, the three components of service levels are Came, -ispatch &riorit" and 8aximum dispatch wait time. K8aximum dispatch wait timeP is the amount of time a tas) can wait in 7ueue and this ensures no tas) waits forever

A .-ispatching #as)s on a node *. #he (oad $alancer chec)s different resource provision thresholds on the node depending on the -ispatch mode set. If dispatching the tas) causes an" threshold to e exceeded, the (oad $alancer places the tas) in the dispatch 7ueue, and it dispatches the tas) later ,. #he (oad $alancer dispatches all tas)s to the node that runs the master Integration Service process $. -ispatching #as)s on a grid, *. #he (oad $alancer verifies which nodes are currentl" running and ena led

,. #he (oad $alancer identifies nodes that have the &owerCenter resources re7uired " the tas)s in the wor)flow ;. #he (oad $alancer verifies that the resource provision thresholds on each candidate node are not exceeded. If dispatching the tas) causes a threshold to e exceeded, the (oad $alancer places the tas) in the dispatch 7ueue, and it dispatches the tas) later <. #he (oad $alancer selects a node ased on the dispatch mode 6. !ata Transformation -anager 5!T-6 Process %hen the wor)flow reaches a session, the Integration Service &rocess starts the -#8 process. #he -#8 is the process associated with the session tas). #he -#8 process performs the following tas)s0

!etrieves and validates session information from the repositor". Dalidates source and target code pages. Derifies connection o /ect permissions. &erforms pushdown optimi5ation when the session is configured for pushdown optimi5ation. Adds partitions to the session when the session is configured for d"namic partitioning. 3xpands the service process varia les, session parameters, and mapping varia les and parameters. Creates the session log. !uns pre+session shell commands, stored procedures, and S2(. Sends a re7uest to start wor)er -#8 processes on other nodes when the session is configured to run on a grid. Creates and runs mapping, reader, writer, and transformation threads to extract, transform, and load data !uns post+session stored procedures, S2(, and shell commands and sends post+ session email After the session is complete, reports execution result to IS&

&ictorial !epresentation of %or)flow execution0

*. A &owerCenter Client re7uest IS to start wor)flow ,. IS starts IS& ;. IS& consults ($ to select node <. IS& starts -#8 in node selected " ($.