4/2/2015

InformaticaDynamicLookupCache

InformaticaDynamicLookupCache
WrittenbySauravMitra

LastUpdated:17September2014

A LookUp cache does not change its data once built. But what if the underlying table upon
whichlookupwasdonechangesthedataafterthelookupcacheiscreated?Isthereawayso
thatthecachealwaysremainuptodateeveniftheunderlyingtablechanges?

WhydoweneedDynamicLookupCache?
Let'sthinkaboutthisscenario.Youareloadingyourtargettablethroughamapping.Insidethe
mapping you have a Lookup and in the Lookup, you are actually looking up the same target
tableyouareloading.
Youmayaskme,"So?What'sthebigdeal?Wealldoitquiteoften...".Andyesyouareright.
There is no "big deal" because Informatica (generally) caches the lookup table in the very
beginning of the mapping, so whatever record getting inserted to the target table through the
mapping,willhavenoeffectontheLookupcache.
The lookup will still hold the previously cached data, even if the underlying target table is
changing.
ButwhatifyouwantyourInformaticaLookupcachetogetupdatedasandwhenthedata
intheunderlyingtargettablechanges?

What if you want your lookup cache to always show the exact snapshot of the data in your
targettableatthatpointintime?Clearlythisrequirementwillnotbefullfilledincaseyouuse
astaticcache.Youwillneedadynamiccachetohandlethis.
Butinwhichscenariowillsomeoneneedtouseadynamiccache?Tounderstandthis,let'sfirst
understandastaticcachescenario.

StaticLookupCacheScenario
Let's suppose you run a retail business and maintain all your customer information in a
customer master table (RDBMS table). Every night, all the customers from your customer
mastertableisloadedintoaCustomerDimensiontableinyourdatawarehouse.Yoursource
customertableisatransactionsystemtable,probablyin3rdnormalform,anddoesnotstore
history. Meaning, if a customer changes his address, the old address is updated with the new
address.
But your data warehouse table stores the history (may be in the form of SCD TypeII). There
is a map that loads your data warehouse table from the source table. Typically you do a
Lookupontarget(staticcache)andcheckwitheveryincomingcustomerrecordtodetermineif
the customer is already existing in target or not. If the customer is not already existing in
target, you conclude the customer is new and INSERT the record whereas if the customer is
alreadyexisting,youmaywanttoupdatethetargetrecordwiththisnewrecord(iftherecord
is updated). This scenario commonly known as 'UPSERT' (update else insert) scenario is
illustratedbelow.
http://dwbi.org/etl/informatica/138dynamiclookupcache

1/9

4/2/2015

InformaticaDynamicLookupCache

AstaticLookupCachetodetermineifasourcerecordisneworupdatable
Youdon'tneeddynamicLookupcachefortheabovetypeofscenario.

DynamicLookupCacheScenario
Notice in the previous example I mentioned that your source table is an RDBMS table.
Generallyspeaking,thisensuresthatyoursourcetabledoesnothaveanyduplicaterecord.
But, What if you had a flat file as source with many duplicate records in the same bunch of
datathatyouaretryingtoload?(OrevenaRDBMStablemayalsocontainduplicaterecords)
WouldthescenariobesameifthebunchofdataIamloadingcontainsduplicate?

UnfortunatelyNot.Let'sunderstandwhyfromthebelowillustration.Asyoucanseebelow,the
new customer "Linda" has been entered twice in the source system most likely mistakenly.
The customer "Linda" is not present in your target system and hence does not exist in the
targetsidelookupcache.
When you try to load the target table, Informatica processes row 3 and inserts it to target as
customer"Linda"doesnotexistintarget.ThenInformaticaprocessesrow4andagaininserts
"Linda" into target since Informatica lookup's static cache can not detect that the customer
"Linda"hasalreadybeeninserted.Thisresultsintoduplicaterowsintarget.

Theproblemarisingfromabovescenariocanberesolvedbyusingdynamiclookupcache
Herearesomemoreexampleswhenyoumayconsiderusingdynamiclookup,
Updating a master customer table with both new and updated customer information
comingtogetherasshownabove
Loading data into a slowly changing dimension table and a fact table at the same time.
Remember, you typically lookup the dimension while loading to fact. So you load
http://dwbi.org/etl/informatica/138dynamiclookupcache

2/9

4/2/2015

InformaticaDynamicLookupCache

dimension table before loading fact table. But using dynamic lookup, you can load both
simultaneously.
Loading data from a file with many duplicate records and to eliminate duplicate records
intargetbyupdatingaduplicaterowi.e.keepingthemostrecentrowortheinitialrow
Loading the same data from multiple sources using a single mapping. Just consider the
previousRetailbusinessexample.IfyouhavemorethanoneshopsandLindahasvisited
two of your shops for the first time, customer record Linda will come twice during the
sameload.

Howdoesdynamiclookupcachework
Once you have configured your lookup to use dynamic cache (we will see below how to do
that), when Integration Service reads a row from the source, it updates the lookup cache by
performingoneofthefollowingactions:
Insertstherowintothecache:Iftheincomingrowisnotinthecache,theIntegration
ServiceinsertstherowinthecachebasedoninputportsorgeneratedSequenceID.The
IntegrationServiceflagstherowasinsert.
Updatestherowinthecache: If the row exists in the cache, the Integration Service
updatestherowinthecachebasedontheinputports.TheIntegrationServiceflags the
rowasupdate.
Makesnochangetothecache:Thishappenswhentherowexistsinthecacheandthe
lookupisconfiguredorspecifiedToInsertNewRowsonlyor,therowisnotinthecache
and lookup is configured to update existing rows only or, the row is in the cache, but
basedonthelookupcondition,nothingchanges.TheIntegrationServiceflagstherowas
unchanged.
NoticethatIntegrationServiceactuallyflagstherowsbasedontheabovethreeconditions.
Andthat'sagreatthing,because,ifyouknowtheflagyoucanactuallyreroutetherowto
achievedifferentlogic.

Fortunately, as soon as you create a dynamic lookup Informatica adds one extra port to the
lookup.Thisnewportiscalled:
NewLookupRow
Using the value of this port, the rows can be routed for insert, update or to do nothing. You
justneedtouseaRouterorFiltertransformationfollowedbyanUpdateStrategy.
Oh,forgottotellyoutheactualvaluesthatyoucanexpectinNewLookupRowportare:
0=IntegrationServicedoesnotupdateorinserttherowinthecache.
1=IntegrationServiceinsertstherowintothecache.
2=IntegrationServiceupdatestherowinthecache.
When the Integration Service reads a row, it changes the lookup cache depending on the
resultsofthelookupqueryandtheLookuptransformationpropertiesyoudefine.Itassignsthe
value 0, 1, or 2 to the NewLookupRow port to indicate if it inserts or updates the row in the
cache,ormakesnochange.

http://dwbi.org/etl/informatica/138dynamiclookupcache

3/9

4/2/2015

InformaticaDynamicLookupCache

ConfiguringaDynamicLookupMappingExample
Ok, I design a mapping for you to show Dynamic lookup implementation. I have given a full
screenshotofthemapping.Sincethescreenshotisslightlybigger,soIlinkitbelow.Justclick
toexpandtheimage.

http://dwbi.org/etl/informatica/138dynamiclookupcache

4/9

4/2/2015

InformaticaDynamicLookupCache

If you check the mapping screenshot, there I have used a router to reroute the INSERT group
and UPDATE group. The router screenshot is also given below. New records are routed to the
INSERTgroupandexistingrecordsareroutedtotheUPDATEgroup.

http://dwbi.org/etl/informatica/138dynamiclookupcache

5/9

4/2/2015

InformaticaDynamicLookupCache

DynamicLookupSequenceID
While using a dynamic lookup cache, we must associate each lookup/output port with an
input/output port or a sequence ID. The Integration Service uses the data in the associated
port to insert or update rows in the lookup cache. The Designer associates the input/output
portswiththelookup/outputportsusedinthelookupcondition.
WhenweselectSequenceIDintheAssociatedPortcolumn,theIntegrationServicegenerates
asequenceIDforeachrowitinsertsintothelookupcache.
When the Integration Service creates the dynamic lookup cache, it tracks the range of values
inthecacheassociatedwithanyportusingasequenceIDanditgeneratesakeyfortheport
byincrementingthegreatestsequenceIDexistingvaluebyone,whentheinsertinganewrow
ofdataintothecache.
When the Integration Service reaches the maximum number for a generated sequence ID, it
starts over at one and increments each sequence ID by one until it reaches the smallest
existing value minus one. If the Integration Service runs out of unique sequence ID numbers,
thesessionfails.

DynamicLookupPorts
The lookup/output port output value depends on whether we choose to output old or new
valueswhentheIntegrationServiceupdatesarow:
Outputoldvaluesonupdate:The Integration Service outputs the value that existed in
thecachebeforeitupdatedtherow.
Outputnewvaluesonupdate:TheIntegrationServiceoutputstheupdatedvaluethatit
writesinthecache.Thelookup/outputportvaluematchestheinput/outputportvalue.
Note:WecanconfiguretooutputoldornewvaluesusingtheOutputOldValueOnUpdate
transformationproperty.

HandlingNULLindynamicLookUp
If the input value is NULL and we select the Ignore Null inputs for Update property for the
associated input port, the input value does not equal the lookup value or the value out of the
input/output port. When you select the Ignore Null property, the lookup cache and the target
tablemightbecomeunsynchronizedifyoupassnullvaluestothetarget.Youmustverifythat
youdonotpassnullvaluestothetarget.
When you update a dynamic lookup cache and target table, the source data might contain
somenullvalues.TheIntegrationServicecanhandlethenullvaluesinthefollowingways:
Insertnullvalues:TheIntegrationServiceusesnullvaluesfromthesourceandupdates
thelookupcacheandtargettableusingallvaluesfromthesource.
Ignore Null inputs for Update property : The Integration Service ignores the null
values in the source and updates the lookup cache and target table using only the not
nullvaluesfromthesource.

http://dwbi.org/etl/informatica/138dynamiclookupcache

6/9

4/2/2015

InformaticaDynamicLookupCache

Ifweknowthesourcedatacontainsnullvalues,andwedonotwanttheIntegrationServiceto
update the lookup cache or target with null values, then we need to check the Ignore Null
propertyforthecorrespondinglookup/outputport.
WhenwechoosetoignoreNULLs,wemustverifythatweoutputthesamevaluestothetarget
that the Integration Service writes to the lookup cache. We can Configure the mapping based
on the value we want the Integration Service to output from the lookup/output ports when it
updates a row in the cache, so that lookup cache and the target table might not become
unsynchronized.
New values. Connect only lookup/output ports from the Lookup transformation to the
target.
Old values. Add an Expression transformation after the Lookup transformation and
before the Filter or Router transformation. Add output ports in the Expression
transformationforeachportinthetargettableandcreateexpressionstoensurethatwe
donotoutputnullinputvaluestothetarget.

SomeotherdetailsaboutDynamicLookup
When we run a session that uses a dynamic lookup cache, the Integration Service compares
thevaluesinalllookupportswiththevaluesintheirassociatedinputportsbydefault.
It compares the values to determine whether or not to update the row in the lookup cache.
Whenavalueinaninputportdiffersfromthevalueinthelookupport,theIntegrationService
updatestherowinthecache.
Butwhatifwedon'twanttocompareallports?
We can choose the ports we want the Integration Service to ignore when it compares ports.
The Designer only enables this property for lookup/output ports when the port is not used in
thelookupcondition.Wecanimproveperformancebyignoringsomeportsduringcomparison.
(Learn how to improve performance of lookup transformation (/etl/informatica/158tuning
informaticalookup)here)
Wemightwanttodothiswhenthesourcedataincludesacolumnthatindicateswhetherornot
the row contains data we need to update. Select the IgnoreinComparisonpropertyfor all
lookupportsexcepttheportthatindicateswhetherornottoupdatetherowinthecache
andtargettable.
Note:WemustconfiguretheLookuptransformationtocompareatleastoneportelsethe
IntegrationServicefailsthesessionwhenweignoreallports.

Prev(/etl/informatica/139activelookuptransformation)
Next(/etl/informatica/137thebenefitanddisadvantageofinformaticapersistentcache
lookup)

Popular
Top20SQLInterviewQuestionswithAnswers(/database/sql/72top20sqlinterview
questionswithanswers)
http://dwbi.org/etl/informatica/138dynamiclookupcache

7/9

4/2/2015

InformaticaDynamicLookupCache

BestInformaticaInterviewQuestions&Answers(/etl/informatica/131importantpractical
interviewquestions)
Top50DataWarehousingInterviewQuestionswithAnswers(/datamodelling/dimensional
model/58top50dwbiinterviewquestionswithanswers)
Top50DWBIInterviewQuestionswithAnswersPart2(/datamodelling/dimensional
model/59top50dwbiinterviewquestionswithanswerspart2)
The101GuidetoDimensionalDataModeling(/datamodelling/dimensionalmodel/1
dimensionalmodelingguide)
SQLDataQueryLanguage(/database/sql/43sqldql)

AlsoRead
HowtouseLookupandJoininSAPDataServices(/etl/sapdataservices/91lookupandjoin
insapdataservices)
HowtoimplementSCDType3inDataServices(/etl/sapdataservices/105howto
implementscdtype3indataservices)
SSISAggregateTransform(/etl/ssis/126ssisaggregatetransform)
SSISLookupTransform(/etl/ssis/128ssislookuptransform)
BestInformaticaInterviewQuestions&Answers(/etl/informatica/131importantpractical
interviewquestions)

Haveaquestiononthissubject?
Feelfreetoaskyourquestiontoourexpertcommunitymembersandgetclarifiedon
yourdoubtsinaminute.Askingaquestionorengaginginameaningfultechnical
discussionisbotheasyandrewarding.Let'sdoitnow.

Askaquestiononthissubject

AreyouonTwitter?
Startfollowingus.Thiswaywewillalwayskeepyouupdatedwithwhat'shappeningin
DataAnalyticscommunity
Follow@dwbic

AboutUs
http://dwbi.org/etl/informatica/138dynamiclookupcache

8/9

4/2/2015

InformaticaDynamicLookupCache

DataWarehousingandBusinessIntelligenceOrganizationAdvancing
BusinessIntelligence
DWBI.orgisaprofessionalinstitutioncreatedandendorsedbyveteranBIandData
Analyticsprofessionalsfortheadvancementofdatadrivenintelligence
JoinUs(/dwbi.org/component/easysocial/login)|Submitanarticle(/contribute)|
ContactUs(/contact)

Copyright
(https://creativecommons.org/licenses/byncsa/4.0/)
Exceptwhereotherwisenoted,contentsofDWBI.ORGbyIntellipLLP(http://intellip.com)
islicensedunderaCreativeCommonsAttributionNonCommercialShareAlike4.0
InternationalLicense.
PrivacyPolicy(/privacy)|TermsofUse(/terms)

Getintouch
(https://www.facebook.com/datawarehousing)
(https://twitter.com/dwbiconcepts)
(https://www.linkedin.com/company/dwbiconcepts)
(https://www.youtube.com/dwbiconcepts)
(https://plus.google.com/b/105042632846858744029)

Security
(https://www.beyondsecurity.com/vulnerabilityscanner
verification/dwbi.org)

http://dwbi.org/etl/informatica/138dynamiclookupcache

9/9