!"#$%&'(!

&)*#+,"&-&
.#+#,/+,&0*"&$#0#&1#!"*(23"&0*!(2,*&
/.45"."+0#0/(+&#+$&4!($2)0/(+&
uavld M Walker
3
rd
lnLernauonal Clua 81C Conference
klev, ukralne - 3
rd
!une 2008
•! uaLa Warehouslna ls a maLure Lechnoloav
–! lor mosL oraanlsauons:
•! lL ls now 6789&!"#&:; Lhev wlll lmplemenL a soluuon
•! Larlv adopLers are bulldlna 2
nd
and 3
rd
aenerauon
soluuons
–! lrom Lhe vendors:
•! 1here ls sumclenL core Lechnoloav for mosL
oraanlsauons Lo develop a vlable soluuon
•! 1here ls enouah lnvesLed ln 8&u for lnnovauon Lo
overcome anv remalnlna lssues
•! vendor consolldauon ls ellmlnauna Lhe weakesL plavers
and developlna lnLearaLed LoolseLs
1uesdav, !une 3, 2008
© 2008 uaLa ManaaemenL & Warehouslna
hup://www.daLamamL.com
Þaae 2
•! 1here ls Lhe compleLe ranae of achlevemenL
–! 1he besL daLa warehouses
•! 1hese are verv, verv successful
•! 1hev are embedded lnLo Lhe buslness process
•! 1hev are consldered a mlsslon crlucal svsLem
–! 1he averaae daLa warehouse
•! uevelopmenLs oûen run over cosL and ume
•! Soluuons are oûen expenslve Lo operaLe and malnLaln
•! Þro[ecL repuLauons are oûen saved bv a small number of verv bla
wlns frequenLlv from an unexpecLed dlrecuon
•! 8uslnesses are falllna Lo explolL Lhe full soluuon and could
slanlñcanLlv lmprove Lhelr reLurn on lnvesLmenL bv manaalna
chanae beuer raLher Lhan anv Lechnoloalcal chanae
–! 1he worsL daLa warehouses
•! nobodv admlLs Lo unsuccessful lmplemenLauons
•! 1he buslness won'L leL ao even Lhouah Lhe svsLem ls cosuna Lhem
monev
1uesdav, !une 3, 2008
© 2008 uaLa ManaaemenL & Warehouslna
hup://www.daLamamL.com
Þaae 3
•! ?our uaLa Warehouse ls <78&=:>>?> of
?our Craanlsauon
–! Þoor buslness processes wlll oûen mean LhaL Lhe
buslness does noL sLore Lhe daLa lL needs
–! Þoor chanae manaaemenL wlLhln Lhe oraanlsauon
wlll resulL ln poor chanae manaaemenL of Lhe daLa
warehouse
–! Þoor daLa quallLv ln Lhe daLa warehouse reßecLs
poor daLa quallLv ln Lhe source svsLems
–! Þoor requlremenLs reßecL Lhe lnablllLv of Lhe
oraanlsauon Lo express whaL lL ls Lrvlna Lo do
1uesdav, !une 3, 2008
© 2008 uaLa ManaaemenL & Warehouslna
hup://www.daLamamL.com
Þaae 4
•! Slnce Lhe daLa warehouse ls a mlrror of Lhe
oraanlsauon lL wlll noL provlde a Lhe soluuon
Lo our currenL buslness lssues
•! lL wlll provlde clarlLv abouL where vour
buslness lssues are and show vou where vou
need Lo acL Lo lmprove Lhlnas
•! lor Lxample: lf vou can'L ldenufv vour producL
hlerarchv from vour source svsLems how can
vou be sure LhaL vour sales people are
eñecuvelv selllna Lhe rlahL producL ?
1uesdav, !une 3, 2008
© 2008 uaLa ManaaemenL & Warehouslna
hup://www.daLamamL.com
Þaae 3
•! uaLa warehouses are consLanLlv ln a sLaLe of ßux
–! 1he cosL of bulld ls small bv comparlson wlLh Lhe
operauonal cosL needed Lo supporL and chanae Lhe
svsLem. 1hls requlres full-ume lona-Lerm resourclna
•! 1here are seven Lvpes of chanae añecuna Lhe
daLa warehouse :
–! user urlven Chanae (LnhancemenLs & 8equlremenLs)
–! LxLernal SvsLems Chanae
–! lnLernal SvsLems Chanae
–! 8lsk ManaaemenL
–! uaLa SLewardshlp
–! AlerL & 1rend MonlLorlna
–! lssue ManaaemenL
1uesdav, !une 3, 2008
© 2008 uaLa ManaaemenL & Warehouslna
hup://www.daLamamL.com
Þaae 6
1uesdav, !une 3, 2008
© 2008 uaLa ManaaemenL & Warehouslna
hup://www.daLamamL.com
Þaae 7
#<&078&3@=8&0:=8&
0
:
=
8
&
!8AB:>8=89<C&
D8>C:?9&EFGG&
#9@HIC:C&
JB:HK&
ELM&
08C<&
/=NH8=89<&
!8AB:>8=89<C&
D8>C:?9&EFGM&
!8AB:>8=89<C&
D8>C:?9&EEOE&
!8AB:>8=89<C&
D8>C:?9&EEOP&
1he dlñerence beLween whaL was
requlred when Lhe developmenL
sLarLed and whaL ls requlred when Lhe
developmenL ls dellvered
Cvercome bv:
a) accepuna and embraclna lL
b) communlcauna wlLh Lhe users so
Lhev undersLand Lhe ume laa
c) dellverlna ln fasL, small lncremenLs
078&
!8AB:>8=89<&
,@N&
•! 1hlnas ouLslde Lhe dlrecL conLrol of Lhe daLa
warehouse Leam e.a. uparades Lo source svsLems
•! Lxample:
–! A cllenL had 3 malnframes, each of whlch could onlv
perform a malnLenance updaLe once a quarLer. lor Lhe
daLa warehouse LhaL meanL 12 chanaes a vear, or one
chanae Lo analvse, deslan, bulld, LesL and lmplemenL everv
monLh
–! 1hev also had 30 unlx plauorms LhaL acLed as sources and
were far less conLrolled - Lhese resulLed ln 100's of
chanaes per monLh Lo Lhe daLa warehouse
•! 1he more conLrolled Lhe operauonal envlronmenLs Lhe
beuer Lhe chanae manaaemenL of Lhe daLa warehouse
envlronmenL
•! A lack of conLrol leads Lo lssues and consequenLlv more
ume spenL dolna lssue manaaemenL
1uesdav, !une 3, 2008
© 2008 uaLa ManaaemenL & Warehouslna
hup://www.daLamamL.com
Þaae 8
•! Chanaes LhaL Lhe daLa warehouse Leam need
Lo make e.a. uparades or re-facLorlna
–! Lxample: noL applvlna a serles of producL
uparades because Lhere ºlsn'L anv ume" means
LhaL aL some polnL Lhe Lhe producL ln quesuon
becomes un-supporLed.
–! lssues LhaL now arlses wlLh Lhe pro[ecL wlll now
requlres a ma[or uparade and Lhese lnevlLablv
come abouL aL a crlucal polnL ln Lhe buslness cvcle
•! Lnsurlna LhaL lnLernal chanaes are planned
means more shorLer planned ouLaaes raLher
Lhan ma[or unplanned svsLems downume
1uesdav, !une 3, 2008
© 2008 uaLa ManaaemenL & Warehouslna
hup://www.daLamamL.com
Þaae 9
•! 8lsk Mluaauon leads Lo chanae
–! Lxamples:
•! Currencv: 1he swlLch from nauonal currencles Lo Lhe
Luro, 1urkev removlna 000,000 from Lhelr currencv,
ukralne removlna 00,000 from Lhelr currencv, eLc.
•! Calendar: ?2k and handllna leap vears
•! 8uslness: Meraers, Acqulsluons, ue-meraers, ulsplosals
•! 8eaulaLorv: 8eaulaLorv requlremenLs
•! 1hese same rlsks wlll also be añecuna Lhe
source svsLems buL aL dlñerenL umes
–! Lxample:
•! uoes a currencv chanae requlre dual sLoraae and musL
lL be chanaed before anv of Lhe source svsLems?
1uesdav, !une 3, 2008
© 2008 uaLa ManaaemenL & Warehouslna
hup://www.daLamamL.com
Þaae 10
Application ETL ETL ETL
Data
Marts
Data
Warehouse
Staging
Area
Application
Database
Change
Request for
application
to improve
Data Quality
on input
D
a
t
a

C
l
e
a
n
s
i
n
g
D
a
t
a

C
l
e
a
n
s
i
n
g
D
a
t
a

P
r
o
f
i
l
i
n
g
M
a
n
u
a
l

D
a
t
a

F
i
x
e
s
M
a
n
u
a
l

D
a
t
a

F
i
x
(
R
a
r
e
l
y
)
A
u
d
i
t
N
e
w

R
u
l
e
s
R
e
s
u
l
t
s
D
a
t
a

P
r
o
f
i
l
i
n
g
R
e
s
u
l
t
s
A
u
d
i
t
N
e
w

R
u
l
e
s
Data Stewards
© 2006 http://www.datamgmt.com
Data Management & Warehousing
Continuous Data Quality Process
Operational Systems,
Data Warehousing and ETL
Data Quality Tools
(Profiling and Cleansing)
Business Process,
Metadata & Results
1uesdav, !une 3, 2008
© 2008 uaLa ManaaemenL & Warehouslna
hup://www.daLamamL.com
Þaae 11
Þrocesses LhaL dellver aood quallLv daLa añecL chanae across Lhe buslness and
push chanae back Lo Lhe source svsLems and Lo Lhe people who creaLe Lhe daLa
1uesdav, !une 3, 2008
© 2008 uaLa ManaaemenL & Warehouslna
hup://www.daLamamL.com
Þaae 12
29<8C<8K&
"05&4@<Q7&
29R<>@:98K&(N8>@<?>C&
2C:9S&3?B>Q8&3IC<8=C&
•! SvsLems monlLored consLanLlv for alerLs and
Lrends ldenufv chanaes LhaL are requlred
•! Lxample from a currenL cllenL:
•! uC MonlLorlna: 10,000 daLa polnLs
•! L1L MonlLorlna: 2,300 daLa polnLs
•! AlerLs bv 8SS feed and SMS messaaes
•! 1he worsL Lvpe of chanae as lL has sllpped
Lhrouah evervLhlna else
•! Crlucal LhaL lL ls manaaed
–! use an lssue Lracklna svsLem
•! Commerclal or Cpen Source (8uazllla/1rac)
–! 8ecord "D"!%0*/+,&
•! Cood svsLems record 1000's of lssues a vear, each of whlch
needs Lo be addressed and closed
•! 8ecomes an lnvaluable source of manaaemenL lnformauon,
Lrends, audlL and accounLablllLv
–! lnLearaLe as much as posslble
•! 1asks, lssues, 8lsks, LnhancemenLs, uefecLs, 1esL cases and
documenLauon all sLored ln one place.
•! lnformauon ln a wlkl and documenLs and code under sLrlcL
verslon conLrol (e.a. svn or cvs)
•! Lxample: hup://pro[ecLs.daLamamL.com/demo
1uesdav, !une 3, 2008
© 2008 uaLa ManaaemenL & Warehouslna
hup://www.daLamamL.com
Þaae 13
•! An undersLandlna of º1echnlcal uebL"
–! Some assessmenL of Lhe cosL and lmpllcauons of acuons
elLher ln Lhe daLa warehouse envlronmenL or upsLream
•! A Leam sLrucLure LhaL encouraaes learnlna
–! 1eams LhaL work LoaeLher reaularlv, roLaLe Lhrouah
funcuons and an envlronmenL LhaL values learnlna from
lL's mlsLakes
•! Well deñned llahL-welahL processes
–! nelLher belna ued up ln a bureaucracv nor an anarchlc
mess. 8aLher Lhe ablllLv Lo operaLe freelv ln a conLrolled
envlronmenL
•! 1ools LhaL allow vou Lo manaae Lhe llahL-welahL
process
–! verslon ConLrol Soûware, LnhancemenL, 1ask, 8lsk and
lssue 1racklna, Cood meanlnaful documenLauon
1uesdav, !une 3, 2008
© 2008 uaLa ManaaemenL & Warehouslna
hup://www.daLamamL.com
Þaae 14
•! 1echnlcal uebL
–! 1he obllaauon LhaL an oraanlzauon lncurs when lL
chooses a deslan or consLrucuon approach LhaL's
expedlenL ln Lhe shorL Lerm buL LhaL lncreases
complexlLv and ls more cosLlv ln Lhe lona Lerm.
–! Can occur accldenLallv (when a mlsLake ls made)
or dellberaLelv (when a consclence declslon ls
made Lo puL someLhlna oñ)
•! 1he secreL Lo a successful lmplemenLauon ls
Lo conunuallv work Lo manaae and reduce Lhe
debL, [usL llke debL ln real llfe
1uesdav, !une 3, 2008
© 2008 uaLa ManaaemenL & Warehouslna
hup://www.daLamamL.com
Þaae 13
•! non uebL
–! leaLure backloa, deferred feaLures, cuL feaLures, eLc. noL all
lncompleLe work ls debL. 1hese aren'L debL, because Lhev don'L requlre
lnLeresL pavmenLs.
•! uebL
–! uebL lncurred unlnLenuonallv usuallv due Lo low quallLv work (Lhe
unlnsured)
–! uebL lncurred lnLenuonallv
•! ShorL-Lerm debL, usuallv lncurred reacuvelv, for Lacucal reasons
–! lndlvlduallv ldenuñable shorLcuLs (Lhe car loan)
–! numerous unv shorLcuLs (credlL card expendlLure)
•! Lona-Lerm debL, usuallv lncurred proacuvelv, for sLraLealc reasons (Lhe
morLaaae)
•! uebL requlres re-pavmenL 6:<7&:9<8>8C<&
–! uelavlna repavmenL alwavs cosLs more
1uesdav, !une 3, 2008
© 2008 uaLa ManaaemenL & Warehouslna
hup://www.daLamamL.com
Þaae 16
8equlremenLs Analvsls ueslan uevelopmenL 1esuna lmplemenLauon Þroducuon
lssues
-!Supporuna mosL
recenLlv
developed code
MalnLenance
-!uolna malnLenance of
exlsuna funcuonal areas
LnhancemenLs
-!ueveloplna new
funcuonal areas
1uesdav, !une 3, 2008
© 2008 uaLa ManaaemenL & Warehouslna
hup://www.daLamamL.com
Þaae 17
•! 1eam ConslsLencv
•! Þumna small aroups
of people LoaeLher ln
reaular Leams
•! 1eam 8oLauon
•! 1he Leam LhaL
ñnlshes an
enhancemenL
becomes Lhe new
lssue Leam
•! 1he lssue Leam move
on Lo do
malnLenance
•! 1he malnLenance
Leam moves on Lo
develop Lhe nexL
enhancemenLs
•! 1eam 8esponslblllLv
•! no '8lame Came' for
mlsLakes buL learnlna
and lmprovlna as a
resulL
1uesdav, !une 3, 2008
© 2008 uaLa ManaaemenL & Warehouslna
hup://www.daLamamL.com
18
LnhancemenLs and malnLenance processes are
requlred Lo ensure Lhe communlcauon of Lhe
oraanlsauon's requlremenLs of Lhe daLa
warehouse and Lo ensure LhaL Lhose
requlremenLs are cosLed and prlorlused ln such
a wav as Lo dellver maxlmum beneñL.
1uesdav, !une 3, 2008
© 2008 uaLa ManaaemenL & Warehouslna
hup://www.daLamamL.com
18
1he lssue manaaemenL process ls requlred Lo
ensure LhaL all lssues found wlLh buslness
lnLelllaence ouLpuLs are capLured and
resolved conslsLenLlv, allowlna for kÞl capLure
and process vlslblllLv for acuve manaaemenL.
1uesdav, !une 3, 2008
© 2008 uaLa ManaaemenL & Warehouslna
hup://www.daLamamL.com
19
1he requlremenLs process ls requlred Lo ensure LhaL Lhe user
requlremenLs are meL eñecuvelv bv creauna a sLaLemenL of whaL ls
requlred and a process for chanalna Lhe deñnluon of whaL ls requlred
1he enhancemenL requesL packaalna process ls requlred Lo ensure
LhaL Lhe scope ls llmlLed Lo user requlremenLs whlch dellver
beneñL aL reasonable cosL and pro[ecL umescales are predlcLable.
1he L1L deslan & bulld process ls requlred Lo ensure LhaL Lhe soluuon ls a
rellable, robusL, accuraLe and performanL daLa loadlna process ls
deslaned and bulld.
1he reporL deslan and bulld process ls requlred Lo ensure LhaL Lhe
soluuon ls a rellable, robusL, accuraLe and easv Lo use reporuna
soluuon ls deslaned and bulld.
1he Lesuna process ls requlred Lo ensure LhaL Lhe soluuon ls a robusL
and accuraLe reßecuon of Lhe requlremenLs, LhaL chanaes and lssues
are handled prompLlv, and performance ls adequaLe.
1uesdav, !une 3, 2008
© 2008 uaLa ManaaemenL & Warehouslna
hup://www.daLamamL.com
20
1he daLa quallLv process ls requlred Lo measure, conLrol and lmprove
Lhe quallLv of daLa held ln Lhe daLa warehouse, reacuna Lo new
lssues qulcklv and ensurlna source svsLems are responslble for Lhe
daLa Lhev provlde.
1he daLa model process ls requlred Lo develop and
malnLaln boLh Lhe loalcal and phvslcal daLa models
aaalnsL seL sLandards, and allaned wlLh meLadaLa.
1he daLa llfecvcle process ls requlred Lo alve sLrucLure Lo
Lhe addlna and removlna daLa from Lhe daLa warehouse,
and malnLalnlna a balance beLween buslness, hardware,
cosL and compllance conslderauons.
1he daLa securlLv process ls requlred Lo ensure LhaL Lhe
oraanlzauon complles wlLh leaal requlremenLs relaLed Lo daLa
proLecuon, and LhaL Lhe daLa warehouse complles wlLh all Lhe
oraanlzauons' lnLernal daLa pollcles.
1he meLadaLa process ls requlred Lo ensure meLadaLa ls
capLured conslsLenLlv as parL of warehouse developmenL, such
LhaL lL can be eñecuvelv used ln appllcauons lncludlna daLa
quallLv checklna, daLa proñllna and daLa securlLv.
!une 3, 08 Þaae 21
•! ueveloplna Lhe processes and ldenufvlna Lhe
rlahL people dellvers daLa warehouse
aovernance and allows chanae Lo be
eñecuvelv manaaed
–! Þeople
•! Covernance ls bullL around people and Lhelr
undersLandlna and wllllnaness Lo operaLe a process. ln
order Lo dellver aovernance we musL deñne Lhe roles
and responslblllues for lndlvlduals and forums ln whlch
lndlvlduals can communlcaLe and work as Leams
–! Þrocesses
•! Þrocesses are used Lo descrlbe how a mechanlsm (e.a.
MeLadaLa, uaLa CuallLv, eLc.) ls dellvered
© 2008 uaLa ManaaemenL & Warehouslna
hup://www.daLamamL.com
!une 3, 08 Þaae 22
•! Þeople
–! 8oles and
8esponslblllues
•! ueñned
responslblllues
•! AccounLablllLv
–! lorums
•! Þurpose for each
forum and lLs
communlcauon
meLhod
•! AuLhorlLv for a forum
Lo make declslons
•! ÞaruclpanLs who
should conLrlbuLe Lo
Lhe forum
•! Þrocesses
–! MeLhodoloales
•! uescrlpuon of Lhe
process
•! Llnks Lo and
compllance wlLh
sLandard processes
•! use of sLandard
documenLauon
–! SLandards
•! 8eference
documenLs for Lhe
conslsLenL use of l1
–! 1ools
•! 1ools Lo supporL
pro[ecLs
•! 1ools Lo supporL
operauonal area
–! Compllance
•! Collecuon and
analvsls of meLrlcs
•! AudlLs of pro[ecLs
© 2008 uaLa ManaaemenL & Warehouslna
hup://www.daLamamL.com
•! 1he daLa warehouse wlll acL as a mlrror of vour oraanlsauon
–! noL evervLhlna vou wlll see ln Lhe mlrror wlll be pleasanL
•! 1he daLa warehouse wlll be a source of consLanL chanae
–! Successfullv compleuna Lhe lnlual bulld ls onlv Lhe ñrsL sLep on a lona
[ournev
•! 1he rlahL aovernance wlll help vou manaae chanae
–! 8uL lneñecuve conLrols wlll lead Lo unconLrollable cosLs LhaL qulcklv
lnLroduce delavs and devalue Lhe soluuon
•! CuallLv ls free - provlded vou are prepared Lo lnvesL ln lL
–! 1he lnvesLmenL requlred Lo dellver quallLv soluuons ls nearlv alwavs
repald bv Lhe beneñLs LhaL quallLv dellvers
–! ShorL Lerm and Lacucal soluuons alwavs run up Lechnlcal debL LhaL
makes Lhe LoLal cosL of ownershlp hlaher ln Lhe lona Lerm
1uesdav, !une 3, 2008
© 2008 uaLa ManaaemenL & Warehouslna
hup://www.daLamamL.com
Þaae 23
•! WhlLe Þapers:
–! hup://www.daLamamL.com
–! Cvervlew ArchlLecLure for LnLerprlse uaLa
Warehouses
–! uaLa Warehouse Covernance
–! uaLa Warehouse uocumenLauon 8oadmap
–! Pow uaLa Works
•! Þro[ecL Servlces
–! hup://pro[ecLs.daLamamL.com
–! hup://pro[ecLs.daLamamL.com/demo
–! hup://pro[ecLs.daLamamL.com/daLamamL
•! MonlLorlna uaLa CuallLv & L1L
–! uaLa ManaaemenL & Warehouslna LoolseL
1uesdav, !une 3, 2008
© 2008 uaLa ManaaemenL & Warehouslna
hup://www.daLamamL.com
Þaae 24
!"#$%&'(!&)*#+,"&-&
.#+#,/+,&0*"&$#0#&1#!"*(23"&0*!(2,*&
/.45"."+0#0/(+&#+$&4!($2)0/(+&
0*#+T %(2&
uavld M Walker
3
rd
lnLernauonal Clua 81C Conference
klev, ukralne - 3
rd
!une 2008