DB2 APD : Enabling Data Analysis

Dominique Evans, DB2 A

vance Pro!"em Determination, IB# evans $ca%i!m%com

SNAPDIFF Comprehension for coherent measurements

1

Presentation goals

PA'( I
I entif) the pro!"em Demonstrate simp"e SNAPDIFF Intro uce notion of section *+roup, Fi"ters

PA'( II
Comp"e- SNAPDIFF Importe fi"es .oine fi"es #eta ata an formu"ae /oa an un"oa

/afs/tor/u/evansd/public/bin/snapdiff

&afs&tor&u&evans &pu!"ic&!in&snap iff

The problem
1% E-periments create "ar+e amounts of measurement ata 2% Data is hetero+eneous% 0% 1sefu" ata *a"most, a"2a)s invo"ves mu"tip"e measurement captures over time interva"s% *e%+% snapshots,% 3% (he num!er of atasets rapi ") e-cee s human a!i"it) to +ain much insi+ht or 4no2"e +e from the ra2 fi"es 5% Automate ata capture scripts can easi") +enerate o6ens or hun re s of reports

&afs&tor&u&evans &pu!"ic&!in&snap iff

(ime 8

(ime 91

(ime 92

Snapshots Snapshots

Snapshots Snapshots

Snapshots Snapshots

!2p !2p

!2p !2p

!2p !2p

7ther

7ther

7ther

&afs&tor&u&evans &pu!"ic&!in&snap iff

A Key to analysis

;hat is chan+in+ over time < B) ho2 much < In re"ation to 2hat other measurements<

&afs&tor&u&evans &pu!"ic&!in&snap iff

A solution. With absolute minimal editing , recogni ing a !ide array o" input datasets.
'eco+ni6e Name > ?a"ue pairs *Name !ecomes canonica" 2ithin the scope of a stu ) session @ i%e% it is invariant an if more than one source spea4s to the measurement, the names must a+ree, or !e coerce to o so *norma"i6e ,,% I+nore a"" other ata Si eA!)Asi e presentation of ata Fi"ters Interactive fee" , * B1( te-t termina" simp"icit) A no BC$DE F1I * i%e% accept irt) an mess) input atasets , *eas) to spot chan+es in va"ues,

Fast , eas) , simp"e *for a hoc usa+e an)2a),

=

E-tensi!"e resu"ts
= &afs&tor&u&evans &pu!"ic&!in&snap iff

Additionally
'eco+ni6e the time a-is Assume that the ata is coherent *an in phase more or "ess, Sort an) output co"umn A""o2 ata to !e presente !) ro2, or !) co"umn *a"on+ the time a-is, or o2n the time a-is, A""o2 +ranu"arit) of ata co""ection to !e mo ifie *s4ip measuremts,

&afs&tor&u&evans &pu!"ic&!in&snap iff

snapdi""

&afs&tor&u&evans &pu!"ic&!in&snap iff

#uic$ e%ample
Customer comp"aintJ A #u"tiAhour !enchmar4 A DB2 I%G Statement Concentrator A 'un +oes south after hour 2 hours #easurementsJ Inc"u e a series of )namic statement snapshots, ever) 18 minutes, appen e in a sin+"e fi"e% 2hi"e K 8 L @ o !2p )n MM !2p N )n%out @ s"eep 088 @ one 'esu"tJ 0335=HI #a) 28 10J8G !2p N )n%out
I

09 me+ is a"rea ) p"ent) of ata for humans to ea" 2ith%

I &afs&tor&u&evans &pu!"ic&!in&snap iff

Ouic4 ata pee4J

;e notice that the pac4a+e cache stan6a repeats, separate !) "ar+e statement umps% /etPs +rep that out%
18

18

&afs&tor&u&evans &pu!"ic&!in&snap iff

Ouic4 ata pee4J

Better% ;e can at "east scro"" o2n the *in this case 0G, atasets an see if 2e can see an)thin+% (his is pro!a!") as far as 2e usua"") +et usin+ stan ar too"s%
11 &afs&tor&u&evans &pu!"ic&!in&snap iff
11

snapdi""
e+rep Ap QRD)namic CacheJSRData!ase partitionT S se Ae Ts&K8AIL& > C&T M miniNsnaps *Simp") a s Q > Q , to create name > va"ue pairs , % !2p N )n%out

12

12

&afs&tor&u&evans &pu!"ic&!in&snap iff

10

10

&afs&tor&u&evans &pu!"ic&!in&snap iff

What are the delta &alues'

13

13

&afs&tor&u&evans &pu!"ic&!in&snap iff

A richer e%ample. Analysis o" slo! (AD) secondary.

snap iff U%poo"NK8AILK8AIL AVoin poo"Nsi6es Aimport U% !NK8AILK8AIL Aimport U%Ne uNsnapshotN8K8AILK8AIL Ameta &(#P&evans &meta Aun"oa poo"N !Ne u

snap iff Aroh T/o+ +ap runnin+ avera+eT A48 A"oa poo"N !Ne u Aon")pivot Apf Tper secon Sprefetch%UsecSNDe"ta Num!er 2rite "o+ I7sSNPrimar) Stan

!) E-tent /a+SNInterva" compute "o+ +apW

15

15

&afs&tor&u&evans &pu!"ic&!in&snap iff

1=

1=

&afs&tor&u&evans &pu!"ic&!in&snap iff

A di""erent &ie!, same data:

snap iff Acorr T/o+ +ap runnin+ avera+eT A48 A"oa poo"N !Ne u Aon")pivot Apf TNDe"ta%UC7DS1%U !2shre %Utota"SNDe"ta%UC7DS1%U !2re Aon")+"o!a"

o2%Utota"SNDe"ta%UC7DS1%U !2ha rs%Utota"T

1G

1G

&afs&tor&u&evans &pu!"ic&!in&snap iff

1H

1H

&afs&tor&u&evans &pu!"ic&!in&snap iff

Bac$ to snapdi"" basics.

Simp"est usa+e is 2 fi"es, e%+% 2 ata!ase snapshots Defau"t isp"a) is !) atapoint ro2, #easurements in co"umns i%e% time a"on+ - a-is *snap iff etects snapshot timestamps,

1I

1I

&afs&tor&u&evans &pu!"ic&!in&snap iff

*impli"y. +et rid o" sections. *ort . +et rid uninteresting data.

28

28

&afs&tor&u&evans &pu!"ic&!in&snap iff

E%actly the same, but ,delta display -includes percent by de"ault.

21

21

&afs&tor&u&evans &pu!"ic&!in&snap iff

What is /section0 '

-a simple /group by0.

?arious snapshots have repeatin+ +roups Snap iff automatica"") etects certain +roups * ata!ase, !ufferpoo", ta!"espace,% Datapoints 2ithin sections are qua"ifie !) the section name Can turn off 2ith nosection Can ec"are an) atapoint a section e"imiter usin+ Asection Qmeasurement nameW *2here Qmeasurement nameW is a re+e- that 2i""" !e use to i entif) a atapoint name so ma4e it uniqueE,

22

22

&afs&tor&u&evans &pu!"ic&!in&snap iff

Ta$e a bu""erpool snapshot "or instance:

1se a restricti&e "ilter -"or clarity.. 2section /Bu""erpool name0 is automatically detected.

20

20

&afs&tor&u&evans &pu!"ic&!in&snap iff

*o !hat is ,p"' -abbre&iation o" 2p"ilter. /Post -data ingestion. "ilter0. A fi"ter app"ie after a"" atasets are in+este *an have ha meta ru"es app"ie if present,% A"so f *Afi"ter, , operates on incomin+ ataset * roppin+ atapoints that o not qua"if),% Ear"iest fi"ter% Af an pf usua"") i entica" resu"ts un"ess usin+ fancier options *A meta, Aimport ,% Af is faster *since rops the ata much sooner, , !ut "oss) for meta processin+% Asfi"ter Xre+e-pM , fi"ters 2ho"e sections% e%+%J
23

23

&afs&tor&u&evans &pu!"ic&!in&snap iff

3iltering sections

B(;J formatte

ata can !e turne offJ noformat

25

25

&afs&tor&u&evans &pu!"ic&!in&snap iff

,import .
A

(o! to e%tend the data.

s Qne2W ata to primar) atasets

e%+% import ata!aseNsnapshot to !ufferpoo"Nsnapshot Sections not reco+ni6e *mi+ht&pro!a!") 2i"" enhance this, A"" importe +roups% ata is Q+"o!a"W *unqua"ifie , re+ar "ess of interna" repeatin+ atapoints roppe

Dup"icates in import

An) name > va"ue ata can !e importe Data shou" !e coherent *i%e% re"ate , An) num!er of separate sources can !e importe at once Can import one ataset *e%+% constants,, or count* primar) , atasets *coherent measuremets,
2= &afs&tor&u&evans &pu!"ic&!in&snap iff

2=

2G

2G

&afs&tor&u&evans &pu!"ic&!in&snap iff

24oin is an ,import !ith relationship through the ,section

A""o2 ata to !e a e to section specific ata, *rather than +"o!a"") as 2ith import, Foo e-amp"e is !ufferpoo" snapshot@ since the ra2 snapshot oes not inc"u e pa+e si6e *E, % A QBufferpoo" Pa+e si6eW atapoints usin+ Voin%

A""o2s correct computation of !)tes from ra2 snapshot ata%

2H

2H

&afs&tor&u&evans &pu!"ic&!in&snap iff

24oin : +oing bac$ to the bu""erpool snapshots e%ample

CaveatJ Do not use nosame E

2I

2I

&afs&tor&u&evans &pu!"ic&!in&snap iff

Primary data duplicate processing

Choice of severa" po"iciesJ A up K i+nore S ha"t S rep"ace S a++re+ate S se+re+ate Y * efau"t se+re+ate,

1% I+nore * rops ups, 2% Za"t *stops 2ith error on up, 0% 'ep"ace *so "ast up 2ins, 3% A++re+ate *sum, 5% Se+re+ate *automatica"") +enerate mo ifie name *ND1PNn, ,

08

08

&afs&tor&u&evans &pu!"ic&!in&snap iff

01

01

&afs&tor&u&evans &pu!"ic&!in&snap iff

P567T &ie!s
Freat for o!servin+ "ar+e num!ers of measurements Best 2ith ver) se"ective fi"ters *so to fit on the screen,

02

02

&afs&tor&u&evans &pu!"ic&!in&snap iff

8eta measurements , and "ormulae

A""o2 e-pressions to !e app"ie to measurement ata 7peran s are atapoint names Eva"uate on a per ataset !asis AF(E' f *Afi"ter, % BEF7'E pf *Apfi"ter, Each ataset can reference any pre&ious ataset measurement*s,

00

00

&afs&tor&u&evans &pu!"ic&!in&snap iff

03

03

&afs&tor&u&evans &pu!"ic&!in&snap iff

05

05

&afs&tor&u&evans &pu!"ic&!in&snap iff

0=

0=

&afs&tor&u&evans &pu!"ic&!in&snap iff

snap iff A< for comp"ete *more or "ess, he"p !2service ;i4i comin+ soon% /ocation of script *Per",J &afs&tor&u&evans &pu!"ic&!in&snap iff

9A*T *95DE

*T7P (E)E
0G

0G

&afs&tor&u&evans &pu!"ic&!in&snap iff