Data Services Training

FSS

Mc Lc
Chng 1. Tng Quan v Data Services...............................................................................................3
I.

Mt S Khi Nim C Bn .......................................................................................................3

1.

Data services .......................................................................................................................3

2.

Kin trc Data services ........................................................................................................3

II.

Cc Cng C Trong Desinger ...................................................................................................9

1.

Cc i Tng Trong Th Vin Chung ..................................................................................9

2.

Ca S Lm Vic ..................................................................................................................9

3.

Khai Bo Bin v Tham S.................................................................................................. 10

4.

S Dng Hm v Script ...................................................................................................... 12

5.

Validation menu................................................................................................................ 14

6.

Palette .............................................................................................................................. 15

Chng 2. Gii Thiu Cc Engine Thng S Dng Trong Tranform .................................................. 16

I.

Tng Quan Chc Nng ca Mi Engine .................................................................................. 16

1. Data_Transfer ...................................................................................................................... 16
2. Date Generation .................................................................................................................. 16
3. Effective Data ...................................................................................................................... 16
4. Hierarchy_Flattening ............................................................................................................ 16
5. History_Preserving ............................................................................................................... 17
6. Key_Generation ................................................................................................................... 18
7. Map_CDC_Operation: .......................................................................................................... 18
8. Pivot .................................................................................................................................... 20
9. Reverse_Pivot ...................................................................................................................... 20
10.Table_Comparison ............................................................................................................... 20
11.XML Pipeline........................................................................................................................ 21
12.V D Slow Data Change Loi 2 ............................................................................................. 21

Chng 3. Gii Thiu Cc Engine Thng S Dng Trong Flatform ................................................... 23

I.

Tng quan v chc nng ca mi engine................................................................................ 23

L Th Phng Thanh

Data Services Training

FSS

1. Case ..................................................................................................................................... 23
2. Map_Operation .................................................................................................................... 23
3. Merge................................................................................................................................... 25
4. Query ................................................................................................................................... 26
5. Row Generation .................................................................................................................... 26
7. SQL ....................................................................................................................................... 26
8. Validation ............................................................................................................................. 26

L Th Phng Thanh

Data Services Training

FSS

Chng 1. Tng Quan v Data Services

I. Mt S Khi Nim C Bn
1.

Data services
a. nh Ngha: Data services l mt gii php hiu qu v cht lng
v tch hp d liu.
b. Mc ch: Cung cp mt mi trng pht trin, thc thi, qun l
trong vic tch hp, vn chuyn, ci tin, phn tn d liu mi ni v
mi lc.
c. Li ch
Gim gi thnh, tit kim thi gian.
H tr x l khng hp nht gia cc mng d liu, cho php vic
x l d liu t nhiu nh vendor, a phng v cu trc.
Kh nng tng tc(Interoperability) v linh hot(Flexibility) ca
dch v Web (Web services) da trn cc ng dng cho php tch hp
n cc h thng front-end trong vic x l thi gian thc.
Phn tch, chun ha, lm sch d liu.
So snh v hp nht cc bn ghi.
H tr nhiu ngi dng v bo mt d liu tt.
Vi cc chc nng lu du gip ch cho cc lp trnh vin c th
theo di d liu, ng thi xc nh nhanh nhng tc ng bo tr, sa
li.

2.

Kin trc Data services

a. Cc Thnh Phn Trong Data Service

L Th Phng Thanh

Data Services Training

FSS

Hnh 1. Kin Trc Data Services

Management Console: Gm cc vng iu khin sau:
o Administrator:

Hnh 2. Giao din Administrator

Lp lch, iu khin v thc thi cc batch job.
Cu hnh, khi ng, dng cc dch v realtime.
Qun l cc user.
o Auto Documentation
L Th Phng Thanh

Data Services Training

FSS

Hnh 3. Giao din Auto Document

Quan st, phn tch, in cc biu din bng
ha ca cc i tng.
o Data Validation

L Th Phng Thanh

Data Services Training

FSS

Hnh 4. Giao din Data Validation

Xem cc thng k t cc quy lut xc nhn hp
l d liu
o Impact & Lineage Analysis

Hnh 5. Giao din Impact & Lineage Analysis

Xc nh tc ng ca cc thay i.
o Operational Dashboard

L Th Phng Thanh

Data Services Training

FSS

Hnh 6. Giao din Operational Dashboard

Thng k cc s thc thi v trng thi.
o Data Quality

Hnh 7. Giao din Data Quality

Bo co s to cc report khc nhau t
Data Quality Transform

L Th Phng Thanh

Data Services Training

FSS

Designer:
L mt giao din ha kt ni vi local repository
lc khi to. S dng xc nh nhng g data s c
x l thng qua vic to ca datastores v file formats.
Cc metadata c lu trong repository.
To, duy tr v thc hin cc cng vic (jobs),
workflows, dataflows
Cho php ngi s dng ko th cc d liu/file v
chuyn n d liu ch.
Repository: l kho cha metadata, c 4 loi repository:
Local : lu cc d liu c to t designer ti my
local.
Central: kho d liu trung tm, h tr mi trng mi
nhiu ngi dng, lu cc thng tin ging kho local,
ko th thc thi cc cng vic v khng kt ni no n
mt job server.
Profiler: lu cc d liu lin quan n cc profiling
task t designer tool khi ngi s dng chay ct hoc
cc lin quan n profiling task. Profiler kt hp vi
job server.
Cleansing Packages: tham chiu n DQD, lu cc t
in ngn ng c th cng nh cc t in t to cho
vic phn tch d liu.
The Data Services service
Engine
Job Server
L Th Phng Thanh

Data Services Training

FSS
Services
Access Server
Adapters
The Address Server

II. Cc Cng C Trong Desinger

1.

Cc i Tng Trong Th Vin Chung

Project: T chc cng vic.

Jobs: Thc thi cc lung cng vic.

Work flows: Ci t th t ca cc lung cng vic.

Data flows: Lu cc kt ni ngun n ch.

Transform: Thay i d liu da theo quy lut.

Datastore tables: D liu ch/ ngun theo RDBMS.

Cung cp kt ni vi d liu ngun

Import metadata t d liu ngun

c v ghi d liu n cc bng

S thay i cu trc c th khng c t ng cp

nht: v d nh thm mt ct vo bng.vv.

C 3 loi datastore:

Database datastores : ly d liu metadata t RDBMS nh

Oracle, MSSQL, IBM DB2, MYSQL.
Application datastores : ly d liu metadata t ERP nh
SAP, Siebel, JD Edwards, BEA, PeopleSoft, vv..
Adapter datastores : ly d liu t cc ng dng.

File formats: Cc loi d liu ch/ngun khc (txt, csv, excel,

xml).
2.

L Th Phng Thanh

Ca S Lm Vic

Data Services Training

FSS

Project Area: ch hin th duy nht mt project c chn

trong Local Object Library.

Hnh 8.
Trong Project Area gm:
Designer: Hin th cc i tng lm vic.
Monitor: Hin th cc cng vic m job server thc hin. Nu
thc thi b li c biu tng gch cho mu , ch lu t khi
designer start.
Log: Lu tt c lch s cng vic m jobserver thc hin i
vi tng job trong d n (Project) ang m.
3.

Khai Bo Bin v Tham S

Tool->Variables

L Th Phng Thanh

10

Data Services Training

FSS

Hnh 9.

Ti mc Project, Job c th to (insert) Variables v Global Variables mi,

khng c parameter. Mc Mc dataflow ch c to Parameters mi.
Global Variable: To ti mc Job, cho php gn bin ti khung ca s khai
bo, hoc trong tab Global variable khi m thuc tnh (Properities) Job. S
dng nhiu ni. Ga tr ban u s b thay i nu trong qu trnh x l c
bc chnh sa gi tr. Hin th trong Query Transform.
Variable: To ti mc Job v Work Flow, khng c hin th trong
dataflow (Query transform). C th nhn gi tr t mt parameter ch
thng qua gn trong trong script mc workflow.
Parameter: mc WF v DF cho php gn gi tr t 1 Variable, hin th
trong Query Transform.
C 3 loi Parameter:
Input: bt buc phi khi to n mt gi tr trc hoc n mt bin
(bin c th cha c gi tr).
Output: phi c gn n mt bin (variable), bin ny c th c gn
n gi tr c th hoc khng cn. (Ko bt buc). Ko gn c gi tr trc
tip.
Output/input: Tng t vi Output?

L Th Phng Thanh

11

Data Services Training

4.

FSS

S Dng Hm v Script
a. Hm:
Dng x l cc bin u vo v tr v mt gi tr
u ra. Ga tr u vo hoc u ra c th bin hoc ct.
Hm c s dng trong cc scripts, Transforms,
Conditional v cc hm do ngi dng t to.
-Cc gi hm chun trong Desinger:

Functions

Description
Thc hin tnh ton cc tnh ton trn gi tr

L Th Phng Thanh

Aggregate Functions

s (avg,count,count_distinct,max,min,sum).

Conversion Functions

Cc hm i kiu d liu ca gi tr.

Custom Functions

Cc hm c nh ngha bi ngi dng.

Database Functions

Performs operations specific databases

Date Functions

Thc hin tnh ton v chuyn i kiu trn

12

Data Services Training

FSS
cc gi tr date.
Performs operations specific to your Data

Environment Functions

Services environment
Cc hm thc hin cc vic kt mt iu
kin (lookup), nhiu iu kin (lookup_ext),

Lookup Functions

gii hn ca iu kin (lookup_seq).

Cc hm s hc phc tp trn gi tr s :
abs, ceil, floor, ln, log, mod, power, rand,

Math Functions

rand_ext, round, sqrt, trunk.

Miscellaneous Functions

Performs various operations

Sap Function
Thc hin cc hm x l{ d liu chui, k{
String Functions

t.

System Functions

Performs system operations

Cc hm xc thc kiu c th ca gi tr, cc
hm tr v gi tr 0 hoc 1 (is_valid_date,

Validation

is_valid_double, is_valid_int..vv)

b. Script
Script c s dng gi hm v gn gi tr cho cc
bin trong WorkFlow.
V d, c th s dng hm SQL trong script xc
nh thi gian cp nht cui cng i vi mt bng v gn
gi tr n mt bin.
Sau gn bin ny cho mt tham s (parammeter)
vo mt data flow v xc nh cc dng t d liu ngun.
Mt script c th gm cc lnh sau:
L Th Phng Thanh

13

Data Services Training

FSS
o Gi hm.
o IF
o While
o Gn
o Ton t
Cc c php c bn trong script:
o Mi dng kt thc bng du chm phy (;).
o Tn bin bt u vi k dollar ($).
o Bin kiu chui t trong du nhy n (').
o Cc comment bt u vi du thng (#).
o Gi hm lun phi ch nh parameter thm ch
nu hm chng s dng parameter no.
V d:
$Todate= sysdate();
$G_Startdate=
END_DATE

sql('demo_target','SELECT
FROM

JOB_STATUS

WHERE

JOB_NAME= {$Jobname}');
Print('The end date is: [sysdate()+1]');

5.

Validation menu

Validate:Xc thc cc i tng trong vng lm vic hin ti

hoc cc i tng trong Job trc khi thc thi ng dng.

Show ATL: Xem bng read-only ca ngn ng c kt hp

vi Job.

Display Optimized SQL: hin th SQL m Data Services sinh ra

cho mt dataflow chn.

L Th Phng Thanh

14

Data Services Training

6.

FSS

Palette

Ti mc Project, Job: Thanh palette hin th cc cng c:

Hnh 10.
Pointer, WF, DF, Script, Conditional, While Loop, Try, Catch, Annotation.
Ti mc Data Flow: Thanh palette hin th cc cng c:

Hnh 11.
Pointer, DF, ABAP data flow, Query Transform, Template Table, Template
XML, Annotation.

L Th Phng Thanh

15

Data Services Training

FSS

Chng 2. Gii Thiu Cc Engine Thng S Dng

Trong Tranform
I. Tng Quan Chc Nng ca Mi Engine
1. Data_Transfer
Cho php mt lung d liu x l chia thnh hai lung d liu ph v
nhm gim hot ng tiu th ti nguyn my ch. Data transfer import
d liu ra mt file ring trong qu trnh x l.
2. Date Generation
To ra mt ct in gi tr ngy da trn ngy bt u v ngy kt thc vi
mt h s tng.
3. Effective Data
c to da trn kha chnh, to thm mt ct vi gi tr effect_to, c
th tnh ton s ngy effect t mt gi tr effect_date c sn.
4. Hierarchy_Flattening
Flattens d liu phn cp thnh cc bng quan h trong star schema.
Hierarchy_Flattening c th c c hai chiu dc v ngang.
V d:
Cho mt bng gm employee_id v manager_id, HF c th gip xc
nh nhng ngi qun l ca mi nhn vin theo cp bc sau y:

L Th Phng Thanh

16

Data Services Training

FSS

Hnh 12.

Hnh 13.
Kt qu nh hnh 13. Nhn vin id 100 l manager cao nht.
104 c ngi qun l 103, 103 c qun l l 102, 102 c qun l l 100.

5. History_Preserving

L Th Phng Thanh

17

Data Services Training

FSS

Chuyn i hng gn c l Update cp nht thm INSERT, do , gi tr

ban u c lu ch, khi ch nh ni m ct s tm v cp nht d
liu.
6. Key_Generation
To ra kha mi cho ngun d liu, bt u t mt gi tr ln nht ca
kha hin c trong bng bn ch nh.
7. Map_CDC_Operation:
Map_CDC_OPERATION thng c dng h tr CDC, cng c
ny c nhng thay i DML ngun nh Insert/Update/Delete v
nh x n bng tng ng trong h thng ch.
Mt bng CDC lun c 2 ct c t tn:
DI_OPERATION_TYPE: lu m DML tng ng vi insert, update,
delete. Cc m c k kiu: I (INSERT), B(tin update), U(update,
hu update), D(delele).
DI_SEQUENCE_SEQUENCE: gi ct lu thng tin th t ca cc lnh
DML. V d nu mun cp nht hoc xa dng trc khi insert, th
phi c mt th t thc thi cc lnh.
V d: Cho bng gc c tn l T_MAPCDC c ni dung nh sau:

Hnh 14.
Bng ch TL_CDC

L Th Phng Thanh

18

Data Services Training

FSS

Hnh 15.
Vi DataFlow SC_MAP_CDC:

Hnh 16.

Hnh 17.

L Th Phng Thanh

19

Data Services Training

FSS

Kt qu

Dng th 0 c operation B, m id 101 b i thnh Thanh va

VietNam t dng th 2 c operation => Dng c ch B s b
gi update cc gi tr (khng phi kha) t dng c
operation U gn nht. Dng c ch U gn nht khng
c update m gi nguyn gi tr c.
Dng th 1, operation B gi nguyn.?
Dng th 2, operation U c update trc cho dng
th 0 nn gi nguyn.
Dng th 3, operation B gi nguyn. ?
8. Pivot
Chuyn ct thnh dng.
9. Reverse_Pivot
Chuyn dng thnh ct.
10. Table_Comparison
So snh hai tp hp d liu v ch ra s khc bit gia chng l mt tp
hp d liu vi cc dng gn c INSERT v UPDATE.
C ba phng php ci t i vi transform ny:
Row_by_Row select: truy vn SQL trn bng so snh s thc thi mi ln
khi n nhn mt dng u vo. Phng thc ny kh chm, ch nn s
dng vi lung data c s lng dng d liu t.

L Th Phng Thanh

20

Data Services Training

FSS

Cached comparision table: Data Service s ti ton b bng so snh. DS

gi SQL query cho mi dng u vo, tuy nhin lc ny bng so snh
c t trong vng nh hoc phn trang. V vy m khng cn gi truy
vn SQL trong c s d liu. La chn ny nhanh hn nhiu so vi Rowby- row select. Nu bng so snh c hng triu dng, nhng ch t 5 dng
trong Table comparision transform th la chn ny cng khng c nhiu
hiu qu hn Row-by- Row select.
Sorted input: Phng thc ny s c mt yu cu trc khi lm vic: sp
xp li dng theo chiu tng da trn kha chnh (kha ko vo 'Input
primary key columns'). Hu ht cc trng hp yu cu insert mt Query
transform trc Table Comparision tranform. Phng thc ny nhanh hn
'Cached comparison table'. Khi c xp xp, data service ch cn c
bng so snh mt ln.
V d: mt bng so snh khong 250 000 dng. Nu chn 'Row-by-Row
select' th thi gian chy 7-8 pht, chn 'Cached comparison table' mt 1012 giy, chn 'Sorted input' mt 7 giy.
Filter: Gii hn s dng t bng so snh, ch ly nhng tp d liu tha
iu kin filter lm u vo cho Table Comparison transform.
11. XML Pipeline
X l{ u vo XML
12. V D Slow Changing Dimension Loi 2
Query: load cc d liu mun ly t bng gc
Table Comparision: so snh gia hai bng ti cc ct chn
History_Preserving: gi li tt c ni dung gc, thm dng mi vo bng
ch nu c s thay i trn cc ct so snh.
Key_generation: sinh kha surrogate cho bng gc.

L Th Phng Thanh

21

Data Services Training

FSS

Hnh 14.
Kt qu:

Hnh 15.

L Th Phng Thanh

22

Data Services Training

FSS

Chng 3. Gii Thiu Cc Engine Thng S Dng

Trong Flatform
II. Tng quan v chc nng ca mi engine
1. Case
n gin ho cc nhnh logic trong cc lung d liu bng cch cng c
trng hp hoc ra quyt nh logic trong mt chuyn i. ng dn c
nh ngha trong mt bng biu.
2. Map_Operation
Cho php chuyn i gia cc thao tc normal, insert, update, delete
c d liu u ra mong mun:
Update

Thay i gi tr dng target

Normal

To dng mi trong target

To dng mi trong target (C
s so snh gia 2 bng gc v

L Th Phng Thanh

Insert

ch)

Delete

Xa dng t bng target

Discard

Xa dng t qu trnh x l{

23

Data Services Training

FSS

Hnh 14.
Ct Region_Name, dng th 3 ca s bn tri t bng gc (Regions)
c update thnh Asiad so vi bng ch cng Region_ID tng ng.
Xt cc trng hp trong Map_Operation:

Hnh 15.
Kt qu:

L Th Phng Thanh

24

Data Services Training

FSS

Hnh 16.

Hnh 17.
Input:

Hnh 18.

Output:

3. Merge
Thng nht cc d liu t hai hay nhiu ngun vo mt ngun ch duy nht.
Khi merge cc d liu vo mt bng, bt buc cc thuc tnh ca cc bng d
liu phi ging tn v di ca datatypes.
L Th Phng Thanh

25

Data Services Training

FSS

4. Query
Ly mt tp d liu tha mn iu kin c ch nh. Mt truy vn chuyn
i tng t nh mt cu lnh SQL SELECT.
5. Row Generation
To ra mt ct in gi tr s nguyn bt u t s 0 v tng dn bi mt
gi tr cui cng c ch nh.
7. SQL
Thc hin cc hot ng truy vn SQL c ch nh.
8. Validation
m bo rng cc d liu bt kz giai on trong data flow u tha iu
kin. C th lc ra hoc thay th d liu khng thch hp.

L Th Phng Thanh

26