2015 International Conference on Intelligent Transportation, Big Data & Smart City

7KH GHVLJQ DQG LPSOHPHQWDWLRQ RI ODUJH GDWD KLJKSHUIRUPDQFH VRUWLQJ DOJRULWKP

=KDQJ +DLMLDQ
 %HLMLQJ ,QIRUPDWLRQ 7HFKQRORJ\ &ROOHJH %HLMLQJ  &KLQD
#TTFRP

$EVWUDFW 7KLV DUWLFOH PDLQO\ DLPV DW WKH VRUWLQJ DOJRULWKP
RI WKH HUD RI ELJ GDWD DQDO\VLV IRU WKH UDQNLQJ PHWKRG RI D
VHULHV RI FRUUHVSRQGLQJ UHVHDUFK DQG IRU WKH UHDOL]DWLRQ RI
WKH DOJRULWKP LQ WKLV VHULHV
.H\ ZRUGV %LJ GDWD +LJK SHUIRUPDQFH 6RUWLQJ
DOJRULWKP
7KH DGYHQW RI WKH HUD RI ELJ GDWD PDNHV WKH GDWD LQ
WKH GDWDEDVH VWRUDJH LV PRUH DQG PRUH ELJ WKH
H[SRQHQWLDOO\ VRUW WLPH PDNHV WKH DPRXQW RI GDWD ORQJHU
UXQQLQJ WLPH DQG DIIHFW WKH ZRUNLQJ HIILFLHQF\ D VRUW RI
JHQHUDO IRUP ZLOO QHHG WR VSHQG D ORQJ WLPH WR FRPSOHWH D
IXOO VRUW EHKDYLRU IRU VRPH VXSHU ODUJH DPRXQW RI GDWD
IRUP LV UHTXLUHG IRU D IHZ GD\V WLPH WR ILQLVK VRUWLQJ
7KH PRVW LPSRUWDQW IXQFWLRQ RI WKH GDWDEDVH
EHFRPHV D IXQFWLRQ RI WKH VRUWLQJ VHTXHQFH LQ WKH
GDWDEDVH XVLQJ WKH UROH FDQ QRW EH UHSODFHG DQG XVH LW
PRUH JHQHUDOO\ 7R HQKDQFH WKH RYHUDOO SHUIRUPDQFH RI
WKH GDWDEDVH V PDLQ UROH LV WKH LPSURYHPHQW RI WKH
SHUIRUPDQFH RI WKH VRUW 'DWDEDVH VXSSRUW ELJ GDWD LV WKH
NH\ WR WKH ELJ GDWD VRUWLQJ %DVHG RQ XVHU V KDUGZDUH DQG
VRIWZDUH LQIRUPDWLRQ HIILFLHQW VRUWLQJ DOJRULWKP DGRSWHG
FDQ WKH LPSURYHPHQW RI WKH XVDELOLW\ DQG HDVH RI XVH RI
WKH GDWDEDVH DQG WKH XVHU WR WKH KDUGZDUH RI VKLHOGLQJ
'LIIHUHQW DSSOLFDWLRQ LV GLIIHUHQW WR WKH UHTXLUHPHQW RI
VRUWLQJ LW UHTXLUHV DQ HIILFLHQW VRUWLQJ DOJRULWKP WR PHHW
WKH RUGHU UHTXLUHPHQWV HQVXUH WKH QRUPDO RSHUDWLRQ RI WKH
DSSOLFDWLRQ 8VXDOO\ D VROXWLRQ XVLQJ D VRUWLQJ DOJRULWKP
LV YHUWLFDO KRUL]RQWDO SDUWLWLRQLQJ DQG SDUDOOHO WHFKQRORJ\
%XW DW WKLV VWDJH RI WHFKQRORJLFDO GHYHORSPHQW VSHHG
UHODWLYH WR WKH LQIODWLRQ GDWD LV GHYHORSLQJ YHU\ UDSLGO\ LQ
RUGHU WR DFKLHYH FOXVWHU DUFKLWHFWXUH WKLV VFKHPH D ODUJH
QXPEHU RI PDQSRZHU LV UHTXLUHG WKH FRUUHVSRQGLQJ
WHFKQRORJ\ LV VXSHULRU 6R VROYH WKH SUREOHP RI ELJ GDWD LV
WKH EDVLV RI WKH VLQJOH QRGH VRUW RI KXJH DPRXQWV RI GDWD
KDV LQFUHDVHG VR WKH HQG ELJ GDWD LV WKH NH\ SUREOHP LQ
KDUGZDUH DXWRPDWLF VRUWLQJ VWUDWHJ\ DOJRULWKP
ĉ7+( 352%/(0 $1$/<6,6 2) 0$66 '$7$
6257,1* $/*25,7+0
$ 6RUWLQJ DOJRULWKP DQDO\VLV RI WKH SUREOHP
%LJ GDWD VRUWLQJ SUREOHP RI SKDVHV KDYH PHUJH DQG
PHPRU\ EORFN VRUWLQJ SKDVH KHUH WR FRQVWUXFW D VLPSOH
GDWD VRUWLQJ WHVW ,Q WKH ZKROH SURFHVV RI RUGHULQJ LQ 
PE\WH RI 0(0 GDWD IORZ ZKHQ VLJQV DUH  PE\WH
DQDO\VLV RI &38 ,2 ,25 ,2: VSHFLILF DQDO\VLV
)LJXUH  6LPSOH ,2 &38 ODUJH DPRXQW RI GDWD VRUWLQJ
DOJRULWKP DQG 0(0 GLDJUDP RUGLQDWH IRU UHDG VSHHG 
PE\OHV DEVFLVVD IRU WV
7KH VWXG\ IRXQG WKDW WKH PHPRU\ EORFN VRUWLQJ
SKDVH LV DFWXDOO\ D SURFHVV RI FLUFXODWLRQ SURFHVV KHUH LV D
F\FOH SHULRG ,Q WKLV F\FOH IRU LQILQLWH DPSOLILFDWLRQ
REYLRXV FDQ VHH GDWD UHDG DQG ZULWH ,2 XWLOL]DWLRQ VWDJH
LV UHODWLYHO\ ODUJH DW WKLV WLPH RI WKH &38 XWLOL]DWLRQ UDWH
LV UHODWLYHO\ ORZ &RQYHUVHO\ ZKHQ PHPRU\ RUGHULQJ
SKDVH RI WKH &38 XWLOL]DWLRQ LV UHODWLYHO\ KLJK WKH ,2 LV
LQ D VWDWH RI IUHH 6R RIWHQ WKH SUREOHPV DV IROORZV WR
UHDG WKH GDWD EORFN QDPHO\ WKH ,2 SHDN WKH &38
XWLOL]DWLRQ UDWH DOVR WR UHDFK  WKLV WLPH ,2 XWLOL]DWLRQ
UDWLR LV UHODWLYHO\ KLJK ZKHQ WKH &38 XWLOL]DWLRQ LV ORZ
PDNH IXOO XVH RI &38 UHVRXUFHV LV DQ XUJHQW QHHG WR VROYH
SUREOHPV 0HPRU\ EORFN VRUWLQJ SKDVH DIWHU WKH &38
XWLOL]DWLRQ UDWH UHDFKHG  WKH ,2 LV LQ D VWDWH RI IUHH
WKH KLJKHU WKH &38 XWLOL]DWLRQ WKH PRUH ,2 ORZ
XWLOL]DWLRQ VR DW WKLV VWDJH VKRXOG WKLQN WKH TXHVWLRQ LV
ZKHWKHU WKH ,2 FDQ EH LGOH ,Q GHDOLQJ ZLWK PHPRU\
VWDJH WDVN LQVHUWLRQ SHULRGLF LGOH &38

978-1-5090-0464-5/16
/16 $31.00 © 2016 IEEE
$31.00 © 2016 IEEE 850
DOI 10.1109/ICITBS.2015.215

Authorized licensed use limited to: Ingrid Nurtanio. Downloaded on October 01,2020 at 02:35:11 UTC from IEEE Xplore. Restrictions apply.
% 7KH SUREOHP RI RSHQ VRXUFH GDWDEDVH V\VWHP
)RU RSHQ VRXUFH 0\VTO GDWDEDVH RQHWLPH DOORFDWLRQ
ZLWKLQ DOO VRUW RI H[LVWHQFH EHIRUH RUGHULQJ IRU VRUWLQJ DUH
GLUHFWO\ UHDG ZULWH D TXLFN VRUW UHVXOWV LQ &57 ILQDOO\
XQLILHG PHUJH GLVRUGHUO\ GDWD PHUJH LQWR RUGHUO\ UHVXOWV
7KH HQWLUH SURFHVV RI RUGHULQJ ,2 XWLOL]DWLRQ UDWH LV QRW
KLJK EXW ZULWH WKH VWDJH RI XWLOL]DWLRQ LV UHODWLYHO\ KLJK PHPRU\ PHUJH VRUW VWDJH VWDJH DQG EORFNLQJ WKH PHPRU\
1HHG WR VSHQG D ORW RI WLPH RQ PHPRU\ RUGHULQJ WKLV ZLOO
PDNH WKH XVH RI ,2 VWDWXV LQ WKH LGOH VWDWH )RU WKH HIIHFW
RQ WKH RYHUDOO SHUIRUPDQFH RI WKH ZKROH GLVN LV ,2: DQG
WKH XVH RI ,25 RYHUODS WRJHWKHU
:LOO EH VXEMHFW WR GRPHVWLF LQGHSHQGHQW FRQWUROODEOH
GDWDEDVH WKH PDLQ GDWD RI  JE\WH FRUUHVSRQGLQJ VRUW LQ
WKH SURFHVV RI WKH HQWLUH GDWDEDVH VRUWLQJ YDULRXV
UHVRXUFHV XWLOL]DWLRQ VLWXDWLRQ DV VKRZQ LQ ILJXUH 
&DQ EH VHHQ IURP WKH JUDSK VRUWLQJ ZDV MXVW WKH
EHJLQQLQJ RI ,2 UHDG WKH UHVRXUFH ZLOO EH VOLJKWO\ EHWWHU
RWKHU WLPHV XVH UDWHV DUH UHODWLYHO\ ORZ WKH PDLQ UHDVRQ
OLHV LQ WKH KHDS VRUW DUH PDLQO\ FRQFHQWUDWHG LQ WKH XVH RI
WKH ILUVW VWDJH DQG ,2 XVH LV WKH UHVXOW RI FHQWUDOL]HG ZD\
RI IORZLQJ ZDWHU WR GULYH :H DW WKH WLPH RI WHVWLQJ RSHQ
VRXUFH GDWDEDVH V\VWHP RQ GDWDEDVH V\VWHP IRU D ZDQW WR
HQRXJK WR UHDFK WKH OLPLWV RI WKH VRUW VRUWLQJ DOJRULWKP
FDQ EH LPSOHPHQWHGDŽ
)LJXUH  2ULJLQDO GDWDEDVH RI GRPHVWLF ODUJH QXPEHU RI
VRUWLQJ SURFHVV RI HDFK UHVRXUFH XVDJH RUGLQDWH &38
RFFXSDQF\  
Ċ7+( 29(5$// '(6,*1 0$66 '$7$ 6257,1*
$/*25,7+0
$ 6\VWHP DQDO\VLV RI WKH RYHUDOO VWUXFWXUH
7KH GDWD SURFHVVLQJ PHWKRG LV PDLQO\ WDVN GULYH ZD\
LQ LQ VRUWLQJ ILOH WKH PDLQ WKUHDG LV QRW LPPHGLDWHO\
EHJDQ DIWHU WDVN H[HFXWLRQ EXGJHW ILUVW XVHU SDUDPHWHU
FDOFXODWLRQ DPRQJ PXOWLSOH ,25 WDVN HVWDEOLVK D ,25
851
Authorized licensed use limited to: Ingrid Nurtanio. Downloaded on October 01,2020 at 02:35:11 UTC from IEEE Xplore. Restrictions apply.
WKUHDG DORQH IRU WKH PDMRU RI ,25 WDVN IRU VSHFLDO
SURFHVVLQJ WKH PDLQ WKUHDG ZLOO QRZ RWKHU RSHUDWLRQV
DQG GHDO ZLWK WKH WDVN DVVLJQHG WR WKH ,25 6XFK D
SURFHVV LV PDLQO\ PDQLIHVWHG LQ WKH IROORZLQJ ILJXUH 
% 'HVLJQ PHPRU\ RUGHULQJ VWDJH
0HPRU\ VRUWLQJ VWDJHV DUH GLYLGHG LQWR WZR VWDJHV
EORFN VRUWLQJ SKDVH LV WKH PDLQ WKUHDG LV PDLQO\
UHVSRQVLEOH IRU ,25 WDVN ,25 WKUHDG ZRUN LV PDLQO\
GULYHQ E\ WKH ,25 WDVN IRU EXIIHU GDWD WR DQDO\]H WKH PDLQ
WKUHDG QHHG WR DGG PDWFK WR WKH WDVN SULRULWL]H ZRUN WDVN
GULYHQ WKUHDG JURXS FRQVWUXFWV WKH HTXLYDOHQW VRUW NH\V
ZLWK HTXLYDOHQW VRUW NH\V IRU TXLFN VRUW 0HPRU\ PHUJH
VWDJH PDLQO\ LQ IDVW PHUJH VRUW VWDJH FRQVWUXFWHG EDVHG
RQ HTXLYDOHQW VRUW NH\V WR LPSURYH HIILFLHQF\ RI PHUJH
7R SDVV F\FOH WR FRPSOHWH WKH ZKROH PHPRU\ VRUWLQJ LV
SHUIRUPHG PHPRU\ EORFN VRUWLQJ DQG PHPRU\ PHUJH
SURFHVV
& 'HVLJQ RI &57 PHUJH SKDVH
7KLV VWDJH LV UHODWLYHO\ FRPSOH[ WKLV VWDJH RI WKH ZRUN DUH
,2: ,25 WR SDUWLFLSDWH LQ WKH ZRUN WR LPSURYH WKH ,2
SHUIRUPDQFH FDQ DV IDU DV SRVVLEOH WR UHDG DQG ZULWH ,2
WR UHGXFH FRQIOLFW WR UDLVH LWV ,2 UHVRXUFHV XWLOL]DWLRQ IRU
DV\QFKURQRXV SURFHVVLQJ LV RQH RI WKH ZD\ RI ,25 6R
EXIIHU  VSHDNLQJ UHDGLQJ DQG ZULWLQJ LV QHHGHG LQ WKH
ZKROH SURFHVV RI PHUJLQJ QDPHO\ ZRUN EXIIHU DQG WKH
EXIIHU
)LJXUH  %LJ GDWD VRUWLQJ DOJRULWKP GHVLJQ
ċ7+( .(< $/*25,7+0 $1' 86,1* 7+(
7(&+12/2*<
$ 4XLFN VRUW DOJRULWKP LV LPSURYHG
7R PDNH IXOO XVH RI &38 UHVRXUFHV E\ VRUWLQJ WKH
HQWLUH ODUJH DPRXQW RI GDWD LQWR VPDOOHU WDVNV KDUGZDUH
ORDG FDSDFLW\ LQ HTXLOLEULXP DV SRVVLEOH VR WKDW LW FDQ EH
YHU\ JRRG WR DYRLG ZDLWLQJ IRU KDUGZDUH SURFHVVLQJ
KDUGZDUH 7KLV DUWLFOH 86(6 WKH ZD\ RI GDWD SURFHVVLQJ LV
WDVN GULYH PRGH PDLQ WHFKQRORJ\ $,2 GRXEOH EXIIHU
PHFKDQLVP WDVN GULYHQ VXFK WHFKQRORJ\ WR PDNH IXOO XVH
RI V\VWHP UHVRXUFHV
% (TXLYDOHQW VRUW NH\ VWUXFWXUH
)RU TXLFN VRUWLQJ DOJRULWKP LQ RUGHU WR DFKLHYH WKH
UHFXUVLYH SURFHVV PDLQO\ 86(6 WKH ZD\ LV WKH ZD\ RI
FLUFXODWLRQ VWDFNV PHPRU\ VRUW RI ZD\ LV WKH G\QDPLF
WXPEOH GDWD EORFN IRU WKH RSWLPL]DWLRQ RI WKH VRUW NH\V E\
WKH RULJLQDO WKUHDG LV PDLQO\ FRQVWUXFWHG HTXLYDOHQW VRUW
NH\V WR VRUW NH\V 4XLFN VRUW LV PDLQO\ XVLQJ WKH VRUW NH\V
OHW WKH VRUWLQJ HIILFLHQF\ UHDFKHV WKH H[WUHPH YDOXH
& 8VH FRPSUHVVLRQ SHUIRUPDQFH OLPLWV
'XULQJ WKH HQWLUH SURFHVV RI RUGHULQJ WKH ,2 LV WKH
ERWWOHQHFN RI WKH VRUWLQJ SURFHVV D IDLUO\ ODUJH DPRXQW RI
,2 GDWD RQ WKH SUHPLVH RI JXDUDQWHH WKH LQIRUPDWLRQ LV LQ
JRRG FRQGLWLRQ E\ ORVVOHVV FRPSUHVVLRQ DOJRULWKP IRU
FRPSUHVVHG LQWHUPHGLDWH UHVXOWV LQ UHDGLQJ DQG ZULWLQJ
IRU WKH VWRUDJH VSDFH LV UHGXFHG ,Q RUGHU WR UHGXFH WKH
DPRXQW RI ,2 FDQ XVH UHGXQGDQW &38 LQ ]RQH ,2
JHQHUDOO\ ZLOO RQO\ FRPSUHVVHG JHQHUDOO\ DV ZHOO DV WKH
DERYH GDWD WKH PDLQ UHDVRQ LV WKDW WKH LQWHUPHGLDWH GDWD
LV SDUW RI WKH RUGHU WKH VKULQNDJH DQG VWUXFWXUHG GDWD FDQ
EH DERXW WR ULVH WR PRUH WKDQ  FDQ DOVR EH WKH ,2
EHGGLQJ IDFH WR UHGXFH WKH DPRXQW RI GDWD PDNH WKH
ZKROH RUGHU WR HQKDQFH WKH RYHUDOO SHUIRUPDQFH
Č7+( ',)),&8/7,(6 ,1 7+( 62/87,21
$ 3URFHVVLQJ EXIIHU ERXQGDU\
%XIIHU IURP ILOH UHDG RXW GDWD UHFRUGV LV SDUW RI WKH
UHFRUGV LQ D EXIIHU RQ WKH RWKHU SDUW LV LQ IURQW RI WKH GDWD
EXIIHU %XIIHU GDWD UHDG LV PDLQO\ XVLQJ WKH PDLQ WKUHDG
ZDLWLQJ IRU OHLVXUH WLPH DQDO\]H WKH GDWD EXIIHU GDWD WKH
UHVW RI WKH GDWD LQ D EXIIHU DQDO\VLV 5HOHYDQW GDWD UHFRUG
LV PDLQO\ DERXW WKH RXWSXW EXIIHU UHFRUGV ZULWWHQ WR WKH
EXIIHU WR MRLQ WKH ,2: WDVN OLVW LQ WKH TXHXH ,2: WDVN WR
FUHDWH ZULWH WKH UHVW RI WKH GDWD LQ D EXIIHU
% 7R FRQVWUXFW VSHFLDO VRUW NH\ GDWD W\SH
)RU FDQ W ZLWK WKH PHWKRG RI VWULQJ VWUXFWXUH HTXLYDOHQW
VRUW NH\ W\SHV DQG IORDWLQJSRLQW W\SHV PDLQO\ KDYH WLPH
2Q WKH EDVLV RI WKH VWUXFWXUH VRUW NH\V ZLWK IORDWLQJ SRLQW
1XPEHUV ELW IORDWLQJSRLQW VWRUDJH VWUXFWXUH DV VKRZQ
LQ ILJXUH 
852
Authorized licensed use limited to: Ingrid Nurtanio. Downloaded on October 01,2020 at 02:35:11 UTC from IEEE Xplore. Restrictions apply.
)LJXUH  )ORDWLQJ SRLQW GDWD VWUXFWXUH
č6800$5<
7KHVH DOJRULWKPV LPSOHPHQWHG LQ WKH GDWDEDVH WR WKH
UHTXLUHPHQW RI DOJRULWKP LV WKH JHQHUDOLW\ DQG UDWLRQDOLW\
WR PDNH DSSURSULDWH VDFULILFH RQ WKH SHUIRUPDQFH RI WKH
SURJUDP IRU GRXEOHSUHFLVLRQ IORDWLQJSRLQW QXPEHU LQ
WKH GDWDEDVH LQWHJHU VWULQJ DQG IORDWLQJSRLQW SURFHVVLQJ
DFFRUGLQJO\ 0RUHRYHU WKH PRVW LPSRUWDQW WKLQJ LV WKH
PHPRU\ UHIHUHQFH SRLQW DQG WKHQ WR WKH EHQFKPDUN
FRPSUHVVLRQ DOJRULWKP LV QRW DOORZHG WR XVH WKH ,2 WKH
LQFUHDVH RI WKH DPRXQW RI GDWD LV D NLQG RI LQYLVLEOH
:KHQ IRUHLJQ ZULWH GDWD VLPSOH DQG HFRQRPLFDO
FRPSUHVVLRQ DOJRULWKP LV D NLQG RI PHWKRG ZKLFK DUH
IUHTXHQWO\ XVHG IRU WKH ,2 RQ WKH UXQQLQJ VSHHG LV D ELJ
LQFUHDVH HYHQ WR D FHUWDLQ OLPLW )RU ODUJHU PHPRU\
PDFKLQH FDQ NHHS SDUW RI WKH DSSURSULDWH GDWD LQ WKH
PHPRU\ WR WKH FRUUHVSRQGLQJ GLVN ,2 GDWD YROXPH
UHGXFWLRQ IRU D ODUJH SURSRUWLRQ RI SHUIRUPDQFH
DFFRUGLQJO\
7KH DGYHQW RI WKH HUD RI ELJ GDWD PDNLQJ DOO NLQGV RI
ODUJHVFDOH JURXS FRPSDQ\ IRU FOXVWHU VROXWLRQ DWWHQWLRQ
PRUH DQG PRUH KLJK EXW LQ WKH FXUUHQW GRPHVWLF GDWDEDVH
LQ WHUPV RI SOD\ LV QR ZD\ WR PHHW WKH QHHGV RI GDWD
FOXVWHULQJ UHDVRQ IRU VLQJOH QRGH GDWD SURFHVVLQJ DELOLW\
RI DVFHQVLRQ DW WKLV VWDJH LV DQ XUJHQW QHHG WR VROYH WKH
SUREOHP
5()(5(1&(
>@&KHQ+RQJ\DQ :DQ-XQZHL :DQJ4L %LJ GDWD
GHVLJQ DQG LPSOHPHQWDWLRQ RI KLJK SHUIRUPDQFH VRUWLQJ
DOJRULWKP >-@ -RXUQDO RI PHDVXUHPHQW DQG FRQWURO 

>@;X+DL[LDR 0D-LQJ :X4L'%6&$1 DOJRULWKP LQ
WKH DSSOLFDWLRQ RI KLJK SHUIRUPDQFH FRPSXWLQJ FHQWHU
XVHU FODVVLILFDWLRQ VWXG\ >-@ MRXUQDO RI MLOLQ XQLYHUVLW\
 
>@&KHQ%DRSLQJ +LJK UHSHWLWLRQ UDWH GDWD RI TXLFN VRUW
>-@ (OHFWURQLF VFLHQFH DQG WHFKQRORJ\  
>@6KDQJ<DOLQJ 4LQ)HQJ +LJKO\ HIIHFWLYH TXLFN VRUW
DOJRULWKP >-@ &RPSXWHU HQJLQHHULQJ  
>@-LD;LDRIHQJ/DUJH DPRXQW RI GDWD UHDOWLPH
LQIRUPDWLRQ H[FKDQJH EDVHG RQ LRW VWUDWHJ\ UHVHDUFK >-@
(JRYHUQPHQW  
>@6RQJ+RQJVKH )X<L ;LDR/HL =KDQJ/L[LD
'HQJ=KDQKRQJ 7KH ZD\ RI PHUJLQJ PXOWLWKUHDGHG
TXLFN VRUW DOJRULWKP >-@ &RPSXWHU HGXFDWLRQ  

7KH ,PSRUWDWLRQ DQG 'HYHORSPHQW RI +LJK&DOLEHU

7DOHQWV 3URMHFW RI %HLMLQJ 0XQLFLSDO ,QVWLWXWLRQV

853

Authorized licensed use limited to: Ingrid Nurtanio. Downloaded on October 01,2020 at 02:35:11 UTC from IEEE Xplore. Restrictions apply.