2018 21st International Conference on Information Fusion (FUSION)

'HHS/HDUQLQJIRU0LOLWDU\,PDJH&DSWLRQLQJ
6XEUDWD'DV /DOLW-DLQ $UXS'DV
Data Science DivisioQ Northeastern University Data Science DivisioQ
Machine Analytics %RVWRQ0$86$ Alphaserve Technologies
%HOPRQW0$86$ MDLQO#KXVN\QHXHGX 1HZ<RUN1<86$
VGDV#PDFKLQHDQDO\WLFVFRP DGDV#DOSKDVHUYHLWFR

Abstract² 86 'HSDUWPHQW RI 'HIHQVH 'R' ELJ GDWD LV SURFHVVLQJ DQG H[WUDFWLQJ RI DFWLRQDEOH EXVLQHVV LQWHOOLJHQFH
H[WHQVLYHO\ PXOWLPRGDO DQG PXOWLSOH LQWHOOLJHQFH PXOWL,17 8QOLNHDW\SLFDOEXVLQHVVVHFWRUZKHUHELJGDWDLVFRPSULVHGRI
ZKHUH VWUXFWXUHG VHQVRU DQG XQVWUXFWXUHG DXGLR YLGHR DQG ODUJHO\ KRPRJHQHRXV WUDQVDFWLRQDO VWUXFWXUHG UHODWLRQDO GDWD
WH[WXDO,65,QWHOOLJHQFH6XUYHLOODQFHDQG5HFRQQDLVVDQFHGDWD
DUHJHQHUDWHGE\QXPHURXVDLUJURXQGDQGVSDFHERUQHVHQVRUV
DORQJZLWKKXPDQLQWHOOLJHQFH'DWDIXVLRQDWDOOOHYHOV³UHPDLQV
D FKDOOHQJLQJ WDVN´ :KLOH WKHUH DUH DOJRULWKPLF VWRYHSLSHG
V\VWHPV WKDW ZRUN ZHOO RQ LQGLYLGXDO PRGDOLWLHV WKHUH LV QR
V\VWHP WR GDWH WKDW LV PLVVLRQ DQG VRXUFH DJQRVWLFV DQG FDQ
DORQJ ZLWK WH[WXDO FRQWHQW IURP VXUYH\V EORJV DQG VRFLDO
PHGLD'R'ELJGDWDLVH[WHQVLYHO\PXOWLPRGDODQGPXOWL,17
ZKHUH VWUXFWXUHG VHQVRU DQG XQVWUXFWXUHG DXGLR YLGHR DQG
WH[WXDO ,65 GDWD LV JHQHUDWHG E\ QXPHURXV DLU JURXQG DQG
VSDFHERUQHVHQVRUVDORQJZLWKKXPDQLQWHOOLJHQFH'DWDIXVLRQ
'DVDWDOO OHYHOV³UHPDLQVDFKDOOHQJLQJWDVN´VWDUWLQJ
VHDPOHVVO\ LQWHJUDWH DQG FRUUHODWH PXOWL,17 GDWD WKDW LQFOXGHV
WH[WXDO K\SHUVSHFWUDO DQG YLGHR FRQWHQW 7KH FRQVLGHUDEOH
ZLWK WUDFNLQJ DQG DQRPDO\ GHWHFWLRQ LQ FOXWWHUHG DQG GHQVH
YROXPH DQG YHORFLW\ DVSHFWV RI ELJ GDWD RQO\ FRPSRXQG WKH HQYLURQPHQWV /HYHO  DEVWUDFWLRQ DQG FRUUHODWLRQ RI
DIRUHPHQWLRQHG HQFRXQWHUHG LQ IXVLRQ :H KDYH GHYHORSHG WKH LQIRUPDWLRQ IRU KLJKOHYHO VLWXDWLRQ DQG WKUHDW DVVHVVPHQWV
FRQFHSWRI³GHHSIXVLRQ´EDVHGRQGHHSOHDUQLQJPRGHOVDGDSWHG /HYHOV    DQG WKHQ IROORZLQJ WKURXJK RQ FROOHFWLRQ
WRSURFHVVPXOWLSOHPRGDOLWLHVRIELJGDWD5DWKHUWKDQUHGXFLQJ PDQDJHPHQW WR VXSSRUW IXVLRQ E\ RSWLPL]LQJ VHQVRU XVDJH
HDFK PRGDOLW\ LQGHSHQGHQWO\ DQG IXVLQJ DW D KLJKHUOHYHO PRGHO /HYHO:KLOHWKHUHDUHDOJRULWKPLFVWRYHSLSHGV\VWHPVWKDW
IHDWXUHOHYHOIXVLRQWKHGHHSIXVLRQDSSURDFKJHQHUDWHVDVHWRI
PXOWLPRGDO IHDWXUHV WKHUHE\ PDLQWDLQLQJ WKH FRUH SURSHUWLHV RI
WKH GLVVLPLODU VLJQDOV DQG UHVXOWLQJ LQ IXVHG PRGHOV RI KLJKHU
DFFXUDF\:HKDYHLQLWLDWHGWZRGHHSIXVLRQH[SHULPHQWV±RQHLV
WRDXWRPDWLFDOO\JHQHUDWHWKHFDSWLRQRIDQLPDJHWRKHOSDQDO\VWV
WDJJLQJ DQG FDSWLRQLQJ ODUJH YROXPHV RI LPDJHV JDWKHUHG IURP
FROOHFWLRQ SODWIRUPV DQG WKH RWKHU LV DQ DXGLRYLVXDO VSHHFK
ZRUNZHOORQLQGLYLGXDOPRGDOLWLHVWKHUHLVQRV\VWHPWRGDWH
WKDW LV PLVVLRQ DQG VRXUFH DJQRVWLFV DQG FDQ VHDPOHVVO\
LQWHJUDWH DQG FRUUHODWH PXOWL,17 GDWD WKDW LQFOXGHV WH[WXDO
K\SHUVSHFWUDODQGYLGHRFRQWHQW7KHFRQVLGHUDEOHYROXPHDQG
YHORFLW\DVSHFWVRIELJGDWDRQO\FRPSRXQGWKHDIRUHPHQWLRQHG
HQFRXQWHUHGLQIXVLRQ
FODVVLILFDWLRQ ZLWK SRWHQWLDO DSSOLFDWLRQV WR OLSUHDGLQJ DQG
HQKDQFHGREMHFWWUDFNLQJ
7KLV SDSHU SUHVHQWV WKH SURRIRIFRQFHSW GHPRQVWUDWLRQ IRU
FDSWLRQ JHQHUDWLRQ 7KH JHQHUDWLYH PRGHO LV EDVHG RQ D GHHS
UHFXUUHQW DUFKLWHFWXUH FRPELQHG ZLWK WKH SUHWUDLQHG LPDJHWR
YHFWRU PRGHO ,QFHSWLRQ 9 YLD D &RQYROXWLRQDO 1HXUDO 1HWZRUN
5HFHQW DGYDQFHV LQ GHHS OHDUQLQJ /H&XQ HW DO 
SURYLGH D WUHPHQGRXV RSSRUWXQLW\ WR WDNH D ³ZKROLVWLF´
PRGHOOLQJ DSSURDFK WR PXOWLPRGDO GDWD LQ RUGHU WR H[WUDFW
VKDUHG IHDWXUH UHSUHVHQWDWLRQ WR SHUIRUP D UDQJH RI WDVNV
VXSSRUWLQJKLJKHUOHYHOIXVLRQVXFKDVGHWHFWLQJDQRPDOLHVDQG
FKDQJHV DQG GHWHFWLQJ DQG FODVVLI\LQJ WDUJHWV :H LQWURGXFH
WKHFRQFHSWRIGHHSIXVLRQEDVHGRQVXFKGHHSOHDUQLQJPRGHOV
&11DQGWKHZRUGWRYHFWRUVPRGHOZRUGYHFYLDDVNLSJUDP DGDSWHG WR SURFHVV PXOWLSOH PRGDOLWLHV DW YDULRXV WLPH
PRGHO:HPDNHXVHRIWKH)OLFNU.GDWDVHWH[WHQGHGZLWKVRPH UHVROXWLRQV5DWKHUWKDQUHGXFLQJHDFKPRGDOLW\LQGHSHQGHQWO\
PLOLWDU\ VSHFLILF LPDJHV WR PDNH WKH GHPRQVWUDWLRQ PRUH
UHOHYDQWWRWKH'R'GRPDLQ
7KHGHWDLOHGUHVXOWVIURPWKHLPDJHFDSWLRQLQJH[SHULPHQWLV
SUHVHQWHG KHUH 7KH FDSWLRQV DUH JHQHUDWHG IURP WHVW LPDJH DUH
VXEMHFWLYHO\ HYDOXDWHG DQG WKH %/(8 ELOLQJXDO HYDOXDWLRQ
DQG IXVLQJ DW D KLJKHU OHYHO PRGHO IHDWXUHOHYHO IXVLRQ WKH
GHHS IXVLRQ DSSURDFK JHQHUDWHV D VHW RI PXOWLPRGDO IHDWXUHV
WKHUHE\ PDLQWDLQLQJ WKH FRUH SURSHUWLHV RI WKH GLVVLPLODU
VLJQDOV DQG UHVXOWLQJ LQ IXVHG PRGHOV RI KLJKHU DFFXUDF\
0RUHRYHU LQVWHDG RI WKH FXUUHQW RQHZD\ IHHG RI ORZOHYHO
XQGHUVWXG\ VFRUHV DUH FRPSDUHG DQG IRXQG VXEVWDQWLDO
LPSURYHPHQWV
Keywords— Data Fusion; Deep Learning; CNN; word2vec;
Inception V3; Flickr8K
, ,1752'8&7,21
7KH DGYHQW RIELJ GDWD SRVHV VLJQLILFDQW FKDOOHQJHV DFURVV
DOO LQGXVWULDO VHFWRUV LQ WHUPV RI HIILFLHQW VWRUDJH DQG WKH

'HHS)XVLRQ“LVDUHJLVWHUHGWUDGHPDUNRI0DFKLQH$QDO\WLFV,QF
978-0-9964527-7-9 ©2018 ISIF
WUDFNLQJGDWDWRWKHKLJKOHYHOIXVLRQDOJRULWKPVZHHQYLVLRQD
V\QHUJLVWLFIHHGEDFNLQGHHSIXVLRQ,QRXUDSSURDFKORZOHYHO
WUDFNLQJ IHHGV LPSURYHG VLWXDWLRQ DZDUHQHVV DQG KLJKOHYHO
VLWXDWLRQ DVVHVVPHQW JXLGHV WKH ORZOHYHO WUDFNLQJ E\
LQIRUPLQJ LW RI G\QDPLF FRQWH[WXDO FKDQJHV LQ WKH WUDFNHG
HQYLURQPHQW7KHIHHGEDFNLPSURYHVWKHWUDFNLQJSHUIRUPDQFH
ZKLFK LQ WXUQ SURYLGHV EHWWHU WUDFN GDWD IRU WKH VLWXDWLRQ
GHYHORSPHQWDQGXQGHUVWDQGLQJSURFHVVHV
'HHSIXVLRQZLOODOVRDOORZPRGHOOLQJQRUPDOLW\DWDYHU\
ODUJH VFDOH WR GHWHFW VXFK DQRPDOLHV DV WKRVH GHYLDWHG IURP
QRUPDOLW\ 7R JLYH RQH H[DPSOH WKH GHHS IXVLRQ DSSURDFK LV
ZHOO HTXLSSHG WR GHDO ZLWK RQH FODVV FODVVLILFDWLRQ SUREOHPV
2165
 2018 21st International Conference on Information Fusion (FUSION)

LQYROYLQJ PRGHOOLQJ RI QRUPDOLW\ IURP ELOOLRQV RI PXOWL,17 GLVWLQJXLVKWKHWDUJHWVSHFLILFFRPELQDWLRQVRIWKHPFDQ7KH

GDWD SRLQWV 7KLV NLQG RI SUREOHP LV ZHOO VXLWHG IRU GHHS WDVN RI GHHS QHWZRUN WUDLQLQJ LV WR ILQG WKH RSWLPDO SL[HO
OHDUQLQJ LQ DQ XQVXSHUYLVHG PDQQHU DV LW LV LPSRVVLEOH WR FRPELQDWLRQVWKDWOHDGWREHVWGHWHFWLRQUHVXOWV1RWHWKDWWKHVH
VSHFLI\ DOO SRVVLEOH SDWWHUQV RI FRUUHODWHG IHDWXUHV DPRQJ RSWLPDO SL[HO FRPELQDWLRQV DUH IHDWXUHV WKDW DUH FRQVWUXFWHG
VHQVRUV RI WZR GLIIHUHQW PRGDOLWLHV VXFK DV DXGLR DQG YLGHR DXWRPDWLFDOO\IURPWKHWUDLQLQJH[DPSOHVDVRSSRVHGWREHLQJ
+RZHYHUWKH\FDQEHFDSWXUHGYLDELPRGDOGHHSDXWRHQFRGHUV KDQGGHVLJQHGIRUWUDGLWLRQDOOHDUQLQJDOJRULWKPV
LQ DQ XQVXSHUYLVHG PDQQHU EDVHG RQ ODUJH YROXPHV RI GDWD
FROOHFWHGRYHUWLPH7KHPRGHOFDQWKHQEHHIIHFWLYHO\XVHGWR
GHWHFW DQRPDOLHV :KLOH FRPPHUFLDO DSSOLFDWLRQV PD\ UHTXLUH
LGHQWLI\LQJLQIRUPDWLRQZLWKUHSHDWDEOHSDWWHUQVHPHUJLQJDQG
DQRPDORXVLQIRUPDWLRQLVPRUHXVHIXOIRU'R'DSSOLFDWLRQV
'HHS/HDUQLQJLQGHHSIXVLRQDOVROLQNVPDFKLQHYLVLRQDQG
WH[W DQDO\VLV VPDUWO\ ZKHUH LPDJH LQWHOOLJHQFH ,0,1,7 DQG
KXPDQ LQWHOOLJHQFH +80,17 DUH IHG LQ ODUJH YROXPHV WR
WUDLQ D GHHS QHWZRUN WR WDJ LPDJHV RU JHQHUDWLQJ FDSWLRQV IRU
H[DPSOH +\EULG GHHS OHDUQLQJ PRGHOV LQ GHHS IXVLRQ DOVR )LJXUH&11$UFKLWHFWXUH/H&XQHWDO
OLQNV PDFKLQH YLVLRQ DQG WH[W DQDO\VLV VPDUWO\ ZKHUH ,0,1,7
DQG+80,17DUHIHGLQODUJHYROXPHVWRWUDLQDGHHSQHWZRUN 5HFXUUHQW1HXUDO1HWZRUNV511VDVVKRZQLQ)LJXUH
WR WDJ LPDJHV IRU H[DPSOH $QRWKHU DSSOLFDWLRQ LQ WDUJHW DUH D ULFK FODVV RI G\QDPLF PRGHOV WKDW KDYH EHHQ XVHG WR
UHFRJQLWLRQ XVLQJ VLJQDO LQWHOOLJHQFH 6,*,17 DQG ,0,17 LV JHQHUDWHVHTXHQFHVLQGRPDLQRIWH[WSURFHVVLQJ6XWVNHYHUHW
WROHDUQYLGHRUHSUHVHQWDWLRQVWKDWWU\WRUHFRQVWUXFWDXGLR
,, %$&.*5281',1'((3/($51,1*
'HHS OHDUQLQJ WHFKQRORJ\ DUH EHLQJ HPSOR\HG ZLWK
DOPDFKLQHWUDQVODWLRQ&KRHWDODQGLQRWKHU
GRPDLQV 511V FDQ EH WUDLQHG IRU VHTXHQFH JHQHUDWLRQ E\
SURFHVVLQJ UHDO GDWD VHTXHQFHV RQH VWHS DW D WLPH DQG
SUHGLFWLQJ ZKDW FRPHV QH[W $VVXPLQJ WKH SUHGLFWLRQV DUH
SUREDELOLVWLFQRYHOVHTXHQFHVFDQEHJHQHUDWHGIURPDWUDLQHG
LQFUHDVLQJ IUHTXHQF\ LQ UHFHQW \HDUV %HQJLR HW DO  QHWZRUN E\ LWHUDWLYHO\ VDPSOLQJ IURP WKH QHWZRUN
V RXWSXW
9DULRXV GHHS OHDUQLQJ PRGHOV KDYH DFKLHYHG UHPDUNDEOH GLVWULEXWLRQ WKHQ IHHGLQJ LQ WKH VDPSOH DV LQSXW DW WKH QH[W
UHVXOWV LQ FRPSXWHU YLVLRQ .UL]KHYVN\ HW DO  UHPRWH VWHS )LJXUH  LV DQ 511 LV D YHU\ GHHS IHHGIRUZDUG QHXUDO
VHQVLQJ=KXHWDOWDUJHWFODVVLILFDWLRQLQ6$5LPDJHV QHWZRUN WKDW KDV D OD\HU IRU HDFK WLPHVWHS ,WV ZHLJKWV DUH
&KHQ HW DO  DQG VSHHFK UHFRJQLWLRQ *UDYHV HW DO VKDUHG DFURVV WLPH 511V FDQ OHDUQ WR PDS VHTXHQFHV IRU
 'DKO HW DO  LQ UHFHQW \HDUV ,Q WKH GRPDLQ RI ZKLFKWKHDOLJQPHQWEHWZHHQWKHLQSXWVDQGRXWSXWVLVNQRZQ
QDWXUDOODQJXDJHSURFHVVLQJGHHSOHDUQLQJPHWKRGVDUHXVHGWR DKHDG RI WLPH 6XWVNHYHU HW DO  KRZHYHU LW FDQ EH
OHDUQ ZRUG YHFWRU UHSUHVHQWDWLRQV WKURXJK QHXUDO ODQJXDJH GLIILFXOWWRWUDLQWKHPWROHDUQORQJUDQJHGHSHQGHQFLHVGXHWR
PRGHOV 0LNRORY HW DO  DQG SHUIRUPLQJ FRPSRVLWLRQ WKHYDQLVKLQJJUDGLHQWSUREOHP+RFKUHLWHUHWDO
RYHU WKH OHDUQHG ZRUGYHFWRUV IRU FODVVLILFDWLRQ &ROOREHUW HW
DO
&RQYROXWLRQDOQHXUDOQHWZRUNV&11IRUH[DPSOHXWLOL]H
OD\HUVZLWKFRQYROYLQJILOWHUVWKDWDUHDSSOLHGWRORFDOIHDWXUHV
/H&XQHWDO&11LVZLGHO\XVHGIRULPDJHWDVNVDQG
LVFXUUHQWO\VWDWHRIWKHDUWIRUREMHFWUHFRJQLWLRQDQGGHWHFWLRQ
2ULJLQDOO\ LQYHQWHG IRU FRPSXWHU YLVLRQ &11 PRGHOV KDYH
VXEVHTXHQWO\ EHHQ VKRZQ WR EH HIIHFWLYH IRU 1/3 DQG KDYH
DFKLHYHG H[FHOOHQW UHVXOWV LQ VHPDQWLF SDUVLQJ <LK HW DO
 VHDUFK TXHU\ UHWULHYDO 6KHQ HW DO  VHQWHQFH )LJXUH511PRGHO6XWVNHYHUHWDO
PRGHOOLQJ .DOFKEUHQQHU HW DO  DQG RWKHU WUDGLWLRQDO
1/3WDVNV&ROOREHUWHWDO /RQJ 6KRUW7HUP 0HPRU\ /670 +RFKUHLWHU DQG
,Q WKH FRQWH[W RI &11 DV VKRZQ LQ )LJXUH  XQOLNH
WUDGLWLRQDO LPDJH SURFHVVLQJ KDQGFUDIWHG IHDWXUHV DUH QR
ORQJHU UHTXLUHG 5DZ REVHUYDWLRQV LQ LPDJH SL[HOV DUH LQSXW
GLUHFWO\ WR GHHS QHWZRUN DORQJ ZLWK WKH WUDLQLQJ ODEHOV
LQGLFDWLQJ WKH H[SHFWHG RXWFRPH IRU HDFK LQSXW 7KH QHWZRUN
DUFKLWHFWXUH FRQWDLQV OD\HUV WKDW DLP WR FDSWXUH WKH RSWLPDO
FRQILJXUDWLRQV RI UDZ REVHUYDWLRQV VXFK WKDW WKH QXPEHU RI
FRUUHFW LGHQWLILFDWLRQV DJDLQVW SURYLGHG ODEHOV LV PD[LPL]HG
6FKPLGKXEHU  DV VKRZQ LQ )LJXUH  LQFRUSRUDWH
H[SOLFLWO\ FRQWUROODEOH PHPRU\ XQLWV WR OHDUQ ORQJUDQJH
WHPSRUDO GHSHQGHQFLHV $W WKH FRUH RI WKH /670 PRGHO LV D
PHPRU\FHOOZKLFKHQFRGHVDWHYHU\WLPHVWHSWKHNQRZOHGJH
RIWKHLQSXWVWKDWKDYHEHHQREVHUYHGXSWRWKDWVWHS7KHFHOOLV
PRGXODWHG E\ JDWHV ZKLFK DUH DOO VLJPRLGDO 7KH JDWHV
GHWHUPLQHZKHWKHUWKH/670NHHSVWKHYDOXHIURPWKHJDWHRU
GLVFDUGVLW7KHIROORZLQJWKUHHJDWHVHQDEOHWKH/670WROHDUQ
)RU H[DPSOH LQGLYLGXDO SL[HO YDOXHV LQ DQ LPDJH GR QRW FRPSOH[ORQJWHUPGHSHQGHQFLHVWKHLQSXWJDWHLFRQWUROV
FRQWDLQVXIILFLHQWFXHVWRGHFLGHZKHWKHUWKHLPDJHFRQWDLQVD ZKHWKHUWKH/670FRQVLGHUVLWVFXUUHQWLQSXW WKHIRUJHW
WDUJHWREMHFWRUQRW+RZHYHUFRPELQDWLRQVRISL[HOVJURXSHG
LQGLIIHUHQWZD\VFDQFUHDWHDPRGHOWKDWDFFXUDWHO\GHWHFWVWKH JDWH I DOORZV WKH /670 WR IRUJHW LWV SUHYLRXV PHPRU\ 
GHVLUHG WDUJHW 7DNHQ WRJHWKHU ZKLOH UDZ LQSXWV FDQQRW DQGWKHRXWSXWJDWHRGHFLGLQJKRZPXFKRIWKHPHPRU\WR

2166
 2018 21st International Conference on Information Fusion (FUSION)

WUDQVIHUWRWKHKLGGHQVWDWH 7KHUHFXUUHQWHTXDWLRQVDUHWKHQ ,,, '((3)86,21$5&+,7(&785(

GHILQHGDVIROORZV 7KH VFRSH RI GHHS IXVLRQ LV ODUJH 7KLV VHFWLRQ GHWDLOV WKH
DUFKLWHFWXUH RI RQH DSSOLFDWLRQ RI GHHS IXVLRQ QDPHO\ LPDJH
FDSWLRQLQJ )RU LPDJH FDSWLRQLQJ K\EULG 511&11 GHHS
OHDUQLQJPRGHOOLQNVPDFKLQHYLVLRQDQGWH[WDQDO\VLVVPDUWO\
ZKHUH,0,1,7DQG+80,17DUHIHGLQODUJHYROXPHVWRWUDLQ
WKHK\EULGPRGHO

A. Automated Image Captioning

7KHUH KDV EHHQ DQ LQFUHDVLQJ LQWHUHVW LQ VWLOO LPDJH
FDSWLRQLQJ LQ QDWXUDO ODQJXDJH )DUKDGL HW DO  XVH
GHWHFWLRQV WR LQIHU D WULSOHW RI VFHQH HOHPHQWV ZKLFK LV
FRQYHUWHGWRWH[WXVLQJWHPSODWHV6LPLODUO\)HQJHWDO
VWDUW RII ZLWKGHWHFWLRQV DQGSLHFH WRJHWKHUD ILQDO GHVFULSWLRQ
XVLQJSKUDVHVFRQWDLQLQJGHWHFWHGREMHFWVDQGUHODWLRQVKLSV$
 PRUH FRPSOH[ JUDSK RI GHWHFWLRQV EH\RQG WULSOHWV LV XVHG E\
.XONDUQLHWDOEXWZLWKWHPSODWHEDVHGWH[WJHQHUDWLRQ
)LJXUH/670&HOO*UHIIHWDO 0RUH SRZHUIXO ODQJXDJH PRGHOV EDVHG RQ ODQJXDJH SDUVLQJ
KDYHEHHQXVHG.X]QHWVRYDHWDO
 2XU LQLWLDO DSSURDFK DQG LWV H[WHQVLRQ DQG LPSOHPHQWDWLRQ
LV DORQJ WKH OLQH RI WKH 6KRZ DQG 7HOO PRGHO 9LQ\OV HW DO
   ,W ZRUNV E\ ILUVW ³HQFRGLQJ´ DQ LPDJH LQWR D IL[HG
 OHQJWK YHFWRU UHSUHVHQWDWLRQ DQG WKHQ ³GHFRGLQJ´ WKH
UHSUHVHQWDWLRQ LQWR D QDWXUDO ODQJXDJH GHVFULSWLRQ 7KH LPDJH
 HQFRGHU LV D &11 $V PHQWLRQHG LQ 9LQ\OV HW DO  WKH
 FKRLFHRIQHWZRUNLVWKH,QFHSWLRQYLPDJHUHFRJQLWLRQPRGHO
SUHWUDLQHG RQ WKH ,/695&&/6 LPDJH FODVVLILFDWLRQ
 GDWDVHW 6]HJHG\ HW DO  7KH GHFRGHU LV D /670
ZKHUH WKH RSHUDWRUV DQG IXQFWLRQV V g DQG h
DUH DV GHVFULEHG LQ WKH OHJHQG ER[ RI WKH ILJXUH 7KH ZHLJKW
PDWULFHVGHQRWHGE\ DUHWKHWUDLQHGSDUDPHWHUV
'HHS OHDUQLQJ PRGHOV DOVR KDYH DQ H[FHOOHQW FDSDELOLW\ WR
WUDQVIHU NQRZOHGJH OHDUQHG IURP RQH SUREOHP WR UHODWHG
SUREOHPV WKHUHE\ JUHDWO\ UHGXFLQJ WKH DPRXQW RI WLPH DQG
H[DPSOHVQHHGHGWRSHUIRUPRQDQHZSUREOHP
QHWZRUN 7KH /670 QHWZRUN LV WUDLQHG DV D ODQJXDJH PRGHO
FRQGLWLRQHGRQWKHLPDJHHQFRGLQJ:RUGVLQWKHFDSWLRQVDUH
UHSUHVHQWHG ZLWK DQ HPEHGGLQJ PRGHO (DFK ZRUG LQ WKH
YRFDEXODU\ LV DVVRFLDWHG ZLWK D IL[HGOHQJWK YHFWRU
UHSUHVHQWDWLRQ WKDW LV OHDUQHG GXULQJ WUDLQLQJ )LJXUH 
LOOXVWUDWHVWKHPRGHODUFKLWHFWXUH,QWKLVGLDJUDP^[[`
DUH WKH ZRUG HPEHGGLQJ YHFWRUV RI WKH FDSWLRQ 7KH RXWSXWV
^K K ` RI WKH /670 DUH SUREDELOLW\ GLVWULEXWLRQV
JHQHUDWHGE\WKHPRGHOIRUWKHQH[WZRUGLQWKHVHQWHQFH

Inceptionv3Ͳ DeepCNN

Agroupof
soldiersare
standingin
frontofa
building
LSTM LSTM LSTM LSTM
Twomenare
posingfora
photograph
LanguageGenerationRNN

)LJXUH+\EULG&11511DUFKLWHFWXUHIRULPDJHFDSWLRQLQJ

2167
 2018 21st International Conference on Information Fusion (FUSION)
'XULQJ WKH ILUVW SKDVH RI WUDLQLQJ WKH SDUDPHWHUV RI WKH
,QFHSWLRQ Y PRGHO DUH NHSW IL[HG LW LV VLPSO\ D VWDWLF LPDJH
HQFRGHU IXQFWLRQ $ VLQJOH WUDLQDEOH OD\HU LV DGGHG RQ WRS RI
WKH,QFHSWLRQYPRGHOWRWUDQVIRUPWKHLPDJHHPEHGGLQJLQWR
WKH ZRUG HPEHGGLQJ YHFWRU VSDFH 7KH PRGHO LV WUDLQHG ZLWK
UHVSHFW WR WKH SDUDPHWHUV RI WKH ZRUG HPEHGGLQJV WKH
SDUDPHWHUV RI WKH OD\HU RQ WRS RI ,QFHSWLRQ Y DQG WKH
SDUDPHWHUV RI WKH /670 ,Q WKH VHFRQG SKDVH RI WUDLQLQJ DOO
SDUDPHWHUV  LQFOXGLQJ WKH SDUDPHWHUV RI ,QFHSWLRQ Y  DUH
WUDLQHG WR MRLQWO\ ILQHWXQH WKH LPDJH HQFRGHU DQG WKH /670
*LYHQDWUDLQHGPRGHODQGDQLPDJHFDSWLRQVDUHJHQHUDWHGYLD
DEHDPVHDUFK&DSWLRQVDUHJHQHUDWHGZRUGE\ZRUGZKHUHDW
HDFKVWHSWWKHVHWRIVHQWHQFHVDOUHDG\JHQHUDWHGZLWKOHQJWKW
DUHXVHGWRJHQHUDWHDQHZVHWRIVHQWHQFHVZLWKOHQJWKWE\
NHHSLQJ RQO\ D FHUWDLQ QXPEHU RI FDQGLGDWHV VSHFLILHG DV WKH
DUH PRUH FRPSXWDWLRQDOO\ H[SHQVLYH VR ILUVW GRLQJ D î
FRQYROXWLRQ UHGXFHV WKH GLPHQVLRQDOLW\ RI LWV IHDWXUH PDS
SDVVLQJ WKH UHVXOWLQJ IHDWXUH PDS WKURXJK D UHOX DQG WKHQ
GRLQJ WKH ODUJHU FRQYROXWLRQV 7KH î FRQYROXWLRQ LV NH\
EHFDXVH LW ZLOO EH XVHG WR UHGXFH WKH GLPHQVLRQDOLW\ RI LWV
IHDWXUHPDS7KHFKDQJHWR,QFHSWLRQYZDVWKDWWKH\UHSODFHG
WKH[FRQYROXWLRQVE\WZRVXFFHVVLYH[FRQYROXWLRQVDQG
DSSOLHG SRROLQJ ,Q ,QFHSWLRQ Y WKH IXOO\ FRQQHFWHG OD\HU RI
WKH DX[LOLDU\ FODVVLILHU LV DOVRQRUPDOL]HG QRW MXVW
FRQYROXWLRQV 2QH SDUWLFXODU LQFDUQDWLRQ RI ,QFHSWLRQ
DUFKLWHFWXUH*RRJ/H1HWDOD\HUVGHHSQHWZRUNDVVKRZQLQ
)LJXUH
C. Word2vec for Word Embedding
7KLV PRGHO ZRUGYHF LV XVHG IRU OHDUQLQJ YHFWRU
EHDPVL]HHJ
UHSUHVHQWDWLRQV RI ZRUGV FDOOHG word embeddings ZKLFK
5HSUHVHQWLQJ ZRUGV DV XQLTXH GLVFUHWH LGV OHDGV WR GDWD FRPHV LQ WZR IODYRUV WKH &RQWLQXRXV %DJRI:RUGV PRGHO
VSDUVLW\ DQG XVXDOO\ PHDQV WKDW ZH PD\ QHHG PRUH GDWD LQ &%2: DQG WKH 6NLS*UDP PRGHO 0LNRORY HW DO «
RUGHU WR VXFFHVVIXOO\ WUDLQ VWDWLVWLFDO PRGHOV 8VLQJ YHFWRU &%2:SUHGLFWVWDUJHWZRUGVHJ³PDW´IURPVRXUFHFRQWH[W
UHSUHVHQWDWLRQV FDQ RYHUFRPH VRPH RI WKHVH REVWDFOHV ,Q RXU ZRUGV ³WKH FDW VLWV RQ WKH´ ZKLOH WKH VNLSJUDP GRHV WKH
RQJRLQJIROORZRQH[SHULPHQWDWLRQZHPDNHXVHRIZRUGYHF LQYHUVHSUHGLFWLQJVRXUFHFRQWH[WZRUGVIURPWKHWDUJHWZRUGV
SUHWUDLQHGPRGHOVIRUZRUGHPEHGGLQJYHFWRUVZKHUHLQZRUGV
DUHSURMHFWHGIURPWKHFXUUHQWVSDUVHRI9HQFRGLQJKHUH9 
LVWKHYRFDEXODU\VL]HRQWRDORZHUGLPHQVLRQDOYHFWRUVSDFH
YLDDKLGGHQOD\HUDUHHVVHQWLDOO\IHDWXUHH[WUDFWRUVWKDWHQFRGH
VHPDQWLFIHDWXUHVRIZRUGVLQWKHLUGLPHQVLRQV$VLPSOH&11
ZLWKRQHOD\HURIFRQYROXWLRQRQWRSRIZRUGYHFWRUVREWDLQHG
IURP DQ XQVXSHUYLVHG QHXUDO ODQJXDJH PRGHO 7KHVH YHFWRUV
ZHUHWUDLQHGE\0LNRORYHWDORQELOOLRQZRUGVRI
*RRJOH 1HZV DQG DUH SXEOLFO\ DYDLODEOH ,Q VXFK GHQVH
UHSUHVHQWDWLRQVVHPDQWLFDOO\FORVHZRUGVDUHOLNHZLVHFORVH²
LQ (XFOLGHDQ RU FRVLQH GLVWDQFH²LQ WKH ORZHU GLPHQVLRQDO
YHFWRUVSDFH

B. Inception v3 for Image Encoding


7KHRULJLQDOGHHS&11,QFHSWLRQDUFKLWHFWXUH6]HJHG\HW
DOZDVGHYHORSHGIRUFODVVLILFDWLRQDQGGHWHFWLRQLQWKH )LJXUHZRUGYHFPRGHO
,PDJH1HW /DUJH6FDOH 9LVXDO 5HFRJQLWLRQ &KDOOHQJH 
,/695& 7KH PDLQ KDOOPDUN RI WKLV DUFKLWHFWXUH LV
7KH &%2: DQG VNLSJUDP PRGHOV DUH WUDLQHG XVLQJ D
LQFUHDVHG WKH GHSWK DQG ZLGWK RI WKH QHWZRUN IRU LPSURYHG
ELQDU\ FODVVLILFDWLRQ REMHFWLYH ORJLVWLF UHJUHVVLRQ WR
XWLOL]DWLRQ RI WKH FRPSXWLQJ UHVRXUFHV )LJXUH  VKRZV WKH
GLVFULPLQDWH WKH UHDO WDUJHW ZRUGV IURP LPDJLQDU\ QRLVH
DUFKLWHFWXUHRIDVLQJOHLQFHSWLRQPRGXOH
ZRUGV LQ WKH VDPH FRQWH[W )LJXUH  LV IRU D &%2: PRGHO
 2QFHWKHPRGHOLVWUDLQHGWKHKLGGHQYHFWRUUHSUHVHQWDWLRQIRU
D JLYHQ ZRUG RU D VHTXHQFH RI ZRUGV SURYLGHV WKH
FRUUHVSRQGLQJ YHFWRU UHSUHVHQWDWLRQ ZKLFK DUH WKHQ LQSXW WR
DQGWUDLQWKH511LQ)LJXUH

,9 352727<3(,03/(0(17$7,21$1'75$,1,1*
7KHUH DUH PDQ\ RSHQ VRXUFH GHHS OHDUQLQJ OLEUDULHV LQ 5
3\WKRQ DQG -DYD EXW RXU LPSOHPHQWDWLRQ LV EDVHG RQ 3\WKRQ
EDVHG 7HQVRU)ORZ $EDGL HW DO  7HQVRU)ORZ FRUH LV
ZULWWHQ LQ & 7KHUH DUH PDQ\  RWKHU  V\VWHPV  WKDW  DUH
FRPSDUDEOH  LQ YDULRXV ZD\V ZLWK 7HQVRU)ORZ  7KHDQR
 %HUJVWUDHWDO7RUFK&ROOREHUWHWDODQG&DIIH
)LJXUH2ULJLQDO,QFHSWLRQPRGXOH
7KH PRGXOH XVHV î î DQG î FRQYROXWLRQV DORQJ
ZLWKDîPD[SRROLQJ7KHODUJHUFRQYROXWLRQVîDQGî
-LDHWDODUHDIHZV\VWHPVGHVLJQHGSULPDULO\IRUWKH
WUDLQLQJRIGHHSQHWZRUNVEXW7HQVRU)ORZVXSSRUWVGLVWULEXWHG
FRPSXWLQJ 7HQVRU)ORZ DQG 7KHDQR XVH D GHFODUDWLYH
SURJUDPPLQJSDUDGLJPPHDQLQJVSHFLI\LQJZKDWQHHGVWREH

2168
 2018 21st International Conference on Information Fusion (FUSION)
FRPSXWHG UDWKHU WKDQ KRZ H[DFWO\ DQG LQ ZKDW RUGHU WKHVH
FRPSXWDWLRQV DUH WR EH SHUIRUPHG 7KH ODWWHU LV WKH FDVH LQ
LPSHUDWLYHSURJUDPPLQJOLNH-DYDRUSURJUDPPLQJLQVFULSWLQJ
ODQJXDJHV OLNH 5 DQG 3\WKRQ $ GHHS OHDUQLQJ PRGHO LV WKHQ
UHSUHVHQWHG E\ DQ DEVWUDFW V\PEROLF FRPSXWDWLRQDO JUDSK
ZKLFK FDQ EH RSWLPL]HG IRU SHUIRUPDQFH ,QGLYLGXDO SDUWV RI
VXFKDGDWDGULYHQJUDSKFDQEHWUDQVODWHGWREHH[HFXWHGE\D
FRPSXWHUSURFHVVRURUJUDSKLFVFKLS
6RPH OLEUDULHV RIIHU DGYDQWDJHV LQ WHUPV RI XWLOL]LQJ SUH
x YDOLGBZLQGRZ     2QO\ SLFN GHY VDPSOHV LQ WKH
KHDGRIWKHGLVWULEXWLRQ
x QXPBVDPSOHG 1XPEHURIQHJDWLYHH[DPSOHVWR
VDPSOH
7UDLQLQJ LV VHW WR  HSRFKV 2QFH WKH WUDLQLQJ RI
ZRUG9HFPDWUL[LVFRPSOHWHGLWFRPHVEDFNWRWKHSURFHVVLQJ
RIWKHLPDJHVGDWDVHWDQGVDYHVLWWRDILOHZKLFKZHFRXOGXVH
ODWHU
WUDLQHG PRGHOV VXFK DV &DIIH & 7RUFK /XD IHDWXUHV 1RZ RQFH WKH GDWD LV SURFHVVHG WR VXLW WKH QHHGV RI
IXQFWLRQDOH[WHQVLELOLW\7KHDQR3\WKRQSURYLGHVDORZOHYHO 7HQVRU)ORZ ZH DUH QRZ UHDG\ WR WUDLQ WKH PRGHO :H KDYH
OLEUDU\IRUFRQVWUXFWLQJPDWKHPDWLFDOH[SUHVVLRQVDQGKLJKHU FKDQJHG WKH VKRZ DQG WHOO PRGHO S\WKRQ ILOH VSHFLILFDOO\
OHYHOZUDSSHUV.HUDV&KROOHWZLWKLQWXLWLYHLQWHUIDFHV FKDQJHG WKH GHILQLWLRQ RI FHUWDLQ SURFHGXUH
IRUPRGHOOHYHOPDQLSXODWLRQ7KHUHDUH-DYDEDVHGWRROVVXFK
DV GHHSOHDUQLQJM DQG +2 ZLWK SODWIRUP LQGHSHQGHQW $3,V
+2SURYLGHVLQWHUIDFHVWR3\WKRQ56FDODDQGRWKHUV0DQ\
WRROVRIIHU*38EDVHGVSHHGXSV
,Q RXU SURRIRIFRQFHSW GHPRQVWUDWLRQ IRU FDSWLRQ
JHQHUDWLRQ WKH JHQHUDWLYH PRGHO LV EDVHG RQ D GHHS UHFXUUHQW
DUFKLWHFWXUH FRPELQHG ZLWK WKH SUHWUDLQHG LPDJHWRYHFWRU
PRGHO,QFHSWLRQ9YLDD&11DQGWKHZRUGWRYHFWRUVPRGHO
ZRUGYHFYLDDVNLSJUDPPRGHO7KHPRGHOLVLPSOHPHQWHGLQ
EXLOGBVHTBHPEHGGLQJV WR LQFOXGH ZRUGYHF HPEHGGLQJ
PDWUL[ JHQHUDWHG HDUOLHU $IWHU WKH PRGHO LV WUDLQHG DIWHU 
PLOOLRQ HSRFKV RYHU D SHULRG RI  GD\V ZH FDQ VHH WKH
LQIHUHQFHE\UXQQLQJWKHLQIHUHQFHPRGXOH
9 5(68/76$1'(9$/8$7,21
)LJXUHEHORZSURYLGHVVRPHH[DPSOHLPDJHVDORQJZLWK
RULJLQDOOPW[WDQGRXUPRGLILHGFDSWLRQV$TXLFNVXEMHFWLYH
HYDOXDWLRQ FRPSDULQJ RULJLQDO ZLWK PRGLILHG FDSWLRQV UHYHDOV
3\WKRQ EDVHG RQ WKH LPW[W FRGH RI 9LQ\DOV HW DO  EXW WKDWWKHODWWHUDUHEHWWHUVXLWHGLQPRVWFDVHV)RU%/(8VFRUH
H[WHQVLYHO\ PRGLILHG WR LQFRUSRUDWH RXU ZRUGYHFEDVHG FDOFXODWLRQ IRU WKH HYDOXDWLRQ VHW ZKLFK LQFOXGHV WKHVH
HQFRGLQJ:HPDNHXVHRIWKH)OLFNU.GDWDVHWH[WHQGHGZLWK
VRPHPLOLWDU\VSHFLILFLPDJHVWRPDNHWKHGHPRQVWUDWLRQPRUH
UHOHYDQW WR WKH 'R' GRPDLQ :H KDYH FDUULHG RXW VRPH
H[WHQVLYH HYDOXDWLRQ EXW FDSWLRQV OLNH WKH RQHV LQ )LJXUH 
JHQHUDWHGE\WKHPRGHOLQWXLWLYHO\PDNHVHQVHVRIDU
:H DUH XVLQJ WKH LPW[W RULJLQDO FRGH IURP WKH *LW+XE
OLEUDU\ DV WKH EDVH DQG PRGLI\LQJ WKH FRGH WR LQSXW RXU
FKDQJHV6LQFHZHDUHXVLQJ)OLFNU'DWDVHWDQGQRW026&22
GDWDVHW ZH PDGH DGMXVWPHQW LQ WKH SUHSDULQJ WKH GDWDVHW )RU
JHQHUDWLQJ VHTXHQFH VWUHDP IRU UHDGLQJ 7) IRUPDW DQG
FDSWLRQVFRGHLVDOWHUHGWRVXLWRXWGDWDQHHGV2QFHWKHGDWDLV
VDYHG LQ 7)5HFRUG IRUPDW ZH DUH DOVR ODXQFKLQJ WKH
JHQHUDWLRQ RI D (PEHGGLQJ 9HFWRU S\WKRQ ILOH ZKLFK ZLOO
HVVHQWLDOO\ WDNH WKH YRFDEXODU\ VL]H DQG RWKHU FRQILJXUDWLRQ
SDUDPHWHUWRFUHDWHWKHHPEHGGLQJYHFWRUV
7KH IROORZLQJ SDUDPHWHUV DUH VHW RI WKH ZRUGYHF
FRQILJXUDWLRQ
x EDWFKBVL]H 
x HPEHGGLQJBVL]H 'LPHQVLRQRIWKHHPEHGGLQJ
YHFWRU
x VNLSBZLQGRZ +RZPDQ\ZRUGVWRFRQVLGHUOHIW
DQGULJKW
x QXPBVNLSV +RZPDQ\WLPHVWRUHXVHDQLQSXWWR
JHQHUDWHDODEHO
:H SLFN D UDQGRP YDOLGDWLRQ VHW WR VDPSOH QHDUHVW
LPDJHV ZH KDYH XVHG 1/7. SDFNDJH 2XU VFRULQJ  YV
WKH VFRULQJ RQ RULJLQDO  LV DW WKH FRUSXV OHYHO ZLWK QR
VPRRWKLQJ DQG  ZHLJKWV IRU JUDP DSSURDFK %/(8 VFRUH
UHOLHVRQQXPEHURIZRUGVLQWKHFDQGLGDWHYVWKHUHIHUHQFH,W
IDYRUVWKHFDQGLGDWHKDYLQJVKRUWHQOHQJWKDVSHUWKHIRUPXODLW
ZRUNV RQ DOWKRXJK EUHYLW\ SHQDOW\ WDNHV FDUH RI LW $OVR ZH
DOZD\V JHW DQ HUURU PHVVDJH IRU VXFK H[DPSOHV DV QR FRXQW
ZDV IRXQG IRU JUDP RU VR ZH KDYH NHSW PDWFKLQJ RI 
JUDPV,IWKLVRUGHURIQJUDPUHWXUQVFRXQWVWKHKLJKHURUGHU
QJUDP ZRXOG DOVR UHWXUQ  WKXV EUHDNLQJ WKH ORRS KHUH 2XU
JHQHUDO REVHUYDWLRQ LV WKDW ZKHQ RXU %/(8 VFRUH ZDV ORZHU
EHFDXVHVHQWHQFHVUHSHDWDQGLVVLPLODU
9, &21&/86,216
,Q WKLV SDSHU ZH KDYH GHYHORSHG DQG GHPRQVWUDWHG WKH
GHHS IXVLRQ FRQFHSW IRU LPDJH FDSWLRQLQJ XVLQJ D K\EULG
&11511 GHHS QHWZRUN PRGHOV :H KDYH PDGH XVH RI
,QFHSWLRQ DQG ZRUGYHF PRGHOV IRU HQFRGLQJ WUDLQLQJ LPDJHV
DQGWH[WXDOFDSWLRQVUHVSHFWLYHO\:HKDYHHYDOXDWHGWKHUHVXOW
ERWKVXEMHFWLYHO\DQGREMHFWLYHO\ZLWK%/(8VFRUH:HSODQWR
FRQWLQXHHQKDQFLQJWKHSHUIRUPDQFHRIWKLVK\EULGGHHSIXVLRQ
PRGHORYHUWKHQH[WIHZPRQWKVXVLQJDODUJHUFRUSXV
:H SODQ IRU DQRWKHU H[SHULPHQW RQ DXGLRYLVXDO VSHHFK
FODVVLILFDWLRQ ZLWK SRWHQWLDO DSSOLFDWLRQV WR OLSUHDGLQJ DQG
HQKDQFHGREMHFWWUDFNLQJLQHQYLURQPHQWZLWKRFFOXVLRQRIRQH
RI WKH PRGDOLWLHV )RU DXGLRYLGHR FODVVLILFDWLRQ ZH SODQ WR
PDNHXVHRIDFRPELQDWLRQRIGHHSDXWRHQFRGHUVDVGHVFULEHG
LQ 1JLDP HW DO  7KH DUFKLWHFWXUH RI WKH ELPRGDO GHHS
QHLJKERXUV+HUHZHOLPLWWKHYDOLGDWLRQVDPSOHVWRWKHZRUGV
DXWRHQFRGHU IRU WKH DXGLRYLVXDO FODVVLILHU LV EDVHG RQ
WKDWKDYHDORZQXPHULF,'ZKLFKE\FRQVWUXFWLRQDUHDOVRWKH
GHQRLVLQJDXWRHQFRGHUV9LQFHQWHWDO
PRVWIUHTXHQW

x YDOLGBVL]H     5DQGRP VHW RI ZRUGV WR HYDOXDWH
VLPLODULW\RQ

2169
 2018 21st International Conference on Information Fusion (FUSION)

,PDJH 2ULJLQDO&DSWLRQ 0RGLILHG&DSWLRQ

 DZKLWHGRJUXQVRQWKHEHDFK  DEURZQDQGZKLWHGRJLV
S  UXQQLQJWKURXJKWKHVDQG
 DZKLWHGRJLVUXQQLQJRQWKH S 
EHDFKS   DEURZQDQGZKLWHGRJLV
 DZKLWHGRJZLWKEURZQNLFN UXQQLQJWKURXJKWKHGLUW
UXQVRQWKHVDQGS  S 
 DEURZQDQGZKLWHGRJLV
UXQQLQJWKURXJKWKHZDWHU
 S 
 DERWWOHEDUHRYHUVL]HG  DERWWOHODQGLQWKHWUXFN
S  S 
 DERWWOHODQGLQWKHWUXFN  DERWWOHODQGRQWKHWUXFN
S  S 
 DERWWOHNLGRYHUWKHJURXSRI  DPLOLWDU\ERWWOHNLGRYHUWKH
SLOHS  RFHDQS 


 
 DVROGLHULVZDONLQJZLWKD  DVROGLHULVSOD\LQJZLWKDGRJ
URFN\LQKLVKDQG S 
S   VROGLHULVVWDQGLQJZLWKKLV
 WKHVROGLHULVKROGLQJDURFN\ URFN\S 
S   VROGLHULVVWDQGLQJZLWKURFN\LQ
 WKHVROGLHULVKROGLQJDEODFN KLVKDQGS 
URFN\LQKLVKDQG
S 

 EDVNHWEDOODUHPRVWO\LQEDW  DVROGLHULVZDONLQJZLWKD
DSSURDFKHV81.! URFN\LQDVQRZERDUGDZD\
S  S 
 EDVNHWEDOODUHPRVWO\LQDWXOLSV  DVROGLHULVZDONLQJZLWKD
RIEDWFDWFKHUS  URFN\LQDXQLIRUPVIO\
 DVROGLHULVO\LQJRQWKHJURXQG S 
ZLWKDURFN\LQKLVKDQG  DVROGLHULVZDONLQJZLWKD
S  URFN\LQDVQRZERDUGIO\
S 

 DVWXQWWUDLQLQJLVUXQQLQJ  DVROGLHULVSOD\LQJZLWKDGRJ
WKURXJKWKHJUDVV S 
S   DVROGLHULVVWDQGLQJZLWKDGRJ
 DVWXQWWUDLQLQJLVUXQQLQJ S 
WKURXJKDVWLFNDUHD  DVROGLHULVVWDQGLQJZLWKKLV
S  URFN\S 
 DVWXQWWUDLQLQJGRJLVUXQQLQJ
WKURXJKWKHJUDVV
S 

 
)LJXUH,PDJHFDSWLRQLQJUHVXOWV

2170
 2018 21st International Conference on Information Fusion (FUSION)

5()(5(1&(6 >@ .UL]KHYVN\$6XWVNHYHU,DQG+LQWRQ*³,PDJH1HW

&ODVVLILFDWLRQ ZLWK 'HHS &RQYROXWLRQDO 1HXUDO 1HWZRUNV´
Advances in Neural Information Processing SystemsSS±
>@ $EDGL 0 $JDUZDO $ %DUKDP 3 %UHYGR ( &KHQ = 
&LWUR & &RUUDGR *6 'DYLV $ 'HDQ - 'HYLQ 0 HW DO
 ³7HQVRU)ORZ /DUJHVFDOH PDFKLQH OHDUQLQJ RQ >@ .XONDUQL * 3UHPUDM 9 'KDU 6 /L 6 &KRL < %HUJ $
KHWHURJHQHRXVV\VWHPV´ DQG%HUJ7³%DE\WDON8QGHUVWDQGLQJDQGJHQHUDWLQJ
VLPSOHLPDJHGHVFULSWLRQV´Proceedings of CVPR
>@ %DKGDQDX'&KR.DQG%HQJLR<³1HXUDOPDFKLQH
WUDQVODWLRQ E\ MRLQWO\ OHDUQLQJ WR DOLJQ DQG WUDQVODWH´ >@ .X]QHWVRYD 3 2UGRQH] 9 %HUJ 7 DQG &KRL < 
KWWSVDU[LYRUJDEV ³7UHHWDON &RPSRVLWLRQ DQG FRPSUHVVLRQ RI WUHHV IRU LPDJH
GHVFULSWLRQV´ACL9RO
>@ %HQJLR < +LQWRQ * /H&XQ <  ³'HHS /HDUQLQJ´
1DWXUHYROSS± >@ /H&XQ < %RWWRX / %HQJLR < DQG +DIIQHU 3 
³*UDGLHQWEDVHG OHDUQLQJ DSSOLHG WR GRFXPHQW UHFRJQLWLRQ´
>@ %HUJVWUD-%UHXOHX[2%DVWLHQ)/DPEOLQ33DVFDQX5 Proceedings of the IEEE9ROSS±
'HVMDUGLQV * 7XULDQ - :DUGH)DUOH\ ' DQG %HQJLR <
³7KHDQR$&38DQG*380DWK&RPSLOHULQ3\WKRQ´ >@ /H&XQ<%HQJLR<DQG+LQWRQ*'HHSOHDUQLQJ
Proceedings of the Python for scientific computing conference 1DWXUH9ROSS±
(SciPy)9ROSS$XVWLQ7; >@ 0DUWLQH] + DQG <DQQDNDNLV *  ³'HHS PXOWLPRGDO
>@ &DPSV9DOOV*³)URP.HUQHO0HWKRGVWR'HHS1HXUDO IXVLRQ &RPELQLQJ GLVFUHWH HYHQWV DQG FRQWLQXRXV VLJQDOV´
1HWZRUNVLQ5HPRWH6HQVLQJ'DWD3URFHVVLQJ´ 3URFHHGLQJVRIWKHWK,QWHUQDWLRQDO&RQIHUHQFHRQ0XOWLPRGDO
,QWHUDFWLRQSS
>@ &KHQ 6 :DQJ + ;X ) DQG -LQ <  ³7DUJHW
&ODVVLILFDWLRQ8VLQJWKH'HHS&RQYROXWLRQDO1HWZRUNVIRU6$5 >@ 0LNRORY 7 &KHQ . &RUUDGR * DQG 'HDQ - 
,PDJHV´ LQ IEEE Transactions on Geoscience and Remote ³(IILFLHQW (VWLPDWLRQ RI :RUG 5HSUHVHQWDWLRQV LQ 9HFWRU
Sensing9RO1RSS 6SDFH´KWWSVDU[LYRUJDEV
>@ &KR . YDQ 0HUULHQERHU % *XOFHKUH & %RXJDUHV ) >@ 0LNRORY76XWVNHYHU,&KHQ.&RUUDGR*DQG'HDQ-
6FKZHQN + DQG %HQJLR <  ³/HDUQLQJ SKUDVH ³'LVWULEXWHG5HSUHVHQWDWLRQVRI:RUGVDQG3KUDVHVDQG
UHSUHVHQWDWLRQV XVLQJ 511 HQFRGHUGHFRGHU IRU VWDWLVWLFDO WKHLU&RPSRVLWLRQDOLW\´Proceedings of NIPS
PDFKLQHWUDQVODWLRQ´3URFHHGLQJVRI(01/3 >@ 1HYHURYD 1 :ROI & 7D\ORU * DQG 1HERXW ) 
>@ &KROOHW)³.HUDV7KHDQREDVHG'HHS/HDUQLQJOLEUDU\ ³0XOWLVFDOH GHHS OHDUQLQJ IRU JHVWXUH GHWHFWLRQ DQG
´LQ*LW+XE ORFDOL]DWLRQ´ (&&9 &KD/HDUQ :RUNVKRS RQ /RRNLQJ DW
3HRSOH
>@ &ROOREHUW 5 :HVWRQ - %RWWRX / .DUOHQ 0 .DYXNFXJOX
. DQG .XNVD 3  ³1DWXUDO /DQJXDJH 3URFHVVLQJ >@ 1JLDP-.KRVOD$.LP01DP-/HH+DQG1J$<
$OPRVW IURP 6FUDWFK´ Journal of Machine Learning  ³0XOWLPRGDO GHHS OHDUQLQJ´ 3URFHHGLQJV RI WKH WK
Research9ROSS± ,QWHUQDWLRQDO &RQIHUHQFH RQ 0DFKLQH /HDUQLQJ ,&0/
SDJHV±
>@ &ROOREHUW 5 %HQJLR 6 DQG 0DULHWKR] -  ³7RUFK $
PRGXODU   PDFKLQH   OHDUQLQJ VRIWZDUH   OLEUDU\´ 7HFKQLFDO >@ 2¶&RQQRU 3 1HLO ' /LX 6 'HOEUXFN 7 DQG 3IHLIIHU 0
5HSRUW,',$3  ³5HDOWLPH FODVVLILFDWLRQ DQG VHQVRU IXVLRQ ZLWK D
VSLNLQJGHHSEHOLHIQHWZRUN´)URQWLHUVLQQHXURVFLHQFH
>@ 'DKO * <X ' 'HQJ / DQG $FHUR $  ³&RQWH[W
GHSHQGHQW SUHWUDLQHG GHHS QHXUDO QHWZRUNV IRU ODUJH >@ 6FKPLGKXEHU - DQG +RFKUHLWHU 6  ³/RQJ VKRUWWHUP
YRFDEXODU\ VSHHFK UHFRJQLWLRQ´ IEEE Trans. Audio, Speech, PHPRU\ ´ Neural Computation 9RO  SS ±
Language Process 
>@ 'DV 6  +LJK/HYHO 'DWD )XVLRQ $UWHFK +RXVH >@ 6WHLQEHUJ $ 1 %RZPDQ &/DQG :KLWH -U ) ( 
1RUZRRG0$ ³5HYLVLRQVWRWKH-'/GDWDIXVLRQVPRGHOV´Proceedings of the
3rd NATO/IRIS Conference4XHEHF&LW\&DQDGD
>@ (LWHO $ 6SULQJHQEHUJ - 6SLQHOOR / 5LHGPLOOHU 0 DQG
%XUJDUG :  ³0XOWLPRGDO 'HHS /HDUQLQJ IRU 5REXVW >@ 6XWVNHYHU , 0DUWHQV - DQG +LQWRQ *  ³*HQHUDWLQJ
5*%' 2EMHFW 5HFRJQLWLRQ´ 3URFHHGLQJV RI WKH ,(((56- WH[WZLWKUHFXUUHQWQHXUDOQHWZRUNV´Proceedings of the ICML
,QWHUQDWLRQDO &RQIHUHQFH RQ ,QWHOOLJHQW 5RERWV DQG 6\VWHPV >@ 6]HJHG\&/LX:-LD<6HUPDQHW35HHG6$QJXHORY
,526 ' (UKDQ ' 9DQKRXFNH 9 DQG 5DELQRYLFK $ 
>@ )DUKDGL$+HMUDWL06DGHJKL0<RXQJ35DVKWFKLDQ& ³*RLQJ 'HHSHU ZLWK &RQYROXWLRQV´ Computer Vision and
+RFNHQPDLHU-DQG)RUV\WK'³(YHU\SLFWXUHWHOOVD Pattern Recognition&935
VWRU\ *HQHUDWLQJ VHQWHQFHV IURP LPDJHV´ Proceedings of >@ 6]HJHG\&9DQKRXFNH9,RIIH66KOHQV-DQG:RMQD=
ECCV  ³5HWKLQNLQJ WKH ,QFHSWLRQ $UFKLWHFWXUH IRU &RPSXWHU
>@ )HQJ ) /L 5 DQG :DQJ ;  ³&RQVWUXFWLQJ 9LVLRQ´LQDU[LY
KLHUDUFKLFDO LPDJHWDJV ELPRGDO UHSUHVHQWDWLRQV IRU ZRUG WDJV >@ 9LQFHQW 3 /DURFKHOOH + %HQJLR < DQG 0DQ]DJRO 3
DOWHUQDWLYH FKRLFH´ &R55 DQG :RUNVKRS RQ &KDOOHQJHV LQ  ³([WUDFWLQJ DQG FRPSRVLQJ UREXVW IHDWXUHV ZLWK
5HSUHVHQWDWLRQ/HDUQLQJ,&0/ GHQRLVLQJDXWRHQFRGHUV´International Conference on Machine
>@ *UDYHV $ 0RKDPHG +LQWRQ *  ³6SHHFK UHFRJQLWLRQ Learning,&0/SS
ZLWKGHHSUHFXUUHQWQHXUDOQHWZRUNV´Proceedings of ICASSP >@ 9LQ\DOV 2 7RVKHY $ %HQJLR 6 DQG (UKDQ ' 
>@ *UHII . 6ULYDVWDYD 5 .RXWQLN - 6WHXQHEULQN % DQG ³6KRZ DQG 7HOO /HVVRQV OHDUQHG IURP WKH  06&2&2
6FKPLGKXEHU -  ³/670 $ 6HDUFK 6SDFH 2G\VVH\´ ,PDJH &DSWLRQLQJ &KDOOHQJH´ IEEE Transaction on Pattern
IEEE Transactions on Neural Networks and Learning Systems Analysis and Machine Intelligence
,VVXH KWWSVJLWKXEFRPWHQVRUIORZPRGHOVWUHHPDVWHULPW[W
>@ -LD < 6KHOKDPHU ( 'RQDKXH - .DUD\HY 6 /RQJ - >@ :KLWH-U)(³$PRGHOIRUGDWDIXVLRQ´3URFHHGLQJV
*LUVKLFN 5 *XDGDUUDPD 6 DQG  'DUUHOO 7  ³&DIIH RIWKHVW1DWLRQDO6\PSRVLXPRQ6HQVRU)XVLRQ9RO
&RQYROXWLRQDO DUFKLWHFWXUH IRU IDVW IHDWXUH HPEHGGLQJ´ >@ =KX ; 7XLD ' 0RX / ;LD *6 =KDQJ / ;X ) DQG
Proceedings of the ACM International Conference on )UDXQGRUIHU )  ³'HHS /HDUQLQJ LQ 5HPRWH 6HQVLQJ $
MultimediaSS±$&0 5HYLHZ´,(((*HRVFLHQFHDQG5HPRWH6HQVLQJ0DJD]LQH



2171