2015 29th Brazilian Symposium on Software Engineering

7RZDUGVDQDSSURDFKPDWFKLQJ&0'DQG'65WR
LPSURYHWKH$FDGHPLD,QGXVWU\VRIWZDUH
GHYHORSPHQWSDUWQHUVKLS
$FDVHRIDJLOHDQG8;LQWHJUDWLRQ
-RHOPD&KRPD

/XFLDQD$0=DLQD

7LDJR6LOYDGD6LOYD

8)6&DU&DPSXV6RURFDED
5GY-RmR/6DQWRV.P
6RURFDED63%UD]LO
MFKRPD#XIVFDUEU

8)6&DU&DPSXV6RURFDED
5GY-RmR/6DQWRV.P
6RURFDED63%UD]LO
O]DLQD#XIVFDUEU




,&781,)(63
8QLYHUVLGDGH)HGHUDOGH6mR3DXOR
6mR-RVpGRV&DPSRV63%UD]LO
VLOYDGDVLOYD#XQLIHVSEU


  
  
   



     
       





   

 
  



  

  
   
  
 
   


 
   
    
  

   

 
 !
"# 
  
 !"
$   %!
#  
  %  !  $&       "

&  !"#!"
$'


  
    
(   
   
 
& )
* 

#)*$     
   + 

     
    !" 
    
 
& 

   
    )*   ")%  
 
 
 )*   

 
    

7KHDFWLYHFRRSHUDWLRQDPRQJUHVHDUFKHUVDQGSUDFWLWLRQHUV
LV FRQVLGHUHG DQ LPSRUWDQW IDFWRU IRU WKH VXFFHVVIXO LQWURGXF
WLRQRIQHZWHFKQRORJ\LQWKHRUJDQL]DWLRQ>@,QWKLVFRQWH[W
WHFKQRORJ\UHIHUVWRDQ\DUWLIDFWVSURGXFHGE\DFWLYLWLHVZLWKLQ
VRIWZDUH GHYHORSPHQW LQFOXGLQJ FRQFHSWV WRROV WHFKQLTXHV
PHWKRGVDQGPHWKRGRORJLHV$UWLIDFWVDUHHVVHQWLDOWRWKHVRIW
ZDUH SURFHVV EHFDXVH WKH\ FDQ UHSRUW DOO LQIRUPDWLRQ FUHDWHG
RYHUDSHULRGRIGHYHORSPHQWDFWLYLWLHV
7KLV SDSHU SUHVHQWV DQ DSSURDFK FDOOHG 6RIW&R'H5 6RIW
ZDUH &RRSHUDWLYH 'HVLJQ 5HVHDUFK WR HQFRXUDJH WKH SDUWQHU
VKLSEHWZHHQ,QGXVWU\DQG$FDGHPLDEULQJLQJFORVHUUHVHDUFK
HUVWRSUDFWLWLRQHUVRIVRIWZDUHHQJLQHHULQJ,QRXUILUVWH[SHUL
PHQWVWKHDSSURDFKSURYHGWREHIHDVLEOHLQDUHVHDUFKSURMHFW
ZKLFKDLPHGWRLQWHJUDWLQJ8VHUH;SHULHQFH8;PHWKRGVLQWR
DJLOH SUDFWLFHV 2QH RI WKH UHVHDUFK SURMHFW JRDOV ZDV WKH GH
YHORSPHQW RI DUWLIDFWV WR LQWHJUDWH WKH YLHZSRLQW RI 8; LQWR
6&580 DJLOH PHWKRGRORJ\ 7KH 6RIW&R'H5 DSSURDFK DO
ORZHGWKHUHVHDUFKHUVWRZRUNFORVHO\LQFRRSHUDWLRQZLWKSUDF
WLWLRQHUV RI WKH VRIWZDUH GHYHORSPHQW DUHD LQ RUGHU WR OLQN
VFLHQWLILFNQRZOHGJH$FDGHPLDWRWKHUHDOQHHGVRIWKH,QGXV
WU\

   

 
 

  




 

 !


"

, ,1752'8&7,21
7KH QHHG IRU VKDULQJ H[SHULHQFHV DQG NQRZOHGJH EHWZHHQ
$FDGHPLD DQG VRIWZDUH ,QGXVWU\ KDV EHHQ EHFRPLQJ LQFUHDV
LQJO\HYLGHQW+RZHYHUWKH$FDGHPLD,QGXVWU\LQWHUDFWLRQKDV
GHPRQVWUDWHG GLIILFXOWLHV RQ HVWDEOLVKLQJ DQ HIIHFWLYH UHODWLRQ
VKLSEHFDXVHWKH\XVXDOO\KDYHGLIIHUHQWYLHZSRLQWV

8VXDOO\WKHWHDPV8;DQG6&580PDNHXVHRIGLIIHUHQW

W\SHVRIDUWLIDFWVWRVXSSRUWWKHLUDFWLYLWLHVDQGWRFRPPXQLFDWH
LQIRUPDWLRQ SURGXFHG WKURXJKRXW WKH GHYHORSPHQW SURFHVV
$UWLIDFWV GHILQHG LQ 6&580 DUH LQWHQGHG WR PD[LPL]H WKH
WUDQVSDUHQF\ RI LQIRUPDWLRQ DQG VXSSRUW GHFLVLRQV GXULQJ D
VRIWZDUHGHYHORSPHQWSURMHFW>@:KLOHWKH8;DUWLIDFWVDUHWR
SURYLGH LQIRUPDWLRQ UHJDUGLQJ LQWHUDFWLRQ DQG XVHU UHTXLUH
PHQWV DQG WR RIIHU GHVLJQ VROXWLRQV >@ 2XU H[SHULHQFH ZDV
EDVHGRQWKHEXLOGLQJRIDUWLIDFWVZKLFKEURXJKWYDOXHIRUERWK
8;DQG6&580WHDPVGXULQJWKHVRIWZDUHGHYHORSPHQWSUR
FHVV

0HFKDQLVPVKDYHEHHQSURSRVHGWREDODQFLQJWKHQHHGVRI
UHVHDUFK DQG LQGXVWU\ WKURXJKRXW WKH GHYHORSPHQW VRIWZDUH
SURFHVV>@DQGDOVRWRSURPRWHDVWUDWHJLFDOLJQPHQWEHWZHHQ
$FDGHPLDDQG,QGXVWU\IURPZKLFKUHVHDUFKVKRXOGEHGULYHQ
E\WKHQHHGVDQGLQWHUHVWVRIWKHVRIWZDUH,QGXVWU\>@
)URPWKLVSHUVSHFWLYHUHVHDUFKHUVQHHGWRGHYHORSVNLOOVWR
EHWWHUXQGHUVWDQGSUDFWLFDOSUREOHPVREVHUYHGLQUHDOVFHQDULRV
IURPWKHH[SHULHQFHRIWHDPVZLWKLQWKHVRIWZDUHGHYHORSPHQW
HQYLURQPHQW )XUWKHUPRUH UHVHDUFKHUV HQJDJHG LQ SURMHFWV
ZLWK,QGXVWU\DUHHQFRXUDJHGWRSHUIRUPDQDFWLYHUROHGXULQJ
WKH LQYHVWLJDWLRQLQ RUGHU WRPD[LPL]H WKH XVH RI WKH UHVHDUFK
UHVXOWVLQWKHSUDFWLFH
978-1-4673-9272-3/15 $31.00 2015 IEEE
DOI 10.1109/SBES.2015.18

7KH UHPDLQGHU RI WKLV SDSHU LV RUJDQL]HG DV IROORZV 6HF
WLRQ ,, LQWURGXFHV WKH EDVLV RI RXU SURSRVDO  DQ RYHUYLHZ RI
WZR UHVHDUFK PHWKRGV DQG UHODWHG ZRUNV 6HFWLRQ ,,, RXWOLQHV
WKH SURSRVDO DSSURDFK 6HFWLRQ ,9 SUHVHQWV WKH DSSURDFK DS
SOLHGLQSUDFWLFH6HFWLRQ9GLVFXVVHVWKHRXWFRPHVRIWKHSUDF
51

LQWKHDUHDRI,QIRUPDWLRQ6\VWHPV,6EXWLWKDVEHHQLPSOLF
LWO\SUDFWLFHGLQ6(IRUGHFDGHV>@

WLFHV DQG ILQDOO\ 6HFWLRQ 9, SUHVHQWV WKH FRQFOXVLRQV DQG

SRLQWVRXWGLUHFWLRQVIRUIXUWKHUZRUNV

+HYQHU HW DO >@ SURSRVHG D FRQFHSWXDO IUDPHZRUNFRP

ELQLQJ'6DQG%HKDYLRUDO6FLHQFHWRGULYHUHVHDUFKLQWKHILHOG
RI ,6 ,Q DQRWKHU SDSHU +HYQHU >@ GHILQHG WKUHH F\FOHV WR
HQKDQFH WKH XQGHUVWDQGLQJ RI '65 7KHVH UHVHDUFK F\FOHV
ZLWKLQ '65 IUDPHZRUN SURSRVHG E\ WKH DXWKRU DUH VKRZQ LQ
)LJXUH$FFRUGLQJWRKLPWKH5HOHYDQFH&\FOHFRQQHFWVWKH
FRQWH[WXDO(QYLURQPHQWRIWKHUHVHDUFKSURMHFWDWRWKH'65
DFWLYLWLHV E 7KH PDLQIRFXV RI WKH 5HOHYDQFH &\FOH LV L WR
FDSWXUH SUREOHPV WR EH DGGUHVVHG RU UHTXLUHPHQWV IRU WKH UH
VHDUFKDQGLLWRSURYLGHGHVLJQVROXWLRQDUWLIDFWVIRUWKHHQYL
URQPHQW RI VWXG\ DQG HYDOXDWLRQ LQ WKH DSSOLFDWLRQ GRPDLQ
7KH 5LJRU &\FOH FRQQHFWV WKH '65 DFWLYLWLHV E ZLWK WKH
.QRZOHGJH%DVHFWKDWVXSSOLHVWKHUHVHDUFKSURMHFWHQVXULQJ
LQQRYDWLRQ E\ SURYLGLQJ H[LVWLQJ NQRZOHGJH IRU WKH UHVHDUFK
7KH.QRZOHGJH%DVHFFRQVLVWVRIIRXQGDWLRQVH[LVWLQJH[SH
ULHQFHVDQGH[SHUWLVHDQGH[LVWLQJDUWLIDFWVDQGSURFHVVHV7KH
PDLQIRFXVRI5LJRU&\FOHLVWRSURYLGHDSSOLFDEOHNQRZOHGJH
IRU'65DFWLYLWLHVEDQGWRIHHGEDFNWKHXSGDWHGNQRZOHGJH
WR HQULFK WKH .QRZOHGJH %DVH 7KH 'HVLJQ &\FOH ORFDWHG LQ
WKHFHQWHURI)LJXUHRSHUDWHVEHWZHHQWKHDFWLYLWLHVRIEXLOG
LQJ DQG HYDOXDWLQJ WR GHVLJQ DUWLIDFWV DQG SURFHVVHV IURP WKH
UHVHDUFK 7KHVH WKUHH F\FOHV PXVW EH FOHDUO\ LGHQWLILDEOH LQ D
UHVHDUFK SURMHFW VLQFH GHVLJQ DFWLYLWLHV PXVW EH FRQYLQFLQJO\
EDVHGRQUHOHYDQFHDQGULJRU

,, %$&.*5281'
,Q WKLV VHFWLRQ ZH LQLWLDOO\ SURYLGH DQ RYHUYLHZ RI WZR
PHWKRGV WKDW EDVHG WKH SURSRVDO DSSURDFK 7KHUHIRUH ZH SUH
VHQWUHODWHGZRUNV
$ $FWLRQ5HVHDUFKDQG&RRSHUDWLYH0HWKRG'HYHORSPHQW
$FWLRQ5HVHDUFK$5LVDQLWHUDWLYHSURFHVVLQYROYLQJUH
VHDUFKHUVDQGSUDFWLWLRQHUVDFWLQJWRJHWKHURQDSDUWLFXODUF\FOH
RI DFWLYLWLHV LQFOXGLQJ SUREOHP GLDJQRVLV DFWLRQ LQWHUYHQWLRQ
DQGUHIOHFWLYHOHDUQLQJDVGHILQHGE\$YLVRQHWDO>@
$5 KDV KLVWRULFDO URRWV LQ WKH VRFLRWHFKQLFDO PRYHPHQW
ZKRVH RULJLQ LV DVVRFLDWHG ZLWK .XUW /HZLQ
V DSSOLFDWLRQ RI
H[SHULPHQWDOORJLFDQGVRFLDOSV\FKRORJLFDOWKHRU\IRUSUDFWLFDO
VRFLDO SUREOHPV LQWKH V>@ 6LQFH WKHQ $5KDV HYROYHG
DQG KDV EHHQ DSSOLHG LQ VHYHUDO VFLHQWLILF DUHDV HJ QXUVLQJ
HGXFDWLRQ PDQDJHPHQW LQIRUPDWLRQ V\VWHPV 6DQWRV DQG
7UDYDVVRV>@VWDWHWKDWWKHUHLVDQLQFUHDVLQJWHQGHQF\RQXVLQJ
$5 LQ 6( WR DGGUHVV GLIIHUHQW UHVHDUFK WRSLFV $FFRUGLQJ WR
WKHP $5 KDV EHHQ VHHQ DV DQ DOWHUQDWLYH PHWKRGRORJ\ WKDW
VWLPXODWHV LQGHSWK LQYHVWLJDWLRQ RI WKH SUDFWLFHV RI 6( LQ DQ
LQGXVWULDOFRQWH[WLQWHQVLI\LQJWKHH[HFXWLRQRIUHOHYDQWVWXGLHV
DQGDGGLQJYDOXHWRWKHUHVXOWV+RZHYHUWKH\KDYHLGHQWLILHG
WKDWWKHULJRXUDQGFRQWURORI$5VWXGLHVVKRXOGEHLPSURYHGLQ
6(VWXGLHV
,QRXUSURSRVDOZHKDYHXVHGWKHFRQFHSWVRIWKHDSSURDFK
SURSRVHGE\'LWWULFK>@FDOOHG&RRSHUDWLYH0HWKRG'HYHORS
PHQW &0' ZKLFK LV DQ DGDSWDWLRQ RI $5 WKDW IRFXVHV RQ
VRIWZDUH GHYHORSPHQW SUDFWLFHV &0' FRPELQHV TXDOLWDWLYH
VRFLDOVFLHQFHILHOGZRUNZLWKSUREOHPRULHQWHGPHWKRGVWHFK
QLTXHDQGSURFHVVLPSURYHPHQW7KH&0'UHVHDUFKSURFHVVLV
PRGHOOHGDVHYROXWLRQDU\F\FOHVFRQVLVWLQJRIWKUHHSKDVHV
8QGHUVWDQGLQJ3UDFWLFH'HOLEHUDWH,PSURYHPHQWVDQG
,PSOHPHQW DQG 2EVHUYH ,PSURYHPHQWV 7KH UHVHDUFK EHJLQV
ZLWKDSKDVHRI8QGHUVWDQGLQJ3UDFWLFHZLWKTXDOLWDWLYHHPSLU
LFDO LQYHVWLJDWLRQV DLPLQJ DW XQGHUVWDQGLQJ DQG H[SODLQLQJ
SUDFWLFHV IURP D SUDFWLWLRQHUV YLHZSRLQW LGHQWLI\LQJ DVSHFWV
WKDWDUHSUREOHPDWLFIRUWKRVHLQYROYHGLQSURMHFW7KHVHFRQG
SKDVH RI 'HOLEHUDWH ,PSURYHPHQWV UHVXOWV RQ PHDVXUHV WKDW
FDQEHH[SHFWHGWRLPSURYHWKHVLWXDWLRQLGHQWLILHGLQWKHILUVW
SKDVH DGGUHVVHG E\ WKH FRRSHUDWLRQ EHWZHHQ UHVHDUFKHUV DQG
WKH SUDFWLWLRQHUV ,Q WKH WKLUG SKDVH ,PSOHPHQW DQG 2EVHUYH
,PSURYHPHQWVILUVWO\WKHLPSURYHPHQWVDUHLPSOHPHQWHGDQG
WKHUHVHDUFKHUVIROORZWKHLPSURYHPHQWVDVSDUWLFLSDQWREVHUY
HUV $IWHUZDUGV WKH UHVXOWV DUH HYDOXDWHG WRJHWKHU ZLWK WKH
SUDFWLWLRQHUVLQYROYHG7KHWKUHHSKDVHVFDQEHUHSHDWHGO\DS
SOLHGLQWKHVDPHFRQWH[W


)LJ 'HVLJQ6FLHQFH5HVHDUFK&\FOHV6RXUFH>@

& 5HODWHG:RUN
$FFRUGLQJWR'LWWULFKHWDO>@&0'KDVEHHQDSSOLHGLQ
GLIIHUHQW UHVHDUFK SURMHFWV RXWOLQLQJ GLIIHUHQW DVSHFWV ZLWKLQ
VRIWZDUH HQJLQHHULQJ HJ GHVLJQ RI IOH[LEOH DQG DGDSWDEOH
VRIWZDUH DJLOH GHYHORSPHQW IRU HJRYHUQPHQW DSSOLFDWLRQV
DQG WKH LQWHJUDWLRQ RI LQWHUDFWLRQ GHVLJQ DQG VRIWZDUH GHYHO
RSPHQW5HFHQWO\$O%DLNDQG0LOOHU>@KDYHLPSOHPHQW
HG &0' LQ DQ LQYHVWLJDWH VWXG\ LQ RUGHU WR XQGHUVWDQG WKH
SRWHQWLDOUHDVRQVEHKLQGWKHUHODWLYHO\ORZVXFFHVVUDWHRIOHDQ
LQLWLDWLYHV LQ ,7 RUJDQL]DWLRQV $V D UHVXOW WKH\ SURSRVHG D
QHZPRGHOWRFODVVLI\ZDVWHVLQ,7RUJDQL]DWLRQV$UGLWRHWDO
>@DSSOLHG&0'LQFRPSDQLHVIURPVRXWKHUQ,WDO\DLPLQJ
WR LQTXLUH WKH LPSDFW RI 8; PHWKRGV LQWR WKH VRIWZDUH
GHYHORSPHQW SUDFWLFHV  LQYROYLQJ SUDFWLWLRQHUV DQG ORRNLQJ
WKH DVSHFWV IURP LQVLGH WKH FRPSDQLHV 7KH DXWKRXUV
FRQILUPHG WKDW PDQ\ FRPSDQLHV HLWKHU QHJOHFW XVDELOLW\ DQG
8;RUFDQQRWSURSHUO\FRQVLGHUWKHP
5HJDUGLQJWKH'65$GLNDULHWDO>@DGRSWHGWKH+HYQHU
HW DOV IUDPHZRUN >@ WR GHYHORS DQ DSSURDFK FDOOHG /LWWOH
'HVLJQ 8S )URQW /'8) DLPLQJ WR LQWHJUDWH WKH 8VHU

% 'HVLJQ6FLHQFH5HVHDUFK
'HVLJQ6FLHQFH'6RULJLQDOO\DURVHIURP+HUEHUW6LPRQV
ZRUN HQWLWOHG 7KH 6FLHQFHV RI WKH $UWLILFLDO SXEOLVKHG LQ
 ZKRVH FRQFHSW LV DQFKRUHG WR PDQ\ SURIHVVLRQDO GLVFL
SOLQHV OLNH DUFKLWHFWXUH HFRQRPLFV DQG HQJLQHHULQJ >@ '6
DGGUHVVHV WKH UHVHDUFK E\ EXLOGLQJ DQG HYDOXDWLQJ DUWLIDFWV
ZKLFKDUHGHVLJQHGWRPHHWEXVLQHVVQHHGVLGHQWLILHGLQDSDU
WLFXODU HQYLURQPHQW >@ 7KH DUWLIDFWV W\SLFDOO\ SURGXFHG E\
'65HVHDUFK'65DUHUHODWHGWRFRQFHSWVPRGHOVPHWKRGV
DQGLQVWDQWLDWLRQV>@'65LVDQDSSURDFKFRPPRQO\DGRSWHG

52

&RQVLGHULQJWKHDIRUHPHQWLRQHGSHUVSHFWLYHVRIFRPELQLQJ
$5DQG'65ZHKDYHFKRVHQWRIROORZDVLPLODUDSSURDFKWR
WKHSURSRVDOVRI%DVNHUYLOOHHWDO>@DQG6HLQHWDO>@ZLWK
UHJDUGVWRWKHUHVHDUFKDFWLYLWLHVWKDWDUHVLJQLILFDQWO\RYHUODS
SLQJ+RZHYHUZHFKRVHWKH&0'PHWKRGEDVHGRQ$5WKDW
ZDV SURSRVHG E\ 'LWWULFK >@  EHFDXVH LW LV D PRUH DGKHUHQW
PHWKRG WR WKH 6( SUDFWLFHV IRFXVLQJ RQ VKRS IORRU VRIWZDUH
GHYHORSPHQW SUDFWLFHV ,Q DGGLWLRQ WKH &0' DSSURDFK SUR
YLGHVDVWUXFWXUHGUHVHDUFKSURFHVVZKLFKFOHDUO\GHILQHVWKUHH
SKDVHV RI UHVHDUFK IRU XQGHUVWDQGLQJ WKH SUDFWLFH DQG IRU WKH
GHOLEHUDWLRQLPSOHPHQWDWLRQDQGHYDOXDWLRQRILPSURYHPHQWV
7KH&0'DSSURDFKSURYLGHVIXUWKHUJXLGDQFHVXFKDVWKHXVH
RI HWKQRJUDSKLF VWXGLHV IRU XQGHUVWDQGLQJ D SUDFWLFH IURP D
SUDFWLWLRQHUV
 YLHZSRLQW ,QWHUYLHZV GRFXPHQW DQDO\VLV DQG
ZRUNVKRSVDUHVRPHRIWKHGDWDJDWKHULQJWHFKQLTXHVFRPPRQ
O\XVHGDVTXDOLWDWLYHPHWKRGVIRUWKHHPSLULFDOILHOGZRUN7KH
&0'DSSURDFKGRHVQRWH[FOXGHTXDQWLWDWLYHPHWKRGVZKHWKHU
WKH\ DSSHDU WR EH XVHIXO WR FRPSOHPHQW WKH TXDOLWDWLYH GDWD
HJGHVFULSWLYHVWDWLVWLFV%HVLGHVWKH&0'HQFRXUDJHVVKRUW
LWHUDWLYH F\FOHV WR SXW WKH LPSURYHPHQWV LQ SUDFWLFH 7KLV LV
WRWDOO\DGKHUHQWWRWKHDJLOHSULQFLSDOV

&HQWHUHG 'HVLJQ SHUVSHFWLYH LQWR $JLOH 5HTXLUHPHQWV (QJL

QHHULQJ7KH/'8)DSSURDFKZDVHYDOXDWHGLQWZRDJLOHSUR
MHFWV DQG LQ WKH HYDOXDWLRQ RI WKH DSSURDFK WKH TXDOLWLHV RI
HDVLHUWROHDUQDQGHDVLHUWRXVHZHUHSRLQWHGRXWE\WKHXVHUV
ZKR XVHG WKH SURGXFWV GHYHORSHG IURP /'8) 5HFHQWO\
5RGUtJXH]HWDO>@KDYHDOVRDSSOLHG+HYQHUHWDOVIUDPH
ZRUNLQWKHLUSURMHFWVWRH[SORUHWKHPDLQHOHPHQWVWKDWFKDU
DFWHUL]H WKH FRPELQDWLRQ RI $JLOH DQG /HDQ PHWKRGV LQ VRIW
ZDUHGHYHORSPHQW7KH\LOOXVWUDWHKRZ'65FDQEHDSSOLHGLQ
HPSLULFDO VRIWZDUH HQJLQHHULQJ UHSRUWLQJ LQ OHVVRQV OHDUQHG
WKDWVXFKUHVHDUFKPHWKRGVKDYHDJRRGSRWHQWLDOIRUVXSSRUW
LQJFROODERUDWLRQEHWZHHQ,QGXVWU\DQG$FDGHPLD
6RPHDSSURDFKHVFRPELQLQJ$5DQG'65KDYHEHHQSUR
SRVHG WRGULYH UHVHDUFK LQ WKH ILHOG RI ,6 >@ >@ >@ %DV
NHUYLOOH HW DO >@ SURSRVHG 6RIW '65 DSSURDFK WKDW LQFRUSR
UDWHV DVSHFWV RI 6RIW 6\VWHPV 0HWKRGRORJ\ WKDW LV D IRUP RI
$5 LQWR D '65 SURFHVV 6HLQ HW DO >@ SUHVHQWHG D PHWKRG
FDOOHG$FWLRQ'HVLJQ5HVHDUFK$'5WKDWGHILQHVIRXUVWDJHV
IRUSHUIRUPLQJDUHVHDUFK3UREOHP)RUPXODWLRQ%XLOG
LQJ,QWHUYHQWLRQDQG(YDOXDWLRQ5HIOHFWLRQDQG/HDUQLQJ
DQG  )RUPDOL]DWLRQ RI /HDUQLQJ %RWK DSSURDFKHV  6RIW
'65 DQG $'5  DUH FODVVLILHG DV GHVLJQRULHQWHG DFWLRQ UH
VHDUFK,QDGLIIHUHQWZD\:LHULQJDDQG0RUDOL>@SURSRVHG
DPHWKRGFDOOHG7HFKQLFDO$FWLRQ5HVHDUFK7$5WKDWIROORZV
DQDUWLIDFWGULYHQDSSURDFKLQZKLFKWKH$5LVXVHGRQO\LQWKH
HYDOXDWLRQRIWKHDUWLIDFWV
YLDELOLW\LQDJLYHQFRQWH[W

7KH'65FRQFHSWVZHUHDGGHGWRWKHSURSRVDODSSURDFKLQ
RUGHU WR SURYLGH JXLGHOLQHV WR SURGXFH DUWLIDFWV RI YDOXH DQ
H[WUHPHO\UHOHYDQWLVVXHWRWKHDJLOHWHDPVWKDWZLOOEHXVHGWR
VXSSRUWWKHDFWLYLWLHVRI6(SURIHVVLRQDOV7KH'65GUDZVWKH
DWWHQWLRQ RI UHVHDUFKHUV ERWK IRU DVSHFWV UHJDUGLQJ SUREOHP
UHOHYDQFH DV ZHOO DV WKRVH WKDW FRQFHUQ WKH UHVHDUFKV ULJRU
7KHREMHFWLYHRI'65LVWRGHYHORSDUWLIDFWVWRVROYHSUREOHPV
WKDWDUHUHOHYDQWZLWKLQDVSHFLILFFRQWH[WEDVHGRQWKHDSSOLFD
WLRQ RI ULJRURXV PHWKRGV ERWK RQ EXLOGLQJ DV ZHOO DV LQ WKH
HYDOXDWLRQRIDUWLIDFWV

,,, 7+(62)7&2'(5$3352$&+
)URP WKH QDWXUDO FRPSDWLELOLW\ EHWZHHQ $5 DQG '65 ZH
FDQ REVHUYH WKDW ERWK DUH  LQWHUYHQWLRQLVW PHWKRGRORJLHV
LQWHUYHQHVUDWKHUWKDQVWXG\DSKHQRPHQRQDIWHUWKHIDFWDQG
 WKH\ DUH FORVHO\ UHODWHG WR SUREOHPVROYLQJ DQG LQYROYLQJ
WKH HYDOXDWLRQ RI WKH VROXWLRQV SURSRVHG >@ 0RUHRYHU ERWK
PHWKRGV DUH EDVHG RQ WKH DUJXPHQW WKDW WKH UHVHDUFK VKRXOG
HYHU FRQWULEXWH IRU ERWK WR $FDGHPLD DV ZHOO DV WR ,QGXVWU\
$FFRUGLQJWR%DVNHUYLOOH>@DOWKRXJKGRLQJGHVLJQVFLHQFH
PD\ORRNDORWOLNHGRLQJDFWLRQUHVHDUFKWKHUHDUHIXQGDPHQ
WDO GLIIHUHQFHV EHWZHHQ WKHP VXFK DV  $5 LV IRFXVHG RQ
SUREOHP VROYLQJ WKURXJK VRFLDO DQG RUJDQL]DWLRQDO FKDQJH
ZKLOH'65LVIRFXVHGRQSUREOHPVROYLQJE\FUHDWLQJDQGSRVL
WLRQLQJ DQ DUWLIDFW LQ D QDWXUDO VHWWLQJ DQG  $5 LV FOHDUO\
FHQWHUHG RQ GLVFRYHU\WKURXJKDFWLRQ ZKLOH '65 LV FOHDUO\
FHQWHUHGRQGLVFRYHU\WKURXJKGHVLJQ

)LJXUH  VKRZV RXU DSSURDFK ZKLFK ZH KDYH FDOOHG

6RIW&R'H56RIWZDUH&RRSHUDWLYH'HVLJQ5HVHDUFK

,LYDUL DQG 9HQDEOH >@ LGHQWLILHG WKH IROORZLQJ OHYHOV RI

RYHUODSSLQJIRUDFWLYLWLHVSHUIRUPHGLQWKHSURFHVVHVRI$5DQG
'65QRQRYHUODSSLQJVRPHZKDWRYHUODSSLQJDQG
VLJQLILFDQWO\ RYHUODSSLQJ 1RQRYHUODSSLQJ RFFXUV ZKHQ WKH
'65LVDGRSWHGIRUVROYLQJDSXUHO\WHFKQLFDOSUREOHPRUHYHQ
WRVROYHDVRFLRWHFKQLFDOSUREOHPLQZKLFKWKHUHLVQRLQWHUHVW
LQGLVFRYHU\WKURXJKDFWLRQ7KHVHFRQGFDVHVRPHZKDWRYHU
ODSSLQJ  FRQFHUQV WR WKH XVH RI $5 WR HYDOXDWH '65 DV HJ
ZKHQDQDFWLRQUHVHDUFKHULVLQWHUHVWHGRQHYDOXDWLQJDQH[LVW
LQJ WHFKQRORJLFDO VROXWLRQ ZLWKRXW HQJDJLQJ KLPVHOI RQ WKH
FRQVWUXFWLRQ RI DUWLIDFWV 7KH VLJQLILFDQW RYHUODS LV ZKHQ WKH
UHVHDUFKHUDGRSWHG$5DLPLQJWRGLVFRYHU\VRPHWKLQJWKURXJK
DFWLRQDQGDGGLWLRQDOO\VKHLVZRUNLQJRQWKHFRQVWUXFWLRQRI
LQQRYDWLYH DUWLIDFWV DGGUHVVHG IRU FOLHQWV SUREOHP VROYLQJ ,Q
WKLV FDVH WKH UHVHDUFK LQWHUHVW LQFOXGHV ERWK WKH EXLOGLQJ DQG
HYDOXDWLRQ RI DUWLIDFWV ZKLOH ZRUNLQJ LQ FORVH FROODERUDWLRQ
ZLWKWKHSUDFWLWLRQHUVRIWKHFRPSDQ\


)LJ 6RIW&R'H56RIWZDUH&RRSHUDWLYH'HVLJQ5HVHDUFK

,Q WKH 6RIW&R'H5 DSSURDFK ZH KDYH FRQVLGHUHG WKH WKUHH

VWHSV RI &0' LQ WHUPV RI DQ LWHUDWLYH F\FOH 8QGHUVWDQGLQJ
3UDFWLFH'HOLEHUDWH,PSURYHPHQWVDDQG,PSOHPHQWDQG
2EVHUYH,PSURYHPHQWVD)URPWKHEHVWSUDFWLFHVRISURGXF
LQJ YDOXDEOH DUWLIDFWV '65 ZH DGGHG DQ DFWLYLW\ WR %XLOG

53

,9 7+(62)7&2'(5,135$&7,&(
7KLV VHFWLRQ SUHVHQWV WKH JHQHUDO VHWWLQJV RI WKH HQYLURQ
PHQW LQ ZKLFK WKH 6RIW&R'H5 KDV EHHQ DSSOLHG :H DOVR GH
VFULEH WKH WZR F\FOHV DSSOLHG IRU DQVZHULQJ WKH VSHFLILF UH
VHDUFKTXHVWLRQVLQYROYLQJWKHEXLOGLQJDQGHYDOXDWLQJRIWKUHH
DUWLIDFWV WKDW LQWHJUDWH 8; PHWKRGV LQWR WKH HQYLURQPHQW RI
DJLOHGHYHORSPHQWVRIWZDUH

$UWLIDFWV E DQG WR (YDOXDWH $UWLIDFW E 7KH LQWHUVHFWLRQ

$% RI LQIRUPDWLRQ EHWZHHQ ,QGXVWU\ LQWHUHVWV DQG QHHGV
DQG$FDGHPLDIRXQGDWLRQVZLOOSURYLGHJXLGHOLQHVWKURXJKDOO
WKHVWDJHVRIWKHUHVHDUFKF\FOH
7KHILUVWVWHSRIWKH6RIW&R'H5FRQVLVWVRI8QGHUVWDQGLQJ
3UDFWLFHVIURPWKHSUDFWLWLRQHUVYLHZSRLQWDLPLQJWRLGHQ
WLI\WKHQHHGVIRUQHZVROXWLRQVRULPSURYHPHQWVWKDWVKRXOGEH
DOLJQHGWRWKH LQWHUHVWV RI ,QGXVWU\ $ 7KH IRXQGDWLRQV IURP
WKH $FDGHPLD % FRQWULEXWH VR WKDW WKH UHVHDUFKHU LGHQWLILHV
FRQFHSWV WKDW DUH XVHGRUQRW LQSUDFWLFH LQ FRQWUDVW WR RWKHU
WKHRULHV DQG WHFKQRORJLHV WKDW KDYH DOUHDG\ EHHQ HPSOR\HG LQ
RWKHU FRPSDQLHV (WKQRJUDSKLF VWXGLHV DQG LQWHUYLHZV DUH WKH
TXDOLWDWLYHWHFKQLTXHVPRUHXVHGWRJDWKHUGDWDRQSUDFWLFHVRI
VRIWZDUHGHYHORSPHQW$UHVHDUFKTXHVWLRQ54FDQEHDQWLF
LSDWHGE\UHVHDUFKHUVHDUO\LQWKLVSKDVHRUFDQEHIRUPXODWHG
GXULQJWKHILUVWVWDJH7KH54VKRXOGHPHUJHEDVHGRQLVVXHV
REVHUYHGDERXWWKHSUDFWLFHLQRUGHUWRGULYHWKHQH[WVWHSV
DQG

$ 3URMHFW&RQWH[WDQG5HVHDUFK7KHPH
7KH LVVXHV DGGUHVVHG LQ WKLV DUWLFOH DUH SDUWLDO UHVXOWV RI D
UHVHDUFKSURMHFWWKDWKDVEHHQGHYHORSHGLQSDUWQHUVKLSZLWKD
FRPSDQ\ WKDW GHYHORSV (53 (QWHUSULVH 5HVRXUFH 3ODQQLQJ
V\VWHPV&XUUHQWO\WKHFRPSDQ\KDVHPSOR\HHVDSSUR[L
PDWHO\FXVWRPHUVDQGPRUHWKDQXVHUV7KHFRP
SDQ\KDVEHHQDFWLYHLQVHYHUDOPDUNHWVHJPHQWVVXFKDV0DQX
IDFWXULQJ /RJLVWLFV DQG $JULEXVLQHVV ,Q UHFHQW \HDUV WKH H[
HFXWLYHVRIWKHFRPSDQ\KDYHFRQFHQWUDWHGLWVHIIRUWVWRHYROYH
WKHLUVRIWZDUHGHYHORSPHQWSURFHVVHVLQFRUSRUDWLQJEHVWSUDF
WLFHV IRU WKH GHYHORSPHQW RI XVHU LQWHUIDFHV 8, E\ XVHU
FHQWHUHG GHVLJQ 7KH\ EHOLHYH WKDW LQWHJUDWLQJ 8; SUDFWLFHV
LQWRWKHGHYHORSPHQWRI(53V\VWHPVLVDNH\DVSHFWWRILOODQ
LPSRUWDQWJDSRIWKHGHYHORSPHQWSURFHVVDQGFRQVHTXHQWO\D
ZD\WRVWDQGRXWLQWKHLUPDUNHWVHJPHQW

7KH WULJJHU IRU WKH QH[W VWHS LV WKH 54 IRUPXODWHG LQ WKH
SUHYLRXV SKDVH ZKLFK VKRXOG JXLGH WKH DFWLRQ WR 'HOLEHUDWH
,PSURYHPHQWV D 'XULQJ WKLV VHFRQG VWDJH $FDGHPLD DQG
,QGXVWU\ PHPEHUV PHHW WRDQDO\]H WKH UHVXOWV JDWKHUHGE\WKH
UHVHDUFKHU UHJDUGLQJ WKH 8QGHUVWDQGLQJ 3UDFWLFH  7RJHWKHU
$FDGHPLDDQG,QGXVWU\PHPEHUVWKH\EXLOGDVFHQDULRZKLFK
UHSURGXFHVWKHSUREOHPVLWXDWLRQWKDWWKH\WDUJHWWRVROYHEDVHG
RQWKH54SUHYLRXVO\GHILQHG%HIRUHSURSRVLQJDQ\VROXWLRQ
SUDFWLWLRQHUV DQG UHVHDUFKHU VKRXOG WKRXJKWIXOO\ H[DPLQH WKH
VFHQDULR DQG WKHQ WKH DQVZHUV FRPH XS IURP WKH LQWHUVHFWLRQ
RILQIRUPDWLRQDQGNQRZOHGJHRI,QGXVWU\DQG$FDGHPLDH[SH
ULHQFHV ,Q WKLV VWHS WKH DUWLIDFWV RI YDOXH UHTXLUHG WR VXSSRUW
WKH SUDFWLWLRQHUV
 ZRUN DUH LGHQWLILHG DFFRUGLQJ WR WZR YLHZ
SRLQWVWHFKQLFDOIXQFWLRQWKHDFDGHPLFYLHZDQGRISUDJPDW
LF IXQFWLRQ WKH ,QGXVWU\ YLHZ 6XEVHTXHQWO\ WKHVH DUWLIDFWV
VKRXOGEHHODERUDWHGDQG %XLOWEE\UHVHDUFKHUVDQGSUDFWL
WLRQHUV7KLVVWHSLVF\FOLFDOWRDOORZWKHGLVFXVVLRQWRLPSURYH
WKHDUWLIDFWV

7KHUHVHDUFKWKHPHLV,QWHUDFWLRQ'HVLJQDGGUHVVHGWR(53
6\VWHPV 7KH REMHFWLYH RI WKH UHVHDUFK LV WR SURSRVH PHWKRGV
DQG DUWLIDFWV VXLWDEOH WR WKH (53 V\VWHPV GRPDLQ LQWHJUDWLQJ
8; FRQFHSWV ZLWKLQ WKH 6&580 :H UHVHDUFKHUV KDYH
ZRUNHG LQ FROODERUDWLRQ ZLWK WKH FRPSDQ\
V SUDFWLWLRQHUV LQ
PDQ\ DFWLRQV DLPLQJ WR SURSRVH IRU LQVWDQFH L PHFKDQLVPV
DQGWHFKQLTXHVIRULGHQWLI\LQJWKHQHHGVRIXVHUVLLVWDQGDUGV
DQG JXLGHOLQHV IRU WKH GHYHORSPHQW RI XVHU LQWHUIDFHV LLL
HIIHFWLYH XVDELOLW\ HYDOXDWLRQ PHWKRGV IRU (53 V\VWHPV DQG
LY KRZ WR LPSURYH WKH FRPPXQLFDWLRQ RI XVDELOLW\ LVVXHV
EHWZHHQ6&580DQG8;WHDPV
,Q UHFHQW \HDUV 8; KDV EHHQ VSUHDG ZLGHO\ DQG ZHOO UH
FHLYHG E\ $FDGHPLD DQG ,QGXVWU\ 7KLV LQFUHDVLQJ LQWHUHVW LQ
8; FDQ EH H[SODLQHG E\ WKH IDFW WKDW LW LV D EURDGHU FRQFHSW
WKDQWKHFRQFHSWRIXVDELOLW\>@,QDGGLWLRQWKHLQVHUWLRQRI
8; PHWKRGV LQWR WKH VRIWZDUH GHYHORSPHQW SURFHVV KDV EHHQ
VHHQDVDQLPSRUWDQWFRPSHWLWLYHDGYDQWDJHIRUFRPSDQLHV8;
LV D FRQFHSW WKDW HQFRPSDVVHV DOO DVSHFWV RI XVHU LQWHUDFWLRQ
IRFXVLQJWRVROYHSUREOHPVLQDQLQWHUGLVFLSOLQDU\KROLVWLFDQG
WDUJHWHG ZD\ IRU DFKLHYLQJ D GHHS XQGHUVWDQGLQJ RI EHKDYLRU
FRJQLWLRQ DELOLWLHV GHVLUHV DQG KXPDQFRQWH[W >@ 8VHU H;
SHULHQFH'HVLJQ8;'DLPVWRSURYLGHDSOHDVDQWH[SHULHQFH
IRU XVHUV E\ PHDQV RI VLPSOLFLW\ HOHJDQFH HIILFLHQF\ HIIHF
WLYHQHVVDQGHPRWLRQDOVDWLVIDFWLRQRIWKHSHRSOH7KH+XPDQ
&RPSXWHU,QWHUDFWLRQ+&,ILHOGLQWXUQSURYLGHVWHFKQLTXHV
DQGPHWKRGRORJLHVWKDWDOORZXVWRVWXG\DQGGHYHORSWKHXVHU
H[SHULHQFHIURPWKHSHUVSHFWLYHRIWKH,QWHUDFWLRQ'HVLJQ

,Q WKH WKLUG VWHS ,QGXVWU\ PHPEHUV GHILQH DQG FKRRVH D

SUDFWLFDOSURMHFWWKURXJKZKLFKLWZLOOEHSRVVLEOHWR,PSOHPHQW
DQG 2EVHUYH ,PSURYHPHQWV D DQG (YDOXDWH $UWLIDFWV E
7KHDUWLIDFWVFDQEHHYDOXDWHGLQWHUPVRISUDFWLFDOLW\HIILFLHQ
F\RUHIIHFWLYHQHVVIRUH[DPSOH0DQ\HYDOXDWLRQPHWKRGVFDQ
EH DGRSWHG LQGLYLGXDOO\ RU LQ FRPELQDWLRQ VXFK DV ODERUDWRU\
H[SHULPHQWV ILHOG H[SHULPHQWV VLPXODWLRQV FDVH VWXGLHV VXU
YH\VDQGHWKQRJUDSKLFVWXGLHV>@,QVRPHFDVHVDQH[SHUL
PHQWDO YDOLGDWLRQ LV VWURQJO\ UHFRPPHQGHG EHIRUH WR SXW WKH
LPSURYHPHQWVLQSUDFWLFHHVSHFLDOO\WRPLQLPL]HWKHLPSDFWRI
LQWURGXFLQJWKHQHZWHFKQRORJ\DUWLIDFWLQSUDFWLFH7KHHYDO
XDWLRQ SKDVH VKRXOGSURYLGH IHHGEDFN UHJDUGLQJ WKH EXLOW DUWL
IDFWV DQG DQ\ RWKHU LPSURYHPHQWV WKDW ZHUH LPSOHPHQWHG
1HZ LQWHUDFWLRQV PD\ EH UHTXLUHG WR UHILQH WKH DUWLIDFWV RU
HYHQWRVXJJHVWQHZLPSURYHPHQWV7KH6RIW&R'H5DSSURDFK
KDVDQLWHUDWLYHDQGF\FOLFDOQDWXUH1HZUHVHDUFKTXHVWLRQFDQ
EHJHQHUDWHGLQHDFKF\FOHEDVHGRQWKHOHVVRQVOHDUQHG7KXV
WKHF\FOHFDQEHLQFUHPHQWDODGGLQJLQIRUPDWLRQIRUHDFKQHZ
F\FOHRUVSLUDOVLQFHWKHLQIRUPDWLRQFDQDULVHGXULQJWKHHYDO
XDWLRQSKDVH

7KHLQWHJUDWLRQRI8;PHWKRGVLQWRVRIWZDUHGHYHORSPHQW
SURFHVVHV EDVHG RQ DJLOH PHWKRGRORJLHV LV QRW DQ HDV\ WDVN
,QGHHGDJLOHVRIWZDUHGHYHORSPHQWSUDFWLFHVKDYHIHDWXUHVWKDW
IRVWHU WKH LQWHJUDWLRQ HJ LWHUDWLYH SURFHVVHV IRFXV RQ WKH
TXDOLW\ DQG ZRUN LQ FROODERUDWLRQ ZLWK GLIIHUHQW VWDNHKROGHUV
>@+RZHYHU WKHUH DUH PDQ\ LVVXHV LQZKLFKLW FDQFRPSOL
FDWH VXFK LQWHJUDWLRQ >@ HJ SURPRWLQJ V\QHUJ\ DQG HVWDE
OLVKLQJ D JRRG FRPPXQLFDWLRQ EHWZHHQ WKH DJLOH DQG WKH 8;
WHDPV >@ ,Q SDUWLFXODU WKLV LVVXH LQYROYHV GHDOLQJ ZLWK WKH

54

PLQGVHWRIWKHSUDFWLWLRQHUVDERXWWKHLPSRUWDQFHRIWKH8;LQ
WKH VRIWZDUH GHYHORSPHQW SURFHVV WKURXJK ZRUNVKRSV
HYROYLQJWKH8;WHDPDQG6&580WHDP,QWKLVF\FOHDQGWR
KHOSXVRQLQYHVWLJDWLRQRIRXUILUVW5454ZHFDUULHGRXW
D ZRUNVKRS HQWLWOHG 8VDELOLW\ +HXULVWLFV IRU (53 V\VWHP
ZKHUH ZH LQWURGXFHG WKH 1LHOVHQV KHXULVWLFV DQG SHUVRQDV
FRQFHSWV$IWHUWKDWZHLQWHUYLHZHGWKHSUDFWLWLRQHUVZKRKDG
DWWHQGHGLQWKHZRUNVKRS

GLIIHUHQFH RI PLQGVHW EHWZHHQ DJLOH GHYHORSHUV DQG 8; GH

VLJQHUV ZKR XVH GLIIHUHQW SUDFWLFHV DQG H[SUHVV LQ WKHLU SDU
WLFXODUZD\IURPWKHLUNQRZOHGJHDUHDV
1H[WZHGHVFULEHWZRF\FOHVSHUIRUPHGIURPWKH6RIW&R'
H5DSSURDFKWKDWKDYHEURXJKWUHVXOWVDQGFRQWULEXWLRQVWRWKH
UHVHDUFKRQLQWHUDFWLRQGHVLJQDGGUHVVHGWR(536\VWHPV
% )LUVW&\FOH
2XU ILUVW H[SHULHQFH ZLWK WKH 6RIW&R'H5 DSSURDFK DURVH
IURPDQLVVXHWKDWWKHLQGXVWU\SDUWQHUUHSRUWHGWRXVWKHGLIIL
FXOW\ RI FRPPXQLFDWLRQ EHWZHHQ 8; DQG 6&580 WHDPV GXH
WRGLIIHUHQFHVLQYRFDEXODU\7DNLQJWKLVLVVXHLQWRDFFRXQWZH
FRQVLGHUWKHXVHRIFRQFHSWVWKDWFRXOGDSSUR[LPDWHWKHGLIIHU
HQWYLHZSRLQWV2XUILUVWLQVLJKWZDVWKHXVHRIWZRFRQFHSWV
SHUVRQDV>@DQG1LHOVHQ
VKHXULVWLFV>@DVDSRVVLEOHVROX
WLRQ  &RQVLGHULQJ WKH SUREOHPVLWXDWLRQ DQG WKH XVH RI WKH
FRQFHSWV PHQWLRQHG ZH KDYH GHILQHG WZR UHVHDUFK TXHVWLRQV
WKDWJXLGHGWKHILUVWF\FOHRIUHVHDUFK


54&DQWKHFRQFHSWVRISHUVRQDVDQG1LHOVHQ
VKHX
ULVWLFVOHYHORXWWKHDZDUHQHVVDQGFRQFHUQVRQXVDELO
LW\ DVSHFWV RI 8; GHVLJQHUV DQG GHYHORSHUV SUR
JUDPPHUVDQGWHVWHUV"

54 +RZ FRXOG SHUVRQDV DQG 1LHOVHQV KHXULVWLFV

FRQFHSWV EH XVHG DV D FRPPRQ YRFDEXODU\ IRU WKH
FRPPXQLFDWLRQEHWZHHQWKH8;DQG6&580WHDPV"

7KH ZRUNVKRS
V JRDO ZDV VSUHDGLQJ WKH FRQFHSWV DQG
SUDFWLFHV RI WKH DSSOLFDWLRQ RI SHUVRQDV DQG 1LHOVHQ
V
KHXULVWLFVLQRUGHUWRLGHQWLI\WKHYDOXHWKDWWKLVFRQFHSWPD\
KDYHWRWKHSDUWLFLSDQWV7KHSUDFWLWLRQHUVZKRSDUWLFLSDWHG
LQ WKH ZRUNVKRS ZHUH IURP GLIIHUHQW DUHDV GHYHORSHUV 
WHVWHUV  DQDO\VWV  WHFKQLFDO OHDGHUV  DQG VRIWZDUH
DUFKLWHFWV   0RVW SDUWLFLSDQWV KDG OHVV WKDQ ILYH \HDUV RI
H[SHULHQFH LQ VRIWZDUH GHYHORSPHQW DQG  KDG OHVV WKDQ
ILYH\HDUVRIH[SHULHQFHLQGHYHORSLQJ(53V\VWHPV2QO\
SDUWLFLSDQWVKDGPRUHWKDQWHQ\HDUVRIH[SHULHQFHLQ
VRIWZDUH GHYHORSPHQW DQG  SDUWLFLSDQWV  KDG
H[SHULHQFH LQ WKH GHYHORSPHQW RI (53 V\VWHPV 'XULQJ WKH
ZRUNVKRS WKH SDUWLFLSDQWV FRXOG ZRUN ZLWK WKH FRQFHSWV LQ
SUDFWLFDOH[DPSOHVLQ(53V\VWHPVFHQDULRV
$LPLQJ WR LQYHVWLJDWH WKH LPSDFW RI WKH ZRUNVKRS LQ WKH
GDLO\SUDFWLFHRIWKHSUDFWLWLRQHUVZHFDUULHGRXWDIHZZHHNV
DIWHU WKH ZRUNVKRS DQ LQWHUYLHZ WR REVHUYH L WKH H[SHUWLVH
DQG VNLOOV REWDLQHG IURP WKH ZRUNVKRS DQG LL WKH FRQWULEX
WLRQ RI WKH ZRUNVKRS LQ FKDQJLQJ WKH SDUWLFLSDQWV
 YLHZSRLQW
RQWKHXVDELOLW\DVSHFWVLQWKHVRIWZDUHGHYHORSPHQWHVSHFLDO
O\ LQ (53 V\VWHPV 7DEOH  ZH SUHVHQW DQ RYHUYLHZ RI WKH
LQWHUYLHZHHVSURILOH

 8QGHUVWDQGLQJ3UDFWLFH
:HVWDUWHGEDVHGRQ3HUVRQDVDQG1LHOVHQ
VKHXULVWLFVFRQ
FHSWVZKLFKDUHUHFRJQL]HGE\WKH8;H[SHUWVDVJRRGSUDF
WLFHVWRNHHSWKHVRIWZDUHGHYHORSPHQWIRFXVRQWKHQHHGVRI
WKHHQGXVHUDQGRQWKHVRIWZDUHXVDELOLW\>@>@3HUVRQDV
DUH XVHG LQ WKH VRIWZDUH GHYHORSPHQW LQGXVWU\ WR GHVFULEH D
JURXS RI UHDO XVHUV DQG WKHLU IHDWXUHV WKURXJK K\SRWKHWLFDO
DUFKHW\SHV7KHXVHRISHUVRQDVFDQJXLGHWKHGHYHORSPHQWRI
LQWHUDFWLRQ VFHQDULRV DQGRU EH XVHG WR GHVFULEH WKH W\SLFDO
WDVNVLQXVDELOLW\WHVWLQJ>@7KHDSSOLFDWLRQRIWKHSHUVRQDV
WHFKQLTXHKDVSURYHGWKDWGHVLJQLQJIRUDVPDOOVHWRISHUVRQDV
FDQ PHHW D VLJQLILFDQW QXPEHU RI XVHUV GXH WR WKHLU VLPLODU
JRDOV DQG IHDWXUHV 7KH FRQVWUXFWLRQ RI SHUVRQDV VKRXOG EH
JURXQGHGLQWKHUHVHDUFKRQWDUJHWXVHUVZKLFKVRPHWLPHVFDQ
UHTXLUHDORQJWLPH+RZHYHUSHUVRQDVKDYHEHHQDGDSWHGWR
EH OLJKW DQG OHDQ DLPLQJ WR KHOS DJLOH GHYHORSHUV DQG RWKHU
VWDNHKROGHUVWRHOLFLWWKHXVHUUHTXLUHPHQWV>@

7$%/(,

1LHOVHQ
VKHXULVWLFV>@DUHXVDELOLW\JXLGHOLQHVFRPPRQO\
XVHG WR VXSSRUW WKH FULWLFDO DQDO\VLV LQ XVDELOLW\ LQVSHFWLRQV
KHXULVWLFHYDOXDWLRQEXWDOVRFDQEHXVHGWRGULYHWKHGHVLJQ
RILQWHUDFWLYHLQWHUIDFHVDQGDOVRWRFRQILUPSUREOHPVLGHQWL
ILHGLQXVDELOLW\WHVWLQJ >@7KHWHQ1LHOVHQ
VKHXULVWLFVDUH
9LVLELOLW\ RI V\VWHP VWDWXV + 0DWFK EHWZHHQ V\VWHP DQG
WKHUHDOZRUOG+&RQWURODQGIUHHGRPIRUXVHU+&RQ
VLVWHQF\DQGVWDQGDUGV+(UURUSUHYHQWLRQ+5HFRJQL
WLRQUDWKHUWKDQUHFDOO+)OH[LELOLW\DQGHIILFLHQF\RIXVH
+$HVWKHWLFDQGPLQLPDOLVWGHVLJQ++HOSXVHUVUHFRJ
QL]HGLDJQRVHDQGUHFRYHUIURPHUURUV++HOSDQGGRFX
PHQWDWLRQ+

29(59,(:2),17(59,(:6


#


 $


 


#
$ !%


!




&
%

#
$ !%


!





#

$


3

7HVWHU

\HDUV

\HDUV

PLQV

3

'HYHORSHU

\HDUV

\HDUV

PLQV

3

7HVWHU

\HDUV

\HDUV

PLQV

3

'HYHORSHU

\HDUV

\HDUV

PLQV

3

7HVWHU

\HDUV

\HDUV

PLQV

3

'HYHORSHU

\HDUV

\HDUV

PLQV

3

$UFKLWHFW

\HDUV

\HDUV

PLQV

3

7HVWHU

\HDUV

\HDUV

PLQV

3

$UFKLWHFW

\HDUV

\HDUV

PLQV

3

'HYHORSHU

\HDUV

\HDUV

PLQV

7KHLQWHUYLHZVZHUHDXGLRUHFRUGHGZLWKWKHSULRUSHUPLV
VLRQRIWKHLQWHUYLHZHHVDQGODWHUWUDQVFULEHGWRDUHSRUW'XU
LQJ  GD\ ZH SHUIRUPHG D VHPLVWUXFWXUHG LQWHUYLHZ LQFOXG
LQJTXHVWLRQVWKDWVXPPDUL]HWKUHHDVSHFWVLWKHLUYLHZSRLQW
RIXVDELOLW\DVSHFWVLLWKHLUSHUFHSWLRQIRUSUDFWLFDODSSOLFD
WLRQRI WKH 1LHOVHQ
V KHXULVWLFV DQGSHUVRQDV LQ (53 V\VWHPV
DQGLLLWKHLUHYDOXDWLRQUHJDUGLQJWKHLPSURYHPHQWVRQ(53

&RQVLGHULQJ RXU PDLQ JRDO LQ WKH UHVHDUFK SURMHFW RI

,QWHUDFWLRQ'HVLJQWR(536\VWHPVZHKDYHEHHQZRUNHGWKH

55

2YHUDOO ZH QRWHG WKDW WKH VXEMHFWV DQG DFWLYLWLHV RI WKH
ZRUNVKRSV KDYH EHHQ HQRXJK WR SRVLWLYHO\ LQIOXHQFH WKH SDU
WLFLSDQWV
 PLQGVHW DERXW WKH LPSRUWDQFH RI XVDELOLW\ LQ (53
V\VWHPV :H DOVR REVHUYHG WKDW 1LHOVHQV KHXULVWLFV FDQ VHW
LPSRUWDQW WRSLFV RQ XVDELOLW\ LQ SHRSOH
V PLQG %DVHG RQ WKH
LQWHUYLHZV RXWFRPHV ZH IRXQG HYLGHQFHV WR DQVZHU RXU ILUVW
TXHVWLRQ547KHSDUWLFLSDQWVFRQVLGHUHGWKDWSHUVRQDVDQG
1LHOVHQ
V KHXULVWLFV FDQ JXLGH SURGXFW GHYHORSPHQW DQG FDQ
DOVREHXVHGDVDFRPPRQYRFDEXODU\DPRQJWKHGHYHORSHUV
WHVWHUVDQG8;GHVLJQHUV

V\VWHPV EURXJKW E\ WKH DSSOLFDWLRQ RI XVDELOLW\ LVVXHV :H

DQDO\]HG TXDOLWDWLYHO\ WKH SDUWLFLSDQWV
 DQVZHUV FDWHJRUL]LQJ
WKHPLQWKHWKUHHSRLQWVOLVWHGDERYH
5HJDUGLQJ WKHILUVWREVHUYHGDVSHFW DERXW L YLHZSRLQWRI
XVDELOLW\   DVSHFWV WKH PDMRULW\ RI WKH UHVSRQGHQWV VD\ WKDW
WKH\ DUH PRUH FDUHIXO DQG SD\ DWWHQWLRQ LQ WKH VLPSOH GHWDLOV
WKDWFDQLPSURYHWKHXVDELOLW\RIWKHV\VWHP7KH\UHSRUWHGWR
EHPRUHFRQFHUQLQJRQWKHQHHGVDQGSHUFHSWLRQVRIWKHXVHUV
WKHSHUVRQDV6RPHSDUWLFLSDQWVFRPPHQWVZHUH
,SD\PRUHDWWHQWLRQDQGUHFRJQL]HWKDWVLPSOHSRLQWV
DUH LPSRUWDQW LPSURYHPHQWV ZKLFK ZH VKRXOG GR  WKH
ZRUNVKRSRSHQVP\PLQGRIXVHULQWHUDFWLRQ>3@

 'HOLEHUDWH,PSURYHPHQWVDQG%XLOG$UWLIDFW
&RQVLGHULQJ WKH RXWFRPHV RI WKH ZRUNVKRS DQG WKH
LQWHUYLHZV  UHVHDUFKHUV PHW ZLWK SURMHFW OHDGHUV DQG WKH 8;
GHVLJQHU WR HYDOXDWH LPSDFW RI WKH XVH RI DUWLIDFWV DSSO\LQJ
WKHVHWZRFRQFHSWVSHUVRQDVDQG1LHOVHQ
VKHXULVWLFVLQWKH
FRPSDQ\ SURMHFWV 7KH SURMHFW OHDGHUV SRLQWHG RXW WKDW WKH
KDUG HIIRUWV RQ EXLOGLQJ RI SHUVRQDV  UHTXLULQJ D SULRU
NQRZOHGJH RI WKH SRWHQWLDO XVHUV  FRXOG PDNH WKH SHUVRQDV
DGRSWLRQLQIHDVLEOH+RZHYHUZHIRXQGWKDWWKHFRPSDQ\KDG
D NQRZOHGJH RI WKH XVHUV
 SURILOH IURP LWV DIWHUVDOHV
GHSDUWPHQW 7KHQ WKH SHUVRQDV FRXOG EH GHVLJQHG IURP WKLV
SUHYLRXV NQRZOHGJH $QG PDQ\ RI WKH SHUVRQDV FRXOG EH
UHXVHG LQ GLIIHUHQW SURMHFWV 7KHQ WKH OHDGHUV DJUHHG ZLWK XV
$QRWKHULPSRUWDQWSUDFWLFDOSRLQWZHGLVFRYHUHGZDVWKDWWKH
8; WHDPV XVXDOO\ FDUULHG RXW XVDELOLW\ WHVWV ZLWK SURSRVHG
LQWHUIDFHV FROOHFWHG WKH SUREOHPV DQG UHSRUWHG WKH SURSRVDO
IRUWKH6&580WHDPIURPDORZILGHOLW\SURWRW\SHVWDWLQJWKH
GHWDLOVRUDOO\RUE\DQQRWDWLRQVLQQDWXUDOODQJXDJH$FFRUGLQJ
WR8;DQG6&580WHDPVWKLVPHFKDQLVPRIFRPPXQLFDWLRQ
KDVEHHQFDXVHGPLVXQGHUVWDQGLQJEHWZHHQWKHWHDPV

QRZZKHQZHORRNDWDQHZVRIWZDUHLQWHUIDFHZHWU\
WR LPLWDWH WKH SHUVRQDV LQWHUDFWLRQ ZH DUH DEOH WR LGHQWLI\
XVDELOLW\SUREOHPV>3@
 ZH DUH ZRUNLQJ RQ FKDQJHV RI HUURU PHVVDJHV RI D
SURMHFWEHFDXVHZHQRWHGWKHPHVVDJHVKDGQRVLJQLILFDQFHWR
WKHXVHU>3@
:LWKUHJDUGVRQWKHVHFRQGDVSHFWLLWKHLUSHUFHSWLRQIRU
SUDFWLFDODSSOLFDWLRQRIWKH1LHOVHQ
VKHXULVWLFVDQGSHUVRQDV
LQ (53 V\VWHP DOO UHVSRQGHQWV DJUHH ZLWK WKH XVHIXOQHVV RI
WKH KHXULVWLFV DQG SHUVRQDV LQ WKH SUDFWLFH RI (53 GHYHORS
PHQW7KH\EHOLHYHWKDWKHXULVWLFVFDQJXLGHWKHXVDELOLW\YHUL
ILFDWLRQDQGDLGWKHGHYHORSHULQILQGLQJDOWHUQDWLYHVWRVROYH
XVDELOLW\LVVXHV7KHSHUVRQDVKDYHDSV\FKRORJLFDOLQIOXHQFH
EHFDXVHWKH\KDYHDFRQFUHWHSHUVRQWKDWUHSUHVHQWVWKHXVHUV
6RPHSDUWLFLSDQWVFRPPHQWVZHUH
WKHKHXULVWLFVDQGSHUVRQDVPDGHP\ZRUNHDVLHU
ZH GHYHORSHG D QHZ XVHU LQWHUIDFH DSSO\LQJ WKH FRQFHSWV RI
FRORUVLQDFDSWLRQ:HKDGQRWGHFLGHGWRXVHWKHPUDQGRPO\
ZHKDGORRNHGIRUFRORUSDWWHUQV>3@

)URPWKHGLVFXVVLRQVEHWZHHQ$FDGHPLDDQG,QGXVWU\DQG
DOVR WR DQVZHU WKH VHFRQG UHVHDUFK TXHVWLRQ 54 ZH KDYH
SURSRVHGEXLOGLQJWZRDUWLIDFWVE

 ,Q P\ RSLQLRQ WKH KHXULVWLFV VKRZHG WR PH WKDW P\
FRQFHUQV VKRXOG QRW EH RQO\ DERXW WKH IXQFWLRQDO DVSHFWV ,
DOVR KDYH WR FRQVLGHU WKH VFUHHQ WKH FRQWHQW DUUDQJHPHQW
SDWWHUQ HWF  $IWHU VHHLQJ WKH SHUVRQDV , EHFRPH PRUH
FULWLFDODERXWWKHXVHULQWHUDFWLRQLVVXHV>3@

D/HDQ 3HUVRQDV UHSUHVHQWV D VHW RI HQGXVHUV

GHVFULEHG EULHIO\ IURP SKRWR QDPH DJH H[SHUWLVH DUHD
NQRZOHGJHVNLOOVDQGDELOLWLHVJRDOVPRWLYHVDQGFRQFHUQV
DQG XVDJH SDWWHUQV FRQWH[W DQG XVDJH KDELWV EDVHG RQ
VLPSOLILHG VWUXFWXUH SURSRVDO IRU SHUVRQDV VSHFLILFDWLRQ
SUHVHQWHG E\ 0LOOHU DQG :LOOLDPV >@ :H FDOOHG WKHP
+\SRWKHVHVRI3HUVRQDV+3

WKHKHXULVWLFVFRXOGEHXVHGDVDFKHFNOLVWGXULQJWKH
VRIWZDUHWHVWV>3@
)LQDOO\WKHWKLUGDVSHFWLLLWKHLUHYDOXDWLRQUHJDUGLQJWKH
LPSURYHPHQWV RQ (53 V\VWHPV EURXJKW E\ WKH DSSOLFDWLRQ RI
XVDELOLW\ LVVXHV DOWKRXJK WKH (53 V\VWHP ZDV FRPSOH[ WKH
UHVSRQGHQWVEHOLHYHWKDWLWLVSRVVLEOHWRPDNHWKHXVHUH[SHUL
HQFHPRUHHQMR\DEOH6RPHH[DPSOHVRIFRPPHQWVZHUH

E3URWRFRO IRU FRPPXQLFDWLRQ RI GHVLJQ VROXWLRQV DQG

XVDELOLW\UHFRPPHQGDWLRQVWKHQHZDUWLIDFWLVJHQHUDWHGIURP
WKH DQDO\VLV RI WKH UHVXOWV RI WKH XVDELOLW\ WHVWLQJ VHVVLRQ
)LUVWO\WKHLVVXHVLGHQWLILHGGXULQJWKHXVDELOLW\WHVWLQJVHVVLRQ
DUH OLVWHG UHPRYLQJ WKH GXSOLFDWH LVVXHV WKH VDPH LVVXHV
SRLQWHG RXW E\ PRUH WKDQ RQH SHUVRQD DQG IDOVH SRVLWLYHV
LVVXHVWKDWZHUHQRWFRQVLGHUHGUHDO6HFRQGO\HDFKLVVXHLV
DQDO\]HGDQGVROXWLRQVDQGRUUHFRPPHQGDWLRQVDUHSURSRVHG
IRUWKHP7KLUGO\WKHSHUVRQDVDIIHFWHGE\HDFKLVVXHVKRXOG
EH SLQSRLQWHG /DVWO\ HDFK LVVXH LV PDSSHG WR 1LHOVHQ
V
KHXULVWLFV)RUPDOL]LQJWKHSURWRFROLWHPVFDQEHUHSUHVHQWHG
LQ(TXDWLRQ

6RPH SHRSOH DQG VRPH SURIHVVLRQDOV EHOLHYH WKDW LQ ELJ

DQG FRPSOH[ V\VWHPV OLNH (53 V\VWHPV WKH XVDELOLW\ LV QRW
LPSRUWDQW 7KH\ WKLQN WKH XVHU LV IDPLOLDU ZLWK WKH LGHD RI D
GLIILFXOWV\VWHP2QWKHRWKHUVLGH,GRQRWDJUHH(YHQWKRXJK
(53V\VWHPVDUHFRPSOH[DQGUHTXLUHFRPSOLFDWHGSURFHVVHV
,WKLQNRXUMREDVGHYHORSHUVLVWRIDFLOLWDWHWKHXVHULQWHUDF
WLRQ WKURXJK WKH XVDELOLW\ KHXULVWLFV DSSOLFDWLRQ IRU H[DP
SOH>3@
7KH(53V\VWHPLVUHDOO\FRPSOH[,KDYHQRGRXEWDERXW
LW EXW LW LV DOZD\V SRVVLEOH WR LPSURYH WKH XVHU LQWHUDFWLRQ
>3@




56

-
 ./.0 1. .2.
 3.45 3.5- .6
ZKHUH

DJDLQ'LIIHUHQWO\IURPWKHILUVWFDVHVWXG\WKH6&580WHDP
 ZKR KDG QR GRXEWV UHJDUGLQJ WKH UHVXOWV DQG UHFRPPHQGD
WLRQVIL[HGDOOWKHLWHPVUHSRUWHGWKURXJKWKHSURWRFRO

GHVFULSWLRQ^UHFRPPHQGDWLRQVROXWLRQ`
1+VXEVHW ^QKQK1+`

#7$

D


E

1+ ^+++`



F

+3VXEVHW ^KSKS +3`



G

+3 ^K\S3HUVRQDK\S3HUVRQDM`

H

1+ 1LHOVHQV+HXULVWLFV+3 +\SRWKHVHVRI3HUVRQDV


 ,PSOHPHQWLQJ DQG 2EVHUYLQJ ,PSURYHPHQWV DQG
(YDOXDWLQJ$UWLIDFWV
7KH QHZ DUWLIDFWV ZHUH SUHVHQWHG WR SURMHFW OHDGHUV 8;
GHVLJQHU DQG 6&580 WHDP LQ D SUHYLRXVO\ VFKHGXOHG PHHW
LQJ$LPLQJWRYDOLGDWHWKHWZREXLOWDUWLIDFWV/HDQ3HUVRQDV
DQG 3URWRFRO  ZH GHFLGHG DORQJ ZLWK SUDFWLWLRQHUV HYDOXDWH
WKHPEWKURXJKWZRFDVHVWXGLHVLQZKLFKXVDELOLW\WHVWLQJ
ZDV SHUIRUPHG RQ DFWXDO UHGHVLJQ SURMHFW 7KH SURMHFW JRDO
ZDVWRGHYHORSGHVLJQIRUDKLJKILGHOLW\SURWRW\SHWRWKHVXE
PRGXOHRI+XPDQ5HVRXUFHV:HEEDVHGXVHGIRUWKHUHJLV
WUDWLRQ RI HPSOR\HHV 1RWHZRUWK\ WKH 8; GHVLJQHU DQG
6&580 WHDP WKDW DWWHQGHG WKH YDOLGDWLRQ DOVR KDG
NQRZOHGJHRIWKH1LHOVHQ
VKHXULVWLFVDQGSHUVRQDVVLQFHWKH\
KDGDWWHQGHGLQWKHZRUNVKRSV

)LJ /HDQSHUVRQDGHVFULEHGE\8;GHVLJQHU

7KH XVH RI WKH QHZ DUWLIDFWV SHUVRQDV DQG SURWRFRO ZDV
LPSOHPHQWHG RQO\ LQ WKH VHFRQG FDVH VWXG\ LQ RUGHU WR FRP
SDUHWKHLUXVHDGYDQWDJHV7KXVLQWKHILUVWFDVHVWXG\ZHGLG
QRWLQWHUIHUHLQWKHWHDPV
ZRUNDFWLQJLQREVHUYHUVUROHIRU
GD\V7KH 8; GHVLJQHU UHFUXLWHG ILYH XVHUV RI GLIIHUHQW SUR
ILOHV DQG SHUIRUPHG WKH XVDELOLW\ WHVWLQJ SOD\LQJ WKH UROH RI
WHVW PRGHUDWRU $IWHU WKH WHVW VKH DQDO\]HG WHQ LVVXHV LGHQWL
ILHG GXULQJ WKLV WHVWLQJ DQG WKHQ VKH VWDWHG WKH LVVXHV RQ D
UHSRUW ZULWWHQ LQ QDWXUDO ODQJXDJH DQG GHOLYHUHG LW WR WKH
6&580WHDPGXULQJDQLQIRUPDOPHHWLQJ7KH6&580WHDP
KDV IL[HG VL[ RI WKH WHQ UHSRUWHG LVVXHV 7KH\ H[SODLQHG WKDW
WKHRWKHUIRXUZHUHUHFXUUHQWLVVXHV7KH8;GHVLJQHUGLGQRW
DJUHHZLWKWKHP


)LJ 7HPSODWHXVHGWRGHVFULEHVROXWLRQVDQGUHFRPPHQGDWLRQWRXVDELOLW\
LVVXHV

$IWHU WKH VHFRQG FDVH VWXG\ ZH PHW WKH 6&580 WHDP WR
FROOHFW WKHLU RSLQLRQ DERXW WKH QHZ DUWLIDFWV 7KH WHDP FRP
PHQWHGWKDWWKHSURWRFROIRUUHSRUWLQJWKHLWHPVWKURXJKKHXULV
WLFVDOORZHGWKHPWRSHUFHLYHWKHUHDOLPSDFWRIHDFKLGHQWLILHG
LVVXHVRWKH\GRQRWKDGGRXEWVRQWKHDGMXVWPHQWVWKH\VKRXOG
IL[:LWKUHJDUGVWRWKHSHUVRQDVWKH\DJUHHWKDWWKH\FRXOGVHH
WKHDSSOLFDWLRQIRFXVLQJRQWKHWDUJHWDXGLHQFH

,Q WKH VHFRQG FDVH VWXG\ ZH SDUWLFLSDWHG DORQJ ZLWK WKH
8; DQG 6&580 WHDPV LQ WKH XVDELOLW\ WHVW XVLQJ SHUVRQDV
DQG WKH SURWRFRO WR UHSRUW WKH XVDELOLW\ LVVXHV )RU WKLV LQWHU
YHQWLRQ ZHXVHG WKHQHZYHUVLRQ RI WKH IXQFWLRQDO SURWRW\SH
IL[HGE\WKH6&580WHDP%HIRUHWKHUHFUXLWPHQWRIWKHXV
HUVWKH8;GHVLJQHUEXLOWWKHWKUHHOHDQSHUVRQDV XVLQJWKH
WHPSODWH DQG FKHFNLQJ VRPH GHWDLOV ZLWK WKH DIWHUVDOHV GH
SDUWPHQW7KHSHUVRQDVZHUHFUHDWHGWRUHSUHVHQWWKUHHGLIIHU
HQWOHYHOVRIH[SHUWLVHUHJDUGLQJWKHXVHRIWHFKQRORJ\EDVLF
LQWHUPHGLDWHDQGDGYDQFHG7KHSHUVRQDVZHUHXVHGWRUHFUXLW
VL[ HQGXVHUV DOO SDUWLFLSDWLQJ IRU WKH ILUVW WLPH )LJXUH 
VKRZVDQH[DPSOHRIOHDQSHUVRQDGHVFULEHGE\8;GHVLJQHU
7KHXVDELOLW\WHVWVZHUHSHUIRUPHGDQGWKHRXWFRPHVZHUH
DQDO\]HG LQ D VLQJOH GD\ )ROORZLQJ VWHSV WR UHYLHZ XVDELOLW\
LVVXHVDQGJHQHUDWHWKHSURWRFROEDVHGRQWKH(TXDWLRQWKH
8; GHVLJQHU GHVFULEHG WKH VROXWLRQV DQG UHFRPPHQGDWLRQ
DFFRUGLQJ WR WKH WHPSODWH VKRZHG LQ )LJXUH  DQG WKHUHSRUW
ZDV GHOLYHUHG WR WKH 6&580 WHDP ZLWKRXW DQ\ H[SODQDWLRQ
,Q WKLV XVDELOLW\ WHVW ZHUH UHYHDOHG WKUHH QHZ LVVXHV DQG WKH
IRXU LVVXHV ZKLFK KDG QRW EHHQ IL[HG ZHUH FRQILUPHG RQFH

& 6HFRQG&\FOH
:KHQWKHSURWRFRODUWLIDFWZDVSXWLQWRSUDFWLFHLQDSUR
MHFW ZLWK WKH SDUWLFLSDWLRQ RI 3URGXFW 2ZQHUV 32 ZH QRWHG
WKDWWKH\KDGGLIILFXOWLHVLQXQGHUVWDQGLQJWKHSURWRFRO2EVHUY
LQJWKHSUDFWLFHWKHUHVHDUFKHUVIRXQGWKDWIRUWKH32WKHXVH
RI 8VHU 6WRU\ 86 ZDV PRUH QDWXUDO &RQVLGHULQJ WKLV LVVXH
WKHUHVHDUFKHUVKDYHVXJJHVWHGDQDGDSWDWLRQRI86LQFRUSRUDW
LQJWKHYLVLRQRIWKHXVHUH[SHULHQFH7KHVHFRQGUHVHDUFKF\FOH
ZDVVWDUWHGLQRUGHUWRDQVZHUWKHIROORZLQJTXHVWLRQ


54 +RZ FRXOG SHUVRQDV DQG 1LHOVHQV KHXULVWLFV

FRQFHSWVEHLQFRUSRUDWHGLQWRWKH86"

 8QGHUVWDQGLQJ3UDFWLFHV
,QWKHILUVWSKDVHWKH$FDGHPLDPHPEHUVKDYHFDUULHGRXWD
WHFKQLFDOOLWHUDWXUHVXUYH\DLPLQJWRLQYHVWLJDWHWKHXVHRI86
LQDJLOHSUDFWLFHV86LVRQHWKHPRVWSRSXODUDUWLIDFWVFUHDWHG
IRU FRQYH\LQJ DJLOH UHTXLUHPHQWV 86 LV XVHG WR FDSWXUH FXV
WRPHUUHTXLUHPHQWVHPSKDVL]LQJXVHUJRDOVEULHIO\H[SODLQLQJ
WKHXVHUSHUFHSWLRQDQGIRFXVLQJRQZKDWLVQHHGHGWREHGRQH

57

>@7KH\DUHRIWHQZULWWHQRQLQGH[FDUGVRUVWLFN\QRWHVDQG
W\SLFDOO\IROORZDVLPSOHWHPSODWHDVVKRZHGLQ)LJXUH


' (
%!$
)

$VD8-
9,ZDQWQHHG89&VRWKDW84:.
 !
ZLOOEHPHW

$VD8  
9,ZDQW89VRWKDW8
9

)LJ ,QLWLDOSURSRVDOWR8VHU;6WRU\WHPSODWH


)LJ 8VHU6WRU\WHPSODWH

7KH LQLWLDO SURSRVDO ZDV GLVFXVVHG EHWZHHQ WKH UHVHDUFK

HUV 32V 6FUXP 0DVWHU DQG 8; GHVLJQHU )URP WKH GLVFXV
VLRQV EHWZHHQ $FDGHPLD DQG ,QGXVWU\ WKH DUWLIDFW KDG EHHQ
LPSURYHG%XLOG$UWLIDFWQGYHUVLRQ7KHVHFRQGYHUVLRQRI
8VHU; 6WRU\ H[SDQGV WKH 86 IURP WKH RXWFRPHV RI WKH GDWD
FROOHFWHGLQWKHXVHUUHVHDUFKSKDVH8VHUUHVHDUFKIRFXVHVRQ
XQGHUVWDQGLQJXVHUEHKDYLRUVQHHGVDQGPRWLYDWLRQVWKURXJK
REVHUYDWLRQ WHFKQLTXHV WDVN DQDO\VLV DQG RWKHU IHHGEDFN
PHWKRGRORJLHVHJWKHXVDELOLW\WHVWLQJSHUIRUPHGRQWKHILUVW
F\FOH7KH8VHU;6WRULHVDUHWROGIURPWKHSHUVSHFWLYHRIWKH
SHUVRQD ZKR QHHGV D SDUWLFXODU FRQGLWLRQ IRU LQWHUDFWLRQ
6XFK DFRQGLWLRQFDQPHHWPXOWLSOHSHUVRQDV7KHVWRULHVGH
VFULEHDQLQWHUDFWLYHSURFHVVZKHUHLQWKHSHUVRQDKDVDJRDOWR
DFKLHYH IRU WKLV VKH DFWV RQ WKH LQWHUIDFH LQWHUDFWLRQ WR
SHUIRUPWDVNVVWHSVIHDWXUHVWRHIIHFWWKHDFWLRQLQDSDUWLFX
ODU FRQWH[W XVDJH SDWWHUQ 7KH SHUVRQD ZLOO DVVHVV ZKHWKHU
WKHREMHFWLYHZDVDFKLHYHGLQWHUSUHWLQJV\VWHPIHHGEDFN$LP
LQJWRYHULI\ZKHWKHUVWRULHVZHUHGHYHORSHGVXFKWKDWLWH[DFW
O\ PHW WKH XVHU LQWHUDFWLRQ QHHGV WKH DFFHSWDQFH FULWHULD
VKRXOG GHVFULEH WKH DFWLRQ WKH VHW RI FRQGLWLRQV DQG WKH
1LHOVHQ
V KHXULVWLFV DFWLRQIHHGEDFN WKDW ZLOO EH VDWLVILHG
RQFHWKHJRDOLVVXFFHVVIXOO\DFKLHYHG)LJXUHVKRZVVHFRQG
YHUVLRQWRWHPSODWHRI8VHU;6WRU\

$FFHSWDQFHFULWHULDDUHFRPPRQO\DGGHGWR86DGGUHVVLQJ
WRJXLGHDFFHSWDQFHWHVWLQJ$FFHSWDQFHWHVWLQJLVWKHSURFHVV
RI YHULI\LQJ ZKHWKHU VWRULHV ZHUH GHYHORSHG VXFK WKDW HDFK
ZRUNVH[DFWO\WKHZD\WKHFXVWRPHUH[SHFWHGLWWRZRUN,WLVD
PHFKDQLVPXVHGWRJXDUDQWHHVWKHIXQFWLRQDOLW\RIWKHVRIWZDUH
UHTXLUHPHQWV>@
$V D UHVXOW RI OLWHUDWXUH UHYLHZ VRPH UHFRPPHQGDWLRQV
ZHUH IRXQG UHJDUGLQJ GHDOLQJ ZLWK XVDELOLW\ 8; DQG 86
WRJHWKHU0RUHQRDQG<DJH>@KDYHLGHQWLILHGWKUHHZD\V
WR LQFRUSRUDWH IXQFWLRQDO XVDELOLW\ UHTXLUHPHQWV ZLWK 86 
DGGLQJ QHZ VWRULHV WR UHSUHVHQW WKH UHTXLUHPHQWV WKDW DUH
GLUHFWO\ GHULYHG IURP XVDELOLW\ FDOOHG XVDELOLW\ VWRULHV 
DGGLQJRUPRGLI\LQJWDVNVLQ86GHWDLOLQJDVQHHGHGDQG
DGGLQJ RU PRGLI\LQJ DFFHSWDQFH FULWHULD $FFRUGLQJ WKHVH
DXWKRUV WKHUH DUH XVDELOLW\ UHFRPPHQGDWLRQV WKDW KDYH
SRVLWLYHLPSDFWRQWKHILQDOTXDOLW\RIXVHRIVRIWZDUHV\VWHPV
FDQ EH FRQVLGHUHG DV IXQFWLRQDO XVDELOLW\ UHTXLUHPHQWV WKDW
FRPSOHPHQW WKH WUDGLWLRQDO UHTXLUHPHQWV DQG FDQ EH
GRFXPHQWHG DV 86  WKH XVDELOLW\ VWRULHV  EHFDXVH ERWK DUH
VLPLODU %DUNVGDOH HW DO >@ KDYH XVHG FRQFHSWXDO PDSV WR
GHVLJQDFRPSOHWHSLFWXUHWRWKHXVHUGHWDLOLQJWKHFRQQHFWLRQV
EHWZHHQ86DQGVFHQDULRV6FHQDULRVLVDGHVLJQDUWLIDFWXVHG
WRGHVFULEHKRZDSDUWLFXODUXVHUXVHVWKHV\VWHPIRUDVSHFLILF
WDVNFRQVLGHULQJWKHFRQWH[WDQGHQYLURQPHQWZKHUHLWZLOOEH
KHOG >@ 7KH FRQFHSWXDO PDSSLQJ ZDV LQLWLDOO\ SURSRVHG WR
PLWLJDWHDJLOHDQG8;WHDPFRQIOLFWWRFRPPXQLFDWHXVDELOLW\
DQG LQWHUDFWLRQ UHTXLUHPHQWV RQ DQ DJLOH SURMHFW 6RKDLE DQG
.KDQ>@DOVRUHFRPPHQGHGWKHXVHRIVFHQDULRVDORQJZLWK
86 LQ H[SORUDWLRQ SKDVH DQG KHXULVWLF HYDOXDWLRQ GXULQJ
DFFHSWDQFHWHVWLQJ


' (
%!$
)
$V D 8 
9 , ZDQW  QHHG 89& IRUWKLV 8 
 9& WKURXJK
ZKHQ;8 9<8 = 9>,HYDOXDWHWKDWP\JRDOZDVDFKLHYHGZKHQ
89

!

*
&KHFNV8 9WKURXJK8  9WRVDWLVI\84?

 #$9RIDFWLRQDQG84?

 #$9RIIHHGEDFN

,Q DGGLWLRQ WR WKH WHFKQLFDO OLWHUDWXUH UHYLHZ WKH

UHVHDUFKHUV GLG DQ HWKQRJUDSKLF VWXG\ LQ RUGHU WR XQGHUVWDQG
KRZ86ZHUHEHLQJGHYHORSHGE\32V,WZDVIRXQGWKDWWKHVH
ZHUHGHYHORSHGE\WUDGLWLRQDOPHWKRGVDVLQ)LJXUH

)LJ 8VHU;6WRU\WHPSODWHUHILQHGYHUVLRQ

7KHZRUNVKRSHQWLWOHG,QWHUDFWLRQ6WRULHVZDVRUJDQL]HG
LQRUGHUWRPDNHDZDUPXS IRUWKHZULWLQJRIVWRULHV DGGLQJ
WKH YLVLRQ RI 8; WKURXJK WKH XVH RI SHUVRQDV DQG 1LHOVHQ
V
KHXULVWLFV6L[32VDWWHQGHGWKHZRUNVKRS$OOWKHSDUWLFLSDQWV
KDGPRUHWKDQWHQ\HDUVRIH[SHULHQFHLQWKH,7ILHOG+RZHY
HUILYHRIWKHVL[SDUWLFLSDQWVKDGOLWWOHH[SHULHQFHZLWKDJLOH
PHWKRGRORJ\OHVVWKDQD\HDU

 'HOLEHUDWH,PSURYHPHQWVDQG%XLOG$UWLIDFW
)URPWKHRXWFRPHVRIWKHOLWHUDWXUHUHYLHZDQGWKHHWKQR
JUDSKLFVWXG\ZHSURSRVHGWRWKHWHDPVDJUDPPDULQFRUSR
UDWLQJSHUVRQDVDQG1LHOVHQ
VKHXULVWLFVWKDWVKRXOGEHDGRSW
HGRQZULWLQJWKH867KHJUDPPDUZDVGULYHQWRDQVZHUWKH
WKLUG UHVHDUFK TXHVWLRQ 54 DQG ZH FDOOHG WKH QHZ VWRU\
WHPSODWHRI8VHU;6WRU\)LJXUHVKRZVWKHLQLWLDOSURSRVDOWR
WKH8VHU;6WRU\%XLOG$UWLIDFWVWYHUVLRQ,QILUVWYHUVLRQ
RIDUWLIDFWWKHWUDGLWLRQDOJUDPPDUZDVPRGLILHGE\UHSODFLQJ
W\SHRIXVHU!E\SHUVRQDV!DQGVRPHUHDVRQ!E\
1LHOVHQ
V KHXULVWLFV! WR KLJKOLJKW IURP XVDELOLW\ SRLQW RI
YLHZWKHSRVLWLYHLPSDFWRIXVHULQWHUDFWLRQXVHUV
JRDOV

7KHZRUNVKRSZDVGLYLGHGLQWRLH[SODQDWLRQRIWKHFRQ
FHSW RI SHUVRQDV KHXULVWLF LQVSHFWLRQ XVLQJ 1LHOVHQ
V KHXULV
WLFV DQG 86 LL SUHVHQWDWLRQ RI WKH DUWLIDFWV /HDQ 3HUVRQDV
DQG8VHU;6WRU\DQGLLLDQH[HUFLVHRIZULWLQJVWRULHVIURP
WKHSURSRVHGWHPSODWHLQFOXGLQJWKHDFFHSWDQFHFULWHULD6RPH
GDWDFROOHFWHGZLWKHQGXVHUVGXULQJDZRUNVKRSDERXW8VHU
5HVHDUFK WHFKQLTXHV SUHYLRXVO\ SHUIRUPHG ZLWK DQRWKHU
JURXSRISDUWLFLSDQWVZHUHSURYLGHGDVVXSSOHPHQWDU\PDWHUL
DOWRVXSSRUWWKHSURSRVHGH[HUFLVH6RPHSHUVRQDVZHUHDOVR
LQFOXGHGLQWKLVPDWHULDO$WWKHHQGRIWKHZRUNVKRSDFWLYLWLHV




58

ZHGLVFXVVHGZLWKWKH32VWKHQH[WVWHSIRULPSOHPHQWLQJWKH
VWRULHVLQUHDOSURMHFWV

WRWKH5LJRU&\FOHWREHJURXQGHGRQDFDGHPLFIRXQGDWLRQV
%IROORZLQJIURPWKHJXLGHOLQHVRIWKH'65

 ,PSOHPHQWLQJ DQG 2EVHUYLQJ ,PSURYHPHQWV DQG

(YDOXDWLQJ$UWLIDFWV
,QWKLVVWHSWKH32VKDGDSHULRGRIRQHPRQWKWRLPSOH
PHQW WKH 8VHU; 6WRU\ LQ WR RQH RI WKHLU SURMHFWV D $IWHU
WKLV SHULRG UHVHDUFKHUV FDUULHG RXW LQGLYLGXDOV LQWHUYLHZV
ZLWKWKH32VWRFROOHFWWKHLUH[SHULHQFHVZLWKWKHLPSOHPHQWD
WLRQ RI 8VHU; 6WRULHV (YDOXDWH $UWLIDFW )LJXUH  VKRZV DQ
LPSOHPHQWHG 8VHU;6WRU\IRUWKHUHGHVLJQRISDUWUHODWHG IRU
WKHLVVXDQFHRIUHSRUWVIRUD7D[%RRNNHHSLQJVXEPRGXOH

5RGUtJXH] HW DO >@ SUHVHQWHG WKHLU UHVHDUFK WKURXJK WKH

'65 EXW WKH UHSRUWHG DFWLYLWLHV FRXOG FRQILJXUH SHUIHFWO\ DF
WLRQUHVHDUFKF\FOHV7KHDUWLIDFWVZHUHEXLOWLQFORVHFROODERUD
WLRQ ZLWK SUDFWLWLRQHUV DQG WKH UHVHDUFK FURVVHG WKH SKDVHV WR
GLDJQRVHWKHSUREOHPSURSRVHVROXWLRQVDQGWRLPSOHPHQWDQG
DVVHVV LPSURYHPHQWV ,Q WKLV FDVH UHVHDUFKHUV KDYH FKRVHQ WR
FRPSOHPHQWWKHUHVHDUFKZLWKIRFXVJURXSVHVVLRQVUDWKHUWKDQ
DGRSW DFWLRQ UHVHDUFK DV D PRUH FRPSUHKHQVLYH PHWKRGRORJ\
7KH 6RIW&R'H5 DSSURDFK VHHPV WR EH IHDVLEOH IRU UHVHDUFK
ZRUNZLWKVXFKFKDUDFWHULVWLFVDQGHVSHFLDOO\ZKHQLWRFFXUVLQ
HQYLURQPHQWVWKDWDGRSWDJLOHPHWKRGV)XUWKHUPRUHWKHLWHUD
WLYH QDWXUH RI WKH SURSRVDO DSSURDFK LV FRPSDWLEOH ZLWK DJLOH
HQYLURQPHQWV EHFDXVH LW LQWHQGV WR GHOLYHU YDOXDEOH DUWLIDFWV
DGDSWLQJTXLFNO\WRFKDQJHVHYROYLQJIURPQHZUHVHDUFKTXHV
WLRQV 'LIIHUHQWO\ WR WKH FDVHV GLVFXVVHG DERYH ZH FDQ VWDWH
WKDWWKHZRUNVSUHVHQWHGE\$GLNDULHWDO>@DQG$UGLWRHWDO
>@ GR QRW ILW ZLWK DQ DSSURDFK FRPELQLQJ WKH WZR UHVHDUFK
PHWKRGV


' +, - ..

!/+% /$

  /HR :DONHU!    LVVXH ILQDQFLDO UHSRUWLQJ DQG EDODQFH
VKHHWVILOWHULQJE\DJHQWV!
 WKHV\VWHPDOORZPHWRFKRRVHWKH
DJHQWWKDW,ZDQWWRILOWHU! 
<>IRULVVXLQJWKHUHSRUW!
UHJDUGOHVV RI WKH RUJDQL]DWLRQ WR ZKLFK , DP SODFHG LQ WKH V\VWHP LW
EHLQJ VXEVLGLDU\ RU FRQVROLGDWRU!@        
WKHUHSRUWRQO\OLVWWKHODXQFKHVFDUULHGIRUWKHVHOHFWHG
DJHQW!

5HJDUGLQJWKHDSSURDFKHVWKDWFRPELQH$5DQG'65>@
>@>@ZHSRLQWRXWWKDWDOOWKHVHLQLWLDWLYHVZHUHRULJLQDWHG
LQWKHILHOGRI,6ZKLFKIRFXVLVWRVHHNVROXWLRQVWREXVLQHVV
QHHGVIURPDV\VWHPLFWKLQNLQJ7KHUHVHDUFKLQ6(LQWXUQLV
FORVHO\ UHODWHG WR VRIWZDUH GHYHORSPHQW DFWLYLWLHV ZKLFK FDQ
EHEHWWHUXQGHUVWRRGIURPWKHSUDFWLWLRQHUVYLHZSRLQWV7KHUH
IRUHWKHFKRLFHRIWKH&0'PHWKRGFRQWULEXWHVWRDQDSSURDFK
WKDWLVLQWHQGHGWRJXLGHUHVHDUFKZLWKLQWKHVRIWZDUHGHYHORS
PHQWHQYLURQPHQWVKRSIORRUDQGKHDYLO\ LQIOXHQFHGE\FR
RSHUDWLRQEHWZHHQUHVHDUFKHUVDQGSUDFWLWLRQHUV


!

*
WKHV\VWHPZLOOYDOLGDWHLIWKDWDJHQWFRGHFDQEHXVHGIRUWKH
VHOHFWHG RUJDQL]DWLRQ ! 
  ILOWHULQJ E\ DJHQW FRGH 9   
+! &+!
WKHV\VWHPVKRXOGGLVSOD\DJHQWQDPHQH[WWRWKHFKRVHQFRGH
! 
FKRRVLQJDQDJHQWHLWKHUE\DJHQWFRGHRUVHDUFKLQJ! 
 +! &+!
)LJ 8VHU;6WRU\DQGDFFHSWDQFHFULWHULDLPSOHPHQWHGE\32

0RVWRIWKH32VDSSURYHGWKHXVHRIWKHLQWHUDFWLRQVWRULHV
DVLQWKHSURSRVHGWHPSODWH7KH\KDYHVKDUHGWKH8VHU;6WRU\
ZLWKWKH6&580WHDPVZKRUHDFWHGSRVLWLYHO\+RZHYHUWZR
32V KDG QRW LPSOHPHQWHGWKH 8VHU; 6WRULHV LQWKHLU SURMHFWV
VLQFH WKH\ ZHUH ZRUNLQJ RQ VPDOO FKDQJHV WKDW ZHUH UHODWHG
H[FOXVLYHO\ WR EXVLQHVV UXOHV OHJDO UHTXLUHPHQWV DQG VXFK
FKDQJHVZRXOGKDYHKDGQRLPSDFWRQXVHULQWHUDFWLRQ)XUWKHU
WHVWVZLOOEHQHHGHGWRHYDOXDWHWKHF\FOHLQZKLFK8VHU;6WR
ULHV DUH ZULWWHQ IURP LWHPV UHSRUWHG LQ WKH SURWRFRO IRU WKH
FRPPXQLFDWLRQ RI XVDELOLW\ UHFRPPHQGDWLRQV )LUVWO\ WKH
LWHPVUHSRUWHGLQWKHSURWRFROVKRXOGEHGLVFXVVHGEHWZHHQ8;
GHVLJQHUDQG6&580WHDP7KHQZKHWKHUWKH6&580WHDP
DJUHHVZLWKWKHUHFRPPHQGHGLWHPWKHLWHPZLOOEHZULWWHQLQ
8VHU; 6WRU\ IRUP ZLWK WKH SDUWLFLSDWLRQ RI WKH 8; GHVLJQHU
2WKHUZLVH WKH UHFRPPHQGDWLRQ LV GLVFDUGHG IRU H[DPSOH E\
WHFKQLFDOOLPLWDWLRQVSUHYHQWLQJLWWREHLPSOHPHQWHG&XUUHQW
O\ZHDUHLQYHVWLJDWLQJWKHOHYHORIGHWDLOZLWKWKHDFFHSWDQFH
FULWHULD DQG DOVR ZH KDYH EHHQ ZRUNLQJ LQ WKH GUDIWLQJ RI
JXLGHOLQHVWREHWWHUWDUJHWWKHXVHRIWKHLQWHUDFWLYHVWRULHV

1HYHUWKHOHVVVRPHOLPLWDWLRQVRIRXUDSSURDFKLVUHODWHGWR
DSSOLFDWLRQ FRQWH[W %HVLGHV WKLV WKH GHSHQGHQFH RI WKH FRP
SDQ\VVFKHGXOHEULQJVGLIILFXOW\WRNHHSWKHSDFHRILWHUDWLRQV
)LUVW RI DOO UHVHDUFKHUV QHHG WKH FRPSDQ\
V SULRU FRQVHQW WR
LQWHUDFW ZLWK WKH SUDFWLWLRQHUV GXULQJ D VXIILFLHQW SHULRG RI LW
HUDWLRQV WR XQGHUVWDQG WKH H[LVWLQJ SUDFWLFH GLVFXVV LPSURYH
PHQWV DQG HYDOXDWH ZKDW KDV EHHQ LPSOHPHQWHG +RZHYHU D
JUHDWHU QXPEHU RI LWHUDWLRQV VRPHWLPHV LW FDQ EH GLIILFXOW WR
NHHSDVFKHGXOHZLWKRXWXQH[SHFWHGHYHQWVLQYROYLQJWKHSUDF
WLWLRQHUV 6RPHWLPHV WKH SUDFWLWLRQHUV QHHG WR VROYH XUJHQW
PDWWHUEUHDNLQJWKHLWHUDWLRQVHYROXWLRQ
9, &21&/86,216$1')87+(5:25.
%\PHDQVRI6RIW&R'H5DQDSSURDFKPDWFKLQJ&0'DQG
'65PHWKRGVZHKDYHEHHQDEOHWRDFKLHYHRXUPDLQUHVHDUFK
JRDOV 7KH PDLQFRQWULEXWLRQRI WKLV DSSURDFKLV WRHQFRXUDJH
WKH $FDGHPLD,QGXVWU\ SDUWQHUVKLS ZKLFK WKH FRUH SURSRVDO
KDV EHHQ WR JXLGH WKH DFDGHPLF NQRZOHGJH WR PHHW WKH UHDO
QHHGV RI WKH 6RIWZDUH ,QGXVWU\ 7KH DXWKRUV EHOLHYH WKDW WKH
DSSURDFK DOORZV WKH GHHSO\ REVHUYDWLRQ RI SUREOHPV WKDW VWLOO
KDYH QRW EHHQ WKRURXJKO\ LQYHVWLJDWHG DFDGHPLFDOO\ ,Q WKLV
ZRUNIRULQVWDQFHZHGHWHFWHGDQLVVXHLQWKHFRPPXQLFDWLRQ
FDUULHG RXW EHWZHHQ 8; DQG DJLOH WHDPV ZKR KDYH GLIIHUHQW
SHUVSHFWLYHVRQXVDELOLW\DQGWKHXVHUUHVHDUFK
$VDIXWXUHZRUNZHDUHSODQQLQJWRDSSO\WKLVDSSURDFKWR
QHZUHVHDUFKSURMHFWV LQRUGHUWRYHULI\ZKHWKHUWKLVSURSRVDO
FDQEHH[WHQGHGWRRWKHUFRQWH[WVLQFOXGLQJRWKHUPDUNHWVHJ
PHQWVRULQD GLIIHUHQWGHYHORSPHQWSURFHVVWKDQDJLOH PHWK
RGRORJ\

9 ',6&866,21
7KH &0' DSSOLHG E\ $O%DLN DQG 0LOOHU >@ VRPHKRZ
HQFDSVXODWHV LQ WKH VHFRQG DQG WKLUG VWHS RI WKH PHWKRG WKH
SURFHVV RI GHVLJQ DQG GHYHORSPHQW RI DQ DUWLIDFW PRGHO WR
LGHQWLI\DQGHOLPLQDWHZDVWHEDVHGXSRQWKHWLPHDQGUHVRXUFHV
DYDLODEOH WR OHDQ LQLWLDWLYHV LQ ,7 FRQWH[WV ,Q WKH 6RIW&R'H5
DSSURDFKKRZHYHUWKHGHVLJQSURFHVVRIDUWLIDFWVLVKLJKOLJKW
HG VKRZLQJ WKDW WKH DFWLYLWLHV WR EXLOG E DQG HYDOXDWH E
DUWLIDFWV DUH FRQQHFWHG WR WKH 5HOHYDQFH &\FOH WR PHHW WKH
LQWHUHVWVDQGQHHGVRI,QGXVWU\$DVZHOODVEHLQJFRQQHFWHG

59

$&.12:/('*0(17

GHVLJQVFLHQFHIRUEULGJLQJWKHFROODERUDWLRQJDSEHWZHHQLQGXVWU\DQG
DFDGHPLDLQHPSLULFDOVRIWZDUHHQJLQHHULQJ,Q3URFHHGLQJVRIWKHQG
,QWHUQDWLRQDO :RUNVKRS RQ &RQGXFWLQJ (PSLULFDO 6WXGLHV LQ ,QGXVWU\
+\GHUDEDG,QGLD

:HZRXOGOLNHWRWKDQNWKHFRPSDQ\DQGWKHSUDFWLWLRQHUV
ZKRZHUHSDUWQHUVRIWKLVUHVHDUFKSURMHFW

>@ 5 %DVNHUYLOOH - 3ULHV+HMH DQG - 9HQDEOH 6RIW GHVLJQ VFLHQFH
PHWKRGRORJ\,Q'(65,67
0DOYHUQ3D86$

5()(5(1&(6

>@ 06HLQ2+HQIULGVVRQ63XUDR05RVVLDQG5/LQGJUHQ$FWLRQ
GHVLJQUHVHDUFK0,64XDUWHUO\YROQSS

>@ )6KXOO-&DUYHUDQG*+7UDYDVVRV$QHPSLULFDOPHWKRGRORJ\IRU
LQWURGXFLQJVRIWZDUHSURFHVVHV$&06,*62)76RIWZDUH(QJLQHHULQJ
1RWHVYROQSS

>@ 5:LHULQJDDQG$0RUDOL7HFKQLFDODFWLRQUHVHDUFKDVDYDOLGDWLRQ
PHWKRG LQ LQIRUPDWLRQ V\VWHPV GHVLJQ VFLHQFH ,Q 'HVLJQ 6FLHQFH
5HVHDUFK LQ ,QIRUPDWLRQ 6\VWHPV $GYDQFHV LQ 7KHRU\ DQG 3UDFWLFH
/DV9HJDV1HYDGD86$

>@ * 6DQWRV $ 5RFKD 7 &RQWH 0 %DUFHOORV DQG 5 3ULNODGQLFNL
$OLJQPHQW EHWZHHQ $FDGHP\ DQG ,QGXVWU\ $ 9LUWXRXV &\FOH WR
3URPRWH ,QQRYDWLRQ ,Q WK %UD]LOLDQ 6\PSRVLXP RQ 6RIWZDUH
(QJLQHHULQJ6%(6LQ3RUWXJXHVH1DWDO%UD]LO

>@ 5 &ROH 6 3XUDR 0 5RVVL DQG 0 6HLQ %HLQJ SURDFWLYH ZKHUH
DFWLRQUHVHDUFKPHHWVGHVLJQUHVHDUFK,Q,&,63URFHHGLQJV/DV
9HJDV1986$

>@ ' 5RPEDFK DQG 5 $FKDW] 5HVHDUFK &ROODERUDWLRQV EHWZHHQ

$FDGHPLD DQG ,QGXVWU\ ,Q 3URFHHGLQJ RI )26(  )XWXUH RI
6RIWZDUH(QJLQHHULQJ0LQQHDSROLV0LQQHVRWD

>@ ( /DZ 9 5RWR $ 3 9HUPHHUHQ - .RUW DQG 0 +DVVHQ]DKO
7RZDUGVDVKDUHGGHILQLWLRQRIXVHUH[SHULHQFH,Q&+,


>@ . 6FKZDEHU DQG - 6XWKHUODQG 7KH VFUXP JXLGH 6FUXP $OOLDQFH


>@ -1LHOVHQDQG'1RUPDQ7KH'HILQLWLRQRI8VHU([SHULHQFH
>2QOLQH@ $YDLODEOH KWWSZZZQQJURXSFRPDUWLFOHVGHILQLWLRQXVHU
H[SHULHQFH

>@ - 0 %URZQ * /LQGJDDUG DQG 5 %LGGOH &ROODERUDWLYH (YHQWV DQG
6KDUHG$UWHIDFWV$JLOH,QWHUDFWLRQ'HVLJQHUVDQG'HYHORSHUV:RUNLQJ
7RZDUG&RPPRQ$LPV,Q$JLOH&RQIHUHQFH$*,/(6DOW/DNH
&LW\8WDK

>@ 0,VRPXUVX$6LURWNLQ39ROWWLDQG0+DORQHQ8VHU([SHULHQFH
'HVLJQ*RHV$JLOHLQ/HDQ7UDQVIRUPDWLRQ $&DVH6WXG\,Q$JLOH
&RQIHUHQFH$*,/('DOODV7H[DV

>@ ' $YLVRQ ) /DX 0 0\HUV DQG 3 1LHOVHQ $FWLRQ 5HVHDUFK
&RPPXQLFDWLRQVRIWKH$&0YROQSS

>@ '6DODK5)3DLJHDQG3&DLUQV$V\VWHPDWLFOLWHUDWXUHUHYLHZIRU
DJLOH GHYHORSPHQW SURFHVVHV DQG XVHU FHQWUHG GHVLJQ LQWHJUDWLRQ ,Q
3URFHHGLQJV RI WKH WK ,QWHUQDWLRQDO &RQIHUHQFH RQ (YDOXDWLRQ DQG
$VVHVVPHQWLQ6RIWZDUH(QJLQHHULQJ($6(
1HZ<RUN

>@ :&DUU3KLORVRSK\PHWKRGRORJ\DQGDFWLRQUHVHDUFKYROQ
SS
>@ 3606DQWRVDQG*+7UDYDVVRV$FWLRQ5HVHDUFK&DQ6ZLQJWKH
%DODQFH LQ ([SHULPHQWDO 6RIWZDUH (QJLQHHULQJ $GYDQFHV LQ
&RPSXWHUVYROSS

>@ $&RRSHU55HLPDQQDQG'&URQLQ$ERXW)DFH7KH(VVHQWLDOVRI
,QWHUDFWLRQ'HVLJQ:LOH\3XEOLVKLQJ
>@ -1LHOVHQ+RZWR&RQGXFWD+HXULVWLF(YDOXDWLRQE>2QOLQH@
KWWSZZZQQJURXSFRPDUWLFOHVKRZWRFRQGXFWD
$YDLODEOH
KHXULVWLFHYDOXDWLRQ

>@ < 'LWWULFK 'RLQJ

(PSLULFDO 5HVHDUFK LQ 6RIWZDUH (QJLQHHULQJ 
ILQGLQJ D SDWK EHWZHHQ XQGHUVWDQGLQJ LQWHUYHQWLRQ DQG PHWKRG
GHYHORSPHQW
,Q6RFLDOWKLQNLQJ6RIWZDUHSUDFWLFH5.)OR\G(G
0,73UHVV

>@ $)OVWDG(/&/DZDQG.+RUQEN2XWOLHUVLQXVDELOLW\WHVWLQJ
+RZWRWUHDWXVDELOLW\SUREOHPVIRXQGIRURQO\RQHWHVWSDUWLFLSDQW",Q
3URFHHGLQJV RI WKH WK 1RUGLF &RQIHUHQFH RQ +XPDQ&RPSXWHU
,QWHUDFWLRQ0DNLQJ6HQVH7KURXJK

>@ 5 %DVNHUYLOOH :KDW GHVLJQ VFLHQFH LV QRW (XURSHDQ -RXUQDO RI
,QIRUPDWLRQ6\VWHPVYROQSS
>@ -9HQDEOH$)UDPHZRUNIRU'HVLJQ6FLHQFH5HVHDUFK$FWLYLWLHV,Q
3URFHHGLQJV RI WKH ,QIRUPDWLRQ 5HVRXUFH 0DQDJHPHQW $VVRFLDWLRQ
&RQIHUHQFH:DVKLQJWRQ'&86$

>@ / &DEDOOHUR $ 0 0RUHQR DQG $ 6HIIDK 3HUVRQD DV D 7RRO WR
,QYROYLQJ +XPDQ LQ $JLOH 0HWKRGV &RQWULEXWLRQV IURP +&, DQG
0DUNHWLQJ +XPDQ&HQWHUHG 6RIWZDUH (QJLQHHULQJ SS 


>@ 670DUFKDQG*)6PLWK'HVLJQDQGQDWXUDOVFLHQFHUHVHDUFKRQ
LQIRUPDWLRQ WHFKQRORJ\ 'HFLVLRQ VXSSRUW V\VWHPV YRO  Q  SS


>@ * 0LOOHU DQG / :LOOLDPV  3HUVRQDV 0RYLQJ %H\RQG 5ROH%DVHG
5HTXLUHPHQWV (QJLQHHULQJ ,Q 0LFURVRIW DQG 1RUWK &DUROLQD 6WDWH
8QLYHUVLW\

>@ -,LYDULDQG-9HQDEOH$FWLRQUHVHDUFKDQGGHVLJQVFLHQFHUHVHDUFK
VHHPLQJO\ VLPLODU EXW GHFLVLYHO\ GLVVLPLODU ,Q (&,6   WK
(XURSHDQ&RQIHUHQFHRQ,QIRUPDWLRQ6\VWHPV9HURQD,WDO\

>@ ,,QD\DW666DOLP60DUF]DN0'DQHYDDQG66KDPVKLUEDQG$
V\VWHPDWLFOLWHUDWXUHUHYLHZRQDJLOHUHTXLUHPHQWVHQJLQHHULQJSUDFWLFHV
DQGFKDOOHQJHV&RPSXWHUVLQ+XPDQ%HKDYLRU

>@ $ 5 +HYQHU 6 7 0DUFK - 3DUN DQG 6 5DP 'HVLJQ 6FLHQFH LQ
,QIRUPDWLRQ 6\VWHPV 5HVHDUFK 0,6 4XDUWHUO\ YRO  Q  SS 


>@ 0 &RKQ 8VHU 6WRULHV $SSOLHG IRU $JLOH 6RIWZDUH 'HYHORSPHQW 
HG%RVWRQ0$$GGLVRQ:HVOH\

>@ $ +HYQHU $ 7KUHH &\FOH 9LHZ RI 'HVLJQ 6FLHQFH 5HVHDUFK
6FDQGLQDYLDQ-RXUQDORI,QIRUPDWLRQ6\VWHPVYROQSS


>@ $ 0 0RUHQR DQG $ <DJH $JLOH 8VHU 6WRULHV (QULFKHG ZLWK
8VDELOLW\ ,Q $JLOH 3URFHVVHV LQ 6RIWZDUH (QJLQHHULQJ DQG ([WUHPH
3URJUDPPLQJ0DOP|6ZHGHQ

>@ < 'LWWULFK . 5|QNN| - (ULNVVRQ & +DQVVRQ DQG 2 /LQGHEHUJ
&RRSHUDWLYH PHWKRG GHYHORSPHQW (PSLULFDO 6RIWZDUH (QJLQHHULQJ
YROQSS

>@ -%DUNVGDOH(5DJDQDQG'0F&ULFNDUG(DVLQJ7HDP8VDELOLW\$
&RQFHSW0DSSLQJ$SSURDFK,Q$JLOH&RQIHUHQFH&KLFDJR,/

>@ 2 $O%DLN DQG - 0LOOHU :DVWH LGHQWLILFDWLRQ DQG HOLPLQDWLRQ LQ
LQIRUPDWLRQ WHFKQRORJ\ RUJDQL]DWLRQV (PSLULFDO 6RIWZDUH
(QJLQHHULQJYROQSS

>@ < -LD 0 . /DUXVGRWWLU DQG c &DMDQGHU 7KH 8VDJH RI 8VDELOLW\
7HFKQLTXHV LQ 6FUXP 3URMHFWV ,Q ,QWHUQDWLRQDO &RQIHUHQFH RQ
+XPDQ&HQWHUHG6RIWZDUH(QJLQHHULQJ7RXORXVH)UDQFH

>@ & $UGLWR 3 %XRQR ' &DLYDQR 0 ) &RVWDELOH DQG 5 /DQ]LORWWL
,QYHVWLJDWLQJDQGSURPRWLQJ8;SUDFWLFHLQLQGXVWU\$QH[SHULPHQWDO
VWXG\ ,QWHUQDWLRQDO -RXUQDO RI +XPDQ&RPSXWHU 6WXGLHV YRO  Q
SS

>@ 26RKDLEDQG..KDQ,QFRUSRUDWLQJ'LVFRXQW8VDELOLW\LQ([WUHPH
3URJUDPPLQJ ,QWHUQDWLRQDO -RXUQDO RI 6RIWZDUH (QJLQHHULQJ DQG ,WV
$SSOLFDWLRQVYROSS

>@ 6 $GLNDUL & 0F'RQDOG DQG - &DPSEHOO /LWWOH GHVLJQ XSIURQW $
GHVLJQVFLHQFHDSSURDFKWRLQWHJUDWLQJXVDELOLW\LQWRDJLOHUHTXLUHPHQWV
HQJLQHHULQJ,Q+XPDQFRPSXWHULQWHUDFWLRQ1HZWUHQGV
>@ 3 5RGUtJXH] 3 .XYDMD DQG 0 2LYR /HVVRQV OHDUQHG RQ DSSO\LQJ

60