You are on page 1of 12



7HVWGULYHQGHYHORSPHQW:LNLSHGLDWKHIUHHHQF\FORSHGLD

7HVWGULYHQGHYHORSPHQW
)URP:LNLSHGLDWKHIUHHHQF\FORSHGLD

7HVWGULYHQGHYHORSPHQW 7'' LVDVRIWZDUHGHYHORSPHQWSURFHVVWKDWUHOLHVRQWKHUHSHWLWLRQRIDYHU\


VKRUWGHYHORSPHQWF\FOHILUVWWKHGHYHORSHUZULWHVDQ LQLWLDOO\IDLOLQJ DXWRPDWHGWHVWFDVHWKDWGHILQHVD
GHVLUHGLPSURYHPHQWRUQHZIXQFWLRQWKHQSURGXFHVWKHPLQLPXPDPRXQWRIFRGHWRSDVVWKDWWHVWDQG
ILQDOO\UHIDFWRUVWKHQHZFRGHWRDFFHSWDEOHVWDQGDUGV.HQW%HFNZKRLVFUHGLWHGZLWKKDYLQJGHYHORSHGRU

UHGLVFRYHUHG
>@WKHWHFKQLTXHVWDWHGLQWKDW7''HQFRXUDJHVVLPSOHGHVLJQVDQGLQVSLUHV
FRQILGHQFH>@
7HVWGULYHQGHYHORSPHQWLVUHODWHGWRWKHWHVWILUVWSURJUDPPLQJFRQFHSWVRIH[WUHPHSURJUDPPLQJEHJXQ
LQ>@EXWPRUHUHFHQWO\KDVFUHDWHGPRUHJHQHUDOLQWHUHVWLQLWVRZQULJKW>@
3URJUDPPHUVDOVRDSSO\WKHFRQFHSWWRLPSURYLQJDQGGHEXJJLQJOHJDF\FRGHGHYHORSHGZLWKROGHU
WHFKQLTXHV>@

&RQWHQWV
 7HVWGULYHQGHYHORSPHQWF\FOH
 $GGDWHVW
 5XQDOOWHVWVDQGVHHLIWKHQHZRQHIDLOV
 :ULWHVRPHFRGH
 5XQWHVWV
 5HIDFWRUFRGH
 5HSHDW
 'HYHORSPHQWVW\OH
 .HHSWKHXQLWVPDOO
 %HVWSUDFWLFHV
 7HVWVWUXFWXUH
 ,QGLYLGXDOEHVWSUDFWLFHV
 3UDFWLFHVWRDYRLGRUDQWLSDWWHUQV
 %HQHILWV
KWWSVHQZLNLSHGLDRUJZLNL7HVWGULYHQBGHYHORSPHQW





7HVWGULYHQGHYHORSPHQW:LNLSHGLDWKHIUHHHQF\FORSHGLD

 /LPLWDWLRQV
 7HVWGULYHQZRUN
 7''DQG$7''
 7''DQG%''
 &RGHYLVLELOLW\
 6RIWZDUHIRU7''
 [8QLWIUDPHZRUNV
 7$3UHVXOWV
 )DNHVPRFNVDQGLQWHJUDWLRQWHVWV
 7''IRUFRPSOH[V\VWHPV
 'HVLJQLQJIRUWHVWDELOLW\
 0DQDJLQJWHVWVIRUODUJHWHDPV
 6HHDOVR
 5HIHUHQFHV
 ([WHUQDOOLQNV

7HVWGULYHQGHYHORSPHQWF\FOH
7KHIROORZLQJVHTXHQFHLVEDVHGRQWKHERRN7HVW'ULYHQ
'HYHORSPHQWE\([DPSOH>@

$GGDWHVW
,QWHVWGULYHQGHYHORSPHQWHDFKQHZIHDWXUHEHJLQVZLWKZULWLQJD
WHVW7RZULWHDWHVWWKHGHYHORSHUPXVWFOHDUO\XQGHUVWDQGWKH
IHDWXUH
VVSHFLILFDWLRQDQGUHTXLUHPHQWV7KHGHYHORSHUFDQ
$JUDSKLFDOUHSUHVHQWDWLRQRIWKH
DFFRPSOLVKWKLVWKURXJKXVHFDVHVDQGXVHUVWRULHVWRFRYHUWKH
7HVW'ULYHQ'HYHORSPHQWOLIHF\FOH
UHTXLUHPHQWVDQGH[FHSWLRQFRQGLWLRQVDQGFDQZULWHWKHWHVWLQ
ZKDWHYHUWHVWLQJIUDPHZRUNLVDSSURSULDWHWRWKHVRIWZDUH
HQYLURQPHQW,WFRXOGEHDPRGLILHGYHUVLRQRIDQH[LVWLQJWHVW7KLVLVDGLIIHUHQWLDWLQJIHDWXUHRIWHVW
GULYHQGHYHORSPHQWYHUVXVZULWLQJXQLWWHVWVDIWHUWKHFRGHLVZULWWHQLWPDNHVWKHGHYHORSHUIRFXVRQWKH
UHTXLUHPHQWVEHIRUHZULWLQJWKHFRGHDVXEWOHEXWLPSRUWDQWGLIIHUHQFH
KWWSVHQZLNLSHGLDRUJZLNL7HVWGULYHQBGHYHORSPHQW





7HVWGULYHQGHYHORSPHQW:LNLSHGLDWKHIUHHHQF\FORSHGLD

5XQDOOWHVWVDQGVHHLIWKHQHZRQHIDLOV
7KLVYDOLGDWHVWKDWWKHWHVWKDUQHVVLVZRUNLQJFRUUHFWO\WKDWWKHQHZWHVWGRHVQRWPLVWDNHQO\SDVVZLWKRXW
UHTXLULQJDQ\QHZFRGHDQGWKDWWKHUHTXLUHGIHDWXUHGRHVQRWDOUHDG\H[LVW7KLVVWHSDOVRWHVWVWKHWHVW
LWVHOILQWKHQHJDWLYHLWUXOHVRXWWKHSRVVLELOLW\WKDWWKHQHZWHVWDOZD\VSDVVHVDQGWKHUHIRUHLVZRUWKOHVV
7KHQHZWHVWVKRXOGDOVRIDLOIRUWKHH[SHFWHGUHDVRQ7KLVVWHSLQFUHDVHVWKHGHYHORSHU
VFRQILGHQFHWKDWWKH
XQLWWHVWLVWHVWLQJWKHFRUUHFWFRQVWUDLQWDQGSDVVHVRQO\LQLQWHQGHGFDVHV

:ULWHVRPHFRGH
7KHQH[WVWHSLVWRZULWHVRPHFRGHWKDWFDXVHVWKHWHVWWRSDVV7KHQHZFRGHZULWWHQDWWKLVVWDJHLVQRW
SHUIHFWDQGPD\IRUH[DPSOHSDVVWKHWHVWLQDQLQHOHJDQWZD\7KDWLVDFFHSWDEOHEHFDXVHLWZLOOEH
LPSURYHGDQGKRQHGLQ6WHS
$WWKLVSRLQWWKHRQO\SXUSRVHRIWKHZULWWHQFRGHLVWRSDVVWKHWHVWQRIXUWKHU DQGWKHUHIRUHXQWHVWHG
IXQFWLRQDOLW\VKRXOGEHSUHGLFWHGQRU
DOORZHGIRU
DWDQ\VWDJH

5XQWHVWV
,IDOOWHVWFDVHVQRZSDVVWKHSURJUDPPHUFDQEHFRQILGHQWWKDWWKHQHZFRGHPHHWVWKHWHVWUHTXLUHPHQWV
DQGGRHVQRWEUHDNRUGHJUDGHDQ\H[LVWLQJIHDWXUHV,IWKH\GRQRWWKHQHZFRGHPXVWEHDGMXVWHGXQWLOWKH\
GR

5HIDFWRUFRGH
7KHJURZLQJFRGHEDVHPXVWEHFOHDQHGXSUHJXODUO\GXULQJWHVWGULYHQGHYHORSPHQW1HZFRGHFDQEH
PRYHGIURPZKHUHLWZDVFRQYHQLHQWIRUSDVVLQJDWHVWWRZKHUHLWPRUHORJLFDOO\EHORQJV'XSOLFDWLRQPXVW
EHUHPRYHG2EMHFWFODVVPRGXOHYDULDEOHDQGPHWKRGQDPHVVKRXOGFOHDUO\UHSUHVHQWWKHLUFXUUHQW
SXUSRVHDQGXVHDVH[WUDIXQFWLRQDOLW\LVDGGHG$VIHDWXUHVDUHDGGHGPHWKRGERGLHVFDQJHWORQJHUDQG
RWKHUREMHFWVODUJHU7KH\EHQHILWIURPEHLQJVSOLWDQGWKHLUSDUWVFDUHIXOO\QDPHGWRLPSURYHUHDGDELOLW\
DQGPDLQWDLQDELOLW\ZKLFKZLOOEHLQFUHDVLQJO\YDOXDEOHODWHULQWKHVRIWZDUHOLIHF\FOH,QKHULWDQFH
KLHUDUFKLHVPD\EHUHDUUDQJHGWREHPRUHORJLFDODQGKHOSIXODQGSHUKDSVWREHQHILWIURPUHFRJQLVHG
GHVLJQSDWWHUQV7KHUHDUHVSHFLILFDQGJHQHUDOJXLGHOLQHVIRUUHIDFWRULQJDQGIRUFUHDWLQJFOHDQFRGH>@>@
%\FRQWLQXDOO\UHUXQQLQJWKHWHVWFDVHVWKURXJKRXWHDFKUHIDFWRULQJSKDVHWKHGHYHORSHUFDQEHFRQILGHQW
WKDWSURFHVVLVQRWDOWHULQJDQ\H[LVWLQJIXQFWLRQDOLW\
7KHFRQFHSWRIUHPRYLQJGXSOLFDWLRQLVDQLPSRUWDQWDVSHFWRIDQ\VRIWZDUHGHVLJQ,QWKLVFDVHKRZHYHULW
DOVRDSSOLHVWRWKHUHPRYDORIDQ\GXSOLFDWLRQEHWZHHQWKHWHVWFRGHDQGWKHSURGXFWLRQFRGHIRUH[DPSOH
PDJLFQXPEHUVRUVWULQJVUHSHDWHGLQERWKWRPDNHWKHWHVWSDVVLQ6WHS

5HSHDW
6WDUWLQJZLWKDQRWKHUQHZWHVWWKHF\FOHLVWKHQUHSHDWHGWRSXVKIRUZDUGWKHIXQFWLRQDOLW\7KHVL]HRIWKH
VWHSVVKRXOGDOZD\VEHVPDOOZLWKDVIHZDVWRHGLWVEHWZHHQHDFKWHVWUXQ,IQHZFRGHGRHVQRW
UDSLGO\VDWLVI\DQHZWHVWRURWKHUWHVWVIDLOXQH[SHFWHGO\WKHSURJUDPPHUVKRXOGXQGRRUUHYHUWLQ
SUHIHUHQFHWRH[FHVVLYHGHEXJJLQJ&RQWLQXRXVLQWHJUDWLRQKHOSVE\SURYLGLQJUHYHUWLEOHFKHFNSRLQWV:KHQ
KWWSVHQZLNLSHGLDRUJZLNL7HVWGULYHQBGHYHORSPHQW





7HVWGULYHQGHYHORSPHQW:LNLSHGLDWKHIUHHHQF\FORSHGLD

XVLQJH[WHUQDOOLEUDULHVLWLVLPSRUWDQWQRWWRPDNHLQFUHPHQWVWKDWDUHVRVPDOODVWREHHIIHFWLYHO\PHUHO\
WHVWLQJWKHOLEUDU\LWVHOI>@XQOHVVWKHUHLVVRPHUHDVRQWREHOLHYHWKDWWKHOLEUDU\LVEXJJ\RULVQRW
VXIILFLHQWO\IHDWXUHFRPSOHWHWRVHUYHDOOWKHQHHGVRIWKHVRIWZDUHXQGHUGHYHORSPHQW

'HYHORSPHQWVW\OH
7KHUHDUHYDULRXVDVSHFWVWRXVLQJWHVWGULYHQGHYHORSPHQWIRUH[DPSOHWKHSULQFLSOHVRINHHSLWVLPSOH
VWXSLG .,66 DQG<RXDUHQ
WJRQQDQHHGLW <$*1, %\IRFXVLQJRQZULWLQJRQO\WKHFRGHQHFHVVDU\WR
SDVVWHVWVGHVLJQVFDQRIWHQEHFOHDQHUDQGFOHDUHUWKDQLVDFKLHYHGE\RWKHUPHWKRGV>@,Q7HVW'ULYHQ
'HYHORSPHQWE\([DPSOH.HQW%HFNDOVRVXJJHVWVWKHSULQFLSOH)DNHLWWLOO\RXPDNHLW
7RDFKLHYHVRPHDGYDQFHGGHVLJQFRQFHSWVXFKDVDGHVLJQSDWWHUQWHVWVDUHZULWWHQWKDWJHQHUDWHWKDW
GHVLJQ7KHFRGHPD\UHPDLQVLPSOHUWKDQWKHWDUJHWSDWWHUQEXWVWLOOSDVVDOOUHTXLUHGWHVWV7KLVFDQEH
XQVHWWOLQJDWILUVWEXWLWDOORZVWKHGHYHORSHUWRIRFXVRQO\RQZKDWLVLPSRUWDQW
:ULWLQJWKHWHVWVILUVW7KHWHVWVVKRXOGEHZULWWHQEHIRUHWKHIXQFWLRQDOLW\WKDWLVWREHWHVWHG7KLVKDVEHHQ
FODLPHGWRKDYHPDQ\EHQHILWV,WKHOSVHQVXUHWKDWWKHDSSOLFDWLRQLVZULWWHQIRUWHVWDELOLW\DVWKH
GHYHORSHUVPXVWFRQVLGHUKRZWRWHVWWKHDSSOLFDWLRQIURPWKHRXWVHWUDWKHUWKDQDGGLQJLWODWHU,WDOVR
HQVXUHVWKDWWHVWVIRUHYHU\IHDWXUHJHWZULWWHQ$GGLWLRQDOO\ZULWLQJWKHWHVWVILUVWOHDGVWRDGHHSHUDQG
HDUOLHUXQGHUVWDQGLQJRIWKHSURGXFWUHTXLUHPHQWVHQVXUHVWKHHIIHFWLYHQHVVRIWKHWHVWFRGHDQGPDLQWDLQV
DFRQWLQXDOIRFXVRQVRIWZDUHTXDOLW\>@:KHQZULWLQJIHDWXUHILUVWFRGHWKHUHLVDWHQGHQF\E\GHYHORSHUV
DQGRUJDQLVDWLRQVWRSXVKWKHGHYHORSHURQWRWKHQH[WIHDWXUHHYHQQHJOHFWLQJWHVWLQJHQWLUHO\7KHILUVW
7''WHVWPLJKWQRWHYHQFRPSLOHDWILUVWEHFDXVHWKHFODVVHVDQGPHWKRGVLWUHTXLUHVPD\QRW\HWH[LVW
1HYHUWKHOHVVWKDWILUVWWHVWIXQFWLRQVDVWKHEHJLQQLQJRIDQH[HFXWDEOHVSHFLILFDWLRQ>@
(DFKWHVWFDVHIDLOVLQLWLDOO\7KLVHQVXUHVWKDWWKHWHVWUHDOO\ZRUNVDQGFDQFDWFKDQHUURU2QFHWKLVLV
VKRZQWKHXQGHUO\LQJIXQFWLRQDOLW\FDQEHLPSOHPHQWHG7KLVKDVOHGWRWKHWHVWGULYHQGHYHORSPHQW
PDQWUDZKLFKLVUHGJUHHQUHIDFWRUZKHUHUHGPHDQVIDLODQGJUHHQPHDQVSDVV7HVWGULYHQ
GHYHORSPHQWFRQVWDQWO\UHSHDWVWKHVWHSVRIDGGLQJWHVWFDVHVWKDWIDLOSDVVLQJWKHPDQGUHIDFWRULQJ
5HFHLYLQJWKHH[SHFWHGWHVWUHVXOWVDWHDFKVWDJHUHLQIRUFHVWKHGHYHORSHU
VPHQWDOPRGHORIWKHFRGHERRVWV
FRQILGHQFHDQGLQFUHDVHVSURGXFWLYLW\

.HHSWKHXQLWVPDOO
)RU7''DXQLWLVPRVWFRPPRQO\GHILQHGDVDFODVVRUDJURXSRIUHODWHGIXQFWLRQVRIWHQFDOOHGDPRGXOH
.HHSLQJXQLWVUHODWLYHO\VPDOOLVFODLPHGWRSURYLGHFULWLFDOEHQHILWVLQFOXGLQJ
5HGXFHGGHEXJJLQJHIIRUW:KHQWHVWIDLOXUHVDUHGHWHFWHGKDYLQJVPDOOHUXQLWVDLGVLQWUDFNLQJ
GRZQHUURUV
6HOIGRFXPHQWLQJWHVWV6PDOOWHVWFDVHVDUHHDVLHUWRUHDGDQGWRXQGHUVWDQG>@
$GYDQFHGSUDFWLFHVRIWHVWGULYHQGHYHORSPHQWFDQOHDGWRDFFHSWDQFHWHVWGULYHQGHYHORSPHQW $7''
DQG6SHFLILFDWLRQE\H[DPSOHZKHUHWKHFULWHULDVSHFLILHGE\WKHFXVWRPHUDUHDXWRPDWHGLQWRDFFHSWDQFH
WHVWVZKLFKWKHQGULYHWKHWUDGLWLRQDOXQLWWHVWGULYHQGHYHORSPHQW 87'' SURFHVV>@7KLVSURFHVV
HQVXUHVWKHFXVWRPHUKDVDQDXWRPDWHGPHFKDQLVPWRGHFLGHZKHWKHUWKHVRIWZDUHPHHWVWKHLUUHTXLUHPHQWV
:LWK$7''WKHGHYHORSPHQWWHDPQRZKDVDVSHFLILFWDUJHWWRVDWLVI\WKHDFFHSWDQFHWHVWVZKLFKNHHSV
WKHPFRQWLQXRXVO\IRFXVHGRQZKDWWKHFXVWRPHUUHDOO\ZDQWVIURPHDFKXVHUVWRU\
KWWSVHQZLNLSHGLDRUJZLNL7HVWGULYHQBGHYHORSPHQW





7HVWGULYHQGHYHORSPHQW:LNLSHGLDWKHIUHHHQF\FORSHGLD

%HVWSUDFWLFHV
7HVWVWUXFWXUH
(IIHFWLYHOD\RXWRIDWHVWFDVHHQVXUHVDOOUHTXLUHGDFWLRQVDUHFRPSOHWHGLPSURYHVWKHUHDGDELOLW\RIWKHWHVW
FDVHDQGVPRRWKVWKHIORZRIH[HFXWLRQ&RQVLVWHQWVWUXFWXUHKHOSVLQEXLOGLQJDVHOIGRFXPHQWLQJWHVWFDVH
$FRPPRQO\DSSOLHGVWUXFWXUHIRUWHVWFDVHVKDV  VHWXS  H[HFXWLRQ  YDOLGDWLRQDQG  FOHDQXS
6HWXS3XWWKH8QLW8QGHU7HVW 887 RUWKHRYHUDOOWHVWV\VWHPLQWKHVWDWHQHHGHGWRUXQWKHWHVW
([HFXWLRQ7ULJJHUGULYHWKH887WRSHUIRUPWKHWDUJHWEHKDYLRUDQGFDSWXUHDOORXWSXWVXFKDV
UHWXUQYDOXHVDQGRXWSXWSDUDPHWHUV7KLVVWHSLVXVXDOO\YHU\VLPSOH
9DOLGDWLRQ(QVXUHWKHUHVXOWVRIWKHWHVWDUHFRUUHFW7KHVHUHVXOWVPD\LQFOXGHH[SOLFLWRXWSXWV
FDSWXUHGGXULQJH[HFXWLRQRUVWDWHFKDQJHVLQWKH887 8$7
&OHDQXS5HVWRUHWKH887RUWKHRYHUDOOWHVWV\VWHPWRWKHSUHWHVWVWDWH7KLVUHVWRUDWLRQSHUPLWV
DQRWKHUWHVWWRH[HFXWHLPPHGLDWHO\DIWHUWKLVRQH>@

,QGLYLGXDOEHVWSUDFWLFHV
,QGLYLGXDOEHVWSUDFWLFHVVWDWHVWKDWRQHVKRXOG
6HSDUDWHFRPPRQVHWXSDQGWHDUGRZQORJLFLQWRWHVWVXSSRUWVHUYLFHVXWLOL]HGE\WKHDSSURSULDWHWHVW
FDVHV
.HHSHDFKWHVWRUDFOHIRFXVHGRQRQO\WKHUHVXOWVQHFHVVDU\WRYDOLGDWHLWVWHVW
'HVLJQWLPHUHODWHGWHVWVWRDOORZWROHUDQFHIRUH[HFXWLRQLQQRQUHDOWLPHRSHUDWLQJV\VWHPV7KH
FRPPRQSUDFWLFHRIDOORZLQJDSHUFHQWPDUJLQIRUODWHH[HFXWLRQUHGXFHVWKHSRWHQWLDOQXPEHU
RIIDOVHQHJDWLYHVLQWHVWH[HFXWLRQ
7UHDW\RXUWHVWFRGHZLWKWKHVDPHUHVSHFWDV\RXUSURGXFWLRQFRGH,WDOVRPXVWZRUNFRUUHFWO\IRU
ERWKSRVLWLYHDQGQHJDWLYHFDVHVODVWDORQJWLPHDQGEHUHDGDEOHDQGPDLQWDLQDEOH
*HWWRJHWKHUZLWK\RXUWHDPDQGUHYLHZ\RXUWHVWVDQGWHVWSUDFWLFHVWRVKDUHHIIHFWLYHWHFKQLTXHVDQG
FDWFKEDGKDELWV,WPD\EHKHOSIXOWRUHYLHZWKLVVHFWLRQGXULQJ\RXUGLVFXVVLRQ>@

3UDFWLFHVWRDYRLGRUDQWLSDWWHUQV
+DYLQJWHVWFDVHVGHSHQGRQV\VWHPVWDWHPDQLSXODWHGIURPSUHYLRXVO\H[HFXWHGWHVWFDVHV
'HSHQGHQFLHVEHWZHHQWHVWFDVHV$WHVWVXLWHZKHUHWHVWFDVHVDUHGHSHQGHQWXSRQHDFKRWKHULV
EULWWOHDQGFRPSOH[([HFXWLRQRUGHUVKRXOGQRWEHSUHVXPHG%DVLFUHIDFWRULQJRIWKHLQLWLDOWHVW
FDVHVRUVWUXFWXUHRIWKH887FDXVHVDVSLUDORILQFUHDVLQJO\SHUYDVLYHLPSDFWVLQDVVRFLDWHGWHVWV
,QWHUGHSHQGHQWWHVWV,QWHUGHSHQGHQWWHVWVFDQFDXVHFDVFDGLQJIDOVHQHJDWLYHV$IDLOXUHLQDQHDUO\
WHVWFDVHEUHDNVDODWHUWHVWFDVHHYHQLIQRDFWXDOIDXOWH[LVWVLQWKH887LQFUHDVLQJGHIHFWDQDO\VLV
DQGGHEXJHIIRUWV
7HVWLQJSUHFLVHH[HFXWLRQEHKDYLRUWLPLQJRUSHUIRUPDQFH
%XLOGLQJDOONQRZLQJRUDFOHV$QRUDFOHWKDWLQVSHFWVPRUHWKDQQHFHVVDU\LVPRUHH[SHQVLYHDQG
EULWWOHRYHUWLPH7KLVYHU\FRPPRQHUURULVGDQJHURXVEHFDXVHLWFDXVHVDVXEWOHEXWSHUYDVLYHWLPH
VLQNDFURVVWKHFRPSOH[SURMHFW>@
7HVWLQJLPSOHPHQWDWLRQGHWDLOV
6ORZUXQQLQJWHVWV

%HQHILWV
KWWSVHQZLNLSHGLDRUJZLNL7HVWGULYHQBGHYHORSPHQW





7HVWGULYHQGHYHORSPHQW:LNLSHGLDWKHIUHHHQF\FORSHGLD

$VWXG\IRXQGWKDWXVLQJ7''PHDQWZULWLQJPRUHWHVWVDQGLQWXUQSURJUDPPHUVZKRZURWHPRUH
WHVWVWHQGHGWREHPRUHSURGXFWLYH>@+\SRWKHVHVUHODWLQJWRFRGHTXDOLW\DQGDPRUHGLUHFWFRUUHODWLRQ
EHWZHHQ7''DQGSURGXFWLYLW\ZHUHLQFRQFOXVLYH>@
3URJUDPPHUVXVLQJSXUH7''RQQHZ JUHHQILHOG SURMHFWVUHSRUWHGWKH\RQO\UDUHO\IHOWWKHQHHGWR
LQYRNHDGHEXJJHU8VHGLQFRQMXQFWLRQZLWKDYHUVLRQFRQWUROV\VWHPZKHQWHVWVIDLOXQH[SHFWHGO\
UHYHUWLQJWKHFRGHWRWKHODVWYHUVLRQWKDWSDVVHGDOOWHVWVPD\RIWHQEHPRUHSURGXFWLYHWKDQGHEXJJLQJ>@
7HVWGULYHQGHYHORSPHQWRIIHUVPRUHWKDQMXVWVLPSOHYDOLGDWLRQRIFRUUHFWQHVVEXWFDQDOVRGULYHWKH
GHVLJQRIDSURJUDP>@%\IRFXVLQJRQWKHWHVWFDVHVILUVWRQHPXVWLPDJLQHKRZWKHIXQFWLRQDOLW\LVXVHG
E\FOLHQWV LQWKHILUVWFDVHWKHWHVWFDVHV 6RWKHSURJUDPPHULVFRQFHUQHGZLWKWKHLQWHUIDFHEHIRUHWKH
LPSOHPHQWDWLRQ7KLVEHQHILWLVFRPSOHPHQWDU\WR'HVLJQE\&RQWUDFWDVLWDSSURDFKHVFRGHWKURXJKWHVW
FDVHVUDWKHUWKDQWKURXJKPDWKHPDWLFDODVVHUWLRQVRUSUHFRQFHSWLRQV
7HVWGULYHQGHYHORSPHQWRIIHUVWKHDELOLW\WRWDNHVPDOOVWHSVZKHQUHTXLUHG,WDOORZVDSURJUDPPHUWR
IRFXVRQWKHWDVNDWKDQGDVWKHILUVWJRDOLVWRPDNHWKHWHVWSDVV([FHSWLRQDOFDVHVDQGHUURUKDQGOLQJDUH
QRWFRQVLGHUHGLQLWLDOO\DQGWHVWVWRFUHDWHWKHVHH[WUDQHRXVFLUFXPVWDQFHVDUHLPSOHPHQWHGVHSDUDWHO\
7HVWGULYHQGHYHORSPHQWHQVXUHVLQWKLVZD\WKDWDOOZULWWHQFRGHLVFRYHUHGE\DWOHDVWRQHWHVW7KLVJLYHV
WKHSURJUDPPLQJWHDPDQGVXEVHTXHQWXVHUVDJUHDWHUOHYHORIFRQILGHQFHLQWKHFRGH
:KLOHLWLVWUXHWKDWPRUHFRGHLVUHTXLUHGZLWK7''WKDQZLWKRXW7''EHFDXVHRIWKHXQLWWHVWFRGHWKH
WRWDOFRGHLPSOHPHQWDWLRQWLPHFRXOGEHVKRUWHUEDVHGRQDPRGHOE\0OOHUDQG3DGEHUJ>@/DUJH
QXPEHUVRIWHVWVKHOSWROLPLWWKHQXPEHURIGHIHFWVLQWKHFRGH7KHHDUO\DQGIUHTXHQWQDWXUHRIWKHWHVWLQJ
KHOSVWRFDWFKGHIHFWVHDUO\LQWKHGHYHORSPHQWF\FOHSUHYHQWLQJWKHPIURPEHFRPLQJHQGHPLFDQG
H[SHQVLYHSUREOHPV(OLPLQDWLQJGHIHFWVHDUO\LQWKHSURFHVVXVXDOO\DYRLGVOHQJWK\DQGWHGLRXVGHEXJJLQJ
ODWHULQWKHSURMHFW
7''FDQOHDGWRPRUHPRGXODUL]HGIOH[LEOHDQGH[WHQVLEOHFRGH7KLVHIIHFWRIWHQFRPHVDERXWEHFDXVH
WKHPHWKRGRORJ\UHTXLUHVWKDWWKHGHYHORSHUVWKLQNRIWKHVRIWZDUHLQWHUPVRIVPDOOXQLWVWKDWFDQEH
ZULWWHQDQGWHVWHGLQGHSHQGHQWO\DQGLQWHJUDWHGWRJHWKHUODWHU7KLVOHDGVWRVPDOOHUPRUHIRFXVHGFODVVHV
ORRVHUFRXSOLQJDQGFOHDQHULQWHUIDFHV7KHXVHRIWKHPRFNREMHFWGHVLJQSDWWHUQDOVRFRQWULEXWHVWRWKH
RYHUDOOPRGXODUL]DWLRQRIWKHFRGHEHFDXVHWKLVSDWWHUQUHTXLUHVWKDWWKHFRGHEHZULWWHQVRWKDWPRGXOHVFDQ
EHVZLWFKHGHDVLO\EHWZHHQPRFNYHUVLRQVIRUXQLWWHVWLQJDQGUHDOYHUVLRQVIRUGHSOR\PHQW
%HFDXVHQRPRUHFRGHLVZULWWHQWKDQQHFHVVDU\WRSDVVDIDLOLQJWHVWFDVHDXWRPDWHGWHVWVWHQGWRFRYHU
HYHU\FRGHSDWK)RUH[DPSOHIRUD7''GHYHORSHUWRDGGDQEUDQFKWRDQH[LVWLQJVWDWHPHQWWKH
GHYHORSHUZRXOGILUVWKDYHWRZULWHDIDLOLQJWHVWFDVHWKDWPRWLYDWHVWKHEUDQFK$VDUHVXOWWKHDXWRPDWHG
WHVWVUHVXOWLQJIURP7''WHQGWREHYHU\WKRURXJKWKH\GHWHFWDQ\XQH[SHFWHGFKDQJHVLQWKHFRGH
V
EHKDYLRXU7KLVGHWHFWVSUREOHPVWKDWFDQDULVHZKHUHDFKDQJHODWHULQWKHGHYHORSPHQWF\FOHXQH[SHFWHGO\
DOWHUVRWKHUIXQFWLRQDOLW\
0DGH\VNL>@SURYLGHGDQHPSLULFDOHYLGHQFH YLDDVHULHVRIODERUDWRU\H[SHULPHQWVZLWKRYHU
GHYHORSHUV UHJDUGLQJWKHVXSHULRULW\RIWKH7''SUDFWLFHRYHUWKHFODVVLF7HVW/DVWDSSURDFKZLWKUHVSHFW
WRWKHORZHUFRXSOLQJEHWZHHQREMHFWV &%2 7KHPHDQHIIHFWVL]HUHSUHVHQWVDPHGLXP EXWFORVHWRODUJH
HIIHFWRQWKHEDVLVRIPHWDDQDO\VLVRIWKHSHUIRUPHGH[SHULPHQWVZKLFKLVDVXEVWDQWLDOILQGLQJ,WVXJJHVWV
DEHWWHUPRGXODUL]DWLRQ LHDPRUHPRGXODUGHVLJQ HDVLHUUHXVHDQGWHVWLQJRIWKHGHYHORSHGVRIWZDUH
SURGXFWVGXHWRWKH7''SURJUDPPLQJSUDFWLFH>@0DGH\VNLDOVRPHDVXUHGWKHHIIHFWRIWKH7''SUDFWLFH
KWWSVHQZLNLSHGLDRUJZLNL7HVWGULYHQBGHYHORSPHQW





7HVWGULYHQGHYHORSPHQW:LNLSHGLDWKHIUHHHQF\FORSHGLD

RQXQLWWHVWVXVLQJEUDQFKFRYHUDJH %& DQGPXWDWLRQVFRUHLQGLFDWRU 06, >@>@>@ZKLFKDUHLQGLFDWRUV


RIWKHWKRURXJKQHVVDQGWKHIDXOWGHWHFWLRQHIIHFWLYHQHVVRIXQLWWHVWVUHVSHFWLYHO\7KHHIIHFWVL]HRI7''
RQEUDQFKFRYHUDJHZDVPHGLXPLQVL]HDQGWKHUHIRUHLVFRQVLGHUHGVXEVWDQWLYHHIIHFW>@

/LPLWDWLRQV
7HVWGULYHQGHYHORSPHQWGRHVQRWSHUIRUPVXIILFLHQWWHVWLQJLQVLWXDWLRQVZKHUHIXOOIXQFWLRQDOWHVWVDUH
UHTXLUHGWRGHWHUPLQHVXFFHVVRUIDLOXUHGXHWRH[WHQVLYHXVHRIXQLWWHVWV>@([DPSOHVRIWKHVHDUHXVHU
LQWHUIDFHVSURJUDPVWKDWZRUNZLWKGDWDEDVHVDQGVRPHWKDWGHSHQGRQVSHFLILFQHWZRUNFRQILJXUDWLRQV
7''HQFRXUDJHVGHYHORSHUVWRSXWWKHPLQLPXPDPRXQWRIFRGHLQWRVXFKPRGXOHVDQGWRPD[LPL]HWKH
ORJLFWKDWLVLQWHVWDEOHOLEUDU\FRGHXVLQJIDNHVDQGPRFNVWRUHSUHVHQWWKHRXWVLGHZRUOG>@
0DQDJHPHQWVXSSRUWLVHVVHQWLDO:LWKRXWWKHHQWLUHRUJDQL]DWLRQEHOLHYLQJWKDWWHVWGULYHQGHYHORSPHQWLV
JRLQJWRLPSURYHWKHSURGXFWPDQDJHPHQWPD\IHHOWKDWWLPHVSHQWZULWLQJWHVWVLVZDVWHG>@
8QLWWHVWVFUHDWHGLQDWHVWGULYHQGHYHORSPHQWHQYLURQPHQWDUHW\SLFDOO\FUHDWHGE\WKHGHYHORSHUZKRLV
ZULWLQJWKHFRGHEHLQJWHVWHG7KHUHIRUHWKHWHVWVPD\VKDUHEOLQGVSRWVZLWKWKHFRGHLIIRUH[DPSOHD
GHYHORSHUGRHVQRWUHDOL]HWKDWFHUWDLQLQSXWSDUDPHWHUVPXVWEHFKHFNHGPRVWOLNHO\QHLWKHUWKHWHVWQRUWKH
FRGHZLOOYHULI\WKRVHSDUDPHWHUV$QRWKHUH[DPSOHLIWKHGHYHORSHUPLVLQWHUSUHWVWKHUHTXLUHPHQWVIRUWKH
PRGXOHKHLVGHYHORSLQJWKHFRGHDQGWKHXQLWWHVWVKHZULWHVZLOOERWKEHZURQJLQWKHVDPHZD\
7KHUHIRUHWKHWHVWVZLOOSDVVJLYLQJDIDOVHVHQVHRIFRUUHFWQHVV
$KLJKQXPEHURISDVVLQJXQLWWHVWVPD\EULQJDIDOVHVHQVHRIVHFXULW\UHVXOWLQJLQIHZHUDGGLWLRQDO
VRIWZDUHWHVWLQJDFWLYLWLHVVXFKDVLQWHJUDWLRQWHVWLQJDQGFRPSOLDQFHWHVWLQJ
7HVWVEHFRPHSDUWRIWKHPDLQWHQDQFHRYHUKHDGRIDSURMHFW%DGO\ZULWWHQWHVWVIRUH[DPSOHRQHVWKDW
LQFOXGHKDUGFRGHGHUURUVWULQJVRUDUHWKHPVHOYHVSURQHWRIDLOXUHDUHH[SHQVLYHWRPDLQWDLQ7KLVLV
HVSHFLDOO\WKHFDVHZLWKIUDJLOHWHVWV>@7KHUHLVDULVNWKDWWHVWVWKDWUHJXODUO\JHQHUDWHIDOVHIDLOXUHVZLOOEH
LJQRUHGVRWKDWZKHQDUHDOIDLOXUHRFFXUVLWPD\QRWEHGHWHFWHG,WLVSRVVLEOHWRZULWHWHVWVIRUORZDQG
HDV\PDLQWHQDQFHIRUH[DPSOHE\WKHUHXVHRIHUURUVWULQJVDQGWKLVVKRXOGEHDJRDOGXULQJWKHFRGH
UHIDFWRULQJSKDVHGHVFULEHGDERYH
:ULWLQJDQGPDLQWDLQLQJDQH[FHVVLYHQXPEHURIWHVWVFRVWVWLPH$OVRPRUHIOH[LEOHPRGXOHV ZLWKOLPLWHG
WHVWV PLJKWDFFHSWQHZUHTXLUHPHQWVZLWKRXWWKHQHHGIRUFKDQJLQJWKHWHVWV)RUWKRVHUHDVRQVWHVWLQJIRU
RQO\H[WUHPHFRQGLWLRQVRUDVPDOOVDPSOHRIGDWDFDQEHHDVLHUWRDGMXVWWKDQDVHWRIKLJKO\GHWDLOHGWHVWV
+RZHYHUGHYHORSHUVFRXOGEHZDUQHGDERXWRYHUWHVWLQJWRDYRLGWKHH[FHVVLYHZRUNEXWLWPLJKWUHTXLUH
DGYDQFHGVNLOOVLQVDPSOLQJRUIDFWRUDQDO\VLV
7KHOHYHORIFRYHUDJHDQGWHVWLQJGHWDLODFKLHYHGGXULQJUHSHDWHG7''F\FOHVFDQQRWHDVLO\EHUHFUHDWHGDW
DODWHUGDWH7KHUHIRUHWKHVHRULJLQDORUHDUO\WHVWVEHFRPHLQFUHDVLQJO\SUHFLRXVDVWLPHJRHVE\7KH
WDFWLFLVWRIL[LWHDUO\$OVRLIDSRRUDUFKLWHFWXUHDSRRUGHVLJQRUDSRRUWHVWLQJVWUDWHJ\OHDGVWRDODWH
FKDQJHWKDWPDNHVGR]HQVRIH[LVWLQJWHVWVIDLOWKHQLWLVLPSRUWDQWWKDWWKH\DUHLQGLYLGXDOO\IL[HG0HUHO\
GHOHWLQJGLVDEOLQJRUUDVKO\DOWHULQJWKHPFDQOHDGWRXQGHWHFWDEOHKROHVLQWKHWHVWFRYHUDJH

7HVWGULYHQZRUN
KWWSVHQZLNLSHGLDRUJZLNL7HVWGULYHQBGHYHORSPHQW





7HVWGULYHQGHYHORSPHQW:LNLSHGLDWKHIUHHHQF\FORSHGLD

7HVWGULYHQGHYHORSPHQWKDVEHHQDGRSWHGRXWVLGHRIVRIWZDUHGHYHORSPHQWLQERWKSURGXFWDQGVHUYLFH
WHDPVDVWHVWGULYHQZRUN>@6LPLODUWR7''QRQVRIWZDUHWHDPVGHYHORSTXDOLW\FRQWUROFKHFNV XVXDOO\
PDQXDOWHVWVUDWKHUWKDQDXWRPDWHGWHVWV IRUHDFKDVSHFWRIWKHZRUNSULRUWRFRPPHQFLQJ7KHVH4&
FKHFNVDUHWKHQXVHGWRLQIRUPWKHGHVLJQDQGYDOLGDWHWKHDVVRFLDWHGRXWFRPHV7KHVL[VWHSVRIWKH7''
VHTXHQFHDUHDSSOLHGZLWKPLQRUVHPDQWLFFKDQJHV
$GGDFKHFNUHSODFHV$GGDWHVW
5XQDOOFKHFNVUHSODFHV5XQDOOWHVWV
'RWKHZRUNUHSODFHV:ULWHVRPHFRGH
5XQDOOFKHFNVUHSODFHV5XQWHVWV
&OHDQXSWKHZRUNUHSODFHV5HIDFWRUFRGH
5HSHDW

7''DQG$7''
7HVW'ULYHQ'HYHORSPHQWLVUHODWHGWREXWGLIIHUHQWIURPDFFHSWDQFHWHVWGULYHQGHYHORSPHQW $7'' >@
7''LVSULPDULO\DGHYHORSHUVWRROWRKHOSFUHDWHZHOOZULWWHQXQLWRIFRGH IXQFWLRQFODVVRUPRGXOH WKDW
FRUUHFWO\SHUIRUPVDVHWRIRSHUDWLRQV$7''LVDFRPPXQLFDWLRQWRROEHWZHHQWKHFXVWRPHUGHYHORSHU
DQGWHVWHUWRHQVXUHWKDWWKHUHTXLUHPHQWVDUHZHOOGHILQHG7''UHTXLUHVWHVWDXWRPDWLRQ$7''GRHVQRW
DOWKRXJKDXWRPDWLRQKHOSVZLWKUHJUHVVLRQWHVWLQJ7HVWVXVHGLQ7''FDQRIWHQEHGHULYHGIURP$7''
WHVWVVLQFHWKHFRGHXQLWVLPSOHPHQWVRPHSRUWLRQRIDUHTXLUHPHQW$7''WHVWVVKRXOGEHUHDGDEOHE\WKH
FXVWRPHU7''WHVWVGRQRWQHHGWREH

7''DQG%''
%'' %HKDYLRUGULYHQGHYHORSPHQW FRPELQHVSUDFWLFHVIURP7''DQGIURP$7''>@,WLQFOXGHVWKH
SUDFWLFHRIZULWLQJWHVWVILUVWEXWIRFXVHVRQWHVWVZKLFKGHVFULEHEHKDYLRUUDWKHUWKDQWHVWVZKLFKWHVWDXQLW
RILPSOHPHQWDWLRQ7RROVVXFKDV0VSHFDQG6SHFIORZSURYLGHDV\QWD[ZKLFKDOORZQRQSURJUDPPHUVWR
GHILQHWKHEHKDYLRUVZKLFKGHYHORSHUVFDQWKHQWUDQVODWHLQWRDXWRPDWHGWHVWV

&RGHYLVLELOLW\
7HVWVXLWHFRGHFOHDUO\KDVWREHDEOHWRDFFHVVWKHFRGHLWLVWHVWLQJ2QWKHRWKHUKDQGQRUPDOGHVLJQ
FULWHULDVXFKDVLQIRUPDWLRQKLGLQJHQFDSVXODWLRQDQGWKHVHSDUDWLRQRIFRQFHUQVVKRXOGQRWEH
FRPSURPLVHG7KHUHIRUHXQLWWHVWFRGHIRU7''LVXVXDOO\ZULWWHQZLWKLQWKHVDPHSURMHFWRUPRGXOHDVWKH
FRGHEHLQJWHVWHG
,QREMHFWRULHQWHGGHVLJQWKLVVWLOOGRHVQRWSURYLGHDFFHVVWRSULYDWHGDWDDQGPHWKRGV7KHUHIRUHH[WUD
ZRUNPD\EHQHFHVVDU\IRUXQLWWHVWV,Q-DYDDQGRWKHUODQJXDJHVDGHYHORSHUFDQXVHUHIOHFWLRQWRDFFHVV
SULYDWHILHOGVDQGPHWKRGV>@$OWHUQDWLYHO\DQLQQHUFODVVFDQEHXVHGWRKROGWKHXQLWWHVWVVRWKH\KDYH
YLVLELOLW\RIWKHHQFORVLQJFODVV
VPHPEHUVDQGDWWULEXWHV,QWKH1(7)UDPHZRUNDQGVRPHRWKHU
SURJUDPPLQJODQJXDJHVSDUWLDOFODVVHVPD\EHXVHGWRH[SRVHSULYDWHPHWKRGVDQGGDWDIRUWKHWHVWVWR
DFFHVV

KWWSVHQZLNLSHGLDRUJZLNL7HVWGULYHQBGHYHORSPHQW





7HVWGULYHQGHYHORSPHQW:LNLSHGLDWKHIUHHHQF\FORSHGLD

,WLVLPSRUWDQWWKDWVXFKWHVWLQJKDFNVGRQRWUHPDLQLQWKHSURGXFWLRQFRGH,Q&DQGRWKHUODQJXDJHV
FRPSLOHUGLUHFWLYHVVXFKDV
FDQEHSODFHGDURXQGVXFKDGGLWLRQDOFODVVHVDQGLQGHHG
DOORWKHUWHVWUHODWHGFRGHWRSUHYHQWWKHPEHLQJFRPSLOHGLQWRWKHUHOHDVHGFRGH7KLVPHDQVWKHUHOHDVHG
FRGHLVQRWH[DFWO\WKHVDPHDVZKDWZDVXQLWWHVWHG7KHUHJXODUUXQQLQJRIIHZHUEXWPRUHFRPSUHKHQVLYH
HQGWRHQGLQWHJUDWLRQWHVWVRQWKHILQDOUHOHDVHEXLOGFDQHQVXUH DPRQJRWKHUWKLQJV WKDWQRSURGXFWLRQ
FRGHH[LVWVWKDWVXEWO\UHOLHVRQDVSHFWVRIWKHWHVWKDUQHVV
7KHUHLVVRPHGHEDWHDPRQJSUDFWLWLRQHUVRI7''GRFXPHQWHGLQWKHLUEORJVDQGRWKHUZULWLQJVDVWR
ZKHWKHULWLVZLVHWRWHVWSULYDWHPHWKRGVDQGGDWDDQ\ZD\6RPHDUJXHWKDWSULYDWHPHPEHUVDUHDPHUH
LPSOHPHQWDWLRQGHWDLOWKDWPD\FKDQJHDQGVKRXOGEHDOORZHGWRGRVRZLWKRXWEUHDNLQJQXPEHUVRIWHVWV
7KXVLWVKRXOGEHVXIILFLHQWWRWHVWDQ\FODVVWKURXJKLWVSXEOLFLQWHUIDFHRUWKURXJKLWVVXEFODVVLQWHUIDFH
ZKLFKVRPHODQJXDJHVFDOOWKHSURWHFWHGLQWHUIDFH>@2WKHUVVD\WKDWFUXFLDODVSHFWVRIIXQFWLRQDOLW\PD\
EHLPSOHPHQWHGLQSULYDWHPHWKRGVDQGWHVWLQJWKHPGLUHFWO\RIIHUVDGYDQWDJHRIVPDOOHUDQGPRUHGLUHFW
XQLWWHVWV>@>@

6RIWZDUHIRU7''
7KHUHDUHPDQ\WHVWLQJIUDPHZRUNVDQGWRROVWKDWDUHXVHIXOLQ7''

[8QLWIUDPHZRUNV
'HYHORSHUVPD\XVHFRPSXWHUDVVLVWHGWHVWLQJIUDPHZRUNVVXFKDV[8QLWFUHDWHGLQWRFUHDWHDQG
DXWRPDWLFDOO\UXQWKHWHVWFDVHV;XQLWIUDPHZRUNVSURYLGHDVVHUWLRQVW\OHWHVWYDOLGDWLRQFDSDELOLWLHVDQG
UHVXOWUHSRUWLQJ7KHVHFDSDELOLWLHVDUHFULWLFDOIRUDXWRPDWLRQDVWKH\PRYHWKHEXUGHQRIH[HFXWLRQ
YDOLGDWLRQIURPDQLQGHSHQGHQWSRVWSURFHVVLQJDFWLYLW\WRRQHWKDWLVLQFOXGHGLQWKHWHVWH[HFXWLRQ7KH
H[HFXWLRQIUDPHZRUNSURYLGHGE\WKHVHWHVWIUDPHZRUNVDOORZVIRUWKHDXWRPDWLFH[HFXWLRQRIDOOV\VWHP
WHVWFDVHVRUYDULRXVVXEVHWVDORQJZLWKRWKHUIHDWXUHV>@

7$3UHVXOWV
7HVWLQJIUDPHZRUNVPD\DFFHSWXQLWWHVWRXWSXWLQWKHODQJXDJHDJQRVWLF7HVW$Q\WKLQJ3URWRFROFUHDWHGLQ


)DNHVPRFNVDQGLQWHJUDWLRQWHVWV
8QLWWHVWVDUHVRQDPHGEHFDXVHWKH\HDFKWHVWRQHXQLWRIFRGH$FRPSOH[PRGXOHPD\KDYHDWKRXVDQG
XQLWWHVWVDQGDVLPSOHPRGXOHPD\KDYHRQO\WHQ7KHWHVWVXVHGIRU7''VKRXOGQHYHUFURVVSURFHVV
ERXQGDULHVLQDSURJUDPOHWDORQHQHWZRUNFRQQHFWLRQV'RLQJVRLQWURGXFHVGHOD\VWKDWPDNHWHVWVUXQ
VORZO\DQGGLVFRXUDJHGHYHORSHUVIURPUXQQLQJWKHZKROHVXLWH,QWURGXFLQJGHSHQGHQFLHVRQH[WHUQDO
PRGXOHVRUGDWDDOVRWXUQVXQLWWHVWVLQWRLQWHJUDWLRQWHVWV,IRQHPRGXOHPLVEHKDYHVLQDFKDLQRI
LQWHUUHODWHGPRGXOHVLWLVQRWVRLPPHGLDWHO\FOHDUZKHUHWRORRNIRUWKHFDXVHRIWKHIDLOXUH
:KHQFRGHXQGHUGHYHORSPHQWUHOLHVRQDGDWDEDVHDZHEVHUYLFHRUDQ\RWKHUH[WHUQDOSURFHVVRUVHUYLFH
HQIRUFLQJDXQLWWHVWDEOHVHSDUDWLRQLVDOVRDQRSSRUWXQLW\DQGDGULYLQJIRUFHWRGHVLJQPRUHPRGXODUPRUH
WHVWDEOHDQGPRUHUHXVDEOHFRGH>@7ZRVWHSVDUHQHFHVVDU\
KWWSVHQZLNLSHGLDRUJZLNL7HVWGULYHQBGHYHORSPHQW





7HVWGULYHQGHYHORSPHQW:LNLSHGLDWKHIUHHHQF\FORSHGLD

:KHQHYHUH[WHUQDODFFHVVLVQHHGHGLQWKHILQDOGHVLJQDQLQWHUIDFHVKRXOGEHGHILQHGWKDWGHVFULEHV
WKHDFFHVVDYDLODEOH6HHWKHGHSHQGHQF\LQYHUVLRQSULQFLSOHIRUDGLVFXVVLRQRIWKHEHQHILWVRIGRLQJ
WKLVUHJDUGOHVVRI7''
7KHLQWHUIDFHVKRXOGEHLPSOHPHQWHGLQWZRZD\VRQHRIZKLFKUHDOO\DFFHVVHVWKHH[WHUQDOSURFHVV
DQGWKHRWKHURIZKLFKLVDIDNHRUPRFN)DNHREMHFWVQHHGGROLWWOHPRUHWKDQDGGDPHVVDJHVXFKDV
3HUVRQREMHFWVDYHGWRDWUDFHORJDJDLQVWZKLFKDWHVWDVVHUWLRQFDQEHUXQWRYHULI\FRUUHFW
EHKDYLRXU0RFNREMHFWVGLIIHULQWKDWWKH\WKHPVHOYHVFRQWDLQWHVWDVVHUWLRQVWKDWFDQPDNHWKHWHVW
IDLOIRUH[DPSOHLIWKHSHUVRQ
VQDPHDQGRWKHUGDWDDUHQRWDVH[SHFWHG
)DNHDQGPRFNREMHFWPHWKRGVWKDWUHWXUQGDWDRVWHQVLEO\IURPDGDWDVWRUHRUXVHUFDQKHOSWKHWHVW
SURFHVVE\DOZD\VUHWXUQLQJWKHVDPHUHDOLVWLFGDWDWKDWWHVWVFDQUHO\XSRQ7KH\FDQDOVREHVHWLQWR
SUHGHILQHGIDXOWPRGHVVRWKDWHUURUKDQGOLQJURXWLQHVFDQEHGHYHORSHGDQGUHOLDEO\WHVWHG,QDIDXOWPRGH
DPHWKRGPD\UHWXUQDQLQYDOLGLQFRPSOHWHRUQXOOUHVSRQVHRUPD\WKURZDQH[FHSWLRQ)DNHVHUYLFHV
RWKHUWKDQGDWDVWRUHVPD\DOVREHXVHIXOLQ7''$IDNHHQFU\SWLRQVHUYLFHPD\QRWLQIDFWHQFU\SWWKH
GDWDSDVVHGDIDNHUDQGRPQXPEHUVHUYLFHPD\DOZD\VUHWXUQ)DNHRUPRFNLPSOHPHQWDWLRQVDUH
H[DPSOHVRIGHSHQGHQF\LQMHFWLRQ
$7HVW'RXEOHLVDWHVWVSHFLILFFDSDELOLW\WKDWVXEVWLWXWHVIRUDV\VWHPFDSDELOLW\W\SLFDOO\DFODVVRU
IXQFWLRQWKDWWKH887GHSHQGVRQ7KHUHDUHWZRWLPHVDWZKLFKWHVWGRXEOHVFDQEHLQWURGXFHGLQWRD
V\VWHPOLQNDQGH[HFXWLRQ/LQNWLPHVXEVWLWXWLRQLVZKHQWKHWHVWGRXEOHLVFRPSLOHGLQWRWKHORDGPRGXOH
ZKLFKLVH[HFXWHGWRYDOLGDWHWHVWLQJ7KLVDSSURDFKLVW\SLFDOO\XVHGZKHQUXQQLQJLQDQHQYLURQPHQWRWKHU
WKDQWKHWDUJHWHQYLURQPHQWWKDWUHTXLUHVGRXEOHVIRUWKHKDUGZDUHOHYHOFRGHIRUFRPSLODWLRQ7KH
DOWHUQDWLYHWROLQNHUVXEVWLWXWLRQLVUXQWLPHVXEVWLWXWLRQLQZKLFKWKHUHDOIXQFWLRQDOLW\LVUHSODFHGGXULQJ
WKHH[HFXWLRQRIDWHVWFDVHV7KLVVXEVWLWXWLRQLVW\SLFDOO\GRQHWKURXJKWKHUHDVVLJQPHQWRINQRZQIXQFWLRQ
SRLQWHUVRUREMHFWUHSODFHPHQW
7HVWGRXEOHVDUHRIDQXPEHURIGLIIHUHQWW\SHVDQGYDU\LQJFRPSOH[LWLHV
'XPP\$GXPP\LVWKHVLPSOHVWIRUPRIDWHVWGRXEOH,WIDFLOLWDWHVOLQNHUWLPHVXEVWLWXWLRQE\
SURYLGLQJDGHIDXOWUHWXUQYDOXHZKHUHUHTXLUHG
6WXE$VWXEDGGVVLPSOLVWLFORJLFWRDGXPP\SURYLGLQJGLIIHUHQWRXWSXWV
6S\$VS\FDSWXUHVDQGPDNHVDYDLODEOHSDUDPHWHUDQGVWDWHLQIRUPDWLRQSXEOLVKLQJDFFHVVRUVWR
WHVWFRGHIRUSULYDWHLQIRUPDWLRQDOORZLQJIRUPRUHDGYDQFHGVWDWHYDOLGDWLRQ
0RFN$PRFNLVVSHFLILHGE\DQLQGLYLGXDOWHVWFDVHWRYDOLGDWHWHVWVSHFLILFEHKDYLRUFKHFNLQJ
SDUDPHWHUYDOXHVDQGFDOOVHTXHQFLQJ
6LPXODWRU$VLPXODWRULVDFRPSUHKHQVLYHFRPSRQHQWSURYLGLQJDKLJKHUILGHOLW\DSSUR[LPDWLRQRI
WKHWDUJHWFDSDELOLW\ WKHWKLQJEHLQJGRXEOHG $VLPXODWRUW\SLFDOO\UHTXLUHVVLJQLILFDQWDGGLWLRQDO
GHYHORSPHQWHIIRUW>@
$FRUROODU\RIVXFKGHSHQGHQF\LQMHFWLRQLVWKDWWKHDFWXDOGDWDEDVHRURWKHUH[WHUQDODFFHVVFRGHLVQHYHU
WHVWHGE\WKH7''SURFHVVLWVHOI7RDYRLGHUURUVWKDWPD\DULVHIURPWKLVRWKHUWHVWVDUHQHHGHGWKDW
LQVWDQWLDWHWKHWHVWGULYHQFRGHZLWKWKHUHDOLPSOHPHQWDWLRQVRIWKHLQWHUIDFHVGLVFXVVHGDERYH7KHVHDUH
LQWHJUDWLRQWHVWVDQGDUHTXLWHVHSDUDWHIURPWKH7''XQLWWHVWV7KHUHDUHIHZHURIWKHPDQGWKH\PXVWEH
UXQOHVVRIWHQWKDQWKHXQLWWHVWV7KH\FDQQRQHWKHOHVVEHLPSOHPHQWHGXVLQJWKHVDPHWHVWLQJIUDPHZRUN
VXFKDV[8QLW
,QWHJUDWLRQWHVWVWKDWDOWHUDQ\SHUVLVWHQWVWRUHRUGDWDEDVHVKRXOGDOZD\VEHGHVLJQHGFDUHIXOO\ZLWK
FRQVLGHUDWLRQRIWKHLQLWLDODQGILQDOVWDWHRIWKHILOHVRUGDWDEDVHHYHQLIDQ\WHVWIDLOV7KLVLVRIWHQ
DFKLHYHGXVLQJVRPHFRPELQDWLRQRIWKHIROORZLQJWHFKQLTXHV
KWWSVHQZLNLSHGLDRUJZLNL7HVWGULYHQBGHYHORSPHQW





7HVWGULYHQGHYHORSPHQW:LNLSHGLDWKHIUHHHQF\FORSHGLD

7KHPHWKRGZKLFKLVLQWHJUDOWRPDQ\WHVWIUDPHZRUNV
H[FHSWLRQKDQGOLQJVWUXFWXUHVZKHUHDYDLODEOH
'DWDEDVHWUDQVDFWLRQVZKHUHDWUDQVDFWLRQDWRPLFDOO\LQFOXGHVSHUKDSVDZULWHDUHDGDQGDPDWFKLQJ
GHOHWHRSHUDWLRQ
7DNLQJDVQDSVKRWRIWKHGDWDEDVHEHIRUHUXQQLQJDQ\WHVWVDQGUROOLQJEDFNWRWKHVQDSVKRWDIWHU
HDFKWHVWUXQ7KLVPD\EHDXWRPDWHGXVLQJDIUDPHZRUNVXFKDV$QWRU1$QWRUDFRQWLQXRXV
LQWHJUDWLRQV\VWHPVXFKDV&UXLVH&RQWURO
,QLWLDOLVLQJWKHGDWDEDVHWRDFOHDQVWDWHEHIRUHWHVWVUDWKHUWKDQFOHDQLQJXSDIWHUWKHP7KLVPD\EH
UHOHYDQWZKHUHFOHDQLQJXSPD\PDNHLWGLIILFXOWWRGLDJQRVHWHVWIDLOXUHVE\GHOHWLQJWKHILQDOVWDWHRI
WKHGDWDEDVHEHIRUHGHWDLOHGGLDJQRVLVFDQEHSHUIRUPHG

7''IRUFRPSOH[V\VWHPV
([HUFLVLQJ7''RQODUJHFKDOOHQJLQJV\VWHPVUHTXLUHVDPRGXODUDUFKLWHFWXUHZHOOGHILQHGFRPSRQHQWV
ZLWKSXEOLVKHGLQWHUIDFHVDQGGLVFLSOLQHGV\VWHPOD\HULQJZLWKPD[LPL]DWLRQRISODWIRUPLQGHSHQGHQFH
7KHVHSURYHQSUDFWLFHV\LHOGLQFUHDVHGWHVWDELOLW\DQGIDFLOLWDWHWKHDSSOLFDWLRQRIEXLOGDQGWHVW
DXWRPDWLRQ>@

'HVLJQLQJIRUWHVWDELOLW\
&RPSOH[V\VWHPVUHTXLUHDQDUFKLWHFWXUHWKDWPHHWVDUDQJHRIUHTXLUHPHQWV$NH\VXEVHWRIWKHVH
UHTXLUHPHQWVLQFOXGHVVXSSRUWIRUWKHFRPSOHWHDQGHIIHFWLYHWHVWLQJRIWKHV\VWHP(IIHFWLYHPRGXODU
GHVLJQ\LHOGVFRPSRQHQWVWKDWVKDUHWUDLWVHVVHQWLDOIRUHIIHFWLYH7''
+LJK&RKHVLRQHQVXUHVHDFKXQLWSURYLGHVDVHWRIUHODWHGFDSDELOLWLHVDQGPDNHVWKHWHVWVRIWKRVH
FDSDELOLWLHVHDVLHUWRPDLQWDLQ
/RZ&RXSOLQJDOORZVHDFKXQLWWREHHIIHFWLYHO\WHVWHGLQLVRODWLRQ
3XEOLVKHG,QWHUIDFHVUHVWULFW&RPSRQHQWDFFHVVDQGVHUYHDVFRQWDFWSRLQWVIRUWHVWVIDFLOLWDWLQJWHVW
FUHDWLRQDQGHQVXULQJWKHKLJKHVWILGHOLW\EHWZHHQWHVWDQGSURGXFWLRQXQLWFRQILJXUDWLRQ
$NH\WHFKQLTXHIRUEXLOGLQJHIIHFWLYHPRGXODUDUFKLWHFWXUHLV6FHQDULR0RGHOLQJZKHUHDVHWRIVHTXHQFH
FKDUWVLVFRQVWUXFWHGHDFKRQHIRFXVLQJRQDVLQJOHV\VWHPOHYHOH[HFXWLRQVFHQDULR7KH6FHQDULR0RGHO
SURYLGHVDQH[FHOOHQWYHKLFOHIRUFUHDWLQJWKHVWUDWHJ\RILQWHUDFWLRQVEHWZHHQFRPSRQHQWVLQUHVSRQVHWRD
VSHFLILFVWLPXOXV(DFKRIWKHVH6FHQDULR0RGHOVVHUYHVDVDULFKVHWRIUHTXLUHPHQWVIRUWKHVHUYLFHVRU
IXQFWLRQVWKDWDFRPSRQHQWPXVWSURYLGHDQGLWDOVRGLFWDWHVWKHRUGHUWKDWWKHVHFRPSRQHQWVDQGVHUYLFHV
LQWHUDFWWRJHWKHU6FHQDULRPRGHOLQJFDQJUHDWO\IDFLOLWDWHWKHFRQVWUXFWLRQRI7''WHVWVIRUDFRPSOH[
V\VWHP>@

0DQDJLQJWHVWVIRUODUJHWHDPV
,QDODUJHUV\VWHPWKHLPSDFWRISRRUFRPSRQHQWTXDOLW\LVPDJQLILHGE\WKHFRPSOH[LW\RILQWHUDFWLRQV
7KLVPDJQLILFDWLRQPDNHVWKHEHQHILWVRI7''DFFUXHHYHQIDVWHULQWKHFRQWH[WRIODUJHUSURMHFWV
+RZHYHUWKHFRPSOH[LW\RIWKHWRWDOSRSXODWLRQRIWHVWVFDQEHFRPHDSUREOHPLQLWVHOIHURGLQJSRWHQWLDO
JDLQV,WVRXQGVVLPSOHEXWDNH\LQLWLDOVWHSLVWRUHFRJQL]HWKDWWHVWFRGHLVDOVRLPSRUWDQWVRIWZDUHDQG
VKRXOGEHSURGXFHGDQGPDLQWDLQHGZLWKWKHVDPHULJRUDVWKHSURGXFWLRQFRGH
&UHDWLQJDQGPDQDJLQJWKHDUFKLWHFWXUHRIWHVWVRIWZDUHZLWKLQDFRPSOH[V\VWHPLVMXVWDVLPSRUWDQWDVWKH
FRUHSURGXFWDUFKLWHFWXUH7HVWGULYHUVLQWHUDFWZLWKWKH887WHVWGRXEOHVDQGWKHXQLWWHVWIUDPHZRUN>@
KWWSVHQZLNLSHGLDRUJZLNL7HVWGULYHQBGHYHORSPHQW





7HVWGULYHQGHYHORSPHQW:LNLSHGLDWKHIUHHHQF\FORSHGLD

5HWULHYHGIURPKWWSVHQZLNLSHGLDRUJZLQGH[SKS"WLWOH 7HVWGULYHQBGHYHORSPHQW ROGLG 


&DWHJRULHV ([WUHPHSURJUDPPLQJ 6RIWZDUHGHYHORSPHQWSKLORVRSKLHV 6RIWZDUHGHYHORSPHQWSURFHVV
6RIWZDUHWHVWLQJ

KWWSVHQZLNLSHGLDRUJZLNL7HVWGULYHQBGHYHORSPHQW