6RIWZDUH(QJLQHHULQJ,VVXHVLQ%LJ'DWD$SSOLFDWLRQ
'HYHORSPHQW
=L\D.DUDND\D
'HSDUWPHQWRI&RPSXWHU(QJLQHHULQJ
$WLOLP8QLYHUVLW\
$QNDUD7XUNH\
]L\DNDUDND\D#DWLOLPHGXWU
$EVWUDFW²%LJ 'DWD KDV EHFRPH RQH RI WKH PRVW LPSRUWDQW
FRQFHSWVWKDWLVEHLQJVWXGLHGLQ&RPSXWHU6RIWZDUH(QJLQHHULQJ
7KH GDWD SURGXFHG LQ UHFHQW \HDUV KDYH LQFUHDVHG UDSLGO\ DQG
H[SRQHQWLDOO\ QHFHVVLWDWLQJ WKH VROXWLRQ RI PDMRU SUREOHPV VXFK
DVWKHFROOHFWLRQSURFHVVLQJDQGVWRUDJHRIKXJHYROXPHRIGDWD
%LJ 'DWD )UDPHZRUNV DUH GHYHORSHG VSHFLILFDOO\ WR VROYH WKHVH
SUREOHPV WKDW IDFLOLWDWHV DSSOLFDWLRQ GHYHORSHUV E\ SURYLGLQJ
RSSRUWXQLWLHV WR FROOHFW SURFHVV PDQDJH PRQLWRU DQG DQDO\]H
WKHVH GDWD $ IHZ H[DPSOHV RI WKHVH IUDPHZRUNV DUH +DGRRS
6SDUN 6WRUP DQG )OLQN ZKLFK DUH GHYHORSHG E\ 6RIWZDUH
(QJLQHHUVDVRSHQVRXUFHSURMHFWV$OWKRXJKWKHFKDOOHQJHVUDLVHG
IURP FRRUGLQDWLRQ RI ,7 UHVRXUFHV VXFK DV KXJH DPRXQWV RI
FRPSXWDWLRQ SRZHU VWRUDJH DUHD PHPRU\ DQG QHWZRUN
EDQGZLGWK LQ D GLVWULEXWHG PDQQHU VROYHG E\ WKHVH IUDPHZRUNV
WKHUH VWLOO UHPDLQV PDQ\ 6RIWZDUH (QJLQHHULQJ SUREOHPV LQ
DSSOLFDWLRQ GHYHORSPHQW SKDVH HYHQ LI WKH\ EDVHG RQ WKHVH
IUDPHZRUNV +LJK VFDODELOLW\ IDXOW WROHUDQFH IOH[LELOLW\
UHOLDELOLW\DQGWHVWDELOLW\FDQEHOLVWHGDVWKHPDLQLVVXHVQHHGWR
EHFDUHIXOO\FRQVLGHUHGLQWHUPVRI6RIWZDUH(QJLQHHULQJ,QWKLV
SDSHU ZH ILUVW FODULI\ WKH WHUPV )UDPHZRUN$SSOLFDWLRQ DQG
WKHQ WKH RYHUYLHZ LQIRUPDWLRQ DERXW %LJ 'DWD DQG UHODWHG
IUDPHZRUNVDUHJLYHQEHIRUHHPSKDVL]LQJWKHSUREOHPVDULVLQJLQ
WHUPVRI6RIWZDUH(QJLQHHULQJ1HYHUWKHOHVVZHWULHGWRSURYLGH
JXLGDQFHWRWKHSHRSOHZKRZRXOGGHYHORSVRIWZDUHIRU%LJ'DWD
DQGWULHGWRJLYHWKHIXUWKHUUHVHDUFKJXLGDQFH
.H\ZRUGV²%LJ 'DWD &KDUDFWHULVWLFV 6RIWZDUH (QJLQHHULQJ
)UDPHZRUNV3UREOHPV'DWD6FLHQFH'DWD(QJLQHHULQJ
, ,1752'8&7,21
%LJ GDWD LV DERXW DQ HQRUPRXV DPRXQW RI GDWD VHWV WKDW
FDQQRW EH SURFHVVHG E\ XVLQJ FRQYHQWLRQDO GDWDEDVHV DQG
FRQYHQWLRQDOWRROV>@7KHUHDUHPDQ\XVHFDVHVRI%LJ'DWD
WKDWDUHSURYHGKRZZHOOLWFDQEHXVHGWRJDLQRSSRUWXQLW\WR
GRPLQDWHWKHLU FRPSHWLWRUV>@ $PRQJ WKH RWKHUV KHDOWKFDUH
WHOHFRPPXQLFDWLRQ HFRPPHUFH VRFLDO PHGLD EDQN WUDGLQJ
DQGVWUHDPLQJYLGHRVHUYLFHVDUHDIHZRIWKHP(YHQLWFDQEH
XVHG WR HQKDQFH WKH EXVLQHVV DQG SURGXFWLRQ SURFHVVHV )RU
H[DPSOH <RX7XEH KDV PRUH WKDQ D ELOOLRQ VXEVFULEHG XVHUV
ZKR LV YLHZLQJ H[WUHPH QXPEHU RI YLGHRV SHU GD\ 7KH\ DUH
XWLOL]LQJ %LJ 'DWD LQ RUGHU WR GHWHFW XVHU SDWWHUQV DQG RIIHU
YLGHRVWKDWFDQEHLQWHUHVWRILQGLYLGXDOXVHUV0RUHRYHUWKHVH
XVHUV DOVR XSORDGLQJ WKRXVDQGV RI YLGHRV SHU KRXU ZKLFK DUH
SURFHVVHG E\ WKHLU GLVWULEXWHG V\VWHPV )DFHERRN LV DQRWKHU
PRVWSRSXODU
@?A/,/96AB/C@6C/@ ,? D6,#CC E.C,? 
H[DPSOHRIVRFLDOPHGLDWKDWKDVPRUHWKDQELOOLRQRIXVHUV
DOORYHUWKHZRUOG7KHLUXVHUVSURGXFHPLOOLRQVRIVKDULQJSHU
GD\VXFKDVWH[WVSKRWRVDQGYLGHRV$PRQJWKHRWKHUWKLQJV
)DFHERRN PDNH XVH RI %LJ 'DWD LQ RUGHU WR GR IDFH GHWHFWLRQ
DQG RIIHU WR WDJ SHUVRQV LQ DQ XSORDGHG SLFWXUH DOPRVW LQ
PLOOLVHFRQGV DQG WKH\ DOVR RIIHU WR MRLQ WR JURXS RU SDJH
ZKLFK DUH H[SHFWHG RI LQWHUHVW RI XVHU 7KH\ XVH PDFKLQH
OHDUQLQJ DQG GHHS OHDUQLQJ DOJRULWKPV IRU WKHVH SXUSRVHV
ZRUNLQJRQWKHLU%LJ'DWDHQYLURQPHQW
%LJ GDWD SURYLGHV RSSRUWXQLWLHV WR FROOHFW VWRUH SURFHVV
DQGPDQDJHWKHKXJHDPRXQWRIGDWDZLWKLQDUHDVRQDEOHWLPH
7R KDYH VXFK LPSRUWDQW EHQHILWV WKHUH ZDV PDQ\ FKDOOHQJHV
WKDW6RIWZDUHHQJLQHHUVVKRXOGRYHUFRPH$OWKRXJKWKH\IRXQG
DZD\RIGRLQJWKLVWKH\VWLOOPDNHUHVHDUFKWRRSWLPL]HWKHP
WRILQGQHZZD\V%LJGDWDDSSOLFDWLRQVDUHSXVKLQJWKHOLPLWV
RI 6RIWZDUH (QJLQHHULQJ RQ PXOWLSOH KRUL]RQV 6XFFHVVIXO
VROXWLRQ VSDQ WKH GHVLJQ RI GDWD GLVWULEXWLRQ DQG GHSOR\PHQW
DUFKLWHFWXUHV>@
7RGD\¶V VROXWLRQV DUH PRVWO\ EDVHG RQ GLVWULEXWHG
FRPSXWLQJ DQG GLVWULEXWHG GDWD PDQDJHPHQW LQ DGGLWLRQ WR
QHZ SURJUDPPLQJ PRGHOV VXFK DV 0DS5HGXFH GHYHORSHG E\
*RRJOH
$FFRUGLQJ WR FKDQJLQJ WHFKQRORJ\ PDQ\ RUJDQL]DWLRQV
KDYHHLWKHUSODQRUZRUNRQPDQDJLQJWKHLUHQRUPRXVDPRXQW
RIGDWDWREHVWRUHGDQGSURFHVVHGLQRUGHUWRJDLQDGYDQWDJHLQ
JOREDOFRPSHWLWLRQ>@
$V VWDWHG DERYH WKHUH DUH PDQ\ ELJ GDWD IUDPHZRUNV
DOUHDG\ GHYHORSHG E\ 6RIWZDUH (QJLQHHUV DQG DUH EHLQJ
FRQWLQXRXVO\ LPSURYHG E\ XVLQJ QHZ WHFKQLTXHV :H NQRZ
WKDW QHZ WHFKQRORJLHV DQG LPSURYHPHQWV RQ H[LVWLQJ
WHFKQRORJLHV DOZD\V VWHP IURP KXPDQ QHHGV 7KHUH ZRXOG
QHYHUEHDVROXWLRQRIDQ\NLQGZLWKRXWKDYLQJGLIILFXOWLHVDQG
SUREOHPV7KLVLVWUXHRIFRXUVHIRUELJGDWDWHFKQRORJLHV>@
,Q WKLV VWXG\ ZH DUH WR DQDO\]H WKH PRVW LPSRUWDQW LVVXHV
WKDWZHKDYHWRRYHUFRPHGXULQJWKHGHYHORSPHQWD%LJ'DWD
$SSOLFDWLRQ $OWKRXJK WKHUH DUH PDQ\ NLQGV RI LVVXHV UHODWHG
WR 6RIWZDUH 'HYHORSPHQW /LIH &\FOH LQ >@ DQG >@
'LVWULEXWLRQ 'DWD DQG 'HSOR\PHQW LQ >@ DQG >@
'HYHORSPHQW 7HDP DQG 3URFHVV LQ >@ 5HTXLUHPHQWV LQ >@
5LVF 0DQDJHPHQW LQ >@ 2SHUDWLRQDO /RJ $QDO\VLV LQ >@
DQG

8%0.¶QG,QWHUQDWLRQDO&RQIHUHQFHRQ&RPSXWHU6FLHQFHDQG(QJLQHHULQJ 

'DWD7\SHV$QDO\VLV0HWKRGV'DWD6HFXULW\DQG$SSOLFDWLRQ DQG PHPRU\ LQWHQVLYLW\ LY WHVWLQJ Y UHOLDELOLW\ YL
LQ>@ZHKDYHOLPLWHGWKLVSDSHUZLWKWKHVFRSHRIKDYLQJWKH IDXOWWROHUDQFHYLLVHFXULW\DQGYLLLH[HFXWLRQORJDQDO\VLV
PRVW LPSRUWDQW  LVVXHV L VFDODELOLW\ LL  DYDLODELOLW\ LLL
GDWD
7KH PDLQ SXUSRVH DQG FRQWULEXWLRQ RI WKLV SDSHU LV WR
LQIRUP WKH UHVHDUFKHUV DERXW WKH VXEMHFW¶V QHHGV ZH EHOLHYH
PRUHHPSLULFDOVWXG\LQ6RIWZDUH(QJLQHHULQJGRPDLQIRUWKH
GHYHORSPHQWRI%LJ'DWD$SSOLFDWLRQ
7KHUHVWRIWKHSDSHULVRUJDQL]HGDVIROORZV1H[WVHFWLRQ
JLYHV DQ RYHUYLHZ LQIRUPDWLRQ DERXW WKH %LJ 'DWD
FKDUDFWHULVWLFV %LJ 'DWD )UDPHZRUNV DQG LWV UHODWLRQV WR
6RIWZDUH(QJLQHHULQJGLVFLSOLQH7KHQLQWKHUGVHFWLRQZHWU\
WR VXPPDUL]H WKH PRVW LPSRUWDQW FKDOOHQJHV DQG LVVXHV WKDW
6RIWZDUH (QJLQHHUV IDFHG DQGRU EHLQJ IDFHG GXULQJ WKH
SURGXFWLRQ RI %LJ 'DWD $SSOLFDWLRQV 7KH SDSHU HQGV ZLWK
GLVFXVVLRQDQGFRQFOXVLRQVHFWLRQ
,, %,*'$7$
$ 'LIIHUHQFHEHWZHHQ)UDPHZRUNDQG$SSOLFDWLRQ
%HIRUH JRLQJ PRUH LQWR GHSWK RI %LJ 'DWD )UDPHZRUN
FKDUDFWHULVWLFVDQGWKHLUGHVFULSWLRQVLWZRXOGEHEHWWHUWRKDYH
FRQYHQWLRQ DERXW ³ZKDW )UDPHZRUN DQG $SSOLFDWLRQ LV´ DQG
³ZKDW DUH WKH GLIIHUHQFH´ $V NQRZQ DSSOLFDWLRQ LV D
FRPSOHWHO\ ILQLVKHG VRIWZDUH GHYHORSHG IRU VRPH SXUSRVH
ZKLFK LV GLUHFWO\ XVHG E\ HQGXVHU 2Q WKH RWKHU KDQG
IUDPHZRUN LV VRPHWLPHV FDOOHG SDUWLDOO\ FRPSOHWH RU VHPL
FRPSOHWH DSSOLFDWLRQV 7KDW LV LW SURYLGHV PDQ\ LPSRUWDQW
IHDWXUHV DQG OLEUDULHV IRU DSSOLFDWLRQ GHYHORSHU WR HDVH WKHLU
VRIWZDUHGHYHORSPHQWSURFHVV
:LNLSHGLD GHILQHV D VRIWZDUH IUDPHZRUN DV ³$ VRIWZDUH
IUDPHZRUN LV D UHXVDEOH GHVLJQ IRU D VRIWZDUH V\VWHP RU
LVVXHV
9DULHW\ PHDQV FROOHFWLQJ GLIIHUHQW W\SHV RI GDWD IURP
GLIIHUHQW VRXUFHV VXFK DV VHQVRU VRFLDO QHWZRUN DQG PRELOH
SKRQH RSHUDWRUV 7KHVH YDULRXV GDWD PD\ EH FDWHJRUL]HG DV
VWUXFWXUHGVHPLVWUXFWXUHGDQGXQVWUXFWXUHG
9HORFLW\ UHIHUV WR WKH VSHHG RI GDWD SURGXFWLRQ IURP D
VLQJOHRUPDQ\VRXUFHV)RUH[DPSOHWKLQNDERXWDVPDUWFLW\
GDWD WKDW DUH EHLQJ SURGXFHG DQG FROOHFWHG IURP KXQGUHGV RI
VHFXULW\FDPHUDVWKRXVDQGVRIVHQVRUVHWF
9HUDFLW\LVWKHPHDQLQJIXOQHVVFRUUHFWQHVVDQGUHOLDELOLW\
RI GDWD ,I WKH GDWD FROOHFWHG IURP D IHZ VHQVRUV WKDW DUH
FRUUXSWHG WKH SURFHVVLQJ RI VXFK GDWD ZRXOG SURGXFH ZURQJ
UHVXOWV*DUEDJHLQJDUEDJHRXW
& %LJ'DWD)UDPHZRUNV
$VWKHGHILQLWLRQRI%LJGDWDVWDWHVLWLVDOPRVWLPSRVVLEOH
WR SURFHVV WKDW PXFK RI GDWD E\ XVLQJ WUDGLWLRQDO WHFKQLTXHV
DQGWHFKQRORJLHV7KHUHIRUHQHZWHFKQRORJLHVDQGWHFKQLTXHV
DUH QHHGHG WR SURFHVV PDQDJH DQG DQDO\]H KXJH DPRXQW RI
GDWD VHWV )RU WKDW SXUSRVH WKHUH DUH PDQ\ %LJ 'DWD
)UDPHZRUNVGHYHORSHGHLWKHUDVSURSULHWDU\RUDVRSHQVRXUFH
VRIWZDUH7KHPRVWIDPRXVRSHQVRXUFHVROXWLRQVDUH+DGRRS
6SDUN 6WRUP DQG )OLQN ZKLFK DUH EHLQJ XVHG WR VROYH PDQ\
VRIWZDUH HQJLQHHULQJ SUREOHPV LQ GDWDLQWHQVLYH DQG &38
LQWHQVLYHELJGDWDDSSOLFDWLRQV>@7KH\KDYHEHHQGHVLJQHGLQ
GLIIHUHQW ZD\V GHSHQGLQJ RQ WKHLU LQWHQW RI XVH IRU H[DPSOH
+DGRRS LV GHVLJQHG WR SURFHVV VWD\LQJ GDWD LQ EDWFKRULHQWHG
ZD\ ZKLOH WKH RWKHU DUH GHVLJQHG HLWKHU WR SURFHVV VWUHDPLQJ
RQO\GDWDRUIRUERWKSXUSRVHDWWKHVDPHWLPH
VXEV\VWHP $ VRIWZDUH IUDPHZRUN PD\ LQFOXGH VXSSRUW 'HSHQGLQJ RQ WKH SXUSRVH DQG W\SHV RI GDWD SURFHVVLQJ
SURJUDPV FRGH OLEUDULHV D VFULSWLQJ ODQJXDJH RU RWKHU WHFKQLTXHV IUDPHZRUNV FDQ VLPSO\ EH FDWHJRUL]HG LQWR WKUHH
VRIWZDUH WR KHOS GHYHORS DQG JOXH WRJHWKHU WKH GLIIHUHQW QDPHO\ EDWFKRQO\ VXFK DV +DGRRS VWUHDPRQO\ VXFK DV
FRPSRQHQWV RI D VRIWZDUH SURMHFW 9DULRXV SDUWV RI WKH
IUDPHZRUNPD\EHH[SRVHGWKURXJKDQ$3,´
2QH RI WKH PRVW LPSRUWDQW FKDUDFWHULVWLFV RI IUDPHZRUN LV
WKH ³LQYHUVLRQ RI FRQWURO´ 1RUPDOO\ QR RQH GHYHORS HYHU\
SDUWRIDQDSSOLFDWLRQIURPWKHVFUDWFK:HHLWKHUXVHOLEUDULHV
DQGRU IUDPHZRUNV ,I RXU DSSOLFDWLRQ FDOOV D IXQFWLRQPHWKRG
SURYLGHG E\ D OLEUDULHV RXU VRIWZDUH LV LQ FKDUJH RI FRQWURO
6WRUPDQGK\EULGVXFKDV6SDUNDQG)OLQNIUDPHZRUNV:H
DUH PDNLQJ WKLV FDWHJRUL]DWLRQ EHFDXVH RI WKHLU FKDOOHQJHV
ZRXOGGLIIHU
%HIRUH JLYLQJ LQIRUPDWLRQ DERXW LQGLYLGXDO IUDPHZRUNV LW
LV JRRG WR VWDUW ZLWK WKH SURSHUWLHV RI WKHVH FDWHJRULHV ZKLFK
DUHFRPPRQIRUDOO
%XWWKHFRQWUROLVLQYHUWHGLIZHDUHXVLQJDIHDWXUHSURYLGHGE\
IUDPHZRUN 7KDW LV ZH SURYLGH D FRPSRQHQW DQG IUDPHZRUN
FDOOVDQGFRQWUROVWKHIORZ)UDPHZRUNVDUHDOZD\VGHYHORSHG
IRUDVSHFLILFGRPDLQRUDSSOLFDWLRQW\SH
$IWHU VWDWLQJ WKLV FRQYHQWLRQ ZH FDQ FRQWLQXH
ZLWKFKDUDFWHULVWLFVRI%LJ'DWDDQGUHODWHG)UDPHZRUNV
 %DWFKRQO\'DWD3URFHVVLQJ
7KHKLVWRU\RIEDWFKRULHQWHGGDWDSURFHVVLQJJRHVEH\RQG
WKH %LJ 'DWD WLOO GDWD ZDUHKRXVH VROXWLRQV 'DWD VFLHQWLVWV
ZHUH FUHDWLQJ GDWD ZDUHKRXVH XVLQJ WUDGLWLRQDO WHFKQRORJLHV
ZKLFKFRQWDLQVDOOWKHGDWDWKDWDUHUHOHYDQWIRUWKHLUSXUSRVHV
DQGZHUHUXQQLQJWKHLU%,%XVLQHVV,QWHOOLJHQFHDQDO\WLFVMRE
HQGLQJLQGD\VRUHYHQLQZHHNV6LQFHWKHWHFKQRORJ\XVHGIRU
% 7KH&KDUDFWHULVWLFVRI%LJ'DWD WKDW SXUSRVH ZDV EDVHG RQ 5HODWLRQDO 'DWDEDVH 0DQDJHPHQW
'DWD SURFHVVHG E\ %LJ 'DWD IUDPHZRUNV KDYH FRPPRQ
FKDUDFWHULVWLFV NQRZQ DV PDQ\ 9¶V 5HFHQWO\ WKH QXPEHU RI
9¶V DUH LQFUHDVHG XS WR PRUH WKDQ  EXW WR XQGHUVWDQG ZKDW
%LJ'DWDPHDQVLWLVHQRXJKWRNQRZPRVWLPSRUWDQW9¶V>@
9ROXPHLVGHILQHGDVWKHVL]HRIGDWDWKDWLVEHLQJ
SURGXFHGDQGSURFHVVHGZKLFKLVHQRUPRXVO\ODUJH
6\VWHPV 5'%06 PRVWO\ ZDV QRW VXSSRUWLQJ GLVWULEXWHG
FRPSXWLQJ WKH H[HFXWLRQ WLPH ZDV VR ORQJ $OWKRXJK WKH\
ZHUH VXSSRUWLQJ YHUWLFDOVFDODELOLW\LH DGGLQJ QHZ UHVRXUFHV
WRWKHFRPSXWHUEHLQJLQXVHG%LJ'DWDVL]HVDUHVRH[WUHPHO\
ODUJH WKDW ZH IDFHG ZLWK D FKDOOHQJH RI KDYLQJ FRVWHIIHFWLYH
VROXWLRQ 6WRUDJH VSDFH ZDV DQRWKHU FKDOOHQJH VLQFH LW LV WRR
FRVWO\WR

8%0.¶QG,QWHUQDWLRQDO&RQIHUHQFHRQ&RPSXWHU6FLHQFHDQG(QJLQHHULQJ 

VWRUHWKDWVL]HRIGDWDRQDVLQJOHVWRUDJHXQLWZKLOHNHHSLQJ
LWUHVLOLHQWDJDLQVWIDLOXUHV
6ROXWLRQ WR WKHVH SUREOHPV LV SURYLGHG E\ %LJ 'DWD
)UDPHZRUNVVXFKDV+DGRRSZKLFKLVFRPSOHWHO\VFDODEOHLQ
7KH WKLUG FRPSRQHQW <HW $QRWKHU 5HVRXUFH 0DQDJHU 
<$51 LV DQRWKHU LPSRUWDQW FRPSRQHQW WKDW KDV WKH
UHVSRQVLELOLW\ RI UHVRXUFH PDQDJHPHQW UHVRXUFH XWLOL]DWLRQ
MREVFKHGXOLQJDQGNHHSLQJWUDFNRIMREDQGWDVNSURFHVVLQJLQ
RUGHU WR SURYLGH IDXOWWROHUDQW V\VWHP 7KLV FRPSRQHQW
HLWKHU GLUHFWLRQ YHUWLFDO RU KRUL]RQWDO UREXVW GLVWULEXWHG
IOH[LEOH DQG IDXOWWROHUDQW VROXWLRQ UXQQLQJ RQ FRVWHIIHFWLYH
FRPPRGLW\FRPSXWHUV
7KHPDLQSURSHUWLHVRIEDWFKRQO\GDWDSURFHVVLQJDUH DYDLODEOHFRPSXWHUV
ā 7KH GDWD LV LQ VWDWLF VWDWH WKDW LV WKH GDWD LV DOUHDG\
VWRUHGDQGDYDLODEOHIRUSURFHVVLQJ
ā ,PPXWDEOH GDWD LV LQ LWV ILQDO VWDWH DQG PRVWO\ UHDG
RSHUDWLRQLVGRQHRQLW
PRQLWRUV UHVRXUFHV EHLQJ XVHG RQ FOXVWHU FRPSXWHUV DQG MREV
UXQQLQJ ZLWKLQ WKH FRQWDLQHUV VR WKDW LQ FDVH RI DQ\ IDLOXUH LW
PDQDJHV WKH SURFHVVLQJ UHPDLQLQJ GDWD E\ XVLQJ RWKHU
:LWK DOO WKHVH IHDWXUHV DQG WKH RWKHU VXSSRUWLQJ WRROV DQG
OLEUDULHV ZKLFK ZH FDOO WKH IRXUWK FRPSRQHQW LW LV DLPHG WR
SURYLGHFRVWHIIHFWLYHVROXWLRQIRUPDQDJLQJDQGSURFHVVLQJRI
ODUJH YROXPHV RI GDWD LQ DGGLWLRQ WR HQVXUH YDULRXV GDWD
DQDO\WLFVEHFDUULHGRXWDVHIILFLHQWDVSRVVLEOH
ā 'DWDVL]HLVH[WUHPHO\ODUJHVRWKDWWKHRQO\RSWLRQLVWR
SURFHVVWKHPLQEDWFKRULHQWHGZD\
ā 'DWD LV ERXQGHG HYHQ IRU H[WUDODUJH GDWD LW KDV D
OLPLWHGVL]H
$SDFKH+DGRRS
,WLVWKHPRVWSRSXODURSHQVRXUFHVRIWZDUHLPSOHPHQWDWLRQ
RI %LJ 'DWD )UDPHZRUN WKDW SURYLGHV EDWFK RULHQWHG
GLVWULEXWHG VFDODEOH DQG UHOLDEOH GDWD SURFHVVLQJ +DGRRS LV
HVSHFLDOO\ GHYHORSHG IRU WKH SXUSRVH RI EDWFK SURFHVVLQJ RQ
 6WUHDPRQO\'DWD3URFHVVLQJ
6WUHDPSURFHVVLQJUHIHUVWRWKHRSHUDWLRQRISURFHVVLQJGDWD
LPPHGLDWHO\ZKHQLWLVUHFHLYHG,QWKDWVHQVHWKHGDWDZRXOG
EHHYHQWOLNHGDWDVXFKDVVHQVRUGDWD*36GDWDRUZKDWHYHU
WKHVRXUFHEXWWKHLPSRUWDQWWKLQJKHUHLVWKDWWKHUHZRXOGEH
QRQH RU PLQLPDO VWDWH WUDQVLWLRQ EHWZHHQ WKH VXEVHTXHQWO\
UHFHLYHGGDWD,IWKHRSHUDWLRQQHHGVWRFRPSXWHDOOGDWDEHIRUH
SURGXFLQJ WKH UHVXOWV EDWFKRULHQWHG IUDPHZRUNV ZRXOG EH
PRUHVXLWDEOHUDWKHUWKDQVWUHDPRULHQWHGRQHV7KHUHDUHWZR
GLIIHUHQW ZD\V RI SURFHVVLQJ VWUHDP HLWKHU SURFHVV
VWD\LQJ GDWD ,W FRQVLVWV RI IRXU FRPPRQ FRPSRQHQWV QDPHO\ VLPXOWDQHRXVO\ DV LQ )/LQN DQG 6WRUP RU SURFHVV LQ PLFUR
0DS5HGXFH +')6 <$51 DQG RWKHU VXSSRUWLQJ WRROV VXFK
DV+%DVH6TRRS3LJ+LYHHWF>@
0DS5HGXFHLVDSURJUDPPLQJPRGHOGHYHORSHGE\JRRJOH
DQGWKHRSHQVRXUFHLPSOHPHQWDWLRQRIWKLVPRGHOLVRQHRIWKH
PRVWLPSRUWDQWFRPSRQHQWVRIWKH+DGRRS)UDPHZRUNZKLFK
LV DOVR FDOOHG WKH +DGRRS (QJLQH %\ XVLQJ 0DS5HGXFH
SURJUDPPLQJ PRGHO +DGRRS DGGUHVVHV DOO WKH GHWDLOV RI
KDQGOLQJ GLVWULEXWHG FRPSXWLQJ DQG GDWD ORFDOLW\ WR HOLPLQDWH
WKHFRPSOH[LWLHVRIGLVWULEXWHGGDWDSURFHVVLQJ
7KHVHFRQGPDMRUFRPSRQHQWRI+DGRRSLVWKHGLVWULEXWHG
EHQFKVHWWLQJVDVLQ6SDUN
7KHPRVWLPSRUWDQWFKDUDFWHULVWLFVRIVWUHDPSURFHVVLQJDUH
ā 'DWDLVSURFHVVHGLQUHDOWLPHRULQQHDUUHDOWLPHDQG
LWLVFRQWLQXRXVO\IORZVLQWRV\VWHP
ā 'DWDUHFRUGVDUHSURFHVVHGLWHPZLVHRQHRUDIHZDWD
WLPH
'DWDLVXQERXQGHGWKDWLVWKHSURFHVVZRXOGQHYHUEH
ā
HQGHGZLWKRXWXVHULQWHUYHQWLRQRULQWHQWLRQDOVWRS
ILOHV\VWHP+')6ZKLFKLVERUURZHGIURP*RRJOH¶VVROXWLRQ
ā /RZODWHQF\LVWKHPRVWVLJQLILFDQWIHDWXUH
*)6 *RRJOH )LOH 6\VWHP 8VLQJ +')6 WKH GDWD VL]H RI
WHUDE\WH SHWDE\WH DQG HYHQ ([DE\WH DQG =HWWDE\WH FDQ EH
KDQGOHG RQ FRPPRGLW\ KDUGZDUH GHSOR\HG LQ FOXVWHUHG
HQYLURQPHQW 2EYLRXVO\ LW ZRXOG EH YHU\ GLIILFXOW WR KDQGOH
VXFKDPRXQWVRIGDWDE\VWRULQJWKHPRQDVLQJOHVWRUDJHXQLW
DQG WR WUDQVIHU WKHP WKURXJK QHWZRUN LQ RUGHU WR SURFHVV RQ
GLVWULEXWHG FRPSXWLQJ HQYLURQPHQWV 7R WKLV HQG WKH VROXWLRQ
LV VLPSO\ QRW RQO\ GLVWULEXWLQJ WKH FRPSXWDWLRQ EXW DOVR
GLVWULEXWLQJ WKH GDWD RQ ZKLFK WKH FRPSXWDWLRQ ZRXOG RFFXUV
:KHQVDYLQJGDWDLQWRDILOH+')6SDUWLWLRQVGDWDLQWREORFNV
6WRUP
$SDFKH 6WRUP LV D IUDPHZRUN LQWHQWLRQDOO\ GHVLJQHG WR
SURFHVV VWUHDPLQJ GDWD LQ UHDOWLPH E\ XVLQJ GLVWULEXWHG
FRPSXWLQJ ZLWK ORZODWHQF\ 8QOLNH 6SDUN 6WRUP LV SXUH
VWUHDPRQO\GDWDSURFHVVLQJIUDPHZRUN7KHUHIRUHWKHGDWDLV
SURFHVVHGDVVRRQDVLWIORZVLQWRV\VWHPZLWKRXWDQ\GHOD\RU
ODWHQF\ 7KLV SXWV 6WRUP DKHDG RI WKH 6SDUN LQ D VWUHDP GDWD
XVXDOO\0%LQVL]HDQGVHQGVHDFKEORFNWREHKRVWHGLQD SURFHVVLQJ $V OLNH RWKHU IUDPHZRUNV 6WRUP LV D GLVWULEXWHG
VWRUDJH PHGLXP RI GLIIHUHQW FRPSXWHUV 0RUHRYHU WR LQVXUH IDXOWWROHUDQWDQGVFDODEOHIUDPHZRUN
DJDLQVWGDWDORVVLQFDVHRIIDLOXUHVRUFRUUXSWLRQWKHUHSOLFDV
RIHDFKEORFNRIGDWDLVXVXDOO\NHSWRQDWOHDVWWKUHHGLIIHUHQW
FRPSXWHUV 7KH PRVW LPSRUWDQW DGYDQWDJH RI XVLQJ +')6 LV
WKDW GXULQJ WKH FRPSXWDWLRQ +DGRRS GRHV QRW UHTXLUH
WUDQVIHUULQJGDWDRYHUWKHQHWZRUNDVVXFKWKHFRGHWRSURFHVV
GDWD LV VSDZQHG RQ WKH VDPH FRPSXWHUV ZKHUH WKH GDWD LV
ORFDOO\ VWRUHG 1HYHUWKHOHVV ,2 SHUIRUPDQFH LV JUHDWO\
LPSURYHGZKHQWKHGDWDLVSURFHVVHGRQHDFKPDFKLQHLQVPDOO
EORFNV
6WRUP XVHV '$* 'LUHFWHG $F\FOLF *UDSKV ZKLFK LV
FDOOHG ³WRSRORJ\´ LQ WKHLU MDUJRQ LQ RUGHU WR RUFKHVWUDWH LWV
WDVNV7KLVHQDEOHVXVHUVWRDSSO\DQ\NLQGRIILOWHUVFOHDQLQJ
DOJRULWKPV HWF EHIRUH VHQGLQJ WKH GDWD WXSOH WR DQDO\WLFV
DOJRULWKPV2WKHUMDUJRQVXVHGLQ6WRUPDUHVSRXWVWUHDPDQG
EROWV 6WUHDP UHIHUV WR XQERXQGHG GDWD VWUHDP WKDW LV IORZLQJ
LQWR V\VWHP VSRXW LV OLNH D PHVVDJH TXHXH ZKHUH WKH GDWD
FRPLQJ IURP VWUHDP ILUVW HQWHUV LQWR 6WRUP 6WRUP VXSSRUWV
LQWHJUDWLRQZLWKPDQ\

8%0.¶QG,QWHUQDWLRQDO&RQIHUHQFHRQ&RPSXWHU6FLHQFHDQG(QJLQHHULQJ 

PHVVDJH TXHXH V\VWHPV DQG GDWDEDVHV IRU WKDW SXUSRVH
7KHEROWVDUHWKHLQGLYLGXDOWDVNVRURSHUDWLRQVH[HFXWHGLQ
6WRUP
$VVWDWHGRQLWVZHEVLWH6WRUPKDVPDQ\XVHFDVHVUHDO
WLPH DQDO\WLFV RQOLQH PDFKLQH OHDUQLQJ FRQWLQXRXV
FRPSXWDWLRQGLVWULEXWHG53&(7/DQGPRUH6WRUPLVIDVWD
EHQFKPDUN FORFNHG LW DW RYHU D PLOOLRQ WXSOHV SURFHVVHG SHU
VHFRQG SHU QRGH ,W LV VFDODEOH IDXOWWROHUDQW JXDUDQWHHV \RXU
GDWDZLOOEHSURFHVVHGDQGLVHDV\WRVHWXSDQGRSHUDWH>@
7KH PRVW LPSRUWDQW GLIILFXOWLHV LQ VXFK V\VWHP LV
UHSDUWLWLRQLQJ WKH VWUHDPV EHWZHHQ HDFK VWDJH RI WKH
FRPSXWDWLRQ
KHOS WR VROYH LPSRUWDQW EXVLQHVV SUREOHPV WKHUH DUH PDQ\
6RIWZDUH (QJLQHHULQJ LVVXHV DULVHV >@ 7KH PRVW FRPPRQ
LVVXHVDUH
6FDODELOLW\ ,VVXH $V VWDWHG DERYH WKH VRIWZDUH VKRXOG EH
DEOHWRVROYHWKHSUREOHPVHYHQLIWKHYROXPHRIGDWDEHFRPH
LQFUHGLEO\KXJH7KHUHIRUHWKHVRIWZDUHSURGXFHGE\6RIWZDUH
(QJLQHHUV VKRXOG EH DEOH WR VFDOH GHSHQGLQJ RQ WKH SUREOHP
VL]H 7KHUH DUH WZR W\SHV RI VXFK VFDOLQJ FDQ EH DSSOLHG
YHUWLFDO DQG KRUL]RQWDO ,Q FRQYHQWLRQDO ZD\ RI VROYLQJ VXFK
SUREOHPZHZHUHPRVWO\XVLQJYHUWLFDOVFDOLQJZKHUHZHDGG
DGGLWLRQDO UHVRXUFHV WR WKH FRPSXWHU WKDW LV UXQQLQJ RXU
DSSOLFDWLRQVXFKDVLQFUHDVLQJWKHQXPEHURI&38LQFUHDVLQJ
WKH PHPRU\ DQGRU VWRUDJH XQLWV ZKHQHYHU QHHGHG
)XUWKHUPRUH WUDGLWLRQDO UHODWLRQDO GDWDEDVH V\VWHPV KDYH
 +\EULGEDWFKDQGVWUHDP'DWD3URFHVVLQJ
$V H[SHFWHG K\EULG VROXWLRQ SURYLGHV ERWK ZD\ RI GDWD
SURFHVVLQJ EDWFK DQGRU VWUHDP ,W ZRXOG EH REYLRXV WKDW WKH
FKDUDFWHULVWLFV RI WKDW NLQG ZRXOG DOVR FRQVWLWXWHV WKH PL[ RI
WKRVH FKDUDFWHULVWLFV EHORQJLQJ WR EDWFKRQO\ DQG VWUHDPRQO\
IUDPHZRUNV
6SDUN
7KH LQDELOLW\ RI +DGRRS WR RSHUDWH RQ VWUHDPLQJ GDWD KDV
LPPHGLDWHO\ UHVXOWHG DV D ELJ UHTXLUHPHQW DV VRRQ DV WKH
+DGRRS LV DQQRXQFHG 7R WKLV HQG VRIWZDUH GHYHORSHUV KDYH
GHYHORSHGWKH6SDUN)UDPHZRUNZKLFKFDQSHUIRUPODUJHGDWD
RSHUDWLRQV RQ ERWK VWD\LQJ DQG VWUHDPLQJ GDWD E\ XVLQJ
+DGRRS¶VDUFKLWHFWXUH'DWDSURFHVVLQJPHFKDQLVPRI6SDUNLV
DOPRVW LGHQWLFDO WR +DGRRS EXW LW XVHV PLFUREDWFKLQJ
DUFKLWHFWXUH WR KDQGOH WKH VWUHDPLQJ GDWD 6SDUN KDV D VWURQJ
LQPHPRU\ GDWD RSHUDWLRQ E\ KDYLQJ D VSHFLDOO\ GHYHORSHG
LQKHUHQW YHUWLFDOVFDOLQJ OLPLWDWLRQV >@ %XW ZKHQ WKH GDWD
VL]HJURZVWKDWVROXWLRQEHFRPHVVRH[SHQVLYHWKDWLWEHFRPHV
LQIHDVLEOH 6R WKHUH DUH WZR W\SHV RI VFDODELOLW\ LVVXH LQ %LJ
'DWD 6\VWHPV 2QH LV DERXW WKH VFDODELOLW\ RI FRPSXWDWLRQDO
RSHUDWLRQVWKHRWKHULVDERXWWKHGDWDVWRUDJH
,Q%LJ'DWD6\VWHPVWKHVFDOLQJRSHUDWLRQLVDOPRVWDOZD\V
GRQHLQKRUL]RQWDOO\VFDOLQJZKLFKPHDQVDGGLQJQHZHQWLW\RI
FRPPRGLW\ FRPSXWHU RU RWKHU UHVRXUFHV WR WKH FOXVWHU
6RIWZDUHGHYHORSHGIRUWKLVSXUSRVHPD\QHHGWRRSHUDWHHYHQ
RQWHQVRIWKRXVDQGVRIFRPSXWHUVLQWKHFOXVWHU%XWWKHQWKH
FKDOOHQJH DSSHDUV KRZ FDQ EH VXUH DERXW WKH VFDODELOLW\ RI
FRPSXWDWLRQ ZLWKRXW GHSOR\LQJ DSSOLFDWLRQ RQ VXFK D ELJ
FOXVWHU WKDW ZRXOG VXVWDLQ XQGHU GDWDLQWHQVLYH ZRUNORDGV" ,Q
WKDW FDVH GHVLJQ RI DSSOLFDWLRQ DQG DOJRULWKPV EHFRPH WKH
FUXFLDO SDUW RI 6RIWZDUH 'HYHORSPHQW 3URFHVV %XW WKHQ WKH
RWKHU TXHVWLRQ DULVH ³ZRXOG WKH FRQYHQWLRQDO VRIWZDUH
DUFKLWHFWXUH EH VWURQJ HQRXJK WR KDYH VXFK VXVWDLQDELOLW\ DQG
FRPSRQHQW QDPHG DV 5HVLOLHQW 'LVWULEXWHG 'DWDVHW 5''
>@6WUHDPLQJGDWDLVSXWLQWRDSLSHOLQHDQGWKHGDWDUHFHLYHG
DWWKHXVHUGHILQHGWLPHLQWHUYDOVLVH[WUDFWHGIURPWKHWDLODQG
SURFHVVHG'XULQJWKHSURFHVVWKHQHZGDWDFRQWLQXHVWRDUULYH
DW WKH SLSHOLQH V\VWHP 7KHVH WLPH ZLQGRZV DUH XVXDOO\ VKRUW
LQWHUYDOV PHDVXUHG E\ VHFRQGV %HVLGHV WKH WLPH ZLQGRZ WKH
GDWD DPRXQW ZLQGRZ WHFKQLTXH FDQ DOVR EH XWLOL]HG )RU
H[DPSOHLQHDFKF\FOH;DPRXQWRIWKHGDWDZKLFKZLOOEHD
VFDODELOLW\"´ > @ 0RVW SUREDEO\ WLJKWO\FRXSOLQJ RI
FRPSRQHQWZRXOGEHQHHGHGZKLFKUHTXLUHVVRPHH[WHQVLRQLQ
WKHWUDGLWLRQDOVRIWZDUHDUFKLWHFWXUHV
$OWKRXJK%LJ'DWDIUDPHZRUNVDUHGHVLJQHGLQDZD\WKDW
WKH FOXVWHU FDQ EH VFDOHG RQ GHPDQG WKH GHYHORSHG VRIWZDUH
LWVHOIPXVWDOVREHZHOOVFDODEOH7RXVHWKHVFDODELOLW\IHDWXUH
RI IUDPHZRUNV VRIWZDUH HQJLQHHU PXVW FUHDWH VXLWDEOH
DOJRULWKPVDQGFRGHV
VPDOO DPRXQW LV WDNHQ DQG SURFHVVHG 6LQFH 6SDUN SURFHVVHV
VPDOO DPRXQWV RI GDWD LQ HDFK F\FOH WKH GDWD LV VWRUHG LQ
PHPRU\DQGDQLQPHPRU\RSHUDWLRQLVSHUIRUPHG7KLVEULQJV
DERXW  WR  WLPHV IDVWHU SHUIRUPDQFH ZKHQ FRPSDUHG WR
+DGRRS:KLOH6SDUN
VPRVWLPSRUWDQWDGYDQWDJHLVWRSHUIRUP GHQRUPDOL]HGGDWD>@
LQPHPRU\ SURFHVVLQJ WKLV LV DOVR WKH PRVW VLJQLILFDQW
GLVDGYDQWDJH 7KH GDWD EHLQJ SURFHVVHG KDV WR ILW LQWR WKH
PHPRU\WKDWLVWKHWRWDODPRXQWRIPHPRU\RIWKHFRPSXWHU
FOXVWHUWREHXVHGPXVWEHJUHDWHUWKDQWKHWRWDODPRXQWRIGDWD
EHLQJ SURFHVVHG 2WKHUZLVH WKH VSHHG REWDLQHG IURP WKH LQ
PHPRU\SURFHVVLQJZRXOGEHGHFUHDVHG
6SDUN XVXDOO\ XVHV WKH FRPSRQHQWV RI +DGRRS VR WKDW LQ
PRVW FDVHV ERWK 6SDUN DQG +DGRRS LV LQVWDOOHG RQ WKH VDPH
HQYLURQPHQW
,,, ,668(6$1'&+$//(1*(6
'DWD SURFHVVLQJ LQFOXGHV GLIIHUHQW W\SH RI DQDO\WLFV ZKLFK
FDQ EH FDWHJRUL]HG DV GHVFULSWLYH GLDJQRVWLF SUHGLFWLYH DQG
SUHVFULSWLYH DQDO\WLFV $OWKRXJK WKHVH DQDO\WLFV XVHG LQ %LJ
'DWD
2QWKHRWKHUKDQGVRIWZDUHHQJLQHHUPXVWWDFNOHZLWKWKH
GLVWULEXWHG GDWD VWRUDJH FKDOOHQJHV 7KH\ PXVW WUDGH
FRQVLVWHQF\DJDLQVWDYDLODELOLW\GLVWULEXWLRQDJDLQVWPRGHUDWH
SHUIRUPDQFHDQGVWULFWO\QRUPDOL]HGGDWDDJDLQVWLQWHQWLRQDOO\
$YDLODELOLW\ ,VVXH :LWKRXW XVLQJ GLVWULEXWHG FRPSXWLQJ
DQGGDWDGLVWULEXWLRQPHFKDQLVPLWZRXOGQHDUO\EHLPSRVVLEOH
WRSURGXFHDVROXWLRQLQ%LJ'DWD7KDWZRXOGUHVXOWLQDQRWKHU
PRVWFRPPRQLVVXHRIKDYLQJSRVVLEOHIDLOXUHLQWKHFOXVWHU,W
LV GLIILFXOW WR WUDFN IDLOHG FRPSRQHQWV DQG SURGXFH UHVLOLHQW
VROXWLRQ ZLWKRXW KDYLQJ VXSSRUW IURP IUDPHZRUN :KHQ DQ\
SUREOHP RFFXUV RQ WKH V\VWHP VRIWZDUH VKRXOG SURFHHG
ZLWKRXW DQG HUURU 7KLV NLQG RI SUREOHPV DUH KDQGOHG
LQKHUHQWO\ E\ UHVRXUFH PDQDJHU DQG VFKHGXOHU RI %LJ 'DWD
IUDPHZRUNV>@%XWWKHQZKDWLIWKHIDLOXUHUHDVRQLVXQNQRZQ
DQG RFFXUV ZKHQ WKH V\VWHP LV VFDOHG WR D ODUJH QXPEHU RI
FOXVWHU",QWKDWVFDOHLWLVDOPRVWLPSRVVLEOHWRXVHSURILOLQJRU
GHEXJJLQJ WRROV 6R WKH QHHG IRU ORJ DQDO\]LQJ RSHUDWLRQ
DSSHDUVZKLFKLWVHOILVD%LJ'DWDSUREOHP

8%0.¶QG,QWHUQDWLRQDO&RQIHUHQFHRQ&RPSXWHU6FLHQFHDQG(QJLQHHULQJ 

 $VVXPHWKDWZHKDYHFUHDWHGDQDSSOLFDWLRQZKLFKUHFHLYHV
ZKROH GDWD RI D VPDUW FLW\ LH VHQVRUV GDWD YLGHR GDWD HWF
DQG ZH REVHUYHG DQ XQH[SHFWHG UDQGRP IDLOXUH ZLWKLQ DQ
XQNQRZQFRPSRQHQWZKLFKUHVXOWVLQZDVWLQJKXJHDPRXQWRI
UHVRXUFH &DQ ZH VWRS WKH V\VWHP" 0RVW SUREDEO\ QR
$YDLODELOLW\ RI VXFK V\VWHPV LV FUXFLDO 7KDW EULQJV 6RIWZDUH
(QJLQHHU WR WKLQN PRUH DERXW WKH 6RIWZDUH $UFKLWHFWXUH DQG
GHSOR\PHQWPHWKRGRORJLHV>@
'DWD,QWHQVLYHDQG0HPRU\,QWHQVLYHQHVV,VVXH%LJGDWD
LQFOXGHV KXJH DPRXQW RI GDWD ZKLFK WDNHV WRR PXFK WLPH WR
UHDG IURP VWRUDJH 7R VSHHGXS WKLV SURFHVV VRIWZDUH PXVW EH
DEOH WR UHDG GDWD LQ SDUDOOHO +DQGOLQJ WKH SDUDOOHO UHDGZULWH
RSHUDWLRQ EHFRPHV D ELJ FKDOOHQJH IRU SURJUDPPHUV /LNH LQ
VFDODELOLW\ LVVXH KRUL]RQWDO VFDODELOLW\ LV QHFHVVDU\ IRU WKH
VROXWLRQ ZKLFK PHDQV D GLVWULEXWHG GDWD VWRUDJH V\VWHPV DUH
QHHGHG:RXOG\RXSUHIHUWRFUHDWHDQGXVHDVWRUDJHFOXVWHU"
0RVW SUREDEO\ QR :H DUH XVLQJ GLVWULEXWHG FRPSXWLQJ IRU
VFDODELOLW\RIFRPSXWDWLRQSRZHUDQGLQPRVWFDVHV%LJ'DWD
)UDPHZRUNVVWRUHVGDWDSDUWLWLRQVWRWKHFRPSXWDWLRQQRGHV
%XWWKHQDQRWKHUFKDOOHQJHDSSHDUV\RXUDSSOLFDWLRQPXVW
EH DZDUH RI GDWD UHSOLFDWLRQ DQG KDQGOH LQFRQVLVWHQFLHV IURP
FRQIOLFWLQJUHSOLFDXSGDWHV
7HVWLQJ,VVXH,WLVVRPHWLPHVDOPRVWLPSRVVLEOHWRFUHDWHD
,9 ',6&866,21$1'&21&/86,21
7KH VL]H RI GDWD LV JURZLQJ XS LQFUHGLEO\ ,W LV GLIILFXOW WR
PDQDJH DQG FRQWURO KXJH GDWD VHWV XVLQJ FRQYHQWLRQDO
GDWDEDVH PDQDJHPHQW V\VWHPV 7KLV GLIILFXOW\ EULQJV WKH
FRQFHSWLQWR6RIWZDUH%LJ(QJLQHHULQJ'DWDDUHD,QSURGXFLQJ
ELJ GDWD DSSOLFDWLRQV VRIWZDUH HQJLQHHUV VKRXOG KDQGOH WKH
SUREOHPVLQVSLUHGIURPWKHGLVWULEXWHGDQGSDUDOOHOFRPSXWLQJ
WHFKQLTXHV $OWKRXJK PRVW RI SUREOHPV DUH KDQGOHG WKHVHE\
WKHKHOSRIELJGDWDV\VWHPVWKHUHUHPDLQVWROHUDQWLPSRUWDQW
HIILFLHQWSUREOHPVWKDWVKRXOGEHDGGUHVVHGGXULQJDSSOLFDWLRQ
GHYHORSPHQWSURFHVVVFDODELOLW\UHOLDELOLW\DQGVXFKWHVWLQJDV
LVVXHV(VSHFLDOO\WKHWHVWLQJLVVXHZRXOGEHKDUGWRGLVDSSHDU
IURP ELJ GDWD EHFDXVH RI XQH[SHFWHG GDWD VRIWZDUH DQG
V\VWHPXQNQRZQ IHDWXUHV UHVLGLQJ LQ GDWD EHLQJ SURGXFHG LQ
UHDOWLPHRSHUDWLRQV
5()(5(1&(6
>@ & *RNKDQ = .DUDND\D $ <D]LFL 6\VWHPDWLF 0DSSLQJ 6WXG\ RQ
3HUIRUPDQFH 6FDODELOLW\ LQ %LJ 'DWD RQ &ORXG 8VLQJ 90 DQG
&RQWDLQHU ,),3 $GYDQFHV LQ ,QIRUPDWLRQ DQG &RPPXQLFDWLRQ
7HFKQRORJ\9ROSS
>@ 9'.XPDUDQG3$OHQFDU6RIWZDUHHQJLQHHULQJIRUELJGDWDSURMHFWV
JRRG VDPSOH GDWD WR WHVW %LJ 'DWD $SSOLFDWLRQ VLQFH ZH DUH
ZRUNLQJ ZLWK KXJH DPRXQW DQG YDU\LQJ GDWD $GGLWLRQDOO\ LW
PD\ QRW EH SRVVLEOH WR KDYH UHDO GDWD ZKHQ WHVWLQJ 7R VRPH
H[WHQWWKLVSUREOHPFDQEHVROYHGE\XVLQJV\QWKHWLFGDWDEXW
WKHQ ZH QHHG WR JXHVV DOO XQNQRZQV WKDW PD\ RFFXUV LQ GDWD
'RPDLQV PHWKRGRORJLHV DQG JDSV %LJ 'DWD %LJ 'DWD  ,(((
,QWHUQDWLRQDO&RQIHUHQFHRQ,(((
>@ , *RUWRQ DQG - .OHLQ ³'LVWULEXWLRQ 'DWD 'HSOR\PHQW 6RIWZDUH
$UFKLWHFWXUH &RQYHUJHQFH LQ %LJ 'DWD 6\VWHPV´ ,((( 6RIWZDUH SS
0D\-XQH
>@ , *RUWRQ $% %HQHU DQG$ 0RFNXV 6RIWZDUH HQJLQHHULQJ IRU ELJ
7KLV PD\ EH WKH PRVW LPSRUWDQW DQG PRVW GLIILFXOW SUREOHP GDWDV\VWHPV,(((6RIWZDUH
WKDW 6RIWZDUH (QJLQHHUV ZRXOG EH IDFHG ZLWK ZKHQ FUHDWLQJ >@ . 0$QGHUVRQ (PEUDFH WKH &KDOOHQJHV 6RIWZDUH (QJLQHHULQJ LQ D
%LJ'DWDDSSOLFDWLRQV %LJ 'DWD :RUOG ,((($&0 VW ,QWHUQDWLRQDO:RUNVKRS RQ %LJ 'DWD
6RIWZDUH(QJLQHHULQJ
5HOLDELOLW\,VVXH5HOLDELOLW\RIELJGDWDVRIWZDUHFDQQRWEH
>@ 1,67 ³1,67 %LJ 'DWD ,QWHURSHUDELOLW\ )UDPHZRUN
JXDUDQWHHG ZLWKRXW GHSOR\LQJ DSSOLFDWLRQ IURP WHVWLQJ 9ROXPH'HILQLWLRQV´ 1,67 6SHFLDO 3XEOLFDWLRQ  1,67 %LJ 'DWD
HQYLURQPHQW WR RSHUDWLRQ EHFDXVH WKH EHKDYLRU RI GLVWULEXWHG
DSSOLFDWLRQPD\GLIIHULQUHDORSHUDWLRQDQGRQGLIIHUHQWVL]HRI
FOXVWHUV 7KLV PHDQV WKDW VRIWZDUH HQJLQHHU PXVW HPSOR\ DQG
WHVW WKHLU VRIWZDUH LQ UHDOWLPH RSHUDWLRQV ZLWK UHDOWLPH GDWD
EHIRUH GHSOR\PHQW RQ RSHUDWLRQDO V\VWHPV 7KDW SURFHVV PD\
WDNHGD\VRUZHHNV
([HFXWLRQ /RJV ,VVXH %LJ GDWD DSSOLFDWLRQ GHDOV ZLWK
KXJH DPRXQW DQG KLJK YHORFLW\ LQJUHVV DQG HJUHVV GDWD
SURFHVVLQJ ZKHUH KXJH DPRXQW RI ORJ ILOHV VKRXOG DOVR EH
FUHDWHG 7KDW ORJV PD\ DOVR EH XVHG E\ DSSOLFDWLRQ LWVHOI
6RIWZDUHHQJLQHHUVPXVWWDFNOHZLWKVWUHDPLQJWKHVHORJGDWD
DQDO\]LQJLWDQGDFWLQJUHVSHFWLYHO\>@
)DXOW 7ROHUDQFH ,VVXH :KHQ D QRGH LQ ELJ GDWD FOXVWHU
3XEOLF :RUNLQJ *URXS 1%'3:* 'HILQLWLRQV DQG 7D[RQRPLHV
6XEJURXS6HS85/KWWSG[GRLRUJ1,6763
>@ -$O-DURRGL DQG 1 0RKDPHG &KDUDFWHULVWLFV DQG 5HTXLUHPHQWV RI
%LJ'DWD$QDO\WLFV$SSOLFDWLRQV,(((QG,QWHUQDWLRQDO&RQIHUHQFHRQ
&ROODERUDWLRQDQG,QWHUQHW&RPSXWLQJ,(((&RPSXWHU6RFLHW\SS

>@ $SDFKH +DGRRS 2IILFLDO:HE 6LWH KDGRRSDSDFKHRUJ /DVW YLVLWHG RQ
-XQH
>@ $SDFKH6WRUP2IILFLDO:HE6LWHVWRUPDSDFKHRUJ/DVWYLVLWHGRQ-XQH

>@ $SDFKH6SDUN2IILFLDO:HE6LWHVSDUNDSDFKHRUJ/DVWYLVLWHGRQ-XQH

>@ 3- 6DGDODJH DQG 0 )RZOHU ³1R64/ 'LVWLOOHG´ $GGLVRQ:HVOH\
3URIHVVLRQDO
>@ & +RQJ0HL 5 .D]PDQ DQG 6 +D]L\HY 6WUDWHJLF SURWRW\SLQJ IRU
FRUUXSWHGRUVWRSSHGIRUVRPHUHDVRQQRWRQO\ZHORVHWKHWDVN
H[HFXWLRQV EXW DOVR GDWD ORFDOLW\ SUREOHP DQGRU GDWD ORVW
SUREOHPPD\RFFXU)RUWKDWUHDVRQWKHGDWDEORFNVVKRXOGEH
UHSOLFDWHG WR VHYHUDO QRGHV RQ WKH FOXVWHU 7KH FDSDELOLW\ RI
IUDPHZRUNV HVSHFLDOO\ WKH 'LVWULEXWHG )LOH 6\VWHP FDQ EH
HPSOR\HGWRWROHUDWHWKLVLVVXH
GHYHORSLQJELJGDWDV\VWHPV,(((6RIWZDUH
>@ $ 0LUDQVN\\ $ +DPRX/KDGM ( &LDOLQL $ /DUVVRQ 2SHUDWLRQDO
/RJ $QDO\VLV IRU %LJ 'DWD 6\VWHPV &KDOOHQJHV DQG 6ROXWLRQV ,(((
6RIWZDUH 3XEOLVKHG E\ WKH ,((( &RPSXWHU 6RFLHW\ SS 0DUFK
$SULO
>@ -$O-DURRGL%+ROOHLQ10RKDPHG$SSO\LQJVRIWZDUHHQJLQHHULQJ
SURFHVVHV IRU ELJ GDWD DQDO\WLFV DSSOLFDWLRQV GHYHORSPHQW  ,(((
WK $QQXDO &RPSXWLQJ DQG &RPPXQLFDWLRQ :RUNVKRS DQG &RQIHUHQFH
6HFXULW\ ,VVXHV %LJ 'DWD SURFHVVLQJ HQYLURQPHQWV DUH &&:&<HDU3DJHV'2,&&:&
PRVWO\ PXOWLWHQDQW HQYLURQPHQW WKDW PHDQV PDQ\ GLIIHUHQW >@ =/Y+6RQJ3%DVDQWD9DO$6WHHG0-R1H[W*HQHUDWLRQ
WHQDQWV FDQ GHSOR\ WKHLU DSSOLFDWLRQ RQ WKH VDPH %LJ 'DWD %LJ'DWD $QDO\WLFV 6WDWH RI WKH $UW &KDOOHQJHV DQG )XWXUH 5HVHDUFK
FOXVWHU,QWKDWFDVHVRIWZDUHHQJLQHHUVPXVWWDFNOHZLWKWKHLU 7RSLFV ,((( 7UDQVDFWLRQV RQ ,QGXVWULDO ,QIRUPDWLFV <HDU 
GDWDVHFXULW\ 9ROXPH  ,VVXH  3DJHV    '2,
7,,

8%0.¶QG,QWHUQDWLRQDO&RQIHUHQFHRQ&RPSXWHU6FLHQFHDQG(QJLQHHULQJ