You are on page 1of 24

Mikio Braun Realtime Personalization and Recommendation with Stream Mining streamdrill

Realtime personalization
and recommendation with
stream mining
Mikio Braun
@mikiobraun
streamdrill
Berlin Buzzwords 2014
Mikio Braun Realtime Personalization and Recommendation with Stream Mining streamdrill
Reacting to user behaior
Mikio Braun Realtime Personalization and Recommendation with Stream Mining streamdrill
!ow"
Mikio Braun Realtime Personalization and Recommendation with Stream Mining streamdrill
Mikio Braun Realtime Personalization and Recommendation with Stream Mining streamdrill
Making #rogress
Mikio Braun Realtime Personalization and Recommendation with Stream Mining streamdrill
$etting rid o% e&actness
Mikio Braun Realtime Personalization and Recommendation with Stream Mining streamdrill
'm#ortant users
Mikio Braun Realtime Personalization and Recommendation with Stream Mining streamdrill
Stream Mining to the rescue

Stream mining algorithms(

answer )stream *ueries+ with ,nite resources

-.#ical e&am#les(

how often does an item a##ear in a stream"

how many distinct elements are in the stream"

what are the top-k most frequent items"


Continuous Stream of Data
Bounded Resource
Analyzer
Stream Queries
Mikio Braun Realtime Personalization and Recommendation with Stream Mining streamdrill
!ea. !itters /a0k0a0 -o#1k2

3ount actiities oer large item sets /millions4


een more4 e0g0 'P addresses4 -witter users2

'nterested in most actie elements onl.0


Metwally, Agrawal, Abbadi, Efficient computation of Frequent and Top-k Elements in Data Streams, Internation Conference
on Database Theory, 2!
frank
paul
an
feli!
leo
ale!
"#
"$
%
#
&
$
'i!ed tables of counts
Case "( element already in data base
Case $( new element
paul ")$ "$ "&
nico ale! $
nico &
Mikio Braun Realtime Personalization and Recommendation with Stream Mining streamdrill
3ount1Min Sketches

Summarize histograms oer large %eature


sets

5ike bloom ,lters4 but better

6uer.( -ake minimum oer all hash %unctions


* * & *
" " * $
* $ * *
* & # $
* # & $
$ ) # *
" & + &
* $ * %
m bins
n different
,as, functions
-pdates for new entry
Query result( "
./ Cormode and S/ Mut,ukris,nan/ "n impro#ed data stream summary$ The count-min sketch and its applications%
0A123 $**), 4/ Algorit,m ##5"6( #%7+# 5$**#6 /
Mikio Braun Realtime Personalization and Recommendation with Stream Mining streamdrill
!ea. !itters oer -ime1
7indow
1ime
DB

8ee# old count


data #eriodicall.

9lternatie(
:&#onential deca.
Mikio Braun Realtime Personalization and Recommendation with Stream Mining streamdrill
'nde&ing 3olumns
page referer IP score
8inde!/,tml google "&/$)/&$/"$ "*
8post8"$& facebook )&/"&/)&/9+ :
8inde!/,tml twitter 9/9$/$&/) 9
8about/,tml google "&/$)/&$/"$ &
/// /// /// //
Mikio Braun Realtime Personalization and Recommendation with Stream Mining streamdrill
Storing ;ata in -rends
Mikio Braun Realtime Personalization and Recommendation with Stream Mining streamdrill
Storing ;ata in -rends
Mikio Braun Realtime Personalization and Recommendation with Stream Mining streamdrill
Streamdrill

Realtime 9nal.sis Solutions

3ore :ngine(

!ea. !itters counting < e&#onential deca.

Instant counts = to#1k results oer time windows

'n1Memor.

written in Scala

Modules

Pro,ling and -rending

Recommendations

3ount ;istinct
Mikio Braun Realtime Personalization and Recommendation with Stream Mining streamdrill
Realtime >ser Pro,les
Mikio Braun Realtime Personalization and Recommendation with Stream Mining streamdrill
Realtime >ser Pro,les
Mikio Braun Realtime Personalization and Recommendation with Stream Mining streamdrill
Realtime >ser Pro,les
Mikio Braun Realtime Personalization and Recommendation with Stream Mining streamdrill
Realtime >ser Pro,les
Mikio Braun Realtime Personalization and Recommendation with Stream Mining streamdrill
Realtime user #ro,les

Process 10k eents ? second on one


machine

-rack about 1 Million counts #er 1 $B

Shard b. user %or higher accurac.


Mikio Braun Realtime Personalization and Recommendation with Stream Mining streamdrill
Realtime Recommendation
Mikio Braun Realtime Personalization and Recommendation with Stream Mining streamdrill
Realtime Recommendation
Mikio Braun Realtime Personalization and Recommendation with Stream Mining streamdrill
Realtime Recommendation

Pi#e in eents4 get recommendations

Seed b. sam#ling #ast clicks

9utomaticall. ada#ts oer time

@umber o% alternaties /user based4 item


based4 trends4 categories2
Mikio Braun Realtime Personalization and Recommendation with Stream Mining streamdrill
Summar.

;itch e&actness A 9##ro&imate with


stream mining

-rends to store all kinds o% counting


structures

React to realtime user behaior with


managable resources