Welcome to Scribd, the world's digital library. Read, publish, and share books and documents. See more
Download
Standard view
Full view
of .
Look up keyword
Like this
2Activity
0 of .
Results for:
No results containing your search query
P. 1
Min Hash

Min Hash

Ratings: (0)|Views: 354|Likes:
Published by Ubermensch Jo

More info:

Published by: Ubermensch Jo on Jun 21, 2010
Copyright:Attribution Non-commercial

Availability:

Read on Scribd mobile: iPhone, iPad and Android.
download as PDF, TXT or read online from Scribd
See more
See less

06/21/2010

pdf

text

original

 
1
Finding Near Duplicates
(Adapted from slides and materialfrom Rajeev Motwaniand Jeff Ullman)
Set Similarity
Set Similarity
(Jaccardmeasure)
View sets as columns of a matrix; one row foreach element in the universe. a
ij
= 1 indicatespresence of item i in set j
Example
 ji ji jiJ
CCCC)C,(Csim
UI
=
C
1
C
2
0 11 01 1
sim
J
(C
1
,C
2
) = 2/5 = 0.4
0 01 10 1
Identifying Similar Sets?
Signature Idea
HashcolumnsC
i
tosignaturesig(C
i
)
sim
 J
(C
i
,C
 j
)approximated bysim
H
(sig(C
i
),sig(C
 j
))
Naïve Approach
Sample P rows uniformly at random
Definesig(C
i
)as P bits of C
i
in sample
Problem
sparsity
would miss interesting part of columns
sample would get only 0’s in columns
Key Observation
For columns C
i
, C
 j
,four types of rows
C
i
C
 j
A
11
B
10
C
01
D
00
Overload notation:A = # of rows of type A
Claim
CBAA)C,(Csim
 jiJ
++=
Min Hashing
Randomlypermuterows
Hashh(C
i
) = index of first row with 1 incolumn C
i
SuprisingProperty
Why?
Both areA/(A+B+C)
Look down columnsC
i
, C
 j
until firstnon-Type-Drow
h(C
i
) = h(C
 j
)

type A row
( )
 jiJ ji
C,Csim)h(C)h(CP
==
Min-Hash Signatures
PickPrandom row permutations
MinHashSignature
sig(C)= list of Pindexes of first rows with 1 in column C
Similarity of signatures
Letsim
H
(sig(C
i
),sig(C
 j
))= fraction of permutations where MinHashvalues agree
ObserveE[sim
H
(sig(C
i
),sig(C
 j
))]=sim
 J
(C
i
,C
 j
)

Activity (2)

You've already reviewed this. Edit your review.
1 thousand reads
1 hundred reads

You're Reading a Free Preview

Download
scribd
/*********** DO NOT ALTER ANYTHING BELOW THIS LINE ! ************/ var s_code=s.t();if(s_code)document.write(s_code)//-->