Algorithms for duplicatedocuments
Andrei BroderIBM Researchabroder@us.ibm.com
2A. Broder –Algorithms fornear-duplicate documentsFebruary 18, 2005
FingerprintingFingerprinting
(discussed last week)(discussed last week)
•
Fingerprints are short tags for larger objects.
•
Notations
•
Properties
objectsallofsetThe
=Ω
tfingerprintheoflenghtThe
=
k
{ }
functiontingfingerprinA
k
f
1,0:
→Ω
( ) ( )( ) ( )
( )
k
B A B f A f
B A B f A f
21Pr
≈≠=≠
⇒
≠
3A. Broder –Algorithms fornear-duplicate documentsFebruary 18, 2005
Fingerprinting schemesFingerprinting schemes
•
Fingerprints vs hashing
u
For hashing I want good distribution so bins will beequally filled
u
For fingerprints I don’t want any collisions = much longerhashes but the distribution does not matter!
•
Cryptographically secure:
u
MD2, MD4, MD5, SHS, etc
u
relatively slow
•
Rabin’s scheme
u
Based on polynomial arithmetic
u
Very fast (1 table lookup + 1 xor + 1 shift) /byte
u
Nice extra-properties
Search History:
Result 00 of 00
00 results for result for
p.
Clustering the Web for Algorithms for Duplicate Documents Prince Ton)
955
Reads
0
Readcasts
0
Embed Views
Recommended
More From This User
Systematic Search Algorithms
Notes
Load more
