Image-based Retrieval using a Camera Phone

Sam S. Tsai*, David M. Chen*, J atinder P. Singh

, and Bernd Girod*

*Information Systems Laboratory, Stanford University,

Deutsche Telekom Laboratories

Mobile Image Matching Application
Feature-based
I M t hi
F t
Wireless
Network
Query
Image Matching
Features
Information
S dx
S
Scalable Vocabulary Tree (SVT)
[Nister and Stewenius, CVPR 2006]
Select Top
Match
S dy
S |dx|
S |dy|
Speeded Up Robust Features (SURF)
[Bay et al., ECCV 2006]
Geometric
Consistency
Check
Rate-efficient Image Retrieval Mobile CD/DVD
Robust MultiviewSVTs
Send Feature Mode
Camera Client Server Coding
SURF
Extraction
Image
Matching
Rate-efficient Image Retrieval Mobile CD/DVD
Recognition System
Robust Multiview SVTs
Front View SVT
Project
Client Nokia N95
ARM11 332MHz 64 MB RAM
Network EDGE <236 kbps
S X 2GH 4 GB RAM
Transform Quantization Entropy Coding
Scalable
Vocabulary
Tree (SVT)
Geometric
Consistency
Check
Interest
Point
Detection
SURF
Descriptor
Generation
Front Top Bottom Right Left
Top View SVT
Server Xeon 2GHz 4 GB RAM
Database 10k+CD entries
9k+DVD entries
Reference
D. Chen, S. S. Tsai, V. Chandrasekhar, G. Takacs, J . Singh, and B. Girod,
"Robust image retrieval usingmultiviewscalable vocabularytrees", SPIE
Send ImageMode
Select Top
Match
Select Top
Match
Select Top
Match
Select Top
Match
Select Top
Match
Geometric
Consistency Check
Robust image retrieval using multiview scalable vocabularytrees , SPIE
Visual Communications and Image Processing (VCIP), San J ose,
California, J anuary2009.
V. Chandrasekhar, G. Takacs, D. Chen, S. S. Tsai, J . Singh, and B. Girod,
"Transformcoding of image feature descriptors", SPIE Visual
Communcations and Image Processing (VCIP), San J ose, California,
J anuary2009.
S. S. Tsai, D. Chen, J . Singh, and B. Girod, "Rate-efficient, real-time CD
cover recognition on a camera-phone", ACM Multimedia (ACM MM),
Vancouver, British Columbia, Canada, October 2008.
Send Image Mode
Camera Client Server
SURF
Extraction
Image
Matching
Mode Size of query
Feature 3-6 KB
Image 30-50 KB
200 -400 feature per images
Inverted Index Coding for Scalable Image Matching
David Chen*, Sam Tsai*, Vijay Chandrasekhar*, Gabriel Takacs*,
Ramakrishna Vedantham

Radek Grzeszczuk

and Bernd Girod* Ramakrishna Vedantham

, Radek Grzeszczuk

, and Bernd Girod

*Department of Electrical Engineering, Stanford University

Nokia Research Center, Palo Alto

Mobile Image Search Inverted Index Coding
Wireless
Network
Query
Coding of Image Identifiers Coding of Fractional Counts
Differential coding of image identifiers
d
11
= i
11
, d
12
= i
12
i
11
, , d
1N
= i
1N
- i
1(N -1)
d
21
= i
21
, d
22
= i
22
i
21
, , d
2N
= i
2N
- i
2(N -1)
Differential coding of image identifiers
d
11
= i
11
, d
12
= i
12
i
11
, , d
1N
= i
1N
- i
1(N -1)
d
21
= i
21
, d
22
= i
22
i
21
, , d
2N
= i
2N
- i
2(N -1)
1
2
1
2
1
2
Fractional counts from soft binning Fractional counts from soft binning
|
|
.
|

\
|
=
2
2
1
1
exp
1
o
d
C
w
d
1
d
2
Information

Entropy coding of consecutive differences Entropy coding of consecutive differences

Very non-uniform and
peaky distribution suggests
that variable-length coding
is much more efficient than
fi d l th di
|
|
.
|

\
|
=
2
2
2
2
exp
1
o
d
C
w
|
|
.
|

\
|
=
2
2
3
3
exp
1
o
d
C
w
2
d
3
Lloyd-Max quantization of fractional counts Lloyd-Max quantization of fractional counts
Vocabulary Tree and Inverted Index
Experimental Results
fixed-length coding:
Arithmetic coding
Carryover coding
RBUC coding
Vocabulary Tree and Inverted Index
Experimental Results
1 2 K
Vocabulary Tree
i
11
i
12
i
1N1
c c c
Uncompressed index
4 bytes for image identifier
4 bytes for fractional count

sumof list lengths

Uncompressed index
4 bytes for image identifier
4 bytes for fractional count

sumof list lengths

K
i
N
~5x savings
~5 sec savings
Inverted Index
i
21
i
22
i
2N2
c
11
c
12
c
1N1
c
21
c
22
c
2N2
sum of list lengths .
=
total memory usage
example: 1 million image
database requires 10 GB
sum of list lengths .
=
total memory usage
example: 1 million image
database requires 10 GB

= i
i
1
Database of 1 million CD/DVD/book cover images