Image Retrieval: Current

Techniques, Promising
Directions, and Open Issues
Yong Rui, Thomas Huang and Shih-Fu
Chang
Published in the Journal of Visual
Communication and Image
Representation.
Presented by: Deepak Bote

Presentation Outline

History of image retrieval Issues faced

Solution Content-based image retrieval
Feature extraction
Multidimensional indexing
Current Systems
Open issues
Conclusion

History of Image
Retrieval

Traditional text-based image search

engines

Manual annotation of images

Use text-based retrieval methods
Water lilies

E.g.
Flowers in a
pond
<Its biological
name>

Limitations of text-based
approach

Problem of image annotation

Problem of human perception

Large volumes of databases

Valid only for one language with image
retrieval this limitation should not exist
Subjectivity of human perception
Too much responsibility on the end-user

Problem of deeper (abstract) needs

Queries that cannot be described at all, but

tap into the visual features of images.

Outline

History of image retrieval Issues faced

Solution Content-based image retrieval
Feature extraction
Multidimensional indexing
Current Systems
Open issues
Conclusion

What is CBIR?

Images have rich content.

This content can be extracted as
various content features:

Mean color , Color Histogram etc

Take the responsibility of forming

the query away from the user.
Each image will now be described by
its own features.

CBIR A sample search

query

User wants to search for, say, many rose

images
He submits an existing rose picture as query.
He submits his own sketch of rose as query.

The system will extract image features for

this query.
It will compare these features with that of
other images in a database.
Relevant results will be displayed to the
user.

Sample Query

Sample CBIR
architecture

Outline

History of image retrieval Issues faced

Solution Content-based image retrieval
Feature extraction
Multidimensional indexing
Current Systems
Open issues
Conclusion

Feature Extraction

What are image features?

Primitive features
Mean color (RGB)
Color Histogram

Semantic features

General features

Color Layout, texture etc

Domain specific features

Face recognition, fingerprint matching

etc

Mean Color

Pixel Color Information: R, G, B

Mean component (R,G or B)=
Sum of that component for all pixels
Number of pixels

Pixel

Histogram

Frequency count of each individual

color
Most commonly used color feature
representation

Image

Corresponding
histogram

Color Layout

Need for Color Layout

How it works:

Global color features give too many false

positives
Divide whole image into sub-blocks
Extract features from each sub-block

Can we go one step further?

Divide into regions based on color feature
concentration
This process is called segmentation.

Example: Color layout

** Image adapted from Smith and Chang : Single Color Extraction

Texture

Texture innate property of all surfaces

Clouds, trees, bricks, hair etc

Refers to visual patterns of homogeneity

Does not result from presence of single color
Most accepted classification of textures
based on psychology studies Tamura
representation

Coarseness

Linelikeness

Contrast

Regularity

Directionality

Roughness

Segmentation issues

Considered as a difficult problem

Not reliable
Segments regions, but not objects
Different requirements from
segmentation:
Shape extraction: High Accuracy required
Layout features: Coarse segmentation
may be enough

Presentation Outline

History of image retrieval Issues faced

Solution Content-based image retrieval
Feature extraction
Multidimensional indexing
Current Systems
Open issues
Conclusion

Problem of high
dimensions
Mean Color = RGB = 3 dimensional
vector
Color Histogram = 256 dimensions
Effective storage and speedy retrieval
needed
Traditional data-structures not sufficient
R-trees, SR-Trees etc

2-dimensional space
Point A

D2

D1

3-dimensional space

Now, imagine

An N-dimensional
box!!
We want to conduct a
nearest neighbor
query.
R-trees are designed
for speedy retrieval
of results for such
purposes
Designed by
Guttmann in 1984

Presentation Outline

History of image retrieval Issues faced

Solution Content-based image retrieval
Feature extraction
Multidimensional indexing
Current Systems
Open issues
Conclusion

IBMs QBIC

QBIC Query by Image Content

First commercial CBIR system.
Model system influenced many others.
Uses color, texture, shape features
Text-based search can also be combined.
Uses R*-trees for indexing

QBIC Search by color

** Images courtesy : Yong Rao

QBIC Search by shape

** Images courtesy : Yong Rao

QBIC Query by sketch

** Images courtesy : Yong Rao

Virage

Developed by Virage inc.

Like QBIC, supports queries based
on color, layout, texture
Supports arbitrary combinations of
these features with weights attached
to each
This gives users more control over
the search process

VisualSEEk

Research prototype University of

Columbia
Mainly different because it considers
spatial relationships between objects.
Global features like mean color, color
histogram can give many false positives
Matching spatial relationships between
objects and visual features together
result in a powerful search.

ISearch my own system

ISearch my own system

ISearch my own system

Feature selection in
ISearch

Database Admin facility in

ISearch

Presentation Outline

History of image retrieval Issues faced

Solution Content-based image retrieval
Feature extraction
Multidimensional indexing
Current Systems
Open issues
Conclusion

Open issues

Gap between low level features and

high-level concepts
Human in the loop interactive
systems
Retrieval speed most research
prototypes can handle only a few
thousand images.
A reliable test-bed and measurement
criterion, please!

Presentation Outline

History of image retrieval Issues faced

Solution Content-based image retrieval
Feature extraction
Multidimensional indexing
Current Systems
Open issues
Conclusion

Conclusion

Satisfactory progress, but still

A long way to go!!

Acknowledgements

Dr. Padma Mundur

Mr. Yong Rao
Mr. Sumit Jain, Software Engineer,
KPIT Cummins, India
Mr. Ajay Joglekar, Software
Engineer, Veritas India.

References

Y. Rui, T. S. Huang, and S.-F. Chang,

Image retrieval: Current techniques,
promising directions, and open issues

S. Jain, A. Joglekar, and D. Bote,

ISearch: A Content-based Image
Retrieval (CBIR) Engine, as Bachelor
of Computer Engineering final year
thesis, Pune University, 2002

THANK YOU!!!
Questions?