Lecture 12

Content-Based Image Retrieval

Slides by:
Deepak Bote, Xiaoguang Feng,
David A. Forsyth, Clark F. Olson,
Yossi Rubner, Linda G. Shapiro
 Image databases

Large collections of image (and video) occur in many

applications:
• Stock photos and footage
• Military
• World-wide web
• Medical imaging
• Space exploration
• Surveillance
• Many others…

2
 Image databases

Most collections of images and videos are not image

databases.
• No DBMS manages the data
• No facility for complex queries is available

This is a rapidly expanding area of interest in computer

vision:
• How can we find a particular image that we are interested in?
• Can we locate images that meet some description?
• How can we organize the collection meaningfully?
• Can we extract new information by exploring the collection
automatically?

3
History

4
 Image retrieval by annotation
Traditional text-based image search engines
Manual annotation of images
Use text-based retrieval methods

Water lilies
E.g.

Flowers in a pond

<Its biological name>

5
 Limitations of annotation
Problem of image annotation
Large volumes of databases
Valid only for one language – with image retrieval this
limitation should not exist
Problem of human perception
Subjectivity of human perception
Too much responsibility on the end-user
Problem of deeper (abstract) needs
Queries that cannot be described at all, but tap into the
visual features of images

6
 Image retrieval
Image have rich content that can be used for retrieval!

The problem that has been examined the most is: How can I
find images that meet some description or are like an example?

This is often called “content-based image retrieval” (CBIR).

Image content is used, rather than metadata.

Using example images:

• Takes the responsibility of forming the query away from the user.
• Allows each image to be described by its own features.
Mean color, histogram, etc.

7
 Measures

No system is perfect. Usually measured using two criteria:

• Recall: percentage of correct items found
• Precision: percentage of items found that are correct

Which is more important?

8
 User interaction
Due to the imperfections in current
methods, most systems have a user
“in the loop”.

Retrieve-refine-retrieve cycle

Y. Rubner, C. Tomasi, L.J. Guibas,

“A metric for distributions with
applications to image databases”,
Intl. Conf. Computer Vision. ©
1998, IEEE.
 Features for retrieval
What features are used for
retrieving images?
• Color
- Mean
- Overall distribution
- Relative locations
• Texture
- Linear filters
- Textures of textures (!?)
• Shape
- Sketches
- Segmented objects
• Others

10
 Color histograms

An early (and still popular)

similarity measurement uses
color histograms.
• The RGB (or another) color
space is discretized into bins.
• For each bin, a count is
maintained on the number of
pixels that fall into the bin
(since they have the right color)

Once constructed, the

histograms can be compared
using several metrics.
UC Berkeley Digital Library Project.
http://elib.cs.berkeley.edu/

11
 Color histograms

The QBIC system (IBM) was the first commercial

system created.

It uses color, texture, shape, location, and keywords.

The QBIC color histogram distance is:

dhist(I,Q) = (h(I) - h(Q))T A (h(I) - h(Q))

• h(I) is a K-bin histogram of a database image

• h(Q) is a K-bin histogram of the query image
• A is a K x K similarity matrix

12
QBIC – Search by color

** Images courtesy : Yong Rao

13
QBIC – Search by shape

** Images courtesy : Yong Rao

14
QBIC – Query by sketch

** Images courtesy : Yong Rao

15
Histogram distances

16
Histogram distances

17
Earth mover’s distance

18
Earth mover’s distance

19
 Layout templates
Color histograms have no
information about pixel locations.
This yields false positives.

We can add spatial information by

using a template of the image.

Example: Blue at top (sky), white

below (snowy mountains), dark at
bottom (mountains or lake)

Simplest implementation would

P. Lipson, E. Grimson, P, Sinha, “Configuration
divide image into regular blocks. based scene classification and image indexing”,
IEEE Conf. on Computer Vision and Pattern
Recognition. © 1997 IEEE.

20
 Color layout

** Image adapted from Smith and Chang : Single Color Extraction and Image Query
21
 Color correlograms
Pick any pixel p1 of color Ci in the image I, at distance k away
from p1 pick another pixel p2, what is the probability that p2 is
also of color Ci?

Red ? Jing Huang, S. Ravi

Kumar, Mandar Mitra,
Wei-Jing Zhu, and Ramin
k P2 Zabih. Image indexing
using color
correlograms. In IEEE
P1 Conference on Computer
Vision and Pattern
Recognition, pages 762--
Image: I 768, 1997.

22
 Color correlograms

The auto-correlogram of image I for color Ci , distance k:

 ( I )  Pr[ p2  I Ci | p1  I Ci , | p1  p2 | k ]
(k )
Ci

Integrates both color and spatial information.

Efficient algorithms exist to compute this.

23
Color correlograms

Jing Huang, S. Ravi

Kumar, Mandar Mitra,
Wei-Jing Zhu, and Ramin
Zabih. Image indexing
using color
correlograms. In IEEE
Conference on Computer
Vision and Pattern
Recognition, pages 762--
768, 1997.

24
 Retrieval with texture
Can compute local texture by:
• Filtering with various kernels
• Taking the signature at each pixel
as the filter response at the pixel
for each kernel
How should texture signatures be
used:
• Histograms
- Image motion causes problems
- Can consider various motions
separately, but more expensive
• Texture segmentation
- Compare average texture
signature within regions
25
Retrieval with texture

Yossi Rubner, Carlo

Tomasi. Texture-Based
Image Retrieval Without
Segmentation. IEEE
International Conference
on Computer Vision,
Kerkyra, Greece,
September 1999, pages
1018-1024.

26
 Retrieval with texture

Yossi Rubner, Carlo

Tomasi. Texture-Based
Image Retrieval Without
Segmentation. IEEE
International Conference
on Computer Vision,
Kerkyra, Greece,
September 1999, pages
1018-1024.

27
Texture of textures

Positive examples (top) and

retrieved results (bottom)

K. Tieu, P. Viola, “Boosting image

retrieval”, IEEE Conf. on Computer
Vision and Pattern Recognition. ©
2000 IEEE.

28
Texture of textures

K. Tieu, P. Viola, “Boosting image

retrieval”, IEEE Conf. on Computer
Vision and Pattern Recognition. ©
2000 IEEE.

29
 Texture of textures

The best filters to use

for each class are
learned using boosting.

K. Tieu, P. Viola, “Boosting image

retrieval”, IEEE Conf. on Computer
Vision and Pattern Recognition. ©
2000 IEEE.

30
Texture of textures

31
 Retrieval using shape
We can do even better if we
include shape in the query.

How do we know what shapes

are in each image in the
database?
• Segmentation

Blobworld (UC-Berkeley) uses

color, texture, shape, and
location to find good matches.

Background can be disregarded,

if desired.

UC Berkeley Digital Library Project. http://elib.cs.berkeley.edu/

32
 Shape measures
Global shape measures However, global measures
include: are not ideal in many
• Boundary length situations.
• Area enclosed
• Boundary curvature
(overall or histogram)
• Moments
• Projections onto axes
• Tangent angle histogram

33