ELE 492: Image Processing

Assoc. Prof. Seniha Esen Yüksel

Lecture: Human Visual System and Spatial Filtering

Hacettepe University
Department of Electrical and Electronics Engineering
Today

 Human Visual System

 Image Acquisition
 Spatial Processing

2
Illustration of Human Eye

The amount of light entering the eye is controlled by the pupil, which dilates and contracts
accordingly. The cornea and lens, whose shape is adjusted by the ciliary body, focus the
light on the retina, where receptors convert it into nerve signals that pass to the brain. A
mesh of blood vessels, the choroid, supplies the retina with oxygen and sugar.
http://psych.athabascau.ca/html/Psych402/Biotutorials/22/intro.shtml

3
Human Visual System (HVS)

 Light passes through cornea, iris, lens and form image on retina.
 Two types of photoreceptors on retina:
 Cones cluster at Fovea, detect color at bright light - photopic vision
 Rods spread at back of eye, general vision - scotopic vision

Adapted from DIP, Gonzalez

4
HVS: Weber ratio

 HVS’s sensitivity to intensity difference

differ at different background intensities.
 Weber ratio: I/I: Just noticeable intensity
difference versus background intensity.

I I+ I

Adapted from YuHen Hu

5
HVS: Obtained image

 How many different shades can you see?

Adapted from http://www.etf.ac.me/

6
HVS: Exercise
 Sensitivity to grayscale
 Create an 16x16 image with 256 shades of gray.
 Try to detect different shades
 MATLAB program
clear
k=0;
for m=1:16
for n=1:16
A(m,n)=k;
k=k+1;
end
end
pcolor(A),colormap(gray(256)),shading interp

Adapted from http://www.etf.ac.me/

7
HVS: Simultaneous Contrast

 The perceived brightness of inner circle are

different due to different background intensity
levels even they are identical.

Adapted from YuHen Hu

8
 Radiance, Luminance and Brightness
 Radiance is the total amount of energy that flows
from the light source, measured in watts (W).
 Luminance, measured in lumens (lm), gives a
measure of the amount of energy an observer
perceives from a light source.
 Light emitted from a source operating in the far
infrared region of the spectrum could have significant
energy (radiance), but an observer would hardly
perceive it; its luminance would be almost zero.
 Brightness is a subjective descriptor of light
perception that is practically impossible to
measure. 9
HVS: Mach Band Effect

 Perceived
Brightness
changes
around
strong
edges.

Adapted from DIP, Gonzalez

10
HVS: Optical Illusions

 We do not know
everything about
the eye, for instance
how the optical
illusions are formed.
 However, we can
use them in movies,
arts, graphics,
photorealism etc.

Adapted from DIP, Gonzalez

11
HVS: Temporal Vision

 Perceived spatial resolution reduces sharply

at scene change
 Flicker fusion: the basis of movie and TV

Adapted from YuHen Hu

12
Image Acquisition

Goes through
sampling (spatial)
and quantization
(gray levels)

13
Sampling in 1D

 Decide which values to take.

 Is this uniform sampling?

14
Quantization in 1D

 Quantize the signal into one of the closest 5

levels

15
Sampling and Quantization in 2D

real image sampled quantized sampled &

quantized

16
 Sampling and Quantization in 2D

pixel grid column index

row index

real image sampled quantized sampled &

quantized

17
 Sampling
Take the average
within each square.

I C  ,   I S ( r, c )
continuous image sampled image

Adapted from Richard Alan Peters II

18
 Sampling
Take the average
within each square.

I C  ,   I S ( r, c )
continuous image sampled image

Adapted from Richard Alan Peters II

19
 Sampling
Take the average
within each
square.

I C  ,   I S ( r, c )
continuous image sampled image

Adapted from Richard Alan Peters II

20
 Sampling
Take the average
within each
square.

I C  ,   I S ( r, c )
continuous image sampled image

Adapted from Richard Alan Peters II

21
Digital Image Formation: Quantization

discrete color output

continuous colors
mapped to a finite,
discrete set of colors.

continuous color input

Adapted from Richard Alan Peters II

22
Digital image generation

23
Digital Images

 Monochrome images have 1 value per

pixel; color images have 3 values per
pixel.
a grid of squares,
each of which
contains a single
color

each square is
called a pixel (for
picture element)

24
Pixels

Pixel Location: p = (r , c)
Pixel : [ p, I(p)]
Pixel Value: I(p) = I(r , c)

Adapted from Richard Alan Peters II, Vanderbilt Univ.

25
Subsampling (down a size)

26
Resampling (up a number of sizes)

27
Interpolation
 Shrinking and zooming are image resampling methods.
 Interpolation is the process of using known data to
estimate values at unknown locations

28
Nearest Neighbor Interpolation

 Causes distortion of straight edges.

29
Bilinear interpolation
 (but not really linear!)
 Uses the four neighbors to estimate the
intensity at a given location.
 Let (x,y) be the coordinate at which we want
to assign a gray level value
 Let u(x,y) be the intensity value of that
location.
 u(x,y) = ax + by + cxy + d
 The coefs. a, b, c, d are found from the 4
equations in 4 unknowns obtained from the 4
nearest neighbours of (x,y). 30
Bicubic interpolation

 u(x,y) = sum ( sum (aij xi yj ))

 Over the 16 nearest neighbors.

 Coefficients are determined from the sixteen
equations in 16 unknowns.

 Yields sharper results.

31
Image Enhancement

 The principal objective of enhancement is to

process an image so that the result is more
suitable than the original for a specific
application.
 Enhancement can be done in
 Spatial domain,
 Frequency domain.
 Common reasons for enhancement include
 Improving visual quality,
 Improving machine recognition accuracy.
32
Image Enhancement: Using image
averaging for noise reduction
 Suppose g(x, y) is a corrupted image formed
by the addition of noise, h(x, y), to a
noiseless image f (x, y); that is,
 g(x, y) = f(x, y) + h(x, y)
where at every pair of coordinates (x,y) the
noise is uncorrelated and has zero average
value.
 Objective: reduce the noise of the output
image by adding a set of noisy input images,
{gi( x, y)}.
33
Using image averaging for noise
reduction
 Averaging K different noisy images,

34
Using image averaging for noise
reduction
 As K increases, variability of the pixel values
at each location (x, y) decreases.
 g(x, y) approaches the noiseless image f (x, y)
 In order to avoid blurring and other artifacts in
the output (average) image, it is necessary
that the images be registered (i.e., spatially
aligned).

35
36
Image Enhancement

 First, we will consider point processing where

enhancement at any point depends only on
the image value at that point.
 For gray level images, we will use a
transformation function of the form
s = T(r)
 where “r” is the original pixel value and “s” is
the new value after enhancement.

37
Image Enhancement

 Image negative: s = L-1-r

 where [0, L-1] is the intensity range

38
Image Enhancement
 Log Transformations: s = c log(1+r)
 Where c is a constant and r≥0

39
Log transformations
 Maps a narrow range of low intensity values
in the input to a wider range of output levels

40
Image Enhancement
 Gamma transformation s = c rγ
 where c and γ are positive constants

41
Gamma transformations
 What can you say about the resulting image?
(darker, lighter etc?)
 If γ = 1
 If γ = 2.5
 γ = 0.2

42
Gamma transformations
 What can you say about the resulting image?
(darker, lighter etc?)

>1
Compresses dark values
Expands bright values

<1
Expandsdark values
Compresses bright values

43
Gamma
correction

44
 Gamma correction

Many TV’s
have this
correction
now

45
Image Enhancement
 Contrast stretching: expand the range of
intensity levels in an image so that it spans
the full intensity range of the display device.

46
Contrast stretching

47
Intensity level slicing
 Two options
 Do a black& white, highlight the regions of interest
 Just brighten or darken the desired range of
intensities

48
Intensity level slicing

49
Bit plane slicing

 Digital numbers are composed of bits.

 Intensity of each pixel in a 256-level gray
scale image is composed of 8 bits (one byte)
 If we plot the bits as binary images, we can
see the contribution of each bit

50
Bit plane slicing

51
Bit plane slicing

52
The Image Histogram

Occurrence
(# of pixels)

Gray Level

 H(k) specifies the # of pixels with gray-value k

 Let N be the number of pixels: N   H (k )
k
 P(k) = H(k)/N defines the normalized histogram
k
 C (k )   H (i) defines the accumulated histogram
i 1
15000

10000

5000
Histogram

0
1 2 3 4 5 6 7 8 9 10
gray level

0.25

0.2

0.15

Normalized Histogram
0.1

0.05

0
1 2 3 4 5 6 7 8 9 10
gray level

0.8

0.6
Accumulated Histogram
0.4

0.2

0
1 2 3 4 5 6 7 8 9 10
gray level
Histogram processing

55
Histogram

Note: Images with uniformly

distributed histograms have
higher contrast and high
dynamic range

56
 P(I)
0.1
Original image

P(I)

Decreasing 0.1

contrast

P(I)
0.1
Increasing average

I
Histogram Equalization

 Visual discrimination between objects depends on the their

gray-level separation.
 How can we improve discrimination after image has been
acquired?

Hard to discriminate Doesn’t help This is better

What about this?

Histogram Equalization

 For a better visual discrimination of an image we would like

to re-assign gray-levels so that gray-level resource will be
optimally assigned.
 Our goal: finding a gray-level transformation M(v)
such that:
 The histogram Hb is as flat as possible.
 The order of gray-levels is maintained.
 The histogram bars are not fragmented.
Hist. Eq.

0 1 2 3 4 5 6 7 8 9 10 11 old
0 0 0 1 1 1 3 5 8 9 9 11 new
 Hist. Eq.
30 (30) 30

25 25
(22)
(20) (21)
20 20

Original15 15
Goal
10 10
(5)
5 (4) (4) (4) 5
(3) (3) (3)
(1)
0 0
0 1 2 3 4 5 6 7 8 9 10 11 0 1 2 3 4 5 6 7 8 9 10 11

30

25

20

15 Result
0 1 2 3 4 5 6 7 8 9 10 11 old
10
0 0 0 1 1 1 3 5 8 9 9 11 new
5

0
0 1 2 3 4 5 6 7 8 9 10 11
Hist. Eq.: Example 1

3500 3500

3000 3000

2500 2500

2000 2000

1500 1500

1000 1000

500 500

0 0
0 50 100 150 200 250 0 50 100 150 200 250

Original Equalized
Hist. Eq.: Example 2

3000 3000
2500 2500
2000 2000
1500 1500
1000 1000
500 500
0 0
0 50 100 150 200 250 0 50 100 150 200 250

Original Equalized
Hist. Eq.: Example 3
Histogram Equalization

65
Histogram

 The Histogram of a digital image is a function :

h(rk )  nk

where rk is the kth gray level

nk is the number of pixels having gray level rk
Histogram

 Example:

0 0 2 2
1 1 2 5
1 1 3 4
2 2 3 4
Normalized Histogram
 Normally, we normalize h(rk) by n, where n is the total number of
pixels
h(rk ) nk
p(rk )  
 So, we have
n n
L 1 L 1
nk

k 0
p(rk )  
k 0 n
1

 p(rk) can be sought of as the probability of a pixel to have a certain

value rk
Normalized Histogram

 Example: n=16

0 0 2 2
1 1 2 5
1 1 3 4
2 2 3 4
Histogram Equalization

 Define a transformation s= T(r)

with
r
T (r )   pr ( w)dw
0

where pr(r) is the probability histogram of

image r
Histogram Equalization

 Now lets calculate ps(s)

 Let s=T(r) be continuous and differentiable

dr
p s ( s )  pr ( r )
ds
r Leibniz rule: The
ds d d
 T (r )   pr ( w)dw derivative of an integral
with respect to its upper
dr dr dr 0 limit is the integrand
evaluated at the limit
ds
 pr ( r )
dr What does this tell us about
ps(s)?
Histogram Equalization

 So,

dr 1

Then ds pr (r )

1
p s ( s )  pr ( r )  1
pr ( r )
r
Which means that using the transformation T (r )   p (w)dw
r
the resulting probability is uniform independent 0

of the original image

ps ( s )  1
Histogram Equalization: in discrete form
 Find the normalized image histogram pr(rk)

 where MN is the total number of pixels in the image, and

nk denotes the number of pixels that have intensity rk

 Compute the transformation

 where L is the number of possible intensity levels in the

image (e.g., 256 for an 8-bit image
73
Histogram Equalization Summary

74
Example

75
Histogram Equalization

Transformation Functions
Ex: histogram values

77
Ex:

78
Ex:

79
Try
1. Compute the histogram for the following example
2. Obtain the histogram equalization curve
3. Perform Histogram Equalization
4. Plot the new image and the new histogram
5. What can you say about it?

0 0 2 2
1 1 2 5
1 1 3 4
2 2 3 4
Some examples from the web

Sujin Philip @ Univ. of Utah

81
 Some examples from the web

Nic Snailum at University of East London

82
Histogram Equalization
Is it always a good idea?
Histogram Equalization
Histogram Matching

85
Histogram Matching (specification)

 Enhancement based on a uniform histogram is not the

best approach
 It is useful sometimes to specify the shape of the histogram that we wish
to have
 Generate a processed image that has a specified
histogram
 Suitable for interactive image enhancement
 Difficulty--build a meaningful histogram

86
Histogram matching
 Transforms an image A so that its histogram will
match that of another image B.
 Usage: before comparing two images of the same
scene (change detection) acquired under different
lighting conditions or different camera parameters.

source target mapped source

Histogram Matching

1. Equalize the histogramr of the input image

T1(r) = (L-1) ò p1(w)dw
0

2. Histogram equalize the desired output histogram

r
T2 (r) = (L-1) ò p2 (w)dw
0
3. Histogram matching can be carried out by the
following point operation:
-1
s= T(r) = T (T1(r))
2
Histogram Matching (specification)
 pr(r): estimated from the given input image
with gray values r.
 pz(z): the specified pdf that we wish the
output image to have.
r
s  T (r )   pr ( w)dw
0
z
G ( z )   p z (t )dt  s
0

G ( z )  T (r )
z  G 1 ( s )  G 1 T (r )
 An image with a specified pdf can be
obtained from an input image by using the
following procedure:
 Obtain the transformation function T(r)
 Obtain the transformation function G(z)
 Obtain the inverse transfer function G-1
 Obtain the output image by applying this
transformation to all of the pixels in the input
image
 The result of this procedure will be an image
whose gray levels, z, have the specified pdf
pz(z).
 The discrete formulation:

k k nj
sk  T (rk )   pr (rj )   k  0,1,2,..., L  1
j 0 j 0 n
k
vk  G ( z k )   p z ( z i )  s k k  0,1,2,..., L  1
i 0

z k  G 1 T (rk ) k  0,1,2,..., L  1
z k  G 1 sk  k  0,1,2,..., L  1

vk  s k
v=s
Example

Got the s values from the previous example

93
Histogram Matching: Discussion
 Histogram matching produces the optimal monotonic
mapping so that the resulting histogram will be as close
as possible to the target histogram.
 This does not necessarily imply similar images.

hist. match

result
Slide credit: Y. Hel-Or
 In cases where corresponding colors between
images are not “consistent” this mapping may fail:

from: S. Kagarlitsky: M.Sc. thesis 2010.

Local Enhancement

 Instead of calculating the histogram for the

whole image and then do histogram
equalization,
 First divide the image into blocks
 Perform histogram equalization on each block
Local Histogram Equalization

97
Local Histogram Equalization
Histogram Statistics for Image
Enhancement
•Instead of using histogram directly, we can use instead some
statistical parameters obtainable directly from the histogram.

•n-th moment of r about its mean:

L 1
n (r )   (ri  m) n p(ri )
i 0

where m is the mean value of r:

L 1
m   ri p(ri )
i 0
•µ0=1, µ1=0, µ2=σ2(r) : variance.

99
Histogram Statistics for Image
Enhancement

100
•The global mean and variance are measured over an entire
image and are useful for gross adjustments of overall intensity
and contrast.

•Local mean and variance can be used to make adjustment in a

predefined region about each pixel.

•Let (x,y) be the coordinates of a pixel in an image and let Sxy

denote a neighborhood of specified size, centered at (x,y). The
mean and variance can be computed as:

mS xy  r s ,t p(rs ,t )  S2xy  
( s ,t )S xy
(rs ,t  mS ) 2 p(rs ,t )
xy
( s ,t )S xy

•Where rs,t is the gray level at coordinates (s,t) in the

neighborhood and p(rs,t) is the neighborhood normalized
histogram corresponding to that value of gray level.
Local mean is a measure of average gray level in neighborhood
Sxy and the variance is a measure of contrast in that
neighborhood.

Filament at the back is not visible. Local enhancement by

contrast manipulation is an ideal approach.
Dark areas should be enhanced while leaving the light areas as
unchanged.
A measure of whether an area is relatively light or dark is to
compare the local average gray level m to the average image
S xy
gray level M G .

The pixel (x,y) is selected as a candidate for processing if

mS xy  k0 M G where 0 ≤ k0 ≤ 1.
Since we also need to enhance low contrast areas, candidate pixels
can be selected by
 S  k2 DG
xy where DG is the global standard deviation and k2
is a positive constant.

k2 will be greater than 1 if we are interested in enhancing light

areas and less than 1 for dark areas.
 We need to restrict the lowest values also.

k1DG ≤ S xy
with k1 < k2
A pixel at (x,y) that meets all the conditions for local
enhancement is processed simply by multiplying it by a
specified constant E to increase the value of its gray level
relative to the rest of the image.

The values of pixels that do not meet the enhancement

conditions are left unchanged.
As a summary:
E  f ( x, y) if mS xy  k 0 M G and k1DG   S xy  k2 DG

g ( x, y )   f ( x, y ) otherwise

 What can you learn from a histogram?
Image Statistics
EI    I (i, j )   k H k    k Pk 
1 1
• The image mean:
N i, j N k k

• Generally: Eg k    g k  Pk  k: gray levels

k

• The image s.t.d. :  

 I   E I  EI 2  E I 2   E 2 I 
0.25

0.2
 
where E I 2   k 2 Pk 
k

0.15

0.1

0.05

0
1 2 3 4 5 6 7 8 9 10
gray level
 Image Entropy
EntropyI    Pk  log Pk 
k

• The image entropy specifies the uncertainty in the image

values.
• Measures the averaged amount of information required to
encode the image values.
0.7

0.6

0.5

0.4
entropy(P)

0.3

0.2

0.1

0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
P
Entropy of a 2 values variable
• An infrequent event provides more information than a
frequent event
• Entropy is a measure of histogram dispersion

1000

800

600

400

200

0 50 100 150 200 250

entropy=7.4635

10000

9000

8000

7000

6000

5000

4000

3000

2000

1000

0 50 100 150 200 250

entropy=0
 Histogram Usage
• Digitizing parameters
• Measuring image properties:
– Average
– Variance
– Entropy
– Contrast
– Area (for a given gray-level range)
• Threshold selection
• Image distance
• Image Enhancement
– Histogram equalization
– Histogram stretching
– Histogram matching
 Example: Auto-Focus
• In some optical equipment (e.g. slide projectors) inappropriate
lens position creates a blurred (“out-of-focus”) image
• We would like to automatically adjust the lens
• How can we measure the amount of blurring?
 4000

3000

2000

1000

0
0 10 20 30 40 50 60

4000

3000

2000

1000

0
0 10 20 30 40 50 60

• Image mean is not affected by blurring

• Image s.t.d. (entropy) is decreased by blurring
• Algorithm: Adjust lens according the changes in the histogram
s.t.d.
 Thresholding
knew
F(k)
255

kold
255
Threshold value
 Threshold Selection
Original Image Binary Image

Threshold too low Threshold too high

 Segmentation using Thresholding
Original Histogram
1500

1000

500

0
0 100 200

50 75

Threshold = 50 Threshold = 75
Segmentation using Thresholding
Original Histogram
1500

1000

500

0
0 100 200

21

Threshold = 21
 Adaptive Thresholding
• Thresholding is space
variant.
• How can we choose
the the local threshold
values?
 Adaptive Histogram
• In many cases histograms are needed for local areas in an
image
• Examples:
– Pattern detection
– adaptive enhancement
– adaptive thresholding
– tracking
 Histogram based image distance
• Problem: Given two images A and B whose (normalized)
histogram are PA and PB define the distance D(A,B) between
the images.

• Example Usage:
– Tracking
– Image retrieval
– Registration
– Detection
– Many more ...

Porkili 05
 Option 1: Minkowski Distance
1/ p
 p
D p  A, B    PA (k )  PB (k ) 
k 

• Problem: distance may not reflect the perceived

dissimilarity:

<
 Option 2: Kullback-Leibler (KL) Distance

PA k 
DKL  A || B    PA k  log
k PB k 

• Measures the amount of added information needed to encode

image A based on the histogram of image B.
• Non-symmetric: DKL(A,B)DKL(B,A)
• Suffers from the same drawback of the Minkowski distance.
 Option 3: The Earth Mover Distance (EMD)

• Suggested by Rubner & Tomasi 98

• Defines as the minimum amount of “work” needed to
transform histogram HA towards HB
• The term dij defines the “ground distance” between gray-
levels i and j.
• The term F={fij} is an admissible flow from HA(i) to HB(j)

>
 Option 3: The Earth Mover Distance (EMD)

From: Pete Barnum

 Option 3: The Earth Mover Distance (EMD)

From: Pete Barnum

 Option 3: The Earth Mover Distance (EMD)

From: Pete Barnum

 Option 3: The Earth Mover Distance (EMD)

(amount moved)

From: Pete Barnum

 Option 3: The Earth Mover Distance (EMD)

work=(amount moved) * (distance

moved)

From: Pete Barnum

 Option 3: The Earth Mover Distance (EMD)

DEMD ( A, B)  min  f ij  d ij
F
i j

s.t. f ij  0 ; PB k    f ik ; PA k    f ki
i i

• Constraints:
– Move earth only from A to B
– After move PA will be equal to PB
– Cannot send more “earth” than there is
• Can be solved using Linear Programming
• Can be applied in high dim. histograms (color).
 Special case: EMD in 1D

• Define CA and CB as the accumulated histograms of image A

and B respectively:

DEMD  A, B    C A k   CB k 
k

PA CA

PB CB

CA-
CB
 What can you not do with histograms?
P(I) P(I)
1
1

0.5

I I
H(I) H(I)
0.1
0.1

I I
Pixel permutation of
the left image

• The image histogram does not fully represent the image

• It does not have spatial information