What is Image Segmentation?

There are many definitions. Three common ones are

 Image Segmentation is the process of isolating
objects of interest from the rest of the scene.
(Castleman )

 Image segmentation is the process of

partitioning ( 分 割 ) an image into non-
intersecting region such that each region is
homogeneous ( 相 似 的 ) and the union of no two
adjacent regions is homogeneous. (Pal )

1
What is Image Segmentation?
 Image segmentation is to divide an image into
parts that have a strong correlation with
objects or areas of the real world contained in
the image. (Watt )

 In brief, segmentation is to subdivide an image

into its constituent ( 组成的 ) regions or objects.
― Segmentation should stop when the objects of
interest in an application have been isolated.

2
Image Processing Flow based on
Image Segmentation

Image Feature
Classification
Segmentation Extraction

Input Object Feature Object

Image Image Vector Type

3
Why is it difficult?
In general autonomous ( 自 主 的 )
segmentation is one of the most difficult
tasks in image processing. It is difficult beca
use of many reasons. Here are some typical
obstacles:
 Non-uniform illumination
 No control of the environment
 Inadequate model of the object of interest
 Noise

 etc. 4
Principal approaches
 Segmentation methods can be divided into three groups
according to the dominant features they employ.
 Segmentation based on global knowledge about an image;
― The knowledge is usually represented by a histogram of
image features.
 Edge-based segmentations;
― Utilizing edge detection processes to find a closed
boundary so that an inside and an outside can be
defined.
 Region-based segmentations;
― This techniques proceed by dividing the image into
regions that exhibit similar properties.
5
2 basis properties of intensity values

 Segmentation algorithms generally are based

on one of 2 basis properties of intensity val
ues
 Discontinuity

― topartition an image based on abrupt ( 突 然

的 ) changes in intensity (such as edges)
 Similarity

― to partition an image into regions that are sim

ilar according to a set of predefined criteria.

6
Detection of Discontinuities
 detect the three basic types of gray level
discontinuities
 points , lines , edges

 the common way is to run a mask through

the image

7
Contents
 Thresholding
 Point Detection
 Line Detection
 Edge-based Segmentation
 Region-based Segmentation

8
7.1 Thresholding
 Thresholding is a labeling operation on a
gray scale image that distinguishes pixels of
a higher intensity from pixels with a lower
intensity value.
 The output of thresholding usually is a
binary image.
 This technique is particularly useful for
scenes which contain solid objects on a
uniform contrasting background.
9
Classification of Thresholding
 Thresholding can be viewed as an operation
that involves tests against a function T of t
he form:

T  T  x , y, p  x, y  , f  x , y  

 where p(x,y) denotes some local property of this

point.

10
Classification of Thresholding
 When T depends on
 only f(x,y) : only on gray-level values

􀀾  Global thresholding
 both f(x,y) and p(x,y) : on gray-level values and
its neighbors

 Local thresholding
x and y (in addition)

 Dynamic thresholding
T  T  x , y, p  x, y  , f  x , y  
11
Basic Global Thresholding
See “basic_global_thre.m”

Original image Histogram

Solution
Solution:: use
use TT midway
midway between
between the
the max
max and
and min
min
gray
gray levels
levels
12
Basic Global Thresholding
 Let light objects in dark background
 To extract the objects:
 Select a “T” that separates the objects from th
e background
 i.e. any (x,y) for which f(x,y)>T is an object poi
nt.

 A thresholded image:
1 if f  x , y   T background
g  x, y   
0 if f  x , y   T foreground
13
Heuristic Global Thresholding
1. Select an initial estimate for T.

2. Segment the image using T. This will produce two

groups of pixels: G1 consisting of all pixels with gr
ay level values < T and G2 consisting of pixels wit
h gray level values  T

3. Compute the average gray level values μ1 and μ2 f

or the pixels in regions G1 and G2

4. Compute a new threshold value: T=0.5(μ1+μ2)

5. Repeat steps 2 through 4 until the difference in
T in successive iterations is smaller than a predef
ined parameter T0. see”h….m” 14
Basic Adaptive Thresholding
 subdivide original image into small areas.
 utilize a different threshold to segment eac
h subimages.
 since the threshold used for each pixel dep
ends on the location of the pixel in terms o
f the subimages, this type of thresholding i
s adaptive.

See “Adaptive_threshold.m”
15
Multilevel Thresholding

 a point (x,y) belongs to

 an object class, if T1 < f(x,y) ≤ T2

 another object class, if f(x,y) > T2

 to background if f(x,y) ≤ T1 16
 The histogram of an image containing two pr
incipal brightness regions can be considered
an estimate of the brightness probability de
nsity function p(z).
 p(z) is the sum (or mixture) of two unimodal ( 单
峰的 ) densities (one for light, one for dark regio
ns).

p  z   P1 p1  z   P2 p2  z 
P1  P2  1

17
Optimal Thresholding
 If the form of the
densities is known
or assumed, in
terms of minimum
error, determining
an optimal
p  z   P1 p1  z   P2 p2  z 
threshold for
 P2  1
P1segmenting the
image is possible.

18
Optimal Thresholding
 Probability of erroneously

19
Optimal Thresholding
 Minimum error
 Differentiating ( 微 分 ) E(T) with respect to T
(using Leibniz’s rule) and equating the result to 0

find T which makes

20
Optimal Thresholding
 Minimum error

 Specially,if P1 = P2 then the optimum threshold

is where the curve p1(z) and p2(z) intersect

21
Optimal Thresholding

For example,
 Let PDF=Gaussian density : p1(z) and p2(z)

 where μ1 and σ12 are the mean and variance of

the Gaussian density of one object
 μ2 and σ22 are the mean and variance of the
Gaussian density of the other object 22
Optimal Thresholding
 Quadratic equation ( 二次方程 )

23
Problems of Thresholding

Original image Thresholded image

24
Problems of Thresholding

(a) Exact threshold

segmentation

(b) Threshold too low

(c) Threshold too high

25
7.2 Point Detection
 a point has been detected at the location
on which the mark is centered if
1 1 1
|R|≥T
1 8 1
1 1 1
 where T is a nonnegative threshold
R is the sum of products of the coefficients
with the gray levels contained in the region
encompassed by the mark

26
7.2 Point Detection
 Note that the mark is the same as the mas
k of Laplacian Operation (in chapter 3)

1 1 1 0 1 0
1 8 1 1 4 1
1 1 1 0 1 0

 The only differences that are considered i

nterest are those large enough (as determin
ed by T) to be considered isolated points.
27
Example

28
7.3 Line Detection
1 1 1 1 1 2 1 2 1 2 1 1
2 2 2 1 2 1 1 2 1 1 2 1
1 1 1 2 1 1 1 2 1 1 1 2
Horizontal 45 Vertical 45
 Horizontal mask will result with max response when
a line passed through the middle row of the mask
with a constant background.
 the similar idea is used with other masks.
 Note: the preferred direction of each mask is
weighted with a larger coefficient (i.e.,2) than
other possible directions. 29
Idea 1 of Line Detection
 Apply every masks on the image
 let R1, R2, R3, R4 denotes the response of the
horizontal, +45 degree, vertical and -45 degree
masks, respectively.

 if, at a certain point in the image

|Ri|>|Rj|,
 for all j≠i, that point is said to be more lik
ely associated with a line in the direction of
mask i.
30
Idea 2 of Line Detection
 Alternatively, if we are interested in detect
ing all lines in an image in the direction defi
ned by a given mask, we simply run the mas
k through the image and threshold the absol
ute value of the result.

 After thresholding, the points that are left

are the strongest responses, which, for line
s one pixel thick, correspond closest to the
direction defined by the mask.
31
Example

32
7.4 Edge-based Segmentation
 Edge-based segmentations rely on edges
found in an image by edge detecting
operators
 these edges mark image locations of
discontinuities in gray level
 Edge detection is the most common approach for
detecting meaningful discontinuities in gray level

 There are a large group of methods based

on information about edges in the image 33
What is edge?
 Edge is where change occurs
 Change is measured by derivative in 1D
― Biggest change, derivative has maximum magni
tude
 Or 2nd derivative is zero.
 we discuss approaches for implementing
― first-order derivative (Gradient operator)
― second-order derivative (Laplacian operator)
― we have introduced both derivatives in chapter 3
― Here, we will talk only about their properties for edge
detection. 34
What is edge?
 In other words, an edge is a set of
connected pixels
 that lie on the boundary between two regions
with relatively distinct gray-level properties.

 Note: edge vs. boundary

― an edge is a “local” concept.
― whereas a region boundary, owing to the way
it is defined, is a more global idea.
35
Ideal and Ramp ( 斜坡 ) Edges

because of
optics, sampling,
image acquisition
imperfection
36
Thick and Thin Edge
 The slope of the ramp is inversely proportional to
the degree of blurring in the edge.
 Namely, we no longer have a thin (one pixel thick) path.

 Instead, an edge point now is any point contained in

the ramp, and an edge would then be a set of such
points that are connected.
 The thickness is determined by the length of the ramp.
 The length is determined by the slope, which is in turn
determined by the degree of blurring.

 Blurred edges tend to be thick and sharp edges

tend to be thin.
37
First and Second derivatives ( 导数 )

Gray-level
profile

First
derivate

the signs of the

derivatives would be
Second
reversed for an edge derivate
that transitions from
light to dark
38
Second derivatives

 an undesirable feature
 produces 2 values for every edge in an image

 zero-crossing property
 an imaginary straight line joining the extreme
positive and negative values of the second
derivative would cross zero near the midpoint of
the edge.
 quite useful for locating the centers of thick
edges.
39
Basic idea of edge detection
 A profile is defined perpendicularly to the
edge direction and the results are
interpreted.
 The magnitude of the first derivative is used to
detect an edge (if a point is on a ramp)

 The sign of the second derivative can determine

whether an edge pixel is on the dark or light
side of an edge.

40
Review of First Derivate z1 z2 z3
z4 z5 z6
 Gradient Operator: f  G x  G y
z7 z8 z9
 simplest approximation, 22
G x  z6  z5 G y  z8  z5
 Roberts cross-gradientoperators, 22
1 0 0 1
G x  z9  z 5 G y  z8  z 6
0 1 1 0
 Sobel operators, 33
G x   z7  2 z8  z9    z1  2 z 2  z3 
1 2 1 1 0 1
G y   z3  2 z6  z9    z1  2 z 4  z7  0 0 0 2 0 2
1 2 1 1 0 1
41
Edge direction and strength
 Let α(x,y) represents the direction angle of
the vector f at (x,y)

α(x,y)=tan-1(Gy/Gx)
 The direction of an edge at (x,y) perpendic
ular ( 垂直 ) to the direction of the gradient
vector at that point
 The edge strength is given by the gradient
magnitude
f  G x  G y
2 2

42
Gradient Masks

1 2 1 1 0 1
1 0 0 1
0 0 0 2 0 2
0 1 1 0
1 2 1 1 0 1
Roberts
Sobel

1 1 1 1 0 1
0 0 0 1 0 1
1 1 1 1 0 1
Prewitt
43
Diagonal edges with Prewitt
and Sobel masks

0 1 1 1 1 0
1 0 1 1 0 1
1 1 0 0 1 1
Prewitt 44
Review of Second Derivate
 Laplacian Operator
 f  x  1, y   f  x  1, y   
 f 
2

 f  x , y  1  f  x , y  1  4 f  x , y  

0 1 0 1 1 1
Laplacian 1 4 1 1 8 1
Mask 0 1 0 1 1 1

45
Example of edge detection
 See Matlab Help--demo--toolbox--image p
rocessing--analysis…--Edge detection

 Note: The Laplacian is seldom used in practi

ce, because:
 unacceptably sensitive to noise (as second-order
derivative)
 produces double edges
 unable to detect edge direction
46
Canny edge detector
 The most powerful edge-detection method
 It differs from the other edge-detection
methods in that
 it uses two different thresholds (to detect
strong and weak edges),
 and includes the weak edges in the output only if
they are connected to strong edges.

 This method is therefore less likely than

the others to be "fooled" by noise, and
more likely to detect true weak edges. 47
Laplacian of Gaussian
 Laplacian combined with smoothing to find e
dges via zero-crossing.

 r 2
  2
  r 2

 h  
2
 exp   2 
   2 
4

r 2  x2  y2

  determines the degr

ee of blurring that oc
curs.
48
Laplacian of Gaussian (Mexican hat)

0 0 1 0 0
0 1 2 1 0
1 2 16 2 1
0 1 2 1 0
0 0 1 0 0
The coefficient must sum to zero
49
Edge Detection and Segmentation
 Image resulting from edge detection cannot
be used as a segmentation result.

 Supplementary processing steps must follow

to combine edges into edge chains that
correspond better with borders (boundary)
in the image.

50
7.5 Region-based Segmentation
 Goal: find regions that
 PPare “homogeneous”
((RRi)i) is
is aa logical b
logical predicate
predicate
y some criterion property defined
property defined over
over the
the
points in
points set RRi
in set
Segmentation is a process that partitions R
i

 For example, PP((RRi)i) == TRUE
For example, TRUE
into n subregions R1, R2, …, Rn such that:
if all
if all pixel in RRi i have
pixel in have the
the ss
ame gray
ame gray level
level

51
Two methods of Region Segmentation
 Region Growing
 Region Splitting

Region growing is the opposite of the split

and merge approach.

52
Region Growing
 The objective of segmentation is to
partition an image into regions
A region is a connected component with some
uniformity (say gray-levels or texture)

 In region growing, we start with a set of

“seed” points, growing by appending to each
seed’s neighbor pixels, if they have similar
properties, such as specific ranges of gray
level, and ‘8-connected neighbor’
 Need: initialization, similarity criterion 53
Steps of Region Growing
 Start by choosing an arbitrary seed pixel
and compare it with neighbor pixels.
 When seed and neighbor pixels are similar a
nd 8-connected neighbor, region is grown
from the seed pixel by adding neighbor
pixels.
 When the growth of one region stops,
choose another seed pixel which does not
yet belong to any region and start again.
54
 Region growing
 An initial set of small areas
are iteratively merged
according to similarity
constraints.


55
Example: defective welds ( 焊缝 ) detecting

 X ray of weld (the horizontal dar

k region) containing several cracks
and porosities ( 孔 , the bright, w
hite streaks running horizontally t
hrough the image)
Seed points

 We need initial seed points to gro

w into regions
 On looking at the histogram an
d image, cracks are bright
 Select all pixels having value o
f 255 as seeds

56
Criterion:
 There is a valley at around 190 in the histogram.
A pixel should have a value>190 to be considered
as a part of region to the seed point.
 The pixel should be a 8-connected neighbor to at
least one pixel in that region

Result of region growing and boundaries of defects 57

Region Splitting
 The opposite approach to region merging.
A top-down approach and starts with the assump
tion that the entire image is homogeneous.
 If this is not true, the image is split into four s
ub images
 This splitting procedure is repeated recursively
( 递归的 ) until the image is split into homogeneou
s regions.

 Need: homogeneity criterion, split rule 58

Region Splitting
 Disadvantage
 they create regions that may be adjacent and
homogeneous, but not merged.

59
Region Splitting and Merging

Procedure:

1. Split image into 4 disjointed quadrants, fo

r any region Ri, P(Ri)=FALSE

2. Merge any adjacent regions Rj and Rk, for

which P(Rj∪Rk)=TRUE

3. Stop when no further splitting or merging i

s possible.

60
Region Splitting and Merging

Quadtree

( 四叉树 )

61
Example

Original image Thresholded image Result of Splitting

and Merging

 P(Ri) = TRUE if at least 80% of the pixels in Ri hav

e the property |zj-mi|≤2σi
 where zj is the gray level of the jth pixel in Ri

 mi is the mean gray level of that region

 σi is the standard deviation of the gray levels in Ri
62