Professional Documents
Culture Documents
Image Segmentation: Chin-Ya Huang, Mon-Ju Wu ECE 533 Final Project, Fall 2006 University of Wisconsin-Madison
Image Segmentation: Chin-Ya Huang, Mon-Ju Wu ECE 533 Final Project, Fall 2006 University of Wisconsin-Madison
Nowadays, sports programs are among the most popular programs, and there is no
doubt that viewers’ interest is concentrated on the athletes. Therefore, demand for
image segmentation of sport scenes is very high in terms of both visual compression
and image handling using extracted athletes. In this project, we introduce a basic idea
about color information and edge extraction to achieve the image segmentation. The
color information helps obtain the texture information of the target image while the
edge extraction detects the boundary of the target image. By combining these, the target
image can be correctly segmented and represent. Besides, because color information
and edge extraction can use basic image processing methods, they can not only
demonstrate what textbook claims but also make us realize their function works. We
segmented an image based on the object while some can segment automatically.
Nowadays, no one can point out which the optimal solution is due to different
constraints. In [1], a similarity close measure was used to classify the belonging of the
pixels, and then used region growing to get the object. Unfortunately, it required a set of
markers, and if there is an unknown image, it is hard to differentiate which part should
be segmented. Linking the area information and the color histogram were considered
for building video databases based on objects [2]. However, the color information has
to be given first, and it is not useful for the life application. A genetic algorithm adapted
environmental conditions [3], but it took time learning. In [4], a two-step approach to
segmentation, and improved active shape models, line-lanes and live-wires, intelligent
scissors, core-atoms, active appearance models. However, there were still two
encountered. The authors in [5] proposed a graph-based method, the cut ratio is defined
following the idea of NP-hard as the ratio of the corresponding sums of two different
weights of edges along the cut boundary and models the mean affinity between the
segments separated by the boundary per unit boundary length. It allows efficient
it is necessary to locate in the image where the objects are [6]. The homogeneity
between two pixels and the distance function are included to measure the segmented
approximation mean square error RMSE and the number of distinct region tried to
not only to understand its principle but also to implement. Comparing the basic edge
detection method, which is get from Matlab commend, with our algorithm, there are
Figure 1: (a) Matlab “edge” command (left) (b) Proposed algorithm (right)
Figure 1 (a) and Figure 2 (a) show the segmented result by Matlab “edge” command,
and Figure 1 (b) and Figure 2 (b) show our proposed algorithm. It is pointed out that
although the proposed algorithm extracts some undesired parts into the segmentation
result, it represents almost the entire target image while the Matlab “edge” command
seems to show the boundary of the target image only. It is because the proposed
algorithm considers the color information at the same time in order to recover some
disconnected parts while the Matlab “edge” command actually has problem
Figure 3: (a) Matlab “edge” command (left) (b) Proposed algorithm (right)
command (a), and the proposed algorithm (b). It can be shown that the Matlab “edge”
command works better than the proposed algorithm because it can correctly extract
the whole boundary. However, the algorithm of Matlab can only extract the image
with clear vision. In other words, it has problem extracting the boundaries with
blurred image and cannot pass the pixel information of those area. On the other hand,
the algorithm we proposed can connect some of the disconnected boundary line
Basically, the proposed algorithm has better performance than the basic edge
automatically. The more detail about the proposed algorithm will be present in the
“Discussion” section.
In our algorithms, there are some criteria. First of all, we need to be aware of the
target image which we would like to segment out. Second, the background image has
to be blurred and the color of the target image should be different to that of background
image as much as possible. Moreover, we expect the appendages of the target image
Our approach is to obtain color information of the target image and boundary
extraction separately and simultaneously. We apply the character of HSI to acquire the
information of the pixels of the target image. In the mean time, we use the Matlab
“edge” and “imfill” command to extract the boundary and fill the image region whose
boundaries make a closure. Afterwards, we combine them by getting the union of the
two results. Finally, we perform and final modification and remove the noise. The
1) Firstly, acquire the color information and the edge information separately.
4) Combine the above results by getting the union of (2) and (3).
setting a threshold based on the pixel value of the hue, saturation, and intensity (H, S, I)
separately, these color information of the object can represent the parts with the image
close to these color information. The character of HSI is used to analyze color because
they are the three components of the original color. Sine the hue, saturation, and
intensity are independent of one another, we can use them to process the image
separately without worrying the correlation of them. On the other hand, if the
character of RGB is used instead, the color of the segmented results will change
Afterwards, to compensate the lack of the boundary when segmenting the image
by the character of HSI, an algorithm is used to extract the boundaries of the target
image. Therefore, we use the Matlab “edge” command to extract the approximate
boundaries. After getting the approximate boundaries, we want to get the information
of the pixels which reside in the extracted boundaries. However, the Matlab “edge”
command has problem full extracting all the boundaries and making all the
boundaries a closure. Therefore, when performing the Matlab “infill” command, only
the image region whose surrounding boundaries make a closure would be filled. In
order to fix it, the dilation is performed to connect the disconnected boundaries. After
the dilation, performing “imfill” would fill the image regions more effectively.
Then we combine the images segmented out from the above two methods by
getting the union of the two images so both the boundary information and color
information could be obtained. However, getting the union of them won’t increase the
noise significantly because when we perform the Matlab “edge” on the image whose
background image is blurred, it eliminates most of the background image and only
the boundaries of the target image, and then we intersect the combines image with the
dilated boundaries to eliminate the trivial noise that doesn’t reside in the target image.
The result of the intersection is the final image. The flowchart in Figure 5 and Figure 6
shows the detail of the “Texture Extraction” and “Shape Extraction” block.
are selected and partially segmented from the original image. The original images and
Figure 7: The original (upper) and processed (lower) image of “Image 1”.
Figure 9: The original (left) and processed (right) image of “Image 3”.
Figure 11: The original (left) and processed (right) image of “Image 5”.
Figure 14: The original (upper) and processed (lower) image of “Image 8”.
Figure 15: The original (upper) and processed (lower) image of “Image 9”.
Figure 18: The original (upper) and processed (lower) image of “Image 12”.
Figure 19: The original (left) and processed (right) image of “Image 13”.
Figure 21: The original (upper) and processed (lower) image of “Image 15”.
From the images above, it is obvious that for those images whose background
images are blurred can be segmented with nice quality. It is because when the Matlab
“edge” command is applied to extract the boundary of the image, only the partial
images with clear boundary would be extracted. Therefore, by performing the Matlab
clear foreground images would be left. After getting the approximate boundaries, we
want to get the information of the pixels which reside in the extracted boundaries.
However, the Matlab “edge” command has problem full extracting all the boundaries
and making all the boundaries a closure. Therefore, when performing the Matlab
“infill” command, only the image region whose surrounding boundaries make a
closure would be filled. In order to fix it, the dilation is performed to connect the
disconnected boundaries. After the dilation, performing “imfill” would fill the image
Other than using the existent Matlab code, the characters of HSI (hue, saturation,
and intensity) is also applied to segment the image. Only the information of the pixels
with similar HSI character to the foreground image would be kept. In order to apply
the characters of HSI to segment the image, partial images of the foreground image is
cut to calculate the average value of the hue, saturation, and intensity, and then the
deviation of the HSI value is set so create different ranges of the hue, saturation, and
intensity. Afterwards, the HSI value of every pixels of the image is examined. Only
the information of the pixel whose HSI value falls in the range will stay. Otherwise,
its information will be set to zero. By applying the character of HSI to image
segmentation, we can obtain all the information of the pixel whose HSI value is
both of the information, we get the union of the results from the above two steps so
the result from different methods would be combined. The reason why we get the
union of the results instead of the intersection is we only got partial images from the
two different methods mentioned above. So only by getting the union of them can we
from the entire image. However, getting the union of them won’t increase the noise
significantly because when we perform the Matlab “edge” on the image whose
background image is blurred, it eliminates most of the background image and only
The image of the union of results of the two methods above contains most of
information of the target image and some noise. In order to eliminate the noise, we try
to connect the disconnect boundaries by dilating the boundaries a few times, and then
we intersect the combines image with the dilated boundaries to eliminate the trivial
noise that doesn’t reside in the target image. The result of the intersection is the final
image.
time. However, it still has some flaws and limitations that need to be fixed and
First of all, the target image must be known. Since the method which applies HSI
to image segmentation is to cut the partial images of HSI and calculate the HSI values,
Second, the target image must have clear vision instead of blurred image. Other
than applying the character of HSI to the image segmentation, another method applied
to segment the image is to use the Matlab “edge” command to extract the approximate
boundaries. Therefore, when the background image is clear, this method will also
the foreground image mostly. Since the Matlab “imfill” command is afterwards
performed to fill the image region whose boundaries make a closure, the image with
clear boundaries will be extracted, including both the foreground and the background
images.
Figure 23: The original (left) and processed (right) image of “Image 17”.
Figure 23 and Figure 24 are two example of the failure of extracting the target image.
Due to the clearness of both the foreground and the background images, we have
Another limitation of our algorithm is that the appendages of the target image
cannot cross over each other. When the appendages of the target image cross over
each other, the boundaries of the appendages would make a closure. In other words,
image which resides within the closure of the boundaries of the appendages would be
considered as part of the foreground image so their pixel information would be passed
Figure 25: The original (upper) and processed (lower) image of “Image 19”.
Figure 25 and Figure 26 are two example of the failure of extracting the target image.
Due to the cross over of appendages, part of the background image is considered as
the target image. Take Figure 25 for instrance, the turf is part of the background image
cross over each other, it is considered as part of the target image and segmented out
mistakenly. Same thing happens to Figure 26. Since the waist of the two female
athletes makes a closure, the partial water image surrounded by the waist of two
female athletes is considered as part of the target image and segmented out
mistakenly.
“edge” command can only extract the approximate boundaries, the interpolation
Figure 27 for instance, the edge (Figure 27 left) along of the left leg of the soccer
player in the image is disconnected. After performing the Matlab “imfill” command,
not all the image region of the target image is filled (Figure 27 right). Therefore, if an
be connected so that the Matlab “imfill” command would fill all the target images. We
Figure 27: The edge of “Image 2” by performing the Matlab “edge” command
Moreover, other than using the character of HSI to extract the image, it will be
helpful to use other characters to extract the image so the segmented pixel information
of the target image will be more solid. Even though the character of HSI extracts most
background image whose HSI value fall in the setting range. Therefore, if we can
come up with other character to segment the image, we should be able to remove part
of the noise but keep the completeness of the target image in the mean time.
With our current algorithm of noise removal, it does remove most of the noise from
the background image. However, it has its own limitation and flaws. Since the dilation
background image surrounding the target image will also be extracted while they
actually should be eliminated. This noise relates to the disconnected boundary line
would not have to perform dilation to connect the disconnected boundary line
segments and the background image surrounding the target image will not be
extracted. Furthermore, the Matlab “edge” command will also extract edge the
background image which has clear vision and cannot be eliminated, so a more
segmentation.
[3] B. Bhanu, S. Lee, and J. Ming. ‘Adaptive image segmentation using a genetic
algorithm,’ IEEE Transactions on systems, man, and cybernetics, vol. 25, No. 12, Dec.
1995.
[4] M. Brejl, and M. Sonka, “Object localization and border detection criteria design in
[5] S. Wang, and J. M. Siskind, “Image segmentation with ratio cut,” IEEE
Transaction on pattern analysis and machine intelligence, vol. 25, No. 6, Jun. 2003.
[7] M. Tabb and N. Ahuja, “Multiscale Image Segmentation by Integrated Edge and
Region Detection, ” IEEE Transactions on image processing, vol. 6, no. 5, May. 1997.
systems, man and cybernetics, vol. 36, no. 2, pp. 366-378, Apr. 2006.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Segment the "uniform" from the image.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Call the "Hue" function to obtain the range of the hue value and pass the
% high and low end values back to the program.
% hLow: lower end of the range of the hue.
% hHigh: higher end of the range of the hue.
[hLow, hHigh] = Hue(rgb, 0.05);
[sLow, sHigh] = Saturation(rgb, 0.2);
[iLow, iHigh] = Intensity(rgb, 0.2);
% Screen through every pixel in the "tempImage" matrix. If the pixel value
% is NOT zero, then pass the pixel value to the corresponding location of
% the "sImage" matrix. Otherwise, don't pass any value.
for i=1:row
for j=1:col
if tempImage(i,j)>0
fImage(i,j) = tempImage(i,j);
end
end
end
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Segment the "leg" from the image.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Call the "Hue" function to obtain the range of the hue value and pass the
% high and low end values back to the program.
% hLow: lower end of the range of the hue.
% hHigh: higher end of the range of the hue.
[hLow, hHigh] = Hue(rgb, 0.02);
[sLow, sHigh] = Saturation(rgb, 0.2);
[iLow, iHigh] = Intensity(rgb, 0.1);
% Screen through every pixel in the "tempImage" matrix. If the pixel value
% is NOT zero, then pass the pixel value to the corresponding location of
% the "sImage" matrix. Otherwise, don't pass any value.
for i=1:row
for j=1:col
if tempImage(i,j)>0
fImage(i,j) = tempImage(i,j); % Pass the "leg"
information to "fImage"
end
end
end
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Segment the "face" from the image.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Call the "Hue" function to obtain the range of the hue value and pass the
% high and low end values back to the program.
% hLow: lower end of the range of the hue.
% hHigh: higher end of the range of the hue.
[hLow, hHigh] = Hue(rgb, 0.02);
[sLow, sHigh] = Saturation(rgb, 0.2);
[iLow, iHigh] = Intensity(rgb, 0.1);
% Screen through every pixel in the "tempImage" matrix. If the pixel value
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Segment the "socks" from the image.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Call the "Hue" function to obtain the range of the hue value and pass the
% high and low end values back to the program.
% hLow: lower end of the range of the hue.
% hHigh: higher end of the range of the hue.
[hLow, hHigh] = Hue(rgb, 0.05);
[sLow, sHigh] = Saturation(rgb, 0.2);
[iLow, iHigh] = Intensity(rgb, 0.1);
% Screen through every pixel in the "tempImage" matrix. If the pixel value
% is NOT zero, then pass the pixel value to the corresponding location of
% the "sImage" matrix. Otherwise, don't pass any value.
for i=1:row
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Combine the segmented image information.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% sImage: the matrix used to store the segmented image after combining the
% result from "EdgeDetection.m" and "Segmentation.m"
sImage = zeros(row, col, layer);
% Screen through every pixel in the "combine" matrix. If its pixel value is
% 1, then the pixel information in the original image will be passed to the
% new matrix, "sImage". Otherwise, the pixel vakue in that corresponding
% location of sImage stays 0.
for i=1:row
for j=1:col
for k=1:layer
if combine(i,j)==1
sImage(i,j,1) = H(i,j);
sImage(i,j,2) = S(i,j);
sImage(i,j,3) = I(i,j);
end
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Convert the image back to RGB format and do the final modification.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Convert the image of "sImage" matrix from HSI format back to RGB format.
% sRGB: the image converted from HSI to RGB format
sRGB = hsv2rgb(sImage);
% Convert the image of "finalImage" matrix from HSI format back to RGB format.
% finalImageRGB: the image converted from HSI to RGB format
sRGB = hsv2rgb(sImage);
finalImageRGB = hsv2rgb(finalImage);
% Convert the image into gray scale, R, G, and B. Perform the "edge"
% function to get the edge in different format.
% E_W: whole edge information without segregating the image.
% E_R: edge information of the Red.
% E_G: edge information of the Green.
% E_B: edge information of the Blue.
B=rgb2gray(A);
E_W=edge(B,'prewitt',0.1);
E_R=edge((A(:,:,1)));
E_G=edge((A(:,:,2)));
E_B=edge((A(:,:,3)));
% E: the combination of the edge information of R, G, and B.
E=E_R+E_G+E_B;
LPF=ones(5,5);
[row col]=size(E);
result=zeros(row, col);
for i=3:row-2
for j=3:col-2
result(i,j)=sum(sum(LPF.*E(i-2:i+2,j-2:j+2)));
if (result(i,j)>0)
result(i,j)=1;
end
end
end
% Combine the information of the "fImage" matrix and the "edge" matrix.
HSI = rgb2hsv(RGB);
H = HSI(:,:,1);
S = HSI(:,:,2);
I = HSI(:,:,3);
edge = edge;
% Now caompare the "fImage" matrix with the "edge" matrix from
% "EdgeDetection.m", and then perform and "imfill" again. Aftwerwards,
% return the combined and filled image.
cImage = imfill(cImage);
___________________________ ___________________________
___________________________
Date