52

nd
International Symposium ELMAR-2010, 15-17 September 2010, Zadar, Croatia
Corner Sharpening With Modified Harris Corner
Detection to Localize Eyes in Facial Images
Moein Lak
1
, AmirMohamad Soleimani Yazdi
2
1, 2
Islamic Azad University, South Tehran Branch, Technical & Engineering Faculty, P.O. Box: 11365-4435, Tehran, Iran
M.lak@ieee.org
Abstract Harris corner detection algorithm has been widely
used to find corners in images; it has been used for eye
localization in combination with filters. In this paper a new
strategy based on filters combination approach is suggested for
eyes localization in which filters are used to find and highlight
corners of region with local maximum intensity referred here as
sharp region. The filtering process proposed here utilizes Harris
corner detector in combination with Homomorphic and Tophat-
Bothat morphologic filtering. The use of Tophat-Bothat filter
enhances image at the first stage resulting in a highlighted region
with maximal differences in intensity level after which
application of Homomorphic filter extracts the desired region
and suppresses others. The proposed method was tested in
different applications including biometrics and industrial
(mechanical) image analysis in which we used sharp region
approach for corner detection. Experimental results, including
application on facial image analysis on both color and grayscale,
all indicate high performance can be achieved using the proposed
method.
Keywords-Harris Corner Detection; Image Analysis; Eye
Localization.
I. INTRODUCTION
ocalization of eye position in facial images has always
been considered to play an important role in many
systems such as: face recognition, gaze tracking, and iris
recognition, etc. By locating the position of eyes, the gaze can
be determined. Detecting eyes facilitates the task of locating
other facial features such as eyelids, nose and mouth needed
for recognition tasks. Eye detection is invaluable in
determining the orientation of the face; model based video
coding, face normalization for passport images, gaze detection
and human computer interaction. Significance of locating eyes
in facial features is due to their relatively constant intraocular
distance.
Various algorithms for eye detection can be generally
classified as follows. At the first stage, the rough estimate of
eye region is obtained which includes also imposing a priori
knowledge on the face image so that eye windows can be
located. This is followed by fine tuning of the eye zone to
localize eye position. In this paper, we describe an algorithm
for eyes detection that is robust against numerous factors
including changes in light conditions, visual angle, noise,
contrast and head orientation. At this stage, background of
images must have no texture but its color can be mixed. When
the filters are applied to the image, only regions belonging to
the corners with maximum difference in intensity such as eyes,
nostrils, furrows in hair, collar of shirts, pocket of shirt, will
have an improved contrast considering density as well as
accumulation of intensity, it would then be possible to select
the region of pair of eyes by using a suitable threshold. Now
projecting the intensity values across axis of the image, it
would be possible to identify the exact bounding box of each
eye. While other methods are also considered to find
approximate region of eyes such as projections and finding
peak amplitude, our algorithm finds the eye zone globally. In
this algorithm, while other corners are also detected as seen in
Fig.1 (f), but they are mostly of vertical shape; however the
eye shape is horizontal. Using this simple remedy, the eyes
region is obtained globally by binary enhancements via
morphologic filters and labeling the components of image.
The Section II summarizes various techniques that have
been utilized for eye detection. In section III and IV, details of
proposed method are described. In Section V the experimental
results are illustrated. Finally the paper is concluded in section
VI.
II. BRIEF OVERVIEW OF RELATED WORKS
There has been several research works dealing with eye
detection and eye region localization during the last decade
and numerous approaches have been proposed.[1,2,3,4,5]
These techniques are based on the use of texture, shape and
color information or a combination of them for eye detection.
Generally the approaches are categorized according to image
segmentation, grayscale projection, edge detection, template
matching and deformable template matching. Lam et al. [1]
improves deformable templates matching method by eye
corners. Vezhnevets et al. [2] estimates approximate eyelid
contours based on facial features (eye corners, iris border
points). Iris center, radius and upper eyelid are then detected
where by filtering the resulted image, the outliers are removed.
This is followed by polynomial curve fitting to boundary
points. Projection functions [3] [4] [5] have also been shown
in different publications. Reinders et al. [6] used eyes position
template matching applied on a sequence of video images.
Some other modified versions of projection method are
introduced by Feng and Yeun [3], Zhou and Geng [4]. Huang
et al. [7] applied optimal wavelet packets for eye
representation and radial basis functions for subsequent
classification of facial areas into eye and non-eye regions.
L
27
52
nd
International Symposium ELMAR-2010, 15-17 September 2010, Zadar, Croatia
Yuille et al. [8] proposed deformable templates in locating
human eye. Saber et al. [9] used geometrical structure of facial
images to estimate the location of eyes. Real-time eye
detection [10] [11] [12] [13] using infrared images are utilized
to capture the physiological properties of eyes. Gabor Filters
[14], principal component analysis [15] [16], Eigeneyes [17],
SVM based architectures [18], and Neural Networks classifier
[19] are also used in some literatures. Current eye detection
methods can be divided into two categories: active and passive
eye detection [23]. The active detection methods use special
illumination and IR cameras to quickly locate pupil centers.
The disadvantages are that they need special lighting sources
and have more false detections within an outdoor
environment. Passive methods directly detect eyes from
images within visual spectrum and normal illumination
condition. Niu et al [24] used 2D cascade adaboost for eye
localization. The two cascade classifiers bootstrap positive and
negative samples. A good survey on recent works is available
in [25].
III. SYSTEM ARCHITECTURE AND METHODOLOGY
As previously indicate, proposed algorithm is composed of
two major steps. At the first stage, coarse estimation of eye
pair is obtained through the sequential filtering. For color
images, by converting the image to Lab space and applying
Homomorphic filter and Harris corner detector- combined
with Tophat-Bothat filters, the main facial template image will
be generated. By trimming the output image with morphologic
filters and using a suitable enhancement, a texture-based
thresholding is applied on the image to obtain corresponding
binary image. At this stage, projection is done on the binary
image to localize eye pair and remove the outliers. Finally by
components labeling and removing horizontal components, the
final binary image is produced. By Using morphologic
operations one can easily find the bounding box and centroid
of eyes. (Fig.1). Some of the related theoretical considerations
of the main steps are described below.
IV. IMAGE PROCESSING AND ENHANCEMENT
A brief explanation of Homomorphic filters, morphologic
filters and Harris corner detector is first given. Then in each
step the output of algorithm is shown and the necessary details
of each step are included. We used luminance part of image as
input by simply converting the image into Lab space.
A. Homomorphic Filtering
The illumination-reflectance model of image can be used as
the basis for a frequency domain procedure that is useful for
improving the appearance of an image by simultaneous
brightness range compression and contrast enhancement [20].
An image ) , ( y x f can be expressed in terms of its
illumination and reflectance components as follows:
) , ( ) , ( ) , ( y x r y x i y x f = (1)
However, the product (separable) of the image representation
in spatial domain as given above, does not yield separable
from of representation in Fourier domain for illumination and
reflectance because the Fourier transform of the product of
two functions is not separable. The low spatial frequency
illumination is separated from the high frequency reflectance
by Fourier high-pass filtering. In general a high-pass filter is
used to separate and suppress low frequency components
while still passing the high frequency components in the
signal, if the two types of signals are additive, i.e., the actual
signal is the sum of the two types of signals. However, in this
illumination/reflection problem low-frequency illumination is
multiplied, instead of being added, to the high-frequency
reflectance. In order to be able to use the usual high-pass filter,
the logarithmic operation is needed to convert the
multiplication to addition. Therefore we use the following
formula for image representation on which Fourier transform
is applied:
) , ( ln ) , ( ln ) , ( ln y x r y x i y x f + = (2)
We can now apply filtering separately in the frequency
domain as given below:
) , ( ) , ( ) , ( v u R v u I v u Z + = (3)
Where ) , ( v u I and ) , ( v u R are the Fourier transforms of
equation (2). If we process ) , ( v u Z by means of a filter
function ) , ( v u H and then apply inverse Fourier transform the
equation converts to:
) , ( ) , ( ) , ( y x r y x i y x s ' + ' = (4)
Where
)} , ( ) , ( {
1
) , (
)} , ( ) , ( {
1
) , (
v u R v u H y x r
v u I v u H y x i

= '

= '
(5)
Finally, as ) , ( y x z was formed by taking the logarithm of
the original image ) , ( y x f , the inverse operation yields the
desired enhanced image. The illumination component of an
image is generally characterized by slow spatial variations,
while the reflectance component tends to vary abruptly,
particularly at the junctions of dissimilar objects. These
characteristics lead to associating the low frequencies of the
Fourier transform of the logarithm of an image with
illumination and the high frequencies with reflectance.
Although these associations are rough approximations, they
can be used to improve image enhancement results.
B. Harris Corner Detection
The Harris corner detector is a popular point detector due to
28
52
nd
International Symposium ELMAR-2010, 15-17 September 2010, Zadar, Croatia
its strong invariance to rotation, scale, illumination variation
and image noise [2]. The Harris corner detector is based on the
local auto-correlation function of a signal; where the local
auto-correlation function measures the local changes of the
signal with patches shifted by a small amount in different
directions. Harris and Stephens improved upon Moravec's
corner detector by considering the differential of the corner
score with respect to direction directly, instead of using shifted
patches. Let this image be shown by I. Consider taking an
image patch over the area ) , ( v u and shifting it by ) , ( y x . The
SSD (sum of squared differences) between these two patches,
S is given by:
_ _ =
u v
y v x u I v u I S
2
)) , ( ) , ( ( (6)
The Harris matrix is found by taking the second derivative
(the Hessian) of S around ) 0 , 0 ( ) , ( = y x . A is given by:
(
(
(

=
) (
2
) (
2
y I
y
I
x
I
y
I
x
I x I
A (7)
where angle brackets denote averaging (summation over ) , ( v u
), and the typical notation for partial derivatives is used. If a
circular window (or circularly weighted window, such as a
Gaussian) is used, then the response will be isotropic.
Although there are two versions of Harris corner detector
descriptors, we used the following equations as corner
detector:
) (
2
) (
2
2
) (
2
). (
2
y I x I
xy
I y I x I
Cor
+

= (8)
C. Tophat-Bothat filter
Morphology is a broad set of image processing operations
that process images based on shapes. Morphological
operations apply a structuring element to an input image,
creating an output image of the same size. Instead of
constructing a dot product of a mask array and a pixel array,
morphological processes use set theory operations such as
intersection (AND), union (OR), and complement (NOT) to
combine pixels logically into a resulting pixel value.
Morphological operators work much like spatial convolution
based on values of the neighboring pixels. Bothat filtering is
combination of closing and subtracting of original image but
Tophat filtering is combination of opening and subtracting.
Bothat = Subtract the input image from its closing
Tophat = Subtract the opening from the input image
We used union of two filters by subtracting the Tophat
filtered image plus original image from Bothat output.
V. EXPERIMENTAL RESULT
In this section the results of each block in Fig.1 are
explained. The results demonstrate the performance of the
algorithm. Because of very low intensity of the output in some
steps, for good visualization we show the complement of
output. Fig.2 (a) shows the original image in RGB space. We
perform conversion to Lab space and use luminance
component of image which is shown in Fig.2 (b). In Fig.2 (c)
the complement of output from Homomorphic and Harris
filters is depicted. You can see that only the sharp region
explained in section I is highlighted but the contrast of these
regions is poor. By performing the combined Tophat-Bothat
filters (Fig.2 (d)) and complements enhancement with
Homomorphic filter which is shown in Fig.2 (e), the images is
enhanced. By finding the maximum intensity of image and
performing threshold (Fig.2 (f)), only the sharp regions are
found. Because the layout of eye is in vertical shape (the
length of eye is bigger than its height), by morphologic
labeling and searching for those components that satisfies this
constraint, many components of filtered image are removed.
The output is shown in Fig.2 (g). The projection against two
versus results precisely eye pair location because in every
image the biggest projection is happened in eyes. Finally by
applying morphologic labeling easily the bounding box and
centroid of each eye are obtained. The reminder of images can
be followed in Fig.2 (h-l).
Our method was performed on different databases as
follows:
- ORL face database [22] which contains 400 facial
images from 40 distinct subjects. There are 10
different images of each subject ( 92 112 pixels).
- Our color facial database (passport and ID card images)
that contains 220 images from 110 subjects (
240 320 pixels)
- Additional images from internet.
- Animation images.
The eye pair candidates can be selected successfully in all
cases whether face patterns are in different scale, expression,
and illumination conditions. The system was implemented by
MATLAB, 2.2GHz Intel processor, 256M RAM. Fig.3 shows
some results of eyes location. A perfect success rate on all
images combined with superior results on animation images
indicated the strength of our new method. The average time
for localization was only 0.47 seconds on our database and
0.17 seconds on ORL.
VI. CONCLUSION
In this paper a new method based on regional synthesis is
proposed for eye localization. The algorithm generally
highlighted sharp area with corners. Using combined
Homomorphic, Harris and Tophat-Bothat filtering with
29
52
nd
International Symposium ELMAR-2010, 15-17 September 2010, Zadar, Croatia
morp
with
intel
Then
deter
each
on b
pictu
of th
[1]
[2]
[3]
[4]
[5]
[6]
[7]
[8]
[9]
[10]
[11]
[12]
[13]
[14]
[15]
[16]
[17]
[18]
[19]
[20]
phologic ope
h some mino
lligent searchi
n easily the b
rmined with m
h labeled regio
both grayscale
ures show the
he proposed m
K.M. Lam, H. Y
Images, in Pat.
V. Vezhnevets a
Extraction, in
2003.
G.C. Feng and
Application to E
Let, vol. 19, pp. 8
Z.H. Zhou, and
Pat. Rec., vol. 37
R. Thilak Kuma
Using Color Cue
Processing ICIP
M.J.T. Reinders
Automatic Codeb
Computing and I
J. Huang and H
Packets and Rad
13, 1999.
A.L. Yuille, P.W
Faces Using Def
pp. 99-111, 1992
E. Saber and A
Feature Extracti
Functions, in Pa
X. Liu, F. Xu an
for Driver Obs
Intelligent Vehic
Z. Zhu, K. Fujim
under Various L
Eye Tracking Re
A. Haro, M. Fli
Using Their Phy
Proc. IEEE CVP
C. Morimoto,
Detection of Ey
Interfaces, San F
S.A. Sirohey and
Linear and Nonli
K. Talmi and J.
Interactive Ster
Communication,
E. Hjelms and J.
Proc. 11th Scand
A. Pentland, B. M
Eigenspaces for
Vision and Patte
J. Huang, D. Ii, X
Detection Using
ASI on Face R
Verlag, pp. 528-5
M.J.T. Reinders
Features in Imag
Conf. Automatic
R.C. Gonzalez
Edition. Addison
erators empha
or outliers. B
ing in binary i
bounding box
morphologic o
on in the bina
e and color fac
e high accurac
method.
REFER
Yan, Locating a
Rec. J., vol. 29, p
and A. Degtiarev
Proc. Graphicon
P.C. Yuen, V
ye Detection for
899-906, 1998.
X. Geng, Proje
7, pp. 1049-1056,
ar, S. Kumar Raj
es and Projection
P, vol. 3, Rocheste
s, Eye Trackin
book Generation
Imaging, Heijen N
H. Wechsler, E
dial Basis Functio
W. Hallinan and
formable Templa
2.
A.M. Tekalp, Fr
ion Using Color
at. Rec. Let., vol.
nd K. Fujimura,
ervation under
le Symposium, V
mura and Q. Ji, R
Light Conditions
esearch &Applica
ickner and I. Es
ysiological Prope
PR, Hilton Head I
D. Koons, A.
es and Faces, in
Francisco CA, pp
d A. Rosenfeld,
inear Filters, Pa
Liu, Eye and G
reoscopic Displa
vol. 14, Berlin G
Wroldsen, Reco
dinavian Conf. Im
Moghaddam and
Face Recognitio
ern Recognition, S
X. Shao and H. W
g Support Vector
Recognition: Fro
536.
s, R.W.C. Koch
ge Sequences Usi
Face and Gestur
and R.E. Wood
n-Wesley Longma
asize particul
By morpholo
image only ey
x and centroi
operators that
ary image. Ex
cial images as
cy and compu
RENCES
and Extracting th
pp. 771-779, 199
va, Robust and
n-2003, Moscow
Variance Project
Human Face Rec
ection Functions
, 2004.
a, A.G. Ramakri
n Functions, in P
er New York USA
ng by Ttemplate
Scheme, in 3rd
Netherlands, pp.
Eye Detection Us
ons, in Int. J. P
D.S. Cohen, Fe
ates, in Int. J. Co
ontal-View Face
r, Shape and S
19, (1998) 669-6
Real-Time Eye D
Various Light C
Versailles France,
Real-Time Eye D
and Face Orienta
ations, NewOrlean
ssa, Detecting a
erties, Dynamics
sland, South Caro
Amir and M.
n Proc. Worksho
. 117-120, 1998.
Eye Detection i
at. Rec. J., vol. 34
Gaze Tracking fo
ays, in Signa
Germany, pp. 799
ognizing Faces fr
mage Analysis, 19
d T. Starner, Vie
on, in Proc. IEE
Seattle WA, pp. 8
Wechsler, Pose D
Machines (SVM
om Theory to A
and J.J. Gerbra
ing Neural Netwo
re Recognition, 1
ds, Digital Imag
an Publishing Co
larly eye loc
ogic labeling
ye pair is obta
id of each bo
find propertie
xperimental re
s well as anim
utational effici
he Eye in Human
6.
Accurate Eye Co
w, Russia , pp. 8
tion Function an
cognition, in Pat
for Eye Detectio
ishnan, Eye Det
Proc. Int. Conf. I
A, pp. 337-340, 2
e Matching Usin
d. Conf. Adv. Scho
85-91, 1997.
sing Optimal W
Pat. Rec. and A. I
eature Extraction
omputer Vision, v
e Detection and
Symmetry Based
680.
Detection and Tra
Conditions, in
pp. 18-20, 2002.
Detection and Tra
ations, in ACM
ns USA. 2002.
and Tracking Ey
, and Appearanc
olina, 2000.
Flickner, Real
op on Perceptual
in a Face image
4, pp. 1367-1391,
for Visually Cont
l Processing. I
9-810, 1999.
rom the Eyes On
999.
ew-Based and Mo
EE Int. Conf. Com
84-91, 1994.
Discrimination an
Ms) , in Proc. N
Applications, Sp
ands, Tracking
orks, in Proc. 2n
996.
ge Processing, S
o., Inc, 2001.
ation
and
ained.
ox is
es for
esults
mated
iency
n Face
ontour
81-84,
nd its
t. Rec.
on, in
tection
Image
2002.
ng an
ool for
Wavelet
I., vol.
n from
vol. 8,
Facial
d Cost
acking
IEEE
acking
Symp.
yes By
ce, in
l-Time
l User
Using
2001.
trolled
Image
ly, in
odular
mputer
nd Eye
NATO-
pringer
Facial
nd Int.
Second
[21]
[22]
[23]
[24]
[25]
Fig. 2. The resu
right, up to dow
component of
Homomorphic fi
Homomorphic f
image, threshold
output after labe
detection, morph
C. Schmid, R. M
Detectors, Int. J
The ORL Datab
JI Qiang, H. We
eye detection
Understanding,
N. Zhiheng, S.
Cascaded AdaB
Conference on P
P. Canpadelli, R
in the book
Communication
September 2006
Input Image
Trimmi
Binariz
Thresholding
Fig. 1. Flow
ults of algorithm
wn): original ima
image, complem
ilters, Tophat-Bot
filtered image,
ded image with m
eling and removi
hologic labeling, e
Mohr and C. Ba
J. Computer Visio
ase of Faces. http
echsler, A. Duch
and tracking,
pp. 13, 2005.
Shiguang, Y.
Boost for Eye Lo
Pattern Recogniti
R. Lanzarotti and
of The Funda
and the Biom
6.
Homomorp
C
Ho
ing &
zation
Project
chart of system fo
m. (Figures (a)-(l
age in RGB form
ment of applyin
that filtered imag
adjusted comple
maximum intensi
ing, projection pr
eye detection.
auckhage, Evalu
on, vol. 37, pp. 1
p://www.uk.resea
howski and M. F
in Computer
Shengye, C. Xi
ocalization, in P
ion, 2006.
G. Lipori, Eye
mentals of Ver
etrical Issue, N
rphic Har
Complement
omomorphic
tion
E
for eye localizatio
l) from left to
mat, luminance
ng Harris and
ge, complement
ement contrast
ity of adjusted,
rofile, eye pair
uation of Interest
51-172, 2000.
arch.att.com:pub/d
lickner, Special
r Vision and
ilin and g. Wen
Proc. 18th Intern
localization: a su
rbal and Non-
NATO Science
rris Corner detection
Tophat- Botha
Eye Localization
on.
t Point
data/
l issue:
Image
n, 2D
nationl
urvey,
-verbal
Series,
n
t
n
30
52
nd
International Symposium ELMAR-2010, 15-17 September 2010, Zadar, Croatia





Fig. 3. Samples of eye localization. Color images are from
our database, grayscale images or from ORL database and
3 animated images from internet. Eyes in all images are
localized very well.
31