JOURNAL OF COMPUTING, VOLUME 2, ISSUE 7, JULY 2010, ISSN 2151-9617

HTTPS://SITES.GOOGLE.COM/SITE/JOURNALOFCOMPUTING/
WWW.JOURNALOFCOMPUTING.ORG 158

Distance Parameter Based Facial Expression

Recognition Using Neural Networks
Anitha C, M K Venkatesha and B Suryanarayana Adiga

Abstract— In any facial expression recognition system, the location and extraction of features is the first task to be
accomplished. Extraction of features using feature points recognition is one of the most widely used technique. The proposed
work is capable of recognizing facial expressions with only eight parameters from a set of ten feature points. The system
identifies the basic emotions: Happiness, Sadness, Surprise, Disgust, Anger and Fear. Distance parameters are used for the
recognition of expressions. The distance parameters are derived from the feature points extracted from the images. The use of
distance parameters makes the system invariant to pose and illumination to a certain extent. The parameter set is trained and
tested using a feed-forward neural network.

Index Terms— basic emotions, distance parameters, facial expression recognition, feed-forward neural network.

——————————  ——————————

1 INTRODUCTION
techniques which, is extensively used in the facial expres-

H
UMAN faces are non rigid objects with a high
degree of variability in size, shape, color, and tex-
ture. Any automated system for face and facial
gesture recognition will have immense potential in crimi-
nal identification, surveillance, missing children retrieval,
office security, credit card verification, video document
retrieval, telecommunication, high-definition television,
medicine, human–computer interfaces, multimedia facial
queries, and low-bandwidth transmission of facial data.
There are several techniques to identify the expression on
one’s face: relative displacement of features (opening the
mouth) [1]; quasi-textural changes (furrowing the brow,
wrinkles) [2]; changes in skin hue (blushing) [1] and the
time course of these signals. This paper proposes a me-
thod of recognition of facial expressions closely based on
the relative displacement of the features of expression.
The images considered are gray-scale images. The fea-
ture-related distance measurement between feature
points is the essence of the proposed recognition system.
The consideration of distance between feature points
makes the recognition system invariant to pose and illu-
mination to a certain extent. The consideration of ratios of
these distance parameters is the uniqueness about this
recognition system. The classifier used is a feed-forward
neural network that categorizes the given input into one
of the six basic expressions. Earlier work using a simple
rule-based system for expression recognition was at-
tempted [16] which inspired the proposed work.
2 RELATED WORK
Feature-based extraction and recognition is one of the
————————————————
 Anitha C is with BMS College of Engineering, Bangalore, Karnataka, subjects in the database are 97 university students. They
India.
 M K Venkatesha is with RNS Institute of Technology, Bangalore, Karna-
taka, India.
 B Suryanarayana Adiga is with the TCS Ltd, Bangalore, Karnataka, In- and three percent were Asian or Latino. Image sequences
dia.
sion recognition systems. The feature extraction technique
based on facial feature locations (eyes, nose, mouth, et al)
are most convenient for usage mainly because they are
pose invariant and to some extent illumination invariant
[3]. In the field of face recognition in [4], [5], [6] feature-
based location and extraction techniques are used. The
feature-based methods can be completely automated or
manually coded. The number of feature points considered
in the proposed system is only 10 while compared to [7]
and [8] where the points considered were 11 and 83 re-
spectively. The ten feature points are transformed into
eight distance parameters which are used for the expres-
sion recognition. Use of distance parameters in 3D was
proposed in [7]. The work proposed in [9] uses the con-
cept of distance for a fuzzy regression model. The recog-
nition system can have different types of classifiers based
on the feature extraction techniques. Most of the recogni-
tion models have classifiers based on Neural Networks
[10], [11], or a traditional pattern recognition technique
such as SVM [10], rule-based [12], expert systems [13] et
al. The proposed facial expression recognition system is a
rule-based system that makes it simpler and achieves
higher rate of accuracy. Initial work was carried out by
the same authors using a rule-based system with the same
distance parameters [16]. The proposed system makes use
of neural networks for classification purpose.
3 DATABASE
For the evaluation of the proposed expression recognition
model, the Cohn-Kanade AU-Coded Facial Expression
Database is used [14]. The database includes approx-
imately 2000 image sequences from over 200 subjects. The
ranged in age from 18 to 30 years, sixty-five percent
whom were female, 15 percent were African-American,
from neutral to target display were digitized into 640 by
480 or 490 pixel arrays with 8-bit precision for grayscale
JOURNAL OF COMPUTING, VOLUME 2, ISSUE 7, JULY 2010, ISSN 2151-9617
HTTPS://SITES.GOOGLE.COM/SITE/JOURNALOFCOMPUTING/
WWW.JOURNALOFCOMPUTING.ORG
values. The Cohn-Kanade facial expression database is
one of the most widely used 2D database for facial ex-
pression evaluation. The expressions in the data base are
coded using the Facial Action Coding System. There are
also 3D databases in use such as [15].
159
The distance parameters for each subject in the database
were stored as an array in a file. Each expression of the
subject is a separate array in the file. So every subject has
two set of files: one containing the neutral (no expression)
parameters and the other containing the set of expression
parameters.

4 PROPOSED WORK
The proposed system is an extension work of [16] by the
same authors. The recognition system that is capable of
recognizing all the six basic expressions using the defined
eight parameters. The feature extraction process used is
based on identification of feature points on the image.
The recognizer is trained with the Cohn-Kanade facial
expression database. A feed-forward neural network is
used for this purpose. The data file used was created for
the rule-based system in [16], for the expression images to
be recognized. The creation of the data file was done sub-
ject-wise. The data file contains the eight parameters viz.,
P1 to P8 with the following meaning:
©Jeffrey Cohn

TABLE 1 Fig 1: Ten marked feature points on the image. The white
PARAMETER SET DESCRIPTION
spots seen on the image are the fearture points consi-
Parameter Description Coordinate dered.
value
P1 Distance between eyelids of y Unlike the previous work in [16] where the set of rules
the left eye governing each of the six emotions were based directly on
the distance parameters, in the proposed system we con-
P2 Distance between the brow x
sider the ratio between the distance parameters as input
corners of the left eyebrow
to the classifier. The distance parameters stored in the
P3 Distance between the left x data files were used to calculate the ratio patterns that
and the right brows at nasal will be used for training and testing purpose.
corner
P4 Distance between the brow y R E /Nj Nj 0 (1)
center and the top-eyelid
For i = 1, 6 (six expressions) and j = 1, 8 (eight parame-
P5 Lip corner distance x
ters). Where, Eij corresponds to the jth distance parameter
of the ith expression, Nj corresponds to the jth distance pa-
P6 Distance between left lip y rameter of expression less image (of the same subject) and
corner and upper lip center Rij corresponds to the resultant ratio of jth distance para-
meter of the ith expression.
P7 Distance between right lip y
corner and upper lip center
The feature extraction process is a feature-based method.
P8 Distance between upper lip y The ten feature points are manually extracted from each
center and lower lip center image. The feature extraction model calculates the dis-
tance between the feature points generated. The recogniz-
er model classifies one of the universal expressions on the
face.
These distances were measured for each image with
marked feature points. The feature points considered are The expression recognition algorithm comprises the fol-
as follows: three positions on the left brow (two corners lowing steps:
and a center), one position on the right brow (nasal corner
only), two positions on the left eyelids (top center and
bottom center) and four positions on the lips (upper lip 1. Extraction of the apex images from the database.
center, left lip corner, lower lip center and right lip cor- The database used here contains a video se-
ner) as seen in Fig 1. These feature points are the mini- quence, the first image with no expression and
mum required for any basic expression to be recognized. the last image containing the expression (apex).
JOURNAL OF COMPUTING, VOLUME 2, ISSUE 7, JULY 2010, ISSN 2151-9617
HTTPS://SITES.GOOGLE.COM/SITE/JOURNALOFCOMPUTING/
WWW.JOURNALOFCOMPUTING.ORG 160
2. Creation of data files for the neutral and the ex-
pression images subject-wise. 60% for training purpose
3. Generating the patterns based on equation (1) 20% for validation purpose
and storing it as input matrix for the neural net- (Generalizing the network and to stop training
work. before over fitting)
4. Train the two-layer neural network with the in- 20% for testing purpose
put ratio pattern and the target. (Is independent testing of network generaliza-
tion).
The sample input and target matrices value is as shown
below: The confusion matrix for training is as shown below:

TABLE 2
TABLE 4
SAMPLE INPUT PATTERN
CONFUSION MATRIX FOR TRAINING SET
1.5 0.75 2.0 1.5 0.5 2.0 2.0 3.5
0.75 1.25 1.05 1.0 1.25 1.05 1.05 0.75
Target  Surprise Sad Happy Fear Angry Disgust
0.8 1.05 0.8 1.0 1.5 1.05 1.05 1.05
Output
1.05 1.05 1.05 0.8 1.25 0.95 0.95 1.05
Surprise 100 0 0 0 0 0
0.75 1.15 0.75 0.75 0.8 1.25 1.25 0.6
Sad 0 100 0 0 0 0
0.8 1.2 0.75 0.75 0.8 1.2 1.2 1.2
Happy 0 0 90 10 0 0
Fear 0 0 0 100 0 0
Angry 0 10 0 0 90 0
This set of six ratios was selected from the input ratio pat-
tern that was given to the neural network classifier. Disgust 0 0 0 10 0 90

The target set used for training the neural network is Average = 94.4%

shown in table 3. The diagonal ones’ correspond to one of

the six expressions in the following order:
Surprise (first row), Sadness (second row), Happiness
The cause of misclassification between happy and fear,
(third row), Fear (fourth row), Anger (fifth row) and Dis-
anger and sadness, and disgust and fear is due to close-
gust (sixth row).
ness in the ratio patterns.
During the validation process 20% of the database image
TABLE 3 feature points were considered. The confusion matrix is
as shown below:
TARGET MATRIX
1 0 0 0 0 0
0 1 0 0 0 0
TABLE 5
0 0 1 0 0 0
0 0 0 1 0 0
CONFUSION MATRIX FOR VALIDATION SET
0 0 0 0 1 0 Target  Surprise Sad Happy Fear Angry Disgust
0 0 0 0 0 1 Output
Surprise 100 0 0 0 0 0
Sad 0 50 0 0 50 0
Happy 0 0 100 0 0 0
The neural network uses the Scaled Conjugate Gradient
Fear 0 0 0 100 0 0
algorithm, for training purpose. The scaled conjugate
Angry 0 0 0 0 100 0
gradient algorithm is based on conjugate directions but
Disgust 0 0 0 0 0 100
does not perform line search for all iterations unlike other
conjugate gradient algorithms. The scaled conjugate gra- Average = 90.7%

dient routine may require more iteration to converge than

the other conjugate gradient algorithms, but the number
of computations is significantly reduced in iterations be-
cause no line search is performed.
5 RESULTS
The Cohn-Kanade AU-Coded Facial Expression Database
was used for evaluation purpose. The training data was
randomly divided into three sets:
The testing of the remaining 20% of the ratio patterns was
performed. The results showed a drastic reduction in the
classification of fear and disgust. This is due to the close-
ness in the ratio patterns, as the ratio patterns were ran-
domly divided into three sets. There would be an im-
provement if the division of ratio patterns were done in a
sequential way.
JOURNAL OF COMPUTING, VOLUME 2, ISSUE 7, JULY 2010, ISSN 2151-9617
HTTPS://SITES.GOOGLE.COM/SITE/JOURNALOFCOMPUTING/
WWW.JOURNALOFCOMPUTING.ORG 161
ponding column thereby resulting in correct classifica-
tion.

TABLE 6
CONFUSION MATRIX FOR TESTING SET 6 DISCUSSION
Target  Surprise Sad Happy Fear Angry Disgust The results obtained in [16] were improved to some ex-
Output tent using the ratio patterns instead of considering the
Surprise 100 0 0 0 0 0 distance parameters directly. When the results of the pro-
Sad 0 100 0 0 0 0 posed system were compared with that of the results ob-
Happy 0 0 100 0 0 0
tained in [16], it was observed that all the expressions
Fear 0 33.3 0 66.7 0 0
except Anger and Disgust recognition had improved.
Angry 0 0 0 0 100 0
Table 9 summarizes the comparison.
Disgust 0 0 0 50 0 50
The recognition rate for Surprise, Sadness, Happiness and
Average = 93.3%
Fear expressions have improved significantly. But there is
a reduction in the recognition rates of Anger and disgust
Summarizing the results obtained from all the three stag- thereby decreasing the average recognition rate by 0.3%
es viz., training, and validation and testing, the confusion in the proposed system. Also the usage of a neural net-
matrix is as shown below. work classifier rather than a set of rule-based binary clas-
sifier [16], has enabled to produce a better system. The
randomness in the selection of ratio patterns for training,
TABLE 7 validation and testing may be the cause for the reduction
in the accuracy rates of anger and disgust.
CONFUSION MATRIX FOR THE ENTIRE PROCESS
Target  Surprise Sad Happy Fear Angry Disgust
Output
TABLE 9
Surprise 100 0 0 0 0 0
Sad 0 91.7 0 0 8.3 0 RULE-BASED SYSTEM VS
Happy 0 0 92.9 7.1 0 0 NEURAL NETWORK –BASED SYSTEM
Fear 0 9.1 0 90.9 0 0 System Rule-based system Neural network –
Angry 0 7.7 0 0 92.3 0 Expression based system
Disgust 0 0 0 13.3 0 86.7
Surprise 98.3% 100%
Average = 92.3%
Sad 90.9% 91.7%
Happy 92.6% 92.9%
For the sample input ratio patterns shown in table 2, the Fear 86.6% 90.9%
output produced by the trained system as one of the six Angry 96% 92.3%
expressions is as shown in table 8. Disgust 92% 86.7%
Average 92.7% 92.4%

TABLE 8
CORRESPONDING OUTPUT FOR SAMPLE INPUT
7 CONCLUSION AND FUTURE WORK
0.99 0.0006 0.0001 7.5e-05 0.007 0.027
Obtaining an accuracy of 92.4% and 92.7% [16], is in
0.001 0.990 0.001 0.118 0.308 0.045
itself good enough with just eight parameters. Further
3.9e-05 0.023 0.98 0.166 9.7e-08 4.8e-07
improvement can be seen with automating the extrac-
9.1e-06 0.182 0.19 0.694 0.0005 0.0008
tion process. The fact that “typically the expression on
6.3e-05 7.0e-06 0.0008 0.0003 0.546 0.031
any normal face is symmetrical” is being exploited
0.005 0.050 1.1e-05 0.007 0.253 0.706 here. The data file contents were carefully classified to
a nominal eight parameters per image considering the
coordinate dependent-variations. The performance of
The column-wise correspondence of expressions/ emo- the proposed system and [16] is convincing to con-
tions is as follows: Surprise, Sadness, Happiness, Fear, clude that the required number of parameters is as
Anger and Disgust. It can be observed by the above table nominal as eight. The future work lies in the automa-
that fear, anger and disgust values are slightly less than tion of marking of the ten feature points which is cur-
compared to surprise, sad and happy. This particular rently manually coded. The other classifiers may also
sample shows that for the given input ratio patterns the be considered in future.
diagonal elements are the maximum values in the corres-
JOURNAL OF COMPUTING, VOLUME 2, ISSUE 7, JULY 2010, ISSN 2151-9617
HTTPS://SITES.GOOGLE.COM/SITE/JOURNALOFCOMPUTING/
WWW.JOURNALOFCOMPUTING.ORG 162

ACKNOWLEDGMENT [14] Kanade, T., Cohn, J. F., and Tian, Y, “Comprehensive database
for facial expression analysis,” Fourth IEEE International Confe-
The authors would like to thank Kanade T, Cohn J.F and
rence on Automatic Face and Gesture Recognition, Grenoble,
Tian Y for providing the Cohn-Kanade facial expression
France, 2000, pp 46-53.
database. The authors would also like to thank the de-
[15] Lijun Yin, Xiaozhou Wei, Yi Sun, Jun Wang and Matthew J.
partment of Electronics and Communication Engineering,
Rosato, “A 3D Facial Expression Database for Facial Behavior
BMS College of Engineering, Bangalore, for their support
Research,” Eighth IEEE International Conference on Face and Ges-
in carriying out this work.
ture Recognition, 2008.
[16] Anitha C, M K Venkatesha and B Suryanarayana Adiga, “Rule-
based facial expression recognition using distance parameters,”
International Conference on Computer and Information Technology,
REFERENCES Italy, Rome. WASET Publications, April 2010.
[1] Lijun Yin and Anup Basu, “Color-based Mouth Shape Tracking
for Synthesizing Realistic Facial Expressions,” IEEE Internation-
Anitha C received her BE degree in Computer Science and Engi-
al Conference on Image Processing, 2002.
neering from Bangalore University, Karnataka, India in 1998 and her
[2] Lijun Yin, Sergey Royt, Matt T. Yourst and Anup Basu, “Recog- Masters Degree in Computer Science and Engineering from Visves-
nizing Facial Expressions using Active Textures with Wrin- waraya Technological University, Belgaum, Karnataka, India, in
kles,” IEEE International Conference on Multimedia and Expo, 2003. She is currently a Research Scholar in Electronics and Com-
munications Engineering department and is also working as a faculty
2003.
for Information Science and Engineering department, BMS College
[3] Ralph Gross, Jianbo Shi and Jeffrey F Cohn, “Quo Vadis Face of Engineering, Bangalore, India. Her research interests include
Recognition,” IEEE Conference on Computer Vision and Pattern Image Processing, Pattern Recognition and Human-Computer Inte-
Recognition, 2001. raction. She is a Student member of the IEEE.
[4] Paola Campadelli, Raffaella Lanzarotti and Giuseppe Lipori,
Dr. M K Venkatesha received his BE degree in Electronics and
“Automatic facial feature Extraction for Face recognition,” Face Communication Engineering from the University of Mysore, Karnata-
Recognition (2007) I-Tech Education and Publishing, Vienna, Aus- ka, India. He received his Masters Degree from University of Manito-
tria, pp 31-58. ISBN-978-3-902613-03-5 ba, Canada, on a Manitoba Hydro fellowship. He is the recipient of J
[5] Manjunath B S, Chellappa R, and Von der Malsburg C, “A Fea- C Bose Gold Medal award. He obtained his PhD in 1989 from the
University of Mysore, Karnataka. He is currently heading the RNS
ture-based face recognition,” IEEE International Conference on Institute of Technology, Bangalore, India. His research interests
Computer Vision and Pattern Recognition (CVPR), 1992 include Digital Signal Processing and its applications, Power Elec-
[6] Pinto-Elias and R. Sossa-Azuela, J. H, “Automatic Facial Fea- tronics, Digital Image Processing.
ture Detection and Location,” IEEE International Conference on
Pattern Recognition, 1998, Vol.14, pages 1360-1364. IEEE Com- Dr. B Suryanarayana Adiga received his BE degree in Electrical
puter Society Press. Engineering in 1968 and Masters Degree in Industrial Electronics
[7] Hamit Soyel and Hasan Demirel, “Facial Expression Recogni- from KREC, Suratkal in 1972. He received his PhD from Indian Insti-
tute of Science, Bangalore, India in 1988. He is currently working as
tion using 3D Facial Feature Distances,” LNCS Book Series, Im- Principal Engineer/ Consultant Supervisor for Tata Consultancy Se-
age Analysis and Recognition, Springer, 2007 vices, Bangalore, India. He has received the Institute of Electronics
[8] Hao Tang and Thomas S Huang, “3D Facial Expression Recog- and Telecommunications Engineer’s Gold Medal for excellence in
Avionics. His research interests include Digital Signal Processing,
nition Based on Properties of Line Segments Connecting Facial
Digital Image Processing, Control Systems and Information Coding.
Feature Points,” IEEE International Conference on Face and Gesture
Recognition, 2008.
[9] Liang-Hsuan Chen and Chan-Ching Hsueh, “Fuzzy Regression
Models using Least-Squares method based on the Concept of
Distance,” IEEE Transactions on Fuzzy Systems, Vol.17, No.6,
2009.
[10] Christian Martin, Uwe Werner and Horst-Michael Gross, “A
Real-Time Facial Expression Recognition System based on Ac-
tive Appearance Models using Gray Image and Edge Images,”
Eighth IEEE International Conference on Face and Gesture Recogni-
tion, 2008.
[11] Hiroshi Kobayashi and Fumio Hara, “Dynamic Recognition of
Basic Facial Expressions by Discrete-Time Recurrent Neural
Networks,” International Joint Conference on Neural Networks,
1993.
[12] Lawrence S. Chen, Thomas S. Huang, Tsutomu Miyasato and
Ryohei Nakatsu, “Multimodal Human Emotion/Expression
Recognition,” IEEE International Conference on Face and Gesture
Recognition, 1998.
[13] Maja Pantic and Leon J.M. Rothkranth, “Self Adaptive Expert
System for Facial Expression analysis,” IEEE International Confe-
rence on Systems, Man and Cybernetics, 2000.