2015 International Conference on Pervasive Computing (ICPC)
A Genetic Algorithm-Based 3D Feature Selection for
Lip Reading
Sunil Sudam Morade
PhD Student,
Department of Electronics Engineering,
SVNIT, Surat, India.
ssm.eltx@gmail.com
Abstract—In lip reading, selection of features play crucial
role. In lip reading applications database is video, so 3
Dimensional transformation is appropriate to extract lip motion
information. State of art the lip reading is based on frame
normalization and frame wise feature extraction. However this is
not appropriate due to chances of information loss during frame
normalization. Also all the frames cannot be considered equally
as they bear varying motion information. In this paper 3D
transform based method is proposed for feature extraction.
These features are the input to Genetic Algorithm (GA) model
for discriminative analysis. Genetic Algorithm is used for
dimensionality reduction and to improve the performance of the
classifiers at low cost of computation. Both testing and training
time for classifier is reduced by compact feature size. For
experimentation of digit utterances CUAVE and Tulips database
are used. The results obtained are compared with various feature
selectors from WEKA software. It is found that from
classification accuracy point of view proposed method is better
than others.
Keywords— BPNN, Feature Selection, Genetic Algorithm,
KNN, SVM, 3D-DWT, 3D-DCT.
I. INTRODUCTION
Visual speech recognition is a technique used to identify the
speech by lip movement. Lip reading is also called visual
speech recognition. Best lip reader can understand the speech
by lip movement. Hearing impaired person understand the
speech by lip movement. From long days people know that the
lip movement having speech information. In Many
applications such as speech in noisy area, places where you do
not have to speak and disaster condition (earthquake) visual
speech is important.
State of art literatures on appearance model are many, out of
which few noteworthy literatures are cited here for basic
understanding of challenges in lip reading paradigm. E.
Petajan [1] experimented on lip-reading to enhance speech
recognition by using visual information. Potamianos et al. [2]
used linear image transforms namely PCA, DWT and DCT
transform techniques. R. Seymour et al. [3] compared image
transform features in visual speech recognition of clean and
corrupted videos. They evaluated PCA, DCT, Fast Discrete
Curvalet Transform (FDCT), and Linear Discriminant
Analysis (LDA) methods. The classification performance
parameters such as specificity, sensitivity and accuracy are
978-1-4799-6272-3/15/$31.00(c)2015 IEEE
Authorized licensed use limited to: KIIT University. Downloaded on July 17,2023 at 04:24:18 UTC from IEEE Xplore. Restrictions apply.
Suprava Patnaik
Professor,
Department of E and TC Engineering,
Xavier Institute of Engineering,
Mahim, Mumbai, India.
suprava_patnaik@yahoo.com
used to test classifiers. Meyor et al. [4] used DCT transform
technique for pixel information of continuous digit recognition
and proposed different fusion techniques for audio and video
feature data. They found that Word Error Rate (WER) is more
for continuous digit recognition.
High dimensional feature set can negatively affect the
performance of pattern or image recognition systems. In other
words, too many features sometimes reduce the classification
accuracy of the recognition system since some of the features
may be redundant and non- informative [5]. In machine
learning , feature selection, which is also called variable
selection or variable subset selection, is the process of
obtaining a subset of relevant features is useful in many
application. There are lots of techniques available for
obtaining such subsets. Some of these techniques include
Principal Component Analysis (PCA), Particle Swarm
Optimization (PSO) and Genetic Algorithm (GA) [6]. More
often, lots of researchers in recent times have employed
Waikato Environment for Knowledge Analysis (WEKA)
software for dimensionality reduction. However, WEKA
software is static in its feature selection approach as the users
cannot change the configuration of the concerned feature
selectors [7]. GA has been known to be a very adaptive and
efficient method of feature selection as reported by [6]. It is to
be noted that feature selection is inherently a multi-objective
problem with two main goals: 1) minimizing both the number
of features 2) reduce classification error.
Frame work of proposed lip reading model is described in
section-II. Section III deals with GA based Feature selection.
Experimentation results and description of test corpus is given
in section-IV. Finally section-V is based on our conclusion and
scope for future work.
II. PROPOSED LIP READING FRAMEWORK
A typical lip reading system consists of three major stages:
video frame normalization and lip localization, feature
extraction, and the final step is classifier. In our proposed
model one more stage can be added as feature selection. Fig. 1
shows the major steps used in the proposed lip reading process.
 Features
Normalised extraction Classifiers
Frames and Lip by 3 D DWT/ (SVM,BPNN
localisation 3 D DCT , KNN)
Fig. 1. Lip reading process
A. Video Segmentation and Lip Contour Localization
There are large inter and intra subject variations in
utterance of a digit and this results in difference in the number
of frames for each utterance. We have used audio analysis,
using Pratt software to segment the time duration and the
associated video frames of each digit which is uttered. On an
average 16 frames are sufficient for utterance of any digit
between 0-9.Out of 16 frames we have selected 10 significant
frames. We have used Adaboost algorithm for face and mouth
detection. A sample result is shown in Fig. 2(a-b).
Fig. 2. (a) Detection of face and lip area for CUAVE datbase s02m(b) Lip
portion
B. Frame Feature Extraction
K. Min et al. used 3-D DCT for feature extraction and 3-D
HMM as classifier [9]. In order to consider dynamic feature 3-
D DCT features are important.
While 2-D DWT is used for computing the feature vector.
Goal is to select only those coefficients which play the
dominant role in the representation of lip motion. In standard 2-
D wavelet decomposition based approach, each level of
filtering splits the input image into four parts via pair of low-
pass and high-pass filters with respect to column vectors and
row vectors of the image matrix. Then the low-spatial
frequency sub-image is selected for further decomposition.
After few levels of decomposition the lowest spatial-frequency
approximation sub-image, is extracted as the feature vector.The
3-D DWT is like a 1-D DWT in three directions. Lip reading is
a video processing application. To use the wavelet transform
for volume and video processing, a 3-D version of filter banks
are implemented. In 3-D DWT, the 1D analysis filter bank is
applied in turn to each of the three dimensions [10, 11].
Equations 1 and 2 show the output 1 D filter output for one
direction image signal. Filter coefficients are derived using
Dmey wavelet.
2𝑛
𝑊𝑙 (𝑛, 𝑗) = � 𝐼(𝑚, 𝑗 − 1) ∗ ℎ(2𝑛 − 𝑚) (1)
𝑚=0
Authorized licensed use limited to: KIIT University. Downloaded on July 17,2023 at 04:24:18 UTC from IEEE Xplore. Restrictions apply.
2𝑛
𝑊ℎ (𝑛, 𝑗) = � 𝐼(𝑚, 𝑗 − 1) ∗ 𝑔(2𝑛 − 𝑚) (2)
𝑚=0
where W (n, j) is wavelet output. h (n) and g(n) are the filter
impulse response of low pass and high pass filter, j is the
current level, n is the current input index and I(n, j − 1) is the
input signal. The results from [8] motivated us to select Dmey
wavelet, as in lip reading application as the speech
information is extracted from visual information.
III. FEATURE SELECTION
A. Genetic Algorithm (GA)
Genetic Algorithm (GA) is an optimization technique, a
population-based and algorithmic search heuristic methods
that mimic natural evolution process of man [12, 13]. The
operations in a GA are iterative procedures manipulating one
population of chromosomes (solution candidates) to produce a
new population through genetic functional such as crossover
and mutation. S. Sivanandam [14] used the terminology
between human genetic and GA. Table 1 is a comparison
between human genetic and GA. The fitnesses of the solution
candidates (chromosomes) are evaluated using a function
commonly referred to as objective or fitness function. The
formulation of the fitness function depends on the problem
being solved. In relation to this article, maximizing
classification accuracy is equivalent to minimizing the
classification error rate. Fig. 3 shows Selection of 3D-DWT or
3D-DCT features using Genetic Algorithm.
Number of features Number of
164 after 3D DWT features ≤
Genetic
/3D DCT 164/149 (..F2,
Algorithm
F5,…….F8,F21)
(F1,F2……………. (GA)
F164)/(F1,F2……
)
Fig. 3. Selection of 3D-DWT or 3D-DCT features using Genetic Algorithm
The GA operates on binary search space as the chromosomes
are bit strings. The GA manipulates the finite binary
population in similar to human natural evolution. To begin
with, an initial population is created (mostly random) and
evaluated using a fitness function. For binary chromosome
employed in this work, a gene value ’1’ depicts that the
particular feature indexed by the position of the ’1’is selected.
Otherwise, (i.e. if it is ’0’), the feature is not selected for
chromosomal evaluation. Using the positional index of
features indexed by the ’1s’, the chromosomes are then ranked
and based on the rankings, the top n fittest kids (Elitism of size
n) are selected to survive to the next generation. After the elite
kids are pushed automatically to the next generation, the
remaining kids (individuals) in the current population are
allowed to genetically pass through the functional crossover
and mutation to form crossover and mutation kids
respectively. The three (3) kids viz elite, crossover and population. Tournament selection is performed iteration until
mutation then form the new population (new generation). the new population is filled up.
Crossover (a genetic functional) is a combination of two e. Crossover function
individuals (chromosomes) to form a crossover kids. Mutation In this XOR operation is performed on the two parent
operator on the other hand, is used for genetic perturbation of chromosomes since they are binary
the genes in each chromosome through bits flipping depending 𝑐𝑜𝑘𝑖𝑑𝑠(𝑛) = 𝑝1 𝑥𝑜𝑟 𝑝2 (4)
on the mutation probability. Using the steps in Fig. 6, GA-
based feature selection is explained in this section [6]. Where n is an index that is from 1 to number of kids, p1 is first
parent chromosome and p2 is second parent chromosome.
TABLE 1 COMPARATIVE TERMINOLOGY BETWEEN HUMAN GENETIC AND
GA.
f. Mutation function
Mutation is genetic perturbation of individuals in a
S.No. Human genetic GA terminology population. Mutation ensures genetic diversity and searching
1 Chromosomes Bit strings of broader solution. In this paper uniform mutation is used and
2 Genes Features GA generates set of random numbers from uniform
3 Allele Feature value distribution.
4 Locus Bit position
g. Termination of GA
5 Genotype Encoded string
6 pheotype Decoded genotype Once the GA reaches the optimum solution, it stops. TWO
stopping conditions are applicable: i) Maximum number of
generation. Here this value is 100. ii) Stall generation limit. Its
TABLE 2 PARAMETERS USED IN GA value is 0.000001.

Sr. No. GA Parameter Value B. WEKA Correlation-based Feature Selection ( CFS)

1 Population size
2 Genome length
3 Population Type
4 Fitness function
5 No. of generations
6 Crossover
7 Crossover Probability
8 Mutation Probability
9 Selection scheme
10 Elite Count
.
164
164 The two WEKA evaluators used are correlation feature
Bit strings selection subset evaluator (CFS-CE) and ranker i.e.
KNN –based classifier error Information Gain (IG). CFS uses a search algorithm like most
100 of the feature selector algorithm used. This algorithm also
Arithmetic evaluates the merit of feature subset. Selection of features is on
.8
0.1
the basis of correlation with the class and yet uncorrelated to
Tournament of size 2 each other [7]. WEKA ranker (IG) is filter-based technique to
2 reduce feature size.

B. Selection of features using GA based

Table 2 shows the configuration of GA. Seven step criteria
for implementation GA is shown in Fig. 6. Description of each
step is given below
a. Generation of starting population
Starting population for this algorithm is a matrix of dimension
Population size x chromosome length generated by random
binary digit.
b. Generation of children for new population
Elite count ≤ population size. With its size of 2, GA
automatic picks the top two best chromosomes and push them
auto to the next generation.
c.New population on which GA performs are
i) Elitism ii) Cross over iii) Mutation
If total population is 100 and Elite = 2 then Crossover is 98 x
Fig. 4. Plot of Fitness value vs generation for 3D DWT features ( CUAVE
0.8=72 and remaining is mutation i.e. 20. data base)
c. Fitness evaluation
Equation (3) is used for fitness evaluation
𝑎
𝑓 = + 𝑒 −1/𝑁 (3)
𝑁
Where 𝑎 is KNN classification error and N is cardinality of
feature vector.
d. Selection mechanism
Tournament of size 2 is used in this work due to its simplicity,
speed, and efficiency [6]. Two chromosomes are selected from

Authorized licensed use limited to: KIIT University. Downloaded on July 17,2023 at 04:24:18 UTC from IEEE Xplore. Restrictions apply.
 TABLE 3 RECO. ACCURACY USING GA AND IG FEATURE SELECTION FOR
3D-DWT FEATURES. (CUAVE AND TULIP DATABASE)

Database CUAVE Tulips

S.No. Selector+Classifier Reco. Reco.
Accuracies(%) Accuracies
(%)
1 GA+BPNN 78.42 73.95
2 GA+SVM 73.71 69.7
3 GA+KNN 70.85 63.54
4 IG+BPNN 68.28 72.91
5 IG+SVM 68.14 67.5
6 IG+KNN 69.42 61.45
7 CFS+BPNN 73.42 73.9
8 CFS+SVM 69 68.5

Fig. 5. Plot of Fitness value vs generation for 3-D DCT ( CUAVE data 9 CFS+KNN 70.28 61.45
base)

TABLE 4 RECO. ACCURACY USING GA AND IG FEATURE SELECTION FOR 3D-

IV. CORPUS AND RESULT
DCT FEATURES (CUAVE DATABASE)

Database CUAVE
A. CUAVE database S.No. Selector+Classifier Reco. Accuracies(%)
1 GA+BPNN 74.00
CUAVE [15] (Clemson University Audio Visual 2 GA+SVM 72.85
Experiments) was recorded by E. K. Pattererson of 3 GA+KNN 68.28
Department of Electrical and Computer Engineering, Clemson 4 IG+BPNN 69.14
University, US. The database was recorded in an isolated 5 IG+SVM 72.41
sound booth at a resolution of 720 x 480 with the NTSC 6 IG+KNN 70.28
7 CFS+BPNN 70
standard of 29.97 frames per second using 1 Megapixel-CCD
8 CFS+SVM 71.41
camera. This database is a speaker-independent database 9 CFS+KNN 70.28
consisting of connected and continuous digits spoken in
different situations. The database consists of two major V. CONCLUSION
sections: one of speaker pairs and the other one of
individuals.
In most cases, the difference in the classification accuracy
B. Tulips Database reported by the two approaches is very small. The features
Tulips are a small audiovisual database of 12 subjects saying selected by both method on the CUAVE and Tulips database
the first 4 digits in English. Subjects are undergraduate respectively. In overall, both the GA method and WEKA-CFS
students from the Cognitive Science Program at UCSD. The which are wrapper-based feature selectors produced better
database was compiled at R. Movellan's laboratory. Tulips classification accuracy than WEKA ranker (IG) which is
contains the video files in 100 x 75 pixel 8 bit gray level. filter-based. The main advantage of the method herein lies in
Each frame corresponds to 30 frames per second. R. Movellan the area of controllability as the GA can be fine-tuned to
presents work on speaker independent visual speech produce better results all the time by changing the fitness
recognition system and used simple HMM as a classifier used functions. Of all the features selected. With the application of
Tulips database of 1 to 4 digits for testing result [16]. GA for dimensionality reduction, more discriminating features
were obtained. After feature selection with GA the
C. Results performance of 3D-DWT is better for BPNN classifier. In
Results of GA are compared to two WEKA based feature future combination of 3D DWT and 3D DCT are used get
reduction technique The reduced feature vector are tested most discriminative feature between them.
classifiers such as SVM, BPNN, KNN. Figs. (4-5) show the
convergence of GA based on the chosen fitness function.
From the Figs. (4-5) fitness evaluation value is small for 3D
DWT as compared to 3D DCT. This is also reflected in
performance of 3D DWT. The performance of 3D DWT is
better as compared to 3D DCT. After GA based feature
selection features are reduced to 32% of original features.

Authorized licensed use limited to: KIIT University. Downloaded on July 17,2023 at 04:24:18 UTC from IEEE Xplore. Restrictions apply.
 Start

Normalised frames for digits

Feature Vectors using 3D

DWT / DCT

N X 164 chromosomes

Fitness function

Selection of parent
chromosomes

Crossover Mutation
Elite Kids Kids Kids

Best Termination
Chromosom condition

Reduce
feature vector Fitness
Evaluation

End
New Population

Fig. 6. Flowchart for GA based feature selection

REFERENCES Department of Information Engineering and Computer Science,

[1] E. D. Petajan, Automatic lip-reading to enhance speech
recognition, Ph.D. Thesis University of Illinois, 1984.
[2] Potamianos, H. Graf, and E. Cosatto, “An image transform
approach for HMM based automatic lip reading,” International
Conference on Image Processing, 173–177, 1998.
[3] R. Seymour, D. Stewart, and Ji Ming, “Comparison of image
transform-based features for visual speech recognition in clean
and corrupted videos,” EURASIP Journal on Video Processing,
Vol. 2008, 1-9, 2008.
[4] G. F. Meyor, J. B. Mulligan and S. M. Wuerger, “Continuous
audio-visual using N test decision fusion,” Elsevier Journal on
Information Fusion, 91-100, 2004.
[5] B. L. Persello “A novel approach to the selection of robust and
invariant features for classification of hyperspectral images,”
University of Trento, 2010.
[6] B. Oluleye , A. Leisa, J. Leng, D. Dean, “A genetic algorithm-
based Feature Selection”, IJECCE, Vol.5, 899-905, 2014.
[7] M. Hall, E. Frank, G. Holmes, B. Pfahringer. P. Reutemann and
H. Ian “The WEKA Data Mining Software: An Update; SIGKDD
Explorations,” Vol. 11, 1, 2009.
[8] V. long and L. Gang “Selection of the best wavelet base for
speech signal” IEEE Intelligent multimedia, video and speech
processing, 2004.
[9] K. Min and M. Fac, “A lip reading method based 3-D DCT and
3-D HMM” IEEE conf. ICIOE, 115-119, 2012.
[10] M. C.Weeks “Architectures For The 3-D Discrete Wavelet
Transform” Ph.D. Thesis University of Southwestern Louisiana
,1998.

Authorized licensed use limited to: KIIT University. Downloaded on July 17,2023 at 04:24:18 UTC from IEEE Xplore. Restrictions apply.
[11] S. Morade and S. Patnaik, “Lip reading by using 3-D Discrete
Wavelet Transform with Dmey wavelet” , IJIP, Vol 8, 385-396,
2014.
[12] J.Tian, Q. Hu, X. Ma and M.Ha “An Improved KPCA/GA-SVM
Classication Model for Plant Leaf Disease Recognition Journal of
Computational Information Systems” 18, 7737-7745, 2012.
[13] Melanie M. “An Introduction to Genetic Algorithms A Bradford
Book” The MIT Press, 1999.
[14] N. Sivanandam and S. N. Deepa,” Introduction to Genetic
Algorithms Springer-Verlag” , Berlin, Heidelberg, 2008
Authorized licensed use limited to: KIIT University. Downloaded on July 17,2023 at 04:24:18 UTC from IEEE Xplore. Restrictions apply.
[15] E. Patterson, S. Gurbuz, Z. Tufekci and J. Gowdy, “CUAVE: a
new audio-visual database for multimodal human computer-
interface research”, Proceedings of IEEE International
conference on Acoustics, speech and Signal Processing, 2017-
2020, 2002.
[16] J. R. Movellan “ Visual Speech Recognition with Stochastic
Networks” Advances in Neural Information Processing Systems,
MIT Pess Cambridge, Vol. 7, 1995.