Multimodal fusion of Face and Palm using CLBP and Haralick Features

Vani Hiremani, Asst. Prof Vijeeta Patil, Asst. Prof

Amity University, K.L.E Institute of Technology,
Jharkhand Hubballi, Karnataka

Abstract - Biometrics is a field of science and technology whose goal is the identification of a person’s identity based on
his or her physiological or behavioral characteristics. Commonly used physiological characteristics in biometrics are
fingerprint, palmprint, hand geometry, face, iris, retina and ear. Behavioral characteristics include signature, voice, lip
movement, gait, gestures and keystroke dynamics. Among these palmprint recognition and face recognition has received
considerable research interest because of many attractive advantages, such as rich and smooth texture feature, stable line
feature, low-resolution imaging, low-cost capturing devices, and easy self-positioning. face identification has attracted
many researchers due to its unique advantage such as non-contact process for feature acquisition. Variations in
illumination, pose and expression are major challenges of face recognition especially when images are taken as gray
scale. To alleviate these challenges to some extent many research works have been carried out by considering color
images and they have yielded better face recognition rate. A method for recognizing face using color local texture
features is described. Experimental results show that Face identification approaches using color local texture features
impressively yield better recognition rates than Face recognition approaches using only color or texture information.
Particularly, compared with grayscale texture features, the proposed color local texture features are able to provide
excellent recognition rates for face images taken under severe variations in illumination and also for low resolution face
images and palmprint recognition is expected to have a wide range of potential security applications, such as access
control, network security, and social security. The other biometric system uses palmprint as trait for the identification
and authentication of persons. The goal of the project is to design an efficient biometric system to extract Haralick
features and use of probabilistic neural networks for authentication using palmprint biometric trait. PolyUdatabase
samples are obtained from around 200 users each user’s 2 samples are acquired. This palm print biometric detects the
fake palmprint made of POP (plaster of paris) and differentiates between living and non-living based on the entropy
feature. Experimental results depict that the eleven Haralick feature values are obtained in implementation stage and
efficient accuracy is attained.

Key Words: Haralick features, Palmprint, Matching, Color space conversion, Local binary pattern, Color local texture
features, Face recognition, Manhattan distance, probabilistic neural network (PNN).
1. INTRODUCTION

In day-to-day lifestyle business, commercial and work activities are computerized. E-commerce applications like e-
Banking, e-shopping, e-learning and security applications like building access demand, maintaining security reception and
office, real time and efficient personal identification work on approaches like lore based approach use “something that
you simply know”(examples like passwords and PINs) and Token based approach use “something that you simply
have”(examples like passports and credit cards) both are tedious , time consuming, ineffective, expensive and difficult to
recollect. These disadvantages have led to Biometric recognition and verification systems which may be segmented into
three types: Texture Based, Line Based and Code Based. when line and coding methods are weighed together texture
method is best because line based method use only the principle lines of the palm are found difficult to acknowledge the
person [1] and coding based methods are complex in nature whereas texture based methods are known for the unique
characteristics of the palm and provides the adverse leads to contributing to the accuracy rate of the system. Imposter palm
prints, those made from plaster and olefin material are often misused by systems and their knowledge is often leaked.
These problems are resolved in proposed system using Haralick features whose main advantage is aliveness detection [2].
The Haralick features are the grey level co-occurrence matrices and are 11 in number which are further detailed within the
following sections.
In current age face recognition has received unique attention from both research communities and therefore the market,
but still remains challenging in real time applications. Varieties of typical algorithms [3] are presented to meet the
challenges and being categorized into appearance based, model-based schemes and template matching schemes [4]. In
appearance-based methods, object is represented in terms of several views, in model-based scheme face model is made to
capture facial variations using prior knowledge of face, whereas template matching method uses the co-relation between
input images and stored standard patterns of faces or features. Existing face recognition systems with gray-scaled images
lead to worst face recognition rate in case of low resolution, variations in illumination and shape cues. To address this
complementary effect is taken by using color and texture information. The opponent color [5] texture features that capture
the texture patterns of spatial interactions between spectral channels are incorporated into the generation of color local
binary pattern (CLBP) [6]. Local texture feature are powerful face descriptors because they're believed to be more robust
to variations of facial pose, expression, etc. especially, local binary pattern (LBP) texture features have proven to be
highly discriminative for FR because of different levels of locality. There has been a limited but increasing amount of
labor on the color aspects of textured image analysis [7-10]. The success rate indicates that color information can play a
complementary role in texture analysis and classification/recognition and consequently are often used to enhance
classification/recognition performance.
There are several challenges in recognition systems beginning from image acquisition to complex environment. These
issues are often broadly categorized as follows: separating the image from an unknown background, extracting the region
of interest from the image consistently during a translation, rotation, and scale invariant fashion and recognizing matches
correctly regardless of differences in lighting or blurriness, dust particles are thoroughly addressed within the course of
this work.

2. PALMPRINT BIOMETRIC SYSTEM: Using Haralick features

The proposed model for palmprint biometric system consists of 5 modules (as shown in figure 1): Image Acquisition,
Preprocessing, Feature Extraction, Template matching and Decision in both training and testing phase.

Figure 1: Proposed model

2.1. IMAGE ACQUISITION

In this phase we have collected palm print samples from Mspalmprint and PolyUdatabase database. The Mspalmprint
database consists of large scale multispectral palm print images captured under blue, green, red and near-infrared (NIR)
illuminations. These images are collected from 250 volunteers, including 195 males and 55 females. The age distribution
is from 20 to 60 years old. As per 6 images per candidate, this database contains 6,000 images of 500 unique palms for
one illumination condition. The PolyUdatabase contains high resolution palm print images around 772 distinguishing
images.

2.2. PREPROCESSING

This phase defines as the process of high lightning and enhancing the palm images. Here the contrast and brightness of
the images are adjusted as per convenience and also the dust and noise particles are removed from training samples.

The stages in the Preprocessing are as follows:

1. As per the need apply either a low pass or high pass filter show in eq. (1) & (2) to the original image.

xfilt = filter (a, [1 a-1], x) …eq.(1)

where a = T/τ, T = the time between samples, and τ (tau) is the filter time constant.
And the corresponding high-pass filter is,

xfilt = filter ([1-a, a-1], [1, a-1], x) …eq.(2)

2. The negation of an image with grey levels in the range [0, v-1] is obtained by the negative transformation which is
given by the expressions,

v-1–r …eq.(3)

This expression results in reversing of the grey level intensities of the image thereby producing a negative image. The
output of this function is directly mapped into the grey level co-occurrence matrices which act as basic steps for Haralick
feature extraction consisting values from 0 to v-1. This includes in the second stage of preprocessing as shown in the
figure 6.

3. The last stage is the extraction of region of interest [ROI] as shown in the figure 7 by noting the points near to fingers
and using axes operations to separate the ROI.

(a) (b)

Marked image

(c)

2) Palm ROI
(d)
Figure. 2. (a). Original image (b) Filtered image (c). Negated enhanced image (d) extraction of ROI

2.3. Feature Extraction

Haralick features mainly concentrate on the “aliveness detection” in the sense it works out on the fake palmprints those
made of plaster and olefin material differentiates between them. Thus, helps to avoid the unauthorized and illegal use
of biometric systems. In this section the Haralick features are extracted from the ROI which resulted in the preprocessing
module. Haralick features extracted from an image texture are obtained from the grey-level co- occurrence matrices
(GLCMs) that contain information about the statistical distribution of grey levels in the analyzed image [13], [14].

The co-occurrence matrix P(d, d', δ, Θ), size D x D, where D is a cardinality of a set of quantized grey-level values LD of
an image, contains at the position (d, d'), a number of occurrences of a pair of pixels that are at a distance, in the direction
Θ = 0.0, 300, 700, 1500, where a unique pixel has a grey-level value d ∈ LD, and consecutive pixel has a grey-level
value d' ∈ LD. The GLCM has to be normalized in such a wa y that each element of the matrix is divided by
R, where R is the sum of all the elements in the matrix. Based on the normalized GLCM, Haralick has proposed 14
statistical features, among these we concentrate only on the 11 features that can be calculated for each δ and Θ detailing
below from eq. (4) to eq. (14):
Notations:
Qi, j = i, jth entry in a normalized gray-tone spatial dependence matrix, = Qi, j/ R or P(i,j)=(I,j)th entry in a normalized gray-tone spatial dependence matrix. pxi
Ng
= ith entry in the marginal-probability matrix obtained by summing the rows of Q (i, j) = ∑ Q ( ⅈ , j ), Ng is the Number of distinct gray levels in the quantized
J =1
image.

The haralick features are represented as follows

1.Angular second moment

2
f 1=∑ ∑ ( Q ( i , j ) ) …
i j
eq. (4)
2. Contrast
Ng−1 Ng Ng
2
f 2= ∑ n ∑ ∑ Q (i , j) …
n=0 i=1 j=1
eq. (5)
3. Correlation

…eq. (6)
4.Sum of squares : variance

...eq. (7)
5.Inverse difference moment

…eq. (8)
6.Sum average

…eq. (9)
7. Sum variance

…eq. (10)
8. Sum entropy

…eq. (11)
9. Entropy
 …eq. (12)
10. Difference variance

…eq. (13)
11. Difference entropy

…eq. (14)
2.4. MATCHING

The use of probabilistic neural network classifier in the training and testing phase in the matching module is done. A
Probabilistic Neural Network (PNN) is largely a classifier which maps any input pattern to a wide number of
classifications and segmentations that can be produced into a more function approximates. Thus, a PNN is an
implementation of an algorithm called KDA-kernel discriminant analysis where the operations are organized into a
multilayered feed forward network with four layers that are used in this palmprint biometric system.

 Input layer

 Pattern layer

 Summation layer

 Output layer

In our system the input layer of PNN compares with the trained samples and within the pattern layer, the haralick
features extracted are dealt with the summation for comparison with the tested samples and output layer identifies
because the recognized person which was trained earlier. a number of the benefits of PNN are fast training process,
orders of magnitude faster than backpropagation an inherently parallel structure bound to converge to an optimal
classifier because the size of the representative training set increases, no local minimal issues, training samples are often
added or removed without extensive retraining.

3. Face Recognition system: Using Color Local Texture Features

The steps involved in proposed color FR framework using color local texture features are shown in Figure 3. It consists of 5 modules
like face image acquisition, face detection (i.e. ROI segmentation), color space conversion, feature extraction and
classification/identification with the assistance of Manhattan distance classifier.

Figure. 3 Framework for FR based on color local texture features.

In presented method, the important time face image is acquired from the user through the online camera. Firstly, the acquired color
face images are converted into grayscale then preprocessed to get rid of the noise. The noise free grayscale face image is converted
into binary image. Further, region of interest (ROI) face image is segmented/detected from the background of binary image. By
cropping and resizing the detected face image form the first color image, is converted in to YRGB color space. Each channel of
YRGB color space is split in to the local regions. The LBP function is applied to each block of the color component images
separately. The histogram of every block is integrated to border the feature vector. so as to spot the claimed user, the feature vectors
extracted from all the users of the system are matched with the feature vector of the testing face image. The detailed description of
every module is presented within the following subsections.
3.1. Image acquisition

The first phase of any security system is to accumulate the user face remotely from the camera as shown in figure 4. an
internet camera i.e., Hp-101 with a maximum resolution of two mega pixels is employed to accumulate the face image of
320x 240 pixel resolution of size. The color face database, in RGB color space is constructed by capturing faces from
100 users. so as to create the face database of users within the age between 10 years to 60 years, 10 face images of each
user is acquired with the variation within the illumination, face and pose. A real-time face image is captured within the
testing phase of our proposed work.

Figure 4. Image capturing through webcam.

2.2. Face image alignment and detection

A face image captured through web camera is 320x240 pixel resolution of size. The captured colored face image may
contain the complex background, variation in pose, variation in illumination. It’s required to eliminate the background
and detect the region of interest (ROI) i.e. face image and enhance the detected face image. Hence, the face image
alignment and detection phase is used for this purpose. Alignment is administered in-order to detect and localize the
region of interest i.e. face. Initially, preprocessing methods are applied on captured image before detection. Firstly the
color face image is converted into grayscale image successively to scale back the complexity and to extend the
processing speed. Further, grayscale image intensity is subtracted from original color image intensities to retain color
information. For the resulting face image in-order to preserve edges and to get rid of salt-pepper noise, the median
filter is applied. A color map/image intensity values are adjusted to fill-in all small gaps and blacken the isolated white
pixels and to scale back the blurring of edges. Further the resultant face image is converted into binary image with
intensity value

Figure 10. ROI of face image.

Figure 11. Color Space Conversion (a) Y color component (b) R color component (c) G color component (d) B color component.
 Figure 1 2 . LBP operation performed on each of the color component image (a) Y color component (b) R color component
(c) G color component (d) B color component.

[0 (black), 1 (white)], where “0” represents non skin region and “1” skin region. The face detection is performed by
employing the bounding box on the ROI face image by tracing with 8-connected components method. Further, the ROI
face image is cropped and resized to 128 × 128 pixel resolution of size as depicted in figure 10.

2.3 Color space conversion

Each of the colour space has different discriminating power, to reinforce discriminating power of color space for better
face recognition it's needed to convert weaker color spaces into powerful ones. The RGB color space is used within the
proposed work, as color space normalization technique (CSN) has significantly and consistently improved the
discriminating power of RGB color space. The cropped and resized RGB color image is converted into R, G, and B
channel images to retain the essential color information. The Y channel image is chosen from the YCbCr color space in-
order to urge the essential gray scale information during the LBP operation for skin region detection. The Y channel
conversion is expressed in equation (1),

Y = 0.318 R + 0.487G + 0.124B (1)

The resulting images from the colour space conversion i.e. Y color component, R color component, G color component
and B color component respectively are as shown in figure 11.

2.4 Extraction of Color Local Texture Features

A method of extracting the proposed color local texture features from a color image i.e., LBP is described during this
section. Each color component image is split into 256 cells (16 16 rows and columns respectively) i.e., each cell
consists of 8 8 pixels resolutions. The LBP function is applied to every block of the colour component image. The
resulting images after employing the LBP function is as shown in figure 12. The feature vector is made from the 256
intensity values computed from the histogram generated using Y, R, G and B color components of the individual
instances of the face images. six instances of face images are used for phase1 called as training. Hence, the dimensions
of template for 100 users are 600x 256.
The local binary pattern (LBP) texture operator has become a well-liked approach in various applications thanks to
its discriminative power. The LBP is additionally found to be a strong feature for texture classification and robustness
to

monotonic grayscale changes caused (i.e., by illumination variations), and its computational simplicity which makes
it possible to research images in challenging real-time scenarios.
The LBP feature vector, in its simplest form, is made within the following manner:

Step 1. Divide the Y, R, G and B color component face images into 256 cells i.e., 8x8 pixel resolution for every cell.
Step 2. Compare center pixel of a cell with its 8 neighboring pixels along a circular manner in clockwise direction.
Step 3. If the middle pixel’s value is bigger than its surrounding neighbor, then label with “1” otherwise, label with
“0”.
Step 4. Compute the histogram, over the cell, of the frequency of every “number” occurring.
Step 5. Normalize the histogram obtained from Step 4.
Step 6. Combine the remaining normalized histograms of all cells together to produce the feature vector.

During the development of LBP feature vector, the bilinear interpolation method is adopted on LBP grid to estimate
the values of neighbors that don't fall exactly on pixels. Since correlation between pixels decreases with distance, much
of the feel information in a picture are often obtained from local neighborhoods.
Further, the opponent color texture features also are extracted by capturing the feel patterns of spatial interactions
between spectral channels All pairs of color channels i.e., opponent colors are perceived as opposing pairs by humans:
red-green and yellow-blue. within the opponent color LBP, the LBP operator is applied on each color channel
separately. additionally, each pair of color channels is employed in collecting opponent color patterns in order that the
middle pixel for an area and therefore the neighborhood itself are taken from different color channels. In OCLBP, the
middle pixel is con- sidered from the red channel. Hence collectively three inter-channel LBP histograms and 6 intra-
channel histograms are extracted. Therefore, the nine-opponent final color texture features are extracted from the
face image.
2.5 Combining color local texture features

This section suggests the way of mixing (i.e. fusion) color local texture features for achieving the simplest FR perfor-
mance. the color local texture feature vector constructed from the Y, R, G and B color channels i.e. 256 histogram
features and therefore the opponent color texture features extracted from three inter-channels and 6 intra-channels i.e. 9
histogram features are concatenated. Hence, the ultimate feature vector constructed for every instance of the user face
has 265 histogram features. The matching module compares the feature vector extracted from the testing face image
with the stored template constructed during the training using Manhattan distance. Manhattan distance is formulated as
below:
n Σ
d= |xi − yi | (2)
i=1
where n=265 is that the dimension of the feature vector, xi is that the i th component of the sample feature vector,
and yi is that the i th component of the template feature vector.
The matching score level fusion method is that the commonest approach thanks to the convenience in accessing and
mixing the scores generated by different matches. The matching scores generated by the matching module are finally
used for decision. because the user authentication/identification is one-to-many matching, the feature vector of the
individual are compared with the feature vectors of each individual stored within the template database. The Manhattan
distance between input face image and therefore the feature vectors of all the face images within the database are
calculated. Finally the simplest Matching face image is identified using the score of minimum Manhattan distance.

3.Experimental Results

In the training phase of our proposed model the extraction of the 11 haralick features for the polyudatabase is obtained
within the TABLE 1 prescribed. The table contains individual person’s two samples and values of haralick features when
considered an equivalent two samples for one person there's a touch variation within the values but somehow it differs
tons when considered for both different person supported this we differentiate one person from another person.the
system is tested for the 100 samples of polyUdatabase.

The features are extracted are explained here angular moment feature may be a measure of the smoothness of the
image. and measure of homogeneity of a picture. This scenario will contain a couple of gray levels, GLCM with
only a few but high values called as P(i, j). Thus, the sum of squares are going to be high.inverse difference moment
is additionally influenced by the homogeneity of the image. due to the weighting factor it'll get small contributions
from inhomogeneous areas. Contrast may be a measure of the intensity contrast between a pixel and its neighbor over
the entire image. The variance may be a parameter describing partially either the particular probability distribution of
an observed population of numbers(pixels), or the theoretical probability distribution of a sample (a not-fully-observed
population) of numbers(pixels).Correlation may be a measure of gray level linear dependence between the pixels at
the required positions relative to every other.and the other feature variance feature puts relatively high weights on the
weather that differ from the typical value. Entropy may be a statistical measure of randomness which will be wont to
characterize the feel of the input image.Thus depending upon the varying values of those haralick features our system
successfully authenticates and verifies the person’s identity with the approximated results mentioned within the next
section.
 TABLE 2: HARALICK FEATURES EXTRACTED OF DIFFERENT SAMPLES
Samples Person 1 sample 1 Person 1 sample 2 Person2 sample 1 Person 2 sample 2
Angular second moment 0.17 0.20 0.23 0.24
Contrast 79.85 75.60 72.70 72.67
Correlation 0.25 0.20 0.3 0.20
Sum of Squares 198 199.2 196.6 192.08
Sum variance (var.) 395.84 397.14 389.4 389.39

Sum entropy -0.13 -0.20 -0.19 -0.19

Entropy 4.79 4.56 4.3 4.29

Diff. var. 0.72 0.73 0.78 0.75
Diff. entropy -0.17 -0.18 -0.26 -0.21
Sum average 396.69 394.98 390.6 388.3

The proposed work is administered by capturing face image through webcam, initially, during training six face images

from every user captured in several constraints like variations in illumination, change within the countenance and pose.

During the testing, from each user four instances of face are used with various constraints. during this experiment, the

Y, R, G and B color channels were adopted for extracting color texture features.

The developed method is tested with four face images taken from each user with variation in illumination, pose and

expressions; using CLBP features the real Acceptance Rate (GAR) of around 96% is achieved. The performance of the

proposed model is measured using GAR and False Acceptance Rate (FAR) [15]. GAR is that the ratio of total

number of correctly identified users by the system to the entire number of users attempted for identification. it's stated

as follows:

Number of Correctly Identified Users

GAR = 100% (3)

Total Number of Attempts

FAR is the ratio of number of wrongly identified users to the total number of users attempted for identification.

Number of Wrongly Identified Users

FAR 100% (4)
=
Total Number of Attempts

Hence, by using the color local texture features the identification accuracy of around 96% is achieved and false accep- tance rate of

2% is obtained for the 25 users for real time face images. The performance of the proposed system is tabulated in

Table 3. Face recognition performance

 FAR % GAR %

1 25 4 96 4 2% 96%
2 50 4 190 10 5% 95%
3 75 4 284 16 5.3% 94.6%
4 100 4 379 21 5.2% 94.7%
”
 References

[1]J. You, W. Li, and D. Zhang, “Hierarchical Palmprint Identification via Multiple Feature Extraction,” Pattern
Recognition, vol. 35, no. 4, pp. 847-859, 2002.

[2] D. Zhang, Z. Guo, G. Lu, L. Zhang, Y. Liu, W. Zuo, “Online joint palmprint and palmvein verification”,
Expert System with Applications, doi:10.1016/j.eswa.2010.08.052, 2010.

[3] S.Palanikumar , M.Sasikumar And J.Rajeesh ,2011, ” Palm Print enhancement Using Discrete Curvelet
Transform,” International Journal Of Computer Science Issues (IJCSI)July 2011, Vol. 8 Issue 4, Pp.313,July 2011

[4]Proyecto Fin de Carrera and Ion Marques, 2010, “Face Recognition Algorithms”. June 16, 2010

[5] Rein Hsu, Anil K. Jain and Mohamed Abdel, 2002, “Face Detection in Color Images”. IEEE Transaction on
Pattern Analysis and Machine Intelligence, Vol. 24, No. 5, pp. 696–706, 2002.

[6]Jain and G. Healey, 1998, “A Multi-scale Representation Including Opponent Color Features for Texture
Recognition”, IEEE Transaction on Image Processing, Vol. 7, No. 1, pp. 124–128, Jan. 1998

[7] Topi Maenpaa and Matti Pietikainen, 2004, “Texture Analysis with Local Binary Patterns”, WSPC. May 13,
2004.
[8] T. Maenpaa and M. Pietikainen, 2004, “Classification with Color and Texture: Jointly or Separately,” Pattern
Recognition, Vol. 37, No. 8, pp. 1629–1640, August 2004.

[9] Drimbarean and P. F. Whelan, 2001, “Experiments in Color Texture Analysis”, Pattern Recognit. Lett., Vol.
22, No. 10, pp. 1161–1167, August 2001.

[10] Paschos, 2001, “Perceptually Uniform Color Spaces for Color Texture Analysis: An Empirical Evaluation,”
IEEE Transaction on Image Processing, Vol. 10, No. 6, pp. 932–937, June 2001.

[11]Wai Kin Kong and David Zhang,2002 “Palmprint Texture Analysis based on Low-Resolution Images for
Personal Authentication,”proceeding icpr '02 proceedings of the 16th international conference on pattern
recognition (icpr'02) vol. 3 ,pp. 30807 IEEE computer society, 2002

[12]Jun chang,changshui Zhang,Gang Rong 2001, “Palm Print Recognition Using Crease,”image processing, 2001.
proceedings. 2001 international conference on 2001 vol:3, pp: 234 – 237, 2001

[13] Anil K. Jain,And Jianjiang Feng ,2009 “Latent Palmprint Matching,” Ieee Trans. Pami Vol. 31, No. 6, Pp.
1032-1047, June, 2009.

[14]Rina Su, Yongping Zhang, Jianbo Fan, And Shaojing Fan ,2009 “ Palmprint Recognition Based On Gabor
Transforms And Invariant Moments,” Proceedings Of The Second Symposium International Computer Science
And Computational Technology(ISCSCT ’09) Pp. 254-257,2009

[15]Shanumukhappa A. Angadi and Sanjeevakumar M. Hatture, 2011, “A Novel Spectral Graph Theoretic
Approach to User Identification Using Hand Geometry”, International Journal of Machine Intelligence, Vol. 3,
Issue 4, pp. 282–288, December 2011