See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/251315883

Markov chain model for multimodal biometric rank fusion

Article  in  Signal Image and Video Processing · January 2011

DOI: 10.1007/s11760-011-0226-8

CITATIONS READS
35 100

Some of the authors of this publication are also working on these related projects:

Biometric image quality and face recognition View project

Social Behavioral Biometrics - Online User Recognition View project

All content following this page was uploaded by Marina L. Gavrilova on 01 October 2020.

The user has requested enhancement of the downloaded file.

 1

Markov Chain Model for Multimodal Biometric Rank Fusion

Md. Maruf Monwar and Marina Gavrilova

Department of Computer Science, University of Calgary

2500 University Drive NW, Calgary, AB T3B 2V3, Canada
Phone: 1 (403) 220 3532; Fax: 1 (403) 284 4707
e-mail: {mmmonwar, marina}@cpsc.ucalgary.ca

Abstract - Multimodal biometric aims at increasing the reliability of biometric systems through utilizing
more than one biometric in decision-making process. An effective fusion scheme is necessary for combining
information from various sources. Such information can be integrated at several distinct levels, such as sensor
level, feature level, match score level, rank level and decision level. In this paper, we present a multimodal
biometric system utilizing face, iris and ear features through rank level fusion method using novel Markov
Chain approach. We first apply Fisherimage technique to face and ear image databases for recognition and
Hough transform and Hamming distance techniques for iris image recognition. The main contribution is in
introducing Markov Chain approach for biometric rank aggregation. One of the distinctive features of this
method is that it satisfies the Condorcet criterion, which is essential in any fair rank aggregation system. The
experimentation shows superiority of the proposed approach to a other recently introduced rank aggregation
methods. The developed system can be effectively used by of security and intelligence services for controlling
access to prohibited areas and protecting important national or public information.
Key words - Pattern recognition, Multimodal biometric system, Rank level fusion, Markov chain.

1 Introduction

The biometric-based controlled access to the protected resources has emerged to offer higher
security and convenience to both government and private sectors. The optimal biometric recognition
method is one having the properties of distinctiveness, universality, permanence, acceptability, collect
ability, and resistance to circumvention [1]. No existing biometric system simultaneously meets all of
these requirements; however the use of more than one biometric can help to develop a system which
approaches those goals. The advantages of multimodal systems stem from the fact that there are
multiple sources of information. The most prominent implications of this are increased accuracy,
fewer enrolment problems and enhanced security.
The most immediate advantage of multimodal authentication is increased recognition accuracy.
Multimodal systems fuse information from more than one source, each of which offers additional
evidence about the authenticity of an identity claim. Therefore, one can have more confidence in the
authentication results [2].
Multibiometric systems can address the non-universality problem and reduce the FTER (Failure-
to-Enroll Rate) and FTCR (Failure-to-Capture Rate). For instance, it is estimated that 2% of the
population may not be able to provide a fingerprint due to medical/genetic conditions, accidental
destruction, or temporary damage [3]. That group of persons can still be recognized using other
biometric traits, like face or iris, in a multimodal biometric system. Poor quality data is a common
cause of enrolment errors. The multimodal approach provides the system with a ‘second chance’ to
obtain or match a sample of sufficient quality, thereby increasing the robustness of the system.
Enrolling multimodal biometrics has some drawbacks too. Multimodal biometrics enrolment process
is more complex and requires more time than a unimodal system.
 2

Multimodal biometric systems are more resistant to spoof attacks because it is difficult for the
attacker to simultaneously spoof multiple biometric sources. Further, a multimodal biometric system
can easily incorporate a challenge-response mechanism during biometric acquisition by acquiring a
subset of the traits in some random order. Such mechanism will ensure that the system is interacting
with a live user.
All multimodal biometric systems need a fusion module that takes individual data and combines it
in order to obtain the authentication result: impostor or genuine user. The fusion strategies are divided
into two main categories: pre-mapping fusion (before matching) and post-mapping fusion (after
matching) [4]. The first strategy deals with the sensor data fusion level and feature vector fusion level.
These techniques are not commonly used due to many implementation problems [5]. The second
strategy is realized through the decision level fusion, based on some algorithms which combine single
decisions for each component system, or through the matching score level fusion, which combines the
matching scores of each component system, or through the rank level fusion, which is used when the
output of each biometric system is a subset of possible matches (i.e., identities) sorted in decreasing
order of confidence. All the above after-mapping fusion schemes can also be defined as classifier
fusion schemes as information obtained from different classifiers is integrated.
Figure 1 shows a sample multimodal biometric system which recognizes an identity using face and
palmprint biometric traits. The sample system uses post-matching fusion scheme to integrate the
matching results.

Fig. 1. Sample multimodal biometric system architecture

In this paper, we present a novel method based on Markov Chain for rank level fusion multimodal
system comprising face, ear and iris biometrics. The reason behind using the rank level fusion is that
fusion at the rank level is a significantly understudied problem, which has a high potential for efficient
consolidation of ranked information obtained from multiple unimodal biometric matchers [6].
Very few researchers investigated this level of biometric integration and hence we prefer to
incorporate this fusion scheme in this multimodal biometric system. We further introduce a new
Markov Chain [7] approach for fusing rank information in multimodal biometric system, which has
improvements over other rank level methods in accuracy and reliability.
The rest of the paper is organized as follows. In section 2, we provide an overview of relevant
research on multimodal biometric systems. Section 3 discusses generic system architecture for various
rank level fusion approaches. Section 4 presents a new Markov chain approach and Section 5
illustrates the experimental setup. Section 6 discusses the conclusion and the future direction of
research.
 3

2 Literature Review

Due to some problems associated with the unimodal biometric system, such as little variation of
biometric samples over the whole population, large intra-variability over time, some features not
being present in the population etc., the use of multimodal biometrics has gained popularity in the
recent years. The main objective of a multimodal biometric system is to improve the recognition
performance of the system and to make the system robust [8].
Several approaches have been proposed and developed for multimodal biometric authentication
system with different biometric traits and with different fusion mechanisms. In 2001, Cattin et al.
developed a sensor fusion based multimodal biometric system for medium security applications
which authenticates people based on their gait [9]. They used four classifiers for gait feature
acquisition and the Bayes Risk Criterion which subsequently integrates the multiple classifiers. Their
multimodal approach significantly increases recognition robustness and reliability.
In 2004, Feng et al. [10] developed a system using face and palmprint using feature level fusion
strategy. For feature level fusion, authors used the feature concatenation approach. It is based on two
algorithms for classification – principal component analysis (PCA) and independent component
analysis (ICA). In 2006, Li et al. [11] introduced a new feature metric – handmetric, which is a
combination of palmprint, hand shape and knuckle print. They integrated the features of these three
biometrics using feature level fusion based on kernel principle component analysis (KPCA), which is
a combination of kernel projection and PCA dimension reduction. They used four fusion rules for
combining feature – sum, product, min and max.
In 1998, a bimodal approach was proposed by Hong and Jain for a Principal Component Analysis
(PCA) based face and minutiae-based fingerprint identification system with a fusion method at the
match score level (integrating the matching scores of different classifiers and make decision based on
the consensus matching scores) [12]. In 2005, Jain et al. proposed a multimodal approach for face,
fingerprint and hand geometry, with fusion at the score level [8]. The matching approaches for these
modalities are minutiae-based matcher for fingerprint, which has similarity scores as output, PCA-
based algorithm for face recognition, which has Euclidean distance as output, and a 14-dimensional
features vector for hand-geometry, which also has Euclidean distance as output. Seven score
normalization techniques (simple distance-t-similarity transformation with no change in scale, min-
max normalization, z-score normalization, median-MAD normalization, double sigmoid
normalization, Tanh normalization and Parzen normalization) and three fusion techniques on the
normalized scores (simple-sum-rule, max-rule and min-rule) were tested in this study. Except for one
normalization technique (the median-MAD), all fusion approaches outperform the monomodal
approaches. In 2008, Nandakumar et al. [13] presented a multimodal biometric system using
likelihood-ratio based match score level fusion. Authors proposed a framework for optimal
combination of match scores, based on the likelihood ratio test, which models the distributions of
genuine and impostor match scores as finite Gaussian mixture model.
In 2000, Frischholz and Dieckmann developed a commercial multimodal approach, BioID, for a
model-based face classifier, a vector quantization (VQ)-based voice classifier and an optical-flow-
based lip movement classifier for verifying persons [14]. Lip motion and face images were extracted
from a video sequence and the voice from an audio signal. Weighted sum rule and majority voting
approaches of decision level fusion method were used for fusion.
In 2009, Monwar and Gavrilova [15] studied logistic regression base rank level fusion method
for multimodal biometric system. They used face, ear and signature biometric traits to evaluate the
performance of rank level fusion. They assigned weights to classifiers based on the assumption that
their face classifier produces better results than other two classifiers and achieved very good equal
error rate (EER) for logistic regression rank level fusion method.
From the above discussion, it can be concluded that many multimodal biometric systems with
various methods and strategies have been proposed over the last decade to achieve higher accuracy
performance (some of the systems are summarized in Table 1). Thus, aiming at the same issue, i.e., to
reduce false acceptance and false rejection error rates, we develop a multibiometric system
incorporating three unimodal biometrics (face, ear and iris). To obtain the consensus rank of the
individuals by consolidating the ranking outputs produced by three unimodal experts, we
 4

Table 1. Examples of multimodal biometric systems.

Year Modalities Fused Authors Fusion Level Fusion Approach

1998 Face and fingerprint Hong and Jain [12] Match score Product rule
2000 Face, voice and leap Frischholz and Decision Weighted sum rule; Majority
movement Dieckmann [14] voting
2001 Gait Cattin et al. [9] Sensor Bayes risk criterion
2003 Face, fingerprint and Fierrez-Aguilar et al. Match score SVM-based learning
signature [16] approach
2003 face, fingerprint and Ross and Jain [17] Match Score Sum-rule, decision tree and
hand geometry linear discriminant function
2004 Face and palmprint Feng et al. [10] Feature Feature concatenation
2005 Face, fingerprint and Jain et al. [8] Match score Simple-sum-rule, max-rule
hand geometry and min-rule
2006 Palmprint, hand Li et al. [11] Feature Sum, product, min and max
shape and
knuckleprint
2008 Fingerprint, face and Nandakumar et al. Match score Likelihood ratio
hand geometry [13]
2009 Face, ear, signature Monwar and Rank Borda count, logistic
Gavrilova [15] regression, highest rank

study the performance of different rank level fusion methods, such as highest rank, Borda count,
logistic regression and a newly introduced Markov chain method. This approach brings a new
dimension to the current ways of biometric rank aggregation and can be effectively used in real-life
security applications.

3 Rank Level Fusion Methodology

Rank-level fusion is used in identification systems and is applicable when the individual matcher’s
output is a ranking of the “candidates” in the template database. The system is expected to assign a
higher rank to a template that is more similar to the query. Very few methods for consolidation of
biometric rank information can be found in the literature, as it is still an understudied problem. Three
methods described by Ho, Hull, and Srihari in [18] for making the final decision in a general multiple
classifier system, can be used for rank level fusion in multimodal biometric systems. These three
methods are highest rank, Borda count and logistic regression methods. Recently, Nandakumar and
others introduced Bayesian approach for rank level fusion [19]. All of these methods for rank level
fusion are briefly discussed in the next subsections.

3.1 Highest Rank Method

The highest rank method is good for combining a small number of specialized matchers and hence
can be effectively used for a multimodal biometric system where the individual matchers are the best.
In this method, the consensus ranking is obtained by sorting the identities according to their highest
rank.
Suppose we have m classifiers which assign ranks to all classes. Then the consensus rank of a
particular class is obtained by the following equation:
m
Consensus rank, Rc = min Ri (1)
i =1

The final identity authentication ranking is then obtained by sorting the consensus ranking of each
class in the ascending order.
 5

The advantage of this method is the ability to utilize the strength of each matcher. Even if only one
matcher assigns the highest rank to the correct user, it is still very likely that the correct user will
receive the highest rank after reordering. The disadvantage of this method is that the final ranking
may have many ties (which can be broken randomly). The number of classes sharing the same ranks
depends on the number of classifiers used. Due to this property this method cannot be a good choice
for a security critical multimodal biometric system [15].

3.2 Borda Count Method

The Borda count [20] method is the most widely used rank aggregation method and uses the sum
of the ranks assigned by individual matchers to calculate the final rank. Suppose we have m
classifiers. Then the consensus rank of a particular class is obtained by the following equation:

m
Consensus rank, Rc = ∑R
i =1
i (2)

The final identity authentication ranking is then obtained by sorting the consensus ranking of each
class in the ascending order.
This method assumes that the ranks assigned to the users by the individual matchers are
statistically independent and the performances of all three matchers are equal. The magnitude for the
Borda count for each class measures the strength of agreement by the three matchers that the input
pattern belongs to that class. The advantage of this method is that it is easy to implement and requires
no training stage. These properties made the Borda count method feasible to incorporate in
multimodal biometric systems. The disadvantage of this method is that it does not take into account
the differences in the individual matcher’s capabilities and assumes that all the matchers perform
equally, which is usually not the case in most real biometric systems.

3.3 Logistic Regression Method

The logistic regression method, which is a variation of the Borda count method, calculates the
weighted sum of the individual ranks. In this method, the final consensus rank is obtained by sorting
the identities according to the summation of their rankings obtained from individual matchers
multiplied by the assigned weight.
Suppose we have m classifiers which assign ranks to all classes. Then the consensus rank of a
particular class is obtained by the following equation:

m
Consensus rank, Rc = ∑W R
i =1
i i (3)

where, Wi is the weight assigned to the i-th classifier. The final identity authentication ranking is
then obtained by sorting the consensus ranking of each class in the ascending order.
The weight to be assigned to the different matchers is determined by a ‘logit’ function using
logistic regression [21]. This method is very useful when the different matchers have significant
differences in their accuracies but requires a training phase to determine the weights which can be
computationally expensive. Also one of the key factors that have direct effect on the performance of a
biometric system is the quality of the biometric samples. Hence the single matchers’ performance can
vary with different sample sets which make the weights allocating process more challenging and
inappropriate weight allocation can eventually reduce the recognition performance of this multimodal
biometric system (using logistic regression) compared to unimodal matchers. So, in some cases,
logistic regression method cannot be employed for rank aggregation.
 6

3.4 Bayesian Approach

Bayesian approach for biometric rank fusion is based on Bayes decision theory. This approach
uses the rank distribution (probability that a identity is assigned a rank by an individual matcher is a
true identity), which can be estimated when the marginal genuine and impostor match score densities
are known [19]. This estimation requires two assumptions - (i) match scores of the individual users
are independent and (ii) match score distributions of different users are identical. The consensus rank
is obtained as the product of the posterior probabilities of the individual matchers.
If we have m classifiers, then the consensus rank of a particular class is obtained by the following
equation:

m
Consensus rank, Rc = ∏ P (R )
i =1
i i (4)

where, Pi ( Ri ) is the posteriori probability that the identity (class) which is assigned rank Ri by
the i-th classifier is the true identity. The final identity authentication ranking is then obtained by
sorting the consensus ranking of each class in the descending order.
The size of the multimodal biometric database is usually huge and thus only the top few results are
usually considered for the final reordered ranking. Hence, a very common scenario of a rank based
multimodal biometric system is that some results may rank at top by a few classifiers and the rest of
the classifiers do not even output the result. In this situation, the above approaches cannot produce
good recognition performance.
To deal with these shortcomings, in this multimodal biometric research, we introduce Markov
chain rank aggregation method to find the consensus rank for person identification. Previously, this
approach has successfully been used in web search [7]. Due to the ease in implementation and its
successful usage in the web ranking, we decide to employ Markov chain approach for multimodal
biometric rank fusion.

4 Markov Chain Approach to Rank Level Fusion

We consider the biometric rank aggregation similarly to a voting mechanism. In a voting method
evaluation, the most important thing is to ensure the fairness of the voting system. Among the fairness
criteria, the two most important criteria are Condorcet Winner Criterion and the Condorcet Loser
Criterion [22].

Definition 4.1: Condorcet Winner Criterion - If there exists an alternative a which would win in
pairwise votes against each other alternative, then a should be declared the winner of the election.
Note that there is not necessarily such an alternative a. This alternative is called the Condorcet
winner.

Definition 4.2: Condorcet Loser Criterion - If there exists an alternative a which would loose in
pairwise votes against each other alternative, then a should not be declared the winner of the election.

None of the approaches described in section 3 ensure the election of Condorcet Winner. This
motivates us to employ Markov chain approach for biometric rank fusion in this multimodal biometric
system.

Definition 4.3: Markov Chain – Markov chain is a random process or set of states in which the
probability that a certain future state will occur depends only on the present or immediately preceding
state of the system, and not on the events leading up to the present state.

In the Markov chain biometric rank aggregation method, it is assumed that there exists a Markov
chain on the enrolled identities and the order relations between those identities in the ranking lists
 7

(obtained from different biometric matchers) represent the transitions in the Markov chain. The
stationary distribution of the Markov chain is then utilized to rank the entities [7]. The construction of
the consensus ranking list from the Markov chain can be summarized as below:

Step 1: Map the set of ranked lists to a single Markov chain, where
one node of the chain represents single identity in the
initial ranking lists.

Step 2: Compute the stationary distribution on the Markov chain.

Step 3: Rank the identities based on the stationary distribution.

That is, the node with the highest score in the stationary
distribution is given the top rank, and so on until the node
with the lowest score in the stationary distribution which is
given the last rank.

The proposed Markov chain approach for biometric rank aggregation has several advantages. This
method handles the partial ranking list very well and provides a more holistic viewpoint of comparing
all candidates against each other. To do so, this method uses only the available comparisons (in the
partial lists) between the identities to determine the transition probabilities and exploit the
connectivity of the chain to infer comparison outcomes between pairs that were not explicitly ranked
by any of the matcher. The Markov chain method also handles the uneven comparison, i.e., when the
results of the initial ranking lists are very much different. Heuristics for combining rankings are
motivated by some underlying principle and the Markov chain model can be viewed as the natural
extensions of those heuristics. For example, Borda’s method is based on the idea “more wins is
better.” It is natural to extend this and say “more wins against good players is even better,” and so on,
and iteratively refine the ordering produced by a heuristic. Some Markov chain models for biometric
rank aggregation can be viewed as the natural extensions of Borda’s method, sorting by geometric
mean or Copeland’s method (sort the candidates by the number of pairwise majority wins minus
pairwise majority losses. This amounts to sorting the nodes in the majority graph by outdegree minus
indegree) [23].
There are four specific Markov chains [7], which can be used for biometric rank aggregation. Then
the next state The four Markov chains are defined as follows (suppose the current state of a Markov
chain is a):

Definition 4.3.1: MC1 - Choose an identity b uniformly from multiset of all identities that were
ranked at least as high as a by some classifier.
Probability to stay at a: ~ average rank of a.

Definition 4.3.2: MC2 - Choose a classifier i uniformly at random and pick uniformly at random
from among the identities that the i-th classifier ranked at least as high as a.

Definition 4.3.3: MC3 - Choose a classifier i uniformly at random and pick uniformly at random
an identity b. If i-th classifier ranked b higher than a, go to b. Otherwise, stay in a.

Definition 4.3.4: MC4 - Choose an identity b uniformly at random. If most classifiers ranked b
higher than a, go to b. Otherwise, stay in a. So, the rank of a ~ # of “pairwise contests” a wins.
Among those four methods, the last method satisfies the Copeland method and according the
literature, the best performing one [7].

Definition 4.4: Copeland method – Sort the identities by the number of pairwise majority wins
minus pairwise majority losses. Copeland’s method satisfies the extended Condorcet condition, can
be computed in Ο(n 2 k ) time and is generalized by MC4.
_
 8

Fig. 2. Markov chain and the transition matrix constructed from three ranking lists based on MC4.

Figure 2 shows a Markov chain with its transition matrix build on MC4. For this example, let us
assume that four persons are to be classified by three classifiers. But each classifier outputs only the
first three results of their ranking list (i.e., each classifier outputs a partial list). From these partial
lists, a full list has been created. The missing items in the list can be inserted randomly or by
examining the partial lists. Based on these full lists, a transition matrix is created and then a Markov
chain is constructed according to MC4. The final ranking list (which satisfies the Condorcet criterion)
can be obtained by applying the Copeland method, i.e., by sorting the nodes in the majority graph
(Markov chain) by outdegree minus indegrree. The figure also shows that if we apply the Borda count
method to the lists, we obtain a final list which does not satisfy the Condorcet criteria.

5 The Proposed System

The design of a multimodal biometric system is strongly dependent on the application scenario. A
number of multimodal biometric systems have been proposed over the last decade that differ in terms
of their architecture, the number and choice of biometric modalities, the level at which the evidence is
accumulated, and the methods used for the integration or fusion of information.
This section presents the Markov chain based rank level fusion technique, and describes how
multimodal biometric system combining face, ear and iris as biometric traits, can be implemented.
Although ear is not a frequently used biometric trait, we selected it because we prefer to use
biometrics from the similar region of the human body due to accessibility of the data. Moreover, ear is
a viable new class of biometrics since it has desirable properties, such as universality, uniqueness, and
permanence [24].
All biometric traits, used by the system, are stored as images. For face and ear image recognition,
Fisherimage technique, which is a combination of PCA and LDA (linear discriminant analysis) [25]
has been used. Researchers have demonstrated that the LDA based algorithms outperform the PCA
algorithm for many different tasks when the image database has a lot of variation due to background
noise, image shift, occlusion of objects, scaling of the image, and illumination change [26]. As our
face and ear datasets contains face and ear images of the same subject with certain illumination
change, fisherimage method has been used to enhance individual biometric performance.
For iris recognition, the iris part of the eye image (from inside the limbus (outer boundary) and
outside the pupil (inner boundary)) are localized. Some methods proposed in the literature for iris
localization are using integrodifferential operator [27], Hough transform [28], active contour models
[29] and elastic model [30]. Among these, Hough transform and integrodifferential operator
 9

techniques have been most popular. Due to the simplicity, in the proposed multimodal biometric
system, Hough transform technique has been utilized for iris localization. After localizing the region
of interest, the Rubber Sheet Model [27] is used for un-wrapping the iris image (see Figure 3).

Definition 5.1: Rubber sheet model – the model that maps the iris image from Cartesian
coordinates to the normalized polar coordinates.

(a) (b)
Fig. 3 (a) Rubber sheet model (b) Rubber sheet model applied on iris database.

The rubber sheet model remaps each point within the iris region to a pair of polar coordinates (r,
Ө), where r is on the interval [0, 1] and Ө is an angle [0, 2π]. The remapping of the iris region from
(x,y) Cartesian coordinates to the normalized non-concentric polar representation is modelled as –

I (x(r, Ө), y(r, Ө)) I(r, Ө) (5)

with x(r, Ө) = (1-r)xp(Ө) + rx1(Ө)

y(r, Ө) = (1-r)yp(Ө) + ry1(Ө) (6)

Where I(x, y) is the iris region image, (x, y) are the original Cartesian coordinates, (r, Ө) are the
corresponding normalized polar coordinates, and xp, yp and x1, y1 and are the coordinates of the pupil
and iris boundaries along the Ө direction. The rubber sheet model takes into account pupil dilation
and size inconsistencies in order to produce a normalized representation with constant dimensions. In
this way, the iris region is modelled as a flexible rubber sheet anchored at the iris boundary with the
pupil centre as the reference point.
Next we encode the iris features by convolving the normalized iris pattern with 1D Log-Gabor
wavelets. The 2D normalized pattern is broken up into a number of 1D-signals and then these 1D
signals are convolved with 1D Gabor wavelets. The rows of the 2D normalized pattern are taken as
the 1D signal where each row corresponds to a circular ring on the iris region. The intensity values at
known noise areas in the normalized pattern are set to the average intensity of surrounding pixels to
prevent influence of noise in the output of the filtering. The output of filtering is then phase quantized
to four levels, with each filter producing two bits of data for each phase [27]. The output of phase
quantization is chosen to be a grey code, so that when going from one quadrant to another, only 1 bit
changes. After encoding, the binary data is available and is compared using Hamming distance
method.
 10

Definition 5.2: Hamming distance - The number of bits which differ between two binary strings.
More formally,

Hamming distance is defined as

Iris i ⊗ Iris t I Mask i I Mask t

HD = (7)
Mask i I Mask t

Where, Irisi and Irist are two irises to be compared and Maski and Maskt are mask for the two
irises. The masking is used according to the portions of segmented image which is not a part of an iris
(eyelash etc.).
The method is chosen for its simplicity over other methods such as weighted Euclidean distance
method and normalized correlation method.

Fig. 4. Proposed system architecture.

A detailed diagram of the developed multimodal biometric system is shown in Figure 4. During
enrolment phase, face, ear and iris images are acquired and then processed according to the training
and classification algorithms. Fisherimage method has been used for face and ear recognition. For
this, two projection matrices have been created, one for the face and one for the ear, whose
components can be viewed as images, referred to as fisherimages. These two projection matrices are
 11

the face templates and the ear templates. For iris recognition, first iris is extracted from the eye images
and then a binary iris code is generated from each iris using 2D Gabor filter. These iris codes are the
iris templates.
In identification phase, face and ear images have recognized measuring the Euclidian distance
between the test image and the images in the fisherfaces and fisherears. For iris, the Hamming
distance is calculated between the codes generated from the test iris with the iris codes in the
database. In each of the three cases, five identities are obtained as output that will be ranked according
to their distances. The identities of these three ranking list are then integrated using the rank level
fusion approach to find out a consensus rank of the identities. The identity at the top of the consensus
ranking list is then identified as the desired identity. For rank level fusion, in addition to evaluate
highest rank, Borda count and logistic regression approaches, we introduce Markov chain approach to
find out the consensus ranking from the three ranking lists.

6 Experiments and Results

This section describes the databases that are used to validate the proposed system and to quantify
the results obtained.
Due to the inherent cost and effort associated with constructing a multimodal database, most of
the multimodal biometric systems, employ a virtual database, which contains records created by
consistent pairing a user from one unimodal database (e.g., face) with a user from another database
(e.g., iris) [1]. The creation of virtual users is based on the assumption that different biometric traits of
the same person are independent [31]. In this work, we make the same assumption and use a virtual
database which contains data from three unimodal databases for iris, ear and face.
For iris, we use the CASIA Iris Image Database (ver 1.0) maintained by the Chinese Academy of
Science [32]. This version of CASIA database includes 756 black and white iris images from 108 eyes
(hence 108 classes). For each eye, 7 images are captured in two sessions, where three samples are
collected in the first session and four in the second session.
The ear images are from the USTB database [33]. The database contains ear images with
illumination variation and orientation variation and individuals were invited to be seated 2m from the
camera and change his/her face orientation. The images are 300 x 400 pixels in size. Due to the
different orientation and image pattern, the ear images from this database need normalization.
For face, the popular Facial Recognition Technology (FERET) database [34] is used, which
documentation lists 24 facial image categories. There are 14,051 images of 1199 person that are 256 x
384 in size. There are various face images with expression, pose and illumination variation.
To build the virtual multimodal database for the proposed system, all the classes (subjects) of each
datasets have been numbered. Then we randomly select same classes from each three datasets. The
images within the same class of three datasets are then paired to form a single class of our virtual
multimodal database. Half of the classes are chosen for training purpose and the rest are used for
testing purpose. Figure 5 shows a portion of our virtual multimodal database created from CASIA iris
dataset, FERET face dataset and USTB ear dataset.
After experimentation, we observe the results by plotting the recognition values on a Cumulative
Match Characteristic (CMC) curve. CMC curve is used to summarize the identification rate at
different rank values. As rank level fusion method can only be used for identification, we introduce
the identification rate which is the proportion of times the identity determined by the system is the
true identity of the user providing the query biometric sample. If the biometric system outputs the
identities of the top x matches, the rank-x identification rate is defined as the proportion of times the
true identity of the user is contained in the top m matching identities.
Figure 6 shows the CMC curves of the individual face, ear and iris matchers for Markov chain
and logistic regression rank fusion approaches. We investigate highest rank, Borda count, logistic
regression and Markov chain approaches on this virtual multimodal database and obtain the best
identification rates through Markov chain approach (+98.29%). Among the other three, logistic
regression approach is better (+97%). As the performances of our individual matchers are not equal,
 12

Fig 5. A small portion of our virtual multimodal database

hence the highest rank and Borda count approaches do not produce satisfactory identification results.
So, we report only the identification rates of Markov chain and logistic regression approaches. Also,
we report the identification rates of the face, iris and ear matchers on the CMC curves to show the
differences.

Fig. 6. CMC curve of Markov chain and logistic regression rank fusion approaches along with face, iris and
ear unimodal systems.

To compare the Markov chain method for rank fusion, we applied it on our second set of virtual
multimodal database (figure 7). In this database, we use a public domain ear database [35] which
contains 102 gray scale images (6 images for 17 subjects). The images were captured with a grey
scale CCD camera Kappa CF 4 (focal 16 mm, objective 25.5 mm, f-number 1.4-16) using the
program Vitec Multimedia Imager for VIDEO NT v1.52. Each raw image has a resolution of 384 x
288 pixels and 256-bit grey scales. The camera was at around 1.5 m from the subject. Six views of the
left profile from each subject were taken under uniform, diffuse lighting. Slight changes in the head
position were encouraged from image to image. There were 17 different subjects, all students or
professors at the Faculty of Informatics of the Technical University of Madrid. The raw images were
then cropped and rotated for uniformity (to a ratio height:width of 1.6), and slightly brightened
(gamma = 1.5 approx.), using the xv program in a Linux system.
The face data in our second virtual multimodal database is from the University of Essex, UK
Computer Vision Science Research Project [36]. There are 395 subjects in this face dataset with each
having 20 face images and almost all of them are undergraduate students (age range is 18-20). Each
image has a resolution of 180 x 200 pixels. The subjects are both male and female and the background
of the image is plain green. The lighting and expression variations in the image are very minimal.
Also there is no individual hairstyle variation as the images were taken in a single session. In figure 7,
the face images are distorted due to the requirements of the face dataset creator.
 13

Fig 7. A small portion of our second virtual multimodal database

The iris dataset in this database is from the Department of Computer Science at Palacky University
in Olomouc, Czech Republic [37]. This iris database contains 3 iris images of left eye and 3 iris
images of right eye of 64 subjects. The iris images are of 24 bit – RGB and have a resolution of 576 x
768 pixels. The irises were scanned by TOPCON TRC50IA optical device connected with SONY
DXC-950P 3CCD camera.
The CMC curves for Markov chain and logistic regression rank fusion method and the unimodal
matchers applied on our second virtual multimodal database are plotted on figure 8. In this database,
face dataset is comparatively clear and has very low background and lighting variation. This reflects
in the unimodal matchers’ results which show that the face matcher has the accuracy of 93.12%. The
Markov chain rank level fusion has the highest recognition performance with an accuracy of 95.98%.
This reduction in recognition performance compared to our virtual multimodal database one is due to
the individual matcher’s performance. As Markov chain method works on the ranking list produced
by the individual matchers, so performance reduction of unimodal matchers will result in performance
reduction by Markov chain rank fusion approach.

Fig. 8. CMC curve for our second virtual multimodal datasets.

Quick response is another performance issue of an identity authentication system. In any

overcrowded place, such as airports, amusement parks, cineplexes etc., an identity authentication
system with slower response time would not work. For this reason, in many real-world applications,
performance of a system needs to be compromised with response time. Due to processing large
amount of data, a multimodal biometric system normally takes longer time to make decision than a
single biometric system. Thus, the deployment of any single biometric system or multibiometric
system depends on the application area and the robustness of the system.
We compare the enrolment time and the authentication time for the developed system. The
experimental results show that iris enrolment and authentication is the fastest among the three
biometrics used in the system. In the case of multibiometrics, the highest rank fusion method is the
 14

fastest followed by Borda count method, logistic regression method and Markov chain method. The
reason behind this is that in Markov chain rank level fusion approach, the system needs to process
more data than in the other three rank fusion approaches. Table 2 shows the comparison of enrolment
and authentication time between different approaches.

Table 2. Response time comparison for different unimodal and multimodal biometric system.

Approaches Enrollment Time (min) Authentication Time (min)

Face/Ear 1.03 + 0.15 0.72 + 0.22

Iris .83 + 0.11 0.59 + 0.16
Highest rank 2.59 + 0.29 0.91 + 0.06
Borda count 2.59 + 0.29 0.99 + 0.11
Logistic regression 2.59 + 0.29 1.10 + 0.12
Markov chain 2.59 + 0.29 1.39 + 0.63

EER is another evaluation technique which is heavily used to measure the performance of any
biometric authentication system which is the value of the position in the receiver operating
characteristics (ROC) curve where FAR is equal to FRR. Figure 9 shows the FAR against the FRR for
different approaches of this multimodal biometric system (for first virtual multimodal database).

Fig. 9. Receiver operating characteristics (ROC) curves for different rank level fusion approaches.

From the above figure, it can be seen that rank fusion techniques result in better EER than single
biometric system. Note that in this figure, only the ROC curve of iris is shown as this is the best
performing single biometric for our first virtual multimodal database. Among the different rank fusion
approaches that we examined, the Markov chain method outperforms other approaches.
The authentication outcomes of any multimodal biometric system depend on a number of
different attributes – biometric traits used, type and quality of datasets, algorithm used, application
area etc. Hence, we compare our results with results of some other multimodal biometric systems
reported in the literature which is shown in table 3 to evaluate the performance of different
multimodal biometric system built on different algorithms and with different biometric traits. From
the comparison results, it is clear that rank level fusion with Markov chain approach can certainly
increase the recognition rate of any multibiometric systems.
 15

Table 3. Comparison of Different Multibiometric System

Systems Biometric Identifiers Fusion level and Equal Error

approaches Rate (EER)

Current system Face, Ear, Iris Rank; 1.71%

(2010) Markov chain
Garcia-Salicetti et Signature, Voice Match score; 1.88%
al. (2005) [38] Arithmetic Mean Rule
with Min-Max normalization
Kumar and Zhang Palmprint Match score; 3.20%
(2004) [39] (Texture-based, Line- Product of Sum (POS)
based, Appearance-based) rule
Nandakumar et al. Fingerprint Match Score: 3.39
(2007) [40] (Minutiae-based and Quality Weighted Sum
Texture-based) (QWS) rule

7 Conclusions

The design of a multimodal biometric system is a challenging task due to heterogeneity of the
biometric sources in terms of the type of information, the magnitude of information content,
correlation among the different sources and conflicting performance requirements of the practical
applications. Extensive research has been done to identify better methods to combine the information
obtained from multiple sources. In this paper, we combine face, ear and iris biometric information
using rank level fusion method. We introduce Markov chain approach for biometric rank fusion and
obtain better identification rate over other rank fusion approaches. Thus, Markov chain method can be
a reliable solution of integrating biometric ranking lists to obtain a consensus rank list and can be
effectively used in various security scenarios. The proposed system can be efficiently employed in the
real-time security requirements with the inclusion of a special eigenimage based method for face or
ear detection and recognition technique in case of occluded or covered face or ear images (for iris, we
need specialized cameras).

References

1. A. Ross, K. Nandakumar, A. K. Jain, Handbook of multibiometrics, New York, Springer, 2006.

2. T. Dunstone, N. Yager, Biometric system and data analysis: Design, evaluation, and data mining.
Springer, New York, 2009.

3. D. Maio, D. Maltoni, R. Cappelli, J. L. Wayman, A. K. Jain, FVC2004: Third finfprint verification

competition, in proc. Intl. Conf. Biometric Authenication, pp 1-7, 2004.

4. K. Revett, Behavioral biometrics: A remote access approach, West Sussex, UK, Wiley, 2008.

5. U. M. Bubeck, Multibiometric authentication – An overview of recent developments. San Diego

15
16

University, 2003.
Retrieved on June 8, 2007 from www.thuktun.org/cs574/papers/multibiometrics.pdf.

6. J. Bhatnagar, A. Kumar, N. Saggar, A novel approach to improve biometric recognition using rank
level fusion, in proc. IEEE Conf. Computer Vision Pattern Recog., pp 1-6, 2007.

7. C. Dwork, R. Kumar, M. Naor, D. Sivakumar, Rank aggregation methods for the web, in proc. 10th
Intl. WWW Conf., 613–622, 2001.

8. K. Jain, K. Nandakumar, A. Ross, Score normalization in multimodal biometric systems, Pattern

Recog., 38, 2270-2285, 2005.

9. P. C. Cattin, D. Zlatnik, R. Borer, Sensor fusion for a biometric system using gait , in proc.
Multisensor Fusion and Integration for Intelligent Systems, 2001, pp 233-238, 2001.

10. G. Feng, K. Dong, D. Hu, D. Zhang, When faces re-combined with palmprints: A novel biometric
fusion strategy, in proc. 1st Intl Conf. Biometric Authentication, pp 701-707, 2004.

11. Q. Li, Z. Qiu, D. Sun, Feature-Level Fusion of Hand Biometrics for Personal Verification Based on
Kernel PCA, in LNCS 3832, pp 744 – 750, Springer, 2006.

12. L. Hong, A. K. Jain, Integrating faces and fingerprints for personal identification, IEEE Trans.
PAMI 20(12), 1295–1307, 1998.

13. K. Nandakumar, Y. Chen S. C. Dass, A. K. Jain, Likelihood ratio-based biometric score fusion,
IEEE Trans. PAMI 30(2), 342-347, 2008.

14. R. Frischholz, U. Dieckmann, BioID: A multimodal biometric identification system, IEEE

Computer 33(2), 64-68, 2000.

15. M. M. Monwar, M. Gavrilova, A multimodal biometric system using rank level fusion approach,
IEEE Trans. SMC - B: Cybernetics, 39(4), 867-878, 2009.

16. J. Fierrez-Aguilar et al., A comparative evaluation of fusion strategies for multimodal biometric
verification, in LNCS 2688, pp 830–837, 2003.

17. A. Ross, A. K. Jain, Information fusion in biometrics, Pattern Recog. Letters 24, 2115–2125, 2003.

18. T. K. Ho, J. J. Hull, S. N. Srihari, Decision combination in multiple classifier systems, IEEE Trans.
PAMI 16 (1), 66-75, 1994.

19. K. Nandakumar, A. K. Jain, A. Ross, Fusion in multibiometric identification systems: What about
the missing data?, in LNCS 5558, pp 743–752, Springer, 2009.

20. J. C. Borda, M´emoire sur les ´elections au scrutin. Histoire de l’Acad´emie Royale des Sciences,
France, 1781.

21. A. Agresti, An introduction to categorical data analysis, Wiley-Interscience, 2nd edition, 2007.

16
 17

22. M.-J. Condorcet, E´ssai sur l’application de l’analyse a` la probabilite´ des de´cisions rendues a` la
pluralite´ desvoix, 1785.

23. H. Copeland, A reasonable social welfare function. Mimeo, University of Michigan, USA, 1951.

24. M. Burge, W. Burger, Ear biometrics. In Biometrics: Personal identification in a networked society,
Jain AK, Bolle R, Pankanti S (Eds.) Kluwer Acedemic Publishers, 1998.

25. W. Zhao, R. Chellappa, J. Phillips, A. Rosenfeld, Face recognition: A literature survey, ACM
Computing Surveys, 35(4), 399–458, 2003.

26. W. Zhao, R. Chellappa, N. Nandhakumar, Empirical performance analysis of linear discriminant

classifiers, in proc. Conf. Computer Vision Pattern Recog., pp 164–169, 1994.

27. J. G. Daugman, How iris recognition works, IEEE Trans. Circuits Sys. Video Tech. 14(1), 21-30,
2004.

28. R. Wildes, Iris recognition: An emerging biometric technology, in proc. IEEE, 85(9), 1348-1363,
1997.

29. N. Ritter, Location of the pupil-iris border in slit-lamp images of the cornea, in proc. Intl. Conf.
Image Analysis Proc., 1999.

30. J. Kim, S. Cho, D. Kim, S.-T. Chung, Iris recognition using a low level of details, in LNCS 4292,
pp 196-204, Springer, 2006.

31. M. M. Monwar, M. Gavrilova, FES: A system of combining face, ear and signature biometrics
using rank level fusion, in proc. 5th IEEE Int. Conf. IT: New Generations, pp 922-927, 2008.

32. CASIA: CASIA iris image database, 2004.

Retrieved on October 2008 from www.sinobiometrics.com.

33. USTB ear database, China.

Retrieved on January 14, 2006 from http://www.ustb.edu.cn/resb/.

34. P. J. Phillips, H. Moon, P. Rauss, The FERET database and evaluation procedure for face
recognition algorithms, Image Vision Computing 16(5), 295-306, 1998.

35. C. Perpinan, Compression neural networks for feature extraction: Application to human recognition
from ear images, M. S. thesis, Faculty of Informatics, Technical University of Madrid, Spain, 1995.

36. Face database, Computer Vision Science Research Project, University of Essex, UK, 2008.
Retrieved on November 20, 2010 from http://cswww.essex.ac.uk/mv/allfaces/index.html.

37. M. Dobeš, J. Martinek, D. Skoupil, Z. Dobešová, J. Pospíšil, Human eye localization using the
modified Hough transform. Optik 117(10), 468-473.

38. S. Garcia-Salicetti et al., Multimodal biometric score fusion: The mean rule vs. support vector
classifiers, in proc. 13th European Signal Proc. Conf. 2005.
17
 18

39. Kumar, D. Zhang, Palmprint authentication using multiple classifiers, in proc. SPIE Conf.
Biometric Tech. Human Identification, pp 20-29, 2004.

40. K. Nandakumar, A. Ross, A. K. Jain, Incorporating ancillary information in multibiometric

systems, in Handbook of biometrics, New York: Springer, pp 335-355, 2007.

Md. Maruf Monwar is a PhD student of Computer Science at the University of

Calgary. He received his B.Sc. (Hons.) and M.Sc. in Computer Science &
Technology at the University of Rajshahi, Bangladesh and M.Sc. in Computer
Science at the University of Northern BC, Canada. He is an Assistant Professor at
the Department of Computer Science & Engineering, University of Rajshahi,
Bangladesh, and currently is on study leave. His primary research interests include
biometric fusion, expression recognition and biological data processing. He is one of
the recipients of the very prestigious inaugural NSERC Vanier CGS Scholarship. He
served as a guest editor of the International Journal of Biometrics, published by
Indersciences.

Marina L. Gavrilova is an Associate Professor in the Department of Computer

Science, University of Calgary. Dr. Gavrilova research interests lie in the area of
computatio- -al geometry, image processing, optimization, spatial and biometric
modeling. Prof. Gavrilova is founder and co-director of two innovative research
laboratories: the Biometric Technologies Laboratory: Modeling and Simulation
and the SPARCS Laboratory for Spatial Analysis in Computational Sciences. Prof.
Gavrilova publication list includes over 90 journal and conference papers, edited
special issues, books and book chapters. Prof. Gavrilova acted as a Co-Chair of the
International Workshop on Biometric Technologies BT 2004, Calgary, Alberta,
Canada, 2004, an Overall Chair of the Third International Conference on Voronoi
Diagrams in Science and Engineering and as Organizational Chair of WADS 09.
Prof. Gavrilova is an Editor-in-Chief of the LNCS Transactions on Computational Science Journal, Springer-
Verlag and serves on the Editorial Board for the International Journal of Computational Sciences and
Engineering, CAD/CAM Journal and Journal of Biometrics.

18

View publication stats