Information Fusion 91 (2023) 694–712

Contents lists available at ScienceDirect

Information Fusion
journal homepage: www.elsevier.com/locate/inffus

A survey of identity recognition via data fusion and feature learning

Zhen Qin a,b ,∗, Pengbiao Zhao a,b , Tianming Zhuang a,b , Fuhu Deng a,b ,∗∗, Yi Ding a,b,c ,∗∗,
Dajiang Chen a,b ,∗∗
a Network and Data Security Key Laboratory of Sichuan Province, University of Electronic Science and Technology of China, China
b School of Information and Software Engineering, University of Electronic Science and Technology of China, China
c
Ningbo WebKing Technology Joint Stock Co.,Ltd, China

ARTICLE INFO ABSTRACT

Keywords: With the rapid development of the Mobile Internet and the Industrial Internet of Things, a variety of
Identity recognition applications put forward an urgent demand for user and device identity recognition. Digital identity with
Data fusion hidden characteristics is essential for both individual users and physical devices. With the assistance of
Multimodal
multimodalities as well as fusion strategies, identity recognition can be more reliable and robust. In this survey,
Mobile internet
we turn to investigate the concepts and limitations of unimodal identity recognition, the motivation, and
Industrial internet of things
Feature learning
advantages of multimodal identity recognition, and summarize the recognition technologies and applications
via feature level, match score level, decision level, and rank level data fusion strategies. Additionally, we also
discuss the security concerns and future research orientations of learning-based identity recognition, which
enables researchers to achieve a better understanding of the current status of this field and select future
research directions. This survey summarizes and expands the fusion processing technologies and methods for
multi-source and multimodality data, and provides theoretical support for their applications in complicated
scenarios. In addition, it enables researchers to achieve a better understanding of the current research status
of this field and select proper future research directions.

1. Introduction
As the Mobile Internet and Industrial Internet of Things developed
rapidly, the scale of users in the social network becomes larger. At the
same time, the era of the interconnection of all things introduces a great
number of intelligent terminal devices. With the help of these devices,
people can freely access the network to complete relevant network
services, leading to mutual interaction and cooperation. Under this
circumstance, the significance of user and device identity recognition
tends to be increasingly prominent, which has a wide range of practical
needs and research values.
However, merely relying on a single feature for the user or device
identity recognition faces severe challenges, specifically, it is easy to
be counterfeited and stolen. For example, identity recognition based
on passwords or keys has been a mature and effective method. The
security of passwords or keys completely depends on their confiden-
tiality. However, the security threats in the open environment are
becoming increasingly prominent. Users are vulnerable to surveillance
when entering passwords, or when using the keyboard to enter the
password in case of computer poisoning, the trojan horse program
will record the keyboard input. Therefore, Rational use of combined
features for identity recognition to overcome the shortcomings of single
feature, so as to effectively improve the security and performance of the
system.
The concept of modality has been gradually introduced to this re-
search field. A modality refers to the way in which something happens
or is experienced. To be more specific, such as hearing sounds, feeling
textures, viewing objects, and smelling scents, all these sensory actions
can be regarded as channels for humans to feel and communicate
with the outside world. The related research is often described as
multimodality if including such modes. In this paper, we summarize
modal into three aspects: human private characteristics [1–3] (e.g.,
fingerprint, iris), individual features collected by physical devices (e.g.,
personal signature, walking action), and physical fingerprints (e.g.,
device hardware that commonly used in the Internet of Things) (see
Fig. 1).
However, in the practical application of single feature recognition,
there exist the following four shortcomings. The first is the similarity
between classes. For the recognition system with a large amount of

∗ Corresponding author at: Network and Data Security Key Laboratory of Sichuan Province, University of Electronic Science and Technology of China, China.
∗∗ Corresponding authors.
E-mail addresses: qinzhen@uestc.edu.cn (Z. Qin), pengbiaozhao@std.uestc.edu.cn (P. Zhao), tianming.zhuang@std.uestc.edu.cn (T. Zhuang),
fuhu.deng@uestc.edu.cn (F. Deng), yi.ding@uestc.edu.cn (Y. Ding), djchen@uestc.edu.cn (D. Chen).

https://doi.org/10.1016/j.inffus.2022.10.032
Received 12 December 2021; Received in revised form 30 October 2022; Accepted 31 October 2022
Available online 7 November 2022
1566-2535/© 2022 The Author(s). Published by Elsevier B.V. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).
Z. Qin et al.
Fig. 1. Presentation of multiple modalities.
data, the features between different users may overlap, which will
seriously affect the recognition result. The second is the variability of
characteristics. Due to the reasons of equipment or users, the same
characteristics of the same user might be mutable with times, which
lacks robustness. The third comes to be sensor noise. Due to the defects
of sensors and other hardware equipment or environmental factors, the
collected data is likely to be disturbed by a large amount of noise,
which affects the recognition accuracy to a large extent. The last is the
non-universality of features. Some users are unable to provide specific
features due to private reasons. For instance, people engaged in manual
labor may not be able to provide clear fingerprint images, and the
deaf cannot provide audio information for voice recognition. In general,
the improvement of the accuracy and reliability requirements has been
raised in identity recognition nowadays, unimodal identity recognition
system cannot meet present demands.
In order to move beyond such limitations, researchers proposed
an identity recognition scheme using multi modalities, also known as
multimodal identity recognition. For example, during the fingerprint
recognition procedure, it is difficult to extract effective information
from unclear fingerprints. The multimodal identity recognition system
is capable of obtaining other types of biometric features to make up
for the deficiency. Then, by fusing different kinds of features [4–8],
the influence of external noise can also be effectively reduced to an
acceptable extent, while the recognition accuracy can be improved
concurrently. Furthermore, the multi-feature system can also improve
the anti-counterfeiting ability of the system, so as to improve the
security of the system. the use of multiple biometrics can increase the
difficulty for intruders to construct different features at the same time.
Generally speaking, the identity recognition system based on multi-
modality shows the advantages of high reliability, strong robustness,
and wide practicability.
The existing surveys related to multimodality fusion [9–12] mainly
focus on the commonly-used biometrical characteristics. As a supple-
ment, this review focuses on research based on the multimodality data
fusion to achieve identity recognition, discusses the feature represen-
tation of physiological biometrics, behavioral biometrics, and physical
fingerprint in the field, and reviewed the complex scenario modal
695
Information Fusion 91 (2023) 694–712
processing and fusion method, further analyzes the recognition security
concerns, as well as explores research challenges and future directions.
This paper presents an overview of the current status of research on
multimodal identity recognition. The main work of this paper can be
summarized as:
• The modalities of physiological biometrics, behavioral biometrics,
and physical fingerprint for identity recognition are investigated,
while the advantages and disadvantages of unimodal identity
recognition are illustrated.
• The fusion strategies of combining and leveraging feature level,
match score level, decision level, and rank level semantic infor-
mation for multimodal identity recognition are summarized and
their application scenarios are also presented.
• The security concerns introduced by adversarial attacks towards
learning-based identity recognition are studied, and future re-
search orientations of multimodal identity recognition are dis-
cussed.
Sections 2 and 3 introduce unimodal identity recognition and its
limitations. Sections 4 to 6 detail the fusion strategies, multimodal
combination, and security analysis of multimodal identity recognition,
respectively. The challenges and future directions are discussed in
Section 7. Section 8 concludes this work.
2. An introduction of unimodal identity recognition
Unimodality means that the person or system relies on one or only
one source of identity feature for identification. There are various
features that can be selected as unimodal features. In this section, we
will provide a detailed description of the commonly used unimodality,
following the three classifications mentioned above.
2.1. Physiological biometrics
Traditional identity recognition systems based on individual char-
acteristics, also known as biometric systems, often only use a single
biological feature for recognition. It can be said that biometric identi-
fication is the oldest and most widely used identification method. The
biometric system usually includes two processes: the registration phase
and the recognition phase.
In the registration phase, the system accepts the user’s biometric
features as input, then preprocesses them and finally extracts features
from them. The extracted features will be stored in the system database
The recognition stage accepts the biometrics of users and then
mainly does the feature extraction like the register stage. After finishing
the feature extraction, the system will put this extracted feature to the
database to calculate the matching score and identification.
There is various type of biometrics feature that can be extracted
and are proven useful and reliable. Brief comments about them can be
found in Table 1. In this work, we separate them as face-based modal
recognition and fingerprint-based modal recognition.
2.1.1. Face-based unimodal recognition
Face biometrics has always been one of the crucial biometric fea-
tures, and its application has extensive areas such as security, educa-
tion, entertainment, health and law enforcement. Face recognition can
be divided into two types: image-based and video-based.
Image-based face recognition implements the identification from
a single image or frame. Appearance-based face recognition was first
introduced by Murase and Nayar [13], which detects the face re-
gion as a whole and managed to present it to a lower dimensional
subspace. Model-based face recognition includes 2D methods and 3D
methods. The most famous 2D model is the Active Appearance Model
(AAM) [14–16], which is a statistical appearance method by con-
trolling a set of shape and gray level parameters. AAM employs an
iterative matching algorithm by utilizing the relationship between the
Z. Qin et al.
Table 1
Evaluation of physiological characteristics for unimodal identity recognition.
Type Advantage
Face Convenient and efficient can recognize multiple
faces at the same time; no need to direct contact
with the device. High recognition accuracy.
Iris Non-contact, clean and avoids the infection of
diseases; impossible to be copied and modified.
High reliability
Ear Stable structure and will not be changed by the
age, emotion and facial expressions; small size and
constant color cost less storage capacity
Fingerprint Fast recognition speed, most convenient biometrics;
easy to promote and widely used; low rate of
misjudgement and rejection; easy to operate
Palm print Easily be placed in the correct position of the
scanner; measurement is weakly influenced by the
dirt and scars.
Table 2
Evaluation of behavioral characteristics for unimodal identity recognition.
Behavioral Advantage
biometrics
Voice Convenient and can be realized by only a simple
microphone; relatively low cost, easy to
commercialize.
Handwriting Non-invasive, easy to collect signature data,
relatively zero cost
Gait Low cost of acquisition device, farther collected
distance, gait is not easy to disguise, convenient to
use
image error and model parameter perturbations. 3D face recognition
uses Three-dimension facial characteristic and features such as surface
normal [17–19]. Texture based face recognition uses local feature
descriptors such as Histogram of Oriented Gradients (HOG) [20–22],
Local Binary Pattern (LBP) [23] and Scale-invariant Feature Transform
(SIFT) [24,25].
Video-based face recognition refers to the use of the dynamic and
movement of facial features to do identification and recognition. Set-
based methods [26] treated video or frames as a set of image samples
such as frame aggregation [27–29], frame selection [30]. Sequenced-
based methods [31] use the temporal information from a video. Spatio-
temporal methods [32,33] utilize both motion information and texture
information.
In addition to the comprehensive use of faces, there is also some
biometrics for single facial organs, such as the eyes and ears.
The structure of the eye includes the sclera, pupil, cornea, and iris.
Among them, the iris has extensive details that can be used for identifi-
cation. Iris recognition gained much attention in different areas such as
airplane security, industrial areas, and medical institutes. A traditional
iris recognition system includes those procedures: pre-processing, iris
segmentation, iris normalization, feature extraction, feature selection,
classification, and decision.
The iris segmentation stage aims to reduce the residual parts and
maintain the iris part. Traditional approaches include Geodesic Active
Contour (GAC) methods [34–36], Super-pixel Segmentation (SPS) [37],
and Morphological [38]. Also, there are several deep learning-based
approaches such as U-Net [39,40], VGG [41], R-CNN [42–44] and other
CNN based model [45,46].
The Iris feature extraction stage is used to obtain the specific
features from obtaining the iris region. Traditional approaches include
local feature extraction and global feature extraction. Local feature
extraction uses local features methods such as texture features, statis-
tical features, Scale-invariant Feature Transform (SIFT) descriptor [47]
and local binary pattern(LBP) [48,49]. Global feature extraction uses
Information Fusion 91 (2023) 694–712
Disadvantage Application field
Get influenced by many factors such as lighting Public management, security and
condition camera angle and low resolutions. anti-theft, e-commerce
transactions, asset protection
Difficult to miniature hardware devices; expensive Coal industry, access control
and hard to promote on large scale; slow system, judicial security
recognition process
Easily affected by posture, occlusion, lighting and Forensic area
other issues; low privacy and medium accuracy
Sensitive to the humidity and cleanliness of the Smartphone unlock, criminal
fingers; difficult to image for some people with investigation, household
few or no fingerprints. registration management
Uniqueness is not very strong, similarity of human Access control system, clerk
palms is relatively high attendance management
Disadvantage Application filed
Voice changes are easily affected by sickness; National social security system
easily imitated and stolen by impostors; noise
sound around the recognition environment
Easy to be imitated, and the handwriting is Business signature, financial area
affected by the ink material and the user’s mood
at that time
Relatively low reliability, slow collection speed Video surveillance system
global features methods such as Binarized Statistical Image Features
(BSIF) [50] and Histogram [51]. Deep learning-based approaches use
CNN based model citegalbally2013image,czajka2015pupil such as VGG
model [52], ResNet model [53] and InceptionV3 model [54,55].
The iris feature classification stage is the final stage of the iris
recognition system, classification aims to match the level of the given
sample and sample already stored in the system database. Tradition
approaches include k-NN method [56] and SVM method [57]. Deep
learning-based approaches uses CNN based model [58,59], FCN based
model [60,61], DBN model [62,63] and ResNet based model [64].
Unlike other common biometrics like fingerprint and face, the ear
can also be used to do identification. The shape of people’s ears presents
its uniqueness just like human facial expressions. The structure of the
ear includes Scapha, Helix, concha, Lobule, and so on. These plentiful
features can be used for recognition. The ear identification system
needs two main stages. Detection stage and Recognition stage.
The ear identification system initially needs to locate and detect
the specific ear images from the original face images. 2D ear detection
has various methods such as active contour model [65], AdaBoost
algorithm [66–68], structural ear image [69–71] and Canny edge de-
tector [72]. Besides, 3D ear detection approach includes Histograms
of Categorized Shapes (HCS) [73,74], Entropic Binary Particle Swarm
Optimization (EBPSO) [75] and other feature extractions [76,77].
Ear recognition stages denote that the system utilizes the richness
of features of the human ear scheme to do identification. Popular
2D recognition includes sparse representation [78–81], Discrete Cosine
Transforms (DCT) [82], Gabor features [83–86] and neural network
methods [87–89]. Also, there are several recognition approaches for 3D
ear images such as Structure from Motion (SFM), Shape from Shading
(SFS) [90], Shape-based Interest Point Descriptor (SIP) [91] and Local
Salient Shape Feature [92].
2.1.2. Fingerprint-based modal recognition
The fingerprint is undoubtedly one of the most famous biological
characteristics, owing to its uniqueness and consistency. A fingerprint is
696
Z. Qin et al.
Table 3
Summary of device characteristics used for unimodal identity recognition.
Radio frequency Methods
fingerprint
Transient-based RF Bayesian Step Change Detection
fingerprint
Bayesian Ramp Change Detection
Mean Change Point Detection
Phase Detection
Steady-based RF Passive Radiometric Device Identification
fingerprint System
Other approaches Power Spectrum Density
an impression left by the friction ridges of a human finger. Fingerprint
representation schemes can be classified as below.
Global feature based representation represents the global pattern
of the ridges in a fingerprint and includes global ridge-line frequency
orientation images and core points, etc.
Local feature based representation [93–97] pays much attention to
the local minutiae part of the fingerprint such as local ridge orientations
and local ridge frequency. These distinctive representations outperform
the global counterparts.
Transform-based representation [98–100] uses transform methods
such as Digital Wavelet Transform features or Digital Cosine Transform
to represent the fingerprint.
Palm print is the most frequently used body part in people’s daily
lives, like the fingers, and has a wide variety of biometric features
for identity recognition. The principal lines of palm prints include the
thenar region, interdigital region and hypothenar region. There are sev-
eral feature extractions such as line based approaches, which develop
edge detection methods to extract palm lines [101–107], subspace-
based approaches, which involve principal component analysis (PCA),
independent component analysis (ICA), linear discriminant analysis
(LDA) [108–113] respectively and statistical approaches including
global statistical [114] and local statistical approaches [115–117].
2.2. Behavioral biometrics
In addition to the extensive use of human physiological charac-
teristics, some human behavioral characteristics can also be used for
identification. In this subsection, we present three of the most studied
behavioral biometrics as shown in Table 2: voice, handwriting and gait
identity recognition.
2.2.1. Voice-based identity recognition
Human’s voice is unique and can provide biometrics characteris-
tics for systems to determine identity. The voice identification system
includes voice collection, voice preprocessing, feature extraction and
feature classification stages.
With the development of machine learning and deep learning, more
and more ML/DL methods get involved in voice identification. Machine
learning approaches such as Gaussian mixture model [118–120], deci-
sion Tree [121,122], support vector machine [123], k-nearest neigh-
bor [123] and Bayes method [124]. Deep learning approaches such as
CNN [125], RNN [126] and autoencoder [127].
2.2.2. Handwriting-based identity recognition
Handwriting is a signature for human identification. Even the same
person cannot write two identical handwritings and the style and shape
of people’s handwriting are influenced by some subjective factors like
people’s moods and object factors like the material of the signing pen.
Therefore, it is necessary to identify and recognize handwriting. Hand-
writing identification has been adopted by commercial transactions and
some governments.
There are three types of features: statistical features, structural fea-
tures and model-based features. Statistical features are geometric and
Information Fusion 91 (2023) 694–712
Advantage Disadvantage
High detection rate Complex computation
Simplify the modulation scheme, resilient to Possible to be effected by the
mobility traveling nodes
Uniqueness to wifi devices, effective to only work well on IEEE 802.11a
amplitude transient detection signals
statistical measurements for classification. It contains global features
and local features. Global features [128–132] describe the global trait
of the handwriting images. Local features [133] describe the key points
and details of the handwriting images.
The structure features presents local structure and topology of char-
acteristics on writing such as loops, dots and edges. There are sev-
eral structural features [133,134] such as fragments, graphemes and
strokes.
Model-based features extract by deep learning-based model from the
image data. The common model for feature extraction is CNN based
model [135–138] and RNN based model [139,140].
2.2.3. Gait-based identity recognition
Gait-based recognition is an active behavioral biometric recognition
that aims to identify people by his or her walking posture. Due to its
ease of disguise and noncontact, gait recognition has been used in the
field of surveillance. There are two types of gait recognition. One is
vision-based and the other is sensor-based gait recognition.
Vision-based gait recognition refers to the use of cameras to acquire
images of human walking actions. Vision-based gait feature extraction
can be classified into Model-based methods and model-free methods.
Model-based methods use body composition or motion model, such as
structural model [141] and motion model [142], to describe walking
pattern. Model-free methods extract the representation of the mo-
tion gait images without any model. The common representation are
appearance-based representation [143–145], distribution-based repre-
sentation [146] and transformation-based representation [147].
Sensor-based gait recognition refers to the use of sensor equip-
ment such as inertial sensors and wearable sensors. After collecting
the sensor data of gait signals, normally it needs data extraction,
cycle segmentation and then conducts gait identification. Recently deep
learning has gain extensive success in gait recognition such as CNN
model [148–150], LSTM model [151,152] and GAN model [153].
2.3. Physical fingerprint
With the emerging rise of IoT, device identification has become
increasingly important and necessary. In this section, we briefly in-
troduce the radio frequency (RF) fingerprint and some other device
authentication solutions as shown in Table 3.
2.3.1. Radio frequency fingerprint
Physical devices can be identified by unique radio circuitry. Radio
fingerprint is used to identify physical wireless devices by extracting
specific structures in the electromagnetic waves emitted from the trans-
mitters. RF fingerprinting includes three main categories: transient-
based RF fingerprinting, steady-based RF fingerprinting and other ap-
proaches.
Transient-based radio frequency fingerprint identification technol-
ogy uses the switch from off to on status of the transmitter that occurs
before the actual data transmission of the signal. These methods include
Bayesian Step Change Detection (BSCD) [154], Bayesian Ramp Change
697
Z. Qin et al.
Fig. 2. Literature statistics of unimodal identity recognition.
Detection (BRCD) [155], Mean Change Point Detection (MCPD) [155],
Phase Detection (PD) [156] and Transient Envelope [157].
The steady-state-based method focuses on the unique features ex-
tracted from the modulation part of the signal. Some approaches were
proposed such as Passive Radiometric Device Identification System
(PARADIS) [158], which used five specific signal features: frequency
error, synchronized correlation, phase errors, I/Q origin offset, and
magnitude.
Some of the other methods usually utilize other attributes of the
signal and logical layer, such as measuring the power spectrum density
(PSD) [159] to identify wireless devices.
2.3.2. Other device recognition solutions
There are some other solutions to authenticate the device by tak-
ing full advantage of the physical trait. Chen et al. [160] proposed
a lightweight device authentication protocol speaker-to-microphone
(S2M) by using the frequency response of a speaker and a microphone
from two wireless IoT devices as the acoustic hardware fingerprint. Lin
et al. [161] introduced a device-free authentication system to validate
human identity using WIFI signals via only utilizing a laptop and a
Commodity Off-The-Shelf (COTS) router. Aneja et al. [162] found that
device fingerprints can be identified via Inter Arrival Time (IAT) for
packets and then propose four models to capture packet information
on the router based on the layers and methodology used.
According to statistics in Web of Science database, the number of
papers published on different unimodal identification methods in the
past five years is shown in Fig. 2.
3. Motivation: multimodal identity recognition design
This section begins with a discussion of the limitations of unimodal
recognition techniques, followed by a brief introduction to common
frameworks and evaluation metrics for multimodal identity recogni-
tion.
3.1. Limitation of unimodal recognition
Although the current unimodal recognition technology has a certain
degree of accuracy and has been fully utilized in some fields, there are
still some drawbacks and limitations.
Difficult to obtain intact data: Typically, unimodal data is ac-
quired through the sensor, but when the sensor is not regularly main-
tained or modality is not sufficiently intact, the data may be disturbed
by noise during acquisition. For example, a fingerprint scanning device
698
Information Fusion 91 (2023) 694–712
Fig. 3. Diagram of the multimodal identity recognition process.
may collect incomplete fingerprint data, if collector’s fingerprint is not
cleaned before scanning, or if the finger is molting.
Not convenient: Unimodal identification requires users to fully
display this modality in front of the device or sensor. However, this
ideal situation is difficult to achieve in some realistic scenarios. For
example, the outbreak of the covid-19 has forced people to wear masks
while traveling, but when performing face recognition, they must take
off the masks. One potential alternative solution is iris recognition,
nevertheless, the accuracy of recognition will also have been affected
negatively if the user happens to wear contact lenses.
Not universal: Some users may be not able to provide specific
biometric features due to personal factors. For example, some disabled
people who have lost their tongue cannot provide the voice information
for voice recognition. Besides, gait recognition and iris recognition do
not work for those people who have some physiological disease with
legs and eyes.
Easy to forge: Due to the singularity of the modalities used for
identification, it is likely that an imposter is able to spoof identification
systems by imitating or stealing certain biometric characteristics of
others. For example, a face recognition system will be fooled by some
specific impostors who wear 3D mask.
3.2. Motivation for multimodality
In addition to the modalities introduced in Section 2, some unique
signals possessed by the human body can also be collected by various
sensors equipped on smartphones and wearable devices. For exam-
ple, the information data collected by gravity sensors, gyroscopes,
touchscreens and other devices in smartphones can reflect the unique
habits of cellphone users. Wearable devices can detect the user’s heart-
beat, gait and other behavioral characteristics. Next-generation sensors
can even collect electroencephalography (EEG) and electromyogram
(EMG) for identification. In our survey, the identification accuracy
of using these modalities alone is not high, but when they are used
for multimodal identity recognition tasks, they can effectively en-
hance recognition and greatly reduce the constraints of the recognition
environment.
Based on the above, the use of multimodality for identity recog-
nition is very necessary and urgent. It can solve the problems of
unimodal recognition mentioned above. First, when multimodal fea-
tures are used for recognition, the combination of them can weaken the
effect of feature overlap and improve the recognition accuracy. Second,
some of modal feature tend to change dramatically in terms of the
spatio-temporal variability of features. When a modality changes, the
importance of that feature in the recognition system decreases, which
can effectively counteract the degradation of performance. The third
point is that when multiple modalities are introduced when observing
and describing the same object, some complementary information can
Z. Qin et al. Information Fusion 91 (2023) 694–712

be obtained to obtain more robust results in the prediction process. 3.4. Description of evaluation metrics
And in a scenario where a modality is not easily accessible or falls into
failure, the rest of the modal information can continue to provide the A brief overview of the commonly used evaluation metrics for
basic information for prediction and avoid system failure. Finally, the identity recognition is presented below.
multimodal identity recognition system has better defense performance
for various attacks. It is difficult for a single tampered modality to play True positive (TP): The system correctly predicted positive samples
a significant role in many modalities to influence the final decision, as positive.
and even during the identification process, the system can also locate True negative (TN): The system correctly predicted negative sam-
the modal position of the attack by analyzing the abnormal modal ples as negative.
characteristics.
False positive (FP): The system wrongly predicted negative sam-
3.3. Framework and technology ples as positive.
False negative (FN): The system wrongly predicted positive sam-
In recent years, a large number of multimodal models have
ples as negative.
emerged, which have been widely applied due to its robustness and
reliability when encountering modality changes. A typical diagram Recall: The recall for one class c is the proportion of predicted
of the multimodal identity recognition system is given in Fig. 3. positive examples in all the positive samples (True positive and false
Firstly, the information is collected from the modalities by various negative).
devices to get the initial data. After that, pre-processing operations
𝑇 𝑃𝑐
are performed on the data, which generally include abnormal data 𝑅𝑒𝑐𝑎𝑙𝑙𝑐 = (1)
cleaning, noise cleaning and other operations. Then the corresponding 𝑇 𝑃𝑐 + 𝐹 𝑁𝑐
feature extractors of different modalities are used, either by using
Accuracy (ACC) : The rate of the correctly accepted and correctly
manually designed algorithms or by using the most popular deep
rejected matches in the whole system.
neural networks to extract features. Finally, multiple modal features are
fused to form a multi-dimensional information feature database with 𝑇𝑃 + 𝑇𝑁
𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦 = (2)
multiple users to provide subsequent classification support. When a 𝑇𝑃 + 𝑇𝑁 + 𝐹𝑃 + 𝐹𝑁
user needs to be identified and authenticated, the data is also collected
Precision (PRE): The percentage of true positive examples in the
from the specified modalities, and after the pre-processing operation,
sample predicted by the system to be positive.
the unique modal features obtained are compared with the feature
data in the database by the matching algorithm to finally obtain the 𝑇𝑃
𝑃 𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 = (3)
identification results. It is worth noting that, the repeated steps here 𝑇𝑃 + 𝐹𝑃
are used to display different modes, such as mode A to mode Z. These
Unweighted average recall (UAR) : In order to reduce the
three processing steps are usually required to obtain data features
weighted deviation, the accuracy of the balance is called the un-
for training and recognition. However, although there are the same
data processing steps, each modality will generally have a variety of weighted average recall rate, which is the average of the positive recall
different algorithms for each step. rate and the negative recall rate.
The repeated steps here are used to display different modes, such as ∑|𝐶|
𝑟𝑒𝑐𝑎𝑙𝑙𝑐
mode A to mode Z. These three processing steps are usually required 𝑈 𝐴𝑅 = 𝑐=1 (4)
|𝐶|
to obtain data features for training and recognition. However, although
there are the same data processing steps, each modality will generally False acceptance rate (FAR) : The rate of the system wrongly
have a variety of different algorithms for each step. accept the false subjects as correct subjects, which measures the per-
The process from the acquisition of information to the final recogni-
centage of wrongly accepted matches.
tion result is essentially the processing of feature information, in which
the process generally contains these important steps: (1) representation 𝑁𝐹 𝐴
𝐹 𝐴𝑅 = × 100% (5)
of information: representing and summarizing multimodal data in a 𝑁𝐼𝑅𝐴
way that exploits the complementarity and redundancy of multiple where NFA refers to the number of times the wrong object is judged as
modalities; (2) fusion: combining information from multiple modalities the correct object, and NIRA is the number of times the wrong object
for classification or prediction. is judged.
Multimodality is a representation of data using information from
False reject rate (FRR) : The ratio of the system wrongly reject
multiple such entities, often with multiple representations, which can
the correct subjects as false subjects, which measures the percentage of
be an image, a piece of audio, a piece of text or other forms. There
false rejected matches.
are many difficulties: how to combine heterogeneous data; how to deal
with different types of noise; how to deal with missing or anoma- 𝑁𝐹 𝑅
𝐹 𝑅𝑅 = × 100% (6)
lous data, and which are more efficient ways to represent modal 𝑁𝐺𝑅𝐴
features. Currently, most images are represented using features learned where NFR refers to the number of times the correct object is judged as
from trained convolutional neural networks or other network architec- the wrong object, and NGRA is the number of times the wrong object
tures [163]. In the audio domain, many artificially designed acoustic is judged.
features have also been replaced by deep neural networks driven
by speech data [164] and recurrent neural networks for language Equal Error Rate (EER) : Equal Error Rate (EER) is the point on the
analysis [165]. ROC curve that corresponds to the same probability of misclassifying
Multimodal fusion is the combination of information from multiple a positive sample or a negative sample. This point is obtained by
modalities that is used to obtain a classified category or predicted intersecting the ROC curve with the diagonal of the unit square.
outcome, with techniques dating back to work done 32 years ago [166].
Currently, the use of neural networks is a very popular approach to 4. Fusion methods for multimodal identity recognition
deal with multimodal fusion. However, in recent deep neural network
models, the boundary between the representation of modal features and Multimodal fusion is one of the key research points. It can process
fusion has gradually blurred. Therefore, in the subsequent discussion and fuse various types of data information obtained from different
section of this paper, we will focus on the introduction and discussion fusion methods and integrate it into a stable multimodal representa-
of multimodal fusion techniques. tion. In this process, the information from two or more modalities is

699
Z. Qin et al. Information Fusion 91 (2023) 694–712

Fig. 4. Illustration of four fusion levels for multimodal identity recognition.

Table 4
Related work of data fusion in feature level for identity recognition.
Reference Year Modalities Methods Dataset Metrics
A feature selection algorithm to
[167] 2012 Audio reduce acoustic information and FAU Aibo Corpus UAR:45.08%
one classifier used to analyze
Feature extraction with Linear
Face and
[168] 2013 Discriminant Analysis and Feature – ACC:97.50%
signature
selection using GA
FVC_2000 DB1A EER:6.49%
[169] 2018 Fingerprint Multi-level feature fusion
SDUMLA_FP EER:7.29%
PEC_FP EER:4.23%
The cepstral coefficients and
[170] 2019 Voice and face statistical coefficients for voice and – EER:2.81%
Eigenface and SVM for face
PRE:99.96%
[171] 2019 Echo and face Deep Neural Networks –
Recall:88.84%
The nearest neighbor algorithm CASIA V1.0 (iris) ACC:98.33%
[172] 2021 Iris and fingerprint and speedup robust feature for and CASIA V5.0 FAR:0.74%
feature extraction (fingerprint) FRR 0.93%

integrated and it can be used to complete the classification or regression
task.
For different fusion levels of multimodal identity recognition, the
current fusion methods can be divided into four parts: feature level
fusion, match score level fusion, decision level fusion, and rank level
fusion, which will be discussed in detail in this section.
4.1. Feature level fusion
The process of combining features vectors obtained through differ-
ent sensors, different biometrics to obtain a new feature vector and fed
it into a machine learning classifier for further classification, is called
feature level fusion. The process of feature level fusion is shown in
feature level fusion in Fig. 4.
Feature level fusion contains more biological information in theory,
but because the dimension and size of feature vectors of different modes
may not be compatible, low compatibility may result in poor inference
performance, so normalization is needed before the fusion process.
Poonguzhali et al. [169] proposed a two-level feature extraction and
fusion method to extract and fuse fingerprint features at two levels
of features. The level 1 feature, level 2 feature, and fused two-level
features are evaluated at three different experiment stages. Then the
three kinds of features were further fused to form a fourth kind of fused
feature. The result shows the combined feature vectors are more ade-
quate in the inference process in terms of EER and DI metrics. Kumar
et al. [172] proposed a feature level method, in which biometrics of
iris and fingerprint are fused. In the process of feature extraction, the
nearest neighbor algorithm and speedup robust feature (SURF) is used
on both fingerprint and Iris data. Then, in the feature selection process,
the extracted features are both optimized by the GA algorithm. Then
the iris and fingerprint data are trained by the ANN algorithm.
Jaswal et al. [173] first combined backtracking search algorithm
and 2D2LDA to fuse fingerprint and palmprint information at feature
level. In finger dorsal patterns identification, Attia et al. [174] combine
the information extracted from the finger dorsal surface image with the
major and minor knuckle pattern regions in feature level.
In Bokade’s research work [175], three modalities of face, ear, and
palmprint were fused to form a feature vector with reduced dimension.
The dimension reduction and feature extraction were achieved by prin-
cipal component analysis and extracted feature vectors were further
concatenated to form a combined feature. The combined feature was
further used for the training and inference process. The GAR has been
improved compared with unimodal approaches.
More applications of feature level fusion is shown in Table 4,
where the vacant part in the column of the dataset indicates that the
corresponding paper uses self-collected non-public datasets.
In summary, the advantages of feature-level fusion are that it
achieves considerable information compression, facilitates real-time
processing. Because the extracted features are directly related to the
decision analysis, the fusion results maximize the feature informa-
tion required for decision analysis. Feature-level fusion generally uses
distributed or centralized fusion systems.

700
Z. Qin et al. Information Fusion 91 (2023) 694–712

Table 5
Related work of data fusion in match score level for identity recognition.
Reference Year Modalities Methods Dataset Metrics
Image and Weighted-summation operation for
[176] 2008 – EER:2.13%
voice matching scores
Different SIFT and ORB CASIA Multispectral
[177] 2014 features of features extraction Palmprint Image EER:0.36%
palm veins based Database V1.0
NIST BSSR1 EER:0.75%
Face+fingerprint, Particle Swarm Optimization and
[178] 2015
face+speech Genetic Algorithm XM2VTS EER:1.85%
BANCA EER:10.4%
FAR:1.5%
K-means clustering, CASIA-IrisV4
FRR:3.89%
[179] 2020 Iris and fingerprint Decision Tree
and Fuzzy Logic FAR:2.5%
CASIA-FingerprintV5
FRR:5%

Table 6
Related work of data fusion in decision level for identity recognition.
Reference Year Modalities Methods Dataset Metrics
Acoustic and
Naive-Bayes, Support Vector
[167] 2012 linguistic FAU Aibo Corpus UAR: 41.74%
Machine and Logistic Model Tree
features
Fingerprints
Multi-finger feature encrypted by
[180] 2015 of different – FAR:0.59%
hash function
fingers
Face and Joint Encryption and Compression
[181] 2016 FEI and NIST ACC:92%
fingerprint technique for biometric data
Face and Wavelet sub-bands, Nearest palmprint database
[182] 2020 ACC: 98.45%
Palmprint Neighbor Classifier ORL+Yale, IIT-Delhi

4.2. Match score level fusion
Match score level fusion is also called confidence level fusion. It is
different from feature layer fusion and is carried out after matching.
The match module in the identity recognition process will deliver a
match score after the match process, as shown in match score level
fusion in Fig. 4. This score can reflect the similarity between the
inputted feature vector and the existing feature vector in the database.
By fusing the match scores of different modalities, the input object can
be recognized. The fusion process can be completed by a variety of
statistical learning techniques.
Normalization operation is often needed before matching score level
fusion to avoid incompatibility caused by different measurements of
match scores of different modalities. Through a series of Normalization
methods, match scores from different modalities can be converted to
the same arithmetic range.
Dalila et al. [178] proposed to use a hybrid GA-PSO approach in
the process of match score level fusion. By the weighted sum rule,
the hybrid GA-PSO optimized the weights which are associated with
the modalities in the fusion process to obtain the ideal ERR values.
The result demonstrated that the hybrid approach outperforms the
traditional approaches and both GA and PSO approaches in terms of
ERR in three different datasets.
Aizi et al. [179] proposed a match score level fusion strategy to
fuse the iris and the fingerprint. These two modalities were processed
separately to form score vectors. Then the fusion is applied at the score
level. For each modality, the score ranges are split into three zones of
interest corresponding to the proposed identification method. Then two
kinds of methods were used in the region of interest. The first is based
on the decision tree and the weighted sum (BCC), and the second is
based on the fuzzy logic method(BFL).
Sandhya et al. [183] proposed a match score level fusion method
for protecting fingerprint and palmprint templates. A binary vector is
generated for each biometric and the match score of each biometric is
calculated. Then score-level fusion is performed with T-operators.
The applications of match score level fusion are shown in Table 5.
To sum up, the fusion of match scores can reduce the influence of
various biometric data, complex recognition processes, and underlying
diversity. At the same time, it can preserve the measurement of indi-
vidual feature similarity, which can be used in the recognition system
of legal and illegal users.
4.3. Decision level fusion
In the decision level fusion, different biometrics are classified in-
dependently to obtain the classification decision (accept/reject). These
decisions from different modalities are fused to make the final decision,
as shown in decision level fusion in Fig. 4. The fusing process can
be implemented by a variety of methods, such as AND rule, OR rule,
Majority voting, decision table, Bayesian decision, etc [189].
Three kinds of decision level fusion methods were proposed [182],
naming Local Decision Fusion method (LDF), Global Decision Fusion
method (GDF), and Local-Global Decision Fusion method (LGDF),
which is performed by utilizing both local and global information.
The information was extracted by utilizing wavelet sub-bands of high
and low frequency. Then the nearest neighbor classifier was used to
classify the different frequencies of sub-bands. The resulting classes are
combined by means of weighted majority voting. The LDF and GDF
methods realized 9.4%, 11%, 10.6%, and 11.5% of average recognition
rates increase than unimodal. The LGDF method outperforms the
feature-score hybrid fusion, and a higher best average recognition rate
of 6.75% is achieved.
The applications of decision level fusion are shown in Table 6.
Because of the high flexibility of decision level fusion in information
processing, the system has low requirements on data transmission band-
width, which can effectively reflect the different types of information
from each side of the environment and target and process asynchronous
information. However, the decision level fusion ignores the low-level
interactions between multiple modalities.

701
Z. Qin et al.
Table 7
Related work of data fusion in rank level for identity recognition.
Reference Year Modalities Methods
Borda Count, Weighted Borda
Multiple Count, Highest and product of
[184] 2010
palmprints ranks, Bucklin Majority Voting, and
a new nonlinear approach
[185] 2011 Palmprints Logistic Regression
Borda Count and Logistic
[186] 2017 Gait, face
Regression
Highest Rank, Borda Count,
Different
[187] 2017 Logistic Regression/Weighted
fingers
Borda Count
Social Weighted Borda Count,
[188] 2020 behavioral Highest
traits Rank method
4.4. Rank level fusion
Rank level fusion is another specific type of decision level fu-
sion [12]. Rank level fusion is the process of reordering the rank list
from different unimodal match modules, thus making a final decision,
as shown in rank level fusion in Fig. 4. The reorder process can be
achieved by several methods, such as Borda Count, Weighted Borda
Count, Bucklin Majority Voting, etc. Since the rank list obtained from
unimodal match modules is the rank of possible objects registered in
the database, the compatibility problem of different modalities can be
avoided.
Rahman et al. [186] proposed a face extraction method based on
the HOG feature. Then K-nearest neighbors algorithm is used to classify
the samples by both face and gait features. Then the rank level fusion
is performed by Borda count and logistic regression approaches. The
two kinds of approaches achieved 93.99% and 96.67% of accuracy. The
applications of rank level fusion are shown in Table 7.
Sing et al. [190] proposed a rank level fusion strategy for creating
fuzzy ranks with Gaussian function based on the confidence of clas-
sifiers. This method can reflect associations between the outputs of
different classifiers. The fuzzy ranks are then fused by the confidence
factors of the classifier to produce the final result in the recognition
process.
In summary, as Renu et al. suggested [191], rank level fusion is an
adequate option in a multimodal identification system, as problems that
come from normalization and incompatibility are not as prominent as
that in score level fusion, and can fuse more information than decision
level fusion.
As mentioned in the article [192,193], the fusion performed before
matching, such as sensor layer fusion and feature layer fusion, can be
referred to as prior to matching fusion. The fusion after matching, such
as matching score level fusion, decision level fusion, and rank level
fusion, can be called after matching fusion.
It is worth mentioning that the fusion methods are also divided into
early fusion (feature-based), late fusion (decision-based), and hybrid
fusion [194].
In early fusion, the low-level information is considered. Features
of different biometrics can be effectively extracted and integrated to
boost the performance and achieve a better effect. Although different
modalities are usually highly correlated, this correlation is difficult to
extract at the feature level. Hinton et al. [195] believe that the corre-
lation between the information contained by different data streams is
certain at higher levels. Martínez et al. [196] proposed that early fusion
of multimodal data does not adequately demonstrate complementarity
between modalities, but may lead to input from redundant vectors.
In late fusion, fusions at a high level are performed. Usually, values
obtained by different unimodal are fused to get the final result under
a certain fusion method. As a high-level fusion, the results are of high
flexibility, strong anti-interference, and good fault tolerance. However,
702
Information Fusion 91 (2023) 694–712
Dataset Metrics
NIST BSSR1, IITD
touchless palmprint RR:99.41%
database
FAR:0.05%
PolyU database
EER:0.01%
Kinect gait and face
ACC:96.67%
datasets
Sfax-Miracl hand
RR: 99.04%
database
A social interaction
database of 241 ACC:99.45%
Twitter users
due to the high requirements for data preprocessing and feature ex-
traction, the cost of decision-level integration is relatively high, which
is also an existing shortcoming, which needs to be further studied and
solved.
Hybrid fusion tries to adopt the advantages of the two kinds of
fusion above in a common framework. And it has already been used in
the fields of multimodal speaker identification and multimedia event
detection.
5. Application: Multimodal combination
As mentioned above, multimodal identification technologies have
a very important role in several domain scenarios. According to the
modal features used in the real environment, identification technolo-
gies can be classified into the following two basic types: biometric
identification based on biometric features and device identification
based on hardware physical features. In recent years, based on the
research and application of biometric features, especially multimodal
biometric features, a variety of multimodal biometric systems with
different features and different fusion mechanisms have been proposed
and developed. At present, the research on biometric recognition is
mainly focused on the recognition of people. Biometric features can
be divided into two categories: one is physiological features, such as
facial features, fingerprints, and palmprint features, and the other is
behavioral features, such as voice and gait features.
In principle, it is natural that the more features used in recognition,
the deeper the portrayal of the entity and the higher the recognition
accuracy. However, from the perspective of practical application, due
to various environmental factors, technical factors, economic factors,
etc., the latest research has mainly focused on the fusion of the main
features rather than adding all features to the recognition system. In
this section, we will sort out the above situations separately, describe
the latest research progress in recognition scenarios, and summarize the
modal combinations commonly used in multimodal identity recognition
research.
5.1. Identity recognition based on physiological characteristics
In many studies, the most commonly used features can be divided
into facial features and hand features.
Yuan et al. [197] applied full space linear discriminant analysis
(FSLDA) to the recognition of ear images, face images, and combined
ear and face images. A multi-domain expert information fusion scheme
based on the rank fusion integration method was proposed [198],
which mainly integrates the results of different biometric matches by
using a new rank fusion method based on the characteristics of the face,
ear, and signature. S. Ribaric et al. [199] presented a bimodal biometric
verification system for physical access control based on the features of
the palmprint and the face.
Z. Qin et al.
Fig. 5. Statistical display of different multimodal combinations based on human
features.
In addition, the most unique feature of facial features is iris informa-
tion. In terms of physiology, this feature is unique, so there are many
recognition methods based on this feature. Haripraath et al. [200] in-
troduced a multimodal biometric system of iris and palmprint based on
wavelet packet analysis. The visible texture of a person’s iris and palm-
print is encoded into a compact two-dimensional wavelet packet coeffi-
cient sequence to generate a ’feature vector code’ for feature matching.
Chiara Galdi used simultaneous interpreting of image differences from
different sensors to solve the problem of sensor interoperability [201].
For hand characteristics, Li et al. [202] used the feature level fusion
method to fuse the hand biometrics including palmprint, hand shape
and finger joint print, and named them hand features . An efficient
matching algorithm based on phase correlation function (PCF) was
proposed [203]. The algorithm used palmprint and finger knuckle print
(FKP) to match. Attia et al. [204] use score level fusion method to
extract distinctive features from fingerprint and palmprint information
using PCANet depth learning method. Then multiclass SVM is utilized
to calculate the corresponding matching score. Finally, combine the
matching scores by different rules.
The modality combinations used in the above article are shown
in Fig. 5. The face is the most used modality (the video is basically
taken for human faces), and it can be used for identity recognition
together with almost all other modalities. It can be said that face is
one of the most important modalities in the task of identity recognition.
Secondly, palm prints, fingerprints, and signatures are also modalities
that have been studied a lot, while ears and iris are often used to assist
in verification with other modalities. It can also be seen that the most
researched combination is face and voice, including the combination of
audio and video, which is also one of the current research hotspots.
In addition to this, the face can be considered a complement to
the ear feature when it is combined with the ear together. Also, face
modality is often combined with hand features, signature, and voice
features, and this combination of two features is a more common modal
combination at present, which is also related to the current practical
application scenario. The combination of two modalities can improve
the accuracy of recognition and increase security, while also taking into
account the convenience of use. For the combination of three modalities
or even more modalities, it is more often applied in more stringent
recognition scenarios, while having higher requirements for fusion and
recognition algorithms.
5.2. Identity recognition based on sensor data
In addition to physiological features, dynamic features can also be
used for identification.
One common feature is video or throat movement data of the
speaker collected by special sensors. Rahul et al. [205] solved the
703
Information Fusion 91 (2023) 694–712
Table 8
Demonstration of sensor-based multimodal fusion methods for identity recognition.
Reference Modalities used Remarks (Pros/Cons)
Vocal cord vibration Limited perceived
[213]
and lip movement distance.
High security, not easy
[214] EEG and keystroke dynamics to be deceived, but small
sample data set.
Password,
The accuracy needs to
[215] keystroke behavior
be improved.
and gesture sliding
Moving steps and Rich data set and accurate
[216]
heart rate recognition rate.
Heart rate, gait and Implicit verification
[217]
breathing without interaction.
Identification of the same
[212] Lip language person using multiple
languages.
problem of dimension increase after the fusion of these features by
introducing modularity in attributes. A new multimodal short-term
memory (LSTM) architecture was described, which seamlessly unified
the visual and auditory modes from the beginning of each sequence
input [206]. The key idea is to extend the traditional LSTM by sharing
weights not only in time steps, but also in modes. Xin et al. [207]
proposed an effective attention guided depth audio face fusion method
to detect active speakers. Abozaid et al. [170] proposed a multimodal
biometric recognition method based on face and speech recognition
fusion. Through feature fusion and score fusion, the speech and face
biometric systems are combined into a single multimodal biometric
system. A special body conduction sensor throat microphone (TM)
was studied for joint speech vitality detection (VLD) and automatic
speaker verification (ASV) [208]. The work in [209] combined the
advantages of lip movement and speech, captures these two biometric
characteristics simultaneously through the built-in audio device on
the smartphone, and integrates them at the data level, solving the
shortcomings of the original system.
Robert et al. [210] combined facial features with dynamic features
for identity recognition, which can effectively resist the environmental
impact when collecting features. A fine tuning of multimodal correla-
tion in speaker verification system was proposed, which uses sound and
face, but no face is needed after training, and only voice and audio are
used as speaker verification system [211]. Nawaz et al. [212] studied
the relationship between faces and sounds in multiple languages spoken
by the same group. A security method was presented for verification of
smart home speakers in the Internet of things using millimeter wave
(MMW) radar [213] (see Table 8).
In addition, in practical application scenarios, there was often a lot
of identity recognition research based on biometric characteristics ob-
tained through mobile devices. For example, Dee et al. [218] used the
sensor’s touch pressure, positioning, and timing functions to develop
biometrics. Gesture keys based on gravity sensors and passwords were
used for verification [219]. A fuzzy authentication framework for gait
information based on a neural network was proposed [220], using a
3-D wearable acceleration sensor with orthogonal sensing direction to
measure a specific body position. A characterized biometric systems
based on motion and position sensor data is proposed [221]. And
a verification scheme combining password, keystroke behavior, and
gesture sliding was also proposed [215].
Meanwhile, identity recognition is also an important part of the
research on wearable devices. Vhaduri et al. [216] proposed a hidden
identity verification mechanism that uses three coarse-grained bio-
metric combinations: number of steps moved, heart rate and calorie
burn, or metabolic amount. Similarly, an verification mechanism was
proposed in [217] based on audio signals of heart rate, gait and
breathing.
Z. Qin et al.
Finally, the rapid development of sensor technology enables con-
sumer electronics (CE) research groups including manufacturers to
embed various practical sensors into handheld devices. In the next
generation CE equipment, in addition to traditional sensors such as
inertial measurement unit (IMU), camera, fingerprint, and proximity,
future sensors such as EEG and EMG are also included. A novel multi-
modal biometric system was proposed by combining EEG and keystroke
dynamics [214]. This method is not easy to be affected by false signals
and has high security. Behera et al. [222] proposed a new method for
air-based identity recognition by analyzing finger movement and brain
activity using sensors in a new generation CE device. The proposed
system in [223] extended finger input verification from touch screens to
any physical surface of IoT devices (such as intelligent access systems
and IoT devices) based on a touch-sensing technology with vibration
signals. It combines cryptography, behavioral and physiological char-
acteristics, and surface dependencies to provide a low-cost, tangible,
and enhanced security solution.
Compared with the static features of the human body, the modalities
that can reflect the human body features collected by sensors are often
rich in diversity and versatility and have better fusion capability at the
fusion level, which can be combined with other multiple modalities.
However, there are still problems such as insufficient differentiation
of modal data, difficulty in feature extraction, expensive hardware
acquisition equipment, and low universality.
5.3. Device identification
The rapid development of the Internet of things involves all as-
pects and effectively promotes the intelligent development of various
infrastructure fields. However, the frequent use of Internet of things
devices exposes more vulnerabilities. In the research of the security
field of the Internet of things, it is necessary to study the reliable trusted
verification of devices, especially in today’s technical environment, it
is necessary and important to carry out multi-level authentication for
highly vulnerable devices.
Sun et al. [224] proposed a unified privacy protection device discov-
ery and verification mechanism for heterogeneous D2D terminals based
on identity prefix encryption and ECDH technology. Sharma et al. [225]
proposed a multi-level security mechanism with password based, One
Time Password (OTP) based, and certificate-based. Gope et al. [226]
proposed a lightweight, privacy-protected two-factor authentication
scheme for IoT devices based on passwords and Physically Uncloneable
Functions (PUFs). Zhang et al. [227] can be based on the channel status
information of Wearable Proxy Devices (WPD) and IMD in WBANs,
using the special characteristics of received signal strength (RSS) ratio,
Distinguish legitimate users from attackers. A lightweight device veri-
fication protocol named Speaker–Microphone (S2M) is proposed [160]
that utilizes the frequency response of the speaker and microphone of
two wireless IoT devices as an acoustic hardware fingerprint.
6. Security analysis for learning-based identity recognition
Most of the identification methods mentioned above use relevant
knowledge in the field of machine learning. However, the above meth-
ods rarely take into account security issues while providing identifica-
tion services. Research shows that they are vulnerable to adversarial
attacks [228–230], which will cause errors in the identity recognition
results.
As is shown in Fig. 6, it describes the process of how adversarial
attacks work. Researchers obtain some relevant features of Mary, such
as her facial information, audio signals, and text data. Those features
are used to be the input samples. However, the attacker adds some
random noise to the samples, which results in some new adversarial
samples. Then the attacker uses these adversarial samples as input data
for the neural network. Due to the presence of noise, the identification
704
Information Fusion 91 (2023) 694–712
network cannot classify the input data correctly and get the wrong out-
put result, Lucy. This process is widely used in numerous multimodal
identity recognition frameworks. Here we divide the security of identity
recognition into two parts: adversarial attack and adversarial defense.
By introducing some common attack methods and defense measures to
analyze the reasons for the attacks and how to remedy them .
6.1. Adversarial attack
In the introduction of identity recognition methods above, deep
learning is widely used and the identity recognition realized by neural
networks has reached a very high accuracy, which is more robust
to random noise than other machine learning methods [231], but
in terms of adversarial samples, the neural network cannot achieve
satisfactory results or even achieve correct recognition. The adversarial
sample [232] is obtained by adding specific noise on the basis of the
original sample. The difference between the two samples is so small
that humans cannot distinguish the difference with the naked eye. But
the adversarial sample will deceive the classifier of the neural network,
and use it as a sample input will get the wrong output result. The
adversarial attack is to use adversarial samples to attack the network,
and this kind of attack is difficult to prevent.
In the past, research on adversarial samples mainly focused on
interference classification. The methods are mainly embodied as per-
turbing samples in advance, e.g. Nguyen et al. [230] used evolutionary
algorithms and gradient ascent to generate deceptive images. The
generated adversarial samples may not be recognized by humans at
all, but Deep Neural Networks (DNN) can identify and classify them
with high confidence. In this regard, Sabour et al. [233] proposed to
slightly perturb the image in the middle layer of the DNN to generate
a new adversarial sample. There is no significant difference between
the adversarial sample and the original sample, but the internal repre-
sentation of the adversarial sample is very similar to the interference
image that realizes the perturbation. Goodfellow et al. [234] introduced
the Fast Gradient Sign Method (FGSM) algorithm which uses network
gradients to generate adversarial samples. The required gradient infor-
mation can be calculated by backpropagation of the model. By adding
gradients to the input samples, the loss value is increased during model
training so as to realize the adversarial attack.
Poudel et al. [235] introduced a black box adversarial attack frame-
work in which the attack model is regarded as a black box. The attacker
can input the public dataset into the model and obtain the output
result, build an alternative model based on the input and output and
use the adversarial samples generated by the alternative model for
attacks. Through the above introduction, we know that adversarial
attacks pose a great threat to the application of deep learning, but in
turn Madryin et al. [236] pointed out that deep learning is not mature
enough from the perspective of the robustness of neural networks and
a Projected Gradient Descent (PGD) attack method that uses local first-
order information is proposed to prove his view. Croce et al. [237]
proposed to test the existing adversarial defense performance through
adversarial attacks. The article introduced two variants based on the
above PGD attack, and then combined the proposed new attack with
the existing two attacks to form an attack set for testing the robustness
of the adversarial defense.
Based on the fact that there was no effective method to calculate the
slight perturbations in the adversarial sample at that time, Moosavi-
Dezfooli et al. [238] proposed the DeepFool algorithm as a tool to
accurately calculate the adversarial perturbations, and thus evaluate
the robustness of the DNN classifier to the perturbations. From the
article, we can see that appropriate fine-tuning based on DeepFool eval-
uation results can improve network performance. Since the DeepFool
algorithm can only calculate against perturbations for a single image,
on this basis Moosavi-Dezfooli et al. [239] proposed an algorithm
for calculating general perturbations, through which perturbations can
Z. Qin et al.
Fig. 6. Process of adversarial attack towards learning-based identity recognition. The network above can be regarded as any identification neural network. The yellow and blue
layers of the network are convolution layers used to extract features from input samples, the gray spots are full connection layers used to realize dimension reduction of intermediate
data and the green spots are classifiers which decide the result of output. (For interpretation of the references to color in this figure legend, the reader is referred to the web
version of this article.)
be generated for any image. The experiment proves that The gen-
eral perturbations can be well extended to other network models. Su
et al. [240] studied an adversarial attack method that can achieve
network interference by modifying only one pixel in extreme cases.
Therefore, a one-pixel adversarial perturbations generation method
based on a differential evolution algorithm is proposed. The attack
only needs the classification probability label output by the model and
does not need internal information such as the network structure of the
model, so this method is also regarded as a semi-black box attack.
As one of the adversarial attacks, a Decision-based attack only needs
top-1 labels returned by the target network to generate adversarial
samples. Based on this, Li et al. [241] first came up with f-mixup
method to produce adversarial samples in the frequency domain and
then proposed f-attack as a new decision-based attack which achieves
SOAT(state of the art) performance. Based on the fact that there are a
few researches on robustness analyses of clustering algorithms against
adversarial noise, Emanuele et al. [242] designed a new black-box
gradient-free attack to fool clustering algorithms. The noise generated
by a novel optimization algorithm is proved to be effective for super-
vised algorithms as well. Chen et al. [243] were the first to propose
AoA (Attack on Attention) which is a semantic feature shared by DNNs
(Deep Neural Networks). By converting the cross entropy loss into
the attention loss, AoA greatly improved its transferability and could
be easily combined with other transferability-enhancement techniques.
The dataset with 50 000 adversarial samples generated by AOA is
named as DAmageNet, which can be considered as a benchmark for
robustness testing and adversarial training.
In next subsection, we will introduce an adversarial defense method
called defensive distillation. This method has excellent performance
against the current adversarial attacks, but Carlini et al. [244] be-
lieved that this does not mean that defensive distillation significantly
improves the robustness of neural network, and proposed three new
adversarial attack algorithms based on different distance metrics (zero
norm, two norm, and infinite norm). Neither the original network
nor the defensively distilled network can withstand the attacks of the
three algorithms. Sharmin et al. [245] used single-step and multi-
step FGSM to attack Artificial Neural Networks (ANN) and Spiking
Neural Networks (SNN), conducted comprehensive analyses of the two
networks based on their performance and proposed a framework for
generating adversarial attacks based on SNN. Experimental results show
that attacks generated from SNN using this framework are stronger than
those generated from ANN.
6.2. Adversarial defense
In view of the fact that adversarial samples will greatly affect the
recognition and classification results of deep learning models, adver-
sarial attacks have become a major security risk for neural networks.
This has also led more researchers to focus on the field of adversarial
705
Information Fusion 91 (2023) 694–712
defense. Compared to the attack method, the realization of defense
technology appears to be more open. Defense technology based on
various methods has been proposed to resist adversarial attacks. These
methods have significantly enhanced the robustness of the network and
achieved good performance.
Papernot et al. [246] proposed a defense mechanism called defen-
sive distillation based on knowledge distillation, which can significantly
reduce the success rate of adversarial samples against neural network
attacks and increase the difficulty of generating adversarial samples.
After the introduction of defensive distillation, the generation efficiency
of adversarial samples under DNN conditions was reduced from 95% to
0.5%. In the previous studies on resisting adversarial samples, most of
the methods were used to improve the DNN model. In the previous
section, Madry et al. [236] have introduced the PGD attack method,
but unlike the predecessors, they use adversarial attacks for adversarial
training from the perspective of robust optimization. Training neu-
ral networks through PGD attacks can greatly improve the network’s
performance and improve the network’s resistance to various types of
Adversarial attacks.
Hendrycks et al. [247] adopted self-supervised learning to improve
the uncertainty and robustness of the model. Through research, it is
found that the self-supervised method has achieved significant improve-
ments in robustness, label damage, and common input damage. Juuti
et al. [248] proposed the first general technology PRADA to detect
model extraction attacks. By analyzing the distribution of users’ con-
tinuous queries, it can identify deviations from the normal distribution
to detect extraction attacks. Experiments show that it can detect all the
past model extracts attacks.
With open-set recognition becoming an important aspect of deep
learning, it has been widely used in the real world and proved to be
vulnerable to adversarial samples. Therefore, it is essential to protect
it from adversarial attacks. Shao et al. [249] proposed an OSDN-CAML
(Open-Set Defense Network with Clean-Adversarial Mutual Learning)
to defend against adversarial attacks. Extensive experiments have been
done to prove the superiority of this method in solving the problem
above.
Through the previous work, people have realized that adversarial
training is an effective means to resist adversarial attacks, but almost
all adversarial training methods in the past have focused on global
defense. For this, Chen et al. [250] proposed the use of smooth ad-
versarial training (SAT) to solve the problem of specific targets being
attacked in actual scenarios. It includes two strategies: smooth distilla-
tion and smooth cross-entropy loss function, which helps SAT achieve
the most advanced defense capabilities in different confrontation at-
tacks. Yuan et al. [251] proposed a new method of network defense
using adversarial dual network learning of random non-linear image
transformation. Random non-linear image transformation is used to in-
terfere and destroy network noise, and the cleaning network is designed
to restore the image content destroyed by the graphics transformation.
Z. Qin et al.
Sheikholeslami et al. [252] introduced a randomization-based sampling
mechanism for adversarial defense. This method samples the network
middle layer data in the test phase and uses variance minimization
to process sampling results to obtain the sampling probability. Test
samples with high classification uncertainty are regarded as adversarial
samples.
In addition, there is another research direction that faces similar
security issues: digital watermarking is the embedding of a set of
information into another data entity, which sounds similar to the
way of fighting attacks. In this regard, Quiring et al. [253] linked
the offensive and defensive technology of digital watermarking with
the research on the adversarial offensive and adversarial defensive.
The article shows that the defensive strategy of digital watermark-
ing can effectively resist model extraction attacks. Similarly, related
technologies that strengthen machine learning in adversarial defense
have achieved good results in defending against watermarking oracle
attacks. Akhtar et al. [229] proposed a special framework for the
general adversarial perturbations in [239], which is used to detect
general perturbations in images and remove image perturbations to
achieve accurate network recognition. The framework uses a Support
Vector Machine (SVM) to detect perturbations and restores the image
through a Perturbation Rectifying Network (PRN). Among the various
defense methods proposed above, adversarial training has proved to
be the most effective method so far, but Wang et al. [254] found
that some correct samples that were misclassified during the training
process had a significant impact on the defense effect. Based on this,
a method called Misclassification Aware adveRsarial Training (MART)
is proposed, which distinguishes misclassified and correctly classified
samples during the training process.
7. Discussion
7.1. Challenges
While dealing with data from multiple information sources in var-
ious complicated scenes, it is likely to face the challenge of low-
quality data, including data incompleteness, data inconsistency, limited
computing resources, and so on.
Data Incompleteness. In some cases, the difficulty to collect samples
of certain categories results in a small-scale dataset. The approaches to
resolving the problem include migration learning, data enhancement, a
priori knowledge, and so on. For example, the idea of transfer learning
can be adapted to train a neural network with larger datasets of
similar fields, and then fine-tune the pre-training network with smaller
collected data sets. Data enhancement can also be applied to rotate,
translate or generate a generic adversarial network (GAN) to expand
the scale of the training datasets.
Data Inconsistency. While in the actual training process, the data
under some specific situations are difficult to collect, so the alternative
is used to replace the ones. Thus, this method might lead to a mismatch
of the data during the training and testing phase. At the same time,
due to limitations of objective factors, data inconsistency is also likely
to occur in the collection procedure. To solve the problem, more data
close to the needs can be synthesized through data expansion. Or we
can also unify all data to another distribution according to the actual
demands.
Limited Computational resources. Under complicated situations (such
as edge computing scenarios in industrial Internet and mobile Internet),
computing resources may be limited. Therefore, it is quite necessary to
develop lightweight neural networks to meet the requirements. There
exists four main approaches, to be specific, the artificial design of
lightweight neural network model, automatic design of neural network
based on neural network architecture search, CNN model compression
and automatic model compression based on AutoML. The lightweight
model is required to maintain normal inference performance under
706
Information Fusion 91 (2023) 694–712
complicated circumstances. Most of the lightweight model structures
are hand-crafted, while the performance of the model is completely
dependent on personal ability. We need an automatic method based on
neural network architecture search to realize the design of a lightweight
model.
7.2. Future directions
Multimodal fusion identity recognition ensures that user informa-
tion is safe through multimodal multiple guarantees. At the same time,
it can adapt to application changes in complicated scenes, as well as
letting the identification results more accurate. In the future, several
directions can be further explored.
The convenience and stability of modal acquisition. Some biometrics
such as voice, heart sound and ECG, are easily disturbed by environ-
ment, noise and other factors. External interference will cause the loss
of information, further affecting the accuracy of the recognition result.
Based on this, flexible wearable devices can be designed to collect
and improve the robustness of the feature acquisition system. Besides,
physical devices are the main source of feature acquisition, so it is
significant to choose. Selecting equipment with high stability, high
cost performance, low energy consumption and low noise interference
will help improve the accuracy of recognition and certification at
the hardware and peripheral levels. Especially for some elders, these
equipment must be convenient to use, quick to transport and provide
an excellent user experience.
Different processing approaches for different modes. Different modali-
ties are presented in various ways, containing different semantic infor-
mation. In this paper, we present three various models. The approach to
extracting semantic information more effectively from modals remains
a problem. Finding a general model for different modes or raising
corresponding processing methods in terms of their characteristics.
These research directions need to be further explored.
The selection of fusion strategies. In this paper, we describe four
kinds of fusion methods, feature level fusion, match score level fusion,
decision level fusion, and rank level fusion. However, there has no
unified standard for the selection of a fusion strategy. Whether there
exist differences in fusion methods cor responding to biological charac-
teristics and physical equipment characteristics is worth discussing. In
addition to the existing several fusion strategies, other levels of fusion
methods that can better combine the semantic information and improve
the recognition results, or deal with low spatial resolution, reasonable
design of evoked scenarios, and small sample size are also required
to be further explored. Adaptive fusion, soft biometrics and quality
information fusion, user-specific fusion, or online learning-based fusion
might be promising directions.
Multimodal security and privacy. Multi-biometrics has contributed
significantly to enhance the identity recognition performance. At the
same time, it is likely to invade and disclose privacy. The high and
rising amount of multi-biometrics used in identity recognition tasks will
increase the risk and concerns. A related challenge is to balance secu-
rity and privacy in a reasonable way. Specifically, utilizing a privacy
security scheme may lead to a decline in identity recognition accuracy.
Thus, targeted encryption algorithms and anti-attack methods can be
further investigated in this research field.
Data feature space compression. Multi-dimensional features can be
extracted from multi-modal data. In the process of data fusion and
matching, it is necessary to calculate the most representative features
for classification and recognition, so as to obtain a group of precise
feature combinations. This is aimed to achieve the effect that different
pattern points are far away and similar pattern points are close. There-
fore, a cost-effective compression method in feature space during the
process of feature matching and fusion is worth to be explored.
Z. Qin et al.
8. Conclusion
As a key technique for identification in Mobile Internet and Indus-
trial Internet of Things, learning-based multimodal identity recognition
can combine and leverage abundant modality features to achieve im-
proved identification accuracy, compared with unimodal manner. This
review has provided an introduction to the modalities used for identity
recognition as well as its limitations. Moreover, multimodal identity
recognition has been discussed, including a presentation and discussion
of four fusion strategies for multimodal features and a summary of
multimodal combinations applied to different scenarios. Furthermore,
the security concerns of multimodal identity recognition have been
analyzed, and so does the current challenges and potential solutions
for multimodal identification.
Moreover, by exploiting complementary information between dif-
ferent modalities, it is capable of satisfying the requirements of identity
recognition applications in complex scenarios.
CRediT authorship contribution statement
Zhen Qin: Conceptualization, Writing – review & editing, Super-
vision, Funding acquisition. Pengbiao Zhao: Investigation, Writing –
original draft, Writing – review & editing. Tianming Zhuang: Investi-
gation, Writing – original draft, Visualization. Fuhu Deng: Methodol-
ogy, Writing – review & editing. Yi Ding: Resources, Writing – review
& editing, Project administration. Dajiang Chen: Resources, Writing –
review & editing, Project administration.
Declaration of competing interest
The authors declare that they have no known competing finan-
cial interests or personal relationships that could have appeared to
influence the work reported in this paper.
Data availability
No data was used for the research described in the article.
Acknowledgments
This work was supported in part by the National Natural Science
Foundation of China (No. 62072074, No. 62076054, No. 62027827,
No. 61902054, No. 62002047), the Frontier Science and Technology
Innovation Projects of National Key R&D Program (No. 2019QY1405),
the Sichuan Science and Technology Innovation Platform and Talent
Plan (No. 2020JDJQ0020, No. 2022JDJQ0039), the Sichuan Science
and Technology Support Plan (No. 2020YFSY0010, No. 2022YFQ0045,
No. 2022YFS0220, No. 2019YJ0636, No. 2021YFG0131), and the
Medico-Engineering Cooperation Funds from University of Electronic
Science and Technology of China (No. ZYGX2021YGLH212, No.
ZYGX2022YGRH012).
References
[1] A.K. Jain, S.Z. Li, Handbook of Face Recognition, vol. 1, Springer, 2011.
[2] K.W. Bowyer, M.J. Burge, Handbook of Iris Recognition, Springer, 2016.
[3] D. Maltoni, D. Maio, A.K. Jain, S. Prabhakar, Handbook of Fingerprint
Recognition, Springer Science & Business Media, 2009.
[4] M. Singh, R. Singh, A. Ross, A comprehensive overview of biometric fusion,
Inf. Fusion 52 (2019) 187–205.
[5] G. Muhammad, F. Alshehri, F. Karray, A. El Saddik, M. Alsulaiman, T.H.
Falk, A comprehensive survey on multimodal medical signals fusion for smart
healthcare systems, Inf. Fusion 76 (2021) 355–375.
[6] G. Slanzi, G. Pizarro, J.D. Velásquez, Biometric information fusion for web user
navigation and preferences analysis: An overview, Inf. Fusion 38 (2017) 12–21.
[7] J. Li, Q. Wang, Multi-modal bioelectrical signal fusion analysis based on
different acquisition devices and scene settings: Overview, challenges, and novel
orientation, Inf. Fusion 79 (2022) 229–247.
707
Information Fusion 91 (2023) 694–712
[8] D. Ramachandram, G.W. Taylor, Deep multimodal learning: A survey on recent
advances and trends, IEEE Signal Process. Mag. 34 (6) (2017) 96–108.
[9] M. Ghayoumi, A review of multimodal biometric systems: Fusion methods
and their applications, in: 2015 IEEE/ACIS 14th International Conference on
Computer and Information Science, ICIS, IEEE, 2015, pp. 131–136.
[10] I. Olade, H.-n. Liang, C. Fleming, A review of multimodal facial biomet-
ric authentication methods in mobile devices and their application in head
mounted displays, in: 2018 IEEE SmartWorld, Ubiquitous Intelligence &
Computing, Advanced & Trusted Computing, Scalable Computing & Commu-
nications, Cloud & Big Data Computing, Internet of People and Smart City
Innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI), IEEE, 2018,
pp. 1997–2004.
[11] R. Ryu, S. Yeom, S.-H. Kim, D. Herbert, Continuous multimodal biometric
authentication schemes: a systematic review, IEEE Access (2021).
[12] S.K. Choudhary, A.K. Naik, Multimodal biometric authentication with secured
templates—A review, in: 2019 3rd International Conference on Trends in
Electronics and Informatics, ICOEI, IEEE, 2019, pp. 1062–1069.
[13] H. Murase, S.K. Nayar, Visual learning and recognition of 3-D objects from
appearance, Int. J. Comput. Vis. 14 (1) (1995) 5–24.
[14] T.F. Cootes, G.J. Edwards, C.J. Taylor, Active appearance models, IEEE Trans.
Pattern Anal. Mach. Intell. 23 (6) (2001) 681–685.
[15] Y. Weiwei, Face recognition using constrained active appearance model, in:
2009 Third International Symposium on Intelligent Information Technology
Application Workshops, IEEE, 2009, pp. 348–351.
[16] M.A. Khan, C. Xydeas, H. Ahmed, Multi-model AAM framework for face image
modeling, in: 2013 18th International Conference on Digital Signal Processing,
DSP, IEEE, 2013, pp. 1–5.
[17] B. Gökberk, M.O. İrfanoğlu, L. Akarun, 3D shape-based face representation
and feature extraction for face recognition, Image Vis. Comput. 24 (8) (2006)
857–869.
[18] H. Song, U. Yang, S. Lee, K. Sohn, 3D face recognition based on facial shape
indexes with dynamic programming, in: International Conference on Biometrics,
Springer, 2006, pp. 99–105.
[19] B. Gokberk, H. Dutagaci, A. Ulas, L. Akarun, B. Sankur, Representation plurality
and fusion for 3-D face recognition, IEEE Trans. Syst. Man Cybern. B 38 (1)
(2008) 155–173.
[20] N. Dalal, B. Triggs, Histograms of oriented gradients for human detection,
in: 2005 IEEE Computer Society Conference on Computer Vision and Pattern
Recognition (CVPR’05), 1, Ieee, 2005, pp. 886–893.
[21] O. Déniz, G. Bueno, J. Salido, F. De la Torre, Face recognition using histograms
of oriented gradients, Pattern Recognit. Lett. 32 (12) (2011) 1598–1603.
[22] T.-T. Do, E. Kijak, Face recognition using co-occurrence histograms of oriented
gradients, in: 2012 IEEE International Conference on Acoustics, Speech and
Signal Processing, ICASSP, IEEE, 2012, pp. 1301–1304.
[23] T. Ojala, M. Pietikäinen, D. Harwood, A comparative study of texture measures
with classification based on featured distributions, Pattern Recognit. 29 (1)
(1996) 51–59.
[24] D.G. Lowe, Distinctive image features from scale-invariant keypoints, Int. J.
Comput. Vis. 60 (2) (2004) 91–110.
[25] D. Huang, C. Shan, M. Ardabilian, Y. Wang, L. Chen, Local binary patterns
and its application to facial image analysis: a survey, IEEE Trans. Syst. Man.
Cybern. C 41 (6) (2011) 765–781.
[26] J.R. Barr, K.W. Bowyer, P.J. Flynn, S. Biswas, Face recognition from video: A
review, Int. J. Pattern Recognit. Artif. Intell. 26 (05) (2012) 1266002.
[27] C. Ding, D. Tao, Trunk-branch ensemble convolutional neural networks for
video-based face recognition, IEEE Trans. Pattern Anal. Mach. Intell. 40 (4)
(2017) 1002–1014.
[28] F. Mokhayeri, E. Granger, G.-A. Bilodeau, Domain-specific face synthesis for
video face recognition from a single sample per person, IEEE Trans. Inf.
Forensics Secur. 14 (3) (2018) 757–772.
[29] F. Mokhayeri, E. Granger, A paired sparse representation model for robust face
recognition from a single sample, Pattern Recognit. 100 (2020) 107129.
[30] W. Wang, R. Wang, Z. Huang, S. Shan, X. Chen, Discriminant analysis on
Riemannian manifold of Gaussian distributions for face recognition with image
sets, in: Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition, 2015, pp. 2048–2057.
[31] R.E. Haamer, K. Kulkarni, N. Imanpour, M.A. Haque, E. Avots, M. Breisch, K.
Nasrollahi, S. Escalera, C. Ozcinar, X. Baro, et al., Changes in facial expression
as biometric: a database and benchmarks of identification, in: 2018 13th IEEE
International Conference on Automatic Face & Gesture Recognition (FG 2018),
IEEE, 2018, pp. 621–628.
[32] S.T. Kim, D.H. Kim, Y.M. Ro, Spatio-temporal representation for face authen-
tication by using multi-task learning with human attributes, in: 2016 IEEE
International Conference on Image Processing, ICIP, IEEE, 2016, pp. 2996–3000.
[33] F. Hajati, M. Tavakolian, S. Gheisari, Y. Gao, A.S. Mian, Dynamic texture
comparison using derivative sparse representation: Application to video-based
face recognition, IEEE Trans. Hum.-Mach. Syst. 47 (6) (2017) 970–982.
Z. Qin et al.
[34] V. Caselles, R. Kimmel, G. Sapiro, Geodesic active contours, in: Proceedings of
IEEE International Conference on Computer Vision, IEEE, 1995, pp. 694–699.
[35] V. Caselles, R. Kimmel, G. Sapiro, Geodesic active contours, Int. J. Comput.
Vis. 22 (1) (1997) 61–79.
[36] M. Mustafa, A.J. Mohamad Faizal, M.R. Mohd Shafry, A. Illiasaak, Implemen-
tation of geodesic active contour approach for pigment spots segmentation on
the iris surface, J. Comput. Sci. 12 (11) (2016) 564–571, http://dx.doi.org/10.
3844/jcssp.2016.564.571.
[37] N. Susitha, R. Subban, Reliable pupil detection and iris segmentation algorithm
based on SPS, Cogn. Syst. Res. 57 (2019) 78–84.
[38] T.M. Khan, D.G. Bailey, M.A. Khan, Y. Kong, Real-time iris segmentation
and its implementation on FPGA, J. Real-Time Image Process. 17 (5) (2020)
1089–1102.
[39] W. Zhang, X. Lu, Y. Gu, Y. Liu, X. Meng, J. Li, A robust iris segmentation
scheme based on improved U-net, IEEE Access 7 (2019) 85082–85089.
[40] X. Wu, L. Zhao, Study on iris segmentation algorithm based on dense U-net,
IEEE Access 7 (2019) 123959–123968.
[41] S. Lian, Z. Luo, Z. Zhong, X. Lin, S. Su, S. Li, Attention guided U-net for accurate
iris segmentation, J. Vis. Commun. Image Represent. 56 (2018) 296–304.
[42] S.M. Patil, R.R. Jha, A. Nigam, IpSegNet: deep convolutional neural network
based segmentation framework for iris and pupil, in: 2017 13th International
Conference on Signal-Image Technology & Internet-Based Systems, SITIS, IEEE,
2017, pp. 184–191.
[43] M. Korobkin, G. Odinokikh, Y. Efimov, I. Solomatin, I. Matveev, Iris segmen-
tation in challenging conditions, Pattern Recognit. Image Anal. 28 (4) (2018)
652–657.
[44] P. Rot, Ž. Emeršič, V. Struc, P. Peer, Deep multi-class eye segmentation for
ocular biometrics, in: 2018 IEEE International Work Conference on Bioinspired
Intelligence, IWOBI, IEEE, 2018, pp. 1–8.
[45] N. Liu, H. Li, M. Zhang, J. Liu, Z. Sun, T. Tan, Accurate iris segmentation
in non-cooperative environments using fully convolutional networks, in: 2016
International Conference on Biometrics, ICB, IEEE, 2016, pp. 1–8.
[46] Y. He, S. Wang, K. Pei, M. Liu, J. Lai, Visible spectral iris segmentation via
deep convolutional network, in: Chinese Conference on Biometric Recognition,
Springer, 2017, pp. 428–435.
[47] J. Geng, Y. Li, T. Chian, SIFT based iris feature extraction and matching,
in: Geoinformatics 2007: Geospatial Information Science, 6753, International
Society for Optics and Photonics, 2007, p. 67532F.
[48] Z. He, Z. Sun, T. Tan, Z. Wei, Efficient iris spoof detection via boosted local
binary patterns, in: International Conference on Biometrics, Springer, 2009, pp.
1080–1090.
[49] H. Zhang, Z. Sun, T. Tan, Contact lens detection based on weighted LBP, in:
2010 20th International Conference on Pattern Recognition, IEEE, 2010, pp.
4279–4282.
[50] K.B. Raja, R. Raghavendra, C. Busch, Binarized statistical features for improved
iris and periocular recognition in visible spectrum, in: 2nd International
Workshop on Biometrics and Forensics, IEEE, 2014, pp. 1–6.
[51] H. Demirel, G. Anbarjafari, Iris recognition system using combined histogram
statistics, in: 2008 23rd International Symposium on Computer and Information
Sciences, IEEE, 2008, pp. 1–4.
[52] K. Nguyen, C. Fookes, A. Ross, S. Sridharan, Iris recognition with off-the-shelf
CNN features: A deep learning perspective, IEEE Access 6 (2017) 18848–18855.
[53] D. Menotti, G. Chiachia, A. Pinto, W.R. Schwartz, H. Pedrini, A.X. Falcao, A.
Rocha, Deep representations for iris, face, and fingerprint spoofing detection,
IEEE Trans. Inf. Forensics Secur. 10 (4) (2015) 864–879.
[54] C.-W. Tan, A. Kumar, Integrating ocular and iris descriptors for fake iris image
detection, in: 2nd International Workshop on Biometrics and Forensics, IEEE,
2014, pp. 1–4.
[55] P. Silva, E. Luz, R. Baeta, H. Pedrini, A.X. Falcao, D. Menotti, An approach
to iris contact lens detection based on deep image representations, in: 2015
28th SIBGRAPI Conference on Graphics, Patterns and Images, IEEE, 2015, pp.
157–164.
[56] X. Liu, Y. Bai, Y. Luo, Z. Yang, Y. Liu, Iris recognition in visible spectrum based
on multi-layer analogous convolution and collaborative representation, Pattern
Recognit. Lett. 117 (2019) 66–73.
[57] H. Rai, A. Yadav, Iris recognition using combined support vector machine and
hamming distance approach, Expert Syst. Appl. 41 (2) (2014) 588–593.
[58] Y. Du, T. Bourlai, J. Dawson, Automated classification of mislabeled near-
infrared left and right iris images using convolutional neural networks, in:
2016 IEEE 8th International Conference on Biometrics Theory, Applications and
Systems, BTAS, IEEE, 2016, pp. 1–6.
[59] F. Marra, G. Poggi, C. Sansone, L. Verdoliva, A deep learning approach for iris
sensor model identification, Pattern Recognit. Lett. 113 (2018) 46–53.
[60] Z. Zhao, A. Kumar, Towards more accurate iris recognition using deeply learned
spatially corresponding features, in: Proceedings of the IEEE International
Conference on Computer Vision, 2017, pp. 3809–3818.
708
Information Fusion 91 (2023) 694–712
[61] F. Gaxiola, P. Melin, F. Valdez, J.R. Castro, Person recognition with modular
deep neural network using the iris biometric measure, in: Fuzzy Logic Aug-
mentation of Neural and Optimization Algorithms: Theoretical Aspects and Real
Applications, Springer, 2018, pp. 69–80.
[62] M. Baqar, A. Ghani, A. Aftab, S. Arbab, S. Yasin, Deep belief networks for iris
recognition based on contour detection, in: 2016 International Conference on
Open Source Systems & Technologies, ICOSST, IEEE, 2016, pp. 72–77.
[63] F. He, Y. Han, H. Wang, J. Ji, Y. Liu, Z. Ma, Deep learning architecture for iris
recognition based on optimal gabor filters and deep belief network, J. Electron.
Imaging 26 (2) (2017) 023005.
[64] Y. Lu, H. Pan, Application of iris images in racial classifications based on dilate
convolution and residual network, IEEE Access 7 (2019) 182395–182405.
[65] R. Deepak, A.V. Nayak, K. Manikantan, Ear detection using active contour
model, in: 2016 International Conference on Emerging Trends in Engineering,
Technology and Science, ICETETS, IEEE, 2016, pp. 1–7.
[66] S.M. Islam, M. Bennamoun, R. Davies, Fast and fully automatic ear detection
using cascaded adaboost, in: 2008 IEEE Workshop on Applications of Computer
Vision, IEEE, 2008, pp. 1–6.
[67] L. Yuan, F. Zhang, Ear detection based on improved adaboost algorithm, in:
2009 International Conference on Machine Learning and Cybernetics, 4, IEEE,
2009, pp. 2414–2417.
[68] A. Abaza, C. Hebert, M.A.F. Harrison, Fast learning ear detection for real-
time surveillance, in: 2010 Fourth IEEE International Conference on Biometrics:
Theory, Applications and Systems, BTAS, IEEE, 2010, pp. 1–6.
[69] S. Prakash, U. Jayaraman, P. Gupta, Connected component based technique for
automatic ear detection, in: 2009 16th IEEE International Conference on Image
Processing, ICIP, IEEE, 2009, pp. 2741–2744.
[70] S. Prakash, U. Jayaraman, P. Gupta, Ear localization using hierarchical clus-
tering, in: Optics and Photonics in Global Homeland Security V and Biometric
Technology for Human Identification VI, 7306, SPIE, 2009, pp. 361–369.
[71] S. Prakash, P. Gupta, An efficient ear localization technique, Image Vis. Comput.
30 (1) (2012) 38–50.
[72] S. Ansari, P. Gupta, Localization of ear using outer helix curve of the ear,
in: 2007 International Conference on Computing: Theory and Applications
(ICCTA’07), IEEE, 2007, pp. 688–692.
[73] J. Zhou, S. Cadavid, M. Abdel-Mottaleb, Histograms of categorized shapes for
3D ear detection, in: 2010 Fourth IEEE International Conference on Biometrics:
Theory, Applications and Systems, BTAS, IEEE, 2010, pp. 1–6.
[74] A. Pflug, P.M. Back, C. Busch, Towards making HCS ear detection robust
against rotation, in: 2012 IEEE International Carnahan Conference on Security
Technology, ICCST, IEEE, 2012, pp. 90–96.
[75] M.R. Ganesh, R. Krishna, K. Manikantan, S. Ramachandran, Entropy based
binary particle swarm optimization and classification for ear detection, Eng.
Appl. Artif. Intell. 27 (2014) 115–128.
[76] P. Chidananda, P. Srinivas, K. Manikantan, S. Ramachandran, Entropy-
cum-hough-transform-based ear detection using ellipsoid particle swarm
optimization, Mach. Vis. Appl. 26 (2) (2015) 185–203.
[77] J. Lei, X. You, M. Abdel-Mottaleb, Automatic ear landmark localization,
segmentation, and pose classification in range images, IEEE Trans. Syst. Man
Cybern. 46 (2) (2015) 165–176.
[78] L. Yuan, W. Liu, Y. Li, Non-negative dictionary based sparse representation
classification for ear recognition with occlusion, Neurocomputing 171 (2016)
540–550.
[79] R. Khorsandi, S. Cadavid, M. Abdel-Mottaleb, Ear recognition via sparse
representation and gabor filters, in: 2012 IEEE Fifth International Conference on
Biometrics: Theory, Applications and Systems, BTAS, IEEE, 2012, pp. 278–282.
[80] R. Khorsandi, M. Abdel-Mottaleb, Gender classification using 2-D ear images
and sparse representation, in: 2013 IEEE Workshop on Applications of Computer
Vision, WACV, IEEE, 2013, pp. 461–466.
[81] R. Khorsandi, A. Taalimi, M. Abdel-Mottaleb, Robust biometrics recognition
using joint weighted dictionary learning and smoothed L0 norm, in: 2015 IEEE
7th International Conference on Biometrics Theory, Applications and Systems,
BTAS, IEEE, 2015, pp. 1–6.
[82] T. Ying, Z. Debin, Z. Baihuan, Ear recognition based on weighted wavelet
transform and DCT, in: The 26th Chinese Control and Decision Conference
(2014 CCDC), IEEE, 2014, pp. 4410–4414.
[83] K. Soni, S.K. Gupta, U. Kumar, S.L. Agrwal, A new gabor wavelet transform
feature extraction technique for ear biometric recognition, in: 2014 6th IEEE
Power India International Conference, PIICON, IEEE, 2014, pp. 1–3.
[84] A. Tahmasebi, H. Pourghassem, H. Mahdavi-Nasab, An ear identification system
using local-gabor features and knn classifier, in: 2011 7th Iranian Conference
on Machine Vision and Image Processing, IEEE, 2011, pp. 1–4.
[85] A. Kumar, C. Wu, Automated human identification using ear imaging, Pattern
Recognit. 45 (3) (2012) 956–968.
[86] B. Arbab-Zavar, M.S. Nixon, Robust log-gabor filter for ear biometrics, in: 2008
19th International Conference on Pattern Recognition, IEEE, 2008, pp. 1–4.
Z. Qin et al.
[87] D. Sánchez, P. Melin, Optimization of modular granular neural networks using
hierarchical genetic algorithms for human recognition using the ear biometric
measure, Eng. Appl. Artif. Intell. 27 (2014) 41–56.
[88] F.N. Sibai, A. Nuaimi, A. Maamari, R. Kuwair, Ear recognition with feed-forward
artificial neural networks, Neural Comput. Appl. 23 (5) (2013) 1265–1273.
[89] S. Banerjee, A. Chatterjee, Image set based ear recognition using novel dic-
tionary learning and classification scheme, Eng. Appl. Artif. Intell. 55 (2016)
37–46.
[90] S. Cadavid, M. Abdel-Mottaleb, Human identification based on 3D ear mod-
els, in: 2007 First IEEE International Conference on Biometrics: Theory,
Applications, and Systems, IEEE, 2007, pp. 1–6.
[91] J. Lei, J. Zhou, M. Abdel-Mottaleb, A novel shape-based interest point descriptor
(SIP) for 3D ear recognition, in: 2013 IEEE International Conference on Image
Processing, IEEE, 2013, pp. 4176–4180.
[92] X. Sun, G. Wang, 3D ear matching using local salient shape feature, in: 2013
International Conference on Computer-Aided Design and Computer Graphics,
IEEE, 2013, pp. 377–378.
[93] A. Gyaourova, A. Ross, A novel coding scheme for indexing fingerprint patterns,
in: Joint IAPR International Workshops on Statistical Techniques in Pattern
Recognition (SPR) and Structural and Syntactic Pattern Recognition, SSPR,
Springer, 2008, pp. 755–764.
[94] P. Tuyls, A.H. Akkermans, T.A. Kevenaar, G.-J. Schrijen, A.M. Bazen,
R.N. Veldhuis, Practical biometric authentication with template protection,
in: International Conference on Audio-and Video-Based Biometric Person
Authentication, Springer, 2005, pp. 436–446.
[95] J. Bringer, V. Despiegel, Binary feature vector fingerprint representation
from minutiae vicinities, in: 2010 Fourth IEEE International Conference on
Biometrics: Theory, Applications and Systems, BTAS, IEEE, 2010, pp. 1–6.
[96] L. Sha, F. Zhao, X. Tang, Improved fingercode for filterbank-based fingerprint
matching, in: Proceedings 2003 International Conference on Image Processing
(Cat. No. 03CH37429), 2, IEEE, 2003, pp. II–895.
[97] H.İ. Öztürk, B. Selbes, Y. Artan, MinNet: Minutia patch embedding network
for automated latent fingerprint recognition, in: Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition, 2022, pp. 1627–1635.
[98] M. Tico, P. Kuosmanen, J. Saarinen, Wavelet domain features for fingerprint
recognition, Electron. Lett. 37 (1) (2001) 21–22.
[99] T. Amornraksa, S. Tachaphetpiboon, Fingerprint recognition using DCT features,
Electron. Lett. 42 (9) (2006) 522–523.
[100] H. Xu, R.N. Veldhuis, A.M. Bazen, T.A. Kevenaar, T.A. Akkermans, B. Gokberk,
Fingerprint verification using spectral minutiae representations, IEEE Trans. Inf.
Forensics Secur. 4 (3) (2009) 397–409.
[101] X. Wu, D. Zhang, K. Wang, B. Huang, Palmprint classification using principal
lines, Pattern Recognit. 37 (10) (2004) 1987–1998.
[102] X. Wu, K. Wang, D. Zhang, Palmprint recognition using directional line
energy feature, in: Proceedings of the 17th International Conference on Pattern
Recognition, 2004. ICPR 2004, 4, IEEE, 2004, pp. 475–478.
[103] F. Li, M.K. Leung, X. Yu, Palmprint identification using hausdorff distance, in:
IEEE International Workshop on Biomedical Circuits and Systems, 2004, IEEE,
2004, pp. S3–3.
[104] D.-S. Huang, W. Jia, D. Zhang, Palmprint verification based on principal lines,
Pattern Recognit. 41 (4) (2008) 1316–1328.
[105] J. Canny, A computational approach to edge detection, IEEE Trans. Pattern
Anal. Mach. Intell. PAMI-8 (6) (1986) 679–698, http://dx.doi.org/10.1109/
TPAMI.1986.4767851.
[106] X. Wu, D. Zhang, K. Wang, Fusion of phase and orientation information for
palmprint authentication, Pattern Anal. Appl. 9 (2) (2006) 103–111.
[107] Z. Sun, T. Tan, Y. Wang, S.Z. Li, Ordinal palmprint represention for personal
identification [represention read representation], in: 2005 IEEE Computer
Society Conference on Computer Vision and Pattern Recognition (CVPR’05),
1, IEEE, 2005, pp. 279–284.
[108] T. Connie, A.T.B. Jin, M.G.K. Ong, D.N.C. Ling, An automated palmprint
recognition system, Image Vis. Comput. 23 (5) (2005) 501–515.
[109] X. Wu, D. Zhang, K. Wang, Fisherpalms based palmprint recognition, Pattern
Recognit. Lett. 24 (15) (2003) 2829–2838.
[110] G. Lu, D. Zhang, K. Wang, Palmprint recognition using eigenpalms features,
Pattern Recognit. Lett. 24 (9–10) (2003) 1463–1467.
[111] L. Zhang, D. Zhang, Characterization of palmprints by wavelet signatures via
directional context modeling, IEEE Trans. Syst. Man Cybern. B 34 (3) (2004)
1335–1347.
[112] S. Ribaric, I. Fratric, A biometric identification system based on eigenpalm and
eigenfinger features, IEEE Trans. Pattern Anal. Mach. Intell. 27 (11) (2005)
1698–1709.
[113] A. Kumar, D. Zhang, Combining fingerprint, palmprint and hand-shape for
user authentication, in: 18th International Conference on Pattern Recognition
(ICPR’06), 4, IEEE, 2006, pp. 549–552.
[114] J.S. Noh, K.H. Rhee, Palmprint identification algorithm using hu invariant mo-
ments and otsu binarization, in: Fourth Annual ACIS International Conference
on Computer and Information Science (ICIS’05), IEEE, 2005, pp. 94–99.
709
Information Fusion 91 (2023) 694–712
[115] J. You, W.-K. Kong, D. Zhang, K.H. Cheung, On hierarchical palmprint coding
with multiple features for personal identification in large databases, IEEE Trans.
Circuits Syst. Video Technol. 14 (2) (2004) 234–243.
[116] X.-Q. Wu, K.-Q. Wang, D. Zhang, Wavelet based palm print recognition, in:
Proceedings. International Conference on Machine Learning and Cybernetics, 3,
IEEE, 2002, pp. 1253–1257.
[117] A. Kumar, H.C. Shen, Palmprint identification using palmcodes, in: Third
International Conference on Image and Graphics (ICIG’04), IEEE, 2004, pp.
258–261.
[118] D.A. Reynolds, R.C. Rose, Robust text-independent speaker identification using
Gaussian mixture speaker models, IEEE Trans. Speech Audio Process. 3 (1)
(1995) 72–83.
[119] S. Chakroborty, G. Saha, Improved text-independent speaker identification using
fused MFCC & IMFCC feature sets based on Gaussian filter, Int. J. Signal Process.
5 (1) (2009) 11–19.
[120] X. Zhao, Y. Wang, D. Wang, Robust speaker identification in noisy and
reverberant conditions, IEEE/ACM Trans. Audio. Speech. Lang. Process. 22 (4)
(2014) 836–845.
[121] K. Manikandan, E. Chandra, Speaker identification using a novel prosody with
fuzzy based hierarchical decision tree approach, Indian J. Sci. Technol. 9 (44)
(2016).
[122] P. Dhakal, P. Damacharla, A.Y. Javaid, V. Devabhaktuni, A near real-time
automatic speaker recognition architecture for voice-based user interface, Mach.
Learn. Knowl. Extr. 1 (1) (2019) 504–520.
[123] Z.K. Abdul, Kurdish speaker identification based on one dimensional convo-
lutional neural network, Comput. Methods Differ. Equ. 7 (4 (Special Issue))
(2019) 566–572.
[124] A. Indumathi, E. Chandra, Speaker identification using bagging techniques, in:
2015 International Conference on Computers, Communications, and Systems,
ICCCS, IEEE, 2015, pp. 223–229.
[125] Z. Zhang, L. Wang, A. Kai, T. Yamada, W. Li, M. Iwahashi, Deep neural network-
based bottleneck feature and denoising autoencoder-based dereverberation for
distant-talking speaker identification, EURASIP J. Audio Speech Music Process.
2015 (1) (2015) 1–13.
[126] L. Dovydaitis, V.E. Rudžionis, Building LSTM neural network based speaker
identification system, Comput. Sci. Techn. 6 (1) (2018) 574–580.
[127] C. Zhang, K. Koishida, J.H. Hansen, Text-independent speaker verification based
on triplet convolutional neural network embeddings, IEEE/ACM Trans. Audio
Speech Language Process. 26 (9) (2018) 1633–1644.
[128] L. Schomaker, M. Bulacu, Automatic writer identification using connected-
component contours and edge-based features of uppercase western script, IEEE
Trans. Pattern Anal. Mach. Intell. 26 (6) (2004) 787–798.
[129] D. Bertolini, L.S. Oliveira, R. Sabourin, Multi-script writer identification using
dissimilarity, in: 2016 23rd International Conference on Pattern Recognition,
ICPR, IEEE, 2016, pp. 3025–3030.
[130] A. Seropian, M. Grimaldi, N. Vincent, Writer identification based on the fractal
construction of a reference base, in: ICDAR, 2003, pp. 1163–1167.
[131] A. Brink, M. Bulacu, L. Schomaker, How much handwritten text is needed
for text-independent writer verification and identification, in: 2008 19th
International Conference on Pattern Recognition, IEEE, 2008, pp. 1–4.
[132] M. Bulacu, L. Schomaker, Text-independent writer identification and verifica-
tion using textural and allographic features, IEEE Trans. Pattern Anal. Mach.
Intell. 29 (4) (2007) 701–717.
[133] G. Leedham, S. Chachra, Writer identification using innovative binarised
features of handwritten numerals, in: Seventh International Conference on
Document Analysis and Recognition, 2003. Proceedings, IEEE, 2003, pp.
413–416.
[134] V. Pervouchine, G. Leedham, Extraction and analysis of forensic document
examiner features used for writer identification, Pattern Recognit. 40 (3) (2007)
1004–1013.
[135] S. Fiel, R. Sablatnig, Writer identification and retrieval using a convolutional
neural network, in: International Conference on Computer Analysis of Images
and Patterns, Springer, 2015, pp. 26–37.
[136] L. Xing, Y. Qiao, Deepwriter: A multi-stream deep CNN for text-independent
writer identification, in: 2016 15th International Conference on Frontiers in
Handwriting Recognition, ICFHR, IEEE, 2016, pp. 584–589.
[137] R. Nasuno, S. Arai, Writer identification for offline japanese handwritten
character using convolutional neural network, in: Proceedings of the 5th IIAE
(Institute of Industrial Applications Engineers) International Conference on
Intelligent Systems and Image Processing, 2017, pp. 94–97.
[138] V. Christlein, D. Bernecker, F. Hönig, A. Maier, E. Angelopoulou, Writer
identification using GMM supervectors and exemplar-SVMs, Pattern Recognit.
63 (2017) 258–267.
[139] X.-Y. Zhang, G.-S. Xie, C.-L. Liu, Y. Bengio, End-to-end online writer identi-
fication with recurrent neural network, IEEE Trans. Hum.-Mach. Syst. 47 (2)
(2016) 285–292.
Z. Qin et al.
[140] Y. Zhu, Y. Wang, An offline text-independent writer identification system
with sae feature extraction, in: 2016 International Conference on Progress in
Informatics and Computing, PIC, IEEE, 2016, pp. 432–436.
[141] J.-H. Yoo, M.S. Nixon, Automated markerless analysis of human gait motion
for recognition and classification, Etri J. 33 (2) (2011) 259–266.
[142] W. Lu, W. Zong, W. Xing, E. Bao, Gait recognition based on joint distribution
of motion angles, J. Vis. Lang. Comput. 25 (6) (2014) 754–763.
[143] A. Roy, S. Sural, J. Mukherjee, Gait recognition using pose kinematics and pose
energy image, Signal Process. 92 (3) (2012) 780–792.
[144] D.-Y. Huang, T.-W. Lin, W.-C. Hu, C.-H. Cheng, Gait recognition based on gabor
wavelets and modified gait energy image for human identification, J. Electron.
Imaging 22 (4) (2013) 043039.
[145] E.-H. Zhang, H.-B. Ma, J.-W. Lu, Y.-J. Chen, Gait recognition using dynamic gait
energy and PCA+ LPP method, in: 2009 International Conference on Machine
Learning and Cybernetics, 1, IEEE, 2009, pp. 50–53.
[146] C.P. Lee, A.W. Tan, S.C. Tan, Gait recognition with transient binary patterns,
J. Vis. Commun. Image Represent. 33 (2015) 69–77.
[147] S.D. Choudhury, T. Tjahjadi, Silhouette-based gait recognition using procrustes
shape analysis and elliptic Fourier descriptors, Pattern Recognit. 45 (9) (2012)
3414–3426.
[148] M. Zeng, L.T. Nguyen, B. Yu, O.J. Mengshoel, J. Zhu, P. Wu, J. Zhang,
Convolutional neural networks for human activity recognition using mobile
sensors, in: 6th International Conference on Mobile Computing, Applications
and Services, IEEE, 2014, pp. 197–205.
[149] Z. Wu, Y. Huang, L. Wang, X. Wang, T. Tan, A comprehensive study on cross-
view gait based human identification with deep cnns, IEEE Trans. Pattern Anal.
Mach. Intell. 39 (2) (2016) 209–226.
[150] N. Takemura, Y. Makihara, D. Muramatsu, T. Echigo, Y. Yagi, On in-
put/output architectures for convolutional neural network-based cross-view gait
recognition, IEEE Trans. Circuits Syst. Video Technol. 29 (9) (2017) 2708–2719.
[151] F.J. Ordóñez, D. Roggen, Deep convolutional and lstm recurrent neural net-
works for multimodal wearable activity recognition, Sensors 16 (1) (2016)
115.
[152] L. Wu, C. Shen, A. van den Hengel, Convolutional lstm networks for video-based
person re-identification, 1, (11) 2016, arXiv preprint arXiv:1606.01609.
[153] S. Yu, H. Chen, E.B. Garcia Reyes, N. Poh, Gaitgan: Invariant gait feature
extraction using generative adversarial networks, in: Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition Workshops, 2017, pp.
30–37.
[154] O. Tekbas, N. Serinken, O. Ureten, An experimental performance evaluation
of a novel radio-transmitter identification system under diverse environmental
conditions, Can. J. Electr. Comput. Eng. 29 (3) (2004) 203–209.
[155] O. Ureten, N. Serinken, Bayesian detection of wi-fi transmitter RF fingerprints,
Electron. Lett. 41 (6) (2005) 373–374.
[156] K.B. Rasmussen, S. Capkun, Implications of radio fingerprinting on the security
of sensor networks, in: 2007 Third International Conference on Security and
Privacy in Communications Networks and the Workshops-SecureComm 2007,
IEEE, 2007, pp. 331–340.
[157] S. Xu, L. Xu, Z. Xu, B. Huang, Individual radio transmitter identification
based on spurious modulation characteristics of signal envelop, in: MILCOM
2008-2008 IEEE Military Communications Conference, IEEE, 2008, pp. 1–5.
[158] V. Brik, S. Banerjee, M. Gruteser, S. Oh, Wireless device identification with ra-
diometric signatures, in: Proceedings of the 14th ACM International Conference
on Mobile Computing and Networking, 2008, pp. 116–127.
[159] W.C. Suski II, M.A. Temple, M.J. Mendenhall, R.F. Mills, Using spectral finger-
prints to improve wireless network security, in: IEEE GLOBECOM 2008-2008
IEEE Global Telecommunications Conference, IEEE, 2008, pp. 1–5.
[160] D. Chen, N. Zhang, Z. Qin, X. Mao, Z. Qin, X. Shen, X.-Y. Li, S2M: A
lightweight acoustic fingerprints-based wireless device authentication protocol,
IEEE Internet Things J. 4 (1) (2016) 88–100.
[161] C. Lin, J. Hu, Y. Sun, F. Ma, L. Wang, G. Wu, Wiau: An accurate device-free
authentication system with ResNet, in: 2018 15th Annual IEEE International
Conference on Sensing, Communication, and Networking, SECON, IEEE, 2018,
pp. 1–9.
[162] S. Aneja, N. Aneja, M.S. Islam, IoT device fingerprint using deep learning,
in: 2018 IEEE International Conference on Internet of Things and Intelligence
System, IOTAIS, IEEE, 2018, pp. 174–179.
[163] A. Krizhevsky, I. Sutskever, G.E. Hinton, Imagenet classification with deep
convolutional neural networks, Adv. Neural Inf. Process. Syst. 25 (2012).
[164] G. Hinton, L. Deng, D. Yu, G.E. Dahl, A.-r. Mohamed, N. Jaitly, A. Senior, V.
Vanhoucke, P. Nguyen, T.N. Sainath, et al., Deep neural networks for acoustic
modeling in speech recognition: The shared views of four research groups, IEEE
Signal Process. Mag. 29 (6) (2012) 82–97.
[165] G. Trigeorgis, F. Ringeval, R. Brueckner, E. Marchi, M.A. Nicolaou, B. Schuller,
S. Zafeiriou, Adieu features? end-to-end speech emotion recognition using a
deep convolutional recurrent network, in: 2016 IEEE International Confer-
ence on Acoustics, Speech and Signal Processing, ICASSP, IEEE, 2016, pp.
5200–5204.
710
Information Fusion 91 (2023) 694–712
[166] B.P. Yuhas, M.H. Goldstein, T.J. Sejnowski, Integration of acoustic and visual
speech signals using neural networks, IEEE Commun. Mag. 27 (11) (1989)
65–71.
[167] S. Planet, I. Iriondo, Comparison between decision-level and feature-level fusion
of acoustic and linguistic features for spontaneous emotion recognition, in: 7th
Iberian Conference on Information Systems and Technologies (CISTI 2012),
IEEE, 2012, pp. 1–6.
[168] S. Awang, R. Yusof, M.F. Zamzuri, R. Arfa, Feature level fusion of face and
signature using a modified feature selection technique, in: 2013 International
Conference on Signal-Image Technology & Internet-Based Systems, IEEE, 2013,
pp. 706–713.
[169] N. Poonguzhali, M. Ezhilarasan, A framework for level-1 and level-2 feature
level fusion, in: 2018 Ieee International Conference on System, Computation,
Automation and Networking (Icscan), IEEE, 2018, pp. 1–6.
[170] A. Abozaid, A. Haggag, H. Kasban, M. Eltokhy, Multimodal biometric scheme
for human authentication technique based on voice and face recognition fusion,
Multimedia Tools Appl. 78 (12) (2019) 16345–16361.
[171] B. Zhou, Z. Xie, F. Ye, Multi-modal face authentication using deep visual
and acoustic features, in: ICC 2019-2019 IEEE International Conference on
Communications, ICC, IEEE, 2019, pp. 1–6.
[172] T. Kumar, S. Bhushan, S. Jangra, Ann trained and WOA optimized feature-level
fusion of iris and fingerprint, Mater. Today: Proceedings 51 (2022) 1–11.
[173] G. Jaswal, R.C. Poonia, Selection of optimized features for fusion of palm print
and finger knuckle-based person authentication, Expert Syst. 38 (2021).
[174] A. Attia, Z. Akhtar, Y. Chahir, Feature-level fusion of major and minor dorsal
finger knuckle patterns for person authentication, Signal, Image Video Process.
15 (4) (2021) 851–859.
[175] G.U. Bokade, R.D. Kanphade, Secure multimodal biometric authentication
using face, palmprint and ear: a feature level fusion approach, in: 2019
10th International Conference on Computing, Communication and Networking
Technologies, ICCCNT, IEEE, 2019, pp. 1–5.
[176] D.-S. Kim, K.-S. Hong, Multimodal biometric authentication using teeth image
and voice in mobile environment, IEEE Trans. Consum. Electron. 54 (4) (2008)
1790–1797.
[177] X. Yan, F. Deng, W. Kang, Palm vein recognition based on multi-algorithm and
score-level fusion, in: 2014 Seventh International Symposium on Computational
Intelligence and Design, 1, IEEE, 2014, pp. 441–444.
[178] C. Dalila, H. Imane, N.-A. Amine, Multimodal score-level fusion using hybrid
ga-pso for multibiometric system, Informatica 39 (2) (2015).
[179] K. Aizi, M. Ouslim, Score level fusion in multi-biometric identification based
on zones of interest, J. King Saud University-Comput. Inf. Sci. 34 (1) (2022)
1498–1509.
[180] C. Li, J. Hu, J. Pieprzyk, W. Susilo, A new biocryptosystem-oriented security
analysis framework and implementation of multibiometric cryptosystems based
on decision level fusion, IEEE Trans. Inf. Forensics Secur. 10 (6) (2015)
1193–1206.
[181] A.K. Naik, R.S. Holambe, Joint encryption and compression scheme for a
multimodal telebiometric system, Neurocomputing 191 (2016) 69–81.
[182] D.R. Devi, K.N. Rao, Decision level fusion schemes for a multimodal biometric
system using local and global wavelet features, in: 2020 IEEE Interna-
tional Conference on Electronics, Computing and Communication Technologies,
CONECCT, IEEE, 2020, pp. 1–6.
[183] M. Sandhya, Y. Sreenivasa Rao, S. Biswajeet, V. Dilip Kumar, M. Anup Kumar,
A score-level fusion method for protecting fingerprint and palmprint templates,
in: Security and Privacy, Springer, 2021, pp. 1–11.
[184] A. Kumar, S. Shekhar, Personal identification using multibiometrics rank-level
fusion, IEEE Trans. Syst. Man Cybern. C 41 (5) (2010) 743–752.
[185] A. Tahmasebi, H. Pourghasem, H. Mahdavi-Nasab, A novel rank-level fusion for
multispectral palmprint identification system, in: 2011 International Conference
on Intelligent Computation and Bio-Medical Instrumentation, IEEE, 2011, pp.
208–211.
[186] M.W. Rahman, F.T. Zohra, M.L. Gavrilova, Rank level fusion for kinect gait and
face biometrie identification, in: 2017 IEEE Symposium Series on Computational
Intelligence, SSCI, IEEE, 2017, pp. 1–7.
[187] S. Ben Jemaa, M. Hammami, H. Ben-Abdallah, Finger surfaces recognition using
rank level fusion, Comput. J. 60 (7) (2017) 969–985.
[188] S.N. Tumpa, M.L. Gavrilova, Score and rank level fusion algorithms for social
behavioral biometrics, IEEE Access 8 (2020) 157663–157675.
[189] M.V. Kumar, R. Srikantaswamy, Comparative analysis of distinct fusion levels
in multimodal biometrics, Int. J. Comput. Appl. 4 (2015) 1–4.
[190] J.K. Sing, A. Dey, M. Ghosh, Confidence factor weighted Gaussian function
induced parallel fuzzy rank-level fusion for inference and its application to
face recognition, Inf. Fusion 47 (2019) 60–71.
[191] R. Sharma, S. Das, P. Joshi, Rank level fusion in multibiometric systems, in:
2015 Fifth National Conference on Computer Vision, Pattern Recognition, Image
Processing and Graphics, NCVPRIPG, IEEE, 2015, pp. 1–4.
Z. Qin et al.
[192] D.R. Devi, K.N. Rao, A multimodal biometric system using partition based dwt
and rank level fusion, in: 2016 IEEE International Conference on Computational
Intelligence and Computing Research, ICCIC, IEEE, 2016, pp. 1–5.
[193] S. Chaudhary, R. Nath, A multimodal biometric recognition system based on
fusion of palmprint, fingerprint and face, in: 2009 International Conference
on Advances in Recent Technologies in Communication and Computing, IEEE,
2009, pp. 596–600.
[194] T. Baltrušaitis, C. Ahuja, L.-P. Morency, Multimodal machine learning: A survey
and taxonomy, IEEE Trans. Pattern Anal. Mach. Intell. 41 (2) (2018) 423–443.
[195] G.E. Hinton, R.R. Salakhutdinov, Reducing the dimensionality of data with
neural networks, Science 313 (5786) (2006) 504–507.
[196] H.P. Martínez, G.N. Yannakakis, Deep multimodal fusion: Combining discrete
events and continuous signals, in: Proceedings of the 16th International
Conference on Multimodal Interaction, 2014, pp. 34–41.
[197] L. Yuan, Z.-C. Mu, X.-N. Xu, Multimodal recognition based on face and ear, in:
2007 International Conference on Wavelet Analysis and Pattern Recognition, 3,
IEEE, 2007, pp. 1203–1207.
[198] M.M. Monwar, M.L. Gavrilova, Multimodal biometric system using rank-level
fusion approach, IEEE Trans. Syst. Man Cybern. B 39 (4) (2009) 867–878.
[199] S. Ribaric, I. Fratric, K. Kis, A biometric verification system based on the
fusion of palmprint and face features, in: ISPA 2005. Proceedings of the 4th
International Symposium on Image and Signal Processing and Analysis, 2005,
IEEE, 2005, pp. 12–17.
[200] S. Hariprasath, T. Prabakar, Multimodal biometric recognition using iris fea-
ture extraction and palmprint features, in: IEEE-International Conference on
Advances in Engineering, Science and Management (ICAESM-2012), IEEE, 2012,
pp. 174–179.
[201] C. Galdi, M. Nappi, J.-L. Dugelay, Multimodal authentication on smartphones:
Combining iris and sensor recognition for a double check of user identity,
Pattern Recognit. Lett. 82 (2016) 144–153.
[202] Q. Li, Z. Qiu, D. Sun, Feature-level fusion of hand biometrics for personal
verification based on kernel PCA, in: International Conference on Biometrics,
Springer, 2006, pp. 744–750.
[203] A. Meraoumia, S. Chitroub, A. Bouridane, Fusion of finger-knuckle-print and
palmprint for an efficient multi-biometric system of person recognition, in: 2011
IEEE International Conference on Communications, ICC, IEEE, 2011, pp. 1–5.
[204] A. Attia, S. Mazaa, Z. Akhtar, Y. Chahir, Deep learning-driven palmprint and
finger knuckle pattern-based multimodal person recognition system, Multimedia
Tools Appl. 81 (8) (2022) 10961–10980.
[205] R. Kala, H. Vazirani, A. Shukla, R. Tiwari, Fusion of speech and face by
enhanced modular neural network, in: International Conference on Information
Systems, Technology and Management, Springer, 2010, pp. 363–372.
[206] J. Ren, Y. Hu, Y.-W. Tai, C. Wang, L. Xu, W. Sun, Q. Yan, Look, listen and
learn—A multimodal LSTM for speaker identification, in: Proceedings of the
AAAI Conference on Artificial Intelligence, AAAI ’16, AAAI Press, 2016, pp.
3581–3587.
[207] X. Liu, J. Geng, H. Ling, Y.-m. Cheung, Attention guided deep audio-face fusion
for efficient speaker naming, Pattern Recognit. 88 (2019) 557–568.
[208] M. Sahidullah, D.A.L. Thomsen, R.G. Hautamäki, T. Kinnunen, Z.-H. Tan, R.
Parts, M. Pitkänen, Robust voice liveness detection and speaker verification
using throat microphones, IEEE/ACM Trans. Audio Speech Lang. Process. 26
(1) (2017) 44–56.
[209] L. Wu, J. Yang, M. Zhou, Y. Chen, Q. Wang, LVID: A multimodal biometrics
authentication system on smartphones, IEEE Trans. Inf. Forensics Secur. 15
(2019) 1572–1585.
[210] R.W. Frischholz, U. Dieckmann, BiolD: a multimodal biometric identification
system, Computer 33 (2) (2000) 64–68.
[211] S. Shon, J.R. Glass, Multimodal association for speaker verification, in:
INTERSPEECH, 2020, pp. 2247–2251.
[212] S. Nawaz, M.S. Saeed, P. Morerio, A. Mahmood, I. Gallo, M.H. Yousaf, A.
Del Bue, Cross-modal speaker verification and recognition: A multilingual
perspective, in: Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, 2021, pp. 1682–1691.
[213] Y. Dong, Y.-D. Yao, Secure mmwave-radar-based speaker verification for IoT
smart home, IEEE Internet Things J. 8 (5) (2020) 3500–3511.
[214] A. Rahman, M.E. Chowdhury, A. Khandakar, S. Kiranyaz, K.S. Zaman, M.B.I.
Reaz, M.T. Islam, M. Ezeddin, M.A. Kadir, Multimodal EEG and keystroke
dynamics based biometric system using machine learning algorithms, IEEE
Access 9 (2021) 94625–94643.
[215] K.-W. Tse, K. Hung, Behavioral biometrics scheme with keystroke and swipe
dynamics for user authentication on mobile platform, in: 2019 IEEE 9th
Symposium on Computer Applications & Industrial Electronics, ISCAIE, IEEE,
2019, pp. 125–130.
[216] S. Vhaduri, C. Poellabauer, Multi-modal biometric-based implicit authentication
of wearable device users, IEEE Trans. Inf. Forensics Secur. 14 (12) (2019)
3116–3125.
711
Information Fusion 91 (2023) 694–712
[217] W. Cheung, S. Vhaduri, Context-dependent implicit authentication for wearable
device users, in: 2020 IEEE 31st Annual International Symposium on Personal,
Indoor and Mobile Radio Communications, IEEE, 2020, pp. 1–7.
[218] T. Dee, I. Richardson, A. Tyagi, Continuous transparent mobile device touch-
screen soft keyboard biometric authentication, in: 2019 32nd International
Conference on VLSI Design and 2019 18th International Conference on
Embedded Systems, VLSID, IEEE, 2019, pp. 539–540.
[219] L. Xie, H. Xian, X. Tang, W. Guo, F. Hang, N. Fang, G-key: An authentica-
tion technique for mobile devices based on gravity sensors, in: 2019 IEEE
International Conference on Power Data Science, ICPDS, IEEE, 2019, pp.
126–129.
[220] Z. Qin, G. Huang, H. Xiong, Z. Qin, K.-K.R. Choo, A fuzzy authentication system
based on neural network learning and extreme value statistics, IEEE Trans.
Fuzzy Syst. 29 (3) (2019) 549–559.
[221] D. Frolova, A. Epishkina, K. Kogos, Mobile user authentication using keystroke
dynamics, in: 2019 European Intelligence and Security Informatics Conference,
EISIC, IEEE, 2019, p. 140.
[222] S.K. Behera, P. Kumar, D.P. Dogra, P.P. Roy, A robust biometric authentication
system for handheld electronic devices by intelligently combining 3D finger
motions and cerebral responses, IEEE Trans. Consum. Electron. 67 (1) (2021)
58–67.
[223] X. Yang, S. Yang, J. Liu, C. Wang, Y. Chen, N. Saxena, Enabling finger-touch-
based mobile user authentication via physical vibrations on IoT devices, IEEE
Trans. Mob. Comput. (2021).
[224] Y. Sun, J. Cao, M. Ma, H. Li, B. Niu, F. Li, Privacy-preserving device discovery
and authentication scheme for D2D communication in 3GPP 5G HetNet, in:
2019 International Conference on Computing, Networking and Communications,
ICNC, IEEE, 2019, pp. 425–431.
[225] D.K. Sharma, N. Baghel, S. Agarwal, Multiple degree authentication in sensible
homes basedon iot device vulnerability, in: 2020 International Conference on
Power Electronics & IoT Applications in Renewable Energy and Its Control,
PARC, IEEE, 2020, pp. 539–543.
[226] P. Gope, B. Sikdar, Lightweight and privacy-preserving two-factor authen-
tication scheme for IoT devices, IEEE Internet Things J. 6 (1) (2018)
580–589.
[227] Z. Zhang, X. Xu, S. Han, Y. Liang, C. Liu, Wearable proxy device-assisted
authentication request filtering for implantable medical devices, in: 2020 IEEE
Wireless Communications and Networking Conference, WCNC, IEEE, 2020, pp.
1–6.
[228] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, R.
Fergus, Intriguing properties of neural networks, 2013, arXiv preprint arXiv:
1312.6199.
[229] N. Akhtar, J. Liu, A. Mian, Defense against universal adversarial perturbations,
in: Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition, 2018, pp. 3389–3398.
[230] A. Nguyen, J. Yosinski, J. Clune, Deep neural networks are easily fooled: High
confidence predictions for unrecognizable images, in: Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition, 2015, pp. 427–436.
[231] R. Feinman, R.R. Curtin, S. Shintre, A.B. Gardner, Detecting adversarial samples
from artifacts, 2017, arXiv preprint arXiv:1703.00410.
[232] N. Akhtar, A. Mian, Threat of adversarial attacks on deep learning in computer
vision: A survey, Ieee Access 6 (2018) 14410–14430.
[233] S. Sabour, Y. Cao, F. Faghri, D.J. Fleet, Adversarial manipulation of deep
representations, 2015, arXiv preprint arXiv:1511.05122.
[234] I.J. Goodfellow, J. Shlens, C. Szegedy, Explaining and harnessing adversarial
examples, 2014, arXiv preprint arXiv:1412.6572.
[235] B. Poudel, W. Li, Black-box adversarial attacks on network-wide multi-
step traffic state prediction models, in: 2021 IEEE International Intelligent
Transportation Systems Conference, ITSC, IEEE, 2021, pp. 3652–3658.
[236] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, A. Vladu, Towards deep learning
models resistant to adversarial attacks, 2017, arXiv preprint arXiv:1706.06083.
[237] F. Croce, M. Hein, Reliable evaluation of adversarial robustness with an
ensemble of diverse parameter-free attacks, in: International Conference on
Machine Learning, PMLR, 2020, pp. 2206–2216.
[238] S.-M. Moosavi-Dezfooli, A. Fawzi, P. Frossard, Deepfool: a simple and accurate
method to fool deep neural networks, in: Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition, 2016, pp. 2574–2582.
[239] S.-M. Moosavi-Dezfooli, A. Fawzi, O. Fawzi, P. Frossard, Universal adversarial
perturbations, in: Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, 2017, pp. 1765–1773.
[240] J. Su, D.V. Vargas, K. Sakurai, One pixel attack for fooling deep neural
networks, IEEE Trans. Evol. Comput. 23 (5) (2019) 828–841.
[241] X.-C. Li, X.-Y. Zhang, F. Yin, C.-L. Liu, Decision-based adversarial attack with
frequency mixup, IEEE Trans. Inf. Forensics Secur. 17 (2022) 1038–1052.
[242] A.E. Cinà, A. Torcinovich, M. Pelillo, A black-box adversarial attack for
poisoning clustering, Pattern Recognit. 122 (2022) 108306.
Z. Qin et al.
[243] S. Chen, Z. He, C. Sun, J. Yang, X. Huang, Universal adversarial attack on
attention and the resulting dataset damagenet, IEEE Trans. Pattern Anal. Mach.
Intell. (2020).
[244] N. Carlini, D. Wagner, Towards evaluating the robustness of neural networks,
in: 2017 Ieee Symposium on Security and Privacy (Sp), IEEE, 2017, pp. 39–57.
[245] S. Sharmin, P. Panda, S.S. Sarwar, C. Lee, W. Ponghiran, K. Roy, A compre-
hensive analysis on adversarial robustness of spiking neural networks, in: 2019
International Joint Conference on Neural Networks, IJCNN, IEEE, 2019, pp.
1–8.
[246] N. Papernot, P. McDaniel, X. Wu, S. Jha, A. Swami, Distillation as a defense
to adversarial perturbations against deep neural networks, in: 2016 IEEE
Symposium on Security and Privacy, SP, IEEE, 2016, pp. 582–597.
[247] D. Hendrycks, M. Mazeika, S. Kadavath, D. Song, Using self-supervised learning
can improve model robustness and uncertainty, Adv. Neural Inf. Process. Syst.
32 (2019).
[248] M. Juuti, S. Szyller, S. Marchal, N. Asokan, PRADA: protecting against DNN
model stealing attacks, in: 2019 IEEE European Symposium on Security and
Privacy (EuroS&P), IEEE, 2019, pp. 512–527.
712
Information Fusion 91 (2023) 694–712
[249] R. Shao, P. Perera, P.C. Yuen, V.M. Patel, Open-set adversarial defense
with clean-adversarial mutual learning, Int. J. Comput. Vis. 130 (4) (2022)
1070–1087.
[250] J. Chen, X. Lin, H. Xiong, Y. Wu, H. Zheng, Q. Xuan, Smoothing adversarial
training for gnn, IEEE Trans. Comput. Soc. Syst. 8 (3) (2020) 618–629.
[251] J. Yuan, Z. He, Adversarial dual network learning with randomized image
transform for restoring attacked images, IEEE Access 8 (2020) 22617–22624.
[252] F. Sheikholeslami, S. Jain, G.B. Giannakis, Efficient randomized defense against
adversarial attacks in deep convolutional neural networks, in: ICASSP 2019-
2019 IEEE International Conference on Acoustics, Speech and Signal Processing,
ICASSP, IEEE, 2019, pp. 3277–3281.
[253] E. Quiring, D. Arp, K. Rieck, Forgotten siblings: Unifying attacks on machine
learning and digital watermarking, in: 2018 IEEE European Symposium on
Security and Privacy (EuroS&P), IEEE, 2018, pp. 488–502.
[254] Y. Wang, D. Zou, J. Yi, J. Bailey, X. Ma, Q. Gu, Improving adversarial robustness
requires revisiting misclassified examples, in: International Conference on
Learning Representations, 2019.