2018 24th International Conference on Pattern Recognition (ICPR)
Beijing, China, August 20-24, 2018
Air Signature Recognition using Deep
Convolutional Neural Network-Based Sequential
Model
S. K. Behera, A. K. Dash and D. P. Dogra
School of Electrical Sciences
Indian Institute of Technology
Bhubaneswar 752050, India
Email: {sb29, akd10, dpdogra}@iitbbs.ac.in
Abstract—Deep convolutional neural networks are becoming
extremely popular in classification, especially when the inputs are
non-sequential in nature. Though it seems unrealistic to adopt
such networks as sequential classifiers, however, researchers have
started to use them for applications that primarily deal with
sequential data. It is possible, if the sequential data can be
represented in the conventional way the inputs are provided in
CNNs. Signature recognition is one of the important tasks for
biometric applications. Signatures represent the signer’s identity.
Air signatures can make traditional biometric systems more
secure and robust than conventional pen-paper or stylus guided
interfaces. In this paper, we propose a new set of geometrical
features to represent 3D air signatures captured using Leap
motion sensor. The features are then arranged such that they can
be fed to a deep convolutional neural network architecture with
application specific tuning of the model parameters. It has been
observed that the proposed features in combination with the CNN
architecture can act as a good sequential classifier when tested on
a moderate size air signature dataset. Experimental results reveal
that the proposed biometric system performs better as compared
to the state-of-the-art geometrical features with average accuracy
improvement of 4%.
I. I NTRODUCTION
In biometric authentication systems, “Signature” recognition
is one of the important modes of identification [1], [2]. Due
to the specific behavioral property, every person’s signature
bears uniqueness. Usability is one of the main advantages of
signature-based biometric authentication systems. The hand-
written inputs, which can be collected easily on papers or
through electronic devices, are therefore quite popular in
authentication [3], [4]. However, due to security reasons, it
is always desirable to enhance the robustness of traditional
signature acquisition systems by other means. This has led
to new ways of signature acquisition systems including sty-
lus/smart pen based interface or 3D air signature setups. In
3D air signature setups, people are encourage to put their
signatures in air and the inputs are captured through sensors
built with visible light camera, IR camera, or depth measuring
devices. Specifically, air signature recognition in 3D has two
advantages over traditional 2D acquisition systems. They are
robust to shouder surfing and 3D features are stronger due to
978-1-5386-3788-3/18/$31.00 ©2018 IEEE
Authorized licensed use limited to: UNIVERSITE DE SFAX. Downloaded on July 10,2023 at 15:58:54 UTC from IEEE Xplore. Restrictions apply.
P. P. Roy
Department Computer Science & Engineering
Indian Institute of Technology
Roorkee 247667, India
Email: proy.fcs@iitr.ac.in
the presence of the third dimension in the input data [5], [6],
[7].
In offline mode [4], people use scanner for digitization, then
analyze shapes and contours of the signatures. The online or
dynamic signatures [3], use a digitizing tablet (pen-sensitive
computer display) where the users sign by a smart pen or
stylus. The sensors like Leap motion, Kinet, or RealSense are
also being used to capture air signatures in 3D. Such setups
are portable and can be integrated with modern handheld
devices easily. In all of the above acquisition systems, features
such as position in x-axis, y-axis, z-axis, slope, direction,
curvature,velocity, acceleration, pressure, azimuth angle or
altitude angles of the pen with the device, are extracted
from the stored data for recognition purpose [8], [9]. In
the next stage, sequential classifiers are trained using the
extracted features to make appropriate models that are used
during recognition [2], [10]. In dynamic time warping (DTW)-
based approach, signatures are directly matched using elastic
distance [11]. In case of Hidden Markov Model (HMM), finite
state machines are used to generate a probabilistic model
that is used during matching [12]. The matching techniques
based on DTW [11], Minimal Variance Matching (MVM) [13],
Support vector machine (SVM) [14], and Neural Networks
(NN) [15] are commonly used for signature recognition.
Apart from the above classifiers, deep learning frameworks
are becoming extremely popular as baseline classifiers at a
faster rate. Especially, deep neural networks are often being
used for learning models that aim to model high-level ab-
stractions in data [16], [17], [18]. Due to multiple layers,
deep learning architecture is build for automating feature
learning. Therefore, through the deep learning framework,
data are represented by a hierarchy of features from lower-
level to higher-level. Even such architectures that are suitable
for representing 2 or higher dimentional data like image or
video, are being used for sequential classification tasks [19],
[20], [21]. For example, the human activity recognition (HAR)
proposed by Yang et al. [22] has successfully utilized such a
deep learning framework on multi-channel time series signals
acquired from a set of body-worn inertial sensors that capture
3525
 Fig. 1. Proposed methodology of air signature recognition system.

predefined human activities. Therefore, we have intuitively

adopted such a deep convolutional neural network architecture
for classification of 3D air signatures.
However, selection of appropriate feature-set is highly im-
portant to make the recognition perfect. Therefore, researchers
have proposed various types of features that preserve static [6],
[23] as well as dynamic [24] characteristics of human signa-
tures. It has been observed that, geometrical characteristics
can be quite handy to define the uniqueness of signatures.
In this paper, we have introduced a new geometrical feature-
set that is capable of preserving the uniqueness of human
signatures. The features are then arranged such that they can
be fed to a deep convolutional neural network architecture with
application specific tuning of the model parameters. It has been
observed that the proposed features in combination with the
Fig. 2. A typical experimental setup using Leap motion interface that has
CNN architecture can act as a good sequential classifier. This been used for air signature acquisition during biometric authentication.
has been done with a primary goal to design a robust biometric
authentication system that is better than existing systems in
terms of accuracy and ease of implementation. Fig. 1 presents the volunteers involved in this study. The Leap motion sensor
the overview of the proposed air signature recognition system. is equipped with infra red camera that creates an inverted
In the quest of accomplishing the above goal, we have made pyramid field of view over the surface of the sensor. Therefore,
the following technical contributions in this field of study: when the palm is placed within this interaction space, its in-
• We have proposed a new geometrical feature-set that built API can track finger tip movements during air signatures.
accounts for the preservation of internal structure with During signature registration, the users are asked to sign using
the help of 3D orientations and normalized distances of their index finger as typically we do while performing air
the boundary points from the center of the signature. signatures over the interaction space. Instantaneous finger tip
• We introduce the application of deep convolutional neural positions are obtained and the 3D trajectories are assumed as
network as a sequential classifiers to recognize 3D air the signature of the intended user. In the pre-processing stage,
signatures by arranging the geometrical features. coordinate values are scaled between 0 to 1 and then re-sized
• An exhaustive experimental evaluation of the proposed to a fixed length using the process as depicted in Fig 3.
geometrical feature-set and the customized architecture
in the context of 3D air signature recognition.
The rest of the paper is organized as follows. In the next
section, we present the feature extraction steps. The architec-
ture of the deep CNN as a sequence classifier applicable to
signature recognition, has been discussed in Section III. Ex-
perimental results and comparative evaluations are presented
in Section IV. Finally, conclusions are presented in Section V.
Fig. 3. Pre-processing of the raw signature data.
II. E XTRACTION OF G EOMETRICAL F EATURES
A Leap motion-based 3D signature acquisition setup as A typical signature with N sequential points represented
depicted in Fig. 2 has been used for collecting air signatures of by S as given in (1), where pi = (xi , yi , zi ) represents the

3526

Authorized licensed use limited to: UNIVERSITE DE SFAX. Downloaded on July 10,2023 at 15:58:54 UTC from IEEE Xplore. Restrictions apply.
instantaneous position of the finger tip at ith instance.
S = [p1 , p2 , p3 , ......, pN ]T
A. Geometrical Features
Though raw coordinates can be used for training, how-
ever, often these values are corrupted with noises such as
inaccurate tracking or perspective errors. Therefore, high-level
features are often preferred instead of the raw coordinates.
High-level feature such as local direction, curvature, skew,
lineness, velocity or acceleration are often used to extract
meaningful information about the curvy signatures [6], [23],
[24]. Geometrical features based on well-known structures
such as convex hulls or polygonal structures can also be
extracted to represent signature in high dimension space. In
this work, we have emphasized on designing a feature-set that
is suitable for 3D air signatures. We have intuitively developed
this feature assuming as if someone is positioned at the center
of the 3D structure. Viewing perspective may provide valuable
information. The concept is depicted in Fig. 4.
Fig. 4. A pictorial depiction of perspective parameter estimation and geo-
metrical feature extraction.
Our method starts by locating the centroid (c) of the 3D
signature and then estimating distance and angular parameters
over the entire curve. To avoid data loss, we estimate the local
features on each point (pi ) of the signature curve (S). First,
we draw a straight line from c to pi . The normalized distance
(di ) between these points is recorded as depicted in Fig. 4. The
normalized distance is assumed to be unique for the same user
due to its rotational invariance property. Next, we compute the
direction cosigns of this line segment across three coordinate
axes. Let the line segment cp ~ i makes ∠α, ∠β, and ∠γ with x-
axis, y-axis, and z-axis, respectively. We estimate the direction
cosines of the line and let the triplet be represented as given
in (2), where li , mi , and ni represent the values of direction
cosines.
< li , mi , ni >=< cos α, cos β, cos γ >
Authorized licensed use limited to: UNIVERSITE DE SFAX. Downloaded on July 10,2023 at 15:58:54 UTC from IEEE Xplore. Restrictions apply.
We then estimate the change in direction cosines between
successive points on the signature curve. It is done as follows.
(1) Let pi and pi+1 represent two consecutive points on the curve
such that
pi = (xi , yi , zi )
and
pi+1 = (xi+1 , yi+1 , zi+1 ).
The change in direction cosines across each axis are computed
and represented as follows:
δli = li − li+1
δmi = mi − mi+1
δni = ni − ni+1 .
Next, we construct a four dimensional high-level feature
representation of the signature at ith position as given in (3),
where δ values are defined earlier.
ξi = [di , δli , δmi , δni ] (3)
Finally, we obtain the complete high-level feature repre-
sentation of the signature as given in (4). This is then used
to prepare the input for deep convolutional neural networks
which is discussed in the following section.
FS = [ξ1 , ξ2 , ξ3 .......ξN ] (4)
III. S EQUENTIAL M ODEL OF CNN FOR S IGNATURE
R ECOGNITION
This section presents the proposed methodology for recog-
nition of 3D air signatures captured using Leap motion sensor
and represented using (4). We call such representation as high-
level features. These feature vectors have been passed through
the sequential models of a deep convolutional neural network
for recognition. The proposed architecture of the network is
described hereafter.
A. Convolutional Neural Network
The traditional architecture of the Convolutional Neural
Network (CNN) usually consists of two major components.
First part is meant for self-learning the features from the raw
data known as the feature extractor stage. The second part
is for performing the classification through a fully connected
multilayer perceptron (MLP). The feature extractor stage is
a collection of a number of stages of similar layer structure,
where each stage consists of a combination of two cascaded-
layers: convolution and activation layer followed by a pooling
layer. However, such networks are mainly suitable for process-
ing image data. Therefore, representation of sequential data in
appropriate form is extremely important. This is discussed in
(2) next section.
3527
 Fig. 5. Architecture view of the signature classification based on deep convolutional neural networks.
1) Adopted Architecture: Unlike the image classifications
tasks using CNN where the input is given in the form of 2D
image pixels, the input for this work is one dimensional. These
are collection of high-level points points extracted from the 3D
air signatures. In order to fit into the architecture of the CNN
model mentioned in Fig. 5, the input data representing the 3D
air signature has been restructured and the subsequent stages
are described below.
Input: The inputs are represented by x ∈ Rn×1×m with n
as the number of samples and m represents the number of
data corresponding to each sample.
Convolution Layer: This step is to perform the convolution
operation between the input received from the previous layer
and the kernel (filter). Generally, multiple kernels are consid-
ered to be used in a convolution layer. Lets, there be ni kernels
used in a convolution layer Ci and we represent the kernel by
κ, then the output of the ith convolution layer with the input
xi is given in (5), where ⊗ is the convolution operation and
bin is the bias.
xi+1 = κn ⊗ x + i
bin ; with n = n1 , n2 , . . . , ni
n
Activation Layer: In order to bring in the concept of non-
linearity inside the neural network, activation functions are
used. The activation functions help to learn complex models.
The most widely used activation functions are sigmoid, tanh
and ReLU. In this paper, ReLU activation has been used across
all layers. The ReLU activation is defined by
ReLU (x) = max(0, x).
Authorized licensed use limited to: UNIVERSITE DE SFAX. Downloaded on July 10,2023 at 15:58:54 UTC from IEEE Xplore. Restrictions apply.
Pooling Layer: Pooling is most widely used to reduce the
spatial size of the output of convolution layer without affecting
its depth. The purpose of pooling layer is to reduce the number
of parameters and hence the computation load is reduced. Max
pooling and average pooling are two of the most widely used
pooling techniques from which, max pooling has been adopted
in this paper.
Flattening and Fully Connected Layer: After a series of
convolution, activation and pooling layers, the feature extractor
stage is completed. The next stage is to perform the classifi-
cation using an fully connected layer. The output of the last
pooling layer is converted in to a vector which is given as
an input to the fully connected layer. The output of the fully
connected layer are generally converted into probabilities and
the popular way to achieve this is through softmax function.
IV. R ESULTS AND D ISCUSSIONS
A. Data Description
A signature dataset has been created for conducting the
(5) experiments and to verify the proposed methodology. The
dataset has been made available to the research community1 . In
our study, 50 users were involved for recording air signatures
using the above mentioned experimental setup, where each
user registered their signature 14 times out of which 12
samples were used for training and rest for testing. Therefore,
a total of 600 signatures (50 X 12 = 600) were used for
training and 100 signature (50 X 2 = 100) were used for
1 https://goo.gl/f9Cao5
3528
 TABLE I proposed feature-set. We have recorded better performance us-
ACCURACY RESULTS USING THE PROPOSED AND EXISTING FEATURES ing the proposed deep learning framework using the proposed
Process Feature-Set Accuracy (%) high-level feature-set as compared to some of the popular
Existing feature 93 classifiers. These comparative results are presented in Table II.
Validation Proposed feature 94
( Existing + Proposed ) feature 98
Existing feature 91
Testing Proposed feature 92
( Existing + Proposed ) feature 95

testing. One of the key objectives of our method is to test

whether less number of signature samples can actually work
in a CNN framework. We were surprised to get exciting
results even with such less number of samples/class. Actually,
this is a positive aspect of our design since registering more
signatures/user is not a practical approach. A user may not
be comfortable to provide more number of signature samples.
However, typical CNN architecture requires 1000s of samples
per class. Therefore, building a network with substantially
less number of samples is a big achievement of the present
research. In the next section, we present the results obtained
using the above dataset.

B. Signature Recognition Results

The TensorFlow framework of deep learning has been used
for implementing the CNN model as described in Section III.
In the training process, we have used 128×1×4 (with sequence Fig. 6. Accuracy variation during training with the increase in number of
length = 128 and 4 number of high-level features) in the first iterations.
layer. In the output layer, we have obtained 50 class probability
values.
In between the input and output layers, 4 groups of layers
including convolution, activation and pooling have been used
alternatively. The number of kernels (filters) have been used
corresponding to the 4 convolution layers are n1 = 32, n2 =
64, n3 = 128 and n4 = 256, respectively. The max-pooling
operations of 4 pooling layers reduce the size of the inputs
from 128 to 8 without changing the depth. Thus, it reduces
the input size immediately before the fully connected layer to
8 × 256. We have obtained 94% validation accuracy during
training process. While testing with the remaining data, we
have recorded accuracy of 92%. Summary of the results is
presented in Table I.
Apart from the above results, we have also experimented
with a few popular existing high-level features such as writing
direction or curvature at each point of the signature. These
features have been proposed by Jaeger et al. [23] and Behera
et al. [6]. We have obtained 93% (validation) and 91% (testing)
accuracy, respectively. Finally, we have applied the proposed
feature-set combined with the existing features. In this setting,
we have recorded significantly better performance (98%) in the
validation while training and (95%) in testing. These results Fig. 7. Loss values during training with the increase in number of iterations.
are described in Table I.
Figs. 6 and 7 show how the accuracy and loss rates vary
with the increase in number of iterations during training, both C. Results by Varying Number of Samples
using the proposed as well as existing features. Also, we have We have also conducted experiments by varying number of
tested using different classifiers to establish the robustness of signatures/user in the training phase. Fig. 8 suggests that both

3529

Authorized licensed use limited to: UNIVERSITE DE SFAX. Downloaded on July 10,2023 at 15:58:54 UTC from IEEE Xplore. Restrictions apply.
 TABLE II
T EST ACCURACY RESULTS USING DIFFERENT CLASSIFIERS
Process Feature-Set Accuracy (%)
HMM LSTM CNN
Existing feature 90 88 91
Testing Proposed feature 91 90 92
( Existing + Proposed ) feature 93 92 95
validation and testing accuracy increase with the increase in
number of samples/user. Therefore, we can assume that using
proposed work, performance can be improved by increasing
number of signatures/user in the training process. However,
it may also be observed that, beyond 10 samples the rate of
increase in accuracy is not significant. Therefore, our goal
of keeping less number of samples (due to impracticality of
collective more samples/user) is fitting into the present context.
Fig. 8. Accuracy by varying the number of signatures/user.
V. C ONCLUSION
The paper proposes a new high-level feature-set for air
signature recognition captured using Leap motion sensor. The
features are designed with the help of geometrical analysis
of the curved signature structures. We have used a CNN
based sequential model for recognition of the signatures.
Experimental results reveal that, using the proposed feature-set
on the adopted deep CNN architecture, Performs of signature
recognition has improved. However, we have observed that
samples/user may not be as high as it is required in typical
neural network setups. The rate of improvement is not signif-
icant beyond a certain number of samples/user, which is an
added advantage of the proposed system.
Several extensions of the present work are possible. For
examples, developing robust biometric techniques, designing
of algorithms for automatic gesture recognition, and proposing
novel human computer interface, are possible.
R EFERENCES
[1] D. Impedovo and G. Pirlo. Automatic signature verification - the state of
the art. IEEE Transactions on Systems Man and Cybernetics, 38(5):609–
635, 2008.
3530
Authorized licensed use limited to: UNIVERSITE DE SFAX. Downloaded on July 10,2023 at 15:58:54 UTC from IEEE Xplore. Restrictions apply.
[2] R. Sabourin, G. Genest, and F. Preteux. Off-line signature verification
by local granulometric size distributions. IEEE Trans. Pattern Analysis
Machine Intelligence, 19(9):976–988, 1997.
[3] D. S. Guru and H. N. Prakash. Online signature verification and
recognition: An approach based on symbolic representation. IEEE
Transaction on Pattern Analysis and Machine Intelligence, 31(6):1059–
1073, 2009.
[4] A. Shanker and A. Rajagopalan. Off-line signature verification using
dtw. Pattern Recognition Letter, 28(12):1407–1414, 2007.
[5] S. K. Behera, D. P. Dogra, and P. P. Roy. Fast recognition and
verification of 3d air signatures using convex hulls. Expert Systems
with Applications, 100:106–119, 2018.
[6] S. K. Behera, D. P. Dogra, and P. P. Roy. Analysis of 3d signatures
recorded using leap motion sensor. Multimedia Tools and Applications,
pages 1–26, 2017.
[7] S. K. Behera, P. Kumar, D. P. Dogra, and P. P. Roy. Fast signature
spotting in continuous air writing. In In Proc. of Fifteenth IAPR
International Conference on Machine Vision Applications (MVA), pages
314–317. IEEE, 2017.
[8] A.K. Jain, F. Griess, and S. Colonnel. On-line signature verification.
Pattern Recognition, 35:2963–2972, 2002.
[9] L. L. Lee, T. Berger, and E. Aviczer. Reliable on-line signature verifi-
cation systems. IEEE Trans. Pattern Analysis and Machine Intelligence,
18(6):643–649, 1996.
[10] V. Nguyen, M. Blumenstein, and G. Leedham. Global features for the
off-line signature verification problem. In ICDAR, pages 1300–1304,
2009.
[11] I. Nakanishi amd H. Sakamoto, N. Nishiguchi, Y. Itoh, and Y. Fukui.
Multimatcher on-line signature verification system in dwt domain.
IEICE Trans Fundam, page 178185, 2006.
[12] E. M. Nel, J. du Preez, and B. Herbst. Estimating the pen trajectories
of static signatures using hidden markov models. IEEE Transactions on
Pattern Analysis and Machine Intelligence, 27:1733–1746, 2005.
[13] L. J. Latecki, V. Megalooikonomou, Q. Wang, and D. Yu. An elastic
partial shape matching technique. Pattern Recognition, 40(11):3069 –
3080, 2007.
[14] K. Alister and B. Yanikoglu. Identity authentication using improved
on-line signature verification method. Pattern Recognition Letters,
26(18):2400–2408, 2005.
[15] J.P. Draouhard, R. Sabourin, and M. Godbout. A neural network
approaches to on-line signature verification using directional pdf. Pattern
Recognition, 29:415–424, 1996.
[16] Y. Bengio et al. Learning deep architectures for ai. Foundations and
trends R in Machine Learning, 2(1):1–127, 2009.
[17] L. Deng. A tutorial survey of architectures, algorithms, and applications
for deep learning. APSIPA Transactions on Signal and Information
Processing, 3, 2014.
[18] X. Chai, Z. Liu, F. Yin, Z. Liu, and X. Chen. Two streams recurrent
neural networks for large-scale continuous gesture recognition. In In
Proc. of 23rd International Conference on Pattern Recognition (ICPR),
pages 31–36. IEEE, 2016.
[19] P. Wang, W. Li, S. Liu, Z. Gao, C. Tang, and P. Ogunbona. Large-scale
isolated gesture recognition using convolutional neural networks. In In
Proc. of 23rd International Conference onPattern Recognition (ICPR),
pages 7–12. IEEE, 2016.
[20] P. Wang, W. Li, S. Liu, Y. Zhang, Z. Gao, and P. Ogunbona. Large-scale
continuous gesture recognition using convolutional neural networks. In
In Proc. of 23rd International Conference onPattern Recognition (ICPR),
pages 13–18. IEEE, 2016.
[21] N. C. Camgoz, S. Hadfield, O. Koller, and R. Bowden. Using con-
volutional 3d neural networks for user-independent continuous gesture
recognition. In In Proc. of 23rd International Conference on Pattern
Recognition (ICPR), pages 49–54. IEEE, 2016.
[22] J. Yang, M. N. Nguyen, P. P. San, X. Li, and S. Krishnaswamy. Deep
convolutional neural networks on multichannel time series for human
activity recognition. In In Proc. of the Twenty-Fourth International Joint
Conference on Artificial Intelligence (IJCAI), pages 3995–4001, 2015.
[23] S. Jaeger, S. Manke, and A. Waibel. Npen++: An on-line handwriting
recognition system. In 7th International Workshop on Frontiers in
Handwriting Recognition, pages 249–260, 2000.
[24] S. Rashidi, A. Fallah, and F. Towhidkhah. Authentication based on
signature verification using position, velocity, acceleration and jerk sig-
nals. In Proc. of IEEE 9th International ISC Conference on Information
Security and Cryptology (ISCISC), pages 26–31. IEEE, 2012.