Speaker Identification Based Proxy Attendance

Detection System
Naman Gupta, Shikha Jain
Department of Computer Science and Information Technology
Jaypee Institute of Information Technology
Noida, India
namang500.ng@gmail.com, shi_81@rediffmail.com

Abstract—Recently, voice biometrics has gained
popularity due to its large number of applications. The
paper presents Siamese Network and CIFAR network-
based architecture for speaker identification. It uses the
voice metrics and voice features independent of the
uttered words. The model is trained on a dataset
containing hundreds of audio clips of 10 speakers. The
model achieved a test accuracy of 88% and training
accuracy of 98.5%. The proposed model is used for
proxy attendance detection in the classroom.
Keywords: Voice Biometrics; Siamese Network,
CIFAR network; Speaker Identification; Proxy
Attendance Detection
I.INTRODUCTION
Machine learning and artificial intelligence are
transforming the world rapidly, by automating
various manual processes. Voice biometrics is one
of them. It has gained much popularity in recent
years. Voice biometrics is a technology used to
match personal voice pattern and verify the
speaker's identity using only voice. Speaker
identification is an emerging research area in
speech technology.
There are a large number of applications of
speaker identification systems such as banking,
telecommunication systems, security devices,
forensic department and many more. The current
open fields for voice biometrics are Government
Immigrant Check-in, Customer Service
Authentication, and Employee Workforce
Management Check-in.
For a human, it is very easy to verify a person just
by hearing the known voices. But for machines, it
is indeed a difficult task to achieve high accuracy
comparable to a human. A lot of work [1-12] has
already been done in this area. However, either
good accuracy is not achieved or the model is
text-dependent [1][4][6-7]. Such models are
dependent on text or the words a person utters.
Therefore, having a speaker recognizing model to
achieve high accuracy independent of the gender
and the native languages is still an open research
problem. Most of the researchers [10][13] have
used MFCC, delta, and delta-delta as the feature
extraction methods in context to speech. These
feature extraction methods have gained high
popularity in the field of speaker verification and
recognition with beats and tempo counted as
secondary features. Nevertheless, the use of CNN
in speaker verification simplifies the task of
feature extraction by finding out those voice
patterns in human voices at its own. The
architecture of Siamese Network uses the shared
weights of CNN’s for finding similar and different
features between same and two different persons.
CNN mainly consists of three layers: the
convolutional layer, the pooling layer and the
dense layer. The convolutional layer is used to
find important features by matching it with
corresponding filter patterns. The pooling layer is
used to capture any translational, rotational and
scaling invariance. The dense layer is the fully
connected layer used for classification. Thus,
CNN is robust to any type of manipulations which
can be carried out on the images. The filters used
in CNN are used to find particular patterns in
images which are important for classification,
object detection or recognition task.
Heigold et al. [6] used techniques to infer the use
of triplet loss function for a dataset with triplet
pairs consisting of an anchor, positive and
negative samples. The anchor is the sample which
is required to test on positive voice recording of
the same person. The negative samples are those
which are hard for the model to verify speaker
identity. Thus, we need to make our model to
have much far distance from the negative sample
and at the same time have the much closer
distance to positive sample. In this paper, we have
used hundreds of audio recording of ten different
speakers with five male and five female voices.
The dataset is thus, constructed using various

978-1-5386-9436-7/19/$31.00 ©2019 IEEE

175

Authorized licensed use limited to: MAULANA AZAD NATIONAL INSTITUTE OF TECHNOLOGY. Downloaded on February 08,2023 at 06:14:25 UTC from IEEE Xplore. Restrictions apply.
 combinations of same and different speaker pairs
irrespective of their gender and language. The
dataset contains ten pairs where both audio clips
are of same speaker and ten pairs where audio
recording are of different speakers.
The paper mainly focuses on the speaker
identification based on the voice metrics and
voice features. The model is designed using
modified CIFAR network architecture as a shared
weight part on Siamese network. The proposed
model is used for proxy attendance detection in
the classroom.
The paper is organized as follows: Section 2
discusses the related work followed by critical
analysis in Section 3. The proposed model and the
results are discussed in details in Section 4 and 5
respectively. The paper is concluded in Section 6
followed by future research directives in Section
7.
II.RELATED WORK
The task of speaker recognition model is to extract
those feature vector embeddings that are most
required to verify persons’ identity. This has been
an area of keen interest for many researchers and
there have been great improvements in the way
machines are able to authenticate persons’
identity. Earlier, many different approaches were
used to tackle the problem. For instance, the
rigorous research was done on pattern matching of
general text-dependent word samples spoken by
familiar persons on a telecom. HMM (Hidden
Markov’s Model) model was tested over 10
speakers produced an accuracy of around 68% to
76% [9]. Recently, the most advanced systems are
accurate enough to achieve very good accuracy
and at the same time acquiring 200 voices related
features. Nammous et al. [3] proposed a method
for linear prediction which smoothen the speech
signals transformed by Fast Fourier Transform
(FFT) used for transition from time-amplitude to
the amplitude-frequency domain. The circulant
(Töeplitz) matrices are calculated from these
points to extract their minimal eigenvalues and
eigenvectors, which are used to produce the
unique feature vector for each audio sample.
These features are then fed to Radial Basis
Function Neural Network.
Residual Conv-Nets are used in text-independent
speaker verification for training deep network [4].
176
Authorized licensed use limited to: MAULANA AZAD NATIONAL INSTITUTE OF TECHNOLOGY. Downloaded on February 08,2023 at 06:14:25 UTC from IEEE Xplore. Restrictions apply.
Around, 19 MFCC(Mel Frequency Cepstral
Coefficients) and log-energies features are first
normalized along mean and variance and then
combined with tempo, delta, delta-delta, and
rhythmic features to produce quality feature
vectors.
Residual architecture is an arrangement of several
residual units. The residual unit is a mapping with
x l +1 = x l + F(x l , w 1 ) where x l and x l +1 are the
unit’s input and output. Additive "shortcut
connection" allows the network to satisfy the
basic property: there is no harm in making model
deep of layers. Thus, it becomes easy to train
large networks. The model referred at [14] was
trained on dataset 300 different speaker (143
female and 157 male) in different environment
uttering same text. Adaptive Moment optimizer
with learning rate set at 10e-4 is used for training
Network is trained to discriminate between all
speakers in training set using the softmax layer
and categorical cross-entropy loss function.
Earlier 512 units of i-vectors were used to find a
person’s authenticity and identity.
For achieving high accuracy on deep residual
CNN, it requires large training data and deep layer
architectures and well-tuned hyper-parameters.
The following steps are used for features
extraction using MFCC:
1. Split an original data (audio file) into small
partitions (30 sec into many small partitions of
20 ms)
2. Calculate each frame’s density of the power
spectrum.
3. Then, filter bank calculates energy and sums
up the energy of each filter.
4. Then, calculate the logarithm of all filter bank
energies.
5. Calculate the Direct Cosine Transform of all
filter banks energies.
Recently, a dataset Vox-Celeb is proposed [2]
which were extracted by running model on
thousands of YouTube videos of different celebs
along with matching the lips of the speaker using
LipNet model. Another researcher [1] gives
intuition about triplet-loss training examples. The
triplets are namely Anchor (testing data), Positive
(similar recording of a person), Negative (similar
 recording of a different person which can closely
mimic the anchor person). The main aim is to
decrease the similarity distance between the
anchor and positive and increase the dissimilarity
distance between the anchor and the negative
III.CRITICAL ANALYSIS
Speaker authentication has been a problem
that has been addressed by many researchers in
recent times. For this a variety of approaches have
been adopted. Some researchers have used
VGG16 CNN model [2] to extract features from
mel-spectrograms from voice samples. This model
is robust to invariance and is capable of extracting
important features. Yet, the problem with VGG16
model is that it has high time complexity as
requires a lot of computation and memory to load
model architecture and its weights along with
heavy paired dataset. Researchers have been
trying to improve the performance of model but
the problem of high time complexity has not been
tackled.
IV.PROPOSED ALGORITHM
This paper proposes a speaker identification
model using modified CIFAR network
architecture as shared weight part on Siamese
network. The beauty of the model is that it uses
minimal calculations and less memory. For
accurate identification of speakers, first step is to
extract the important features in terms of vector
embeddings. These feature vectors actually
represents the pattern in the voice of the speaker
independent of the uttered words. Next step is to
Fig 1: Modified CIFAR architecture
177
Authorized licensed use limited to: MAULANA AZAD NATIONAL INSTITUTE OF TECHNOLOGY. Downloaded on February 08,2023 at 06:14:25 UTC from IEEE Xplore. Restrictions apply.
train the model to find similarity between voice
patterns of two different speakers. For this, a
dataset consisting of ten different speakers with
equal number of male and female speakers is
used. This dataset contains hundreds of audio
recordings of speakers in different languages.
The model is implemented using Google Colab
platform using K80 Tesla GPU and i7 7th gen core
processor. The original CIFAR network is having
10 neurons in the last layer. Here, instead of 10
neurons, 128 neurons are used to generate more
detailed feature vector (as shown in figure 1).
Nevertheless, in the learning phase, a special type
of neural network, Siamese Network (SNN) is
used. The advantage of using SNN is that it uses a
concept of distance-based loss. As a result it
generates better embeddings. Moreover, this kind
of network learns to differentiate between two
inputs rather than simply classifying them. The
proposed architecture is shown in figure 2.
Finally, it generates zero indicating different
speakers and one indicating same speaker. Based
upon the error, the weights are updated. The
learning rate is set to 0.0001 for the initial hundred
epochs. As the model approaches to near optimum
level, the learning rate was decreased ten times to
0.00001 so that it can achieve global optimum.
The model is trained using ADAM optimizer using
batch normalization at each layer along with
dropout set to 0.2 for dense layer connected just
after convolution layer for minimizing the chances
of over fitting.
 1 if input A and input
feature embeddings with speaker 3 as can be seen
B are same else 0 by comparing F1-Score.

Sigmoid
Activation

Vector embedding Vector embedding

for input A for input B

CIFAR Network Shared Weights CIFAR Network

Input A Input B Fig 3: Classification Report of ten speakers

(Image of Voice (Image of Voice
Melspectrogram) Melspectrogram)

Table 1 shows some of the test cases of speaker

Fig 2: Proposed architecture identification. It can be seen that the model is able
Once, the model is trained and reached the to identify the speaker correctly. Moreover, it is
global optimum and achieved promising accuracy able to detect the proxy attendance as well. The
then the last layer of the model was popped out in model has achieved a decent test accuracy of 88%
order to give 128 hidden unit layer which acts as and training accuracy of 98.5%. Table 2 shows the
feature vector embedding. The 1 X 128 shaped comparative performance of CIFAR based
vector-embedding of different speakers is the proposed model with VGG16 based model. It can
quality feature of the voice of that particular be seen that the proposed model outperform the
speaker. The vector embedding of all the speakers existing model in terms of time complexity and
was extracted using Siamese model and is stored accuracy both.
in the database. Table 1: Test Cases
Input Output
Once the embeddings are generated for each Input Audio file: Expected Speaker:
speaker and the model is trained to differentiate Speaker_1.wav Speaker_1
between any two speakers, the model is saved for Name called for: Speaker_1 Prediction:
the actual use of speaker identification. For Person Verified.... Speaker
speaker identification, the Mel-spectrogram of the no. 1
speaker is passed as input to the trained model to Input Audio file: Expected Speaker:
Speaker_3.wav Speaker_3
generate its 1 X 128 vector embedding. Then, the
Name called for: Speaker_4 Prediction: Beware, It’s a
cosine similarity is used to determine the similarity proxy.... This is
between the current speakers and all the already Speaker no. 3
processed speakers at real time to identify the Input Audio file: Expected Speaker:
speaker. Here, we considered threshold parameter Speaker_8.wav Speaker_8
to be equal to 0.75. The proposed model is used Name called for: Speaker_8 Prediction:
for proxy attendance detection in the classroom. Person Verified.... Speaker
no. 8
V.RESULTS Input Audio file: Expected Speaker:
Speaker_6.wav Speaker_6
For the verifying the speaker, the vector Name called for: Speaker_6 Prediction:
embedding were compared between various Person Verified.... Speaker
speakers. Figure 3 presents the generated no. 6
classification report. It can be observed that the
model is able to identify 9 speakers with high F1-
Score except speaker 6. It is because of similar

178

Authorized licensed use limited to: MAULANA AZAD NATIONAL INSTITUTE OF TECHNOLOGY. Downloaded on February 08,2023 at 06:14:25 UTC from IEEE Xplore. Restrictions apply.

Table 2: Comparative Performance
Parameters VGG 16 Proposed
Based Model Model
No of Epochs 3000 2500
Training Time 3 Hrs 1.5 Hrs
Accuracy 81% 88%
VI. CONCLUSION
The paper presents a speaker identification
model based on the voice metrics and voice
features independent of the uttered words. The
model is designed using modified CIFAR network
architecture as shared weight part on Siamese
network generating a feature vector of size 1 X
128. The model is trained on dataset containing
hundreds of audio clips of 10 speakers The model
achieved a decent test accuracy of 88% and
training accuracy of 98.5%. Thus, Model was
accurate to verify 8 speakers independently. Due
to some correlation in voices between speaker 3
and speaker 6, the model was sometimes mistaken
for some voice recordings. The model can be used
in a number of applications such as banking,
telecommunication systems, security devices,
forensic department. Here, it is used as a proxy
attendance detection model in the classroom.
VII.FUTURE WORK
In future, we intend to work upon the accuracy
metric suitable for large scale speaker
identification with end to end deployment. This
requires a different arrangement of dataset with
anchor, positive and negative samples trained on
triplet loss function and having really deep neural
network with latest neural network architecture.
VIII.REFERENCES
[1] C. Li, X. Ma, B. Jiang, X. Li, X. Zhang, X. Liu, Y.
Cao, A. Kannan, and Z. Zhu. "Deep speaker: an end-
to-end neural speaker embedding system." arXiv
preprint arXiv:1705.02304 , 2017.
[2] A. Nagrani, J. S. Chung, and A. Zisserman.
"Voxceleb: a large-scale speaker identification
dataset." arXiv preprint arXiv:1706.08612, 2017.
[3] M. K. Nammous, A. Szczepański, and K. Saeed. "An
exploratory research on text-independent speaker
recognition." In International Conference on Hybrid
`
179
Authorized licensed use limited to: MAULANA AZAD NATIONAL INSTITUTE OF TECHNOLOGY. Downloaded on February 08,2023 at 06:14:25 UTC from IEEE Xplore. Restrictions apply.
Artificial Intelligence Systems, pp. 412-419.
Springer, Berlin, Heidelberg, 2011.
[4] E. Malykh, S. Novoselov, and O. Kudashev. "On
residual CNN in text-dependent speaker verification
task." In International Conference on Speech and
Computer, pp. 593-601. Springer, Cham, 2017.
[5] S. Narang, and D. Gupta. "Speech feature extraction
techniques: a review." International Journal of
Computer Science and Mobile Computing 4, no. 3,
2015, pp: 107-114.
[6] G. Heigold, I. Moreno, S. Bengio, and N. Shazeer.
"End-to-end text-dependent speaker verification." In
2016 IEEE International Conference on Acoustics,
Speech and Signal Processing (ICASSP), pp. 5115-
5119. IEEE, 2016.
[7] D. K. Burton, “Text-dependent speaker verification
using vector quan-tization source coding,” IEEE
Trans. Acoust., Speech, Signal Process.,vol. ASSP-
35, pp. 133, 1987.
[8] H. Salehghaffari, "Speaker Verification using
Convolutional Neural Networks." arXiv preprint
arXiv:1803.05427, 2018.
[9] U. Mahola, F. V. Nelwamondo, and T. Marwala.
"HMM Speaker Identification Using Linear and
Non-linear Merging Techniques." arXiv preprint
arXiv:0705.1585 , 2007.
[10] P. Annesi, R. Basili, R. Gitto, A. Moschitti, and R.
Petitti. "Audio feature engineering for automatic
music genre classification." In Large Scale Semantic
Access to Content (Text, Image, Video, and Sound),
pp. 702-711. Le Centre De Hautes Etudes
Internationales D'Informatique Documentaire, 2007.
[11] D. Snyder, D. Garcia-Romero, D. Povey, and S.
Khudanpur. "Deep Neural Network Embeddings for
Text-Independent Speaker Verification." In
Interspeech, pp. 999-1003. 2017.
[12] K. Chen, and A. Salman. "Extracting speaker-
specific information with a regularized siamese deep
network." In Advances in Neural Information
Processing Systems, pp. 298-306. 2011.
[13] N. Dave, "Feature extraction methods LPC, PLP and
MFCC in speech recognition." International journal
for advance research in engineering and technology
1, no. 6, 2013, pp: 1-4.
[14] S. Safavi, and I. Mporas. "Improving performance of
speaker identification systems using score level
fusion of two modes of operation." In International
Conference on Speech and Computer, pp. 438-444.
Springer, Cham, 2017.