Some Investigations on Speaker Identification from Speech
Segments using Deep Learning
Abhik Pramanik Arka prabha Saha
BEE,4th Year
Department of Electrical Engineering
Jadavpur University
Kolkata 700032
Abstract: Audio-visual recognition (AVR) has been
considered as a solution for speech recognition tasks
when the audio is corrupted, as well as a visual
recognition method used for speaker verification in
multi-speaker scenarios. The approach of AVR
systems is to leverage the extracted information from
one modality to improve the recognition ability of the
other modality by complementing the missing
information. Speaker recognition models that utilize
neural networks to map utterances where distances
reflect similarity between speakers have driven the
recent progress in the speaker recognition task.
However, there is still a significant performance gap
between training set and unseen speakers. Here we
have employed certain time and frequency domain
techniques to investigate the features of the speech
segments which is then applied in a VGG model with
classification loss to estimate the speaker of the
speech segment.
Index Terms—Convolutional Networks, Deep Learning,
Audio-visual Recognition.
1. Introduction:
Speaker recognition under noisy and unconstrained
conditions is an extremely challenging topic.
Applications of speaker recognition are many and
varied, ranging from authentication in high-security
systems and forensic tests, to searching for persons in
large corpora of speech data. All such tasks require
high speaker recognition performance under ‘real
world’ conditions. This is an extremely difficult task
due to both extrinsic and intrinsic variations; extrinsic
variations include background chatter and music,
laughter, reverberation, channel and microphone
Ankan Kumar Bhunia
effects; while intrinsic variations are factors inherent
to the speaker themselves such as age, accent,
emotion, intonation and manner of speaking,
amongst others. Deep Convolutional Neural
Networks (CNNs) have given rise to substantial
improvements in speech recognition, computer
vision and related fields due to their ability to deal
with real world, noisy datasets without the need for
handcrafted features. One of the most important
ingredients for the success of such methods,
however, is the availability of large training datasets
Speaker recognition with only a few and short
utterances is a challenging problem. Recently many
approaches have been proposed for speaker
identification using deep neural networks but they
assume availability of large amount of training data.
Moreover, the deep learning pipelines attempting to
learn from short utterances are, in general, based on
i-vectors or Mel-frequency cepstral coefficients
(MFCCs). While MFCCs are known for susceptibility to
noise, the performance of i-vectors tends to suffer in
case of short utterances. Moreover, it has been
shown that convolutional neural networks (CNNs) are
able to mitigate the noise susceptibility of i-vectors,
MFCCs and have been successfully used for speaker
recognition. Since convolutional neural networks are
data-hungry, and are able to exploit structured
information such as images very effectively, recently
large-scale speaker recognition datasets have been
made publicly available where benchmarks setup on
CNNs with spectrogram as input have been shown to
perform very well. However, their effectiveness
 Figure 1 Architecture of the speaker recognition model
tolearn or generalize with limited amount of data
(few-shots) and short utterances is not established
very well. Few shot learning paradigms have recently
been effectively applied for audio processing.
However, their effectiveness to speech processing
especially speaker recognition is still unknown. We
choose VGGNet as the base architectures and
evaluate them under various settings. While CNNs
have shown to perform very well, but they are not
able to exploit the spatial relationships within an
image. CNNs are not able to leverage the spatial
information within spectrograms such as between
pitch and formant. Therefore, we utilized VGG
networks for speech command recognition. However,
there are two problems in applying VGG network for
speaker recognition. First is that their applicability to
complex data is yet to be established and hence the
generalization ability. Second, they are extremely
computationally intensive. The projected feature
vector from the embedding space are then subjected
to L2 loss to learn from the constrained data. The
entire pipeline is trained end-to-end.
2. Related Work:
Lip reading and audio-visual speech recognition
(AVSR) are highly correlated such that the relevant
information of one modality can improve the
recognition of the other modality in any of the two
aforementioned applications. DNNs have been
employed for fusing speech and visual modalities for
audio visual automatic speech recognition (AV-ASR)
[10]. Moreover, in [11], a connectionist HMM system
is introduced for AVSR and a pre-trained CNN is used
to classify phonemes. Some researchers have used
CNNs to predict and generate phonemes [12] or
visemes [13] without considering the word level
prediction. Phonemes are the smallest
distinguishable unit of an audio stream which are
combined to create a spoken word, and a viseme is its
corresponding visual equivalent. For recognizing full
words, a Long Short-Term Memory (LSTM) classifier
with Discrete Cosine Transform (DCT) and Deep
Bottleneck Features (DBF) is trained [14]. Similarly,
LSTM with Histogram of Oriented Gradients (HOG)
features are used also for phrase recognition [15].
One of the most challenging applications of audio-
visual recognition is the audio-video synchronization
for which the audio-visual matching ability is
required. The research efforts related to this paper
consist of different audio-visual recognition tasks
which try to find the correspondence between the
two modalities. Different approaches have been
employed for tackling the audio-visual matching
problem. Some are based on data-driven approaches,
such as using DNN classifiers to determine the off-
sync time [16], [17] and some are based on Canonical
Correlation Analysis (CCA) [18] and Co-Inertia Analysis
(CoIA).
3. Methodology:
Our aim is to move from techniques that require
traditional hand-crafted features, to a CNN
architecture that can choose the features required for
the task of speaker recognition. This allows us to
minimise the pre-processing of the audio data and
hence avoid losing valuable information in the
process. We create a CNN by modifying an existing
VGG architecture and train it on spectrograms from
50 unique speakers.
3.1 Architecture:
All audio is first converted to single-channel, 16-bit
streams at a 16kHz sampling rate for consistency.
Spectrograms are then generated in a sliding window
fashion using a hamming window of width 25ms and
step 10ms. This gives spectrograms of size 512 x 300
for 3 seconds of speech. Mean and variance
normalisation is performed on every frequency bin of
the spectrum. No other speech-specific pre-
processing (e.g. silence removal, voice activity
detection, or removal of unvoiced speech) is used.
These short time magnitude spectrograms are then
used as input to the CNN. Architecture. Since speaker
identification under a closed set can be treated as a
multiple-class classification problem, we base our
architecture on the VGG-M CNN, known for good
classification performance on image data, with
modifications to adapt to the spectrogram input. The
fully connected fc6 layer of dimension 9 × 8 (support
in both dimensions) is replaced by two layers – a fully
connected layer of 9×1 (support in the frequency
domain) and an average pool layer with support 1 ×
n, where n depends on the length of the input speech
segment (for example for a 3 second segment, n = 8).
This makes the network invariant to temporal
position but not frequency, and at the same time
keeps the output dimensions the same as those of the
original fully connected layer. This also reduces the
number of parameters from 319M in VGG-M to 67M
in our network, which helps avoid overfitting.
3.2 Spectrogram Construction:
The Speech signal is a non-stationary signal. So the
normal Fourier analysis will not give proper result.
Thus we have to use STFT or Short Time Fourier
Transform that gives the frequency spectrum of the
signal as a function of time as well as frequency. The
spectrum so obtained is called spectrogram. But STFT
has a disadvantage that the proper frequency
resolution and time resolution cannot be decided
properly. So we can use wavelet transform using any
wavelets like Harr, Mexican Hat, Daubechis, Mathieu
etc. The visual representation of the transform so
obtained is called scalogram.
So, for spectrogram First we convert all audio to
single-channel, 16-bit streams at a 16 kHz sampling
rate for consistency. The spectrograms are then
generated by sliding window protocol by using a
hamming window. The width of the hamming window
is 25 ms with a step size of 10 ms. This gives
spectrograms of size 128 (number of FFT features) x
300 for 3 seconds of randomly sampled speech for
each audio. Subsequently, each frequency bin is
normalized (mean, variance). the spectrograms are
constructed from the raw audio input i.e. no pre-
processing such as silence removal etc. is performed.
4. Experiments:
First, we will discuss about the dataset we have used
and some of the salient implementation details. Then,
we will present our results of our proposed model.
4.1 Datasets:
For our experiments, we have used a simple dataset
available on Kaggle. It consists of 50 speakers’ audio
data with length more than 1 hour for each. This
dataset can be used for speaker recognition kind of
problems. This dataset was scraped from YouTube and
Librivox. Each speaker has 80 – 90 tracks. For the
training purpose we have used different numbers of
tracks i.e., 10, 20, 30, 40, 50 tracks per speaker and the
rest of the tracks were used to test our model.
4.2 Implementation Details:
We implement our framework in TensorFlow on a
server with Nvidia Titan X GPU. Optimization of the
network is done with Adam Optimizer. The model is
trained for 20k iterations with batch size 32 and
learning rate 0.001. The weight decay regularization
parameter is fixed to 5 × 10−4. The computational cost
increases with the length of the images resulting more
time to converge. Also, batch size was reduced to 32
to accommodate the GPU’s memory. During
evaluation we noticed that each file takes roughly
85ms on average on GeForce Titan X.
Table I. Experimental results using different number of
training samples used per speaker.
Training data/speaker Accuracy (%)
10 51.3
20 71.3
30 81.3
40 85.3
50 94.2
4.3 Results:
In Table I, the results of the method is shown.
According to the obtained data, it is evident that
accuracy of our model decreases with decreasing
training samples. We have seen that if we use 10
samples per classes, the results become 51.3%.
5. Future Extension:
The method has several challenges. First of all, in
reality, we will have access to a very limited number of
samples per speaker. Thus, it is not possible to train
the model with that limited examples. Secondly, the
method uses an end-to-end training strategy. Thus, it
is difficult to use the model for new classes. There is a
requirement of such model that is easily scalable.
Keeping these disadvantages in our mind, we have
formulated a few-shot learning techniques. This
model can be used to obtain a feature representation
of the audio files using some metric learning methods.
Now, we will discuss the theoretical foundation of our
future work based on few-shot learning. We are
assuming that for each speaker, only a few very short
utterances are available. Specifically, we learn from 1
to 5 small utterances of 3 seconds each. Apart from
computational and technical improvisations obtained
by solving under such constraints, this setting also
allows enrolment of speakers to be easier and more
practical.
5.1 Few shot learning using Prototypical Loss:
In few shot learning, at test time we have to classify
test samples among K new classes given very few
labeled examples of each class. We are provided with
D dimensional embeddings Xi ⋳ ℝD for each input Xi
and its corresponding label Yi where
Yi ⋳ {1, . . . ,K}. The objective is to compute a M
dimensional representation i.e. the prototype of each
class ak ⋳ ℝM. The embedding is computed via a
function f𝜙 : ℝD → ℝM where f𝜙 indicates a deep
neural network and _ indicates its parameters. The
prototype ak is the mean of the support points for a
class. By using a distance function d, a distribution
over classes is learned for a query point q, and is given
as
exp(−𝑑(𝑓∅ (𝑞),𝑎𝑘 ))
𝑝∅ ( 𝑦 = (𝑘|𝑞) = ∑𝑘′ exp(−𝑑(𝑓∅ (𝑞),𝑎𝑘′ ))
At train time, we minimize negative log probability of
the positive class. The training data is generated by
randomly selecting a smaller subset from the training
set. Then from each class, we choose a subset of
samples which are considered as support points while
the rest are considered as query points. The flow
diagram for few shot learning is shown in figure above.
6. Conclusion:
In order to establish benchmark performance, we
develop a VGG CNN architecture with the ability to
deal with variable length audio inputs. We have shown
the results on a speech dataset collected from 50
speakers. We have discussed some of the
disadvantages of the method. We also formulated our
future extension of our work using few-shot learning
technique.
7. References:
1. D. Snyder, D. Garcia-Romero, G. Sell, D. Povey,
and S. Khudanpur, “X-vectors: Robust dnn
embeddings for speaker recognition,” in 2018
IEEE International Conference on Acoustics,
Speech and Signal Processing (ICASSP). IEEE, 2018,
pp. 5329–5333.
2. D. Snyder, D. Garcia-Romero, D. Povey, and S.
Khudanpur, “Deep neural network embeddings
for text-independent speaker verification.” in
Interspeech, 2017, pp. 999–1003.
3. Y. Lei, N. Scheffer, L. Ferrer, and M. McLaren, “A
novel scheme for speaker recognition using a
phonetically-aware deep neural network,” in
2014 IEEE International Conference on Acoustics,
Speech and Signal Processing (ICASSP). IEEE, 2014,
pp. 1695–1699.
4. J. Guo, N. Xu, K. Qian, Y. Shi, K. Xu, Y. Wu, and A.
Alwan, “Deep neural network based i-vector
mapping for speaker verification using short
utterances,” Speech Communication, vol. 105, pp.
92–102, 2018.
5. X. Zhao and D. Wang, “Analyzing noise robustness
of mfcc and gfcc features in speaker
identification,” in 2013 IEEE international
conference on acoustics, speech and signal
processing. IEEE, 2013, pp. 7204–7208.
6. A. Kanagasundaram, R. Vogt, D. B. Dean, S.
Sridharan, and M. W. Mason, “I-vector based
speaker recognition on short utterances,” in
Proceedings of the 12th Annual Conference of the
International Speech Communication Association.
International Speech Communication Association
(ISCA), 2011, pp. 2341– 2344.
7. M. McLaren, Y. Lei, N. Scheffer, and L. Ferrer,
“Application of convolutional neural networks to
speaker recognition in noisy conditions,” in
Fifteenth Annual Conference of the International
Speech Communication Association, 2014.
8. Z. Liu, Z. Wu, T. Li, J. Li, and C. Shen, “Gmm and
cnn hybrid method for short utterance speaker
recognition,” IEEE Transactions on Industrial
Informatics, vol. 14, no. 7, pp. 3244–3252, 2018.
9. Y. Mroueh, E. Marcheret, and V. Goel, “Deep
multimodal learning for audio-visual speech
recognition,” in 2015 IEEE International Conference on
Acoustics, Speech and Signal Processing (ICASSP), April
2015, pp. 2130–2134.
10. K. Noda, Y. Yamaguchi, K. Nakadai, H. G. Okuno, and T.
Ogata, “Audio-visual speech recognition using deep
learning,” Applied Intelligence, vol. 42, no. 4, pp. 722–
737, Jun. 2015. [Online]. Available:
http://dx.doi.org/10.1007/s10489-014-0629-7
11. K. Noda, Y. Yamaguchi, K. Nakadai, H. Okuno, and T.
Ogata, Lip reading using convolutional neural network.
International Speech and Communication Association,
2014, pp. 1149–1153.
12. O. Koller, H. Ney, and R. Bowden, “Deep learning of
mouth shapes for sign language,” in 2015 IEEE
International Conference on Computer Vision
Workshop (ICCVW), Dec 2015, pp. 477–483.
13. S. Petridis and M. Pantic, “Deep complementary
bottleneck features for visual speech recognition,” in
2016 IEEE International Conference on Acoustics,
Speech and Signal Processing (ICASSP), March 2016,
pp. 2304–2308.
14. M. Wand, J. Koutnk, and J. Schmidhuber, “Lipreading
with long shortterm memory,” in 2016 IEEE
International Conference on Acoustics, Speech and
Signal Processing (ICASSP), March 2016, pp. 6115–
6119
15. J. S. Chung and A. Zisserman, “Out of time: automated
lip sync in the wild,” in Workshop on Multi-view Lip-
reading, ACCV, 2016.
16. E. Marcheret, G. Potamianos, J. Vopicka, and V. Goel,
“Detecting audiovisual synchrony using deep neural
networks,” in INTERSPEECH, 2015.
17. M. E. Sargin, Y. Yemez, E. Erzin, and A. M. Tekalp,
“Audiovisual synchronization and fusion using
canonical correlation analysis,” IEEE Transactions on
Multimedia, vol. 9, no. 7, pp. 1396–1403, 2007.