Digital Signal Processing 22 (2012) 544–553

Contents lists available at SciVerse ScienceDirect

Digital Signal Processing

www.elsevier.com/locate/dsp

A hierarchical language identification system for Indian languages

S. Jothilakshmi ∗ , V. Ramalingam, S. Palanivel
Department of Computer Science and Engineering, Annamalai University, Annamalainagar 608 002, India

a r t i c l e i n f o a b s t r a c t

Article history: Automatic spoken Language IDentification (LID) is the task of identifying the language from a short
Available online 27 January 2012 duration of speech signal uttered by an unknown speaker. In this work, an attempt has been made
to develop a two level language identification system for Indian languages using acoustic features. In
Keywords:
the first level, the system identifies the family of the spoken language, and then it is fed to the second
Language identification
Mel frequency cepstral coefficients
level which aims at identifying the particular language in the corresponding family. The performance
Shifted delta cepstral coefficients of the system is analyzed for various acoustic features and different classifiers. The suitable acoustic
Hidden Markov model feature and the pattern classification model are suggested for effective identification of Indian languages.
Gaussian mixture model The system has been modeled using hidden Markov model (HMM), Gaussian mixture model (GMM) and
Neural networks artificial neural networks (ANN). We studied the discriminative power of the system for the features mel
Indian languages frequency cepstral coefficients (MFCC), MFCC with delta and acceleration coefficients and shifted delta
cepstral (SDC) coefficients. Then the LID performance as a function of the different training and testing
set sizes has been studied. To carry out the experiments, a new database has been created for 9 Indian
languages. It is shown that GMM based LID system using MFCC with delta and acceleration coefficients
is performing well with 80.56% accuracy. The performance of GMM based LID system with SDC is also
considerable.
© 2012 Elsevier Inc. All rights reserved.

1. Introduction
Automatic spoken Language IDentification (LID) is the task of iden-
tifying the language from a short duration of speech signal uttered
by an unknown speaker [1,2]. In general, applications of language
identification systems can be categorized into two classes, namely
front-end for machines and front-end for human listeners.
As a front-end for machines, the LID systems find many appli-
cations. In the case of a multi-language speech recognition system,
if the language of the input signal is not known, it will be required
to run speech recognizers of several languages in parallel, which
is computationally very expensive. In this scenario, an LID system
can be used as a front-end to reduce the number of speech rec-
ognizers to be activated, by considering the first n-best languages
declared by the LID system.
Another application in this category is speech-to-speech trans-
lation system, in which the source language is to be converted to
that of the target language. Using an LID system as a front-end
to identify the language of the input (source) speech signal, the
speech recognizers’ performance can be drastically improved. An
important application in this category is at public service points,
where information access is enabled by voice. In countries like
* Corresponding author.
E-mail addresses: jothi.sekar@gmail.com (S. Jothilakshmi), aucsevr@yahoo.com
(V. Ramalingam), spal_yughu@yahoo.com (S. Palanivel).
India, China, South Africa, Vietnam, etc., where a multitude of lan-
guages coexist, it is not always possible to have trained operators
who can assist in different languages, at all public service points.
An automatic language identification system can be extremely use-
ful in such places to provide assistance. Once the language of the
speaker is identified, further responses from the system can be
given in the language of the speaker itself.
As a front-end for human listeners, an LID system can play an
important role in routing an international telephone call to a hu-
man operator who is fluent in the language of the caller. In all the
above mentioned applications, the LID system can be extremely
useful if the performance of the system is reasonably good. In the
recent past, a number of significant results have been obtained in
the area of language identification [3–5].
The LID systems are categorized into two types namely acoustic
LID and phonotactic LID [6]. The acoustic LID determines the lan-
guage directly on the basis of features derived from the speech sig-
nal [7]. In phonotactic LID, speech is first transcribed by phoneme
recognizer into strings or graphs of phonemes. On these, “lan-
guage” models are trained to capture statistics of couples and
triples of phonemes.
It is common for language identification systems to need large
sets of phonemically labeled data for training, usually one set
for each language to be identified [8–10,3]. Such an approach to
the language identification problem is known as Phonemic Recog-
nition followed by Language Modeling (PRLM) [9,11–13]. While
PRLM algorithm is effective, the phonemic labeling required can

1051-2004/$ – see front matter © 2012 Elsevier Inc. All rights reserved.
doi:10.1016/j.dsp.2011.11.008
 S. Jothilakshmi et al. / Digital Signal Processing 22 (2012) 544–553
be a laborious and time consuming process [3], and can also cre-
ate extensibility issues due to the large amount of time and effort
required to add new language models. It is therefore advantageous
for systems to be developed that perform the discrimination task
without this phonemic modeling prerequisite. Attempts at devel-
oping acoustic based models have resulted in the use of techniques
such as vector quantization (VQ) [14], support vector machines
(SVM) [15], and Gaussian mixture model (GMM) [16,17].
The aim of the proposed work is to develop LID system for In-
dian languages which do not require any linguistic information, i.e.,
the system should be less complex and the inclusion of a new
language should be a trivial task. But as far as Indian languages
are concerned, since most of the languages have the same origin,
even though there are few unique sounds, all the languages share
a common set of phonemes. In this work, our intention is to sug-
gest more suitable acoustic features and modeling techniques for
Indian languages to give a reasonable performance.
This paper describes a two level identification system for In-
dian languages using acoustic features. In the first level, the system
identifies the family of the spoken language, and then it is fed
to the second level which aims at identifying the particular lan-
guage in the corresponding family. The commonly used pattern
recognition models in the literature for LID are hidden Markov
model (HMM), GMM, artificial neural networks (ANN) and SVM.
The proposed system has been modeled using HMM, GMM and
ANN for Indian languages. Any pattern recognition task will rely
on a proper choice of features in order to perform well. The most
widely used features for LID are mel frequency cepstral coefficients
(MFCC). Traditionally, language and speaker identification tasks use
feature vectors containing cepstra and delta and acceleration cep-
stra. Recently, however, the shifted delta cepstrum (SDC) has been
found to exhibit superior performance to the delta and acceler-
ation cepstra in a number of language identification studies due
to its ability to incorporate additional temporal information, span-
ning multiple frames, into the feature vector. The performances of
MFCC and SDC features are compared in this work. To carry out
the experiments, a new database has been created for 9 Indian
languages.
This paper is organized as follows: Section 2 briefly reviews ex-
traction of acoustic features used for LID. In Section 3, we briefly
review modeling techniques used for LID. The proposed method is
presented in detail in Section 4. Section 5 describes the database
and the experimental results. Section 6 concludes the paper.
2. Acoustic features
The most widely used features for LID are MFCC features. Tra-
ditionally, language and speaker identification tasks use feature
vectors containing cepstra and delta and acceleration cepstra. Re-
cently, however, the SDC has been found to exhibit superior per-
formance to the delta and acceleration cepstra in a number of LID
studies due to its ability to incorporate additional temporal infor-
mation, spanning multiple frames, into the feature vector. The per-
formances of MFCC and SDC features are compared in this work.
2.1. Mel frequency cepstral coefficients
Mel frequency cepstral coefficients have proved to be one of the
most successful feature representations in speech related recogni-
tion tasks. The mel-cepstrum exploits auditory principles, as well
as the decorrelating property of the cepstrum [18]. The procedure
of MFCC computation is shown in Fig. 1 and is described as fol-
lows:
• Preemphasis: The digitized speech signal S = s(n), n = 1, . . . , N t
(where N t is the total number of samples in the speech
545
Fig. 1. Extraction of MFCC from speech signal.
stream) is put through a low order digital system to spec-
trally flatten the signal and to make it less susceptible to finite
precision effects later in the signal processing. The output of
the preemphasis network, ŝ(n) is related to the input s(n) by
the difference equation
ŝ(n) = s(n) − α s(n − 1) (1)
where α is the scaling factor which varies from 0 to 1. The
most common value for α is around 0.95.
• Frame blocking: Speech analysis usually assumes that the sig-
nal properties change relatively slowly with time. This allows
examination of a short time window of speech to extract pa-
rameters presumed to remain fixed for the duration of the
window. Thus, to model dynamic parameters, we must divide
the signal into successive windows or analysis frames, so that
the parameters can be calculated often enough to follow the
relevant changes. In this step the preemphasized speech sig-
nal, ŝ(n) is blocked into frames of N samples, with adjacent
frames being separated by M samples. If we denote the lth
frame speech by ŝl (n), and there are L frames within the en-
tire speech signal, then
ŝl (n) = ŝ( Ml + n),
n = 0, 1 , . . . , N − 1 , l = 0, 1 , . . . , L − 1 (2)
We used a frame rate of 125 frames/s, where each frame was
16 ms in duration with an overlap of 50 percent between ad-
jacent frames.
• Windowing: The next step in the processing is to window each
individual frame so as to minimize the signal discontinuities
at the beginning and end of the frame. The window must be
selected to taper the signal to zero at the beginning and end
of each frame. If we define the window as w (n), 0  n  N − 1,
then the result of windowing the signal is
s̃l (n) = ŝl (n) w (n), 0n N −1 (3)
The Hamming window is used for our work, which has the
form
 
2π n
w (n) = 0.54 − 0.46 cos , 0n N −1 (4)
N −1
• Computing spectral coefficients: The spectral coefficients of the
windowed frames are computed using fast Fourier transform
(FFT), as follows:

N −1
2π
X (k) = s̃l (n) exp− jk( N
)n
, 0k N −1 (5)
n =0
• Computing mel spectral coefficients: The spectral coefficients of
each frame are then weighted by a series of filter frequency
responses whose center frequencies and bandwidths roughly
match those of the auditory critical band filters. These filters
follow the mel scale whereby band edges and center frequen-
cies of the filters are linear for low frequency and logarith-
mically increase with increasing frequency. These filters are
546 S. Jothilakshmi et al. / Digital Signal Processing 22 (2012) 544–553
Fig. 2. Mel scale filter bank.
called as mel-scale filters and collectively a mel-scale filter
bank. As shown in Fig. 2, the filters used are triangular and
they are equally spaced along the mel-scale which is defined
by
 
f
Mel( f ) = 2595 log10 1 +
700
where f is the linear scale frequency and Mel( f ) is the corre-
sponding mel scale frequency. Each short term Fourier trans-
form (STFT) magnitude coefficient is multiplied by the corre-
sponding filter gain and the results are accumulated.
• Computing mel frequency cepstral coefficients: The discrete cosine
transform (DCT) is applied to the log of the mel spectral co-
efficients to obtain the mel frequency cepstral coefficients as
follows:
  
M f −1
 
2 (2i + 1)mπ
x(m) = E (i ) cos
Mf 2M f
i =0
m = 1, . . . , M f
where M f = number of filters in the filter bank. Finally cep-
stral mean subtraction is performed to reduce the channel
effects.
2.2. Shifted delta cepstral coefficients
The use of the shifted delta cepstral feature vectors allows a
pseudo-prosodic feature vector to be computed without having
to explicitly find or model the prosodic structure of the speech
signal [19]. A shifting delta operation is applied to frame based
acoustic feature vectors in order to create the new combined fea-
ture vectors for each frame.
The SDC coefficients are computed, for a cepstral frame at
time t, according to:
cn (t , i ) = cn (t + i P + D ) − cn (t + i P − D ),
n = 0, . . . , N f − 1 , i = 0, . . . , k b − 1
where n is the nth cepstral coefficient, D is the lag of the deltas,
P is the distance between successive delta computations, and i is
the SDC block number. The final feature vector is obtained by con-
catenation of kb blocks of N f parameters.
The computation of the shifted delta feature vectors is a rel-
atively simple procedure. The process is as follows: The cepstral
feature vectors are first computed as described above. Then, the
acoustic feature vectors spaced D sample frames apart are first dif-
ferenced. Then kb differenced feature vector frames, and spaced P
frames apart are then stacked to form a new feature vector. Fig. 3
gives a graphical depiction of this process.
The configuration 7–1–3–7 for N f − D − P − kb has been used
in this work to extract SDC feature vector. For each frame, with
7 direct MFCC coefficients, 49 SDC coefficients are appended, so
totally 56 coefficients are used.
3. Acoustic modeling for LID
Language recognition can be seen as a classification problem
with each language representing a class. Many classifier designs
exist in the machine learning literature for high-dimension vector
classification. In this work, we used three conventional techniques,
namely HMM, GMM and ANN. We now describe the three classi-
fiers in more detail.
3.1. Hidden Markov model
The hidden Markov model (HMM) consists of two stage stochas-
tic processes: One is an unobservable Markov chain with a finite
number of states, an initial state probability distribution and a
state transition probability matrix, and the other is a set of proba-
bility density functions associated with each state. The probability
density function can be either discrete (discrete HMM) or continu-
(6) ous (continuous HMM) [20]. The continuous HMM is characterized
by the following [21]:
• N x , the number of states in the model. We denote the individ-
ual states as s = {s1 , s2 , . . . , s N x }, and the state at time t as qt .
• The state-transition probability distribution A = {ai j }, where
ai j = P [qt +1 = s j |qt = si ], 1  i, j  N x (9)
defines the probability of transition from state si to s j at
time t with the constraint
Nx
a i j = 1, 1  i  Nx (10)
,
j =1
(7) • The observation probability density function in state i, B =
{bi (O)}, where
M

b i (O) = C ik p (O, μik , Σik ), 1  i  Nx (11)
k =1
where C ik is the mixture coefficient for kth mixture compo-
nent in state i. M is the number of components in a Gaussian
mixture model, and p (O, μik , Σik ) is a Gaussian probability
density function with mean (μik ) and covariance (Σik ).
• The initial state distribution π = {πi }, where
πi = P [q1 = si ], 1  i  N x (12)
Given appropriate values of N x , M, A, B, and π , the HMM can be
used as both a generator of observations, and as a model of how
a given observation sequence O = (o1 o2 . . . o T ) was generated by
an appropriate HMM, where T is the length of the observation
sequence. The compact notation λ = (A, B, π ) is used to indicate
(8)
the complete parameter set of the model. A 5 state (N x = 5) left–
right HMM with two mixtures (M = 2) in each state has been
used for modeling each language feature vectors. In the ergodic
HMM, any state can be reached from any other state in a single
step, i.e., ai j > 0 for all i , j.
3.2. Gaussian mixture models
The basis for using Gaussian mixture model (GMM) is that
the distribution of feature vectors extracted from an individual’s
speech data can be modeled by a mixture of Gaussian densi-
ties [20]. For an N f dimensional feature vector x, the mixture
density function for class s is defined as
M
  
p x/λs = αis f is (x) (13)
i =1
 S. Jothilakshmi et al. / Digital Signal Processing 22 (2012) 544–553 547

Fig. 3. Calculation of the shifted delta feature vectors.

The mixture density function is a weighted linear combination
of M component unimodal Gaussian densities f is (.). Each Gaus-
sian density function f is (.) is parameterized by the mean vector μis
and the covariance matrix Σ is using
1 curve. This allows a GMM to discriminate between classes that lie
f is (x) = 
(2π ) N f |Σ is |
 
1  T  − 1   of languages are one example of this.
× exp − x − μis Σ is x − μis
2
where (Σ is )−1 and |Σ is | denote the inverse and determinant
of the covariance matrix Σ is , respectively. The mixture weights
M
(α1s , α2s , . . . , αM i =1 αi = 1. Collectively,
s s
) satisfy the constraint
the parameters of the class s model λs are denoted as λs =
{αis , μis , Σ is }, i = 1, 2, . . . , M. The number of mixture components
is chosen empirically for a given data set. The parameters of GMM
are estimated using the iterative expectation-maximization (EM)
algorithm [22].
In language identification task, a GMM is created for each lan-
guage. Under GMM assumption, the likelihood of a feature vector
is represented by a weighted sum of multi-variant Gaussian den-
sity. During the recognition, an unknown speech utterance is rep-
resented by a sequence of feature vectors, and the log-likelihood
produced by the model is calculated.
Because of the quasi-parametric nature of GMMs, arbitrary fea-
ture space segmentations can be modeled to very high accuracy
given enough training data and enough mixture components. With
a GMM, the user is not restricted to specific functional forms, as
in truly parametric modeling. Yet the size of the model only grows
with the complexity of the problem being solved, unlike fully non-
parametric methods. Also, GMMs are capable of modeling the den-
sity of data points along an arbitrary feature space segmentation
on the same feature space curve, but have different densities along
that curve. The acoustic feature vectors of the different phonemes
(14)
For the most part, the phonemes of the languages lie within the
same bounded surface of the feature space, with the differentiat-
ing factor between languages being the distribution of data points
along the feature space surface. While these traits of GMMs are ex-
ceptionally noteworthy, one must be careful to ensure that enough
training data and mixture components are used to accurately de-
scribe the classification boundary in the feature space accurately.
The more the mixture components are used, the more accurate
the model can become. But increasing the number of mixtures also
increases the amount of training data and time required for pro-
cessing. Such trade-offs should be considered when using GMMs.
In this experiment, 256 mixtures are used.
3.3. Artificial neural network
Artificial neural networks are capable of learning complex map-
pings between inputs and outputs, and are particularly useful
when the underlying statistics of the considered task are not well-
understood. A multi-layer perceptron typically has three layers of
548 S. Jothilakshmi et al. / Digital Signal Processing 22 (2012) 544–553
Fig. 4. Effect of acoustic features for various test utterance duration. (a) 3 s. (b) 10 s. (c) 30 s.
nodes [23,24]. The input layer takes feature vectors as inputs. Each
node in the output layer corresponds to a language. Therefore, we
have as many nodes in the first layer as the dimensionality of the
feature vector. The non-linear functions at the hidden and output
nodes are sigmoid. For an L c -class LID task, we have L c nodes in
the output layer. The number of nodes in the hidden layer is cho-
sen in an empirical manner to yield a good compromise between
generalization and regression of the training data. During training,
the output node target corresponding to the language of the input
vector is set to one, and the other output node targets are set to
zero. The training can be done by an error back-propagation algo-
rithm.
4. LID system
In this work, a hierarchical language identification system has
been proposed for Indian languages. The languages of India be-
long to four major linguistic families namely Indo Aryan, Dravid-
ian, Austro-Asiatic and Tibeto-Burman [25]. The largest of these in
terms of speakers is Indo Aryan which is spoken by 75.28% of the
people. The second largest is the Dravidian family which is spo-
ken by 22.5% of the people whereas Austro-Asiatic is spoken by
1.13% of the people and Tibeto-Burman by 0.97% [26]. So, nearly
98.0% of the people in India speak languages from Aryan fam-
ily and Dravidian family. Hence, the proposed two level system
is designed to identify the languages of the Aryan and Dravidian
families. In the first level, the system identifies the family of the
spoken language, and then it is fed to the second level which aims
at identifying the particular language in the corresponding fam-
ily. The acoustic systems are an interesting compromise between
complexity and performance. This work explores the use of differ-
ent types of acoustic features for Indian languages identification
for three types of classifiers namely HMM, GMM and ANN.
5. Experimental results
All the experiments described in this paper were conducted on
our own database. It comprises broadcast news in 9 languages. In
the Dravidian family, all four languages namely Tamil (Ta), Telugu
(Te), Kannada (Ka) and Malayalam (Mal) languages are used. In the
Aryan family, the major languages namely Hindi (Hi), Bengali (Be),
Marathi (Ma), Gujarati (Gu), and Punjabi (Pu) are selected. This
database contains a total of 9 hours of broadcast data from Do-
ordarshan television network which is broadcasting programs in
all regional languages of India.
While training the models, it is important to be consistent with
the amount of training data used for training. For the majority of
the experiments conducted, training feature vectors for each lan-
guage to be modeled are randomly selected as the training set. The
testing and training sets for all of the experiments presented here
are mutually exclusive, and approximately equal in size.
Prior to automatic LID, the speech signals are first pre-processed
by removing the silences using short term energy function. This
system consists primarily of three steps: Feature extraction, Family
identification, and Language identification. In the first step, from
the speech data of all the languages under consideration, the fea-
tures are extracted from all the frames, with a frame duration of
20 ms and frame-shift of 10 ms.
Three key topics are addressed by conducting a systematic se-
ries of experiments. For the topic of feature extraction, we studied
the discriminative power of the system for the features MFCC,
MFCC with delta and acceleration coefficients and shifted delta
cepstral coefficients. Then the LID performance as a function of the
different training and testing set sizes has been studied. Finally the
performance of the proposed approach is compared for different
modeling techniques namely HMM, GMM and neural networks.
5.1. HMM based LID system
When modeling an acoustic system with HMMs, the model
size is dictated by two major settings, the number of states and
the number of Gaussian mixtures per state. For our experiments,
5 state HMMs with 2 mixtures/state are chosen. Separate HMM
models are created for the Aryan and Dravidian families and for
each language.
The features used for comparison are 19 MFCC coefficients, 39
MFCC with delta and acceleration coefficients and 7 MFCC coef-
ficients concatenated with 49 SDC coefficients. The usual practice
is to use SDC coefficients without concatenating with the original
cepstral features. Doing so, however, leads to significant degrada-
tion in performance. Figs. 4(a), (b) and (c) show the effect of dif-
ferent features on the performance of the HMM based system. The
results show that the MFCC with delta and acceleration coefficients
and SDC with MFCC coefficients are the best acoustic features to
classify the Indian languages. This is due to their ability to incor-
porate temporal information. The highest performance obtained by
MFCC features is 46.45% and MFCC with delta and acceleration co-
efficients is 50.56%, whereas 48.44% is obtained by SDC features.
It is well known that the size of the training set often greatly
affects the performance of a pattern classification system. The ex-
periments are carried out with various speech segments for each
language to study the effects of training set size. The training set
is randomly selected for each language. The aim of this work is to
design a less complex acoustic system for Indian language identi-
fication. Hence, the performance of the system with 200 seconds,
 S. Jothilakshmi et al. / Digital Signal Processing 22 (2012) 544–553 549

Fig. 5. Effect of training set size for various features. (a) MFCC features. (b) MFCC with delta and acceleration features. (c) SDC features.

Fig. 6. Effect of test utterance duration for various training sets. (a) 200 s. (b) 300 s. (c) 400 s.

Table 1
Language-wise performance of the HMM based LID system (in %).

Be Gu Hi Ma Pu Ka Mal Ta Te Average
HMM based 30.0 35.0 20.0 35.0 40.0 100 50.0 85.0 60.0 50.56

300 seconds and 400 seconds of speech segments from each lan- Table 2
guage has been studied. The variation in the performance of MFCC, False rejection (FR) and false acceptance (FA) (in %).
MFCC with delta and acceleration and SDC features for these train- Language False rejection False acceptance
ing sets is analyzed. The average LID error rate as a function Be 70.0 0.0
of the training data size for HMM based classifier is shown in Gu 65.0 0.0
Figs. 5(a), (b) and (c). The results show that for all the features, Hi 80.0 2.5
the average error rate reduces by increasing the training set size. Ma 65.0 1.9
Pu 60.0 1.9
Selecting segments that have enough information for identify-
Ka 0.0 4.4
ing the language is clearly needed to achieve better performance. Mal 50.0 9.4
As the duration of the test segments varies greatly, an analysis has Ta 15.0 28.13
been made on the performance of this LID system with different Te 40.0 7.5
durations of test signals. The results are shown in Figs. 6(a), (b)
and (c). As expected, the performance of the system improves
• Performance for the Dravidian family languages is good when
when using longer segments. The improvement is clearly signifi-
compared to Aryan family.
cant when selecting segments of duration above 20 seconds.
• Malayalam and Tamil (7th and 8th rows in Table 2 and 7th
The best performance of the LID using HMM based acoustic
and 8th column in Table 3) are severely biased.
system is given in Table 1. This is achieved for MFCC with delta
and acceleration coefficients. The training set size is 400 seconds • Even though the performance for the language Kannada is
for each language and the test utterance duration is 30 seconds. good, it is biased and other languages are severely confused
The average performance is severely affected by the poorer perfor- with this.
mance of the Aryan languages.
The observations made on the false acceptance and false rejec- The family-wise performance of HMM based system for various
tion analysis (Table 2) and the confusion matrix (Table 3) show features is given in Table 4. The observations show that the HMM
that there is a strong bias with some of the languages. The specific system is performing well for the Dravidian family when compared
observations are given below: to Aryan family.
550 S. Jothilakshmi et al. / Digital Signal Processing 22 (2012) 544–553

Fig. 7. Effect of acoustic features for various test utterance duration. (a) 3 s. (b) 10 s. (c) 30 s.

Table 3 MFCC with delta and acceleration coefficients is 80.56% whereas

Confusion matrix (LID system using HMM). Rows correspond to the actual class of 78.59% is obtained by SDC features.
the test data, columns to the assigned class of the test data.
As discussed in Section 5.1, we proceed with various speech
Be Gu Hi Ma Pu Ka Mal Ta Te segments for each language to study the effects of training set size
Be 3 6 0 0 0 3 3 0 5 with GMM classifier. The performance of the GMM based LID sys-
Gu 0 0 3 7 0 7 3 0 0 tem with 200 seconds, 300 seconds and 400 seconds of speech
Hi 0 0 0 0 4 8 0 4 4 segments from each language has been studied. The variation in
Ma 0 0 7 0 0 7 3 3 0
Pu 8 0 0 0 0 6 0 0 6
the performance of MFCC, MFCC with delta and acceleration and
Ka 0 0 0 0 0 20 0 0 0 SDC features for these training sets is analyzed. The average LID
Mal 0 0 0 0 0 0 10 10 0 error rate as a function of the training data size for GMM based
Ta 0 0 0 0 0 0 0 17 3 system is shown in Figs. 8(a), (b) and (c). The results show that for
Te 0 0 4 0 0 0 0 4 12
all the features, the average error rate reduces by increasing the
training set size.
Table 4 As the duration of the test segments varies greatly, an analysis
Family-wise average performance of HMM based system (in %). has been made on the performance of this GMM based LID system
Features Aryan Dravidian with different durations of test signals. The results are shown in
Figs. 9(a), (b) and (c). As expected, the performance of the system
MFCC 27.0 39.03
MFCC + D + A 32.0 73.75 improves when using longer segments. The improvement is clearly
SDC 36.66 55.0 significant when selecting segments of duration above 20 seconds.
The best performance of the LID using GMM based acoustic sys-
tem is 80.56% which is given in Table 5. This is achieved for MFCC
5.2. GMM based LID system with delta and acceleration coefficients. The training set size is 400
seconds for each language and the test utterance duration is 30
This section presents and discusses the experimental data that seconds.
are collected on the GMM based LID task. When modeling an The observations made on the false acceptance and false rejec-
acoustic system with GMMs, the model size is dictated by the tion analysis (Table 6) and the confusion matrix (Table 7) show
number of Gaussian mixtures. In this work, 256 number of mix- that there is a strong bias with some of the languages. The specific
tures are used for the number of heuristic reasons. observations are given below:

• It gives a reasonable and consistent estimate for the number
of phonemes that can be expected in any given language.
• Keeping the number of mixtures low allows faster algorithm
speed and GMM training.
• It reduces memory constrains on the system as compared to
using higher number of mixture components.
• It requires less training data to obtain accurate results than
using higher mixture orders.
As discussed in Section 5.1, this section compares the different
types of features namely MFCC, MFCC with delta and acceleration
and SDC features for language identification with Gaussian mixture
models. The features used for comparison are 19 MFCC coefficients,
39 MFCC with delta and acceleration coefficients and 7 MFCC co-
efficients concatenated with 49 SDC coefficients. Figs. 7(a), (b)
and (c) show the effect of different features on the performance
of the GMM based system. The results show that the MFCC with
delta and acceleration coefficients are performing well with GMM.
The highest performance obtained by MFCC features is 65.15% and
• Performance for the Dravidian family languages is good when
compared to that for the Aryan family.
• Bengali, Tamil and Telugu (1st, 7th and 8th rows in Table 6
and 1st, 7th and 8th column in Table 7) are severely biased.
• Even though the performance for the languages Gujarati and
Kannada is good, they are not biased and other languages are
not that much confused with these languages.
The best family-wise performance of GMM based system for
various features is given in Table 8. The experiments show that
the performance of MFCC with delta and acceleration coefficients
and SDC features is good. The observations show that the system
is performing well for the Dravidian family when compared to the
Aryan family.
5.3. Neural networks based LID system
This section presents and discusses the experimental data that
are collected on the neural networks based LID system. The struc-
 S. Jothilakshmi et al. / Digital Signal Processing 22 (2012) 544–553 551

Fig. 8. Effect of training set size for various features. (a) MFCC features. (b) MFCC with delta and acceleration features. (c) SDC features.

Fig. 9. Effect of test utterance duration for various training sets. (a) 200 s. (b) 300 s. (c) 400 s.

Table 5
Language-wise performance of the GMM based LID system (in %).

Be Gu Hi Ma Pu Ka Mal Ta Te Average
GMM based 45.0 100 80.0 80.0 80.0 80.0 100 100 60.0 80.56

Table 6 Table 8
False rejection (FR) and false acceptance (FA) (in %). Family-wise performance of GMM based system (in %).

Language False rejection False acceptance Features Aryan Dravidian

Be 55.0 8.1 MFCC 62.13 68.93
Gu 0.0 0 .0 MFCC + D + A 77.0 85.0
Hi 20.0 0 .0 SDC 73.46 85.0
Ma 20.0 0 .0
Pu 20.0 0 .0
Ka 20.0 0.63 ture of the neural network for MFCC features is 19L u 39N u 29N u
Mal 0.0 1.25 (excluding output layer). The network structure used for MFCC
Ta 0.0 5.0
Te 40.0 6.9
with delta and acceleration coefficients is 39L u 59N u 49N u (ex-
cluding output layer). The structure of the neural network for SDC
features is 56L u 68N u 58N u (excluding output layer) where L u
denotes a linear unit, and N u denotes a non-linear unit. The non-
Table 7 linear units use tanh(s) as the activation function, where s is the
Confusion matrix (LID system using GMM). Rows correspond to the actual class of
activation value of the unit. The back propagation learning algo-
the test data, columns to the assigned class of the test data.
rithm is used to adjust the weights of the network to minimize
Be Gu Hi Ma Pu Ka Mal Ta Te the mean square error for each acoustic feature vector.
Be 9 0 0 0 0 0 0 2 9 The experiments are carried out by varying the size of the
Gu 0 20 0 0 0 0 0 0 0 training set and test utterance duration as done for HMM and
Hi 0 0 16 0 0 0 0 4 0
Ma 1 0 0 16 0 1 0 2 0
GMM based LID systems. From the results it is observed that
Pu 0 0 0 0 16 0 2 0 2 the performance of this system is very poor for all features. The
Ka 4 0 0 0 0 16 0 0 0 highest performance obtained is 11.72%. There is no significant im-
Mal 0 0 0 0 0 0 20 0 0 provement in the performance even though the size of the training
Ta 0 0 0 0 0 0 0 20 0
set and the test utterance duration are varied. Moreover, it takes
Te 8 0 0 0 0 0 0 0 12
longer training time when compared to the training time taken by
552 S. Jothilakshmi et al. / Digital Signal Processing 22 (2012) 544–553
HMM and GMM based systems. The system is heavily biased with
particular languages most of the time. Hence, it is concluded that
the neural network is not a suitable classifier to classify Indian lan-
guages.
5.4. Discussion
Based on the results obtained from this hierarchical LID system,
we can conclude that for identifying Indian languages MFCC with
delta and acceleration coefficients outperforms the other feature
vector types examined. The performance of the SDC features also
is good. But the computational complexity of SDC is high when
compared to MFCC with delta and acceleration coefficients. While
comparing the performance of the models used in this analysis,
GMM is performing better than HMM and neural networks. The
highest performance of GMM based system is 80.56%, HMM based
system is 50.56% and neural network based system is 11.72%. These
results are obtained with limited amount of training data.
A thorough analysis of the error was performed on the above-
described LID methods. It is noticed that the error in identifying
the languages correctly is due to the following reasons.
• The Indian languages are similar in nature.
• The recorded speech signals are news bulletins and are
recorded within a week. As they are recorded from Doordar-
shan network, most of the information is similar and few of
the words are also common for many languages. Hence, the
confusions are more.
6. Conclusion and future work
In this paper, a hierarchical approach is proposed for spoken LID
of Indian languages. This method identifies the family of the lan-
guage and then identifies the particular language in the family. So
the number of comparisons required to identify a language is less.
As this is an acoustic LID system, the complexity of the system is
less. The LID task is carried out using different acoustic features
namely MFCC, MFCC with delta and acceleration coefficients and
SDC features. To identify the best modeling technique to classify
Indian languages, the LID system is developed using HMM, GMM
and neural networks and the results are compared. It is shown that
GMM based LID system using MFCC with delta and acceleration
coefficients is performing well with 80.56% accuracy. The perfor-
mance of GMM based LID system with SDC is also considerable. In
this system, only acoustic features are used. If prosodic features are
combined, better performance can be expected. It is noticed that
the automatic LID system is biased toward some languages. An at-
tempt can be made to address this. In the Aryan family, only 5
languages are considered. Further experiments can be carried out
on a larger set of broadcast sources and more languages like Oriya,
Kashmiri and Urdu.
References
[1] J.-L. Rouas, I. Trancoso, C. Viana, M. Abreu, Language and variety verification on
broadcast news for Portuguese, Speech Commun. 50 (2008) 965–979.
[2] T. Nagarajan, Implicit systems for spoken language identification, PhD thesis,
Department of Computer Science and Engrg., Indian Institute of Technology
Madras, 2004.
[3] M.A. Zissman, Comparison of four approaches to automatic language identifi-
cation of telephone speech, IEEE Trans. Speech Audio Process. 4 (1996) 31–44.
[4] A.K.V.S. Jayaram, V. Ramasubramanian, T.V. Sreenivas, Language identification
using parallel sub-word recognition, in: Proc. of IEEE Int’l Conf. Acoustics,
Speech and Signal Processing, 2003, pp. 32–35.
[5] T. Nagarajan, H.A. Murthy, Language identification using acoustic log-
likelihoods of syllable-like units, Speech Commun. 48 (2006) 913–926.
[6] R. Tong, B. Ma, D. Zhu, H. Li, E. Siong Chng, Integrating acoustic, prosodic and
phonotactic features for spoken language identification, in: Proc. ICASSP, 2006,
pp. 205–208.
[7] R. Ng, T. Lee, C.-C. Leung, B. Ma, H. Li, Analysis and selection of prosodic
features for language identification, in: Proc. of Int’l Conf. Asian Language Pro-
cessing, 2009, IALP ’09, 2009, pp. 123–128.
[8] Y.K. Muthusamy, E. Barnard, R.A. Cole, Reviewing automatic language identifi-
cation, IEEE Signal Process. Mag. (1994) 33–41.
[9] P.A. Torres-Carrasquillo, T.P. Gleason, W.M. Campbell, D.A. Reynolds, E. Singer,
Acoustic, phonetic, and discriminative approaches to automatic language recog-
nition, in: Proc. Eurospeech in Geneva, Switzerland, ISCA, 2003, pp. 1345–
1348.
[10] M.A. Zissman, Automatic language identification using Gaussian mixture and
hidden Markov models, in: Proc. of IEEE Int’l Conf. Acoustics, Speech and Signal
Processing, 1993, pp. 399–402.
[11] L. Wang, E. Ambikairajah, E.H.C. Choi, Multi-lingual phoneme recognition and
language identification using phonotactic information, in: Proc. of IEEE Int’l
Conf. on Pattern Recognition, 2006.
[12] H. Li, B. Ma, C.H. Lee, A vector space modeling approach to spoken language
identification, IEEE Trans. Audio, Speech Language Process. 15 (1) (2007) 271–
284.
[13] K.C. Sim, H. Li, On acoustic diversification front-end for spoken language iden-
tification, IEEE Trans. Audio, Speech Language Process. 16 (2008) 1029–1037.
[14] T. Nagarajan, H.A. Murthy, A pair-wise multiple codebook approach to implicit
language identification, in: Workshop on Spoken Language Processing, TIFR, In-
dia, 2003, pp. 101–108.
[15] P.A. Torres-Carrasquillo, T.P. Gleason, W.M. Campbell, D.A. Reynolds, E. Singer,
Language recognition with support vector machines, in: Proc. Odyssey: The
Speaker and Language Recognition Workshop in Toledo, Spain, ISCA, 2004,
pp. 41–44.
[16] T.P. Gleason, P.A. Torres-Carrasquillo, D.A. Reynolds, Dialect identification using
Gaussian mixture models, in: Proc. Odyssey: The Speaker and Language Recog-
nition Workshop in Toledo, Spain, ISCA, 2004, pp. 297–300.
[17] C.-Y. Lin, H.-C. Wang, Language identification using pitch contour information
in the Ergodic Markov model, in: Proc. of IEEE Int’l Conf. Acoustics, Speech and
Signal Processing, 2006, pp. I-193–I-196.
[18] S.B. Davis, P. Mermelstein, Comparison of parametric representations for mono-
syllabic word recognition in continuously spoken sentences, IEEE Trans. Acoust.
Speech Signal Process. 28 (1980) 357–366.
[19] B. Yin, E. Ambikairajah, F. Chen, Combining cepstral and prosodic features in
language identification, in: Proc. of IEEE Int’l Conf. Pattern Recognition, 2006.
[20] S. Palanivel, Person authentication using speech, face and visual speech, PhD
thesis, Department of Computer Science and Engrg., Indian Institute of Tech-
nology Madras, 2004.
[21] L. Rabiner, A tutorial on hidden Markov models and selected applications in
speech recognition, Proc. IEEE 77 (2) (1989) 257–287.
[22] R. Redner, H. Walker, Mixture densities, maximum likelihood and the EM algo-
rithm, SIAM Rev. 26 (1984) 195–239.
[23] B. Yegnanarayana, Artificial Neural Networks, Prentice Hall, New Delhi, 1999.
[24] V. Ramalingam, A study of advertisement effectiveness using neural networks,
PhD thesis, Annamalai University, Department of Computer Science and Engrg.,
2006.
[25] www.ciil-spokencorpus.net [Online, accessed on 20 January 2009].
[26] http://www.lisindia.net/ [Online, accessed on 1 February 2011].
Dr. S. Jothilakshmi received the B.E. degree in
Electronics and Communication Engineering from
Govt. College of Engineering, Salem in 1994. She re-
ceived the M.E. degree in Computer Science and Engi-
neering from Annamalai University in the year 2005.
She has been with Annamalai University, since 1999.
She completed her Ph.D. degree in Computer Science
and Engineering at Annamalai University in 2011. She
published 6 papers in international journals and con-
ferences. Her research interest includes speech processing, image and
video processing, and pattern classification.
Dr. V. Ramalingam received the M.Sc. degree in
Statistics from Annamalai University in 1980. He re-
ceived the M.Tech degree in Computer Applications
from Indian Institute of Technology Delhi in the year
1995. He has been with Annamalai University, since
1982. He completed his Ph.D. degree in Computer
Science and Engineering at Annamalai University in
2006. He published more than 40 papers in interna-
tional conferences and journals. His research interest
includes image and video processing, natural language processing and
neural networks.
 S. Jothilakshmi et al. / Digital Signal Processing 22 (2012) 544–553 553

Dr. S. Palanivel received the B.E. (Hons) degree in
Computer Science and Engineering from Mookambigai
College of Engineering in 1989. He received the M.E.
degree in Computer Science and Engineering from
Government College of Technology in the year 1994.
He has been with Annamalai University, since 1994.
He completed his Ph.D. degree in Computer Science and Engineering at
Indian Institute of Technology Madras under quality improvement pro-
gramme (QIP) sponsored by Annamalai University, in the year 2005. He
published 26 papers in international journals and 14 in international
conferences. His research interest includes speech processing, image and
video processing, pattern classification techniques.