Available online at www.sciencedirect.

com

Speech Communication 77 (2016) 53–64

www.elsevier.com/locate/specom

Maxout neurons for deep convolutional and LSTM neural networks

in speech recognition
Meng Cai∗, Jia Liu
Tsinghua National Laboratory for Information Science and Technology, Department of Electronic Engineering, Tsinghua University, Beijing 100084, China
Received 1 July 2015; received in revised form 7 November 2015; accepted 7 December 2015
Available online 17 December 2015

Abstract
Deep neural networks (DNNs) have achieved great success in acoustic modeling for speech recognition. However, DNNs with sigmoid neurons
may suffer from the vanishing gradient problem during training. Maxout neurons are promising alternatives to sigmoid neurons. The activation
of a maxout neuron is obtained by selecting the maximum value within a local region, which results in constant gradients during the training
process. In this paper, we combine the maxout neurons with two popular DNN structures for acoustic modeling, namely the convolutional neural
network (CNN) and the long short-term memory (LSTM) recurrent neural network (RNN). The optimal network structures and training strategies
for the models are explored. Experiments are conducted on the benchmark data sets released under the IARPA Babel Program. The proposed
models achieve 2.5–6.0% relative improvements over their corresponding CNN or LSTM RNN baselines across six language collections. The
state-of-the-art results on these data sets are achieved after system combination.
© 2015 Elsevier B.V. All rights reserved.

Keywords: Maxout neuron; Convolutional neural network; Long short-term memory; Acoustic modeling; Speech recognition.

1. Introduction consistent improvements as the DNN acoustic models (Deng

Automatic speech recognition (ASR) is undergoing a very
fast developing period in recent years. The application of deep
neural networks (DNNs) in acoustic modeling is a strong force
in driving this development. Traditionally, acoustic models are
dominated by Gaussian mixture model-hidden Markov mod-
els (GMM-HMMs). The initial success of DNN-based acoustic
models is on the phone recognition task on the TIMIT corpus
(Mohamed et al., 2012). Then DNNs quickly scale up to large
vocabulary continuous speech recognition (Dahl et al., 2012).
Compared to conventional GMM-based systems, the error rates
of DNN-based systems drop by as large as 33% relatively on
the well-known Switchboard phone-call transcription bench-
mark (Seide et al., 2011). These results are soon confirmed and
extended by many research groups (Hinton et al., 2012). Few
technologies in the history of ASR have brought such large and
∗ Corresponding author.
E-mail addresses: cai-m10@mails.tsinghua.edu.cn, caimeng06@foxmail.
com (M. Cai), liuj@tsinghua.edu.cn (J. Liu).
and Huang, 2004).
The most widely used DNN model is the feed-forward DNN
with fully-connected weights and sigmoid neurons. Despite its
simplicity, there are two issues that might be the drawback of
this model. First, the feed-forward fully-connected DNNs do
not have structures that explicitly model the prior knowledge
of speech signals. The prior knowledge includes the local prop-
erties within the speech frames, long-time dependencies among
the speech frames, etc. Second, the sigmoid neurons of the DNN
may cause the vanishing gradient problem during the stochastic
gradient descent (SGD) training (Glorot and Bengio, 2010). This
is because the gradient of the sigmoid function tends to be very
small if its input value is not around zero. The two issues have
attracted attentions of researchers in both the speech community
and the machine learning community.
Previous works on exploiting prior knowledge of speech sig-
nals for DNNs mainly focus on two aspects. The first aspect
is discovering local properties within the speech frames. The
speech signals exhibit some well-known local properties in the
spectrum, such as the speech formants. The convolutional neural
network (CNN) (LeCun et al., 1998) is proved to be successful at

http://dx.doi.org/10.1016/j.specom.2015.12.003
0167-6393/© 2015 Elsevier B.V. All rights reserved.
54 M. Cai, J. Liu / Speech Communication 77 (2016) 53–64
modeling the local structures of the input data. There are three
key features of CNNs. First, the lower layers of the CNN are
split into a set of local receptive fields, so that different spatial
regions of the input data can be processed differently. Second,
the weights connecting the receptive fields in adjacent layers are
often replicated for robust parameter estimation. Third, pooling
layers are often used in conjunction with the convolutional lay-
ers to reduce the signal variations. The CNN has been shown
to outperform DNNs in acoustic modeling, both in small vo-
cabulary tasks (Abdel-Hamid et al., 2012; Toth, 2013) and large
vocabulary tasks (Sainath et al., 2013a, b).
The second aspect of exploiting prior knowledge of speech
signals is utilizing long-time dependencies among the speech
frames. Feed-forward neural networks discover cross-frame in-
formation by concatenating input features in a context window.
However, the length of the context window has to be fixed. Some
works use the recurrent neural networks (RNNs) (Vinyals et al.,
2012; Weng et al., 2014, Saon et al., 2014) to capture longer
context and report better performance than feed-forward DNNs,
especially for noise robust tasks. But the time dependencies
learned by standard RNNs are still limited due to the vanishing
and exploding gradient problem (Bengio et al., 1994). Recent
innovations in acoustic modeling explore the use of long short-
term memory (LSTM) RNNs (Gers et al., 2002). The recurrent
cells in LSTM RNNs are linearly connected in order to solve the
vanishing gradient problem. The input signals, recurrent signals
and output signals in LSTM RNNs are controlled by gate signals
to achieve precise timing. The LSTM RNN-based systems have
achieved state-of-the-art results on the TIMIT phone recognition
task in 2013 (Graves et al., 2013a). Then LSTM RNN-HMM hy-
brid models are applied to large vocabulary tasks (Graves et al.,
2013b). After that, researchers in Google extend the LSTM RNN
acoustic model to big data and achieve impressive results on
voice search tasks (Sak et al., 2014a, 2014b). The good per-
formances of LSTM RNNs on various tasks demonstrate their
ability to learn long-time dependencies from data.
To solve the vanishing gradient problem of the sigmoid neu-
rons, previous solutions mostly fall into two categories. The
first solution is to use pre-training methods for the sigmoid neu-
rons. Existing pre-training algorithms include deep belief net-
work (DBN) pre-training (Hinton et al., 2006), denoising au-
toencoders (Vincent et al., 2008), discriminative pre-training
(Seide et al., 2011), etc. These methods aim at finding a good
initialization for DNNs, so that the final model will have better
convergence with SGD training. However, pre-training methods
introduce more computational cost and they are not directly ap-
plicable to neural network architectures other than feed-forward
DNNs, e.g. CNNs or LSTM RNNs. The other solution to the
vanishing gradient problem is to use neuron types that have con-
stant gradients. Rectified linear unit (ReLU) (Nair and Hinton,
2010; Glorot et al., 2011) is a kind of neuron that produces
constant gradients. The activation function of the ReLU has the
form y = max(x, 0), so its gradient is either 1 or 0, depending
on the input value. Later, the maxout neuron is proposed for
machine learning tasks (Goodfellow et al., 2013). This neuron
type achieves its nonlinearity by selecting the maximum value
within a group of linear pieces, so its gradient is 1 if the piece
has the maximum value and 0 otherwise. The maxout network
has obtained state-of-the-art results in many data classification
tasks (Goodfellow et al., 2013). It has also shown good results
in acoustic modeling for speech recognition (Cai et al., 2013;
Miao et al., 2013).
In this paper, we aim at building DNN structures for acoustic
modeling that make use of prior knowledge in the speech sig-
nals and at the same time solve the vanishing gradient problem.
The solution we propose is to combine the maxout neurons with
both the CNN and the LSTM RNN. The convolutional structure
and the LSTM recurrent structure of the networks capture infor-
mation within and among the speech frames, while the maxout
neurons are able to solve the vanishing gradient problem. The
resulting models are referred to as the convolutional maxout
neural network (CMNN) and the recurrent maxout neural net-
work (RMNN). This work greatly extends our previous works
(Cai et al., 2013, 2014) in four aspects. First, the RMNN model
is proposed to capture long-term dependencies in speech, which
has not been adopted previously. Second, we try various alter-
native network structures and training strategies to explore the
optimal configuration. Third, extensive experiments are carried
out on the Babel benchmark data sets, including six languages.
Fourth, the system combination results for the CMNNs and the
RMNNs are presented, which achieve state-of-the-art perfor-
mances on these data sets.
The remainder of this paper is organized as follows. In
Section 2, a description of the baseline CNN and LSTM RNN
is given. In Section 3, we introduce the CMNN and the RMNN
acoustic models. In Section 4, we give our experimental details
and results. Finally, the conclusions are presented in Section 5.
2. Baseline CNN and LSTM RNN
In this work, the CNN acoustic models and the LSTM RNN
acoustic models are employed as the baseline systems.
2.1. CNN acoustic model
The CNN aims at discovering local structures in the input
data. The speech signal exhibits local similarities in both the
spectral and temporal dimension, making the application of
CNN acoustic models possible. An example of the CNN applied
to speech signals is illustrated in Fig. 1. In the figure, the light
and shade regions along the frequency axis represent the speech
formants. The CNN has a set of locally-connected weights in
the lower convolutional layers to model different regions in the
speech spectrogram.
To apply the CNN acoustic model for speech signals, the
spectral structures of the speech features have to be preserved.
The Mel filterbanks or the power spectrum features (Sainath
et al., 2013) are good choices for the CNN. The input fea-
tures are first divided into N non-overlapping frequency bands
{vi |i ∈ 0, . . . , N − 1}. A group of bands {vi+r |r ∈ 0, . . . , s − 1}
produces the activations hi through the weight connections and
nonlinear transforms. If full weight-sharing is used, then hi is
 M. Cai, J. Liu / Speech Communication 77 (2016) 53–64
Fig. 1. An illustration of the CNN applied to Mel filterbank features of the
speech signals.
computed as
 s−1 
 Fig. 2. An illustration of the LSTM memory block. The red lines stand for
hi = θ WrT · vi+r + b ,
r=0
where {Wr |r ∈ 0, . . . , s − 1} and b are weights and biases repli-
cated in the convolutional layer, θ is the nonlinear transform.
The s is called band width. The r is the index of the bands. The
weights and biases shift d bands every time, producing a group
of activation bands {h j | j ∈ 0 . . . M − 1}. The d is called band
shift, which is typically 1. The meanings of band width s and
band shift d are also illustrated in Fig. 1.
The max-pooling layer is often used after the convolutional
layer to reduce the variance of frequency shift. The form of the
max-pooling is
pmi = max hmj ,
j∈i·k,...,(i+1)·k−1
where pmi is the output of the max-pooling operation, i and j are
the indexes of the bands, m is the index of the neurons in a band
and k is the pooling size.
After several layers of convolution or max-pooling, the bands
are concatenated and fully-connected weights are applied just
like conventional DNNs.
2.2. LSTM RNN acoustic model
RNNs have the advantage of modeling longer context than
DNNs. The speech signals are non-stationary time series, the
context information is important for speech recognition. Con-
ventional RNNs have cyclic connections in the hidden layers
to model temporal correlations. But modeling long-time depen-
dencies using conventional RNN is difficult due to the vanish-
ing and exploding gradients in the SGD training (Bengio et al.,
1994). The LSTM RNN is an elegant solution to model long-time
dependencies.
The architecture of the LSTM RNN is depicted in Fig. 2.
The major difference of the LSTM RNN from conventional
RNN is that the recurrent connections are linear in the LSTM
RNN. This connection is controlled by the signal ft called for-
get gate. Because the recurrent connections in LSTM RNN do
55
(1) the connections with weights. The dashed lines stand for recurrent connections.
(For interpretation of the references to color in this figure legend, the reader is
referred to the web version of this article).
not go through nonlinear transforms, the gradients will be back-
propagated smoothly through the recurrent connections using
SGD. In this way the model can learn long-time dependencies.
In addition to the forget gate, there are also the input gate it
and the output gate ot in the LSTM RNN to control the input and
output signals. Let xt and yt be the input and output signal for an
LSTM network at time t. Let ct and mt be the cell activation and
the output activation, respectively. The forward pass from xt to
yt follows the equations according to Gers et al. (2002), Graves
(2) et al. (2013a) and Sak et al. (2014a):
it = σ (Wix xt + Wim mt−1 + Wic ct−1 + bi ) (3)
ft = σ (W f x xt + W f m mt−1 + W f c ct−1 + b f ) (4)
ct = ft  ct−1 + it  g(Wcx xt + Wcm mt−1 + bc ) (5)
ot = σ (Wox xt + Wom mt−1 + Woc ct + bo ) (6)
mt = ot  h(ct ) (7)
yt = φ(Wym mt + by ) (8)
where Wcx , Wix , Wfx and Wox are the weights connected to
the LSTM inputs, Wcm , Wim , Wfm and Wom are the weights
connected to the LSTM activations, bc , bi , bf and bo are the
biases, Wic , Wfc and Woc are diagonal peephole connections
from the cell to the gate signals. Wym and by are the weights
and biases in the final output layer. The σ , g and h are nonlinear
functions. The sigmoid function is typically used for σ and the
tanh function is typically used for g and h. The  is element-
wise product of the vectors and the φ is the softmax function in
the output layer.
The standard LSTM RNN has one hidden layer. Later, deep
LSTM RNNs are proved to outperform the standard LSTM RNN
for speech recognition (Graves et al., 2013a; Sak et al., 2014a).
56 M. Cai, J. Liu / Speech Communication 77 (2016) 53–64
Fig. 3. An illustration of the maxout neurons.
Moreover, linear recurrent projection layers are found to be use-
ful for reducing the number of parameters at no loss of accu-
racy (Sak et al., 2014a). The resulting model is referred to as
deep LSTMP (long short-term memory projected) (Sak et al.,
2014a). The LSTMP architecture is employed in the baseline
LSTM RNN in this paper.
3. CMNN and RMNN
In this section we extend the exploration of the CMNN (con-
volutional maxout neural network) in our previous work (Cai
et al., 2014). We also propose the new RMNN (recurrent max-
out neural network).
3.1. Maxout neurons
The maxout neurons are promising alternatives to the sigmoid
neurons. Conventional sigmoid neurons, though most widely-
used in DNNs, may cause the vanishing gradient problem during
SGD training. The maxout neurons effectively deal with the van-
ishing gradient problem by producing constant gradient during
SGD training.
The structure of the maxout neuron is illustrated in Fig. 3.
Each maxout neuron consists of several pieces of alternative
activations. The output of a maxout neuron is the maximum
value within its piece group, i.e.,
hli = max zli j ,
j∈1,...,k
where hli is the i-th maxout neuron output in the l-th layer, k
is the number of pieces for the maxout neurons, and zli j is the
j-th piece activation of the i-th neuron in the l-th layer. The
piece activations zl are connected directly to the neurons of
the previous layer through the weight matrix Wl and the bias
vector bl :
zl = WlT hl−1 + bl .
Neither the computation of zl nor the computation of hl includes
nonlinear transforms such as sigmoid or tanh. The nonlinearity
of the maxout neuron is achieved by the maximum value selec-
tion process as in Eq. (9). This nonlinearity can also be viewed
as a feature selection process.
In the SGD training stage, the gradient for the maxout neuron
is computed as

∂hli 1 if zli j ≥ zlis , ∀s ∈ 1, . . . ,k
= (11)
∂zli j 0 otherwise
This equation shows that the gradient of the maxout neuron is 1
for the piece with the maximum activation, and 0 otherwise. By
producing constant gradients during SGD training, the vanishing
gradient problem is naturally solved in the maxout networks.
Therefore the optimization of deep maxout neural networks is
easier compared to conventional sigmoid neural networks.
3.2. Dropout training
Because the maxout neurons have better optimization per-
formance, they can effectively deal with the problem of un-
derfitting. However, neural networks with maxout neurons are
sometimes prone to overfitting. Dropout (Srivastava et al., 2014)
is an effective regularization method to control the overfitting
problem of neural networks. Dropout is also particularly effec-
tive for the maxout networks because of better model averaging
(Goodfellow et al., 2013).
The dropout regularization includes different strategies for
the training phase and the testing phase. During the SGD training
phase, the activation hl is obtained by applying a binary mask
ml to the original activations:
hl = ml  θ (WlT hl−1 + bl ), (12)
where θ stands for the neuron nonlinearity such as sigmoid or
maxout,  is element-wise product of the vectors. The elements
of the binary mask ml obeys the distribution Bernoulli(1 − r).
The hyperparameter r is the ratio of the number of neurons to be
omitted, which is often called the dropout rate. Lower dropout
rate preserves more information, while higher dropout rate per-
forms more aggressive regularization.
During the dropout testing phase, the neuron activations are
no longer omitted. But the neuron activations are scaled down
by the coefficient 1 − r to compensate for the dropout training.
This strategy is a simple way to perform model averaging (Li
et al., 2013) and to improve the model generalization ability.
In this paper, we would like to explore the optimal dropout
strategy for the CMNN and the RMNN.
(9)
3.3. CMNN
Compared with fully-connected DNNs, CNN acoustic mod-
els have the advantage of discovering local spectral characteris-
tics within speech frames. However, the CNNs may still suffer
from the vanishing gradient problem during SGD training. A
natural way to deal with the vanishing gradient problem for the
CNN is to apply the maxout neurons. The goal of the CMNN
(10) (convolutional maxout neural network) is to combine the advan-
tages of both the CNN and the maxout neurons.
In the CMNN, the convolutional structure, the max-pooling
operation and the higher fully-connected layers are preserved
from CNNs. But all the sigmoid neurons are replaced by the max-
out neurons. The lower convolutional layers learn local proper-
ties of speech signals, while the maxout neurons provide better
 M. Cai, J. Liu / Speech Communication 77 (2016) 53–64
Fig. 4. An illustration of maxout and max-pooling applied to the convolutional
layers of the CMNN.
optimization performance. Fig. 4 illustrates the difference be-
tween the maxout operation and the max-pooling operation. In
the convolutional layers, the maxout operation selects the max-
imum value within a local spatial region in the same band. The
max-pooling is applied after the convolutional operation and the
maxout operation. The output of the max-pooling operation is
the maximum value from different bands.
The combination of CNNs and maxout neurons has been
explored in our previous work (Cai et al., 2014) and several other
works (Toth, 2014; Miao and Metze, 2014). Our models differ
from the models of Miao and Metze (2014) because the maxout
neurons in Miao and Metze (2014) are only applied to the fully-
connected layers in the models, not the convolutional layers. In
Toth (2014), the models are applied to the phone recognition task
on the TIMIT corpus. While in this work, we apply the CMNN
to the large vocabulary speech recognition tasks. The CMNN in
this work also extends our previous work in Cai et al. (2014) as
we explore the optimal dropout strategy for the CMNN. We also
compare the CMNN models with the model structures in Miao
and Metze (2014) in the experiments.
3.4. RMNN
The LSTM RNN acoustic model is good at modeling long-
time dependencies of the speech signals. When unfolded in time,
the LSTM RNN has deep structures in both the time domain
and the spatial domain. In the time domain, the LSTM RNN can
be viewed as a very deep neural network (the number of lay-
ers is the same as the number of speech frames) with shared
weights. The LSTM RNN deals with the vanishing gradient
in the time domain by using linear recurrent connections with
gate signals. Thus the history states of the LSTM can have a
long-term impact. However, the LSTM RNN does not explic-
itly deal with the vanishing gradient problem in the spatial do-
main. In Sak et al. (2014a), it is discovered that the optimal
57
Fig. 5. The structure of the RMNN.
performance is achieved by an LSTM RNN with 2 or 3 hidden
layers. But the optimal number of hidden layers are more for
DNNs than LSTM RNNs, typically 7 for the Switchboard task
(Seide et al., 2011). The optimal numbers of hidden layers for
the maxout networks are even more (Goodfellow et al., 2013;
Cai et al., 2013). Because the maxout neurons are good solutions
to the vanishing gradient problems for DNNs, it is intuitively
beneficial to increase the depth of the LSTM RNN acoustic mod-
els using the maxout neurons.
Our proposed RMNN (recurrent maxout neural network) is
a hybrid deep model combining the LSTM RNN and the max-
out neurons. Its structure is illustrated in Fig. 5. The LSTMP
structure (Sak et al., 2014a) is used in the lower layers of the
RMNN to model long-time dependencies of the input signals.
The fully-connected layers with maxout neurons are used af-
ter several LSTMP recurrent layers. Finally the softmax layer
produces the output.
Besides the increased network depth, another benefit of the
RMNN is that dropout training can be applied to the max-
out neurons. Traditionally, the dropout regularization is mostly
used with feed-forward neural networks. The work in Zaremba
et al. (2014) has discovered that dropout can be effective for
RNNs when it is only applied to the non-recurrent connec-
tions. But applying the dropout regularization to the recurrent
connections is inappropriate because the dropout regulariza-
tion tends to damage the long-term memories learned by the
RNNs. This discovery inspires us to explore the dropout reg-
ularization to the maxout neurons in the fully-connected part
of the RMNN, as the maxout neurons perform better model
averaging in conjunction with dropout training. This is the
first time the maxout neurons and the dropout training strat-
egy are used for LSTM RNN models in speech recognition
tasks.
4. Experiments
In this section, we present the detailed experiments for the
CMNN and the RMNN acoustic models.
58 M. Cai, J. Liu / Speech Communication 77 (2016) 53–64

Table 1
The corpus size and characteristics of the languages.

Language Release ID Training hours Dev hours Characteristics

Cantonese IARPA-babel101b-v0.4c 140.7 10.0 With 6 tones.
Pashto IARPA-babel104b-v0.4bY 77.3 10.1 Arabic language. Used in Afghanistan and
Pakistan. With primary and secondary stress.
Turkish IARPA-babel105b-v0.4 76.3 10.0 Arabic language. With primary stress.
Tagalog IARPA-babel106-v0.2g 83.7 9.9 Used in Philippines.
Vietnamese IARPA-babel107b-v0.7 87.1 10.0 With 6 tone.
Tamil IARPA-babel204b-v1.1b 62.3 10.1 Used in India, Sri Lanka and Singapore.

4.1. Data corpus Table 2

The baseline configurations of the languages. #: number of. AM: acoustic model.
LM: language model.
This effort uses language collections released under the
IARPA Babel Program. We experiment with 6 different lan- Language Vocabulary size # phonemes # AM states LM perplexity
guages. These languages include Cantonese, Pashto, Turkish, Cantonese 25,416 217 4610 127.6
Tagalog, Vietnamese and Tamil. The FullLP training sets are Pashto 22,484 127 4940 159.7
used for all these languages. Most of the training audio is con- Turkish 48,914 84 4866 431.6
Tagalog 26,319 139 4894 157.3
versational telephone speech, while some of the training audio Vietnamese 6,961 238 4647 134.8
is scripted read-style speech. All the development audio is tele- Tamil 64,601 34 4630 883.6
phone speech. We only use the telephone speech for training,
as adding scripted training data is found to degrade the perfor-
mance. The corpus size and characteristics of the languages are
listed in Table 1. dow of 11 frames (5 on the left and 5 on the right) is used for
the DNN input features. So the dimension of the DNN input
4.2. Baseline setup layer is 1320. The alignments for the DNN training are gener-
ated by the GMM-HMM with SAT and BMMI discriminative
The pronunciation lexicons are processed based on the re- training. The dimension of the DNN output layer are the same
leased lexicons and the language specific peculiarities (LSP) as the number of acoustic model states shown in Table 2. The
documents. The phonemes with tonal markers are used for Can- baseline DNN-HMM acoustic models have feed-forward fully-
tonese and Vietnamese. The phonemes with stress markers are connected structures with sigmoid neurons. We use 6 hidden
used for Pashto and Turkish. Trigram language models (LM) layers with 1500 neurons per hidden layer. The networks pa-
are trained using the transcriptions of the FullLP training packs rameters are initialized using the scheme introduced in Glorot
with modified Kneser–Ney smoothing. and Bengio (2010). DBN pre-training is then performed for 3
The baseline acoustic models are built with the Kaldi toolkit epochs. The SGD training is used to optimize the network pa-
(Povey et al., 2011). The training procedure are the same for rameters based on the cross entropy criterion. We use a batch size
all the six languages. The GMM-HMM acoustic models are of 128 and an initial learning rate of 0.000625 per frame. The
first trained using 13-dimensional PLP features concatenated network is trained for 20 epochs. The learning rate is halved at
with 3-dimensional pitch features (Ghahremani et al., 2014). the end of an epoch if the frame accuracy on the development set
The features are extracted with a frame length of 25 ms and a decreases. The momentum value is set to 0.0 for the first epoch
frame shift of 10 ms. After mean normalization per conversa- and 0.9 for the rest of the epochs. We use the Kaldi decoder with
tion side, 9 consecutive features frames are concatenated and the the beam set to 11.0 and the acoustic weight set to 0.083333
feature dimension is reduced to 40 by LDA plus a global semi- in all of our evaluation experiments. The performances of the
tied covariance transform. Next, the GMM-HMMs are trained by models are measured by token error rates (TER), which is char-
speaker adaptive training (SAT) based on the feature-space max- acter error rate (CER) for Cantonese, syllable word error rate
imum likelihood linear regression (fMLLR) transforms for the (SyllableWER) for Vietnamese and word error rate (WER) for
LDA features. The GMM-HMMs are further enhanced by dis- Pashto, Turkish, Tagalog and Tamil. The results of the baseline
criminative training using the boosted maximum mutual infor- models are given in Table 3. These results are comparable with
mation (BMMI) criterion. Approximately 5000 acoustic states
and 75,000 Gaussian mixtures are used for the GMM-HMMs.
Table 3
Some baseline configurations, including the vocabulary size, the
The token error rates (TER) (%) of the baseline models for all the languages.
number of phonemes, the number of acoustic model states and ML: maximum likelihood. BMMI: boosted maximum mutual information.
the language model perplexity, are given in Table 2.
Model Cantonese Pashto Turkish Tagalog Vietnamese Tamil
The features for the DNN-HMM acoustic models are 40-
dimensional Mel filterbank features concatenated with first- and GMM-HMM ML 51.6 59.4 58.0 59.7 60.6 72.9
second-order derivatives. The features are normalized to zero GMM-HMM BMMI 47.7 56.0 53.9 54.2 57.0 70.2
DNN 44.8 51.2 47.6 49.8 53.1 66.7
mean and unit variance per conversation side. A context win-
 M. Cai, J. Liu / Speech Communication 77 (2016) 53–64
the results published in various other works (Cui et al., 2013;
Karafiat et al., 2013; Tsakalidis et al., 2014; Chen et al., 2014).
4.3. Results of the CMNN
The results of the CMNN acoustic models are presented in
this section. We first explore different network structures and
training strategies for Vietnamese. Then we give the results using
the optimal strategy for all the languages.
4.3.1. The optimal dropout strategy
In our previous work, it is discovered that the CMNN models
trained in conjunction with the dropout strategy produce sig-
nificant gains compared to the baseline CNN models. It is also
shown that the dropout regularization is effective when it is only
applied to the fully-connected hidden layers of the CMNN (Cai
et al., 2014). In this work, we take a step further to explore the
optimal dropout strategy for CMNN. More specifically, dropout
for all or part of the fully-connected hidden layers of the CMNN
are tried and the results are compared.
We first train a standard CNN model as the baseline for the
CMNN models. The CNN model contains two convolutional
hidden layers with 768 filters, one max-pooling hidden layer
with a pooling size of 3 between the convolutional layers, and 3
fully-connected hidden layers with 1500 nodes. The input fea-
tures for the CNN are the same as those for the DNN, namely 40-
dimensional Mel filterbanks with delta and delta-delta deriva-
tives. A context window of 11 frames is applied. So the input
feature is divided into 40 bands with 33 feature maps per band.
The filters in the first convolutional layer have an input band
width of 8 and an input band shift of 1. The filters in the second
convolutional layer have an input band width of 4 and an input
band shift of 1. We use an initial learning rate of 0.00078125
per frame and a batch size of 256. The CNN is trained for 20
epochs. The momentum value increases from 0.0 to 0.5 linearly
with every epoch.
The CMNN acoustic model is configured to have approxi-
mately the same number of parameters as the CNN model. The
first convolutional operation has an input band width of 8, an
input band shift of 1 and 512 filters. Then a maxout operation
with 2 pieces is applied to reduce the number of feature maps to
256 in each band. A max-pooling layer with a pooling size of 3
follows the maxout operation. Another convolutional operation
is applied on top of the max-pooling layer, which has an input
band width of 4, an input band shift of 1 and 512 filters. The
second convolutional hidden layer is also followed by a max-
out operation with 2 pieces. Finally, 5 fully-connected hidden
layers with 1000 maxout neurons and 2 pieces per neuron are
applied, followed by the softmax output layer. Hence there are
totally 6 fully-connected hidden layers in the CMNN. The initial
learning rate for the CMNN is set to 0.0003125 per frame. The
momentum increases linearly from 0.0 to 0.5.
We try to apply dropout training to different parts of the fully-
connected layers of the CMNN to explore the optimal dropout
strategy. Heuristically, it may be considered beneficial to apply
dropout to the upper layers of the CMNN, as the neurons in the
upper layers are more discriminative. We try to apply dropout
59
Table 4
TERs (%) of different dropout strategies for the CNN models and the CMNN
models for Vietnamese.
Model Dropout layer TER
CNN None 52.5
CNN Top 2 52.4
CNN Top 4 51.8
CMNN None 53.8
CMNN Top 2 52.0
CMNN Top 4 50.6
CMNN Top 6 49.9
to the top 2, the top 4 and the top 6 hidden layers of the CMNN.
A dropout rate of 0.2 is used. The CMNNs are trained for at
most 25 epochs. The results are shown in Table 4. The results of
dropout training for the CNN models are also shown in Table 4,
for which we try to apply dropout to the top 2 and the top 4
hidden layers. However, the CNN models with dropout training
requires up to 40 epochs to converge, making their training time
much longer than the CMNN models. Detailed comparisons of
the training time are given in Section 4.5.
The results in Table 4 show that for both the CNN and the
CMNN, the optimal dropout strategy is to apply dropout to all
the fully-connected hidden layers. The CNN model with the
optimal dropout strategy brings 1.3% relative improvement over
the CNN without dropout, while the CMNN model with the
optimal dropout strategy brings 7.2% relative improvement over
the CMNN without dropout. It is also shown that the TER of the
best CMNN model with dropout training is 3.7% relatively lower
than the best CNN, and is 5.0% relatively lower than the baseline
CNN without dropout training. These results demonstrate the
effectiveness of dropout training for the CMNNs.
4.3.2. Maxout neurons for the convolutional layers
The CMNN models in the previous experiments use max-
out neurons for both the convolutional and the fully-connected
hidden layers. In Miao and Metze (2014), another method to
combine the CNN and the maxout neurons is used, in which
sigmoid neurons are used for the convolutional hidden lay-
ers, and maxout neurons are used for the fully-connected hid-
den layers. In this experiment we compare the CMNN models
with maxout neurons or sigmoid neurons for the convolutional
layers.
The structure of the CMNN with sigmoid convolutional
layers is configured as follows. The first convolutional operation
has an input band width of 8, an input band shift of 1 and 768
filters. Then the max-pooling with a pooling size of 3 is applied.
The sigmoid nonlinearity is used for the output of the max-
pooling layer. Another convolutional operation is applied after
the sigmoid nonlinearity, which has an input band width of 4, an
input band shift of 1 and 384 filters. The sigmoid nonlinearity is
also applied to the output of the second convolutional operation.
Then 5 fully-connected hidden layers with 1000 maxout neurons
and 2 pieces per neuron are applied before the softmax output
layer. The hyperparameters for SGD training is the same as the
previous CMNN, namely an initial learning rate of 0.0003125
per frame and a momentum value increasing from 0.0 to 0.5.
60 M. Cai, J. Liu / Speech Communication 77 (2016) 53–64

Table 5 4.4.1. The optimal network structure

A comparison of the CMNN models with sigmoid or maxout neurons for the Since the RMNNs are deep in both the time domain the spatial
convolutional layers. The TERs (%) are for Vietnamese.
domain, there are different structures that may have an impact
Model Convolutional layer type TER on the performance of the models. For example, the number
CMNN Sigmoid 50.9 of recurrent hidden layers may affect the ability of the mod-
CMNN Maxout 49.9 els to learn temporal relationships, and the number of fully-
connected hidden layers may affect the ability of the models
to learn feature transforms. It is also interesting to compare the
We also use dropout training for all the 6 fully-connected results of the RMNNs and the results of deep LSTMP RNNs
hidden layers, as it is found to be optimal for CMNN. Training with extra fully-connected sigmoid or ReLU neurons, which are
lasts for 25 epochs. alternative model structures that are deeper than the LSTMP
The results for the CMNN with sigmoid or maxout convo- RNNs.
lutional layers are shown in Table 5. These results show a per- To give a solid performance comparison, we first train a stan-
formance gap between the CMNN with sigmoid convolutional dard LSTMP RNN as in Sak et al. (2014a). The model has 3
layers and the CMNN with maxout convolutional layers. The LSTMP layers, 1024 cells per layer and a linear projection of
reason for the superior performance of the maxout neurons is 512 dimensions for each layer. To keep the feature type con-
that the maxout neurons are easier to optimize than the sigmoid sistent with previous models, we use a single frame of Mel
neurons. The constant gradients of the maxout neurons during filterbank feature with 120 dimensions (40-dimensional filter-
SGD training is effective to deal with the vanishing gradient bank with delta and double delta) as the input to the LSTMP
problem, especially for the lower layers of the neural networks. RNN. The target label is delayed for 5 frames, i.e., the infor-
mation of previous feature frames and 5 future feature frames
4.3.3. Results for all the languages is used to predict the current label. Training is performed us-
After explorations conducted for Vietnamese, we now give ing truncated backpropagation through time (BPTT) algorithm.
the results for all the 6 languages. The network structures and the A fixed subsequence length of 20 frames is used in the BPTT
training process are the same as those mentioned in the previous training. A stream of 20 subsequences from different utterances
experiments. The CMNN models are trained with dropout for are processed in parallel for efficient GPU computation. Af-
all the fully-connected hidden layers. The results are shown in ter the training of a subsequence from an utterance finishes,
Table 6. The results for the CNN models with or without dropout the initial status of the LSTM is set to the last status of the
training are both presented. finished subsequence if the next subsequence begins from the
The results in Table 6 show that the CMNN models outper- same utterance, or is reset if a subsequence begins from a new
form the CNN models across a variety of different languages. utterance. We use an initial learning rate of 0.0008 per frame.
Compared with the baseline CNN models, the relative improve- The momentum increases linearly from 0.0 to 0.5 with every
ments of the CMNN models are between 4.3% and 6.0%. The training epoch. The LSTM RNN is trained for a maximum of
results also show that the CMNNs with maxout convolutional 20 epochs.
layers have better performances than the CMNNs with sigmoid We try different RMNN structures that vary in the number
convolutional layers, confirming the discoveries in the previous of recurrent hidden layers and the number of fully-connected
experiments. maxout hidden layers. Two RMNNs are trained in our experi-
ments. One has 1 LSTMP recurrent layer and 5 maxout layers.
The other has 2 LSTMP recurrent layers and 4 maxout layers.
4.4. Results of the RMNN The LSTMP layers used in the RMNNs have 1024 cells and a
linear projection of 512 dimensions. The maxout layers have
In this section, we present the results of the proposed 1000 neurons with 2 pieces. Like the LSTM RNNs, the input to
RMNN acoustic models. Different model structures are first the RMNN is a single frame of Mel filterbank feature, while the
tried for Vietnamese, then the results for all the languages are training labels are delayed for 5 frames. We also use the subse-
given. quence length of 20 and the streams of 20 subsequences for the
BPTT training. The initial learning rate is set to 0.0003125 per
Table 6 frame and the momentum increases from 0.0 to 0.5. Besides, a
TERs (%) of the CNN models and the CMNN models for all the languages. dropout rate of 0.2 is used for the fully-connected layers. The
The “dropout fc” stands for dropout training for all the fully-connected hidden RMNNs are trained for a maximum of 30 epochs.
layers. The “sigmoid conv” stands for sigmoid neurons for the convolutional
Besides the RMNN, there are other ways to make the LSTMP
hidden layers.
RNNs deep in the spatial domain. We try to create deep RNN
Model Cantonese Pashto Turkish Tagalog Vietnamese Tamil models by adding fully-connected sigmoid or ReLU neurons
CNN 43.6 51.5 48.3 49.6 52.5 67.2 on top of the LSTMP layers of the RNN, as comparisons to
CNN, dropout fc 46.1 52.0 47.6 49.6 51.8 66.0 the RMNN models. We refer to the models as DRNNs (deep
CMNN, sigmoid conv 43.4 50.6 46.7 48.8 50.9 65.1 recurrent neural networks). Three DRNNs are trained in the ex-
CMNN 41.7 49.3 45.4 47.2 49.9 64.2
periments. The first model has 1 LSTMP recurrent layer and 2
 M. Cai, J. Liu / Speech Communication 77 (2016) 53–64 61

Fig. 6. A performance comparison of different acoustic models applied to all the languages.

Table 7
A comparison of RNNs with different structures. The TERs (%) are for Viet-
namese.
Model Structure TER
LSTM 3 LSTMP 47.8
RMNN 1 LSTMP + 5 maxout fc 46.9
RMNN 2 LSTMP + 4 maxout fc 46.0
DRNN 1 LSTMP + 2 sigmoid fc 50.5
DRNN 2 LSTMP + 2 sigmoid fc 50.0
DRNN 1 LSTMP + 5 ReLU fc 47.8
sigmoid layers. The second model has 2 LSTMP recurrent lay-
ers and 2 sigmoid layers. The LSTMP layers have 1024 cells
and a linear projection of 768 dimensions. The sigmoid layers
have 1500 neurons. The hyperparameters for the BPTT training
for these two models are the same as those for the LSTMP RNN
except for an initial learning rate of 0.000625 per frame. The
third model has 1 LSTMP recurrent layer and 5 ReLU layers.
The LSTMP layer has 1024 cells and a linear projection of 512
dimensions. The ReLU layer has 1500 neurons. The hyperpa-
rameters for the BPTT training for this model are the same as
those for the RMNN.
The results of the RNN acoustic models are presented in
Table 7. The results show that the TER of the LSTM model
is 47.8%, which is a very strong baseline. Both the RMNNs
with 1 or 2 recurrent hidden layers achieve better results than
the LSTM. Results also show that the RMNN with 2 recur-
rent hidden layers has the best performance, achieving a TER of
46.0%. This result is 3.8% better than the LSTM relatively. How-
ever, the DRNNs with fully-connected sigmoid neurons do not
achieve gains over the baseline. The performance of the DRNN
with fully-connected ReLU neurons is on par with the baseline
LSTMP RNN. The results indicate that it is not trivial to build
deep RNNs. The better optimization performance of the maxout
neurons and the better model averaging with the dropout training
are beneficial for deep models.
4.4.2. Results for all the languages
The RMNN models give good results for Vietnamese. It is
more convincing to see how the RMNN models perform for other
languages. We conduct experiments for all the 6 languages using
Table 8
TERs (%) of the LSTM, the RMNN and the DRNN models for all the languages.
Model Cantonese Pashto Turkish Tagalog Vietnamese Tamil
LSTM 40.7 50.5 47.4 47.9 47.8 65.0
RMNN 39.0 48.1 44.9 45.7 46.0 63.4
DRNN 45.2 51.3 49.5 50.8 50.0 68.1
the best network structure explored before, i.e., the RMNN mod-
els with 2 recurrent hidden layers and 4 maxout fully-connected
layers. The standard LSTM models with 3 recurrent layers and
the DRNN models with 2 recurrent layers and 2 sigmoid fully-
connected layers are also trained. The hyperparameters for the
BPTT are the same as those in the previous experiments except
for Cantonese, for which we use a dropout rate r of 0.1 as there
is more data. The results are given in Table 8.
The results show that the RMNN models achieve good per-
formances for different languages. The token error rates of the
RMNN models are 2.5–5.3% relatively lower than the LSTM
models. The DRNNs, however, only show degraded perfor-
mances compared to the LSTM models. The experiments for
all the languages demonstrate the effectiveness of the RMNN
models. We also give the performance comparison of differ-
ent acoustic models in Fig. 6. The figure clearly shows that the
proposed RMNN acoustic models achieve the best performance
compared to other models for all the languages, outperforming
the baseline DNNs by 4.9–13.4% relatively.
4.5. Training time
Apart from the token error rates, another matter to take into
consideration is the training time. We use highly efficient GPU
implementations for all the models in our experiments. The train-
ing time per epoch and the number of required epochs for all the
models are given in Tables 9 and 10, respectively. The experi-
ments are carried out on NVIDIA Tesla K20m GPUs.
From the training time presented in Tables 9 and 10, we can
see that the baseline DNN models cost the least training time.
The training time per epoch for the CNN models is between
3 and 4 times slower than the DNNs. This is because the local
weight connections in the convolutional structures require many
62 M. Cai, J. Liu / Speech Communication 77 (2016) 53–64

Table 9
Training time per epoch (hours) of different acoustic models for all the languages.
a
Model Cantonese Pashto Turkish Tagalog Vietnamese Tamil
DNN 2.35 1.32 1.12 1.43 1.22 1.04
CNN 7.07 4.61 3.83 4.98 4.22 3.70
CNN, dropout 7.13 4.68 3.90 5.04 4.37 3.79
CMNN 7.09 4.62 4.05 4.87 4.30 3.13
LSTM 5.33 3.17 3.07 3.44 3.22 2.93
RMNN 4.78 2.93 2.81 3.11 3.00 2.19

Table 10
Number of required training epochs of different acoustic models for all the
languages.

Model Cantonese Pashto Turkish Tagalog Vietnamese Tamil

DNN 20 19 20 20 18 18
CNN 20 20 20 19 20 19
CNN, dropout 27 40 38 40 40 40
CMNN 24 25 24 25 24 25
LSTM 13 12 11 14 13 14
RMNN 22 27 29 30 28 23

small matrix multiplications, which are less efficient for GPUs. b

The training time per epoch for the CNNs with dropout training
is on par with that of the CMNNs. However, the CNNs with
dropout training need much more training epochs to converge
than the CMNNs. The CNNs with dropout training require up
to 40 training epochs, which are too computationally expen-
sive. The training time per epoch for the LSTM models and
the RMNN models is about 2 or 3 times slower than the DNNs,
which is more computationally efficient than the CNNs. The par-
allel processing of the feature streams from different utterances
enables efficient training of the LSTM recurrent structures.
Fig. 7 illustrates the frame accuracies for different models
on the training set and the development set for Vietnamese. The
figure shows that the CMNN model have better development
set accuracy than the baseline DNN and CNN, because dropout
training improves its generalization ability. The CNN with
dropout training shows small and steady improvements with ev-
ery training epoch, but it converges very slowly. The figure also
shows that the accuracies of the LSTM and the RMNN are much
better than other models. This is because the recurrent weight Fig. 7. The frame accuracy for different models on (a) the training set and (b)
connections are able to model longer frame context. The CMNN the development set for Vietnamese.
and the RMNN both have quick convergence on the develop-
ment set in spite of the dropout regularization, which shows the toolkit. The results for all the languages are given in Table 11.
good optimization performance of the maxout neurons. We first try to combine the CNN models and the CMNN mod-
els, both of which have the convolutional structures. The relative
4.6. System combination improvements of the combined results are between −0.2% and
3.0% compared to the CMNN models. Then the LSTM models
The previous experiments show the effectiveness of the and the RMNN models are combined, both of which have the re-
CMNN models and the RMNN models. Since the CMNN and current connections. The relative improvements of the combined
the RMNN make use of the prior knowledge of different aspects results are between 1.0% and 4.5% compared to the RMNN
of speech signals (i.e., the local spectral properties within frames models. These results suggest that the differences between the
and the long-term dependencies among frames), they may have LSTM models and the RMNN models are greater than the dif-
complementary information. In this experiment we try to com- ferences between the CNN models and the CMNN models. This
bine the results of different systems. is indeed the case since the RMNN requires the maxout neurons
We use the lattice-based system combination method pro- and extra fully-connected hidden layers compared to the LSTM,
posed in Xu et al. (2011), which is implemented in the Kaldi whereas the CMNN only requires the maxout neurons compared
 M. Cai, J. Liu / Speech Communication 77 (2016) 53–64 63

Table 11
TERs (%) for system combination. The optimal results are shown in bold face.

Model Cantonese Pashto Turkish Tagalog Vietnamese Tamil

DNN 44.8 51.2 47.6 49.8 53.1 66.7
CNN 43.6 51.5 48.3 49.6 52.5 67.2
CMNN 41.7 49.3 45.4 47.2 49.9 64.2
LSTM 40.7 50.5 47.4 47.9 47.8 65.0
RMNN 39.0 48.1 44.9 45.7 46.0 63.4
CNN + CMNN 41.8 48.4 45.1 45.9 48.4 63.8
LSTM + RMNN 38.2 47.6 43.5 44.2 43.9 61.7
CMNN + RMNN 38.4 46.1 42.3 42.9 44.1 60.9
DNN + CNN + LSTM 40.3 48.1 44.1 44.7 46.8 62.6
DNN + CMNN + RMNN 39.3 46.5 42.7 43.4 45.5 61.2
CNN + CMNN + LSTM + RMNN 38.2 46.3 42.2 42.7 44.0 60.7
DNN + CNN + LSTM + CMNN + RMNN 38.7 46.6 42.4 43.0 44.9 60.9

to the CNN. The proposed CMNN models and RMNN models
are also combined. The results are better than previous com-
binations for Pashto, Turkish, Tagalog and Tamil, but a little
worse than the LSTM+RMNN results for Cantonese and Viet-
namese. We believe this is because both the Cantonese and the
Vietnamese are tonal languages. The pitch in tonal languages
is relatively stable. So that the speech in tonal languages ex-
hibits longer frame dependencies than the speech in non-tonal
languages. The recurrent structures in the LSTMP RNNs and
the RMNNs are suitable to model long frame dependencies.
Then we combine three baseline systems, i.e., the DNN, the
CNN and the LSTM models. We also combine the DNNs with
the CMNNs and RMNNs for comparison. The results show that
the latter combinations achieve relative improvements of 2.2%
to 3.3% over the former combinations. These results again con-
firm the effectiveness of the proposed models compared to the
baselines. We further try to combine 4 systems and 5 systems.
The best results are shown in bold face in Table 11. To the au-
thors’ knowledge, these are the state-of-the-art results on the
data sets.
the CMNN is about 3–4 times slower than the DNN of the same
size. The training time per epoch for the RMNN is about 2–3
times slower than the DNN of the same size. (5) System combi-
nations using the CNN, the LSTM, the CMNN and the RMNN
models give state-of-the-art results for all the 6 languages. In the
future we want to do multilingual training (Knill et al., 2013).
We would also like to try more sophisticated CNN structures
(Abdel-Hamid et al., 2014; Sainath et al., 2015) and try to de-
sign new models that combine the advantages of both the CMNN
models and the RMNN models.
Acknowledgments
The authors would like to thank anonymous reviewers for
improving the paper. This work is supported by National Nat-
ural Science Foundation of China under Grant Nos. 61273268,
61370034, 61403224 and 61005017.
References

5. Conclusions
In this paper, we extend our previous work on the convolu-
tional maxout neural network (CMNN) acoustic model and pro-
pose the new recurrent maxout neural network (RMNN) acoustic
model. Both the models take advantage of the maxout neurons,
while they model the prior knowledge of the speech signals
in different aspects. We compare alternative model structures
and explore various dropout strategies for the CMNNs and the
RMNNs. The performances of the models are evaluated for 6
different languages on the IARPA Babel data sets. System com-
binations are carried out using different models. Our main dis-
coveries include: (1) Dropout training works well for both the
CMNN and the RMNN. The optimal strategy is to apply dropout
to all the fully-connected layers of the models. (2) Replacing the
maxout neurons by the sigmoid neurons in the convolutional part
of the CMNNs or the fully-connected part of the RMNNs only
results in degraded performance. (3) The relative TER reductions
of the CMNNs over the CNNs are between 4.3% and 6.0%. The
relative TER reductions of the RMNNs over the LSTM RNNs
are between 2.5% and 5.3%. (4) The training time per epoch for
Abdel-Hamid, O., Mohamed, A., Jiang, H., Penn, G., 2012. Applying convolu-
tional neural networks concepts to hybrid NN-HMM model for speech recog-
nition. In: Proceedings of International Conference on Acoustics, Speech and
Signal Processing. ICASSP. Kyoto, Japan, pp. 4277–4280.
Abdel-Hamid, O., Mohamed, A., Jiang, H., Deng, L., Penn, G., Yu, D., 2014.
Convolutional neural networks for speech recognition. IEEE/ACM Trans.
Audio Speech Lang. Process. 22 (10), 1533–1545.
Bengio, Y., Simard, P., Frasconi, P., 1994. Learning long-term dependencies
with gradient descent is difficult. IEEE Trans. Neural Netw. 5 (2), 157–166.
Cai, M., Shi, Y., Liu, J., 2013. Deep maxout neural networks for speech recogni-
tion. In: Proceedings of Automatic Speech Recognition and Understanding.
ASRU. Olomouc, Czech Republic, pp. 291–296.
Cui, J., Cui, X., Ramabhadran, B., Kim, J., Kingsbury, B., Mamou, J., Mangu, L.,
Picheny, M., Sainath, T.N., Sethy, A., 2013. Developing speech recognition
systems for corpus indexing under the IARPA Babel program. In: Proceed-
ings of International Conference on Acoustics, Speech and Signal Process-
ing. ICASSP. Vancouver, Canada, pp. 6753–6757.
Cai, M., Shi, Y., Kang, J., Liu, J., Su, T., 2014. Convolutional maxout neural net-
works for low-resource speech recognition. In: Proceedings of International
Symposium on Chinese Spoken Language Processing. ISCSLP. Singapore,
pp. 133–137.
Chen, N.F., Sivadas, S., Lim, B.P., Ngo, H.G., Xu, H., Pham, V.T., Ma, B.,
Li, H., 2014. Strategies for Vietnamese keyword search. In: Proceedings
of International Conference on Acoustics, Speech and Signal Processing.
ICASSP. Florence, Italy, pp. 4121–4125.
64 M. Cai, J. Liu / Speech Communication 77 (2016) 53–64
Dahl, G.E., Yu, D., Deng, L., Acero, A., 2012. Context-dependent pre-trained
deep neural networks for large-vocabulary speech recognition. IEEE Trans.
Audio Speech Lang. Process. 20 (1), 30–42.
Deng, L., Huang, X., 2004. Challenges in adopting speech recognition. Com-
mun. ACM 47 (1), 69–75.
Gers, F.A., Schraudolph, N.N., Schmidhuber, J., 2002. Learning precise timing
with LSTM recurrent networks. J. Mach. Learn. Res. 3, 115–143.
Ghahremani, P., BabaAli, B., Povey, D., Riedhammer, K., Trmal, J.,
Khudanpur, S., 2014. A pitch extraction algorithm tuned for automatic
speech recognition. In: Proceedings of International Conference on Acous-
tics, Speech and Signal Processing. ICASSP. Florence, Italy, pp. 2494–2498.
Glorot, X., Bengio, Y., 2010. Understanding the difficulty of training deep feed-
forward neural networks. J. Mach. Learn. Res. 9, 249–256.
Glorot, X., Bordes, A., Bengio, Y., 2011. Deep sparse rectifier neural networks.
J. Mach. Learn. Res. 15, 315–323.
Goodfellow, I.J., Warde-Farley, D., Mirza, M., Courville, A., Bengio, Y., 2013.
Maxout networks. In: Proceedings of International Conference on Machine
Learning. ICML. Atlanta, USA, pp. 2356–2364.
Graves, A., Mohamed, A., Hinton, G.E., 2013a. Speech recognition with deep
recurrent neural networks. In: Proceedings of International Conference on
Acoustics, Speech and Signal Processing. ICASSP. Vancouver, Canada,
pp. 6645–6649.
Graves, A., Jaitly, N., Mohamed, A., 2013b. Hybrid speech recognition with deep
bidirectional LSTM. In: Proceedings of Automatic Speech Recognition and
Understanding. ASRU. Olomouc, Czech Republic, pp. 273–278.
Hinton, G.E., Osindero, S., Teh, Y.-W., 2006. A fast learning algorithm for deep
belief nets. Neural Comput. 18 (7), 1527–1554.
Hinton, G.E., Deng, L., Yu, D., Dahl, G.E., Mohamed, A., Jaitly, N., Senior, A.,
Vanhoucke, V., Nguyen, P., Sainath, T.N., Kingsbury, B., 2012. Deep neural
networks for acoustic modeling in speech recognition: the shared views of
four research groups. IEEE Signal Process. Mag. 29 (6), 82–97.
Karafiat, M., Grezl, F., Hannemann, M., Vesely, K., Cernocky, J.H., 2013. BUT
Babel system for spontaneous Cantonese. In: Proceedings of Interspeech.
Lyon, France, pp. 2589–2593.
Knill, K.M., Gales, M.J.F., Rath, S.P., Woodland, P.C., Zhang, C., Zhang, S.-X.,
2013. Investigation of multilingual deep neural networks for spoken term
detection. In: Proceedings of Automatic Speech Recognition and Under-
standing. ASRU. Olomouc, Czech Republic, pp. 138–143.
LeCun, Y., Bottou, L., Bengio, Y., Haffner, P., 1998. Gradient-based learning
applied to document recognition. Proc. IEEE 86 (11), 2278–2324.
Li, J., Wang, X., Xu, B., 2013. Understanding the dropout strategy and analyzing
its effectiveness on LVCSR. In: Proceedings of International Conference
on Acoustics, Speech and Signal Processing. ICASSP. Vancouver, Canada,
pp. 7614–7618.
Miao, Y., Metze, F., Rawat, S., 2013. Deep maxout networks for low-resource
speech recognition. In: Proceedings of Automatic Speech Recognition and
Understanding. ASRU. Olomouc, Czech Republic, pp. 398–403.
Miao, Y., Metze, F., 2014. Improving language-universal feature extraction with
deep maxout and convolutional neural networks. In: Proceedings of Inter-
speech. Singapore, pp. 800–804.
Mohamed, A., Dahl, G.E., Hinton, G.E., 2012. Acoustic modeling using deep
belief networks. IEEE Trans. Audio Speech Lang. Process. 20 (1), 14–
22.
Nair, V., Hinton, G.E., 2010. Rectified linear units improve restricted Boltz-
mann machines. In: Proceedings of International Conference on Machine
Learning. ICML. Haifa, Israel, pp. 807–814.
Povey, D., Ghoshal, A., Boulianne, G., Burget, L., Glembek, O., Goel, N.,
Hannemann, M., Motlicek, P., Qian, Y., Schwarz, P., Silovsky, J.,
Stemmer, G., Vesely, K., 2011. The Kaldi speech recognition toolkit. In:
Proceedings of Automatic Speech Recognition and Understanding. ASRU.
Waikoloa, USA.
Sainath, T.N., Mohamed, A., Kingsbury, B., Ramabhadran, B., 2013a. Deep
convolutional neural networks for LVCSR. In: Proceedings of International
Conference on Acoustics, Speech and Signal Processing. ICASSP. Vancou-
ver, Canada, pp. 8614–8618.
Sainath, T.N., Kingsbury, B., Mohamed, A., Dahl, G.E., Saon, G., Soltau, H.,
Beran, T., Aravkin, A., Ramabhadran, B., 2013b. Improvements to deep
convolutional neural networks for LVCSR. In: Proceedings of Automatic
Speech Recognition and Understanding. ASRU. Olomouc, Czech Republic,
pp. 315–320.
Sak, H., Senior, A., Beaufays, F., 2014a. Long short-term memory recurrent neu-
ral network architectures for large scale acoustic modeling. In: Proceedings
of Interspeech. Singapore, pp. 338–342.
Sak, H., Vinyals, O., Heigold, G., Senior, A., McDermott, E., Monga, R.,
Mao, M., 2014b. Sequence discriminative distributed training of long short-
term memory recurrent neural networks. In: Proceedings of Interspeech.
Singapore, pp. 1209–1213.
Sainath, T.N., Kingsbury, B., Mohamed, A., Ramabhadran, B., 2013. Learning
filter banks within a deep neural network framework. In: Proceedings of
Automatic Speech Recognition and Understanding. ASRU. Olomouc, Czech
Republic, pp. 297–302.
Sainath, T.N., Kingsbury, B., Saon, G., Soltau, H., Mohamed, A., Dahl, G.E.,
Ramabhadran, B., 2015. Deep convolutional neural networks for large-scale
speech tasks. Neural Netw. 64, 39–48.
Saon, G., Soltau, H., Emami, A., Picheny, M., 2014. Unfolded recurrent neural
networks for speech recognition. In: Proceedings of Interspeech. Singapore,
pp. 343–347.
Seide, F., Li, G., Yu, D., 2011. Conversational speech transcription using context-
dependent deep neural networks. In: Proceedings of Interspeech. Florence,
Italy, pp. 437–440.
Seide, F., Li, G., Chen, X., Yu, D., 2011. Feature engineering in context-
dependent deep neural networks for conversational speech transcription. In:
Proceedings of Automatic Speech Recognition and Understanding. ASRU.
Waikoloa, USA, pp. 24–29.
Srivastava, N., Hinton, G.E., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.R.,
2014. Dropout: a simple way to prevent neural networks from overfitting. J.
Mach. Learn. Res. 15, 1929–1958.
Toth, L., 2013. Convolutional deep rectifier neural nets for phone recognition.
In: Proceedings of Interspeech. Lyon, France, pp. 1722–1726.
Toth, L., 2014. Convolutional deep maxout networks for phone recognition. In:
Proceedings of Interspeech. Singapore, pp. 1078–1082.
Tsakalidis, S., Hsiao, R., Karakos, D., Ng, T., Ranjan, S., Saikumar, G.,
Zhang, L., Nguyen, L., Schwartz, R., Makhoul, J., 2014. The 2013 BBN
Vietnamese telephone speech keyword spotting system. In: Proceedings
of International Conference on Acoustics, Speech and Signal Processing.
ICASSP. Florence, Italy, pp. 7829–7833.
Vincent, P., Larochelle, H., Bengio, Y., Manzegol, P.-A., 2008. Extracting and
composing robust features with denoising autoencoders. In: Proceedings of
International Conference on Machine Learning. ICML. Helsinki, Finland,
pp. 1096–1103.
Vinyals, O., Ravuri, S.V., Povey, D., 2012. Revisiting recurrent neural networks
for robust ASR. In: Proceedings of International Conference on Acoustics,
Speech and Signal Processing. ICASSP. Kyoto, Japan, pp. 4085–4088.
Weng, C., Yu, D., Watanabe, S., Juang, B.-H., 2014. Recurrent deep neural
networks for robust speech recognition. In: Proceedings of International
Conference on Acoustics, Speech and Signal Processing. ICASSP. Florence,
Italy, pp. 5532–5536.
Xu, H., Povey, D., Mangu, L., Zhu, J., 2011. Minimum Bayes Risk decoding and
system combination based on a recursion for edit distance. Comput. Speech
Lang. 25 (4), 802–828.
Zaremba, W., Sutskever, I., Vinyals, O., Recurrent neural network regularization,
arXiv:1409.2329 (2014).