IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 25, NO.
12, DECEMBER 2017 2313
Whispered Speech Recognition Using Deep
Denoising Autoencoder and Inverse Filtering
Đorđe T. Grozdić and Slobodan T. Jovičić
Abstract—Due to the profound differences between acoustic
characteristics of neutral and whispered speech, the performance
of traditional automatic speech recognition (ASR) systems trained
on neutral speech degrades significantly when whisper is applied.
In order to deeply analyze this mismatched train/test situation and
to develop an efficient way for whisper recognition, this study first
analyzes acoustic characteristics of whispered speech, addresses
the problems of whispered speech recognition in mismatched
conditions, and then proposes a new robust cepstral features and
preprocessing approach based on deep denoising autoencoder
(DDAE) that enhance whisper recognition. The experimental
results confirm that Teager-energy-based cepstral features,
especially TECCs, are more robust and better whisper descriptors
than traditional Mel-frequency cepstral coefficients (MFCC).
Further detailed analysis of cepstral distances, distributions of
cepstral coefficients, confusion matrices, and experiments with
inverse filtering, prove that voicing in speech stimuli is the main
cause of word misclassification in mismatched train/test scenarios.
The new framework based on DDAE and TECC feature, signif-
icantly improves whisper recognition accuracy and outperforms
traditional MFCC and GMM-HMM (Gaussian mixture density—
Hidden Markov model) baseline, resulting in an absolute 31%
improvement of whisper recognition accuracy. The achieved word
recognition rate in neutral/whisper scenario is 92.81%.
Index Terms—Automatic speech recognition, deep denoising
autoencoder, inverse filtering, Teager-energy operator, whispered
speech recognition.
I. INTRODUCTION
ODERN-DAY automatic speech recognition (ASR) sys-
M tems have shown good performances and evident com-
mercial utilization, while, at the same time they have displayed
a considerable number of weaknesses, flaws, and problems
when practically utilized. In order to accomplish more efficient
and better quality of human-computer interaction, these prob-
lems have yet to be solved. Despite much research concerned
with ASR systems’ improvement over the last years, in some
Manuscript received October 31, 2016; revised May 21, 2017; accepted June
17, 2017. Date of current version November 27, 2017. This work was supported
by the Serbian Ministry of Education, Science and Technological Development,
under Grants TR 32032 and OI 178027. The guest editor coordinating the
review of this manuscript and approving it for publication was Dr. Prof. Tanja
Schultz . (Corresponding author: Đorđe T. Grozdić.)
Đ. T. Grozdić is with the School of Electrical Engineering, University of
Belgrade, Belgrade 11120, Serbia (e-mail: djordjegrozdic@gmail.com).
S. T. Jovičić is with the School of Electrical Engineering, University of
Belgrade, Belgrade 11120, Serbia, and also with the Life Activities Advance-
ment Center, Laboratory for Forensic Acoustics and Phonetics, Belgrade 11000,
Serbia (e-mail: jovicic@etf.rs).
Color versions of one or more of the figures in this paper are available online
at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/TASLP.2017.2738559
2329-9290 © 2017 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications standards/publications/rights/index.html for more information.
situations these systems still remain both significantly sensi-
tive and unreliable. The ASR performance may be affected by
various factors including: speech type and quality, individual
speaker characteristics, such as speech rate and style, dialect,
the vocal tract anatomy, the speaker’s psycho-physical state etc.
Furthermore, other influence types may also occur, particularly
originating from the surrounding environment, and entailing:
ambient noise, reverberation, loudness etc. Apart from the neu-
tral mode, speech also occurs in different modalities, such as
emotional speech, the Lombard-effect speech, as well as whis-
pered speech. All these above-mentioned atypical speech modes
present challenging problems in ASR.
Whisper is a specific form of verbal communication that is
frequently utilized in different situations. Firstly, it is employed
to make a discreet and intimate atmosphere in conversation,
and, secondly, it is used to protect some confidential and private
information from uninvolved parties. Besides, speakers whisper
when they do not want to disturb other people, for example in
the library, or during a business meeting, but also in criminal
activities, e.g., when criminals try to disguise their identity.
Nevertheless, in spite of the conscious production of a whisper,
it may occur due to health problems such as chronic disease of
the larynx structures [1].
Whisper has become a research topic of interest, essentially
important for speech technologies, mainly because of its sub-
stantial difference compared to normally phonated (neutral)
speech, and this is the case primarily due to the glottal vibra-
tions’ absence, noisy structure and lower SNR (Signal-to-Noise
Ratio) [2]. Namely, the most current neutral speech oriented
interfaces of ASR systems are not capable of handling such
an acoustic mismatch. Therefore, the performance of neutral-
trained ASR systems degrades significantly when whisper is
applied. Nonetheless, the research on the automatic whisper
recognition is still at the beginning, and there have been only a
few studies so far. The literature shows several approaches that
have been proposed to alleviate this acoustic mismatch through
model adaptation [3]–[7], feature transformations [5], or using
alternative sensing technologies such as throat microphone [8].
Also, there is an audio-visual approach to isolated word recogni-
tion under whisper speech condition [9]. A vast majority of stud-
ies on whisper recognition use GMM-HMM (Gaussian mixture
density—Hidden Markov model) system and traditional MFCC
(Mel-Frequency Cepstral Coefficients) or PLP (Perceptual Lin-
ear Prediction) features.
Our contribution to the literature is to demonstrate that by
using state-of-the-art acoustic modeling techniques such as
2314 IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 25, NO. 12, DECEMBER 2017
deep denoising autoencoder (DDAE) a significant increase in
whisper recognition performance can be attained without the
need for model adaptation. The DDAE is applied to transform
cepstral feature vectors of whispered speech, into clean cepstral
feature vectors of neutral speech. More specifically, feature
extraction by a DDAE is achieved by training the deep neural
network to predict original neutral speech features from
pseudo-whisper features that are artificially generated by in-
verse filtering of neutral speech data. Acquired cepstral features
are then concatenated with original cepstral features and then
processed with a conventional GMM-HMM recognizer to
conduct an isolated word recognition task. The advantage of
our feature mechanism is that whisper-robust features are easily
acquired by a rather simple mechanism. Regarding the input
feature vectors, three different types of cepstral coefficients
were tested: traditional MFCC (Mel-Frequency Cepstral
Coefficients) and two more recent cepstral coefficients, the
TECC (Teager Energy Cepstral Coefficients) and the TEMFCC
(Teager Energy based Mel-Frequency Cepstral Coefficients)
[10]. These two features use the nonlinear TEO (Teager-Energy
Operator) [9], [10] which serves for Teager energy calcula-
tion. So far, nonlinear TEO-based features introduced very
promising results in ASR of quiet and unvoiced murmured
speech as well in speech classification under stress and noisy
conditions [9]–[12]. On the other side, the characteristics of
whispered speech might be considered to have similarities with
non-audible murmur, noise corrupted speech and speech under
stress. Due to these similarities, it was expected that the TECC
and the TEMFCC could be good descriptors of whispered
speech. The experiments on whispered speech recognition were
conducted on a special database that was particularly created
for this study, entitled Whi-Spe [13], which contains 10000
audio recordings of parallel neutral and whispered speech.
Another important task of this study was to analyze the confu-
sion in mismatched train/test conditions. The situation in which
a speaker is in front of the ASR system and switches from neu-
tral speech to whisper provides particularly interesting results
of neutral/whisper scenario that corresponds to real situation. In
order to better understand these mismatched scenarios and to de-
velop system which can provide better whisper recognition un-
der such conditions, confusion in word recognition was deeply
analyzed. The analysis of acoustic characteristics, distributions
of cepstral coefficients, cepstral distances, as well as analysis
of confusion matrices were done. Based on these results, a new
speech signal pre-processing approach was proposed. This ap-
proach includes TECC features, inverse filtering, and DDAE,
which reduces the difference between neutral speech and whis-
per and improves the word recognition rates in mismatched
scenarios.
The reminder of this study is organized into seven sections. In
Section II, we briefly review related work on whisper recogni-
tion. Section III introduces the speech corpus that was specially
constructed for this study and describes the nature of whispered
speech. Here, the analyses of whisper characteristics and differ-
ences compared to neutral speech are presented. In Section IV,
we introduce the general framework of the proposed ASR sys-
tem and describe feature extraction, DDAE, and GMM-HMM
recognizer. In the next section, we conduct isolated word recog-
nition experiments to evaluate different cepstral features, inverse
filtering and application of deep denoising autoencoder in mis-
matched train/test conditions. Finally, conclusions are presented
in Section VI.
II. RELATED WORK
Automatic recognition of whispered speech is an ongoing
and active field of research which is hindered by the lack of
suitable and systematically collected corpora. Currently, there
are only several existing databases of parallel neutral and whis-
pered speech collected for English [13], [14], [15], Japanese [3],
Serbian [13] and Mandarin language [17]. Most of them have
a small or medium-sized vocabulary, while only a few of them
are transcribed and phonetically balanced.
One of the first experiments with automatic whisper recog-
nition was started by researchers at the University of Nagoya
[3]. They intended to develop a speech recognizer specifically
capable of handling whisper on cell phones in noisy conditions.
Using HMM technique and MFCC features, they analyzed dif-
ferent mismatched train/test scenarios with three speech modes:
whisper, low-voice speech, and neutral speech. The results of
these experiments show severe degradation in ASR when using
mismatch data. However, the outstanding result is that the whis-
pered speech model (ASR model trained on whisper) performs
as well on either type of speech. This phenomenon, among
other things, will be explained and experimentally proved in
our study. Next, the authors [3] confirm that covering the mouth
and the cell-phone with a hand can cause the SNR increase
in a noisy environment and may improve whisper and low-
voice speech recognition to some extent. It was also shown that
ASR systems that are trained on neutral speech can be adapted
to whisper recognition by using a small amount of whispered
speech data, which can improve whisper recognition. Utiliz-
ing such speaking-style-independent model yields the whisper
recognition accuracy of about 66%.
Later studies of whisper recognition attempt to reduce this
acoustic mismatch in different ways, for example through model
adaptation [4]–[6], [17] and feature transformations [5]. There
are also studies focused on front-end feature extraction strate-
gies [14], [18] and front-end filter bank redistribution based on
the subband relevance [15]. The efficiency of Vocal Tract Length
Normalization (VTLN) [20] and shift transform [21] for whis-
per recognition was also investigated in [20]. Several studies
deal with ASR model adaptation to whisper recognition when
small amounts of whisper are available [19], [21]. In [20] Vector
Taylor Series (VTS) based approach to pseudo-whisper adapta-
tion sample generation was investigated. All above-mentioned
papers use HMM as ASR system. There are only two stud-
ies that investigate neural network-based approach in whisper
recognition [16], [22]. Also, there is an audio-visual approach
for isolated digits recognition under whisper and neutral speech
[9]. In terms of feature extraction, only basic MFCC and PLP
features were tested in whisper recognition.
Although some improvement is achieved in each study, suc-
cessful and usable recognition of whisper is still an unsolved
problem and an open research topic.
GROZDIĆ AND JOVIČIĆ: WHISPERED SPEECH RECOGNITION USING DEEP DENOISING AUTOENCODER AND INVERSE FILTERING 2315

III. WHISPER
A. Corpus of Neutral/Whispered Speech
Given the fact that there is a lack of extensive, appropriate
and publicly available databases in this area, the special speech
corpus for Serbian language, entitled “Whi-Spe” (abbreviated
from Whispered Speech), was developed for the purpose of this
study. The corpus consists of two parts - 5000 audio recordings
of spoken isolated words in neutral speech and 5000 recordings
of the same words in whisper. The corpus contains 50 differ-
ent words that were taken as a part of GEES speech database
[24] and achieve the balance of linguistic features. Both whis-
pered speech and neutral speech were collected from 5 male
and 5 female speakers, with typical pronunciation of speech
and whisper, and correct hearing. Each speaker had read all 50
words ten times in both speech modes, so the Whi-Spe corpus
contains 10000 recorded words. The recording process was car-
ried out under quiet laboratory conditions in a sound booth with
a high-quality omnidirectional microphone. Speech data were Fig. 1. Waveforms, spectrograms and spectrums of word “pijatsa” (“market”
in English) in neutral (left) and whispered speech (right).
digitized using a sampling frequency of 22050 Hz with 16 bits
per sample. More pieces of information concerning the Whi-Spe
corpus can be found in [13].

B. Acoustic Analysis of Whispered Speech

Whisper represents an alternative speech mode that is quite
different from neutral speech with regard to its characteris-
tics, nature and generation mechanism. The main peculiarity
of whisper’s generation mechanism is the absence of vocal cord
vibration. The vocal tract shape differs from neutral speech
and implies narrowed pharyngeal cavity, different placement
of tongue, lips and mouth opening [25]. As a result of differ-
ent vocal tract shape and articulation organs’ specific behavior,
whisper has distinctive acoustic characteristics. In order to more Fig. 2. The relationship between cepstral distance (between words in neutral
and whispered speech) and the ratio of unvoiced consonants.
closely describe this acoustic mismatch that occurs in ASR sys-
tems, several time-domain and spectral-domain examples are
presented. Fig. 1 shows comparisons of waveforms, spectro-
grams and long-term average spectrums of a word in neutral from the long-term average spectrums, in which whisper, in op-
and whispered speech. We can see a clear difference in ampli- position to neutral speech, has much flatter spectral slope [3],
tude intensity from the time waveforms. Since there is the lack [24], [26].
of sonority, amplitudes of voiced phonemes (vowels chiefly) are In order to quantify the difference between the same words
considerably lower in whisper, while the amplitudes of unvoiced in neutral speech and whisper, as a measure of cepstral dissim-
phonemes show similar intensity for both modes [3]. Whisper ilarity, the cepstral distance can be used. The cepstral distance
has completely noisy structure and conspicuously lower energy. (CD) is defined as:

The whispered speech duration is slightly longer (it is more  N  2
obvious in longer utterances), which is also one of its character- 10  2
 (n ) (w )
CD = ci − ci [dB], (1)
istics [25], [26]. ln(10) i=1
The spectrograms’ analysis shows the other differences be-
(n ) (w )
tween neutral and whispered speech. Although the vocal tract where ci and ci are the cepstral coefficients (MFCC) for
shape is different in whisper, spectrograms save the most impor- the neutral and the whispered speech, respectively, and N is the
tant spectral speech characteristics. Despite the noisy structure, fixed number of cepstral coefficients per word. The average CD
the spectral concentrates of some phonemes can still be ob- for each word from Whi-Spe corpus is calculated and presented
served. The most important spectral changes are perceived in in Fig. 2 showing the correlation with the ratio of unvoiced
vowels. In whisper the lower formants’ locations are shifted to consonants per word (unvoiced/phoneme count).
higher frequencies [3], [24], [27]. In contrast to vowels, un- The straight line of best fit shows strong and negative corre-
voiced consonants are not significantly changed in their spectral lation (Pearson’s coefficient −0.834), meaning that the words
domain [3]. The lack of sonority in whisper is also observed with the lower ratio of unvoiced consonants have the higher
2316 IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 25, NO. 12, DECEMBER 2017
Fig. 3. Normalized distributions of c0 and c1 coefficients in neutral and
whispered speech.
CD and vice versa. For example, a word with 3 unvoiced
consonants and total 4 phonemes (ratio 0.75) has the lowest
CD, 4.7 dB. Conversely, the words without unvoiced conso-
nants (ratio 0) have the highest CD or dissimilarity, on average
6.7 dB. This indicates that characteristics of vowels and voiced
consonants change more significantly than those of unvoiced
consonants [3].
Cepstral coefficients can be further analyzed and compared
between neutral and whispered speech. For example distribu-
tions of the first two cepstral coefficients, c0 and c1 , can be used
to verify the energy and spectral slope in Whi-Spe database.
To be specific, c0 incorporates information about signal energy,
while c1 is related to the spectral slope [15]. Fig. 3 shows c0
and c1 distributions in neutral and whispered speech, as well as
distributions of their unvoiced and voiced parts.
The ratio of unvoiced to voiced speech segments in the Whi-
Spe neutral speech recordings is 37.65/62.35%, respectively,
and 99.2/0.8% in whispered recordings. The left part of the
Fig. 3 presents c0 distributions which reflect the energy levels.
The distributions of neutral speech show that dominant voiced
segments tend to reach higher energy (higher c0 values) than
the unvoiced. In whisper, unvoiced segments are dominant and
the overall distributions are shifted to the left indicating lower
energies compared to neutral speech.
The right part of Fig. 3 shows distributions of c1 which reflect
the spectral slope. In neutral speech, voiced distribution occu-
pies the highest c1 values, i.e., the steepest slopes. Following the
intuition, unvoiced segments are centered together at lower c1 s
(flatter slopes). In whisper, all distributions are shifted to the left
(lower c1 s). This reflects that the overall spectral slope is flatter
compared to neutral speech. In contrast to distributions of c0 ,
here we have a noticeable difference in distributions’ positions.
When compared with neutral speech, all these specific dif-
ferences render whisper a particular challenge for recognition
by means of traditional ASR systems that have been primarily
Fig. 4. Architecture of the proposed ASR system. The system is com-
posed of: (a) Feature extractor. (b) Deep denoising autoencoder (DDAE).
(c) GMM-HMM.
designed for normally phonated speech. Some of these differ-
ences can be reduced, i.e., differences in energy level. This
problem can be solved by closer microphone position while
whispering, or by simple signal amplification under good SNR
conditions. However, spectral differences still remain, e.g., dif-
ferences in spectral slopes, and require their proper modification
in order to be adjusted for ASR.
IV. MODEL
A schematic diagram of the proposed ASR system is shown
in Fig. 4.
The proposed framework consists of two feature extractors
to process audio signals and one GMM-HMM. The first extrac-
tor serves for MFCC, TECC, and TEMFCC cepstral features
extraction. The second extractor is in the form of a deep denois-
ing autoencoder (DDAE) which filters out the effects of whis-
pered speech and reconstructs clean neutral speech features.
Accordingly, DDAE outputs are extracted to serve as the robust
cepstral features for mismatched whispered speech. Finally, a
GMM-HMM recognizes isolated words by binding acquired
cepstral feature sequences from both extractors.
A. Feature Extraction
Three types of cepstral features are tested in this study:
Mel-frequency Cepstral Coefficient (MFCC), Teager En-
ergy Cepstral Coefficient (TECC) and Teager-Energy based
Mel-frequency Cepstral Coefficient (TEMFCC). These features
are extracted from 25 ms time frames with a step size of 10 ms.
The MFCCs are traditional cepstral features that are used in ASR
and their extraction procedure is performed in the following
steps: (1) the discrete Fourier transform (DFT) of speech signal
frame is computed, (2) the power spectrum is calculated, (3) the
Mel-filterbank consisting of 30 triangular shaped filters is ap-
plied, (4) the signal output log energy of each filter is estimated,
(5) the discrete cosine transform (DCT) is taken and the cepstral
coefficients are obtained, and (6) only the first 12 MFCCs are
extracted. Finally, feature vectors are channel normalized us-
ing cepstral mean subtraction (CMS) to remove convolutional
GROZDIĆ AND JOVIČIĆ: WHISPERED SPEECH RECOGNITION USING DEEP DENOISING AUTOENCODER AND INVERSE FILTERING
distortions caused by characteristics of communication chan-
nels or recording devices. However, CMS is partially effective
in reducing the effects of additive environmental noise.
The other two cepstral coefficient types are relatively new and
more robust to noise interference. Moreover, they are still not
tested on whispered speech recognition. Their basic extraction
algorithm is similar to MFCC feature, but includes one signifi-
cant difference in terms of energy estimation. For TEMFCC and
TECC calculation, the nonlinear Teager-energy operator (TEO)
is used [29], which serves for estimation of the Teager-energy
in place of the standard energy calculation method (Standard-
Energy Operator, SEO −x(t)2 ). The main idea behind using
TEO is nonlinear modeling of speech production. Namely, the
traditional acoustic theory assumes that airflow from the vocal
tract propagates as a plane wave. However, this assumption may
not hold since the flow is actually separate and concomitant vor-
tices are distributed throughout the vocal tract. The true source of
speech production is actually the vortex-flow interactions which
are nonlinear [30]. Since whispered speech has nonlinear and
extremely turbulent airflow, TEO provides an efficient way for
signal processing. Contrary to the traditional signal energy ap-
proximation, which only takes into account the kinetic energy of
the signal’s source, TEO computes the “true” total source energy
of a resonance signal, and incorporates both amplitude and fre-
quency information [10]. In other words, the total source energy,
i.e., the sum of the potential and kinetic energy, is proportional to
the product amplitude and frequency squared which is nothing
but the true energy required to generate the speech signal [31].
These supplementary information can improve time-frequency
description of rapid energy changes within a glottal cycle [29]
and improve the representation of formant information in the
feature vectors. For continuous-time signals, TEO is defined
as [32]:
Ψ(x(t)) = ẋ(t)2 − x(t)ẍ(t), (2)
where ẋ(t) and ẍ(t) are first and second-time derivatives of
speech signal x(t). The TEO’s good feature is also a relatively
easy and fast way of computing it, which could be of benefit
to real-time continuous speech recognition. For discrete time
signal is nearly instantaneous and requires only three adjacent
signal samples at each time instance [32]:
Ψ(x[n]) = x[n]2 − x[n − 1]x[n + 1]. (3)
In addition to the TEO utilization, the TECC feature is dis-
tinctive by one more characteristic in comparison to MFCC and
TEMFCC features, and that is the use of Gammatone filterbank
[10] instead of Mel-filterbank. In this research, the Gammatone
filterbank contains the same number of filters (L = 30) as in
the Mel-filterbank in the process of MFCC and TEMFCC fea-
tures extraction. The Gammatone filters are asymmetric, non-
constant-Q type, smoother and broader compared to Mel’s fil-
ters that are asymmetric, triangular and of constant-Q type. The
Gammatone filterbank is auditory motivated filterbank created
with the aim to simulate the nature of cochlea and basilar mem-
brane, and thereby human sound perception. It is also superior
to the Mel-filterbank and provides higher robustness in situation
when signal is degraded with additive noise and other harmful
2317
interferences [10]. Recent studies have shown that the human
physiology dictates that the auditory filter bandwidths are given
by the ERB (Equivalent Rectangular Bandwidth) function [10]:
ERB(fc ) = 6.23(fc /1000) + 93.39(fc /1000) + 28.52, (4)
where fc is the central frequency in Hz. Then, the impulse
response of the asymmetrical Gammatone filter is given by:
g(t) = Atη −1 e(−2π bE R B (f c )t) cos(2πfc t), (5)
where A., b and η (the filter order) are the Gammatone design
parameters, and fc is the filter central frequency.
At the end, in order to describe speech signal dynamics more
thoroughly, the MFCC, TECC, and TEMFCC feature vectors
were concatenated with their first (delta) and second-time (delta-
delta) derivatives. In this way, every frame was represented with
36-dimensional vector (12 static features + 12 delta + 12 delta-
delta).
B. Denoising Autoencoder (DAE)
The deep autoencoder is a special type of the DNN (Deep
Neural Network) whose output is a reconstruction of the input
which is commonly utilized for dimensionality compression and
feature extraction [32]. Typically, an autoencoder has an input
layer which represents the original feature vectors, one or more
hidden layers that represent the transformed feature, and an out-
put layer which matches the input layer for reconstruction. The
dimension of the hidden layers can be either smaller than the
input dimension when the goal is feature compression, or larger
when the goal is mapping the feature to a higher-dimensional
space. An autoencoder tries to find deterministic mapping be-
tween input units and hidden nodes by means of a nonlinear
function fΘ (x):
y = fθ (x) = f1 (W x + b), (6)
where W is a d × d weight matrix, b is a bias vector, and f1 (.)
is a nonlinear function such as sigmoid or tanh. This mapping is
called the encoder. The latent representation y is then mapped
back to reconstruct the input signal with:
z = fθ  (y) = f2 (W  y + b ), (7)
where W is a d × d weight matrix, b is a bias vector, and
f2 (.) is either nonlinear function, such as sigmoid or tanh, or a
linear function. This mapping is called the decoder. The goal of
training is to minimize squared error function:
L(x, z) = x − z2 . (8)
To prevent autoencoder from learning the trivial identity map-
ping function, some constraints are usually applied during the
training. For example adding Gaussian noise to the input signal
(which is applied in our study) or using the “dropout” trick by
randomly forcing certain values to be zero at the input data.
Such autoencoder is known as denoising autoencoder (DAE)
[32]. DAE shares the same structure as autoencoder, but in-
put data is a deteriorated version of the output data. In other
words, DAE use feature mapping to convert corrupted input data
(x̂ - input signal) into clean output (x - teacher signal). In our
2318 IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 25, NO. 12, DECEMBER 2017
case, we use deep DAE to transform cepstral feature vectors
of whispered speech, into clean cepstral feature vectors of neu-
tral speech. The proposed architecture of DDAE is illustrated
in Fig. 4 and consists of two encoder layers with sigmoid func-
tions and one decoder layer with linear function. The input
layer has 11 × 36 nodes while output has 36 nodes. This means
that we use 11 contiguous frames with 36 cepstral coefficients
as x̂ to encode, and use only the corresponding middle frame
with 36 clean cepstral features as x to fine-tune. The output
36-dimensional feature vector is joined with the original 36-
dimensional feature vector from feature extractor and together
fed to the GMM-HMM, as shown in Fig. 4.
In terms of training, with parallel deteriorated and clean
speech data, a DDAE can be pre-trained on pseudo-whisper
cepstral features and fine-tuned by neutral speech cepstral fea-
tures. To be precise, in our experiments pseudo-whisper sam-
ples are obtained by inverse filtering [34] of neutral speech
samples and adding random Gaussian noise with 10 dB SNR.
Such pseudo-whisper data is used in the pre-training process,
while neutral speech data is applied for fine-tuning of DDAE.
This is a standard way to train DDAE. There are several rea-
sons for adding random noise. First, with adding Gaussian noise,
inverse-filtered signal becomes more similar to whisper in terms
of its noisy nature, and the model learned that way would be
robust to the same kind of distortions in the test data. Second,
since the noise is added randomly, the DDAE avoid learning
the trivial identity solution. Third after all, each distorted input
sample is different, which greatly increases the training set size
and thus can additionally alleviate the overfitting problem. In
this way, the rich nonlinear structure in the DDAE can be used
to learn an efficient transfer function which suppresses whisper
characteristics in speech while keeping enough phonetically dis-
criminative information to generate good reconstructed neutral
speech features. This can be of great importance for large vo-
cabulary continuous speech recognition systems, where ground
truth labels are hard to obtain (i.e., in the whisper case).
Pre-training consists of learning a stack of restricted
Boltzmann machines (RBMs), each having only one layer of
feature detectors. After learning one RBM, the status of the
learned hidden units given the training data can be used as fea-
ture vectors for the second RBM layer. The contrastive diver-
gence method [35] can be used to learn the second RBM in the
same fashion. Then, the status of the hidden units of the second
RBM can be used as the feature vectors for the third RBM, etc.
This layer-by-layer learning can be repeated for many times.
After the pre-training, the RBMs are unrolled to create a deep
autoencoder, which is then fine-tuned using backpropagation of
error derivatives. Backpropagation modifies the weights of the
network to reduce the error of the teacher signal and the output
value when a pair of signals (input signal and the ideal teacher
signal) are given.
C. GMM-HMM
We used a standard GMM-HMM system trained and tested
with the Hidden Markov model toolkit (HTK) [36]. The acous-
tic model contains 5 states in total (3 of which are emitting)
TABLE I
WORD RECOGNITION RATE (%) IN MATCHED TRAIN/TEST SCENARIOS
ACHIEVED BY GMM-HMM SYSTEM USING DIFFERENT CEPSTRAL FEATURES
Matched train/test Matched train/test
Feature neutral/neutral whisper/whisper
MFCC 98.52 97.80
TECC 99.41 99.01∗
TEMFCC 99.27 98.13∗
MFCC + Δ 99.65 99.24
TECC + Δ 99.92 99.85∗
TEMFCC + Δ 99.86 99.28∗
MFCC + Δ + ΔΔ 99.77 99.53
TECC + Δ + ΔΔ 99.95 99.94∗∗
TEMFCC + Δ + ΔΔ 99.91 99.60∗
(p < 0.05 ∗ ; p < 0.01 ∗∗ ; p < 0.006 ∗∗∗; Confidence interval = 95%)
with strictly left-right structure and 16 mixture components.
The number of training cycles in embedded re-estimation was
restricted to 5 and the variance floor for Gaussian probability
density functions was set to 1%. The initial model parameters
were estimated using the flat-start method, in which models
were initialized with global mean and variance. In the testing
phase, Viterbi algorithm was applied to determine the most
likely model that best matched each test utterance.
V. RESULTS
The following subsections present results of isolated word
recognition obtained through 10-fold cross-validation.
A. ASR performance Evaluation—Matched
Train/Test Scenarios
This section presents the results of speaker-dependent word
recognition in matched train/test scenarios − neutral/neutral and
whisper/whisper, performed on the whole Whi-Spe database
and using GMM-HMM system without denoising autoencoder
features. The word recognition results achieved by GMM-HMM
system are presented in Table I.
As expected for matched data, the recognition efficiency in
both speech modes is very high. Although the “ceiling effect”
has been reached, the recognition accuracies show two slight
tendencies. Firstly, MFCC based features show a bit less recog-
nition rate in both speech modes compared to the other two
features. Secondly, in whisper recognition, the TECC based fea-
tures give the best results (TECC: 99.01%; TECC+Δ: 99.85%;
TECC+Δ+ΔΔ: 99.94%) which predicts the fact that TEO and
Gammatone filterbank better describe whisper characteristics. In
order to prove these statements, a statistical test was needed. For
this purpose, the two-tailed Wilcoxon signed-rank test was used
to analyze statistical significance of these small differences in
achieved average word recognition accuracies between MFCC
feature and other cepstral features. P-values for Wilcoxon test
are presented with asterisks in Table II. The obtained results con-
firm for all speakers that Teager-based features are statistically
significant (p<0.05) in whisper recognition. TECC+Δ+ΔΔ
feature shows the highest statistical significance among the other
features (p = 0.007). However, in neutral/neutral scenario the
choice of different features does not show any significance.
GROZDIĆ AND JOVIČIĆ: WHISPERED SPEECH RECOGNITION USING DEEP DENOISING AUTOENCODER AND INVERSE FILTERING
TABLE II
WORD RECOGNITION RATE (%) IN MISMATCHED TRAIN/TEST SCENARIOS
ACHIEVED BY GMM-HMM SYSTEM USING DIFFERENT CEPSTRAL FEATURES.
Feature Neutral/whisper Whisper/neutral
MFCC 45.19
TECC 54.42 ∗∗∗
TEMFCC 47.85 ∗
MFCC + Δ 59.67
TECC + Δ 65.29 ∗∗∗
TEMFCC + Δ 59.94 ∗
MFCC + Δ + ΔΔ 61.81
TECC + Δ + ΔΔ 71.33 ∗∗
TEMFCC + Δ + ΔΔ 63.71
(p < 0.05 ∗ ; p < 0.01 ∗∗ ; p < 0.006 ∗∗∗; Confidence interval = 95%)
B. ASR Performance Evaluation—Mismatched
Train/Test Scenarios
The usual problem of the ASR systems occurs when speaker
switches from neutral speech to whisper, or vice versa. Such a
problem is also noticeable and related to mismatched train/test
scenarios in the application of GMM-HMM systems. This ex-
periment investigates exactly that situation. The analysis was
carried out for the whispered speech recognition with the ASR
system that was trained to recognize neutral speech, and vice
versa, neutral speech recognition with the ASR system that was
trained to recognize the whispered speech. The averaged results
achieved by GMM-HMM without using denoising autoencoder
features for all speakers are presented in Table II.
In contrast to the matched train/test scenarios, mismatched
scenarios show significantly lower word recognition accuracies.
However, there are three important observations. Firstly, adding
dynamic features to their static features results in the significant
improvement of word recognition in both speech modes for all
tree speech features: MFCC, TEMFCC and TECC. Secondly, in
whisper/neutral scenario the TECC based features show much
higher word recognition rate than other two features. This is
even more noticeable in neutral/whisper scenario. For example,
in case of using static features alone, TECC achieves 54.42%
in word recognition accuracy compared to 47.85% and 45.19%
when TEMFCC and MFCC are applied respectively. When delta
features are added, TECC+Δ feature again shows better perfor-
mance with 65.29% of correct word recognition in contrast to
59.94% (TEMFCC+Δ) and 59.67% (MFCC+Δ). The best per-
formance is achieved when both delta and delta-delta features
are used, and in that case TECC+Δ+ΔΔ feature once again
demonstrates the best result with word recognition accuracy of
71.33% compared to 63.71% (TEMFCC) and 61.81% (MFCC).
These observations are statistically confirmed with two-tailed
Wilcoxon test (p-values are marked with asterisks in Table II)
which highlights once again that the TECC feature is the one that
best characterizes whisper. Thirdly, the TECC features show the
smallest difference between these two scenarios at the level of
6.02%, especially when delta, delta-delta features are added, in
comparison to MFCC with 11.43% and TEMFCC with 11.33%.
Better insight in the relation between all train/test scenarios is
illustrated in Fig. 5.
2319
62.13
68.51
63.57
67.22 Fig. 5. Word recognition rate (%) for different train/test scenarios in case of
73.25 expanded set of features (feature+Δ+ΔΔ).
69.43
73.24
77.35
75.04
Fig. 6. Confusion matrices of the word recognition in two train/test scenarios
(neutral/whisper on the left and whisper/neutral on the right) using MFCC
feature. The color bar scale to the right indicates probability ranges from 1
down to 0.
The relative relations of our results (particularly the better
recognition results in the whisper/neutral mismatch condition
over the neutral/whisper mismatch condition) are in agreement
with the results of the other two studies performed with the
HMM systems [3], [17]. Namely, in Ito’s work [3], HMM sys-
tem also gives higher word recognition rate in whisper/neutral
scenario (53%) compared to neutral/whisper scenario (27%).
The same tendency is observed in [18] where word recognition
rate in whisper/neutral scenario is 48.36% compared to 36.24%
in neutral/whisper scenario. This phenomenon will be discussed
in the following sections.
C. Confusion Analysis
This Section describes the analysis of confusion in word
recognition in mismatched train/test scenarios. As an illustration
Fig. 6 presents the word recognition confusion matrices when
MFCC feature was utilized.
The vertical axis on the matrices marks the input stimuli
words, while the horizontal axis denotes the output recognized
words. In the graphical representations of confusion matrices,
the gray scale on their right side indicates the accuracy of word
recognition. So, the darker shade denotes a higher possibility
in word recognition, where black indicates maximum probabil-
ity (p = 1) and white indicates minimal probability (p = 0).
The dark diagonals are clearly highlighted, and they represent
correctly classified words. In addition to the diagonals, slightly
brighter vertical lines and dots are observed, particularly in the
neutral/whisper scenario, which indicate distinct confusions be-
tween words in whisper and neutral speech.
The analysis of confusion in neutral/whisper scenario re-
vealed several highly misclassified stimuli words and some of
them are marked with arrows in confusion matrix. It was found
2320 IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 25, NO. 12, DECEMBER 2017

out that these confusions occur in most cases when two words
start with the same fricative and contain several consonants such
as /s/, /z/, /ts/ that are very similar in neutral and whisper mode.
For example fricative /z/ is voiced pair of /s/ and in whisper,
due to the lack of sonority, their confusion is very often because
of their similar articulation and spectral structure. However, the
mentioned confusions are not noticeable in whisper/neutral sce-
nario. The following question arises: Why don’t they occur in
the whisper/neutral scenario? The possible answer is that in
whisper/neutral scenario ASR system is trained with whisper
that has completely unvoiced sounds, while as an input comes
speech with both voiced and unvoiced sounds. In this situation,
for ASR system it is sufficient to match patterns of the unvoiced
sounds, while the voiced sounds are surplus. On the other hand, Fig. 7. Example of inverse filtering on a speech sample in neutral speech: (a)
in neutral/whisper scenario ASR system expects to get both FFT spectrum of a speech. (b) LPC spectral envelope. (c) FFT spectrum after
voiced and unvoiced sounds, but instead gets only unvoiced inverse filtering. (d) frequency response of inverse filter IF(z).
sounds of whisper. With these attitudes, whisper/neutral sce-
nario displays a higher recognition score than neutral/whisper
scenario (shown in Fig. 5), and consequently lower recognition
confusion, as indicated by Fig. 6.
However, the confusion matrices in the case of TECC feature
usage have a smaller number of misclassified words. This fact
points out that TECC feature is better for whisper recognition
than MFCC feature. The reason for this lower confusion level
is different kind of information extracted from the words in
speech and whisper as a consequence of using the TEO and the
Gammatone filterbank.
D. Reduction of Confusion Using Inverse Filtering
The phenomenon of better recognition results in the
whisper/neutral mismatch condition over the neutral/whisper Fig. 8. LTAS of Whi-Spe database, before and after inverse filtering.
mismatch condition has also been noted in two studies with
HMM systems [3], [18], but its deeper analysis has not been
performed. While looking for an explanation, the following where H(z) is the transfer function of vocal tract, ai are the LPC
hypothesis was brought - The ASR system that was trained coefficients of an utterance, and p = 10 is the order of the LPC
on whisper has better recognition in whisper/neutral scenario filter. It is known that inverse filter is reciprocal of the all-pole
than the ASR system that was trained on neutral speech in filter H(z) [37]. Hence, the frequency response of inverse filter is
scenario neutral/whisper, because the most of whisper features reciprocal to the LPC spectrum envelope of an utterance as it is
are contained in neutral speech, which is not the opposite case. illustrated in Fig. 7. In this way, inverse filter performs spectrum
Correspondingly, the ASR system that was trained on neutral flattening for each word from the Whi-Spe database.
speech was trained with speech features the basis of which The final result of inverse filtering can be perceived from the
is composed of voicing that does not exist in whisper, and long-term average spectrums (LTAS) of Whi-Spe database in
therefore whisper is less recognized. Fig. 8.
The way to test this hypothesis is to reduce the voicing influ- Comparing the shapes of LTAS before and after inverse filter-
ence in neutral speech, and to make it more similar to whisper ing, we can see that after the filtering, also known as spectrum
in terms of acoustic characteristics, and after such modification “whitening” [38], the spectrums of speech and whisper are flat-
to apply it in ASR training. According to Fig. 1, the whis- tened and more similar. To prove this achieved spectral similar-
pered speech spectrum is very flat. On the other hand, in neutral ity, the CDs between speech and whisper stimuli are measured
speech, voicing is dominant in domain of the first four formants once again. The average CD for all words before inverse filter-
of voiced phonemes, primarily vowels, i.e., at lower frequencies ing was 6.19 dB, while the new one has noticeable less value,
below 5 kHz. In order to make speech spectrum more simi- 3.45 dB.
lar to whisper, it is necessary to reduce this spectral tilt. For The changes of energy and spectral slope can be evaluated by
this purpose, in the pre-processing stage we added inverse filter c0 and c1 cepstral distributions that are presented in Fig. 9. As we
given by: can see, the energy level (c0 values) in both speech modes drops
off slightly because of the suppression of spectral components

p
IF (z) = 1/H(z) = 1 − ai z −i , (9) during inverse filtering (compare with Fig. 3). Nevertheless, the
i=1 c0 distributions are centered at the same position one above the
GROZDIĆ AND JOVIČIĆ: WHISPERED SPEECH RECOGNITION USING DEEP DENOISING AUTOENCODER AND INVERSE FILTERING
Fig. 9. Normalized distributions of c0 and c1 coefficients in neutral and
whispered speech after inverse filtering.
TABLE III
WORD RECOGNITION RATE (%) IN MISMATCHED TRAIN/TEST SCENARIOS
BEFORE AND AFTER INVERSE FILTERING ACHIEVED BY GMM-HMM
Neutral/ Whisper/
whisper neutral
Features Before After Before After
MFCC + Δ + ΔΔ 62.81 72.45∗∗∗ 73.24 75.25
TECC + Δ + ΔΔ 71.33 76.57∗∗∗ 77.35 79.21
TEMFCC + Δ + ΔΔ 63.71 71.42∗∗∗ 75.04 75.16
(p < 0.05 ∗ ; p < 0.01 ∗∗ ; p < 0.006 ∗∗∗; Confidence interval = 95%)
other. On the right side, c1 distributions are also centered at the
same position. This means that spectral slope in neutral speech
is now flattened and matched with the one in whispered speech.
The shift of c1 distribution in neutral speech is obvious when
compared Figs. 3 and 8. However, these spectral changes didn’t
harm the formant structure in terms of formant locations and
their visibility. With stimuli spectrum corrected in such a way,
the experiments in word recognition were repeated.
The new recognition results in matched scenarios achieved
by GMM-HMM after inverse filtering are very similar to those
in Table I. For example in neutral/neutral scenario the average
recognition accuracy is 99.82%, while in whisper/whisper is
99.76% (when TECC+Δ+ΔΔ are used). The new results of
mismatched scenarios are presented in Table III.
Three facts are important to be noticed in mismatched scenar-
ios. Firstly, all results of word recognition after inverse filtering
are better. According to p-values from Wilcoxon test, this im-
provement is statistically significant only in neutral/whisper sce-
nario, which is a consequence of suppression of voicing effects
during inverse filtering. Secondly, the TECC feature once again
shows the best accuracy, the maximum of 76.57%. Thirdly,
the difference in word recognition rates between mismatched
scenarios is much less. This difference is minimal for the TECC
feature and it is 2.64%. These results prove the hypothesis that
the voicing in speech stimuli is the main cause of difference in
word recognition accuracies in mismatched scenarios.
2321
TABLE IV
WORD RECOGNITION RATE (%) ACHIEVED BY PROPOSED ASR SYSTEM AND
APPLICATION OF DDAE-BASED CEPSTRAL FEATURES
Feature Neutral/neutral Neutral/whisper
MFCC + Δ + ΔΔ + DDAE 99.83 85.97 ∗∗
TECC + Δ + ΔΔ + DDAE 99.94 92.81 ∗∗
TEMFCC + Δ + ΔΔ + DDAE 99.87 87.73 ∗∗∗
DDAE (MFCC) 94.17 84.89 ∗∗
DDAE (TECC) 95.83 92.74 ∗∗
DDAE (TEMFCC) 95.47 85.97 ∗∗
(p < 0.05 ∗ ; p < 0.01 ∗∗ ; p < 0.006 ∗∗∗; Confidence interval = 95%)
E. Application of Denoising Autoencoder
As explained in previous sections, inverse filtering can be
easily applied for fast generation of pseudo-whisper samples
from neutral speech samples and applied in training of DDAE.
This section presents the results of a new proposed ASR frame-
work (illustrated in Fig. 4) which enables better whisper recog-
nition in mismatched scenarios. The results are presented in
Table IV. For simplicity, only the results of neutral-trained rec-
ognizer models are presented, which are the most important
Difference in from the standpoint of a real ASR application.
scenarios The results demonstrate three things. Firstly, the fusion
of standard cepstral features and DDAE-based cepstral fea-
Before After
tures considerably improves whisper recognition in mismatched
10.43 2.80 condition while keeping high performances in matched neu-
6.02 2.64
12.33 3.74
tral/neutral scenario. The obtained word recognition rates in
neutral/whisper scenario are 85.97%, 92.81% and 87.73% for
MFCC, TECC and TEMFCC features respectively. This im-
provement in word recognition accuracy in comparison with
the results from Table II is statistically confirmed with Wilcoxon
signed-rank test. Secondly, using only DDAE features proves
that improvement in mismatched scenario comes from DDAE
features itself and not from the larger number of parameters.
However, using only DDAE features shows slightly worse re-
sults in neutral/neutral scenario. Thirdly, TECC feature once
again demonstrates the highest score, proving to be the best
feature for whisper recognition.
VI. CONCLUSION
Lately applications of deep learning with its strong mod-
eling power helped to quickly spread the success of DDAE
in various industrial and machine learning tasks such as: im-
age recognition, optical character recognition, computer vision,
natural language and text processing, information retrieval, au-
tomatic speech recognition, etc. This paper extends this trend
and demonstrates how a DDAE can be applied to solve a com-
plex problem in ASR such as whispered speech recognition. We
propose a new and more advanced approach based on DDAE
for isolated word recognition, which can also be applied in
continuous speech recognition. Using a database of neutral
and whispered speech (Whi-Spe), this article evaluates whis-
pered speech recognition performance in mismatched train/test
conditions and demonstrates how DDAE with the help of in-
verse filtering can effectively filter out the effects of whispered
2322 IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 25, NO. 12, DECEMBER 2017
speech and reconstruct neutral cepstral features leading to sig-
nificant performance gain in whisper recognition. The efficiency
of this approach was demonstrated through a comparative study
with the conventional GMM-HMM speech recognizer and three
types of cepstral features: MFCC, TECC and TEMFCC.
Experimental results confirmed that the proposed model
presents several advantageous characteristics such as (i) sig-
nificantly lowered word error rate in mismatched train/test con-
ditions together with high performances in matched train/test
scenarios, (ii) easily acquirable whisper-robust features, and
(iii) no need for real whisper data in training process or model
adaptation. Furthermore, the experimental results demonstrate
that usage of Teager-based cepstral features outperforms tra-
ditional MFCC features in whisper recognition accuracy by
nearly 10%. Thus, combining TECC features with DDAE ap-
proach gives the best results and shows that the proposed frame-
work can considerably reduce recognition errors and improve
whisper recognition performance by 31% over the traditional
HTK-MFCC baseline and thereby achieve word recognition ac-
curacy of 92.81%.
REFERENCES
[1] S. T. Jovičić and Z. Šarić, “Acoustic analysis of consonants in whispered
speech,” J. Voice, vol. 22, pp. 263–274, 2008.
[2] R. Morris, “Enhancement and recognition of whispered speech,” Ph.D.
dissertation, School Elect. Comput. Eng., Georgia Inst. Technol., Atlanta,
GA, USA, 2003.
[3] T. Ito, K. Takeda, and F. Itakura, “Analysis and recognition of whispered
speech,” Speech Commun., vol. 45, pp. 139–152, 2005.
[4] B. P. Lim, “Computational differences between whispered and non-
whispered speech,” Ph.D. dissertation, Dept. Elect. Comput. Eng., Univ.
Illinois at Urbana-Champaign, Champaign, IL, USA, 2011.
[5] C. Y. Yang, G. Brown, L. Lu, J. Yamagishi, and S. King, “Noise-robust
whispered speech recognition using a non-audible-murmur microphone
with VTS compensation,” in Proc. 8th Int. Symp. Chinese Spoken Lang.
Process., Hong Kong, China, Dec. 2012, pp. 220–223.
[6] A. Mathur, S. M. Reddy, and R. M. Hegde, “Significance of parametric
spectral ratio methods in detection and recognition of whispered speech,”
EURASIP J. Adv. Signal Process, vol. 2012, p. 157, 2012.
[7] S. Ghaffarzadegan, S. Boril, and J. H. L. Hansen, “Generative modeling of
pseudo-target domain adaptation samples for whispered speech recogni-
tion,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., Brisbane,
Australia, Apr. 2015, pp. 5024–5024.
[8] S. Jou, T. Schultz, and A. Waibel, “Adaptation for soft whisper recognition
using a throat microphone,” in Proc. INTERSPEECH, Jeju Island, South
Korea, Oct. 2004, pp. 5–8.
[9] F. Tao and C. Busso, “Lipreading approach for isolated digits recogni-
tion under whisper and neutral speech,” in Proc. ISCA INTERSPEECH,
Singapore, Sep. 2014, pp. 1154–1158.
[10] D. Dimitriadis, P. Maragos, and A. Potamianos, “Auditory teager energy
cepstrum coefficients for robust speech recognition,” in Proc. EUSPICO,
Nice, France, Jul. 2005, pp. 3013–3016.
[11] P. Heracleous, “Using teager energy cepstrum and HMM distances,” Int.
J. Inf. Commun. Eng., vol. 5, no. 1, pp. 31–37, 2009.
[12] G. Zhou, J. Hansen, and J. Kaiser, “Classification of speech under stress
based on features derived from the nonlinear Teager energy operator,” in
Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., Seattle, WA, USA,
May 1998, pp. 549–552.
[13] B. Marković, S. T. Jovičić, J. Galić, and Đ. Grozdić, “Whispered
speech database: Design, processing and application,” in Proc. 16th
Int. Conf. Text, Speech, Dialogue, Pilsen, Czech Republic, Sep. 2013,
pp. 591–598.
[14] C. Zhang and J. H. L. Hansen, “Whisper-Island detection based on un-
supervised segmentation with entropy-based speech feature processing,”
IEEE Trans. Audio, Speech, Lang. Process., vol. 19, no. 4, pp. 883–894,
May 2011.
[15] S. Ghaffarzadegan, H. Boril, and J. H. L. Hansen, “UT-Vocal Effort II:
Analysis and constrained-lexicon recognition of whispered speech,” in
Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., Florence, Italy,
May 2014, pp. 2544–2548.
[16] T. Tran, S. Mariooryad, and C. Busso, “Audiovisual corpus to analyze
whisper speech,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process.,
Vancouver, BC, Canada, May 2013, pp. 8101–8105.
[17] P. X. Lee et al., “A whispered mandarin corpus for speech tech-
nology applications,” in Proc. INTERSPEECH, Singapore, Sep. 2014,
pp. 1598–1602.
[18] J. Galić, S. T. Jovičić, Đ. Grozdić, and B. Marković, “HTK-based recogni-
tion of whispered speech,” in Proc. Int. Conf. Speech Comput., Novi Sad,
Serbia, Oct. 2014, pp. 251–258.
[19] C. Zhang and J. H. L. Hansen, “Advancements in whisper-island detection
using the linear predictive residual,” in Proc. IEEE Int. Conf. Acoust.,
Speech, Signal Process., Dallas, TX, USA, Mar. 2010, pp. 5170–5173.
[20] S. Ghaffarzadegan, H. Boril, and J. H.L. Hansen, “Model and feature
based compensation for whispered speech recognition,” in Proc. INTER-
SPEECH, Singapore, Sep. 2014, pp. 2420–2424.
[21] H. Bořil and J. H. L. Hansen, “Unsupervised equalization of lombard
effect for speech recognition in noisy adverse environments,” IEEE Trans.
Audio, Speech, Lang. Process., vol. 18, no. 6, pp. 1379–1393, Aug. 2010.
[22] S. Ghaffarzadegan, H. Boril, and J. H. L. Hansen, “Generative modeling
of pseudo-target domain adaptation samples for whispered speech recog-
nition,” IEEE/ACM Trans. Audio, Speech, Lang. Process., vol. 24, no. 10,
pp. 1705–1720, Oct. 2016.
[23] Đ. T. Grozdić, B. Markovic, J. Galic, and S. T. Jovičić, “Application of
neural networks in whispered speech recognition,” in Proc. 20th Telecom-
mun. Forum, Belgrade, Serbia, Nov. 2012, pp. 728–731.
[24] S. T. Jovičić, Z. Kašić, M. Đorđević, and M. Rajković, “Serbian emotional
speech database: Design, processing and evaluation,” in Proc. 9th Conf.
Speech Comput., St. Petersburg, Russia, Sep. 2004, pp. 77–81.
[25] S. T. Jovicic, “Formant feature differences between whispered and
voiced sustained vowels,” Acta Acust. United With Acust., vol. 84,
pp. 739–743, 1998.
[26] X. Fan and J. H. L. Hansen, “Speaker identification within whispered
speech audio streams,” IEEE Trans. Audio, Speech, Lang. Process.,
vol. 19, no. 5, pp. 1408–1421, Jul. 2011.
[27] C. Zhang and J. H. L. Hansen, “Analysis and classification of speech
mode: Whispered through shouted,” in Proc. INTERSPEECH, Antwerp,
Belgium, Aug. 2007, pp. 2396–2399.
[28] K. J. Kallail and F. W. Emanuel, “Formant-frequency differences between
isolated whispered and phonated vowel samples produced by adult female
subjects,” J. Speech Hear. Res., vol. 27, pp. 245–251, 1984.
[29] J. F. Kaiser, “Some useful properties of Teager’s energy operators,” in
Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., Minneapolis, MN,
USA, Apr. 1993, pp. 149–152.
[30] H. M. Teager, “Some observations on oral air flow during phonation,”
IEEE Trans. Acoust., Speech, Signal Process., vol. 28, no. 5, pp. 599–601,
Oct. 1980.
[31] P. Maragos, J. F. Kaiser, and T. F. Quatieri, “Energy separation in signal
modulations with application to speech analysis,” IEEE Trans. Signal
Process., vol. 41, no. 10, pp. 3024–3051, Oct. 1993.
[32] E. Kvedalen, “Signal processing using the Teager energy operator and
other nonlinear operators,” Candies Scientific Thesis, Univ. Oslo, Norway,
2003.
[33] P. Vincent, H. Larochelle, Y. Bengio, and P. A. Manzagol, “Extracting and
composing robust features with denoising autoencoders,” in Proc. 25th
Int. Conf. Mach. Learn., Helsinki, Finland, Jul. 2008, pp. 1096–1103.
[34] Đ. T. Grozdić, S. T. Jovičić, J. Galić, and B. Marković, “Application of
inverse filtering in enhancement of whisper recognition,” in Proc. IEEE
Neural Netw. Appl. Elect. Eng., Belgrade, Serbia, Nov. 2014, pp. 157–162.
[35] G. E. Hinton, “Training products of experts by minimizing contrastive
divergence,” Neural Comput., vol. 14, pp. 1771–1800, 2002.
[36] S. Young et al., The HTK Book (for HTK Version 3.2). Cambridge Univ.,
Eng. Dept., Cambridge, U.K., 2002.
[37] C. J. Leggetter and P. C. Woodland, “Flexible speaker adaptation us-
ing maximum likelihood linear regression,” in Proc. ARPA Spoken Lang.
Technol. Workshop, Austin, TX, USA, Jan. 1995, pp. 110–115.
[38] D. Havelock, S. Kuwano, and M. Vorländer, Handbook of Signal Process-
ing in Acoustics. New York, NY, USA: Springer, 2008.
Authors’ photographs and biographies not available at the time of publication.