Intelligent Systems with Applications 10–11 (2021) 20 0 049

Contents lists available at ScienceDirect

Intelligent Systems with Applications

journal homepage: www.elsevier.com/locate/iswa

A Sequence-to-Sequence Approach for Remaining Useful Lifetime

Estimation Using Attention-augmented Bidirectional LSTM
Sayed Rafay Bin Shah a,∗, Gavneet Singh Chadha a, Andreas Schwung a, Steven X. Ding b
a
Department of Automation Technology, South Westphalia University of Applied Sciences, Luebecker Ring 2, Soest 59494 Germany
b
Department of Automatic Control and Complex Systems, University of Duisburg-Essen, Bismarckstrasse 81, Duisburg 47057 Germany

a r t i c l e i n f o a b s t r a c t

Article history: We propose a novel sequence-to-sequence prediction approach for the estimation of the remaining use-
Received 31 May 2021 ful lifetime (RUL) of technical components. The approach is based on deep recurrent neural network
Revised 8 July 2021
structures, namely bidirectional Long Short Term Memory (LSTM) networks, which we augment with an
Accepted 20 July 2021
attention mechanism to allow for a more fine-grained information flow between the input and output
Available online 31 July 2021
sequence. Using the base architecture as a reference, we experiment with various forms of attention
Keywords: mechanisms as well as different forms of additional input embeddings. Further, we analyse the impact of
Remaining useful lifetime estimation the sequence length on the estimation quality. We apply our approach to the well known C-MAPSS data
Encoder-decoder networks set previously serving as a benchmark dataset for RUL prediction. We obtain state of the art results on
Bidirectional long-short term memory the data set and provide a thorough hyperparameter study that underlines, that more simple but well
Attention mechanism tuned architecture can achieve comparable or better performance than highly complex architectures.
© 2021 The Authors. Published by Elsevier Ltd.
This is an open access article under the CC BY-NC-ND license
(http://creativecommons.org/licenses/by-nc-nd/4.0/)

1. Introduction predictive maintenance can be a challenging and time-consuming

The requirement for manufacturing environments and machines
to run in a 24/7 setting poses high production management chal-
lenges. Due to the required high workload, machines’ uninten-
tional standstill should be avoided, while intentional standstill
times should be reduced as much as possible. A common approach
to prevent unintentional standstill is the incorporation of recur-
ring maintenance actions in the production schedule. This peri-
odic maintenance prevents machines from severe breakdowns but
results in the machine’s potential downtimes due to unnecessary
maintenance actions. Predictive maintenance approaches provide a
solution based on a thorough condition monitoring Martin (1994);
Nandi et al. (2005) which generally uses a model of the degra-
dation process in the component allowing for a more detailed
prediction of future downtimes Isermann (2005). This predictive
maintenance results in various advantages such as economic inven-
tory management for spare parts, prediction-based planned main-
tenance of failure-prone equipment, conditional monitoring and
life cycle optimisation of a system. However, deriving models for
∗ ing Goodfellow et al. (2016).
Corresponding author.
E-mail address: shah.sayedrafaybin@fh-swf.de (S.R.B. Shah).
task.
In parallel, the upcoming digitisation augments the modern
production environment with more and more information about
its machines’ actual status. Using various sensor readings and mea-
surements all over the process allows for a detailed view of the
overall plant condition, continuous monitoring of the production
process, and assists in analysing the production’s possible weak-
nesses. Consequently, the obtained data can be used for data-
based maintenance purposes. Various approaches have been pro-
posed for data-based maintenance, see Jardine et al. (2006); Tsui
et al. (2015) for overviews. One of the critical methodologies of
data-based maintenance is estimating the remaining useful life-
time (RUL) of a machine using historical data Lei et al. (2018).
RUL is the estimation of the duration of an item, product, com-
ponent, or a system for its intended purpose before it needs to be
replaced. RUL estimation includes modelling the degradation pat-
tern in a system which can arise due to either normal operation
(no fault symptoms) or after the detection of a fault. Therefore, an
efficient RUL estimation approach guarantees a safe operation of a
product till till the end of its life. Among the various approaches
for data-driven RUL estimation, neural networks appeared to be of
particular use due to recent advances in the field of deep learn-

https://doi.org/10.1016/j.iswa.2021.20 0 049
2667-3053/© 2021 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/)
S.R.B. Shah, G.S. Chadha, A. Schwung et al.
Neural networks have been researched in different application
fields including modelling Esen et al. (2008); Hikmet Esen et al.
(2009, 2017), forecasting Hikmet Esen et al. (2008a,b), machine
health monitoring Zhao et al. (2019a) and RUL estimation. Mainly,
convolutional neural networks (CNN) Yann LeCun et al. (1990) and
recurrent neural networks (RNN) such as long short term memo-
ries (LSTM) Hochreiter and Schmidhuber (1997), and gated recur-
rent units (GRU) Junyoung Chung et al. (2014) have provided more
dynamic and efficient ways of performing RUL estimations.
This paper introduces a novel deep learning architecture for re-
maining useful lifetime estimation that operates directly on raw
data sets. Notably, we propose to apply sequence-to-sequence ar-
chitectures Sutskever (Vinyals) best known from natural language
processing tasks to extract the complex degradation pattern inher-
ent to RUL estimation from a machine’s sensor readings. To this
end, we opt for a bidirectional LSTM as the base network, which
we operate in an encoder-decoder (ED) style manner. Following the
recent improvements to encoder-decoder architectures provided by
attention mechanisms, we augment the architecture with different
attention modules differing mainly in mapping input and hidden
variables to the attention keys and queries. Further, we provide a
detailed discussion on the impact of different ED architecture hy-
perparameters on the RUL estimation performance and the input
encoding required for the sequence processing. The approach is
applied to the well known C-MAPSS Saxena et al. (06/10/2008 -
09/10/2008) benchmark dataset and compared with other meth-
ods. The results obtained with the proposed architecture perform
superior or on par with actual state-of-the-art approaches under-
lining the approach’s applicability. Particularly, we show that even
simpler architectures can result in comparable or better perfor-
mance than more complex architectures with a high number of
parameters.
The contributions of the paper can be summarised as follows:
• We present a novel approach for remaining useful life-
time estimation based on attention augmented sequence-
to-sequence neural networks using bidirectional LSTMs. The
sequence-to-sequence framework appears highly suitable to
cover the inherent nonlinear trend analysis problem under-
lying RUL estimations.
• A novel Shared Kernel Convolution (SKC) Neural Network ap-
proach as an encoder to the bidirectional LSTM has been
proposed to model the inter-dependencies among the differ-
ent input variables.
• A universally applicable sliding window approach for target
sequence generation has been presented to better represent
the health status of a system.
• We discuss architectural variations, including different atten-
tion mechanisms operating on the hidden representation of
the bidirectional LSTM and various types of input embed-
dings. Further, we analyse the impact of the sequence length
on the prediction accuracy and provide a thorough analysis
of hyperparameters, resulting in guidelines on architectural
components’ suitability.
• We apply the approach to the well known C-MAPSS data
set serving as a very challenging benchmark study in pre-
vious works. We report state-of-the-art results on the data
set with comparably lightweight and straightforward archi-
tectures.
The paper is organised as follows. Section 2 discusses work re-
lated to our approach. Section 3 presents preliminaries concerning
the considered neural network structures. Section 4 presents the
novel approach for RUL estimation using RNNs followed by the de-
scription of the training data set generation for the Sequence-to-
Sequence NN in 5 and the dataset description 6. In Section 7 we
2
Intelligent Systems with Applications 10–11 (2021) 200049
provide results on the C-MAPPS data set. Section 8 concludes the
paper.
2. Related work
We present an overview on remaining useful lifetime estima-
tion methods with particular emphasis on the recently prevalent
deep learning approaches while we refer to Jardine et al. (2006);
Tsui et al. (2015) for overviews on classical techniques. A review
on the application of deep learning approaches for prognostics and
health management is presented in Khan and Yairi (2018).
Notably, two types of Deep Neural Networks (DNN) have been
used in RUL estimation, namely RNNs and CNNs. Additionally, var-
ious architectures combining both network types have been pro-
posed in the literature. In Zheng et al. (2017), a stacked LSTM
network combined with multiple fully connected (FC) layers has
been presented while in Wang et al. (2018) a stacked bidirec-
tional LSTM network with additional Fully Connected (FC) lay-
ers for RUL prediction of C-MAPSS dataset is shown. Alternatively,
Zhao et al. (2019b) first construct trend features that are fed to
a stacked LSTM network for predicting RULs. The approach pro-
posed in Listou Ellefsen et al. (2019) introduces an unsupervised
pre-training stage using Restricted Boltzmann Machine (RBM) to
extract complex raw input features and subsequently employ a ge-
netic algorithm to tune the hyperparameters. Following the pre-
training, supervised training is performed for RUL prediction. A
vanilla LSTM neural network model along with dynamic differen-
tial technology to get good RUL prediction accuracy is proposed
in Wu et al. (2018). A bidirectional LSTM model for RUL estimation
of the C-MAPPS dataset is presented in Elsheikh et al. (2019).
To further strengthen RNN-based encoder-decoder layers’ learn-
ing abilities, attention mechanisms have been proposed in Dzmitry
Bahdanau et al. (2015) and Luong et al. (2015) to allow each de-
coder state to attend to all the encoder hidden states before gen-
erating the next output. Please refer to Chaudhari et al. (2019b) for
a detailed overview of the different types of attention methods.
While these approaches concentrate on machine translation tasks,
Chen et al. (2021) use such attention mechanism for RUL predic-
tion, where handcrafted features are extracted, concatenated with
the LSTM output and fed to a regression layer. The attention-based
LSTM encoder-decoder network proposed in Ragab et al. (2020) re-
constructs the input data and generates RUL using a parallel RUL
predictor layer. The RUL predictor layer receives dual latent fea-
ture representations, i.e., the attended encoder latent representa-
tions and the latent decoder representations as inputs.
A survey of 1D CNN models for condition monitoring in ma-
chines is presented in Kiranyaz et al. (2021). The first use of
CNNs for RUL estimation dates back to Sateesh Babu et al. (2016),
while Li et al. (2018) performs convolution operation with ker-
nels of unit width. These unit width convolution kernels allow for
kernel weights sharing across raw sensors, thereby enhancing the
network’s ability to learn abstract feature information. Recently,
a generalized dilation CNN methodology for RUL estimation has
been proposed in Gavneet Singh Chadha et al. (2021) to model
long term time dependencies. An attention-based CNN approach
is proposed in Tan and Teo (2021) where the CNN extract features
across multiple temporal axes fed to an attention layer to predict
the RUL. The work in Tan and Teo (2021) replaces the Softmax ac-
tivation with a Sigmoid activation in the attention mechanism to
add multivariate RUL estimation features. The Global Luong con-
catenation method to calculate attention alignment scores is pro-
posed in Paulo Roberto De Oliveira Da Costa et al. (2020) for RUL
prediction for the C-MAPSS dataset. An attention CNN-LSTM ar-
chitecture is introduced in Zhang et al. (2017), where a CNN is
used for raw input feature extraction before feeding into a stacked
LSTM network. The LSTM output from all time-steps is attended
S.R.B. Shah, G.S. Chadha, A. Schwung et al. Intelligent Systems with Applications 10–11 (2021) 200049

to the last hidden state via an attention layer to generate a rota-

tory machine’s RUL finally. The study Kong et al. (2019) proposes
a hybrid CNN-LSTM RUL predictor where raw sensor data is pre-
processed to generate a 1D-health indicator matrix. The study Peng
et al. (2021) extracts spatial and temporal features from CNN and
LSTM networks that are fused and passed to another CNN layer to
predict RULs of C-MAPSS sub-datasets FD001 and FD003. Another
hybrid network of parallel CNN and LSTM paths is proposed in Li
et al. (2019b), to reduce the influence of CNN extracted features on
series-connected LSTMs. However, the model adds LSTM network
that processes the previous CNN and LSTM paths’ fused outputs
and predicts RUL values.
Encoder-decoder models for C-MAPSS RUL prediction are pro-
posed in two separate works by Liu et al. (2019) and Liu et al.
(2021). In the model presented in Liu et al. (2019), the encoder
is made up of stacked bidirectional LSTM layers, followed by mul-
tiple CNNs with an intermediate pooling, and the decoder is sim-
ply a network of three FC layers. The proposed architecture in Liu
et al. (2021) utilises a self-attention mechanism for feature extrac-
tion. The feature-attended output is fed to a bidirectional GRU-CNN
encoder, followed by a decoder with flattening and FC layers.
In contrast to existing methods, this study proposes a Fig. 1. Structure of the LSTM cell.
Sequence-to-Sequence (Seq2Seq) RUL estimation model wherein
not a point estimate of the RUL is predicted, rather a sequence
estimate is predicted. Furthermore, this work compares the differ-
ent attention methods and encoder models for the RUL estimation
ct = C˜t  it + ct−1  ft , (4)
task, which is missing in the literature. It must also be noted, that
most of the good performing architectures generally require a high
amount of learnable parameters, which renders their application
difficult for small RUL data sets. Therefore, we also provide a thor- ot = σ (Woxt + Roht−1 + bo ), (5)
ough comparison of the number of parameters of our proposed
models and the models provided in the literature.
ht = ψ (ht )  ot , (6)
3. Recurrent Neural Networks Architectures
where, xt , ht−1 and ct−1 are the new input, previous hidden and
previous cell states respectively. The input gate it from Eq. (1) de-
We consider long-short term memory networks as the back-
termines how much new input information should be added or
bone network architecture for processing sequential data and CNN
left out while creating the new cell state. Contrarily, the forget
for input encoding. Particularly, we employ bidirectional LSTMs,
gate ft from Eq. (2) decides how much memory from the pre-
first as they generally provide better performance scores, second
vious state should be retained or forgotten. In Eq. (3), a can-
as they allow for a change of view on the sequence prediction task
didate for the new cell state, C˜t is generated. Results of point-
as discussed in Gavneet Singh Chadha et al. (2020). Note that other
wise multiplication between it , C˜t and, ft , ct−1 are added to
bidirectional, gated RNNs like GRUs can be used similarly.
generate a new cell state, ct in Eq. (4). The output gate, ot in
Eq. (5) determines how much of the new information should be
3.1. Long-Short Term Memory
used as instantaneous LSTM cell output. New hidden state, ht
is formed by passing the previous hidden state, ht−1 through a
Originally, LSTMs have been proposed in Hochreiter and
hyperbolic tangent activation, ψ , and then performing pointwise
Schmidhuber (1997). Recently, LSTMs have shown great success in
multiplication with the newly generated output, ot as shown in
various domains, including speech recognition and machine trans-
Eq. (6). The weight matrices Wc , Wi , W f , Wo ∈ RH×I , Rc , Ri , R f , Ro ∈
lation. The complete network is obtained similarly to the classical
RNN by stacking LSTM layers. However, the structure of the neural RH×H and vectors bc , bi , b f , bo ∈ RH are the input, recurrent and
units varies considerably. A schematic of the LSTM is illustrated in bias weights, respectively where H denotes the hidden dimension
Fig. 1. As can be seen, the LSTM includes three gates, namely the of LSTM blocks and I is the dimensionality of the input vector. The
input it , forget ft and output gate ot , a cell state ct and σ , γ and initial states are given by h0 ∈ RH and c0 ∈ RH .
ψ are pointwise activation functions. In the vanilla LSTM, logistic In contrast to the classical RNN, additional memory cells ct are
sigmoid functions σ (x ) = 1/1 + e−x are used for the gate activa- introduced in the LSTM framework. Additionally, gates are incor-
tions, while the hyperbolic tangent γ (x ) = ψ (x ) = tanh(x ) is used porated to control the information flow from input, previous time
as candidate cell state and new hidden state activations. step and state, allowing the memory cell to keep its full informa-
The vector equations for the LSTM layer in the forward pass can tion even over many time steps. Hence, the LSTM can remove or
be written as add information during long periods.

it = σ (Wi xt + Ri ht−1 + bi ), (1)

3.2. Bidirectional LSTM

Bidirectional LSTM (BiLSTM) structures use two independent

ft = σ (W f xt + R f ht−1 + b f ), (2)
LSTM networks, where the first network computes data in the
−
→
usual forward sequential order, ht and the second network in the
←
−
C˜t = γ (Wc xt + Rc ht−1 + bc ), (3) reverse order, ht Graves and Schmidhuber (2005). At any timestep,

3
S.R.B. Shah, G.S. Chadha, A. Schwung et al.
Fig. 2. Structure of the bidirectional LSTM.
the output from each of the forward and backward cells are con-
catenated to produce a single output, yt . A bidirectional LSTM net-
work is shown in Fig. 2. Both the forward and reverse LSTM net-
works are initiated with the same hidden and cell states. At any
timestep t, the equations involved in a BiLSTM cell are Yu et al.
(2015),
−−→ − → ←−− ← − dimensional movement of CNN-1D kernels over time Albawi et al.
yˆt = Why · ht + Why · ht + by
−−→ ←−−
where, Why and Why denote the forward and reverse hidden to out-
put weights and by denote the output bias.
By providing a reversed copy of the input data, the individual
LSTM cells can learn the context from future information. Hence
the network, at any time, can process both the past and future
information compared to the unidirectional LSTM’s ability to pro-
cess only past information. Additionally, BiLSTMs are computation-
ally inexpensive since they use conventional BPTT training method
like LSTM networks Graves and Schmidhuber (2005); Lipton et al.
(2015). This work makes use of the bidirectional feature in all the
designed LSTM networks.
3.3. Shared Kernel Convolutional 2D Neural Networks
Convolutional 2D neural networks (CNN-2D) apply a two-
dimensional moving filter across 3D-input data to create output
feature maps by capturing the data’s spatial and temporal rep-
Fig. 3. Shared Kernel Convolutional 2D Neural Network.
Intelligent Systems with Applications 10–11 (2021) 200049
resentations. We use CNN-2D architectures similar to Li et al.
(2018) for extracting temporal features across each spatial dimen-
sion shown in Fig. 3 which serves as an input encoding upstream
of the LSTM.
Feature maps Nft refer to the no. of channels present in a cer-
tain data. As we consider time-series data, the no. of input chan-
nels equals to 1. After each 2D-convolutional layer, the features
extracted from the input are mapped by the corresponding filters
into multiple channels through a non-linear activation function. In
Fig. 3, the input is a 3D data with timesteps, Ntw = t1 , t2 ,., tT , fea-
tures, Nft = f1 , f2 ,., fF and, a channel size of 1.
The information extracted from this input by a CNN-2D(1)
(1 )
N ×N ×N
layer is mapped into an output of shape Z1 ∈ R f m tw f t , with
Nfm (1) feature maps Li et al. (2018); Wu (2017). The no. of
timesteps, Ntw and no. of features, Nft are preserved by the use of
padding.
Filters or kernels move over the input data and extract in-
formation into output feature maps. In CNN-2D architectures, a
kernel is 2-dimensional in shape (KL ×KW ), and they move over
both the temporal and feature dimensions, compared to the 1-
(7)
(2017). KL and KW are the height and width of the kernel. The
number of kernels involved in a convolutional layer also depends
on the no. of feature maps. For Nfm =2, there will be two ker-
nels of the same size, extracting input information to map into
their respective feature maps. This is illustrated in Fig. 3. These
kernels contain trainable weights, and they help extract local or
global information based on their size compared to the input
sample. The shared-kernel (SK) 2D-CNN architecture used in this
work requires the kernel to extract temporal information from
each sensor data individually and finally create feature maps based
on all the sensors’ information. This is accomplished by setting
the kernel’s width, KW to 1, i.e., each of the kernels spans over
only one feature dimension at a time. This structure is shown in
Fig. 3, where the kernels on the input data are marked in red
and blue.
For a kernel of size (KL ×1), spanning from the i-th timestep up
to the (i+KL -1)-th timestep, of an input sequence x = x1 , x2 ,., xT
with features, f = 0, 1,., F, the learned feature representation from
temporal dimension along a single feature, Zi f mapped from on a
4
S.R.B. Shah, G.S. Chadha, A. Schwung et al.
Fig. 4. LSTM ED Sequence RUL Predictor.
single output feature map is given by Gu et al. (2018),
Zi f = [W T · xi:i+KL −1 + b] f
where W is the kernel weight from the input to the output fea-
out
×N f m in ×KL ×KW
ture map and, W ∈ RN f m , b is the bias and b ∈
out
Nfm
R . In the case of SK 2D-CNN with one input channel, W ∈
out
RN f m ×1×KL ×1 . For n = 0,1,... N output feature maps, the final con-
volved output is given by,
0:N
Cout = [Zi0 , Zi1 , Zi2 , ., ZiF ]0:N
The convolution output is flattened and passed through
a fully connected (FC) layer with weight matrix, Wflatten ∈
R(N f m ·Ntw ·N f t )×(Ntw ·N f t ) in order to combine all the extracted infor-
mation across all feature maps. The FC output is then reshaped to
create a vector of the original input size, i.e., RNtw ×N f t .
4. Sequence-to-sequence Neural Network based RUL estimation
In this section, we present the novel sequence-to-sequence neu-
ral network architecture for RUL estimation. To this end, we first
give an overview of the proposed architecture and subsequently,
we explain the individual components in detail.
4.1. Overview on the sequence-to-sequence architecture
Fig. 4 shows the architecture of the novel proposed sequence-
to-sequence prediction based neural network. As its core, we pro-
pose an encoder-decoder structure where a bidirectional LSTM rep-
resents the encoder. Simultaneously, the decoder is composed of a
unidirectional LSTM and a fully connected layer to form the out-
put of the network predicting a sequence of RUL estimation val-
ues. Additionally, we propose using the attention mechanism that
augments the network with relevance weights between the infor-
mation on each decoder time-step and the encoder’s entire hidden
state representation. We experiment with two types of encoder-
decoder attention mechanisms, namely Dzmitry Bahdanau et al.
(2015) and Luong et al. (2015). Optionally, we further add in-
put encoding provided by a shared kernel CNN as described in
Section 3.3. In the following, we discuss the components of the
architecture in detail.
4.2. Encoder-Decoder sequence-to-sequence predictor
As the core architecture of our proposed Sequence-to-Sequence
RUL Predictor, we employ an LSTM-based encoder-decoder (LSTM
Intelligent Systems with Applications 10–11 (2021) 200049
ED) structure. The ability of sequence mapping of variable input-
output lengths, i.e., Ti =To is exploited in the model shown in Fig. 4.
(8)
The encoder is a BiLSTM network where the initial hidden and cell
states are initialized with zeros. After processing the input infor-
−
→ ←−
mation in recurrent method, the final hidden states, hT and hT
from both directions are concatenated to form a latent representa-
tion, s0 . At any direction, hT ∈ RL×de where L denotes the number
of stacked LSTM layers (L=1 in our case), and de denotes encoder
hidden dimension. Therefore, the combined latent representation
(9) is s0 ∈ R1×2de . Complex degradation pattern information from the
input sequence is encoded in this latent representation, which is to
be decoded and mapped into corresponding RUL points. Hence, the
hidden state of the first LSTM cell in the decoder is initialized by
this encoded latent representation. The input to the first decoder
LSTM cell is a tensor of zeros, which serves as a start-of-sequence
< SOS > token. The output from every decoder LSTM cell is passed
through a FC layer with weight, W∈ Rdd ×1 , where dd is the decoder
hidden dimension and 1 is the RUL dimension. The output is the
predicted RUL in that decoder timestep and it is passed on as the
input to the next decoder LSTM cell. This recurrent operation con-
tinues until outputs from all pre-defined number of decoder LSTM
cells, To are generated. The FC weights and biases are shared across
all the decoder LSTM timesteps.
4.3. Shared CNN encoder-decoder sequence-to-sequence predictor
As an input of the multivariate time series input data results
in potentially high dimensional encoder hidden dimension, we op-
tionally augment the encoder BiLSTM with an additional input em-
bedding provided by a CNN-2D architecture with shared kernels
(SKC) as introduced in Section 3.3. The combined model is referred
to in this work as the Shared-Kernel Convolutional LSTM Encoder-
Decoder (SKC LSTM ED) sequence RUL predictor and is shown in
Fig. 5.
The CNN layer potentially extracts complex features from the
raw input before feeding them to the BiLSTM encoder. The SKC
layer kernel is of length, KL and unit width, KW = 1. The no. of
feature maps are Nfm . Non-linear Leaky ReLU activation is applied
after the CNN layer. Zero padding of appropriate lengths is applied
along each feature map’s temporal edges to prevent information
loss in the sliding kernels. The CNN layer’s output is flattened to
rearrange the information from the extracted feature maps in a
single column. The flattened tensor is then passed through a FC
layer with weight W ∈ R(N f m ·Ntw ·N f t )×(Ntw ·N f t ) , where, Ntw and Nft
are the no. of timesteps and no. of features respectively in the in-
5
S.R.B. Shah, G.S. Chadha, A. Schwung et al.
Fig. 5. SKC LSTM ED Sequence RUL Predictor.
put feature map. The output is then reshaped to a form that re-
sembles the shape of the input feature map. This is done to enable
the bidirectional LSTM encoder to perform the recurrent operation
on an equivalent number of input time-steps and features as in the
raw input tensor.
4.4. Attention Mechanism in Neural Networks
Attention mechanism in deep neural networks refers to “At-
tending to” certain parts of the input for generating output. As dis-
cussed in the previous sections, vanilla RNN or LSTM structures use
the input data’s temporal dynamics and map it to sequential out-
put data. However, there is still the question of relevance between
the output generated at a particular time-step and the input se-
quence used to generate that output. Even though LSTM reduces
the effects of vanishing and exploding gradients for very long se-
quences, it does not eliminate them entirely. Moreover, NN archi-
tectures such as RNN, LSTM or CNN can fall short of processing
highly complex feature representations to generate accurate out-
puts. The attention mechanism counters this problem by finding
the relevance between the output data it needs to generate and
the encoder’s hidden state representations. The attention structure
creates a context vector from encoder hidden states. It then allows
the decoder to use the context vector to generate a more precise
and relevant output Chaudhari et al. (2019a); Dzmitry Bahdanau
et al. (2015); Luong et al. (2015); Swati Meshram (2019). Hence,
attention structures are simply an addition to the existing encoder-
decoder networks. The different types of attention methodologies
and their use in this work are detailed in the following sections.
4.4.1. Bahdanau Attention
The attention model proposed by Dzmitry Bahdanau et al.
(2015) calculates the decoder output, yˆ j at any time j based on the
i=1,2,...,T encoder timesteps as follows:
1. Alignment Score, Sij
Si j = Walign · tanh · [(Wh · h̄i ); (Ws · s j−1 )]
where Wh ∈ Rde ×dd
and Ws ∈ Rdd ×dd
are the weight matri-
ces for all encoder hidden states, h̄i and the corresponding
previous decoder hidden state, sj-1 respectively and tanh is
ex −e−x
the hyperbolic tangent function with tanh(x ) = ex +e−x
2. Attention Weights, Wij
Wi j = So f tmax(Si j )
Intelligent Systems with Applications 10–11 (2021) 200049
where
exi
So f tmax(xi ) = (12)

K
xj
e
j=1
3. Context Vector, Cij
Ci j = WiTj · h̄i (13)
4. Attention Combined, Idecoder, j-1 This step performs a
weighted concatenation with weight matrix, Wapplied ∈
R(dd +dRUL )×dd , between the context vector, Cij and the
decoder output from previous timestep, yˆ j−1 .
Idecoder, j−1 = Wapplied · [Ci j ; yˆ j−1 ] (14)
And finally,
yˆ j = LSTM((Idecoder, j−1 ), s j−1 ) (15)
where LSTM is the decoder in the model.
It can be deduced from Eq. (10)–(15), that the Bahdanau atten-
tion model computes the relevance of all encoder hidden states,
h̄i w.r.t the previous decoder hidden state, sj-1 to decide the in-
put, Idecoder, j-1 for the current decoder timestep. The input is passed
through the decoder LSTM cell at that timestep to generate an out-
put, yˆ j . Moreover, this method deploys a weighted additive align-
ment technique Dzmitry Bahdanau et al. (2015). These are the key
differences between this attention technique and the Luong atten-
tion model which is detailed in the next section.
4.4.2. Luong attention
The Luong attention model deploys an approach similar to
Bahdanau attention, which considers all encoder hidden states to
calculate the context vector and attention weights Luong et al.
(2015) also called the global attention method. However, this
(10) model also suggests using a more localised approach, where the
context and attention weights are computed by focusing on a
windowed segment of the encoder hidden states. Such attention
is known as local attention or window-based attention since the
mechanism deploys a window of a certain length upon the encoder
states, depending upon a position token, pi received from the cur-
rent decoder time-step. The selection of the window length and its
(11) centre can be made in two ways Luong et al. (2015):
6
S.R.B. Shah, G.S. Chadha, A. Schwung et al.
Fig. 6. LSTM ED Sequence RUL Predictor with Bahdanau Attention.
• Monotonic Alignment: A fixed window length is selected
where the center point, pi depends upon the current de-
coder position. The window moves along with the progress
of each decoder time-step to the next. For an encoder with i
timesteps and decoder with j timesteps, the window length,
Lwindow in the local attention models in this work is calcu-
lated as,
Lwindow = i − j + 1
• Predictive Alignment: This method predicts the alignment
window by applying a Gaussian distribution centred around
pt . Predictive alignment is not applied in this work.
Regardless of global or local attention, the Luong attention
model also follows the four previously mentioned steps but in a
different manner. For simplicity, the steps are shown as follows
concerning global attention Luong et al. (2015).
1. Alignment Score, Sij The alignment score, Sij in Luong atten-
tion model can be calculated by any of the following three
methods:
⎧ i.e. degradation information, for the corresponding RUL prediction
⎨h̄i · s j , Dot
Si j = Wh (h̄i ) · s j , General
⎩
tanh(Walign .[h̄i ; s j ] ), Concat enat e
Where, Wh ∈ Rde ×dd , h̄i is the complete encoder hidden
states and sj is the current decoder hidden state.
2. Attention Weights, Wij
Wi j = So f tmax(Si j )
3. Context Vector, Cij
Ci j = WiTj · h̄i
4. Attention Combined, yˆ j First, the current decoder LSTM
timestep, Dj generates a candidate output, ȳ j based on the
previous decoder output, yˆ j−1 and hidden state, sj-1 .
ȳ j = LSTM(yˆ j−1 , s j−1 )
Then, a weighted concatenation with weight matrix, Wapplied
∈ R(de +dd )×dRUL is performed between the context vector, Cij
and the candidate decoder output from the current timestep,
ȳ j to generate the final attended output, yˆ j .
yˆ j = Wapplied · [Ci j ; ȳ j ]
Intelligent Systems with Applications 10–11 (2021) 200049
It is evident from Eq. (17)–(21), that the Luong attention model
computes the relevance of all encoder hidden states, h̄i w.r.t the
current decoder hidden state, sj , and candidate output, ȳ j to decide
the output, yˆ j for the current decoder timestep. This method also
provides the flexibility to deploy multiplicative, weighted multi-
plicative and weighted additive alignment calculations Luong et al.
(2015). The possibility of aligning the decoder representations with
a windowed segment of the encoder hidden states reduces com-
(16) putation cost and focuses more on the relevant input informa-
tion. Global and local attention techniques, along with the differ-
ent alignment formulas, have been used in developing encoder-
decoder architectures in this work, and their performances will be
compared in the experimental section.
4.5. Attention enhanced encoder-decoder predictor
As previously discussed, the attention mechanism generates rel-
evance weights between the information on each decoder time-
step and the encoder’s entire hidden state representation. Hence,
it allows stronger emphasis on the beneficial input information,
and improves the prediction performance. We test the proposed
(17) models with the two types of encoder-decoder attention mecha-
nisms proposed in Dzmitry Bahdanau et al. (2015) and Luong et al.
(2015). In both architectures, the encoder remains the stated BiL-
STM network with zero initialisation of the hidden and cell states.
The final hidden states are concatenated and then used to initialise
the decoder hidden state. The LSTM ED architectures with atten-
tion are explained in the following.
(18)
4.5.1. RUL-Prediction using Bahdanau Attention
The proposed LSTM ED RUL predictor with Bahdanau attention
(19) is shown in Fig. 6. The mechanism of the Bahdanau Attention block
is implemented according to Section 4.4.1. At any decoder timestep
sj-1 , an alignment score is created between the entire encoder hid-
den representation, h̄i ∈ RTi ×2de and the instantaneous decoder hid-
den state sj-1 ∈ R1×dd . This score is softmaxed and element-wise
(20) multiplication is performed between the softmaxed score and h̄i to
generate a context vector. A weighted addition between the con-
text vector and the instantaneous decoder output, yˆ j−1 creates the
input for the next decoder cell, sj . Bahdanau attention is performed
to create attended inputs for LSTM operation. Hence, every decoder
timestep output is its respective predicted RUL, as well as the can-
(21) didate to be passed to the attention block for generating the next
7
S.R.B. Shah, G.S. Chadha, A. Schwung et al.
Fig. 7. LSTM ED Sequence RUL Predictor with Luong Attention.
timestep input. Thus, for every recurring decoder timestep, the at-
tention mechanism generates attended inputs for that decoder cell
using the predicted RUL and hidden state from the previous step,
until the end of the sequence is reached. The collective outputs
from all decoder timesteps represent the predicted sequence RUL.
4.5.2. RUL-Prediction using Luong Attention
The proposed LSTM ED RUL predictor with Luong attention is
shown in Fig. 7. Luong Attention mechanism is implemented ac-
cording to Section 4.4.2. At any decoder timestep sj-1 , an alignment
score is created between the entire encoder hidden representa-
tion, h̄i ∈ RTi ×2de and the instantaneous decoder hidden state, sj-1 ∈
R1×dd . This score is softmaxed and element-wise multiplication is
performed between the softmaxed score and h̄i to generate a con-
text vector. The decoder cell sj-1 creates a candidate output and a
weighted addition between the candidate output and the context
vector generates the final predicted RUL, yˆ j−1 . This predicted RUL,
yˆ j−1 also serves as the input to the next decoder timestep sj . Thus
for every candidate output generated by a decoder timestep, Lu-
ong attention block creates a final predicted output and the next
timestep input by attending to the encoder hidden representation
with respect to the candidate output.
Luong attention mechanism can also attend to a localised seg-
ment of the encoder hidden states instead of the global hidden
states by performing local attention. In local attention mode, a po-
sitional token, Pj is sent to the attention block from the decoder
hidden state based on which a local window is generated using
monotonic alignment (Section 4.4.2). The window ”rolls over” the
encoder hidden states with the progression of each decoder time-
step until the end of the sequence is reached. The collected RULs
from the decoder’s recurrent operation is the sequence RUL pre-
diction of the Luong attention-based LSTM ED network.
4.6. Teacher Forcing
In encoder-decoder models dealing with sequence-to-sequence
predictions, teacher forcing (TF) is a technique used in training that
feeds the ground truth label from the previous time-step instead
of the model predicted value as the input for processing the sub-
sequent time-step output. TF is a commonly used feature in NLP
applications and is used in recurrent learning in RNNs Williams
and Zipser (1989). NN models that heavily depend on previously
8
Intelligent Systems with Applications 10–11 (2021) 200049
decoded outputs can often suffer from slow convergence due to
processing faulty predictions at the initial training stages. The ac-
cumulation of error for more extended periods can be avoided by
feeding the true label instead of the predicted outcome. Apart from
recurrent models, TF can also be used in auto-regressive models,
e.g, transformer since, it also operates on the principle of encoder-
decoder structure. However, studies in Bengio et al. (2015); Goyal
et al. (2016) have shown models to be suffering from ”Exposure
Bias” during inference where the model fails to provide accurate
and stable predictions due to its high dependency on exposed
ground truth labels. As the model moves farther away from its own
predicted outputs, the discrepancy between training and inference
increases. This can be avoided by setting a TF-free running ratio,
where the model will be trained for a certain period with right-
shifted ground truth labels and for the rest of the time with its
predictions from the previous time step.
5. Sliding Window approach and sequence target RUL
generation
Training the sequence-to-sequence model in this work requires
feeding the input data in batches of sub-sequences. These sub-
sequences optimise the model’s learning process since it learns to
map the corresponding health status from a short sequence of lo-
calised data Ye and Keogh (2009). This sub-sequences creation is
achieved by using sliding windows, where a window of a specific
length moves across the input signal over time, captures the infor-
mation at a certain instance and feeds it to the model to predict
the corresponding RUL. The step size of the window is termed in
this work as stride. Additionally, the novelty of this work is the
generation of RUL outputs in a sequence instead of a single data
point. This is proposed based on the hypothesis that a model can
better represent an engine’s health status in a sequence of RUL
labels. Sudden degradation or fluctuations occurring at a specific
input time window can be more easily mapped to a sequence of
RULs instead of averaging the learned information into a unit. The
RUL labels created by using the piece-wise linear function are un-
folded to a specific size to achieve this. This size depends on the
length of the target sliding window, v. The space between two tar-
get RUL labels is filled with v-1 equidistant interval points. Hence,
an output RUL sequence from the initial to the final cycle of length,
S.R.B. Shah, G.S. Chadha, A. Schwung et al. Intelligent Systems with Applications 10–11 (2021) 200049

Table 1
C-MAPSS Dataset Sensor Parameters Saxena et al. (06/10/2008
- 09/10/2008).

Sensor Parameter Unit

◦
Fig. 8. Unfolded Target Labels for Sequence RUL generation with v = 5. S1 Total temperature at fan inlet R
◦
S2 Total temperature at LPC outlet R
◦
S3 Total temperature at HPC outlet R
◦
S4 Total temperature at LPT outlet R
S5 Pressure at fan inlet psia
S6 Total pressure in bypass-duct psia
S7 Total pressure at HPC outlet psia
S8 Physical fan speed rpm
S9 Physical core speed rpm
S10 Engine pressure ratio –
S11 Static pressure at HPC outlet psia
S12 Ratio of Fuel Flow to S12 parameter pps/psi
S13 Corrected fan speed rpm
S14 Corrected core speed rpm
S15 Bypass Ratio –
S16 Burner fuel-air ratio –
S17 Bleed Enthalpy –
S18 Demanded fan speed rpm
S19 Demanded corrected fan speed rpm
S20 HPT coolant blend lbm/s
S21 LPT coolant blend lbm/s

dataset Saxena et al. (06/10/2008 - 09/10/2008). The Turbofan En-

Fig. 9. Sliding Window operation and corresponding Sequence target RULs.
to is unfolded to a sequence of length, To = to × v. An example is
illustrated in Fig. 8 for v = 5.
An illustration of sliding window operation and corresponding
sequence target RULs is shown in Fig. 9. Consider a sensor input
data, I ∈ RTi , and corresponding unfolded RUL labels O ∈ RTo , of
lengths Ti and To respectively. u and v are the sliding window
lengths of the input and target outputs. If Ti =To , then, u=v. Let,
the maximum number of times the respective windows slide over
the input and output data is N. The input sliding window always
follows a constant stride of 1 whereas, the output window stride is
equal to the length of the output window i.e., v. This is performed
to ensure that the input sequences fed to the model contain a con-
tinuous and non-interrupted flow of temporal information. Hence,
the edge of the input window slides over one cycle at a time. To
map this accurately to the output RUL sequence, the final point in
an output window must contain the integer RUL value correspond-
ing to the windowed input data, and its preceding values in that
RUL window are the equidistant interval points.
6. Dataset and Evaluation Metrics
In the following, we discuss the results obtained for the RUL es-
timation framework. We first present the considered data set, fol-
lowed by showing the results and comparing the existing state of
the art.
6.1. C-MAPSS Dataset
Commercial Modular Aero-Propulsion System Simulation or C-
MAPSS is a model-based simulation program prepared by NASA.
This program provides the simulation of a large commercial tur-
bofan engine used in aircraft propulsion Frederick et al. (2007).
The C-MAPSS program was used to generate the Turbofan En-
gine Degradation Simulation Data Set and the PHM 2008 Challenge
gine Degradation dataset (referred to as C-MAPSS dataset hence-
forth) is used to evaluate the model performances in this work.
The C-MAPSS program was simulated in four different work-
ing environments to generate the C-MAPSS dataset with four sub-
datasets, namely FD001, FD002, FD003 and FD004. Each sub-
dataset consists of a train and a test dataset, with corresponding
RUL data containing the last run-time for each test engine data.
In each of the four sub-datasets, the engines start operating in a
healthy state and run until the system fails due to gradual degra-
dation. In the test samples, the sensor measurements are trimmed
from a certain period (as provided in the RUL database) before sys-
tem failure. That corresponding period is taken for evaluation and
scoring. Initial wear is considered in all the train, and test engines
Chatterjee and Litt (2003); Saxena et al. (06/10/2008 - 09/10/2008).
Each of the train and test C-MAPSS sub-datasets contains out-
puts from 26 sensors (Table 1). These measurements are sensor
responses to 14 different health parameter inputs that simulate
degradation scenarios in any engine’s rotating components Saxena
et al. (06/10/2008 - 09/10/2008). For simplicity, the sensor data are
referred to as the way they appear in the sensor columns in the
raw dataset, i.e., the column ”Sensor” in Table 1. The engines in
the C-MAPSS dataset are operated under different operational set-
tings and fault modes. Sub-datasets FD001 and FD003 work with
one operational setting, resulting in one unique operating condi-
tion whereas, FD002 and FD004 operate with three different oper-
ational settings. The three operational settings result in six unique
operating conditions. This variation in operating conditions creates
fluctuations in sensor readings since data points in consecutive cy-
cles may not belong to the same operating condition. The opera-
tional settings, no. of resulting sub-conditions and fault modes for
each sub-datasets, are shown in Table 2.
It is evident from the raw sensor data that not all signals
show a clear degradation trend that may optimise the performance
of an NN model. Sensors S1, S5, S6, S10, S16, S18, S19 show ei-
ther a non-continuous or monotonous pattern in their readings.
These monotonous readings are excluded from all train and test
sub-datasets. Additionally, the inclusion of the history of operat-
ing regimes as features along with the selected sensor readings to
optimise model performance is suggested in Sateesh Babu et al.
(2016). The approach is implemented for the sub-dataset FD002.

9
S.R.B. Shah, G.S. Chadha, A. Schwung et al. Intelligent Systems with Applications 10–11 (2021) 200049

Table 2
Operational Information of C-MAPSS Dataset .

Sub-Dataset Total Cycles Operational Settings Operating Conditions Fault Modes

Train Test

FD001 100 100 Sea Level 1 HPC Degradation

FD002 260 259 Altitude, Throttle Resolver Angle, Mach 6 HPC Degradation
FD003 100 100 Sea Level 1 HPC Degradation Fan Degradation
FD004 249 248 Altitude, Throttle Resolver Angle, Mach 6 HPC Degradation Fan Degradation

each engine under a sub-dataset is provided in a separate database.

The no. of healthy cycles, cycleshealthy of an engine, e of sub-dataset,
d is determined as
(e,d ) (e,d )
cycleshealthy = cyclestotal + RUL(fe,d ) − RULi (22)

cyclesd(e,d )
egrading
(e,d )
= cyclestotal (e,d )
− cycleshealthy . (23)

6.3. Root mean-Squared error (RMSE)

Loss functions represent how a predicted output from a NN

Fig. 10. Piece-wise Linear Function for RUL Labels.
Further, we use the Min-Max Scaler approach to scale the data
within a range of [-1, 1]. However, due to the six different oper-
ating conditions in FD002 and FD004, the magnitude and distri-
bution of the data points within each condition vary and hence,
they require to be scaled separately Bektas et al. (2019). The in-
dividual sensor data from each of the six conditions are segre-
gated, scaled and then combined according to their original pre-
scaling positions. The same scaling technique is separately applied
to FD002 and FD004. Since FD001 and FD003 operate under one
operating condition, the multi-regime scaling is not required in
these datasets.
model compares against the actual desired output. These functions
are differentiable and are used to calculate the gradients in order
to update weights while training. The most common loss functions
used in regression are the Mean-Squared Error (MSE), Root Mean-
Squared Error (RMSE), and Mean Absolute Error (MAE) Botchkarev
(2019).
The RMSE loss function is used for sequence RUL prediction
task with C-MAPSS dataset in this work and is defined as

1
N
RMSE = (yˆi − yi )2 (24)
N
i=1
where N is the number of predicted samples, yˆi and yi are the pre-
dicted and target outputs respectively.
The predicted sequence RUL for a specific engine consists of val-
ues in the decimal range, beginning after the preceding RUL se-
quence and ending with the final RUL for that cycle e.g. [194.8,
194.6, 194.4, 194.2, 194]. This is explained in detail in the Section 5.
Additionally, RMSE is an important metric for comparison against
other model performances proposed in the literature for the C-
MAPSS dataset.

6.2. Piece-wise RUL target function

6.4. RUL Evaluation Metrics
It is not desirable to hypothesize a linearly degrading life cy-
The scoring functions Saxena et al. (06/10/2008 - 09/10/2008)
cle since the beginning of the operation, as it implies that a ma-
for the regression task of the C-MAPSS dataset is given by
chine deteriorates linearly with usage. A suitable alternative to lin-  di
early degrading RUL is suggested in Heimes (2008), where the life-
e a1 − 1, di < 0
cycle is divided into two distinct phases- the healthy and constant si = di (25)
phase and, the linearly degrading phase. This approach is shown e a2 − 1, di ≥ 0
in Fig. 10, where it sets a limit to the maximum possible RUL. The
machine operates with a healthy status for a certain number of 
N
S= si (26)
cycles and then, starts linearly degrading to the end RUL point.
i=1
This approach is called the piece-wise linear RUL function estima-
tor. Upon experimentation in Heimes (2008), the suitable maxi- where, a1 = 13, a2 = 10, di = Predicted RUL - Target RUL for the
mum RUL limit was found to be between 120 and 130 cycles. In i-th testing engine, si = score for the i-th engine, N = total no.
this work, this is termed as the start-of-life or initial RUL, RULi of engines in a test sub-dataset and S is the score for that sub-
and is taken as 120 cycles. The point at which the machine tran- dataset Li et al. (2018). The scoring function for C-MAPSS dataset
sitions from the healthy phase to the degrading phase is termed can be termed as an inverse scoring function i.e. a lower cumula-
as the knee-point, Pointknee and it varies for different engines since tive score signifies better model performance and a higher score
the number of cycles, cyclestotal for each engine is different. In the signifies the opposite. Hence, late predictions are penalized more
training dataset, the end-of-life or final RUL, RULf is at 0 for all than early predictions for an Error of the same absolute magni-
engines. In the test dataset, since the sensor data feed for the en- tude. Additionally, the scoring function depends solely on the RUL
gines are stopped at certain points before functional failure, RULf generated for the final input window, and not on the RULs for the
and Pointknee are not the same throughout all the engines. RULf for earlier stages of operation.

10
S.R.B. Shah, G.S. Chadha, A. Schwung et al. Intelligent Systems with Applications 10–11 (2021) 200049

Table 3
Hyperparameters for ED RUL Predictors.

Hyperparameters Range

Attention Models Non-Attention Models

Predicted RUL Length (TRUL ) {10, 15, 20} {10, 15, 20}
Batch Size (BS) {1024} {256, 512}
SKC Kernel Length (KL ) {3, 5} {3, 5}
Encoder Hidden Size (de ) {32, 48, 64} {40, 64}
Decoder Hidden Size (dd ) {64, 96, 128} {80, 128}
Learning Rate (LR) {1e-1, 5e-2, 1e-2} {5e-2, 1e-2}
Teacher Forcing (TF%) {0, 30, 40} {0, 30, 40}

Table 4
Best Hyperparameters for ED Models.

Hyperparameters Model Architectures

LSTM Bahd. AT ED SKC LSTM Bahd. AT ED LSTM ED

TRUL 10 10 20
BS 1024 1024 512
KL - 3 -
de 32 64 40
dd 64 128 80
LR 1e-2 1e-2 1e-2
TF% 30 40 0

7. Results and Comparison

In this section, we present the results of the proposed architec-

tures on the C-MAPPS dataset. We first report results for the differ-
ent attention and encoder variations. Second, we present ablation
studies on the impact of different hyperparameter settings. Third,
we compare the results with existing approaches in the literature.
Unless stated otherwise, the Adam Kingma and Ba (2015) opti-
mizer is used for the gradient descent algorithm and for any initial
learning rate (LR), a milestone of 125 epochs is set after which,
the initial LR is reduced to its 10% of value. The attention aug-
mented models are trained for 200 epochs while the non-attention
for 150 since they started to overfit the training set. All the weights
and biases are initialized using Xavier Uniform Glorot and Ben-
gio (2010) distribution. The hyperparameters and their respective
ranges for training the different ED Seq2Seq RUL predictor mod-
els are shown in Table 3. The table includes the details for at-
tention as well as non-attention models. These ranges for hyper-
parameters were based on a random search Bergstra and Bengio
(2012) methodology. The input sequence length, Ti is chosen to be
30, 20, 38 and 18 for the sub-datasets FD0 01, FD0 02, FD0 03 and
FD004 respectively.
All the models are tested with the different hyperparameters
and the best performing hyperparameters for each of the models
are shown in Table 4. In the next subsections, we will underline
the impact of some of these hyperparameters.
Fig. 11. LSTM Encoder Decoder network with different types of Attention.
mance can be observed from Luong Dot product Global attention
and Luong Concat Global attention. The model performance deteri-
orates when local attention is used. Therefore, it can be concluded,
that for this RUL estimation task, the LSTM Bahadanau AT ED per-
forms the best. Therefore, this model along with its variants will
be used for further analysis in this work.
7.2. Effect of Attention and No Attention with encoder decoder
architectures
The LSTM ED model without attention is evaluated in this sec-
tion. The different hyperparameters for this model are shown in
Table 3. As illustrated in Fig. 12, the model performance with at-
tention improves significantly on three of the four sub-datasets
when compared to the LSTM ED model without attention. There-
fore, the attention augmented models improve the predictive per-
formance of the Seq2Seq models.

7.1. Effect of type of attention with encoder-decoder architectures
The performance of the LSTM encoder-decoder model with dif-
ferent attention types (LSTM AT ED) is evaluated in this section.
The LSTM AT ED model is separately evaluated with Bahdanau at-
tention and the three variants of Luong attention (Dot product,
General and Concatenation) for both Global(Gl.) and Local (L) at-
tention operations. The hyperparameters for each of the models
are shown in Table 4. As illustrated in Fig. 11, the model with
Bahdanau attention performs excellently and much better than
the other attention types. A very significant improvement is ob-
served on sub-datasets FD0 01, FD0 03 and FD0 04 compared to the
scores from previous models while maintaining a very good score
in FD002. Among the Luong attention variants, the best perfor-
7.3. Comparison among different encoder structures in the
encoder-decoder architectures
The SKC LSTM ED model adds a feature extracting SKC layer on
top of the LSTM Bahadanau AT ED layer as discussed in Section 3.3.
The best performing hyperparameters for achieving the optimum
performance from this model are shown in Table 4.
The inclusion of a CNN layer requires new additional hyperpa-
rameters to be optimized. A fixed output feature map, N f m = 8 is
selected while kernel lengths of 3 and 5 are inspected. All other
training parameters remain the same as the LSTM ED model. The
scores in the SKC LSTM Bahdanau AT model deteriorate compared
to the LSTM Bahdanau AT ED model since it fails to strike a balance
in any case throughout the four sub-datasets. Therefore, it can be

11
S.R.B. Shah, G.S. Chadha, A. Schwung et al. Intelligent Systems with Applications 10–11 (2021) 200049

Fig. 12. Analysis of Bahdanau Attention in LSTM Encoder Decoder Network.

Fig. 14. Effect of Hidden Sizes (de , dd ) in LSTM Bahdanau Attention Encoder De-
coder Network.

Fig. 13. Analysis of SKC Layer in LSTM Bahdanau Attention Encoder Decoder Net-
work.

concluded, that the added parameters from the convolutional layer

harm the performance in this case.

7.4. Effect of changing hidden sizes in the encoder-decoder

architectures
The hidden dimension sizes, de and dd , are of significant impor-
tance in attention models. This is seen in Fig. 14 where the LSTM
Bahdanau AT ED model displays a higher sensitivity to de and dd .
Keeping all other hyperparameters constant, increasing de and dd
deteriorates the model performance across all four sub-datasets.
Again, the added overload of the number of parameters in the
LSTM layer leads to a degradation in the generalization capability
of the model.
7.5. Effect of output sequence length on encoder-decoder
architectures
The length of the output sequence length, TRUL is a hyperpa-
rameter of significant importance in attention models as it repre-
sents the output sampling frequency. In the case of the C-MAPSS
dataset, the output sequence length of 10 works the best across all
Fig. 15. Effect of predicted sequence RUL lengths (TRUL ) in LSTM Bahdanau Atten-
tion Encoder Decoder Network.
operating conditions. A comparison among the different lengths is
illustrated in Fig. 15 where increasing the sequence length reduces
the performance across all the operating conditions.
7.6. Effect of Teacher Forcing on encoder-decoder architectures
The importance of teacher forcing (TF) while training the LSTM
Bahdanau AT ED model is shown in Fig. 16. There is no clear trend
to be detected in this case as increasing the TF% sometimes de-
grades the performance but also can lead to an improvement in
performance on some operating conditions. Therefore, no conclu-
sive evidence on the effect of TF can be provided here.

12
S.R.B. Shah, G.S. Chadha, A. Schwung et al. Intelligent Systems with Applications 10–11 (2021) 200049

Table 5
Comparison of Model Performance (Scores) with Related Literature.

Model Description Score

FD001 FD002 FD003 FD004

Deep LSTM Zheng et al. (2017) 338 4450 852 5550

Stacked BiLSTM Wang et al. (2018) 295 4130 317 5430
Handcrafted Features + LSTM Chen et al. (2021) 322 - - 5649
CEEMD + DLSTM Zhao et al. (2019b) 262 6953 452 15069
RBM + LSTM Listou Ellefsen et al. (2019) 231 3366 251 2840
CNN Sateesh Babu et al. (2016) 1287 13,570 1596 7886
CNN + Attention Tan and Teo (2021) 198 1144 251 2072
HICNN-LSTM-NN Kong et al. (2019) 303 3440 1420 4630
FCLCNN Peng et al. (2021) 204 - 234 -
BLCNN Liu et al. (2019) 302 1558 381 3859
DAG Li et al. (2019a) 229 2730 535 3370
DCNN Li et al. (2018) 274 10,412 284 12466
AGCNN (Self-Att.) Liu et al. (2021) 225 1492 227 3392
LSTM + Luong Att. Paulo Roberto De Oliveira Da Costa et al. (2020) 320 2102 223 3100
BiLSTM 473 1223 676 2684
LSTM ED 254 898 243 2161
SKC LSTM ED 296 1182 185 2096
LSTM Bahd. AT ED 215 999 187 2081
LSTM Luong Concat (Gl.) AT ED 228 1063 214 2234
SKC LSTM Bahd. AT ED 318 1682 216 2111
SKC LSTM Luong(Gl.) AT ED 280 1224 200 2228
SKC LSTM Luong(Loc.) AT ED 294 1751 244 2462

encoder, which is decoded to predict a sequence of RULs. The in-

Fig. 16. Effect of Teacher Forcing (TF%) in LSTM Bahdanau Attention Encoder De-
coder Network.
7.7. Qualitative Proposed Model Performance Analysis
The advantage of predicting sequence RULs with encoder-
decoder models can be observed by comparing the scores be-
tween the LSTM ED architecture to the vanilla BiLSTM model in
Table 5. The significant improvement in performance can be at-
tributed to the generation of latent space representation by the
clusion of a Bahdanau attention layer in the LSTM ED helps the
model to find the most relevant input for the next decoder step.
Conversely, the Luong attention layer focuses its operation on the
decoder output to generate a final output using attention opera-
tion. According to Table 5, the best set of balanced scores between
the proposed models can be found in the LSTM Bahdanau AT ED
model whereas, the LSTM Luong Concat (Gl.) AT ED model dete-
riorates in performance. The decoder LSTM cells perform a much
better task at generating final RULs when fed with relevant infor-
mation from Bahdanau attention, instead of generating intermedi-
ate output from raw encoder latent representations and allowing
the Luong attention layer to produce final RULs.
The inclusion of a feature extracting SKC layer to the LSTM
ED model at the cost of increased number of trainable parame-
ters, fails to improve the model performance any further. As ob-
served in Table 5, this persists even when SKC LSTM ED models
are enhanced with attention layers. In every case, the LSTM en-
coder tends to perform much better when fed with raw sequential
input data rather than a feature extracted input from a preceding
SKC layer. An example of RUL predictions by the proposed models
compared to the actual RUL of engine ID 100 of sub-dataset FD001
are plotted in Fig. 17. The BiLSTM model follows the overall degra-
dation pattern of the engine, however, the predicted RULs deviate
significantly from the true RULs throughout the cycles (Fig. 17a).
The LSTM ED model improves this deviation with certain outliers
in the healthy phase of the prediction horizon (Fig. 17b). The pre-
diction capability of the model improves when incorporated with
a Bahdanau attention layer (Fig. 17c). The LSTM Bahdanau AT ED
model also minimizes the occurrence of a large over or under-
prediction, which is not observed in the performance of other ar-
chitectures. Further models such as the LSTM Luong Cat Gl. AT ED
(Fig. 17d), SKC LSTM ED (Fig. 17e) and SKC LSTM AT ED models
with Bahdanau and Luong attention layers (Fig. 17f-h), also main-
tain a good prediction capability, however, with increased fluctua-
tions throughout the prediction horizon.
The testing loss comparison between the LSTM ED model, the
LSTM Bahdanau AT ED model and LSTM Luong Concat Gl. AT ED
is shown in Fig. 18. Although all three models eventually converge
to the same minimum loss, the main difference lies in the initial
epochs. Luong attention takes place at the end segment of every

13
S.R.B. Shah, G.S. Chadha, A. Schwung et al. Intelligent Systems with Applications 10–11 (2021) 200049

Fig. 17. Remaining Useful Lifetime (RUL) Estimation Plots by Proposed Models for sub-dataset FD001 Engine ID 100.

14
S.R.B. Shah, G.S. Chadha, A. Schwung et al. Intelligent Systems with Applications 10–11 (2021) 200049

Fig. 18. Testing Loss comparison between L STM ED, L STM Bahdanau AT ED and L STM Luong Concat Gl. AT ED architectures for all C-MAPSS sub-datasets.

Fig. 19. Testing Loss comparison between L STM Bahdanau AT ED and SKC L STM Bahdanau AT ED architectures for all C-MAPSS sub-datasets.

decoder cell, allowing each candidate RUL to directly attend to the
encoder hidden representations. This evidently benefits the model
as it begins performing at a much lower loss magnitude than com-
pared to LSTM ED or LSTM Bahdanau AT ED. LSTM AT ED mod-
els, especially with Bahdanau attention, appear to converge to the
minimum loss magnitude in the latter epochs than compared to
the vanilla LSTM ED model. The testing loss comparison between
LSTM Bahdanau AT ED and SKC LSTM Bahdanau AT ED is shown
in Fig. 19. While both models require a fair amount of epochs to
converge to a minimum loss point, the SKC LSTM Bahdanau AT
ED accomplishes this faster compared to the LSTM Bahdanau AT
ED. This reflects the contribution of a feature extracting SKC layer
that helps the model to reach early convergence and predict RULs
that are closer to the actual values from an earlier stage. These
loss plots reflect the overall prediction capability of the models
throughout the life cycle of the engines. These, however, do not
represent the score-based performance of the model architectures
since, the scoring function depends on the final RUL values of each
engine instead of the RUL at every instance.
7.8. Comparison with existing literature
The sequence RUL predictor models presented in this work
are evaluated against the literature model performances based on
their generated scores (Table 5) for the C-MAPSS Turbofan En-
gine Dataset. As evident in Table 5, the proposed LSTM ED and
the LSTM Bahdanau AT ED model outperforms the best literature
score Tan and Teo (2021) for FD002. The proposed SKC LSTM ED
and LSTM Bahdanau AT ED model outperform the best literature
score Paulo Roberto De Oliveira Da Costa et al. (2020) for FD003.

15
S.R.B. Shah, G.S. Chadha, A. Schwung et al. Intelligent Systems with Applications 10–11 (2021) 200049

Table 6
Comparison of Model Performance (RMSE) with Related Literature.

Model Description RMSE

FD001 FD002 FD003 FD004

Deep LSTM Zheng et al. (2017) 16.14 24.49 16.18 28.17

Stacked BiLSTM Wang et al. (2018) 13.65 23.18 13.74 24.86
Handcrafted Features + LSTM Chen et al. (2021) 14.53 _ _ 27.08
CEEMD + DLSTM Zhao et al. (2019b) 14.72 29 17.72 33.43
RBM + LSTM Listou Ellefsen et al. (2019) 12.56 22.73 12.10 22.66
CNN Sateesh Babu et al. (2016) 18.45 30.29 19.82 29.16
CNN + Attention Tan and Teo (2021) 11.48 17.25 12.31 20.58
HI CNN-LSTM-NN Kong et al. (2019) 16.13 20.46 17.12 23.26
FCLCNN Peng et al. (2021) 11.17 _ 9.99 _
BLCNN Liu et al. (2019) 13.18 19.09 13.75 20.97
DAG Li et al. (2019a) 11.96 20.34 12.46 22.43
DCNN Li et al. (2018) 12.64 22.36 12.64 23.31
AGCNN (Self-Attention) Liu et al. (2021) 12.42 19.43 13.39 21.50
LSTM + Luong Attention Paulo Roberto De Oliveira Da Costa et al. (2020) 13.95 17.65 12.72 20.21
Transformer Encoder + Gated CNN Mo et al. (2021) 11.27 22.81 11.42 24.86
BiLSTM 15.87 16.59 15.10 14.36
LSTM ED 11.67 15.78 10.86 12.98
SKC LSTM ED 11.71 16.76 10.05 13.62
LSTM Bahdanau AT ED 11.69 15.47 11.42 13.24
LSTM Luong Concat (Global) AT ED 13.23 17.60 10.98 13.92
SKC LSTM Bahdanau AT ED 12.30 18.51 9.86 13.78
SKC LSTM Luong Concat (Global) AT ED 12.02 15.60 9.8 14.11
SKC LSTM Luong Concat (Local) AT ED 12.04 17.33 9.5 13.8

are shown in Table 6. Even in this case, the proposed LSTM Bah-
danau AT ED model RMSE is either very close to, or much lower
than the lowest RMSE from the literature for all sub-datasets. In
addition, the attention-based SKC LSTM ED also produce excellent
losses, reflecting the ability of the attention mechanism to attend
to specific parts of the input for predicting nearly accurate RULs
throughout the prediction horizon.
The average score on all the operating conditions in the C-
MAPSS dataset in relation to the total number of trainable param-
eters in the best performing models from literature and our best
performing model is shown in Fig. 20. The best performing LSTM
Bahdanau AT ED based models require a similar number of train-
able parameters as the ones in literature but performs drastically
better with an average score of 847. The obtained results are supe-
rior to the current state of the art.

8. Conclusions

Fig. 20. Average score of our best architecture LSTM Bahdanau Attention Encoder
Decoder Network on the C-MAPSS dataset compared to existing work in literature.
The performance of the LSTM Bahdanau AT ED model for FD001
and FD004 is very closely matched to the best literature score
in Tan and Teo (2021).
A thorough observation of Table 5 confirms that the proposed
LSTM Bahdanau AT ED model outmatches every literature model
when providing a highly balanced performance throughout all the
C-MAPSS sub-datasets. Among the literature, the attention-based
temporal CNN architecture Tan and Teo (2021) provides a good
overall performance. Additionally, an overall comparison between
attention and non-attention architectures in Table 5 shows the sig-
nificant improvement yielded by the attention-based models. The
Transformer encoder model with Gated CNN Mo et al. (2021) does
not deploy the scoring function as a performance metric and
hence, scores from this literature are not available for comparison.
A distinctive analysis can be made by comparing the models
based on their performances throughout the engines life cycles.
This is evaluated by the RMSE losses of the models during the test-
ing phase. The RMSE for all the implemented and literature models
In this paper we propose a novel approach for remaining useful
lifetime estimation based on sequence-to-sequence LSTM encoder-
decoder structures augmented with Bahdanau and Luong atten-
tion. The different Luong attention variants are examined in both
global and local format. The contributions of a Shared kernel CNN
layer for feature extraction in the LSTM-based attention and non-
attention encoder-decoders are also evaluated. Not all attention
variants perform equally good, with Bahdanau attention outper-
forming the rest in terms of balanced sub-dataset scores.
A novel network architecture is proposed which allows for the
reading of sequences of the multidimensional time series using
a sliding window for the input as well as the output sequences.
This significantly increases the sample efficiency of the training
procedure and hence allows for estimation of RUL also in sparse
data sets with strongly limited test cases of machine degradation.
Therefore, the primary hypothesis of enhanced performance with
a sequence of RUL is achieved in this work. A comparison of the
obtained results with the actual state-of-the-art results from the
literature shows the superior performance of the proposed archi-
tectures.
In future work, we will further optimize the training procedure
for the proposed architecture especially by using deeper CNN mod-

16
S.R.B. Shah, G.S. Chadha, A. Schwung et al.
els with more feature maps and other kernel lengths. Furthermore,
we will test alternative neural network architecture, particularly
transformer networks, for their ability to tackle remaining useful
lifetime estimation problems.
Declaration of competing interest
• All authors have participated in the conception of the
manuscript, including the analysis and interpretation of the re-
sults, drafting and revising the article for important intellectual
content.
• The enclosed manuscript has not been submitted to, nor is con-
currently under review at another journal or publishing venue.
• The authors have no affiliation or involvement with any orga-
nization or entity with a direct or indirect financial interest in
the subject matter discussed in the manuscript.
• The name of the authors below have affiliations or involvement
with organizations with a direct or indirect financial interest in
the subject matter discussed in the manuscript
CRediT authorship contribution statement
Sayed Rafay Bin Shah: Software, Conceptualization, Method-
ology, Validation, Formal analysis, Resources, Writing – original
draft, Writing – review & editing, Visualization. Gavneet Singh
Chadha: Conceptualization, Methodology, Validation, Formal anal-
ysis, Resources, Writing – original draft, Writing – review &
editing, Project administration. Andreas Schwung: Conceptualiza-
tion, Writing – original draft, Supervision, Project administration.
Steven X. Ding: Supervision, Project administration.
References
Albawi, S., Mohammed, T. A., & Al-Zawi, S. (2017). Understanding of a convolutional
neural network. In I. C. o. E. &. Technology (Ed.), Proceedings of 2017 Interna-
tional Conference on Engineering & Technology (icet’2017) (pp. 1–6). Piscataway,
NJ: IEEE. https://doi.org/10.1109/ICEngTechnol.2017.8308186.
Bektas, O., Jones, J. A., Sankararaman, S., Roychoudhury, I., & Goebel, K. (2019). A
neural network filtering approach for similarity-based remaining useful life es-
timation. The International Journal of Advanced Manufacturing Technology, 101(1–
4), 87–103. https://doi.org/10.10 07/s0 0170-018-2874-0.
Bengio, S., Vinyals, O., Jaitly, N., & Shazeer, N. (2015). Scheduled sampling for se-
quence prediction with recurrent neural networks. In Proceedings of the 28th
International Conference on Neural Information Processing Systems - volume 1. In
NIPS’15 (p. 11711179). Cambridge, MA, USA: MIT Press.
Bergstra, J., & Bengio, Y. (2012). Random search for hyper-parameter optimization.
Journal of Machine Learning Research, 13(Feb), 281–305.
Botchkarev, A. (2019). A new typology design of performance metrics to measure
errors in machine learning regression algorithms. Interdisciplinary Journal of In-
formation, Knowledge, and Management, 14, 045–076. https://doi.org/10.28945/
4184.
Chatterjee, S., & Litt, J. (2003). Online model parameter estimation of jet engine
degradation for autonomous propulsion control. In Guidance, Navigation, and
Control and Co-Located Conferences. https://doi.org/10.2514/6.2003-5425.
Chaudhari, S., Polatkan, G., Ramanath, R., & Mithal, V. (2019a). An attentive survey
of attention models. ArXiv, abs/1904.02874.
Chaudhari, S., Polatkan, G., Ramanath, R., & Mithal, V. (2019b). An attentive survey
of attention models. arXiv preprint arXiv:1904.02874.
Chen, Z., Wu, M., Zhao, R., Guretno, F., Yan, R., & Li, X. (2021). Machine remain-
ing useful life prediction via an attention-Based deep learning approach. IEEE
Transactions on Industrial Electronics, 68(3), 2521–2531. https://doi.org/10.1109/
TIE.2020.2972443.
Dzmitry Bahdanau, Kyunghyun Cho, & Yoshua Bengio (2015). Neural machine trans-
lation by jointly learning to align and translate. CoRR, abs/1409.0473.
Elsheikh, A., Yacout, S., & Ouali, M.-S. (2019). Bidirectional handshaking lstm for re-
maining useful life prediction. Neurocomputing, 323, 148–156. https://doi.org/10.
1016/j.neucom.2018.09.076.
Esen, H., Inalli, M., Sengur, A., & Esen, M. (2008). Artificial neural networks and
adaptive neuro-fuzzy assessments for ground-coupled heat pump system. En-
ergy and Buildings, 40(6), 1074–1083. https://doi.org/10.1016/j.enbuild.2007.10.
002.
Frederick, D. K., DeCastro, J. A., & Litt, J. S. (2007). Users Guide for the Commercial
Modular Aero-Propulsion System Simulation (C-MAPSS).
Gavneet Singh Chadha, Ambarish Panambilly, Andreas Schwung, & Steven X. Ding
(2020). Bidirectional deep recurrent neural networks for process fault classifica-
tion. ISA Transactions, 106, 330–342. https://doi.org/10.1016/j.isatra.2020.07.011.
17
Intelligent Systems with Applications 10–11 (2021) 200049
Gavneet Singh Chadha, Utkarsh Panara, Andreas Schwung, & Steven X. Ding
(2021). Generalized dilation convolutional neural networks for remaining use-
ful lifetime estimation. Neurocomputing, 452, 182–199. https://doi.org/10.1016/j.
neucom.2021.04.109.
Glorot, X., & Bengio, Y. (2010). Understanding the difficulty of training deep feed-
forward neural networks. In Proceedings of the Thirteenth International Confer-
ence on Artificial Intelligence and Statistics. In Proceedings of Machine Learning
Research: 9 (pp. 249–256). PMLR.
Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. Adaptive computation
and machine learning. Cambridge, Massachusetts: The MIT Press.
Goyal, A., Lamb, A., Zhang, Y., Zhang, S., Courville, A., & Bengio, Y. (2016). Professor
forcing: A new algorithm for training recurrent networks. In Proceedings of the
30th International Conference on Neural Information Processing Systems. In NIPS’16
(p. 46084616). Red Hook, NY, USA: Curran Associates Inc..
Graves, A., & Schmidhuber, J. (2005). Framewise phoneme classification with bidi-
rectional LSTM and other neural network architectures. Neural Networks : The
Official Journal of the International Neural Network Society, 18(5–6), 602–610.
https://doi.org/10.1016/j.neunet.2005.06.042.
Gu, J., Wang, Z., Kuen, J., Ma, L., Shahroudy, A., Shuai, B., Liu, T., Wang, X., Wang, G.,
Cai, J., & Chen, T. (2018). Recent advances in convolutional neural networks. Pat-
tern Recognition, 77, 354–377. https://doi.org/10.1016/j.patcog.2017.10.013.
Heimes, F. O. (2008). Recurrent neural networks for remaining useful life estimation.
In 2008 International Conference on Prognostics and Health Management (pp. 1–
6). https://doi.org/10.1109/PHM.2008.4711422.
Hikmet Esen, Filiz Ozgen, Mehmet Esen, & Abdulkadir Sengur (2009). Artificial neu-
ral network and wavelet neural network approaches for modelling of a solar air
heater. Expert Systems with Applications, 36(8), 11240–11248. https://doi.org/10.
1016/j.eswa.2009.02.073.
Hikmet Esen, Mehmet Esen, & Onur Ozsolak (2017). Modelling and experimental
performance analysis of solar-assisted ground source heat pump system. Journal
of Experimental & Theoretical Artificial Intelligence, 29(1), 1–17. https://doi.org/10.
1080/0952813X.2015.1056242.
Hikmet Esen, Mustafa Inalli, Abdulkadir Sengur, & Mehmet Esen (2008a). Forecast-
ing of a ground-coupled heat pump performance using neural networks with
statistical data weighting pre-processing. International Journal of Thermal Sci-
ences, 47(4), 431–441. https://doi.org/10.1016/j.ijthermalsci.20 07.03.0 04.
Hikmet Esen, Mustafa Inalli, Abdulkadir Sengur, & Mehmet Esen (2008b). Perfor-
mance prediction of a ground-coupled heat pump system using artificial neural
networks. Expert Systems with Applications, 35(4), 1940–1948. https://doi.org/10.
1016/j.eswa.2007.08.081.
Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computa-
tion, 9(8), 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735.
Isermann, R. (2005). Model-based fault-detection and diagnosis – status and ap-
plications. Annual Reviews in Control, 29(1), 71–85. https://doi.org/10.1016/j.
arcontrol.20 04.12.0 02.
Jardine, A. K., Lin, D., & Banjevic, D. (2006). A review on machinery diagnos-
tics and prognostics implementing condition-based maintenance. Mechanical
Systems and Signal Processing, 20(7), 1483–1510. https://doi.org/10.1016/j.ymssp.
2005.09.012.
Junyoung Chung, Çaglar Gülçehre, KyungHyun Cho, & Yoshua Bengio (2014). Empir-
ical evaluation of gated recurrent neural networks on sequence modeling. CoRR,
abs/1412.3555.
Khan, S., & Yairi, T. (2018). A review on the application of deep learning in sys-
tem health management. Mechanical Systems and Signal Processing, 107, 241–
265. https://doi.org/10.1016/j.ymssp.2017.11.024.
Kingma, D. P., & Ba, J. (2015). Adam: A Method for stochastic optimization. CoRR,
abs/1412.6980.
Kiranyaz, S., Avci, O., Abdeljaber, O., Ince, T., Gabbouj, M., & Inman, D. J. (2021).
1D convolutional neural networks and applications: A survey. Mechanical Sys-
tems and Signal Processing, 151, 107398. https://doi.org/10.1016/j.ymssp.2020.
107398.
Kong, Z., Cui, Y., Xia, Z., & Lv, H. (2019). Convolution and long short-Term mem-
ory hybrid deep neural networks for remaining useful life prognostics. Applied
Sciences, 9(19), 4156. https://doi.org/10.3390/app9194156.
Lei, Y., Li, N., Guo, L., Li, N., Yan, T., & Lin, J. (2018). Machinery health prognostics:
A systematic review from data acquisition to rul prediction. Mechanical Systems
and Signal Processing, 104, 799–834. https://doi.org/10.1016/j.ymssp.2017.11.016.
Li, J., Li, X., & He, D. (2019a). A directed acyclic graph network combined with
CNN and LSTM for remaining useful life prediction. IEEE Access, 7, 75464–75475.
https://doi.org/10.1109/ACCESS.2019.2919566.
Li, X., Ding, Q., & Sun, J.-Q. (2018). Remaining useful life estimation in prognostics
using deep convolution neural networks. Reliability Engineering & System Safety,
172, 1–11. https://doi.org/10.1016/j.ress.2017.11.021.
Li, X., Zhang, W., & Ding, Q. (2019b). Deep learning-based remaining useful life es-
timation of bearings using multi-scale feature extraction. Reliability Engineering
& System Safety, 182, 208–218. https://doi.org/10.1016/j.ress.2018.11.011.
Lipton, Z. C., Berkowitz, J., & Elkan, C. (2015). A Critical Review of Recurrent Neural
Networks for Sequence Learning. https://arxiv.org/pdf/1506.0 0 019.
Listou Ellefsen, A., Bjørlykhaug, E., Æsøy, V., Ushakov, S., & Zhang, H. (2019). Re-
maining useful life predictions for turbofan engine degradation using semi-
Supervised deep architecture. Reliability Engineering & System Safety, 183, 240–
251. https://doi.org/10.1016/j.ress.2018.11.027.
Liu, H., Liu, Z., Jia, W., & Lin, X. (2019). A novel deep learning-based encoder-decoder
model for remaining useful life prediction. In 2019 International Joint Conference
on Neural Networks (ijcnn) (pp. 1–8). IEEE. https://doi.org/10.1109/IJCNN.2019.
8852129.
S.R.B. Shah, G.S. Chadha, A. Schwung et al.
Liu, H., Liu, Z., Jia, W., & Lin, X. (2021). Remaining useful life prediction using a
novel feature-attention-based end-to-end approach. IEEE Transactions on Indus-
trial Informatics, 17(2), 1197–1207. https://doi.org/10.1109/TII.2020.2983760.
Luong, T., Pham, H., & Manning, C. D. (2015). Effective Approaches to Attention-
based Neural Machine Translation. In Proceedings of the 2015 Conference on Em-
pirical Methods in Natural Language Processing (pp. 1412–1421). Lisbon, Portugal:
Association for Computational Linguistics. https://doi.org/10.18653/v1/D15-1166.
Martin, K. F. (1994). A review by discussion of condition monitoring and fault diag-
nosis in machine tools. International Journal of Machine Tools and Manufacture,
34(4), 527–551. https://doi.org/10.1016/0890- 6955(94)90083- 3.
Mo, Y., Wu, Q., Li, X., & Huang, B. (2021). Remaining useful life estimation via trans-
former encoder enhanced by a gated convolutional unit. Journal of Intelligent
Manufacturing. https://doi.org/10.1007/s10845- 021- 01750- x.
Nandi, S., Toliyat, H. A., & Li, X. (2005). Condition monitoring and fault diagnosis of
electrical motors—A review. IEEE Transactions on Energy Conversion, 20(4), 719–
729. https://doi.org/10.1109/TEC.2005.847955.
Paulo Roberto De Oliveira Da Costa, Alp Akcay, Yingqian Zhang, & Uzay Kay-
mak (2020). Attention and long short-Term memory network for remaining use-
ful lifetime predictions of turbofan engine degradation. International Journal of
Prognostics and Health Management, 10, 12.
Peng, C., Chen, Y., Chen, Q., Tang, Z., Li, L., & Gui, W. (2021). A remaining useful life
prognosis of turbofan engine using temporal and spatial feature fusion. Sensors
(Basel, Switzerland), 21(2). https://doi.org/10.3390/s21020418.
Ragab, M., Chen, Z., Wu, M., Kwoh, C.-K., Yan, R., & Li, X. (2020). Attention
Sequence to Sequence Model for Machine Remaining Useful Life Prediction.
https://arxiv.org/pdf/2007.09868.
Sateesh Babu, G., Zhao, P., & Li, X.-L. (2016). Deep convolutional neural network
based regression approach for estimation of remaining useful life. In Database
Systems for Advanced Applications (pp. 214–228). Springer International Publish-
ing. https://doi.org/10.1007/978- 3- 319- 32025- 0_14.
Saxena, A., Goebel, K., Simon, D., & Eklund, N. (06/10/2008 - 09/10/2008). Dam-
age propagation modeling for aircraft engine run-to-failure simulation. In 2008
International Conference on Prognostics and Health Management (pp. 1–9). IEEE.
https://doi.org/10.1109/PHM.2008.4711414.
Sutskever, I., Vinyals, O., & Le V, Q. Sequence to sequence learning with neural net-
works. http://arxiv.org/pdf/1409.3215v3.
Swati Meshram (2019). Survey on attention neural network models for natural lan-
guage processing: 8. https://doi.org/10.15680/IJIRSET.2019.0810037.
Tan, W. M., & Teo, T. H. (2021). Remaining useful life prediction using temporal con-
volution with attention. Artificial Intelligence (AI), 2(1), 48–70. https://doi.org/10.
3390/ai2010 0 05.
Tsui, K. L., Chen, N., Zhou, Q., Hai, Y., & Wang, W. (2015). Prognostics and health
management: A review on data driven approaches. Mathematical Problems in
Engineering, 2015(6), 1–17. https://doi.org/10.1155/2015/793161.
18
Intelligent Systems with Applications 10–11 (2021) 200049
Wang, J., Wen, G., Yang, S., & Liu, Y. (2018). Remaining useful life estimation in
prognostics using deep bidirectional LSTM neural network. In 2018 Prognos-
tics and System Health Management Conference (phm-chongqing) (pp. 1037–1042).
https://doi.org/10.1109/PHM-Chongqing.2018.00184.
Williams, R. J., & Zipser, D. (1989). A learning algorithm for continually running fully
recurrent neural networks. Neural Computation, 1(2), 270–280. https://doi.org/
10.1162/neco.1989.1.2.270.
Wu, J. (2017). Introduction to convolutional neural networks. National Key Lab for
Novel Software Technology. Nanjing University. China, 5, 23.
Wu, Y., Yuan, M., Dong, S., Lin, L., & Liu, Y. (2018). Remaining useful life estimation
of engineered systems using vanilla lstm neural networks. Neurocomputing, 275,
167–179. https://doi.org/10.1016/j.neucom.2017.05.063.
Yann LeCun, Bernhard E. Boser, John S. Denker, Donnie Henderson, Richard E.
Howard, Wayne E. Hubbard, & Lawrence D. Jackel (1990). Handwritten digit
recognition with a back-propagation network denver, colorado, usa, november
27–30, 1989]. In David S. Touretzky (Ed.), Advances in Neural Information Pro-
cessing Systems 2 (pp. 396–404). Morgan Kaufmann.
Ye, L., & Keogh, E. (2009). Time Series Shapelets: A New Primitive for Data Mining.
In KDD ’09 (p. 947956). New York, NY, USA: Association for Computing Machin-
ery. https://doi.org/10.1145/1557019.1557122.
Yu, Z., Ramanarayanan, V., Suendermann-Oeft, D., Wang, X., Zechner, K., Chen, L.,
Tao, J., Ivanou, A., & Qian, Y. (2015). Using bidirectional LSTM recurrent neural
networks to learn high-level abstractions of sequential features for automated
scoring of non-native spontaneous speech. In 2015 IEEE Workshop on Automatic
Speech Recognition and Understanding (asru) (pp. 338–345). IEEE. https://doi.org/
10.1109/ASRU.2015.7404814.
Zhang, H., Zhang, Q., Shao, S., Niu, T., & Yang, X. (2017). Attention-Based LSTM net-
work for rotatory machine remaining useful life prediction. IEEE Access, 1–9.
https://doi.org/10.1109/ACCESS.2020.3010066.
Zhao, R., Yan, R., Chen, Z., Mao, K., Wang, P., & Gao, R. X. (2019a). Deep learning and
its applications to machine health monitoring. Mechanical Systems and Signal
Processing, 115, 213–237. https://doi.org/10.1016/j.ymssp.2018.05.050.
Zhao, S., Zhang, Y., Wang, S., Zhou, B., & Cheng, C. (2019b). A recurrent neural net-
work approach for remaining useful life prediction utilizing a novel trend fea-
tures construction method. Measurement, 146, 279–288. https://doi.org/10.1016/
j.measurement.2019.06.004.
Zheng, S., Ristovski, K., Farahat, A., & Gupta, C. (2017). Long short-term memory net-
work for remaining useful life estimation. In 2017 IEEE International Conference
on Prognostics and Health Management (icphm) (pp. 88–95). https://doi.org/10.
1109/ICPHM.2017.7998311.