Progress in Nuclear Energy 152 (2022) 104401

Contents lists available at ScienceDirect

Progress in Nuclear Energy

journal homepage: www.elsevier.com/locate/pnucene

Development of deep reinforcement learning-based fault diagnosis method

for rotating machinery in nuclear power plants
Gensheng Qian *, Jingquan Liu
Department of Engineering Physics, Tsinghua University, Beijing, 100084, China

A R T I C L E I N F O A B S T R A C T

Keywords: Rotating machinery fault can cause accidents like loss of flow or turbine trip that seriously threaten the operation
Deep reinforcement learning safety of nuclear power plants (NPPs). Artificial intelligence algorithms, like machine learning or deep learning
Small sample methods, can implement fault diagnosis by sample learning with no reliance on fault mechanism or physics
Rotating machinery
model of the equipment. However, the accumulated fault samples are small due to high operation safety re­
Fault diagnosis
Nuclear power plant
quirements of the plant. Small sample learning is challenging and usually leads to degradation of model per­
formance. The emerging deep reinforcement learning (DRL) algorithm can incorporate the advantages of
automatic feature extraction from deep learning algorithm and interactive learning from reinforcement learning
algorithm, is expected to have better learning ability and robustness. In this paper, two DRL fault diagnosis
models are proposed and compared. Experiment results show that the proposed models can achieve very high
diagnosis accuracy of over 99% and outperform all the baseline models (support vector machine, convolutional
neural network and gated recurrent unit neural network) in all test cases in this paper.

1. Introduction
Rotating machinery, such as the primary coolant pump, condensate
pump, feedwater pump and steam turbine, are the key equipment to
ensure the stable and safe operation of the entire thermal-hydraulic
circulation system in nuclear power plants (NPPs). Due to the long-
term operation under harsh environment of high temperature, high
pressure, high load and varying working conditions, the components of
rotating machinery inevitably incur damage such as corrosion, wear and
cracks, and the risk of fault increases gradually. Some typical rotating
machinery faults can cause reactor system accidents, e.g., loss of flow
accident (LOFA), loss of feed water accident (LOFW) and turbine trip,
which would result in significant economic loss, and even evolve into
nuclear disasters if not handled properly. Effective condition monitoring
and fault diagnosis techniques can help improve the safety, reliability
and economy of NPPs (Ma and Jiang 2011).
Vibration measurement is an effective method for condition moni­
toring in NPPs (Ayo-Imoru and Cilliers 2018; Koo and Kim 2000; Lebold
et al., 2004). Traditional vibration analysis methods include analysis of
the frequency spectrum, orbit plot, bode plot, 3D waterfall spectra plot
and amplitude-phase plot (Sinha 2008), which are labor-intensive and
rely heavily on the analysts’ experience. With the rapid development of
data mining and artificial intelligence techniques (Jordan and Mitchell
2015), data-driven machine learning algorithms can provide a feasible
solution for intelligent fault diagnosis.
Generally, machine learning algorithms focus on how to learn and
obtain useful insights to make predictions and decisions from datasets
(Jordan and Mitchell 2015). Machine learning-based fault diagnosis
methods do not rely on the fault mechanism or physics model that are
hard to build or not clear for complicated industrial system or equip­
ment. For building a machine leaning fault diagnosis model, signal
processing or statistical methods are required to extract data features at
first, then, appropriate classifier model, e.g., random forest, support
vector machine (SVM), is built to fit the mapping relationship between
data features and their fault labels. A well-trained model can predict the
condition type of the monitoring equipment based on new collected
sensor data automatically, thus reduces the workload of analysts. Many
studies use advanced signal processing and machine learning methods
for rotating machinery fault diagnosis in NPPs, e.g., artificial neural
network (ANN) method for reactor coolant pump fault diagnosis (Koo
and Kim 2000), K-Nearest neighbors (KNN) for turbo-generator rotor
fault diagnosis (Biet 2013), Hilbert-Huang Transform (HHT) for
flywheel fault feature extraction (Liu et al., 2015), Teager energy
operator (TEO) for bearing fault frequencies identification (Zhu et al.,

* Corresponding author.
E-mail address: qgs19@mails.tsinghua.edu.cn (G. Qian).

https://doi.org/10.1016/j.pnucene.2022.104401
Received 8 June 2022; Received in revised form 23 August 2022; Accepted 24 August 2022
Available online 5 September 2022
0149-1970/© 2022 Elsevier Ltd. All rights reserved.
G. Qian and J. Liu
2021), SVM (Feng et al., 2018), random forest and Adaboost (Zhong and
Ban 2022a) for bearing and gear fault diagnosis.
In the above mentioned studies, signal analysis methods, like Wigner
Distribution (WD), Variational Mode Decomposition (VMD), Empirical
Mode Decomposition (EMD) and wavelet packet transform (WPT), are
used for feature extraction at first. In the diagnosis framework of classic
machine learning method, feature extraction and fault diagnosis pro­
cedures are executed separately. Generally, feature extraction inevitably
loses some information in the original data that may be helpful for fault
diagnosis. Therefore, the classic machine learning model performance is
heavily limited by the manual feature extraction experience. The latest
research topic, deep learning algorithms, such as convolutional neural
network (CNN), can directly process raw data and automatically extract
abstract features with a deep network structure (LeCun et al. 2015).
Deep learning can integrate feature extraction and fault diagnosis in one
end-to-end framework, reducing the limit and reliance on manual
feature extraction. For example, (Duan et al., 2020) proposed a fault
diagnosis method for air compressors in NPPs based on vibration
observation window (VOW) and CNN model. (Zou et al., 2021) pre­
sented a one-dimensional CNN (1D-CNN) fault diagnosis method for
bearing. (Chen et al., 2020) developed two 1D-CNN models for rotor and
bearing fault diagnosis, which can reliably identify simulated 48 ma­
chine health conditions and even previously unlearned faults. (Zhang
et al., 2021) proposed a Gated Recurrent Unit neural network (GRU)-­
based rotating machinery fault diagnosis model with residual connec­
tion and learning rate decay strategies.
The machine learning and deep learning-based fault diagnosis
methods mentioned above are all based on labeled sample learning (i.e.,
supervised learning). Usually, the upper limit of the diagnostic ability in
the fitted function (mapping relationship) is determined by the sample
size and quality. When the sample size reduces or the samples are
disturbed by noise, it will lead to underfitting or overfitting problem and
model performance degradation. In industrial practice, especially in the
nuclear industry, which requires high safety and reliability of equip­
ment, the plant operate in a normal condition most of the time, and the
accumulated fault samples are small. Small sample learning is chal­
lenging. Development of fault diagnosis model under small sample is a
hot research topic at present (Pan et al., 2021).
Data augmentation (Li et al., 2022; Qian and Liu 2022) and transfer
learning (Zhong, Fu, and Lin 2019; Zhong and Ban 2022b) are two viable
solutions that deal small sample learning. Data augmentation methods
expand the sample size by over-sampling or generating models to yield
synthetic data similar with existing samples. The transfer learning
approach first pre-trains the model using datasets of similar equipment
or other field’s dataset (e.g., ImageNet, a general image dataset) to make
the model have certain diagnostic capability, and then fine-tunes it on
the target small sample dataset to implement the target diagnosis task.
Overall, data augmentation and transfer learning methods can improve
the diagnostic capability by optimizing the fitted mapping function via
synthetic data or data from other equipment or even other fields.
Reinforcement learning (RL) is another branch of artificial intelli­
gence research topic. RL model learns by interacting with the environ­
ment, more specifically, RL model continuously adapts its behavior in
order to maximize the feedback reward signal from the environment
during the training (Sutton and Barto 2018). Compared to the super­
vised machine learning algorithms, RL learns an optimal policy (i.e., a
response mechanism that yields the most reward) rather than fit the
mapping function between samples and labels. Deep reinforcement
learning (DRL), which combines the advantages of automatic feature
extraction capability from deep learning and interactive learning capa­
bility from RL, is a revolutionary progress in artificial intelligence field
and show promise to solve complex real-world problems (Arulkumaran
et al., 2017). For instance, DRL model can play video games and get
scores at human player level (Mnih et al., 2013). The DRL-based
AlphaGo computer program defeated a world champion in the game
of Go (Silver et al., 2017). In nuclear industry, DRL-based framework
2
Progress in Nuclear Energy 152 (2022) 104401
was developed for safety function status check task (Park et al. 2020),
control automation in the heat-up mode of a NPP (Park et al., 2022), and
large-scale design optimization of boiling water reactor bundles
(Radaideh et al. 2021), etc.
Recently, a growing number of studies applied DRL to fault diagnosis
problems. For example, (Ding et al., 2019) proposed a DRL method
based on sparse auto-encoder (SAE) for bearing and pump fault diag­
nosis, which has comparable performance with the deep learning model
SAE-softmax. (Wang et al., 2022) developed a planetary gearbox fault
diagnosis method based on time-frequency representation (TFR) and a
CNN-based DRL model, which shows good performance under
multi-work conditions. (Li et al., 2021) developed a DRL model based on
capsule neural network (Cap-net) and an online feature dictionary
method, which can adapt to fault diagnosis tasks of variable working
conditions. (Zisheng Wang and Xuan 2021) proposed a 1D-CNN based
DRL method to implement the compound fault diagnosis under heavy
background noise of bearing and tool. (Fan et al. 2022) presented a
DRL-based fault diagnosis method called “DiagSelect” for sample
imbalance scenario, which selects suitable samples from the initial
training set to reduce the imbalance that helps improve the model
performance.
As shown from the above studies, the DRL model can achieve
excellent fault diagnosis performance in scenarios such as strong noise,
varying working conditions and sample imbalance. However, fault
diagnosis study of DRL model under small sample scenario is relatively
lacking. Many studies in existing publications focus on developing CNN-
based DRL fault diagnosis model. Since CNN and GRU are 2 popular
deep learning models, it is an important work to evaluate the perfor­
mance difference between CNN-based and GRU-based DRL fault diag­
nosis models. In addition, according to our literature survey, there are
no relevant publications (by July 2022) discussing the application of
DRL-based fault diagnosis method for NPPs.
To fill the research gap, in this paper, we propose two DRL fault
diagnosis models. The model training process, diagnostic effectiveness,
performance comparison under small sample scenario are carefully
analyzed. The main contribution of this paper are summarized as
follows:
1) This paper is likely to be the first DRL-based method study for NPP
fault diagnosis.
2) Two DRL models are presented for fault diagnosis of rotating ma­
chinery based on CNN and GRU model, respectively. And compari­
son experiments are designed for evaluating their performance with
baseline models.
3) The fault diagnosis effectiveness is carefully studied and elaborated
by analysis of the training accuracy curve, cumulative reward,
single-sample identification process and hidden layer visualization.
4) The proposed DRL models can achieve excellent fault diagnosis
performance with over 99% accuracy, and are more robust in small
sample scenarios than baseline models, making them more suitable
for fault diagnosis applications in NPP.
The rest of this paper are organized as follows: Section 2 introduces
the background knowledge of CNN, GRU, DRL and the proposed fault
diagnosis method in this paper. Section 3 describes two fault experi­
mental datasets of rotating machinery and the data processing, model
parameter setting and evaluating method in detail. Section 4 analyzes
and discusses the model training process and evaluation results. Section
5 concludes the research of this paper.
2. Methodology
2.1. CNN
CNN is designed to process data in the form of multiple arrays and
has been successfully applied in image, video, speech and audio
G. Qian and J. Liu
processing (LeCun et al. 2015). CNN has four key innovations, namely
local connections, shared weights, spatial pooling and deep layer
structure, and its key technical elements include convolution, pooling,
nonlinear activation, fully-connected (FC) layer processing and softmax
regression, as shown in Fig. 1. The original vibration sensor data is 1D
time-series data, so this paper uses a 1D-CNN model, whose main
functional features are similar to 2D-CNN (usually for 2D image format
data processing).
The convolution layer contains multiple convolution kernels (also
called filters) that scan and convolve the input data to obtain feature
maps. The pooling layer performs a subsampling operation to reduce the
spatial resolution of the feature map, allowing the network to extract
location-invariant features. Maximum pooling method is used in this
paper, as shown in Fig. 1(b), which takes the maximum value in each
pooling window as the output. After several convolutional and pooling
layers, the abstract features of the input data are automatically extrac­
ted. Then, the feature maps are flattened into a vector, and the subse­
quent processing is performed using FC layers. Appropriate activation
function is used after the convolutional layer and FC layer to improve
the nonlinear fitting ability of the network. Commonly used activation
functions are sigmoid function, hyperbolic tangent function (tanh), and
rectified linear unit (ReLU) function, etc. (Rasamoelina et al. 2020),
which are shown in Fig. 1(c ~ e).
For solving fault diagnosis problems, the output layer uses the soft­
max regression (see Fig. 1(g)) to convert the output results into a
probability distribution, and the final diagnosis result is determined by
the neuron (or called node) with the maximum probability. The cross-
entropy loss function (see Fig. 1(h)) can quantify the difference be­
tween the predicted distribution and the ground true label distribution,
and is used as the optimization objective of the network model. All
network models in this paper are optimized by the popular Adam al­
gorithm (Kingma et al., 2015).
By combining these basic elements, a variety of CNN models can be
designed. An effective CNN-based fault diagnosis model design is given
in this paper, as shown in Fig. 2, and the model performance analysis are
introduced in Section 4.
2.2. GRU
GRU network is a modified and improved variant of early vanilla
recurrent neural network (RNN) that excels in processing and extracting
Fig. 1. Key conceptions in CNN model.
3
Progress in Nuclear Energy 152 (2022) 104401
features from sequential data (Chung et al., 2014). It has been successful
applied in machine translation, speech recognition, etc. (Ravanelli et al.,
2018). The key technical elements of the GRU model include the
recurrent information transfer and two gating mechanisms (update gate
and reset gate), as shown in Fig. 3. The GRU model considers the input
data as a sequence stream (or flow), and the network output depends on
the current input data and last hidden state (the learned features from
the whole history inputs), as shown in Fig. 3(a). The reset and update
gates are responsible for manipulating the information flow in the
recurrent units (cells), ensuring that important features and long-time
dependencies can be accumulated in the hidden state, and the unim­
portant or unnecessary information can be reset and removed in time.
An effective GRU-based fault diagnosis model design is given in this
paper, as shown in Fig. 4. The hidden state of all time steps of the GRU
layer is taken and spliced into a feature vector, which contains the
feature information of the overall input sequence. FC layers are added at
the back-end of the network for subsequent processing and diagnosis
output. The model performance analysis are introduced in Section 4.
2.3. DRL
DRL is a kind of composite framework of deep learning and rein­
forcement learning (RL) that uses deep neural networks (DNNs) as the
intelligent agent to perceive the environment and make appropriate
decisions, with feedback rewards from the environment used to train the
agent in a RL way. AlphaGo is the most famous DRL model that defeated
the human world champion in the Go game (Silver et al., 2017). Since
playing Go game is a much more difficult task than image classification,
DRL can master it, which shows a revolutionary progress in artificial
intelligence. Therefore, DRL is considered to have stronger intelligence
than deep learning and has the potential to solve complex real-world
problems, e.g., electric power system control (Glavic 2019) and cyber
security (Nguyen and Reddi 2021). This section provides a brief intro­
duction to the basic principle of DRL.
As shown in Fig. 5, the RL framework consists of a learning agent, the
environment and their interactions in terms of states, actions and re­
wards. Agent is an intelligent body, which can make decisions and take
actions based on a certain policy. Environment is the interaction object
of the agent. State s represents one state of the environment, which can
be observed and sensed by the agent. Action a represents the agent’s
behavior, which has some influence in the environment. Reward r
G. Qian and J. Liu
Fig. 2. Schematic diagram of a 1D-CNN based fault diagnosis model.
Fig. 3. Key conceptions in GRU model.
represents the feedback signal from the environment, which has some
meaning for the agent, like reward or penalty. The agent aims to learn an
optimal policy π(a|s) from the interaction experience. The optimal pol­
icy means how to choose the most appropriate action a based on the
current observing state s to obtain the optimal cumulative reward r from
the environment.
Generally, the function Q(s, a) is used to represent the expectation of
reward by choosing the action a at state s with policy π.
Q(s, a) = E[rt |st = s, at = a; π] (1)
and its value is updated iteratively using the following equation (Sutton
and Barto 2018):
( )
(2)
′ ′ ′
Q (s, a) = Q(s, a) + α r(s, a) + γmaxQ(s , a ) − Q(s, a)
′
a using a copy of DQN to predict the target Q value, which delays the
where α is the learning rate. γ is the discount factor, which takes value
from 0 to 1, and indicates the delay discount reward for the current
action. When taking 0, only the reward at the current step is considered.
DNN can be used as a function approximator to estimate the Q
function as Q(s, a) ≈ Q(s, a; θ),where θ is the weight parameters of the
DNN. Such a network is also called DQN and was first proposed in 2013
by Mnih et al., (2013). The DQN can take full advantage of the DNN’s
4
Progress in Nuclear Energy 152 (2022) 104401
powerful ability to sense the environment state from raw sensor data.
For example, the DQN can make appropriate actions based on the
screenshot images of the Atari video games and achieve a level com­
parable to professional human game testers on 49 different games (Mnih
et al., 2015).
Nowadays, 3 important techniques are used in the training of DQN:
ε-greedy exploration mechanism, experience replay (ER) method and
network separation (Mnih et al., 2015). The ε-greedy mechanism sets a
small probability threshold ε, which is a trade-off between exploration
and exploitation and allows the DQN to make random actions or make a
decision greedily. The ER method, which randomly takes small batches
of experience from the experience pool to train and update the model
parameters, makes full use of the experience pool and reduces the cor­
relation of 2 consecutive input experience. Network separation means
update of Q value and improve the model stability. The detailed training
procedure is described in Appendix Algorithm A1.
2.4. The proposed method
Fault diagnosis is a classification decision problem and can be
simulated as a guessing game (Ding et al., 2019; Wang et al., 2022). The
agent is the game player, which aims to guess the fault type correctly.
G. Qian and J. Liu Progress in Nuclear Energy 152 (2022) 104401

Fig. 4. Schematic diagram of a GRU-based fault diagnosis model.

The environment consists of a large number of samples of sensor data,

Fig. 5. Basic conceptions of reinforcement learning.
each state corresponding to one sample. For a K-class fault diagnosis
problem, the guessing action space is defined as {0, 1, …, K-1}, where
0 represents normal and other numbers, such as k, represents the k-th
type of fault. The agent will get a reward signal when it guesses the right
answer, otherwise, a penalty signal. We expect that after many rounds of
guessing game, the agent can learn an optimal policy to guess the correct
condition type using the sensor data from the monitoring equipment.
The proposed fault diagnosis method is shown in Fig. 6. The envi­
ronment emulator is constructed by dataset D = {xi , yi }Ni=1 (x is the
sensor data, y is the fault type, and N is the number of samples). The
agent (i.e., DQN) is constructed using a CNN or GRU network, which
extracts and senses the sample features and tries to make a correct guess.
The agent is further divided into Policy-Net and Target-Net, which have
the same network structure and synchronize the parameters periodi­
cally. To simplify the description, the Policy-Net and Target-Net are
denoted by fθ and fθ′ , where θ and θ are the sets of weight parameters of
′

the corresponding networks. The Policy-Net outputs the best estimated

Fig. 6. The proposed DRL-based fault diagnosis model, where Policy-Net and Target-Net are implemented by 1D-CNN or GRU model in this paper.

5
G. Qian and J. Liu
action a and Q value based on the observing state s (see Eq. (3)). The
Target-Net outputs the estimated Q’ value corresponding to the next
state s (see Eq. (4)).
′
(a, Q) = fθ (s)
′ ′
Q = fθ′ (s )
The details of one interaction process are described as follows. For a
state s (i.e., a vibration data sample), s is first processed using Fast
Fourier Transform (FFT) (Cooley and Tukey 1965) to obtain the fre­
quency spectrum, and then the action is taken according to the ε-greedy
mechanism. With probability ε, randomly taking an action a and with
probability 1-ε , the spectrum is input to Policy-Net to obtain the pre­
dicted best estimated action a. The environment gives a reward signal r
according to the action a and switches to a new state s randomly. We
′ Zhang, and Tang 2018)
define the reward signal as 1 when the guess is correct, otherwise, the
reward is − 1. At the end of each game, the experience tuple (s, a, r, s ) is
′
stored in the experience pool. ER method is used to update the param­
eters of Policy-Net. Specifically, the Mean Square Error (MSE) loss
function L(θ) is calculated based on an experience tuple (st , at , rt , st+1 )
sampled from the experience pool:
L(θ) = [yt − fθ (st , at )]2
′
yt = rt + γ maxa′ fθ′ (st+1 , a )
where fθ (st , at ) is the Q value predicted by Policy-Net, rt is the reward
value recorded in the experience, and fθ′ (st+1 , a ) is the Q value at step
′
t+1 predicted by the Target-Net.
In the training process, T guesses are taken as one episode (epoch),
and the cumulative reward value is recorded. Ideally, after adequate
episodes of training, the cumulative reward value in each episode should
gradually increase until it converges to T (i.e., the game rounds). The
cumulative reward can be monitored to determine whether the model
has been trained enough or not.
For ease of description, the DRL models proposed in this paper are
named as CNN-RL and GRU-RL based on the internal network type of the
Agent, respectively. The performance comparison and validation are
described in detail in Section 4.
3. Case study
3.1. Case description
Bearings and gearboxes are common and important rotating ma­
chinery components used in NPPs for motors, pumps, fans and turbines
(Ma and Jiang 2011; Sinha 2008; Smith et al. 2007). However, for
commercial or other considerations, there is no public rotating ma­
chinery fault dataset from NPPs now. As an alternative, related works
(Feng et al., 2018; Miki and Demachi 2020; Zhichao Wang et al., 2022;
Zhong and Ban 2022a, 2022b) used public datasets for their proposed
rotating machinery fault diagnosis method validation of NPPs. Usually,
public datasets are good benchmarks for method validation, and it also
facilitates the reproduction and propagation of the proposed methods.
On the other hand, validating the model performance on laboratory
dataset is necessary. Only when the model can perform well on labo­
ratory datasets does it have the possibility and potential to show good
performance on real industrial datasets. Therefore, in this paper, 2
public fault experiment datasets are selected for method validation.
They are from 2 types of typical rotating machinery components, and
both contain sensor data under normal and multiple abnormal operating
conditions, which are suitable for constructing multi-classification ex­
periments for fault diagnosis. The validation process and results in this
paper can provide a valuable reference for further engineering appli­
cations in NPPs. The selected fault datasets are briefly described as
Progress in Nuclear Energy 152 (2022) 104401
follows:
1) Case Western Reserve University (CWRU) bearing fault dataset
(Bearing Data Center | Case School of Engineering | Case Western
(3) Reserve University n.d.)
(4)
The experiment setup is shown in Fig. 7(a). The experiments
measured the vibration acceleration signals of motor bearings under
normal and fault states. The sampling frequencies include 12 kHz and
48 kHz. Single point fault was introduced in the inner race, outer race
and rolling body of the bearing in different experiments. Fault diameters
include 0.18, 0.36, and 0.54 mm, etc.
2) University of Connecticut (UOC) gearbox fault dataset (P. Cao,
The experiment setup is shown in Fig. 7(b). The experiments
measured the vibration signals of two-stage gearbox in normal and fault
states. The sampling frequency was 20 kHz and the operating conditions
included health, missing tooth, root crack, spalling, and chipping tip of 5
severity levels.
(5) 3.2. Data processing
(6) In this paper, we take the drive-end sensor data in the CWRU dataset
with the rotation speed of 1797 rpm and sampling frequency of 12 kHz
to construct dataset A. 1024 data points are taken as one sample. In the
UOC dataset, 1800 data points are taken as one sample to construct
dataset B. Detailed information of the dataset are shown in Table 1. 10
and 9 samples (one sample from different condition types) are selected
from datasets A and B, respectively, and their original vibration signals
and frequency spectrum processed by FFT are plotted. As shown in Fig. 8
(a)(c), the vibration time domain waveforms vary drastically and appear
to be cluttered. The frequency spectrum shown in Fig. 8(b)(d) indicate
the vibration characteristics more clearly, which help the subsequent
information processing and classification. So frequency spectrum is used
as input to the network model.
The raw frequency spectrum is a one-dimensional vector and needs
to be resized its shape accordingly when used as network model input.
For CNN and CNN-RL models, the spectrum will be reshaped to a three-
dimensional tensor shaped as [B, L, C], where B represents the batch size
of training samples, L represents the sample length, and C represents the
number of input channel (equals to 1 in this paper). Similarly, for the
GRU and GRU-RL models, the spectrum will be reshaped to a three-
dimensional tensor shaped as [B, T, S], where T is the number of time
steps and S is the sequence length of data in each time step. B, L, C, T,
and S are set empirically and by trial-and-error. The specific values used
in this paper are shown in Section 3.3 and Table 2.
3.3. Model setting
We use TensorFlow (v2.3.0) (Abadi et al., 2016) for network model
development. After much trial-and-error, the parameters shown in
Table 2 were finally selected. Due to the symmetry of the FFT processing
results, the input size is half of the original vibration signal sample
length, which are 512 for dataset A and 900 for dataset B, respectively.
The other parameters are set as follows: the experience pool size is 300,
the replay size is 32, the game rounds in per episode (epoch) is 32, the
update periodicity of the target network is 10 training steps, the initial
value of ε is 0.5, εtotal is 50, and εmin is 0.01. Adam optimizer (Kingma
et al., 2015) is selected for the network parameters updating. The
learning rate is taken as 0.001 and the batch size is taken as 32.
The discount factor γ (taking value from 0 to 1) in the DRL model is
an important parameter that reflects how much attention the model pays
to the delayed rewards of actions. For solving control problems, such as
playing Atari video games, consecutive input samples in an episode,
6
G. Qian and J. Liu Progress in Nuclear Energy 152 (2022) 104401

Fig. 7. Experimental setup (a) CWRU bearing fault experiment (Bearing Data Center | Case School of Engineering | Case Western Reserve University n.d.) (b) UOC
gearbox fault experiment (Cao et al. 2018).

Table 1
Detailed information of fault datasets in case study.
Dataset Machinery type Fault type Sample size Sample length Label

A Rolling bearing H,IR1,IR2,IR3, 100 in each class, 1000 in total 1024 0,1,2,3,4,5,6,7,8,9
OR1,OR2,OR3,
B1,B2,B3
B Gear H,MT,RC,SP, 208 in each class, 1872 in total 1800 0,1,2,3,4,5,6,7,8
CT1,CT2,CT3,CT4,CT5

H= Normal (i.e., health).

IR1~IR3 = Inner race fault with size 0.18 mm, 0.36 mm, 0.54 mm.
OR1~OR3 = Outer race fault with size 0.18 mm, 0.36 mm, 0.54 mm.
B1~B3=Ball fault with size 0.18 mm, 0.36 mm, 0.54 mm.
MT = missing tooth, RC = root crack, SP = spalling, CT1~CT5 = chipping tip with 5 severity levels.
The label field corresponds to the value of the fault type field.

corresponding to game screenshots at different moments, are strongly
correlated, and the game action at the previous step has a great influence
on the next action. So the model needs to pay much attention to the
delayed reward of each action, and γ is usually taken as 0.99 (Mnih et al.,
2015). However, in the fault diagnosis scenario of this paper, each
sample contains enough fault information for the DRL model to make
type prediction action at a time, which indicates that the guessing ac­
tions are relatively independent. So the model should focus more on the
current reward and the discount factor γ should take a small value.
Results of the reference (Wang et al., 2022) show that the model per­
formance tends to decrease when γ increases from 0.1 to 0.9. In the
reference (Ding et al., 2019), γ takes the value of 0. Therefore, the dis­
count factor γ is taken as 0 in this paper.
The baseline models chosen in this paper are CNN, GRU and SVM.
The architecture of CNN and GRU model are shown in Figs. 2 and 4,
respectively. To enhance comparability, the main hyper-parameter set­
tings of CNN and GRU models are consistent with the corresponding
DRL model described in Table 2. SVM is a classic machine learning
model with powerful nonlinear fitting ability and small sample learning
capability (Cortes and Vapnik 1995; Yang et al. 2007). Feature engi­
neering approach similar to (Zhong and Ban 2022a) is selected. The
normalized vibration energy percentage distribution is extracted by
3-layer WPT and are used as input to the SVM model. Scikit-learn
(v1.0.1) (Pedregosa et al., 2011) is used to build SVM model and its
hyper-parameters (e.g, regularization parameter, kernel parameter) is
automatically optimized by the “hyperopt” toolkit in Python (Bergstra
et al., 2015).
3.4. Model evaluation
To verify that the model can be applied to multiple rotating
machinery, we evaluate model performance on 2 fault datasets,
respectively. During the evaluation process, dataset A or B is further
divided into 3 sub-datasets: a training set, a validation set and a test set
in the sample size ratio of 5:2:3. The models are trained on the training
set. The validation set is used to mark the model performance during
training and select the best model for every case. The test set is not
involved in the model training and parameter optimization, and is only
used to evaluate the model performance finally.
We use accuracy (see Eq. (7)) on the test set as the main evaluation
metric. The accuracy can reflect the comprehensive classification ability
of the model for all sample types. In each case, 10 experiments are
conducted and the mean test accuracy is taken as the evaluation result.
∑N
I(̃y = yi )
acc = i=1 i × 100% (7)
N
where I( ⋅) is the indicator function, which return 1 if condition is valid
and 0 otherwise. N is the test set size. ̃
yi and yi are the predicted and true
label of the i-th sample, respectively.
4. Results and discussion
4.1. Analysis of DRL training process
In this section, the training process of DRL is analyzed in detail to
verify the effectiveness of fault diagnosis. Specific contents include the
analysis of training accuracy curve, cumulated reward, single-sample
identification process and hidden layer visualization. According to our
experimental results, the training processes of CNN-RL and GRU-RL
show high similarity, and GRU-RL can achieve better performance
(see Section 4.2 and 4.3), so the training process of GRU-RL model is
selected for presentation here.

7
G. Qian and J. Liu Progress in Nuclear Energy 152 (2022) 104401

Fig. 8. Visualization of samples, 10 types on dataset A and 9 types on dataset B (a) (c) vibration time-domain waveform; (b) (d) frequency spectrum, y label
represents the abbreviation of fault type name.

Table 2
Main parameter settings of the proposed DRL and deep learning models.
Model Parameter Value

dataset A dataset B

CNN, CNN-RL Input size 512 × 1 900 × 1

Conv-1 4@3 × 1 (4 filters with same size 3 × 1)
Pool-1 Max pooling, size 2 × 1, stride 2
Conv-2 8@3 × 1
Pool-2 Max pooling, size 2 × 1, stride 2
FC layer [1024,32] [1800,32]
GRU, GRU-RL Input size 32 30
Time step 16 30
Hidden units 32 32
FC layer [512,32] [900,32]
CNN-RL,GRU-RL Target output size 1 1
(Q value)
Policy output size 10 9
(Action)
CNN, GRU Output size 10 9

The number of training epoch (episode) is set to 200, and the training
accuracy at the end of each epoch is recorded. Training accuracy reflects
the model’s ability of feature extraction and pattern recognition on the
training set. As shown in Fig. 9(a)(c), the training accuracy of the GRU
model rises faster than the GRU-RL model, reaching convergence status
first. This is due to the fact that supervised learning is a strong feedback
training mechanism that uses the sample labels to guide the updating of
GRU model parameters. While GRU-RL model is training in a weak
feedback mechanism that uses scalar reward signal (taken − 1 or 1) to
evaluate the model behavior and update its parameters in the direction
that may maximize cumulative reward in the interaction. For the model
stability, as can be seen in the embedded small plot in Fig. 9(a), GRU-RL
model behaves more stably when it converges, while the accuracy of
GRU model fluctuates. Although the training process of the GRU-RL
model is slower, its performance is more stable after reaching
convergence status.
Next, the variation characteristics of the cumulative reward values
are analyzed. In each epoch, the GRU-RL model performs 32 rounds of
guessing games. It gets a reward of 1 point when the guess is correct, and
− 1 point (penalty signal) for an incorrect guess. Thus, the cumulative
reward value is taken in the range of [− 32, 32], where 32 represents
entirely correct, − 32 means entirely wrong. A larger reward value im­
plies a more accurate guessing policy or capability embodied in the
model. Fig. 9(b)(d) shows the 5 epoch-mean cumulative reward value of
the GRU-RL model during the training process. As the model is contin­
uously trained, the cumulative reward value gradually increases and
finally converges to 32. It implies that the model can learn the correct
classification policy and can complete the guessing game excellently.
We further analysis the training process of the GRU-RL model from
the perspective of single-sample identification. One fixed sample is

8
G. Qian and J. Liu Progress in Nuclear Energy 152 (2022) 104401

Fig. 9. Training accuracy and mean cumulative reward on (a) (b) dataset A, (c) (d) dataset B.

Fig. 10. Identification results of the GRU-RL model for a single fixed sample on (a) Dataset A, (b) Dataset B.

selected from datasets A and B, respectively. At the end of each training
epoch, the type of the fixed sample is predicted and recorded. Fig. 10
shows the identification results for the fixed samples at different training
stages. The horizontal coordinate represents the epoch number and the
vertical coordinate represents the identification result. At the beginning,
the model is in the exploration and trial-and-error phase, and the pre­
diction results are randomly distributed. After enough training epochs,
the model prediction results gradually converge to the true labels. This
re-verifies that the GRU-RL model can learn the correct classification
strategy on the 2 different rotating machinery datasets.
Now, we use the t-SNE algorithm (Van Der Maaten and Hinton,
2008) to downscale and visualize the first FC layer state of the GRU-RL
model. The FC layer contains sample feature information extracted by
the model. As shown in Fig. 11, each point in the figure represents a
sample, and different colors represent different sample labels (condition
types). Different types of samples are scattered in relatively independent
local regions, which indicates that the GRU-RL model can accurately
extract the abstract representation of fault features in the selected
datasets.

9
G. Qian and J. Liu Progress in Nuclear Energy 152 (2022) 104401

Fig. 11. 2D visualization of the GRU-RL model’s first FC layer state by t-SNE on (a) dataset A, (b) dataset B.

4.2. Model performance comparison

Table 3
Model performance under initial training sample size.
This section compares the test accuracy of different models at the
Model Dataset initial sample size. Dataset A has 500 training set samples and dataset B
A (Bearing) B (Gear) has 936 training set samples. The performance of the 5 models is
SVM 89.47 ± 1.38 90.27 ± 1.00 compared according to the evaluation process described in Section 3.4.
CNN 98.73 ± 1.81 95.31 ± 1.62 Table 3 shows the accuracy evaluation results of all models on the test
GRU 99.90 ± 0.20 99.11 ± 0.65 set. The order of model performance superiority on the 2 datasets both
CNN-RL 99.95 ± 0.15 99.61 ± 0.50 are: GRU-RL > CNN-RL > GRU > CNN > SVM. The 2 presented DRL
GRU-RL 99.98 ± 0.05 99.95 ± 0.11
models can achieve very high accuracy of over 99%, showing excellent
Note: mean test accuracy ± standard deviation [%] of 10 experiments. diagnostic capability.
The DRL model has the ability to learn interactively with the envi­
ronment and integrates the advantages of automatic feature extraction
from the deep learning model, allowing the model to deeply mine and
understand the essential pattern features of the environment (i.e., fault

Fig. 12. Model performance under small sample scenarios, test accuracy and its standard deviation on (a) (c) dataset A, (b) (d) dataset B.

10
G. Qian and J. Liu
datasets) and learn better fault diagnosis strategies than the deep
learning model.
SVM model uses the vibration signal energy features extracted by the
WPT algorithm, which may lose a part of the important fault informa­
tion contained in the original signal, compared to the other four models
(they use the original spectral information as input). Therefore, the SVM
has the lowest accuracy, which is consistent with the common sense.
Nevertheless, SVM achieves or approaches 90% accuracy and has good
diagnostic performance.
4.3. Performance under small samples
In real industry environment, with high safety requirements, espe­
cially for nuclear industry and NPPs, the rotating machinery operates in
normal state most of the time, forming a fault diagnosis problem under
small samples. Model performance under small sample scenario is more
informative for engineering applications. In this section, small sample
scenario is simulated by selecting a small percentage (5%–30%) of the
training set for model training.
Fig. 12 shows the evaluation results under different training sample
ratio (TSR). On dataset A, the accuracy of CNN-RL, GRU-RL and GRU
models change less when the TSR decreases from 30% to 5%, while the
accuracy of CNN and SVM models show a larger degree of decrease,
which reflects that they are more sensitive to sample reduction. The
accuracy of the 2 DRL models are higher than the other baseline models
at different TSRs. Overall, the GRU-RL was slightly better than the CNN-
RL model, with smaller standard deviation and better stability. When the
TSR was 5%, GRU-RL achieves an average of 93.9% accuracy and CNN-
RL achieves an average of 95.8% accuracy, both of which achieve very
high diagnostic accuracy in the small sample cases.
There is a similar pattern on dataset B. When the TSR decreases from
30% to 5%, the accuracy of both 2 DRL models is higher than the other
baseline models. When the TSR was 5%, GRU-RL achieves an average of
97.3% accuracy and CNN-RL achieves an average of 89.1% accuracy,
both of which achieve very high diagnostic accuracy in the small sample
cases.
The proposed DRL method has a strong fault diagnosis capability in
the case of small samples, surpassing the deep learning models CNN and
GRU and the classic SVM model (although SVM has a strong small
sample learning capability). The DRL model is trained by interacting
with the environment continuously. By lots of trial-and-error, DRL
adapts its behavior and decision policy to maximize the reward signal
from the environment. During this complex training process, DRL model
can learn more essential features of the environment (i.e., fault dataset)
and focus on the most critical information in the data, rather than simply
fit the function relationship of samples and their labels. Therefore, DRL
can achieve better performance.
With the network architecture setting in this paper, GRU-RL out­
performs the CNN-RL model, especially on dataset B. This may be
attributed to the mechanism of GRU network to process information
recurrently. The vibration signal spectrum has strong sequence features.
GRU can hold the temporal information features of the whole sequence
in the extracted feature vector. In contrast, CNN focuses more on the
local features of the data and lacks the mining of the overall temporal
features of the information.
4.4. Discussions
Since the verification datasets in this paper are from laboratory, but
not the real datasets from NPPs, we cannot guarantee that the proposed
model can achieve 99% accuracy in real situations. To see the bright
side, this paper gives a specific and effective model architecture design
and some valuable experiment results of DRL-based fault diagnosis
method for rotating machinery. The experimental results in this paper
can demonstrate the relative superiority of the model performance to
some extent. This information may be very useful and valuable for NPP
11
Progress in Nuclear Energy 152 (2022) 104401
engineers and lay a foundation for further study. For further engineering
applications, there are two more points, i.e., data source and sample size
(total number of samples, even sample classes), which need to be dis­
cussed here.
For point 1: data source. The datasets used in this paper are from
laboratories (with low noise and less disturbance compared to real in­
dustrial measurements), the monitored equipment types may differ from
the specific ones used in NPPs, and the sensors and signal acquisition
frequency may also exist discrepancies, which result in the trained
model in this paper cannot directly deployed and delivered for NPP use.
However, the modeling idea and process can be directly migrated and
provide valuable reference for NPP engineers of equipment
maintenance.
For point 2: data size. Compared to image datasets, such as ImageNet
which has more than 15 million images and over 22,000 categories
available for building deep learning models (Krizhevsky et al. 2017), the
training set used in this paper is small, whose maximum size is 500 (10
categories) for dataset A and 936 (9 categories) for dataset B. Related
publications also use similar size of training set to build deep learning
fault diagnosis models, such as (Gan et al. 2016; Guo et al. 2016) both
used 500 samples for model training and (Cao, Zhang, and Tang 2018)
used 747 samples for model training. This proves that such training set
size can meet the requirements for building deep learning-based rotating
machinery fault diagnosis model, at least for laboratory datasets.
Two more important issues related to data size are overfitting
problem and generalization ability. Overfitting is a common problem in
deep learning model training. Due to the powerful fitting ability of deep
learning models, it is easy to fit unimportant features and noise in the
dataset, leading to overfitting. When overfitting occurs, the model has
high accuracy on the training set but low accuracy on the test set. In the
experimental results of this paper, the proposed deep learning and DRL
models can achieve high accuracy both in the training set and the test set
most of the time. They do not show significant overfitting until the size
of the training set was reduced to a certain level (e.g., TSR = 5%).
Generalization ability refers to the ability of the model to adapt to
unknown data. Test accuracy is a measure of the model generalization
ability. Although the proposed DRL models achieve high test accuracy,
their generalization ability to real NPP datasets still needs further study
and validation. Since the high safety requirements in NPPs and con­
ducting equipment fault experiments are very expensive, small sample
problem is an objective situation and will last for years. Future studies
should dedicate to not only the advanced model development, but also
the large-scale fault dataset establishment for NPPs.
5. Conclusions
In this paper, we propose 2 DRL fault diagnosis models based on CNN
and GRU models, analysis the model training process carefully, and
compare the performance between them and with 3 baseline models. 2
experimental fault datasets of bearing and gear are selected as case
studies. The results indicate that:
1) the DRL model converges more slowly than deep learning model
but has better stability after reaching convergence status; 2) the pro­
posed 2 DRL models outperform traditional deep learning models (CNN,
GRU) and classic machine learning model (SVM) in both initial sample
size and small sample scenarios. This is attributed to the combination
advantages of powerful feature extraction from deep learning and
interactive learning ability from reinforcement learning, which moti­
vates the models to learn more essential features with the weak reward
feedback signal; 3) Based on the network architecture design in this
paper, the GRU-RL performance is slightly better than the CNN-RL
model, which may be attributed to the intrinsic ability of the GRU
network to extract the overall sequence information of the vibration
spectrum and the splicing of all hidden states in the recurrent unit.
Subsequent studies will further investigate the fault diagnosis per­
formance of the DRL model under varying working conditions, strong
G. Qian and J. Liu Progress in Nuclear Energy 152 (2022) 104401

environment noise and sample imbalance scenarios using real equip­ interests or personal relationships that could have appeared to influence
ment datasets or simulated system-level fault datasets and promote the the work reported in this paper.
engineering application of the DRL model in NPPs. In addition, there are
still many interesting topics and challenges, such as Monte Carlo tree Data availability
search technique (Silver et al., 2017) in DRL is not introduced for fault
diagnosis studies by now, model training efficiency needs to be Data will be made available on request.
improved (Ding et al., 2019; Wang et al., 2022) and network architec­
ture optimization problem (Cao et al., 2022). In the future study, we will Acknowledgement
also consider promoting these techniques.
We appreciate the editors and anonymous reviewers very much for
Declaration of competing interest their precious time and valuable comments.

The authors declare that they have no known competing financial

Appendix

Appendix Algorithm A1 describes the training process of DQN models.

References Chen, S., et al., 2020. Robust deep learning-based diagnosis of mixed faults in rotating
machinery. IEEE ASME Trans. Mechatron. 25 (5), 2167–2176.
Chung, Junyoung, Gulcehre, Caglar, Cho, Kyunghyun, Bengio, Yoshua, 2014. Empirical
Abadi, TensorFlow, 2016. A system for large-scale machine learning. In: Proceedings of
evaluation of gated recurrent neural networks on sequence modeling. In: NIPS 2014
the 12th USENIX Conference on Operating Systems Design and Implementation,
Workshop on Deep Learning, December 2014.
OSDI’16, USA. USENIX Association, 265–83.
Cooley, James W., Tukey, John W., 1965. An algorithm for the machine calculation of
Arulkumaran, K., Deisenroth, M.P., Brundage, M., Bharath, A.A., 2017. Deep
complex fourier series. Math. Comput. 19, 297–301.
reinforcement learning: a brief survey. IEEE Signal Process. Mag. 34 (6), 26–38.
Cortes, Corinna, Vapnik, Vladimir, 1995. Support-vector networks. Mach. Learn. 20 (3),
Ayo-Imoru, R.M., Cilliers, A.C., 2018. A survey of the state of condition-based
273–297. https://doi.org/10.1007/BF00994018.
maintenance (CBM) in the nuclear power industry. Ann. Nucl. Energy 112, 177–188.
Ding, Yu, et al., 2019. intelligent fault diagnosis for rotating machinery using deep Q-
https://www.sciencedirect.com/science/article/pii/S0306454917303365.
network based health state classification: a deep reinforcement learning approach.
Bearing Data Center. Case School of engineering | case western Reserve university
Adv. Eng. Inf. 42, 100977. https://www.sciencedirect.com/science/article/pii/S14
[WWW Document], n.d. https://engineering.case.edu/bearingdatacenter. (Accessed
74034619305506.
2 July 2022).
Duan, Q., et al., 2020. fault diagnosis of air compressor in nuclear power plant based on
Bergstra, James, et al., 2015. Hyperopt: a Python library for model selection and
vibration observation window. IEEE Access 8, 222274–222284.
hyperparameter optimization. Comput. Sci. Discov. 8 (1), 014008 https://doi.org/
Fan, S., Zhang, X., Song, Z., 2022. Imbalanced sample selection with deep reinforcement
10.1088/1749-4699/8/1/014008.
learning for fault diagnosis. IEEE Trans. Ind. Inf. 18 (4), 2518–2527.
Biet, M., 2013. Rotor faults diagnosis using feature selection and nearest neighbors rule:
Feng, Yi, et al., 2018. Pump Bearing Fault Detection Based on EMD and SVM.” Volume 1:
application to a turbogenerator. IEEE Trans. Ind. Electron. 60 (9), 4063–4073.
Operations and Maintenance, Engineering, Modifications, Life Extension, Life Cycle,
Cao, P., Zhang, S., Tang, J., 2018. Preprocessing-free gear fault diagnosis using small
and Balance of Plant; Instrumentation and Control (I&C) and Influence of Human
datasets with deep convolutional neural network-based transfer learning. IEEE
Factors; Innovative Nuclear Power Plant Design and SMRs.
Access 6, 26241–26253.
Gan, Meng, Wang, Cong, Zhu, Chang‫׳‬an, 2016. Construction of hierarchical diagnosis
Cao, Jie, Ma, Jialin, Huang, Dailin, Yu, Ping, 2022. Finding the optimal multilayer
network based on deep learning and its application in The fault pattern recognition
network structure through reinforcement learning in fault diagnosis. Measurement
of rolling element bearings. Mech. Syst. Signal Process. 72 (73), 92–104. https://
188, 110377. https://www.sciencedirect.com/science/article/pii/S0263224
www.sciencedirect.com/science/article/pii/S0888327015005312.
121012707.

12
G. Qian and J. Liu
Glavic, Mevludin, 2019. Deep) reinforcement learning for electric power system control
and related problems: a short review and perspectives. Annu. Rev. Control 48,
22–35. https://www.sciencedirect.com/science/article/pii/S1367578819301014.
Guo, Xiaojie, Chen, Liang, Shen, Changqing, 2016. Hierarchical adaptive deep
convolution neural network and its application to bearing fault diagnosis.
Measurement 93, 490–502. https://www.sciencedirect.com/science/article/pii/
S0263224116304249.
Jordan, M.I., Mitchell, T.M., 2015. Machine learning: trends, perspectives, and prospects.
Science 349 (6245), 255–260. https://doi.org/10.1126/science.aaa8415.
Kingma, Diederik, P., Jimmy, Ba, 2015. Adam: A Method for Stochastic Optimization,
p. 6980. CoRR abs/1412.
Koo, In Soo, Kim, Whan Woo, 2000. The development of reactor coolant pump vibration
monitoring and a diagnostic system in the nuclear power plant. ISA (Instrum. Soc.
Am.) Trans. 39 (3), 309–316. https://www.sciencedirect.com/science/article/pii/
S0019057800000197.
Krizhevsky, Alex, Sutskever, Ilya, Hinton, Geoffrey E., 2017. ImageNet classification with
deep convolutional neural networks. Commun. ACM 60 (6), 84–90. https://doi.org/
10.1145/3065386.
Lebold, M.S., et al., 2004. Using torsional vibration analysis as a synergistic method for
crack detection in rotating equipment. In: 2004 IEEE Aerospace Conference
Proceedings (IEEE Cat. No.04TH8720), vol. 6, pp. 3517–3527.
LeCun, Yann, Bengio, Yoshua, Hinton, Geoffrey, 2015. Deep learning. Nature 521
(7553), 436–444. https://doi.org/10.1038/nature14539.
Li, G., et al., 2021. Deep reinforcement learning-based online domain adaptation method
for fault diagnosis of rotating machinery. IEEE ASME Trans. Mechatron. 1–10.
Li, Wei, et al., 2022. Multi-mode data augmentation and fault diagnosis of rotating
machinery using modified ACGAN designed with new framework. Adv. Eng. Inf. 52,
101552. https://www.sciencedirect.com/science/article/pii/S1474034622000271.
Liu, Meiru, et al., 2015. Vibration signal analysis of main coolant pump flywheel based
on hilbert–huang transform. Nucl. Eng. Technol. 47 (2), 219–225. https://www.sci
encedirect.com/science/article/pii/S1738573315000108.
Ma, Jianping, Jiang, Jin, 2011. Applications of fault detection and diagnosis methods in
nuclear power plants: a review. Prog. Nucl. Energy 53 (3), 255–266. https://www.
sciencedirect.com/science/article/pii/S0149197010001769.
Miki, Daisuke, Demachi, Kazuyuki, 2020. bearing fault diagnosis using weakly
supervised long short-term memory. J. Nucl. Sci. Technol. 57 (9), 1091–1100.
https://doi.org/10.1080/00223131.2020.1761473.
Mnih, Volodymyr, et al., 2013. Playing Atari with Deep Reinforcement Learning. ArXiv
abs/1312.5602.
Mnih, Volodymyr, et al., 2015. Human-level control through deep reinforcement
learning. Nature 518 (7540), 529–533. https://doi.org/10.1038/nature14236.
Nguyen, T.T., Reddi, V.J., 2021. Deep reinforcement learning for cyber security. IEEE
Transact. Neural Networks Learn. Syst. 1–17.
Pan, Tongyang, et al., 2021. Generative Adversarial Network in Mechanical Fault
Diagnosis under Small Sample: A Systematic Review on Applications and Future
Perspectives. ISA Transactions. https://www.sciencedirect.com/science/article/pii/
S0019057821006169.
Park, JaeKwan, Kim, TaekKyu, Seong, SeungHwan, 2020. Providing support to operators
for monitoring safety functions using reinforcement learning. Prog. Nucl. Energy
118, 103123. https://www.sciencedirect.com/science/article/pii/S0149197
01930232X.
Park, JaeKwan, Kim, TaekKyu, Seong, SeungHwan, Koo, SeoRyong, 2022. Control
automation in the heat-up mode of a nuclear power plant using reinforcement
learning. Prog. Nucl. Energy 145, 104107. https://www.sciencedirect.com/science/
article/pii/S0149197021004595.
Pedregosa, Fabian, et al., 2011. Scikit-learn: machine learning in Python. J. Mach. Learn.
Res. 12 (null), 2825–2830.
Qian, Gensheng, Liu, Jingquan, 2022. fault diagnosis based on conditional generative
adversarial networks in nuclear power plants. Ann. Nucl. Energy 176, 109267.
https://www.sciencedirect.com/science/article/pii/S0306454922003024.
13
Progress in Nuclear Energy 152 (2022) 104401
Radaideh, Majdi, I., Forget, Benoit, Shirvan, Koroush, 2021. Large-scale design
optimisation of boiling water reactor bundles with neuroevolution. Ann. Nucl.
Energy 160, 108355. https://www.sciencedirect.com/science/article/pii/S0306454
921002310.
Rasamoelina, Andrinandrasana David, Adjailia, Fouzia, Peter, Sinčák, 2020. A review of
activation function for artificial neural network. In: 2020 IEEE 18th World
Symposium on Applied Machine Intelligence and Informatics (Sami), pp. 281–286.
Ravanelli, M., Brakel, P., Omologo, M., Bengio, Y., 2018. Light gated recurrent units for
speech recognition. IEEE Transactions on Emerging Topics in Computational
Intelligence 2 (2), 92–102.
Silver, David, et al., 2017. Mastering the game of Go without human knowledge. Nature
550 (7676), 354–359. https://doi.org/10.1038/nature24270.
Sinha, Jyoti K., 2008. Vibration-based diagnosis techniques used in nuclear power plants:
an overview of experiences. Nucl. Eng. Des. 238 (9), 2439–2452. https://www.sci
encedirect.com/science/article/pii/S0029549308001556.
Smith, H.R., Wiedenbrug, E., Lind, M., 2007. Rotating element bearing diagnostics in a
nuclear power plant: comparing vibration and torque techniques. In: 2007 IEEE
International Symposium on Diagnostics for Electric Machines, Power Electronics
and Drives, , 17–22.
Sutton, R.S., Barto, A.G., 2018. Reinforcement Learning: an Introduction, second ed.
Cambridge.
Van Der Maaten, Laurens, Hinton, Geoffrey, 2008. Visualizing data using T-SNE. J. Mach.
Learn. Res. 9 (2605), 2579–2605.
Wang, Zisheng, Xuan, Jianping, 2021. intelligent fault recognition framework by using
deep reinforcement learning with one dimension convolution and improved actor-
critic algorithm. Adv. Eng. Inf. 49, 101315. https://www.sciencedirect.com/scienc
e/article/pii/S1474034621000689.
Wang, H., et al., 2022. intelligent fault diagnosis for planetary gearbox using time-
frequency representation and deep reinforcement learning. IEEE ASME Trans.
Mechatron. 27 (2), 985–998.
Wang, Zhichao, et al., 2022. Cross-domain fault diagnosis of rotating machinery in
nuclear power plant based on improved domain adaptation method. J. Nucl. Sci.
Technol. 59 (1), 67–77. https://doi.org/10.1080/00223131.2021.1953630.
Yang, Junyan, Zhang, Youyun, Zhu, Yongsheng, 2007. intelligent fault diagnosis of
rolling element bearing based on SVMs and fractal dimension. Mech. Syst. Signal
Process. 21 (5), 2012–2024. https://www.sciencedirect.com/science/article/pii/
S0888327006002251.
Zhang, Yahui, et al., 2021. fault diagnosis of rotating machinery based on recurrent
neural networks. Measurement 171, 108774. https://www.sciencedirect.com/scienc
e/article/pii/S0263224120312732.
Zhong, Xianping, Ban, Heng, 2022a. Crack fault diagnosis of rotating machine in nuclear
power plant based on ensemble learning. Ann. Nucl. Energy 168, 108909. https://
www.sciencedirect.com/science/article/pii/S0306454921007866.
Zhong, Xianping, Ban, Heng, 2022b. Pre-trained network-based transfer learning: a
small-sample machine learning approach to nuclear power plant classification
problem. Ann. Nucl. Energy 175, 109201. https://www.sciencedirect.com/science/
article/pii/S0306454922002365.
Zhong, Shi-sheng, Fu, Song, Lin, Lin, 2019. A novel gas turbine fault diagnosis method
based on transfer learning with CNN. Measurement 137, 435–453. https://www.sci
encedirect.com/science/article/pii/S0263224119300405.
Zhu, Shaomin, et al., 2021. Feature extraction for early fault detection in rotating
machinery of nuclear power plants based on adaptive VMD and teager energy
operator. Ann. Nucl. Energy 160, 108392. https://www.sciencedirect.com/science/
article/pii/S0306454921002681.
Zou, Fengqian, et al., 2021. An anti-noise one-dimension convolutional neural network
learning model applying on bearing fault diagnosis. Measurement 186, 110236.
https://www.sciencedirect.com/science/article/pii/S0263224121011453.