This article has been accepted for publication in a future issue of this journal, but has not been
fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TII.2020.3004232, IEEE
Transactions on Industrial Informatics
A Reinforcement Learning-based Network Traffic
Prediction Mechanism in Intelligent Internet of
Things
Laisen Nie, Zhaolong Ning, Mohammad S. Obaidat, Life Fellow, IEEE, Balqies Sadoun, Huizhi Wang,
Shengtao Li, Lei Guo, and Guoyin Wang
Abstract—Intelligent Internet of Things (IIoT) is comprised of
various wireless and wired networks for industrial applications,
which makes it complex and heterogeneous. The openness of
IIoT has led to the intractable problems of network security
and management. Many network security and management
functions rely on network traffic prediction techniques, such as
anomaly detection and predictive network planning. Predicting
IIoT network traffic is significantly difficult because its frequently
updated topology and diversified services lead to irregular
network traffic fluctuations. Motivated by these observations,
we proposed a reinforcement learning-based mechanism. We
modelled the network traffic prediction problem as a Markov
decision process, and then predicted network traffic by Monte-
Carlo Q-learning. Furthermore, we addressed the real-time
requirement of the proposed mechanism and we proposed a
residual-based dictionary learning algorithm to improve the
complexity of Monte-Carlo Q-learning. Finally, the effectiveness
of our mechanism was evaluated using real network traffic.
Index Terms—Intelligent Internet of Things, network traffic
prediction, reinforcement learning, industrial applications.
I. I NTRODUCTION
L. Nie is with the School of Electronics and Information, Northwestern
Polytechnical University, Xi’an, 710072, China, also with the National Mobile
Communications Research Laboratory, Southeast University, Nanjing 210096,
China, and also with the Qingdao Research Institute of Northwestern Poly-
technical University, Qingdao, China. Email: nielaisen@nwpu.edu.cn.
Z. Ning (Corresponding author) is with the National Mobile Communica-
tions Research Laboratory, Southeast University, Nanjing 210096, China, also
with the Chongqing Key Laboratory of Mobile Communications Technology,
Chongqing University of Posts and Telecommunications, Chongqing 400065,
China, and also with the School of Software, Dalian University of Technology,
Dalian 116024, China. Email: zhaolongning@dlut.edu.cn.
M. Obaidat is with the College of Computing and Informatics, The
University of Sharjah, Sharjah 27272, UAE; The King Abdullah II School of
Information Technology, The University of Jordan, Amman 11942, Jordan;
and University of Science and Technology Beijing, Beijing 100083, China.
Email: msobaidat@gmail.com.
B. Sadoun is with College of Engineering, University of Sharjah, Sharjah
27272, UAE and with College of Engineering, Al-Balqa Applied University,
Al-Salt 19117, Jordan. Email: sadounbalqies@gmail.com.
H. Wang is with the School of Electronics and Information,
Northwestern Polytechnical University, Xi’an, 710072, China. Email:
wanghuizhi@mail.nwpu.edu.cn.
S. Li is with the School of Information Science and Engineering, Shandong
Normal University, Jinan, China. Email: saintaolee@sdnu.edu.cn.
L. Guo is with School of Communication and Information Engineering,
Chongqing University of Posts and Telecommunications, Chongqing 400000,
China. Email: guolei@cqupt.edu.cn.
G. Wang is with the Chongqing Key Laboratory of Computational Intel-
ligence, Chongqing University of Posts and Telecommunications, Chongqing
400065, China. Email: wanggy@cqupt.edu.cn.
1551-3203 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: University of Exeter. Downloaded on June 26,2020 at 20:54:25 UTC from IEEE Xplore. Restrictions apply.
1
T O provide support to Intelligent Internet of Things (IIoTs)
with adequate throughput for industrial applications, the
5G-enabled communication network, which is involved in
IIoTs, is expected to be complex and heterogeneous with a
dense deployment of infrastructures. Aiming at distinct kinds
of industrial applications, various access techniques have been
used in 5G-enabled networks to guarantee the quality-of-
service and quality-of-experience of IIoTs [1]–[4]. In this case,
the network security and management obtain unprecedented
attentions. Nevertheless, the openness of 5G heterogeneous
networks (5G HetNets) arises the main challenges in network
security. Meanwhile, the dense deployment of infrastructure
using smaller cells requires more efficient management mech-
anisms [5]–[7].
Network traffic prediction is one of the most important
bases for network security and management functions, such
as intrusion detection and anomaly detection [8]–[10].Network
traffic information can be used in anomaly detection resulted
by network attacks or malfunctions. In a wireless network,
network traffic information is adopted for both safety-related
and non-safety-related applications, such as predictive network
planning, energy-efficient routing, and aggregation traffic de-
tection [11], [12].
There are a great number of researchers exploring network
traffic prediction mechanisms for traditional networks, such
as IP backbone and cellular networks [13], [14]. Statistical
model-based mechanisms, where a linear (or non-linear) model
is built to extract the features of traffic flows as a prior, have
been proposed to predict network traffic in various scenarios.
Unfortunately, these prediction mechanisms are not suitable
for IIoTs to empower industrial systems. The primary reason
for this is that the statistical characteristics of network traffic in
IIoTs is very different to those of other networks. For instance,
IP backbone network traffic often obeys a regular and periodic
profile. On the contrary, the traffic flows of IIoTs have many
irregular fluctuations, increasing the difficulty of prediction.
Machine learning has been acknowledged as an available
solution for the problem of network traffic prediction in
traditional networks. Nevertheless, there are several challenges
in machine learning-based network traffic prediction when
these mechanisms are utilized in the same way for IIoT traffic
prediction [15], [16]. The main challenges can be summarized
as follows:
• In an IIoT, infrastructure is deployed much more densely
than in other networks. The number of end-to-end net-
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TII.2020.3004232, IEEE
Transactions on Industrial Informatics
work traffic increases with the number of nodes. Hence,
the computational complexity of predicting IIoT traffic
by way of machine learning is greater, which also makes
real-time prediction more difficult.
• Machine learning-based network traffic prediction con-
sumes many computing and memory resources for net-
work management. This is because these algorithms must
sample a prior of network traffic as a training data set.
• Collecting such prior network traffic also consumes the
resources of network edge. The IIoT is built by vari-
ous access techniques, but not all of these can support
sufficient resources for network traffic sampling. For
instance, the energy of nodes in a wireless sensor network
cannot provide sufficient power to deploy network traffic
sampling.
Thereby, though network traffic prediction is crucial for vari-
ous perspectives and many prediction mechanisms have been
proposed, it is still an intractable challenge to predict the
end-to-end network traffic in IIoTs. Existing network traffic
prediction mechanisms mainly make a trade-off between the
cost of the algorithm and its accuracy.
Motivated by the above challenges, we focus on the problem
of end-to-end network traffic prediction in IIoTs, and propose
a mechanism based on Reinforcement Learning (RL) for
real-time traffic prediction. Addressing the issues of high
consumption caused by learning a large-scale training data set
and sampling a great deal of network traffic, we propose a
Monte-Carlo Q-learning algorithm. In details, we first model
the end-to-end network traffic as a Markov Decision Process
(MDP) via the prior of network traffic. Then, we predict
the traffic flow using the proposed Monte-Carlo Q-learning
algorithm. Nevertheless, the volume of each traffic flow is
tremendous, making defining a series of actions impossible.
Therefore, to reduce the computational complexity of the
proposed algorithm, we propose a residual-based adaptive
dictionary learning algorithm to project the network traffic to
another space. In this many, network traffic can be denoted
by few coefficients, which decreases the number of actions as
a result of the reduced computational complexity. The main
contributions of this paper can be summarized as follows:
• We consider the irregular fluctuations of network traffic
in IIoTs, and propose an efficient algorithm based on
RL to extract short-term time-varying features of network
traffic We combine Monte-Carlo learning, Q-learning, and
Kullback-Leibler (KL) divergence [17] for prediction.
• With the aim of running the prediction mechanism in real
time, a greedy adaptive dictionary learning algorithm is
proposed, where a residual term is defined via the `0 - and
`2 -norms to make the coefficients fewer and smaller so
that the number of actions and computational complexity
are reduced.
• An evaluation of the proposed mechanism using real
network traffic data sets via our testbed and the GÉANT
network is implemented. Numerical results indicate that
the Monte-Carlo Q-learning-based mechanism can cap-
ture the short-term features of traffic flows in IIoTs better.
The remainder of this paper is organized as follows. Section
1551-3203 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: University of Exeter. Downloaded on June 26,2020 at 20:54:25 UTC from IEEE Xplore. Restrictions apply.
2
II reviews related work. Section III presents the system model
of network traffic prediction. In Section IV, we present the RL-
based network traffic prediction mechanism for IIoTs. Section
V describes our evaluation of the proposed mechanism. Sec-
tion VI concludes our work and indicates future work.
II. R ELATED W ORK
A. Statistical Model-based Network Traffic Prediction
Statistical model-based mechanisms have been widely de-
veloped to predict network traffic [18]–[21]. These models
track the linear or non-linear features of end-to-end network
traffic for prediction. Typically, linear time series methods,
e.g., autoregressive model and autoregressive moving average
model, are used to model end-to-end network traffic for
prediction. Besides, the GARCH model in [19] was utilized
to extract non-linear features of traffic flows. In a wireless
network, traffic reveals a large number of irregular fluctuations.
Meanwhile, due to the random access, researchers usually
focus on short-term prediction of traffic flows. Therefore,
modeling short-range dependence is crucial for traffic pre-
diction. However, a single statistical model cannot match the
complex statistical features of traffic flows. As a result, hybrid
model-based mechanisms have appeared in the literature for
feature extraction. In [22], the authors took advantage of the
stationary wavelet transform, quantum genetic algorithm, and
backpropagation neural network to predict traffic flow in a
wireless network. The Markov model is a prevalent method
to extract the statistical features of end-to-end network traffic.
The authors in [23] utilized Markov chains with tensors to
deal with the problem of network traffic prediction, and a
multivariate multi-order Markov transition was proposed for
accurate multi-modal prediction. They first defined the tensor
unified product, and then proposed the general multivariate
multi-order Markov model. After that, a modified multi-step
transition tensor is put forward to predict network traffic.
B. Machine Learning-enabled Network Traffic Prediction
Currently, machine learning techniques have been widely
used in network traffic prediction to capture various features
of network traffic. To manage various networks jointly, the
software defined network has gradually become a common
networking technique. The authors in [24] studied network
traffic prediction for a mobile metro-core network based on
a software defined network. They constructed a feedforward
neural network with 3 layers to predict network traffic. The
short-range dependence was also considered. The authors
in [25] focused on the spatio-temporal features of wireless net-
work traffic, and a recurrent neural network was designed for
network traffic prediction. The authors not only considered the
short-range dependence but also the long-range dependence of
traffic flows. The Long Short-Term Memory (LSTM)-based
approach was deployed in optical data center networks to
predict the traces of traffic flows [26]. This LSTM-based deep
architecture contained 4 LSTM blocks as the hidden layer.
Similarly, the authors in [27] designed a deep architecture
based on the LSTM recurrent neural network with 2 hidden
layers to capture the spatio-temporal features of network
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TII.2020.3004232, IEEE
Transactions on Industrial Informatics
traffic for prediction in large-scale backbone networks. Each
hidden layer consisted of two LSTM blocks. Considering the
contradiction between real-time online prediction and the mass
of training data set required to train the deep architecture,
the authors in [28] took advantage of the transfer learning
technique to reduce the training time for the network traffic
prediction in cellular networks. LSTM was also adopted to
construct the deep architecture. Consequently, an effective
inter-cluster transfer learning approach was proposed, which
can reuse extracted features efficiently. The authors in [29]
studied the problem of network traffic prediction in software
defined network-enabled optical networks. They modeled the
problem of network traffic prediction using a Gaussian process
regression model; the proposed method took into account
the short-range and long-range dependencies of traffic flows
at the same time. The authors in [30] dealt with network
traffic prediction using an integer linear programming model.
Afterwards, a heuristic algorithm was designed to calculate
the NP-complete problem. The network traffic can be denoted
by a matrix, and it can be viewed as an image. Thereby,
many methods based on Convolutional Neural Network (CNN)
have emerged to deal with the problem of network traffic
prediction. The authors in [31] built a CNN to predict end-to-
end network traffic in a citywide cellular network. Different
from previous methods based on CNN, a densely connected
CNN was proposed, where more matrices were concurrently
added as input data to extract the spatial and temporal features
of the traffic matrix. However, due to the dense deployment of
infrastructure and larger energy consumption of sampling with
prior information, these methods are not suitable for IIoTs.
III. S YSTEM M ODEL
End-to-end network traffic can be denoted by a matrix [32].
This is a crucial input parameter for several kinds of network
management and security functions. In a traffic matrix, each
row vector describes the volume of traffic flow between
possible Origin-Destination (OD) node pairs, where the nodes
can be routers, switches, or point of presences. We denote
this matrix by X in this paper, and we denote each element of
the traffic matrix by X (n, t), where n is the index of an OD
flow and t denotes time slot. If the IIoT contains N nodes,
then n = 1, 2, 3, ..., N 2 . This consideration is established when
we consider an OD flow enters and exits our network at the
same node; otherwise, n = 1, 2, 3, ..., N (N − 1). Without loss of
generally, in this paper, we set n = 1, 2, 3, ..., N 2 . Meanwhile,
we assume that the traffic matrix X records the volume of
network traffic with T time slots, namely t = 1, 2, 3, ..., T.
The network traffic yields various statistical features. These
features can be grouped into three dimensions: temporal,
spatial, and spatio-temporal. The temporal feature represents
the time-varying of an OD flow, and the spatial feature defines
the relationship between two OD flows influenced by the
locations of users and servers in the logical topology. The
spatio-temporal feature indicates how close adjacent elements
in the traffic matrix are to each other in size. In our network
traffic prediction mechanism, we mainly take into account the
time-varying feature of the traffic matrix. Hence, to present
1551-3203 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: University of Exeter. Downloaded on June 26,2020 at 20:54:25 UTC from IEEE Xplore. Restrictions apply.
3
our network traffic prediction mechanism legibly and simply,
we define an OD flow as a row vector Xt and its elements as
Xt . Here, we use time slot as a unit, which means that X (n, t)
is not the volume of network traffic at a point in time; rather,
it declares the average value of network traffic during a time
interval, typically 10 or 15 min [33].
Network traffic prediction can be formulated as the problem
of calculating the traffic matrix X according to a prior of the
traffic matrix. If we denote a traffic element of any OD flow
by Xt , then the problem of network traffic prediction can be
regarded as calculating Xt from the previous network traffic
data (Xt−1, Xt−2, ..., Xt−W ). Generally, the statistical model-
based network traffic prediction mechanism is to deal with
the following conditional probability:
p (Xt |Xt−1, Xt−2, ..., Xt−W ) , (1)
where W is the length of the prior network traffic. The
statistical model-based network traffic prediction mechanism
often assumes that Eq. (1) obeys certain distributions, such as
the multifractal wavelet model and generalized autoregressive
conditional heteroskedasticity model. Indeed, these mecha-
nisms pursue a traffic flow via the prior of network traffic
in the sense that a function is fitted, which can be denoted by:
Xt = f (Xt−1, Xt−2, ..., Xt−W ) . (2)
Unfortunately, this function represents the time-varying feature
of traffic flows, and it changes depending on the services
supported by our network. Therefore, we explore RL to deal
with the problem of network traffic prediction defined by Eq.
(2).
IV. R EINFORCEMENT L EARNING - BASED N ETWORK
T RAFFIC P REDICTION
In this section, we introduce the designed network traffic
prediction mechanism based on RL and dictionary learning.
This mechanism contains two parts. First, we propose a
network traffic prediction approach that combines the Monte-
Carlo learning and the Q-learning. In our mechanism, we
predict each OD flow independently. Thereby, we mainly
present the predicting process of an OD flow as an example
for convenience. In view of the computation complexity of
the proposed mechanism, we further propose a modified
approach based on dictionary learning to improve its real-time
performance.
A. Monte-Carlo Q-learning-based Network Traffic Prediction
RL [34] is one of the most prevalent machine learning
methods. It calculates a policy by an agent implementing
iterative actions to research the best policy with the maximum
reward. In detail, RL takes advantage of a sample under the
environment to guide the next action. Meanwhile, the relative
reward can be achieved during this process, and then the
environment can be updated according to the reward. The RL
can be defined as an MDP denoted by hS, A, P, R, γi, where
S is the set of states and A is the set of actions. R is the
immediate reward from the current state to the next according
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TII.2020.3004232, IEEE
Prior
a∗
X t−1
(X t −1
,a ∗
Monte-Carlo Q-Learning
(X ′T −W
,aT′ −W ,rT′−W +1 , X T′ −W +1 ..., X T′ −1, aT′ −1, rT′ , X T′ )
Training Dataset
Fig. 1. Network traffic prediction based on RL.
to the given policy with environment. P is the transition
probability matrix, and γ is the discount factor [35].
Q-learning has been employed and widely developed in RL.
For an agent with state s (s ∈ S), when it moves to the
next state s̃ (s̃ ∈ S) after implementing an action a (a ∈ A)
according to policy π (s, a), an immediate reward R (s, a) can
be achieved. The reward with respect to the current state s by
taking action a, denoted by Q (s, a), is a weight sum via the
immediate reward R (s, a) of moving to the next state s̃, which
can be calculated by:
 
Q (s, a) = R (s, a) + γ max Q (s̃, ã) , (3)
where ã ∈ Ã is the action set under the next state s̃. In fact, Q-
learning can be regarded as an optimization problem that finds
an optimal policy to maximize the reward Q (s, a). During the
iterative process of finding the maximum reward, the reward
is updated by a weighted sum:
  
Q (s, a) ← (1 − α) Q (s, a) + α R (s, a) + γ max Q (s̃, ã) ,
(4)
where α ∈ [0, 1]. During each iteration, the sampling of
state can be chosen via exploration-only and exploitation-only
approaches. The former is suitable for sampling with large
range and selects a sampling of state according to the uniform
distribution. By contrast, the latter chooses the sampling with
the current maximum cumulative reward.
We first assume that Xt ∈ [Xmin, Xmax ], where Xmin and
Xmax are the minimum and maximum values of traffic flow,
respectively. Xmin and Xmax are known to us because they can
be determined from the prior of network traffic. This assump-
tion is established due to the prediction of trends of traffic
flows. Namely, predicting an OD flow can be transformed
to predicting its variation profile instead of predicting each
numerical value exactly. This is because the network security
and management functions using predicted network traffic
always have fault-tolerance for prediction error. In practice,
the variation trends of OD flows are suitable enough, and do
not have a negative impact on deploying any network security
and management functions.
1551-3203 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: University of Exeter. Downloaded on June 26,2020 at 20:54:25 UTC from IEEE Xplore. Restrictions apply.
Transactions on Industrial Informatics
4
Predictor For network traffic prediction, we define the sampling of OD
flows with two elements as (Xt−1, Xt ), where Xt−1 and Xt are
Xt the prior of network traffic and relative predictor, respectively.
) The problem of network traffic prediction is computing Xt
when Xt−1 is known. In this case, we define the problem of
network traffic prediction as the following optimization model:
min K L (p (Xt |Xt−1 ) kp0 (X 0 t |X 0 t−1 ) )
s.t. Xt ∈ [Xmin, Xmax ] ,
X 0 t ∈ [X 0 min, X 0 max ] , (5)
0 i.i.d
p, p ∼ PX .
In Eq. (5), p and p0, which are independent and identically
distributed, denote the distributions of the predicted and prior
network traffic data sets, respectively. Note that the constraint
i.i.d
p, p0 ∼ PX mainly declares that p and p0 are independent
and identically distributed. The distribution PX is unknown
to us. Meanwhile, because the problem of network traffic
prediction can be viewed as calculating a conditional prob-
ability, we use the KL divergence denoted by K L (·) as the
metric between two distributions [17]. Then, the problem of
network traffic prediction is to search the optimal probability
distribution p∗ (Xt |Xt−1 ) with the minimized KL divergence.
Furthermore, this optimization model is convex, and the trace
of the objective function is a parabola with respect to p. To
solve this convex optimization model, we propose our RL-
based mechanism. The framework of the proposed mechanism
is illustrated in Fig. 1. Calculating the predictor Xt from the
known network traffic Xt−1 is regarded as an action carried out
according to a specified policy π (Xt−1, a). Consequently, the
ã ∈ Ã
next issue is how to determine an optimal policy π (Xt−1, a∗ )
with the maximum reward and implement the action a∗ to
achieve the predictor Xt . To deal with this problem, the
Monte-Carlo Q-learning algorithm based on KL divergence
is proposed to calculate the optimal policy π (Xt−1, a∗ ).
The KL divergence-based Monte-Carlo Q-learning algo-
rithm is employed to determine the optimal policy through
a training data set denoted by X 0. Each element of X 0
ã ∈ A
is the network traffic with W + 1 time slots denoted by
(X 0T −W , X 0T −W +1 ..., X 0T −1, X 0T ). We consider that the net-
0
work traffic moves from State X j−1 to State X j0 by imple-
0
menting Action a j−1 according to a policy and receives an
immediate reward r j0. Then, we are able to generate a sequence:
(X 0T −W , a 0T −W , r 0T −W +1, X 0T −W +1 ..., X 0T −1, a 0T −1, r 0T , X 0T ) .
(6)
The generated sequence depends on an initialized reward R. In
our mechanism, the reward R is a (X 0 max − X 0 min + 1)-order
square matrix whose rows and columns are indexed by the
values of traffic flows, i.e.,
 r 0 X 0 mi n,X 0 mi n r 0 X 0 mi n,X 0 mi n +1 · · · r 0 X 0 mi n,X 0 ma x

 0
r0 · · · r 0 X 0 mi n +1,X 0 ma x

 r X 0 mi n +1,X 0 mi n
X 0 mi n +1,X 0 mi n +1 
.
.. .. ..
 
. . .


 ··· 

 r 0 X 0 ,X 0 r 0 X 0 ma x ,X 0 mi n +1 · · · r 0 X 0 ma x ,X 0 ma x 
 ma x mi n
(7)
The element rs01,s2 (s1, s2 ∈ [X 0 min, X 0 max ]) represents the
immediate reward when the agent moves from State s1 to
State s2 by implementing action as01,s2 . Eq. (6) declares the
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TII.2020.3004232, IEEE
Transactions on Industrial Informatics
5

state and reward sequence of MDP with the state space Algorithm 1 Network Traffic Prediction Based on Monte-
Xs0 = [X 0 min, X 0 max ]. The reward of this sequence is defined Carlo Q-learning
as the mean of all immediate rewards: Require: training data set X 0 with P time slots; state space
T Xs0 = [X 0 min, X 0 max ]; number of iterations Z.
1 Õ
Ensure: π (X 0, a 0)
Rc = r 0 j, (8)
W j=T −W +1 1: Q ← [](X 0 ma x −X 0 mi n +1)×(X 0 ma x −X 0 mi n +1)
2: rs01,s2 ← 1, s1, s2 ∈ Xs0
where r 0 j denotes the reward when the agent moves from State
0 to State X 0 . After that, we can update the policy π (X 0, a 0 ) 3: for s1 ∈ X 0 do
X j−1 j 4: rs01,s2 ← rs01,s2 + 1
using Algorithm 1. At this time, we can predict Xt according to
5: end for 0
the optimal policy π (Xt−1, a∗ ). In this algorithm, the reward r s ,s
6: rs01,s2 ← |X 01|−12
R is initialized according to the training data set, and each
7: for z = 1, 2, ..., Z do
element rs01,s2 records the ratio of actions from State s1 to
8: Generate a series shown by Eq. (6) from training data
State s2 in the training data set. Namely, we also ensure that
set X 0
all the elements in the initialized reward R are non-zero, and
r0s , s 9: for i = T − W, T − W + 1, ..., T − 1 do
we set rs01,s2 = 1. After that, we set rs01,s2 = |X 01|−12 , where |X 0 | T
Rc = T1−i r0j
Í
is the number of elements in the training data set. To obtain 10:
j=i
the optimal policy, Q (X 0i, a 0i ) is updated by: 11: Update Q (X 0i, a 0i ) according to Eq. (9)
Q (X 0i, a 0i ) ← (1 − α) Q (X 0i, a 0i ) 12: end for
+α (R (X 0i, a 0i ) + γ (max Q (X 0i+1, a 0i+1 ))) 13: if ε < E then
+K L (p (Xi+1 |Xi ) kp0 (X 0i+1 |X 0i ) ). 14: π (X 0, a 0) ← arg max Q (X 0, ã 0)
ã0
(9) 15: else
Under this update rule, the convergence can be guarantee 16: π (X 0, a 0) ← π (X 0, ã 0) , ã 0 ∼ U (X 0 max, X 0 min )
to find optimal solution with probability 1. We denote the 17: end if
optimal policy by Q∗ (Xt−1, at−1 ), and then we define ∆Q = 18: end for
Q∗ (Xt−1, at−1 ) − Q (Xt−1, at−1 ). According to Eq. (9), we have
the following: Prior Predictor
∆Q = (1 − α) ∆Q+ a∗
X t−1 Xt
= α (R + γ (max Q (Xt , at ))
, (10)
−Q∗ (Xt−1, at−1 ) + α−1 K L(p kp0 ) (
π Xt −1, a ∗ )
where R = R (Xt−1, at−1 ) and K L (p∗ kp0 ) = 0. We define
F (Xt−1, at−1 ) as: Monte-Carlo Q-Learning
F (Xt−1, at−1 ) = R + γ (max Q (Xt , at ))
. (11)
−Q∗ (Xt−1, at−1 ) + α−1 K L (p kp0 )
From Theorem 1 in [36], the random iterative process ∆Q (α T −W
,aT′ −W ,rT′−W +1 , αT −W +1 ..., αT −1, aT′ −1, rT′ , αT )

converges to 0 with probability 1 under the following two

assumptions: (α T −W
, αT −W +1 ..., αT −1 , αT )

kE {F (Xt−1, at−1 )}kW ≤γk∆QkW , 

(
(12)
var {F (Xt−1, at−1 )} ≤ C 1 + k∆QkW
2
,
Adaptive Dictionary Learning
where γ < 1 and the constant C > 0. k·kW refers to some
weighted maximum norm [36]. Obviously, after adding the KL
(X ′ T −W
, XT′ −W +1 ..., XT′ −1, XT′ )
divergence shown by Eq. (9), these two assumptions can still
be established for the infinite norm because the KL divergence Training Data Set
is nonnegative.
Meanwhile, both the exploration-only and exploitation-only Fig. 2. Network traffic prediction based on RL and adaptive dictionary
approaches are used to update the policy. The variable ε, which learning.
obeys the uniform distribution ε ∼ U (0, 1), is employed to
determine the optimal tradeoff between these two approaches.
Q-learning algorithm is massive resulting in poor real-time
performance. In other words, if the volume of an OD flow
B. Dictionary Learning for Traffic Matrix is [X 0 min, X 0 max ], then the cumulative reward Q (X 0, a 0) is a
Nowadays, an IIoT supports many network applications and (X 0 max − X 0 min + 1)-order square matrix, which will occupy
services for industrial applications. Consequently, the volume a significant amount of memory for deploying the proposed
of network traffic has increased exponentially. Furthermore, network traffic prediction mechanism.
the state space of the KL divergence-based Monte-Carlo To address this issue, we propose a dictionary learning

1551-3203 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: University of Exeter. Downloaded on June 26,2020 at 20:54:25 UTC from IEEE Xplore. Restrictions apply.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TII.2020.3004232, IEEE
Transactions on Industrial Informatics
6

nl ← arg min η = φ nl 0 + φ nl 2 . During each iteration,

algorithm to project the network traffic to a transform domain,
where the coefficients are fewer and smaller than those of
the time domain. This mapping process can decrease the
implementation time of the proposed network traffic prediction
mechanism. The framework of the modified RL-based network
traffic prediction mechanism is illustrated in Fig. 2. In detail,
the modified mechanism first conducts a dictionary learning
algorithm over the traffic matrix to acquire the coefficient
matrix. Each row of the coefficient matrix represents the trace
of an OD flow in the transform domain. After that, we use
the KL divergence-based Monte-Carlo Q-learning algorithm
to predict these coefficients instead of the network traffic in
time domain.
At this moment, the problem of dictionary learning is to
find an orthogonal matrix D such that the following equation
holds:
X 0 = D α, (13)
where α is the coefficient with respect to dictionary D. Re-
searchers have proposed many methods to create a dictionary
based on given data. These methods consist of the dictionary
selection and dictionary learning. The typical dictionary selec-
tion contains wavelet basis and Fourier basis. The dictionary
do
φ nl
0 +
φ nl
2
learning constructs a dictionary by an alternating optimization
strategy. Existing dictionary learning strategies always look for
φ nl l
a dictionary to make the data sparse, but the size of coefficient
is usually explicitly neglected. Both signal representation and
image processing generally focus on the sparsity of the coeffi-
cient after decomposition. Different from the computationally
expensive algorithms for sparse representation, we investigate
a sparse decomposition strategy that takes into account the
sparse representation and size of coefficients jointly.
In our dictionary learning algorithm, we define a minimum
index to match the sparse representation and minimize coeffi-
cients.
Definition 1: The minimum index is defined as:
η = k c n k 0 + k c n k 2,
where k·k 0 and k·k 2 are the `0 - and `2 -norms, respectively.
The vector c n is the nth column of the training data set X 0,
and n = 1, 2, ..., N 2 .
The minimum index measures two features of coefficients.
The `0 -norm in Eq. (14) derives the sparsity of the coefficients,
and the `2 -norm is used to keep the coefficients as small as
possible. Note that we build a novel prior of the traffic matrix,
denoted by X 00, collected from X 0 to construct the orthogonal
matrix D. This novel X 00 consists of N 2 time slots, that is the
matrix X 00 is N 2 × N 2 .
φ nl
0 +
φ nl
2 .
Our adaptive dictionary learning algorithm gains an or-
thogonal matrix by multiple h iterations. During each iteration
i Hence, by several greedy iterations, the proposed adaptive
l, a residual term Φl = φ1l , φ2l , ..., φ lN 2 is defined in our
2
algorithm, where φ nl ∈ R N is the nth column of the residual
term. The dictionary learning algorithm is shown in Algorithm
2. The notation nl denotes the nth column of the residual term
at the lth iteration. In the dictionary learning algorithm, the
residual term is initialized as the prior of the traffic matrix
X 00. The notation N l , which is initialized as a null set, records
the indexes of the columns of the residual term selected by

n<N l
the column of the residual term with the minimum index is
determined and normalized to gain an orthogonal basis. From
Eq. (15), we observe that the proposed dictionary learning-
based solution in fact calculates an orthogonal basis by way
of the Schmidt orthogonalization.
φ nl+1 ← φ nl − d l < d l , φ nl > . (15)
Hence, the computational complexity is much lower than
that of other state-of-the-art methods, such as the Principal
Component Analysis (PCA) [32] and K-SVD. Besides, during
each iteration, one column of the training data is selected
with minimum index independently. Thereby, we can obtain
the optimal convergence by way of this greedy algorithm.
Meanwhile, according to the definition of the minimum index
and the greedy adaptive dictionary learning algorithm, we have
the following proposition.
Algorithm 2 Residual-based Dictionary Learning
Require: Φ0 = X 00; D0 = []; l = 0; N 0 = ∅.
Ensure: Dl+1
1: for l = 0, 1, 2..., N2 − 1
nl ← arg min η =
2:
l
.n<N
d l ← φ nl l
3:
2
4: Dl+1 ← [Dl, d l ]
5: N l+1 ← N l ∪ {nl }
6: for n = 1, 2..., N 2 do
7: φ nl+1 ← φ nl − d l < d l , φ nl >
8: end for
9: end for
Proposition 1: The minimum index can make the coeffi-
cients small and sparse, which will decrease the state space in
the Monte-Carlo Q-learning algorithm.
Proof: The function of the minimum index can be achieved
(14)
by means of the definitions of `0 -norm, `2 -norm and the
greedy dictionary learning algorithm based on the Schmidt
orthogonalization. The `0 -norm is defined as the number of
non-zero elements in a vector, and the `2 -norm reflects its
energy. In this case, when we define the residual term as the
prior of the traffic matrix, the equation φ nl − d l < d l , φ nl >
will implement the Schmidt orthogonalization in the sense that
all remaining residual terms are orthogonal to d l . At the same
time, during the next iteration, the optimal orthogonal vector
with the minimum `0 -norm and `2 -norm can be obtained by:
nl ← arg min η =

(16)
n<N l
dictionary learning algorithm is able to find the optimal
dictionary in the sense that the coefficients obey the optimal
sum of the `0 -norm and `2 -norm. That is, the coefficients
are sufficiently sparse and small. From Algorithm 1, we see
that the computation complexity is O (Z). The size of Z
belongs to the state space [X 0 min, X 0 max ]. Namely, because the
network traffic is nonnegative integers, the number of states is
(X 0 max − X 0 min + 1). In fact, the network traffic can be equal

1551-3203 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: University of Exeter. Downloaded on June 26,2020 at 20:54:25 UTC from IEEE Xplore. Restrictions apply.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TII.2020.3004232, IEEE
Transactions on Industrial Informatics
to 0 occasionally. Then, there are Xmax 0 states in the Monte-
Carlo Q-learning. We must set Z  Xmax 0 for the algorithm to
always be valid. In addition, the cumulative reward Q (X 0, a 0)
is a (X 0 max − X 0 min + 1)×(X 0 max − X 0 min + 1) matrix. Above
all, it is significantly difficult to use the Monte-Carlo Q-
0
learning algorithm without making Xmax sufficiently small.
By contrast, after projecting the original network traffic into a
transform domain, the real-time performance can be improved
obviously.
V. N UMERICAL R ESULTS
A. Network Traffic Data Set
To evaluate the proposed RL-based network traffic pre-
diction mechanism, we built a testbed including 12 nodes.
The constructed wireless network supports various services
consisting of video, voice, etc. The topology of the testbed is
generated by the Open Shortest Path First (OSPF) algorithm,
and then the weights of the OSPF are defined as the general
urban path loss model, as shown in [37]. We sample the end-
to-end network traffic data with 10100 time slots. Namely,
the traffic matrix is 144 × 10100. We employ the first 10000
time slots as the training data set for the KL divergence-
based Monte-Carlo Q-learning algorithm. In the training data
set with 10000 time slots, the first 144 time slots are used
to implement the proposed dictionary learning algorithm.
Besides, the GÉANT backbone network traffic was employed
in our simulations. It is made up of 23 nodes and 120 links,
and records 3360 samples of network traffic. The first 3000
time slots are used as the training data set, and the first
800 samplings of the training data set are for the proposed
residual-based adaptive dictionary learning algorithm. We train
the proposed framework shown in Algorithm 1 in batch, and
the size of batch is 10. Moreover, we set Z = 10, E = 0.4,
α = 0.5, and γ = 0.01 in Algorithm 1. These parameters are
set empirically to obtain the lowest prediction error. We used
MATLAB R2018a to conduct all evaluations. The evaluations
were conducted on a 64-bit Windows 7 machine running with
an Intel Xeon W-2102 (2.9 GHz) and 32 GB RAM.
In traditional large-scale backbone networks, the PCA
method and the Sparsity Regularized Matrix Factorization
(SRMF) [38] method are state-of-the-art techniques for end-to-
end network traffic prediction. The former takes into account
the power laws of a traffic matrix, and predicts the network
traffic in the principal component domain. The latter utilizes
the spatio-temporal features of a traffic matrix, that is, the
neighboring elements are close to each other in value. It
can implement both network traffic estimation and prediction
by specifying a certain configuration. Hence, in this section,
we compare our network traffic prediction mechanism with
PCA and SRMF. Moreover, we also take into account the
LSTM method mentioned in [26] to evaluate the proposed
mechanism.
B. Prediction Error Analysis
To subsequently assess the overall performance of the
proposed prediction mechanism, we define the Spatial Relative
1551-3203 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: University of Exeter. Downloaded on June 26,2020 at 20:54:25 UTC from IEEE Xplore. Restrictions apply.
7
Error (SRE) and Temporal Relative Error (TRE) as:
 SRE (n) = k kX(n,t) k k 2 ,
 X̂(n,t)−X(n,t)


2 (17)
 T RE (t) = k kX(n,t) k k 2 ,
 X̂(n,t)−X(n,t)
2

where X̂ (n, t) is the predicted network traffic with respect to
X (n, t). Figs. 3(a) and 3(b) respectively show the SREs and
TREs of four network traffic prediction mechanisms for the
data set via our testbed, respectively. In Fig. 3(a), the x- and
y-axes represent the ordered flows from smallest to largest
according to their mean and the SREs, respectively. In our
testbed, the volumes of several OD flows are zero, and we
delete these OD flows so that k X (n, t)k 2 , 0. The SREs of the
four mechanisms decrease as the size of the flow increases.
The RL mechanism has high SREs for OD 5, 11, and 15.
By contrast, SRMF and LSTM have larger SREs for partial
OD flows compared to the other two approaches. The means
of SREs for RL, PCA, SRMF, and LSTM are 3.20, 8.74,
8.16, and 9.31, respectively. Fig. 3(b) shows the TREs of RL,
PCA, SRMF, and LSTM. Obviously, RL has the lowest TRE
among the four approaches, and PCA has the highest TRE.
The means of TREs for RL, PCA, SRMF, and LSTM are 0.55,
1.28, 1.09, and 1.20, respectively. We observe that the TREs
of RL show some fluctuations in size. Namely, some TREs
are close to zero, while others are much higher. The profile
of the flow in our testbed yields much more fluctuations,
meaning it requires many more iterations to update the optimal
policy. In other words, if we make Z larger, the TREs will be
smoother. However, the means of the TREs do not decrease as
Z increases. As a result, we mainly consider the computational
complexity rather than the fluctuations of TREs. Figs. 3(c)
and 3(d) represent the evaluations in the GÉANT data set. We
observe that the SRE of SRMF is enormous for several OD
flows, and the SREs of RL, PCA, and LSTM are relatively low
in comparison. Some traffic flows in the GÉANT backbone
network have sharp fluctuations. The SRMF method assumes
that the values of two adjacent elements are similar, which can
lead to large SREs for predicting sharp fluctuations. Thereby,
from Fig 3(c), we obverse TREs of SRMF close to order 105 .
The LSTM method achieves the best TRE for the GÉANT
data set, as shown by Fig. 3(d). LSTM has an outstanding
capability of learning long-range dependence, which is suitable
for predict long-term traffic, such as in the GÉANT backbone
network. The means of TREs for RL, PCA, SRMF, and LSTM
are 0.58, 0.86, 0.78, and 0.34, respectively.
To observe the SRE and TRE intuitively, we present the
Cumulative Distribution Functions (CDF) of SREs and TREs
in Fig. 4. In Fig. 4(a), for RL, PCA, SRMF and LSTM in
our testbed data set, the SREs are less than 2.3, 19.0, 16.0,
and 1.28, respectively, with respect to 80% of all the OD
flows. In Fig. 4(b), the TREs are less than 0.7, 1.4, 1.3, and
1.4, respectively, for 80% of the time slots. From Figs. 4(c)
and 4(d), it can bee seen that SRMF is better in terms of TRE
compared to PCA, but worse in terms of SRE. For LSTM and
RL, they mainly focus on the temporal features of network
traffic. Hence, their improvements are large in TRE but small
in SRE. Comparing with LSTM, RL prefers to predict short-
term traffic over long-term traffic. Hence, the TRE of LSTM
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TII.2020.3004232, IEEE
Transactions on Industrial Informatics
8

300
RL 200 RL
100 0.8 PCA
0
0 20 40 60 80 100 120 SRMF
300 0.6 LSTM

CDF
PCA

200
100
0.4
0
0 20 40 60 80 100 120
0.2
LSTM SRMF

300
200
100
0
0
0 20 40 60 80 100 120 0 50 100 150 200 250
300 SRE
200
100
0
(a) CDF of SRE in testbed.
0 20 40 60 80 100 120

(a) SREs in testbed. 1

RL
PCA
2
SRMF
RL

1 LSTM

CDF
0 0.5
10 20 30 40 50 60 70 80 90 100
2
PCA

1
0
10 20 30 40 50 60 70 80 90 100 0
2
LSTM SRMF

0 0.5 1 1.5 2
1 TRE
0
10 20 30 40 50 60 70 80 90 100 (b) CDF of TRE in testbed.
2
1
1
0
10 20 30 40 50 60 70 80 90 100 RL
PCA
(b) TREs in testbed. SRMF

CDF
LSTM
0.5
104
10
RL

5
0
0 50 100 150 200 250 300 350 400 450 500

10
104 0
PCA

0 0.5 1 1.5 2 2.5 3

5
0
SRE 105
0 50 100 150 200 250 300 350 400 450 500
4
10 (c) CDF of SRE in GÉANT.
LSTM SRMF

10
5
0
0 50 100 150 200 250 300 350 400 450 500
104 RL
10
0.8 PCA
5
0
SRMF
0.6
CDF

0 50 100 150 200 250 300 350 400 450 500 LSTM

(c) SREs in GÉANT. 0.4

0.2
RL

1 0
0
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8
50 100 150 200 250 300 350
TRE
PCA

1 (d) CDF of TRE in GÉANT.

0
50 100 150 200 250 300 350
Fig. 4. CDFs of SREs and TREs in our testbed and GÉANT.
LSTM SRMF

0
50 100 150 200 250 300 350

where error (n) = X̂ (n, t) − X (n, t). Fig. 5(a) shows the bias
1

0
50 100 150 200 250 300 350
of each OD flow for the testbed data set. The x-axis denotes
(d) TREs in GÉANT.
the identities of OD flows arranged in descending order with
Fig. 3. SREs and TREs in our testbed and GÉANT. respect to their averages. We observe that the bias of PCA is
much larger than that of the others when predicting for small
OD flows. The reason for this is that some components with
is lower than that of RL in the GÉANT data set but higher respect to small singular values are neglected in the sense that
for the testbed data set. just a small number of principal components with respect to
It is well known that biased estimators sometimes can have larger singular values are retained to approximate the traffic
lower variance, and then estimates are closer to the true value matrix. SRMF extracts the features between two neighboring
than those of unbiased estimators. Consequently, we evalu- elements in a traffic matrix, making its bias consistent. LSTM
ate the performance of RL aiming at short-range dependent shows an overestimation for the largest flow. For the RL-
and long-range dependences separately, respectively. For this based network traffic prediction mechanism, it mainly takes
purpose, the prediction bias and relative sample Standard advantage of the temporal features of the traffic matrix. In
Deviation (SD) [32] are involved in our experiments, which each iteration, the policy is updated according to a generated
are defined as follows: sequence from the prior of an OD flow. Hence, the RL-based
T   mechanism is independently over-estimate and under-estimate
bias(n) = T1 X̂ (n, t) − X (n, t) ,

for diverse OD flows. In addition, RL has the lowest bias

 Í



 s t=1 among these four approaches. Fig. 5(b) displays the bias for
T
(18)
 SD(n) = each flow against its sample standard deviation (or variance)
1 Í
(error(n) − bias(n))2,

T −1


 t=1 for the testbed data set. The variance of LSTM is lowest among


1551-3203 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: University of Exeter. Downloaded on June 26,2020 at 20:54:25 UTC from IEEE Xplore. Restrictions apply.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TII.2020.3004232, IEEE
Transactions on Industrial Informatics
9

1000 RL and PCA methods’ error variances are relatively high,

RL
PCA while the SRMF and LSTM methods maintain relatively
500 SRMF
LSTM
low variances. Obviously, PCA shows entirely different error
Bias

variances for two data sets. Although the profiles of traffic

0
flows in GÉANT are smoother, there are still several enormous
-500
low-high transitions, which cause higher variance.
0 50 100 150 Finally, we evaluate the performance of the four approaches
Flow ID for two data sets by referring to the performance improvement
(a) Bias of predicted results in testbed. ratio, which is defined as:
1000
RL
Í2 Í
N T   N2 T  
 X̂a (n, t) − X (n, t) − Í Í  X̂b (n, t) − X (n, t)
PCA
n=1 t=1 n=1 t=1
500 SRMF R= ,
LSTM Í2 Í
N T
Bias

 
 X̂a (n, t) − X (n, t)
0
n=1 t=1
(19)
-500 where X̂a (n, t) and X̂b (n, t) are the respective predictors of
0 1 2 3 4 5 6 7 8 9 10 two algorithms. The performance ratios of RL versus PCA,
SD in Error 105
SRMF, and LSTM in our testbed data set are 73.40%, 58.59%,
(b) Bias versus SD in testbed.
and 63.18%. For the GÉANT data set, the performance ratios
105
of RL versus PCA and SRMF are 15.90% and 51.81%,
5 RL respectively. The RL method has no performance improvement
PCA to LSTM. This is because RL and LSTM are designed for
SRMF
LSTM short-term and long-term traffic, respectively. As a result,
Bias

0
RL does not perform well compared with LSTM for traffic
prediction of any backbone network.
-5
0 100 200 300 400 500 600
Flow ID
VI. C ONCLUSIONS AND F UTURE W ORKS
(c) Bias of predicted results in GÉANT. This paper investigates the problem of end-to-end network
traffic prediction in IIoTs for industrial applications. Although
105
5
machine learning-based methods are widely used to capture
RL
PCA
the statistical features of traffic flows for network traffic
SRMF prediction, there are some issues when they are applied to
Bias

0 IIoTs directly. These issues mainly consist of high computa-
-5
0 0.5 1 1.5 2 2.5
SD in Error
(d) Bias versus SD in GÉANT.
Fig. 5. Biases and SD in our testbed and GÉANT.
the four approaches. RL and SRMF have few larger sample
standard deviations. Therefore, they tend to capture individual
flow elements. In theory, as mentioned before, both RL and
SRMF predict network traffic via adjacent elements. On the
contrary, PCA and LSTM obtain a predictor from a series of
previous flow elements. We can obtain similar conclusions via
the evaluation of the GÉANT backbone network, as shown
in Figs. 5(c) and 5(d). The volume of network traffic in
GÉANT is much larger than that in our testbed. Consequently,
compared with the testbed data set, the absolute biases of
all four algorithms are much larger in GÉANT. The SRMF
method shows an overestimation for the largest OD flows,
with the bias increasing with flow size. The PCA method is
quite different from the other three methods; underestimation
appears for small flows. From Fig. 5(d), we observe that the
LSTM
tional complexity and large consumption caused by the direct
measurement of large scale training data set. Motivated by
these observations, we proposed an RL-based network traffic
3 3.5 4 4.5
1010
prediction mechanism with lower computational complexity
and a smaller scale training data set. Aiming at a small
training data set, we took advantage of RL to predict network
traffic, and proposed a KL divergence-based Monte-Carlo Q-
learning algorithm. Furthermore, to reduce the computational
complexity of the proposed algorithm, a dictionary learn-
ing mechanism was also proposed. The proposed RL-based
network traffic prediction mechanism was evaluated on real
network traffic from our testbed and the GÉANT network.
According to the evaluation, the proposed mechanism can
predict short-term traffic effectively. The RL-based network
traffic prediction mechanism can improve prediction accuracy
efficiently. However, some works are still necessary to improve
the performance of the proposed algorithm adequately, which
will be our main work in the future.
For our future work, the proposed mechanism can first be
expanded for long-term traffic prediction by designing a novel
RL algorithm. Besides, the spatial and spatio-temporal features
can be considered as additional constraints to improve the
prediction error. For instance, we can define a novel immediate
reward by a deep learning architecture, e.g., deep neural

1551-3203 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: University of Exeter. Downloaded on June 26,2020 at 20:54:25 UTC from IEEE Xplore. Restrictions apply.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TII.2020.3004232, IEEE
Transactions on Industrial Informatics
network and CNN for spatial and spatio-temporal feature
extraction, respectively.
VII. ACKNOWLEDGEMENTS
This work was supported in part by the National Key R&D
Program of China under Grant 2018YFE0206800, in part by
the National Natural Science Foundation of China under Grant
61701406 and Grant 61971084, in part by the National Natural
Science Foundation of Chongqing under Grant cstc2019jcyj-
msxmX0208, in part by the open research fund of National
Mobile Communications Research Laboratory, Southeast Uni-
versity, under Grant 2020D05, and in part by the Applied Basic
Research Programs of Qingdao City under Grant 18-2-2-36-
jch.
R EFERENCES
[1] Z. Ning, X. Wang, J. J. P. C. Rodrigues, and F. Xia, “Joint computation
offloading, power allocation, and channel assignment for 5G-enabled
traffic management systems,” IEEE Transactions on Industrial Infor-
matics, vol. 15, no. 5, pp. 3058–3067, May 2019.
[2] Q. He, X. Wang, Z. Lei, M. Huang, Y. Cai, and L. Ma, “TIFIM: A two-
stage iterative framework for influence maximization in social networks,”
Applied Mathematics and Computation, vol. 354, pp. 338 – 352, 2019.
[3] D. Zhang, L. Tan, J. Ren, M. K. Awad, S. Zhang, Y. Zhang, and
P. Wan, “Near-optimal and truthful online auction for computation
offloading in green edge-computing systems,” IEEE Transactions on
Mobile Computing, vol. 19, no. 4, pp. 880–893, 2020.
[4] H. Yang, X. Xie, and M. Kadoch, “Intelligent resource management
based on reinforcement learning for ultra-reliable and low-latency IoV
communication networks,” IEEE Transactions on Vehicular Technology,
vol. 68, no. 5, pp. 4157–4169, May 2019.
[5] X. Wang, Z. Ning, and L. Wang, “Offloading in Internet of Vehicles: A
fog-enabled real-time traffic management system,” IEEE Transactions
on Industrial Informatics, vol. 14, no. 10, pp. 4568–4578, Oct 2018.
[6] T. Otoshi, Y. Ohsita, M. Murata, Y. Takahashi, K. Ishibashi, K. Sh-
iomoto, and T. Hashimoto, “Hierarchical model predictive traffic engi-
neering,” IEEE/ACM Transactions on Networking, vol. 26, no. 4, pp.
1754–1767, Aug 2018.
[7] H. Yang, A. Alphones, W. Zhong, C. Chen, and X. Xie, “Learning-based
energy-efficient resource management by heterogeneous RF/VLC for
ultra-reliable low-latency industrial IoT networks,” IEEE Transactions
on Industrial Informatics, pp. 1–1, 2019.
[8] Y. Zhang and H. Chiang, “Enhanced ELITE-load: A novel CMPSOATT
methodology constructing short-term load forecasting model for indus-
trial applications,” IEEE Transactions on Industrial Informatics, vol. 16,
no. 4, pp. 2325–2334, April 2020.
[9] J. Ren, D. Zhang, S. He, Y. Zhang, and T. Li, “A survey on end-
edge-cloud orchestrated network computing paradigms: Transparent
computing, mobile edge computing, fog computing, and cloudlet,” ACM
Computing Surveys, vol. 52, no. 6, pp. 125:1–125:36, 2020.
[10] Z. Ning, X. Hu, Z. Chen, M. Zhou, B. Hu, J. Cheng, and M. S. Obaidat,
“A cooperative quality-aware service access system for Social Internet
of Vehicles,” IEEE Internet of Things Journal, vol. 5, no. 4, pp. 2506–
2517, Aug 2018.
[11] Z. Ning, P. Dong, X. Wang, M. S. Obaidat, X. Hu, L. Guo, Y. Guo,
J. Huang, B. Hu, and Y. Li, “When deep reinforcement learning meets
5G-enabled vehicular networks: A distributed offloading framework for
traffic big data,” IEEE Transactions on Industrial Informatics, vol. 16,
no. 2, pp. 1352–1361, 2020.
[12] X. Wang, Z. Zhou, F. Xiao, K. Xing, Z. Yang, Y. Liu, and C. Peng,
“Spatio-temporal analysis and prediction of cellular traffic in metropo-
lis,” IEEE Transactions on Mobile Computing, vol. 18, no. 9, pp. 2190–
2202, Sep. 2019.
[13] C. Backfrieder, G. Ostermayer, and C. F. Mecklenbräuker, “Increased
traffic flow through node-based bottleneck prediction and V2X com-
munication,” IEEE Transactions on Intelligent Transportation Systems,
vol. 18, no. 2, pp. 349–363, Feb 2017.
[14] D. Chen, “Research on traffic flow prediction in the big data environment
based on the improved RBF neural network,” IEEE Transactions on
Industrial Informatics, vol. 13, no. 4, pp. 2000–2008, Aug 2017.
1551-3203 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: University of Exeter. Downloaded on June 26,2020 at 20:54:25 UTC from IEEE Xplore. Restrictions apply.
10
[15] Z. Ning, P. Dong, X. Wang, X. Hu, L. Guo, B. Hu, Y. Guo, T. Qiu,
and R. Y. K. Kwok, “Mobile edge computing enabled 5G health mon-
itoring for Internet of Medical Things: A decentralized game theoretic
approach,” IEEE Journal on Selected Areas in Communications, 2020.
[16] J. Zhang, X. Hu, Z. Ning, E. Ngai, L. Zhou, J. Wei, J. Cheng, and
B. Hu, “Energy-latency tradeoff for energy-aware offloading in mobile
edge computing networks,” IEEE Internet of Things Journal, vol. 5,
no. 4, pp. 2633–2645, Aug 2018.
[17] P. Yang and B. Chen, “Robust Kullback-Leibler divergence and universal
hypothesis testing for continuous distributions,” IEEE Transactions on
Information Theory, vol. 65, no. 4, pp. 2360–2373, April 2019.
[18] Y. Xu, F. Yin, W. Xu, J. Lin, and S. Cui, “Wireless traffic prediction with
scalable gaussian process: Framework, algorithms, and verification,”
IEEE Journal on Selected Areas in Communications, vol. 37, no. 6,
pp. 1291–1306, June 2019.
[19] M. Akintunde, P. Kgosi, and D. Shangodoyin, “Evaluation of GARCH
model adequacy in forecasting non-linear economic time series data,”
Journal of Computations and Modelling, vol. 3, no. 2, pp. 1–20, 2013.
[20] C. Xiang, P. Qu, and X. Qu, “Network traffic prediction based on MK-
SVR,” Journal of Information and Computational Science, vol. 12, no. 8,
pp. 3185–3197, 2015.
[21] M. Joshi and T. Hadi, “A review of network traffic analysis and
prediction techniques,” Computer Science, 2015.
[22] Y. Liu, B. Li, X. Sun, and Z. Zhou, “A fusion model of SWT, QGA
and BP neural network for wireless network traffic prediction,” in 2013
15th IEEE International Conference on Communication Technology,
Nov 2013, pp. 769–774.
[23] H. Liu, L. T. Yang, J. Chen, M. Ye, J. Ding, and L. Kuang, “Multivariate
multi-order markov multi-modal prediction with its applications in net-
work traffic management,” IEEE Transactions on Network and Service
Management, vol. 16, no. 3, pp. 828–841, Sep. 2019.
[24] R. Alvizu, S. Troia, G. Maier, and A. Pattavina, “Matheuristic with
machine-learning-based prediction for software-defined mobile metro-
core networks,” IEEE/OSA Journal of Optical Communications and
Networking, vol. 9, no. 9, pp. D19–D30, Sep. 2017.
[25] C. Qiu, Y. Zhang, Z. Feng, P. Zhang, and S. Cui, “Spatio-temporal
wireless traffic prediction with recurrent neural network,” IEEE Wireless
Communications Letters, vol. 7, no. 4, pp. 554–557, Aug 2018.
[26] S. K. Singh and A. Jukan, “Machine-learning-based prediction for
resource (re)allocation in optical data center networks,” IEEE/OSA
Journal of Optical Communications and Networking, vol. 10, no. 10,
pp. D12–D28, Oct 2018.
[27] J. Zhao, H. Qu, J. Zhao, and D. Jiang, “Towards traffic matrix prediction
with LSTM recurrent neural networks,” Electronics Letters, vol. 54,
no. 9, pp. 566–568, 2018.
[28] C. Zhang, H. Zhang, J. Qiao, D. Yuan, and M. Zhang, “Deep transfer
learning for intelligent cellular traffic prediction based on cross-domain
big data,” IEEE Journal on Selected Areas in Communications, vol. 37,
no. 6, pp. 1389–1401, June 2019.
[29] G. Choudhury, D. Lynch, G. Thakur, and S. Tse, “Two use cases of
machine learning for SDN-enabled IP/optical networks: traffic matrix
prediction and optical path performance prediction [invited],” IEEE/OSA
Journal of Optical Communications and Networking, vol. 10, no. 10, pp.
D52–D62, Oct 2018.
[30] F. Morales, M. Ruiz, L. Gifre, L. M. Contreras, V. Lopez, and L. Velasco,
“Virtual network topology adaptability based on data analytics for
traffic prediction,” IEEE/OSA Journal of Optical Communications and
Networking, vol. 9, no. 1, pp. A35–A45, Jan 2017.
[31] C. Zhang, H. Zhang, D. Yuan, and M. Zhang, “Citywide cellular traffic
prediction based on densely connected convolutional neural networks,”
IEEE Communications Letters, vol. 22, no. 8, pp. 1656–1659, Aug 2018.
[32] A. Soule, A. Lakhina, N. Taft, K. Papagiannaki, K. Salamatian, A. Nucci,
M. Crovella, and C. Diot, “Traffic matrices: balancing measurements,
inference and modeling,” in Proceedings of SIGMETRICS 2005, 2005,
pp. 362–373.
[33] F. Xu, Y. Lin, J. Huang, D. Wu, H. Shi, J. Song, and Y. Li, “Big
data driven mobile traffic understanding and forecasting: A time series
approach,” IEEE Transactions on Services Computing, vol. 9, no. 5, pp.
796–805, 2016.
[34] S. Gelly and D. Silver, “Monte-carlo tree search and rapid action value
estimation in computer go,” Artificial Intelligence, vol. 175, no. 11, pp.
1856 – 1875, 2011.
[35] C. B. Browne, E. Powley, D. Whitehouse, S. M. Lucas, P. I. Cowling,
P. Rohlfshagen, S. Tavener, D. Perez, S. Samothrakis, and S. Colton,
“A survey of monte carlo tree search methods,” IEEE Transactions on
Computational Intelligence and AI in Games, vol. 4, no. 1, pp. 1–43,
March 2012.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TII.2020.3004232, IEEE
Transactions on Industrial Informatics
[36] T. Jaakkola, M. I. Jordan, and S. P. Singh, “On the convergence of
stochastic iterative dynamic programming algorithms,” Neural Compu-
tation, vol. 6, no. 6, pp. 1185–1201, 1994.
[37] J. Andrusenko, R. L. Miller, J. A. Abrahamson, N. M. M. Emanuelli,
R. S. Pattay, and R. M. Shuford, “VHF general urban path loss model
for short range ground-to-ground communications,” IEEE Transactions
on Antennas and Propagation, vol. 56, no. 10, pp. 3302–3310, 2008.
[38] M. Roughan, Y. Zhang, W. Willinger, and L. Qiu, “Spatio-temporal
compressive sensing and Internet traffic matrices (extended version),”
IEEE Transactions on Networking, vol. 20, no. 3, pp. 662–676, 2012.
Laisen Nie received his Ph.D. degree in commu-
nication and information system from Northeastern
University, Shenyang, China, in 2016. Now He is
an Associate Professor in School of Electronics and
Information, Northwestern Polytechnical University,
Xi’an, China. His research interests include network
measurement, network security, and cognitive net-
works.
Zhaolong Ning (M’14-SM’18) received the M.S.
and Ph.D. degrees from Northeastern University,
Shenyang, China, in 2011 and 2014, respectively.
From 2013 to 2014, he was a Research Fellow
at Kyushu University, Japan. He is an Associate
Professor in Dalian University of Technology, China,
and a Distinguished Professor in Chongqing Univer-
sity of Posts and Telecommunications. His research
interests include Internet of Things, edge computing,
and vehicular networks. Dr. Ning has published over
120 scientific papers in international journals and
conferences. He serves as an associate editor or guest editor of several
journals, such as IEEE Transactions on Industrial Informatics, The Computer
Journal and so on.
Mohammad S. Obaidat (S’85-M’86-SM’91-F’05)
Dr. Mohammad S. Obaidat [F’05] received his Ph.D.
degree in Computer Engineering with a minor in
Computer Science from The Ohio State University,
Columbus, USA. He has published To Date (2019)
about 1,000 refereed technical articles-About half
of them are journal articles, over 70 books, and
about 70 Book Chapters. He is Editor-in-Chief of
3 scholarly journals and an editor of many other
international journals. He is the founding Editor-in
Chief of Wiley Security and Privacy Journal. He is
now the Founding Dean of the College in Computing and Informatics at
The University of Sharjah, UAE. Among his previous positions are Advisor
to the President of Philadelphia University for Research, Development and
Information Technology, President and Chair of Board of Directors of the
Society for Molding and Simulation International, SCS, Senior Vice President
of SCS, Dean of the College of Engineering at Prince Sultan University, Chair
and tenured Professor at the Department of Computer and Information Science
and Director of the MS Graduate Program in Data Analytics at Fordham
university, Chair and tenured Professor of the Department of Computer
Science and Director of the Graduate Program at Monmouth University is a
tenured Full Professor at King Abdullah II School of Information Technology,
University of Jordan, The PR of China Ministry of Education Distinguished
Overseas Professor at the University of Science and Technology Beijing,
China and an Honorary Distinguished Professor at the Amity University-
A Global University. He has chaired numerous (Over 160) international
conferences and has given numerous (Over 160) keynote speeches worldwide.
He founded or co-founded four international conferences. He has served
as ABET/CSAB evaluator and on IEEE CS Fellow Evaluation Committee.
He has served as IEEE CS Distinguished Speaker/Lecturer and an ACM
Distinguished Lecturer. Since 2004 has been serving as an SCS Distinguished
Lecturer. He received many best paper awards for his papers including
ones from IEEE ICC, IEEE Globecom, AICSA, CITS, SPECTS, DCNET
International conferences. He also received Best Paper awards from IEEE
Systems Journal in 2018 and in 2019 (2 Best Paper Awards). In 2020, he
received 4 best paper awards from IEEE Systems Journal. He also received
many other worldwide awards for his technical contributions. He also received
the SCS Outstanding Service Award. He was awarded the IEEE CITS Hall
of Fame Distinguished and Eminent Award. He is a Life Fellow of IEEE and
a Fellow of SCS.
1551-3203 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: University of Exeter. Downloaded on June 26,2020 at 20:54:25 UTC from IEEE Xplore. Restrictions apply.
11
Balqies Sadoun received her M.S. in Civil Engineer-
ing from the Ecole Nationale Des Travaux Publique
de LEtat, Lyon, France. She received another M.S.
and Ph.D. degrees from The Ohio State University,
Columbus, Ohio, USA, where show was on a schol-
arship by JUST. Currently, she is a Professor at
the College of Engineering, University of Sharjah,
UAE and a professor in the Department of Surveying
Engineering and Geomatics and Civil Engineering,
Faculty of Engineering, at the Al-Balqa Applied
University, Jordan. She has published good number
of refereed journal and conference papers. She served as a Department Chair
more than once. She worked at City University of New York, USA, and
Jordan University of Science and Technology in Jordan. Her research interests
include: modeling and simulation, Geographical Information Systems (GIS),
Wireless Navigation Systems, Data Analytics, Transportation Systems, Smart
Homes and Cities, among others.
Huizhi Wang is now studying communication engi-
neering at Northwestern Polytechnical University for
undergraduate degree. Her research interests include
machine learning and network measurement.
Shengtao Li received his M.S. degree in operational
research and cybernetics from Ludong University,
China, in 2010, and Ph.D. degree in control theory
and control engineering from Northeastern Univer-
sity, China in 2013. He is currently an associate pro-
fessor in the school of information science and engi-
neering, Shandong Normal University. His research
interests include nonlinear systems theory, optimal
switch-time control theory of switched stochastic
systems, and robust control of systems with time-
delay.
Lei Guo (M’06) received the Ph.D. degree from
the University of Electronic Science and Technol-
ogy of China, Chengdu, China, in 2006. He is a
Full Professor with the School of Communication
and Information Engineering, Chongqing Univer-
sity of Posts and Telecommunications, Chongqing,
China. He has authored or co-authored more than
200 technical papers in international journals and
conferences. His current research interests include
communication networks, optical communications,
and wireless communications. Dr. Guo is currently
serving as an Editor for several international journals.
Guoyin Wang (SM’03) received the B.S., M.S., and
Ph.D. degrees in computer science and technology
from Xi’an Jiaotong University, Xi’an, China, in
1992, 1994, and 1996, respectively.
During 1998−1999, he was a Visiting Scholar
with the University of North Texas, Denton,
TX, USA, and the University of Regina, Regina,
SK, Canada. Since 1996, he has been with the
Chongqing University of Posts and Telecommuni-
cations, Chongqing, China, where he is currently a
Professor, the Director of the Chongqing Key Labo-
ratory of Computational Intelligence, and the Dean of the School of Graduate.
He was appointed as the Director of the Institute of Electronic Information
Technology, Chongqing Institute of Green and Intelligent Technology, CAS,
China, in 2011. He is the author of ten books, the Editor of dozens of
proceedings of international and national conferences, and has more than
200 reviewed research publications. His research interests include rough set,
granular computing, knowledge technology, data mining, neural network, and
cognitive computing.