Journal of Information Security and Applications 59 (2021) 102821

Contents lists available at ScienceDirect

Journal of Information Security and Applications

journal homepage: www.elsevier.com/locate/jisa

A novel Byzantine fault tolerance consensus for Green IoT with intelligence
based on reinforcement
Peng Chen (first author) a ,1 , Dezhi Han (first author) a ,1 , Tien-Hsiung Weng b , Kuan-Ching Li b ,∗,
Arcangelo Castiglione c
a
Department of Computer Science and Technology, Shanghai Maritime University, Shanghai 200120, China
b Department of Computer Science and Information Engr. (CSIE), Providence University, Taichung 43301, Taiwan
c Department of Computer Science, University of Salerno, Via Giovanni Paolo II, 132, 84084, Fisciano, Salerno, Italy

ARTICLE INFO ABSTRACT

Keywords: To enhance the consensus performance of Blockchain in the Green Internet of Things (G-IoT) and improve
Blockchain the static network structure and communication overheads in the Practical Byzantine Fault Tolerance (PBFT)
Green IoT consensus algorithm, in this paper, we propose a Credit Reinforce Byzantine Fault Tolerance (CRBFT) consensus
Byzantine fault tolerance
algorithm by using reinforcement learning. The CRBFT algorithm divides the nodes into three types, each with
Reinforcement learning
different responsibilities: master node, sub-nodes, and candidate nodes, and sets the credit attribute to the
Consensus algorithm
Smart city
node. The node’s credit can be adjusted adaptively through the reinforcement learning algorithm, which can
dynamically change the state of nodes. CRBFT algorithm can automatically identify malicious nodes and invalid
nodes, making them exit from the consensus network. Experimental results show that the CRBFT algorithm can
effectively improve the consensus network’s security. Besides, compared with the PBFT algorithm, in CRBFT,
the consensus delay is reduced by about 40%, and the traffic overhead is reduced by more than 45%. This
reduction is conducive to save energy and reduce emissions.

1. Introduction
Green Internet of Things (G-IoT) is a new design concept of the
Internet of Things (IoT) that optimizes network devices and introduces
new technologies to achieve energy saving, pollution reduction, and
lower operating costs [1]. In recent years, with the development of
the IoT technology, energy consumption problems in network mainte-
nance, communication overhead, and data management have become
increasingly prominent. Therefore, the demand for G-IoT is becoming
more and more urgent [2]. A Blockchain is a list of records linked and
protected by cryptography that holds the following main characteris-
tics: decentralization, distribution, peer-to-peer (P2P) architecture [3].
The essence of a Blockchain is a distributed database technology [4].
More precisely, through cryptography, consensus algorithm, distributed
storage, and point-to-point technology, each node in the network main-
tains the network data’s consistency and validity and constructs a
decentralized distributed system [5]. The use of Blockchain in the
IoT can enhance security and mitigate energy consumption problems
caused by centralized servers [6,7]. As a result, the combination of IoT
and blockchain can effectively reduce energy consumption, which is
conducive to IoT energy saving and reducing operation costs and is in
line with the design concept of G-IoT. As the technological foundation
of smart cities, the IoT provides smart cities with urban perception
capabilities. Therefore, the development of G-IoT contributes to the
sustainable development of smart cities [8,9].
The consensus algorithm is an essential part of Blockchain technol-
ogy, and it is necessary to solve Byzantine faults. A suitable consensus
algorithm is conducive to reducing consensus energy consumption and
resource waste. A Byzantine fault takes its name from the Byzantine
general problem, proposed by Lamport [10]. The Byzantine general
problem is typically used to characterize consistency problems in dis-
tributed systems and represents the underlying issue for the Blockchain
consensus Algorithm. More precisely, the Byzantine general problem
in a computer system can be expressed as follows: assuming there is
a reliable channel for the transmission of messages, how to avoid the
influence of malicious nodes in the system, so that the whole system
can run well without the impact of such nodes, and ensure the integrity,
reliability, and consistency of information data [11]?
The Blockchain consensus algorithm aims to make the transaction
data verified by more than half of the nodes. Research on consensus
algorithms has long been underway. Pease and Lamport first proposed

∗ Corresponding author.
E-mail addresses: dzhan@shmtu.edu.cn (D. Han), thweng@pu.edu.tw (T.-H. Weng), kuancli@pu.edu.tw (K.-C. Li), arcastiglione@unisa.it (A. Castiglione).
1
Authors Peng Chen and Dezhi Han contributed equally to this work.

https://doi.org/10.1016/j.jisa.2021.102821

Available online 22 April 2021

2214-2126/© 2021 Published by Elsevier Ltd.
P. Chen et al.
the Byzantine Fault Tolerance (BFT) algorithm in the 1980s [12]. This
algorithm relies on the mutual transmission of information between
the nodes to reach a deterministic consensus result. However, BFT is
not practical since, in this algorithm, the complexity of the messages
exchanged between the nodes is exponential. The process of joining and
exiting nodes requires special processing. In 1993, Cynthia Dwork and
Moni Naor first proposed the proof-of-work (PoW) algorithm [13]. In
this algorithm, the client needs to perform a certain amount of compli-
cated calculations. Therefore, this algorithm requires more nodes and
computing power, resulting in a long transaction time. In 1999, Miguel
Castro and Barbara Liskov improved the BFT by proposing the Practical
Byzantine Fault Tolerance (PBFT) algorithm [14]. The PBFT inherited
the advantages of BFT while reducing the algorithm complexity from
exponential to polynomial. However, the PBFT algorithm adopts the
Client–Server (C/S) structure [15], and its consensus nodes are fixed,
making it unable to perceive the changes in the number of nodes
dynamically. Therefore, with the further development of Blockchain
technology, many researchers have proposed improved algorithms to
overcome the limitations of the PBFT algorithm.
In particular, Malkhi et al. introduce the Flexible BFT [16], a new
approach for BFT consensus which is resilient to higher malicious levels
than possible in a pure Byzantine fault model. But the new fault model
it proposed cannot predict what these replicas would do if they can vio-
late safety. Duan et al. present hBFT, a hybrid, Byzantine fault-tolerant
algorithm [17], which can detect and identify faulty clients. But if
clients are participating, it is necessary to ensure that the master node
is correct to maintain performance. Liu introduces a Dynamic autho-
rization of the Byzantine fault-tolerant (DDBFT) algorithm [18], which
applies the Delegated Proof of Stake (DPoS) algorithm to PBFT. The
DDBFT algorithm is dynamic. This characteristic improves throughput
and reduces the delay. On the other hand, it causes the system blocking
when the block size (transmitted data size) exceeds the node processing
capacity, resulting in wasting resources. Li and Zhang in [19] describe a
Group-Hierarchy (GH) algorithm based on PBFT, which divides replicas
into groups. Each group executes the normal-case operation of PBFT
concurrently to reach a local consensus. Then, every group’s primary
as the consensus representative reaches an agreement with other pri-
maries to reach a global consensus. This algorithm reaches a consensus
faster than PBFT but does not deal with malicious nodes, wasting many
resources, and weakening system stability. The above algorithms have
some drawbacks, especially in resource consumption, which do not
meet the G-IoT requirements.
In recent years, with the coming of the research upsurge of machine
learning (ML), the combination of Blockchain and machine learning
has attracted researchers’ attention [9,20,21]. For example, machine
learning applies to the design of fair data transaction protocol based on
Blockchain [22], and deep learning (DL) finds application in financial
investment research based on Blockchain [23]–[24]. Reinforcement
learning (RL) is a behavior-based machine learning method, which can
adjust its behavior through the interaction between agent and envi-
ronment. More precisely, RL has attracted much attention in artificial
intelligence (AI) for its excellent decision-making ability [25]–[26].
Based on the above problems, in this paper, we propose a Credit
Reinforcement Byzantine Fault Tolerance (CRBFT) consensus algorithm
based on the RL algorithm, where ‘‘credit’’ is the quantitative per-
formance of the credibility of each consensus node, that is, ‘‘credit’’
is an indicator of whether a node can participate in the consensus.
Besides, a node with a higher credit value has a higher priority to
participate in the consensus. The CRBFT consensus algorithm combines
RL with the PBFT algorithm, using the RL algorithm to identify ma-
licious nodes and invalid nodes automatically. Furthermore, CRBFT
is reliable, efficient, safe, and dynamic. It can reduce communication
overhead and consensus energy consumption, which is conducive to
G-IoT development.
In detail, the main contributions of this paper are as follows:
2
Journal of Information Security and Applications 59 (2021) 102821
(1) We apply for the first time reinforcement learning to the design
of the BFT consensus algorithm, introduce the concept of credit,
and propose a reinforcement-based BFT consensus algorithm,
CRBFT. With the proposed algorithm, the network has cognitive
intelligence, which allows it to identify automatically malicious
nodes and failed nodes, adaptively adjust the credit of nodes
so that the state between nodes can be dynamically adjusted
with time. It reduces the interference of malicious nodes to the
consensus process and improving the security of the consensus
network;
(2) We improve the PBFT algorithm. Change from the C/S paradigm
to the P2P paradigm, remove the confirmation phase, and per-
form synchronous verification when the master node changes.
It makes the algorithm conform to decentralization, reduces
communication overhead, and lowers consensus energy con-
sumption.
(3) The experimental results show that compared with the tra-
ditional PBFT consensus algorithm, the CRBFT algorithm we
proposed significantly reduces the consensus delay. In the pro-
posed algorithm, the performance is less affected by the increase
in the number of nodes.
The remaining of this paper is organized as follows. Section 2
introduces some preliminary knowledge and background necessary to
understand the consensus algorithm design better. Section 3 presents
the design ideas and details concerning the CRBFT consensus algorithm
based on reinforcement learning. In Section 4, we show the experi-
mental results achieved through simulation. Finally, in Section 5, we
provide some concluding remarks and future research prospects.
2. Preliminaries and problem formulation
To better illustrate the design details of the Credit Reinforcement
Byzantine Fault Tolerance (CRBFT) consensus algorithm proposed in
this paper, in this section, we introduce the basic knowledge of the
Practical Byzantine Fault Tolerance (PBFT) algorithm and provide the
concepts underlying the reinforcement learning (RL) algorithm used in
our proposal.
2.1. Practical Byzantine Fault Tolerance (PBFT)
Lamport et al. showed that an effective Byzantine Fault Tolerant
algorithm exists when the number of traitors (which can make mis-
takes) in the Byzantine system does not exceed 1/3 of the total amount.
Conversely, if it exceeds 1/3, there is no guarantee that the system
will achieve a consistent result [27]. Therefore, the solution to the
Byzantine problem can be determined if Eq. (1) is satisfied,
𝑛 ≥ 3𝑓 + 1 (1)
where 𝑛 is the total number of nodes, and 𝑓 is the Byzantine nodes’
maximum tolerable number.
The PBTF algorithm requires that all nodes have the same result of
operation execution under the same given service state and parameters,
and all nodes must start execution from the same state. Under this
constraint, even if there are failed nodes, the PBFT algorithm agrees
on the total execution order of all non-failed nodes, thereby ensuring
safety [28]. The PBFT algorithm divides all nodes into three types: a
client node, a master node, and a backup node. Again, it divides the
consensus process into three stages: pre-preparation stage, preparation
stage, and confirmation stage. We show the consensus process of the
PBFT algorithm in Fig. 1.
The result of the process is not valid until the client node receives
the same result from at least 𝑓 + 1 normal nodes that do not make
mistakes.
P. Chen et al.
Fig. 1. The PBFT algorithm consensus process.
2.2. Reinforcement learning (RL)
Reinforcement learning (RL) is a type of machine learning inspired
by animal psychology and combined with psychology, control theory,
and other related subjects [29]–[30]. In simple terms, RL is a cyclic
process in which an agent takes action to change its state to gain reward
and interact with the environment. Markov Decision Processes (MDPs)
provide a framework for the study of RL. A typical MDPs problem can
be expressed as a five-tuple (𝑆, 𝐴, 𝑃 , 𝑅, 𝛾), where 𝑆 is a set of states,
and 𝐴 is a set of actions. The transition probability 𝑃 describes the
probability distribution of the agent’s transition from the current state
𝑠 ∈ 𝑆 to other states after the action of 𝑎 ∈ 𝐴. 𝑅 is the reward function,
which defines the reward for the action 𝑎 ∈ 𝐴. 𝛾 is a discount factor that
is mainly used to balance current and future rewards. The objective of
MDPs is to get the maximum return 𝐺𝑡 when taking the corresponding
action 𝑎 under state 𝑠. The return 𝐺𝑡 is the total discounted reward from
time-step 𝑡. Formally, 𝐺𝑡 is defined by Eq. (2),
𝐺𝑡 = 𝑅𝑡+1 + 𝛾 1 𝑅𝑡+2 + 𝛾 2 𝑅𝑡+3 + ⋯ (2)
where 0 < 𝛾 < 1, 𝑅𝑡+1 represents the reward of the state 𝑠𝑡 to 𝑠𝑡+1 . The
optimal strategy to obtain the maximum return can be found by solving
the optimal Bellman equation, which is defined as follows
∑ adapt better the Byzantine fault-tolerant consensus algorithm to the
𝑣∗ (𝑠) = 𝑚𝑎𝑥𝑎 𝑞 ∗ (𝑠, 𝑎) = 𝑚𝑎𝑥𝑎 (𝑅𝑎𝑠 + 𝛾 𝑎 ∗ ′
𝑃𝑠𝑠′ 𝑣 (𝑠 )) (3)
𝑠′
∑
𝑞 ∗ (𝑠, 𝑎) = 𝑚𝑎𝑥𝜋 𝑞 𝜋 (𝑠, 𝑎) = 𝑅𝑎𝑠 + 𝛾 𝑎
𝑃𝑠𝑠 ∗ ′ ′
′ 𝑚𝑎𝑥𝑎′ 𝑞 (𝑠 , 𝑎 ) (4)
𝑠′
where 𝑣∗ (𝑠) represents the optimal long-term value of the state 𝑠, that
is, the value of the state in which all possible actions are considered,
and the optimal action is selected; strategy 𝜋 is a method for agent to
choose actions; 𝑞 𝜋 (𝑠, 𝑎) is the action-value function, which represents
the expected return of using strategy 𝜋 and taking action 𝑎 in state 𝑠;
𝑞 ∗ (𝑠, 𝑎) represents the optimal value among the action-value functions
generated under all strategies; 𝑅𝑎𝑠 is the expected return of taking action
𝑎 in state 𝑠; 𝑃𝑠𝑠𝑎 is the probability of taking action 𝑎 from state 𝑠 to state
′
′
𝑠.
We can solve the optimal Bellman equation by the Temporal-
Difference (TD) algorithm [31]. The TD algorithm is a model-free RL
algorithm, which updates the state value 𝑉 (𝑠𝑡 ) by predicting the TD
target, where TD target= 𝑅𝑡+1 + 𝛾𝑉 (𝑠𝑡+1 ).
𝑉 (𝑠𝑡 ) ← 𝑉 (𝑠𝑡 ) + 𝛼[𝑅𝑡+1 + 𝛾𝑉 (𝑠𝑡+1 ) − 𝑉 (𝑠𝑡 )]. (5)
The purpose of the TD algorithm update is to minimize the error
between the final predicted value and the true value, to reduce its error.
TD error 𝛿𝑡 is defined by Eq. (6):
𝛿𝑡 ≐ 𝑅𝑡+1 + 𝛾𝑉 (𝑠𝑡+1 ) − 𝑉 (𝑠𝑡 ). (6)
The Actor-Critic method [32] is an effective RL algorithm. In this
method, the Actor selects the action according to the current system
state. On the other hand, the Critic gives the corresponding value
evaluation according to the current system state and the action selected
by the Actor, parameterizes the action-value function, and updates its
3
Journal of Information Security and Applications 59 (2021) 102821
Fig. 2. Schematic diagram for implementations of the actor-critic.
parameters by reward 𝑅𝑡+1 to get a higher return. The Actor then
directs the update of the action based on the value derived from the
Critic. Then, the objective function is constructed, and the parameters
are updated iteratively to make the output meet the threshold of the
objective function. The final output is the approximate optimal solution
of the Bellman equation. Therefore, the Actor-Critic method can solve
the optimal Bellman equation adaptively, as shown by the diagram in
Fig. 2.
3. Credit Reinforcement Byzantine Fault Tolerance consensus al-
gorithm design
In this section, we provide a detailed description of our proposal.
More precisely, we first present the design approach underlying the
Credit Reinforcement Byzantine Fault Tolerance (CRBFT) consensus
algorithm, and then we give the relative details.
3.1. Algorithm design approach
In the Blockchain, the primary purpose is to reach a consensus
on the Blockchain transaction information across the entire network,
which does not involve the consensus request’s order. Therefore, to
Blockchain, we propose a Credit Reinforcement Byzantine Fault Tol-
erance (CRBFT) consensus algorithm.
First of all, we adapt the PBFT algorithm to make it compatible
with the Blockchain system’s effective application. More precisely, we
propose the following improvements to the PBFT algorithm:
1. According to the Blockchain decentralized architecture, we shift
from the Client–Server (C/S) paradigm to the Peer-to-Peer (P2P)
paradigm. Thus, there is no client in the system;
2. We divide the consensus node into three types: master node,
sub-node, and candidate node;
3. We set the ‘‘credit’’ attribute for consensus nodes. In this way,
the system can dynamically divide the types of consensus nodes,
and nodes can dynamically join and leave the system;
4. We remove the confirmation phase from the consensus process
of the PBFT algorithm and carry out the synchronous verification
process when the master node changes;
5. We perform adaptive consensus confirmation and reputation
adjustment based on reinforcement learning.
We remark that removing the verification phase of the protocol re-
duces the network’s bandwidth to a certain extent. However, to remove
state inconsistency and ensure the system consistency after the master
node change, the synchronous verification process is included after the
change of such a node. More precisely, the synchronous verification
process works as follows. The sub-node sends a synchronization request
to the new master node to verify whether the master node number
is consistent. After the synchronization is successful, the master node
sends backup data to the sub-node, and then the sub-node verifies
the validity of the backup data. Through this operation, we reduce
P. Chen et al.
communication overhead and improve consensus efficiency, without
compromising the fault tolerance of the system.
Furthermore, according to the output of the action performed by
reinforcement learning, the consensus system adjusts the credit of
the nodes participating in the consensus each time. The node that
successfully participates in the consensus is that whose verification
message positively impacts the successful release of the block by the
master node. Nodes that successfully participate in the consensus will
get a ‘‘reward’’ so that the successful nodes have higher credit and are
more likely to participate in the next consensus. Conversely, malicious
nodes and failed nodes will be ‘‘punished’’. The credit of such nodes
will decrease with the increase of the consensus. Besides, when a node’s
credit is lower than a given threshold (base value), such a node will
quit the consensus. In this way, the system can effectively identify
malicious nodes and failed nodes and dynamically add and delete
the nodes participating in the consensus. Consequently, these design
choices make the consensus environment more in line with current
network scenarios and make it more stable and secure while reducing
resource consumption.
3.2. Algorithm design content
3.2.1. Algorithm notation and preprocessing
The basic parameters used in the system are defined as follows:
• Set ‘‘credit’’ 𝐶 as the index to measure the reliability of con-
sensus nodes (i.e., the possibility of successfully participating in
consensus), and take it as the main basis to divide the consensus
nodes;
• Consensus nodes are divided into three types: master node, sub-
nodes, and candidate nodes. The master node is the node with
the highest credit. The number of consensus nodes is 𝑁, where
𝑁 ≥ 3𝐹 + 1 and 𝐹 is the maximum number of malicious nodes
that the system can tolerate. In general, 𝑁 takes 3𝐹 + 1;
• Preset the value of the admission credit 𝐶𝑏𝑎𝑠𝑒 . 𝑁 − 1 candidate
nodes with a credit 𝐶 ≥ 𝐶𝑏𝑎𝑠𝑒 are selected as sub-nodes. Nodes
with a credit lower than 𝐶𝑏𝑎𝑠𝑒 or newly added nodes are selected
as candidate nodes.
3.2.2. An overview of the CRBFT algorithm
When the system is initialized, all the consensus nodes’ credit is
set to 𝐶𝑏𝑎𝑠𝑒 , and 𝑁 consensus nodes are randomly selected as sub-
nodes. The master node is then elected from the sub-nodes, and the
number 𝑝 is assigned to it. Since the Blockchain’s consensus system
is P2P-based and decentralized, each consensus node needs to start in
the same state. This assumption means that the data stored by each
consensus node needs to be consistent. Ensuring consistency requires
data backup and verification. After the data backup and verification
phases are complete, the system begins the consensus process. We
remark that in the CRBFT algorithm, message delivery uses digital
signature techniques and the SHA-256 algorithm to ensure the integrity
and authenticity of the message.
The master node initiates the consensus process and sets the master
node consensus interval to 𝛥𝑡. The details of the CRBFT algorithm are
as follows:
1. The initiator starts its transaction by signing this transaction and
broadcasting it to the consensus node;
2. The consensus node verifies the legality of the received transac-
tion. If it is legal, the transaction information is cached, and the
master node generates a block. If the received transaction is not
legal, it is directly discarded;
3. The master node 𝑝 generates a proposal message for the gen-
erated block after 𝛥𝑡 and assigns a unique number 𝑛 to this
proposal. This number is increased with each new proposal
generated. Then the master node 𝑝 broadcasts this proposal
message to the sub-node to request the sub-node to participate
4
Journal of Information Security and Applications 59 (2021) 102821
Fig. 3. The consensus process of the CRBFT.
in the consensus. The message format is ≪ 𝑃 𝑟𝑜𝑝𝑜𝑠𝑎𝑙, 𝑝, 𝑛, 𝑠𝑚 , 𝑑 >
, 𝑏𝑙𝑜𝑐𝑘 >, where 𝑠𝑚 is the message signature and 𝑑 is the message
digest computed using the SHA -256 algorithm;
4. The sub-node first checks the message signature, proposal num-
ber, and master node number of the proposal. If the verification
fails, the sub-node discards the proposal message. On the other
hand, if the verification succeeds, the sub-node sends a veri-
fication message to the master node. The verification message
format is ⟨𝐶𝑜𝑛𝑠𝑒𝑠𝑢𝑠𝐶𝑜𝑚, 𝑝, 𝑛, 𝑠𝑚 , 𝑑, 𝑟𝑒, 𝑘, 𝐶⟩, where 𝑘 is the sub-
node number, 𝐶 is the reputation of node 𝑘, and 𝑟𝑒 denotes the
verification type of the sub-node 𝑘 to the proposal message di-
gest. 𝑟𝑒 can be as simple as either a ‘‘1’’ or a ‘‘−1’’, corresponding
to ‘‘true’’ or ‘‘false’’, respectively, where ‘‘true’’ means that the
proposal message-digest 𝑑 is consistent with the data cached by
the sub-node. Otherwise, ‘‘false’’ means inconsistency and that
the sub-node suspects of the master node as well;
5. The master node then uses the reinforcement learning to perform
the consensus confirmation of the received verification message.
If the master node receives 2𝐹 confirmation messages, it con-
siders that consensus has been reached and publishes a block.
If the master node receives 2𝐹 suspicious messages, the con-
sensus network broadcasts the message of changing the master
node and reselects the master node. If not enough messages are
received during the consensus timeout, all nodes discard the
block generated during the consensus process, replace the master
node, and reduce the ‘‘credit’’ of the replaced master node. Then,
reselect the master node, but no longer select the original master
node. Finally, start a new round of consensus. If the master
node can successfully publish the block, after publishing the
block, such a node sends the credit adjustment message to the
sub-node;
6. When the consensus node receives the published block, it con-
siders this round of consensus complete. Then such a node
updates the credit according to the adjusted information, clears
the cache, updates the consensus node, and starts a new round.
The consensus process of the CRBFT is shown in Fig. 3, and the
CRBFT algorithm information flow is shown in Fig. 4.
3.2.3. Reinforcement learning consensus confirmation
Consensus confirmation combines the Actor-Critic method, TD algo-
rithm, and neural networks. It is mainly composed of an action network
that produces actions and a critic network, which is used to evaluate
the actions. The main function of Actor-Critic is to judge whether the
sub-nodes are in agreement and provide a basis for credit adjustment.
In Fig. 5, we show the neural network structure for the critic network.
The structure of the action network is essentially the same as the critic
network. More precisely, both networks adopt the back propagation
(BP) neural network with a hidden layer.
Take 𝑟𝑒𝑘 and 𝐶𝑘 in the 𝑘𝑡ℎ verification message as the input of the
action network, and get the output 𝑢(𝑘). The critic network takes 𝑟𝑒𝑘 ,
𝐶𝑘 , and 𝑢(𝑘) as inputs, and obtains the output 𝐽 (𝑘) as the approximate
value of the return 𝐺𝑡 in Eq. (2), which is used to estimate the output of
the critic network approximately. Therefore, this approximation is used
P. Chen et al. Journal of Information Security and Applications 59 (2021) 102821

Fig. 6. Schematic diagram for implementations of the CRBFT.

In the critic network, the output 𝐽 (𝑘) is defined as follows:

𝑛𝑖𝑛 +1
∑
𝜑𝑖 (𝑘) = 𝑤(1)
𝑐 (𝑘)𝑥𝑗 (𝑘), 𝑖 = 1, … , 𝑁ℎ (9)
𝑖𝑗
Fig. 4. The CRBFT algorithm flowchart. 𝑗=1
1 − 𝑒𝑥𝑝−𝜑𝑖 (𝑘)
𝜙𝑖 (𝑘) = , 𝑖 = 1, … , 𝑁ℎ (10)
1 + 𝑒𝑥𝑝−𝜑𝑖 (𝑘)
𝑁ℎ
∑
𝐽 (𝑘) = 𝑤(2)
𝑐 (𝑘)𝜙𝑖 (𝑘) (11)
𝑖
𝑖=1

where 𝜑𝑖 is the 𝑖𝑡ℎ hidden node input of the critic network; 𝑛𝑖𝑛 + 1 is
the total number of inputs into the critic network, including the analog
action value from the action network; 𝜙𝑖 is the corresponding output;
𝑁ℎ is the total number of hidden nodes in the critic network; 𝑤𝑐 is the
weight vector in the critic network.
According to the error propagation equation of the back propaga-
tion algorithm and the chain rule, the gradient of the neural network
objective function to the weights can be obtained. We summarize the
Fig. 5. Schematic diagram for the implementation of a nonlinear critic network using
a feedforward network with one hidden layer.
adaptation of the critic network as follows:

• 𝛥𝑤(2)
𝑐 (hidden to output layer)

𝜕𝐸𝑐 (𝑘)
to quantify the output of 𝑢(𝑘) better. Then, compare the action output 𝛥𝑤(2)
𝑐 (𝑘) = 𝑙𝑐 (𝑘)[− ]
𝑢(𝑘) and 𝑟𝑒𝑘 to select the reinforcement signal 𝑟(𝑘). If 𝑢(𝑘) and 𝑟𝑒𝑘 have
𝑖
𝜕𝑤(2)
𝑐𝑖 (𝑘)

the same sign, the output is regarded as successful, and 𝑟(𝑘) is ‘‘0’’. On 𝜕𝐸 (𝑘) 𝜕𝐽 (𝑘) (12)
= −𝑙𝑐 (𝑘)[ 𝑐 ]
the other hand, if the sign is different, it is considered to be a failure, 𝜕𝐽 (𝑘) 𝜕𝑤(2) (𝑘)
𝑐𝑖
and 𝑟(𝑘) is ‘‘1’’. = −𝑙𝑐 (𝑘)[𝛼𝑒𝑐 (𝑘)𝜙𝑖 (𝑘)]
In the initial stage of consensus, the weights of the action network
• 𝛥𝑤(1)
𝑐 (input to hidden layer)
and critic network are random. During the neural network’s learning
process, the critic network uses the 𝑟(𝑘) to update the weights and 𝜕𝐸𝑐 (𝑘)
𝛥𝑤(1)
𝑐 (𝑘) = 𝑙𝑐 (𝑘)[− ]
obtain the optimal 𝐽 (𝑘). The action network then uses the optimal 𝐽 (𝑘) 𝑖𝑗
𝜕𝑤(1)
𝑐𝑖𝑗 (𝑘)
to update its weights and get the optimal output. The neural network’s 𝜕𝐸𝑐 (𝑘) 𝜕𝐽 (𝑘) 𝜕𝜙𝑖 (𝑘) 𝜕𝜑𝑖 (𝑘)
= −𝑙𝑐 (𝑘)[ ] (13)
detailed design process will be given in parts A and B of this subsection. 𝜕𝐽 (𝑘) 𝜕𝜙𝑖 (𝑘) 𝜕𝜑𝑖 (𝑘) 𝜕𝑤(1) (𝑘)
𝑐𝑖𝑗
In Fig. 6, we show a diagram characterizing the process for tuning the
1
Actor-Critic method’s parameters. = −𝛼𝑙𝑐 (𝑘)𝑒𝑐 (𝑘)𝑤(2) 2
𝑐𝑖 (𝑘) ⋅ [ 2 (1 − 𝜙𝑖 (𝑘))]𝑥𝑗 (𝑘)

A. Critic network. The learning objective of the critic network is to
minimize the error between the approximation value of the value func-
tion and the actual value, while optimizing the maximum return. The
prediction error function 𝑒𝑐 (𝑘) of the critic network and the minimized
objective function 𝐸𝑐 (𝑘) is defined as follows:
𝑒𝑐 (𝑘) = 𝛼𝐽 (𝑘) − [𝐽 (𝑘 − 1) − 𝑟(𝑘)]
1 2
𝐸𝑐 (𝑘) = 𝑒 (𝑘).
2 𝑐
where 𝑙𝑐 (𝑘) > 0 is the learning rate of the critic network when
dealing with 𝑖𝑡ℎ message. We remark that this rate usually de-
creases over time to a small value.
B. Action Network. Previously we defined that if the output is regarded
as successful, 𝑟(𝑘) is ‘‘0’’. That is, ‘‘0’’ was defined as the reinforcement
signal for ‘‘success’’. To satisfy the Bellman equation and maximize the
(7)
state value function, the ultimate learning target of the action network,
denoted by 𝑈𝑐 , is set to ‘‘0’’ in the algorithm. Through observation, we
(8) found that the parameter adjustment principle of the action network

5
P. Chen et al. Journal of Information Security and Applications 59 (2021) 102821

is to indirectly back-propagate the error between 𝐽 and 𝑈𝑐 . We define Table 1

Experimental environment.
𝑒𝑎 (𝑘) and 𝐸𝑎 (𝑘) as follows:
CPU Intel(R) Core (TM) i7-6500U @2.60 GHz
Hardware
𝑒𝑎 (𝑘) = 𝐽 (𝑘) − 𝑈𝑐 (𝑘) (14) Memory 8 GB RAM
Operating System Window 10
Software
1 2 Simulation Software Pycharm 2019.2 (Python 3.8)
𝐸𝑎 (𝑘) = 𝑒 (𝑘). (15)
2 𝑎
The action network adopts a neural network structure, which is
similar to the critic network shown in Fig. 5. In detail, the relevant 4.1. Simulation setup
equations characterizing the action network are defined as follows:
𝑛𝑖𝑛
∑ The experimental environment for performance testing is shown in
𝑚𝑖 (𝑘) = 𝑤(1)
𝑎 (𝑘)𝑥𝑗 (𝑘), 𝑖 = 1, … , 𝑁ℎ (16) Table 1. The rules for adjusting the credit of the CRBFT algorithm are
𝑖𝑗
𝑗=1
as follows:
1 − 𝑒𝑥𝑝−𝑚𝑖 (𝑘)
𝑧𝑖 (𝑘) = , 𝑖 = 1, … , 𝑁ℎ (17)
1 + 𝑒𝑥𝑝−𝑚𝑖 (𝑘) 1. The initial credit of all nodes, denoted as 𝐶𝑏𝑎𝑠𝑒 , is set to 3;
𝑁ℎ
∑ 2. If the master node completes a consensus, its credit is incre-
ℎ(𝑘) = 𝑤(2)
𝑎 (𝑘)𝑧𝑖 (𝑘) (18) mented by 0.2. If the sub-node’s verification type is ‘‘true’’, the
𝑖
𝑖=1 credit of this sub-node is incremented by 0.1. On the other
1 − 𝑒𝑥𝑝−ℎ(𝑘) hand, if the verification type is ‘‘false’’, this sub-node’s credit
𝑢(𝑘) = (19)
1 + 𝑒𝑥𝑝−ℎ(𝑘) is decremented by 0.1. Also, the credit is reduced by 0.1 to a
where 𝑚𝑖 is input of the 𝑖𝑡ℎ hidden node of the action network, 𝑧𝑖 is sub-node that has not responded over time;
the corresponding output; ℎ is the input of the output node; 𝑤𝑎 is the 3. If the sub-node suspects that the master node is successful, it sets
weight vector in the action network. the credit of the master node to 2;
Similar to the critic network, the parameters update rules of the 4. If the master node does not complete the consensus within the
action network are summarized in Eqs. (20)–(21). timeout, the master node’s credit is decremented by 1.

Notice that the Actor-Critic structure implements the calculation of

• 𝛥𝑤(2)
𝑎 (hidden to output layer)
the credit adjustment value in step 2. The relation between the output
𝜕𝐸𝑎 (𝑘) 𝑢 and 𝛥𝐶 is defined by Eq. (22):
𝛥𝑤(2)
𝑎 (𝑘) = 𝑙𝑎 (𝑘)[− ] {
𝑖
𝜕𝑤(2)
𝑎𝑖 (𝑘) 𝑚𝑎𝑔 𝑢≥0
𝛥𝐶 = (22)
𝜕𝐸 (𝑘) 𝜕𝐽 (𝑘) 𝜕𝑢(𝑘) 𝜕ℎ(𝑘) −2𝑚𝑎𝑔 𝑢<0
= −𝑙𝑎 (𝑘)[ 𝑎 ]
𝜕𝐽 (𝑘) 𝜕𝑢(𝑘) 𝜕ℎ(𝑘) 𝜕𝑤(2) (𝑘) where the 𝑚𝑎𝑔 is the adjustment rate of credit, and it can be set freely.
𝑎𝑖
𝑁ℎ
∑ In this paper, we set it to 0.1.
1 1
= 𝑒𝑎 (𝑘)[ (1 − 𝑢2 (𝑘))]𝑧𝑖 (𝑘) [ 𝑤2𝑐 (𝑘)(1 − 𝜙2 (𝑘))𝑤(1)
𝑐𝑖 ,𝑛+1
(𝑘)] In addition, the specific settings of the neural network are as fol-
2 𝑖=1
2 𝑖
lows:
(20)
- 𝑙𝑐 (0) 0.3, initial learning rate of the critic network;
• 𝛥𝑤(1)
𝑎 (input to hidden layer) - 𝑙𝑎 (0) 0.3, initial learning rate of the action network;
- 𝑙𝑐 (𝑡) learning rate of the critic network at time 𝑡, that is
𝜕𝐸𝑎 (𝑘)
𝛥𝑤(1)
𝑎 (𝑘) = 𝑙𝑎 (𝑘)[− ] decreased by 0.05 every five-time steps, until it reaches 0.005
𝑖𝑗
𝜕𝑤(1)
𝑎𝑖𝑗 (𝑘) and it stays at 0.005 after that;
𝜕𝐸𝑎 (𝑘) 𝜕𝐽 (𝑘) 𝜕𝑢(𝑘) 𝜕ℎ(𝑘) 𝜕𝑧𝑖 (𝑘) 𝜕𝑚𝑖 (𝑘) - 𝑙𝑎 (𝑡) learning rate of the action network at time 𝑡, that is
= −𝑙𝑎 (𝑘)[ ] decreased by 0.05 every five-time steps, until it reaches 0.005
𝜕𝐽 (𝑘) 𝜕𝑢(𝑘) 𝜕ℎ(𝑘) 𝜕𝑧𝑖 (𝑘) 𝜕𝑚𝑖 (𝑘) 𝜕𝑤(1) (𝑘)
𝑎𝑖𝑗
and it stays at 0.005 after that;
1 1
= −𝑙𝑎 (𝑘)𝑒𝑎 (𝑘)[ (1 − 𝑢2 (𝑘))] ⋅ 𝑤2𝑎 (𝑘)[ (1 − 𝑧2𝑖 (𝑘))]𝑥𝑗 (𝑘) - 𝑁𝑐 50, internal cycle of the critic network;
2 𝑖 2
𝑁ℎ - 𝑁𝑎 100, internal cycle of the action network;
∑ 1 2
⋅ [ 𝑤𝑐 (𝑘)(1 − 𝜙2 (𝑘))𝑤(1)
𝑐𝑖 ,𝑛+1
(𝑘)] - 𝑇𝑐 0.05, internal training error threshold for the critic net-
𝑖=1
2 𝑖 work (the threshold of 𝐸𝑐 (𝑘));
(21) - 𝑇𝑎 0.005, internal training error threshold for the action
network (the threshold of 𝐸𝑎 (𝑘));
where 𝑙𝑎 (𝑘) > 0 is the learning rate of the action network in - 𝑁ℎ 6, number of the hidden nodes.
dealing with the 𝑖𝑡ℎ message.

We define the list NT to record nodes where 𝑟𝑒 is ‘‘1’’, and the list
NF to record nodes where 𝑟𝑒 is ‘‘−1’’. The consensus is confirmed when
the number of nodes in NT is greater than or equal to 2𝐹 . Conversely,
when the number of nodes in NF is greater than or equal to 2𝐹 , the
sub-node should not trust the master node.
4. Results and discussion
This section first explains the simulation settings. Then, we test
the performance of the CRBFT algorithm and compare the CRBFT
algorithm with other related works to verify its effectiveness and
availability. Finally, we list two algorithm applications.
4.2. Performance evaluation
4.2.1. Algorithm performance test
In the experiment, we set 𝐹 to 3, 𝑁 to 13, and we consider 10
consensuses. Again, we set node 3 to become malicious at the 3𝑟𝑑
consensus, node 2 to become failed at the 4𝑡ℎ consensus, and node 5
to become malicious at the 6𝑡ℎ consensus. We show the credit trends
of each node in Fig. 7.
In Fig. 7, we can observe that in the 3𝑟𝑑 consensus, node 3 becomes
a malicious node and its credit decreases. When the credit of node 3 is
lower than 𝐶𝑏𝑎𝑠𝑒 , node 11 changes from a candidate node to a sub-node,
and node 3 becomes a candidate node. Node 2 becomes a failed node at
the 4𝑡ℎ consensus, and the credit also decreases, but the decrease rate

6
P. Chen et al.
Fig. 7. Credit trend of nodes during the 10 consensus.
Fig. 8. Comparison diagram of single consensus time.
is slower than malicious nodes. When the credit is lower than 𝐶𝑏𝑎𝑠𝑒 , it
becomes a candidate node.
We can conclude that the proposed algorithm can adaptively adjust
the credit of consensus nodes, effectively identify malicious nodes and
failed nodes, and dynamically adjust the types of nodes, improving con-
sensus security. It can also be observed that it will give the successful
sub-node priority to take part in the next consensus, and the master
node has the advantage of being the master node next time.
The comparison of a single consensus time is shown in Fig. 8. We
can observe that the first consensus takes the longest time, and with
the learning of the neural network, the consensus time decreases until
the learning ends. The consensus time stabilizes at around 200 ms.
4.2.2. Delay test
Consensus delay is an important index to measure the speed of the
consensus algorithm. A low consensus delay can make the transaction
be confirmed quickly. Thus, the Blockchain results in being more secure
and practical. More precisely, the consensus delay 𝑇𝑑 tested in this
paper is a consensus completion time, and it is defined by Eq. (23):
𝑇𝑑 = 𝑇𝑡𝑐 − −𝑇𝑡𝑟
where 𝑇𝑡𝑟 is the transaction start time, and 𝑇𝑡𝑐 is the consensus comple-
tion time. Then, 𝐹 is set to values 1, 2, and 3, and 𝑁 is set to values
4, 7, and 10, respectively. The average value is obtained through 10
tests and compared with the PBFT algorithm. The statistical results are
shown in Fig. 9.
In Fig. 9, we can observe that with the increase of nodes, the
consensus delay of both algorithms will increase, but the increase of
the PBFT is higher than the CRBFT, so the rise of consensus nodes has
a more significant impact on PBFT. Besides, using the same number of
nodes, the consensus delay of CRBFT is significantly lower than that of
PBFT, which is reduced by about 40%.
4.2.3. Communication frequency comparison
According to the PBFT algorithm execution flow, the number of
messages in each phase is simplified to get the Eq. (24) as follows:
𝑍𝑃 𝐵𝐹 𝑇 = 2𝑛 ⋅ (𝑛 − 1)
Journal of Information Security and Applications 59 (2021) 102821
Fig. 9. Consensus delay comparison between the two algorithms.
where 𝑛 is the number of nodes participating in the consensus. Simi-
larly, we can get the number of single consensus messages for CRBFT
as follow:
𝑍𝐶𝑅𝐵𝐹 𝑇 = 4𝑛 − 3 (25)
The total amount of messages in GH [19] is given as below:
𝑛2 ∑
𝑗=𝑔
𝑍𝐺𝐻 = 2 +2 𝑀𝑗2 + 2𝑔 2 + 𝑔(𝑛 − 2) − 𝑛 (26)
𝑔 𝑗=1
where 𝑔 is the number of groups, 𝑀𝑗 is the number of nodes for the 𝑗𝑡ℎ
group. We set the number of nodes in each group to 2 in the experiment.
We compare the number of messages between these algorithms with
a different number of nodes, as shown in Fig. 10. More precisely,
as shown in Fig. 10, the number of messages exchanged by these
three algorithms increases as the number of nodes increases, but PBFT
increases faster. Although GH has fewer messages than PBFT, it still
(23) has more messages than CRBFT. In other words, CRBFT has a small
increase, compared with PBFT and GH, the number of messages is
reduced by more than 45%. It can effectively reduce the communica-
tion frequency of a single consensus process and reduce the resource
consumption of the network.
4.3. Algorithm application
4.3.1. Smart contracts in power system
Blockchain technology can realize more accurate and reliable per-
ception, transmission and recording of power systems, and more intel-
ligent and efficient mining, integration, and analysis of physical char-
acteristics and internal connections. Besides, Blockchain technology
can provide an open, transparent, and credible platform for electricity
market transactions, reducing contract execution risks and regulatory
costs. Simultaneously, it assists people in thinking and decision-making
(24) in the power system and its related social systems, which improves the
7
P. Chen et al.
Fig. 10. The number of messages between the related algorithms.
Fig. 11. The Blockchain mode of power system state estimation.
management level [33]. Its related applications are, for example, the
Blockchain mode of power system state estimation shown in Fig. 11.
Fig. 11 is a block chain-based power system state estimation model.
In Fig. 11, the data center and users form a blockchain power system,
and they both have communication and verification functions. The
data center receives power state estimation data from users and stores
the data. If an attacker wants to tamper with the information of a
data center or a single user, it needs to attack most users in the
blockchain system at the same time, which requires a huge cracking
cost. Therefore, the blockchain model can better prevent data intrusion
and form a data protection barrier.
More precisely, combining Fig. 11 and Blockchain technology
knowledge, we can apply the CPBFT algorithm to power systems.
Based on Blockchain data’s tamper-proof characteristics, Blockchain
nodes store energy consumption information collected from the IoT’s
intelligent metering devices, and then consumers in the Blockchain self-
execute smart contracts through the CRBFT algorithm. When the smart
contract legally satisfies all the judgment conditions, and the consensus
node reaches a consensus, the smart contract automatically enforces
and improves transaction flexibility and efficiency. Concurrently, the
increase or decrease of credibility in the CRBFT algorithm is used as
rewards and punishments to balance energy demand and grid energy
production rules, thereby balancing energy supply and demand. When
there is an external attack, the CRBFT algorithm can screen out the
error node and take defensive measures to ensure the information
security of the power system. The CRBFT algorithm ensures each
transaction’s legitimacy, guarantees the power system’s safety, and is
conducive to the construction and development of smart grids.
4.3.2. Transaction settlement and clearing in finance
In the field of clearing and settlement, the traditional transaction
model is that both parties keep separate accounts, and as the data is
recorded by each other, the authenticity is difficult to guarantee. In
contrast, the data in the blockchain is distributed, and each node has
access to all the transaction information, and once changes are detected
the whole network can be notified to prevent tampering.
8
Journal of Information Security and Applications 59 (2021) 102821
We can apply the CRBFT consensus algorithm to the financial
blockchain, then its transaction process and clearing process are syn-
chronized in real time, and the bookkeeping initiated by the seller
must be approved by the buyer to complete the transaction. More
importantly, in the transaction process, the CRBFT consensus algorithm
can dynamically adjust the credibility of the transaction object, which
is conducive to creating a safe and credible clearing and settlement
environment.
5. Conclusion and future work
This paper proposes a Credit Reinforcement Byzantine Fault Toler-
ance Consensus (CRBFT) algorithm based on reinforcement learning. In
the CRBFT algorithm, the credit attribute is set to the node. The node
credit is then adaptively adjusted through the reinforcement learning
algorithm to have a specific network cognitive ability. The proposed
algorithm can automatically identify malicious nodes and failure nodes
in the consensus network, thus improving consensus network security,
reducing consensus delay, stimulating energy saving, and emission
reduction. Moreover, compared with the PBFT algorithm, the CRBFT
algorithm has lower consensus delay, fewer communication times, and
less network resource consumption. These characteristics make our
proposal suitable for constructing Green IoT and promoting smart cities’
development.
As future research, we want to optimize the algorithm details, for
example, parameter selection in credit adjustment rules. In particular,
we intend to find new solutions to speed up the learning process
and stability of the neural network, besides further improving the
algorithm’s security, to apply it to different types of Blockchain in
IoT, such as DAG-structured blockchains [34]–[35]. Meanwhile, we
intend to explore the possibility of applying the Blockchain in the smart
grid to improve the algorithm further and promote the smart grid’s
development.
Declaration of competing interest
The authors declare that they have no known competing finan-
cial interests or personal relationships that could have appeared to
influence the work reported in this paper.
Acknowledgment
This study was funded by the National Natural Science Foundation
of China, under grants 61873160 and 61672338.
References
[1] Huang J, Meng Y, Gong X, Liu Y, Duan Q. A novel deployment scheme for green
internet of things. IEEE Internet Things J 2014;1(2):196–205. http://dx.doi.org/
10.1109/JIOT.2014.2301819.
[2] Zhang X, Huang Y, Wang WB. Green internet of things: Requirements,
development status and key technologies. Telecommun Sci 2012;28(8):96–104.
[3] Chang J, Han F. Blockchain: from digital currency to credit society. Beijing:
China CITIC Press; 2016.
[4] Yuan Y, Wang FY. Blockchain: The state of the art and future trends. Acta
Automat Sinica 2016.
[5] Bonneau J, Miller A, Clark J, Narayanan A, Kroll JA, Felten EW. SoK: Research
perspectives and challenges for bitcoin and cryptocurrencies. In: 2015 IEEE
Symposium on security and privacy, 2015, pp. 104–121.
[6] Tian Q, Han D, Li K-C, Liu X, Duan L, Castiglione A. An intrusion detection
approach based on improved deep belief network. Appl Intell 2020;50:3162–78.
[7] Xiao L, Han D, Meng X, Liang W, Li K-C. A secure framework for data sharing
in private blockchain-based WBANs. IEEE Access 2020;8:153956–68.
[8] Dorri A, Kanhere SS, Jurdak R, Gauravaram P. Blockchain for IoT security and
privacy: The case study of a smart home. In: 2017 IEEE international conference
on pervasive computing and communications workshops (PerCom Workshops), 2017.
[9] Liang W, Zhang D, Lei X, Tang M, Li K-C, Zomaya A. Circuit copyright
blockchain: Blockchain-based homomorphic encryption for IP circuit protection.
IEEE Trans Emerg Top Comput 2020. http://dx.doi.org/10.1109/TETC.2020.
2993032.
P. Chen et al.
[10] Drdobbs. The byzantine generals problem. Acm Trans Program Lang Syst
1982;4(3):382–401.
[11] Lamport L. Seminal research document related to the field of byzantine fault
tolerance. 1982.
[12] Pease M, Shostak R, Lamport L. Reaching agreement in the presence of faults.
J ACM 1980;27:228–34.
[13] Dwork C, Naor M. Pricing via processing or combatting junk mail. In: Annual
international cryptology conference, 1992.
[14] Castro M, Liskov B. Practical byzantine fault tolerance. Acm Trans Comput Syst
2002;20(4):398–461.
[15] Androutsellis-Theotokis S, Spinellis D. A survey of peer-to-peer content
distribution technologies. ACM Comput Surv 2004;36(4):335–71.
[16] Malkhi D, Nayak K, Ren L. Flexible byzantine fault tolerance. In: Proceedings of
the 2019 ACM SIGSAC conference on computer and communications security.
CCS ’19, New York, NY, USA: Association for Computing Machinery; 2019, p.
1041–53. http://dx.doi.org/10.1145/3319535.3354225.
[17] Duan S, Peisert S, Levitt KN. hBFT: Speculative byzantine fault tolerance with
minimum cost. IEEE Trans Dependable Secure Comput 2015;12(1):58–70. http:
//dx.doi.org/10.1109/TDSC.2014.2312331.
[18] Liu XF. Research on blockchain performance improvement based on byzantine
fault tolerance consensus algorithm based on dynamic authorization. [Ph.D.
thesis], Hangzhou: Zhejiang University; 2017.
[19] Li QW. Research on consensus efficiency based on practical byzantine fault
tolerance. In: ICMIC. Guiyang; 2018.
[20] Liang W, Fan Y, Li K-C, Zhang D, Gaudiot J-L. Secure data storage and
recovery in industrial blockchain network environments. IEEE Trans Ind Inf
2020;16(10):6543–52.
[21] Liang W, Huang W, Long J, Zhang K, Li K-C, Zhang D. Deep reinforcement
learning for resource protection and real-time detection in lot environment. IEEE
Internet Things J 2020;7(7):6392–401.
Journal of Information Security and Applications 59 (2021) 102821
[22] Zhao YQ, Yu Y, Li YN, Han G, Du XJ. Machine learning based privacy-preserving
fair data trading in big data market. Inform Sci 2019;478:449–60.
[23] Xie MH, Li HY, Zhao YJ. Blockchain financial investment based on deep learning
network algorithm. J Comput Appl Math 2020;372:112723.
[24] Pang XW, Zhou YQ, Wang P, Lin WW, Chang V. An innovative neural network
approach for stock market prediction. J Supercomput 2020;76(3):2098–118.
[25] Mendel JM, Mclaren RW. Reinforcement learning control and pattern recognition
systems. In: A prelude to neural networks. 1970.
[26] Busoniu L, Babuska R, Schutter BD, Ernst D. Reinforcement learning and dynamic
programming using function approximators, first ed.. USA: CRC Press, Inc.; 2010.
[27] Pease M, Shostak R, Lamport L. Reaching agreement in the presence of faults.
J ACM 1980;27:228–34.
[28] Reiter MK. A secure group membership protocol. IEEE Trans Softw Eng
1996;22(1):P.31–42.
[29] Sutton R, Barto A. Reinforcement learning: An introduction (adaptive
computation and machine learning). 1998.
[30] Lee D, Seo H, Jung MW. Neural basis of reinforcement learning and decision
making. Ann Rev Neuroence 2012;35(1):287.
[31] Sutton, Richard S. Learning to predict by the methods of temporal differences.
Mach Learn 1988;3(1):9–44.
[32] Werbos P. Approximate dynamic programming for real-time control and neural
modeling. 1992.
[33] Wang S, Guo CX, Feng B, Zhang H, Du ZD. Application of blockchain
technology in power system: Prospects and ideas. Autom Electr Power Syst
2020;44(11):10–24.
[34] Suhail S, Hussain R, Khan A, Hong CS. Orchestrating product provenance
story: When IOTA ecosystem meets electronics supply chain space. Comput
Ind 2020;123:103334. http://dx.doi.org/10.1016/j.compind.2020.103334, https:
//www.sciencedirect.com/science/article/pii/S0166361520305686.
[35] Suhail S, Hussain R, Jurdak R, Hong CS. Trustworthy digital twins in the
industrial internet of things with blockchain. 2020, arXiv:2010.12168.