IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS, VOL. 19, NO.
3, MARCH 2020
Adaptive Resource Allocation in Future Wireless
Networks With Blockchain and Mobile
Edge Computing
Fengxian Guo , F. Richard Yu , Fellow, IEEE, Heli Zhang , Hong Ji , Senior Member, IEEE,
Mengting Liu , and Victor C. M. Leung , Fellow, IEEE
Abstract— In this paper, we present a blockchain-based mobile
edge computing (B-MEC) framework for adaptive resource allo-
cation and computation offloading in future wireless networks,
where the blockchain works as an overlaid system to provide
management and control functions. In this framework, how to
reach a consensus between the nodes while simultaneously guar-
anteeing the performance of both MEC and blockchain systems
is a major challenge. Meanwhile, resource allocation, block size,
and the number of consecutive blocks produced by each producer
are critical to the performance of B-MEC. Therefore, an adaptive
resource allocation and block generation scheme is proposed.
To improve the throughput of the overlaid blockchain system
and the quality of services (QoS) of the users in the underlaid
MEC system, spectrum allocation, size of the blocks, and number
of producing blocks for each producer are formulated as a joint
optimization problem, where the time-varying wireless links and
computation capacity of the MEC servers are considered. Since
this problem is intractable using traditional methods, we resort
to the deep reinforcement learning approach. Simulation results
show the effectiveness of the proposed approach by comparing
with other baseline methods.
Index Terms— Mobile edge computing, computation offloading,
blockchain, deep reinforcement learning.
I. I NTRODUCTION
T HE progressive miniaturization of hardware is enabling
the massive deployment of smart mobile devices [1].
Manuscript received January 6, 2019; revised May 27, 2019, September 4,
2019, and November 11, 2019; accepted November 12, 2019. Date of
publication December 9, 2019; date of current version March 10, 2020. This
work was supported in part by the National Natural Science Foundation of
China under Grant 61671088 and Grant 61771070, in part by the Beijing
University of Posts and Telecommunications (BUPT) Excellent Ph.D. Students
Foundation under Grant CX2018201, and in part by the Canadian Natural
Sciences and Engineering Research Council under Grant RGPIN-2019-06348.
The associate editor coordinating the review of this article and approving it
for publication was L. Le. (Corresponding author: Hong Ji.)
F. Guo, H. Zhang, and H. Ji are with the Key Laboratory of Universal
Wireless Communications, Ministry of Education, Beijing University of
Posts and Telecommunications, Beijing 100876, China (e-mail: fengxianguo@
bupt.edu.cn; zhangheli@bupt.edu.cn; jihong@bupt.edu.cn).
F. R. Yu is with the Department of Systems and Computer
Engineering, Carleton University, Ottawa, ON K1S 5B6, Canada (e-mail:
richard.yu@carleton.ca).
M. Liu is with the Beijing Key Laboratory of Space-ground Interconnec-
tion and Convergence, Beijing University of Posts and Telecommunications,
Beijing 100876, China (e-mail: liumengting@bupt.edu.cn).
V. C. M. Leung is with the College of Computer Science and Software
Engineering, Shenzhen University, Shenzhen 518060, China, and also with the
Department of Electrical and Computer Engineering, The University of British
Columbia, Vancouver, BC V6T 1Z4, Canada (e-mail: vleung@ieee.org).
Color versions of one or more of the figures in this article are available
online at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/TWC.2019.2956519
1536-1276 © 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See https://www.ieee.org/publications/rights/index.html for more information.
1689
Meanwhile, new applications are developing in the directions
of Internet of things (IoT), Internet of vehicles (IoV), e-
healthcare, tactile Internet and so on. However, the deployment
of these applications is restricted by the energy, memory size,
computation resources of mobile devices [2]. These emerging
applications with requirements in terms of intensive compu-
tational capacity and sensitive latency can rely on advanced
improved wireless technologies and computation offloading.
Future wireless networks are required to not only support
massive wireless access but also offer the provisioning of
computation offloading for mobile users.
To meet the demands of mobile users, future wireless
networks will become more heterogeneous and dense. In
the growth of more capable wireless networks, the scarcity
of spectrum is always an impediment along the evolution
of cellular networks from the first generation (1G) to the
upcoming fifth generation (5G) [3]. One reason is the binary
quality of the current spectrum access approach, i.e., licensed
and un-licensed, which is an intentional set of policy choices.
To improve the spectrum efficiency, dynamic spectrum access
becomes the norm. However, with an unprecedented level of
network densification in the future, the spectrum management
is of high complexity. Thus, smarter and more decentralized
dynamic spectrum access techniques are preferred.
In future communication networks, the edge clouds will be
deployed in the heterogeneous network and able to provide
computation offloading services to users [4]. One of the
promising paradigms is mobile edge computing (MEC) [5].
Many outstanding works have been done on computation
offloading [6]–[10], in which resource allocation, collabora-
tion, offloading strategy and pricing algorithm are investigated.
However, some problems in this distributed and distrusted
environment are failed to be considered. First, it is impractical
to deploy or collaborate all the system resources, e.g., caching,
computing, networking, due to the self-deployment nature and
coexistence of multiple radio access service providers (SPs)
and edge cloud vendors, which is precisely required by the
logical system of traditional MEC. Second, there is no trusted
entity in the system to audit the computation offloading
process or ensure the proper and surefire payments to the
SPs and edge cloud vendors. Third, privacy is often cited
as one of the key concerns in cloud adoption, especially
when sensitive or personal information is outsourced to the
edge cloud vendors. Few cloud SPs can be fully trusted
1690 IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS, VOL. 19, NO. 3, MARCH 2020

by end-users [11]. Hence, new expectations are set for a

decentralized, self-organized, trusted computation offloading
system.
As the core technology behind Bitcoin and Ethereum,
blockchain has ben gained popularity in academia and indus-
try [12], which is a secured, shared and distributed ledger
in essence. It allows two parties in a peer-to-peer network
to communicate and exchange resources where the decisions
are made distributedly by the majority rather than a single
centralized one [13]. Thanks to the decentralization and other
inspiring features of blockchain, it is considered as a candidate
technology to establish a secure and self-organized MEC
ecosystem for future wireless networks.
To address these challenges, in this paper, we propose
a novel blockchain-based MEC framework (B-MEC) for
resource allocation in future wireless networks with MEC.
A blockchain is a kind of distributed ledger without any
centralized trusted auditors [14], which is public immutable
and append-only [15]. Attracted by the fantastic characteristics
of blockchains in terms of decentralization, anonymity and
trust, researchers have developed significant research interests Fig. 1. Blockchain-enabled MEC in the heterogeneous wireless networks.
in blockchains, e.g., resource allocation for [16], [17], secure
data storage and sharing in vehicular edge networks [18],
security in IoT [19], [20], and electric vehicle networks [21]. blockchains and MEC. Section III introduces the system
In this paper, we focus on the adaptive resource allocation model, followed by the consensus protocol in Section IV. The
issue in future wireless networks with blockchains and MEC. performance of MEC and blockchain system is presented in
Our contributions are summarized as follows: Section V. In Section VI, the main problem is formulated.
• We develop a B-MEC framework for adaptive resource
It is solved with a novel DRL approach in Section VII.
allocation and computation offloading in future wire- Section VIII presents and discusses the simulation results.
less networks, considering the issues arose from the Finally, conclusion and future works are given in Section IX.
property of heterogeneous wireless networks and MEC.
This framework deploys a consensus protocol based on II. S YSTEM D ESCRIPTION
practical Byzantine fault tolerance (PBFT) and delegated In this section, we first describe the framework of future
proof of stake (DPoS). wireless networks with blockchains and MEC, in which some
• The details and theoretical analysis of the consensus pro- concepts are included. Then some challenges in this frame-
tocol are presented, where the computation task execution work are discussed.
is self-organized by smart contracts. The performance of
the blockchain system is given, i.e., throughput, time to
A. Future Wireless Networks With Blockchain and MEC
finality, decentralization, and security.
• By jointly considering the computation task execution in As shown in Fig. 1, there are three layers in this frame-
smart contracts and blockchain consensus maintenance, work, i.e., users, edge networks (i.e., heterogeneous wireless
we formulate the spectrum allocation, block size, number networks with MEC), and blockchain. Before describing this
of consecutive blocks produced by the block producers as framework, some concepts are illustrated as follows.
an optimization problem, which is described as a markov • Clients: mobile users in this system, which submit
decision process (MDP) by defining state space, action offloading requests to the B-MEC system.
space, and reward function. The goal of the main problem • Replicas or block producers: BSs with MEC servers,
is to optimize the performance of the joint MEC and which are selected from the BSs and provide offloading
blockchain system. services by using blockchain technology.
• To handle the high dynamics of this system, we propose • Primary node: the one selected from all replicas, which
to solve this problem with a novel deep reinforcement is authorized to produce blocks at a certain time period.
learning (DRL) approach, with double-dueling deep Q • Backup nodes or validators: the block producers except
network (DQN). Google TenserFlow is used to implement the primary node, which play the role of validators.
the double-dueling DQN. • Transactions: the offloading requests generated by the
• Simulation results show the effectiveness of the proposed users, which are handled by the block producers.
approach with various parameters by comparing with In this paper, the blockchain serves as an overlaid sys-
other baseline algorithms. tem to provide management and control functions to the
The rest of this paper is organized as follows. Section II underlaid MEC system. With the blockchain, data delivery
describes the framework of future wireless networks with and computation task execution are self-organized by smart
GUO et al.: ADAPTIVE RESOURCE ALLOCATION IN FUTURE WIRELESS NETWORKS WITH BLOCKCHAIN AND MEC
contracts [22], which can provide an incentive to ensure the
interests of different parties in a trust-less computing market.
The transaction records the computation offloading requests
from the mobile users. These transaction records are jointly
approved by the consensus nodes selected from the blockchain
nodes, and then digitally stored in all nodes’ local blockchain
replicas. Since the mobile devices usually have limited storage,
the full ledger is stored only in the blockchain nodes, i.e., the
edge servers in this paper. Fortunately, the blockchain provides
publicly accessible records to the users about the transactions
in this network, which are inevitably encrypted to provide
privacy guarantee.
In this network, the offloading requests are listened by
all the BSs. When it is its own turn to produce blocks,
the primary node validates and processes the transactions with
smart contracts. Then the computing results are sent back
to the users and the transactions are packaged into a new
block. After that, it comes to a consensus procedure, which
will be described in details in section IV. After the consensus
is reached among the nodes, the block is appended to the
blockchain, which means the block reaches finality.
B. Key Challenges of Computation Offloading in This
Framework
Considering the time-varying of the wireless channels and
spectrum efficiency, an adaptive spectrum allocation scheme
is needed. Although the blockchain maintains non-repudiation
and non-tampering properties, it is difficult to use it in wireless
networks directly, since the currently employed blockchain
technology doesn’t take into the limitation of resources and
the time-varying features of wireless networks into consid-
eration. Hence, an adaptive consensus protocol is required
for a wireless blockchain network. Furthermore, since the
blockchain provides management and control functions to
the MEC system, the performance of the blockchain is vital
to ensure the quality of experience (QoE) of the mobile
users. Hence, we need to guarantee the performance of the
blockchain, i.e., throughput, decentralization, time to finality,
and security, while concurrently ensuring the performance of
MEC, e.g., quality of services (QoS) of the users.
III. S YSTEM M ODEL
In this section, we introduce the system model in this work.
We first present the network model, then the communication
model and computation model are introduced. In this paper,
a discrete-time slotted system is applied, where time is parti-
tioned into discrete time periods T = {1, . . . , t, . . . , T }, and
each time period t has a constant duration Ṫ .
A. Network Model
As seen in Fig. 1, we assume that there are I BSs and
N block producers in the heterogeneous wireless networks.
Especially, the block producers are selected from the BSs,
which are indexed by B = {B1 , . . . , Bn , . . . , BN }. Each of
them is equipped with a MEC server, the computation capacity
of which is denoted by FBn (in Hz). Also, there are M
1691
users located around the BSs, each of which has a number of
computational tasks (e.g., online games, navigation, VR, health
monitor and so on) to be completed. The users are denoted by
U = {U1 , . . . , Um , . . . , UM }. To complete the tasks, the users
are supposed to choose to offload the tasks to the BSs. In this
paper, we don’t consider the local execution on the mobile
devices.
In the blockchain system, the consensus protocol adopts
the idea of both DPoS and PBFT. As noted, the selection of
block producers is important in DPoS, which has been well
studied in existing works [23]. Thus, we didn’t consider it
in this paper. PBFT provides safety and liveness while there
are less than (N − 1)/3 faulty nodes. In the B-MEC system,
assume that the N block producers take turns to produce K
blocks within interval Ṫ (in seconds), the size of which is SB
(in bits). K varies across different time periods to account for
the time-varying characteristics of wireless networks.
As noted, computation offloading in this system includes
four phases, 1) submitting the offloading requests to the
blockchain system, 2) executing smart contracts by the block
producers, 3) sending back the results to the users, 4) reaching
consensus among the block producers. As noted, it involves a
communication model and a computation model, which will
be presented next.
B. Communication Model
In the B-MEC system, it involves two types of wireless
links: 1) data transmission from the users to the BSs, 2) mes-
sage delivery among the BSs.
To capture the characteristics of the time-varying wireless
channels in this system, we resort to finite state Markov
channel (FSMC) model. In this paper, we define the channel
state according to the received signal-to-noise ratio (SNR). The
amplitude of the SNR of the received signals are partitioned
into L non-overlapping levels, and there are L + 1 correspond-
ing thresholds {hl , l = 0, 1, 2, . . . , L}. Generally, h0 and hL
are known, which denote the minimum and maximum mea-
sured values of the SNR. The available state set of the finite
markov chain is given by H = {H1 , H2 , . . . , Hl , . . . , HL }.
Let γ denote the channel state, the realization of which at time
period t is denoted by Γ(t). In details, we have Γ(t) = Hl ,
when there is γ ∈ {hl−1 , hl ).
We assume the channel is block-fading, where the received
SNR is constant during one time period, but evolves between
different time periods according to a set of Markov transition
probabilities. Let pl,g = P r{Γt+1 = Hl |Γt = Hg } be
the transition probability from state Hg to Hl , where l, g ∈
{1, . . . , L}. Then the L × L transition probability matrix can
be defined as P = [pl,g ]L×L .
Based on the presented FSMC model, let ΓUm ,Bn be the
SNR from user Um to BS Bn , and ΓBn ,Bn be the SNR from
BS Bn to BS Bn .
Multicast OFDMA [24] is considered in this paper, where
a sub-channel is used by one transmitter to multicast the
same message to several receivers. Assume that there exists
E sub-channels with the same bandwidth W0 over the whole
spectrum bandwidth W , where there is E ≥ M +N . As noted,
1692 IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS, VOL. 19, NO. 3, MARCH 2020
each user and BS have the chance to be a transmitter of one
multicast group. Assume that the user Um is allocated with
WUm sub-channels, and BS Bn is assigned with WBn sub-
channels. Due to the limited wireless spectrum, the following
constraint should be met
 
WUm W0 + WWBn W0 ≤ W, (1)
Um ∈U Bn ∈B
which means the spectrum allocated to all users and BSs
should not exceed the total bandwidth in this system.
Hence, the received data rate of user Um at BS Bn is
RUm ,Bn = WUm W0 log(1 + ΓUm ,Bn ). (2)
The received data rate of BS Bn to Bn is expressed by
RBn ,Bn = WBn W0 log(1 + ΓBn ,Bn ). (3)
Since each BS receives the broadcasting messages from all
the users and the other BSs, the backhaul data rate should not
exceed the received data rate. Thus, we have
 
RUm ,Bn + RBn ,Bn ≥ RBn ,Bn . (4)
Um ∈U n ∈B/{n}
C. Computation Model
In this system, the blocks are generated continuously with-
out waiting for confirmation. This results in the fact that there
may be several un-confirmed blocks existing in this system
at the same time, on which it needs to be reached consensus.
Hence, each BS needs to process different computation tasks
from different blocks, i.e., executing smart contracts, gener-
ating and verifying the signatures, generating and verifying
MACs. After one BS receives a message, it contains certain
computation tasks. Assume that the computation task of mes-
sage s is denoted by a 2-tuple Is = {ds , fs }, where ds denotes
data size of the message s, while fs is the total CPU cycles
to complete this task. Particularly, define s ∈ {p, v}, where p
denotes producing while v denotes verifying.
Denote the computation capacity assigned to one message
by Fs (in Hz or CPU cycles per second). In this system,
one BS may process different messages at the same time. For
example, the primary node needs to produce blocks while ver-
ifying the other messages from the other BSs. It results in the
fact that we don’t exactly know the computational resources
for each message at the next time instant. Hence, we model
the evolution of Fs as a finite state Markov process. The
computation capacity is partitioned into Y non-overlapping
levels, which is expressed as C = {C1 , . . . , CY }. The real-
ization of Fs at time period t is Ψs (t) = Cy , when there is
Fs ∈ [cy−1 , cy ]. There are Cy ∈ C and c0 = 0 < c1 < . . . <
cY = Fmax , where Fmax denotes the maximum computation
capacity of the processing BS.
For simplicity, we assume that the computation capacity
allocated to message s is constant in one specific time period,
but it evolves to the next state according to the transition
probability. Let qx,y denote the probability that Ψs (t) moves
from state x to state y at time period t. The Y × Y state
transition probability matrix is defined as Q = [qx,y ]Y ×Y ,
where we have qx,y = P r(Ψs (t + 1) = y|Ψs (t) = x) and x,
y ∈ C.
Based on the above model, define the computation resources
assigned from BS Bn to message s is ΨBn ,s (t). The execution
time for completing task Is at BS Bn can be calculated by
fs
TBn ,s = . (5)
ΨBn ,s (t)
IV. A DAPTIVE C ONSENSUS P ROTOCOL
In this section, we first present the adaptive consensus
protocol proposed in this paper, which is based on PBFT.
A. Overview of the Consensus Protocol
After the new block is generated, it is then broadcasted
to the validators. When the validators verify the new block,
the signatures of them are added to the block. After reaching
consensus, the new block will be appended to the blockchain.
A signed block contains the block number, block size,
block header, signed block summary and transactions. In the
block header, it contains the version of the block, the hash
of the previous block, a timestamp (i.e., the creation time
of the new block), Markle root of all the transactions, and
IDs of the validators. Signed block summary contains the
summary of the transactions, e.g., the number of the included
transactions, the structure of the transactions. Worth noting,
it only contains the summary of the transactions, while the
integral transactions are attached at the end of the block. The
transactions mainly contain the transaction number, ID, scope,
smart contract, signature and MAC. Here the smart contract is
used to complete the computation tasks offloaded by the users,
and the required transaction data represents what is required
to complete the computation tasks, e.g., associated program
codes or captured images.
B. Theoretical Analysis
In the following, we give detailed steps and theoretical
analysis of the consensus protocol. Compared with traditional
systems without blockchain, this blockchain system also needs
to validate the signatures and MACs except for executing
the offloaded tasks, which can be treated as a computation
overhead to execute the transactions. Since smart contracts
are in charge of the execution of offloaded tasks, we assume
that the computation of the offloaded tasks is included in the
execution of smart contracts. It is assumed that executing the
smart contract for one transaction, generating or verifying one
signature, generating or verifying one MAC require α, β, and
θ CPU cycles, respectively. Based on PBFT, the consensus
protocol consists of five steps as the following.
1) Request: In one time period, the users submit offloading
requests to the replicas, and the requests are as transactions
stored in the pending pool. The primary node, which is one
of the replicas, selects its’ preferred transactions. For one
transaction, the primary node will first verify its signature.
If valid, it then verifies the transaction’s MAC. If still valid,
the primary node will execute smart contracts for this transac-
tion. After verifying the preferred transactions, all transactions,
computing results and other important information will be
GUO et al.: ADAPTIVE RESOURCE ALLOCATION IN FUTURE WIRELESS NETWORKS WITH BLOCKCHAIN AND MEC 1693

packaged into the new block. This procedure occurs within matching prepare messages from the other replicas, it enters
the block interval Ṫ /K. the next step.
Assume that the average size of one transaction is denoted In this phase, the transmission cost is caused by sending the
by . In this phase, the transmission latency in this phase prepare message to all other replicas, which can be calculated
tr
Treq can be expressed by by
tr  SB
Treq = max { }. (6) tr
Tpre = max { }. (10)
Um ∈U RUm ,Bp 
Bn ,Bn ∈B/{Bp },Bn =Bn RBn ,Bn
where RUm ,Bp denotes the transmission rate from user Um to For the computation cost, the primary node needs to verify
the primary node Bp . 2f signatures and MACs from the other replicas, which can
Considering the size of one block, the maximum number be expressed by Δpre,Bp = 2f (β + θ). For the other backup
of transactions that can be included in a block is SB /. nodes, each needs to generate a signature and N − 1 MACs
Uncivil execution is assumed that a g fraction of transactions for the prepare message. Then 2f signatures and MACs are
submitted by the clients are correct [25]. In this phase, required to be validated. Hence, the computation cost at the
the primary node needs to verify the signatures and MACs backup nodes Bn (= Bp ) can be given by Δpre,Bn = β +
for SB /g transactions and it also needs to execute smart (N − 1)θ + 2f (β + θ). Hence, the computation latency in this
contracts for SB / transactions. Hence, the computation cost phase is
at the primary node is Δreq,Bp = SB g (β+θ)
+ S Bα
. Thus, Δpre,Bn
c
the computation delay is Tpre = max { }. (11)
Bn ∈B FBn ,v
c Δreq,Bp
Treq = . (7) 4) Commit: Following receipt 2f matching prepare mes-
FBp ,p
sages from the other replicas that are consistent with the
As noted, there is no computation cost at the backup nodes. pre-prepare message, each replica sends a commit message
2) Pre-Prepare: After producing the new block, the primary to all the others, which includes the ID and signature of the
node multicasts the signed block along with a pre-prepare replica. Once upon receipt 2f matching commit messages,
message to all the backup nodes for validation, where the it enters the next step.
pre-prepare message contains the ID, signature of the primary To deliver the commit messages, the transmission latency
node and hashed result of the new block. Since the smart con- can be expressed by
tract is in charge of the execution of the offloaded computation SB
tasks, the backup nodes need to make sure that the offloading Tctr = max { }. (12)
Bn ,Bn ∈B,Bn =Bn RBn ,Bn
tasks are actually executed by the primary node except for
validating the identities and the economic parts. In this case, In this phase, each replica needs to generate 1 signature and
the intuitive method is to implement the smart contracts and N −1 MACs to form the commit messages. After receiving the
compare the computation results. commit messages, each replica needs to verify 2f signatures
Hence, after receiving the pre-prepare message and the new and MACs. Hence, the computation cost at each replica is
block, the backup nodes first verify the signature and MAC of Δc,Bn = β + (N − 1)θ + 2f (β + θ). Hence, the computation
the block, then the signatures and MACs of the transactions. latency in this phase is
Different from the work in [20], then smart contracts are Δc,Bn
executed by the backup nodes to validate the transactions. Tcc = max { }. (13)
Bn ∈B FBn ,v
If the pre-prepare message is accepted by some backup node,
5) Reply: After collecting 2f matching commit messages,
it enters the next step.
tr the new block becomes a valid one and it will be appended to
In this phase, the transmission latency Tprep can be calcu-
the blockchain. A reply message will be delivered, in which
lated by
the signature, ID, computation result for the offloading task
tr SB
Tprep = max { }. (8) are included. Different from the original BFT protocol [25],
Bn ∈B/{Bp } RBp ,Bn the reply message is delivered to the primary node, instead of
As noted, the primary node needs to generate one signature the clients, due to the mobile devices’ limited memory size.
and N − 1 MACs in this phase, which is given by Δprep,Bp = In this phase, the transmission cost is
β + (N − 1)θ. The computation cost at the backup nodes is SB
Δprep,Bn = β + θ + SB (α + β + θ), where there is Bn = Bp . Trtr = max { }. (14)
Bn ∈B/{Bp } RBn ,Bp
Hence, the computation latency in this phase is
For the computation cost, each backup node needs to
c Δprep,Bn
Tprep = max { }. (9) generate SB / signatures, and SB / MACs for the primary
Bn ∈B FBn ,v node, which can be given by Δr,Bn = SB (β + θ). For the
3) Prepare: After verifying the new block, each backup primary node, it needs to verify 2f signatures and MACs,
node sends a prepare message to all the other replicas, in which the computation cost of which is given by Δr,Bp = 2f (β +θ).
the replica ID and the signature are contained. Each replica Hence, the computation latency in this phase is
will check the prepare message to make sure that it is Δr,Bn
consistent with the pre-prepare message. Once upon receipt 2f Trc = max { }. (15)
Bn ∈B FBn ,v
1694 IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS, VOL. 19, NO. 3, MARCH 2020
V. P ERFORMANCE A NALYSIS
In this section, we give details of the performance of the
MEC system and blockchain system. For the MEC system,
the QoS of the users in terms of delay is given, which
is the time from submitting the requests to receiving the
results. In the blockchain system, the most important criterions
to measure the system performance are throughput, time to
finality, decentralization, and security. To address the four-way
trade-off issue, the four properties will be presented in this
section.
A. Performance of MEC
To measure the QoS of the users, the delay experienced
by the users is introduced, which consists of three parts,
submitting the requests to the BS, executing the offloading
tasks (executing smart contracts), sending back the results
to the users. As analyzed in Section IV.B, the transmission
tr
latency Treq to submitting the requests is as expression (6).
As analyzed in Section IV.B, the primary node first verifies
the signature and MAC of the requests, then executes the
offloading requests. The computation cost for one offloading
request is α + β + θ. The processing latency consists of two
parts, queuing delay and executing delay. For each transaction,
the executing delay is
α+β+θ
Te = . (16)
FBp ,p
Hence, the average queuing delay can be expressed by
1 SB
Tq = ( − 1)T e . (17)
2 
In this work, we don’t consider the sending back procedure
as in [9], since the size of the output may be much smaller than
the input data, which corresponds to many practical scenarios,
such as virus detection, face recognition, and video analysis.
Hence, the average delay experienced by the users are
tr
TU = Treq + T e + T q. (18)
B. Performance of the Blockchain System
1) Throughput: The throughput of the blockchain system
can be measured by the number of transactions that can be
processed successfully in unit time, which is related to two
procedures, i.e., block generation, and consensus reaching.
When producing a block, it is limited by the block size and
the processing capacity of the primary node. Considering the
block size, the transactions that can be included into the block
per time is
SB K
Ξ(SB , K) = . (19)
Ṫ
The computation cost of producing one block is shown in
Section IV.B. We assume that the computation resources of the
primary node that are assigned to produce blocks is Fp,p Hz.
Considering the limited computation resources, the following
constraint should be met
SB (β + θ)/g + α Ṫ
≤ , (20)
 Fp,p K block producers, we consider the number of blocks that each
the left of which denotes the processing time to produce a
block.
Since the primary node produces K blocks continuously,
the last several blocks may be ignored due to the propagation
delay to the next primary node. We assume the transmission
data rate from the current primary node to the next one is
Rp,p+1 . Hence, the number of ignored blocks can be calculated
by
SB /Rp,p+1
IB(SB , K) =  − 1. (21)
Ṫ /K
As noted, there is IB ≤ K, which is apparent that the system
won’t missing more blocks than produced ones.
The throughput of the consensus protocol can be expressed
by
(K − IB) SB SB K
Υ(SB , K, W) = Ξ = (K − +1), (22)
K Ṫ Rp,p Ṫ
where W = {WUm , WB,n , Um ∈ U, Bn ∈ B} denotes the
spectrum allocation profile. Υ denotes the number of transac-
tions that can be included into the blocks and transmitted to
the next primary node successfully.
2) Time to Finality/Confirmation Latency: To guarantee the
security of the transactions, it is essential to prevent the
transactions to be arbitrarily changed or reversed. Time to
finality is the time that the transactions can’t be revoked
once committed to the blockchain, which is important to
some real-time applications. Longer delay frustrates users and
makes applications built on a blockchain less competitive with
existing non-blockchain alternatives.
Time to finality T f includes two parts, time for propagation
T and time for computation T c .
p
T f = T p + T c. (23)
Assume that each transmission procedure should be done
within a timeout τtr . As discussed in section IV.B, the prop-
agation time can be calculated by
T p = ttr tr tr tr tr
req + tprep + tpre + tc + tr
tr tr tr
= min{Treq , τtr } + min{Tprep , τtr } + min{Tpre , τtr }
+ min{Tctr , τtr } + min{Trtr , τtr }. (24)
In this consensus protocol, it involves five procedures. The
computation cost and computation latency for each procedure
are shown in section IV.B. We assume that each message
should be processed within a timeout τc . Thus, for computation
latency T c , we have
T c = tcreq + tcprep + tcpre + tcc + tcr
c c c
= min{Treq , τc } + min{Tprep , τc } + min{Tpre , τc }
c c
+ min{Tc , τc } + min{Tr , τc }. (25)
3) Decentralization: To characterize the decentralization of
the blockchain systems, we resort to Gini coefficient, which
is often used as a gauge of economic inequality, measuring
income distribution or wealth distribution among a popula-
tion [26]. The definition to measure inequality is based on
Lorenz curve [27]. Focusing on the decentralization of the
GUO et al.: ADAPTIVE RESOURCE ALLOCATION IN FUTURE WIRELESS NETWORKS WITH BLOCKCHAIN AND MEC
replica produces over time, the set of which is denoted by
K = {K(1), K(2), . . . , K(T )}. Hence, the Gini coefficient of
the distribution among K is expressed by
 
t∈T  |K(t) − K(t )|
G(K) =  t ∈T
2 t∈T t ∈B K(t)
  
t∈T t ∈T |K(t) − K(t )|
=  . (26)
2N t∈T K(t)
Note that there is G(K) ∈ [0, 1]. The smaller the value of
the Gini coefficient is, the more decentralized the blockchain
system is. A Gini coefficient of zero expresses perfect equality,
where all values in K are the same. It means every replica
produces the same number of blocks in a round. A Gini
coefficient of 1 expresses maximal inequality among values.
For example, only one replica produces several blocks, but the
other replicas don’t have a chance or can’t produce one block.
In this case, the blockchain system becomes totally centralized,
which violates the idea of the blockchain, a distributed ledger.
To ensure the decentralization of the blockchain system,
we have the following constraint
G(K) ≤ η, (27)
where η ∈ [0, 1] denotes the thresholds of decentralization in
terms of K.
4) Security: To guarantee the security of the transactions,
it is essential to prevent the transactions to be arbitrarily
changed or reversed. As such, finality is vital when designing a
blockchain consensus protocol. In PBFT-based consensus pro-
tocol, absolute finality can be provided when a 2/3 fraction of
nodes are honest. So the number of loyal nodes is essential to
guarantee the security of the consensus protocol. To guarantee
the security of the system, the following constraint should be
met
N −1
f≤ . (28)
3
In another word, to prevent from revoking or modifying a
transaction, the number of malicious nodes should not exceed
(N − 1)/3. In this paper, we don’t consider the security
problem of the consensus protocol. In another word, the above
condition is assumed to be satisfied already.
VI. P ROBLEM F ORMULATION
In order to improve the throughput of this system, we need
to jointly optimize spectrum allocation, block size, number of
blocks produced by each replica. Since it is intractable to solve
this problem with the traditional methods, we resort to DRL,
which will be introduced in the next section. To implement
the approach, we formulate the joint optimization problem as
a MDP, where the state, action, and reward function are defined
as follows.
A. State
Let S = {s(t), t ∈ T } be the system state space, where s(t)
denotes the state at time period t. Here st evolves across T .
The network state consists of the SNR between the users and
the BSs, SNR between different BSs, computing resources
1695
assigned by the BSs to different messages, primary node ID.
Hence, the network state s(t) at time period t is expressed by
s(t) = {ΓU1 (t), . . . , ΓUm (t), . . . , ΓUM (t);
ΓB1 (t), . . . , ΓBn (t), . . . , ΓBN (t);
ΨB1 (t), . . . , ΨBn (t), . . . , ΨBN (t); Bp (t)}, (29)
where there are ΓUm (t) = {ΓUm ,Bn (t), Bn ∈ B},
ΓBn (t) = {ΓBn ,Bn (t), Bn ∈ B, Bn = Bn }, and ΨBn (t) =
{ΨBn ,s (t)}. In this paper, the primary node in each time
period is known as a priori.
B. Action
In this paper, we focus on spectrum allocation, block size,
number of successive blocks produced by one block producer.
Let A = {A(t), t ∈ T } be the system action space. Here A(t)
denotes the action at time period t, which can be expressed
by
A(t) = {WU1 (t), . . . , WUm (t), . . . , WUM (t);
WB1 (t), . . . , WBn (t), . . . , WBN (t);
SB (t); K(t)}, (30)
where the first two rows denote the spectrum allocation indica-
tors for the users and BSs, respectively. Particularly, we have
WUm ∈ {1, . . . , E} and WBn ∈ {1, . . . , E}. Considering the
limited wireless resources, the capacity constraint should be
met, which is shown in expression (1). Here, SB (t) denotes
the block size at time period t, and K(t) denotes the number
of successive blocks at time period t. Especially, the replicas
take turns to produce blocks, there is only one primary node
in a certain time period. For the primary node Bp (t) at time
period t, it produces K(t) blocks. To simplify this problem,
we discretize the action space, where SB and K are selected
in the set SB and K.
C. Reward Function
In this paper, we aim to maximize the performance of the
joint MEC and blockchain system by making decisions on the
action space. The reward function is designed to be
max R(SB , K, W)
SB ,K,W
s.t. C1 : Tkf (t) ≤ Tmax , ∀k ∈ K(t)
C2 : G(K) ≤ η
 
C3 : WUm + WWBn ≤ E
Um ∈U Bn ∈B
 
C4 : RUm ,Bn + RBn ,Bn ≥ RBn ,Bn
Um ∈U n ∈B/{n}
(31)
T 
where R(SB , K, W) = t =t t −t r(t) denotes the long term
reward over the time periods T . Here ∈ [0, 1) is the discount
rate, which indicates the weight of the future reward. For
fixed t, the bigger is, the more influence the future reward

r(t) has. As noted, with fixed , t −t approaches zero when

t − t is large enough, which means that the future reward has
less impact on the long term reward with the time going on.
1696 IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS, VOL. 19, NO. 3, MARCH 2020

In the proposed problem, Tkf (t) in constraint C1 denotes the where y(t) is the target Q value, which can be estimated by
time to finality of the k-th block produced in time period t.
y(t) = r(t)+ max Q(s(t+1), a(t+1); θ−(t+1)). (35)
Constraints C1 and C2 represents the limitations on time to a(t+1)
finality, and decentralization, respectively. C3 denotes the allo-
Here, the target DQN is updated every G steps,
cated sub-channels should not exceed the total system wireless
i.e., θ− (t) = θt−G .
resources. C4 denotes the backhaul capacity constraint. As
3) Beyond DRL: To improve the performance of DRL, two
noted, constraints C1 ∼ C4 may not be met, which means the
important techniques, double DQN, and dueling DQN, are
whole system may have a low system performance. Adopting
applied in this work, which will be described next.
the idea of penalty function, we define the immediate reward
a) Double DQN: To handle the problem of overestima-
r(t) as
⎧ tions of Q values, double DQN is proposed by Hado van
⎨ϑ Υ(t) + ϑ 1 , when C1 ∼ C4 is satisfied, Hasselt [30], the idea of which is to decompose the selection
B M
r(t) = TU from the estimation for the actions. In mathematics, it can be
⎩0, otherwise. expressed by
(32)
y DoubleDQN = r + Q(s , argmaxQ(s , a; θ); θ− ), (36)
where ϑB and ϑM ∈ [0, 1] are the weights corresponding to
which selects the actions according to online weights θ, while
the blockchain system and MEC system. And there is ϑB +
the estimation is based on the current values. This simple trick
ϑM = 1. Note that the weights can be dynamic, which indicate
can help yield more accurate estimations, thus improving the
the dynamic preference on these two systems. For ease of
performance of DRL.
modeling, assume that the weights remain stationary within
b) Dueling DQN: Motivated by the fact that not every
one time period, while can be changed over different periods.
action affects the state when in some state, dueling DQN is
VII. P ROPOSED L EARNING A PPROACH proposed [31], the idea of which is to decompose the Q value
into two parts, the value of being in that state V (s) and the
In this section, we first introduce necessary background
advantage of taking that action at that state A(s, a). The idea
related to DRL, then present the approach to solve the con-
can be expressed by
sidered problem.
Q(s, a) = A(s, a) + V (s). (37)
A. DRL Background
In this case, dueling DQN can intuitively learn which state
1) RL: RL is a branch of machine learning, in which the is more valuable without learning the effect of each action at
agent learns the optimal policy by interacting with an unknown that state. By doing so, it can help find more reliable Q values
environment to maximize the expected long term reward [28]. for each action and accelerate the training process.
A RL agent can be modeled as a MDP. The way that an agent
acts in a MDP framework is as follows. Given the state s(t) ∈
B. Proposed Algorithm
S in environment X, the agent takes an action a(t) from the
legal set A at each time step. To solve the proposed problem, an offline DRL-based
After taking the action from the given state, it enters the approach is proposed. In this approach, double-dueling DQN
next state s(t + 1) according to the state transition probability model is first trained to learn the optimal policies in an
P (s(t + 1)|s(t), a(t)). At the same time, it receives an instant offline way. After the model is trained, it can be used by the
reward r(t). In RL, the objective is defined as the expected B-MEC system to jointly allocate the wireless resources and
long term reward, which is decide the size of the blocks and the number of consecutive
T
produced blocks for each replica in an online way. In this way,
  it avoids long training time compared to the online learning
R(t) = t −t r(t), (33)
t =t
approach.
In each training step, the state information is sent to the
where ∈ [0, 1] is the discount factor on the future rewards.
Q network, and the Q network sends back the optimal action
2) DRL: Recently, many researches have shown that deep
a∗ (t) at each time step. Action selection follows the ε-greedy
learning can be combined with RL to solve problems with
policy. The transitions, i.e., experience, from all the trains
high dimensional raw data input, which is referred to DRL
are accumulated in the experience replay buffer in parallel.
[29]. In the training process of DRL, it utilizes a deep neural
A mini-batch of samples are selected from the experience
network (DNN) called DQN to derive the relationship between
replay buffer to train the Q network parameter θ. The target Q
the action-state pair and the Q function Q(s, a; θ), in which θ
network parameter θ− is updated every G steps, i.e., copying
represents the weights of the neural networks. DQN is trained
from the main Q network. The training process is shown in
by updating θ in each iteration to approximate the real Q
Algorithm 1, in which there are two points to be specified.
values. This is achieved until two improved technique are
First, the Q values are first divided into two parts, the value
applied in DQN, experience replay and the target network.
being in the state V (s) and the advantage of taking that action
Furthermore, the main DQN is trained towards the target
in that state A(s, a). In the last layer of DQN, combine these
DQN by minimizing the loss function, which is defined as
two parts into one Q value. Second, when updating the target
Loss(θ(t)) = E[(y(t) − Q(s(t), a(t); θ(t)))], (34) Q network, a learning rate α is introduced, where α ∈ [0, 1] is
GUO et al.: ADAPTIVE RESOURCE ALLOCATION IN FUTURE WIRELESS NETWORKS WITH BLOCKCHAIN AND MEC
Algorithm 1 Offline DRL-Based Performance Optimization
for B-MEC
1: Input: Maximum training episode Emax , maximum steps
Hmax in each episode, mini-batch size U , initial learning
rate α, exploration probability ε, discount rate .
2: Initialization
3: Initialize the state of the B-MEC system s1 , set ε = 1
4: Initialize the experience replay buffer
5: Initialize the main DQN with random weights with θ.
6: Initialize the target DQN with weights θ − = θ.
7: for episode = 1, . . . , Emax do
8: for t = 1, . . . , Hmax do
9: Choose a random probability p,
10: if p > ε then
11: a(t) = a∗ (t) = arg maxa Q(x, a; θ),
12: else
13: randomly select an action at = a∗ (t).
14: end if
15: Decrease exploration probability ε
16: Execute action a(t) in the system, and observe the
reward r(t) and the next state s(t + 1).
17: Store the experience (s(t), a(t), r(t), s(t + 1)) into
the experience replay buffer.
18: Sample a mini-batch of size U
19: Calculate the target Q-value through expression (36)
20: Update the main DQN by minimizing the loss L(θ)
in expression (34), and perform a gradient descent step on
L(θ) with respect to θ.
21: Every G steps, update the target DQN parameters
with learning rate α, θ− = αθ + (1 − α)θ− every G steps
22: Update the learning rate according to the optimizer
(e.g., Adam, Adagrad)
23: end for
24: end for
the weight to adjust the preference on the current and previous
learning values.
In each training step, the system state transits into a new
state according to the system transition probability after an
action is performed. And the reward can be observed based
on the reward function. After the states, actions, the reward
function, transition probabilities, and constraints of B-MEC
system are identified, the optimal policy can be learned off-
line. In order to obtain the optimal solution, the states, actions,
reward function, and constraints are identified in Section VI.
As noted, the transition probabilities and reward need to be
identified when conducting the simulation, while both of them
are not needed when carrying out the Q networks in a real
B-MEC system.
C. Complexity Analysis
Next, we analyze the computational complexity and space
complexity of the proposed DRL-based algorithm for practical
scenarios where there are tens and even hundreds of users and
BSs in a small area.
1697
Theorems 1: In practical scenarios, the computational
complexity of the proposed training DRL algorithm is
O(E (N +M) ) or O(N + M )(E−(N +M)) .
Proof of Theorem 1: To prove the above theorem and thus
analyze the complexity of the proposed algorithm, one must
consider the size of the state function of the system as
well as the action space at each state vector [32]. As such,
based on the action space definition, the system needs to
update each user and BS’s spectrum allocation indicator,
the block size and number of blocks produced by each
block producer, and, thus, its actions is also a function of
channel association vector, block size level, and block number
level.
For each state, the action of the system is a function of
channel association vector, block size level, and block number
level. Nevertheless, the number of possible channel association
of the users and BSs in the system is much more than
the number of possible block size level and block number
level. Therefore, one can focus on the number of possible
channel association of the users and BSs only for analyzing the
convergence complexity of the proposed training algorithm,
by the law of large numbers. Consequently, the computational
complexity of the proposed algorithm is O(E (N +M) ) when
the system update the channel allocation indicator of the N
block producers and M users with E sub-channels. In this
paper, we assume that the number of sub-channels is more than
the total number of users and BSs, i.e., E > N + M . Thus,
from another perspective, the computation complexity can be
also expressed by O(N + M )(E−(N +M)) . This completes the
proof. 
From Theorem 1, we can conclude that the conver-
gence speed of the proposed training algorithm is strongly
related to the state space dimension. It is of signifi-
cant importance to note here that there exists a tradeoff
between the computational complexity of the proposed DRL
training algorithm and the resulting network performance
[32]. Worth noted, the complexity of the proposed algo-
rithm can be ignored in this paper, due to the training
process, which is the most costly, is conducted in an offline
way.
Theorems 2: The space complexity of the proposed
DRL-based algorithm is O(SAHmax ), where S is the number
of states, A is the number of actions, and Hmax is the number
of steps in one episode.
Proof of Theorem 2: According to [33], space complexity
is measured by the amount of memory required to implement
the algorithm. Inferred from the work in [34], the space
complexity is related to the number of states, the number
of actions, and the number of steps per episode. In this
paper, the number of states can be expressed by S = (N +
M )(E−(N +M)) × Y N × N . The number of actions can be
calculated by A = (N + M ) × |SB | × |K|. In this paper,
the number of maximum steps in each episode is Hmax ,
as defined in Algorithm 1.
In this case, the space complexity to implement the proposed
algorithm can be expressed by O(SAHmax ) = O((N +
M )(E−(N +M)+1) × Y N × N × |SB | × |K| × Hmax ). This
completes the proof. 
1698 IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS, VOL. 19, NO. 3, MARCH 2020

TABLE I
T HE S IMULATION PARAMETERS

Fig. 2. Reward under different learning rates.

For double-dueling DQN, we use four full-connected layers

in the main network and target network. The first 3 layers have
VIII. S IMULATION R ESULTS AND D ISCUSSION 256, 256 and 128 neurons, respectively. The forth neural layer
is split into Advantage (action advantage) and Value (state
In this section, we use computer simulation to demonstrate
value) functions, which comes from the idea of dueling DQN.
the effectiveness of the proposed scheme. First, the simulation
In the last layer, the Advantage and Value functions are merged
settings are presented. Then it follows the simulation results
as the Q value. The other parameters of the proposed approach
with various parameters.
are presented in Table I.
To evaluate the effectiveness of the proposed approach,
A. Simulation Setting we first compare the performance achieved by DQN, double
We conduct our simulation on a GPU based server, which DQN, dueling DQN, and the utilized double-dueling DQN.
has four GTX 1080 TI NVIDIA GPUs, a 128G RAM, and Then we choose four comparison algorithms: 1) the proposed
an Intel Xeon CPU. The software environment is TensorFlow scheme with fixed spectrum allocation, in which the learning
1.8.0 with Python 3.6 on Ubuntu 18.04 LTS. agent only needs to determine the optimal block size and the
For the wireless network, there are 4 BSs, surrounded by number of consecutively produced blocks for each replica;
3 mobile users. Each BS is equipped with a MEC server. Since 2) the proposed scheme with fixed block size, in which
the block producer selection is not considered in this paper, the block size is set to be 1 MB; 3) the proposed scheme
all the BSs are assumed to be selected as block producers. with fixed block number, in which each block producer can
The computation resources assigned to each message are from produce a fixed number of blocks at one time period, i.e., 4;
the set {200, 500, 1000} GHz, the transition probability of 4) the existing static scheme, in which the decisions are made
which is Q = [0.7, 0.2, 0.1; 0.2, 0.1, 0.7; 0.1, 0.7, 0.2]. The through maximizing the immediate reward. Last, we choose
channel state also follows MDP. We assume three scenarios two existing algorithms in other works, the proportional fair-
with different SNR state settings, labeled as SN R1 , SN R2 , ness utility based algorithm (PFA) widely used for resource
and SN R3 . There are three states in the first scenario with allocation [36] and the random selection algorithm (RSA)
SN R1 , low (Hl = 1), medium (Hl = 7), and high (Hl = 15), in [37], where an action is randomly selected to execute in
the transition probability of which is each step.
⎡ ⎤
0.6 0.3 0.1
P1 = ⎣0.3 0.1 0.6⎦ . (38) B. Simulation Results and Discussion
0.1 0.6 0.3
The performance of double-dueling DQN based scheme
The second scenario with SN R2 has the following setting, with different learning rates is shown in Fig. 2. From this
five states {1, 3, 7, 15, 31} and transition probabilities figure, we can observe that the convergence is faster with
⎡ ⎤ the learning rate equal to 0.001 than that with learning rate
0.6 0.4 0.2 0.1 0.05
⎢0.05 0.6 0.4 0.2 0.1 ⎥ 0.0001. The learning rate adjusts the weights of the current and
⎢ ⎥
⎢
P2 = ⎢ 0.1 0.05 0.6 0.4 0.2 ⎥ previous learning value. A larger learning rate means that the
⎥. (39)
⎣ 0.2 0.1 0.05 0.6 0.4 ⎦ learning agent will place more emphasis on the current learn-
0.4 0.2 0.1 0.05 0.6 ing value, and vice versa. In another word, a bigger learning
rate denotes a longer learning step. However, a big learning
In the third scenario with SN R3 , there are ten states rate can result in a local optimum point and miss the global
{1, 2, 3, 5, 7, 11, 15, 23, 31, 47}, the transition probability of optimum point due to the big learning step. Hence, the learning
which is as the settings in [35]. The other related settings rate should be carefully chosen, which is set to be 0.001 in
are illustrated in Table I. this paper.
GUO et al.: ADAPTIVE RESOURCE ALLOCATION IN FUTURE WIRELESS NETWORKS WITH BLOCKCHAIN AND MEC
Fig. 3. Reward under different discount rates.
Fig. 4. Performance of the proposed approach with different mini-batch
sizes.
The effects of different discount rates on the performance
of the proposed approach are shown in Fig. 3. In DRL,
the actions are chosen by optimizing the long term reward,
where the future rewards are discounted by multiplying the
discount rate, as defined in equation (33). The learning agent
will choose the action maximizing the current reward with a
small discount rate and vice versa. Since the current action
would influence future rewards in this paper, the long term
reward increases with the discount rate growing. However,
it is meaningless to put too many weights on the future in
an unstable system. Also, it would incur high computational
complexity. To explore a tradeoff between the performance
and the computational complexity, an appropriate discount
rate should be chosen, which is set as 0.9 in the rest of the
simulations.
Fig. 4 shows the effects of the mini-batch size of the pro-
posed approach on the convergence performance. The x-axis
denotes the training steps and the y-axis represents the value
of loss function. The mini-batch size indicates how many
experience cases are used to train the Q network in each
training step. We can observe from Fig. 4 that the convergence
is faster with the mini-batch size growing, which is because
that more experiences are used to train the Q network with
a bigger mini-batch size. Similar to the other parameters,
an appropriate mini-batch size should be chosen, which is set
to be 64 in the rest of the simulations.
1699
Fig. 5. Comparison of DQN, double DQN, dueling DQN, double-dueling
DQN.
Fig. 6. Convergence of difference comparison algorithms.
Different DRL methods are compared in Fig. 5. For fairness,
all methods adopt the same simulation parameters. The Y-axis
represents the value of loss function, which represents the gap
to approximate the Q function, while the x-axis represents
the training steps. First, we can see that double-dueling DQN
converges first. Second, double-dueling DQN performs a more
accurate approximation of the Q function than the other
three DRL methods. That is because dueling DQN divides
the Q function into the state value function and the action
advantage function, which allows a better approximation of the
Q values and enables faster convergence. Furthermore, double
DQN selects the actions according to the online weights,
which mitigates the overestimation compared with traditional
DQN. It also results in more accurate approximation.
Fig. 6 shows the convergence of different schemes, where
the y-axis denotes the long term reward. First, the existing
static scheme converges firstly, but obtains the worst perfor-
mance in terms of reward. That is because that the decisions
are made according to the current reward, which needs less
training steps. As a result, it doesn’t consider the effects of the
current action on the future rewards, which obviously obtains
the lowest reward. Second, the proposed scheme maintains
higher long term reward than the other three schemes. With
the adaptive spectrum allocation policy, the latency can be
reduced. With an adaptive block size and a properly chosen
number of producing blocks, the throughput of blockchain
1700 IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS, VOL. 19, NO. 3, MARCH 2020
Fig. 7. Long term reward vs. average transaction size.
Fig. 8. Long term reward vs. time to finality threshold.
system can be improved. All these aspects contribute to better
system performance. Hence, the proposed scheme, which
jointly optimizes these aspects, obtains the best performance
of all.
Fig. 7 depicts the relationship between the system perfor-
mance and the average transaction size. This figure can be
used to show the performance of the proposed approach with
different types of transactions, which are corresponding to
different offloading tasks. With the average transaction size
increasing, the system performance decreases. The reasons
are 1) one block can contain fewer transactions when the
transaction size rises; 2) the transmission latency from the
users to the BSs increases, which lowers the performance
of MEC system. Focusing on the comparison of different
schemes, the proposed scheme obtains the highest long term
reward with the variation of average transaction size, then
follows the proposed scheme with fixed spectrum allocation,
fixed block size and fixed block number, and the existing static
scheme has the worst performance. The reasons are described
as before.
Fig. 8 shows the relationship between the system per-
formance and time to finality threshold. One observation is
that the long term reward increases with the time to finality
threshold raising and intends to be steady finally. With a
flexible latency constraint, less punishment may be added to
the reward, which naturally leads to a higher performance.
However, when the time to finality threshold becomes big
Fig. 9. Rewards with different numbers of BSs and users.
enough, it has little influence on the reward. Another observa-
tion is that the scheme with fixed block size acts better than
that with fixed block number. The reason is that it can adjust
the block number, which in turn adjusts the block interval
to deal with the strict time to finality threshold and avoids
missing blocks. Furthermore, we can find that the existing
static scheme has the poorest performance, which reveals
the superiority of double-dueling DQN based schemes. The
proposed scheme maintains the best performance of all as well.
Fig. 9 plots the reward to show the scalability and the the
ability of the framework in dealing with dynamic changes
of users in the system, where the number of BSs is fixed,
i.e., 5, and the number of users varies from 2 to 20. In the
simulation, we first train the algorithm offline in the scenario
with 20 users and 5 BSs. When the offline-trained algorithm
is applied online with changing number of users, the extra
parameters in the states and actions are set to be null to cope
with the smaller number of users. In another word, the results
in Fig. 9 are obtained online using the algorithm that is
trained offline. From the simulations, we can first observe
that the long-term reward decreases with increasing number
of users. The reasons are: 1) the average resources decrease,
resulting in higher latency, and 2) more blocks may be ignored,
inducing smaller throughput. Second, we can see that the
proposed algorithm outperforms the other existing algorithms,
which shows the superiority of the proposed scheme, even
with a large number of users. Third, the results show that
the offline-trained algorithm can work effectively online under
changing conditions.
In Fig. 10, the performance of the proposed approach is ver-
ified by comparing with the other two baseline algorithms in
three scenarios, where the SNR state settings vary in different
scenarios. In the simulation, we train the proposed framework
again when the dynamics of the SNR have changed. First,
the scenario with SN R3 achieves the best reward. That is
because the system can obtain a better SNR state in this
scenario, which introduce larger throughput and lower laten-
cies, resulting in larger rewards. Second, it can be seen that
the proposed scheme becomes less stable when the number
of SNR states increase. The reason is due to the mismatch
between the states and actions. Third, there is little difference
GUO et al.: ADAPTIVE RESOURCE ALLOCATION IN FUTURE WIRELESS NETWORKS WITH BLOCKCHAIN AND MEC
Fig. 10. Rewards under different SNR state settings.
Fig. 11. Throughput and latency (seconds) versus the number of users.
on the convergence speeds of these algorithms. That is because
the number of states, actions, and steps in one episode is rather
larger than the state set’s dimension, which can be ignored by
the law of large numbers. Fourth, the proposed scheme obtains
the best performance, due to the superiority of DRL, which
also shows the ability of the proposed scheme to adapt to
different network dynamics.
At last, we analyze the throughput and latency of the
proposed algorithm, as shown in Fig. 11, where the number
of BSs is fixed to 10 and the number of users ranges from
5 to 25. Similar to Fig. 9, the results in Fig. 11 are also
obtained online using the offline-trained algorithm, and it
also shows that the proposed approach of training offline
and operating online can work well in practice. Obviously,
the throughput decreases and the latency increases with the
number of users increasing. First, the resources are fixed.
So the average amount of resources decreases with the number
of users going up. Second, the queuing delay grows when the
number of users raises, which induces larger latency.
IX. C ONCLUSION AND F UTURE W ORK
In this paper, we developed a novel blockchain-based frame-
work for resource allocation in future wireless networks with
MEC. With blockchain, the data delivery and computation
execution on the edge servers are self-organized by smart
contracts. A consensus protocol in this distributed wireless
network was proposed, along with the details and theoretical
analysis. The performance of MEC and blockchain system was
1701
given, i.e., the latency of the users, throughput, time to finality,
decentralization, security. To improve the performance of the
joint MEC and blockchain system, the main problem was for-
mulated, where the spectrum allocation, block size, and block
number were optimized. Since this problem is intractable using
traditional methods, we resorted to a novel double-dueling
deep Q learning approach to solve this problem. Simulation
results demonstrated the effectiveness of the proposed scheme
by comparing with the other baseline schemes. Future work
is in progress to consider caching in the proposed framework.
R EFERENCES
[1] V. Sharma, I. You, F. Palmieri, D. N. K. Jayakody, and J. Li, “Secure
and energy-efficient handover in fog networks using blockchain-based
DMM,” IEEE Commun. Mag., vol. 56, no. 5, pp. 22–31, May 2018.
[2] X. Tao, K. Ota, M. Dong, H. Qi, and K. Li, “Performance guaranteed
computation offloading for mobile-edge cloud computing,” IEEE Wire-
less Commun. Lett., vol. 6, no. 6, pp. 774–777, Dec. 2017.
[3] D. M. Kalathil and R. Jain, “Spectrum sharing through contracts
for cognitive radios,” IEEE Trans. Mobile Comput., vol. 12, no. 10,
pp. 1999–2011, Oct. 2013.
[4] J. Zheng, Y. Cai, Y. Wu, and X. Shen, “Dynamic computation offloading
for mobile cloud computing: A stochastic game-theoretic approach,”
IEEE Trans. Mobile Comput., vol. 18, no. 4, pp. 771–786, Apr. 2019.
[5] Y. C. Hu, M. Patel, D. Sabella, N. Sprecher, and V. Young, “Mobile
edge computing—A key technology towards 5G,” ETSI White Paper,
vol. 11, no. 11, pp. 1–16, 2015.
[6] J. Feng, Q. Pei, F. R. Yu, X. Chu, and B. Shang, “Computation offloading
and resource allocation for wireless powered mobile edge computing
with latency constraint,” IEEE Wireless Commun. Lett., vol. 8, no. 5,
pp. 1320–1323, Oct. 2019.
[7] Y. Liu, F. R. Yu, X. Li, H. Ji, and V. C. M. Leung, “Distributed resource
allocation and computation offloading in fog and cloud networks with
non-orthogonal multiple access,” IEEE Trans. Vehi. Technol., vol. 67,
no. 12, pp. 12137–12151, Dec. 2018.
[8] H. Guo, J. Liu, and H. Qin, “Collaborative mobile edge computation
offloading for IoT over fiber-wireless networks,” IEEE Netw., vol. 32,
no. 1, pp. 66–71, Jan. 2018.
[9] C. Wang, F. R. Yu, C. Liang, Q. Chen, and L. Tang, “Joint computation
offloading and interference management in wireless cellular networks
with mobile edge computing,” IEEE Trans. Veh. Technol., vol. 66, no. 8,
pp. 7432–7445, Aug. 2017.
[10] M. Liu and Y. Liu, “Price-based distributed offloading for mobile-
edge computing with computation capacity constraints,” IEEE Wireless
Commun. Lett., vol. 7, no. 3, pp. 420–423, Jun. 2018.
[11] K. Yang, X. Jia, and K. Ren, “Secure and verifiable policy update
outsourcing for big data access control in the cloud,” IEEE Trans.
Parallel Distrib. Syst., vol. 26, no. 12, pp. 3461–3470, Dec. 2015.
[12] F. Tschorsch and B. Scheuermann, “Bitcoin and beyond: A technical
survey on decentralized digital currencies,” IEEE Commun. Surveys
Tuts., vol. 18, no. 3, pp. 2084–2123, 3rd Quart., 2016.
[13] T. Salman, M. Zolanvari, A. Erbad, R. Jain, and M. Samaka, “Security
services using blockchains: A state of the art survey,” IEEE Commun.
Surveys Tuts., vol. 21, no. 1, pp. 858–880, 1st Quart., 2018.
[14] F. R. Yu, J. Liu, Y. He, P. Si, and Y. Zhang, “Virtualization for distributed
ledger technology (vDLT),” IEEE Access, vol. 6, pp. 25019–25028,
2018.
[15] T. N. Dinh and M. T. Thai, “AI and blockchain: A disruptive integration,”
Computer, vol. 51, no. 9, pp. 48–53, Sep. 2018.
[16] Y. Liu, F. R. Yu, X. Li, H. Ji, and V. C. M. Leung, “Decentralized
resource allocation for video transcoding and delivery in blockchain-
based system with mobile edge computing,” IEEE Trans. Vehi. Technol.,
vol. 68, no. 11, pp. 11169–11185, Nov. 2019.
[17] M. Liu, F. R. Yu, Y. Teng, V. C. M. Leung, and M. Song, “Distributed
resource allocation in blockchain-based video streaming systems with
mobile edge computing,” IEEE Trans. Wireless Commun., vol. 18, no. 1,
pp. 695–708, Jan. 2019.
[18] J. Kang et al., “Blockchain for secure and efficient data sharing in
vehicular edge computing and networks,” IEEE Internet Things J., vol. 6,
no. 3, pp. 4660–4670, Jun. 2019.
[19] K. Fan, S. Wang, Y. Ren, K. Yang, Z. Yan, H. Li, and Y. Yang,
“Blockchain-based secure time protection scheme in IoT,” IEEE Internet
Things J., vol. 6, no. 3, pp. 4671–4679, Jun. 2019.
1702 IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS, VOL. 19, NO. 3, MARCH 2020
[20] C. Qiu, F. R. Yu, H. Yao, C. Jiang, F. Xu, and C. Zhao, “Blockchain-
based software-defined industrial Internet of Things: A dueling deep
Q-learning approach,” IEEE Internet Things J., vol. 6, no. 3,
pp. 4627–4639, Jun. 2019.
[21] H. Liu et al., “Blockchain-enabled security in electric vehicles cloud and
edge computing,” IEEE Netw., vol. 32, no. 3, pp. 78–83, May 2018.
[22] G. Wood et al., “Ethereum: A secure decentralised generalised transac-
tion ledger (eip-150 revision),” Ethereum Project Yellow Paper, vol. 151,
no. 2017, pp. 1–32, 2017.
[23] M. Liu, F. R. Yu, Y. Teng, V. C. M. Leung, and M. Song, “Performance
optimization for blockchain-enabled industrial Internet of Things (IIoT)
systems: A deep reinforcement learning approach,” IEEE Trans. Ind.
Informat., vol. 15, no. 6, pp. 3559–3570, Jun. 2019.
[24] V. D. Papoutsis and S. A. Kotsopoulos, “Chunk-based resource alloca-
tion in multicast OFDMA systems with average BER constraint,” IEEE
Commun. Lett., vol. 15, no. 5, pp. 551–553, May 2011.
[25] A. Clement, E. Wong, L. Alvisi, M. Dahlin, and M. Marchetti, “Making
byzantine fault tolerant systems tolerate byzantine faults,” in Proc. 6th
NSDI, 2009, pp. 153–168.
[26] F. Cowell, Measuring Inequality. Oxford, U.K.: Oxford Univ. Press,
2011.
[27] F. Wenli, H. Ping, and L. Zhigang, “Multi-attribute node importance
evaluation method based on Gini-coefficient in complex power grids,”
IET Gener., Transmiss. Distrib., vol. 10, no. 9, pp. 2027–2034, Jun. 2016.
[28] L. P. Kaelbling, M. L. Littman, and A. W. Moore, “Reinforcement
learning: A survey,” J. Artif. Intell. Res., vol. 4, no. 1, pp. 237–285,
Jan. 1996.
[29] V. Mnih et al., “Human-level control through deep reinforcement learn-
ing,” Nature, vol. 518, no. 7540, p. 529, 2015.
[30] H. V. Hasselt, “Double Q-learning,” in Proc. Adv. Neural Inf. Process.
Syst. 23, 2010, pp. 2613–2621. [Online]. Available: http://papers.nips.cc/
paper/3964-double-q-learning.pdf
[31] Z. Wang, T. Schaul, M. Hessel, H. Van Hasselt, M. Lanctot, and
N. de Freitas, “Dueling network architectures for deep reinforcement
learning,” 2015, arXiv:1511.06581. [Online]. Available: https://arxiv.
org/abs/1511.06581
[32] U. Challita, W. Saad, and C. Bettstetter, “Interference management
for cellular-connected UAVs: A deep reinforcement learning approach,”
IEEE Trans. Wireless Commun., vol. 18, no. 4, pp. 2125–2140,
Apr. 2019.
[33] A. L. Strehl, L. Li, E. Wiewiora, J. Langford, and M. L. Littman,
“PAC model-free reinforcement learning,” in Proc. 23rd Int. Conf. Mach.
Learn., 2006, pp. 881–888.
[34] C. Jin, Z. A. Zhu, S. Bubeck, and M. I. Jordan, “Is Q-learning provably
efficient?” in Proc. NIPS, 2018, pp. 4863–4873.
[35] Y. He et al., “Deep-reinforcement-learning-based optimization for cache-
enabled opportunistic interference alignment wireless networks,” IEEE
Trans. Veh. Technol., vol. 66, no. 11, pp. 10433–10445, Nov. 2017.
[36] U. Challita, L. Dong, and W. Saad, “Proactive resource management for
LTE in unlicensed spectrum: A deep learning perspective,” IEEE Trans.
Wireless Commun., vol. 17, no. 7, pp. 4674–4689, Jul. 2018.
[37] J. Zhu, Y. Song, D. Jiang, and H. Song, “A new deep-Q-learning-based
transmission scheduling mechanism for the cognitive Internet of Things,”
IEEE Internet Things J., vol. 5, no. 4, pp. 2375–2385, Aug. 2018.
Fengxian Guo received the B.E. degree in commu-
nications from Zhengzhou University (ZZU), China,
in 2015. She is currently pursuing the Ph.D. degree
with the School of Information and Communica-
tion Engineering, Beijing University of Posts and
Telecommunications (BUPT), Beijing, China. She
was with The University of British Columbia, Van-
couver, Canada, and Carleton University, Ottawa,
Canada, as a Visiting Ph.D. Student from Septem-
ber 2018 to September 2019. Her current research
interests include future wireless networks, mobile
edge computing, blockchain, and machine learning.
F. Richard Yu (S’00–M’04–SM’08–F’18) received
the Ph.D. degree in electrical engineering from The
University of British Columbia (UBC) in 2003.
From 2002 to 2006, he was with Ericsson, Lund,
Sweden, and a start-up in California, USA. He joined
Carleton University in 2007, where he is currently
a Professor. His research interests include con-
nected/autonomous vehicles, security, artificial intel-
ligence, distributed ledger technology, and wireless
cyber-physical systems.
Dr. Yu is a registered Professional Engineer in the
province of Ontario, Canada, and a fellow of the Institution of Engineering and
Technology (IET). He is an elected member of the Board of Governors of the
IEEE VTS. He received the IEEE TCGCC Best Journal Paper Award in 2019,
the Distinguished Service Awards in 2019 and 2016, the Outstanding Lead-
ership Award in 2013, the Carleton Research Achievement Award in 2012,
the Ontario Early Researcher Award (formerly Premiers Research Excellence
Award) in 2011, the Excellent Contribution Award at IEEE/IFIP TrustCom
2010, the Leadership Opportunity Fund Award from Canada Foundation of
Innovation in 2009, and the Best Paper Awards at IEEE ICNC 2018, VTC
2017 Spring, ICC 2014, Globecom 2012, IEEE/IFIP TrustCom 2009, and
International Conference on Networking 2005. He has served as the Technical
Program Committee (TPC) Co-Chair of numerous conferences. He serves on
the editorial boards of several journals, including as a Co-Editor-in-Chief
for Ad Hoc and Sensor Wireless Networks, an Area Editor for the IEEE
C OMMUNICATIONS S URVEYS AND T UTORIALS , a Lead Series Editor for
the IEEE T RANSACTIONS ON V EHICULAR T ECHNOLOGY, and the IEEE
T RANSACTIONS ON G REEN C OMMUNICATIONS AND N ETWORKING. He is
an IEEE Distinguished Lecturer of Vehicular Technology Society (VTS) and
Communications Society.
Heli Zhang received the B.S. degree in commu-
nication engineering from Central South University
in 2009 and the Ph.D. degree in communication
and information system from the Beijing University
of Posts and Telecommunications (BUPT) in 2014.
From 2014 to 2018, she was a Lecturer with the
School of Information and Communication Engi-
neering, BUPT, where she has been an Associate
Professor since 2018. Her research interests include
heterogeneous networks, long-term evolution/fifth
generation, and the Internet of Things.
Dr. Zhang participated in many National projects funded by National
Science and Technology Major Project, National 863 High-tech, and National
Natural Science Foundation of China, and cooperated with many corporations
in research. She has been a reviewer for Journals of IEEE Wireless Commu-
nications, IEEE Communication Magazine, the IEEE T RANSACTIONS ON
V EHICULAR T ECHNOLOGY, the IEEE C OMMUNICATION L ETTERS , and the
IEEE T RANSACTIONS ON N ETWORKING.
Hong Ji (SM’09) received the B.S. degree in
communications engineering and the M.S. and
Ph.D. degrees in information and communications
engineering from the Beijing University of Posts
and Telecommunications (BUPT), Beijing, China,
in 1989, 1992, and 2002, respectively. In 2006, she
was a Visiting Scholar with The University of British
Columbia, Vancouver, BC, Canada. She is currently
a Professor with BUPT. She has authored more than
300 journals/conference papers. Several of her arti-
cles had been selected for Best paper. Her research
interests include wireless networks and mobile systems, including cloud
computing, machine learning, intelligent networks, green communications,
radio access, ICT applications, system architectures, management algorithms,
and performance evaluations.
Dr. Ji is serving on the Editorial Boards of the IEEE T RANSAC -
TIONS ON G REEN C OMMUNICATIONS AND N ETWORKING and Interna-
tional Journal of Communication Systems (Wiley). She has served as the
Co-Chair for Chinacom’11 and a member of the Technical Program Com-
mittee of WCNC’19/15/14/12, Globecom’17/16/15/14/13/12/11/10, ISCIT’17,
CITS’16/15/12, WCSP’15, ICC’20/13/12/11, ICCC’13/12, PIMRC’12/11,
IEEE VTC’12S, and Mobi-World’11. She was a Guest Editor of International
Journal of Communication Systems, (Wiley) Special Issue on Mobile Internet:
Content, Security and Terminal.
GUO et al.: ADAPTIVE RESOURCE ALLOCATION IN FUTURE WIRELESS NETWORKS WITH BLOCKCHAIN AND MEC 1703

Mengting Liu received the Ph.D. degree from Victor C. M. Leung (S’75–M’89–SM’97–F’03)
the Beijing University of Posts and Telecommu- is currently a Distinguished Professor of computer
nications (BUPT), Beijing, China, in 2019. From science and software engineering with Shenzhen
2017 to 2018, she was a Visiting Ph.D. Student University, Shenzhen, China, and a Professor Emeri-
with The University of British Columbia, Vancouver, tus with The University of British Columbia (UBC),
BC, Canada. Her current research interests include Vancouver, BC, Canada. Before he retired from
blockchain technology, deep reinforcement learning, UBC in 2018, he was a Professor of electrical
resource allocation, mobile edge computing systems, and computer engineering and the holder of the
and stochastic geometry theory. TELUS Mobility Research Chair there. He has coau-
thored more than 1300 journals/conference papers
and book chapters. His research is in the broad areas
of wireless networks and mobile systems. He is serving on the Editorial
Boards of the IEEE T RANSACTIONS ON G REEN C OMMUNICATIONS AND
N ETWORKING, the IEEE T RANSACTIONS ON C LOUD C OMPUTING, IEEE
A CCESS , the IEEE N ETWORK, and several other journals. He is a fellow of the
Royal Society of Canada, Canadian Academy of Engineering, and Engineering
Institute of Canada. He received the IEEE Vancouver Section Centennial
Award, the 2011 UBC Killam Research Prize, the 2017 Canadian Award
for Telecommunications Research, and the 2018 IEEE TCGCC Distinguished
Technical Achievement Recognition Award. He has coauthored articles that
received the 2017 IEEE ComSoc Fred W. Ellersick Prize, the 2017 IEEE
Systems Journal Best Paper Award, the 2018 IEEE CSIM Best Journal Paper
Award, and the 2019 IEEE TCGCC Best Journal Paper Award. He is named
in the current Clarivate Analytics list of Highly Cited Researchers.