TRANSACTIONS ON EMERGING TELECOMMUNICATIONS TECHNOLOGIES

Trans. Emerging Tel. Tech. (2016)

Published online in Wiley Online Library (wileyonlinelibrary.com). DOI: 10.1002/ett.3046

RESEARCH ARTICLE

Demand-aware resource allocation for ultra-dense

small cell networks: an interference-separation
clustering-based solution
Junfei Qiu1 , Qihui Wu1 , Yuhua Xu1*, Youming Sun2 and Ducheng Wu1
1 College of Communications Engineering, PLA University of Science and Technology, Nanjing, China
2 National Digital Switching System Engineering and Technological Research Center, Zhengzhou, China

ABSTRACT
In this paper, we present a novel clustering-based resource allocation framework for downlink transmission in ultra-dense
small cell networks. Specifically, we first model a combinatorial optimisation problem that jointly considers subchannel
and power allocation and user traffic demand in terms of a large-scale network scenario. Unfortunately, the huge commu-
nication overhead and computational complexity make it challenging for traditional centralised/distributed solutions. To
address this issue, we propose an interference-separation clustering-based scheme to divide the massive small cells into
smaller groups with different priorities, which reduces the network scale. Different from the existing cluster construction
scheme, the proposed clustering method effectively avoids the inter-cluster interference through coordination. Then, for
a given cluster configuration, we formulate the distributed resource allocation problem as a local interaction game where
the utility of each player comprises not only its own profits but also the interests of neighbours. We prove the existence of
Nash equilibrium for the formulated game and design a hierarchical learning algorithm to achieve the Nash equilibrium,
which only needs local information exchange. Finally, simulation results validate that the proposed solution outperforms
some other existing approaches and is more suitable for large-scale networks. Copyright © 2016 John Wiley & Sons, Ltd.

*Correspondence
Y. Xu, College of Communications Engineering, PLA University of Science and Technology, Nanjing, China.
E-mail: yuhuaenator@gmail.com

Received 5 December 2015; Revised 23 March 2016; Accepted 19 April 2016

1. INTRODUCTION appropriate for future ultra-dense networks, because of the

Small cells have recently emerged as a promising technol-
ogy to address coverage and capacity issues in wireless
systems, which can significantly improve the efficiency of
frequency reuse and spectrum sharing [1]. What is more, it
is expected that the evolving 5G networks will be made up
of hundreds of heterogeneous small cells [2]. However, the
dense deployment of small cell access points (SAPs) raises
several technical issues amongst which interference man-
agement remains the most challenging [3]. Both cross-tier
and co-tier interference will diminish the system perfor-
mance gains. Thus, effective interference mitigation and
resource allocation for small cell networks have attracted
increasing interests from researchers.
Several resource allocation approaches have been pro-
posed in the literature to mitigate interference in small
cell networks [4–6], which are mainly classified into cen-
tralised and distributed solutions. However, most of the
existing schemes used in the traditional systems are not
following reasons: (i) although conventional centralised
solutions can achieve optimal/near-optimal performance,
global information is often not available owing to the ran-
dom deployment of massive SAPs in dense small cell
networks. In addition, huge computational overhead also
makes them impractical. Therefore, based on the afore-
mentioned analysis, decentralised approaches with less
information requirement should be preferred. (ii) Existing
distributed methods, on the other hand, have superiority of
low complexity, but the convergence speed of algorithms
will be very slow in a large-scale network scenario with
tens and even hundreds of nodes, leading to inefficiency.
Therefore, it is urgent and important to study more effi-
cient resource allocation strategies for hyper-dense small
cell networks.
To harness the limitations of traditional optimisation
approaches, in this article, we present a novel interference-
separation clustering-based joint subchannel and power
allocation scheme for downlink transmission of dense

Copyright © 2016 John Wiley & Sons, Ltd.

 J. Qiu et al.
small cell networks, taking the user demand into considera-
tion at the same time. Concretely speaking, to surmount the
obstacles stemming from large-scale network scenarios,
we resort to clustering technique that can decompose the
original resource allocation problem into smaller subprob-
lems for each cluster, reducing the network complexity.
Specifically, based on the proposed clustering scheme,
massive SAPs can be divided into two parts with differ-
ent priorities, that is, split nodes with high priority and
cluster nodes with low priority. These split nodes consti-
tute the buffer zones connecting any two adjacent clusters,
which can effectively avoid the inter-cluster interference.
Afterwards, in terms of the two kinds of SAP groups,
we formulate the distributed resource allocation problem
as a local interaction game, taking into account the real
traffic demands of users and the interests of surrounding
neighbour users, which is proved to be an exact poten-
tial game. Moreover, to achieve the Nash equilibria of the
game, a hierarchical learning algorithm based on stochas-
tic learning automata (SLA) is also presented, which does
not rely on global information or additional centralised
network architecture, making it suitable for dense small
cell networks.
To summarise, the main contributions of this article are
as follows:
 We model the joint subchannel and power allocation
for extremely dense small cell networks as a combi-
natorial optimisation problem, in which the real traffic
demands of users are considered.
 We propose a new interference-separation clustering-
based resource allocation framework to divide the
large-scale network into smaller modules with dif-
ferent priorities, where a hierarchical fashion is per-
formed to achieve the resource assignment according
to the priority.
 Based on the network configuration obtained from
clustering, we formulate the distributed resource allo-
cation problem as a local interaction game with the
SAPs as the players. Particularly, a local reciprocal
marginal utility is designed to avoid the inherent lim-
itation of selfish behaviours of users and improve the
efficiency of the social welfare.
 A distributed algorithm is proposed to reach the Nash
equilibrium (NE) of the formulated game, and its con-
vergence performance is analysed through theoretical
proof and validated by simulations.
The remainder of the paper is organised as follows. In
Section 2, we review the related work and highlight the dif-
ferences between this work and existing work. In Section 3,
we present the system model as well as the problem
formulation. In Section 4, we formulate an interference-
separation clustering-based local interaction game model
and investigate the properties of its NE points. A hier-
archical learning algorithm is proposed in Section 5 to
achieve NE points of the game. Section 6 presents sim-
ulation results and discussion. Conclusions are drawn in
Section 7.
2. RELATED WORK
The problem of resource allocation in small cell net-
works has been extensively studied in the literature. Some
existing works [7–10] investigated applying the dynamic
policies to perform frequency assignment. However, these
studies neglected the power control strategies, leading to
attenuate the efficiency of resource allocation and spatial
reuse. In fact, power control [11, 12] is also an effective
way to mitigate co-channel interference, which needs to be
considered as well.
To this end, several centralised architectures and opti-
misation approaches, taking into account joint channel and
power allocation, have been adopted to improve the sys-
tem efficiency. In [13], the authors proposed a coalition
game-based resource allocation algorithm with centralised
operation and management for small cell networks, aim-
ing at improving the network capacity. A novel dynamic
interference avoidance scheme to prevent excessive inter-
cell interference was investigated in [14], where a central
entity was applied for the resolution of the conflicting
requests. A. Elsherif et al. in [15] studied the problem of
interference management and coarse resource allocation in
a femtocell cellular network and proposed to use a cen-
tral controller to achieve coordination through an adaptive
graph colouring (GC) approach. However, although the
overall system performance is improved by the centralised
schemes, along with which, signalling cost amongst users
and networks and computational complexity increase. As
a result, these methods are generally just suitable for
small-sized networks.
On the other hand, there has been significant interest in
applying distributed approaches based on game theory to
address the problem of resource allocation and interference
mitigation in small cells [16]. P. Semasinghe et al. in [17]
proposed an evolutionary game theory-based distributed
subcarrier and power allocation and used an approach
based on stochastic geometry to analyse the stability of
the equilibrium of the game. In [18], the authors formu-
lated the subchannel assignment problem for the hybrid
overlay/underlay cognitive femtocell networks as a coali-
tion formation game and designed a modified recursive
core algorithm to achieve a stable and efficient allocation.
Furthermore, Z. Zhang et al. extended the general non-
overlapping coalition game models to the formulation with
overlapping coalitions in charge of interference manage-
ment in small cell networks in [19]. Zhang et al. [20]
considered joint the subchannel scheduling and power allo-
cation for cognitive small cell networks using a cooperative
bargaining game model. Applying Stackelberg game the-
ory to address the resource allocation problems for both
the macrocell and small cells was presented in [21–23].
Nevertheless, the earlier-described works do not take into
account the real traffic demands of users, reducing the
spectrum utilisation efficiency. In addition, although game
theory-based decentralised methods have better dynamic
adaptability compared with centralised mechanisms, with
Trans. Emerging Tel. Tech. (2016) © 2016 John Wiley & Sons, Ltd.
DOI: 10.1002/ett
 J. Qiu et al.
the increase of network size, the algorithms will converge
slowly and the system performance cannot be guaranteed.
We can notice that while many different solutions for
resource allocation in small cell networks have been pro-
posed, as the literature survey earlier describes, it is hard
to scale these methods to large-scale networks. In fact,
extremely (hyper/ultra) dense wireless networks with tens
or even multiple hundreds of nodes with small cells will be
a trend, and several works have presented some prospects
about them [24–26]. However, how to achieve resource
management to mitigate interference in such dense net-
works is a challenging issue because of the random deploy-
ment of small cells in large quantities. Recently, there are
few studies applying the concept of clustering to dense
small cell networks, for example, [27] and [28]. In [27],
the authors designed a distributed clustering algorithm to
group the femtocells into disjoint clusters. Then, within
each cluster, a femtocell was elected as a cluster head
to perform resource assignment in a centralised manner.
To investigate the optimal cluster configuration, a central
entity such as a femto gateway was utilised to be respon-
sible for the clustering subproblem in [28], whereas, the
cluster head was responsible for subchannel and power
allocation. However, these traditional clustering methods
have two drawbacks: (i) because there exists inter-cluster
interference, the performances of cluster-edge users are rel-
atively bad. (ii) The burden on cluster head is very high
because of the intra-cluster centralised resource allocation.
To tackle these issues, Virgile Garcia et al. had investigated
applying the macrodiversity-coordinated multipoint trans-
mission with user-centric adaptive clustering to improve
the system performance of dense cellular networks
[29], which provides us with well heuristic exploration
and research.
Different from the related works in the literature, in
this paper, we propose a fire-new clustering method in
which any two adjacent clusters are separated by the buffer
zones, avoiding the inter-cluster interference, that is, clus-
ters are not only disjoint but also interference-separation.
In addition, we investigate more practical scenarios taking
user demand diversity into consideration and present a dis-
tributed subchannel and power allocation solution based on
game theory.
3. SYSTEM MODEL AND PROBLEM
FORMULATION
3.1. System model
We consider a dense small cell network that consists of a
set of N randomly deployed SAPs. There are Ln user equip-
ments (UEs) associated with SAP n, 1 6 n 6 N, and
without loss of generality, we assume that at most one UE
is active within the coverage of the SAP at the time-instant
considered. We denote by n an active UE served by SAP
n. SAPs and UEs are all equipped with single antennas. It is
assumed that the small cell and macrocell networks operate
on orthogonal channels, and there are M orthogonal sub-
Trans. Emerging Tel. Tech. (2016) © 2016 John Wiley & Sons, Ltd.
DOI: 10.1002/ett
channels available for the SAPs. We consider the universal
frequency reuse deployment in which every small cell
shares the whole bandwidth. For presentation, denote the
SAP set as N , that is, N D f1, : : : , Ng and the available
subchannel set as M, that is, M D f1, : : : , Mg. In this
paper, we focus on the study of downlink communications
from SAPs to the users.
Suppose a UE n is connected with SAP n and let ın, m
n
be the subchannel allocation indicator to denote whether
subchannel m is allocated to the user n within cell n. Sim-
ilar to previous studies in [30] and [31], we consider that
one UE can only transmit data on single subchannel in
each time slot. Our analysis can also be easily extended to
multi-subchannel transmission model. Thus, the instanta-
neous received signal-to-interference noise ratio of UE n
over subchannel m is written as
m pm Gm
ın,
m n n n,n
n,n
D PN m m m
(1)
i D 1,i¤n ıi,i pi Gi,n C 2
where pmn is the transmit power of SAP n over subchan-
nel m and Gm n,n is the channel power gain from SAP
n to UE n over subchannel m. In this work, without
loss of generality, we assume that the bandwidth of each
subchannel is less than the coherence bandwidth of the
channel so that flat fading is considered in each subchan-
nel. The corresponding achievable data rate is given by the
following:
 
Rm m
n,n D B log2 1 C n,n (2)
where B is the subchannel bandwidth.
3.2. Problem formulation
Because of the dense deployment of SAPs, the co-channel
interference amongst small cells becomes severe, which
majorly limits the system performance. Here, we mainly
investigate the joint subchannel and power allocation, tak-
ing the demands of the users into account at the same time,
to maximise the system performance in a particular slot.
Definition 1. A spectrum vector characterises the set
of subchannels used by SAPs to transmit data: f D
.f1 , f2 , : : : , fn , : : : , fN /, where Œfn D fn . Noting that we
consider the universal frequency reuse deployment in
which every cell shares the whole bandwidth, the subchan-
nel set M D f1, : : : , Mg is available for each SAP and
fn 2 M. Therefore, the constraint set of spectrum vectors
(i.e. the subchannel selection strategy space) is given by
M D ffjfn 2 f1, 2, : : : , m, : : : , Mg, 8n 2 N g.
Definition 2. A transmit power vector characterises
the transmit power values used by each SAP to com-
municate with its respective attached UEs: p D
.p1 , p2 , : : : , pn , : : : , pN /, where Œpn D pn . Here, we con-
sider the discrete power control. Note that in real practice,

discrete power control can benefit over continuous power
control in the transmitter design and the overhead of
information exchange [32, 33]. We assume each SAP
can use L > 2 different power levels for transmission,
namely, f1 Pn,max , 2 Pn,max , : : : , L Pn,max g, where 0 D
1 < 2 < : : : L D 1. Then, the constraint set
of transmit power vectors is given by P D fpjpn 2
f1 Pn,max , 2 Pn,max , : : : , L Pn,max g, 8n 2 N g.
Small cells are coordinated to jointly determine the opti-
mal spectrum and transmit power vectors, which maximise
the system utility. Thus, the resource allocation problem
can now be formulated as
.fopt , popt / D arg max U0
f2M,p2P
where U0 is the system utility function. If we take into
account the user traffic demand, without loss of the gener-
ality, the form of the satisfaction of user i can be defined
as [34]
 Ri
, R < Di
ui D Di i (4)
1, otherwise
where Ri denotes the achievable data rate for the user i and
Di is the required transmission rate. Then, from the
P system
optimisation point of view, we can have U0 D ui and
i2N
problem (3) can be transformed into
P
N intractable problem. In this work, we present a new cluster-
.P1/ : arg max ui .pi , pi , fi , fi /
iD1
subject to
.a/ fi 2 f1, 2, : : : , Mg, 8i 2 N
.b/ pi 2 f1 Pi,max , 2 Pi,max , : : : , L Pi,max g, 8i 2 N
m 2 f0, 1g, 8i 2 N , 8m 2 M
.c/ ıi,i
where pi and fi represent the power and subchan-
nel allocation strategy profiles of all the SAPs excluding
SAP i. Constraints (a) and (b) denote the subchannel and
power allocation strategy space, respectively. Constraint
(c) restricts that each element of the allocation matrix is a
binary variable. Specifically, ui is a coupling product that
needs to combine the formulas (1), (2) and (4) to calculate.
Remark 1. The sum-utility optimal joint subchannel and
power allocation problem for a dense small cell network
belongs to a class of combinatorial optimisation problems;
finding the globally optimal solution .fopt , popt / is non-
deterministic polynomial hard (NP-hard), and it will be
more complex in the large-scale networks. For example,
suppose a network scenario with 50 SAPs, where each SAP
determines the selection strategies from six subchannels
and four power levels, the number of all possible strategy
selection profiles will be 450  650 D 1.0246  1069 ,
which is a huge number beyond the processing abilities of
general algorithms. Therefore, standard optimisation tech-
niques cannot be applied directly that necessitates new
methods to tackle this issue.
J. Qiu et al.
4. INTERFERENCE-SEPARATION
CLUSTERING BASED LOCAL
INTERACTION GAME APPROACH
To address the aforementioned problem (5), in this
section, we present a distributed solution by apply-
ing the interference-separation clustering-based game-
theoretic approach. First, in terms of the hyper-dense
network scenario, we propose a new clustering method to
group the massive SAPs into two smaller parts with differ-
ent priorities, that is, ‘split messengers (SMs)’ with high
priority and ‘cluster members (CMs)’ with low priority.
Second, based on the cluster configuration, we design a
demand-aware local interaction game model to achieve the
(3) distributed subchannel and power allocation. According to
the resource access priority, a hierarchical learning scheme
is performed between the SM group and CM group. What
is more, because the clusters are mutually independent,
resource assignment for CMs is accomplished in parallel.
This framework can be illustrated using the diagram shown
in Figure 1.
4.1. Distributed interference-separation
cluster formulation
Although there are some preliminary studies [27, 28]
applying clustering technique to dense small cell networks,
how to address the inter-cluster interference is still an
ing scheme, in which the clusters are not only disjoint but
also interference-separation. The distributed interference-
separation cluster formulation algorithm is described in
Algorithm 1.
The main idea of the algorithm is as follows: each SAP
can compute its own interference degree (i.e. the num-
(5)
ber of interfering neighbours) by sensing the environment.
If a SAP has highest degree of interfering neighbours, it
will elect itself as the first cluster head and inform its
neighbours to constitute the first cluster. Then, those SAPs
receiving the broadcast information from the first clus-
ter will be denoted as SMs. Remove the aforementioned
SAPs from the node list and sort the list of the remainder
SAPs decreasingly according to their interference degree.
The second cluster and corresponding SMs can generate
in the light of the previous rules. Repeat the procedures
hereinbefore until completing the clustering.
After accomplishing the distributed interference-
separation cluster formulation through local information
exchange, a large-scale network is divided into two parts,
including out-cluster SAPs (i.e. SMs) and inner-cluster
SAPs (i.e. CMs). These SMs compose the buffer zones
that can effectively eliminate the interference between
two neighbouring clusters. Here, we assume the SMs
have the high priority to gain the resource in the network
and low priority for CMs. Once satisfying these SMs’
requirements, the remainder clusters will become mutually
independent and a cluster is now a resource allocation
unit. Because of interference separation between two
Trans. Emerging Tel. Tech. (2016) © 2016 John Wiley & Sons, Ltd.
DOI: 10.1002/ett
 J. Qiu et al.

Figure 1. Diagram of the proposed interference-separation clustering-based game-theoretic framework. SMs, split messengers;
CMs, cluster members.

Algorithm 1 Distributed interference-separation clustering arbitrary neighbouring clusters, these CMs in each cluster
algorithm can achieve resource allocation in parallel while avoiding
mutual influences.
Initialization: Given a network topology with N SAPs, A simple example is given in Figure 2. Specifically, in
denoted by a list S D fs1 , s2 , : : : sN g, and jSj D N, Figure 2(a), the initial network consists of 14 SAPs and
set the iteration index t D 0, subscript index i 2 the interference relationships amongst the small cells are
f1, 2, : : : , Ng. shown by the black dotted line. In Figure 2(b), according to
1: Loop for t D 0, : : :, the clustering rules, massive SAPs are divided into two cat-
2: for i D 1 to N egories: CMs and out-cluster nodes (i.e. SMs). Here, there
3: si establishes a list of interfering neighbor SAPs are three clusters (plotted by the green dotted line), c1 , c2
by exploiting its attached users’ measurement reports, and c3 . SAPs 4 and 11 are denoted by SMs that can avoid
and then transmits and shares the associated interfering interference between c1 and c2 , c2 and c3 , respectively.
list with the corresponding neighbors. For each cluster, the overall-network resource is available.
4: end Note that SAPs 4 and 11 have the higher priority; if they
5: for i D 1 to N have occupied several subchannels in the network, their
6: if si has the highest degree of interfering neigh- neighbouring SAPs will try to refrain from accessing them
bors in the system, then to avoid collisions. In other words, to improve the system
7: si elects itself as the first cluster head CH1 and efficiency, SAPs 3, 5 and 6 should avert using the same
informs its neighbors channels as SAP 4, while with regard to SAPs 9 and 12, it
8: si and its neighboring SAPs are considered is advisable to prevent interference for SAP 11 happening.
as cluster members (CMs), constituting the first cluster
c1 ; some other SAPs will elect themselves as SMs if Remark 2. PIt is noted that the original system utility
is U0 D ui , after clustering, which can be rewrit-
receiving the broadcasts of CMs from c1 i2N
9: Removing these SMs and CMs from S jSP
SM j jCj P
P jci j
10: while jSj > 0 ten as U0 D ui C uj , where SSM denotes
11: Sort the list of remainder SAPs decreas- iD1 iD1jD1
the SM set, C denotes the cluster sets and ci is the ith
ingly according to their interference degree; repeat the jCj
similar executions like 6 to 9 cluster, [i D 1 ci D SCM , and SCM denotes the CM set.
From the perspective of strategy selection, we can have
12: end
.M  L/N possible profiles for the initial network, while
13: else
in terms of the clustering configuration, that is, .M 
14: i C 1, go back to 5 jCj
P
15: end if L/jSSM j C .M  L/jci j , in which M is the number of
16: end iD1
17: End loop subchannels and L is the number of power levels. Because
in a large-scale network, jSSM j < N and jci j < N hold,

Trans. Emerging Tel. Tech. (2016) © 2016 John Wiley & Sons, Ltd.
DOI: 10.1002/ett
 J. Qiu et al.

Figure 2. Distributed interference-separation cluster formulation. (a) Original network topology and (b) interference-separation
clustering. SM, split messenger; CM, cluster member.

jCj
P It is worth noting that other metrics can also be used
.M  L/jSSM j C .M  L/jci j  .M  L/N follows. to decide the interference graph (e.g. based on the ratio
iD1
In other words, the proposed interference-separation clus- between the experienced interference and desired signal
tering framework dramatically reduces the computational [32, 33] or further considering the traffic load). However,
there would be little variation in the following game model
overhead and improves the system efficiency.
and the main conclusions. Because the optimal construc-
tion of the interference graph is not the focus of this
4.2. Local interaction game for work, we adopt the metric simply based on the geographic
demand-aware SAP coordination location [35, 36].
In this section, we will apply a game-theoretic self-
4.2.1. Game model.
organising approach to analyse the intra-cluster resource
Based on the interference graph, a local interaction
allocation.† Because of the lower coverage of SAPs, the
game to achieve cluster satisfaction maximisation is pro-
signal transmitted by a SAP causes significant interference
posed. Because the clusters are mutually independent, all
only to the UEs located in a few neighbouring cells. An
the clusters can achieve resource allocation simultane-
interference graph can be introduced to characterise the
ously in parallel. In terms of a given cluster ck .ck 2 N /
local interference relationships amongst the small cells.
consisting of jck j .jc˚k j  N/ SAPs, the game is formally 
Here, we use a simple distance-determined model for pre-
denoted by G D ck , fFl ˝ Pl gl2ck , fJl gl2ck , fUl gl2ck ,
sentation. The interference distance of an arbitrary SAP n
where ck D f1, : : : , jck jg is the set of players (i.e.
can be indicated as d0 . Denote Jn as the set of neighbour-
SAPs) in the cluster ck , Fl ˝ Pl is the set of avail-
ing SAPs of SAP n, that is, Jn D fi 2 N : dn,i 6 d0 g,
able joint power and subchannel allocation strategies
where dn,i is the distance between SAP n and SAP i. There-
for player l, Jl is the neighbouring SAP set of player
fore, the potential interference relationship can be now
l and Ul is the utility function of player l. To improve
characterised by an interference graph G D fV, Eg, where
the efficiency of the game, motivated by the utility
V is the vertex set (the SAP set) and E is the edge set, that
design in [36], a local reciprocal marginal util-
is, V D f1, : : : , Ng and E D f.i, j/ji 2 N , j 2 N , di,j 6 d0 g.
ity function for each player l is defined as follows:
   
†
Ul pl , pJl , fl , fJl D ulPpl , pJl , fl , fJl 
In fact, we can regard the SM group as a special cluster, in which the
C ui pi , pJi , fi , fJi
distributed subchannel and power allocation scheme is similar to the i2J
Pl   (6)
case for the CMs. Therefore, we term the resource allocation for SMs  ui pi , pJi nl , fi , fJi nl
and CMs as intra-cluster resource assignment uniformly. i2Jl

Trans. Emerging Tel. Tech. (2016) © 2016 John Wiley & Sons, Ltd.
DOI: 10.1002/ett
 J. Qiu et al.

where pJl 2 PJl and fJl 2 FJl denote the power and
subchannel allocation strategy profiles of player l’s neigh-
bours, respectively. A1 nA2 means that A2 is excluded from
A1 ; hence, pJi nl and fJi nl are the strategy profiles of player
i’s neighbours when SAP l gives up competing for powers
and subchannels.
Note that the aforementioned defined utility function is
composed of three parts: the individual utility of player l,
the sum utility of its neighbours and the aggregate utility
of l’s neighbours at the time of without l’s competition. In
other words, each SAP will consider the payoffs of itself as
well as its neighbours during the strategy selection, rather
than considering itself only. Then, the local interaction
game is expressed as follows:
First, we construct a potential function as follows:
X
ˆ.pl , pl , fl , fl / D ul .pl , pl , fl , fl / (10)
l2ck
where pl and fl represent the power and subchannel
allocation strategy profiles of all the SAPs in the cluster ck
excluding SAP l, respectively. It is shown that the potential
function is equal to the sum-utility of a cluster.
(a) Only power selection strategy changes: Suppose that
an arbitrary player in the cluster ck , say l, unilaterally
changes its transmit power from pl to p0 l , then the
change in potential function is given by

   
.G/ : arg max Ul pl , pJl , fl , fJl , 8l 2 ck , ck 2 C (7) ˆ p0 l , pl , fl , fl  ˆ.pl , pl , fl , fl /
pl 2Pl ,fl 2Fl X   X
D ul p0 l , pl , fl , fl  ul .pl , pl , fl , fl /
l2ck l2ck
4.2.2. Analysis of Nash equilibrium.  0
  
D ul p l , pJl , fl , fJl  ul pl , pJl , fl , fJl
The properties of the proposed local interaction game X    
are characterised as follows. C ui pi , p0 Ji , fi , fJi  ui pi , pJi , fi , fJi
i2Jl
X     
Definition 3. Nash equilibrium: A joint power C ui pi , p0J i , fi , fJi ui pi , pJi , fi , fJi
and subchannel allocation profile .p , f /
 D i2ck nJl ,i¤l
     
p1 , p2 , : : : , pjc j , f1 , f2 , : : : , fjc j is a pure strategy NE
k k
(11)
point of G if and only if no player can improve its utility by Based on the interference graph, we can know that if
deviating unilaterally, that is, i 2 fck nfJl , lgg, p0 Ji D pJi , when only l changes its
transmit power. Thus, the following equation holds:
   
Ul pl , pJl , fl , fJl > Ul pl , pJl , fl , fJl  
(8) ˆ p0 l , pl , fl , fl  ˆ .pl , pl , fl , fl /
 0   
8l 2 ck , 8pl 2 Pl , fl 2 Fl , pl ¤ pl , fl ¤ fl D ul p l , pJl , fl , fJl  ul pl , pJl , fl , fJl
X    
C ui pi , p0 Ji , fi , fJi  ui pi , pJi , fi , fJi
Definition 4. Exact potential game: A resource allo- i2Jl
cation game is an exact potential game if there exists a (12)
potential function ˆ : .p1 ˝ f1 /  .p2 ˝ f2/      On the other hand, the change of individual utility
pck ˝ fck ! R such that for all l 2 ck , pl , pl 2 Pl and function caused by this unilateral change is given by
fl , fl 2 Fl , the following holds:
   
Ul p0 l , pJl , fl , fJl  Ul pl , pJl , fl , fJl
     
Ul .pl , pl , fl , fl /  Ul p0 l , pl , f 0 l , fl D ul p0 l , pJl , fl , fJl  ul pl , pJl , fl , fJl
  (9) X    
D ˆ.pl , pl , fl , fl /  ˆ p0 l , pl , f 0 l , fl C ui pi , p0 Ji , fi , fJi  ui pi , pJi , fi , fJi
i2Jl
That is, the change in the utility function caused by the (13)
unilateral action change of an arbitrary player is equal to Then, based on Equations (12) and (13), we can
that in the potential function. obtain
 
Theorem 1. The demand-aware joint power and sub- ˆ p0 l , pl , fl , fl  ˆ .pl , pl , fl , fl /
 0    (14)
channel allocation game G is an exact potential game that D Ul p l , pJl , fl , fJl  Ul pl , pJl , fl , fJl
has at least one pure strategy NE. The optimal solution
of the cluster satisfaction maximisation problem is a pure (b) Only subchannel selection strategy
 changes: Given

strategy NE of G. transmit power vector p D p1 , p2 , : : : , pjck j , the
subchannel access problem decouples across SAPs.
Proof. The following proof procedure is similar to the The set of competing neighbouring players, who may
idea given in [37]. The difference is that our work consid- compete for subchannels with player l, can be given
ers the case of joint power and subchannel allocation, while as Il D fi 2 Jl : fi D fl g. The selected subchan-
Xu et al. [37] just took into account the spectrum access. nel sets of SAPs in Il contain at least one of available

Trans. Emerging Tel. Tech. (2016) © 2016 John Wiley & Sons, Ltd.
DOI: 10.1002/ett
 J. Qiu et al.

subchannels of SAP l. In this context, only the SAPs pl ! p0 l

in Il can influence the satisfaction of SAP l in terms fl ! f 0 l  
of subchannel access. Thus, it can be found that Y.pl , fl / ! Y p0 l , f 0 l
0
pl !p l   fl !f 0 l  
) Y.pl , fl / ! Y p0 l , fl ! Y p0 l , f 0 l
    „ ƒ‚ … „ ƒ‚ …
ul pl , pJl , fl , fJl D ul pl , pJl , fl , fIl (15) step 1 step 2
(19)
where Y./ denotes a general function. Steps 1 and
In addition, if i 2 fck nfJl , lgg or i 2 fJl nIl g, we 2 have the similar properties to the aforementioned
can obtain f0 Ji D fJi D f0 Ii D fIi when only l discussions in the cases of (a) and (b); therefore,
changes its subchannel selection. Thus, suppose the synthesising the analysis from them, we can acquire
power of SAP l is given, if it unilaterally changes its  
subchannel selection from fl to f 0 l , the corresponding ˆ p0 l , pl , f 0 l , fl  ˆ .pl , pl , fl , fl /
 0    (20)
change in individual utility function is expressed as D Ul p l , pJl , f 0 l , fJl  Ul pl , pJl , fl , fJl

    It is shown from Equations (14), (18) and (20) that the

Ul pl , pJl , f 0 l , fJl  Ul pl , pJl , fl , fJl
    change in individual utility function caused by any player’s
D Ul pl , pIl , f 0 l , fIl  Ul pl , pIl , fl , fIl unilateral deviation equals to the change in the potential
   
D ul pl , pIl , f 0 l , fIl  ul pl , pIl , fl , fIl function. Thus, according to the definition of exact poten-
X     tial game given in [38], the demand-aware local interaction
C ui pi , pIi , fi , f0 Ii  ui pi , pIi , fi , fIi game G for a given cluster ck .ck 2 N / is an exact poten-
i2Il tial game with the whole cluster satisfaction Uck serving
(16)
as the potential function. Exact potential game is a special
The change in the potential function caused by this kind of game because it admits several promising prop-
unilateral change is given by erties, amongst which the most important two aspects are
as follows:
 
ˆ pl , pl , f 0 l , fl  ˆ.pl , pl , fl , fl /
     Potential game has one pure strategy NE at least.
D ul pl , pIl , f 0 l , fIl  ul pl , pIl , fl , fIl  Local or global maxima of potential function consti-
X    
C ui pi , pIi , fi , f0 Ii  ui pi , pIi , fi , fIi tutes a pure strategy NE.
i2Il
X      Thus, Theorem 1 is proved based on the aforementioned
C ui pi , pIi , fi , f0 Ii  ui pi , pIi , fi , fIi properties.
i2Jl nIl
X     
C ui pi , pIi , fi , f0 Ii ui pi , pIi , fi , fIi Remark 3. With respect to the formulated game, we
i2ck nJl ,i¤l do not adhere to the assumption of selfishness of users
    as in traditional game models; instead, a local reciprocal
D ul pl , pIl , f 0 l , fIl  ul pl , pIl , fl , fIl
X  marginal utility function is designed in which each player
  
C ui pi , pIi , fi , f0 Ii  ui pi , pIi , fi , fIi takes into account the payoffs of itself as well as its neigh-
i2Il bours. In this context, the inherent limitation due to selfish
(17) behaviours can be avoided and the system efficiency would
Then, from Equations (16) and (17), we have the be improved. Such a case applies the benefits of local infor-
following equation: mation exchange well to prevent the ‘tragedy of commons’
[39] happening.
 
ˆ pl , pl , f 0 l , fl  ˆ .pl , pl , fl , fl / 5. CLUSTERING-BASED
    (18)
D Ul pl , pJl , f 0 l , fJl  Ul pl , pJl , fl , fJl HIERARCHICAL SLA ALGORITHM
FOR ACHIEVING NE
(c) Power and subchannel selection strategies change It is known that there are some commonly used learning
simultaneously: Imagine the power and subchannel algorithms in the literature converging to pure strategy NE
selection strategies of an arbitrary SAP l change at points of potential games, for example, spatial adaptive
the same time, that is, from pl and fl to p0 l and play [37], no-regret learning [23] and best response [32].
f 0 l , then the potential function will transform  from However, these algorithms cannot be applied in the con-
ˆ.pl , pl , fl , fl / to ˆ p0 l , pl , f 0 l , fl . Because of sidered dense small cell networks because they are origi-
the non-coupling relationship between the alterations nally designed for static environment and require complete
of power and subchannel, the following derivation information of all other players’ selected actions in each
process holds: iteration. Due to the clustering scheme, there are two kinds

Trans. Emerging Tel. Tech. (2016) © 2016 John Wiley & Sons, Ltd.
DOI: 10.1002/ett
 J. Qiu et al.
of SAP groups with different priorities existing in the net-
work, that is, CMs and SMs. Therefore, in this section,
motivated by the underlying idea of SLA [40, 41], we
propose a clustering-based hierarchical learning algorithm
(C-HLA) for distributed subchannel and power allocation,
in which each player makes decisions autonomously.
5.1. Algorithm description
To characterise the proposed C-HLA for resource alloca-
tion, we extend the local interaction game G to a mixed
strategy form and give the following definitions. Let Q D
.q1 , : : : ,qN / denote the mixed strategy profile, in which
qn D qn1 , : : : , qnK , 8n 2 N , m 2 M, is the strategy
selection probability vector of SAP n (K D L  M is the
number of available strategies) and qnk denotes the proba-
bility with which SAP n selects strategy k. The underlying
ideas of the proposed C-HLA can be summarised as fol-
lows: (i) based on the clustering configuration, the original
SAP set S can be divided into two parts, that is, SSM with
high priority for SMs and SCM with low priority for CMs.
Thus, the algorithm is performed in a hierarchical man-
ner. (ii) For a SM n 2 SSM , it can learn from its individual
action–reward experiences with local influences from other
neighbouring SMs and adjust its behaviour towards sta-
ble state. (iii) After self-organising subchannel and power
selections for SMs, decentralised resource assignment for
CMs in each cluster is carried out in parallel, where strate-
gies changes of SAPs in cluster ci will not cause impacts
on other SAPs attached to cj , 8ci , cj 2 SCM , i ¤ j. The
proposed C-HLA is described in Algorithm 2.
5.2. Discussion about convergence of the
proposed hierarchical learning algorithm
In this subsection, we provide a discussion on the conver-
gence of the proposed algorithm towards NE. Specifically,
its asymptotical convergence property is characterised by
the following theorem.
Theorem 2. When the learning parameter goes suffi-
ciently small, that is, b ! 0, the proposed hierarchical
learning algorithm asymptotically converges to a pure
strategy NE point of the formulated local interaction game.
Proof. Because of the hierarchical characteristics, in
terms of a local area with several correlated SAPs, such
as the SMs or the SAPs in a cluster, we can obtain the
existence of NE of the local game from the previous
analysis. Because the SAPs in the clusters achieve the
resource allocation in a parallel manner and update their
strategies according to the similar rules as the SMs, the
local convergence property for each cluster follows that
of SMs. Therefore, we first present a discussion about the
convergence of the proposed algorithm for SMs.
In terms of the SMs with the high priority, the resource
allocation is performed based on the scheme of SLA. For-
tunately, the convergence of SLA towards pure strategy
Trans. Emerging Tel. Tech. (2016) © 2016 John Wiley & Sons, Ltd.
DOI: 10.1002/ett
Algorithm 2 The proposed C-HLA for distributed resource
allocation
Initialization: denote the strategy selection matrix by K
with L  M dimensions, jKj D K; set the time slot
t D 0, and the initial strategy selection probability vec-
tor qnk .t/ D 1=K, K D L  M, 8n 2 N , m 2 M.
Loop for t D 0, 1, 2, : : :
For the SMs with the high priority level
1) At the beginning, each SM n (n 2 SSM ) selects a
strategy an .t/ according to its current strategy selection
probability vector qn .t/. Then, each SM n receives the
random reward Un .t/ specified by (6) through exchang-
ing information with its neighbouring SMs.
2) All the SMs update their mixed strategies according
to the following rule:
qnk .t C 1/ D qnk .t/ C bUn .t/.1  qnk .t//, k D an .t/
qnk .t C 1/ D qnk .t/ C bUn .t/qnk .t/, k ¤ an .t/
(21)
where 0 < b < 1 is the learning parameter.
3) If 8n 2 SSM , there exists a component of qn .t/
which is approaching one, e.g., larger than 0.99, stop;
otherwise, go to step 1).
For the CMs with the low priority level
4) With respect to the CMs, the distributed resource
allocation is carried out
 in parallel in each
 cluster. Each
jCj
CM j in a cluster ci [i D 1 ci D SCM performs strat-
egy selection in the light of its current strategy selection
probability, and then will receive a feedback Uj .t/ also
specified by (6), which is jointly determined by the
strategy selections of SMs and the behaviours of the
neighbouring CMs in the same cluster.
5) All the CMs update their mixed strategies accord-
ing to the similar rules in (21) and stop if the selection
probabilities on any strategy are larger than a predefined
threshold, e.g., 0.99; otherwise, go to step 4).
End loop
Nash equilibria of exact potential games, with the condi-
tion of the learning parameter b ! 0, has been rigorously
proved in [40]. In methodology, the differences in this work
are summarised as follows: (i) all players only consider
spectrum access during the strategy updating in [40], while
the action profile also includes the power assignment in our
work. (ii) To improve the system efficiency, in this paper,
local information exchange is applied in which each player
takes into account the incomes of itself as well as its neigh-
bours. In fact, variations of strategy space and utility design
will not affect the essential attributes of convergence of the
algorithm. Therefore, following similar lines for the proof
given in [40], the convergence of the proposed hierarchi-
cal learning algorithm for SMs can be achieved. To avoid
unnecessary repetition, here, the details are omitted. Fur-
thermore, with regard to the CMs, the relevant proofs are
also convinced because they utilise the same rules as the
 J. Qiu et al.
SMs to adjust their strategies. Therefore, according to the
aforementioned analysis, Theorem 2 can be proved.
Through the previous discussions, we can know that
based on the clustering configuration, the proposed algo-
rithm with sufficiently small learning parameter can
asymptotically converge to a pure strategy NE point of
the formulated local interaction game. Furthermore, it is
noted from Theorem 1 that the optimal solution of the local
group (such as for SMs or an arbitrary cluster) utility max-
imisation problem is a pure strategy NE. Therefore, from
the statistical average perspective, the attainable local total
utility with the learning algorithm when b ! 0, is close
to the locally optimal utility. Because the achievable sys-
tem utility of the proposed scheme is the sum of each local
group utility, it can be expected that our method would
obtain considerably desirable performance.
Remark 4. We can notice that the proposed interference-
separation clustering-based solution can achieve resource
allocation in parallel, in which each cluster becomes a
resource assignment unit. This scheme greatly accelerates
the speed of the traditional learning algorithms, which
is validated in the section of simulation results. What
is more, the proposed clustering-based parallel process-
ing framework can also scale to other existing distributed
learning methods, improving their convergence speeds to
be suitable for large-scale network scenarios.
5.3. Computational complexity analysis
The proposed scheme involves two main stages, that is,
interference-separation clustering and distributed learning.
Therefore, we need to calculate their respective complex-
ities. At first, in terms of the clustering stage, each SAP
needs two multiplications, one addition and one exponent
operation to calculate the distance between each other and
then N  1 comparisons to construct the list of interfer-
ing neighbours. Thus, the complexity of establishing an
interference graph for N SAPs is O.N 2 /. Next, it needs
N comparisons to elect the first cluster head CH1 , N  1
comparisons to determine the other CMs of the first clus-
ter c1 and jc1 j.N  jc1 j/ comparisons to pick out the
SMs for c1 , continuing the succeeding operations in the
same way with much lower complexity, which can be neg-
ligible, until completing the clustering. In summary, the
total complexity for the clustering stage approximates to
be O.N 2 /.
Then, on the basis of the cluster configuration, all the
SAPs are divided into the SM and CM groups. In the imple-
mentation phase of the learning algorithm, if the SAP l
belongs to the SSM (i.e. the SM set),ˇ as for ˇ the computa-
tion of the utility value ul , it needs ˇJlSM ˇ C 1 additions
ˇ ˇ
and ˇJlSM ˇ C 3 multiplications (divisions) to first compute
the information rate and then one comparison to calcu-
late the utility value, where JlSM denotes the neighbouring
ˇ ˇ
SMs for the SAP l. Next, it needs two ˇJlSM ˇ  1 addi-
tions to compute the reward Ul specified by Equation (6).
Therefore, the total complexity for computing the random
ˇ ˇ2 
reward is O ˇJlSM ˇ . In addition, the process of strategy
updating for the algorithm is based on the SLA. Accord-
ing to Zheng et al. [42], the computational complexity for
SLA is O.jAl j/, where Al is the strategy space of the
SAP l. Thus, when the subchannel number is M and the
power level number is L, then, jAl j D M  L, that is,
SLA’s complexity will be O.M  L/. Therefore, for a
SAP l .l 2 SSM /, in total, the computational complex-
ityfor the operation procedure
 of the learning algorithm is
ˇ ˇ2
O ˇJlSM ˇ  M  L . Instead, if the SAP l is affiliated
with the SCM (i.e. the CM set) after clustering, the pro-
cedure of analysis is similar to the case as a SM, which
is omitted for concision and brevity. The difference is that
a CM needs to consider the influences from both neigh-
bouring CMs and SMs. Therefore, the total computational
complexity in the algorithm execution stage for a CM is
O.jJl j2  jAl j/.
The aforementioned analysis presents the computational
complexity for each iteration of the proposed scheme.
In addition, the whole computational complexity also
depends on the number of iterations of the learning algo-
rithm needed for convergence (i.e. convergence speed).
Nevertheless, there is a trade-off between the achiev-
able performance and convergence speed of the proposed
learning algorithm. On the one hand, we have proved in
Theorem 2 that the probability of achieving the pure NE
by our proposed algorithm would be close to 1 when the
learning parameter b ! 0; however, it cannot be obtained
in finite number of iterations in practical operations [40].
On the other hand, if b is not sufficiently small for real
application, there may exist performance loss that will be
demonstrated in the simulation section. In addition, it is
worth mentioning that the greatest advantage of our pro-
posed scheme is that it can obtain a very satisfactory
system performance with a faster convergence speed in
the large-scale networks. Of course, achieving this purpose
also needs some energy cost, which includes the overhead
of information exchange between the neighbouring SAPs
in the phase of clustering the learning algorithm execution.
6. SIMULATION RESULTS
AND DISCUSSION
In this section, we evaluate the performance of the pro-
posed hierarchical joint subchannel and power allocation
scheme. The total spectrum is divided into several orthog-
onal subchannels with the bandwidth of 200 KHz. Gm n,n
is the channel power gain from SAP n to its UE n on
 
subchannel m, which is expressed as Gm n,n D dn,n ,
where dn,n is the distance between SAP n to UE n ,  is
set to be 2 and the noise power experienced at each receiver
is assumed identical and has a power of 174 dBm. Simi-
lar to the previous studies [5, 33], in this work, we assume
that the bandwidth of each subchannel is less than the
coherence bandwidth of the channel. In this context, our
Trans. Emerging Tel. Tech. (2016) © 2016 John Wiley & Sons, Ltd.
DOI: 10.1002/ett
 J. Qiu et al.
proposed algorithm is still efficient when considering the
small-scale channel fading. The radius of a small cell is
10 m, and the interference range is 50 m.
The simulation results mainly include the following
four parts. In the first part, we present an example of
interference-separation clustering scheme. In the second
part, we illustrate the convergence behaviour of the pro-
posed algorithm. In the third part, we compare the achiev-
able system utility of the following three resource alloca-
tion schemes in small network scenarios: (i) the proposed
scheme, (ii) the exhaustive search and (iii) the random
allocation (RA) approach. In the fourth part, we provide
a performance evaluation from three aspects in large net-
work scenarios, that is, system utility, convergence speed
and personal properties.
6.1. An example of the
interference-separation clustering scheme
In Figure 3, we provide an example to characterise the
interference-separation clustering scheme. In Figure 3(a),
we assume that there are 20 SAPs randomly deploying in
an area of 200 m  200 m, where the small circle is the
SAP and the dotted line denotes the interference relation-
ship. In Figure 3(b), the clustering configuration is given,
in which the green dotted line denotes the cluster, and the
small circles with different colours show that the SAPs
belong to different groups, that is, the SAPs with red colour
denote the CMs, while those of blue colours are SMs. From
Figure 3(c), we can note that, based on the clustering, SAPs
6, 7, 15, 17, 19 and 20 are grouped into the SMs and the
remaining SAPs are denoted by the CMs, constituting four
clusters. Furthermore, SAPs 9, 13, 1 and 2 are made up of
the cluster head for respective cluster. In the following sim-
ulations, the clustering scheme is performed according to
the case in Figure 3.
Figure 3. Simulation results about an example of interference-separation clustering with 20 SAPs in the network. (a) Original net-
work topology, (b) clustering configuration and (c) clustering results. SAP, small cell access point; SM, split messenger; CM, cluster
member; CH, cluster head.
Trans. Emerging Tel. Tech. (2016) © 2016 John Wiley & Sons, Ltd.
DOI: 10.1002/ett
6.2. Convergence of the proposed
clustering based hierarchical learning
algorithm
To show the convergence of the proposed scheme, we
consider a dense network scenario with random deploy-
ment of 40 SAPs in a 200 m  200 m square area.
There are three subchannels and four power levels .p D
f40 mw, 60 mw, 80 mw, 100 mwg/ available in the sys-
tem. The traffic demands of UEs are varied in (0, 3]
Mbps with the learning parameter of the proposed algo-
rithm is b D 0.15. Figure 4 depicts the evolution
of the strategy selection probabilities of the proposed
learning algorithm for an arbitrarily SAP. We plot six
cases out of 12 from the strategy set. It is noted that
the strategy selection probability vector evolves from
f1=12, 1=12 , 1=12 , 1=12 , 1=12 , 1=12g to f0, 0, 0, 0, 1, 0g
in about 30 iterations. In other words, the SAP finally
selects strategy pn5 for transmission. The evolutions of the
number of SAPs on each subchannel and power level are
shown in Figures 5 and 6, respectively. It is noted that
the system converges in about 110 iterations for subchan-
nel variation and 90 iterations for power level variation.
These results validate the convergence of the proposed
hierarchical learning algorithm.
6.3. Numerical results for small scenarios
Because the global optimum cannot be found by existing
computing techniques in large-scale networks, a small sce-
nario with several small cells varied from four to eight
existing in an area of dimensions 100 m  100 m is con-
sidered. Three subchannels and three power levels .p D
f40 mw, 60 mw, 80 mwg/ are available for the network.
The traffic demands of UEs are heterogeneous with dis-
tribution in (0, 3] Mbps, and the learning parameter of
the proposed algorithm is set to b D 0.15. In this con-
text, we compare our schemes with exhaustive search and
RA algorithm. It is clear that the achievable system util-
ity of the exhaustive search serves as an upper bound. On
 J. Qiu et al.
Figure 4. The evolution of the strategy selection probabilities of
a small cell access point.
Figure 5. The evolution of the number of small cell access
points (SAPs) on each subchannel.
Figure 6. The evolution of the number of small cell access
points (SAPs) on each power level.
Figure 7. Comparison of the achievable system utility of three
resource allocation schemes in small scenarios. SAP, small cell
access points.
the other hand, in the RA algorithm, each SAP selects an
arbitrary strategy for data transmission, without informa-
tion exchange and channel availability statistics known in
priori.
Figure 7 shows the comparison results of the achievable
system utility when increasing the number of SAPs. For the
proposed learning algorithm and RA approach, the simula-
tion results are obtained by independently simulating 1000
trials and then taking the average results. It is noted that
the proposed learning algorithm significantly outperforms
the RA approach and yields almost the same performance
as the global optimum.
6.4. Numerical results for big scenarios
To validate the advantages of the proposed scheme in large-
scale network scenario, in this subsection, we provide a
performance evaluation from the following perspectives.
(1) System utility comparison: We consider a net-
work deployed in the square region of 200 m 
200 m with three subchannels and four power lev-
els p D f40 mw, 60 mw, 80 mw, 100 mwg. The traffic
demands of UEs are heterogeneous with distribution
in (0, 3] Mbps, and the learning parameter of the pro-
posed algorithm is set to b D 0.15. For comparison,
we apply four other schemes for benchmarks.
 Global SLA (GSLA) scheme [40]: all the play-
ers in the network update their own strategies
according to the rules of SLA simultaneously,
that is, global SAPs participate in the resource
competition. In order to investigate the advan-
tage of the presented local reciprocal marginal
utility, for this scheme, we consider the fol-
lowing two cases. Case 1: the utility function
Trans. Emerging Tel. Tech. (2016) © 2016 John Wiley & Sons, Ltd.
DOI: 10.1002/ett
 J. Qiu et al.
of each SAP is the proposed local reciprocal
marginal utility. Denote case 2 by applying
the traditional selfish utility pattern, in which
each player tries to maximise its own profits,
ignoring the payoffs of its neighbours.
 GC algorithm [43]: the two arbitrary vertices
connected by one edge cannot use the same
colour (i.e. subchannel), if possible. The trans-
mit power is allocated equally amongst the
small cells (e.g. 40 mw).
 Traditional clustering-based allocation (TC-A)
scheme [27]: the cluster formation is established
through information exchange between one-
hop neighbours, where there exists inter-cluster
interference. Based on the TC-A scheme,
resource allocation in each cluster is performed
by the distributed algorithm for SMs, while
applying the proposed local reciprocal marginal
utility as the utility function for the SAPs.
 RA scheme: each SAP selects a strategy ran-
domly regardless of other SAPs’ actions.
Figure 8 shows the variation in the achievable sys-
tem utility with the number of SAPs. The results
are obtained by simulating 1000 independent tri-
als and then taking the expected values. It is seen
that the achievable system utility of the proposed
scheme is greater than that of the GC, TC-A and
RA approaches. Furthermore, as the number of the
SAPs increases, the performance gains get much big-
ger. The reason is that the co-tier interference will
become severer with the increasing of small cell den-
sity, while the aforementioned three methods lack
efficient coordination mechanisms.
Figure 8. Performance comparison of different schemes
under large-scale networks. GSLA, global stochastic learning
automata; GC, graph colouring; TC-A, traditional clustering-
based allocation; RA, random allocation; SAP, small cell access
point.
Trans. Emerging Tel. Tech. (2016) © 2016 John Wiley & Sons, Ltd.
DOI: 10.1002/ett
Specifically, in terms of the different schemes, we
give the following analyses: (i) it is noted that the
achievable system utility of the proposed clustering-
based SLA algorithm is very close to that of tradi-
tional GSLA algorithm with the proposed utility and
higher than that of other four schemes, which val-
idates the performance guarantee of our presented
method. In addition, comparing the achievable sys-
tem performances of the GLSA scheme with two
different utility functions, we can note that the system
performance of applying the proposed local recip-
rocal marginal utility is better than that of using
traditional selfish utility pattern. This shows the supe-
riority of the designed utility function. (ii) With
regard to the TC-A scheme, it is lack of effective res-
olution to the inter-cluster interference that weakens
the profits of the SAPs located at the edge of clus-
ters, even though this method may be suitable for
dense networks. (iii) For the GC algorithm, in spite of
achieving less channel occupancy collision through
effective colouring scheme, it ignores the benefits of
applying power control. (iv) In terms of the RA algo-
rithm, it is easy to know the system utility would
be very low if selecting strategies randomly. These
results show the advantages of our proposed scheme
in terms of system utility over some other exist-
ing methods for resource allocation in large-scale
networks.
(2) Convergence speed comparison: In dense network
scenarios, distributed learning algorithms often need
many iterations to guarantee the convergence. In
order to demonstrate the superiority of our method
in terms of convergence speed, we compare the con-
vergence speed of the proposed C-HLA with the
GSLA scheme [40] from a statistical perspective.
The parameter setup is similar to the scenario in the
previous system utility comparison. Figure 9 shows
Figure 9. Convergence speed comparison for different network
scales. SLA, stochastic learning automata; CDF, cumulative
distribution function.
 J. Qiu et al.

the cumulative distribution function of the iterations

needing to converge to the stable state of the formu-
lated game for different network scales. It is noted
that for a given network scale (e.g. N D 30), the con-
vergence speed of the proposed scheme is faster than
that of the GSLA as expected. Moreover, when the
network scales up, the convergence speed of the pro-
posed algorithm decreases slightly, whereas that of
GSLA decreases significantly. The results show the
advantage of our method over GSLA in the matter of
convergence speed.
From Figures 8 and 9, we can note that our pro-
posed interference-separation clustering-based learn-
ing algorithm can yield almost the same performance
as the traditional global learning algorithm, with a
faster convergence speed.
(3) Impact of parameter variation: To investigate the
impact of learning parameter b on the utility, Figure 11. System utility versus traffic demand. SAP, small cell
Figure 10 shows the variation in system utility with access point.
the learning parameter. We consider a scenario with
40 SAPs, 3 subchannels and 4 power levels p D
f40 mw, 60 mw, 80 mw, 100 mwg. It is clear from the bers. In this case, the effect of traffic demand on the system
figure that the system with lower learning parameter utility is presented in Figure 11. We notice that, for a
can obtain higher utility at last but needs more itera- given network scale, as the requirement increases, the sys-
tions to converge. In fact, the choice of the value of tem utility decreases. The reason is that the competition
b involves a trade-off between speed and accuracy. amongst SAPs will become severe with the increase of user
Smaller learning parameter implies a slower conver- demands, which attenuates the system performance.
gence speed, while the algorithm may converge to
To summarise, the simulation results validate the con-
the non-NE pure strategy with the bigger b. Thus, the
vergence of the proposed hierarchical learning algorithm in
choice of b is application-dependent and should be
dense small cell networks. Also, they show that our method
performed by practical experiment or training.
can achieve satisfactory system performance in both small
and big scenarios. What is more, the presented scheme
The traffic demands for different users in previous sim-
ulations are set to be heterogeneous, that is, taking the has faster convergence speed owing to the parallel pro-
demand differentiation into consideration. In the follow- cessing framework. In a word, our proposed interference-
ing, we investigate the case that the demands of users separation clustering-based game-theoretic solution has
are same and increase from 0.5 to 3 Mbps. There are great advantages over several existing resource allocation
three subchannels and four power levels, that is, p D approaches and is more suitable for large-scale networks.
f40 mw, 60 mw, 80 mw, 100 mwg, with different SAP num-
7. CONCLUSIONS

In this paper, we have investigated the joint subchannel and

Figure 10. System utility versus learning parameter.
power allocation problem in hyper-dense small cell net-
works, taking into account the traffic demands of users at
the same time. To cope with the challenges brought by the
large scale and lack of centralised control, we proposed
a novel interference-separation clustering-based resource
allocation scheme, in which the SAPs were divided into
two groups with different priorities to select their own
strategies (i.e. subchannel and power) autonomously in a
hierarchical manner. Based on the cluster configuration,
a local interaction game was formulated to characterise
the local influence relationships amongst the SAPs, where
each SAP considered the payoffs of itself as well as its
neighbours. The game was proved to be an exact potential
game, and the NE was achieved by using a decentralised
learning algorithm with local information exchange.

Trans. Emerging Tel. Tech. (2016) © 2016 John Wiley & Sons, Ltd.
DOI: 10.1002/ett
 J. Qiu et al.
Simulation results validated the performance gains of the
proposed solution over several existing methods for large-
scale networks.
ACKNOWLEDGEMENT
This work was supported by the National Natural Science
Foundation of China under grant nos. 61401508, 61172062
and 61301160.
REFERENCES
1. Fehske AJ, Viering I, Voigt J, Sartori C, Redana S,
Fettweis GP. Small-cell self-organizing wireless net-
works. Proceedings of the IEEE 2014; 102 (3):
334–350.
2. Demestichas P, Georgakopoulos A, Karvounas D,
Tsagkaris K, Stavroulaki V, Lu J, Xiong C, Yao J. 5G
on the horizon: key challenges for the radio-access net-
work. IEEE Vehicular Technology Magazine 2013; 8(3):
47–53.
3. Langar R, Secci S, Boutaba R, Pujolle G. An operations
research game approach for resource and power alloca-
tion in cooperative femtocell networks. IEEE Transac-
tions on Mobile Computing 2015; 14(4): 675–687.
4. Son K, Lee S, Yi Y, Chong S. REFIM: a practical inter-
ference management in heterogeneous wireless access
networks. IEEE Journal on Selected Areas in Communi-
cations 2011; 29(6): 1260–1272.
5. López-Pérez D, Chu X, Zhang J. Dynamic downlink
frequency and power allocation in OFDMA cellular net-
works. IEEE Transactions on Communications 2012;
60(10): 2904–2914.
6. Huang JW, Krishnamurthy V. Cognitive base stations
in LTE/3GPP femtocells: a correlated equilibrium game-
theoretic approach. IEEE Transactions on Communica-
tions 2011; 59(12): 3485–3493.
7. Meng Y, Li J, Li H, Pan M. A transformed conflict
graph based resource allocation scheme combining inter-
ference alignment in OFDMA femtocell networks. IEEE
Transactions on Vehicular Technology 2015; 64 (10):
4728–4737.
8. Kim SJ, Cho IK. Graph-based dynamic channel assign-
ment scheme for femtocell networks. IEEE Communica-
tions Letters 2013; 17(9): 1718–1721.
9. Xu C, Sheng M, Wang X, Wang C, Li J. Distributed
subchannel allocation for interference mitigation in
OFDMA femtocells: a utility-based learning approach.
IEEE Transactions on Vehicular Technology 2015;
64(6): 2463–2475.
10. Chen D, Jiang T, Zhang Z. Frequency partitioning
methods to mitigate cross-tier interference in two-tier
Trans. Emerging Tel. Tech. (2016) © 2016 John Wiley & Sons, Ltd.
DOI: 10.1002/ett
femtocell networks. IEEE Transactions on Vehicular
Technology 2015; 64(5): 1793–1805.
11. Apandi NIA, Hardjawana W, Vucetic B. A distributed
cooperative power allocation scheme for small cell net-
works. In IEEE Wireless Communications and Network-
ing Conference (WCNC), New Orleans, United States,
2015; 1882–1887.
12. Zhao N, Yu FR, Sun H. Adaptive energy-efficient power
allocation in green interference-alignment-based wire-
less networks. IEEE Transactions on Vehicular Technol-
ogy 2015; 64(9): 4268–4281.
13. Tian H. Cooperative resource allocation for self-healing
in small cell networks. IEEE Communications Letters
2015; 19(7): 1221–1224.
14. Rahman M, Yanikomeroglu H. Enhancing cell-edge per-
formance: a downlink dynamic interference avoidance
scheme with inter-cell coordination. IEEE Transactions
on Wireless Communications 2010; 9(4): 1414–1425.
15. Elsherif A, Chen WP, Ito A, Ding Z. Adaptive resource
allocation for interference management in small cell net-
works. IEEE Transactions on Communications 2015;
63(6): 2107–2125.
16. Xu Y, Wang J, Wu Q, Du Z, Shen L, Anpalagan A. A
game theoretic perspective on self-organizing optimiza-
tion for cognitive small cells. IEEE Communications
Magazine 2015; 53(7): 100–108.
17. Semasinghe P, Hossain E, Zhu K. An evolutionary game
for distributed resource allocation in self-organizing
small cells. IEEE Transactions on Mobile Computing
2015; 14(2): 274–287.
18. Ma B, Cheung M, Wong V, Huang J. Hybrid over-
lay/underlay cognitive femtocell networks: a game
theoretic approach. IEEE Transactions on Wireless Com-
munications 2015; 14(6): 3259–3270.
19. Zhang Z, Song L, Han Z, Saad W. Coalitional games
with overlapping coalitions for interference management
in small cell networks. IEEE Transactions on Wireless
Communications 2014; 13(5): 2659–2669.
20. Zhang H, Jiang C, Beaulieu N, Chu X, Wang X, Quek
T. Resource allocation for cognitive small cell networks:
a cooperative bargaining game theoretic approach. IEEE
Transactions on Wireless Communications 2015; 14(6):
3481–3493.
21. Wang Y, Wang X, Wang L. Low-complexity Stackelberg
game approach for energy-efficient resource allocation
in heterogeneous networks. IEEE Communications Let-
ters 2014; 18(11): 2011–2014.
22. Rose L, Belmega EV, Saad W, Debbah M. Pricing in het-
erogeneous wireless networks: hierarchical games and
dynamics. IEEE Transactions on Wireless Communica-
tions 2014; 13(9): 4985–5001.
23. Bu S, Yu FR, Yanikomeroglu H. Interference-aware
energy-efficient resource allocation for OFDMA-based
heterogeneous networks with incomplete channels state
 J. Qiu et al.
information. IEEE Transactions on Vehicular Technol-
ogy 2015; 64(3): 1036–1050.
24. Hwang I, Song B, Soliman SS. A holistic view on hyper-
dense heterogeneous and small cell networks. IEEE
Communications Magazine 2013; 51(6): 20–27.
25. Bhushan N, Li J, Malladi D, Gilmore R, Brenner D,
Damnjanovic A, Sukhavasi R, Patel C, Geirhofer S.
Network densification: the dominant theme for wire-
less evolution into 5G. IEEE Communications Magazine
2014; 52(2): 82–89.
26. Yunas S, Valkama M, Niemela J. Spectral and energy
efficiency of ultra-dense networks under different
deployment strategies. IEEE Communications Magazine
2015; 53(1): 90–100.
27. Hatoum A, Langar R, Aitsaadi N, Boutaba R, Pujolle G.
Cluster-based resource management in OFDMA femto-
cell networks with QoS guarantees. IEEE Transactions
on Vehicular Technology 2014; 63(5): 2378–2391.
28. Abdelnasser A, Hossain E, Kim DI. Clustering and
resource allocation for dense femtocells in a two-tier cel-
lular OFDMA network. IEEE Transactions on Wireless
Communications 2014; 13(3): 1628–1641.
29. Garcia V, Zhou Y, Shi J. Coordinated multipoint trans-
mission in dense cellular networks with user-centric
adaptive clustering. IEEE Transactions on Wireless
Communications 2014; 13(8): 4297–4308.
30. Pateromichelakis E, Shariat M, Quddus A, Dianati M,
Tafazolli R. Dynamic clustering framework for multi-
cell scheduling in dense small cell networks. IEEE
Communications Letters 2013; 17(9): 1802–1805.
31. Zhang Z, Song L, Han Z, Saad W. Coalitional games
with overlapping coalitions for interference management
in small cell networks. IEEE Transactions on Wireless
Communications 2014; 13(5): 2659–2669.
32. Zhang H, Venturino L, Prasad N, Li P, Rangarajan S,
Wang X. Weighted sum-rate maximization in multi-cell
networks via coordinated scheduling and discrete power
control. IEEE Journal on Selected Areas in Communica-
tions 2011; 29(6): 1214–1224.
33. Zheng J, Cai Y, Liu Y, Xu Y, Duan B, Shen X. Optimal
power allocation and user scheduling in multicell net-
works: base station cooperation using a game-theoretic
approach. IEEE Transactions on Wireless Communica-
tions 2014; 13(12): 6928–6942.
34. Meng Y, Li JD, Li HY, Liu P. Graph-based user satisfac-
tion aware fair resource allocation in OFDMA femtocell
networks. IEEE Transactions on Vehicular Technology
2015; 64(5): 2165–2169.
35. Chang RY, Tao Z, Zhang J, Kuo CJ. Multicell OFDMA
downlink resource allocation using a graphic frame-
work. IEEE Transactions on Vehicular Technology 2009;
58(7): 3494–3507.
36. Wu Q, Wu D, Xu Y, Wang J. Demand-aware multi-
channel opportunistic spectrum access: a local interac-
tion game approach with reduced information exchange.
IEEE Transactions on Vehicular Technology 2015;
64(10): 4899–4904.
37. Xu Y, Wang J, Wu Q, Anpalagan A, Yao Y. Opportunis-
tic spectrum access in cognitive radio networks: global
optimization using local interaction games. IEEE Jour-
nal of Selected Topics in Signal Processing 2012; 6(2):
180–194.
38. Monderer D, Shapley LS. Potential games. Games and
Economic Behavior 1996; 14: 124–143.
39. Haddad M, Elayoubi SE, Altman E, Altman Z. A hybrid
approach for radio resource management in heteroge-
neous cognitive networks. IEEE Journal on Selected
Areas in Communications 2011; 29(4): 831–842.
40. Xu Y, Wang J, Wu Q, Anpalagan A, Yao Y. Opportunis-
tic spectrum access in unknown dynamic environment:
a game-theoretic stochastic learning solution. IEEE
Transactions on Wireless Communications 2012; 11(4):
1380–1391.
41. Xu Y, Wu Q, Wang J, Shen L, Anpalagan A. Robust mul-
tiuser sequential channel sensing and access in dynamic
cognitive radio networks: potential games and stochas-
tic learning. IEEE Transactions on Vehicular Technology
2015; 64(8): 3594–3607.
42. Zheng J, Cai Y, Shen X, Zheng Z, Yang W. Green energy
optimization in energy harvesting wireless sensor net-
works. IEEE Communications Magazine 2015; 53(11):
150–157.
43. Sadr S, Adve RS. Partially-distributed resource allo-
cation in small-cell networks. IEEE Transactions on
Wireless Communications 2014; 13(12): 6851–6862.
Trans. Emerging Tel. Tech. (2016) © 2016 John Wiley & Sons, Ltd.
DOI: 10.1002/ett