This article has been accepted for publication in a future issue of this journal, but has not been

fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2017.2778145, IEEE Access

IEEE ACCESS 1

A Cost-optimized Resource Provisioning

Policy for Heterogeneous Cloud Environments
Xin Chen, Feng Ding, Tiantian Zhang, Gang Hou, and Lan Lan,

Abstract—To avoid the drawbacks of a pricing mechanism in heterogeneous cloud environments that considers only single
resources, we propose a multi-resource combinatorial pricing mechanism in this paper. This approach jointly considers the
principal resources (i.e., CPU, memory, storage and bandwidth) with the goal of minimizing the total cost. Then, a cost-optimized
resource provisioning policy (CORP P ) based on game theory is applied to this mechanism that considers the Nash equilibrium
between cloud users as well as the Stackelberg equilibrium between users and the cloud provider. The experimental results
show that the proposed combinatorial pricing mechanism is more suitable in a heterogeneous cloud environment when running
various types of cloudlets. Moreover, CORPP is effective in reducing users’ total costs when using both random cloudlets and
PlanetLab cloudlets.

Index Terms—heterogeneous cloud; combinatorial pricing; resource provisioning; game theory; Nash equilibrium; Stackelberg
equilibrium.

1 I NTRODUCTION resource insufficiency during peak demand periods for vir-

computing is a recent business model. With the
C LOUD
development of smart computing environments, cloud
computing can be considered as a natural improvement on
parallel computing, distributed computing and grid comput-
ing. The resources at cloud service providers (CSP) owns
are leased by different cloud users (CU), who can apply and
release their cloud computing resources dynamically [1-3].
Because there are different types of cloudlets, the resources
a task relies on are also vary. For example, a calculation
task relies on a CPU more than on other resources, a
matrix operation may focus primarily on memory, a cloud
storage service depends on available storage and bandwidth
resources, and so on.
Resource pricing is a foundational concern when s-
tudying a cloud computing provisioning policy. However,
researchers have mostly considered only single-resource
pricing mechanisms (i.e., for CPU, memory or network).
For example, Li et al. [4] proposed a scheduling strategy to
maximize the benefits of a CSP’s network resources based
on a network resource bidirectional auction mechanism in
a federated selfish cloud environment. Many similarities
exist between the cloud computing environment and the
free market in economic theory; thus, many scholars have
proposed methods based on market mechanisms to solve
the cloud computing provisioning problem. Goiri et al. [5]
proposed a federated cloud architecture for CSPs based
on economics. Toosi et al. [6] proposed an approach to
outsource or terminate cloud tasks to solve the problem of
• X. Chen is with the School of Electronics and Information Engineering,
Liaoning University of Technology, Jinzhou, China.
• F. Ding, T. Zhang and G. Hou are with the School of Software
Technology, Dalian University of Technology, Dalian, China; email:
dingf2017@gmail.com
• L. Lan is with the International Studies of College, Shenyang University,
Shenyang, China.
2169-3536 (c) 2017 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
tual machines in a federated cloud environment. Moreover,
most articles based on the market economy model focus
on maximizing the benefits accrued by the CSP without
considering the purchasing costs of CUs.
However, as mentioned before, the tasks executing on
a computing platform rely on different resources. Conse-
quently, a single-resource pricing mechanism has natural
drawbacks: it can lead to higher costs for some CUs or
cause unnecessary purchases. There are two possible rea-
sons for this phenomenon: (1) multiple CUs can share cloud
computing resources, but traditional resource provisioning
policies do not consider the impact on CUs when using
common resources; (2) existing studies based on market
economy game theory do not consider the influence of CSP
pricing strategy or resource provisioning policy change on
CUs [7,8].
Therefore, the sustainability of cloud computing sys-
tem urgently requires a multi-dimensional resource pricing
mechanism for heterogeneous cloud computing resources
and a resource provisioning policy that simultaneously
considers different CUs and the interplay between CUs
and the CSP. In this paper, we propose a multi-dimensional
resource combinatorial pricing mechanism that jointly con-
siders the principal resources (i.e., CPU, memory, storage
and bandwidth) with the goal of minimizing the total cost.
Furthermore, a cost-optimized resource provisioning pol-
icy (CORPP) in a heterogeneous cloud environment is
defined based on game theory that maximizes system utility
while minimizing the total cost to CUs.
The rest of this paper is organized as follows. In Sec-
tion 2, we introduce the resource pricing mechanism and
provisioning policy and then analyze the game relations
in CORPP in Section 3. The experiments and results are
presented in Section 4, and final conclusions and future
research directions are given in Section 5.
 This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2017.2778145, IEEE Access

IEEE ACCESS 2

2 R ESOURCE PRICING MECHANISM AND combined price of CPU, memory, storage, and bandwidth
PROVISIONING POLICY for server Hi .
2.1 Problem description
2.3 Cost-optimized resource provisioning policy
In a cloud computing system, CUs submit resource require-
(CORP P )
ments and different types of cloud tasks to a CSP. The cloud
computing resource distributor consists of a pricing mecha- The global resource provisioning policy of a CSP is defined
nism, a statistical mechanism, a task scheduler, and a virtual as Φ = (ϕ1 , ϕ2 , ..., ϕn )T and determines the provisioning
machine manager. The pricing mechanism sets the price for of CU’s resources. In CORP P , ϕj is the policy of Uj
multi-dimensional combinations and is responsible for pric- (1 ≤ j ≤ n), expressed as (rj1 , rj2 , ..., rjm )T , in which
ing different types of resources from different cloud servers. rji = (cpuij , memij , stgeij , bwji ) (1 ≤ i ≤ m). The total
The pricing mechanism is responsible for recording each resources required by all CUs on server H∑ i is denoted by
n
CU’s purchases and the revenue generated by each cloud Ri = (T cpui , T memi , T stgei , T bwi ) = j=1 rji , and is
server. The task scheduler is responsible for scheduling restricted to be no greater than the maximum capacity of
different cloud tasks. CORPP includes two types of cloud the corresponding resources of RESi , i.e.,
tasks: (1) User tasks (those submitted by a CU); and (2) 

 T cpui ≤ CP Ui
System tasks (those required to manage the normal opera- 
T memi ≤ M EMi
tion of cloud computing system). Different tasks running on (1)

 T stgei ≤ ST GEi
different cloud servers will affect a CU’s purchasing costs. 
T bwi ≤ BWi
The virtual machine manager is responsible for dynamically
creating, starting, suspending, and deleting virtual machines where 1 ≤ i ≤ m. The function set of the server’s cost is
based on the CUs’ resource requirements. A cloud data defined as
center is heterogeneous, containing a variety of different
types of cloud servers that provide CPU, memory, storage, F = {F1 (·), F2 (·), ..., Fm (·)}, (2)
network bandwidth and other physical resources required
to run the virtual machines. The structure of a cloud in which Fi (rji ) represents the cost of Uj using CPU,
computing system is shown in Figure 1. memory, storage, and bandwidth resources rji on server
The cloud task scenario we consider in this paper is as Hi , and the cost of all CUs generated on the server Hi is
follows. Multiple CUs submit resource requests as well denoted by Fi (Ri ). From the above definition we can see
as cloud tasks to the CSP at the same time. During the that for policy Φ to be feasible, we must meet the following
duration of a lease period, all the cloud tasks from CUs conditions:
can be completed. Thus, we can make the following three  ∑m i
assumptions: (1) all the resource types in the system (e.g.,  i=1 rj = ωj
rji ≥ 0 (3)
CPU, memory, storage, and bandwidth) are independent; 
(2) The CSP has a sufficient number of resources to meet RESi − Ri ≥ 0
the resource requirements of all its CUs within a lease for each i ∈ [1, m] and j ∈ [1, n].
period; and (3) each CU behaves selfishly; the CSP wishes Formula 3 represents the fact that the sum of all server
to retain its CUs by offering relatively low prices to improve resources assigned to Uj should be equal to Uj ’s demand,
its reputation in the cloud computing market and, thus, i.e., ωj , and that the total demand on each resource should
attract more CUs. be limited to the maximum capacity of the corresponding
resource of each server.
2.2 Multi-dimensional resources combinatorial Then, the price that Uj pays is the total cost of all the
pricing mechanism server resource Uj uses, i.e.,

Assume that a heterogeneous cloud data center has m ∑m

U CΦ (j) = ψ(i, j) · Fi (Ri ). (4)
cloud servers, denoted as H = {H1 , H2 , ..., Hm }. The i=1
maximum CPU, memory, storage and bandwidth capac- Here, ψ(i, j) = 1 when Uj uses server Hi ; conversely,
ity of each server Hi is defined as a vector RESi = ψ(i, j) = 0. Because Formula 4 presents the purchasing
(CP Ui , M EMi , ST GEi , BWi ). U = {U1 , U2 , ..., Un } cost of Uj , maximizing −U CΦ (j) means that Uj maxi-
denotes n CUs all using the cloud computing resources mizes its utility function.
simultaneously. The resource requirement of U is Ω = The cost of the entire cloud computing system is deter-
(ω1 , ω2 , ..., ωn )T , in which the CPU, memory, storage, and mined by the total costs T CΦ of the global provisioning
bandwidth requirements of Uj are expressed as ωj = policy Φ, i.e.,
(cpuj , memj , stgej , bwj ), respectively. For different types
∑m
of cloud servers, the prices of different resource types vary; T CΦ = Fi (Ri ). (5)
the multi-dimensional resource combinatorial pricing in this i=1

paper is accurate for the four resources of each cloud server. Using the same logic, maximizing −T CΦ means that we
We define P (RESi ) = (Picpu , Pimem , Pistge , Pibw ) as the obtain the smallest total system cost.
2169-3536 (c) 2017 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
 This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2017.2778145, IEEE Access

IEEE ACCESS 3

Fig. 1. The structure of a cloud computing system

3 G AME THEORY IN CORP P can be achieved in CORP P , we define the cost-optimized

problem as follows:
Market economic game theory can be used to analyze
the relationships between multiple CUs as well as the
∑n ∑m T
relationship between CUs and the CSP with the goal of M inimize T CΦ = rji · (P (Ri )) (6)
finding the most suitable resource provisioning policy for j=1 i=1

the cloud computing system. The relationships in CORP P s.t.,  ∑m i

are shown in Figure 2, where the CSPs are responsible  i=1 rj = ωj
for the cloud server and for selecting the CPU, memory, rji ≥ 0 (7)

storage and bandwidth resources required to run the cloud RESi − Ri ≥ 0
computing system, and the CUs apply to use the required where 1 ≤ i ≤ m and 1 ≤ j ≤ n.
resources. Because the resources are shared by CUs, there Furthermore, Formula 6 can be re-written as follows:
is a competition between multiple CUs. Moreover, there is ∑m T
also a cost game between the CUs and the CSP. T CΦ = Ri · (P (Ri )) (8)
i=1

According to Formula 2, we have

3.1 Game between multiple CUs Fi (Ri ) = Ri · (P (Ri ))T
∑n T
In CORP P , a user Uj adjusts its strategy selfishly; max- = j=1 rji · (P (Ri ))
imizing the utility function −U CΦ (j) minimizes its own ∑n
= j=1 (cpuij · Picpu + memij · Pimem + stgeij · Pistge + b
costs. However, because multiple CUs share cloud com- (9)
puting resources, this strategy increases the costs U CΦ (k) Which leads to the following theorem.
of other Uk (k ̸= j). Because CUs are selfish, each CU ∑n T
Theorem 1: Fi (Ri ) = j=1 rji · (P (Ri )) is a convex
tries to minimize its own purchasing costs; thus, the CUs function on Rn .
continually change their strategies until they can no longer Proof :
reduce the cost. This dynamic adjustment continues until Let x, y ∈ Rn and α ∈ [0, 1], then we have
the system reaches a stable state, i.e., no CU can reduce
costs by changing its strategy unilaterally. This stable state
is known as a Nash equilibrium. F
∑i [α · rxi + (1 − α) · ryi ]
n
= [α · rxi + (1 − α) · ryi ] · (P (Ri ))T
Definition 1: A feasible global resource provisioning ∑x,y=1
n T ∑n T
x=1 α · rx · (P (Ri )) + y=1 (1 − α) · ry · (P (Ri ))
i i
policy Φ = (ϕ1 , ϕ2 , ..., ϕn )T can be considered as having =
achieved Nash equilibrium when the following condition is = αFi (rxi ) + (1 − α)Fi (ryi ),
met; this provisioning policy is named the Nash Provision- Then, according to the definition of a convex function,
ing policy. we have proved this theorem.
′
If Uj ∈ U , and for any other Φ , we have U CΦ (j) ≤ Therefore, the above nonlinear optimization problem is
U CΦ′ (j), and U CΦ (k) = U CΦ′ (k) (k ̸= j). a convex programming problem, and any local optimal
In the Nash equilibrium state, no CU will change its cur- solution is a global optimal solution. Therefore, Formula
rent policy. However, although Nash equilibrium represents 8 can be further rewritten as follows:
a stable state of the system, the total costs T CΦ may not ∑m
be optimal. To explore the best possible performance that T CΦ = Fk (Rk ). (10)
i=1
2169-3536 (c) 2017 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
 This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2017.2778145, IEEE Access
IEEE ACCESS
Fig. 2. Game relationship in CORPP
The representation method and the constraint conditions
of T CΦ are similar to the formula for solving the selfish
routing [9]. One needs only to make the corresponding
change: replacing the ”path” in selfish routing in [9] with
”cloud users” in CORP P , and using ”physical resources”
in CORP P to represent the ”side” in [9]. Finally, to obtain
the ”global resources provisioning policy” in CORP P ,
we need to find the ”flow” in [9]. Then, the equivalence
between CORP P and selfish routing can be derived from
their respective optimization formulas. When T CΦ uses this
equivalence, based on T heorem 3.2 in [9], we obtain the
following theorem.
Theorem 2: When the various resource cost functions of
(H, F, Φ) are continuously non-decreasing, CORP P can
reach the Nash equilibrium. Furthermore, the Nash provi-
sioning policy of CORP P involves solving the following
optimization problem to minimize T CΦ under Constraint
7, in which the server cost function is
∫ cpu ∫ memi mem
Fi (Ri ) = Ri · ( 0 i ficpu (t)dt, 0 fi (t)dt,
∫ stgei stge ∫ bwi bw
0
f i (t)dt, 0
f i (t)dt),
∫ cpui cpu (11)
where 0 fi (t)dt represents the functional relation
between the price and the time of the CPU on server Hi .
The other three integral functions are similar.
3.2 Game between CUs and CSP
To overcome the local optimal in Nash equilibrium for
arbitrary cost functions, we consider a variant of the Nash
equilibrium in CORP P , i.e., the Stackelberg equilibrium.
In this case, the game includes two types of players: a
leader, whose goal is to optimize the utility of the whole
system, and a group of participants, who are selfish; each
type attempts to maximize its own utility. The leader cannot
directly determine the participants’ strategies; however, the
leader can influence the followers’ strategies through his or
her own strategy. Therefore, the leader wants to choose a
strategy that causes the system to achieve an approximately
optimal solution. Consequently, CORP P can be modeled
after Stackelberg equilibrium, where the CSP plays the
leader role and tries to maximize the system utility, while
2169-3536 (c) 2017 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
4
the CUs can be seen as the participants, who selfishly
attempt to reduce their own purchasing costs.
1 2 m T
We use ΦL = (rL , rL , ..., rL ) to represent the resource
provisioning policy of the CSP (i.e., the leader’s policy in
i
the system), where rL represents the CPU, memory, stor-
age, and bandwidth resources used by the leader on server
Hi . The total amount ∑m ofi resources used by the leader is
defined as ωL = i=1 rL . The Stackelberg global resource
provisioning policy is defined as (ϕ1 , ϕ2 , ..., ϕn , ϕL )T , the
total amount
∑n of resources used on server Hi is RiS =
rL + j=1 rj , which must meet RESi − Ri s ≥ 0.
i i
Therefore, each CU can see that the resources on the
server Hi are reduced due to the leader’s use, which affects
their equilibrium. The leader’s ωL can be assigned to any
task that he controls, such as user task scheduling and
management, infrastructure management, and so on.
A Stackelberg game achieves equilibrium in the fol-
lowing circumstance: the leader selects a strategy and
the followers cannot unilaterally change their strategies to
reduce their costs. The definition of Stackelberg equilibrium
in SCORP P is as follows.
Definition: For SCORPP (H, F, Φ, ΦL ), the Stack-
elberg global resource provisioning policy ΦS =
(ϕ1 , ϕ2 , ..., ϕn , ϕL )T , if Φ = (ϕ1 , ϕ2 , ..., ϕn )T is the
Nash equilibrium for CORP P (H, F N , Φ), where F N =
{FiN |FiN (x) = Fi (x + rL i
)}, then ΦS achieves Stackelberg
equilibrium. The complete provisioning policy is called the
Nash-Stackelberg provisioning policy.
When the cost function assigned by the Stackelberg
policy is the lowest, then the resource provisioning policy
is optimal. For best performance, Stackelberg equilibrium
relies on the leader’s strategy; in other words, the leader’s
strategy also determines the system’s performance. In this
paper, we consider two kinds of leader strategies: the Aloof
strategy and the Least Cost First (LCF) strategy [10].
Aloof strategy: In this strategy, the CSP ignores the
CUs’ strategies; instead, it tries to find a provisioning
strategy for ΦL in which ΦL is optimal for CORP P
(H, F, ΦL ), where ΦL = {ωL }. In the Aloof strategy,
the leader does not consider the followers’ strategies; he
determines the resource provisioning policy independently.
LCF strategy: In the LCF strategy, the leader uses the
resources that other clients tend to reject: those whose
 This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
IEEE ACCESS
prices are higher. The following algorithm is used to find
the LCF strategy.
leftmargin=*,labelsep=3mm
1) The leader first solves Formula 6 to obtain the opti-
mal provisioning policy Φ∗ of CORP P (H, F, Φ∗ ),
recorded as Φ∗ = (ω1 , ω2 , ..., ωn , ωL )T . This allo-
cation is not necessarily balanced for CUs; that is,
some Uj may reduce their costs by changing their
strategies.
2) Sort the resources to make F1 (Φ∗ (1))
F2 (Φ∗ (2)) ≤ ... ∑ ≤ Fm (Φ∗ (m)) and find the
smallest i∗ , so that i=i∗ Φ∗ (i) ≤ ωL . Here, Φ∗ (i)
m the total cost based on combinatorial pricing. But as the
represents the resources that the optimal provisioning
policy Φ∗ allocates on server Hi .
3) The leader’s strategy ΦSL ∗ = (rL 1 2
, rL m T
, ..., rL ) is
 ∗as follows:
determined
 Φ (i),∑ i > i∗
ωL − k=k∗ Φ (k), k = k ∗
∗
i m
rL =

0, k < k∗
In the LCF strategy, the leader adopts the (m − k ∗ )
servers with the highest costs, which induces the followers
to achieve Nash equilibrium. According to T heorem 4.2 in
[10], the following conclusions can be obtained. When ΦSL
is the LCF strategy of CORP P (H, F, Φ)’s leader, which
includes one method that provides ΦSL for the followers’
Nash and another one providing Φ∗ for the CORP P
(H, F, Φ)’s optimum, then CT(ΦSF ,ΦSL ) ≤ α1 CTΦ∗ , where
α represents the proportion that the leader’s tasks account
for of the entire system task. This indicates that the worst
case for the LCF policy is α1 times the optimal provisioning
policy.
4 E XPERIMENT AND RESULTS
4.1 Experiment configuration
The proposed CORP P algorithm is implemented on the
CloudSim 3.0.3 platform [11]. The experiment simulates
four types of virtual machines commonly used by Amazon
EC2, namely, H-VM, E-VM, S-VM and M-VM, whose
configurations are shown in Table 1. The cloud data center
has 100 servers; 50 are HP ProLiant ML110 G4s, and the
others are G5s. The G4s are configured with a Xeon-3040
@ 1860 MHz, 2 cores, and 4 GB of RAM, while the G5s
contain a Xeon-3075 @ 2660 MHz, 2 cores, and 4 GB of
RAM.
Our experiments use two types of cloud tasks: CPU-
intensive tasks and PlanetLab tasks that can be considered
as network I/O-intensive tasks [12,13]. The experimental
simulation cycle is 24 h; for all cloud tasks, the MIPS range
is between 2.0 ∗ 107 and 4.0 ∗ 107 .
4.2 Results analysis
The experiment first compares the difference between a
CPU-pricing mechanism and the combinatorial pricing
mechanism with different types of virtual machines and
different quantities of CUs when running either CPU in-
tensive tasks or network I/O-intensive tasks. When the
ratio between the CPU-pricing result and the combinatorial
2169-3536 (c) 2017 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
10.1109/ACCESS.2017.2778145, IEEE Access
5
pricing result is sufficiently high (which we define as 60%),
we say the CPU-pricing mechanism is sufficient because
CPU pricing plays a prime role, and the single-resource
pricing mechanism is much simpler than the combinatorial
pricing mechanism. The experiments are run several times
and the average is taken as the result.
Figure 3 shows the total costs of the two pricing mecha-
nisms when employed with different numbers of users and
four types of virtual machines running CPU-intensive tasks.
≤ When using M-VM with the lower CPU configuration, the
total cost based on CPU pricing is approximately 42% of
CPU configuration in the virtual machines improves (i.e.,
when using S-VM, E-VM or H-VM), the proportion of
the total costs based on the CPU to the total costs based
on combinatorial pricing gradually increases, from 60%, to
73%, and finally, to 78%, respectively. This result indicates
that when the cloud task is using more powerful CPU
configurations such as H-VM, it is reasonable to use the
CPU-pricing mechanism as the pricing method. In contrast,
when using a less-powerful CPU configuration such as M-
VM, the prices under this single-resource mechanism are
less than half the total costs generated by the combinatorial
pricing. This result shows that CPU cannot be the only
condition considered by a pricing strategy.
Similar scenarios occur in Figure 4, in which network
I/O-intensive tasks are being executed. The ratios between
the CPU-pricing results and the combinatorial pricing re-
sults are 35%, 47%, 51%, and 40% when using M-VM,
S-VM, E-VM, and H-VM configurations, respectively. This
result indicates that the single CPU-pricing mechanism
is completely unsuitable for pricing network I/O-intensive
tasks.
From the above analysis, the CPU-pricing mechanism
considers only one resource; consequently, it may be suit-
able for CPU-intensive tasks, but cannot be applied to other
types of tasks, such as I/O-intensive tasks. Therefore, the
multi-dimensional resource combinatorial pricing mecha-
nism based on the CPU, memory, storage and network
bandwidth and proposed in this paper is more suitable for
pricing cloud computing resources.
For our combinatorial pricing mechanism experiments,
we executed randomly generated tasks and PlanetLab cloud
tasks and then compared the total costs using four resource
provisioning policies: the optimal provisioning policy, Nash
provisioning policy, Nash-Stackelberg provisioning poli-
cy and Round-Robin policy. Moreover, we compared the
highest income obtained by a single cloud server under
these four policies. Among these four resource provisioning
policies, the first three are based on game theory, where the
optimal provisioning policy is theoretically optimal, while
the leader in the Nash-Stackelberg provisioning policy uses
the LCF strategy, with α = 0.1. The last policy, Round-
Robin, is not a game-based policy.
Figure 5 and Figure 6 respectively show the total costs
and highest income of a single server with randomly
generated tasks generated by four different provisioning
policies under the combinatorial pricing mechanism. From
 This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2017.2778145, IEEE Access

IEEE ACCESS 6

TABLE 1
Four types of virtual machines used by Amazon EC2

VM types CPU (MIPS) Memory (GB) Storage (GB) Bandwidth (MB)

H-VM 2500 0.85 100 100
E-VM 2000 3.75 100 1000
S-VM 1000 1.7 100 1000
M-VM 500 0.633 100 100

Fig. 3. The total costs of CPU intensive tasks

Fig. 4. The total costs of I/O intensive tasks

Figure 5, the highest total cost is generated by the Round- resources and the selfish characteristics of CUs. In addition,
Robin provisioning policy, while the results from the Nash the Round-Robin provisioning policy does not consider the
and Nash-Stackelberg provisioning policies are both close games between the CUs, or the CUs and the CSP.
to the cost of the optimal provisioning policy. Compared to The results of the above experiments with PlanetLab
the costs of the Round-Robin provisioning policy, the other cloud tasks are shown in Figure 7 and Figure 8. From
three results average 22% lower. Figure 6) shows that the Figure 7, the total costs of Round-Robin provisioning
highest costs obtained by a single cloud server are very policy are much higher than those of the other three
close to each other among the last three policies, but under policies; in the worst case, the Round-Robin provisioning
the Round-Robin policy, the results can be 2 times those policy result is approximately 2.6 times as large as the
of the other three policies. This result shows that in the optimal provisioning policy result. However, the total costs
Round-Robin provisioning policy, CUs cannot obtain the obtained by Nash provisioning policy and Nash-Stackelberg
resources at a lower price on this cloud server, resulting in provisioning policy are close to the optimal provisioning
an increase in the purchasing cost. This result may occur policy result, and they average 54% below the Round-Robin
because of the pricing differences of heterogeneous cloud provisioning policy costs. Similar to Figure 6, Figure 8
2169-3536 (c) 2017 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
 This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2017.2778145, IEEE Access

IEEE ACCESS 7

Fig. 5. Total costs with randomly generated tasks

Fig. 6. Highest income of a single server with randomly generated tasks

shows the highest income from a single cloud server under
different resource provisioning policies and number of CUs
when running PlanetLab cloud tasks. These results show
that the proposed provisioning policy based on a multi-
dimensional pricing mechanism is also applicable to real
tasks.
5 C ONCLUSION
In this study, we first designed a multi-dimensional resource
pricing mechanism for heterogeneous cloud computing sys-
tems that overcomes the drawback of single resource pric-
ing mechanisms, which are unsuitable for the diversity of
cloud computing tasks. Then, we proposed a cost-optimized
resource provisioning policy (CORP P ) based on game
theory, which considers interactions between the CUs and
between the CUs and the CSP to avoid unnecessarily high
costs. In these games, CORP P simultaneously considers
the existence of Nash equilibrium between the CUs, and
Stackelberg equilibrium between the CUs and the CSP.
2169-3536 (c) 2017 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
The experimental results showed multi-dimensional re-
source combinatorial pricing mechanism is more suitable
for describing a real cloud environment executing various
types of cloud tasks. For both randomly generated cloud
tasks or actual PlanetLab cloud tasks, CORPP reduced
the total costs of all CUs efficiently compared to the
existing Round-Robin strategy. Future work should focus
on strategies that include games between multiple CSPs or
on other heterogeneous cloud data centers objectives, such
as saving energy.
ACKNOWLEDGMENT
This work is supported by Doctoral Research Foundation
of Liaoning under grant No. 20170520306, Social Sci-
ence Planning Foundation of Liaoning under grant No.
L15AGL005 and National Science Foundation of China
under grant No. 61771090.
 This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2017.2778145, IEEE Access

IEEE ACCESS 8

Fig. 7. Total costs with PlanetLab tasks

Fig. 8. Highest income of a single cloud server with PlanetLab tasks

R EFERENCES [8] Q. Zhang, L. T. Yang, X. Liu, Z. Chen, and P. Li, ”A Tucker

[1] Q. Zhang, L. T. Yang, Z. Chen, and P. Li, ”PPHOPCM: Privacy-
preserving High-order Possibilistic c-Means Algorithm for Big Data
Clustering with Cloud Computing,” IEEE Transactions on Big Data,
2017, DOI: DOI: 10.1109/TBDATA.2017.2701816.
[2] Q. Zhang, L. T. Yang, Z. Chen, P. Li, and M. J. Deen, ”Privacy-
preserving Double-projection Deep Computation Model with Crowd-
sourcing on Cloud for Big Data Feature Learning,” IEEE Internet of
Things Journal, 2017, DOI: 10.1109/JIOT.2017.2732735.
[3] P. Li, Z. Chen, L. T. Yang, Q. Zhang, and M. J. Deen, ”Deep
Convolutional Computation Model for Feature Learning on Big Data
in Internet of Things,” IEEE Transactions on Industrial Informatics,
2017, DOI: 10.1109/TII.2017.2739340.
[4] H. Li, C. Wu, Z. Li, and F. C. M. Lau, ”Profit-maximizing Virtual
Machine Trading in a Federation of Selfish Clouds,” in Proceedings of
IEEE International Conference on Computer Communications, 2013,
pp. 25-29.
[5] I. Goiri, J. Guitart, and J. Torres, ”Economic Model of a Cloud
Provider Operating in a Federated Cloud,” Information Systems Fron-
tiers, vol. 14, no. 4, pp. 827-843, 2012.
[6] A. N. Toosi, R. N. Calheiros, R. K. Thulasiram, and R. Buyya,
”Resource Provisioning Policies to Increase IaaS Provider’s Profit in a
Federated Cloud Environment,” in Proceedings of IEEE International
Conference on High Performance Computing and Communications,
2011, pp. 279-287.
[7] M. A. Arfeen, K. Pawlikowski, and A. Willig, ”A Framework for
Resource Allocation Strategies in Cloud Computing Environment.” in
Proceedings of IEEE Computer Software and Applications Conference
Workshops, 2011, pp. 261-266.
2169-3536 (c) 2017 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Deep Computation Model for Mobile Multimedia Feature Learning,”
ACM Transactions on Multimedia Computing, Communications, and
Applications, vol. 13, no. 3s, pp. 39:1-39:18, 2017.
[9] T. Roughgarden and E. Tardos, ”How Bad is Selfish Routing?,”
Journal of the ACM, vol. 49, no. 2, pp. 236-259, 2002.
[10] T. Roughgarden , ”Stackelberg Scheduling Strategies,” SIAM Journal
on Computing, vol. 33, no. 2, pp. 332-350, 2004.
[11] R. N. Calheiros, R. Ranjan, A. Beloglazov, A. F. De Rose Cesar,
and R. Buyya, ”CloudSim: a Toolkit for Modeling and Simulation of
Cloud Computing Environments and Evaluation of Resource Provi-
sioning Algorithms,” Software: Practice and Experience, vol. 41, no.
1, pp. 23-50, 2011.
[12] Q. Zhang, L. T. Yang, Z. Chen, and F. Xia, ”A High-order Possibilis-
tic C-means Algorithm for Clustering Incomplete Multimedia Data,”
IEEE Systems Journal, vol. 11, no. 4, pp. 2160-2169, 2017.
[13] H. Zhou, Victor C. M. Leung, C. Zhu, S. Xu, and J. Fan, ”Predicting
Temporal Social Contact Patterns for Data Forwarding in Opportunis-
tic Mobile Networks,” IEEE Transactions on Vehicular Technology,
vol. 66, no. 11, pp. 10372-10383, 2017.
 This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2017.2778145, IEEE Access

IEEE ACCESS 9

Xin Chen received his B.E. degree in com-

puter science from Dalian University of Tech-
nology, China, in 2005 and Ph. D degree
in information science from Poznan Univer-
sity of Technology, Poland, in 2014. He is
currently an associate professor of school
of electronic and information engineering in
Liaoning University of Technology, China. His
research interests include combination opti-
mization, especially scheduling problem and
algorithm design.

Feng Ding received his B.E. degree in Power

Engineering from Dalian University of Tech-
nology, China, in 2002 and Ph. D degree
in Management Science and Engineering
from Dalian University of Technology, China,
in 2009. He is currently an associate pro-
fessor in School of software Technology in
Dalian University of Technology, China. His
research interests include e-government, e-
commerce, decision support, and informa-
tion security.

Tiantian Zhang received her B.E. degree in

software engineering from Dalian University
of Technology, China, in 2017 and she is cur-
rently a master student of School of software
technology in Dalian University of Technol-
ogy, China. Her research interests include
scheduling problem and algorithm design.

Gang Hou received his B.E. degree in com-

puter science from Dalian University of Tech-
nology, China, in 2005 and Ph. D degree in
software engineering from Dalian University
of Technology, China, in 2017. He is currently
an assistant professor of school of software
technology in Dalian University of Technol-
ogy, China. His research interests include
formal methods, model checking and trust-
worthy software.

Lan Lan received her B.E. degree in in-

ternational trade from Victoria University of
Technology, Australia, in 2002 and Master
degree in industrial relations and human re-
source management from University of Syd-
ney, Australia, in 2006. She is currently an
assistant professor of International School
in Shenyang University, China. Her research
interests include cloud computing and re-
source management.

2169-3536 (c) 2017 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See
http://www.ieee.org/publications_standards/publications/rights/index.html for more information.